2023-09-22T20:47:19.116 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-22T20:47:19.116 DEBUG:teuthology.run:Teuthology command: teuthology --name mkogan-2023-09-22_17:38:01-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi} --verbose --owner scheduled_mkogan@teuthology --archive /home/teuthworker/archive/mkogan-2023-09-22_17:38:01-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7401166 -- /home/teuthworker/archive/mkogan-2023-09-22_17:38:01-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7401166/orig.config.yaml 2023-09-22T20:47:19.152 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-22T20:47:19.234 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mkogan-2023-09-22_17:38:01-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7401166 branch: rgw-wip-t62141-valg-supp_i008 description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi} email: mkogan@redhat.com first_in_suite: false job_id: '7401166' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: mkogan-2023-09-22_17:38:01-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: rgw-wip-t62141-valg-supp_i008 ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: 03616b4330080cadf20be440d92e33ca90de254a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 03616b4330080cadf20be440d92e33ca90de254a rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: a zones: - endpoints: - c1.client.0 is_default: true is_master: true name: a1 - endpoints: - c1.client.1 name: a2 - enabled_features: - resharding endpoints: - c2.client.0 is_default: true name: b zones: - endpoints: - c2.client.0 is_default: true is_master: true name: b1 - endpoints: - c2.client.1 name: b2 rgw-multisite-tests: args: - tests.py workunit: branch: wip-qa-rgw-verify-ubuntu-003 sha1: 691e03976a4d019c58bad99761055f3266556055 owner: scheduled_mkogan@teuthology priority: 150 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 1924 sha1: 03616b4330080cadf20be440d92e33ca90de254a sleep_before_teardown: 0 suite: rgw suite_branch: wip-qa-rgw-verify-ubuntu-003 suite_path: /home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa suite_relpath: qa suite_repo: https://github.com/mkogan1/ceph.git suite_sha1: 691e03976a4d019c58bad99761055f3266556055 targets: smithi160.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJc+XzPRS2TrfwsLPPiLo/tNCa2RGt6AcCR/nSya29UpF+GTgm+TxeVeHmZSk/cnLRECsVqSxIMEtf26RoWCMWI= smithi204.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEqnq9KNsYblkwRNcJhKaEyCeQLeszT9O0Xr2v9A/EES2mM7cdfy9fws30xx5mDF1ikl+kZ3V+/mYuCmhT9Lpvw= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-22_17:38:01 tube: smithi user: mkogan verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-22T20:47:19.234 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa; will attempt to use it 2023-09-22T20:47:19.235 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks 2023-09-22T20:47:19.235 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-22T20:47:19.236 INFO:teuthology.task.internal:Checking packages... 2023-09-22T20:47:19.258 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '03616b4330080cadf20be440d92e33ca90de254a' 2023-09-22T20:47:19.258 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-22T20:47:19.258 INFO:teuthology.packaging:ref: None 2023-09-22T20:47:19.258 INFO:teuthology.packaging:tag: None 2023-09-22T20:47:19.258 INFO:teuthology.packaging:branch: rgw-wip-t62141-valg-supp_i008 2023-09-22T20:47:19.258 INFO:teuthology.packaging:sha1: 03616b4330080cadf20be440d92e33ca90de254a 2023-09-22T20:47:19.259 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=rgw-wip-t62141-valg-supp_i008 2023-09-22T20:47:19.397 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6192-g03616b43-1jammy 2023-09-22T20:47:19.421 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-22T20:47:19.426 INFO:teuthology.task.internal:no buildpackages task found 2023-09-22T20:47:19.426 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-22T20:47:19.432 INFO:teuthology.task.internal:Saving configuration 2023-09-22T20:47:19.445 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-22T20:47:19.451 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-22T20:47:19.477 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi160.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-09-22_17:38:01-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7401166', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-22 20:41:35.308549', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILGKRnEZEPXu1L06qPcubInZS3hTnkRpZokZJ2OptxvA'} 2023-09-22T20:47:19.501 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-09-22_17:38:01-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7401166', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-22 20:41:35.306891', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICeMh1LXsYkxvW4ToLo7uCPVehpCaE3cVDsC/xMz9YKp'} 2023-09-22T20:47:19.501 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-22T20:47:19.509 INFO:teuthology.task.internal:roles: ubuntu@smithi160.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2023-09-22T20:47:19.509 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2023-09-22T20:47:19.509 INFO:teuthology.run_tasks:Running task console_log... 2023-09-22T20:47:19.584 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f93778a3040>, signals=[15]) 2023-09-22T20:47:19.584 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-22T20:47:19.590 INFO:teuthology.task.internal:Opening connections... 2023-09-22T20:47:19.590 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi160.front.sepia.ceph.com 2023-09-22T20:47:19.592 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T20:47:19.670 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2023-09-22T20:47:19.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T20:47:19.744 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-22T20:47:19.751 DEBUG:teuthology.orchestra.run.smithi160:> uname -m 2023-09-22T20:47:19.756 INFO:teuthology.orchestra.run.smithi160.stdout:x86_64 2023-09-22T20:47:19.756 DEBUG:teuthology.orchestra.run.smithi160:> cat /etc/os-release 2023-09-22T20:47:19.801 INFO:teuthology.orchestra.run.smithi160.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-09-22T20:47:19.801 INFO:teuthology.orchestra.run.smithi160.stdout:NAME="Ubuntu" 2023-09-22T20:47:19.801 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_ID="22.04" 2023-09-22T20:47:19.801 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-09-22T20:47:19.801 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_CODENAME=jammy 2023-09-22T20:47:19.802 INFO:teuthology.orchestra.run.smithi160.stdout:ID=ubuntu 2023-09-22T20:47:19.802 INFO:teuthology.orchestra.run.smithi160.stdout:ID_LIKE=debian 2023-09-22T20:47:19.802 INFO:teuthology.orchestra.run.smithi160.stdout:HOME_URL="https://www.ubuntu.com/" 2023-09-22T20:47:19.802 INFO:teuthology.orchestra.run.smithi160.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-09-22T20:47:19.802 INFO:teuthology.orchestra.run.smithi160.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-09-22T20:47:19.802 INFO:teuthology.orchestra.run.smithi160.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-09-22T20:47:19.802 INFO:teuthology.orchestra.run.smithi160.stdout:UBUNTU_CODENAME=jammy 2023-09-22T20:47:19.803 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server 2023-09-22T20:47:19.828 DEBUG:teuthology.orchestra.run.smithi204:> uname -m 2023-09-22T20:47:19.833 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2023-09-22T20:47:19.833 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2023-09-22T20:47:19.878 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-09-22T20:47:19.879 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="Ubuntu" 2023-09-22T20:47:19.879 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="22.04" 2023-09-22T20:47:19.879 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-09-22T20:47:19.879 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_CODENAME=jammy 2023-09-22T20:47:19.879 INFO:teuthology.orchestra.run.smithi204.stdout:ID=ubuntu 2023-09-22T20:47:19.879 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE=debian 2023-09-22T20:47:19.879 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://www.ubuntu.com/" 2023-09-22T20:47:19.879 INFO:teuthology.orchestra.run.smithi204.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-09-22T20:47:19.879 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-09-22T20:47:19.879 INFO:teuthology.orchestra.run.smithi204.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-09-22T20:47:19.879 INFO:teuthology.orchestra.run.smithi204.stdout:UBUNTU_CODENAME=jammy 2023-09-22T20:47:19.880 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2023-09-22T20:47:19.903 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-22T20:47:19.912 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-22T20:47:19.917 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-22T20:47:19.917 DEBUG:teuthology.orchestra.run.smithi160:> test '!' -e /home/ubuntu/cephtest 2023-09-22T20:47:19.919 DEBUG:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2023-09-22T20:47:19.926 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-22T20:47:19.932 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-22T20:47:19.933 DEBUG:teuthology.orchestra.run.smithi160:> test -z $(ls -A /var/lib/ceph) 2023-09-22T20:47:19.967 DEBUG:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2023-09-22T20:47:19.974 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-22T20:47:20.028 INFO:teuthology.run_tasks:Running task kernel... 2023-09-22T20:47:20.041 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-22T20:47:20.042 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-22T20:47:20.042 DEBUG:teuthology.orchestra.run.smithi160:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-22T20:47:20.042 DEBUG:teuthology.orchestra.run.smithi204:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-22T20:47:20.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T20:47:20.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T20:47:20.047 DEBUG:teuthology.orchestra.run.smithi160:> uname -r 2023-09-22T20:47:20.048 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2023-09-22T20:47:20.094 INFO:teuthology.orchestra.run.smithi160.stdout:5.15.0-56-generic 2023-09-22T20:47:20.094 INFO:teuthology.task.kernel:Running kernel on smithi160: 5.15.0-56-generic 2023-09-22T20:47:20.095 DEBUG:teuthology.orchestra.run.smithi160:> sudo apt-get clean 2023-09-22T20:47:20.096 INFO:teuthology.orchestra.run.smithi204.stdout:5.15.0-56-generic 2023-09-22T20:47:20.096 INFO:teuthology.task.kernel:Running kernel on smithi204: 5.15.0-56-generic 2023-09-22T20:47:20.096 DEBUG:teuthology.orchestra.run.smithi204:> sudo apt-get clean 2023-09-22T20:47:20.216 DEBUG:teuthology.orchestra.run.smithi204:> sudo apt-get update 2023-09-22T20:47:20.235 DEBUG:teuthology.orchestra.run.smithi160:> sudo apt-get update 2023-09-22T20:47:20.370 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-09-22T20:47:20.370 INFO:teuthology.orchestra.run.smithi204.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-09-22T20:47:20.370 INFO:teuthology.orchestra.run.smithi204.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-09-22T20:47:20.371 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-09-22T20:47:20.373 INFO:teuthology.orchestra.run.smithi160.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-09-22T20:47:20.373 INFO:teuthology.orchestra.run.smithi160.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-09-22T20:47:20.441 INFO:teuthology.orchestra.run.smithi204.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-09-22T20:47:20.455 INFO:teuthology.orchestra.run.smithi160.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-09-22T20:47:20.594 INFO:teuthology.orchestra.run.smithi204.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [802 kB] 2023-09-22T20:47:20.603 INFO:teuthology.orchestra.run.smithi160.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [328 kB] 2023-09-22T20:47:20.676 INFO:teuthology.orchestra.run.smithi160.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,012 kB] 2023-09-22T20:47:20.676 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [493 kB] 2023-09-22T20:47:20.676 INFO:teuthology.orchestra.run.smithi160.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [802 kB] 2023-09-22T20:47:20.696 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [328 kB] 2023-09-22T20:47:20.702 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [169 kB] 2023-09-22T20:47:20.710 INFO:teuthology.orchestra.run.smithi160.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [169 kB] 2023-09-22T20:47:20.731 INFO:teuthology.orchestra.run.smithi204.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.3 kB] 2023-09-22T20:47:20.731 INFO:teuthology.orchestra.run.smithi204.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [30.0 kB] 2023-09-22T20:47:20.731 INFO:teuthology.orchestra.run.smithi204.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [882 kB] 2023-09-22T20:47:20.732 INFO:teuthology.orchestra.run.smithi160.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.3 kB] 2023-09-22T20:47:20.732 INFO:teuthology.orchestra.run.smithi160.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [30.0 kB] 2023-09-22T20:47:20.732 INFO:teuthology.orchestra.run.smithi160.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [882 kB] 2023-09-22T20:47:20.734 INFO:teuthology.orchestra.run.smithi204.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [142 kB] 2023-09-22T20:47:20.735 INFO:teuthology.orchestra.run.smithi204.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [536 B] 2023-09-22T20:47:20.735 INFO:teuthology.orchestra.run.smithi204.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [785 kB] 2023-09-22T20:47:20.738 INFO:teuthology.orchestra.run.smithi160.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [142 kB] 2023-09-22T20:47:20.740 INFO:teuthology.orchestra.run.smithi160.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [536 B] 2023-09-22T20:47:20.740 INFO:teuthology.orchestra.run.smithi160.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [785 kB] 2023-09-22T20:47:20.748 INFO:teuthology.orchestra.run.smithi160.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [559 kB] 2023-09-22T20:47:20.751 INFO:teuthology.orchestra.run.smithi204.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [559 kB] 2023-09-22T20:47:20.756 INFO:teuthology.orchestra.run.smithi204.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,012 kB] 2023-09-22T20:47:20.757 INFO:teuthology.orchestra.run.smithi160.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [143 kB] 2023-09-22T20:47:20.760 INFO:teuthology.orchestra.run.smithi160.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.7 kB] 2023-09-22T20:47:20.761 INFO:teuthology.orchestra.run.smithi204.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [143 kB] 2023-09-22T20:47:20.762 INFO:teuthology.orchestra.run.smithi204.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.7 kB] 2023-09-22T20:47:20.762 INFO:teuthology.orchestra.run.smithi204.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-09-22T20:47:20.763 INFO:teuthology.orchestra.run.smithi160.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-09-22T20:47:20.763 INFO:teuthology.orchestra.run.smithi204.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-09-22T20:47:20.766 INFO:teuthology.orchestra.run.smithi160.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-09-22T20:47:20.767 INFO:teuthology.orchestra.run.smithi160.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-09-22T20:47:20.767 INFO:teuthology.orchestra.run.smithi160.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [493 kB] 2023-09-22T20:47:20.771 INFO:teuthology.orchestra.run.smithi160.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [227 kB] 2023-09-22T20:47:20.774 INFO:teuthology.orchestra.run.smithi160.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [15.6 kB] 2023-09-22T20:47:20.775 INFO:teuthology.orchestra.run.smithi160.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [30.3 kB] 2023-09-22T20:47:20.776 INFO:teuthology.orchestra.run.smithi160.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [898 kB] 2023-09-22T20:47:20.783 INFO:teuthology.orchestra.run.smithi204.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-09-22T20:47:20.783 INFO:teuthology.orchestra.run.smithi204.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-09-22T20:47:20.786 INFO:teuthology.orchestra.run.smithi160.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-09-22T20:47:20.791 INFO:teuthology.orchestra.run.smithi204.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [227 kB] 2023-09-22T20:47:20.801 INFO:teuthology.orchestra.run.smithi204.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [15.6 kB] 2023-09-22T20:47:20.801 INFO:teuthology.orchestra.run.smithi204.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [898 kB] 2023-09-22T20:47:20.803 INFO:teuthology.orchestra.run.smithi160.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [145 kB] 2023-09-22T20:47:20.807 INFO:teuthology.orchestra.run.smithi160.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [536 B] 2023-09-22T20:47:20.807 INFO:teuthology.orchestra.run.smithi160.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [984 kB] 2023-09-22T20:47:20.826 INFO:teuthology.orchestra.run.smithi204.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [30.3 kB] 2023-09-22T20:47:20.826 INFO:teuthology.orchestra.run.smithi204.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [145 kB] 2023-09-22T20:47:20.829 INFO:teuthology.orchestra.run.smithi160.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [655 kB] 2023-09-22T20:47:20.830 INFO:teuthology.orchestra.run.smithi204.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [536 B] 2023-09-22T20:47:20.830 INFO:teuthology.orchestra.run.smithi204.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [655 kB] 2023-09-22T20:47:20.840 INFO:teuthology.orchestra.run.smithi160.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [215 kB] 2023-09-22T20:47:20.844 INFO:teuthology.orchestra.run.smithi160.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [21.8 kB] 2023-09-22T20:47:20.845 INFO:teuthology.orchestra.run.smithi160.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-09-22T20:47:20.845 INFO:teuthology.orchestra.run.smithi160.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-09-22T20:47:20.846 INFO:teuthology.orchestra.run.smithi160.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-09-22T20:47:20.848 INFO:teuthology.orchestra.run.smithi204.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [984 kB] 2023-09-22T20:47:20.861 INFO:teuthology.orchestra.run.smithi204.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [215 kB] 2023-09-22T20:47:20.864 INFO:teuthology.orchestra.run.smithi204.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [21.8 kB] 2023-09-22T20:47:20.865 INFO:teuthology.orchestra.run.smithi204.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-09-22T20:47:20.865 INFO:teuthology.orchestra.run.smithi204.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-09-22T20:47:20.866 INFO:teuthology.orchestra.run.smithi204.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-09-22T20:47:20.866 INFO:teuthology.orchestra.run.smithi204.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [476 B] 2023-09-22T20:47:20.867 INFO:teuthology.orchestra.run.smithi204.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.5 kB] 2023-09-22T20:47:20.867 INFO:teuthology.orchestra.run.smithi204.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.3 kB] 2023-09-22T20:47:20.871 INFO:teuthology.orchestra.run.smithi160.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [476 B] 2023-09-22T20:47:20.872 INFO:teuthology.orchestra.run.smithi160.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.3 kB] 2023-09-22T20:47:20.873 INFO:teuthology.orchestra.run.smithi160.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.5 kB] 2023-09-22T20:47:20.874 INFO:teuthology.orchestra.run.smithi160.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-09-22T20:47:20.874 INFO:teuthology.orchestra.run.smithi160.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-09-22T20:47:20.886 INFO:teuthology.orchestra.run.smithi204.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-09-22T20:47:20.887 INFO:teuthology.orchestra.run.smithi204.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-09-22T20:47:20.887 INFO:teuthology.orchestra.run.smithi204.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-09-22T20:47:20.888 INFO:teuthology.orchestra.run.smithi204.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-09-22T20:47:20.892 INFO:teuthology.orchestra.run.smithi160.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-09-22T20:47:20.893 INFO:teuthology.orchestra.run.smithi160.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-09-22T20:47:20.893 INFO:teuthology.orchestra.run.smithi160.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-09-22T20:47:20.894 INFO:teuthology.orchestra.run.smithi160.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [640 B] 2023-09-22T20:47:20.907 INFO:teuthology.orchestra.run.smithi204.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-09-22T20:47:20.907 INFO:teuthology.orchestra.run.smithi204.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [640 B] 2023-09-22T20:47:25.861 INFO:teuthology.orchestra.run.smithi160.stdout:Fetched 9,196 kB in 1s (6,737 kB/s) 2023-09-22T20:47:25.868 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 9,196 kB in 1s (6,625 kB/s) 2023-09-22T20:47:27.103 INFO:teuthology.orchestra.run.smithi160.stdout:Reading package lists... 2023-09-22T20:47:27.120 DEBUG:teuthology.orchestra.run.smithi160:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-22T20:47:27.132 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2023-09-22T20:47:27.150 DEBUG:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-22T20:47:27.170 INFO:teuthology.orchestra.run.smithi160.stdout:Reading package lists... 2023-09-22T20:47:27.201 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2023-09-22T20:47:27.333 INFO:teuthology.orchestra.run.smithi160.stdout:Building dependency tree... 2023-09-22T20:47:27.334 INFO:teuthology.orchestra.run.smithi160.stdout:Reading state information... 2023-09-22T20:47:27.364 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2023-09-22T20:47:27.364 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2023-09-22T20:47:27.473 INFO:teuthology.orchestra.run.smithi160.stdout:The following packages were automatically installed and are no longer required: 2023-09-22T20:47:27.473 INFO:teuthology.orchestra.run.smithi160.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-09-22T20:47:27.473 INFO:teuthology.orchestra.run.smithi160.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-09-22T20:47:27.474 INFO:teuthology.orchestra.run.smithi160.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-09-22T20:47:27.474 INFO:teuthology.orchestra.run.smithi160.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-09-22T20:47:27.474 INFO:teuthology.orchestra.run.smithi160.stdout: python2.7-minimal 2023-09-22T20:47:27.474 INFO:teuthology.orchestra.run.smithi160.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-22T20:47:27.475 INFO:teuthology.orchestra.run.smithi160.stdout:The following additional packages will be installed: 2023-09-22T20:47:27.475 INFO:teuthology.orchestra.run.smithi160.stdout: linux-generic linux-headers-5.15.0-84 linux-headers-5.15.0-84-generic 2023-09-22T20:47:27.475 INFO:teuthology.orchestra.run.smithi160.stdout: linux-headers-generic linux-image-5.15.0-84-generic 2023-09-22T20:47:27.475 INFO:teuthology.orchestra.run.smithi160.stdout: linux-modules-5.15.0-84-generic linux-modules-extra-5.15.0-84-generic 2023-09-22T20:47:27.476 INFO:teuthology.orchestra.run.smithi160.stdout:Suggested packages: 2023-09-22T20:47:27.476 INFO:teuthology.orchestra.run.smithi160.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-09-22T20:47:27.477 INFO:teuthology.orchestra.run.smithi160.stdout:Recommended packages: 2023-09-22T20:47:27.477 INFO:teuthology.orchestra.run.smithi160.stdout: thermald 2023-09-22T20:47:27.502 INFO:teuthology.orchestra.run.smithi204.stdout:The following packages were automatically installed and are no longer required: 2023-09-22T20:47:27.502 INFO:teuthology.orchestra.run.smithi204.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-09-22T20:47:27.502 INFO:teuthology.orchestra.run.smithi204.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-09-22T20:47:27.503 INFO:teuthology.orchestra.run.smithi204.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-09-22T20:47:27.503 INFO:teuthology.orchestra.run.smithi204.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-09-22T20:47:27.503 INFO:teuthology.orchestra.run.smithi204.stdout: python2.7-minimal 2023-09-22T20:47:27.503 INFO:teuthology.orchestra.run.smithi204.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-22T20:47:27.504 INFO:teuthology.orchestra.run.smithi204.stdout:The following additional packages will be installed: 2023-09-22T20:47:27.504 INFO:teuthology.orchestra.run.smithi204.stdout: linux-generic linux-headers-5.15.0-84 linux-headers-5.15.0-84-generic 2023-09-22T20:47:27.504 INFO:teuthology.orchestra.run.smithi204.stdout: linux-headers-generic linux-image-5.15.0-84-generic 2023-09-22T20:47:27.504 INFO:teuthology.orchestra.run.smithi204.stdout: linux-modules-5.15.0-84-generic linux-modules-extra-5.15.0-84-generic 2023-09-22T20:47:27.505 INFO:teuthology.orchestra.run.smithi204.stdout:Suggested packages: 2023-09-22T20:47:27.505 INFO:teuthology.orchestra.run.smithi204.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-09-22T20:47:27.505 INFO:teuthology.orchestra.run.smithi204.stdout:Recommended packages: 2023-09-22T20:47:27.505 INFO:teuthology.orchestra.run.smithi204.stdout: thermald 2023-09-22T20:47:27.526 INFO:teuthology.orchestra.run.smithi160.stdout:The following NEW packages will be installed: 2023-09-22T20:47:27.526 INFO:teuthology.orchestra.run.smithi160.stdout: linux-headers-5.15.0-84 linux-headers-5.15.0-84-generic 2023-09-22T20:47:27.526 INFO:teuthology.orchestra.run.smithi160.stdout: linux-image-5.15.0-84-generic linux-modules-5.15.0-84-generic 2023-09-22T20:47:27.527 INFO:teuthology.orchestra.run.smithi160.stdout: linux-modules-extra-5.15.0-84-generic 2023-09-22T20:47:27.528 INFO:teuthology.orchestra.run.smithi160.stdout:The following packages will be upgraded: 2023-09-22T20:47:27.528 INFO:teuthology.orchestra.run.smithi160.stdout: linux-generic linux-headers-generic linux-image-generic 2023-09-22T20:47:27.553 INFO:teuthology.orchestra.run.smithi204.stdout:The following NEW packages will be installed: 2023-09-22T20:47:27.553 INFO:teuthology.orchestra.run.smithi204.stdout: linux-headers-5.15.0-84 linux-headers-5.15.0-84-generic 2023-09-22T20:47:27.553 INFO:teuthology.orchestra.run.smithi204.stdout: linux-image-5.15.0-84-generic linux-modules-5.15.0-84-generic 2023-09-22T20:47:27.554 INFO:teuthology.orchestra.run.smithi204.stdout: linux-modules-extra-5.15.0-84-generic 2023-09-22T20:47:27.555 INFO:teuthology.orchestra.run.smithi204.stdout:The following packages will be upgraded: 2023-09-22T20:47:27.555 INFO:teuthology.orchestra.run.smithi204.stdout: linux-generic linux-headers-generic linux-image-generic 2023-09-22T20:47:27.582 INFO:teuthology.orchestra.run.smithi160.stdout:3 upgraded, 5 newly installed, 0 to remove and 265 not upgraded. 2023-09-22T20:47:27.582 INFO:teuthology.orchestra.run.smithi160.stdout:Need to get 113 MB of archives. 2023-09-22T20:47:27.582 INFO:teuthology.orchestra.run.smithi160.stdout:After this operation, 582 MB of additional disk space will be used. 2023-09-22T20:47:27.582 INFO:teuthology.orchestra.run.smithi160.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-84-generic amd64 5.15.0-84.93 [22.5 MB] 2023-09-22T20:47:27.733 INFO:teuthology.orchestra.run.smithi204.stdout:3 upgraded, 5 newly installed, 0 to remove and 265 not upgraded. 2023-09-22T20:47:27.733 INFO:teuthology.orchestra.run.smithi204.stdout:Need to get 113 MB of archives. 2023-09-22T20:47:27.734 INFO:teuthology.orchestra.run.smithi204.stdout:After this operation, 582 MB of additional disk space will be used. 2023-09-22T20:47:27.734 INFO:teuthology.orchestra.run.smithi204.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-84-generic amd64 5.15.0-84.93 [22.5 MB] 2023-09-22T20:47:28.017 INFO:teuthology.orchestra.run.smithi160.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-84-generic amd64 5.15.0-84.93 [11.5 MB] 2023-09-22T20:47:28.118 INFO:teuthology.orchestra.run.smithi160.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-84-generic amd64 5.15.0-84.93 [63.9 MB] 2023-09-22T20:47:28.970 INFO:teuthology.orchestra.run.smithi160.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.84.81 [1,698 B] 2023-09-22T20:47:28.970 INFO:teuthology.orchestra.run.smithi160.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.84.81 [2,476 B] 2023-09-22T20:47:28.970 INFO:teuthology.orchestra.run.smithi160.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-84 all 5.15.0-84.93 [12.3 MB] 2023-09-22T20:47:29.089 INFO:teuthology.orchestra.run.smithi160.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-84-generic amd64 5.15.0-84.93 [2,853 kB] 2023-09-22T20:47:29.098 INFO:teuthology.orchestra.run.smithi204.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-84-generic amd64 5.15.0-84.93 [11.5 MB] 2023-09-22T20:47:29.111 INFO:teuthology.orchestra.run.smithi160.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.84.81 [2,330 B] 2023-09-22T20:47:29.443 INFO:teuthology.orchestra.run.smithi204.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-84-generic amd64 5.15.0-84.93 [63.9 MB] 2023-09-22T20:47:29.482 INFO:teuthology.orchestra.run.smithi160.stdout:Fetched 113 MB in 2s (71.7 MB/s) 2023-09-22T20:47:29.652 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package linux-modules-5.15.0-84-generic. 2023-09-22T20:47:31.597 INFO:teuthology.orchestra.run.smithi204.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.84.81 [1,698 B] 2023-09-22T20:47:31.597 INFO:teuthology.orchestra.run.smithi204.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.84.81 [2,476 B] 2023-09-22T20:47:31.597 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-84 all 5.15.0-84.93 [12.3 MB] 2023-09-22T20:47:31.699 INFO:teuthology.orchestra.run.smithi160.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-09-22T20:47:31.704 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../0-linux-modules-5.15.0-84-generic_5.15.0-84.93_amd64.deb ... 2023-09-22T20:47:31.965 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-modules-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-22T20:47:32.473 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-84-generic amd64 5.15.0-84.93 [2,853 kB] 2023-09-22T20:47:32.607 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.84.81 [2,330 B] 2023-09-22T20:47:33.015 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 113 MB in 5s (22.4 MB/s) 2023-09-22T20:47:33.205 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-modules-5.15.0-84-generic. 2023-09-22T20:47:33.991 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package linux-image-5.15.0-84-generic. 2023-09-22T20:47:34.012 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../1-linux-image-5.15.0-84-generic_5.15.0-84.93_amd64.deb ... 2023-09-22T20:47:34.108 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-image-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-22T20:47:34.569 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-84-generic. 2023-09-22T20:47:34.590 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-84-generic_5.15.0-84.93_amd64.deb ... 2023-09-22T20:47:34.627 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-modules-extra-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-22T20:47:35.228 INFO:teuthology.orchestra.run.smithi204.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-09-22T20:47:35.232 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../0-linux-modules-5.15.0-84-generic_5.15.0-84.93_amd64.deb ... 2023-09-22T20:47:35.319 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-modules-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-22T20:47:37.712 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-image-5.15.0-84-generic. 2023-09-22T20:47:37.734 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../1-linux-image-5.15.0-84-generic_5.15.0-84.93_amd64.deb ... 2023-09-22T20:47:37.827 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-image-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-22T20:47:38.299 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-84-generic. 2023-09-22T20:47:38.319 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-84-generic_5.15.0-84.93_amd64.deb ... 2023-09-22T20:47:38.357 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-modules-extra-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-22T20:47:40.783 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../3-linux-generic_5.15.0.84.81_amd64.deb ... 2023-09-22T20:47:40.911 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-generic (5.15.0.84.81) over (5.15.0.56.54) ... 2023-09-22T20:47:41.226 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.84.81_amd64.deb ... 2023-09-22T20:47:41.331 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-image-generic (5.15.0.84.81) over (5.15.0.56.54) ... 2023-09-22T20:47:41.617 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package linux-headers-5.15.0-84. 2023-09-22T20:47:41.628 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../5-linux-headers-5.15.0-84_5.15.0-84.93_all.deb ... 2023-09-22T20:47:41.667 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-headers-5.15.0-84 (5.15.0-84.93) ... 2023-09-22T20:47:44.841 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../3-linux-generic_5.15.0.84.81_amd64.deb ... 2023-09-22T20:47:44.968 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-generic (5.15.0.84.81) over (5.15.0.56.54) ... 2023-09-22T20:47:45.251 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.84.81_amd64.deb ... 2023-09-22T20:47:45.347 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-image-generic (5.15.0.84.81) over (5.15.0.56.54) ... 2023-09-22T20:47:45.658 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-headers-5.15.0-84. 2023-09-22T20:47:45.680 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../5-linux-headers-5.15.0-84_5.15.0-84.93_all.deb ... 2023-09-22T20:47:45.725 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-headers-5.15.0-84 (5.15.0-84.93) ... 2023-09-22T20:47:46.980 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package linux-headers-5.15.0-84-generic. 2023-09-22T20:47:47.916 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../6-linux-headers-5.15.0-84-generic_5.15.0-84.93_amd64.deb ... 2023-09-22T20:47:47.919 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-headers-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-22T20:47:50.043 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.84.81_amd64.deb ... 2023-09-22T20:47:50.167 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-headers-generic (5.15.0.84.81) over (5.15.0.56.54) ... 2023-09-22T20:47:50.499 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-headers-5.15.0-84 (5.15.0-84.93) ... 2023-09-22T20:47:50.596 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-headers-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-22T20:47:50.714 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-headers-generic (5.15.0.84.81) ... 2023-09-22T20:47:50.867 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-image-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-22T20:47:51.440 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-headers-5.15.0-84-generic. 2023-09-22T20:47:51.477 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../6-linux-headers-5.15.0-84-generic_5.15.0-84.93_amd64.deb ... 2023-09-22T20:47:51.514 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-headers-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-22T20:47:52.186 INFO:teuthology.orchestra.run.smithi160.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-09-22T20:47:52.187 INFO:teuthology.orchestra.run.smithi160.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-09-22T20:47:52.187 INFO:teuthology.orchestra.run.smithi160.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-84-generic 2023-09-22T20:47:52.187 INFO:teuthology.orchestra.run.smithi160.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-84-generic 2023-09-22T20:47:52.344 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-modules-extra-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-22T20:47:53.754 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-modules-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-22T20:47:54.134 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.84.81_amd64.deb ... 2023-09-22T20:47:54.260 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-headers-generic (5.15.0.84.81) over (5.15.0.56.54) ... 2023-09-22T20:47:54.584 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-headers-5.15.0-84 (5.15.0-84.93) ... 2023-09-22T20:47:54.698 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-headers-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-22T20:47:54.799 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-headers-generic (5.15.0.84.81) ... 2023-09-22T20:47:54.928 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-image-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-22T20:47:55.104 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-image-generic (5.15.0.84.81) ... 2023-09-22T20:47:55.222 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-generic (5.15.0.84.81) ... 2023-09-22T20:47:55.349 INFO:teuthology.orchestra.run.smithi160.stdout:Processing triggers for linux-image-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-22T20:47:55.418 INFO:teuthology.orchestra.run.smithi160.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-09-22T20:47:55.418 INFO:teuthology.orchestra.run.smithi160.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-84-generic 2023-09-22T20:47:56.235 INFO:teuthology.orchestra.run.smithi204.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-09-22T20:47:56.235 INFO:teuthology.orchestra.run.smithi204.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-09-22T20:47:56.236 INFO:teuthology.orchestra.run.smithi204.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-84-generic 2023-09-22T20:47:56.236 INFO:teuthology.orchestra.run.smithi204.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-84-generic 2023-09-22T20:47:56.405 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-modules-extra-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-22T20:47:57.857 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-modules-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-22T20:47:59.266 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-image-generic (5.15.0.84.81) ... 2023-09-22T20:47:59.392 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-generic (5.15.0.84.81) ... 2023-09-22T20:47:59.519 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for linux-image-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-22T20:47:59.585 INFO:teuthology.orchestra.run.smithi204.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-09-22T20:47:59.585 INFO:teuthology.orchestra.run.smithi204.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-84-generic 2023-09-22T20:48:09.603 INFO:teuthology.orchestra.run.smithi160.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-09-22T20:48:09.604 INFO:teuthology.orchestra.run.smithi160.stdout:Sourcing file `/etc/default/grub' 2023-09-22T20:48:09.622 INFO:teuthology.orchestra.run.smithi160.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-22T20:48:09.639 INFO:teuthology.orchestra.run.smithi160.stdout:Generating grub configuration file ... 2023-09-22T20:48:09.897 INFO:teuthology.orchestra.run.smithi160.stdout:Found linux image: /boot/vmlinuz-5.15.0-84-generic 2023-09-22T20:48:09.910 INFO:teuthology.orchestra.run.smithi160.stdout:Found initrd image: /boot/initrd.img-5.15.0-84-generic 2023-09-22T20:48:10.229 INFO:teuthology.orchestra.run.smithi160.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-09-22T20:48:10.235 INFO:teuthology.orchestra.run.smithi160.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-09-22T20:48:10.303 INFO:teuthology.orchestra.run.smithi160.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-09-22T20:48:10.308 INFO:teuthology.orchestra.run.smithi160.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-09-22T20:48:10.468 INFO:teuthology.orchestra.run.smithi160.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-09-22T20:48:10.468 INFO:teuthology.orchestra.run.smithi160.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-09-22T20:48:10.468 INFO:teuthology.orchestra.run.smithi160.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-09-22T20:48:10.513 INFO:teuthology.orchestra.run.smithi160.stdout:done 2023-09-22T20:48:10.953 DEBUG:teuthology.orchestra.run.smithi160:> dpkg -s linux-image-generic 2023-09-22T20:48:10.970 INFO:teuthology.orchestra.run.smithi160.stdout:Package: linux-image-generic 2023-09-22T20:48:10.970 INFO:teuthology.orchestra.run.smithi160.stdout:Status: install ok installed 2023-09-22T20:48:10.970 INFO:teuthology.orchestra.run.smithi160.stdout:Priority: optional 2023-09-22T20:48:10.971 INFO:teuthology.orchestra.run.smithi160.stdout:Section: kernel 2023-09-22T20:48:10.971 INFO:teuthology.orchestra.run.smithi160.stdout:Installed-Size: 20 2023-09-22T20:48:10.971 INFO:teuthology.orchestra.run.smithi160.stdout:Maintainer: Ubuntu Kernel Team 2023-09-22T20:48:10.971 INFO:teuthology.orchestra.run.smithi160.stdout:Architecture: amd64 2023-09-22T20:48:10.971 INFO:teuthology.orchestra.run.smithi160.stdout:Source: linux-meta 2023-09-22T20:48:10.971 INFO:teuthology.orchestra.run.smithi160.stdout:Version: 5.15.0.84.81 2023-09-22T20:48:10.971 INFO:teuthology.orchestra.run.smithi160.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-84), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-09-22T20:48:10.971 INFO:teuthology.orchestra.run.smithi160.stdout:Depends: linux-image-5.15.0-84-generic, linux-modules-extra-5.15.0-84-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-22T20:48:10.971 INFO:teuthology.orchestra.run.smithi160.stdout:Recommends: thermald 2023-09-22T20:48:10.971 INFO:teuthology.orchestra.run.smithi160.stdout:Description: Generic Linux kernel image 2023-09-22T20:48:10.971 INFO:teuthology.orchestra.run.smithi160.stdout: This package will always depend on the latest generic kernel image 2023-09-22T20:48:10.971 INFO:teuthology.orchestra.run.smithi160.stdout: available. 2023-09-22T20:48:10.972 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-84-generic 2023-09-22T20:48:10.972 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-22T20:48:10.972 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-22T20:48:10.972 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2023-09-22T20:48:10.972 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi160.front.sepia.ceph.com, path=None, version=distro) 2023-09-22T20:48:10.972 DEBUG:teuthology.orchestra.run.smithi160:> sudo apt-get clean 2023-09-22T20:48:11.147 DEBUG:teuthology.orchestra.run.smithi160:> sudo apt-get update 2023-09-22T20:48:11.267 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-09-22T20:48:11.277 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-09-22T20:48:11.294 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-09-22T20:48:11.470 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-09-22T20:48:12.548 INFO:teuthology.orchestra.run.smithi160.stdout:Reading package lists... 2023-09-22T20:48:12.565 DEBUG:teuthology.orchestra.run.smithi160:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-22T20:48:12.624 INFO:teuthology.orchestra.run.smithi160.stdout:Reading package lists... 2023-09-22T20:48:12.804 INFO:teuthology.orchestra.run.smithi160.stdout:Building dependency tree... 2023-09-22T20:48:12.805 INFO:teuthology.orchestra.run.smithi160.stdout:Reading state information... 2023-09-22T20:48:12.964 INFO:teuthology.orchestra.run.smithi160.stdout:linux-image-generic is already the newest version (5.15.0.84.81). 2023-09-22T20:48:12.964 INFO:teuthology.orchestra.run.smithi160.stdout:The following packages were automatically installed and are no longer required: 2023-09-22T20:48:12.965 INFO:teuthology.orchestra.run.smithi160.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-09-22T20:48:12.965 INFO:teuthology.orchestra.run.smithi160.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-09-22T20:48:12.965 INFO:teuthology.orchestra.run.smithi160.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-09-22T20:48:12.965 INFO:teuthology.orchestra.run.smithi160.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-09-22T20:48:12.965 INFO:teuthology.orchestra.run.smithi160.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-09-22T20:48:12.965 INFO:teuthology.orchestra.run.smithi160.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-22T20:48:13.004 INFO:teuthology.orchestra.run.smithi160.stdout:0 upgraded, 0 newly installed, 0 to remove and 265 not upgraded. 2023-09-22T20:48:13.006 DEBUG:teuthology.orchestra.run.smithi160:> dpkg -s linux-image-generic 2023-09-22T20:48:13.068 INFO:teuthology.orchestra.run.smithi160.stdout:Package: linux-image-generic 2023-09-22T20:48:13.068 INFO:teuthology.orchestra.run.smithi160.stdout:Status: install ok installed 2023-09-22T20:48:13.069 INFO:teuthology.orchestra.run.smithi160.stdout:Priority: optional 2023-09-22T20:48:13.069 INFO:teuthology.orchestra.run.smithi160.stdout:Section: kernel 2023-09-22T20:48:13.069 INFO:teuthology.orchestra.run.smithi160.stdout:Installed-Size: 20 2023-09-22T20:48:13.069 INFO:teuthology.orchestra.run.smithi160.stdout:Maintainer: Ubuntu Kernel Team 2023-09-22T20:48:13.069 INFO:teuthology.orchestra.run.smithi160.stdout:Architecture: amd64 2023-09-22T20:48:13.069 INFO:teuthology.orchestra.run.smithi160.stdout:Source: linux-meta 2023-09-22T20:48:13.069 INFO:teuthology.orchestra.run.smithi160.stdout:Version: 5.15.0.84.81 2023-09-22T20:48:13.069 INFO:teuthology.orchestra.run.smithi160.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-84), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-09-22T20:48:13.069 INFO:teuthology.orchestra.run.smithi160.stdout:Depends: linux-image-5.15.0-84-generic, linux-modules-extra-5.15.0-84-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-22T20:48:13.069 INFO:teuthology.orchestra.run.smithi160.stdout:Recommends: thermald 2023-09-22T20:48:13.069 INFO:teuthology.orchestra.run.smithi160.stdout:Description: Generic Linux kernel image 2023-09-22T20:48:13.069 INFO:teuthology.orchestra.run.smithi160.stdout: This package will always depend on the latest generic kernel image 2023-09-22T20:48:13.070 INFO:teuthology.orchestra.run.smithi160.stdout: available. 2023-09-22T20:48:13.070 DEBUG:teuthology.orchestra.run.smithi160:> mktemp 2023-09-22T20:48:13.113 INFO:teuthology.orchestra.run.smithi160.stdout:/tmp/tmp.ee8kTA6V3N 2023-09-22T20:48:13.113 DEBUG:teuthology.orchestra.run.smithi160:> sudo cp /boot/grub/grub.cfg /tmp/tmp.ee8kTA6V3N 2023-09-22T20:48:13.169 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0666 /tmp/tmp.ee8kTA6V3N 2023-09-22T20:48:13.293 DEBUG:teuthology.orchestra.remote:smithi160:/tmp/tmp.ee8kTA6V3N is 10KB 2023-09-22T20:48:13.305 DEBUG:teuthology.orchestra.run.smithi160:> rm -fr /tmp/tmp.ee8kTA6V3N 2023-09-22T20:48:13.311 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-09-22T20:48:13.365 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T20:48:13.365 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-09-22T20:48:13.365 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-09-22T20:48:13.433 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-84-generic 2023-09-22T20:48:13.434 DEBUG:teuthology.orchestra.run.smithi160:> sudo update-grub 2023-09-22T20:48:13.776 INFO:teuthology.orchestra.run.smithi204.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-09-22T20:48:13.776 INFO:teuthology.orchestra.run.smithi204.stdout:Sourcing file `/etc/default/grub' 2023-09-22T20:48:13.791 INFO:teuthology.orchestra.run.smithi204.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-22T20:48:13.809 INFO:teuthology.orchestra.run.smithi204.stdout:Generating grub configuration file ... 2023-09-22T20:48:14.071 INFO:teuthology.orchestra.run.smithi204.stdout:Found linux image: /boot/vmlinuz-5.15.0-84-generic 2023-09-22T20:48:14.085 INFO:teuthology.orchestra.run.smithi204.stdout:Found initrd image: /boot/initrd.img-5.15.0-84-generic 2023-09-22T20:48:14.381 INFO:teuthology.orchestra.run.smithi204.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-09-22T20:48:14.386 INFO:teuthology.orchestra.run.smithi204.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-09-22T20:48:14.453 INFO:teuthology.orchestra.run.smithi204.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-09-22T20:48:14.458 INFO:teuthology.orchestra.run.smithi204.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-09-22T20:48:14.465 INFO:teuthology.orchestra.run.smithi160.stderr:Sourcing file `/etc/default/grub' 2023-09-22T20:48:14.467 INFO:teuthology.orchestra.run.smithi160.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-22T20:48:14.469 INFO:teuthology.orchestra.run.smithi160.stderr:Generating grub configuration file ... 2023-09-22T20:48:14.612 INFO:teuthology.orchestra.run.smithi204.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-09-22T20:48:14.612 INFO:teuthology.orchestra.run.smithi204.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-09-22T20:48:14.612 INFO:teuthology.orchestra.run.smithi204.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-09-22T20:48:14.645 INFO:teuthology.orchestra.run.smithi160.stderr:Found linux image: /boot/vmlinuz-5.15.0-84-generic 2023-09-22T20:48:14.651 INFO:teuthology.orchestra.run.smithi160.stderr:Found initrd image: /boot/initrd.img-5.15.0-84-generic 2023-09-22T20:48:14.656 INFO:teuthology.orchestra.run.smithi204.stdout:done 2023-09-22T20:48:14.873 INFO:teuthology.orchestra.run.smithi160.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-09-22T20:48:14.878 INFO:teuthology.orchestra.run.smithi160.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-09-22T20:48:14.919 INFO:teuthology.orchestra.run.smithi160.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-09-22T20:48:14.924 INFO:teuthology.orchestra.run.smithi160.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-09-22T20:48:15.035 INFO:teuthology.orchestra.run.smithi160.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-09-22T20:48:15.035 INFO:teuthology.orchestra.run.smithi160.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-09-22T20:48:15.035 INFO:teuthology.orchestra.run.smithi160.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-09-22T20:48:15.060 INFO:teuthology.orchestra.run.smithi160.stderr:done 2023-09-22T20:48:15.070 DEBUG:teuthology.orchestra.run.smithi160:> sudo shutdown -r now 2023-09-22T20:48:15.110 DEBUG:teuthology.orchestra.run.smithi204:> dpkg -s linux-image-generic 2023-09-22T20:48:15.129 INFO:teuthology.orchestra.run.smithi204.stdout:Package: linux-image-generic 2023-09-22T20:48:15.130 INFO:teuthology.orchestra.run.smithi204.stdout:Status: install ok installed 2023-09-22T20:48:15.130 INFO:teuthology.orchestra.run.smithi204.stdout:Priority: optional 2023-09-22T20:48:15.130 INFO:teuthology.orchestra.run.smithi204.stdout:Section: kernel 2023-09-22T20:48:15.130 INFO:teuthology.orchestra.run.smithi204.stdout:Installed-Size: 20 2023-09-22T20:48:15.130 INFO:teuthology.orchestra.run.smithi204.stdout:Maintainer: Ubuntu Kernel Team 2023-09-22T20:48:15.130 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: amd64 2023-09-22T20:48:15.130 INFO:teuthology.orchestra.run.smithi204.stdout:Source: linux-meta 2023-09-22T20:48:15.130 INFO:teuthology.orchestra.run.smithi204.stdout:Version: 5.15.0.84.81 2023-09-22T20:48:15.130 INFO:teuthology.orchestra.run.smithi204.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-84), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-09-22T20:48:15.131 INFO:teuthology.orchestra.run.smithi204.stdout:Depends: linux-image-5.15.0-84-generic, linux-modules-extra-5.15.0-84-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-22T20:48:15.131 INFO:teuthology.orchestra.run.smithi204.stdout:Recommends: thermald 2023-09-22T20:48:15.131 INFO:teuthology.orchestra.run.smithi204.stdout:Description: Generic Linux kernel image 2023-09-22T20:48:15.131 INFO:teuthology.orchestra.run.smithi204.stdout: This package will always depend on the latest generic kernel image 2023-09-22T20:48:15.131 INFO:teuthology.orchestra.run.smithi204.stdout: available. 2023-09-22T20:48:15.131 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-84-generic 2023-09-22T20:48:15.131 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-22T20:48:15.131 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-22T20:48:15.132 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2023-09-22T20:48:15.132 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi204.front.sepia.ceph.com, path=None, version=distro) 2023-09-22T20:48:15.132 DEBUG:teuthology.orchestra.run.smithi204:> sudo apt-get clean 2023-09-22T20:48:15.335 DEBUG:teuthology.orchestra.run.smithi204:> sudo apt-get update 2023-09-22T20:48:15.445 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-09-22T20:48:15.446 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-09-22T20:48:15.455 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-09-22T20:48:15.471 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-09-22T20:48:16.710 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2023-09-22T20:48:16.728 DEBUG:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-22T20:48:16.779 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2023-09-22T20:48:16.941 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2023-09-22T20:48:16.941 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2023-09-22T20:48:17.083 INFO:teuthology.orchestra.run.smithi204.stdout:linux-image-generic is already the newest version (5.15.0.84.81). 2023-09-22T20:48:17.083 INFO:teuthology.orchestra.run.smithi204.stdout:The following packages were automatically installed and are no longer required: 2023-09-22T20:48:17.083 INFO:teuthology.orchestra.run.smithi204.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-09-22T20:48:17.083 INFO:teuthology.orchestra.run.smithi204.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-09-22T20:48:17.083 INFO:teuthology.orchestra.run.smithi204.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-09-22T20:48:17.084 INFO:teuthology.orchestra.run.smithi204.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-09-22T20:48:17.084 INFO:teuthology.orchestra.run.smithi204.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-09-22T20:48:17.084 INFO:teuthology.orchestra.run.smithi204.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-22T20:48:17.120 INFO:teuthology.orchestra.run.smithi204.stdout:0 upgraded, 0 newly installed, 0 to remove and 265 not upgraded. 2023-09-22T20:48:17.122 DEBUG:teuthology.orchestra.run.smithi204:> dpkg -s linux-image-generic 2023-09-22T20:48:17.180 INFO:teuthology.orchestra.run.smithi204.stdout:Package: linux-image-generic 2023-09-22T20:48:17.180 INFO:teuthology.orchestra.run.smithi204.stdout:Status: install ok installed 2023-09-22T20:48:17.181 INFO:teuthology.orchestra.run.smithi204.stdout:Priority: optional 2023-09-22T20:48:17.181 INFO:teuthology.orchestra.run.smithi204.stdout:Section: kernel 2023-09-22T20:48:17.181 INFO:teuthology.orchestra.run.smithi204.stdout:Installed-Size: 20 2023-09-22T20:48:17.181 INFO:teuthology.orchestra.run.smithi204.stdout:Maintainer: Ubuntu Kernel Team 2023-09-22T20:48:17.181 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: amd64 2023-09-22T20:48:17.181 INFO:teuthology.orchestra.run.smithi204.stdout:Source: linux-meta 2023-09-22T20:48:17.181 INFO:teuthology.orchestra.run.smithi204.stdout:Version: 5.15.0.84.81 2023-09-22T20:48:17.181 INFO:teuthology.orchestra.run.smithi204.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-84), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-09-22T20:48:17.181 INFO:teuthology.orchestra.run.smithi204.stdout:Depends: linux-image-5.15.0-84-generic, linux-modules-extra-5.15.0-84-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-22T20:48:17.181 INFO:teuthology.orchestra.run.smithi204.stdout:Recommends: thermald 2023-09-22T20:48:17.181 INFO:teuthology.orchestra.run.smithi204.stdout:Description: Generic Linux kernel image 2023-09-22T20:48:17.181 INFO:teuthology.orchestra.run.smithi204.stdout: This package will always depend on the latest generic kernel image 2023-09-22T20:48:17.182 INFO:teuthology.orchestra.run.smithi204.stdout: available. 2023-09-22T20:48:17.182 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2023-09-22T20:48:17.226 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.CP7zSpWhUo 2023-09-22T20:48:17.226 DEBUG:teuthology.orchestra.run.smithi204:> sudo cp /boot/grub/grub.cfg /tmp/tmp.CP7zSpWhUo 2023-09-22T20:48:17.283 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.CP7zSpWhUo 2023-09-22T20:48:17.393 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.CP7zSpWhUo is 10KB 2023-09-22T20:48:17.406 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.CP7zSpWhUo 2023-09-22T20:48:17.411 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-09-22T20:48:17.466 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T20:48:17.466 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-09-22T20:48:17.466 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-09-22T20:48:17.530 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-84-generic 2023-09-22T20:48:17.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo update-grub 2023-09-22T20:48:18.672 INFO:teuthology.orchestra.run.smithi204.stderr:Sourcing file `/etc/default/grub' 2023-09-22T20:48:18.673 INFO:teuthology.orchestra.run.smithi204.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-22T20:48:18.676 INFO:teuthology.orchestra.run.smithi204.stderr:Generating grub configuration file ... 2023-09-22T20:48:18.848 INFO:teuthology.orchestra.run.smithi204.stderr:Found linux image: /boot/vmlinuz-5.15.0-84-generic 2023-09-22T20:48:18.854 INFO:teuthology.orchestra.run.smithi204.stderr:Found initrd image: /boot/initrd.img-5.15.0-84-generic 2023-09-22T20:48:19.074 INFO:teuthology.orchestra.run.smithi204.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-09-22T20:48:19.079 INFO:teuthology.orchestra.run.smithi204.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-09-22T20:48:19.118 INFO:teuthology.orchestra.run.smithi204.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-09-22T20:48:19.123 INFO:teuthology.orchestra.run.smithi204.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-09-22T20:48:19.279 INFO:teuthology.orchestra.run.smithi204.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-09-22T20:48:19.279 INFO:teuthology.orchestra.run.smithi204.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-09-22T20:48:19.279 INFO:teuthology.orchestra.run.smithi204.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-09-22T20:48:19.299 INFO:teuthology.orchestra.run.smithi204.stderr:done 2023-09-22T20:48:19.308 DEBUG:teuthology.orchestra.run.smithi204:> sudo shutdown -r now 2023-09-22T20:48:45.097 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2023-09-22T20:48:45.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-09-22T20:48:45.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T20:48:49.319 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2023-09-22T20:48:49.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2023-09-22T20:48:49.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T20:49:03.610 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2023-09-22T20:49:07.710 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2023-09-22T20:49:12.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-09-22T20:49:12.615 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T20:49:15.674 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2023-09-22T20:49:16.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2023-09-22T20:49:16.712 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T20:49:19.770 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2023-09-22T20:49:27.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-09-22T20:49:27.683 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T20:49:31.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2023-09-22T20:49:31.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T20:49:46.106 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2023-09-22T20:50:01.130 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-09-22T20:50:01.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T20:50:01.906 DEBUG:teuthology.orchestra.run.smithi160:> true 2023-09-22T20:50:02.925 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-09-22T20:50:02.926 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "5.15.0-84-generic"... 2023-09-22T20:50:02.926 DEBUG:teuthology.orchestra.run.smithi160:> uname -r 2023-09-22T20:50:02.931 INFO:teuthology.orchestra.run.smithi160.stdout:5.15.0-84-generic 2023-09-22T20:50:02.931 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-84-generic vs 5.15.0-84-generic 2023-09-22T20:50:02.931 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-22T20:50:02.931 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-09-22T20:50:03.258 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2023-09-22T20:50:03.932 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2023-09-22T20:50:03.932 DEBUG:teuthology.orchestra.run.smithi160:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-22T20:50:04.077 INFO:teuthology.orchestra.run.smithi160.stdout:ttyS1 2023-09-22T20:50:04.098 DEBUG:teuthology.parallel:result is None 2023-09-22T20:50:18.259 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2023-09-22T20:50:18.260 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T20:50:18.789 DEBUG:teuthology.orchestra.run.smithi204:> true 2023-09-22T20:50:19.336 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2023-09-22T20:50:19.336 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "5.15.0-84-generic"... 2023-09-22T20:50:19.336 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2023-09-22T20:50:19.381 INFO:teuthology.orchestra.run.smithi204.stdout:5.15.0-84-generic 2023-09-22T20:50:19.382 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-84-generic vs 5.15.0-84-generic 2023-09-22T20:50:19.382 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-22T20:50:19.382 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-09-22T20:50:20.383 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2023-09-22T20:50:20.383 DEBUG:teuthology.orchestra.run.smithi204:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-22T20:50:20.458 INFO:teuthology.orchestra.run.smithi204.stdout:ttyS1 2023-09-22T20:50:20.491 DEBUG:teuthology.parallel:result is None 2023-09-22T20:50:20.491 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-22T20:50:20.498 INFO:teuthology.task.internal:Creating test directory... 2023-09-22T20:50:20.498 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-22T20:50:20.501 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-22T20:50:20.507 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-22T20:50:20.515 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-22T20:50:20.522 INFO:teuthology.task.internal:Creating archive directory... 2023-09-22T20:50:20.522 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-22T20:50:20.549 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-22T20:50:20.612 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-22T20:50:20.620 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-22T20:50:20.620 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-22T20:50:20.623 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-22T20:50:20.643 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-22T20:50:20.649 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-22T20:50:20.652 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-22T20:50:20.657 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-22T20:50:20.659 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-22T20:50:20.666 INFO:teuthology.task.internal:Configuring sudo... 2023-09-22T20:50:20.666 DEBUG:teuthology.orchestra.run.smithi160:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-22T20:50:20.705 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-22T20:50:20.725 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-22T20:50:20.733 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-22T20:50:20.734 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-22T20:50:20.769 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-22T20:50:20.775 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-22T20:50:20.821 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-22T20:50:20.869 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T20:50:20.869 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-22T20:50:20.932 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-22T20:50:20.938 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-22T20:50:20.986 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T20:50:20.986 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-22T20:50:21.050 DEBUG:teuthology.orchestra.run.smithi160:> sudo service rsyslog restart 2023-09-22T20:50:21.054 DEBUG:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2023-09-22T20:50:21.161 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-22T20:50:21.170 INFO:teuthology.task.internal:Starting timer... 2023-09-22T20:50:21.170 INFO:teuthology.run_tasks:Running task pcp... 2023-09-22T20:50:21.187 INFO:teuthology.run_tasks:Running task selinux... 2023-09-22T20:50:21.201 DEBUG:teuthology.task.selinux:Excluding smithi160: OS 'ubuntu' does not support SELinux 2023-09-22T20:50:21.201 DEBUG:teuthology.task.selinux:Excluding smithi204: OS 'ubuntu' does not support SELinux 2023-09-22T20:50:21.201 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-22T20:50:21.201 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-09-22T20:50:21.201 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-22T20:50:21.201 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-22T20:50:21.224 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-09-22T20:50:21.229 INFO:teuthology.repo_utils:Fetching main from origin 2023-09-22T20:50:21.409 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-22T20:50:21.424 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-22T20:50:21.425 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi160.front.sepia.ceph.com,smithi204.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-22T20:57:57.569 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi160.front.sepia.ceph.com'), Remote(name='ubuntu@smithi204.front.sepia.ceph.com')] 2023-09-22T20:57:57.593 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-09-22T20:57:57.594 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T20:57:57.667 DEBUG:teuthology.orchestra.run.smithi160:> true 2023-09-22T20:57:58.649 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-09-22T20:57:58.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2023-09-22T20:57:58.650 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T20:57:58.725 DEBUG:teuthology.orchestra.run.smithi204:> true 2023-09-22T20:57:58.822 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2023-09-22T20:57:58.823 INFO:teuthology.run_tasks:Running task clock... 2023-09-22T20:57:58.837 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-22T20:57:58.837 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-22T20:57:58.838 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-22T20:57:58.840 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-22T20:57:58.841 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-22T20:57:58.869 INFO:teuthology.orchestra.run.smithi160.stdout:22 Sep 20:57:58 ntpd[27850]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-09-22T20:57:58.869 INFO:teuthology.orchestra.run.smithi160.stdout:22 Sep 20:57:58 ntpd[27850]: Command line: ntpd -gq 2023-09-22T20:57:58.869 INFO:teuthology.orchestra.run.smithi160.stdout:22 Sep 20:57:58 ntpd[27850]: ---------------------------------------------------- 2023-09-22T20:57:58.869 INFO:teuthology.orchestra.run.smithi160.stdout:22 Sep 20:57:58 ntpd[27850]: ntp-4 is maintained by Network Time Foundation, 2023-09-22T20:57:58.869 INFO:teuthology.orchestra.run.smithi160.stdout:22 Sep 20:57:58 ntpd[27850]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-09-22T20:57:58.870 INFO:teuthology.orchestra.run.smithi160.stdout:22 Sep 20:57:58 ntpd[27850]: corporation. Support and training for ntp-4 are 2023-09-22T20:57:58.870 INFO:teuthology.orchestra.run.smithi160.stdout:22 Sep 20:57:58 ntpd[27850]: available at https://www.nwtime.org/support 2023-09-22T20:57:58.870 INFO:teuthology.orchestra.run.smithi160.stdout:22 Sep 20:57:58 ntpd[27850]: ---------------------------------------------------- 2023-09-22T20:57:58.870 INFO:teuthology.orchestra.run.smithi160.stdout:22 Sep 20:57:58 ntpd[27850]: proto: precision = 0.063 usec (-24) 2023-09-22T20:57:58.870 INFO:teuthology.orchestra.run.smithi160.stdout:22 Sep 20:57:58 ntpd[27850]: basedate set to 2022-02-04 2023-09-22T20:57:58.871 INFO:teuthology.orchestra.run.smithi160.stdout:22 Sep 20:57:58 ntpd[27850]: gps base set to 2022-02-06 (week 2196) 2023-09-22T20:57:58.871 INFO:teuthology.orchestra.run.smithi160.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-22T20:57:58.871 INFO:teuthology.orchestra.run.smithi160.stdout:22 Sep 20:57:58 ntpd[27850]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-22T20:57:58.871 INFO:teuthology.orchestra.run.smithi160.stdout:22 Sep 20:57:58 ntpd[27850]: restrict ::: KOD does nothing without LIMITED. 2023-09-22T20:57:58.872 INFO:teuthology.orchestra.run.smithi160.stderr:restrict ::: KOD does nothing without LIMITED. 2023-09-22T20:57:58.873 INFO:teuthology.orchestra.run.smithi160.stdout:22 Sep 20:57:58 ntpd[27850]: Listen and drop on 0 v6wildcard [::]:123 2023-09-22T20:57:58.873 INFO:teuthology.orchestra.run.smithi160.stdout:22 Sep 20:57:58 ntpd[27850]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-09-22T20:57:58.873 INFO:teuthology.orchestra.run.smithi160.stdout:22 Sep 20:57:58 ntpd[27850]: Listen normally on 2 lo 127.0.0.1:123 2023-09-22T20:57:58.873 INFO:teuthology.orchestra.run.smithi160.stdout:22 Sep 20:57:58 ntpd[27850]: Listen normally on 3 enp3s0f1 172.21.15.160:123 2023-09-22T20:57:58.873 INFO:teuthology.orchestra.run.smithi160.stdout:22 Sep 20:57:58 ntpd[27850]: Listen normally on 4 lo [::1]:123 2023-09-22T20:57:58.873 INFO:teuthology.orchestra.run.smithi160.stdout:22 Sep 20:57:58 ntpd[27850]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f5b%5]:123 2023-09-22T20:57:58.874 INFO:teuthology.orchestra.run.smithi160.stdout:22 Sep 20:57:58 ntpd[27850]: Listening on routing socket on fd #22 for interface updates 2023-09-22T20:57:58.887 INFO:teuthology.orchestra.run.smithi204.stdout:22 Sep 20:57:58 ntpd[27817]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-09-22T20:57:58.888 INFO:teuthology.orchestra.run.smithi204.stdout:22 Sep 20:57:58 ntpd[27817]: Command line: ntpd -gq 2023-09-22T20:57:58.888 INFO:teuthology.orchestra.run.smithi204.stdout:22 Sep 20:57:58 ntpd[27817]: ---------------------------------------------------- 2023-09-22T20:57:58.888 INFO:teuthology.orchestra.run.smithi204.stdout:22 Sep 20:57:58 ntpd[27817]: ntp-4 is maintained by Network Time Foundation, 2023-09-22T20:57:58.888 INFO:teuthology.orchestra.run.smithi204.stdout:22 Sep 20:57:58 ntpd[27817]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-09-22T20:57:58.888 INFO:teuthology.orchestra.run.smithi204.stdout:22 Sep 20:57:58 ntpd[27817]: corporation. Support and training for ntp-4 are 2023-09-22T20:57:58.888 INFO:teuthology.orchestra.run.smithi204.stdout:22 Sep 20:57:58 ntpd[27817]: available at https://www.nwtime.org/support 2023-09-22T20:57:58.888 INFO:teuthology.orchestra.run.smithi204.stdout:22 Sep 20:57:58 ntpd[27817]: ---------------------------------------------------- 2023-09-22T20:57:58.889 INFO:teuthology.orchestra.run.smithi204.stdout:22 Sep 20:57:58 ntpd[27817]: proto: precision = 0.044 usec (-24) 2023-09-22T20:57:58.889 INFO:teuthology.orchestra.run.smithi204.stdout:22 Sep 20:57:58 ntpd[27817]: basedate set to 2022-02-04 2023-09-22T20:57:58.889 INFO:teuthology.orchestra.run.smithi204.stdout:22 Sep 20:57:58 ntpd[27817]: gps base set to 2022-02-06 (week 2196) 2023-09-22T20:57:58.889 INFO:teuthology.orchestra.run.smithi204.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-22T20:57:58.890 INFO:teuthology.orchestra.run.smithi204.stdout:22 Sep 20:57:58 ntpd[27817]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-22T20:57:58.890 INFO:teuthology.orchestra.run.smithi204.stdout:22 Sep 20:57:58 ntpd[27817]: restrict ::: KOD does nothing without LIMITED. 2023-09-22T20:57:58.890 INFO:teuthology.orchestra.run.smithi204.stderr:restrict ::: KOD does nothing without LIMITED. 2023-09-22T20:57:58.891 INFO:teuthology.orchestra.run.smithi204.stdout:22 Sep 20:57:58 ntpd[27817]: Listen and drop on 0 v6wildcard [::]:123 2023-09-22T20:57:58.891 INFO:teuthology.orchestra.run.smithi204.stdout:22 Sep 20:57:58 ntpd[27817]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-09-22T20:57:58.891 INFO:teuthology.orchestra.run.smithi204.stdout:22 Sep 20:57:58 ntpd[27817]: Listen normally on 2 lo 127.0.0.1:123 2023-09-22T20:57:58.891 INFO:teuthology.orchestra.run.smithi204.stdout:22 Sep 20:57:58 ntpd[27817]: Listen normally on 3 enp3s0f1 172.21.15.204:123 2023-09-22T20:57:58.891 INFO:teuthology.orchestra.run.smithi204.stdout:22 Sep 20:57:58 ntpd[27817]: Listen normally on 4 lo [::1]:123 2023-09-22T20:57:58.892 INFO:teuthology.orchestra.run.smithi204.stdout:22 Sep 20:57:58 ntpd[27817]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:77ab%5]:123 2023-09-22T20:57:58.892 INFO:teuthology.orchestra.run.smithi204.stdout:22 Sep 20:57:58 ntpd[27817]: Listening on routing socket on fd #22 for interface updates 2023-09-22T20:57:59.888 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:57:59 ntpd[27817]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:57:59.888 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:57:59 ntpd[27817]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:57:59.888 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:57:59 ntpd[27817]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:57:59.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:57:59 ntpd[27817]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:57:59.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:57:59 ntpd[27817]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-22T20:57:59.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:57:59 ntpd[27817]: can't open /var/log/ntpstats/peerstats.20230922: Permission denied 2023-09-22T20:58:00.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:00 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:00.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:00 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:00.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:00 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:00.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:00 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:00.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:00 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:00.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:00 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:00.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:00 ntpd[27850]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-22T20:58:00.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:00 ntpd[27850]: can't open /var/log/ntpstats/peerstats.20230922: Permission denied 2023-09-22T20:58:00.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:00 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:00.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:00 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:00.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:00 ntpd[27850]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-22T20:58:00.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:00 ntpd[27850]: can't open /var/log/ntpstats/peerstats.20230922: Permission denied 2023-09-22T20:58:01.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:01 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:01.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:01 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:01.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:01 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:01.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:01 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:01.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:01 ntpd[27850]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-22T20:58:01.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:01 ntpd[27850]: can't open /var/log/ntpstats/peerstats.20230922: Permission denied 2023-09-22T20:58:01.888 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:01 ntpd[27817]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:01.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:01 ntpd[27817]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:01.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:01 ntpd[27817]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:01.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:01 ntpd[27817]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:01.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:01 ntpd[27817]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:01.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:01 ntpd[27817]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:01.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:01 ntpd[27817]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-22T20:58:01.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:01 ntpd[27817]: can't open /var/log/ntpstats/peerstats.20230922: Permission denied 2023-09-22T20:58:02.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:02 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:02.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:02 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:02.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:02 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:02.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:02 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:02.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:02 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:02.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:02 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:02.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:02 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:02.872 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:02 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:02.872 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:02 ntpd[27850]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-22T20:58:02.872 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:02 ntpd[27850]: can't open /var/log/ntpstats/peerstats.20230922: Permission denied 2023-09-22T20:58:02.872 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:02 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:02.872 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:02 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:02.872 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:02 ntpd[27850]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-22T20:58:02.872 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:02 ntpd[27850]: can't open /var/log/ntpstats/peerstats.20230922: Permission denied 2023-09-22T20:58:02.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:02 ntpd[27817]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:02.890 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:02 ntpd[27817]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:02.890 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:02 ntpd[27817]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:02.890 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:02 ntpd[27817]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:02.890 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:02 ntpd[27817]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:02.890 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:02 ntpd[27817]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:02.890 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:02 ntpd[27817]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-22T20:58:02.891 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:02 ntpd[27817]: can't open /var/log/ntpstats/peerstats.20230922: Permission denied 2023-09-22T20:58:02.891 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:02 ntpd[27817]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:02.891 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:02 ntpd[27817]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:02.891 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:02 ntpd[27817]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-22T20:58:02.891 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:02 ntpd[27817]: can't open /var/log/ntpstats/peerstats.20230922: Permission denied 2023-09-22T20:58:03.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:03 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:03.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:03 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:03.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:03 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:03.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:03 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:03.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:03 ntpd[27850]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-22T20:58:03.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:03 ntpd[27850]: can't open /var/log/ntpstats/peerstats.20230922: Permission denied 2023-09-22T20:58:03.888 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:03 ntpd[27817]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:03.888 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:03 ntpd[27817]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:03.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:03 ntpd[27817]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:03.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:03 ntpd[27817]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:03.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:03 ntpd[27817]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-22T20:58:03.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:03 ntpd[27817]: can't open /var/log/ntpstats/peerstats.20230922: Permission denied 2023-09-22T20:58:04.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:04 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:04.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:04 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:04.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:04 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:04.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:04 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:04.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:04 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:04.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:04 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:04.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:04 ntpd[27850]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-22T20:58:04.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:04 ntpd[27850]: can't open /var/log/ntpstats/peerstats.20230922: Permission denied 2023-09-22T20:58:04.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:04 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:04.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:04 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:04.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:04 ntpd[27850]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-22T20:58:04.872 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:04 ntpd[27850]: can't open /var/log/ntpstats/peerstats.20230922: Permission denied 2023-09-22T20:58:04.888 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:04 ntpd[27817]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:04.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:04 ntpd[27817]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:04.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:04 ntpd[27817]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:04.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:04 ntpd[27817]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:04.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:04 ntpd[27817]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:04.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:04 ntpd[27817]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:04.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:04 ntpd[27817]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-22T20:58:04.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:04 ntpd[27817]: can't open /var/log/ntpstats/peerstats.20230922: Permission denied 2023-09-22T20:58:04.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:04 ntpd[27817]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:04.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:04 ntpd[27817]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:04.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:04 ntpd[27817]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-22T20:58:04.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:04 ntpd[27817]: can't open /var/log/ntpstats/peerstats.20230922: Permission denied 2023-09-22T20:58:05.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:05 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:05.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:05 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:05.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:05 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:05.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:05 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:05.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:05 ntpd[27850]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-22T20:58:05.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:05 ntpd[27850]: can't open /var/log/ntpstats/peerstats.20230922: Permission denied 2023-09-22T20:58:05.888 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:05 ntpd[27817]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:05.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:05 ntpd[27817]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:05.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:05 ntpd[27817]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:05.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:05 ntpd[27817]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:05.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:05 ntpd[27817]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-22T20:58:05.889 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:05 ntpd[27817]: can't open /var/log/ntpstats/peerstats.20230922: Permission denied 2023-09-22T20:58:05.889 INFO:teuthology.orchestra.run.smithi204.stdout:22 Sep 20:58:05 ntpd[27817]: ntpd: time slew +0.001599 s 2023-09-22T20:58:05.889 INFO:teuthology.orchestra.run.smithi204.stdout:ntpd: time slew +0.001599s 2023-09-22T20:58:05.890 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:05 ntpd[27817]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-09-22T20:58:05.890 INFO:teuthology.orchestra.run.smithi204.stderr:22 Sep 20:58:05 ntpd[27817]: can't open /var/log/ntpstats/loopstats.20230922: Permission denied 2023-09-22T20:58:05.980 INFO:teuthology.orchestra.run.smithi204.stdout: remote refid st t when poll reach delay offset jitter 2023-09-22T20:58:05.980 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================== 2023-09-22T20:58:05.980 INFO:teuthology.orchestra.run.smithi204.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-09-22T20:58:05.981 INFO:teuthology.orchestra.run.smithi204.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-09-22T20:58:05.981 INFO:teuthology.orchestra.run.smithi204.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-09-22T20:58:05.981 INFO:teuthology.orchestra.run.smithi204.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-09-22T20:58:06.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:06 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:06.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:06 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:06.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:06 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:06.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:06 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:06.870 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:06 ntpd[27850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-22T20:58:06.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:06 ntpd[27850]: can't open /var/log/ntpstats/rawstats.20230922: Permission denied 2023-09-22T20:58:06.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:06 ntpd[27850]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-22T20:58:06.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:06 ntpd[27850]: can't open /var/log/ntpstats/peerstats.20230922: Permission denied 2023-09-22T20:58:06.871 INFO:teuthology.orchestra.run.smithi160.stdout:22 Sep 20:58:06 ntpd[27850]: ntpd: time slew -0.002037 s 2023-09-22T20:58:06.871 INFO:teuthology.orchestra.run.smithi160.stdout:ntpd: time slew -0.002037s 2023-09-22T20:58:06.871 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:06 ntpd[27850]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-09-22T20:58:06.872 INFO:teuthology.orchestra.run.smithi160.stderr:22 Sep 20:58:06 ntpd[27850]: can't open /var/log/ntpstats/loopstats.20230922: Permission denied 2023-09-22T20:58:06.956 INFO:teuthology.orchestra.run.smithi160.stdout: remote refid st t when poll reach delay offset jitter 2023-09-22T20:58:06.956 INFO:teuthology.orchestra.run.smithi160.stdout:============================================================================== 2023-09-22T20:58:06.956 INFO:teuthology.orchestra.run.smithi160.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-09-22T20:58:06.956 INFO:teuthology.orchestra.run.smithi160.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-09-22T20:58:06.956 INFO:teuthology.orchestra.run.smithi160.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-09-22T20:58:06.956 INFO:teuthology.orchestra.run.smithi160.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-09-22T20:58:06.957 INFO:teuthology.run_tasks:Running task install... 2023-09-22T20:58:06.967 DEBUG:teuthology.task.install:project ceph 2023-09-22T20:58:06.967 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '03616b4330080cadf20be440d92e33ca90de254a'}} 2023-09-22T20:58:06.967 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '03616b4330080cadf20be440d92e33ca90de254a'} 2023-09-22T20:58:06.968 INFO:teuthology.task.install:Using flavor: default 2023-09-22T20:58:06.973 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-22T20:58:06.973 INFO:teuthology.task.install:extra packages: [] 2023-09-22T20:58:06.974 DEBUG:teuthology.orchestra.run.smithi160:> sudo apt-key list | grep Ceph 2023-09-22T20:58:06.974 DEBUG:teuthology.orchestra.run.smithi204:> sudo apt-key list | grep Ceph 2023-09-22T20:58:07.038 INFO:teuthology.orchestra.run.smithi204.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-09-22T20:58:07.065 INFO:teuthology.orchestra.run.smithi204.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-09-22T20:58:07.065 INFO:teuthology.orchestra.run.smithi204.stdout:uid [ unknown] Ceph.com (release key) 2023-09-22T20:58:07.065 INFO:teuthology.orchestra.run.smithi160.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-09-22T20:58:07.066 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-09-22T20:58:07.067 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=03616b4330080cadf20be440d92e33ca90de254a 2023-09-22T20:58:07.101 INFO:teuthology.orchestra.run.smithi160.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-09-22T20:58:07.101 INFO:teuthology.orchestra.run.smithi160.stdout:uid [ unknown] Ceph.com (release key) 2023-09-22T20:58:07.101 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-09-22T20:58:07.101 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=03616b4330080cadf20be440d92e33ca90de254a 2023-09-22T20:58:07.209 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default/ 2023-09-22T20:58:07.209 INFO:teuthology.task.install.deb:Package version is 18.0.0-6192-g03616b43-1jammy 2023-09-22T20:58:07.242 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default/ 2023-09-22T20:58:07.243 INFO:teuthology.task.install.deb:Package version is 18.0.0-6192-g03616b43-1jammy 2023-09-22T20:58:07.318 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T20:58:07.318 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-09-22T20:58:07.332 DEBUG:teuthology.orchestra.run.smithi204:> sudo apt-get update 2023-09-22T20:58:07.353 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T20:58:07.354 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-09-22T20:58:07.367 DEBUG:teuthology.orchestra.run.smithi160:> sudo apt-get update 2023-09-22T20:58:07.552 INFO:teuthology.orchestra.run.smithi204.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy InRelease 2023-09-22T20:58:07.574 INFO:teuthology.orchestra.run.smithi204.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-09-22T20:58:07.597 INFO:teuthology.orchestra.run.smithi204.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy Release.gpg 2023-09-22T20:58:07.618 INFO:teuthology.orchestra.run.smithi204.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,410 B] 2023-09-22T20:58:07.637 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-09-22T20:58:07.638 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-09-22T20:58:07.645 INFO:teuthology.orchestra.run.smithi160.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy InRelease 2023-09-22T20:58:07.659 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-09-22T20:58:07.659 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-09-22T20:58:07.659 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-09-22T20:58:07.682 INFO:teuthology.orchestra.run.smithi160.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-09-22T20:58:07.701 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-09-22T20:58:07.720 INFO:teuthology.orchestra.run.smithi160.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy Release.gpg 2023-09-22T20:58:07.734 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-09-22T20:58:07.756 INFO:teuthology.orchestra.run.smithi160.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,410 B] 2023-09-22T20:58:07.783 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-09-22T20:58:07.800 INFO:teuthology.orchestra.run.smithi160.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-09-22T20:58:07.814 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-09-22T20:58:07.987 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 28.6 kB in 0s (59.3 kB/s) 2023-09-22T20:58:08.015 INFO:teuthology.orchestra.run.smithi160.stdout:Fetched 28.6 kB in 0s (60.1 kB/s) 2023-09-22T20:58:08.819 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2023-09-22T20:58:08.834 DEBUG:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6192-g03616b43-1jammy cephadm=18.0.0-6192-g03616b43-1jammy ceph-mds=18.0.0-6192-g03616b43-1jammy ceph-mgr=18.0.0-6192-g03616b43-1jammy ceph-common=18.0.0-6192-g03616b43-1jammy ceph-fuse=18.0.0-6192-g03616b43-1jammy ceph-test=18.0.0-6192-g03616b43-1jammy ceph-volume=18.0.0-6192-g03616b43-1jammy radosgw=18.0.0-6192-g03616b43-1jammy python3-rados=18.0.0-6192-g03616b43-1jammy python3-rgw=18.0.0-6192-g03616b43-1jammy python3-cephfs=18.0.0-6192-g03616b43-1jammy python3-rbd=18.0.0-6192-g03616b43-1jammy libcephfs2=18.0.0-6192-g03616b43-1jammy libcephfs-dev=18.0.0-6192-g03616b43-1jammy librados2=18.0.0-6192-g03616b43-1jammy librbd1=18.0.0-6192-g03616b43-1jammy rbd-fuse=18.0.0-6192-g03616b43-1jammy 2023-09-22T20:58:08.884 INFO:teuthology.orchestra.run.smithi160.stdout:Reading package lists... 2023-09-22T20:58:08.893 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2023-09-22T20:58:08.900 DEBUG:teuthology.orchestra.run.smithi160:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6192-g03616b43-1jammy cephadm=18.0.0-6192-g03616b43-1jammy ceph-mds=18.0.0-6192-g03616b43-1jammy ceph-mgr=18.0.0-6192-g03616b43-1jammy ceph-common=18.0.0-6192-g03616b43-1jammy ceph-fuse=18.0.0-6192-g03616b43-1jammy ceph-test=18.0.0-6192-g03616b43-1jammy ceph-volume=18.0.0-6192-g03616b43-1jammy radosgw=18.0.0-6192-g03616b43-1jammy python3-rados=18.0.0-6192-g03616b43-1jammy python3-rgw=18.0.0-6192-g03616b43-1jammy python3-cephfs=18.0.0-6192-g03616b43-1jammy python3-rbd=18.0.0-6192-g03616b43-1jammy libcephfs2=18.0.0-6192-g03616b43-1jammy libcephfs-dev=18.0.0-6192-g03616b43-1jammy librados2=18.0.0-6192-g03616b43-1jammy librbd1=18.0.0-6192-g03616b43-1jammy rbd-fuse=18.0.0-6192-g03616b43-1jammy 2023-09-22T20:58:08.952 INFO:teuthology.orchestra.run.smithi160.stdout:Reading package lists... 2023-09-22T20:58:09.079 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2023-09-22T20:58:09.080 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2023-09-22T20:58:09.114 INFO:teuthology.orchestra.run.smithi160.stdout:Building dependency tree... 2023-09-22T20:58:09.115 INFO:teuthology.orchestra.run.smithi160.stdout:Reading state information... 2023-09-22T20:58:09.240 INFO:teuthology.orchestra.run.smithi204.stdout:The following packages were automatically installed and are no longer required: 2023-09-22T20:58:09.240 INFO:teuthology.orchestra.run.smithi204.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-09-22T20:58:09.240 INFO:teuthology.orchestra.run.smithi204.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-09-22T20:58:09.240 INFO:teuthology.orchestra.run.smithi204.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-09-22T20:58:09.240 INFO:teuthology.orchestra.run.smithi204.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-09-22T20:58:09.241 INFO:teuthology.orchestra.run.smithi204.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-09-22T20:58:09.241 INFO:teuthology.orchestra.run.smithi204.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-22T20:58:09.241 INFO:teuthology.orchestra.run.smithi204.stdout:The following additional packages will be installed: 2023-09-22T20:58:09.241 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-09-22T20:58:09.241 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-09-22T20:58:09.242 INFO:teuthology.orchestra.run.smithi204.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-09-22T20:58:09.242 INFO:teuthology.orchestra.run.smithi204.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-09-22T20:58:09.242 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-09-22T20:58:09.242 INFO:teuthology.orchestra.run.smithi204.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-09-22T20:58:09.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-09-22T20:58:09.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-09-22T20:58:09.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-09-22T20:58:09.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-09-22T20:58:09.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-09-22T20:58:09.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-09-22T20:58:09.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-09-22T20:58:09.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-09-22T20:58:09.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-09-22T20:58:09.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-09-22T20:58:09.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-09-22T20:58:09.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-09-22T20:58:09.244 INFO:teuthology.orchestra.run.smithi204.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-09-22T20:58:09.244 INFO:teuthology.orchestra.run.smithi204.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-09-22T20:58:09.244 INFO:teuthology.orchestra.run.smithi204.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-09-22T20:58:09.244 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-09-22T20:58:09.244 INFO:teuthology.orchestra.run.smithi204.stdout:Suggested packages: 2023-09-22T20:58:09.244 INFO:teuthology.orchestra.run.smithi204.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-09-22T20:58:09.244 INFO:teuthology.orchestra.run.smithi204.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-09-22T20:58:09.244 INFO:teuthology.orchestra.run.smithi204.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-09-22T20:58:09.244 INFO:teuthology.orchestra.run.smithi204.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-09-22T20:58:09.244 INFO:teuthology.orchestra.run.smithi204.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-09-22T20:58:09.245 INFO:teuthology.orchestra.run.smithi204.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-09-22T20:58:09.245 INFO:teuthology.orchestra.run.smithi204.stdout: python3-watchdog 2023-09-22T20:58:09.245 INFO:teuthology.orchestra.run.smithi204.stdout:Recommended packages: 2023-09-22T20:58:09.245 INFO:teuthology.orchestra.run.smithi204.stdout: btrfs-tools 2023-09-22T20:58:09.258 INFO:teuthology.orchestra.run.smithi160.stdout:The following packages were automatically installed and are no longer required: 2023-09-22T20:58:09.259 INFO:teuthology.orchestra.run.smithi160.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-09-22T20:58:09.259 INFO:teuthology.orchestra.run.smithi160.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-09-22T20:58:09.262 INFO:teuthology.orchestra.run.smithi160.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-09-22T20:58:09.262 INFO:teuthology.orchestra.run.smithi160.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-09-22T20:58:09.262 INFO:teuthology.orchestra.run.smithi160.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-09-22T20:58:09.263 INFO:teuthology.orchestra.run.smithi160.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-22T20:58:09.263 INFO:teuthology.orchestra.run.smithi160.stdout:The following additional packages will be installed: 2023-09-22T20:58:09.263 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-09-22T20:58:09.263 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-09-22T20:58:09.263 INFO:teuthology.orchestra.run.smithi160.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-09-22T20:58:09.263 INFO:teuthology.orchestra.run.smithi160.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-09-22T20:58:09.263 INFO:teuthology.orchestra.run.smithi160.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-09-22T20:58:09.263 INFO:teuthology.orchestra.run.smithi160.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-09-22T20:58:09.263 INFO:teuthology.orchestra.run.smithi160.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-09-22T20:58:09.263 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-09-22T20:58:09.263 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-09-22T20:58:09.263 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-09-22T20:58:09.264 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-09-22T20:58:09.264 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-09-22T20:58:09.264 INFO:teuthology.orchestra.run.smithi160.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-09-22T20:58:09.264 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-09-22T20:58:09.264 INFO:teuthology.orchestra.run.smithi160.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-09-22T20:58:09.264 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-09-22T20:58:09.264 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-09-22T20:58:09.264 INFO:teuthology.orchestra.run.smithi160.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-09-22T20:58:09.264 INFO:teuthology.orchestra.run.smithi160.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-09-22T20:58:09.264 INFO:teuthology.orchestra.run.smithi160.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-09-22T20:58:09.264 INFO:teuthology.orchestra.run.smithi160.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-09-22T20:58:09.264 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-09-22T20:58:09.264 INFO:teuthology.orchestra.run.smithi160.stdout:Suggested packages: 2023-09-22T20:58:09.265 INFO:teuthology.orchestra.run.smithi160.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-09-22T20:58:09.265 INFO:teuthology.orchestra.run.smithi160.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-09-22T20:58:09.265 INFO:teuthology.orchestra.run.smithi160.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-09-22T20:58:09.265 INFO:teuthology.orchestra.run.smithi160.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-09-22T20:58:09.265 INFO:teuthology.orchestra.run.smithi160.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-09-22T20:58:09.265 INFO:teuthology.orchestra.run.smithi160.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-09-22T20:58:09.265 INFO:teuthology.orchestra.run.smithi160.stdout: python3-watchdog 2023-09-22T20:58:09.265 INFO:teuthology.orchestra.run.smithi160.stdout:Recommended packages: 2023-09-22T20:58:09.265 INFO:teuthology.orchestra.run.smithi160.stdout: btrfs-tools 2023-09-22T20:58:09.337 INFO:teuthology.orchestra.run.smithi204.stdout:The following NEW packages will be installed: 2023-09-22T20:58:09.337 INFO:teuthology.orchestra.run.smithi204.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-09-22T20:58:09.337 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-09-22T20:58:09.338 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-09-22T20:58:09.338 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-09-22T20:58:09.338 INFO:teuthology.orchestra.run.smithi204.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-09-22T20:58:09.338 INFO:teuthology.orchestra.run.smithi204.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-09-22T20:58:09.339 INFO:teuthology.orchestra.run.smithi204.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-09-22T20:58:09.339 INFO:teuthology.orchestra.run.smithi204.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-09-22T20:58:09.339 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-09-22T20:58:09.339 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-09-22T20:58:09.339 INFO:teuthology.orchestra.run.smithi204.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-09-22T20:58:09.339 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-09-22T20:58:09.339 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-09-22T20:58:09.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-09-22T20:58:09.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-09-22T20:58:09.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-09-22T20:58:09.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-09-22T20:58:09.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-09-22T20:58:09.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-09-22T20:58:09.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-09-22T20:58:09.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-09-22T20:58:09.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-09-22T20:58:09.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-09-22T20:58:09.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-09-22T20:58:09.340 INFO:teuthology.orchestra.run.smithi204.stdout: socat xmlstarlet 2023-09-22T20:58:09.356 INFO:teuthology.orchestra.run.smithi160.stdout:The following NEW packages will be installed: 2023-09-22T20:58:09.356 INFO:teuthology.orchestra.run.smithi160.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-09-22T20:58:09.356 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-09-22T20:58:09.357 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-09-22T20:58:09.357 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-09-22T20:58:09.357 INFO:teuthology.orchestra.run.smithi160.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-09-22T20:58:09.357 INFO:teuthology.orchestra.run.smithi160.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-09-22T20:58:09.358 INFO:teuthology.orchestra.run.smithi160.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-09-22T20:58:09.358 INFO:teuthology.orchestra.run.smithi160.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-09-22T20:58:09.358 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-09-22T20:58:09.358 INFO:teuthology.orchestra.run.smithi160.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-09-22T20:58:09.358 INFO:teuthology.orchestra.run.smithi160.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-09-22T20:58:09.358 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-09-22T20:58:09.358 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-09-22T20:58:09.358 INFO:teuthology.orchestra.run.smithi160.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-09-22T20:58:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-09-22T20:58:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-09-22T20:58:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-09-22T20:58:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-09-22T20:58:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-09-22T20:58:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-09-22T20:58:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-09-22T20:58:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-09-22T20:58:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-09-22T20:58:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-09-22T20:58:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: socat xmlstarlet 2023-09-22T20:58:09.399 INFO:teuthology.orchestra.run.smithi204.stdout:0 upgraded, 109 newly installed, 0 to remove and 253 not upgraded. 2023-09-22T20:58:09.399 INFO:teuthology.orchestra.run.smithi204.stdout:Need to get 183 MB of archives. 2023-09-22T20:58:09.399 INFO:teuthology.orchestra.run.smithi204.stdout:After this operation, 824 MB of additional disk space will be used. 2023-09-22T20:58:09.399 INFO:teuthology.orchestra.run.smithi204.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-09-22T20:58:09.423 INFO:teuthology.orchestra.run.smithi160.stdout:0 upgraded, 109 newly installed, 0 to remove and 253 not upgraded. 2023-09-22T20:58:09.423 INFO:teuthology.orchestra.run.smithi160.stdout:Need to get 183 MB of archives. 2023-09-22T20:58:09.424 INFO:teuthology.orchestra.run.smithi160.stdout:After this operation, 824 MB of additional disk space will be used. 2023-09-22T20:58:09.424 INFO:teuthology.orchestra.run.smithi160.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-09-22T20:58:09.465 INFO:teuthology.orchestra.run.smithi204.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-09-22T20:58:09.490 INFO:teuthology.orchestra.run.smithi160.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-09-22T20:58:09.507 INFO:teuthology.orchestra.run.smithi204.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-09-22T20:58:09.511 INFO:teuthology.orchestra.run.smithi204.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-09-22T20:58:09.528 INFO:teuthology.orchestra.run.smithi204.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-09-22T20:58:09.532 INFO:teuthology.orchestra.run.smithi160.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-09-22T20:58:09.536 INFO:teuthology.orchestra.run.smithi160.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-09-22T20:58:09.555 INFO:teuthology.orchestra.run.smithi160.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-09-22T20:58:09.562 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6192-g03616b43-1jammy [3,415 kB] 2023-09-22T20:58:09.592 INFO:teuthology.orchestra.run.smithi160.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6192-g03616b43-1jammy [3,415 kB] 2023-09-22T20:58:09.597 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-09-22T20:58:09.602 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-09-22T20:58:09.615 INFO:teuthology.orchestra.run.smithi204.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-09-22T20:58:09.620 INFO:teuthology.orchestra.run.smithi204.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-09-22T20:58:09.620 INFO:teuthology.orchestra.run.smithi204.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-09-22T20:58:09.621 INFO:teuthology.orchestra.run.smithi204.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-09-22T20:58:09.630 INFO:teuthology.orchestra.run.smithi204.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-09-22T20:58:09.631 INFO:teuthology.orchestra.run.smithi204.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-09-22T20:58:09.631 INFO:teuthology.orchestra.run.smithi204.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-09-22T20:58:09.639 INFO:teuthology.orchestra.run.smithi160.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-09-22T20:58:09.639 INFO:teuthology.orchestra.run.smithi204.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-09-22T20:58:09.644 INFO:teuthology.orchestra.run.smithi160.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-09-22T20:58:09.660 INFO:teuthology.orchestra.run.smithi160.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-09-22T20:58:09.663 INFO:teuthology.orchestra.run.smithi204.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-09-22T20:58:09.664 INFO:teuthology.orchestra.run.smithi204.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-09-22T20:58:09.664 INFO:teuthology.orchestra.run.smithi204.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-09-22T20:58:09.665 INFO:teuthology.orchestra.run.smithi160.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-09-22T20:58:09.665 INFO:teuthology.orchestra.run.smithi204.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-09-22T20:58:09.666 INFO:teuthology.orchestra.run.smithi204.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-09-22T20:58:09.666 INFO:teuthology.orchestra.run.smithi204.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-09-22T20:58:09.666 INFO:teuthology.orchestra.run.smithi160.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-09-22T20:58:09.667 INFO:teuthology.orchestra.run.smithi160.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-09-22T20:58:09.669 INFO:teuthology.orchestra.run.smithi204.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-09-22T20:58:09.669 INFO:teuthology.orchestra.run.smithi204.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-09-22T20:58:09.675 INFO:teuthology.orchestra.run.smithi204.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-09-22T20:58:09.676 INFO:teuthology.orchestra.run.smithi160.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-09-22T20:58:09.677 INFO:teuthology.orchestra.run.smithi160.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-09-22T20:58:09.677 INFO:teuthology.orchestra.run.smithi160.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-09-22T20:58:09.679 INFO:teuthology.orchestra.run.smithi160.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-09-22T20:58:09.690 INFO:teuthology.orchestra.run.smithi204.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-09-22T20:58:09.691 INFO:teuthology.orchestra.run.smithi204.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-09-22T20:58:09.692 INFO:teuthology.orchestra.run.smithi204.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-09-22T20:58:09.692 INFO:teuthology.orchestra.run.smithi204.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-09-22T20:58:09.694 INFO:teuthology.orchestra.run.smithi204.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-09-22T20:58:09.694 INFO:teuthology.orchestra.run.smithi204.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-09-22T20:58:09.695 INFO:teuthology.orchestra.run.smithi204.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-09-22T20:58:09.701 INFO:teuthology.orchestra.run.smithi204.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-09-22T20:58:09.701 INFO:teuthology.orchestra.run.smithi204.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-09-22T20:58:09.702 INFO:teuthology.orchestra.run.smithi160.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-09-22T20:58:09.702 INFO:teuthology.orchestra.run.smithi160.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-09-22T20:58:09.703 INFO:teuthology.orchestra.run.smithi160.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-09-22T20:58:09.704 INFO:teuthology.orchestra.run.smithi160.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-09-22T20:58:09.704 INFO:teuthology.orchestra.run.smithi160.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-09-22T20:58:09.719 INFO:teuthology.orchestra.run.smithi204.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-09-22T20:58:09.720 INFO:teuthology.orchestra.run.smithi204.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-09-22T20:58:09.722 INFO:teuthology.orchestra.run.smithi204.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-09-22T20:58:09.724 INFO:teuthology.orchestra.run.smithi204.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-09-22T20:58:09.725 INFO:teuthology.orchestra.run.smithi204.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-09-22T20:58:09.726 INFO:teuthology.orchestra.run.smithi160.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-09-22T20:58:09.726 INFO:teuthology.orchestra.run.smithi204.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-09-22T20:58:09.727 INFO:teuthology.orchestra.run.smithi204.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-09-22T20:58:09.730 INFO:teuthology.orchestra.run.smithi160.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-09-22T20:58:09.731 INFO:teuthology.orchestra.run.smithi160.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-09-22T20:58:09.732 INFO:teuthology.orchestra.run.smithi160.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-09-22T20:58:09.732 INFO:teuthology.orchestra.run.smithi160.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-09-22T20:58:09.742 INFO:teuthology.orchestra.run.smithi204.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-09-22T20:58:09.743 INFO:teuthology.orchestra.run.smithi204.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-09-22T20:58:09.745 INFO:teuthology.orchestra.run.smithi204.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-09-22T20:58:09.754 INFO:teuthology.orchestra.run.smithi160.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-09-22T20:58:09.755 INFO:teuthology.orchestra.run.smithi160.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-09-22T20:58:09.756 INFO:teuthology.orchestra.run.smithi160.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-09-22T20:58:09.758 INFO:teuthology.orchestra.run.smithi160.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-09-22T20:58:09.759 INFO:teuthology.orchestra.run.smithi160.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-09-22T20:58:09.794 INFO:teuthology.orchestra.run.smithi160.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-09-22T20:58:09.805 INFO:teuthology.orchestra.run.smithi160.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-09-22T20:58:09.805 INFO:teuthology.orchestra.run.smithi160.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-09-22T20:58:09.806 INFO:teuthology.orchestra.run.smithi160.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-09-22T20:58:09.807 INFO:teuthology.orchestra.run.smithi160.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-09-22T20:58:09.821 INFO:teuthology.orchestra.run.smithi204.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-09-22T20:58:09.822 INFO:teuthology.orchestra.run.smithi204.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-09-22T20:58:09.823 INFO:teuthology.orchestra.run.smithi204.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-09-22T20:58:09.824 INFO:teuthology.orchestra.run.smithi204.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-09-22T20:58:09.824 INFO:teuthology.orchestra.run.smithi204.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-09-22T20:58:09.826 INFO:teuthology.orchestra.run.smithi204.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-09-22T20:58:09.839 INFO:teuthology.orchestra.run.smithi160.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-09-22T20:58:09.843 INFO:teuthology.orchestra.run.smithi160.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-09-22T20:58:09.845 INFO:teuthology.orchestra.run.smithi160.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-09-22T20:58:09.847 INFO:teuthology.orchestra.run.smithi160.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-09-22T20:58:09.849 INFO:teuthology.orchestra.run.smithi160.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-09-22T20:58:09.856 INFO:teuthology.orchestra.run.smithi204.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-09-22T20:58:09.861 INFO:teuthology.orchestra.run.smithi204.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-09-22T20:58:09.862 INFO:teuthology.orchestra.run.smithi204.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-09-22T20:58:09.883 INFO:teuthology.orchestra.run.smithi204.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-09-22T20:58:09.883 INFO:teuthology.orchestra.run.smithi204.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-09-22T20:58:09.884 INFO:teuthology.orchestra.run.smithi160.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-09-22T20:58:09.884 INFO:teuthology.orchestra.run.smithi204.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-09-22T20:58:09.885 INFO:teuthology.orchestra.run.smithi160.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-09-22T20:58:09.890 INFO:teuthology.orchestra.run.smithi160.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-09-22T20:58:09.919 INFO:teuthology.orchestra.run.smithi204.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-09-22T20:58:09.919 INFO:teuthology.orchestra.run.smithi204.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-09-22T20:58:09.919 INFO:teuthology.orchestra.run.smithi204.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-09-22T20:58:09.920 INFO:teuthology.orchestra.run.smithi204.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-09-22T20:58:09.921 INFO:teuthology.orchestra.run.smithi204.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-09-22T20:58:09.925 INFO:teuthology.orchestra.run.smithi204.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-09-22T20:58:09.955 INFO:teuthology.orchestra.run.smithi204.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-09-22T20:58:09.957 INFO:teuthology.orchestra.run.smithi204.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-09-22T20:58:09.958 INFO:teuthology.orchestra.run.smithi204.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-09-22T20:58:09.962 INFO:teuthology.orchestra.run.smithi204.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-09-22T20:58:09.979 INFO:teuthology.orchestra.run.smithi204.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-09-22T20:58:09.985 INFO:teuthology.orchestra.run.smithi204.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-09-22T20:58:09.988 INFO:teuthology.orchestra.run.smithi204.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-09-22T20:58:09.988 INFO:teuthology.orchestra.run.smithi204.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-09-22T20:58:09.989 INFO:teuthology.orchestra.run.smithi204.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-09-22T20:58:09.989 INFO:teuthology.orchestra.run.smithi204.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-09-22T20:58:10.005 INFO:teuthology.orchestra.run.smithi160.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-09-22T20:58:10.007 INFO:teuthology.orchestra.run.smithi160.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-09-22T20:58:10.009 INFO:teuthology.orchestra.run.smithi160.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-09-22T20:58:10.010 INFO:teuthology.orchestra.run.smithi160.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-09-22T20:58:10.010 INFO:teuthology.orchestra.run.smithi204.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-09-22T20:58:10.010 INFO:teuthology.orchestra.run.smithi160.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-09-22T20:58:10.011 INFO:teuthology.orchestra.run.smithi204.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-09-22T20:58:10.012 INFO:teuthology.orchestra.run.smithi160.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-09-22T20:58:10.020 INFO:teuthology.orchestra.run.smithi204.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-09-22T20:58:10.020 INFO:teuthology.orchestra.run.smithi204.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-09-22T20:58:10.036 INFO:teuthology.orchestra.run.smithi160.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-09-22T20:58:10.038 INFO:teuthology.orchestra.run.smithi160.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-09-22T20:58:10.038 INFO:teuthology.orchestra.run.smithi160.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-09-22T20:58:10.048 INFO:teuthology.orchestra.run.smithi204.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-09-22T20:58:10.050 INFO:teuthology.orchestra.run.smithi204.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-09-22T20:58:10.051 INFO:teuthology.orchestra.run.smithi204.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-09-22T20:58:10.080 INFO:teuthology.orchestra.run.smithi160.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-09-22T20:58:10.083 INFO:teuthology.orchestra.run.smithi160.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-09-22T20:58:10.085 INFO:teuthology.orchestra.run.smithi160.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-09-22T20:58:10.085 INFO:teuthology.orchestra.run.smithi160.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-09-22T20:58:10.086 INFO:teuthology.orchestra.run.smithi160.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-09-22T20:58:10.086 INFO:teuthology.orchestra.run.smithi160.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-09-22T20:58:10.087 INFO:teuthology.orchestra.run.smithi160.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-09-22T20:58:10.087 INFO:teuthology.orchestra.run.smithi160.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-09-22T20:58:10.092 INFO:teuthology.orchestra.run.smithi160.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-09-22T20:58:10.111 INFO:teuthology.orchestra.run.smithi160.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-09-22T20:58:10.113 INFO:teuthology.orchestra.run.smithi160.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-09-22T20:58:10.114 INFO:teuthology.orchestra.run.smithi160.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-09-22T20:58:10.120 INFO:teuthology.orchestra.run.smithi160.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-09-22T20:58:10.127 INFO:teuthology.orchestra.run.smithi160.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-09-22T20:58:10.135 INFO:teuthology.orchestra.run.smithi160.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-09-22T20:58:10.141 INFO:teuthology.orchestra.run.smithi160.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-09-22T20:58:10.141 INFO:teuthology.orchestra.run.smithi160.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-09-22T20:58:10.143 INFO:teuthology.orchestra.run.smithi160.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-09-22T20:58:10.143 INFO:teuthology.orchestra.run.smithi160.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-09-22T20:58:10.157 INFO:teuthology.orchestra.run.smithi160.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-09-22T20:58:10.159 INFO:teuthology.orchestra.run.smithi160.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-09-22T20:58:10.174 INFO:teuthology.orchestra.run.smithi160.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-09-22T20:58:10.174 INFO:teuthology.orchestra.run.smithi160.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-09-22T20:58:10.175 INFO:teuthology.orchestra.run.smithi160.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-09-22T20:58:10.180 INFO:teuthology.orchestra.run.smithi160.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-09-22T20:58:10.181 INFO:teuthology.orchestra.run.smithi160.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-09-22T20:58:10.463 INFO:teuthology.orchestra.run.smithi204.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6192-g03616b43-1jammy [3,120 kB] 2023-09-22T20:58:10.764 INFO:teuthology.orchestra.run.smithi204.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6192-g03616b43-1jammy [670 kB] 2023-09-22T20:58:10.830 INFO:teuthology.orchestra.run.smithi204.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6192-g03616b43-1jammy [345 kB] 2023-09-22T20:58:10.861 INFO:teuthology.orchestra.run.smithi204.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6192-g03616b43-1jammy [23.0 kB] 2023-09-22T20:58:10.862 INFO:teuthology.orchestra.run.smithi204.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6192-g03616b43-1jammy [170 kB] 2023-09-22T20:58:10.868 INFO:teuthology.orchestra.run.smithi204.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6192-g03616b43-1jammy [52.4 kB] 2023-09-22T20:58:10.870 INFO:teuthology.orchestra.run.smithi204.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6192-g03616b43-1jammy [318 kB] 2023-09-22T20:58:10.900 INFO:teuthology.orchestra.run.smithi204.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6192-g03616b43-1jammy [6,191 kB] 2023-09-22T20:58:10.919 INFO:teuthology.orchestra.run.smithi160.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6192-g03616b43-1jammy [3,120 kB] 2023-09-22T20:58:11.251 INFO:teuthology.orchestra.run.smithi204.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6192-g03616b43-1jammy [102 kB] 2023-09-22T20:58:11.266 INFO:teuthology.orchestra.run.smithi204.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6192-g03616b43-1jammy [428 kB] 2023-09-22T20:58:11.279 INFO:teuthology.orchestra.run.smithi204.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6192-g03616b43-1jammy [22.6 MB] 2023-09-22T20:58:11.330 INFO:teuthology.orchestra.run.smithi160.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6192-g03616b43-1jammy [670 kB] 2023-09-22T20:58:11.376 INFO:teuthology.orchestra.run.smithi160.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6192-g03616b43-1jammy [345 kB] 2023-09-22T20:58:11.419 INFO:teuthology.orchestra.run.smithi160.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6192-g03616b43-1jammy [23.0 kB] 2023-09-22T20:58:11.420 INFO:teuthology.orchestra.run.smithi160.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6192-g03616b43-1jammy [170 kB] 2023-09-22T20:58:11.458 INFO:teuthology.orchestra.run.smithi160.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6192-g03616b43-1jammy [52.4 kB] 2023-09-22T20:58:11.460 INFO:teuthology.orchestra.run.smithi160.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6192-g03616b43-1jammy [318 kB] 2023-09-22T20:58:11.505 INFO:teuthology.orchestra.run.smithi160.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6192-g03616b43-1jammy [6,191 kB] 2023-09-22T20:58:12.167 INFO:teuthology.orchestra.run.smithi160.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6192-g03616b43-1jammy [102 kB] 2023-09-22T20:58:12.169 INFO:teuthology.orchestra.run.smithi160.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6192-g03616b43-1jammy [428 kB] 2023-09-22T20:58:12.239 INFO:teuthology.orchestra.run.smithi160.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6192-g03616b43-1jammy [22.6 MB] 2023-09-22T20:58:12.400 INFO:teuthology.orchestra.run.smithi204.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6192-g03616b43-1jammy [6,840 kB] 2023-09-22T20:58:12.698 INFO:teuthology.orchestra.run.smithi204.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6192-g03616b43-1jammy [228 kB] 2023-09-22T20:58:12.703 INFO:teuthology.orchestra.run.smithi204.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6192-g03616b43-1jammy [117 kB] 2023-09-22T20:58:12.706 INFO:teuthology.orchestra.run.smithi204.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6192-g03616b43-1jammy [1,481 kB] 2023-09-22T20:58:12.758 INFO:teuthology.orchestra.run.smithi204.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6192-g03616b43-1jammy [9,857 kB] 2023-09-22T20:58:13.219 INFO:teuthology.orchestra.run.smithi204.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6192-g03616b43-1jammy [31.6 MB] 2023-09-22T20:58:14.658 INFO:teuthology.orchestra.run.smithi160.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6192-g03616b43-1jammy [6,840 kB] 2023-09-22T20:58:14.819 INFO:teuthology.orchestra.run.smithi204.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6192-g03616b43-1jammy [3,598 B] 2023-09-22T20:58:14.819 INFO:teuthology.orchestra.run.smithi204.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6192-g03616b43-1jammy [821 kB] 2023-09-22T20:58:14.836 INFO:teuthology.orchestra.run.smithi204.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6192-g03616b43-1jammy [2,157 kB] 2023-09-22T20:58:14.876 INFO:teuthology.orchestra.run.smithi204.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6192-g03616b43-1jammy [248 kB] 2023-09-22T20:58:14.893 INFO:teuthology.orchestra.run.smithi204.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6192-g03616b43-1jammy [128 kB] 2023-09-22T20:58:14.897 INFO:teuthology.orchestra.run.smithi204.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6192-g03616b43-1jammy [1,883 kB] 2023-09-22T20:58:14.987 INFO:teuthology.orchestra.run.smithi204.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6192-g03616b43-1jammy [8,615 kB] 2023-09-22T20:58:15.283 INFO:teuthology.orchestra.run.smithi160.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6192-g03616b43-1jammy [228 kB] 2023-09-22T20:58:15.317 INFO:teuthology.orchestra.run.smithi160.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6192-g03616b43-1jammy [117 kB] 2023-09-22T20:58:15.320 INFO:teuthology.orchestra.run.smithi160.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6192-g03616b43-1jammy [1,481 kB] 2023-09-22T20:58:15.394 INFO:teuthology.orchestra.run.smithi204.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6192-g03616b43-1jammy [3,728 B] 2023-09-22T20:58:15.394 INFO:teuthology.orchestra.run.smithi204.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6192-g03616b43-1jammy [47.9 MB] 2023-09-22T20:58:15.466 INFO:teuthology.orchestra.run.smithi160.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6192-g03616b43-1jammy [9,857 kB] 2023-09-22T20:58:16.534 INFO:teuthology.orchestra.run.smithi160.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6192-g03616b43-1jammy [31.6 MB] 2023-09-22T20:58:17.874 INFO:teuthology.orchestra.run.smithi204.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6192-g03616b43-1jammy [234 kB] 2023-09-22T20:58:17.952 INFO:teuthology.orchestra.run.smithi204.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6192-g03616b43-1jammy [28.4 kB] 2023-09-22T20:58:17.953 INFO:teuthology.orchestra.run.smithi204.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6192-g03616b43-1jammy [12.2 MB] 2023-09-22T20:58:18.385 INFO:teuthology.orchestra.run.smithi204.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6192-g03616b43-1jammy [80.7 kB] 2023-09-22T20:58:18.937 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 183 MB in 9s (20.3 MB/s) 2023-09-22T20:58:19.076 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libfmt8:amd64. 2023-09-22T20:58:19.091 INFO:teuthology.orchestra.run.smithi160.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6192-g03616b43-1jammy [3,598 B] 2023-09-22T20:58:19.255 INFO:teuthology.orchestra.run.smithi160.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6192-g03616b43-1jammy [821 kB] 2023-09-22T20:58:19.256 INFO:teuthology.orchestra.run.smithi160.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6192-g03616b43-1jammy [2,157 kB] 2023-09-22T20:58:19.256 INFO:teuthology.orchestra.run.smithi160.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6192-g03616b43-1jammy [248 kB] 2023-09-22T20:58:19.256 INFO:teuthology.orchestra.run.smithi160.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6192-g03616b43-1jammy [128 kB] 2023-09-22T20:58:19.256 INFO:teuthology.orchestra.run.smithi160.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6192-g03616b43-1jammy [1,883 kB] 2023-09-22T20:58:19.256 INFO:teuthology.orchestra.run.smithi204.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 166009 files and directories currently installed.) 2023-09-22T20:58:19.257 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-09-22T20:58:19.266 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-09-22T20:58:19.302 INFO:teuthology.orchestra.run.smithi160.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6192-g03616b43-1jammy [8,615 kB] 2023-09-22T20:58:19.594 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-09-22T20:58:19.620 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-09-22T20:58:19.661 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-09-22T20:58:19.731 INFO:teuthology.orchestra.run.smithi160.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6192-g03616b43-1jammy [3,728 B] 2023-09-22T20:58:19.731 INFO:teuthology.orchestra.run.smithi160.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6192-g03616b43-1jammy [47.9 MB] 2023-09-22T20:58:20.031 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-09-22T20:58:20.058 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-09-22T20:58:20.106 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-09-22T20:58:20.626 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-09-22T20:58:20.652 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-09-22T20:58:20.692 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-09-22T20:58:21.154 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-09-22T20:58:21.181 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-09-22T20:58:21.237 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-09-22T20:58:21.716 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-09-22T20:58:21.742 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-09-22T20:58:21.782 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-09-22T20:58:22.119 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-09-22T20:58:22.145 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-09-22T20:58:22.185 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-09-22T20:58:22.189 INFO:teuthology.orchestra.run.smithi160.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6192-g03616b43-1jammy [234 kB] 2023-09-22T20:58:22.194 INFO:teuthology.orchestra.run.smithi160.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6192-g03616b43-1jammy [28.4 kB] 2023-09-22T20:58:22.195 INFO:teuthology.orchestra.run.smithi160.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6192-g03616b43-1jammy [12.2 MB] 2023-09-22T20:58:22.591 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-09-22T20:58:22.617 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-09-22T20:58:22.666 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-09-22T20:58:22.865 INFO:teuthology.orchestra.run.smithi160.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/rgw-wip-t62141-valg-supp_i008/03616b4330080cadf20be440d92e33ca90de254a/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6192-g03616b43-1jammy [80.7 kB] 2023-09-22T20:58:22.994 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package librados2. 2023-09-22T20:58:23.020 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../008-librados2_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:23.060 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librados2 (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:23.420 INFO:teuthology.orchestra.run.smithi160.stdout:Fetched 183 MB in 13s (13.6 MB/s) 2023-09-22T20:58:23.496 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package librbd1. 2023-09-22T20:58:23.523 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../009-librbd1_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:23.539 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libfmt8:amd64. 2023-09-22T20:58:23.562 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librbd1 (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:23.633 INFO:teuthology.orchestra.run.smithi160.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 166009 files and directories currently installed.) 2023-09-22T20:58:23.638 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-09-22T20:58:23.721 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-09-22T20:58:23.974 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libcephfs2. 2023-09-22T20:58:23.990 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-09-22T20:58:24.000 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:24.017 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-09-22T20:58:24.032 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libcephfs2 (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:24.049 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-09-22T20:58:24.310 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-rados. 2023-09-22T20:58:24.325 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:24.368 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-rados (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:24.369 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-09-22T20:58:24.395 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-09-22T20:58:24.435 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-09-22T20:58:24.679 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-09-22T20:58:24.705 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6192-g03616b43-1jammy_all.deb ... 2023-09-22T20:58:24.745 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-ceph-argparse (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:24.755 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-09-22T20:58:24.781 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-09-22T20:58:24.821 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-09-22T20:58:24.964 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-cephfs. 2023-09-22T20:58:24.980 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:25.014 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-cephfs (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:25.200 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-09-22T20:58:25.226 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-09-22T20:58:25.266 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-09-22T20:58:25.318 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-ceph-common. 2023-09-22T20:58:25.344 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6192-g03616b43-1jammy_all.deb ... 2023-09-22T20:58:25.411 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-ceph-common (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:25.636 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-09-22T20:58:25.662 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-09-22T20:58:25.703 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-09-22T20:58:26.029 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-wcwidth. 2023-09-22T20:58:26.039 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-09-22T20:58:26.056 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-09-22T20:58:26.065 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-09-22T20:58:26.096 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-09-22T20:58:26.105 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-09-22T20:58:26.374 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-prettytable. 2023-09-22T20:58:26.401 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-09-22T20:58:26.433 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-09-22T20:58:26.440 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-09-22T20:58:26.448 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-09-22T20:58:26.483 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-09-22T20:58:26.710 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-rbd. 2023-09-22T20:58:26.725 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:26.768 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-rbd (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:26.811 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package librados2. 2023-09-22T20:58:26.838 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../008-librados2_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:26.878 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking librados2 (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:27.113 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-09-22T20:58:27.139 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-09-22T20:58:27.179 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-09-22T20:58:27.347 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package librbd1. 2023-09-22T20:58:27.373 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../009-librbd1_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:27.414 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking librbd1 (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:27.490 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package librgw2. 2023-09-22T20:58:27.517 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../019-librgw2_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:27.549 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librgw2 (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:27.800 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libcephfs2. 2023-09-22T20:58:27.826 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:27.866 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libcephfs2 (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:28.051 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-rgw. 2023-09-22T20:58:28.078 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:28.118 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-rgw (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:28.143 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-rados. 2023-09-22T20:58:28.159 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:28.215 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-rados (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:28.438 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package liboath0:amd64. 2023-09-22T20:58:28.464 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-09-22T20:58:28.504 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-09-22T20:58:28.671 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-09-22T20:58:28.697 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6192-g03616b43-1jammy_all.deb ... 2023-09-22T20:58:28.738 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-ceph-argparse (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:28.824 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libradosstriper1. 2023-09-22T20:58:28.850 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:28.890 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libradosstriper1 (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:28.981 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-cephfs. 2023-09-22T20:58:29.008 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:29.048 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-cephfs (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:29.294 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-common. 2023-09-22T20:58:29.320 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:29.351 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-ceph-common. 2023-09-22T20:58:29.360 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-common (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:29.377 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6192-g03616b43-1jammy_all.deb ... 2023-09-22T20:58:29.417 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-ceph-common (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:29.746 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-wcwidth. 2023-09-22T20:58:29.772 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-09-22T20:58:29.812 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-09-22T20:58:30.140 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-prettytable. 2023-09-22T20:58:30.166 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-09-22T20:58:30.206 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-09-22T20:58:30.467 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-rbd. 2023-09-22T20:58:30.482 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:30.517 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-rbd (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:30.691 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-base. 2023-09-22T20:58:30.718 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:30.767 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-base (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:30.871 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-09-22T20:58:30.896 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-09-22T20:58:30.937 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-09-22T20:58:31.264 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package librgw2. 2023-09-22T20:58:31.290 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../019-librgw2_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:31.331 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking librgw2 (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:31.545 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-09-22T20:58:31.572 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-09-22T20:58:31.620 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-09-22T20:58:31.858 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-rgw. 2023-09-22T20:58:31.885 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:31.925 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-rgw (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:31.998 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-cheroot. 2023-09-22T20:58:32.025 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-09-22T20:58:32.073 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-09-22T20:58:32.245 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package liboath0:amd64. 2023-09-22T20:58:32.271 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-09-22T20:58:32.302 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-09-22T20:58:32.426 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-09-22T20:58:32.453 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-09-22T20:58:32.493 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-09-22T20:58:32.614 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libradosstriper1. 2023-09-22T20:58:32.640 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:32.680 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libradosstriper1 (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:32.821 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-jaraco.text. 2023-09-22T20:58:32.847 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-09-22T20:58:32.887 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-09-22T20:58:33.108 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-common. 2023-09-22T20:58:33.134 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:33.174 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-common (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:33.223 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-09-22T20:58:33.250 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-09-22T20:58:33.298 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-09-22T20:58:33.676 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-tempora. 2023-09-22T20:58:33.703 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-09-22T20:58:33.751 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-09-22T20:58:34.062 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-portend. 2023-09-22T20:58:34.089 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-09-22T20:58:34.129 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-09-22T20:58:34.432 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-09-22T20:58:34.459 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-09-22T20:58:34.498 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-09-22T20:58:34.597 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-base. 2023-09-22T20:58:34.624 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:34.673 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-base (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:34.885 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-cherrypy3. 2023-09-22T20:58:34.912 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-09-22T20:58:34.943 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-09-22T20:58:35.288 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-natsort. 2023-09-22T20:58:35.315 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-09-22T20:58:35.326 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-09-22T20:58:35.353 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-09-22T20:58:35.355 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-09-22T20:58:35.392 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-09-22T20:58:35.662 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-cheroot. 2023-09-22T20:58:35.675 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-logutils. 2023-09-22T20:58:35.688 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-09-22T20:58:35.701 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-09-22T20:58:35.729 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-09-22T20:58:35.741 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-09-22T20:58:36.040 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-09-22T20:58:36.067 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-09-22T20:58:36.106 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-09-22T20:58:36.228 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-markupsafe. 2023-09-22T20:58:36.254 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-09-22T20:58:36.294 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-09-22T20:58:36.409 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-jaraco.text. 2023-09-22T20:58:36.435 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-09-22T20:58:36.475 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-09-22T20:58:36.580 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-mako. 2023-09-22T20:58:36.607 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-09-22T20:58:36.647 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-09-22T20:58:36.762 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-09-22T20:58:36.788 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-09-22T20:58:36.828 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-09-22T20:58:37.017 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-simplegeneric. 2023-09-22T20:58:37.043 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-09-22T20:58:37.073 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-tempora. 2023-09-22T20:58:37.085 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-09-22T20:58:37.092 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-09-22T20:58:37.114 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-09-22T20:58:37.417 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-portend. 2023-09-22T20:58:37.444 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-09-22T20:58:37.470 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-singledispatch. 2023-09-22T20:58:37.483 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-09-22T20:58:37.496 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-09-22T20:58:37.544 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-09-22T20:58:37.820 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-09-22T20:58:37.846 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-09-22T20:58:37.872 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-webob. 2023-09-22T20:58:37.886 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-09-22T20:58:37.899 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-09-22T20:58:37.964 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-09-22T20:58:38.173 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-cherrypy3. 2023-09-22T20:58:38.199 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-09-22T20:58:38.231 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-09-22T20:58:38.317 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-waitress. 2023-09-22T20:58:38.344 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-09-22T20:58:38.387 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-09-22T20:58:38.576 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-natsort. 2023-09-22T20:58:38.602 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-09-22T20:58:38.642 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-09-22T20:58:38.687 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-tempita. 2023-09-22T20:58:38.713 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-09-22T20:58:38.753 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-09-22T20:58:38.954 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-logutils. 2023-09-22T20:58:38.981 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-09-22T20:58:39.020 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-09-22T20:58:39.106 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-paste. 2023-09-22T20:58:39.133 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-09-22T20:58:39.181 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-09-22T20:58:39.457 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-markupsafe. 2023-09-22T20:58:39.483 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-09-22T20:58:39.518 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-09-22T20:58:39.523 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-09-22T20:58:39.533 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-09-22T20:58:39.576 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-09-22T20:58:39.826 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-mako. 2023-09-22T20:58:39.852 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-09-22T20:58:39.892 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-09-22T20:58:39.912 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pastedeploy. 2023-09-22T20:58:39.939 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-09-22T20:58:39.987 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-09-22T20:58:40.214 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-simplegeneric. 2023-09-22T20:58:40.241 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-09-22T20:58:40.272 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-09-22T20:58:40.307 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-webtest. 2023-09-22T20:58:40.333 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-09-22T20:58:40.373 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-09-22T20:58:40.575 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-singledispatch. 2023-09-22T20:58:40.602 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-09-22T20:58:40.641 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-09-22T20:58:40.693 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pecan. 2023-09-22T20:58:40.720 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-09-22T20:58:40.760 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-09-22T20:58:40.927 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-webob. 2023-09-22T20:58:40.954 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-09-22T20:58:40.985 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-09-22T20:58:41.140 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-certifi. 2023-09-22T20:58:41.166 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-09-22T20:58:41.206 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-09-22T20:58:41.297 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-waitress. 2023-09-22T20:58:41.324 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-09-22T20:58:41.358 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-09-22T20:58:41.659 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-idna. 2023-09-22T20:58:41.683 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-tempita. 2023-09-22T20:58:41.685 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-09-22T20:58:41.710 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-09-22T20:58:41.725 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-idna (3.3-1) ... 2023-09-22T20:58:41.749 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-09-22T20:58:42.336 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-paste. 2023-09-22T20:58:42.337 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-09-22T20:58:42.337 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-09-22T20:58:42.439 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-09-22T20:58:42.466 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-09-22T20:58:42.505 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-09-22T20:58:42.887 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pastedeploy. 2023-09-22T20:58:43.074 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-09-22T20:58:43.074 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-urllib3. 2023-09-22T20:58:43.074 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-09-22T20:58:43.075 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-09-22T20:58:43.266 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-09-22T20:58:43.365 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-requests. 2023-09-22T20:58:43.392 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-09-22T20:58:43.440 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-09-22T20:58:43.818 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-werkzeug. 2023-09-22T20:58:43.836 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-webtest. 2023-09-22T20:58:43.845 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-09-22T20:58:43.862 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-09-22T20:58:43.893 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-09-22T20:58:43.902 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-09-22T20:58:44.155 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pecan. 2023-09-22T20:58:44.182 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-09-22T20:58:44.196 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-09-22T20:58:44.212 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6192-g03616b43-1jammy_all.deb ... 2023-09-22T20:58:44.213 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-09-22T20:58:44.246 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:44.560 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-certifi. 2023-09-22T20:58:44.586 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-09-22T20:58:44.634 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-09-22T20:58:44.661 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:44.701 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:44.784 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-09-22T20:58:44.995 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-openssl. 2023-09-22T20:58:45.022 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-09-22T20:58:45.062 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-09-22T20:58:45.104 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-idna. 2023-09-22T20:58:45.130 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-09-22T20:58:45.178 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-idna (3.3-1) ... 2023-09-22T20:58:45.373 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr. 2023-09-22T20:58:45.400 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:45.440 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:45.498 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-urllib3. 2023-09-22T20:58:45.525 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-09-22T20:58:45.565 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-09-22T20:58:45.843 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mon. 2023-09-22T20:58:45.870 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:45.909 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-requests. 2023-09-22T20:58:45.910 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mon (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:45.936 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-09-22T20:58:45.976 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-09-22T20:58:46.254 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-werkzeug. 2023-09-22T20:58:46.269 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-09-22T20:58:46.303 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-09-22T20:58:46.574 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-09-22T20:58:46.589 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6192-g03616b43-1jammy_all.deb ... 2023-09-22T20:58:46.631 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:46.897 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-osd. 2023-09-22T20:58:46.912 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:46.947 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-osd (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:47.022 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-09-22T20:58:47.046 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:47.088 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:47.397 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-openssl. 2023-09-22T20:58:47.426 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-09-22T20:58:47.472 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-09-22T20:58:47.750 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mgr. 2023-09-22T20:58:47.765 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:47.800 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mgr (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:48.161 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mon. 2023-09-22T20:58:48.187 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:48.227 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mon (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:48.758 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph. 2023-09-22T20:58:48.773 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../059-ceph_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:48.816 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:49.111 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-fuse. 2023-09-22T20:58:49.138 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:49.177 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-fuse (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:49.180 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-osd. 2023-09-22T20:58:49.207 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:49.247 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-osd (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:49.556 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mds. 2023-09-22T20:58:49.583 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:49.622 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mds (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:49.975 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package cephadm. 2023-09-22T20:58:50.002 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../062-cephadm_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:50.041 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking cephadm (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:50.345 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-asyncssh. 2023-09-22T20:58:50.371 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-09-22T20:58:50.411 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-09-22T20:58:50.739 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-babel-localedata. 2023-09-22T20:58:50.766 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-09-22T20:58:50.805 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-09-22T20:58:51.041 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph. 2023-09-22T20:58:51.056 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../059-ceph_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:51.091 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:51.369 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-fuse. 2023-09-22T20:58:51.395 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:51.435 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-fuse (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:51.803 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-babel. 2023-09-22T20:58:51.805 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mds. 2023-09-22T20:58:51.830 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-09-22T20:58:51.832 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:51.860 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-09-22T20:58:51.872 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mds (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:52.216 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package cephadm. 2023-09-22T20:58:52.231 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-jinja2. 2023-09-22T20:58:52.231 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../062-cephadm_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:58:52.257 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-09-22T20:58:52.266 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking cephadm (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:52.297 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-09-22T20:58:52.569 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-asyncssh. 2023-09-22T20:58:52.595 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-09-22T20:58:52.635 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-09-22T20:58:52.676 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-09-22T20:58:52.703 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6192-g03616b43-1jammy_all.deb ... 2023-09-22T20:58:52.742 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:52.921 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python-babel-localedata. 2023-09-22T20:58:52.948 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-09-22T20:58:52.988 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-09-22T20:58:53.146 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-bcrypt. 2023-09-22T20:58:53.173 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-09-22T20:58:53.212 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-09-22T20:58:53.540 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-repoze.lru. 2023-09-22T20:58:53.567 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-09-22T20:58:53.607 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-09-22T20:58:53.934 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-babel. 2023-09-22T20:58:53.950 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-09-22T20:58:53.960 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-routes. 2023-09-22T20:58:53.984 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-09-22T20:58:53.987 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-09-22T20:58:54.035 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-09-22T20:58:54.262 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-jinja2. 2023-09-22T20:58:54.277 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-09-22T20:58:54.313 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-09-22T20:58:54.338 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-09-22T20:58:54.354 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6192-g03616b43-1jammy_all.deb ... 2023-09-22T20:58:54.388 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:54.624 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-09-22T20:58:54.639 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6192-g03616b43-1jammy_all.deb ... 2023-09-22T20:58:54.674 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:54.775 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-09-22T20:58:54.790 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-09-22T20:58:54.833 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-09-22T20:58:55.127 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-bcrypt. 2023-09-22T20:58:55.154 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-09-22T20:58:55.193 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-09-22T20:58:55.346 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-joblib. 2023-09-22T20:58:55.372 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-09-22T20:58:55.412 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-09-22T20:58:55.496 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-repoze.lru. 2023-09-22T20:58:55.523 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-09-22T20:58:55.563 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-09-22T20:58:55.732 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-09-22T20:58:55.759 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-09-22T20:58:55.798 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-09-22T20:58:55.882 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-routes. 2023-09-22T20:58:55.909 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-09-22T20:58:55.949 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-09-22T20:58:56.110 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-sklearn. 2023-09-22T20:58:56.137 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-09-22T20:58:56.176 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-09-22T20:58:56.294 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-09-22T20:58:56.320 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6192-g03616b43-1jammy_all.deb ... 2023-09-22T20:58:56.360 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:56.772 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-09-22T20:58:56.799 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-09-22T20:58:56.846 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-09-22T20:58:57.119 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-09-22T20:58:57.146 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6192-g03616b43-1jammy_all.deb ... 2023-09-22T20:58:57.185 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:57.350 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-joblib. 2023-09-22T20:58:57.377 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-09-22T20:58:57.416 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-09-22T20:58:57.745 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-09-22T20:58:57.772 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-09-22T20:58:57.803 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-09-22T20:58:58.056 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-sklearn. 2023-09-22T20:58:58.083 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-09-22T20:58:58.122 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-09-22T20:58:58.189 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-cachetools. 2023-09-22T20:58:58.216 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-09-22T20:58:58.255 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-09-22T20:58:58.524 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pyasn1. 2023-09-22T20:58:58.540 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-09-22T20:58:58.574 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-09-22T20:58:58.797 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-09-22T20:58:58.825 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6192-g03616b43-1jammy_all.deb ... 2023-09-22T20:58:58.864 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:58:58.878 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-09-22T20:58:58.905 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-09-22T20:58:58.944 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-09-22T20:58:59.231 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-rsa. 2023-09-22T20:58:59.258 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-09-22T20:58:59.297 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-rsa (4.8-1) ... 2023-09-22T20:58:59.651 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-google-auth. 2023-09-22T20:58:59.678 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-09-22T20:58:59.717 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-09-22T20:58:59.850 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-cachetools. 2023-09-22T20:58:59.878 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-09-22T20:58:59.916 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-09-22T20:59:00.037 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-09-22T20:59:00.064 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-09-22T20:59:00.103 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-09-22T20:59:00.228 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pyasn1. 2023-09-22T20:59:00.416 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-09-22T20:59:00.416 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-09-22T20:59:00.475 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-websocket. 2023-09-22T20:59:00.502 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-09-22T20:59:00.533 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-09-22T20:59:00.639 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-09-22T20:59:00.666 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-09-22T20:59:00.706 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-09-22T20:59:00.862 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-kubernetes. 2023-09-22T20:59:00.889 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-09-22T20:59:00.928 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-09-22T20:59:01.017 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-rsa. 2023-09-22T20:59:01.044 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-09-22T20:59:01.083 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-rsa (4.8-1) ... 2023-09-22T20:59:01.395 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-google-auth. 2023-09-22T20:59:01.422 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-09-22T20:59:01.461 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-09-22T20:59:01.592 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-09-22T20:59:01.608 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6192-g03616b43-1jammy_all.deb ... 2023-09-22T20:59:01.650 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:01.789 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-09-22T20:59:01.816 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-09-22T20:59:01.856 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-09-22T20:59:01.979 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libonig5:amd64. 2023-09-22T20:59:02.006 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-09-22T20:59:02.045 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-09-22T20:59:02.243 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-websocket. 2023-09-22T20:59:02.271 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-09-22T20:59:02.348 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libjq1:amd64. 2023-09-22T20:59:02.368 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-09-22T20:59:02.376 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-09-22T20:59:02.406 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-09-22T20:59:02.688 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-kubernetes. 2023-09-22T20:59:02.716 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-09-22T20:59:02.754 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-09-22T20:59:02.775 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package jq. 2023-09-22T20:59:02.791 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-09-22T20:59:02.834 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-09-22T20:59:03.103 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package socat. 2023-09-22T20:59:03.131 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-09-22T20:59:03.170 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-09-22T20:59:03.385 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-09-22T20:59:03.401 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6192-g03616b43-1jammy_all.deb ... 2023-09-22T20:59:03.435 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:03.481 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package xmlstarlet. 2023-09-22T20:59:03.497 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-09-22T20:59:03.539 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-09-22T20:59:03.754 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libonig5:amd64. 2023-09-22T20:59:03.782 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-09-22T20:59:03.821 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-09-22T20:59:03.913 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-test. 2023-09-22T20:59:03.929 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:59:03.971 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-test (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:04.149 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libjq1:amd64. 2023-09-22T20:59:04.176 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-09-22T20:59:04.215 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-09-22T20:59:04.450 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package jq. 2023-09-22T20:59:04.466 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-09-22T20:59:04.501 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-09-22T20:59:04.753 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package socat. 2023-09-22T20:59:04.780 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-09-22T20:59:04.819 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-09-22T20:59:05.140 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package xmlstarlet. 2023-09-22T20:59:05.155 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-09-22T20:59:05.198 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-09-22T20:59:05.543 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-test. 2023-09-22T20:59:05.559 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:59:05.601 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-test (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:06.217 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-volume. 2023-09-22T20:59:06.244 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6192-g03616b43-1jammy_all.deb ... 2023-09-22T20:59:06.283 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-volume (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:06.596 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libcephfs-dev. 2023-09-22T20:59:06.612 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:59:06.654 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libcephfs-dev (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:07.025 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package nvme-cli. 2023-09-22T20:59:07.052 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-09-22T20:59:07.091 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-09-22T20:59:07.639 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-09-22T20:59:07.666 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-09-22T20:59:07.713 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-09-22T20:59:07.805 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-volume. 2023-09-22T20:59:07.821 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6192-g03616b43-1jammy_all.deb ... 2023-09-22T20:59:07.863 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-volume (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:08.133 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-iniconfig. 2023-09-22T20:59:08.160 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-09-22T20:59:08.201 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libcephfs-dev. 2023-09-22T20:59:08.208 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-09-22T20:59:08.217 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:59:08.259 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libcephfs-dev (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:08.562 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package nvme-cli. 2023-09-22T20:59:08.577 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pastescript. 2023-09-22T20:59:08.589 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-09-22T20:59:08.605 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-09-22T20:59:08.628 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-09-22T20:59:08.669 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-09-22T20:59:08.976 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-09-22T20:59:09.003 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-09-22T20:59:09.023 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pluggy. 2023-09-22T20:59:09.033 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-09-22T20:59:09.050 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-09-22T20:59:09.192 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-09-22T20:59:09.401 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-psutil. 2023-09-22T20:59:09.428 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-09-22T20:59:09.445 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-iniconfig. 2023-09-22T20:59:09.461 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-09-22T20:59:09.467 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-09-22T20:59:09.503 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-09-22T20:59:09.764 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pastescript. 2023-09-22T20:59:09.792 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-09-22T20:59:09.821 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-py. 2023-09-22T20:59:09.839 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-09-22T20:59:09.850 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-09-22T20:59:09.895 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-py (1.10.0-1) ... 2023-09-22T20:59:10.159 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pluggy. 2023-09-22T20:59:10.187 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-09-22T20:59:10.216 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pygments. 2023-09-22T20:59:10.225 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-09-22T20:59:10.243 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-09-22T20:59:10.299 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-09-22T20:59:10.728 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pyinotify. 2023-09-22T20:59:10.755 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-09-22T20:59:10.786 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-psutil. 2023-09-22T20:59:10.794 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-09-22T20:59:10.814 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-09-22T20:59:10.853 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-09-22T20:59:11.097 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-toml. 2023-09-22T20:59:11.125 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-09-22T20:59:11.164 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-09-22T20:59:11.197 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-py. 2023-09-22T20:59:11.225 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-09-22T20:59:11.280 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-py (1.10.0-1) ... 2023-09-22T20:59:11.450 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pytest. 2023-09-22T20:59:11.478 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-09-22T20:59:11.516 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-09-22T20:59:11.584 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pygments. 2023-09-22T20:59:11.611 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-09-22T20:59:11.650 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-09-22T20:59:11.854 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-simplejson. 2023-09-22T20:59:11.881 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-09-22T20:59:11.921 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-09-22T20:59:12.104 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pyinotify. 2023-09-22T20:59:12.132 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-09-22T20:59:12.148 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-09-22T20:59:12.162 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-09-22T20:59:12.164 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-09-22T20:59:12.198 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-09-22T20:59:12.432 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-toml. 2023-09-22T20:59:12.459 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-09-22T20:59:12.489 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-09-22T20:59:12.767 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pytest. 2023-09-22T20:59:12.795 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-09-22T20:59:12.834 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-09-22T20:59:12.960 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package radosgw. 2023-09-22T20:59:12.988 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../107-radosgw_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:59:13.026 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking radosgw (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:13.120 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-simplejson. 2023-09-22T20:59:13.148 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-09-22T20:59:13.178 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-09-22T20:59:13.439 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-09-22T20:59:13.467 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-09-22T20:59:13.506 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-09-22T20:59:13.780 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package rbd-fuse. 2023-09-22T20:59:13.795 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:59:13.838 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking rbd-fuse (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:14.158 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-09-22T20:59:14.226 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package radosgw. 2023-09-22T20:59:14.254 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../107-radosgw_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:59:14.284 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking radosgw (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:14.393 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-09-22T20:59:14.519 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-09-22T20:59:14.797 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-09-22T20:59:15.029 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package rbd-fuse. 2023-09-22T20:59:15.057 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6192-g03616b43-1jammy_amd64.deb ... 2023-09-22T20:59:15.095 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking rbd-fuse (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:15.139 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-09-22T20:59:15.391 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-09-22T20:59:15.590 INFO:teuthology.orchestra.run.smithi204.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-09-22T20:59:15.650 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-09-22T20:59:15.655 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up cephadm (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:15.776 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-09-22T20:59:16.223 INFO:teuthology.orchestra.run.smithi204.stdout:Adding system user cephadm....done 2023-09-22T20:59:16.252 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-09-22T20:59:16.547 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-09-22T20:59:16.643 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-09-22T20:59:16.922 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-09-22T20:59:17.093 INFO:teuthology.orchestra.run.smithi160.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-09-22T20:59:17.186 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up cephadm (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:17.281 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-09-22T20:59:17.408 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-09-22T20:59:17.755 INFO:teuthology.orchestra.run.smithi160.stdout:Adding system user cephadm....done 2023-09-22T20:59:17.875 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-09-22T20:59:17.901 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-09-22T20:59:18.153 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-09-22T20:59:18.160 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-09-22T20:59:18.287 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-py (1.10.0-1) ... 2023-09-22T20:59:18.419 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-09-22T20:59:18.545 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-09-22T20:59:18.597 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-09-22T20:59:18.796 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-09-22T20:59:18.957 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-09-22T20:59:19.056 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-09-22T20:59:19.182 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-py (1.10.0-1) ... 2023-09-22T20:59:19.208 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-09-22T20:59:19.459 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-09-22T20:59:19.483 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-09-22T20:59:19.710 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-09-22T20:59:19.852 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-09-22T20:59:20.020 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-09-22T20:59:20.119 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-09-22T20:59:20.296 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-ceph-argparse (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:20.395 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-09-22T20:59:20.547 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-09-22T20:59:20.646 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-09-22T20:59:20.656 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-09-22T20:59:20.908 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-09-22T20:59:20.908 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-09-22T20:59:21.067 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-09-22T20:59:21.158 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-ceph-argparse (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:21.418 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-09-22T20:59:21.443 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-09-22T20:59:21.569 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-09-22T20:59:21.652 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-09-22T20:59:21.837 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-09-22T20:59:21.920 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-09-22T20:59:22.046 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-09-22T20:59:22.105 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-09-22T20:59:22.389 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-09-22T20:59:22.413 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-09-22T20:59:22.539 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-09-22T20:59:22.816 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-09-22T20:59:23.016 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-09-22T20:59:23.075 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-09-22T20:59:23.267 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-09-22T20:59:23.368 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-09-22T20:59:23.393 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-09-22T20:59:23.519 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-09-22T20:59:23.829 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-09-22T20:59:24.011 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-09-22T20:59:24.205 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-09-22T20:59:24.270 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-09-22T20:59:24.396 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-09-22T20:59:24.440 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-09-22T20:59:24.522 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-09-22T20:59:24.792 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-09-22T20:59:24.848 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-09-22T20:59:25.093 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-09-22T20:59:25.233 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-09-22T20:59:25.436 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-09-22T20:59:25.476 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-09-22T20:59:25.688 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-09-22T20:59:25.813 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-ceph-common (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:25.844 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-09-22T20:59:26.115 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-idna (3.3-1) ... 2023-09-22T20:59:26.128 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-09-22T20:59:26.424 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-09-22T20:59:26.472 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-09-22T20:59:26.731 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-09-22T20:59:26.840 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-ceph-common (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:27.141 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-idna (3.3-1) ... 2023-09-22T20:59:27.451 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-09-22T20:59:27.563 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-09-22T20:59:27.681 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-toml (0.10.2-1) ... 2023-09-22T20:59:27.932 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-09-22T20:59:28.050 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-09-22T20:59:28.343 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-09-22T20:59:28.636 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-09-22T20:59:28.656 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-09-22T20:59:28.790 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-toml (0.10.2-1) ... 2023-09-22T20:59:28.896 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-09-22T20:59:29.050 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-09-22T20:59:29.139 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-09-22T20:59:29.176 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-09-22T20:59:29.248 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-09-22T20:59:29.302 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-09-22T20:59:29.374 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-09-22T20:59:29.628 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-09-22T20:59:29.667 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-09-22T20:59:29.879 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-09-22T20:59:29.918 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-09-22T20:59:30.130 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-09-22T20:59:30.202 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-09-22T20:59:30.256 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-09-22T20:59:30.366 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-09-22T20:59:30.462 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-09-22T20:59:30.659 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-09-22T20:59:30.705 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-09-22T20:59:30.910 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-09-22T20:59:30.956 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-09-22T20:59:31.057 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-09-22T20:59:31.160 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-09-22T20:59:31.333 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-09-22T20:59:31.411 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-09-22T20:59:31.459 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-09-22T20:59:31.744 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-09-22T20:59:31.846 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-09-22T20:59:32.045 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-09-22T20:59:32.122 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-09-22T20:59:32.248 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-09-22T20:59:32.288 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-09-22T20:59:32.524 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-09-22T20:59:32.650 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-09-22T20:59:32.698 INFO:teuthology.orchestra.run.smithi204.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-09-22T20:59:32.934 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-09-22T20:59:32.949 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-09-22T20:59:33.067 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-09-22T20:59:33.211 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-09-22T20:59:33.469 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-09-22T20:59:33.470 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-09-22T20:59:33.736 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-portend (3.0.0-1) ... 2023-09-22T20:59:33.878 INFO:teuthology.orchestra.run.smithi160.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-09-22T20:59:33.996 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-09-22T20:59:34.122 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-09-22T20:59:34.131 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-09-22T20:59:34.266 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-09-22T20:59:34.440 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-09-22T20:59:34.642 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-09-22T20:59:34.741 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-09-22T20:59:34.893 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-portend (3.0.0-1) ... 2023-09-22T20:59:35.000 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-09-22T20:59:35.126 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-rsa (4.8-1) ... 2023-09-22T20:59:35.127 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-09-22T20:59:35.253 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-09-22T20:59:35.394 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-09-22T20:59:35.596 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-09-22T20:59:35.670 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-09-22T20:59:35.897 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-09-22T20:59:35.930 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-09-22T20:59:36.173 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-09-22T20:59:36.299 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-rsa (4.8-1) ... 2023-09-22T20:59:36.323 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-09-22T20:59:36.575 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-09-22T20:59:36.616 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-09-22T20:59:36.851 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-09-22T20:59:36.944 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-09-22T20:59:37.070 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librados2 (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:37.102 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-09-22T20:59:37.187 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librgw2 (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:37.288 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:37.398 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libcephfs2 (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:37.487 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-09-22T20:59:37.524 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libradosstriper1 (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:37.744 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-09-22T20:59:37.788 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-09-22T20:59:38.020 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librbd1 (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:38.132 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-09-22T20:59:38.147 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-modules-core (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:38.241 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up librados2 (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:38.264 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-fuse (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:38.351 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up librgw2 (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:38.452 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:38.469 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-22T20:59:38.469 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-22T20:59:38.578 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libcephfs2 (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:38.704 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libradosstriper1 (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:38.915 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-09-22T20:59:38.991 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libcephfs-dev (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:39.117 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-rados (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:39.192 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up librbd1 (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:39.243 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-09-22T20:59:39.318 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mgr-modules-core (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:39.443 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-fuse (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:39.653 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-22T20:59:39.654 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-22T20:59:40.128 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libcephfs-dev (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:40.279 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-rados (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:40.414 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-09-22T20:59:40.456 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-rbd (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:40.557 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up rbd-fuse (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:40.667 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-rgw (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:40.793 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-cephfs (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:40.919 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-common (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:41.462 INFO:teuthology.orchestra.run.smithi204.stdout:Adding group ceph....done 2023-09-22T20:59:41.652 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-rbd (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:41.778 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up rbd-fuse (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:41.839 INFO:teuthology.orchestra.run.smithi204.stdout:Adding system user ceph....done 2023-09-22T20:59:41.904 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-rgw (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:41.981 INFO:teuthology.orchestra.run.smithi204.stdout:Setting system user ceph properties....done 2023-09-22T20:59:41.995 INFO:teuthology.orchestra.run.smithi204.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-09-22T20:59:42.022 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-cephfs (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:42.126 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-09-22T20:59:42.139 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-common (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:42.487 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-09-22T20:59:42.682 INFO:teuthology.orchestra.run.smithi160.stdout:Adding group ceph....done 2023-09-22T20:59:43.029 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-test (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:43.059 INFO:teuthology.orchestra.run.smithi160.stdout:Adding system user ceph....done 2023-09-22T20:59:43.163 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up radosgw (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:43.202 INFO:teuthology.orchestra.run.smithi160.stdout:Setting system user ceph properties....done 2023-09-22T20:59:43.217 INFO:teuthology.orchestra.run.smithi160.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-09-22T20:59:43.346 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-09-22T20:59:43.419 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-22T20:59:43.419 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-22T20:59:43.715 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-09-22T20:59:44.199 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-base (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:44.274 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-test (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:44.391 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up radosgw (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:44.512 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-09-22T20:59:44.631 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-22T20:59:44.631 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-22T20:59:45.025 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mds (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:45.262 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-22T20:59:45.262 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-22T20:59:45.326 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-base (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:45.663 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-09-22T20:59:45.761 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:45.980 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-22T20:59:45.981 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-22T20:59:46.203 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mds (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:46.557 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-osd (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:46.557 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-22T20:59:46.557 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-22T20:59:46.801 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-22T20:59:46.801 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-22T20:59:46.913 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mgr (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:47.127 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-22T20:59:47.127 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-22T20:59:47.372 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:47.481 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:47.615 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-osd (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:47.624 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mon (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:47.832 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-22T20:59:47.832 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-22T20:59:47.934 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-22T20:59:47.934 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-22T20:59:48.359 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-cephadm (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:48.416 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:48.510 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:48.517 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:48.652 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mon (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:48.669 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-dashboard (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:48.820 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-volume (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:48.864 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-22T20:59:48.864 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-22T20:59:48.973 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-09-22T20:59:49.411 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mgr-cephadm (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:49.553 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:49.680 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mgr-dashboard (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:49.773 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-09-22T20:59:49.830 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-volume (18.0.0-6192-g03616b43-1jammy) ... 2023-09-22T20:59:49.957 INFO:teuthology.orchestra.run.smithi160.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-09-22T20:59:50.228 INFO:teuthology.orchestra.run.smithi204.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-09-22T20:59:50.232 DEBUG:teuthology.parallel:result is None 2023-09-22T20:59:51.000 INFO:teuthology.orchestra.run.smithi160.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-09-22T20:59:51.464 INFO:teuthology.orchestra.run.smithi160.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-09-22T20:59:51.467 DEBUG:teuthology.parallel:result is None 2023-09-22T20:59:51.467 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=03616b4330080cadf20be440d92e33ca90de254a 2023-09-22T20:59:51.624 DEBUG:teuthology.orchestra.run.smithi160:> dpkg-query -W -f '${Version}' ceph 2023-09-22T20:59:51.643 INFO:teuthology.orchestra.run.smithi160.stdout:18.0.0-6192-g03616b43-1jammy 2023-09-22T20:59:51.643 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6192-g03616b43-1jammy 2023-09-22T20:59:51.644 INFO:teuthology.task.install:The correct ceph version 18.0.0-6192-g03616b43-1jammy is installed. 2023-09-22T20:59:51.644 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=03616b4330080cadf20be440d92e33ca90de254a 2023-09-22T20:59:51.781 DEBUG:teuthology.orchestra.run.smithi204:> dpkg-query -W -f '${Version}' ceph 2023-09-22T20:59:51.800 INFO:teuthology.orchestra.run.smithi204.stdout:18.0.0-6192-g03616b43-1jammy 2023-09-22T20:59:51.800 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6192-g03616b43-1jammy 2023-09-22T20:59:51.800 INFO:teuthology.task.install:The correct ceph version 18.0.0-6192-g03616b43-1jammy is installed. 2023-09-22T20:59:51.801 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-22T20:59:51.801 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T20:59:51.801 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-22T20:59:51.815 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T20:59:51.815 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-22T20:59:51.857 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-22T20:59:51.858 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T20:59:51.858 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/daemon-helper 2023-09-22T20:59:51.870 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-22T20:59:51.929 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T20:59:51.929 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/daemon-helper 2023-09-22T20:59:51.942 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-22T20:59:51.995 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-22T20:59:51.996 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T20:59:51.996 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-22T20:59:52.008 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-22T20:59:52.061 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T20:59:52.061 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-22T20:59:52.073 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-22T20:59:52.127 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-22T20:59:52.127 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T20:59:52.128 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/stdin-killer 2023-09-22T20:59:52.139 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-22T20:59:52.192 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T20:59:52.192 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/stdin-killer 2023-09-22T20:59:52.205 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-22T20:59:52.260 INFO:teuthology.run_tasks:Running task ceph... 2023-09-22T20:59:52.334 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-22T20:59:52.334 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 777 /var/log/ceph 2023-09-22T20:59:52.337 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 777 /var/log/ceph 2023-09-22T20:59:52.349 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-22T20:59:52.349 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-22T20:59:52.391 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-22T20:59:52.403 INFO:tasks.ceph:Creating extra log directories... 2023-09-22T20:59:52.403 DEBUG:teuthology.orchestra.run.smithi160:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-22T20:59:52.451 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-22T20:59:52.472 INFO:tasks.ceph:Creating ceph cluster c1... 2023-09-22T20:59:52.473 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-22T20:59:52.473 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mkogan-2023-09-22_17:38:01-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7401166', 'branch': 'rgw-wip-t62141-valg-supp_i008', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi}', 'email': 'mkogan@redhat.com', 'first_in_suite': False, 'job_id': '7401166', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'mkogan-2023-09-22_17:38:01-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'rgw-wip-t62141-valg-supp_i008'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '03616b4330080cadf20be440d92e33ca90de254a'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '03616b4330080cadf20be440d92e33ca90de254a'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'workunit': {'branch': 'wip-qa-rgw-verify-ubuntu-003', 'sha1': '691e03976a4d019c58bad99761055f3266556055'}}, 'owner': 'scheduled_mkogan@teuthology', 'priority': 150, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 1924, 'sha1': '03616b4330080cadf20be440d92e33ca90de254a', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-qa-rgw-verify-ubuntu-003', 'suite_path': '/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/mkogan1/ceph.git', 'suite_sha1': '691e03976a4d019c58bad99761055f3266556055', 'targets': {'smithi160.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJc+XzPRS2TrfwsLPPiLo/tNCa2RGt6AcCR/nSya29UpF+GTgm+TxeVeHmZSk/cnLRECsVqSxIMEtf26RoWCMWI=', 'smithi204.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEqnq9KNsYblkwRNcJhKaEyCeQLeszT9O0Xr2v9A/EES2mM7cdfy9fws30xx5mDF1ikl+kZ3V+/mYuCmhT9Lpvw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '03616b4330080cadf20be440d92e33ca90de254a'}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-22_17:38:01', 'tube': 'smithi', 'user': 'mkogan', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-22T20:59:52.473 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-09-22T20:59:52.507 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-09-22T20:59:52.512 DEBUG:teuthology.orchestra.run.smithi160:> sudo install -d -m0777 -- /var/run/ceph 2023-09-22T20:59:52.558 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/run/ceph 2023-09-22T20:59:52.570 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T20:59:52.571 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/scratch_devs of=/dev/stdout 2023-09-22T20:59:52.618 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-22T20:59:52.618 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_1 2023-09-22T20:59:52.666 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-22T20:59:52.666 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-22T20:59:52.666 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 885 Links: 1 2023-09-22T20:59:52.666 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-22T20:59:52.667 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2023-09-22 20:54:38.521893333 +0000 2023-09-22T20:59:52.667 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2023-09-22 20:54:38.197896227 +0000 2023-09-22T20:59:52.667 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2023-09-22 20:54:38.197896227 +0000 2023-09-22T20:59:52.667 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: - 2023-09-22T20:59:52.667 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-22T20:59:52.720 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2023-09-22T20:59:52.720 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2023-09-22T20:59:52.720 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000438723 s, 1.2 MB/s 2023-09-22T20:59:52.722 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-22T20:59:52.771 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_2 2023-09-22T20:59:52.817 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-22T20:59:52.818 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-22T20:59:52.818 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 902 Links: 1 2023-09-22T20:59:52.818 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-22T20:59:52.818 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2023-09-22 20:54:39.157887652 +0000 2023-09-22T20:59:52.818 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2023-09-22 20:54:38.773891081 +0000 2023-09-22T20:59:52.818 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2023-09-22 20:54:38.773891081 +0000 2023-09-22T20:59:52.818 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: - 2023-09-22T20:59:52.818 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-22T20:59:52.872 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2023-09-22T20:59:52.872 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2023-09-22T20:59:52.872 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.00042793 s, 1.2 MB/s 2023-09-22T20:59:52.873 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-22T20:59:52.923 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_3 2023-09-22T20:59:52.974 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-22T20:59:52.974 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-22T20:59:52.974 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 911 Links: 1 2023-09-22T20:59:52.974 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-22T20:59:52.974 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2023-09-22 20:54:39.729882543 +0000 2023-09-22T20:59:52.974 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2023-09-22 20:54:39.401885473 +0000 2023-09-22T20:59:52.974 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2023-09-22 20:54:39.401885473 +0000 2023-09-22T20:59:52.974 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: - 2023-09-22T20:59:52.975 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-22T20:59:53.033 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2023-09-22T20:59:53.034 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2023-09-22T20:59:53.034 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000425404 s, 1.2 MB/s 2023-09-22T20:59:53.034 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-22T20:59:53.083 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_4 2023-09-22T20:59:53.129 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-22T20:59:53.130 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-22T20:59:53.130 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 931 Links: 1 2023-09-22T20:59:53.130 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-22T20:59:53.130 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2023-09-22 20:54:40.393876612 +0000 2023-09-22T20:59:53.130 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2023-09-22 20:54:39.969880399 +0000 2023-09-22T20:59:53.130 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2023-09-22 20:54:39.969880399 +0000 2023-09-22T20:59:53.130 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: - 2023-09-22T20:59:53.130 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-22T20:59:53.184 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2023-09-22T20:59:53.184 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2023-09-22T20:59:53.184 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.00041355 s, 1.2 MB/s 2023-09-22T20:59:53.185 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-22T20:59:53.235 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-22T20:59:53.235 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi160.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-22T20:59:53.236 INFO:tasks.ceph:Generating config... 2023-09-22T20:59:53.237 INFO:tasks.ceph:[client] debug rgw = 20 2023-09-22T20:59:53.237 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-09-22T20:59:53.237 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-09-22T20:59:53.237 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-09-22T20:59:53.237 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-09-22T20:59:53.237 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-09-22T20:59:53.237 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-09-22T20:59:53.237 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-09-22T20:59:53.237 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-09-22T20:59:53.238 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-09-22T20:59:53.238 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-09-22T20:59:53.238 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-09-22T20:59:53.238 INFO:tasks.ceph:[client] setgroup = ceph 2023-09-22T20:59:53.238 INFO:tasks.ceph:[client] setuser = ceph 2023-09-22T20:59:53.238 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-09-22T20:59:53.238 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-22T20:59:53.238 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-22T20:59:53.238 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-22T20:59:53.238 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-22T20:59:53.239 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-22T20:59:53.239 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-22T20:59:53.239 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-22T20:59:53.239 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-09-22T20:59:53.239 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-09-22T20:59:53.239 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-09-22T20:59:53.239 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-09-22T20:59:53.239 INFO:tasks.ceph:Setting up c1.mon.a... 2023-09-22T20:59:53.239 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2023-09-22T20:59:53.307 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/c1.keyring 2023-09-22T20:59:53.325 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2023-09-22T20:59:53.401 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-09-22T20:59:53.455 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.160')] 2023-09-22T20:59:53.455 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.160'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-09-22T20:59:53.456 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T20:59:53.456 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-22T20:59:53.501 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.160 --print /home/ubuntu/cephtest/c1.monmap 2023-09-22T20:59:53.580 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T20:59:53.580 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T20:59:53.580 INFO:teuthology.orchestra.run.smithi160.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2023-09-22T20:59:53.581 INFO:teuthology.orchestra.run.smithi160.stdout:monmaptool: generated fsid d0188f7d-2452-4b09-8037-934f8f90e7d1 2023-09-22T20:59:53.581 INFO:teuthology.orchestra.run.smithi160.stdout:setting min_mon_release = pacific 2023-09-22T20:59:53.581 INFO:teuthology.orchestra.run.smithi160.stdout:epoch 0 2023-09-22T20:59:53.581 INFO:teuthology.orchestra.run.smithi160.stdout:fsid d0188f7d-2452-4b09-8037-934f8f90e7d1 2023-09-22T20:59:53.581 INFO:teuthology.orchestra.run.smithi160.stdout:last_changed 2023-09-22T20:59:53.580902+0000 2023-09-22T20:59:53.581 INFO:teuthology.orchestra.run.smithi160.stdout:created 2023-09-22T20:59:53.580902+0000 2023-09-22T20:59:53.581 INFO:teuthology.orchestra.run.smithi160.stdout:min_mon_release 16 (pacific) 2023-09-22T20:59:53.581 INFO:teuthology.orchestra.run.smithi160.stdout:election_strategy: 1 2023-09-22T20:59:53.581 INFO:teuthology.orchestra.run.smithi160.stdout:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2023-09-22T20:59:53.581 INFO:teuthology.orchestra.run.smithi160.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2023-09-22T20:59:53.582 DEBUG:teuthology.orchestra.run.smithi160:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-22T20:59:53.630 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID d0188f7d-2452-4b09-8037-934f8f90e7d1... 2023-09-22T20:59:53.631 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-09-22T20:59:53.675 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-09-22T20:59:53.704 INFO:teuthology.orchestra.run.smithi160.stdout:[global] 2023-09-22T20:59:53.704 INFO:teuthology.orchestra.run.smithi160.stdout: chdir = "" 2023-09-22T20:59:53.704 INFO:teuthology.orchestra.run.smithi160.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-22T20:59:53.704 INFO:teuthology.orchestra.run.smithi160.stdout: auth supported = cephx 2023-09-22T20:59:53.704 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.704 INFO:teuthology.orchestra.run.smithi160.stdout: filestore xattr use omap = true 2023-09-22T20:59:53.705 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.705 INFO:teuthology.orchestra.run.smithi160.stdout: mon clock drift allowed = 1.000 2023-09-22T20:59:53.705 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.705 INFO:teuthology.orchestra.run.smithi160.stdout: osd crush chooseleaf type = 0 2023-09-22T20:59:53.705 INFO:teuthology.orchestra.run.smithi160.stdout: auth debug = true 2023-09-22T20:59:53.705 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.705 INFO:teuthology.orchestra.run.smithi160.stdout: ms die on old message = true 2023-09-22T20:59:53.705 INFO:teuthology.orchestra.run.smithi160.stdout: ms die on bug = true 2023-09-22T20:59:53.706 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.706 INFO:teuthology.orchestra.run.smithi160.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-22T20:59:53.706 INFO:teuthology.orchestra.run.smithi160.stdout: mon pg warn max object skew = 0 2023-09-22T20:59:53.706 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.706 INFO:teuthology.orchestra.run.smithi160.stdout: # disable pg_autoscaler by default for new pools 2023-09-22T20:59:53.706 INFO:teuthology.orchestra.run.smithi160.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-22T20:59:53.706 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.707 INFO:teuthology.orchestra.run.smithi160.stdout: osd pool default size = 2 2023-09-22T20:59:53.707 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.707 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd allow primary affinity = true 2023-09-22T20:59:53.707 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd allow pg remap = true 2023-09-22T20:59:53.707 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on legacy crush tunables = false 2023-09-22T20:59:53.707 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on crush straw calc version zero = false 2023-09-22T20:59:53.707 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on no sortbitwise = false 2023-09-22T20:59:53.707 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on osd down out interval zero = false 2023-09-22T20:59:53.708 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on too few osds = false 2023-09-22T20:59:53.708 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-22T20:59:53.708 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-22T20:59:53.708 INFO:teuthology.orchestra.run.smithi160.stdout: mon_allow_pool_size_one = true 2023-09-22T20:59:53.708 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.708 INFO:teuthology.orchestra.run.smithi160.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-22T20:59:53.708 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.709 INFO:teuthology.orchestra.run.smithi160.stdout: osd default data pool replay window = 5 2023-09-22T20:59:53.709 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.709 INFO:teuthology.orchestra.run.smithi160.stdout: mon allow pool delete = true 2023-09-22T20:59:53.709 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.709 INFO:teuthology.orchestra.run.smithi160.stdout: mon cluster log file level = debug 2023-09-22T20:59:53.709 INFO:teuthology.orchestra.run.smithi160.stdout: debug asserts on shutdown = true 2023-09-22T20:59:53.709 INFO:teuthology.orchestra.run.smithi160.stdout: mon health detail to clog = false 2023-09-22T20:59:53.709 INFO:teuthology.orchestra.run.smithi160.stdout: mon host = 172.21.15.160 2023-09-22T20:59:53.710 INFO:teuthology.orchestra.run.smithi160.stdout: fsid = d0188f7d-2452-4b09-8037-934f8f90e7d1 2023-09-22T20:59:53.710 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.710 INFO:teuthology.orchestra.run.smithi160.stdout:[osd] 2023-09-22T20:59:53.710 INFO:teuthology.orchestra.run.smithi160.stdout: osd journal size = 100 2023-09-22T20:59:53.710 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.710 INFO:teuthology.orchestra.run.smithi160.stdout: osd scrub load threshold = 5.0 2023-09-22T20:59:53.710 INFO:teuthology.orchestra.run.smithi160.stdout: osd scrub max interval = 600 2023-09-22T20:59:53.710 INFO:teuthology.orchestra.run.smithi160.stdout: osd mclock profile = high_recovery_ops 2023-09-22T20:59:53.711 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.711 INFO:teuthology.orchestra.run.smithi160.stdout: osd recover clone overlap = true 2023-09-22T20:59:53.711 INFO:teuthology.orchestra.run.smithi160.stdout: osd recovery max chunk = 1048576 2023-09-22T20:59:53.711 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.711 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug shutdown = true 2023-09-22T20:59:53.711 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug op order = true 2023-09-22T20:59:53.711 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug verify stray on activate = true 2023-09-22T20:59:53.711 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.711 INFO:teuthology.orchestra.run.smithi160.stdout: osd open classes on start = true 2023-09-22T20:59:53.711 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug pg log writeout = true 2023-09-22T20:59:53.711 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.712 INFO:teuthology.orchestra.run.smithi160.stdout: osd deep scrub update digest min age = 30 2023-09-22T20:59:53.712 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.712 INFO:teuthology.orchestra.run.smithi160.stdout: osd map max advance = 10 2023-09-22T20:59:53.712 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.712 INFO:teuthology.orchestra.run.smithi160.stdout: journal zero on create = true 2023-09-22T20:59:53.712 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.712 INFO:teuthology.orchestra.run.smithi160.stdout: filestore ondisk finisher threads = 3 2023-09-22T20:59:53.712 INFO:teuthology.orchestra.run.smithi160.stdout: filestore apply finisher threads = 3 2023-09-22T20:59:53.712 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.712 INFO:teuthology.orchestra.run.smithi160.stdout: bdev debug aio = true 2023-09-22T20:59:53.712 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug misdirected ops = true 2023-09-22T20:59:53.713 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2023-09-22T20:59:53.713 INFO:teuthology.orchestra.run.smithi160.stdout: debug osd = 20 2023-09-22T20:59:53.713 INFO:teuthology.orchestra.run.smithi160.stdout: rocksdb delete range threshold = 0 2023-09-22T20:59:53.713 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.713 INFO:teuthology.orchestra.run.smithi160.stdout:[mgr] 2023-09-22T20:59:53.713 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2023-09-22T20:59:53.713 INFO:teuthology.orchestra.run.smithi160.stdout: debug mgr = 20 2023-09-22T20:59:53.713 INFO:teuthology.orchestra.run.smithi160.stdout: debug mon = 20 2023-09-22T20:59:53.714 INFO:teuthology.orchestra.run.smithi160.stdout: debug auth = 20 2023-09-22T20:59:53.714 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min pgs per osd = 4 2023-09-22T20:59:53.714 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min bytes per osd = 10 2023-09-22T20:59:53.714 INFO:teuthology.orchestra.run.smithi160.stdout: mgr/telemetry/nag = false 2023-09-22T20:59:53.714 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.714 INFO:teuthology.orchestra.run.smithi160.stdout:[mon] 2023-09-22T20:59:53.714 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2023-09-22T20:59:53.714 INFO:teuthology.orchestra.run.smithi160.stdout: debug mon = 20 2023-09-22T20:59:53.715 INFO:teuthology.orchestra.run.smithi160.stdout: debug paxos = 20 2023-09-22T20:59:53.715 INFO:teuthology.orchestra.run.smithi160.stdout: debug auth = 20 2023-09-22T20:59:53.715 INFO:teuthology.orchestra.run.smithi160.stdout: mon data avail warn = 5 2023-09-22T20:59:53.715 INFO:teuthology.orchestra.run.smithi160.stdout: mon mgr mkfs grace = 240 2023-09-22T20:59:53.715 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min pgs per osd = 4 2023-09-22T20:59:53.715 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd reporter subtree level = osd 2023-09-22T20:59:53.715 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd prime pg temp = true 2023-09-22T20:59:53.716 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min bytes per osd = 10 2023-09-22T20:59:53.716 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.716 INFO:teuthology.orchestra.run.smithi160.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-22T20:59:53.716 INFO:teuthology.orchestra.run.smithi160.stdout: auth mon ticket ttl = 660 # 11m 2023-09-22T20:59:53.716 INFO:teuthology.orchestra.run.smithi160.stdout: auth service ticket ttl = 240 # 4m 2023-09-22T20:59:53.716 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.716 INFO:teuthology.orchestra.run.smithi160.stdout: # don't complain about insecure global_id in the test suite 2023-09-22T20:59:53.716 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-22T20:59:53.716 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-22T20:59:53.716 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.716 INFO:teuthology.orchestra.run.smithi160.stdout: # 1m isn't quite enough 2023-09-22T20:59:53.716 INFO:teuthology.orchestra.run.smithi160.stdout: mon_down_mkfs_grace = 2m 2023-09-22T20:59:53.717 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.717 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_filestore_osds = false 2023-09-22T20:59:53.717 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T20:59:53.717 INFO:teuthology.orchestra.run.smithi160.stdout:[client] 2023-09-22T20:59:53.717 INFO:teuthology.orchestra.run.smithi160.stdout: rgw cache enabled = true 2023-09-22T20:59:53.717 INFO:teuthology.orchestra.run.smithi160.stdout: rgw enable ops log = true 2023-09-22T20:59:53.717 INFO:teuthology.orchestra.run.smithi160.stdout: rgw enable usage log = true 2023-09-22T20:59:53.717 INFO:teuthology.orchestra.run.smithi160.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-22T20:59:53.717 INFO:teuthology.orchestra.run.smithi160.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-22T20:59:53.717 INFO:teuthology.orchestra.run.smithi160.stdout: debug rgw = 20 2023-09-22T20:59:53.717 INFO:teuthology.orchestra.run.smithi160.stdout: debug rgw sync = 20 2023-09-22T20:59:53.717 INFO:teuthology.orchestra.run.smithi160.stdout: rgw crypt require ssl = False 2023-09-22T20:59:53.718 INFO:teuthology.orchestra.run.smithi160.stdout: rgw crypt s3 kms backend = testing 2023-09-22T20:59:53.718 INFO:teuthology.orchestra.run.smithi160.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-09-22T20:59:53.718 INFO:teuthology.orchestra.run.smithi160.stdout: rgw curl low speed time = 300 2023-09-22T20:59:53.718 INFO:teuthology.orchestra.run.smithi160.stdout: rgw data log num shards = 4 2023-09-22T20:59:53.718 INFO:teuthology.orchestra.run.smithi160.stdout: rgw md log max shards = 4 2023-09-22T20:59:53.718 INFO:teuthology.orchestra.run.smithi160.stdout: rgw sync data inject err probability = 0.1 2023-09-22T20:59:53.718 INFO:teuthology.orchestra.run.smithi160.stdout: rgw sync log trim interval = 0 2023-09-22T20:59:53.718 INFO:teuthology.orchestra.run.smithi160.stdout: rgw sync meta inject err probability = 0.1 2023-09-22T20:59:53.718 INFO:teuthology.orchestra.run.smithi160.stdout: rgw sync obj etag verify = True 2023-09-22T20:59:53.718 INFO:teuthology.orchestra.run.smithi160.stdout: setgroup = ceph 2023-09-22T20:59:53.718 INFO:teuthology.orchestra.run.smithi160.stdout: setuser = ceph 2023-09-22T20:59:53.719 INFO:teuthology.orchestra.run.smithi160.stdout:[mon.a] 2023-09-22T20:59:53.719 INFO:teuthology.orchestra.run.smithi160.stdout:[client.0] 2023-09-22T20:59:53.719 INFO:teuthology.orchestra.run.smithi160.stdout: rgw data notify interval msec = 0 2023-09-22T20:59:53.719 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.0] 2023-09-22T20:59:53.719 INFO:teuthology.orchestra.run.smithi160.stdout: osd_max_omap_entries_per_request = 10 2023-09-22T20:59:53.719 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.1] 2023-09-22T20:59:53.719 INFO:teuthology.orchestra.run.smithi160.stdout: osd_max_omap_entries_per_request = 1000 2023-09-22T20:59:53.719 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.2] 2023-09-22T20:59:53.719 INFO:teuthology.orchestra.run.smithi160.stdout: osd_max_omap_entries_per_request = 10000 2023-09-22T20:59:53.721 INFO:teuthology.orchestra.run.smithi204.stdout:[global] 2023-09-22T20:59:53.721 INFO:teuthology.orchestra.run.smithi204.stdout: chdir = "" 2023-09-22T20:59:53.721 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-22T20:59:53.721 INFO:teuthology.orchestra.run.smithi204.stdout: auth supported = cephx 2023-09-22T20:59:53.721 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.721 INFO:teuthology.orchestra.run.smithi204.stdout: filestore xattr use omap = true 2023-09-22T20:59:53.721 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.721 INFO:teuthology.orchestra.run.smithi204.stdout: mon clock drift allowed = 1.000 2023-09-22T20:59:53.721 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.721 INFO:teuthology.orchestra.run.smithi204.stdout: osd crush chooseleaf type = 0 2023-09-22T20:59:53.721 INFO:teuthology.orchestra.run.smithi204.stdout: auth debug = true 2023-09-22T20:59:53.722 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.722 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on old message = true 2023-09-22T20:59:53.722 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on bug = true 2023-09-22T20:59:53.722 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.722 INFO:teuthology.orchestra.run.smithi204.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-22T20:59:53.722 INFO:teuthology.orchestra.run.smithi204.stdout: mon pg warn max object skew = 0 2023-09-22T20:59:53.722 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.722 INFO:teuthology.orchestra.run.smithi204.stdout: # disable pg_autoscaler by default for new pools 2023-09-22T20:59:53.722 INFO:teuthology.orchestra.run.smithi204.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-22T20:59:53.722 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.722 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default size = 2 2023-09-22T20:59:53.723 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.723 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow primary affinity = true 2023-09-22T20:59:53.723 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow pg remap = true 2023-09-22T20:59:53.723 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on legacy crush tunables = false 2023-09-22T20:59:53.723 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on crush straw calc version zero = false 2023-09-22T20:59:53.723 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on no sortbitwise = false 2023-09-22T20:59:53.723 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on osd down out interval zero = false 2023-09-22T20:59:53.723 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on too few osds = false 2023-09-22T20:59:53.723 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-22T20:59:53.723 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-22T20:59:53.723 INFO:teuthology.orchestra.run.smithi204.stdout: mon_allow_pool_size_one = true 2023-09-22T20:59:53.723 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.724 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-22T20:59:53.724 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.724 INFO:teuthology.orchestra.run.smithi204.stdout: osd default data pool replay window = 5 2023-09-22T20:59:53.724 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.724 INFO:teuthology.orchestra.run.smithi204.stdout: mon allow pool delete = true 2023-09-22T20:59:53.724 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.724 INFO:teuthology.orchestra.run.smithi204.stdout: mon cluster log file level = debug 2023-09-22T20:59:53.724 INFO:teuthology.orchestra.run.smithi204.stdout: debug asserts on shutdown = true 2023-09-22T20:59:53.724 INFO:teuthology.orchestra.run.smithi204.stdout: mon health detail to clog = false 2023-09-22T20:59:53.724 INFO:teuthology.orchestra.run.smithi204.stdout: mon host = 172.21.15.160 2023-09-22T20:59:53.724 INFO:teuthology.orchestra.run.smithi204.stdout: fsid = d0188f7d-2452-4b09-8037-934f8f90e7d1 2023-09-22T20:59:53.724 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.725 INFO:teuthology.orchestra.run.smithi204.stdout:[osd] 2023-09-22T20:59:53.725 INFO:teuthology.orchestra.run.smithi204.stdout: osd journal size = 100 2023-09-22T20:59:53.725 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.725 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub load threshold = 5.0 2023-09-22T20:59:53.725 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub max interval = 600 2023-09-22T20:59:53.725 INFO:teuthology.orchestra.run.smithi204.stdout: osd mclock profile = high_recovery_ops 2023-09-22T20:59:53.725 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.725 INFO:teuthology.orchestra.run.smithi204.stdout: osd recover clone overlap = true 2023-09-22T20:59:53.725 INFO:teuthology.orchestra.run.smithi204.stdout: osd recovery max chunk = 1048576 2023-09-22T20:59:53.725 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.725 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug shutdown = true 2023-09-22T20:59:53.725 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug op order = true 2023-09-22T20:59:53.725 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify stray on activate = true 2023-09-22T20:59:53.726 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.726 INFO:teuthology.orchestra.run.smithi204.stdout: osd open classes on start = true 2023-09-22T20:59:53.726 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug pg log writeout = true 2023-09-22T20:59:53.726 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.726 INFO:teuthology.orchestra.run.smithi204.stdout: osd deep scrub update digest min age = 30 2023-09-22T20:59:53.726 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.726 INFO:teuthology.orchestra.run.smithi204.stdout: osd map max advance = 10 2023-09-22T20:59:53.726 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.726 INFO:teuthology.orchestra.run.smithi204.stdout: journal zero on create = true 2023-09-22T20:59:53.726 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.726 INFO:teuthology.orchestra.run.smithi204.stdout: filestore ondisk finisher threads = 3 2023-09-22T20:59:53.727 INFO:teuthology.orchestra.run.smithi204.stdout: filestore apply finisher threads = 3 2023-09-22T20:59:53.727 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.727 INFO:teuthology.orchestra.run.smithi204.stdout: bdev debug aio = true 2023-09-22T20:59:53.727 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug misdirected ops = true 2023-09-22T20:59:53.727 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2023-09-22T20:59:53.727 INFO:teuthology.orchestra.run.smithi204.stdout: debug osd = 20 2023-09-22T20:59:53.727 INFO:teuthology.orchestra.run.smithi204.stdout: rocksdb delete range threshold = 0 2023-09-22T20:59:53.727 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.727 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr] 2023-09-22T20:59:53.727 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2023-09-22T20:59:53.727 INFO:teuthology.orchestra.run.smithi204.stdout: debug mgr = 20 2023-09-22T20:59:53.727 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2023-09-22T20:59:53.728 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2023-09-22T20:59:53.728 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2023-09-22T20:59:53.728 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2023-09-22T20:59:53.728 INFO:teuthology.orchestra.run.smithi204.stdout: mgr/telemetry/nag = false 2023-09-22T20:59:53.728 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.728 INFO:teuthology.orchestra.run.smithi204.stdout:[mon] 2023-09-22T20:59:53.728 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2023-09-22T20:59:53.728 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2023-09-22T20:59:53.728 INFO:teuthology.orchestra.run.smithi204.stdout: debug paxos = 20 2023-09-22T20:59:53.728 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2023-09-22T20:59:53.728 INFO:teuthology.orchestra.run.smithi204.stdout: mon data avail warn = 5 2023-09-22T20:59:53.729 INFO:teuthology.orchestra.run.smithi204.stdout: mon mgr mkfs grace = 240 2023-09-22T20:59:53.729 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2023-09-22T20:59:53.729 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd reporter subtree level = osd 2023-09-22T20:59:53.729 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd prime pg temp = true 2023-09-22T20:59:53.729 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2023-09-22T20:59:53.729 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.729 INFO:teuthology.orchestra.run.smithi204.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-22T20:59:53.730 INFO:teuthology.orchestra.run.smithi204.stdout: auth mon ticket ttl = 660 # 11m 2023-09-22T20:59:53.730 INFO:teuthology.orchestra.run.smithi204.stdout: auth service ticket ttl = 240 # 4m 2023-09-22T20:59:53.730 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.730 INFO:teuthology.orchestra.run.smithi204.stdout: # don't complain about insecure global_id in the test suite 2023-09-22T20:59:53.730 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-22T20:59:53.730 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-22T20:59:53.730 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.730 INFO:teuthology.orchestra.run.smithi204.stdout: # 1m isn't quite enough 2023-09-22T20:59:53.730 INFO:teuthology.orchestra.run.smithi204.stdout: mon_down_mkfs_grace = 2m 2023-09-22T20:59:53.731 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.731 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_filestore_osds = false 2023-09-22T20:59:53.731 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T20:59:53.731 INFO:teuthology.orchestra.run.smithi204.stdout:[client] 2023-09-22T20:59:53.731 INFO:teuthology.orchestra.run.smithi204.stdout: rgw cache enabled = true 2023-09-22T20:59:53.731 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable ops log = true 2023-09-22T20:59:53.731 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable usage log = true 2023-09-22T20:59:53.731 INFO:teuthology.orchestra.run.smithi204.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-22T20:59:53.731 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-22T20:59:53.731 INFO:teuthology.orchestra.run.smithi204.stdout: debug rgw = 20 2023-09-22T20:59:53.731 INFO:teuthology.orchestra.run.smithi204.stdout: debug rgw sync = 20 2023-09-22T20:59:53.731 INFO:teuthology.orchestra.run.smithi204.stdout: rgw crypt require ssl = False 2023-09-22T20:59:53.732 INFO:teuthology.orchestra.run.smithi204.stdout: rgw crypt s3 kms backend = testing 2023-09-22T20:59:53.732 INFO:teuthology.orchestra.run.smithi204.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-09-22T20:59:53.732 INFO:teuthology.orchestra.run.smithi204.stdout: rgw curl low speed time = 300 2023-09-22T20:59:53.732 INFO:teuthology.orchestra.run.smithi204.stdout: rgw data log num shards = 4 2023-09-22T20:59:53.732 INFO:teuthology.orchestra.run.smithi204.stdout: rgw md log max shards = 4 2023-09-22T20:59:53.732 INFO:teuthology.orchestra.run.smithi204.stdout: rgw sync data inject err probability = 0.1 2023-09-22T20:59:53.732 INFO:teuthology.orchestra.run.smithi204.stdout: rgw sync log trim interval = 0 2023-09-22T20:59:53.732 INFO:teuthology.orchestra.run.smithi204.stdout: rgw sync meta inject err probability = 0.1 2023-09-22T20:59:53.732 INFO:teuthology.orchestra.run.smithi204.stdout: rgw sync obj etag verify = True 2023-09-22T20:59:53.732 INFO:teuthology.orchestra.run.smithi204.stdout: setgroup = ceph 2023-09-22T20:59:53.732 INFO:teuthology.orchestra.run.smithi204.stdout: setuser = ceph 2023-09-22T20:59:53.733 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.a] 2023-09-22T20:59:53.733 INFO:teuthology.orchestra.run.smithi204.stdout:[client.0] 2023-09-22T20:59:53.733 INFO:teuthology.orchestra.run.smithi204.stdout: rgw data notify interval msec = 0 2023-09-22T20:59:53.733 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.0] 2023-09-22T20:59:53.733 INFO:teuthology.orchestra.run.smithi204.stdout: osd_max_omap_entries_per_request = 10 2023-09-22T20:59:53.733 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.1] 2023-09-22T20:59:53.733 INFO:teuthology.orchestra.run.smithi204.stdout: osd_max_omap_entries_per_request = 1000 2023-09-22T20:59:53.733 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.2] 2023-09-22T20:59:53.733 INFO:teuthology.orchestra.run.smithi204.stdout: osd_max_omap_entries_per_request = 10000 2023-09-22T20:59:53.734 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2023-09-22T20:59:53.734 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2023-09-22T20:59:53.813 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-22T20:59:53.814 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T20:59:53.814 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2023-09-22T20:59:53.819 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T20:59:53.819 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2023-09-22T20:59:53.869 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi160.front.sepia.ceph.com 2023-09-22T20:59:53.870 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T20:59:53.870 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/c1.keyring 2023-09-22T20:59:53.870 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-09-22T20:59:53.933 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T20:59:53.933 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-09-22T20:59:53.981 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi204.front.sepia.ceph.com 2023-09-22T20:59:53.982 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T20:59:53.982 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/c1.keyring 2023-09-22T20:59:53.982 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-09-22T20:59:54.005 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T20:59:54.005 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-09-22T20:59:54.053 INFO:tasks.ceph:Setting up mon nodes... 2023-09-22T20:59:54.053 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-22T20:59:54.053 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2023-09-22T20:59:54.097 INFO:teuthology.orchestra.run.smithi160.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2023-09-22T20:59:54.100 INFO:tasks.ceph:Setting up mds nodes... 2023-09-22T20:59:54.100 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-22T20:59:54.100 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2023-09-22T20:59:54.173 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/c1.client.0.keyring 2023-09-22T20:59:54.185 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2023-09-22T20:59:54.260 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/c1.client.1.keyring 2023-09-22T20:59:54.270 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-22T20:59:54.271 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi160.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-22T20:59:54.271 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2023-09-22T20:59:54.323 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-22T20:59:54.324 INFO:tasks.ceph:role: c1.osd.0 2023-09-22T20:59:54.324 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi160.front.sepia.ceph.com 2023-09-22T20:59:54.324 DEBUG:teuthology.orchestra.run.smithi160:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-22T20:59:54.385 INFO:teuthology.orchestra.run.smithi160.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-22T20:59:54.385 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-22T20:59:54.385 INFO:teuthology.orchestra.run.smithi160.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-22T20:59:54.386 INFO:teuthology.orchestra.run.smithi160.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-22T20:59:54.386 INFO:teuthology.orchestra.run.smithi160.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-22T20:59:54.386 INFO:teuthology.orchestra.run.smithi160.stdout: = sunit=0 swidth=0 blks 2023-09-22T20:59:54.386 INFO:teuthology.orchestra.run.smithi160.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-22T20:59:54.386 INFO:teuthology.orchestra.run.smithi160.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-22T20:59:54.386 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-22T20:59:54.386 INFO:teuthology.orchestra.run.smithi160.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-22T20:59:54.854 INFO:teuthology.orchestra.run.smithi160.stdout:Discarding blocks...Done. 2023-09-22T20:59:54.856 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi160.front.sepia.ceph.com -o noatime 2023-09-22T20:59:54.856 DEBUG:teuthology.orchestra.run.smithi160:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2023-09-22T20:59:54.975 DEBUG:teuthology.orchestra.run.smithi160:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2023-09-22T20:59:54.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T20:59:54.987 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: /sbin/restorecon: command not found 2023-09-22T20:59:54.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2023-09-22T20:59:55.041 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-22T20:59:55.041 INFO:tasks.ceph:role: c1.osd.1 2023-09-22T20:59:55.042 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi160.front.sepia.ceph.com 2023-09-22T20:59:55.042 DEBUG:teuthology.orchestra.run.smithi160:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-22T20:59:55.110 INFO:teuthology.orchestra.run.smithi160.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-22T20:59:55.110 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-22T20:59:55.111 INFO:teuthology.orchestra.run.smithi160.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-22T20:59:55.111 INFO:teuthology.orchestra.run.smithi160.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-22T20:59:55.111 INFO:teuthology.orchestra.run.smithi160.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-22T20:59:55.111 INFO:teuthology.orchestra.run.smithi160.stdout: = sunit=0 swidth=0 blks 2023-09-22T20:59:55.111 INFO:teuthology.orchestra.run.smithi160.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-22T20:59:55.111 INFO:teuthology.orchestra.run.smithi160.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-22T20:59:55.111 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-22T20:59:55.111 INFO:teuthology.orchestra.run.smithi160.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-22T20:59:55.538 INFO:teuthology.orchestra.run.smithi160.stdout:Discarding blocks...Done. 2023-09-22T20:59:55.540 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi160.front.sepia.ceph.com -o noatime 2023-09-22T20:59:55.540 DEBUG:teuthology.orchestra.run.smithi160:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2023-09-22T20:59:55.605 DEBUG:teuthology.orchestra.run.smithi160:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2023-09-22T20:59:55.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T20:59:55.658 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: /sbin/restorecon: command not found 2023-09-22T20:59:55.658 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2023-09-22T20:59:55.713 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-22T20:59:55.713 INFO:tasks.ceph:role: c1.osd.2 2023-09-22T20:59:55.713 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi160.front.sepia.ceph.com 2023-09-22T20:59:55.713 DEBUG:teuthology.orchestra.run.smithi160:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-22T20:59:55.777 INFO:teuthology.orchestra.run.smithi160.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-22T20:59:55.777 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-22T20:59:55.777 INFO:teuthology.orchestra.run.smithi160.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-22T20:59:55.777 INFO:teuthology.orchestra.run.smithi160.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-22T20:59:55.777 INFO:teuthology.orchestra.run.smithi160.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-22T20:59:55.777 INFO:teuthology.orchestra.run.smithi160.stdout: = sunit=0 swidth=0 blks 2023-09-22T20:59:55.777 INFO:teuthology.orchestra.run.smithi160.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-22T20:59:55.777 INFO:teuthology.orchestra.run.smithi160.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-22T20:59:55.777 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-22T20:59:55.777 INFO:teuthology.orchestra.run.smithi160.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-22T20:59:56.245 INFO:teuthology.orchestra.run.smithi160.stdout:Discarding blocks...Done. 2023-09-22T20:59:56.246 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi160.front.sepia.ceph.com -o noatime 2023-09-22T20:59:56.247 DEBUG:teuthology.orchestra.run.smithi160:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2023-09-22T20:59:56.331 DEBUG:teuthology.orchestra.run.smithi160:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2023-09-22T20:59:56.383 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: /sbin/restorecon: command not found 2023-09-22T20:59:56.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T20:59:56.384 DEBUG:teuthology.orchestra.run.smithi160:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2023-09-22T20:59:56.488 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T20:59:56.482+0000 7efe7e155640 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2023-09-22T20:59:56.492 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T20:59:56.486+0000 7efe7e155640 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2023-09-22T20:59:56.492 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T20:59:56.486+0000 7efe7e155640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-09-22T20:59:56.493 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T20:59:56.486+0000 7efe7e155640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-09-22T20:59:56.493 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T20:59:56.486+0000 7efe7e155640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-09-22T20:59:56.496 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T20:59:56.490+0000 7efe7e155640 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2023-09-22T20:59:58.719 DEBUG:teuthology.orchestra.run.smithi160:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2023-09-22T20:59:58.733 DEBUG:teuthology.orchestra.run.smithi160:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2023-09-22T20:59:58.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T20:59:58.818+0000 7f2dabfc2640 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2023-09-22T20:59:58.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T20:59:58.818+0000 7f2dabfc2640 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2023-09-22T20:59:58.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T20:59:58.818+0000 7f2dabfc2640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-09-22T20:59:58.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T20:59:58.818+0000 7f2dabfc2640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-09-22T20:59:58.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T20:59:58.818+0000 7f2dabfc2640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-09-22T20:59:58.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T20:59:58.818+0000 7f2dabfc2640 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2023-09-22T21:00:01.045 DEBUG:teuthology.orchestra.run.smithi160:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2023-09-22T21:00:01.102 DEBUG:teuthology.orchestra.run.smithi160:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2023-09-22T21:00:01.183 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:00:01.178+0000 7fead15de640 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2023-09-22T21:00:01.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:00:01.178+0000 7fead15de640 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2023-09-22T21:00:01.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:00:01.178+0000 7fead15de640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-09-22T21:00:01.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:00:01.178+0000 7fead15de640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-09-22T21:00:01.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:00:01.178+0000 7fead15de640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-09-22T21:00:01.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:00:01.178+0000 7fead15de640 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2023-09-22T21:00:03.397 DEBUG:teuthology.orchestra.run.smithi160:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2023-09-22T21:00:03.454 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-22T21:00:03.454 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T21:00:03.454 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2023-09-22T21:00:03.512 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T21:00:03.512 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2023-09-22T21:00:03.564 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T21:00:03.564 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2023-09-22T21:00:03.616 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T21:00:03.616 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2023-09-22T21:00:03.668 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T21:00:03.668 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2023-09-22T21:00:03.713 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T21:00:03.713 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2023-09-22T21:00:03.762 INFO:tasks.ceph:Adding keys to all mons... 2023-09-22T21:00:03.762 DEBUG:teuthology.orchestra.run.smithi160:> sudo tee -a /etc/ceph/c1.keyring 2023-09-22T21:00:03.817 INFO:teuthology.orchestra.run.smithi160.stdout:[mgr.x] 2023-09-22T21:00:03.818 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBKAA5lPZrSBRAAA7iOnYswwafwOWHHQ6eoig== 2023-09-22T21:00:03.818 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.0] 2023-09-22T21:00:03.818 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBMAA5lXzgkHRAAEAGDa0JPCq6odFAkPSwbMg== 2023-09-22T21:00:03.818 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.1] 2023-09-22T21:00:03.818 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBOAA5laRoHMRAACigmGllXiJlgJjnBmrA30w== 2023-09-22T21:00:03.818 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.2] 2023-09-22T21:00:03.818 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBRAA5lxHbvChAARjhis5mOo8yCzDNf3M869A== 2023-09-22T21:00:03.818 INFO:teuthology.orchestra.run.smithi160.stdout:[client.0] 2023-09-22T21:00:03.818 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBKAA5lha9fChAArxMX2JNiMDoreYL3ukmiTw== 2023-09-22T21:00:03.818 INFO:teuthology.orchestra.run.smithi160.stdout:[client.1] 2023-09-22T21:00:03.819 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBKAA5lZ4uHDxAAYewGkcSyvLHh4OdOgH1ngg== 2023-09-22T21:00:03.819 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-22T21:00:03.905 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T21:00:03.993 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T21:00:04.081 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T21:00:04.170 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-22T21:00:04.214 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-22T21:00:04.300 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-22T21:00:04.300 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2023-09-22T21:00:04.351 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2023-09-22T21:00:04.443 DEBUG:teuthology.orchestra.run.smithi160:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2023-09-22T21:00:04.453 DEBUG:teuthology.orchestra.run.smithi160:> rm -- /home/ubuntu/cephtest/c1.monmap 2023-09-22T21:00:04.501 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2023-09-22T21:00:04.501 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2023-09-22T21:00:04.501 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2023-09-22T21:00:04.546 INFO:tasks.ceph.c1.mon.a:Started 2023-09-22T21:00:04.547 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2023-09-22T21:00:04.547 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2023-09-22T21:00:04.547 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2023-09-22T21:00:04.549 INFO:tasks.ceph.c1.mgr.x:Started 2023-09-22T21:00:04.549 INFO:tasks.ceph:Setting crush tunables to default 2023-09-22T21:00:04.549 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster c1 osd crush tunables default 2023-09-22T21:00:04.780 INFO:teuthology.orchestra.run.smithi160.stderr:adjusted tunables profile to default 2023-09-22T21:00:04.799 INFO:tasks.ceph:check_enable_crimson: False 2023-09-22T21:00:04.799 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2023-09-22T21:00:04.799 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T21:00:04.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2023-09-22T21:00:04.809 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T21:00:04.809 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2023-09-22T21:00:04.868 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T21:00:04.868 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2023-09-22T21:00:04.907 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:04.902+0000 7f3fe74e8180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-22T21:00:04.918 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster c1 osd new 70ac7512-5619-4a29-825b-22268a0449d4 0 2023-09-22T21:00:05.163 INFO:teuthology.orchestra.run.smithi160.stdout:0 2023-09-22T21:00:05.180 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster c1 osd new d071b20e-b201-496c-bde9-a496d154c403 1 2023-09-22T21:00:05.380 INFO:teuthology.orchestra.run.smithi160.stdout:1 2023-09-22T21:00:05.396 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster c1 osd new aa84ccc7-e0be-49ac-8821-2e862cadfac8 2 2023-09-22T21:00:05.423 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:05.418+0000 7f3fe74e8180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-22T21:00:05.590 INFO:teuthology.orchestra.run.smithi160.stdout:2 2023-09-22T21:00:05.602 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:05.598+0000 7f3fe74e8180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-22T21:00:05.608 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2023-09-22T21:00:05.608 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2023-09-22T21:00:05.650 INFO:tasks.ceph.c1.osd.0:Started 2023-09-22T21:00:05.650 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2023-09-22T21:00:05.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2023-09-22T21:00:05.652 INFO:tasks.ceph.c1.osd.1:Started 2023-09-22T21:00:05.653 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2023-09-22T21:00:05.653 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2023-09-22T21:00:05.655 INFO:tasks.ceph.c1.osd.2:Started 2023-09-22T21:00:05.655 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-09-22T21:00:05.832 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:05.832 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":5,"fsid":"d0188f7d-2452-4b09-8037-934f8f90e7d1","created":"2023-09-22T21:00:04.679384+0000","modified":"2023-09-22T21:00:05.588062+0000","last_up_change":"0.000000","last_in_change":"2023-09-22T21:00:05.588062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"70ac7512-5619-4a29-825b-22268a0449d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d071b20e-b201-496c-bde9-a496d154c403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"aa84ccc7-e0be-49ac-8821-2e862cadfac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-22T21:00:05.848 INFO:tasks.ceph.ceph_manager.c1:[] 2023-09-22T21:00:05.848 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-22T21:00:06.150 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-09-22T21:00:06.197 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:06.190+0000 7f3fe74e8180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-22T21:00:06.252 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:06.246+0000 7f3fe74e8180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-22T21:00:06.270 INFO:tasks.ceph.c1.osd.2.smithi160.stderr:2023-09-22T21:00:06.266+0000 7ff0022c4640 -1 Falling back to public interface 2023-09-22T21:00:06.278 INFO:tasks.ceph.c1.osd.1.smithi160.stderr:2023-09-22T21:00:06.274+0000 7f1e00040640 -1 Falling back to public interface 2023-09-22T21:00:06.278 INFO:tasks.ceph.c1.osd.0.smithi160.stderr:2023-09-22T21:00:06.274+0000 7fe09d43f640 -1 Falling back to public interface 2023-09-22T21:00:06.307 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:06.302+0000 7f3fe74e8180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-22T21:00:06.334 INFO:teuthology.misc.health.smithi160.stdout: 2023-09-22T21:00:06.334 INFO:teuthology.misc.health.smithi160.stdout:{"epoch":5,"fsid":"d0188f7d-2452-4b09-8037-934f8f90e7d1","created":"2023-09-22T21:00:04.679384+0000","modified":"2023-09-22T21:00:05.588062+0000","last_up_change":"0.000000","last_in_change":"2023-09-22T21:00:05.588062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"70ac7512-5619-4a29-825b-22268a0449d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d071b20e-b201-496c-bde9-a496d154c403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"aa84ccc7-e0be-49ac-8821-2e862cadfac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-22T21:00:06.350 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-09-22T21:00:06.386 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:06.382+0000 7f3fe74e8180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-22T21:00:06.437 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:06.434+0000 7f3fe74e8180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-22T21:00:06.513 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:06.510+0000 7f3fe74e8180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-22T21:00:06.564 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:06.558+0000 7f3fe74e8180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-22T21:00:06.704 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:06.698+0000 7f3fe74e8180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-22T21:00:06.762 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:06.758+0000 7f3fe74e8180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-22T21:00:07.434 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-09-22T21:00:07.434 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-09-22T21:00:07.434 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr: from numpy import show_config as show_numpy_config 2023-09-22T21:00:07.500 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:07.494+0000 7f3fe74e8180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-22T21:00:07.659 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:07.654+0000 7f3fe74e8180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-22T21:00:07.826 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:07.822+0000 7f3fe74e8180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-22T21:00:07.888 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:07.882+0000 7f3fe74e8180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-22T21:00:08.010 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:08.006+0000 7f3fe74e8180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-22T21:00:08.630 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:08.626+0000 7f3fe74e8180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-22T21:00:08.705 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:08.702+0000 7f3fe74e8180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-22T21:00:08.889 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:08.882+0000 7f3fe74e8180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-22T21:00:08.962 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:08.958+0000 7f3fe74e8180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-22T21:00:09.092 INFO:tasks.ceph.c1.osd.0.smithi160.stderr:2023-09-22T21:00:09.086+0000 7fe09d43f640 -1 osd.0 0 log_to_monitors true 2023-09-22T21:00:09.260 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:09.254+0000 7f3fe74e8180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-22T21:00:09.323 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:09.318+0000 7f3fe74e8180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-22T21:00:09.335 INFO:tasks.ceph.c1.osd.2.smithi160.stderr:2023-09-22T21:00:09.330+0000 7ff0022c4640 -1 osd.2 0 log_to_monitors true 2023-09-22T21:00:09.336 INFO:tasks.ceph.c1.osd.1.smithi160.stderr:2023-09-22T21:00:09.330+0000 7f1e00040640 -1 osd.1 0 log_to_monitors true 2023-09-22T21:00:10.374 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T21:00:10.370+0000 7f3f8ee62640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-09-22T21:00:10.731 INFO:tasks.ceph.c1.osd.0.smithi160.stderr:2023-09-22T21:00:10.726+0000 7fe0993e7640 -1 osd.0 0 waiting for initial osdmap 2023-09-22T21:00:10.749 INFO:tasks.ceph.c1.osd.0.smithi160.stderr:2023-09-22T21:00:10.746+0000 7fe09121c640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-22T21:00:12.174 INFO:tasks.ceph.c1.osd.2.smithi160.stderr:2023-09-22T21:00:12.170+0000 7feffea80640 -1 osd.2 0 waiting for initial osdmap 2023-09-22T21:00:12.191 INFO:tasks.ceph.c1.osd.1.smithi160.stderr:2023-09-22T21:00:12.186+0000 7f1dfc7fc640 -1 osd.1 0 waiting for initial osdmap 2023-09-22T21:00:12.201 INFO:tasks.ceph.c1.osd.2.smithi160.stderr:2023-09-22T21:00:12.198+0000 7feff60a1640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-22T21:00:12.213 INFO:tasks.ceph.c1.osd.1.smithi160.stderr:2023-09-22T21:00:12.210+0000 7f1df3e1d640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-22T21:00:12.652 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-09-22T21:00:12.939 INFO:teuthology.misc.health.smithi160.stdout: 2023-09-22T21:00:12.939 INFO:teuthology.misc.health.smithi160.stdout:{"epoch":9,"fsid":"d0188f7d-2452-4b09-8037-934f8f90e7d1","created":"2023-09-22T21:00:04.679384+0000","modified":"2023-09-22T21:00:12.379349+0000","last_up_change":"2023-09-22T21:00:12.379349+0000","last_in_change":"2023-09-22T21:00:05.588062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"70ac7512-5619-4a29-825b-22268a0449d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6809","nonce":1670521548},{"type":"v1","addr":"172.21.15.160:6811","nonce":1670521548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6813","nonce":1670521548},{"type":"v1","addr":"172.21.15.160:6815","nonce":1670521548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6821","nonce":1670521548},{"type":"v1","addr":"172.21.15.160:6823","nonce":1670521548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6817","nonce":1670521548},{"type":"v1","addr":"172.21.15.160:6819","nonce":1670521548}]},"public_addr":"172.21.15.160:6811/1670521548","cluster_addr":"172.21.15.160:6815/1670521548","heartbeat_back_addr":"172.21.15.160:6823/1670521548","heartbeat_front_addr":"172.21.15.160:6819/1670521548","state":["exists","up"]},{"osd":1,"uuid":"d071b20e-b201-496c-bde9-a496d154c403","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":1361305861},{"type":"v1","addr":"172.21.15.160:6810","nonce":1361305861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":1361305861},{"type":"v1","addr":"172.21.15.160:6814","nonce":1361305861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":1361305861},{"type":"v1","addr":"172.21.15.160:6822","nonce":1361305861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":1361305861},{"type":"v1","addr":"172.21.15.160:6818","nonce":1361305861}]},"public_addr":"172.21.15.160:6810/1361305861","cluster_addr":"172.21.15.160:6814/1361305861","heartbeat_back_addr":"172.21.15.160:6822/1361305861","heartbeat_front_addr":"172.21.15.160:6818/1361305861","state":["exists","up"]},{"osd":2,"uuid":"aa84ccc7-e0be-49ac-8821-2e862cadfac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3973234119},{"type":"v1","addr":"172.21.15.160:6801","nonce":3973234119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3973234119},{"type":"v1","addr":"172.21.15.160:6803","nonce":3973234119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3973234119},{"type":"v1","addr":"172.21.15.160:6807","nonce":3973234119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3973234119},{"type":"v1","addr":"172.21.15.160:6805","nonce":3973234119}]},"public_addr":"172.21.15.160:6801/3973234119","cluster_addr":"172.21.15.160:6803/3973234119","heartbeat_back_addr":"172.21.15.160:6807/3973234119","heartbeat_front_addr":"172.21.15.160:6805/3973234119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T21:00:10.134139+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-22T21:00:12.956 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-09-22T21:00:12.956 INFO:tasks.ceph:Creating RBD pool 2023-09-22T21:00:12.956 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster c1 osd pool create rbd 8 2023-09-22T21:00:14.181 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'rbd' created 2023-09-22T21:00:14.205 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-22T21:00:15.191 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-22T21:00:15.222 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2023-09-22T21:00:15.223 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2023-09-22T21:00:15.223 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2023-09-22T21:00:15.223 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2023-09-22T21:00:15.224 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-22T21:00:15.576 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:15.593 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":4,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":3162430974},{"type":"v1","addr":"172.21.15.160:6825","nonce":3162430974}]},"active_addr":"172.21.15.160:6825/3162430974","active_change":"2023-09-22T21:00:09.342578+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.160:0","nonce":3109281175}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.160:0","nonce":213570127}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.160:0","nonce":357509464}]}]} 2023-09-22T21:00:15.596 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2023-09-22T21:00:15.596 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2023-09-22T21:00:15.596 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-09-22T21:00:15.924 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:15.924 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":11,"fsid":"d0188f7d-2452-4b09-8037-934f8f90e7d1","created":"2023-09-22T21:00:04.679384+0000","modified":"2023-09-22T21:00:15.183180+0000","last_up_change":"2023-09-22T21:00:12.379349+0000","last_in_change":"2023-09-22T21:00:05.588062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-22T21:00:13.279098+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-22T21:00:13.415183+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"70ac7512-5619-4a29-825b-22268a0449d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6809","nonce":1670521548},{"type":"v1","addr":"172.21.15.160:6811","nonce":1670521548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6813","nonce":1670521548},{"type":"v1","addr":"172.21.15.160:6815","nonce":1670521548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6821","nonce":1670521548},{"type":"v1","addr":"172.21.15.160:6823","nonce":1670521548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6817","nonce":1670521548},{"type":"v1","addr":"172.21.15.160:6819","nonce":1670521548}]},"public_addr":"172.21.15.160:6811/1670521548","cluster_addr":"172.21.15.160:6815/1670521548","heartbeat_back_addr":"172.21.15.160:6823/1670521548","heartbeat_front_addr":"172.21.15.160:6819/1670521548","state":["exists","up"]},{"osd":1,"uuid":"d071b20e-b201-496c-bde9-a496d154c403","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":1361305861},{"type":"v1","addr":"172.21.15.160:6810","nonce":1361305861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":1361305861},{"type":"v1","addr":"172.21.15.160:6814","nonce":1361305861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":1361305861},{"type":"v1","addr":"172.21.15.160:6822","nonce":1361305861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":1361305861},{"type":"v1","addr":"172.21.15.160:6818","nonce":1361305861}]},"public_addr":"172.21.15.160:6810/1361305861","cluster_addr":"172.21.15.160:6814/1361305861","heartbeat_back_addr":"172.21.15.160:6822/1361305861","heartbeat_front_addr":"172.21.15.160:6818/1361305861","state":["exists","up"]},{"osd":2,"uuid":"aa84ccc7-e0be-49ac-8821-2e862cadfac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3973234119},{"type":"v1","addr":"172.21.15.160:6801","nonce":3973234119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3973234119},{"type":"v1","addr":"172.21.15.160:6803","nonce":3973234119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3973234119},{"type":"v1","addr":"172.21.15.160:6807","nonce":3973234119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3973234119},{"type":"v1","addr":"172.21.15.160:6805","nonce":3973234119}]},"public_addr":"172.21.15.160:6801/3973234119","cluster_addr":"172.21.15.160:6803/3973234119","heartbeat_back_addr":"172.21.15.160:6807/3973234119","heartbeat_front_addr":"172.21.15.160:6805/3973234119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T21:00:10.134139+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T21:00:10.378492+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T21:00:10.358510+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-22T21:00:15.942 INFO:tasks.ceph.ceph_manager.c1:all up! 2023-09-22T21:00:15.943 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-09-22T21:00:16.232 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:16.233 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":12,"fsid":"d0188f7d-2452-4b09-8037-934f8f90e7d1","created":"2023-09-22T21:00:04.679384+0000","modified":"2023-09-22T21:00:16.193329+0000","last_up_change":"2023-09-22T21:00:12.379349+0000","last_in_change":"2023-09-22T21:00:05.588062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-22T21:00:13.279098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-22T21:00:13.415183+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"70ac7512-5619-4a29-825b-22268a0449d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6809","nonce":1670521548},{"type":"v1","addr":"172.21.15.160:6811","nonce":1670521548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6813","nonce":1670521548},{"type":"v1","addr":"172.21.15.160:6815","nonce":1670521548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6821","nonce":1670521548},{"type":"v1","addr":"172.21.15.160:6823","nonce":1670521548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6817","nonce":1670521548},{"type":"v1","addr":"172.21.15.160:6819","nonce":1670521548}]},"public_addr":"172.21.15.160:6811/1670521548","cluster_addr":"172.21.15.160:6815/1670521548","heartbeat_back_addr":"172.21.15.160:6823/1670521548","heartbeat_front_addr":"172.21.15.160:6819/1670521548","state":["exists","up"]},{"osd":1,"uuid":"d071b20e-b201-496c-bde9-a496d154c403","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":1361305861},{"type":"v1","addr":"172.21.15.160:6810","nonce":1361305861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":1361305861},{"type":"v1","addr":"172.21.15.160:6814","nonce":1361305861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":1361305861},{"type":"v1","addr":"172.21.15.160:6822","nonce":1361305861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":1361305861},{"type":"v1","addr":"172.21.15.160:6818","nonce":1361305861}]},"public_addr":"172.21.15.160:6810/1361305861","cluster_addr":"172.21.15.160:6814/1361305861","heartbeat_back_addr":"172.21.15.160:6822/1361305861","heartbeat_front_addr":"172.21.15.160:6818/1361305861","state":["exists","up"]},{"osd":2,"uuid":"aa84ccc7-e0be-49ac-8821-2e862cadfac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3973234119},{"type":"v1","addr":"172.21.15.160:6801","nonce":3973234119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3973234119},{"type":"v1","addr":"172.21.15.160:6803","nonce":3973234119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3973234119},{"type":"v1","addr":"172.21.15.160:6807","nonce":3973234119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3973234119},{"type":"v1","addr":"172.21.15.160:6805","nonce":3973234119}]},"public_addr":"172.21.15.160:6801/3973234119","cluster_addr":"172.21.15.160:6803/3973234119","heartbeat_back_addr":"172.21.15.160:6807/3973234119","heartbeat_front_addr":"172.21.15.160:6805/3973234119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T21:00:10.134139+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T21:00:10.378492+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T21:00:10.358510+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-22T21:00:16.251 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2023-09-22T21:00:16.251 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2023-09-22T21:00:16.251 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2023-09-22T21:00:16.406 INFO:teuthology.orchestra.run.smithi160.stdout:38654705667 2023-09-22T21:00:16.406 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-09-22T21:00:16.419 INFO:teuthology.orchestra.run.smithi160.stdout:34359738372 2023-09-22T21:00:16.419 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-09-22T21:00:16.420 INFO:teuthology.orchestra.run.smithi160.stdout:38654705668 2023-09-22T21:00:16.420 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-09-22T21:00:16.689 INFO:teuthology.orchestra.run.smithi160.stdout:38654705666 2023-09-22T21:00:16.707 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705666 for osd.1 2023-09-22T21:00:16.742 INFO:teuthology.orchestra.run.smithi160.stdout:34359738370 2023-09-22T21:00:16.752 INFO:teuthology.orchestra.run.smithi160.stdout:38654705667 2023-09-22T21:00:16.760 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738370 for osd.0 2023-09-22T21:00:16.769 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705668 got 38654705667 for osd.2 2023-09-22T21:00:17.708 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-09-22T21:00:17.761 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-09-22T21:00:17.770 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-09-22T21:00:18.014 INFO:teuthology.orchestra.run.smithi160.stdout:38654705666 2023-09-22T21:00:18.031 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705666 for osd.1 2023-09-22T21:00:18.056 INFO:teuthology.orchestra.run.smithi160.stdout:34359738370 2023-09-22T21:00:18.072 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738370 for osd.0 2023-09-22T21:00:18.093 INFO:teuthology.orchestra.run.smithi160.stdout:38654705667 2023-09-22T21:00:18.109 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705668 got 38654705667 for osd.2 2023-09-22T21:00:19.032 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-09-22T21:00:19.073 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-09-22T21:00:19.110 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-09-22T21:00:19.327 INFO:teuthology.orchestra.run.smithi160.stdout:38654705667 2023-09-22T21:00:19.345 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705667 for osd.1 2023-09-22T21:00:19.345 DEBUG:teuthology.parallel:result is None 2023-09-22T21:00:19.367 INFO:teuthology.orchestra.run.smithi160.stdout:34359738372 2023-09-22T21:00:19.384 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.0 2023-09-22T21:00:19.385 DEBUG:teuthology.parallel:result is None 2023-09-22T21:00:19.409 INFO:teuthology.orchestra.run.smithi160.stdout:38654705669 2023-09-22T21:00:19.426 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705668 got 38654705669 for osd.2 2023-09-22T21:00:19.426 DEBUG:teuthology.parallel:result is None 2023-09-22T21:00:19.426 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2023-09-22T21:00:19.426 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-09-22T21:00:19.724 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:19.725 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-09-22T21:00:19.740 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-09-22T21:00:19.348352+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82520,"kb_used_data":1456,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490280,"statfs":{"total":322122547200,"available":322038046720,"internally_reserved":0,"allocated":1490944,"data_stored":1261058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82963795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000498"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:16.340621+0000","last_change":"2023-09-22T21:00:15.197846+0000","last_active":"2023-09-22T21:00:16.340621+0000","last_peered":"2023-09-22T21:00:16.340621+0000","last_clean":"2023-09-22T21:00:16.340621+0000","last_became_active":"2023-09-22T21:00:15.197430+0000","last_became_peered":"2023-09-22T21:00:15.197430+0000","last_unstale":"2023-09-22T21:00:16.340621+0000","last_undegraded":"2023-09-22T21:00:16.340621+0000","last_fullsized":"2023-09-22T21:00:16.340621+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_clean_scrub_stamp":"2023-09-22T21:00:14.173392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:17:41.139460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:16.340702+0000","last_change":"2023-09-22T21:00:15.197243+0000","last_active":"2023-09-22T21:00:16.340702+0000","last_peered":"2023-09-22T21:00:16.340702+0000","last_clean":"2023-09-22T21:00:16.340702+0000","last_became_active":"2023-09-22T21:00:15.196456+0000","last_became_peered":"2023-09-22T21:00:15.196456+0000","last_unstale":"2023-09-22T21:00:16.340702+0000","last_undegraded":"2023-09-22T21:00:16.340702+0000","last_fullsized":"2023-09-22T21:00:16.340702+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_clean_scrub_stamp":"2023-09-22T21:00:14.173392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:22:31.882461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:16.207508+0000","last_change":"2023-09-22T21:00:15.201661+0000","last_active":"2023-09-22T21:00:16.207508+0000","last_peered":"2023-09-22T21:00:16.207508+0000","last_clean":"2023-09-22T21:00:16.207508+0000","last_became_active":"2023-09-22T21:00:15.201317+0000","last_became_peered":"2023-09-22T21:00:15.201317+0000","last_unstale":"2023-09-22T21:00:16.207508+0000","last_undegraded":"2023-09-22T21:00:16.207508+0000","last_fullsized":"2023-09-22T21:00:16.207508+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_clean_scrub_stamp":"2023-09-22T21:00:14.173392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:50:16.319273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:16.340666+0000","last_change":"2023-09-22T21:00:15.198221+0000","last_active":"2023-09-22T21:00:16.340666+0000","last_peered":"2023-09-22T21:00:16.340666+0000","last_clean":"2023-09-22T21:00:16.340666+0000","last_became_active":"2023-09-22T21:00:15.197902+0000","last_became_peered":"2023-09-22T21:00:15.197902+0000","last_unstale":"2023-09-22T21:00:16.340666+0000","last_undegraded":"2023-09-22T21:00:16.340666+0000","last_fullsized":"2023-09-22T21:00:16.340666+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_clean_scrub_stamp":"2023-09-22T21:00:14.173392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T02:37:34.156075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":138,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:16.207056+0000","last_change":"2023-09-22T21:00:15.201269+0000","last_active":"2023-09-22T21:00:16.207056+0000","last_peered":"2023-09-22T21:00:16.207056+0000","last_clean":"2023-09-22T21:00:16.207056+0000","last_became_active":"2023-09-22T21:00:15.200965+0000","last_became_peered":"2023-09-22T21:00:15.200965+0000","last_unstale":"2023-09-22T21:00:16.207056+0000","last_undegraded":"2023-09-22T21:00:16.207056+0000","last_fullsized":"2023-09-22T21:00:16.207056+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_clean_scrub_stamp":"2023-09-22T21:00:14.173392+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:46:20.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:16.340555+0000","last_change":"2023-09-22T21:00:15.197923+0000","last_active":"2023-09-22T21:00:16.340555+0000","last_peered":"2023-09-22T21:00:16.340555+0000","last_clean":"2023-09-22T21:00:16.340555+0000","last_became_active":"2023-09-22T21:00:15.197396+0000","last_became_peered":"2023-09-22T21:00:15.197396+0000","last_unstale":"2023-09-22T21:00:16.340555+0000","last_undegraded":"2023-09-22T21:00:16.340555+0000","last_fullsized":"2023-09-22T21:00:16.340555+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_clean_scrub_stamp":"2023-09-22T21:00:14.173392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T08:10:45.019469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-22T21:00:15.204981+0000","last_change":"2023-09-22T21:00:15.200430+0000","last_active":"2023-09-22T21:00:15.204981+0000","last_peered":"2023-09-22T21:00:15.204981+0000","last_clean":"2023-09-22T21:00:15.204981+0000","last_became_active":"2023-09-22T21:00:15.200151+0000","last_became_peered":"2023-09-22T21:00:15.200151+0000","last_unstale":"2023-09-22T21:00:15.204981+0000","last_undegraded":"2023-09-22T21:00:15.204981+0000","last_fullsized":"2023-09-22T21:00:15.204981+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_clean_scrub_stamp":"2023-09-22T21:00:14.173392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-23T23:57:23.159539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:16.207301+0000","last_change":"2023-09-22T21:00:15.201712+0000","last_active":"2023-09-22T21:00:16.207301+0000","last_peered":"2023-09-22T21:00:16.207301+0000","last_clean":"2023-09-22T21:00:16.207301+0000","last_became_active":"2023-09-22T21:00:15.201424+0000","last_became_peered":"2023-09-22T21:00:15.201424+0000","last_unstale":"2023-09-22T21:00:16.207301+0000","last_undegraded":"2023-09-22T21:00:16.207301+0000","last_fullsized":"2023-09-22T21:00:16.207301+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_clean_scrub_stamp":"2023-09-22T21:00:14.173392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T08:03:22.201955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:16.340571+0000","last_change":"2023-09-22T21:00:15.198389+0000","last_active":"2023-09-22T21:00:16.340571+0000","last_peered":"2023-09-22T21:00:16.340571+0000","last_clean":"2023-09-22T21:00:16.340571+0000","last_became_active":"2023-09-22T21:00:15.197794+0000","last_became_peered":"2023-09-22T21:00:15.197794+0000","last_unstale":"2023-09-22T21:00:16.340571+0000","last_undegraded":"2023-09-22T21:00:16.340571+0000","last_fullsized":"2023-09-22T21:00:16.340571+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_clean_scrub_stamp":"2023-09-22T21:00:14.173392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-23T21:08:54.734741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":696320,"data_stored":618071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]}]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":696320,"data_stored":618071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27120,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830480,"statfs":{"total":107374182400,"available":107346411520,"internally_reserved":0,"allocated":98304,"data_stored":24916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-22T21:00:19.742 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-09-22T21:00:20.017 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:20.017 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-09-22T21:00:20.034 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-09-22T21:00:19.348352+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82520,"kb_used_data":1456,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490280,"statfs":{"total":322122547200,"available":322038046720,"internally_reserved":0,"allocated":1490944,"data_stored":1261058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82963795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000498"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:16.340621+0000","last_change":"2023-09-22T21:00:15.197846+0000","last_active":"2023-09-22T21:00:16.340621+0000","last_peered":"2023-09-22T21:00:16.340621+0000","last_clean":"2023-09-22T21:00:16.340621+0000","last_became_active":"2023-09-22T21:00:15.197430+0000","last_became_peered":"2023-09-22T21:00:15.197430+0000","last_unstale":"2023-09-22T21:00:16.340621+0000","last_undegraded":"2023-09-22T21:00:16.340621+0000","last_fullsized":"2023-09-22T21:00:16.340621+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_clean_scrub_stamp":"2023-09-22T21:00:14.173392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:17:41.139460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:16.340702+0000","last_change":"2023-09-22T21:00:15.197243+0000","last_active":"2023-09-22T21:00:16.340702+0000","last_peered":"2023-09-22T21:00:16.340702+0000","last_clean":"2023-09-22T21:00:16.340702+0000","last_became_active":"2023-09-22T21:00:15.196456+0000","last_became_peered":"2023-09-22T21:00:15.196456+0000","last_unstale":"2023-09-22T21:00:16.340702+0000","last_undegraded":"2023-09-22T21:00:16.340702+0000","last_fullsized":"2023-09-22T21:00:16.340702+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_clean_scrub_stamp":"2023-09-22T21:00:14.173392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:22:31.882461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:16.207508+0000","last_change":"2023-09-22T21:00:15.201661+0000","last_active":"2023-09-22T21:00:16.207508+0000","last_peered":"2023-09-22T21:00:16.207508+0000","last_clean":"2023-09-22T21:00:16.207508+0000","last_became_active":"2023-09-22T21:00:15.201317+0000","last_became_peered":"2023-09-22T21:00:15.201317+0000","last_unstale":"2023-09-22T21:00:16.207508+0000","last_undegraded":"2023-09-22T21:00:16.207508+0000","last_fullsized":"2023-09-22T21:00:16.207508+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_clean_scrub_stamp":"2023-09-22T21:00:14.173392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:50:16.319273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:16.340666+0000","last_change":"2023-09-22T21:00:15.198221+0000","last_active":"2023-09-22T21:00:16.340666+0000","last_peered":"2023-09-22T21:00:16.340666+0000","last_clean":"2023-09-22T21:00:16.340666+0000","last_became_active":"2023-09-22T21:00:15.197902+0000","last_became_peered":"2023-09-22T21:00:15.197902+0000","last_unstale":"2023-09-22T21:00:16.340666+0000","last_undegraded":"2023-09-22T21:00:16.340666+0000","last_fullsized":"2023-09-22T21:00:16.340666+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_clean_scrub_stamp":"2023-09-22T21:00:14.173392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T02:37:34.156075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":138,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:16.207056+0000","last_change":"2023-09-22T21:00:15.201269+0000","last_active":"2023-09-22T21:00:16.207056+0000","last_peered":"2023-09-22T21:00:16.207056+0000","last_clean":"2023-09-22T21:00:16.207056+0000","last_became_active":"2023-09-22T21:00:15.200965+0000","last_became_peered":"2023-09-22T21:00:15.200965+0000","last_unstale":"2023-09-22T21:00:16.207056+0000","last_undegraded":"2023-09-22T21:00:16.207056+0000","last_fullsized":"2023-09-22T21:00:16.207056+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_clean_scrub_stamp":"2023-09-22T21:00:14.173392+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:46:20.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:16.340555+0000","last_change":"2023-09-22T21:00:15.197923+0000","last_active":"2023-09-22T21:00:16.340555+0000","last_peered":"2023-09-22T21:00:16.340555+0000","last_clean":"2023-09-22T21:00:16.340555+0000","last_became_active":"2023-09-22T21:00:15.197396+0000","last_became_peered":"2023-09-22T21:00:15.197396+0000","last_unstale":"2023-09-22T21:00:16.340555+0000","last_undegraded":"2023-09-22T21:00:16.340555+0000","last_fullsized":"2023-09-22T21:00:16.340555+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_clean_scrub_stamp":"2023-09-22T21:00:14.173392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T08:10:45.019469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-22T21:00:15.204981+0000","last_change":"2023-09-22T21:00:15.200430+0000","last_active":"2023-09-22T21:00:15.204981+0000","last_peered":"2023-09-22T21:00:15.204981+0000","last_clean":"2023-09-22T21:00:15.204981+0000","last_became_active":"2023-09-22T21:00:15.200151+0000","last_became_peered":"2023-09-22T21:00:15.200151+0000","last_unstale":"2023-09-22T21:00:15.204981+0000","last_undegraded":"2023-09-22T21:00:15.204981+0000","last_fullsized":"2023-09-22T21:00:15.204981+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_clean_scrub_stamp":"2023-09-22T21:00:14.173392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-23T23:57:23.159539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:16.207301+0000","last_change":"2023-09-22T21:00:15.201712+0000","last_active":"2023-09-22T21:00:16.207301+0000","last_peered":"2023-09-22T21:00:16.207301+0000","last_clean":"2023-09-22T21:00:16.207301+0000","last_became_active":"2023-09-22T21:00:15.201424+0000","last_became_peered":"2023-09-22T21:00:15.201424+0000","last_unstale":"2023-09-22T21:00:16.207301+0000","last_undegraded":"2023-09-22T21:00:16.207301+0000","last_fullsized":"2023-09-22T21:00:16.207301+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_clean_scrub_stamp":"2023-09-22T21:00:14.173392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T08:03:22.201955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:16.340571+0000","last_change":"2023-09-22T21:00:15.198389+0000","last_active":"2023-09-22T21:00:16.340571+0000","last_peered":"2023-09-22T21:00:16.340571+0000","last_clean":"2023-09-22T21:00:16.340571+0000","last_became_active":"2023-09-22T21:00:15.197794+0000","last_became_peered":"2023-09-22T21:00:15.197794+0000","last_unstale":"2023-09-22T21:00:16.340571+0000","last_undegraded":"2023-09-22T21:00:16.340571+0000","last_fullsized":"2023-09-22T21:00:16.340571+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:14.173392+0000","last_clean_scrub_stamp":"2023-09-22T21:00:14.173392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-23T21:08:54.734741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":696320,"data_stored":618071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]}]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":696320,"data_stored":618071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27120,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830480,"statfs":{"total":107374182400,"available":107346411520,"internally_reserved":0,"allocated":98304,"data_stored":24916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-22T21:00:20.035 INFO:tasks.ceph.ceph_manager.c1:clean! 2023-09-22T21:00:20.035 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2023-09-22T21:00:20.035 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2023-09-22T21:00:20.036 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2023-09-22T21:00:20.338 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:20.338 INFO:teuthology.orchestra.run.smithi160.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-09-22T21:00:20.354 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2023-09-22T21:00:20.354 INFO:teuthology.run_tasks:Running task ceph... 2023-09-22T21:00:20.363 INFO:tasks.ceph:Creating ceph cluster c2... 2023-09-22T21:00:20.363 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-22T21:00:20.364 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mkogan-2023-09-22_17:38:01-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7401166', 'branch': 'rgw-wip-t62141-valg-supp_i008', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi}', 'email': 'mkogan@redhat.com', 'first_in_suite': False, 'job_id': '7401166', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'mkogan-2023-09-22_17:38:01-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'rgw-wip-t62141-valg-supp_i008'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '03616b4330080cadf20be440d92e33ca90de254a'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '03616b4330080cadf20be440d92e33ca90de254a'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'workunit': {'branch': 'wip-qa-rgw-verify-ubuntu-003', 'sha1': '691e03976a4d019c58bad99761055f3266556055'}}, 'owner': 'scheduled_mkogan@teuthology', 'priority': 150, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 1924, 'sha1': '03616b4330080cadf20be440d92e33ca90de254a', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-qa-rgw-verify-ubuntu-003', 'suite_path': '/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/mkogan1/ceph.git', 'suite_sha1': '691e03976a4d019c58bad99761055f3266556055', 'targets': {'smithi160.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJc+XzPRS2TrfwsLPPiLo/tNCa2RGt6AcCR/nSya29UpF+GTgm+TxeVeHmZSk/cnLRECsVqSxIMEtf26RoWCMWI=', 'smithi204.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEqnq9KNsYblkwRNcJhKaEyCeQLeszT9O0Xr2v9A/EES2mM7cdfy9fws30xx5mDF1ikl+kZ3V+/mYuCmhT9Lpvw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '03616b4330080cadf20be440d92e33ca90de254a'}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '03616b4330080cadf20be440d92e33ca90de254a'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-22_17:38:01', 'tube': 'smithi', 'user': 'mkogan', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-22T21:00:20.364 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-09-22T21:00:20.366 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-09-22T21:00:20.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo install -d -m0777 -- /var/run/ceph 2023-09-22T21:00:20.414 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/run/ceph 2023-09-22T21:00:20.429 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T21:00:20.429 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2023-09-22T21:00:20.477 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-22T21:00:20.477 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 2023-09-22T21:00:20.529 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-22T21:00:20.529 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-22T21:00:20.529 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 886 Links: 1 2023-09-22T21:00:20.529 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-22T21:00:20.529 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2023-09-22 20:54:38.202863241 +0000 2023-09-22T21:00:20.529 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2023-09-22 20:54:37.874866171 +0000 2023-09-22T21:00:20.530 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2023-09-22 20:54:37.874866171 +0000 2023-09-22T21:00:20.530 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2023-09-22T21:00:20.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-22T21:00:20.584 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2023-09-22T21:00:20.584 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2023-09-22T21:00:20.584 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000443844 s, 1.2 MB/s 2023-09-22T21:00:20.585 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-22T21:00:20.634 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 2023-09-22T21:00:20.680 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-22T21:00:20.680 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-22T21:00:20.681 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 905 Links: 1 2023-09-22T21:00:20.681 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-22T21:00:20.681 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2023-09-22 20:54:38.814857775 +0000 2023-09-22T21:00:20.681 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2023-09-22 20:54:38.446861061 +0000 2023-09-22T21:00:20.681 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2023-09-22 20:54:38.446861061 +0000 2023-09-22T21:00:20.681 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2023-09-22T21:00:20.681 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-22T21:00:20.734 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2023-09-22T21:00:20.734 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2023-09-22T21:00:20.735 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000431018 s, 1.2 MB/s 2023-09-22T21:00:20.736 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-22T21:00:20.786 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 2023-09-22T21:00:20.832 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-22T21:00:20.832 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-22T21:00:20.832 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 922 Links: 1 2023-09-22T21:00:20.833 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-22T21:00:20.833 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2023-09-22 20:54:39.350852987 +0000 2023-09-22T21:00:20.833 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2023-09-22 20:54:39.030855846 +0000 2023-09-22T21:00:20.833 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2023-09-22 20:54:39.030855846 +0000 2023-09-22T21:00:20.833 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2023-09-22T21:00:20.833 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-22T21:00:20.886 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2023-09-22T21:00:20.887 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2023-09-22T21:00:20.887 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000316188 s, 1.6 MB/s 2023-09-22T21:00:20.888 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-22T21:00:20.938 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 2023-09-22T21:00:20.988 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-22T21:00:20.988 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-22T21:00:20.988 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 940 Links: 1 2023-09-22T21:00:20.989 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-22T21:00:20.989 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2023-09-22 20:54:39.974847413 +0000 2023-09-22T21:00:20.989 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2023-09-22 20:54:39.590850843 +0000 2023-09-22T21:00:20.989 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2023-09-22 20:54:39.590850843 +0000 2023-09-22T21:00:20.989 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2023-09-22T21:00:20.989 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-22T21:00:21.043 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2023-09-22T21:00:21.043 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2023-09-22T21:00:21.043 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000388597 s, 1.3 MB/s 2023-09-22T21:00:21.044 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-22T21:00:21.094 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-22T21:00:21.094 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-22T21:00:21.094 INFO:tasks.ceph:Generating config... 2023-09-22T21:00:21.097 INFO:tasks.ceph:[client] debug rgw = 20 2023-09-22T21:00:21.098 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-09-22T21:00:21.098 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-09-22T21:00:21.098 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-09-22T21:00:21.098 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-09-22T21:00:21.098 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-09-22T21:00:21.098 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-09-22T21:00:21.098 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-09-22T21:00:21.098 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-09-22T21:00:21.098 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-09-22T21:00:21.098 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-09-22T21:00:21.099 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-09-22T21:00:21.099 INFO:tasks.ceph:[client] setgroup = ceph 2023-09-22T21:00:21.099 INFO:tasks.ceph:[client] setuser = ceph 2023-09-22T21:00:21.099 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-09-22T21:00:21.099 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-22T21:00:21.099 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-22T21:00:21.099 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-22T21:00:21.099 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-22T21:00:21.099 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-22T21:00:21.099 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-22T21:00:21.099 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-22T21:00:21.099 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-09-22T21:00:21.099 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-09-22T21:00:21.100 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-09-22T21:00:21.100 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-09-22T21:00:21.100 INFO:tasks.ceph:Setting up c2.mon.a... 2023-09-22T21:00:21.100 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2023-09-22T21:00:21.167 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/c2.keyring 2023-09-22T21:00:21.177 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2023-09-22T21:00:21.251 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-09-22T21:00:21.307 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.204')] 2023-09-22T21:00:21.307 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.204'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-09-22T21:00:21.308 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T21:00:21.308 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-22T21:00:21.352 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.204 --print /home/ubuntu/cephtest/c2.monmap 2023-09-22T21:00:21.430 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:00:21.430 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:00:21.430 INFO:teuthology.orchestra.run.smithi204.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2023-09-22T21:00:21.431 INFO:teuthology.orchestra.run.smithi204.stdout:monmaptool: generated fsid 6b89fe4e-6e99-42c4-9d3a-9f184acab2d9 2023-09-22T21:00:21.431 INFO:teuthology.orchestra.run.smithi204.stdout:setting min_mon_release = pacific 2023-09-22T21:00:21.431 INFO:teuthology.orchestra.run.smithi204.stdout:epoch 0 2023-09-22T21:00:21.431 INFO:teuthology.orchestra.run.smithi204.stdout:fsid 6b89fe4e-6e99-42c4-9d3a-9f184acab2d9 2023-09-22T21:00:21.431 INFO:teuthology.orchestra.run.smithi204.stdout:last_changed 2023-09-22T21:00:21.434091+0000 2023-09-22T21:00:21.431 INFO:teuthology.orchestra.run.smithi204.stdout:created 2023-09-22T21:00:21.434091+0000 2023-09-22T21:00:21.431 INFO:teuthology.orchestra.run.smithi204.stdout:min_mon_release 16 (pacific) 2023-09-22T21:00:21.431 INFO:teuthology.orchestra.run.smithi204.stdout:election_strategy: 1 2023-09-22T21:00:21.431 INFO:teuthology.orchestra.run.smithi204.stdout:0: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.a 2023-09-22T21:00:21.431 INFO:teuthology.orchestra.run.smithi204.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2023-09-22T21:00:21.432 DEBUG:teuthology.orchestra.run.smithi204:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-22T21:00:21.480 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 6b89fe4e-6e99-42c4-9d3a-9f184acab2d9... 2023-09-22T21:00:21.481 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-09-22T21:00:21.483 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-09-22T21:00:21.530 INFO:teuthology.orchestra.run.smithi160.stdout:[global] 2023-09-22T21:00:21.530 INFO:teuthology.orchestra.run.smithi160.stdout: chdir = "" 2023-09-22T21:00:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-22T21:00:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: auth supported = cephx 2023-09-22T21:00:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.532 INFO:teuthology.orchestra.run.smithi160.stdout: filestore xattr use omap = true 2023-09-22T21:00:21.532 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.532 INFO:teuthology.orchestra.run.smithi160.stdout: mon clock drift allowed = 1.000 2023-09-22T21:00:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: osd crush chooseleaf type = 0 2023-09-22T21:00:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: auth debug = true 2023-09-22T21:00:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.534 INFO:teuthology.orchestra.run.smithi160.stdout: ms die on old message = true 2023-09-22T21:00:21.535 INFO:teuthology.orchestra.run.smithi160.stdout: ms die on bug = true 2023-09-22T21:00:21.535 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-22T21:00:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: mon pg warn max object skew = 0 2023-09-22T21:00:21.537 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.537 INFO:teuthology.orchestra.run.smithi160.stdout: # disable pg_autoscaler by default for new pools 2023-09-22T21:00:21.537 INFO:teuthology.orchestra.run.smithi160.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-22T21:00:21.537 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.538 INFO:teuthology.orchestra.run.smithi160.stdout: osd pool default size = 2 2023-09-22T21:00:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd allow primary affinity = true 2023-09-22T21:00:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd allow pg remap = true 2023-09-22T21:00:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on legacy crush tunables = false 2023-09-22T21:00:21.542 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on crush straw calc version zero = false 2023-09-22T21:00:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on no sortbitwise = false 2023-09-22T21:00:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on osd down out interval zero = false 2023-09-22T21:00:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on too few osds = false 2023-09-22T21:00:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-22T21:00:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-22T21:00:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: mon_allow_pool_size_one = true 2023-09-22T21:00:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-22T21:00:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.548 INFO:teuthology.orchestra.run.smithi160.stdout: osd default data pool replay window = 5 2023-09-22T21:00:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: mon allow pool delete = true 2023-09-22T21:00:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: mon cluster log file level = debug 2023-09-22T21:00:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: debug asserts on shutdown = true 2023-09-22T21:00:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: mon health detail to clog = false 2023-09-22T21:00:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: mon host = 172.21.15.204 2023-09-22T21:00:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: fsid = 6b89fe4e-6e99-42c4-9d3a-9f184acab2d9 2023-09-22T21:00:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.554 INFO:teuthology.orchestra.run.smithi160.stdout:[osd] 2023-09-22T21:00:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: osd journal size = 100 2023-09-22T21:00:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: osd scrub load threshold = 5.0 2023-09-22T21:00:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: osd scrub max interval = 600 2023-09-22T21:00:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: osd mclock profile = high_recovery_ops 2023-09-22T21:00:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: osd recover clone overlap = true 2023-09-22T21:00:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: osd recovery max chunk = 1048576 2023-09-22T21:00:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug shutdown = true 2023-09-22T21:00:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug op order = true 2023-09-22T21:00:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug verify stray on activate = true 2023-09-22T21:00:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: osd open classes on start = true 2023-09-22T21:00:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug pg log writeout = true 2023-09-22T21:00:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: osd deep scrub update digest min age = 30 2023-09-22T21:00:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: osd map max advance = 10 2023-09-22T21:00:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: journal zero on create = true 2023-09-22T21:00:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: filestore ondisk finisher threads = 3 2023-09-22T21:00:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: filestore apply finisher threads = 3 2023-09-22T21:00:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: bdev debug aio = true 2023-09-22T21:00:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug misdirected ops = true 2023-09-22T21:00:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2023-09-22T21:00:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: debug osd = 20 2023-09-22T21:00:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: rocksdb delete range threshold = 0 2023-09-22T21:00:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.558 INFO:teuthology.orchestra.run.smithi160.stdout:[mgr] 2023-09-22T21:00:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2023-09-22T21:00:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: debug mgr = 20 2023-09-22T21:00:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: debug mon = 20 2023-09-22T21:00:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: debug auth = 20 2023-09-22T21:00:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min pgs per osd = 4 2023-09-22T21:00:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min bytes per osd = 10 2023-09-22T21:00:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: mgr/telemetry/nag = false 2023-09-22T21:00:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.559 INFO:teuthology.orchestra.run.smithi160.stdout:[mon] 2023-09-22T21:00:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2023-09-22T21:00:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: debug mon = 20 2023-09-22T21:00:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: debug paxos = 20 2023-09-22T21:00:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: debug auth = 20 2023-09-22T21:00:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: mon data avail warn = 5 2023-09-22T21:00:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: mon mgr mkfs grace = 240 2023-09-22T21:00:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min pgs per osd = 4 2023-09-22T21:00:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd reporter subtree level = osd 2023-09-22T21:00:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd prime pg temp = true 2023-09-22T21:00:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min bytes per osd = 10 2023-09-22T21:00:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-22T21:00:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: auth mon ticket ttl = 660 # 11m 2023-09-22T21:00:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: auth service ticket ttl = 240 # 4m 2023-09-22T21:00:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: # don't complain about insecure global_id in the test suite 2023-09-22T21:00:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-22T21:00:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-22T21:00:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: # 1m isn't quite enough 2023-09-22T21:00:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: mon_down_mkfs_grace = 2m 2023-09-22T21:00:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_filestore_osds = false 2023-09-22T21:00:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-09-22T21:00:21.561 INFO:teuthology.orchestra.run.smithi160.stdout:[client] 2023-09-22T21:00:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: rgw cache enabled = true 2023-09-22T21:00:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: rgw enable ops log = true 2023-09-22T21:00:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: rgw enable usage log = true 2023-09-22T21:00:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-22T21:00:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-22T21:00:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: debug rgw = 20 2023-09-22T21:00:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: debug rgw sync = 20 2023-09-22T21:00:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: rgw crypt require ssl = False 2023-09-22T21:00:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: rgw crypt s3 kms backend = testing 2023-09-22T21:00:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-09-22T21:00:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: rgw curl low speed time = 300 2023-09-22T21:00:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: rgw data log num shards = 4 2023-09-22T21:00:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: rgw md log max shards = 4 2023-09-22T21:00:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: rgw sync data inject err probability = 0.1 2023-09-22T21:00:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: rgw sync log trim interval = 0 2023-09-22T21:00:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: rgw sync meta inject err probability = 0.1 2023-09-22T21:00:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: rgw sync obj etag verify = True 2023-09-22T21:00:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: setgroup = ceph 2023-09-22T21:00:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: setuser = ceph 2023-09-22T21:00:21.563 INFO:teuthology.orchestra.run.smithi160.stdout:[mon.a] 2023-09-22T21:00:21.563 INFO:teuthology.orchestra.run.smithi160.stdout:[client.0] 2023-09-22T21:00:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: rgw data notify interval msec = 0 2023-09-22T21:00:21.563 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.0] 2023-09-22T21:00:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: osd_max_omap_entries_per_request = 10 2023-09-22T21:00:21.563 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.1] 2023-09-22T21:00:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: osd_max_omap_entries_per_request = 1000 2023-09-22T21:00:21.564 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.2] 2023-09-22T21:00:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: osd_max_omap_entries_per_request = 10000 2023-09-22T21:00:21.564 INFO:teuthology.orchestra.run.smithi204.stdout:[global] 2023-09-22T21:00:21.565 INFO:teuthology.orchestra.run.smithi204.stdout: chdir = "" 2023-09-22T21:00:21.565 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-22T21:00:21.565 INFO:teuthology.orchestra.run.smithi204.stdout: auth supported = cephx 2023-09-22T21:00:21.565 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.565 INFO:teuthology.orchestra.run.smithi204.stdout: filestore xattr use omap = true 2023-09-22T21:00:21.565 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.565 INFO:teuthology.orchestra.run.smithi204.stdout: mon clock drift allowed = 1.000 2023-09-22T21:00:21.565 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.566 INFO:teuthology.orchestra.run.smithi204.stdout: osd crush chooseleaf type = 0 2023-09-22T21:00:21.566 INFO:teuthology.orchestra.run.smithi204.stdout: auth debug = true 2023-09-22T21:00:21.566 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.566 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on old message = true 2023-09-22T21:00:21.566 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on bug = true 2023-09-22T21:00:21.566 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.566 INFO:teuthology.orchestra.run.smithi204.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-22T21:00:21.566 INFO:teuthology.orchestra.run.smithi204.stdout: mon pg warn max object skew = 0 2023-09-22T21:00:21.567 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.567 INFO:teuthology.orchestra.run.smithi204.stdout: # disable pg_autoscaler by default for new pools 2023-09-22T21:00:21.567 INFO:teuthology.orchestra.run.smithi204.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-22T21:00:21.567 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.567 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default size = 2 2023-09-22T21:00:21.567 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.567 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow primary affinity = true 2023-09-22T21:00:21.568 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow pg remap = true 2023-09-22T21:00:21.568 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on legacy crush tunables = false 2023-09-22T21:00:21.568 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on crush straw calc version zero = false 2023-09-22T21:00:21.568 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on no sortbitwise = false 2023-09-22T21:00:21.568 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on osd down out interval zero = false 2023-09-22T21:00:21.568 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on too few osds = false 2023-09-22T21:00:21.568 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-22T21:00:21.568 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-22T21:00:21.568 INFO:teuthology.orchestra.run.smithi204.stdout: mon_allow_pool_size_one = true 2023-09-22T21:00:21.569 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.569 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-22T21:00:21.569 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.569 INFO:teuthology.orchestra.run.smithi204.stdout: osd default data pool replay window = 5 2023-09-22T21:00:21.569 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.569 INFO:teuthology.orchestra.run.smithi204.stdout: mon allow pool delete = true 2023-09-22T21:00:21.569 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.569 INFO:teuthology.orchestra.run.smithi204.stdout: mon cluster log file level = debug 2023-09-22T21:00:21.569 INFO:teuthology.orchestra.run.smithi204.stdout: debug asserts on shutdown = true 2023-09-22T21:00:21.569 INFO:teuthology.orchestra.run.smithi204.stdout: mon health detail to clog = false 2023-09-22T21:00:21.569 INFO:teuthology.orchestra.run.smithi204.stdout: mon host = 172.21.15.204 2023-09-22T21:00:21.570 INFO:teuthology.orchestra.run.smithi204.stdout: fsid = 6b89fe4e-6e99-42c4-9d3a-9f184acab2d9 2023-09-22T21:00:21.570 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.570 INFO:teuthology.orchestra.run.smithi204.stdout:[osd] 2023-09-22T21:00:21.570 INFO:teuthology.orchestra.run.smithi204.stdout: osd journal size = 100 2023-09-22T21:00:21.570 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.570 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub load threshold = 5.0 2023-09-22T21:00:21.570 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub max interval = 600 2023-09-22T21:00:21.570 INFO:teuthology.orchestra.run.smithi204.stdout: osd mclock profile = high_recovery_ops 2023-09-22T21:00:21.570 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.570 INFO:teuthology.orchestra.run.smithi204.stdout: osd recover clone overlap = true 2023-09-22T21:00:21.570 INFO:teuthology.orchestra.run.smithi204.stdout: osd recovery max chunk = 1048576 2023-09-22T21:00:21.571 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.571 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug shutdown = true 2023-09-22T21:00:21.571 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug op order = true 2023-09-22T21:00:21.571 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify stray on activate = true 2023-09-22T21:00:21.571 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.571 INFO:teuthology.orchestra.run.smithi204.stdout: osd open classes on start = true 2023-09-22T21:00:21.571 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug pg log writeout = true 2023-09-22T21:00:21.571 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.571 INFO:teuthology.orchestra.run.smithi204.stdout: osd deep scrub update digest min age = 30 2023-09-22T21:00:21.571 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.572 INFO:teuthology.orchestra.run.smithi204.stdout: osd map max advance = 10 2023-09-22T21:00:21.572 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.572 INFO:teuthology.orchestra.run.smithi204.stdout: journal zero on create = true 2023-09-22T21:00:21.572 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.572 INFO:teuthology.orchestra.run.smithi204.stdout: filestore ondisk finisher threads = 3 2023-09-22T21:00:21.572 INFO:teuthology.orchestra.run.smithi204.stdout: filestore apply finisher threads = 3 2023-09-22T21:00:21.572 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.572 INFO:teuthology.orchestra.run.smithi204.stdout: bdev debug aio = true 2023-09-22T21:00:21.572 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug misdirected ops = true 2023-09-22T21:00:21.573 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2023-09-22T21:00:21.573 INFO:teuthology.orchestra.run.smithi204.stdout: debug osd = 20 2023-09-22T21:00:21.573 INFO:teuthology.orchestra.run.smithi204.stdout: rocksdb delete range threshold = 0 2023-09-22T21:00:21.573 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.573 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr] 2023-09-22T21:00:21.573 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2023-09-22T21:00:21.573 INFO:teuthology.orchestra.run.smithi204.stdout: debug mgr = 20 2023-09-22T21:00:21.573 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2023-09-22T21:00:21.573 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2023-09-22T21:00:21.574 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2023-09-22T21:00:21.574 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2023-09-22T21:00:21.574 INFO:teuthology.orchestra.run.smithi204.stdout: mgr/telemetry/nag = false 2023-09-22T21:00:21.574 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.574 INFO:teuthology.orchestra.run.smithi204.stdout:[mon] 2023-09-22T21:00:21.574 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2023-09-22T21:00:21.574 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2023-09-22T21:00:21.574 INFO:teuthology.orchestra.run.smithi204.stdout: debug paxos = 20 2023-09-22T21:00:21.575 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2023-09-22T21:00:21.575 INFO:teuthology.orchestra.run.smithi204.stdout: mon data avail warn = 5 2023-09-22T21:00:21.575 INFO:teuthology.orchestra.run.smithi204.stdout: mon mgr mkfs grace = 240 2023-09-22T21:00:21.575 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2023-09-22T21:00:21.575 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd reporter subtree level = osd 2023-09-22T21:00:21.575 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd prime pg temp = true 2023-09-22T21:00:21.575 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2023-09-22T21:00:21.575 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.575 INFO:teuthology.orchestra.run.smithi204.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-22T21:00:21.575 INFO:teuthology.orchestra.run.smithi204.stdout: auth mon ticket ttl = 660 # 11m 2023-09-22T21:00:21.576 INFO:teuthology.orchestra.run.smithi204.stdout: auth service ticket ttl = 240 # 4m 2023-09-22T21:00:21.576 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.576 INFO:teuthology.orchestra.run.smithi204.stdout: # don't complain about insecure global_id in the test suite 2023-09-22T21:00:21.576 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-22T21:00:21.576 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-22T21:00:21.576 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.576 INFO:teuthology.orchestra.run.smithi204.stdout: # 1m isn't quite enough 2023-09-22T21:00:21.576 INFO:teuthology.orchestra.run.smithi204.stdout: mon_down_mkfs_grace = 2m 2023-09-22T21:00:21.576 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.576 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_filestore_osds = false 2023-09-22T21:00:21.576 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:21.577 INFO:teuthology.orchestra.run.smithi204.stdout:[client] 2023-09-22T21:00:21.577 INFO:teuthology.orchestra.run.smithi204.stdout: rgw cache enabled = true 2023-09-22T21:00:21.577 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable ops log = true 2023-09-22T21:00:21.577 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable usage log = true 2023-09-22T21:00:21.577 INFO:teuthology.orchestra.run.smithi204.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-22T21:00:21.577 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-22T21:00:21.577 INFO:teuthology.orchestra.run.smithi204.stdout: debug rgw = 20 2023-09-22T21:00:21.577 INFO:teuthology.orchestra.run.smithi204.stdout: debug rgw sync = 20 2023-09-22T21:00:21.577 INFO:teuthology.orchestra.run.smithi204.stdout: rgw crypt require ssl = False 2023-09-22T21:00:21.577 INFO:teuthology.orchestra.run.smithi204.stdout: rgw crypt s3 kms backend = testing 2023-09-22T21:00:21.578 INFO:teuthology.orchestra.run.smithi204.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-09-22T21:00:21.578 INFO:teuthology.orchestra.run.smithi204.stdout: rgw curl low speed time = 300 2023-09-22T21:00:21.578 INFO:teuthology.orchestra.run.smithi204.stdout: rgw data log num shards = 4 2023-09-22T21:00:21.578 INFO:teuthology.orchestra.run.smithi204.stdout: rgw md log max shards = 4 2023-09-22T21:00:21.578 INFO:teuthology.orchestra.run.smithi204.stdout: rgw sync data inject err probability = 0.1 2023-09-22T21:00:21.578 INFO:teuthology.orchestra.run.smithi204.stdout: rgw sync log trim interval = 0 2023-09-22T21:00:21.578 INFO:teuthology.orchestra.run.smithi204.stdout: rgw sync meta inject err probability = 0.1 2023-09-22T21:00:21.579 INFO:teuthology.orchestra.run.smithi204.stdout: rgw sync obj etag verify = True 2023-09-22T21:00:21.579 INFO:teuthology.orchestra.run.smithi204.stdout: setgroup = ceph 2023-09-22T21:00:21.579 INFO:teuthology.orchestra.run.smithi204.stdout: setuser = ceph 2023-09-22T21:00:21.579 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.a] 2023-09-22T21:00:21.579 INFO:teuthology.orchestra.run.smithi204.stdout:[client.0] 2023-09-22T21:00:21.579 INFO:teuthology.orchestra.run.smithi204.stdout: rgw data notify interval msec = 0 2023-09-22T21:00:21.579 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.0] 2023-09-22T21:00:21.579 INFO:teuthology.orchestra.run.smithi204.stdout: osd_max_omap_entries_per_request = 10 2023-09-22T21:00:21.579 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.1] 2023-09-22T21:00:21.580 INFO:teuthology.orchestra.run.smithi204.stdout: osd_max_omap_entries_per_request = 1000 2023-09-22T21:00:21.580 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.2] 2023-09-22T21:00:21.580 INFO:teuthology.orchestra.run.smithi204.stdout: osd_max_omap_entries_per_request = 10000 2023-09-22T21:00:21.581 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2023-09-22T21:00:21.582 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2023-09-22T21:00:21.680 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-22T21:00:21.680 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T21:00:21.680 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2023-09-22T21:00:21.691 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T21:00:21.692 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2023-09-22T21:00:21.748 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi160.front.sepia.ceph.com 2023-09-22T21:00:21.748 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T21:00:21.748 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/c2.keyring 2023-09-22T21:00:21.749 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-09-22T21:00:21.825 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-09-22T21:00:21.826 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-09-22T21:00:21.940 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi204.front.sepia.ceph.com 2023-09-22T21:00:21.940 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T21:00:21.940 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/c2.keyring 2023-09-22T21:00:21.940 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-09-22T21:00:22.010 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T21:00:22.010 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-09-22T21:00:22.113 INFO:tasks.ceph:Setting up mon nodes... 2023-09-22T21:00:22.113 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-22T21:00:22.113 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2023-09-22T21:00:22.204 INFO:teuthology.orchestra.run.smithi204.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2023-09-22T21:00:22.207 INFO:tasks.ceph:Setting up mds nodes... 2023-09-22T21:00:22.207 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-22T21:00:22.207 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2023-09-22T21:00:22.288 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/c2.client.0.keyring 2023-09-22T21:00:22.300 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2023-09-22T21:00:22.386 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/c2.client.1.keyring 2023-09-22T21:00:22.396 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-22T21:00:22.396 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi160.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-22T21:00:22.396 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2023-09-22T21:00:22.454 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-22T21:00:22.454 INFO:tasks.ceph:role: c2.osd.0 2023-09-22T21:00:22.454 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com 2023-09-22T21:00:22.455 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-22T21:00:22.524 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-22T21:00:22.524 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-22T21:00:22.524 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-22T21:00:22.524 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-22T21:00:22.524 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-22T21:00:22.524 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2023-09-22T21:00:22.524 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-22T21:00:22.524 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-22T21:00:22.525 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-22T21:00:22.525 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-22T21:00:22.990 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2023-09-22T21:00:22.992 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2023-09-22T21:00:22.992 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2023-09-22T21:00:23.093 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2023-09-22T21:00:23.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T21:00:23.106 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: /sbin/restorecon: command not found 2023-09-22T21:00:23.106 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2023-09-22T21:00:23.158 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-22T21:00:23.158 INFO:tasks.ceph:role: c2.osd.1 2023-09-22T21:00:23.158 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com 2023-09-22T21:00:23.158 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-22T21:00:23.220 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-22T21:00:23.220 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-22T21:00:23.220 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-22T21:00:23.221 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-22T21:00:23.221 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-22T21:00:23.221 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2023-09-22T21:00:23.221 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-22T21:00:23.221 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-22T21:00:23.221 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-22T21:00:23.221 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-22T21:00:23.681 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2023-09-22T21:00:23.683 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2023-09-22T21:00:23.683 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2023-09-22T21:00:23.840 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2023-09-22T21:00:23.850 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: /sbin/restorecon: command not found 2023-09-22T21:00:23.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T21:00:23.851 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2023-09-22T21:00:23.904 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-22T21:00:23.904 INFO:tasks.ceph:role: c2.osd.2 2023-09-22T21:00:23.904 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com 2023-09-22T21:00:23.904 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-22T21:00:23.972 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-22T21:00:23.973 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-22T21:00:23.973 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-22T21:00:23.973 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-22T21:00:23.973 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-22T21:00:23.973 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2023-09-22T21:00:23.973 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-22T21:00:23.973 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-22T21:00:23.973 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-22T21:00:23.973 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-22T21:00:24.442 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2023-09-22T21:00:24.444 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2023-09-22T21:00:24.444 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2023-09-22T21:00:24.580 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2023-09-22T21:00:24.590 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: /sbin/restorecon: command not found 2023-09-22T21:00:24.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T21:00:24.591 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2023-09-22T21:00:24.697 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:00:24.698+0000 7ff76920e640 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2023-09-22T21:00:24.698 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:00:24.698+0000 7ff76920e640 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2023-09-22T21:00:24.698 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:00:24.698+0000 7ff76920e640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-09-22T21:00:24.698 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:00:24.698+0000 7ff76920e640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-09-22T21:00:24.698 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:00:24.698+0000 7ff76920e640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-09-22T21:00:24.699 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:00:24.698+0000 7ff76920e640 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2023-09-22T21:00:26.910 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2023-09-22T21:00:26.924 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2023-09-22T21:00:27.006 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:00:27.006+0000 7f3a06c74640 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2023-09-22T21:00:27.007 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:00:27.006+0000 7f3a06c74640 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2023-09-22T21:00:27.007 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:00:27.006+0000 7f3a06c74640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-09-22T21:00:27.008 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:00:27.006+0000 7f3a06c74640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-09-22T21:00:27.008 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:00:27.006+0000 7f3a06c74640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-09-22T21:00:27.008 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:00:27.006+0000 7f3a06c74640 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2023-09-22T21:00:29.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2023-09-22T21:00:29.280 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2023-09-22T21:00:29.363 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:00:29.362+0000 7fc51a3f5640 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2023-09-22T21:00:29.364 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:00:29.366+0000 7fc51a3f5640 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2023-09-22T21:00:29.364 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:00:29.366+0000 7fc51a3f5640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-09-22T21:00:29.364 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:00:29.366+0000 7fc51a3f5640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-09-22T21:00:29.365 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:00:29.366+0000 7fc51a3f5640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-09-22T21:00:29.365 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:00:29.366+0000 7fc51a3f5640 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2023-09-22T21:00:31.567 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2023-09-22T21:00:31.620 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-22T21:00:31.621 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T21:00:31.621 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2023-09-22T21:00:31.675 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T21:00:31.675 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2023-09-22T21:00:31.728 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T21:00:31.728 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2023-09-22T21:00:31.784 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T21:00:31.785 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2023-09-22T21:00:31.839 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T21:00:31.945 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2023-09-22T21:00:31.951 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T21:00:31.951 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2023-09-22T21:00:31.996 INFO:tasks.ceph:Adding keys to all mons... 2023-09-22T21:00:31.996 DEBUG:teuthology.orchestra.run.smithi204:> sudo tee -a /etc/ceph/c2.keyring 2023-09-22T21:00:32.050 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr.x] 2023-09-22T21:00:32.050 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBmAA5lYNhcDBAAMia97zqxz1Ig28vGFI8Rew== 2023-09-22T21:00:32.050 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.0] 2023-09-22T21:00:32.051 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBoAA5ls2HKKRAArCEF36PwU2yCg5zQJpbqQw== 2023-09-22T21:00:32.051 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.1] 2023-09-22T21:00:32.051 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBrAA5lRkmZABAAjoNhANs56T7gYHHAIxZLcg== 2023-09-22T21:00:32.051 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.2] 2023-09-22T21:00:32.051 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBtAA5l+R3jFRAArDRI9g3Kn59FSdwryJ0itQ== 2023-09-22T21:00:32.051 INFO:teuthology.orchestra.run.smithi204.stdout:[client.0] 2023-09-22T21:00:32.051 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBmAA5lXylXERAAiiRpa3Bn/PSXdhbG7jmb2Q== 2023-09-22T21:00:32.051 INFO:teuthology.orchestra.run.smithi204.stdout:[client.1] 2023-09-22T21:00:32.051 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBmAA5lhgg8FxAAhzUl9UWhl36sFsklkSxIYQ== 2023-09-22T21:00:32.052 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-22T21:00:32.142 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T21:00:32.184 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T21:00:32.268 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T21:00:32.350 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-22T21:00:32.436 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-22T21:00:32.477 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-22T21:00:32.477 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2023-09-22T21:00:32.529 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2023-09-22T21:00:32.623 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2023-09-22T21:00:32.633 DEBUG:teuthology.orchestra.run.smithi204:> rm -- /home/ubuntu/cephtest/c2.monmap 2023-09-22T21:00:32.679 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2023-09-22T21:00:32.679 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2023-09-22T21:00:32.680 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2023-09-22T21:00:32.725 INFO:tasks.ceph.c2.mon.a:Started 2023-09-22T21:00:32.725 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2023-09-22T21:00:32.725 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2023-09-22T21:00:32.726 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2023-09-22T21:00:32.728 INFO:tasks.ceph.c2.mgr.x:Started 2023-09-22T21:00:32.728 INFO:tasks.ceph:Setting crush tunables to default 2023-09-22T21:00:32.728 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster c2 osd crush tunables default 2023-09-22T21:00:32.949 INFO:teuthology.orchestra.run.smithi204.stderr:adjusted tunables profile to default 2023-09-22T21:00:32.968 INFO:tasks.ceph:check_enable_crimson: False 2023-09-22T21:00:32.968 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2023-09-22T21:00:32.968 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T21:00:32.968 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2023-09-22T21:00:32.977 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T21:00:32.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2023-09-22T21:00:33.029 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-09-22T21:00:33.029 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2023-09-22T21:00:33.060 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:33.062+0000 7fc33f2f5180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-22T21:00:33.069 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster c2 osd new f5197f6e-1a5e-494e-b595-a3d41ca005a8 0 2023-09-22T21:00:33.305 INFO:teuthology.orchestra.run.smithi204.stdout:0 2023-09-22T21:00:33.322 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster c2 osd new d76faa91-6659-47b6-ae04-0fe2619106f6 1 2023-09-22T21:00:33.513 INFO:teuthology.orchestra.run.smithi204.stdout:1 2023-09-22T21:00:33.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster c2 osd new 82e292a4-1030-450e-a9ad-f2e535b5581c 2 2023-09-22T21:00:33.602 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:33.602+0000 7fc33f2f5180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-22T21:00:33.720 INFO:teuthology.orchestra.run.smithi204.stdout:2 2023-09-22T21:00:33.736 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2023-09-22T21:00:33.736 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2023-09-22T21:00:33.738 INFO:tasks.ceph.c2.osd.0:Started 2023-09-22T21:00:33.738 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2023-09-22T21:00:33.739 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2023-09-22T21:00:33.740 INFO:tasks.ceph.c2.osd.1:Started 2023-09-22T21:00:33.740 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2023-09-22T21:00:33.741 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2023-09-22T21:00:33.742 INFO:tasks.ceph.c2.osd.2:Started 2023-09-22T21:00:33.742 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-09-22T21:00:33.775 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:33.774+0000 7fc33f2f5180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-22T21:00:33.920 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:34.044 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":5,"fsid":"6b89fe4e-6e99-42c4-9d3a-9f184acab2d9","created":"2023-09-22T21:00:32.846292+0000","modified":"2023-09-22T21:00:33.720586+0000","last_up_change":"0.000000","last_in_change":"2023-09-22T21:00:33.720586+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f5197f6e-1a5e-494e-b595-a3d41ca005a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d76faa91-6659-47b6-ae04-0fe2619106f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"82e292a4-1030-450e-a9ad-f2e535b5581c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-22T21:00:34.046 INFO:tasks.ceph.ceph_manager.c2:[] 2023-09-22T21:00:34.046 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-22T21:00:34.347 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-09-22T21:00:34.361 INFO:tasks.ceph.c2.osd.2.smithi204.stderr:2023-09-22T21:00:34.362+0000 7f22ec5c0640 -1 Falling back to public interface 2023-09-22T21:00:34.372 INFO:tasks.ceph.c2.osd.1.smithi204.stderr:2023-09-22T21:00:34.374+0000 7f3efd0b2640 -1 Falling back to public interface 2023-09-22T21:00:34.373 INFO:tasks.ceph.c2.osd.0.smithi204.stderr:2023-09-22T21:00:34.374+0000 7f4986223640 -1 Falling back to public interface 2023-09-22T21:00:34.380 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:34.382+0000 7fc33f2f5180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-22T21:00:34.436 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:34.438+0000 7fc33f2f5180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-22T21:00:34.501 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:34.502+0000 7fc33f2f5180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-22T21:00:34.581 INFO:teuthology.misc.health.smithi204.stdout: 2023-09-22T21:00:34.581 INFO:teuthology.misc.health.smithi204.stdout:{"epoch":5,"fsid":"6b89fe4e-6e99-42c4-9d3a-9f184acab2d9","created":"2023-09-22T21:00:32.846292+0000","modified":"2023-09-22T21:00:33.720586+0000","last_up_change":"0.000000","last_in_change":"2023-09-22T21:00:33.720586+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f5197f6e-1a5e-494e-b595-a3d41ca005a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d76faa91-6659-47b6-ae04-0fe2619106f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"82e292a4-1030-450e-a9ad-f2e535b5581c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-22T21:00:34.582 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-09-22T21:00:34.585 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:34.586+0000 7fc33f2f5180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-22T21:00:34.641 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:34.642+0000 7fc33f2f5180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-22T21:00:34.726 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:34.726+0000 7fc33f2f5180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-22T21:00:34.781 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:34.782+0000 7fc33f2f5180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-22T21:00:34.920 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:34.922+0000 7fc33f2f5180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-22T21:00:34.979 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:34.978+0000 7fc33f2f5180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-22T21:00:35.588 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-09-22T21:00:35.589 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-09-22T21:00:35.589 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr: from numpy import show_config as show_numpy_config 2023-09-22T21:00:35.655 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:35.654+0000 7fc33f2f5180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-22T21:00:35.820 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:35.822+0000 7fc33f2f5180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-22T21:00:35.988 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:35.990+0000 7fc33f2f5180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-22T21:00:36.058 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:36.058+0000 7fc33f2f5180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-22T21:00:36.192 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:36.194+0000 7fc33f2f5180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-22T21:00:36.786 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:36.786+0000 7fc33f2f5180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-22T21:00:36.862 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:36.862+0000 7fc33f2f5180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-22T21:00:36.986 INFO:tasks.ceph.c2.osd.2.smithi204.stderr:2023-09-22T21:00:36.986+0000 7f22ec5c0640 -1 osd.2 0 log_to_monitors true 2023-09-22T21:00:37.037 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:37.038+0000 7fc33f2f5180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-22T21:00:37.111 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:37.110+0000 7fc33f2f5180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-22T21:00:37.197 INFO:tasks.ceph.c2.osd.0.smithi204.stderr:2023-09-22T21:00:37.198+0000 7f4986223640 -1 osd.0 0 log_to_monitors true 2023-09-22T21:00:37.206 INFO:tasks.ceph.c2.osd.1.smithi204.stderr:2023-09-22T21:00:37.206+0000 7f3efd0b2640 -1 osd.1 0 log_to_monitors true 2023-09-22T21:00:37.402 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:37.402+0000 7fc33f2f5180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-22T21:00:37.461 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:37.462+0000 7fc33f2f5180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-22T21:00:38.496 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T21:00:38.498+0000 7fc2e6a71640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-09-22T21:00:38.686 INFO:tasks.ceph.c2.osd.2.smithi204.stderr:2023-09-22T21:00:38.686+0000 7f22e8568640 -1 osd.2 0 waiting for initial osdmap 2023-09-22T21:00:38.704 INFO:tasks.ceph.c2.osd.2.smithi204.stderr:2023-09-22T21:00:38.706+0000 7f22e039d640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-22T21:00:40.228 INFO:tasks.ceph.c2.osd.0.smithi204.stderr:2023-09-22T21:00:40.230+0000 7f49829df640 -1 osd.0 0 waiting for initial osdmap 2023-09-22T21:00:40.230 INFO:tasks.ceph.c2.osd.1.smithi204.stderr:2023-09-22T21:00:40.230+0000 7f3ef986e640 -1 osd.1 0 waiting for initial osdmap 2023-09-22T21:00:40.260 INFO:tasks.ceph.c2.osd.1.smithi204.stderr:2023-09-22T21:00:40.262+0000 7f3ef0e8f640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-22T21:00:40.272 INFO:tasks.ceph.c2.osd.0.smithi204.stderr:2023-09-22T21:00:40.274+0000 7f497a000640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-22T21:00:40.884 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-09-22T21:00:41.175 INFO:teuthology.misc.health.smithi204.stdout: 2023-09-22T21:00:41.176 INFO:teuthology.misc.health.smithi204.stdout:{"epoch":9,"fsid":"6b89fe4e-6e99-42c4-9d3a-9f184acab2d9","created":"2023-09-22T21:00:32.846292+0000","modified":"2023-09-22T21:00:40.505770+0000","last_up_change":"2023-09-22T21:00:40.505770+0000","last_in_change":"2023-09-22T21:00:33.720586+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f5197f6e-1a5e-494e-b595-a3d41ca005a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":3893040448},{"type":"v1","addr":"172.21.15.204:6812","nonce":3893040448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":3893040448},{"type":"v1","addr":"172.21.15.204:6816","nonce":3893040448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":3893040448},{"type":"v1","addr":"172.21.15.204:6823","nonce":3893040448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":3893040448},{"type":"v1","addr":"172.21.15.204:6820","nonce":3893040448}]},"public_addr":"172.21.15.204:6812/3893040448","cluster_addr":"172.21.15.204:6816/3893040448","heartbeat_back_addr":"172.21.15.204:6823/3893040448","heartbeat_front_addr":"172.21.15.204:6820/3893040448","state":["exists","up"]},{"osd":1,"uuid":"d76faa91-6659-47b6-ae04-0fe2619106f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":752904203},{"type":"v1","addr":"172.21.15.204:6809","nonce":752904203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6811","nonce":752904203},{"type":"v1","addr":"172.21.15.204:6813","nonce":752904203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6819","nonce":752904203},{"type":"v1","addr":"172.21.15.204:6821","nonce":752904203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6815","nonce":752904203},{"type":"v1","addr":"172.21.15.204:6817","nonce":752904203}]},"public_addr":"172.21.15.204:6809/752904203","cluster_addr":"172.21.15.204:6813/752904203","heartbeat_back_addr":"172.21.15.204:6821/752904203","heartbeat_front_addr":"172.21.15.204:6817/752904203","state":["exists","up"]},{"osd":2,"uuid":"82e292a4-1030-450e-a9ad-f2e535b5581c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":2205208541},{"type":"v1","addr":"172.21.15.204:6801","nonce":2205208541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":2205208541},{"type":"v1","addr":"172.21.15.204:6803","nonce":2205208541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":2205208541},{"type":"v1","addr":"172.21.15.204:6807","nonce":2205208541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":2205208541},{"type":"v1","addr":"172.21.15.204:6805","nonce":2205208541}]},"public_addr":"172.21.15.204:6801/2205208541","cluster_addr":"172.21.15.204:6803/2205208541","heartbeat_back_addr":"172.21.15.204:6807/2205208541","heartbeat_front_addr":"172.21.15.204:6805/2205208541","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T21:00:38.004258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-22T21:00:41.192 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-09-22T21:00:41.193 INFO:tasks.ceph:Creating RBD pool 2023-09-22T21:00:41.193 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster c2 osd pool create rbd 8 2023-09-22T21:00:42.074 INFO:teuthology.orchestra.run.smithi204.stderr:pool 'rbd' created 2023-09-22T21:00:42.096 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-22T21:00:43.077 INFO:teuthology.orchestra.run.smithi204.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-22T21:00:43.109 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2023-09-22T21:00:43.109 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2023-09-22T21:00:43.109 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2023-09-22T21:00:43.110 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2023-09-22T21:00:43.110 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-22T21:00:43.462 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:43.481 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":4,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":4183631385},{"type":"v1","addr":"172.21.15.204:6825","nonce":4183631385}]},"active_addr":"172.21.15.204:6825/4183631385","active_change":"2023-09-22T21:00:37.485966+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.204:0","nonce":2586833471}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.204:0","nonce":4103843606}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.204:0","nonce":2558810931}]}]} 2023-09-22T21:00:43.485 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2023-09-22T21:00:43.485 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2023-09-22T21:00:43.485 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-09-22T21:00:43.794 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:43.794 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":11,"fsid":"6b89fe4e-6e99-42c4-9d3a-9f184acab2d9","created":"2023-09-22T21:00:32.846292+0000","modified":"2023-09-22T21:00:43.073187+0000","last_up_change":"2023-09-22T21:00:40.505770+0000","last_in_change":"2023-09-22T21:00:33.720586+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-22T21:00:41.518266+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-22T21:00:41.565363+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f5197f6e-1a5e-494e-b595-a3d41ca005a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":3893040448},{"type":"v1","addr":"172.21.15.204:6812","nonce":3893040448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":3893040448},{"type":"v1","addr":"172.21.15.204:6816","nonce":3893040448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":3893040448},{"type":"v1","addr":"172.21.15.204:6823","nonce":3893040448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":3893040448},{"type":"v1","addr":"172.21.15.204:6820","nonce":3893040448}]},"public_addr":"172.21.15.204:6812/3893040448","cluster_addr":"172.21.15.204:6816/3893040448","heartbeat_back_addr":"172.21.15.204:6823/3893040448","heartbeat_front_addr":"172.21.15.204:6820/3893040448","state":["exists","up"]},{"osd":1,"uuid":"d76faa91-6659-47b6-ae04-0fe2619106f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":752904203},{"type":"v1","addr":"172.21.15.204:6809","nonce":752904203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6811","nonce":752904203},{"type":"v1","addr":"172.21.15.204:6813","nonce":752904203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6819","nonce":752904203},{"type":"v1","addr":"172.21.15.204:6821","nonce":752904203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6815","nonce":752904203},{"type":"v1","addr":"172.21.15.204:6817","nonce":752904203}]},"public_addr":"172.21.15.204:6809/752904203","cluster_addr":"172.21.15.204:6813/752904203","heartbeat_back_addr":"172.21.15.204:6821/752904203","heartbeat_front_addr":"172.21.15.204:6817/752904203","state":["exists","up"]},{"osd":2,"uuid":"82e292a4-1030-450e-a9ad-f2e535b5581c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":2205208541},{"type":"v1","addr":"172.21.15.204:6801","nonce":2205208541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":2205208541},{"type":"v1","addr":"172.21.15.204:6803","nonce":2205208541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":2205208541},{"type":"v1","addr":"172.21.15.204:6807","nonce":2205208541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":2205208541},{"type":"v1","addr":"172.21.15.204:6805","nonce":2205208541}]},"public_addr":"172.21.15.204:6801/2205208541","cluster_addr":"172.21.15.204:6803/2205208541","heartbeat_back_addr":"172.21.15.204:6807/2205208541","heartbeat_front_addr":"172.21.15.204:6805/2205208541","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T21:00:38.178097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T21:00:38.201047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T21:00:38.004258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-22T21:00:43.812 INFO:tasks.ceph.ceph_manager.c2:all up! 2023-09-22T21:00:43.812 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-09-22T21:00:44.104 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:44.104 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":12,"fsid":"6b89fe4e-6e99-42c4-9d3a-9f184acab2d9","created":"2023-09-22T21:00:32.846292+0000","modified":"2023-09-22T21:00:44.081957+0000","last_up_change":"2023-09-22T21:00:40.505770+0000","last_in_change":"2023-09-22T21:00:33.720586+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-22T21:00:41.518266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-22T21:00:41.565363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f5197f6e-1a5e-494e-b595-a3d41ca005a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":3893040448},{"type":"v1","addr":"172.21.15.204:6812","nonce":3893040448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":3893040448},{"type":"v1","addr":"172.21.15.204:6816","nonce":3893040448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":3893040448},{"type":"v1","addr":"172.21.15.204:6823","nonce":3893040448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":3893040448},{"type":"v1","addr":"172.21.15.204:6820","nonce":3893040448}]},"public_addr":"172.21.15.204:6812/3893040448","cluster_addr":"172.21.15.204:6816/3893040448","heartbeat_back_addr":"172.21.15.204:6823/3893040448","heartbeat_front_addr":"172.21.15.204:6820/3893040448","state":["exists","up"]},{"osd":1,"uuid":"d76faa91-6659-47b6-ae04-0fe2619106f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":752904203},{"type":"v1","addr":"172.21.15.204:6809","nonce":752904203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6811","nonce":752904203},{"type":"v1","addr":"172.21.15.204:6813","nonce":752904203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6819","nonce":752904203},{"type":"v1","addr":"172.21.15.204:6821","nonce":752904203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6815","nonce":752904203},{"type":"v1","addr":"172.21.15.204:6817","nonce":752904203}]},"public_addr":"172.21.15.204:6809/752904203","cluster_addr":"172.21.15.204:6813/752904203","heartbeat_back_addr":"172.21.15.204:6821/752904203","heartbeat_front_addr":"172.21.15.204:6817/752904203","state":["exists","up"]},{"osd":2,"uuid":"82e292a4-1030-450e-a9ad-f2e535b5581c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":2205208541},{"type":"v1","addr":"172.21.15.204:6801","nonce":2205208541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":2205208541},{"type":"v1","addr":"172.21.15.204:6803","nonce":2205208541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":2205208541},{"type":"v1","addr":"172.21.15.204:6807","nonce":2205208541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":2205208541},{"type":"v1","addr":"172.21.15.204:6805","nonce":2205208541}]},"public_addr":"172.21.15.204:6801/2205208541","cluster_addr":"172.21.15.204:6803/2205208541","heartbeat_back_addr":"172.21.15.204:6807/2205208541","heartbeat_front_addr":"172.21.15.204:6805/2205208541","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T21:00:38.178097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T21:00:38.201047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T21:00:38.004258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-22T21:00:44.123 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2023-09-22T21:00:44.124 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2023-09-22T21:00:44.124 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2023-09-22T21:00:44.291 INFO:teuthology.orchestra.run.smithi204.stdout:34359738370 2023-09-22T21:00:44.291 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-09-22T21:00:44.292 INFO:teuthology.orchestra.run.smithi204.stdout:38654705667 2023-09-22T21:00:44.293 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-09-22T21:00:44.294 INFO:teuthology.orchestra.run.smithi204.stdout:38654705667 2023-09-22T21:00:44.294 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-09-22T21:00:44.589 INFO:teuthology.orchestra.run.smithi204.stdout:34359738369 2023-09-22T21:00:44.606 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.2 2023-09-22T21:00:44.612 INFO:teuthology.orchestra.run.smithi204.stdout:38654705666 2023-09-22T21:00:44.627 INFO:teuthology.orchestra.run.smithi204.stdout:38654705666 2023-09-22T21:00:44.628 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705666 for osd.1 2023-09-22T21:00:44.643 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705666 for osd.0 2023-09-22T21:00:45.607 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-09-22T21:00:45.629 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-09-22T21:00:45.644 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-09-22T21:00:45.935 INFO:teuthology.orchestra.run.smithi204.stdout:38654705666 2023-09-22T21:00:45.952 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705666 for osd.1 2023-09-22T21:00:46.000 INFO:teuthology.orchestra.run.smithi204.stdout:34359738369 2023-09-22T21:00:46.017 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.2 2023-09-22T21:00:46.022 INFO:teuthology.orchestra.run.smithi204.stdout:38654705666 2023-09-22T21:00:46.039 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705666 for osd.0 2023-09-22T21:00:46.953 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-09-22T21:00:47.018 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-09-22T21:00:47.040 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-09-22T21:00:47.249 INFO:teuthology.orchestra.run.smithi204.stdout:38654705667 2023-09-22T21:00:47.267 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705667 for osd.1 2023-09-22T21:00:47.267 DEBUG:teuthology.parallel:result is None 2023-09-22T21:00:47.319 INFO:teuthology.orchestra.run.smithi204.stdout:34359738371 2023-09-22T21:00:47.336 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738371 for osd.2 2023-09-22T21:00:47.336 DEBUG:teuthology.parallel:result is None 2023-09-22T21:00:47.354 INFO:teuthology.orchestra.run.smithi204.stdout:38654705667 2023-09-22T21:00:47.370 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705667 for osd.0 2023-09-22T21:00:47.433 DEBUG:teuthology.parallel:result is None 2023-09-22T21:00:47.433 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2023-09-22T21:00:47.433 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-09-22T21:00:47.694 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:47.694 INFO:teuthology.orchestra.run.smithi204.stderr:dumped all 2023-09-22T21:00:47.710 INFO:teuthology.orchestra.run.smithi204.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-09-22T21:00:47.495778+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":83012,"kb_used_data":1448,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314489788,"statfs":{"total":322122547200,"available":322037542912,"internally_reserved":0,"allocated":1482752,"data_stored":1255484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000598"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-22T21:00:43.099550+0000","last_change":"2023-09-22T21:00:43.092473+0000","last_active":"2023-09-22T21:00:43.099550+0000","last_peered":"2023-09-22T21:00:43.099550+0000","last_clean":"2023-09-22T21:00:43.099550+0000","last_became_active":"2023-09-22T21:00:43.092252+0000","last_became_peered":"2023-09-22T21:00:43.092252+0000","last_unstale":"2023-09-22T21:00:43.099550+0000","last_undegraded":"2023-09-22T21:00:43.099550+0000","last_fullsized":"2023-09-22T21:00:43.099550+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_clean_scrub_stamp":"2023-09-22T21:00:42.065747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-23T23:09:48.486719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-22T21:00:43.091007+0000","last_change":"2023-09-22T21:00:43.087686+0000","last_active":"2023-09-22T21:00:43.091007+0000","last_peered":"2023-09-22T21:00:43.091007+0000","last_clean":"2023-09-22T21:00:43.091007+0000","last_became_active":"2023-09-22T21:00:43.087381+0000","last_became_peered":"2023-09-22T21:00:43.087381+0000","last_unstale":"2023-09-22T21:00:43.091007+0000","last_undegraded":"2023-09-22T21:00:43.091007+0000","last_fullsized":"2023-09-22T21:00:43.091007+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_clean_scrub_stamp":"2023-09-22T21:00:42.065747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:41:39.209430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:44.088483+0000","last_change":"2023-09-22T21:00:43.092616+0000","last_active":"2023-09-22T21:00:44.088483+0000","last_peered":"2023-09-22T21:00:44.088483+0000","last_clean":"2023-09-22T21:00:44.088483+0000","last_became_active":"2023-09-22T21:00:43.092226+0000","last_became_peered":"2023-09-22T21:00:43.092226+0000","last_unstale":"2023-09-22T21:00:44.088483+0000","last_undegraded":"2023-09-22T21:00:44.088483+0000","last_fullsized":"2023-09-22T21:00:44.088483+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_clean_scrub_stamp":"2023-09-22T21:00:42.065747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T08:57:17.214072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-22T21:00:43.091062+0000","last_change":"2023-09-22T21:00:43.087768+0000","last_active":"2023-09-22T21:00:43.091062+0000","last_peered":"2023-09-22T21:00:43.091062+0000","last_clean":"2023-09-22T21:00:43.091062+0000","last_became_active":"2023-09-22T21:00:43.087372+0000","last_became_peered":"2023-09-22T21:00:43.087372+0000","last_unstale":"2023-09-22T21:00:43.091062+0000","last_undegraded":"2023-09-22T21:00:43.091062+0000","last_fullsized":"2023-09-22T21:00:43.091062+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_clean_scrub_stamp":"2023-09-22T21:00:42.065747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:00:57.967547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":138,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:44.088603+0000","last_change":"2023-09-22T21:00:43.092845+0000","last_active":"2023-09-22T21:00:44.088603+0000","last_peered":"2023-09-22T21:00:44.088603+0000","last_clean":"2023-09-22T21:00:44.088603+0000","last_became_active":"2023-09-22T21:00:43.091839+0000","last_became_peered":"2023-09-22T21:00:43.091839+0000","last_unstale":"2023-09-22T21:00:44.088603+0000","last_undegraded":"2023-09-22T21:00:44.088603+0000","last_fullsized":"2023-09-22T21:00:44.088603+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_clean_scrub_stamp":"2023-09-22T21:00:42.065747+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-23T23:25:58.154444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-22T21:00:43.091601+0000","last_change":"2023-09-22T21:00:43.090408+0000","last_active":"2023-09-22T21:00:43.091601+0000","last_peered":"2023-09-22T21:00:43.091601+0000","last_clean":"2023-09-22T21:00:43.091601+0000","last_became_active":"2023-09-22T21:00:43.089994+0000","last_became_peered":"2023-09-22T21:00:43.089994+0000","last_unstale":"2023-09-22T21:00:43.091601+0000","last_undegraded":"2023-09-22T21:00:43.091601+0000","last_fullsized":"2023-09-22T21:00:43.091601+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_clean_scrub_stamp":"2023-09-22T21:00:42.065747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T00:57:47.793582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-22T21:00:43.091608+0000","last_change":"2023-09-22T21:00:43.088213+0000","last_active":"2023-09-22T21:00:43.091608+0000","last_peered":"2023-09-22T21:00:43.091608+0000","last_clean":"2023-09-22T21:00:43.091608+0000","last_became_active":"2023-09-22T21:00:43.087698+0000","last_became_peered":"2023-09-22T21:00:43.087698+0000","last_unstale":"2023-09-22T21:00:43.091608+0000","last_undegraded":"2023-09-22T21:00:43.091608+0000","last_fullsized":"2023-09-22T21:00:43.091608+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_clean_scrub_stamp":"2023-09-22T21:00:42.065747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:41:39.209430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:44.088705+0000","last_change":"2023-09-22T21:00:43.095343+0000","last_active":"2023-09-22T21:00:44.088705+0000","last_peered":"2023-09-22T21:00:44.088705+0000","last_clean":"2023-09-22T21:00:44.088705+0000","last_became_active":"2023-09-22T21:00:43.094983+0000","last_became_peered":"2023-09-22T21:00:43.094983+0000","last_unstale":"2023-09-22T21:00:44.088705+0000","last_undegraded":"2023-09-22T21:00:44.088705+0000","last_fullsized":"2023-09-22T21:00:44.088705+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_clean_scrub_stamp":"2023-09-22T21:00:42.065747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:58:36.972600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-22T21:00:43.091989+0000","last_change":"2023-09-22T21:00:43.089609+0000","last_active":"2023-09-22T21:00:43.091989+0000","last_peered":"2023-09-22T21:00:43.091989+0000","last_clean":"2023-09-22T21:00:43.091989+0000","last_became_active":"2023-09-22T21:00:43.087607+0000","last_became_peered":"2023-09-22T21:00:43.087607+0000","last_unstale":"2023-09-22T21:00:43.091989+0000","last_undegraded":"2023-09-22T21:00:43.091989+0000","last_fullsized":"2023-09-22T21:00:43.091989+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_clean_scrub_stamp":"2023-09-22T21:00:42.065747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-23T21:55:04.406955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27692,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":692224,"data_stored":615284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":98304,"data_stored":24916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]}]},{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28196,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829404,"statfs":{"total":107374182400,"available":107345309696,"internally_reserved":0,"allocated":692224,"data_stored":615284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-22T21:00:47.712 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-09-22T21:00:47.976 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:47.976 INFO:teuthology.orchestra.run.smithi204.stderr:dumped all 2023-09-22T21:00:47.992 INFO:teuthology.orchestra.run.smithi204.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-09-22T21:00:47.495778+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":83012,"kb_used_data":1448,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314489788,"statfs":{"total":322122547200,"available":322037542912,"internally_reserved":0,"allocated":1482752,"data_stored":1255484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000598"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-22T21:00:43.099550+0000","last_change":"2023-09-22T21:00:43.092473+0000","last_active":"2023-09-22T21:00:43.099550+0000","last_peered":"2023-09-22T21:00:43.099550+0000","last_clean":"2023-09-22T21:00:43.099550+0000","last_became_active":"2023-09-22T21:00:43.092252+0000","last_became_peered":"2023-09-22T21:00:43.092252+0000","last_unstale":"2023-09-22T21:00:43.099550+0000","last_undegraded":"2023-09-22T21:00:43.099550+0000","last_fullsized":"2023-09-22T21:00:43.099550+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_clean_scrub_stamp":"2023-09-22T21:00:42.065747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-23T23:09:48.486719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-22T21:00:43.091007+0000","last_change":"2023-09-22T21:00:43.087686+0000","last_active":"2023-09-22T21:00:43.091007+0000","last_peered":"2023-09-22T21:00:43.091007+0000","last_clean":"2023-09-22T21:00:43.091007+0000","last_became_active":"2023-09-22T21:00:43.087381+0000","last_became_peered":"2023-09-22T21:00:43.087381+0000","last_unstale":"2023-09-22T21:00:43.091007+0000","last_undegraded":"2023-09-22T21:00:43.091007+0000","last_fullsized":"2023-09-22T21:00:43.091007+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_clean_scrub_stamp":"2023-09-22T21:00:42.065747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:41:39.209430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:44.088483+0000","last_change":"2023-09-22T21:00:43.092616+0000","last_active":"2023-09-22T21:00:44.088483+0000","last_peered":"2023-09-22T21:00:44.088483+0000","last_clean":"2023-09-22T21:00:44.088483+0000","last_became_active":"2023-09-22T21:00:43.092226+0000","last_became_peered":"2023-09-22T21:00:43.092226+0000","last_unstale":"2023-09-22T21:00:44.088483+0000","last_undegraded":"2023-09-22T21:00:44.088483+0000","last_fullsized":"2023-09-22T21:00:44.088483+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_clean_scrub_stamp":"2023-09-22T21:00:42.065747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T08:57:17.214072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-22T21:00:43.091062+0000","last_change":"2023-09-22T21:00:43.087768+0000","last_active":"2023-09-22T21:00:43.091062+0000","last_peered":"2023-09-22T21:00:43.091062+0000","last_clean":"2023-09-22T21:00:43.091062+0000","last_became_active":"2023-09-22T21:00:43.087372+0000","last_became_peered":"2023-09-22T21:00:43.087372+0000","last_unstale":"2023-09-22T21:00:43.091062+0000","last_undegraded":"2023-09-22T21:00:43.091062+0000","last_fullsized":"2023-09-22T21:00:43.091062+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_clean_scrub_stamp":"2023-09-22T21:00:42.065747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:00:57.967547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":138,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:44.088603+0000","last_change":"2023-09-22T21:00:43.092845+0000","last_active":"2023-09-22T21:00:44.088603+0000","last_peered":"2023-09-22T21:00:44.088603+0000","last_clean":"2023-09-22T21:00:44.088603+0000","last_became_active":"2023-09-22T21:00:43.091839+0000","last_became_peered":"2023-09-22T21:00:43.091839+0000","last_unstale":"2023-09-22T21:00:44.088603+0000","last_undegraded":"2023-09-22T21:00:44.088603+0000","last_fullsized":"2023-09-22T21:00:44.088603+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_clean_scrub_stamp":"2023-09-22T21:00:42.065747+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-23T23:25:58.154444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-22T21:00:43.091601+0000","last_change":"2023-09-22T21:00:43.090408+0000","last_active":"2023-09-22T21:00:43.091601+0000","last_peered":"2023-09-22T21:00:43.091601+0000","last_clean":"2023-09-22T21:00:43.091601+0000","last_became_active":"2023-09-22T21:00:43.089994+0000","last_became_peered":"2023-09-22T21:00:43.089994+0000","last_unstale":"2023-09-22T21:00:43.091601+0000","last_undegraded":"2023-09-22T21:00:43.091601+0000","last_fullsized":"2023-09-22T21:00:43.091601+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_clean_scrub_stamp":"2023-09-22T21:00:42.065747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T00:57:47.793582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-22T21:00:43.091608+0000","last_change":"2023-09-22T21:00:43.088213+0000","last_active":"2023-09-22T21:00:43.091608+0000","last_peered":"2023-09-22T21:00:43.091608+0000","last_clean":"2023-09-22T21:00:43.091608+0000","last_became_active":"2023-09-22T21:00:43.087698+0000","last_became_peered":"2023-09-22T21:00:43.087698+0000","last_unstale":"2023-09-22T21:00:43.091608+0000","last_undegraded":"2023-09-22T21:00:43.091608+0000","last_fullsized":"2023-09-22T21:00:43.091608+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_clean_scrub_stamp":"2023-09-22T21:00:42.065747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:41:39.209430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-22T21:00:44.088705+0000","last_change":"2023-09-22T21:00:43.095343+0000","last_active":"2023-09-22T21:00:44.088705+0000","last_peered":"2023-09-22T21:00:44.088705+0000","last_clean":"2023-09-22T21:00:44.088705+0000","last_became_active":"2023-09-22T21:00:43.094983+0000","last_became_peered":"2023-09-22T21:00:43.094983+0000","last_unstale":"2023-09-22T21:00:44.088705+0000","last_undegraded":"2023-09-22T21:00:44.088705+0000","last_fullsized":"2023-09-22T21:00:44.088705+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_clean_scrub_stamp":"2023-09-22T21:00:42.065747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:58:36.972600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-22T21:00:43.091989+0000","last_change":"2023-09-22T21:00:43.089609+0000","last_active":"2023-09-22T21:00:43.091989+0000","last_peered":"2023-09-22T21:00:43.091989+0000","last_clean":"2023-09-22T21:00:43.091989+0000","last_became_active":"2023-09-22T21:00:43.087607+0000","last_became_peered":"2023-09-22T21:00:43.087607+0000","last_unstale":"2023-09-22T21:00:43.091989+0000","last_undegraded":"2023-09-22T21:00:43.091989+0000","last_fullsized":"2023-09-22T21:00:43.091989+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T21:00:42.065747+0000","last_clean_scrub_stamp":"2023-09-22T21:00:42.065747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-23T21:55:04.406955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27692,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":692224,"data_stored":615284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":98304,"data_stored":24916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]}]},{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28196,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829404,"statfs":{"total":107374182400,"available":107345309696,"internally_reserved":0,"allocated":692224,"data_stored":615284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-22T21:00:47.993 INFO:tasks.ceph.ceph_manager.c2:clean! 2023-09-22T21:00:47.993 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2023-09-22T21:00:47.993 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2023-09-22T21:00:47.993 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2023-09-22T21:00:48.293 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-09-22T21:00:48.294 INFO:teuthology.orchestra.run.smithi204.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-09-22T21:00:48.311 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2023-09-22T21:00:48.311 INFO:teuthology.run_tasks:Running task rgw... 2023-09-22T21:00:48.324 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2023-09-22T21:00:48.324 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2023-09-22T21:00:48.325 INFO:tasks.rgw:Creating data pools 2023-09-22T21:00:48.325 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2023-09-22T21:00:48.325 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-09-22T21:00:48.678 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'default.rgw.buckets.data' created 2023-09-22T21:00:48.707 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-09-22T21:00:49.693 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-09-22T21:00:49.742 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-09-22T21:00:50.698 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'default.rgw.buckets.index' created 2023-09-22T21:00:50.735 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-09-22T21:00:51.712 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-09-22T21:00:51.767 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2023-09-22T21:00:51.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-09-22T21:00:52.053 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'default.rgw.buckets.data' already exists 2023-09-22T21:00:52.071 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-09-22T21:00:52.710 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-09-22T21:00:52.735 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-09-22T21:00:53.038 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'default.rgw.buckets.index' already exists 2023-09-22T21:00:53.057 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-09-22T21:00:53.715 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-09-22T21:00:53.737 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2023-09-22T21:00:53.738 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-09-22T21:00:54.820 INFO:teuthology.orchestra.run.smithi204.stderr:pool 'default.rgw.buckets.data' created 2023-09-22T21:00:54.858 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-09-22T21:00:55.839 INFO:teuthology.orchestra.run.smithi204.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-09-22T21:00:55.887 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-09-22T21:00:56.845 INFO:teuthology.orchestra.run.smithi204.stderr:pool 'default.rgw.buckets.index' created 2023-09-22T21:00:56.882 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-09-22T21:00:57.856 INFO:teuthology.orchestra.run.smithi204.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-09-22T21:00:57.899 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2023-09-22T21:00:57.899 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-09-22T21:00:58.188 INFO:teuthology.orchestra.run.smithi204.stderr:pool 'default.rgw.buckets.data' already exists 2023-09-22T21:00:58.206 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-09-22T21:00:58.854 INFO:teuthology.orchestra.run.smithi204.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-09-22T21:00:58.878 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-09-22T21:00:59.175 INFO:teuthology.orchestra.run.smithi204.stderr:pool 'default.rgw.buckets.index' already exists 2023-09-22T21:00:59.193 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-09-22T21:00:59.871 INFO:teuthology.orchestra.run.smithi204.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-09-22T21:00:59.892 DEBUG:tasks.rgw:Pools created 2023-09-22T21:00:59.893 INFO:tasks.rgw:Configuring compression type = random 2023-09-22T21:00:59.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2023-09-22T21:00:59.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2023-09-22T21:00:59.893 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2023-09-22T21:00:59.965 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:00:59.966 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:00:59.979 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:00:59.973+0000 7f584e706a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:01.689 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.685+0000 7f584e706a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:01.690 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.685+0000 7f584e706a40 20 realm 2023-09-22T21:01:01.690 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.685+0000 7f584e706a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:01.690 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.685+0000 7f584e706a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:01.690 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.685+0000 7f584e706a40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-09-22T21:01:01.690 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.685+0000 7f584e706a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:01.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.685+0000 7f584e706a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:01.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.685+0000 7f584e706a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:01.697 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.689+0000 7f584e706a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:01.697 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.693+0000 7f584e706a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:01.697 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.693+0000 7f584e706a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:01.698 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.693+0000 7f584e706a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:01.700 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.693+0000 7f584e706a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:01.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.693+0000 7f584e706a40 10 Creating default zonegroup 2023-09-22T21:01:01.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.697+0000 7f584e706a40 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-09-22T21:01:01.722 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.717+0000 7f584e706a40 10 WARNING: driver->list_zones() returned r=-2 2023-09-22T21:01:01.722 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.717+0000 7f584e706a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:01.722 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.717+0000 7f584e706a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:01.725 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.721+0000 7f584e706a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:01.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.721+0000 7f584e706a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:01.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.721+0000 7f584e706a40 10 could not read realm id: (2) No such file or directory 2023-09-22T21:01:01.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.721+0000 7f584e706a40 10 WARNING: failed to set zone as default, r=-22 2023-09-22T21:01:01.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.721+0000 7f584e706a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:01.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.721+0000 7f584e706a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:01.730 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.725+0000 7f584e706a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:01.731 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.725+0000 7f584e706a40 20 rados_obj.operate() r=0 bl.length=874 2023-09-22T21:01:01.731 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.725+0000 7f584e706a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:01.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.725+0000 7f584e706a40 20 rados_obj.operate() r=0 bl.length=420 2023-09-22T21:01:01.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.725+0000 7f584e706a40 10 Using default name default 2023-09-22T21:01:01.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.725+0000 7f584e706a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:01.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.729+0000 7f584e706a40 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T21:01:01.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.729+0000 7f584e706a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:01.734 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.729+0000 7f584e706a40 20 rados_obj.operate() r=0 bl.length=874 2023-09-22T21:01:01.734 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.729+0000 7f584e706a40 20 zone default found 2023-09-22T21:01:01.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.729+0000 7f584e706a40 4 Realm: () 2023-09-22T21:01:01.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.729+0000 7f584e706a40 4 ZoneGroup: default (7877ce5d-7618-49eb-b79d-d065c36c17f3) 2023-09-22T21:01:01.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.729+0000 7f584e706a40 4 Zone: default (90fa458d-c6a9-49f5-82c5-4bfcfe78369b) 2023-09-22T21:01:01.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.729+0000 7f584e706a40 10 cannot find current period zonegroup using local zonegroup configuration 2023-09-22T21:01:01.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.729+0000 7f584e706a40 20 zonegroup default 2023-09-22T21:01:01.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.729+0000 7f584e706a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:01.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.729+0000 7f584e706a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:01.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:01.729+0000 7f584e706a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:03.719 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:03.713+0000 7f584e706a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:03.719 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:03.713+0000 7f584e706a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:03.725 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:03.717+0000 7f584e706a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:03.725 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:03.717+0000 7f584e706a40 20 started sync module instance, tier type = 2023-09-22T21:01:03.725 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:03.717+0000 7f584e706a40 20 started zone id=90fa458d-c6a9-49f5-82c5-4bfcfe78369b (name=default) with tier type = 2023-09-22T21:01:05.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.789+0000 7f584e706a40 20 add_watcher() i=0 2023-09-22T21:01:05.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.789+0000 7f584e706a40 20 add_watcher() i=1 2023-09-22T21:01:05.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.789+0000 7f584e706a40 20 add_watcher() i=2 2023-09-22T21:01:05.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.789+0000 7f584e706a40 20 add_watcher() i=3 2023-09-22T21:01:05.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.789+0000 7f584e706a40 20 add_watcher() i=4 2023-09-22T21:01:05.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.789+0000 7f584e706a40 20 add_watcher() i=5 2023-09-22T21:01:05.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.789+0000 7f584e706a40 20 add_watcher() i=6 2023-09-22T21:01:05.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.797+0000 7f584e706a40 20 add_watcher() i=7 2023-09-22T21:01:05.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.797+0000 7f584e706a40 2 all 8 watchers are set, enabling cache 2023-09-22T21:01:05.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.797+0000 7f584e706a40 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-09-22T21:01:05.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.797+0000 7f584e706a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-09-22T21:01:05.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.797+0000 7f584e706a40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-22T21:01:05.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.797+0000 7f584e706a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-09-22T21:01:05.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.797+0000 7f584e706a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-09-22T21:01:05.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.797+0000 7f584e706a40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-22T21:01:05.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.797+0000 7f584e706a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-09-22T21:01:05.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.797+0000 7f584e706a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-09-22T21:01:05.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.797+0000 7f584e706a40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-22T21:01:05.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.797+0000 7f584e706a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-09-22T21:01:05.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.797+0000 7f584e706a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-09-22T21:01:05.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.797+0000 7f584e706a40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-22T21:01:05.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.797+0000 7f584e706a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-09-22T21:01:05.804 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.797+0000 7f584e706a40 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-09-22T21:01:05.806 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.801+0000 7f584e706a40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-22T21:01:05.813 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.809+0000 7f5833e91640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T21:01:05.813 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.809+0000 7f584e706a40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T21:01:05.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:05.809+0000 7f584e706a40 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T21:01:08.721 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.713+0000 7f584e706a40 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-09-22T21:01:08.721 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.713+0000 7f584e706a40 5 note: GC not initialized 2023-09-22T21:01:08.786 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.781+0000 7f582f688640 20 reqs_thread_entry: start 2023-09-22T21:01:08.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.781+0000 7f584e706a40 20 init_complete bucket index max shards: 11 2023-09-22T21:01:08.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.781+0000 7f584e706a40 20 Filter name: none 2023-09-22T21:01:08.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.781+0000 7f582d684640 20 reqs_thread_entry: start 2023-09-22T21:01:08.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.809+0000 7f5833e91640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T21:01:08.817 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.813+0000 7f584e706a40 20 remove_watcher() i=0 2023-09-22T21:01:08.817 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.813+0000 7f584e706a40 2 removed watcher, disabling cache 2023-09-22T21:01:08.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.813+0000 7f584e706a40 20 remove_watcher() i=1 2023-09-22T21:01:08.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.817+0000 7f584e706a40 20 remove_watcher() i=2 2023-09-22T21:01:08.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.817+0000 7f584e706a40 20 remove_watcher() i=3 2023-09-22T21:01:08.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.821+0000 7f584e706a40 20 remove_watcher() i=4 2023-09-22T21:01:08.826 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.821+0000 7f584e706a40 20 remove_watcher() i=5 2023-09-22T21:01:08.829 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.825+0000 7f584e706a40 20 remove_watcher() i=6 2023-09-22T21:01:08.831 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.825+0000 7f584e706a40 20 remove_watcher() i=7 2023-09-22T21:01:08.837 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2023-09-22T21:01:08.838 DEBUG:tasks.util.rgw: json result: [] 2023-09-22T21:01:08.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-22T21:01:08.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-22T21:01:08.838 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-09-22T21:01:08.877 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:01:08.878 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:01:08.891 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.885+0000 7fca194d7a40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T21:01:08.891 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.885+0000 7fca194d7a40 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T21:01:08.894 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.889+0000 7fc9fb45b640 20 reqs_thread_entry: start 2023-09-22T21:01:08.905 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"90fa458d-c6a9-49f5-82c5-4bfcfe78369b","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2023-09-22T21:01:08.906 DEBUG:tasks.util.rgw: json result: {'id': '90fa458d-c6a9-49f5-82c5-4bfcfe78369b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-09-22T21:01:08.906 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2023-09-22T21:01:08.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2023-09-22T21:01:08.906 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2023-09-22T21:01:08.978 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:01:08.978 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:01:08.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.985+0000 7fcabdd74a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:08.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.985+0000 7fcabdd74a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:08.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.985+0000 7fcabdd74a40 20 realm 2023-09-22T21:01:08.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.985+0000 7fcabdd74a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:08.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.985+0000 7fcabdd74a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:08.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.985+0000 7fcabdd74a40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-09-22T21:01:08.993 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.985+0000 7fcabdd74a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:08.993 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.985+0000 7fcabdd74a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:08.993 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.985+0000 7fcabdd74a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:08.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.985+0000 7fcabdd74a40 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T21:01:08.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.985+0000 7fcabdd74a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:08.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.989+0000 7fcabdd74a40 20 rados_obj.operate() r=0 bl.length=890 2023-09-22T21:01:08.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:08.989+0000 7fcabdd74a40 20 searching for the correct realm 2023-09-22T21:01:09.005 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 20 RGWRados::pool_iterate: got zone_info.90fa458d-c6a9-49f5-82c5-4bfcfe78369b 2023-09-22T21:01:09.005 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 20 RGWRados::pool_iterate: got zonegroup_info.7877ce5d-7618-49eb-b79d-d065c36c17f3 2023-09-22T21:01:09.005 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T21:01:09.006 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T21:01:09.006 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:09.006 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:09.006 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:09.006 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T21:01:09.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:09.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 20 rados_obj.operate() r=0 bl.length=420 2023-09-22T21:01:09.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 20 zone default found 2023-09-22T21:01:09.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 4 Realm: () 2023-09-22T21:01:09.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 4 ZoneGroup: default (7877ce5d-7618-49eb-b79d-d065c36c17f3) 2023-09-22T21:01:09.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 4 Zone: default (90fa458d-c6a9-49f5-82c5-4bfcfe78369b) 2023-09-22T21:01:09.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 10 cannot find current period zonegroup using local zonegroup configuration 2023-09-22T21:01:09.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 20 zonegroup default 2023-09-22T21:01:09.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:09.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:09.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:09.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:09.010 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:09.010 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:09.010 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 20 started sync module instance, tier type = 2023-09-22T21:01:09.010 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.001+0000 7fcabdd74a40 20 started zone id=90fa458d-c6a9-49f5-82c5-4bfcfe78369b (name=default) with tier type = 2023-09-22T21:01:09.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.013+0000 7fcabdd74a40 20 add_watcher() i=0 2023-09-22T21:01:09.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.013+0000 7fcabdd74a40 20 add_watcher() i=1 2023-09-22T21:01:09.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.013+0000 7fcabdd74a40 20 add_watcher() i=2 2023-09-22T21:01:09.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.013+0000 7fcabdd74a40 20 add_watcher() i=3 2023-09-22T21:01:09.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.013+0000 7fcabdd74a40 20 add_watcher() i=4 2023-09-22T21:01:09.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.013+0000 7fcabdd74a40 20 add_watcher() i=5 2023-09-22T21:01:09.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.013+0000 7fcabdd74a40 20 add_watcher() i=6 2023-09-22T21:01:09.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.013+0000 7fcabdd74a40 20 add_watcher() i=7 2023-09-22T21:01:09.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.013+0000 7fcabdd74a40 2 all 8 watchers are set, enabling cache 2023-09-22T21:01:09.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.017+0000 7fcaa34ff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T21:01:09.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.017+0000 7fcabdd74a40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T21:01:09.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.017+0000 7fcabdd74a40 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T21:01:09.027 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.021+0000 7fcabdd74a40 5 note: GC not initialized 2023-09-22T21:01:09.027 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.021+0000 7fca9ecf6640 20 reqs_thread_entry: start 2023-09-22T21:01:09.027 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.021+0000 7fcabdd74a40 20 init_complete bucket index max shards: 11 2023-09-22T21:01:09.027 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.021+0000 7fcabdd74a40 20 Filter name: none 2023-09-22T21:01:09.027 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.021+0000 7fca9ccf2640 20 reqs_thread_entry: start 2023-09-22T21:01:09.039 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.033+0000 7fcaa34ff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T21:01:09.043 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.037+0000 7fcabdd74a40 20 remove_watcher() i=0 2023-09-22T21:01:09.043 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.037+0000 7fcabdd74a40 2 removed watcher, disabling cache 2023-09-22T21:01:09.046 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.041+0000 7fcabdd74a40 20 remove_watcher() i=1 2023-09-22T21:01:09.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.045+0000 7fcabdd74a40 20 remove_watcher() i=2 2023-09-22T21:01:09.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.049+0000 7fcabdd74a40 20 remove_watcher() i=3 2023-09-22T21:01:09.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.049+0000 7fcabdd74a40 20 remove_watcher() i=4 2023-09-22T21:01:09.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.053+0000 7fcabdd74a40 20 remove_watcher() i=5 2023-09-22T21:01:09.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.053+0000 7fcabdd74a40 20 remove_watcher() i=6 2023-09-22T21:01:09.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.057+0000 7fcabdd74a40 20 remove_watcher() i=7 2023-09-22T21:01:09.066 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2023-09-22T21:01:09.066 DEBUG:tasks.util.rgw: json result: [] 2023-09-22T21:01:09.066 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-22T21:01:09.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-22T21:01:09.067 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-09-22T21:01:09.154 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:01:09.155 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:01:09.167 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.161+0000 7f77b1083a40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T21:01:09.167 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.161+0000 7f77b1083a40 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T21:01:09.170 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:09.165+0000 7f7793007640 20 reqs_thread_entry: start 2023-09-22T21:01:09.181 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"90fa458d-c6a9-49f5-82c5-4bfcfe78369b","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2023-09-22T21:01:09.181 DEBUG:tasks.util.rgw: json result: {'id': '90fa458d-c6a9-49f5-82c5-4bfcfe78369b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-09-22T21:01:09.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2023-09-22T21:01:09.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2023-09-22T21:01:09.181 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2023-09-22T21:01:09.241 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:01:09.241 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:01:09.255 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:09.254+0000 7f40da030a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:10.920 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.922+0000 7f40da030a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:10.920 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.922+0000 7f40da030a40 20 realm 2023-09-22T21:01:10.920 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.922+0000 7f40da030a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:10.921 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.922+0000 7f40da030a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:10.921 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.922+0000 7f40da030a40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-09-22T21:01:10.922 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.922+0000 7f40da030a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:10.922 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.922+0000 7f40da030a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:10.922 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.922+0000 7f40da030a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:10.922 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.922+0000 7f40da030a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:10.922 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.922+0000 7f40da030a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:10.923 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.922+0000 7f40da030a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:10.923 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.922+0000 7f40da030a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:10.923 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.922+0000 7f40da030a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:10.923 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.922+0000 7f40da030a40 10 Creating default zonegroup 2023-09-22T21:01:10.923 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.922+0000 7f40da030a40 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-09-22T21:01:10.941 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.942+0000 7f40da030a40 10 WARNING: driver->list_zones() returned r=-2 2023-09-22T21:01:10.942 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.942+0000 7f40da030a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:10.942 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.942+0000 7f40da030a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:10.945 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.946+0000 7f40da030a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:10.946 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.946+0000 7f40da030a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:10.946 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.946+0000 7f40da030a40 10 could not read realm id: (2) No such file or directory 2023-09-22T21:01:10.946 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.946+0000 7f40da030a40 10 WARNING: failed to set zone as default, r=-22 2023-09-22T21:01:10.946 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.946+0000 7f40da030a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:10.946 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.946+0000 7f40da030a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:10.950 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.950+0000 7f40da030a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:10.951 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.950+0000 7f40da030a40 20 rados_obj.operate() r=0 bl.length=874 2023-09-22T21:01:10.951 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.950+0000 7f40da030a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:10.952 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.954+0000 7f40da030a40 20 rados_obj.operate() r=0 bl.length=420 2023-09-22T21:01:10.952 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.954+0000 7f40da030a40 10 Using default name default 2023-09-22T21:01:10.952 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.954+0000 7f40da030a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:10.952 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.954+0000 7f40da030a40 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T21:01:10.952 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.954+0000 7f40da030a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:10.953 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.954+0000 7f40da030a40 20 rados_obj.operate() r=0 bl.length=874 2023-09-22T21:01:10.953 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.954+0000 7f40da030a40 20 zone default found 2023-09-22T21:01:10.953 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.954+0000 7f40da030a40 4 Realm: () 2023-09-22T21:01:10.953 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.954+0000 7f40da030a40 4 ZoneGroup: default (269c4e9f-5423-43bd-b1e4-cdfc8d5dee66) 2023-09-22T21:01:10.953 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.954+0000 7f40da030a40 4 Zone: default (bd4eec33-6715-4bb0-93f5-4b1092cfc000) 2023-09-22T21:01:10.953 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.954+0000 7f40da030a40 10 cannot find current period zonegroup using local zonegroup configuration 2023-09-22T21:01:10.953 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.954+0000 7f40da030a40 20 zonegroup default 2023-09-22T21:01:10.953 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.954+0000 7f40da030a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:10.954 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.954+0000 7f40da030a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:10.954 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:10.954+0000 7f40da030a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:12.956 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:12.958+0000 7f40da030a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:12.956 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:12.958+0000 7f40da030a40 20 rados->read ofs=0 len=0 2023-09-22T21:01:12.957 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:12.958+0000 7f40da030a40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:12.957 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:12.958+0000 7f40da030a40 20 started sync module instance, tier type = 2023-09-22T21:01:12.957 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:12.958+0000 7f40da030a40 20 started zone id=bd4eec33-6715-4bb0-93f5-4b1092cfc000 (name=default) with tier type = 2023-09-22T21:01:15.023 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.022+0000 7f40da030a40 20 add_watcher() i=0 2023-09-22T21:01:15.023 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.022+0000 7f40da030a40 20 add_watcher() i=1 2023-09-22T21:01:15.024 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.022+0000 7f40da030a40 20 add_watcher() i=2 2023-09-22T21:01:15.024 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.022+0000 7f40da030a40 20 add_watcher() i=3 2023-09-22T21:01:15.024 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.022+0000 7f40da030a40 20 add_watcher() i=4 2023-09-22T21:01:15.024 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.022+0000 7f40da030a40 20 add_watcher() i=5 2023-09-22T21:01:15.024 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.022+0000 7f40da030a40 20 add_watcher() i=6 2023-09-22T21:01:15.026 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.026+0000 7f40da030a40 20 add_watcher() i=7 2023-09-22T21:01:15.026 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.026+0000 7f40da030a40 2 all 8 watchers are set, enabling cache 2023-09-22T21:01:15.027 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.026+0000 7f40da030a40 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-09-22T21:01:15.027 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.026+0000 7f40da030a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-09-22T21:01:15.027 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.026+0000 7f40da030a40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-22T21:01:15.027 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.030+0000 7f40da030a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-09-22T21:01:15.027 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.030+0000 7f40da030a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-09-22T21:01:15.028 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.030+0000 7f40da030a40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-22T21:01:15.028 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.030+0000 7f40da030a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-09-22T21:01:15.028 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.030+0000 7f40da030a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-09-22T21:01:15.028 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.030+0000 7f40da030a40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-22T21:01:15.029 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.030+0000 7f40da030a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-09-22T21:01:15.029 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.030+0000 7f40da030a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-09-22T21:01:15.029 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.030+0000 7f40da030a40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-22T21:01:15.029 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.030+0000 7f40da030a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-09-22T21:01:15.029 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.030+0000 7f40da030a40 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-09-22T21:01:15.032 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.034+0000 7f40da030a40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-22T21:01:15.040 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.042+0000 7f40bf7bb640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T21:01:15.040 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.042+0000 7f40da030a40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T21:01:15.041 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:15.042+0000 7f40da030a40 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T21:01:17.961 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:17.962+0000 7f40da030a40 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-09-22T21:01:17.961 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:17.962+0000 7f40da030a40 5 note: GC not initialized 2023-09-22T21:01:17.966 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:17.966+0000 7f40bafb2640 20 reqs_thread_entry: start 2023-09-22T21:01:17.967 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:17.966+0000 7f40da030a40 20 init_complete bucket index max shards: 11 2023-09-22T21:01:17.967 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:17.966+0000 7f40da030a40 20 Filter name: none 2023-09-22T21:01:17.967 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:17.966+0000 7f40b8fae640 20 reqs_thread_entry: start 2023-09-22T21:01:17.998 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:17.998+0000 7f40bf7bb640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T21:01:18.002 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.002+0000 7f40da030a40 20 remove_watcher() i=0 2023-09-22T21:01:18.002 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.002+0000 7f40da030a40 2 removed watcher, disabling cache 2023-09-22T21:01:18.003 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.006+0000 7f40da030a40 20 remove_watcher() i=1 2023-09-22T21:01:18.009 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.010+0000 7f40da030a40 20 remove_watcher() i=2 2023-09-22T21:01:18.010 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.010+0000 7f40da030a40 20 remove_watcher() i=3 2023-09-22T21:01:18.013 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.014+0000 7f40da030a40 20 remove_watcher() i=4 2023-09-22T21:01:18.015 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.018+0000 7f40da030a40 20 remove_watcher() i=5 2023-09-22T21:01:18.017 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.018+0000 7f40da030a40 20 remove_watcher() i=6 2023-09-22T21:01:18.018 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.018+0000 7f40da030a40 20 remove_watcher() i=7 2023-09-22T21:01:18.026 INFO:teuthology.orchestra.run.smithi204.stdout:[] 2023-09-22T21:01:18.026 DEBUG:tasks.util.rgw: json result: [] 2023-09-22T21:01:18.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-22T21:01:18.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-22T21:01:18.027 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-09-22T21:01:18.067 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:01:18.067 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:01:18.080 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.082+0000 7f92e775da40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T21:01:18.081 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.082+0000 7f92e775da40 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T21:01:18.084 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.086+0000 7f92c96e1640 20 reqs_thread_entry: start 2023-09-22T21:01:18.094 INFO:teuthology.orchestra.run.smithi204.stdout:{"id":"bd4eec33-6715-4bb0-93f5-4b1092cfc000","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2023-09-22T21:01:18.094 DEBUG:tasks.util.rgw: json result: {'id': 'bd4eec33-6715-4bb0-93f5-4b1092cfc000', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-09-22T21:01:18.094 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2023-09-22T21:01:18.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2023-09-22T21:01:18.095 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2023-09-22T21:01:18.166 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:01:18.166 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:01:18.178 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.178+0000 7f0604c9fa40 20 rados->read ofs=0 len=0 2023-09-22T21:01:18.180 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.182+0000 7f0604c9fa40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:18.180 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.182+0000 7f0604c9fa40 20 realm 2023-09-22T21:01:18.180 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.182+0000 7f0604c9fa40 20 rados->read ofs=0 len=0 2023-09-22T21:01:18.180 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.182+0000 7f0604c9fa40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:18.180 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.182+0000 7f0604c9fa40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-09-22T21:01:18.180 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.182+0000 7f0604c9fa40 20 rados->read ofs=0 len=0 2023-09-22T21:01:18.181 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.182+0000 7f0604c9fa40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:18.181 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.182+0000 7f0604c9fa40 20 rados->read ofs=0 len=0 2023-09-22T21:01:18.181 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.182+0000 7f0604c9fa40 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T21:01:18.181 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.182+0000 7f0604c9fa40 20 rados->read ofs=0 len=0 2023-09-22T21:01:18.181 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.182+0000 7f0604c9fa40 20 rados_obj.operate() r=0 bl.length=890 2023-09-22T21:01:18.181 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.182+0000 7f0604c9fa40 20 searching for the correct realm 2023-09-22T21:01:18.193 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 20 RGWRados::pool_iterate: got zonegroup_info.269c4e9f-5423-43bd-b1e4-cdfc8d5dee66 2023-09-22T21:01:18.194 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 20 RGWRados::pool_iterate: got zone_info.bd4eec33-6715-4bb0-93f5-4b1092cfc000 2023-09-22T21:01:18.194 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T21:01:18.194 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T21:01:18.194 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 20 rados->read ofs=0 len=0 2023-09-22T21:01:18.194 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:18.194 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 20 rados->read ofs=0 len=0 2023-09-22T21:01:18.195 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T21:01:18.195 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 20 rados->read ofs=0 len=0 2023-09-22T21:01:18.195 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 20 rados_obj.operate() r=0 bl.length=420 2023-09-22T21:01:18.195 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 20 zone default found 2023-09-22T21:01:18.196 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 4 Realm: () 2023-09-22T21:01:18.196 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 4 ZoneGroup: default (269c4e9f-5423-43bd-b1e4-cdfc8d5dee66) 2023-09-22T21:01:18.196 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 4 Zone: default (bd4eec33-6715-4bb0-93f5-4b1092cfc000) 2023-09-22T21:01:18.196 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 10 cannot find current period zonegroup using local zonegroup configuration 2023-09-22T21:01:18.196 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 20 zonegroup default 2023-09-22T21:01:18.196 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 20 rados->read ofs=0 len=0 2023-09-22T21:01:18.196 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:18.196 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 20 rados->read ofs=0 len=0 2023-09-22T21:01:18.196 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:18.196 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 20 rados->read ofs=0 len=0 2023-09-22T21:01:18.196 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T21:01:18.196 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 20 started sync module instance, tier type = 2023-09-22T21:01:18.197 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.194+0000 7f0604c9fa40 20 started zone id=bd4eec33-6715-4bb0-93f5-4b1092cfc000 (name=default) with tier type = 2023-09-22T21:01:18.224 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.226+0000 7f0604c9fa40 20 add_watcher() i=0 2023-09-22T21:01:18.224 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.226+0000 7f0604c9fa40 20 add_watcher() i=1 2023-09-22T21:01:18.225 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.226+0000 7f0604c9fa40 20 add_watcher() i=2 2023-09-22T21:01:18.225 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.226+0000 7f0604c9fa40 20 add_watcher() i=3 2023-09-22T21:01:18.225 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.226+0000 7f0604c9fa40 20 add_watcher() i=4 2023-09-22T21:01:18.225 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.226+0000 7f0604c9fa40 20 add_watcher() i=5 2023-09-22T21:01:18.225 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.226+0000 7f0604c9fa40 20 add_watcher() i=6 2023-09-22T21:01:18.227 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.226+0000 7f0604c9fa40 20 add_watcher() i=7 2023-09-22T21:01:18.227 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.226+0000 7f0604c9fa40 2 all 8 watchers are set, enabling cache 2023-09-22T21:01:18.229 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.230+0000 7f05ea42a640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T21:01:18.229 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.230+0000 7f0604c9fa40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T21:01:18.229 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.230+0000 7f0604c9fa40 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T21:01:18.234 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.234+0000 7f0604c9fa40 5 note: GC not initialized 2023-09-22T21:01:18.235 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.234+0000 7f05e5c21640 20 reqs_thread_entry: start 2023-09-22T21:01:18.235 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.234+0000 7f0604c9fa40 20 init_complete bucket index max shards: 11 2023-09-22T21:01:18.235 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.234+0000 7f0604c9fa40 20 Filter name: none 2023-09-22T21:01:18.235 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.234+0000 7f05e3c1d640 20 reqs_thread_entry: start 2023-09-22T21:01:18.248 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.250+0000 7f05ea42a640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T21:01:18.250 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.250+0000 7f0604c9fa40 20 remove_watcher() i=0 2023-09-22T21:01:18.250 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.250+0000 7f0604c9fa40 2 removed watcher, disabling cache 2023-09-22T21:01:18.251 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.254+0000 7f0604c9fa40 20 remove_watcher() i=1 2023-09-22T21:01:18.253 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.254+0000 7f0604c9fa40 20 remove_watcher() i=2 2023-09-22T21:01:18.254 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.254+0000 7f0604c9fa40 20 remove_watcher() i=3 2023-09-22T21:01:18.255 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.254+0000 7f0604c9fa40 20 remove_watcher() i=4 2023-09-22T21:01:18.256 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.258+0000 7f0604c9fa40 20 remove_watcher() i=5 2023-09-22T21:01:18.257 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.258+0000 7f0604c9fa40 20 remove_watcher() i=6 2023-09-22T21:01:18.258 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.258+0000 7f0604c9fa40 20 remove_watcher() i=7 2023-09-22T21:01:18.264 INFO:teuthology.orchestra.run.smithi204.stdout:[] 2023-09-22T21:01:18.264 DEBUG:tasks.util.rgw: json result: [] 2023-09-22T21:01:18.264 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-22T21:01:18.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-22T21:01:18.265 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-09-22T21:01:18.343 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:01:18.356 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:01:18.357 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.358+0000 7f71959dba40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T21:01:18.357 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.358+0000 7f71959dba40 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T21:01:18.359 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:18.358+0000 7f717795f640 20 reqs_thread_entry: start 2023-09-22T21:01:18.369 INFO:teuthology.orchestra.run.smithi204.stdout:{"id":"bd4eec33-6715-4bb0-93f5-4b1092cfc000","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2023-09-22T21:01:18.369 DEBUG:tasks.util.rgw: json result: {'id': 'bd4eec33-6715-4bb0-93f5-4b1092cfc000', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-09-22T21:01:18.369 INFO:tasks.rgw:Starting rgw... 2023-09-22T21:01:18.369 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2023-09-22T21:01:18.369 INFO:tasks.rgw:Using beast as radosgw frontend 2023-09-22T21:01:18.370 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-09-22T21:01:18.370 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-09-22T21:01:18.373 INFO:tasks.rgw.c1.client.0:Started 2023-09-22T21:01:18.373 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2023-09-22T21:01:18.373 INFO:tasks.rgw:Using beast as radosgw frontend 2023-09-22T21:01:18.373 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-22T21:01:18.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-22T21:01:18.376 INFO:tasks.rgw.c1.client.1:Started 2023-09-22T21:01:18.376 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2023-09-22T21:01:18.376 INFO:tasks.rgw:Using beast as radosgw frontend 2023-09-22T21:01:18.376 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-09-22T21:01:18.376 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-09-22T21:01:18.413 INFO:tasks.rgw.c2.client.0:Started 2023-09-22T21:01:18.413 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2023-09-22T21:01:18.414 INFO:tasks.rgw:Using beast as radosgw frontend 2023-09-22T21:01:18.414 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-09-22T21:01:18.414 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-09-22T21:01:18.416 INFO:tasks.rgw.c2.client.1:Started 2023-09-22T21:01:18.416 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi160.front.sepia.ceph.com:8000/ 2023-09-22T21:01:18.416 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000/ 2023-09-22T21:01:18.515 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:01:18.515 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:01:18.515 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T21:01:18.516 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T21:01:18.517 INFO:teuthology.orchestra.run.smithi160.stderr:curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8000 after 0 ms: Connection refused 2023-09-22T21:01:19.517 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000/ 2023-09-22T21:01:19.528 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:01:19.528 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:01:19.531 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 68001 0 --:--:-- --:--:-- --:--:-- 71333 2023-09-22T21:01:19.532 INFO:teuthology.orchestra.run.smithi160.stdout:anonymous 2023-09-22T21:01:19.532 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi160.front.sepia.ceph.com:8001/ 2023-09-22T21:01:19.532 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001/ 2023-09-22T21:01:19.582 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:01:19.583 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:01:19.585 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 79672 0 --:--:-- --:--:-- --:--:-- 104k 2023-09-22T21:01:19.586 INFO:teuthology.orchestra.run.smithi160.stdout:anonymous 2023-09-22T21:01:19.586 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi204.front.sepia.ceph.com:8000/ 2023-09-22T21:01:19.586 DEBUG:teuthology.orchestra.run.smithi204:> curl http://smithi204.front.sepia.ceph.com:8000/ 2023-09-22T21:01:19.605 INFO:teuthology.orchestra.run.smithi204.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:01:19.605 INFO:teuthology.orchestra.run.smithi204.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:01:19.634 INFO:teuthology.orchestra.run.smithi204.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 7164 0 --:--:-- --:--:-- --:--:-- 7379 2023-09-22T21:01:19.635 INFO:teuthology.orchestra.run.smithi204.stdout:anonymous 2023-09-22T21:01:19.635 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi204.front.sepia.ceph.com:8001/ 2023-09-22T21:01:19.636 DEBUG:teuthology.orchestra.run.smithi204:> curl http://smithi204.front.sepia.ceph.com:8001/ 2023-09-22T21:01:19.689 INFO:teuthology.orchestra.run.smithi204.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:01:19.690 INFO:teuthology.orchestra.run.smithi204.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:01:19.719 INFO:teuthology.orchestra.run.smithi204.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 7318 0 --:--:-- --:--:-- --:--:-- 7379 2023-09-22T21:01:19.719 INFO:teuthology.orchestra.run.smithi204.stdout:anonymous 2023-09-22T21:01:19.720 INFO:teuthology.run_tasks:Running task rgw-multisite... 2023-09-22T21:01:20.906 INFO:tasks.rgw_multisite:creating realm.. 2023-09-22T21:01:20.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:20.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:20.907 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:01:20.946 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:01:20.947 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:01:20.986 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","name":"test-realm","current_period":"a6099785-5160-4a69-b202-552a1b9cb770","epoch":1} 2023-09-22T21:01:20.986 INFO:tasks.rgw_multisite:creating master zonegroup.. 2023-09-22T21:01:20.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:20.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:20.987 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup a --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:01:21.069 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:01:21.069 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:01:21.087 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:21.081+0000 7f2c8eaeba40 -1 old format 2023-09-22T21:01:21.103 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"a","name":"a","api_name":"","is_master":true,"endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":[]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]} 2023-09-22T21:01:21.103 INFO:tasks.rgw_multisite:creating master zone.. 2023-09-22T21:01:21.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'KT50DM2YGPMJ8OAR', '--secret', 'lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo', '--endpoints', 'http://smithi160.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:21.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'KT50DM2YGPMJ8OAR', '--secret', 'lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo', '--endpoints', 'http://smithi160.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:21.103 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --default --master --access-key KT50DM2YGPMJ8OAR --secret lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo --endpoints http://smithi160.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:01:21.195 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:01:21.195 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:01:21.214 INFO:teuthology.orchestra.run.smithi160.stderr:NOTICE: set zone's realm_id=9e7cb63e-d555-4d20-9c48-393b1c24d1ff 2023-09-22T21:01:21.243 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"f132bdb9-20a7-4073-a932-8f99da20691e","name":"a1","domain_root":"a1.rgw.meta:root","control_pool":"a1.rgw.control","gc_pool":"a1.rgw.log:gc","lc_pool":"a1.rgw.log:lc","log_pool":"a1.rgw.log","intent_log_pool":"a1.rgw.log:intent","usage_log_pool":"a1.rgw.log:usage","roles_pool":"a1.rgw.meta:roles","reshard_pool":"a1.rgw.log:reshard","user_keys_pool":"a1.rgw.meta:users.keys","user_email_pool":"a1.rgw.meta:users.email","user_swift_pool":"a1.rgw.meta:users.swift","user_uid_pool":"a1.rgw.meta:users.uid","otp_pool":"a1.rgw.otp","system_key":{"access_key":"KT50DM2YGPMJ8OAR","secret_key":"lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"a1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"a1.rgw.buckets.data"}},"data_extra_pool":"a1.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","notif_pool":"a1.rgw.log:notif"} 2023-09-22T21:01:21.244 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool create a1.rgw.buckets.data 64 64 --cluster c1 2023-09-22T21:01:21.681 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'a1.rgw.buckets.data' created 2023-09-22T21:01:21.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool application enable a1.rgw.buckets.data rgw --cluster c1 2023-09-22T21:01:22.682 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rgw' on pool 'a1.rgw.buckets.data' 2023-09-22T21:01:22.729 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:22.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:22.730 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:01:22.775 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:01:22.775 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:01:22.799 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"f132bdb9-20a7-4073-a932-8f99da20691e","name":"a1","domain_root":"a1.rgw.meta:root","control_pool":"a1.rgw.control","gc_pool":"a1.rgw.log:gc","lc_pool":"a1.rgw.log:lc","log_pool":"a1.rgw.log","intent_log_pool":"a1.rgw.log:intent","usage_log_pool":"a1.rgw.log:usage","roles_pool":"a1.rgw.meta:roles","reshard_pool":"a1.rgw.log:reshard","user_keys_pool":"a1.rgw.meta:users.keys","user_email_pool":"a1.rgw.meta:users.email","user_swift_pool":"a1.rgw.meta:users.swift","user_uid_pool":"a1.rgw.meta:users.uid","otp_pool":"a1.rgw.otp","system_key":{"access_key":"KT50DM2YGPMJ8OAR","secret_key":"lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"a1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"a1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"a1.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","notif_pool":"a1.rgw.log:notif"} 2023-09-22T21:01:22.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:22.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:22.800 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:01:22.877 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:01:22.877 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:01:22.893 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:22.893+0000 7f725c3a2a40 0 period (a6099785-5160-4a69-b202-552a1b9cb770 does not have zone f132bdb9-20a7-4073-a932-8f99da20691e configured 2023-09-22T21:01:29.883 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","epoch":1,"predecessor_uuid":"a6099785-5160-4a69-b202-552a1b9cb770","sync_status":[],"period_map":{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","zonegroups":[{"id":"a","name":"a","api_name":"","is_master":true,"endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","zones":[{"id":"f132bdb9-20a7-4073-a932-8f99da20691e","name":"a1","endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":false,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"f132bdb9-20a7-4073-a932-8f99da20691e","val":3405200286}]},"master_zonegroup":"a","master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","realm_name":"test-realm","realm_epoch":2} 2023-09-22T21:01:29.883 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-09-22T21:01:29.883 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T21:01:29.925 INFO:tasks.rgw.c1.client.0.smithi160.stdout:2023-09-22T21:01:29.921+0000 7f178f6db640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 37733) UID: 0 2023-09-22T21:01:29.925 INFO:tasks.rgw.c1.client.0.smithi160.stdout:2023-09-22T21:01:29.921+0000 7f1792f22a40 -1 shutting down 2023-09-22T21:01:33.296 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-09-22T21:01:35.985 INFO:tasks.rgw.c1.client.0:Stopped 2023-09-22T21:01:35.986 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-09-22T21:01:35.986 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-09-22T21:01:35.989 INFO:tasks.rgw.c1.client.0:Started 2023-09-22T21:01:35.989 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000 2023-09-22T21:01:36.000 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:01:36.000 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:01:36.000 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T21:01:36.001 INFO:teuthology.orchestra.run.smithi160.stderr:curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8000 after 0 ms: Connection refused 2023-09-22T21:01:36.001 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T21:01:37.002 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000 2023-09-22T21:01:37.014 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:01:37.014 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:01:37.016 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71860 0 --:--:-- --:--:-- --:--:-- 104k 2023-09-22T21:01:37.017 INFO:teuthology.orchestra.run.smithi160.stdout:anonymous 2023-09-22T21:01:37.017 INFO:tasks.rgw_multisite:creating admin user.. 2023-09-22T21:01:37.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'KT50DM2YGPMJ8OAR', '--secret', 'lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:37.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'KT50DM2YGPMJ8OAR', '--secret', 'lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:37.018 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key KT50DM2YGPMJ8OAR --secret lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:01:37.108 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:01:37.108 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:01:37.147 INFO:teuthology.orchestra.run.smithi160.stdout:{"user_id":"realm-admin","display_name":"Realm Admin","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"realm-admin","access_key":"KT50DM2YGPMJ8OAR","secret_key":"lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","system":true,"default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2023-09-22T21:01:37.163 INFO:tasks.rgw_multisite:creating zone.. 2023-09-22T21:01:37.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'KT50DM2YGPMJ8OAR', '--secret', 'lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo', '--endpoints', 'http://smithi160.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:37.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'KT50DM2YGPMJ8OAR', '--secret', 'lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo', '--endpoints', 'http://smithi160.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:37.163 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --access-key KT50DM2YGPMJ8OAR --secret lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo --endpoints http://smithi160.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:01:37.202 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:01:37.202 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:01:37.220 INFO:teuthology.orchestra.run.smithi160.stderr:NOTICE: set zone's realm_id=9e7cb63e-d555-4d20-9c48-393b1c24d1ff 2023-09-22T21:01:37.241 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"2b721fee-a149-45da-925b-502d5a3d6853","name":"a2","domain_root":"a2.rgw.meta:root","control_pool":"a2.rgw.control","gc_pool":"a2.rgw.log:gc","lc_pool":"a2.rgw.log:lc","log_pool":"a2.rgw.log","intent_log_pool":"a2.rgw.log:intent","usage_log_pool":"a2.rgw.log:usage","roles_pool":"a2.rgw.meta:roles","reshard_pool":"a2.rgw.log:reshard","user_keys_pool":"a2.rgw.meta:users.keys","user_email_pool":"a2.rgw.meta:users.email","user_swift_pool":"a2.rgw.meta:users.swift","user_uid_pool":"a2.rgw.meta:users.uid","otp_pool":"a2.rgw.otp","system_key":{"access_key":"KT50DM2YGPMJ8OAR","secret_key":"lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"a2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"a2.rgw.buckets.data"}},"data_extra_pool":"a2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","notif_pool":"a2.rgw.log:notif"} 2023-09-22T21:01:37.242 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool create a2.rgw.buckets.data 64 64 --cluster c1 2023-09-22T21:01:37.675 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'a2.rgw.buckets.data' created 2023-09-22T21:01:37.710 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool application enable a2.rgw.buckets.data rgw --cluster c1 2023-09-22T21:01:38.677 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rgw' on pool 'a2.rgw.buckets.data' 2023-09-22T21:01:38.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:38.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:38.741 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:01:38.777 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:01:38.777 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:01:38.801 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"2b721fee-a149-45da-925b-502d5a3d6853","name":"a2","domain_root":"a2.rgw.meta:root","control_pool":"a2.rgw.control","gc_pool":"a2.rgw.log:gc","lc_pool":"a2.rgw.log:lc","log_pool":"a2.rgw.log","intent_log_pool":"a2.rgw.log:intent","usage_log_pool":"a2.rgw.log:usage","roles_pool":"a2.rgw.meta:roles","reshard_pool":"a2.rgw.log:reshard","user_keys_pool":"a2.rgw.meta:users.keys","user_email_pool":"a2.rgw.meta:users.email","user_swift_pool":"a2.rgw.meta:users.swift","user_uid_pool":"a2.rgw.meta:users.uid","otp_pool":"a2.rgw.otp","system_key":{"access_key":"KT50DM2YGPMJ8OAR","secret_key":"lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"a2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"a2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"a2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","notif_pool":"a2.rgw.log:notif"} 2023-09-22T21:01:38.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:38.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:38.802 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:01:38.877 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:01:38.877 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:01:38.892 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:01:38.890+0000 7fd5071cea40 0 period (03d00cc5-7ad7-4abf-a475-d9ffc8e37106 does not have zone 2b721fee-a149-45da-925b-502d5a3d6853 configured 2023-09-22T21:01:39.001 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-09-22T21:01:45.826 INFO:teuthology.orchestra.run.smithi160.stderr:Sending period to new master zone f132bdb9-20a7-4073-a932-8f99da20691e 2023-09-22T21:01:45.872 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","epoch":2,"predecessor_uuid":"a6099785-5160-4a69-b202-552a1b9cb770","sync_status":[],"period_map":{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","zonegroups":[{"id":"a","name":"a","api_name":"","is_master":true,"endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","zones":[{"id":"2b721fee-a149-45da-925b-502d5a3d6853","name":"a2","endpoints":["http://smithi160.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"f132bdb9-20a7-4073-a932-8f99da20691e","name":"a1","endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"2b721fee-a149-45da-925b-502d5a3d6853","val":3586193783},{"key":"f132bdb9-20a7-4073-a932-8f99da20691e","val":3405200286}]},"master_zonegroup":"a","master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","realm_name":"test-realm","realm_epoch":2} 2023-09-22T21:01:45.873 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-09-22T21:01:45.873 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T21:01:45.916 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:01:45.914+0000 7fc210f68640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 37735) UID: 0 2023-09-22T21:01:45.916 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:01:45.914+0000 7fc2147afa40 -1 shutting down 2023-09-22T21:01:50.313 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-09-22T21:01:51.975 INFO:tasks.rgw.c1.client.1:Stopped 2023-09-22T21:01:51.975 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-22T21:01:51.975 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-22T21:01:51.978 INFO:tasks.rgw.c1.client.1:Started 2023-09-22T21:01:51.978 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T21:01:51.989 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:01:51.989 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:01:51.991 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T21:01:51.991 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T21:01:51.991 INFO:teuthology.orchestra.run.smithi160.stderr:curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T21:01:52.992 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T21:01:53.003 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:01:53.003 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:01:53.006 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 86534 0 --:--:-- --:--:-- --:--:-- 104k 2023-09-22T21:01:53.006 INFO:teuthology.orchestra.run.smithi160.stdout:anonymous 2023-09-22T21:01:53.007 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-09-22T21:01:53.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi160.front.sepia.ceph.com:8000', '--access-key', 'KT50DM2YGPMJ8OAR', '--secret', 'lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:53.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi160.front.sepia.ceph.com:8000', '--access-key', 'KT50DM2YGPMJ8OAR', '--secret', 'lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:53.007 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi160.front.sepia.ceph.com:8000 --access-key KT50DM2YGPMJ8OAR --secret lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-22T21:01:53.052 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:01:53.052 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:01:53.101 INFO:teuthology.orchestra.run.smithi204.stdout:{"id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","name":"test-realm","current_period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","epoch":2} 2023-09-22T21:01:53.101 INFO:tasks.rgw_multisite:creating zonegroup.. 2023-09-22T21:01:53.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:53.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:53.102 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zonegroup set --rgw-zonegroup b --rgw-realm test-realm --default --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-22T21:01:53.134 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:01:53.135 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:01:53.153 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:53.153+0000 7f9871d43a40 -1 old format 2023-09-22T21:01:53.163 INFO:teuthology.orchestra.run.smithi204.stdout:{"id":"b","name":"b","api_name":"","is_master":false,"endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":[]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]} 2023-09-22T21:01:53.163 INFO:tasks.rgw_multisite:creating zone.. 2023-09-22T21:01:53.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'KT50DM2YGPMJ8OAR', '--secret', 'lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo', '--endpoints', 'http://smithi204.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:53.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'KT50DM2YGPMJ8OAR', '--secret', 'lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo', '--endpoints', 'http://smithi204.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:53.163 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --default --master --access-key KT50DM2YGPMJ8OAR --secret lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo --endpoints http://smithi204.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-22T21:01:53.244 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:01:53.244 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:01:53.263 INFO:teuthology.orchestra.run.smithi204.stderr:NOTICE: set zone's realm_id=9e7cb63e-d555-4d20-9c48-393b1c24d1ff 2023-09-22T21:01:53.287 INFO:teuthology.orchestra.run.smithi204.stdout:{"id":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","name":"b1","domain_root":"b1.rgw.meta:root","control_pool":"b1.rgw.control","gc_pool":"b1.rgw.log:gc","lc_pool":"b1.rgw.log:lc","log_pool":"b1.rgw.log","intent_log_pool":"b1.rgw.log:intent","usage_log_pool":"b1.rgw.log:usage","roles_pool":"b1.rgw.meta:roles","reshard_pool":"b1.rgw.log:reshard","user_keys_pool":"b1.rgw.meta:users.keys","user_email_pool":"b1.rgw.meta:users.email","user_swift_pool":"b1.rgw.meta:users.swift","user_uid_pool":"b1.rgw.meta:users.uid","otp_pool":"b1.rgw.otp","system_key":{"access_key":"KT50DM2YGPMJ8OAR","secret_key":"lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"b1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"b1.rgw.buckets.data"}},"data_extra_pool":"b1.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","notif_pool":"b1.rgw.log:notif"} 2023-09-22T21:01:53.287 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph osd pool create b1.rgw.buckets.data 64 64 --cluster c2 2023-09-22T21:01:53.896 INFO:teuthology.orchestra.run.smithi204.stderr:pool 'b1.rgw.buckets.data' created 2023-09-22T21:01:53.934 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph osd pool application enable b1.rgw.buckets.data rgw --cluster c2 2023-09-22T21:01:54.905 INFO:teuthology.orchestra.run.smithi204.stderr:enabled application 'rgw' on pool 'b1.rgw.buckets.data' 2023-09-22T21:01:54.970 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:54.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:54.970 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-22T21:01:55.010 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:01:55.010 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:01:55.035 INFO:teuthology.orchestra.run.smithi204.stdout:{"id":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","name":"b1","domain_root":"b1.rgw.meta:root","control_pool":"b1.rgw.control","gc_pool":"b1.rgw.log:gc","lc_pool":"b1.rgw.log:lc","log_pool":"b1.rgw.log","intent_log_pool":"b1.rgw.log:intent","usage_log_pool":"b1.rgw.log:usage","roles_pool":"b1.rgw.meta:roles","reshard_pool":"b1.rgw.log:reshard","user_keys_pool":"b1.rgw.meta:users.keys","user_email_pool":"b1.rgw.meta:users.email","user_swift_pool":"b1.rgw.meta:users.swift","user_uid_pool":"b1.rgw.meta:users.uid","otp_pool":"b1.rgw.otp","system_key":{"access_key":"KT50DM2YGPMJ8OAR","secret_key":"lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"b1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"b1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"b1.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","notif_pool":"b1.rgw.log:notif"} 2023-09-22T21:01:55.035 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:55.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:01:55.036 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-22T21:01:55.110 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:01:55.110 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:01:55.126 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:01:55.125+0000 7f83aad4fa40 0 period (03d00cc5-7ad7-4abf-a475-d9ffc8e37106 does not have zone bfe1d114-67f9-4e0e-8e45-c23380ed8feb configured 2023-09-22T21:01:56.018 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-09-22T21:02:02.032 INFO:teuthology.orchestra.run.smithi204.stderr:Sending period to new master zone f132bdb9-20a7-4073-a932-8f99da20691e 2023-09-22T21:02:02.129 INFO:teuthology.orchestra.run.smithi204.stdout:{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","epoch":3,"predecessor_uuid":"a6099785-5160-4a69-b202-552a1b9cb770","sync_status":[],"period_map":{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","zonegroups":[{"id":"a","name":"a","api_name":"","is_master":true,"endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","zones":[{"id":"2b721fee-a149-45da-925b-502d5a3d6853","name":"a2","endpoints":["http://smithi160.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"f132bdb9-20a7-4073-a932-8f99da20691e","name":"a1","endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]},{"id":"b","name":"b","api_name":"","is_master":false,"endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","zones":[{"id":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","name":"b1","endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":false,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"2b721fee-a149-45da-925b-502d5a3d6853","val":3586193783},{"key":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","val":4184768938},{"key":"f132bdb9-20a7-4073-a932-8f99da20691e","val":3405200286}]},"master_zonegroup":"a","master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","realm_name":"test-realm","realm_epoch":2} 2023-09-22T21:02:02.129 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-09-22T21:02:02.130 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T21:02:02.172 INFO:tasks.rgw.c2.client.0.smithi204.stdout:2023-09-22T21:02:02.173+0000 7f41bf5ba640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 37628) UID: 0 2023-09-22T21:02:02.172 INFO:tasks.rgw.c2.client.0.smithi204.stdout:2023-09-22T21:02:02.173+0000 7f41c2e01a40 -1 shutting down 2023-09-22T21:02:07.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-09-22T21:02:08.231 INFO:tasks.rgw.c2.client.0:Stopped 2023-09-22T21:02:08.231 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-09-22T21:02:08.231 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-09-22T21:02:08.234 INFO:tasks.rgw.c2.client.0:Started 2023-09-22T21:02:08.234 DEBUG:teuthology.orchestra.run.smithi204:> curl http://smithi204.front.sepia.ceph.com:8000 2023-09-22T21:02:08.246 INFO:teuthology.orchestra.run.smithi204.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:02:08.246 INFO:teuthology.orchestra.run.smithi204.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:02:08.246 INFO:teuthology.orchestra.run.smithi204.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T21:02:08.246 INFO:teuthology.orchestra.run.smithi204.stderr:curl: (7) Failed to connect to smithi204.front.sepia.ceph.com port 8000 after 0 ms: Connection refused 2023-09-22T21:02:08.247 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T21:02:09.248 DEBUG:teuthology.orchestra.run.smithi204:> curl http://smithi204.front.sepia.ceph.com:8000 2023-09-22T21:02:09.259 INFO:teuthology.orchestra.run.smithi204.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:02:09.259 INFO:teuthology.orchestra.run.smithi204.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:02:09.261 INFO:teuthology.orchestra.run.smithi204.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 75061 0 --:--:-- --:--:-- --:--:-- 104k 2023-09-22T21:02:09.262 INFO:teuthology.orchestra.run.smithi204.stdout:anonymous 2023-09-22T21:02:09.262 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-09-22T21:02:09.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi160.front.sepia.ceph.com:8000', '--access-key', 'KT50DM2YGPMJ8OAR', '--secret', 'lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo', '--url', 'http://smithi160.front.sepia.ceph.com:8000', '--access-key', 'KT50DM2YGPMJ8OAR', '--secret', 'lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:02:09.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi160.front.sepia.ceph.com:8000', '--access-key', 'KT50DM2YGPMJ8OAR', '--secret', 'lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo', '--url', 'http://smithi160.front.sepia.ceph.com:8000', '--access-key', 'KT50DM2YGPMJ8OAR', '--secret', 'lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:02:09.263 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi160.front.sepia.ceph.com:8000 --access-key KT50DM2YGPMJ8OAR --secret lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo --url http://smithi160.front.sepia.ceph.com:8000 --access-key KT50DM2YGPMJ8OAR --secret lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-22T21:02:09.353 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:02:09.354 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:02:09.401 INFO:teuthology.orchestra.run.smithi204.stdout:{"id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","name":"test-realm","current_period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","epoch":2} 2023-09-22T21:02:09.401 INFO:tasks.rgw_multisite:creating zone.. 2023-09-22T21:02:09.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'KT50DM2YGPMJ8OAR', '--secret', 'lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo', '--endpoints', 'http://smithi204.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:02:09.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'KT50DM2YGPMJ8OAR', '--secret', 'lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo', '--endpoints', 'http://smithi204.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:02:09.401 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --access-key KT50DM2YGPMJ8OAR --secret lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo --endpoints http://smithi204.front.sepia.ceph.com:8001 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-22T21:02:09.440 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:02:09.440 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:02:09.458 INFO:teuthology.orchestra.run.smithi204.stderr:NOTICE: set zone's realm_id=9e7cb63e-d555-4d20-9c48-393b1c24d1ff 2023-09-22T21:02:09.479 INFO:teuthology.orchestra.run.smithi204.stdout:{"id":"087d4b41-8f81-4835-9e99-026ca13a7b60","name":"b2","domain_root":"b2.rgw.meta:root","control_pool":"b2.rgw.control","gc_pool":"b2.rgw.log:gc","lc_pool":"b2.rgw.log:lc","log_pool":"b2.rgw.log","intent_log_pool":"b2.rgw.log:intent","usage_log_pool":"b2.rgw.log:usage","roles_pool":"b2.rgw.meta:roles","reshard_pool":"b2.rgw.log:reshard","user_keys_pool":"b2.rgw.meta:users.keys","user_email_pool":"b2.rgw.meta:users.email","user_swift_pool":"b2.rgw.meta:users.swift","user_uid_pool":"b2.rgw.meta:users.uid","otp_pool":"b2.rgw.otp","system_key":{"access_key":"KT50DM2YGPMJ8OAR","secret_key":"lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"b2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"b2.rgw.buckets.data"}},"data_extra_pool":"b2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","notif_pool":"b2.rgw.log:notif"} 2023-09-22T21:02:09.479 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph osd pool create b2.rgw.buckets.data 64 64 --cluster c2 2023-09-22T21:02:09.903 INFO:teuthology.orchestra.run.smithi204.stderr:pool 'b2.rgw.buckets.data' created 2023-09-22T21:02:09.938 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph osd pool application enable b2.rgw.buckets.data rgw --cluster c2 2023-09-22T21:02:10.921 INFO:teuthology.orchestra.run.smithi204.stderr:enabled application 'rgw' on pool 'b2.rgw.buckets.data' 2023-09-22T21:02:10.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:02:10.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:02:10.972 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-22T21:02:11.018 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:02:11.019 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:02:11.044 INFO:teuthology.orchestra.run.smithi204.stdout:{"id":"087d4b41-8f81-4835-9e99-026ca13a7b60","name":"b2","domain_root":"b2.rgw.meta:root","control_pool":"b2.rgw.control","gc_pool":"b2.rgw.log:gc","lc_pool":"b2.rgw.log:lc","log_pool":"b2.rgw.log","intent_log_pool":"b2.rgw.log:intent","usage_log_pool":"b2.rgw.log:usage","roles_pool":"b2.rgw.meta:roles","reshard_pool":"b2.rgw.log:reshard","user_keys_pool":"b2.rgw.meta:users.keys","user_email_pool":"b2.rgw.meta:users.email","user_swift_pool":"b2.rgw.meta:users.swift","user_uid_pool":"b2.rgw.meta:users.uid","otp_pool":"b2.rgw.otp","system_key":{"access_key":"KT50DM2YGPMJ8OAR","secret_key":"lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"b2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"b2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"b2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","notif_pool":"b2.rgw.log:notif"} 2023-09-22T21:02:11.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:02:11.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:02:11.045 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-22T21:02:11.076 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:02:11.077 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:02:11.094 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:02:11.092+0000 7ff8dcdc5a40 0 period (03d00cc5-7ad7-4abf-a475-d9ffc8e37106 does not have zone 087d4b41-8f81-4835-9e99-026ca13a7b60 configured 2023-09-22T21:02:13.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-09-22T21:02:18.047 INFO:teuthology.orchestra.run.smithi204.stderr:Sending period to new master zone f132bdb9-20a7-4073-a932-8f99da20691e 2023-09-22T21:02:18.141 INFO:teuthology.orchestra.run.smithi204.stdout:{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","epoch":4,"predecessor_uuid":"a6099785-5160-4a69-b202-552a1b9cb770","sync_status":[],"period_map":{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","zonegroups":[{"id":"a","name":"a","api_name":"","is_master":true,"endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","zones":[{"id":"2b721fee-a149-45da-925b-502d5a3d6853","name":"a2","endpoints":["http://smithi160.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"f132bdb9-20a7-4073-a932-8f99da20691e","name":"a1","endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]},{"id":"b","name":"b","api_name":"","is_master":false,"endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","zones":[{"id":"087d4b41-8f81-4835-9e99-026ca13a7b60","name":"b2","endpoints":["http://smithi204.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","name":"b1","endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"087d4b41-8f81-4835-9e99-026ca13a7b60","val":2135214481},{"key":"2b721fee-a149-45da-925b-502d5a3d6853","val":3586193783},{"key":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","val":4184768938},{"key":"f132bdb9-20a7-4073-a932-8f99da20691e","val":3405200286}]},"master_zonegroup":"a","master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","realm_name":"test-realm","realm_epoch":2} 2023-09-22T21:02:18.142 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-09-22T21:02:18.142 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T21:02:18.188 INFO:tasks.rgw.c2.client.1.smithi204.stdout:2023-09-22T21:02:18.188+0000 7f951a6dc640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 37632) UID: 0 2023-09-22T21:02:18.189 INFO:tasks.rgw.c2.client.1.smithi204.stdout:2023-09-22T21:02:18.188+0000 7f951df23a40 -1 shutting down 2023-09-22T21:02:19.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-09-22T21:02:24.243 INFO:tasks.rgw.c2.client.1:Stopped 2023-09-22T21:02:24.243 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-09-22T21:02:24.243 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-09-22T21:02:24.246 INFO:tasks.rgw.c2.client.1:Started 2023-09-22T21:02:24.246 DEBUG:teuthology.orchestra.run.smithi204:> curl http://smithi204.front.sepia.ceph.com:8001 2023-09-22T21:02:24.257 INFO:teuthology.orchestra.run.smithi204.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:02:24.258 INFO:teuthology.orchestra.run.smithi204.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:02:24.258 INFO:teuthology.orchestra.run.smithi204.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T21:02:24.258 INFO:teuthology.orchestra.run.smithi204.stderr:curl: (7) Failed to connect to smithi204.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T21:02:24.258 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T21:02:24.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-09-22T21:02:25.259 DEBUG:teuthology.orchestra.run.smithi204:> curl http://smithi204.front.sepia.ceph.com:8001 2023-09-22T21:02:25.271 INFO:teuthology.orchestra.run.smithi204.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:02:25.271 INFO:teuthology.orchestra.run.smithi204.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:02:25.274 INFO:teuthology.orchestra.run.smithi204.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 68392 0 --:--:-- --:--:-- --:--:-- 71333 2023-09-22T21:02:25.275 INFO:teuthology.orchestra.run.smithi204.stdout:anonymous 2023-09-22T21:02:25.275 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2023-09-22T21:02:25.275 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2023-09-22T21:02:25.288 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi 2023-09-22T21:02:25.362 INFO:tasks.rgw_multisite_tests:creating test user.. 2023-09-22T21:02:25.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:02:25.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:02:25.363 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:02:25.414 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:02:25.414 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:02:25.459 INFO:teuthology.orchestra.run.smithi160.stdout:{"user_id":"rgw-multisite-test-user","display_name":"Multisite Test User","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"rgw-multisite-test-user","access_key":"RX9T9XC1DLDGURC46TDQ","secret_key":"hT6zqON7vfXAg3X9lAg1pbe32BuYaaz5Vfd17UI4"}],"swift_keys":[],"caps":[{"type":"roles","perm":"*"}],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2023-09-22T21:02:25.475 INFO:rgw_multi.tests:meta checkpoint 2023-09-22T21:02:25.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:02:25.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:02:25.475 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:02:25.508 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:02:25.508 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:02:25.545 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:02:25.561 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2023-09-22T21:02:25.561 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:02:25.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:02:25.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:02:25.562 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:02:25.641 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:02:25.642 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:02:25.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:02:25.674+0000 7ff5faa6ba40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:02:25.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:02:25.674+0000 7ff5faa6ba40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:02:25.693 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:02:25Z"} 2023-09-22T21:02:25.693 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:02:30.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:02:30.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:02:30.694 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:02:30.738 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:02:30.738 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:02:30.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:02:30.774+0000 7ff36140ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:02:30.776 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:02:30.774+0000 7ff36140ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:02:30.793 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:02:30Z"} 2023-09-22T21:02:30.794 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:02:35.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:02:35.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:02:35.795 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:02:35.837 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:02:35.837 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:02:35.874 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:02:35.874+0000 7f470f4b2a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:02:35.875 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:02:35.874+0000 7f470f4b2a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:02:35.892 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:02:35Z"} 2023-09-22T21:02:35.892 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:02:40.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:02:40.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:02:40.893 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:02:40.937 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:02:40.937 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:02:40.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:02:40.970+0000 7f577de53a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:02:40.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:02:40.970+0000 7f577de53a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:02:40.989 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:02:40Z"} 2023-09-22T21:02:40.990 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:02:45.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:02:45.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:02:45.991 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:02:46.035 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:02:46.035 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:02:46.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:02:46.066+0000 7f1e10fe5a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:02:46.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:02:46.070+0000 7f1e10fe5a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:02:46.087 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:02:46Z"} 2023-09-22T21:02:46.087 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:02:51.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:02:51.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:02:51.089 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:02:51.133 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:02:51.133 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:02:51.169 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:02:51.166+0000 7f2f95f2ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:02:51.170 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:02:51.166+0000 7f2f95f2ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:02:51.186 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:02:51Z"} 2023-09-22T21:02:51.187 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:02:56.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:02:56.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:02:56.188 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:02:56.232 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:02:56.233 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:02:56.268 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:02:56.266+0000 7f56d4497a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:02:56.269 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:02:56.266+0000 7f56d4497a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:02:56.286 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:02:56Z"} 2023-09-22T21:02:56.286 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:03:01.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:01.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:01.287 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:03:01.331 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:03:01.332 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:03:01.368 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:01.366+0000 7f9e1b465a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:01.369 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:01.366+0000 7f9e1b465a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:01.385 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:03:01Z"} 2023-09-22T21:03:01.385 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:03:06.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:06.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:06.386 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:03:06.430 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:03:06.430 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:03:06.465 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:06.462+0000 7f624e92aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:06.466 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:06.466+0000 7f624e92aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:06.483 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:03:06Z"} 2023-09-22T21:03:06.483 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:03:11.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:11.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:11.485 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:03:11.529 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:03:11.529 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:03:11.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:11.562+0000 7f0e22148a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:11.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:11.562+0000 7f0e22148a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:11.582 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:03:11Z"} 2023-09-22T21:03:11.583 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:03:16.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:16.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:16.585 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:03:16.627 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:03:16.628 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:03:16.665 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:16.662+0000 7f288bf33a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:16.666 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:16.662+0000 7f288bf33a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:16.682 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:03:16Z"} 2023-09-22T21:03:16.683 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:03:21.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:21.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:21.685 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:03:21.730 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:03:21.731 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:03:21.768 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:21.766+0000 7fe7432d4a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:21.769 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:21.766+0000 7fe7432d4a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:21.786 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:03:21Z"} 2023-09-22T21:03:21.786 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:03:26.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:26.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:26.787 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:03:26.831 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:03:26.831 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:03:26.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:26.866+0000 7f1e2b427a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:26.868 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:26.866+0000 7f1e2b427a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:26.884 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:03:26Z"} 2023-09-22T21:03:26.884 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:03:31.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:31.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:31.885 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:03:31.929 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:03:31.930 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:03:31.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:31.962+0000 7f3c94352a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:31.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:31.962+0000 7f3c94352a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:31.982 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:03:31Z"} 2023-09-22T21:03:31.983 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:03:36.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:36.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:36.984 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:03:37.027 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:03:37.028 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:03:37.064 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:37.062+0000 7f75a205ba40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:37.065 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:37.062+0000 7f75a205ba40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:37.081 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:03:37Z"} 2023-09-22T21:03:37.082 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:03:42.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:42.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:42.083 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:03:42.126 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:03:42.126 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:03:42.163 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:42.162+0000 7f3ab6b2fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:42.164 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:42.162+0000 7f3ab6b2fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:42.181 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:03:42Z"} 2023-09-22T21:03:42.181 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:03:47.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:47.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:47.182 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:03:47.226 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:03:47.227 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:03:47.262 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:47.258+0000 7f8fcd9f1a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:47.262 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:47.262+0000 7f8fcd9f1a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:47.279 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:03:47Z"} 2023-09-22T21:03:47.279 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:03:52.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:52.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:52.281 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:03:52.324 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:03:52.324 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:03:52.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:52.358+0000 7f4209922a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:52.361 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:52.358+0000 7f4209922a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:52.377 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:03:52Z"} 2023-09-22T21:03:52.377 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:03:57.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:57.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:03:57.379 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:03:57.423 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:03:57.423 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:03:57.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:57.457+0000 7fa88debda40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:57.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:03:57.457+0000 7fa88debda40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:03:57.476 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:03:57Z"} 2023-09-22T21:03:57.476 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:04:02.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:02.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:02.477 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:02.520 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:02.521 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:02.558 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:02.557+0000 7f591e84aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:02.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:02.557+0000 7f591e84aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:02.577 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:02Z"} 2023-09-22T21:04:02.578 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:04:07.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:07.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:07.579 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:07.624 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:07.624 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:07.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:07.657+0000 7ffaba879a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:07.661 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:07.657+0000 7ffaba879a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:07.677 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:07Z"} 2023-09-22T21:04:07.677 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:04:12.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:12.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:12.679 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:12.724 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:12.725 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:12.764 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:12.761+0000 7ff262ea8a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:12.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:12.761+0000 7ff262ea8a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:12.781 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:12Z"} 2023-09-22T21:04:12.782 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:04:17.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:17.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:17.784 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:17.826 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:17.826 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:17.866 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:17.861+0000 7f5aea53aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:17.866 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:17.865+0000 7f5aea53aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:17.883 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:17Z"} 2023-09-22T21:04:17.884 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1695416545.459338_27.1 target= 2023-09-22T21:04:22.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:22.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:22.885 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:22.928 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:22.928 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:22.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:22.961+0000 7f89a8672a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:22.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:22.961+0000 7f89a8672a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:22.982 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:22Z"} 2023-09-22T21:04:22.982 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:04:22.982 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-09-22T21:04:22.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:22.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:22.983 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:23.028 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:23.029 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:23.066 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:04:23.065+0000 7f2f7cfa9a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:23.067 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:04:23.065+0000 7f2f7cfa9a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:23.085 INFO:teuthology.orchestra.run.smithi204.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:23Z"} 2023-09-22T21:04:23.085 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-09-22T21:04:23.085 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-09-22T21:04:23.085 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:23.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:23.086 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:23.164 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:23.164 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:23.203 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:04:23.201+0000 7f7a6d329a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:23.203 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:04:23.201+0000 7f7a6d329a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:23.220 INFO:teuthology.orchestra.run.smithi204.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:23Z"} 2023-09-22T21:04:23.220 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-09-22T21:04:23.220 INFO:rgw_multi.tests:meta checkpoint 2023-09-22T21:04:23.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:23.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:23.221 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:23.262 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:23.262 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:23.301 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:04:23.318 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2023-09-22T21:04:23.319 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:04:23.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:23.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:23.319 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:23.397 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:23.397 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:23.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:23.433+0000 7fc01072ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:23.434 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:23.437+0000 7fc01072ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:23.451 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:23Z"} 2023-09-22T21:04:23.451 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:04:23.451 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-09-22T21:04:23.451 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:23.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:23.452 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:23.494 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:23.494 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:23.532 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:04:23.529+0000 7f7aefe06a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:23.533 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:04:23.529+0000 7f7aefe06a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:23.550 INFO:teuthology.orchestra.run.smithi204.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:23Z"} 2023-09-22T21:04:23.550 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-09-22T21:04:23.551 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-09-22T21:04:23.551 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:23.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:23.551 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:23.634 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:23.634 INFO:teuthology.orchestra.run.smithi204.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:23.670 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:04:23.669+0000 7f81f9733a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:23.671 INFO:teuthology.orchestra.run.smithi204.stderr:2023-09-22T21:04:23.669+0000 7f81f9733a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:23.688 INFO:teuthology.orchestra.run.smithi204.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:23Z"} 2023-09-22T21:04:23.688 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-09-22T21:04:23.688 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2023-09-22T21:04:24.052 DEBUG:boto:Using access key provided by client. 2023-09-22T21:04:24.052 DEBUG:boto:Using secret key provided by client. 2023-09-22T21:04:24.053 DEBUG:boto:Using access key provided by client. 2023-09-22T21:04:24.053 DEBUG:boto:Using secret key provided by client. 2023-09-22T21:04:24.054 DEBUG:boto:Using access key provided by client. 2023-09-22T21:04:24.054 DEBUG:boto:Using secret key provided by client. 2023-09-22T21:04:24.055 DEBUG:boto:Using access key provided by client. 2023-09-22T21:04:24.055 DEBUG:boto:Using secret key provided by client. 2023-09-22T21:04:24.056 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-1 2023-09-22T21:04:24.056 DEBUG:boto:path=/ffzdqu-1/ 2023-09-22T21:04:24.056 DEBUG:boto:auth_path=/ffzdqu-1/ 2023-09-22T21:04:24.056 DEBUG:boto:Method: PUT 2023-09-22T21:04:24.057 DEBUG:boto:Path: /ffzdqu-1/ 2023-09-22T21:04:24.057 DEBUG:boto:Data: 2023-09-22T21:04:24.057 DEBUG:boto:Headers: {} 2023-09-22T21:04:24.057 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:24.057 DEBUG:boto:Port: 8000 2023-09-22T21:04:24.058 DEBUG:boto:Params: {} 2023-09-22T21:04:24.058 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:04:24.058 DEBUG:boto:Token: None 2023-09-22T21:04:24.059 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:04:24 GMT /ffzdqu-1/ 2023-09-22T21:04:24.059 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KXiRepQB+OfRUY5NWE4SnBqHG58= 2023-09-22T21:04:24.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KXiRepQB+OfRUY5NWE4SnBqHG58=', 'Content-Length': '0'} 2023-09-22T21:04:26.855 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000086ccbd1ebbf79c20-00650e0158-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:26.855 INFO:rgw_multi.tests:create bucket zone=a2 name=ffzdqu-2 2023-09-22T21:04:26.856 DEBUG:boto:path=/ffzdqu-2/ 2023-09-22T21:04:26.856 DEBUG:boto:auth_path=/ffzdqu-2/ 2023-09-22T21:04:26.856 DEBUG:boto:Method: PUT 2023-09-22T21:04:26.856 DEBUG:boto:Path: /ffzdqu-2/ 2023-09-22T21:04:26.856 DEBUG:boto:Data: 2023-09-22T21:04:26.856 DEBUG:boto:Headers: {} 2023-09-22T21:04:26.856 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:26.857 DEBUG:boto:Port: 8001 2023-09-22T21:04:26.857 DEBUG:boto:Params: {} 2023-09-22T21:04:26.857 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:04:26.857 DEBUG:boto:Token: None 2023-09-22T21:04:26.857 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:04:26 GMT /ffzdqu-2/ 2023-09-22T21:04:26.857 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QB2yTOQrELwq21R076QvaM3UEGY= 2023-09-22T21:04:26.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QB2yTOQrELwq21R076QvaM3UEGY=', 'Content-Length': '0'} 2023-09-22T21:04:29.895 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008da5ca564878dbb1-00650e015a-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:29.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:29.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:29.896 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:29.934 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:29.935 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:29.971 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:04:29.988 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695416666.882702_55.1","last_update":"2023-09-22T21:04:26.882702Z"}] 2023-09-22T21:04:29.989 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '', 3: '1_1695416666.882702_55.1'} 2023-09-22T21:04:29.989 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:04:29.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:29.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:29.989 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:30.064 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:30.064 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:30.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:30.101+0000 7f9d43790a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:30.101 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:30.101+0000 7f9d43790a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:30.118 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416666.825318_48.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:26.825318Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:30Z"} 2023-09-22T21:04:30.118 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416666.825318_48.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:26.825318Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:30Z"} 2023-09-22T21:04:30.118 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416666.825318_48.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:04:26.825318Z', 'realm_epoch': 2}}] 2023-09-22T21:04:30.118 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '', 3: '1_1695416666.882702_55.1'} 2023-09-22T21:04:30.119 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '', 3: '1_1695416666.825318_48.1'} 2023-09-22T21:04:30.119 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1695416666.882702_55.1 target=1_1695416666.825318_48.1 2023-09-22T21:04:35.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:35.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:35.121 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:35.164 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:35.164 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:35.200 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:35.201+0000 7f76250c5a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:35.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:35.201+0000 7f76250c5a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:35.217 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416666.882702_55.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:26.882702Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:35Z"} 2023-09-22T21:04:35.218 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416666.882702_55.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:26.882702Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:35Z"} 2023-09-22T21:04:35.218 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416666.882702_55.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:04:26.882702Z', 'realm_epoch': 2}}] 2023-09-22T21:04:35.218 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '', 3: '1_1695416666.882702_55.1'} 2023-09-22T21:04:35.218 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '', 3: '1_1695416666.882702_55.1'} 2023-09-22T21:04:35.218 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:04:35.219 DEBUG:boto:path=/ffzdqu-1/ 2023-09-22T21:04:35.219 DEBUG:boto:auth_path=/ffzdqu-1/ 2023-09-22T21:04:35.219 DEBUG:boto:Method: HEAD 2023-09-22T21:04:35.219 DEBUG:boto:Path: /ffzdqu-1/ 2023-09-22T21:04:35.219 DEBUG:boto:Data: 2023-09-22T21:04:35.219 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.219 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:35.219 DEBUG:boto:Port: 8000 2023-09-22T21:04:35.219 DEBUG:boto:Params: {} 2023-09-22T21:04:35.220 DEBUG:boto:Token: None 2023-09-22T21:04:35.220 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-1/ 2023-09-22T21:04:35.220 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EyjgQZxSPdSXLHlmEFWauE9QCoA= 2023-09-22T21:04:35.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EyjgQZxSPdSXLHlmEFWauE9QCoA=', 'Content-Length': '0'} 2023-09-22T21:04:35.222 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fcea0c7ae00e621d-00650e0163-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.223 DEBUG:boto:path=/ffzdqu-2/ 2023-09-22T21:04:35.223 DEBUG:boto:auth_path=/ffzdqu-2/ 2023-09-22T21:04:35.223 DEBUG:boto:Method: HEAD 2023-09-22T21:04:35.223 DEBUG:boto:Path: /ffzdqu-2/ 2023-09-22T21:04:35.223 DEBUG:boto:Data: 2023-09-22T21:04:35.223 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.223 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:35.223 DEBUG:boto:Port: 8000 2023-09-22T21:04:35.224 DEBUG:boto:Params: {} 2023-09-22T21:04:35.224 DEBUG:boto:Token: None 2023-09-22T21:04:35.224 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-2/ 2023-09-22T21:04:35.224 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+5QpwZmPrJde/HM1UyRd2itrj2E= 2023-09-22T21:04:35.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+5QpwZmPrJde/HM1UyRd2itrj2E=', 'Content-Length': '0'} 2023-09-22T21:04:35.226 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d35d730eeae758a1-00650e0163-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.226 DEBUG:boto:path=/ffzdqu-1/ 2023-09-22T21:04:35.226 DEBUG:boto:auth_path=/ffzdqu-1/ 2023-09-22T21:04:35.227 DEBUG:boto:Method: HEAD 2023-09-22T21:04:35.227 DEBUG:boto:Path: /ffzdqu-1/ 2023-09-22T21:04:35.227 DEBUG:boto:Data: 2023-09-22T21:04:35.227 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.227 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:35.227 DEBUG:boto:Port: 8001 2023-09-22T21:04:35.227 DEBUG:boto:Params: {} 2023-09-22T21:04:35.227 DEBUG:boto:Token: None 2023-09-22T21:04:35.228 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-1/ 2023-09-22T21:04:35.228 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EyjgQZxSPdSXLHlmEFWauE9QCoA= 2023-09-22T21:04:35.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EyjgQZxSPdSXLHlmEFWauE9QCoA=', 'Content-Length': '0'} 2023-09-22T21:04:35.230 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008cd2ae53041d38fb-00650e0163-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.230 DEBUG:boto:path=/ffzdqu-2/ 2023-09-22T21:04:35.230 DEBUG:boto:auth_path=/ffzdqu-2/ 2023-09-22T21:04:35.230 DEBUG:boto:Method: HEAD 2023-09-22T21:04:35.230 DEBUG:boto:Path: /ffzdqu-2/ 2023-09-22T21:04:35.231 DEBUG:boto:Data: 2023-09-22T21:04:35.231 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.231 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:35.231 DEBUG:boto:Port: 8001 2023-09-22T21:04:35.231 DEBUG:boto:Params: {} 2023-09-22T21:04:35.231 DEBUG:boto:Token: None 2023-09-22T21:04:35.231 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-2/ 2023-09-22T21:04:35.231 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+5QpwZmPrJde/HM1UyRd2itrj2E= 2023-09-22T21:04:35.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+5QpwZmPrJde/HM1UyRd2itrj2E=', 'Content-Length': '0'} 2023-09-22T21:04:35.233 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e621be1bca27f35-00650e0163-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2023-09-22T21:04:35.234 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-3 2023-09-22T21:04:35.235 DEBUG:boto:path=/ffzdqu-3/ 2023-09-22T21:04:35.235 DEBUG:boto:auth_path=/ffzdqu-3/ 2023-09-22T21:04:35.235 DEBUG:boto:Method: PUT 2023-09-22T21:04:35.235 DEBUG:boto:Path: /ffzdqu-3/ 2023-09-22T21:04:35.235 DEBUG:boto:Data: 2023-09-22T21:04:35.235 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.235 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:35.235 DEBUG:boto:Port: 8000 2023-09-22T21:04:35.235 DEBUG:boto:Params: {} 2023-09-22T21:04:35.236 DEBUG:boto:Token: None 2023-09-22T21:04:35.236 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-3/ 2023-09-22T21:04:35.236 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qplKiqj8oKOVZ5d830cRHzTmmz8= 2023-09-22T21:04:35.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qplKiqj8oKOVZ5d830cRHzTmmz8=', 'Content-Length': '0'} 2023-09-22T21:04:35.252 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001eae375184ab132d-00650e0163-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.252 INFO:rgw_multi.tests:create bucket zone=a2 name=ffzdqu-4 2023-09-22T21:04:35.253 DEBUG:boto:path=/ffzdqu-4/ 2023-09-22T21:04:35.253 DEBUG:boto:auth_path=/ffzdqu-4/ 2023-09-22T21:04:35.253 DEBUG:boto:Method: PUT 2023-09-22T21:04:35.253 DEBUG:boto:Path: /ffzdqu-4/ 2023-09-22T21:04:35.253 DEBUG:boto:Data: 2023-09-22T21:04:35.253 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.253 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:35.253 DEBUG:boto:Port: 8001 2023-09-22T21:04:35.253 DEBUG:boto:Params: {} 2023-09-22T21:04:35.254 DEBUG:boto:Token: None 2023-09-22T21:04:35.254 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-4/ 2023-09-22T21:04:35.254 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kTjHkeZRTkzm8sQJi6Oyn+xGesQ= 2023-09-22T21:04:35.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kTjHkeZRTkzm8sQJi6Oyn+xGesQ=', 'Content-Length': '0'} 2023-09-22T21:04:35.288 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000028d2760bff3ef0a3-00650e0163-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:35.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:35.289 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:35.325 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:35.325 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:35.363 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:04:35.385 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695416675.274405_63.1","last_update":"2023-09-22T21:04:35.274405Z"}] 2023-09-22T21:04:35.385 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '', 3: '1_1695416675.274405_63.1'} 2023-09-22T21:04:35.385 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:04:35.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:35.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:35.386 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:35.461 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:35.461 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:35.495 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:35.497+0000 7f73cbf96a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:35.496 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:35.497+0000 7f73cbf96a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:35.512 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416675.274405_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:35.274405Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:35Z"} 2023-09-22T21:04:35.512 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416675.274405_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:35.274405Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:35Z"} 2023-09-22T21:04:35.512 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416675.274405_63.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:04:35.274405Z', 'realm_epoch': 2}}] 2023-09-22T21:04:35.513 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '', 3: '1_1695416675.274405_63.1'} 2023-09-22T21:04:35.513 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '', 3: '1_1695416675.274405_63.1'} 2023-09-22T21:04:35.513 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:04:35.513 DEBUG:boto:path=/ffzdqu-3/ 2023-09-22T21:04:35.513 DEBUG:boto:auth_path=/ffzdqu-3/ 2023-09-22T21:04:35.513 DEBUG:boto:Method: HEAD 2023-09-22T21:04:35.513 DEBUG:boto:Path: /ffzdqu-3/ 2023-09-22T21:04:35.513 DEBUG:boto:Data: 2023-09-22T21:04:35.513 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.514 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:35.514 DEBUG:boto:Port: 8000 2023-09-22T21:04:35.514 DEBUG:boto:Params: {} 2023-09-22T21:04:35.514 DEBUG:boto:Token: None 2023-09-22T21:04:35.514 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-3/ 2023-09-22T21:04:35.514 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Wsbyarc5UCxskcue6cgbvmo1Ews= 2023-09-22T21:04:35.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Wsbyarc5UCxskcue6cgbvmo1Ews=', 'Content-Length': '0'} 2023-09-22T21:04:35.516 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000b37d0a8cdec6e7-00650e0163-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.517 DEBUG:boto:path=/ffzdqu-4/ 2023-09-22T21:04:35.517 DEBUG:boto:auth_path=/ffzdqu-4/ 2023-09-22T21:04:35.517 DEBUG:boto:Method: HEAD 2023-09-22T21:04:35.517 DEBUG:boto:Path: /ffzdqu-4/ 2023-09-22T21:04:35.517 DEBUG:boto:Data: 2023-09-22T21:04:35.517 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.517 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:35.517 DEBUG:boto:Port: 8000 2023-09-22T21:04:35.517 DEBUG:boto:Params: {} 2023-09-22T21:04:35.518 DEBUG:boto:Token: None 2023-09-22T21:04:35.518 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-4/ 2023-09-22T21:04:35.518 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ejg+ORAyKPkojYt31e7H9pTwDTc= 2023-09-22T21:04:35.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ejg+ORAyKPkojYt31e7H9pTwDTc=', 'Content-Length': '0'} 2023-09-22T21:04:35.520 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d36ebb42c7ef64c-00650e0163-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.520 DEBUG:boto:path=/ffzdqu-3/ 2023-09-22T21:04:35.520 DEBUG:boto:auth_path=/ffzdqu-3/ 2023-09-22T21:04:35.520 DEBUG:boto:Method: HEAD 2023-09-22T21:04:35.520 DEBUG:boto:Path: /ffzdqu-3/ 2023-09-22T21:04:35.520 DEBUG:boto:Data: 2023-09-22T21:04:35.521 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.521 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:35.521 DEBUG:boto:Port: 8001 2023-09-22T21:04:35.521 DEBUG:boto:Params: {} 2023-09-22T21:04:35.521 DEBUG:boto:Token: None 2023-09-22T21:04:35.521 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-3/ 2023-09-22T21:04:35.521 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Wsbyarc5UCxskcue6cgbvmo1Ews= 2023-09-22T21:04:35.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Wsbyarc5UCxskcue6cgbvmo1Ews=', 'Content-Length': '0'} 2023-09-22T21:04:35.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007358cecd4980ae76-00650e0163-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.523 DEBUG:boto:path=/ffzdqu-4/ 2023-09-22T21:04:35.524 DEBUG:boto:auth_path=/ffzdqu-4/ 2023-09-22T21:04:35.524 DEBUG:boto:Method: HEAD 2023-09-22T21:04:35.524 DEBUG:boto:Path: /ffzdqu-4/ 2023-09-22T21:04:35.524 DEBUG:boto:Data: 2023-09-22T21:04:35.524 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.524 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:35.524 DEBUG:boto:Port: 8001 2023-09-22T21:04:35.524 DEBUG:boto:Params: {} 2023-09-22T21:04:35.524 DEBUG:boto:Token: None 2023-09-22T21:04:35.525 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-4/ 2023-09-22T21:04:35.525 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ejg+ORAyKPkojYt31e7H9pTwDTc= 2023-09-22T21:04:35.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ejg+ORAyKPkojYt31e7H9pTwDTc=', 'Content-Length': '0'} 2023-09-22T21:04:35.527 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001cd9209bf273a055-00650e0163-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.527 DEBUG:boto:path=/ffzdqu-3/ 2023-09-22T21:04:35.527 DEBUG:boto:auth_path=/ffzdqu-3/ 2023-09-22T21:04:35.527 DEBUG:boto:Method: PUT 2023-09-22T21:04:35.527 DEBUG:boto:Path: /ffzdqu-3/ 2023-09-22T21:04:35.527 DEBUG:boto:Data: 2023-09-22T21:04:35.527 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.527 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:35.528 DEBUG:boto:Port: 8000 2023-09-22T21:04:35.528 DEBUG:boto:Params: {} 2023-09-22T21:04:35.528 DEBUG:boto:Token: None 2023-09-22T21:04:35.528 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-3/ 2023-09-22T21:04:35.528 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qplKiqj8oKOVZ5d830cRHzTmmz8= 2023-09-22T21:04:35.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qplKiqj8oKOVZ5d830cRHzTmmz8=', 'Content-Length': '0'} 2023-09-22T21:04:35.531 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003ae9013c42fd10b6-00650e0163-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.531 DEBUG:boto:path=/ffzdqu-4/ 2023-09-22T21:04:35.531 DEBUG:boto:auth_path=/ffzdqu-4/ 2023-09-22T21:04:35.531 DEBUG:boto:Method: PUT 2023-09-22T21:04:35.531 DEBUG:boto:Path: /ffzdqu-4/ 2023-09-22T21:04:35.531 DEBUG:boto:Data: 2023-09-22T21:04:35.532 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.532 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:35.532 DEBUG:boto:Port: 8000 2023-09-22T21:04:35.532 DEBUG:boto:Params: {} 2023-09-22T21:04:35.532 DEBUG:boto:Token: None 2023-09-22T21:04:35.532 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-4/ 2023-09-22T21:04:35.532 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kTjHkeZRTkzm8sQJi6Oyn+xGesQ= 2023-09-22T21:04:35.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kTjHkeZRTkzm8sQJi6Oyn+xGesQ=', 'Content-Length': '0'} 2023-09-22T21:04:35.535 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d6f593c94da041ba-00650e0163-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.535 DEBUG:boto:path=/ffzdqu-3/ 2023-09-22T21:04:35.535 DEBUG:boto:auth_path=/ffzdqu-3/ 2023-09-22T21:04:35.535 DEBUG:boto:Method: PUT 2023-09-22T21:04:35.535 DEBUG:boto:Path: /ffzdqu-3/ 2023-09-22T21:04:35.536 DEBUG:boto:Data: 2023-09-22T21:04:35.536 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.536 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:35.536 DEBUG:boto:Port: 8001 2023-09-22T21:04:35.536 DEBUG:boto:Params: {} 2023-09-22T21:04:35.536 DEBUG:boto:Token: None 2023-09-22T21:04:35.536 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-3/ 2023-09-22T21:04:35.536 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qplKiqj8oKOVZ5d830cRHzTmmz8= 2023-09-22T21:04:35.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qplKiqj8oKOVZ5d830cRHzTmmz8=', 'Content-Length': '0'} 2023-09-22T21:04:35.542 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e57ed39f3cf7467f-00650e0163-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.542 DEBUG:boto:path=/ffzdqu-4/ 2023-09-22T21:04:35.542 DEBUG:boto:auth_path=/ffzdqu-4/ 2023-09-22T21:04:35.542 DEBUG:boto:Method: PUT 2023-09-22T21:04:35.542 DEBUG:boto:Path: /ffzdqu-4/ 2023-09-22T21:04:35.542 DEBUG:boto:Data: 2023-09-22T21:04:35.542 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.542 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:35.543 DEBUG:boto:Port: 8001 2023-09-22T21:04:35.543 DEBUG:boto:Params: {} 2023-09-22T21:04:35.543 DEBUG:boto:Token: None 2023-09-22T21:04:35.543 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-4/ 2023-09-22T21:04:35.543 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kTjHkeZRTkzm8sQJi6Oyn+xGesQ= 2023-09-22T21:04:35.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kTjHkeZRTkzm8sQJi6Oyn+xGesQ=', 'Content-Length': '0'} 2023-09-22T21:04:35.593 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e2c93f8989b37501-00650e0163-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.593 DEBUG:boto:path=/ffzdqu-3/ 2023-09-22T21:04:35.593 DEBUG:boto:auth_path=/ffzdqu-3/ 2023-09-22T21:04:35.594 DEBUG:boto:Method: HEAD 2023-09-22T21:04:35.594 DEBUG:boto:Path: /ffzdqu-3/ 2023-09-22T21:04:35.594 DEBUG:boto:Data: 2023-09-22T21:04:35.594 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.594 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:35.594 DEBUG:boto:Port: 8000 2023-09-22T21:04:35.594 DEBUG:boto:Params: {} 2023-09-22T21:04:35.594 DEBUG:boto:Token: None 2023-09-22T21:04:35.594 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-3/ 2023-09-22T21:04:35.595 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Wsbyarc5UCxskcue6cgbvmo1Ews= 2023-09-22T21:04:35.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Wsbyarc5UCxskcue6cgbvmo1Ews=', 'Content-Length': '0'} 2023-09-22T21:04:35.598 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000955c8f920a3b9af3-00650e0163-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.599 DEBUG:boto:path=/ffzdqu-4/ 2023-09-22T21:04:35.599 DEBUG:boto:auth_path=/ffzdqu-4/ 2023-09-22T21:04:35.599 DEBUG:boto:Method: HEAD 2023-09-22T21:04:35.599 DEBUG:boto:Path: /ffzdqu-4/ 2023-09-22T21:04:35.599 DEBUG:boto:Data: 2023-09-22T21:04:35.599 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:35.599 DEBUG:boto:Port: 8000 2023-09-22T21:04:35.599 DEBUG:boto:Params: {} 2023-09-22T21:04:35.600 DEBUG:boto:Token: None 2023-09-22T21:04:35.600 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-4/ 2023-09-22T21:04:35.600 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ejg+ORAyKPkojYt31e7H9pTwDTc= 2023-09-22T21:04:35.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ejg+ORAyKPkojYt31e7H9pTwDTc=', 'Content-Length': '0'} 2023-09-22T21:04:35.603 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047339c247dd31fae-00650e0163-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.603 DEBUG:boto:path=/ffzdqu-3/ 2023-09-22T21:04:35.604 DEBUG:boto:auth_path=/ffzdqu-3/ 2023-09-22T21:04:35.604 DEBUG:boto:Method: HEAD 2023-09-22T21:04:35.604 DEBUG:boto:Path: /ffzdqu-3/ 2023-09-22T21:04:35.604 DEBUG:boto:Data: 2023-09-22T21:04:35.604 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.604 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:35.604 DEBUG:boto:Port: 8001 2023-09-22T21:04:35.604 DEBUG:boto:Params: {} 2023-09-22T21:04:35.604 DEBUG:boto:Token: None 2023-09-22T21:04:35.605 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-3/ 2023-09-22T21:04:35.605 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Wsbyarc5UCxskcue6cgbvmo1Ews= 2023-09-22T21:04:35.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Wsbyarc5UCxskcue6cgbvmo1Ews=', 'Content-Length': '0'} 2023-09-22T21:04:35.608 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008cb1f9a0be31a5c0-00650e0163-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.608 DEBUG:boto:path=/ffzdqu-4/ 2023-09-22T21:04:35.608 DEBUG:boto:auth_path=/ffzdqu-4/ 2023-09-22T21:04:35.609 DEBUG:boto:Method: HEAD 2023-09-22T21:04:35.609 DEBUG:boto:Path: /ffzdqu-4/ 2023-09-22T21:04:35.609 DEBUG:boto:Data: 2023-09-22T21:04:35.609 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.609 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:35.609 DEBUG:boto:Port: 8001 2023-09-22T21:04:35.609 DEBUG:boto:Params: {} 2023-09-22T21:04:35.609 DEBUG:boto:Token: None 2023-09-22T21:04:35.609 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-4/ 2023-09-22T21:04:35.610 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ejg+ORAyKPkojYt31e7H9pTwDTc= 2023-09-22T21:04:35.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ejg+ORAyKPkojYt31e7H9pTwDTc=', 'Content-Length': '0'} 2023-09-22T21:04:35.613 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d365fba5cb441950-00650e0163-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:35.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:35.613 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:35.654 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:35.654 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:35.692 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:04:35.709 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695416675.274405_63.1","last_update":"2023-09-22T21:04:35.274405Z"}] 2023-09-22T21:04:35.709 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '', 3: '1_1695416675.274405_63.1'} 2023-09-22T21:04:35.709 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:04:35.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:35.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:35.710 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:35.787 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:35.787 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:35.826 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:35.829+0000 7f4de5ebda40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:35.827 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:35.829+0000 7f4de5ebda40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:35.844 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416675.274405_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:35.274405Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:35Z"} 2023-09-22T21:04:35.844 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416675.274405_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:35.274405Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:35Z"} 2023-09-22T21:04:35.844 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416675.274405_63.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:04:35.274405Z', 'realm_epoch': 2}}] 2023-09-22T21:04:35.844 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '', 3: '1_1695416675.274405_63.1'} 2023-09-22T21:04:35.844 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '', 3: '1_1695416675.274405_63.1'} 2023-09-22T21:04:35.845 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:04:35.845 DEBUG:boto:path=/ffzdqu-3/ 2023-09-22T21:04:35.845 DEBUG:boto:auth_path=/ffzdqu-3/ 2023-09-22T21:04:35.845 DEBUG:boto:Method: HEAD 2023-09-22T21:04:35.845 DEBUG:boto:Path: /ffzdqu-3/ 2023-09-22T21:04:35.845 DEBUG:boto:Data: 2023-09-22T21:04:35.845 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.846 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:35.846 DEBUG:boto:Port: 8000 2023-09-22T21:04:35.846 DEBUG:boto:Params: {} 2023-09-22T21:04:35.846 DEBUG:boto:Token: None 2023-09-22T21:04:35.846 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-3/ 2023-09-22T21:04:35.846 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Wsbyarc5UCxskcue6cgbvmo1Ews= 2023-09-22T21:04:35.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Wsbyarc5UCxskcue6cgbvmo1Ews=', 'Content-Length': '0'} 2023-09-22T21:04:35.849 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e70dc5fed6503633-00650e0163-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.849 DEBUG:boto:path=/ffzdqu-4/ 2023-09-22T21:04:35.849 DEBUG:boto:auth_path=/ffzdqu-4/ 2023-09-22T21:04:35.849 DEBUG:boto:Method: HEAD 2023-09-22T21:04:35.849 DEBUG:boto:Path: /ffzdqu-4/ 2023-09-22T21:04:35.849 DEBUG:boto:Data: 2023-09-22T21:04:35.849 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.850 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:35.850 DEBUG:boto:Port: 8000 2023-09-22T21:04:35.850 DEBUG:boto:Params: {} 2023-09-22T21:04:35.850 DEBUG:boto:Token: None 2023-09-22T21:04:35.850 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-4/ 2023-09-22T21:04:35.850 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ejg+ORAyKPkojYt31e7H9pTwDTc= 2023-09-22T21:04:35.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ejg+ORAyKPkojYt31e7H9pTwDTc=', 'Content-Length': '0'} 2023-09-22T21:04:35.852 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee71285fd4629277-00650e0163-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.853 DEBUG:boto:path=/ffzdqu-3/ 2023-09-22T21:04:35.853 DEBUG:boto:auth_path=/ffzdqu-3/ 2023-09-22T21:04:35.853 DEBUG:boto:Method: HEAD 2023-09-22T21:04:35.853 DEBUG:boto:Path: /ffzdqu-3/ 2023-09-22T21:04:35.853 DEBUG:boto:Data: 2023-09-22T21:04:35.853 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.853 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:35.853 DEBUG:boto:Port: 8001 2023-09-22T21:04:35.854 DEBUG:boto:Params: {} 2023-09-22T21:04:35.854 DEBUG:boto:Token: None 2023-09-22T21:04:35.854 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-3/ 2023-09-22T21:04:35.854 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Wsbyarc5UCxskcue6cgbvmo1Ews= 2023-09-22T21:04:35.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Wsbyarc5UCxskcue6cgbvmo1Ews=', 'Content-Length': '0'} 2023-09-22T21:04:35.856 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f680134ccae9ca3d-00650e0163-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.856 DEBUG:boto:path=/ffzdqu-4/ 2023-09-22T21:04:35.856 DEBUG:boto:auth_path=/ffzdqu-4/ 2023-09-22T21:04:35.857 DEBUG:boto:Method: HEAD 2023-09-22T21:04:35.857 DEBUG:boto:Path: /ffzdqu-4/ 2023-09-22T21:04:35.857 DEBUG:boto:Data: 2023-09-22T21:04:35.857 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.857 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:35.857 DEBUG:boto:Port: 8001 2023-09-22T21:04:35.857 DEBUG:boto:Params: {} 2023-09-22T21:04:35.857 DEBUG:boto:Token: None 2023-09-22T21:04:35.857 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-4/ 2023-09-22T21:04:35.858 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ejg+ORAyKPkojYt31e7H9pTwDTc= 2023-09-22T21:04:35.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ejg+ORAyKPkojYt31e7H9pTwDTc=', 'Content-Length': '0'} 2023-09-22T21:04:35.860 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063b872cbb610ffda-00650e0163-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2023-09-22T21:04:35.860 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-5 2023-09-22T21:04:35.861 DEBUG:boto:path=/ffzdqu-5/ 2023-09-22T21:04:35.861 DEBUG:boto:auth_path=/ffzdqu-5/ 2023-09-22T21:04:35.861 DEBUG:boto:Method: PUT 2023-09-22T21:04:35.861 DEBUG:boto:Path: /ffzdqu-5/ 2023-09-22T21:04:35.861 DEBUG:boto:Data: 2023-09-22T21:04:35.861 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.861 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:35.861 DEBUG:boto:Port: 8000 2023-09-22T21:04:35.861 DEBUG:boto:Params: {} 2023-09-22T21:04:35.862 DEBUG:boto:Token: None 2023-09-22T21:04:35.862 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-5/ 2023-09-22T21:04:35.862 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WiOXAh9lQPrj+XgXuKNaIrWRJNU= 2023-09-22T21:04:35.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WiOXAh9lQPrj+XgXuKNaIrWRJNU=', 'Content-Length': '0'} 2023-09-22T21:04:35.878 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000072eb67caae2a5e27-00650e0163-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.879 INFO:rgw_multi.tests:create bucket zone=a2 name=ffzdqu-6 2023-09-22T21:04:35.879 DEBUG:boto:path=/ffzdqu-6/ 2023-09-22T21:04:35.879 DEBUG:boto:auth_path=/ffzdqu-6/ 2023-09-22T21:04:35.879 DEBUG:boto:Method: PUT 2023-09-22T21:04:35.879 DEBUG:boto:Path: /ffzdqu-6/ 2023-09-22T21:04:35.879 DEBUG:boto:Data: 2023-09-22T21:04:35.879 DEBUG:boto:Headers: {} 2023-09-22T21:04:35.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:35.880 DEBUG:boto:Port: 8001 2023-09-22T21:04:35.880 DEBUG:boto:Params: {} 2023-09-22T21:04:35.880 DEBUG:boto:Token: None 2023-09-22T21:04:35.880 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:04:35 GMT /ffzdqu-6/ 2023-09-22T21:04:35.880 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:43CQkwBC8dLwVx+WrA/Cn0AdisA= 2023-09-22T21:04:35.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:43CQkwBC8dLwVx+WrA/Cn0AdisA=', 'Content-Length': '0'} 2023-09-22T21:04:35.908 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000af2c4b7e4d724ef0-00650e0163-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:35.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:35.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:35.909 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:35.942 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:35.942 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:35.981 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:04:36.000 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695416675.900138_71.1","last_update":"2023-09-22T21:04:35.900138Z"}] 2023-09-22T21:04:36.000 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '', 3: '1_1695416675.900138_71.1'} 2023-09-22T21:04:36.000 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:04:36.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:36.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:36.001 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:36.078 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:36.079 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:36.113 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:36.113+0000 7fde1ef33a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:36.113 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:36.113+0000 7fde1ef33a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:36.129 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416675.900138_71.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:35.900138Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:36Z"} 2023-09-22T21:04:36.129 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416675.900138_71.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:35.900138Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:36Z"} 2023-09-22T21:04:36.129 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416675.900138_71.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:04:35.900138Z', 'realm_epoch': 2}}] 2023-09-22T21:04:36.130 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '', 3: '1_1695416675.900138_71.1'} 2023-09-22T21:04:36.130 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '', 3: '1_1695416675.900138_71.1'} 2023-09-22T21:04:36.130 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:04:36.130 DEBUG:boto:path=/ffzdqu-5/ 2023-09-22T21:04:36.130 DEBUG:boto:auth_path=/ffzdqu-5/ 2023-09-22T21:04:36.130 DEBUG:boto:Method: HEAD 2023-09-22T21:04:36.130 DEBUG:boto:Path: /ffzdqu-5/ 2023-09-22T21:04:36.131 DEBUG:boto:Data: 2023-09-22T21:04:36.131 DEBUG:boto:Headers: {} 2023-09-22T21:04:36.131 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:36.131 DEBUG:boto:Port: 8000 2023-09-22T21:04:36.131 DEBUG:boto:Params: {} 2023-09-22T21:04:36.131 DEBUG:boto:Token: None 2023-09-22T21:04:36.131 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-5/ 2023-09-22T21:04:36.131 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/p0DtriFTA1RZ6N017IgV3zv0oA= 2023-09-22T21:04:36.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/p0DtriFTA1RZ6N017IgV3zv0oA=', 'Content-Length': '0'} 2023-09-22T21:04:36.134 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000797cd155fd35081-00650e0164-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.134 DEBUG:boto:path=/ffzdqu-6/ 2023-09-22T21:04:36.134 DEBUG:boto:auth_path=/ffzdqu-6/ 2023-09-22T21:04:36.134 DEBUG:boto:Method: HEAD 2023-09-22T21:04:36.134 DEBUG:boto:Path: /ffzdqu-6/ 2023-09-22T21:04:36.134 DEBUG:boto:Data: 2023-09-22T21:04:36.134 DEBUG:boto:Headers: {} 2023-09-22T21:04:36.134 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:36.135 DEBUG:boto:Port: 8000 2023-09-22T21:04:36.135 DEBUG:boto:Params: {} 2023-09-22T21:04:36.135 DEBUG:boto:Token: None 2023-09-22T21:04:36.135 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-6/ 2023-09-22T21:04:36.135 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZpEIuI4MjfmW1s1RdNqzPcAHpVs= 2023-09-22T21:04:36.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZpEIuI4MjfmW1s1RdNqzPcAHpVs=', 'Content-Length': '0'} 2023-09-22T21:04:36.137 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008beb92befe328346-00650e0164-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.137 DEBUG:boto:path=/ffzdqu-5/ 2023-09-22T21:04:36.137 DEBUG:boto:auth_path=/ffzdqu-5/ 2023-09-22T21:04:36.137 DEBUG:boto:Method: HEAD 2023-09-22T21:04:36.137 DEBUG:boto:Path: /ffzdqu-5/ 2023-09-22T21:04:36.137 DEBUG:boto:Data: 2023-09-22T21:04:36.137 DEBUG:boto:Headers: {} 2023-09-22T21:04:36.138 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:36.138 DEBUG:boto:Port: 8001 2023-09-22T21:04:36.138 DEBUG:boto:Params: {} 2023-09-22T21:04:36.138 DEBUG:boto:Token: None 2023-09-22T21:04:36.138 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-5/ 2023-09-22T21:04:36.138 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/p0DtriFTA1RZ6N017IgV3zv0oA= 2023-09-22T21:04:36.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/p0DtriFTA1RZ6N017IgV3zv0oA=', 'Content-Length': '0'} 2023-09-22T21:04:36.140 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073bd92c276e7b2af-00650e0164-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.140 DEBUG:boto:path=/ffzdqu-6/ 2023-09-22T21:04:36.141 DEBUG:boto:auth_path=/ffzdqu-6/ 2023-09-22T21:04:36.141 DEBUG:boto:Method: HEAD 2023-09-22T21:04:36.141 DEBUG:boto:Path: /ffzdqu-6/ 2023-09-22T21:04:36.141 DEBUG:boto:Data: 2023-09-22T21:04:36.141 DEBUG:boto:Headers: {} 2023-09-22T21:04:36.141 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:36.141 DEBUG:boto:Port: 8001 2023-09-22T21:04:36.141 DEBUG:boto:Params: {} 2023-09-22T21:04:36.142 DEBUG:boto:Token: None 2023-09-22T21:04:36.142 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-6/ 2023-09-22T21:04:36.142 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZpEIuI4MjfmW1s1RdNqzPcAHpVs= 2023-09-22T21:04:36.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZpEIuI4MjfmW1s1RdNqzPcAHpVs=', 'Content-Length': '0'} 2023-09-22T21:04:36.144 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007dd261b086cf57d2-00650e0164-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.144 DEBUG:boto:path=/ffzdqu-5/ 2023-09-22T21:04:36.144 DEBUG:boto:auth_path=/ffzdqu-5/ 2023-09-22T21:04:36.144 DEBUG:boto:Method: DELETE 2023-09-22T21:04:36.144 DEBUG:boto:Path: /ffzdqu-5/ 2023-09-22T21:04:36.144 DEBUG:boto:Data: 2023-09-22T21:04:36.144 DEBUG:boto:Headers: {} 2023-09-22T21:04:36.145 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:36.145 DEBUG:boto:Port: 8000 2023-09-22T21:04:36.145 DEBUG:boto:Params: {} 2023-09-22T21:04:36.145 DEBUG:boto:Token: None 2023-09-22T21:04:36.145 DEBUG:boto:StringToSign: DELETE Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-5/ 2023-09-22T21:04:36.145 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SOPLosfvdlGcS78DgZ4KgU/G9K4= 2023-09-22T21:04:36.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SOPLosfvdlGcS78DgZ4KgU/G9K4=', 'Content-Length': '0'} 2023-09-22T21:04:36.175 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b8dab4e7d9cdcddd-00650e0164-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.176 DEBUG:boto:path=/ffzdqu-6/ 2023-09-22T21:04:36.176 DEBUG:boto:auth_path=/ffzdqu-6/ 2023-09-22T21:04:36.176 DEBUG:boto:Method: DELETE 2023-09-22T21:04:36.176 DEBUG:boto:Path: /ffzdqu-6/ 2023-09-22T21:04:36.176 DEBUG:boto:Data: 2023-09-22T21:04:36.176 DEBUG:boto:Headers: {} 2023-09-22T21:04:36.176 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:36.176 DEBUG:boto:Port: 8001 2023-09-22T21:04:36.176 DEBUG:boto:Params: {} 2023-09-22T21:04:36.177 DEBUG:boto:Token: None 2023-09-22T21:04:36.177 DEBUG:boto:StringToSign: DELETE Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-6/ 2023-09-22T21:04:36.177 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IbpftPRXxIqoSEBEPpgzAFLnCBI= 2023-09-22T21:04:36.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IbpftPRXxIqoSEBEPpgzAFLnCBI=', 'Content-Length': '0'} 2023-09-22T21:04:36.251 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e52a65026ff5376c-00650e0164-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:36.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:36.252 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:36.285 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:36.285 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:36.316 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:04:36.331 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695416676.216721_79.1","last_update":"2023-09-22T21:04:36.216721Z"}] 2023-09-22T21:04:36.331 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '', 3: '1_1695416676.216721_79.1'} 2023-09-22T21:04:36.331 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:04:36.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:36.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:36.332 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:36.406 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:36.406 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:36.449 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:36.449+0000 7fd9d1975a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:36.449 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:36.449+0000 7fd9d1975a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:36.464 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416676.216721_79.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:36.216721Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:36Z"} 2023-09-22T21:04:36.464 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416676.216721_79.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:36.216721Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:36Z"} 2023-09-22T21:04:36.464 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416676.216721_79.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:04:36.216721Z', 'realm_epoch': 2}}] 2023-09-22T21:04:36.464 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '', 3: '1_1695416676.216721_79.1'} 2023-09-22T21:04:36.464 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '', 3: '1_1695416676.216721_79.1'} 2023-09-22T21:04:36.464 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:04:36.465 DEBUG:boto:path=/ffzdqu-5/ 2023-09-22T21:04:36.465 DEBUG:boto:auth_path=/ffzdqu-5/ 2023-09-22T21:04:36.465 DEBUG:boto:Method: HEAD 2023-09-22T21:04:36.465 DEBUG:boto:Path: /ffzdqu-5/ 2023-09-22T21:04:36.465 DEBUG:boto:Data: 2023-09-22T21:04:36.465 DEBUG:boto:Headers: {} 2023-09-22T21:04:36.465 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:36.465 DEBUG:boto:Port: 8000 2023-09-22T21:04:36.465 DEBUG:boto:Params: {} 2023-09-22T21:04:36.466 DEBUG:boto:Token: None 2023-09-22T21:04:36.466 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-5/ 2023-09-22T21:04:36.466 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/p0DtriFTA1RZ6N017IgV3zv0oA= 2023-09-22T21:04:36.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/p0DtriFTA1RZ6N017IgV3zv0oA=', 'Content-Length': '0'} 2023-09-22T21:04:36.467 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx0000097454aefaf62e015-00650e0164-111c-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.468 DEBUG:boto:path=/ffzdqu-6/ 2023-09-22T21:04:36.468 DEBUG:boto:auth_path=/ffzdqu-6/ 2023-09-22T21:04:36.468 DEBUG:boto:Method: HEAD 2023-09-22T21:04:36.468 DEBUG:boto:Path: /ffzdqu-6/ 2023-09-22T21:04:36.468 DEBUG:boto:Data: 2023-09-22T21:04:36.468 DEBUG:boto:Headers: {} 2023-09-22T21:04:36.469 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:36.469 DEBUG:boto:Port: 8000 2023-09-22T21:04:36.469 DEBUG:boto:Params: {} 2023-09-22T21:04:36.469 DEBUG:boto:Token: None 2023-09-22T21:04:36.469 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-6/ 2023-09-22T21:04:36.469 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZpEIuI4MjfmW1s1RdNqzPcAHpVs= 2023-09-22T21:04:36.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZpEIuI4MjfmW1s1RdNqzPcAHpVs=', 'Content-Length': '0'} 2023-09-22T21:04:36.470 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000996476a3bfc2a04a-00650e0164-111c-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.470 DEBUG:boto:path=/ffzdqu-5/ 2023-09-22T21:04:36.470 DEBUG:boto:auth_path=/ffzdqu-5/ 2023-09-22T21:04:36.471 DEBUG:boto:Method: HEAD 2023-09-22T21:04:36.471 DEBUG:boto:Path: /ffzdqu-5/ 2023-09-22T21:04:36.471 DEBUG:boto:Data: 2023-09-22T21:04:36.471 DEBUG:boto:Headers: {} 2023-09-22T21:04:36.471 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:36.471 DEBUG:boto:Port: 8001 2023-09-22T21:04:36.471 DEBUG:boto:Params: {} 2023-09-22T21:04:36.471 DEBUG:boto:Token: None 2023-09-22T21:04:36.471 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-5/ 2023-09-22T21:04:36.472 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/p0DtriFTA1RZ6N017IgV3zv0oA= 2023-09-22T21:04:36.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/p0DtriFTA1RZ6N017IgV3zv0oA=', 'Content-Length': '0'} 2023-09-22T21:04:36.473 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx000004448f79793cb64ee-00650e0164-1120-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.473 DEBUG:boto:path=/ffzdqu-6/ 2023-09-22T21:04:36.473 DEBUG:boto:auth_path=/ffzdqu-6/ 2023-09-22T21:04:36.473 DEBUG:boto:Method: HEAD 2023-09-22T21:04:36.473 DEBUG:boto:Path: /ffzdqu-6/ 2023-09-22T21:04:36.474 DEBUG:boto:Data: 2023-09-22T21:04:36.474 DEBUG:boto:Headers: {} 2023-09-22T21:04:36.474 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:36.474 DEBUG:boto:Port: 8001 2023-09-22T21:04:36.474 DEBUG:boto:Params: {} 2023-09-22T21:04:36.474 DEBUG:boto:Token: None 2023-09-22T21:04:36.474 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-6/ 2023-09-22T21:04:36.474 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZpEIuI4MjfmW1s1RdNqzPcAHpVs= 2023-09-22T21:04:36.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZpEIuI4MjfmW1s1RdNqzPcAHpVs=', 'Content-Length': '0'} 2023-09-22T21:04:36.475 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000801290d1857b5ff8-00650e0164-1120-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2023-09-22T21:04:36.476 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-7 2023-09-22T21:04:36.476 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:36.476 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:36.477 DEBUG:boto:Method: PUT 2023-09-22T21:04:36.477 DEBUG:boto:Path: /ffzdqu-7/ 2023-09-22T21:04:36.477 DEBUG:boto:Data: 2023-09-22T21:04:36.477 DEBUG:boto:Headers: {} 2023-09-22T21:04:36.477 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:36.477 DEBUG:boto:Port: 8000 2023-09-22T21:04:36.477 DEBUG:boto:Params: {} 2023-09-22T21:04:36.477 DEBUG:boto:Token: None 2023-09-22T21:04:36.477 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-7/ 2023-09-22T21:04:36.478 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GEtFaGD8WeSLYe5reiz5EfxIkJo= 2023-09-22T21:04:36.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GEtFaGD8WeSLYe5reiz5EfxIkJo=', 'Content-Length': '0'} 2023-09-22T21:04:36.492 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008f4bddd93e768a40-00650e0164-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.493 INFO:rgw_multi.tests:create bucket zone=a2 name=ffzdqu-8 2023-09-22T21:04:36.493 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:04:36.493 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:04:36.493 DEBUG:boto:Method: PUT 2023-09-22T21:04:36.493 DEBUG:boto:Path: /ffzdqu-8/ 2023-09-22T21:04:36.493 DEBUG:boto:Data: 2023-09-22T21:04:36.493 DEBUG:boto:Headers: {} 2023-09-22T21:04:36.493 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:36.494 DEBUG:boto:Port: 8001 2023-09-22T21:04:36.494 DEBUG:boto:Params: {} 2023-09-22T21:04:36.494 DEBUG:boto:Token: None 2023-09-22T21:04:36.494 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-8/ 2023-09-22T21:04:36.494 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VSsixDfyPfOAMjOB7vr7YNIgVd0= 2023-09-22T21:04:36.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VSsixDfyPfOAMjOB7vr7YNIgVd0=', 'Content-Length': '0'} 2023-09-22T21:04:36.530 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005a26d2e96d27d710-00650e0164-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.530 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:36.530 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:36.530 DEBUG:boto:Method: HEAD 2023-09-22T21:04:36.530 DEBUG:boto:Path: /ffzdqu-7/ 2023-09-22T21:04:36.530 DEBUG:boto:Data: 2023-09-22T21:04:36.531 DEBUG:boto:Headers: {} 2023-09-22T21:04:36.531 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:36.531 DEBUG:boto:Port: 8000 2023-09-22T21:04:36.531 DEBUG:boto:Params: {} 2023-09-22T21:04:36.531 DEBUG:boto:Token: None 2023-09-22T21:04:36.531 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-7/ 2023-09-22T21:04:36.531 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pYbY54IPoC09wwGM93XoL0V1bVI= 2023-09-22T21:04:36.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pYbY54IPoC09wwGM93XoL0V1bVI=', 'Content-Length': '0'} 2023-09-22T21:04:36.534 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce800be5b71d4524-00650e0164-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.534 DEBUG:boto:path=/ffzdqu-7/myobj 2023-09-22T21:04:36.535 DEBUG:boto:auth_path=/ffzdqu-7/myobj 2023-09-22T21:04:36.535 DEBUG:boto:Method: PUT 2023-09-22T21:04:36.535 DEBUG:boto:Path: /ffzdqu-7/myobj 2023-09-22T21:04:36.535 DEBUG:boto:Data: 2023-09-22T21:04:36.535 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:04:36.535 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:36.535 DEBUG:boto:Port: 8000 2023-09-22T21:04:36.535 DEBUG:boto:Params: {} 2023-09-22T21:04:36.535 DEBUG:boto:Token: None 2023-09-22T21:04:36.536 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-7/myobj 2023-09-22T21:04:36.536 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yMBXj69mDAWj9DpG3QtbeS/d6hI= 2023-09-22T21:04:36.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yMBXj69mDAWj9DpG3QtbeS/d6hI='} 2023-09-22T21:04:36.550 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e76c1566b43562a4-00650e0164-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.550 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:36.550 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:36.550 DEBUG:boto:Method: HEAD 2023-09-22T21:04:36.550 DEBUG:boto:Path: /ffzdqu-7/ 2023-09-22T21:04:36.550 DEBUG:boto:Data: 2023-09-22T21:04:36.551 DEBUG:boto:Headers: {} 2023-09-22T21:04:36.551 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:36.551 DEBUG:boto:Port: 8000 2023-09-22T21:04:36.551 DEBUG:boto:Params: {} 2023-09-22T21:04:36.551 DEBUG:boto:Token: None 2023-09-22T21:04:36.551 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-7/ 2023-09-22T21:04:36.551 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pYbY54IPoC09wwGM93XoL0V1bVI= 2023-09-22T21:04:36.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pYbY54IPoC09wwGM93XoL0V1bVI=', 'Content-Length': '0'} 2023-09-22T21:04:36.554 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000740437b7b1bfd49-00650e0164-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.554 DEBUG:boto:path=/ffzdqu-7/_myobj 2023-09-22T21:04:36.554 DEBUG:boto:auth_path=/ffzdqu-7/_myobj 2023-09-22T21:04:36.554 DEBUG:boto:Method: PUT 2023-09-22T21:04:36.554 DEBUG:boto:Path: /ffzdqu-7/_myobj 2023-09-22T21:04:36.555 DEBUG:boto:Data: 2023-09-22T21:04:36.555 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:04:36.555 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:36.555 DEBUG:boto:Port: 8000 2023-09-22T21:04:36.555 DEBUG:boto:Params: {} 2023-09-22T21:04:36.555 DEBUG:boto:Token: None 2023-09-22T21:04:36.555 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-7/_myobj 2023-09-22T21:04:36.555 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9jvJAS6agMrVm9R2JhEfHeZCJeM= 2023-09-22T21:04:36.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9jvJAS6agMrVm9R2JhEfHeZCJeM='} 2023-09-22T21:04:36.598 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c6648149c73957bb-00650e0164-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.599 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:36.599 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:36.599 DEBUG:boto:Method: HEAD 2023-09-22T21:04:36.599 DEBUG:boto:Path: /ffzdqu-7/ 2023-09-22T21:04:36.599 DEBUG:boto:Data: 2023-09-22T21:04:36.599 DEBUG:boto:Headers: {} 2023-09-22T21:04:36.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:36.599 DEBUG:boto:Port: 8000 2023-09-22T21:04:36.600 DEBUG:boto:Params: {} 2023-09-22T21:04:36.600 DEBUG:boto:Token: None 2023-09-22T21:04:36.600 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-7/ 2023-09-22T21:04:36.600 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pYbY54IPoC09wwGM93XoL0V1bVI= 2023-09-22T21:04:36.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pYbY54IPoC09wwGM93XoL0V1bVI=', 'Content-Length': '0'} 2023-09-22T21:04:36.603 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000314d08bd2bbdf37f-00650e0164-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.603 DEBUG:boto:path=/ffzdqu-7/%3A 2023-09-22T21:04:36.604 DEBUG:boto:auth_path=/ffzdqu-7/%3A 2023-09-22T21:04:36.604 DEBUG:boto:Method: PUT 2023-09-22T21:04:36.604 DEBUG:boto:Path: /ffzdqu-7/%3A 2023-09-22T21:04:36.604 DEBUG:boto:Data: 2023-09-22T21:04:36.604 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:04:36.604 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:36.604 DEBUG:boto:Port: 8000 2023-09-22T21:04:36.604 DEBUG:boto:Params: {} 2023-09-22T21:04:36.604 DEBUG:boto:Token: None 2023-09-22T21:04:36.605 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-7/%3A 2023-09-22T21:04:36.605 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wK6dnphFYB2unzWw+3VPjWXfMk0= 2023-09-22T21:04:36.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wK6dnphFYB2unzWw+3VPjWXfMk0='} 2023-09-22T21:04:36.621 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000452a8c78c42f91df-00650e0164-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.622 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:36.622 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:36.622 DEBUG:boto:Method: HEAD 2023-09-22T21:04:36.622 DEBUG:boto:Path: /ffzdqu-7/ 2023-09-22T21:04:36.622 DEBUG:boto:Data: 2023-09-22T21:04:36.622 DEBUG:boto:Headers: {} 2023-09-22T21:04:36.622 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:36.622 DEBUG:boto:Port: 8000 2023-09-22T21:04:36.622 DEBUG:boto:Params: {} 2023-09-22T21:04:36.623 DEBUG:boto:Token: None 2023-09-22T21:04:36.623 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-7/ 2023-09-22T21:04:36.623 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pYbY54IPoC09wwGM93XoL0V1bVI= 2023-09-22T21:04:36.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pYbY54IPoC09wwGM93XoL0V1bVI=', 'Content-Length': '0'} 2023-09-22T21:04:36.627 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df24a617c199c050-00650e0164-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.627 DEBUG:boto:path=/ffzdqu-7/%26 2023-09-22T21:04:36.627 DEBUG:boto:auth_path=/ffzdqu-7/%26 2023-09-22T21:04:36.627 DEBUG:boto:Method: PUT 2023-09-22T21:04:36.627 DEBUG:boto:Path: /ffzdqu-7/%26 2023-09-22T21:04:36.628 DEBUG:boto:Data: 2023-09-22T21:04:36.628 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:04:36.628 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:36.628 DEBUG:boto:Port: 8000 2023-09-22T21:04:36.628 DEBUG:boto:Params: {} 2023-09-22T21:04:36.628 DEBUG:boto:Token: None 2023-09-22T21:04:36.628 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-7/%26 2023-09-22T21:04:36.628 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vIaLDpIYM1GVvbS4HQAd0l9zY4o= 2023-09-22T21:04:36.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vIaLDpIYM1GVvbS4HQAd0l9zY4o='} 2023-09-22T21:04:36.670 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000821f8f5d437e58e5-00650e0164-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.671 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:04:36.671 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:04:36.671 DEBUG:boto:Method: HEAD 2023-09-22T21:04:36.671 DEBUG:boto:Path: /ffzdqu-8/ 2023-09-22T21:04:36.671 DEBUG:boto:Data: 2023-09-22T21:04:36.671 DEBUG:boto:Headers: {} 2023-09-22T21:04:36.671 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:36.672 DEBUG:boto:Port: 8001 2023-09-22T21:04:36.672 DEBUG:boto:Params: {} 2023-09-22T21:04:36.672 DEBUG:boto:Token: None 2023-09-22T21:04:36.672 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-8/ 2023-09-22T21:04:36.672 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lqlyJgV8Wr/B99fyVrqrJHpvs90= 2023-09-22T21:04:36.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lqlyJgV8Wr/B99fyVrqrJHpvs90=', 'Content-Length': '0'} 2023-09-22T21:04:36.675 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df82f4a01f0c4da6-00650e0164-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.676 DEBUG:boto:path=/ffzdqu-8/myobj 2023-09-22T21:04:36.676 DEBUG:boto:auth_path=/ffzdqu-8/myobj 2023-09-22T21:04:36.676 DEBUG:boto:Method: PUT 2023-09-22T21:04:36.676 DEBUG:boto:Path: /ffzdqu-8/myobj 2023-09-22T21:04:36.676 DEBUG:boto:Data: 2023-09-22T21:04:36.676 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:04:36.676 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:36.676 DEBUG:boto:Port: 8001 2023-09-22T21:04:36.677 DEBUG:boto:Params: {} 2023-09-22T21:04:36.677 DEBUG:boto:Token: None 2023-09-22T21:04:36.677 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-8/myobj 2023-09-22T21:04:36.677 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hsa0FGsuXaiZSfgfmCAbTba+WZM= 2023-09-22T21:04:36.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hsa0FGsuXaiZSfgfmCAbTba+WZM='} 2023-09-22T21:04:36.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e49c5cbc18aabef9-00650e0164-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.694 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:04:36.695 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:04:36.695 DEBUG:boto:Method: HEAD 2023-09-22T21:04:36.695 DEBUG:boto:Path: /ffzdqu-8/ 2023-09-22T21:04:36.695 DEBUG:boto:Data: 2023-09-22T21:04:36.695 DEBUG:boto:Headers: {} 2023-09-22T21:04:36.695 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:36.695 DEBUG:boto:Port: 8001 2023-09-22T21:04:36.695 DEBUG:boto:Params: {} 2023-09-22T21:04:36.697 DEBUG:boto:Token: None 2023-09-22T21:04:36.698 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-8/ 2023-09-22T21:04:36.698 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lqlyJgV8Wr/B99fyVrqrJHpvs90= 2023-09-22T21:04:36.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lqlyJgV8Wr/B99fyVrqrJHpvs90=', 'Content-Length': '0'} 2023-09-22T21:04:36.701 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000edd2f74cb38b67b1-00650e0164-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.702 DEBUG:boto:path=/ffzdqu-8/_myobj 2023-09-22T21:04:36.702 DEBUG:boto:auth_path=/ffzdqu-8/_myobj 2023-09-22T21:04:36.702 DEBUG:boto:Method: PUT 2023-09-22T21:04:36.702 DEBUG:boto:Path: /ffzdqu-8/_myobj 2023-09-22T21:04:36.702 DEBUG:boto:Data: 2023-09-22T21:04:36.702 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:04:36.703 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:36.703 DEBUG:boto:Port: 8001 2023-09-22T21:04:36.703 DEBUG:boto:Params: {} 2023-09-22T21:04:36.703 DEBUG:boto:Token: None 2023-09-22T21:04:36.703 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-8/_myobj 2023-09-22T21:04:36.703 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kAVmLHpLUQX0vB6SMCIqbPlg0A4= 2023-09-22T21:04:36.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kAVmLHpLUQX0vB6SMCIqbPlg0A4='} 2023-09-22T21:04:36.724 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000064a9b0cfee8c0dc-00650e0164-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.724 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:04:36.724 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:04:36.724 DEBUG:boto:Method: HEAD 2023-09-22T21:04:36.724 DEBUG:boto:Path: /ffzdqu-8/ 2023-09-22T21:04:36.725 DEBUG:boto:Data: 2023-09-22T21:04:36.725 DEBUG:boto:Headers: {} 2023-09-22T21:04:36.725 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:36.725 DEBUG:boto:Port: 8001 2023-09-22T21:04:36.725 DEBUG:boto:Params: {} 2023-09-22T21:04:36.725 DEBUG:boto:Token: None 2023-09-22T21:04:36.725 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-8/ 2023-09-22T21:04:36.725 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lqlyJgV8Wr/B99fyVrqrJHpvs90= 2023-09-22T21:04:36.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lqlyJgV8Wr/B99fyVrqrJHpvs90=', 'Content-Length': '0'} 2023-09-22T21:04:36.729 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000acfda31acbca73ca-00650e0164-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.730 DEBUG:boto:path=/ffzdqu-8/%3A 2023-09-22T21:04:36.730 DEBUG:boto:auth_path=/ffzdqu-8/%3A 2023-09-22T21:04:36.730 DEBUG:boto:Method: PUT 2023-09-22T21:04:36.730 DEBUG:boto:Path: /ffzdqu-8/%3A 2023-09-22T21:04:36.730 DEBUG:boto:Data: 2023-09-22T21:04:36.730 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:04:36.730 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:36.730 DEBUG:boto:Port: 8001 2023-09-22T21:04:36.730 DEBUG:boto:Params: {} 2023-09-22T21:04:36.731 DEBUG:boto:Token: None 2023-09-22T21:04:36.731 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-8/%3A 2023-09-22T21:04:36.731 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZHrQmHmcYPeHi5ik/tahrmUHprg= 2023-09-22T21:04:36.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZHrQmHmcYPeHi5ik/tahrmUHprg='} 2023-09-22T21:04:36.774 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e930feed7abb229-00650e0164-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.775 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:04:36.775 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:04:36.775 DEBUG:boto:Method: HEAD 2023-09-22T21:04:36.775 DEBUG:boto:Path: /ffzdqu-8/ 2023-09-22T21:04:36.775 DEBUG:boto:Data: 2023-09-22T21:04:36.775 DEBUG:boto:Headers: {} 2023-09-22T21:04:36.775 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:36.776 DEBUG:boto:Port: 8001 2023-09-22T21:04:36.776 DEBUG:boto:Params: {} 2023-09-22T21:04:36.776 DEBUG:boto:Token: None 2023-09-22T21:04:36.776 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-8/ 2023-09-22T21:04:36.776 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lqlyJgV8Wr/B99fyVrqrJHpvs90= 2023-09-22T21:04:36.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lqlyJgV8Wr/B99fyVrqrJHpvs90=', 'Content-Length': '0'} 2023-09-22T21:04:36.780 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff1ec1ea7c22e51b-00650e0164-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.781 DEBUG:boto:path=/ffzdqu-8/%26 2023-09-22T21:04:36.781 DEBUG:boto:auth_path=/ffzdqu-8/%26 2023-09-22T21:04:36.781 DEBUG:boto:Method: PUT 2023-09-22T21:04:36.781 DEBUG:boto:Path: /ffzdqu-8/%26 2023-09-22T21:04:36.781 DEBUG:boto:Data: 2023-09-22T21:04:36.781 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:04:36.781 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:36.781 DEBUG:boto:Port: 8001 2023-09-22T21:04:36.781 DEBUG:boto:Params: {} 2023-09-22T21:04:36.782 DEBUG:boto:Token: None 2023-09-22T21:04:36.782 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:04:36 GMT /ffzdqu-8/%26 2023-09-22T21:04:36.782 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pLsbK389CUz5DMYEWLRXrOholOg= 2023-09-22T21:04:36.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:04:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pLsbK389CUz5DMYEWLRXrOholOg='} 2023-09-22T21:04:36.794 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a91c50aadf798d15-00650e0164-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:36.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:36.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:36.795 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:36.836 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:36.836 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:36.874 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:04:36.892 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695416676.514203_87.1","last_update":"2023-09-22T21:04:36.514203Z"}] 2023-09-22T21:04:36.892 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '', 3: '1_1695416676.514203_87.1'} 2023-09-22T21:04:36.892 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:04:36.892 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:36.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:04:36.893 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:04:36.969 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:36.969 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:37.004 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:37.005+0000 7f34f0d47a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:37.005 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:37.005+0000 7f34f0d47a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:04:37.022 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416676.514203_87.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:36.514203Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:37Z"} 2023-09-22T21:04:37.022 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416676.514203_87.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:36.514203Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:04:37Z"} 2023-09-22T21:04:37.022 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416676.514203_87.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:04:36.514203Z', 'realm_epoch': 2}}] 2023-09-22T21:04:37.022 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '', 3: '1_1695416676.514203_87.1'} 2023-09-22T21:04:37.023 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '', 3: '1_1695416676.514203_87.1'} 2023-09-22T21:04:37.023 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:04:37.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:04:37.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:04:37.023 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-7 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:04:37.109 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:37.109 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:37.150 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:37.153+0000 7f53c3e66a40 1 waiting to reach incremental sync.. 2023-09-22T21:04:39.434 INFO:tasks.rgw.c1.client.0.smithi160.stdout:2023-09-22T21:04:39.433+0000 7fa9c1675640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:04:39.463 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:04:39.465+0000 7ff4815cb640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:04:42.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:42.153+0000 7f53c3e66a40 1 bucket sync caught up with source: 2023-09-22T21:04:42.153 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.2.6, , 00000000001.5.6, , , , 00000000001.7.6, 00000000001.5.6, , ] 2023-09-22T21:04:42.153 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.2.6, , 00000000001.5.6, , , , 00000000001.7.6, 00000000001.5.6, , ] 2023-09-22T21:04:42.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:42.153+0000 7f53c3e66a40 0 bucket checkpoint complete 2023-09-22T21:04:42.186 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-7 zones={a2, a1} 2023-09-22T21:04:42.187 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:42.187 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:42.187 DEBUG:boto:Method: HEAD 2023-09-22T21:04:42.187 DEBUG:boto:Path: /ffzdqu-7/ 2023-09-22T21:04:42.187 DEBUG:boto:Data: 2023-09-22T21:04:42.187 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.188 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.188 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.188 DEBUG:boto:Params: {} 2023-09-22T21:04:42.188 DEBUG:boto:Token: None 2023-09-22T21:04:42.188 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/ 2023-09-22T21:04:42.188 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dvfX+9+UX8OlpWXsPZike4zuk+s= 2023-09-22T21:04:42.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dvfX+9+UX8OlpWXsPZike4zuk+s=', 'Content-Length': '0'} 2023-09-22T21:04:42.192 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000874d88cdc96d0a3a-00650e016a-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.192 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:42.193 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:42.193 DEBUG:boto:Method: HEAD 2023-09-22T21:04:42.193 DEBUG:boto:Path: /ffzdqu-7/ 2023-09-22T21:04:42.193 DEBUG:boto:Data: 2023-09-22T21:04:42.193 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.193 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.193 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.193 DEBUG:boto:Params: {} 2023-09-22T21:04:42.193 DEBUG:boto:Token: None 2023-09-22T21:04:42.194 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/ 2023-09-22T21:04:42.194 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dvfX+9+UX8OlpWXsPZike4zuk+s= 2023-09-22T21:04:42.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dvfX+9+UX8OlpWXsPZike4zuk+s=', 'Content-Length': '0'} 2023-09-22T21:04:42.197 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc99ac8de7bbfdd0-00650e016a-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.197 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:04:42.197 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:42.197 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:42.197 DEBUG:boto:path=/ffzdqu-7/?versions&max-keys=999 2023-09-22T21:04:42.197 DEBUG:boto:auth_path=/ffzdqu-7/?versions&max-keys=999 2023-09-22T21:04:42.197 DEBUG:boto:Method: GET 2023-09-22T21:04:42.198 DEBUG:boto:Path: /ffzdqu-7/?versions&max-keys=999 2023-09-22T21:04:42.198 DEBUG:boto:Data: 2023-09-22T21:04:42.198 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.198 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.198 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.198 DEBUG:boto:Params: {} 2023-09-22T21:04:42.198 DEBUG:boto:Token: None 2023-09-22T21:04:42.198 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/?versions 2023-09-22T21:04:42.199 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/pznQx8OWXfPxMJ+s+8BCrJyGdE= 2023-09-22T21:04:42.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/pznQx8OWXfPxMJ+s+8BCrJyGdE=', 'Content-Length': '0'} 2023-09-22T21:04:42.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f727627298c6dc3f-00650e016a-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.202 DEBUG:boto:b'ffzdqu-7999false&nulltrue2023-09-22T21:04:36.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-22T21:04:36.610Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-22T21:04:36.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-22T21:04:36.542Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:04:42.243 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-09-22T21:04:42.243 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-09-22T21:04:42.243 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-09-22T21:04:42.243 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-22T21:04:42.243 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:04:42.243 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:42.243 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:42.244 DEBUG:boto:path=/ffzdqu-7/?versions&max-keys=999 2023-09-22T21:04:42.244 DEBUG:boto:auth_path=/ffzdqu-7/?versions&max-keys=999 2023-09-22T21:04:42.244 DEBUG:boto:Method: GET 2023-09-22T21:04:42.244 DEBUG:boto:Path: /ffzdqu-7/?versions&max-keys=999 2023-09-22T21:04:42.244 DEBUG:boto:Data: 2023-09-22T21:04:42.244 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.244 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.244 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.244 DEBUG:boto:Params: {} 2023-09-22T21:04:42.245 DEBUG:boto:Token: None 2023-09-22T21:04:42.245 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/?versions 2023-09-22T21:04:42.245 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/pznQx8OWXfPxMJ+s+8BCrJyGdE= 2023-09-22T21:04:42.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/pznQx8OWXfPxMJ+s+8BCrJyGdE=', 'Content-Length': '0'} 2023-09-22T21:04:42.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6ddfc0f342a37de-00650e016a-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.249 DEBUG:boto:b'ffzdqu-7999false&nulltrue2023-09-22T21:04:36.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-22T21:04:36.610Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-22T21:04:36.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-22T21:04:36.542Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:04:42.250 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-09-22T21:04:42.250 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-09-22T21:04:42.250 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-09-22T21:04:42.250 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-22T21:04:42.250 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:42.250 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:42.250 DEBUG:boto:path=/ffzdqu-7/?versions&max-keys=999 2023-09-22T21:04:42.250 DEBUG:boto:auth_path=/ffzdqu-7/?versions&max-keys=999 2023-09-22T21:04:42.250 DEBUG:boto:Method: GET 2023-09-22T21:04:42.251 DEBUG:boto:Path: /ffzdqu-7/?versions&max-keys=999 2023-09-22T21:04:42.251 DEBUG:boto:Data: 2023-09-22T21:04:42.251 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.251 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.251 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.251 DEBUG:boto:Params: {} 2023-09-22T21:04:42.251 DEBUG:boto:Token: None 2023-09-22T21:04:42.251 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/?versions 2023-09-22T21:04:42.251 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/pznQx8OWXfPxMJ+s+8BCrJyGdE= 2023-09-22T21:04:42.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/pznQx8OWXfPxMJ+s+8BCrJyGdE=', 'Content-Length': '0'} 2023-09-22T21:04:42.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000126c00cb17a07039-00650e016a-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.256 DEBUG:boto:b'ffzdqu-7999false&nulltrue2023-09-22T21:04:36.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-22T21:04:36.610Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-22T21:04:36.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-22T21:04:36.542Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:04:42.256 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:42.256 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:42.256 DEBUG:boto:path=/ffzdqu-7/?versions&max-keys=999 2023-09-22T21:04:42.256 DEBUG:boto:auth_path=/ffzdqu-7/?versions&max-keys=999 2023-09-22T21:04:42.257 DEBUG:boto:Method: GET 2023-09-22T21:04:42.257 DEBUG:boto:Path: /ffzdqu-7/?versions&max-keys=999 2023-09-22T21:04:42.257 DEBUG:boto:Data: 2023-09-22T21:04:42.257 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.257 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.257 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.257 DEBUG:boto:Params: {} 2023-09-22T21:04:42.257 DEBUG:boto:Token: None 2023-09-22T21:04:42.257 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/?versions 2023-09-22T21:04:42.258 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/pznQx8OWXfPxMJ+s+8BCrJyGdE= 2023-09-22T21:04:42.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/pznQx8OWXfPxMJ+s+8BCrJyGdE=', 'Content-Length': '0'} 2023-09-22T21:04:42.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000472a6eb548ebeee9-00650e016a-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.263 DEBUG:boto:b'ffzdqu-7999false&nulltrue2023-09-22T21:04:36.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-22T21:04:36.610Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-22T21:04:36.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-22T21:04:36.542Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:04:42.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-22T21:04:42.263 DEBUG:boto:path=/ffzdqu-7/%26 2023-09-22T21:04:42.263 DEBUG:boto:auth_path=/ffzdqu-7/%26 2023-09-22T21:04:42.263 DEBUG:boto:path=/ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.264 DEBUG:boto:auth_path=/ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.264 DEBUG:boto:Method: GET 2023-09-22T21:04:42.264 DEBUG:boto:Path: /ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.264 DEBUG:boto:Data: 2023-09-22T21:04:42.264 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.264 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.264 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.264 DEBUG:boto:Params: {} 2023-09-22T21:04:42.264 DEBUG:boto:Token: None 2023-09-22T21:04:42.264 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.265 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SJZXk1CuZkVTApThBkQ57Hi2LHw= 2023-09-22T21:04:42.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SJZXk1CuZkVTApThBkQ57Hi2LHw=', 'Content-Length': '0'} 2023-09-22T21:04:42.268 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fe6a7bfc0841fdf-00650e016a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.268 DEBUG:boto:path=/ffzdqu-7/%26 2023-09-22T21:04:42.268 DEBUG:boto:auth_path=/ffzdqu-7/%26 2023-09-22T21:04:42.269 DEBUG:boto:path=/ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.269 DEBUG:boto:auth_path=/ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.269 DEBUG:boto:Method: GET 2023-09-22T21:04:42.269 DEBUG:boto:Path: /ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.269 DEBUG:boto:Data: 2023-09-22T21:04:42.269 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.269 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.269 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.269 DEBUG:boto:Params: {} 2023-09-22T21:04:42.270 DEBUG:boto:Token: None 2023-09-22T21:04:42.270 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.270 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SJZXk1CuZkVTApThBkQ57Hi2LHw= 2023-09-22T21:04:42.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SJZXk1CuZkVTApThBkQ57Hi2LHw=', 'Content-Length': '0'} 2023-09-22T21:04:42.272 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000687065633460aafc-00650e016a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.273 DEBUG:boto:path=/ffzdqu-7/%26 2023-09-22T21:04:42.273 DEBUG:boto:auth_path=/ffzdqu-7/%26 2023-09-22T21:04:42.273 DEBUG:boto:path=/ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.273 DEBUG:boto:auth_path=/ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.273 DEBUG:boto:Method: HEAD 2023-09-22T21:04:42.273 DEBUG:boto:Path: /ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.273 DEBUG:boto:Data: 2023-09-22T21:04:42.273 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.273 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.274 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.274 DEBUG:boto:Params: {} 2023-09-22T21:04:42.274 DEBUG:boto:Token: None 2023-09-22T21:04:42.274 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.274 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vG8EDIGBRykBPTHVIeK7EQmwPzo= 2023-09-22T21:04:42.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vG8EDIGBRykBPTHVIeK7EQmwPzo=', 'Content-Length': '0'} 2023-09-22T21:04:42.277 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069ba3d9f5d60234c-00650e016a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.277 DEBUG:boto:path=/ffzdqu-7/%26 2023-09-22T21:04:42.278 DEBUG:boto:auth_path=/ffzdqu-7/%26 2023-09-22T21:04:42.278 DEBUG:boto:path=/ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.278 DEBUG:boto:auth_path=/ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.278 DEBUG:boto:Method: HEAD 2023-09-22T21:04:42.278 DEBUG:boto:Path: /ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.278 DEBUG:boto:Data: 2023-09-22T21:04:42.279 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.279 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.279 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.279 DEBUG:boto:Params: {} 2023-09-22T21:04:42.279 DEBUG:boto:Token: None 2023-09-22T21:04:42.279 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.280 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vG8EDIGBRykBPTHVIeK7EQmwPzo= 2023-09-22T21:04:42.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vG8EDIGBRykBPTHVIeK7EQmwPzo=', 'Content-Length': '0'} 2023-09-22T21:04:42.282 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8e2250d8a585f6a-00650e016a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-22T21:04:42.283 DEBUG:boto:path=/ffzdqu-7/%26 2023-09-22T21:04:42.283 DEBUG:boto:auth_path=/ffzdqu-7/%26 2023-09-22T21:04:42.283 DEBUG:boto:path=/ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.283 DEBUG:boto:auth_path=/ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.283 DEBUG:boto:Method: GET 2023-09-22T21:04:42.284 DEBUG:boto:Path: /ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.284 DEBUG:boto:Data: 2023-09-22T21:04:42.284 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.284 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.284 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.284 DEBUG:boto:Params: {} 2023-09-22T21:04:42.284 DEBUG:boto:Token: None 2023-09-22T21:04:42.285 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.285 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SJZXk1CuZkVTApThBkQ57Hi2LHw= 2023-09-22T21:04:42.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SJZXk1CuZkVTApThBkQ57Hi2LHw=', 'Content-Length': '0'} 2023-09-22T21:04:42.287 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005966dbaa7a252115-00650e016a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.288 DEBUG:boto:path=/ffzdqu-7/%26 2023-09-22T21:04:42.288 DEBUG:boto:auth_path=/ffzdqu-7/%26 2023-09-22T21:04:42.288 DEBUG:boto:path=/ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.288 DEBUG:boto:auth_path=/ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.288 DEBUG:boto:Method: GET 2023-09-22T21:04:42.288 DEBUG:boto:Path: /ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.288 DEBUG:boto:Data: 2023-09-22T21:04:42.288 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.289 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.289 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.289 DEBUG:boto:Params: {} 2023-09-22T21:04:42.289 DEBUG:boto:Token: None 2023-09-22T21:04:42.289 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/%26?versionId=null 2023-09-22T21:04:42.289 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SJZXk1CuZkVTApThBkQ57Hi2LHw= 2023-09-22T21:04:42.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SJZXk1CuZkVTApThBkQ57Hi2LHw=', 'Content-Length': '0'} 2023-09-22T21:04:42.292 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080b3b107a78a5239-00650e016a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.334 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:42.335 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:42.335 DEBUG:boto:path=/ffzdqu-7/?prefix=%26 2023-09-22T21:04:42.335 DEBUG:boto:auth_path=/ffzdqu-7/?prefix=%26 2023-09-22T21:04:42.335 DEBUG:boto:Method: GET 2023-09-22T21:04:42.335 DEBUG:boto:Path: /ffzdqu-7/?prefix=%26 2023-09-22T21:04:42.335 DEBUG:boto:Data: 2023-09-22T21:04:42.335 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.335 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.335 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.336 DEBUG:boto:Params: {} 2023-09-22T21:04:42.336 DEBUG:boto:Token: None 2023-09-22T21:04:42.336 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/ 2023-09-22T21:04:42.336 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA= 2023-09-22T21:04:42.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA=', 'Content-Length': '0'} 2023-09-22T21:04:42.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9a6bcbcdccb076f-00650e016a-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.341 DEBUG:boto:b'ffzdqu-7&1000false&2023-09-22T21:04:36.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:04:42.341 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:42.341 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:42.341 DEBUG:boto:path=/ffzdqu-7/?prefix=%26 2023-09-22T21:04:42.342 DEBUG:boto:auth_path=/ffzdqu-7/?prefix=%26 2023-09-22T21:04:42.342 DEBUG:boto:Method: GET 2023-09-22T21:04:42.342 DEBUG:boto:Path: /ffzdqu-7/?prefix=%26 2023-09-22T21:04:42.342 DEBUG:boto:Data: 2023-09-22T21:04:42.342 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.342 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.342 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.342 DEBUG:boto:Params: {} 2023-09-22T21:04:42.342 DEBUG:boto:Token: None 2023-09-22T21:04:42.343 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/ 2023-09-22T21:04:42.343 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA= 2023-09-22T21:04:42.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA=', 'Content-Length': '0'} 2023-09-22T21:04:42.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c72837a726261cc-00650e016a-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.347 DEBUG:boto:b'ffzdqu-7&1000false&2023-09-22T21:04:36.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:04:42.348 DEBUG:boto:path=/ffzdqu-7/%26 2023-09-22T21:04:42.348 DEBUG:boto:auth_path=/ffzdqu-7/%26 2023-09-22T21:04:42.348 DEBUG:boto:Method: HEAD 2023-09-22T21:04:42.348 DEBUG:boto:Path: /ffzdqu-7/%26 2023-09-22T21:04:42.348 DEBUG:boto:Data: 2023-09-22T21:04:42.348 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.348 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.348 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.349 DEBUG:boto:Params: {} 2023-09-22T21:04:42.349 DEBUG:boto:Token: None 2023-09-22T21:04:42.349 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/%26 2023-09-22T21:04:42.349 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:04dqTCF8yxXv1pXTFFy2tV4O3PA= 2023-09-22T21:04:42.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:04dqTCF8yxXv1pXTFFy2tV4O3PA=', 'Content-Length': '0'} 2023-09-22T21:04:42.352 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000add46c3bf2ccbcd6-00650e016a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.352 DEBUG:boto:path=/ffzdqu-7/%26 2023-09-22T21:04:42.352 DEBUG:boto:auth_path=/ffzdqu-7/%26 2023-09-22T21:04:42.352 DEBUG:boto:Method: HEAD 2023-09-22T21:04:42.352 DEBUG:boto:Path: /ffzdqu-7/%26 2023-09-22T21:04:42.352 DEBUG:boto:Data: 2023-09-22T21:04:42.353 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.353 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.353 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.353 DEBUG:boto:Params: {} 2023-09-22T21:04:42.353 DEBUG:boto:Token: None 2023-09-22T21:04:42.353 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/%26 2023-09-22T21:04:42.353 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:04dqTCF8yxXv1pXTFFy2tV4O3PA= 2023-09-22T21:04:42.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:04dqTCF8yxXv1pXTFFy2tV4O3PA=', 'Content-Length': '0'} 2023-09-22T21:04:42.356 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003be7d47a72bedba7-00650e016a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-22T21:04:42.356 DEBUG:boto:path=/ffzdqu-7/%26 2023-09-22T21:04:42.356 DEBUG:boto:auth_path=/ffzdqu-7/%26 2023-09-22T21:04:42.356 DEBUG:boto:Method: GET 2023-09-22T21:04:42.356 DEBUG:boto:Path: /ffzdqu-7/%26 2023-09-22T21:04:42.357 DEBUG:boto:Data: 2023-09-22T21:04:42.357 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.357 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.357 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.357 DEBUG:boto:Params: {} 2023-09-22T21:04:42.357 DEBUG:boto:Token: None 2023-09-22T21:04:42.357 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/%26 2023-09-22T21:04:42.357 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:j97Iy7rNY4TWj6aCJcfEUoBJa7A= 2023-09-22T21:04:42.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:j97Iy7rNY4TWj6aCJcfEUoBJa7A=', 'Content-Length': '0'} 2023-09-22T21:04:42.360 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d36a176266a59351-00650e016a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.402 DEBUG:boto:path=/ffzdqu-7/%26 2023-09-22T21:04:42.403 DEBUG:boto:auth_path=/ffzdqu-7/%26 2023-09-22T21:04:42.403 DEBUG:boto:Method: GET 2023-09-22T21:04:42.403 DEBUG:boto:Path: /ffzdqu-7/%26 2023-09-22T21:04:42.403 DEBUG:boto:Data: 2023-09-22T21:04:42.403 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.403 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.404 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.404 DEBUG:boto:Params: {} 2023-09-22T21:04:42.404 DEBUG:boto:Token: None 2023-09-22T21:04:42.404 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/%26 2023-09-22T21:04:42.404 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:j97Iy7rNY4TWj6aCJcfEUoBJa7A= 2023-09-22T21:04:42.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:j97Iy7rNY4TWj6aCJcfEUoBJa7A=', 'Content-Length': '0'} 2023-09-22T21:04:42.407 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000071e5b8e97a0aba4-00650e016a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.408 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:42.408 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:42.408 DEBUG:boto:path=/ffzdqu-7/?prefix=%26 2023-09-22T21:04:42.408 DEBUG:boto:auth_path=/ffzdqu-7/?prefix=%26 2023-09-22T21:04:42.408 DEBUG:boto:Method: GET 2023-09-22T21:04:42.408 DEBUG:boto:Path: /ffzdqu-7/?prefix=%26 2023-09-22T21:04:42.408 DEBUG:boto:Data: 2023-09-22T21:04:42.408 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.408 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.409 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.409 DEBUG:boto:Params: {} 2023-09-22T21:04:42.409 DEBUG:boto:Token: None 2023-09-22T21:04:42.409 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/ 2023-09-22T21:04:42.409 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA= 2023-09-22T21:04:42.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA=', 'Content-Length': '0'} 2023-09-22T21:04:42.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002332465c668a11fd-00650e016a-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.414 DEBUG:boto:b'ffzdqu-7&1000false&2023-09-22T21:04:36.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:04:42.414 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:42.414 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:42.414 DEBUG:boto:path=/ffzdqu-7/?prefix=%26 2023-09-22T21:04:42.414 DEBUG:boto:auth_path=/ffzdqu-7/?prefix=%26 2023-09-22T21:04:42.415 DEBUG:boto:Method: GET 2023-09-22T21:04:42.415 DEBUG:boto:Path: /ffzdqu-7/?prefix=%26 2023-09-22T21:04:42.415 DEBUG:boto:Data: 2023-09-22T21:04:42.415 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.415 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.415 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.415 DEBUG:boto:Params: {} 2023-09-22T21:04:42.415 DEBUG:boto:Token: None 2023-09-22T21:04:42.415 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/ 2023-09-22T21:04:42.416 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA= 2023-09-22T21:04:42.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA=', 'Content-Length': '0'} 2023-09-22T21:04:42.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000500e51c0f7fabd80-00650e016a-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.462 DEBUG:boto:b'ffzdqu-7&1000false&2023-09-22T21:04:36.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:04:42.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-22T21:04:42.463 DEBUG:boto:path=/ffzdqu-7/%3A 2023-09-22T21:04:42.463 DEBUG:boto:auth_path=/ffzdqu-7/%3A 2023-09-22T21:04:42.463 DEBUG:boto:path=/ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.463 DEBUG:boto:auth_path=/ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.464 DEBUG:boto:Method: GET 2023-09-22T21:04:42.464 DEBUG:boto:Path: /ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.464 DEBUG:boto:Data: 2023-09-22T21:04:42.464 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.464 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.464 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.464 DEBUG:boto:Params: {} 2023-09-22T21:04:42.464 DEBUG:boto:Token: None 2023-09-22T21:04:42.464 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.465 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QhL9SYRGWee1l5Rha2XCH1y+KF8= 2023-09-22T21:04:42.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QhL9SYRGWee1l5Rha2XCH1y+KF8=', 'Content-Length': '0'} 2023-09-22T21:04:42.468 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006dcd3e91445a01c8-00650e016a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.468 DEBUG:boto:path=/ffzdqu-7/%3A 2023-09-22T21:04:42.468 DEBUG:boto:auth_path=/ffzdqu-7/%3A 2023-09-22T21:04:42.468 DEBUG:boto:path=/ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.468 DEBUG:boto:auth_path=/ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.468 DEBUG:boto:Method: GET 2023-09-22T21:04:42.469 DEBUG:boto:Path: /ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.469 DEBUG:boto:Data: 2023-09-22T21:04:42.469 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.469 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.469 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.469 DEBUG:boto:Params: {} 2023-09-22T21:04:42.469 DEBUG:boto:Token: None 2023-09-22T21:04:42.469 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.469 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QhL9SYRGWee1l5Rha2XCH1y+KF8= 2023-09-22T21:04:42.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QhL9SYRGWee1l5Rha2XCH1y+KF8=', 'Content-Length': '0'} 2023-09-22T21:04:42.472 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e648dfe0dba456bd-00650e016a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.473 DEBUG:boto:path=/ffzdqu-7/%3A 2023-09-22T21:04:42.473 DEBUG:boto:auth_path=/ffzdqu-7/%3A 2023-09-22T21:04:42.473 DEBUG:boto:path=/ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.473 DEBUG:boto:auth_path=/ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.473 DEBUG:boto:Method: HEAD 2023-09-22T21:04:42.473 DEBUG:boto:Path: /ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.473 DEBUG:boto:Data: 2023-09-22T21:04:42.473 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.473 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.474 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.474 DEBUG:boto:Params: {} 2023-09-22T21:04:42.474 DEBUG:boto:Token: None 2023-09-22T21:04:42.474 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.474 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PG2DiRHUcts3Eb+3+vzYei2+ECM= 2023-09-22T21:04:42.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PG2DiRHUcts3Eb+3+vzYei2+ECM=', 'Content-Length': '0'} 2023-09-22T21:04:42.477 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea88395180b908c3-00650e016a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.477 DEBUG:boto:path=/ffzdqu-7/%3A 2023-09-22T21:04:42.477 DEBUG:boto:auth_path=/ffzdqu-7/%3A 2023-09-22T21:04:42.477 DEBUG:boto:path=/ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.478 DEBUG:boto:auth_path=/ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.478 DEBUG:boto:Method: HEAD 2023-09-22T21:04:42.478 DEBUG:boto:Path: /ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.478 DEBUG:boto:Data: 2023-09-22T21:04:42.478 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.478 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.479 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.479 DEBUG:boto:Params: {} 2023-09-22T21:04:42.479 DEBUG:boto:Token: None 2023-09-22T21:04:42.479 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.479 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PG2DiRHUcts3Eb+3+vzYei2+ECM= 2023-09-22T21:04:42.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PG2DiRHUcts3Eb+3+vzYei2+ECM=', 'Content-Length': '0'} 2023-09-22T21:04:42.482 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018d2cd200be7065c-00650e016a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-22T21:04:42.483 DEBUG:boto:path=/ffzdqu-7/%3A 2023-09-22T21:04:42.483 DEBUG:boto:auth_path=/ffzdqu-7/%3A 2023-09-22T21:04:42.483 DEBUG:boto:path=/ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.483 DEBUG:boto:auth_path=/ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.483 DEBUG:boto:Method: GET 2023-09-22T21:04:42.483 DEBUG:boto:Path: /ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.483 DEBUG:boto:Data: 2023-09-22T21:04:42.484 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.484 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.484 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.484 DEBUG:boto:Params: {} 2023-09-22T21:04:42.484 DEBUG:boto:Token: None 2023-09-22T21:04:42.485 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.485 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QhL9SYRGWee1l5Rha2XCH1y+KF8= 2023-09-22T21:04:42.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QhL9SYRGWee1l5Rha2XCH1y+KF8=', 'Content-Length': '0'} 2023-09-22T21:04:42.488 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd355669fca607e8-00650e016a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.530 DEBUG:boto:path=/ffzdqu-7/%3A 2023-09-22T21:04:42.531 DEBUG:boto:auth_path=/ffzdqu-7/%3A 2023-09-22T21:04:42.531 DEBUG:boto:path=/ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.531 DEBUG:boto:auth_path=/ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.531 DEBUG:boto:Method: GET 2023-09-22T21:04:42.531 DEBUG:boto:Path: /ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.531 DEBUG:boto:Data: 2023-09-22T21:04:42.531 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.532 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.532 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.532 DEBUG:boto:Params: {} 2023-09-22T21:04:42.532 DEBUG:boto:Token: None 2023-09-22T21:04:42.532 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/%3A?versionId=null 2023-09-22T21:04:42.532 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QhL9SYRGWee1l5Rha2XCH1y+KF8= 2023-09-22T21:04:42.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QhL9SYRGWee1l5Rha2XCH1y+KF8=', 'Content-Length': '0'} 2023-09-22T21:04:42.535 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e5b54a9e2bace1c-00650e016a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.535 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:42.536 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:42.536 DEBUG:boto:path=/ffzdqu-7/?prefix=%3A 2023-09-22T21:04:42.536 DEBUG:boto:auth_path=/ffzdqu-7/?prefix=%3A 2023-09-22T21:04:42.536 DEBUG:boto:Method: GET 2023-09-22T21:04:42.536 DEBUG:boto:Path: /ffzdqu-7/?prefix=%3A 2023-09-22T21:04:42.536 DEBUG:boto:Data: 2023-09-22T21:04:42.536 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.536 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.536 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.536 DEBUG:boto:Params: {} 2023-09-22T21:04:42.537 DEBUG:boto:Token: None 2023-09-22T21:04:42.537 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/ 2023-09-22T21:04:42.537 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA= 2023-09-22T21:04:42.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA=', 'Content-Length': '0'} 2023-09-22T21:04:42.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000943d514b3a5585ec-00650e016a-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.541 DEBUG:boto:b'ffzdqu-7:1000false:2023-09-22T21:04:36.610Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:04:42.542 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:42.542 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:42.542 DEBUG:boto:path=/ffzdqu-7/?prefix=%3A 2023-09-22T21:04:42.542 DEBUG:boto:auth_path=/ffzdqu-7/?prefix=%3A 2023-09-22T21:04:42.542 DEBUG:boto:Method: GET 2023-09-22T21:04:42.542 DEBUG:boto:Path: /ffzdqu-7/?prefix=%3A 2023-09-22T21:04:42.542 DEBUG:boto:Data: 2023-09-22T21:04:42.542 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.543 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.543 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.543 DEBUG:boto:Params: {} 2023-09-22T21:04:42.543 DEBUG:boto:Token: None 2023-09-22T21:04:42.543 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/ 2023-09-22T21:04:42.543 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA= 2023-09-22T21:04:42.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA=', 'Content-Length': '0'} 2023-09-22T21:04:42.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000103e18547868222-00650e016a-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.594 DEBUG:boto:b'ffzdqu-7:1000false:2023-09-22T21:04:36.610Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:04:42.595 DEBUG:boto:path=/ffzdqu-7/%3A 2023-09-22T21:04:42.595 DEBUG:boto:auth_path=/ffzdqu-7/%3A 2023-09-22T21:04:42.595 DEBUG:boto:Method: HEAD 2023-09-22T21:04:42.596 DEBUG:boto:Path: /ffzdqu-7/%3A 2023-09-22T21:04:42.596 DEBUG:boto:Data: 2023-09-22T21:04:42.596 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.596 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.596 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.596 DEBUG:boto:Params: {} 2023-09-22T21:04:42.597 DEBUG:boto:Token: None 2023-09-22T21:04:42.597 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/%3A 2023-09-22T21:04:42.597 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9OgYKztxPGVanGIvf935cElUT3k= 2023-09-22T21:04:42.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9OgYKztxPGVanGIvf935cElUT3k=', 'Content-Length': '0'} 2023-09-22T21:04:42.600 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4b8fa134573bf3c-00650e016a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.601 DEBUG:boto:path=/ffzdqu-7/%3A 2023-09-22T21:04:42.601 DEBUG:boto:auth_path=/ffzdqu-7/%3A 2023-09-22T21:04:42.601 DEBUG:boto:Method: HEAD 2023-09-22T21:04:42.601 DEBUG:boto:Path: /ffzdqu-7/%3A 2023-09-22T21:04:42.601 DEBUG:boto:Data: 2023-09-22T21:04:42.601 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.602 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.602 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.602 DEBUG:boto:Params: {} 2023-09-22T21:04:42.602 DEBUG:boto:Token: None 2023-09-22T21:04:42.603 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/%3A 2023-09-22T21:04:42.603 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9OgYKztxPGVanGIvf935cElUT3k= 2023-09-22T21:04:42.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9OgYKztxPGVanGIvf935cElUT3k=', 'Content-Length': '0'} 2023-09-22T21:04:42.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af0b09753f0a5b8e-00650e016a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-22T21:04:42.607 DEBUG:boto:path=/ffzdqu-7/%3A 2023-09-22T21:04:42.607 DEBUG:boto:auth_path=/ffzdqu-7/%3A 2023-09-22T21:04:42.607 DEBUG:boto:Method: GET 2023-09-22T21:04:42.607 DEBUG:boto:Path: /ffzdqu-7/%3A 2023-09-22T21:04:42.607 DEBUG:boto:Data: 2023-09-22T21:04:42.607 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.607 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.607 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.607 DEBUG:boto:Params: {} 2023-09-22T21:04:42.608 DEBUG:boto:Token: None 2023-09-22T21:04:42.608 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/%3A 2023-09-22T21:04:42.608 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AfdH58LW8VyI/xW/pZEKKfEglSA= 2023-09-22T21:04:42.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AfdH58LW8VyI/xW/pZEKKfEglSA=', 'Content-Length': '0'} 2023-09-22T21:04:42.611 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b0e0376fdd342cc-00650e016a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.658 DEBUG:boto:path=/ffzdqu-7/%3A 2023-09-22T21:04:42.659 DEBUG:boto:auth_path=/ffzdqu-7/%3A 2023-09-22T21:04:42.659 DEBUG:boto:Method: GET 2023-09-22T21:04:42.659 DEBUG:boto:Path: /ffzdqu-7/%3A 2023-09-22T21:04:42.659 DEBUG:boto:Data: 2023-09-22T21:04:42.659 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.659 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.659 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.659 DEBUG:boto:Params: {} 2023-09-22T21:04:42.660 DEBUG:boto:Token: None 2023-09-22T21:04:42.660 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/%3A 2023-09-22T21:04:42.660 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AfdH58LW8VyI/xW/pZEKKfEglSA= 2023-09-22T21:04:42.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AfdH58LW8VyI/xW/pZEKKfEglSA=', 'Content-Length': '0'} 2023-09-22T21:04:42.663 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000700824c08d2adc4d-00650e016a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.664 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:42.664 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:42.664 DEBUG:boto:path=/ffzdqu-7/?prefix=%3A 2023-09-22T21:04:42.664 DEBUG:boto:auth_path=/ffzdqu-7/?prefix=%3A 2023-09-22T21:04:42.664 DEBUG:boto:Method: GET 2023-09-22T21:04:42.664 DEBUG:boto:Path: /ffzdqu-7/?prefix=%3A 2023-09-22T21:04:42.664 DEBUG:boto:Data: 2023-09-22T21:04:42.664 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.664 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.665 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.665 DEBUG:boto:Params: {} 2023-09-22T21:04:42.665 DEBUG:boto:Token: None 2023-09-22T21:04:42.665 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/ 2023-09-22T21:04:42.665 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA= 2023-09-22T21:04:42.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA=', 'Content-Length': '0'} 2023-09-22T21:04:42.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfb1f78a1b20916b-00650e016a-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.670 DEBUG:boto:b'ffzdqu-7:1000false:2023-09-22T21:04:36.610Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:04:42.671 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:42.671 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:42.671 DEBUG:boto:path=/ffzdqu-7/?prefix=%3A 2023-09-22T21:04:42.671 DEBUG:boto:auth_path=/ffzdqu-7/?prefix=%3A 2023-09-22T21:04:42.671 DEBUG:boto:Method: GET 2023-09-22T21:04:42.671 DEBUG:boto:Path: /ffzdqu-7/?prefix=%3A 2023-09-22T21:04:42.671 DEBUG:boto:Data: 2023-09-22T21:04:42.671 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.671 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.672 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.672 DEBUG:boto:Params: {} 2023-09-22T21:04:42.672 DEBUG:boto:Token: None 2023-09-22T21:04:42.672 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/ 2023-09-22T21:04:42.672 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA= 2023-09-22T21:04:42.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA=', 'Content-Length': '0'} 2023-09-22T21:04:42.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000646ec95cb912397f-00650e016a-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.677 DEBUG:boto:b'ffzdqu-7:1000false:2023-09-22T21:04:36.610Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:04:42.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-22T21:04:42.678 DEBUG:boto:path=/ffzdqu-7/_myobj 2023-09-22T21:04:42.678 DEBUG:boto:auth_path=/ffzdqu-7/_myobj 2023-09-22T21:04:42.678 DEBUG:boto:path=/ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.678 DEBUG:boto:auth_path=/ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.678 DEBUG:boto:Method: GET 2023-09-22T21:04:42.679 DEBUG:boto:Path: /ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.679 DEBUG:boto:Data: 2023-09-22T21:04:42.679 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.679 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.679 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.679 DEBUG:boto:Params: {} 2023-09-22T21:04:42.679 DEBUG:boto:Token: None 2023-09-22T21:04:42.679 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.679 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IkxGiM/+S6TPGLVDSLhM9z6EL6Y= 2023-09-22T21:04:42.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IkxGiM/+S6TPGLVDSLhM9z6EL6Y=', 'Content-Length': '0'} 2023-09-22T21:04:42.682 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033bda1327e4e2877-00650e016a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.683 DEBUG:boto:path=/ffzdqu-7/_myobj 2023-09-22T21:04:42.683 DEBUG:boto:auth_path=/ffzdqu-7/_myobj 2023-09-22T21:04:42.683 DEBUG:boto:path=/ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.683 DEBUG:boto:auth_path=/ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.683 DEBUG:boto:Method: GET 2023-09-22T21:04:42.683 DEBUG:boto:Path: /ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.683 DEBUG:boto:Data: 2023-09-22T21:04:42.683 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.683 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.684 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.684 DEBUG:boto:Params: {} 2023-09-22T21:04:42.684 DEBUG:boto:Token: None 2023-09-22T21:04:42.684 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.684 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IkxGiM/+S6TPGLVDSLhM9z6EL6Y= 2023-09-22T21:04:42.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IkxGiM/+S6TPGLVDSLhM9z6EL6Y=', 'Content-Length': '0'} 2023-09-22T21:04:42.687 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008aef3a86141bd6c9-00650e016a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.730 DEBUG:boto:path=/ffzdqu-7/_myobj 2023-09-22T21:04:42.730 DEBUG:boto:auth_path=/ffzdqu-7/_myobj 2023-09-22T21:04:42.731 DEBUG:boto:path=/ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.731 DEBUG:boto:auth_path=/ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.731 DEBUG:boto:Method: HEAD 2023-09-22T21:04:42.731 DEBUG:boto:Path: /ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.731 DEBUG:boto:Data: 2023-09-22T21:04:42.731 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.731 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.731 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.731 DEBUG:boto:Params: {} 2023-09-22T21:04:42.732 DEBUG:boto:Token: None 2023-09-22T21:04:42.732 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.732 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fabhFEbMa73n8q7gdudT3faILXM= 2023-09-22T21:04:42.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fabhFEbMa73n8q7gdudT3faILXM=', 'Content-Length': '0'} 2023-09-22T21:04:42.735 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da1bb964bbe5b668-00650e016a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.736 DEBUG:boto:path=/ffzdqu-7/_myobj 2023-09-22T21:04:42.736 DEBUG:boto:auth_path=/ffzdqu-7/_myobj 2023-09-22T21:04:42.736 DEBUG:boto:path=/ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.736 DEBUG:boto:auth_path=/ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.736 DEBUG:boto:Method: HEAD 2023-09-22T21:04:42.736 DEBUG:boto:Path: /ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.736 DEBUG:boto:Data: 2023-09-22T21:04:42.736 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.737 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.737 DEBUG:boto:Params: {} 2023-09-22T21:04:42.737 DEBUG:boto:Token: None 2023-09-22T21:04:42.737 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.737 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fabhFEbMa73n8q7gdudT3faILXM= 2023-09-22T21:04:42.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fabhFEbMa73n8q7gdudT3faILXM=', 'Content-Length': '0'} 2023-09-22T21:04:42.740 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f2b0df1aa954ffc-00650e016a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-22T21:04:42.741 DEBUG:boto:path=/ffzdqu-7/_myobj 2023-09-22T21:04:42.741 DEBUG:boto:auth_path=/ffzdqu-7/_myobj 2023-09-22T21:04:42.741 DEBUG:boto:path=/ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.741 DEBUG:boto:auth_path=/ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.741 DEBUG:boto:Method: GET 2023-09-22T21:04:42.741 DEBUG:boto:Path: /ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.741 DEBUG:boto:Data: 2023-09-22T21:04:42.741 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.741 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.742 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.742 DEBUG:boto:Params: {} 2023-09-22T21:04:42.742 DEBUG:boto:Token: None 2023-09-22T21:04:42.742 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.742 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IkxGiM/+S6TPGLVDSLhM9z6EL6Y= 2023-09-22T21:04:42.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IkxGiM/+S6TPGLVDSLhM9z6EL6Y=', 'Content-Length': '0'} 2023-09-22T21:04:42.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ed901ddd6b07ba5-00650e016a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.786 DEBUG:boto:path=/ffzdqu-7/_myobj 2023-09-22T21:04:42.787 DEBUG:boto:auth_path=/ffzdqu-7/_myobj 2023-09-22T21:04:42.787 DEBUG:boto:path=/ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.787 DEBUG:boto:auth_path=/ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.787 DEBUG:boto:Method: GET 2023-09-22T21:04:42.787 DEBUG:boto:Path: /ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.787 DEBUG:boto:Data: 2023-09-22T21:04:42.787 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.787 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.787 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.788 DEBUG:boto:Params: {} 2023-09-22T21:04:42.788 DEBUG:boto:Token: None 2023-09-22T21:04:42.788 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/_myobj?versionId=null 2023-09-22T21:04:42.788 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IkxGiM/+S6TPGLVDSLhM9z6EL6Y= 2023-09-22T21:04:42.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IkxGiM/+S6TPGLVDSLhM9z6EL6Y=', 'Content-Length': '0'} 2023-09-22T21:04:42.791 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fd7ecd4929c8b6b-00650e016a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.791 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:42.791 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:42.791 DEBUG:boto:path=/ffzdqu-7/?prefix=_myobj 2023-09-22T21:04:42.791 DEBUG:boto:auth_path=/ffzdqu-7/?prefix=_myobj 2023-09-22T21:04:42.792 DEBUG:boto:Method: GET 2023-09-22T21:04:42.792 DEBUG:boto:Path: /ffzdqu-7/?prefix=_myobj 2023-09-22T21:04:42.792 DEBUG:boto:Data: 2023-09-22T21:04:42.792 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.792 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.792 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.792 DEBUG:boto:Params: {} 2023-09-22T21:04:42.792 DEBUG:boto:Token: None 2023-09-22T21:04:42.792 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/ 2023-09-22T21:04:42.793 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA= 2023-09-22T21:04:42.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA=', 'Content-Length': '0'} 2023-09-22T21:04:42.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e4fce8319bf5d41-00650e016a-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.798 DEBUG:boto:b'ffzdqu-7_myobj1000false_myobj2023-09-22T21:04:36.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:04:42.798 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:42.798 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:42.798 DEBUG:boto:path=/ffzdqu-7/?prefix=_myobj 2023-09-22T21:04:42.798 DEBUG:boto:auth_path=/ffzdqu-7/?prefix=_myobj 2023-09-22T21:04:42.798 DEBUG:boto:Method: GET 2023-09-22T21:04:42.799 DEBUG:boto:Path: /ffzdqu-7/?prefix=_myobj 2023-09-22T21:04:42.799 DEBUG:boto:Data: 2023-09-22T21:04:42.799 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.799 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.799 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.799 DEBUG:boto:Params: {} 2023-09-22T21:04:42.799 DEBUG:boto:Token: None 2023-09-22T21:04:42.799 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/ 2023-09-22T21:04:42.799 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA= 2023-09-22T21:04:42.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA=', 'Content-Length': '0'} 2023-09-22T21:04:42.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec9fede1bcf8ec88-00650e016a-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.804 DEBUG:boto:b'ffzdqu-7_myobj1000false_myobj2023-09-22T21:04:36.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:04:42.804 DEBUG:boto:path=/ffzdqu-7/_myobj 2023-09-22T21:04:42.804 DEBUG:boto:auth_path=/ffzdqu-7/_myobj 2023-09-22T21:04:42.805 DEBUG:boto:Method: HEAD 2023-09-22T21:04:42.805 DEBUG:boto:Path: /ffzdqu-7/_myobj 2023-09-22T21:04:42.805 DEBUG:boto:Data: 2023-09-22T21:04:42.805 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.805 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.805 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.805 DEBUG:boto:Params: {} 2023-09-22T21:04:42.805 DEBUG:boto:Token: None 2023-09-22T21:04:42.806 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/_myobj 2023-09-22T21:04:42.806 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7JSpnFZNeuGWMyDR1JYYKv9X24I= 2023-09-22T21:04:42.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7JSpnFZNeuGWMyDR1JYYKv9X24I=', 'Content-Length': '0'} 2023-09-22T21:04:42.809 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c987d7cdc4b5e91b-00650e016a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.809 DEBUG:boto:path=/ffzdqu-7/_myobj 2023-09-22T21:04:42.809 DEBUG:boto:auth_path=/ffzdqu-7/_myobj 2023-09-22T21:04:42.809 DEBUG:boto:Method: HEAD 2023-09-22T21:04:42.809 DEBUG:boto:Path: /ffzdqu-7/_myobj 2023-09-22T21:04:42.809 DEBUG:boto:Data: 2023-09-22T21:04:42.810 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.810 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.810 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.810 DEBUG:boto:Params: {} 2023-09-22T21:04:42.810 DEBUG:boto:Token: None 2023-09-22T21:04:42.810 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/_myobj 2023-09-22T21:04:42.810 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7JSpnFZNeuGWMyDR1JYYKv9X24I= 2023-09-22T21:04:42.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7JSpnFZNeuGWMyDR1JYYKv9X24I=', 'Content-Length': '0'} 2023-09-22T21:04:42.813 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4c2700cbf8ed7f4-00650e016a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-22T21:04:42.813 DEBUG:boto:path=/ffzdqu-7/_myobj 2023-09-22T21:04:42.814 DEBUG:boto:auth_path=/ffzdqu-7/_myobj 2023-09-22T21:04:42.814 DEBUG:boto:Method: GET 2023-09-22T21:04:42.814 DEBUG:boto:Path: /ffzdqu-7/_myobj 2023-09-22T21:04:42.814 DEBUG:boto:Data: 2023-09-22T21:04:42.814 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.814 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.814 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.814 DEBUG:boto:Params: {} 2023-09-22T21:04:42.814 DEBUG:boto:Token: None 2023-09-22T21:04:42.815 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/_myobj 2023-09-22T21:04:42.815 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IWnITfMpK6AsRZWQj/6iJMRxhIA= 2023-09-22T21:04:42.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IWnITfMpK6AsRZWQj/6iJMRxhIA=', 'Content-Length': '0'} 2023-09-22T21:04:42.818 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008cff2650d25ee295-00650e016a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.818 DEBUG:boto:path=/ffzdqu-7/_myobj 2023-09-22T21:04:42.818 DEBUG:boto:auth_path=/ffzdqu-7/_myobj 2023-09-22T21:04:42.818 DEBUG:boto:Method: GET 2023-09-22T21:04:42.819 DEBUG:boto:Path: /ffzdqu-7/_myobj 2023-09-22T21:04:42.819 DEBUG:boto:Data: 2023-09-22T21:04:42.819 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.819 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.819 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.819 DEBUG:boto:Params: {} 2023-09-22T21:04:42.819 DEBUG:boto:Token: None 2023-09-22T21:04:42.819 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/_myobj 2023-09-22T21:04:42.820 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IWnITfMpK6AsRZWQj/6iJMRxhIA= 2023-09-22T21:04:42.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IWnITfMpK6AsRZWQj/6iJMRxhIA=', 'Content-Length': '0'} 2023-09-22T21:04:42.823 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007864a2f8cc7e4ab6-00650e016a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.866 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:42.867 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:42.867 DEBUG:boto:path=/ffzdqu-7/?prefix=_myobj 2023-09-22T21:04:42.867 DEBUG:boto:auth_path=/ffzdqu-7/?prefix=_myobj 2023-09-22T21:04:42.867 DEBUG:boto:Method: GET 2023-09-22T21:04:42.867 DEBUG:boto:Path: /ffzdqu-7/?prefix=_myobj 2023-09-22T21:04:42.867 DEBUG:boto:Data: 2023-09-22T21:04:42.867 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.867 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.867 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.868 DEBUG:boto:Params: {} 2023-09-22T21:04:42.868 DEBUG:boto:Token: None 2023-09-22T21:04:42.868 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/ 2023-09-22T21:04:42.868 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA= 2023-09-22T21:04:42.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA=', 'Content-Length': '0'} 2023-09-22T21:04:42.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e24db9fab0a6ac2c-00650e016a-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.877 DEBUG:boto:b'ffzdqu-7_myobj1000false_myobj2023-09-22T21:04:36.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:04:42.877 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:42.878 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:42.878 DEBUG:boto:path=/ffzdqu-7/?prefix=_myobj 2023-09-22T21:04:42.878 DEBUG:boto:auth_path=/ffzdqu-7/?prefix=_myobj 2023-09-22T21:04:42.878 DEBUG:boto:Method: GET 2023-09-22T21:04:42.878 DEBUG:boto:Path: /ffzdqu-7/?prefix=_myobj 2023-09-22T21:04:42.878 DEBUG:boto:Data: 2023-09-22T21:04:42.878 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.878 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.878 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.879 DEBUG:boto:Params: {} 2023-09-22T21:04:42.879 DEBUG:boto:Token: None 2023-09-22T21:04:42.879 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/ 2023-09-22T21:04:42.879 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA= 2023-09-22T21:04:42.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8iCwDam6XDbTVuDY3DZI1TXLlHA=', 'Content-Length': '0'} 2023-09-22T21:04:42.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068fd75670c5958b5-00650e016a-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.884 DEBUG:boto:b'ffzdqu-7_myobj1000false_myobj2023-09-22T21:04:36.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:04:42.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-22T21:04:42.884 DEBUG:boto:path=/ffzdqu-7/myobj 2023-09-22T21:04:42.884 DEBUG:boto:auth_path=/ffzdqu-7/myobj 2023-09-22T21:04:42.884 DEBUG:boto:path=/ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.885 DEBUG:boto:auth_path=/ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.885 DEBUG:boto:Method: GET 2023-09-22T21:04:42.885 DEBUG:boto:Path: /ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.885 DEBUG:boto:Data: 2023-09-22T21:04:42.885 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.885 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.885 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.885 DEBUG:boto:Params: {} 2023-09-22T21:04:42.885 DEBUG:boto:Token: None 2023-09-22T21:04:42.886 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.886 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MXrB+23GwbjpV7ZWHmSHuICsTW4= 2023-09-22T21:04:42.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MXrB+23GwbjpV7ZWHmSHuICsTW4=', 'Content-Length': '0'} 2023-09-22T21:04:42.889 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc8c0c332abcaffa-00650e016a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.934 DEBUG:boto:path=/ffzdqu-7/myobj 2023-09-22T21:04:42.935 DEBUG:boto:auth_path=/ffzdqu-7/myobj 2023-09-22T21:04:42.935 DEBUG:boto:path=/ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.935 DEBUG:boto:auth_path=/ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.935 DEBUG:boto:Method: GET 2023-09-22T21:04:42.935 DEBUG:boto:Path: /ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.935 DEBUG:boto:Data: 2023-09-22T21:04:42.935 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.935 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.936 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.936 DEBUG:boto:Params: {} 2023-09-22T21:04:42.936 DEBUG:boto:Token: None 2023-09-22T21:04:42.936 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.936 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MXrB+23GwbjpV7ZWHmSHuICsTW4= 2023-09-22T21:04:42.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MXrB+23GwbjpV7ZWHmSHuICsTW4=', 'Content-Length': '0'} 2023-09-22T21:04:42.939 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078d3d0b9d10b47f2-00650e016a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.940 DEBUG:boto:path=/ffzdqu-7/myobj 2023-09-22T21:04:42.940 DEBUG:boto:auth_path=/ffzdqu-7/myobj 2023-09-22T21:04:42.940 DEBUG:boto:path=/ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.940 DEBUG:boto:auth_path=/ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.940 DEBUG:boto:Method: HEAD 2023-09-22T21:04:42.940 DEBUG:boto:Path: /ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.941 DEBUG:boto:Data: 2023-09-22T21:04:42.941 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.941 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.941 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.941 DEBUG:boto:Params: {} 2023-09-22T21:04:42.941 DEBUG:boto:Token: None 2023-09-22T21:04:42.941 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.941 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uLNVtT9vMIxIRghNAreAi6eEX9Q= 2023-09-22T21:04:42.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uLNVtT9vMIxIRghNAreAi6eEX9Q=', 'Content-Length': '0'} 2023-09-22T21:04:42.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000337353a4e32454c4-00650e016a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.945 DEBUG:boto:path=/ffzdqu-7/myobj 2023-09-22T21:04:42.945 DEBUG:boto:auth_path=/ffzdqu-7/myobj 2023-09-22T21:04:42.945 DEBUG:boto:path=/ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.945 DEBUG:boto:auth_path=/ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.945 DEBUG:boto:Method: HEAD 2023-09-22T21:04:42.945 DEBUG:boto:Path: /ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.945 DEBUG:boto:Data: 2023-09-22T21:04:42.945 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.946 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.946 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.946 DEBUG:boto:Params: {} 2023-09-22T21:04:42.946 DEBUG:boto:Token: None 2023-09-22T21:04:42.946 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.946 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uLNVtT9vMIxIRghNAreAi6eEX9Q= 2023-09-22T21:04:42.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uLNVtT9vMIxIRghNAreAi6eEX9Q=', 'Content-Length': '0'} 2023-09-22T21:04:42.949 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e061b009d9557789-00650e016a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-22T21:04:42.949 DEBUG:boto:path=/ffzdqu-7/myobj 2023-09-22T21:04:42.950 DEBUG:boto:auth_path=/ffzdqu-7/myobj 2023-09-22T21:04:42.950 DEBUG:boto:path=/ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.950 DEBUG:boto:auth_path=/ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.950 DEBUG:boto:Method: GET 2023-09-22T21:04:42.950 DEBUG:boto:Path: /ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.950 DEBUG:boto:Data: 2023-09-22T21:04:42.950 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.950 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:42.950 DEBUG:boto:Port: 8001 2023-09-22T21:04:42.951 DEBUG:boto:Params: {} 2023-09-22T21:04:42.951 DEBUG:boto:Token: None 2023-09-22T21:04:42.951 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.951 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MXrB+23GwbjpV7ZWHmSHuICsTW4= 2023-09-22T21:04:42.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MXrB+23GwbjpV7ZWHmSHuICsTW4=', 'Content-Length': '0'} 2023-09-22T21:04:42.954 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b874fc212efaebd-00650e016a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.994 DEBUG:boto:path=/ffzdqu-7/myobj 2023-09-22T21:04:42.995 DEBUG:boto:auth_path=/ffzdqu-7/myobj 2023-09-22T21:04:42.995 DEBUG:boto:path=/ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.995 DEBUG:boto:auth_path=/ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.995 DEBUG:boto:Method: GET 2023-09-22T21:04:42.995 DEBUG:boto:Path: /ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.995 DEBUG:boto:Data: 2023-09-22T21:04:42.995 DEBUG:boto:Headers: {} 2023-09-22T21:04:42.995 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:42.995 DEBUG:boto:Port: 8000 2023-09-22T21:04:42.996 DEBUG:boto:Params: {} 2023-09-22T21:04:42.996 DEBUG:boto:Token: None 2023-09-22T21:04:42.996 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:42 GMT /ffzdqu-7/myobj?versionId=null 2023-09-22T21:04:42.996 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MXrB+23GwbjpV7ZWHmSHuICsTW4= 2023-09-22T21:04:42.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MXrB+23GwbjpV7ZWHmSHuICsTW4=', 'Content-Length': '0'} 2023-09-22T21:04:42.999 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc0f82c16c68323c-00650e016a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:42.999 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:43.000 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:43.000 DEBUG:boto:path=/ffzdqu-7/?prefix=myobj 2023-09-22T21:04:43.000 DEBUG:boto:auth_path=/ffzdqu-7/?prefix=myobj 2023-09-22T21:04:43.000 DEBUG:boto:Method: GET 2023-09-22T21:04:43.000 DEBUG:boto:Path: /ffzdqu-7/?prefix=myobj 2023-09-22T21:04:43.000 DEBUG:boto:Data: 2023-09-22T21:04:43.000 DEBUG:boto:Headers: {} 2023-09-22T21:04:43.000 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:43.000 DEBUG:boto:Port: 8001 2023-09-22T21:04:43.000 DEBUG:boto:Params: {} 2023-09-22T21:04:43.001 DEBUG:boto:Token: None 2023-09-22T21:04:43.001 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:43 GMT /ffzdqu-7/ 2023-09-22T21:04:43.001 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lax1BpRz+IJ+bO+fHoO7xGW0qrg= 2023-09-22T21:04:43.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lax1BpRz+IJ+bO+fHoO7xGW0qrg=', 'Content-Length': '0'} 2023-09-22T21:04:43.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6ccbaef7ad95003-00650e016b-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:43.006 DEBUG:boto:b'ffzdqu-7myobj1000falsemyobj2023-09-22T21:04:36.542Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:04:43.006 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:43.006 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:43.006 DEBUG:boto:path=/ffzdqu-7/?prefix=myobj 2023-09-22T21:04:43.006 DEBUG:boto:auth_path=/ffzdqu-7/?prefix=myobj 2023-09-22T21:04:43.006 DEBUG:boto:Method: GET 2023-09-22T21:04:43.007 DEBUG:boto:Path: /ffzdqu-7/?prefix=myobj 2023-09-22T21:04:43.007 DEBUG:boto:Data: 2023-09-22T21:04:43.007 DEBUG:boto:Headers: {} 2023-09-22T21:04:43.007 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:43.007 DEBUG:boto:Port: 8000 2023-09-22T21:04:43.007 DEBUG:boto:Params: {} 2023-09-22T21:04:43.007 DEBUG:boto:Token: None 2023-09-22T21:04:43.007 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:43 GMT /ffzdqu-7/ 2023-09-22T21:04:43.008 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lax1BpRz+IJ+bO+fHoO7xGW0qrg= 2023-09-22T21:04:43.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lax1BpRz+IJ+bO+fHoO7xGW0qrg=', 'Content-Length': '0'} 2023-09-22T21:04:43.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092b72e3a259d6ccc-00650e016b-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:43.054 DEBUG:boto:b'ffzdqu-7myobj1000falsemyobj2023-09-22T21:04:36.542Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:04:43.055 DEBUG:boto:path=/ffzdqu-7/myobj 2023-09-22T21:04:43.055 DEBUG:boto:auth_path=/ffzdqu-7/myobj 2023-09-22T21:04:43.055 DEBUG:boto:Method: HEAD 2023-09-22T21:04:43.055 DEBUG:boto:Path: /ffzdqu-7/myobj 2023-09-22T21:04:43.055 DEBUG:boto:Data: 2023-09-22T21:04:43.055 DEBUG:boto:Headers: {} 2023-09-22T21:04:43.055 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:43.056 DEBUG:boto:Port: 8001 2023-09-22T21:04:43.056 DEBUG:boto:Params: {} 2023-09-22T21:04:43.056 DEBUG:boto:Token: None 2023-09-22T21:04:43.056 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:43 GMT /ffzdqu-7/myobj 2023-09-22T21:04:43.056 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HYXrYVmipD2DJ8r1B+x1XsEvkMo= 2023-09-22T21:04:43.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HYXrYVmipD2DJ8r1B+x1XsEvkMo=', 'Content-Length': '0'} 2023-09-22T21:04:43.059 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006f6764221c23dde-00650e016b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:43.059 DEBUG:boto:path=/ffzdqu-7/myobj 2023-09-22T21:04:43.059 DEBUG:boto:auth_path=/ffzdqu-7/myobj 2023-09-22T21:04:43.059 DEBUG:boto:Method: HEAD 2023-09-22T21:04:43.060 DEBUG:boto:Path: /ffzdqu-7/myobj 2023-09-22T21:04:43.060 DEBUG:boto:Data: 2023-09-22T21:04:43.060 DEBUG:boto:Headers: {} 2023-09-22T21:04:43.060 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:43.060 DEBUG:boto:Port: 8000 2023-09-22T21:04:43.060 DEBUG:boto:Params: {} 2023-09-22T21:04:43.060 DEBUG:boto:Token: None 2023-09-22T21:04:43.060 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:04:43 GMT /ffzdqu-7/myobj 2023-09-22T21:04:43.060 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HYXrYVmipD2DJ8r1B+x1XsEvkMo= 2023-09-22T21:04:43.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HYXrYVmipD2DJ8r1B+x1XsEvkMo=', 'Content-Length': '0'} 2023-09-22T21:04:43.063 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008feab77bb5bf9b5c-00650e016b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:43.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-22T21:04:43.064 DEBUG:boto:path=/ffzdqu-7/myobj 2023-09-22T21:04:43.064 DEBUG:boto:auth_path=/ffzdqu-7/myobj 2023-09-22T21:04:43.064 DEBUG:boto:Method: GET 2023-09-22T21:04:43.064 DEBUG:boto:Path: /ffzdqu-7/myobj 2023-09-22T21:04:43.064 DEBUG:boto:Data: 2023-09-22T21:04:43.064 DEBUG:boto:Headers: {} 2023-09-22T21:04:43.064 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:43.064 DEBUG:boto:Port: 8001 2023-09-22T21:04:43.064 DEBUG:boto:Params: {} 2023-09-22T21:04:43.065 DEBUG:boto:Token: None 2023-09-22T21:04:43.065 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:43 GMT /ffzdqu-7/myobj 2023-09-22T21:04:43.065 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Hn/aM4cbk7z8dobSaJZXsPQLpyU= 2023-09-22T21:04:43.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Hn/aM4cbk7z8dobSaJZXsPQLpyU=', 'Content-Length': '0'} 2023-09-22T21:04:43.067 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4bc7d1bf284e8cb-00650e016b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:43.110 DEBUG:boto:path=/ffzdqu-7/myobj 2023-09-22T21:04:43.111 DEBUG:boto:auth_path=/ffzdqu-7/myobj 2023-09-22T21:04:43.111 DEBUG:boto:Method: GET 2023-09-22T21:04:43.111 DEBUG:boto:Path: /ffzdqu-7/myobj 2023-09-22T21:04:43.111 DEBUG:boto:Data: 2023-09-22T21:04:43.111 DEBUG:boto:Headers: {} 2023-09-22T21:04:43.111 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:43.111 DEBUG:boto:Port: 8000 2023-09-22T21:04:43.111 DEBUG:boto:Params: {} 2023-09-22T21:04:43.112 DEBUG:boto:Token: None 2023-09-22T21:04:43.112 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:43 GMT /ffzdqu-7/myobj 2023-09-22T21:04:43.112 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Hn/aM4cbk7z8dobSaJZXsPQLpyU= 2023-09-22T21:04:43.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Hn/aM4cbk7z8dobSaJZXsPQLpyU=', 'Content-Length': '0'} 2023-09-22T21:04:43.115 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000572a8c5d7821cb88-00650e016b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:43.115 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:43.115 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:43.116 DEBUG:boto:path=/ffzdqu-7/?prefix=myobj 2023-09-22T21:04:43.116 DEBUG:boto:auth_path=/ffzdqu-7/?prefix=myobj 2023-09-22T21:04:43.116 DEBUG:boto:Method: GET 2023-09-22T21:04:43.116 DEBUG:boto:Path: /ffzdqu-7/?prefix=myobj 2023-09-22T21:04:43.116 DEBUG:boto:Data: 2023-09-22T21:04:43.116 DEBUG:boto:Headers: {} 2023-09-22T21:04:43.116 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:04:43.116 DEBUG:boto:Port: 8001 2023-09-22T21:04:43.116 DEBUG:boto:Params: {} 2023-09-22T21:04:43.117 DEBUG:boto:Token: None 2023-09-22T21:04:43.117 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:43 GMT /ffzdqu-7/ 2023-09-22T21:04:43.117 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lax1BpRz+IJ+bO+fHoO7xGW0qrg= 2023-09-22T21:04:43.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lax1BpRz+IJ+bO+fHoO7xGW0qrg=', 'Content-Length': '0'} 2023-09-22T21:04:43.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a68a009ec2526b1-00650e016b-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:43.121 DEBUG:boto:b'ffzdqu-7myobj1000falsemyobj2023-09-22T21:04:36.542Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:04:43.122 DEBUG:boto:path=/ffzdqu-7/ 2023-09-22T21:04:43.122 DEBUG:boto:auth_path=/ffzdqu-7/ 2023-09-22T21:04:43.122 DEBUG:boto:path=/ffzdqu-7/?prefix=myobj 2023-09-22T21:04:43.122 DEBUG:boto:auth_path=/ffzdqu-7/?prefix=myobj 2023-09-22T21:04:43.122 DEBUG:boto:Method: GET 2023-09-22T21:04:43.122 DEBUG:boto:Path: /ffzdqu-7/?prefix=myobj 2023-09-22T21:04:43.123 DEBUG:boto:Data: 2023-09-22T21:04:43.123 DEBUG:boto:Headers: {} 2023-09-22T21:04:43.123 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:04:43.123 DEBUG:boto:Port: 8000 2023-09-22T21:04:43.123 DEBUG:boto:Params: {} 2023-09-22T21:04:43.123 DEBUG:boto:Token: None 2023-09-22T21:04:43.123 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:04:43 GMT /ffzdqu-7/ 2023-09-22T21:04:43.123 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lax1BpRz+IJ+bO+fHoO7xGW0qrg= 2023-09-22T21:04:43.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:04:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lax1BpRz+IJ+bO+fHoO7xGW0qrg=', 'Content-Length': '0'} 2023-09-22T21:04:43.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d611c0b5f6c242ad-00650e016b-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:04:43.128 DEBUG:boto:b'ffzdqu-7myobj1000falsemyobj2023-09-22T21:04:36.542Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:04:43.128 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-7 zones={a2, a1} 2023-09-22T21:04:43.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:04:43.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:04:43.129 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-8 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:04:43.179 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:04:43.179 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:04:43.221 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:43.221+0000 7fc632c6ca40 1 waiting to reach incremental sync.. 2023-09-22T21:04:48.222 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:48.221+0000 7fc632c6ca40 1 waiting to reach incremental sync.. 2023-09-22T21:04:53.223 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:53.225+0000 7fc632c6ca40 1 waiting to reach incremental sync.. 2023-09-22T21:04:58.224 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:04:58.225+0000 7fc632c6ca40 1 waiting to reach incremental sync.. 2023-09-22T21:05:03.228 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:05:03.229+0000 7fc632c6ca40 1 bucket sync caught up with source: 2023-09-22T21:05:03.228 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.9.6, , 00000000001.6.6, , , , 00000000001.11.6, 00000000001.8.6, , ] 2023-09-22T21:05:03.228 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.9.6, , 00000000001.6.6, , , , 00000000001.11.6, 00000000001.8.6, , ] 2023-09-22T21:05:03.228 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:05:03.229+0000 7fc632c6ca40 0 bucket checkpoint complete 2023-09-22T21:05:03.257 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-8 zones={a1, a2} 2023-09-22T21:05:03.257 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:03.257 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:03.257 DEBUG:boto:Method: HEAD 2023-09-22T21:05:03.258 DEBUG:boto:Path: /ffzdqu-8/ 2023-09-22T21:05:03.258 DEBUG:boto:Data: 2023-09-22T21:05:03.258 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.258 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.258 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.258 DEBUG:boto:Params: {} 2023-09-22T21:05:03.258 DEBUG:boto:Token: None 2023-09-22T21:05:03.258 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/ 2023-09-22T21:05:03.259 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eFAQu05DAD3HDnOGrcJXVrTMAYw= 2023-09-22T21:05:03.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eFAQu05DAD3HDnOGrcJXVrTMAYw=', 'Content-Length': '0'} 2023-09-22T21:05:03.262 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6e17fd0e19a0041-00650e017f-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.262 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:03.262 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:03.263 DEBUG:boto:Method: HEAD 2023-09-22T21:05:03.263 DEBUG:boto:Path: /ffzdqu-8/ 2023-09-22T21:05:03.263 DEBUG:boto:Data: 2023-09-22T21:05:03.263 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.263 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.263 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.263 DEBUG:boto:Params: {} 2023-09-22T21:05:03.263 DEBUG:boto:Token: None 2023-09-22T21:05:03.263 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/ 2023-09-22T21:05:03.264 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eFAQu05DAD3HDnOGrcJXVrTMAYw= 2023-09-22T21:05:03.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eFAQu05DAD3HDnOGrcJXVrTMAYw=', 'Content-Length': '0'} 2023-09-22T21:05:03.267 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b08d836cf32b86cb-00650e017f-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.267 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:05:03.267 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:03.267 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:03.268 DEBUG:boto:path=/ffzdqu-8/?versions&max-keys=999 2023-09-22T21:05:03.268 DEBUG:boto:auth_path=/ffzdqu-8/?versions&max-keys=999 2023-09-22T21:05:03.268 DEBUG:boto:Method: GET 2023-09-22T21:05:03.268 DEBUG:boto:Path: /ffzdqu-8/?versions&max-keys=999 2023-09-22T21:05:03.268 DEBUG:boto:Data: 2023-09-22T21:05:03.268 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.268 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.268 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.268 DEBUG:boto:Params: {} 2023-09-22T21:05:03.268 DEBUG:boto:Token: None 2023-09-22T21:05:03.269 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/?versions 2023-09-22T21:05:03.269 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fG8CbKZDL7qhxyIaklTNLx8ZMlc= 2023-09-22T21:05:03.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fG8CbKZDL7qhxyIaklTNLx8ZMlc=', 'Content-Length': '0'} 2023-09-22T21:05:03.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092cb1fe16009fdcc-00650e017f-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.272 DEBUG:boto:b'ffzdqu-8999false&nulltrue2023-09-22T21:04:36.788Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-22T21:04:36.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-22T21:04:36.708Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-22T21:04:36.683Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:03.273 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-09-22T21:05:03.273 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-09-22T21:05:03.273 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-09-22T21:05:03.273 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-22T21:05:03.273 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:05:03.273 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:03.273 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:03.273 DEBUG:boto:path=/ffzdqu-8/?versions&max-keys=999 2023-09-22T21:05:03.274 DEBUG:boto:auth_path=/ffzdqu-8/?versions&max-keys=999 2023-09-22T21:05:03.274 DEBUG:boto:Method: GET 2023-09-22T21:05:03.274 DEBUG:boto:Path: /ffzdqu-8/?versions&max-keys=999 2023-09-22T21:05:03.274 DEBUG:boto:Data: 2023-09-22T21:05:03.274 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.274 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.274 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.274 DEBUG:boto:Params: {} 2023-09-22T21:05:03.274 DEBUG:boto:Token: None 2023-09-22T21:05:03.275 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/?versions 2023-09-22T21:05:03.275 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fG8CbKZDL7qhxyIaklTNLx8ZMlc= 2023-09-22T21:05:03.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fG8CbKZDL7qhxyIaklTNLx8ZMlc=', 'Content-Length': '0'} 2023-09-22T21:05:03.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfa509bb19b33897-00650e017f-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.278 DEBUG:boto:b'ffzdqu-8999false&nulltrue2023-09-22T21:04:36.788Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-22T21:04:36.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-22T21:04:36.708Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-22T21:04:36.683Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:03.278 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-09-22T21:05:03.279 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-09-22T21:05:03.279 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-09-22T21:05:03.279 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-22T21:05:03.279 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:03.279 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:03.279 DEBUG:boto:path=/ffzdqu-8/?versions&max-keys=999 2023-09-22T21:05:03.279 DEBUG:boto:auth_path=/ffzdqu-8/?versions&max-keys=999 2023-09-22T21:05:03.279 DEBUG:boto:Method: GET 2023-09-22T21:05:03.280 DEBUG:boto:Path: /ffzdqu-8/?versions&max-keys=999 2023-09-22T21:05:03.280 DEBUG:boto:Data: 2023-09-22T21:05:03.280 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.280 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.280 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.280 DEBUG:boto:Params: {} 2023-09-22T21:05:03.280 DEBUG:boto:Token: None 2023-09-22T21:05:03.280 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/?versions 2023-09-22T21:05:03.280 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fG8CbKZDL7qhxyIaklTNLx8ZMlc= 2023-09-22T21:05:03.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fG8CbKZDL7qhxyIaklTNLx8ZMlc=', 'Content-Length': '0'} 2023-09-22T21:05:03.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae7788a2197b24f9-00650e017f-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.284 DEBUG:boto:b'ffzdqu-8999false&nulltrue2023-09-22T21:04:36.788Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-22T21:04:36.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-22T21:04:36.708Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-22T21:04:36.683Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:03.284 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:03.285 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:03.285 DEBUG:boto:path=/ffzdqu-8/?versions&max-keys=999 2023-09-22T21:05:03.285 DEBUG:boto:auth_path=/ffzdqu-8/?versions&max-keys=999 2023-09-22T21:05:03.285 DEBUG:boto:Method: GET 2023-09-22T21:05:03.285 DEBUG:boto:Path: /ffzdqu-8/?versions&max-keys=999 2023-09-22T21:05:03.285 DEBUG:boto:Data: 2023-09-22T21:05:03.285 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.285 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.285 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.285 DEBUG:boto:Params: {} 2023-09-22T21:05:03.286 DEBUG:boto:Token: None 2023-09-22T21:05:03.286 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/?versions 2023-09-22T21:05:03.286 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fG8CbKZDL7qhxyIaklTNLx8ZMlc= 2023-09-22T21:05:03.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fG8CbKZDL7qhxyIaklTNLx8ZMlc=', 'Content-Length': '0'} 2023-09-22T21:05:03.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073a2e508484b4cd0-00650e017f-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.290 DEBUG:boto:b'ffzdqu-8999false&nulltrue2023-09-22T21:04:36.788Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-22T21:04:36.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-22T21:04:36.708Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-22T21:04:36.683Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:03.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-22T21:05:03.291 DEBUG:boto:path=/ffzdqu-8/%26 2023-09-22T21:05:03.291 DEBUG:boto:auth_path=/ffzdqu-8/%26 2023-09-22T21:05:03.291 DEBUG:boto:path=/ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.291 DEBUG:boto:auth_path=/ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.291 DEBUG:boto:Method: GET 2023-09-22T21:05:03.291 DEBUG:boto:Path: /ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.292 DEBUG:boto:Data: 2023-09-22T21:05:03.292 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.292 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.292 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.292 DEBUG:boto:Params: {} 2023-09-22T21:05:03.292 DEBUG:boto:Token: None 2023-09-22T21:05:03.292 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.292 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qTPvxpBNL5WZNlCCcgoWIe7PeRc= 2023-09-22T21:05:03.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qTPvxpBNL5WZNlCCcgoWIe7PeRc=', 'Content-Length': '0'} 2023-09-22T21:05:03.295 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046d9abe3dde26987-00650e017f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.342 DEBUG:boto:path=/ffzdqu-8/%26 2023-09-22T21:05:03.343 DEBUG:boto:auth_path=/ffzdqu-8/%26 2023-09-22T21:05:03.343 DEBUG:boto:path=/ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.343 DEBUG:boto:auth_path=/ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.343 DEBUG:boto:Method: GET 2023-09-22T21:05:03.343 DEBUG:boto:Path: /ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.343 DEBUG:boto:Data: 2023-09-22T21:05:03.343 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.343 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.343 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.343 DEBUG:boto:Params: {} 2023-09-22T21:05:03.344 DEBUG:boto:Token: None 2023-09-22T21:05:03.344 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.344 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qTPvxpBNL5WZNlCCcgoWIe7PeRc= 2023-09-22T21:05:03.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qTPvxpBNL5WZNlCCcgoWIe7PeRc=', 'Content-Length': '0'} 2023-09-22T21:05:03.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007196ae2ef0b22357-00650e017f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.347 DEBUG:boto:path=/ffzdqu-8/%26 2023-09-22T21:05:03.347 DEBUG:boto:auth_path=/ffzdqu-8/%26 2023-09-22T21:05:03.348 DEBUG:boto:path=/ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.348 DEBUG:boto:auth_path=/ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.348 DEBUG:boto:Method: HEAD 2023-09-22T21:05:03.348 DEBUG:boto:Path: /ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.348 DEBUG:boto:Data: 2023-09-22T21:05:03.348 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.348 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.348 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.348 DEBUG:boto:Params: {} 2023-09-22T21:05:03.348 DEBUG:boto:Token: None 2023-09-22T21:05:03.349 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.349 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g2lFEPmoCNvcCHWDIhhrCw73GGA= 2023-09-22T21:05:03.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g2lFEPmoCNvcCHWDIhhrCw73GGA=', 'Content-Length': '0'} 2023-09-22T21:05:03.351 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a8990d530e29217-00650e017f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.352 DEBUG:boto:path=/ffzdqu-8/%26 2023-09-22T21:05:03.352 DEBUG:boto:auth_path=/ffzdqu-8/%26 2023-09-22T21:05:03.352 DEBUG:boto:path=/ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.352 DEBUG:boto:auth_path=/ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.352 DEBUG:boto:Method: HEAD 2023-09-22T21:05:03.352 DEBUG:boto:Path: /ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.352 DEBUG:boto:Data: 2023-09-22T21:05:03.352 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.353 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.353 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.353 DEBUG:boto:Params: {} 2023-09-22T21:05:03.353 DEBUG:boto:Token: None 2023-09-22T21:05:03.353 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.353 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g2lFEPmoCNvcCHWDIhhrCw73GGA= 2023-09-22T21:05:03.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g2lFEPmoCNvcCHWDIhhrCw73GGA=', 'Content-Length': '0'} 2023-09-22T21:05:03.356 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000883fd4f90c715246-00650e017f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-22T21:05:03.356 DEBUG:boto:path=/ffzdqu-8/%26 2023-09-22T21:05:03.356 DEBUG:boto:auth_path=/ffzdqu-8/%26 2023-09-22T21:05:03.356 DEBUG:boto:path=/ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.356 DEBUG:boto:auth_path=/ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.357 DEBUG:boto:Method: GET 2023-09-22T21:05:03.357 DEBUG:boto:Path: /ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.357 DEBUG:boto:Data: 2023-09-22T21:05:03.357 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.357 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.357 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.357 DEBUG:boto:Params: {} 2023-09-22T21:05:03.357 DEBUG:boto:Token: None 2023-09-22T21:05:03.357 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.358 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qTPvxpBNL5WZNlCCcgoWIe7PeRc= 2023-09-22T21:05:03.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qTPvxpBNL5WZNlCCcgoWIe7PeRc=', 'Content-Length': '0'} 2023-09-22T21:05:03.360 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078212b71b2d3a86b-00650e017f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.360 DEBUG:boto:path=/ffzdqu-8/%26 2023-09-22T21:05:03.361 DEBUG:boto:auth_path=/ffzdqu-8/%26 2023-09-22T21:05:03.361 DEBUG:boto:path=/ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.361 DEBUG:boto:auth_path=/ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.361 DEBUG:boto:Method: GET 2023-09-22T21:05:03.361 DEBUG:boto:Path: /ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.361 DEBUG:boto:Data: 2023-09-22T21:05:03.361 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.361 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.361 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.361 DEBUG:boto:Params: {} 2023-09-22T21:05:03.362 DEBUG:boto:Token: None 2023-09-22T21:05:03.362 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/%26?versionId=null 2023-09-22T21:05:03.362 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qTPvxpBNL5WZNlCCcgoWIe7PeRc= 2023-09-22T21:05:03.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qTPvxpBNL5WZNlCCcgoWIe7PeRc=', 'Content-Length': '0'} 2023-09-22T21:05:03.365 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044768fde75e5bea3-00650e017f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.406 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:03.407 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:03.407 DEBUG:boto:path=/ffzdqu-8/?prefix=%26 2023-09-22T21:05:03.407 DEBUG:boto:auth_path=/ffzdqu-8/?prefix=%26 2023-09-22T21:05:03.407 DEBUG:boto:Method: GET 2023-09-22T21:05:03.407 DEBUG:boto:Path: /ffzdqu-8/?prefix=%26 2023-09-22T21:05:03.407 DEBUG:boto:Data: 2023-09-22T21:05:03.407 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.408 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.408 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.408 DEBUG:boto:Params: {} 2023-09-22T21:05:03.408 DEBUG:boto:Token: None 2023-09-22T21:05:03.408 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/ 2023-09-22T21:05:03.408 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA= 2023-09-22T21:05:03.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA=', 'Content-Length': '0'} 2023-09-22T21:05:03.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000b53826701def5b-00650e017f-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.413 DEBUG:boto:b'ffzdqu-8&1000false&2023-09-22T21:04:36.788Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:03.413 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:03.414 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:03.414 DEBUG:boto:path=/ffzdqu-8/?prefix=%26 2023-09-22T21:05:03.414 DEBUG:boto:auth_path=/ffzdqu-8/?prefix=%26 2023-09-22T21:05:03.414 DEBUG:boto:Method: GET 2023-09-22T21:05:03.414 DEBUG:boto:Path: /ffzdqu-8/?prefix=%26 2023-09-22T21:05:03.414 DEBUG:boto:Data: 2023-09-22T21:05:03.414 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.414 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.415 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.415 DEBUG:boto:Params: {} 2023-09-22T21:05:03.415 DEBUG:boto:Token: None 2023-09-22T21:05:03.415 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/ 2023-09-22T21:05:03.415 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA= 2023-09-22T21:05:03.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA=', 'Content-Length': '0'} 2023-09-22T21:05:03.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002598983ef404e912-00650e017f-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.420 DEBUG:boto:b'ffzdqu-8&1000false&2023-09-22T21:04:36.788Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:03.421 DEBUG:boto:path=/ffzdqu-8/%26 2023-09-22T21:05:03.421 DEBUG:boto:auth_path=/ffzdqu-8/%26 2023-09-22T21:05:03.421 DEBUG:boto:Method: HEAD 2023-09-22T21:05:03.421 DEBUG:boto:Path: /ffzdqu-8/%26 2023-09-22T21:05:03.421 DEBUG:boto:Data: 2023-09-22T21:05:03.421 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.422 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.422 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.422 DEBUG:boto:Params: {} 2023-09-22T21:05:03.422 DEBUG:boto:Token: None 2023-09-22T21:05:03.422 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/%26 2023-09-22T21:05:03.422 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:R1xWfiwmij0N63D+SqXSROO30dE= 2023-09-22T21:05:03.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:R1xWfiwmij0N63D+SqXSROO30dE=', 'Content-Length': '0'} 2023-09-22T21:05:03.425 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e5d6fd5173cf503-00650e017f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.425 DEBUG:boto:path=/ffzdqu-8/%26 2023-09-22T21:05:03.425 DEBUG:boto:auth_path=/ffzdqu-8/%26 2023-09-22T21:05:03.425 DEBUG:boto:Method: HEAD 2023-09-22T21:05:03.425 DEBUG:boto:Path: /ffzdqu-8/%26 2023-09-22T21:05:03.426 DEBUG:boto:Data: 2023-09-22T21:05:03.426 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.426 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.426 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.426 DEBUG:boto:Params: {} 2023-09-22T21:05:03.426 DEBUG:boto:Token: None 2023-09-22T21:05:03.426 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/%26 2023-09-22T21:05:03.426 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:R1xWfiwmij0N63D+SqXSROO30dE= 2023-09-22T21:05:03.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:R1xWfiwmij0N63D+SqXSROO30dE=', 'Content-Length': '0'} 2023-09-22T21:05:03.429 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a3838e839607dc9-00650e017f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-22T21:05:03.429 DEBUG:boto:path=/ffzdqu-8/%26 2023-09-22T21:05:03.430 DEBUG:boto:auth_path=/ffzdqu-8/%26 2023-09-22T21:05:03.430 DEBUG:boto:Method: GET 2023-09-22T21:05:03.430 DEBUG:boto:Path: /ffzdqu-8/%26 2023-09-22T21:05:03.430 DEBUG:boto:Data: 2023-09-22T21:05:03.430 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.430 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.430 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.430 DEBUG:boto:Params: {} 2023-09-22T21:05:03.430 DEBUG:boto:Token: None 2023-09-22T21:05:03.431 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/%26 2023-09-22T21:05:03.431 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GKOG52pRuaiLbnFwDvUVEi8Dl8E= 2023-09-22T21:05:03.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GKOG52pRuaiLbnFwDvUVEi8Dl8E=', 'Content-Length': '0'} 2023-09-22T21:05:03.433 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000881fc85f9cbdb1c-00650e017f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.474 DEBUG:boto:path=/ffzdqu-8/%26 2023-09-22T21:05:03.475 DEBUG:boto:auth_path=/ffzdqu-8/%26 2023-09-22T21:05:03.475 DEBUG:boto:Method: GET 2023-09-22T21:05:03.475 DEBUG:boto:Path: /ffzdqu-8/%26 2023-09-22T21:05:03.475 DEBUG:boto:Data: 2023-09-22T21:05:03.475 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.475 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.475 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.475 DEBUG:boto:Params: {} 2023-09-22T21:05:03.476 DEBUG:boto:Token: None 2023-09-22T21:05:03.476 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/%26 2023-09-22T21:05:03.476 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GKOG52pRuaiLbnFwDvUVEi8Dl8E= 2023-09-22T21:05:03.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GKOG52pRuaiLbnFwDvUVEi8Dl8E=', 'Content-Length': '0'} 2023-09-22T21:05:03.479 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006fda9f1d349fd20-00650e017f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.479 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:03.479 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:03.479 DEBUG:boto:path=/ffzdqu-8/?prefix=%26 2023-09-22T21:05:03.479 DEBUG:boto:auth_path=/ffzdqu-8/?prefix=%26 2023-09-22T21:05:03.480 DEBUG:boto:Method: GET 2023-09-22T21:05:03.480 DEBUG:boto:Path: /ffzdqu-8/?prefix=%26 2023-09-22T21:05:03.480 DEBUG:boto:Data: 2023-09-22T21:05:03.480 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.480 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.480 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.480 DEBUG:boto:Params: {} 2023-09-22T21:05:03.480 DEBUG:boto:Token: None 2023-09-22T21:05:03.480 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/ 2023-09-22T21:05:03.481 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA= 2023-09-22T21:05:03.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA=', 'Content-Length': '0'} 2023-09-22T21:05:03.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d788b6ff432d11ed-00650e017f-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.485 DEBUG:boto:b'ffzdqu-8&1000false&2023-09-22T21:04:36.788Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:03.486 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:03.486 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:03.486 DEBUG:boto:path=/ffzdqu-8/?prefix=%26 2023-09-22T21:05:03.486 DEBUG:boto:auth_path=/ffzdqu-8/?prefix=%26 2023-09-22T21:05:03.486 DEBUG:boto:Method: GET 2023-09-22T21:05:03.486 DEBUG:boto:Path: /ffzdqu-8/?prefix=%26 2023-09-22T21:05:03.486 DEBUG:boto:Data: 2023-09-22T21:05:03.486 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.487 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.487 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.487 DEBUG:boto:Params: {} 2023-09-22T21:05:03.487 DEBUG:boto:Token: None 2023-09-22T21:05:03.487 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/ 2023-09-22T21:05:03.487 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA= 2023-09-22T21:05:03.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA=', 'Content-Length': '0'} 2023-09-22T21:05:03.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047123a415f156948-00650e017f-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.534 DEBUG:boto:b'ffzdqu-8&1000false&2023-09-22T21:04:36.788Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:03.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-22T21:05:03.535 DEBUG:boto:path=/ffzdqu-8/%3A 2023-09-22T21:05:03.535 DEBUG:boto:auth_path=/ffzdqu-8/%3A 2023-09-22T21:05:03.535 DEBUG:boto:path=/ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.535 DEBUG:boto:auth_path=/ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.536 DEBUG:boto:Method: GET 2023-09-22T21:05:03.536 DEBUG:boto:Path: /ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.536 DEBUG:boto:Data: 2023-09-22T21:05:03.536 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.536 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.536 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.536 DEBUG:boto:Params: {} 2023-09-22T21:05:03.536 DEBUG:boto:Token: None 2023-09-22T21:05:03.536 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.537 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7gEL2p4pD1uCLXIcQ64LfipkIcw= 2023-09-22T21:05:03.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7gEL2p4pD1uCLXIcQ64LfipkIcw=', 'Content-Length': '0'} 2023-09-22T21:05:03.539 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001bae3b0ba8949383-00650e017f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.540 DEBUG:boto:path=/ffzdqu-8/%3A 2023-09-22T21:05:03.540 DEBUG:boto:auth_path=/ffzdqu-8/%3A 2023-09-22T21:05:03.540 DEBUG:boto:path=/ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.540 DEBUG:boto:auth_path=/ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.540 DEBUG:boto:Method: GET 2023-09-22T21:05:03.540 DEBUG:boto:Path: /ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.541 DEBUG:boto:Data: 2023-09-22T21:05:03.541 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.541 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.541 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.541 DEBUG:boto:Params: {} 2023-09-22T21:05:03.541 DEBUG:boto:Token: None 2023-09-22T21:05:03.541 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.541 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7gEL2p4pD1uCLXIcQ64LfipkIcw= 2023-09-22T21:05:03.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7gEL2p4pD1uCLXIcQ64LfipkIcw=', 'Content-Length': '0'} 2023-09-22T21:05:03.544 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000044f691be1bdba84-00650e017f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.544 DEBUG:boto:path=/ffzdqu-8/%3A 2023-09-22T21:05:03.544 DEBUG:boto:auth_path=/ffzdqu-8/%3A 2023-09-22T21:05:03.545 DEBUG:boto:path=/ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.545 DEBUG:boto:auth_path=/ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.545 DEBUG:boto:Method: HEAD 2023-09-22T21:05:03.545 DEBUG:boto:Path: /ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.545 DEBUG:boto:Data: 2023-09-22T21:05:03.545 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.545 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.545 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.545 DEBUG:boto:Params: {} 2023-09-22T21:05:03.545 DEBUG:boto:Token: None 2023-09-22T21:05:03.546 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.546 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oX6thKjeJxDRKxDtQhDT0HGGHp8= 2023-09-22T21:05:03.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oX6thKjeJxDRKxDtQhDT0HGGHp8=', 'Content-Length': '0'} 2023-09-22T21:05:03.548 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff79770681089390-00650e017f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.549 DEBUG:boto:path=/ffzdqu-8/%3A 2023-09-22T21:05:03.549 DEBUG:boto:auth_path=/ffzdqu-8/%3A 2023-09-22T21:05:03.549 DEBUG:boto:path=/ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.549 DEBUG:boto:auth_path=/ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.549 DEBUG:boto:Method: HEAD 2023-09-22T21:05:03.549 DEBUG:boto:Path: /ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.549 DEBUG:boto:Data: 2023-09-22T21:05:03.549 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.549 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.550 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.550 DEBUG:boto:Params: {} 2023-09-22T21:05:03.550 DEBUG:boto:Token: None 2023-09-22T21:05:03.550 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.550 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oX6thKjeJxDRKxDtQhDT0HGGHp8= 2023-09-22T21:05:03.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oX6thKjeJxDRKxDtQhDT0HGGHp8=', 'Content-Length': '0'} 2023-09-22T21:05:03.553 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e078c5f4b0390e8f-00650e017f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-22T21:05:03.553 DEBUG:boto:path=/ffzdqu-8/%3A 2023-09-22T21:05:03.553 DEBUG:boto:auth_path=/ffzdqu-8/%3A 2023-09-22T21:05:03.553 DEBUG:boto:path=/ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.553 DEBUG:boto:auth_path=/ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.554 DEBUG:boto:Method: GET 2023-09-22T21:05:03.554 DEBUG:boto:Path: /ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.554 DEBUG:boto:Data: 2023-09-22T21:05:03.554 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.554 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.554 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.554 DEBUG:boto:Params: {} 2023-09-22T21:05:03.554 DEBUG:boto:Token: None 2023-09-22T21:05:03.554 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.555 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7gEL2p4pD1uCLXIcQ64LfipkIcw= 2023-09-22T21:05:03.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7gEL2p4pD1uCLXIcQ64LfipkIcw=', 'Content-Length': '0'} 2023-09-22T21:05:03.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c5e95dcba07d80b-00650e017f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.558 DEBUG:boto:path=/ffzdqu-8/%3A 2023-09-22T21:05:03.558 DEBUG:boto:auth_path=/ffzdqu-8/%3A 2023-09-22T21:05:03.558 DEBUG:boto:path=/ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.558 DEBUG:boto:auth_path=/ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.559 DEBUG:boto:Method: GET 2023-09-22T21:05:03.559 DEBUG:boto:Path: /ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.559 DEBUG:boto:Data: 2023-09-22T21:05:03.559 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.559 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.559 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.559 DEBUG:boto:Params: {} 2023-09-22T21:05:03.559 DEBUG:boto:Token: None 2023-09-22T21:05:03.560 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/%3A?versionId=null 2023-09-22T21:05:03.560 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7gEL2p4pD1uCLXIcQ64LfipkIcw= 2023-09-22T21:05:03.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7gEL2p4pD1uCLXIcQ64LfipkIcw=', 'Content-Length': '0'} 2023-09-22T21:05:03.563 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1cfa45427204673-00650e017f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.563 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:03.563 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:03.563 DEBUG:boto:path=/ffzdqu-8/?prefix=%3A 2023-09-22T21:05:03.563 DEBUG:boto:auth_path=/ffzdqu-8/?prefix=%3A 2023-09-22T21:05:03.563 DEBUG:boto:Method: GET 2023-09-22T21:05:03.563 DEBUG:boto:Path: /ffzdqu-8/?prefix=%3A 2023-09-22T21:05:03.564 DEBUG:boto:Data: 2023-09-22T21:05:03.564 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.564 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.564 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.564 DEBUG:boto:Params: {} 2023-09-22T21:05:03.564 DEBUG:boto:Token: None 2023-09-22T21:05:03.564 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/ 2023-09-22T21:05:03.564 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA= 2023-09-22T21:05:03.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA=', 'Content-Length': '0'} 2023-09-22T21:05:03.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebfdf4d92bb795ed-00650e017f-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.569 DEBUG:boto:b'ffzdqu-8:1000false:2023-09-22T21:04:36.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:03.570 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:03.570 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:03.570 DEBUG:boto:path=/ffzdqu-8/?prefix=%3A 2023-09-22T21:05:03.570 DEBUG:boto:auth_path=/ffzdqu-8/?prefix=%3A 2023-09-22T21:05:03.570 DEBUG:boto:Method: GET 2023-09-22T21:05:03.570 DEBUG:boto:Path: /ffzdqu-8/?prefix=%3A 2023-09-22T21:05:03.570 DEBUG:boto:Data: 2023-09-22T21:05:03.570 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.570 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.571 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.571 DEBUG:boto:Params: {} 2023-09-22T21:05:03.571 DEBUG:boto:Token: None 2023-09-22T21:05:03.571 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/ 2023-09-22T21:05:03.571 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA= 2023-09-22T21:05:03.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA=', 'Content-Length': '0'} 2023-09-22T21:05:03.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007987cae79e1723f7-00650e017f-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.618 DEBUG:boto:b'ffzdqu-8:1000false:2023-09-22T21:04:36.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:03.619 DEBUG:boto:path=/ffzdqu-8/%3A 2023-09-22T21:05:03.619 DEBUG:boto:auth_path=/ffzdqu-8/%3A 2023-09-22T21:05:03.619 DEBUG:boto:Method: HEAD 2023-09-22T21:05:03.619 DEBUG:boto:Path: /ffzdqu-8/%3A 2023-09-22T21:05:03.619 DEBUG:boto:Data: 2023-09-22T21:05:03.619 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.620 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.620 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.620 DEBUG:boto:Params: {} 2023-09-22T21:05:03.620 DEBUG:boto:Token: None 2023-09-22T21:05:03.620 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/%3A 2023-09-22T21:05:03.620 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pdnRpBrkRG6j1BxkFHQyKiBcoU8= 2023-09-22T21:05:03.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pdnRpBrkRG6j1BxkFHQyKiBcoU8=', 'Content-Length': '0'} 2023-09-22T21:05:03.623 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000991146a0162701a-00650e017f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.623 DEBUG:boto:path=/ffzdqu-8/%3A 2023-09-22T21:05:03.623 DEBUG:boto:auth_path=/ffzdqu-8/%3A 2023-09-22T21:05:03.624 DEBUG:boto:Method: HEAD 2023-09-22T21:05:03.624 DEBUG:boto:Path: /ffzdqu-8/%3A 2023-09-22T21:05:03.624 DEBUG:boto:Data: 2023-09-22T21:05:03.624 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.624 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.624 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.624 DEBUG:boto:Params: {} 2023-09-22T21:05:03.624 DEBUG:boto:Token: None 2023-09-22T21:05:03.624 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/%3A 2023-09-22T21:05:03.625 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pdnRpBrkRG6j1BxkFHQyKiBcoU8= 2023-09-22T21:05:03.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pdnRpBrkRG6j1BxkFHQyKiBcoU8=', 'Content-Length': '0'} 2023-09-22T21:05:03.627 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000934fb68b5f768978-00650e017f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-22T21:05:03.627 DEBUG:boto:path=/ffzdqu-8/%3A 2023-09-22T21:05:03.628 DEBUG:boto:auth_path=/ffzdqu-8/%3A 2023-09-22T21:05:03.628 DEBUG:boto:Method: GET 2023-09-22T21:05:03.628 DEBUG:boto:Path: /ffzdqu-8/%3A 2023-09-22T21:05:03.628 DEBUG:boto:Data: 2023-09-22T21:05:03.628 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.628 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.628 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.628 DEBUG:boto:Params: {} 2023-09-22T21:05:03.628 DEBUG:boto:Token: None 2023-09-22T21:05:03.629 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/%3A 2023-09-22T21:05:03.629 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UYEl2oTY31ADVJAbiBiZ9p8LlS0= 2023-09-22T21:05:03.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UYEl2oTY31ADVJAbiBiZ9p8LlS0=', 'Content-Length': '0'} 2023-09-22T21:05:03.631 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000016076ae429d9b5b-00650e017f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.674 DEBUG:boto:path=/ffzdqu-8/%3A 2023-09-22T21:05:03.674 DEBUG:boto:auth_path=/ffzdqu-8/%3A 2023-09-22T21:05:03.675 DEBUG:boto:Method: GET 2023-09-22T21:05:03.675 DEBUG:boto:Path: /ffzdqu-8/%3A 2023-09-22T21:05:03.675 DEBUG:boto:Data: 2023-09-22T21:05:03.675 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.675 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.675 DEBUG:boto:Params: {} 2023-09-22T21:05:03.675 DEBUG:boto:Token: None 2023-09-22T21:05:03.675 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/%3A 2023-09-22T21:05:03.676 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UYEl2oTY31ADVJAbiBiZ9p8LlS0= 2023-09-22T21:05:03.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UYEl2oTY31ADVJAbiBiZ9p8LlS0=', 'Content-Length': '0'} 2023-09-22T21:05:03.678 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b5efbd924558782-00650e017f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.679 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:03.679 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:03.679 DEBUG:boto:path=/ffzdqu-8/?prefix=%3A 2023-09-22T21:05:03.679 DEBUG:boto:auth_path=/ffzdqu-8/?prefix=%3A 2023-09-22T21:05:03.679 DEBUG:boto:Method: GET 2023-09-22T21:05:03.679 DEBUG:boto:Path: /ffzdqu-8/?prefix=%3A 2023-09-22T21:05:03.679 DEBUG:boto:Data: 2023-09-22T21:05:03.679 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.679 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.679 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.680 DEBUG:boto:Params: {} 2023-09-22T21:05:03.680 DEBUG:boto:Token: None 2023-09-22T21:05:03.680 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/ 2023-09-22T21:05:03.680 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA= 2023-09-22T21:05:03.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA=', 'Content-Length': '0'} 2023-09-22T21:05:03.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009358ed749939a3db-00650e017f-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.684 DEBUG:boto:b'ffzdqu-8:1000false:2023-09-22T21:04:36.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:03.684 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:03.684 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:03.684 DEBUG:boto:path=/ffzdqu-8/?prefix=%3A 2023-09-22T21:05:03.684 DEBUG:boto:auth_path=/ffzdqu-8/?prefix=%3A 2023-09-22T21:05:03.685 DEBUG:boto:Method: GET 2023-09-22T21:05:03.685 DEBUG:boto:Path: /ffzdqu-8/?prefix=%3A 2023-09-22T21:05:03.685 DEBUG:boto:Data: 2023-09-22T21:05:03.685 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.685 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.685 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.685 DEBUG:boto:Params: {} 2023-09-22T21:05:03.685 DEBUG:boto:Token: None 2023-09-22T21:05:03.685 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/ 2023-09-22T21:05:03.685 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA= 2023-09-22T21:05:03.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA=', 'Content-Length': '0'} 2023-09-22T21:05:03.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010051e89c01bc3f9-00650e017f-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.689 DEBUG:boto:b'ffzdqu-8:1000false:2023-09-22T21:04:36.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:03.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-22T21:05:03.690 DEBUG:boto:path=/ffzdqu-8/_myobj 2023-09-22T21:05:03.690 DEBUG:boto:auth_path=/ffzdqu-8/_myobj 2023-09-22T21:05:03.690 DEBUG:boto:path=/ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.690 DEBUG:boto:auth_path=/ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.690 DEBUG:boto:Method: GET 2023-09-22T21:05:03.690 DEBUG:boto:Path: /ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.691 DEBUG:boto:Data: 2023-09-22T21:05:03.691 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.691 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.691 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.691 DEBUG:boto:Params: {} 2023-09-22T21:05:03.691 DEBUG:boto:Token: None 2023-09-22T21:05:03.691 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.692 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:55bw1oAhXBiOcHVpj1I/7cU9WXo= 2023-09-22T21:05:03.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:55bw1oAhXBiOcHVpj1I/7cU9WXo=', 'Content-Length': '0'} 2023-09-22T21:05:03.696 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a2465b3ae91e517-00650e017f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.697 DEBUG:boto:path=/ffzdqu-8/_myobj 2023-09-22T21:05:03.697 DEBUG:boto:auth_path=/ffzdqu-8/_myobj 2023-09-22T21:05:03.697 DEBUG:boto:path=/ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.697 DEBUG:boto:auth_path=/ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.697 DEBUG:boto:Method: GET 2023-09-22T21:05:03.697 DEBUG:boto:Path: /ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.697 DEBUG:boto:Data: 2023-09-22T21:05:03.697 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.697 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.698 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.698 DEBUG:boto:Params: {} 2023-09-22T21:05:03.698 DEBUG:boto:Token: None 2023-09-22T21:05:03.698 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.698 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:55bw1oAhXBiOcHVpj1I/7cU9WXo= 2023-09-22T21:05:03.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:55bw1oAhXBiOcHVpj1I/7cU9WXo=', 'Content-Length': '0'} 2023-09-22T21:05:03.700 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006117e4584ac2b545-00650e017f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.743 DEBUG:boto:path=/ffzdqu-8/_myobj 2023-09-22T21:05:03.743 DEBUG:boto:auth_path=/ffzdqu-8/_myobj 2023-09-22T21:05:03.743 DEBUG:boto:path=/ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.744 DEBUG:boto:auth_path=/ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.744 DEBUG:boto:Method: HEAD 2023-09-22T21:05:03.744 DEBUG:boto:Path: /ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.744 DEBUG:boto:Data: 2023-09-22T21:05:03.744 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.744 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.744 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.744 DEBUG:boto:Params: {} 2023-09-22T21:05:03.745 DEBUG:boto:Token: None 2023-09-22T21:05:03.745 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.745 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b2ZXbDbqBKaF1cnNB3nCG6o8l+I= 2023-09-22T21:05:03.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b2ZXbDbqBKaF1cnNB3nCG6o8l+I=', 'Content-Length': '0'} 2023-09-22T21:05:03.748 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004dce7a79a65cea06-00650e017f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.748 DEBUG:boto:path=/ffzdqu-8/_myobj 2023-09-22T21:05:03.748 DEBUG:boto:auth_path=/ffzdqu-8/_myobj 2023-09-22T21:05:03.748 DEBUG:boto:path=/ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.749 DEBUG:boto:auth_path=/ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.749 DEBUG:boto:Method: HEAD 2023-09-22T21:05:03.749 DEBUG:boto:Path: /ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.749 DEBUG:boto:Data: 2023-09-22T21:05:03.749 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.749 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.749 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.749 DEBUG:boto:Params: {} 2023-09-22T21:05:03.749 DEBUG:boto:Token: None 2023-09-22T21:05:03.749 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.750 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b2ZXbDbqBKaF1cnNB3nCG6o8l+I= 2023-09-22T21:05:03.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b2ZXbDbqBKaF1cnNB3nCG6o8l+I=', 'Content-Length': '0'} 2023-09-22T21:05:03.752 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e85c4213e07f1642-00650e017f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-22T21:05:03.753 DEBUG:boto:path=/ffzdqu-8/_myobj 2023-09-22T21:05:03.753 DEBUG:boto:auth_path=/ffzdqu-8/_myobj 2023-09-22T21:05:03.753 DEBUG:boto:path=/ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.753 DEBUG:boto:auth_path=/ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.753 DEBUG:boto:Method: GET 2023-09-22T21:05:03.753 DEBUG:boto:Path: /ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.753 DEBUG:boto:Data: 2023-09-22T21:05:03.753 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.754 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.754 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.754 DEBUG:boto:Params: {} 2023-09-22T21:05:03.754 DEBUG:boto:Token: None 2023-09-22T21:05:03.754 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.754 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:55bw1oAhXBiOcHVpj1I/7cU9WXo= 2023-09-22T21:05:03.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:55bw1oAhXBiOcHVpj1I/7cU9WXo=', 'Content-Length': '0'} 2023-09-22T21:05:03.757 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d903c9602b28067c-00650e017f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.802 DEBUG:boto:path=/ffzdqu-8/_myobj 2023-09-22T21:05:03.802 DEBUG:boto:auth_path=/ffzdqu-8/_myobj 2023-09-22T21:05:03.803 DEBUG:boto:path=/ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.803 DEBUG:boto:auth_path=/ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.803 DEBUG:boto:Method: GET 2023-09-22T21:05:03.803 DEBUG:boto:Path: /ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.803 DEBUG:boto:Data: 2023-09-22T21:05:03.803 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.803 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.804 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.804 DEBUG:boto:Params: {} 2023-09-22T21:05:03.804 DEBUG:boto:Token: None 2023-09-22T21:05:03.804 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/_myobj?versionId=null 2023-09-22T21:05:03.804 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:55bw1oAhXBiOcHVpj1I/7cU9WXo= 2023-09-22T21:05:03.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:55bw1oAhXBiOcHVpj1I/7cU9WXo=', 'Content-Length': '0'} 2023-09-22T21:05:03.807 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000946a9cbbf507e61f-00650e017f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.807 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:03.807 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:03.808 DEBUG:boto:path=/ffzdqu-8/?prefix=_myobj 2023-09-22T21:05:03.808 DEBUG:boto:auth_path=/ffzdqu-8/?prefix=_myobj 2023-09-22T21:05:03.808 DEBUG:boto:Method: GET 2023-09-22T21:05:03.808 DEBUG:boto:Path: /ffzdqu-8/?prefix=_myobj 2023-09-22T21:05:03.808 DEBUG:boto:Data: 2023-09-22T21:05:03.808 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.808 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.808 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.809 DEBUG:boto:Params: {} 2023-09-22T21:05:03.809 DEBUG:boto:Token: None 2023-09-22T21:05:03.809 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/ 2023-09-22T21:05:03.809 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA= 2023-09-22T21:05:03.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA=', 'Content-Length': '0'} 2023-09-22T21:05:03.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000645511848890104-00650e017f-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.813 DEBUG:boto:b'ffzdqu-8_myobj1000false_myobj2023-09-22T21:04:36.708Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:03.814 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:03.814 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:03.814 DEBUG:boto:path=/ffzdqu-8/?prefix=_myobj 2023-09-22T21:05:03.814 DEBUG:boto:auth_path=/ffzdqu-8/?prefix=_myobj 2023-09-22T21:05:03.815 DEBUG:boto:Method: GET 2023-09-22T21:05:03.815 DEBUG:boto:Path: /ffzdqu-8/?prefix=_myobj 2023-09-22T21:05:03.815 DEBUG:boto:Data: 2023-09-22T21:05:03.815 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.815 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.815 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.815 DEBUG:boto:Params: {} 2023-09-22T21:05:03.815 DEBUG:boto:Token: None 2023-09-22T21:05:03.816 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/ 2023-09-22T21:05:03.816 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA= 2023-09-22T21:05:03.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA=', 'Content-Length': '0'} 2023-09-22T21:05:03.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000942d3ea07c229f6d-00650e017f-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.820 DEBUG:boto:b'ffzdqu-8_myobj1000false_myobj2023-09-22T21:04:36.708Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:03.821 DEBUG:boto:path=/ffzdqu-8/_myobj 2023-09-22T21:05:03.821 DEBUG:boto:auth_path=/ffzdqu-8/_myobj 2023-09-22T21:05:03.821 DEBUG:boto:Method: HEAD 2023-09-22T21:05:03.821 DEBUG:boto:Path: /ffzdqu-8/_myobj 2023-09-22T21:05:03.821 DEBUG:boto:Data: 2023-09-22T21:05:03.821 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.821 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.821 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.821 DEBUG:boto:Params: {} 2023-09-22T21:05:03.822 DEBUG:boto:Token: None 2023-09-22T21:05:03.822 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/_myobj 2023-09-22T21:05:03.822 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3UbCG6Fwit52wUjfXWAp680iBmM= 2023-09-22T21:05:03.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3UbCG6Fwit52wUjfXWAp680iBmM=', 'Content-Length': '0'} 2023-09-22T21:05:03.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005329f517028cf98c-00650e017f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.825 DEBUG:boto:path=/ffzdqu-8/_myobj 2023-09-22T21:05:03.825 DEBUG:boto:auth_path=/ffzdqu-8/_myobj 2023-09-22T21:05:03.825 DEBUG:boto:Method: HEAD 2023-09-22T21:05:03.825 DEBUG:boto:Path: /ffzdqu-8/_myobj 2023-09-22T21:05:03.825 DEBUG:boto:Data: 2023-09-22T21:05:03.825 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.826 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.826 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.826 DEBUG:boto:Params: {} 2023-09-22T21:05:03.826 DEBUG:boto:Token: None 2023-09-22T21:05:03.826 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/_myobj 2023-09-22T21:05:03.826 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3UbCG6Fwit52wUjfXWAp680iBmM= 2023-09-22T21:05:03.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3UbCG6Fwit52wUjfXWAp680iBmM=', 'Content-Length': '0'} 2023-09-22T21:05:03.829 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d96510818ec3cf6-00650e017f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-22T21:05:03.830 DEBUG:boto:path=/ffzdqu-8/_myobj 2023-09-22T21:05:03.830 DEBUG:boto:auth_path=/ffzdqu-8/_myobj 2023-09-22T21:05:03.830 DEBUG:boto:Method: GET 2023-09-22T21:05:03.830 DEBUG:boto:Path: /ffzdqu-8/_myobj 2023-09-22T21:05:03.830 DEBUG:boto:Data: 2023-09-22T21:05:03.831 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.831 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.831 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.831 DEBUG:boto:Params: {} 2023-09-22T21:05:03.831 DEBUG:boto:Token: None 2023-09-22T21:05:03.831 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/_myobj 2023-09-22T21:05:03.831 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i6sgRkdaYtxE5dqQYj4md4LgQCM= 2023-09-22T21:05:03.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i6sgRkdaYtxE5dqQYj4md4LgQCM=', 'Content-Length': '0'} 2023-09-22T21:05:03.834 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038db5fa4315b90ab-00650e017f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.834 DEBUG:boto:path=/ffzdqu-8/_myobj 2023-09-22T21:05:03.835 DEBUG:boto:auth_path=/ffzdqu-8/_myobj 2023-09-22T21:05:03.835 DEBUG:boto:Method: GET 2023-09-22T21:05:03.835 DEBUG:boto:Path: /ffzdqu-8/_myobj 2023-09-22T21:05:03.835 DEBUG:boto:Data: 2023-09-22T21:05:03.835 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.835 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.835 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.835 DEBUG:boto:Params: {} 2023-09-22T21:05:03.835 DEBUG:boto:Token: None 2023-09-22T21:05:03.836 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/_myobj 2023-09-22T21:05:03.836 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i6sgRkdaYtxE5dqQYj4md4LgQCM= 2023-09-22T21:05:03.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i6sgRkdaYtxE5dqQYj4md4LgQCM=', 'Content-Length': '0'} 2023-09-22T21:05:03.839 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4d08bc03bcf51ed-00650e017f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.882 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:03.883 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:03.883 DEBUG:boto:path=/ffzdqu-8/?prefix=_myobj 2023-09-22T21:05:03.883 DEBUG:boto:auth_path=/ffzdqu-8/?prefix=_myobj 2023-09-22T21:05:03.883 DEBUG:boto:Method: GET 2023-09-22T21:05:03.883 DEBUG:boto:Path: /ffzdqu-8/?prefix=_myobj 2023-09-22T21:05:03.883 DEBUG:boto:Data: 2023-09-22T21:05:03.883 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.883 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.883 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.883 DEBUG:boto:Params: {} 2023-09-22T21:05:03.884 DEBUG:boto:Token: None 2023-09-22T21:05:03.884 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/ 2023-09-22T21:05:03.884 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA= 2023-09-22T21:05:03.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA=', 'Content-Length': '0'} 2023-09-22T21:05:03.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031c6b9a794731745-00650e017f-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.889 DEBUG:boto:b'ffzdqu-8_myobj1000false_myobj2023-09-22T21:04:36.708Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:03.889 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:03.889 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:03.890 DEBUG:boto:path=/ffzdqu-8/?prefix=_myobj 2023-09-22T21:05:03.890 DEBUG:boto:auth_path=/ffzdqu-8/?prefix=_myobj 2023-09-22T21:05:03.890 DEBUG:boto:Method: GET 2023-09-22T21:05:03.890 DEBUG:boto:Path: /ffzdqu-8/?prefix=_myobj 2023-09-22T21:05:03.890 DEBUG:boto:Data: 2023-09-22T21:05:03.890 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.890 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.890 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.890 DEBUG:boto:Params: {} 2023-09-22T21:05:03.890 DEBUG:boto:Token: None 2023-09-22T21:05:03.891 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/ 2023-09-22T21:05:03.891 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA= 2023-09-22T21:05:03.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNq+Wv3evnKGl/d5aTbY8ToylYA=', 'Content-Length': '0'} 2023-09-22T21:05:03.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000769954bc3eb0b5e2-00650e017f-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.896 DEBUG:boto:b'ffzdqu-8_myobj1000false_myobj2023-09-22T21:04:36.708Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:03.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-22T21:05:03.897 DEBUG:boto:path=/ffzdqu-8/myobj 2023-09-22T21:05:03.897 DEBUG:boto:auth_path=/ffzdqu-8/myobj 2023-09-22T21:05:03.897 DEBUG:boto:path=/ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.897 DEBUG:boto:auth_path=/ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.897 DEBUG:boto:Method: GET 2023-09-22T21:05:03.897 DEBUG:boto:Path: /ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.898 DEBUG:boto:Data: 2023-09-22T21:05:03.898 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.898 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.898 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.898 DEBUG:boto:Params: {} 2023-09-22T21:05:03.898 DEBUG:boto:Token: None 2023-09-22T21:05:03.899 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.899 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VTkA9pCtbF+3g2zCD6aUzs9gEr0= 2023-09-22T21:05:03.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VTkA9pCtbF+3g2zCD6aUzs9gEr0=', 'Content-Length': '0'} 2023-09-22T21:05:03.902 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078b690ddd0e93e3c-00650e017f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.946 DEBUG:boto:path=/ffzdqu-8/myobj 2023-09-22T21:05:03.947 DEBUG:boto:auth_path=/ffzdqu-8/myobj 2023-09-22T21:05:03.947 DEBUG:boto:path=/ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.947 DEBUG:boto:auth_path=/ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.947 DEBUG:boto:Method: GET 2023-09-22T21:05:03.947 DEBUG:boto:Path: /ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.948 DEBUG:boto:Data: 2023-09-22T21:05:03.948 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.948 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.948 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.948 DEBUG:boto:Params: {} 2023-09-22T21:05:03.948 DEBUG:boto:Token: None 2023-09-22T21:05:03.948 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.949 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VTkA9pCtbF+3g2zCD6aUzs9gEr0= 2023-09-22T21:05:03.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VTkA9pCtbF+3g2zCD6aUzs9gEr0=', 'Content-Length': '0'} 2023-09-22T21:05:03.951 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024e07e8689a40f80-00650e017f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.952 DEBUG:boto:path=/ffzdqu-8/myobj 2023-09-22T21:05:03.952 DEBUG:boto:auth_path=/ffzdqu-8/myobj 2023-09-22T21:05:03.952 DEBUG:boto:path=/ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.952 DEBUG:boto:auth_path=/ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.952 DEBUG:boto:Method: HEAD 2023-09-22T21:05:03.953 DEBUG:boto:Path: /ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.953 DEBUG:boto:Data: 2023-09-22T21:05:03.953 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.953 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.953 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.953 DEBUG:boto:Params: {} 2023-09-22T21:05:03.954 DEBUG:boto:Token: None 2023-09-22T21:05:03.954 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.954 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JKBXiop6CEN8tkamZM4e5YcXY3g= 2023-09-22T21:05:03.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JKBXiop6CEN8tkamZM4e5YcXY3g=', 'Content-Length': '0'} 2023-09-22T21:05:03.957 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b0a7f7e304900a4-00650e017f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.957 DEBUG:boto:path=/ffzdqu-8/myobj 2023-09-22T21:05:03.958 DEBUG:boto:auth_path=/ffzdqu-8/myobj 2023-09-22T21:05:03.958 DEBUG:boto:path=/ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.958 DEBUG:boto:auth_path=/ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.958 DEBUG:boto:Method: HEAD 2023-09-22T21:05:03.958 DEBUG:boto:Path: /ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.958 DEBUG:boto:Data: 2023-09-22T21:05:03.958 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.958 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.959 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.959 DEBUG:boto:Params: {} 2023-09-22T21:05:03.959 DEBUG:boto:Token: None 2023-09-22T21:05:03.959 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.959 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JKBXiop6CEN8tkamZM4e5YcXY3g= 2023-09-22T21:05:03.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JKBXiop6CEN8tkamZM4e5YcXY3g=', 'Content-Length': '0'} 2023-09-22T21:05:03.962 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b7fc2ca339e0705-00650e017f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-22T21:05:03.963 DEBUG:boto:path=/ffzdqu-8/myobj 2023-09-22T21:05:03.963 DEBUG:boto:auth_path=/ffzdqu-8/myobj 2023-09-22T21:05:03.963 DEBUG:boto:path=/ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.963 DEBUG:boto:auth_path=/ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.963 DEBUG:boto:Method: GET 2023-09-22T21:05:03.963 DEBUG:boto:Path: /ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.963 DEBUG:boto:Data: 2023-09-22T21:05:03.963 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.963 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:03.963 DEBUG:boto:Port: 8000 2023-09-22T21:05:03.964 DEBUG:boto:Params: {} 2023-09-22T21:05:03.964 DEBUG:boto:Token: None 2023-09-22T21:05:03.964 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.964 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VTkA9pCtbF+3g2zCD6aUzs9gEr0= 2023-09-22T21:05:03.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VTkA9pCtbF+3g2zCD6aUzs9gEr0=', 'Content-Length': '0'} 2023-09-22T21:05:03.967 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d8e89cbbbb7fb68-00650e017f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:03.967 DEBUG:boto:path=/ffzdqu-8/myobj 2023-09-22T21:05:03.967 DEBUG:boto:auth_path=/ffzdqu-8/myobj 2023-09-22T21:05:03.967 DEBUG:boto:path=/ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.968 DEBUG:boto:auth_path=/ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.968 DEBUG:boto:Method: GET 2023-09-22T21:05:03.968 DEBUG:boto:Path: /ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.968 DEBUG:boto:Data: 2023-09-22T21:05:03.968 DEBUG:boto:Headers: {} 2023-09-22T21:05:03.968 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:03.968 DEBUG:boto:Port: 8001 2023-09-22T21:05:03.968 DEBUG:boto:Params: {} 2023-09-22T21:05:03.968 DEBUG:boto:Token: None 2023-09-22T21:05:03.969 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:03 GMT /ffzdqu-8/myobj?versionId=null 2023-09-22T21:05:03.969 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VTkA9pCtbF+3g2zCD6aUzs9gEr0= 2023-09-22T21:05:03.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VTkA9pCtbF+3g2zCD6aUzs9gEr0=', 'Content-Length': '0'} 2023-09-22T21:05:03.971 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c86c259546817a23-00650e017f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:04.014 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:04.015 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:04.015 DEBUG:boto:path=/ffzdqu-8/?prefix=myobj 2023-09-22T21:05:04.015 DEBUG:boto:auth_path=/ffzdqu-8/?prefix=myobj 2023-09-22T21:05:04.015 DEBUG:boto:Method: GET 2023-09-22T21:05:04.015 DEBUG:boto:Path: /ffzdqu-8/?prefix=myobj 2023-09-22T21:05:04.015 DEBUG:boto:Data: 2023-09-22T21:05:04.015 DEBUG:boto:Headers: {} 2023-09-22T21:05:04.015 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:04.015 DEBUG:boto:Port: 8000 2023-09-22T21:05:04.015 DEBUG:boto:Params: {} 2023-09-22T21:05:04.016 DEBUG:boto:Token: None 2023-09-22T21:05:04.016 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:04 GMT /ffzdqu-8/ 2023-09-22T21:05:04.016 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Gh2TKrRj8iK3DyVWYcqIfbkvTKQ= 2023-09-22T21:05:04.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Gh2TKrRj8iK3DyVWYcqIfbkvTKQ=', 'Content-Length': '0'} 2023-09-22T21:05:04.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f18d055efca0578-00650e0180-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:04.021 DEBUG:boto:b'ffzdqu-8myobj1000falsemyobj2023-09-22T21:04:36.683Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:04.021 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:04.022 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:04.022 DEBUG:boto:path=/ffzdqu-8/?prefix=myobj 2023-09-22T21:05:04.022 DEBUG:boto:auth_path=/ffzdqu-8/?prefix=myobj 2023-09-22T21:05:04.022 DEBUG:boto:Method: GET 2023-09-22T21:05:04.022 DEBUG:boto:Path: /ffzdqu-8/?prefix=myobj 2023-09-22T21:05:04.022 DEBUG:boto:Data: 2023-09-22T21:05:04.022 DEBUG:boto:Headers: {} 2023-09-22T21:05:04.022 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:04.022 DEBUG:boto:Port: 8001 2023-09-22T21:05:04.022 DEBUG:boto:Params: {} 2023-09-22T21:05:04.023 DEBUG:boto:Token: None 2023-09-22T21:05:04.023 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:04 GMT /ffzdqu-8/ 2023-09-22T21:05:04.023 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Gh2TKrRj8iK3DyVWYcqIfbkvTKQ= 2023-09-22T21:05:04.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Gh2TKrRj8iK3DyVWYcqIfbkvTKQ=', 'Content-Length': '0'} 2023-09-22T21:05:04.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000801ecf484cfc11c5-00650e0180-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:04.027 DEBUG:boto:b'ffzdqu-8myobj1000falsemyobj2023-09-22T21:04:36.683Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:04.027 DEBUG:boto:path=/ffzdqu-8/myobj 2023-09-22T21:05:04.028 DEBUG:boto:auth_path=/ffzdqu-8/myobj 2023-09-22T21:05:04.028 DEBUG:boto:Method: HEAD 2023-09-22T21:05:04.028 DEBUG:boto:Path: /ffzdqu-8/myobj 2023-09-22T21:05:04.028 DEBUG:boto:Data: 2023-09-22T21:05:04.028 DEBUG:boto:Headers: {} 2023-09-22T21:05:04.028 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:04.028 DEBUG:boto:Port: 8000 2023-09-22T21:05:04.028 DEBUG:boto:Params: {} 2023-09-22T21:05:04.028 DEBUG:boto:Token: None 2023-09-22T21:05:04.029 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:04 GMT /ffzdqu-8/myobj 2023-09-22T21:05:04.029 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vAKwV1x3pHUjbpK3HHosVz8wLWc= 2023-09-22T21:05:04.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vAKwV1x3pHUjbpK3HHosVz8wLWc=', 'Content-Length': '0'} 2023-09-22T21:05:04.031 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bd6884853c2e9d9-00650e0180-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:04.032 DEBUG:boto:path=/ffzdqu-8/myobj 2023-09-22T21:05:04.032 DEBUG:boto:auth_path=/ffzdqu-8/myobj 2023-09-22T21:05:04.032 DEBUG:boto:Method: HEAD 2023-09-22T21:05:04.032 DEBUG:boto:Path: /ffzdqu-8/myobj 2023-09-22T21:05:04.032 DEBUG:boto:Data: 2023-09-22T21:05:04.032 DEBUG:boto:Headers: {} 2023-09-22T21:05:04.032 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:04.032 DEBUG:boto:Port: 8001 2023-09-22T21:05:04.032 DEBUG:boto:Params: {} 2023-09-22T21:05:04.032 DEBUG:boto:Token: None 2023-09-22T21:05:04.033 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:04 GMT /ffzdqu-8/myobj 2023-09-22T21:05:04.033 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vAKwV1x3pHUjbpK3HHosVz8wLWc= 2023-09-22T21:05:04.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vAKwV1x3pHUjbpK3HHosVz8wLWc=', 'Content-Length': '0'} 2023-09-22T21:05:04.035 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf2d23e1fc6c6a53-00650e0180-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:04.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-22T21:05:04.036 DEBUG:boto:path=/ffzdqu-8/myobj 2023-09-22T21:05:04.036 DEBUG:boto:auth_path=/ffzdqu-8/myobj 2023-09-22T21:05:04.036 DEBUG:boto:Method: GET 2023-09-22T21:05:04.036 DEBUG:boto:Path: /ffzdqu-8/myobj 2023-09-22T21:05:04.036 DEBUG:boto:Data: 2023-09-22T21:05:04.036 DEBUG:boto:Headers: {} 2023-09-22T21:05:04.036 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:04.036 DEBUG:boto:Port: 8000 2023-09-22T21:05:04.036 DEBUG:boto:Params: {} 2023-09-22T21:05:04.036 DEBUG:boto:Token: None 2023-09-22T21:05:04.037 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:04 GMT /ffzdqu-8/myobj 2023-09-22T21:05:04.037 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1coLx3l3mpxmxAPP0btbUzYbZqs= 2023-09-22T21:05:04.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1coLx3l3mpxmxAPP0btbUzYbZqs=', 'Content-Length': '0'} 2023-09-22T21:05:04.039 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000770ad07cf1d1fd7b-00650e0180-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:04.086 DEBUG:boto:path=/ffzdqu-8/myobj 2023-09-22T21:05:04.087 DEBUG:boto:auth_path=/ffzdqu-8/myobj 2023-09-22T21:05:04.087 DEBUG:boto:Method: GET 2023-09-22T21:05:04.087 DEBUG:boto:Path: /ffzdqu-8/myobj 2023-09-22T21:05:04.087 DEBUG:boto:Data: 2023-09-22T21:05:04.087 DEBUG:boto:Headers: {} 2023-09-22T21:05:04.087 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:04.087 DEBUG:boto:Port: 8001 2023-09-22T21:05:04.087 DEBUG:boto:Params: {} 2023-09-22T21:05:04.087 DEBUG:boto:Token: None 2023-09-22T21:05:04.088 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:04 GMT /ffzdqu-8/myobj 2023-09-22T21:05:04.088 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1coLx3l3mpxmxAPP0btbUzYbZqs= 2023-09-22T21:05:04.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1coLx3l3mpxmxAPP0btbUzYbZqs=', 'Content-Length': '0'} 2023-09-22T21:05:04.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:04:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006dd1e55ea70d61b4-00650e0180-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:04.091 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:04.091 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:04.092 DEBUG:boto:path=/ffzdqu-8/?prefix=myobj 2023-09-22T21:05:04.092 DEBUG:boto:auth_path=/ffzdqu-8/?prefix=myobj 2023-09-22T21:05:04.092 DEBUG:boto:Method: GET 2023-09-22T21:05:04.092 DEBUG:boto:Path: /ffzdqu-8/?prefix=myobj 2023-09-22T21:05:04.092 DEBUG:boto:Data: 2023-09-22T21:05:04.092 DEBUG:boto:Headers: {} 2023-09-22T21:05:04.092 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:04.092 DEBUG:boto:Port: 8000 2023-09-22T21:05:04.092 DEBUG:boto:Params: {} 2023-09-22T21:05:04.092 DEBUG:boto:Token: None 2023-09-22T21:05:04.093 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:04 GMT /ffzdqu-8/ 2023-09-22T21:05:04.093 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Gh2TKrRj8iK3DyVWYcqIfbkvTKQ= 2023-09-22T21:05:04.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Gh2TKrRj8iK3DyVWYcqIfbkvTKQ=', 'Content-Length': '0'} 2023-09-22T21:05:04.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fba69459fc14c447-00650e0180-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:04.098 DEBUG:boto:b'ffzdqu-8myobj1000falsemyobj2023-09-22T21:04:36.683Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:04.098 DEBUG:boto:path=/ffzdqu-8/ 2023-09-22T21:05:04.098 DEBUG:boto:auth_path=/ffzdqu-8/ 2023-09-22T21:05:04.098 DEBUG:boto:path=/ffzdqu-8/?prefix=myobj 2023-09-22T21:05:04.099 DEBUG:boto:auth_path=/ffzdqu-8/?prefix=myobj 2023-09-22T21:05:04.099 DEBUG:boto:Method: GET 2023-09-22T21:05:04.099 DEBUG:boto:Path: /ffzdqu-8/?prefix=myobj 2023-09-22T21:05:04.099 DEBUG:boto:Data: 2023-09-22T21:05:04.099 DEBUG:boto:Headers: {} 2023-09-22T21:05:04.099 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:04.099 DEBUG:boto:Port: 8001 2023-09-22T21:05:04.099 DEBUG:boto:Params: {} 2023-09-22T21:05:04.099 DEBUG:boto:Token: None 2023-09-22T21:05:04.100 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:04 GMT /ffzdqu-8/ 2023-09-22T21:05:04.100 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Gh2TKrRj8iK3DyVWYcqIfbkvTKQ= 2023-09-22T21:05:04.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Gh2TKrRj8iK3DyVWYcqIfbkvTKQ=', 'Content-Length': '0'} 2023-09-22T21:05:04.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca957bdf8d9c8d64-00650e0180-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:04.147 DEBUG:boto:b'ffzdqu-8myobj1000falsemyobj2023-09-22T21:04:36.683Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:04.147 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-8 zones={a1, a2} 2023-09-22T21:05:04.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2023-09-22T21:05:04.148 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-9 2023-09-22T21:05:04.149 DEBUG:boto:path=/ffzdqu-9/ 2023-09-22T21:05:04.149 DEBUG:boto:auth_path=/ffzdqu-9/ 2023-09-22T21:05:04.149 DEBUG:boto:Method: PUT 2023-09-22T21:05:04.149 DEBUG:boto:Path: /ffzdqu-9/ 2023-09-22T21:05:04.149 DEBUG:boto:Data: 2023-09-22T21:05:04.149 DEBUG:boto:Headers: {} 2023-09-22T21:05:04.149 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:04.149 DEBUG:boto:Port: 8000 2023-09-22T21:05:04.149 DEBUG:boto:Params: {} 2023-09-22T21:05:04.150 DEBUG:boto:Token: None 2023-09-22T21:05:04.150 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:05:04 GMT /ffzdqu-9/ 2023-09-22T21:05:04.150 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1Oj/ULlmkKBUVRAK1yKWkP1Jw+E= 2023-09-22T21:05:04.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1Oj/ULlmkKBUVRAK1yKWkP1Jw+E=', 'Content-Length': '0'} 2023-09-22T21:05:04.187 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000034b5e3f4503abb8a-00650e0180-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:04.187 INFO:rgw_multi.tests:create bucket zone=a2 name=ffzdqu-10 2023-09-22T21:05:04.187 DEBUG:boto:path=/ffzdqu-10/ 2023-09-22T21:05:04.188 DEBUG:boto:auth_path=/ffzdqu-10/ 2023-09-22T21:05:04.188 DEBUG:boto:Method: PUT 2023-09-22T21:05:04.188 DEBUG:boto:Path: /ffzdqu-10/ 2023-09-22T21:05:04.188 DEBUG:boto:Data: 2023-09-22T21:05:04.188 DEBUG:boto:Headers: {} 2023-09-22T21:05:04.188 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:04.188 DEBUG:boto:Port: 8001 2023-09-22T21:05:04.188 DEBUG:boto:Params: {} 2023-09-22T21:05:04.188 DEBUG:boto:Token: None 2023-09-22T21:05:04.189 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:05:04 GMT /ffzdqu-10/ 2023-09-22T21:05:04.189 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FO05JNj7yBqFH1U3e0EDO1tkGvU= 2023-09-22T21:05:04.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FO05JNj7yBqFH1U3e0EDO1tkGvU=', 'Content-Length': '0'} 2023-09-22T21:05:04.256 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cb09ca0d333c6f82-00650e0180-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:04.256 DEBUG:boto:path=/ffzdqu-9/ 2023-09-22T21:05:04.257 DEBUG:boto:auth_path=/ffzdqu-9/ 2023-09-22T21:05:04.257 DEBUG:boto:Method: HEAD 2023-09-22T21:05:04.257 DEBUG:boto:Path: /ffzdqu-9/ 2023-09-22T21:05:04.257 DEBUG:boto:Data: 2023-09-22T21:05:04.257 DEBUG:boto:Headers: {} 2023-09-22T21:05:04.257 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:04.257 DEBUG:boto:Port: 8000 2023-09-22T21:05:04.257 DEBUG:boto:Params: {} 2023-09-22T21:05:04.257 DEBUG:boto:Token: None 2023-09-22T21:05:04.258 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:04 GMT /ffzdqu-9/ 2023-09-22T21:05:04.258 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Yl88hLHqHz7BU/pLgPTYaODwZKA= 2023-09-22T21:05:04.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Yl88hLHqHz7BU/pLgPTYaODwZKA=', 'Content-Length': '0'} 2023-09-22T21:05:04.265 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff316e5b8fc6d27e-00650e0180-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:04.266 DEBUG:boto:path=/ffzdqu-9/myobj 2023-09-22T21:05:04.266 DEBUG:boto:auth_path=/ffzdqu-9/myobj 2023-09-22T21:05:04.266 DEBUG:boto:Method: PUT 2023-09-22T21:05:04.266 DEBUG:boto:Path: /ffzdqu-9/myobj 2023-09-22T21:05:04.266 DEBUG:boto:Data: 2023-09-22T21:05:04.266 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:05:04.266 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:04.266 DEBUG:boto:Port: 8000 2023-09-22T21:05:04.267 DEBUG:boto:Params: {} 2023-09-22T21:05:04.267 DEBUG:boto:Token: None 2023-09-22T21:05:04.267 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:05:04 GMT /ffzdqu-9/myobj 2023-09-22T21:05:04.267 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m+yl0tNDFwfL/Va/jPKT2IAhxag= 2023-09-22T21:05:04.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:05:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m+yl0tNDFwfL/Va/jPKT2IAhxag='} 2023-09-22T21:05:04.283 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000018e4585e8b3a41ff-00650e0180-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:04.283 DEBUG:boto:path=/ffzdqu-10/ 2023-09-22T21:05:04.283 DEBUG:boto:auth_path=/ffzdqu-10/ 2023-09-22T21:05:04.283 DEBUG:boto:Method: HEAD 2023-09-22T21:05:04.283 DEBUG:boto:Path: /ffzdqu-10/ 2023-09-22T21:05:04.283 DEBUG:boto:Data: 2023-09-22T21:05:04.284 DEBUG:boto:Headers: {} 2023-09-22T21:05:04.284 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:04.284 DEBUG:boto:Port: 8001 2023-09-22T21:05:04.284 DEBUG:boto:Params: {} 2023-09-22T21:05:04.284 DEBUG:boto:Token: None 2023-09-22T21:05:04.284 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:04 GMT /ffzdqu-10/ 2023-09-22T21:05:04.284 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DpnqgxYo0BJpV29+qdTUnlWjXKg= 2023-09-22T21:05:04.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DpnqgxYo0BJpV29+qdTUnlWjXKg=', 'Content-Length': '0'} 2023-09-22T21:05:04.290 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002290851bf1e43332-00650e0180-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:04.291 DEBUG:boto:path=/ffzdqu-10/myobj 2023-09-22T21:05:04.291 DEBUG:boto:auth_path=/ffzdqu-10/myobj 2023-09-22T21:05:04.291 DEBUG:boto:Method: PUT 2023-09-22T21:05:04.291 DEBUG:boto:Path: /ffzdqu-10/myobj 2023-09-22T21:05:04.291 DEBUG:boto:Data: 2023-09-22T21:05:04.291 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:05:04.291 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:04.291 DEBUG:boto:Port: 8001 2023-09-22T21:05:04.292 DEBUG:boto:Params: {} 2023-09-22T21:05:04.292 DEBUG:boto:Token: None 2023-09-22T21:05:04.292 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:05:04 GMT /ffzdqu-10/myobj 2023-09-22T21:05:04.292 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:o4O+5IPc1rIC76ze7E+PxyTaIU4= 2023-09-22T21:05:04.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:05:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:o4O+5IPc1rIC76ze7E+PxyTaIU4='} 2023-09-22T21:05:04.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb2f4d7bd055a709-00650e0180-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:04.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:05:04.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:05:04.319 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:05:04.363 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:05:04.364 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:05:04.403 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:05:04.420 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695416704.230032_44.1","last_update":"2023-09-22T21:05:04.230032Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:05:04.421 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416704.230032_44.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:05:04.421 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:05:04.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:05:04.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:05:04.423 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:05:04.501 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:05:04.502 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:05:04.537 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:05:04.537+0000 7f17f376da40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:05:04.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:05:04.541+0000 7f17f376da40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:05:04.555 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695416704.230032_44.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.230032Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:05:04Z"} 2023-09-22T21:05:04.555 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695416704.230032_44.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.230032Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:05:04Z"} 2023-09-22T21:05:04.555 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695416704.230032_44.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.230032Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:05:04.555 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416704.230032_44.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:05:04.555 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416704.230032_44.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:05:04.555 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:05:04.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:05:04.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:05:04.556 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:05:04.649 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:05:04.649 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:05:04.693 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:05:04.693+0000 7f2ee3b19a40 1 waiting to reach incremental sync.. 2023-09-22T21:05:09.694 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:05:09.697+0000 7f2ee3b19a40 1 waiting to reach incremental sync.. 2023-09-22T21:05:14.695 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:05:14.697+0000 7f2ee3b19a40 1 waiting to reach incremental sync.. 2023-09-22T21:05:19.696 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:05:19.697+0000 7f2ee3b19a40 1 waiting to reach incremental sync.. 2023-09-22T21:05:24.699 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:05:24.701+0000 7f2ee3b19a40 1 bucket sync caught up with source: 2023-09-22T21:05:24.700 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2023-09-22T21:05:24.700 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.8.6, , , , , , , , , ] 2023-09-22T21:05:24.700 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:05:24.701+0000 7f2ee3b19a40 0 bucket checkpoint complete 2023-09-22T21:05:24.728 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-9 zones={a2, a1} 2023-09-22T21:05:24.728 DEBUG:boto:path=/ffzdqu-9/ 2023-09-22T21:05:24.729 DEBUG:boto:auth_path=/ffzdqu-9/ 2023-09-22T21:05:24.729 DEBUG:boto:Method: HEAD 2023-09-22T21:05:24.729 DEBUG:boto:Path: /ffzdqu-9/ 2023-09-22T21:05:24.729 DEBUG:boto:Data: 2023-09-22T21:05:24.729 DEBUG:boto:Headers: {} 2023-09-22T21:05:24.729 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:24.729 DEBUG:boto:Port: 8001 2023-09-22T21:05:24.729 DEBUG:boto:Params: {} 2023-09-22T21:05:24.730 DEBUG:boto:Token: None 2023-09-22T21:05:24.730 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:24 GMT /ffzdqu-9/ 2023-09-22T21:05:24.730 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sFmEWFIrUSG3EOtgwYlJq2bKMSg= 2023-09-22T21:05:24.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sFmEWFIrUSG3EOtgwYlJq2bKMSg=', 'Content-Length': '0'} 2023-09-22T21:05:24.734 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b67f0b80c39f1eac-00650e0194-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:05:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:24.734 DEBUG:boto:path=/ffzdqu-9/ 2023-09-22T21:05:24.734 DEBUG:boto:auth_path=/ffzdqu-9/ 2023-09-22T21:05:24.734 DEBUG:boto:Method: HEAD 2023-09-22T21:05:24.734 DEBUG:boto:Path: /ffzdqu-9/ 2023-09-22T21:05:24.734 DEBUG:boto:Data: 2023-09-22T21:05:24.735 DEBUG:boto:Headers: {} 2023-09-22T21:05:24.735 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:24.735 DEBUG:boto:Port: 8000 2023-09-22T21:05:24.735 DEBUG:boto:Params: {} 2023-09-22T21:05:24.735 DEBUG:boto:Token: None 2023-09-22T21:05:24.735 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:24 GMT /ffzdqu-9/ 2023-09-22T21:05:24.735 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sFmEWFIrUSG3EOtgwYlJq2bKMSg= 2023-09-22T21:05:24.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sFmEWFIrUSG3EOtgwYlJq2bKMSg=', 'Content-Length': '0'} 2023-09-22T21:05:24.739 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000fa9bb4ae1e7d665-00650e0194-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:05:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:24.740 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:05:24.740 DEBUG:boto:path=/ffzdqu-9/ 2023-09-22T21:05:24.740 DEBUG:boto:auth_path=/ffzdqu-9/ 2023-09-22T21:05:24.740 DEBUG:boto:path=/ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:24.740 DEBUG:boto:auth_path=/ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:24.740 DEBUG:boto:Method: GET 2023-09-22T21:05:24.740 DEBUG:boto:Path: /ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:24.741 DEBUG:boto:Data: 2023-09-22T21:05:24.741 DEBUG:boto:Headers: {} 2023-09-22T21:05:24.741 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:24.741 DEBUG:boto:Port: 8001 2023-09-22T21:05:24.741 DEBUG:boto:Params: {} 2023-09-22T21:05:24.741 DEBUG:boto:Token: None 2023-09-22T21:05:24.741 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:24 GMT /ffzdqu-9/?versions 2023-09-22T21:05:24.741 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ok954PG8ebMwCikB3SHBEcRWpd8= 2023-09-22T21:05:24.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ok954PG8ebMwCikB3SHBEcRWpd8=', 'Content-Length': '0'} 2023-09-22T21:05:24.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2b1a79a7021fb2d-00650e0194-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:24.745 DEBUG:boto:b'ffzdqu-9999falsemyobjnulltrue2023-09-22T21:05:04.274Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:24.745 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-22T21:05:24.746 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:05:24.746 DEBUG:boto:path=/ffzdqu-9/ 2023-09-22T21:05:24.746 DEBUG:boto:auth_path=/ffzdqu-9/ 2023-09-22T21:05:24.746 DEBUG:boto:path=/ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:24.746 DEBUG:boto:auth_path=/ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:24.746 DEBUG:boto:Method: GET 2023-09-22T21:05:24.746 DEBUG:boto:Path: /ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:24.746 DEBUG:boto:Data: 2023-09-22T21:05:24.746 DEBUG:boto:Headers: {} 2023-09-22T21:05:24.747 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:24.747 DEBUG:boto:Port: 8000 2023-09-22T21:05:24.747 DEBUG:boto:Params: {} 2023-09-22T21:05:24.747 DEBUG:boto:Token: None 2023-09-22T21:05:24.747 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:24 GMT /ffzdqu-9/?versions 2023-09-22T21:05:24.747 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ok954PG8ebMwCikB3SHBEcRWpd8= 2023-09-22T21:05:24.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ok954PG8ebMwCikB3SHBEcRWpd8=', 'Content-Length': '0'} 2023-09-22T21:05:24.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050455f4433ceae96-00650e0194-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:24.794 DEBUG:boto:b'ffzdqu-9999falsemyobjnulltrue2023-09-22T21:05:04.274Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:24.795 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-22T21:05:24.795 DEBUG:boto:path=/ffzdqu-9/ 2023-09-22T21:05:24.795 DEBUG:boto:auth_path=/ffzdqu-9/ 2023-09-22T21:05:24.795 DEBUG:boto:path=/ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:24.795 DEBUG:boto:auth_path=/ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:24.795 DEBUG:boto:Method: GET 2023-09-22T21:05:24.795 DEBUG:boto:Path: /ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:24.795 DEBUG:boto:Data: 2023-09-22T21:05:24.796 DEBUG:boto:Headers: {} 2023-09-22T21:05:24.796 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:24.796 DEBUG:boto:Port: 8001 2023-09-22T21:05:24.796 DEBUG:boto:Params: {} 2023-09-22T21:05:24.796 DEBUG:boto:Token: None 2023-09-22T21:05:24.796 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:24 GMT /ffzdqu-9/?versions 2023-09-22T21:05:24.796 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ok954PG8ebMwCikB3SHBEcRWpd8= 2023-09-22T21:05:24.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ok954PG8ebMwCikB3SHBEcRWpd8=', 'Content-Length': '0'} 2023-09-22T21:05:24.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1c835e537777cec-00650e0194-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:24.800 DEBUG:boto:b'ffzdqu-9999falsemyobjnulltrue2023-09-22T21:05:04.274Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:24.801 DEBUG:boto:path=/ffzdqu-9/ 2023-09-22T21:05:24.801 DEBUG:boto:auth_path=/ffzdqu-9/ 2023-09-22T21:05:24.801 DEBUG:boto:path=/ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:24.801 DEBUG:boto:auth_path=/ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:24.801 DEBUG:boto:Method: GET 2023-09-22T21:05:24.801 DEBUG:boto:Path: /ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:24.801 DEBUG:boto:Data: 2023-09-22T21:05:24.802 DEBUG:boto:Headers: {} 2023-09-22T21:05:24.802 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:24.802 DEBUG:boto:Port: 8000 2023-09-22T21:05:24.802 DEBUG:boto:Params: {} 2023-09-22T21:05:24.802 DEBUG:boto:Token: None 2023-09-22T21:05:24.802 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:24 GMT /ffzdqu-9/?versions 2023-09-22T21:05:24.802 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ok954PG8ebMwCikB3SHBEcRWpd8= 2023-09-22T21:05:24.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ok954PG8ebMwCikB3SHBEcRWpd8=', 'Content-Length': '0'} 2023-09-22T21:05:24.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a77ff8dae758b94e-00650e0194-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:24.807 DEBUG:boto:b'ffzdqu-9999falsemyobjnulltrue2023-09-22T21:05:04.274Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:24.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-22T21:05:24.807 DEBUG:boto:path=/ffzdqu-9/myobj 2023-09-22T21:05:24.807 DEBUG:boto:auth_path=/ffzdqu-9/myobj 2023-09-22T21:05:24.807 DEBUG:boto:path=/ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.807 DEBUG:boto:auth_path=/ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.808 DEBUG:boto:Method: GET 2023-09-22T21:05:24.808 DEBUG:boto:Path: /ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.808 DEBUG:boto:Data: 2023-09-22T21:05:24.808 DEBUG:boto:Headers: {} 2023-09-22T21:05:24.808 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:24.808 DEBUG:boto:Port: 8001 2023-09-22T21:05:24.808 DEBUG:boto:Params: {} 2023-09-22T21:05:24.808 DEBUG:boto:Token: None 2023-09-22T21:05:24.808 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:24 GMT /ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.809 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9AQmpG3PGEzqCr/tU6hVJraQchI= 2023-09-22T21:05:24.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9AQmpG3PGEzqCr/tU6hVJraQchI=', 'Content-Length': '0'} 2023-09-22T21:05:24.811 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000101fadeedf29e646-00650e0194-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:24.858 DEBUG:boto:path=/ffzdqu-9/myobj 2023-09-22T21:05:24.859 DEBUG:boto:auth_path=/ffzdqu-9/myobj 2023-09-22T21:05:24.859 DEBUG:boto:path=/ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.859 DEBUG:boto:auth_path=/ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.859 DEBUG:boto:Method: GET 2023-09-22T21:05:24.859 DEBUG:boto:Path: /ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.859 DEBUG:boto:Data: 2023-09-22T21:05:24.859 DEBUG:boto:Headers: {} 2023-09-22T21:05:24.859 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:24.859 DEBUG:boto:Port: 8000 2023-09-22T21:05:24.859 DEBUG:boto:Params: {} 2023-09-22T21:05:24.860 DEBUG:boto:Token: None 2023-09-22T21:05:24.860 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:24 GMT /ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.860 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9AQmpG3PGEzqCr/tU6hVJraQchI= 2023-09-22T21:05:24.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9AQmpG3PGEzqCr/tU6hVJraQchI=', 'Content-Length': '0'} 2023-09-22T21:05:24.863 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6f98fc8f8bb2a28-00650e0194-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:24.863 DEBUG:boto:path=/ffzdqu-9/myobj 2023-09-22T21:05:24.863 DEBUG:boto:auth_path=/ffzdqu-9/myobj 2023-09-22T21:05:24.864 DEBUG:boto:path=/ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.864 DEBUG:boto:auth_path=/ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.864 DEBUG:boto:Method: HEAD 2023-09-22T21:05:24.864 DEBUG:boto:Path: /ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.864 DEBUG:boto:Data: 2023-09-22T21:05:24.864 DEBUG:boto:Headers: {} 2023-09-22T21:05:24.864 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:24.864 DEBUG:boto:Port: 8001 2023-09-22T21:05:24.864 DEBUG:boto:Params: {} 2023-09-22T21:05:24.865 DEBUG:boto:Token: None 2023-09-22T21:05:24.865 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:24 GMT /ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.865 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lzdzTVDK1ctCkW9mxOtI85zod9E= 2023-09-22T21:05:24.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lzdzTVDK1ctCkW9mxOtI85zod9E=', 'Content-Length': '0'} 2023-09-22T21:05:24.867 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008dbda7580576c45e-00650e0194-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:24.868 DEBUG:boto:path=/ffzdqu-9/myobj 2023-09-22T21:05:24.868 DEBUG:boto:auth_path=/ffzdqu-9/myobj 2023-09-22T21:05:24.868 DEBUG:boto:path=/ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.868 DEBUG:boto:auth_path=/ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.868 DEBUG:boto:Method: HEAD 2023-09-22T21:05:24.868 DEBUG:boto:Path: /ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.869 DEBUG:boto:Data: 2023-09-22T21:05:24.869 DEBUG:boto:Headers: {} 2023-09-22T21:05:24.869 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:24.869 DEBUG:boto:Port: 8000 2023-09-22T21:05:24.869 DEBUG:boto:Params: {} 2023-09-22T21:05:24.869 DEBUG:boto:Token: None 2023-09-22T21:05:24.869 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:24 GMT /ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.869 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lzdzTVDK1ctCkW9mxOtI85zod9E= 2023-09-22T21:05:24.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lzdzTVDK1ctCkW9mxOtI85zod9E=', 'Content-Length': '0'} 2023-09-22T21:05:24.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000622a2193ef5ddc11-00650e0194-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:24.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-22T21:05:24.872 DEBUG:boto:path=/ffzdqu-9/myobj 2023-09-22T21:05:24.873 DEBUG:boto:auth_path=/ffzdqu-9/myobj 2023-09-22T21:05:24.873 DEBUG:boto:path=/ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.873 DEBUG:boto:auth_path=/ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.873 DEBUG:boto:Method: GET 2023-09-22T21:05:24.873 DEBUG:boto:Path: /ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.873 DEBUG:boto:Data: 2023-09-22T21:05:24.873 DEBUG:boto:Headers: {} 2023-09-22T21:05:24.873 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:24.873 DEBUG:boto:Port: 8001 2023-09-22T21:05:24.873 DEBUG:boto:Params: {} 2023-09-22T21:05:24.874 DEBUG:boto:Token: None 2023-09-22T21:05:24.874 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:24 GMT /ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.874 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9AQmpG3PGEzqCr/tU6hVJraQchI= 2023-09-22T21:05:24.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9AQmpG3PGEzqCr/tU6hVJraQchI=', 'Content-Length': '0'} 2023-09-22T21:05:24.877 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cae3597efc8a501-00650e0194-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:24.877 DEBUG:boto:path=/ffzdqu-9/myobj 2023-09-22T21:05:24.877 DEBUG:boto:auth_path=/ffzdqu-9/myobj 2023-09-22T21:05:24.877 DEBUG:boto:path=/ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.878 DEBUG:boto:auth_path=/ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.878 DEBUG:boto:Method: GET 2023-09-22T21:05:24.878 DEBUG:boto:Path: /ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.878 DEBUG:boto:Data: 2023-09-22T21:05:24.878 DEBUG:boto:Headers: {} 2023-09-22T21:05:24.878 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:24.878 DEBUG:boto:Port: 8000 2023-09-22T21:05:24.878 DEBUG:boto:Params: {} 2023-09-22T21:05:24.878 DEBUG:boto:Token: None 2023-09-22T21:05:24.879 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:24 GMT /ffzdqu-9/myobj?versionId=null 2023-09-22T21:05:24.879 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9AQmpG3PGEzqCr/tU6hVJraQchI= 2023-09-22T21:05:24.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9AQmpG3PGEzqCr/tU6hVJraQchI=', 'Content-Length': '0'} 2023-09-22T21:05:24.882 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000707c1aba858ee508-00650e0194-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:24.922 DEBUG:boto:path=/ffzdqu-9/ 2023-09-22T21:05:24.923 DEBUG:boto:auth_path=/ffzdqu-9/ 2023-09-22T21:05:24.923 DEBUG:boto:path=/ffzdqu-9/?prefix=myobj 2023-09-22T21:05:24.923 DEBUG:boto:auth_path=/ffzdqu-9/?prefix=myobj 2023-09-22T21:05:24.923 DEBUG:boto:Method: GET 2023-09-22T21:05:24.923 DEBUG:boto:Path: /ffzdqu-9/?prefix=myobj 2023-09-22T21:05:24.923 DEBUG:boto:Data: 2023-09-22T21:05:24.923 DEBUG:boto:Headers: {} 2023-09-22T21:05:24.923 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:24.923 DEBUG:boto:Port: 8001 2023-09-22T21:05:24.923 DEBUG:boto:Params: {} 2023-09-22T21:05:24.924 DEBUG:boto:Token: None 2023-09-22T21:05:24.924 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:24 GMT /ffzdqu-9/ 2023-09-22T21:05:24.924 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Z9P2BnMATiWPj/p2xCTm53IhXsI= 2023-09-22T21:05:24.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Z9P2BnMATiWPj/p2xCTm53IhXsI=', 'Content-Length': '0'} 2023-09-22T21:05:24.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee617522c7208bc5-00650e0194-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:24.929 DEBUG:boto:b'ffzdqu-9myobj1000falsemyobj2023-09-22T21:05:04.274Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:24.929 DEBUG:boto:path=/ffzdqu-9/ 2023-09-22T21:05:24.929 DEBUG:boto:auth_path=/ffzdqu-9/ 2023-09-22T21:05:24.930 DEBUG:boto:path=/ffzdqu-9/?prefix=myobj 2023-09-22T21:05:24.930 DEBUG:boto:auth_path=/ffzdqu-9/?prefix=myobj 2023-09-22T21:05:24.930 DEBUG:boto:Method: GET 2023-09-22T21:05:24.930 DEBUG:boto:Path: /ffzdqu-9/?prefix=myobj 2023-09-22T21:05:24.930 DEBUG:boto:Data: 2023-09-22T21:05:24.930 DEBUG:boto:Headers: {} 2023-09-22T21:05:24.930 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:24.930 DEBUG:boto:Port: 8000 2023-09-22T21:05:24.930 DEBUG:boto:Params: {} 2023-09-22T21:05:24.931 DEBUG:boto:Token: None 2023-09-22T21:05:24.931 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:24 GMT /ffzdqu-9/ 2023-09-22T21:05:24.931 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Z9P2BnMATiWPj/p2xCTm53IhXsI= 2023-09-22T21:05:24.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Z9P2BnMATiWPj/p2xCTm53IhXsI=', 'Content-Length': '0'} 2023-09-22T21:05:24.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3513fc5092196c7-00650e0194-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:24.936 DEBUG:boto:b'ffzdqu-9myobj1000falsemyobj2023-09-22T21:05:04.274Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:24.936 DEBUG:boto:path=/ffzdqu-9/myobj 2023-09-22T21:05:24.936 DEBUG:boto:auth_path=/ffzdqu-9/myobj 2023-09-22T21:05:24.936 DEBUG:boto:Method: HEAD 2023-09-22T21:05:24.937 DEBUG:boto:Path: /ffzdqu-9/myobj 2023-09-22T21:05:24.937 DEBUG:boto:Data: 2023-09-22T21:05:24.937 DEBUG:boto:Headers: {} 2023-09-22T21:05:24.937 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:24.937 DEBUG:boto:Port: 8001 2023-09-22T21:05:24.937 DEBUG:boto:Params: {} 2023-09-22T21:05:24.937 DEBUG:boto:Token: None 2023-09-22T21:05:24.937 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:24 GMT /ffzdqu-9/myobj 2023-09-22T21:05:24.937 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y604eEoJ7HYncObDl0d1clnmB0g= 2023-09-22T21:05:24.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y604eEoJ7HYncObDl0d1clnmB0g=', 'Content-Length': '0'} 2023-09-22T21:05:24.940 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c58768b563bc3e6b-00650e0194-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:24.940 DEBUG:boto:path=/ffzdqu-9/myobj 2023-09-22T21:05:24.940 DEBUG:boto:auth_path=/ffzdqu-9/myobj 2023-09-22T21:05:24.940 DEBUG:boto:Method: HEAD 2023-09-22T21:05:24.941 DEBUG:boto:Path: /ffzdqu-9/myobj 2023-09-22T21:05:24.941 DEBUG:boto:Data: 2023-09-22T21:05:24.941 DEBUG:boto:Headers: {} 2023-09-22T21:05:24.941 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:24.941 DEBUG:boto:Port: 8000 2023-09-22T21:05:24.941 DEBUG:boto:Params: {} 2023-09-22T21:05:24.941 DEBUG:boto:Token: None 2023-09-22T21:05:24.941 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:24 GMT /ffzdqu-9/myobj 2023-09-22T21:05:24.941 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y604eEoJ7HYncObDl0d1clnmB0g= 2023-09-22T21:05:24.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y604eEoJ7HYncObDl0d1clnmB0g=', 'Content-Length': '0'} 2023-09-22T21:05:24.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4eb8e3235de6d66-00650e0194-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:24.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-22T21:05:24.945 DEBUG:boto:path=/ffzdqu-9/myobj 2023-09-22T21:05:24.945 DEBUG:boto:auth_path=/ffzdqu-9/myobj 2023-09-22T21:05:24.945 DEBUG:boto:Method: GET 2023-09-22T21:05:24.945 DEBUG:boto:Path: /ffzdqu-9/myobj 2023-09-22T21:05:24.945 DEBUG:boto:Data: 2023-09-22T21:05:24.945 DEBUG:boto:Headers: {} 2023-09-22T21:05:24.945 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:24.945 DEBUG:boto:Port: 8001 2023-09-22T21:05:24.945 DEBUG:boto:Params: {} 2023-09-22T21:05:24.945 DEBUG:boto:Token: None 2023-09-22T21:05:24.946 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:24 GMT /ffzdqu-9/myobj 2023-09-22T21:05:24.946 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F/aZD0Cgh4LBKpmfB+rdDD2WUCM= 2023-09-22T21:05:24.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F/aZD0Cgh4LBKpmfB+rdDD2WUCM=', 'Content-Length': '0'} 2023-09-22T21:05:24.948 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000faf564f59d9ce4aa-00650e0194-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:24.990 DEBUG:boto:path=/ffzdqu-9/myobj 2023-09-22T21:05:24.991 DEBUG:boto:auth_path=/ffzdqu-9/myobj 2023-09-22T21:05:24.991 DEBUG:boto:Method: GET 2023-09-22T21:05:24.991 DEBUG:boto:Path: /ffzdqu-9/myobj 2023-09-22T21:05:24.991 DEBUG:boto:Data: 2023-09-22T21:05:24.991 DEBUG:boto:Headers: {} 2023-09-22T21:05:24.991 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:24.991 DEBUG:boto:Port: 8000 2023-09-22T21:05:24.991 DEBUG:boto:Params: {} 2023-09-22T21:05:24.991 DEBUG:boto:Token: None 2023-09-22T21:05:24.992 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:24 GMT /ffzdqu-9/myobj 2023-09-22T21:05:24.992 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F/aZD0Cgh4LBKpmfB+rdDD2WUCM= 2023-09-22T21:05:24.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F/aZD0Cgh4LBKpmfB+rdDD2WUCM=', 'Content-Length': '0'} 2023-09-22T21:05:24.995 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b947da60d3878e47-00650e0194-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:24.995 DEBUG:boto:path=/ffzdqu-9/ 2023-09-22T21:05:24.995 DEBUG:boto:auth_path=/ffzdqu-9/ 2023-09-22T21:05:24.995 DEBUG:boto:path=/ffzdqu-9/?prefix=myobj 2023-09-22T21:05:24.995 DEBUG:boto:auth_path=/ffzdqu-9/?prefix=myobj 2023-09-22T21:05:24.995 DEBUG:boto:Method: GET 2023-09-22T21:05:24.996 DEBUG:boto:Path: /ffzdqu-9/?prefix=myobj 2023-09-22T21:05:24.996 DEBUG:boto:Data: 2023-09-22T21:05:24.996 DEBUG:boto:Headers: {} 2023-09-22T21:05:24.996 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:24.996 DEBUG:boto:Port: 8001 2023-09-22T21:05:24.996 DEBUG:boto:Params: {} 2023-09-22T21:05:24.996 DEBUG:boto:Token: None 2023-09-22T21:05:24.996 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:24 GMT /ffzdqu-9/ 2023-09-22T21:05:24.996 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Z9P2BnMATiWPj/p2xCTm53IhXsI= 2023-09-22T21:05:24.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Z9P2BnMATiWPj/p2xCTm53IhXsI=', 'Content-Length': '0'} 2023-09-22T21:05:25.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b239ae5508ffd64-00650e0195-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.001 DEBUG:boto:b'ffzdqu-9myobj1000falsemyobj2023-09-22T21:05:04.274Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:25.002 DEBUG:boto:path=/ffzdqu-9/ 2023-09-22T21:05:25.002 DEBUG:boto:auth_path=/ffzdqu-9/ 2023-09-22T21:05:25.002 DEBUG:boto:path=/ffzdqu-9/?prefix=myobj 2023-09-22T21:05:25.002 DEBUG:boto:auth_path=/ffzdqu-9/?prefix=myobj 2023-09-22T21:05:25.002 DEBUG:boto:Method: GET 2023-09-22T21:05:25.002 DEBUG:boto:Path: /ffzdqu-9/?prefix=myobj 2023-09-22T21:05:25.002 DEBUG:boto:Data: 2023-09-22T21:05:25.003 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.003 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:25.003 DEBUG:boto:Port: 8000 2023-09-22T21:05:25.003 DEBUG:boto:Params: {} 2023-09-22T21:05:25.003 DEBUG:boto:Token: None 2023-09-22T21:05:25.003 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-9/ 2023-09-22T21:05:25.003 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3ZRrhLLsExoMF/fXwYjcu/xaHro= 2023-09-22T21:05:25.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3ZRrhLLsExoMF/fXwYjcu/xaHro=', 'Content-Length': '0'} 2023-09-22T21:05:25.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a07e18b76b1b7bb5-00650e0195-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.050 DEBUG:boto:b'ffzdqu-9myobj1000falsemyobj2023-09-22T21:05:04.274Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:25.051 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-9 zones={a2, a1} 2023-09-22T21:05:25.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:05:25.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:05:25.051 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:05:25.101 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:05:25.102 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:05:25.145 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:05:25.145+0000 7f2e291b1a40 1 bucket sync caught up with source: 2023-09-22T21:05:25.145 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2023-09-22T21:05:25.145 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.8.6, , , , , , , , , ] 2023-09-22T21:05:25.145 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:05:25.145+0000 7f2e291b1a40 0 bucket checkpoint complete 2023-09-22T21:05:25.161 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-10 zones={a1, a2} 2023-09-22T21:05:25.161 DEBUG:boto:path=/ffzdqu-10/ 2023-09-22T21:05:25.161 DEBUG:boto:auth_path=/ffzdqu-10/ 2023-09-22T21:05:25.161 DEBUG:boto:Method: HEAD 2023-09-22T21:05:25.161 DEBUG:boto:Path: /ffzdqu-10/ 2023-09-22T21:05:25.161 DEBUG:boto:Data: 2023-09-22T21:05:25.162 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.162 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:25.162 DEBUG:boto:Port: 8000 2023-09-22T21:05:25.162 DEBUG:boto:Params: {} 2023-09-22T21:05:25.162 DEBUG:boto:Token: None 2023-09-22T21:05:25.162 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-10/ 2023-09-22T21:05:25.162 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oqkPFEXPqcxbocy8sVtmzCzZFpA= 2023-09-22T21:05:25.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oqkPFEXPqcxbocy8sVtmzCzZFpA=', 'Content-Length': '0'} 2023-09-22T21:05:25.165 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036c545c947bd0439-00650e0195-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.165 DEBUG:boto:path=/ffzdqu-10/ 2023-09-22T21:05:25.165 DEBUG:boto:auth_path=/ffzdqu-10/ 2023-09-22T21:05:25.165 DEBUG:boto:Method: HEAD 2023-09-22T21:05:25.165 DEBUG:boto:Path: /ffzdqu-10/ 2023-09-22T21:05:25.165 DEBUG:boto:Data: 2023-09-22T21:05:25.165 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.166 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:25.166 DEBUG:boto:Port: 8001 2023-09-22T21:05:25.166 DEBUG:boto:Params: {} 2023-09-22T21:05:25.166 DEBUG:boto:Token: None 2023-09-22T21:05:25.166 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-10/ 2023-09-22T21:05:25.166 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oqkPFEXPqcxbocy8sVtmzCzZFpA= 2023-09-22T21:05:25.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oqkPFEXPqcxbocy8sVtmzCzZFpA=', 'Content-Length': '0'} 2023-09-22T21:05:25.168 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5503abe3743e1dc-00650e0195-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.168 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:05:25.169 DEBUG:boto:path=/ffzdqu-10/ 2023-09-22T21:05:25.169 DEBUG:boto:auth_path=/ffzdqu-10/ 2023-09-22T21:05:25.169 DEBUG:boto:path=/ffzdqu-10/?versions&max-keys=999 2023-09-22T21:05:25.169 DEBUG:boto:auth_path=/ffzdqu-10/?versions&max-keys=999 2023-09-22T21:05:25.169 DEBUG:boto:Method: GET 2023-09-22T21:05:25.169 DEBUG:boto:Path: /ffzdqu-10/?versions&max-keys=999 2023-09-22T21:05:25.169 DEBUG:boto:Data: 2023-09-22T21:05:25.169 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.169 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:25.170 DEBUG:boto:Port: 8000 2023-09-22T21:05:25.170 DEBUG:boto:Params: {} 2023-09-22T21:05:25.170 DEBUG:boto:Token: None 2023-09-22T21:05:25.170 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-10/?versions 2023-09-22T21:05:25.170 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JDZki0rL/HsSSaBd2dxjFCJeGdA= 2023-09-22T21:05:25.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JDZki0rL/HsSSaBd2dxjFCJeGdA=', 'Content-Length': '0'} 2023-09-22T21:05:25.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d15648b1aa43b5f-00650e0195-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.173 DEBUG:boto:b'ffzdqu-10999falsemyobjnulltrue2023-09-22T21:05:04.301Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:25.173 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-22T21:05:25.173 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:05:25.173 DEBUG:boto:path=/ffzdqu-10/ 2023-09-22T21:05:25.174 DEBUG:boto:auth_path=/ffzdqu-10/ 2023-09-22T21:05:25.174 DEBUG:boto:path=/ffzdqu-10/?versions&max-keys=999 2023-09-22T21:05:25.174 DEBUG:boto:auth_path=/ffzdqu-10/?versions&max-keys=999 2023-09-22T21:05:25.174 DEBUG:boto:Method: GET 2023-09-22T21:05:25.174 DEBUG:boto:Path: /ffzdqu-10/?versions&max-keys=999 2023-09-22T21:05:25.174 DEBUG:boto:Data: 2023-09-22T21:05:25.174 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.174 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:25.174 DEBUG:boto:Port: 8001 2023-09-22T21:05:25.175 DEBUG:boto:Params: {} 2023-09-22T21:05:25.175 DEBUG:boto:Token: None 2023-09-22T21:05:25.175 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-10/?versions 2023-09-22T21:05:25.175 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JDZki0rL/HsSSaBd2dxjFCJeGdA= 2023-09-22T21:05:25.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JDZki0rL/HsSSaBd2dxjFCJeGdA=', 'Content-Length': '0'} 2023-09-22T21:05:25.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d0de552425e4345-00650e0195-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.178 DEBUG:boto:b'ffzdqu-10999falsemyobjnulltrue2023-09-22T21:05:04.301Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:25.178 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-22T21:05:25.178 DEBUG:boto:path=/ffzdqu-10/ 2023-09-22T21:05:25.178 DEBUG:boto:auth_path=/ffzdqu-10/ 2023-09-22T21:05:25.178 DEBUG:boto:path=/ffzdqu-10/?versions&max-keys=999 2023-09-22T21:05:25.178 DEBUG:boto:auth_path=/ffzdqu-10/?versions&max-keys=999 2023-09-22T21:05:25.179 DEBUG:boto:Method: GET 2023-09-22T21:05:25.179 DEBUG:boto:Path: /ffzdqu-10/?versions&max-keys=999 2023-09-22T21:05:25.179 DEBUG:boto:Data: 2023-09-22T21:05:25.179 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.179 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:25.179 DEBUG:boto:Port: 8000 2023-09-22T21:05:25.179 DEBUG:boto:Params: {} 2023-09-22T21:05:25.179 DEBUG:boto:Token: None 2023-09-22T21:05:25.180 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-10/?versions 2023-09-22T21:05:25.180 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JDZki0rL/HsSSaBd2dxjFCJeGdA= 2023-09-22T21:05:25.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JDZki0rL/HsSSaBd2dxjFCJeGdA=', 'Content-Length': '0'} 2023-09-22T21:05:25.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000389eb40d14037021-00650e0195-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.182 DEBUG:boto:b'ffzdqu-10999falsemyobjnulltrue2023-09-22T21:05:04.301Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:25.182 DEBUG:boto:path=/ffzdqu-10/ 2023-09-22T21:05:25.182 DEBUG:boto:auth_path=/ffzdqu-10/ 2023-09-22T21:05:25.183 DEBUG:boto:path=/ffzdqu-10/?versions&max-keys=999 2023-09-22T21:05:25.183 DEBUG:boto:auth_path=/ffzdqu-10/?versions&max-keys=999 2023-09-22T21:05:25.183 DEBUG:boto:Method: GET 2023-09-22T21:05:25.183 DEBUG:boto:Path: /ffzdqu-10/?versions&max-keys=999 2023-09-22T21:05:25.183 DEBUG:boto:Data: 2023-09-22T21:05:25.183 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.183 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:25.183 DEBUG:boto:Port: 8001 2023-09-22T21:05:25.183 DEBUG:boto:Params: {} 2023-09-22T21:05:25.184 DEBUG:boto:Token: None 2023-09-22T21:05:25.184 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-10/?versions 2023-09-22T21:05:25.184 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JDZki0rL/HsSSaBd2dxjFCJeGdA= 2023-09-22T21:05:25.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JDZki0rL/HsSSaBd2dxjFCJeGdA=', 'Content-Length': '0'} 2023-09-22T21:05:25.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc4b7e2d2f66e20c-00650e0195-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.186 DEBUG:boto:b'ffzdqu-10999falsemyobjnulltrue2023-09-22T21:05:04.301Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:25.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-22T21:05:25.187 DEBUG:boto:path=/ffzdqu-10/myobj 2023-09-22T21:05:25.187 DEBUG:boto:auth_path=/ffzdqu-10/myobj 2023-09-22T21:05:25.187 DEBUG:boto:path=/ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.187 DEBUG:boto:auth_path=/ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.187 DEBUG:boto:Method: GET 2023-09-22T21:05:25.187 DEBUG:boto:Path: /ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.187 DEBUG:boto:Data: 2023-09-22T21:05:25.188 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.188 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:25.188 DEBUG:boto:Port: 8000 2023-09-22T21:05:25.188 DEBUG:boto:Params: {} 2023-09-22T21:05:25.188 DEBUG:boto:Token: None 2023-09-22T21:05:25.188 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.188 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vOBhMKrtlTMg/draOnhWM6c8+Zk= 2023-09-22T21:05:25.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vOBhMKrtlTMg/draOnhWM6c8+Zk=', 'Content-Length': '0'} 2023-09-22T21:05:25.190 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001039e625d1824b47-00650e0195-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.191 DEBUG:boto:path=/ffzdqu-10/myobj 2023-09-22T21:05:25.191 DEBUG:boto:auth_path=/ffzdqu-10/myobj 2023-09-22T21:05:25.191 DEBUG:boto:path=/ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.191 DEBUG:boto:auth_path=/ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.191 DEBUG:boto:Method: GET 2023-09-22T21:05:25.191 DEBUG:boto:Path: /ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.191 DEBUG:boto:Data: 2023-09-22T21:05:25.191 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.191 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:25.192 DEBUG:boto:Port: 8001 2023-09-22T21:05:25.192 DEBUG:boto:Params: {} 2023-09-22T21:05:25.192 DEBUG:boto:Token: None 2023-09-22T21:05:25.192 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.192 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vOBhMKrtlTMg/draOnhWM6c8+Zk= 2023-09-22T21:05:25.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vOBhMKrtlTMg/draOnhWM6c8+Zk=', 'Content-Length': '0'} 2023-09-22T21:05:25.194 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f43d6b1507907fbb-00650e0195-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.194 DEBUG:boto:path=/ffzdqu-10/myobj 2023-09-22T21:05:25.194 DEBUG:boto:auth_path=/ffzdqu-10/myobj 2023-09-22T21:05:25.194 DEBUG:boto:path=/ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.194 DEBUG:boto:auth_path=/ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.194 DEBUG:boto:Method: HEAD 2023-09-22T21:05:25.195 DEBUG:boto:Path: /ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.195 DEBUG:boto:Data: 2023-09-22T21:05:25.195 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.195 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:25.195 DEBUG:boto:Port: 8000 2023-09-22T21:05:25.195 DEBUG:boto:Params: {} 2023-09-22T21:05:25.195 DEBUG:boto:Token: None 2023-09-22T21:05:25.195 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.195 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3wtgtyMXRaNTkCH0az4f0G9wL0w= 2023-09-22T21:05:25.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3wtgtyMXRaNTkCH0az4f0G9wL0w=', 'Content-Length': '0'} 2023-09-22T21:05:25.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f56b0a16b71e7d85-00650e0195-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.197 DEBUG:boto:path=/ffzdqu-10/myobj 2023-09-22T21:05:25.197 DEBUG:boto:auth_path=/ffzdqu-10/myobj 2023-09-22T21:05:25.197 DEBUG:boto:path=/ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.198 DEBUG:boto:auth_path=/ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.198 DEBUG:boto:Method: HEAD 2023-09-22T21:05:25.198 DEBUG:boto:Path: /ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.198 DEBUG:boto:Data: 2023-09-22T21:05:25.198 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.198 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:25.198 DEBUG:boto:Port: 8001 2023-09-22T21:05:25.198 DEBUG:boto:Params: {} 2023-09-22T21:05:25.198 DEBUG:boto:Token: None 2023-09-22T21:05:25.199 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.199 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3wtgtyMXRaNTkCH0az4f0G9wL0w= 2023-09-22T21:05:25.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3wtgtyMXRaNTkCH0az4f0G9wL0w=', 'Content-Length': '0'} 2023-09-22T21:05:25.201 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e01d6af856418bb4-00650e0195-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-22T21:05:25.201 DEBUG:boto:path=/ffzdqu-10/myobj 2023-09-22T21:05:25.201 DEBUG:boto:auth_path=/ffzdqu-10/myobj 2023-09-22T21:05:25.201 DEBUG:boto:path=/ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.201 DEBUG:boto:auth_path=/ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.202 DEBUG:boto:Method: GET 2023-09-22T21:05:25.202 DEBUG:boto:Path: /ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.202 DEBUG:boto:Data: 2023-09-22T21:05:25.202 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.202 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:25.202 DEBUG:boto:Port: 8000 2023-09-22T21:05:25.202 DEBUG:boto:Params: {} 2023-09-22T21:05:25.202 DEBUG:boto:Token: None 2023-09-22T21:05:25.202 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.203 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vOBhMKrtlTMg/draOnhWM6c8+Zk= 2023-09-22T21:05:25.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vOBhMKrtlTMg/draOnhWM6c8+Zk=', 'Content-Length': '0'} 2023-09-22T21:05:25.205 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ca08191a8b3ebb0-00650e0195-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.205 DEBUG:boto:path=/ffzdqu-10/myobj 2023-09-22T21:05:25.205 DEBUG:boto:auth_path=/ffzdqu-10/myobj 2023-09-22T21:05:25.205 DEBUG:boto:path=/ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.205 DEBUG:boto:auth_path=/ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.205 DEBUG:boto:Method: GET 2023-09-22T21:05:25.205 DEBUG:boto:Path: /ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.205 DEBUG:boto:Data: 2023-09-22T21:05:25.206 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.206 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:25.206 DEBUG:boto:Port: 8001 2023-09-22T21:05:25.206 DEBUG:boto:Params: {} 2023-09-22T21:05:25.206 DEBUG:boto:Token: None 2023-09-22T21:05:25.206 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-10/myobj?versionId=null 2023-09-22T21:05:25.206 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vOBhMKrtlTMg/draOnhWM6c8+Zk= 2023-09-22T21:05:25.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vOBhMKrtlTMg/draOnhWM6c8+Zk=', 'Content-Length': '0'} 2023-09-22T21:05:25.208 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007aa17cec159d6b3-00650e0195-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.209 DEBUG:boto:path=/ffzdqu-10/ 2023-09-22T21:05:25.209 DEBUG:boto:auth_path=/ffzdqu-10/ 2023-09-22T21:05:25.209 DEBUG:boto:path=/ffzdqu-10/?prefix=myobj 2023-09-22T21:05:25.209 DEBUG:boto:auth_path=/ffzdqu-10/?prefix=myobj 2023-09-22T21:05:25.209 DEBUG:boto:Method: GET 2023-09-22T21:05:25.209 DEBUG:boto:Path: /ffzdqu-10/?prefix=myobj 2023-09-22T21:05:25.209 DEBUG:boto:Data: 2023-09-22T21:05:25.209 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.209 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:25.210 DEBUG:boto:Port: 8000 2023-09-22T21:05:25.210 DEBUG:boto:Params: {} 2023-09-22T21:05:25.210 DEBUG:boto:Token: None 2023-09-22T21:05:25.210 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-10/ 2023-09-22T21:05:25.210 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:88kBk5rMkpThTv3sRxc2UxP+SAU= 2023-09-22T21:05:25.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:88kBk5rMkpThTv3sRxc2UxP+SAU=', 'Content-Length': '0'} 2023-09-22T21:05:25.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc65a6701990ce85-00650e0195-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.213 DEBUG:boto:b'ffzdqu-10myobj1000falsemyobj2023-09-22T21:05:04.301Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:25.213 DEBUG:boto:path=/ffzdqu-10/ 2023-09-22T21:05:25.213 DEBUG:boto:auth_path=/ffzdqu-10/ 2023-09-22T21:05:25.213 DEBUG:boto:path=/ffzdqu-10/?prefix=myobj 2023-09-22T21:05:25.213 DEBUG:boto:auth_path=/ffzdqu-10/?prefix=myobj 2023-09-22T21:05:25.213 DEBUG:boto:Method: GET 2023-09-22T21:05:25.214 DEBUG:boto:Path: /ffzdqu-10/?prefix=myobj 2023-09-22T21:05:25.214 DEBUG:boto:Data: 2023-09-22T21:05:25.214 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.214 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:25.214 DEBUG:boto:Port: 8001 2023-09-22T21:05:25.214 DEBUG:boto:Params: {} 2023-09-22T21:05:25.214 DEBUG:boto:Token: None 2023-09-22T21:05:25.214 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-10/ 2023-09-22T21:05:25.215 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:88kBk5rMkpThTv3sRxc2UxP+SAU= 2023-09-22T21:05:25.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:88kBk5rMkpThTv3sRxc2UxP+SAU=', 'Content-Length': '0'} 2023-09-22T21:05:25.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e556d093b7e247b-00650e0195-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.218 DEBUG:boto:b'ffzdqu-10myobj1000falsemyobj2023-09-22T21:05:04.301Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:25.218 DEBUG:boto:path=/ffzdqu-10/myobj 2023-09-22T21:05:25.218 DEBUG:boto:auth_path=/ffzdqu-10/myobj 2023-09-22T21:05:25.218 DEBUG:boto:Method: HEAD 2023-09-22T21:05:25.218 DEBUG:boto:Path: /ffzdqu-10/myobj 2023-09-22T21:05:25.218 DEBUG:boto:Data: 2023-09-22T21:05:25.218 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.219 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:25.219 DEBUG:boto:Port: 8000 2023-09-22T21:05:25.219 DEBUG:boto:Params: {} 2023-09-22T21:05:25.219 DEBUG:boto:Token: None 2023-09-22T21:05:25.219 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-10/myobj 2023-09-22T21:05:25.219 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Pj+E+hnvvAUc3fM7N8MSSgi5buA= 2023-09-22T21:05:25.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Pj+E+hnvvAUc3fM7N8MSSgi5buA=', 'Content-Length': '0'} 2023-09-22T21:05:25.221 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058df365ae957a90f-00650e0195-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.221 DEBUG:boto:path=/ffzdqu-10/myobj 2023-09-22T21:05:25.221 DEBUG:boto:auth_path=/ffzdqu-10/myobj 2023-09-22T21:05:25.221 DEBUG:boto:Method: HEAD 2023-09-22T21:05:25.221 DEBUG:boto:Path: /ffzdqu-10/myobj 2023-09-22T21:05:25.222 DEBUG:boto:Data: 2023-09-22T21:05:25.222 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.222 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:25.222 DEBUG:boto:Port: 8001 2023-09-22T21:05:25.222 DEBUG:boto:Params: {} 2023-09-22T21:05:25.222 DEBUG:boto:Token: None 2023-09-22T21:05:25.222 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-10/myobj 2023-09-22T21:05:25.222 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Pj+E+hnvvAUc3fM7N8MSSgi5buA= 2023-09-22T21:05:25.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Pj+E+hnvvAUc3fM7N8MSSgi5buA=', 'Content-Length': '0'} 2023-09-22T21:05:25.224 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4012b5481857957-00650e0195-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-22T21:05:25.225 DEBUG:boto:path=/ffzdqu-10/myobj 2023-09-22T21:05:25.225 DEBUG:boto:auth_path=/ffzdqu-10/myobj 2023-09-22T21:05:25.225 DEBUG:boto:Method: GET 2023-09-22T21:05:25.225 DEBUG:boto:Path: /ffzdqu-10/myobj 2023-09-22T21:05:25.225 DEBUG:boto:Data: 2023-09-22T21:05:25.225 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.225 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:25.225 DEBUG:boto:Port: 8000 2023-09-22T21:05:25.225 DEBUG:boto:Params: {} 2023-09-22T21:05:25.226 DEBUG:boto:Token: None 2023-09-22T21:05:25.226 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-10/myobj 2023-09-22T21:05:25.226 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+u6hF0t/zO9R3TKZ2blfZWKlawU= 2023-09-22T21:05:25.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+u6hF0t/zO9R3TKZ2blfZWKlawU=', 'Content-Length': '0'} 2023-09-22T21:05:25.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9dd98705f1311fb-00650e0195-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.228 DEBUG:boto:path=/ffzdqu-10/myobj 2023-09-22T21:05:25.228 DEBUG:boto:auth_path=/ffzdqu-10/myobj 2023-09-22T21:05:25.228 DEBUG:boto:Method: GET 2023-09-22T21:05:25.228 DEBUG:boto:Path: /ffzdqu-10/myobj 2023-09-22T21:05:25.228 DEBUG:boto:Data: 2023-09-22T21:05:25.228 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.228 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:25.229 DEBUG:boto:Port: 8001 2023-09-22T21:05:25.229 DEBUG:boto:Params: {} 2023-09-22T21:05:25.229 DEBUG:boto:Token: None 2023-09-22T21:05:25.229 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-10/myobj 2023-09-22T21:05:25.229 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+u6hF0t/zO9R3TKZ2blfZWKlawU= 2023-09-22T21:05:25.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+u6hF0t/zO9R3TKZ2blfZWKlawU=', 'Content-Length': '0'} 2023-09-22T21:05:25.231 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000098819afc67c86a76-00650e0195-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.231 DEBUG:boto:path=/ffzdqu-10/ 2023-09-22T21:05:25.231 DEBUG:boto:auth_path=/ffzdqu-10/ 2023-09-22T21:05:25.231 DEBUG:boto:path=/ffzdqu-10/?prefix=myobj 2023-09-22T21:05:25.231 DEBUG:boto:auth_path=/ffzdqu-10/?prefix=myobj 2023-09-22T21:05:25.232 DEBUG:boto:Method: GET 2023-09-22T21:05:25.232 DEBUG:boto:Path: /ffzdqu-10/?prefix=myobj 2023-09-22T21:05:25.232 DEBUG:boto:Data: 2023-09-22T21:05:25.232 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.232 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:25.232 DEBUG:boto:Port: 8000 2023-09-22T21:05:25.232 DEBUG:boto:Params: {} 2023-09-22T21:05:25.232 DEBUG:boto:Token: None 2023-09-22T21:05:25.232 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-10/ 2023-09-22T21:05:25.233 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:88kBk5rMkpThTv3sRxc2UxP+SAU= 2023-09-22T21:05:25.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:88kBk5rMkpThTv3sRxc2UxP+SAU=', 'Content-Length': '0'} 2023-09-22T21:05:25.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a8d0110b44e0cbd-00650e0195-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.236 DEBUG:boto:b'ffzdqu-10myobj1000falsemyobj2023-09-22T21:05:04.301Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:25.236 DEBUG:boto:path=/ffzdqu-10/ 2023-09-22T21:05:25.236 DEBUG:boto:auth_path=/ffzdqu-10/ 2023-09-22T21:05:25.236 DEBUG:boto:path=/ffzdqu-10/?prefix=myobj 2023-09-22T21:05:25.236 DEBUG:boto:auth_path=/ffzdqu-10/?prefix=myobj 2023-09-22T21:05:25.237 DEBUG:boto:Method: GET 2023-09-22T21:05:25.237 DEBUG:boto:Path: /ffzdqu-10/?prefix=myobj 2023-09-22T21:05:25.237 DEBUG:boto:Data: 2023-09-22T21:05:25.237 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.237 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:25.237 DEBUG:boto:Port: 8001 2023-09-22T21:05:25.237 DEBUG:boto:Params: {} 2023-09-22T21:05:25.237 DEBUG:boto:Token: None 2023-09-22T21:05:25.237 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-10/ 2023-09-22T21:05:25.238 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:88kBk5rMkpThTv3sRxc2UxP+SAU= 2023-09-22T21:05:25.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:88kBk5rMkpThTv3sRxc2UxP+SAU=', 'Content-Length': '0'} 2023-09-22T21:05:25.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be34b10ef1d8aba0-00650e0195-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.282 DEBUG:boto:b'ffzdqu-10myobj1000falsemyobj2023-09-22T21:05:04.301Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:05:25.283 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-10 zones={a1, a2} 2023-09-22T21:05:25.283 DEBUG:boto:path=/ffzdqu-9/ 2023-09-22T21:05:25.283 DEBUG:boto:auth_path=/ffzdqu-9/ 2023-09-22T21:05:25.283 DEBUG:boto:Method: HEAD 2023-09-22T21:05:25.283 DEBUG:boto:Path: /ffzdqu-9/ 2023-09-22T21:05:25.284 DEBUG:boto:Data: 2023-09-22T21:05:25.284 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.284 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:25.284 DEBUG:boto:Port: 8000 2023-09-22T21:05:25.284 DEBUG:boto:Params: {} 2023-09-22T21:05:25.284 DEBUG:boto:Token: None 2023-09-22T21:05:25.284 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-9/ 2023-09-22T21:05:25.284 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PnxoV1J43kWru4/lmEfKs0jcxZY= 2023-09-22T21:05:25.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PnxoV1J43kWru4/lmEfKs0jcxZY=', 'Content-Length': '0'} 2023-09-22T21:05:25.288 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e247fa373ddc3323-00650e0195-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.288 DEBUG:boto:path=/ffzdqu-9/myobj 2023-09-22T21:05:25.289 DEBUG:boto:auth_path=/ffzdqu-9/myobj 2023-09-22T21:05:25.289 DEBUG:boto:Method: HEAD 2023-09-22T21:05:25.289 DEBUG:boto:Path: /ffzdqu-9/myobj 2023-09-22T21:05:25.289 DEBUG:boto:Data: 2023-09-22T21:05:25.289 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.289 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:25.289 DEBUG:boto:Port: 8000 2023-09-22T21:05:25.289 DEBUG:boto:Params: {} 2023-09-22T21:05:25.289 DEBUG:boto:Token: None 2023-09-22T21:05:25.290 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-9/myobj 2023-09-22T21:05:25.290 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kQ2WTmzOYb+/E16uyxbDpLjuRFI= 2023-09-22T21:05:25.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kQ2WTmzOYb+/E16uyxbDpLjuRFI=', 'Content-Length': '0'} 2023-09-22T21:05:25.292 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ddc105235f701ab6-00650e0195-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.293 DEBUG:boto:path=/ffzdqu-9/myobj 2023-09-22T21:05:25.293 DEBUG:boto:auth_path=/ffzdqu-9/myobj 2023-09-22T21:05:25.293 DEBUG:boto:Method: DELETE 2023-09-22T21:05:25.293 DEBUG:boto:Path: /ffzdqu-9/myobj 2023-09-22T21:05:25.293 DEBUG:boto:Data: 2023-09-22T21:05:25.293 DEBUG:boto:Headers: {} 2023-09-22T21:05:25.293 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:25.293 DEBUG:boto:Port: 8000 2023-09-22T21:05:25.293 DEBUG:boto:Params: {} 2023-09-22T21:05:25.294 DEBUG:boto:Token: None 2023-09-22T21:05:25.294 DEBUG:boto:StringToSign: DELETE Fri, 22 Sep 2023 21:05:25 GMT /ffzdqu-9/myobj 2023-09-22T21:05:25.294 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1+nISgaMiQHjn5S7x0yBMQOKW+s= 2023-09-22T21:05:25.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1+nISgaMiQHjn5S7x0yBMQOKW+s=', 'Content-Length': '0'} 2023-09-22T21:05:25.303 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a50de5e7d2aca1e8-00650e0195-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:25.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:05:25.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:05:25.304 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:05:25.352 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:05:25.352 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:05:25.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:05:25.397+0000 7f31996a8a40 1 waiting for incremental sync to catch up: 2023-09-22T21:05:25.394 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2023-09-22T21:05:25.394 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.10.6, , , , , , , , , ] 2023-09-22T21:05:30.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:05:30.397+0000 7f31996a8a40 1 waiting for incremental sync to catch up: 2023-09-22T21:05:30.397 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2023-09-22T21:05:30.397 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.10.6, , , , , , , , , ] 2023-09-22T21:05:35.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:05:35.401+0000 7f31996a8a40 1 waiting for incremental sync to catch up: 2023-09-22T21:05:35.398 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2023-09-22T21:05:35.398 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.10.6, , , , , , , , , ] 2023-09-22T21:05:40.400 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:05:40.401+0000 7f31996a8a40 1 waiting for incremental sync to catch up: 2023-09-22T21:05:40.400 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2023-09-22T21:05:40.401 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.10.6, , , , , , , , , ] 2023-09-22T21:05:45.402 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:05:45.404+0000 7f31996a8a40 1 bucket sync caught up with source: 2023-09-22T21:05:45.403 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.10.6, , , , , , , , , ] 2023-09-22T21:05:45.403 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.10.6, , , , , , , , , ] 2023-09-22T21:05:45.403 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:05:45.404+0000 7f31996a8a40 0 bucket checkpoint complete 2023-09-22T21:05:45.433 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-9 zones={a2, a1} 2023-09-22T21:05:45.433 DEBUG:boto:path=/ffzdqu-9/ 2023-09-22T21:05:45.433 DEBUG:boto:auth_path=/ffzdqu-9/ 2023-09-22T21:05:45.433 DEBUG:boto:Method: HEAD 2023-09-22T21:05:45.434 DEBUG:boto:Path: /ffzdqu-9/ 2023-09-22T21:05:45.434 DEBUG:boto:Data: 2023-09-22T21:05:45.434 DEBUG:boto:Headers: {} 2023-09-22T21:05:45.434 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:45.434 DEBUG:boto:Port: 8001 2023-09-22T21:05:45.434 DEBUG:boto:Params: {} 2023-09-22T21:05:45.435 DEBUG:boto:Token: None 2023-09-22T21:05:45.435 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:45 GMT /ffzdqu-9/ 2023-09-22T21:05:45.435 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Acvti+SkxIl4i7vBDwUTcgZjLMY= 2023-09-22T21:05:45.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Acvti+SkxIl4i7vBDwUTcgZjLMY=', 'Content-Length': '0'} 2023-09-22T21:05:45.438 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b66bc88662c792af-00650e01a9-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:45.439 DEBUG:boto:path=/ffzdqu-9/ 2023-09-22T21:05:45.439 DEBUG:boto:auth_path=/ffzdqu-9/ 2023-09-22T21:05:45.439 DEBUG:boto:Method: HEAD 2023-09-22T21:05:45.439 DEBUG:boto:Path: /ffzdqu-9/ 2023-09-22T21:05:45.439 DEBUG:boto:Data: 2023-09-22T21:05:45.439 DEBUG:boto:Headers: {} 2023-09-22T21:05:45.439 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:45.440 DEBUG:boto:Port: 8000 2023-09-22T21:05:45.440 DEBUG:boto:Params: {} 2023-09-22T21:05:45.440 DEBUG:boto:Token: None 2023-09-22T21:05:45.440 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:45 GMT /ffzdqu-9/ 2023-09-22T21:05:45.440 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Acvti+SkxIl4i7vBDwUTcgZjLMY= 2023-09-22T21:05:45.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Acvti+SkxIl4i7vBDwUTcgZjLMY=', 'Content-Length': '0'} 2023-09-22T21:05:45.443 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0ca42b31a9ca9dd-00650e01a9-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:45.443 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:05:45.443 DEBUG:boto:path=/ffzdqu-9/ 2023-09-22T21:05:45.444 DEBUG:boto:auth_path=/ffzdqu-9/ 2023-09-22T21:05:45.444 DEBUG:boto:path=/ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:45.444 DEBUG:boto:auth_path=/ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:45.444 DEBUG:boto:Method: GET 2023-09-22T21:05:45.444 DEBUG:boto:Path: /ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:45.444 DEBUG:boto:Data: 2023-09-22T21:05:45.444 DEBUG:boto:Headers: {} 2023-09-22T21:05:45.445 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:45.445 DEBUG:boto:Port: 8001 2023-09-22T21:05:45.445 DEBUG:boto:Params: {} 2023-09-22T21:05:45.445 DEBUG:boto:Token: None 2023-09-22T21:05:45.445 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:45 GMT /ffzdqu-9/?versions 2023-09-22T21:05:45.445 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kDFk/pTeJwLY1+Ce/OZHMEb8/ps= 2023-09-22T21:05:45.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kDFk/pTeJwLY1+Ce/OZHMEb8/ps=', 'Content-Length': '0'} 2023-09-22T21:05:45.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099b2cac86b75a0a7-00650e01a9-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:45.449 DEBUG:boto:b'ffzdqu-9999false' 2023-09-22T21:05:45.450 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:05:45.450 DEBUG:boto:path=/ffzdqu-9/ 2023-09-22T21:05:45.450 DEBUG:boto:auth_path=/ffzdqu-9/ 2023-09-22T21:05:45.450 DEBUG:boto:path=/ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:45.450 DEBUG:boto:auth_path=/ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:45.450 DEBUG:boto:Method: GET 2023-09-22T21:05:45.450 DEBUG:boto:Path: /ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:45.451 DEBUG:boto:Data: 2023-09-22T21:05:45.451 DEBUG:boto:Headers: {} 2023-09-22T21:05:45.451 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:45.451 DEBUG:boto:Port: 8000 2023-09-22T21:05:45.451 DEBUG:boto:Params: {} 2023-09-22T21:05:45.451 DEBUG:boto:Token: None 2023-09-22T21:05:45.451 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:45 GMT /ffzdqu-9/?versions 2023-09-22T21:05:45.451 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kDFk/pTeJwLY1+Ce/OZHMEb8/ps= 2023-09-22T21:05:45.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kDFk/pTeJwLY1+Ce/OZHMEb8/ps=', 'Content-Length': '0'} 2023-09-22T21:05:45.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6695a184391aef1-00650e01a9-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:45.498 DEBUG:boto:b'ffzdqu-9999false' 2023-09-22T21:05:45.499 DEBUG:boto:path=/ffzdqu-9/ 2023-09-22T21:05:45.499 DEBUG:boto:auth_path=/ffzdqu-9/ 2023-09-22T21:05:45.499 DEBUG:boto:path=/ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:45.499 DEBUG:boto:auth_path=/ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:45.499 DEBUG:boto:Method: GET 2023-09-22T21:05:45.500 DEBUG:boto:Path: /ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:45.500 DEBUG:boto:Data: 2023-09-22T21:05:45.500 DEBUG:boto:Headers: {} 2023-09-22T21:05:45.500 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:45.500 DEBUG:boto:Port: 8001 2023-09-22T21:05:45.500 DEBUG:boto:Params: {} 2023-09-22T21:05:45.500 DEBUG:boto:Token: None 2023-09-22T21:05:45.500 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:45 GMT /ffzdqu-9/?versions 2023-09-22T21:05:45.501 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kDFk/pTeJwLY1+Ce/OZHMEb8/ps= 2023-09-22T21:05:45.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kDFk/pTeJwLY1+Ce/OZHMEb8/ps=', 'Content-Length': '0'} 2023-09-22T21:05:45.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbfef7b7d0799136-00650e01a9-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:45.505 DEBUG:boto:b'ffzdqu-9999false' 2023-09-22T21:05:45.505 DEBUG:boto:path=/ffzdqu-9/ 2023-09-22T21:05:45.506 DEBUG:boto:auth_path=/ffzdqu-9/ 2023-09-22T21:05:45.506 DEBUG:boto:path=/ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:45.506 DEBUG:boto:auth_path=/ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:45.506 DEBUG:boto:Method: GET 2023-09-22T21:05:45.506 DEBUG:boto:Path: /ffzdqu-9/?versions&max-keys=999 2023-09-22T21:05:45.506 DEBUG:boto:Data: 2023-09-22T21:05:45.506 DEBUG:boto:Headers: {} 2023-09-22T21:05:45.506 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:05:45.506 DEBUG:boto:Port: 8000 2023-09-22T21:05:45.507 DEBUG:boto:Params: {} 2023-09-22T21:05:45.507 DEBUG:boto:Token: None 2023-09-22T21:05:45.507 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:05:45 GMT /ffzdqu-9/?versions 2023-09-22T21:05:45.507 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kDFk/pTeJwLY1+Ce/OZHMEb8/ps= 2023-09-22T21:05:45.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kDFk/pTeJwLY1+Ce/OZHMEb8/ps=', 'Content-Length': '0'} 2023-09-22T21:05:45.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ded6be4fcd796830-00650e01a9-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:45.512 DEBUG:boto:b'ffzdqu-9999false' 2023-09-22T21:05:45.512 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-9 zones={a2, a1} 2023-09-22T21:05:45.512 DEBUG:boto:path=/ffzdqu-10/ 2023-09-22T21:05:45.513 DEBUG:boto:auth_path=/ffzdqu-10/ 2023-09-22T21:05:45.513 DEBUG:boto:Method: HEAD 2023-09-22T21:05:45.513 DEBUG:boto:Path: /ffzdqu-10/ 2023-09-22T21:05:45.513 DEBUG:boto:Data: 2023-09-22T21:05:45.513 DEBUG:boto:Headers: {} 2023-09-22T21:05:45.513 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:45.513 DEBUG:boto:Port: 8001 2023-09-22T21:05:45.513 DEBUG:boto:Params: {} 2023-09-22T21:05:45.513 DEBUG:boto:Token: None 2023-09-22T21:05:45.514 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:45 GMT /ffzdqu-10/ 2023-09-22T21:05:45.514 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:apq52fs22OwXZV83wAe4k2GATJo= 2023-09-22T21:05:45.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:apq52fs22OwXZV83wAe4k2GATJo=', 'Content-Length': '0'} 2023-09-22T21:05:45.517 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ebae00f82af2156-00650e01a9-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:45.517 DEBUG:boto:path=/ffzdqu-10/myobj 2023-09-22T21:05:45.517 DEBUG:boto:auth_path=/ffzdqu-10/myobj 2023-09-22T21:05:45.518 DEBUG:boto:Method: HEAD 2023-09-22T21:05:45.518 DEBUG:boto:Path: /ffzdqu-10/myobj 2023-09-22T21:05:45.518 DEBUG:boto:Data: 2023-09-22T21:05:45.518 DEBUG:boto:Headers: {} 2023-09-22T21:05:45.518 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:45.518 DEBUG:boto:Port: 8001 2023-09-22T21:05:45.518 DEBUG:boto:Params: {} 2023-09-22T21:05:45.518 DEBUG:boto:Token: None 2023-09-22T21:05:45.518 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:05:45 GMT /ffzdqu-10/myobj 2023-09-22T21:05:45.519 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BeLXu5U96uqwd0ncV7eGP5Psoqc= 2023-09-22T21:05:45.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BeLXu5U96uqwd0ncV7eGP5Psoqc=', 'Content-Length': '0'} 2023-09-22T21:05:45.521 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:05:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000839dc4cfc7536506-00650e01a9-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:45.521 DEBUG:boto:path=/ffzdqu-10/myobj 2023-09-22T21:05:45.521 DEBUG:boto:auth_path=/ffzdqu-10/myobj 2023-09-22T21:05:45.521 DEBUG:boto:Method: DELETE 2023-09-22T21:05:45.522 DEBUG:boto:Path: /ffzdqu-10/myobj 2023-09-22T21:05:45.522 DEBUG:boto:Data: 2023-09-22T21:05:45.522 DEBUG:boto:Headers: {} 2023-09-22T21:05:45.522 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:05:45.522 DEBUG:boto:Port: 8001 2023-09-22T21:05:45.522 DEBUG:boto:Params: {} 2023-09-22T21:05:45.522 DEBUG:boto:Token: None 2023-09-22T21:05:45.522 DEBUG:boto:StringToSign: DELETE Fri, 22 Sep 2023 21:05:45 GMT /ffzdqu-10/myobj 2023-09-22T21:05:45.522 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lC0zhG1WRs/7K6pUB/shG6WxiZQ= 2023-09-22T21:05:45.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:05:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lC0zhG1WRs/7K6pUB/shG6WxiZQ=', 'Content-Length': '0'} 2023-09-22T21:05:45.534 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e9807f55c05150dc-00650e01a9-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:05:45.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:05:45.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:05:45.535 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:05:45.584 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:05:45.585 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:05:45.627 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:05:45.628+0000 7ff4e9159a40 1 waiting for incremental sync to catch up: 2023-09-22T21:05:45.627 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2023-09-22T21:05:45.627 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2023-09-22T21:05:50.629 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:05:50.628+0000 7ff4e9159a40 1 waiting for incremental sync to catch up: 2023-09-22T21:05:50.816 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2023-09-22T21:05:50.817 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2023-09-22T21:05:55.631 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:05:55.632+0000 7ff4e9159a40 1 waiting for incremental sync to catch up: 2023-09-22T21:05:55.631 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2023-09-22T21:05:55.631 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2023-09-22T21:06:00.633 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:06:00.632+0000 7ff4e9159a40 1 bucket sync caught up with source: 2023-09-22T21:06:00.633 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.11.6, , , , , , , , , ] 2023-09-22T21:06:00.633 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2023-09-22T21:06:00.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:06:00.632+0000 7ff4e9159a40 0 bucket checkpoint complete 2023-09-22T21:06:00.663 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-10 zones={a1, a2} 2023-09-22T21:06:00.663 DEBUG:boto:path=/ffzdqu-10/ 2023-09-22T21:06:00.664 DEBUG:boto:auth_path=/ffzdqu-10/ 2023-09-22T21:06:00.664 DEBUG:boto:Method: HEAD 2023-09-22T21:06:00.664 DEBUG:boto:Path: /ffzdqu-10/ 2023-09-22T21:06:00.664 DEBUG:boto:Data: 2023-09-22T21:06:00.664 DEBUG:boto:Headers: {} 2023-09-22T21:06:00.664 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:06:00.664 DEBUG:boto:Port: 8000 2023-09-22T21:06:00.664 DEBUG:boto:Params: {} 2023-09-22T21:06:00.664 DEBUG:boto:Token: None 2023-09-22T21:06:00.665 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:06:00 GMT /ffzdqu-10/ 2023-09-22T21:06:00.665 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H4QhJOINgcrtUG7AqB36LfQL2Ko= 2023-09-22T21:06:00.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H4QhJOINgcrtUG7AqB36LfQL2Ko=', 'Content-Length': '0'} 2023-09-22T21:06:00.668 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005816db0628b10455-00650e01b8-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:00.669 DEBUG:boto:path=/ffzdqu-10/ 2023-09-22T21:06:00.669 DEBUG:boto:auth_path=/ffzdqu-10/ 2023-09-22T21:06:00.669 DEBUG:boto:Method: HEAD 2023-09-22T21:06:00.669 DEBUG:boto:Path: /ffzdqu-10/ 2023-09-22T21:06:00.669 DEBUG:boto:Data: 2023-09-22T21:06:00.669 DEBUG:boto:Headers: {} 2023-09-22T21:06:00.669 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:06:00.669 DEBUG:boto:Port: 8001 2023-09-22T21:06:00.670 DEBUG:boto:Params: {} 2023-09-22T21:06:00.670 DEBUG:boto:Token: None 2023-09-22T21:06:00.670 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:06:00 GMT /ffzdqu-10/ 2023-09-22T21:06:00.670 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H4QhJOINgcrtUG7AqB36LfQL2Ko= 2023-09-22T21:06:00.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H4QhJOINgcrtUG7AqB36LfQL2Ko=', 'Content-Length': '0'} 2023-09-22T21:06:00.673 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035222baed83be762-00650e01b8-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:00.673 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:06:00.674 DEBUG:boto:path=/ffzdqu-10/ 2023-09-22T21:06:00.674 DEBUG:boto:auth_path=/ffzdqu-10/ 2023-09-22T21:06:00.674 DEBUG:boto:path=/ffzdqu-10/?versions&max-keys=999 2023-09-22T21:06:00.674 DEBUG:boto:auth_path=/ffzdqu-10/?versions&max-keys=999 2023-09-22T21:06:00.674 DEBUG:boto:Method: GET 2023-09-22T21:06:00.674 DEBUG:boto:Path: /ffzdqu-10/?versions&max-keys=999 2023-09-22T21:06:00.674 DEBUG:boto:Data: 2023-09-22T21:06:00.674 DEBUG:boto:Headers: {} 2023-09-22T21:06:00.674 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:06:00.674 DEBUG:boto:Port: 8000 2023-09-22T21:06:00.675 DEBUG:boto:Params: {} 2023-09-22T21:06:00.675 DEBUG:boto:Token: None 2023-09-22T21:06:00.675 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:06:00 GMT /ffzdqu-10/?versions 2023-09-22T21:06:00.675 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:N/ZRIL7LIxikdlDv+ecvbB33R20= 2023-09-22T21:06:00.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:N/ZRIL7LIxikdlDv+ecvbB33R20=', 'Content-Length': '0'} 2023-09-22T21:06:00.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008809e85464c2bd0-00650e01b8-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:00.679 DEBUG:boto:b'ffzdqu-10999false' 2023-09-22T21:06:00.679 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:06:00.679 DEBUG:boto:path=/ffzdqu-10/ 2023-09-22T21:06:00.680 DEBUG:boto:auth_path=/ffzdqu-10/ 2023-09-22T21:06:00.680 DEBUG:boto:path=/ffzdqu-10/?versions&max-keys=999 2023-09-22T21:06:00.680 DEBUG:boto:auth_path=/ffzdqu-10/?versions&max-keys=999 2023-09-22T21:06:00.680 DEBUG:boto:Method: GET 2023-09-22T21:06:00.680 DEBUG:boto:Path: /ffzdqu-10/?versions&max-keys=999 2023-09-22T21:06:00.680 DEBUG:boto:Data: 2023-09-22T21:06:00.680 DEBUG:boto:Headers: {} 2023-09-22T21:06:00.680 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:06:00.681 DEBUG:boto:Port: 8001 2023-09-22T21:06:00.681 DEBUG:boto:Params: {} 2023-09-22T21:06:00.681 DEBUG:boto:Token: None 2023-09-22T21:06:00.681 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:06:00 GMT /ffzdqu-10/?versions 2023-09-22T21:06:00.681 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:N/ZRIL7LIxikdlDv+ecvbB33R20= 2023-09-22T21:06:00.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:N/ZRIL7LIxikdlDv+ecvbB33R20=', 'Content-Length': '0'} 2023-09-22T21:06:00.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000def9ca1cc6a4b896-00650e01b8-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:00.685 DEBUG:boto:b'ffzdqu-10999false' 2023-09-22T21:06:00.685 DEBUG:boto:path=/ffzdqu-10/ 2023-09-22T21:06:00.685 DEBUG:boto:auth_path=/ffzdqu-10/ 2023-09-22T21:06:00.685 DEBUG:boto:path=/ffzdqu-10/?versions&max-keys=999 2023-09-22T21:06:00.685 DEBUG:boto:auth_path=/ffzdqu-10/?versions&max-keys=999 2023-09-22T21:06:00.685 DEBUG:boto:Method: GET 2023-09-22T21:06:00.686 DEBUG:boto:Path: /ffzdqu-10/?versions&max-keys=999 2023-09-22T21:06:00.686 DEBUG:boto:Data: 2023-09-22T21:06:00.686 DEBUG:boto:Headers: {} 2023-09-22T21:06:00.686 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:06:00.686 DEBUG:boto:Port: 8000 2023-09-22T21:06:00.686 DEBUG:boto:Params: {} 2023-09-22T21:06:00.686 DEBUG:boto:Token: None 2023-09-22T21:06:00.686 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:06:00 GMT /ffzdqu-10/?versions 2023-09-22T21:06:00.687 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:N/ZRIL7LIxikdlDv+ecvbB33R20= 2023-09-22T21:06:00.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:N/ZRIL7LIxikdlDv+ecvbB33R20=', 'Content-Length': '0'} 2023-09-22T21:06:00.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaa012573bab94f2-00650e01b8-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:00.734 DEBUG:boto:b'ffzdqu-10999false' 2023-09-22T21:06:00.735 DEBUG:boto:path=/ffzdqu-10/ 2023-09-22T21:06:00.735 DEBUG:boto:auth_path=/ffzdqu-10/ 2023-09-22T21:06:00.735 DEBUG:boto:path=/ffzdqu-10/?versions&max-keys=999 2023-09-22T21:06:00.735 DEBUG:boto:auth_path=/ffzdqu-10/?versions&max-keys=999 2023-09-22T21:06:00.735 DEBUG:boto:Method: GET 2023-09-22T21:06:00.735 DEBUG:boto:Path: /ffzdqu-10/?versions&max-keys=999 2023-09-22T21:06:00.735 DEBUG:boto:Data: 2023-09-22T21:06:00.736 DEBUG:boto:Headers: {} 2023-09-22T21:06:00.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:06:00.736 DEBUG:boto:Port: 8001 2023-09-22T21:06:00.736 DEBUG:boto:Params: {} 2023-09-22T21:06:00.736 DEBUG:boto:Token: None 2023-09-22T21:06:00.736 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:06:00 GMT /ffzdqu-10/?versions 2023-09-22T21:06:00.736 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:N/ZRIL7LIxikdlDv+ecvbB33R20= 2023-09-22T21:06:00.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:N/ZRIL7LIxikdlDv+ecvbB33R20=', 'Content-Length': '0'} 2023-09-22T21:06:00.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d820df894bed4e5-00650e01b8-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:00.741 DEBUG:boto:b'ffzdqu-10999false' 2023-09-22T21:06:00.741 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-10 zones={a1, a2} 2023-09-22T21:06:00.742 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2023-09-22T21:06:00.743 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-11 2023-09-22T21:06:00.743 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:06:00.743 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:06:00.743 DEBUG:boto:Method: PUT 2023-09-22T21:06:00.743 DEBUG:boto:Path: /ffzdqu-11/ 2023-09-22T21:06:00.743 DEBUG:boto:Data: 2023-09-22T21:06:00.743 DEBUG:boto:Headers: {} 2023-09-22T21:06:00.743 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:06:00.744 DEBUG:boto:Port: 8000 2023-09-22T21:06:00.744 DEBUG:boto:Params: {} 2023-09-22T21:06:00.744 DEBUG:boto:Token: None 2023-09-22T21:06:00.744 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:06:00 GMT /ffzdqu-11/ 2023-09-22T21:06:00.744 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YCnwexJr8i0mr1OrfFDl4cyLM3c= 2023-09-22T21:06:00.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YCnwexJr8i0mr1OrfFDl4cyLM3c=', 'Content-Length': '0'} 2023-09-22T21:06:00.782 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006b1adad8b3c1c22f-00650e01b8-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:00.783 INFO:rgw_multi.tests:create bucket zone=a2 name=ffzdqu-12 2023-09-22T21:06:00.783 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:06:00.783 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:06:00.783 DEBUG:boto:Method: PUT 2023-09-22T21:06:00.783 DEBUG:boto:Path: /ffzdqu-12/ 2023-09-22T21:06:00.783 DEBUG:boto:Data: 2023-09-22T21:06:00.783 DEBUG:boto:Headers: {} 2023-09-22T21:06:00.783 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:06:00.784 DEBUG:boto:Port: 8001 2023-09-22T21:06:00.784 DEBUG:boto:Params: {} 2023-09-22T21:06:00.784 DEBUG:boto:Token: None 2023-09-22T21:06:00.784 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:06:00 GMT /ffzdqu-12/ 2023-09-22T21:06:00.784 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3Mdjfomf5pXfTeckG149AXCqehQ= 2023-09-22T21:06:00.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3Mdjfomf5pXfTeckG149AXCqehQ=', 'Content-Length': '0'} 2023-09-22T21:06:00.857 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f26aec8549f479e2-00650e01b8-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:00.857 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:06:00.857 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:06:00.857 DEBUG:boto:path=/ffzdqu-11/?versioning 2023-09-22T21:06:00.858 DEBUG:boto:auth_path=/ffzdqu-11/?versioning 2023-09-22T21:06:00.858 DEBUG:boto:Method: PUT 2023-09-22T21:06:00.858 DEBUG:boto:Path: /ffzdqu-11/?versioning 2023-09-22T21:06:00.858 DEBUG:boto:Data: Enabled Disabled 2023-09-22T21:06:00.858 DEBUG:boto:Headers: {} 2023-09-22T21:06:00.858 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:06:00.858 DEBUG:boto:Port: 8000 2023-09-22T21:06:00.859 DEBUG:boto:Params: {} 2023-09-22T21:06:00.859 DEBUG:boto:Token: None 2023-09-22T21:06:00.859 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:06:00 GMT /ffzdqu-11/?versioning 2023-09-22T21:06:00.859 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7xf0UMijrmHK3jZqlyCQN7mmy8s= 2023-09-22T21:06:00.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7xf0UMijrmHK3jZqlyCQN7mmy8s=', 'Content-Length': '228'} 2023-09-22T21:06:00.869 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003238edb12616fef9-00650e01b8-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:00.870 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:06:00.870 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:06:00.870 DEBUG:boto:path=/ffzdqu-12/?versioning 2023-09-22T21:06:00.870 DEBUG:boto:auth_path=/ffzdqu-12/?versioning 2023-09-22T21:06:00.870 DEBUG:boto:Method: PUT 2023-09-22T21:06:00.870 DEBUG:boto:Path: /ffzdqu-12/?versioning 2023-09-22T21:06:00.870 DEBUG:boto:Data: Enabled Disabled 2023-09-22T21:06:00.870 DEBUG:boto:Headers: {} 2023-09-22T21:06:00.870 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:06:00.871 DEBUG:boto:Port: 8001 2023-09-22T21:06:00.871 DEBUG:boto:Params: {} 2023-09-22T21:06:00.871 DEBUG:boto:Token: None 2023-09-22T21:06:00.871 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:06:00 GMT /ffzdqu-12/?versioning 2023-09-22T21:06:00.871 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0ZaSl88631Yp0pka1O3x+tK6MVY= 2023-09-22T21:06:00.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0ZaSl88631Yp0pka1O3x+tK6MVY=', 'Content-Length': '228'} 2023-09-22T21:06:00.892 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ae0ec658dfc7a074-00650e01b8-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:00.892 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:06:00.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:06:00.893 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:06:00.932 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:06:00.933 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:06:00.970 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:06:00.990 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695416760.885477_57.1","last_update":"2023-09-22T21:06:00.885477Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:06:00.990 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416760.885477_57.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:06:00.991 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:06:00.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:06:00.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:06:00.992 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:06:01.077 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:06:01.077 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:06:01.116 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:06:01.116+0000 7ff8438d6a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:06:01.117 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:06:01.116+0000 7ff8438d6a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:06:01.133 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695416760.762875_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:06:00.762875Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:06:01Z"} 2023-09-22T21:06:01.133 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695416760.762875_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:06:00.762875Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:06:01Z"} 2023-09-22T21:06:01.134 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695416760.762875_46.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:06:00.762875Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:06:01.134 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416760.885477_57.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:06:01.134 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416760.762875_46.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:06:01.134 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695416760.885477_57.1 target=1_1695416760.762875_46.1 2023-09-22T21:06:06.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:06:06.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:06:06.136 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:06:06.179 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:06:06.179 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:06:06.216 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:06:06.216+0000 7f122fca5a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:06:06.217 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:06:06.216+0000 7f122fca5a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:06:06.234 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695416760.885477_57.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:06:00.885477Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:06:06Z"} 2023-09-22T21:06:06.235 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695416760.885477_57.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:06:00.885477Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:06:06Z"} 2023-09-22T21:06:06.235 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695416760.885477_57.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:06:00.885477Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:06:06.235 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416760.885477_57.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:06:06.235 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416760.885477_57.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:06:06.235 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:06:06.235 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:06:06.235 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:06:06.236 DEBUG:boto:Method: HEAD 2023-09-22T21:06:06.236 DEBUG:boto:Path: /ffzdqu-11/ 2023-09-22T21:06:06.236 DEBUG:boto:Data: 2023-09-22T21:06:06.236 DEBUG:boto:Headers: {} 2023-09-22T21:06:06.236 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:06:06.236 DEBUG:boto:Port: 8000 2023-09-22T21:06:06.236 DEBUG:boto:Params: {} 2023-09-22T21:06:06.236 DEBUG:boto:Token: None 2023-09-22T21:06:06.236 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:06:06 GMT /ffzdqu-11/ 2023-09-22T21:06:06.237 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9D6yEM7sqIvEn8JfeNGKjAovY4E= 2023-09-22T21:06:06.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9D6yEM7sqIvEn8JfeNGKjAovY4E=', 'Content-Length': '0'} 2023-09-22T21:06:06.239 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fd7d3b4e5b9961a-00650e01be-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:06.239 DEBUG:boto:path=/ffzdqu-11/dummy 2023-09-22T21:06:06.240 DEBUG:boto:auth_path=/ffzdqu-11/dummy 2023-09-22T21:06:06.240 DEBUG:boto:Method: PUT 2023-09-22T21:06:06.240 DEBUG:boto:Path: /ffzdqu-11/dummy 2023-09-22T21:06:06.240 DEBUG:boto:Data: 2023-09-22T21:06:06.240 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-22T21:06:06.240 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:06:06.240 DEBUG:boto:Port: 8000 2023-09-22T21:06:06.240 DEBUG:boto:Params: {} 2023-09-22T21:06:06.240 DEBUG:boto:Token: None 2023-09-22T21:06:06.241 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 22 Sep 2023 21:06:06 GMT /ffzdqu-11/dummy 2023-09-22T21:06:06.241 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YJElOjX3yJrYJp5uKnMS1BQQMfg= 2023-09-22T21:06:06.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:06:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YJElOjX3yJrYJp5uKnMS1BQQMfg='} 2023-09-22T21:06:06.255 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx'), ('x-amz-request-id', 'tx00000751dfcd9331023bf-00650e01be-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:06.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:06:06.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:06:06.256 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:06:06.323 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:06:06.323 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:06:06.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:06:06.364+0000 7ff4a8dcfa40 1 waiting to reach incremental sync.. 2023-09-22T21:06:11.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:06:11.368+0000 7ff4a8dcfa40 1 waiting to reach incremental sync.. 2023-09-22T21:06:16.367 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:06:16.368+0000 7ff4a8dcfa40 1 waiting to reach incremental sync.. 2023-09-22T21:06:21.370 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:06:21.372+0000 7ff4a8dcfa40 1 bucket sync caught up with source: 2023-09-22T21:06:21.370 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.8.11, , , , , ] 2023-09-22T21:06:21.370 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000002.8.11, , , , , ] 2023-09-22T21:06:21.370 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:06:21.372+0000 7ff4a8dcfa40 0 bucket checkpoint complete 2023-09-22T21:06:21.405 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:06:21.405 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:06:21.405 DEBUG:boto:Method: HEAD 2023-09-22T21:06:21.405 DEBUG:boto:Path: /ffzdqu-12/ 2023-09-22T21:06:21.405 DEBUG:boto:Data: 2023-09-22T21:06:21.406 DEBUG:boto:Headers: {} 2023-09-22T21:06:21.406 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:06:21.406 DEBUG:boto:Port: 8001 2023-09-22T21:06:21.406 DEBUG:boto:Params: {} 2023-09-22T21:06:21.406 DEBUG:boto:Token: None 2023-09-22T21:06:21.406 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:06:21 GMT /ffzdqu-12/ 2023-09-22T21:06:21.406 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JqZc8thH+B9CTET5r9EYRBVgcWM= 2023-09-22T21:06:21.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JqZc8thH+B9CTET5r9EYRBVgcWM=', 'Content-Length': '0'} 2023-09-22T21:06:21.410 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1f74a3efc7c76f2-00650e01cd-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:21.410 DEBUG:boto:path=/ffzdqu-12/dummy 2023-09-22T21:06:21.411 DEBUG:boto:auth_path=/ffzdqu-12/dummy 2023-09-22T21:06:21.411 DEBUG:boto:Method: PUT 2023-09-22T21:06:21.411 DEBUG:boto:Path: /ffzdqu-12/dummy 2023-09-22T21:06:21.411 DEBUG:boto:Data: 2023-09-22T21:06:21.411 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-22T21:06:21.411 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:06:21.411 DEBUG:boto:Port: 8001 2023-09-22T21:06:21.411 DEBUG:boto:Params: {} 2023-09-22T21:06:21.411 DEBUG:boto:Token: None 2023-09-22T21:06:21.412 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 22 Sep 2023 21:06:21 GMT /ffzdqu-12/dummy 2023-09-22T21:06:21.412 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cfIgQmqOKncwO/QVcouPQcF574A= 2023-09-22T21:06:21.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:06:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cfIgQmqOKncwO/QVcouPQcF574A='} 2023-09-22T21:06:21.435 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'boYngPv33ng8HsND7dSXZTE12I7WmDD'), ('x-amz-request-id', 'tx0000097443ca224c3152d-00650e01cd-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:21.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:06:21.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:06:21.436 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:06:21.485 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:06:21.485 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:06:21.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:06:21.528+0000 7f9a2aa02a40 1 waiting to reach incremental sync.. 2023-09-22T21:06:26.528 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:06:26.528+0000 7f9a2aa02a40 1 waiting to reach incremental sync.. 2023-09-22T21:06:31.529 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:06:31.528+0000 7f9a2aa02a40 1 waiting to reach incremental sync.. 2023-09-22T21:06:36.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:06:36.532+0000 7f9a2aa02a40 1 waiting to reach incremental sync.. 2023-09-22T21:06:41.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:06:41.532+0000 7f9a2aa02a40 1 bucket sync caught up with source: 2023-09-22T21:06:41.533 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.16.11, , , , , ] 2023-09-22T21:06:41.533 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000002.16.11, , , , , ] 2023-09-22T21:06:41.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:06:41.532+0000 7f9a2aa02a40 0 bucket checkpoint complete 2023-09-22T21:06:41.561 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:06:41.561 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:06:41.561 DEBUG:boto:Method: HEAD 2023-09-22T21:06:41.561 DEBUG:boto:Path: /ffzdqu-11/ 2023-09-22T21:06:41.561 DEBUG:boto:Data: 2023-09-22T21:06:41.562 DEBUG:boto:Headers: {} 2023-09-22T21:06:41.562 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:06:41.562 DEBUG:boto:Port: 8000 2023-09-22T21:06:41.562 DEBUG:boto:Params: {} 2023-09-22T21:06:41.562 DEBUG:boto:Token: None 2023-09-22T21:06:41.562 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:06:41 GMT /ffzdqu-11/ 2023-09-22T21:06:41.562 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wjdn6SVVBpuGHYEsQXDYdBeYZBU= 2023-09-22T21:06:41.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wjdn6SVVBpuGHYEsQXDYdBeYZBU=', 'Content-Length': '0'} 2023-09-22T21:06:41.566 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ecef5665f668f0a7-00650e01e1-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:41.567 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:06:41.567 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:06:41.567 DEBUG:boto:Method: PUT 2023-09-22T21:06:41.567 DEBUG:boto:Path: /ffzdqu-11/obj-a1 2023-09-22T21:06:41.567 DEBUG:boto:Data: 2023-09-22T21:06:41.567 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:06:41.567 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:06:41.567 DEBUG:boto:Port: 8000 2023-09-22T21:06:41.568 DEBUG:boto:Params: {} 2023-09-22T21:06:41.568 DEBUG:boto:Token: None 2023-09-22T21:06:41.568 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:06:41 GMT /ffzdqu-11/obj-a1 2023-09-22T21:06:41.568 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5FW1ogKkRTG/aelU/5ILR5sIDJQ= 2023-09-22T21:06:41.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:06:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5FW1ogKkRTG/aelU/5ILR5sIDJQ='} 2023-09-22T21:06:41.610 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc'), ('x-amz-request-id', 'tx00000e59e3064d2812836-00650e01e1-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:41.610 DEBUG:rgw_multi.tests:version1 id=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:06:41.611 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:06:41.611 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:06:41.611 DEBUG:boto:Method: PUT 2023-09-22T21:06:41.611 DEBUG:boto:Path: /ffzdqu-11/obj-a1 2023-09-22T21:06:41.611 DEBUG:boto:Data: 2023-09-22T21:06:41.611 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:06:41.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:06:41.611 DEBUG:boto:Port: 8000 2023-09-22T21:06:41.612 DEBUG:boto:Params: {} 2023-09-22T21:06:41.612 DEBUG:boto:Token: None 2023-09-22T21:06:41.612 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 22 Sep 2023 21:06:41 GMT /ffzdqu-11/obj-a1 2023-09-22T21:06:41.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FnHh55D83EaC0Bn/PeHBVQVNzPo= 2023-09-22T21:06:41.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:06:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FnHh55D83EaC0Bn/PeHBVQVNzPo='} 2023-09-22T21:06:41.629 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '2TpbOK7pTyOUEu6mS0jmrAIzKPmpPq8'), ('x-amz-request-id', 'tx00000c09c66563596ecea-00650e01e1-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:41.629 DEBUG:rgw_multi.tests:version2 id=2TpbOK7pTyOUEu6mS0jmrAIzKPmpPq8 2023-09-22T21:06:41.630 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:06:41.630 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:06:41.630 DEBUG:boto:path=/ffzdqu-11/obj-a1?versionId=2TpbOK7pTyOUEu6mS0jmrAIzKPmpPq8 2023-09-22T21:06:41.630 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1?versionId=2TpbOK7pTyOUEu6mS0jmrAIzKPmpPq8 2023-09-22T21:06:41.630 DEBUG:boto:Method: DELETE 2023-09-22T21:06:41.630 DEBUG:boto:Path: /ffzdqu-11/obj-a1?versionId=2TpbOK7pTyOUEu6mS0jmrAIzKPmpPq8 2023-09-22T21:06:41.630 DEBUG:boto:Data: 2023-09-22T21:06:41.630 DEBUG:boto:Headers: {} 2023-09-22T21:06:41.630 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:06:41.630 DEBUG:boto:Port: 8000 2023-09-22T21:06:41.631 DEBUG:boto:Params: {} 2023-09-22T21:06:41.631 DEBUG:boto:Token: None 2023-09-22T21:06:41.631 DEBUG:boto:StringToSign: DELETE Fri, 22 Sep 2023 21:06:41 GMT /ffzdqu-11/obj-a1?versionId=2TpbOK7pTyOUEu6mS0jmrAIzKPmpPq8 2023-09-22T21:06:41.631 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QvkCmAQp6DbPOWNTJdEAGhRUGNY= 2023-09-22T21:06:41.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QvkCmAQp6DbPOWNTJdEAGhRUGNY=', 'Content-Length': '0'} 2023-09-22T21:06:41.652 DEBUG:boto:Response headers: [('x-amz-version-id', '2TpbOK7pTyOUEu6mS0jmrAIzKPmpPq8'), ('x-amz-request-id', 'tx0000056566d862f08db89-00650e01e1-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:41.652 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:06:41.652 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:06:41.653 DEBUG:boto:Method: PUT 2023-09-22T21:06:41.653 DEBUG:boto:Path: /ffzdqu-11/obj-a1 2023-09-22T21:06:41.653 DEBUG:boto:Data: 2023-09-22T21:06:41.653 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:06:41.653 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:06:41.653 DEBUG:boto:Port: 8000 2023-09-22T21:06:41.653 DEBUG:boto:Params: {} 2023-09-22T21:06:41.653 DEBUG:boto:Token: None 2023-09-22T21:06:41.653 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 22 Sep 2023 21:06:41 GMT /ffzdqu-11/obj-a1 2023-09-22T21:06:41.654 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:01vusBqSX7UiVPdodXVTt/gRUnw= 2023-09-22T21:06:41.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:06:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:01vusBqSX7UiVPdodXVTt/gRUnw='} 2023-09-22T21:06:41.698 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'oq9flLJoQTjcxFANstyrATZkSd-R1SI'), ('x-amz-request-id', 'tx00000ebf035da9f0224ce-00650e01e1-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:41.699 DEBUG:rgw_multi.tests:version3 id=oq9flLJoQTjcxFANstyrATZkSd-R1SI 2023-09-22T21:06:41.699 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:06:41.699 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:06:41.699 DEBUG:boto:path=/ffzdqu-11/obj-a1?versionId=oq9flLJoQTjcxFANstyrATZkSd-R1SI 2023-09-22T21:06:41.699 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1?versionId=oq9flLJoQTjcxFANstyrATZkSd-R1SI 2023-09-22T21:06:41.699 DEBUG:boto:Method: DELETE 2023-09-22T21:06:41.699 DEBUG:boto:Path: /ffzdqu-11/obj-a1?versionId=oq9flLJoQTjcxFANstyrATZkSd-R1SI 2023-09-22T21:06:41.700 DEBUG:boto:Data: 2023-09-22T21:06:41.700 DEBUG:boto:Headers: {} 2023-09-22T21:06:41.700 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:06:41.700 DEBUG:boto:Port: 8000 2023-09-22T21:06:41.700 DEBUG:boto:Params: {} 2023-09-22T21:06:41.700 DEBUG:boto:Token: None 2023-09-22T21:06:41.700 DEBUG:boto:StringToSign: DELETE Fri, 22 Sep 2023 21:06:41 GMT /ffzdqu-11/obj-a1?versionId=oq9flLJoQTjcxFANstyrATZkSd-R1SI 2023-09-22T21:06:41.700 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oX35BQ0uVZ5gI1M0afPzWZLNY7g= 2023-09-22T21:06:41.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oX35BQ0uVZ5gI1M0afPzWZLNY7g=', 'Content-Length': '0'} 2023-09-22T21:06:41.727 DEBUG:boto:Response headers: [('x-amz-version-id', 'oq9flLJoQTjcxFANstyrATZkSd-R1SI'), ('x-amz-request-id', 'tx00000b738f7f09d5a8625-00650e01e1-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:41.727 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:06:41.727 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:06:41.727 DEBUG:boto:Method: HEAD 2023-09-22T21:06:41.727 DEBUG:boto:Path: /ffzdqu-11/ 2023-09-22T21:06:41.727 DEBUG:boto:Data: 2023-09-22T21:06:41.727 DEBUG:boto:Headers: {} 2023-09-22T21:06:41.727 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:06:41.728 DEBUG:boto:Port: 8001 2023-09-22T21:06:41.728 DEBUG:boto:Params: {} 2023-09-22T21:06:41.728 DEBUG:boto:Token: None 2023-09-22T21:06:41.728 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:06:41 GMT /ffzdqu-11/ 2023-09-22T21:06:41.728 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wjdn6SVVBpuGHYEsQXDYdBeYZBU= 2023-09-22T21:06:41.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wjdn6SVVBpuGHYEsQXDYdBeYZBU=', 'Content-Length': '0'} 2023-09-22T21:06:41.734 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c5cabdb3ebb6f1a-00650e01e1-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:41.734 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:06:41.735 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:06:41.735 DEBUG:boto:Method: PUT 2023-09-22T21:06:41.735 DEBUG:boto:Path: /ffzdqu-11/obj-a2 2023-09-22T21:06:41.735 DEBUG:boto:Data: 2023-09-22T21:06:41.735 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:06:41.735 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:06:41.735 DEBUG:boto:Port: 8001 2023-09-22T21:06:41.735 DEBUG:boto:Params: {} 2023-09-22T21:06:41.735 DEBUG:boto:Token: None 2023-09-22T21:06:41.736 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:06:41 GMT /ffzdqu-11/obj-a2 2023-09-22T21:06:41.736 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aEjmYosHqMcvf196mN+9XmlCsu8= 2023-09-22T21:06:41.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:06:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aEjmYosHqMcvf196mN+9XmlCsu8='} 2023-09-22T21:06:41.768 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr'), ('x-amz-request-id', 'tx00000f7a2d8bfea00b7a5-00650e01e1-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:41.768 DEBUG:rgw_multi.tests:version1 id=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:06:41.769 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:06:41.769 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:06:41.769 DEBUG:boto:Method: PUT 2023-09-22T21:06:41.769 DEBUG:boto:Path: /ffzdqu-11/obj-a2 2023-09-22T21:06:41.769 DEBUG:boto:Data: 2023-09-22T21:06:41.769 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:06:41.769 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:06:41.770 DEBUG:boto:Port: 8001 2023-09-22T21:06:41.770 DEBUG:boto:Params: {} 2023-09-22T21:06:41.770 DEBUG:boto:Token: None 2023-09-22T21:06:41.770 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 22 Sep 2023 21:06:41 GMT /ffzdqu-11/obj-a2 2023-09-22T21:06:41.770 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cD0Z+y9lf2+mO7zVB8c2fDSkaBM= 2023-09-22T21:06:41.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:06:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cD0Z+y9lf2+mO7zVB8c2fDSkaBM='} 2023-09-22T21:06:41.793 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Zv7bp1S-LpPK.rBltuqcHcoSCnLWVQd'), ('x-amz-request-id', 'tx000004adef9bbbe407617-00650e01e1-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:41.793 DEBUG:rgw_multi.tests:version2 id=Zv7bp1S-LpPK.rBltuqcHcoSCnLWVQd 2023-09-22T21:06:41.793 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:06:41.793 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:06:41.794 DEBUG:boto:path=/ffzdqu-11/obj-a2?versionId=Zv7bp1S-LpPK.rBltuqcHcoSCnLWVQd 2023-09-22T21:06:41.794 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2?versionId=Zv7bp1S-LpPK.rBltuqcHcoSCnLWVQd 2023-09-22T21:06:41.794 DEBUG:boto:Method: DELETE 2023-09-22T21:06:41.794 DEBUG:boto:Path: /ffzdqu-11/obj-a2?versionId=Zv7bp1S-LpPK.rBltuqcHcoSCnLWVQd 2023-09-22T21:06:41.794 DEBUG:boto:Data: 2023-09-22T21:06:41.794 DEBUG:boto:Headers: {} 2023-09-22T21:06:41.794 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:06:41.794 DEBUG:boto:Port: 8001 2023-09-22T21:06:41.794 DEBUG:boto:Params: {} 2023-09-22T21:06:41.795 DEBUG:boto:Token: None 2023-09-22T21:06:41.795 DEBUG:boto:StringToSign: DELETE Fri, 22 Sep 2023 21:06:41 GMT /ffzdqu-11/obj-a2?versionId=Zv7bp1S-LpPK.rBltuqcHcoSCnLWVQd 2023-09-22T21:06:41.795 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VhAHFm16HaNYnmNBxMloqKMkLP0= 2023-09-22T21:06:41.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VhAHFm16HaNYnmNBxMloqKMkLP0=', 'Content-Length': '0'} 2023-09-22T21:06:41.818 DEBUG:boto:Response headers: [('x-amz-version-id', 'Zv7bp1S-LpPK.rBltuqcHcoSCnLWVQd'), ('x-amz-request-id', 'tx00000107cb15492c9d27c-00650e01e1-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:41.818 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:06:41.818 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:06:41.819 DEBUG:boto:Method: PUT 2023-09-22T21:06:41.819 DEBUG:boto:Path: /ffzdqu-11/obj-a2 2023-09-22T21:06:41.819 DEBUG:boto:Data: 2023-09-22T21:06:41.819 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:06:41.819 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:06:41.819 DEBUG:boto:Port: 8001 2023-09-22T21:06:41.819 DEBUG:boto:Params: {} 2023-09-22T21:06:41.819 DEBUG:boto:Token: None 2023-09-22T21:06:41.820 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 22 Sep 2023 21:06:41 GMT /ffzdqu-11/obj-a2 2023-09-22T21:06:41.820 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7EM5ZfAQyPgv0P9AZixJxRCqtdI= 2023-09-22T21:06:41.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:06:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7EM5ZfAQyPgv0P9AZixJxRCqtdI='} 2023-09-22T21:06:41.863 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Bt-MJPbmT3okHjylylw5x4LsgeZyOJK'), ('x-amz-request-id', 'tx000003262a224097c0162-00650e01e1-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:41.863 DEBUG:rgw_multi.tests:version3 id=Bt-MJPbmT3okHjylylw5x4LsgeZyOJK 2023-09-22T21:06:41.863 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:06:41.863 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:06:41.864 DEBUG:boto:path=/ffzdqu-11/obj-a2?versionId=Bt-MJPbmT3okHjylylw5x4LsgeZyOJK 2023-09-22T21:06:41.864 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2?versionId=Bt-MJPbmT3okHjylylw5x4LsgeZyOJK 2023-09-22T21:06:41.864 DEBUG:boto:Method: DELETE 2023-09-22T21:06:41.864 DEBUG:boto:Path: /ffzdqu-11/obj-a2?versionId=Bt-MJPbmT3okHjylylw5x4LsgeZyOJK 2023-09-22T21:06:41.864 DEBUG:boto:Data: 2023-09-22T21:06:41.864 DEBUG:boto:Headers: {} 2023-09-22T21:06:41.865 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:06:41.865 DEBUG:boto:Port: 8001 2023-09-22T21:06:41.865 DEBUG:boto:Params: {} 2023-09-22T21:06:41.865 DEBUG:boto:Token: None 2023-09-22T21:06:41.865 DEBUG:boto:StringToSign: DELETE Fri, 22 Sep 2023 21:06:41 GMT /ffzdqu-11/obj-a2?versionId=Bt-MJPbmT3okHjylylw5x4LsgeZyOJK 2023-09-22T21:06:41.866 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Z1KMRcYW/OTruR70MkbTMXTjcaQ= 2023-09-22T21:06:41.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Z1KMRcYW/OTruR70MkbTMXTjcaQ=', 'Content-Length': '0'} 2023-09-22T21:06:41.890 DEBUG:boto:Response headers: [('x-amz-version-id', 'Bt-MJPbmT3okHjylylw5x4LsgeZyOJK'), ('x-amz-request-id', 'tx00000579b15ac2eece9d2-00650e01e1-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:41.891 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:06:41.891 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:06:41.891 DEBUG:boto:Method: HEAD 2023-09-22T21:06:41.891 DEBUG:boto:Path: /ffzdqu-12/ 2023-09-22T21:06:41.891 DEBUG:boto:Data: 2023-09-22T21:06:41.891 DEBUG:boto:Headers: {} 2023-09-22T21:06:41.891 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:06:41.891 DEBUG:boto:Port: 8000 2023-09-22T21:06:41.892 DEBUG:boto:Params: {} 2023-09-22T21:06:41.892 DEBUG:boto:Token: None 2023-09-22T21:06:41.892 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:06:41 GMT /ffzdqu-12/ 2023-09-22T21:06:41.892 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7JZTaO+R8ITL2MuvuiWTfgZA+rw= 2023-09-22T21:06:41.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7JZTaO+R8ITL2MuvuiWTfgZA+rw=', 'Content-Length': '0'} 2023-09-22T21:06:41.895 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cad117199f461a65-00650e01e1-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:41.896 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:06:41.896 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:06:41.896 DEBUG:boto:Method: PUT 2023-09-22T21:06:41.896 DEBUG:boto:Path: /ffzdqu-12/obj-a1 2023-09-22T21:06:41.896 DEBUG:boto:Data: 2023-09-22T21:06:41.896 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:06:41.896 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:06:41.897 DEBUG:boto:Port: 8000 2023-09-22T21:06:41.897 DEBUG:boto:Params: {} 2023-09-22T21:06:41.897 DEBUG:boto:Token: None 2023-09-22T21:06:41.897 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:06:41 GMT /ffzdqu-12/obj-a1 2023-09-22T21:06:41.897 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gD7JuTS1nGf7MFK8Y858NBNwoG4= 2023-09-22T21:06:41.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:06:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gD7JuTS1nGf7MFK8Y858NBNwoG4='} 2023-09-22T21:06:41.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL'), ('x-amz-request-id', 'tx00000703837d434d35aa1-00650e01e1-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:41.922 DEBUG:rgw_multi.tests:version1 id=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:06:41.922 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:06:41.922 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:06:41.922 DEBUG:boto:Method: PUT 2023-09-22T21:06:41.922 DEBUG:boto:Path: /ffzdqu-12/obj-a1 2023-09-22T21:06:41.922 DEBUG:boto:Data: 2023-09-22T21:06:41.923 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:06:41.923 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:06:41.923 DEBUG:boto:Port: 8000 2023-09-22T21:06:41.923 DEBUG:boto:Params: {} 2023-09-22T21:06:41.923 DEBUG:boto:Token: None 2023-09-22T21:06:41.923 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 22 Sep 2023 21:06:41 GMT /ffzdqu-12/obj-a1 2023-09-22T21:06:41.923 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YYcRAY9I58YRKWIFY2HWd05j3kQ= 2023-09-22T21:06:41.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:06:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YYcRAY9I58YRKWIFY2HWd05j3kQ='} 2023-09-22T21:06:41.966 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '.vO5m0JCm4yZXbOilzbNQm2fg-ZENFD'), ('x-amz-request-id', 'tx00000078fa550b11e442c-00650e01e1-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:41.967 DEBUG:rgw_multi.tests:version2 id=.vO5m0JCm4yZXbOilzbNQm2fg-ZENFD 2023-09-22T21:06:41.967 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:06:41.967 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:06:41.967 DEBUG:boto:path=/ffzdqu-12/obj-a1?versionId=.vO5m0JCm4yZXbOilzbNQm2fg-ZENFD 2023-09-22T21:06:41.967 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1?versionId=.vO5m0JCm4yZXbOilzbNQm2fg-ZENFD 2023-09-22T21:06:41.967 DEBUG:boto:Method: DELETE 2023-09-22T21:06:41.967 DEBUG:boto:Path: /ffzdqu-12/obj-a1?versionId=.vO5m0JCm4yZXbOilzbNQm2fg-ZENFD 2023-09-22T21:06:41.967 DEBUG:boto:Data: 2023-09-22T21:06:41.967 DEBUG:boto:Headers: {} 2023-09-22T21:06:41.968 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:06:41.968 DEBUG:boto:Port: 8000 2023-09-22T21:06:41.968 DEBUG:boto:Params: {} 2023-09-22T21:06:41.968 DEBUG:boto:Token: None 2023-09-22T21:06:41.968 DEBUG:boto:StringToSign: DELETE Fri, 22 Sep 2023 21:06:41 GMT /ffzdqu-12/obj-a1?versionId=.vO5m0JCm4yZXbOilzbNQm2fg-ZENFD 2023-09-22T21:06:41.968 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PCn64xXicUDiJvc9XrZxey1ii20= 2023-09-22T21:06:41.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PCn64xXicUDiJvc9XrZxey1ii20=', 'Content-Length': '0'} 2023-09-22T21:06:41.991 DEBUG:boto:Response headers: [('x-amz-version-id', '.vO5m0JCm4yZXbOilzbNQm2fg-ZENFD'), ('x-amz-request-id', 'tx00000cd28f6f0c437d5f2-00650e01e1-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:41.991 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:06:41.992 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:06:41.992 DEBUG:boto:Method: PUT 2023-09-22T21:06:41.992 DEBUG:boto:Path: /ffzdqu-12/obj-a1 2023-09-22T21:06:41.992 DEBUG:boto:Data: 2023-09-22T21:06:41.992 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:06:41.992 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:06:41.992 DEBUG:boto:Port: 8000 2023-09-22T21:06:41.992 DEBUG:boto:Params: {} 2023-09-22T21:06:41.992 DEBUG:boto:Token: None 2023-09-22T21:06:41.993 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 22 Sep 2023 21:06:41 GMT /ffzdqu-12/obj-a1 2023-09-22T21:06:41.993 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i9AWbT61xrMVG/F9ubjeKS+1gL4= 2023-09-22T21:06:41.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:06:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i9AWbT61xrMVG/F9ubjeKS+1gL4='} 2023-09-22T21:06:42.017 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'qVYTayjH3e4C4SyFl6M6z3u8PfskYY6'), ('x-amz-request-id', 'tx000003de481a6d266b20e-00650e01e1-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:42.017 DEBUG:rgw_multi.tests:version3 id=qVYTayjH3e4C4SyFl6M6z3u8PfskYY6 2023-09-22T21:06:42.017 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:06:42.017 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:06:42.017 DEBUG:boto:path=/ffzdqu-12/obj-a1?versionId=qVYTayjH3e4C4SyFl6M6z3u8PfskYY6 2023-09-22T21:06:42.017 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1?versionId=qVYTayjH3e4C4SyFl6M6z3u8PfskYY6 2023-09-22T21:06:42.018 DEBUG:boto:Method: DELETE 2023-09-22T21:06:42.018 DEBUG:boto:Path: /ffzdqu-12/obj-a1?versionId=qVYTayjH3e4C4SyFl6M6z3u8PfskYY6 2023-09-22T21:06:42.018 DEBUG:boto:Data: 2023-09-22T21:06:42.018 DEBUG:boto:Headers: {} 2023-09-22T21:06:42.018 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:06:42.018 DEBUG:boto:Port: 8000 2023-09-22T21:06:42.018 DEBUG:boto:Params: {} 2023-09-22T21:06:42.018 DEBUG:boto:Token: None 2023-09-22T21:06:42.018 DEBUG:boto:StringToSign: DELETE Fri, 22 Sep 2023 21:06:42 GMT /ffzdqu-12/obj-a1?versionId=qVYTayjH3e4C4SyFl6M6z3u8PfskYY6 2023-09-22T21:06:42.019 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Psa80OzMU10xR67LFA/kXB9Vk1U= 2023-09-22T21:06:42.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Psa80OzMU10xR67LFA/kXB9Vk1U=', 'Content-Length': '0'} 2023-09-22T21:06:42.045 DEBUG:boto:Response headers: [('x-amz-version-id', 'qVYTayjH3e4C4SyFl6M6z3u8PfskYY6'), ('x-amz-request-id', 'tx000005564d8da4a1fdcd9-00650e01e2-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:42.045 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:06:42.045 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:06:42.045 DEBUG:boto:Method: HEAD 2023-09-22T21:06:42.045 DEBUG:boto:Path: /ffzdqu-12/ 2023-09-22T21:06:42.045 DEBUG:boto:Data: 2023-09-22T21:06:42.046 DEBUG:boto:Headers: {} 2023-09-22T21:06:42.046 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:06:42.046 DEBUG:boto:Port: 8001 2023-09-22T21:06:42.046 DEBUG:boto:Params: {} 2023-09-22T21:06:42.046 DEBUG:boto:Token: None 2023-09-22T21:06:42.046 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:06:42 GMT /ffzdqu-12/ 2023-09-22T21:06:42.046 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JwaLORAicjCRKcNPWGDEaghzFH0= 2023-09-22T21:06:42.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JwaLORAicjCRKcNPWGDEaghzFH0=', 'Content-Length': '0'} 2023-09-22T21:06:42.051 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002726e060b60ee8d9-00650e01e2-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:42.051 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:06:42.051 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:06:42.051 DEBUG:boto:Method: PUT 2023-09-22T21:06:42.051 DEBUG:boto:Path: /ffzdqu-12/obj-a2 2023-09-22T21:06:42.051 DEBUG:boto:Data: 2023-09-22T21:06:42.052 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:06:42.052 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:06:42.052 DEBUG:boto:Port: 8001 2023-09-22T21:06:42.052 DEBUG:boto:Params: {} 2023-09-22T21:06:42.052 DEBUG:boto:Token: None 2023-09-22T21:06:42.052 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:06:42 GMT /ffzdqu-12/obj-a2 2023-09-22T21:06:42.052 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZOTKeqp149f30Ys71duQ/gmH3xw= 2023-09-22T21:06:42.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:06:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZOTKeqp149f30Ys71duQ/gmH3xw='} 2023-09-22T21:06:42.078 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'u7s2N-8HbpXwMBMoersKLSCgHbn-fnr'), ('x-amz-request-id', 'tx000001eb06b30a5d9704b-00650e01e2-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:42.078 DEBUG:rgw_multi.tests:version1 id=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:06:42.078 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:06:42.079 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:06:42.079 DEBUG:boto:Method: PUT 2023-09-22T21:06:42.079 DEBUG:boto:Path: /ffzdqu-12/obj-a2 2023-09-22T21:06:42.079 DEBUG:boto:Data: 2023-09-22T21:06:42.079 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:06:42.079 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:06:42.079 DEBUG:boto:Port: 8001 2023-09-22T21:06:42.079 DEBUG:boto:Params: {} 2023-09-22T21:06:42.080 DEBUG:boto:Token: None 2023-09-22T21:06:42.080 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 22 Sep 2023 21:06:42 GMT /ffzdqu-12/obj-a2 2023-09-22T21:06:42.080 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7AJPZ4O/yJux89Qq5yQ+mn7BkrA= 2023-09-22T21:06:42.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:06:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7AJPZ4O/yJux89Qq5yQ+mn7BkrA='} 2023-09-22T21:06:42.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '3M5qWn2lcrF2xUvk-RyGoy6DtSN5s.d'), ('x-amz-request-id', 'tx00000ddad7aac59ef2f3c-00650e01e2-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:42.123 DEBUG:rgw_multi.tests:version2 id=3M5qWn2lcrF2xUvk-RyGoy6DtSN5s.d 2023-09-22T21:06:42.123 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:06:42.123 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:06:42.123 DEBUG:boto:path=/ffzdqu-12/obj-a2?versionId=3M5qWn2lcrF2xUvk-RyGoy6DtSN5s.d 2023-09-22T21:06:42.123 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2?versionId=3M5qWn2lcrF2xUvk-RyGoy6DtSN5s.d 2023-09-22T21:06:42.123 DEBUG:boto:Method: DELETE 2023-09-22T21:06:42.123 DEBUG:boto:Path: /ffzdqu-12/obj-a2?versionId=3M5qWn2lcrF2xUvk-RyGoy6DtSN5s.d 2023-09-22T21:06:42.124 DEBUG:boto:Data: 2023-09-22T21:06:42.124 DEBUG:boto:Headers: {} 2023-09-22T21:06:42.124 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:06:42.124 DEBUG:boto:Port: 8001 2023-09-22T21:06:42.124 DEBUG:boto:Params: {} 2023-09-22T21:06:42.124 DEBUG:boto:Token: None 2023-09-22T21:06:42.124 DEBUG:boto:StringToSign: DELETE Fri, 22 Sep 2023 21:06:42 GMT /ffzdqu-12/obj-a2?versionId=3M5qWn2lcrF2xUvk-RyGoy6DtSN5s.d 2023-09-22T21:06:42.124 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TDClsu9NvR0nONWnnnQBfoQQdT4= 2023-09-22T21:06:42.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TDClsu9NvR0nONWnnnQBfoQQdT4=', 'Content-Length': '0'} 2023-09-22T21:06:42.150 DEBUG:boto:Response headers: [('x-amz-version-id', '3M5qWn2lcrF2xUvk-RyGoy6DtSN5s.d'), ('x-amz-request-id', 'tx00000b74fde6d5beb1d5f-00650e01e2-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:42.150 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:06:42.151 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:06:42.151 DEBUG:boto:Method: PUT 2023-09-22T21:06:42.151 DEBUG:boto:Path: /ffzdqu-12/obj-a2 2023-09-22T21:06:42.151 DEBUG:boto:Data: 2023-09-22T21:06:42.151 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:06:42.151 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:06:42.151 DEBUG:boto:Port: 8001 2023-09-22T21:06:42.152 DEBUG:boto:Params: {} 2023-09-22T21:06:42.152 DEBUG:boto:Token: None 2023-09-22T21:06:42.152 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 22 Sep 2023 21:06:42 GMT /ffzdqu-12/obj-a2 2023-09-22T21:06:42.152 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:X3PttLj9qVe/Atj15RoaHRPK3cg= 2023-09-22T21:06:42.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:06:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:X3PttLj9qVe/Atj15RoaHRPK3cg='} 2023-09-22T21:06:42.174 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'vqxLPs5P.2xaJVo7IvJvEKflQJH9EtQ'), ('x-amz-request-id', 'tx0000011e0c9fdb2576fc3-00650e01e2-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:42.175 DEBUG:rgw_multi.tests:version3 id=vqxLPs5P.2xaJVo7IvJvEKflQJH9EtQ 2023-09-22T21:06:42.175 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:06:42.175 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:06:42.175 DEBUG:boto:path=/ffzdqu-12/obj-a2?versionId=vqxLPs5P.2xaJVo7IvJvEKflQJH9EtQ 2023-09-22T21:06:42.175 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2?versionId=vqxLPs5P.2xaJVo7IvJvEKflQJH9EtQ 2023-09-22T21:06:42.175 DEBUG:boto:Method: DELETE 2023-09-22T21:06:42.175 DEBUG:boto:Path: /ffzdqu-12/obj-a2?versionId=vqxLPs5P.2xaJVo7IvJvEKflQJH9EtQ 2023-09-22T21:06:42.175 DEBUG:boto:Data: 2023-09-22T21:06:42.176 DEBUG:boto:Headers: {} 2023-09-22T21:06:42.176 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:06:42.176 DEBUG:boto:Port: 8001 2023-09-22T21:06:42.176 DEBUG:boto:Params: {} 2023-09-22T21:06:42.176 DEBUG:boto:Token: None 2023-09-22T21:06:42.176 DEBUG:boto:StringToSign: DELETE Fri, 22 Sep 2023 21:06:42 GMT /ffzdqu-12/obj-a2?versionId=vqxLPs5P.2xaJVo7IvJvEKflQJH9EtQ 2023-09-22T21:06:42.176 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KZf/6boUOKX9X8qCTzgG1QS6g1o= 2023-09-22T21:06:42.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:06:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KZf/6boUOKX9X8qCTzgG1QS6g1o=', 'Content-Length': '0'} 2023-09-22T21:06:42.200 DEBUG:boto:Response headers: [('x-amz-version-id', 'vqxLPs5P.2xaJVo7IvJvEKflQJH9EtQ'), ('x-amz-request-id', 'tx00000b480e6f4a374a683-00650e01e2-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:06:42.200 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-11 2023-09-22T21:06:42.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:06:42.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:06:42.201 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:06:42.249 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:06:42.250 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:06:42.294 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:06:42.296+0000 7fe227c86a40 1 waiting for incremental sync to catch up: 2023-09-22T21:06:42.294 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.8.11, , , , , ] 2023-09-22T21:06:42.294 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.24.6, ] 2023-09-22T21:06:47.296 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:06:47.296+0000 7fe227c86a40 1 waiting for incremental sync to catch up: 2023-09-22T21:06:47.296 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.8.11, , , , , ] 2023-09-22T21:06:47.297 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.24.6, ] 2023-09-22T21:06:52.298 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:06:52.300+0000 7fe227c86a40 1 waiting for incremental sync to catch up: 2023-09-22T21:06:52.299 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.8.11, , , , , ] 2023-09-22T21:06:52.299 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.24.6, ] 2023-09-22T21:06:57.300 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:06:57.300+0000 7fe227c86a40 1 waiting for incremental sync to catch up: 2023-09-22T21:06:57.301 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.8.11, , , , , ] 2023-09-22T21:06:57.301 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.24.6, ] 2023-09-22T21:07:02.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:02.304+0000 7fe227c86a40 1 bucket sync caught up with source: 2023-09-22T21:07:02.303 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000003.10.10, , , , 00000000010.24.6, ] 2023-09-22T21:07:02.303 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.24.6, ] 2023-09-22T21:07:02.303 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:02.304+0000 7fe227c86a40 0 bucket checkpoint complete 2023-09-22T21:07:02.332 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-11 2023-09-22T21:07:02.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:07:02.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:07:02.332 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:07:02.381 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:07:02.382 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:07:02.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:02.424+0000 7f8d1244ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:07:02.425 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.22.11, , , 00000000010.62.6, , ] 2023-09-22T21:07:02.425 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000002.22.11, , , 00000000010.62.6, 00000000002.34.11, ] 2023-09-22T21:07:07.427 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:07.428+0000 7f8d1244ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:07:07.427 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.22.11, , , 00000000010.62.6, , ] 2023-09-22T21:07:07.427 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000002.22.11, , , 00000000010.62.6, 00000000002.34.11, ] 2023-09-22T21:07:12.429 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:12.432+0000 7f8d1244ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:07:12.430 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.22.11, , , 00000000010.62.6, , ] 2023-09-22T21:07:12.430 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000002.22.11, , , 00000000010.62.6, 00000000002.34.11, ] 2023-09-22T21:07:17.432 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:17.432+0000 7f8d1244ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:07:17.432 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.22.11, , , 00000000010.62.6, , ] 2023-09-22T21:07:17.432 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000002.22.11, , , 00000000010.62.6, 00000000002.34.11, ] 2023-09-22T21:07:22.435 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:22.436+0000 7f8d1244ea40 1 bucket sync caught up with source: 2023-09-22T21:07:22.435 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.22.11, , , 00000000010.62.6, 00000000002.34.11, ] 2023-09-22T21:07:22.435 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000002.22.11, , , 00000000010.62.6, 00000000002.34.11, ] 2023-09-22T21:07:22.435 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:22.436+0000 7f8d1244ea40 0 bucket checkpoint complete 2023-09-22T21:07:22.461 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-11 zones={a2, a1} 2023-09-22T21:07:22.462 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:22.462 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:22.462 DEBUG:boto:Method: HEAD 2023-09-22T21:07:22.462 DEBUG:boto:Path: /ffzdqu-11/ 2023-09-22T21:07:22.462 DEBUG:boto:Data: 2023-09-22T21:07:22.463 DEBUG:boto:Headers: {} 2023-09-22T21:07:22.463 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:22.463 DEBUG:boto:Port: 8001 2023-09-22T21:07:22.463 DEBUG:boto:Params: {} 2023-09-22T21:07:22.463 DEBUG:boto:Token: None 2023-09-22T21:07:22.463 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:22 GMT /ffzdqu-11/ 2023-09-22T21:07:22.464 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:L4x3O3vpJazpFLXO9X+Z1B6+wfU= 2023-09-22T21:07:22.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:L4x3O3vpJazpFLXO9X+Z1B6+wfU=', 'Content-Length': '0'} 2023-09-22T21:07:22.467 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035e8fd00ced15fe2-00650e020a-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:22.467 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:22.467 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:22.467 DEBUG:boto:Method: HEAD 2023-09-22T21:07:22.467 DEBUG:boto:Path: /ffzdqu-11/ 2023-09-22T21:07:22.468 DEBUG:boto:Data: 2023-09-22T21:07:22.468 DEBUG:boto:Headers: {} 2023-09-22T21:07:22.468 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:22.468 DEBUG:boto:Port: 8000 2023-09-22T21:07:22.468 DEBUG:boto:Params: {} 2023-09-22T21:07:22.468 DEBUG:boto:Token: None 2023-09-22T21:07:22.468 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:22 GMT /ffzdqu-11/ 2023-09-22T21:07:22.469 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:L4x3O3vpJazpFLXO9X+Z1B6+wfU= 2023-09-22T21:07:22.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:L4x3O3vpJazpFLXO9X+Z1B6+wfU=', 'Content-Length': '0'} 2023-09-22T21:07:22.472 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d81b3cbce7f7b3ca-00650e020a-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:22.472 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:07:22.472 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:22.472 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:22.473 DEBUG:boto:path=/ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:22.473 DEBUG:boto:auth_path=/ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:22.473 DEBUG:boto:Method: GET 2023-09-22T21:07:22.473 DEBUG:boto:Path: /ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:22.473 DEBUG:boto:Data: 2023-09-22T21:07:22.473 DEBUG:boto:Headers: {} 2023-09-22T21:07:22.474 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:22.474 DEBUG:boto:Port: 8001 2023-09-22T21:07:22.474 DEBUG:boto:Params: {} 2023-09-22T21:07:22.474 DEBUG:boto:Token: None 2023-09-22T21:07:22.474 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:22 GMT /ffzdqu-11/?versions 2023-09-22T21:07:22.474 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wdv0IDA+BJdqqEtxsuNxhOjuXKg= 2023-09-22T21:07:22.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wdv0IDA+BJdqqEtxsuNxhOjuXKg=', 'Content-Length': '0'} 2023-09-22T21:07:22.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039e3a060b6debdb7-00650e020a-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:22.479 DEBUG:boto:b'ffzdqu-11999falsedummyDtjOyJANiXCj7.Vh58J.RwdyLtGvcNxtrue2023-09-22T21:06:06.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a18uA2UlGwMDN2SZPYYFpbhdTZDMaHDlctrue2023-09-22T21:06:41.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a22Ghp8GTpfzBw6qT8Hie8IpGLuimDYjrtrue2023-09-22T21:06:41.742Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:22.479 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-22T21:07:22.480 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-09-22T21:07:22.480 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-09-22T21:07:22.480 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:07:22.480 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:22.480 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:22.480 DEBUG:boto:path=/ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:22.480 DEBUG:boto:auth_path=/ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:22.481 DEBUG:boto:Method: GET 2023-09-22T21:07:22.481 DEBUG:boto:Path: /ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:22.481 DEBUG:boto:Data: 2023-09-22T21:07:22.481 DEBUG:boto:Headers: {} 2023-09-22T21:07:22.481 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:22.481 DEBUG:boto:Port: 8000 2023-09-22T21:07:22.481 DEBUG:boto:Params: {} 2023-09-22T21:07:22.481 DEBUG:boto:Token: None 2023-09-22T21:07:22.481 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:22 GMT /ffzdqu-11/?versions 2023-09-22T21:07:22.482 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wdv0IDA+BJdqqEtxsuNxhOjuXKg= 2023-09-22T21:07:22.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wdv0IDA+BJdqqEtxsuNxhOjuXKg=', 'Content-Length': '0'} 2023-09-22T21:07:22.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000802b6bf744cee968-00650e020a-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:22.486 DEBUG:boto:b'ffzdqu-11999falsedummyDtjOyJANiXCj7.Vh58J.RwdyLtGvcNxtrue2023-09-22T21:06:06.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a18uA2UlGwMDN2SZPYYFpbhdTZDMaHDlctrue2023-09-22T21:06:41.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a22Ghp8GTpfzBw6qT8Hie8IpGLuimDYjrtrue2023-09-22T21:06:41.742Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:22.486 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-22T21:07:22.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-09-22T21:07:22.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-09-22T21:07:22.487 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:22.487 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:22.487 DEBUG:boto:path=/ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:22.487 DEBUG:boto:auth_path=/ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:22.487 DEBUG:boto:Method: GET 2023-09-22T21:07:22.487 DEBUG:boto:Path: /ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:22.487 DEBUG:boto:Data: 2023-09-22T21:07:22.487 DEBUG:boto:Headers: {} 2023-09-22T21:07:22.487 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:22.488 DEBUG:boto:Port: 8001 2023-09-22T21:07:22.488 DEBUG:boto:Params: {} 2023-09-22T21:07:22.488 DEBUG:boto:Token: None 2023-09-22T21:07:22.488 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:22 GMT /ffzdqu-11/?versions 2023-09-22T21:07:22.488 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wdv0IDA+BJdqqEtxsuNxhOjuXKg= 2023-09-22T21:07:22.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wdv0IDA+BJdqqEtxsuNxhOjuXKg=', 'Content-Length': '0'} 2023-09-22T21:07:22.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8e7f1a103b4fc93-00650e020a-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:22.492 DEBUG:boto:b'ffzdqu-11999falsedummyDtjOyJANiXCj7.Vh58J.RwdyLtGvcNxtrue2023-09-22T21:06:06.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a18uA2UlGwMDN2SZPYYFpbhdTZDMaHDlctrue2023-09-22T21:06:41.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a22Ghp8GTpfzBw6qT8Hie8IpGLuimDYjrtrue2023-09-22T21:06:41.742Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:22.492 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:22.492 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:22.493 DEBUG:boto:path=/ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:22.493 DEBUG:boto:auth_path=/ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:22.493 DEBUG:boto:Method: GET 2023-09-22T21:07:22.493 DEBUG:boto:Path: /ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:22.493 DEBUG:boto:Data: 2023-09-22T21:07:22.493 DEBUG:boto:Headers: {} 2023-09-22T21:07:22.493 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:22.494 DEBUG:boto:Port: 8000 2023-09-22T21:07:22.494 DEBUG:boto:Params: {} 2023-09-22T21:07:22.494 DEBUG:boto:Token: None 2023-09-22T21:07:22.494 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:22 GMT /ffzdqu-11/?versions 2023-09-22T21:07:22.494 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wdv0IDA+BJdqqEtxsuNxhOjuXKg= 2023-09-22T21:07:22.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wdv0IDA+BJdqqEtxsuNxhOjuXKg=', 'Content-Length': '0'} 2023-09-22T21:07:22.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bd6557099ad91cf-00650e020a-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:22.498 DEBUG:boto:b'ffzdqu-11999falsedummyDtjOyJANiXCj7.Vh58J.RwdyLtGvcNxtrue2023-09-22T21:06:06.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a18uA2UlGwMDN2SZPYYFpbhdTZDMaHDlctrue2023-09-22T21:06:41.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a22Ghp8GTpfzBw6qT8Hie8IpGLuimDYjrtrue2023-09-22T21:06:41.742Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:22.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-22T21:07:22.499 DEBUG:boto:path=/ffzdqu-11/dummy 2023-09-22T21:07:22.499 DEBUG:boto:auth_path=/ffzdqu-11/dummy 2023-09-22T21:07:22.499 DEBUG:boto:path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.499 DEBUG:boto:auth_path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.499 DEBUG:boto:Method: GET 2023-09-22T21:07:22.499 DEBUG:boto:Path: /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.499 DEBUG:boto:Data: 2023-09-22T21:07:22.500 DEBUG:boto:Headers: {} 2023-09-22T21:07:22.500 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:22.500 DEBUG:boto:Port: 8001 2023-09-22T21:07:22.500 DEBUG:boto:Params: {} 2023-09-22T21:07:22.500 DEBUG:boto:Token: None 2023-09-22T21:07:22.500 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:22 GMT /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.501 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6tLlq/NuW6/NQqob2q/gHY757Ks= 2023-09-22T21:07:22.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6tLlq/NuW6/NQqob2q/gHY757Ks=', 'Content-Length': '0'} 2023-09-22T21:07:22.503 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('x-amz-version-id', 'DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f91580d78129566c-00650e020a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:22.503 DEBUG:boto:path=/ffzdqu-11/dummy 2023-09-22T21:07:22.503 DEBUG:boto:auth_path=/ffzdqu-11/dummy 2023-09-22T21:07:22.503 DEBUG:boto:path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.503 DEBUG:boto:auth_path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.504 DEBUG:boto:Method: GET 2023-09-22T21:07:22.504 DEBUG:boto:Path: /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.504 DEBUG:boto:Data: 2023-09-22T21:07:22.504 DEBUG:boto:Headers: {} 2023-09-22T21:07:22.504 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:22.504 DEBUG:boto:Port: 8000 2023-09-22T21:07:22.504 DEBUG:boto:Params: {} 2023-09-22T21:07:22.504 DEBUG:boto:Token: None 2023-09-22T21:07:22.505 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:22 GMT /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.505 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6tLlq/NuW6/NQqob2q/gHY757Ks= 2023-09-22T21:07:22.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6tLlq/NuW6/NQqob2q/gHY757Ks=', 'Content-Length': '0'} 2023-09-22T21:07:22.507 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('x-amz-version-id', 'DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000031a6a0eb1e837b27-00650e020a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:22.507 DEBUG:boto:path=/ffzdqu-11/dummy 2023-09-22T21:07:22.507 DEBUG:boto:auth_path=/ffzdqu-11/dummy 2023-09-22T21:07:22.508 DEBUG:boto:path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.508 DEBUG:boto:auth_path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.508 DEBUG:boto:Method: HEAD 2023-09-22T21:07:22.508 DEBUG:boto:Path: /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.508 DEBUG:boto:Data: 2023-09-22T21:07:22.508 DEBUG:boto:Headers: {} 2023-09-22T21:07:22.508 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:22.508 DEBUG:boto:Port: 8001 2023-09-22T21:07:22.508 DEBUG:boto:Params: {} 2023-09-22T21:07:22.509 DEBUG:boto:Token: None 2023-09-22T21:07:22.509 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:22 GMT /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.509 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cELe5jUStBTYFwKMRqxSFj5EKmk= 2023-09-22T21:07:22.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cELe5jUStBTYFwKMRqxSFj5EKmk=', 'Content-Length': '0'} 2023-09-22T21:07:22.511 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('x-amz-version-id', 'DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000030af3d023ee09f75-00650e020a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:22.511 DEBUG:boto:path=/ffzdqu-11/dummy 2023-09-22T21:07:22.512 DEBUG:boto:auth_path=/ffzdqu-11/dummy 2023-09-22T21:07:22.512 DEBUG:boto:path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.512 DEBUG:boto:auth_path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.512 DEBUG:boto:Method: HEAD 2023-09-22T21:07:22.512 DEBUG:boto:Path: /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.512 DEBUG:boto:Data: 2023-09-22T21:07:22.512 DEBUG:boto:Headers: {} 2023-09-22T21:07:22.513 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:22.513 DEBUG:boto:Port: 8000 2023-09-22T21:07:22.513 DEBUG:boto:Params: {} 2023-09-22T21:07:22.513 DEBUG:boto:Token: None 2023-09-22T21:07:22.513 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:22 GMT /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.513 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cELe5jUStBTYFwKMRqxSFj5EKmk= 2023-09-22T21:07:22.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cELe5jUStBTYFwKMRqxSFj5EKmk=', 'Content-Length': '0'} 2023-09-22T21:07:22.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('x-amz-version-id', 'DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b6640f22ecae1164-00650e020a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:22.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-22T21:07:22.516 DEBUG:boto:path=/ffzdqu-11/dummy 2023-09-22T21:07:22.516 DEBUG:boto:auth_path=/ffzdqu-11/dummy 2023-09-22T21:07:22.516 DEBUG:boto:path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.516 DEBUG:boto:auth_path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.516 DEBUG:boto:Method: GET 2023-09-22T21:07:22.516 DEBUG:boto:Path: /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.516 DEBUG:boto:Data: 2023-09-22T21:07:22.516 DEBUG:boto:Headers: {} 2023-09-22T21:07:22.516 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:22.517 DEBUG:boto:Port: 8001 2023-09-22T21:07:22.517 DEBUG:boto:Params: {} 2023-09-22T21:07:22.517 DEBUG:boto:Token: None 2023-09-22T21:07:22.517 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:22 GMT /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.517 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6tLlq/NuW6/NQqob2q/gHY757Ks= 2023-09-22T21:07:22.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6tLlq/NuW6/NQqob2q/gHY757Ks=', 'Content-Length': '0'} 2023-09-22T21:07:22.519 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('x-amz-version-id', 'DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000634081dbb31a89ff-00650e020a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:22.519 DEBUG:boto:path=/ffzdqu-11/dummy 2023-09-22T21:07:22.520 DEBUG:boto:auth_path=/ffzdqu-11/dummy 2023-09-22T21:07:22.520 DEBUG:boto:path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.520 DEBUG:boto:auth_path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.520 DEBUG:boto:Method: GET 2023-09-22T21:07:22.520 DEBUG:boto:Path: /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.520 DEBUG:boto:Data: 2023-09-22T21:07:22.520 DEBUG:boto:Headers: {} 2023-09-22T21:07:22.520 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:22.520 DEBUG:boto:Port: 8000 2023-09-22T21:07:22.520 DEBUG:boto:Params: {} 2023-09-22T21:07:22.521 DEBUG:boto:Token: None 2023-09-22T21:07:22.521 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:22 GMT /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.521 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6tLlq/NuW6/NQqob2q/gHY757Ks= 2023-09-22T21:07:22.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6tLlq/NuW6/NQqob2q/gHY757Ks=', 'Content-Length': '0'} 2023-09-22T21:07:22.523 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('x-amz-version-id', 'DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000096d692a8e715461d-00650e020a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:22.523 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:22.523 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:22.523 DEBUG:boto:path=/ffzdqu-11/?prefix=dummy 2023-09-22T21:07:22.524 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=dummy 2023-09-22T21:07:22.524 DEBUG:boto:Method: GET 2023-09-22T21:07:22.524 DEBUG:boto:Path: /ffzdqu-11/?prefix=dummy 2023-09-22T21:07:22.524 DEBUG:boto:Data: 2023-09-22T21:07:22.524 DEBUG:boto:Headers: {} 2023-09-22T21:07:22.524 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:22.524 DEBUG:boto:Port: 8001 2023-09-22T21:07:22.524 DEBUG:boto:Params: {} 2023-09-22T21:07:22.525 DEBUG:boto:Token: None 2023-09-22T21:07:22.525 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:22 GMT /ffzdqu-11/ 2023-09-22T21:07:22.525 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/MPgNNjVxi6otw8pJlZW2jJx0tU= 2023-09-22T21:07:22.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/MPgNNjVxi6otw8pJlZW2jJx0tU=', 'Content-Length': '0'} 2023-09-22T21:07:22.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000875ea6f790cf4070-00650e020a-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:22.570 DEBUG:boto:b'ffzdqu-11dummy1000falsedummy2023-09-22T21:06:06.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:22.644 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:22.789 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:22.789 DEBUG:boto:path=/ffzdqu-11/?prefix=dummy 2023-09-22T21:07:22.789 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=dummy 2023-09-22T21:07:22.789 DEBUG:boto:Method: GET 2023-09-22T21:07:22.789 DEBUG:boto:Path: /ffzdqu-11/?prefix=dummy 2023-09-22T21:07:22.789 DEBUG:boto:Data: 2023-09-22T21:07:22.789 DEBUG:boto:Headers: {} 2023-09-22T21:07:22.790 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:22.790 DEBUG:boto:Port: 8000 2023-09-22T21:07:22.790 DEBUG:boto:Params: {} 2023-09-22T21:07:22.790 DEBUG:boto:Token: None 2023-09-22T21:07:22.790 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:22 GMT /ffzdqu-11/ 2023-09-22T21:07:22.790 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/MPgNNjVxi6otw8pJlZW2jJx0tU= 2023-09-22T21:07:22.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/MPgNNjVxi6otw8pJlZW2jJx0tU=', 'Content-Length': '0'} 2023-09-22T21:07:22.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec9c1467c392a140-00650e020a-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:22.796 DEBUG:boto:b'ffzdqu-11dummy1000falsedummy2023-09-22T21:06:06.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:22.796 DEBUG:boto:path=/ffzdqu-11/dummy 2023-09-22T21:07:22.797 DEBUG:boto:auth_path=/ffzdqu-11/dummy 2023-09-22T21:07:22.797 DEBUG:boto:Method: HEAD 2023-09-22T21:07:22.797 DEBUG:boto:Path: /ffzdqu-11/dummy 2023-09-22T21:07:22.797 DEBUG:boto:Data: 2023-09-22T21:07:22.797 DEBUG:boto:Headers: {} 2023-09-22T21:07:22.797 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:22.797 DEBUG:boto:Port: 8001 2023-09-22T21:07:22.797 DEBUG:boto:Params: {} 2023-09-22T21:07:22.797 DEBUG:boto:Token: None 2023-09-22T21:07:22.798 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:22 GMT /ffzdqu-11/dummy 2023-09-22T21:07:22.798 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gfU8BTxfx3Q+KtNBn4nLfyKbFes= 2023-09-22T21:07:22.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gfU8BTxfx3Q+KtNBn4nLfyKbFes=', 'Content-Length': '0'} 2023-09-22T21:07:22.801 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('x-amz-version-id', 'DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dca087466ba3a0ac-00650e020a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:22.802 DEBUG:boto:path=/ffzdqu-11/dummy 2023-09-22T21:07:22.802 DEBUG:boto:auth_path=/ffzdqu-11/dummy 2023-09-22T21:07:22.802 DEBUG:boto:Method: HEAD 2023-09-22T21:07:22.802 DEBUG:boto:Path: /ffzdqu-11/dummy 2023-09-22T21:07:22.803 DEBUG:boto:Data: 2023-09-22T21:07:22.803 DEBUG:boto:Headers: {} 2023-09-22T21:07:22.803 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:22.803 DEBUG:boto:Port: 8000 2023-09-22T21:07:22.803 DEBUG:boto:Params: {} 2023-09-22T21:07:22.803 DEBUG:boto:Token: None 2023-09-22T21:07:22.803 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:22 GMT /ffzdqu-11/dummy 2023-09-22T21:07:22.804 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gfU8BTxfx3Q+KtNBn4nLfyKbFes= 2023-09-22T21:07:22.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gfU8BTxfx3Q+KtNBn4nLfyKbFes=', 'Content-Length': '0'} 2023-09-22T21:07:22.807 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('x-amz-version-id', 'DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d78fddda6e205e42-00650e020a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:22.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-22T21:07:22.807 DEBUG:boto:path=/ffzdqu-11/dummy 2023-09-22T21:07:22.808 DEBUG:boto:auth_path=/ffzdqu-11/dummy 2023-09-22T21:07:22.808 DEBUG:boto:path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.808 DEBUG:boto:auth_path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.808 DEBUG:boto:Method: GET 2023-09-22T21:07:22.808 DEBUG:boto:Path: /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.808 DEBUG:boto:Data: 2023-09-22T21:07:22.808 DEBUG:boto:Headers: {} 2023-09-22T21:07:22.808 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:22.808 DEBUG:boto:Port: 8001 2023-09-22T21:07:22.808 DEBUG:boto:Params: {} 2023-09-22T21:07:22.809 DEBUG:boto:Token: None 2023-09-22T21:07:22.809 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:22 GMT /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.809 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6tLlq/NuW6/NQqob2q/gHY757Ks= 2023-09-22T21:07:22.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6tLlq/NuW6/NQqob2q/gHY757Ks=', 'Content-Length': '0'} 2023-09-22T21:07:22.812 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('x-amz-version-id', 'DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bbe93029250ff66f-00650e020a-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:22.812 DEBUG:boto:path=/ffzdqu-11/dummy 2023-09-22T21:07:22.812 DEBUG:boto:auth_path=/ffzdqu-11/dummy 2023-09-22T21:07:22.812 DEBUG:boto:path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.812 DEBUG:boto:auth_path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.812 DEBUG:boto:Method: GET 2023-09-22T21:07:22.812 DEBUG:boto:Path: /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.812 DEBUG:boto:Data: 2023-09-22T21:07:22.813 DEBUG:boto:Headers: {} 2023-09-22T21:07:22.813 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:22.813 DEBUG:boto:Port: 8000 2023-09-22T21:07:22.813 DEBUG:boto:Params: {} 2023-09-22T21:07:22.813 DEBUG:boto:Token: None 2023-09-22T21:07:22.813 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:22 GMT /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:22.813 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6tLlq/NuW6/NQqob2q/gHY757Ks= 2023-09-22T21:07:22.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6tLlq/NuW6/NQqob2q/gHY757Ks=', 'Content-Length': '0'} 2023-09-22T21:07:22.817 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('x-amz-version-id', 'DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000def00e22c8a37448-00650e020a-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:22.817 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:22.818 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:22.818 DEBUG:boto:path=/ffzdqu-11/?prefix=dummy 2023-09-22T21:07:22.818 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=dummy 2023-09-22T21:07:22.818 DEBUG:boto:Method: GET 2023-09-22T21:07:22.818 DEBUG:boto:Path: /ffzdqu-11/?prefix=dummy 2023-09-22T21:07:22.818 DEBUG:boto:Data: 2023-09-22T21:07:22.818 DEBUG:boto:Headers: {} 2023-09-22T21:07:22.818 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:22.818 DEBUG:boto:Port: 8001 2023-09-22T21:07:22.818 DEBUG:boto:Params: {} 2023-09-22T21:07:22.819 DEBUG:boto:Token: None 2023-09-22T21:07:22.819 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:22 GMT /ffzdqu-11/ 2023-09-22T21:07:22.819 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/MPgNNjVxi6otw8pJlZW2jJx0tU= 2023-09-22T21:07:22.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/MPgNNjVxi6otw8pJlZW2jJx0tU=', 'Content-Length': '0'} 2023-09-22T21:07:22.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e25b2b7c15b9ad3-00650e020a-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:22.824 DEBUG:boto:b'ffzdqu-11dummy1000falsedummy2023-09-22T21:06:06.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:22.825 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:22.825 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:22.825 DEBUG:boto:path=/ffzdqu-11/?prefix=dummy 2023-09-22T21:07:22.825 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=dummy 2023-09-22T21:07:22.825 DEBUG:boto:Method: GET 2023-09-22T21:07:22.825 DEBUG:boto:Path: /ffzdqu-11/?prefix=dummy 2023-09-22T21:07:22.825 DEBUG:boto:Data: 2023-09-22T21:07:22.825 DEBUG:boto:Headers: {} 2023-09-22T21:07:22.825 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:22.825 DEBUG:boto:Port: 8000 2023-09-22T21:07:22.826 DEBUG:boto:Params: {} 2023-09-22T21:07:22.826 DEBUG:boto:Token: None 2023-09-22T21:07:22.826 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:22 GMT /ffzdqu-11/ 2023-09-22T21:07:22.826 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/MPgNNjVxi6otw8pJlZW2jJx0tU= 2023-09-22T21:07:22.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/MPgNNjVxi6otw8pJlZW2jJx0tU=', 'Content-Length': '0'} 2023-09-22T21:07:22.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009acc0ff12b279aa2-00650e020a-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:22.874 DEBUG:boto:b'ffzdqu-11dummy1000falsedummy2023-09-22T21:06:06.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:23.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-22T21:07:23.313 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:07:23.313 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:07:23.313 DEBUG:boto:path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.313 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.313 DEBUG:boto:Method: GET 2023-09-22T21:07:23.313 DEBUG:boto:Path: /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.313 DEBUG:boto:Data: 2023-09-22T21:07:23.313 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.313 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:23.314 DEBUG:boto:Port: 8001 2023-09-22T21:07:23.314 DEBUG:boto:Params: {} 2023-09-22T21:07:23.314 DEBUG:boto:Token: None 2023-09-22T21:07:23.314 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.314 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Zrx54d3nxSps3MtnGqoY33yXBNc= 2023-09-22T21:07:23.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Zrx54d3nxSps3MtnGqoY33yXBNc=', 'Content-Length': '0'} 2023-09-22T21:07:23.317 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016c1edacd0ae9400-00650e020b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.321 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:07:23.321 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:07:23.321 DEBUG:boto:path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.321 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.322 DEBUG:boto:Method: GET 2023-09-22T21:07:23.322 DEBUG:boto:Path: /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.322 DEBUG:boto:Data: 2023-09-22T21:07:23.322 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.322 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:23.323 DEBUG:boto:Port: 8000 2023-09-22T21:07:23.323 DEBUG:boto:Params: {} 2023-09-22T21:07:23.323 DEBUG:boto:Token: None 2023-09-22T21:07:23.323 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.323 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Zrx54d3nxSps3MtnGqoY33yXBNc= 2023-09-22T21:07:23.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Zrx54d3nxSps3MtnGqoY33yXBNc=', 'Content-Length': '0'} 2023-09-22T21:07:23.327 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000761eb7dbdcf93ff-00650e020b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.327 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:07:23.327 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:07:23.327 DEBUG:boto:path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.328 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.328 DEBUG:boto:Method: HEAD 2023-09-22T21:07:23.328 DEBUG:boto:Path: /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.328 DEBUG:boto:Data: 2023-09-22T21:07:23.328 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.328 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:23.329 DEBUG:boto:Port: 8001 2023-09-22T21:07:23.329 DEBUG:boto:Params: {} 2023-09-22T21:07:23.329 DEBUG:boto:Token: None 2023-09-22T21:07:23.329 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.329 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:V0Gp3VlUGmBc8QJgs3j+aUFvuoA= 2023-09-22T21:07:23.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:V0Gp3VlUGmBc8QJgs3j+aUFvuoA=', 'Content-Length': '0'} 2023-09-22T21:07:23.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003a8bf6568b84798-00650e020b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.332 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:07:23.332 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:07:23.333 DEBUG:boto:path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.333 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.333 DEBUG:boto:Method: HEAD 2023-09-22T21:07:23.333 DEBUG:boto:Path: /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.333 DEBUG:boto:Data: 2023-09-22T21:07:23.333 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.333 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:23.333 DEBUG:boto:Port: 8000 2023-09-22T21:07:23.334 DEBUG:boto:Params: {} 2023-09-22T21:07:23.334 DEBUG:boto:Token: None 2023-09-22T21:07:23.334 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.334 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:V0Gp3VlUGmBc8QJgs3j+aUFvuoA= 2023-09-22T21:07:23.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:V0Gp3VlUGmBc8QJgs3j+aUFvuoA=', 'Content-Length': '0'} 2023-09-22T21:07:23.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000603e9506d85b1257-00650e020b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-22T21:07:23.337 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:07:23.337 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:07:23.337 DEBUG:boto:path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.338 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.338 DEBUG:boto:Method: GET 2023-09-22T21:07:23.338 DEBUG:boto:Path: /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.338 DEBUG:boto:Data: 2023-09-22T21:07:23.338 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.338 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:23.339 DEBUG:boto:Port: 8001 2023-09-22T21:07:23.339 DEBUG:boto:Params: {} 2023-09-22T21:07:23.339 DEBUG:boto:Token: None 2023-09-22T21:07:23.339 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.339 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Zrx54d3nxSps3MtnGqoY33yXBNc= 2023-09-22T21:07:23.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Zrx54d3nxSps3MtnGqoY33yXBNc=', 'Content-Length': '0'} 2023-09-22T21:07:23.342 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000062b59fc77948077-00650e020b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.342 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:07:23.342 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:07:23.342 DEBUG:boto:path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.343 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.343 DEBUG:boto:Method: GET 2023-09-22T21:07:23.343 DEBUG:boto:Path: /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.343 DEBUG:boto:Data: 2023-09-22T21:07:23.343 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.343 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:23.343 DEBUG:boto:Port: 8000 2023-09-22T21:07:23.343 DEBUG:boto:Params: {} 2023-09-22T21:07:23.344 DEBUG:boto:Token: None 2023-09-22T21:07:23.344 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.344 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Zrx54d3nxSps3MtnGqoY33yXBNc= 2023-09-22T21:07:23.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Zrx54d3nxSps3MtnGqoY33yXBNc=', 'Content-Length': '0'} 2023-09-22T21:07:23.347 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef970e958e247f9b-00650e020b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.347 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:23.347 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:23.348 DEBUG:boto:path=/ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:23.348 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:23.348 DEBUG:boto:Method: GET 2023-09-22T21:07:23.348 DEBUG:boto:Path: /ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:23.348 DEBUG:boto:Data: 2023-09-22T21:07:23.348 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.348 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:23.349 DEBUG:boto:Port: 8001 2023-09-22T21:07:23.349 DEBUG:boto:Params: {} 2023-09-22T21:07:23.349 DEBUG:boto:Token: None 2023-09-22T21:07:23.349 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/ 2023-09-22T21:07:23.349 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eDcNL1v+8wJIvRd79WF4sfZpN7c= 2023-09-22T21:07:23.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eDcNL1v+8wJIvRd79WF4sfZpN7c=', 'Content-Length': '0'} 2023-09-22T21:07:23.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066a32950294f92a5-00650e020b-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.355 DEBUG:boto:b'ffzdqu-11obj-a11000falseobj-a12023-09-22T21:06:41.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:23.355 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:23.356 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:23.356 DEBUG:boto:path=/ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:23.356 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:23.356 DEBUG:boto:Method: GET 2023-09-22T21:07:23.356 DEBUG:boto:Path: /ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:23.356 DEBUG:boto:Data: 2023-09-22T21:07:23.356 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.356 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:23.357 DEBUG:boto:Port: 8000 2023-09-22T21:07:23.357 DEBUG:boto:Params: {} 2023-09-22T21:07:23.357 DEBUG:boto:Token: None 2023-09-22T21:07:23.357 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/ 2023-09-22T21:07:23.357 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eDcNL1v+8wJIvRd79WF4sfZpN7c= 2023-09-22T21:07:23.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eDcNL1v+8wJIvRd79WF4sfZpN7c=', 'Content-Length': '0'} 2023-09-22T21:07:23.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006230e52667665bf1-00650e020b-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.363 DEBUG:boto:b'ffzdqu-11obj-a11000falseobj-a12023-09-22T21:06:41.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:23.363 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:07:23.363 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:07:23.364 DEBUG:boto:Method: HEAD 2023-09-22T21:07:23.364 DEBUG:boto:Path: /ffzdqu-11/obj-a1 2023-09-22T21:07:23.364 DEBUG:boto:Data: 2023-09-22T21:07:23.364 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.364 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:23.364 DEBUG:boto:Port: 8001 2023-09-22T21:07:23.365 DEBUG:boto:Params: {} 2023-09-22T21:07:23.365 DEBUG:boto:Token: None 2023-09-22T21:07:23.365 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/obj-a1 2023-09-22T21:07:23.365 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LeqLA19NTFDL1Gr0kSqBR+VHKdU= 2023-09-22T21:07:23.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LeqLA19NTFDL1Gr0kSqBR+VHKdU=', 'Content-Length': '0'} 2023-09-22T21:07:23.369 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010d00804503cf1b2-00650e020b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.369 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:07:23.369 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:07:23.369 DEBUG:boto:Method: HEAD 2023-09-22T21:07:23.370 DEBUG:boto:Path: /ffzdqu-11/obj-a1 2023-09-22T21:07:23.370 DEBUG:boto:Data: 2023-09-22T21:07:23.370 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.370 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:23.370 DEBUG:boto:Port: 8000 2023-09-22T21:07:23.370 DEBUG:boto:Params: {} 2023-09-22T21:07:23.370 DEBUG:boto:Token: None 2023-09-22T21:07:23.370 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/obj-a1 2023-09-22T21:07:23.370 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LeqLA19NTFDL1Gr0kSqBR+VHKdU= 2023-09-22T21:07:23.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LeqLA19NTFDL1Gr0kSqBR+VHKdU=', 'Content-Length': '0'} 2023-09-22T21:07:23.374 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fca4b92ed26469de-00650e020b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-22T21:07:23.374 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:07:23.375 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:07:23.375 DEBUG:boto:path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.375 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.375 DEBUG:boto:Method: GET 2023-09-22T21:07:23.375 DEBUG:boto:Path: /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.375 DEBUG:boto:Data: 2023-09-22T21:07:23.375 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.375 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:23.375 DEBUG:boto:Port: 8001 2023-09-22T21:07:23.375 DEBUG:boto:Params: {} 2023-09-22T21:07:23.376 DEBUG:boto:Token: None 2023-09-22T21:07:23.376 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.376 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Zrx54d3nxSps3MtnGqoY33yXBNc= 2023-09-22T21:07:23.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Zrx54d3nxSps3MtnGqoY33yXBNc=', 'Content-Length': '0'} 2023-09-22T21:07:23.378 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0926c39d7b76703-00650e020b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.422 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:07:23.423 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:07:23.423 DEBUG:boto:path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.423 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.423 DEBUG:boto:Method: GET 2023-09-22T21:07:23.423 DEBUG:boto:Path: /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.423 DEBUG:boto:Data: 2023-09-22T21:07:23.423 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.423 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:23.424 DEBUG:boto:Port: 8000 2023-09-22T21:07:23.424 DEBUG:boto:Params: {} 2023-09-22T21:07:23.424 DEBUG:boto:Token: None 2023-09-22T21:07:23.424 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:23.424 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Zrx54d3nxSps3MtnGqoY33yXBNc= 2023-09-22T21:07:23.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Zrx54d3nxSps3MtnGqoY33yXBNc=', 'Content-Length': '0'} 2023-09-22T21:07:23.429 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e98cc37e8abdffc3-00650e020b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.430 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:23.430 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:23.430 DEBUG:boto:path=/ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:23.430 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:23.430 DEBUG:boto:Method: GET 2023-09-22T21:07:23.431 DEBUG:boto:Path: /ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:23.431 DEBUG:boto:Data: 2023-09-22T21:07:23.431 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.431 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:23.431 DEBUG:boto:Port: 8001 2023-09-22T21:07:23.431 DEBUG:boto:Params: {} 2023-09-22T21:07:23.431 DEBUG:boto:Token: None 2023-09-22T21:07:23.432 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/ 2023-09-22T21:07:23.432 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eDcNL1v+8wJIvRd79WF4sfZpN7c= 2023-09-22T21:07:23.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eDcNL1v+8wJIvRd79WF4sfZpN7c=', 'Content-Length': '0'} 2023-09-22T21:07:23.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f38574322f0cf860-00650e020b-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.437 DEBUG:boto:b'ffzdqu-11obj-a11000falseobj-a12023-09-22T21:06:41.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:23.438 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:23.438 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:23.438 DEBUG:boto:path=/ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:23.438 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:23.438 DEBUG:boto:Method: GET 2023-09-22T21:07:23.438 DEBUG:boto:Path: /ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:23.438 DEBUG:boto:Data: 2023-09-22T21:07:23.439 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.439 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:23.439 DEBUG:boto:Port: 8000 2023-09-22T21:07:23.439 DEBUG:boto:Params: {} 2023-09-22T21:07:23.439 DEBUG:boto:Token: None 2023-09-22T21:07:23.439 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/ 2023-09-22T21:07:23.439 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eDcNL1v+8wJIvRd79WF4sfZpN7c= 2023-09-22T21:07:23.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eDcNL1v+8wJIvRd79WF4sfZpN7c=', 'Content-Length': '0'} 2023-09-22T21:07:23.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d683ad68e3d047e0-00650e020b-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.444 DEBUG:boto:b'ffzdqu-11obj-a11000falseobj-a12023-09-22T21:06:41.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:23.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-22T21:07:23.445 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:07:23.445 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:07:23.445 DEBUG:boto:path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.445 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.445 DEBUG:boto:Method: GET 2023-09-22T21:07:23.445 DEBUG:boto:Path: /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.445 DEBUG:boto:Data: 2023-09-22T21:07:23.445 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.446 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:23.446 DEBUG:boto:Port: 8001 2023-09-22T21:07:23.446 DEBUG:boto:Params: {} 2023-09-22T21:07:23.446 DEBUG:boto:Token: None 2023-09-22T21:07:23.446 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.446 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ENpgw9YCSg2lWEqX2db2v2mqpR4= 2023-09-22T21:07:23.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ENpgw9YCSg2lWEqX2db2v2mqpR4=', 'Content-Length': '0'} 2023-09-22T21:07:23.449 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000735933d7fd5242ff-00650e020b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.449 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:07:23.449 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:07:23.450 DEBUG:boto:path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.450 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.450 DEBUG:boto:Method: GET 2023-09-22T21:07:23.450 DEBUG:boto:Path: /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.450 DEBUG:boto:Data: 2023-09-22T21:07:23.450 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.450 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:23.450 DEBUG:boto:Port: 8000 2023-09-22T21:07:23.450 DEBUG:boto:Params: {} 2023-09-22T21:07:23.450 DEBUG:boto:Token: None 2023-09-22T21:07:23.451 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.451 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ENpgw9YCSg2lWEqX2db2v2mqpR4= 2023-09-22T21:07:23.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ENpgw9YCSg2lWEqX2db2v2mqpR4=', 'Content-Length': '0'} 2023-09-22T21:07:23.455 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a84ad644f149f8c2-00650e020b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.455 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:07:23.455 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:07:23.455 DEBUG:boto:path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.455 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.455 DEBUG:boto:Method: HEAD 2023-09-22T21:07:23.456 DEBUG:boto:Path: /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.456 DEBUG:boto:Data: 2023-09-22T21:07:23.456 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:23.456 DEBUG:boto:Port: 8001 2023-09-22T21:07:23.456 DEBUG:boto:Params: {} 2023-09-22T21:07:23.456 DEBUG:boto:Token: None 2023-09-22T21:07:23.456 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.457 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MjOiIfNTn6e26wqqRY0YyvYBkxY= 2023-09-22T21:07:23.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MjOiIfNTn6e26wqqRY0YyvYBkxY=', 'Content-Length': '0'} 2023-09-22T21:07:23.459 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098a5ed3f08603954-00650e020b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.460 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:07:23.460 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:07:23.460 DEBUG:boto:path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.460 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.460 DEBUG:boto:Method: HEAD 2023-09-22T21:07:23.460 DEBUG:boto:Path: /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.460 DEBUG:boto:Data: 2023-09-22T21:07:23.460 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.461 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:23.461 DEBUG:boto:Port: 8000 2023-09-22T21:07:23.461 DEBUG:boto:Params: {} 2023-09-22T21:07:23.461 DEBUG:boto:Token: None 2023-09-22T21:07:23.461 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.461 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MjOiIfNTn6e26wqqRY0YyvYBkxY= 2023-09-22T21:07:23.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MjOiIfNTn6e26wqqRY0YyvYBkxY=', 'Content-Length': '0'} 2023-09-22T21:07:23.464 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000599115bb22109d2b-00650e020b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-22T21:07:23.464 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:07:23.464 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:07:23.465 DEBUG:boto:path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.465 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.465 DEBUG:boto:Method: GET 2023-09-22T21:07:23.465 DEBUG:boto:Path: /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.465 DEBUG:boto:Data: 2023-09-22T21:07:23.465 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.465 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:23.465 DEBUG:boto:Port: 8001 2023-09-22T21:07:23.465 DEBUG:boto:Params: {} 2023-09-22T21:07:23.466 DEBUG:boto:Token: None 2023-09-22T21:07:23.466 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.466 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ENpgw9YCSg2lWEqX2db2v2mqpR4= 2023-09-22T21:07:23.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ENpgw9YCSg2lWEqX2db2v2mqpR4=', 'Content-Length': '0'} 2023-09-22T21:07:23.469 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017487f8bd439dc00-00650e020b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.510 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:07:23.510 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:07:23.511 DEBUG:boto:path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.511 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.511 DEBUG:boto:Method: GET 2023-09-22T21:07:23.511 DEBUG:boto:Path: /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.511 DEBUG:boto:Data: 2023-09-22T21:07:23.511 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:23.511 DEBUG:boto:Port: 8000 2023-09-22T21:07:23.512 DEBUG:boto:Params: {} 2023-09-22T21:07:23.512 DEBUG:boto:Token: None 2023-09-22T21:07:23.512 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.512 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ENpgw9YCSg2lWEqX2db2v2mqpR4= 2023-09-22T21:07:23.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ENpgw9YCSg2lWEqX2db2v2mqpR4=', 'Content-Length': '0'} 2023-09-22T21:07:23.515 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af6df1d8f88cb4dc-00650e020b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.515 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:23.516 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:23.516 DEBUG:boto:path=/ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:23.516 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:23.516 DEBUG:boto:Method: GET 2023-09-22T21:07:23.516 DEBUG:boto:Path: /ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:23.516 DEBUG:boto:Data: 2023-09-22T21:07:23.516 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.516 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:23.516 DEBUG:boto:Port: 8001 2023-09-22T21:07:23.516 DEBUG:boto:Params: {} 2023-09-22T21:07:23.517 DEBUG:boto:Token: None 2023-09-22T21:07:23.517 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/ 2023-09-22T21:07:23.517 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eDcNL1v+8wJIvRd79WF4sfZpN7c= 2023-09-22T21:07:23.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eDcNL1v+8wJIvRd79WF4sfZpN7c=', 'Content-Length': '0'} 2023-09-22T21:07:23.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000251588153cdcbe67-00650e020b-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.523 DEBUG:boto:b'ffzdqu-11obj-a21000falseobj-a22023-09-22T21:06:41.742Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:23.523 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:23.523 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:23.523 DEBUG:boto:path=/ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:23.523 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:23.523 DEBUG:boto:Method: GET 2023-09-22T21:07:23.524 DEBUG:boto:Path: /ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:23.524 DEBUG:boto:Data: 2023-09-22T21:07:23.524 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.524 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:23.524 DEBUG:boto:Port: 8000 2023-09-22T21:07:23.524 DEBUG:boto:Params: {} 2023-09-22T21:07:23.524 DEBUG:boto:Token: None 2023-09-22T21:07:23.524 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/ 2023-09-22T21:07:23.524 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eDcNL1v+8wJIvRd79WF4sfZpN7c= 2023-09-22T21:07:23.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eDcNL1v+8wJIvRd79WF4sfZpN7c=', 'Content-Length': '0'} 2023-09-22T21:07:23.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000201e8e586bacd1ad-00650e020b-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.530 DEBUG:boto:b'ffzdqu-11obj-a21000falseobj-a22023-09-22T21:06:41.742Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:23.530 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:07:23.531 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:07:23.531 DEBUG:boto:Method: HEAD 2023-09-22T21:07:23.531 DEBUG:boto:Path: /ffzdqu-11/obj-a2 2023-09-22T21:07:23.531 DEBUG:boto:Data: 2023-09-22T21:07:23.531 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.531 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:23.532 DEBUG:boto:Port: 8001 2023-09-22T21:07:23.532 DEBUG:boto:Params: {} 2023-09-22T21:07:23.532 DEBUG:boto:Token: None 2023-09-22T21:07:23.532 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/obj-a2 2023-09-22T21:07:23.532 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Z7tk61MosKCQjaXX/QI+1W/l04= 2023-09-22T21:07:23.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Z7tk61MosKCQjaXX/QI+1W/l04=', 'Content-Length': '0'} 2023-09-22T21:07:23.535 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000671ac6038112b983-00650e020b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.536 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:07:23.536 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:07:23.536 DEBUG:boto:Method: HEAD 2023-09-22T21:07:23.536 DEBUG:boto:Path: /ffzdqu-11/obj-a2 2023-09-22T21:07:23.536 DEBUG:boto:Data: 2023-09-22T21:07:23.536 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.536 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:23.537 DEBUG:boto:Port: 8000 2023-09-22T21:07:23.537 DEBUG:boto:Params: {} 2023-09-22T21:07:23.537 DEBUG:boto:Token: None 2023-09-22T21:07:23.537 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/obj-a2 2023-09-22T21:07:23.537 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Z7tk61MosKCQjaXX/QI+1W/l04= 2023-09-22T21:07:23.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Z7tk61MosKCQjaXX/QI+1W/l04=', 'Content-Length': '0'} 2023-09-22T21:07:23.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7055e8aa6f244b2-00650e020b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-22T21:07:23.542 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:07:23.542 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:07:23.542 DEBUG:boto:path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.542 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.542 DEBUG:boto:Method: GET 2023-09-22T21:07:23.542 DEBUG:boto:Path: /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.542 DEBUG:boto:Data: 2023-09-22T21:07:23.542 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.543 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:23.543 DEBUG:boto:Port: 8001 2023-09-22T21:07:23.543 DEBUG:boto:Params: {} 2023-09-22T21:07:23.543 DEBUG:boto:Token: None 2023-09-22T21:07:23.543 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.543 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ENpgw9YCSg2lWEqX2db2v2mqpR4= 2023-09-22T21:07:23.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ENpgw9YCSg2lWEqX2db2v2mqpR4=', 'Content-Length': '0'} 2023-09-22T21:07:23.546 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb4c5c60375ae2b4-00650e020b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.586 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:07:23.587 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:07:23.587 DEBUG:boto:path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.587 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.587 DEBUG:boto:Method: GET 2023-09-22T21:07:23.587 DEBUG:boto:Path: /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.587 DEBUG:boto:Data: 2023-09-22T21:07:23.587 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.587 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:23.587 DEBUG:boto:Port: 8000 2023-09-22T21:07:23.588 DEBUG:boto:Params: {} 2023-09-22T21:07:23.588 DEBUG:boto:Token: None 2023-09-22T21:07:23.588 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:23.588 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ENpgw9YCSg2lWEqX2db2v2mqpR4= 2023-09-22T21:07:23.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ENpgw9YCSg2lWEqX2db2v2mqpR4=', 'Content-Length': '0'} 2023-09-22T21:07:23.591 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000286c6657a7b5088d-00650e020b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.591 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:23.592 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:23.592 DEBUG:boto:path=/ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:23.592 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:23.592 DEBUG:boto:Method: GET 2023-09-22T21:07:23.592 DEBUG:boto:Path: /ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:23.592 DEBUG:boto:Data: 2023-09-22T21:07:23.592 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.592 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:23.592 DEBUG:boto:Port: 8001 2023-09-22T21:07:23.592 DEBUG:boto:Params: {} 2023-09-22T21:07:23.593 DEBUG:boto:Token: None 2023-09-22T21:07:23.593 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/ 2023-09-22T21:07:23.593 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eDcNL1v+8wJIvRd79WF4sfZpN7c= 2023-09-22T21:07:23.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eDcNL1v+8wJIvRd79WF4sfZpN7c=', 'Content-Length': '0'} 2023-09-22T21:07:23.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097394f870a4578ae-00650e020b-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.598 DEBUG:boto:b'ffzdqu-11obj-a21000falseobj-a22023-09-22T21:06:41.742Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:23.599 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:23.599 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:23.599 DEBUG:boto:path=/ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:23.599 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:23.599 DEBUG:boto:Method: GET 2023-09-22T21:07:23.599 DEBUG:boto:Path: /ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:23.600 DEBUG:boto:Data: 2023-09-22T21:07:23.600 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.600 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:23.600 DEBUG:boto:Port: 8000 2023-09-22T21:07:23.600 DEBUG:boto:Params: {} 2023-09-22T21:07:23.600 DEBUG:boto:Token: None 2023-09-22T21:07:23.600 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-11/ 2023-09-22T21:07:23.600 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eDcNL1v+8wJIvRd79WF4sfZpN7c= 2023-09-22T21:07:23.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eDcNL1v+8wJIvRd79WF4sfZpN7c=', 'Content-Length': '0'} 2023-09-22T21:07:23.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011dcf8f73d46f204-00650e020b-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.646 DEBUG:boto:b'ffzdqu-11obj-a21000falseobj-a22023-09-22T21:06:41.742Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:23.647 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-11 zones={a2, a1} 2023-09-22T21:07:23.647 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-12 2023-09-22T21:07:23.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:07:23.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:07:23.648 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:07:23.701 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:07:23.701 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:07:23.745 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:23.744+0000 7f5587c6ba40 1 bucket sync caught up with source: 2023-09-22T21:07:23.745 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.3.11, , , 00000000002.11.11, 00000000010.21.6, ] 2023-09-22T21:07:23.745 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000002.3.11, , , 00000000002.11.11, 00000000010.21.6, ] 2023-09-22T21:07:23.745 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:23.744+0000 7f5587c6ba40 0 bucket checkpoint complete 2023-09-22T21:07:23.762 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-12 2023-09-22T21:07:23.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:07:23.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:07:23.763 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:07:23.805 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:07:23.806 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:07:23.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:23.848+0000 7f84b1377a40 1 bucket sync caught up with source: 2023-09-22T21:07:23.846 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000003.18.10, , , 00000000010.82.6, 00000000002.18.11, ] 2023-09-22T21:07:23.847 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000003.18.10, , , 00000000010.82.6, 00000000002.18.11, ] 2023-09-22T21:07:23.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:23.848+0000 7f84b1377a40 0 bucket checkpoint complete 2023-09-22T21:07:23.862 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-12 zones={a2, a1} 2023-09-22T21:07:23.862 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:23.862 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:23.862 DEBUG:boto:Method: HEAD 2023-09-22T21:07:23.862 DEBUG:boto:Path: /ffzdqu-12/ 2023-09-22T21:07:23.862 DEBUG:boto:Data: 2023-09-22T21:07:23.863 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.863 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:23.863 DEBUG:boto:Port: 8001 2023-09-22T21:07:23.863 DEBUG:boto:Params: {} 2023-09-22T21:07:23.863 DEBUG:boto:Token: None 2023-09-22T21:07:23.864 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-12/ 2023-09-22T21:07:23.864 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:282SauuRgjrFBY0GnFqlvXupRD8= 2023-09-22T21:07:23.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:282SauuRgjrFBY0GnFqlvXupRD8=', 'Content-Length': '0'} 2023-09-22T21:07:23.936 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000151349a176414473-00650e020b-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.936 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:23.937 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:23.937 DEBUG:boto:Method: HEAD 2023-09-22T21:07:23.937 DEBUG:boto:Path: /ffzdqu-12/ 2023-09-22T21:07:23.937 DEBUG:boto:Data: 2023-09-22T21:07:23.937 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.937 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:23.937 DEBUG:boto:Port: 8000 2023-09-22T21:07:23.937 DEBUG:boto:Params: {} 2023-09-22T21:07:23.938 DEBUG:boto:Token: None 2023-09-22T21:07:23.938 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-12/ 2023-09-22T21:07:23.938 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:282SauuRgjrFBY0GnFqlvXupRD8= 2023-09-22T21:07:23.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:282SauuRgjrFBY0GnFqlvXupRD8=', 'Content-Length': '0'} 2023-09-22T21:07:23.942 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f31702228ac210f4-00650e020b-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.942 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:07:23.942 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:23.943 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:23.943 DEBUG:boto:path=/ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:23.943 DEBUG:boto:auth_path=/ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:23.943 DEBUG:boto:Method: GET 2023-09-22T21:07:23.943 DEBUG:boto:Path: /ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:23.943 DEBUG:boto:Data: 2023-09-22T21:07:23.943 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.944 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:23.944 DEBUG:boto:Port: 8001 2023-09-22T21:07:23.944 DEBUG:boto:Params: {} 2023-09-22T21:07:23.944 DEBUG:boto:Token: None 2023-09-22T21:07:23.944 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-12/?versions 2023-09-22T21:07:23.945 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7YDR3KuQ8mBYgJl3KW1J3Hg1Vdw= 2023-09-22T21:07:23.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7YDR3KuQ8mBYgJl3KW1J3Hg1Vdw=', 'Content-Length': '0'} 2023-09-22T21:07:23.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3d11c06bbcd18c8-00650e020b-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.949 DEBUG:boto:b'ffzdqu-12999falsedummyboYngPv33ng8HsND7dSXZTE12I7WmDDtrue2023-09-22T21:06:21.418Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1ENtJQRcMHYYCQC9B1Taipgnc1aUE.HLtrue2023-09-22T21:06:41.903Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2u7s2N-8HbpXwMBMoersKLSCgHbn-fnrtrue2023-09-22T21:06:42.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:23.950 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-22T21:07:23.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-09-22T21:07:23.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-09-22T21:07:23.950 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:07:23.950 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:23.951 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:23.951 DEBUG:boto:path=/ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:23.951 DEBUG:boto:auth_path=/ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:23.951 DEBUG:boto:Method: GET 2023-09-22T21:07:23.951 DEBUG:boto:Path: /ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:23.951 DEBUG:boto:Data: 2023-09-22T21:07:23.951 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.951 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:23.951 DEBUG:boto:Port: 8000 2023-09-22T21:07:23.951 DEBUG:boto:Params: {} 2023-09-22T21:07:23.952 DEBUG:boto:Token: None 2023-09-22T21:07:23.952 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-12/?versions 2023-09-22T21:07:23.952 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7YDR3KuQ8mBYgJl3KW1J3Hg1Vdw= 2023-09-22T21:07:23.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7YDR3KuQ8mBYgJl3KW1J3Hg1Vdw=', 'Content-Length': '0'} 2023-09-22T21:07:23.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000967fbdb35f02164d-00650e020b-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.957 DEBUG:boto:b'ffzdqu-12999falsedummyboYngPv33ng8HsND7dSXZTE12I7WmDDtrue2023-09-22T21:06:21.418Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1ENtJQRcMHYYCQC9B1Taipgnc1aUE.HLtrue2023-09-22T21:06:41.903Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2u7s2N-8HbpXwMBMoersKLSCgHbn-fnrtrue2023-09-22T21:06:42.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:23.957 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-22T21:07:23.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-09-22T21:07:23.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-09-22T21:07:23.958 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:23.958 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:23.958 DEBUG:boto:path=/ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:23.958 DEBUG:boto:auth_path=/ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:23.958 DEBUG:boto:Method: GET 2023-09-22T21:07:23.958 DEBUG:boto:Path: /ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:23.959 DEBUG:boto:Data: 2023-09-22T21:07:23.959 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.959 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:23.959 DEBUG:boto:Port: 8001 2023-09-22T21:07:23.959 DEBUG:boto:Params: {} 2023-09-22T21:07:23.959 DEBUG:boto:Token: None 2023-09-22T21:07:23.959 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-12/?versions 2023-09-22T21:07:23.960 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7YDR3KuQ8mBYgJl3KW1J3Hg1Vdw= 2023-09-22T21:07:23.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7YDR3KuQ8mBYgJl3KW1J3Hg1Vdw=', 'Content-Length': '0'} 2023-09-22T21:07:23.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdfe1f04f5585d1c-00650e020b-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.964 DEBUG:boto:b'ffzdqu-12999falsedummyboYngPv33ng8HsND7dSXZTE12I7WmDDtrue2023-09-22T21:06:21.418Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1ENtJQRcMHYYCQC9B1Taipgnc1aUE.HLtrue2023-09-22T21:06:41.903Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2u7s2N-8HbpXwMBMoersKLSCgHbn-fnrtrue2023-09-22T21:06:42.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:23.965 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:23.965 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:23.965 DEBUG:boto:path=/ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:23.965 DEBUG:boto:auth_path=/ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:23.965 DEBUG:boto:Method: GET 2023-09-22T21:07:23.965 DEBUG:boto:Path: /ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:23.965 DEBUG:boto:Data: 2023-09-22T21:07:23.965 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.965 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:23.966 DEBUG:boto:Port: 8000 2023-09-22T21:07:23.966 DEBUG:boto:Params: {} 2023-09-22T21:07:23.966 DEBUG:boto:Token: None 2023-09-22T21:07:23.966 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-12/?versions 2023-09-22T21:07:23.966 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7YDR3KuQ8mBYgJl3KW1J3Hg1Vdw= 2023-09-22T21:07:23.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7YDR3KuQ8mBYgJl3KW1J3Hg1Vdw=', 'Content-Length': '0'} 2023-09-22T21:07:23.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009db0cec88eef0e65-00650e020b-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.971 DEBUG:boto:b'ffzdqu-12999falsedummyboYngPv33ng8HsND7dSXZTE12I7WmDDtrue2023-09-22T21:06:21.418Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1ENtJQRcMHYYCQC9B1Taipgnc1aUE.HLtrue2023-09-22T21:06:41.903Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2u7s2N-8HbpXwMBMoersKLSCgHbn-fnrtrue2023-09-22T21:06:42.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:23.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-22T21:07:23.971 DEBUG:boto:path=/ffzdqu-12/dummy 2023-09-22T21:07:23.971 DEBUG:boto:auth_path=/ffzdqu-12/dummy 2023-09-22T21:07:23.972 DEBUG:boto:path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.972 DEBUG:boto:auth_path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.972 DEBUG:boto:Method: GET 2023-09-22T21:07:23.972 DEBUG:boto:Path: /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.972 DEBUG:boto:Data: 2023-09-22T21:07:23.972 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.972 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:23.972 DEBUG:boto:Port: 8001 2023-09-22T21:07:23.972 DEBUG:boto:Params: {} 2023-09-22T21:07:23.972 DEBUG:boto:Token: None 2023-09-22T21:07:23.973 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.973 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vSIcs52xaMR91vRj835vPRWjwsk= 2023-09-22T21:07:23.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vSIcs52xaMR91vRj835vPRWjwsk=', 'Content-Length': '0'} 2023-09-22T21:07:23.975 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('x-amz-version-id', 'boYngPv33ng8HsND7dSXZTE12I7WmDD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dd39870262bac1a1-00650e020b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.976 DEBUG:boto:path=/ffzdqu-12/dummy 2023-09-22T21:07:23.976 DEBUG:boto:auth_path=/ffzdqu-12/dummy 2023-09-22T21:07:23.976 DEBUG:boto:path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.976 DEBUG:boto:auth_path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.976 DEBUG:boto:Method: GET 2023-09-22T21:07:23.976 DEBUG:boto:Path: /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.976 DEBUG:boto:Data: 2023-09-22T21:07:23.976 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.977 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:23.977 DEBUG:boto:Port: 8000 2023-09-22T21:07:23.977 DEBUG:boto:Params: {} 2023-09-22T21:07:23.977 DEBUG:boto:Token: None 2023-09-22T21:07:23.977 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.977 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vSIcs52xaMR91vRj835vPRWjwsk= 2023-09-22T21:07:23.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vSIcs52xaMR91vRj835vPRWjwsk=', 'Content-Length': '0'} 2023-09-22T21:07:23.980 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('x-amz-version-id', 'boYngPv33ng8HsND7dSXZTE12I7WmDD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000048f9e847fa1b4e43-00650e020b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.981 DEBUG:boto:path=/ffzdqu-12/dummy 2023-09-22T21:07:23.981 DEBUG:boto:auth_path=/ffzdqu-12/dummy 2023-09-22T21:07:23.981 DEBUG:boto:path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.981 DEBUG:boto:auth_path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.981 DEBUG:boto:Method: HEAD 2023-09-22T21:07:23.981 DEBUG:boto:Path: /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.981 DEBUG:boto:Data: 2023-09-22T21:07:23.981 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.981 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:23.982 DEBUG:boto:Port: 8001 2023-09-22T21:07:23.982 DEBUG:boto:Params: {} 2023-09-22T21:07:23.982 DEBUG:boto:Token: None 2023-09-22T21:07:23.982 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.982 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aGk55s2tdD6vv9un5ifTRX+Ogfo= 2023-09-22T21:07:23.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aGk55s2tdD6vv9un5ifTRX+Ogfo=', 'Content-Length': '0'} 2023-09-22T21:07:23.984 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('x-amz-version-id', 'boYngPv33ng8HsND7dSXZTE12I7WmDD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aa9fe572a4a4a6cd-00650e020b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.984 DEBUG:boto:path=/ffzdqu-12/dummy 2023-09-22T21:07:23.985 DEBUG:boto:auth_path=/ffzdqu-12/dummy 2023-09-22T21:07:23.985 DEBUG:boto:path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.985 DEBUG:boto:auth_path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.985 DEBUG:boto:Method: HEAD 2023-09-22T21:07:23.985 DEBUG:boto:Path: /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.985 DEBUG:boto:Data: 2023-09-22T21:07:23.985 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.985 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:23.985 DEBUG:boto:Port: 8000 2023-09-22T21:07:23.985 DEBUG:boto:Params: {} 2023-09-22T21:07:23.986 DEBUG:boto:Token: None 2023-09-22T21:07:23.986 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.986 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aGk55s2tdD6vv9un5ifTRX+Ogfo= 2023-09-22T21:07:23.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aGk55s2tdD6vv9un5ifTRX+Ogfo=', 'Content-Length': '0'} 2023-09-22T21:07:23.988 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('x-amz-version-id', 'boYngPv33ng8HsND7dSXZTE12I7WmDD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bdda7d77af5f7c9e-00650e020b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-22T21:07:23.989 DEBUG:boto:path=/ffzdqu-12/dummy 2023-09-22T21:07:23.989 DEBUG:boto:auth_path=/ffzdqu-12/dummy 2023-09-22T21:07:23.989 DEBUG:boto:path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.989 DEBUG:boto:auth_path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.989 DEBUG:boto:Method: GET 2023-09-22T21:07:23.989 DEBUG:boto:Path: /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.990 DEBUG:boto:Data: 2023-09-22T21:07:23.990 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.990 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:23.990 DEBUG:boto:Port: 8001 2023-09-22T21:07:23.990 DEBUG:boto:Params: {} 2023-09-22T21:07:23.990 DEBUG:boto:Token: None 2023-09-22T21:07:23.990 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.990 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vSIcs52xaMR91vRj835vPRWjwsk= 2023-09-22T21:07:23.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vSIcs52xaMR91vRj835vPRWjwsk=', 'Content-Length': '0'} 2023-09-22T21:07:23.993 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('x-amz-version-id', 'boYngPv33ng8HsND7dSXZTE12I7WmDD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000282bb3a4f9900932-00650e020b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.993 DEBUG:boto:path=/ffzdqu-12/dummy 2023-09-22T21:07:23.993 DEBUG:boto:auth_path=/ffzdqu-12/dummy 2023-09-22T21:07:23.993 DEBUG:boto:path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.993 DEBUG:boto:auth_path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.993 DEBUG:boto:Method: GET 2023-09-22T21:07:23.994 DEBUG:boto:Path: /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.994 DEBUG:boto:Data: 2023-09-22T21:07:23.994 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.994 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:23.994 DEBUG:boto:Port: 8000 2023-09-22T21:07:23.994 DEBUG:boto:Params: {} 2023-09-22T21:07:23.994 DEBUG:boto:Token: None 2023-09-22T21:07:23.994 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:23.994 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vSIcs52xaMR91vRj835vPRWjwsk= 2023-09-22T21:07:23.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vSIcs52xaMR91vRj835vPRWjwsk=', 'Content-Length': '0'} 2023-09-22T21:07:23.997 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('x-amz-version-id', 'boYngPv33ng8HsND7dSXZTE12I7WmDD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c89f47737f9e2029-00650e020b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:23.997 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:23.997 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:23.997 DEBUG:boto:path=/ffzdqu-12/?prefix=dummy 2023-09-22T21:07:23.997 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=dummy 2023-09-22T21:07:23.997 DEBUG:boto:Method: GET 2023-09-22T21:07:23.998 DEBUG:boto:Path: /ffzdqu-12/?prefix=dummy 2023-09-22T21:07:23.998 DEBUG:boto:Data: 2023-09-22T21:07:23.998 DEBUG:boto:Headers: {} 2023-09-22T21:07:23.998 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:23.998 DEBUG:boto:Port: 8001 2023-09-22T21:07:23.998 DEBUG:boto:Params: {} 2023-09-22T21:07:23.998 DEBUG:boto:Token: None 2023-09-22T21:07:23.998 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:23 GMT /ffzdqu-12/ 2023-09-22T21:07:23.999 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kTUNaydkRY95dRXZyHLjZ1jm98Y= 2023-09-22T21:07:23.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kTUNaydkRY95dRXZyHLjZ1jm98Y=', 'Content-Length': '0'} 2023-09-22T21:07:24.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000788a68c32ba016e6-00650e020c-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.042 DEBUG:boto:b'ffzdqu-12dummy1000falsedummy2023-09-22T21:06:21.418Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:24.043 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:24.043 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:24.043 DEBUG:boto:path=/ffzdqu-12/?prefix=dummy 2023-09-22T21:07:24.043 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=dummy 2023-09-22T21:07:24.043 DEBUG:boto:Method: GET 2023-09-22T21:07:24.043 DEBUG:boto:Path: /ffzdqu-12/?prefix=dummy 2023-09-22T21:07:24.043 DEBUG:boto:Data: 2023-09-22T21:07:24.043 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.044 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.044 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.044 DEBUG:boto:Params: {} 2023-09-22T21:07:24.044 DEBUG:boto:Token: None 2023-09-22T21:07:24.044 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/ 2023-09-22T21:07:24.044 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k= 2023-09-22T21:07:24.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k=', 'Content-Length': '0'} 2023-09-22T21:07:24.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a063b4c8e2578d29-00650e020c-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.049 DEBUG:boto:b'ffzdqu-12dummy1000falsedummy2023-09-22T21:06:21.418Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:24.050 DEBUG:boto:path=/ffzdqu-12/dummy 2023-09-22T21:07:24.050 DEBUG:boto:auth_path=/ffzdqu-12/dummy 2023-09-22T21:07:24.050 DEBUG:boto:Method: HEAD 2023-09-22T21:07:24.050 DEBUG:boto:Path: /ffzdqu-12/dummy 2023-09-22T21:07:24.050 DEBUG:boto:Data: 2023-09-22T21:07:24.050 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.050 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.050 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.050 DEBUG:boto:Params: {} 2023-09-22T21:07:24.051 DEBUG:boto:Token: None 2023-09-22T21:07:24.051 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/dummy 2023-09-22T21:07:24.051 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pXWj0NdiE1uJj28SnfW5qgiz2eo= 2023-09-22T21:07:24.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pXWj0NdiE1uJj28SnfW5qgiz2eo=', 'Content-Length': '0'} 2023-09-22T21:07:24.054 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('x-amz-version-id', 'boYngPv33ng8HsND7dSXZTE12I7WmDD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bd42b3465707b533-00650e020c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.054 DEBUG:boto:path=/ffzdqu-12/dummy 2023-09-22T21:07:24.055 DEBUG:boto:auth_path=/ffzdqu-12/dummy 2023-09-22T21:07:24.055 DEBUG:boto:Method: HEAD 2023-09-22T21:07:24.055 DEBUG:boto:Path: /ffzdqu-12/dummy 2023-09-22T21:07:24.055 DEBUG:boto:Data: 2023-09-22T21:07:24.055 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.055 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.055 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.055 DEBUG:boto:Params: {} 2023-09-22T21:07:24.055 DEBUG:boto:Token: None 2023-09-22T21:07:24.056 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/dummy 2023-09-22T21:07:24.056 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pXWj0NdiE1uJj28SnfW5qgiz2eo= 2023-09-22T21:07:24.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pXWj0NdiE1uJj28SnfW5qgiz2eo=', 'Content-Length': '0'} 2023-09-22T21:07:24.059 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('x-amz-version-id', 'boYngPv33ng8HsND7dSXZTE12I7WmDD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000068f0a7907b511527-00650e020c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-22T21:07:24.059 DEBUG:boto:path=/ffzdqu-12/dummy 2023-09-22T21:07:24.060 DEBUG:boto:auth_path=/ffzdqu-12/dummy 2023-09-22T21:07:24.060 DEBUG:boto:path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:24.060 DEBUG:boto:auth_path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:24.060 DEBUG:boto:Method: GET 2023-09-22T21:07:24.060 DEBUG:boto:Path: /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:24.060 DEBUG:boto:Data: 2023-09-22T21:07:24.060 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.060 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.060 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.060 DEBUG:boto:Params: {} 2023-09-22T21:07:24.061 DEBUG:boto:Token: None 2023-09-22T21:07:24.061 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:24.061 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WDtM5oSRukZxLeEtw7+2xWIDObM= 2023-09-22T21:07:24.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WDtM5oSRukZxLeEtw7+2xWIDObM=', 'Content-Length': '0'} 2023-09-22T21:07:24.063 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('x-amz-version-id', 'boYngPv33ng8HsND7dSXZTE12I7WmDD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000817edea4da9cd96b-00650e020c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.064 DEBUG:boto:path=/ffzdqu-12/dummy 2023-09-22T21:07:24.064 DEBUG:boto:auth_path=/ffzdqu-12/dummy 2023-09-22T21:07:24.064 DEBUG:boto:path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:24.064 DEBUG:boto:auth_path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:24.064 DEBUG:boto:Method: GET 2023-09-22T21:07:24.064 DEBUG:boto:Path: /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:24.064 DEBUG:boto:Data: 2023-09-22T21:07:24.064 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.065 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.065 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.065 DEBUG:boto:Params: {} 2023-09-22T21:07:24.065 DEBUG:boto:Token: None 2023-09-22T21:07:24.065 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:24.065 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WDtM5oSRukZxLeEtw7+2xWIDObM= 2023-09-22T21:07:24.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WDtM5oSRukZxLeEtw7+2xWIDObM=', 'Content-Length': '0'} 2023-09-22T21:07:24.068 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('x-amz-version-id', 'boYngPv33ng8HsND7dSXZTE12I7WmDD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ccc43f391d17d400-00650e020c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.068 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:24.068 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:24.069 DEBUG:boto:path=/ffzdqu-12/?prefix=dummy 2023-09-22T21:07:24.069 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=dummy 2023-09-22T21:07:24.069 DEBUG:boto:Method: GET 2023-09-22T21:07:24.069 DEBUG:boto:Path: /ffzdqu-12/?prefix=dummy 2023-09-22T21:07:24.069 DEBUG:boto:Data: 2023-09-22T21:07:24.069 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.069 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.069 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.069 DEBUG:boto:Params: {} 2023-09-22T21:07:24.069 DEBUG:boto:Token: None 2023-09-22T21:07:24.070 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/ 2023-09-22T21:07:24.070 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k= 2023-09-22T21:07:24.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k=', 'Content-Length': '0'} 2023-09-22T21:07:24.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004028f0b4ff033a14-00650e020c-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.118 DEBUG:boto:b'ffzdqu-12dummy1000falsedummy2023-09-22T21:06:21.418Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:24.119 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:24.119 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:24.119 DEBUG:boto:path=/ffzdqu-12/?prefix=dummy 2023-09-22T21:07:24.119 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=dummy 2023-09-22T21:07:24.119 DEBUG:boto:Method: GET 2023-09-22T21:07:24.119 DEBUG:boto:Path: /ffzdqu-12/?prefix=dummy 2023-09-22T21:07:24.119 DEBUG:boto:Data: 2023-09-22T21:07:24.119 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.119 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.120 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.120 DEBUG:boto:Params: {} 2023-09-22T21:07:24.120 DEBUG:boto:Token: None 2023-09-22T21:07:24.120 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/ 2023-09-22T21:07:24.120 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k= 2023-09-22T21:07:24.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k=', 'Content-Length': '0'} 2023-09-22T21:07:24.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b58f9170f2efa516-00650e020c-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.125 DEBUG:boto:b'ffzdqu-12dummy1000falsedummy2023-09-22T21:06:21.418Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:24.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-22T21:07:24.126 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.126 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.126 DEBUG:boto:path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.126 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.126 DEBUG:boto:Method: GET 2023-09-22T21:07:24.126 DEBUG:boto:Path: /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.126 DEBUG:boto:Data: 2023-09-22T21:07:24.126 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.126 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.126 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.127 DEBUG:boto:Params: {} 2023-09-22T21:07:24.127 DEBUG:boto:Token: None 2023-09-22T21:07:24.127 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.127 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sFEmDyvAiPZ88mASpWF0OwzvVro= 2023-09-22T21:07:24.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sFEmDyvAiPZ88mASpWF0OwzvVro=', 'Content-Length': '0'} 2023-09-22T21:07:24.130 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', 'ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008428911575fa43d1-00650e020c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.130 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.130 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.130 DEBUG:boto:path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.131 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.131 DEBUG:boto:Method: GET 2023-09-22T21:07:24.131 DEBUG:boto:Path: /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.131 DEBUG:boto:Data: 2023-09-22T21:07:24.131 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.131 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.131 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.131 DEBUG:boto:Params: {} 2023-09-22T21:07:24.131 DEBUG:boto:Token: None 2023-09-22T21:07:24.132 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.132 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sFEmDyvAiPZ88mASpWF0OwzvVro= 2023-09-22T21:07:24.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sFEmDyvAiPZ88mASpWF0OwzvVro=', 'Content-Length': '0'} 2023-09-22T21:07:24.134 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', 'ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072da969a0e421491-00650e020c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.178 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.179 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.179 DEBUG:boto:path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.179 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.179 DEBUG:boto:Method: HEAD 2023-09-22T21:07:24.179 DEBUG:boto:Path: /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.179 DEBUG:boto:Data: 2023-09-22T21:07:24.179 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.179 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.179 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.180 DEBUG:boto:Params: {} 2023-09-22T21:07:24.180 DEBUG:boto:Token: None 2023-09-22T21:07:24.180 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.180 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XQbeQUTt4tScFjNR80AkB1cPP10= 2023-09-22T21:07:24.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XQbeQUTt4tScFjNR80AkB1cPP10=', 'Content-Length': '0'} 2023-09-22T21:07:24.183 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', 'ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000380b84f818601478-00650e020c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.184 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.184 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.184 DEBUG:boto:path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.184 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.184 DEBUG:boto:Method: HEAD 2023-09-22T21:07:24.184 DEBUG:boto:Path: /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.185 DEBUG:boto:Data: 2023-09-22T21:07:24.185 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.185 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.185 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.185 DEBUG:boto:Params: {} 2023-09-22T21:07:24.185 DEBUG:boto:Token: None 2023-09-22T21:07:24.185 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.186 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XQbeQUTt4tScFjNR80AkB1cPP10= 2023-09-22T21:07:24.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XQbeQUTt4tScFjNR80AkB1cPP10=', 'Content-Length': '0'} 2023-09-22T21:07:24.189 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', 'ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a20c1d7b0aaa16c-00650e020c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-22T21:07:24.189 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.189 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.189 DEBUG:boto:path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.189 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.190 DEBUG:boto:Method: GET 2023-09-22T21:07:24.190 DEBUG:boto:Path: /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.190 DEBUG:boto:Data: 2023-09-22T21:07:24.190 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.190 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.190 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.190 DEBUG:boto:Params: {} 2023-09-22T21:07:24.190 DEBUG:boto:Token: None 2023-09-22T21:07:24.190 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.191 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sFEmDyvAiPZ88mASpWF0OwzvVro= 2023-09-22T21:07:24.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sFEmDyvAiPZ88mASpWF0OwzvVro=', 'Content-Length': '0'} 2023-09-22T21:07:24.193 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', 'ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b024f621d4a54db2-00650e020c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.234 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.235 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.235 DEBUG:boto:path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.235 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.235 DEBUG:boto:Method: GET 2023-09-22T21:07:24.235 DEBUG:boto:Path: /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.235 DEBUG:boto:Data: 2023-09-22T21:07:24.235 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.235 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.235 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.236 DEBUG:boto:Params: {} 2023-09-22T21:07:24.236 DEBUG:boto:Token: None 2023-09-22T21:07:24.236 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.236 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sFEmDyvAiPZ88mASpWF0OwzvVro= 2023-09-22T21:07:24.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sFEmDyvAiPZ88mASpWF0OwzvVro=', 'Content-Length': '0'} 2023-09-22T21:07:24.239 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', 'ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d997d470979c69b3-00650e020c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.239 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:24.239 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:24.240 DEBUG:boto:path=/ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:24.240 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:24.240 DEBUG:boto:Method: GET 2023-09-22T21:07:24.240 DEBUG:boto:Path: /ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:24.240 DEBUG:boto:Data: 2023-09-22T21:07:24.240 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.240 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.240 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.241 DEBUG:boto:Params: {} 2023-09-22T21:07:24.241 DEBUG:boto:Token: None 2023-09-22T21:07:24.241 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/ 2023-09-22T21:07:24.241 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k= 2023-09-22T21:07:24.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k=', 'Content-Length': '0'} 2023-09-22T21:07:24.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caf908159b22b766-00650e020c-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.286 DEBUG:boto:b'ffzdqu-12obj-a11000falseobj-a12023-09-22T21:06:41.903Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:24.287 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:24.287 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:24.287 DEBUG:boto:path=/ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:24.287 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:24.287 DEBUG:boto:Method: GET 2023-09-22T21:07:24.287 DEBUG:boto:Path: /ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:24.287 DEBUG:boto:Data: 2023-09-22T21:07:24.288 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.288 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.288 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.288 DEBUG:boto:Params: {} 2023-09-22T21:07:24.288 DEBUG:boto:Token: None 2023-09-22T21:07:24.288 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/ 2023-09-22T21:07:24.288 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k= 2023-09-22T21:07:24.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k=', 'Content-Length': '0'} 2023-09-22T21:07:24.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1e0b77f68df1398-00650e020c-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.293 DEBUG:boto:b'ffzdqu-12obj-a11000falseobj-a12023-09-22T21:06:41.903Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:24.293 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.293 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.293 DEBUG:boto:Method: HEAD 2023-09-22T21:07:24.294 DEBUG:boto:Path: /ffzdqu-12/obj-a1 2023-09-22T21:07:24.294 DEBUG:boto:Data: 2023-09-22T21:07:24.294 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.294 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.294 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.294 DEBUG:boto:Params: {} 2023-09-22T21:07:24.294 DEBUG:boto:Token: None 2023-09-22T21:07:24.294 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/obj-a1 2023-09-22T21:07:24.294 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tnYn+Iq/OHRlwyYfk2JE8idWXRI= 2023-09-22T21:07:24.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tnYn+Iq/OHRlwyYfk2JE8idWXRI=', 'Content-Length': '0'} 2023-09-22T21:07:24.298 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', 'ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c406e88d3302b951-00650e020c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.298 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.299 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.299 DEBUG:boto:Method: HEAD 2023-09-22T21:07:24.299 DEBUG:boto:Path: /ffzdqu-12/obj-a1 2023-09-22T21:07:24.299 DEBUG:boto:Data: 2023-09-22T21:07:24.299 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.299 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.299 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.299 DEBUG:boto:Params: {} 2023-09-22T21:07:24.299 DEBUG:boto:Token: None 2023-09-22T21:07:24.300 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/obj-a1 2023-09-22T21:07:24.300 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tnYn+Iq/OHRlwyYfk2JE8idWXRI= 2023-09-22T21:07:24.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tnYn+Iq/OHRlwyYfk2JE8idWXRI=', 'Content-Length': '0'} 2023-09-22T21:07:24.303 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', 'ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8cebbdb75e70ed3-00650e020c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-22T21:07:24.304 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.304 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.304 DEBUG:boto:path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.304 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.304 DEBUG:boto:Method: GET 2023-09-22T21:07:24.304 DEBUG:boto:Path: /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.304 DEBUG:boto:Data: 2023-09-22T21:07:24.304 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.304 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.305 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.305 DEBUG:boto:Params: {} 2023-09-22T21:07:24.305 DEBUG:boto:Token: None 2023-09-22T21:07:24.305 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.305 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sFEmDyvAiPZ88mASpWF0OwzvVro= 2023-09-22T21:07:24.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sFEmDyvAiPZ88mASpWF0OwzvVro=', 'Content-Length': '0'} 2023-09-22T21:07:24.308 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', 'ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c49a0ddba9b2d5f-00650e020c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.350 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.350 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.350 DEBUG:boto:path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.351 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.351 DEBUG:boto:Method: GET 2023-09-22T21:07:24.351 DEBUG:boto:Path: /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.351 DEBUG:boto:Data: 2023-09-22T21:07:24.351 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.351 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.351 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.351 DEBUG:boto:Params: {} 2023-09-22T21:07:24.351 DEBUG:boto:Token: None 2023-09-22T21:07:24.352 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:24.352 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sFEmDyvAiPZ88mASpWF0OwzvVro= 2023-09-22T21:07:24.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sFEmDyvAiPZ88mASpWF0OwzvVro=', 'Content-Length': '0'} 2023-09-22T21:07:24.354 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', 'ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ce959bba45f3197-00650e020c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.355 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:24.355 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:24.355 DEBUG:boto:path=/ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:24.355 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:24.355 DEBUG:boto:Method: GET 2023-09-22T21:07:24.355 DEBUG:boto:Path: /ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:24.355 DEBUG:boto:Data: 2023-09-22T21:07:24.356 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.356 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.356 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.356 DEBUG:boto:Params: {} 2023-09-22T21:07:24.356 DEBUG:boto:Token: None 2023-09-22T21:07:24.356 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/ 2023-09-22T21:07:24.356 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k= 2023-09-22T21:07:24.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k=', 'Content-Length': '0'} 2023-09-22T21:07:24.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c231f38ab2139d37-00650e020c-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.361 DEBUG:boto:b'ffzdqu-12obj-a11000falseobj-a12023-09-22T21:06:41.903Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:24.362 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:24.362 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:24.362 DEBUG:boto:path=/ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:24.362 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:24.362 DEBUG:boto:Method: GET 2023-09-22T21:07:24.362 DEBUG:boto:Path: /ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:24.362 DEBUG:boto:Data: 2023-09-22T21:07:24.363 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.363 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.363 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.363 DEBUG:boto:Params: {} 2023-09-22T21:07:24.363 DEBUG:boto:Token: None 2023-09-22T21:07:24.363 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/ 2023-09-22T21:07:24.363 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k= 2023-09-22T21:07:24.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k=', 'Content-Length': '0'} 2023-09-22T21:07:24.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000156654cd0cf9c1db-00650e020c-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.410 DEBUG:boto:b'ffzdqu-12obj-a11000falseobj-a12023-09-22T21:06:41.903Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:24.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-22T21:07:24.411 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.411 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.411 DEBUG:boto:path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.411 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.411 DEBUG:boto:Method: GET 2023-09-22T21:07:24.411 DEBUG:boto:Path: /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.412 DEBUG:boto:Data: 2023-09-22T21:07:24.412 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.412 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.412 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.412 DEBUG:boto:Params: {} 2023-09-22T21:07:24.412 DEBUG:boto:Token: None 2023-09-22T21:07:24.412 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.412 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cwUQW/3CeFpk7GXV06J6cbBm56g= 2023-09-22T21:07:24.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cwUQW/3CeFpk7GXV06J6cbBm56g=', 'Content-Length': '0'} 2023-09-22T21:07:24.415 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('x-amz-version-id', 'u7s2N-8HbpXwMBMoersKLSCgHbn-fnr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009777936378feec2b-00650e020c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.416 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.416 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.416 DEBUG:boto:path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.416 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.416 DEBUG:boto:Method: GET 2023-09-22T21:07:24.416 DEBUG:boto:Path: /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.416 DEBUG:boto:Data: 2023-09-22T21:07:24.416 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.416 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.417 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.417 DEBUG:boto:Params: {} 2023-09-22T21:07:24.417 DEBUG:boto:Token: None 2023-09-22T21:07:24.417 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.417 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cwUQW/3CeFpk7GXV06J6cbBm56g= 2023-09-22T21:07:24.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cwUQW/3CeFpk7GXV06J6cbBm56g=', 'Content-Length': '0'} 2023-09-22T21:07:24.420 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('x-amz-version-id', 'u7s2N-8HbpXwMBMoersKLSCgHbn-fnr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e40d45d6b6ba965-00650e020c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.421 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.421 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.421 DEBUG:boto:path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.421 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.421 DEBUG:boto:Method: HEAD 2023-09-22T21:07:24.421 DEBUG:boto:Path: /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.421 DEBUG:boto:Data: 2023-09-22T21:07:24.421 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.422 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.422 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.422 DEBUG:boto:Params: {} 2023-09-22T21:07:24.422 DEBUG:boto:Token: None 2023-09-22T21:07:24.422 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.422 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TbUV3reXnW4/igo10CR+iEu/KaM= 2023-09-22T21:07:24.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TbUV3reXnW4/igo10CR+iEu/KaM=', 'Content-Length': '0'} 2023-09-22T21:07:24.425 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('x-amz-version-id', 'u7s2N-8HbpXwMBMoersKLSCgHbn-fnr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077603f11c21a33e5-00650e020c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.425 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.425 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.426 DEBUG:boto:path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.426 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.426 DEBUG:boto:Method: HEAD 2023-09-22T21:07:24.426 DEBUG:boto:Path: /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.426 DEBUG:boto:Data: 2023-09-22T21:07:24.426 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.426 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.426 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.426 DEBUG:boto:Params: {} 2023-09-22T21:07:24.427 DEBUG:boto:Token: None 2023-09-22T21:07:24.427 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.427 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TbUV3reXnW4/igo10CR+iEu/KaM= 2023-09-22T21:07:24.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TbUV3reXnW4/igo10CR+iEu/KaM=', 'Content-Length': '0'} 2023-09-22T21:07:24.429 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('x-amz-version-id', 'u7s2N-8HbpXwMBMoersKLSCgHbn-fnr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1d91df5c31af2d9-00650e020c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-22T21:07:24.430 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.430 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.430 DEBUG:boto:path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.430 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.430 DEBUG:boto:Method: GET 2023-09-22T21:07:24.430 DEBUG:boto:Path: /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.430 DEBUG:boto:Data: 2023-09-22T21:07:24.431 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.431 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.431 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.431 DEBUG:boto:Params: {} 2023-09-22T21:07:24.431 DEBUG:boto:Token: None 2023-09-22T21:07:24.431 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.431 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cwUQW/3CeFpk7GXV06J6cbBm56g= 2023-09-22T21:07:24.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cwUQW/3CeFpk7GXV06J6cbBm56g=', 'Content-Length': '0'} 2023-09-22T21:07:24.434 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('x-amz-version-id', 'u7s2N-8HbpXwMBMoersKLSCgHbn-fnr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fcf1a017c2c86487-00650e020c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.478 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.478 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.479 DEBUG:boto:path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.479 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.479 DEBUG:boto:Method: GET 2023-09-22T21:07:24.479 DEBUG:boto:Path: /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.479 DEBUG:boto:Data: 2023-09-22T21:07:24.479 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.479 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.479 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.479 DEBUG:boto:Params: {} 2023-09-22T21:07:24.480 DEBUG:boto:Token: None 2023-09-22T21:07:24.480 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.480 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cwUQW/3CeFpk7GXV06J6cbBm56g= 2023-09-22T21:07:24.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cwUQW/3CeFpk7GXV06J6cbBm56g=', 'Content-Length': '0'} 2023-09-22T21:07:24.483 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('x-amz-version-id', 'u7s2N-8HbpXwMBMoersKLSCgHbn-fnr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008862405a3356ff73-00650e020c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.483 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:24.483 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:24.484 DEBUG:boto:path=/ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:24.484 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:24.484 DEBUG:boto:Method: GET 2023-09-22T21:07:24.484 DEBUG:boto:Path: /ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:24.484 DEBUG:boto:Data: 2023-09-22T21:07:24.484 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.484 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.484 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.484 DEBUG:boto:Params: {} 2023-09-22T21:07:24.485 DEBUG:boto:Token: None 2023-09-22T21:07:24.485 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/ 2023-09-22T21:07:24.485 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k= 2023-09-22T21:07:24.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k=', 'Content-Length': '0'} 2023-09-22T21:07:24.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031602bd1fc39d6a5-00650e020c-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.489 DEBUG:boto:b'ffzdqu-12obj-a21000falseobj-a22023-09-22T21:06:42.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:24.490 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:24.490 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:24.490 DEBUG:boto:path=/ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:24.490 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:24.490 DEBUG:boto:Method: GET 2023-09-22T21:07:24.490 DEBUG:boto:Path: /ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:24.490 DEBUG:boto:Data: 2023-09-22T21:07:24.491 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.491 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.491 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.491 DEBUG:boto:Params: {} 2023-09-22T21:07:24.491 DEBUG:boto:Token: None 2023-09-22T21:07:24.491 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/ 2023-09-22T21:07:24.491 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k= 2023-09-22T21:07:24.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k=', 'Content-Length': '0'} 2023-09-22T21:07:24.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e97e29e88dedd858-00650e020c-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.538 DEBUG:boto:b'ffzdqu-12obj-a21000falseobj-a22023-09-22T21:06:42.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:24.541 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.541 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.541 DEBUG:boto:Method: HEAD 2023-09-22T21:07:24.541 DEBUG:boto:Path: /ffzdqu-12/obj-a2 2023-09-22T21:07:24.541 DEBUG:boto:Data: 2023-09-22T21:07:24.541 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.541 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.541 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.542 DEBUG:boto:Params: {} 2023-09-22T21:07:24.542 DEBUG:boto:Token: None 2023-09-22T21:07:24.542 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/obj-a2 2023-09-22T21:07:24.542 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qYhapQ7sF0PwWlv32fuvqG6Gktg= 2023-09-22T21:07:24.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qYhapQ7sF0PwWlv32fuvqG6Gktg=', 'Content-Length': '0'} 2023-09-22T21:07:24.546 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('x-amz-version-id', 'u7s2N-8HbpXwMBMoersKLSCgHbn-fnr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004dda6145c0e8050a-00650e020c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.546 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.546 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.547 DEBUG:boto:Method: HEAD 2023-09-22T21:07:24.547 DEBUG:boto:Path: /ffzdqu-12/obj-a2 2023-09-22T21:07:24.547 DEBUG:boto:Data: 2023-09-22T21:07:24.547 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.547 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.547 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.547 DEBUG:boto:Params: {} 2023-09-22T21:07:24.547 DEBUG:boto:Token: None 2023-09-22T21:07:24.547 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/obj-a2 2023-09-22T21:07:24.548 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qYhapQ7sF0PwWlv32fuvqG6Gktg= 2023-09-22T21:07:24.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qYhapQ7sF0PwWlv32fuvqG6Gktg=', 'Content-Length': '0'} 2023-09-22T21:07:24.551 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('x-amz-version-id', 'u7s2N-8HbpXwMBMoersKLSCgHbn-fnr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004cc94207aef84206-00650e020c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-22T21:07:24.552 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.552 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.552 DEBUG:boto:path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.552 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.552 DEBUG:boto:Method: GET 2023-09-22T21:07:24.552 DEBUG:boto:Path: /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.553 DEBUG:boto:Data: 2023-09-22T21:07:24.553 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.553 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.553 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.553 DEBUG:boto:Params: {} 2023-09-22T21:07:24.553 DEBUG:boto:Token: None 2023-09-22T21:07:24.553 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.553 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cwUQW/3CeFpk7GXV06J6cbBm56g= 2023-09-22T21:07:24.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cwUQW/3CeFpk7GXV06J6cbBm56g=', 'Content-Length': '0'} 2023-09-22T21:07:24.556 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('x-amz-version-id', 'u7s2N-8HbpXwMBMoersKLSCgHbn-fnr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2764c8c9f61d0f3-00650e020c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.556 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.557 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.557 DEBUG:boto:path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.557 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.557 DEBUG:boto:Method: GET 2023-09-22T21:07:24.557 DEBUG:boto:Path: /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.557 DEBUG:boto:Data: 2023-09-22T21:07:24.557 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.557 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.557 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.558 DEBUG:boto:Params: {} 2023-09-22T21:07:24.558 DEBUG:boto:Token: None 2023-09-22T21:07:24.558 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:24.558 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cwUQW/3CeFpk7GXV06J6cbBm56g= 2023-09-22T21:07:24.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cwUQW/3CeFpk7GXV06J6cbBm56g=', 'Content-Length': '0'} 2023-09-22T21:07:24.561 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('x-amz-version-id', 'u7s2N-8HbpXwMBMoersKLSCgHbn-fnr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9612023a9882fb0-00650e020c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.561 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:24.561 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:24.562 DEBUG:boto:path=/ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:24.562 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:24.562 DEBUG:boto:Method: GET 2023-09-22T21:07:24.562 DEBUG:boto:Path: /ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:24.562 DEBUG:boto:Data: 2023-09-22T21:07:24.562 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.562 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.562 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.562 DEBUG:boto:Params: {} 2023-09-22T21:07:24.563 DEBUG:boto:Token: None 2023-09-22T21:07:24.563 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/ 2023-09-22T21:07:24.563 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k= 2023-09-22T21:07:24.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k=', 'Content-Length': '0'} 2023-09-22T21:07:24.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019c1c7541b0437ff-00650e020c-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.568 DEBUG:boto:b'ffzdqu-12obj-a21000falseobj-a22023-09-22T21:06:42.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:24.568 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:24.568 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:24.568 DEBUG:boto:path=/ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:24.568 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:24.568 DEBUG:boto:Method: GET 2023-09-22T21:07:24.569 DEBUG:boto:Path: /ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:24.569 DEBUG:boto:Data: 2023-09-22T21:07:24.569 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.569 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.569 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.569 DEBUG:boto:Params: {} 2023-09-22T21:07:24.569 DEBUG:boto:Token: None 2023-09-22T21:07:24.569 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/ 2023-09-22T21:07:24.569 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k= 2023-09-22T21:07:24.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EWv75vYwdRltSz/ZmWEO96qzF5k=', 'Content-Length': '0'} 2023-09-22T21:07:24.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006153f5ca190ee20-00650e020c-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.574 DEBUG:boto:b'ffzdqu-12obj-a21000falseobj-a22023-09-22T21:06:42.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:24.575 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-12 zones={a2, a1} 2023-09-22T21:07:24.575 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:24.575 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:24.575 DEBUG:boto:Method: HEAD 2023-09-22T21:07:24.575 DEBUG:boto:Path: /ffzdqu-11/ 2023-09-22T21:07:24.575 DEBUG:boto:Data: 2023-09-22T21:07:24.575 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.576 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.576 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.576 DEBUG:boto:Params: {} 2023-09-22T21:07:24.576 DEBUG:boto:Token: None 2023-09-22T21:07:24.576 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-11/ 2023-09-22T21:07:24.576 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mhnYmw1ezAJrcakNzxWUEE/Nll4= 2023-09-22T21:07:24.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mhnYmw1ezAJrcakNzxWUEE/Nll4=', 'Content-Length': '0'} 2023-09-22T21:07:24.580 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c73e75888e387ea-00650e020c-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.580 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:24.581 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:24.581 DEBUG:boto:path=/ffzdqu-11/?versions&max-keys=999&prefix=obj-a1 2023-09-22T21:07:24.581 DEBUG:boto:auth_path=/ffzdqu-11/?versions&max-keys=999&prefix=obj-a1 2023-09-22T21:07:24.581 DEBUG:boto:Method: GET 2023-09-22T21:07:24.581 DEBUG:boto:Path: /ffzdqu-11/?versions&max-keys=999&prefix=obj-a1 2023-09-22T21:07:24.581 DEBUG:boto:Data: 2023-09-22T21:07:24.581 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.581 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.581 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.581 DEBUG:boto:Params: {} 2023-09-22T21:07:24.582 DEBUG:boto:Token: None 2023-09-22T21:07:24.582 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-11/?versions 2023-09-22T21:07:24.582 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RoeVFuENbITMIwvtZJRfG8Qyy00= 2023-09-22T21:07:24.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RoeVFuENbITMIwvtZJRfG8Qyy00=', 'Content-Length': '0'} 2023-09-22T21:07:24.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003913a4188ca6b66c-00650e020c-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.630 DEBUG:boto:b'ffzdqu-11obj-a1999falseobj-a18uA2UlGwMDN2SZPYYFpbhdTZDMaHDlctrue2023-09-22T21:06:41.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:24.631 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:07:24.631 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:07:24.631 DEBUG:boto:path=/ffzdqu-11/obj-a1?acl 2023-09-22T21:07:24.631 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1?acl 2023-09-22T21:07:24.631 DEBUG:boto:Method: PUT 2023-09-22T21:07:24.631 DEBUG:boto:Path: /ffzdqu-11/obj-a1?acl 2023-09-22T21:07:24.631 DEBUG:boto:Data: 2023-09-22T21:07:24.631 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-22T21:07:24.631 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.632 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.632 DEBUG:boto:Params: {} 2023-09-22T21:07:24.632 DEBUG:boto:Token: None 2023-09-22T21:07:24.632 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:07:24 GMT x-amz-acl:public-read /ffzdqu-11/obj-a1?acl 2023-09-22T21:07:24.632 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ntSZHeE4Oxbv8kHOrMr0UyYlH2M= 2023-09-22T21:07:24.632 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ntSZHeE4Oxbv8kHOrMr0UyYlH2M=', 'Content-Length': '0'} 2023-09-22T21:07:24.646 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dab0bc5d277dbc56-00650e020c-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.646 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:24.646 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:24.647 DEBUG:boto:Method: HEAD 2023-09-22T21:07:24.647 DEBUG:boto:Path: /ffzdqu-11/ 2023-09-22T21:07:24.647 DEBUG:boto:Data: 2023-09-22T21:07:24.647 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.647 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.647 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.647 DEBUG:boto:Params: {} 2023-09-22T21:07:24.647 DEBUG:boto:Token: None 2023-09-22T21:07:24.647 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-11/ 2023-09-22T21:07:24.648 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mhnYmw1ezAJrcakNzxWUEE/Nll4= 2023-09-22T21:07:24.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mhnYmw1ezAJrcakNzxWUEE/Nll4=', 'Content-Length': '0'} 2023-09-22T21:07:24.652 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d1dc660471f2952-00650e020c-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.652 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:24.652 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:24.652 DEBUG:boto:path=/ffzdqu-11/?versions&max-keys=999&prefix=obj-a2 2023-09-22T21:07:24.652 DEBUG:boto:auth_path=/ffzdqu-11/?versions&max-keys=999&prefix=obj-a2 2023-09-22T21:07:24.653 DEBUG:boto:Method: GET 2023-09-22T21:07:24.653 DEBUG:boto:Path: /ffzdqu-11/?versions&max-keys=999&prefix=obj-a2 2023-09-22T21:07:24.653 DEBUG:boto:Data: 2023-09-22T21:07:24.653 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.653 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.653 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.653 DEBUG:boto:Params: {} 2023-09-22T21:07:24.653 DEBUG:boto:Token: None 2023-09-22T21:07:24.653 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-11/?versions 2023-09-22T21:07:24.654 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RoeVFuENbITMIwvtZJRfG8Qyy00= 2023-09-22T21:07:24.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RoeVFuENbITMIwvtZJRfG8Qyy00=', 'Content-Length': '0'} 2023-09-22T21:07:24.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000471e361404692d53-00650e020c-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.702 DEBUG:boto:b'ffzdqu-11obj-a2999falseobj-a22Ghp8GTpfzBw6qT8Hie8IpGLuimDYjrtrue2023-09-22T21:06:41.742Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:24.703 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:07:24.703 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:07:24.703 DEBUG:boto:path=/ffzdqu-11/obj-a2?acl 2023-09-22T21:07:24.703 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2?acl 2023-09-22T21:07:24.703 DEBUG:boto:Method: PUT 2023-09-22T21:07:24.703 DEBUG:boto:Path: /ffzdqu-11/obj-a2?acl 2023-09-22T21:07:24.703 DEBUG:boto:Data: 2023-09-22T21:07:24.703 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-22T21:07:24.703 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.704 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.704 DEBUG:boto:Params: {} 2023-09-22T21:07:24.704 DEBUG:boto:Token: None 2023-09-22T21:07:24.704 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:07:24 GMT x-amz-acl:public-read /ffzdqu-11/obj-a2?acl 2023-09-22T21:07:24.704 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hiplianPbcKvvD7R/czhUjXLMjk= 2023-09-22T21:07:24.704 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hiplianPbcKvvD7R/czhUjXLMjk=', 'Content-Length': '0'} 2023-09-22T21:07:24.719 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e2ec36f7f86c8696-00650e020c-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.719 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:24.720 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:24.720 DEBUG:boto:Method: HEAD 2023-09-22T21:07:24.720 DEBUG:boto:Path: /ffzdqu-12/ 2023-09-22T21:07:24.720 DEBUG:boto:Data: 2023-09-22T21:07:24.720 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.720 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.720 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.720 DEBUG:boto:Params: {} 2023-09-22T21:07:24.720 DEBUG:boto:Token: None 2023-09-22T21:07:24.721 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/ 2023-09-22T21:07:24.721 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7+WXNZ0MIZbUbPSPKZIMoBhGWbw= 2023-09-22T21:07:24.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7+WXNZ0MIZbUbPSPKZIMoBhGWbw=', 'Content-Length': '0'} 2023-09-22T21:07:24.725 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cecc4f6653b3cbbd-00650e020c-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.725 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:24.725 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:24.725 DEBUG:boto:path=/ffzdqu-12/?versions&max-keys=999&prefix=obj-a1 2023-09-22T21:07:24.725 DEBUG:boto:auth_path=/ffzdqu-12/?versions&max-keys=999&prefix=obj-a1 2023-09-22T21:07:24.725 DEBUG:boto:Method: GET 2023-09-22T21:07:24.725 DEBUG:boto:Path: /ffzdqu-12/?versions&max-keys=999&prefix=obj-a1 2023-09-22T21:07:24.726 DEBUG:boto:Data: 2023-09-22T21:07:24.726 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.726 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.726 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.726 DEBUG:boto:Params: {} 2023-09-22T21:07:24.726 DEBUG:boto:Token: None 2023-09-22T21:07:24.726 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/?versions 2023-09-22T21:07:24.726 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/bu/t/vtXxJFrg2YXvfI+CAuzRA= 2023-09-22T21:07:24.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/bu/t/vtXxJFrg2YXvfI+CAuzRA=', 'Content-Length': '0'} 2023-09-22T21:07:24.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032491d917344b76f-00650e020c-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.732 DEBUG:boto:b'ffzdqu-12obj-a1999falseobj-a1ENtJQRcMHYYCQC9B1Taipgnc1aUE.HLtrue2023-09-22T21:06:41.903Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:24.732 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.732 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:07:24.732 DEBUG:boto:path=/ffzdqu-12/obj-a1?acl 2023-09-22T21:07:24.732 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1?acl 2023-09-22T21:07:24.733 DEBUG:boto:Method: PUT 2023-09-22T21:07:24.733 DEBUG:boto:Path: /ffzdqu-12/obj-a1?acl 2023-09-22T21:07:24.733 DEBUG:boto:Data: 2023-09-22T21:07:24.733 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-22T21:07:24.733 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:24.733 DEBUG:boto:Port: 8000 2023-09-22T21:07:24.733 DEBUG:boto:Params: {} 2023-09-22T21:07:24.733 DEBUG:boto:Token: None 2023-09-22T21:07:24.733 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:07:24 GMT x-amz-acl:public-read /ffzdqu-12/obj-a1?acl 2023-09-22T21:07:24.734 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JIVb/dSQqs+7JhTDw81mCr24s28= 2023-09-22T21:07:24.734 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JIVb/dSQqs+7JhTDw81mCr24s28=', 'Content-Length': '0'} 2023-09-22T21:07:24.749 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007a50beabaceae76d-00650e020c-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.749 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:24.749 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:24.750 DEBUG:boto:Method: HEAD 2023-09-22T21:07:24.750 DEBUG:boto:Path: /ffzdqu-12/ 2023-09-22T21:07:24.750 DEBUG:boto:Data: 2023-09-22T21:07:24.750 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.751 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.751 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.751 DEBUG:boto:Params: {} 2023-09-22T21:07:24.751 DEBUG:boto:Token: None 2023-09-22T21:07:24.751 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/ 2023-09-22T21:07:24.752 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7+WXNZ0MIZbUbPSPKZIMoBhGWbw= 2023-09-22T21:07:24.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7+WXNZ0MIZbUbPSPKZIMoBhGWbw=', 'Content-Length': '0'} 2023-09-22T21:07:24.756 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7a0e260e85ce32d-00650e020c-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.756 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:24.756 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:24.757 DEBUG:boto:path=/ffzdqu-12/?versions&max-keys=999&prefix=obj-a2 2023-09-22T21:07:24.757 DEBUG:boto:auth_path=/ffzdqu-12/?versions&max-keys=999&prefix=obj-a2 2023-09-22T21:07:24.757 DEBUG:boto:Method: GET 2023-09-22T21:07:24.757 DEBUG:boto:Path: /ffzdqu-12/?versions&max-keys=999&prefix=obj-a2 2023-09-22T21:07:24.757 DEBUG:boto:Data: 2023-09-22T21:07:24.757 DEBUG:boto:Headers: {} 2023-09-22T21:07:24.757 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.757 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.757 DEBUG:boto:Params: {} 2023-09-22T21:07:24.758 DEBUG:boto:Token: None 2023-09-22T21:07:24.758 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:24 GMT /ffzdqu-12/?versions 2023-09-22T21:07:24.758 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/bu/t/vtXxJFrg2YXvfI+CAuzRA= 2023-09-22T21:07:24.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/bu/t/vtXxJFrg2YXvfI+CAuzRA=', 'Content-Length': '0'} 2023-09-22T21:07:24.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d270a421f890402c-00650e020c-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.806 DEBUG:boto:b'ffzdqu-12obj-a2999falseobj-a2u7s2N-8HbpXwMBMoersKLSCgHbn-fnrtrue2023-09-22T21:06:42.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:24.807 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.807 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:07:24.807 DEBUG:boto:path=/ffzdqu-12/obj-a2?acl 2023-09-22T21:07:24.807 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2?acl 2023-09-22T21:07:24.807 DEBUG:boto:Method: PUT 2023-09-22T21:07:24.807 DEBUG:boto:Path: /ffzdqu-12/obj-a2?acl 2023-09-22T21:07:24.807 DEBUG:boto:Data: 2023-09-22T21:07:24.807 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-22T21:07:24.807 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:24.808 DEBUG:boto:Port: 8001 2023-09-22T21:07:24.808 DEBUG:boto:Params: {} 2023-09-22T21:07:24.808 DEBUG:boto:Token: None 2023-09-22T21:07:24.808 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:07:24 GMT x-amz-acl:public-read /ffzdqu-12/obj-a2?acl 2023-09-22T21:07:24.808 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XHPfyi3lxFFWv3/IfoANw43qgZA= 2023-09-22T21:07:24.808 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XHPfyi3lxFFWv3/IfoANw43qgZA=', 'Content-Length': '0'} 2023-09-22T21:07:24.823 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006c35e757dcf89ead-00650e020c-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:07:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:24.823 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-11 2023-09-22T21:07:24.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:07:24.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:07:24.823 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:07:24.873 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:07:24.874 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:07:24.918 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:24.919+0000 7f1878e15a40 1 waiting for incremental sync to catch up: 2023-09-22T21:07:24.918 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.7.11, 00000000010.24.6, ] 2023-09-22T21:07:24.918 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000003.10.10, , , 00000000002.7.11, 00000000011.27.6, ] 2023-09-22T21:07:29.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:29.919+0000 7f1878e15a40 1 waiting for incremental sync to catch up: 2023-09-22T21:07:29.921 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.7.11, 00000000010.24.6, ] 2023-09-22T21:07:29.921 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000003.10.10, , , 00000000002.7.11, 00000000011.27.6, ] 2023-09-22T21:07:34.923 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:34.923+0000 7f1878e15a40 1 waiting for incremental sync to catch up: 2023-09-22T21:07:34.923 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.7.11, 00000000010.24.6, ] 2023-09-22T21:07:34.923 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000003.10.10, , , 00000000002.7.11, 00000000011.27.6, ] 2023-09-22T21:07:39.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:39.923+0000 7f1878e15a40 1 bucket sync caught up with source: 2023-09-22T21:07:39.925 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.7.11, 00000000011.27.6, ] 2023-09-22T21:07:39.925 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000003.10.10, , , 00000000002.7.11, 00000000011.27.6, ] 2023-09-22T21:07:39.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:39.923+0000 7f1878e15a40 0 bucket checkpoint complete 2023-09-22T21:07:39.953 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-11 2023-09-22T21:07:39.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:07:39.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:07:39.954 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:07:40.003 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:07:40.003 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:07:40.045 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:40.047+0000 7f4c70eaea40 1 waiting for incremental sync to catch up: 2023-09-22T21:07:40.045 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.22.11, , , 00000000010.62.6, 00000000002.34.11, ] 2023-09-22T21:07:40.046 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000002.22.11, , , 00000000011.85.6, 00000000002.34.11, ] 2023-09-22T21:07:45.047 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:45.047+0000 7f4c70eaea40 1 bucket sync caught up with source: 2023-09-22T21:07:45.048 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.22.11, , , 00000000011.85.6, 00000000002.34.11, ] 2023-09-22T21:07:45.048 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000002.22.11, , , 00000000011.85.6, 00000000002.34.11, ] 2023-09-22T21:07:45.048 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:45.047+0000 7f4c70eaea40 0 bucket checkpoint complete 2023-09-22T21:07:45.079 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-11 zones={a2, a1} 2023-09-22T21:07:45.079 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:45.080 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:45.080 DEBUG:boto:Method: HEAD 2023-09-22T21:07:45.080 DEBUG:boto:Path: /ffzdqu-11/ 2023-09-22T21:07:45.080 DEBUG:boto:Data: 2023-09-22T21:07:45.080 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.080 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.080 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.080 DEBUG:boto:Params: {} 2023-09-22T21:07:45.081 DEBUG:boto:Token: None 2023-09-22T21:07:45.081 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/ 2023-09-22T21:07:45.081 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9fsPXEMuhXXYOYP87Urngl/aZZA= 2023-09-22T21:07:45.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9fsPXEMuhXXYOYP87Urngl/aZZA=', 'Content-Length': '0'} 2023-09-22T21:07:45.085 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fd364799f863e08-00650e0221-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.085 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:45.085 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:45.086 DEBUG:boto:Method: HEAD 2023-09-22T21:07:45.086 DEBUG:boto:Path: /ffzdqu-11/ 2023-09-22T21:07:45.086 DEBUG:boto:Data: 2023-09-22T21:07:45.086 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.086 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.086 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.086 DEBUG:boto:Params: {} 2023-09-22T21:07:45.086 DEBUG:boto:Token: None 2023-09-22T21:07:45.087 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/ 2023-09-22T21:07:45.087 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9fsPXEMuhXXYOYP87Urngl/aZZA= 2023-09-22T21:07:45.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9fsPXEMuhXXYOYP87Urngl/aZZA=', 'Content-Length': '0'} 2023-09-22T21:07:45.090 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c14794877fc7d17-00650e0221-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.090 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:07:45.090 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:45.091 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:45.091 DEBUG:boto:path=/ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:45.091 DEBUG:boto:auth_path=/ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:45.091 DEBUG:boto:Method: GET 2023-09-22T21:07:45.091 DEBUG:boto:Path: /ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:45.091 DEBUG:boto:Data: 2023-09-22T21:07:45.091 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.092 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.092 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.092 DEBUG:boto:Params: {} 2023-09-22T21:07:45.092 DEBUG:boto:Token: None 2023-09-22T21:07:45.092 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/?versions 2023-09-22T21:07:45.092 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y2b68AnpFY4DkNHOxuUe4EnpoiY= 2023-09-22T21:07:45.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y2b68AnpFY4DkNHOxuUe4EnpoiY=', 'Content-Length': '0'} 2023-09-22T21:07:45.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bd54cfb520da375-00650e0221-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.097 DEBUG:boto:b'ffzdqu-11999falsedummyDtjOyJANiXCj7.Vh58J.RwdyLtGvcNxtrue2023-09-22T21:06:06.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a18uA2UlGwMDN2SZPYYFpbhdTZDMaHDlctrue2023-09-22T21:06:41.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a22Ghp8GTpfzBw6qT8Hie8IpGLuimDYjrtrue2023-09-22T21:06:41.742Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:45.098 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-22T21:07:45.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-09-22T21:07:45.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-09-22T21:07:45.098 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:07:45.098 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:45.099 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:45.099 DEBUG:boto:path=/ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:45.099 DEBUG:boto:auth_path=/ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:45.099 DEBUG:boto:Method: GET 2023-09-22T21:07:45.099 DEBUG:boto:Path: /ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:45.099 DEBUG:boto:Data: 2023-09-22T21:07:45.099 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.099 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.099 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.099 DEBUG:boto:Params: {} 2023-09-22T21:07:45.100 DEBUG:boto:Token: None 2023-09-22T21:07:45.100 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/?versions 2023-09-22T21:07:45.100 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y2b68AnpFY4DkNHOxuUe4EnpoiY= 2023-09-22T21:07:45.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y2b68AnpFY4DkNHOxuUe4EnpoiY=', 'Content-Length': '0'} 2023-09-22T21:07:45.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fd3b2fa47c9b745-00650e0221-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.104 DEBUG:boto:b'ffzdqu-11999falsedummyDtjOyJANiXCj7.Vh58J.RwdyLtGvcNxtrue2023-09-22T21:06:06.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a18uA2UlGwMDN2SZPYYFpbhdTZDMaHDlctrue2023-09-22T21:06:41.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a22Ghp8GTpfzBw6qT8Hie8IpGLuimDYjrtrue2023-09-22T21:06:41.742Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:45.105 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-22T21:07:45.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-09-22T21:07:45.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-09-22T21:07:45.105 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:45.105 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:45.105 DEBUG:boto:path=/ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:45.105 DEBUG:boto:auth_path=/ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:45.106 DEBUG:boto:Method: GET 2023-09-22T21:07:45.106 DEBUG:boto:Path: /ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:45.106 DEBUG:boto:Data: 2023-09-22T21:07:45.106 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.106 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.106 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.106 DEBUG:boto:Params: {} 2023-09-22T21:07:45.106 DEBUG:boto:Token: None 2023-09-22T21:07:45.106 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/?versions 2023-09-22T21:07:45.107 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y2b68AnpFY4DkNHOxuUe4EnpoiY= 2023-09-22T21:07:45.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y2b68AnpFY4DkNHOxuUe4EnpoiY=', 'Content-Length': '0'} 2023-09-22T21:07:45.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003f57aa6c2b8ed56-00650e0221-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.111 DEBUG:boto:b'ffzdqu-11999falsedummyDtjOyJANiXCj7.Vh58J.RwdyLtGvcNxtrue2023-09-22T21:06:06.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a18uA2UlGwMDN2SZPYYFpbhdTZDMaHDlctrue2023-09-22T21:06:41.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a22Ghp8GTpfzBw6qT8Hie8IpGLuimDYjrtrue2023-09-22T21:06:41.742Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:45.112 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:45.112 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:45.112 DEBUG:boto:path=/ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:45.112 DEBUG:boto:auth_path=/ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:45.112 DEBUG:boto:Method: GET 2023-09-22T21:07:45.112 DEBUG:boto:Path: /ffzdqu-11/?versions&max-keys=999 2023-09-22T21:07:45.113 DEBUG:boto:Data: 2023-09-22T21:07:45.113 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.113 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.113 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.113 DEBUG:boto:Params: {} 2023-09-22T21:07:45.113 DEBUG:boto:Token: None 2023-09-22T21:07:45.113 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/?versions 2023-09-22T21:07:45.113 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y2b68AnpFY4DkNHOxuUe4EnpoiY= 2023-09-22T21:07:45.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y2b68AnpFY4DkNHOxuUe4EnpoiY=', 'Content-Length': '0'} 2023-09-22T21:07:45.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001349099573049b25-00650e0221-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.119 DEBUG:boto:b'ffzdqu-11999falsedummyDtjOyJANiXCj7.Vh58J.RwdyLtGvcNxtrue2023-09-22T21:06:06.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a18uA2UlGwMDN2SZPYYFpbhdTZDMaHDlctrue2023-09-22T21:06:41.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a22Ghp8GTpfzBw6qT8Hie8IpGLuimDYjrtrue2023-09-22T21:06:41.742Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:45.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-22T21:07:45.119 DEBUG:boto:path=/ffzdqu-11/dummy 2023-09-22T21:07:45.120 DEBUG:boto:auth_path=/ffzdqu-11/dummy 2023-09-22T21:07:45.120 DEBUG:boto:path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.120 DEBUG:boto:auth_path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.120 DEBUG:boto:Method: GET 2023-09-22T21:07:45.120 DEBUG:boto:Path: /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.120 DEBUG:boto:Data: 2023-09-22T21:07:45.120 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.120 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.121 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.121 DEBUG:boto:Params: {} 2023-09-22T21:07:45.121 DEBUG:boto:Token: None 2023-09-22T21:07:45.121 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.121 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YufYHiDoDKiXkuTV0D8yik7mI2A= 2023-09-22T21:07:45.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YufYHiDoDKiXkuTV0D8yik7mI2A=', 'Content-Length': '0'} 2023-09-22T21:07:45.124 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('x-amz-version-id', 'DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e164bfdd35514636-00650e0221-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.125 DEBUG:boto:path=/ffzdqu-11/dummy 2023-09-22T21:07:45.125 DEBUG:boto:auth_path=/ffzdqu-11/dummy 2023-09-22T21:07:45.125 DEBUG:boto:path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.125 DEBUG:boto:auth_path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.125 DEBUG:boto:Method: GET 2023-09-22T21:07:45.125 DEBUG:boto:Path: /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.126 DEBUG:boto:Data: 2023-09-22T21:07:45.126 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.126 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.126 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.126 DEBUG:boto:Params: {} 2023-09-22T21:07:45.126 DEBUG:boto:Token: None 2023-09-22T21:07:45.127 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.127 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YufYHiDoDKiXkuTV0D8yik7mI2A= 2023-09-22T21:07:45.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YufYHiDoDKiXkuTV0D8yik7mI2A=', 'Content-Length': '0'} 2023-09-22T21:07:45.129 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('x-amz-version-id', 'DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001ec127ffbb4df132-00650e0221-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.130 DEBUG:boto:path=/ffzdqu-11/dummy 2023-09-22T21:07:45.130 DEBUG:boto:auth_path=/ffzdqu-11/dummy 2023-09-22T21:07:45.130 DEBUG:boto:path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.130 DEBUG:boto:auth_path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.130 DEBUG:boto:Method: HEAD 2023-09-22T21:07:45.130 DEBUG:boto:Path: /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.131 DEBUG:boto:Data: 2023-09-22T21:07:45.131 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.131 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.131 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.131 DEBUG:boto:Params: {} 2023-09-22T21:07:45.131 DEBUG:boto:Token: None 2023-09-22T21:07:45.131 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.131 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iHBdIZhgErA7D2/8JZAYc/4advM= 2023-09-22T21:07:45.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iHBdIZhgErA7D2/8JZAYc/4advM=', 'Content-Length': '0'} 2023-09-22T21:07:45.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('x-amz-version-id', 'DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000249eeb106839cb7b-00650e0221-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.135 DEBUG:boto:path=/ffzdqu-11/dummy 2023-09-22T21:07:45.135 DEBUG:boto:auth_path=/ffzdqu-11/dummy 2023-09-22T21:07:45.135 DEBUG:boto:path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.135 DEBUG:boto:auth_path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.135 DEBUG:boto:Method: HEAD 2023-09-22T21:07:45.135 DEBUG:boto:Path: /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.135 DEBUG:boto:Data: 2023-09-22T21:07:45.135 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.135 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.136 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.136 DEBUG:boto:Params: {} 2023-09-22T21:07:45.136 DEBUG:boto:Token: None 2023-09-22T21:07:45.136 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.136 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iHBdIZhgErA7D2/8JZAYc/4advM= 2023-09-22T21:07:45.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iHBdIZhgErA7D2/8JZAYc/4advM=', 'Content-Length': '0'} 2023-09-22T21:07:45.139 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('x-amz-version-id', 'DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c6cf15416fc005d9-00650e0221-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-22T21:07:45.139 DEBUG:boto:path=/ffzdqu-11/dummy 2023-09-22T21:07:45.139 DEBUG:boto:auth_path=/ffzdqu-11/dummy 2023-09-22T21:07:45.139 DEBUG:boto:path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.139 DEBUG:boto:auth_path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.139 DEBUG:boto:Method: GET 2023-09-22T21:07:45.140 DEBUG:boto:Path: /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.140 DEBUG:boto:Data: 2023-09-22T21:07:45.140 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.140 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.140 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.140 DEBUG:boto:Params: {} 2023-09-22T21:07:45.140 DEBUG:boto:Token: None 2023-09-22T21:07:45.140 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.140 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YufYHiDoDKiXkuTV0D8yik7mI2A= 2023-09-22T21:07:45.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YufYHiDoDKiXkuTV0D8yik7mI2A=', 'Content-Length': '0'} 2023-09-22T21:07:45.143 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('x-amz-version-id', 'DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001c97e086fbcc5b00-00650e0221-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.143 DEBUG:boto:path=/ffzdqu-11/dummy 2023-09-22T21:07:45.143 DEBUG:boto:auth_path=/ffzdqu-11/dummy 2023-09-22T21:07:45.143 DEBUG:boto:path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.144 DEBUG:boto:auth_path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.144 DEBUG:boto:Method: GET 2023-09-22T21:07:45.144 DEBUG:boto:Path: /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.144 DEBUG:boto:Data: 2023-09-22T21:07:45.144 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.144 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.144 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.144 DEBUG:boto:Params: {} 2023-09-22T21:07:45.144 DEBUG:boto:Token: None 2023-09-22T21:07:45.145 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.145 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YufYHiDoDKiXkuTV0D8yik7mI2A= 2023-09-22T21:07:45.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YufYHiDoDKiXkuTV0D8yik7mI2A=', 'Content-Length': '0'} 2023-09-22T21:07:45.147 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('x-amz-version-id', 'DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000011cd3de92c027442-00650e0221-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.148 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:45.148 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:45.148 DEBUG:boto:path=/ffzdqu-11/?prefix=dummy 2023-09-22T21:07:45.148 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=dummy 2023-09-22T21:07:45.148 DEBUG:boto:Method: GET 2023-09-22T21:07:45.148 DEBUG:boto:Path: /ffzdqu-11/?prefix=dummy 2023-09-22T21:07:45.148 DEBUG:boto:Data: 2023-09-22T21:07:45.149 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.149 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.149 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.149 DEBUG:boto:Params: {} 2023-09-22T21:07:45.149 DEBUG:boto:Token: None 2023-09-22T21:07:45.149 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/ 2023-09-22T21:07:45.149 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g= 2023-09-22T21:07:45.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g=', 'Content-Length': '0'} 2023-09-22T21:07:45.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019c60060456dd97b-00650e0221-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.198 DEBUG:boto:b'ffzdqu-11dummy1000falsedummy2023-09-22T21:06:06.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:45.199 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:45.199 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:45.199 DEBUG:boto:path=/ffzdqu-11/?prefix=dummy 2023-09-22T21:07:45.199 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=dummy 2023-09-22T21:07:45.200 DEBUG:boto:Method: GET 2023-09-22T21:07:45.200 DEBUG:boto:Path: /ffzdqu-11/?prefix=dummy 2023-09-22T21:07:45.200 DEBUG:boto:Data: 2023-09-22T21:07:45.200 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.200 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.200 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.200 DEBUG:boto:Params: {} 2023-09-22T21:07:45.200 DEBUG:boto:Token: None 2023-09-22T21:07:45.201 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/ 2023-09-22T21:07:45.201 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g= 2023-09-22T21:07:45.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g=', 'Content-Length': '0'} 2023-09-22T21:07:45.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ca6ac75726f4faf-00650e0221-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.205 DEBUG:boto:b'ffzdqu-11dummy1000falsedummy2023-09-22T21:06:06.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:45.206 DEBUG:boto:path=/ffzdqu-11/dummy 2023-09-22T21:07:45.206 DEBUG:boto:auth_path=/ffzdqu-11/dummy 2023-09-22T21:07:45.206 DEBUG:boto:Method: HEAD 2023-09-22T21:07:45.206 DEBUG:boto:Path: /ffzdqu-11/dummy 2023-09-22T21:07:45.206 DEBUG:boto:Data: 2023-09-22T21:07:45.206 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.206 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.206 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.206 DEBUG:boto:Params: {} 2023-09-22T21:07:45.207 DEBUG:boto:Token: None 2023-09-22T21:07:45.207 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/dummy 2023-09-22T21:07:45.207 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lDXIn/LZXHSX5ILN6VUH0dapPBk= 2023-09-22T21:07:45.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lDXIn/LZXHSX5ILN6VUH0dapPBk=', 'Content-Length': '0'} 2023-09-22T21:07:45.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('x-amz-version-id', 'DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000041f0d14dcab87322-00650e0221-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.211 DEBUG:boto:path=/ffzdqu-11/dummy 2023-09-22T21:07:45.211 DEBUG:boto:auth_path=/ffzdqu-11/dummy 2023-09-22T21:07:45.211 DEBUG:boto:Method: HEAD 2023-09-22T21:07:45.211 DEBUG:boto:Path: /ffzdqu-11/dummy 2023-09-22T21:07:45.211 DEBUG:boto:Data: 2023-09-22T21:07:45.211 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.211 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.211 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.211 DEBUG:boto:Params: {} 2023-09-22T21:07:45.212 DEBUG:boto:Token: None 2023-09-22T21:07:45.212 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/dummy 2023-09-22T21:07:45.212 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lDXIn/LZXHSX5ILN6VUH0dapPBk= 2023-09-22T21:07:45.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lDXIn/LZXHSX5ILN6VUH0dapPBk=', 'Content-Length': '0'} 2023-09-22T21:07:45.215 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('x-amz-version-id', 'DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000059786c35fc304bfa-00650e0221-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-22T21:07:45.216 DEBUG:boto:path=/ffzdqu-11/dummy 2023-09-22T21:07:45.216 DEBUG:boto:auth_path=/ffzdqu-11/dummy 2023-09-22T21:07:45.216 DEBUG:boto:path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.216 DEBUG:boto:auth_path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.216 DEBUG:boto:Method: GET 2023-09-22T21:07:45.216 DEBUG:boto:Path: /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.216 DEBUG:boto:Data: 2023-09-22T21:07:45.216 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.216 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.217 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.217 DEBUG:boto:Params: {} 2023-09-22T21:07:45.217 DEBUG:boto:Token: None 2023-09-22T21:07:45.217 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.217 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YufYHiDoDKiXkuTV0D8yik7mI2A= 2023-09-22T21:07:45.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YufYHiDoDKiXkuTV0D8yik7mI2A=', 'Content-Length': '0'} 2023-09-22T21:07:45.220 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('x-amz-version-id', 'DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004dd3f12473aab87a-00650e0221-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.220 DEBUG:boto:path=/ffzdqu-11/dummy 2023-09-22T21:07:45.221 DEBUG:boto:auth_path=/ffzdqu-11/dummy 2023-09-22T21:07:45.221 DEBUG:boto:path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.221 DEBUG:boto:auth_path=/ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.221 DEBUG:boto:Method: GET 2023-09-22T21:07:45.221 DEBUG:boto:Path: /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.221 DEBUG:boto:Data: 2023-09-22T21:07:45.221 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.221 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.222 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.222 DEBUG:boto:Params: {} 2023-09-22T21:07:45.222 DEBUG:boto:Token: None 2023-09-22T21:07:45.222 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/dummy?versionId=DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx 2023-09-22T21:07:45.222 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YufYHiDoDKiXkuTV0D8yik7mI2A= 2023-09-22T21:07:45.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YufYHiDoDKiXkuTV0D8yik7mI2A=', 'Content-Length': '0'} 2023-09-22T21:07:45.225 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:06 GMT'), ('x-amz-version-id', 'DtjOyJANiXCj7.Vh58J.RwdyLtGvcNx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007a97ac4e8d05c02e-00650e0221-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.225 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:45.225 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:45.225 DEBUG:boto:path=/ffzdqu-11/?prefix=dummy 2023-09-22T21:07:45.226 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=dummy 2023-09-22T21:07:45.226 DEBUG:boto:Method: GET 2023-09-22T21:07:45.226 DEBUG:boto:Path: /ffzdqu-11/?prefix=dummy 2023-09-22T21:07:45.226 DEBUG:boto:Data: 2023-09-22T21:07:45.226 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.226 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.226 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.227 DEBUG:boto:Params: {} 2023-09-22T21:07:45.227 DEBUG:boto:Token: None 2023-09-22T21:07:45.227 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/ 2023-09-22T21:07:45.227 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g= 2023-09-22T21:07:45.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g=', 'Content-Length': '0'} 2023-09-22T21:07:45.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000067efe575cdc5275-00650e0221-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.274 DEBUG:boto:b'ffzdqu-11dummy1000falsedummy2023-09-22T21:06:06.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:45.275 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:45.275 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:45.275 DEBUG:boto:path=/ffzdqu-11/?prefix=dummy 2023-09-22T21:07:45.275 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=dummy 2023-09-22T21:07:45.275 DEBUG:boto:Method: GET 2023-09-22T21:07:45.275 DEBUG:boto:Path: /ffzdqu-11/?prefix=dummy 2023-09-22T21:07:45.276 DEBUG:boto:Data: 2023-09-22T21:07:45.276 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.276 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.276 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.276 DEBUG:boto:Params: {} 2023-09-22T21:07:45.276 DEBUG:boto:Token: None 2023-09-22T21:07:45.276 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/ 2023-09-22T21:07:45.276 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g= 2023-09-22T21:07:45.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g=', 'Content-Length': '0'} 2023-09-22T21:07:45.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c6bc5181d0343db-00650e0221-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.282 DEBUG:boto:b'ffzdqu-11dummy1000falsedummy2023-09-22T21:06:06.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:45.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-22T21:07:45.282 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:07:45.282 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:07:45.282 DEBUG:boto:path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.282 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.282 DEBUG:boto:Method: GET 2023-09-22T21:07:45.283 DEBUG:boto:Path: /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.283 DEBUG:boto:Data: 2023-09-22T21:07:45.283 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.283 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.283 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.283 DEBUG:boto:Params: {} 2023-09-22T21:07:45.283 DEBUG:boto:Token: None 2023-09-22T21:07:45.283 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.283 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pjcJDPpSQlDcAwyoqTSlgOn27XE= 2023-09-22T21:07:45.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pjcJDPpSQlDcAwyoqTSlgOn27XE=', 'Content-Length': '0'} 2023-09-22T21:07:45.286 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc155f00cdcecdff-00650e0221-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.330 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:07:45.330 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:07:45.331 DEBUG:boto:path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.331 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.331 DEBUG:boto:Method: GET 2023-09-22T21:07:45.331 DEBUG:boto:Path: /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.331 DEBUG:boto:Data: 2023-09-22T21:07:45.331 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.331 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.331 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.331 DEBUG:boto:Params: {} 2023-09-22T21:07:45.332 DEBUG:boto:Token: None 2023-09-22T21:07:45.332 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.332 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pjcJDPpSQlDcAwyoqTSlgOn27XE= 2023-09-22T21:07:45.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pjcJDPpSQlDcAwyoqTSlgOn27XE=', 'Content-Length': '0'} 2023-09-22T21:07:45.335 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de09bac02963f163-00650e0221-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.335 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:07:45.335 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:07:45.336 DEBUG:boto:path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.336 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.336 DEBUG:boto:Method: HEAD 2023-09-22T21:07:45.336 DEBUG:boto:Path: /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.336 DEBUG:boto:Data: 2023-09-22T21:07:45.336 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.336 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.337 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.337 DEBUG:boto:Params: {} 2023-09-22T21:07:45.337 DEBUG:boto:Token: None 2023-09-22T21:07:45.337 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.337 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6miSe1ylQLb+vATwrZytW90TDkI= 2023-09-22T21:07:45.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6miSe1ylQLb+vATwrZytW90TDkI=', 'Content-Length': '0'} 2023-09-22T21:07:45.341 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000962ece61dd2633eb-00650e0221-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.341 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:07:45.342 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:07:45.342 DEBUG:boto:path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.342 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.342 DEBUG:boto:Method: HEAD 2023-09-22T21:07:45.342 DEBUG:boto:Path: /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.342 DEBUG:boto:Data: 2023-09-22T21:07:45.342 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.342 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.342 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.342 DEBUG:boto:Params: {} 2023-09-22T21:07:45.343 DEBUG:boto:Token: None 2023-09-22T21:07:45.343 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.343 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6miSe1ylQLb+vATwrZytW90TDkI= 2023-09-22T21:07:45.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6miSe1ylQLb+vATwrZytW90TDkI=', 'Content-Length': '0'} 2023-09-22T21:07:45.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8df4e5e15a97b94-00650e0221-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-22T21:07:45.346 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:07:45.346 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:07:45.346 DEBUG:boto:path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.346 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.346 DEBUG:boto:Method: GET 2023-09-22T21:07:45.347 DEBUG:boto:Path: /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.347 DEBUG:boto:Data: 2023-09-22T21:07:45.347 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.347 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.347 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.347 DEBUG:boto:Params: {} 2023-09-22T21:07:45.347 DEBUG:boto:Token: None 2023-09-22T21:07:45.347 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.347 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pjcJDPpSQlDcAwyoqTSlgOn27XE= 2023-09-22T21:07:45.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pjcJDPpSQlDcAwyoqTSlgOn27XE=', 'Content-Length': '0'} 2023-09-22T21:07:45.350 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9650648c96efe0d-00650e0221-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.394 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:07:45.395 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:07:45.395 DEBUG:boto:path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.395 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.395 DEBUG:boto:Method: GET 2023-09-22T21:07:45.395 DEBUG:boto:Path: /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.395 DEBUG:boto:Data: 2023-09-22T21:07:45.395 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.396 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.396 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.396 DEBUG:boto:Params: {} 2023-09-22T21:07:45.396 DEBUG:boto:Token: None 2023-09-22T21:07:45.396 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.396 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pjcJDPpSQlDcAwyoqTSlgOn27XE= 2023-09-22T21:07:45.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pjcJDPpSQlDcAwyoqTSlgOn27XE=', 'Content-Length': '0'} 2023-09-22T21:07:45.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa6305eeecabc00c-00650e0221-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.400 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:45.400 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:45.400 DEBUG:boto:path=/ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:45.400 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:45.400 DEBUG:boto:Method: GET 2023-09-22T21:07:45.400 DEBUG:boto:Path: /ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:45.401 DEBUG:boto:Data: 2023-09-22T21:07:45.401 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.401 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.401 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.401 DEBUG:boto:Params: {} 2023-09-22T21:07:45.401 DEBUG:boto:Token: None 2023-09-22T21:07:45.401 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/ 2023-09-22T21:07:45.402 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g= 2023-09-22T21:07:45.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g=', 'Content-Length': '0'} 2023-09-22T21:07:45.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055fabf0caa403f8c-00650e0221-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.454 DEBUG:boto:b'ffzdqu-11obj-a11000falseobj-a12023-09-22T21:06:41.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:45.455 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:45.455 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:45.455 DEBUG:boto:path=/ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:45.455 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:45.455 DEBUG:boto:Method: GET 2023-09-22T21:07:45.456 DEBUG:boto:Path: /ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:45.456 DEBUG:boto:Data: 2023-09-22T21:07:45.456 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.456 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.456 DEBUG:boto:Params: {} 2023-09-22T21:07:45.456 DEBUG:boto:Token: None 2023-09-22T21:07:45.456 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/ 2023-09-22T21:07:45.457 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g= 2023-09-22T21:07:45.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g=', 'Content-Length': '0'} 2023-09-22T21:07:45.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002aa58aef09f32520-00650e0221-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.462 DEBUG:boto:b'ffzdqu-11obj-a11000falseobj-a12023-09-22T21:06:41.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:45.462 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:07:45.462 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:07:45.462 DEBUG:boto:Method: HEAD 2023-09-22T21:07:45.462 DEBUG:boto:Path: /ffzdqu-11/obj-a1 2023-09-22T21:07:45.463 DEBUG:boto:Data: 2023-09-22T21:07:45.463 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.463 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.463 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.463 DEBUG:boto:Params: {} 2023-09-22T21:07:45.463 DEBUG:boto:Token: None 2023-09-22T21:07:45.463 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/obj-a1 2023-09-22T21:07:45.463 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LNV7Q2kGeAiiewPsoOEJNakVrL8= 2023-09-22T21:07:45.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LNV7Q2kGeAiiewPsoOEJNakVrL8=', 'Content-Length': '0'} 2023-09-22T21:07:45.467 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7432387349a195f-00650e0221-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.467 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:07:45.467 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:07:45.467 DEBUG:boto:Method: HEAD 2023-09-22T21:07:45.468 DEBUG:boto:Path: /ffzdqu-11/obj-a1 2023-09-22T21:07:45.468 DEBUG:boto:Data: 2023-09-22T21:07:45.468 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.468 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.468 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.468 DEBUG:boto:Params: {} 2023-09-22T21:07:45.468 DEBUG:boto:Token: None 2023-09-22T21:07:45.468 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/obj-a1 2023-09-22T21:07:45.469 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LNV7Q2kGeAiiewPsoOEJNakVrL8= 2023-09-22T21:07:45.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LNV7Q2kGeAiiewPsoOEJNakVrL8=', 'Content-Length': '0'} 2023-09-22T21:07:45.472 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef7728eb3265dcb1-00650e0221-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-22T21:07:45.472 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:07:45.473 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:07:45.473 DEBUG:boto:path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.473 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.473 DEBUG:boto:Method: GET 2023-09-22T21:07:45.473 DEBUG:boto:Path: /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.473 DEBUG:boto:Data: 2023-09-22T21:07:45.473 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.474 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.474 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.474 DEBUG:boto:Params: {} 2023-09-22T21:07:45.474 DEBUG:boto:Token: None 2023-09-22T21:07:45.474 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.474 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pjcJDPpSQlDcAwyoqTSlgOn27XE= 2023-09-22T21:07:45.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pjcJDPpSQlDcAwyoqTSlgOn27XE=', 'Content-Length': '0'} 2023-09-22T21:07:45.477 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000805e49e75b14e0fc-00650e0221-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.518 DEBUG:boto:path=/ffzdqu-11/obj-a1 2023-09-22T21:07:45.519 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1 2023-09-22T21:07:45.519 DEBUG:boto:path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.519 DEBUG:boto:auth_path=/ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.519 DEBUG:boto:Method: GET 2023-09-22T21:07:45.519 DEBUG:boto:Path: /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.519 DEBUG:boto:Data: 2023-09-22T21:07:45.519 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.519 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.519 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.520 DEBUG:boto:Params: {} 2023-09-22T21:07:45.520 DEBUG:boto:Token: None 2023-09-22T21:07:45.520 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/obj-a1?versionId=8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc 2023-09-22T21:07:45.520 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pjcJDPpSQlDcAwyoqTSlgOn27XE= 2023-09-22T21:07:45.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pjcJDPpSQlDcAwyoqTSlgOn27XE=', 'Content-Length': '0'} 2023-09-22T21:07:45.523 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '8uA2UlGwMDN2SZPYYFpbhdTZDMaHDlc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e583aca2b976148-00650e0221-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.523 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:45.523 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:45.524 DEBUG:boto:path=/ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:45.524 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:45.524 DEBUG:boto:Method: GET 2023-09-22T21:07:45.524 DEBUG:boto:Path: /ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:45.524 DEBUG:boto:Data: 2023-09-22T21:07:45.524 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.524 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.524 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.524 DEBUG:boto:Params: {} 2023-09-22T21:07:45.524 DEBUG:boto:Token: None 2023-09-22T21:07:45.525 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/ 2023-09-22T21:07:45.525 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g= 2023-09-22T21:07:45.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g=', 'Content-Length': '0'} 2023-09-22T21:07:45.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009757688579c2edde-00650e0221-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.570 DEBUG:boto:b'ffzdqu-11obj-a11000falseobj-a12023-09-22T21:06:41.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:45.571 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:45.571 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:45.571 DEBUG:boto:path=/ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:45.571 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:45.571 DEBUG:boto:Method: GET 2023-09-22T21:07:45.572 DEBUG:boto:Path: /ffzdqu-11/?prefix=obj-a1 2023-09-22T21:07:45.572 DEBUG:boto:Data: 2023-09-22T21:07:45.572 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.572 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.572 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.572 DEBUG:boto:Params: {} 2023-09-22T21:07:45.572 DEBUG:boto:Token: None 2023-09-22T21:07:45.573 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/ 2023-09-22T21:07:45.573 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g= 2023-09-22T21:07:45.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g=', 'Content-Length': '0'} 2023-09-22T21:07:45.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008da5d6594ea21f35-00650e0221-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.578 DEBUG:boto:b'ffzdqu-11obj-a11000falseobj-a12023-09-22T21:06:41.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:45.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-22T21:07:45.579 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:07:45.579 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:07:45.579 DEBUG:boto:path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.579 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.579 DEBUG:boto:Method: GET 2023-09-22T21:07:45.579 DEBUG:boto:Path: /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.579 DEBUG:boto:Data: 2023-09-22T21:07:45.579 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.579 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.579 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.580 DEBUG:boto:Params: {} 2023-09-22T21:07:45.580 DEBUG:boto:Token: None 2023-09-22T21:07:45.580 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.580 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/XcglN9yRzYWSfbChLCeAVprqrU= 2023-09-22T21:07:45.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/XcglN9yRzYWSfbChLCeAVprqrU=', 'Content-Length': '0'} 2023-09-22T21:07:45.583 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000948267012869ad29-00650e0221-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.583 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:07:45.583 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:07:45.583 DEBUG:boto:path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.584 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.584 DEBUG:boto:Method: GET 2023-09-22T21:07:45.584 DEBUG:boto:Path: /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.584 DEBUG:boto:Data: 2023-09-22T21:07:45.584 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.584 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.584 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.584 DEBUG:boto:Params: {} 2023-09-22T21:07:45.585 DEBUG:boto:Token: None 2023-09-22T21:07:45.585 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.585 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/XcglN9yRzYWSfbChLCeAVprqrU= 2023-09-22T21:07:45.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/XcglN9yRzYWSfbChLCeAVprqrU=', 'Content-Length': '0'} 2023-09-22T21:07:45.588 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009295bdb2ff5dd0a4-00650e0221-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.588 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:07:45.588 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:07:45.588 DEBUG:boto:path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.588 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.588 DEBUG:boto:Method: HEAD 2023-09-22T21:07:45.589 DEBUG:boto:Path: /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.589 DEBUG:boto:Data: 2023-09-22T21:07:45.589 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.589 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.589 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.589 DEBUG:boto:Params: {} 2023-09-22T21:07:45.589 DEBUG:boto:Token: None 2023-09-22T21:07:45.589 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.590 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+LN55xGOm7koKWOFNopXW9sUiig= 2023-09-22T21:07:45.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+LN55xGOm7koKWOFNopXW9sUiig=', 'Content-Length': '0'} 2023-09-22T21:07:45.592 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004240afe1c0b4863c-00650e0221-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.592 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:07:45.593 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:07:45.593 DEBUG:boto:path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.593 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.593 DEBUG:boto:Method: HEAD 2023-09-22T21:07:45.593 DEBUG:boto:Path: /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.593 DEBUG:boto:Data: 2023-09-22T21:07:45.593 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.593 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.593 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.593 DEBUG:boto:Params: {} 2023-09-22T21:07:45.594 DEBUG:boto:Token: None 2023-09-22T21:07:45.594 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.594 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+LN55xGOm7koKWOFNopXW9sUiig= 2023-09-22T21:07:45.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+LN55xGOm7koKWOFNopXW9sUiig=', 'Content-Length': '0'} 2023-09-22T21:07:45.596 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ee869b55d47d6e9-00650e0221-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-22T21:07:45.597 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:07:45.597 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:07:45.597 DEBUG:boto:path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.597 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.597 DEBUG:boto:Method: GET 2023-09-22T21:07:45.597 DEBUG:boto:Path: /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.597 DEBUG:boto:Data: 2023-09-22T21:07:45.598 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.598 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.598 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.598 DEBUG:boto:Params: {} 2023-09-22T21:07:45.598 DEBUG:boto:Token: None 2023-09-22T21:07:45.598 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.598 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/XcglN9yRzYWSfbChLCeAVprqrU= 2023-09-22T21:07:45.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/XcglN9yRzYWSfbChLCeAVprqrU=', 'Content-Length': '0'} 2023-09-22T21:07:45.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032d934aa58bd303d-00650e0221-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.601 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:07:45.601 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:07:45.601 DEBUG:boto:path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.602 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.602 DEBUG:boto:Method: GET 2023-09-22T21:07:45.602 DEBUG:boto:Path: /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.602 DEBUG:boto:Data: 2023-09-22T21:07:45.602 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.602 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.602 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.602 DEBUG:boto:Params: {} 2023-09-22T21:07:45.603 DEBUG:boto:Token: None 2023-09-22T21:07:45.603 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.603 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/XcglN9yRzYWSfbChLCeAVprqrU= 2023-09-22T21:07:45.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/XcglN9yRzYWSfbChLCeAVprqrU=', 'Content-Length': '0'} 2023-09-22T21:07:45.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000092cd0adf5bd7e53a-00650e0221-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.646 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:45.647 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:45.647 DEBUG:boto:path=/ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:45.647 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:45.647 DEBUG:boto:Method: GET 2023-09-22T21:07:45.647 DEBUG:boto:Path: /ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:45.647 DEBUG:boto:Data: 2023-09-22T21:07:45.647 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.647 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.647 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.647 DEBUG:boto:Params: {} 2023-09-22T21:07:45.648 DEBUG:boto:Token: None 2023-09-22T21:07:45.648 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/ 2023-09-22T21:07:45.648 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g= 2023-09-22T21:07:45.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g=', 'Content-Length': '0'} 2023-09-22T21:07:45.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a1585030d4a5edd-00650e0221-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.653 DEBUG:boto:b'ffzdqu-11obj-a21000falseobj-a22023-09-22T21:06:41.742Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:45.653 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:45.654 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:45.654 DEBUG:boto:path=/ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:45.654 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:45.654 DEBUG:boto:Method: GET 2023-09-22T21:07:45.654 DEBUG:boto:Path: /ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:45.654 DEBUG:boto:Data: 2023-09-22T21:07:45.654 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.654 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.654 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.654 DEBUG:boto:Params: {} 2023-09-22T21:07:45.655 DEBUG:boto:Token: None 2023-09-22T21:07:45.655 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/ 2023-09-22T21:07:45.655 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g= 2023-09-22T21:07:45.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g=', 'Content-Length': '0'} 2023-09-22T21:07:45.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007063242d382d3787-00650e0221-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.660 DEBUG:boto:b'ffzdqu-11obj-a21000falseobj-a22023-09-22T21:06:41.742Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:45.660 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:07:45.660 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:07:45.660 DEBUG:boto:Method: HEAD 2023-09-22T21:07:45.661 DEBUG:boto:Path: /ffzdqu-11/obj-a2 2023-09-22T21:07:45.661 DEBUG:boto:Data: 2023-09-22T21:07:45.661 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.661 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.661 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.661 DEBUG:boto:Params: {} 2023-09-22T21:07:45.661 DEBUG:boto:Token: None 2023-09-22T21:07:45.661 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/obj-a2 2023-09-22T21:07:45.661 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SpEmr2ze+p8VYSPsl8ml5iDsw8I= 2023-09-22T21:07:45.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SpEmr2ze+p8VYSPsl8ml5iDsw8I=', 'Content-Length': '0'} 2023-09-22T21:07:45.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000483af2968b2e73b0-00650e0221-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.665 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:07:45.665 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:07:45.665 DEBUG:boto:Method: HEAD 2023-09-22T21:07:45.666 DEBUG:boto:Path: /ffzdqu-11/obj-a2 2023-09-22T21:07:45.666 DEBUG:boto:Data: 2023-09-22T21:07:45.666 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.666 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.666 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.666 DEBUG:boto:Params: {} 2023-09-22T21:07:45.666 DEBUG:boto:Token: None 2023-09-22T21:07:45.666 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/obj-a2 2023-09-22T21:07:45.666 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SpEmr2ze+p8VYSPsl8ml5iDsw8I= 2023-09-22T21:07:45.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SpEmr2ze+p8VYSPsl8ml5iDsw8I=', 'Content-Length': '0'} 2023-09-22T21:07:45.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ed803e1546edb27-00650e0221-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-22T21:07:45.670 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:07:45.670 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:07:45.671 DEBUG:boto:path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.671 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.671 DEBUG:boto:Method: GET 2023-09-22T21:07:45.671 DEBUG:boto:Path: /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.671 DEBUG:boto:Data: 2023-09-22T21:07:45.671 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.671 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.671 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.671 DEBUG:boto:Params: {} 2023-09-22T21:07:45.671 DEBUG:boto:Token: None 2023-09-22T21:07:45.672 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.672 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/XcglN9yRzYWSfbChLCeAVprqrU= 2023-09-22T21:07:45.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/XcglN9yRzYWSfbChLCeAVprqrU=', 'Content-Length': '0'} 2023-09-22T21:07:45.675 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000fbb441d5ffd1ad0-00650e0221-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.718 DEBUG:boto:path=/ffzdqu-11/obj-a2 2023-09-22T21:07:45.719 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2 2023-09-22T21:07:45.719 DEBUG:boto:path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.719 DEBUG:boto:auth_path=/ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.719 DEBUG:boto:Method: GET 2023-09-22T21:07:45.719 DEBUG:boto:Path: /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.719 DEBUG:boto:Data: 2023-09-22T21:07:45.719 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.719 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.719 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.719 DEBUG:boto:Params: {} 2023-09-22T21:07:45.720 DEBUG:boto:Token: None 2023-09-22T21:07:45.720 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/obj-a2?versionId=2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr 2023-09-22T21:07:45.720 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/XcglN9yRzYWSfbChLCeAVprqrU= 2023-09-22T21:07:45.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/XcglN9yRzYWSfbChLCeAVprqrU=', 'Content-Length': '0'} 2023-09-22T21:07:45.723 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', '2Ghp8GTpfzBw6qT8Hie8IpGLuimDYjr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a26f193fce1f3d9-00650e0221-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.766 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:45.767 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:45.767 DEBUG:boto:path=/ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:45.767 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:45.767 DEBUG:boto:Method: GET 2023-09-22T21:07:45.767 DEBUG:boto:Path: /ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:45.767 DEBUG:boto:Data: 2023-09-22T21:07:45.767 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.767 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:45.767 DEBUG:boto:Port: 8001 2023-09-22T21:07:45.767 DEBUG:boto:Params: {} 2023-09-22T21:07:45.768 DEBUG:boto:Token: None 2023-09-22T21:07:45.768 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/ 2023-09-22T21:07:45.768 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g= 2023-09-22T21:07:45.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g=', 'Content-Length': '0'} 2023-09-22T21:07:45.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1900210e4b2c3c5-00650e0221-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.774 DEBUG:boto:b'ffzdqu-11obj-a21000falseobj-a22023-09-22T21:06:41.742Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:45.774 DEBUG:boto:path=/ffzdqu-11/ 2023-09-22T21:07:45.774 DEBUG:boto:auth_path=/ffzdqu-11/ 2023-09-22T21:07:45.774 DEBUG:boto:path=/ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:45.775 DEBUG:boto:auth_path=/ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:45.775 DEBUG:boto:Method: GET 2023-09-22T21:07:45.775 DEBUG:boto:Path: /ffzdqu-11/?prefix=obj-a2 2023-09-22T21:07:45.775 DEBUG:boto:Data: 2023-09-22T21:07:45.775 DEBUG:boto:Headers: {} 2023-09-22T21:07:45.775 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:45.775 DEBUG:boto:Port: 8000 2023-09-22T21:07:45.775 DEBUG:boto:Params: {} 2023-09-22T21:07:45.775 DEBUG:boto:Token: None 2023-09-22T21:07:45.776 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:45 GMT /ffzdqu-11/ 2023-09-22T21:07:45.776 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g= 2023-09-22T21:07:45.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CjrVymr0yTS5jIFCL0pNbvVpu9g=', 'Content-Length': '0'} 2023-09-22T21:07:45.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004442cbf1e9af23c-00650e0221-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:45.826 DEBUG:boto:b'ffzdqu-11obj-a21000falseobj-a22023-09-22T21:06:41.742Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:45.827 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-11 zones={a2, a1} 2023-09-22T21:07:45.827 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-12 2023-09-22T21:07:45.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:07:45.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:07:45.827 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:07:45.878 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:07:45.879 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:07:45.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:45.923+0000 7fb6b29a2a40 1 bucket sync caught up with source: 2023-09-22T21:07:45.922 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.3.11, , , 00000000002.11.11, 00000000011.24.6, ] 2023-09-22T21:07:45.922 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000002.3.11, , , 00000000002.11.11, 00000000011.24.6, ] 2023-09-22T21:07:45.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:45.923+0000 7fb6b29a2a40 0 bucket checkpoint complete 2023-09-22T21:07:45.938 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-12 2023-09-22T21:07:45.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:07:45.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:07:45.938 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:07:45.979 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:07:45.979 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:07:46.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:46.019+0000 7f5d4be55a40 1 bucket sync caught up with source: 2023-09-22T21:07:46.018 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000003.18.10, , , 00000000011.87.6, 00000000002.18.11, ] 2023-09-22T21:07:46.019 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000003.18.10, , , 00000000011.87.6, 00000000002.18.11, ] 2023-09-22T21:07:46.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:46.019+0000 7f5d4be55a40 0 bucket checkpoint complete 2023-09-22T21:07:46.034 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-12 zones={a2, a1} 2023-09-22T21:07:46.034 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:46.034 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:46.034 DEBUG:boto:Method: HEAD 2023-09-22T21:07:46.034 DEBUG:boto:Path: /ffzdqu-12/ 2023-09-22T21:07:46.034 DEBUG:boto:Data: 2023-09-22T21:07:46.035 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.035 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.035 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.035 DEBUG:boto:Params: {} 2023-09-22T21:07:46.035 DEBUG:boto:Token: None 2023-09-22T21:07:46.035 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/ 2023-09-22T21:07:46.035 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:j9tGkeTCVrI6gmp7qE2Ny/uDnss= 2023-09-22T21:07:46.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:j9tGkeTCVrI6gmp7qE2Ny/uDnss=', 'Content-Length': '0'} 2023-09-22T21:07:46.037 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000980d04731d0c89bf-00650e0222-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.038 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:46.038 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:46.038 DEBUG:boto:Method: HEAD 2023-09-22T21:07:46.038 DEBUG:boto:Path: /ffzdqu-12/ 2023-09-22T21:07:46.038 DEBUG:boto:Data: 2023-09-22T21:07:46.038 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.038 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.038 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.039 DEBUG:boto:Params: {} 2023-09-22T21:07:46.039 DEBUG:boto:Token: None 2023-09-22T21:07:46.039 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/ 2023-09-22T21:07:46.039 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:j9tGkeTCVrI6gmp7qE2Ny/uDnss= 2023-09-22T21:07:46.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:j9tGkeTCVrI6gmp7qE2Ny/uDnss=', 'Content-Length': '0'} 2023-09-22T21:07:46.041 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004cf4625dccfbdeb4-00650e0222-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.041 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:07:46.041 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:46.041 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:46.042 DEBUG:boto:path=/ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:46.042 DEBUG:boto:auth_path=/ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:46.042 DEBUG:boto:Method: GET 2023-09-22T21:07:46.042 DEBUG:boto:Path: /ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:46.042 DEBUG:boto:Data: 2023-09-22T21:07:46.042 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.042 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.042 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.042 DEBUG:boto:Params: {} 2023-09-22T21:07:46.042 DEBUG:boto:Token: None 2023-09-22T21:07:46.043 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/?versions 2023-09-22T21:07:46.043 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jMEYGcnvIBfSkFotjgEU41A7GZc= 2023-09-22T21:07:46.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jMEYGcnvIBfSkFotjgEU41A7GZc=', 'Content-Length': '0'} 2023-09-22T21:07:46.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c49b3a17439be9cd-00650e0222-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.046 DEBUG:boto:b'ffzdqu-12999falsedummyboYngPv33ng8HsND7dSXZTE12I7WmDDtrue2023-09-22T21:06:21.418Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1ENtJQRcMHYYCQC9B1Taipgnc1aUE.HLtrue2023-09-22T21:06:41.903Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2u7s2N-8HbpXwMBMoersKLSCgHbn-fnrtrue2023-09-22T21:06:42.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:46.046 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-22T21:07:46.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-09-22T21:07:46.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-09-22T21:07:46.047 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:07:46.047 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:46.047 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:46.047 DEBUG:boto:path=/ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:46.047 DEBUG:boto:auth_path=/ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:46.048 DEBUG:boto:Method: GET 2023-09-22T21:07:46.048 DEBUG:boto:Path: /ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:46.048 DEBUG:boto:Data: 2023-09-22T21:07:46.048 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.048 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.048 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.048 DEBUG:boto:Params: {} 2023-09-22T21:07:46.049 DEBUG:boto:Token: None 2023-09-22T21:07:46.049 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/?versions 2023-09-22T21:07:46.049 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jMEYGcnvIBfSkFotjgEU41A7GZc= 2023-09-22T21:07:46.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jMEYGcnvIBfSkFotjgEU41A7GZc=', 'Content-Length': '0'} 2023-09-22T21:07:46.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4d3a402721d667b-00650e0222-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.052 DEBUG:boto:b'ffzdqu-12999falsedummyboYngPv33ng8HsND7dSXZTE12I7WmDDtrue2023-09-22T21:06:21.418Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1ENtJQRcMHYYCQC9B1Taipgnc1aUE.HLtrue2023-09-22T21:06:41.903Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2u7s2N-8HbpXwMBMoersKLSCgHbn-fnrtrue2023-09-22T21:06:42.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:46.053 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-22T21:07:46.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-09-22T21:07:46.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-09-22T21:07:46.053 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:46.053 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:46.053 DEBUG:boto:path=/ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:46.053 DEBUG:boto:auth_path=/ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:46.054 DEBUG:boto:Method: GET 2023-09-22T21:07:46.054 DEBUG:boto:Path: /ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:46.054 DEBUG:boto:Data: 2023-09-22T21:07:46.054 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.054 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.054 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.054 DEBUG:boto:Params: {} 2023-09-22T21:07:46.054 DEBUG:boto:Token: None 2023-09-22T21:07:46.054 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/?versions 2023-09-22T21:07:46.055 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jMEYGcnvIBfSkFotjgEU41A7GZc= 2023-09-22T21:07:46.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jMEYGcnvIBfSkFotjgEU41A7GZc=', 'Content-Length': '0'} 2023-09-22T21:07:46.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dff464f6cc5b3f5-00650e0222-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.057 DEBUG:boto:b'ffzdqu-12999falsedummyboYngPv33ng8HsND7dSXZTE12I7WmDDtrue2023-09-22T21:06:21.418Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1ENtJQRcMHYYCQC9B1Taipgnc1aUE.HLtrue2023-09-22T21:06:41.903Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2u7s2N-8HbpXwMBMoersKLSCgHbn-fnrtrue2023-09-22T21:06:42.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:46.057 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:46.058 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:46.058 DEBUG:boto:path=/ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:46.058 DEBUG:boto:auth_path=/ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:46.058 DEBUG:boto:Method: GET 2023-09-22T21:07:46.058 DEBUG:boto:Path: /ffzdqu-12/?versions&max-keys=999 2023-09-22T21:07:46.058 DEBUG:boto:Data: 2023-09-22T21:07:46.058 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.058 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.058 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.059 DEBUG:boto:Params: {} 2023-09-22T21:07:46.059 DEBUG:boto:Token: None 2023-09-22T21:07:46.059 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/?versions 2023-09-22T21:07:46.059 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jMEYGcnvIBfSkFotjgEU41A7GZc= 2023-09-22T21:07:46.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jMEYGcnvIBfSkFotjgEU41A7GZc=', 'Content-Length': '0'} 2023-09-22T21:07:46.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008336f6c80e2dfff2-00650e0222-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.061 DEBUG:boto:b'ffzdqu-12999falsedummyboYngPv33ng8HsND7dSXZTE12I7WmDDtrue2023-09-22T21:06:21.418Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1ENtJQRcMHYYCQC9B1Taipgnc1aUE.HLtrue2023-09-22T21:06:41.903Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2u7s2N-8HbpXwMBMoersKLSCgHbn-fnrtrue2023-09-22T21:06:42.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:46.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-22T21:07:46.062 DEBUG:boto:path=/ffzdqu-12/dummy 2023-09-22T21:07:46.062 DEBUG:boto:auth_path=/ffzdqu-12/dummy 2023-09-22T21:07:46.062 DEBUG:boto:path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.062 DEBUG:boto:auth_path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.063 DEBUG:boto:Method: GET 2023-09-22T21:07:46.063 DEBUG:boto:Path: /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.063 DEBUG:boto:Data: 2023-09-22T21:07:46.063 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.063 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.063 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.063 DEBUG:boto:Params: {} 2023-09-22T21:07:46.063 DEBUG:boto:Token: None 2023-09-22T21:07:46.063 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.064 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wQ6iNb6SnVKwW8jpZhhwpGkPr18= 2023-09-22T21:07:46.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wQ6iNb6SnVKwW8jpZhhwpGkPr18=', 'Content-Length': '0'} 2023-09-22T21:07:46.065 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('x-amz-version-id', 'boYngPv33ng8HsND7dSXZTE12I7WmDD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002526464953085580-00650e0222-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.065 DEBUG:boto:path=/ffzdqu-12/dummy 2023-09-22T21:07:46.065 DEBUG:boto:auth_path=/ffzdqu-12/dummy 2023-09-22T21:07:46.066 DEBUG:boto:path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.066 DEBUG:boto:auth_path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.066 DEBUG:boto:Method: GET 2023-09-22T21:07:46.066 DEBUG:boto:Path: /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.066 DEBUG:boto:Data: 2023-09-22T21:07:46.066 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.066 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.066 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.066 DEBUG:boto:Params: {} 2023-09-22T21:07:46.067 DEBUG:boto:Token: None 2023-09-22T21:07:46.067 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.067 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wQ6iNb6SnVKwW8jpZhhwpGkPr18= 2023-09-22T21:07:46.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wQ6iNb6SnVKwW8jpZhhwpGkPr18=', 'Content-Length': '0'} 2023-09-22T21:07:46.068 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('x-amz-version-id', 'boYngPv33ng8HsND7dSXZTE12I7WmDD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000349cae96e2a275e9-00650e0222-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.069 DEBUG:boto:path=/ffzdqu-12/dummy 2023-09-22T21:07:46.069 DEBUG:boto:auth_path=/ffzdqu-12/dummy 2023-09-22T21:07:46.069 DEBUG:boto:path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.069 DEBUG:boto:auth_path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.069 DEBUG:boto:Method: HEAD 2023-09-22T21:07:46.069 DEBUG:boto:Path: /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.069 DEBUG:boto:Data: 2023-09-22T21:07:46.069 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.069 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.070 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.070 DEBUG:boto:Params: {} 2023-09-22T21:07:46.070 DEBUG:boto:Token: None 2023-09-22T21:07:46.070 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.070 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ph/zpno3ndUnqTRtFmRpHRXzed4= 2023-09-22T21:07:46.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ph/zpno3ndUnqTRtFmRpHRXzed4=', 'Content-Length': '0'} 2023-09-22T21:07:46.072 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('x-amz-version-id', 'boYngPv33ng8HsND7dSXZTE12I7WmDD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007e03656aa9e2db6e-00650e0222-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.072 DEBUG:boto:path=/ffzdqu-12/dummy 2023-09-22T21:07:46.072 DEBUG:boto:auth_path=/ffzdqu-12/dummy 2023-09-22T21:07:46.072 DEBUG:boto:path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.072 DEBUG:boto:auth_path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.073 DEBUG:boto:Method: HEAD 2023-09-22T21:07:46.073 DEBUG:boto:Path: /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.073 DEBUG:boto:Data: 2023-09-22T21:07:46.073 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.073 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.073 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.073 DEBUG:boto:Params: {} 2023-09-22T21:07:46.074 DEBUG:boto:Token: None 2023-09-22T21:07:46.074 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.074 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ph/zpno3ndUnqTRtFmRpHRXzed4= 2023-09-22T21:07:46.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ph/zpno3ndUnqTRtFmRpHRXzed4=', 'Content-Length': '0'} 2023-09-22T21:07:46.076 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('x-amz-version-id', 'boYngPv33ng8HsND7dSXZTE12I7WmDD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000f70ff00c9278ac7-00650e0222-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-22T21:07:46.077 DEBUG:boto:path=/ffzdqu-12/dummy 2023-09-22T21:07:46.077 DEBUG:boto:auth_path=/ffzdqu-12/dummy 2023-09-22T21:07:46.077 DEBUG:boto:path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.077 DEBUG:boto:auth_path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.077 DEBUG:boto:Method: GET 2023-09-22T21:07:46.077 DEBUG:boto:Path: /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.078 DEBUG:boto:Data: 2023-09-22T21:07:46.078 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.078 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.078 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.078 DEBUG:boto:Params: {} 2023-09-22T21:07:46.078 DEBUG:boto:Token: None 2023-09-22T21:07:46.079 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.079 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wQ6iNb6SnVKwW8jpZhhwpGkPr18= 2023-09-22T21:07:46.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wQ6iNb6SnVKwW8jpZhhwpGkPr18=', 'Content-Length': '0'} 2023-09-22T21:07:46.081 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('x-amz-version-id', 'boYngPv33ng8HsND7dSXZTE12I7WmDD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000050fbd0b04498d05-00650e0222-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.081 DEBUG:boto:path=/ffzdqu-12/dummy 2023-09-22T21:07:46.081 DEBUG:boto:auth_path=/ffzdqu-12/dummy 2023-09-22T21:07:46.081 DEBUG:boto:path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.081 DEBUG:boto:auth_path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.081 DEBUG:boto:Method: GET 2023-09-22T21:07:46.082 DEBUG:boto:Path: /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.082 DEBUG:boto:Data: 2023-09-22T21:07:46.082 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.082 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.082 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.082 DEBUG:boto:Params: {} 2023-09-22T21:07:46.083 DEBUG:boto:Token: None 2023-09-22T21:07:46.083 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.083 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wQ6iNb6SnVKwW8jpZhhwpGkPr18= 2023-09-22T21:07:46.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wQ6iNb6SnVKwW8jpZhhwpGkPr18=', 'Content-Length': '0'} 2023-09-22T21:07:46.085 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('x-amz-version-id', 'boYngPv33ng8HsND7dSXZTE12I7WmDD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000216deea76ae716db-00650e0222-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.085 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:46.085 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:46.085 DEBUG:boto:path=/ffzdqu-12/?prefix=dummy 2023-09-22T21:07:46.086 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=dummy 2023-09-22T21:07:46.086 DEBUG:boto:Method: GET 2023-09-22T21:07:46.086 DEBUG:boto:Path: /ffzdqu-12/?prefix=dummy 2023-09-22T21:07:46.086 DEBUG:boto:Data: 2023-09-22T21:07:46.086 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.086 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.087 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.087 DEBUG:boto:Params: {} 2023-09-22T21:07:46.087 DEBUG:boto:Token: None 2023-09-22T21:07:46.087 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/ 2023-09-22T21:07:46.087 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o= 2023-09-22T21:07:46.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o=', 'Content-Length': '0'} 2023-09-22T21:07:46.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b26a48d8f602ddec-00650e0222-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.091 DEBUG:boto:b'ffzdqu-12dummy1000falsedummy2023-09-22T21:06:21.418Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:46.091 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:46.091 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:46.092 DEBUG:boto:path=/ffzdqu-12/?prefix=dummy 2023-09-22T21:07:46.092 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=dummy 2023-09-22T21:07:46.092 DEBUG:boto:Method: GET 2023-09-22T21:07:46.092 DEBUG:boto:Path: /ffzdqu-12/?prefix=dummy 2023-09-22T21:07:46.092 DEBUG:boto:Data: 2023-09-22T21:07:46.092 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.093 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.093 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.093 DEBUG:boto:Params: {} 2023-09-22T21:07:46.093 DEBUG:boto:Token: None 2023-09-22T21:07:46.093 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/ 2023-09-22T21:07:46.094 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o= 2023-09-22T21:07:46.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o=', 'Content-Length': '0'} 2023-09-22T21:07:46.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078dfd5c1fb17d44e-00650e0222-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.097 DEBUG:boto:b'ffzdqu-12dummy1000falsedummy2023-09-22T21:06:21.418Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:46.097 DEBUG:boto:path=/ffzdqu-12/dummy 2023-09-22T21:07:46.097 DEBUG:boto:auth_path=/ffzdqu-12/dummy 2023-09-22T21:07:46.098 DEBUG:boto:Method: HEAD 2023-09-22T21:07:46.098 DEBUG:boto:Path: /ffzdqu-12/dummy 2023-09-22T21:07:46.098 DEBUG:boto:Data: 2023-09-22T21:07:46.098 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.098 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.098 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.098 DEBUG:boto:Params: {} 2023-09-22T21:07:46.099 DEBUG:boto:Token: None 2023-09-22T21:07:46.099 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/dummy 2023-09-22T21:07:46.099 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Yb3I1oszNO1dSt6l54qGXN8R2Mc= 2023-09-22T21:07:46.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Yb3I1oszNO1dSt6l54qGXN8R2Mc=', 'Content-Length': '0'} 2023-09-22T21:07:46.101 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('x-amz-version-id', 'boYngPv33ng8HsND7dSXZTE12I7WmDD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e62c990b5b389c69-00650e0222-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.101 DEBUG:boto:path=/ffzdqu-12/dummy 2023-09-22T21:07:46.101 DEBUG:boto:auth_path=/ffzdqu-12/dummy 2023-09-22T21:07:46.102 DEBUG:boto:Method: HEAD 2023-09-22T21:07:46.102 DEBUG:boto:Path: /ffzdqu-12/dummy 2023-09-22T21:07:46.102 DEBUG:boto:Data: 2023-09-22T21:07:46.102 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.102 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.102 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.102 DEBUG:boto:Params: {} 2023-09-22T21:07:46.102 DEBUG:boto:Token: None 2023-09-22T21:07:46.103 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/dummy 2023-09-22T21:07:46.103 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Yb3I1oszNO1dSt6l54qGXN8R2Mc= 2023-09-22T21:07:46.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Yb3I1oszNO1dSt6l54qGXN8R2Mc=', 'Content-Length': '0'} 2023-09-22T21:07:46.105 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('x-amz-version-id', 'boYngPv33ng8HsND7dSXZTE12I7WmDD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000712c12ddc466ca80-00650e0222-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-22T21:07:46.106 DEBUG:boto:path=/ffzdqu-12/dummy 2023-09-22T21:07:46.106 DEBUG:boto:auth_path=/ffzdqu-12/dummy 2023-09-22T21:07:46.106 DEBUG:boto:path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.106 DEBUG:boto:auth_path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.106 DEBUG:boto:Method: GET 2023-09-22T21:07:46.107 DEBUG:boto:Path: /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.107 DEBUG:boto:Data: 2023-09-22T21:07:46.107 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.107 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.107 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.107 DEBUG:boto:Params: {} 2023-09-22T21:07:46.107 DEBUG:boto:Token: None 2023-09-22T21:07:46.107 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.108 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wQ6iNb6SnVKwW8jpZhhwpGkPr18= 2023-09-22T21:07:46.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wQ6iNb6SnVKwW8jpZhhwpGkPr18=', 'Content-Length': '0'} 2023-09-22T21:07:46.109 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('x-amz-version-id', 'boYngPv33ng8HsND7dSXZTE12I7WmDD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d42d3d8bfb95d898-00650e0222-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.110 DEBUG:boto:path=/ffzdqu-12/dummy 2023-09-22T21:07:46.110 DEBUG:boto:auth_path=/ffzdqu-12/dummy 2023-09-22T21:07:46.110 DEBUG:boto:path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.110 DEBUG:boto:auth_path=/ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.110 DEBUG:boto:Method: GET 2023-09-22T21:07:46.111 DEBUG:boto:Path: /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.111 DEBUG:boto:Data: 2023-09-22T21:07:46.111 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.111 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.111 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.111 DEBUG:boto:Params: {} 2023-09-22T21:07:46.111 DEBUG:boto:Token: None 2023-09-22T21:07:46.111 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/dummy?versionId=boYngPv33ng8HsND7dSXZTE12I7WmDD 2023-09-22T21:07:46.111 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wQ6iNb6SnVKwW8jpZhhwpGkPr18= 2023-09-22T21:07:46.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wQ6iNb6SnVKwW8jpZhhwpGkPr18=', 'Content-Length': '0'} 2023-09-22T21:07:46.113 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:21 GMT'), ('x-amz-version-id', 'boYngPv33ng8HsND7dSXZTE12I7WmDD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aedd32f23d055081-00650e0222-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.113 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:46.114 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:46.114 DEBUG:boto:path=/ffzdqu-12/?prefix=dummy 2023-09-22T21:07:46.114 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=dummy 2023-09-22T21:07:46.114 DEBUG:boto:Method: GET 2023-09-22T21:07:46.114 DEBUG:boto:Path: /ffzdqu-12/?prefix=dummy 2023-09-22T21:07:46.114 DEBUG:boto:Data: 2023-09-22T21:07:46.114 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.115 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.115 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.115 DEBUG:boto:Params: {} 2023-09-22T21:07:46.115 DEBUG:boto:Token: None 2023-09-22T21:07:46.115 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/ 2023-09-22T21:07:46.115 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o= 2023-09-22T21:07:46.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o=', 'Content-Length': '0'} 2023-09-22T21:07:46.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d8e2de9cc996a33-00650e0222-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.118 DEBUG:boto:b'ffzdqu-12dummy1000falsedummy2023-09-22T21:06:21.418Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:46.118 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:46.119 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:46.119 DEBUG:boto:path=/ffzdqu-12/?prefix=dummy 2023-09-22T21:07:46.119 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=dummy 2023-09-22T21:07:46.119 DEBUG:boto:Method: GET 2023-09-22T21:07:46.119 DEBUG:boto:Path: /ffzdqu-12/?prefix=dummy 2023-09-22T21:07:46.119 DEBUG:boto:Data: 2023-09-22T21:07:46.119 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.120 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.120 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.120 DEBUG:boto:Params: {} 2023-09-22T21:07:46.120 DEBUG:boto:Token: None 2023-09-22T21:07:46.120 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/ 2023-09-22T21:07:46.120 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o= 2023-09-22T21:07:46.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o=', 'Content-Length': '0'} 2023-09-22T21:07:46.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086cfe98ddc9f4139-00650e0222-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.123 DEBUG:boto:b'ffzdqu-12dummy1000falsedummy2023-09-22T21:06:21.418Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:46.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-22T21:07:46.123 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:07:46.124 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:07:46.124 DEBUG:boto:path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.124 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.124 DEBUG:boto:Method: GET 2023-09-22T21:07:46.124 DEBUG:boto:Path: /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.124 DEBUG:boto:Data: 2023-09-22T21:07:46.124 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.124 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.124 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.125 DEBUG:boto:Params: {} 2023-09-22T21:07:46.125 DEBUG:boto:Token: None 2023-09-22T21:07:46.125 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.125 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYvYibgm+hOOV9VeRotxD01LLjY= 2023-09-22T21:07:46.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYvYibgm+hOOV9VeRotxD01LLjY=', 'Content-Length': '0'} 2023-09-22T21:07:46.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', 'ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000108bfe61eba400f3-00650e0222-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.127 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:07:46.127 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:07:46.127 DEBUG:boto:path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.127 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.128 DEBUG:boto:Method: GET 2023-09-22T21:07:46.128 DEBUG:boto:Path: /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.128 DEBUG:boto:Data: 2023-09-22T21:07:46.128 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.128 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.128 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.128 DEBUG:boto:Params: {} 2023-09-22T21:07:46.128 DEBUG:boto:Token: None 2023-09-22T21:07:46.129 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.129 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYvYibgm+hOOV9VeRotxD01LLjY= 2023-09-22T21:07:46.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYvYibgm+hOOV9VeRotxD01LLjY=', 'Content-Length': '0'} 2023-09-22T21:07:46.131 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', 'ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000200b6cd81c00a29a-00650e0222-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.131 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:07:46.132 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:07:46.132 DEBUG:boto:path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.132 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.132 DEBUG:boto:Method: HEAD 2023-09-22T21:07:46.132 DEBUG:boto:Path: /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.132 DEBUG:boto:Data: 2023-09-22T21:07:46.132 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.132 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.132 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.133 DEBUG:boto:Params: {} 2023-09-22T21:07:46.133 DEBUG:boto:Token: None 2023-09-22T21:07:46.133 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.133 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1UWIKz6IH6aDVg/wPswvtEUBRw4= 2023-09-22T21:07:46.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1UWIKz6IH6aDVg/wPswvtEUBRw4=', 'Content-Length': '0'} 2023-09-22T21:07:46.135 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', 'ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000160aabb5d4f4e1d-00650e0222-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.135 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:07:46.135 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:07:46.136 DEBUG:boto:path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.136 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.136 DEBUG:boto:Method: HEAD 2023-09-22T21:07:46.136 DEBUG:boto:Path: /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.136 DEBUG:boto:Data: 2023-09-22T21:07:46.136 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.136 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.137 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.137 DEBUG:boto:Params: {} 2023-09-22T21:07:46.137 DEBUG:boto:Token: None 2023-09-22T21:07:46.137 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.137 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1UWIKz6IH6aDVg/wPswvtEUBRw4= 2023-09-22T21:07:46.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1UWIKz6IH6aDVg/wPswvtEUBRw4=', 'Content-Length': '0'} 2023-09-22T21:07:46.139 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', 'ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfeaf5b1d5a88136-00650e0222-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-22T21:07:46.140 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:07:46.140 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:07:46.140 DEBUG:boto:path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.140 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.140 DEBUG:boto:Method: GET 2023-09-22T21:07:46.141 DEBUG:boto:Path: /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.141 DEBUG:boto:Data: 2023-09-22T21:07:46.141 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.141 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.141 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.141 DEBUG:boto:Params: {} 2023-09-22T21:07:46.141 DEBUG:boto:Token: None 2023-09-22T21:07:46.142 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.142 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYvYibgm+hOOV9VeRotxD01LLjY= 2023-09-22T21:07:46.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYvYibgm+hOOV9VeRotxD01LLjY=', 'Content-Length': '0'} 2023-09-22T21:07:46.144 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', 'ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045760cfefcc78c8e-00650e0222-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.186 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:07:46.186 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:07:46.187 DEBUG:boto:path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.187 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.187 DEBUG:boto:Method: GET 2023-09-22T21:07:46.187 DEBUG:boto:Path: /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.187 DEBUG:boto:Data: 2023-09-22T21:07:46.187 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.187 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.187 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.187 DEBUG:boto:Params: {} 2023-09-22T21:07:46.187 DEBUG:boto:Token: None 2023-09-22T21:07:46.188 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.188 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYvYibgm+hOOV9VeRotxD01LLjY= 2023-09-22T21:07:46.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYvYibgm+hOOV9VeRotxD01LLjY=', 'Content-Length': '0'} 2023-09-22T21:07:46.190 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', 'ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027f7394c8fc961e7-00650e0222-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.191 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:46.191 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:46.191 DEBUG:boto:path=/ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:46.191 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:46.191 DEBUG:boto:Method: GET 2023-09-22T21:07:46.191 DEBUG:boto:Path: /ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:46.192 DEBUG:boto:Data: 2023-09-22T21:07:46.192 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.192 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.192 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.192 DEBUG:boto:Params: {} 2023-09-22T21:07:46.192 DEBUG:boto:Token: None 2023-09-22T21:07:46.192 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/ 2023-09-22T21:07:46.192 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o= 2023-09-22T21:07:46.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o=', 'Content-Length': '0'} 2023-09-22T21:07:46.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbbe064c022a8a53-00650e0222-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.197 DEBUG:boto:b'ffzdqu-12obj-a11000falseobj-a12023-09-22T21:06:41.903Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:46.198 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:46.198 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:46.198 DEBUG:boto:path=/ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:46.198 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:46.199 DEBUG:boto:Method: GET 2023-09-22T21:07:46.199 DEBUG:boto:Path: /ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:46.199 DEBUG:boto:Data: 2023-09-22T21:07:46.199 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.199 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.199 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.199 DEBUG:boto:Params: {} 2023-09-22T21:07:46.199 DEBUG:boto:Token: None 2023-09-22T21:07:46.200 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/ 2023-09-22T21:07:46.200 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o= 2023-09-22T21:07:46.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o=', 'Content-Length': '0'} 2023-09-22T21:07:46.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cea19e34eeb03ec2-00650e0222-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.204 DEBUG:boto:b'ffzdqu-12obj-a11000falseobj-a12023-09-22T21:06:41.903Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:46.205 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:07:46.205 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:07:46.205 DEBUG:boto:Method: HEAD 2023-09-22T21:07:46.205 DEBUG:boto:Path: /ffzdqu-12/obj-a1 2023-09-22T21:07:46.205 DEBUG:boto:Data: 2023-09-22T21:07:46.205 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.206 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.206 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.206 DEBUG:boto:Params: {} 2023-09-22T21:07:46.206 DEBUG:boto:Token: None 2023-09-22T21:07:46.206 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/obj-a1 2023-09-22T21:07:46.206 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2L87vTT76cGProjJQwaHdEqLAwU= 2023-09-22T21:07:46.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2L87vTT76cGProjJQwaHdEqLAwU=', 'Content-Length': '0'} 2023-09-22T21:07:46.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', 'ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd2b42ccd1b40c14-00650e0222-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.210 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:07:46.210 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:07:46.211 DEBUG:boto:Method: HEAD 2023-09-22T21:07:46.211 DEBUG:boto:Path: /ffzdqu-12/obj-a1 2023-09-22T21:07:46.211 DEBUG:boto:Data: 2023-09-22T21:07:46.211 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.211 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.211 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.211 DEBUG:boto:Params: {} 2023-09-22T21:07:46.211 DEBUG:boto:Token: None 2023-09-22T21:07:46.211 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/obj-a1 2023-09-22T21:07:46.212 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2L87vTT76cGProjJQwaHdEqLAwU= 2023-09-22T21:07:46.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2L87vTT76cGProjJQwaHdEqLAwU=', 'Content-Length': '0'} 2023-09-22T21:07:46.215 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', 'ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f444ee6be1fc7f7-00650e0222-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-22T21:07:46.216 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:07:46.216 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:07:46.216 DEBUG:boto:path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.216 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.216 DEBUG:boto:Method: GET 2023-09-22T21:07:46.216 DEBUG:boto:Path: /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.216 DEBUG:boto:Data: 2023-09-22T21:07:46.216 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.216 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.216 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.217 DEBUG:boto:Params: {} 2023-09-22T21:07:46.217 DEBUG:boto:Token: None 2023-09-22T21:07:46.217 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.217 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYvYibgm+hOOV9VeRotxD01LLjY= 2023-09-22T21:07:46.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYvYibgm+hOOV9VeRotxD01LLjY=', 'Content-Length': '0'} 2023-09-22T21:07:46.220 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', 'ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d90d56b0913c17e-00650e0222-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.262 DEBUG:boto:path=/ffzdqu-12/obj-a1 2023-09-22T21:07:46.262 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1 2023-09-22T21:07:46.263 DEBUG:boto:path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.263 DEBUG:boto:auth_path=/ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.263 DEBUG:boto:Method: GET 2023-09-22T21:07:46.263 DEBUG:boto:Path: /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.263 DEBUG:boto:Data: 2023-09-22T21:07:46.263 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.263 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.263 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.263 DEBUG:boto:Params: {} 2023-09-22T21:07:46.264 DEBUG:boto:Token: None 2023-09-22T21:07:46.264 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/obj-a1?versionId=ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL 2023-09-22T21:07:46.264 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYvYibgm+hOOV9VeRotxD01LLjY= 2023-09-22T21:07:46.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYvYibgm+hOOV9VeRotxD01LLjY=', 'Content-Length': '0'} 2023-09-22T21:07:46.267 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:41 GMT'), ('x-amz-version-id', 'ENtJQRcMHYYCQC9B1Taipgnc1aUE.HL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ff9ac69a1678e18-00650e0222-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.267 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:46.267 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:46.267 DEBUG:boto:path=/ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:46.267 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:46.268 DEBUG:boto:Method: GET 2023-09-22T21:07:46.268 DEBUG:boto:Path: /ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:46.268 DEBUG:boto:Data: 2023-09-22T21:07:46.268 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.268 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.268 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.268 DEBUG:boto:Params: {} 2023-09-22T21:07:46.269 DEBUG:boto:Token: None 2023-09-22T21:07:46.269 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/ 2023-09-22T21:07:46.269 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o= 2023-09-22T21:07:46.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o=', 'Content-Length': '0'} 2023-09-22T21:07:46.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000015ddae46c2434f4-00650e0222-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.314 DEBUG:boto:b'ffzdqu-12obj-a11000falseobj-a12023-09-22T21:06:41.903Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:46.315 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:46.315 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:46.315 DEBUG:boto:path=/ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:46.315 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:46.315 DEBUG:boto:Method: GET 2023-09-22T21:07:46.315 DEBUG:boto:Path: /ffzdqu-12/?prefix=obj-a1 2023-09-22T21:07:46.315 DEBUG:boto:Data: 2023-09-22T21:07:46.315 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.315 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.316 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.316 DEBUG:boto:Params: {} 2023-09-22T21:07:46.316 DEBUG:boto:Token: None 2023-09-22T21:07:46.316 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/ 2023-09-22T21:07:46.316 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o= 2023-09-22T21:07:46.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o=', 'Content-Length': '0'} 2023-09-22T21:07:46.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063c9e8fbaa97b9cc-00650e0222-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.321 DEBUG:boto:b'ffzdqu-12obj-a11000falseobj-a12023-09-22T21:06:41.903Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:46.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-22T21:07:46.322 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:07:46.322 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:07:46.322 DEBUG:boto:path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.322 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.322 DEBUG:boto:Method: GET 2023-09-22T21:07:46.322 DEBUG:boto:Path: /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.322 DEBUG:boto:Data: 2023-09-22T21:07:46.322 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.322 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.323 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.323 DEBUG:boto:Params: {} 2023-09-22T21:07:46.323 DEBUG:boto:Token: None 2023-09-22T21:07:46.323 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.323 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WZq+VGGRVg0FMpr7Rvc3rqdCjnE= 2023-09-22T21:07:46.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WZq+VGGRVg0FMpr7Rvc3rqdCjnE=', 'Content-Length': '0'} 2023-09-22T21:07:46.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('x-amz-version-id', 'u7s2N-8HbpXwMBMoersKLSCgHbn-fnr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2b900e7f4f411a3-00650e0222-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.326 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:07:46.326 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:07:46.326 DEBUG:boto:path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.327 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.327 DEBUG:boto:Method: GET 2023-09-22T21:07:46.327 DEBUG:boto:Path: /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.327 DEBUG:boto:Data: 2023-09-22T21:07:46.327 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.327 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.327 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.327 DEBUG:boto:Params: {} 2023-09-22T21:07:46.327 DEBUG:boto:Token: None 2023-09-22T21:07:46.328 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.328 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WZq+VGGRVg0FMpr7Rvc3rqdCjnE= 2023-09-22T21:07:46.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WZq+VGGRVg0FMpr7Rvc3rqdCjnE=', 'Content-Length': '0'} 2023-09-22T21:07:46.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('x-amz-version-id', 'u7s2N-8HbpXwMBMoersKLSCgHbn-fnr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c96e51dfaddfa3d-00650e0222-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.374 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:07:46.374 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:07:46.375 DEBUG:boto:path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.375 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.375 DEBUG:boto:Method: HEAD 2023-09-22T21:07:46.375 DEBUG:boto:Path: /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.375 DEBUG:boto:Data: 2023-09-22T21:07:46.375 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.375 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.375 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.375 DEBUG:boto:Params: {} 2023-09-22T21:07:46.376 DEBUG:boto:Token: None 2023-09-22T21:07:46.376 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.376 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nADHn19gDqHtZLWWb6ZI0b9TcfE= 2023-09-22T21:07:46.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nADHn19gDqHtZLWWb6ZI0b9TcfE=', 'Content-Length': '0'} 2023-09-22T21:07:46.379 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('x-amz-version-id', 'u7s2N-8HbpXwMBMoersKLSCgHbn-fnr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf034c87072098da-00650e0222-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.379 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:07:46.379 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:07:46.379 DEBUG:boto:path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.379 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.380 DEBUG:boto:Method: HEAD 2023-09-22T21:07:46.380 DEBUG:boto:Path: /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.380 DEBUG:boto:Data: 2023-09-22T21:07:46.380 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.380 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.380 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.380 DEBUG:boto:Params: {} 2023-09-22T21:07:46.380 DEBUG:boto:Token: None 2023-09-22T21:07:46.380 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.381 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nADHn19gDqHtZLWWb6ZI0b9TcfE= 2023-09-22T21:07:46.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nADHn19gDqHtZLWWb6ZI0b9TcfE=', 'Content-Length': '0'} 2023-09-22T21:07:46.383 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('x-amz-version-id', 'u7s2N-8HbpXwMBMoersKLSCgHbn-fnr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a25e7f854aa4856-00650e0222-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-22T21:07:46.384 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:07:46.384 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:07:46.384 DEBUG:boto:path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.384 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.384 DEBUG:boto:Method: GET 2023-09-22T21:07:46.384 DEBUG:boto:Path: /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.385 DEBUG:boto:Data: 2023-09-22T21:07:46.385 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.385 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.385 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.385 DEBUG:boto:Params: {} 2023-09-22T21:07:46.385 DEBUG:boto:Token: None 2023-09-22T21:07:46.385 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.385 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WZq+VGGRVg0FMpr7Rvc3rqdCjnE= 2023-09-22T21:07:46.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WZq+VGGRVg0FMpr7Rvc3rqdCjnE=', 'Content-Length': '0'} 2023-09-22T21:07:46.388 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('x-amz-version-id', 'u7s2N-8HbpXwMBMoersKLSCgHbn-fnr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df32e1980e29b504-00650e0222-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.388 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:07:46.388 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:07:46.389 DEBUG:boto:path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.389 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.389 DEBUG:boto:Method: GET 2023-09-22T21:07:46.389 DEBUG:boto:Path: /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.389 DEBUG:boto:Data: 2023-09-22T21:07:46.389 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.389 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.389 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.389 DEBUG:boto:Params: {} 2023-09-22T21:07:46.390 DEBUG:boto:Token: None 2023-09-22T21:07:46.390 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.390 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WZq+VGGRVg0FMpr7Rvc3rqdCjnE= 2023-09-22T21:07:46.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WZq+VGGRVg0FMpr7Rvc3rqdCjnE=', 'Content-Length': '0'} 2023-09-22T21:07:46.392 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('x-amz-version-id', 'u7s2N-8HbpXwMBMoersKLSCgHbn-fnr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bd49b34f6e2b32b-00650e0222-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.438 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:46.439 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:46.439 DEBUG:boto:path=/ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:46.439 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:46.439 DEBUG:boto:Method: GET 2023-09-22T21:07:46.439 DEBUG:boto:Path: /ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:46.439 DEBUG:boto:Data: 2023-09-22T21:07:46.439 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.439 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.439 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.440 DEBUG:boto:Params: {} 2023-09-22T21:07:46.440 DEBUG:boto:Token: None 2023-09-22T21:07:46.440 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/ 2023-09-22T21:07:46.440 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o= 2023-09-22T21:07:46.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o=', 'Content-Length': '0'} 2023-09-22T21:07:46.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a0c04d62df5930f-00650e0222-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.445 DEBUG:boto:b'ffzdqu-12obj-a21000falseobj-a22023-09-22T21:06:42.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:46.445 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:46.446 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:46.446 DEBUG:boto:path=/ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:46.446 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:46.446 DEBUG:boto:Method: GET 2023-09-22T21:07:46.446 DEBUG:boto:Path: /ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:46.446 DEBUG:boto:Data: 2023-09-22T21:07:46.446 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.446 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.446 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.447 DEBUG:boto:Params: {} 2023-09-22T21:07:46.447 DEBUG:boto:Token: None 2023-09-22T21:07:46.447 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/ 2023-09-22T21:07:46.447 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o= 2023-09-22T21:07:46.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o=', 'Content-Length': '0'} 2023-09-22T21:07:46.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b818859ec379c6fc-00650e0222-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.498 DEBUG:boto:b'ffzdqu-12obj-a21000falseobj-a22023-09-22T21:06:42.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:46.499 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:07:46.499 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:07:46.499 DEBUG:boto:Method: HEAD 2023-09-22T21:07:46.499 DEBUG:boto:Path: /ffzdqu-12/obj-a2 2023-09-22T21:07:46.499 DEBUG:boto:Data: 2023-09-22T21:07:46.499 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.499 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.499 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.500 DEBUG:boto:Params: {} 2023-09-22T21:07:46.500 DEBUG:boto:Token: None 2023-09-22T21:07:46.500 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/obj-a2 2023-09-22T21:07:46.500 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:d1iaMG1+ZbkO4EMrjiF110h9r60= 2023-09-22T21:07:46.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:d1iaMG1+ZbkO4EMrjiF110h9r60=', 'Content-Length': '0'} 2023-09-22T21:07:46.503 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('x-amz-version-id', 'u7s2N-8HbpXwMBMoersKLSCgHbn-fnr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca6e7f8c8f39243e-00650e0222-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.504 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:07:46.504 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:07:46.504 DEBUG:boto:Method: HEAD 2023-09-22T21:07:46.504 DEBUG:boto:Path: /ffzdqu-12/obj-a2 2023-09-22T21:07:46.504 DEBUG:boto:Data: 2023-09-22T21:07:46.504 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.504 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.505 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.505 DEBUG:boto:Params: {} 2023-09-22T21:07:46.505 DEBUG:boto:Token: None 2023-09-22T21:07:46.505 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/obj-a2 2023-09-22T21:07:46.505 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:d1iaMG1+ZbkO4EMrjiF110h9r60= 2023-09-22T21:07:46.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:d1iaMG1+ZbkO4EMrjiF110h9r60=', 'Content-Length': '0'} 2023-09-22T21:07:46.508 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('x-amz-version-id', 'u7s2N-8HbpXwMBMoersKLSCgHbn-fnr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f229187d0775cf5-00650e0222-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-22T21:07:46.509 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:07:46.509 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:07:46.509 DEBUG:boto:path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.509 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.509 DEBUG:boto:Method: GET 2023-09-22T21:07:46.509 DEBUG:boto:Path: /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.509 DEBUG:boto:Data: 2023-09-22T21:07:46.510 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.510 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.510 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.510 DEBUG:boto:Params: {} 2023-09-22T21:07:46.510 DEBUG:boto:Token: None 2023-09-22T21:07:46.510 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.510 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WZq+VGGRVg0FMpr7Rvc3rqdCjnE= 2023-09-22T21:07:46.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WZq+VGGRVg0FMpr7Rvc3rqdCjnE=', 'Content-Length': '0'} 2023-09-22T21:07:46.513 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('x-amz-version-id', 'u7s2N-8HbpXwMBMoersKLSCgHbn-fnr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004740393075d5a933-00650e0222-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.554 DEBUG:boto:path=/ffzdqu-12/obj-a2 2023-09-22T21:07:46.555 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2 2023-09-22T21:07:46.555 DEBUG:boto:path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.555 DEBUG:boto:auth_path=/ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.555 DEBUG:boto:Method: GET 2023-09-22T21:07:46.555 DEBUG:boto:Path: /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.555 DEBUG:boto:Data: 2023-09-22T21:07:46.555 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.555 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.556 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.556 DEBUG:boto:Params: {} 2023-09-22T21:07:46.556 DEBUG:boto:Token: None 2023-09-22T21:07:46.556 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/obj-a2?versionId=u7s2N-8HbpXwMBMoersKLSCgHbn-fnr 2023-09-22T21:07:46.556 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WZq+VGGRVg0FMpr7Rvc3rqdCjnE= 2023-09-22T21:07:46.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WZq+VGGRVg0FMpr7Rvc3rqdCjnE=', 'Content-Length': '0'} 2023-09-22T21:07:46.559 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:06:42 GMT'), ('x-amz-version-id', 'u7s2N-8HbpXwMBMoersKLSCgHbn-fnr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3727f3b5636ace5-00650e0222-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.603 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:46.603 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:46.603 DEBUG:boto:path=/ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:46.603 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:46.603 DEBUG:boto:Method: GET 2023-09-22T21:07:46.603 DEBUG:boto:Path: /ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:46.603 DEBUG:boto:Data: 2023-09-22T21:07:46.604 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.604 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:07:46.604 DEBUG:boto:Port: 8001 2023-09-22T21:07:46.604 DEBUG:boto:Params: {} 2023-09-22T21:07:46.604 DEBUG:boto:Token: None 2023-09-22T21:07:46.604 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/ 2023-09-22T21:07:46.604 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o= 2023-09-22T21:07:46.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o=', 'Content-Length': '0'} 2023-09-22T21:07:46.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f302509e6df034b7-00650e0222-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.609 DEBUG:boto:b'ffzdqu-12obj-a21000falseobj-a22023-09-22T21:06:42.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:46.610 DEBUG:boto:path=/ffzdqu-12/ 2023-09-22T21:07:46.610 DEBUG:boto:auth_path=/ffzdqu-12/ 2023-09-22T21:07:46.610 DEBUG:boto:path=/ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:46.610 DEBUG:boto:auth_path=/ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:46.610 DEBUG:boto:Method: GET 2023-09-22T21:07:46.611 DEBUG:boto:Path: /ffzdqu-12/?prefix=obj-a2 2023-09-22T21:07:46.611 DEBUG:boto:Data: 2023-09-22T21:07:46.611 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.611 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.611 DEBUG:boto:Params: {} 2023-09-22T21:07:46.611 DEBUG:boto:Token: None 2023-09-22T21:07:46.611 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-12/ 2023-09-22T21:07:46.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o= 2023-09-22T21:07:46.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fpW60n+pgqKxddmEhL/y2acdy/o=', 'Content-Length': '0'} 2023-09-22T21:07:46.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce1be667dafb35f7-00650e0222-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.662 DEBUG:boto:b'ffzdqu-12obj-a21000falseobj-a22023-09-22T21:06:42.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:07:46.663 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-12 zones={a2, a1} 2023-09-22T21:07:46.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2023-09-22T21:07:46.664 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:07:46.664 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:07:46.664 DEBUG:boto:Method: PUT 2023-09-22T21:07:46.665 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:07:46.665 DEBUG:boto:Data: 2023-09-22T21:07:46.665 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.665 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.665 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.665 DEBUG:boto:Params: {} 2023-09-22T21:07:46.665 DEBUG:boto:Token: None 2023-09-22T21:07:46.665 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-13/ 2023-09-22T21:07:46.666 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:741xDUIvPpVvqTNYeAXK1x13cH4= 2023-09-22T21:07:46.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:741xDUIvPpVvqTNYeAXK1x13cH4=', 'Content-Length': '0'} 2023-09-22T21:07:46.702 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003ad1e3f86854cef6-00650e0222-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.702 DEBUG:rgw_multi.tests:created bucket=ffzdqu-13 2023-09-22T21:07:46.702 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:07:46.702 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:07:46.703 DEBUG:boto:path=/ffzdqu-13/?versioning 2023-09-22T21:07:46.703 DEBUG:boto:auth_path=/ffzdqu-13/?versioning 2023-09-22T21:07:46.703 DEBUG:boto:Method: PUT 2023-09-22T21:07:46.703 DEBUG:boto:Path: /ffzdqu-13/?versioning 2023-09-22T21:07:46.703 DEBUG:boto:Data: Enabled Disabled 2023-09-22T21:07:46.703 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.703 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.703 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.703 DEBUG:boto:Params: {} 2023-09-22T21:07:46.703 DEBUG:boto:Token: None 2023-09-22T21:07:46.704 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-13/?versioning 2023-09-22T21:07:46.704 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7m5vak2LRuujup7S/LQkm9JUXYY= 2023-09-22T21:07:46.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7m5vak2LRuujup7S/LQkm9JUXYY=', 'Content-Length': '228'} 2023-09-22T21:07:46.715 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001a3779adf74d92d6-00650e0222-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:07:46.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:07:46.716 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:07:46.763 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:07:46.764 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:07:46.803 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:07:46.821 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695416866.717400_63.1","last_update":"2023-09-22T21:07:46.717400Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:07:46.821 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416866.717400_63.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:07:46.821 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:07:46.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:07:46.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:07:46.822 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:07:46.905 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:07:46.905 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:07:46.946 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:46.947+0000 7fc67f2faa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:07:46.947 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:46.947+0000 7fc67f2faa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:07:46.965 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695416866.717400_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:07:46.717400Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:07:46Z"} 2023-09-22T21:07:46.966 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695416866.717400_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:07:46.717400Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:07:46Z"} 2023-09-22T21:07:46.966 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695416866.717400_63.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:07:46.717400Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:07:46.966 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416866.717400_63.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:07:46.966 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416866.717400_63.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:07:46.966 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:07:46.966 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:07:46.967 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:07:46.967 DEBUG:boto:Method: HEAD 2023-09-22T21:07:46.967 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:07:46.967 DEBUG:boto:Data: 2023-09-22T21:07:46.967 DEBUG:boto:Headers: {} 2023-09-22T21:07:46.967 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.967 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.967 DEBUG:boto:Params: {} 2023-09-22T21:07:46.967 DEBUG:boto:Token: None 2023-09-22T21:07:46.968 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-13/ 2023-09-22T21:07:46.968 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:htJUolyMcQn71R+jB2nLh9lwidc= 2023-09-22T21:07:46.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:htJUolyMcQn71R+jB2nLh9lwidc=', 'Content-Length': '0'} 2023-09-22T21:07:46.970 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c355787f33643e5-00650e0222-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.971 DEBUG:boto:path=/ffzdqu-13/dummy 2023-09-22T21:07:46.971 DEBUG:boto:auth_path=/ffzdqu-13/dummy 2023-09-22T21:07:46.971 DEBUG:boto:Method: PUT 2023-09-22T21:07:46.971 DEBUG:boto:Path: /ffzdqu-13/dummy 2023-09-22T21:07:46.971 DEBUG:boto:Data: 2023-09-22T21:07:46.971 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-22T21:07:46.971 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:07:46.971 DEBUG:boto:Port: 8000 2023-09-22T21:07:46.971 DEBUG:boto:Params: {} 2023-09-22T21:07:46.971 DEBUG:boto:Token: None 2023-09-22T21:07:46.972 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 22 Sep 2023 21:07:46 GMT /ffzdqu-13/dummy 2023-09-22T21:07:46.972 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7gDdvGTUlz/ToAVSMtIEqz4t0f0= 2023-09-22T21:07:46.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:07:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7gDdvGTUlz/ToAVSMtIEqz4t0f0='} 2023-09-22T21:07:46.985 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L'), ('x-amz-request-id', 'tx0000048254bfe70ac83aa-00650e0222-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:07:46.986 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-13 2023-09-22T21:07:46.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:07:46.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:07:46.987 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:07:47.050 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:07:47.050 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:07:47.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:47.091+0000 7f305a5f9a40 1 waiting to reach incremental sync.. 2023-09-22T21:07:52.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:52.091+0000 7f305a5f9a40 1 waiting to reach incremental sync.. 2023-09-22T21:07:57.092 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:07:57.091+0000 7f305a5f9a40 1 waiting to reach incremental sync.. 2023-09-22T21:08:02.095 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:08:02.095+0000 7f305a5f9a40 1 bucket sync caught up with source: 2023-09-22T21:08:02.096 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.11.11, , , , , ] 2023-09-22T21:08:02.096 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000002.11.11, , , , , ] 2023-09-22T21:08:02.096 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:08:02.095+0000 7f305a5f9a40 0 bucket checkpoint complete 2023-09-22T21:08:02.125 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-13 2023-09-22T21:08:02.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:08:02.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:08:02.125 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:08:02.217 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:08:02.217 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:08:02.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:08:02.259+0000 7f73ba178a40 1 waiting to reach incremental sync.. 2023-09-22T21:08:07.259 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:08:07.259+0000 7f73ba178a40 1 waiting to reach incremental sync.. 2023-09-22T21:08:12.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:08:12.259+0000 7f73ba178a40 1 waiting to reach incremental sync.. 2023-09-22T21:08:17.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:08:17.263+0000 7f73ba178a40 1 waiting to reach incremental sync.. 2023-09-22T21:08:22.264 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:08:22.263+0000 7f73ba178a40 1 bucket sync caught up with source: 2023-09-22T21:08:22.264 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.27.11, , , , , ] 2023-09-22T21:08:22.265 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000002.27.11, , , , , ] 2023-09-22T21:08:22.265 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:08:22.263+0000 7f73ba178a40 0 bucket checkpoint complete 2023-09-22T21:08:22.293 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-13 zones={a2, a1} 2023-09-22T21:08:22.293 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.293 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.293 DEBUG:boto:Method: HEAD 2023-09-22T21:08:22.293 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:22.294 DEBUG:boto:Data: 2023-09-22T21:08:22.294 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.294 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.294 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.294 DEBUG:boto:Params: {} 2023-09-22T21:08:22.294 DEBUG:boto:Token: None 2023-09-22T21:08:22.294 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/ 2023-09-22T21:08:22.294 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI= 2023-09-22T21:08:22.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI=', 'Content-Length': '0'} 2023-09-22T21:08:22.298 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008dd775d74197722e-00650e0246-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.298 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.298 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.299 DEBUG:boto:Method: HEAD 2023-09-22T21:08:22.299 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:22.299 DEBUG:boto:Data: 2023-09-22T21:08:22.299 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.299 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.299 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.299 DEBUG:boto:Params: {} 2023-09-22T21:08:22.299 DEBUG:boto:Token: None 2023-09-22T21:08:22.299 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/ 2023-09-22T21:08:22.300 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI= 2023-09-22T21:08:22.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI=', 'Content-Length': '0'} 2023-09-22T21:08:22.303 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088de49c9d005c8ad-00650e0246-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.303 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:08:22.303 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.303 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.303 DEBUG:boto:path=/ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:22.303 DEBUG:boto:auth_path=/ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:22.303 DEBUG:boto:Method: GET 2023-09-22T21:08:22.304 DEBUG:boto:Path: /ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:22.304 DEBUG:boto:Data: 2023-09-22T21:08:22.304 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.304 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.304 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.304 DEBUG:boto:Params: {} 2023-09-22T21:08:22.304 DEBUG:boto:Token: None 2023-09-22T21:08:22.304 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/?versions 2023-09-22T21:08:22.304 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:K5IYM1o2uI0n74TQuhekOAp5cZI= 2023-09-22T21:08:22.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:K5IYM1o2uI0n74TQuhekOAp5cZI=', 'Content-Length': '0'} 2023-09-22T21:08:22.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000313afaf2e9779909-00650e0246-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.308 DEBUG:boto:b'ffzdqu-13999falsedummyEH3tK3P9S8trmvB9h-eP4MX-b1Uhu-Ltrue2023-09-22T21:07:46.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:22.309 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-22T21:08:22.309 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:08:22.309 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.309 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.309 DEBUG:boto:path=/ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:22.309 DEBUG:boto:auth_path=/ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:22.309 DEBUG:boto:Method: GET 2023-09-22T21:08:22.309 DEBUG:boto:Path: /ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:22.310 DEBUG:boto:Data: 2023-09-22T21:08:22.310 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.310 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.310 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.310 DEBUG:boto:Params: {} 2023-09-22T21:08:22.310 DEBUG:boto:Token: None 2023-09-22T21:08:22.310 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/?versions 2023-09-22T21:08:22.310 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:K5IYM1o2uI0n74TQuhekOAp5cZI= 2023-09-22T21:08:22.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:K5IYM1o2uI0n74TQuhekOAp5cZI=', 'Content-Length': '0'} 2023-09-22T21:08:22.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000946b9448f3fc9be2-00650e0246-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.354 DEBUG:boto:b'ffzdqu-13999falsedummyEH3tK3P9S8trmvB9h-eP4MX-b1Uhu-Ltrue2023-09-22T21:07:46.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:22.355 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-22T21:08:22.355 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.355 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.355 DEBUG:boto:path=/ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:22.355 DEBUG:boto:auth_path=/ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:22.355 DEBUG:boto:Method: GET 2023-09-22T21:08:22.355 DEBUG:boto:Path: /ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:22.355 DEBUG:boto:Data: 2023-09-22T21:08:22.355 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.355 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.356 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.356 DEBUG:boto:Params: {} 2023-09-22T21:08:22.356 DEBUG:boto:Token: None 2023-09-22T21:08:22.356 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/?versions 2023-09-22T21:08:22.356 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:K5IYM1o2uI0n74TQuhekOAp5cZI= 2023-09-22T21:08:22.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:K5IYM1o2uI0n74TQuhekOAp5cZI=', 'Content-Length': '0'} 2023-09-22T21:08:22.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c1998bf5891dcfb-00650e0246-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.361 DEBUG:boto:b'ffzdqu-13999falsedummyEH3tK3P9S8trmvB9h-eP4MX-b1Uhu-Ltrue2023-09-22T21:07:46.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:22.361 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.361 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.361 DEBUG:boto:path=/ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:22.361 DEBUG:boto:auth_path=/ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:22.361 DEBUG:boto:Method: GET 2023-09-22T21:08:22.362 DEBUG:boto:Path: /ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:22.362 DEBUG:boto:Data: 2023-09-22T21:08:22.362 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.362 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.362 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.362 DEBUG:boto:Params: {} 2023-09-22T21:08:22.362 DEBUG:boto:Token: None 2023-09-22T21:08:22.362 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/?versions 2023-09-22T21:08:22.362 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:K5IYM1o2uI0n74TQuhekOAp5cZI= 2023-09-22T21:08:22.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:K5IYM1o2uI0n74TQuhekOAp5cZI=', 'Content-Length': '0'} 2023-09-22T21:08:22.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c39b81f31e76425f-00650e0246-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.366 DEBUG:boto:b'ffzdqu-13999falsedummyEH3tK3P9S8trmvB9h-eP4MX-b1Uhu-Ltrue2023-09-22T21:07:46.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:22.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-22T21:08:22.367 DEBUG:boto:path=/ffzdqu-13/dummy 2023-09-22T21:08:22.367 DEBUG:boto:auth_path=/ffzdqu-13/dummy 2023-09-22T21:08:22.367 DEBUG:boto:path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.367 DEBUG:boto:auth_path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.367 DEBUG:boto:Method: GET 2023-09-22T21:08:22.367 DEBUG:boto:Path: /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.367 DEBUG:boto:Data: 2023-09-22T21:08:22.367 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.368 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.368 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.368 DEBUG:boto:Params: {} 2023-09-22T21:08:22.368 DEBUG:boto:Token: None 2023-09-22T21:08:22.368 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.368 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MiSMW9Ls4s0AXs5pS4N3E4ttSbU= 2023-09-22T21:08:22.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MiSMW9Ls4s0AXs5pS4N3E4ttSbU=', 'Content-Length': '0'} 2023-09-22T21:08:22.371 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('x-amz-version-id', 'EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005cb207b26567f6c6-00650e0246-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.371 DEBUG:boto:path=/ffzdqu-13/dummy 2023-09-22T21:08:22.371 DEBUG:boto:auth_path=/ffzdqu-13/dummy 2023-09-22T21:08:22.371 DEBUG:boto:path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.371 DEBUG:boto:auth_path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.371 DEBUG:boto:Method: GET 2023-09-22T21:08:22.372 DEBUG:boto:Path: /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.372 DEBUG:boto:Data: 2023-09-22T21:08:22.372 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.372 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.372 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.372 DEBUG:boto:Params: {} 2023-09-22T21:08:22.372 DEBUG:boto:Token: None 2023-09-22T21:08:22.372 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.372 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MiSMW9Ls4s0AXs5pS4N3E4ttSbU= 2023-09-22T21:08:22.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MiSMW9Ls4s0AXs5pS4N3E4ttSbU=', 'Content-Length': '0'} 2023-09-22T21:08:22.375 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('x-amz-version-id', 'EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ec76a2df8fa737ca-00650e0246-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.375 DEBUG:boto:path=/ffzdqu-13/dummy 2023-09-22T21:08:22.375 DEBUG:boto:auth_path=/ffzdqu-13/dummy 2023-09-22T21:08:22.375 DEBUG:boto:path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.375 DEBUG:boto:auth_path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.376 DEBUG:boto:Method: HEAD 2023-09-22T21:08:22.376 DEBUG:boto:Path: /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.376 DEBUG:boto:Data: 2023-09-22T21:08:22.376 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.376 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.376 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.376 DEBUG:boto:Params: {} 2023-09-22T21:08:22.376 DEBUG:boto:Token: None 2023-09-22T21:08:22.376 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.376 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TB9MmgmGbHh3zcUGwGxB1Q4c/ok= 2023-09-22T21:08:22.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TB9MmgmGbHh3zcUGwGxB1Q4c/ok=', 'Content-Length': '0'} 2023-09-22T21:08:22.379 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('x-amz-version-id', 'EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000bd18ec526a261b7-00650e0246-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.379 DEBUG:boto:path=/ffzdqu-13/dummy 2023-09-22T21:08:22.379 DEBUG:boto:auth_path=/ffzdqu-13/dummy 2023-09-22T21:08:22.379 DEBUG:boto:path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.380 DEBUG:boto:auth_path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.380 DEBUG:boto:Method: HEAD 2023-09-22T21:08:22.380 DEBUG:boto:Path: /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.380 DEBUG:boto:Data: 2023-09-22T21:08:22.380 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.380 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.380 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.380 DEBUG:boto:Params: {} 2023-09-22T21:08:22.380 DEBUG:boto:Token: None 2023-09-22T21:08:22.380 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.381 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TB9MmgmGbHh3zcUGwGxB1Q4c/ok= 2023-09-22T21:08:22.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TB9MmgmGbHh3zcUGwGxB1Q4c/ok=', 'Content-Length': '0'} 2023-09-22T21:08:22.383 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('x-amz-version-id', 'EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f3e9fbcb3d7fb7a7-00650e0246-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-22T21:08:22.383 DEBUG:boto:path=/ffzdqu-13/dummy 2023-09-22T21:08:22.383 DEBUG:boto:auth_path=/ffzdqu-13/dummy 2023-09-22T21:08:22.384 DEBUG:boto:path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.384 DEBUG:boto:auth_path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.384 DEBUG:boto:Method: GET 2023-09-22T21:08:22.384 DEBUG:boto:Path: /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.384 DEBUG:boto:Data: 2023-09-22T21:08:22.384 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.384 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.384 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.384 DEBUG:boto:Params: {} 2023-09-22T21:08:22.384 DEBUG:boto:Token: None 2023-09-22T21:08:22.384 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.385 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MiSMW9Ls4s0AXs5pS4N3E4ttSbU= 2023-09-22T21:08:22.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MiSMW9Ls4s0AXs5pS4N3E4ttSbU=', 'Content-Length': '0'} 2023-09-22T21:08:22.388 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('x-amz-version-id', 'EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fbee1d92abfa9a0b-00650e0246-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.388 DEBUG:boto:path=/ffzdqu-13/dummy 2023-09-22T21:08:22.388 DEBUG:boto:auth_path=/ffzdqu-13/dummy 2023-09-22T21:08:22.388 DEBUG:boto:path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.388 DEBUG:boto:auth_path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.388 DEBUG:boto:Method: GET 2023-09-22T21:08:22.388 DEBUG:boto:Path: /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.388 DEBUG:boto:Data: 2023-09-22T21:08:22.388 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.389 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.389 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.389 DEBUG:boto:Params: {} 2023-09-22T21:08:22.389 DEBUG:boto:Token: None 2023-09-22T21:08:22.389 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.389 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MiSMW9Ls4s0AXs5pS4N3E4ttSbU= 2023-09-22T21:08:22.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MiSMW9Ls4s0AXs5pS4N3E4ttSbU=', 'Content-Length': '0'} 2023-09-22T21:08:22.392 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('x-amz-version-id', 'EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ad4f0ef68d8535b6-00650e0246-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.392 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.392 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.392 DEBUG:boto:path=/ffzdqu-13/?prefix=dummy 2023-09-22T21:08:22.392 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=dummy 2023-09-22T21:08:22.392 DEBUG:boto:Method: GET 2023-09-22T21:08:22.392 DEBUG:boto:Path: /ffzdqu-13/?prefix=dummy 2023-09-22T21:08:22.393 DEBUG:boto:Data: 2023-09-22T21:08:22.393 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.393 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.393 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.393 DEBUG:boto:Params: {} 2023-09-22T21:08:22.393 DEBUG:boto:Token: None 2023-09-22T21:08:22.393 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/ 2023-09-22T21:08:22.393 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yW2dGHCGJFqj7B75XBbZFAI1dh4= 2023-09-22T21:08:22.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yW2dGHCGJFqj7B75XBbZFAI1dh4=', 'Content-Length': '0'} 2023-09-22T21:08:22.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000690613ed77f5ff1f-00650e0246-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.438 DEBUG:boto:b'ffzdqu-13dummy1000falsedummy2023-09-22T21:07:46.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:22.439 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.439 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.439 DEBUG:boto:path=/ffzdqu-13/?prefix=dummy 2023-09-22T21:08:22.439 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=dummy 2023-09-22T21:08:22.439 DEBUG:boto:Method: GET 2023-09-22T21:08:22.439 DEBUG:boto:Path: /ffzdqu-13/?prefix=dummy 2023-09-22T21:08:22.439 DEBUG:boto:Data: 2023-09-22T21:08:22.439 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.439 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.439 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.440 DEBUG:boto:Params: {} 2023-09-22T21:08:22.440 DEBUG:boto:Token: None 2023-09-22T21:08:22.440 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/ 2023-09-22T21:08:22.440 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yW2dGHCGJFqj7B75XBbZFAI1dh4= 2023-09-22T21:08:22.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yW2dGHCGJFqj7B75XBbZFAI1dh4=', 'Content-Length': '0'} 2023-09-22T21:08:22.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c54e7ebab38d6c48-00650e0246-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.445 DEBUG:boto:b'ffzdqu-13dummy1000falsedummy2023-09-22T21:07:46.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:22.445 DEBUG:boto:path=/ffzdqu-13/dummy 2023-09-22T21:08:22.445 DEBUG:boto:auth_path=/ffzdqu-13/dummy 2023-09-22T21:08:22.445 DEBUG:boto:Method: HEAD 2023-09-22T21:08:22.445 DEBUG:boto:Path: /ffzdqu-13/dummy 2023-09-22T21:08:22.445 DEBUG:boto:Data: 2023-09-22T21:08:22.445 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.445 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.446 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.446 DEBUG:boto:Params: {} 2023-09-22T21:08:22.446 DEBUG:boto:Token: None 2023-09-22T21:08:22.446 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/dummy 2023-09-22T21:08:22.446 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rfoJAMpGnHauullzIq6dhjddgKM= 2023-09-22T21:08:22.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rfoJAMpGnHauullzIq6dhjddgKM=', 'Content-Length': '0'} 2023-09-22T21:08:22.450 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('x-amz-version-id', 'EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006a4c7ef35f15022a-00650e0246-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.450 DEBUG:boto:path=/ffzdqu-13/dummy 2023-09-22T21:08:22.450 DEBUG:boto:auth_path=/ffzdqu-13/dummy 2023-09-22T21:08:22.450 DEBUG:boto:Method: HEAD 2023-09-22T21:08:22.450 DEBUG:boto:Path: /ffzdqu-13/dummy 2023-09-22T21:08:22.450 DEBUG:boto:Data: 2023-09-22T21:08:22.451 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.451 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.451 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.451 DEBUG:boto:Params: {} 2023-09-22T21:08:22.451 DEBUG:boto:Token: None 2023-09-22T21:08:22.451 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/dummy 2023-09-22T21:08:22.451 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rfoJAMpGnHauullzIq6dhjddgKM= 2023-09-22T21:08:22.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rfoJAMpGnHauullzIq6dhjddgKM=', 'Content-Length': '0'} 2023-09-22T21:08:22.455 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('x-amz-version-id', 'EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000031bc16719fdb479e-00650e0246-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-22T21:08:22.455 DEBUG:boto:path=/ffzdqu-13/dummy 2023-09-22T21:08:22.455 DEBUG:boto:auth_path=/ffzdqu-13/dummy 2023-09-22T21:08:22.455 DEBUG:boto:path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.455 DEBUG:boto:auth_path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.455 DEBUG:boto:Method: GET 2023-09-22T21:08:22.456 DEBUG:boto:Path: /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.456 DEBUG:boto:Data: 2023-09-22T21:08:22.456 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.456 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.456 DEBUG:boto:Params: {} 2023-09-22T21:08:22.456 DEBUG:boto:Token: None 2023-09-22T21:08:22.456 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.456 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MiSMW9Ls4s0AXs5pS4N3E4ttSbU= 2023-09-22T21:08:22.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MiSMW9Ls4s0AXs5pS4N3E4ttSbU=', 'Content-Length': '0'} 2023-09-22T21:08:22.459 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('x-amz-version-id', 'EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008eb4cc5c44d88483-00650e0246-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.459 DEBUG:boto:path=/ffzdqu-13/dummy 2023-09-22T21:08:22.459 DEBUG:boto:auth_path=/ffzdqu-13/dummy 2023-09-22T21:08:22.459 DEBUG:boto:path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.460 DEBUG:boto:auth_path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.460 DEBUG:boto:Method: GET 2023-09-22T21:08:22.460 DEBUG:boto:Path: /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.460 DEBUG:boto:Data: 2023-09-22T21:08:22.460 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.460 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.460 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.460 DEBUG:boto:Params: {} 2023-09-22T21:08:22.460 DEBUG:boto:Token: None 2023-09-22T21:08:22.460 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:22.461 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MiSMW9Ls4s0AXs5pS4N3E4ttSbU= 2023-09-22T21:08:22.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MiSMW9Ls4s0AXs5pS4N3E4ttSbU=', 'Content-Length': '0'} 2023-09-22T21:08:22.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('x-amz-version-id', 'EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005a43b8d73f2b17fa-00650e0246-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.463 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.464 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.464 DEBUG:boto:path=/ffzdqu-13/?prefix=dummy 2023-09-22T21:08:22.464 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=dummy 2023-09-22T21:08:22.464 DEBUG:boto:Method: GET 2023-09-22T21:08:22.464 DEBUG:boto:Path: /ffzdqu-13/?prefix=dummy 2023-09-22T21:08:22.464 DEBUG:boto:Data: 2023-09-22T21:08:22.464 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.464 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.464 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.464 DEBUG:boto:Params: {} 2023-09-22T21:08:22.465 DEBUG:boto:Token: None 2023-09-22T21:08:22.465 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/ 2023-09-22T21:08:22.465 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yW2dGHCGJFqj7B75XBbZFAI1dh4= 2023-09-22T21:08:22.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yW2dGHCGJFqj7B75XBbZFAI1dh4=', 'Content-Length': '0'} 2023-09-22T21:08:22.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079db3a02e810131a-00650e0246-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.470 DEBUG:boto:b'ffzdqu-13dummy1000falsedummy2023-09-22T21:07:46.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:22.470 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.471 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.471 DEBUG:boto:path=/ffzdqu-13/?prefix=dummy 2023-09-22T21:08:22.471 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=dummy 2023-09-22T21:08:22.471 DEBUG:boto:Method: GET 2023-09-22T21:08:22.471 DEBUG:boto:Path: /ffzdqu-13/?prefix=dummy 2023-09-22T21:08:22.471 DEBUG:boto:Data: 2023-09-22T21:08:22.471 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.471 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.471 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.471 DEBUG:boto:Params: {} 2023-09-22T21:08:22.472 DEBUG:boto:Token: None 2023-09-22T21:08:22.472 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/ 2023-09-22T21:08:22.472 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yW2dGHCGJFqj7B75XBbZFAI1dh4= 2023-09-22T21:08:22.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yW2dGHCGJFqj7B75XBbZFAI1dh4=', 'Content-Length': '0'} 2023-09-22T21:08:22.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba4a8498fb676b80-00650e0246-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.518 DEBUG:boto:b'ffzdqu-13dummy1000falsedummy2023-09-22T21:07:46.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:22.519 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-13 zones={a2, a1} 2023-09-22T21:08:22.519 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.519 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.519 DEBUG:boto:Method: HEAD 2023-09-22T21:08:22.519 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:22.519 DEBUG:boto:Data: 2023-09-22T21:08:22.519 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.519 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.519 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.519 DEBUG:boto:Params: {} 2023-09-22T21:08:22.520 DEBUG:boto:Token: None 2023-09-22T21:08:22.520 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/ 2023-09-22T21:08:22.520 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI= 2023-09-22T21:08:22.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI=', 'Content-Length': '0'} 2023-09-22T21:08:22.524 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000724820a1f18e124b-00650e0246-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.524 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:22.524 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:22.524 DEBUG:boto:Method: PUT 2023-09-22T21:08:22.525 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:22.525 DEBUG:boto:Data: 2023-09-22T21:08:22.525 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:08:22.525 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.525 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.525 DEBUG:boto:Params: {} 2023-09-22T21:08:22.525 DEBUG:boto:Token: None 2023-09-22T21:08:22.525 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/obj 2023-09-22T21:08:22.525 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y= 2023-09-22T21:08:22.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y='} 2023-09-22T21:08:22.552 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'hn7HL8wXg0x8WecE.JHMlcrta8XT0VL'), ('x-amz-request-id', 'tx00000d95e88253501facb-00650e0246-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.552 DEBUG:rgw_multi.tests:zone=a1 version=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:22.552 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.552 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.552 DEBUG:boto:Method: HEAD 2023-09-22T21:08:22.552 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:22.553 DEBUG:boto:Data: 2023-09-22T21:08:22.553 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.553 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.553 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.553 DEBUG:boto:Params: {} 2023-09-22T21:08:22.553 DEBUG:boto:Token: None 2023-09-22T21:08:22.553 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/ 2023-09-22T21:08:22.553 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI= 2023-09-22T21:08:22.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI=', 'Content-Length': '0'} 2023-09-22T21:08:22.557 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2f97387e23dc8ad-00650e0246-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.558 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:22.558 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:22.558 DEBUG:boto:Method: PUT 2023-09-22T21:08:22.558 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:22.558 DEBUG:boto:Data: 2023-09-22T21:08:22.558 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:08:22.558 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.558 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.558 DEBUG:boto:Params: {} 2023-09-22T21:08:22.559 DEBUG:boto:Token: None 2023-09-22T21:08:22.559 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/obj 2023-09-22T21:08:22.559 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y= 2023-09-22T21:08:22.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y='} 2023-09-22T21:08:22.602 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM'), ('x-amz-request-id', 'tx0000013d2dc0121887145-00650e0246-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.603 DEBUG:rgw_multi.tests:zone=a2 version=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:22.603 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.603 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.603 DEBUG:boto:Method: HEAD 2023-09-22T21:08:22.603 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:22.603 DEBUG:boto:Data: 2023-09-22T21:08:22.603 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.603 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.603 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.603 DEBUG:boto:Params: {} 2023-09-22T21:08:22.604 DEBUG:boto:Token: None 2023-09-22T21:08:22.604 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/ 2023-09-22T21:08:22.604 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI= 2023-09-22T21:08:22.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI=', 'Content-Length': '0'} 2023-09-22T21:08:22.608 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d632dc09f1717b6-00650e0246-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.608 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:22.608 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:22.608 DEBUG:boto:Method: PUT 2023-09-22T21:08:22.608 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:22.609 DEBUG:boto:Data: 2023-09-22T21:08:22.609 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:08:22.609 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.609 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.609 DEBUG:boto:Params: {} 2023-09-22T21:08:22.609 DEBUG:boto:Token: None 2023-09-22T21:08:22.609 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/obj 2023-09-22T21:08:22.609 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y= 2023-09-22T21:08:22.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y='} 2023-09-22T21:08:22.654 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc'), ('x-amz-request-id', 'tx00000b27fe3aa8a7ad311-00650e0246-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.655 DEBUG:rgw_multi.tests:zone=a1 version=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:22.655 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.655 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.655 DEBUG:boto:Method: HEAD 2023-09-22T21:08:22.655 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:22.655 DEBUG:boto:Data: 2023-09-22T21:08:22.655 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.655 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.655 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.655 DEBUG:boto:Params: {} 2023-09-22T21:08:22.656 DEBUG:boto:Token: None 2023-09-22T21:08:22.656 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/ 2023-09-22T21:08:22.656 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI= 2023-09-22T21:08:22.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI=', 'Content-Length': '0'} 2023-09-22T21:08:22.660 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008615101809554403-00650e0246-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.660 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:22.660 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:22.660 DEBUG:boto:Method: PUT 2023-09-22T21:08:22.660 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:22.660 DEBUG:boto:Data: 2023-09-22T21:08:22.661 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:08:22.661 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.661 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.661 DEBUG:boto:Params: {} 2023-09-22T21:08:22.661 DEBUG:boto:Token: None 2023-09-22T21:08:22.661 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/obj 2023-09-22T21:08:22.661 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y= 2023-09-22T21:08:22.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y='} 2023-09-22T21:08:22.685 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '2t60n7HyrucItuaaoFxjoHWIjdIOj1g'), ('x-amz-request-id', 'tx000009aadff0fbfd354d4-00650e0246-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.685 DEBUG:rgw_multi.tests:zone=a2 version=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:22.686 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.686 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.686 DEBUG:boto:Method: HEAD 2023-09-22T21:08:22.686 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:22.686 DEBUG:boto:Data: 2023-09-22T21:08:22.686 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.686 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.686 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.686 DEBUG:boto:Params: {} 2023-09-22T21:08:22.687 DEBUG:boto:Token: None 2023-09-22T21:08:22.687 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/ 2023-09-22T21:08:22.687 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI= 2023-09-22T21:08:22.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI=', 'Content-Length': '0'} 2023-09-22T21:08:22.691 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000877ccec507e4ea76-00650e0246-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.691 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:22.691 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:22.691 DEBUG:boto:Method: PUT 2023-09-22T21:08:22.691 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:22.692 DEBUG:boto:Data: 2023-09-22T21:08:22.692 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:08:22.692 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.692 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.692 DEBUG:boto:Params: {} 2023-09-22T21:08:22.692 DEBUG:boto:Token: None 2023-09-22T21:08:22.692 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/obj 2023-09-22T21:08:22.692 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y= 2023-09-22T21:08:22.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y='} 2023-09-22T21:08:22.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd'), ('x-amz-request-id', 'tx00000dfa8e910961a87d2-00650e0246-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.715 DEBUG:rgw_multi.tests:zone=a1 version=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:22.715 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.715 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.715 DEBUG:boto:Method: HEAD 2023-09-22T21:08:22.715 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:22.715 DEBUG:boto:Data: 2023-09-22T21:08:22.715 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.715 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.715 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.715 DEBUG:boto:Params: {} 2023-09-22T21:08:22.716 DEBUG:boto:Token: None 2023-09-22T21:08:22.716 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/ 2023-09-22T21:08:22.716 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI= 2023-09-22T21:08:22.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI=', 'Content-Length': '0'} 2023-09-22T21:08:22.720 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000447ae800b0409e7b-00650e0246-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.720 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:22.720 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:22.720 DEBUG:boto:Method: PUT 2023-09-22T21:08:22.720 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:22.721 DEBUG:boto:Data: 2023-09-22T21:08:22.721 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:08:22.721 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.721 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.721 DEBUG:boto:Params: {} 2023-09-22T21:08:22.721 DEBUG:boto:Token: None 2023-09-22T21:08:22.721 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/obj 2023-09-22T21:08:22.721 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y= 2023-09-22T21:08:22.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y='} 2023-09-22T21:08:22.770 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Ux-o.jzhWCEq59rWr2ALKdruobR.tsY'), ('x-amz-request-id', 'tx000008369e22b2b59b3c4-00650e0246-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.771 DEBUG:rgw_multi.tests:zone=a2 version=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:22.771 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.771 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.771 DEBUG:boto:Method: HEAD 2023-09-22T21:08:22.771 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:22.771 DEBUG:boto:Data: 2023-09-22T21:08:22.771 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.771 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.771 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.772 DEBUG:boto:Params: {} 2023-09-22T21:08:22.772 DEBUG:boto:Token: None 2023-09-22T21:08:22.772 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/ 2023-09-22T21:08:22.772 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI= 2023-09-22T21:08:22.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI=', 'Content-Length': '0'} 2023-09-22T21:08:22.777 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c9b9b508575c7d2-00650e0246-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.777 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:22.777 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:22.777 DEBUG:boto:Method: PUT 2023-09-22T21:08:22.777 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:22.777 DEBUG:boto:Data: 2023-09-22T21:08:22.777 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:08:22.778 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.778 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.778 DEBUG:boto:Params: {} 2023-09-22T21:08:22.778 DEBUG:boto:Token: None 2023-09-22T21:08:22.778 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/obj 2023-09-22T21:08:22.778 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y= 2023-09-22T21:08:22.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y='} 2023-09-22T21:08:22.822 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'MWb3f8.clEGzrytonhqa4UHE-3nWzBc'), ('x-amz-request-id', 'tx00000cb95e23b0c1af75e-00650e0246-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.823 DEBUG:rgw_multi.tests:zone=a1 version=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:22.823 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.823 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.823 DEBUG:boto:Method: HEAD 2023-09-22T21:08:22.823 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:22.823 DEBUG:boto:Data: 2023-09-22T21:08:22.823 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.823 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.823 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.823 DEBUG:boto:Params: {} 2023-09-22T21:08:22.824 DEBUG:boto:Token: None 2023-09-22T21:08:22.824 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/ 2023-09-22T21:08:22.824 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI= 2023-09-22T21:08:22.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI=', 'Content-Length': '0'} 2023-09-22T21:08:22.828 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000209efbc648cbbdfd-00650e0246-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.828 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:22.828 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:22.828 DEBUG:boto:Method: PUT 2023-09-22T21:08:22.828 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:22.828 DEBUG:boto:Data: 2023-09-22T21:08:22.828 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:08:22.828 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.829 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.829 DEBUG:boto:Params: {} 2023-09-22T21:08:22.829 DEBUG:boto:Token: None 2023-09-22T21:08:22.829 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/obj 2023-09-22T21:08:22.829 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y= 2023-09-22T21:08:22.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y='} 2023-09-22T21:08:22.852 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3'), ('x-amz-request-id', 'tx00000910ae4c7003a56d7-00650e0246-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.852 DEBUG:rgw_multi.tests:zone=a2 version=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:22.852 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.853 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.853 DEBUG:boto:Method: HEAD 2023-09-22T21:08:22.853 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:22.853 DEBUG:boto:Data: 2023-09-22T21:08:22.853 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.853 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.853 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.853 DEBUG:boto:Params: {} 2023-09-22T21:08:22.853 DEBUG:boto:Token: None 2023-09-22T21:08:22.853 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/ 2023-09-22T21:08:22.854 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI= 2023-09-22T21:08:22.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI=', 'Content-Length': '0'} 2023-09-22T21:08:22.858 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8dd8b19ce3b4e54-00650e0246-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.858 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:22.858 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:22.858 DEBUG:boto:Method: PUT 2023-09-22T21:08:22.858 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:22.858 DEBUG:boto:Data: 2023-09-22T21:08:22.858 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:08:22.859 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.859 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.859 DEBUG:boto:Params: {} 2023-09-22T21:08:22.859 DEBUG:boto:Token: None 2023-09-22T21:08:22.859 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/obj 2023-09-22T21:08:22.859 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y= 2023-09-22T21:08:22.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y='} 2023-09-22T21:08:22.882 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'X5s9z0mNXbqm37pFm1vpfPexuZJVGd0'), ('x-amz-request-id', 'tx000009978406d65bd111d-00650e0246-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.882 DEBUG:rgw_multi.tests:zone=a1 version=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:22.882 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.883 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.883 DEBUG:boto:Method: HEAD 2023-09-22T21:08:22.883 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:22.883 DEBUG:boto:Data: 2023-09-22T21:08:22.883 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.883 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.883 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.883 DEBUG:boto:Params: {} 2023-09-22T21:08:22.883 DEBUG:boto:Token: None 2023-09-22T21:08:22.883 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/ 2023-09-22T21:08:22.884 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI= 2023-09-22T21:08:22.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI=', 'Content-Length': '0'} 2023-09-22T21:08:22.887 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f2ae4c3f578db76-00650e0246-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.888 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:22.888 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:22.888 DEBUG:boto:Method: PUT 2023-09-22T21:08:22.888 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:22.888 DEBUG:boto:Data: 2023-09-22T21:08:22.888 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:08:22.888 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.888 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.889 DEBUG:boto:Params: {} 2023-09-22T21:08:22.889 DEBUG:boto:Token: None 2023-09-22T21:08:22.889 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/obj 2023-09-22T21:08:22.889 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y= 2023-09-22T21:08:22.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y='} 2023-09-22T21:08:22.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ZmK2SagIILGhSA.9md7mKoG8ffxTZzd'), ('x-amz-request-id', 'tx0000004ee4897dc06c8ff-00650e0246-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.935 DEBUG:rgw_multi.tests:zone=a2 version=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:22.935 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.935 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.935 DEBUG:boto:Method: HEAD 2023-09-22T21:08:22.935 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:22.935 DEBUG:boto:Data: 2023-09-22T21:08:22.935 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.935 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.935 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.935 DEBUG:boto:Params: {} 2023-09-22T21:08:22.936 DEBUG:boto:Token: None 2023-09-22T21:08:22.936 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/ 2023-09-22T21:08:22.936 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI= 2023-09-22T21:08:22.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI=', 'Content-Length': '0'} 2023-09-22T21:08:22.940 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a63116eea8ef6419-00650e0246-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.940 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:22.940 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:22.940 DEBUG:boto:Method: PUT 2023-09-22T21:08:22.940 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:22.940 DEBUG:boto:Data: 2023-09-22T21:08:22.940 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:08:22.940 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:22.941 DEBUG:boto:Port: 8000 2023-09-22T21:08:22.941 DEBUG:boto:Params: {} 2023-09-22T21:08:22.941 DEBUG:boto:Token: None 2023-09-22T21:08:22.941 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/obj 2023-09-22T21:08:22.941 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y= 2023-09-22T21:08:22.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y='} 2023-09-22T21:08:22.986 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2'), ('x-amz-request-id', 'tx0000059b194fd4c942c60-00650e0246-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.987 DEBUG:rgw_multi.tests:zone=a1 version=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:22.987 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:22.987 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:22.987 DEBUG:boto:Method: HEAD 2023-09-22T21:08:22.987 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:22.987 DEBUG:boto:Data: 2023-09-22T21:08:22.987 DEBUG:boto:Headers: {} 2023-09-22T21:08:22.987 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.987 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.987 DEBUG:boto:Params: {} 2023-09-22T21:08:22.988 DEBUG:boto:Token: None 2023-09-22T21:08:22.988 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/ 2023-09-22T21:08:22.988 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI= 2023-09-22T21:08:22.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gJR68gPRimOorez3WyOWHj+qFTI=', 'Content-Length': '0'} 2023-09-22T21:08:22.992 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7e5e1406aba7bad-00650e0246-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:22.992 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:22.993 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:22.993 DEBUG:boto:Method: PUT 2023-09-22T21:08:22.993 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:22.993 DEBUG:boto:Data: 2023-09-22T21:08:22.993 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:08:22.993 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:22.993 DEBUG:boto:Port: 8001 2023-09-22T21:08:22.993 DEBUG:boto:Params: {} 2023-09-22T21:08:22.993 DEBUG:boto:Token: None 2023-09-22T21:08:22.993 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:08:22 GMT /ffzdqu-13/obj 2023-09-22T21:08:22.994 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y= 2023-09-22T21:08:22.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:08:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xxKTR+hwBjBxmqTn1pScPJmHP6Y='} 2023-09-22T21:08:23.016 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp'), ('x-amz-request-id', 'tx0000022cc5f7223eb93f1-00650e0246-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:23.016 DEBUG:rgw_multi.tests:zone=a2 version=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:23.016 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:23.016 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:23.016 DEBUG:boto:Method: HEAD 2023-09-22T21:08:23.017 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:23.017 DEBUG:boto:Data: 2023-09-22T21:08:23.017 DEBUG:boto:Headers: {} 2023-09-22T21:08:23.017 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:23.017 DEBUG:boto:Port: 8000 2023-09-22T21:08:23.017 DEBUG:boto:Params: {} 2023-09-22T21:08:23.017 DEBUG:boto:Token: None 2023-09-22T21:08:23.017 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:23 GMT /ffzdqu-13/ 2023-09-22T21:08:23.017 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CepcKHaQne6XPUcUtv4Q5Whnf1s= 2023-09-22T21:08:23.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CepcKHaQne6XPUcUtv4Q5Whnf1s=', 'Content-Length': '0'} 2023-09-22T21:08:23.022 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e0a56891945d11f-00650e0247-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:23.022 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:23.022 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:23.022 DEBUG:boto:Method: PUT 2023-09-22T21:08:23.022 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:23.022 DEBUG:boto:Data: 2023-09-22T21:08:23.022 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:08:23.022 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:23.023 DEBUG:boto:Port: 8000 2023-09-22T21:08:23.023 DEBUG:boto:Params: {} 2023-09-22T21:08:23.023 DEBUG:boto:Token: None 2023-09-22T21:08:23.023 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:08:23 GMT /ffzdqu-13/obj 2023-09-22T21:08:23.023 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DD9iksGUV8nMUw7fQeuAYEqOrMo= 2023-09-22T21:08:23.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:08:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DD9iksGUV8nMUw7fQeuAYEqOrMo='} 2023-09-22T21:08:23.045 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'nkf1H6kXofD14zadwSPLBP-vHM76qTM'), ('x-amz-request-id', 'tx000004c29bb075ec63b63-00650e0247-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:23.045 DEBUG:rgw_multi.tests:zone=a1 version=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:23.046 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:23.046 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:23.046 DEBUG:boto:Method: HEAD 2023-09-22T21:08:23.046 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:23.046 DEBUG:boto:Data: 2023-09-22T21:08:23.046 DEBUG:boto:Headers: {} 2023-09-22T21:08:23.046 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:23.046 DEBUG:boto:Port: 8001 2023-09-22T21:08:23.046 DEBUG:boto:Params: {} 2023-09-22T21:08:23.046 DEBUG:boto:Token: None 2023-09-22T21:08:23.047 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:23 GMT /ffzdqu-13/ 2023-09-22T21:08:23.047 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CepcKHaQne6XPUcUtv4Q5Whnf1s= 2023-09-22T21:08:23.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CepcKHaQne6XPUcUtv4Q5Whnf1s=', 'Content-Length': '0'} 2023-09-22T21:08:23.051 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000595c38a885a7fc76-00650e0247-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:23.051 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:23.051 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:23.051 DEBUG:boto:Method: PUT 2023-09-22T21:08:23.052 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:23.052 DEBUG:boto:Data: 2023-09-22T21:08:23.052 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:08:23.052 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:23.052 DEBUG:boto:Port: 8001 2023-09-22T21:08:23.052 DEBUG:boto:Params: {} 2023-09-22T21:08:23.052 DEBUG:boto:Token: None 2023-09-22T21:08:23.052 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:08:23 GMT /ffzdqu-13/obj 2023-09-22T21:08:23.052 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DD9iksGUV8nMUw7fQeuAYEqOrMo= 2023-09-22T21:08:23.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:08:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DD9iksGUV8nMUw7fQeuAYEqOrMo='} 2023-09-22T21:08:23.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'KeL89-xXRu815nqON693z3wYbeHRSpw'), ('x-amz-request-id', 'tx00000245449e2a22ceb88-00650e0247-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:23.095 DEBUG:rgw_multi.tests:zone=a2 version=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:23.095 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:23.095 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:23.095 DEBUG:boto:Method: HEAD 2023-09-22T21:08:23.095 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:23.095 DEBUG:boto:Data: 2023-09-22T21:08:23.095 DEBUG:boto:Headers: {} 2023-09-22T21:08:23.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:23.095 DEBUG:boto:Port: 8000 2023-09-22T21:08:23.095 DEBUG:boto:Params: {} 2023-09-22T21:08:23.096 DEBUG:boto:Token: None 2023-09-22T21:08:23.096 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:23 GMT /ffzdqu-13/ 2023-09-22T21:08:23.096 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CepcKHaQne6XPUcUtv4Q5Whnf1s= 2023-09-22T21:08:23.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CepcKHaQne6XPUcUtv4Q5Whnf1s=', 'Content-Length': '0'} 2023-09-22T21:08:23.100 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0700efbd813119b-00650e0247-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:23.100 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:23.100 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:23.100 DEBUG:boto:Method: PUT 2023-09-22T21:08:23.100 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:23.101 DEBUG:boto:Data: 2023-09-22T21:08:23.101 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:08:23.101 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:23.101 DEBUG:boto:Port: 8000 2023-09-22T21:08:23.101 DEBUG:boto:Params: {} 2023-09-22T21:08:23.101 DEBUG:boto:Token: None 2023-09-22T21:08:23.101 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:08:23 GMT /ffzdqu-13/obj 2023-09-22T21:08:23.101 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DD9iksGUV8nMUw7fQeuAYEqOrMo= 2023-09-22T21:08:23.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:08:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DD9iksGUV8nMUw7fQeuAYEqOrMo='} 2023-09-22T21:08:23.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY'), ('x-amz-request-id', 'tx0000098c23a6f8c0e5af6-00650e0247-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:23.147 DEBUG:rgw_multi.tests:zone=a1 version=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:23.147 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:23.147 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:23.147 DEBUG:boto:Method: HEAD 2023-09-22T21:08:23.147 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:23.147 DEBUG:boto:Data: 2023-09-22T21:08:23.147 DEBUG:boto:Headers: {} 2023-09-22T21:08:23.147 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:23.147 DEBUG:boto:Port: 8001 2023-09-22T21:08:23.147 DEBUG:boto:Params: {} 2023-09-22T21:08:23.148 DEBUG:boto:Token: None 2023-09-22T21:08:23.148 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:23 GMT /ffzdqu-13/ 2023-09-22T21:08:23.148 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CepcKHaQne6XPUcUtv4Q5Whnf1s= 2023-09-22T21:08:23.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CepcKHaQne6XPUcUtv4Q5Whnf1s=', 'Content-Length': '0'} 2023-09-22T21:08:23.151 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036fe34217a28f1e3-00650e0247-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:23.152 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:23.152 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:23.152 DEBUG:boto:Method: PUT 2023-09-22T21:08:23.152 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:23.152 DEBUG:boto:Data: 2023-09-22T21:08:23.152 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:08:23.152 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:23.152 DEBUG:boto:Port: 8001 2023-09-22T21:08:23.152 DEBUG:boto:Params: {} 2023-09-22T21:08:23.153 DEBUG:boto:Token: None 2023-09-22T21:08:23.153 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:08:23 GMT /ffzdqu-13/obj 2023-09-22T21:08:23.153 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DD9iksGUV8nMUw7fQeuAYEqOrMo= 2023-09-22T21:08:23.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:08:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DD9iksGUV8nMUw7fQeuAYEqOrMo='} 2023-09-22T21:08:23.175 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD'), ('x-amz-request-id', 'tx000008e35ac9f92dbe204-00650e0247-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:23.175 DEBUG:rgw_multi.tests:zone=a2 version=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:23.175 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:23.176 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:23.176 DEBUG:boto:Method: HEAD 2023-09-22T21:08:23.176 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:23.176 DEBUG:boto:Data: 2023-09-22T21:08:23.176 DEBUG:boto:Headers: {} 2023-09-22T21:08:23.176 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:23.176 DEBUG:boto:Port: 8000 2023-09-22T21:08:23.176 DEBUG:boto:Params: {} 2023-09-22T21:08:23.176 DEBUG:boto:Token: None 2023-09-22T21:08:23.176 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:23 GMT /ffzdqu-13/ 2023-09-22T21:08:23.177 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CepcKHaQne6XPUcUtv4Q5Whnf1s= 2023-09-22T21:08:23.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CepcKHaQne6XPUcUtv4Q5Whnf1s=', 'Content-Length': '0'} 2023-09-22T21:08:23.181 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007593f6ef3423b094-00650e0247-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:23.181 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:23.181 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:23.181 DEBUG:boto:Method: PUT 2023-09-22T21:08:23.181 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:23.181 DEBUG:boto:Data: 2023-09-22T21:08:23.181 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:08:23.182 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:23.182 DEBUG:boto:Port: 8000 2023-09-22T21:08:23.182 DEBUG:boto:Params: {} 2023-09-22T21:08:23.182 DEBUG:boto:Token: None 2023-09-22T21:08:23.182 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:08:23 GMT /ffzdqu-13/obj 2023-09-22T21:08:23.182 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DD9iksGUV8nMUw7fQeuAYEqOrMo= 2023-09-22T21:08:23.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:08:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DD9iksGUV8nMUw7fQeuAYEqOrMo='} 2023-09-22T21:08:23.205 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H'), ('x-amz-request-id', 'tx000006bc8266713838592-00650e0247-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:23.205 DEBUG:rgw_multi.tests:zone=a1 version=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:23.205 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:23.206 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:23.206 DEBUG:boto:Method: HEAD 2023-09-22T21:08:23.206 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:23.206 DEBUG:boto:Data: 2023-09-22T21:08:23.206 DEBUG:boto:Headers: {} 2023-09-22T21:08:23.206 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:23.206 DEBUG:boto:Port: 8001 2023-09-22T21:08:23.206 DEBUG:boto:Params: {} 2023-09-22T21:08:23.206 DEBUG:boto:Token: None 2023-09-22T21:08:23.207 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:23 GMT /ffzdqu-13/ 2023-09-22T21:08:23.207 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CepcKHaQne6XPUcUtv4Q5Whnf1s= 2023-09-22T21:08:23.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CepcKHaQne6XPUcUtv4Q5Whnf1s=', 'Content-Length': '0'} 2023-09-22T21:08:23.211 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a43cd7bf1bffcfdd-00650e0247-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:23.211 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:23.211 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:23.211 DEBUG:boto:Method: PUT 2023-09-22T21:08:23.211 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:23.212 DEBUG:boto:Data: 2023-09-22T21:08:23.212 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:08:23.212 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:23.212 DEBUG:boto:Port: 8001 2023-09-22T21:08:23.212 DEBUG:boto:Params: {} 2023-09-22T21:08:23.212 DEBUG:boto:Token: None 2023-09-22T21:08:23.212 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:08:23 GMT /ffzdqu-13/obj 2023-09-22T21:08:23.212 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DD9iksGUV8nMUw7fQeuAYEqOrMo= 2023-09-22T21:08:23.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:08:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DD9iksGUV8nMUw7fQeuAYEqOrMo='} 2023-09-22T21:08:23.254 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Lk20u5mUDXrc4IGPE.LofVWMUByRmdS'), ('x-amz-request-id', 'tx0000052b37abcbfb18d0f-00650e0247-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:23.255 DEBUG:rgw_multi.tests:zone=a2 version=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:23.255 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:23.255 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:23.255 DEBUG:boto:Method: HEAD 2023-09-22T21:08:23.255 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:23.255 DEBUG:boto:Data: 2023-09-22T21:08:23.255 DEBUG:boto:Headers: {} 2023-09-22T21:08:23.255 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:23.255 DEBUG:boto:Port: 8000 2023-09-22T21:08:23.256 DEBUG:boto:Params: {} 2023-09-22T21:08:23.256 DEBUG:boto:Token: None 2023-09-22T21:08:23.256 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:23 GMT /ffzdqu-13/ 2023-09-22T21:08:23.256 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CepcKHaQne6XPUcUtv4Q5Whnf1s= 2023-09-22T21:08:23.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CepcKHaQne6XPUcUtv4Q5Whnf1s=', 'Content-Length': '0'} 2023-09-22T21:08:23.260 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e75df367f4ba5385-00650e0247-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:23.260 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:23.260 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:23.261 DEBUG:boto:Method: PUT 2023-09-22T21:08:23.261 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:23.261 DEBUG:boto:Data: 2023-09-22T21:08:23.261 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:08:23.261 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:23.261 DEBUG:boto:Port: 8000 2023-09-22T21:08:23.261 DEBUG:boto:Params: {} 2023-09-22T21:08:23.261 DEBUG:boto:Token: None 2023-09-22T21:08:23.261 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:08:23 GMT /ffzdqu-13/obj 2023-09-22T21:08:23.261 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DD9iksGUV8nMUw7fQeuAYEqOrMo= 2023-09-22T21:08:23.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:08:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DD9iksGUV8nMUw7fQeuAYEqOrMo='} 2023-09-22T21:08:23.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N'), ('x-amz-request-id', 'tx0000007640f4656bd8576-00650e0247-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:23.311 DEBUG:rgw_multi.tests:zone=a1 version=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:23.311 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:23.311 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:23.311 DEBUG:boto:Method: HEAD 2023-09-22T21:08:23.311 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:23.311 DEBUG:boto:Data: 2023-09-22T21:08:23.311 DEBUG:boto:Headers: {} 2023-09-22T21:08:23.311 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:23.311 DEBUG:boto:Port: 8001 2023-09-22T21:08:23.311 DEBUG:boto:Params: {} 2023-09-22T21:08:23.311 DEBUG:boto:Token: None 2023-09-22T21:08:23.312 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:23 GMT /ffzdqu-13/ 2023-09-22T21:08:23.312 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CepcKHaQne6XPUcUtv4Q5Whnf1s= 2023-09-22T21:08:23.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CepcKHaQne6XPUcUtv4Q5Whnf1s=', 'Content-Length': '0'} 2023-09-22T21:08:23.315 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a07bfe9c57241ee5-00650e0247-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:23.316 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:23.316 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:23.316 DEBUG:boto:Method: PUT 2023-09-22T21:08:23.316 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:23.316 DEBUG:boto:Data: 2023-09-22T21:08:23.316 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-22T21:08:23.316 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:23.316 DEBUG:boto:Port: 8001 2023-09-22T21:08:23.317 DEBUG:boto:Params: {} 2023-09-22T21:08:23.317 DEBUG:boto:Token: None 2023-09-22T21:08:23.317 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 22 Sep 2023 21:08:23 GMT /ffzdqu-13/obj 2023-09-22T21:08:23.317 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DD9iksGUV8nMUw7fQeuAYEqOrMo= 2023-09-22T21:08:23.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:08:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DD9iksGUV8nMUw7fQeuAYEqOrMo='} 2023-09-22T21:08:23.339 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-amz-request-id', 'tx00000ba7585f209865a66-00650e0247-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:23.340 DEBUG:rgw_multi.tests:zone=a2 version=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:23.340 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-13 2023-09-22T21:08:23.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:08:23.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:08:23.340 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:08:23.389 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:08:23.389 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:08:23.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:08:23.435+0000 7f185b5b2a40 1 waiting for incremental sync to catch up: 2023-09-22T21:08:23.433 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.11.11, , , , , ] 2023-09-22T21:08:23.433 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000003.13.10, , , 00000000020.40.12, , ] 2023-09-22T21:08:28.435 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:08:28.435+0000 7f185b5b2a40 1 waiting for incremental sync to catch up: 2023-09-22T21:08:28.435 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.11.11, , , , , ] 2023-09-22T21:08:28.435 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000003.13.10, , , 00000000020.40.12, , ] 2023-09-22T21:08:33.437 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:08:33.439+0000 7f185b5b2a40 1 waiting for incremental sync to catch up: 2023-09-22T21:08:33.438 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.11.11, , , , , ] 2023-09-22T21:08:33.438 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000003.13.10, , , 00000000020.40.12, , ] 2023-09-22T21:08:38.440 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:08:38.439+0000 7f185b5b2a40 1 waiting for incremental sync to catch up: 2023-09-22T21:08:38.440 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.11.11, , , , , ] 2023-09-22T21:08:38.440 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000003.13.10, , , 00000000020.40.12, , ] 2023-09-22T21:08:43.442 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:08:43.443+0000 7f185b5b2a40 1 bucket sync caught up with source: 2023-09-22T21:08:43.442 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000003.13.10, , , 00000000020.40.12, , ] 2023-09-22T21:08:43.442 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000003.13.10, , , 00000000020.40.12, , ] 2023-09-22T21:08:43.443 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:08:43.443+0000 7f185b5b2a40 0 bucket checkpoint complete 2023-09-22T21:08:43.472 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-13 2023-09-22T21:08:43.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:08:43.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:08:43.473 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:08:43.523 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:08:43.523 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:08:43.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:08:43.567+0000 7fa4f864ea40 1 bucket sync caught up with source: 2023-09-22T21:08:43.567 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.27.11, , , 00000000031.72.9, , ] 2023-09-22T21:08:43.567 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000002.27.11, , , 00000000031.72.9, , ] 2023-09-22T21:08:43.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:08:43.567+0000 7fa4f864ea40 0 bucket checkpoint complete 2023-09-22T21:08:43.583 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-13 zones={a2, a1} 2023-09-22T21:08:43.583 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:43.583 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:43.583 DEBUG:boto:Method: HEAD 2023-09-22T21:08:43.583 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:43.584 DEBUG:boto:Data: 2023-09-22T21:08:43.584 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.584 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:43.584 DEBUG:boto:Port: 8001 2023-09-22T21:08:43.584 DEBUG:boto:Params: {} 2023-09-22T21:08:43.584 DEBUG:boto:Token: None 2023-09-22T21:08:43.584 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/ 2023-09-22T21:08:43.584 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FzRTqP56LENdGgnEqVAzpwWZ3qw= 2023-09-22T21:08:43.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FzRTqP56LENdGgnEqVAzpwWZ3qw=', 'Content-Length': '0'} 2023-09-22T21:08:43.587 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2c5c719c211549d-00650e025b-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.587 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:43.587 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:43.587 DEBUG:boto:Method: HEAD 2023-09-22T21:08:43.587 DEBUG:boto:Path: /ffzdqu-13/ 2023-09-22T21:08:43.588 DEBUG:boto:Data: 2023-09-22T21:08:43.588 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.588 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:43.588 DEBUG:boto:Port: 8000 2023-09-22T21:08:43.588 DEBUG:boto:Params: {} 2023-09-22T21:08:43.588 DEBUG:boto:Token: None 2023-09-22T21:08:43.588 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/ 2023-09-22T21:08:43.588 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FzRTqP56LENdGgnEqVAzpwWZ3qw= 2023-09-22T21:08:43.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FzRTqP56LENdGgnEqVAzpwWZ3qw=', 'Content-Length': '0'} 2023-09-22T21:08:43.591 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094e232f5c979015e-00650e025b-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.591 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:08:43.591 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:43.591 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:43.591 DEBUG:boto:path=/ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:43.591 DEBUG:boto:auth_path=/ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:43.592 DEBUG:boto:Method: GET 2023-09-22T21:08:43.592 DEBUG:boto:Path: /ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:43.592 DEBUG:boto:Data: 2023-09-22T21:08:43.592 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.592 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:43.592 DEBUG:boto:Port: 8001 2023-09-22T21:08:43.592 DEBUG:boto:Params: {} 2023-09-22T21:08:43.592 DEBUG:boto:Token: None 2023-09-22T21:08:43.592 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/?versions 2023-09-22T21:08:43.593 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SWoRHcFRDr9SynP+8iFnlEZtZUk= 2023-09-22T21:08:43.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SWoRHcFRDr9SynP+8iFnlEZtZUk=', 'Content-Length': '0'} 2023-09-22T21:08:43.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019e64b36d50f3f1d-00650e025b-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.596 DEBUG:boto:b'ffzdqu-13999falsedummyEH3tK3P9S8trmvB9h-eP4MX-b1Uhu-Ltrue2023-09-22T21:07:46.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVPOyc5MFRGSz2wBgBorKv2oh8K4FSnytrue2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjwzrpoM6PF-asZD0RLHOKLOYe9ma4z8Nfalse2023-09-22T21:08:23.268Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLk20u5mUDXrc4IGPE.LofVWMUByRmdSfalse2023-09-22T21:08:23.219Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvqVKHaLLoJZurTgAvh6O-aPYcyOZI.Hfalse2023-09-22T21:08:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZHOtFSYtYeqSDp3B48tnX9aoyhE-LVYfalse2023-09-22T21:08:23.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtrwlNr-EtLMj-wIVlmYw9dBWQcMpEiDfalse2023-09-22T21:08:23.159Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKeL89-xXRu815nqON693z3wYbeHRSpwfalse2023-09-22T21:08:23.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnkf1H6kXofD14zadwSPLBP-vHM76qTMfalse2023-09-22T21:08:23.030Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2false2023-09-22T21:08:22.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6-gODrZ6LdnRGbTNPrF.c3yj-YHJffpfalse2023-09-22T21:08:23.000Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjX5s9z0mNXbqm37pFm1vpfPexuZJVGd0false2023-09-22T21:08:22.866Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZmK2SagIILGhSA.9md7mKoG8ffxTZzdfalse2023-09-22T21:08:22.895Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMWb3f8.clEGzrytonhqa4UHE-3nWzBcfalse2023-09-22T21:08:22.785Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3false2023-09-22T21:08:22.836Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUx-o.jzhWCEq59rWr2ALKdruobR.tsYfalse2023-09-22T21:08:22.728Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZFg17GpJzZzJQ2hycn0YSoQcRmEUKXdfalse2023-09-22T21:08:22.699Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2t60n7HyrucItuaaoFxjoHWIjdIOj1gfalse2023-09-22T21:08:22.668Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVi5UgYUqMidEG29Nj6qJbNhM09I8CJcfalse2023-09-22T21:08:22.616Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhn7HL8wXg0x8WecE.JHMlcrta8XT0VLfalse2023-09-22T21:08:22.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vMfalse2023-09-22T21:08:22.565Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:43.597 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-22T21:08:43.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.599 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:08:43.599 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:43.599 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:43.599 DEBUG:boto:path=/ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:43.599 DEBUG:boto:auth_path=/ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:43.599 DEBUG:boto:Method: GET 2023-09-22T21:08:43.600 DEBUG:boto:Path: /ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:43.600 DEBUG:boto:Data: 2023-09-22T21:08:43.600 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.600 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:43.600 DEBUG:boto:Port: 8000 2023-09-22T21:08:43.600 DEBUG:boto:Params: {} 2023-09-22T21:08:43.600 DEBUG:boto:Token: None 2023-09-22T21:08:43.600 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/?versions 2023-09-22T21:08:43.600 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SWoRHcFRDr9SynP+8iFnlEZtZUk= 2023-09-22T21:08:43.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SWoRHcFRDr9SynP+8iFnlEZtZUk=', 'Content-Length': '0'} 2023-09-22T21:08:43.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fce02842cd69c1e3-00650e025b-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.603 DEBUG:boto:b'ffzdqu-13999falsedummyEH3tK3P9S8trmvB9h-eP4MX-b1Uhu-Ltrue2023-09-22T21:07:46.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVPOyc5MFRGSz2wBgBorKv2oh8K4FSnytrue2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjwzrpoM6PF-asZD0RLHOKLOYe9ma4z8Nfalse2023-09-22T21:08:23.268Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLk20u5mUDXrc4IGPE.LofVWMUByRmdSfalse2023-09-22T21:08:23.219Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvqVKHaLLoJZurTgAvh6O-aPYcyOZI.Hfalse2023-09-22T21:08:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZHOtFSYtYeqSDp3B48tnX9aoyhE-LVYfalse2023-09-22T21:08:23.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtrwlNr-EtLMj-wIVlmYw9dBWQcMpEiDfalse2023-09-22T21:08:23.159Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKeL89-xXRu815nqON693z3wYbeHRSpwfalse2023-09-22T21:08:23.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnkf1H6kXofD14zadwSPLBP-vHM76qTMfalse2023-09-22T21:08:23.030Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2false2023-09-22T21:08:22.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6-gODrZ6LdnRGbTNPrF.c3yj-YHJffpfalse2023-09-22T21:08:23.000Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjX5s9z0mNXbqm37pFm1vpfPexuZJVGd0false2023-09-22T21:08:22.866Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZmK2SagIILGhSA.9md7mKoG8ffxTZzdfalse2023-09-22T21:08:22.895Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMWb3f8.clEGzrytonhqa4UHE-3nWzBcfalse2023-09-22T21:08:22.785Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3false2023-09-22T21:08:22.836Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUx-o.jzhWCEq59rWr2ALKdruobR.tsYfalse2023-09-22T21:08:22.728Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZFg17GpJzZzJQ2hycn0YSoQcRmEUKXdfalse2023-09-22T21:08:22.699Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2t60n7HyrucItuaaoFxjoHWIjdIOj1gfalse2023-09-22T21:08:22.668Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVi5UgYUqMidEG29Nj6qJbNhM09I8CJcfalse2023-09-22T21:08:22.616Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhn7HL8wXg0x8WecE.JHMlcrta8XT0VLfalse2023-09-22T21:08:22.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vMfalse2023-09-22T21:08:22.565Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:43.604 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-22T21:08:43.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:08:43.607 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:43.607 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:43.607 DEBUG:boto:path=/ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:43.607 DEBUG:boto:auth_path=/ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:43.607 DEBUG:boto:Method: GET 2023-09-22T21:08:43.607 DEBUG:boto:Path: /ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:43.607 DEBUG:boto:Data: 2023-09-22T21:08:43.607 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.607 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:43.607 DEBUG:boto:Port: 8001 2023-09-22T21:08:43.608 DEBUG:boto:Params: {} 2023-09-22T21:08:43.608 DEBUG:boto:Token: None 2023-09-22T21:08:43.608 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/?versions 2023-09-22T21:08:43.608 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SWoRHcFRDr9SynP+8iFnlEZtZUk= 2023-09-22T21:08:43.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SWoRHcFRDr9SynP+8iFnlEZtZUk=', 'Content-Length': '0'} 2023-09-22T21:08:43.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c483a16529a6b041-00650e025b-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.611 DEBUG:boto:b'ffzdqu-13999falsedummyEH3tK3P9S8trmvB9h-eP4MX-b1Uhu-Ltrue2023-09-22T21:07:46.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVPOyc5MFRGSz2wBgBorKv2oh8K4FSnytrue2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjwzrpoM6PF-asZD0RLHOKLOYe9ma4z8Nfalse2023-09-22T21:08:23.268Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLk20u5mUDXrc4IGPE.LofVWMUByRmdSfalse2023-09-22T21:08:23.219Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvqVKHaLLoJZurTgAvh6O-aPYcyOZI.Hfalse2023-09-22T21:08:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZHOtFSYtYeqSDp3B48tnX9aoyhE-LVYfalse2023-09-22T21:08:23.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtrwlNr-EtLMj-wIVlmYw9dBWQcMpEiDfalse2023-09-22T21:08:23.159Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKeL89-xXRu815nqON693z3wYbeHRSpwfalse2023-09-22T21:08:23.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnkf1H6kXofD14zadwSPLBP-vHM76qTMfalse2023-09-22T21:08:23.030Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2false2023-09-22T21:08:22.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6-gODrZ6LdnRGbTNPrF.c3yj-YHJffpfalse2023-09-22T21:08:23.000Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjX5s9z0mNXbqm37pFm1vpfPexuZJVGd0false2023-09-22T21:08:22.866Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZmK2SagIILGhSA.9md7mKoG8ffxTZzdfalse2023-09-22T21:08:22.895Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMWb3f8.clEGzrytonhqa4UHE-3nWzBcfalse2023-09-22T21:08:22.785Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3false2023-09-22T21:08:22.836Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUx-o.jzhWCEq59rWr2ALKdruobR.tsYfalse2023-09-22T21:08:22.728Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZFg17GpJzZzJQ2hycn0YSoQcRmEUKXdfalse2023-09-22T21:08:22.699Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2t60n7HyrucItuaaoFxjoHWIjdIOj1gfalse2023-09-22T21:08:22.668Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVi5UgYUqMidEG29Nj6qJbNhM09I8CJcfalse2023-09-22T21:08:22.616Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhn7HL8wXg0x8WecE.JHMlcrta8XT0VLfalse2023-09-22T21:08:22.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vMfalse2023-09-22T21:08:22.565Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:43.612 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:43.612 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:43.613 DEBUG:boto:path=/ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:43.613 DEBUG:boto:auth_path=/ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:43.613 DEBUG:boto:Method: GET 2023-09-22T21:08:43.613 DEBUG:boto:Path: /ffzdqu-13/?versions&max-keys=999 2023-09-22T21:08:43.613 DEBUG:boto:Data: 2023-09-22T21:08:43.613 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.613 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:43.613 DEBUG:boto:Port: 8000 2023-09-22T21:08:43.613 DEBUG:boto:Params: {} 2023-09-22T21:08:43.614 DEBUG:boto:Token: None 2023-09-22T21:08:43.614 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/?versions 2023-09-22T21:08:43.614 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SWoRHcFRDr9SynP+8iFnlEZtZUk= 2023-09-22T21:08:43.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SWoRHcFRDr9SynP+8iFnlEZtZUk=', 'Content-Length': '0'} 2023-09-22T21:08:43.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bdf4ca152c3296c-00650e025b-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.617 DEBUG:boto:b'ffzdqu-13999falsedummyEH3tK3P9S8trmvB9h-eP4MX-b1Uhu-Ltrue2023-09-22T21:07:46.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVPOyc5MFRGSz2wBgBorKv2oh8K4FSnytrue2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjwzrpoM6PF-asZD0RLHOKLOYe9ma4z8Nfalse2023-09-22T21:08:23.268Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLk20u5mUDXrc4IGPE.LofVWMUByRmdSfalse2023-09-22T21:08:23.219Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvqVKHaLLoJZurTgAvh6O-aPYcyOZI.Hfalse2023-09-22T21:08:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZHOtFSYtYeqSDp3B48tnX9aoyhE-LVYfalse2023-09-22T21:08:23.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtrwlNr-EtLMj-wIVlmYw9dBWQcMpEiDfalse2023-09-22T21:08:23.159Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKeL89-xXRu815nqON693z3wYbeHRSpwfalse2023-09-22T21:08:23.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnkf1H6kXofD14zadwSPLBP-vHM76qTMfalse2023-09-22T21:08:23.030Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2false2023-09-22T21:08:22.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6-gODrZ6LdnRGbTNPrF.c3yj-YHJffpfalse2023-09-22T21:08:23.000Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjX5s9z0mNXbqm37pFm1vpfPexuZJVGd0false2023-09-22T21:08:22.866Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZmK2SagIILGhSA.9md7mKoG8ffxTZzdfalse2023-09-22T21:08:22.895Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMWb3f8.clEGzrytonhqa4UHE-3nWzBcfalse2023-09-22T21:08:22.785Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3false2023-09-22T21:08:22.836Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUx-o.jzhWCEq59rWr2ALKdruobR.tsYfalse2023-09-22T21:08:22.728Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZFg17GpJzZzJQ2hycn0YSoQcRmEUKXdfalse2023-09-22T21:08:22.699Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2t60n7HyrucItuaaoFxjoHWIjdIOj1gfalse2023-09-22T21:08:22.668Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVi5UgYUqMidEG29Nj6qJbNhM09I8CJcfalse2023-09-22T21:08:22.616Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhn7HL8wXg0x8WecE.JHMlcrta8XT0VLfalse2023-09-22T21:08:22.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vMfalse2023-09-22T21:08:22.565Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:43.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-22T21:08:43.619 DEBUG:boto:path=/ffzdqu-13/dummy 2023-09-22T21:08:43.619 DEBUG:boto:auth_path=/ffzdqu-13/dummy 2023-09-22T21:08:43.619 DEBUG:boto:path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.619 DEBUG:boto:auth_path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.620 DEBUG:boto:Method: GET 2023-09-22T21:08:43.620 DEBUG:boto:Path: /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.620 DEBUG:boto:Data: 2023-09-22T21:08:43.620 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.620 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:43.620 DEBUG:boto:Port: 8001 2023-09-22T21:08:43.620 DEBUG:boto:Params: {} 2023-09-22T21:08:43.620 DEBUG:boto:Token: None 2023-09-22T21:08:43.620 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.621 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lmNy7Dj46Z85iQAIF43fvrbR8q4= 2023-09-22T21:08:43.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lmNy7Dj46Z85iQAIF43fvrbR8q4=', 'Content-Length': '0'} 2023-09-22T21:08:43.623 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('x-amz-version-id', 'EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000071f0d3d6ac05bc9e-00650e025b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.623 DEBUG:boto:path=/ffzdqu-13/dummy 2023-09-22T21:08:43.623 DEBUG:boto:auth_path=/ffzdqu-13/dummy 2023-09-22T21:08:43.623 DEBUG:boto:path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.623 DEBUG:boto:auth_path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.623 DEBUG:boto:Method: GET 2023-09-22T21:08:43.624 DEBUG:boto:Path: /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.624 DEBUG:boto:Data: 2023-09-22T21:08:43.624 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.624 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:43.624 DEBUG:boto:Port: 8000 2023-09-22T21:08:43.624 DEBUG:boto:Params: {} 2023-09-22T21:08:43.624 DEBUG:boto:Token: None 2023-09-22T21:08:43.624 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.625 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lmNy7Dj46Z85iQAIF43fvrbR8q4= 2023-09-22T21:08:43.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lmNy7Dj46Z85iQAIF43fvrbR8q4=', 'Content-Length': '0'} 2023-09-22T21:08:43.626 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('x-amz-version-id', 'EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ee4a222250e94c08-00650e025b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.626 DEBUG:boto:path=/ffzdqu-13/dummy 2023-09-22T21:08:43.627 DEBUG:boto:auth_path=/ffzdqu-13/dummy 2023-09-22T21:08:43.627 DEBUG:boto:path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.627 DEBUG:boto:auth_path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.627 DEBUG:boto:Method: HEAD 2023-09-22T21:08:43.627 DEBUG:boto:Path: /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.627 DEBUG:boto:Data: 2023-09-22T21:08:43.627 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.627 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:43.627 DEBUG:boto:Port: 8001 2023-09-22T21:08:43.628 DEBUG:boto:Params: {} 2023-09-22T21:08:43.628 DEBUG:boto:Token: None 2023-09-22T21:08:43.628 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.628 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OE498BUdRB4YMQ8k8i9JPw08LIo= 2023-09-22T21:08:43.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OE498BUdRB4YMQ8k8i9JPw08LIo=', 'Content-Length': '0'} 2023-09-22T21:08:43.630 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('x-amz-version-id', 'EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fbe3e536ee2f3b68-00650e025b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.630 DEBUG:boto:path=/ffzdqu-13/dummy 2023-09-22T21:08:43.631 DEBUG:boto:auth_path=/ffzdqu-13/dummy 2023-09-22T21:08:43.631 DEBUG:boto:path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.631 DEBUG:boto:auth_path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.631 DEBUG:boto:Method: HEAD 2023-09-22T21:08:43.631 DEBUG:boto:Path: /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.631 DEBUG:boto:Data: 2023-09-22T21:08:43.631 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.631 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:43.631 DEBUG:boto:Port: 8000 2023-09-22T21:08:43.631 DEBUG:boto:Params: {} 2023-09-22T21:08:43.632 DEBUG:boto:Token: None 2023-09-22T21:08:43.632 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.632 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OE498BUdRB4YMQ8k8i9JPw08LIo= 2023-09-22T21:08:43.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OE498BUdRB4YMQ8k8i9JPw08LIo=', 'Content-Length': '0'} 2023-09-22T21:08:43.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('x-amz-version-id', 'EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000709f81582d4d2733-00650e025b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-22T21:08:43.634 DEBUG:boto:path=/ffzdqu-13/dummy 2023-09-22T21:08:43.635 DEBUG:boto:auth_path=/ffzdqu-13/dummy 2023-09-22T21:08:43.635 DEBUG:boto:path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.635 DEBUG:boto:auth_path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.635 DEBUG:boto:Method: GET 2023-09-22T21:08:43.635 DEBUG:boto:Path: /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.635 DEBUG:boto:Data: 2023-09-22T21:08:43.635 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.635 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:43.635 DEBUG:boto:Port: 8001 2023-09-22T21:08:43.636 DEBUG:boto:Params: {} 2023-09-22T21:08:43.636 DEBUG:boto:Token: None 2023-09-22T21:08:43.636 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.636 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lmNy7Dj46Z85iQAIF43fvrbR8q4= 2023-09-22T21:08:43.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lmNy7Dj46Z85iQAIF43fvrbR8q4=', 'Content-Length': '0'} 2023-09-22T21:08:43.638 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('x-amz-version-id', 'EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000085e8b4ebc9db2378-00650e025b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.639 DEBUG:boto:path=/ffzdqu-13/dummy 2023-09-22T21:08:43.639 DEBUG:boto:auth_path=/ffzdqu-13/dummy 2023-09-22T21:08:43.639 DEBUG:boto:path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.639 DEBUG:boto:auth_path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.639 DEBUG:boto:Method: GET 2023-09-22T21:08:43.639 DEBUG:boto:Path: /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.639 DEBUG:boto:Data: 2023-09-22T21:08:43.640 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.640 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:43.640 DEBUG:boto:Port: 8000 2023-09-22T21:08:43.640 DEBUG:boto:Params: {} 2023-09-22T21:08:43.640 DEBUG:boto:Token: None 2023-09-22T21:08:43.640 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.640 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lmNy7Dj46Z85iQAIF43fvrbR8q4= 2023-09-22T21:08:43.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lmNy7Dj46Z85iQAIF43fvrbR8q4=', 'Content-Length': '0'} 2023-09-22T21:08:43.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('x-amz-version-id', 'EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007990fdc8d5c8fc0a-00650e025b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.643 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:43.643 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:43.643 DEBUG:boto:path=/ffzdqu-13/?prefix=dummy 2023-09-22T21:08:43.643 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=dummy 2023-09-22T21:08:43.643 DEBUG:boto:Method: GET 2023-09-22T21:08:43.643 DEBUG:boto:Path: /ffzdqu-13/?prefix=dummy 2023-09-22T21:08:43.643 DEBUG:boto:Data: 2023-09-22T21:08:43.643 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.644 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:43.644 DEBUG:boto:Port: 8001 2023-09-22T21:08:43.644 DEBUG:boto:Params: {} 2023-09-22T21:08:43.644 DEBUG:boto:Token: None 2023-09-22T21:08:43.644 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/ 2023-09-22T21:08:43.644 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M9F29gEjFgL/zPot1+6ZyPewwxA= 2023-09-22T21:08:43.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M9F29gEjFgL/zPot1+6ZyPewwxA=', 'Content-Length': '0'} 2023-09-22T21:08:43.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a4b24bd8d86fa7d-00650e025b-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.694 DEBUG:boto:b'ffzdqu-13dummy1000falsedummy2023-09-22T21:07:46.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:43.695 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:43.695 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:43.695 DEBUG:boto:path=/ffzdqu-13/?prefix=dummy 2023-09-22T21:08:43.695 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=dummy 2023-09-22T21:08:43.695 DEBUG:boto:Method: GET 2023-09-22T21:08:43.695 DEBUG:boto:Path: /ffzdqu-13/?prefix=dummy 2023-09-22T21:08:43.696 DEBUG:boto:Data: 2023-09-22T21:08:43.696 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.696 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:43.696 DEBUG:boto:Port: 8000 2023-09-22T21:08:43.696 DEBUG:boto:Params: {} 2023-09-22T21:08:43.696 DEBUG:boto:Token: None 2023-09-22T21:08:43.696 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/ 2023-09-22T21:08:43.697 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M9F29gEjFgL/zPot1+6ZyPewwxA= 2023-09-22T21:08:43.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M9F29gEjFgL/zPot1+6ZyPewwxA=', 'Content-Length': '0'} 2023-09-22T21:08:43.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3849af8bfa4fc4e-00650e025b-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.702 DEBUG:boto:b'ffzdqu-13dummy1000falsedummy2023-09-22T21:07:46.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:43.702 DEBUG:boto:path=/ffzdqu-13/dummy 2023-09-22T21:08:43.702 DEBUG:boto:auth_path=/ffzdqu-13/dummy 2023-09-22T21:08:43.702 DEBUG:boto:Method: HEAD 2023-09-22T21:08:43.702 DEBUG:boto:Path: /ffzdqu-13/dummy 2023-09-22T21:08:43.703 DEBUG:boto:Data: 2023-09-22T21:08:43.703 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.703 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:43.703 DEBUG:boto:Port: 8001 2023-09-22T21:08:43.703 DEBUG:boto:Params: {} 2023-09-22T21:08:43.703 DEBUG:boto:Token: None 2023-09-22T21:08:43.703 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/dummy 2023-09-22T21:08:43.703 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+N5Z5bwXX0nWP4DmymBPo44NA8w= 2023-09-22T21:08:43.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+N5Z5bwXX0nWP4DmymBPo44NA8w=', 'Content-Length': '0'} 2023-09-22T21:08:43.707 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('x-amz-version-id', 'EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003713a8f3fb246e52-00650e025b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.707 DEBUG:boto:path=/ffzdqu-13/dummy 2023-09-22T21:08:43.707 DEBUG:boto:auth_path=/ffzdqu-13/dummy 2023-09-22T21:08:43.707 DEBUG:boto:Method: HEAD 2023-09-22T21:08:43.708 DEBUG:boto:Path: /ffzdqu-13/dummy 2023-09-22T21:08:43.708 DEBUG:boto:Data: 2023-09-22T21:08:43.708 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.708 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:43.708 DEBUG:boto:Port: 8000 2023-09-22T21:08:43.708 DEBUG:boto:Params: {} 2023-09-22T21:08:43.708 DEBUG:boto:Token: None 2023-09-22T21:08:43.708 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/dummy 2023-09-22T21:08:43.708 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+N5Z5bwXX0nWP4DmymBPo44NA8w= 2023-09-22T21:08:43.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+N5Z5bwXX0nWP4DmymBPo44NA8w=', 'Content-Length': '0'} 2023-09-22T21:08:43.712 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('x-amz-version-id', 'EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000227e8dc53b21709f-00650e025b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-22T21:08:43.712 DEBUG:boto:path=/ffzdqu-13/dummy 2023-09-22T21:08:43.712 DEBUG:boto:auth_path=/ffzdqu-13/dummy 2023-09-22T21:08:43.713 DEBUG:boto:path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.713 DEBUG:boto:auth_path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.713 DEBUG:boto:Method: GET 2023-09-22T21:08:43.713 DEBUG:boto:Path: /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.713 DEBUG:boto:Data: 2023-09-22T21:08:43.713 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.713 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:43.713 DEBUG:boto:Port: 8001 2023-09-22T21:08:43.713 DEBUG:boto:Params: {} 2023-09-22T21:08:43.713 DEBUG:boto:Token: None 2023-09-22T21:08:43.714 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.714 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lmNy7Dj46Z85iQAIF43fvrbR8q4= 2023-09-22T21:08:43.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lmNy7Dj46Z85iQAIF43fvrbR8q4=', 'Content-Length': '0'} 2023-09-22T21:08:43.716 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('x-amz-version-id', 'EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000010223f3ab041889b-00650e025b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.717 DEBUG:boto:path=/ffzdqu-13/dummy 2023-09-22T21:08:43.717 DEBUG:boto:auth_path=/ffzdqu-13/dummy 2023-09-22T21:08:43.717 DEBUG:boto:path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.717 DEBUG:boto:auth_path=/ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.717 DEBUG:boto:Method: GET 2023-09-22T21:08:43.717 DEBUG:boto:Path: /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.717 DEBUG:boto:Data: 2023-09-22T21:08:43.717 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.717 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:43.718 DEBUG:boto:Port: 8000 2023-09-22T21:08:43.718 DEBUG:boto:Params: {} 2023-09-22T21:08:43.718 DEBUG:boto:Token: None 2023-09-22T21:08:43.718 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/dummy?versionId=EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L 2023-09-22T21:08:43.718 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lmNy7Dj46Z85iQAIF43fvrbR8q4= 2023-09-22T21:08:43.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lmNy7Dj46Z85iQAIF43fvrbR8q4=', 'Content-Length': '0'} 2023-09-22T21:08:43.721 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:07:46 GMT'), ('x-amz-version-id', 'EH3tK3P9S8trmvB9h-eP4MX-b1Uhu-L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000073439b77ea1b1729-00650e025b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.721 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:43.721 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:43.721 DEBUG:boto:path=/ffzdqu-13/?prefix=dummy 2023-09-22T21:08:43.721 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=dummy 2023-09-22T21:08:43.722 DEBUG:boto:Method: GET 2023-09-22T21:08:43.722 DEBUG:boto:Path: /ffzdqu-13/?prefix=dummy 2023-09-22T21:08:43.722 DEBUG:boto:Data: 2023-09-22T21:08:43.722 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.722 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:43.722 DEBUG:boto:Port: 8001 2023-09-22T21:08:43.722 DEBUG:boto:Params: {} 2023-09-22T21:08:43.722 DEBUG:boto:Token: None 2023-09-22T21:08:43.722 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/ 2023-09-22T21:08:43.723 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M9F29gEjFgL/zPot1+6ZyPewwxA= 2023-09-22T21:08:43.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M9F29gEjFgL/zPot1+6ZyPewwxA=', 'Content-Length': '0'} 2023-09-22T21:08:43.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009379d646f9e0a856-00650e025b-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.727 DEBUG:boto:b'ffzdqu-13dummy1000falsedummy2023-09-22T21:07:46.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:43.728 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:43.728 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:43.728 DEBUG:boto:path=/ffzdqu-13/?prefix=dummy 2023-09-22T21:08:43.728 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=dummy 2023-09-22T21:08:43.728 DEBUG:boto:Method: GET 2023-09-22T21:08:43.728 DEBUG:boto:Path: /ffzdqu-13/?prefix=dummy 2023-09-22T21:08:43.728 DEBUG:boto:Data: 2023-09-22T21:08:43.728 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.728 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:43.728 DEBUG:boto:Port: 8000 2023-09-22T21:08:43.729 DEBUG:boto:Params: {} 2023-09-22T21:08:43.729 DEBUG:boto:Token: None 2023-09-22T21:08:43.729 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/ 2023-09-22T21:08:43.729 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M9F29gEjFgL/zPot1+6ZyPewwxA= 2023-09-22T21:08:43.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M9F29gEjFgL/zPot1+6ZyPewwxA=', 'Content-Length': '0'} 2023-09-22T21:08:43.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003342dafa0761f224-00650e025b-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.734 DEBUG:boto:b'ffzdqu-13dummy1000falsedummy2023-09-22T21:07:46.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:43.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:43.735 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:43.735 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:43.735 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.735 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.735 DEBUG:boto:Method: GET 2023-09-22T21:08:43.735 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.735 DEBUG:boto:Data: 2023-09-22T21:08:43.736 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:43.736 DEBUG:boto:Port: 8001 2023-09-22T21:08:43.736 DEBUG:boto:Params: {} 2023-09-22T21:08:43.736 DEBUG:boto:Token: None 2023-09-22T21:08:43.736 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.736 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dbRsFlQMQJUdg73cKlAVCyr4aps= 2023-09-22T21:08:43.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dbRsFlQMQJUdg73cKlAVCyr4aps=', 'Content-Length': '0'} 2023-09-22T21:08:43.739 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059da3842a3dfef21-00650e025b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.786 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:43.786 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:43.787 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.787 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.787 DEBUG:boto:Method: GET 2023-09-22T21:08:43.787 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.787 DEBUG:boto:Data: 2023-09-22T21:08:43.787 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.787 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:43.787 DEBUG:boto:Port: 8000 2023-09-22T21:08:43.787 DEBUG:boto:Params: {} 2023-09-22T21:08:43.788 DEBUG:boto:Token: None 2023-09-22T21:08:43.788 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.788 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dbRsFlQMQJUdg73cKlAVCyr4aps= 2023-09-22T21:08:43.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dbRsFlQMQJUdg73cKlAVCyr4aps=', 'Content-Length': '0'} 2023-09-22T21:08:43.791 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000690936321366f46f-00650e025b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.791 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:43.791 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:43.791 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.792 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.792 DEBUG:boto:Method: HEAD 2023-09-22T21:08:43.792 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.792 DEBUG:boto:Data: 2023-09-22T21:08:43.792 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.792 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:43.792 DEBUG:boto:Port: 8001 2023-09-22T21:08:43.792 DEBUG:boto:Params: {} 2023-09-22T21:08:43.792 DEBUG:boto:Token: None 2023-09-22T21:08:43.792 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.793 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bJ7OhNlOY3saO9mvB7qKZIOEb0E= 2023-09-22T21:08:43.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bJ7OhNlOY3saO9mvB7qKZIOEb0E=', 'Content-Length': '0'} 2023-09-22T21:08:43.795 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000871b98fa2d447c64-00650e025b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.796 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:43.796 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:43.796 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.796 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.796 DEBUG:boto:Method: HEAD 2023-09-22T21:08:43.796 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.796 DEBUG:boto:Data: 2023-09-22T21:08:43.797 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.797 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:43.797 DEBUG:boto:Port: 8000 2023-09-22T21:08:43.797 DEBUG:boto:Params: {} 2023-09-22T21:08:43.797 DEBUG:boto:Token: None 2023-09-22T21:08:43.797 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.797 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bJ7OhNlOY3saO9mvB7qKZIOEb0E= 2023-09-22T21:08:43.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bJ7OhNlOY3saO9mvB7qKZIOEb0E=', 'Content-Length': '0'} 2023-09-22T21:08:43.800 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052aec86d5717538f-00650e025b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:43.801 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:43.801 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:43.801 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.801 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.801 DEBUG:boto:Method: GET 2023-09-22T21:08:43.801 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.801 DEBUG:boto:Data: 2023-09-22T21:08:43.801 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.801 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:43.801 DEBUG:boto:Port: 8001 2023-09-22T21:08:43.802 DEBUG:boto:Params: {} 2023-09-22T21:08:43.802 DEBUG:boto:Token: None 2023-09-22T21:08:43.802 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.802 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dbRsFlQMQJUdg73cKlAVCyr4aps= 2023-09-22T21:08:43.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dbRsFlQMQJUdg73cKlAVCyr4aps=', 'Content-Length': '0'} 2023-09-22T21:08:43.805 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000755ab1b055266f23-00650e025b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.805 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:43.805 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:43.806 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.806 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.806 DEBUG:boto:Method: GET 2023-09-22T21:08:43.806 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.806 DEBUG:boto:Data: 2023-09-22T21:08:43.807 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.807 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:43.807 DEBUG:boto:Port: 8000 2023-09-22T21:08:43.807 DEBUG:boto:Params: {} 2023-09-22T21:08:43.807 DEBUG:boto:Token: None 2023-09-22T21:08:43.807 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.807 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dbRsFlQMQJUdg73cKlAVCyr4aps= 2023-09-22T21:08:43.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dbRsFlQMQJUdg73cKlAVCyr4aps=', 'Content-Length': '0'} 2023-09-22T21:08:43.811 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052d39f543bb5a77b-00650e025b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.858 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:43.859 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:43.859 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:43.859 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:43.859 DEBUG:boto:Method: GET 2023-09-22T21:08:43.859 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:43.859 DEBUG:boto:Data: 2023-09-22T21:08:43.859 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.859 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:43.859 DEBUG:boto:Port: 8001 2023-09-22T21:08:43.860 DEBUG:boto:Params: {} 2023-09-22T21:08:43.860 DEBUG:boto:Token: None 2023-09-22T21:08:43.860 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/ 2023-09-22T21:08:43.860 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M9F29gEjFgL/zPot1+6ZyPewwxA= 2023-09-22T21:08:43.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M9F29gEjFgL/zPot1+6ZyPewwxA=', 'Content-Length': '0'} 2023-09-22T21:08:43.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4971e64c83fe2bf-00650e025b-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.865 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:43.866 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:43.866 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:43.866 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:43.866 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:43.866 DEBUG:boto:Method: GET 2023-09-22T21:08:43.866 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:43.866 DEBUG:boto:Data: 2023-09-22T21:08:43.867 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.867 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:43.867 DEBUG:boto:Port: 8000 2023-09-22T21:08:43.867 DEBUG:boto:Params: {} 2023-09-22T21:08:43.867 DEBUG:boto:Token: None 2023-09-22T21:08:43.867 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/ 2023-09-22T21:08:43.867 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M9F29gEjFgL/zPot1+6ZyPewwxA= 2023-09-22T21:08:43.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M9F29gEjFgL/zPot1+6ZyPewwxA=', 'Content-Length': '0'} 2023-09-22T21:08:43.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086d9162d911c0783-00650e025b-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.872 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:43.873 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:43.873 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:43.873 DEBUG:boto:Method: HEAD 2023-09-22T21:08:43.873 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:43.873 DEBUG:boto:Data: 2023-09-22T21:08:43.873 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.873 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:43.873 DEBUG:boto:Port: 8001 2023-09-22T21:08:43.873 DEBUG:boto:Params: {} 2023-09-22T21:08:43.874 DEBUG:boto:Token: None 2023-09-22T21:08:43.874 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/obj 2023-09-22T21:08:43.874 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HBjKr48oZLMUJo92ONN1WMU6LQ8= 2023-09-22T21:08:43.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HBjKr48oZLMUJo92ONN1WMU6LQ8=', 'Content-Length': '0'} 2023-09-22T21:08:43.880 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d140cf5aa700dc6-00650e025b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.880 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:43.880 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:43.880 DEBUG:boto:Method: HEAD 2023-09-22T21:08:43.880 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:43.881 DEBUG:boto:Data: 2023-09-22T21:08:43.881 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.881 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:43.881 DEBUG:boto:Port: 8000 2023-09-22T21:08:43.881 DEBUG:boto:Params: {} 2023-09-22T21:08:43.881 DEBUG:boto:Token: None 2023-09-22T21:08:43.881 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/obj 2023-09-22T21:08:43.881 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HBjKr48oZLMUJo92ONN1WMU6LQ8= 2023-09-22T21:08:43.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HBjKr48oZLMUJo92ONN1WMU6LQ8=', 'Content-Length': '0'} 2023-09-22T21:08:43.887 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5b0ca0a5c803c08-00650e025b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:43.887 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:43.887 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:43.887 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.887 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.888 DEBUG:boto:Method: GET 2023-09-22T21:08:43.888 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.888 DEBUG:boto:Data: 2023-09-22T21:08:43.888 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.888 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:43.888 DEBUG:boto:Port: 8001 2023-09-22T21:08:43.888 DEBUG:boto:Params: {} 2023-09-22T21:08:43.888 DEBUG:boto:Token: None 2023-09-22T21:08:43.889 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.889 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dbRsFlQMQJUdg73cKlAVCyr4aps= 2023-09-22T21:08:43.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dbRsFlQMQJUdg73cKlAVCyr4aps=', 'Content-Length': '0'} 2023-09-22T21:08:43.891 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044841c12688ea013-00650e025b-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.934 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:43.935 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:43.935 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.935 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.935 DEBUG:boto:Method: GET 2023-09-22T21:08:43.935 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.936 DEBUG:boto:Data: 2023-09-22T21:08:43.936 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.936 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:43.936 DEBUG:boto:Port: 8000 2023-09-22T21:08:43.936 DEBUG:boto:Params: {} 2023-09-22T21:08:43.936 DEBUG:boto:Token: None 2023-09-22T21:08:43.936 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:43.936 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dbRsFlQMQJUdg73cKlAVCyr4aps= 2023-09-22T21:08:43.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dbRsFlQMQJUdg73cKlAVCyr4aps=', 'Content-Length': '0'} 2023-09-22T21:08:43.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ea6c700f40bccd9-00650e025b-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.982 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:43.983 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:43.983 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:43.983 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:43.983 DEBUG:boto:Method: GET 2023-09-22T21:08:43.983 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:43.983 DEBUG:boto:Data: 2023-09-22T21:08:43.983 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.984 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:43.984 DEBUG:boto:Port: 8001 2023-09-22T21:08:43.984 DEBUG:boto:Params: {} 2023-09-22T21:08:43.984 DEBUG:boto:Token: None 2023-09-22T21:08:43.984 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/ 2023-09-22T21:08:43.985 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M9F29gEjFgL/zPot1+6ZyPewwxA= 2023-09-22T21:08:43.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M9F29gEjFgL/zPot1+6ZyPewwxA=', 'Content-Length': '0'} 2023-09-22T21:08:43.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045c78fe8ca54e0e0-00650e025b-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.990 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:43.990 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:43.991 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:43.991 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:43.991 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:43.991 DEBUG:boto:Method: GET 2023-09-22T21:08:43.991 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:43.991 DEBUG:boto:Data: 2023-09-22T21:08:43.991 DEBUG:boto:Headers: {} 2023-09-22T21:08:43.992 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:43.992 DEBUG:boto:Port: 8000 2023-09-22T21:08:43.992 DEBUG:boto:Params: {} 2023-09-22T21:08:43.992 DEBUG:boto:Token: None 2023-09-22T21:08:43.992 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:43 GMT /ffzdqu-13/ 2023-09-22T21:08:43.992 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M9F29gEjFgL/zPot1+6ZyPewwxA= 2023-09-22T21:08:43.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M9F29gEjFgL/zPot1+6ZyPewwxA=', 'Content-Length': '0'} 2023-09-22T21:08:43.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3276105700ba7a3-00650e025b-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:43.998 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:43.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:43.998 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:43.999 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:43.999 DEBUG:boto:path=/ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:43.999 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:43.999 DEBUG:boto:Method: GET 2023-09-22T21:08:43.999 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:43.999 DEBUG:boto:Data: 2023-09-22T21:08:44.000 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.000 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.000 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.000 DEBUG:boto:Params: {} 2023-09-22T21:08:44.000 DEBUG:boto:Token: None 2023-09-22T21:08:44.001 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:44.001 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TgmKVXHVgwnSbDB1QbqRhMVwjHo= 2023-09-22T21:08:44.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TgmKVXHVgwnSbDB1QbqRhMVwjHo=', 'Content-Length': '0'} 2023-09-22T21:08:44.004 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8c8f7d780ee59a5-00650e025c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.050 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.051 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.051 DEBUG:boto:path=/ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:44.051 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:44.051 DEBUG:boto:Method: GET 2023-09-22T21:08:44.051 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:44.051 DEBUG:boto:Data: 2023-09-22T21:08:44.052 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.052 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.052 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.052 DEBUG:boto:Params: {} 2023-09-22T21:08:44.052 DEBUG:boto:Token: None 2023-09-22T21:08:44.052 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:44.053 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TgmKVXHVgwnSbDB1QbqRhMVwjHo= 2023-09-22T21:08:44.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TgmKVXHVgwnSbDB1QbqRhMVwjHo=', 'Content-Length': '0'} 2023-09-22T21:08:44.055 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060811dca7792a59e-00650e025c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.056 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.056 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.056 DEBUG:boto:path=/ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:44.056 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:44.056 DEBUG:boto:Method: HEAD 2023-09-22T21:08:44.057 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:44.057 DEBUG:boto:Data: 2023-09-22T21:08:44.057 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.057 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.057 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.057 DEBUG:boto:Params: {} 2023-09-22T21:08:44.058 DEBUG:boto:Token: None 2023-09-22T21:08:44.058 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:44.058 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UDk9+qZfN4VGFumH0oMuo78dbkY= 2023-09-22T21:08:44.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UDk9+qZfN4VGFumH0oMuo78dbkY=', 'Content-Length': '0'} 2023-09-22T21:08:44.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009adb5262fe91a8f-00650e025c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.061 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.061 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.061 DEBUG:boto:path=/ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:44.061 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:44.062 DEBUG:boto:Method: HEAD 2023-09-22T21:08:44.062 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:44.062 DEBUG:boto:Data: 2023-09-22T21:08:44.062 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.062 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.062 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.062 DEBUG:boto:Params: {} 2023-09-22T21:08:44.063 DEBUG:boto:Token: None 2023-09-22T21:08:44.063 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:44.063 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UDk9+qZfN4VGFumH0oMuo78dbkY= 2023-09-22T21:08:44.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UDk9+qZfN4VGFumH0oMuo78dbkY=', 'Content-Length': '0'} 2023-09-22T21:08:44.066 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b04e43d40357d950-00650e025c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:44.066 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.067 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.067 DEBUG:boto:path=/ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:44.067 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:44.067 DEBUG:boto:Method: GET 2023-09-22T21:08:44.067 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:44.067 DEBUG:boto:Data: 2023-09-22T21:08:44.067 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.068 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.068 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.068 DEBUG:boto:Params: {} 2023-09-22T21:08:44.068 DEBUG:boto:Token: None 2023-09-22T21:08:44.068 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:44.068 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TgmKVXHVgwnSbDB1QbqRhMVwjHo= 2023-09-22T21:08:44.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TgmKVXHVgwnSbDB1QbqRhMVwjHo=', 'Content-Length': '0'} 2023-09-22T21:08:44.071 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007041903cffc4a405-00650e025c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.114 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.114 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.115 DEBUG:boto:path=/ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:44.115 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:44.115 DEBUG:boto:Method: GET 2023-09-22T21:08:44.115 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:44.115 DEBUG:boto:Data: 2023-09-22T21:08:44.115 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.115 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.116 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.116 DEBUG:boto:Params: {} 2023-09-22T21:08:44.116 DEBUG:boto:Token: None 2023-09-22T21:08:44.116 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N 2023-09-22T21:08:44.116 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TgmKVXHVgwnSbDB1QbqRhMVwjHo= 2023-09-22T21:08:44.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TgmKVXHVgwnSbDB1QbqRhMVwjHo=', 'Content-Length': '0'} 2023-09-22T21:08:44.119 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'wzrpoM6PF-asZD0RLHOKLOYe9ma4z8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086391d27066fb77e-00650e025c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.119 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:44.119 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:44.120 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.120 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.120 DEBUG:boto:Method: GET 2023-09-22T21:08:44.120 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.120 DEBUG:boto:Data: 2023-09-22T21:08:44.120 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.120 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.121 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.121 DEBUG:boto:Params: {} 2023-09-22T21:08:44.121 DEBUG:boto:Token: None 2023-09-22T21:08:44.121 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/ 2023-09-22T21:08:44.121 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk= 2023-09-22T21:08:44.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk=', 'Content-Length': '0'} 2023-09-22T21:08:44.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086b87005d2a587d1-00650e025c-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.170 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:44.171 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:44.171 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:44.171 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.171 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.171 DEBUG:boto:Method: GET 2023-09-22T21:08:44.171 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.171 DEBUG:boto:Data: 2023-09-22T21:08:44.172 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.172 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.172 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.172 DEBUG:boto:Params: {} 2023-09-22T21:08:44.172 DEBUG:boto:Token: None 2023-09-22T21:08:44.172 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/ 2023-09-22T21:08:44.172 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk= 2023-09-22T21:08:44.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk=', 'Content-Length': '0'} 2023-09-22T21:08:44.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e05d79623025f49a-00650e025c-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.177 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:44.178 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.178 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.178 DEBUG:boto:Method: HEAD 2023-09-22T21:08:44.178 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:44.178 DEBUG:boto:Data: 2023-09-22T21:08:44.178 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.178 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.178 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.178 DEBUG:boto:Params: {} 2023-09-22T21:08:44.179 DEBUG:boto:Token: None 2023-09-22T21:08:44.179 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj 2023-09-22T21:08:44.179 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0xxL5qAGd3Ad7R7iYfAZ096tA78= 2023-09-22T21:08:44.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0xxL5qAGd3Ad7R7iYfAZ096tA78=', 'Content-Length': '0'} 2023-09-22T21:08:44.184 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087a25a98c94c903f-00650e025c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.185 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.185 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.185 DEBUG:boto:Method: HEAD 2023-09-22T21:08:44.185 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:44.185 DEBUG:boto:Data: 2023-09-22T21:08:44.185 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.185 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.186 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.186 DEBUG:boto:Params: {} 2023-09-22T21:08:44.186 DEBUG:boto:Token: None 2023-09-22T21:08:44.186 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj 2023-09-22T21:08:44.186 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0xxL5qAGd3Ad7R7iYfAZ096tA78= 2023-09-22T21:08:44.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0xxL5qAGd3Ad7R7iYfAZ096tA78=', 'Content-Length': '0'} 2023-09-22T21:08:44.191 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a22fcf0658c49d48-00650e025c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:44.192 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.192 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.192 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.192 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.192 DEBUG:boto:Method: GET 2023-09-22T21:08:44.192 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.193 DEBUG:boto:Data: 2023-09-22T21:08:44.193 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.193 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.193 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.193 DEBUG:boto:Params: {} 2023-09-22T21:08:44.193 DEBUG:boto:Token: None 2023-09-22T21:08:44.193 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.193 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UNgTa524z+ShGslUAj5GAynaRt4= 2023-09-22T21:08:44.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UNgTa524z+ShGslUAj5GAynaRt4=', 'Content-Length': '0'} 2023-09-22T21:08:44.196 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5d9f20337dad94c-00650e025c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.197 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.197 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.197 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.197 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.197 DEBUG:boto:Method: GET 2023-09-22T21:08:44.197 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.197 DEBUG:boto:Data: 2023-09-22T21:08:44.197 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.197 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.198 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.198 DEBUG:boto:Params: {} 2023-09-22T21:08:44.198 DEBUG:boto:Token: None 2023-09-22T21:08:44.198 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.198 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UNgTa524z+ShGslUAj5GAynaRt4= 2023-09-22T21:08:44.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UNgTa524z+ShGslUAj5GAynaRt4=', 'Content-Length': '0'} 2023-09-22T21:08:44.201 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001fc7195311836ee2-00650e025c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.242 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:44.243 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:44.243 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.243 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.243 DEBUG:boto:Method: GET 2023-09-22T21:08:44.243 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.243 DEBUG:boto:Data: 2023-09-22T21:08:44.243 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.243 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.243 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.243 DEBUG:boto:Params: {} 2023-09-22T21:08:44.244 DEBUG:boto:Token: None 2023-09-22T21:08:44.244 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/ 2023-09-22T21:08:44.244 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk= 2023-09-22T21:08:44.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk=', 'Content-Length': '0'} 2023-09-22T21:08:44.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e43a27be3b297328-00650e025c-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.249 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:44.249 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:44.250 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:44.250 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.250 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.250 DEBUG:boto:Method: GET 2023-09-22T21:08:44.250 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.250 DEBUG:boto:Data: 2023-09-22T21:08:44.251 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.251 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.251 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.251 DEBUG:boto:Params: {} 2023-09-22T21:08:44.251 DEBUG:boto:Token: None 2023-09-22T21:08:44.251 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/ 2023-09-22T21:08:44.252 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk= 2023-09-22T21:08:44.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk=', 'Content-Length': '0'} 2023-09-22T21:08:44.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063c0a6486ac8975f-00650e025c-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.257 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:44.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:44.258 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.258 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.258 DEBUG:boto:path=/ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.258 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.258 DEBUG:boto:Method: GET 2023-09-22T21:08:44.258 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.258 DEBUG:boto:Data: 2023-09-22T21:08:44.259 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.259 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.259 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.259 DEBUG:boto:Params: {} 2023-09-22T21:08:44.259 DEBUG:boto:Token: None 2023-09-22T21:08:44.259 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.259 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yAZDnFnvq89iG4+++dskJ/XJkII= 2023-09-22T21:08:44.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yAZDnFnvq89iG4+++dskJ/XJkII=', 'Content-Length': '0'} 2023-09-22T21:08:44.262 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'Lk20u5mUDXrc4IGPE.LofVWMUByRmdS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c48b534b1be8cec5-00650e025c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.302 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.302 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.303 DEBUG:boto:path=/ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.303 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.303 DEBUG:boto:Method: GET 2023-09-22T21:08:44.303 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.303 DEBUG:boto:Data: 2023-09-22T21:08:44.303 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.303 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.303 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.303 DEBUG:boto:Params: {} 2023-09-22T21:08:44.304 DEBUG:boto:Token: None 2023-09-22T21:08:44.304 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.304 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yAZDnFnvq89iG4+++dskJ/XJkII= 2023-09-22T21:08:44.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yAZDnFnvq89iG4+++dskJ/XJkII=', 'Content-Length': '0'} 2023-09-22T21:08:44.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'Lk20u5mUDXrc4IGPE.LofVWMUByRmdS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9a46f28ae6c9667-00650e025c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.350 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.351 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.351 DEBUG:boto:path=/ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.351 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.351 DEBUG:boto:Method: HEAD 2023-09-22T21:08:44.351 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.351 DEBUG:boto:Data: 2023-09-22T21:08:44.351 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.351 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.351 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.352 DEBUG:boto:Params: {} 2023-09-22T21:08:44.352 DEBUG:boto:Token: None 2023-09-22T21:08:44.352 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.352 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H2eMNVSNd+Rr+03L2wikNUAawsk= 2023-09-22T21:08:44.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H2eMNVSNd+Rr+03L2wikNUAawsk=', 'Content-Length': '0'} 2023-09-22T21:08:44.355 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'Lk20u5mUDXrc4IGPE.LofVWMUByRmdS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2fa9fa09c249e61-00650e025c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.355 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.355 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.355 DEBUG:boto:path=/ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.355 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.356 DEBUG:boto:Method: HEAD 2023-09-22T21:08:44.356 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.356 DEBUG:boto:Data: 2023-09-22T21:08:44.356 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.356 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.356 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.356 DEBUG:boto:Params: {} 2023-09-22T21:08:44.356 DEBUG:boto:Token: None 2023-09-22T21:08:44.356 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.357 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H2eMNVSNd+Rr+03L2wikNUAawsk= 2023-09-22T21:08:44.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H2eMNVSNd+Rr+03L2wikNUAawsk=', 'Content-Length': '0'} 2023-09-22T21:08:44.359 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'Lk20u5mUDXrc4IGPE.LofVWMUByRmdS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049e868adcac05d0e-00650e025c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:44.360 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.360 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.360 DEBUG:boto:path=/ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.360 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.360 DEBUG:boto:Method: GET 2023-09-22T21:08:44.360 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.360 DEBUG:boto:Data: 2023-09-22T21:08:44.360 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.361 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.361 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.361 DEBUG:boto:Params: {} 2023-09-22T21:08:44.361 DEBUG:boto:Token: None 2023-09-22T21:08:44.361 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.361 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yAZDnFnvq89iG4+++dskJ/XJkII= 2023-09-22T21:08:44.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yAZDnFnvq89iG4+++dskJ/XJkII=', 'Content-Length': '0'} 2023-09-22T21:08:44.364 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'Lk20u5mUDXrc4IGPE.LofVWMUByRmdS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016dd4c0159dd1609-00650e025c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.364 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.365 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.365 DEBUG:boto:path=/ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.365 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.365 DEBUG:boto:Method: GET 2023-09-22T21:08:44.365 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.365 DEBUG:boto:Data: 2023-09-22T21:08:44.365 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.365 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.365 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.365 DEBUG:boto:Params: {} 2023-09-22T21:08:44.366 DEBUG:boto:Token: None 2023-09-22T21:08:44.366 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=Lk20u5mUDXrc4IGPE.LofVWMUByRmdS 2023-09-22T21:08:44.366 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yAZDnFnvq89iG4+++dskJ/XJkII= 2023-09-22T21:08:44.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yAZDnFnvq89iG4+++dskJ/XJkII=', 'Content-Length': '0'} 2023-09-22T21:08:44.369 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'Lk20u5mUDXrc4IGPE.LofVWMUByRmdS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a964e9c81acfa6c1-00650e025c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.414 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:44.415 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:44.415 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.415 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.415 DEBUG:boto:Method: GET 2023-09-22T21:08:44.415 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.415 DEBUG:boto:Data: 2023-09-22T21:08:44.415 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.415 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.416 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.416 DEBUG:boto:Params: {} 2023-09-22T21:08:44.416 DEBUG:boto:Token: None 2023-09-22T21:08:44.416 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/ 2023-09-22T21:08:44.416 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk= 2023-09-22T21:08:44.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk=', 'Content-Length': '0'} 2023-09-22T21:08:44.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077b99e31179c9036-00650e025c-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.421 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:44.422 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:44.422 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:44.422 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.422 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.422 DEBUG:boto:Method: GET 2023-09-22T21:08:44.422 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.422 DEBUG:boto:Data: 2023-09-22T21:08:44.423 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.423 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.423 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.423 DEBUG:boto:Params: {} 2023-09-22T21:08:44.423 DEBUG:boto:Token: None 2023-09-22T21:08:44.423 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/ 2023-09-22T21:08:44.423 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk= 2023-09-22T21:08:44.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk=', 'Content-Length': '0'} 2023-09-22T21:08:44.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5556b85a0b92c47-00650e025c-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.428 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:44.429 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.429 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.429 DEBUG:boto:Method: HEAD 2023-09-22T21:08:44.429 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:44.429 DEBUG:boto:Data: 2023-09-22T21:08:44.429 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.429 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.429 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.430 DEBUG:boto:Params: {} 2023-09-22T21:08:44.430 DEBUG:boto:Token: None 2023-09-22T21:08:44.430 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj 2023-09-22T21:08:44.430 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0xxL5qAGd3Ad7R7iYfAZ096tA78= 2023-09-22T21:08:44.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0xxL5qAGd3Ad7R7iYfAZ096tA78=', 'Content-Length': '0'} 2023-09-22T21:08:44.436 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9afed75136144a8-00650e025c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.436 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.436 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.436 DEBUG:boto:Method: HEAD 2023-09-22T21:08:44.436 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:44.436 DEBUG:boto:Data: 2023-09-22T21:08:44.437 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.437 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.437 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.437 DEBUG:boto:Params: {} 2023-09-22T21:08:44.437 DEBUG:boto:Token: None 2023-09-22T21:08:44.437 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj 2023-09-22T21:08:44.437 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0xxL5qAGd3Ad7R7iYfAZ096tA78= 2023-09-22T21:08:44.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0xxL5qAGd3Ad7R7iYfAZ096tA78=', 'Content-Length': '0'} 2023-09-22T21:08:44.443 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e269d2c79989b88-00650e025c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:44.444 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.444 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.444 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.444 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.444 DEBUG:boto:Method: GET 2023-09-22T21:08:44.444 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.444 DEBUG:boto:Data: 2023-09-22T21:08:44.444 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.445 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.445 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.445 DEBUG:boto:Params: {} 2023-09-22T21:08:44.445 DEBUG:boto:Token: None 2023-09-22T21:08:44.445 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.445 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UNgTa524z+ShGslUAj5GAynaRt4= 2023-09-22T21:08:44.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UNgTa524z+ShGslUAj5GAynaRt4=', 'Content-Length': '0'} 2023-09-22T21:08:44.448 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e862d51fbfc7dd9-00650e025c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.490 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.490 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.491 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.491 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.491 DEBUG:boto:Method: GET 2023-09-22T21:08:44.491 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.491 DEBUG:boto:Data: 2023-09-22T21:08:44.491 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.491 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.491 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.491 DEBUG:boto:Params: {} 2023-09-22T21:08:44.492 DEBUG:boto:Token: None 2023-09-22T21:08:44.492 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.492 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UNgTa524z+ShGslUAj5GAynaRt4= 2023-09-22T21:08:44.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UNgTa524z+ShGslUAj5GAynaRt4=', 'Content-Length': '0'} 2023-09-22T21:08:44.495 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007767bc4299543571-00650e025c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.538 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:44.539 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:44.539 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.539 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.539 DEBUG:boto:Method: GET 2023-09-22T21:08:44.539 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.539 DEBUG:boto:Data: 2023-09-22T21:08:44.539 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.539 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.539 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.539 DEBUG:boto:Params: {} 2023-09-22T21:08:44.540 DEBUG:boto:Token: None 2023-09-22T21:08:44.540 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/ 2023-09-22T21:08:44.540 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk= 2023-09-22T21:08:44.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk=', 'Content-Length': '0'} 2023-09-22T21:08:44.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087cb536bcc787667-00650e025c-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.546 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:44.546 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:44.546 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:44.547 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.547 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.547 DEBUG:boto:Method: GET 2023-09-22T21:08:44.547 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.547 DEBUG:boto:Data: 2023-09-22T21:08:44.547 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.547 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.547 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.547 DEBUG:boto:Params: {} 2023-09-22T21:08:44.548 DEBUG:boto:Token: None 2023-09-22T21:08:44.548 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/ 2023-09-22T21:08:44.548 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk= 2023-09-22T21:08:44.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk=', 'Content-Length': '0'} 2023-09-22T21:08:44.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043d5e3bf0f5fde63-00650e025c-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.553 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:44.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:44.554 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.554 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.554 DEBUG:boto:path=/ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.554 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.555 DEBUG:boto:Method: GET 2023-09-22T21:08:44.555 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.555 DEBUG:boto:Data: 2023-09-22T21:08:44.555 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.555 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.555 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.555 DEBUG:boto:Params: {} 2023-09-22T21:08:44.556 DEBUG:boto:Token: None 2023-09-22T21:08:44.556 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.556 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2gKKhSXDbh90yIJpj9ug2jLVwiE= 2023-09-22T21:08:44.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2gKKhSXDbh90yIJpj9ug2jLVwiE=', 'Content-Length': '0'} 2023-09-22T21:08:44.559 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042c69a3cee56d613-00650e025c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.602 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.602 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.603 DEBUG:boto:path=/ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.603 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.603 DEBUG:boto:Method: GET 2023-09-22T21:08:44.603 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.603 DEBUG:boto:Data: 2023-09-22T21:08:44.603 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.603 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.603 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.603 DEBUG:boto:Params: {} 2023-09-22T21:08:44.604 DEBUG:boto:Token: None 2023-09-22T21:08:44.604 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.604 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2gKKhSXDbh90yIJpj9ug2jLVwiE= 2023-09-22T21:08:44.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2gKKhSXDbh90yIJpj9ug2jLVwiE=', 'Content-Length': '0'} 2023-09-22T21:08:44.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066271fbf0c8f9b3c-00650e025c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.607 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.607 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.607 DEBUG:boto:path=/ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.607 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.607 DEBUG:boto:Method: HEAD 2023-09-22T21:08:44.608 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.608 DEBUG:boto:Data: 2023-09-22T21:08:44.608 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.608 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.608 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.608 DEBUG:boto:Params: {} 2023-09-22T21:08:44.608 DEBUG:boto:Token: None 2023-09-22T21:08:44.608 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.608 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5gn7Cx5/ws47Wg+f3OvyZ9BQZaE= 2023-09-22T21:08:44.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5gn7Cx5/ws47Wg+f3OvyZ9BQZaE=', 'Content-Length': '0'} 2023-09-22T21:08:44.611 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b8404e922ae5ce0-00650e025c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.611 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.612 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.612 DEBUG:boto:path=/ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.612 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.612 DEBUG:boto:Method: HEAD 2023-09-22T21:08:44.612 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.612 DEBUG:boto:Data: 2023-09-22T21:08:44.612 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.612 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.612 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.612 DEBUG:boto:Params: {} 2023-09-22T21:08:44.613 DEBUG:boto:Token: None 2023-09-22T21:08:44.613 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.613 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5gn7Cx5/ws47Wg+f3OvyZ9BQZaE= 2023-09-22T21:08:44.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5gn7Cx5/ws47Wg+f3OvyZ9BQZaE=', 'Content-Length': '0'} 2023-09-22T21:08:44.615 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084117dfea089fa13-00650e025c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:44.616 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.616 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.616 DEBUG:boto:path=/ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.616 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.616 DEBUG:boto:Method: GET 2023-09-22T21:08:44.616 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.616 DEBUG:boto:Data: 2023-09-22T21:08:44.617 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.617 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.617 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.617 DEBUG:boto:Params: {} 2023-09-22T21:08:44.617 DEBUG:boto:Token: None 2023-09-22T21:08:44.617 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.617 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2gKKhSXDbh90yIJpj9ug2jLVwiE= 2023-09-22T21:08:44.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2gKKhSXDbh90yIJpj9ug2jLVwiE=', 'Content-Length': '0'} 2023-09-22T21:08:44.620 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3d91b0ad59bdbf8-00650e025c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.662 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.662 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.663 DEBUG:boto:path=/ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.663 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.663 DEBUG:boto:Method: GET 2023-09-22T21:08:44.663 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.663 DEBUG:boto:Data: 2023-09-22T21:08:44.663 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.663 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.663 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.663 DEBUG:boto:Params: {} 2023-09-22T21:08:44.664 DEBUG:boto:Token: None 2023-09-22T21:08:44.664 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H 2023-09-22T21:08:44.664 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2gKKhSXDbh90yIJpj9ug2jLVwiE= 2023-09-22T21:08:44.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2gKKhSXDbh90yIJpj9ug2jLVwiE=', 'Content-Length': '0'} 2023-09-22T21:08:44.667 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'vqVKHaLLoJZurTgAvh6O-aPYcyOZI.H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065615ab19f8ebc0a-00650e025c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.667 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:44.667 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:44.667 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.667 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.668 DEBUG:boto:Method: GET 2023-09-22T21:08:44.668 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.668 DEBUG:boto:Data: 2023-09-22T21:08:44.668 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.668 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.668 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.668 DEBUG:boto:Params: {} 2023-09-22T21:08:44.668 DEBUG:boto:Token: None 2023-09-22T21:08:44.668 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/ 2023-09-22T21:08:44.669 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk= 2023-09-22T21:08:44.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk=', 'Content-Length': '0'} 2023-09-22T21:08:44.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a0795a6c7c6c828-00650e025c-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.674 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:44.675 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:44.675 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:44.675 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.675 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.675 DEBUG:boto:Method: GET 2023-09-22T21:08:44.675 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.675 DEBUG:boto:Data: 2023-09-22T21:08:44.676 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.676 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.676 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.676 DEBUG:boto:Params: {} 2023-09-22T21:08:44.676 DEBUG:boto:Token: None 2023-09-22T21:08:44.676 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/ 2023-09-22T21:08:44.676 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk= 2023-09-22T21:08:44.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk=', 'Content-Length': '0'} 2023-09-22T21:08:44.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025a05daff04ccec4-00650e025c-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.722 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:44.723 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.723 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.723 DEBUG:boto:Method: HEAD 2023-09-22T21:08:44.723 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:44.723 DEBUG:boto:Data: 2023-09-22T21:08:44.724 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.724 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.724 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.724 DEBUG:boto:Params: {} 2023-09-22T21:08:44.724 DEBUG:boto:Token: None 2023-09-22T21:08:44.724 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj 2023-09-22T21:08:44.724 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0xxL5qAGd3Ad7R7iYfAZ096tA78= 2023-09-22T21:08:44.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0xxL5qAGd3Ad7R7iYfAZ096tA78=', 'Content-Length': '0'} 2023-09-22T21:08:44.730 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000480a3b660f5badf9-00650e025c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.731 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.731 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.731 DEBUG:boto:Method: HEAD 2023-09-22T21:08:44.731 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:44.731 DEBUG:boto:Data: 2023-09-22T21:08:44.731 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.731 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.731 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.732 DEBUG:boto:Params: {} 2023-09-22T21:08:44.732 DEBUG:boto:Token: None 2023-09-22T21:08:44.732 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj 2023-09-22T21:08:44.732 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0xxL5qAGd3Ad7R7iYfAZ096tA78= 2023-09-22T21:08:44.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0xxL5qAGd3Ad7R7iYfAZ096tA78=', 'Content-Length': '0'} 2023-09-22T21:08:44.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ec8c5be0339b3bc-00650e025c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:44.738 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.738 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.738 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.739 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.739 DEBUG:boto:Method: GET 2023-09-22T21:08:44.739 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.739 DEBUG:boto:Data: 2023-09-22T21:08:44.739 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.739 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.739 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.739 DEBUG:boto:Params: {} 2023-09-22T21:08:44.739 DEBUG:boto:Token: None 2023-09-22T21:08:44.739 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.740 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UNgTa524z+ShGslUAj5GAynaRt4= 2023-09-22T21:08:44.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UNgTa524z+ShGslUAj5GAynaRt4=', 'Content-Length': '0'} 2023-09-22T21:08:44.742 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023bcce3df21dffe0-00650e025c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.743 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.743 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.743 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.743 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.743 DEBUG:boto:Method: GET 2023-09-22T21:08:44.743 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.743 DEBUG:boto:Data: 2023-09-22T21:08:44.743 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.744 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.744 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.744 DEBUG:boto:Params: {} 2023-09-22T21:08:44.744 DEBUG:boto:Token: None 2023-09-22T21:08:44.744 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.744 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UNgTa524z+ShGslUAj5GAynaRt4= 2023-09-22T21:08:44.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UNgTa524z+ShGslUAj5GAynaRt4=', 'Content-Length': '0'} 2023-09-22T21:08:44.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2741090da096f17-00650e025c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.790 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:44.791 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:44.791 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.791 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.791 DEBUG:boto:Method: GET 2023-09-22T21:08:44.791 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.791 DEBUG:boto:Data: 2023-09-22T21:08:44.791 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.791 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.791 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.792 DEBUG:boto:Params: {} 2023-09-22T21:08:44.792 DEBUG:boto:Token: None 2023-09-22T21:08:44.792 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/ 2023-09-22T21:08:44.792 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk= 2023-09-22T21:08:44.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk=', 'Content-Length': '0'} 2023-09-22T21:08:44.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca16b4c489142e14-00650e025c-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.798 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:44.798 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:44.798 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:44.798 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.799 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.799 DEBUG:boto:Method: GET 2023-09-22T21:08:44.799 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.799 DEBUG:boto:Data: 2023-09-22T21:08:44.799 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.799 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.799 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.799 DEBUG:boto:Params: {} 2023-09-22T21:08:44.799 DEBUG:boto:Token: None 2023-09-22T21:08:44.799 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/ 2023-09-22T21:08:44.800 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk= 2023-09-22T21:08:44.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk=', 'Content-Length': '0'} 2023-09-22T21:08:44.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038953001b2b1a3ea-00650e025c-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.846 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:44.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:44.847 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.847 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.847 DEBUG:boto:path=/ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.847 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.847 DEBUG:boto:Method: GET 2023-09-22T21:08:44.848 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.848 DEBUG:boto:Data: 2023-09-22T21:08:44.848 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.848 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.848 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.848 DEBUG:boto:Params: {} 2023-09-22T21:08:44.848 DEBUG:boto:Token: None 2023-09-22T21:08:44.848 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.849 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6CDL0W9dc8rEy15GlQv2eDRxN+M= 2023-09-22T21:08:44.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6CDL0W9dc8rEy15GlQv2eDRxN+M=', 'Content-Length': '0'} 2023-09-22T21:08:44.851 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094921066309f9511-00650e025c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.894 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.895 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.895 DEBUG:boto:path=/ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.895 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.895 DEBUG:boto:Method: GET 2023-09-22T21:08:44.895 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.895 DEBUG:boto:Data: 2023-09-22T21:08:44.895 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.895 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.896 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.896 DEBUG:boto:Params: {} 2023-09-22T21:08:44.896 DEBUG:boto:Token: None 2023-09-22T21:08:44.896 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.896 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6CDL0W9dc8rEy15GlQv2eDRxN+M= 2023-09-22T21:08:44.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6CDL0W9dc8rEy15GlQv2eDRxN+M=', 'Content-Length': '0'} 2023-09-22T21:08:44.899 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f6c0f9de13abf8d-00650e025c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.899 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.899 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.900 DEBUG:boto:path=/ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.900 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.900 DEBUG:boto:Method: HEAD 2023-09-22T21:08:44.900 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.900 DEBUG:boto:Data: 2023-09-22T21:08:44.900 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.900 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.900 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.901 DEBUG:boto:Params: {} 2023-09-22T21:08:44.901 DEBUG:boto:Token: None 2023-09-22T21:08:44.901 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.901 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Gq5SspWn26QtYmzstosW3JcldMU= 2023-09-22T21:08:44.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Gq5SspWn26QtYmzstosW3JcldMU=', 'Content-Length': '0'} 2023-09-22T21:08:44.903 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b439a44a17be2171-00650e025c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.904 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.904 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.904 DEBUG:boto:path=/ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.904 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.904 DEBUG:boto:Method: HEAD 2023-09-22T21:08:44.904 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.904 DEBUG:boto:Data: 2023-09-22T21:08:44.904 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.905 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.905 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.905 DEBUG:boto:Params: {} 2023-09-22T21:08:44.905 DEBUG:boto:Token: None 2023-09-22T21:08:44.905 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.905 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Gq5SspWn26QtYmzstosW3JcldMU= 2023-09-22T21:08:44.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Gq5SspWn26QtYmzstosW3JcldMU=', 'Content-Length': '0'} 2023-09-22T21:08:44.907 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d6724bbc79f3f57-00650e025c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:44.908 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.908 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.908 DEBUG:boto:path=/ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.908 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.908 DEBUG:boto:Method: GET 2023-09-22T21:08:44.908 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.908 DEBUG:boto:Data: 2023-09-22T21:08:44.909 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.909 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.909 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.909 DEBUG:boto:Params: {} 2023-09-22T21:08:44.909 DEBUG:boto:Token: None 2023-09-22T21:08:44.909 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.909 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6CDL0W9dc8rEy15GlQv2eDRxN+M= 2023-09-22T21:08:44.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6CDL0W9dc8rEy15GlQv2eDRxN+M=', 'Content-Length': '0'} 2023-09-22T21:08:44.912 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec906fd44afd9411-00650e025c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.954 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.954 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.955 DEBUG:boto:path=/ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.955 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.955 DEBUG:boto:Method: GET 2023-09-22T21:08:44.955 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.955 DEBUG:boto:Data: 2023-09-22T21:08:44.955 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.955 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.955 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.956 DEBUG:boto:Params: {} 2023-09-22T21:08:44.956 DEBUG:boto:Token: None 2023-09-22T21:08:44.956 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY 2023-09-22T21:08:44.956 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6CDL0W9dc8rEy15GlQv2eDRxN+M= 2023-09-22T21:08:44.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6CDL0W9dc8rEy15GlQv2eDRxN+M=', 'Content-Length': '0'} 2023-09-22T21:08:44.959 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'ZHOtFSYtYeqSDp3B48tnX9aoyhE-LVY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005db19759f5ca34b7-00650e025c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.960 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:44.960 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:44.960 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.960 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.960 DEBUG:boto:Method: GET 2023-09-22T21:08:44.960 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.961 DEBUG:boto:Data: 2023-09-22T21:08:44.961 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.961 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.961 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.961 DEBUG:boto:Params: {} 2023-09-22T21:08:44.961 DEBUG:boto:Token: None 2023-09-22T21:08:44.961 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/ 2023-09-22T21:08:44.962 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk= 2023-09-22T21:08:44.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk=', 'Content-Length': '0'} 2023-09-22T21:08:44.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003849d91a0f71cc8c-00650e025c-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.967 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:44.968 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:44.968 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:44.968 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.968 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.968 DEBUG:boto:Method: GET 2023-09-22T21:08:44.968 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:44.969 DEBUG:boto:Data: 2023-09-22T21:08:44.969 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.969 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.969 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.969 DEBUG:boto:Params: {} 2023-09-22T21:08:44.969 DEBUG:boto:Token: None 2023-09-22T21:08:44.969 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/ 2023-09-22T21:08:44.970 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk= 2023-09-22T21:08:44.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iUyselmksXMBFANrtmDikKWlirk=', 'Content-Length': '0'} 2023-09-22T21:08:44.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dbbd7d177be6045-00650e025c-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.974 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:44.975 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.975 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.975 DEBUG:boto:Method: HEAD 2023-09-22T21:08:44.975 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:44.975 DEBUG:boto:Data: 2023-09-22T21:08:44.975 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.976 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.976 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.976 DEBUG:boto:Params: {} 2023-09-22T21:08:44.976 DEBUG:boto:Token: None 2023-09-22T21:08:44.976 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj 2023-09-22T21:08:44.976 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0xxL5qAGd3Ad7R7iYfAZ096tA78= 2023-09-22T21:08:44.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0xxL5qAGd3Ad7R7iYfAZ096tA78=', 'Content-Length': '0'} 2023-09-22T21:08:44.981 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006eddc4ee87e2b267-00650e025c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.982 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.982 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.982 DEBUG:boto:Method: HEAD 2023-09-22T21:08:44.982 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:44.982 DEBUG:boto:Data: 2023-09-22T21:08:44.982 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.983 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:44.983 DEBUG:boto:Port: 8000 2023-09-22T21:08:44.983 DEBUG:boto:Params: {} 2023-09-22T21:08:44.983 DEBUG:boto:Token: None 2023-09-22T21:08:44.983 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj 2023-09-22T21:08:44.983 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0xxL5qAGd3Ad7R7iYfAZ096tA78= 2023-09-22T21:08:44.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0xxL5qAGd3Ad7R7iYfAZ096tA78=', 'Content-Length': '0'} 2023-09-22T21:08:44.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000049e125639145a3-00650e025c-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:44.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:44.989 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:44.989 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:44.989 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.989 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.990 DEBUG:boto:Method: GET 2023-09-22T21:08:44.990 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.990 DEBUG:boto:Data: 2023-09-22T21:08:44.990 DEBUG:boto:Headers: {} 2023-09-22T21:08:44.990 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:44.990 DEBUG:boto:Port: 8001 2023-09-22T21:08:44.990 DEBUG:boto:Params: {} 2023-09-22T21:08:44.990 DEBUG:boto:Token: None 2023-09-22T21:08:44.991 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:44 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:44.991 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UNgTa524z+ShGslUAj5GAynaRt4= 2023-09-22T21:08:44.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UNgTa524z+ShGslUAj5GAynaRt4=', 'Content-Length': '0'} 2023-09-22T21:08:44.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a67491fc4468dec5-00650e025c-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.034 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.035 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.035 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.035 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.035 DEBUG:boto:Method: GET 2023-09-22T21:08:45.035 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.035 DEBUG:boto:Data: 2023-09-22T21:08:45.035 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.035 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.036 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.036 DEBUG:boto:Params: {} 2023-09-22T21:08:45.036 DEBUG:boto:Token: None 2023-09-22T21:08:45.036 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.036 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YcwbyCZiJGANYAsmS4glk+zOQ20= 2023-09-22T21:08:45.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YcwbyCZiJGANYAsmS4glk+zOQ20=', 'Content-Length': '0'} 2023-09-22T21:08:45.040 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be541cde8f1945e1-00650e025d-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.086 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:45.087 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:45.087 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.087 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.087 DEBUG:boto:Method: GET 2023-09-22T21:08:45.087 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.087 DEBUG:boto:Data: 2023-09-22T21:08:45.087 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.087 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.087 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.088 DEBUG:boto:Params: {} 2023-09-22T21:08:45.088 DEBUG:boto:Token: None 2023-09-22T21:08:45.088 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/ 2023-09-22T21:08:45.088 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ= 2023-09-22T21:08:45.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ=', 'Content-Length': '0'} 2023-09-22T21:08:45.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d82a27500e68d64-00650e025d-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.094 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:45.094 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:45.094 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:45.094 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.094 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.095 DEBUG:boto:Method: GET 2023-09-22T21:08:45.095 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.095 DEBUG:boto:Data: 2023-09-22T21:08:45.095 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.095 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.095 DEBUG:boto:Params: {} 2023-09-22T21:08:45.095 DEBUG:boto:Token: None 2023-09-22T21:08:45.096 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/ 2023-09-22T21:08:45.096 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ= 2023-09-22T21:08:45.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ=', 'Content-Length': '0'} 2023-09-22T21:08:45.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006572201a396c45d0-00650e025d-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.101 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:45.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:45.102 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.102 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.102 DEBUG:boto:path=/ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.102 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.102 DEBUG:boto:Method: GET 2023-09-22T21:08:45.102 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.102 DEBUG:boto:Data: 2023-09-22T21:08:45.102 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.102 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.103 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.103 DEBUG:boto:Params: {} 2023-09-22T21:08:45.103 DEBUG:boto:Token: None 2023-09-22T21:08:45.103 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.103 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PdG9fJ8a6VEX4F7XgsMrSd4MNME= 2023-09-22T21:08:45.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PdG9fJ8a6VEX4F7XgsMrSd4MNME=', 'Content-Length': '0'} 2023-09-22T21:08:45.106 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007df15b98e9a58ea7-00650e025d-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.106 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.106 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.106 DEBUG:boto:path=/ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.106 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.107 DEBUG:boto:Method: GET 2023-09-22T21:08:45.107 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.107 DEBUG:boto:Data: 2023-09-22T21:08:45.107 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.107 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.107 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.107 DEBUG:boto:Params: {} 2023-09-22T21:08:45.107 DEBUG:boto:Token: None 2023-09-22T21:08:45.108 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.108 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PdG9fJ8a6VEX4F7XgsMrSd4MNME= 2023-09-22T21:08:45.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PdG9fJ8a6VEX4F7XgsMrSd4MNME=', 'Content-Length': '0'} 2023-09-22T21:08:45.111 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df8e87ac0ea3f32a-00650e025d-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.154 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.155 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.155 DEBUG:boto:path=/ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.155 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.155 DEBUG:boto:Method: HEAD 2023-09-22T21:08:45.155 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.156 DEBUG:boto:Data: 2023-09-22T21:08:45.156 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.156 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.156 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.156 DEBUG:boto:Params: {} 2023-09-22T21:08:45.157 DEBUG:boto:Token: None 2023-09-22T21:08:45.157 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.158 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b/vpJynp+/aiqH856pucvPpaO+w= 2023-09-22T21:08:45.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b/vpJynp+/aiqH856pucvPpaO+w=', 'Content-Length': '0'} 2023-09-22T21:08:45.161 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009fcfb0c0797e089c-00650e025d-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.162 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.162 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.162 DEBUG:boto:path=/ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.162 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.162 DEBUG:boto:Method: HEAD 2023-09-22T21:08:45.162 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.162 DEBUG:boto:Data: 2023-09-22T21:08:45.163 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.163 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.163 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.163 DEBUG:boto:Params: {} 2023-09-22T21:08:45.163 DEBUG:boto:Token: None 2023-09-22T21:08:45.163 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.163 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b/vpJynp+/aiqH856pucvPpaO+w= 2023-09-22T21:08:45.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b/vpJynp+/aiqH856pucvPpaO+w=', 'Content-Length': '0'} 2023-09-22T21:08:45.166 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a7a94d1419643cf-00650e025d-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:45.167 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.167 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.167 DEBUG:boto:path=/ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.167 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.167 DEBUG:boto:Method: GET 2023-09-22T21:08:45.167 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.167 DEBUG:boto:Data: 2023-09-22T21:08:45.167 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.168 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.168 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.168 DEBUG:boto:Params: {} 2023-09-22T21:08:45.168 DEBUG:boto:Token: None 2023-09-22T21:08:45.168 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.168 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PdG9fJ8a6VEX4F7XgsMrSd4MNME= 2023-09-22T21:08:45.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PdG9fJ8a6VEX4F7XgsMrSd4MNME=', 'Content-Length': '0'} 2023-09-22T21:08:45.171 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa5d369211e24f25-00650e025d-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.171 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.171 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.171 DEBUG:boto:path=/ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.172 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.172 DEBUG:boto:Method: GET 2023-09-22T21:08:45.172 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.172 DEBUG:boto:Data: 2023-09-22T21:08:45.172 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.172 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.172 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.172 DEBUG:boto:Params: {} 2023-09-22T21:08:45.172 DEBUG:boto:Token: None 2023-09-22T21:08:45.173 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD 2023-09-22T21:08:45.173 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PdG9fJ8a6VEX4F7XgsMrSd4MNME= 2023-09-22T21:08:45.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PdG9fJ8a6VEX4F7XgsMrSd4MNME=', 'Content-Length': '0'} 2023-09-22T21:08:45.176 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'trwlNr-EtLMj-wIVlmYw9dBWQcMpEiD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f832025755c4660-00650e025d-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.218 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:45.219 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:45.219 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.219 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.219 DEBUG:boto:Method: GET 2023-09-22T21:08:45.219 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.219 DEBUG:boto:Data: 2023-09-22T21:08:45.219 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.219 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.219 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.220 DEBUG:boto:Params: {} 2023-09-22T21:08:45.220 DEBUG:boto:Token: None 2023-09-22T21:08:45.220 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/ 2023-09-22T21:08:45.220 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ= 2023-09-22T21:08:45.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ=', 'Content-Length': '0'} 2023-09-22T21:08:45.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a3c6e7a82020210-00650e025d-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.226 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:45.227 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:45.227 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:45.227 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.227 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.227 DEBUG:boto:Method: GET 2023-09-22T21:08:45.227 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.227 DEBUG:boto:Data: 2023-09-22T21:08:45.227 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.228 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.228 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.228 DEBUG:boto:Params: {} 2023-09-22T21:08:45.228 DEBUG:boto:Token: None 2023-09-22T21:08:45.228 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/ 2023-09-22T21:08:45.228 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ= 2023-09-22T21:08:45.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ=', 'Content-Length': '0'} 2023-09-22T21:08:45.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049893b9e7b2ea957-00650e025d-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.233 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:45.233 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.233 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.234 DEBUG:boto:Method: HEAD 2023-09-22T21:08:45.234 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:45.234 DEBUG:boto:Data: 2023-09-22T21:08:45.234 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.234 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.234 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.234 DEBUG:boto:Params: {} 2023-09-22T21:08:45.234 DEBUG:boto:Token: None 2023-09-22T21:08:45.235 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj 2023-09-22T21:08:45.235 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:utN/6pwgoKVf54dKwe+FEkaWhm8= 2023-09-22T21:08:45.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:utN/6pwgoKVf54dKwe+FEkaWhm8=', 'Content-Length': '0'} 2023-09-22T21:08:45.240 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051981577e8f3f6fa-00650e025d-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.240 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.240 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.240 DEBUG:boto:Method: HEAD 2023-09-22T21:08:45.241 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:45.241 DEBUG:boto:Data: 2023-09-22T21:08:45.241 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.241 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.241 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.241 DEBUG:boto:Params: {} 2023-09-22T21:08:45.241 DEBUG:boto:Token: None 2023-09-22T21:08:45.241 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj 2023-09-22T21:08:45.242 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:utN/6pwgoKVf54dKwe+FEkaWhm8= 2023-09-22T21:08:45.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:utN/6pwgoKVf54dKwe+FEkaWhm8=', 'Content-Length': '0'} 2023-09-22T21:08:45.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5d72d264c50da67-00650e025d-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:45.247 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.248 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.248 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.248 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.248 DEBUG:boto:Method: GET 2023-09-22T21:08:45.248 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.248 DEBUG:boto:Data: 2023-09-22T21:08:45.248 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.248 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.248 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.248 DEBUG:boto:Params: {} 2023-09-22T21:08:45.249 DEBUG:boto:Token: None 2023-09-22T21:08:45.249 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.249 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YcwbyCZiJGANYAsmS4glk+zOQ20= 2023-09-22T21:08:45.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YcwbyCZiJGANYAsmS4glk+zOQ20=', 'Content-Length': '0'} 2023-09-22T21:08:45.252 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b33a2674107f3e8f-00650e025d-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.294 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.295 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.295 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.295 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.295 DEBUG:boto:Method: GET 2023-09-22T21:08:45.295 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.295 DEBUG:boto:Data: 2023-09-22T21:08:45.295 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.295 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.295 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.295 DEBUG:boto:Params: {} 2023-09-22T21:08:45.296 DEBUG:boto:Token: None 2023-09-22T21:08:45.296 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.296 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YcwbyCZiJGANYAsmS4glk+zOQ20= 2023-09-22T21:08:45.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YcwbyCZiJGANYAsmS4glk+zOQ20=', 'Content-Length': '0'} 2023-09-22T21:08:45.299 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c3bb1cbb53f34c4-00650e025d-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.342 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:45.343 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:45.343 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.343 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.343 DEBUG:boto:Method: GET 2023-09-22T21:08:45.343 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.343 DEBUG:boto:Data: 2023-09-22T21:08:45.343 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.343 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.343 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.343 DEBUG:boto:Params: {} 2023-09-22T21:08:45.344 DEBUG:boto:Token: None 2023-09-22T21:08:45.344 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/ 2023-09-22T21:08:45.344 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ= 2023-09-22T21:08:45.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ=', 'Content-Length': '0'} 2023-09-22T21:08:45.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1d6fad84a83bda1-00650e025d-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.350 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:45.350 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:45.350 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:45.350 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.351 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.351 DEBUG:boto:Method: GET 2023-09-22T21:08:45.351 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.351 DEBUG:boto:Data: 2023-09-22T21:08:45.351 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.351 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.351 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.351 DEBUG:boto:Params: {} 2023-09-22T21:08:45.351 DEBUG:boto:Token: None 2023-09-22T21:08:45.352 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/ 2023-09-22T21:08:45.352 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ= 2023-09-22T21:08:45.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ=', 'Content-Length': '0'} 2023-09-22T21:08:45.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085537e2bc3b50f5f-00650e025d-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.357 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:45.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:45.357 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.358 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.358 DEBUG:boto:path=/ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.358 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.358 DEBUG:boto:Method: GET 2023-09-22T21:08:45.358 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.358 DEBUG:boto:Data: 2023-09-22T21:08:45.358 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.358 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.358 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.358 DEBUG:boto:Params: {} 2023-09-22T21:08:45.359 DEBUG:boto:Token: None 2023-09-22T21:08:45.359 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.359 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JMftkqKJBrR82htKguNLNoWq0kI= 2023-09-22T21:08:45.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JMftkqKJBrR82htKguNLNoWq0kI=', 'Content-Length': '0'} 2023-09-22T21:08:45.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'KeL89-xXRu815nqON693z3wYbeHRSpw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f51648f83fcc0edd-00650e025d-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.362 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.362 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.362 DEBUG:boto:path=/ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.362 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.362 DEBUG:boto:Method: GET 2023-09-22T21:08:45.362 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.363 DEBUG:boto:Data: 2023-09-22T21:08:45.363 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.363 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.363 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.363 DEBUG:boto:Params: {} 2023-09-22T21:08:45.363 DEBUG:boto:Token: None 2023-09-22T21:08:45.363 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.363 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JMftkqKJBrR82htKguNLNoWq0kI= 2023-09-22T21:08:45.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JMftkqKJBrR82htKguNLNoWq0kI=', 'Content-Length': '0'} 2023-09-22T21:08:45.366 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'KeL89-xXRu815nqON693z3wYbeHRSpw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d4175c75431716a-00650e025d-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.414 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.415 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.415 DEBUG:boto:path=/ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.415 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.415 DEBUG:boto:Method: HEAD 2023-09-22T21:08:45.415 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.415 DEBUG:boto:Data: 2023-09-22T21:08:45.415 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.416 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.416 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.416 DEBUG:boto:Params: {} 2023-09-22T21:08:45.416 DEBUG:boto:Token: None 2023-09-22T21:08:45.416 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.416 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AmUAPqN2qYkRL07nGpGPA7WxecY= 2023-09-22T21:08:45.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AmUAPqN2qYkRL07nGpGPA7WxecY=', 'Content-Length': '0'} 2023-09-22T21:08:45.419 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'KeL89-xXRu815nqON693z3wYbeHRSpw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005fdaaab09a91e39-00650e025d-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.419 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.420 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.420 DEBUG:boto:path=/ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.420 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.420 DEBUG:boto:Method: HEAD 2023-09-22T21:08:45.420 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.420 DEBUG:boto:Data: 2023-09-22T21:08:45.420 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.420 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.421 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.421 DEBUG:boto:Params: {} 2023-09-22T21:08:45.421 DEBUG:boto:Token: None 2023-09-22T21:08:45.421 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.421 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AmUAPqN2qYkRL07nGpGPA7WxecY= 2023-09-22T21:08:45.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AmUAPqN2qYkRL07nGpGPA7WxecY=', 'Content-Length': '0'} 2023-09-22T21:08:45.424 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'KeL89-xXRu815nqON693z3wYbeHRSpw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ad904fdd049c89c-00650e025d-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:45.425 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.425 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.425 DEBUG:boto:path=/ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.425 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.425 DEBUG:boto:Method: GET 2023-09-22T21:08:45.425 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.425 DEBUG:boto:Data: 2023-09-22T21:08:45.425 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.425 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.425 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.426 DEBUG:boto:Params: {} 2023-09-22T21:08:45.426 DEBUG:boto:Token: None 2023-09-22T21:08:45.426 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.426 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JMftkqKJBrR82htKguNLNoWq0kI= 2023-09-22T21:08:45.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JMftkqKJBrR82htKguNLNoWq0kI=', 'Content-Length': '0'} 2023-09-22T21:08:45.429 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'KeL89-xXRu815nqON693z3wYbeHRSpw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f22e2d4ba28a84df-00650e025d-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.429 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.429 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.429 DEBUG:boto:path=/ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.429 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.429 DEBUG:boto:Method: GET 2023-09-22T21:08:45.429 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.429 DEBUG:boto:Data: 2023-09-22T21:08:45.430 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.430 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.430 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.430 DEBUG:boto:Params: {} 2023-09-22T21:08:45.430 DEBUG:boto:Token: None 2023-09-22T21:08:45.430 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=KeL89-xXRu815nqON693z3wYbeHRSpw 2023-09-22T21:08:45.430 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JMftkqKJBrR82htKguNLNoWq0kI= 2023-09-22T21:08:45.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JMftkqKJBrR82htKguNLNoWq0kI=', 'Content-Length': '0'} 2023-09-22T21:08:45.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'KeL89-xXRu815nqON693z3wYbeHRSpw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006588ff49743e1823-00650e025d-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.474 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:45.475 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:45.475 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.475 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.475 DEBUG:boto:Method: GET 2023-09-22T21:08:45.475 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.475 DEBUG:boto:Data: 2023-09-22T21:08:45.475 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.475 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.475 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.476 DEBUG:boto:Params: {} 2023-09-22T21:08:45.476 DEBUG:boto:Token: None 2023-09-22T21:08:45.476 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/ 2023-09-22T21:08:45.476 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ= 2023-09-22T21:08:45.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ=', 'Content-Length': '0'} 2023-09-22T21:08:45.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ce18ac203352a6a-00650e025d-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.482 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:45.482 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:45.483 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:45.483 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.483 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.483 DEBUG:boto:Method: GET 2023-09-22T21:08:45.483 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.483 DEBUG:boto:Data: 2023-09-22T21:08:45.483 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.484 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.484 DEBUG:boto:Params: {} 2023-09-22T21:08:45.484 DEBUG:boto:Token: None 2023-09-22T21:08:45.484 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/ 2023-09-22T21:08:45.484 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ= 2023-09-22T21:08:45.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ=', 'Content-Length': '0'} 2023-09-22T21:08:45.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acfcb42cd141daa3-00650e025d-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.489 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:45.490 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.490 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.490 DEBUG:boto:Method: HEAD 2023-09-22T21:08:45.490 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:45.490 DEBUG:boto:Data: 2023-09-22T21:08:45.490 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.490 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.491 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.491 DEBUG:boto:Params: {} 2023-09-22T21:08:45.491 DEBUG:boto:Token: None 2023-09-22T21:08:45.491 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj 2023-09-22T21:08:45.491 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:utN/6pwgoKVf54dKwe+FEkaWhm8= 2023-09-22T21:08:45.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:utN/6pwgoKVf54dKwe+FEkaWhm8=', 'Content-Length': '0'} 2023-09-22T21:08:45.496 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aee784178786f5e3-00650e025d-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.497 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.497 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.497 DEBUG:boto:Method: HEAD 2023-09-22T21:08:45.497 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:45.497 DEBUG:boto:Data: 2023-09-22T21:08:45.497 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.497 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.497 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.497 DEBUG:boto:Params: {} 2023-09-22T21:08:45.498 DEBUG:boto:Token: None 2023-09-22T21:08:45.498 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj 2023-09-22T21:08:45.498 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:utN/6pwgoKVf54dKwe+FEkaWhm8= 2023-09-22T21:08:45.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:utN/6pwgoKVf54dKwe+FEkaWhm8=', 'Content-Length': '0'} 2023-09-22T21:08:45.503 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007710d8340982dd67-00650e025d-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:45.504 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.504 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.504 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.504 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.504 DEBUG:boto:Method: GET 2023-09-22T21:08:45.505 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.505 DEBUG:boto:Data: 2023-09-22T21:08:45.505 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.505 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.505 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.505 DEBUG:boto:Params: {} 2023-09-22T21:08:45.505 DEBUG:boto:Token: None 2023-09-22T21:08:45.505 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.505 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YcwbyCZiJGANYAsmS4glk+zOQ20= 2023-09-22T21:08:45.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YcwbyCZiJGANYAsmS4glk+zOQ20=', 'Content-Length': '0'} 2023-09-22T21:08:45.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005084f273ce80cab8-00650e025d-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.550 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.551 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.551 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.551 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.551 DEBUG:boto:Method: GET 2023-09-22T21:08:45.551 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.551 DEBUG:boto:Data: 2023-09-22T21:08:45.551 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.551 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.551 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.551 DEBUG:boto:Params: {} 2023-09-22T21:08:45.552 DEBUG:boto:Token: None 2023-09-22T21:08:45.552 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.552 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YcwbyCZiJGANYAsmS4glk+zOQ20= 2023-09-22T21:08:45.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YcwbyCZiJGANYAsmS4glk+zOQ20=', 'Content-Length': '0'} 2023-09-22T21:08:45.555 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000145121db63c77270-00650e025d-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.598 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:45.599 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:45.599 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.599 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.599 DEBUG:boto:Method: GET 2023-09-22T21:08:45.599 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.599 DEBUG:boto:Data: 2023-09-22T21:08:45.599 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.599 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.600 DEBUG:boto:Params: {} 2023-09-22T21:08:45.600 DEBUG:boto:Token: None 2023-09-22T21:08:45.600 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/ 2023-09-22T21:08:45.600 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ= 2023-09-22T21:08:45.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ=', 'Content-Length': '0'} 2023-09-22T21:08:45.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000018471e43690db1c-00650e025d-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.605 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:45.606 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:45.606 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:45.606 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.606 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.606 DEBUG:boto:Method: GET 2023-09-22T21:08:45.606 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.606 DEBUG:boto:Data: 2023-09-22T21:08:45.607 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.607 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.607 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.607 DEBUG:boto:Params: {} 2023-09-22T21:08:45.607 DEBUG:boto:Token: None 2023-09-22T21:08:45.607 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/ 2023-09-22T21:08:45.607 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ= 2023-09-22T21:08:45.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ=', 'Content-Length': '0'} 2023-09-22T21:08:45.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025ed20ba4e4b4df6-00650e025d-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.613 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:45.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:45.613 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.613 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.614 DEBUG:boto:path=/ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.614 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.614 DEBUG:boto:Method: GET 2023-09-22T21:08:45.614 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.614 DEBUG:boto:Data: 2023-09-22T21:08:45.614 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.614 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.614 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.614 DEBUG:boto:Params: {} 2023-09-22T21:08:45.615 DEBUG:boto:Token: None 2023-09-22T21:08:45.615 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.615 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ilVm9mBaV/ypsCMVV9DznhT7GOM= 2023-09-22T21:08:45.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ilVm9mBaV/ypsCMVV9DznhT7GOM=', 'Content-Length': '0'} 2023-09-22T21:08:45.618 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'nkf1H6kXofD14zadwSPLBP-vHM76qTM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be6f53f80428fe36-00650e025d-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.658 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.658 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.659 DEBUG:boto:path=/ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.659 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.659 DEBUG:boto:Method: GET 2023-09-22T21:08:45.659 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.659 DEBUG:boto:Data: 2023-09-22T21:08:45.659 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.659 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.659 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.659 DEBUG:boto:Params: {} 2023-09-22T21:08:45.659 DEBUG:boto:Token: None 2023-09-22T21:08:45.660 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.660 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ilVm9mBaV/ypsCMVV9DznhT7GOM= 2023-09-22T21:08:45.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ilVm9mBaV/ypsCMVV9DznhT7GOM=', 'Content-Length': '0'} 2023-09-22T21:08:45.663 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'nkf1H6kXofD14zadwSPLBP-vHM76qTM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7f1c4c10395005e-00650e025d-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.663 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.663 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.663 DEBUG:boto:path=/ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.663 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.663 DEBUG:boto:Method: HEAD 2023-09-22T21:08:45.664 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.664 DEBUG:boto:Data: 2023-09-22T21:08:45.664 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.664 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.664 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.664 DEBUG:boto:Params: {} 2023-09-22T21:08:45.664 DEBUG:boto:Token: None 2023-09-22T21:08:45.664 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.664 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8GqefCC0VZlrbZ5X5X23ZFux7cY= 2023-09-22T21:08:45.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8GqefCC0VZlrbZ5X5X23ZFux7cY=', 'Content-Length': '0'} 2023-09-22T21:08:45.667 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'nkf1H6kXofD14zadwSPLBP-vHM76qTM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004308a6d2cddf3bb3-00650e025d-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.667 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.667 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.668 DEBUG:boto:path=/ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.668 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.668 DEBUG:boto:Method: HEAD 2023-09-22T21:08:45.668 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.668 DEBUG:boto:Data: 2023-09-22T21:08:45.668 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.668 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.668 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.668 DEBUG:boto:Params: {} 2023-09-22T21:08:45.669 DEBUG:boto:Token: None 2023-09-22T21:08:45.669 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.669 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8GqefCC0VZlrbZ5X5X23ZFux7cY= 2023-09-22T21:08:45.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8GqefCC0VZlrbZ5X5X23ZFux7cY=', 'Content-Length': '0'} 2023-09-22T21:08:45.671 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'nkf1H6kXofD14zadwSPLBP-vHM76qTM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b82101379d2c8a99-00650e025d-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:45.672 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.672 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.672 DEBUG:boto:path=/ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.672 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.672 DEBUG:boto:Method: GET 2023-09-22T21:08:45.673 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.673 DEBUG:boto:Data: 2023-09-22T21:08:45.673 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.673 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.673 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.673 DEBUG:boto:Params: {} 2023-09-22T21:08:45.673 DEBUG:boto:Token: None 2023-09-22T21:08:45.673 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.674 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ilVm9mBaV/ypsCMVV9DznhT7GOM= 2023-09-22T21:08:45.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ilVm9mBaV/ypsCMVV9DznhT7GOM=', 'Content-Length': '0'} 2023-09-22T21:08:45.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'nkf1H6kXofD14zadwSPLBP-vHM76qTM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c57903c301eb3a69-00650e025d-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.718 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.718 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.719 DEBUG:boto:path=/ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.719 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.719 DEBUG:boto:Method: GET 2023-09-22T21:08:45.719 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.719 DEBUG:boto:Data: 2023-09-22T21:08:45.719 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.719 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.719 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.719 DEBUG:boto:Params: {} 2023-09-22T21:08:45.719 DEBUG:boto:Token: None 2023-09-22T21:08:45.720 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=nkf1H6kXofD14zadwSPLBP-vHM76qTM 2023-09-22T21:08:45.720 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ilVm9mBaV/ypsCMVV9DznhT7GOM= 2023-09-22T21:08:45.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ilVm9mBaV/ypsCMVV9DznhT7GOM=', 'Content-Length': '0'} 2023-09-22T21:08:45.722 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'nkf1H6kXofD14zadwSPLBP-vHM76qTM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000513b5eea5cbcc934-00650e025d-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.723 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:45.723 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:45.723 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.723 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.723 DEBUG:boto:Method: GET 2023-09-22T21:08:45.723 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.724 DEBUG:boto:Data: 2023-09-22T21:08:45.724 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.724 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.724 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.724 DEBUG:boto:Params: {} 2023-09-22T21:08:45.724 DEBUG:boto:Token: None 2023-09-22T21:08:45.724 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/ 2023-09-22T21:08:45.724 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ= 2023-09-22T21:08:45.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ=', 'Content-Length': '0'} 2023-09-22T21:08:45.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c214c902bce1b9c9-00650e025d-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.731 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:45.731 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:45.731 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:45.731 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.731 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.732 DEBUG:boto:Method: GET 2023-09-22T21:08:45.732 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.732 DEBUG:boto:Data: 2023-09-22T21:08:45.732 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.732 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.732 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.732 DEBUG:boto:Params: {} 2023-09-22T21:08:45.732 DEBUG:boto:Token: None 2023-09-22T21:08:45.732 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/ 2023-09-22T21:08:45.733 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ= 2023-09-22T21:08:45.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ=', 'Content-Length': '0'} 2023-09-22T21:08:45.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000973fe277695d03ba-00650e025d-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.782 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:45.783 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.783 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.783 DEBUG:boto:Method: HEAD 2023-09-22T21:08:45.783 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:45.783 DEBUG:boto:Data: 2023-09-22T21:08:45.783 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.783 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.784 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.784 DEBUG:boto:Params: {} 2023-09-22T21:08:45.784 DEBUG:boto:Token: None 2023-09-22T21:08:45.784 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj 2023-09-22T21:08:45.784 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:utN/6pwgoKVf54dKwe+FEkaWhm8= 2023-09-22T21:08:45.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:utN/6pwgoKVf54dKwe+FEkaWhm8=', 'Content-Length': '0'} 2023-09-22T21:08:45.791 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000212dccd8071063c4-00650e025d-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.791 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.791 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.791 DEBUG:boto:Method: HEAD 2023-09-22T21:08:45.791 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:45.792 DEBUG:boto:Data: 2023-09-22T21:08:45.792 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.792 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.792 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.792 DEBUG:boto:Params: {} 2023-09-22T21:08:45.792 DEBUG:boto:Token: None 2023-09-22T21:08:45.792 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj 2023-09-22T21:08:45.792 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:utN/6pwgoKVf54dKwe+FEkaWhm8= 2023-09-22T21:08:45.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:utN/6pwgoKVf54dKwe+FEkaWhm8=', 'Content-Length': '0'} 2023-09-22T21:08:45.798 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff5cf3686170adc7-00650e025d-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:45.799 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.799 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.799 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.799 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.799 DEBUG:boto:Method: GET 2023-09-22T21:08:45.799 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.799 DEBUG:boto:Data: 2023-09-22T21:08:45.800 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.800 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.800 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.800 DEBUG:boto:Params: {} 2023-09-22T21:08:45.800 DEBUG:boto:Token: None 2023-09-22T21:08:45.800 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.800 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YcwbyCZiJGANYAsmS4glk+zOQ20= 2023-09-22T21:08:45.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YcwbyCZiJGANYAsmS4glk+zOQ20=', 'Content-Length': '0'} 2023-09-22T21:08:45.803 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f1f02b79d5cbbd3-00650e025d-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.803 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.804 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.804 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.804 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.804 DEBUG:boto:Method: GET 2023-09-22T21:08:45.804 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.804 DEBUG:boto:Data: 2023-09-22T21:08:45.804 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.804 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.804 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.804 DEBUG:boto:Params: {} 2023-09-22T21:08:45.805 DEBUG:boto:Token: None 2023-09-22T21:08:45.805 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:45.805 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YcwbyCZiJGANYAsmS4glk+zOQ20= 2023-09-22T21:08:45.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YcwbyCZiJGANYAsmS4glk+zOQ20=', 'Content-Length': '0'} 2023-09-22T21:08:45.808 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8488605618e584e-00650e025d-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.851 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:45.851 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:45.851 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.851 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.851 DEBUG:boto:Method: GET 2023-09-22T21:08:45.851 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.851 DEBUG:boto:Data: 2023-09-22T21:08:45.851 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.852 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.852 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.852 DEBUG:boto:Params: {} 2023-09-22T21:08:45.852 DEBUG:boto:Token: None 2023-09-22T21:08:45.852 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/ 2023-09-22T21:08:45.852 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ= 2023-09-22T21:08:45.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ=', 'Content-Length': '0'} 2023-09-22T21:08:45.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec82a27c36ec3093-00650e025d-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.858 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:45.858 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:45.858 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:45.858 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.858 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.859 DEBUG:boto:Method: GET 2023-09-22T21:08:45.859 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:45.859 DEBUG:boto:Data: 2023-09-22T21:08:45.859 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.859 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.859 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.859 DEBUG:boto:Params: {} 2023-09-22T21:08:45.859 DEBUG:boto:Token: None 2023-09-22T21:08:45.859 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/ 2023-09-22T21:08:45.860 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ= 2023-09-22T21:08:45.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aEjS1QCO4K6/rb/kELlZ7yWQDrQ=', 'Content-Length': '0'} 2023-09-22T21:08:45.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007103c75a9a1cabd3-00650e025d-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.906 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:45.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:45.907 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.907 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.907 DEBUG:boto:path=/ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:45.907 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:45.908 DEBUG:boto:Method: GET 2023-09-22T21:08:45.908 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:45.908 DEBUG:boto:Data: 2023-09-22T21:08:45.908 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.908 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.908 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.908 DEBUG:boto:Params: {} 2023-09-22T21:08:45.908 DEBUG:boto:Token: None 2023-09-22T21:08:45.908 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:45.909 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TYZTGvLrZ6y6vN5kcf0XSYf+B08= 2023-09-22T21:08:45.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TYZTGvLrZ6y6vN5kcf0XSYf+B08=', 'Content-Length': '0'} 2023-09-22T21:08:45.912 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', '0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df132e204c296c33-00650e025d-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.954 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.954 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.955 DEBUG:boto:path=/ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:45.955 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:45.955 DEBUG:boto:Method: GET 2023-09-22T21:08:45.955 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:45.955 DEBUG:boto:Data: 2023-09-22T21:08:45.955 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.955 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.955 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.955 DEBUG:boto:Params: {} 2023-09-22T21:08:45.956 DEBUG:boto:Token: None 2023-09-22T21:08:45.956 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:45.956 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TYZTGvLrZ6y6vN5kcf0XSYf+B08= 2023-09-22T21:08:45.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TYZTGvLrZ6y6vN5kcf0XSYf+B08=', 'Content-Length': '0'} 2023-09-22T21:08:45.959 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', '0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff95d6af0e2fb588-00650e025d-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.959 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.959 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.959 DEBUG:boto:path=/ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:45.959 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:45.960 DEBUG:boto:Method: HEAD 2023-09-22T21:08:45.960 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:45.960 DEBUG:boto:Data: 2023-09-22T21:08:45.960 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.960 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.960 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.960 DEBUG:boto:Params: {} 2023-09-22T21:08:45.960 DEBUG:boto:Token: None 2023-09-22T21:08:45.960 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:45.961 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qd0NlCOaOU2ZfR5lxR6DQyK0Rmw= 2023-09-22T21:08:45.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qd0NlCOaOU2ZfR5lxR6DQyK0Rmw=', 'Content-Length': '0'} 2023-09-22T21:08:45.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', '0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8d14d67c72006f3-00650e025d-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.964 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.964 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.964 DEBUG:boto:path=/ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:45.964 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:45.964 DEBUG:boto:Method: HEAD 2023-09-22T21:08:45.964 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:45.964 DEBUG:boto:Data: 2023-09-22T21:08:45.964 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.964 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:45.965 DEBUG:boto:Port: 8000 2023-09-22T21:08:45.965 DEBUG:boto:Params: {} 2023-09-22T21:08:45.965 DEBUG:boto:Token: None 2023-09-22T21:08:45.965 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:45.965 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qd0NlCOaOU2ZfR5lxR6DQyK0Rmw= 2023-09-22T21:08:45.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qd0NlCOaOU2ZfR5lxR6DQyK0Rmw=', 'Content-Length': '0'} 2023-09-22T21:08:45.968 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', '0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000457e31e52a03e912-00650e025d-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:45.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:45.968 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:45.968 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:45.968 DEBUG:boto:path=/ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:45.968 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:45.969 DEBUG:boto:Method: GET 2023-09-22T21:08:45.969 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:45.969 DEBUG:boto:Data: 2023-09-22T21:08:45.969 DEBUG:boto:Headers: {} 2023-09-22T21:08:45.969 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:45.969 DEBUG:boto:Port: 8001 2023-09-22T21:08:45.969 DEBUG:boto:Params: {} 2023-09-22T21:08:45.969 DEBUG:boto:Token: None 2023-09-22T21:08:45.969 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:45 GMT /ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:45.969 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TYZTGvLrZ6y6vN5kcf0XSYf+B08= 2023-09-22T21:08:45.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TYZTGvLrZ6y6vN5kcf0XSYf+B08=', 'Content-Length': '0'} 2023-09-22T21:08:45.972 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', '0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055776a5acff2ad91-00650e025d-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.014 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:46.015 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:46.015 DEBUG:boto:path=/ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:46.015 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:46.015 DEBUG:boto:Method: GET 2023-09-22T21:08:46.015 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:46.015 DEBUG:boto:Data: 2023-09-22T21:08:46.015 DEBUG:boto:Headers: {} 2023-09-22T21:08:46.015 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:46.015 DEBUG:boto:Port: 8000 2023-09-22T21:08:46.016 DEBUG:boto:Params: {} 2023-09-22T21:08:46.016 DEBUG:boto:Token: None 2023-09-22T21:08:46.016 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:46 GMT /ffzdqu-13/obj?versionId=0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2 2023-09-22T21:08:46.016 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ie/R7uCLuylrf15x9Hgt0iuoaYI= 2023-09-22T21:08:46.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ie/R7uCLuylrf15x9Hgt0iuoaYI=', 'Content-Length': '0'} 2023-09-22T21:08:46.019 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', '0fF1FMqyAFa5LbZHJipKNHRKyHWq6v2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d160155f30b931b6-00650e025e-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.019 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:46.020 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:46.020 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:46.020 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:46.020 DEBUG:boto:Method: GET 2023-09-22T21:08:46.020 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:46.020 DEBUG:boto:Data: 2023-09-22T21:08:46.020 DEBUG:boto:Headers: {} 2023-09-22T21:08:46.020 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:46.020 DEBUG:boto:Port: 8001 2023-09-22T21:08:46.020 DEBUG:boto:Params: {} 2023-09-22T21:08:46.021 DEBUG:boto:Token: None 2023-09-22T21:08:46.021 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:46 GMT /ffzdqu-13/ 2023-09-22T21:08:46.021 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9Wg4uzZIKiRO0ZT8WbJYE8Hg2Fs= 2023-09-22T21:08:46.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9Wg4uzZIKiRO0ZT8WbJYE8Hg2Fs=', 'Content-Length': '0'} 2023-09-22T21:08:46.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087aa8dbdfdd7899f-00650e025e-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.026 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:46.027 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:46.027 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:46.027 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:46.027 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:46.027 DEBUG:boto:Method: GET 2023-09-22T21:08:46.028 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:46.028 DEBUG:boto:Data: 2023-09-22T21:08:46.028 DEBUG:boto:Headers: {} 2023-09-22T21:08:46.028 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:46.028 DEBUG:boto:Port: 8000 2023-09-22T21:08:46.028 DEBUG:boto:Params: {} 2023-09-22T21:08:46.028 DEBUG:boto:Token: None 2023-09-22T21:08:46.028 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:46 GMT /ffzdqu-13/ 2023-09-22T21:08:46.028 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9Wg4uzZIKiRO0ZT8WbJYE8Hg2Fs= 2023-09-22T21:08:46.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9Wg4uzZIKiRO0ZT8WbJYE8Hg2Fs=', 'Content-Length': '0'} 2023-09-22T21:08:46.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1709755e8008aa6-00650e025e-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.034 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:46.034 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:46.034 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:46.034 DEBUG:boto:Method: HEAD 2023-09-22T21:08:46.034 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:46.034 DEBUG:boto:Data: 2023-09-22T21:08:46.035 DEBUG:boto:Headers: {} 2023-09-22T21:08:46.035 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:46.035 DEBUG:boto:Port: 8001 2023-09-22T21:08:46.035 DEBUG:boto:Params: {} 2023-09-22T21:08:46.035 DEBUG:boto:Token: None 2023-09-22T21:08:46.035 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:46 GMT /ffzdqu-13/obj 2023-09-22T21:08:46.035 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZCLIcGUHnM3SQxdlD/KHIyQkbD4= 2023-09-22T21:08:46.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZCLIcGUHnM3SQxdlD/KHIyQkbD4=', 'Content-Length': '0'} 2023-09-22T21:08:46.040 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc5072bc99faaffc-00650e025e-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.041 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:46.041 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:46.041 DEBUG:boto:Method: HEAD 2023-09-22T21:08:46.041 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:46.041 DEBUG:boto:Data: 2023-09-22T21:08:46.041 DEBUG:boto:Headers: {} 2023-09-22T21:08:46.041 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:46.041 DEBUG:boto:Port: 8000 2023-09-22T21:08:46.041 DEBUG:boto:Params: {} 2023-09-22T21:08:46.042 DEBUG:boto:Token: None 2023-09-22T21:08:46.042 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:46 GMT /ffzdqu-13/obj 2023-09-22T21:08:46.042 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZCLIcGUHnM3SQxdlD/KHIyQkbD4= 2023-09-22T21:08:46.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZCLIcGUHnM3SQxdlD/KHIyQkbD4=', 'Content-Length': '0'} 2023-09-22T21:08:46.047 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000083c27faa9d8f797a-00650e025e-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:46.048 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:46.048 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:46.048 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:46.048 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:46.048 DEBUG:boto:Method: GET 2023-09-22T21:08:46.048 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:46.049 DEBUG:boto:Data: 2023-09-22T21:08:46.049 DEBUG:boto:Headers: {} 2023-09-22T21:08:46.049 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:46.049 DEBUG:boto:Port: 8001 2023-09-22T21:08:46.049 DEBUG:boto:Params: {} 2023-09-22T21:08:46.049 DEBUG:boto:Token: None 2023-09-22T21:08:46.049 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:46 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:46.049 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P39+Dxy5bTnAqYNTMMki4z6hGfk= 2023-09-22T21:08:46.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P39+Dxy5bTnAqYNTMMki4z6hGfk=', 'Content-Length': '0'} 2023-09-22T21:08:46.052 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006350858be99eaaef-00650e025e-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.094 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:46.094 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:46.095 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:46.095 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:46.095 DEBUG:boto:Method: GET 2023-09-22T21:08:46.095 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:46.095 DEBUG:boto:Data: 2023-09-22T21:08:46.095 DEBUG:boto:Headers: {} 2023-09-22T21:08:46.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:46.095 DEBUG:boto:Port: 8000 2023-09-22T21:08:46.095 DEBUG:boto:Params: {} 2023-09-22T21:08:46.096 DEBUG:boto:Token: None 2023-09-22T21:08:46.096 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:46 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:46.096 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P39+Dxy5bTnAqYNTMMki4z6hGfk= 2023-09-22T21:08:46.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P39+Dxy5bTnAqYNTMMki4z6hGfk=', 'Content-Length': '0'} 2023-09-22T21:08:46.099 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ebe769099ba7fd5b-00650e025e-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.146 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:46.147 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:46.147 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:46.147 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:46.147 DEBUG:boto:Method: GET 2023-09-22T21:08:46.147 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:46.147 DEBUG:boto:Data: 2023-09-22T21:08:46.147 DEBUG:boto:Headers: {} 2023-09-22T21:08:46.148 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:46.148 DEBUG:boto:Port: 8001 2023-09-22T21:08:46.149 DEBUG:boto:Params: {} 2023-09-22T21:08:46.149 DEBUG:boto:Token: None 2023-09-22T21:08:46.149 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:46 GMT /ffzdqu-13/ 2023-09-22T21:08:46.149 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9Wg4uzZIKiRO0ZT8WbJYE8Hg2Fs= 2023-09-22T21:08:46.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9Wg4uzZIKiRO0ZT8WbJYE8Hg2Fs=', 'Content-Length': '0'} 2023-09-22T21:08:46.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dc9e83dee3642b8-00650e025e-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.155 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:46.156 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:46.156 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:46.156 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:46.156 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:46.156 DEBUG:boto:Method: GET 2023-09-22T21:08:46.156 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:46.156 DEBUG:boto:Data: 2023-09-22T21:08:46.157 DEBUG:boto:Headers: {} 2023-09-22T21:08:46.157 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:46.157 DEBUG:boto:Port: 8000 2023-09-22T21:08:46.157 DEBUG:boto:Params: {} 2023-09-22T21:08:46.157 DEBUG:boto:Token: None 2023-09-22T21:08:46.157 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:46 GMT /ffzdqu-13/ 2023-09-22T21:08:46.157 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9Wg4uzZIKiRO0ZT8WbJYE8Hg2Fs= 2023-09-22T21:08:46.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9Wg4uzZIKiRO0ZT8WbJYE8Hg2Fs=', 'Content-Length': '0'} 2023-09-22T21:08:46.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b39191ece5c33ae3-00650e025e-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.163 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:46.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:46.163 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:46.163 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:46.164 DEBUG:boto:path=/ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.164 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.164 DEBUG:boto:Method: GET 2023-09-22T21:08:46.164 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.164 DEBUG:boto:Data: 2023-09-22T21:08:46.164 DEBUG:boto:Headers: {} 2023-09-22T21:08:46.164 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:46.164 DEBUG:boto:Port: 8001 2023-09-22T21:08:46.165 DEBUG:boto:Params: {} 2023-09-22T21:08:46.165 DEBUG:boto:Token: None 2023-09-22T21:08:46.165 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:46 GMT /ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.165 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OE/bJGZ30msJRCNqOg/cdajYd+Q= 2023-09-22T21:08:46.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OE/bJGZ30msJRCNqOg/cdajYd+Q=', 'Content-Length': '0'} 2023-09-22T21:08:46.168 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', '6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060fa4b721cf6d0ab-00650e025e-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.168 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:46.168 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:46.168 DEBUG:boto:path=/ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.168 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.169 DEBUG:boto:Method: GET 2023-09-22T21:08:46.169 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.169 DEBUG:boto:Data: 2023-09-22T21:08:46.169 DEBUG:boto:Headers: {} 2023-09-22T21:08:46.169 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:46.169 DEBUG:boto:Port: 8000 2023-09-22T21:08:46.169 DEBUG:boto:Params: {} 2023-09-22T21:08:46.169 DEBUG:boto:Token: None 2023-09-22T21:08:46.170 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:46 GMT /ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.170 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OE/bJGZ30msJRCNqOg/cdajYd+Q= 2023-09-22T21:08:46.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OE/bJGZ30msJRCNqOg/cdajYd+Q=', 'Content-Length': '0'} 2023-09-22T21:08:46.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', '6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e31b8eae3ffe0362-00650e025e-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.218 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:46.219 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:46.219 DEBUG:boto:path=/ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.219 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.219 DEBUG:boto:Method: HEAD 2023-09-22T21:08:46.219 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.219 DEBUG:boto:Data: 2023-09-22T21:08:46.219 DEBUG:boto:Headers: {} 2023-09-22T21:08:46.219 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:46.219 DEBUG:boto:Port: 8001 2023-09-22T21:08:46.220 DEBUG:boto:Params: {} 2023-09-22T21:08:46.220 DEBUG:boto:Token: None 2023-09-22T21:08:46.220 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:46 GMT /ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.220 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LwoPl5Mri166HB6cVJE+3FRiSPw= 2023-09-22T21:08:46.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LwoPl5Mri166HB6cVJE+3FRiSPw=', 'Content-Length': '0'} 2023-09-22T21:08:46.223 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', '6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c119ec721596134-00650e025e-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.224 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:46.224 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:46.224 DEBUG:boto:path=/ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.224 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.224 DEBUG:boto:Method: HEAD 2023-09-22T21:08:46.225 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.225 DEBUG:boto:Data: 2023-09-22T21:08:46.225 DEBUG:boto:Headers: {} 2023-09-22T21:08:46.225 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:46.225 DEBUG:boto:Port: 8000 2023-09-22T21:08:46.225 DEBUG:boto:Params: {} 2023-09-22T21:08:46.225 DEBUG:boto:Token: None 2023-09-22T21:08:46.225 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:46 GMT /ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.226 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LwoPl5Mri166HB6cVJE+3FRiSPw= 2023-09-22T21:08:46.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LwoPl5Mri166HB6cVJE+3FRiSPw=', 'Content-Length': '0'} 2023-09-22T21:08:46.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', '6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066b72d446d1ed4e7-00650e025e-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:46.229 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:46.229 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:46.229 DEBUG:boto:path=/ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.229 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.229 DEBUG:boto:Method: GET 2023-09-22T21:08:46.229 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.229 DEBUG:boto:Data: 2023-09-22T21:08:46.230 DEBUG:boto:Headers: {} 2023-09-22T21:08:46.230 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:46.230 DEBUG:boto:Port: 8001 2023-09-22T21:08:46.230 DEBUG:boto:Params: {} 2023-09-22T21:08:46.230 DEBUG:boto:Token: None 2023-09-22T21:08:46.230 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:46 GMT /ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.230 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OE/bJGZ30msJRCNqOg/cdajYd+Q= 2023-09-22T21:08:46.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OE/bJGZ30msJRCNqOg/cdajYd+Q=', 'Content-Length': '0'} 2023-09-22T21:08:46.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', '6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000797b632141669109-00650e025e-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.234 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:46.234 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:46.234 DEBUG:boto:path=/ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.234 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.234 DEBUG:boto:Method: GET 2023-09-22T21:08:46.234 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.234 DEBUG:boto:Data: 2023-09-22T21:08:46.235 DEBUG:boto:Headers: {} 2023-09-22T21:08:46.235 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:46.235 DEBUG:boto:Port: 8000 2023-09-22T21:08:46.235 DEBUG:boto:Params: {} 2023-09-22T21:08:46.235 DEBUG:boto:Token: None 2023-09-22T21:08:46.235 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:46 GMT /ffzdqu-13/obj?versionId=6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp 2023-09-22T21:08:46.235 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OE/bJGZ30msJRCNqOg/cdajYd+Q= 2023-09-22T21:08:46.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OE/bJGZ30msJRCNqOg/cdajYd+Q=', 'Content-Length': '0'} 2023-09-22T21:08:46.238 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', '6-gODrZ6LdnRGbTNPrF.c3yj-YHJffp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053b0d0c42bf20118-00650e025e-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.278 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:46.279 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:46.279 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:46.279 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:46.279 DEBUG:boto:Method: GET 2023-09-22T21:08:46.279 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:46.279 DEBUG:boto:Data: 2023-09-22T21:08:46.279 DEBUG:boto:Headers: {} 2023-09-22T21:08:46.279 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:46.279 DEBUG:boto:Port: 8001 2023-09-22T21:08:46.280 DEBUG:boto:Params: {} 2023-09-22T21:08:46.280 DEBUG:boto:Token: None 2023-09-22T21:08:46.280 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:46 GMT /ffzdqu-13/ 2023-09-22T21:08:46.280 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9Wg4uzZIKiRO0ZT8WbJYE8Hg2Fs= 2023-09-22T21:08:46.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9Wg4uzZIKiRO0ZT8WbJYE8Hg2Fs=', 'Content-Length': '0'} 2023-09-22T21:08:46.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bcc313dc639b68e-00650e025e-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.286 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:46.287 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:46.287 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:46.287 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:46.287 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:46.287 DEBUG:boto:Method: GET 2023-09-22T21:08:46.287 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:46.287 DEBUG:boto:Data: 2023-09-22T21:08:46.287 DEBUG:boto:Headers: {} 2023-09-22T21:08:46.288 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:46.288 DEBUG:boto:Port: 8000 2023-09-22T21:08:46.288 DEBUG:boto:Params: {} 2023-09-22T21:08:46.288 DEBUG:boto:Token: None 2023-09-22T21:08:46.288 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:46 GMT /ffzdqu-13/ 2023-09-22T21:08:46.288 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9Wg4uzZIKiRO0ZT8WbJYE8Hg2Fs= 2023-09-22T21:08:46.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9Wg4uzZIKiRO0ZT8WbJYE8Hg2Fs=', 'Content-Length': '0'} 2023-09-22T21:08:46.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f434d941a2771e0f-00650e025e-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.294 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:46.294 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:46.295 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:46.295 DEBUG:boto:Method: HEAD 2023-09-22T21:08:46.295 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:46.295 DEBUG:boto:Data: 2023-09-22T21:08:46.295 DEBUG:boto:Headers: {} 2023-09-22T21:08:46.295 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:46.295 DEBUG:boto:Port: 8001 2023-09-22T21:08:46.295 DEBUG:boto:Params: {} 2023-09-22T21:08:46.296 DEBUG:boto:Token: None 2023-09-22T21:08:46.296 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:46 GMT /ffzdqu-13/obj 2023-09-22T21:08:46.296 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZCLIcGUHnM3SQxdlD/KHIyQkbD4= 2023-09-22T21:08:46.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZCLIcGUHnM3SQxdlD/KHIyQkbD4=', 'Content-Length': '0'} 2023-09-22T21:08:46.302 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ea77cf6d15d47f6-00650e025e-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.302 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:46.303 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:46.303 DEBUG:boto:Method: HEAD 2023-09-22T21:08:46.303 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:46.303 DEBUG:boto:Data: 2023-09-22T21:08:46.303 DEBUG:boto:Headers: {} 2023-09-22T21:08:46.303 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:46.303 DEBUG:boto:Port: 8000 2023-09-22T21:08:46.303 DEBUG:boto:Params: {} 2023-09-22T21:08:46.304 DEBUG:boto:Token: None 2023-09-22T21:08:46.304 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:46 GMT /ffzdqu-13/obj 2023-09-22T21:08:46.304 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZCLIcGUHnM3SQxdlD/KHIyQkbD4= 2023-09-22T21:08:46.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZCLIcGUHnM3SQxdlD/KHIyQkbD4=', 'Content-Length': '0'} 2023-09-22T21:08:46.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a127312aa37fa846-00650e025e-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:46.310 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:46.310 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:46.310 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:46.310 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:46.310 DEBUG:boto:Method: GET 2023-09-22T21:08:46.310 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:46.310 DEBUG:boto:Data: 2023-09-22T21:08:46.311 DEBUG:boto:Headers: {} 2023-09-22T21:08:46.311 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:46.311 DEBUG:boto:Port: 8001 2023-09-22T21:08:46.311 DEBUG:boto:Params: {} 2023-09-22T21:08:46.311 DEBUG:boto:Token: None 2023-09-22T21:08:46.311 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:46 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:46.311 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P39+Dxy5bTnAqYNTMMki4z6hGfk= 2023-09-22T21:08:46.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P39+Dxy5bTnAqYNTMMki4z6hGfk=', 'Content-Length': '0'} 2023-09-22T21:08:46.314 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057c73ee354c7f8e5-00650e025e-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.354 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:46.355 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:46.355 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:46.355 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:46.355 DEBUG:boto:Method: GET 2023-09-22T21:08:46.355 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:46.355 DEBUG:boto:Data: 2023-09-22T21:08:46.355 DEBUG:boto:Headers: {} 2023-09-22T21:08:46.355 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:46.356 DEBUG:boto:Port: 8000 2023-09-22T21:08:46.356 DEBUG:boto:Params: {} 2023-09-22T21:08:46.356 DEBUG:boto:Token: None 2023-09-22T21:08:46.356 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:46 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:46.356 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P39+Dxy5bTnAqYNTMMki4z6hGfk= 2023-09-22T21:08:46.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P39+Dxy5bTnAqYNTMMki4z6hGfk=', 'Content-Length': '0'} 2023-09-22T21:08:46.359 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd57835f038865dd-00650e025e-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:46.402 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:47.099 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:47.099 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.099 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.099 DEBUG:boto:Method: GET 2023-09-22T21:08:47.099 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.099 DEBUG:boto:Data: 2023-09-22T21:08:47.099 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.099 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:47.099 DEBUG:boto:Port: 8001 2023-09-22T21:08:47.100 DEBUG:boto:Params: {} 2023-09-22T21:08:47.100 DEBUG:boto:Token: None 2023-09-22T21:08:47.100 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/ 2023-09-22T21:08:47.100 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q= 2023-09-22T21:08:47.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q=', 'Content-Length': '0'} 2023-09-22T21:08:47.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000977312eda9f5a9b0-00650e025f-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.105 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:47.106 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:47.106 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:47.106 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.106 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.106 DEBUG:boto:Method: GET 2023-09-22T21:08:47.107 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.107 DEBUG:boto:Data: 2023-09-22T21:08:47.107 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.107 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:47.107 DEBUG:boto:Port: 8000 2023-09-22T21:08:47.107 DEBUG:boto:Params: {} 2023-09-22T21:08:47.107 DEBUG:boto:Token: None 2023-09-22T21:08:47.107 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/ 2023-09-22T21:08:47.107 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q= 2023-09-22T21:08:47.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q=', 'Content-Length': '0'} 2023-09-22T21:08:47.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000694b19f75e00dc4a-00650e025f-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.112 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:47.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:47.113 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.113 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.113 DEBUG:boto:path=/ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.113 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.113 DEBUG:boto:Method: GET 2023-09-22T21:08:47.113 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.114 DEBUG:boto:Data: 2023-09-22T21:08:47.114 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.114 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:47.114 DEBUG:boto:Port: 8001 2023-09-22T21:08:47.114 DEBUG:boto:Params: {} 2023-09-22T21:08:47.114 DEBUG:boto:Token: None 2023-09-22T21:08:47.114 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.114 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VTO10T4hFYtHUb7k1kriFKsSHWI= 2023-09-22T21:08:47.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VTO10T4hFYtHUb7k1kriFKsSHWI=', 'Content-Length': '0'} 2023-09-22T21:08:47.118 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'X5s9z0mNXbqm37pFm1vpfPexuZJVGd0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc5bbe8425065252-00650e025f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.118 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.118 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.118 DEBUG:boto:path=/ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.118 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.118 DEBUG:boto:Method: GET 2023-09-22T21:08:47.119 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.119 DEBUG:boto:Data: 2023-09-22T21:08:47.119 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.119 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:47.119 DEBUG:boto:Port: 8000 2023-09-22T21:08:47.119 DEBUG:boto:Params: {} 2023-09-22T21:08:47.119 DEBUG:boto:Token: None 2023-09-22T21:08:47.119 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.119 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VTO10T4hFYtHUb7k1kriFKsSHWI= 2023-09-22T21:08:47.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VTO10T4hFYtHUb7k1kriFKsSHWI=', 'Content-Length': '0'} 2023-09-22T21:08:47.122 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'X5s9z0mNXbqm37pFm1vpfPexuZJVGd0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036cac0ec175543b7-00650e025f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.122 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.123 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.123 DEBUG:boto:path=/ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.123 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.123 DEBUG:boto:Method: HEAD 2023-09-22T21:08:47.123 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.123 DEBUG:boto:Data: 2023-09-22T21:08:47.123 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.123 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:47.123 DEBUG:boto:Port: 8001 2023-09-22T21:08:47.123 DEBUG:boto:Params: {} 2023-09-22T21:08:47.124 DEBUG:boto:Token: None 2023-09-22T21:08:47.124 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.124 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cilOIBHPShBNdF7MZYqIDp4qTLQ= 2023-09-22T21:08:47.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cilOIBHPShBNdF7MZYqIDp4qTLQ=', 'Content-Length': '0'} 2023-09-22T21:08:47.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'X5s9z0mNXbqm37pFm1vpfPexuZJVGd0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bedfdf983c247c8f-00650e025f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.127 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.127 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.127 DEBUG:boto:path=/ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.127 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.127 DEBUG:boto:Method: HEAD 2023-09-22T21:08:47.128 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.128 DEBUG:boto:Data: 2023-09-22T21:08:47.128 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.128 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:47.128 DEBUG:boto:Port: 8000 2023-09-22T21:08:47.128 DEBUG:boto:Params: {} 2023-09-22T21:08:47.128 DEBUG:boto:Token: None 2023-09-22T21:08:47.128 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.128 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cilOIBHPShBNdF7MZYqIDp4qTLQ= 2023-09-22T21:08:47.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cilOIBHPShBNdF7MZYqIDp4qTLQ=', 'Content-Length': '0'} 2023-09-22T21:08:47.131 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'X5s9z0mNXbqm37pFm1vpfPexuZJVGd0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001733899a144fedf6-00650e025f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:47.132 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.132 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.132 DEBUG:boto:path=/ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.132 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.132 DEBUG:boto:Method: GET 2023-09-22T21:08:47.132 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.132 DEBUG:boto:Data: 2023-09-22T21:08:47.132 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.132 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:47.132 DEBUG:boto:Port: 8001 2023-09-22T21:08:47.133 DEBUG:boto:Params: {} 2023-09-22T21:08:47.133 DEBUG:boto:Token: None 2023-09-22T21:08:47.133 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.133 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VTO10T4hFYtHUb7k1kriFKsSHWI= 2023-09-22T21:08:47.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VTO10T4hFYtHUb7k1kriFKsSHWI=', 'Content-Length': '0'} 2023-09-22T21:08:47.136 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'X5s9z0mNXbqm37pFm1vpfPexuZJVGd0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b30871ff12d2f42-00650e025f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.137 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.137 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.137 DEBUG:boto:path=/ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.137 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.137 DEBUG:boto:Method: GET 2023-09-22T21:08:47.137 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.137 DEBUG:boto:Data: 2023-09-22T21:08:47.137 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.137 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:47.138 DEBUG:boto:Port: 8000 2023-09-22T21:08:47.138 DEBUG:boto:Params: {} 2023-09-22T21:08:47.138 DEBUG:boto:Token: None 2023-09-22T21:08:47.138 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=X5s9z0mNXbqm37pFm1vpfPexuZJVGd0 2023-09-22T21:08:47.138 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VTO10T4hFYtHUb7k1kriFKsSHWI= 2023-09-22T21:08:47.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VTO10T4hFYtHUb7k1kriFKsSHWI=', 'Content-Length': '0'} 2023-09-22T21:08:47.141 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'X5s9z0mNXbqm37pFm1vpfPexuZJVGd0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063eb5da652f6b2b7-00650e025f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.182 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:47.183 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:47.183 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.183 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.183 DEBUG:boto:Method: GET 2023-09-22T21:08:47.183 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.183 DEBUG:boto:Data: 2023-09-22T21:08:47.183 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.183 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:47.183 DEBUG:boto:Port: 8001 2023-09-22T21:08:47.184 DEBUG:boto:Params: {} 2023-09-22T21:08:47.184 DEBUG:boto:Token: None 2023-09-22T21:08:47.184 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/ 2023-09-22T21:08:47.184 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q= 2023-09-22T21:08:47.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q=', 'Content-Length': '0'} 2023-09-22T21:08:47.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088cf3b3097334564-00650e025f-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.190 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:47.190 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:47.190 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:47.190 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.191 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.191 DEBUG:boto:Method: GET 2023-09-22T21:08:47.191 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.191 DEBUG:boto:Data: 2023-09-22T21:08:47.191 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.191 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:47.191 DEBUG:boto:Port: 8000 2023-09-22T21:08:47.191 DEBUG:boto:Params: {} 2023-09-22T21:08:47.191 DEBUG:boto:Token: None 2023-09-22T21:08:47.191 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/ 2023-09-22T21:08:47.192 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q= 2023-09-22T21:08:47.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q=', 'Content-Length': '0'} 2023-09-22T21:08:47.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0c9b0e44fd5d8ac-00650e025f-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.196 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:47.197 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.197 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.197 DEBUG:boto:Method: HEAD 2023-09-22T21:08:47.197 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:47.197 DEBUG:boto:Data: 2023-09-22T21:08:47.197 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.197 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:47.197 DEBUG:boto:Port: 8001 2023-09-22T21:08:47.198 DEBUG:boto:Params: {} 2023-09-22T21:08:47.198 DEBUG:boto:Token: None 2023-09-22T21:08:47.198 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj 2023-09-22T21:08:47.198 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vY5ZK89lo50+/QnGSDcoXl5x488= 2023-09-22T21:08:47.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vY5ZK89lo50+/QnGSDcoXl5x488=', 'Content-Length': '0'} 2023-09-22T21:08:47.204 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b65a46be684eb6cd-00650e025f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.204 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.204 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.204 DEBUG:boto:Method: HEAD 2023-09-22T21:08:47.204 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:47.204 DEBUG:boto:Data: 2023-09-22T21:08:47.204 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.204 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:47.205 DEBUG:boto:Port: 8000 2023-09-22T21:08:47.205 DEBUG:boto:Params: {} 2023-09-22T21:08:47.205 DEBUG:boto:Token: None 2023-09-22T21:08:47.205 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj 2023-09-22T21:08:47.205 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vY5ZK89lo50+/QnGSDcoXl5x488= 2023-09-22T21:08:47.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vY5ZK89lo50+/QnGSDcoXl5x488=', 'Content-Length': '0'} 2023-09-22T21:08:47.211 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eeca3353feba995a-00650e025f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:47.211 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.211 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.211 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:47.211 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:47.212 DEBUG:boto:Method: GET 2023-09-22T21:08:47.212 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:47.212 DEBUG:boto:Data: 2023-09-22T21:08:47.414 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.414 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:47.414 DEBUG:boto:Port: 8001 2023-09-22T21:08:47.415 DEBUG:boto:Params: {} 2023-09-22T21:08:47.415 DEBUG:boto:Token: None 2023-09-22T21:08:47.415 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:47.415 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XMqvlZwrMxUwTGYUAYWu2xEt0k0= 2023-09-22T21:08:47.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XMqvlZwrMxUwTGYUAYWu2xEt0k0=', 'Content-Length': '0'} 2023-09-22T21:08:47.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b304f0d3c6acde19-00650e025f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.418 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.418 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.419 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:47.419 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:47.419 DEBUG:boto:Method: GET 2023-09-22T21:08:47.419 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:47.419 DEBUG:boto:Data: 2023-09-22T21:08:47.419 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.419 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:47.419 DEBUG:boto:Port: 8000 2023-09-22T21:08:47.419 DEBUG:boto:Params: {} 2023-09-22T21:08:47.420 DEBUG:boto:Token: None 2023-09-22T21:08:47.420 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:47.420 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XMqvlZwrMxUwTGYUAYWu2xEt0k0= 2023-09-22T21:08:47.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XMqvlZwrMxUwTGYUAYWu2xEt0k0=', 'Content-Length': '0'} 2023-09-22T21:08:47.422 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c763cecab5aec30b-00650e025f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.423 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:47.423 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:47.423 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.423 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.423 DEBUG:boto:Method: GET 2023-09-22T21:08:47.423 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.423 DEBUG:boto:Data: 2023-09-22T21:08:47.423 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.424 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:47.424 DEBUG:boto:Port: 8001 2023-09-22T21:08:47.424 DEBUG:boto:Params: {} 2023-09-22T21:08:47.424 DEBUG:boto:Token: None 2023-09-22T21:08:47.424 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/ 2023-09-22T21:08:47.424 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q= 2023-09-22T21:08:47.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q=', 'Content-Length': '0'} 2023-09-22T21:08:47.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c672341dcdb18511-00650e025f-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.430 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:47.430 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:47.430 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:47.430 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.430 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.431 DEBUG:boto:Method: GET 2023-09-22T21:08:47.431 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.431 DEBUG:boto:Data: 2023-09-22T21:08:47.431 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.431 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:47.431 DEBUG:boto:Port: 8000 2023-09-22T21:08:47.431 DEBUG:boto:Params: {} 2023-09-22T21:08:47.431 DEBUG:boto:Token: None 2023-09-22T21:08:47.431 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/ 2023-09-22T21:08:47.432 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q= 2023-09-22T21:08:47.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q=', 'Content-Length': '0'} 2023-09-22T21:08:47.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e64f567cac3473dc-00650e025f-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.478 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:47.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:47.479 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.479 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.479 DEBUG:boto:path=/ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.479 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.479 DEBUG:boto:Method: GET 2023-09-22T21:08:47.480 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.480 DEBUG:boto:Data: 2023-09-22T21:08:47.480 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.480 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:47.480 DEBUG:boto:Port: 8001 2023-09-22T21:08:47.480 DEBUG:boto:Params: {} 2023-09-22T21:08:47.480 DEBUG:boto:Token: None 2023-09-22T21:08:47.480 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.480 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ess/sSli49w4BhPNHUnvTmJhQpY= 2023-09-22T21:08:47.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ess/sSli49w4BhPNHUnvTmJhQpY=', 'Content-Length': '0'} 2023-09-22T21:08:47.483 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'ZmK2SagIILGhSA.9md7mKoG8ffxTZzd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ef9bf2201017050-00650e025f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.571 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.572 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.572 DEBUG:boto:path=/ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.572 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.572 DEBUG:boto:Method: GET 2023-09-22T21:08:47.572 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.572 DEBUG:boto:Data: 2023-09-22T21:08:47.572 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.572 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:47.572 DEBUG:boto:Port: 8000 2023-09-22T21:08:47.573 DEBUG:boto:Params: {} 2023-09-22T21:08:47.573 DEBUG:boto:Token: None 2023-09-22T21:08:47.573 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.573 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ess/sSli49w4BhPNHUnvTmJhQpY= 2023-09-22T21:08:47.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ess/sSli49w4BhPNHUnvTmJhQpY=', 'Content-Length': '0'} 2023-09-22T21:08:47.576 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'ZmK2SagIILGhSA.9md7mKoG8ffxTZzd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066deb2e9b1eee5c1-00650e025f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.576 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.577 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.577 DEBUG:boto:path=/ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.577 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.577 DEBUG:boto:Method: HEAD 2023-09-22T21:08:47.577 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.577 DEBUG:boto:Data: 2023-09-22T21:08:47.577 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.577 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:47.577 DEBUG:boto:Port: 8001 2023-09-22T21:08:47.578 DEBUG:boto:Params: {} 2023-09-22T21:08:47.578 DEBUG:boto:Token: None 2023-09-22T21:08:47.578 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.578 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kvhLaeV/SUVY8nmd8w+kkmEVuMM= 2023-09-22T21:08:47.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kvhLaeV/SUVY8nmd8w+kkmEVuMM=', 'Content-Length': '0'} 2023-09-22T21:08:47.581 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'ZmK2SagIILGhSA.9md7mKoG8ffxTZzd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6fd6ea754e51fee-00650e025f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.581 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.582 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.582 DEBUG:boto:path=/ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.582 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.582 DEBUG:boto:Method: HEAD 2023-09-22T21:08:47.582 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.582 DEBUG:boto:Data: 2023-09-22T21:08:47.582 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.582 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:47.582 DEBUG:boto:Port: 8000 2023-09-22T21:08:47.582 DEBUG:boto:Params: {} 2023-09-22T21:08:47.583 DEBUG:boto:Token: None 2023-09-22T21:08:47.583 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.583 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kvhLaeV/SUVY8nmd8w+kkmEVuMM= 2023-09-22T21:08:47.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kvhLaeV/SUVY8nmd8w+kkmEVuMM=', 'Content-Length': '0'} 2023-09-22T21:08:47.586 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'ZmK2SagIILGhSA.9md7mKoG8ffxTZzd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5228965284dc0e4-00650e025f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:47.586 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.586 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.586 DEBUG:boto:path=/ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.586 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.587 DEBUG:boto:Method: GET 2023-09-22T21:08:47.587 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.587 DEBUG:boto:Data: 2023-09-22T21:08:47.587 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.587 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:47.587 DEBUG:boto:Port: 8001 2023-09-22T21:08:47.587 DEBUG:boto:Params: {} 2023-09-22T21:08:47.587 DEBUG:boto:Token: None 2023-09-22T21:08:47.587 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.588 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ess/sSli49w4BhPNHUnvTmJhQpY= 2023-09-22T21:08:47.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ess/sSli49w4BhPNHUnvTmJhQpY=', 'Content-Length': '0'} 2023-09-22T21:08:47.590 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'ZmK2SagIILGhSA.9md7mKoG8ffxTZzd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0bc1d39995845b7-00650e025f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.590 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.591 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.591 DEBUG:boto:path=/ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.591 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.591 DEBUG:boto:Method: GET 2023-09-22T21:08:47.591 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.591 DEBUG:boto:Data: 2023-09-22T21:08:47.591 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.591 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:47.591 DEBUG:boto:Port: 8000 2023-09-22T21:08:47.591 DEBUG:boto:Params: {} 2023-09-22T21:08:47.591 DEBUG:boto:Token: None 2023-09-22T21:08:47.592 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=ZmK2SagIILGhSA.9md7mKoG8ffxTZzd 2023-09-22T21:08:47.592 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ess/sSli49w4BhPNHUnvTmJhQpY= 2023-09-22T21:08:47.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ess/sSli49w4BhPNHUnvTmJhQpY=', 'Content-Length': '0'} 2023-09-22T21:08:47.594 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'ZmK2SagIILGhSA.9md7mKoG8ffxTZzd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b9d7bb991d85cee-00650e025f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.595 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:47.595 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:47.595 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.595 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.595 DEBUG:boto:Method: GET 2023-09-22T21:08:47.595 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.595 DEBUG:boto:Data: 2023-09-22T21:08:47.595 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.595 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:47.596 DEBUG:boto:Port: 8001 2023-09-22T21:08:47.596 DEBUG:boto:Params: {} 2023-09-22T21:08:47.596 DEBUG:boto:Token: None 2023-09-22T21:08:47.596 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/ 2023-09-22T21:08:47.596 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q= 2023-09-22T21:08:47.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q=', 'Content-Length': '0'} 2023-09-22T21:08:47.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db3bad4be8f16d55-00650e025f-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.602 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:47.602 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:47.602 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:47.602 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.602 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.603 DEBUG:boto:Method: GET 2023-09-22T21:08:47.603 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.603 DEBUG:boto:Data: 2023-09-22T21:08:47.603 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.603 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:47.603 DEBUG:boto:Port: 8000 2023-09-22T21:08:47.603 DEBUG:boto:Params: {} 2023-09-22T21:08:47.603 DEBUG:boto:Token: None 2023-09-22T21:08:47.603 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/ 2023-09-22T21:08:47.604 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q= 2023-09-22T21:08:47.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q=', 'Content-Length': '0'} 2023-09-22T21:08:47.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007498852cc03e45d9-00650e025f-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.609 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:47.726 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.726 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.727 DEBUG:boto:Method: HEAD 2023-09-22T21:08:47.727 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:47.727 DEBUG:boto:Data: 2023-09-22T21:08:47.727 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.727 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:47.727 DEBUG:boto:Port: 8001 2023-09-22T21:08:47.727 DEBUG:boto:Params: {} 2023-09-22T21:08:47.727 DEBUG:boto:Token: None 2023-09-22T21:08:47.727 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj 2023-09-22T21:08:47.727 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vY5ZK89lo50+/QnGSDcoXl5x488= 2023-09-22T21:08:47.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vY5ZK89lo50+/QnGSDcoXl5x488=', 'Content-Length': '0'} 2023-09-22T21:08:47.734 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3a5fb5f9d9ff1b0-00650e025f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.735 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.735 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.735 DEBUG:boto:Method: HEAD 2023-09-22T21:08:47.735 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:47.735 DEBUG:boto:Data: 2023-09-22T21:08:47.735 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.735 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:47.735 DEBUG:boto:Port: 8000 2023-09-22T21:08:47.735 DEBUG:boto:Params: {} 2023-09-22T21:08:47.736 DEBUG:boto:Token: None 2023-09-22T21:08:47.736 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj 2023-09-22T21:08:47.736 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vY5ZK89lo50+/QnGSDcoXl5x488= 2023-09-22T21:08:47.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vY5ZK89lo50+/QnGSDcoXl5x488=', 'Content-Length': '0'} 2023-09-22T21:08:47.741 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4e479dcf46abbbe-00650e025f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:47.742 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.742 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.742 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:47.742 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:47.742 DEBUG:boto:Method: GET 2023-09-22T21:08:47.742 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:47.742 DEBUG:boto:Data: 2023-09-22T21:08:47.742 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.742 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:47.743 DEBUG:boto:Port: 8001 2023-09-22T21:08:47.743 DEBUG:boto:Params: {} 2023-09-22T21:08:47.743 DEBUG:boto:Token: None 2023-09-22T21:08:47.743 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:47.743 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XMqvlZwrMxUwTGYUAYWu2xEt0k0= 2023-09-22T21:08:47.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XMqvlZwrMxUwTGYUAYWu2xEt0k0=', 'Content-Length': '0'} 2023-09-22T21:08:47.746 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9dbd79698e21704-00650e025f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.786 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.786 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.786 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:47.787 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:47.787 DEBUG:boto:Method: GET 2023-09-22T21:08:47.787 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:47.787 DEBUG:boto:Data: 2023-09-22T21:08:47.787 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.787 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:47.787 DEBUG:boto:Port: 8000 2023-09-22T21:08:47.787 DEBUG:boto:Params: {} 2023-09-22T21:08:47.787 DEBUG:boto:Token: None 2023-09-22T21:08:47.787 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:47.788 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XMqvlZwrMxUwTGYUAYWu2xEt0k0= 2023-09-22T21:08:47.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XMqvlZwrMxUwTGYUAYWu2xEt0k0=', 'Content-Length': '0'} 2023-09-22T21:08:47.790 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac8081c71f9116a7-00650e025f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.834 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:47.834 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:47.835 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.835 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.835 DEBUG:boto:Method: GET 2023-09-22T21:08:47.835 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.835 DEBUG:boto:Data: 2023-09-22T21:08:47.835 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.835 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:47.835 DEBUG:boto:Port: 8001 2023-09-22T21:08:47.835 DEBUG:boto:Params: {} 2023-09-22T21:08:47.835 DEBUG:boto:Token: None 2023-09-22T21:08:47.836 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/ 2023-09-22T21:08:47.836 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q= 2023-09-22T21:08:47.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q=', 'Content-Length': '0'} 2023-09-22T21:08:47.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b14b9335ec2ff2a8-00650e025f-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.842 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:47.842 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:47.842 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:47.842 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.842 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.843 DEBUG:boto:Method: GET 2023-09-22T21:08:47.843 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.843 DEBUG:boto:Data: 2023-09-22T21:08:47.843 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.843 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:47.843 DEBUG:boto:Port: 8000 2023-09-22T21:08:47.843 DEBUG:boto:Params: {} 2023-09-22T21:08:47.843 DEBUG:boto:Token: None 2023-09-22T21:08:47.843 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/ 2023-09-22T21:08:47.844 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q= 2023-09-22T21:08:47.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q=', 'Content-Length': '0'} 2023-09-22T21:08:47.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caebcc09d19537e0-00650e025f-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.849 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:47.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:47.849 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.849 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.849 DEBUG:boto:path=/ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.850 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.850 DEBUG:boto:Method: GET 2023-09-22T21:08:47.850 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.850 DEBUG:boto:Data: 2023-09-22T21:08:47.850 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.850 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:47.850 DEBUG:boto:Port: 8001 2023-09-22T21:08:47.850 DEBUG:boto:Params: {} 2023-09-22T21:08:47.850 DEBUG:boto:Token: None 2023-09-22T21:08:47.850 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.851 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yE1vj1wXyEE1O5lhEa3JUvYdRF8= 2023-09-22T21:08:47.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yE1vj1wXyEE1O5lhEa3JUvYdRF8=', 'Content-Length': '0'} 2023-09-22T21:08:47.854 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'MWb3f8.clEGzrytonhqa4UHE-3nWzBc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5ef9adab3839c1d-00650e025f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.898 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.898 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.899 DEBUG:boto:path=/ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.899 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.899 DEBUG:boto:Method: GET 2023-09-22T21:08:47.899 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.899 DEBUG:boto:Data: 2023-09-22T21:08:47.899 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.899 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:47.899 DEBUG:boto:Port: 8000 2023-09-22T21:08:47.899 DEBUG:boto:Params: {} 2023-09-22T21:08:47.899 DEBUG:boto:Token: None 2023-09-22T21:08:47.900 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.900 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yE1vj1wXyEE1O5lhEa3JUvYdRF8= 2023-09-22T21:08:47.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yE1vj1wXyEE1O5lhEa3JUvYdRF8=', 'Content-Length': '0'} 2023-09-22T21:08:47.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'MWb3f8.clEGzrytonhqa4UHE-3nWzBc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064d91ebc417ec607-00650e025f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.903 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.903 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.903 DEBUG:boto:path=/ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.903 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.903 DEBUG:boto:Method: HEAD 2023-09-22T21:08:47.903 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.903 DEBUG:boto:Data: 2023-09-22T21:08:47.903 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.904 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:47.904 DEBUG:boto:Port: 8001 2023-09-22T21:08:47.904 DEBUG:boto:Params: {} 2023-09-22T21:08:47.904 DEBUG:boto:Token: None 2023-09-22T21:08:47.904 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.904 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YuccJBARVKLmk5IiDVzuiLIzgH4= 2023-09-22T21:08:47.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YuccJBARVKLmk5IiDVzuiLIzgH4=', 'Content-Length': '0'} 2023-09-22T21:08:47.907 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'MWb3f8.clEGzrytonhqa4UHE-3nWzBc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ef4bec7305f9d92-00650e025f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.907 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.907 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.907 DEBUG:boto:path=/ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.907 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.907 DEBUG:boto:Method: HEAD 2023-09-22T21:08:47.908 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.908 DEBUG:boto:Data: 2023-09-22T21:08:47.908 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.908 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:47.908 DEBUG:boto:Port: 8000 2023-09-22T21:08:47.908 DEBUG:boto:Params: {} 2023-09-22T21:08:47.908 DEBUG:boto:Token: None 2023-09-22T21:08:47.908 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.908 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YuccJBARVKLmk5IiDVzuiLIzgH4= 2023-09-22T21:08:47.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YuccJBARVKLmk5IiDVzuiLIzgH4=', 'Content-Length': '0'} 2023-09-22T21:08:47.911 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'MWb3f8.clEGzrytonhqa4UHE-3nWzBc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8b55e177396a811-00650e025f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:47.911 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.911 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.911 DEBUG:boto:path=/ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.912 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.912 DEBUG:boto:Method: GET 2023-09-22T21:08:47.912 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.912 DEBUG:boto:Data: 2023-09-22T21:08:47.912 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.912 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:47.912 DEBUG:boto:Port: 8001 2023-09-22T21:08:47.912 DEBUG:boto:Params: {} 2023-09-22T21:08:47.912 DEBUG:boto:Token: None 2023-09-22T21:08:47.912 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.913 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yE1vj1wXyEE1O5lhEa3JUvYdRF8= 2023-09-22T21:08:47.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yE1vj1wXyEE1O5lhEa3JUvYdRF8=', 'Content-Length': '0'} 2023-09-22T21:08:47.916 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'MWb3f8.clEGzrytonhqa4UHE-3nWzBc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000093017bdc40d2059-00650e025f-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.958 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:47.958 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:47.959 DEBUG:boto:path=/ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.959 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.959 DEBUG:boto:Method: GET 2023-09-22T21:08:47.959 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.959 DEBUG:boto:Data: 2023-09-22T21:08:47.959 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.959 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:47.959 DEBUG:boto:Port: 8000 2023-09-22T21:08:47.959 DEBUG:boto:Params: {} 2023-09-22T21:08:47.959 DEBUG:boto:Token: None 2023-09-22T21:08:47.960 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/obj?versionId=MWb3f8.clEGzrytonhqa4UHE-3nWzBc 2023-09-22T21:08:47.960 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yE1vj1wXyEE1O5lhEa3JUvYdRF8= 2023-09-22T21:08:47.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yE1vj1wXyEE1O5lhEa3JUvYdRF8=', 'Content-Length': '0'} 2023-09-22T21:08:47.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'MWb3f8.clEGzrytonhqa4UHE-3nWzBc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3c31c416ddec713-00650e025f-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.963 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:47.963 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:47.963 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.963 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.963 DEBUG:boto:Method: GET 2023-09-22T21:08:47.963 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.964 DEBUG:boto:Data: 2023-09-22T21:08:47.964 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.964 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:47.964 DEBUG:boto:Port: 8001 2023-09-22T21:08:47.964 DEBUG:boto:Params: {} 2023-09-22T21:08:47.964 DEBUG:boto:Token: None 2023-09-22T21:08:47.964 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/ 2023-09-22T21:08:47.964 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q= 2023-09-22T21:08:47.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q=', 'Content-Length': '0'} 2023-09-22T21:08:47.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001af065cacd94aaf6-00650e025f-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:47.970 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:47.970 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:47.970 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:47.970 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.970 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.971 DEBUG:boto:Method: GET 2023-09-22T21:08:47.971 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:47.971 DEBUG:boto:Data: 2023-09-22T21:08:47.971 DEBUG:boto:Headers: {} 2023-09-22T21:08:47.971 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:47.971 DEBUG:boto:Port: 8000 2023-09-22T21:08:47.971 DEBUG:boto:Params: {} 2023-09-22T21:08:47.971 DEBUG:boto:Token: None 2023-09-22T21:08:47.971 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:47 GMT /ffzdqu-13/ 2023-09-22T21:08:47.971 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q= 2023-09-22T21:08:47.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i+7MgXCgdYGSg3e2BHInGXaXW5Q=', 'Content-Length': '0'} 2023-09-22T21:08:47.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8477ca579761643-00650e025f-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.022 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:48.023 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.023 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.023 DEBUG:boto:Method: HEAD 2023-09-22T21:08:48.023 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:48.023 DEBUG:boto:Data: 2023-09-22T21:08:48.023 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.023 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.023 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.024 DEBUG:boto:Params: {} 2023-09-22T21:08:48.024 DEBUG:boto:Token: None 2023-09-22T21:08:48.024 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj 2023-09-22T21:08:48.024 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kpIgYfQnHTNUVPDH0rRgHGUEjYg= 2023-09-22T21:08:48.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kpIgYfQnHTNUVPDH0rRgHGUEjYg=', 'Content-Length': '0'} 2023-09-22T21:08:48.030 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e073ff4e3d54975-00650e0260-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.030 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.030 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.030 DEBUG:boto:Method: HEAD 2023-09-22T21:08:48.030 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:48.030 DEBUG:boto:Data: 2023-09-22T21:08:48.030 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.031 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.031 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.031 DEBUG:boto:Params: {} 2023-09-22T21:08:48.031 DEBUG:boto:Token: None 2023-09-22T21:08:48.031 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj 2023-09-22T21:08:48.031 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kpIgYfQnHTNUVPDH0rRgHGUEjYg= 2023-09-22T21:08:48.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kpIgYfQnHTNUVPDH0rRgHGUEjYg=', 'Content-Length': '0'} 2023-09-22T21:08:48.036 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da6f945abd2ee3da-00650e0260-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:48.037 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.037 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.037 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.037 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.037 DEBUG:boto:Method: GET 2023-09-22T21:08:48.037 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.038 DEBUG:boto:Data: 2023-09-22T21:08:48.038 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.038 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.038 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.038 DEBUG:boto:Params: {} 2023-09-22T21:08:48.038 DEBUG:boto:Token: None 2023-09-22T21:08:48.038 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.038 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QBIV6ZRmTbRiq3FN82Px5H2SsS4= 2023-09-22T21:08:48.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QBIV6ZRmTbRiq3FN82Px5H2SsS4=', 'Content-Length': '0'} 2023-09-22T21:08:48.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a446a4ffa2efde6-00650e0260-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.041 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.041 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.041 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.041 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.042 DEBUG:boto:Method: GET 2023-09-22T21:08:48.042 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.042 DEBUG:boto:Data: 2023-09-22T21:08:48.042 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.042 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.042 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.042 DEBUG:boto:Params: {} 2023-09-22T21:08:48.042 DEBUG:boto:Token: None 2023-09-22T21:08:48.042 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.043 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QBIV6ZRmTbRiq3FN82Px5H2SsS4= 2023-09-22T21:08:48.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QBIV6ZRmTbRiq3FN82Px5H2SsS4=', 'Content-Length': '0'} 2023-09-22T21:08:48.045 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000915ddf548e6d2c93-00650e0260-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.086 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:48.086 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:48.087 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.087 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.087 DEBUG:boto:Method: GET 2023-09-22T21:08:48.087 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.087 DEBUG:boto:Data: 2023-09-22T21:08:48.087 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.087 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.087 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.087 DEBUG:boto:Params: {} 2023-09-22T21:08:48.087 DEBUG:boto:Token: None 2023-09-22T21:08:48.088 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/ 2023-09-22T21:08:48.088 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k= 2023-09-22T21:08:48.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k=', 'Content-Length': '0'} 2023-09-22T21:08:48.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6f6252cb8304367-00650e0260-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.093 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:48.093 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:48.094 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:48.094 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.094 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.094 DEBUG:boto:Method: GET 2023-09-22T21:08:48.094 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.094 DEBUG:boto:Data: 2023-09-22T21:08:48.094 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.094 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.094 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.094 DEBUG:boto:Params: {} 2023-09-22T21:08:48.095 DEBUG:boto:Token: None 2023-09-22T21:08:48.095 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/ 2023-09-22T21:08:48.095 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k= 2023-09-22T21:08:48.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k=', 'Content-Length': '0'} 2023-09-22T21:08:48.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd777e29ea1dcc44-00650e0260-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.146 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:48.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:48.147 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.147 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.147 DEBUG:boto:path=/ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.147 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.147 DEBUG:boto:Method: GET 2023-09-22T21:08:48.147 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.147 DEBUG:boto:Data: 2023-09-22T21:08:48.147 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.148 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.148 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.148 DEBUG:boto:Params: {} 2023-09-22T21:08:48.148 DEBUG:boto:Token: None 2023-09-22T21:08:48.148 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.148 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:47FyrZQXUX/7eFqfG3ons6hgrpw= 2023-09-22T21:08:48.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:47FyrZQXUX/7eFqfG3ons6hgrpw=', 'Content-Length': '0'} 2023-09-22T21:08:48.151 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003ca2db30dd41447-00650e0260-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.151 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.151 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.152 DEBUG:boto:path=/ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.152 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.152 DEBUG:boto:Method: GET 2023-09-22T21:08:48.152 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.152 DEBUG:boto:Data: 2023-09-22T21:08:48.152 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.152 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.152 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.152 DEBUG:boto:Params: {} 2023-09-22T21:08:48.152 DEBUG:boto:Token: None 2023-09-22T21:08:48.153 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.153 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:47FyrZQXUX/7eFqfG3ons6hgrpw= 2023-09-22T21:08:48.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:47FyrZQXUX/7eFqfG3ons6hgrpw=', 'Content-Length': '0'} 2023-09-22T21:08:48.156 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5e297b0e721e1dc-00650e0260-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.202 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.203 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.203 DEBUG:boto:path=/ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.203 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.203 DEBUG:boto:Method: HEAD 2023-09-22T21:08:48.203 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.203 DEBUG:boto:Data: 2023-09-22T21:08:48.203 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.203 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.205 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.205 DEBUG:boto:Params: {} 2023-09-22T21:08:48.205 DEBUG:boto:Token: None 2023-09-22T21:08:48.205 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.205 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tUJ+yPK+uyfqhSfu2uH49tgcr/U= 2023-09-22T21:08:48.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tUJ+yPK+uyfqhSfu2uH49tgcr/U=', 'Content-Length': '0'} 2023-09-22T21:08:48.208 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008dcdae623db2eb52-00650e0260-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.208 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.209 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.209 DEBUG:boto:path=/ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.209 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.209 DEBUG:boto:Method: HEAD 2023-09-22T21:08:48.209 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.209 DEBUG:boto:Data: 2023-09-22T21:08:48.209 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.209 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.209 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.209 DEBUG:boto:Params: {} 2023-09-22T21:08:48.210 DEBUG:boto:Token: None 2023-09-22T21:08:48.210 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.210 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tUJ+yPK+uyfqhSfu2uH49tgcr/U= 2023-09-22T21:08:48.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tUJ+yPK+uyfqhSfu2uH49tgcr/U=', 'Content-Length': '0'} 2023-09-22T21:08:48.213 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052f2d53f3d527e89-00650e0260-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:48.213 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.213 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.213 DEBUG:boto:path=/ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.213 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.213 DEBUG:boto:Method: GET 2023-09-22T21:08:48.214 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.214 DEBUG:boto:Data: 2023-09-22T21:08:48.214 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.214 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.214 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.214 DEBUG:boto:Params: {} 2023-09-22T21:08:48.214 DEBUG:boto:Token: None 2023-09-22T21:08:48.214 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.214 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:47FyrZQXUX/7eFqfG3ons6hgrpw= 2023-09-22T21:08:48.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:47FyrZQXUX/7eFqfG3ons6hgrpw=', 'Content-Length': '0'} 2023-09-22T21:08:48.217 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098184ba665be792e-00650e0260-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.262 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.262 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.263 DEBUG:boto:path=/ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.263 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.263 DEBUG:boto:Method: GET 2023-09-22T21:08:48.263 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.263 DEBUG:boto:Data: 2023-09-22T21:08:48.263 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.263 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.263 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.263 DEBUG:boto:Params: {} 2023-09-22T21:08:48.263 DEBUG:boto:Token: None 2023-09-22T21:08:48.264 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3 2023-09-22T21:08:48.264 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:47FyrZQXUX/7eFqfG3ons6hgrpw= 2023-09-22T21:08:48.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:47FyrZQXUX/7eFqfG3ons6hgrpw=', 'Content-Length': '0'} 2023-09-22T21:08:48.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'tQ6vaDx6FoFq3NNXxkWlMoMiEQdm8j3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1e40dcd7efed926-00650e0260-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.310 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:48.310 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:48.311 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.311 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.311 DEBUG:boto:Method: GET 2023-09-22T21:08:48.311 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.311 DEBUG:boto:Data: 2023-09-22T21:08:48.311 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.311 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.311 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.311 DEBUG:boto:Params: {} 2023-09-22T21:08:48.312 DEBUG:boto:Token: None 2023-09-22T21:08:48.312 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/ 2023-09-22T21:08:48.312 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k= 2023-09-22T21:08:48.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k=', 'Content-Length': '0'} 2023-09-22T21:08:48.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cb16428b1885e58-00650e0260-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.318 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:48.318 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:48.319 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:48.319 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.319 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.319 DEBUG:boto:Method: GET 2023-09-22T21:08:48.319 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.319 DEBUG:boto:Data: 2023-09-22T21:08:48.319 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.319 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.319 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.320 DEBUG:boto:Params: {} 2023-09-22T21:08:48.320 DEBUG:boto:Token: None 2023-09-22T21:08:48.320 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/ 2023-09-22T21:08:48.320 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k= 2023-09-22T21:08:48.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k=', 'Content-Length': '0'} 2023-09-22T21:08:48.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a69ddacf709c2693-00650e0260-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.366 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:48.367 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.367 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.367 DEBUG:boto:Method: HEAD 2023-09-22T21:08:48.367 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:48.367 DEBUG:boto:Data: 2023-09-22T21:08:48.367 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.367 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.368 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.368 DEBUG:boto:Params: {} 2023-09-22T21:08:48.368 DEBUG:boto:Token: None 2023-09-22T21:08:48.368 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj 2023-09-22T21:08:48.368 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kpIgYfQnHTNUVPDH0rRgHGUEjYg= 2023-09-22T21:08:48.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kpIgYfQnHTNUVPDH0rRgHGUEjYg=', 'Content-Length': '0'} 2023-09-22T21:08:48.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000114f41658f602b3e-00650e0260-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.374 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.374 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.374 DEBUG:boto:Method: HEAD 2023-09-22T21:08:48.374 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:48.374 DEBUG:boto:Data: 2023-09-22T21:08:48.374 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.374 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.375 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.375 DEBUG:boto:Params: {} 2023-09-22T21:08:48.375 DEBUG:boto:Token: None 2023-09-22T21:08:48.375 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj 2023-09-22T21:08:48.375 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kpIgYfQnHTNUVPDH0rRgHGUEjYg= 2023-09-22T21:08:48.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kpIgYfQnHTNUVPDH0rRgHGUEjYg=', 'Content-Length': '0'} 2023-09-22T21:08:48.380 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004958d580a8efca79-00650e0260-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:48.381 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.381 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.381 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.381 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.381 DEBUG:boto:Method: GET 2023-09-22T21:08:48.381 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.381 DEBUG:boto:Data: 2023-09-22T21:08:48.382 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.382 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.382 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.382 DEBUG:boto:Params: {} 2023-09-22T21:08:48.383 DEBUG:boto:Token: None 2023-09-22T21:08:48.383 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.383 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QBIV6ZRmTbRiq3FN82Px5H2SsS4= 2023-09-22T21:08:48.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QBIV6ZRmTbRiq3FN82Px5H2SsS4=', 'Content-Length': '0'} 2023-09-22T21:08:48.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d92e70ff8b5c3ac-00650e0260-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.387 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.387 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.387 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.387 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.387 DEBUG:boto:Method: GET 2023-09-22T21:08:48.387 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.387 DEBUG:boto:Data: 2023-09-22T21:08:48.387 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.388 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.388 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.388 DEBUG:boto:Params: {} 2023-09-22T21:08:48.388 DEBUG:boto:Token: None 2023-09-22T21:08:48.388 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.388 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QBIV6ZRmTbRiq3FN82Px5H2SsS4= 2023-09-22T21:08:48.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QBIV6ZRmTbRiq3FN82Px5H2SsS4=', 'Content-Length': '0'} 2023-09-22T21:08:48.391 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015b7e27060075df2-00650e0260-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.438 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:48.439 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:48.439 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.439 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.439 DEBUG:boto:Method: GET 2023-09-22T21:08:48.439 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.439 DEBUG:boto:Data: 2023-09-22T21:08:48.439 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.439 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.439 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.440 DEBUG:boto:Params: {} 2023-09-22T21:08:48.440 DEBUG:boto:Token: None 2023-09-22T21:08:48.440 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/ 2023-09-22T21:08:48.440 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k= 2023-09-22T21:08:48.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k=', 'Content-Length': '0'} 2023-09-22T21:08:48.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d329dc9776170ff-00650e0260-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.446 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:48.446 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:48.446 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:48.446 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.446 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.446 DEBUG:boto:Method: GET 2023-09-22T21:08:48.447 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.447 DEBUG:boto:Data: 2023-09-22T21:08:48.447 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.447 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.447 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.447 DEBUG:boto:Params: {} 2023-09-22T21:08:48.447 DEBUG:boto:Token: None 2023-09-22T21:08:48.447 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/ 2023-09-22T21:08:48.447 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k= 2023-09-22T21:08:48.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k=', 'Content-Length': '0'} 2023-09-22T21:08:48.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c05a17f7b465257-00650e0260-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.494 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:48.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:48.495 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.496 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.496 DEBUG:boto:path=/ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.496 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.496 DEBUG:boto:Method: GET 2023-09-22T21:08:48.497 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.497 DEBUG:boto:Data: 2023-09-22T21:08:48.497 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.497 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.497 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.497 DEBUG:boto:Params: {} 2023-09-22T21:08:48.497 DEBUG:boto:Token: None 2023-09-22T21:08:48.498 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.498 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bHW9q/2Wf0dIJN6/+UQFWWnsWzY= 2023-09-22T21:08:48.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bHW9q/2Wf0dIJN6/+UQFWWnsWzY=', 'Content-Length': '0'} 2023-09-22T21:08:48.501 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'Ux-o.jzhWCEq59rWr2ALKdruobR.tsY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aee04d5c546f72ce-00650e0260-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.502 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.502 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.502 DEBUG:boto:path=/ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.502 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.502 DEBUG:boto:Method: GET 2023-09-22T21:08:48.502 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.502 DEBUG:boto:Data: 2023-09-22T21:08:48.503 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.503 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.503 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.503 DEBUG:boto:Params: {} 2023-09-22T21:08:48.503 DEBUG:boto:Token: None 2023-09-22T21:08:48.503 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.503 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bHW9q/2Wf0dIJN6/+UQFWWnsWzY= 2023-09-22T21:08:48.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bHW9q/2Wf0dIJN6/+UQFWWnsWzY=', 'Content-Length': '0'} 2023-09-22T21:08:48.506 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'Ux-o.jzhWCEq59rWr2ALKdruobR.tsY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb36aed519499e08-00650e0260-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.546 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.547 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.547 DEBUG:boto:path=/ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.547 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.548 DEBUG:boto:Method: HEAD 2023-09-22T21:08:48.548 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.548 DEBUG:boto:Data: 2023-09-22T21:08:48.548 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.548 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.548 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.549 DEBUG:boto:Params: {} 2023-09-22T21:08:48.549 DEBUG:boto:Token: None 2023-09-22T21:08:48.549 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.550 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4JAqoCyjb6F6vnrFgoL2otQdwRc= 2023-09-22T21:08:48.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4JAqoCyjb6F6vnrFgoL2otQdwRc=', 'Content-Length': '0'} 2023-09-22T21:08:48.553 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'Ux-o.jzhWCEq59rWr2ALKdruobR.tsY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c08cee4fc6c51c8-00650e0260-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.553 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.553 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.554 DEBUG:boto:path=/ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.554 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.554 DEBUG:boto:Method: HEAD 2023-09-22T21:08:48.554 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.554 DEBUG:boto:Data: 2023-09-22T21:08:48.554 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.554 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.555 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.555 DEBUG:boto:Params: {} 2023-09-22T21:08:48.555 DEBUG:boto:Token: None 2023-09-22T21:08:48.555 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.555 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4JAqoCyjb6F6vnrFgoL2otQdwRc= 2023-09-22T21:08:48.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4JAqoCyjb6F6vnrFgoL2otQdwRc=', 'Content-Length': '0'} 2023-09-22T21:08:48.558 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'Ux-o.jzhWCEq59rWr2ALKdruobR.tsY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019adebb4ebe41d4f-00650e0260-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:48.559 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.559 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.559 DEBUG:boto:path=/ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.559 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.559 DEBUG:boto:Method: GET 2023-09-22T21:08:48.559 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.559 DEBUG:boto:Data: 2023-09-22T21:08:48.559 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.559 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.559 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.560 DEBUG:boto:Params: {} 2023-09-22T21:08:48.560 DEBUG:boto:Token: None 2023-09-22T21:08:48.560 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.560 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bHW9q/2Wf0dIJN6/+UQFWWnsWzY= 2023-09-22T21:08:48.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bHW9q/2Wf0dIJN6/+UQFWWnsWzY=', 'Content-Length': '0'} 2023-09-22T21:08:48.563 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'Ux-o.jzhWCEq59rWr2ALKdruobR.tsY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000845230f14ecfcbee-00650e0260-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.563 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.563 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.563 DEBUG:boto:path=/ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.563 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.564 DEBUG:boto:Method: GET 2023-09-22T21:08:48.564 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.564 DEBUG:boto:Data: 2023-09-22T21:08:48.564 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.564 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.564 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.564 DEBUG:boto:Params: {} 2023-09-22T21:08:48.564 DEBUG:boto:Token: None 2023-09-22T21:08:48.564 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=Ux-o.jzhWCEq59rWr2ALKdruobR.tsY 2023-09-22T21:08:48.565 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bHW9q/2Wf0dIJN6/+UQFWWnsWzY= 2023-09-22T21:08:48.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bHW9q/2Wf0dIJN6/+UQFWWnsWzY=', 'Content-Length': '0'} 2023-09-22T21:08:48.567 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'Ux-o.jzhWCEq59rWr2ALKdruobR.tsY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c8eaaaec2974fb7-00650e0260-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.610 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:48.611 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:48.611 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.611 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.611 DEBUG:boto:Method: GET 2023-09-22T21:08:48.611 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.611 DEBUG:boto:Data: 2023-09-22T21:08:48.611 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.611 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.611 DEBUG:boto:Params: {} 2023-09-22T21:08:48.612 DEBUG:boto:Token: None 2023-09-22T21:08:48.612 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/ 2023-09-22T21:08:48.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k= 2023-09-22T21:08:48.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k=', 'Content-Length': '0'} 2023-09-22T21:08:48.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007735c04876f070fa-00650e0260-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.618 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:48.618 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:48.618 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:48.618 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.619 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.619 DEBUG:boto:Method: GET 2023-09-22T21:08:48.619 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.619 DEBUG:boto:Data: 2023-09-22T21:08:48.620 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.620 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.620 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.620 DEBUG:boto:Params: {} 2023-09-22T21:08:48.620 DEBUG:boto:Token: None 2023-09-22T21:08:48.621 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/ 2023-09-22T21:08:48.621 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k= 2023-09-22T21:08:48.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k=', 'Content-Length': '0'} 2023-09-22T21:08:48.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000069814403a52e2e9-00650e0260-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.670 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:48.671 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.671 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.671 DEBUG:boto:Method: HEAD 2023-09-22T21:08:48.671 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:48.672 DEBUG:boto:Data: 2023-09-22T21:08:48.672 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.672 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.672 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.672 DEBUG:boto:Params: {} 2023-09-22T21:08:48.672 DEBUG:boto:Token: None 2023-09-22T21:08:48.672 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj 2023-09-22T21:08:48.673 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kpIgYfQnHTNUVPDH0rRgHGUEjYg= 2023-09-22T21:08:48.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kpIgYfQnHTNUVPDH0rRgHGUEjYg=', 'Content-Length': '0'} 2023-09-22T21:08:48.678 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097226b52cce0e58f-00650e0260-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.679 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.679 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.679 DEBUG:boto:Method: HEAD 2023-09-22T21:08:48.679 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:48.679 DEBUG:boto:Data: 2023-09-22T21:08:48.679 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.680 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.680 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.680 DEBUG:boto:Params: {} 2023-09-22T21:08:48.680 DEBUG:boto:Token: None 2023-09-22T21:08:48.680 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj 2023-09-22T21:08:48.681 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kpIgYfQnHTNUVPDH0rRgHGUEjYg= 2023-09-22T21:08:48.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kpIgYfQnHTNUVPDH0rRgHGUEjYg=', 'Content-Length': '0'} 2023-09-22T21:08:48.686 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e8d3917390aff49-00650e0260-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:48.687 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.688 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.688 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.688 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.688 DEBUG:boto:Method: GET 2023-09-22T21:08:48.689 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.689 DEBUG:boto:Data: 2023-09-22T21:08:48.689 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.689 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.690 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.690 DEBUG:boto:Params: {} 2023-09-22T21:08:48.691 DEBUG:boto:Token: None 2023-09-22T21:08:48.691 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.691 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QBIV6ZRmTbRiq3FN82Px5H2SsS4= 2023-09-22T21:08:48.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QBIV6ZRmTbRiq3FN82Px5H2SsS4=', 'Content-Length': '0'} 2023-09-22T21:08:48.694 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086703538c7ada145-00650e0260-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.695 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.695 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.696 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.696 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.696 DEBUG:boto:Method: GET 2023-09-22T21:08:48.696 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.696 DEBUG:boto:Data: 2023-09-22T21:08:48.697 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.697 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.697 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.697 DEBUG:boto:Params: {} 2023-09-22T21:08:48.697 DEBUG:boto:Token: None 2023-09-22T21:08:48.698 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.698 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QBIV6ZRmTbRiq3FN82Px5H2SsS4= 2023-09-22T21:08:48.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QBIV6ZRmTbRiq3FN82Px5H2SsS4=', 'Content-Length': '0'} 2023-09-22T21:08:48.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000237cbe5d97798739-00650e0260-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.746 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:48.746 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:48.747 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.747 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.747 DEBUG:boto:Method: GET 2023-09-22T21:08:48.747 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.747 DEBUG:boto:Data: 2023-09-22T21:08:48.747 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.747 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.747 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.748 DEBUG:boto:Params: {} 2023-09-22T21:08:48.748 DEBUG:boto:Token: None 2023-09-22T21:08:48.748 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/ 2023-09-22T21:08:48.748 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k= 2023-09-22T21:08:48.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k=', 'Content-Length': '0'} 2023-09-22T21:08:48.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d259fc81da3cebd5-00650e0260-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.756 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:48.756 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:48.756 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:48.757 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.757 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.757 DEBUG:boto:Method: GET 2023-09-22T21:08:48.757 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.757 DEBUG:boto:Data: 2023-09-22T21:08:48.757 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.757 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.757 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.757 DEBUG:boto:Params: {} 2023-09-22T21:08:48.757 DEBUG:boto:Token: None 2023-09-22T21:08:48.758 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/ 2023-09-22T21:08:48.758 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k= 2023-09-22T21:08:48.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k=', 'Content-Length': '0'} 2023-09-22T21:08:48.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097a9c773b8266aa4-00650e0260-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.810 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:48.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:48.811 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.811 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.811 DEBUG:boto:path=/ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.811 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.811 DEBUG:boto:Method: GET 2023-09-22T21:08:48.811 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.811 DEBUG:boto:Data: 2023-09-22T21:08:48.811 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.812 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.812 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.812 DEBUG:boto:Params: {} 2023-09-22T21:08:48.812 DEBUG:boto:Token: None 2023-09-22T21:08:48.812 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.812 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:008K9uadtMokIWij+ECMWxU5ogk= 2023-09-22T21:08:48.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:008K9uadtMokIWij+ECMWxU5ogk=', 'Content-Length': '0'} 2023-09-22T21:08:48.817 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1f160b53445fb27-00650e0260-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.859 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.860 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.860 DEBUG:boto:path=/ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.860 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.860 DEBUG:boto:Method: GET 2023-09-22T21:08:48.860 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.860 DEBUG:boto:Data: 2023-09-22T21:08:48.860 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.862 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.862 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.863 DEBUG:boto:Params: {} 2023-09-22T21:08:48.863 DEBUG:boto:Token: None 2023-09-22T21:08:48.866 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.866 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:008K9uadtMokIWij+ECMWxU5ogk= 2023-09-22T21:08:48.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:008K9uadtMokIWij+ECMWxU5ogk=', 'Content-Length': '0'} 2023-09-22T21:08:48.870 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000083ebde77d507c7e6-00650e0260-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.870 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.870 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.871 DEBUG:boto:path=/ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.871 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.871 DEBUG:boto:Method: HEAD 2023-09-22T21:08:48.871 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.871 DEBUG:boto:Data: 2023-09-22T21:08:48.871 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.872 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.872 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.872 DEBUG:boto:Params: {} 2023-09-22T21:08:48.872 DEBUG:boto:Token: None 2023-09-22T21:08:48.872 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.873 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rw0VDX7boVcEGheBcF8M6m62yS0= 2023-09-22T21:08:48.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rw0VDX7boVcEGheBcF8M6m62yS0=', 'Content-Length': '0'} 2023-09-22T21:08:48.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2a9039181b13098-00650e0260-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.875 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.876 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.876 DEBUG:boto:path=/ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.876 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.876 DEBUG:boto:Method: HEAD 2023-09-22T21:08:48.876 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.876 DEBUG:boto:Data: 2023-09-22T21:08:48.876 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.877 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.877 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.877 DEBUG:boto:Params: {} 2023-09-22T21:08:48.877 DEBUG:boto:Token: None 2023-09-22T21:08:48.877 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.877 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rw0VDX7boVcEGheBcF8M6m62yS0= 2023-09-22T21:08:48.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rw0VDX7boVcEGheBcF8M6m62yS0=', 'Content-Length': '0'} 2023-09-22T21:08:48.881 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2ffd9a62a58a75f-00650e0260-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:48.881 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.881 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.881 DEBUG:boto:path=/ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.882 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.882 DEBUG:boto:Method: GET 2023-09-22T21:08:48.882 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.882 DEBUG:boto:Data: 2023-09-22T21:08:48.882 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.882 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.882 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.882 DEBUG:boto:Params: {} 2023-09-22T21:08:48.882 DEBUG:boto:Token: None 2023-09-22T21:08:48.883 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.883 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:008K9uadtMokIWij+ECMWxU5ogk= 2023-09-22T21:08:48.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:008K9uadtMokIWij+ECMWxU5ogk=', 'Content-Length': '0'} 2023-09-22T21:08:48.886 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9dcbe3a4f2a0a4f-00650e0260-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.930 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.931 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.931 DEBUG:boto:path=/ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.931 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.931 DEBUG:boto:Method: GET 2023-09-22T21:08:48.931 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.932 DEBUG:boto:Data: 2023-09-22T21:08:48.932 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.932 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.932 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.932 DEBUG:boto:Params: {} 2023-09-22T21:08:48.932 DEBUG:boto:Token: None 2023-09-22T21:08:48.933 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd 2023-09-22T21:08:48.933 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:008K9uadtMokIWij+ECMWxU5ogk= 2023-09-22T21:08:48.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:008K9uadtMokIWij+ECMWxU5ogk=', 'Content-Length': '0'} 2023-09-22T21:08:48.936 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'ZFg17GpJzZzJQ2hycn0YSoQcRmEUKXd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d0ef1762be2e637-00650e0260-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.936 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:48.937 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:48.937 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.937 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.937 DEBUG:boto:Method: GET 2023-09-22T21:08:48.937 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.937 DEBUG:boto:Data: 2023-09-22T21:08:48.937 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.938 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.938 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.938 DEBUG:boto:Params: {} 2023-09-22T21:08:48.938 DEBUG:boto:Token: None 2023-09-22T21:08:48.938 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/ 2023-09-22T21:08:48.939 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k= 2023-09-22T21:08:48.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k=', 'Content-Length': '0'} 2023-09-22T21:08:48.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d34ef1f47aaffaf-00650e0260-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.944 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:48.945 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:48.945 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:48.945 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.945 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.946 DEBUG:boto:Method: GET 2023-09-22T21:08:48.946 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:48.946 DEBUG:boto:Data: 2023-09-22T21:08:48.946 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.946 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.946 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.947 DEBUG:boto:Params: {} 2023-09-22T21:08:48.947 DEBUG:boto:Token: None 2023-09-22T21:08:48.947 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/ 2023-09-22T21:08:48.947 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k= 2023-09-22T21:08:48.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8jwgNCTCvvqQZ2qAvU4q7UAWV5k=', 'Content-Length': '0'} 2023-09-22T21:08:48.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdec82ac6d3edb5d-00650e0260-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.952 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:48.953 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.953 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.953 DEBUG:boto:Method: HEAD 2023-09-22T21:08:48.953 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:48.953 DEBUG:boto:Data: 2023-09-22T21:08:48.954 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.954 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.954 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.954 DEBUG:boto:Params: {} 2023-09-22T21:08:48.954 DEBUG:boto:Token: None 2023-09-22T21:08:48.955 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj 2023-09-22T21:08:48.955 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kpIgYfQnHTNUVPDH0rRgHGUEjYg= 2023-09-22T21:08:48.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kpIgYfQnHTNUVPDH0rRgHGUEjYg=', 'Content-Length': '0'} 2023-09-22T21:08:48.960 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ff66ad2d5e53c6d-00650e0260-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.961 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.961 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.961 DEBUG:boto:Method: HEAD 2023-09-22T21:08:48.961 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:48.961 DEBUG:boto:Data: 2023-09-22T21:08:48.962 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.962 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:48.962 DEBUG:boto:Port: 8000 2023-09-22T21:08:48.962 DEBUG:boto:Params: {} 2023-09-22T21:08:48.962 DEBUG:boto:Token: None 2023-09-22T21:08:48.963 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj 2023-09-22T21:08:48.963 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kpIgYfQnHTNUVPDH0rRgHGUEjYg= 2023-09-22T21:08:48.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kpIgYfQnHTNUVPDH0rRgHGUEjYg=', 'Content-Length': '0'} 2023-09-22T21:08:48.969 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000887ffa6d7226c3b5-00650e0260-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:48.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:48.969 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:48.969 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:48.970 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.970 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.970 DEBUG:boto:Method: GET 2023-09-22T21:08:48.970 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.970 DEBUG:boto:Data: 2023-09-22T21:08:48.970 DEBUG:boto:Headers: {} 2023-09-22T21:08:48.970 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:48.970 DEBUG:boto:Port: 8001 2023-09-22T21:08:48.970 DEBUG:boto:Params: {} 2023-09-22T21:08:48.971 DEBUG:boto:Token: None 2023-09-22T21:08:48.971 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:48 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:48.971 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QBIV6ZRmTbRiq3FN82Px5H2SsS4= 2023-09-22T21:08:48.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QBIV6ZRmTbRiq3FN82Px5H2SsS4=', 'Content-Length': '0'} 2023-09-22T21:08:48.974 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fce0ce54a294f918-00650e0260-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.014 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.015 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.015 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.015 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.015 DEBUG:boto:Method: GET 2023-09-22T21:08:49.015 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.016 DEBUG:boto:Data: 2023-09-22T21:08:49.016 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.016 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.016 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.016 DEBUG:boto:Params: {} 2023-09-22T21:08:49.016 DEBUG:boto:Token: None 2023-09-22T21:08:49.017 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.017 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S3vxCCoYC4YolzeyQuZXgXoxMbw= 2023-09-22T21:08:49.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S3vxCCoYC4YolzeyQuZXgXoxMbw=', 'Content-Length': '0'} 2023-09-22T21:08:49.020 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093d02b2540430c80-00650e0261-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.062 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:49.063 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:49.063 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.063 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.063 DEBUG:boto:Method: GET 2023-09-22T21:08:49.063 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.063 DEBUG:boto:Data: 2023-09-22T21:08:49.064 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.064 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.064 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.064 DEBUG:boto:Params: {} 2023-09-22T21:08:49.064 DEBUG:boto:Token: None 2023-09-22T21:08:49.065 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/ 2023-09-22T21:08:49.065 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU= 2023-09-22T21:08:49.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU=', 'Content-Length': '0'} 2023-09-22T21:08:49.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ffa9464585ea497-00650e0261-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.071 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:49.071 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:49.071 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:49.071 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.071 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.071 DEBUG:boto:Method: GET 2023-09-22T21:08:49.072 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.072 DEBUG:boto:Data: 2023-09-22T21:08:49.072 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.072 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.072 DEBUG:boto:Params: {} 2023-09-22T21:08:49.072 DEBUG:boto:Token: None 2023-09-22T21:08:49.072 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/ 2023-09-22T21:08:49.073 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU= 2023-09-22T21:08:49.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU=', 'Content-Length': '0'} 2023-09-22T21:08:49.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004763e660c1c54937-00650e0261-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.078 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:49.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:49.078 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.078 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.079 DEBUG:boto:path=/ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.079 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.079 DEBUG:boto:Method: GET 2023-09-22T21:08:49.079 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.079 DEBUG:boto:Data: 2023-09-22T21:08:49.079 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.079 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.079 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.079 DEBUG:boto:Params: {} 2023-09-22T21:08:49.079 DEBUG:boto:Token: None 2023-09-22T21:08:49.080 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.080 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P9t5lkWpX92y93mbVRdq+ORJuDs= 2023-09-22T21:08:49.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P9t5lkWpX92y93mbVRdq+ORJuDs=', 'Content-Length': '0'} 2023-09-22T21:08:49.082 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', '2t60n7HyrucItuaaoFxjoHWIjdIOj1g'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000c1e5997089eba6-00650e0261-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.083 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.083 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.083 DEBUG:boto:path=/ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.083 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.083 DEBUG:boto:Method: GET 2023-09-22T21:08:49.083 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.083 DEBUG:boto:Data: 2023-09-22T21:08:49.083 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.084 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.084 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.084 DEBUG:boto:Params: {} 2023-09-22T21:08:49.084 DEBUG:boto:Token: None 2023-09-22T21:08:49.084 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.084 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P9t5lkWpX92y93mbVRdq+ORJuDs= 2023-09-22T21:08:49.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P9t5lkWpX92y93mbVRdq+ORJuDs=', 'Content-Length': '0'} 2023-09-22T21:08:49.087 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', '2t60n7HyrucItuaaoFxjoHWIjdIOj1g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053fcf9b53ae96bb6-00650e0261-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.130 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.131 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.131 DEBUG:boto:path=/ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.131 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.131 DEBUG:boto:Method: HEAD 2023-09-22T21:08:49.131 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.131 DEBUG:boto:Data: 2023-09-22T21:08:49.131 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.132 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.132 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.132 DEBUG:boto:Params: {} 2023-09-22T21:08:49.132 DEBUG:boto:Token: None 2023-09-22T21:08:49.132 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.132 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bb/XAZPqiWpBwTiUMtGx8/FQ9Zo= 2023-09-22T21:08:49.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bb/XAZPqiWpBwTiUMtGx8/FQ9Zo=', 'Content-Length': '0'} 2023-09-22T21:08:49.135 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', '2t60n7HyrucItuaaoFxjoHWIjdIOj1g'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a827d7767ec8a56d-00650e0261-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.135 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.135 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.136 DEBUG:boto:path=/ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.136 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.136 DEBUG:boto:Method: HEAD 2023-09-22T21:08:49.136 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.136 DEBUG:boto:Data: 2023-09-22T21:08:49.136 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.136 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.136 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.137 DEBUG:boto:Params: {} 2023-09-22T21:08:49.137 DEBUG:boto:Token: None 2023-09-22T21:08:49.137 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.137 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bb/XAZPqiWpBwTiUMtGx8/FQ9Zo= 2023-09-22T21:08:49.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bb/XAZPqiWpBwTiUMtGx8/FQ9Zo=', 'Content-Length': '0'} 2023-09-22T21:08:49.140 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', '2t60n7HyrucItuaaoFxjoHWIjdIOj1g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009cd397481b9640dd-00650e0261-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:49.140 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.140 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.140 DEBUG:boto:path=/ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.140 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.140 DEBUG:boto:Method: GET 2023-09-22T21:08:49.141 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.141 DEBUG:boto:Data: 2023-09-22T21:08:49.141 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.141 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.141 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.141 DEBUG:boto:Params: {} 2023-09-22T21:08:49.141 DEBUG:boto:Token: None 2023-09-22T21:08:49.141 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.141 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P9t5lkWpX92y93mbVRdq+ORJuDs= 2023-09-22T21:08:49.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P9t5lkWpX92y93mbVRdq+ORJuDs=', 'Content-Length': '0'} 2023-09-22T21:08:49.145 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', '2t60n7HyrucItuaaoFxjoHWIjdIOj1g'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c71336ac6090d2a-00650e0261-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.145 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.145 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.145 DEBUG:boto:path=/ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.145 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.146 DEBUG:boto:Method: GET 2023-09-22T21:08:49.146 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.146 DEBUG:boto:Data: 2023-09-22T21:08:49.146 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.146 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.146 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.146 DEBUG:boto:Params: {} 2023-09-22T21:08:49.146 DEBUG:boto:Token: None 2023-09-22T21:08:49.146 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=2t60n7HyrucItuaaoFxjoHWIjdIOj1g 2023-09-22T21:08:49.147 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P9t5lkWpX92y93mbVRdq+ORJuDs= 2023-09-22T21:08:49.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P9t5lkWpX92y93mbVRdq+ORJuDs=', 'Content-Length': '0'} 2023-09-22T21:08:49.149 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', '2t60n7HyrucItuaaoFxjoHWIjdIOj1g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ba61e8bffa7dd12-00650e0261-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.194 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:49.195 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:49.195 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.195 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.195 DEBUG:boto:Method: GET 2023-09-22T21:08:49.195 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.195 DEBUG:boto:Data: 2023-09-22T21:08:49.195 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.195 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.196 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.196 DEBUG:boto:Params: {} 2023-09-22T21:08:49.196 DEBUG:boto:Token: None 2023-09-22T21:08:49.196 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/ 2023-09-22T21:08:49.196 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU= 2023-09-22T21:08:49.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU=', 'Content-Length': '0'} 2023-09-22T21:08:49.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4e356d19f2713ca-00650e0261-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.202 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:49.202 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:49.202 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:49.202 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.203 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.203 DEBUG:boto:Method: GET 2023-09-22T21:08:49.203 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.203 DEBUG:boto:Data: 2023-09-22T21:08:49.203 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.203 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.203 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.203 DEBUG:boto:Params: {} 2023-09-22T21:08:49.203 DEBUG:boto:Token: None 2023-09-22T21:08:49.204 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/ 2023-09-22T21:08:49.204 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU= 2023-09-22T21:08:49.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU=', 'Content-Length': '0'} 2023-09-22T21:08:49.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d4e76ea5f892177-00650e0261-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.209 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:49.209 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.209 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.209 DEBUG:boto:Method: HEAD 2023-09-22T21:08:49.209 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:49.210 DEBUG:boto:Data: 2023-09-22T21:08:49.210 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.210 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.210 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.210 DEBUG:boto:Params: {} 2023-09-22T21:08:49.210 DEBUG:boto:Token: None 2023-09-22T21:08:49.210 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj 2023-09-22T21:08:49.210 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1NL5pvoew7kuOYgPymZTx/KqYeY= 2023-09-22T21:08:49.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1NL5pvoew7kuOYgPymZTx/KqYeY=', 'Content-Length': '0'} 2023-09-22T21:08:49.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000031f30fcdd15a356-00650e0261-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.216 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.216 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.216 DEBUG:boto:Method: HEAD 2023-09-22T21:08:49.217 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:49.217 DEBUG:boto:Data: 2023-09-22T21:08:49.217 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.217 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.217 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.217 DEBUG:boto:Params: {} 2023-09-22T21:08:49.217 DEBUG:boto:Token: None 2023-09-22T21:08:49.217 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj 2023-09-22T21:08:49.217 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1NL5pvoew7kuOYgPymZTx/KqYeY= 2023-09-22T21:08:49.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1NL5pvoew7kuOYgPymZTx/KqYeY=', 'Content-Length': '0'} 2023-09-22T21:08:49.223 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094d0862e38f2f921-00650e0261-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:49.224 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.224 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.224 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.224 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.224 DEBUG:boto:Method: GET 2023-09-22T21:08:49.224 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.224 DEBUG:boto:Data: 2023-09-22T21:08:49.224 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.224 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.224 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.225 DEBUG:boto:Params: {} 2023-09-22T21:08:49.225 DEBUG:boto:Token: None 2023-09-22T21:08:49.225 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.225 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S3vxCCoYC4YolzeyQuZXgXoxMbw= 2023-09-22T21:08:49.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S3vxCCoYC4YolzeyQuZXgXoxMbw=', 'Content-Length': '0'} 2023-09-22T21:08:49.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062f066669de5badb-00650e0261-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.270 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.270 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.271 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.271 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.271 DEBUG:boto:Method: GET 2023-09-22T21:08:49.271 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.271 DEBUG:boto:Data: 2023-09-22T21:08:49.271 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.271 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.271 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.271 DEBUG:boto:Params: {} 2023-09-22T21:08:49.272 DEBUG:boto:Token: None 2023-09-22T21:08:49.272 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.272 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S3vxCCoYC4YolzeyQuZXgXoxMbw= 2023-09-22T21:08:49.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S3vxCCoYC4YolzeyQuZXgXoxMbw=', 'Content-Length': '0'} 2023-09-22T21:08:49.275 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000722a7f6c3f7f1836-00650e0261-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.318 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:49.319 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:49.319 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.319 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.319 DEBUG:boto:Method: GET 2023-09-22T21:08:49.319 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.319 DEBUG:boto:Data: 2023-09-22T21:08:49.319 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.319 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.319 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.320 DEBUG:boto:Params: {} 2023-09-22T21:08:49.320 DEBUG:boto:Token: None 2023-09-22T21:08:49.320 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/ 2023-09-22T21:08:49.320 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU= 2023-09-22T21:08:49.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU=', 'Content-Length': '0'} 2023-09-22T21:08:49.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000138094f66f658868-00650e0261-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.325 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:49.326 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:49.326 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:49.326 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.326 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.326 DEBUG:boto:Method: GET 2023-09-22T21:08:49.327 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.327 DEBUG:boto:Data: 2023-09-22T21:08:49.327 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.327 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.327 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.327 DEBUG:boto:Params: {} 2023-09-22T21:08:49.327 DEBUG:boto:Token: None 2023-09-22T21:08:49.327 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/ 2023-09-22T21:08:49.327 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU= 2023-09-22T21:08:49.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU=', 'Content-Length': '0'} 2023-09-22T21:08:49.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f878b59117d17fea-00650e0261-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.332 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:49.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:49.333 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.333 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.333 DEBUG:boto:path=/ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.333 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.333 DEBUG:boto:Method: GET 2023-09-22T21:08:49.333 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.333 DEBUG:boto:Data: 2023-09-22T21:08:49.333 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.334 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.334 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.334 DEBUG:boto:Params: {} 2023-09-22T21:08:49.334 DEBUG:boto:Token: None 2023-09-22T21:08:49.334 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.334 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9q/m54f+BnSaPNZWSSkCV3mSr6U= 2023-09-22T21:08:49.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9q/m54f+BnSaPNZWSSkCV3mSr6U=', 'Content-Length': '0'} 2023-09-22T21:08:49.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019a00ad4cdce7c81-00650e0261-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.378 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.378 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.378 DEBUG:boto:path=/ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.379 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.379 DEBUG:boto:Method: GET 2023-09-22T21:08:49.379 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.379 DEBUG:boto:Data: 2023-09-22T21:08:49.379 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.379 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.379 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.379 DEBUG:boto:Params: {} 2023-09-22T21:08:49.379 DEBUG:boto:Token: None 2023-09-22T21:08:49.380 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.380 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9q/m54f+BnSaPNZWSSkCV3mSr6U= 2023-09-22T21:08:49.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9q/m54f+BnSaPNZWSSkCV3mSr6U=', 'Content-Length': '0'} 2023-09-22T21:08:49.382 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000328b7d8bfd932f8-00650e0261-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.383 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.383 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.383 DEBUG:boto:path=/ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.383 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.383 DEBUG:boto:Method: HEAD 2023-09-22T21:08:49.383 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.383 DEBUG:boto:Data: 2023-09-22T21:08:49.383 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.384 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.384 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.384 DEBUG:boto:Params: {} 2023-09-22T21:08:49.384 DEBUG:boto:Token: None 2023-09-22T21:08:49.384 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.384 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9ThZPDTE1S8wrOK3PrKlhXfHNkY= 2023-09-22T21:08:49.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9ThZPDTE1S8wrOK3PrKlhXfHNkY=', 'Content-Length': '0'} 2023-09-22T21:08:49.387 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a5e271e2d3e0224-00650e0261-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.387 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.387 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.387 DEBUG:boto:path=/ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.387 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.387 DEBUG:boto:Method: HEAD 2023-09-22T21:08:49.387 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.388 DEBUG:boto:Data: 2023-09-22T21:08:49.388 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.388 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.388 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.388 DEBUG:boto:Params: {} 2023-09-22T21:08:49.388 DEBUG:boto:Token: None 2023-09-22T21:08:49.388 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.388 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9ThZPDTE1S8wrOK3PrKlhXfHNkY= 2023-09-22T21:08:49.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9ThZPDTE1S8wrOK3PrKlhXfHNkY=', 'Content-Length': '0'} 2023-09-22T21:08:49.391 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2f5aae571799b98-00650e0261-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:49.391 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.391 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.391 DEBUG:boto:path=/ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.392 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.392 DEBUG:boto:Method: GET 2023-09-22T21:08:49.392 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.392 DEBUG:boto:Data: 2023-09-22T21:08:49.392 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.392 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.392 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.392 DEBUG:boto:Params: {} 2023-09-22T21:08:49.392 DEBUG:boto:Token: None 2023-09-22T21:08:49.392 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.393 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9q/m54f+BnSaPNZWSSkCV3mSr6U= 2023-09-22T21:08:49.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9q/m54f+BnSaPNZWSSkCV3mSr6U=', 'Content-Length': '0'} 2023-09-22T21:08:49.395 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000fa35054175085f8-00650e0261-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.438 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.439 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.439 DEBUG:boto:path=/ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.439 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.439 DEBUG:boto:Method: GET 2023-09-22T21:08:49.439 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.439 DEBUG:boto:Data: 2023-09-22T21:08:49.439 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.439 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.439 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.440 DEBUG:boto:Params: {} 2023-09-22T21:08:49.440 DEBUG:boto:Token: None 2023-09-22T21:08:49.440 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc 2023-09-22T21:08:49.440 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9q/m54f+BnSaPNZWSSkCV3mSr6U= 2023-09-22T21:08:49.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9q/m54f+BnSaPNZWSSkCV3mSr6U=', 'Content-Length': '0'} 2023-09-22T21:08:49.443 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'Vi5UgYUqMidEG29Nj6qJbNhM09I8CJc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095e9e3ad051fbc52-00650e0261-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.443 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:49.443 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:49.443 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.443 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.443 DEBUG:boto:Method: GET 2023-09-22T21:08:49.444 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.444 DEBUG:boto:Data: 2023-09-22T21:08:49.444 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.444 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.444 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.444 DEBUG:boto:Params: {} 2023-09-22T21:08:49.444 DEBUG:boto:Token: None 2023-09-22T21:08:49.444 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/ 2023-09-22T21:08:49.444 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU= 2023-09-22T21:08:49.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU=', 'Content-Length': '0'} 2023-09-22T21:08:49.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b15d4fd304d4cf1c-00650e0261-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.450 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:49.450 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:49.451 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:49.451 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.451 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.451 DEBUG:boto:Method: GET 2023-09-22T21:08:49.451 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.451 DEBUG:boto:Data: 2023-09-22T21:08:49.451 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.451 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.451 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.452 DEBUG:boto:Params: {} 2023-09-22T21:08:49.452 DEBUG:boto:Token: None 2023-09-22T21:08:49.452 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/ 2023-09-22T21:08:49.452 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU= 2023-09-22T21:08:49.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU=', 'Content-Length': '0'} 2023-09-22T21:08:49.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c7c0bd5f5eaaf32-00650e0261-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.498 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:49.499 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.499 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.499 DEBUG:boto:Method: HEAD 2023-09-22T21:08:49.499 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:49.499 DEBUG:boto:Data: 2023-09-22T21:08:49.499 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.500 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.500 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.500 DEBUG:boto:Params: {} 2023-09-22T21:08:49.500 DEBUG:boto:Token: None 2023-09-22T21:08:49.500 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj 2023-09-22T21:08:49.500 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1NL5pvoew7kuOYgPymZTx/KqYeY= 2023-09-22T21:08:49.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1NL5pvoew7kuOYgPymZTx/KqYeY=', 'Content-Length': '0'} 2023-09-22T21:08:49.506 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000855807b7976775dc-00650e0261-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.506 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.506 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.506 DEBUG:boto:Method: HEAD 2023-09-22T21:08:49.506 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:49.507 DEBUG:boto:Data: 2023-09-22T21:08:49.507 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.507 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.507 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.507 DEBUG:boto:Params: {} 2023-09-22T21:08:49.507 DEBUG:boto:Token: None 2023-09-22T21:08:49.507 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj 2023-09-22T21:08:49.508 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1NL5pvoew7kuOYgPymZTx/KqYeY= 2023-09-22T21:08:49.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1NL5pvoew7kuOYgPymZTx/KqYeY=', 'Content-Length': '0'} 2023-09-22T21:08:49.513 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060f927ef077683cc-00650e0261-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:49.514 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.514 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.514 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.514 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.514 DEBUG:boto:Method: GET 2023-09-22T21:08:49.514 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.514 DEBUG:boto:Data: 2023-09-22T21:08:49.514 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.514 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.514 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.515 DEBUG:boto:Params: {} 2023-09-22T21:08:49.515 DEBUG:boto:Token: None 2023-09-22T21:08:49.515 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.515 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S3vxCCoYC4YolzeyQuZXgXoxMbw= 2023-09-22T21:08:49.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S3vxCCoYC4YolzeyQuZXgXoxMbw=', 'Content-Length': '0'} 2023-09-22T21:08:49.518 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064e3cd9d3c12761c-00650e0261-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.518 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.518 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.518 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.519 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.519 DEBUG:boto:Method: GET 2023-09-22T21:08:49.519 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.519 DEBUG:boto:Data: 2023-09-22T21:08:49.519 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.519 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.519 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.519 DEBUG:boto:Params: {} 2023-09-22T21:08:49.520 DEBUG:boto:Token: None 2023-09-22T21:08:49.520 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.520 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S3vxCCoYC4YolzeyQuZXgXoxMbw= 2023-09-22T21:08:49.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S3vxCCoYC4YolzeyQuZXgXoxMbw=', 'Content-Length': '0'} 2023-09-22T21:08:49.522 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047af4752ea2b671e-00650e0261-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.566 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:49.566 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:49.567 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.567 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.567 DEBUG:boto:Method: GET 2023-09-22T21:08:49.567 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.567 DEBUG:boto:Data: 2023-09-22T21:08:49.567 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.567 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.567 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.567 DEBUG:boto:Params: {} 2023-09-22T21:08:49.568 DEBUG:boto:Token: None 2023-09-22T21:08:49.568 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/ 2023-09-22T21:08:49.568 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU= 2023-09-22T21:08:49.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU=', 'Content-Length': '0'} 2023-09-22T21:08:49.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045a0d584edc21576-00650e0261-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.573 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:49.574 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:49.574 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:49.574 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.574 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.574 DEBUG:boto:Method: GET 2023-09-22T21:08:49.574 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.574 DEBUG:boto:Data: 2023-09-22T21:08:49.574 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.575 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.575 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.575 DEBUG:boto:Params: {} 2023-09-22T21:08:49.575 DEBUG:boto:Token: None 2023-09-22T21:08:49.575 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/ 2023-09-22T21:08:49.575 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU= 2023-09-22T21:08:49.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU=', 'Content-Length': '0'} 2023-09-22T21:08:49.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8e511bf2a4eeab3-00650e0261-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.622 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:49.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:49.623 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.623 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.623 DEBUG:boto:path=/ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.623 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.623 DEBUG:boto:Method: GET 2023-09-22T21:08:49.623 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.623 DEBUG:boto:Data: 2023-09-22T21:08:49.624 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.624 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.624 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.624 DEBUG:boto:Params: {} 2023-09-22T21:08:49.624 DEBUG:boto:Token: None 2023-09-22T21:08:49.624 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.624 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7DcY5UfoCx+vEkVIeXmjYSbPGp4= 2023-09-22T21:08:49.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7DcY5UfoCx+vEkVIeXmjYSbPGp4=', 'Content-Length': '0'} 2023-09-22T21:08:49.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'hn7HL8wXg0x8WecE.JHMlcrta8XT0VL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b647151d1061573-00650e0261-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.670 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.670 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.671 DEBUG:boto:path=/ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.671 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.671 DEBUG:boto:Method: GET 2023-09-22T21:08:49.671 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.671 DEBUG:boto:Data: 2023-09-22T21:08:49.671 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.671 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.671 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.671 DEBUG:boto:Params: {} 2023-09-22T21:08:49.672 DEBUG:boto:Token: None 2023-09-22T21:08:49.672 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.672 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7DcY5UfoCx+vEkVIeXmjYSbPGp4= 2023-09-22T21:08:49.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7DcY5UfoCx+vEkVIeXmjYSbPGp4=', 'Content-Length': '0'} 2023-09-22T21:08:49.675 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'hn7HL8wXg0x8WecE.JHMlcrta8XT0VL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f51a78f2658edb25-00650e0261-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.675 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.675 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.675 DEBUG:boto:path=/ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.675 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.675 DEBUG:boto:Method: HEAD 2023-09-22T21:08:49.675 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.676 DEBUG:boto:Data: 2023-09-22T21:08:49.676 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.676 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.676 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.676 DEBUG:boto:Params: {} 2023-09-22T21:08:49.676 DEBUG:boto:Token: None 2023-09-22T21:08:49.676 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.676 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DfOX4WTh9dshq44GYCngTPG2YGw= 2023-09-22T21:08:49.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DfOX4WTh9dshq44GYCngTPG2YGw=', 'Content-Length': '0'} 2023-09-22T21:08:49.679 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'hn7HL8wXg0x8WecE.JHMlcrta8XT0VL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000373d35c14a344a64-00650e0261-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.679 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.679 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.679 DEBUG:boto:path=/ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.680 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.680 DEBUG:boto:Method: HEAD 2023-09-22T21:08:49.680 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.680 DEBUG:boto:Data: 2023-09-22T21:08:49.680 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.680 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.680 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.680 DEBUG:boto:Params: {} 2023-09-22T21:08:49.680 DEBUG:boto:Token: None 2023-09-22T21:08:49.680 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.681 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DfOX4WTh9dshq44GYCngTPG2YGw= 2023-09-22T21:08:49.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DfOX4WTh9dshq44GYCngTPG2YGw=', 'Content-Length': '0'} 2023-09-22T21:08:49.683 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'hn7HL8wXg0x8WecE.JHMlcrta8XT0VL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2e312adc835e983-00650e0261-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:49.684 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.684 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.684 DEBUG:boto:path=/ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.684 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.684 DEBUG:boto:Method: GET 2023-09-22T21:08:49.684 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.684 DEBUG:boto:Data: 2023-09-22T21:08:49.684 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.684 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.684 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.685 DEBUG:boto:Params: {} 2023-09-22T21:08:49.685 DEBUG:boto:Token: None 2023-09-22T21:08:49.685 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.685 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7DcY5UfoCx+vEkVIeXmjYSbPGp4= 2023-09-22T21:08:49.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7DcY5UfoCx+vEkVIeXmjYSbPGp4=', 'Content-Length': '0'} 2023-09-22T21:08:49.688 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'hn7HL8wXg0x8WecE.JHMlcrta8XT0VL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b314b4665c4f5e02-00650e0261-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.730 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.731 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.731 DEBUG:boto:path=/ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.731 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.731 DEBUG:boto:Method: GET 2023-09-22T21:08:49.731 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.731 DEBUG:boto:Data: 2023-09-22T21:08:49.731 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.731 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.731 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.731 DEBUG:boto:Params: {} 2023-09-22T21:08:49.732 DEBUG:boto:Token: None 2023-09-22T21:08:49.732 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=hn7HL8wXg0x8WecE.JHMlcrta8XT0VL 2023-09-22T21:08:49.732 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7DcY5UfoCx+vEkVIeXmjYSbPGp4= 2023-09-22T21:08:49.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7DcY5UfoCx+vEkVIeXmjYSbPGp4=', 'Content-Length': '0'} 2023-09-22T21:08:49.735 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'hn7HL8wXg0x8WecE.JHMlcrta8XT0VL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004605ef5bc734515e-00650e0261-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.735 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:49.735 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:49.735 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.735 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.736 DEBUG:boto:Method: GET 2023-09-22T21:08:49.736 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.736 DEBUG:boto:Data: 2023-09-22T21:08:49.736 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.736 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.736 DEBUG:boto:Params: {} 2023-09-22T21:08:49.736 DEBUG:boto:Token: None 2023-09-22T21:08:49.736 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/ 2023-09-22T21:08:49.737 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU= 2023-09-22T21:08:49.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU=', 'Content-Length': '0'} 2023-09-22T21:08:49.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0fa96e763aad824-00650e0261-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.745 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:49.745 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:49.745 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:49.745 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.746 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.746 DEBUG:boto:Method: GET 2023-09-22T21:08:49.746 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.746 DEBUG:boto:Data: 2023-09-22T21:08:49.746 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.746 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.746 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.746 DEBUG:boto:Params: {} 2023-09-22T21:08:49.746 DEBUG:boto:Token: None 2023-09-22T21:08:49.747 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/ 2023-09-22T21:08:49.747 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU= 2023-09-22T21:08:49.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU=', 'Content-Length': '0'} 2023-09-22T21:08:49.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004454b64999c59a6e-00650e0261-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.754 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:49.754 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.754 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.754 DEBUG:boto:Method: HEAD 2023-09-22T21:08:49.754 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:49.754 DEBUG:boto:Data: 2023-09-22T21:08:49.755 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.755 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.755 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.755 DEBUG:boto:Params: {} 2023-09-22T21:08:49.755 DEBUG:boto:Token: None 2023-09-22T21:08:49.755 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj 2023-09-22T21:08:49.755 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1NL5pvoew7kuOYgPymZTx/KqYeY= 2023-09-22T21:08:49.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1NL5pvoew7kuOYgPymZTx/KqYeY=', 'Content-Length': '0'} 2023-09-22T21:08:49.761 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000580ec80804a34b3b-00650e0261-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.761 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.762 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.762 DEBUG:boto:Method: HEAD 2023-09-22T21:08:49.762 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:49.762 DEBUG:boto:Data: 2023-09-22T21:08:49.762 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.762 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.762 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.762 DEBUG:boto:Params: {} 2023-09-22T21:08:49.762 DEBUG:boto:Token: None 2023-09-22T21:08:49.763 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj 2023-09-22T21:08:49.763 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1NL5pvoew7kuOYgPymZTx/KqYeY= 2023-09-22T21:08:49.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1NL5pvoew7kuOYgPymZTx/KqYeY=', 'Content-Length': '0'} 2023-09-22T21:08:49.768 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e336020750a9e637-00650e0261-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:49.769 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.769 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.770 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.770 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.770 DEBUG:boto:Method: GET 2023-09-22T21:08:49.770 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.771 DEBUG:boto:Data: 2023-09-22T21:08:49.771 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.771 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.771 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.771 DEBUG:boto:Params: {} 2023-09-22T21:08:49.772 DEBUG:boto:Token: None 2023-09-22T21:08:49.772 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.772 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S3vxCCoYC4YolzeyQuZXgXoxMbw= 2023-09-22T21:08:49.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S3vxCCoYC4YolzeyQuZXgXoxMbw=', 'Content-Length': '0'} 2023-09-22T21:08:49.775 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c387e4570f25ec6-00650e0261-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.818 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.819 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.819 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.819 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.819 DEBUG:boto:Method: GET 2023-09-22T21:08:49.819 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.819 DEBUG:boto:Data: 2023-09-22T21:08:49.819 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.820 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.820 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.820 DEBUG:boto:Params: {} 2023-09-22T21:08:49.820 DEBUG:boto:Token: None 2023-09-22T21:08:49.820 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:49.820 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S3vxCCoYC4YolzeyQuZXgXoxMbw= 2023-09-22T21:08:49.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S3vxCCoYC4YolzeyQuZXgXoxMbw=', 'Content-Length': '0'} 2023-09-22T21:08:49.823 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc8ac3efa2639406-00650e0261-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.866 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:49.866 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:49.867 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.867 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.867 DEBUG:boto:Method: GET 2023-09-22T21:08:49.867 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.867 DEBUG:boto:Data: 2023-09-22T21:08:49.867 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.867 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.867 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.867 DEBUG:boto:Params: {} 2023-09-22T21:08:49.867 DEBUG:boto:Token: None 2023-09-22T21:08:49.868 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/ 2023-09-22T21:08:49.868 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU= 2023-09-22T21:08:49.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU=', 'Content-Length': '0'} 2023-09-22T21:08:49.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000438dd1641925fecb-00650e0261-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.874 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:49.874 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:49.874 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:49.874 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.875 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.875 DEBUG:boto:Method: GET 2023-09-22T21:08:49.875 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.875 DEBUG:boto:Data: 2023-09-22T21:08:49.875 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.875 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.875 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.875 DEBUG:boto:Params: {} 2023-09-22T21:08:49.875 DEBUG:boto:Token: None 2023-09-22T21:08:49.875 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/ 2023-09-22T21:08:49.876 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU= 2023-09-22T21:08:49.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU=', 'Content-Length': '0'} 2023-09-22T21:08:49.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8f1e3f09d92af80-00650e0261-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.881 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:49.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:49.881 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.882 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.882 DEBUG:boto:path=/ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.882 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.882 DEBUG:boto:Method: GET 2023-09-22T21:08:49.882 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.882 DEBUG:boto:Data: 2023-09-22T21:08:49.882 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.882 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.882 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.882 DEBUG:boto:Params: {} 2023-09-22T21:08:49.883 DEBUG:boto:Token: None 2023-09-22T21:08:49.883 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.883 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22yDfxCHf19ctuo7GIDVWLN2EV4= 2023-09-22T21:08:49.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22yDfxCHf19ctuo7GIDVWLN2EV4=', 'Content-Length': '0'} 2023-09-22T21:08:49.886 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca78fc10eff8f016-00650e0261-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.886 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.886 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.886 DEBUG:boto:path=/ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.886 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.886 DEBUG:boto:Method: GET 2023-09-22T21:08:49.886 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.887 DEBUG:boto:Data: 2023-09-22T21:08:49.887 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.887 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.887 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.887 DEBUG:boto:Params: {} 2023-09-22T21:08:49.887 DEBUG:boto:Token: None 2023-09-22T21:08:49.887 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.887 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22yDfxCHf19ctuo7GIDVWLN2EV4= 2023-09-22T21:08:49.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22yDfxCHf19ctuo7GIDVWLN2EV4=', 'Content-Length': '0'} 2023-09-22T21:08:49.890 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ce2480f644a8b52-00650e0261-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.934 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.934 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.935 DEBUG:boto:path=/ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.935 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.935 DEBUG:boto:Method: HEAD 2023-09-22T21:08:49.935 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.935 DEBUG:boto:Data: 2023-09-22T21:08:49.935 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.935 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.935 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.935 DEBUG:boto:Params: {} 2023-09-22T21:08:49.935 DEBUG:boto:Token: None 2023-09-22T21:08:49.936 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.936 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0k2MlSyym0NdAIPtN9W1zTzqh+0= 2023-09-22T21:08:49.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0k2MlSyym0NdAIPtN9W1zTzqh+0=', 'Content-Length': '0'} 2023-09-22T21:08:49.938 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025f57b721721af2f-00650e0261-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.939 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.939 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.939 DEBUG:boto:path=/ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.939 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.939 DEBUG:boto:Method: HEAD 2023-09-22T21:08:49.939 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.939 DEBUG:boto:Data: 2023-09-22T21:08:49.939 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.939 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.940 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.940 DEBUG:boto:Params: {} 2023-09-22T21:08:49.940 DEBUG:boto:Token: None 2023-09-22T21:08:49.940 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.940 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0k2MlSyym0NdAIPtN9W1zTzqh+0= 2023-09-22T21:08:49.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0k2MlSyym0NdAIPtN9W1zTzqh+0=', 'Content-Length': '0'} 2023-09-22T21:08:49.943 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f260b3a58be7f99-00650e0261-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:49.943 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.943 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.943 DEBUG:boto:path=/ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.943 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.943 DEBUG:boto:Method: GET 2023-09-22T21:08:49.944 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.944 DEBUG:boto:Data: 2023-09-22T21:08:49.944 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.944 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.944 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.944 DEBUG:boto:Params: {} 2023-09-22T21:08:49.944 DEBUG:boto:Token: None 2023-09-22T21:08:49.944 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.944 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22yDfxCHf19ctuo7GIDVWLN2EV4= 2023-09-22T21:08:49.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22yDfxCHf19ctuo7GIDVWLN2EV4=', 'Content-Length': '0'} 2023-09-22T21:08:49.947 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e938c89579263da-00650e0261-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.947 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:49.947 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:49.947 DEBUG:boto:path=/ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.948 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.948 DEBUG:boto:Method: GET 2023-09-22T21:08:49.948 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.948 DEBUG:boto:Data: 2023-09-22T21:08:49.948 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.948 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:49.948 DEBUG:boto:Port: 8000 2023-09-22T21:08:49.948 DEBUG:boto:Params: {} 2023-09-22T21:08:49.948 DEBUG:boto:Token: None 2023-09-22T21:08:49.948 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/obj?versionId=nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM 2023-09-22T21:08:49.949 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22yDfxCHf19ctuo7GIDVWLN2EV4= 2023-09-22T21:08:49.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22yDfxCHf19ctuo7GIDVWLN2EV4=', 'Content-Length': '0'} 2023-09-22T21:08:49.951 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:22 GMT'), ('x-amz-version-id', 'nJHDh5xJ5xSZF6I3lmFHoo1.MEBF9vM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045ea92123e8ebd83-00650e0261-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:49.994 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:49.994 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:49.995 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.995 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.995 DEBUG:boto:Method: GET 2023-09-22T21:08:49.995 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:49.995 DEBUG:boto:Data: 2023-09-22T21:08:49.995 DEBUG:boto:Headers: {} 2023-09-22T21:08:49.995 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:49.995 DEBUG:boto:Port: 8001 2023-09-22T21:08:49.995 DEBUG:boto:Params: {} 2023-09-22T21:08:49.995 DEBUG:boto:Token: None 2023-09-22T21:08:49.996 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:49 GMT /ffzdqu-13/ 2023-09-22T21:08:49.996 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU= 2023-09-22T21:08:49.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:slXvc7xOTAfghwKOmCVzN7hKSYU=', 'Content-Length': '0'} 2023-09-22T21:08:50.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aebc4682daa0609-00650e0261-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:50.002 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:50.002 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:50.002 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:50.002 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:50.003 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:50.003 DEBUG:boto:Method: GET 2023-09-22T21:08:50.003 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:50.003 DEBUG:boto:Data: 2023-09-22T21:08:50.003 DEBUG:boto:Headers: {} 2023-09-22T21:08:50.003 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:50.003 DEBUG:boto:Port: 8000 2023-09-22T21:08:50.003 DEBUG:boto:Params: {} 2023-09-22T21:08:50.003 DEBUG:boto:Token: None 2023-09-22T21:08:50.003 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:50 GMT /ffzdqu-13/ 2023-09-22T21:08:50.004 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CHF+g3IDM+uat19Vx5inkPphDNE= 2023-09-22T21:08:50.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CHF+g3IDM+uat19Vx5inkPphDNE=', 'Content-Length': '0'} 2023-09-22T21:08:50.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000748c6b880b9f6d1a-00650e0262-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:50.009 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:50.009 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:50.009 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:50.009 DEBUG:boto:Method: HEAD 2023-09-22T21:08:50.010 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:50.010 DEBUG:boto:Data: 2023-09-22T21:08:50.010 DEBUG:boto:Headers: {} 2023-09-22T21:08:50.010 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:50.010 DEBUG:boto:Port: 8001 2023-09-22T21:08:50.010 DEBUG:boto:Params: {} 2023-09-22T21:08:50.010 DEBUG:boto:Token: None 2023-09-22T21:08:50.010 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:50 GMT /ffzdqu-13/obj 2023-09-22T21:08:50.010 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mWfABvPmrZOXCO1Ni6hlrF+uyxM= 2023-09-22T21:08:50.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mWfABvPmrZOXCO1Ni6hlrF+uyxM=', 'Content-Length': '0'} 2023-09-22T21:08:50.016 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ad5dba8b1395707-00650e0262-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:50.016 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:50.016 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:50.016 DEBUG:boto:Method: HEAD 2023-09-22T21:08:50.017 DEBUG:boto:Path: /ffzdqu-13/obj 2023-09-22T21:08:50.017 DEBUG:boto:Data: 2023-09-22T21:08:50.017 DEBUG:boto:Headers: {} 2023-09-22T21:08:50.017 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:50.017 DEBUG:boto:Port: 8000 2023-09-22T21:08:50.017 DEBUG:boto:Params: {} 2023-09-22T21:08:50.017 DEBUG:boto:Token: None 2023-09-22T21:08:50.017 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:50 GMT /ffzdqu-13/obj 2023-09-22T21:08:50.017 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mWfABvPmrZOXCO1Ni6hlrF+uyxM= 2023-09-22T21:08:50.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mWfABvPmrZOXCO1Ni6hlrF+uyxM=', 'Content-Length': '0'} 2023-09-22T21:08:50.023 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a709e60c59ace67-00650e0262-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:50.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:08:50.023 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:50.023 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:50.024 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:50.024 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:50.024 DEBUG:boto:Method: GET 2023-09-22T21:08:50.024 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:50.024 DEBUG:boto:Data: 2023-09-22T21:08:50.024 DEBUG:boto:Headers: {} 2023-09-22T21:08:50.024 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:50.024 DEBUG:boto:Port: 8001 2023-09-22T21:08:50.024 DEBUG:boto:Params: {} 2023-09-22T21:08:50.024 DEBUG:boto:Token: None 2023-09-22T21:08:50.025 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:50 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:50.025 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:15WC5+aW9eXAQU6aY0Os5tTNp/Y= 2023-09-22T21:08:50.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:15WC5+aW9eXAQU6aY0Os5tTNp/Y=', 'Content-Length': '0'} 2023-09-22T21:08:50.027 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b182d31941eb86e-00650e0262-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:50.070 DEBUG:boto:path=/ffzdqu-13/obj 2023-09-22T21:08:50.071 DEBUG:boto:auth_path=/ffzdqu-13/obj 2023-09-22T21:08:50.071 DEBUG:boto:path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:50.071 DEBUG:boto:auth_path=/ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:50.071 DEBUG:boto:Method: GET 2023-09-22T21:08:50.071 DEBUG:boto:Path: /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:50.072 DEBUG:boto:Data: 2023-09-22T21:08:50.072 DEBUG:boto:Headers: {} 2023-09-22T21:08:50.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:50.072 DEBUG:boto:Port: 8000 2023-09-22T21:08:50.072 DEBUG:boto:Params: {} 2023-09-22T21:08:50.073 DEBUG:boto:Token: None 2023-09-22T21:08:50.073 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:50 GMT /ffzdqu-13/obj?versionId=VPOyc5MFRGSz2wBgBorKv2oh8K4FSny 2023-09-22T21:08:50.073 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:15WC5+aW9eXAQU6aY0Os5tTNp/Y= 2023-09-22T21:08:50.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:15WC5+aW9eXAQU6aY0Os5tTNp/Y=', 'Content-Length': '0'} 2023-09-22T21:08:50.076 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:23 GMT'), ('x-amz-version-id', 'VPOyc5MFRGSz2wBgBorKv2oh8K4FSny'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000463709fa76072c1-00650e0262-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:50.122 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:50.123 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:50.123 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:50.123 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:50.123 DEBUG:boto:Method: GET 2023-09-22T21:08:50.123 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:50.123 DEBUG:boto:Data: 2023-09-22T21:08:50.123 DEBUG:boto:Headers: {} 2023-09-22T21:08:50.123 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:08:50.124 DEBUG:boto:Port: 8001 2023-09-22T21:08:50.124 DEBUG:boto:Params: {} 2023-09-22T21:08:50.124 DEBUG:boto:Token: None 2023-09-22T21:08:50.124 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:50 GMT /ffzdqu-13/ 2023-09-22T21:08:50.124 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CHF+g3IDM+uat19Vx5inkPphDNE= 2023-09-22T21:08:50.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CHF+g3IDM+uat19Vx5inkPphDNE=', 'Content-Length': '0'} 2023-09-22T21:08:50.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095d19e9112be6e21-00650e0262-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:50.130 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:50.130 DEBUG:boto:path=/ffzdqu-13/ 2023-09-22T21:08:50.130 DEBUG:boto:auth_path=/ffzdqu-13/ 2023-09-22T21:08:50.130 DEBUG:boto:path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:50.130 DEBUG:boto:auth_path=/ffzdqu-13/?prefix=obj 2023-09-22T21:08:50.131 DEBUG:boto:Method: GET 2023-09-22T21:08:50.131 DEBUG:boto:Path: /ffzdqu-13/?prefix=obj 2023-09-22T21:08:50.131 DEBUG:boto:Data: 2023-09-22T21:08:50.131 DEBUG:boto:Headers: {} 2023-09-22T21:08:50.131 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:50.132 DEBUG:boto:Port: 8000 2023-09-22T21:08:50.132 DEBUG:boto:Params: {} 2023-09-22T21:08:50.132 DEBUG:boto:Token: None 2023-09-22T21:08:50.133 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:08:50 GMT /ffzdqu-13/ 2023-09-22T21:08:50.133 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CHF+g3IDM+uat19Vx5inkPphDNE= 2023-09-22T21:08:50.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CHF+g3IDM+uat19Vx5inkPphDNE=', 'Content-Length': '0'} 2023-09-22T21:08:50.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065193ef324843b2f-00650e0262-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:50.138 DEBUG:boto:b'ffzdqu-13obj1000falseobj2023-09-22T21:08:23.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:08:50.139 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-13 zones={a2, a1} 2023-09-22T21:08:50.139 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-09-22T21:08:50.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:08:50.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:08:50.139 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:08:50.182 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:08:50.182 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:08:50.244 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"00000000000000000000:00000000000000000651","last_update":"2023-09-22T21:04:42.948773Z"},{"marker":"00000000000000000000:00000000000000001491","last_update":"2023-09-22T21:08:32.257204Z"},{"marker":"00000000000000000000:00000000000000002204","last_update":"2023-09-22T21:08:19.579516Z"},{"marker":"00000000000000000000:00000000000000000929","last_update":"2023-09-22T21:04:39.576975Z"}] 2023-09-22T21:08:50.245 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000001491', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000000929'} 2023-09-22T21:08:50.245 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-22T21:08:50.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:08:50.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:08:50.245 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:08:50.326 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:08:50.326 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:08:50.382 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:42.948773Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001491","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:08:32.257204Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:08:19.579516Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000929","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:39.576975Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:08:50Z"} 2023-09-22T21:08:50.382 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:42.948773Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001491","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:08:32.257204Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:08:19.579516Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000929","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:39.576975Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:08:50Z"} 2023-09-22T21:08:50.382 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:04:42.948773Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001491', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:08:32.257204Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:08:19.579516Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:04:39.576975Z'}}] 2023-09-22T21:08:50.383 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000001491', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000000929'} 2023-09-22T21:08:50.383 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000001491', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000000929'} 2023-09-22T21:08:50.383 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-22T21:08:50.383 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-09-22T21:08:50.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:08:50.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:08:50.383 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:08:50.458 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:08:50.458 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:08:50.519 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"00000000000000000000:00000000000000000651","last_update":"2023-09-22T21:04:39.605415Z"},{"marker":"00000000000000000000:00000000000000002055","last_update":"2023-09-22T21:08:32.302594Z"},{"marker":"00000000000000000000:00000000000000001499","last_update":"2023-09-22T21:07:59.650851Z"},{"marker":"00000000000000000000:00000000000000000929","last_update":"2023-09-22T21:04:40.693030Z"}] 2023-09-22T21:08:50.519 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000002055', 2: '00000000000000000000:00000000000000001499', 3: '00000000000000000000:00000000000000000929'} 2023-09-22T21:08:50.520 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-09-22T21:08:50.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:08:50.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:08:50.520 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:08:50.601 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:08:50.601 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:08:50.655 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10476880264182873081},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:39.605415Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002055","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:08:32.302594Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001499","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:07:59.650851Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000929","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:40.693030Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:08:50Z"} 2023-09-22T21:08:50.656 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10476880264182873081},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:39.605415Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002055","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:08:32.302594Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001499","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:07:59.650851Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000929","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:40.693030Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:08:50Z"} 2023-09-22T21:08:50.656 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:04:39.605415Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002055', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:08:32.302594Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:07:59.650851Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:04:40.693030Z'}}] 2023-09-22T21:08:50.656 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000002055', 2: '00000000000000000000:00000000000000001499', 3: '00000000000000000000:00000000000000000929'} 2023-09-22T21:08:50.656 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000002055', 2: '00000000000000000000:00000000000000001499', 3: '00000000000000000000:00000000000000000929'} 2023-09-22T21:08:50.656 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-09-22T21:08:50.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2023-09-22T21:08:50.657 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:08:50.657 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:08:50.658 DEBUG:boto:Method: PUT 2023-09-22T21:08:50.658 DEBUG:boto:Path: /ffzdqu-14/ 2023-09-22T21:08:50.658 DEBUG:boto:Data: 2023-09-22T21:08:50.658 DEBUG:boto:Headers: {} 2023-09-22T21:08:50.658 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:50.658 DEBUG:boto:Port: 8000 2023-09-22T21:08:50.658 DEBUG:boto:Params: {} 2023-09-22T21:08:50.658 DEBUG:boto:Token: None 2023-09-22T21:08:50.659 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:08:50 GMT /ffzdqu-14/ 2023-09-22T21:08:50.659 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:z4ecT8DbxKHJR3KBkL7Jg+n/jmY= 2023-09-22T21:08:50.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:z4ecT8DbxKHJR3KBkL7Jg+n/jmY=', 'Content-Length': '0'} 2023-09-22T21:08:50.678 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007498535326ad4ed4-00650e0262-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:50.678 DEBUG:rgw_multi.tests:created bucket=ffzdqu-14 2023-09-22T21:08:50.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:08:50.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:08:50.679 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:08:50.736 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:08:50.736 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:08:50.781 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:08:50.797 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695416930.680776_69.1","last_update":"2023-09-22T21:08:50.680776Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:08:50.797 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416930.680776_69.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:08:50.798 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:08:50.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:08:50.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:08:50.798 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:08:50.875 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:08:50.875 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:08:50.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:08:50.911+0000 7f3b0f1b5a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:08:50.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:08:50.911+0000 7f3b0f1b5a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:08:50.928 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695416930.672113_65.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:08:50.672113Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:08:50Z"} 2023-09-22T21:08:50.929 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695416930.672113_65.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:08:50.672113Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:08:50Z"} 2023-09-22T21:08:50.929 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695416930.672113_65.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:08:50.672113Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:08:50.929 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416930.680776_69.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:08:50.929 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416930.672113_65.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:08:50.929 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695416930.680776_69.1 target=1_1695416930.672113_65.1 2023-09-22T21:08:55.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:08:55.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:08:55.931 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:08:55.973 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:08:55.974 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:08:56.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:08:56.010+0000 7f1487296a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:08:56.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:08:56.010+0000 7f1487296a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:08:56.027 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695416930.680776_69.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:08:50.680776Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:08:56Z"} 2023-09-22T21:08:56.028 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695416930.680776_69.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:08:50.680776Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:08:56Z"} 2023-09-22T21:08:56.028 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695416930.680776_69.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:08:50.680776Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:08:56.028 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416930.680776_69.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:08:56.028 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416930.680776_69.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:08:56.028 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:08:56.029 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:08:56.029 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:08:56.029 DEBUG:boto:Method: HEAD 2023-09-22T21:08:56.029 DEBUG:boto:Path: /ffzdqu-14/ 2023-09-22T21:08:56.029 DEBUG:boto:Data: 2023-09-22T21:08:56.029 DEBUG:boto:Headers: {} 2023-09-22T21:08:56.029 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:56.029 DEBUG:boto:Port: 8000 2023-09-22T21:08:56.029 DEBUG:boto:Params: {} 2023-09-22T21:08:56.030 DEBUG:boto:Token: None 2023-09-22T21:08:56.030 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:08:56 GMT /ffzdqu-14/ 2023-09-22T21:08:56.030 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VsqbbOYbbVZ7VczzdE2zDpvfiqM= 2023-09-22T21:08:56.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:08:56 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VsqbbOYbbVZ7VczzdE2zDpvfiqM=', 'Content-Length': '0'} 2023-09-22T21:08:56.033 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cdc263ec73564b68-00650e0268-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:08:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:56.033 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:08:56.033 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:08:56.033 DEBUG:boto:Method: PUT 2023-09-22T21:08:56.033 DEBUG:boto:Path: /ffzdqu-14/obj 2023-09-22T21:08:56.033 DEBUG:boto:Data: 2023-09-22T21:08:56.033 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-22T21:08:56.034 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:08:56.034 DEBUG:boto:Port: 8000 2023-09-22T21:08:56.034 DEBUG:boto:Params: {} 2023-09-22T21:08:56.034 DEBUG:boto:Token: None 2023-09-22T21:08:56.034 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 22 Sep 2023 21:08:56 GMT /ffzdqu-14/obj 2023-09-22T21:08:56.034 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+jY4nrEKHI3Fg7pvkjFCX3qKDpQ= 2023-09-22T21:08:56.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:08:56 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+jY4nrEKHI3Fg7pvkjFCX3qKDpQ='} 2023-09-22T21:08:56.041 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002751e3d14540790a-00650e0268-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:08:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:08:56.042 DEBUG:rgw_multi.tests:created initial version id=None 2023-09-22T21:08:56.042 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-14 2023-09-22T21:08:56.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:08:56.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:08:56.042 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:08:56.112 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:08:56.113 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:08:56.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:08:56.154+0000 7fd014cf2a40 1 waiting to reach incremental sync.. 2023-09-22T21:09:01.157 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:01.154+0000 7fd014cf2a40 1 bucket sync caught up with source: 2023-09-22T21:09:01.157 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.6.6, , ] 2023-09-22T21:09:01.157 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.6.6, , ] 2023-09-22T21:09:01.157 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:01.154+0000 7fd014cf2a40 0 bucket checkpoint complete 2023-09-22T21:09:01.182 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-14 2023-09-22T21:09:01.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:09:01.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:09:01.182 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:09:01.232 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:09:01.232 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:09:01.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:01.274+0000 7f96d7646a40 1 waiting to reach incremental sync.. 2023-09-22T21:09:06.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:06.274+0000 7f96d7646a40 1 bucket sync caught up with source: 2023-09-22T21:09:06.277 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.21.6, , ] 2023-09-22T21:09:06.277 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.21.6, , ] 2023-09-22T21:09:06.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:06.274+0000 7f96d7646a40 0 bucket checkpoint complete 2023-09-22T21:09:06.304 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-14 zones={a2, a1} 2023-09-22T21:09:06.304 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:06.304 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:06.304 DEBUG:boto:Method: HEAD 2023-09-22T21:09:06.305 DEBUG:boto:Path: /ffzdqu-14/ 2023-09-22T21:09:06.305 DEBUG:boto:Data: 2023-09-22T21:09:06.305 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.305 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:06.305 DEBUG:boto:Port: 8001 2023-09-22T21:09:06.305 DEBUG:boto:Params: {} 2023-09-22T21:09:06.305 DEBUG:boto:Token: None 2023-09-22T21:09:06.305 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/ 2023-09-22T21:09:06.306 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jUVAsP6zrSvBrOCJkJ5eXJJ6we8= 2023-09-22T21:09:06.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jUVAsP6zrSvBrOCJkJ5eXJJ6we8=', 'Content-Length': '0'} 2023-09-22T21:09:06.309 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfe1adcd132f89fa-00650e0272-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.309 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:06.309 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:06.310 DEBUG:boto:Method: HEAD 2023-09-22T21:09:06.310 DEBUG:boto:Path: /ffzdqu-14/ 2023-09-22T21:09:06.310 DEBUG:boto:Data: 2023-09-22T21:09:06.310 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.310 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:06.310 DEBUG:boto:Port: 8000 2023-09-22T21:09:06.310 DEBUG:boto:Params: {} 2023-09-22T21:09:06.310 DEBUG:boto:Token: None 2023-09-22T21:09:06.311 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/ 2023-09-22T21:09:06.311 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jUVAsP6zrSvBrOCJkJ5eXJJ6we8= 2023-09-22T21:09:06.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jUVAsP6zrSvBrOCJkJ5eXJJ6we8=', 'Content-Length': '0'} 2023-09-22T21:09:06.313 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f115cd5bd876de9-00650e0272-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.314 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:09:06.314 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:06.314 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:06.314 DEBUG:boto:path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:06.314 DEBUG:boto:auth_path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:06.314 DEBUG:boto:Method: GET 2023-09-22T21:09:06.314 DEBUG:boto:Path: /ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:06.315 DEBUG:boto:Data: 2023-09-22T21:09:06.315 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.315 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:06.315 DEBUG:boto:Port: 8001 2023-09-22T21:09:06.315 DEBUG:boto:Params: {} 2023-09-22T21:09:06.315 DEBUG:boto:Token: None 2023-09-22T21:09:06.315 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/?versions 2023-09-22T21:09:06.315 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/MNqmwOwZeKOYHz/bI7liI+9k9c= 2023-09-22T21:09:06.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/MNqmwOwZeKOYHz/bI7liI+9k9c=', 'Content-Length': '0'} 2023-09-22T21:09:06.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce9a779443dab130-00650e0272-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.319 DEBUG:boto:b'ffzdqu-14999falseobjnulltrue2023-09-22T21:08:56.041Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:09:06.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:09:06.320 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:09:06.320 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:06.320 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:06.320 DEBUG:boto:path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:06.320 DEBUG:boto:auth_path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:06.321 DEBUG:boto:Method: GET 2023-09-22T21:09:06.321 DEBUG:boto:Path: /ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:06.321 DEBUG:boto:Data: 2023-09-22T21:09:06.321 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.321 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:06.321 DEBUG:boto:Port: 8000 2023-09-22T21:09:06.321 DEBUG:boto:Params: {} 2023-09-22T21:09:06.321 DEBUG:boto:Token: None 2023-09-22T21:09:06.321 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/?versions 2023-09-22T21:09:06.322 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/MNqmwOwZeKOYHz/bI7liI+9k9c= 2023-09-22T21:09:06.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/MNqmwOwZeKOYHz/bI7liI+9k9c=', 'Content-Length': '0'} 2023-09-22T21:09:06.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf1b844452a61a84-00650e0272-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.325 DEBUG:boto:b'ffzdqu-14999falseobjnulltrue2023-09-22T21:08:56.041Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:09:06.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:09:06.326 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:06.326 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:06.326 DEBUG:boto:path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:06.326 DEBUG:boto:auth_path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:06.326 DEBUG:boto:Method: GET 2023-09-22T21:09:06.327 DEBUG:boto:Path: /ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:06.327 DEBUG:boto:Data: 2023-09-22T21:09:06.327 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.327 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:06.327 DEBUG:boto:Port: 8001 2023-09-22T21:09:06.327 DEBUG:boto:Params: {} 2023-09-22T21:09:06.327 DEBUG:boto:Token: None 2023-09-22T21:09:06.327 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/?versions 2023-09-22T21:09:06.327 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/MNqmwOwZeKOYHz/bI7liI+9k9c= 2023-09-22T21:09:06.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/MNqmwOwZeKOYHz/bI7liI+9k9c=', 'Content-Length': '0'} 2023-09-22T21:09:06.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e8ed8b0ce2f93ef-00650e0272-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.331 DEBUG:boto:b'ffzdqu-14999falseobjnulltrue2023-09-22T21:08:56.041Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:09:06.331 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:06.331 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:06.332 DEBUG:boto:path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:06.332 DEBUG:boto:auth_path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:06.332 DEBUG:boto:Method: GET 2023-09-22T21:09:06.332 DEBUG:boto:Path: /ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:06.332 DEBUG:boto:Data: 2023-09-22T21:09:06.332 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.332 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:06.332 DEBUG:boto:Port: 8000 2023-09-22T21:09:06.332 DEBUG:boto:Params: {} 2023-09-22T21:09:06.333 DEBUG:boto:Token: None 2023-09-22T21:09:06.333 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/?versions 2023-09-22T21:09:06.333 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/MNqmwOwZeKOYHz/bI7liI+9k9c= 2023-09-22T21:09:06.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/MNqmwOwZeKOYHz/bI7liI+9k9c=', 'Content-Length': '0'} 2023-09-22T21:09:06.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066c7118f1388f200-00650e0272-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.337 DEBUG:boto:b'ffzdqu-14999falseobjnulltrue2023-09-22T21:08:56.041Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:09:06.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:09:06.338 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:06.338 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:06.338 DEBUG:boto:path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.338 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.338 DEBUG:boto:Method: GET 2023-09-22T21:09:06.338 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.338 DEBUG:boto:Data: 2023-09-22T21:09:06.338 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.338 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:06.339 DEBUG:boto:Port: 8001 2023-09-22T21:09:06.339 DEBUG:boto:Params: {} 2023-09-22T21:09:06.339 DEBUG:boto:Token: None 2023-09-22T21:09:06.339 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.339 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yH+d79SRzWyPRRHzh9yXby7S0Uw= 2023-09-22T21:09:06.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yH+d79SRzWyPRRHzh9yXby7S0Uw=', 'Content-Length': '0'} 2023-09-22T21:09:06.341 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000045abdd6015f76bcf-00650e0272-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.342 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:06.342 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:06.342 DEBUG:boto:path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.342 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.342 DEBUG:boto:Method: GET 2023-09-22T21:09:06.342 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.342 DEBUG:boto:Data: 2023-09-22T21:09:06.342 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.343 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:06.343 DEBUG:boto:Port: 8000 2023-09-22T21:09:06.343 DEBUG:boto:Params: {} 2023-09-22T21:09:06.343 DEBUG:boto:Token: None 2023-09-22T21:09:06.343 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.343 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yH+d79SRzWyPRRHzh9yXby7S0Uw= 2023-09-22T21:09:06.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yH+d79SRzWyPRRHzh9yXby7S0Uw=', 'Content-Length': '0'} 2023-09-22T21:09:06.345 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001b6d846ff86df628-00650e0272-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.346 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:06.346 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:06.346 DEBUG:boto:path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.346 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.346 DEBUG:boto:Method: HEAD 2023-09-22T21:09:06.346 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.346 DEBUG:boto:Data: 2023-09-22T21:09:06.346 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.346 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:06.347 DEBUG:boto:Port: 8001 2023-09-22T21:09:06.347 DEBUG:boto:Params: {} 2023-09-22T21:09:06.347 DEBUG:boto:Token: None 2023-09-22T21:09:06.347 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.347 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yTZU1fPCKihICM6dRxCAoK54wBg= 2023-09-22T21:09:06.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yTZU1fPCKihICM6dRxCAoK54wBg=', 'Content-Length': '0'} 2023-09-22T21:09:06.349 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000cf6c348054299c2-00650e0272-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.349 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:06.350 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:06.350 DEBUG:boto:path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.350 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.350 DEBUG:boto:Method: HEAD 2023-09-22T21:09:06.350 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.350 DEBUG:boto:Data: 2023-09-22T21:09:06.350 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.350 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:06.350 DEBUG:boto:Port: 8000 2023-09-22T21:09:06.351 DEBUG:boto:Params: {} 2023-09-22T21:09:06.351 DEBUG:boto:Token: None 2023-09-22T21:09:06.351 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.351 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yTZU1fPCKihICM6dRxCAoK54wBg= 2023-09-22T21:09:06.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yTZU1fPCKihICM6dRxCAoK54wBg=', 'Content-Length': '0'} 2023-09-22T21:09:06.353 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c925f4d791466b9b-00650e0272-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:09:06.354 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:06.354 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:06.354 DEBUG:boto:path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.354 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.354 DEBUG:boto:Method: GET 2023-09-22T21:09:06.354 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.354 DEBUG:boto:Data: 2023-09-22T21:09:06.354 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.354 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:06.355 DEBUG:boto:Port: 8001 2023-09-22T21:09:06.355 DEBUG:boto:Params: {} 2023-09-22T21:09:06.355 DEBUG:boto:Token: None 2023-09-22T21:09:06.355 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.355 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yH+d79SRzWyPRRHzh9yXby7S0Uw= 2023-09-22T21:09:06.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yH+d79SRzWyPRRHzh9yXby7S0Uw=', 'Content-Length': '0'} 2023-09-22T21:09:06.357 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007935dddd1a1bfd74-00650e0272-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.358 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:06.358 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:06.358 DEBUG:boto:path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.358 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.358 DEBUG:boto:Method: GET 2023-09-22T21:09:06.358 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.358 DEBUG:boto:Data: 2023-09-22T21:09:06.359 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.359 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:06.359 DEBUG:boto:Port: 8000 2023-09-22T21:09:06.359 DEBUG:boto:Params: {} 2023-09-22T21:09:06.359 DEBUG:boto:Token: None 2023-09-22T21:09:06.359 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:06.359 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yH+d79SRzWyPRRHzh9yXby7S0Uw= 2023-09-22T21:09:06.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yH+d79SRzWyPRRHzh9yXby7S0Uw=', 'Content-Length': '0'} 2023-09-22T21:09:06.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eaa2b29b9470f642-00650e0272-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.362 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:06.362 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:06.362 DEBUG:boto:path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:06.362 DEBUG:boto:auth_path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:06.362 DEBUG:boto:Method: GET 2023-09-22T21:09:06.362 DEBUG:boto:Path: /ffzdqu-14/?prefix=obj 2023-09-22T21:09:06.363 DEBUG:boto:Data: 2023-09-22T21:09:06.363 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.363 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:06.363 DEBUG:boto:Port: 8001 2023-09-22T21:09:06.363 DEBUG:boto:Params: {} 2023-09-22T21:09:06.363 DEBUG:boto:Token: None 2023-09-22T21:09:06.363 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/ 2023-09-22T21:09:06.363 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lPZC3OZcPG662QwEo0KaBQCHppM= 2023-09-22T21:09:06.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lPZC3OZcPG662QwEo0KaBQCHppM=', 'Content-Length': '0'} 2023-09-22T21:09:06.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009af78b9377fd3da2-00650e0272-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.367 DEBUG:boto:b'ffzdqu-14obj1000falseobj2023-09-22T21:08:56.041Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:09:06.368 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:06.368 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:06.368 DEBUG:boto:path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:06.368 DEBUG:boto:auth_path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:06.368 DEBUG:boto:Method: GET 2023-09-22T21:09:06.368 DEBUG:boto:Path: /ffzdqu-14/?prefix=obj 2023-09-22T21:09:06.368 DEBUG:boto:Data: 2023-09-22T21:09:06.368 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.369 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:06.369 DEBUG:boto:Port: 8000 2023-09-22T21:09:06.369 DEBUG:boto:Params: {} 2023-09-22T21:09:06.369 DEBUG:boto:Token: None 2023-09-22T21:09:06.369 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/ 2023-09-22T21:09:06.369 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lPZC3OZcPG662QwEo0KaBQCHppM= 2023-09-22T21:09:06.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lPZC3OZcPG662QwEo0KaBQCHppM=', 'Content-Length': '0'} 2023-09-22T21:09:06.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000293ceab597b7668f-00650e0272-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.373 DEBUG:boto:b'ffzdqu-14obj1000falseobj2023-09-22T21:08:56.041Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:09:06.374 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:06.374 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:06.374 DEBUG:boto:Method: HEAD 2023-09-22T21:09:06.374 DEBUG:boto:Path: /ffzdqu-14/obj 2023-09-22T21:09:06.374 DEBUG:boto:Data: 2023-09-22T21:09:06.374 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.374 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:06.374 DEBUG:boto:Port: 8001 2023-09-22T21:09:06.375 DEBUG:boto:Params: {} 2023-09-22T21:09:06.375 DEBUG:boto:Token: None 2023-09-22T21:09:06.375 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/obj 2023-09-22T21:09:06.375 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nY6Pux+69xATo980i5WBPEI4VJQ= 2023-09-22T21:09:06.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nY6Pux+69xATo980i5WBPEI4VJQ=', 'Content-Length': '0'} 2023-09-22T21:09:06.377 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ed9bedc8f9c45cf6-00650e0272-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.378 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:06.378 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:06.378 DEBUG:boto:Method: HEAD 2023-09-22T21:09:06.378 DEBUG:boto:Path: /ffzdqu-14/obj 2023-09-22T21:09:06.378 DEBUG:boto:Data: 2023-09-22T21:09:06.378 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.378 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:06.378 DEBUG:boto:Port: 8000 2023-09-22T21:09:06.378 DEBUG:boto:Params: {} 2023-09-22T21:09:06.379 DEBUG:boto:Token: None 2023-09-22T21:09:06.379 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/obj 2023-09-22T21:09:06.379 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nY6Pux+69xATo980i5WBPEI4VJQ= 2023-09-22T21:09:06.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nY6Pux+69xATo980i5WBPEI4VJQ=', 'Content-Length': '0'} 2023-09-22T21:09:06.381 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000aef492f9b9ec318-00650e0272-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:09:06.381 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:06.381 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:06.381 DEBUG:boto:Method: GET 2023-09-22T21:09:06.381 DEBUG:boto:Path: /ffzdqu-14/obj 2023-09-22T21:09:06.382 DEBUG:boto:Data: 2023-09-22T21:09:06.382 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.382 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:06.382 DEBUG:boto:Port: 8001 2023-09-22T21:09:06.382 DEBUG:boto:Params: {} 2023-09-22T21:09:06.382 DEBUG:boto:Token: None 2023-09-22T21:09:06.382 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/obj 2023-09-22T21:09:06.382 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ISw4xIBZ4XGUisX6WgBhS9XFbxw= 2023-09-22T21:09:06.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ISw4xIBZ4XGUisX6WgBhS9XFbxw=', 'Content-Length': '0'} 2023-09-22T21:09:06.385 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000077483f9bf370f9b0-00650e0272-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.385 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:06.385 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:06.386 DEBUG:boto:Method: GET 2023-09-22T21:09:06.386 DEBUG:boto:Path: /ffzdqu-14/obj 2023-09-22T21:09:06.386 DEBUG:boto:Data: 2023-09-22T21:09:06.386 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.386 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:06.386 DEBUG:boto:Port: 8000 2023-09-22T21:09:06.386 DEBUG:boto:Params: {} 2023-09-22T21:09:06.386 DEBUG:boto:Token: None 2023-09-22T21:09:06.386 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/obj 2023-09-22T21:09:06.387 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ISw4xIBZ4XGUisX6WgBhS9XFbxw= 2023-09-22T21:09:06.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ISw4xIBZ4XGUisX6WgBhS9XFbxw=', 'Content-Length': '0'} 2023-09-22T21:09:06.389 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fbaa63046fefc114-00650e0272-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.389 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:06.389 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:06.389 DEBUG:boto:path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:06.389 DEBUG:boto:auth_path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:06.390 DEBUG:boto:Method: GET 2023-09-22T21:09:06.390 DEBUG:boto:Path: /ffzdqu-14/?prefix=obj 2023-09-22T21:09:06.390 DEBUG:boto:Data: 2023-09-22T21:09:06.390 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.390 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:06.390 DEBUG:boto:Port: 8001 2023-09-22T21:09:06.390 DEBUG:boto:Params: {} 2023-09-22T21:09:06.390 DEBUG:boto:Token: None 2023-09-22T21:09:06.390 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/ 2023-09-22T21:09:06.391 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lPZC3OZcPG662QwEo0KaBQCHppM= 2023-09-22T21:09:06.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lPZC3OZcPG662QwEo0KaBQCHppM=', 'Content-Length': '0'} 2023-09-22T21:09:06.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d994c8200770396-00650e0272-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.438 DEBUG:boto:b'ffzdqu-14obj1000falseobj2023-09-22T21:08:56.041Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:09:06.439 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:06.439 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:06.439 DEBUG:boto:path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:06.439 DEBUG:boto:auth_path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:06.439 DEBUG:boto:Method: GET 2023-09-22T21:09:06.440 DEBUG:boto:Path: /ffzdqu-14/?prefix=obj 2023-09-22T21:09:06.440 DEBUG:boto:Data: 2023-09-22T21:09:06.440 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.440 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:06.440 DEBUG:boto:Port: 8000 2023-09-22T21:09:06.440 DEBUG:boto:Params: {} 2023-09-22T21:09:06.440 DEBUG:boto:Token: None 2023-09-22T21:09:06.440 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/ 2023-09-22T21:09:06.441 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lPZC3OZcPG662QwEo0KaBQCHppM= 2023-09-22T21:09:06.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lPZC3OZcPG662QwEo0KaBQCHppM=', 'Content-Length': '0'} 2023-09-22T21:09:06.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a7745af8f233f9d-00650e0272-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.446 DEBUG:boto:b'ffzdqu-14obj1000falseobj2023-09-22T21:08:56.041Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:09:06.449 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-14 zones={a2, a1} 2023-09-22T21:09:06.449 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:06.449 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:06.449 DEBUG:boto:path=/ffzdqu-14/?versioning 2023-09-22T21:09:06.449 DEBUG:boto:auth_path=/ffzdqu-14/?versioning 2023-09-22T21:09:06.449 DEBUG:boto:Method: PUT 2023-09-22T21:09:06.449 DEBUG:boto:Path: /ffzdqu-14/?versioning 2023-09-22T21:09:06.450 DEBUG:boto:Data: Enabled Disabled 2023-09-22T21:09:06.450 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.450 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:06.450 DEBUG:boto:Port: 8000 2023-09-22T21:09:06.450 DEBUG:boto:Params: {} 2023-09-22T21:09:06.450 DEBUG:boto:Token: None 2023-09-22T21:09:06.450 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/?versioning 2023-09-22T21:09:06.450 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nWcoV5GFZg+pKs67uYnwYahT3Pc= 2023-09-22T21:09:06.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nWcoV5GFZg+pKs67uYnwYahT3Pc=', 'Content-Length': '228'} 2023-09-22T21:09:06.463 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000060a25a51032739c7-00650e0272-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:09:06.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:09:06.464 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:09:06.506 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:09:06.506 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:09:06.546 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:09:06.567 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695416946.465659_71.1","last_update":"2023-09-22T21:09:06.465659Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:09:06.567 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416946.465659_71.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:09:06.567 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:09:06.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:09:06.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:09:06.568 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:09:06.643 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:09:06.643 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:09:06.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:06.678+0000 7fcc53ef7a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:09:06.681 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:06.682+0000 7fcc53ef7a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:09:06.698 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695416946.465659_71.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:09:06.465659Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:09:06Z"} 2023-09-22T21:09:06.698 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695416946.465659_71.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:09:06.465659Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:09:06Z"} 2023-09-22T21:09:06.699 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695416946.465659_71.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:09:06.465659Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:09:06.699 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416946.465659_71.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:09:06.699 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416946.465659_71.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:09:06.699 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:09:06.699 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:06.699 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:06.699 DEBUG:boto:Method: HEAD 2023-09-22T21:09:06.700 DEBUG:boto:Path: /ffzdqu-14/ 2023-09-22T21:09:06.700 DEBUG:boto:Data: 2023-09-22T21:09:06.700 DEBUG:boto:Headers: {} 2023-09-22T21:09:06.700 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:06.700 DEBUG:boto:Port: 8000 2023-09-22T21:09:06.700 DEBUG:boto:Params: {} 2023-09-22T21:09:06.700 DEBUG:boto:Token: None 2023-09-22T21:09:06.700 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/ 2023-09-22T21:09:06.701 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jUVAsP6zrSvBrOCJkJ5eXJJ6we8= 2023-09-22T21:09:06.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jUVAsP6zrSvBrOCJkJ5eXJJ6we8=', 'Content-Length': '0'} 2023-09-22T21:09:06.703 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf9dcc39c3221c37-00650e0272-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.703 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:06.703 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:06.703 DEBUG:boto:Method: PUT 2023-09-22T21:09:06.704 DEBUG:boto:Path: /ffzdqu-14/obj 2023-09-22T21:09:06.704 DEBUG:boto:Data: 2023-09-22T21:09:06.704 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-22T21:09:06.704 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:06.704 DEBUG:boto:Port: 8000 2023-09-22T21:09:06.704 DEBUG:boto:Params: {} 2023-09-22T21:09:06.704 DEBUG:boto:Token: None 2023-09-22T21:09:06.705 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 22 Sep 2023 21:09:06 GMT /ffzdqu-14/obj 2023-09-22T21:09:06.705 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:faEPZ8GkB7sIZ1IzhJqmOcZIkxM= 2023-09-22T21:09:06.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:09:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:faEPZ8GkB7sIZ1IzhJqmOcZIkxM='} 2023-09-22T21:09:06.715 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7'), ('x-amz-request-id', 'tx0000097e06d893eef6d93-00650e0272-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:06.715 DEBUG:rgw_multi.tests:created new version id=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:06.715 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-14 2023-09-22T21:09:06.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:09:06.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:09:06.716 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:09:06.783 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:09:06.783 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:09:06.826 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:06.826+0000 7f80fd472a40 1 waiting for incremental sync to catch up: 2023-09-22T21:09:06.827 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.6.6, , ] 2023-09-22T21:09:06.827 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.9.13, , ] 2023-09-22T21:09:11.829 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:11.826+0000 7f80fd472a40 1 waiting for incremental sync to catch up: 2023-09-22T21:09:11.829 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.6.6, , ] 2023-09-22T21:09:11.829 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.9.13, , ] 2023-09-22T21:09:16.830 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:16.830+0000 7f80fd472a40 1 waiting for incremental sync to catch up: 2023-09-22T21:09:16.831 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.6.6, , ] 2023-09-22T21:09:16.831 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.9.13, , ] 2023-09-22T21:09:21.833 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:21.834+0000 7f80fd472a40 1 bucket sync caught up with source: 2023-09-22T21:09:21.833 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.9.13, , ] 2023-09-22T21:09:21.834 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.9.13, , ] 2023-09-22T21:09:21.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:21.834+0000 7f80fd472a40 0 bucket checkpoint complete 2023-09-22T21:09:21.863 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-14 2023-09-22T21:09:21.863 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:09:21.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:09:21.863 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:09:21.953 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:09:21.953 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:09:21.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:21.994+0000 7f17e1ba8a40 1 waiting for incremental sync to catch up: 2023-09-22T21:09:21.995 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.21.6, , ] 2023-09-22T21:09:21.995 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.26.13, , ] 2023-09-22T21:09:26.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:26.998+0000 7f17e1ba8a40 1 waiting for incremental sync to catch up: 2023-09-22T21:09:26.997 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.21.6, , ] 2023-09-22T21:09:26.998 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.26.13, , ] 2023-09-22T21:09:31.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:31.998+0000 7f17e1ba8a40 1 waiting for incremental sync to catch up: 2023-09-22T21:09:32.000 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.21.6, , ] 2023-09-22T21:09:32.000 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.26.13, , ] 2023-09-22T21:09:37.001 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:37.002+0000 7f17e1ba8a40 1 waiting for incremental sync to catch up: 2023-09-22T21:09:37.002 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.21.6, , ] 2023-09-22T21:09:37.002 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.26.13, , ] 2023-09-22T21:09:41.761 INFO:tasks.rgw.c1.client.0.smithi160.stdout:2023-09-22T21:09:41.762+0000 7fa9c1675640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:09:42.004 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:42.002+0000 7f17e1ba8a40 1 bucket sync caught up with source: 2023-09-22T21:09:42.004 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.26.13, , ] 2023-09-22T21:09:42.004 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.26.13, , ] 2023-09-22T21:09:42.005 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:42.002+0000 7f17e1ba8a40 0 bucket checkpoint complete 2023-09-22T21:09:42.033 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-14 zones={a2, a1} 2023-09-22T21:09:42.034 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:42.034 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:42.034 DEBUG:boto:Method: HEAD 2023-09-22T21:09:42.034 DEBUG:boto:Path: /ffzdqu-14/ 2023-09-22T21:09:42.034 DEBUG:boto:Data: 2023-09-22T21:09:42.034 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.034 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:42.035 DEBUG:boto:Port: 8001 2023-09-22T21:09:42.035 DEBUG:boto:Params: {} 2023-09-22T21:09:42.035 DEBUG:boto:Token: None 2023-09-22T21:09:42.035 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/ 2023-09-22T21:09:42.035 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jYqC28vLiiewmY5trHE8HnVLDWg= 2023-09-22T21:09:42.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jYqC28vLiiewmY5trHE8HnVLDWg=', 'Content-Length': '0'} 2023-09-22T21:09:42.039 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc101d3a3ba08291-00650e0296-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.039 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:42.039 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:42.040 DEBUG:boto:Method: HEAD 2023-09-22T21:09:42.040 DEBUG:boto:Path: /ffzdqu-14/ 2023-09-22T21:09:42.040 DEBUG:boto:Data: 2023-09-22T21:09:42.040 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.040 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:42.040 DEBUG:boto:Port: 8000 2023-09-22T21:09:42.040 DEBUG:boto:Params: {} 2023-09-22T21:09:42.040 DEBUG:boto:Token: None 2023-09-22T21:09:42.040 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/ 2023-09-22T21:09:42.041 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jYqC28vLiiewmY5trHE8HnVLDWg= 2023-09-22T21:09:42.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jYqC28vLiiewmY5trHE8HnVLDWg=', 'Content-Length': '0'} 2023-09-22T21:09:42.044 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f38d05b010b080d4-00650e0296-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.044 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:09:42.044 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:42.044 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:42.044 DEBUG:boto:path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:42.044 DEBUG:boto:auth_path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:42.045 DEBUG:boto:Method: GET 2023-09-22T21:09:42.045 DEBUG:boto:Path: /ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:42.045 DEBUG:boto:Data: 2023-09-22T21:09:42.045 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.045 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:42.045 DEBUG:boto:Port: 8001 2023-09-22T21:09:42.045 DEBUG:boto:Params: {} 2023-09-22T21:09:42.045 DEBUG:boto:Token: None 2023-09-22T21:09:42.045 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/?versions 2023-09-22T21:09:42.046 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1aca0nGRD+Qtdym7EwU9SJzCfH8= 2023-09-22T21:09:42.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1aca0nGRD+Qtdym7EwU9SJzCfH8=', 'Content-Length': '0'} 2023-09-22T21:09:42.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebd42bd63dc6f1e4-00650e0296-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.049 DEBUG:boto:b'ffzdqu-14999falseobjJ8i90YbXhRfq0RtMmWPl-WhwfaK0Om7true2023-09-22T21:09:06.711Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-09-22T21:08:56.041Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:09:42.050 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:09:42.050 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:09:42.050 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:09:42.050 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:42.051 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:42.051 DEBUG:boto:path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:42.051 DEBUG:boto:auth_path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:42.051 DEBUG:boto:Method: GET 2023-09-22T21:09:42.051 DEBUG:boto:Path: /ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:42.051 DEBUG:boto:Data: 2023-09-22T21:09:42.051 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.051 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:42.051 DEBUG:boto:Port: 8000 2023-09-22T21:09:42.051 DEBUG:boto:Params: {} 2023-09-22T21:09:42.052 DEBUG:boto:Token: None 2023-09-22T21:09:42.052 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/?versions 2023-09-22T21:09:42.052 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1aca0nGRD+Qtdym7EwU9SJzCfH8= 2023-09-22T21:09:42.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1aca0nGRD+Qtdym7EwU9SJzCfH8=', 'Content-Length': '0'} 2023-09-22T21:09:42.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c6eb3c13dfc0730-00650e0296-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.098 DEBUG:boto:b'ffzdqu-14999falseobjJ8i90YbXhRfq0RtMmWPl-WhwfaK0Om7true2023-09-22T21:09:06.711Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-09-22T21:08:56.041Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:09:42.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:09:42.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:09:42.099 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:42.099 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:42.099 DEBUG:boto:path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:42.100 DEBUG:boto:auth_path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:42.100 DEBUG:boto:Method: GET 2023-09-22T21:09:42.100 DEBUG:boto:Path: /ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:42.100 DEBUG:boto:Data: 2023-09-22T21:09:42.100 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.100 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:42.100 DEBUG:boto:Port: 8001 2023-09-22T21:09:42.100 DEBUG:boto:Params: {} 2023-09-22T21:09:42.100 DEBUG:boto:Token: None 2023-09-22T21:09:42.101 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/?versions 2023-09-22T21:09:42.101 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1aca0nGRD+Qtdym7EwU9SJzCfH8= 2023-09-22T21:09:42.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1aca0nGRD+Qtdym7EwU9SJzCfH8=', 'Content-Length': '0'} 2023-09-22T21:09:42.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044ee3978544651fc-00650e0296-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.105 DEBUG:boto:b'ffzdqu-14999falseobjJ8i90YbXhRfq0RtMmWPl-WhwfaK0Om7true2023-09-22T21:09:06.711Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-09-22T21:08:56.041Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:09:42.105 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:42.106 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:42.106 DEBUG:boto:path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:42.106 DEBUG:boto:auth_path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:42.106 DEBUG:boto:Method: GET 2023-09-22T21:09:42.106 DEBUG:boto:Path: /ffzdqu-14/?versions&max-keys=999 2023-09-22T21:09:42.106 DEBUG:boto:Data: 2023-09-22T21:09:42.106 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.106 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:42.106 DEBUG:boto:Port: 8000 2023-09-22T21:09:42.107 DEBUG:boto:Params: {} 2023-09-22T21:09:42.107 DEBUG:boto:Token: None 2023-09-22T21:09:42.107 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/?versions 2023-09-22T21:09:42.107 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1aca0nGRD+Qtdym7EwU9SJzCfH8= 2023-09-22T21:09:42.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1aca0nGRD+Qtdym7EwU9SJzCfH8=', 'Content-Length': '0'} 2023-09-22T21:09:42.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d1c8ec307507378-00650e0296-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.111 DEBUG:boto:b'ffzdqu-14999falseobjJ8i90YbXhRfq0RtMmWPl-WhwfaK0Om7true2023-09-22T21:09:06.711Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-09-22T21:08:56.041Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:09:42.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:09:42.112 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:42.112 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:42.112 DEBUG:boto:path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.112 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.112 DEBUG:boto:Method: GET 2023-09-22T21:09:42.112 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.112 DEBUG:boto:Data: 2023-09-22T21:09:42.112 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.112 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:42.113 DEBUG:boto:Port: 8001 2023-09-22T21:09:42.113 DEBUG:boto:Params: {} 2023-09-22T21:09:42.113 DEBUG:boto:Token: None 2023-09-22T21:09:42.113 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.113 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PzuRu3yO9uEZ8ur+eInt2jqGSGM= 2023-09-22T21:09:42.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PzuRu3yO9uEZ8ur+eInt2jqGSGM=', 'Content-Length': '0'} 2023-09-22T21:09:42.116 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('x-amz-version-id', 'J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006600125105666f48-00650e0296-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.116 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:42.116 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:42.116 DEBUG:boto:path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.116 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.117 DEBUG:boto:Method: GET 2023-09-22T21:09:42.117 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.117 DEBUG:boto:Data: 2023-09-22T21:09:42.117 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.117 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:42.117 DEBUG:boto:Port: 8000 2023-09-22T21:09:42.117 DEBUG:boto:Params: {} 2023-09-22T21:09:42.117 DEBUG:boto:Token: None 2023-09-22T21:09:42.117 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.118 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PzuRu3yO9uEZ8ur+eInt2jqGSGM= 2023-09-22T21:09:42.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PzuRu3yO9uEZ8ur+eInt2jqGSGM=', 'Content-Length': '0'} 2023-09-22T21:09:42.120 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('x-amz-version-id', 'J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001ccfc858a795d3ee-00650e0296-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.120 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:42.121 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:42.121 DEBUG:boto:path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.121 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.121 DEBUG:boto:Method: HEAD 2023-09-22T21:09:42.121 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.121 DEBUG:boto:Data: 2023-09-22T21:09:42.121 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.121 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:42.121 DEBUG:boto:Port: 8001 2023-09-22T21:09:42.121 DEBUG:boto:Params: {} 2023-09-22T21:09:42.122 DEBUG:boto:Token: None 2023-09-22T21:09:42.122 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.122 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oWWgGvd6mb03MCtwJQ8OYTaRL5c= 2023-09-22T21:09:42.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oWWgGvd6mb03MCtwJQ8OYTaRL5c=', 'Content-Length': '0'} 2023-09-22T21:09:42.125 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('x-amz-version-id', 'J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dceaa3626d721510-00650e0296-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.125 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:42.125 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:42.125 DEBUG:boto:path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.125 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.126 DEBUG:boto:Method: HEAD 2023-09-22T21:09:42.126 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.126 DEBUG:boto:Data: 2023-09-22T21:09:42.126 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.126 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:42.126 DEBUG:boto:Port: 8000 2023-09-22T21:09:42.126 DEBUG:boto:Params: {} 2023-09-22T21:09:42.126 DEBUG:boto:Token: None 2023-09-22T21:09:42.126 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.127 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oWWgGvd6mb03MCtwJQ8OYTaRL5c= 2023-09-22T21:09:42.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oWWgGvd6mb03MCtwJQ8OYTaRL5c=', 'Content-Length': '0'} 2023-09-22T21:09:42.129 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('x-amz-version-id', 'J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000022138fccdc3ddcdc-00650e0296-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:09:42.130 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:42.130 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:42.130 DEBUG:boto:path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.130 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.130 DEBUG:boto:Method: GET 2023-09-22T21:09:42.130 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.130 DEBUG:boto:Data: 2023-09-22T21:09:42.130 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.130 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:42.131 DEBUG:boto:Port: 8001 2023-09-22T21:09:42.131 DEBUG:boto:Params: {} 2023-09-22T21:09:42.131 DEBUG:boto:Token: None 2023-09-22T21:09:42.131 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.131 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PzuRu3yO9uEZ8ur+eInt2jqGSGM= 2023-09-22T21:09:42.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PzuRu3yO9uEZ8ur+eInt2jqGSGM=', 'Content-Length': '0'} 2023-09-22T21:09:42.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('x-amz-version-id', 'J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000061fd8b5f119d9337-00650e0296-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.134 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:42.134 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:42.134 DEBUG:boto:path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.134 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.135 DEBUG:boto:Method: GET 2023-09-22T21:09:42.135 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.135 DEBUG:boto:Data: 2023-09-22T21:09:42.135 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.135 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:42.135 DEBUG:boto:Port: 8000 2023-09-22T21:09:42.135 DEBUG:boto:Params: {} 2023-09-22T21:09:42.135 DEBUG:boto:Token: None 2023-09-22T21:09:42.135 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.136 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PzuRu3yO9uEZ8ur+eInt2jqGSGM= 2023-09-22T21:09:42.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PzuRu3yO9uEZ8ur+eInt2jqGSGM=', 'Content-Length': '0'} 2023-09-22T21:09:42.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('x-amz-version-id', 'J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c1e01a01b819a049-00650e0296-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.138 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:42.139 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:42.139 DEBUG:boto:path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.139 DEBUG:boto:auth_path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.139 DEBUG:boto:Method: GET 2023-09-22T21:09:42.139 DEBUG:boto:Path: /ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.139 DEBUG:boto:Data: 2023-09-22T21:09:42.139 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.139 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:42.139 DEBUG:boto:Port: 8001 2023-09-22T21:09:42.139 DEBUG:boto:Params: {} 2023-09-22T21:09:42.140 DEBUG:boto:Token: None 2023-09-22T21:09:42.140 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/ 2023-09-22T21:09:42.140 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7RFwJQC0PXeo3TPDZHjZXWx/X64= 2023-09-22T21:09:42.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7RFwJQC0PXeo3TPDZHjZXWx/X64=', 'Content-Length': '0'} 2023-09-22T21:09:42.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d979f2c44493f5f-00650e0296-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.186 DEBUG:boto:b'ffzdqu-14obj1000falseobj2023-09-22T21:09:06.711Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:09:42.187 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:42.187 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:42.187 DEBUG:boto:path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.187 DEBUG:boto:auth_path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.187 DEBUG:boto:Method: GET 2023-09-22T21:09:42.187 DEBUG:boto:Path: /ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.187 DEBUG:boto:Data: 2023-09-22T21:09:42.188 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.188 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:42.188 DEBUG:boto:Port: 8000 2023-09-22T21:09:42.188 DEBUG:boto:Params: {} 2023-09-22T21:09:42.188 DEBUG:boto:Token: None 2023-09-22T21:09:42.188 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/ 2023-09-22T21:09:42.188 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7RFwJQC0PXeo3TPDZHjZXWx/X64= 2023-09-22T21:09:42.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7RFwJQC0PXeo3TPDZHjZXWx/X64=', 'Content-Length': '0'} 2023-09-22T21:09:42.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005caea032a675cd68-00650e0296-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.193 DEBUG:boto:b'ffzdqu-14obj1000falseobj2023-09-22T21:09:06.711Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:09:42.194 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:42.194 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:42.194 DEBUG:boto:Method: HEAD 2023-09-22T21:09:42.194 DEBUG:boto:Path: /ffzdqu-14/obj 2023-09-22T21:09:42.194 DEBUG:boto:Data: 2023-09-22T21:09:42.195 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.195 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:42.195 DEBUG:boto:Port: 8001 2023-09-22T21:09:42.195 DEBUG:boto:Params: {} 2023-09-22T21:09:42.195 DEBUG:boto:Token: None 2023-09-22T21:09:42.195 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/obj 2023-09-22T21:09:42.195 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UYsZhhHYO6XGwB1Ss70VW3ujatc= 2023-09-22T21:09:42.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UYsZhhHYO6XGwB1Ss70VW3ujatc=', 'Content-Length': '0'} 2023-09-22T21:09:42.199 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('x-amz-version-id', 'J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000060dcebc38c82dc9c-00650e0296-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.199 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:42.199 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:42.199 DEBUG:boto:Method: HEAD 2023-09-22T21:09:42.199 DEBUG:boto:Path: /ffzdqu-14/obj 2023-09-22T21:09:42.200 DEBUG:boto:Data: 2023-09-22T21:09:42.200 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.200 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:42.200 DEBUG:boto:Port: 8000 2023-09-22T21:09:42.200 DEBUG:boto:Params: {} 2023-09-22T21:09:42.200 DEBUG:boto:Token: None 2023-09-22T21:09:42.200 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/obj 2023-09-22T21:09:42.200 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UYsZhhHYO6XGwB1Ss70VW3ujatc= 2023-09-22T21:09:42.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UYsZhhHYO6XGwB1Ss70VW3ujatc=', 'Content-Length': '0'} 2023-09-22T21:09:42.204 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('x-amz-version-id', 'J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c450c8f31631c919-00650e0296-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:09:42.204 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:42.204 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:42.204 DEBUG:boto:path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.204 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.205 DEBUG:boto:Method: GET 2023-09-22T21:09:42.205 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.205 DEBUG:boto:Data: 2023-09-22T21:09:42.205 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.205 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:42.205 DEBUG:boto:Port: 8001 2023-09-22T21:09:42.205 DEBUG:boto:Params: {} 2023-09-22T21:09:42.205 DEBUG:boto:Token: None 2023-09-22T21:09:42.205 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.206 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PzuRu3yO9uEZ8ur+eInt2jqGSGM= 2023-09-22T21:09:42.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PzuRu3yO9uEZ8ur+eInt2jqGSGM=', 'Content-Length': '0'} 2023-09-22T21:09:42.208 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('x-amz-version-id', 'J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004af3d55370a6b81d-00650e0296-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.209 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:42.209 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:42.209 DEBUG:boto:path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.209 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.209 DEBUG:boto:Method: GET 2023-09-22T21:09:42.209 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.209 DEBUG:boto:Data: 2023-09-22T21:09:42.209 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.209 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:42.210 DEBUG:boto:Port: 8000 2023-09-22T21:09:42.210 DEBUG:boto:Params: {} 2023-09-22T21:09:42.210 DEBUG:boto:Token: None 2023-09-22T21:09:42.210 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.210 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PzuRu3yO9uEZ8ur+eInt2jqGSGM= 2023-09-22T21:09:42.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PzuRu3yO9uEZ8ur+eInt2jqGSGM=', 'Content-Length': '0'} 2023-09-22T21:09:42.213 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('x-amz-version-id', 'J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004bd80ca1faebe138-00650e0296-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.213 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:42.213 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:42.213 DEBUG:boto:path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.213 DEBUG:boto:auth_path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.214 DEBUG:boto:Method: GET 2023-09-22T21:09:42.214 DEBUG:boto:Path: /ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.214 DEBUG:boto:Data: 2023-09-22T21:09:42.214 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.214 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:42.214 DEBUG:boto:Port: 8001 2023-09-22T21:09:42.214 DEBUG:boto:Params: {} 2023-09-22T21:09:42.214 DEBUG:boto:Token: None 2023-09-22T21:09:42.214 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/ 2023-09-22T21:09:42.215 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7RFwJQC0PXeo3TPDZHjZXWx/X64= 2023-09-22T21:09:42.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7RFwJQC0PXeo3TPDZHjZXWx/X64=', 'Content-Length': '0'} 2023-09-22T21:09:42.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c741cbbe444a7bcb-00650e0296-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.262 DEBUG:boto:b'ffzdqu-14obj1000falseobj2023-09-22T21:09:06.711Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:09:42.263 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:42.263 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:42.263 DEBUG:boto:path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.263 DEBUG:boto:auth_path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.264 DEBUG:boto:Method: GET 2023-09-22T21:09:42.264 DEBUG:boto:Path: /ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.264 DEBUG:boto:Data: 2023-09-22T21:09:42.264 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.264 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:42.264 DEBUG:boto:Port: 8000 2023-09-22T21:09:42.264 DEBUG:boto:Params: {} 2023-09-22T21:09:42.264 DEBUG:boto:Token: None 2023-09-22T21:09:42.265 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/ 2023-09-22T21:09:42.265 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7RFwJQC0PXeo3TPDZHjZXWx/X64= 2023-09-22T21:09:42.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7RFwJQC0PXeo3TPDZHjZXWx/X64=', 'Content-Length': '0'} 2023-09-22T21:09:42.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068b3e9dbd4ae9f96-00650e0296-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.270 DEBUG:boto:b'ffzdqu-14obj1000falseobj2023-09-22T21:09:06.711Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:09:42.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:09:42.270 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:42.270 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:42.271 DEBUG:boto:path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.271 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.271 DEBUG:boto:Method: GET 2023-09-22T21:09:42.271 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.271 DEBUG:boto:Data: 2023-09-22T21:09:42.271 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.271 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:42.271 DEBUG:boto:Port: 8001 2023-09-22T21:09:42.271 DEBUG:boto:Params: {} 2023-09-22T21:09:42.271 DEBUG:boto:Token: None 2023-09-22T21:09:42.272 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.272 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2y0HCYcKYwyHwH/v/KvUx+gtsVk= 2023-09-22T21:09:42.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2y0HCYcKYwyHwH/v/KvUx+gtsVk=', 'Content-Length': '0'} 2023-09-22T21:09:42.274 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000090df5c8aacedffad-00650e0296-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.275 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:42.275 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:42.275 DEBUG:boto:path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.275 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.275 DEBUG:boto:Method: GET 2023-09-22T21:09:42.275 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.275 DEBUG:boto:Data: 2023-09-22T21:09:42.275 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.276 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:42.276 DEBUG:boto:Port: 8000 2023-09-22T21:09:42.276 DEBUG:boto:Params: {} 2023-09-22T21:09:42.276 DEBUG:boto:Token: None 2023-09-22T21:09:42.276 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.276 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2y0HCYcKYwyHwH/v/KvUx+gtsVk= 2023-09-22T21:09:42.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2y0HCYcKYwyHwH/v/KvUx+gtsVk=', 'Content-Length': '0'} 2023-09-22T21:09:42.279 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a0554b7c7e4af77d-00650e0296-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.279 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:42.279 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:42.279 DEBUG:boto:path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.279 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.280 DEBUG:boto:Method: HEAD 2023-09-22T21:09:42.280 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.280 DEBUG:boto:Data: 2023-09-22T21:09:42.280 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.280 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:42.280 DEBUG:boto:Port: 8001 2023-09-22T21:09:42.280 DEBUG:boto:Params: {} 2023-09-22T21:09:42.280 DEBUG:boto:Token: None 2023-09-22T21:09:42.280 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.281 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HD3MlA+N0/N+pjdQW9ECy8/eZOk= 2023-09-22T21:09:42.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HD3MlA+N0/N+pjdQW9ECy8/eZOk=', 'Content-Length': '0'} 2023-09-22T21:09:42.283 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a1605e38f614d88c-00650e0296-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.283 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:42.284 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:42.284 DEBUG:boto:path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.284 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.284 DEBUG:boto:Method: HEAD 2023-09-22T21:09:42.284 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.284 DEBUG:boto:Data: 2023-09-22T21:09:42.284 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.284 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:42.284 DEBUG:boto:Port: 8000 2023-09-22T21:09:42.284 DEBUG:boto:Params: {} 2023-09-22T21:09:42.285 DEBUG:boto:Token: None 2023-09-22T21:09:42.285 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.285 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HD3MlA+N0/N+pjdQW9ECy8/eZOk= 2023-09-22T21:09:42.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HD3MlA+N0/N+pjdQW9ECy8/eZOk=', 'Content-Length': '0'} 2023-09-22T21:09:42.287 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000034e7dfbe7285e221-00650e0296-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:09:42.288 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:42.288 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:42.288 DEBUG:boto:path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.288 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.288 DEBUG:boto:Method: GET 2023-09-22T21:09:42.288 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.288 DEBUG:boto:Data: 2023-09-22T21:09:42.289 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.289 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:42.289 DEBUG:boto:Port: 8001 2023-09-22T21:09:42.289 DEBUG:boto:Params: {} 2023-09-22T21:09:42.289 DEBUG:boto:Token: None 2023-09-22T21:09:42.289 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.289 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2y0HCYcKYwyHwH/v/KvUx+gtsVk= 2023-09-22T21:09:42.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2y0HCYcKYwyHwH/v/KvUx+gtsVk=', 'Content-Length': '0'} 2023-09-22T21:09:42.292 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d573ea1a8450a698-00650e0296-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.292 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:42.292 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:42.292 DEBUG:boto:path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.292 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.293 DEBUG:boto:Method: GET 2023-09-22T21:09:42.293 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.293 DEBUG:boto:Data: 2023-09-22T21:09:42.293 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.293 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:42.293 DEBUG:boto:Port: 8000 2023-09-22T21:09:42.293 DEBUG:boto:Params: {} 2023-09-22T21:09:42.293 DEBUG:boto:Token: None 2023-09-22T21:09:42.293 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/obj?versionId=null 2023-09-22T21:09:42.294 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2y0HCYcKYwyHwH/v/KvUx+gtsVk= 2023-09-22T21:09:42.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2y0HCYcKYwyHwH/v/KvUx+gtsVk=', 'Content-Length': '0'} 2023-09-22T21:09:42.296 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:08:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005d61e1aa94c722d0-00650e0296-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.297 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:42.297 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:42.297 DEBUG:boto:path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.297 DEBUG:boto:auth_path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.297 DEBUG:boto:Method: GET 2023-09-22T21:09:42.297 DEBUG:boto:Path: /ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.297 DEBUG:boto:Data: 2023-09-22T21:09:42.297 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.297 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:42.298 DEBUG:boto:Port: 8001 2023-09-22T21:09:42.298 DEBUG:boto:Params: {} 2023-09-22T21:09:42.298 DEBUG:boto:Token: None 2023-09-22T21:09:42.298 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/ 2023-09-22T21:09:42.298 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7RFwJQC0PXeo3TPDZHjZXWx/X64= 2023-09-22T21:09:42.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7RFwJQC0PXeo3TPDZHjZXWx/X64=', 'Content-Length': '0'} 2023-09-22T21:09:42.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000791d3f2e79cad36a-00650e0296-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.350 DEBUG:boto:b'ffzdqu-14obj1000falseobj2023-09-22T21:09:06.711Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:09:42.351 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:42.351 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:42.351 DEBUG:boto:path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.352 DEBUG:boto:auth_path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.352 DEBUG:boto:Method: GET 2023-09-22T21:09:42.352 DEBUG:boto:Path: /ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.352 DEBUG:boto:Data: 2023-09-22T21:09:42.352 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.352 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:42.353 DEBUG:boto:Port: 8000 2023-09-22T21:09:42.353 DEBUG:boto:Params: {} 2023-09-22T21:09:42.353 DEBUG:boto:Token: None 2023-09-22T21:09:42.353 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/ 2023-09-22T21:09:42.353 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7RFwJQC0PXeo3TPDZHjZXWx/X64= 2023-09-22T21:09:42.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7RFwJQC0PXeo3TPDZHjZXWx/X64=', 'Content-Length': '0'} 2023-09-22T21:09:42.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df44db8bc6438ff9-00650e0296-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.359 DEBUG:boto:b'ffzdqu-14obj1000falseobj2023-09-22T21:09:06.711Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:09:42.359 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:42.359 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:42.359 DEBUG:boto:Method: HEAD 2023-09-22T21:09:42.359 DEBUG:boto:Path: /ffzdqu-14/obj 2023-09-22T21:09:42.360 DEBUG:boto:Data: 2023-09-22T21:09:42.360 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.360 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:42.360 DEBUG:boto:Port: 8001 2023-09-22T21:09:42.360 DEBUG:boto:Params: {} 2023-09-22T21:09:42.360 DEBUG:boto:Token: None 2023-09-22T21:09:42.360 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/obj 2023-09-22T21:09:42.360 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UYsZhhHYO6XGwB1Ss70VW3ujatc= 2023-09-22T21:09:42.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UYsZhhHYO6XGwB1Ss70VW3ujatc=', 'Content-Length': '0'} 2023-09-22T21:09:42.364 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('x-amz-version-id', 'J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a1e22b9087ae3c82-00650e0296-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.364 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:42.365 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:42.365 DEBUG:boto:Method: HEAD 2023-09-22T21:09:42.365 DEBUG:boto:Path: /ffzdqu-14/obj 2023-09-22T21:09:42.365 DEBUG:boto:Data: 2023-09-22T21:09:42.365 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.365 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:42.365 DEBUG:boto:Port: 8000 2023-09-22T21:09:42.365 DEBUG:boto:Params: {} 2023-09-22T21:09:42.365 DEBUG:boto:Token: None 2023-09-22T21:09:42.366 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/obj 2023-09-22T21:09:42.366 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UYsZhhHYO6XGwB1Ss70VW3ujatc= 2023-09-22T21:09:42.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UYsZhhHYO6XGwB1Ss70VW3ujatc=', 'Content-Length': '0'} 2023-09-22T21:09:42.369 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('x-amz-version-id', 'J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002dfab89a16f0dcf9-00650e0296-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:09:42.370 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:42.370 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:42.370 DEBUG:boto:path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.370 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.370 DEBUG:boto:Method: GET 2023-09-22T21:09:42.370 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.370 DEBUG:boto:Data: 2023-09-22T21:09:42.370 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.371 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:42.371 DEBUG:boto:Port: 8001 2023-09-22T21:09:42.371 DEBUG:boto:Params: {} 2023-09-22T21:09:42.371 DEBUG:boto:Token: None 2023-09-22T21:09:42.371 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.371 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PzuRu3yO9uEZ8ur+eInt2jqGSGM= 2023-09-22T21:09:42.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PzuRu3yO9uEZ8ur+eInt2jqGSGM=', 'Content-Length': '0'} 2023-09-22T21:09:42.374 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('x-amz-version-id', 'J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000000ba435a064ec869-00650e0296-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.374 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:42.374 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:42.375 DEBUG:boto:path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.375 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.375 DEBUG:boto:Method: GET 2023-09-22T21:09:42.375 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.375 DEBUG:boto:Data: 2023-09-22T21:09:42.375 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.375 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:42.375 DEBUG:boto:Port: 8000 2023-09-22T21:09:42.375 DEBUG:boto:Params: {} 2023-09-22T21:09:42.375 DEBUG:boto:Token: None 2023-09-22T21:09:42.376 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:09:42.376 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PzuRu3yO9uEZ8ur+eInt2jqGSGM= 2023-09-22T21:09:42.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PzuRu3yO9uEZ8ur+eInt2jqGSGM=', 'Content-Length': '0'} 2023-09-22T21:09:42.378 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('x-amz-version-id', 'J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000946a2d05bbbd7983-00650e0296-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.379 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:42.379 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:42.379 DEBUG:boto:path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.379 DEBUG:boto:auth_path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.379 DEBUG:boto:Method: GET 2023-09-22T21:09:42.379 DEBUG:boto:Path: /ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.380 DEBUG:boto:Data: 2023-09-22T21:09:42.380 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.380 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:09:42.380 DEBUG:boto:Port: 8001 2023-09-22T21:09:42.380 DEBUG:boto:Params: {} 2023-09-22T21:09:42.380 DEBUG:boto:Token: None 2023-09-22T21:09:42.380 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/ 2023-09-22T21:09:42.380 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7RFwJQC0PXeo3TPDZHjZXWx/X64= 2023-09-22T21:09:42.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7RFwJQC0PXeo3TPDZHjZXWx/X64=', 'Content-Length': '0'} 2023-09-22T21:09:42.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc48b34ee0c8b447-00650e0296-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.385 DEBUG:boto:b'ffzdqu-14obj1000falseobj2023-09-22T21:09:06.711Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:09:42.385 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:42.385 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:42.386 DEBUG:boto:path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.386 DEBUG:boto:auth_path=/ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.386 DEBUG:boto:Method: GET 2023-09-22T21:09:42.386 DEBUG:boto:Path: /ffzdqu-14/?prefix=obj 2023-09-22T21:09:42.386 DEBUG:boto:Data: 2023-09-22T21:09:42.386 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.386 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:42.386 DEBUG:boto:Port: 8000 2023-09-22T21:09:42.386 DEBUG:boto:Params: {} 2023-09-22T21:09:42.387 DEBUG:boto:Token: None 2023-09-22T21:09:42.387 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/ 2023-09-22T21:09:42.387 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7RFwJQC0PXeo3TPDZHjZXWx/X64= 2023-09-22T21:09:42.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7RFwJQC0PXeo3TPDZHjZXWx/X64=', 'Content-Length': '0'} 2023-09-22T21:09:42.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002626691d2ee4e3cf-00650e0296-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.392 DEBUG:boto:b'ffzdqu-14obj1000falseobj2023-09-22T21:09:06.711Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:09:42.392 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-14 zones={a2, a1} 2023-09-22T21:09:42.392 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:42.392 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:42.393 DEBUG:boto:path=/ffzdqu-14/?versioning 2023-09-22T21:09:42.393 DEBUG:boto:auth_path=/ffzdqu-14/?versioning 2023-09-22T21:09:42.393 DEBUG:boto:Method: PUT 2023-09-22T21:09:42.393 DEBUG:boto:Path: /ffzdqu-14/?versioning 2023-09-22T21:09:42.393 DEBUG:boto:Data: Suspended Disabled 2023-09-22T21:09:42.393 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.393 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:42.393 DEBUG:boto:Port: 8000 2023-09-22T21:09:42.393 DEBUG:boto:Params: {} 2023-09-22T21:09:42.394 DEBUG:boto:Token: None 2023-09-22T21:09:42.394 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/?versioning 2023-09-22T21:09:42.394 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uyeAq4ZfeCP2FE29cUWerBOAolY= 2023-09-22T21:09:42.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uyeAq4ZfeCP2FE29cUWerBOAolY=', 'Content-Length': '230'} 2023-09-22T21:09:42.406 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000011832553ac02d83f-00650e0296-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:09:42.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:09:42.407 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:09:42.448 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:09:42.448 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:09:42.486 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:09:42.503 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695416982.408513_73.1","last_update":"2023-09-22T21:09:42.408513Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:09:42.503 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416982.408513_73.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:09:42.504 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:09:42.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:09:42.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:09:42.504 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:09:42.590 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:09:42.590 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:09:42.627 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:42.626+0000 7fc819594a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:09:42.627 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:42.626+0000 7fc819594a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:09:42.644 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695416982.408513_73.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:09:42.408513Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:09:42Z"} 2023-09-22T21:09:42.645 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695416982.408513_73.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:09:42.408513Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:09:42Z"} 2023-09-22T21:09:42.645 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695416982.408513_73.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:09:42.408513Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:09:42.645 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416982.408513_73.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:09:42.645 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695416982.408513_73.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:09:42.645 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:09:42.645 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:09:42.646 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:09:42.646 DEBUG:boto:Method: HEAD 2023-09-22T21:09:42.646 DEBUG:boto:Path: /ffzdqu-14/ 2023-09-22T21:09:42.646 DEBUG:boto:Data: 2023-09-22T21:09:42.646 DEBUG:boto:Headers: {} 2023-09-22T21:09:42.646 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:42.646 DEBUG:boto:Port: 8000 2023-09-22T21:09:42.646 DEBUG:boto:Params: {} 2023-09-22T21:09:42.646 DEBUG:boto:Token: None 2023-09-22T21:09:42.647 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/ 2023-09-22T21:09:42.647 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jYqC28vLiiewmY5trHE8HnVLDWg= 2023-09-22T21:09:42.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jYqC28vLiiewmY5trHE8HnVLDWg=', 'Content-Length': '0'} 2023-09-22T21:09:42.649 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007761976972b490e-00650e0296-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.649 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:09:42.650 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:09:42.650 DEBUG:boto:Method: PUT 2023-09-22T21:09:42.650 DEBUG:boto:Path: /ffzdqu-14/obj 2023-09-22T21:09:42.650 DEBUG:boto:Data: 2023-09-22T21:09:42.650 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-22T21:09:42.650 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:09:42.650 DEBUG:boto:Port: 8000 2023-09-22T21:09:42.650 DEBUG:boto:Params: {} 2023-09-22T21:09:42.650 DEBUG:boto:Token: None 2023-09-22T21:09:42.651 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 22 Sep 2023 21:09:42 GMT /ffzdqu-14/obj 2023-09-22T21:09:42.651 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MSmn+cm6ulR7NjCFsUAFF5yW1YI= 2023-09-22T21:09:42.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:09:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MSmn+cm6ulR7NjCFsUAFF5yW1YI='} 2023-09-22T21:09:42.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a923113555f3d2ff-00650e0296-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:09:42.662 DEBUG:rgw_multi.tests:created null version id=None 2023-09-22T21:09:42.663 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-14 2023-09-22T21:09:42.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:09:42.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:09:42.663 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:09:42.730 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:09:42.730 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:09:42.772 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:42.770+0000 7f4fe8d1fa40 1 waiting for incremental sync to catch up: 2023-09-22T21:09:42.772 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.9.13, , ] 2023-09-22T21:09:42.772 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.13.18, , ] 2023-09-22T21:09:47.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:47.774+0000 7f4fe8d1fa40 1 waiting for incremental sync to catch up: 2023-09-22T21:09:47.775 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.9.13, , ] 2023-09-22T21:09:47.775 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.13.18, , ] 2023-09-22T21:09:52.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:52.778+0000 7f4fe8d1fa40 1 waiting for incremental sync to catch up: 2023-09-22T21:09:52.778 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.9.13, , ] 2023-09-22T21:09:52.778 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.13.18, , ] 2023-09-22T21:09:57.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:09:57.778+0000 7f4fe8d1fa40 1 waiting for incremental sync to catch up: 2023-09-22T21:09:57.780 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.9.13, , ] 2023-09-22T21:09:57.780 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.13.18, , ] 2023-09-22T21:10:00.507 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:10:00.506+0000 7ff4815cb640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:10:02.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:10:02.782+0000 7f4fe8d1fa40 1 bucket sync caught up with source: 2023-09-22T21:10:02.782 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000005.13.18, , ] 2023-09-22T21:10:02.782 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.13.18, , ] 2023-09-22T21:10:02.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:10:02.782+0000 7f4fe8d1fa40 0 bucket checkpoint complete 2023-09-22T21:10:02.814 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-14 2023-09-22T21:10:02.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:10:02.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:10:02.815 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:10:02.909 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:10:02.909 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:10:02.958 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:10:02.958+0000 7f64a878fa40 1 waiting for incremental sync to catch up: 2023-09-22T21:10:02.959 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.26.13, , ] 2023-09-22T21:10:02.959 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.32.18, , ] 2023-09-22T21:10:07.961 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:10:07.962+0000 7f64a878fa40 1 waiting for incremental sync to catch up: 2023-09-22T21:10:07.961 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.26.13, , ] 2023-09-22T21:10:07.961 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.32.18, , ] 2023-09-22T21:10:12.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:10:12.962+0000 7f64a878fa40 1 waiting for incremental sync to catch up: 2023-09-22T21:10:12.963 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.26.13, , ] 2023-09-22T21:10:12.963 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.32.18, , ] 2023-09-22T21:10:17.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:10:17.966+0000 7f64a878fa40 1 waiting for incremental sync to catch up: 2023-09-22T21:10:17.965 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.26.13, , ] 2023-09-22T21:10:17.965 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.32.18, , ] 2023-09-22T21:10:22.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:10:22.966+0000 7f64a878fa40 1 waiting for incremental sync to catch up: 2023-09-22T21:10:22.967 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.26.13, , ] 2023-09-22T21:10:22.967 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.32.18, , ] 2023-09-22T21:10:27.969 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:10:27.970+0000 7f64a878fa40 1 waiting for incremental sync to catch up: 2023-09-22T21:10:27.969 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.26.13, , ] 2023-09-22T21:10:27.969 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.32.18, , ] 2023-09-22T21:10:32.971 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:10:32.969+0000 7f64a878fa40 1 waiting for incremental sync to catch up: 2023-09-22T21:10:32.971 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.26.13, , ] 2023-09-22T21:10:32.971 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.32.18, , ] 2023-09-22T21:10:37.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:10:37.973+0000 7f64a878fa40 1 waiting for incremental sync to catch up: 2023-09-22T21:10:37.974 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.26.13, , ] 2023-09-22T21:10:37.974 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.32.18, , ] 2023-09-22T21:10:42.976 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:10:42.973+0000 7f64a878fa40 1 bucket sync caught up with source: 2023-09-22T21:10:42.976 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000005.32.18, , ] 2023-09-22T21:10:42.976 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.32.18, , ] 2023-09-22T21:10:42.976 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:10:42.973+0000 7f64a878fa40 0 bucket checkpoint complete 2023-09-22T21:10:43.007 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-14 zones={a2, a1} 2023-09-22T21:10:43.007 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:10:43.007 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:10:43.007 DEBUG:boto:Method: HEAD 2023-09-22T21:10:43.007 DEBUG:boto:Path: /ffzdqu-14/ 2023-09-22T21:10:43.007 DEBUG:boto:Data: 2023-09-22T21:10:43.008 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.008 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:43.008 DEBUG:boto:Port: 8001 2023-09-22T21:10:43.008 DEBUG:boto:Params: {} 2023-09-22T21:10:43.008 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:10:43.008 DEBUG:boto:Token: None 2023-09-22T21:10:43.009 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/ 2023-09-22T21:10:43.009 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:758qiKbVs8XQGZaePd4Ciw0zpMI= 2023-09-22T21:10:43.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:758qiKbVs8XQGZaePd4Ciw0zpMI=', 'Content-Length': '0'} 2023-09-22T21:10:43.015 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000263eb8d326b61084-00650e02d3-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.015 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:10:43.015 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:10:43.015 DEBUG:boto:Method: HEAD 2023-09-22T21:10:43.015 DEBUG:boto:Path: /ffzdqu-14/ 2023-09-22T21:10:43.015 DEBUG:boto:Data: 2023-09-22T21:10:43.016 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.016 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:43.016 DEBUG:boto:Port: 8000 2023-09-22T21:10:43.016 DEBUG:boto:Params: {} 2023-09-22T21:10:43.016 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:10:43.016 DEBUG:boto:Token: None 2023-09-22T21:10:43.016 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/ 2023-09-22T21:10:43.017 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:758qiKbVs8XQGZaePd4Ciw0zpMI= 2023-09-22T21:10:43.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:758qiKbVs8XQGZaePd4Ciw0zpMI=', 'Content-Length': '0'} 2023-09-22T21:10:43.021 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000053ccabc9f515702-00650e02d3-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.021 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:10:43.021 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:10:43.021 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:10:43.021 DEBUG:boto:path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:10:43.021 DEBUG:boto:auth_path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:10:43.022 DEBUG:boto:Method: GET 2023-09-22T21:10:43.022 DEBUG:boto:Path: /ffzdqu-14/?versions&max-keys=999 2023-09-22T21:10:43.022 DEBUG:boto:Data: 2023-09-22T21:10:43.022 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.022 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:43.022 DEBUG:boto:Port: 8001 2023-09-22T21:10:43.022 DEBUG:boto:Params: {} 2023-09-22T21:10:43.022 DEBUG:boto:Token: None 2023-09-22T21:10:43.022 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/?versions 2023-09-22T21:10:43.023 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:643Gz/j7htKO6Xawah0fYtBzsKY= 2023-09-22T21:10:43.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:643Gz/j7htKO6Xawah0fYtBzsKY=', 'Content-Length': '0'} 2023-09-22T21:10:43.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b6b456cf18a8eef-00650e02d3-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.027 DEBUG:boto:b'ffzdqu-14999falseobjnulltrue2023-09-22T21:09:42.657Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJ8i90YbXhRfq0RtMmWPl-WhwfaK0Om7false2023-09-22T21:09:06.711Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:10:43.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:10:43.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:10:43.027 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:10:43.028 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:10:43.028 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:10:43.028 DEBUG:boto:path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:10:43.028 DEBUG:boto:auth_path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:10:43.028 DEBUG:boto:Method: GET 2023-09-22T21:10:43.028 DEBUG:boto:Path: /ffzdqu-14/?versions&max-keys=999 2023-09-22T21:10:43.028 DEBUG:boto:Data: 2023-09-22T21:10:43.028 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.029 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:43.029 DEBUG:boto:Port: 8000 2023-09-22T21:10:43.029 DEBUG:boto:Params: {} 2023-09-22T21:10:43.029 DEBUG:boto:Token: None 2023-09-22T21:10:43.029 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/?versions 2023-09-22T21:10:43.029 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:643Gz/j7htKO6Xawah0fYtBzsKY= 2023-09-22T21:10:43.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:643Gz/j7htKO6Xawah0fYtBzsKY=', 'Content-Length': '0'} 2023-09-22T21:10:43.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddb05ff5d61a12fe-00650e02d3-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.034 DEBUG:boto:b'ffzdqu-14999falseobjnulltrue2023-09-22T21:09:42.657Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJ8i90YbXhRfq0RtMmWPl-WhwfaK0Om7false2023-09-22T21:09:06.711Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:10:43.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:10:43.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:10:43.034 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:10:43.034 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:10:43.035 DEBUG:boto:path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:10:43.035 DEBUG:boto:auth_path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:10:43.035 DEBUG:boto:Method: GET 2023-09-22T21:10:43.035 DEBUG:boto:Path: /ffzdqu-14/?versions&max-keys=999 2023-09-22T21:10:43.035 DEBUG:boto:Data: 2023-09-22T21:10:43.035 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.035 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:43.035 DEBUG:boto:Port: 8001 2023-09-22T21:10:43.035 DEBUG:boto:Params: {} 2023-09-22T21:10:43.035 DEBUG:boto:Token: None 2023-09-22T21:10:43.036 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/?versions 2023-09-22T21:10:43.036 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:643Gz/j7htKO6Xawah0fYtBzsKY= 2023-09-22T21:10:43.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:643Gz/j7htKO6Xawah0fYtBzsKY=', 'Content-Length': '0'} 2023-09-22T21:10:43.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060f95f7ba3f377e9-00650e02d3-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.039 DEBUG:boto:b'ffzdqu-14999falseobjnulltrue2023-09-22T21:09:42.657Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJ8i90YbXhRfq0RtMmWPl-WhwfaK0Om7false2023-09-22T21:09:06.711Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:10:43.039 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:10:43.040 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:10:43.040 DEBUG:boto:path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:10:43.040 DEBUG:boto:auth_path=/ffzdqu-14/?versions&max-keys=999 2023-09-22T21:10:43.040 DEBUG:boto:Method: GET 2023-09-22T21:10:43.040 DEBUG:boto:Path: /ffzdqu-14/?versions&max-keys=999 2023-09-22T21:10:43.040 DEBUG:boto:Data: 2023-09-22T21:10:43.040 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.040 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:43.040 DEBUG:boto:Port: 8000 2023-09-22T21:10:43.041 DEBUG:boto:Params: {} 2023-09-22T21:10:43.041 DEBUG:boto:Token: None 2023-09-22T21:10:43.041 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/?versions 2023-09-22T21:10:43.041 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:643Gz/j7htKO6Xawah0fYtBzsKY= 2023-09-22T21:10:43.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:643Gz/j7htKO6Xawah0fYtBzsKY=', 'Content-Length': '0'} 2023-09-22T21:10:43.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bfe1a615ef54c1c-00650e02d3-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.045 DEBUG:boto:b'ffzdqu-14999falseobjnulltrue2023-09-22T21:09:42.657Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJ8i90YbXhRfq0RtMmWPl-WhwfaK0Om7false2023-09-22T21:09:06.711Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:10:43.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:10:43.045 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:10:43.045 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:10:43.045 DEBUG:boto:path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.045 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.046 DEBUG:boto:Method: GET 2023-09-22T21:10:43.046 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.046 DEBUG:boto:Data: 2023-09-22T21:10:43.046 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.046 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:43.046 DEBUG:boto:Port: 8001 2023-09-22T21:10:43.046 DEBUG:boto:Params: {} 2023-09-22T21:10:43.046 DEBUG:boto:Token: None 2023-09-22T21:10:43.047 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.047 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SXvvyzaX7o+O0fKvURGSOZBIkLE= 2023-09-22T21:10:43.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SXvvyzaX7o+O0fKvURGSOZBIkLE=', 'Content-Length': '0'} 2023-09-22T21:10:43.049 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c49938713c3bcce-00650e02d3-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.049 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:10:43.049 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:10:43.049 DEBUG:boto:path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.050 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.050 DEBUG:boto:Method: GET 2023-09-22T21:10:43.050 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.050 DEBUG:boto:Data: 2023-09-22T21:10:43.050 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.050 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:43.050 DEBUG:boto:Port: 8000 2023-09-22T21:10:43.050 DEBUG:boto:Params: {} 2023-09-22T21:10:43.050 DEBUG:boto:Token: None 2023-09-22T21:10:43.051 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.051 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SXvvyzaX7o+O0fKvURGSOZBIkLE= 2023-09-22T21:10:43.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SXvvyzaX7o+O0fKvURGSOZBIkLE=', 'Content-Length': '0'} 2023-09-22T21:10:43.053 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f18691512d463e4a-00650e02d3-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.053 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:10:43.053 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:10:43.054 DEBUG:boto:path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.054 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.054 DEBUG:boto:Method: HEAD 2023-09-22T21:10:43.054 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.054 DEBUG:boto:Data: 2023-09-22T21:10:43.054 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.054 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:43.054 DEBUG:boto:Port: 8001 2023-09-22T21:10:43.054 DEBUG:boto:Params: {} 2023-09-22T21:10:43.055 DEBUG:boto:Token: None 2023-09-22T21:10:43.055 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.055 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Tj/kSmh+vXMGH8Pet/tdrKB99Xs= 2023-09-22T21:10:43.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Tj/kSmh+vXMGH8Pet/tdrKB99Xs=', 'Content-Length': '0'} 2023-09-22T21:10:43.057 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009b7984f7202d8110-00650e02d3-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.057 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:10:43.057 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:10:43.057 DEBUG:boto:path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.058 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.058 DEBUG:boto:Method: HEAD 2023-09-22T21:10:43.058 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.058 DEBUG:boto:Data: 2023-09-22T21:10:43.058 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.058 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:43.058 DEBUG:boto:Port: 8000 2023-09-22T21:10:43.058 DEBUG:boto:Params: {} 2023-09-22T21:10:43.058 DEBUG:boto:Token: None 2023-09-22T21:10:43.059 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.059 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Tj/kSmh+vXMGH8Pet/tdrKB99Xs= 2023-09-22T21:10:43.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Tj/kSmh+vXMGH8Pet/tdrKB99Xs=', 'Content-Length': '0'} 2023-09-22T21:10:43.061 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000053b5fff9549a0715-00650e02d3-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:10:43.061 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:10:43.062 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:10:43.062 DEBUG:boto:path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.062 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.062 DEBUG:boto:Method: GET 2023-09-22T21:10:43.062 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.062 DEBUG:boto:Data: 2023-09-22T21:10:43.062 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.062 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:43.062 DEBUG:boto:Port: 8001 2023-09-22T21:10:43.063 DEBUG:boto:Params: {} 2023-09-22T21:10:43.063 DEBUG:boto:Token: None 2023-09-22T21:10:43.063 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.063 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SXvvyzaX7o+O0fKvURGSOZBIkLE= 2023-09-22T21:10:43.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SXvvyzaX7o+O0fKvURGSOZBIkLE=', 'Content-Length': '0'} 2023-09-22T21:10:43.065 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000364cebaa0ed4d822-00650e02d3-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.065 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:10:43.065 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:10:43.065 DEBUG:boto:path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.066 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.066 DEBUG:boto:Method: GET 2023-09-22T21:10:43.066 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.066 DEBUG:boto:Data: 2023-09-22T21:10:43.066 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.066 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:43.066 DEBUG:boto:Port: 8000 2023-09-22T21:10:43.066 DEBUG:boto:Params: {} 2023-09-22T21:10:43.066 DEBUG:boto:Token: None 2023-09-22T21:10:43.067 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/obj?versionId=null 2023-09-22T21:10:43.067 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SXvvyzaX7o+O0fKvURGSOZBIkLE= 2023-09-22T21:10:43.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SXvvyzaX7o+O0fKvURGSOZBIkLE=', 'Content-Length': '0'} 2023-09-22T21:10:43.069 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b3aeca5ce216db61-00650e02d3-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.069 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:10:43.069 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:10:43.069 DEBUG:boto:path=/ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.069 DEBUG:boto:auth_path=/ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.070 DEBUG:boto:Method: GET 2023-09-22T21:10:43.070 DEBUG:boto:Path: /ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.070 DEBUG:boto:Data: 2023-09-22T21:10:43.070 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.070 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:43.070 DEBUG:boto:Port: 8001 2023-09-22T21:10:43.070 DEBUG:boto:Params: {} 2023-09-22T21:10:43.070 DEBUG:boto:Token: None 2023-09-22T21:10:43.070 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/ 2023-09-22T21:10:43.071 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:l8ypx9gRnndHcnba1BG0s91SNYY= 2023-09-22T21:10:43.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:l8ypx9gRnndHcnba1BG0s91SNYY=', 'Content-Length': '0'} 2023-09-22T21:10:43.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dd9b2382720de83-00650e02d3-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.118 DEBUG:boto:b'ffzdqu-14obj1000falseobj2023-09-22T21:09:42.657Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:10:43.119 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:10:43.119 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:10:43.119 DEBUG:boto:path=/ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.119 DEBUG:boto:auth_path=/ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.119 DEBUG:boto:Method: GET 2023-09-22T21:10:43.119 DEBUG:boto:Path: /ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.119 DEBUG:boto:Data: 2023-09-22T21:10:43.120 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.120 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:43.120 DEBUG:boto:Port: 8000 2023-09-22T21:10:43.120 DEBUG:boto:Params: {} 2023-09-22T21:10:43.120 DEBUG:boto:Token: None 2023-09-22T21:10:43.120 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/ 2023-09-22T21:10:43.120 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:l8ypx9gRnndHcnba1BG0s91SNYY= 2023-09-22T21:10:43.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:l8ypx9gRnndHcnba1BG0s91SNYY=', 'Content-Length': '0'} 2023-09-22T21:10:43.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e86860029750c8bf-00650e02d3-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.125 DEBUG:boto:b'ffzdqu-14obj1000falseobj2023-09-22T21:09:42.657Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:10:43.125 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:10:43.126 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:10:43.126 DEBUG:boto:Method: HEAD 2023-09-22T21:10:43.126 DEBUG:boto:Path: /ffzdqu-14/obj 2023-09-22T21:10:43.126 DEBUG:boto:Data: 2023-09-22T21:10:43.126 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.126 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:43.126 DEBUG:boto:Port: 8001 2023-09-22T21:10:43.126 DEBUG:boto:Params: {} 2023-09-22T21:10:43.127 DEBUG:boto:Token: None 2023-09-22T21:10:43.127 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/obj 2023-09-22T21:10:43.127 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nXevQf/dJbka2H4nyQADaglRLEo= 2023-09-22T21:10:43.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nXevQf/dJbka2H4nyQADaglRLEo=', 'Content-Length': '0'} 2023-09-22T21:10:43.129 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000049560eb93c6a3e4b-00650e02d3-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.130 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:10:43.130 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:10:43.130 DEBUG:boto:Method: HEAD 2023-09-22T21:10:43.130 DEBUG:boto:Path: /ffzdqu-14/obj 2023-09-22T21:10:43.130 DEBUG:boto:Data: 2023-09-22T21:10:43.130 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.131 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:43.131 DEBUG:boto:Port: 8000 2023-09-22T21:10:43.131 DEBUG:boto:Params: {} 2023-09-22T21:10:43.131 DEBUG:boto:Token: None 2023-09-22T21:10:43.131 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/obj 2023-09-22T21:10:43.131 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nXevQf/dJbka2H4nyQADaglRLEo= 2023-09-22T21:10:43.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nXevQf/dJbka2H4nyQADaglRLEo=', 'Content-Length': '0'} 2023-09-22T21:10:43.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b89e5517635269da-00650e02d3-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:10:43.134 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:10:43.134 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:10:43.134 DEBUG:boto:Method: GET 2023-09-22T21:10:43.135 DEBUG:boto:Path: /ffzdqu-14/obj 2023-09-22T21:10:43.135 DEBUG:boto:Data: 2023-09-22T21:10:43.135 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.135 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:43.135 DEBUG:boto:Port: 8001 2023-09-22T21:10:43.135 DEBUG:boto:Params: {} 2023-09-22T21:10:43.135 DEBUG:boto:Token: None 2023-09-22T21:10:43.135 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/obj 2023-09-22T21:10:43.136 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vC/XxRpJoYoJsTrZJtV8U6kovfQ= 2023-09-22T21:10:43.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vC/XxRpJoYoJsTrZJtV8U6kovfQ=', 'Content-Length': '0'} 2023-09-22T21:10:43.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000165173d421d14fd2-00650e02d3-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.138 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:10:43.139 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:10:43.139 DEBUG:boto:Method: GET 2023-09-22T21:10:43.139 DEBUG:boto:Path: /ffzdqu-14/obj 2023-09-22T21:10:43.139 DEBUG:boto:Data: 2023-09-22T21:10:43.139 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.139 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:43.139 DEBUG:boto:Port: 8000 2023-09-22T21:10:43.139 DEBUG:boto:Params: {} 2023-09-22T21:10:43.140 DEBUG:boto:Token: None 2023-09-22T21:10:43.140 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/obj 2023-09-22T21:10:43.140 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vC/XxRpJoYoJsTrZJtV8U6kovfQ= 2023-09-22T21:10:43.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vC/XxRpJoYoJsTrZJtV8U6kovfQ=', 'Content-Length': '0'} 2023-09-22T21:10:43.142 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002328d6800f92285c-00650e02d3-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.143 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:10:43.143 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:10:43.143 DEBUG:boto:path=/ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.143 DEBUG:boto:auth_path=/ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.143 DEBUG:boto:Method: GET 2023-09-22T21:10:43.143 DEBUG:boto:Path: /ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.143 DEBUG:boto:Data: 2023-09-22T21:10:43.144 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.144 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:43.144 DEBUG:boto:Port: 8001 2023-09-22T21:10:43.144 DEBUG:boto:Params: {} 2023-09-22T21:10:43.144 DEBUG:boto:Token: None 2023-09-22T21:10:43.144 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/ 2023-09-22T21:10:43.144 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:l8ypx9gRnndHcnba1BG0s91SNYY= 2023-09-22T21:10:43.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:l8ypx9gRnndHcnba1BG0s91SNYY=', 'Content-Length': '0'} 2023-09-22T21:10:43.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe2823fb0207ae63-00650e02d3-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.149 DEBUG:boto:b'ffzdqu-14obj1000falseobj2023-09-22T21:09:42.657Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:10:43.149 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:10:43.149 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:10:43.150 DEBUG:boto:path=/ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.150 DEBUG:boto:auth_path=/ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.150 DEBUG:boto:Method: GET 2023-09-22T21:10:43.150 DEBUG:boto:Path: /ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.150 DEBUG:boto:Data: 2023-09-22T21:10:43.150 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.150 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:43.150 DEBUG:boto:Port: 8000 2023-09-22T21:10:43.151 DEBUG:boto:Params: {} 2023-09-22T21:10:43.151 DEBUG:boto:Token: None 2023-09-22T21:10:43.151 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/ 2023-09-22T21:10:43.151 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:l8ypx9gRnndHcnba1BG0s91SNYY= 2023-09-22T21:10:43.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:l8ypx9gRnndHcnba1BG0s91SNYY=', 'Content-Length': '0'} 2023-09-22T21:10:43.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082e84475b22e3c48-00650e02d3-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.198 DEBUG:boto:b'ffzdqu-14obj1000falseobj2023-09-22T21:09:42.657Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:10:43.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:10:43.199 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:10:43.199 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:10:43.199 DEBUG:boto:path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.199 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.199 DEBUG:boto:Method: GET 2023-09-22T21:10:43.199 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.200 DEBUG:boto:Data: 2023-09-22T21:10:43.200 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.200 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:43.200 DEBUG:boto:Port: 8001 2023-09-22T21:10:43.200 DEBUG:boto:Params: {} 2023-09-22T21:10:43.200 DEBUG:boto:Token: None 2023-09-22T21:10:43.200 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.200 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:crdUC0+c3RGm6epm3oQhBw2uwdw= 2023-09-22T21:10:43.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:crdUC0+c3RGm6epm3oQhBw2uwdw=', 'Content-Length': '0'} 2023-09-22T21:10:43.203 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('x-amz-version-id', 'J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000366e481eaac97117-00650e02d3-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.203 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:10:43.204 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:10:43.204 DEBUG:boto:path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.204 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.204 DEBUG:boto:Method: GET 2023-09-22T21:10:43.204 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.204 DEBUG:boto:Data: 2023-09-22T21:10:43.204 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.204 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:43.204 DEBUG:boto:Port: 8000 2023-09-22T21:10:43.204 DEBUG:boto:Params: {} 2023-09-22T21:10:43.205 DEBUG:boto:Token: None 2023-09-22T21:10:43.205 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.205 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:crdUC0+c3RGm6epm3oQhBw2uwdw= 2023-09-22T21:10:43.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:crdUC0+c3RGm6epm3oQhBw2uwdw=', 'Content-Length': '0'} 2023-09-22T21:10:43.207 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('x-amz-version-id', 'J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005c8fe7e2e5439628-00650e02d3-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.208 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:10:43.208 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:10:43.208 DEBUG:boto:path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.208 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.208 DEBUG:boto:Method: HEAD 2023-09-22T21:10:43.208 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.208 DEBUG:boto:Data: 2023-09-22T21:10:43.209 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.209 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:43.209 DEBUG:boto:Port: 8001 2023-09-22T21:10:43.209 DEBUG:boto:Params: {} 2023-09-22T21:10:43.209 DEBUG:boto:Token: None 2023-09-22T21:10:43.209 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.209 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LaFHX6E1izLfToGB0+8iHB5lRuU= 2023-09-22T21:10:43.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LaFHX6E1izLfToGB0+8iHB5lRuU=', 'Content-Length': '0'} 2023-09-22T21:10:43.212 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('x-amz-version-id', 'J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000033da9a4d5a26fb63-00650e02d3-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.213 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:10:43.213 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:10:43.213 DEBUG:boto:path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.213 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.213 DEBUG:boto:Method: HEAD 2023-09-22T21:10:43.213 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.213 DEBUG:boto:Data: 2023-09-22T21:10:43.213 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.214 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:43.214 DEBUG:boto:Port: 8000 2023-09-22T21:10:43.214 DEBUG:boto:Params: {} 2023-09-22T21:10:43.214 DEBUG:boto:Token: None 2023-09-22T21:10:43.214 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.214 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LaFHX6E1izLfToGB0+8iHB5lRuU= 2023-09-22T21:10:43.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LaFHX6E1izLfToGB0+8iHB5lRuU=', 'Content-Length': '0'} 2023-09-22T21:10:43.217 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('x-amz-version-id', 'J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009a8f56c186d68283-00650e02d3-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:10:43.217 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:10:43.218 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:10:43.218 DEBUG:boto:path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.218 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.218 DEBUG:boto:Method: GET 2023-09-22T21:10:43.218 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.218 DEBUG:boto:Data: 2023-09-22T21:10:43.218 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.218 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:43.218 DEBUG:boto:Port: 8001 2023-09-22T21:10:43.219 DEBUG:boto:Params: {} 2023-09-22T21:10:43.219 DEBUG:boto:Token: None 2023-09-22T21:10:43.219 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.219 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:crdUC0+c3RGm6epm3oQhBw2uwdw= 2023-09-22T21:10:43.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:crdUC0+c3RGm6epm3oQhBw2uwdw=', 'Content-Length': '0'} 2023-09-22T21:10:43.221 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('x-amz-version-id', 'J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008b99706fdee62b1c-00650e02d3-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.222 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:10:43.222 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:10:43.222 DEBUG:boto:path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.222 DEBUG:boto:auth_path=/ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.222 DEBUG:boto:Method: GET 2023-09-22T21:10:43.222 DEBUG:boto:Path: /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.222 DEBUG:boto:Data: 2023-09-22T21:10:43.223 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.223 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:43.223 DEBUG:boto:Port: 8000 2023-09-22T21:10:43.223 DEBUG:boto:Params: {} 2023-09-22T21:10:43.223 DEBUG:boto:Token: None 2023-09-22T21:10:43.223 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/obj?versionId=J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7 2023-09-22T21:10:43.223 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:crdUC0+c3RGm6epm3oQhBw2uwdw= 2023-09-22T21:10:43.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:crdUC0+c3RGm6epm3oQhBw2uwdw=', 'Content-Length': '0'} 2023-09-22T21:10:43.226 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:06 GMT'), ('x-amz-version-id', 'J8i90YbXhRfq0RtMmWPl-WhwfaK0Om7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fd16cd92d3cd7b3b-00650e02d3-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.226 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:10:43.226 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:10:43.227 DEBUG:boto:path=/ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.227 DEBUG:boto:auth_path=/ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.227 DEBUG:boto:Method: GET 2023-09-22T21:10:43.227 DEBUG:boto:Path: /ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.227 DEBUG:boto:Data: 2023-09-22T21:10:43.227 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.227 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:43.227 DEBUG:boto:Port: 8001 2023-09-22T21:10:43.228 DEBUG:boto:Params: {} 2023-09-22T21:10:43.228 DEBUG:boto:Token: None 2023-09-22T21:10:43.228 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/ 2023-09-22T21:10:43.228 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:l8ypx9gRnndHcnba1BG0s91SNYY= 2023-09-22T21:10:43.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:l8ypx9gRnndHcnba1BG0s91SNYY=', 'Content-Length': '0'} 2023-09-22T21:10:43.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078a6413c737294fc-00650e02d3-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.278 DEBUG:boto:b'ffzdqu-14obj1000falseobj2023-09-22T21:09:42.657Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:10:43.279 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:10:43.279 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:10:43.279 DEBUG:boto:path=/ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.279 DEBUG:boto:auth_path=/ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.279 DEBUG:boto:Method: GET 2023-09-22T21:10:43.279 DEBUG:boto:Path: /ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.280 DEBUG:boto:Data: 2023-09-22T21:10:43.280 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.280 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:43.280 DEBUG:boto:Port: 8000 2023-09-22T21:10:43.280 DEBUG:boto:Params: {} 2023-09-22T21:10:43.280 DEBUG:boto:Token: None 2023-09-22T21:10:43.280 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/ 2023-09-22T21:10:43.280 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:l8ypx9gRnndHcnba1BG0s91SNYY= 2023-09-22T21:10:43.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:l8ypx9gRnndHcnba1BG0s91SNYY=', 'Content-Length': '0'} 2023-09-22T21:10:43.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005626bb95393118ae-00650e02d3-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.286 DEBUG:boto:b'ffzdqu-14obj1000falseobj2023-09-22T21:09:42.657Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:10:43.286 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:10:43.286 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:10:43.286 DEBUG:boto:Method: HEAD 2023-09-22T21:10:43.287 DEBUG:boto:Path: /ffzdqu-14/obj 2023-09-22T21:10:43.287 DEBUG:boto:Data: 2023-09-22T21:10:43.287 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.287 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:43.287 DEBUG:boto:Port: 8001 2023-09-22T21:10:43.287 DEBUG:boto:Params: {} 2023-09-22T21:10:43.287 DEBUG:boto:Token: None 2023-09-22T21:10:43.287 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/obj 2023-09-22T21:10:43.287 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nXevQf/dJbka2H4nyQADaglRLEo= 2023-09-22T21:10:43.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nXevQf/dJbka2H4nyQADaglRLEo=', 'Content-Length': '0'} 2023-09-22T21:10:43.290 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000108062c377a6e996-00650e02d3-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.290 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:10:43.291 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:10:43.291 DEBUG:boto:Method: HEAD 2023-09-22T21:10:43.291 DEBUG:boto:Path: /ffzdqu-14/obj 2023-09-22T21:10:43.291 DEBUG:boto:Data: 2023-09-22T21:10:43.291 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.291 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:43.291 DEBUG:boto:Port: 8000 2023-09-22T21:10:43.291 DEBUG:boto:Params: {} 2023-09-22T21:10:43.291 DEBUG:boto:Token: None 2023-09-22T21:10:43.292 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/obj 2023-09-22T21:10:43.292 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nXevQf/dJbka2H4nyQADaglRLEo= 2023-09-22T21:10:43.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nXevQf/dJbka2H4nyQADaglRLEo=', 'Content-Length': '0'} 2023-09-22T21:10:43.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000035d8e6bf8ec76c62-00650e02d3-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:10:43.295 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:10:43.295 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:10:43.295 DEBUG:boto:Method: GET 2023-09-22T21:10:43.295 DEBUG:boto:Path: /ffzdqu-14/obj 2023-09-22T21:10:43.295 DEBUG:boto:Data: 2023-09-22T21:10:43.295 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.295 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:43.296 DEBUG:boto:Port: 8001 2023-09-22T21:10:43.296 DEBUG:boto:Params: {} 2023-09-22T21:10:43.296 DEBUG:boto:Token: None 2023-09-22T21:10:43.296 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/obj 2023-09-22T21:10:43.296 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vC/XxRpJoYoJsTrZJtV8U6kovfQ= 2023-09-22T21:10:43.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vC/XxRpJoYoJsTrZJtV8U6kovfQ=', 'Content-Length': '0'} 2023-09-22T21:10:43.298 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006549b186fd271629-00650e02d3-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.299 DEBUG:boto:path=/ffzdqu-14/obj 2023-09-22T21:10:43.299 DEBUG:boto:auth_path=/ffzdqu-14/obj 2023-09-22T21:10:43.299 DEBUG:boto:Method: GET 2023-09-22T21:10:43.299 DEBUG:boto:Path: /ffzdqu-14/obj 2023-09-22T21:10:43.299 DEBUG:boto:Data: 2023-09-22T21:10:43.299 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.299 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:43.299 DEBUG:boto:Port: 8000 2023-09-22T21:10:43.300 DEBUG:boto:Params: {} 2023-09-22T21:10:43.300 DEBUG:boto:Token: None 2023-09-22T21:10:43.300 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/obj 2023-09-22T21:10:43.300 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vC/XxRpJoYoJsTrZJtV8U6kovfQ= 2023-09-22T21:10:43.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vC/XxRpJoYoJsTrZJtV8U6kovfQ=', 'Content-Length': '0'} 2023-09-22T21:10:43.303 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:09:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000a8d18f7c0902a83-00650e02d3-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.303 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:10:43.303 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:10:43.303 DEBUG:boto:path=/ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.303 DEBUG:boto:auth_path=/ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.303 DEBUG:boto:Method: GET 2023-09-22T21:10:43.303 DEBUG:boto:Path: /ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.303 DEBUG:boto:Data: 2023-09-22T21:10:43.304 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.304 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:43.304 DEBUG:boto:Port: 8001 2023-09-22T21:10:43.304 DEBUG:boto:Params: {} 2023-09-22T21:10:43.304 DEBUG:boto:Token: None 2023-09-22T21:10:43.304 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/ 2023-09-22T21:10:43.304 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:l8ypx9gRnndHcnba1BG0s91SNYY= 2023-09-22T21:10:43.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:l8ypx9gRnndHcnba1BG0s91SNYY=', 'Content-Length': '0'} 2023-09-22T21:10:43.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000956c6e01ada67fa-00650e02d3-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.350 DEBUG:boto:b'ffzdqu-14obj1000falseobj2023-09-22T21:09:42.657Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:10:43.351 DEBUG:boto:path=/ffzdqu-14/ 2023-09-22T21:10:43.351 DEBUG:boto:auth_path=/ffzdqu-14/ 2023-09-22T21:10:43.351 DEBUG:boto:path=/ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.351 DEBUG:boto:auth_path=/ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.351 DEBUG:boto:Method: GET 2023-09-22T21:10:43.351 DEBUG:boto:Path: /ffzdqu-14/?prefix=obj 2023-09-22T21:10:43.352 DEBUG:boto:Data: 2023-09-22T21:10:43.352 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.352 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:43.352 DEBUG:boto:Port: 8000 2023-09-22T21:10:43.352 DEBUG:boto:Params: {} 2023-09-22T21:10:43.352 DEBUG:boto:Token: None 2023-09-22T21:10:43.352 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-14/ 2023-09-22T21:10:43.353 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:l8ypx9gRnndHcnba1BG0s91SNYY= 2023-09-22T21:10:43.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:l8ypx9gRnndHcnba1BG0s91SNYY=', 'Content-Length': '0'} 2023-09-22T21:10:43.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a056e77cd1bc037-00650e02d3-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.358 DEBUG:boto:b'ffzdqu-14obj1000falseobj2023-09-22T21:09:42.657Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:10:43.358 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-14 zones={a2, a1} 2023-09-22T21:10:43.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2023-09-22T21:10:43.359 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-15 2023-09-22T21:10:43.360 DEBUG:boto:path=/ffzdqu-15/ 2023-09-22T21:10:43.360 DEBUG:boto:auth_path=/ffzdqu-15/ 2023-09-22T21:10:43.360 DEBUG:boto:Method: PUT 2023-09-22T21:10:43.360 DEBUG:boto:Path: /ffzdqu-15/ 2023-09-22T21:10:43.360 DEBUG:boto:Data: 2023-09-22T21:10:43.360 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.360 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:43.360 DEBUG:boto:Port: 8000 2023-09-22T21:10:43.360 DEBUG:boto:Params: {} 2023-09-22T21:10:43.361 DEBUG:boto:Token: None 2023-09-22T21:10:43.361 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-15/ 2023-09-22T21:10:43.361 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2T10t01q8u7PP99NIt1TjSxDPgc= 2023-09-22T21:10:43.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2T10t01q8u7PP99NIt1TjSxDPgc=', 'Content-Length': '0'} 2023-09-22T21:10:43.398 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d42df32f58bd045c-00650e02d3-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.398 INFO:rgw_multi.tests:create bucket zone=a2 name=ffzdqu-16 2023-09-22T21:10:43.398 DEBUG:boto:path=/ffzdqu-16/ 2023-09-22T21:10:43.398 DEBUG:boto:auth_path=/ffzdqu-16/ 2023-09-22T21:10:43.399 DEBUG:boto:Method: PUT 2023-09-22T21:10:43.399 DEBUG:boto:Path: /ffzdqu-16/ 2023-09-22T21:10:43.399 DEBUG:boto:Data: 2023-09-22T21:10:43.399 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.399 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:43.399 DEBUG:boto:Port: 8001 2023-09-22T21:10:43.399 DEBUG:boto:Params: {} 2023-09-22T21:10:43.399 DEBUG:boto:Token: None 2023-09-22T21:10:43.400 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-16/ 2023-09-22T21:10:43.400 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FX4lJiFgjhbwX9NP3DPXNQLmalo= 2023-09-22T21:10:43.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FX4lJiFgjhbwX9NP3DPXNQLmalo=', 'Content-Length': '0'} 2023-09-22T21:10:43.465 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000020f2263d6f90e9b8-00650e02d3-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.465 DEBUG:boto:path=/ffzdqu-15/ 2023-09-22T21:10:43.465 DEBUG:boto:auth_path=/ffzdqu-15/ 2023-09-22T21:10:43.465 DEBUG:boto:path=/ffzdqu-15/?versioning 2023-09-22T21:10:43.465 DEBUG:boto:auth_path=/ffzdqu-15/?versioning 2023-09-22T21:10:43.466 DEBUG:boto:Method: PUT 2023-09-22T21:10:43.466 DEBUG:boto:Path: /ffzdqu-15/?versioning 2023-09-22T21:10:43.466 DEBUG:boto:Data: Enabled Disabled 2023-09-22T21:10:43.466 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.466 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:43.466 DEBUG:boto:Port: 8000 2023-09-22T21:10:43.466 DEBUG:boto:Params: {} 2023-09-22T21:10:43.466 DEBUG:boto:Token: None 2023-09-22T21:10:43.467 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-15/?versioning 2023-09-22T21:10:43.467 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ps9vGGiKlnC5zoHlEqlAYD5vtYk= 2023-09-22T21:10:43.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ps9vGGiKlnC5zoHlEqlAYD5vtYk=', 'Content-Length': '228'} 2023-09-22T21:10:43.479 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000066a1750dc9936fea-00650e02d3-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.480 DEBUG:boto:path=/ffzdqu-16/ 2023-09-22T21:10:43.480 DEBUG:boto:auth_path=/ffzdqu-16/ 2023-09-22T21:10:43.480 DEBUG:boto:path=/ffzdqu-16/?versioning 2023-09-22T21:10:43.480 DEBUG:boto:auth_path=/ffzdqu-16/?versioning 2023-09-22T21:10:43.480 DEBUG:boto:Method: PUT 2023-09-22T21:10:43.480 DEBUG:boto:Path: /ffzdqu-16/?versioning 2023-09-22T21:10:43.480 DEBUG:boto:Data: Enabled Disabled 2023-09-22T21:10:43.481 DEBUG:boto:Headers: {} 2023-09-22T21:10:43.481 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:43.481 DEBUG:boto:Port: 8001 2023-09-22T21:10:43.481 DEBUG:boto:Params: {} 2023-09-22T21:10:43.481 DEBUG:boto:Token: None 2023-09-22T21:10:43.481 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:10:43 GMT /ffzdqu-16/?versioning 2023-09-22T21:10:43.481 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YHle/vaujve2Fz8my71UbpsMiKk= 2023-09-22T21:10:43.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YHle/vaujve2Fz8my71UbpsMiKk=', 'Content-Length': '228'} 2023-09-22T21:10:43.500 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000054c59c2e200c0cd8-00650e02d3-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:43.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:10:43.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:10:43.500 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:10:43.541 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:10:43.541 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:10:43.587 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:10:43.608 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695417043.497936_85.1","last_update":"2023-09-22T21:10:43.497936Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:10:43.609 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417043.497936_85.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:10:43.609 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:10:43.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:10:43.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:10:43.609 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:10:43.690 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:10:43.690 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:10:43.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:10:43.725+0000 7f1ba46aba40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:10:43.727 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:10:43.725+0000 7f1ba46aba40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:10:43.745 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417043.380222_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:10:43.380222Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:10:43Z"} 2023-09-22T21:10:43.745 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417043.380222_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:10:43.380222Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:10:43Z"} 2023-09-22T21:10:43.745 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417043.380222_74.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:10:43.380222Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:10:43.745 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417043.497936_85.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:10:43.745 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417043.380222_74.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:10:43.745 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417043.497936_85.1 target=1_1695417043.380222_74.1 2023-09-22T21:10:48.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:10:48.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:10:48.747 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:10:48.789 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:10:48.790 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:10:48.828 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:10:48.825+0000 7f6e30fc9a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:10:48.829 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:10:48.829+0000 7f6e30fc9a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:10:48.847 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417043.497936_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:10:43.497936Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:10:48Z"} 2023-09-22T21:10:48.847 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417043.497936_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:10:43.497936Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:10:48Z"} 2023-09-22T21:10:48.848 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417043.497936_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:10:43.497936Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:10:48.848 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417043.497936_85.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:10:48.848 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417043.497936_85.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:10:48.848 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:10:48.848 DEBUG:boto:path=/ffzdqu-15/ 2023-09-22T21:10:48.848 DEBUG:boto:auth_path=/ffzdqu-15/ 2023-09-22T21:10:48.848 DEBUG:boto:Method: HEAD 2023-09-22T21:10:48.848 DEBUG:boto:Path: /ffzdqu-15/ 2023-09-22T21:10:48.849 DEBUG:boto:Data: 2023-09-22T21:10:48.849 DEBUG:boto:Headers: {} 2023-09-22T21:10:48.849 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:48.849 DEBUG:boto:Port: 8000 2023-09-22T21:10:48.849 DEBUG:boto:Params: {} 2023-09-22T21:10:48.849 DEBUG:boto:Token: None 2023-09-22T21:10:48.849 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:10:48 GMT /ffzdqu-15/ 2023-09-22T21:10:48.850 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/FuAkMqYp1AQPXdXSmHu9UDJOQw= 2023-09-22T21:10:48.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/FuAkMqYp1AQPXdXSmHu9UDJOQw=', 'Content-Length': '0'} 2023-09-22T21:10:48.852 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a590b2d4e76f755-00650e02d8-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:10:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:48.852 DEBUG:boto:path=/ffzdqu-15/obj 2023-09-22T21:10:48.852 DEBUG:boto:auth_path=/ffzdqu-15/obj 2023-09-22T21:10:48.852 DEBUG:boto:Method: PUT 2023-09-22T21:10:48.853 DEBUG:boto:Path: /ffzdqu-15/obj 2023-09-22T21:10:48.853 DEBUG:boto:Data: 2023-09-22T21:10:48.853 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-22T21:10:48.853 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:48.853 DEBUG:boto:Port: 8000 2023-09-22T21:10:48.853 DEBUG:boto:Params: {} 2023-09-22T21:10:48.853 DEBUG:boto:Token: None 2023-09-22T21:10:48.853 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 22 Sep 2023 21:10:48 GMT /ffzdqu-15/obj 2023-09-22T21:10:48.854 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BJw6QbLUV/sXClGgQqZx+ZyWKnc= 2023-09-22T21:10:48.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:10:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BJw6QbLUV/sXClGgQqZx+ZyWKnc='} 2023-09-22T21:10:48.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'wdhULduirIbqsMn5d.L2a7FG4kVi1gZ'), ('x-amz-request-id', 'tx000001453ce1ec2b3c876-00650e02d8-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:48.866 DEBUG:boto:path=/ffzdqu-15/ 2023-09-22T21:10:48.866 DEBUG:boto:auth_path=/ffzdqu-15/ 2023-09-22T21:10:48.866 DEBUG:boto:Method: HEAD 2023-09-22T21:10:48.866 DEBUG:boto:Path: /ffzdqu-15/ 2023-09-22T21:10:48.867 DEBUG:boto:Data: 2023-09-22T21:10:48.867 DEBUG:boto:Headers: {} 2023-09-22T21:10:48.867 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:48.867 DEBUG:boto:Port: 8000 2023-09-22T21:10:48.867 DEBUG:boto:Params: {} 2023-09-22T21:10:48.867 DEBUG:boto:Token: None 2023-09-22T21:10:48.867 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:10:48 GMT /ffzdqu-15/ 2023-09-22T21:10:48.867 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/FuAkMqYp1AQPXdXSmHu9UDJOQw= 2023-09-22T21:10:48.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/FuAkMqYp1AQPXdXSmHu9UDJOQw=', 'Content-Length': '0'} 2023-09-22T21:10:48.869 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008aaa4b5839ca2936-00650e02d8-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:10:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:48.870 DEBUG:boto:path=/ffzdqu-15/obj 2023-09-22T21:10:48.870 DEBUG:boto:auth_path=/ffzdqu-15/obj 2023-09-22T21:10:48.870 DEBUG:boto:Method: DELETE 2023-09-22T21:10:48.870 DEBUG:boto:Path: /ffzdqu-15/obj 2023-09-22T21:10:48.870 DEBUG:boto:Data: 2023-09-22T21:10:48.870 DEBUG:boto:Headers: {} 2023-09-22T21:10:48.870 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:10:48.871 DEBUG:boto:Port: 8000 2023-09-22T21:10:48.871 DEBUG:boto:Params: {} 2023-09-22T21:10:48.871 DEBUG:boto:Token: None 2023-09-22T21:10:48.871 DEBUG:boto:StringToSign: DELETE Fri, 22 Sep 2023 21:10:48 GMT /ffzdqu-15/obj 2023-09-22T21:10:48.871 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ETcd3vRfogr2Fuso5cwMCsZ+fDM= 2023-09-22T21:10:48.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ETcd3vRfogr2Fuso5cwMCsZ+fDM=', 'Content-Length': '0'} 2023-09-22T21:10:48.879 DEBUG:boto:Response headers: [('x-amz-version-id', 'cbFI3snwFVP-tRtlvdB0Gq7QIRe9-3k'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000fb9d75cacecec1e8-00650e02d8-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:48.879 DEBUG:boto:path=/ffzdqu-16/ 2023-09-22T21:10:48.879 DEBUG:boto:auth_path=/ffzdqu-16/ 2023-09-22T21:10:48.879 DEBUG:boto:Method: HEAD 2023-09-22T21:10:48.879 DEBUG:boto:Path: /ffzdqu-16/ 2023-09-22T21:10:48.879 DEBUG:boto:Data: 2023-09-22T21:10:48.879 DEBUG:boto:Headers: {} 2023-09-22T21:10:48.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:48.879 DEBUG:boto:Port: 8001 2023-09-22T21:10:48.880 DEBUG:boto:Params: {} 2023-09-22T21:10:48.880 DEBUG:boto:Token: None 2023-09-22T21:10:48.880 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:10:48 GMT /ffzdqu-16/ 2023-09-22T21:10:48.880 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6JiVmyuoch0P8I4QUA5yU04kr2Q= 2023-09-22T21:10:48.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6JiVmyuoch0P8I4QUA5yU04kr2Q=', 'Content-Length': '0'} 2023-09-22T21:10:48.882 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b42e998fee11e6d-00650e02d8-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:10:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:48.882 DEBUG:boto:path=/ffzdqu-16/obj 2023-09-22T21:10:48.882 DEBUG:boto:auth_path=/ffzdqu-16/obj 2023-09-22T21:10:48.883 DEBUG:boto:Method: PUT 2023-09-22T21:10:48.883 DEBUG:boto:Path: /ffzdqu-16/obj 2023-09-22T21:10:48.883 DEBUG:boto:Data: 2023-09-22T21:10:48.883 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-22T21:10:48.883 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:48.883 DEBUG:boto:Port: 8001 2023-09-22T21:10:48.883 DEBUG:boto:Params: {} 2023-09-22T21:10:48.883 DEBUG:boto:Token: None 2023-09-22T21:10:48.883 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 22 Sep 2023 21:10:48 GMT /ffzdqu-16/obj 2023-09-22T21:10:48.884 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1+iLO3WCoJk8m3ypm/GMkv4GsK0= 2023-09-22T21:10:48.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:10:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1+iLO3WCoJk8m3ypm/GMkv4GsK0='} 2023-09-22T21:10:48.897 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '1JC9Icw-ucSKkSxsGX754pzA-PCw9bG'), ('x-amz-request-id', 'tx0000017ea7d4967e52ded-00650e02d8-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:48.897 DEBUG:boto:path=/ffzdqu-16/ 2023-09-22T21:10:48.898 DEBUG:boto:auth_path=/ffzdqu-16/ 2023-09-22T21:10:48.898 DEBUG:boto:Method: HEAD 2023-09-22T21:10:48.898 DEBUG:boto:Path: /ffzdqu-16/ 2023-09-22T21:10:48.898 DEBUG:boto:Data: 2023-09-22T21:10:48.898 DEBUG:boto:Headers: {} 2023-09-22T21:10:48.898 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:48.898 DEBUG:boto:Port: 8001 2023-09-22T21:10:48.898 DEBUG:boto:Params: {} 2023-09-22T21:10:48.898 DEBUG:boto:Token: None 2023-09-22T21:10:48.899 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:10:48 GMT /ffzdqu-16/ 2023-09-22T21:10:48.899 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6JiVmyuoch0P8I4QUA5yU04kr2Q= 2023-09-22T21:10:48.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6JiVmyuoch0P8I4QUA5yU04kr2Q=', 'Content-Length': '0'} 2023-09-22T21:10:48.901 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f2f99101f177af9-00650e02d8-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:10:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:48.901 DEBUG:boto:path=/ffzdqu-16/obj 2023-09-22T21:10:48.901 DEBUG:boto:auth_path=/ffzdqu-16/obj 2023-09-22T21:10:48.901 DEBUG:boto:Method: DELETE 2023-09-22T21:10:48.901 DEBUG:boto:Path: /ffzdqu-16/obj 2023-09-22T21:10:48.902 DEBUG:boto:Data: 2023-09-22T21:10:48.902 DEBUG:boto:Headers: {} 2023-09-22T21:10:48.902 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:10:48.902 DEBUG:boto:Port: 8001 2023-09-22T21:10:48.902 DEBUG:boto:Params: {} 2023-09-22T21:10:48.902 DEBUG:boto:Token: None 2023-09-22T21:10:48.902 DEBUG:boto:StringToSign: DELETE Fri, 22 Sep 2023 21:10:48 GMT /ffzdqu-16/obj 2023-09-22T21:10:48.902 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rJAIw/hNF8T5o0hoVJQrY+BPMbw= 2023-09-22T21:10:48.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:10:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rJAIw/hNF8T5o0hoVJQrY+BPMbw=', 'Content-Length': '0'} 2023-09-22T21:10:48.911 DEBUG:boto:Response headers: [('x-amz-version-id', 'Juvk1sb6t2yV.GGrRbOXcJf7E-yZOyz'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000fec8c436e274b460-00650e02d8-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:10:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:10:48.912 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-15 2023-09-22T21:10:48.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:10:48.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:10:48.912 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-15 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:10:48.954 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:10:48.954 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:10:48.998 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:10:48.997+0000 7fbd4f74aa40 1 waiting to reach incremental sync.. 2023-09-22T21:10:53.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:10:54.001+0000 7fbd4f74aa40 1 waiting to reach incremental sync.. 2023-09-22T21:10:59.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:10:59.001+0000 7fbd4f74aa40 1 waiting to reach incremental sync.. 2023-09-22T21:11:04.003 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:04.005+0000 7fbd4f74aa40 1 bucket sync caught up with source: 2023-09-22T21:11:04.004 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.78.12, , ] 2023-09-22T21:11:04.004 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.78.12, , ] 2023-09-22T21:11:04.004 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:04.005+0000 7fbd4f74aa40 0 bucket checkpoint complete 2023-09-22T21:11:04.032 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-15 2023-09-22T21:11:04.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:11:04.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:11:04.032 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-15 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:11:04.124 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:11:04.124 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:11:04.166 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:04.169+0000 7f4f7ff4ca40 1 bucket sync caught up with source: 2023-09-22T21:11:04.166 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.80.6, , ] 2023-09-22T21:11:04.167 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000002.80.6, , ] 2023-09-22T21:11:04.167 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:04.169+0000 7f4f7ff4ca40 0 bucket checkpoint complete 2023-09-22T21:11:04.183 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-15 zones={a2, a1} 2023-09-22T21:11:04.183 DEBUG:boto:path=/ffzdqu-15/ 2023-09-22T21:11:04.183 DEBUG:boto:auth_path=/ffzdqu-15/ 2023-09-22T21:11:04.183 DEBUG:boto:Method: HEAD 2023-09-22T21:11:04.183 DEBUG:boto:Path: /ffzdqu-15/ 2023-09-22T21:11:04.183 DEBUG:boto:Data: 2023-09-22T21:11:04.183 DEBUG:boto:Headers: {} 2023-09-22T21:11:04.183 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:04.184 DEBUG:boto:Port: 8001 2023-09-22T21:11:04.184 DEBUG:boto:Params: {} 2023-09-22T21:11:04.184 DEBUG:boto:Token: None 2023-09-22T21:11:04.184 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:04 GMT /ffzdqu-15/ 2023-09-22T21:11:04.184 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F/EUtnZMJ1ZR5gcq9x7Pu6b/n5A= 2023-09-22T21:11:04.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F/EUtnZMJ1ZR5gcq9x7Pu6b/n5A=', 'Content-Length': '0'} 2023-09-22T21:11:04.187 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a5e203429b13de8-00650e02e8-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:11:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:04.187 DEBUG:boto:path=/ffzdqu-15/ 2023-09-22T21:11:04.187 DEBUG:boto:auth_path=/ffzdqu-15/ 2023-09-22T21:11:04.187 DEBUG:boto:Method: HEAD 2023-09-22T21:11:04.187 DEBUG:boto:Path: /ffzdqu-15/ 2023-09-22T21:11:04.187 DEBUG:boto:Data: 2023-09-22T21:11:04.187 DEBUG:boto:Headers: {} 2023-09-22T21:11:04.187 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:04.188 DEBUG:boto:Port: 8000 2023-09-22T21:11:04.188 DEBUG:boto:Params: {} 2023-09-22T21:11:04.188 DEBUG:boto:Token: None 2023-09-22T21:11:04.188 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:04 GMT /ffzdqu-15/ 2023-09-22T21:11:04.188 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F/EUtnZMJ1ZR5gcq9x7Pu6b/n5A= 2023-09-22T21:11:04.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F/EUtnZMJ1ZR5gcq9x7Pu6b/n5A=', 'Content-Length': '0'} 2023-09-22T21:11:04.190 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a37609c8a7232250-00650e02e8-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:11:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:04.190 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:11:04.191 DEBUG:boto:path=/ffzdqu-15/ 2023-09-22T21:11:04.191 DEBUG:boto:auth_path=/ffzdqu-15/ 2023-09-22T21:11:04.191 DEBUG:boto:path=/ffzdqu-15/?versions&max-keys=999 2023-09-22T21:11:04.191 DEBUG:boto:auth_path=/ffzdqu-15/?versions&max-keys=999 2023-09-22T21:11:04.191 DEBUG:boto:Method: GET 2023-09-22T21:11:04.191 DEBUG:boto:Path: /ffzdqu-15/?versions&max-keys=999 2023-09-22T21:11:04.191 DEBUG:boto:Data: 2023-09-22T21:11:04.191 DEBUG:boto:Headers: {} 2023-09-22T21:11:04.191 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:04.191 DEBUG:boto:Port: 8001 2023-09-22T21:11:04.192 DEBUG:boto:Params: {} 2023-09-22T21:11:04.192 DEBUG:boto:Token: None 2023-09-22T21:11:04.192 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:04 GMT /ffzdqu-15/?versions 2023-09-22T21:11:04.192 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uIfI2YcybqBhEwRkxGT8DxgU0do= 2023-09-22T21:11:04.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uIfI2YcybqBhEwRkxGT8DxgU0do=', 'Content-Length': '0'} 2023-09-22T21:11:04.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1e639583be8a758-00650e02e8-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:04.194 DEBUG:boto:b'ffzdqu-15999falseobjcbFI3snwFVP-tRtlvdB0Gq7QIRe9-3ktrue2023-09-22T21:10:48.878Zrgw-multisite-test-userMultisite Test UserNormalobjwdhULduirIbqsMn5d.L2a7FG4kVi1gZfalse2023-09-22T21:10:48.860Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:11:04.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:11:04.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:11:04.195 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:11:04.195 DEBUG:boto:path=/ffzdqu-15/ 2023-09-22T21:11:04.196 DEBUG:boto:auth_path=/ffzdqu-15/ 2023-09-22T21:11:04.196 DEBUG:boto:path=/ffzdqu-15/?versions&max-keys=999 2023-09-22T21:11:04.196 DEBUG:boto:auth_path=/ffzdqu-15/?versions&max-keys=999 2023-09-22T21:11:04.196 DEBUG:boto:Method: GET 2023-09-22T21:11:04.196 DEBUG:boto:Path: /ffzdqu-15/?versions&max-keys=999 2023-09-22T21:11:04.196 DEBUG:boto:Data: 2023-09-22T21:11:04.196 DEBUG:boto:Headers: {} 2023-09-22T21:11:04.196 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:04.196 DEBUG:boto:Port: 8000 2023-09-22T21:11:04.196 DEBUG:boto:Params: {} 2023-09-22T21:11:04.197 DEBUG:boto:Token: None 2023-09-22T21:11:04.197 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:04 GMT /ffzdqu-15/?versions 2023-09-22T21:11:04.197 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uIfI2YcybqBhEwRkxGT8DxgU0do= 2023-09-22T21:11:04.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uIfI2YcybqBhEwRkxGT8DxgU0do=', 'Content-Length': '0'} 2023-09-22T21:11:04.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b355eea8cffa3cfe-00650e02e8-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:04.199 DEBUG:boto:b'ffzdqu-15999falseobjcbFI3snwFVP-tRtlvdB0Gq7QIRe9-3ktrue2023-09-22T21:10:48.878Zrgw-multisite-test-userMultisite Test UserNormalobjwdhULduirIbqsMn5d.L2a7FG4kVi1gZfalse2023-09-22T21:10:48.860Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:11:04.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:11:04.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:11:04.200 DEBUG:boto:path=/ffzdqu-15/ 2023-09-22T21:11:04.200 DEBUG:boto:auth_path=/ffzdqu-15/ 2023-09-22T21:11:04.200 DEBUG:boto:path=/ffzdqu-15/?versions&max-keys=999 2023-09-22T21:11:04.200 DEBUG:boto:auth_path=/ffzdqu-15/?versions&max-keys=999 2023-09-22T21:11:04.201 DEBUG:boto:Method: GET 2023-09-22T21:11:04.201 DEBUG:boto:Path: /ffzdqu-15/?versions&max-keys=999 2023-09-22T21:11:04.201 DEBUG:boto:Data: 2023-09-22T21:11:04.201 DEBUG:boto:Headers: {} 2023-09-22T21:11:04.201 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:04.201 DEBUG:boto:Port: 8001 2023-09-22T21:11:04.201 DEBUG:boto:Params: {} 2023-09-22T21:11:04.201 DEBUG:boto:Token: None 2023-09-22T21:11:04.201 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:04 GMT /ffzdqu-15/?versions 2023-09-22T21:11:04.202 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uIfI2YcybqBhEwRkxGT8DxgU0do= 2023-09-22T21:11:04.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uIfI2YcybqBhEwRkxGT8DxgU0do=', 'Content-Length': '0'} 2023-09-22T21:11:04.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a266c53308a0cd83-00650e02e8-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:04.204 DEBUG:boto:b'ffzdqu-15999falseobjcbFI3snwFVP-tRtlvdB0Gq7QIRe9-3ktrue2023-09-22T21:10:48.878Zrgw-multisite-test-userMultisite Test UserNormalobjwdhULduirIbqsMn5d.L2a7FG4kVi1gZfalse2023-09-22T21:10:48.860Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:11:04.205 DEBUG:boto:path=/ffzdqu-15/ 2023-09-22T21:11:04.205 DEBUG:boto:auth_path=/ffzdqu-15/ 2023-09-22T21:11:04.205 DEBUG:boto:path=/ffzdqu-15/?versions&max-keys=999 2023-09-22T21:11:04.205 DEBUG:boto:auth_path=/ffzdqu-15/?versions&max-keys=999 2023-09-22T21:11:04.205 DEBUG:boto:Method: GET 2023-09-22T21:11:04.205 DEBUG:boto:Path: /ffzdqu-15/?versions&max-keys=999 2023-09-22T21:11:04.205 DEBUG:boto:Data: 2023-09-22T21:11:04.205 DEBUG:boto:Headers: {} 2023-09-22T21:11:04.206 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:04.206 DEBUG:boto:Port: 8000 2023-09-22T21:11:04.206 DEBUG:boto:Params: {} 2023-09-22T21:11:04.206 DEBUG:boto:Token: None 2023-09-22T21:11:04.206 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:04 GMT /ffzdqu-15/?versions 2023-09-22T21:11:04.206 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uIfI2YcybqBhEwRkxGT8DxgU0do= 2023-09-22T21:11:04.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uIfI2YcybqBhEwRkxGT8DxgU0do=', 'Content-Length': '0'} 2023-09-22T21:11:04.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4e5cfe8e3d51b18-00650e02e8-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:04.250 DEBUG:boto:b'ffzdqu-15999falseobjcbFI3snwFVP-tRtlvdB0Gq7QIRe9-3ktrue2023-09-22T21:10:48.878Zrgw-multisite-test-userMultisite Test UserNormalobjwdhULduirIbqsMn5d.L2a7FG4kVi1gZfalse2023-09-22T21:10:48.860Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:11:04.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:11:04.251 DEBUG:boto:path=/ffzdqu-15/obj 2023-09-22T21:11:04.251 DEBUG:boto:auth_path=/ffzdqu-15/obj 2023-09-22T21:11:04.251 DEBUG:boto:Method: HEAD 2023-09-22T21:11:04.251 DEBUG:boto:Path: /ffzdqu-15/obj 2023-09-22T21:11:04.251 DEBUG:boto:Data: 2023-09-22T21:11:04.251 DEBUG:boto:Headers: {} 2023-09-22T21:11:04.251 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:04.252 DEBUG:boto:Port: 8001 2023-09-22T21:11:04.252 DEBUG:boto:Params: {} 2023-09-22T21:11:04.252 DEBUG:boto:Token: None 2023-09-22T21:11:04.252 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:04 GMT /ffzdqu-15/obj 2023-09-22T21:11:04.252 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GBlrMRN9s83Dg/kLTVVIow/5lck= 2023-09-22T21:11:04.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GBlrMRN9s83Dg/kLTVVIow/5lck=', 'Content-Length': '0'} 2023-09-22T21:11:04.255 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000471a6e8a1435e16b-00650e02e8-1120-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:04.255 DEBUG:boto:path=/ffzdqu-15/obj 2023-09-22T21:11:04.255 DEBUG:boto:auth_path=/ffzdqu-15/obj 2023-09-22T21:11:04.255 DEBUG:boto:Method: HEAD 2023-09-22T21:11:04.255 DEBUG:boto:Path: /ffzdqu-15/obj 2023-09-22T21:11:04.255 DEBUG:boto:Data: 2023-09-22T21:11:04.256 DEBUG:boto:Headers: {} 2023-09-22T21:11:04.256 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:04.256 DEBUG:boto:Port: 8000 2023-09-22T21:11:04.256 DEBUG:boto:Params: {} 2023-09-22T21:11:04.256 DEBUG:boto:Token: None 2023-09-22T21:11:04.256 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:04 GMT /ffzdqu-15/obj 2023-09-22T21:11:04.256 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GBlrMRN9s83Dg/kLTVVIow/5lck= 2023-09-22T21:11:04.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GBlrMRN9s83Dg/kLTVVIow/5lck=', 'Content-Length': '0'} 2023-09-22T21:11:04.260 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000e05dfc6d735a76c8-00650e02e8-111c-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:04.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:11:04.260 DEBUG:boto:path=/ffzdqu-15/obj 2023-09-22T21:11:04.260 DEBUG:boto:auth_path=/ffzdqu-15/obj 2023-09-22T21:11:04.260 DEBUG:boto:path=/ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.260 DEBUG:boto:auth_path=/ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.260 DEBUG:boto:Method: GET 2023-09-22T21:11:04.261 DEBUG:boto:Path: /ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.261 DEBUG:boto:Data: 2023-09-22T21:11:04.261 DEBUG:boto:Headers: {} 2023-09-22T21:11:04.261 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:04.261 DEBUG:boto:Port: 8001 2023-09-22T21:11:04.261 DEBUG:boto:Params: {} 2023-09-22T21:11:04.261 DEBUG:boto:Token: None 2023-09-22T21:11:04.261 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:04 GMT /ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.261 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:73knXeMTo8VK2TvVoL1OtGQopjk= 2023-09-22T21:11:04.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:73knXeMTo8VK2TvVoL1OtGQopjk=', 'Content-Length': '0'} 2023-09-22T21:11:04.264 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:10:48 GMT'), ('x-amz-version-id', 'wdhULduirIbqsMn5d.L2a7FG4kVi1gZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000016f8b25ce7297f18-00650e02e8-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:04.264 DEBUG:boto:path=/ffzdqu-15/obj 2023-09-22T21:11:04.264 DEBUG:boto:auth_path=/ffzdqu-15/obj 2023-09-22T21:11:04.265 DEBUG:boto:path=/ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.265 DEBUG:boto:auth_path=/ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.265 DEBUG:boto:Method: GET 2023-09-22T21:11:04.265 DEBUG:boto:Path: /ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.265 DEBUG:boto:Data: 2023-09-22T21:11:04.265 DEBUG:boto:Headers: {} 2023-09-22T21:11:04.265 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:04.265 DEBUG:boto:Port: 8000 2023-09-22T21:11:04.265 DEBUG:boto:Params: {} 2023-09-22T21:11:04.265 DEBUG:boto:Token: None 2023-09-22T21:11:04.266 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:04 GMT /ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.266 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:73knXeMTo8VK2TvVoL1OtGQopjk= 2023-09-22T21:11:04.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:73knXeMTo8VK2TvVoL1OtGQopjk=', 'Content-Length': '0'} 2023-09-22T21:11:04.268 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:10:48 GMT'), ('x-amz-version-id', 'wdhULduirIbqsMn5d.L2a7FG4kVi1gZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000470609d59ec867a9-00650e02e8-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:04.268 DEBUG:boto:path=/ffzdqu-15/obj 2023-09-22T21:11:04.268 DEBUG:boto:auth_path=/ffzdqu-15/obj 2023-09-22T21:11:04.269 DEBUG:boto:path=/ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.269 DEBUG:boto:auth_path=/ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.269 DEBUG:boto:Method: HEAD 2023-09-22T21:11:04.269 DEBUG:boto:Path: /ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.269 DEBUG:boto:Data: 2023-09-22T21:11:04.269 DEBUG:boto:Headers: {} 2023-09-22T21:11:04.269 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:04.269 DEBUG:boto:Port: 8001 2023-09-22T21:11:04.269 DEBUG:boto:Params: {} 2023-09-22T21:11:04.270 DEBUG:boto:Token: None 2023-09-22T21:11:04.270 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:04 GMT /ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.270 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/UdW4UAmWPU7A+4pRXjUWkdmCQA= 2023-09-22T21:11:04.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/UdW4UAmWPU7A+4pRXjUWkdmCQA=', 'Content-Length': '0'} 2023-09-22T21:11:04.272 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:10:48 GMT'), ('x-amz-version-id', 'wdhULduirIbqsMn5d.L2a7FG4kVi1gZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eb6bee85c84506b4-00650e02e8-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:04.272 DEBUG:boto:path=/ffzdqu-15/obj 2023-09-22T21:11:04.272 DEBUG:boto:auth_path=/ffzdqu-15/obj 2023-09-22T21:11:04.272 DEBUG:boto:path=/ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.273 DEBUG:boto:auth_path=/ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.273 DEBUG:boto:Method: HEAD 2023-09-22T21:11:04.273 DEBUG:boto:Path: /ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.273 DEBUG:boto:Data: 2023-09-22T21:11:04.273 DEBUG:boto:Headers: {} 2023-09-22T21:11:04.273 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:04.273 DEBUG:boto:Port: 8000 2023-09-22T21:11:04.273 DEBUG:boto:Params: {} 2023-09-22T21:11:04.273 DEBUG:boto:Token: None 2023-09-22T21:11:04.274 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:04 GMT /ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.274 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/UdW4UAmWPU7A+4pRXjUWkdmCQA= 2023-09-22T21:11:04.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/UdW4UAmWPU7A+4pRXjUWkdmCQA=', 'Content-Length': '0'} 2023-09-22T21:11:04.276 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:10:48 GMT'), ('x-amz-version-id', 'wdhULduirIbqsMn5d.L2a7FG4kVi1gZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000038a44ccbf2cb545d-00650e02e8-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:04.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:11:04.276 DEBUG:boto:path=/ffzdqu-15/obj 2023-09-22T21:11:04.276 DEBUG:boto:auth_path=/ffzdqu-15/obj 2023-09-22T21:11:04.276 DEBUG:boto:path=/ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.277 DEBUG:boto:auth_path=/ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.277 DEBUG:boto:Method: GET 2023-09-22T21:11:04.277 DEBUG:boto:Path: /ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.277 DEBUG:boto:Data: 2023-09-22T21:11:04.277 DEBUG:boto:Headers: {} 2023-09-22T21:11:04.277 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:04.277 DEBUG:boto:Port: 8001 2023-09-22T21:11:04.277 DEBUG:boto:Params: {} 2023-09-22T21:11:04.277 DEBUG:boto:Token: None 2023-09-22T21:11:04.278 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:04 GMT /ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.278 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:73knXeMTo8VK2TvVoL1OtGQopjk= 2023-09-22T21:11:04.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:73knXeMTo8VK2TvVoL1OtGQopjk=', 'Content-Length': '0'} 2023-09-22T21:11:04.280 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:10:48 GMT'), ('x-amz-version-id', 'wdhULduirIbqsMn5d.L2a7FG4kVi1gZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000078ab580a0f9654e8-00650e02e8-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:04.280 DEBUG:boto:path=/ffzdqu-15/obj 2023-09-22T21:11:04.280 DEBUG:boto:auth_path=/ffzdqu-15/obj 2023-09-22T21:11:04.281 DEBUG:boto:path=/ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.281 DEBUG:boto:auth_path=/ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.281 DEBUG:boto:Method: GET 2023-09-22T21:11:04.281 DEBUG:boto:Path: /ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.281 DEBUG:boto:Data: 2023-09-22T21:11:04.281 DEBUG:boto:Headers: {} 2023-09-22T21:11:04.281 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:04.281 DEBUG:boto:Port: 8000 2023-09-22T21:11:04.281 DEBUG:boto:Params: {} 2023-09-22T21:11:04.282 DEBUG:boto:Token: None 2023-09-22T21:11:04.282 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:04 GMT /ffzdqu-15/obj?versionId=wdhULduirIbqsMn5d.L2a7FG4kVi1gZ 2023-09-22T21:11:04.282 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:73knXeMTo8VK2TvVoL1OtGQopjk= 2023-09-22T21:11:04.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:73knXeMTo8VK2TvVoL1OtGQopjk=', 'Content-Length': '0'} 2023-09-22T21:11:04.284 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:10:48 GMT'), ('x-amz-version-id', 'wdhULduirIbqsMn5d.L2a7FG4kVi1gZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000715d96d8293fdef6-00650e02e8-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:04.285 DEBUG:boto:path=/ffzdqu-15/ 2023-09-22T21:11:04.285 DEBUG:boto:auth_path=/ffzdqu-15/ 2023-09-22T21:11:04.285 DEBUG:boto:path=/ffzdqu-15/?prefix=obj 2023-09-22T21:11:04.285 DEBUG:boto:auth_path=/ffzdqu-15/?prefix=obj 2023-09-22T21:11:04.285 DEBUG:boto:Method: GET 2023-09-22T21:11:04.285 DEBUG:boto:Path: /ffzdqu-15/?prefix=obj 2023-09-22T21:11:04.285 DEBUG:boto:Data: 2023-09-22T21:11:04.285 DEBUG:boto:Headers: {} 2023-09-22T21:11:04.285 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:04.285 DEBUG:boto:Port: 8001 2023-09-22T21:11:04.286 DEBUG:boto:Params: {} 2023-09-22T21:11:04.286 DEBUG:boto:Token: None 2023-09-22T21:11:04.286 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:04 GMT /ffzdqu-15/ 2023-09-22T21:11:04.286 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YQuqwvx3/OuMTYXZ+PvEz3ks/JI= 2023-09-22T21:11:04.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YQuqwvx3/OuMTYXZ+PvEz3ks/JI=', 'Content-Length': '0'} 2023-09-22T21:11:04.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de31ddcbbeecf55f-00650e02e8-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:04.330 DEBUG:boto:b'ffzdqu-15obj1000false' 2023-09-22T21:11:04.331 DEBUG:boto:path=/ffzdqu-15/ 2023-09-22T21:11:04.331 DEBUG:boto:auth_path=/ffzdqu-15/ 2023-09-22T21:11:04.331 DEBUG:boto:path=/ffzdqu-15/?prefix=obj 2023-09-22T21:11:04.331 DEBUG:boto:auth_path=/ffzdqu-15/?prefix=obj 2023-09-22T21:11:04.331 DEBUG:boto:Method: GET 2023-09-22T21:11:04.331 DEBUG:boto:Path: /ffzdqu-15/?prefix=obj 2023-09-22T21:11:04.332 DEBUG:boto:Data: 2023-09-22T21:11:04.332 DEBUG:boto:Headers: {} 2023-09-22T21:11:04.332 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:04.332 DEBUG:boto:Port: 8000 2023-09-22T21:11:04.332 DEBUG:boto:Params: {} 2023-09-22T21:11:04.332 DEBUG:boto:Token: None 2023-09-22T21:11:04.332 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:04 GMT /ffzdqu-15/ 2023-09-22T21:11:04.333 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YQuqwvx3/OuMTYXZ+PvEz3ks/JI= 2023-09-22T21:11:04.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YQuqwvx3/OuMTYXZ+PvEz3ks/JI=', 'Content-Length': '0'} 2023-09-22T21:11:04.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff687b51cb5273d2-00650e02e8-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:04.338 DEBUG:boto:b'ffzdqu-15obj1000false' 2023-09-22T21:11:04.338 DEBUG:boto:path=/ffzdqu-15/obj 2023-09-22T21:11:04.338 DEBUG:boto:auth_path=/ffzdqu-15/obj 2023-09-22T21:11:04.338 DEBUG:boto:Method: HEAD 2023-09-22T21:11:04.339 DEBUG:boto:Path: /ffzdqu-15/obj 2023-09-22T21:11:04.339 DEBUG:boto:Data: 2023-09-22T21:11:04.339 DEBUG:boto:Headers: {} 2023-09-22T21:11:04.339 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:04.339 DEBUG:boto:Port: 8001 2023-09-22T21:11:04.339 DEBUG:boto:Params: {} 2023-09-22T21:11:04.340 DEBUG:boto:Token: None 2023-09-22T21:11:04.340 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:04 GMT /ffzdqu-15/obj 2023-09-22T21:11:04.340 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GBlrMRN9s83Dg/kLTVVIow/5lck= 2023-09-22T21:11:04.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GBlrMRN9s83Dg/kLTVVIow/5lck=', 'Content-Length': '0'} 2023-09-22T21:11:04.343 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000d48793f54feacff4-00650e02e8-1120-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:04.343 DEBUG:boto:path=/ffzdqu-15/obj 2023-09-22T21:11:04.344 DEBUG:boto:auth_path=/ffzdqu-15/obj 2023-09-22T21:11:04.344 DEBUG:boto:Method: HEAD 2023-09-22T21:11:04.344 DEBUG:boto:Path: /ffzdqu-15/obj 2023-09-22T21:11:04.344 DEBUG:boto:Data: 2023-09-22T21:11:04.344 DEBUG:boto:Headers: {} 2023-09-22T21:11:04.344 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:04.344 DEBUG:boto:Port: 8000 2023-09-22T21:11:04.344 DEBUG:boto:Params: {} 2023-09-22T21:11:04.344 DEBUG:boto:Token: None 2023-09-22T21:11:04.345 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:04 GMT /ffzdqu-15/obj 2023-09-22T21:11:04.345 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GBlrMRN9s83Dg/kLTVVIow/5lck= 2023-09-22T21:11:04.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GBlrMRN9s83Dg/kLTVVIow/5lck=', 'Content-Length': '0'} 2023-09-22T21:11:04.348 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f3bf74926e027601-00650e02e8-111c-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:04.349 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-15 zones={a2, a1} 2023-09-22T21:11:04.349 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-16 2023-09-22T21:11:04.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:11:04.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:11:04.349 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:11:04.399 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:11:04.399 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:11:04.440 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:04.445+0000 7f6bf4f7ca40 1 waiting to reach incremental sync.. 2023-09-22T21:11:09.442 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:09.445+0000 7f6bf4f7ca40 1 waiting to reach incremental sync.. 2023-09-22T21:11:14.443 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:14.445+0000 7f6bf4f7ca40 1 waiting to reach incremental sync.. 2023-09-22T21:11:19.443 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:19.445+0000 7f6bf4f7ca40 1 waiting to reach incremental sync.. 2023-09-22T21:11:24.446 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:24.449+0000 7f6bf4f7ca40 1 bucket sync caught up with source: 2023-09-22T21:11:24.447 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.19.6, , ] 2023-09-22T21:11:24.447 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000002.19.6, , ] 2023-09-22T21:11:24.447 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:24.449+0000 7f6bf4f7ca40 0 bucket checkpoint complete 2023-09-22T21:11:24.475 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-16 2023-09-22T21:11:24.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:11:24.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:11:24.475 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:11:24.565 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:11:24.565 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:11:24.609 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:24.613+0000 7f4e190aca40 1 bucket sync caught up with source: 2023-09-22T21:11:24.609 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.47.12, , ] 2023-09-22T21:11:24.610 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.47.12, , ] 2023-09-22T21:11:24.610 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:24.613+0000 7f4e190aca40 0 bucket checkpoint complete 2023-09-22T21:11:24.625 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-16 zones={a2, a1} 2023-09-22T21:11:24.625 DEBUG:boto:path=/ffzdqu-16/ 2023-09-22T21:11:24.625 DEBUG:boto:auth_path=/ffzdqu-16/ 2023-09-22T21:11:24.626 DEBUG:boto:Method: HEAD 2023-09-22T21:11:24.626 DEBUG:boto:Path: /ffzdqu-16/ 2023-09-22T21:11:24.626 DEBUG:boto:Data: 2023-09-22T21:11:24.626 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.626 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:24.626 DEBUG:boto:Port: 8001 2023-09-22T21:11:24.626 DEBUG:boto:Params: {} 2023-09-22T21:11:24.627 DEBUG:boto:Token: None 2023-09-22T21:11:24.627 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-16/ 2023-09-22T21:11:24.627 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TofuDu3xuhoaFc+pl4dXKQUJw5w= 2023-09-22T21:11:24.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TofuDu3xuhoaFc+pl4dXKQUJw5w=', 'Content-Length': '0'} 2023-09-22T21:11:24.629 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007a5d49111b5ee68-00650e02fc-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.630 DEBUG:boto:path=/ffzdqu-16/ 2023-09-22T21:11:24.630 DEBUG:boto:auth_path=/ffzdqu-16/ 2023-09-22T21:11:24.630 DEBUG:boto:Method: HEAD 2023-09-22T21:11:24.630 DEBUG:boto:Path: /ffzdqu-16/ 2023-09-22T21:11:24.630 DEBUG:boto:Data: 2023-09-22T21:11:24.630 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.630 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:24.630 DEBUG:boto:Port: 8000 2023-09-22T21:11:24.630 DEBUG:boto:Params: {} 2023-09-22T21:11:24.631 DEBUG:boto:Token: None 2023-09-22T21:11:24.631 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-16/ 2023-09-22T21:11:24.631 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TofuDu3xuhoaFc+pl4dXKQUJw5w= 2023-09-22T21:11:24.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TofuDu3xuhoaFc+pl4dXKQUJw5w=', 'Content-Length': '0'} 2023-09-22T21:11:24.633 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034290d20959625b1-00650e02fc-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.633 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:11:24.634 DEBUG:boto:path=/ffzdqu-16/ 2023-09-22T21:11:24.634 DEBUG:boto:auth_path=/ffzdqu-16/ 2023-09-22T21:11:24.634 DEBUG:boto:path=/ffzdqu-16/?versions&max-keys=999 2023-09-22T21:11:24.634 DEBUG:boto:auth_path=/ffzdqu-16/?versions&max-keys=999 2023-09-22T21:11:24.634 DEBUG:boto:Method: GET 2023-09-22T21:11:24.634 DEBUG:boto:Path: /ffzdqu-16/?versions&max-keys=999 2023-09-22T21:11:24.634 DEBUG:boto:Data: 2023-09-22T21:11:24.634 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.634 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:24.635 DEBUG:boto:Port: 8001 2023-09-22T21:11:24.635 DEBUG:boto:Params: {} 2023-09-22T21:11:24.635 DEBUG:boto:Token: None 2023-09-22T21:11:24.635 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-16/?versions 2023-09-22T21:11:24.635 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wcuR3Po4LcG8aM7sFmHSXB+yUME= 2023-09-22T21:11:24.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wcuR3Po4LcG8aM7sFmHSXB+yUME=', 'Content-Length': '0'} 2023-09-22T21:11:24.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d68cbfd60add626b-00650e02fc-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.637 DEBUG:boto:b'ffzdqu-16999falseobjJuvk1sb6t2yV.GGrRbOXcJf7E-yZOyztrue2023-09-22T21:10:48.910Zrgw-multisite-test-userMultisite Test UserNormalobj1JC9Icw-ucSKkSxsGX754pzA-PCw9bGfalse2023-09-22T21:10:48.891Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:11:24.638 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:11:24.638 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:11:24.638 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:11:24.639 DEBUG:boto:path=/ffzdqu-16/ 2023-09-22T21:11:24.639 DEBUG:boto:auth_path=/ffzdqu-16/ 2023-09-22T21:11:24.639 DEBUG:boto:path=/ffzdqu-16/?versions&max-keys=999 2023-09-22T21:11:24.639 DEBUG:boto:auth_path=/ffzdqu-16/?versions&max-keys=999 2023-09-22T21:11:24.639 DEBUG:boto:Method: GET 2023-09-22T21:11:24.639 DEBUG:boto:Path: /ffzdqu-16/?versions&max-keys=999 2023-09-22T21:11:24.639 DEBUG:boto:Data: 2023-09-22T21:11:24.639 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.639 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:24.639 DEBUG:boto:Port: 8000 2023-09-22T21:11:24.640 DEBUG:boto:Params: {} 2023-09-22T21:11:24.640 DEBUG:boto:Token: None 2023-09-22T21:11:24.640 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-16/?versions 2023-09-22T21:11:24.640 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wcuR3Po4LcG8aM7sFmHSXB+yUME= 2023-09-22T21:11:24.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wcuR3Po4LcG8aM7sFmHSXB+yUME=', 'Content-Length': '0'} 2023-09-22T21:11:24.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a8e6376ca106b5a-00650e02fc-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.642 DEBUG:boto:b'ffzdqu-16999falseobjJuvk1sb6t2yV.GGrRbOXcJf7E-yZOyztrue2023-09-22T21:10:48.910Zrgw-multisite-test-userMultisite Test UserNormalobj1JC9Icw-ucSKkSxsGX754pzA-PCw9bGfalse2023-09-22T21:10:48.891Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:11:24.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:11:24.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:11:24.643 DEBUG:boto:path=/ffzdqu-16/ 2023-09-22T21:11:24.643 DEBUG:boto:auth_path=/ffzdqu-16/ 2023-09-22T21:11:24.643 DEBUG:boto:path=/ffzdqu-16/?versions&max-keys=999 2023-09-22T21:11:24.643 DEBUG:boto:auth_path=/ffzdqu-16/?versions&max-keys=999 2023-09-22T21:11:24.643 DEBUG:boto:Method: GET 2023-09-22T21:11:24.644 DEBUG:boto:Path: /ffzdqu-16/?versions&max-keys=999 2023-09-22T21:11:24.644 DEBUG:boto:Data: 2023-09-22T21:11:24.644 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.644 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:24.644 DEBUG:boto:Port: 8001 2023-09-22T21:11:24.644 DEBUG:boto:Params: {} 2023-09-22T21:11:24.644 DEBUG:boto:Token: None 2023-09-22T21:11:24.644 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-16/?versions 2023-09-22T21:11:24.644 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wcuR3Po4LcG8aM7sFmHSXB+yUME= 2023-09-22T21:11:24.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wcuR3Po4LcG8aM7sFmHSXB+yUME=', 'Content-Length': '0'} 2023-09-22T21:11:24.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000357053c76d580847-00650e02fc-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.647 DEBUG:boto:b'ffzdqu-16999falseobjJuvk1sb6t2yV.GGrRbOXcJf7E-yZOyztrue2023-09-22T21:10:48.910Zrgw-multisite-test-userMultisite Test UserNormalobj1JC9Icw-ucSKkSxsGX754pzA-PCw9bGfalse2023-09-22T21:10:48.891Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:11:24.647 DEBUG:boto:path=/ffzdqu-16/ 2023-09-22T21:11:24.647 DEBUG:boto:auth_path=/ffzdqu-16/ 2023-09-22T21:11:24.648 DEBUG:boto:path=/ffzdqu-16/?versions&max-keys=999 2023-09-22T21:11:24.648 DEBUG:boto:auth_path=/ffzdqu-16/?versions&max-keys=999 2023-09-22T21:11:24.648 DEBUG:boto:Method: GET 2023-09-22T21:11:24.648 DEBUG:boto:Path: /ffzdqu-16/?versions&max-keys=999 2023-09-22T21:11:24.648 DEBUG:boto:Data: 2023-09-22T21:11:24.648 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.648 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:24.648 DEBUG:boto:Port: 8000 2023-09-22T21:11:24.648 DEBUG:boto:Params: {} 2023-09-22T21:11:24.649 DEBUG:boto:Token: None 2023-09-22T21:11:24.649 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-16/?versions 2023-09-22T21:11:24.649 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wcuR3Po4LcG8aM7sFmHSXB+yUME= 2023-09-22T21:11:24.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wcuR3Po4LcG8aM7sFmHSXB+yUME=', 'Content-Length': '0'} 2023-09-22T21:11:24.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdb5f92ff52067a8-00650e02fc-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.652 DEBUG:boto:b'ffzdqu-16999falseobjJuvk1sb6t2yV.GGrRbOXcJf7E-yZOyztrue2023-09-22T21:10:48.910Zrgw-multisite-test-userMultisite Test UserNormalobj1JC9Icw-ucSKkSxsGX754pzA-PCw9bGfalse2023-09-22T21:10:48.891Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:11:24.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:11:24.652 DEBUG:boto:path=/ffzdqu-16/obj 2023-09-22T21:11:24.652 DEBUG:boto:auth_path=/ffzdqu-16/obj 2023-09-22T21:11:24.652 DEBUG:boto:Method: HEAD 2023-09-22T21:11:24.653 DEBUG:boto:Path: /ffzdqu-16/obj 2023-09-22T21:11:24.653 DEBUG:boto:Data: 2023-09-22T21:11:24.653 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.653 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:24.653 DEBUG:boto:Port: 8001 2023-09-22T21:11:24.653 DEBUG:boto:Params: {} 2023-09-22T21:11:24.653 DEBUG:boto:Token: None 2023-09-22T21:11:24.653 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-16/obj 2023-09-22T21:11:24.653 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MSDJpTf6QX+s46MprBRAPH0PzGs= 2023-09-22T21:11:24.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MSDJpTf6QX+s46MprBRAPH0PzGs=', 'Content-Length': '0'} 2023-09-22T21:11:24.655 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000763b02195f236963-00650e02fc-1120-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.655 DEBUG:boto:path=/ffzdqu-16/obj 2023-09-22T21:11:24.656 DEBUG:boto:auth_path=/ffzdqu-16/obj 2023-09-22T21:11:24.656 DEBUG:boto:Method: HEAD 2023-09-22T21:11:24.656 DEBUG:boto:Path: /ffzdqu-16/obj 2023-09-22T21:11:24.656 DEBUG:boto:Data: 2023-09-22T21:11:24.656 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.656 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:24.656 DEBUG:boto:Port: 8000 2023-09-22T21:11:24.656 DEBUG:boto:Params: {} 2023-09-22T21:11:24.656 DEBUG:boto:Token: None 2023-09-22T21:11:24.656 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-16/obj 2023-09-22T21:11:24.657 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MSDJpTf6QX+s46MprBRAPH0PzGs= 2023-09-22T21:11:24.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MSDJpTf6QX+s46MprBRAPH0PzGs=', 'Content-Length': '0'} 2023-09-22T21:11:24.658 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000004a645920da9ef971-00650e02fc-111c-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:11:24.659 DEBUG:boto:path=/ffzdqu-16/obj 2023-09-22T21:11:24.659 DEBUG:boto:auth_path=/ffzdqu-16/obj 2023-09-22T21:11:24.659 DEBUG:boto:path=/ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.659 DEBUG:boto:auth_path=/ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.659 DEBUG:boto:Method: GET 2023-09-22T21:11:24.659 DEBUG:boto:Path: /ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.659 DEBUG:boto:Data: 2023-09-22T21:11:24.660 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.660 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:24.660 DEBUG:boto:Port: 8001 2023-09-22T21:11:24.660 DEBUG:boto:Params: {} 2023-09-22T21:11:24.660 DEBUG:boto:Token: None 2023-09-22T21:11:24.660 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.660 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2WT3mPIDDFvUNJdnHtv6ttFHA38= 2023-09-22T21:11:24.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2WT3mPIDDFvUNJdnHtv6ttFHA38=', 'Content-Length': '0'} 2023-09-22T21:11:24.663 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:10:48 GMT'), ('x-amz-version-id', '1JC9Icw-ucSKkSxsGX754pzA-PCw9bG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005c1de905eee4d2d1-00650e02fc-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.663 DEBUG:boto:path=/ffzdqu-16/obj 2023-09-22T21:11:24.663 DEBUG:boto:auth_path=/ffzdqu-16/obj 2023-09-22T21:11:24.663 DEBUG:boto:path=/ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.663 DEBUG:boto:auth_path=/ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.664 DEBUG:boto:Method: GET 2023-09-22T21:11:24.664 DEBUG:boto:Path: /ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.664 DEBUG:boto:Data: 2023-09-22T21:11:24.664 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.664 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:24.664 DEBUG:boto:Port: 8000 2023-09-22T21:11:24.664 DEBUG:boto:Params: {} 2023-09-22T21:11:24.664 DEBUG:boto:Token: None 2023-09-22T21:11:24.664 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.665 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2WT3mPIDDFvUNJdnHtv6ttFHA38= 2023-09-22T21:11:24.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2WT3mPIDDFvUNJdnHtv6ttFHA38=', 'Content-Length': '0'} 2023-09-22T21:11:24.666 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:10:48 GMT'), ('x-amz-version-id', '1JC9Icw-ucSKkSxsGX754pzA-PCw9bG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b800c08292bd3726-00650e02fc-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.666 DEBUG:boto:path=/ffzdqu-16/obj 2023-09-22T21:11:24.667 DEBUG:boto:auth_path=/ffzdqu-16/obj 2023-09-22T21:11:24.667 DEBUG:boto:path=/ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.667 DEBUG:boto:auth_path=/ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.667 DEBUG:boto:Method: HEAD 2023-09-22T21:11:24.667 DEBUG:boto:Path: /ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.667 DEBUG:boto:Data: 2023-09-22T21:11:24.667 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.667 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:24.667 DEBUG:boto:Port: 8001 2023-09-22T21:11:24.667 DEBUG:boto:Params: {} 2023-09-22T21:11:24.668 DEBUG:boto:Token: None 2023-09-22T21:11:24.668 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.668 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BM8nJ3qx4cRF4hEStxxsTC0djsA= 2023-09-22T21:11:24.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BM8nJ3qx4cRF4hEStxxsTC0djsA=', 'Content-Length': '0'} 2023-09-22T21:11:24.670 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:10:48 GMT'), ('x-amz-version-id', '1JC9Icw-ucSKkSxsGX754pzA-PCw9bG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007608147ec0d4ab06-00650e02fc-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.670 DEBUG:boto:path=/ffzdqu-16/obj 2023-09-22T21:11:24.670 DEBUG:boto:auth_path=/ffzdqu-16/obj 2023-09-22T21:11:24.670 DEBUG:boto:path=/ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.670 DEBUG:boto:auth_path=/ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.671 DEBUG:boto:Method: HEAD 2023-09-22T21:11:24.671 DEBUG:boto:Path: /ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.671 DEBUG:boto:Data: 2023-09-22T21:11:24.671 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.671 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:24.671 DEBUG:boto:Port: 8000 2023-09-22T21:11:24.671 DEBUG:boto:Params: {} 2023-09-22T21:11:24.672 DEBUG:boto:Token: None 2023-09-22T21:11:24.672 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.672 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BM8nJ3qx4cRF4hEStxxsTC0djsA= 2023-09-22T21:11:24.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BM8nJ3qx4cRF4hEStxxsTC0djsA=', 'Content-Length': '0'} 2023-09-22T21:11:24.673 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:10:48 GMT'), ('x-amz-version-id', '1JC9Icw-ucSKkSxsGX754pzA-PCw9bG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b3157ce7c32397f2-00650e02fc-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:11:24.674 DEBUG:boto:path=/ffzdqu-16/obj 2023-09-22T21:11:24.674 DEBUG:boto:auth_path=/ffzdqu-16/obj 2023-09-22T21:11:24.674 DEBUG:boto:path=/ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.674 DEBUG:boto:auth_path=/ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.674 DEBUG:boto:Method: GET 2023-09-22T21:11:24.675 DEBUG:boto:Path: /ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.675 DEBUG:boto:Data: 2023-09-22T21:11:24.675 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:24.675 DEBUG:boto:Port: 8001 2023-09-22T21:11:24.675 DEBUG:boto:Params: {} 2023-09-22T21:11:24.675 DEBUG:boto:Token: None 2023-09-22T21:11:24.675 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.675 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2WT3mPIDDFvUNJdnHtv6ttFHA38= 2023-09-22T21:11:24.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2WT3mPIDDFvUNJdnHtv6ttFHA38=', 'Content-Length': '0'} 2023-09-22T21:11:24.677 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:10:48 GMT'), ('x-amz-version-id', '1JC9Icw-ucSKkSxsGX754pzA-PCw9bG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bab9ad06fd3965be-00650e02fc-1120-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.678 DEBUG:boto:path=/ffzdqu-16/obj 2023-09-22T21:11:24.678 DEBUG:boto:auth_path=/ffzdqu-16/obj 2023-09-22T21:11:24.678 DEBUG:boto:path=/ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.678 DEBUG:boto:auth_path=/ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.678 DEBUG:boto:Method: GET 2023-09-22T21:11:24.678 DEBUG:boto:Path: /ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.678 DEBUG:boto:Data: 2023-09-22T21:11:24.678 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.678 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:24.678 DEBUG:boto:Port: 8000 2023-09-22T21:11:24.679 DEBUG:boto:Params: {} 2023-09-22T21:11:24.679 DEBUG:boto:Token: None 2023-09-22T21:11:24.679 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-16/obj?versionId=1JC9Icw-ucSKkSxsGX754pzA-PCw9bG 2023-09-22T21:11:24.679 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2WT3mPIDDFvUNJdnHtv6ttFHA38= 2023-09-22T21:11:24.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2WT3mPIDDFvUNJdnHtv6ttFHA38=', 'Content-Length': '0'} 2023-09-22T21:11:24.681 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:10:48 GMT'), ('x-amz-version-id', '1JC9Icw-ucSKkSxsGX754pzA-PCw9bG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000094c156a9ef3fb78-00650e02fc-111c-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.681 DEBUG:boto:path=/ffzdqu-16/ 2023-09-22T21:11:24.681 DEBUG:boto:auth_path=/ffzdqu-16/ 2023-09-22T21:11:24.681 DEBUG:boto:path=/ffzdqu-16/?prefix=obj 2023-09-22T21:11:24.681 DEBUG:boto:auth_path=/ffzdqu-16/?prefix=obj 2023-09-22T21:11:24.682 DEBUG:boto:Method: GET 2023-09-22T21:11:24.682 DEBUG:boto:Path: /ffzdqu-16/?prefix=obj 2023-09-22T21:11:24.682 DEBUG:boto:Data: 2023-09-22T21:11:24.682 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.682 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:24.682 DEBUG:boto:Port: 8001 2023-09-22T21:11:24.682 DEBUG:boto:Params: {} 2023-09-22T21:11:24.682 DEBUG:boto:Token: None 2023-09-22T21:11:24.682 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-16/ 2023-09-22T21:11:24.683 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+7SVWo6KQX4VU9usFjZmv7Sbe/A= 2023-09-22T21:11:24.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+7SVWo6KQX4VU9usFjZmv7Sbe/A=', 'Content-Length': '0'} 2023-09-22T21:11:24.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000506aafbe56c6a1a6-00650e02fc-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.726 DEBUG:boto:b'ffzdqu-16obj1000false' 2023-09-22T21:11:24.727 DEBUG:boto:path=/ffzdqu-16/ 2023-09-22T21:11:24.727 DEBUG:boto:auth_path=/ffzdqu-16/ 2023-09-22T21:11:24.727 DEBUG:boto:path=/ffzdqu-16/?prefix=obj 2023-09-22T21:11:24.727 DEBUG:boto:auth_path=/ffzdqu-16/?prefix=obj 2023-09-22T21:11:24.727 DEBUG:boto:Method: GET 2023-09-22T21:11:24.727 DEBUG:boto:Path: /ffzdqu-16/?prefix=obj 2023-09-22T21:11:24.727 DEBUG:boto:Data: 2023-09-22T21:11:24.728 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.728 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:24.728 DEBUG:boto:Port: 8000 2023-09-22T21:11:24.728 DEBUG:boto:Params: {} 2023-09-22T21:11:24.728 DEBUG:boto:Token: None 2023-09-22T21:11:24.728 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-16/ 2023-09-22T21:11:24.728 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+7SVWo6KQX4VU9usFjZmv7Sbe/A= 2023-09-22T21:11:24.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+7SVWo6KQX4VU9usFjZmv7Sbe/A=', 'Content-Length': '0'} 2023-09-22T21:11:24.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dfe80bec7740929-00650e02fc-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.733 DEBUG:boto:b'ffzdqu-16obj1000false' 2023-09-22T21:11:24.734 DEBUG:boto:path=/ffzdqu-16/obj 2023-09-22T21:11:24.734 DEBUG:boto:auth_path=/ffzdqu-16/obj 2023-09-22T21:11:24.734 DEBUG:boto:Method: HEAD 2023-09-22T21:11:24.734 DEBUG:boto:Path: /ffzdqu-16/obj 2023-09-22T21:11:24.734 DEBUG:boto:Data: 2023-09-22T21:11:24.734 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.734 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:24.734 DEBUG:boto:Port: 8001 2023-09-22T21:11:24.734 DEBUG:boto:Params: {} 2023-09-22T21:11:24.735 DEBUG:boto:Token: None 2023-09-22T21:11:24.735 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-16/obj 2023-09-22T21:11:24.735 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MSDJpTf6QX+s46MprBRAPH0PzGs= 2023-09-22T21:11:24.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MSDJpTf6QX+s46MprBRAPH0PzGs=', 'Content-Length': '0'} 2023-09-22T21:11:24.738 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000002556660c81328eb5-00650e02fc-1120-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.738 DEBUG:boto:path=/ffzdqu-16/obj 2023-09-22T21:11:24.739 DEBUG:boto:auth_path=/ffzdqu-16/obj 2023-09-22T21:11:24.739 DEBUG:boto:Method: HEAD 2023-09-22T21:11:24.739 DEBUG:boto:Path: /ffzdqu-16/obj 2023-09-22T21:11:24.739 DEBUG:boto:Data: 2023-09-22T21:11:24.739 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.739 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:24.739 DEBUG:boto:Port: 8000 2023-09-22T21:11:24.739 DEBUG:boto:Params: {} 2023-09-22T21:11:24.740 DEBUG:boto:Token: None 2023-09-22T21:11:24.740 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-16/obj 2023-09-22T21:11:24.740 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MSDJpTf6QX+s46MprBRAPH0PzGs= 2023-09-22T21:11:24.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MSDJpTf6QX+s46MprBRAPH0PzGs=', 'Content-Length': '0'} 2023-09-22T21:11:24.743 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000008b33da2dc5a4b3fa-00650e02fc-111c-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.743 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-16 zones={a2, a1} 2023-09-22T21:11:24.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2023-09-22T21:11:24.745 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-17 2023-09-22T21:11:24.745 DEBUG:boto:path=/ffzdqu-17/ 2023-09-22T21:11:24.745 DEBUG:boto:auth_path=/ffzdqu-17/ 2023-09-22T21:11:24.745 DEBUG:boto:Method: PUT 2023-09-22T21:11:24.747 DEBUG:boto:Path: /ffzdqu-17/ 2023-09-22T21:11:24.748 DEBUG:boto:Data: 2023-09-22T21:11:24.748 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.748 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:24.748 DEBUG:boto:Port: 8000 2023-09-22T21:11:24.748 DEBUG:boto:Params: {} 2023-09-22T21:11:24.748 DEBUG:boto:Token: None 2023-09-22T21:11:24.748 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-17/ 2023-09-22T21:11:24.749 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sntKYtRkoFKw7kvmLywMm0EmD6w= 2023-09-22T21:11:24.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sntKYtRkoFKw7kvmLywMm0EmD6w=', 'Content-Length': '0'} 2023-09-22T21:11:24.793 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008ec5c08776b15ba8-00650e02fc-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.793 INFO:rgw_multi.tests:create bucket zone=a2 name=ffzdqu-18 2023-09-22T21:11:24.793 DEBUG:boto:path=/ffzdqu-18/ 2023-09-22T21:11:24.793 DEBUG:boto:auth_path=/ffzdqu-18/ 2023-09-22T21:11:24.794 DEBUG:boto:Method: PUT 2023-09-22T21:11:24.794 DEBUG:boto:Path: /ffzdqu-18/ 2023-09-22T21:11:24.794 DEBUG:boto:Data: 2023-09-22T21:11:24.794 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.794 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:24.794 DEBUG:boto:Port: 8001 2023-09-22T21:11:24.794 DEBUG:boto:Params: {} 2023-09-22T21:11:24.794 DEBUG:boto:Token: None 2023-09-22T21:11:24.795 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-18/ 2023-09-22T21:11:24.795 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:k0SvQdWJ+BPKNfMahVbo7qrwcqM= 2023-09-22T21:11:24.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:k0SvQdWJ+BPKNfMahVbo7qrwcqM=', 'Content-Length': '0'} 2023-09-22T21:11:24.867 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001ae9ef5bfd1ba1d9-00650e02fc-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.867 DEBUG:boto:path=/ffzdqu-17/ 2023-09-22T21:11:24.867 DEBUG:boto:auth_path=/ffzdqu-17/ 2023-09-22T21:11:24.868 DEBUG:boto:path=/ffzdqu-17/?versioning 2023-09-22T21:11:24.868 DEBUG:boto:auth_path=/ffzdqu-17/?versioning 2023-09-22T21:11:24.868 DEBUG:boto:Method: PUT 2023-09-22T21:11:24.868 DEBUG:boto:Path: /ffzdqu-17/?versioning 2023-09-22T21:11:24.868 DEBUG:boto:Data: Enabled Disabled 2023-09-22T21:11:24.868 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.868 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:24.868 DEBUG:boto:Port: 8000 2023-09-22T21:11:24.868 DEBUG:boto:Params: {} 2023-09-22T21:11:24.868 DEBUG:boto:Token: None 2023-09-22T21:11:24.869 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-17/?versioning 2023-09-22T21:11:24.869 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xryDwULJV/A0eceNmIGT0tZSukQ= 2023-09-22T21:11:24.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xryDwULJV/A0eceNmIGT0tZSukQ=', 'Content-Length': '228'} 2023-09-22T21:11:24.881 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004f546af021211157-00650e02fc-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.881 DEBUG:boto:path=/ffzdqu-17/ 2023-09-22T21:11:24.882 DEBUG:boto:auth_path=/ffzdqu-17/ 2023-09-22T21:11:24.882 DEBUG:boto:path=/ffzdqu-17/?versioning 2023-09-22T21:11:24.882 DEBUG:boto:auth_path=/ffzdqu-17/?versioning 2023-09-22T21:11:24.882 DEBUG:boto:Method: PUT 2023-09-22T21:11:24.882 DEBUG:boto:Path: /ffzdqu-17/?versioning 2023-09-22T21:11:24.882 DEBUG:boto:Data: Suspended Disabled 2023-09-22T21:11:24.882 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.882 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:24.882 DEBUG:boto:Port: 8000 2023-09-22T21:11:24.882 DEBUG:boto:Params: {} 2023-09-22T21:11:24.883 DEBUG:boto:Token: None 2023-09-22T21:11:24.883 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-17/?versioning 2023-09-22T21:11:24.883 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xryDwULJV/A0eceNmIGT0tZSukQ= 2023-09-22T21:11:24.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xryDwULJV/A0eceNmIGT0tZSukQ=', 'Content-Length': '230'} 2023-09-22T21:11:24.893 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004cc1d5e7607ea0bf-00650e02fc-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.893 DEBUG:boto:path=/ffzdqu-18/ 2023-09-22T21:11:24.894 DEBUG:boto:auth_path=/ffzdqu-18/ 2023-09-22T21:11:24.894 DEBUG:boto:path=/ffzdqu-18/?versioning 2023-09-22T21:11:24.894 DEBUG:boto:auth_path=/ffzdqu-18/?versioning 2023-09-22T21:11:24.894 DEBUG:boto:Method: PUT 2023-09-22T21:11:24.894 DEBUG:boto:Path: /ffzdqu-18/?versioning 2023-09-22T21:11:24.894 DEBUG:boto:Data: Enabled Disabled 2023-09-22T21:11:24.894 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.894 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:24.894 DEBUG:boto:Port: 8001 2023-09-22T21:11:24.895 DEBUG:boto:Params: {} 2023-09-22T21:11:24.895 DEBUG:boto:Token: None 2023-09-22T21:11:24.895 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-18/?versioning 2023-09-22T21:11:24.895 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uCRoWdQRFFA+93/a1TPNBWSJUsg= 2023-09-22T21:11:24.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uCRoWdQRFFA+93/a1TPNBWSJUsg=', 'Content-Length': '228'} 2023-09-22T21:11:24.911 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005185a68e74de6ae8-00650e02fc-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.913 DEBUG:boto:path=/ffzdqu-18/ 2023-09-22T21:11:24.913 DEBUG:boto:auth_path=/ffzdqu-18/ 2023-09-22T21:11:24.913 DEBUG:boto:path=/ffzdqu-18/?versioning 2023-09-22T21:11:24.913 DEBUG:boto:auth_path=/ffzdqu-18/?versioning 2023-09-22T21:11:24.913 DEBUG:boto:Method: PUT 2023-09-22T21:11:24.913 DEBUG:boto:Path: /ffzdqu-18/?versioning 2023-09-22T21:11:24.913 DEBUG:boto:Data: Suspended Disabled 2023-09-22T21:11:24.913 DEBUG:boto:Headers: {} 2023-09-22T21:11:24.914 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:24.914 DEBUG:boto:Port: 8001 2023-09-22T21:11:24.914 DEBUG:boto:Params: {} 2023-09-22T21:11:24.914 DEBUG:boto:Token: None 2023-09-22T21:11:24.914 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:11:24 GMT /ffzdqu-18/?versioning 2023-09-22T21:11:24.914 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uCRoWdQRFFA+93/a1TPNBWSJUsg= 2023-09-22T21:11:24.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uCRoWdQRFFA+93/a1TPNBWSJUsg=', 'Content-Length': '230'} 2023-09-22T21:11:24.931 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000046fa5e21a28ba995-00650e02fc-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:11:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:24.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:11:24.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:11:24.932 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:11:24.972 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:11:24.972 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:11:25.027 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:11:25.046 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695417084.929246_101.1","last_update":"2023-09-22T21:11:24.929246Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:11:25.046 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417084.929246_101.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:11:25.046 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:11:25.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:11:25.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:11:25.047 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:11:25.124 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:11:25.124 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:11:25.162 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:25.165+0000 7f177fe60a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:11:25.163 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:25.165+0000 7f177fe60a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:11:25.179 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417084.929246_101.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:11:24.929246Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:11:25Z"} 2023-09-22T21:11:25.180 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417084.929246_101.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:11:24.929246Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:11:25Z"} 2023-09-22T21:11:25.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417084.929246_101.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:11:24.929246Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:11:25.180 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417084.929246_101.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:11:25.180 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417084.929246_101.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:11:25.180 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:11:25.181 DEBUG:boto:path=/ffzdqu-17/ 2023-09-22T21:11:25.181 DEBUG:boto:auth_path=/ffzdqu-17/ 2023-09-22T21:11:25.181 DEBUG:boto:Method: HEAD 2023-09-22T21:11:25.181 DEBUG:boto:Path: /ffzdqu-17/ 2023-09-22T21:11:25.181 DEBUG:boto:Data: 2023-09-22T21:11:25.181 DEBUG:boto:Headers: {} 2023-09-22T21:11:25.181 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:25.181 DEBUG:boto:Port: 8000 2023-09-22T21:11:25.181 DEBUG:boto:Params: {} 2023-09-22T21:11:25.182 DEBUG:boto:Token: None 2023-09-22T21:11:25.182 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:25 GMT /ffzdqu-17/ 2023-09-22T21:11:25.182 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jG6aTZ/CA+okZ+HCtLPTp0YvRLg= 2023-09-22T21:11:25.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jG6aTZ/CA+okZ+HCtLPTp0YvRLg=', 'Content-Length': '0'} 2023-09-22T21:11:25.184 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004aeb260bfc737d8-00650e02fd-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:11:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:25.185 DEBUG:boto:path=/ffzdqu-17/obj 2023-09-22T21:11:25.185 DEBUG:boto:auth_path=/ffzdqu-17/obj 2023-09-22T21:11:25.185 DEBUG:boto:Method: PUT 2023-09-22T21:11:25.185 DEBUG:boto:Path: /ffzdqu-17/obj 2023-09-22T21:11:25.185 DEBUG:boto:Data: 2023-09-22T21:11:25.185 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-22T21:11:25.185 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:25.186 DEBUG:boto:Port: 8000 2023-09-22T21:11:25.186 DEBUG:boto:Params: {} 2023-09-22T21:11:25.186 DEBUG:boto:Token: None 2023-09-22T21:11:25.186 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 22 Sep 2023 21:11:25 GMT /ffzdqu-17/obj 2023-09-22T21:11:25.186 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DP87CUhwm6o8/mpe7BCFnutNb4M= 2023-09-22T21:11:25.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:11:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DP87CUhwm6o8/mpe7BCFnutNb4M='} 2023-09-22T21:11:25.193 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000533fb25fa8b1ca17-00650e02fd-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:25.193 DEBUG:boto:path=/ffzdqu-17/ 2023-09-22T21:11:25.193 DEBUG:boto:auth_path=/ffzdqu-17/ 2023-09-22T21:11:25.193 DEBUG:boto:Method: HEAD 2023-09-22T21:11:25.193 DEBUG:boto:Path: /ffzdqu-17/ 2023-09-22T21:11:25.193 DEBUG:boto:Data: 2023-09-22T21:11:25.194 DEBUG:boto:Headers: {} 2023-09-22T21:11:25.194 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:25.194 DEBUG:boto:Port: 8000 2023-09-22T21:11:25.194 DEBUG:boto:Params: {} 2023-09-22T21:11:25.194 DEBUG:boto:Token: None 2023-09-22T21:11:25.194 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:25 GMT /ffzdqu-17/ 2023-09-22T21:11:25.194 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jG6aTZ/CA+okZ+HCtLPTp0YvRLg= 2023-09-22T21:11:25.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jG6aTZ/CA+okZ+HCtLPTp0YvRLg=', 'Content-Length': '0'} 2023-09-22T21:11:25.196 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e589a5d9c9abfaf3-00650e02fd-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:11:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:25.197 DEBUG:boto:path=/ffzdqu-17/obj 2023-09-22T21:11:25.197 DEBUG:boto:auth_path=/ffzdqu-17/obj 2023-09-22T21:11:25.197 DEBUG:boto:Method: DELETE 2023-09-22T21:11:25.197 DEBUG:boto:Path: /ffzdqu-17/obj 2023-09-22T21:11:25.197 DEBUG:boto:Data: 2023-09-22T21:11:25.197 DEBUG:boto:Headers: {} 2023-09-22T21:11:25.197 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:25.197 DEBUG:boto:Port: 8000 2023-09-22T21:11:25.198 DEBUG:boto:Params: {} 2023-09-22T21:11:25.198 DEBUG:boto:Token: None 2023-09-22T21:11:25.198 DEBUG:boto:StringToSign: DELETE Fri, 22 Sep 2023 21:11:25 GMT /ffzdqu-17/obj 2023-09-22T21:11:25.198 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YLKr8gMkHSczaHPbmQ97NADA8LA= 2023-09-22T21:11:25.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YLKr8gMkHSczaHPbmQ97NADA8LA=', 'Content-Length': '0'} 2023-09-22T21:11:25.209 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000b81f5a24c433c7bf-00650e02fd-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:25.209 DEBUG:boto:path=/ffzdqu-18/ 2023-09-22T21:11:25.209 DEBUG:boto:auth_path=/ffzdqu-18/ 2023-09-22T21:11:25.210 DEBUG:boto:Method: HEAD 2023-09-22T21:11:25.210 DEBUG:boto:Path: /ffzdqu-18/ 2023-09-22T21:11:25.210 DEBUG:boto:Data: 2023-09-22T21:11:25.210 DEBUG:boto:Headers: {} 2023-09-22T21:11:25.210 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:25.210 DEBUG:boto:Port: 8001 2023-09-22T21:11:25.210 DEBUG:boto:Params: {} 2023-09-22T21:11:25.210 DEBUG:boto:Token: None 2023-09-22T21:11:25.210 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:25 GMT /ffzdqu-18/ 2023-09-22T21:11:25.211 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vfq6C/IV3nGN8Khd9WhBPtXopnc= 2023-09-22T21:11:25.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vfq6C/IV3nGN8Khd9WhBPtXopnc=', 'Content-Length': '0'} 2023-09-22T21:11:25.213 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a872ba3c667984f6-00650e02fd-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:11:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:25.213 DEBUG:boto:path=/ffzdqu-18/obj 2023-09-22T21:11:25.214 DEBUG:boto:auth_path=/ffzdqu-18/obj 2023-09-22T21:11:25.214 DEBUG:boto:Method: PUT 2023-09-22T21:11:25.214 DEBUG:boto:Path: /ffzdqu-18/obj 2023-09-22T21:11:25.214 DEBUG:boto:Data: 2023-09-22T21:11:25.214 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-22T21:11:25.214 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:25.214 DEBUG:boto:Port: 8001 2023-09-22T21:11:25.214 DEBUG:boto:Params: {} 2023-09-22T21:11:25.214 DEBUG:boto:Token: None 2023-09-22T21:11:25.215 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 22 Sep 2023 21:11:25 GMT /ffzdqu-18/obj 2023-09-22T21:11:25.215 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ce3TQbZOvPyG0eiVNvWy6vqNYgA= 2023-09-22T21:11:25.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:11:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ce3TQbZOvPyG0eiVNvWy6vqNYgA='} 2023-09-22T21:11:25.222 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000088e0d3f49d75ba1e-00650e02fd-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:25.222 DEBUG:boto:path=/ffzdqu-18/ 2023-09-22T21:11:25.222 DEBUG:boto:auth_path=/ffzdqu-18/ 2023-09-22T21:11:25.222 DEBUG:boto:Method: HEAD 2023-09-22T21:11:25.222 DEBUG:boto:Path: /ffzdqu-18/ 2023-09-22T21:11:25.222 DEBUG:boto:Data: 2023-09-22T21:11:25.222 DEBUG:boto:Headers: {} 2023-09-22T21:11:25.222 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:25.223 DEBUG:boto:Port: 8001 2023-09-22T21:11:25.223 DEBUG:boto:Params: {} 2023-09-22T21:11:25.223 DEBUG:boto:Token: None 2023-09-22T21:11:25.223 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:25 GMT /ffzdqu-18/ 2023-09-22T21:11:25.223 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vfq6C/IV3nGN8Khd9WhBPtXopnc= 2023-09-22T21:11:25.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vfq6C/IV3nGN8Khd9WhBPtXopnc=', 'Content-Length': '0'} 2023-09-22T21:11:25.225 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f71caecb9bc078ee-00650e02fd-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:11:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:25.225 DEBUG:boto:path=/ffzdqu-18/obj 2023-09-22T21:11:25.225 DEBUG:boto:auth_path=/ffzdqu-18/obj 2023-09-22T21:11:25.225 DEBUG:boto:Method: DELETE 2023-09-22T21:11:25.225 DEBUG:boto:Path: /ffzdqu-18/obj 2023-09-22T21:11:25.226 DEBUG:boto:Data: 2023-09-22T21:11:25.226 DEBUG:boto:Headers: {} 2023-09-22T21:11:25.226 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:25.226 DEBUG:boto:Port: 8001 2023-09-22T21:11:25.226 DEBUG:boto:Params: {} 2023-09-22T21:11:25.226 DEBUG:boto:Token: None 2023-09-22T21:11:25.226 DEBUG:boto:StringToSign: DELETE Fri, 22 Sep 2023 21:11:25 GMT /ffzdqu-18/obj 2023-09-22T21:11:25.226 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vtWawYHgRuIMIgY7NaARzPmHOTE= 2023-09-22T21:11:25.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vtWawYHgRuIMIgY7NaARzPmHOTE=', 'Content-Length': '0'} 2023-09-22T21:11:25.238 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000002390c2fc5b9ff89e-00650e02fd-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:25.239 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-17 2023-09-22T21:11:25.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:11:25.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:11:25.239 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-17 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:11:25.280 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:11:25.280 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:11:25.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:25.321+0000 7fa845691a40 1 waiting to reach incremental sync.. 2023-09-22T21:11:30.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:30.325+0000 7fa845691a40 1 waiting to reach incremental sync.. 2023-09-22T21:11:35.322 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:35.325+0000 7fa845691a40 1 waiting to reach incremental sync.. 2023-09-22T21:11:40.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:40.325+0000 7fa845691a40 1 waiting to reach incremental sync.. 2023-09-22T21:11:45.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:45.329+0000 7fa845691a40 1 bucket sync caught up with source: 2023-09-22T21:11:45.327 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.10.7, , ] 2023-09-22T21:11:45.327 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.10.7, , ] 2023-09-22T21:11:45.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:45.329+0000 7fa845691a40 0 bucket checkpoint complete 2023-09-22T21:11:45.359 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-17 2023-09-22T21:11:45.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:11:45.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:11:45.360 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-17 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:11:45.450 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:11:45.450 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:11:45.493 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:45.497+0000 7f96dbdc2a40 1 bucket sync caught up with source: 2023-09-22T21:11:45.493 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.30.14, , ] 2023-09-22T21:11:45.493 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.30.14, , ] 2023-09-22T21:11:45.493 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:45.497+0000 7f96dbdc2a40 0 bucket checkpoint complete 2023-09-22T21:11:45.509 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-17 zones={a2, a1} 2023-09-22T21:11:45.510 DEBUG:boto:path=/ffzdqu-17/ 2023-09-22T21:11:45.510 DEBUG:boto:auth_path=/ffzdqu-17/ 2023-09-22T21:11:45.510 DEBUG:boto:Method: HEAD 2023-09-22T21:11:45.510 DEBUG:boto:Path: /ffzdqu-17/ 2023-09-22T21:11:45.510 DEBUG:boto:Data: 2023-09-22T21:11:45.510 DEBUG:boto:Headers: {} 2023-09-22T21:11:45.510 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:45.511 DEBUG:boto:Port: 8001 2023-09-22T21:11:45.511 DEBUG:boto:Params: {} 2023-09-22T21:11:45.511 DEBUG:boto:Token: None 2023-09-22T21:11:45.511 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:45 GMT /ffzdqu-17/ 2023-09-22T21:11:45.511 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5lEFdm6tEl2l6NtAfbCz0eE/1hQ= 2023-09-22T21:11:45.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5lEFdm6tEl2l6NtAfbCz0eE/1hQ=', 'Content-Length': '0'} 2023-09-22T21:11:45.514 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef610aecdaa8372f-00650e0311-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:45.514 DEBUG:boto:path=/ffzdqu-17/ 2023-09-22T21:11:45.514 DEBUG:boto:auth_path=/ffzdqu-17/ 2023-09-22T21:11:45.514 DEBUG:boto:Method: HEAD 2023-09-22T21:11:45.514 DEBUG:boto:Path: /ffzdqu-17/ 2023-09-22T21:11:45.515 DEBUG:boto:Data: 2023-09-22T21:11:45.515 DEBUG:boto:Headers: {} 2023-09-22T21:11:45.515 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:45.515 DEBUG:boto:Port: 8000 2023-09-22T21:11:45.515 DEBUG:boto:Params: {} 2023-09-22T21:11:45.515 DEBUG:boto:Token: None 2023-09-22T21:11:45.515 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:45 GMT /ffzdqu-17/ 2023-09-22T21:11:45.516 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5lEFdm6tEl2l6NtAfbCz0eE/1hQ= 2023-09-22T21:11:45.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5lEFdm6tEl2l6NtAfbCz0eE/1hQ=', 'Content-Length': '0'} 2023-09-22T21:11:45.518 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000abb5830a0488bc35-00650e0311-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:45.518 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:11:45.518 DEBUG:boto:path=/ffzdqu-17/ 2023-09-22T21:11:45.518 DEBUG:boto:auth_path=/ffzdqu-17/ 2023-09-22T21:11:45.518 DEBUG:boto:path=/ffzdqu-17/?versions&max-keys=999 2023-09-22T21:11:45.519 DEBUG:boto:auth_path=/ffzdqu-17/?versions&max-keys=999 2023-09-22T21:11:45.519 DEBUG:boto:Method: GET 2023-09-22T21:11:45.519 DEBUG:boto:Path: /ffzdqu-17/?versions&max-keys=999 2023-09-22T21:11:45.519 DEBUG:boto:Data: 2023-09-22T21:11:45.519 DEBUG:boto:Headers: {} 2023-09-22T21:11:45.519 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:45.519 DEBUG:boto:Port: 8001 2023-09-22T21:11:45.519 DEBUG:boto:Params: {} 2023-09-22T21:11:45.520 DEBUG:boto:Token: None 2023-09-22T21:11:45.520 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:45 GMT /ffzdqu-17/?versions 2023-09-22T21:11:45.520 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2hebgD3A3BoLwJ35gWZRYN7gbeU= 2023-09-22T21:11:45.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2hebgD3A3BoLwJ35gWZRYN7gbeU=', 'Content-Length': '0'} 2023-09-22T21:11:45.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f465acde2cdd5efe-00650e0311-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:45.523 DEBUG:boto:b'ffzdqu-17999falseobjnulltrue2023-09-22T21:11:25.205Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:11:45.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:11:45.523 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:11:45.523 DEBUG:boto:path=/ffzdqu-17/ 2023-09-22T21:11:45.524 DEBUG:boto:auth_path=/ffzdqu-17/ 2023-09-22T21:11:45.524 DEBUG:boto:path=/ffzdqu-17/?versions&max-keys=999 2023-09-22T21:11:45.524 DEBUG:boto:auth_path=/ffzdqu-17/?versions&max-keys=999 2023-09-22T21:11:45.524 DEBUG:boto:Method: GET 2023-09-22T21:11:45.524 DEBUG:boto:Path: /ffzdqu-17/?versions&max-keys=999 2023-09-22T21:11:45.524 DEBUG:boto:Data: 2023-09-22T21:11:45.524 DEBUG:boto:Headers: {} 2023-09-22T21:11:45.524 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:45.525 DEBUG:boto:Port: 8000 2023-09-22T21:11:45.525 DEBUG:boto:Params: {} 2023-09-22T21:11:45.525 DEBUG:boto:Token: None 2023-09-22T21:11:45.525 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:45 GMT /ffzdqu-17/?versions 2023-09-22T21:11:45.525 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2hebgD3A3BoLwJ35gWZRYN7gbeU= 2023-09-22T21:11:45.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2hebgD3A3BoLwJ35gWZRYN7gbeU=', 'Content-Length': '0'} 2023-09-22T21:11:45.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1d7a8a843a7bdd1-00650e0311-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:45.528 DEBUG:boto:b'ffzdqu-17999falseobjnulltrue2023-09-22T21:11:25.205Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:11:45.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:11:45.528 DEBUG:boto:path=/ffzdqu-17/ 2023-09-22T21:11:45.529 DEBUG:boto:auth_path=/ffzdqu-17/ 2023-09-22T21:11:45.529 DEBUG:boto:path=/ffzdqu-17/?versions&max-keys=999 2023-09-22T21:11:45.529 DEBUG:boto:auth_path=/ffzdqu-17/?versions&max-keys=999 2023-09-22T21:11:45.529 DEBUG:boto:Method: GET 2023-09-22T21:11:45.529 DEBUG:boto:Path: /ffzdqu-17/?versions&max-keys=999 2023-09-22T21:11:45.529 DEBUG:boto:Data: 2023-09-22T21:11:45.529 DEBUG:boto:Headers: {} 2023-09-22T21:11:45.529 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:45.529 DEBUG:boto:Port: 8001 2023-09-22T21:11:45.530 DEBUG:boto:Params: {} 2023-09-22T21:11:45.530 DEBUG:boto:Token: None 2023-09-22T21:11:45.530 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:45 GMT /ffzdqu-17/?versions 2023-09-22T21:11:45.530 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2hebgD3A3BoLwJ35gWZRYN7gbeU= 2023-09-22T21:11:45.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2hebgD3A3BoLwJ35gWZRYN7gbeU=', 'Content-Length': '0'} 2023-09-22T21:11:45.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ac6990fbdbb79ae-00650e0311-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:45.533 DEBUG:boto:b'ffzdqu-17999falseobjnulltrue2023-09-22T21:11:25.205Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:11:45.533 DEBUG:boto:path=/ffzdqu-17/ 2023-09-22T21:11:45.533 DEBUG:boto:auth_path=/ffzdqu-17/ 2023-09-22T21:11:45.533 DEBUG:boto:path=/ffzdqu-17/?versions&max-keys=999 2023-09-22T21:11:45.533 DEBUG:boto:auth_path=/ffzdqu-17/?versions&max-keys=999 2023-09-22T21:11:45.533 DEBUG:boto:Method: GET 2023-09-22T21:11:45.534 DEBUG:boto:Path: /ffzdqu-17/?versions&max-keys=999 2023-09-22T21:11:45.534 DEBUG:boto:Data: 2023-09-22T21:11:45.534 DEBUG:boto:Headers: {} 2023-09-22T21:11:45.534 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:45.534 DEBUG:boto:Port: 8000 2023-09-22T21:11:45.534 DEBUG:boto:Params: {} 2023-09-22T21:11:45.534 DEBUG:boto:Token: None 2023-09-22T21:11:45.535 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:11:45 GMT /ffzdqu-17/?versions 2023-09-22T21:11:45.535 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2hebgD3A3BoLwJ35gWZRYN7gbeU= 2023-09-22T21:11:45.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2hebgD3A3BoLwJ35gWZRYN7gbeU=', 'Content-Length': '0'} 2023-09-22T21:11:45.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001599c60aeecb37fc-00650e0311-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:45.537 DEBUG:boto:b'ffzdqu-17999falseobjnulltrue2023-09-22T21:11:25.205Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:11:45.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:11:45.538 DEBUG:boto:path=/ffzdqu-17/obj 2023-09-22T21:11:45.538 DEBUG:boto:auth_path=/ffzdqu-17/obj 2023-09-22T21:11:45.538 DEBUG:boto:Method: HEAD 2023-09-22T21:11:45.538 DEBUG:boto:Path: /ffzdqu-17/obj 2023-09-22T21:11:45.538 DEBUG:boto:Data: 2023-09-22T21:11:45.538 DEBUG:boto:Headers: {} 2023-09-22T21:11:45.538 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:11:45.539 DEBUG:boto:Port: 8001 2023-09-22T21:11:45.539 DEBUG:boto:Params: {} 2023-09-22T21:11:45.539 DEBUG:boto:Token: None 2023-09-22T21:11:45.539 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:45 GMT /ffzdqu-17/obj 2023-09-22T21:11:45.539 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GnJAG/jvVdnqoa2AxrmYzYhNEG8= 2023-09-22T21:11:45.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GnJAG/jvVdnqoa2AxrmYzYhNEG8=', 'Content-Length': '0'} 2023-09-22T21:11:45.541 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000957efc51b1d54a97-00650e0311-1120-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:45.542 DEBUG:boto:path=/ffzdqu-17/obj 2023-09-22T21:11:45.542 DEBUG:boto:auth_path=/ffzdqu-17/obj 2023-09-22T21:11:45.542 DEBUG:boto:Method: HEAD 2023-09-22T21:11:45.542 DEBUG:boto:Path: /ffzdqu-17/obj 2023-09-22T21:11:45.542 DEBUG:boto:Data: 2023-09-22T21:11:45.542 DEBUG:boto:Headers: {} 2023-09-22T21:11:45.542 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:11:45.542 DEBUG:boto:Port: 8000 2023-09-22T21:11:45.543 DEBUG:boto:Params: {} 2023-09-22T21:11:45.543 DEBUG:boto:Token: None 2023-09-22T21:11:45.543 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:11:45 GMT /ffzdqu-17/obj 2023-09-22T21:11:45.543 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GnJAG/jvVdnqoa2AxrmYzYhNEG8= 2023-09-22T21:11:45.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:11:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GnJAG/jvVdnqoa2AxrmYzYhNEG8=', 'Content-Length': '0'} 2023-09-22T21:11:45.545 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000003ad9dbf8fe8ac0a3-00650e0311-111c-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:11:45.545 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-17 zones={a2, a1} 2023-09-22T21:11:45.545 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-18 2023-09-22T21:11:45.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:11:45.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:11:45.545 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-18 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:11:45.587 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:11:45.588 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:11:45.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:45.629+0000 7fc6ad68ba40 1 waiting to reach incremental sync.. 2023-09-22T21:11:50.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:50.633+0000 7fc6ad68ba40 1 waiting to reach incremental sync.. 2023-09-22T21:11:55.629 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:11:55.633+0000 7fc6ad68ba40 1 waiting to reach incremental sync.. 2023-09-22T21:12:00.630 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:12:00.633+0000 7fc6ad68ba40 1 waiting to reach incremental sync.. 2023-09-22T21:12:05.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:12:05.637+0000 7fc6ad68ba40 1 bucket sync caught up with source: 2023-09-22T21:12:05.634 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.15.14, , ] 2023-09-22T21:12:05.634 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.15.14, , ] 2023-09-22T21:12:05.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:12:05.637+0000 7fc6ad68ba40 0 bucket checkpoint complete 2023-09-22T21:12:05.666 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-18 2023-09-22T21:12:05.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:12:05.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:12:05.669 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-18 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:12:05.756 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:05.757 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:05.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:12:05.804+0000 7f0dd7c14a40 1 bucket sync caught up with source: 2023-09-22T21:12:05.800 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.41.7, , ] 2023-09-22T21:12:05.800 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.41.7, , ] 2023-09-22T21:12:05.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:12:05.804+0000 7f0dd7c14a40 0 bucket checkpoint complete 2023-09-22T21:12:05.816 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-18 zones={a2, a1} 2023-09-22T21:12:05.817 DEBUG:boto:path=/ffzdqu-18/ 2023-09-22T21:12:05.817 DEBUG:boto:auth_path=/ffzdqu-18/ 2023-09-22T21:12:05.817 DEBUG:boto:Method: HEAD 2023-09-22T21:12:05.817 DEBUG:boto:Path: /ffzdqu-18/ 2023-09-22T21:12:05.817 DEBUG:boto:Data: 2023-09-22T21:12:05.817 DEBUG:boto:Headers: {} 2023-09-22T21:12:05.817 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:12:05.817 DEBUG:boto:Port: 8001 2023-09-22T21:12:05.817 DEBUG:boto:Params: {} 2023-09-22T21:12:05.818 DEBUG:boto:Token: None 2023-09-22T21:12:05.818 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:12:05 GMT /ffzdqu-18/ 2023-09-22T21:12:05.818 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PBpG70s9nJbtwiGHRW0AK2gN7+I= 2023-09-22T21:12:05.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PBpG70s9nJbtwiGHRW0AK2gN7+I=', 'Content-Length': '0'} 2023-09-22T21:12:05.820 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fe444ffa74c6b5e-00650e0325-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:05.821 DEBUG:boto:path=/ffzdqu-18/ 2023-09-22T21:12:05.821 DEBUG:boto:auth_path=/ffzdqu-18/ 2023-09-22T21:12:05.821 DEBUG:boto:Method: HEAD 2023-09-22T21:12:05.821 DEBUG:boto:Path: /ffzdqu-18/ 2023-09-22T21:12:05.821 DEBUG:boto:Data: 2023-09-22T21:12:05.821 DEBUG:boto:Headers: {} 2023-09-22T21:12:05.821 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:05.821 DEBUG:boto:Port: 8000 2023-09-22T21:12:05.821 DEBUG:boto:Params: {} 2023-09-22T21:12:05.822 DEBUG:boto:Token: None 2023-09-22T21:12:05.822 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:12:05 GMT /ffzdqu-18/ 2023-09-22T21:12:05.822 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PBpG70s9nJbtwiGHRW0AK2gN7+I= 2023-09-22T21:12:05.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PBpG70s9nJbtwiGHRW0AK2gN7+I=', 'Content-Length': '0'} 2023-09-22T21:12:05.824 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f6ec21e635b72dc-00650e0325-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:05.824 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:12:05.824 DEBUG:boto:path=/ffzdqu-18/ 2023-09-22T21:12:05.824 DEBUG:boto:auth_path=/ffzdqu-18/ 2023-09-22T21:12:05.825 DEBUG:boto:path=/ffzdqu-18/?versions&max-keys=999 2023-09-22T21:12:05.825 DEBUG:boto:auth_path=/ffzdqu-18/?versions&max-keys=999 2023-09-22T21:12:05.825 DEBUG:boto:Method: GET 2023-09-22T21:12:05.825 DEBUG:boto:Path: /ffzdqu-18/?versions&max-keys=999 2023-09-22T21:12:05.825 DEBUG:boto:Data: 2023-09-22T21:12:05.825 DEBUG:boto:Headers: {} 2023-09-22T21:12:05.825 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:12:05.825 DEBUG:boto:Port: 8001 2023-09-22T21:12:05.825 DEBUG:boto:Params: {} 2023-09-22T21:12:05.826 DEBUG:boto:Token: None 2023-09-22T21:12:05.826 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:12:05 GMT /ffzdqu-18/?versions 2023-09-22T21:12:05.826 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1Y0MX0e5RVfbbSh9UUE/j8Myazg= 2023-09-22T21:12:05.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1Y0MX0e5RVfbbSh9UUE/j8Myazg=', 'Content-Length': '0'} 2023-09-22T21:12:05.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a483328830cae93f-00650e0325-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:05.828 DEBUG:boto:b'ffzdqu-18999falseobjnulltrue2023-09-22T21:11:25.233Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:12:05.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:12:05.829 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:12:05.829 DEBUG:boto:path=/ffzdqu-18/ 2023-09-22T21:12:05.829 DEBUG:boto:auth_path=/ffzdqu-18/ 2023-09-22T21:12:05.829 DEBUG:boto:path=/ffzdqu-18/?versions&max-keys=999 2023-09-22T21:12:05.829 DEBUG:boto:auth_path=/ffzdqu-18/?versions&max-keys=999 2023-09-22T21:12:05.830 DEBUG:boto:Method: GET 2023-09-22T21:12:05.830 DEBUG:boto:Path: /ffzdqu-18/?versions&max-keys=999 2023-09-22T21:12:05.830 DEBUG:boto:Data: 2023-09-22T21:12:05.830 DEBUG:boto:Headers: {} 2023-09-22T21:12:05.830 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:05.830 DEBUG:boto:Port: 8000 2023-09-22T21:12:05.830 DEBUG:boto:Params: {} 2023-09-22T21:12:05.830 DEBUG:boto:Token: None 2023-09-22T21:12:05.830 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:12:05 GMT /ffzdqu-18/?versions 2023-09-22T21:12:05.831 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1Y0MX0e5RVfbbSh9UUE/j8Myazg= 2023-09-22T21:12:05.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1Y0MX0e5RVfbbSh9UUE/j8Myazg=', 'Content-Length': '0'} 2023-09-22T21:12:05.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f7e64b10810b43c-00650e0325-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:05.833 DEBUG:boto:b'ffzdqu-18999falseobjnulltrue2023-09-22T21:11:25.233Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:12:05.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-22T21:12:05.833 DEBUG:boto:path=/ffzdqu-18/ 2023-09-22T21:12:05.834 DEBUG:boto:auth_path=/ffzdqu-18/ 2023-09-22T21:12:05.834 DEBUG:boto:path=/ffzdqu-18/?versions&max-keys=999 2023-09-22T21:12:05.834 DEBUG:boto:auth_path=/ffzdqu-18/?versions&max-keys=999 2023-09-22T21:12:05.834 DEBUG:boto:Method: GET 2023-09-22T21:12:05.834 DEBUG:boto:Path: /ffzdqu-18/?versions&max-keys=999 2023-09-22T21:12:05.834 DEBUG:boto:Data: 2023-09-22T21:12:05.834 DEBUG:boto:Headers: {} 2023-09-22T21:12:05.834 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:12:05.834 DEBUG:boto:Port: 8001 2023-09-22T21:12:05.835 DEBUG:boto:Params: {} 2023-09-22T21:12:05.835 DEBUG:boto:Token: None 2023-09-22T21:12:05.835 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:12:05 GMT /ffzdqu-18/?versions 2023-09-22T21:12:05.835 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1Y0MX0e5RVfbbSh9UUE/j8Myazg= 2023-09-22T21:12:05.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1Y0MX0e5RVfbbSh9UUE/j8Myazg=', 'Content-Length': '0'} 2023-09-22T21:12:05.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007203e91375681754-00650e0325-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:05.837 DEBUG:boto:b'ffzdqu-18999falseobjnulltrue2023-09-22T21:11:25.233Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:12:05.838 DEBUG:boto:path=/ffzdqu-18/ 2023-09-22T21:12:05.838 DEBUG:boto:auth_path=/ffzdqu-18/ 2023-09-22T21:12:05.838 DEBUG:boto:path=/ffzdqu-18/?versions&max-keys=999 2023-09-22T21:12:05.838 DEBUG:boto:auth_path=/ffzdqu-18/?versions&max-keys=999 2023-09-22T21:12:05.838 DEBUG:boto:Method: GET 2023-09-22T21:12:05.838 DEBUG:boto:Path: /ffzdqu-18/?versions&max-keys=999 2023-09-22T21:12:05.838 DEBUG:boto:Data: 2023-09-22T21:12:05.838 DEBUG:boto:Headers: {} 2023-09-22T21:12:05.839 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:05.839 DEBUG:boto:Port: 8000 2023-09-22T21:12:05.839 DEBUG:boto:Params: {} 2023-09-22T21:12:05.839 DEBUG:boto:Token: None 2023-09-22T21:12:05.839 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:12:05 GMT /ffzdqu-18/?versions 2023-09-22T21:12:05.839 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1Y0MX0e5RVfbbSh9UUE/j8Myazg= 2023-09-22T21:12:05.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1Y0MX0e5RVfbbSh9UUE/j8Myazg=', 'Content-Length': '0'} 2023-09-22T21:12:05.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000189080a98b9c4857-00650e0325-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:05.842 DEBUG:boto:b'ffzdqu-18999falseobjnulltrue2023-09-22T21:11:25.233Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:12:05.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-22T21:12:05.842 DEBUG:boto:path=/ffzdqu-18/obj 2023-09-22T21:12:05.842 DEBUG:boto:auth_path=/ffzdqu-18/obj 2023-09-22T21:12:05.842 DEBUG:boto:Method: HEAD 2023-09-22T21:12:05.842 DEBUG:boto:Path: /ffzdqu-18/obj 2023-09-22T21:12:05.842 DEBUG:boto:Data: 2023-09-22T21:12:05.843 DEBUG:boto:Headers: {} 2023-09-22T21:12:05.843 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:12:05.843 DEBUG:boto:Port: 8001 2023-09-22T21:12:05.843 DEBUG:boto:Params: {} 2023-09-22T21:12:05.843 DEBUG:boto:Token: None 2023-09-22T21:12:05.843 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:12:05 GMT /ffzdqu-18/obj 2023-09-22T21:12:05.843 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xp1tPon0WMIKToSDX8SPQiscxpE= 2023-09-22T21:12:05.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xp1tPon0WMIKToSDX8SPQiscxpE=', 'Content-Length': '0'} 2023-09-22T21:12:05.845 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c5ddf5e447001b5a-00650e0325-1120-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:05.845 DEBUG:boto:path=/ffzdqu-18/obj 2023-09-22T21:12:05.845 DEBUG:boto:auth_path=/ffzdqu-18/obj 2023-09-22T21:12:05.845 DEBUG:boto:Method: HEAD 2023-09-22T21:12:05.846 DEBUG:boto:Path: /ffzdqu-18/obj 2023-09-22T21:12:05.846 DEBUG:boto:Data: 2023-09-22T21:12:05.846 DEBUG:boto:Headers: {} 2023-09-22T21:12:05.846 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:05.846 DEBUG:boto:Port: 8000 2023-09-22T21:12:05.846 DEBUG:boto:Params: {} 2023-09-22T21:12:05.846 DEBUG:boto:Token: None 2023-09-22T21:12:05.846 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:12:05 GMT /ffzdqu-18/obj 2023-09-22T21:12:05.846 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xp1tPon0WMIKToSDX8SPQiscxpE= 2023-09-22T21:12:05.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xp1tPon0WMIKToSDX8SPQiscxpE=', 'Content-Length': '0'} 2023-09-22T21:12:05.848 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000007558bbb853b76d71-00650e0325-111c-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:05.848 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-18 zones={a2, a1} 2023-09-22T21:12:05.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2023-09-22T21:12:05.849 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-19 2023-09-22T21:12:05.849 DEBUG:boto:path=/ffzdqu-19/ 2023-09-22T21:12:05.849 DEBUG:boto:auth_path=/ffzdqu-19/ 2023-09-22T21:12:05.849 DEBUG:boto:Method: PUT 2023-09-22T21:12:05.849 DEBUG:boto:Path: /ffzdqu-19/ 2023-09-22T21:12:05.850 DEBUG:boto:Data: 2023-09-22T21:12:05.850 DEBUG:boto:Headers: {} 2023-09-22T21:12:05.850 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:05.850 DEBUG:boto:Port: 8000 2023-09-22T21:12:05.850 DEBUG:boto:Params: {} 2023-09-22T21:12:05.850 DEBUG:boto:Token: None 2023-09-22T21:12:05.850 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:05 GMT /ffzdqu-19/ 2023-09-22T21:12:05.850 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Kzvbt2PQ2mJz0wWgDvlbunHR1dI= 2023-09-22T21:12:05.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Kzvbt2PQ2mJz0wWgDvlbunHR1dI=', 'Content-Length': '0'} 2023-09-22T21:12:05.869 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000032b0f5d8b4f5915f-00650e0325-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:05.870 INFO:rgw_multi.tests:create bucket zone=a2 name=ffzdqu-20 2023-09-22T21:12:05.870 DEBUG:boto:path=/ffzdqu-20/ 2023-09-22T21:12:05.870 DEBUG:boto:auth_path=/ffzdqu-20/ 2023-09-22T21:12:05.870 DEBUG:boto:Method: PUT 2023-09-22T21:12:05.870 DEBUG:boto:Path: /ffzdqu-20/ 2023-09-22T21:12:05.870 DEBUG:boto:Data: 2023-09-22T21:12:05.870 DEBUG:boto:Headers: {} 2023-09-22T21:12:05.870 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:12:05.871 DEBUG:boto:Port: 8001 2023-09-22T21:12:05.871 DEBUG:boto:Params: {} 2023-09-22T21:12:05.871 DEBUG:boto:Token: None 2023-09-22T21:12:05.871 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:05 GMT /ffzdqu-20/ 2023-09-22T21:12:05.871 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:U0+brlB4YTcahDiRLvtaDJ8d4BQ= 2023-09-22T21:12:05.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:U0+brlB4YTcahDiRLvtaDJ8d4BQ=', 'Content-Length': '0'} 2023-09-22T21:12:05.912 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dcae2d0ceb6a1f9b-00650e0325-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:05.912 DEBUG:boto:Using access key provided by client. 2023-09-22T21:12:05.912 DEBUG:boto:Using secret key provided by client. 2023-09-22T21:12:05.913 DEBUG:boto:Using access key provided by client. 2023-09-22T21:12:05.913 DEBUG:boto:Using secret key provided by client. 2023-09-22T21:12:05.913 DEBUG:boto:Using access key provided by client. 2023-09-22T21:12:05.913 DEBUG:boto:Using secret key provided by client. 2023-09-22T21:12:05.914 DEBUG:boto:Using access key provided by client. 2023-09-22T21:12:05.914 DEBUG:boto:Using secret key provided by client. 2023-09-22T21:12:05.914 INFO:rgw_multi.tests:create bucket zone=b1 name=ffzdqu-21 2023-09-22T21:12:05.914 DEBUG:boto:path=/ffzdqu-21/ 2023-09-22T21:12:05.914 DEBUG:boto:auth_path=/ffzdqu-21/ 2023-09-22T21:12:05.915 DEBUG:boto:Method: PUT 2023-09-22T21:12:05.915 DEBUG:boto:Path: /ffzdqu-21/ 2023-09-22T21:12:05.915 DEBUG:boto:Data: 2023-09-22T21:12:05.915 DEBUG:boto:Headers: {} 2023-09-22T21:12:05.915 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8000 2023-09-22T21:12:05.915 DEBUG:boto:Port: 8000 2023-09-22T21:12:05.915 DEBUG:boto:Params: {} 2023-09-22T21:12:05.915 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:12:05.915 DEBUG:boto:Token: None 2023-09-22T21:12:05.916 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:05 GMT /ffzdqu-21/ 2023-09-22T21:12:05.916 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AFzZ6GE09jMVbVsng8Px9M4/xXM= 2023-09-22T21:12:05.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AFzZ6GE09jMVbVsng8Px9M4/xXM=', 'Content-Length': '0'} 2023-09-22T21:12:05.968 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003989a0cb8f115ba3-00650e0325-1109-b1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:05.969 INFO:rgw_multi.tests:create bucket zone=b2 name=ffzdqu-22 2023-09-22T21:12:05.969 DEBUG:boto:path=/ffzdqu-22/ 2023-09-22T21:12:05.969 DEBUG:boto:auth_path=/ffzdqu-22/ 2023-09-22T21:12:05.969 DEBUG:boto:Method: PUT 2023-09-22T21:12:05.969 DEBUG:boto:Path: /ffzdqu-22/ 2023-09-22T21:12:05.969 DEBUG:boto:Data: 2023-09-22T21:12:05.969 DEBUG:boto:Headers: {} 2023-09-22T21:12:05.969 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8001 2023-09-22T21:12:05.970 DEBUG:boto:Port: 8001 2023-09-22T21:12:05.970 DEBUG:boto:Params: {} 2023-09-22T21:12:05.970 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:12:05.970 DEBUG:boto:Token: None 2023-09-22T21:12:05.970 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:05 GMT /ffzdqu-22/ 2023-09-22T21:12:05.970 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lHparGCbdpuJ+LLAqUAIuE3yhHE= 2023-09-22T21:12:05.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lHparGCbdpuJ+LLAqUAIuE3yhHE=', 'Content-Length': '0'} 2023-09-22T21:12:06.039 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000061ac3bcc5e645965-00650e0325-1110-b2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.039 DEBUG:boto:path=/ffzdqu-19/ 2023-09-22T21:12:06.040 DEBUG:boto:auth_path=/ffzdqu-19/ 2023-09-22T21:12:06.040 DEBUG:boto:path=/ffzdqu-19/?versioning 2023-09-22T21:12:06.040 DEBUG:boto:auth_path=/ffzdqu-19/?versioning 2023-09-22T21:12:06.040 DEBUG:boto:Method: PUT 2023-09-22T21:12:06.040 DEBUG:boto:Path: /ffzdqu-19/?versioning 2023-09-22T21:12:06.040 DEBUG:boto:Data: Enabled Disabled 2023-09-22T21:12:06.040 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.040 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:06.040 DEBUG:boto:Port: 8000 2023-09-22T21:12:06.040 DEBUG:boto:Params: {} 2023-09-22T21:12:06.041 DEBUG:boto:Token: None 2023-09-22T21:12:06.041 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-19/?versioning 2023-09-22T21:12:06.041 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oVoMxpswzHlrvi62DpEfs/DXN6Y= 2023-09-22T21:12:06.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oVoMxpswzHlrvi62DpEfs/DXN6Y=', 'Content-Length': '228'} 2023-09-22T21:12:06.061 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000667634cf1b32e22d-00650e0326-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.061 DEBUG:boto:path=/ffzdqu-19/ 2023-09-22T21:12:06.061 DEBUG:boto:auth_path=/ffzdqu-19/ 2023-09-22T21:12:06.061 DEBUG:boto:path=/ffzdqu-19/?versioning 2023-09-22T21:12:06.061 DEBUG:boto:auth_path=/ffzdqu-19/?versioning 2023-09-22T21:12:06.061 DEBUG:boto:Method: GET 2023-09-22T21:12:06.061 DEBUG:boto:Path: /ffzdqu-19/?versioning 2023-09-22T21:12:06.061 DEBUG:boto:Data: 2023-09-22T21:12:06.062 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.062 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:06.062 DEBUG:boto:Port: 8000 2023-09-22T21:12:06.062 DEBUG:boto:Params: {} 2023-09-22T21:12:06.062 DEBUG:boto:Token: None 2023-09-22T21:12:06.062 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-19/?versioning 2023-09-22T21:12:06.062 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xM28LUr7uvW4lJEjizxjHwFRywk= 2023-09-22T21:12:06.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xM28LUr7uvW4lJEjizxjHwFRywk=', 'Content-Length': '0'} 2023-09-22T21:12:06.064 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cd9b98060dc8671f-00650e0326-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '192'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.106 DEBUG:boto:EnabledDisabled 2023-09-22T21:12:06.107 DEBUG:boto:path=/ffzdqu-20/ 2023-09-22T21:12:06.107 DEBUG:boto:auth_path=/ffzdqu-20/ 2023-09-22T21:12:06.107 DEBUG:boto:path=/ffzdqu-20/?versioning 2023-09-22T21:12:06.107 DEBUG:boto:auth_path=/ffzdqu-20/?versioning 2023-09-22T21:12:06.107 DEBUG:boto:Method: PUT 2023-09-22T21:12:06.107 DEBUG:boto:Path: /ffzdqu-20/?versioning 2023-09-22T21:12:06.107 DEBUG:boto:Data: Enabled Disabled 2023-09-22T21:12:06.107 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.108 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:12:06.108 DEBUG:boto:Port: 8001 2023-09-22T21:12:06.108 DEBUG:boto:Params: {} 2023-09-22T21:12:06.108 DEBUG:boto:Token: None 2023-09-22T21:12:06.108 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-20/?versioning 2023-09-22T21:12:06.108 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UlSEWsAkJmwFYw1SDDTP2dtTP/E= 2023-09-22T21:12:06.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UlSEWsAkJmwFYw1SDDTP2dtTP/E=', 'Content-Length': '228'} 2023-09-22T21:12:06.123 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003b7318be14ef62d8-00650e0326-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.124 DEBUG:boto:path=/ffzdqu-20/ 2023-09-22T21:12:06.124 DEBUG:boto:auth_path=/ffzdqu-20/ 2023-09-22T21:12:06.124 DEBUG:boto:path=/ffzdqu-20/?versioning 2023-09-22T21:12:06.124 DEBUG:boto:auth_path=/ffzdqu-20/?versioning 2023-09-22T21:12:06.124 DEBUG:boto:Method: GET 2023-09-22T21:12:06.124 DEBUG:boto:Path: /ffzdqu-20/?versioning 2023-09-22T21:12:06.124 DEBUG:boto:Data: 2023-09-22T21:12:06.124 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.124 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:12:06.124 DEBUG:boto:Port: 8001 2023-09-22T21:12:06.125 DEBUG:boto:Params: {} 2023-09-22T21:12:06.125 DEBUG:boto:Token: None 2023-09-22T21:12:06.125 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-20/?versioning 2023-09-22T21:12:06.125 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T/ZoGZg+6pqK9TfQDKZzXPFBml0= 2023-09-22T21:12:06.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T/ZoGZg+6pqK9TfQDKZzXPFBml0=', 'Content-Length': '0'} 2023-09-22T21:12:06.126 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009b309ad03b47dc7f-00650e0326-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '192'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.170 DEBUG:boto:EnabledDisabled 2023-09-22T21:12:06.170 DEBUG:boto:path=/ffzdqu-21/ 2023-09-22T21:12:06.171 DEBUG:boto:auth_path=/ffzdqu-21/ 2023-09-22T21:12:06.171 DEBUG:boto:path=/ffzdqu-21/?versioning 2023-09-22T21:12:06.171 DEBUG:boto:auth_path=/ffzdqu-21/?versioning 2023-09-22T21:12:06.171 DEBUG:boto:Method: PUT 2023-09-22T21:12:06.171 DEBUG:boto:Path: /ffzdqu-21/?versioning 2023-09-22T21:12:06.171 DEBUG:boto:Data: Enabled Disabled 2023-09-22T21:12:06.171 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.171 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8000 2023-09-22T21:12:06.171 DEBUG:boto:Port: 8000 2023-09-22T21:12:06.171 DEBUG:boto:Params: {} 2023-09-22T21:12:06.172 DEBUG:boto:Token: None 2023-09-22T21:12:06.172 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-21/?versioning 2023-09-22T21:12:06.172 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:or87kZTKqrkLo++LHylKJd5OYeU= 2023-09-22T21:12:06.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:or87kZTKqrkLo++LHylKJd5OYeU=', 'Content-Length': '228'} 2023-09-22T21:12:06.192 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000081cf3c04919ff77e-00650e0326-1109-b1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.192 DEBUG:boto:path=/ffzdqu-21/ 2023-09-22T21:12:06.193 DEBUG:boto:auth_path=/ffzdqu-21/ 2023-09-22T21:12:06.193 DEBUG:boto:path=/ffzdqu-21/?versioning 2023-09-22T21:12:06.193 DEBUG:boto:auth_path=/ffzdqu-21/?versioning 2023-09-22T21:12:06.193 DEBUG:boto:Method: GET 2023-09-22T21:12:06.193 DEBUG:boto:Path: /ffzdqu-21/?versioning 2023-09-22T21:12:06.193 DEBUG:boto:Data: 2023-09-22T21:12:06.193 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.193 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8000 2023-09-22T21:12:06.193 DEBUG:boto:Port: 8000 2023-09-22T21:12:06.193 DEBUG:boto:Params: {} 2023-09-22T21:12:06.193 DEBUG:boto:Token: None 2023-09-22T21:12:06.194 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-21/?versioning 2023-09-22T21:12:06.194 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SYVCTRgE69xVwyKAZinygmuGLB8= 2023-09-22T21:12:06.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SYVCTRgE69xVwyKAZinygmuGLB8=', 'Content-Length': '0'} 2023-09-22T21:12:06.195 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000073672fe1a955fa4c-00650e0326-1109-b1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '192'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.196 DEBUG:boto:EnabledDisabled 2023-09-22T21:12:06.196 DEBUG:boto:path=/ffzdqu-22/ 2023-09-22T21:12:06.196 DEBUG:boto:auth_path=/ffzdqu-22/ 2023-09-22T21:12:06.196 DEBUG:boto:path=/ffzdqu-22/?versioning 2023-09-22T21:12:06.196 DEBUG:boto:auth_path=/ffzdqu-22/?versioning 2023-09-22T21:12:06.196 DEBUG:boto:Method: PUT 2023-09-22T21:12:06.196 DEBUG:boto:Path: /ffzdqu-22/?versioning 2023-09-22T21:12:06.196 DEBUG:boto:Data: Enabled Disabled 2023-09-22T21:12:06.196 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.196 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8001 2023-09-22T21:12:06.197 DEBUG:boto:Port: 8001 2023-09-22T21:12:06.197 DEBUG:boto:Params: {} 2023-09-22T21:12:06.197 DEBUG:boto:Token: None 2023-09-22T21:12:06.197 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-22/?versioning 2023-09-22T21:12:06.197 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:U1XjSaxqMAwU+OMh28vo9I/YavM= 2023-09-22T21:12:06.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:U1XjSaxqMAwU+OMh28vo9I/YavM=', 'Content-Length': '228'} 2023-09-22T21:12:06.217 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b4089319edc3995f-00650e0326-1110-b2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.217 DEBUG:boto:path=/ffzdqu-22/ 2023-09-22T21:12:06.217 DEBUG:boto:auth_path=/ffzdqu-22/ 2023-09-22T21:12:06.217 DEBUG:boto:path=/ffzdqu-22/?versioning 2023-09-22T21:12:06.217 DEBUG:boto:auth_path=/ffzdqu-22/?versioning 2023-09-22T21:12:06.217 DEBUG:boto:Method: GET 2023-09-22T21:12:06.217 DEBUG:boto:Path: /ffzdqu-22/?versioning 2023-09-22T21:12:06.217 DEBUG:boto:Data: 2023-09-22T21:12:06.217 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.218 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8001 2023-09-22T21:12:06.218 DEBUG:boto:Port: 8001 2023-09-22T21:12:06.218 DEBUG:boto:Params: {} 2023-09-22T21:12:06.218 DEBUG:boto:Token: None 2023-09-22T21:12:06.218 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-22/?versioning 2023-09-22T21:12:06.218 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IX5vcfRIH+2SVsDQr0s52wFb+WU= 2023-09-22T21:12:06.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IX5vcfRIH+2SVsDQr0s52wFb+WU=', 'Content-Length': '0'} 2023-09-22T21:12:06.220 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c6de662d4a95ec7b-00650e0326-1110-b2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '192'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.220 DEBUG:boto:EnabledDisabled 2023-09-22T21:12:06.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2023-09-22T21:12:06.220 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-23 2023-09-22T21:12:06.221 DEBUG:boto:path=/ffzdqu-23/ 2023-09-22T21:12:06.221 DEBUG:boto:auth_path=/ffzdqu-23/ 2023-09-22T21:12:06.221 DEBUG:boto:Method: PUT 2023-09-22T21:12:06.221 DEBUG:boto:Path: /ffzdqu-23/ 2023-09-22T21:12:06.221 DEBUG:boto:Data: 2023-09-22T21:12:06.221 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.221 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:06.221 DEBUG:boto:Port: 8000 2023-09-22T21:12:06.221 DEBUG:boto:Params: {} 2023-09-22T21:12:06.221 DEBUG:boto:Token: None 2023-09-22T21:12:06.222 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-23/ 2023-09-22T21:12:06.222 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PlTYuXPLAlNqZozBq1FWxOn/TVw= 2023-09-22T21:12:06.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PlTYuXPLAlNqZozBq1FWxOn/TVw=', 'Content-Length': '0'} 2023-09-22T21:12:06.302 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ff3c4f40a40103bb-00650e0326-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.303 INFO:rgw_multi.tests:create bucket zone=a2 name=ffzdqu-24 2023-09-22T21:12:06.303 DEBUG:boto:path=/ffzdqu-24/ 2023-09-22T21:12:06.303 DEBUG:boto:auth_path=/ffzdqu-24/ 2023-09-22T21:12:06.303 DEBUG:boto:Method: PUT 2023-09-22T21:12:06.303 DEBUG:boto:Path: /ffzdqu-24/ 2023-09-22T21:12:06.303 DEBUG:boto:Data: 2023-09-22T21:12:06.303 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.303 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:12:06.303 DEBUG:boto:Port: 8001 2023-09-22T21:12:06.303 DEBUG:boto:Params: {} 2023-09-22T21:12:06.304 DEBUG:boto:Token: None 2023-09-22T21:12:06.304 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-24/ 2023-09-22T21:12:06.304 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pbJiYMC4hEJoZqXNFnL5io4u3HU= 2023-09-22T21:12:06.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pbJiYMC4hEJoZqXNFnL5io4u3HU=', 'Content-Length': '0'} 2023-09-22T21:12:06.339 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003d76929b714cb5ca-00650e0326-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.340 INFO:rgw_multi.tests:create bucket zone=b1 name=ffzdqu-25 2023-09-22T21:12:06.340 DEBUG:boto:path=/ffzdqu-25/ 2023-09-22T21:12:06.340 DEBUG:boto:auth_path=/ffzdqu-25/ 2023-09-22T21:12:06.340 DEBUG:boto:Method: PUT 2023-09-22T21:12:06.340 DEBUG:boto:Path: /ffzdqu-25/ 2023-09-22T21:12:06.340 DEBUG:boto:Data: 2023-09-22T21:12:06.340 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.340 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8000 2023-09-22T21:12:06.340 DEBUG:boto:Port: 8000 2023-09-22T21:12:06.340 DEBUG:boto:Params: {} 2023-09-22T21:12:06.341 DEBUG:boto:Token: None 2023-09-22T21:12:06.341 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-25/ 2023-09-22T21:12:06.341 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:irYsjx9Pu0u/ocBDiKww1ZT8VgM= 2023-09-22T21:12:06.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:irYsjx9Pu0u/ocBDiKww1ZT8VgM=', 'Content-Length': '0'} 2023-09-22T21:12:06.389 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002644acf6b45ccd4d-00650e0326-1109-b1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.390 INFO:rgw_multi.tests:create bucket zone=b2 name=ffzdqu-26 2023-09-22T21:12:06.390 DEBUG:boto:path=/ffzdqu-26/ 2023-09-22T21:12:06.390 DEBUG:boto:auth_path=/ffzdqu-26/ 2023-09-22T21:12:06.390 DEBUG:boto:Method: PUT 2023-09-22T21:12:06.390 DEBUG:boto:Path: /ffzdqu-26/ 2023-09-22T21:12:06.390 DEBUG:boto:Data: 2023-09-22T21:12:06.390 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.391 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8001 2023-09-22T21:12:06.391 DEBUG:boto:Port: 8001 2023-09-22T21:12:06.391 DEBUG:boto:Params: {} 2023-09-22T21:12:06.391 DEBUG:boto:Token: None 2023-09-22T21:12:06.391 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-26/ 2023-09-22T21:12:06.391 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P0yCMFmo9Y0Uzaw9uBI/ys9rhYM= 2023-09-22T21:12:06.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P0yCMFmo9Y0Uzaw9uBI/ys9rhYM=', 'Content-Length': '0'} 2023-09-22T21:12:06.463 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000291212ec9ee92b49-00650e0326-1110-b2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.539 DEBUG:boto:path=/ffzdqu-23/ 2023-09-22T21:12:06.539 DEBUG:boto:auth_path=/ffzdqu-23/ 2023-09-22T21:12:06.539 DEBUG:boto:path=/ffzdqu-23/?acl 2023-09-22T21:12:06.539 DEBUG:boto:auth_path=/ffzdqu-23/?acl 2023-09-22T21:12:06.539 DEBUG:boto:Method: GET 2023-09-22T21:12:06.539 DEBUG:boto:Path: /ffzdqu-23/?acl 2023-09-22T21:12:06.539 DEBUG:boto:Data: 2023-09-22T21:12:06.540 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.540 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:06.540 DEBUG:boto:Port: 8000 2023-09-22T21:12:06.540 DEBUG:boto:Params: {} 2023-09-22T21:12:06.540 DEBUG:boto:Token: None 2023-09-22T21:12:06.540 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-23/?acl 2023-09-22T21:12:06.540 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/ITUPOn3lSuVTxY+kpDHjaGaFxc= 2023-09-22T21:12:06.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/ITUPOn3lSuVTxY+kpDHjaGaFxc=', 'Content-Length': '0'} 2023-09-22T21:12:06.542 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d1f67d2d5351df82-00650e0326-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '489'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.543 DEBUG:boto:path=/ffzdqu-23/ 2023-09-22T21:12:06.543 DEBUG:boto:auth_path=/ffzdqu-23/ 2023-09-22T21:12:06.543 DEBUG:boto:path=/ffzdqu-23/?acl 2023-09-22T21:12:06.543 DEBUG:boto:auth_path=/ffzdqu-23/?acl 2023-09-22T21:12:06.543 DEBUG:boto:Method: PUT 2023-09-22T21:12:06.543 DEBUG:boto:Path: /ffzdqu-23/?acl 2023-09-22T21:12:06.543 DEBUG:boto:Data: 2023-09-22T21:12:06.543 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-22T21:12:06.543 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:06.544 DEBUG:boto:Port: 8000 2023-09-22T21:12:06.544 DEBUG:boto:Params: {} 2023-09-22T21:12:06.544 DEBUG:boto:Token: None 2023-09-22T21:12:06.544 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:06 GMT x-amz-acl:public-read /ffzdqu-23/?acl 2023-09-22T21:12:06.544 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:57sfA/5sCYRp7u7NatA7p+qgSWU= 2023-09-22T21:12:06.544 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:57sfA/5sCYRp7u7NatA7p+qgSWU=', 'Content-Length': '0'} 2023-09-22T21:12:06.557 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d19b06466b094b5e-00650e0326-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.557 DEBUG:boto:path=/ffzdqu-23/ 2023-09-22T21:12:06.557 DEBUG:boto:auth_path=/ffzdqu-23/ 2023-09-22T21:12:06.557 DEBUG:boto:path=/ffzdqu-23/?acl 2023-09-22T21:12:06.557 DEBUG:boto:auth_path=/ffzdqu-23/?acl 2023-09-22T21:12:06.558 DEBUG:boto:Method: GET 2023-09-22T21:12:06.558 DEBUG:boto:Path: /ffzdqu-23/?acl 2023-09-22T21:12:06.558 DEBUG:boto:Data: 2023-09-22T21:12:06.558 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.558 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:06.558 DEBUG:boto:Port: 8000 2023-09-22T21:12:06.558 DEBUG:boto:Params: {} 2023-09-22T21:12:06.558 DEBUG:boto:Token: None 2023-09-22T21:12:06.558 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-23/?acl 2023-09-22T21:12:06.559 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/ITUPOn3lSuVTxY+kpDHjaGaFxc= 2023-09-22T21:12:06.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/ITUPOn3lSuVTxY+kpDHjaGaFxc=', 'Content-Length': '0'} 2023-09-22T21:12:06.560 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004b5e4abe098228d3-00650e0326-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '681'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.560 DEBUG:boto:path=/ffzdqu-24/ 2023-09-22T21:12:06.561 DEBUG:boto:auth_path=/ffzdqu-24/ 2023-09-22T21:12:06.561 DEBUG:boto:path=/ffzdqu-24/?acl 2023-09-22T21:12:06.561 DEBUG:boto:auth_path=/ffzdqu-24/?acl 2023-09-22T21:12:06.561 DEBUG:boto:Method: GET 2023-09-22T21:12:06.561 DEBUG:boto:Path: /ffzdqu-24/?acl 2023-09-22T21:12:06.561 DEBUG:boto:Data: 2023-09-22T21:12:06.561 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.561 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:12:06.561 DEBUG:boto:Port: 8001 2023-09-22T21:12:06.562 DEBUG:boto:Params: {} 2023-09-22T21:12:06.562 DEBUG:boto:Token: None 2023-09-22T21:12:06.562 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-24/?acl 2023-09-22T21:12:06.562 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uL/tGiAGUh9uEWQijMBjbyJyUxk= 2023-09-22T21:12:06.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uL/tGiAGUh9uEWQijMBjbyJyUxk=', 'Content-Length': '0'} 2023-09-22T21:12:06.563 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009b5c125a8604088f-00650e0326-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '489'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.564 DEBUG:boto:path=/ffzdqu-24/ 2023-09-22T21:12:06.564 DEBUG:boto:auth_path=/ffzdqu-24/ 2023-09-22T21:12:06.564 DEBUG:boto:path=/ffzdqu-24/?acl 2023-09-22T21:12:06.564 DEBUG:boto:auth_path=/ffzdqu-24/?acl 2023-09-22T21:12:06.564 DEBUG:boto:Method: PUT 2023-09-22T21:12:06.564 DEBUG:boto:Path: /ffzdqu-24/?acl 2023-09-22T21:12:06.565 DEBUG:boto:Data: 2023-09-22T21:12:06.565 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-22T21:12:06.565 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:12:06.565 DEBUG:boto:Port: 8001 2023-09-22T21:12:06.565 DEBUG:boto:Params: {} 2023-09-22T21:12:06.565 DEBUG:boto:Token: None 2023-09-22T21:12:06.565 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:06 GMT x-amz-acl:public-read /ffzdqu-24/?acl 2023-09-22T21:12:06.565 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9w+oiR2e+luMUk+0LU/tyWo2Qlo= 2023-09-22T21:12:06.566 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9w+oiR2e+luMUk+0LU/tyWo2Qlo=', 'Content-Length': '0'} 2023-09-22T21:12:06.591 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002e265682f55abd80-00650e0326-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.592 DEBUG:boto:path=/ffzdqu-24/ 2023-09-22T21:12:06.592 DEBUG:boto:auth_path=/ffzdqu-24/ 2023-09-22T21:12:06.592 DEBUG:boto:path=/ffzdqu-24/?acl 2023-09-22T21:12:06.592 DEBUG:boto:auth_path=/ffzdqu-24/?acl 2023-09-22T21:12:06.592 DEBUG:boto:Method: GET 2023-09-22T21:12:06.592 DEBUG:boto:Path: /ffzdqu-24/?acl 2023-09-22T21:12:06.592 DEBUG:boto:Data: 2023-09-22T21:12:06.593 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.593 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:12:06.593 DEBUG:boto:Port: 8001 2023-09-22T21:12:06.593 DEBUG:boto:Params: {} 2023-09-22T21:12:06.593 DEBUG:boto:Token: None 2023-09-22T21:12:06.593 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-24/?acl 2023-09-22T21:12:06.593 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uL/tGiAGUh9uEWQijMBjbyJyUxk= 2023-09-22T21:12:06.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uL/tGiAGUh9uEWQijMBjbyJyUxk=', 'Content-Length': '0'} 2023-09-22T21:12:06.595 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000de4ba449f6f2cfe5-00650e0326-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '681'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.595 DEBUG:boto:path=/ffzdqu-25/ 2023-09-22T21:12:06.596 DEBUG:boto:auth_path=/ffzdqu-25/ 2023-09-22T21:12:06.596 DEBUG:boto:path=/ffzdqu-25/?acl 2023-09-22T21:12:06.596 DEBUG:boto:auth_path=/ffzdqu-25/?acl 2023-09-22T21:12:06.596 DEBUG:boto:Method: GET 2023-09-22T21:12:06.596 DEBUG:boto:Path: /ffzdqu-25/?acl 2023-09-22T21:12:06.596 DEBUG:boto:Data: 2023-09-22T21:12:06.596 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.596 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8000 2023-09-22T21:12:06.596 DEBUG:boto:Port: 8000 2023-09-22T21:12:06.596 DEBUG:boto:Params: {} 2023-09-22T21:12:06.597 DEBUG:boto:Token: None 2023-09-22T21:12:06.597 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-25/?acl 2023-09-22T21:12:06.597 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hMUdloOYayK0M7AS37JHGSKbU74= 2023-09-22T21:12:06.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hMUdloOYayK0M7AS37JHGSKbU74=', 'Content-Length': '0'} 2023-09-22T21:12:06.598 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004a62f72ca5188b42-00650e0326-1109-b1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '489'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.599 DEBUG:boto:path=/ffzdqu-25/ 2023-09-22T21:12:06.599 DEBUG:boto:auth_path=/ffzdqu-25/ 2023-09-22T21:12:06.599 DEBUG:boto:path=/ffzdqu-25/?acl 2023-09-22T21:12:06.599 DEBUG:boto:auth_path=/ffzdqu-25/?acl 2023-09-22T21:12:06.599 DEBUG:boto:Method: PUT 2023-09-22T21:12:06.599 DEBUG:boto:Path: /ffzdqu-25/?acl 2023-09-22T21:12:06.599 DEBUG:boto:Data: 2023-09-22T21:12:06.600 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-22T21:12:06.600 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8000 2023-09-22T21:12:06.600 DEBUG:boto:Port: 8000 2023-09-22T21:12:06.600 DEBUG:boto:Params: {} 2023-09-22T21:12:06.600 DEBUG:boto:Token: None 2023-09-22T21:12:06.600 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:06 GMT x-amz-acl:public-read /ffzdqu-25/?acl 2023-09-22T21:12:06.600 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sj9jpURr6igwXnfjKlXLeJ7pJDE= 2023-09-22T21:12:06.600 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sj9jpURr6igwXnfjKlXLeJ7pJDE=', 'Content-Length': '0'} 2023-09-22T21:12:06.625 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000068ba593021830b8a-00650e0326-1109-b1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.625 DEBUG:boto:path=/ffzdqu-25/ 2023-09-22T21:12:06.625 DEBUG:boto:auth_path=/ffzdqu-25/ 2023-09-22T21:12:06.626 DEBUG:boto:path=/ffzdqu-25/?acl 2023-09-22T21:12:06.626 DEBUG:boto:auth_path=/ffzdqu-25/?acl 2023-09-22T21:12:06.626 DEBUG:boto:Method: GET 2023-09-22T21:12:06.626 DEBUG:boto:Path: /ffzdqu-25/?acl 2023-09-22T21:12:06.626 DEBUG:boto:Data: 2023-09-22T21:12:06.626 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.626 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8000 2023-09-22T21:12:06.626 DEBUG:boto:Port: 8000 2023-09-22T21:12:06.626 DEBUG:boto:Params: {} 2023-09-22T21:12:06.627 DEBUG:boto:Token: None 2023-09-22T21:12:06.627 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-25/?acl 2023-09-22T21:12:06.627 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hMUdloOYayK0M7AS37JHGSKbU74= 2023-09-22T21:12:06.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hMUdloOYayK0M7AS37JHGSKbU74=', 'Content-Length': '0'} 2023-09-22T21:12:06.628 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f5a05e0a907e5c9e-00650e0326-1109-b1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '681'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.675 DEBUG:boto:path=/ffzdqu-26/ 2023-09-22T21:12:06.675 DEBUG:boto:auth_path=/ffzdqu-26/ 2023-09-22T21:12:06.675 DEBUG:boto:path=/ffzdqu-26/?acl 2023-09-22T21:12:06.675 DEBUG:boto:auth_path=/ffzdqu-26/?acl 2023-09-22T21:12:06.675 DEBUG:boto:Method: GET 2023-09-22T21:12:06.675 DEBUG:boto:Path: /ffzdqu-26/?acl 2023-09-22T21:12:06.675 DEBUG:boto:Data: 2023-09-22T21:12:06.675 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.676 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8001 2023-09-22T21:12:06.676 DEBUG:boto:Port: 8001 2023-09-22T21:12:06.676 DEBUG:boto:Params: {} 2023-09-22T21:12:06.676 DEBUG:boto:Token: None 2023-09-22T21:12:06.676 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-26/?acl 2023-09-22T21:12:06.676 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:itknJtX+ZcI36KirUY/kNmT6YZA= 2023-09-22T21:12:06.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:itknJtX+ZcI36KirUY/kNmT6YZA=', 'Content-Length': '0'} 2023-09-22T21:12:06.678 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000287524b6a3a6ad0c-00650e0326-1110-b2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '489'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.678 DEBUG:boto:path=/ffzdqu-26/ 2023-09-22T21:12:06.678 DEBUG:boto:auth_path=/ffzdqu-26/ 2023-09-22T21:12:06.679 DEBUG:boto:path=/ffzdqu-26/?acl 2023-09-22T21:12:06.679 DEBUG:boto:auth_path=/ffzdqu-26/?acl 2023-09-22T21:12:06.679 DEBUG:boto:Method: PUT 2023-09-22T21:12:06.679 DEBUG:boto:Path: /ffzdqu-26/?acl 2023-09-22T21:12:06.679 DEBUG:boto:Data: 2023-09-22T21:12:06.679 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-22T21:12:06.679 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8001 2023-09-22T21:12:06.679 DEBUG:boto:Port: 8001 2023-09-22T21:12:06.679 DEBUG:boto:Params: {} 2023-09-22T21:12:06.679 DEBUG:boto:Token: None 2023-09-22T21:12:06.680 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:06 GMT x-amz-acl:public-read /ffzdqu-26/?acl 2023-09-22T21:12:06.680 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ikDHAGAKwu2iTQoMeSGuTvuTtE4= 2023-09-22T21:12:06.680 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ikDHAGAKwu2iTQoMeSGuTvuTtE4=', 'Content-Length': '0'} 2023-09-22T21:12:06.700 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b6bc2bc21ccbb4b4-00650e0326-1110-b2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.700 DEBUG:boto:path=/ffzdqu-26/ 2023-09-22T21:12:06.700 DEBUG:boto:auth_path=/ffzdqu-26/ 2023-09-22T21:12:06.700 DEBUG:boto:path=/ffzdqu-26/?acl 2023-09-22T21:12:06.700 DEBUG:boto:auth_path=/ffzdqu-26/?acl 2023-09-22T21:12:06.700 DEBUG:boto:Method: GET 2023-09-22T21:12:06.701 DEBUG:boto:Path: /ffzdqu-26/?acl 2023-09-22T21:12:06.701 DEBUG:boto:Data: 2023-09-22T21:12:06.701 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.701 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8001 2023-09-22T21:12:06.701 DEBUG:boto:Port: 8001 2023-09-22T21:12:06.701 DEBUG:boto:Params: {} 2023-09-22T21:12:06.701 DEBUG:boto:Token: None 2023-09-22T21:12:06.701 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-26/?acl 2023-09-22T21:12:06.701 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:itknJtX+ZcI36KirUY/kNmT6YZA= 2023-09-22T21:12:06.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:itknJtX+ZcI36KirUY/kNmT6YZA=', 'Content-Length': '0'} 2023-09-22T21:12:06.703 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d08dfa157006b120-00650e0326-1110-b2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '681'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2023-09-22T21:12:06.704 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-27 2023-09-22T21:12:06.704 DEBUG:boto:path=/ffzdqu-27/ 2023-09-22T21:12:06.704 DEBUG:boto:auth_path=/ffzdqu-27/ 2023-09-22T21:12:06.704 DEBUG:boto:Method: PUT 2023-09-22T21:12:06.705 DEBUG:boto:Path: /ffzdqu-27/ 2023-09-22T21:12:06.705 DEBUG:boto:Data: 2023-09-22T21:12:06.705 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.705 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:06.705 DEBUG:boto:Port: 8000 2023-09-22T21:12:06.705 DEBUG:boto:Params: {} 2023-09-22T21:12:06.705 DEBUG:boto:Token: None 2023-09-22T21:12:06.705 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-27/ 2023-09-22T21:12:06.705 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZGxM4xIGFgiRPOZjV75EYzGcQwY= 2023-09-22T21:12:06.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZGxM4xIGFgiRPOZjV75EYzGcQwY=', 'Content-Length': '0'} 2023-09-22T21:12:06.750 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006a733bb73cdf9d54-00650e0326-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.750 INFO:rgw_multi.tests:create bucket zone=a2 name=ffzdqu-28 2023-09-22T21:12:06.750 DEBUG:boto:path=/ffzdqu-28/ 2023-09-22T21:12:06.750 DEBUG:boto:auth_path=/ffzdqu-28/ 2023-09-22T21:12:06.750 DEBUG:boto:Method: PUT 2023-09-22T21:12:06.750 DEBUG:boto:Path: /ffzdqu-28/ 2023-09-22T21:12:06.750 DEBUG:boto:Data: 2023-09-22T21:12:06.750 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.751 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:12:06.751 DEBUG:boto:Port: 8001 2023-09-22T21:12:06.751 DEBUG:boto:Params: {} 2023-09-22T21:12:06.751 DEBUG:boto:Token: None 2023-09-22T21:12:06.751 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-28/ 2023-09-22T21:12:06.751 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HNoQsKaTnD7ZMLSY8KY+LT8NRJo= 2023-09-22T21:12:06.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HNoQsKaTnD7ZMLSY8KY+LT8NRJo=', 'Content-Length': '0'} 2023-09-22T21:12:06.819 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009533a4c18e3f0904-00650e0326-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.820 INFO:rgw_multi.tests:create bucket zone=b1 name=ffzdqu-29 2023-09-22T21:12:06.820 DEBUG:boto:path=/ffzdqu-29/ 2023-09-22T21:12:06.820 DEBUG:boto:auth_path=/ffzdqu-29/ 2023-09-22T21:12:06.820 DEBUG:boto:Method: PUT 2023-09-22T21:12:06.820 DEBUG:boto:Path: /ffzdqu-29/ 2023-09-22T21:12:06.820 DEBUG:boto:Data: 2023-09-22T21:12:06.820 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.820 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8000 2023-09-22T21:12:06.821 DEBUG:boto:Port: 8000 2023-09-22T21:12:06.821 DEBUG:boto:Params: {} 2023-09-22T21:12:06.821 DEBUG:boto:Token: None 2023-09-22T21:12:06.821 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-29/ 2023-09-22T21:12:06.821 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OQPkUVtDiAFdIh6ZHLD29Sj8iVE= 2023-09-22T21:12:06.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OQPkUVtDiAFdIh6ZHLD29Sj8iVE=', 'Content-Length': '0'} 2023-09-22T21:12:06.885 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d6ddbcb193aaccb1-00650e0326-1109-b1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.886 INFO:rgw_multi.tests:create bucket zone=b2 name=ffzdqu-30 2023-09-22T21:12:06.886 DEBUG:boto:path=/ffzdqu-30/ 2023-09-22T21:12:06.886 DEBUG:boto:auth_path=/ffzdqu-30/ 2023-09-22T21:12:06.886 DEBUG:boto:Method: PUT 2023-09-22T21:12:06.886 DEBUG:boto:Path: /ffzdqu-30/ 2023-09-22T21:12:06.886 DEBUG:boto:Data: 2023-09-22T21:12:06.886 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.886 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8001 2023-09-22T21:12:06.887 DEBUG:boto:Port: 8001 2023-09-22T21:12:06.887 DEBUG:boto:Params: {} 2023-09-22T21:12:06.887 DEBUG:boto:Token: None 2023-09-22T21:12:06.887 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-30/ 2023-09-22T21:12:06.887 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JQR93Rd4+/ohTHc8M4LS3KQRbpk= 2023-09-22T21:12:06.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JQR93Rd4+/ohTHc8M4LS3KQRbpk=', 'Content-Length': '0'} 2023-09-22T21:12:06.944 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f93185965fb072bf-00650e0326-1110-b2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.944 DEBUG:boto:path=/ffzdqu-27/ 2023-09-22T21:12:06.944 DEBUG:boto:auth_path=/ffzdqu-27/ 2023-09-22T21:12:06.944 DEBUG:boto:path=/ffzdqu-27/?cors 2023-09-22T21:12:06.944 DEBUG:boto:auth_path=/ffzdqu-27/?cors 2023-09-22T21:12:06.945 DEBUG:boto:Method: PUT 2023-09-22T21:12:06.945 DEBUG:boto:Path: /ffzdqu-27/?cors 2023-09-22T21:12:06.945 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-09-22T21:12:06.945 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-09-22T21:12:06.945 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:06.945 DEBUG:boto:Port: 8000 2023-09-22T21:12:06.945 DEBUG:boto:Params: {} 2023-09-22T21:12:06.945 DEBUG:boto:Token: None 2023-09-22T21:12:06.945 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-27/?cors 2023-09-22T21:12:06.946 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rds3pSO6rqfz5cl5ek9hJinq4bY= 2023-09-22T21:12:06.946 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rds3pSO6rqfz5cl5ek9hJinq4bY=', 'Content-Length': '218'} 2023-09-22T21:12:06.955 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f8386cf020125fc4-00650e0326-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.955 DEBUG:boto:path=/ffzdqu-27/ 2023-09-22T21:12:06.955 DEBUG:boto:auth_path=/ffzdqu-27/ 2023-09-22T21:12:06.955 DEBUG:boto:path=/ffzdqu-27/?cors 2023-09-22T21:12:06.956 DEBUG:boto:auth_path=/ffzdqu-27/?cors 2023-09-22T21:12:06.956 DEBUG:boto:Method: GET 2023-09-22T21:12:06.956 DEBUG:boto:Path: /ffzdqu-27/?cors 2023-09-22T21:12:06.956 DEBUG:boto:Data: 2023-09-22T21:12:06.956 DEBUG:boto:Headers: {} 2023-09-22T21:12:06.956 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:06.956 DEBUG:boto:Port: 8000 2023-09-22T21:12:06.956 DEBUG:boto:Params: {} 2023-09-22T21:12:06.956 DEBUG:boto:Token: None 2023-09-22T21:12:06.957 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:12:06 GMT /ffzdqu-27/?cors 2023-09-22T21:12:06.957 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jjGfLeNdwsEkkjm95OZ6kdAWsdA= 2023-09-22T21:12:06.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jjGfLeNdwsEkkjm95OZ6kdAWsdA=', 'Content-Length': '0'} 2023-09-22T21:12:06.958 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002bc3d50c5ca1ab4d-00650e0326-111c-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '266'), ('Date', 'Fri, 22 Sep 2023 21:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:06.998 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-09-22T21:12:06.999 DEBUG:boto:path=/ffzdqu-28/ 2023-09-22T21:12:06.999 DEBUG:boto:auth_path=/ffzdqu-28/ 2023-09-22T21:12:06.999 DEBUG:boto:path=/ffzdqu-28/?cors 2023-09-22T21:12:06.999 DEBUG:boto:auth_path=/ffzdqu-28/?cors 2023-09-22T21:12:06.999 DEBUG:boto:Method: PUT 2023-09-22T21:12:06.999 DEBUG:boto:Path: /ffzdqu-28/?cors 2023-09-22T21:12:06.999 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-09-22T21:12:06.999 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-09-22T21:12:06.999 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:12:07.000 DEBUG:boto:Port: 8001 2023-09-22T21:12:07.000 DEBUG:boto:Params: {} 2023-09-22T21:12:07.000 DEBUG:boto:Token: None 2023-09-22T21:12:07.000 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 22 Sep 2023 21:12:07 GMT /ffzdqu-28/?cors 2023-09-22T21:12:07.000 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0RSiWJ+guNRnUvFclTyLBLiwYbg= 2023-09-22T21:12:07.000 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0RSiWJ+guNRnUvFclTyLBLiwYbg=', 'Content-Length': '218'} 2023-09-22T21:12:08.021 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ed2be5808524de0e-00650e0327-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:08.022 DEBUG:boto:path=/ffzdqu-28/ 2023-09-22T21:12:08.022 DEBUG:boto:auth_path=/ffzdqu-28/ 2023-09-22T21:12:08.022 DEBUG:boto:path=/ffzdqu-28/?cors 2023-09-22T21:12:08.022 DEBUG:boto:auth_path=/ffzdqu-28/?cors 2023-09-22T21:12:08.022 DEBUG:boto:Method: GET 2023-09-22T21:12:08.022 DEBUG:boto:Path: /ffzdqu-28/?cors 2023-09-22T21:12:08.023 DEBUG:boto:Data: 2023-09-22T21:12:08.023 DEBUG:boto:Headers: {} 2023-09-22T21:12:08.023 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:12:08.023 DEBUG:boto:Port: 8001 2023-09-22T21:12:08.023 DEBUG:boto:Params: {} 2023-09-22T21:12:08.023 DEBUG:boto:Token: None 2023-09-22T21:12:08.023 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:12:08 GMT /ffzdqu-28/?cors 2023-09-22T21:12:08.023 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HpFalJb9BSBm5SU93FArl5Ot/4Y= 2023-09-22T21:12:08.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HpFalJb9BSBm5SU93FArl5Ot/4Y=', 'Content-Length': '0'} 2023-09-22T21:12:08.025 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b9228e5832d73a2c-00650e0328-1120-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '266'), ('Date', 'Fri, 22 Sep 2023 21:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:08.025 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-09-22T21:12:08.025 DEBUG:boto:path=/ffzdqu-29/ 2023-09-22T21:12:08.026 DEBUG:boto:auth_path=/ffzdqu-29/ 2023-09-22T21:12:08.026 DEBUG:boto:path=/ffzdqu-29/?cors 2023-09-22T21:12:08.026 DEBUG:boto:auth_path=/ffzdqu-29/?cors 2023-09-22T21:12:08.026 DEBUG:boto:Method: PUT 2023-09-22T21:12:08.026 DEBUG:boto:Path: /ffzdqu-29/?cors 2023-09-22T21:12:08.026 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-09-22T21:12:08.026 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-09-22T21:12:08.026 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8000 2023-09-22T21:12:08.026 DEBUG:boto:Port: 8000 2023-09-22T21:12:08.027 DEBUG:boto:Params: {} 2023-09-22T21:12:08.027 DEBUG:boto:Token: None 2023-09-22T21:12:08.027 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 22 Sep 2023 21:12:08 GMT /ffzdqu-29/?cors 2023-09-22T21:12:08.027 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:acqA3hGBMtzyqGd+6Hj/NLcCb/8= 2023-09-22T21:12:08.027 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:acqA3hGBMtzyqGd+6Hj/NLcCb/8=', 'Content-Length': '218'} 2023-09-22T21:12:09.048 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002aef2abdcc02029c-00650e0328-1109-b1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:09.048 DEBUG:boto:path=/ffzdqu-29/ 2023-09-22T21:12:09.048 DEBUG:boto:auth_path=/ffzdqu-29/ 2023-09-22T21:12:09.048 DEBUG:boto:path=/ffzdqu-29/?cors 2023-09-22T21:12:09.048 DEBUG:boto:auth_path=/ffzdqu-29/?cors 2023-09-22T21:12:09.048 DEBUG:boto:Method: GET 2023-09-22T21:12:09.049 DEBUG:boto:Path: /ffzdqu-29/?cors 2023-09-22T21:12:09.049 DEBUG:boto:Data: 2023-09-22T21:12:09.049 DEBUG:boto:Headers: {} 2023-09-22T21:12:09.049 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8000 2023-09-22T21:12:09.049 DEBUG:boto:Port: 8000 2023-09-22T21:12:09.049 DEBUG:boto:Params: {} 2023-09-22T21:12:09.049 DEBUG:boto:Token: None 2023-09-22T21:12:09.049 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:12:09 GMT /ffzdqu-29/?cors 2023-09-22T21:12:09.049 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UeVr7uwTEpAJS6TAHfmxGsptQqc= 2023-09-22T21:12:09.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UeVr7uwTEpAJS6TAHfmxGsptQqc=', 'Content-Length': '0'} 2023-09-22T21:12:09.051 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b79ef648ebfd55a3-00650e0329-1109-b1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '266'), ('Date', 'Fri, 22 Sep 2023 21:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:09.051 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-09-22T21:12:09.052 DEBUG:boto:path=/ffzdqu-30/ 2023-09-22T21:12:09.052 DEBUG:boto:auth_path=/ffzdqu-30/ 2023-09-22T21:12:09.052 DEBUG:boto:path=/ffzdqu-30/?cors 2023-09-22T21:12:09.052 DEBUG:boto:auth_path=/ffzdqu-30/?cors 2023-09-22T21:12:09.052 DEBUG:boto:Method: PUT 2023-09-22T21:12:09.052 DEBUG:boto:Path: /ffzdqu-30/?cors 2023-09-22T21:12:09.052 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-09-22T21:12:09.052 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-09-22T21:12:09.052 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8001 2023-09-22T21:12:09.053 DEBUG:boto:Port: 8001 2023-09-22T21:12:09.053 DEBUG:boto:Params: {} 2023-09-22T21:12:09.053 DEBUG:boto:Token: None 2023-09-22T21:12:09.053 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 22 Sep 2023 21:12:09 GMT /ffzdqu-30/?cors 2023-09-22T21:12:09.053 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nOXdyzegciwtj2QGTcVqn68L5zI= 2023-09-22T21:12:09.053 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nOXdyzegciwtj2QGTcVqn68L5zI=', 'Content-Length': '218'} 2023-09-22T21:12:10.074 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000005fbeff5269e039b-00650e0329-1110-b2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:10.074 DEBUG:boto:path=/ffzdqu-30/ 2023-09-22T21:12:10.074 DEBUG:boto:auth_path=/ffzdqu-30/ 2023-09-22T21:12:10.075 DEBUG:boto:path=/ffzdqu-30/?cors 2023-09-22T21:12:10.075 DEBUG:boto:auth_path=/ffzdqu-30/?cors 2023-09-22T21:12:10.075 DEBUG:boto:Method: GET 2023-09-22T21:12:10.075 DEBUG:boto:Path: /ffzdqu-30/?cors 2023-09-22T21:12:10.075 DEBUG:boto:Data: 2023-09-22T21:12:10.075 DEBUG:boto:Headers: {} 2023-09-22T21:12:10.075 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8001 2023-09-22T21:12:10.075 DEBUG:boto:Port: 8001 2023-09-22T21:12:10.075 DEBUG:boto:Params: {} 2023-09-22T21:12:10.075 DEBUG:boto:Token: None 2023-09-22T21:12:10.076 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:12:10 GMT /ffzdqu-30/?cors 2023-09-22T21:12:10.076 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yxU1RLnZJPdqTf68IUna1ogp95U= 2023-09-22T21:12:10.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yxU1RLnZJPdqTf68IUna1ogp95U=', 'Content-Length': '0'} 2023-09-22T21:12:10.077 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000048e423a8503297ec-00650e032a-1110-b2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '266'), ('Date', 'Fri, 22 Sep 2023 21:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:10.122 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-09-22T21:12:10.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2023-09-22T21:12:10.123 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-31 2023-09-22T21:12:10.123 DEBUG:boto:path=/ffzdqu-31/ 2023-09-22T21:12:10.123 DEBUG:boto:auth_path=/ffzdqu-31/ 2023-09-22T21:12:10.124 DEBUG:boto:Method: PUT 2023-09-22T21:12:10.124 DEBUG:boto:Path: /ffzdqu-31/ 2023-09-22T21:12:10.124 DEBUG:boto:Data: 2023-09-22T21:12:10.124 DEBUG:boto:Headers: {} 2023-09-22T21:12:10.124 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:10.124 DEBUG:boto:Port: 8000 2023-09-22T21:12:10.124 DEBUG:boto:Params: {} 2023-09-22T21:12:10.124 DEBUG:boto:Token: None 2023-09-22T21:12:10.124 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:10 GMT /ffzdqu-31/ 2023-09-22T21:12:10.125 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2wcJZQ1UxDW+dMUUHp5s+5hSf6Y= 2023-09-22T21:12:10.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2wcJZQ1UxDW+dMUUHp5s+5hSf6Y=', 'Content-Length': '0'} 2023-09-22T21:12:10.163 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c5c26ed3a8ed9ef9-00650e032a-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:10.164 INFO:rgw_multi.tests:create bucket zone=a2 name=ffzdqu-32 2023-09-22T21:12:10.164 DEBUG:boto:path=/ffzdqu-32/ 2023-09-22T21:12:10.164 DEBUG:boto:auth_path=/ffzdqu-32/ 2023-09-22T21:12:10.164 DEBUG:boto:Method: PUT 2023-09-22T21:12:10.164 DEBUG:boto:Path: /ffzdqu-32/ 2023-09-22T21:12:10.164 DEBUG:boto:Data: 2023-09-22T21:12:10.164 DEBUG:boto:Headers: {} 2023-09-22T21:12:10.164 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:12:10.164 DEBUG:boto:Port: 8001 2023-09-22T21:12:10.165 DEBUG:boto:Params: {} 2023-09-22T21:12:10.165 DEBUG:boto:Token: None 2023-09-22T21:12:10.165 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:10 GMT /ffzdqu-32/ 2023-09-22T21:12:10.165 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q+4YTGS5I1PXFHWliZK//5Zjmjg= 2023-09-22T21:12:10.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q+4YTGS5I1PXFHWliZK//5Zjmjg=', 'Content-Length': '0'} 2023-09-22T21:12:10.233 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003b5912fb3c76ef5f-00650e032a-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:10.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:10.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:10.234 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:10.292 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:10.292 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:10.334 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:12:10.352 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695417130.209786_183.1","last_update":"2023-09-22T21:12:10.209786Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:12:10.353 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417130.209786_183.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:12:10.353 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:12:10.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:10.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:10.353 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:10.398 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:10.398 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:10.437 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:12:10.440+0000 7fc4084d0a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:12:10.438 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:12:10.440+0000 7fc4084d0a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:12:10.455 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417126.921434_167.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:06.921434Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:12:10Z"} 2023-09-22T21:12:10.455 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417126.921434_167.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:06.921434Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:12:10Z"} 2023-09-22T21:12:10.455 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417126.921434_167.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:06.921434Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:12:10.455 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417130.209786_183.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:12:10.455 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417126.921434_167.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:12:10.455 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417130.209786_183.1 target=1_1695417126.921434_167.1 2023-09-22T21:12:15.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:15.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:15.457 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:15.501 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:15.501 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:15.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:12:15.540+0000 7fd3f52d9a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:12:15.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:12:15.544+0000 7fd3f52d9a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:12:15.558 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417126.953335_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:06.953335Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:12:15Z"} 2023-09-22T21:12:15.558 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417126.953335_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:06.953335Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:12:15Z"} 2023-09-22T21:12:15.558 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417126.953335_168.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:06.953335Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:12:15.558 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417130.209786_183.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:12:15.559 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417126.953335_168.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:12:15.559 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417130.209786_183.1 target=1_1695417126.953335_168.1 2023-09-22T21:12:20.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:20.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:20.560 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:20.603 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:20.603 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:20.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:12:20.644+0000 7f06d5f18a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:12:20.644 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:12:20.648+0000 7f06d5f18a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:12:20.662 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417130.209786_183.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:10.209786Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:12:20Z"} 2023-09-22T21:12:20.662 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417130.209786_183.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:10.209786Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:12:20Z"} 2023-09-22T21:12:20.662 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417130.209786_183.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:10.209786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:12:20.662 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417130.209786_183.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:12:20.663 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417130.209786_183.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:12:20.663 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:12:20.663 DEBUG:boto:path=/ffzdqu-31/ 2023-09-22T21:12:20.663 DEBUG:boto:auth_path=/ffzdqu-31/ 2023-09-22T21:12:20.663 DEBUG:boto:Method: HEAD 2023-09-22T21:12:20.663 DEBUG:boto:Path: /ffzdqu-31/ 2023-09-22T21:12:20.663 DEBUG:boto:Data: 2023-09-22T21:12:20.663 DEBUG:boto:Headers: {} 2023-09-22T21:12:20.664 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:20.664 DEBUG:boto:Port: 8000 2023-09-22T21:12:20.664 DEBUG:boto:Params: {} 2023-09-22T21:12:20.664 DEBUG:boto:Token: None 2023-09-22T21:12:20.664 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:12:20 GMT /ffzdqu-31/ 2023-09-22T21:12:20.664 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dIOpJSPe84D8JJHOABeJeVZccjY= 2023-09-22T21:12:20.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dIOpJSPe84D8JJHOABeJeVZccjY=', 'Content-Length': '0'} 2023-09-22T21:12:20.667 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000bff140f7f0008ef-00650e0334-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:20.667 DEBUG:boto:path=/ffzdqu-31/foo 2023-09-22T21:12:20.667 DEBUG:boto:auth_path=/ffzdqu-31/foo 2023-09-22T21:12:20.667 DEBUG:boto:Method: PUT 2023-09-22T21:12:20.667 DEBUG:boto:Path: /ffzdqu-31/foo 2023-09-22T21:12:20.668 DEBUG:boto:Data: 2023-09-22T21:12:20.668 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:12:20.668 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:20.668 DEBUG:boto:Port: 8000 2023-09-22T21:12:20.668 DEBUG:boto:Params: {} 2023-09-22T21:12:20.668 DEBUG:boto:Token: None 2023-09-22T21:12:20.668 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Fri, 22 Sep 2023 21:12:20 GMT /ffzdqu-31/foo 2023-09-22T21:12:20.668 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g1oKJXaXxINsMRSxUmzRi29096s= 2023-09-22T21:12:20.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:12:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g1oKJXaXxINsMRSxUmzRi29096s='} 2023-09-22T21:12:20.677 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001fdfaceaac249f46-00650e0334-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:20.677 DEBUG:boto:path=/ffzdqu-31/ 2023-09-22T21:12:20.677 DEBUG:boto:auth_path=/ffzdqu-31/ 2023-09-22T21:12:20.678 DEBUG:boto:Method: DELETE 2023-09-22T21:12:20.678 DEBUG:boto:Path: /ffzdqu-31/ 2023-09-22T21:12:20.678 DEBUG:boto:Data: 2023-09-22T21:12:20.678 DEBUG:boto:Headers: {} 2023-09-22T21:12:20.678 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:20.678 DEBUG:boto:Port: 8000 2023-09-22T21:12:20.678 DEBUG:boto:Params: {} 2023-09-22T21:12:20.678 DEBUG:boto:Token: None 2023-09-22T21:12:20.678 DEBUG:boto:StringToSign: DELETE Fri, 22 Sep 2023 21:12:20 GMT /ffzdqu-31/ 2023-09-22T21:12:20.679 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PXoNNRrlau7x0qRoF1LdU6iKeFg= 2023-09-22T21:12:20.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PXoNNRrlau7x0qRoF1LdU6iKeFg=', 'Content-Length': '0'} 2023-09-22T21:12:20.686 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx00000582b9a39cc87a3cd-00650e0334-111c-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:20.687 DEBUG:boto:path=/ffzdqu-32/ 2023-09-22T21:12:20.687 DEBUG:boto:auth_path=/ffzdqu-32/ 2023-09-22T21:12:20.687 DEBUG:boto:Method: HEAD 2023-09-22T21:12:20.687 DEBUG:boto:Path: /ffzdqu-32/ 2023-09-22T21:12:20.688 DEBUG:boto:Data: 2023-09-22T21:12:20.688 DEBUG:boto:Headers: {} 2023-09-22T21:12:20.688 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:12:20.688 DEBUG:boto:Port: 8001 2023-09-22T21:12:20.688 DEBUG:boto:Params: {} 2023-09-22T21:12:20.688 DEBUG:boto:Token: None 2023-09-22T21:12:20.688 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:12:20 GMT /ffzdqu-32/ 2023-09-22T21:12:20.688 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J1SISKI8CAMsW19UvRoHDuW9uAc= 2023-09-22T21:12:20.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J1SISKI8CAMsW19UvRoHDuW9uAc=', 'Content-Length': '0'} 2023-09-22T21:12:20.691 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000284d398eaac73b53-00650e0334-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:20.691 DEBUG:boto:path=/ffzdqu-32/foo 2023-09-22T21:12:20.691 DEBUG:boto:auth_path=/ffzdqu-32/foo 2023-09-22T21:12:20.691 DEBUG:boto:Method: PUT 2023-09-22T21:12:20.691 DEBUG:boto:Path: /ffzdqu-32/foo 2023-09-22T21:12:20.691 DEBUG:boto:Data: 2023-09-22T21:12:20.691 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:12:20.692 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:12:20.692 DEBUG:boto:Port: 8001 2023-09-22T21:12:20.692 DEBUG:boto:Params: {} 2023-09-22T21:12:20.692 DEBUG:boto:Token: None 2023-09-22T21:12:20.692 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Fri, 22 Sep 2023 21:12:20 GMT /ffzdqu-32/foo 2023-09-22T21:12:20.692 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BKwOe+5W3SkDoKKzGsCvrkDeeUQ= 2023-09-22T21:12:20.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:12:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BKwOe+5W3SkDoKKzGsCvrkDeeUQ='} 2023-09-22T21:12:20.734 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008671dfac2262012a-00650e0334-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:20.735 DEBUG:boto:path=/ffzdqu-32/ 2023-09-22T21:12:20.735 DEBUG:boto:auth_path=/ffzdqu-32/ 2023-09-22T21:12:20.735 DEBUG:boto:Method: DELETE 2023-09-22T21:12:20.735 DEBUG:boto:Path: /ffzdqu-32/ 2023-09-22T21:12:20.735 DEBUG:boto:Data: 2023-09-22T21:12:20.735 DEBUG:boto:Headers: {} 2023-09-22T21:12:20.735 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:12:20.736 DEBUG:boto:Port: 8001 2023-09-22T21:12:20.736 DEBUG:boto:Params: {} 2023-09-22T21:12:20.736 DEBUG:boto:Token: None 2023-09-22T21:12:20.736 DEBUG:boto:StringToSign: DELETE Fri, 22 Sep 2023 21:12:20 GMT /ffzdqu-32/ 2023-09-22T21:12:20.736 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SyYmHpRpAORs0DRWB7+Zl6ynfSg= 2023-09-22T21:12:20.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SyYmHpRpAORs0DRWB7+Zl6ynfSg=', 'Content-Length': '0'} 2023-09-22T21:12:20.747 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx00000a4806b16c3ee2ded-00650e0334-1120-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:20.747 DEBUG:boto:path=/ffzdqu-31/ 2023-09-22T21:12:20.747 DEBUG:boto:auth_path=/ffzdqu-31/ 2023-09-22T21:12:20.747 DEBUG:boto:Method: HEAD 2023-09-22T21:12:20.748 DEBUG:boto:Path: /ffzdqu-31/ 2023-09-22T21:12:20.748 DEBUG:boto:Data: 2023-09-22T21:12:20.748 DEBUG:boto:Headers: {} 2023-09-22T21:12:20.748 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:20.748 DEBUG:boto:Port: 8000 2023-09-22T21:12:20.748 DEBUG:boto:Params: {} 2023-09-22T21:12:20.748 DEBUG:boto:Token: None 2023-09-22T21:12:20.749 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:12:20 GMT /ffzdqu-31/ 2023-09-22T21:12:20.749 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dIOpJSPe84D8JJHOABeJeVZccjY= 2023-09-22T21:12:20.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dIOpJSPe84D8JJHOABeJeVZccjY=', 'Content-Length': '0'} 2023-09-22T21:12:20.752 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083bf7aa4111db216-00650e0334-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:20.752 DEBUG:boto:path=/ffzdqu-32/ 2023-09-22T21:12:20.752 DEBUG:boto:auth_path=/ffzdqu-32/ 2023-09-22T21:12:20.753 DEBUG:boto:Method: HEAD 2023-09-22T21:12:20.753 DEBUG:boto:Path: /ffzdqu-32/ 2023-09-22T21:12:20.753 DEBUG:boto:Data: 2023-09-22T21:12:20.753 DEBUG:boto:Headers: {} 2023-09-22T21:12:20.753 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:20.753 DEBUG:boto:Port: 8000 2023-09-22T21:12:20.753 DEBUG:boto:Params: {} 2023-09-22T21:12:20.754 DEBUG:boto:Token: None 2023-09-22T21:12:20.754 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:12:20 GMT /ffzdqu-32/ 2023-09-22T21:12:20.754 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J1SISKI8CAMsW19UvRoHDuW9uAc= 2023-09-22T21:12:20.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J1SISKI8CAMsW19UvRoHDuW9uAc=', 'Content-Length': '0'} 2023-09-22T21:12:20.757 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000273cb13c663ddc88-00650e0334-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:20.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... ok 2023-09-22T21:12:20.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... SKIP: test_multi_period_incremental_sync skipped. Requires 3 or more zones in master zonegroup. 2023-09-22T21:12:20.759 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-33 2023-09-22T21:12:20.759 DEBUG:boto:path=/ffzdqu-33/ 2023-09-22T21:12:20.759 DEBUG:boto:auth_path=/ffzdqu-33/ 2023-09-22T21:12:20.759 DEBUG:boto:Method: PUT 2023-09-22T21:12:20.759 DEBUG:boto:Path: /ffzdqu-33/ 2023-09-22T21:12:20.760 DEBUG:boto:Data: 2023-09-22T21:12:20.760 DEBUG:boto:Headers: {} 2023-09-22T21:12:20.760 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:20.760 DEBUG:boto:Port: 8000 2023-09-22T21:12:20.760 DEBUG:boto:Params: {} 2023-09-22T21:12:20.760 DEBUG:boto:Token: None 2023-09-22T21:12:20.760 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:20 GMT /ffzdqu-33/ 2023-09-22T21:12:20.761 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ix2HICeDL/ihey7rrINdDPJn8kU= 2023-09-22T21:12:20.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ix2HICeDL/ihey7rrINdDPJn8kU=', 'Content-Length': '0'} 2023-09-22T21:12:20.788 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000075b80abd34b3d445-00650e0334-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:20.788 INFO:rgw_multi.tests:create bucket zone=a2 name=ffzdqu-34 2023-09-22T21:12:20.788 DEBUG:boto:path=/ffzdqu-34/ 2023-09-22T21:12:20.788 DEBUG:boto:auth_path=/ffzdqu-34/ 2023-09-22T21:12:20.789 DEBUG:boto:Method: PUT 2023-09-22T21:12:20.789 DEBUG:boto:Path: /ffzdqu-34/ 2023-09-22T21:12:20.789 DEBUG:boto:Data: 2023-09-22T21:12:20.789 DEBUG:boto:Headers: {} 2023-09-22T21:12:20.789 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:12:20.789 DEBUG:boto:Port: 8001 2023-09-22T21:12:20.789 DEBUG:boto:Params: {} 2023-09-22T21:12:20.789 DEBUG:boto:Token: None 2023-09-22T21:12:20.789 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:12:20 GMT /ffzdqu-34/ 2023-09-22T21:12:20.790 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kB/dwYdTls245GnD9uNzgDALDug= 2023-09-22T21:12:20.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kB/dwYdTls245GnD9uNzgDALDug=', 'Content-Length': '0'} 2023-09-22T21:12:20.845 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006026a34610b9af1a-00650e0334-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:20.846 DEBUG:boto:path=/ffzdqu-33/ 2023-09-22T21:12:20.846 DEBUG:boto:auth_path=/ffzdqu-33/ 2023-09-22T21:12:20.846 DEBUG:boto:Method: HEAD 2023-09-22T21:12:20.846 DEBUG:boto:Path: /ffzdqu-33/ 2023-09-22T21:12:20.846 DEBUG:boto:Data: 2023-09-22T21:12:20.846 DEBUG:boto:Headers: {} 2023-09-22T21:12:20.846 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:20.846 DEBUG:boto:Port: 8000 2023-09-22T21:12:20.847 DEBUG:boto:Params: {} 2023-09-22T21:12:20.847 DEBUG:boto:Token: None 2023-09-22T21:12:20.847 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:12:20 GMT /ffzdqu-33/ 2023-09-22T21:12:20.847 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PD5ulM+SdsAtw0goJsLlO2RQYFs= 2023-09-22T21:12:20.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PD5ulM+SdsAtw0goJsLlO2RQYFs=', 'Content-Length': '0'} 2023-09-22T21:12:20.850 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4a265cf510e698c-00650e0334-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:20.851 DEBUG:boto:path=/ffzdqu-33/key 2023-09-22T21:12:20.851 DEBUG:boto:auth_path=/ffzdqu-33/key 2023-09-22T21:12:20.851 DEBUG:boto:Method: PUT 2023-09-22T21:12:20.851 DEBUG:boto:Path: /ffzdqu-33/key 2023-09-22T21:12:20.851 DEBUG:boto:Data: 2023-09-22T21:12:20.851 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-09-22T21:12:20.852 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:12:20.852 DEBUG:boto:Port: 8000 2023-09-22T21:12:20.852 DEBUG:boto:Params: {} 2023-09-22T21:12:20.852 DEBUG:boto:Token: None 2023-09-22T21:12:20.852 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 22 Sep 2023 21:12:20 GMT /ffzdqu-33/key 2023-09-22T21:12:20.852 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0Zo+GbXdPjR+slvXisoFY6/1sOA= 2023-09-22T21:12:20.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:12:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0Zo+GbXdPjR+slvXisoFY6/1sOA='} 2023-09-22T21:12:20.863 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f88ab0ddd09c8b3f-00650e0334-111c-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:20.863 DEBUG:boto:path=/ffzdqu-34/ 2023-09-22T21:12:20.863 DEBUG:boto:auth_path=/ffzdqu-34/ 2023-09-22T21:12:20.863 DEBUG:boto:Method: HEAD 2023-09-22T21:12:20.863 DEBUG:boto:Path: /ffzdqu-34/ 2023-09-22T21:12:20.863 DEBUG:boto:Data: 2023-09-22T21:12:20.863 DEBUG:boto:Headers: {} 2023-09-22T21:12:20.864 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:12:20.864 DEBUG:boto:Port: 8001 2023-09-22T21:12:20.864 DEBUG:boto:Params: {} 2023-09-22T21:12:20.864 DEBUG:boto:Token: None 2023-09-22T21:12:20.864 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:12:20 GMT /ffzdqu-34/ 2023-09-22T21:12:20.864 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QmzGhDupbdX8Ev0BYkowou0NvgQ= 2023-09-22T21:12:20.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:12:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QmzGhDupbdX8Ev0BYkowou0NvgQ=', 'Content-Length': '0'} 2023-09-22T21:12:20.868 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f09ab0a9352154c8-00650e0334-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:20.868 DEBUG:boto:path=/ffzdqu-34/key 2023-09-22T21:12:20.869 DEBUG:boto:auth_path=/ffzdqu-34/key 2023-09-22T21:12:20.869 DEBUG:boto:Method: PUT 2023-09-22T21:12:20.869 DEBUG:boto:Path: /ffzdqu-34/key 2023-09-22T21:12:20.869 DEBUG:boto:Data: 2023-09-22T21:12:20.869 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-09-22T21:12:20.869 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:12:20.869 DEBUG:boto:Port: 8001 2023-09-22T21:12:20.869 DEBUG:boto:Params: {} 2023-09-22T21:12:20.869 DEBUG:boto:Token: None 2023-09-22T21:12:20.870 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 22 Sep 2023 21:12:20 GMT /ffzdqu-34/key 2023-09-22T21:12:20.870 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WPNgDRXtstgnNA4RimwJ9NKokh0= 2023-09-22T21:12:20.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:12:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WPNgDRXtstgnNA4RimwJ9NKokh0='} 2023-09-22T21:12:20.914 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006c016a97f1f64ef7-00650e0334-1120-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:12:20.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:20.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:20.915 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:20.953 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:20.953 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:20.992 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:12:21.009 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695417140.821551_197.1","last_update":"2023-09-22T21:12:20.821551Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:12:21.010 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417140.821551_197.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:12:21.010 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:12:21.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:21.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:21.010 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:21.089 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:21.089 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:21.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:12:21.128+0000 7fad43edaa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:12:21.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:12:21.128+0000 7fad43edaa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:12:21.142 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.778608_190.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.778608Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:12:21Z"} 2023-09-22T21:12:21.142 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.778608_190.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.778608Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:12:21Z"} 2023-09-22T21:12:21.142 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417140.778608_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:20.778608Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:12:21.142 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417140.821551_197.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:12:21.142 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417140.778608_190.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:12:21.142 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417140.821551_197.1 target=1_1695417140.778608_190.1 2023-09-22T21:12:26.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:26.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:26.144 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:26.188 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:26.188 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:26.224 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:12:26.228+0000 7f86df826a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:12:26.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:12:26.228+0000 7f86df826a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:12:26.242 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:12:26Z"} 2023-09-22T21:12:26.243 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:12:26Z"} 2023-09-22T21:12:26.243 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417140.821551_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:20.821551Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:12:26.243 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417140.821551_197.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:12:26.243 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417140.821551_197.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:12:26.243 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:12:26.243 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-09-22T21:12:26.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:26.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:26.244 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:26.321 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:26.321 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:26.382 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"00000000000000000000:00000000000000000651","last_update":"2023-09-22T21:04:42.948773Z"},{"marker":"00000000000000000000:00000000000000003324","last_update":"2023-09-22T21:12:21.905803Z"},{"marker":"00000000000000000000:00000000000000002204","last_update":"2023-09-22T21:08:19.579516Z"},{"marker":"00000000000000000000:00000000000000001210","last_update":"2023-09-22T21:12:22.975904Z"}] 2023-09-22T21:12:26.383 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003324', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000001210'} 2023-09-22T21:12:26.383 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-22T21:12:26.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:26.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:26.384 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:26.460 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:26.461 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:26.515 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:42.948773Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003042","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:11:41.869437Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:08:19.579516Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001210","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:22.975904Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:12:26Z"} 2023-09-22T21:12:26.515 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:42.948773Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003042","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:11:41.869437Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:08:19.579516Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001210","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:22.975904Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:12:26Z"} 2023-09-22T21:12:26.516 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:04:42.948773Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003042', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:11:41.869437Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:08:19.579516Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:22.975904Z'}}] 2023-09-22T21:12:26.516 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003324', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000001210'} 2023-09-22T21:12:26.516 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003042', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000001210'} 2023-09-22T21:12:26.516 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000003324 target=00000000000000000000:00000000000000003042 2023-09-22T21:12:31.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:31.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:31.518 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:31.564 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:31.564 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:31.620 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:42.948773Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003042","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:11:41.869437Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:08:19.579516Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001210","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:22.975904Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:12:31Z"} 2023-09-22T21:12:31.620 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:42.948773Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003042","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:11:41.869437Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:08:19.579516Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001210","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:22.975904Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:12:31Z"} 2023-09-22T21:12:31.620 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:04:42.948773Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003042', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:11:41.869437Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:08:19.579516Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:22.975904Z'}}] 2023-09-22T21:12:31.621 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003324', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000001210'} 2023-09-22T21:12:31.621 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003042', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000001210'} 2023-09-22T21:12:31.621 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000003324 target=00000000000000000000:00000000000000003042 2023-09-22T21:12:36.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:36.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:36.622 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:36.669 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:36.669 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:36.724 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:42.948773Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003042","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:11:41.869437Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:08:19.579516Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001210","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:22.975904Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:12:36Z"} 2023-09-22T21:12:36.724 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:42.948773Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003042","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:11:41.869437Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:08:19.579516Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001210","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:22.975904Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:12:36Z"} 2023-09-22T21:12:36.725 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:04:42.948773Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003042', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:11:41.869437Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:08:19.579516Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:22.975904Z'}}] 2023-09-22T21:12:36.725 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003324', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000001210'} 2023-09-22T21:12:36.725 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003042', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000001210'} 2023-09-22T21:12:36.725 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000003324 target=00000000000000000000:00000000000000003042 2023-09-22T21:12:41.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:41.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:41.726 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:41.778 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:41.778 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:41.842 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:42.948773Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003324","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:21.905803Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:08:19.579516Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001210","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:22.975904Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:12:41Z"} 2023-09-22T21:12:41.842 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:42.948773Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003324","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:21.905803Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:08:19.579516Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001210","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:22.975904Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:12:41Z"} 2023-09-22T21:12:41.842 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:04:42.948773Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003324', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:21.905803Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:08:19.579516Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:22.975904Z'}}] 2023-09-22T21:12:41.842 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003324', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000001210'} 2023-09-22T21:12:41.842 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003324', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000001210'} 2023-09-22T21:12:41.842 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-22T21:12:41.843 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-09-22T21:12:41.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:41.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:41.843 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:41.877 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:41.877 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:41.935 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"00000000000000000000:00000000000000000651","last_update":"2023-09-22T21:04:39.605415Z"},{"marker":"00000000000000000000:00000000000000004029","last_update":"2023-09-22T21:12:40.611546Z"},{"marker":"00000000000000000000:00000000000000001499","last_update":"2023-09-22T21:07:59.650851Z"},{"marker":"00000000000000000000:00000000000000001210","last_update":"2023-09-22T21:12:21.945596Z"}] 2023-09-22T21:12:41.936 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004029', 2: '00000000000000000000:00000000000000001499', 3: '00000000000000000000:00000000000000001210'} 2023-09-22T21:12:41.936 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-09-22T21:12:41.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:41.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:41.936 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:42.021 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:42.021 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:42.076 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10476880264182873081},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:39.605415Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004029","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:40.611546Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001499","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:07:59.650851Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001210","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:21.945596Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:12:42Z"} 2023-09-22T21:12:42.077 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10476880264182873081},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:39.605415Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004029","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:40.611546Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001499","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:07:59.650851Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001210","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:21.945596Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:12:42Z"} 2023-09-22T21:12:42.077 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:04:39.605415Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004029', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:40.611546Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:07:59.650851Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:21.945596Z'}}] 2023-09-22T21:12:42.077 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004029', 2: '00000000000000000000:00000000000000001499', 3: '00000000000000000000:00000000000000001210'} 2023-09-22T21:12:42.077 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004029', 2: '00000000000000000000:00000000000000001499', 3: '00000000000000000000:00000000000000001210'} 2023-09-22T21:12:42.077 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-09-22T21:12:42.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:42.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:42.078 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:42.152 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:42.187 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:42.212 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"00000000000000000000:00000000000000000651","last_update":"2023-09-22T21:04:42.948773Z"},{"marker":"00000000000000000000:00000000000000003324","last_update":"2023-09-22T21:12:21.905803Z"},{"marker":"00000000000000000000:00000000000000002204","last_update":"2023-09-22T21:08:19.579516Z"},{"marker":"00000000000000000000:00000000000000001210","last_update":"2023-09-22T21:12:22.975904Z"}] 2023-09-22T21:12:42.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:12:42.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:12:42.213 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:12:42.286 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:42.286 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:42.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:42.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:42.352 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:42.431 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:42.431 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:42.485 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2023-09-22T21:12:42.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:42.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:42.486 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:42.562 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:42.563 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:42.617 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2023-09-22T21:12:42.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:42.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:42.618 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:42.696 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:42.696 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:42.752 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2023-09-22T21:12:42.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:42.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:42.753 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:42.828 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:42.829 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:42.879 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2023-09-22T21:12:42.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:42.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:42.880 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:42.946 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:42.946 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:42.998 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"00000000000000000000:00000000000000000651","last_update":"2023-09-22T21:04:42.948773Z"},{"marker":"00000000000000000000:00000000000000003324","last_update":"2023-09-22T21:12:21.905803Z"},{"marker":"00000000000000000000:00000000000000002204","last_update":"2023-09-22T21:08:19.579516Z"},{"marker":"00000000000000000000:00000000000000001210","last_update":"2023-09-22T21:12:22.975904Z"}] 2023-09-22T21:12:42.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:12:42.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:12:42.999 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:12:43.067 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:43.067 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:43.126 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:43.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:43.126 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:43.197 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:43.197 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:43.243 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2023-09-22T21:12:43.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:43.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:43.243 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:43.314 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:43.314 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:43.361 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2023-09-22T21:12:43.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:43.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:43.362 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:43.435 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:43.435 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:43.481 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2023-09-22T21:12:43.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:43.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:12:43.481 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:12:43.563 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:43.563 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:43.610 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2023-09-22T21:12:43.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... ok 2023-09-22T21:12:43.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:12:43.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:12:43.611 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=false --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:12:43.681 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:43.681 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:43.705 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"2b721fee-a149-45da-925b-502d5a3d6853","name":"a2","domain_root":"a2.rgw.meta:root","control_pool":"a2.rgw.control","gc_pool":"a2.rgw.log:gc","lc_pool":"a2.rgw.log:lc","log_pool":"a2.rgw.log","intent_log_pool":"a2.rgw.log:intent","usage_log_pool":"a2.rgw.log:usage","roles_pool":"a2.rgw.meta:roles","reshard_pool":"a2.rgw.log:reshard","user_keys_pool":"a2.rgw.meta:users.keys","user_email_pool":"a2.rgw.meta:users.email","user_swift_pool":"a2.rgw.meta:users.swift","user_uid_pool":"a2.rgw.meta:users.uid","otp_pool":"a2.rgw.otp","system_key":{"access_key":"KT50DM2YGPMJ8OAR","secret_key":"lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"a2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"a2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"a2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","notif_pool":"a2.rgw.log:notif"} 2023-09-22T21:12:43.705 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:12:43.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:12:43.705 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:12:43.782 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:12:43.782 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:12:43.830 INFO:teuthology.orchestra.run.smithi160.stderr:Sending period to new master zone f132bdb9-20a7-4073-a932-8f99da20691e 2023-09-22T21:12:43.879 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","epoch":5,"predecessor_uuid":"a6099785-5160-4a69-b202-552a1b9cb770","sync_status":[],"period_map":{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","zonegroups":[{"id":"a","name":"a","api_name":"","is_master":true,"endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","zones":[{"id":"2b721fee-a149-45da-925b-502d5a3d6853","name":"a2","endpoints":["http://smithi160.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":false,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"f132bdb9-20a7-4073-a932-8f99da20691e","name":"a1","endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]},{"id":"b","name":"b","api_name":"","is_master":false,"endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","zones":[{"id":"087d4b41-8f81-4835-9e99-026ca13a7b60","name":"b2","endpoints":["http://smithi204.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","name":"b1","endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"087d4b41-8f81-4835-9e99-026ca13a7b60","val":2135214481},{"key":"2b721fee-a149-45da-925b-502d5a3d6853","val":3586193783},{"key":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","val":4184768938},{"key":"f132bdb9-20a7-4073-a932-8f99da20691e","val":3405200286}]},"master_zonegroup":"a","master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","realm_name":"test-realm","realm_epoch":2} 2023-09-22T21:12:43.879 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to false 2023-09-22T21:13:43.880 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-35 2023-09-22T21:13:43.881 DEBUG:boto:path=/ffzdqu-35/ 2023-09-22T21:13:43.881 DEBUG:boto:auth_path=/ffzdqu-35/ 2023-09-22T21:13:43.881 DEBUG:boto:Method: PUT 2023-09-22T21:13:43.882 DEBUG:boto:Path: /ffzdqu-35/ 2023-09-22T21:13:43.882 DEBUG:boto:Data: 2023-09-22T21:13:43.882 DEBUG:boto:Headers: {} 2023-09-22T21:13:43.882 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:13:43.882 DEBUG:boto:Port: 8000 2023-09-22T21:13:43.882 DEBUG:boto:Params: {} 2023-09-22T21:13:43.882 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:13:43.883 DEBUG:boto:Token: None 2023-09-22T21:13:43.883 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:13:43 GMT /ffzdqu-35/ 2023-09-22T21:13:43.883 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:87bDA7c1OukyXWIYjzYdChJXYM4= 2023-09-22T21:13:43.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:13:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:87bDA7c1OukyXWIYjzYdChJXYM4=', 'Content-Length': '0'} 2023-09-22T21:13:43.928 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b2b200833dba34ff-00650e0387-1514-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:13:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:13:43.929 DEBUG:boto:path=/ffzdqu-35/testredirect 2023-09-22T21:13:43.929 DEBUG:boto:auth_path=/ffzdqu-35/testredirect 2023-09-22T21:13:43.929 DEBUG:boto:Method: PUT 2023-09-22T21:13:43.929 DEBUG:boto:Path: /ffzdqu-35/testredirect 2023-09-22T21:13:43.929 DEBUG:boto:Data: 2023-09-22T21:13:43.929 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-09-22T21:13:43.929 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:13:43.930 DEBUG:boto:Port: 8000 2023-09-22T21:13:43.930 DEBUG:boto:Params: {} 2023-09-22T21:13:43.930 DEBUG:boto:Token: None 2023-09-22T21:13:43.930 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Fri, 22 Sep 2023 21:13:43 GMT /ffzdqu-35/testredirect 2023-09-22T21:13:43.930 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EbbKxNMM1hrjfh7gIWNW1wSWSNw= 2023-09-22T21:13:43.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:13:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EbbKxNMM1hrjfh7gIWNW1wSWSNw='} 2023-09-22T21:13:43.949 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000221d6de6ebbf5e6a-00650e0387-1514-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:13:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:13:43.949 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:13:43.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:13:43.950 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:13:43.991 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:13:43.991 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:13:44.029 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:13:44.047 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695417223.928828_219.1","last_update":"2023-09-22T21:13:43.928828Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:13:44.048 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:13:44.048 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:13:44.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:13:44.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:13:44.048 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:13:44.132 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:13:44.132 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:13:44.173 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:13:44.176+0000 7fefe190fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:13:44.174 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:13:44.176+0000 7fefe190fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:13:44.191 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:13:44Z"} 2023-09-22T21:13:44.191 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:13:44Z"} 2023-09-22T21:13:44.193 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417140.821551_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:20.821551Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:13:44.193 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:13:44.193 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417140.821551_197.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:13:44.193 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417223.928828_219.1 target=1_1695417140.821551_197.1 2023-09-22T21:13:49.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:13:49.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:13:49.195 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:13:49.237 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:13:49.238 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:13:49.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:13:49.275+0000 7f282ec51a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:13:49.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:13:49.275+0000 7f282ec51a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:13:49.292 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:13:49Z"} 2023-09-22T21:13:49.292 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:13:49Z"} 2023-09-22T21:13:49.292 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417140.821551_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:20.821551Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:13:49.292 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:13:49.293 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417140.821551_197.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:13:49.293 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417223.928828_219.1 target=1_1695417140.821551_197.1 2023-09-22T21:13:54.294 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:13:54.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:13:54.294 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:13:54.337 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:13:54.338 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:13:54.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:13:54.379+0000 7f4c86b85a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:13:54.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:13:54.379+0000 7f4c86b85a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:13:54.394 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:13:54Z"} 2023-09-22T21:13:54.394 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:13:54Z"} 2023-09-22T21:13:54.394 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417140.821551_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:20.821551Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:13:54.394 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:13:54.395 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417140.821551_197.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:13:54.395 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417223.928828_219.1 target=1_1695417140.821551_197.1 2023-09-22T21:13:59.396 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:13:59.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:13:59.396 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:13:59.439 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:13:59.440 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:13:59.476 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:13:59.479+0000 7f9766e54a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:13:59.477 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:13:59.479+0000 7f9766e54a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:13:59.495 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:13:59Z"} 2023-09-22T21:13:59.495 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:13:59Z"} 2023-09-22T21:13:59.495 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417140.821551_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:20.821551Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:13:59.495 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:13:59.495 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417140.821551_197.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:13:59.496 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417223.928828_219.1 target=1_1695417140.821551_197.1 2023-09-22T21:14:04.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:14:04.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:14:04.497 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:14:04.541 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:14:04.541 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:14:04.578 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:14:04.579+0000 7f9596b1ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:14:04.579 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:14:04.579+0000 7f9596b1ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:14:04.596 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:14:04Z"} 2023-09-22T21:14:04.596 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:14:04Z"} 2023-09-22T21:14:04.596 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417140.821551_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:20.821551Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:14:04.597 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:14:04.597 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417140.821551_197.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:14:04.597 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417223.928828_219.1 target=1_1695417140.821551_197.1 2023-09-22T21:14:09.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:14:09.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:14:09.598 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:14:09.641 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:14:09.642 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:14:09.678 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:14:09.679+0000 7f9ada625a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:14:09.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:14:09.679+0000 7f9ada625a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:14:09.696 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:14:09Z"} 2023-09-22T21:14:09.696 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:14:09Z"} 2023-09-22T21:14:09.696 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417140.821551_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:20.821551Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:14:09.697 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:14:09.697 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417140.821551_197.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:14:09.697 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417223.928828_219.1 target=1_1695417140.821551_197.1 2023-09-22T21:14:14.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:14:14.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:14:14.698 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:14:14.741 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:14:14.741 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:14:14.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:14:14.779+0000 7fc9c3ec2a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:14:14.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:14:14.783+0000 7fc9c3ec2a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:14:14.798 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:14:14Z"} 2023-09-22T21:14:14.799 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:14:14Z"} 2023-09-22T21:14:14.799 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417140.821551_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:20.821551Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:14:14.799 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:14:14.799 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417140.821551_197.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:14:14.799 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417223.928828_219.1 target=1_1695417140.821551_197.1 2023-09-22T21:14:19.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:14:19.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:14:19.800 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:14:19.842 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:14:19.842 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:14:19.882 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:14:19.883+0000 7f2253e44a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:14:19.883 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:14:19.883+0000 7f2253e44a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:14:19.901 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:14:19Z"} 2023-09-22T21:14:19.902 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:14:19Z"} 2023-09-22T21:14:19.902 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417140.821551_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:20.821551Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:14:19.902 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:14:19.902 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417140.821551_197.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:14:19.902 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417223.928828_219.1 target=1_1695417140.821551_197.1 2023-09-22T21:14:24.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:14:24.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:14:24.904 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:14:24.947 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:14:24.947 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:14:24.984 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:14:24.987+0000 7f6a9d982a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:14:24.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:14:24.987+0000 7f6a9d982a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:14:25.002 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:14:24Z"} 2023-09-22T21:14:25.002 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:14:24Z"} 2023-09-22T21:14:25.002 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417140.821551_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:20.821551Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:14:25.002 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:14:25.003 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417140.821551_197.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:14:25.003 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417223.928828_219.1 target=1_1695417140.821551_197.1 2023-09-22T21:14:30.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:14:30.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:14:30.004 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:14:30.047 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:14:30.047 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:14:30.084 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:14:30.087+0000 7fd06f792a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:14:30.085 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:14:30.087+0000 7fd06f792a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:14:30.102 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:14:30Z"} 2023-09-22T21:14:30.102 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:14:30Z"} 2023-09-22T21:14:30.102 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417140.821551_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:20.821551Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:14:30.103 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:14:30.103 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417140.821551_197.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:14:30.103 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417223.928828_219.1 target=1_1695417140.821551_197.1 2023-09-22T21:14:35.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:14:35.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:14:35.105 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:14:35.149 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:14:35.149 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:14:35.187 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:14:35.187+0000 7fa6ae49da40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:14:35.188 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:14:35.191+0000 7fa6ae49da40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:14:35.205 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:14:35Z"} 2023-09-22T21:14:35.205 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:14:35Z"} 2023-09-22T21:14:35.206 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417140.821551_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:20.821551Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:14:35.206 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:14:35.206 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417140.821551_197.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:14:35.206 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417223.928828_219.1 target=1_1695417140.821551_197.1 2023-09-22T21:14:40.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:14:40.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:14:40.208 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:14:40.252 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:14:40.252 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:14:40.288 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:14:40.291+0000 7f1b8837ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:14:40.289 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:14:40.291+0000 7f1b8837ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:14:40.306 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:14:40Z"} 2023-09-22T21:14:40.307 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417140.821551_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:20.821551Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:14:40Z"} 2023-09-22T21:14:40.307 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417140.821551_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:20.821551Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:14:40.307 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:14:40.307 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417140.821551_197.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:14:40.307 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417223.928828_219.1 target=1_1695417140.821551_197.1 2023-09-22T21:14:45.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:14:45.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:14:45.309 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:14:45.353 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:14:45.353 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:14:45.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:14:45.391+0000 7ff4b746ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:14:45.390 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:14:45.391+0000 7ff4b746ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:14:45.408 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:14:45Z"} 2023-09-22T21:14:45.409 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:14:45Z"} 2023-09-22T21:14:45.409 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:14:45.409 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:14:45.409 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:14:45.409 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:14:45.410 DEBUG:boto:path=/ffzdqu-35/ 2023-09-22T21:14:45.410 DEBUG:boto:auth_path=/ffzdqu-35/ 2023-09-22T21:14:45.411 DEBUG:boto:Method: HEAD 2023-09-22T21:14:45.411 DEBUG:boto:Path: /ffzdqu-35/ 2023-09-22T21:14:45.411 DEBUG:boto:Data: 2023-09-22T21:14:45.411 DEBUG:boto:Headers: {} 2023-09-22T21:14:45.411 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:14:45.411 DEBUG:boto:Port: 8001 2023-09-22T21:14:45.411 DEBUG:boto:Params: {} 2023-09-22T21:14:45.411 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:14:45.412 DEBUG:boto:Token: None 2023-09-22T21:14:45.412 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:14:45 GMT /ffzdqu-35/ 2023-09-22T21:14:45.412 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:00hWCGN/t3VlxgkTtFjHKRTYdfw= 2023-09-22T21:14:45.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:14:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:00hWCGN/t3VlxgkTtFjHKRTYdfw=', 'Content-Length': '0'} 2023-09-22T21:14:45.418 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013caab8cee5f13b5-00650e03c5-1510-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:14:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:14:45.418 DEBUG:boto:path=/ffzdqu-35/testredirect 2023-09-22T21:14:45.419 DEBUG:boto:auth_path=/ffzdqu-35/testredirect 2023-09-22T21:14:45.419 DEBUG:boto:Method: HEAD 2023-09-22T21:14:45.419 DEBUG:boto:Path: /ffzdqu-35/testredirect 2023-09-22T21:14:45.419 DEBUG:boto:Data: 2023-09-22T21:14:45.419 DEBUG:boto:Headers: {} 2023-09-22T21:14:45.419 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:14:45.419 DEBUG:boto:Port: 8001 2023-09-22T21:14:45.419 DEBUG:boto:Params: {} 2023-09-22T21:14:45.420 DEBUG:boto:Token: None 2023-09-22T21:14:45.420 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:14:45 GMT /ffzdqu-35/testredirect 2023-09-22T21:14:45.420 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VcT34mgSEjSk6UMo1QSRSaVUywM= 2023-09-22T21:14:45.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:14:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VcT34mgSEjSk6UMo1QSRSaVUywM=', 'Content-Length': '0'} 2023-09-22T21:14:45.421 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000912fdfa3f749f1df-00650e03c5-1510-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:14:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:14:45.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=f132bdb9-20a7-4073-a932-8f99da20691e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:14:45.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=f132bdb9-20a7-4073-a932-8f99da20691e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:14:45.422 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone=f132bdb9-20a7-4073-a932-8f99da20691e --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:14:45.489 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:14:45.490 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:14:45.518 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"2b721fee-a149-45da-925b-502d5a3d6853","name":"a2","domain_root":"a2.rgw.meta:root","control_pool":"a2.rgw.control","gc_pool":"a2.rgw.log:gc","lc_pool":"a2.rgw.log:lc","log_pool":"a2.rgw.log","intent_log_pool":"a2.rgw.log:intent","usage_log_pool":"a2.rgw.log:usage","roles_pool":"a2.rgw.meta:roles","reshard_pool":"a2.rgw.log:reshard","user_keys_pool":"a2.rgw.meta:users.keys","user_email_pool":"a2.rgw.meta:users.email","user_swift_pool":"a2.rgw.meta:users.swift","user_uid_pool":"a2.rgw.meta:users.uid","otp_pool":"a2.rgw.otp","system_key":{"access_key":"KT50DM2YGPMJ8OAR","secret_key":"lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"a2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"a2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"a2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","notif_pool":"a2.rgw.log:notif"} 2023-09-22T21:14:45.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:14:45.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:14:45.519 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:14:45.594 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:14:45.594 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:14:45.648 INFO:teuthology.orchestra.run.smithi160.stderr:Sending period to new master zone f132bdb9-20a7-4073-a932-8f99da20691e 2023-09-22T21:14:45.696 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","epoch":6,"predecessor_uuid":"a6099785-5160-4a69-b202-552a1b9cb770","sync_status":[],"period_map":{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","zonegroups":[{"id":"a","name":"a","api_name":"","is_master":true,"endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","zones":[{"id":"2b721fee-a149-45da-925b-502d5a3d6853","name":"a2","endpoints":["http://smithi160.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":false,"sync_from":[],"redirect_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","supported_features":["compress-encrypted","resharding"]},{"id":"f132bdb9-20a7-4073-a932-8f99da20691e","name":"a1","endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]},{"id":"b","name":"b","api_name":"","is_master":false,"endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","zones":[{"id":"087d4b41-8f81-4835-9e99-026ca13a7b60","name":"b2","endpoints":["http://smithi204.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","name":"b1","endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"087d4b41-8f81-4835-9e99-026ca13a7b60","val":2135214481},{"key":"2b721fee-a149-45da-925b-502d5a3d6853","val":3586193783},{"key":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","val":4184768938},{"key":"f132bdb9-20a7-4073-a932-8f99da20691e","val":3405200286}]},"master_zonegroup":"a","master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","realm_name":"test-realm","realm_epoch":2} 2023-09-22T21:14:45.697 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "f132bdb9-20a7-4073-a932-8f99da20691e" 2023-09-22T21:15:45.697 DEBUG:boto:path=/ffzdqu-35/testredirect 2023-09-22T21:15:45.698 DEBUG:boto:auth_path=/ffzdqu-35/testredirect 2023-09-22T21:15:45.698 DEBUG:boto:Method: HEAD 2023-09-22T21:15:45.698 DEBUG:boto:Path: /ffzdqu-35/testredirect 2023-09-22T21:15:45.698 DEBUG:boto:Data: 2023-09-22T21:15:45.698 DEBUG:boto:Headers: {} 2023-09-22T21:15:45.698 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:15:45.699 DEBUG:boto:Port: 8001 2023-09-22T21:15:45.699 DEBUG:boto:Params: {} 2023-09-22T21:15:45.699 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:15:45.699 DEBUG:boto:Token: None 2023-09-22T21:15:45.699 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:15:45 GMT /ffzdqu-35/testredirect 2023-09-22T21:15:45.700 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Lvd5X65dvkP1yiFwPaAwfGhJhNA= 2023-09-22T21:15:45.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:15:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Lvd5X65dvkP1yiFwPaAwfGhJhNA=', 'Content-Length': '0'} 2023-09-22T21:15:45.708 DEBUG:boto:Response headers: [('Location', 'http://smithi160.front.sepia.ceph.com:8000/ffzdqu-35/testredirect?'), ('x-amz-request-id', 'tx0000066956eef1ef80f8d-00650e0401-1599-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:15:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:15:45.709 DEBUG:boto:Redirecting: http://smithi160.front.sepia.ceph.com/ffzdqu-35/testredirect 2023-09-22T21:15:45.709 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': '8000'} 2023-09-22T21:15:45.709 DEBUG:boto:Token: None 2023-09-22T21:15:45.709 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:15:45 GMT /ffzdqu-35/testredirect 2023-09-22T21:15:45.709 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Lvd5X65dvkP1yiFwPaAwfGhJhNA= 2023-09-22T21:15:45.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:15:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Lvd5X65dvkP1yiFwPaAwfGhJhNA=', 'Content-Length': '0'} 2023-09-22T21:15:45.718 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:13:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000007d97b62f7016c14c-00650e0401-1597-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:15:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:15:45.719 DEBUG:boto:path=/ffzdqu-35/testredirect 2023-09-22T21:15:45.719 DEBUG:boto:auth_path=/ffzdqu-35/testredirect 2023-09-22T21:15:45.719 DEBUG:boto:Method: GET 2023-09-22T21:15:45.719 DEBUG:boto:Path: /ffzdqu-35/testredirect 2023-09-22T21:15:45.719 DEBUG:boto:Data: 2023-09-22T21:15:45.719 DEBUG:boto:Headers: {} 2023-09-22T21:15:45.719 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:15:45.719 DEBUG:boto:Port: 8001 2023-09-22T21:15:45.719 DEBUG:boto:Params: {} 2023-09-22T21:15:45.720 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:15:45.720 DEBUG:boto:Token: None 2023-09-22T21:15:45.720 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:15:45 GMT /ffzdqu-35/testredirect 2023-09-22T21:15:45.720 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yAnn2rmxURWa2RYg/48jK9eO960= 2023-09-22T21:15:45.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:15:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yAnn2rmxURWa2RYg/48jK9eO960=', 'Content-Length': '0'} 2023-09-22T21:15:45.724 DEBUG:boto:Response headers: [('Location', 'http://smithi160.front.sepia.ceph.com:8000/ffzdqu-35/testredirect?'), ('x-amz-request-id', 'tx0000072d24b34cd479a57-00650e0401-1599-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:15:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:15:45.724 DEBUG:boto:Redirecting: http://smithi160.front.sepia.ceph.com/ffzdqu-35/testredirect 2023-09-22T21:15:45.724 DEBUG:boto:Token: None 2023-09-22T21:15:45.724 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:15:45 GMT /ffzdqu-35/testredirect 2023-09-22T21:15:45.724 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yAnn2rmxURWa2RYg/48jK9eO960= 2023-09-22T21:15:45.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:15:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yAnn2rmxURWa2RYg/48jK9eO960=', 'Content-Length': '0'} 2023-09-22T21:15:45.728 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:13:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000001f31e9a90effa925-00650e0401-1597-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:15:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:15:45.728 DEBUG:boto:path=/ffzdqu-35/testredirect 2023-09-22T21:15:45.728 DEBUG:boto:auth_path=/ffzdqu-35/testredirect 2023-09-22T21:15:45.728 DEBUG:boto:Method: PUT 2023-09-22T21:15:45.728 DEBUG:boto:Path: /ffzdqu-35/testredirect 2023-09-22T21:15:45.728 DEBUG:boto:Data: 2023-09-22T21:15:45.729 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-09-22T21:15:45.729 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:15:45.729 DEBUG:boto:Port: 8000 2023-09-22T21:15:45.729 DEBUG:boto:Params: {} 2023-09-22T21:15:45.729 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:15:45.729 DEBUG:boto:Token: None 2023-09-22T21:15:45.729 DEBUG:boto:StringToSign: PUT VpBzljOcorCZvRIkX5Nt3A== application/octet-stream Fri, 22 Sep 2023 21:15:45 GMT /ffzdqu-35/testredirect 2023-09-22T21:15:45.730 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QkyUl7N+Lv8E/7NvhJDDfYEp0pw= 2023-09-22T21:15:45.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:15:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QkyUl7N+Lv8E/7NvhJDDfYEp0pw='} 2023-09-22T21:15:45.759 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008b64d718653a111b-00650e0401-1597-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:15:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:15:45.759 DEBUG:boto:path=/ffzdqu-35/testredirect 2023-09-22T21:15:45.760 DEBUG:boto:auth_path=/ffzdqu-35/testredirect 2023-09-22T21:15:45.760 DEBUG:boto:Method: GET 2023-09-22T21:15:45.760 DEBUG:boto:Path: /ffzdqu-35/testredirect 2023-09-22T21:15:45.760 DEBUG:boto:Data: 2023-09-22T21:15:45.760 DEBUG:boto:Headers: {} 2023-09-22T21:15:45.760 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:15:45.760 DEBUG:boto:Port: 8001 2023-09-22T21:15:45.760 DEBUG:boto:Params: {} 2023-09-22T21:15:45.760 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:15:45.760 DEBUG:boto:Token: None 2023-09-22T21:15:45.761 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:15:45 GMT /ffzdqu-35/testredirect 2023-09-22T21:15:45.761 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yAnn2rmxURWa2RYg/48jK9eO960= 2023-09-22T21:15:45.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:15:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yAnn2rmxURWa2RYg/48jK9eO960=', 'Content-Length': '0'} 2023-09-22T21:15:45.765 DEBUG:boto:Response headers: [('Location', 'http://smithi160.front.sepia.ceph.com:8000/ffzdqu-35/testredirect?'), ('x-amz-request-id', 'tx000003d45bbbc653a7c22-00650e0401-1599-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:15:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:15:45.765 DEBUG:boto:Redirecting: http://smithi160.front.sepia.ceph.com/ffzdqu-35/testredirect 2023-09-22T21:15:45.765 DEBUG:boto:Token: None 2023-09-22T21:15:45.765 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:15:45 GMT /ffzdqu-35/testredirect 2023-09-22T21:15:45.765 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yAnn2rmxURWa2RYg/48jK9eO960= 2023-09-22T21:15:45.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:15:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yAnn2rmxURWa2RYg/48jK9eO960=', 'Content-Length': '0'} 2023-09-22T21:15:45.768 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:15:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('x-amz-request-id', 'tx0000088adb7746719fcea-00650e0401-1597-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:15:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:15:45.769 DEBUG:boto:path=/ffzdqu-35/testredirect 2023-09-22T21:15:45.769 DEBUG:boto:auth_path=/ffzdqu-35/testredirect 2023-09-22T21:15:45.769 DEBUG:boto:Method: PUT 2023-09-22T21:15:45.769 DEBUG:boto:Path: /ffzdqu-35/testredirect 2023-09-22T21:15:45.769 DEBUG:boto:Data: 2023-09-22T21:15:45.769 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-09-22T21:15:45.769 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:15:45.769 DEBUG:boto:Port: 8000 2023-09-22T21:15:45.769 DEBUG:boto:Params: {} 2023-09-22T21:15:45.770 DEBUG:boto:Token: None 2023-09-22T21:15:45.770 DEBUG:boto:StringToSign: PUT uk9S5NXZfBvPq4jGr+LM5g== application/octet-stream Fri, 22 Sep 2023 21:15:45 GMT /ffzdqu-35/testredirect 2023-09-22T21:15:45.770 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i7PEBMFCedIQ4NAJa+pF7YDBbJw= 2023-09-22T21:15:45.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:15:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i7PEBMFCedIQ4NAJa+pF7YDBbJw='} 2023-09-22T21:15:45.783 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b772e9f8d4fc87c-00650e0401-1597-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:15:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:15:45.783 DEBUG:boto:path=/ffzdqu-35/testredirect 2023-09-22T21:15:45.783 DEBUG:boto:auth_path=/ffzdqu-35/testredirect 2023-09-22T21:15:45.783 DEBUG:boto:Method: GET 2023-09-22T21:15:45.783 DEBUG:boto:Path: /ffzdqu-35/testredirect 2023-09-22T21:15:45.783 DEBUG:boto:Data: 2023-09-22T21:15:45.783 DEBUG:boto:Headers: {} 2023-09-22T21:15:45.784 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:15:45.784 DEBUG:boto:Port: 8001 2023-09-22T21:15:45.784 DEBUG:boto:Params: {} 2023-09-22T21:15:45.784 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:15:45.784 DEBUG:boto:Token: None 2023-09-22T21:15:45.784 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:15:45 GMT /ffzdqu-35/testredirect 2023-09-22T21:15:45.784 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yAnn2rmxURWa2RYg/48jK9eO960= 2023-09-22T21:15:45.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:15:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yAnn2rmxURWa2RYg/48jK9eO960=', 'Content-Length': '0'} 2023-09-22T21:15:45.788 DEBUG:boto:Response headers: [('Location', 'http://smithi160.front.sepia.ceph.com:8000/ffzdqu-35/testredirect?'), ('x-amz-request-id', 'tx00000a307aa770c74dfd0-00650e0401-1599-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:15:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:15:45.789 DEBUG:boto:Redirecting: http://smithi160.front.sepia.ceph.com/ffzdqu-35/testredirect 2023-09-22T21:15:45.789 DEBUG:boto:Token: None 2023-09-22T21:15:45.789 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:15:45 GMT /ffzdqu-35/testredirect 2023-09-22T21:15:45.789 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yAnn2rmxURWa2RYg/48jK9eO960= 2023-09-22T21:15:45.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:15:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yAnn2rmxURWa2RYg/48jK9eO960=', 'Content-Length': '0'} 2023-09-22T21:15:45.795 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:15:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('x-amz-request-id', 'tx000002cac00fe6e3b362f-00650e0401-1597-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:15:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:15:45.838 DEBUG:boto:path=/ffzdqu-35/testredirect 2023-09-22T21:15:45.838 DEBUG:boto:auth_path=/ffzdqu-35/testredirect 2023-09-22T21:15:45.839 DEBUG:boto:Method: PUT 2023-09-22T21:15:45.839 DEBUG:boto:Path: /ffzdqu-35/testredirect 2023-09-22T21:15:45.839 DEBUG:boto:Data: 2023-09-22T21:15:45.839 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-09-22T21:15:45.839 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:15:45.839 DEBUG:boto:Port: 8000 2023-09-22T21:15:45.839 DEBUG:boto:Params: {} 2023-09-22T21:15:45.839 DEBUG:boto:Token: None 2023-09-22T21:15:45.840 DEBUG:boto:StringToSign: PUT 6aKmvaJMgaK7Z2rfDZvVeg== application/octet-stream Fri, 22 Sep 2023 21:15:45 GMT /ffzdqu-35/testredirect 2023-09-22T21:15:45.840 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+++wCsSBUL4mxm4ePP3fKo7WIpQ= 2023-09-22T21:15:45.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:15:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+++wCsSBUL4mxm4ePP3fKo7WIpQ='} 2023-09-22T21:15:45.853 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c89fd09445f1650e-00650e0401-1597-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:15:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:15:45.853 DEBUG:boto:path=/ffzdqu-35/testredirect 2023-09-22T21:15:45.853 DEBUG:boto:auth_path=/ffzdqu-35/testredirect 2023-09-22T21:15:45.853 DEBUG:boto:Method: GET 2023-09-22T21:15:45.853 DEBUG:boto:Path: /ffzdqu-35/testredirect 2023-09-22T21:15:45.853 DEBUG:boto:Data: 2023-09-22T21:15:45.853 DEBUG:boto:Headers: {} 2023-09-22T21:15:45.853 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:15:45.853 DEBUG:boto:Port: 8001 2023-09-22T21:15:45.854 DEBUG:boto:Params: {} 2023-09-22T21:15:45.854 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:15:45.854 DEBUG:boto:Token: None 2023-09-22T21:15:45.854 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:15:45 GMT /ffzdqu-35/testredirect 2023-09-22T21:15:45.854 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yAnn2rmxURWa2RYg/48jK9eO960= 2023-09-22T21:15:45.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:15:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yAnn2rmxURWa2RYg/48jK9eO960=', 'Content-Length': '0'} 2023-09-22T21:15:45.858 DEBUG:boto:Response headers: [('Location', 'http://smithi160.front.sepia.ceph.com:8000/ffzdqu-35/testredirect?'), ('x-amz-request-id', 'tx000001b46009fb48e6be0-00650e0401-1599-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:15:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:15:45.858 DEBUG:boto:Redirecting: http://smithi160.front.sepia.ceph.com/ffzdqu-35/testredirect 2023-09-22T21:15:45.859 DEBUG:boto:Token: None 2023-09-22T21:15:45.859 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:15:45 GMT /ffzdqu-35/testredirect 2023-09-22T21:15:45.859 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yAnn2rmxURWa2RYg/48jK9eO960= 2023-09-22T21:15:45.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:15:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yAnn2rmxURWa2RYg/48jK9eO960=', 'Content-Length': '0'} 2023-09-22T21:15:45.865 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:15:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('x-amz-request-id', 'tx00000ae3d0bcf9ab35864-00650e0401-1597-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:15:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:15:45.865 DEBUG:boto:path=/ffzdqu-35/testredirect 2023-09-22T21:15:45.865 DEBUG:boto:auth_path=/ffzdqu-35/testredirect 2023-09-22T21:15:45.865 DEBUG:boto:Method: PUT 2023-09-22T21:15:45.865 DEBUG:boto:Path: /ffzdqu-35/testredirect 2023-09-22T21:15:45.865 DEBUG:boto:Data: 2023-09-22T21:15:45.866 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-09-22T21:15:45.866 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:15:45.866 DEBUG:boto:Port: 8000 2023-09-22T21:15:45.866 DEBUG:boto:Params: {} 2023-09-22T21:15:45.866 DEBUG:boto:Token: None 2023-09-22T21:15:45.866 DEBUG:boto:StringToSign: PUT GbNgNQ5MAukCtgY+rsT5Xw== application/octet-stream Fri, 22 Sep 2023 21:15:45 GMT /ffzdqu-35/testredirect 2023-09-22T21:15:45.866 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iFjMeuEpoIQ1yazOIUzomEl/KE4= 2023-09-22T21:15:45.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:15:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iFjMeuEpoIQ1yazOIUzomEl/KE4='} 2023-09-22T21:15:45.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005dc5b4bec6a562fe-00650e0401-1597-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:15:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:15:45.878 DEBUG:boto:path=/ffzdqu-35/testredirect 2023-09-22T21:15:45.879 DEBUG:boto:auth_path=/ffzdqu-35/testredirect 2023-09-22T21:15:45.879 DEBUG:boto:Method: GET 2023-09-22T21:15:45.879 DEBUG:boto:Path: /ffzdqu-35/testredirect 2023-09-22T21:15:45.879 DEBUG:boto:Data: 2023-09-22T21:15:45.879 DEBUG:boto:Headers: {} 2023-09-22T21:15:45.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:15:45.879 DEBUG:boto:Port: 8001 2023-09-22T21:15:45.879 DEBUG:boto:Params: {} 2023-09-22T21:15:45.879 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:15:45.880 DEBUG:boto:Token: None 2023-09-22T21:15:45.880 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:15:45 GMT /ffzdqu-35/testredirect 2023-09-22T21:15:45.880 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yAnn2rmxURWa2RYg/48jK9eO960= 2023-09-22T21:15:45.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:15:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yAnn2rmxURWa2RYg/48jK9eO960=', 'Content-Length': '0'} 2023-09-22T21:15:45.885 DEBUG:boto:Response headers: [('Location', 'http://smithi160.front.sepia.ceph.com:8000/ffzdqu-35/testredirect?'), ('x-amz-request-id', 'tx0000039d34454d309fe2e-00650e0401-1599-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:15:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:15:45.885 DEBUG:boto:Redirecting: http://smithi160.front.sepia.ceph.com/ffzdqu-35/testredirect 2023-09-22T21:15:45.885 DEBUG:boto:Token: None 2023-09-22T21:15:45.885 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:15:45 GMT /ffzdqu-35/testredirect 2023-09-22T21:15:45.885 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yAnn2rmxURWa2RYg/48jK9eO960= 2023-09-22T21:15:45.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:15:45 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yAnn2rmxURWa2RYg/48jK9eO960=', 'Content-Length': '0'} 2023-09-22T21:15:45.891 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:15:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('x-amz-request-id', 'tx00000a7da9d95e3d808b9-00650e0401-1597-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:15:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:15:45.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:15:45.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:15:45.892 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=true --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:15:45.934 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:15:45.934 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:15:45.963 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"2b721fee-a149-45da-925b-502d5a3d6853","name":"a2","domain_root":"a2.rgw.meta:root","control_pool":"a2.rgw.control","gc_pool":"a2.rgw.log:gc","lc_pool":"a2.rgw.log:lc","log_pool":"a2.rgw.log","intent_log_pool":"a2.rgw.log:intent","usage_log_pool":"a2.rgw.log:usage","roles_pool":"a2.rgw.meta:roles","reshard_pool":"a2.rgw.log:reshard","user_keys_pool":"a2.rgw.meta:users.keys","user_email_pool":"a2.rgw.meta:users.email","user_swift_pool":"a2.rgw.meta:users.swift","user_uid_pool":"a2.rgw.meta:users.uid","otp_pool":"a2.rgw.otp","system_key":{"access_key":"KT50DM2YGPMJ8OAR","secret_key":"lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"a2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"a2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"a2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","notif_pool":"a2.rgw.log:notif"} 2023-09-22T21:15:45.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:15:45.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:15:45.964 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:15:46.047 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:15:46.047 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:15:46.101 INFO:teuthology.orchestra.run.smithi160.stderr:Sending period to new master zone f132bdb9-20a7-4073-a932-8f99da20691e 2023-09-22T21:15:46.151 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","epoch":7,"predecessor_uuid":"a6099785-5160-4a69-b202-552a1b9cb770","sync_status":[],"period_map":{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","zonegroups":[{"id":"a","name":"a","api_name":"","is_master":true,"endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","zones":[{"id":"2b721fee-a149-45da-925b-502d5a3d6853","name":"a2","endpoints":["http://smithi160.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","supported_features":["compress-encrypted","resharding"]},{"id":"f132bdb9-20a7-4073-a932-8f99da20691e","name":"a1","endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]},{"id":"b","name":"b","api_name":"","is_master":false,"endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","zones":[{"id":"087d4b41-8f81-4835-9e99-026ca13a7b60","name":"b2","endpoints":["http://smithi204.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","name":"b1","endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"087d4b41-8f81-4835-9e99-026ca13a7b60","val":2135214481},{"key":"2b721fee-a149-45da-925b-502d5a3d6853","val":3586193783},{"key":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","val":4184768938},{"key":"f132bdb9-20a7-4073-a932-8f99da20691e","val":3405200286}]},"master_zonegroup":"a","master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","realm_name":"test-realm","realm_epoch":2} 2023-09-22T21:15:46.151 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to true 2023-09-22T21:16:46.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:16:46.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:16:46.153 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone= --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:16:46.195 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:16:46.195 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:16:46.227 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"2b721fee-a149-45da-925b-502d5a3d6853","name":"a2","domain_root":"a2.rgw.meta:root","control_pool":"a2.rgw.control","gc_pool":"a2.rgw.log:gc","lc_pool":"a2.rgw.log:lc","log_pool":"a2.rgw.log","intent_log_pool":"a2.rgw.log:intent","usage_log_pool":"a2.rgw.log:usage","roles_pool":"a2.rgw.meta:roles","reshard_pool":"a2.rgw.log:reshard","user_keys_pool":"a2.rgw.meta:users.keys","user_email_pool":"a2.rgw.meta:users.email","user_swift_pool":"a2.rgw.meta:users.swift","user_uid_pool":"a2.rgw.meta:users.uid","otp_pool":"a2.rgw.otp","system_key":{"access_key":"KT50DM2YGPMJ8OAR","secret_key":"lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"a2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"a2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"a2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","notif_pool":"a2.rgw.log:notif"} 2023-09-22T21:16:46.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:16:46.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:16:46.228 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:16:46.306 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:16:46.306 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:16:46.365 INFO:teuthology.orchestra.run.smithi160.stderr:Sending period to new master zone f132bdb9-20a7-4073-a932-8f99da20691e 2023-09-22T21:16:46.412 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","epoch":8,"predecessor_uuid":"a6099785-5160-4a69-b202-552a1b9cb770","sync_status":[],"period_map":{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","zonegroups":[{"id":"a","name":"a","api_name":"","is_master":true,"endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","zones":[{"id":"2b721fee-a149-45da-925b-502d5a3d6853","name":"a2","endpoints":["http://smithi160.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"f132bdb9-20a7-4073-a932-8f99da20691e","name":"a1","endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]},{"id":"b","name":"b","api_name":"","is_master":false,"endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","zones":[{"id":"087d4b41-8f81-4835-9e99-026ca13a7b60","name":"b2","endpoints":["http://smithi204.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","name":"b1","endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"087d4b41-8f81-4835-9e99-026ca13a7b60","val":2135214481},{"key":"2b721fee-a149-45da-925b-502d5a3d6853","val":3586193783},{"key":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","val":4184768938},{"key":"f132bdb9-20a7-4073-a932-8f99da20691e","val":3405200286}]},"master_zonegroup":"a","master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","realm_name":"test-realm","realm_epoch":2} 2023-09-22T21:16:46.413 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "" 2023-09-22T21:16:46.707 INFO:tasks.rgw.c1.client.0.smithi160.stdout:2023-09-22T21:16:46.710+0000 7fa9c1675640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:17:46.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... ok 2023-09-22T21:17:46.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:17:46.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:17:46.415 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:17:46.458 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:17:46.458 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:17:46.496 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:17:46.513 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695417223.928828_219.1","last_update":"2023-09-22T21:13:43.928828Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:17:46.513 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:17:46.514 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:17:46.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:17:46.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:17:46.514 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:17:46.590 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:17:46.591 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:17:46.627 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:17:46.628+0000 7fd58b158a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:17:46.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:17:46.632+0000 7fd58b158a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:17:46.645 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:17:46Z"} 2023-09-22T21:17:46.645 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:17:46Z"} 2023-09-22T21:17:46.645 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:17:46.646 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:17:46.646 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:17:46.646 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:17:46.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'KT50DM2YGPMJ8OAR', '--secret', 'lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:46.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'KT50DM2YGPMJ8OAR', '--secret', 'lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:46.646 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --access-key KT50DM2YGPMJ8OAR --secret lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:17:46.726 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:17:46.726 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:17:46.749 INFO:teuthology.orchestra.run.smithi160.stderr:NOTICE: set zone's realm_id=9e7cb63e-d555-4d20-9c48-393b1c24d1ff 2023-09-22T21:17:46.774 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"9a953f47-426a-4852-ad80-868c6f7a6dc0","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"KT50DM2YGPMJ8OAR","secret_key":"lrD7NXRCimLs6LMgwXC8zPDwuWhwW9yo"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","notif_pool":"remove.rgw.log:notif"} 2023-09-22T21:17:46.774 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:46.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:46.775 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:17:46.854 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:17:46.854 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:17:46.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:17:46.872+0000 7f534d7c3a40 0 period (03d00cc5-7ad7-4abf-a475-d9ffc8e37106 does not have zone 9a953f47-426a-4852-ad80-868c6f7a6dc0 configured 2023-09-22T21:17:53.547 INFO:teuthology.orchestra.run.smithi160.stderr:Sending period to new master zone f132bdb9-20a7-4073-a932-8f99da20691e 2023-09-22T21:17:53.602 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","epoch":9,"predecessor_uuid":"a6099785-5160-4a69-b202-552a1b9cb770","sync_status":[],"period_map":{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","zonegroups":[{"id":"a","name":"a","api_name":"","is_master":true,"endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","zones":[{"id":"2b721fee-a149-45da-925b-502d5a3d6853","name":"a2","endpoints":["http://smithi160.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"9a953f47-426a-4852-ad80-868c6f7a6dc0","name":"remove","endpoints":[],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"f132bdb9-20a7-4073-a932-8f99da20691e","name":"a1","endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]},{"id":"b","name":"b","api_name":"","is_master":false,"endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","zones":[{"id":"087d4b41-8f81-4835-9e99-026ca13a7b60","name":"b2","endpoints":["http://smithi204.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","name":"b1","endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"087d4b41-8f81-4835-9e99-026ca13a7b60","val":2135214481},{"key":"2b721fee-a149-45da-925b-502d5a3d6853","val":3586193783},{"key":"9a953f47-426a-4852-ad80-868c6f7a6dc0","val":231323593},{"key":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","val":4184768938},{"key":"f132bdb9-20a7-4073-a932-8f99da20691e","val":3405200286}]},"master_zonegroup":"a","master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","realm_name":"test-realm","realm_epoch":2} 2023-09-22T21:17:53.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:53.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:53.603 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:17:53.682 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:17:53.682 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:17:53.713 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"a","name":"a","api_name":"","is_master":true,"endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","zones":[{"id":"2b721fee-a149-45da-925b-502d5a3d6853","name":"a2","endpoints":["http://smithi160.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"f132bdb9-20a7-4073-a932-8f99da20691e","name":"a1","endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]} 2023-09-22T21:17:53.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:53.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:53.714 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:17:53.799 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:17:53.799 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:17:53.833 INFO:teuthology.orchestra.run.smithi160.stderr:zone name remove not found in zonegroup a 2023-09-22T21:17:53.840 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-09-22T21:17:53.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:53.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:53.841 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:17:53.924 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:17:53.924 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:17:53.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:53.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:53.986 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:17:54.069 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:17:54.070 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:17:54.096 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:17:54.100+0000 7fd388f24a40 0 WARNING: can't generate connection for zone 9a953f47-426a-4852-ad80-868c6f7a6dc0 id remove: no endpoints defined 2023-09-22T21:17:54.183 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","epoch":10,"predecessor_uuid":"a6099785-5160-4a69-b202-552a1b9cb770","sync_status":[],"period_map":{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","zonegroups":[{"id":"a","name":"a","api_name":"","is_master":true,"endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","zones":[{"id":"2b721fee-a149-45da-925b-502d5a3d6853","name":"a2","endpoints":["http://smithi160.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"f132bdb9-20a7-4073-a932-8f99da20691e","name":"a1","endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]},{"id":"b","name":"b","api_name":"","is_master":false,"endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","zones":[{"id":"087d4b41-8f81-4835-9e99-026ca13a7b60","name":"b2","endpoints":["http://smithi204.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","name":"b1","endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"087d4b41-8f81-4835-9e99-026ca13a7b60","val":2135214481},{"key":"2b721fee-a149-45da-925b-502d5a3d6853","val":3586193783},{"key":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","val":4184768938},{"key":"f132bdb9-20a7-4073-a932-8f99da20691e","val":3405200286}]},"master_zonegroup":"a","master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","realm_name":"test-realm","realm_epoch":2} 2023-09-22T21:17:54.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2023-09-22T21:17:54.184 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:54.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:54.184 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:17:54.262 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:17:54.262 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:17:54.295 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"0ff5ba2a-a0c9-4e84-bb37-e02e3cb11fda","name":"remove_zg","api_name":"remove_zg","is_master":false,"endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":[]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]} 2023-09-22T21:17:54.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:54.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:54.295 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:17:54.375 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:17:54.375 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:17:54.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:17:54.460+0000 7f9e4b01ca40 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2023-09-22T21:17:54.465 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"88081928-d3e7-41a7-a898-6c07b4a41367","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","notif_pool":"remove.rgw.log:notif"} 2023-09-22T21:17:54.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:54.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:54.466 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:17:54.548 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:17:54.548 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:17:54.676 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","epoch":11,"predecessor_uuid":"a6099785-5160-4a69-b202-552a1b9cb770","sync_status":[],"period_map":{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","zonegroups":[{"id":"0ff5ba2a-a0c9-4e84-bb37-e02e3cb11fda","name":"remove_zg","api_name":"remove_zg","is_master":false,"endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"88081928-d3e7-41a7-a898-6c07b4a41367","zones":[{"id":"88081928-d3e7-41a7-a898-6c07b4a41367","name":"remove","endpoints":[],"log_meta":false,"log_data":false,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]},{"id":"a","name":"a","api_name":"","is_master":true,"endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","zones":[{"id":"2b721fee-a149-45da-925b-502d5a3d6853","name":"a2","endpoints":["http://smithi160.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"f132bdb9-20a7-4073-a932-8f99da20691e","name":"a1","endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]},{"id":"b","name":"b","api_name":"","is_master":false,"endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","zones":[{"id":"087d4b41-8f81-4835-9e99-026ca13a7b60","name":"b2","endpoints":["http://smithi204.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","name":"b1","endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"087d4b41-8f81-4835-9e99-026ca13a7b60","val":2135214481},{"key":"2b721fee-a149-45da-925b-502d5a3d6853","val":3586193783},{"key":"88081928-d3e7-41a7-a898-6c07b4a41367","val":3459188459},{"key":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","val":4184768938},{"key":"f132bdb9-20a7-4073-a932-8f99da20691e","val":3405200286}]},"master_zonegroup":"a","master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","realm_name":"test-realm","realm_epoch":2} 2023-09-22T21:17:54.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:54.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:54.677 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:17:54.758 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:17:54.758 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:17:54.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:17:54.800+0000 7f7a8fcc3a40 0 NOTICE: removed master_zone of zonegroup remove_zg 2023-09-22T21:17:54.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:17:54.800+0000 7f7a8fcc3a40 0 Removed zone from zonegroup remove_zg 2023-09-22T21:17:54.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:54.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:54.811 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:17:54.891 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:17:54.892 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:17:54.944 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:17:54.948+0000 7fe65f728a40 0 ERROR: zonegroup remove_zg has a non existent master zone 2023-09-22T21:17:54.954 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-09-22T21:17:54.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:54.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:54.955 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:17:55.046 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:17:55.047 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:17:55.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:55.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:17:55.095 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:17:55.140 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:17:55.141 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:17:55.254 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","epoch":12,"predecessor_uuid":"a6099785-5160-4a69-b202-552a1b9cb770","sync_status":[],"period_map":{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","zonegroups":[{"id":"a","name":"a","api_name":"","is_master":true,"endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","zones":[{"id":"2b721fee-a149-45da-925b-502d5a3d6853","name":"a2","endpoints":["http://smithi160.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"f132bdb9-20a7-4073-a932-8f99da20691e","name":"a1","endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]},{"id":"b","name":"b","api_name":"","is_master":false,"endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","zones":[{"id":"087d4b41-8f81-4835-9e99-026ca13a7b60","name":"b2","endpoints":["http://smithi204.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","name":"b1","endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"087d4b41-8f81-4835-9e99-026ca13a7b60","val":2135214481},{"key":"2b721fee-a149-45da-925b-502d5a3d6853","val":3586193783},{"key":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","val":4184768938},{"key":"f132bdb9-20a7-4073-a932-8f99da20691e","val":3405200286}]},"master_zonegroup":"a","master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","realm_name":"test-realm","realm_epoch":2} 2023-09-22T21:17:55.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2023-09-22T21:17:55.255 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-36 2023-09-22T21:17:55.255 DEBUG:boto:path=/ffzdqu-36/ 2023-09-22T21:17:55.255 DEBUG:boto:auth_path=/ffzdqu-36/ 2023-09-22T21:17:55.255 DEBUG:boto:Method: PUT 2023-09-22T21:17:55.256 DEBUG:boto:Path: /ffzdqu-36/ 2023-09-22T21:17:55.256 DEBUG:boto:Data: 2023-09-22T21:17:55.256 DEBUG:boto:Headers: {} 2023-09-22T21:17:55.256 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:17:55.256 DEBUG:boto:Port: 8000 2023-09-22T21:17:55.256 DEBUG:boto:Params: {} 2023-09-22T21:17:55.256 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:17:55.257 DEBUG:boto:Token: None 2023-09-22T21:17:55.257 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:17:55 GMT /ffzdqu-36/ 2023-09-22T21:17:55.257 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tZsNl1EDq5mATKlTYhGdW4Of6Hw= 2023-09-22T21:17:55.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:17:55 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tZsNl1EDq5mATKlTYhGdW4Of6Hw=', 'Content-Length': '0'} 2023-09-22T21:17:55.574 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001349a37841f03fc4-00650e0483-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:17:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:17:55.574 INFO:rgw_multi.tests:create bucket zone=a2 name=ffzdqu-37 2023-09-22T21:17:55.575 DEBUG:boto:path=/ffzdqu-37/ 2023-09-22T21:17:55.575 DEBUG:boto:auth_path=/ffzdqu-37/ 2023-09-22T21:17:55.575 DEBUG:boto:Method: PUT 2023-09-22T21:17:55.575 DEBUG:boto:Path: /ffzdqu-37/ 2023-09-22T21:17:55.575 DEBUG:boto:Data: 2023-09-22T21:17:55.575 DEBUG:boto:Headers: {} 2023-09-22T21:17:55.575 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:17:55.575 DEBUG:boto:Port: 8001 2023-09-22T21:17:55.575 DEBUG:boto:Params: {} 2023-09-22T21:17:55.576 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:17:55.576 DEBUG:boto:Token: None 2023-09-22T21:17:55.576 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:17:55 GMT /ffzdqu-37/ 2023-09-22T21:17:55.576 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7s1Rgs7KhqSXYj/mxfNWVUmVO0s= 2023-09-22T21:17:55.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:17:55 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7s1Rgs7KhqSXYj/mxfNWVUmVO0s=', 'Content-Length': '0'} 2023-09-22T21:17:55.623 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e5e5c7588971a708-00650e0483-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:17:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:17:55.623 INFO:rgw_multi.tests:create bucket zone=b1 name=ffzdqu-38 2023-09-22T21:17:55.623 DEBUG:boto:path=/ffzdqu-38/ 2023-09-22T21:17:55.623 DEBUG:boto:auth_path=/ffzdqu-38/ 2023-09-22T21:17:55.624 DEBUG:boto:Method: PUT 2023-09-22T21:17:55.624 DEBUG:boto:Path: /ffzdqu-38/ 2023-09-22T21:17:55.624 DEBUG:boto:Data: 2023-09-22T21:17:55.624 DEBUG:boto:Headers: {} 2023-09-22T21:17:55.624 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8000 2023-09-22T21:17:55.624 DEBUG:boto:Port: 8000 2023-09-22T21:17:55.624 DEBUG:boto:Params: {} 2023-09-22T21:17:55.624 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:17:55.625 DEBUG:boto:Token: None 2023-09-22T21:17:55.625 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:17:55 GMT /ffzdqu-38/ 2023-09-22T21:17:55.625 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PLGV/mYUN55iWR2+xAej83Hmv24= 2023-09-22T21:17:55.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:17:55 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PLGV/mYUN55iWR2+xAej83Hmv24=', 'Content-Length': '0'} 2023-09-22T21:17:55.671 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000097de9f6851eef507-00650e0483-1172-b1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:17:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:17:55.671 INFO:rgw_multi.tests:create bucket zone=b2 name=ffzdqu-39 2023-09-22T21:17:55.671 DEBUG:boto:path=/ffzdqu-39/ 2023-09-22T21:17:55.671 DEBUG:boto:auth_path=/ffzdqu-39/ 2023-09-22T21:17:55.671 DEBUG:boto:Method: PUT 2023-09-22T21:17:55.672 DEBUG:boto:Path: /ffzdqu-39/ 2023-09-22T21:17:55.672 DEBUG:boto:Data: 2023-09-22T21:17:55.672 DEBUG:boto:Headers: {} 2023-09-22T21:17:55.672 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8001 2023-09-22T21:17:55.672 DEBUG:boto:Port: 8001 2023-09-22T21:17:55.672 DEBUG:boto:Params: {} 2023-09-22T21:17:55.672 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:17:55.673 DEBUG:boto:Token: None 2023-09-22T21:17:55.673 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:17:55 GMT /ffzdqu-39/ 2023-09-22T21:17:55.673 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yTAtUb5zAg49A5HzouiEfWS1pg8= 2023-09-22T21:17:55.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:17:55 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yTAtUb5zAg49A5HzouiEfWS1pg8=', 'Content-Length': '0'} 2023-09-22T21:17:55.711 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fabd01703262cf59-00650e0483-116e-b2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:17:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:17:55.712 DEBUG:boto:path=/ffzdqu-36/ 2023-09-22T21:17:55.712 DEBUG:boto:auth_path=/ffzdqu-36/ 2023-09-22T21:17:55.712 DEBUG:boto:path=/ffzdqu-36/?website 2023-09-22T21:17:55.712 DEBUG:boto:auth_path=/ffzdqu-36/?website 2023-09-22T21:17:55.712 DEBUG:boto:Method: PUT 2023-09-22T21:17:55.712 DEBUG:boto:Path: /ffzdqu-36/?website 2023-09-22T21:17:55.712 DEBUG:boto:Data: index.htmlerror.html 2023-09-22T21:17:55.712 DEBUG:boto:Headers: {} 2023-09-22T21:17:55.712 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:17:55.713 DEBUG:boto:Port: 8000 2023-09-22T21:17:55.713 DEBUG:boto:Params: {} 2023-09-22T21:17:55.713 DEBUG:boto:Token: None 2023-09-22T21:17:55.713 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:17:55 GMT /ffzdqu-36/?website 2023-09-22T21:17:55.713 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KcpDeyV/XEOS7ikr4QPfcznTrEw= 2023-09-22T21:17:55.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:17:55 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KcpDeyV/XEOS7ikr4QPfcznTrEw=', 'Content-Length': '241'} 2023-09-22T21:17:55.714 DEBUG:boto:Response headers: [('Content-Length', '192'), ('x-amz-request-id', 'tx00000e790ddba115a056d-00650e0483-165f-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:17:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:17:55.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2023-09-22T21:17:55.715 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-40 2023-09-22T21:17:55.716 DEBUG:boto:path=/ffzdqu-40/ 2023-09-22T21:17:55.716 DEBUG:boto:auth_path=/ffzdqu-40/ 2023-09-22T21:17:55.716 DEBUG:boto:Method: PUT 2023-09-22T21:17:55.716 DEBUG:boto:Path: /ffzdqu-40/ 2023-09-22T21:17:55.716 DEBUG:boto:Data: 2023-09-22T21:17:55.716 DEBUG:boto:Headers: {} 2023-09-22T21:17:55.716 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:17:55.716 DEBUG:boto:Port: 8000 2023-09-22T21:17:55.716 DEBUG:boto:Params: {} 2023-09-22T21:17:55.717 DEBUG:boto:Token: None 2023-09-22T21:17:55.717 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:17:55 GMT /ffzdqu-40/ 2023-09-22T21:17:55.717 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ye1TXryYwQB/mQ0aLjj908xwhFo= 2023-09-22T21:17:55.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:17:55 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ye1TXryYwQB/mQ0aLjj908xwhFo=', 'Content-Length': '0'} 2023-09-22T21:17:55.737 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000205b07d1ef511b8a-00650e0483-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:17:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:17:55.737 INFO:rgw_multi.tests:create bucket zone=a2 name=ffzdqu-41 2023-09-22T21:17:55.737 DEBUG:boto:path=/ffzdqu-41/ 2023-09-22T21:17:55.738 DEBUG:boto:auth_path=/ffzdqu-41/ 2023-09-22T21:17:55.738 DEBUG:boto:Method: PUT 2023-09-22T21:17:55.738 DEBUG:boto:Path: /ffzdqu-41/ 2023-09-22T21:17:55.738 DEBUG:boto:Data: 2023-09-22T21:17:55.738 DEBUG:boto:Headers: {} 2023-09-22T21:17:55.738 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:17:55.738 DEBUG:boto:Port: 8001 2023-09-22T21:17:55.738 DEBUG:boto:Params: {} 2023-09-22T21:17:55.738 DEBUG:boto:Token: None 2023-09-22T21:17:55.739 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:17:55 GMT /ffzdqu-41/ 2023-09-22T21:17:55.739 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xqhfIP74tAY8d4om0PSaPoFEdeA= 2023-09-22T21:17:55.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:17:55 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xqhfIP74tAY8d4om0PSaPoFEdeA=', 'Content-Length': '0'} 2023-09-22T21:17:55.771 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ebe503f5613d8991-00650e0483-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:17:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:17:55.771 INFO:rgw_multi.tests:create bucket zone=b1 name=ffzdqu-42 2023-09-22T21:17:55.771 DEBUG:boto:path=/ffzdqu-42/ 2023-09-22T21:17:55.771 DEBUG:boto:auth_path=/ffzdqu-42/ 2023-09-22T21:17:55.772 DEBUG:boto:Method: PUT 2023-09-22T21:17:55.772 DEBUG:boto:Path: /ffzdqu-42/ 2023-09-22T21:17:55.772 DEBUG:boto:Data: 2023-09-22T21:17:55.772 DEBUG:boto:Headers: {} 2023-09-22T21:17:55.772 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8000 2023-09-22T21:17:55.772 DEBUG:boto:Port: 8000 2023-09-22T21:17:55.772 DEBUG:boto:Params: {} 2023-09-22T21:17:55.772 DEBUG:boto:Token: None 2023-09-22T21:17:55.772 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:17:55 GMT /ffzdqu-42/ 2023-09-22T21:17:55.773 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iY49rwV2SxS+b2DKFLfTixMPdbM= 2023-09-22T21:17:55.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:17:55 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iY49rwV2SxS+b2DKFLfTixMPdbM=', 'Content-Length': '0'} 2023-09-22T21:17:55.810 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000976aeec5fd1abe17-00650e0483-1172-b1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:17:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:17:55.810 INFO:rgw_multi.tests:create bucket zone=b2 name=ffzdqu-43 2023-09-22T21:17:55.810 DEBUG:boto:path=/ffzdqu-43/ 2023-09-22T21:17:55.810 DEBUG:boto:auth_path=/ffzdqu-43/ 2023-09-22T21:17:55.810 DEBUG:boto:Method: PUT 2023-09-22T21:17:55.811 DEBUG:boto:Path: /ffzdqu-43/ 2023-09-22T21:17:55.811 DEBUG:boto:Data: 2023-09-22T21:17:55.811 DEBUG:boto:Headers: {} 2023-09-22T21:17:55.811 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8001 2023-09-22T21:17:55.811 DEBUG:boto:Port: 8001 2023-09-22T21:17:55.811 DEBUG:boto:Params: {} 2023-09-22T21:17:55.811 DEBUG:boto:Token: None 2023-09-22T21:17:55.811 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:17:55 GMT /ffzdqu-43/ 2023-09-22T21:17:55.812 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:05Cf431zyK1jMpCJiua7g4vNvHg= 2023-09-22T21:17:55.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:17:55 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:05Cf431zyK1jMpCJiua7g4vNvHg=', 'Content-Length': '0'} 2023-09-22T21:17:55.846 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000097932c14a01ba271-00650e0483-116e-b2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:17:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:17:55.846 DEBUG:boto:path=/ffzdqu-40/ 2023-09-22T21:17:55.846 DEBUG:boto:auth_path=/ffzdqu-40/ 2023-09-22T21:17:55.846 DEBUG:boto:path=/ffzdqu-40/?policy 2023-09-22T21:17:55.846 DEBUG:boto:auth_path=/ffzdqu-40/?policy 2023-09-22T21:17:55.847 DEBUG:boto:Method: PUT 2023-09-22T21:17:55.847 DEBUG:boto:Path: /ffzdqu-40/?policy 2023-09-22T21:17:55.847 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-09-22T21:17:55.847 DEBUG:boto:Headers: {} 2023-09-22T21:17:55.847 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:17:55.847 DEBUG:boto:Port: 8000 2023-09-22T21:17:55.847 DEBUG:boto:Params: {} 2023-09-22T21:17:55.847 DEBUG:boto:Token: None 2023-09-22T21:17:55.847 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:17:55 GMT /ffzdqu-40/?policy 2023-09-22T21:17:55.848 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/TwSePh8O26NQ9pJH+PArkT5B5w= 2023-09-22T21:17:55.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:17:55 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/TwSePh8O26NQ9pJH+PArkT5B5w=', 'Content-Length': '97'} 2023-09-22T21:17:55.853 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005ff1c497737dc34d-00650e0483-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:17:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:17:55.854 DEBUG:boto:path=/ffzdqu-40/ 2023-09-22T21:17:55.854 DEBUG:boto:auth_path=/ffzdqu-40/ 2023-09-22T21:17:55.854 DEBUG:boto:path=/ffzdqu-40/?policy 2023-09-22T21:17:55.854 DEBUG:boto:auth_path=/ffzdqu-40/?policy 2023-09-22T21:17:55.854 DEBUG:boto:Method: GET 2023-09-22T21:17:55.854 DEBUG:boto:Path: /ffzdqu-40/?policy 2023-09-22T21:17:55.854 DEBUG:boto:Data: 2023-09-22T21:17:55.855 DEBUG:boto:Headers: {} 2023-09-22T21:17:55.855 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:17:55.855 DEBUG:boto:Port: 8000 2023-09-22T21:17:55.855 DEBUG:boto:Params: {} 2023-09-22T21:17:55.855 DEBUG:boto:Token: None 2023-09-22T21:17:55.855 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:17:55 GMT /ffzdqu-40/?policy 2023-09-22T21:17:55.855 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1phkmAJuINoGaApxTf085p/9Y7k= 2023-09-22T21:17:55.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:17:55 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1phkmAJuINoGaApxTf085p/9Y7k=', 'Content-Length': '0'} 2023-09-22T21:17:55.856 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b114bad0cb99b63e-00650e0483-165f-a1'), ('Content-Type', 'application/json'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '97'), ('Date', 'Fri, 22 Sep 2023 21:17:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:17:55.857 DEBUG:boto:path=/ffzdqu-41/ 2023-09-22T21:17:55.857 DEBUG:boto:auth_path=/ffzdqu-41/ 2023-09-22T21:17:55.857 DEBUG:boto:path=/ffzdqu-41/?policy 2023-09-22T21:17:55.857 DEBUG:boto:auth_path=/ffzdqu-41/?policy 2023-09-22T21:17:55.857 DEBUG:boto:Method: PUT 2023-09-22T21:17:55.857 DEBUG:boto:Path: /ffzdqu-41/?policy 2023-09-22T21:17:55.857 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-09-22T21:17:55.857 DEBUG:boto:Headers: {} 2023-09-22T21:17:55.857 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:17:55.858 DEBUG:boto:Port: 8001 2023-09-22T21:17:55.858 DEBUG:boto:Params: {} 2023-09-22T21:17:55.858 DEBUG:boto:Token: None 2023-09-22T21:17:55.858 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:17:55 GMT /ffzdqu-41/?policy 2023-09-22T21:17:55.858 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QMO/vKY53x+f6VDEWDhM/0CQTU4= 2023-09-22T21:17:55.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:17:55 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QMO/vKY53x+f6VDEWDhM/0CQTU4=', 'Content-Length': '97'} 2023-09-22T21:17:56.877 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000069150c8d25837bc9-00650e0483-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:17:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:17:56.878 DEBUG:boto:path=/ffzdqu-41/ 2023-09-22T21:17:56.878 DEBUG:boto:auth_path=/ffzdqu-41/ 2023-09-22T21:17:56.878 DEBUG:boto:path=/ffzdqu-41/?policy 2023-09-22T21:17:56.878 DEBUG:boto:auth_path=/ffzdqu-41/?policy 2023-09-22T21:17:56.878 DEBUG:boto:Method: GET 2023-09-22T21:17:56.878 DEBUG:boto:Path: /ffzdqu-41/?policy 2023-09-22T21:17:56.878 DEBUG:boto:Data: 2023-09-22T21:17:56.878 DEBUG:boto:Headers: {} 2023-09-22T21:17:56.878 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:17:56.879 DEBUG:boto:Port: 8001 2023-09-22T21:17:56.879 DEBUG:boto:Params: {} 2023-09-22T21:17:56.879 DEBUG:boto:Token: None 2023-09-22T21:17:56.879 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:17:56 GMT /ffzdqu-41/?policy 2023-09-22T21:17:56.879 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HfzzJf68OKtNaqbCu62Vre4f/0c= 2023-09-22T21:17:56.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:17:56 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HfzzJf68OKtNaqbCu62Vre4f/0c=', 'Content-Length': '0'} 2023-09-22T21:17:56.881 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005a2000beeda35740-00650e0484-165b-a2'), ('Content-Type', 'application/json'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '97'), ('Date', 'Fri, 22 Sep 2023 21:17:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:17:56.881 DEBUG:boto:path=/ffzdqu-42/ 2023-09-22T21:17:56.881 DEBUG:boto:auth_path=/ffzdqu-42/ 2023-09-22T21:17:56.881 DEBUG:boto:path=/ffzdqu-42/?policy 2023-09-22T21:17:56.882 DEBUG:boto:auth_path=/ffzdqu-42/?policy 2023-09-22T21:17:56.882 DEBUG:boto:Method: PUT 2023-09-22T21:17:56.882 DEBUG:boto:Path: /ffzdqu-42/?policy 2023-09-22T21:17:56.882 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-09-22T21:17:56.882 DEBUG:boto:Headers: {} 2023-09-22T21:17:56.882 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8000 2023-09-22T21:17:56.882 DEBUG:boto:Port: 8000 2023-09-22T21:17:56.882 DEBUG:boto:Params: {} 2023-09-22T21:17:56.882 DEBUG:boto:Token: None 2023-09-22T21:17:56.883 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:17:56 GMT /ffzdqu-42/?policy 2023-09-22T21:17:56.883 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xCIKa5sIEsMNFiuZxTNK5xzXnmo= 2023-09-22T21:17:56.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:17:56 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xCIKa5sIEsMNFiuZxTNK5xzXnmo=', 'Content-Length': '97'} 2023-09-22T21:17:57.904 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000027f06c5512ef7bd5-00650e0484-1172-b1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:17:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:17:57.904 DEBUG:boto:path=/ffzdqu-42/ 2023-09-22T21:17:57.905 DEBUG:boto:auth_path=/ffzdqu-42/ 2023-09-22T21:17:57.905 DEBUG:boto:path=/ffzdqu-42/?policy 2023-09-22T21:17:57.905 DEBUG:boto:auth_path=/ffzdqu-42/?policy 2023-09-22T21:17:57.905 DEBUG:boto:Method: GET 2023-09-22T21:17:57.905 DEBUG:boto:Path: /ffzdqu-42/?policy 2023-09-22T21:17:57.905 DEBUG:boto:Data: 2023-09-22T21:17:57.905 DEBUG:boto:Headers: {} 2023-09-22T21:17:57.905 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8000 2023-09-22T21:17:57.905 DEBUG:boto:Port: 8000 2023-09-22T21:17:57.906 DEBUG:boto:Params: {} 2023-09-22T21:17:57.906 DEBUG:boto:Token: None 2023-09-22T21:17:57.906 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:17:57 GMT /ffzdqu-42/?policy 2023-09-22T21:17:57.906 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AJWKjnY7FZUH0aUt4WMh+bAuxkI= 2023-09-22T21:17:57.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:17:57 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AJWKjnY7FZUH0aUt4WMh+bAuxkI=', 'Content-Length': '0'} 2023-09-22T21:17:57.908 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000305fe6148e75b457-00650e0485-1172-b1'), ('Content-Type', 'application/json'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '97'), ('Date', 'Fri, 22 Sep 2023 21:17:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:17:57.908 DEBUG:boto:path=/ffzdqu-43/ 2023-09-22T21:17:57.908 DEBUG:boto:auth_path=/ffzdqu-43/ 2023-09-22T21:17:57.908 DEBUG:boto:path=/ffzdqu-43/?policy 2023-09-22T21:17:57.908 DEBUG:boto:auth_path=/ffzdqu-43/?policy 2023-09-22T21:17:57.908 DEBUG:boto:Method: PUT 2023-09-22T21:17:57.909 DEBUG:boto:Path: /ffzdqu-43/?policy 2023-09-22T21:17:57.909 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-09-22T21:17:57.909 DEBUG:boto:Headers: {} 2023-09-22T21:17:57.909 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8001 2023-09-22T21:17:57.909 DEBUG:boto:Port: 8001 2023-09-22T21:17:57.909 DEBUG:boto:Params: {} 2023-09-22T21:17:57.909 DEBUG:boto:Token: None 2023-09-22T21:17:57.909 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:17:57 GMT /ffzdqu-43/?policy 2023-09-22T21:17:57.909 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ds/ENl2+pNhEplxDQGW0Ksvg1eQ= 2023-09-22T21:17:57.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:17:57 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ds/ENl2+pNhEplxDQGW0Ksvg1eQ=', 'Content-Length': '97'} 2023-09-22T21:17:58.932 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000014e7acf6987a79d8-00650e0485-116e-b2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:17:58.932 DEBUG:boto:path=/ffzdqu-43/ 2023-09-22T21:17:58.932 DEBUG:boto:auth_path=/ffzdqu-43/ 2023-09-22T21:17:58.932 DEBUG:boto:path=/ffzdqu-43/?policy 2023-09-22T21:17:58.932 DEBUG:boto:auth_path=/ffzdqu-43/?policy 2023-09-22T21:17:58.932 DEBUG:boto:Method: GET 2023-09-22T21:17:58.933 DEBUG:boto:Path: /ffzdqu-43/?policy 2023-09-22T21:17:58.933 DEBUG:boto:Data: 2023-09-22T21:17:58.933 DEBUG:boto:Headers: {} 2023-09-22T21:17:58.933 DEBUG:boto:Host: smithi204.front.sepia.ceph.com:8001 2023-09-22T21:17:58.933 DEBUG:boto:Port: 8001 2023-09-22T21:17:58.933 DEBUG:boto:Params: {} 2023-09-22T21:17:58.933 DEBUG:boto:Token: None 2023-09-22T21:17:58.933 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:17:58 GMT /ffzdqu-43/?policy 2023-09-22T21:17:58.933 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LvKKe+DV7QxbmkeKSVryejbda+w= 2023-09-22T21:17:58.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:17:58 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LvKKe+DV7QxbmkeKSVryejbda+w=', 'Content-Length': '0'} 2023-09-22T21:17:58.935 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005a4d5cebb0eb0371-00650e0486-116e-b2'), ('Content-Type', 'application/json'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '97'), ('Date', 'Fri, 22 Sep 2023 21:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:17:58.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2023-09-22T21:17:58.936 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-44 2023-09-22T21:17:58.936 DEBUG:boto:path=/ffzdqu-44/ 2023-09-22T21:17:58.936 DEBUG:boto:auth_path=/ffzdqu-44/ 2023-09-22T21:17:58.936 DEBUG:boto:Method: PUT 2023-09-22T21:17:58.936 DEBUG:boto:Path: /ffzdqu-44/ 2023-09-22T21:17:58.936 DEBUG:boto:Data: 2023-09-22T21:17:58.936 DEBUG:boto:Headers: {} 2023-09-22T21:17:58.937 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:17:58.937 DEBUG:boto:Port: 8000 2023-09-22T21:17:58.937 DEBUG:boto:Params: {} 2023-09-22T21:17:58.937 DEBUG:boto:Token: None 2023-09-22T21:17:58.937 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:17:58 GMT /ffzdqu-44/ 2023-09-22T21:17:58.937 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:07LQtgBiF4baLyHIt5bw7Bywi0c= 2023-09-22T21:17:58.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:17:58 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:07LQtgBiF4baLyHIt5bw7Bywi0c=', 'Content-Length': '0'} 2023-09-22T21:17:58.980 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001153ed04748c2439-00650e0486-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:17:58.980 INFO:rgw_multi.tests:create bucket zone=a2 name=ffzdqu-45 2023-09-22T21:17:58.980 DEBUG:boto:path=/ffzdqu-45/ 2023-09-22T21:17:58.980 DEBUG:boto:auth_path=/ffzdqu-45/ 2023-09-22T21:17:58.980 DEBUG:boto:Method: PUT 2023-09-22T21:17:58.981 DEBUG:boto:Path: /ffzdqu-45/ 2023-09-22T21:17:58.981 DEBUG:boto:Data: 2023-09-22T21:17:58.981 DEBUG:boto:Headers: {} 2023-09-22T21:17:58.981 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:17:58.981 DEBUG:boto:Port: 8001 2023-09-22T21:17:58.981 DEBUG:boto:Params: {} 2023-09-22T21:17:58.981 DEBUG:boto:Token: None 2023-09-22T21:17:58.981 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:17:58 GMT /ffzdqu-45/ 2023-09-22T21:17:58.981 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Dljgbd2zo885E7SbwrrxMrNPLnw= 2023-09-22T21:17:58.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:17:58 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Dljgbd2zo885E7SbwrrxMrNPLnw=', 'Content-Length': '0'} 2023-09-22T21:17:59.049 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007024475cfb02f327-00650e0486-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:17:59.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:17:59.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:17:59.050 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:17:59.089 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:17:59.089 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:17:59.129 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:17:59.147 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695417479.028394_332.1","last_update":"2023-09-22T21:17:59.028394Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:17:59.147 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:17:59.147 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:17:59.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:17:59.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:17:59.148 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:17:59.225 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:17:59.226 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:17:59.262 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:17:59.264+0000 7fa0be192a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:17:59.263 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:17:59.264+0000 7fa0be192a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:17:59.280 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:17:59Z"} 2023-09-22T21:17:59.280 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:17:59Z"} 2023-09-22T21:17:59.281 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:17:59.281 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:17:59.281 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:17:59.281 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:18:04.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:04.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:04.283 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:18:04.326 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:18:04.326 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:18:04.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:04.364+0000 7fe09b04fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:04.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:04.368+0000 7fe09b04fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:04.381 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:04Z"} 2023-09-22T21:18:04.381 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:04Z"} 2023-09-22T21:18:04.381 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:18:04.382 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:04.382 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:04.382 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:18:09.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:09.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:09.384 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:18:09.428 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:18:09.428 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:18:09.466 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:09.468+0000 7f8dd488fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:09.467 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:09.472+0000 7f8dd488fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:09.485 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:09Z"} 2023-09-22T21:18:09.485 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:09Z"} 2023-09-22T21:18:09.485 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:18:09.485 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:09.486 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:09.486 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:18:14.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:14.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:14.487 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:18:14.531 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:18:14.531 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:18:14.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:14.571+0000 7f82ac154a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:14.569 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:14.571+0000 7f82ac154a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:14.592 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:14Z"} 2023-09-22T21:18:14.593 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:14Z"} 2023-09-22T21:18:14.593 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:18:14.593 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:14.593 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:14.593 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:18:19.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:19.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:19.595 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:18:19.638 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:18:19.639 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:18:19.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:19.675+0000 7f5395ff4a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:19.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:19.679+0000 7f5395ff4a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:19.694 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:19Z"} 2023-09-22T21:18:19.694 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:19Z"} 2023-09-22T21:18:19.694 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:18:19.694 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:19.695 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:19.695 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:18:24.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:24.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:24.696 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:18:24.740 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:18:24.740 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:18:24.776 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:24.779+0000 7f0798780a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:24.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:24.779+0000 7f0798780a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:24.795 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:24Z"} 2023-09-22T21:18:24.796 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:24Z"} 2023-09-22T21:18:24.796 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:18:24.796 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:24.796 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:24.796 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:18:29.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:29.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:29.798 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:18:29.841 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:18:29.841 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:18:29.880 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:29.883+0000 7f022c1e2a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:29.881 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:29.883+0000 7f022c1e2a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:29.898 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:29Z"} 2023-09-22T21:18:29.899 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:29Z"} 2023-09-22T21:18:29.899 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:18:29.899 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:29.899 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:29.899 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:18:34.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:34.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:34.900 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:18:34.944 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:18:34.944 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:18:34.983 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:34.983+0000 7f2280364a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:34.984 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:34.987+0000 7f2280364a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:35.001 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:34Z"} 2023-09-22T21:18:35.001 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:34Z"} 2023-09-22T21:18:35.001 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:18:35.002 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:35.002 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:35.002 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:18:40.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:40.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:40.003 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:18:40.047 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:18:40.047 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:18:40.085 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:40.087+0000 7fb36e2aba40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:40.086 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:40.087+0000 7fb36e2aba40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:40.104 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:40Z"} 2023-09-22T21:18:40.104 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:40Z"} 2023-09-22T21:18:40.105 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:18:40.105 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:40.105 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:40.105 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:18:45.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:45.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:45.106 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:18:45.149 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:18:45.149 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:18:45.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:45.187+0000 7f2adf7f2a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:45.187 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:45.187+0000 7f2adf7f2a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:45.204 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:45Z"} 2023-09-22T21:18:45.205 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:45Z"} 2023-09-22T21:18:45.205 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:18:45.205 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:45.205 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:45.205 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:18:50.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:50.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:50.207 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:18:50.250 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:18:50.251 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:18:50.288 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:50.291+0000 7fc80a38ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:50.289 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:50.291+0000 7fc80a38ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:50.307 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:50Z"} 2023-09-22T21:18:50.307 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:50Z"} 2023-09-22T21:18:50.307 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:18:50.308 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:50.308 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:50.308 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:18:55.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:55.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:18:55.309 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:18:55.352 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:18:55.352 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:18:55.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:55.391+0000 7ff4dd9cca40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:55.390 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:18:55.391+0000 7ff4dd9cca40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:18:55.407 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:55Z"} 2023-09-22T21:18:55.408 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:18:55Z"} 2023-09-22T21:18:55.408 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:18:55.408 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:55.408 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:18:55.408 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:19:00.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:00.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:00.409 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:19:00.453 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:19:00.453 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:19:00.491 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:00.490+0000 7ff5917b9a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:00.492 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:00.494+0000 7ff5917b9a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:00.509 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:00Z"} 2023-09-22T21:19:00.510 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:00Z"} 2023-09-22T21:19:00.510 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:19:00.510 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:00.510 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:00.510 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:19:05.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:05.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:05.512 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:19:05.554 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:19:05.554 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:19:05.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:05.590+0000 7f94a331ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:05.591 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:05.590+0000 7f94a331ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:05.609 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:05Z"} 2023-09-22T21:19:05.609 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:05Z"} 2023-09-22T21:19:05.609 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:19:05.609 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:05.609 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:05.610 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:19:10.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:10.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:10.611 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:19:10.654 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:19:10.654 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:19:10.693 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:10.694+0000 7f846dcfda40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:10.694 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:10.694+0000 7f846dcfda40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:10.712 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:10Z"} 2023-09-22T21:19:10.712 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:10Z"} 2023-09-22T21:19:10.712 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:19:10.712 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:10.713 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:10.713 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:19:15.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:15.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:15.715 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:19:15.758 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:19:15.759 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:19:15.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:15.798+0000 7fcbe8561a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:15.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:15.798+0000 7fcbe8561a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:15.816 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:15Z"} 2023-09-22T21:19:15.817 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:15Z"} 2023-09-22T21:19:15.817 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:19:15.817 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:15.817 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:15.817 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:19:20.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:20.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:20.818 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:19:20.865 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:19:20.866 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:19:20.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:20.902+0000 7f11b6118a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:20.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:20.906+0000 7f11b6118a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:20.922 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:20Z"} 2023-09-22T21:19:20.923 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:20Z"} 2023-09-22T21:19:20.923 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:19:20.924 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:20.924 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:20.924 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:19:25.925 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:25.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:25.926 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:19:25.969 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:19:25.969 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:19:26.006 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:26.006+0000 7f36fda4ca40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:26.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:26.006+0000 7f36fda4ca40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:26.025 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:26Z"} 2023-09-22T21:19:26.026 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:26Z"} 2023-09-22T21:19:26.026 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:19:26.026 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:26.026 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:26.026 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:19:31.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:31.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:31.028 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:19:31.073 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:19:31.073 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:19:31.110 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:31.110+0000 7f3bd3a85a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:31.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:31.110+0000 7f3bd3a85a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:31.129 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:31Z"} 2023-09-22T21:19:31.130 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:31Z"} 2023-09-22T21:19:31.130 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:19:31.130 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:31.130 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:31.130 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:19:36.131 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:36.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:36.131 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:19:36.176 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:19:36.176 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:19:36.214 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:36.214+0000 7faa23fc9a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:36.215 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:36.214+0000 7faa23fc9a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:36.232 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:36Z"} 2023-09-22T21:19:36.233 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:36Z"} 2023-09-22T21:19:36.233 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:19:36.233 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:36.233 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:36.233 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:19:41.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:41.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:41.235 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:19:41.277 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:19:41.278 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:19:41.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:41.314+0000 7fac183b0a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:41.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:41.314+0000 7fac183b0a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:41.333 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:41Z"} 2023-09-22T21:19:41.333 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:41Z"} 2023-09-22T21:19:41.334 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:19:41.334 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:41.334 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:41.334 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:19:46.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:46.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:46.336 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:19:46.380 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:19:46.380 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:19:46.418 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:46.418+0000 7f18498dea40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:46.419 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:46.418+0000 7f18498dea40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:46.437 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:46Z"} 2023-09-22T21:19:46.438 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:46Z"} 2023-09-22T21:19:46.438 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:19:46.438 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:46.438 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:46.438 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:19:51.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:51.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:51.440 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:19:51.484 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:19:51.484 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:19:51.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:51.521+0000 7f87c9553a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:51.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:51.521+0000 7f87c9553a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:51.540 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:51Z"} 2023-09-22T21:19:51.540 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:51Z"} 2023-09-22T21:19:51.541 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:19:51.541 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:51.541 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:51.541 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:19:56.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:56.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:19:56.543 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:19:56.594 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:19:56.594 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:19:56.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:56.633+0000 7f88941daa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:56.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:19:56.633+0000 7f88941daa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:19:56.652 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:56Z"} 2023-09-22T21:19:56.653 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417223.928828_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:13:43.928828Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:19:56Z"} 2023-09-22T21:19:56.653 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417223.928828_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:13:43.928828Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:19:56.653 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:56.653 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417223.928828_219.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:19:56.653 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417223.928828_219.1 2023-09-22T21:20:01.655 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:01.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:01.655 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:20:01.698 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:20:01.699 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:20:01.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:20:01.733+0000 7fb35d2b9a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:20:01.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:20:01.737+0000 7fb35d2b9a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:20:01.754 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417475.703115_300.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:17:55.703115Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:20:01Z"} 2023-09-22T21:20:01.754 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417475.703115_300.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:17:55.703115Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:20:01Z"} 2023-09-22T21:20:01.754 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417475.703115_300.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:17:55.703115Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:20:01.754 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:20:01.755 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417475.703115_300.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:20:01.755 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417475.703115_300.1 2023-09-22T21:20:06.755 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:06.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:06.756 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:20:06.800 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:20:06.800 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:20:06.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:20:06.845+0000 7f23d2f73a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:20:06.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:20:06.845+0000 7f23d2f73a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:20:06.859 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417475.568653_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:17:55.568653Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:20:06Z"} 2023-09-22T21:20:06.859 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417475.568653_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:17:55.568653Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:20:06Z"} 2023-09-22T21:20:06.859 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417475.568653_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:17:55.568653Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:20:06.859 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:20:06.859 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417475.568653_281.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:20:06.859 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417475.568653_281.1 2023-09-22T21:20:11.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:11.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:11.861 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:20:11.904 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:20:11.905 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:20:11.941 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:20:11.945+0000 7f8e8e076a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:20:11.942 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:20:11.945+0000 7f8e8e076a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:20:11.960 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417475.566947_280.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:17:55.566947Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:20:11Z"} 2023-09-22T21:20:11.960 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417475.566947_280.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:17:55.566947Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:20:11Z"} 2023-09-22T21:20:11.960 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417475.566947_280.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:17:55.566947Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:20:11.960 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:20:11.961 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417475.566947_280.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:20:11.961 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417475.566947_280.1 2023-09-22T21:20:16.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:16.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:16.962 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:20:17.005 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:20:17.006 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:20:17.043 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:20:17.045+0000 7f1b1d91aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:20:17.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:20:17.045+0000 7f1b1d91aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:20:17.061 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417475.604534_286.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:17:55.604534Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:20:17Z"} 2023-09-22T21:20:17.062 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417475.604534_286.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:17:55.604534Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:20:17Z"} 2023-09-22T21:20:17.062 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417475.604534_286.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:17:55.604534Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:20:17.062 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:20:17.062 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417475.604534_286.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:20:17.062 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417479.028394_332.1 target=1_1695417475.604534_286.1 2023-09-22T21:20:22.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:22.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:22.064 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:20:22.108 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:20:22.108 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:20:22.146 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:20:22.149+0000 7f2a2d2a8a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:20:22.147 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:20:22.149+0000 7f2a2d2a8a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:20:22.164 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417479.028394_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:17:59.028394Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:20:22Z"} 2023-09-22T21:20:22.165 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417479.028394_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:17:59.028394Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:20:22Z"} 2023-09-22T21:20:22.165 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417479.028394_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:17:59.028394Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:20:22.165 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:20:22.165 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417479.028394_332.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:20:22.165 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:20:22.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'ffzdqu-44', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:20:22.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'ffzdqu-44', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:20:22.166 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket ffzdqu-44 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:20:22.250 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:20:22.251 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:20:22.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'ffzdqu-45', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:20:22.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'ffzdqu-45', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:20:22.336 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket ffzdqu-45 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:20:22.412 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:20:22.412 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:20:22.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:22.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:22.499 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:20:22.573 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:20:22.573 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:20:22.630 INFO:teuthology.orchestra.run.smithi160.stdout:["bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:0","obj_delete_at_hint.0000000078","meta.history","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:6","data_log.1.ZBHFLxr9ILW.0","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:1","obj_delete_at_hint.0000000026","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:8","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:5","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:2","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:2","sync.error-log.8","obj_delete_at_hint.0000000028","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:7","bucket.full-sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:2","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:2","obj_delete_at_hint.0000000040","sync.error-log.6","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:1","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:8","obj_delete_at_hint.0000000095","bucket.full-sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:4","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:5","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:9","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:3","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:0","obj_delete_at_hint.0000000024","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:5","obj_delete_at_hint.0000000000","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:10","obj_delete_at_hint.0000000031","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:10","data_log.3","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:6","meta.log.03d00cc5-7ad7-4abf-a475-d9ffc8e37106.2","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:10","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:9","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:4","data_loggenerations_metadata","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:9","obj_delete_at_hint.0000000076","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:1","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:10","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:8","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:2","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:2","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:6","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:4","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:3","obj_delete_at_hint.0000000110","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:6","obj_delete_at_hint.0000000096","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:0","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:6","datalog.sync-status.shard.2b721fee-a149-45da-925b-502d5a3d6853.1","bucket.full-sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:1","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:0","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:5","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:7","bilog.trim","obj_delete_at_hint.0000000016","obj_delete_at_hint.0000000041","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:2","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:6","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:3","bucket.full-sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:0","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:7","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:4","sync.error-log.4","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:0","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:3","obj_delete_at_hint.0000000014","bucket.full-sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:9","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:6","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","bucket.full-sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:6","obj_delete_at_hint.0000000074","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:8","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:3","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:6","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","bucket.full-sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13","data_log.1","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:1","datalog.sync-status.shard.2b721fee-a149-45da-925b-502d5a3d6853.3.retry","data_log.2.nmmwAieI+R8.0","bucket.full-sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12","bucket.full-sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","datalog.sync-status.shard.2b721fee-a149-45da-925b-502d5a3d6853.1.retry","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:7","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:2","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:5","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:5","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:5","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:3","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:4","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:10","obj_delete_at_hint.0000000065","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:3","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.full-sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:6","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:4","sync.error-log.7","sync.error-log.3","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:10","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:0","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:0","datalog.sync-status.2b721fee-a149-45da-925b-502d5a3d6853","datalog.sync-status.shard.2b721fee-a149-45da-925b-502d5a3d6853.0","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:5","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:1","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:10","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:7","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:4","obj_delete_at_hint.0000000116","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:3","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:9","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:3","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:9","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:2","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.full-sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:4","obj_delete_at_hint.0000000066","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:7","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:1","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:10","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:1","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:1","obj_delete_at_hint.0000000073","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:10","bucket.full-sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10","obj_delete_at_hint.0000000038","obj_delete_at_hint.0000000118","obj_delete_at_hint.0000000036","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:7","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:9","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:2","obj_delete_at_hint.0000000103","obj_delete_at_hint.0000000119","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:8","obj_delete_at_hint.0000000098","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:1","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:0","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:1","obj_delete_at_hint.0000000027","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:7","obj_delete_at_hint.0000000019","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:2","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:3","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:7","obj_delete_at_hint.0000000039","meta.log.03d00cc5-7ad7-4abf-a475-d9ffc8e37106.3","bucket.full-sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:0","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:5","obj_delete_at_hint.0000000100","datalog.sync-status.shard.2b721fee-a149-45da-925b-502d5a3d6853.3","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:3","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:10","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:4","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:7","obj_delete_at_hint.0000000093","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:8","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:9","obj_delete_at_hint.0000000057","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:9","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:10","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:4","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:2","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:7","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:8","sync.error-log.2","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:0","data_log.0","datalog.sync-status.shard.2b721fee-a149-45da-925b-502d5a3d6853.2","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:10","bucket.full-sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:5","obj_delete_at_hint.0000000012","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:4","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:1","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:8","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:4","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:6","obj_delete_at_hint.0000000111","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:1","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:4","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","bucket.full-sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:5","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:5","obj_delete_at_hint.0000000072","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:5","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:8","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","sync.error-log.1","obj_delete_at_hint.0000000033","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:2","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:8","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:0","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:6","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:5","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:7","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:7","obj_delete_at_hint.0000000108","datalog.sync-status.shard.2b721fee-a149-45da-925b-502d5a3d6853.0.retry","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:2","obj_delete_at_hint.0000000094","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:4","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:6","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:6","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:4","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:9","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:1","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:8","obj_delete_at_hint.0000000013","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:2","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:6","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:7","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:3","obj_delete_at_hint.0000000051","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:9","obj_delete_at_hint.0000000106","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:9","obj_delete_at_hint.0000000017","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:3","obj_delete_at_hint.0000000023","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:9","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:10","bucket.full-sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14","obj_delete_at_hint.0000000107","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:8","obj_delete_at_hint.0000000046","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:9","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:2","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:5","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:1","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:0","data_log.2","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:8","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:10","obj_delete_at_hint.0000000067","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:4","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:2","sync.error-log.9","obj_delete_at_hint.0000000086","sync.error-log.5","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:0","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:8","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:10","data_log.0.lOdK4Ugt2ai.0","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:8","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:7","meta.log.03d00cc5-7ad7-4abf-a475-d9ffc8e37106.0","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:3","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:3","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:5","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:0","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:0","data_log.3.INzdnx96Z23.0","obj_delete_at_hint.0000000124","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:6","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:0","obj_delete_at_hint.0000000001","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:10","obj_delete_at_hint.0000000126","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:8","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:3","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:8","bucket.full-sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:4","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:6","obj_delete_at_hint.0000000097","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:9","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:9","obj_delete_at_hint.0000000042","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:5","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:10","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:7","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:9","obj_delete_at_hint.0000000075","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:1","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:7","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:1","bucket.sync-status.2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:3"] 2023-09-22T21:20:22.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:22.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:22.648 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:20:22.722 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:20:22.722 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:20:22.783 INFO:teuthology.orchestra.run.smithi160.stdout:["obj_delete_at_hint.0000000078","meta.history","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:3","bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:8","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:9","sync.error-log.8","obj_delete_at_hint.0000000028","obj_delete_at_hint.0000000040","bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:6","sync.error-log.6","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:8","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:1","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:8","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:2","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:7","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:0","obj_delete_at_hint.0000000035","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:0","obj_delete_at_hint.0000000037","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:1","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:7","bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14","bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:7","data_log.3","datalog.sync-status.shard.f132bdb9-20a7-4073-a932-8f99da20691e.2","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:6","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:4","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:1","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:3","meta.log.03d00cc5-7ad7-4abf-a475-d9ffc8e37106.2","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:1","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:6","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:5","data_loggenerations_metadata","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:9","datalog.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:4","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:5","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:9","obj_delete_at_hint.0000000005","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:1","obj_delete_at_hint.0000000011","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:3","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:10","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:3","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:4","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:7","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:9","obj_delete_at_hint.0000000088","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:10","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:4","obj_delete_at_hint.0000000110","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:1","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:7","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:9","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:2","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:6","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:5","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:3","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bilog.trim","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:0","obj_delete_at_hint.0000000016","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:6","obj_delete_at_hint.0000000041","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:6","obj_delete_at_hint.0000000018","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:2","obj_delete_at_hint.0000000092","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:7","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:0","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:9","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:7","data_log.3.zFAoPhKdVVF.0","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:3","sync.error-log.4","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:10","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:1","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:8","obj_delete_at_hint.0000000014","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:8","datalog.sync-status.shard.f132bdb9-20a7-4073-a932-8f99da20691e.1","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:8","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:8","obj_delete_at_hint.0000000064","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:4","obj_delete_at_hint.0000000071","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:3","obj_delete_at_hint.0000000074","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:8","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:3","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:2","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","data_log.1","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:8","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:6","obj_delete_at_hint.0000000125","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:5","obj_delete_at_hint.0000000082","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:5","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:9","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:6","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:7","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:6","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:1","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:6","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:4","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:6","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:9","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:8","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:0","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:8","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:7","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:0","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","sync.error-log.7","sync.error-log.3","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:0","data_log.2.OXwLcN83V3Q.0","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:0","mdlog.sync-status.shard.2","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:8","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:3","obj_delete_at_hint.0000000045","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:0","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:7","obj_delete_at_hint.0000000002","obj_delete_at_hint.0000000116","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:4","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:5","obj_delete_at_hint.0000000034","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:2","obj_delete_at_hint.0000000101","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:2","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:5","mdlog.sync-status.shard.1","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:2","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:2","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:9","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:6","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:4","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:10","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:4","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:10","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:3","obj_delete_at_hint.0000000066","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:5","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:0","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18","obj_delete_at_hint.0000000073","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:7","obj_delete_at_hint.0000000038","bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:0","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:0","obj_delete_at_hint.0000000118","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:9","obj_delete_at_hint.0000000036","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:3","obj_delete_at_hint.0000000103","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:4","mdlog.sync-status.shard.3","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:10","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:5","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:0","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:1","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:7","mdlog.sync-status","obj_delete_at_hint.0000000019","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:9","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:2","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:1","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:2","obj_delete_at_hint.0000000039","meta.log.03d00cc5-7ad7-4abf-a475-d9ffc8e37106.3","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:10","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:9","obj_delete_at_hint.0000000100","bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:3","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:5","obj_delete_at_hint.0000000093","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","datalog.sync-status.shard.f132bdb9-20a7-4073-a932-8f99da20691e.1.retry","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:4","bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:0","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:3","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:10","obj_delete_at_hint.0000000054","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:4","obj_delete_at_hint.0000000114","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:9","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:10","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:10","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:8","sync.error-log.2","bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16","data_log.0","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:3","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:10","obj_delete_at_hint.0000000012","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:7","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:8","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:0","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:1","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:10","data_log.1.AL+8ANqQ4bP.0","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:3","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:3","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:2","obj_delete_at_hint.0000000020","bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:7","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:9","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:8","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:7","obj_delete_at_hint.0000000072","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:1","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:5","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","sync.error-log.1","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-14:f132bdb9-20a7-4073-a932-8f99da20691e.4382.14:10","obj_delete_at_hint.0000000033","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:2","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:5","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:8","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:8","obj_delete_at_hint.0000000025","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:10","obj_delete_at_hint.0000000123","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:5","obj_delete_at_hint.0000000108","bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:9","datalog.sync-status.shard.f132bdb9-20a7-4073-a932-8f99da20691e.3.retry","obj_delete_at_hint.0000000094","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:2","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:1","obj_delete_at_hint.0000000013","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:6","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:6","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:4","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:4","obj_delete_at_hint.0000000017","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:1","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:1","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8:6","datalog.sync-status.shard.f132bdb9-20a7-4073-a932-8f99da20691e.0.retry","obj_delete_at_hint.0000000023","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:7","obj_delete_at_hint.0000000107","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:4","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34:9","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:3","obj_delete_at_hint.0000000046","bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-15:f132bdb9-20a7-4073-a932-8f99da20691e.4382.15:7","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:0","bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-34:f132bdb9-20a7-4073-a932-8f99da20691e.4382.34","data_log.2","datalog.sync-status.shard.f132bdb9-20a7-4073-a932-8f99da20691e.0","obj_delete_at_hint.0000000067","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:10","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-32:f132bdb9-20a7-4073-a932-8f99da20691e.4382.32:4","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-12:f132bdb9-20a7-4073-a932-8f99da20691e.4382.12:4","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:5","datalog.sync-status.shard.f132bdb9-20a7-4073-a932-8f99da20691e.3","sync.error-log.9","obj_delete_at_hint.0000000086","sync.error-log.5","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-7:f132bdb9-20a7-4073-a932-8f99da20691e.4382.7:10","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:6","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-17:f132bdb9-20a7-4073-a932-8f99da20691e.4382.17:5","meta.log.03d00cc5-7ad7-4abf-a475-d9ffc8e37106.0","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:1","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:9","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:5","bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-16:f132bdb9-20a7-4073-a932-8f99da20691e.4382.16:10","bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33","bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-8:f132bdb9-20a7-4073-a932-8f99da20691e.4382.8","obj_delete_at_hint.0000000124","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-11:f132bdb9-20a7-4073-a932-8f99da20691e.4382.11:2","obj_delete_at_hint.0000000001","data_log.0.juRq3w627S+.0","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-31:f132bdb9-20a7-4073-a932-8f99da20691e.4382.31:5","obj_delete_at_hint.0000000097","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-10:f132bdb9-20a7-4073-a932-8f99da20691e.4382.10:2","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-9:f132bdb9-20a7-4073-a932-8f99da20691e.4382.9:0","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-13:f132bdb9-20a7-4073-a932-8f99da20691e.4382.13:2","obj_delete_at_hint.0000000042","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-35:f132bdb9-20a7-4073-a932-8f99da20691e.5398.1:6","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-33:f132bdb9-20a7-4073-a932-8f99da20691e.4382.33:1","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-18:f132bdb9-20a7-4073-a932-8f99da20691e.4382.18:2"] 2023-09-22T21:20:22.800 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-09-22T21:20:22.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:22.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:22.801 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:20:22.884 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:20:22.884 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:20:22.938 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"00000000000000000000:00000000000000001211","last_update":"2023-09-22T21:20:22.477842Z"},{"marker":"00000000000000000000:00000000000000004167","last_update":"2023-09-22T21:20:22.479134Z"},{"marker":"00000000000000000000:00000000000000003467","last_update":"2023-09-22T21:20:22.480406Z"},{"marker":"00000000000000000000:00000000000000002055","last_update":"2023-09-22T21:20:22.481688Z"}] 2023-09-22T21:20:22.938 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000001211', 1: '00000000000000000000:00000000000000004167', 2: '00000000000000000000:00000000000000003467', 3: '00000000000000000000:00000000000000002055'} 2023-09-22T21:20:22.939 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-22T21:20:22.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:22.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:22.939 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:20:23.015 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:20:23.016 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:20:23.064 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:42.948773Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003324","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:21.905803Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002625","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:15:46.164882Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001210","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:22.975904Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:20:23Z"} 2023-09-22T21:20:23.065 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:42.948773Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003324","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:21.905803Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002625","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:15:46.164882Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001210","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:22.975904Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:20:23Z"} 2023-09-22T21:20:23.065 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:04:42.948773Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003324', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:21.905803Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002625', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:15:46.164882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:22.975904Z'}}] 2023-09-22T21:20:23.065 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001211', 1: '00000000000000000000:00000000000000004167', 2: '00000000000000000000:00000000000000003467', 3: '00000000000000000000:00000000000000002055'} 2023-09-22T21:20:23.065 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003324', 2: '00000000000000000000:00000000000000002625', 3: '00000000000000000000:00000000000000001210'} 2023-09-22T21:20:23.065 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000001211 target=00000000000000000000:00000000000000000651, shard=1 master=00000000000000000000:00000000000000004167 target=00000000000000000000:00000000000000003324, shard=2 master=00000000000000000000:00000000000000003467 target=00000000000000000000:00000000000000002625, shard=3 master=00000000000000000000:00000000000000002055 target=00000000000000000000:00000000000000001210 2023-09-22T21:20:28.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:28.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:28.066 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:20:28.111 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:20:28.112 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:20:28.169 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:42.948773Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003324","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:21.905803Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002625","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:15:46.164882Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001210","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:22.975904Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:20:28Z"} 2023-09-22T21:20:28.169 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:42.948773Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003324","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:21.905803Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002625","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:15:46.164882Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001210","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:22.975904Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:20:28Z"} 2023-09-22T21:20:28.169 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:04:42.948773Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003324', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:21.905803Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002625', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:15:46.164882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:22.975904Z'}}] 2023-09-22T21:20:28.169 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001211', 1: '00000000000000000000:00000000000000004167', 2: '00000000000000000000:00000000000000003467', 3: '00000000000000000000:00000000000000002055'} 2023-09-22T21:20:28.169 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003324', 2: '00000000000000000000:00000000000000002625', 3: '00000000000000000000:00000000000000001210'} 2023-09-22T21:20:28.170 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000001211 target=00000000000000000000:00000000000000000651, shard=1 master=00000000000000000000:00000000000000004167 target=00000000000000000000:00000000000000003324, shard=2 master=00000000000000000000:00000000000000003467 target=00000000000000000000:00000000000000002625, shard=3 master=00000000000000000000:00000000000000002055 target=00000000000000000000:00000000000000001210 2023-09-22T21:20:33.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:33.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:33.171 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:20:33.217 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:20:33.217 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:20:33.273 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:42.948773Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003324","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:21.905803Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002625","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:15:46.164882Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001210","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:22.975904Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:20:33Z"} 2023-09-22T21:20:33.273 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:42.948773Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003324","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:21.905803Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002625","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:15:46.164882Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001210","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:22.975904Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:20:33Z"} 2023-09-22T21:20:33.274 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:04:42.948773Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003324', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:21.905803Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002625', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:15:46.164882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:22.975904Z'}}] 2023-09-22T21:20:33.274 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001211', 1: '00000000000000000000:00000000000000004167', 2: '00000000000000000000:00000000000000003467', 3: '00000000000000000000:00000000000000002055'} 2023-09-22T21:20:33.274 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003324', 2: '00000000000000000000:00000000000000002625', 3: '00000000000000000000:00000000000000001210'} 2023-09-22T21:20:33.274 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000001211 target=00000000000000000000:00000000000000000651, shard=1 master=00000000000000000000:00000000000000004167 target=00000000000000000000:00000000000000003324, shard=2 master=00000000000000000000:00000000000000003467 target=00000000000000000000:00000000000000002625, shard=3 master=00000000000000000000:00000000000000002055 target=00000000000000000000:00000000000000001210 2023-09-22T21:20:36.569 INFO:tasks.rgw.c2.client.0.smithi204.stdout:2023-09-22T21:20:36.570+0000 7f11298ba640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:20:36.570 INFO:tasks.rgw.c2.client.0.smithi204.stdout:2023-09-22T21:20:36.570+0000 7f11410e9640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-09-22T21:20:36.583 INFO:tasks.rgw.c2.client.1.smithi204.stdout:2023-09-22T21:20:36.586+0000 7fb92d9f1640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:20:36.592 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:20:36.597+0000 7ff4815cb640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:20:38.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:38.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:38.293 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:20:38.338 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:20:38.339 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:20:38.394 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001211","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:20:22.477842Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004167","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:20:22.479134Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003467","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:20:22.480406Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002055","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:20:22.481688Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:20:38Z"} 2023-09-22T21:20:38.395 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001211","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:20:22.477842Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004167","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:20:22.479134Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003467","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:20:22.480406Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002055","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:20:22.481688Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:20:38Z"} 2023-09-22T21:20:38.395 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:20:22.477842Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004167', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:20:22.479134Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003467', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:20:22.480406Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002055', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:20:22.481688Z'}}] 2023-09-22T21:20:38.395 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001211', 1: '00000000000000000000:00000000000000004167', 2: '00000000000000000000:00000000000000003467', 3: '00000000000000000000:00000000000000002055'} 2023-09-22T21:20:38.395 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001211', 1: '00000000000000000000:00000000000000004167', 2: '00000000000000000000:00000000000000003467', 3: '00000000000000000000:00000000000000002055'} 2023-09-22T21:20:38.395 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-22T21:20:38.395 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-09-22T21:20:38.396 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:38.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:38.396 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:20:38.475 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:20:38.476 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:20:38.540 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"00000000000000000000:00000000000000000651","last_update":"2023-09-22T21:04:39.605415Z"},{"marker":"00000000000000000000:00000000000000004029","last_update":"2023-09-22T21:12:40.611546Z"},{"marker":"00000000000000000000:00000000000000001640","last_update":"2023-09-22T21:15:46.472411Z"},{"marker":"00000000000000000000:00000000000000001210","last_update":"2023-09-22T21:12:21.945596Z"}] 2023-09-22T21:20:38.540 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004029', 2: '00000000000000000000:00000000000000001640', 3: '00000000000000000000:00000000000000001210'} 2023-09-22T21:20:38.540 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-09-22T21:20:38.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:38.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:38.541 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:20:38.632 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:20:38.632 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:20:38.688 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10476880264182873081},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:39.605415Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004029","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:40.611546Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001640","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:15:46.472411Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001210","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:21.945596Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:20:38Z"} 2023-09-22T21:20:38.688 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10476880264182873081},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000651","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:04:39.605415Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004029","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:40.611546Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001640","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:15:46.472411Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001210","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:12:21.945596Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:20:38Z"} 2023-09-22T21:20:38.689 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:04:39.605415Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004029', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:40.611546Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:15:46.472411Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:12:21.945596Z'}}] 2023-09-22T21:20:38.689 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004029', 2: '00000000000000000000:00000000000000001640', 3: '00000000000000000000:00000000000000001210'} 2023-09-22T21:20:38.689 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004029', 2: '00000000000000000000:00000000000000001640', 3: '00000000000000000000:00000000000000001210'} 2023-09-22T21:20:38.689 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-09-22T21:20:38.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ok 2023-09-22T21:20:38.691 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-46 2023-09-22T21:20:38.691 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:20:38.691 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:20:38.691 DEBUG:boto:Method: PUT 2023-09-22T21:20:38.691 DEBUG:boto:Path: /ffzdqu-46/ 2023-09-22T21:20:38.692 DEBUG:boto:Data: 2023-09-22T21:20:38.692 DEBUG:boto:Headers: {} 2023-09-22T21:20:38.692 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:20:38.692 DEBUG:boto:Port: 8000 2023-09-22T21:20:38.692 DEBUG:boto:Params: {} 2023-09-22T21:20:38.693 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:20:38.693 DEBUG:boto:Token: None 2023-09-22T21:20:38.693 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:20:38 GMT /ffzdqu-46/ 2023-09-22T21:20:38.694 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HvP+ne0w7QDhHgHmVZYrYDr52T0= 2023-09-22T21:20:38.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:20:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HvP+ne0w7QDhHgHmVZYrYDr52T0=', 'Content-Length': '0'} 2023-09-22T21:20:38.728 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004caab3f4836f7769-00650e0526-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:20:38.729 INFO:rgw_multi.tests:create bucket zone=a2 name=ffzdqu-47 2023-09-22T21:20:38.729 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:20:38.729 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:20:38.729 DEBUG:boto:Method: PUT 2023-09-22T21:20:38.729 DEBUG:boto:Path: /ffzdqu-47/ 2023-09-22T21:20:38.729 DEBUG:boto:Data: 2023-09-22T21:20:38.730 DEBUG:boto:Headers: {} 2023-09-22T21:20:38.730 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:20:38.730 DEBUG:boto:Port: 8001 2023-09-22T21:20:38.730 DEBUG:boto:Params: {} 2023-09-22T21:20:38.730 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:20:38.731 DEBUG:boto:Token: None 2023-09-22T21:20:38.731 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:20:38 GMT /ffzdqu-47/ 2023-09-22T21:20:38.731 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D6iSETFVE8t3mx1lAm+ZX+APDso= 2023-09-22T21:20:38.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:20:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D6iSETFVE8t3mx1lAm+ZX+APDso=', 'Content-Length': '0'} 2023-09-22T21:20:38.763 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ad29d98b786e03a2-00650e0526-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:20:38.763 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:20:38.763 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:20:38.763 DEBUG:boto:Method: HEAD 2023-09-22T21:20:38.763 DEBUG:boto:Path: /ffzdqu-46/ 2023-09-22T21:20:38.763 DEBUG:boto:Data: 2023-09-22T21:20:38.764 DEBUG:boto:Headers: {} 2023-09-22T21:20:38.764 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:20:38.764 DEBUG:boto:Port: 8000 2023-09-22T21:20:38.764 DEBUG:boto:Params: {} 2023-09-22T21:20:38.764 DEBUG:boto:Token: None 2023-09-22T21:20:38.764 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:20:38 GMT /ffzdqu-46/ 2023-09-22T21:20:38.765 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9ahSRPMVS4vKbRaN7Iw/jQ60gEg= 2023-09-22T21:20:38.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:20:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9ahSRPMVS4vKbRaN7Iw/jQ60gEg=', 'Content-Length': '0'} 2023-09-22T21:20:38.767 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000140c4d8b973d2309-00650e0526-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:20:38.767 DEBUG:boto:path=/ffzdqu-46/obj1 2023-09-22T21:20:38.767 DEBUG:boto:auth_path=/ffzdqu-46/obj1 2023-09-22T21:20:38.767 DEBUG:boto:Method: PUT 2023-09-22T21:20:38.767 DEBUG:boto:Path: /ffzdqu-46/obj1 2023-09-22T21:20:38.767 DEBUG:boto:Data: 2023-09-22T21:20:38.768 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:20:38.768 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:20:38.768 DEBUG:boto:Port: 8000 2023-09-22T21:20:38.768 DEBUG:boto:Params: {} 2023-09-22T21:20:38.768 DEBUG:boto:Token: None 2023-09-22T21:20:38.768 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:20:38 GMT /ffzdqu-46/obj1 2023-09-22T21:20:38.768 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZLYNXv1nc6uRK2KcSc30KFg+kuI= 2023-09-22T21:20:38.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:20:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZLYNXv1nc6uRK2KcSc30KFg+kuI='} 2023-09-22T21:20:38.775 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3b9702ed8054020-00650e0526-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:20:38.776 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:20:38.776 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:20:38.776 DEBUG:boto:Method: HEAD 2023-09-22T21:20:38.776 DEBUG:boto:Path: /ffzdqu-46/ 2023-09-22T21:20:38.776 DEBUG:boto:Data: 2023-09-22T21:20:38.776 DEBUG:boto:Headers: {} 2023-09-22T21:20:38.776 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:20:38.776 DEBUG:boto:Port: 8000 2023-09-22T21:20:38.777 DEBUG:boto:Params: {} 2023-09-22T21:20:38.777 DEBUG:boto:Token: None 2023-09-22T21:20:38.777 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:20:38 GMT /ffzdqu-46/ 2023-09-22T21:20:38.777 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9ahSRPMVS4vKbRaN7Iw/jQ60gEg= 2023-09-22T21:20:38.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:20:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9ahSRPMVS4vKbRaN7Iw/jQ60gEg=', 'Content-Length': '0'} 2023-09-22T21:20:38.779 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007938f5fcb05956b8-00650e0526-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:20:38.779 DEBUG:boto:path=/ffzdqu-46/obj2 2023-09-22T21:20:38.779 DEBUG:boto:auth_path=/ffzdqu-46/obj2 2023-09-22T21:20:38.780 DEBUG:boto:Method: PUT 2023-09-22T21:20:38.780 DEBUG:boto:Path: /ffzdqu-46/obj2 2023-09-22T21:20:38.780 DEBUG:boto:Data: 2023-09-22T21:20:38.780 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:20:38.780 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:20:38.780 DEBUG:boto:Port: 8000 2023-09-22T21:20:38.780 DEBUG:boto:Params: {} 2023-09-22T21:20:38.780 DEBUG:boto:Token: None 2023-09-22T21:20:38.780 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:20:38 GMT /ffzdqu-46/obj2 2023-09-22T21:20:38.781 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D9Tjr0gi2PyFam1rcGdvxQWylU4= 2023-09-22T21:20:38.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:20:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D9Tjr0gi2PyFam1rcGdvxQWylU4='} 2023-09-22T21:20:38.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000086dcc989f6f77e0-00650e0526-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:20:38.827 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:20:38.827 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:20:38.827 DEBUG:boto:Method: HEAD 2023-09-22T21:20:38.827 DEBUG:boto:Path: /ffzdqu-46/ 2023-09-22T21:20:38.827 DEBUG:boto:Data: 2023-09-22T21:20:38.827 DEBUG:boto:Headers: {} 2023-09-22T21:20:38.827 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:20:38.827 DEBUG:boto:Port: 8000 2023-09-22T21:20:38.828 DEBUG:boto:Params: {} 2023-09-22T21:20:38.828 DEBUG:boto:Token: None 2023-09-22T21:20:38.828 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:20:38 GMT /ffzdqu-46/ 2023-09-22T21:20:38.828 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9ahSRPMVS4vKbRaN7Iw/jQ60gEg= 2023-09-22T21:20:38.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:20:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9ahSRPMVS4vKbRaN7Iw/jQ60gEg=', 'Content-Length': '0'} 2023-09-22T21:20:38.833 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f02a1adcf802e59-00650e0526-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:20:38.833 DEBUG:boto:path=/ffzdqu-46/obj3 2023-09-22T21:20:38.833 DEBUG:boto:auth_path=/ffzdqu-46/obj3 2023-09-22T21:20:38.834 DEBUG:boto:Method: PUT 2023-09-22T21:20:38.834 DEBUG:boto:Path: /ffzdqu-46/obj3 2023-09-22T21:20:38.834 DEBUG:boto:Data: 2023-09-22T21:20:38.834 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:20:38.834 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:20:38.834 DEBUG:boto:Port: 8000 2023-09-22T21:20:38.834 DEBUG:boto:Params: {} 2023-09-22T21:20:38.834 DEBUG:boto:Token: None 2023-09-22T21:20:38.835 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:20:38 GMT /ffzdqu-46/obj3 2023-09-22T21:20:38.835 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wQcFHrmF3ML26VYAgDggyJvTTKM= 2023-09-22T21:20:38.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:20:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wQcFHrmF3ML26VYAgDggyJvTTKM='} 2023-09-22T21:20:38.849 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da6288c88c7d93f3-00650e0526-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:20:38.849 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:20:38.850 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:20:38.850 DEBUG:boto:Method: HEAD 2023-09-22T21:20:38.850 DEBUG:boto:Path: /ffzdqu-46/ 2023-09-22T21:20:38.850 DEBUG:boto:Data: 2023-09-22T21:20:38.850 DEBUG:boto:Headers: {} 2023-09-22T21:20:38.850 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:20:38.850 DEBUG:boto:Port: 8000 2023-09-22T21:20:38.850 DEBUG:boto:Params: {} 2023-09-22T21:20:38.851 DEBUG:boto:Token: None 2023-09-22T21:20:38.851 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:20:38 GMT /ffzdqu-46/ 2023-09-22T21:20:38.851 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9ahSRPMVS4vKbRaN7Iw/jQ60gEg= 2023-09-22T21:20:38.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:20:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9ahSRPMVS4vKbRaN7Iw/jQ60gEg=', 'Content-Length': '0'} 2023-09-22T21:20:38.853 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dcbd48d72b00c7b4-00650e0526-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:20:38.854 DEBUG:boto:path=/ffzdqu-46/obj4 2023-09-22T21:20:38.854 DEBUG:boto:auth_path=/ffzdqu-46/obj4 2023-09-22T21:20:38.854 DEBUG:boto:Method: PUT 2023-09-22T21:20:38.854 DEBUG:boto:Path: /ffzdqu-46/obj4 2023-09-22T21:20:38.854 DEBUG:boto:Data: 2023-09-22T21:20:38.854 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:20:38.854 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:20:38.855 DEBUG:boto:Port: 8000 2023-09-22T21:20:38.855 DEBUG:boto:Params: {} 2023-09-22T21:20:38.855 DEBUG:boto:Token: None 2023-09-22T21:20:38.855 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:20:38 GMT /ffzdqu-46/obj4 2023-09-22T21:20:38.855 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gJPqWzzMrGs8ljQT4d9FGWRCgWA= 2023-09-22T21:20:38.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:20:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gJPqWzzMrGs8ljQT4d9FGWRCgWA='} 2023-09-22T21:20:38.898 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4c8852b1e1eea2d-00650e0526-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:20:38.899 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:20:38.899 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:20:38.899 DEBUG:boto:Method: HEAD 2023-09-22T21:20:38.899 DEBUG:boto:Path: /ffzdqu-47/ 2023-09-22T21:20:38.899 DEBUG:boto:Data: 2023-09-22T21:20:38.899 DEBUG:boto:Headers: {} 2023-09-22T21:20:38.900 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:20:38.900 DEBUG:boto:Port: 8001 2023-09-22T21:20:38.900 DEBUG:boto:Params: {} 2023-09-22T21:20:38.900 DEBUG:boto:Token: None 2023-09-22T21:20:38.900 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:20:38 GMT /ffzdqu-47/ 2023-09-22T21:20:38.900 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bAb1+bbIUJcPaW4dz11HEI4Ofto= 2023-09-22T21:20:38.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:20:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bAb1+bbIUJcPaW4dz11HEI4Ofto=', 'Content-Length': '0'} 2023-09-22T21:20:38.903 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f08699dee7a96050-00650e0526-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:20:38.903 DEBUG:boto:path=/ffzdqu-47/obj1 2023-09-22T21:20:38.903 DEBUG:boto:auth_path=/ffzdqu-47/obj1 2023-09-22T21:20:38.903 DEBUG:boto:Method: PUT 2023-09-22T21:20:38.903 DEBUG:boto:Path: /ffzdqu-47/obj1 2023-09-22T21:20:38.904 DEBUG:boto:Data: 2023-09-22T21:20:38.904 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:20:38.904 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:20:38.904 DEBUG:boto:Port: 8001 2023-09-22T21:20:38.904 DEBUG:boto:Params: {} 2023-09-22T21:20:38.904 DEBUG:boto:Token: None 2023-09-22T21:20:38.904 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:20:38 GMT /ffzdqu-47/obj1 2023-09-22T21:20:38.904 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y/zY/USvleZCLNOpsdJT5GoiJWk= 2023-09-22T21:20:38.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:20:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y/zY/USvleZCLNOpsdJT5GoiJWk='} 2023-09-22T21:20:38.912 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c6861c41833d0b8e-00650e0526-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:20:38.912 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:20:38.912 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:20:38.912 DEBUG:boto:Method: HEAD 2023-09-22T21:20:38.912 DEBUG:boto:Path: /ffzdqu-47/ 2023-09-22T21:20:38.913 DEBUG:boto:Data: 2023-09-22T21:20:38.913 DEBUG:boto:Headers: {} 2023-09-22T21:20:38.913 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:20:38.913 DEBUG:boto:Port: 8001 2023-09-22T21:20:38.913 DEBUG:boto:Params: {} 2023-09-22T21:20:38.913 DEBUG:boto:Token: None 2023-09-22T21:20:38.913 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:20:38 GMT /ffzdqu-47/ 2023-09-22T21:20:38.913 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bAb1+bbIUJcPaW4dz11HEI4Ofto= 2023-09-22T21:20:38.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:20:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bAb1+bbIUJcPaW4dz11HEI4Ofto=', 'Content-Length': '0'} 2023-09-22T21:20:38.916 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006aa57b88020e02f2-00650e0526-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:20:38.916 DEBUG:boto:path=/ffzdqu-47/obj2 2023-09-22T21:20:38.916 DEBUG:boto:auth_path=/ffzdqu-47/obj2 2023-09-22T21:20:38.916 DEBUG:boto:Method: PUT 2023-09-22T21:20:38.916 DEBUG:boto:Path: /ffzdqu-47/obj2 2023-09-22T21:20:38.916 DEBUG:boto:Data: 2023-09-22T21:20:38.917 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:20:38.917 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:20:38.917 DEBUG:boto:Port: 8001 2023-09-22T21:20:38.917 DEBUG:boto:Params: {} 2023-09-22T21:20:38.917 DEBUG:boto:Token: None 2023-09-22T21:20:38.917 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:20:38 GMT /ffzdqu-47/obj2 2023-09-22T21:20:38.917 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:W54tGSWeNewhHzQV6g9oIROlrYk= 2023-09-22T21:20:38.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:20:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:W54tGSWeNewhHzQV6g9oIROlrYk='} 2023-09-22T21:20:38.966 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e385610b053a69b1-00650e0526-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:20:38.967 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:20:38.967 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:20:38.967 DEBUG:boto:Method: HEAD 2023-09-22T21:20:38.967 DEBUG:boto:Path: /ffzdqu-47/ 2023-09-22T21:20:38.967 DEBUG:boto:Data: 2023-09-22T21:20:38.967 DEBUG:boto:Headers: {} 2023-09-22T21:20:38.968 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:20:38.968 DEBUG:boto:Port: 8001 2023-09-22T21:20:38.968 DEBUG:boto:Params: {} 2023-09-22T21:20:38.968 DEBUG:boto:Token: None 2023-09-22T21:20:38.968 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:20:38 GMT /ffzdqu-47/ 2023-09-22T21:20:38.968 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bAb1+bbIUJcPaW4dz11HEI4Ofto= 2023-09-22T21:20:38.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:20:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bAb1+bbIUJcPaW4dz11HEI4Ofto=', 'Content-Length': '0'} 2023-09-22T21:20:38.971 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000baa1023c56b2679c-00650e0526-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:20:38.972 DEBUG:boto:path=/ffzdqu-47/obj3 2023-09-22T21:20:38.972 DEBUG:boto:auth_path=/ffzdqu-47/obj3 2023-09-22T21:20:38.972 DEBUG:boto:Method: PUT 2023-09-22T21:20:38.972 DEBUG:boto:Path: /ffzdqu-47/obj3 2023-09-22T21:20:38.972 DEBUG:boto:Data: 2023-09-22T21:20:38.972 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:20:38.972 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:20:38.973 DEBUG:boto:Port: 8001 2023-09-22T21:20:38.973 DEBUG:boto:Params: {} 2023-09-22T21:20:38.973 DEBUG:boto:Token: None 2023-09-22T21:20:38.973 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:20:38 GMT /ffzdqu-47/obj3 2023-09-22T21:20:38.973 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cabpiZXFzRT3FM1oWGvN6wbTBwE= 2023-09-22T21:20:38.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:20:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cabpiZXFzRT3FM1oWGvN6wbTBwE='} 2023-09-22T21:20:38.985 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000310836e16075f746-00650e0526-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:20:38.985 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:20:38.985 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:20:38.985 DEBUG:boto:Method: HEAD 2023-09-22T21:20:38.985 DEBUG:boto:Path: /ffzdqu-47/ 2023-09-22T21:20:38.985 DEBUG:boto:Data: 2023-09-22T21:20:38.986 DEBUG:boto:Headers: {} 2023-09-22T21:20:38.986 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:20:38.986 DEBUG:boto:Port: 8001 2023-09-22T21:20:38.986 DEBUG:boto:Params: {} 2023-09-22T21:20:38.986 DEBUG:boto:Token: None 2023-09-22T21:20:38.986 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:20:38 GMT /ffzdqu-47/ 2023-09-22T21:20:38.986 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bAb1+bbIUJcPaW4dz11HEI4Ofto= 2023-09-22T21:20:38.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:20:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bAb1+bbIUJcPaW4dz11HEI4Ofto=', 'Content-Length': '0'} 2023-09-22T21:20:38.990 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cda565d614ef5549-00650e0526-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:20:38.990 DEBUG:boto:path=/ffzdqu-47/obj4 2023-09-22T21:20:38.991 DEBUG:boto:auth_path=/ffzdqu-47/obj4 2023-09-22T21:20:38.991 DEBUG:boto:Method: PUT 2023-09-22T21:20:38.991 DEBUG:boto:Path: /ffzdqu-47/obj4 2023-09-22T21:20:38.991 DEBUG:boto:Data: 2023-09-22T21:20:38.991 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:20:38.991 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:20:38.991 DEBUG:boto:Port: 8001 2023-09-22T21:20:38.992 DEBUG:boto:Params: {} 2023-09-22T21:20:38.992 DEBUG:boto:Token: None 2023-09-22T21:20:38.992 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:20:38 GMT /ffzdqu-47/obj4 2023-09-22T21:20:38.992 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E2ywPFLe2g2fJt61lc7ZkcXL8D0= 2023-09-22T21:20:38.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:20:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E2ywPFLe2g2fJt61lc7ZkcXL8D0='} 2023-09-22T21:20:39.038 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b136abdc33c2048e-00650e0526-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:20:39.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:39.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:39.039 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:20:39.081 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:20:39.081 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:20:39.119 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:20:39.137 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695417638.756529_348.1","last_update":"2023-09-22T21:20:38.756529Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:20:39.137 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417638.756529_348.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:20:39.137 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:20:39.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:39.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:20:39.138 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:20:39.217 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:20:39.217 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:20:39.254 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:20:39.257+0000 7f38fa9fba40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:20:39.255 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:20:39.257+0000 7f38fa9fba40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:20:39.273 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417638.756529_348.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:20:38.756529Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:20:39Z"} 2023-09-22T21:20:39.274 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417638.756529_348.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:20:38.756529Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:20:39Z"} 2023-09-22T21:20:39.274 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417638.756529_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:20:38.756529Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:20:39.274 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417638.756529_348.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:20:39.274 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417638.756529_348.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:20:39.274 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:20:39.274 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-46 2023-09-22T21:20:39.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:20:39.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:20:39.275 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-46 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:20:39.363 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:20:39.363 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:20:39.405 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:20:39.409+0000 7fc1fdcd9a40 1 waiting to reach incremental sync.. 2023-09-22T21:20:44.406 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:20:44.408+0000 7fc1fdcd9a40 1 waiting to reach incremental sync.. 2023-09-22T21:20:49.407 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:20:49.408+0000 7fc1fdcd9a40 1 waiting to reach incremental sync.. 2023-09-22T21:20:54.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:20:54.408+0000 7fc1fdcd9a40 1 waiting to reach incremental sync.. 2023-09-22T21:20:56.582 INFO:tasks.rgw.c1.client.0.smithi160.stdout:2023-09-22T21:20:56.584+0000 7fa9c1675640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:20:59.411 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:20:59.412+0000 7fc1fdcd9a40 1 bucket sync caught up with source: 2023-09-22T21:20:59.411 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.24.6, , 00000000001.26.6, , 00000000001.22.6, , , 00000000001.22.6, , , ] 2023-09-22T21:20:59.412 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.24.6, , 00000000001.26.6, , 00000000001.22.6, , , 00000000001.22.6, , , ] 2023-09-22T21:20:59.412 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:20:59.412+0000 7fc1fdcd9a40 0 bucket checkpoint complete 2023-09-22T21:20:59.443 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-46 2023-09-22T21:20:59.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:20:59.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:20:59.444 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-46 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:20:59.493 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:20:59.493 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:20:59.534 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:20:59.536+0000 7f0fa6375a40 1 waiting to reach incremental sync.. 2023-09-22T21:21:04.537 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:21:04.536+0000 7f0fa6375a40 1 waiting to reach incremental sync.. 2023-09-22T21:21:09.537 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:21:09.540+0000 7f0fa6375a40 1 waiting to reach incremental sync.. 2023-09-22T21:21:14.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:21:14.540+0000 7f0fa6375a40 1 waiting to reach incremental sync.. 2023-09-22T21:21:19.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:21:19.544+0000 7f0fa6375a40 1 bucket sync caught up with source: 2023-09-22T21:21:19.541 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.129.6, , 00000000001.213.6, , 00000000001.157.6, , , 00000000001.191.6, , , ] 2023-09-22T21:21:19.541 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.129.6, , 00000000001.213.6, , 00000000001.157.6, , , 00000000001.191.6, , , ] 2023-09-22T21:21:19.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:21:19.544+0000 7f0fa6375a40 0 bucket checkpoint complete 2023-09-22T21:21:19.576 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-46 zones={a2, a1} 2023-09-22T21:21:19.577 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:19.577 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:19.577 DEBUG:boto:Method: HEAD 2023-09-22T21:21:19.577 DEBUG:boto:Path: /ffzdqu-46/ 2023-09-22T21:21:19.577 DEBUG:boto:Data: 2023-09-22T21:21:19.577 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.577 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:19.577 DEBUG:boto:Port: 8001 2023-09-22T21:21:19.578 DEBUG:boto:Params: {} 2023-09-22T21:21:19.578 DEBUG:boto:Token: None 2023-09-22T21:21:19.578 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/ 2023-09-22T21:21:19.578 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9NA/nrz180l81MMIPMkEwo4usXM= 2023-09-22T21:21:19.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9NA/nrz180l81MMIPMkEwo4usXM=', 'Content-Length': '0'} 2023-09-22T21:21:19.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee2feab95efa8c93-00650e054f-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.582 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:19.582 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:19.583 DEBUG:boto:Method: HEAD 2023-09-22T21:21:19.583 DEBUG:boto:Path: /ffzdqu-46/ 2023-09-22T21:21:19.583 DEBUG:boto:Data: 2023-09-22T21:21:19.583 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.583 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:19.583 DEBUG:boto:Port: 8000 2023-09-22T21:21:19.583 DEBUG:boto:Params: {} 2023-09-22T21:21:19.583 DEBUG:boto:Token: None 2023-09-22T21:21:19.583 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/ 2023-09-22T21:21:19.584 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9NA/nrz180l81MMIPMkEwo4usXM= 2023-09-22T21:21:19.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9NA/nrz180l81MMIPMkEwo4usXM=', 'Content-Length': '0'} 2023-09-22T21:21:19.587 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000052237657e3d3ac1-00650e054f-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.587 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:21:19.588 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:19.588 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:19.588 DEBUG:boto:path=/ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:19.588 DEBUG:boto:auth_path=/ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:19.588 DEBUG:boto:Method: GET 2023-09-22T21:21:19.588 DEBUG:boto:Path: /ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:19.588 DEBUG:boto:Data: 2023-09-22T21:21:19.588 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.588 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:19.588 DEBUG:boto:Port: 8001 2023-09-22T21:21:19.589 DEBUG:boto:Params: {} 2023-09-22T21:21:19.589 DEBUG:boto:Token: None 2023-09-22T21:21:19.589 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/?versions 2023-09-22T21:21:19.589 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9kl20HsidouaL8iweSNsW7gWSbk= 2023-09-22T21:21:19.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9kl20HsidouaL8iweSNsW7gWSbk=', 'Content-Length': '0'} 2023-09-22T21:21:19.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7cc79bf236bae1c-00650e054f-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.594 DEBUG:boto:b'ffzdqu-46999falseobj1nulltrue2023-09-22T21:20:38.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:20:38.787Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:20:38.842Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:20:38.862Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:19.594 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-22T21:21:19.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-22T21:21:19.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-22T21:21:19.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-22T21:21:19.595 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:21:19.595 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:19.595 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:19.595 DEBUG:boto:path=/ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:19.595 DEBUG:boto:auth_path=/ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:19.596 DEBUG:boto:Method: GET 2023-09-22T21:21:19.596 DEBUG:boto:Path: /ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:19.596 DEBUG:boto:Data: 2023-09-22T21:21:19.596 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.596 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:19.596 DEBUG:boto:Port: 8000 2023-09-22T21:21:19.596 DEBUG:boto:Params: {} 2023-09-22T21:21:19.596 DEBUG:boto:Token: None 2023-09-22T21:21:19.596 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/?versions 2023-09-22T21:21:19.597 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9kl20HsidouaL8iweSNsW7gWSbk= 2023-09-22T21:21:19.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9kl20HsidouaL8iweSNsW7gWSbk=', 'Content-Length': '0'} 2023-09-22T21:21:19.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030308f6ec76ff7d0-00650e054f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.601 DEBUG:boto:b'ffzdqu-46999falseobj1nulltrue2023-09-22T21:20:38.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:20:38.787Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:20:38.842Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:20:38.862Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:19.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-22T21:21:19.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-22T21:21:19.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-22T21:21:19.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-22T21:21:19.602 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:19.602 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:19.602 DEBUG:boto:path=/ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:19.602 DEBUG:boto:auth_path=/ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:19.602 DEBUG:boto:Method: GET 2023-09-22T21:21:19.602 DEBUG:boto:Path: /ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:19.602 DEBUG:boto:Data: 2023-09-22T21:21:19.602 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.602 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:19.603 DEBUG:boto:Port: 8001 2023-09-22T21:21:19.603 DEBUG:boto:Params: {} 2023-09-22T21:21:19.603 DEBUG:boto:Token: None 2023-09-22T21:21:19.603 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/?versions 2023-09-22T21:21:19.603 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9kl20HsidouaL8iweSNsW7gWSbk= 2023-09-22T21:21:19.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9kl20HsidouaL8iweSNsW7gWSbk=', 'Content-Length': '0'} 2023-09-22T21:21:19.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017006ab0dcb7c368-00650e054f-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.607 DEBUG:boto:b'ffzdqu-46999falseobj1nulltrue2023-09-22T21:20:38.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:20:38.787Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:20:38.842Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:20:38.862Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:19.608 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:19.608 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:19.608 DEBUG:boto:path=/ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:19.608 DEBUG:boto:auth_path=/ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:19.608 DEBUG:boto:Method: GET 2023-09-22T21:21:19.608 DEBUG:boto:Path: /ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:19.608 DEBUG:boto:Data: 2023-09-22T21:21:19.608 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.609 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:19.609 DEBUG:boto:Port: 8000 2023-09-22T21:21:19.609 DEBUG:boto:Params: {} 2023-09-22T21:21:19.609 DEBUG:boto:Token: None 2023-09-22T21:21:19.609 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/?versions 2023-09-22T21:21:19.609 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9kl20HsidouaL8iweSNsW7gWSbk= 2023-09-22T21:21:19.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9kl20HsidouaL8iweSNsW7gWSbk=', 'Content-Length': '0'} 2023-09-22T21:21:19.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d50ca3f10d9b70c-00650e054f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.614 DEBUG:boto:b'ffzdqu-46999falseobj1nulltrue2023-09-22T21:20:38.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:20:38.787Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:20:38.842Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:20:38.862Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:19.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:21:19.614 DEBUG:boto:path=/ffzdqu-46/obj1 2023-09-22T21:21:19.614 DEBUG:boto:auth_path=/ffzdqu-46/obj1 2023-09-22T21:21:19.614 DEBUG:boto:path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.615 DEBUG:boto:auth_path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.615 DEBUG:boto:Method: GET 2023-09-22T21:21:19.615 DEBUG:boto:Path: /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.615 DEBUG:boto:Data: 2023-09-22T21:21:19.615 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.615 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:19.615 DEBUG:boto:Port: 8001 2023-09-22T21:21:19.615 DEBUG:boto:Params: {} 2023-09-22T21:21:19.615 DEBUG:boto:Token: None 2023-09-22T21:21:19.616 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.616 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YwQRNPkNFxfFqIfarfMYYu5ZgEs= 2023-09-22T21:21:19.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YwQRNPkNFxfFqIfarfMYYu5ZgEs=', 'Content-Length': '0'} 2023-09-22T21:21:19.618 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a97991d732c7e82c-00650e054f-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.619 DEBUG:boto:path=/ffzdqu-46/obj1 2023-09-22T21:21:19.619 DEBUG:boto:auth_path=/ffzdqu-46/obj1 2023-09-22T21:21:19.619 DEBUG:boto:path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.619 DEBUG:boto:auth_path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.619 DEBUG:boto:Method: GET 2023-09-22T21:21:19.619 DEBUG:boto:Path: /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.619 DEBUG:boto:Data: 2023-09-22T21:21:19.619 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.619 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:19.619 DEBUG:boto:Port: 8000 2023-09-22T21:21:19.620 DEBUG:boto:Params: {} 2023-09-22T21:21:19.620 DEBUG:boto:Token: None 2023-09-22T21:21:19.620 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.620 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YwQRNPkNFxfFqIfarfMYYu5ZgEs= 2023-09-22T21:21:19.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YwQRNPkNFxfFqIfarfMYYu5ZgEs=', 'Content-Length': '0'} 2023-09-22T21:21:19.622 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000513fd0c2a44739a8-00650e054f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.666 DEBUG:boto:path=/ffzdqu-46/obj1 2023-09-22T21:21:19.667 DEBUG:boto:auth_path=/ffzdqu-46/obj1 2023-09-22T21:21:19.667 DEBUG:boto:path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.667 DEBUG:boto:auth_path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.667 DEBUG:boto:Method: HEAD 2023-09-22T21:21:19.667 DEBUG:boto:Path: /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.667 DEBUG:boto:Data: 2023-09-22T21:21:19.667 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.667 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:19.667 DEBUG:boto:Port: 8001 2023-09-22T21:21:19.667 DEBUG:boto:Params: {} 2023-09-22T21:21:19.668 DEBUG:boto:Token: None 2023-09-22T21:21:19.668 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.668 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BntHT6/xHC9kFpi+X5AmHOkAeMM= 2023-09-22T21:21:19.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BntHT6/xHC9kFpi+X5AmHOkAeMM=', 'Content-Length': '0'} 2023-09-22T21:21:19.671 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070b45aec5eab09b6-00650e054f-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.671 DEBUG:boto:path=/ffzdqu-46/obj1 2023-09-22T21:21:19.671 DEBUG:boto:auth_path=/ffzdqu-46/obj1 2023-09-22T21:21:19.671 DEBUG:boto:path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.671 DEBUG:boto:auth_path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.672 DEBUG:boto:Method: HEAD 2023-09-22T21:21:19.672 DEBUG:boto:Path: /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.672 DEBUG:boto:Data: 2023-09-22T21:21:19.672 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.672 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:19.672 DEBUG:boto:Port: 8000 2023-09-22T21:21:19.672 DEBUG:boto:Params: {} 2023-09-22T21:21:19.672 DEBUG:boto:Token: None 2023-09-22T21:21:19.672 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.673 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BntHT6/xHC9kFpi+X5AmHOkAeMM= 2023-09-22T21:21:19.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BntHT6/xHC9kFpi+X5AmHOkAeMM=', 'Content-Length': '0'} 2023-09-22T21:21:19.675 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021fe953b99f53bdd-00650e054f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:21:19.676 DEBUG:boto:path=/ffzdqu-46/obj1 2023-09-22T21:21:19.676 DEBUG:boto:auth_path=/ffzdqu-46/obj1 2023-09-22T21:21:19.676 DEBUG:boto:path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.676 DEBUG:boto:auth_path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.676 DEBUG:boto:Method: GET 2023-09-22T21:21:19.676 DEBUG:boto:Path: /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.676 DEBUG:boto:Data: 2023-09-22T21:21:19.676 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.677 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:19.677 DEBUG:boto:Port: 8001 2023-09-22T21:21:19.677 DEBUG:boto:Params: {} 2023-09-22T21:21:19.677 DEBUG:boto:Token: None 2023-09-22T21:21:19.677 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.677 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YwQRNPkNFxfFqIfarfMYYu5ZgEs= 2023-09-22T21:21:19.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YwQRNPkNFxfFqIfarfMYYu5ZgEs=', 'Content-Length': '0'} 2023-09-22T21:21:19.680 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e198169409ad2b5-00650e054f-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.726 DEBUG:boto:path=/ffzdqu-46/obj1 2023-09-22T21:21:19.726 DEBUG:boto:auth_path=/ffzdqu-46/obj1 2023-09-22T21:21:19.727 DEBUG:boto:path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.727 DEBUG:boto:auth_path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.727 DEBUG:boto:Method: GET 2023-09-22T21:21:19.727 DEBUG:boto:Path: /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.727 DEBUG:boto:Data: 2023-09-22T21:21:19.727 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.727 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:19.727 DEBUG:boto:Port: 8000 2023-09-22T21:21:19.727 DEBUG:boto:Params: {} 2023-09-22T21:21:19.728 DEBUG:boto:Token: None 2023-09-22T21:21:19.728 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:19.728 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YwQRNPkNFxfFqIfarfMYYu5ZgEs= 2023-09-22T21:21:19.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YwQRNPkNFxfFqIfarfMYYu5ZgEs=', 'Content-Length': '0'} 2023-09-22T21:21:19.731 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b4bb1a2dfc8823c-00650e054f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.731 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:19.731 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:19.731 DEBUG:boto:path=/ffzdqu-46/?prefix=obj1 2023-09-22T21:21:19.731 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj1 2023-09-22T21:21:19.731 DEBUG:boto:Method: GET 2023-09-22T21:21:19.731 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj1 2023-09-22T21:21:19.732 DEBUG:boto:Data: 2023-09-22T21:21:19.732 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.732 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:19.732 DEBUG:boto:Port: 8001 2023-09-22T21:21:19.732 DEBUG:boto:Params: {} 2023-09-22T21:21:19.732 DEBUG:boto:Token: None 2023-09-22T21:21:19.732 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/ 2023-09-22T21:21:19.732 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2yO6ObytvAIm25ruePYccj4o9Nc= 2023-09-22T21:21:19.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2yO6ObytvAIm25ruePYccj4o9Nc=', 'Content-Length': '0'} 2023-09-22T21:21:19.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f662e6d7d4d2ea9-00650e054f-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.737 DEBUG:boto:b'ffzdqu-46obj11000falseobj12023-09-22T21:20:38.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:19.737 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:19.738 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:19.738 DEBUG:boto:path=/ffzdqu-46/?prefix=obj1 2023-09-22T21:21:19.738 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj1 2023-09-22T21:21:19.738 DEBUG:boto:Method: GET 2023-09-22T21:21:19.738 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj1 2023-09-22T21:21:19.738 DEBUG:boto:Data: 2023-09-22T21:21:19.738 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.738 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:19.738 DEBUG:boto:Port: 8000 2023-09-22T21:21:19.739 DEBUG:boto:Params: {} 2023-09-22T21:21:19.739 DEBUG:boto:Token: None 2023-09-22T21:21:19.739 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/ 2023-09-22T21:21:19.739 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2yO6ObytvAIm25ruePYccj4o9Nc= 2023-09-22T21:21:19.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2yO6ObytvAIm25ruePYccj4o9Nc=', 'Content-Length': '0'} 2023-09-22T21:21:19.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bcf5d3d06a77865-00650e054f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.744 DEBUG:boto:b'ffzdqu-46obj11000falseobj12023-09-22T21:20:38.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:19.744 DEBUG:boto:path=/ffzdqu-46/obj1 2023-09-22T21:21:19.744 DEBUG:boto:auth_path=/ffzdqu-46/obj1 2023-09-22T21:21:19.744 DEBUG:boto:Method: HEAD 2023-09-22T21:21:19.744 DEBUG:boto:Path: /ffzdqu-46/obj1 2023-09-22T21:21:19.744 DEBUG:boto:Data: 2023-09-22T21:21:19.745 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.745 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:19.745 DEBUG:boto:Port: 8001 2023-09-22T21:21:19.745 DEBUG:boto:Params: {} 2023-09-22T21:21:19.745 DEBUG:boto:Token: None 2023-09-22T21:21:19.745 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/obj1 2023-09-22T21:21:19.745 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5Je1Az/1wGAqAgAUokbhXGWR/ds= 2023-09-22T21:21:19.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5Je1Az/1wGAqAgAUokbhXGWR/ds=', 'Content-Length': '0'} 2023-09-22T21:21:19.748 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000377052b880cd0efe-00650e054f-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.748 DEBUG:boto:path=/ffzdqu-46/obj1 2023-09-22T21:21:19.748 DEBUG:boto:auth_path=/ffzdqu-46/obj1 2023-09-22T21:21:19.749 DEBUG:boto:Method: HEAD 2023-09-22T21:21:19.749 DEBUG:boto:Path: /ffzdqu-46/obj1 2023-09-22T21:21:19.749 DEBUG:boto:Data: 2023-09-22T21:21:19.749 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.749 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:19.749 DEBUG:boto:Port: 8000 2023-09-22T21:21:19.749 DEBUG:boto:Params: {} 2023-09-22T21:21:19.749 DEBUG:boto:Token: None 2023-09-22T21:21:19.749 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/obj1 2023-09-22T21:21:19.750 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5Je1Az/1wGAqAgAUokbhXGWR/ds= 2023-09-22T21:21:19.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5Je1Az/1wGAqAgAUokbhXGWR/ds=', 'Content-Length': '0'} 2023-09-22T21:21:19.752 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db49adff9759bd6c-00650e054f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:21:19.753 DEBUG:boto:path=/ffzdqu-46/obj1 2023-09-22T21:21:19.753 DEBUG:boto:auth_path=/ffzdqu-46/obj1 2023-09-22T21:21:19.753 DEBUG:boto:Method: GET 2023-09-22T21:21:19.753 DEBUG:boto:Path: /ffzdqu-46/obj1 2023-09-22T21:21:19.753 DEBUG:boto:Data: 2023-09-22T21:21:19.753 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.753 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:19.753 DEBUG:boto:Port: 8001 2023-09-22T21:21:19.753 DEBUG:boto:Params: {} 2023-09-22T21:21:19.753 DEBUG:boto:Token: None 2023-09-22T21:21:19.754 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/obj1 2023-09-22T21:21:19.754 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kmLxTHzMhN1LUxLSK1oeeVGV3kE= 2023-09-22T21:21:19.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kmLxTHzMhN1LUxLSK1oeeVGV3kE=', 'Content-Length': '0'} 2023-09-22T21:21:19.757 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016cc735db622cc35-00650e054f-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.802 DEBUG:boto:path=/ffzdqu-46/obj1 2023-09-22T21:21:19.803 DEBUG:boto:auth_path=/ffzdqu-46/obj1 2023-09-22T21:21:19.803 DEBUG:boto:Method: GET 2023-09-22T21:21:19.803 DEBUG:boto:Path: /ffzdqu-46/obj1 2023-09-22T21:21:19.803 DEBUG:boto:Data: 2023-09-22T21:21:19.803 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.803 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:19.803 DEBUG:boto:Port: 8000 2023-09-22T21:21:19.803 DEBUG:boto:Params: {} 2023-09-22T21:21:19.804 DEBUG:boto:Token: None 2023-09-22T21:21:19.804 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/obj1 2023-09-22T21:21:19.804 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kmLxTHzMhN1LUxLSK1oeeVGV3kE= 2023-09-22T21:21:19.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kmLxTHzMhN1LUxLSK1oeeVGV3kE=', 'Content-Length': '0'} 2023-09-22T21:21:19.807 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000788876082bda3bd6-00650e054f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.810 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:19.810 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:19.810 DEBUG:boto:path=/ffzdqu-46/?prefix=obj1 2023-09-22T21:21:19.811 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj1 2023-09-22T21:21:19.811 DEBUG:boto:Method: GET 2023-09-22T21:21:19.811 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj1 2023-09-22T21:21:19.811 DEBUG:boto:Data: 2023-09-22T21:21:19.811 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.811 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:19.812 DEBUG:boto:Port: 8001 2023-09-22T21:21:19.812 DEBUG:boto:Params: {} 2023-09-22T21:21:19.812 DEBUG:boto:Token: None 2023-09-22T21:21:19.812 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/ 2023-09-22T21:21:19.813 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2yO6ObytvAIm25ruePYccj4o9Nc= 2023-09-22T21:21:19.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2yO6ObytvAIm25ruePYccj4o9Nc=', 'Content-Length': '0'} 2023-09-22T21:21:19.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe5771ec4496cae3-00650e054f-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.817 DEBUG:boto:b'ffzdqu-46obj11000falseobj12023-09-22T21:20:38.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:19.818 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:19.818 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:19.818 DEBUG:boto:path=/ffzdqu-46/?prefix=obj1 2023-09-22T21:21:19.818 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj1 2023-09-22T21:21:19.818 DEBUG:boto:Method: GET 2023-09-22T21:21:19.818 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj1 2023-09-22T21:21:19.818 DEBUG:boto:Data: 2023-09-22T21:21:19.818 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.818 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:19.819 DEBUG:boto:Port: 8000 2023-09-22T21:21:19.819 DEBUG:boto:Params: {} 2023-09-22T21:21:19.819 DEBUG:boto:Token: None 2023-09-22T21:21:19.819 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/ 2023-09-22T21:21:19.819 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2yO6ObytvAIm25ruePYccj4o9Nc= 2023-09-22T21:21:19.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2yO6ObytvAIm25ruePYccj4o9Nc=', 'Content-Length': '0'} 2023-09-22T21:21:19.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000705b4169acdba07c-00650e054f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.824 DEBUG:boto:b'ffzdqu-46obj11000falseobj12023-09-22T21:20:38.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:19.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:21:19.825 DEBUG:boto:path=/ffzdqu-46/obj2 2023-09-22T21:21:19.825 DEBUG:boto:auth_path=/ffzdqu-46/obj2 2023-09-22T21:21:19.825 DEBUG:boto:path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.825 DEBUG:boto:auth_path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.825 DEBUG:boto:Method: GET 2023-09-22T21:21:19.826 DEBUG:boto:Path: /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.826 DEBUG:boto:Data: 2023-09-22T21:21:19.826 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.827 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:19.827 DEBUG:boto:Port: 8001 2023-09-22T21:21:19.827 DEBUG:boto:Params: {} 2023-09-22T21:21:19.827 DEBUG:boto:Token: None 2023-09-22T21:21:19.827 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.828 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:staPYFr8vfjA8BT4gcLqvi8XW0I= 2023-09-22T21:21:19.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:staPYFr8vfjA8BT4gcLqvi8XW0I=', 'Content-Length': '0'} 2023-09-22T21:21:19.831 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d51b48fd103ffb36-00650e054f-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.831 DEBUG:boto:path=/ffzdqu-46/obj2 2023-09-22T21:21:19.831 DEBUG:boto:auth_path=/ffzdqu-46/obj2 2023-09-22T21:21:19.831 DEBUG:boto:path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.832 DEBUG:boto:auth_path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.832 DEBUG:boto:Method: GET 2023-09-22T21:21:19.832 DEBUG:boto:Path: /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.832 DEBUG:boto:Data: 2023-09-22T21:21:19.832 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.833 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:19.833 DEBUG:boto:Port: 8000 2023-09-22T21:21:19.833 DEBUG:boto:Params: {} 2023-09-22T21:21:19.833 DEBUG:boto:Token: None 2023-09-22T21:21:19.833 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.834 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:staPYFr8vfjA8BT4gcLqvi8XW0I= 2023-09-22T21:21:19.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:staPYFr8vfjA8BT4gcLqvi8XW0I=', 'Content-Length': '0'} 2023-09-22T21:21:19.837 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b2e93e6d12663b7-00650e054f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.837 DEBUG:boto:path=/ffzdqu-46/obj2 2023-09-22T21:21:19.838 DEBUG:boto:auth_path=/ffzdqu-46/obj2 2023-09-22T21:21:19.838 DEBUG:boto:path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.838 DEBUG:boto:auth_path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.838 DEBUG:boto:Method: HEAD 2023-09-22T21:21:19.839 DEBUG:boto:Path: /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.839 DEBUG:boto:Data: 2023-09-22T21:21:19.839 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.839 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:19.840 DEBUG:boto:Port: 8001 2023-09-22T21:21:19.840 DEBUG:boto:Params: {} 2023-09-22T21:21:19.840 DEBUG:boto:Token: None 2023-09-22T21:21:19.841 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.841 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9KPc6Rr73/dBIIQqym42BpAUXuE= 2023-09-22T21:21:19.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9KPc6Rr73/dBIIQqym42BpAUXuE=', 'Content-Length': '0'} 2023-09-22T21:21:19.844 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c42e51a669327cc-00650e054f-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.845 DEBUG:boto:path=/ffzdqu-46/obj2 2023-09-22T21:21:19.845 DEBUG:boto:auth_path=/ffzdqu-46/obj2 2023-09-22T21:21:19.846 DEBUG:boto:path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.846 DEBUG:boto:auth_path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.846 DEBUG:boto:Method: HEAD 2023-09-22T21:21:19.847 DEBUG:boto:Path: /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.847 DEBUG:boto:Data: 2023-09-22T21:21:19.847 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.847 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:19.848 DEBUG:boto:Port: 8000 2023-09-22T21:21:19.848 DEBUG:boto:Params: {} 2023-09-22T21:21:19.848 DEBUG:boto:Token: None 2023-09-22T21:21:19.848 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.849 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9KPc6Rr73/dBIIQqym42BpAUXuE= 2023-09-22T21:21:19.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9KPc6Rr73/dBIIQqym42BpAUXuE=', 'Content-Length': '0'} 2023-09-22T21:21:19.851 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005cf85242f1e17653-00650e054f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:21:19.852 DEBUG:boto:path=/ffzdqu-46/obj2 2023-09-22T21:21:19.852 DEBUG:boto:auth_path=/ffzdqu-46/obj2 2023-09-22T21:21:19.852 DEBUG:boto:path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.852 DEBUG:boto:auth_path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.853 DEBUG:boto:Method: GET 2023-09-22T21:21:19.853 DEBUG:boto:Path: /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.853 DEBUG:boto:Data: 2023-09-22T21:21:19.853 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.853 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:19.854 DEBUG:boto:Port: 8001 2023-09-22T21:21:19.854 DEBUG:boto:Params: {} 2023-09-22T21:21:19.854 DEBUG:boto:Token: None 2023-09-22T21:21:19.854 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.855 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:staPYFr8vfjA8BT4gcLqvi8XW0I= 2023-09-22T21:21:19.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:staPYFr8vfjA8BT4gcLqvi8XW0I=', 'Content-Length': '0'} 2023-09-22T21:21:19.857 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005832755650a28f91-00650e054f-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.898 DEBUG:boto:path=/ffzdqu-46/obj2 2023-09-22T21:21:19.899 DEBUG:boto:auth_path=/ffzdqu-46/obj2 2023-09-22T21:21:19.899 DEBUG:boto:path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.899 DEBUG:boto:auth_path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.899 DEBUG:boto:Method: GET 2023-09-22T21:21:19.899 DEBUG:boto:Path: /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.899 DEBUG:boto:Data: 2023-09-22T21:21:19.899 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.899 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:19.900 DEBUG:boto:Port: 8000 2023-09-22T21:21:19.900 DEBUG:boto:Params: {} 2023-09-22T21:21:19.900 DEBUG:boto:Token: None 2023-09-22T21:21:19.900 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:19.900 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:staPYFr8vfjA8BT4gcLqvi8XW0I= 2023-09-22T21:21:19.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:staPYFr8vfjA8BT4gcLqvi8XW0I=', 'Content-Length': '0'} 2023-09-22T21:21:19.903 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f279e4f0777358f-00650e054f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.903 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:19.904 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:19.904 DEBUG:boto:path=/ffzdqu-46/?prefix=obj2 2023-09-22T21:21:19.904 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj2 2023-09-22T21:21:19.904 DEBUG:boto:Method: GET 2023-09-22T21:21:19.904 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj2 2023-09-22T21:21:19.904 DEBUG:boto:Data: 2023-09-22T21:21:19.904 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.904 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:19.905 DEBUG:boto:Port: 8001 2023-09-22T21:21:19.905 DEBUG:boto:Params: {} 2023-09-22T21:21:19.905 DEBUG:boto:Token: None 2023-09-22T21:21:19.905 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/ 2023-09-22T21:21:19.905 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2yO6ObytvAIm25ruePYccj4o9Nc= 2023-09-22T21:21:19.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2yO6ObytvAIm25ruePYccj4o9Nc=', 'Content-Length': '0'} 2023-09-22T21:21:19.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f1aba2e8b00f4ad-00650e054f-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.910 DEBUG:boto:b'ffzdqu-46obj21000falseobj22023-09-22T21:20:38.787Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:19.910 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:19.910 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:19.911 DEBUG:boto:path=/ffzdqu-46/?prefix=obj2 2023-09-22T21:21:19.911 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj2 2023-09-22T21:21:19.911 DEBUG:boto:Method: GET 2023-09-22T21:21:19.911 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj2 2023-09-22T21:21:19.911 DEBUG:boto:Data: 2023-09-22T21:21:19.911 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.911 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:19.911 DEBUG:boto:Port: 8000 2023-09-22T21:21:19.911 DEBUG:boto:Params: {} 2023-09-22T21:21:19.911 DEBUG:boto:Token: None 2023-09-22T21:21:19.912 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/ 2023-09-22T21:21:19.912 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2yO6ObytvAIm25ruePYccj4o9Nc= 2023-09-22T21:21:19.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2yO6ObytvAIm25ruePYccj4o9Nc=', 'Content-Length': '0'} 2023-09-22T21:21:19.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007040e5eda86cba89-00650e054f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.958 DEBUG:boto:b'ffzdqu-46obj21000falseobj22023-09-22T21:20:38.787Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:19.959 DEBUG:boto:path=/ffzdqu-46/obj2 2023-09-22T21:21:19.959 DEBUG:boto:auth_path=/ffzdqu-46/obj2 2023-09-22T21:21:19.959 DEBUG:boto:Method: HEAD 2023-09-22T21:21:19.959 DEBUG:boto:Path: /ffzdqu-46/obj2 2023-09-22T21:21:19.959 DEBUG:boto:Data: 2023-09-22T21:21:19.959 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.960 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:19.960 DEBUG:boto:Port: 8001 2023-09-22T21:21:19.960 DEBUG:boto:Params: {} 2023-09-22T21:21:19.960 DEBUG:boto:Token: None 2023-09-22T21:21:19.960 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/obj2 2023-09-22T21:21:19.960 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TOi6REuCmX7F9EceblJb2hxoOVA= 2023-09-22T21:21:19.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TOi6REuCmX7F9EceblJb2hxoOVA=', 'Content-Length': '0'} 2023-09-22T21:21:19.963 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024b774c77dca8120-00650e054f-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.963 DEBUG:boto:path=/ffzdqu-46/obj2 2023-09-22T21:21:19.964 DEBUG:boto:auth_path=/ffzdqu-46/obj2 2023-09-22T21:21:19.964 DEBUG:boto:Method: HEAD 2023-09-22T21:21:19.964 DEBUG:boto:Path: /ffzdqu-46/obj2 2023-09-22T21:21:19.964 DEBUG:boto:Data: 2023-09-22T21:21:19.964 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.964 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:19.964 DEBUG:boto:Port: 8000 2023-09-22T21:21:19.964 DEBUG:boto:Params: {} 2023-09-22T21:21:19.964 DEBUG:boto:Token: None 2023-09-22T21:21:19.965 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/obj2 2023-09-22T21:21:19.965 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TOi6REuCmX7F9EceblJb2hxoOVA= 2023-09-22T21:21:19.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TOi6REuCmX7F9EceblJb2hxoOVA=', 'Content-Length': '0'} 2023-09-22T21:21:19.967 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096eb05e0e3f43c62-00650e054f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:19.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:21:19.968 DEBUG:boto:path=/ffzdqu-46/obj2 2023-09-22T21:21:19.968 DEBUG:boto:auth_path=/ffzdqu-46/obj2 2023-09-22T21:21:19.968 DEBUG:boto:Method: GET 2023-09-22T21:21:19.968 DEBUG:boto:Path: /ffzdqu-46/obj2 2023-09-22T21:21:19.968 DEBUG:boto:Data: 2023-09-22T21:21:19.968 DEBUG:boto:Headers: {} 2023-09-22T21:21:19.968 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:19.968 DEBUG:boto:Port: 8001 2023-09-22T21:21:19.968 DEBUG:boto:Params: {} 2023-09-22T21:21:19.969 DEBUG:boto:Token: None 2023-09-22T21:21:19.969 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:19 GMT /ffzdqu-46/obj2 2023-09-22T21:21:19.969 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4UFc3FPm+5/63muFyy84EHGlCio= 2023-09-22T21:21:19.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4UFc3FPm+5/63muFyy84EHGlCio=', 'Content-Length': '0'} 2023-09-22T21:21:19.971 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046625f862f527d86-00650e054f-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.014 DEBUG:boto:path=/ffzdqu-46/obj2 2023-09-22T21:21:20.015 DEBUG:boto:auth_path=/ffzdqu-46/obj2 2023-09-22T21:21:20.015 DEBUG:boto:Method: GET 2023-09-22T21:21:20.015 DEBUG:boto:Path: /ffzdqu-46/obj2 2023-09-22T21:21:20.015 DEBUG:boto:Data: 2023-09-22T21:21:20.015 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.015 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.015 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.015 DEBUG:boto:Params: {} 2023-09-22T21:21:20.016 DEBUG:boto:Token: None 2023-09-22T21:21:20.016 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/obj2 2023-09-22T21:21:20.016 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oSHQRzLpJBvRlBR3FkJklbQsjUA= 2023-09-22T21:21:20.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oSHQRzLpJBvRlBR3FkJklbQsjUA=', 'Content-Length': '0'} 2023-09-22T21:21:20.018 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033039738156c15cf-00650e0550-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.019 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:20.019 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:20.019 DEBUG:boto:path=/ffzdqu-46/?prefix=obj2 2023-09-22T21:21:20.019 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj2 2023-09-22T21:21:20.019 DEBUG:boto:Method: GET 2023-09-22T21:21:20.019 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj2 2023-09-22T21:21:20.019 DEBUG:boto:Data: 2023-09-22T21:21:20.019 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.020 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.020 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.020 DEBUG:boto:Params: {} 2023-09-22T21:21:20.020 DEBUG:boto:Token: None 2023-09-22T21:21:20.020 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/ 2023-09-22T21:21:20.020 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P4iz2px7AQCzxFf2ERWcUDT41uU= 2023-09-22T21:21:20.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P4iz2px7AQCzxFf2ERWcUDT41uU=', 'Content-Length': '0'} 2023-09-22T21:21:20.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a56f6ac7d5fecce-00650e0550-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.070 DEBUG:boto:b'ffzdqu-46obj21000falseobj22023-09-22T21:20:38.787Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:20.071 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:20.071 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:20.071 DEBUG:boto:path=/ffzdqu-46/?prefix=obj2 2023-09-22T21:21:20.071 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj2 2023-09-22T21:21:20.071 DEBUG:boto:Method: GET 2023-09-22T21:21:20.071 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj2 2023-09-22T21:21:20.072 DEBUG:boto:Data: 2023-09-22T21:21:20.072 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.072 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.072 DEBUG:boto:Params: {} 2023-09-22T21:21:20.072 DEBUG:boto:Token: None 2023-09-22T21:21:20.072 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/ 2023-09-22T21:21:20.073 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P4iz2px7AQCzxFf2ERWcUDT41uU= 2023-09-22T21:21:20.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P4iz2px7AQCzxFf2ERWcUDT41uU=', 'Content-Length': '0'} 2023-09-22T21:21:20.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a185e5e0355bd96-00650e0550-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.078 DEBUG:boto:b'ffzdqu-46obj21000falseobj22023-09-22T21:20:38.787Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:20.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:21:20.078 DEBUG:boto:path=/ffzdqu-46/obj3 2023-09-22T21:21:20.078 DEBUG:boto:auth_path=/ffzdqu-46/obj3 2023-09-22T21:21:20.079 DEBUG:boto:path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.079 DEBUG:boto:auth_path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.079 DEBUG:boto:Method: GET 2023-09-22T21:21:20.079 DEBUG:boto:Path: /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.079 DEBUG:boto:Data: 2023-09-22T21:21:20.079 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.079 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.079 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.079 DEBUG:boto:Params: {} 2023-09-22T21:21:20.079 DEBUG:boto:Token: None 2023-09-22T21:21:20.080 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.080 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iggNe/qjCGwhPgw2tZdHvtXG/Zw= 2023-09-22T21:21:20.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iggNe/qjCGwhPgw2tZdHvtXG/Zw=', 'Content-Length': '0'} 2023-09-22T21:21:20.083 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da77512f350c36cf-00650e0550-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.083 DEBUG:boto:path=/ffzdqu-46/obj3 2023-09-22T21:21:20.083 DEBUG:boto:auth_path=/ffzdqu-46/obj3 2023-09-22T21:21:20.083 DEBUG:boto:path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.083 DEBUG:boto:auth_path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.083 DEBUG:boto:Method: GET 2023-09-22T21:21:20.084 DEBUG:boto:Path: /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.084 DEBUG:boto:Data: 2023-09-22T21:21:20.084 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.084 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.084 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.084 DEBUG:boto:Params: {} 2023-09-22T21:21:20.084 DEBUG:boto:Token: None 2023-09-22T21:21:20.084 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.084 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iggNe/qjCGwhPgw2tZdHvtXG/Zw= 2023-09-22T21:21:20.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iggNe/qjCGwhPgw2tZdHvtXG/Zw=', 'Content-Length': '0'} 2023-09-22T21:21:20.087 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fafd1c1d92910c9c-00650e0550-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.088 DEBUG:boto:path=/ffzdqu-46/obj3 2023-09-22T21:21:20.088 DEBUG:boto:auth_path=/ffzdqu-46/obj3 2023-09-22T21:21:20.088 DEBUG:boto:path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.088 DEBUG:boto:auth_path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.088 DEBUG:boto:Method: HEAD 2023-09-22T21:21:20.088 DEBUG:boto:Path: /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.088 DEBUG:boto:Data: 2023-09-22T21:21:20.088 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.089 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.089 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.089 DEBUG:boto:Params: {} 2023-09-22T21:21:20.089 DEBUG:boto:Token: None 2023-09-22T21:21:20.089 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.089 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eD9WTW4ZoEuCrHS9B4Ud/4NzJjk= 2023-09-22T21:21:20.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eD9WTW4ZoEuCrHS9B4Ud/4NzJjk=', 'Content-Length': '0'} 2023-09-22T21:21:20.092 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019a099ee1731a4bb-00650e0550-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.092 DEBUG:boto:path=/ffzdqu-46/obj3 2023-09-22T21:21:20.092 DEBUG:boto:auth_path=/ffzdqu-46/obj3 2023-09-22T21:21:20.092 DEBUG:boto:path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.092 DEBUG:boto:auth_path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.092 DEBUG:boto:Method: HEAD 2023-09-22T21:21:20.093 DEBUG:boto:Path: /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.093 DEBUG:boto:Data: 2023-09-22T21:21:20.093 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.093 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.093 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.093 DEBUG:boto:Params: {} 2023-09-22T21:21:20.093 DEBUG:boto:Token: None 2023-09-22T21:21:20.093 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.093 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eD9WTW4ZoEuCrHS9B4Ud/4NzJjk= 2023-09-22T21:21:20.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eD9WTW4ZoEuCrHS9B4Ud/4NzJjk=', 'Content-Length': '0'} 2023-09-22T21:21:20.096 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000efe794aeda0a4fcd-00650e0550-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:21:20.097 DEBUG:boto:path=/ffzdqu-46/obj3 2023-09-22T21:21:20.097 DEBUG:boto:auth_path=/ffzdqu-46/obj3 2023-09-22T21:21:20.097 DEBUG:boto:path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.098 DEBUG:boto:auth_path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.098 DEBUG:boto:Method: GET 2023-09-22T21:21:20.098 DEBUG:boto:Path: /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.098 DEBUG:boto:Data: 2023-09-22T21:21:20.098 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.099 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.099 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.099 DEBUG:boto:Params: {} 2023-09-22T21:21:20.099 DEBUG:boto:Token: None 2023-09-22T21:21:20.100 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.100 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iggNe/qjCGwhPgw2tZdHvtXG/Zw= 2023-09-22T21:21:20.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iggNe/qjCGwhPgw2tZdHvtXG/Zw=', 'Content-Length': '0'} 2023-09-22T21:21:20.103 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df5d8ce5f027e216-00650e0550-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.146 DEBUG:boto:path=/ffzdqu-46/obj3 2023-09-22T21:21:20.147 DEBUG:boto:auth_path=/ffzdqu-46/obj3 2023-09-22T21:21:20.147 DEBUG:boto:path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.147 DEBUG:boto:auth_path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.147 DEBUG:boto:Method: GET 2023-09-22T21:21:20.147 DEBUG:boto:Path: /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.147 DEBUG:boto:Data: 2023-09-22T21:21:20.147 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.147 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.148 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.148 DEBUG:boto:Params: {} 2023-09-22T21:21:20.148 DEBUG:boto:Token: None 2023-09-22T21:21:20.148 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:20.148 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iggNe/qjCGwhPgw2tZdHvtXG/Zw= 2023-09-22T21:21:20.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iggNe/qjCGwhPgw2tZdHvtXG/Zw=', 'Content-Length': '0'} 2023-09-22T21:21:20.151 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063e41ef11bc7f35e-00650e0550-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.152 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:20.152 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:20.152 DEBUG:boto:path=/ffzdqu-46/?prefix=obj3 2023-09-22T21:21:20.152 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj3 2023-09-22T21:21:20.152 DEBUG:boto:Method: GET 2023-09-22T21:21:20.152 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj3 2023-09-22T21:21:20.152 DEBUG:boto:Data: 2023-09-22T21:21:20.152 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.153 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.153 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.153 DEBUG:boto:Params: {} 2023-09-22T21:21:20.153 DEBUG:boto:Token: None 2023-09-22T21:21:20.153 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/ 2023-09-22T21:21:20.153 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P4iz2px7AQCzxFf2ERWcUDT41uU= 2023-09-22T21:21:20.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P4iz2px7AQCzxFf2ERWcUDT41uU=', 'Content-Length': '0'} 2023-09-22T21:21:20.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007183d264c4c63ce0-00650e0550-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.158 DEBUG:boto:b'ffzdqu-46obj31000falseobj32023-09-22T21:20:38.842Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:20.158 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:20.158 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:20.158 DEBUG:boto:path=/ffzdqu-46/?prefix=obj3 2023-09-22T21:21:20.159 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj3 2023-09-22T21:21:20.159 DEBUG:boto:Method: GET 2023-09-22T21:21:20.159 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj3 2023-09-22T21:21:20.159 DEBUG:boto:Data: 2023-09-22T21:21:20.159 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.159 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.159 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.159 DEBUG:boto:Params: {} 2023-09-22T21:21:20.159 DEBUG:boto:Token: None 2023-09-22T21:21:20.160 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/ 2023-09-22T21:21:20.160 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P4iz2px7AQCzxFf2ERWcUDT41uU= 2023-09-22T21:21:20.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P4iz2px7AQCzxFf2ERWcUDT41uU=', 'Content-Length': '0'} 2023-09-22T21:21:20.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb1c23255c221ca3-00650e0550-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.164 DEBUG:boto:b'ffzdqu-46obj31000falseobj32023-09-22T21:20:38.842Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:20.164 DEBUG:boto:path=/ffzdqu-46/obj3 2023-09-22T21:21:20.165 DEBUG:boto:auth_path=/ffzdqu-46/obj3 2023-09-22T21:21:20.165 DEBUG:boto:Method: HEAD 2023-09-22T21:21:20.165 DEBUG:boto:Path: /ffzdqu-46/obj3 2023-09-22T21:21:20.165 DEBUG:boto:Data: 2023-09-22T21:21:20.165 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.165 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.165 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.165 DEBUG:boto:Params: {} 2023-09-22T21:21:20.165 DEBUG:boto:Token: None 2023-09-22T21:21:20.166 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/obj3 2023-09-22T21:21:20.166 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+kq8oCue7jN7KyU7jNFxbZq2SKM= 2023-09-22T21:21:20.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+kq8oCue7jN7KyU7jNFxbZq2SKM=', 'Content-Length': '0'} 2023-09-22T21:21:20.168 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb6eaed651791af3-00650e0550-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.169 DEBUG:boto:path=/ffzdqu-46/obj3 2023-09-22T21:21:20.169 DEBUG:boto:auth_path=/ffzdqu-46/obj3 2023-09-22T21:21:20.169 DEBUG:boto:Method: HEAD 2023-09-22T21:21:20.169 DEBUG:boto:Path: /ffzdqu-46/obj3 2023-09-22T21:21:20.169 DEBUG:boto:Data: 2023-09-22T21:21:20.169 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.169 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.170 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.170 DEBUG:boto:Params: {} 2023-09-22T21:21:20.170 DEBUG:boto:Token: None 2023-09-22T21:21:20.170 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/obj3 2023-09-22T21:21:20.170 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+kq8oCue7jN7KyU7jNFxbZq2SKM= 2023-09-22T21:21:20.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+kq8oCue7jN7KyU7jNFxbZq2SKM=', 'Content-Length': '0'} 2023-09-22T21:21:20.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073163b1aa4ad6373-00650e0550-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:21:20.173 DEBUG:boto:path=/ffzdqu-46/obj3 2023-09-22T21:21:20.173 DEBUG:boto:auth_path=/ffzdqu-46/obj3 2023-09-22T21:21:20.173 DEBUG:boto:Method: GET 2023-09-22T21:21:20.173 DEBUG:boto:Path: /ffzdqu-46/obj3 2023-09-22T21:21:20.174 DEBUG:boto:Data: 2023-09-22T21:21:20.174 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.174 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.174 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.174 DEBUG:boto:Params: {} 2023-09-22T21:21:20.174 DEBUG:boto:Token: None 2023-09-22T21:21:20.174 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/obj3 2023-09-22T21:21:20.174 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cykRWOFkumQbYHT0aevHBMnvRD0= 2023-09-22T21:21:20.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cykRWOFkumQbYHT0aevHBMnvRD0=', 'Content-Length': '0'} 2023-09-22T21:21:20.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000751dab3709326345-00650e0550-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.222 DEBUG:boto:path=/ffzdqu-46/obj3 2023-09-22T21:21:20.223 DEBUG:boto:auth_path=/ffzdqu-46/obj3 2023-09-22T21:21:20.223 DEBUG:boto:Method: GET 2023-09-22T21:21:20.223 DEBUG:boto:Path: /ffzdqu-46/obj3 2023-09-22T21:21:20.223 DEBUG:boto:Data: 2023-09-22T21:21:20.223 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.223 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.223 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.223 DEBUG:boto:Params: {} 2023-09-22T21:21:20.224 DEBUG:boto:Token: None 2023-09-22T21:21:20.224 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/obj3 2023-09-22T21:21:20.224 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cykRWOFkumQbYHT0aevHBMnvRD0= 2023-09-22T21:21:20.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cykRWOFkumQbYHT0aevHBMnvRD0=', 'Content-Length': '0'} 2023-09-22T21:21:20.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000da5ab20b875688a-00650e0550-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.227 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:20.227 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:20.227 DEBUG:boto:path=/ffzdqu-46/?prefix=obj3 2023-09-22T21:21:20.227 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj3 2023-09-22T21:21:20.228 DEBUG:boto:Method: GET 2023-09-22T21:21:20.228 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj3 2023-09-22T21:21:20.228 DEBUG:boto:Data: 2023-09-22T21:21:20.228 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.228 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.228 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.228 DEBUG:boto:Params: {} 2023-09-22T21:21:20.228 DEBUG:boto:Token: None 2023-09-22T21:21:20.228 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/ 2023-09-22T21:21:20.229 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P4iz2px7AQCzxFf2ERWcUDT41uU= 2023-09-22T21:21:20.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P4iz2px7AQCzxFf2ERWcUDT41uU=', 'Content-Length': '0'} 2023-09-22T21:21:20.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081de471448a21888-00650e0550-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.274 DEBUG:boto:b'ffzdqu-46obj31000falseobj32023-09-22T21:20:38.842Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:20.275 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:20.275 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:20.275 DEBUG:boto:path=/ffzdqu-46/?prefix=obj3 2023-09-22T21:21:20.275 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj3 2023-09-22T21:21:20.275 DEBUG:boto:Method: GET 2023-09-22T21:21:20.275 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj3 2023-09-22T21:21:20.275 DEBUG:boto:Data: 2023-09-22T21:21:20.276 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.276 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.276 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.276 DEBUG:boto:Params: {} 2023-09-22T21:21:20.276 DEBUG:boto:Token: None 2023-09-22T21:21:20.276 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/ 2023-09-22T21:21:20.276 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P4iz2px7AQCzxFf2ERWcUDT41uU= 2023-09-22T21:21:20.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P4iz2px7AQCzxFf2ERWcUDT41uU=', 'Content-Length': '0'} 2023-09-22T21:21:20.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2ddc9f4f278289a-00650e0550-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.281 DEBUG:boto:b'ffzdqu-46obj31000falseobj32023-09-22T21:20:38.842Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:20.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:21:20.282 DEBUG:boto:path=/ffzdqu-46/obj4 2023-09-22T21:21:20.282 DEBUG:boto:auth_path=/ffzdqu-46/obj4 2023-09-22T21:21:20.282 DEBUG:boto:path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.282 DEBUG:boto:auth_path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.282 DEBUG:boto:Method: GET 2023-09-22T21:21:20.282 DEBUG:boto:Path: /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.282 DEBUG:boto:Data: 2023-09-22T21:21:20.282 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.283 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.283 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.283 DEBUG:boto:Params: {} 2023-09-22T21:21:20.283 DEBUG:boto:Token: None 2023-09-22T21:21:20.283 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.283 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gLM7yabuT4ZydezClvG2P54PlD4= 2023-09-22T21:21:20.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gLM7yabuT4ZydezClvG2P54PlD4=', 'Content-Length': '0'} 2023-09-22T21:21:20.286 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4ffa7fc04e03fdf-00650e0550-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.286 DEBUG:boto:path=/ffzdqu-46/obj4 2023-09-22T21:21:20.286 DEBUG:boto:auth_path=/ffzdqu-46/obj4 2023-09-22T21:21:20.287 DEBUG:boto:path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.287 DEBUG:boto:auth_path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.287 DEBUG:boto:Method: GET 2023-09-22T21:21:20.287 DEBUG:boto:Path: /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.287 DEBUG:boto:Data: 2023-09-22T21:21:20.287 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.287 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.287 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.287 DEBUG:boto:Params: {} 2023-09-22T21:21:20.288 DEBUG:boto:Token: None 2023-09-22T21:21:20.288 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.288 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gLM7yabuT4ZydezClvG2P54PlD4= 2023-09-22T21:21:20.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gLM7yabuT4ZydezClvG2P54PlD4=', 'Content-Length': '0'} 2023-09-22T21:21:20.291 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6bbe339379987df-00650e0550-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.334 DEBUG:boto:path=/ffzdqu-46/obj4 2023-09-22T21:21:20.334 DEBUG:boto:auth_path=/ffzdqu-46/obj4 2023-09-22T21:21:20.335 DEBUG:boto:path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.335 DEBUG:boto:auth_path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.335 DEBUG:boto:Method: HEAD 2023-09-22T21:21:20.335 DEBUG:boto:Path: /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.335 DEBUG:boto:Data: 2023-09-22T21:21:20.335 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.335 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.335 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.335 DEBUG:boto:Params: {} 2023-09-22T21:21:20.336 DEBUG:boto:Token: None 2023-09-22T21:21:20.336 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.336 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RrxeeOS/Cbk7245Sg/VUz3woOKk= 2023-09-22T21:21:20.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RrxeeOS/Cbk7245Sg/VUz3woOKk=', 'Content-Length': '0'} 2023-09-22T21:21:20.338 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000394566d291ae2e63-00650e0550-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.339 DEBUG:boto:path=/ffzdqu-46/obj4 2023-09-22T21:21:20.339 DEBUG:boto:auth_path=/ffzdqu-46/obj4 2023-09-22T21:21:20.339 DEBUG:boto:path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.339 DEBUG:boto:auth_path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.339 DEBUG:boto:Method: HEAD 2023-09-22T21:21:20.339 DEBUG:boto:Path: /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.339 DEBUG:boto:Data: 2023-09-22T21:21:20.339 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.340 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.340 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.340 DEBUG:boto:Params: {} 2023-09-22T21:21:20.340 DEBUG:boto:Token: None 2023-09-22T21:21:20.340 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.340 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RrxeeOS/Cbk7245Sg/VUz3woOKk= 2023-09-22T21:21:20.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RrxeeOS/Cbk7245Sg/VUz3woOKk=', 'Content-Length': '0'} 2023-09-22T21:21:20.343 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088b4822e62cdc4b5-00650e0550-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:21:20.343 DEBUG:boto:path=/ffzdqu-46/obj4 2023-09-22T21:21:20.344 DEBUG:boto:auth_path=/ffzdqu-46/obj4 2023-09-22T21:21:20.344 DEBUG:boto:path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.344 DEBUG:boto:auth_path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.344 DEBUG:boto:Method: GET 2023-09-22T21:21:20.344 DEBUG:boto:Path: /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.344 DEBUG:boto:Data: 2023-09-22T21:21:20.344 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.344 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.344 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.344 DEBUG:boto:Params: {} 2023-09-22T21:21:20.345 DEBUG:boto:Token: None 2023-09-22T21:21:20.345 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.345 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gLM7yabuT4ZydezClvG2P54PlD4= 2023-09-22T21:21:20.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gLM7yabuT4ZydezClvG2P54PlD4=', 'Content-Length': '0'} 2023-09-22T21:21:20.348 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e944151f4480b95d-00650e0550-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.394 DEBUG:boto:path=/ffzdqu-46/obj4 2023-09-22T21:21:20.395 DEBUG:boto:auth_path=/ffzdqu-46/obj4 2023-09-22T21:21:20.395 DEBUG:boto:path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.395 DEBUG:boto:auth_path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.395 DEBUG:boto:Method: GET 2023-09-22T21:21:20.395 DEBUG:boto:Path: /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.395 DEBUG:boto:Data: 2023-09-22T21:21:20.395 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.395 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.395 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.396 DEBUG:boto:Params: {} 2023-09-22T21:21:20.396 DEBUG:boto:Token: None 2023-09-22T21:21:20.396 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:20.396 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gLM7yabuT4ZydezClvG2P54PlD4= 2023-09-22T21:21:20.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gLM7yabuT4ZydezClvG2P54PlD4=', 'Content-Length': '0'} 2023-09-22T21:21:20.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d86e3afcf80ff29f-00650e0550-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.399 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:20.399 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:20.399 DEBUG:boto:path=/ffzdqu-46/?prefix=obj4 2023-09-22T21:21:20.400 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj4 2023-09-22T21:21:20.400 DEBUG:boto:Method: GET 2023-09-22T21:21:20.400 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj4 2023-09-22T21:21:20.400 DEBUG:boto:Data: 2023-09-22T21:21:20.400 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.400 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.400 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.400 DEBUG:boto:Params: {} 2023-09-22T21:21:20.400 DEBUG:boto:Token: None 2023-09-22T21:21:20.401 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/ 2023-09-22T21:21:20.401 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P4iz2px7AQCzxFf2ERWcUDT41uU= 2023-09-22T21:21:20.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P4iz2px7AQCzxFf2ERWcUDT41uU=', 'Content-Length': '0'} 2023-09-22T21:21:20.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000086e9f770b9447b2-00650e0550-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.450 DEBUG:boto:b'ffzdqu-46obj41000falseobj42023-09-22T21:20:38.862Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:20.451 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:20.451 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:20.451 DEBUG:boto:path=/ffzdqu-46/?prefix=obj4 2023-09-22T21:21:20.451 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj4 2023-09-22T21:21:20.451 DEBUG:boto:Method: GET 2023-09-22T21:21:20.451 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj4 2023-09-22T21:21:20.451 DEBUG:boto:Data: 2023-09-22T21:21:20.451 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.452 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.452 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.452 DEBUG:boto:Params: {} 2023-09-22T21:21:20.452 DEBUG:boto:Token: None 2023-09-22T21:21:20.452 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/ 2023-09-22T21:21:20.452 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P4iz2px7AQCzxFf2ERWcUDT41uU= 2023-09-22T21:21:20.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P4iz2px7AQCzxFf2ERWcUDT41uU=', 'Content-Length': '0'} 2023-09-22T21:21:20.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a98b8e3f6c3d37d-00650e0550-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.457 DEBUG:boto:b'ffzdqu-46obj41000falseobj42023-09-22T21:20:38.862Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:20.457 DEBUG:boto:path=/ffzdqu-46/obj4 2023-09-22T21:21:20.457 DEBUG:boto:auth_path=/ffzdqu-46/obj4 2023-09-22T21:21:20.458 DEBUG:boto:Method: HEAD 2023-09-22T21:21:20.458 DEBUG:boto:Path: /ffzdqu-46/obj4 2023-09-22T21:21:20.458 DEBUG:boto:Data: 2023-09-22T21:21:20.458 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.458 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.458 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.458 DEBUG:boto:Params: {} 2023-09-22T21:21:20.458 DEBUG:boto:Token: None 2023-09-22T21:21:20.459 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/obj4 2023-09-22T21:21:20.459 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cjVBkggQxSQBI1PthxB/nyepZPE= 2023-09-22T21:21:20.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cjVBkggQxSQBI1PthxB/nyepZPE=', 'Content-Length': '0'} 2023-09-22T21:21:20.461 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000065e69cc8f6e5ddde-00650e0550-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.461 DEBUG:boto:path=/ffzdqu-46/obj4 2023-09-22T21:21:20.462 DEBUG:boto:auth_path=/ffzdqu-46/obj4 2023-09-22T21:21:20.462 DEBUG:boto:Method: HEAD 2023-09-22T21:21:20.462 DEBUG:boto:Path: /ffzdqu-46/obj4 2023-09-22T21:21:20.462 DEBUG:boto:Data: 2023-09-22T21:21:20.462 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.462 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.462 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.462 DEBUG:boto:Params: {} 2023-09-22T21:21:20.463 DEBUG:boto:Token: None 2023-09-22T21:21:20.463 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/obj4 2023-09-22T21:21:20.463 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cjVBkggQxSQBI1PthxB/nyepZPE= 2023-09-22T21:21:20.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cjVBkggQxSQBI1PthxB/nyepZPE=', 'Content-Length': '0'} 2023-09-22T21:21:20.466 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0a01edc5e16318a-00650e0550-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:21:20.466 DEBUG:boto:path=/ffzdqu-46/obj4 2023-09-22T21:21:20.466 DEBUG:boto:auth_path=/ffzdqu-46/obj4 2023-09-22T21:21:20.466 DEBUG:boto:Method: GET 2023-09-22T21:21:20.466 DEBUG:boto:Path: /ffzdqu-46/obj4 2023-09-22T21:21:20.467 DEBUG:boto:Data: 2023-09-22T21:21:20.467 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.467 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.467 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.467 DEBUG:boto:Params: {} 2023-09-22T21:21:20.467 DEBUG:boto:Token: None 2023-09-22T21:21:20.467 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/obj4 2023-09-22T21:21:20.467 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Af0jS7eU9yUkluwnhnAoWsNbaps= 2023-09-22T21:21:20.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Af0jS7eU9yUkluwnhnAoWsNbaps=', 'Content-Length': '0'} 2023-09-22T21:21:20.470 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c035d93a9dbd100e-00650e0550-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.510 DEBUG:boto:path=/ffzdqu-46/obj4 2023-09-22T21:21:20.510 DEBUG:boto:auth_path=/ffzdqu-46/obj4 2023-09-22T21:21:20.511 DEBUG:boto:Method: GET 2023-09-22T21:21:20.511 DEBUG:boto:Path: /ffzdqu-46/obj4 2023-09-22T21:21:20.511 DEBUG:boto:Data: 2023-09-22T21:21:20.511 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.512 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.512 DEBUG:boto:Params: {} 2023-09-22T21:21:20.512 DEBUG:boto:Token: None 2023-09-22T21:21:20.512 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/obj4 2023-09-22T21:21:20.512 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Af0jS7eU9yUkluwnhnAoWsNbaps= 2023-09-22T21:21:20.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Af0jS7eU9yUkluwnhnAoWsNbaps=', 'Content-Length': '0'} 2023-09-22T21:21:20.515 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da85320ad31f88d0-00650e0550-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.516 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:20.516 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:20.516 DEBUG:boto:path=/ffzdqu-46/?prefix=obj4 2023-09-22T21:21:20.516 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj4 2023-09-22T21:21:20.517 DEBUG:boto:Method: GET 2023-09-22T21:21:20.517 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj4 2023-09-22T21:21:20.517 DEBUG:boto:Data: 2023-09-22T21:21:20.517 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.517 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.517 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.518 DEBUG:boto:Params: {} 2023-09-22T21:21:20.518 DEBUG:boto:Token: None 2023-09-22T21:21:20.518 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/ 2023-09-22T21:21:20.518 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P4iz2px7AQCzxFf2ERWcUDT41uU= 2023-09-22T21:21:20.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P4iz2px7AQCzxFf2ERWcUDT41uU=', 'Content-Length': '0'} 2023-09-22T21:21:20.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029b1d0ef8b51a1cb-00650e0550-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.525 DEBUG:boto:b'ffzdqu-46obj41000falseobj42023-09-22T21:20:38.862Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:20.526 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:20.526 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:20.526 DEBUG:boto:path=/ffzdqu-46/?prefix=obj4 2023-09-22T21:21:20.526 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj4 2023-09-22T21:21:20.526 DEBUG:boto:Method: GET 2023-09-22T21:21:20.527 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj4 2023-09-22T21:21:20.527 DEBUG:boto:Data: 2023-09-22T21:21:20.527 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.527 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.527 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.527 DEBUG:boto:Params: {} 2023-09-22T21:21:20.528 DEBUG:boto:Token: None 2023-09-22T21:21:20.528 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-46/ 2023-09-22T21:21:20.528 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P4iz2px7AQCzxFf2ERWcUDT41uU= 2023-09-22T21:21:20.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P4iz2px7AQCzxFf2ERWcUDT41uU=', 'Content-Length': '0'} 2023-09-22T21:21:20.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da51740349ecc3f0-00650e0550-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.574 DEBUG:boto:b'ffzdqu-46obj41000falseobj42023-09-22T21:20:38.862Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:20.575 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-46 zones={a2, a1} 2023-09-22T21:21:20.575 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-47 2023-09-22T21:21:20.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:21:20.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:21:20.576 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-47 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:21:20.626 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:21:20.626 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:21:20.672 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:21:20.672+0000 7f2596204a40 1 bucket sync caught up with source: 2023-09-22T21:21:20.672 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.28.6, , 00000000001.24.6, , 00000000001.25.6, , , 00000000001.20.6, , , ] 2023-09-22T21:21:20.672 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.28.6, , 00000000001.24.6, , 00000000001.25.6, , , 00000000001.20.6, , , ] 2023-09-22T21:21:20.672 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:21:20.672+0000 7f2596204a40 0 bucket checkpoint complete 2023-09-22T21:21:20.689 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-47 2023-09-22T21:21:20.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:21:20.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:21:20.690 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-47 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:21:20.731 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:21:20.731 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:21:20.772 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:21:20.772+0000 7f6004e17a40 1 bucket sync caught up with source: 2023-09-22T21:21:20.773 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.211.6, , 00000000001.305.6, , 00000000001.303.6, , , 00000000001.215.6, , , ] 2023-09-22T21:21:20.773 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.211.6, , 00000000001.305.6, , 00000000001.303.6, , , 00000000001.215.6, , , ] 2023-09-22T21:21:20.773 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:21:20.772+0000 7f6004e17a40 0 bucket checkpoint complete 2023-09-22T21:21:20.789 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-47 zones={a2, a1} 2023-09-22T21:21:20.789 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:20.789 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:20.789 DEBUG:boto:Method: HEAD 2023-09-22T21:21:20.790 DEBUG:boto:Path: /ffzdqu-47/ 2023-09-22T21:21:20.790 DEBUG:boto:Data: 2023-09-22T21:21:20.790 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.790 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.790 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.791 DEBUG:boto:Params: {} 2023-09-22T21:21:20.791 DEBUG:boto:Token: None 2023-09-22T21:21:20.791 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-47/ 2023-09-22T21:21:20.791 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mHyZqgq0E3fFHkHuRC0wZH2Isc4= 2023-09-22T21:21:20.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mHyZqgq0E3fFHkHuRC0wZH2Isc4=', 'Content-Length': '0'} 2023-09-22T21:21:20.794 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e82032db935842e-00650e0550-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.794 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:20.794 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:20.795 DEBUG:boto:Method: HEAD 2023-09-22T21:21:20.795 DEBUG:boto:Path: /ffzdqu-47/ 2023-09-22T21:21:20.795 DEBUG:boto:Data: 2023-09-22T21:21:20.795 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.795 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.795 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.796 DEBUG:boto:Params: {} 2023-09-22T21:21:20.796 DEBUG:boto:Token: None 2023-09-22T21:21:20.796 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-47/ 2023-09-22T21:21:20.796 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mHyZqgq0E3fFHkHuRC0wZH2Isc4= 2023-09-22T21:21:20.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mHyZqgq0E3fFHkHuRC0wZH2Isc4=', 'Content-Length': '0'} 2023-09-22T21:21:20.799 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aeffa4a9672b1630-00650e0550-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.799 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:21:20.799 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:20.799 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:20.799 DEBUG:boto:path=/ffzdqu-47/?versions&max-keys=999 2023-09-22T21:21:20.800 DEBUG:boto:auth_path=/ffzdqu-47/?versions&max-keys=999 2023-09-22T21:21:20.800 DEBUG:boto:Method: GET 2023-09-22T21:21:20.800 DEBUG:boto:Path: /ffzdqu-47/?versions&max-keys=999 2023-09-22T21:21:20.800 DEBUG:boto:Data: 2023-09-22T21:21:20.800 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.801 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.801 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.801 DEBUG:boto:Params: {} 2023-09-22T21:21:20.801 DEBUG:boto:Token: None 2023-09-22T21:21:20.801 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-47/?versions 2023-09-22T21:21:20.802 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Qap+gHMBSItGGCYQ+rydIZvbEZU= 2023-09-22T21:21:20.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Qap+gHMBSItGGCYQ+rydIZvbEZU=', 'Content-Length': '0'} 2023-09-22T21:21:20.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004286264a60603a38-00650e0550-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.804 DEBUG:boto:b'ffzdqu-47999falseobj1nulltrue2023-09-22T21:20:38.912Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:20:38.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:20:38.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:20:38.999Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:20.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-22T21:21:20.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-22T21:21:20.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-22T21:21:20.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-22T21:21:20.806 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:21:20.806 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:20.806 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:20.806 DEBUG:boto:path=/ffzdqu-47/?versions&max-keys=999 2023-09-22T21:21:20.807 DEBUG:boto:auth_path=/ffzdqu-47/?versions&max-keys=999 2023-09-22T21:21:20.807 DEBUG:boto:Method: GET 2023-09-22T21:21:20.807 DEBUG:boto:Path: /ffzdqu-47/?versions&max-keys=999 2023-09-22T21:21:20.807 DEBUG:boto:Data: 2023-09-22T21:21:20.807 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.807 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.808 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.808 DEBUG:boto:Params: {} 2023-09-22T21:21:20.808 DEBUG:boto:Token: None 2023-09-22T21:21:20.808 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-47/?versions 2023-09-22T21:21:20.808 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Qap+gHMBSItGGCYQ+rydIZvbEZU= 2023-09-22T21:21:20.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Qap+gHMBSItGGCYQ+rydIZvbEZU=', 'Content-Length': '0'} 2023-09-22T21:21:20.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072f610a225a572fd-00650e0550-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.811 DEBUG:boto:b'ffzdqu-47999falseobj1nulltrue2023-09-22T21:20:38.912Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:20:38.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:20:38.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:20:38.999Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:20.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-22T21:21:20.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-22T21:21:20.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-22T21:21:20.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-22T21:21:20.813 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:20.813 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:20.813 DEBUG:boto:path=/ffzdqu-47/?versions&max-keys=999 2023-09-22T21:21:20.813 DEBUG:boto:auth_path=/ffzdqu-47/?versions&max-keys=999 2023-09-22T21:21:20.814 DEBUG:boto:Method: GET 2023-09-22T21:21:20.814 DEBUG:boto:Path: /ffzdqu-47/?versions&max-keys=999 2023-09-22T21:21:20.814 DEBUG:boto:Data: 2023-09-22T21:21:20.814 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.814 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.814 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.815 DEBUG:boto:Params: {} 2023-09-22T21:21:20.815 DEBUG:boto:Token: None 2023-09-22T21:21:20.815 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-47/?versions 2023-09-22T21:21:20.815 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Qap+gHMBSItGGCYQ+rydIZvbEZU= 2023-09-22T21:21:20.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Qap+gHMBSItGGCYQ+rydIZvbEZU=', 'Content-Length': '0'} 2023-09-22T21:21:20.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008934b2f720d278ad-00650e0550-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.818 DEBUG:boto:b'ffzdqu-47999falseobj1nulltrue2023-09-22T21:20:38.912Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:20:38.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:20:38.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:20:38.999Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:20.819 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:20.819 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:20.819 DEBUG:boto:path=/ffzdqu-47/?versions&max-keys=999 2023-09-22T21:21:20.819 DEBUG:boto:auth_path=/ffzdqu-47/?versions&max-keys=999 2023-09-22T21:21:20.819 DEBUG:boto:Method: GET 2023-09-22T21:21:20.820 DEBUG:boto:Path: /ffzdqu-47/?versions&max-keys=999 2023-09-22T21:21:20.820 DEBUG:boto:Data: 2023-09-22T21:21:20.820 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.820 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.820 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.821 DEBUG:boto:Params: {} 2023-09-22T21:21:20.821 DEBUG:boto:Token: None 2023-09-22T21:21:20.821 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-47/?versions 2023-09-22T21:21:20.821 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Qap+gHMBSItGGCYQ+rydIZvbEZU= 2023-09-22T21:21:20.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Qap+gHMBSItGGCYQ+rydIZvbEZU=', 'Content-Length': '0'} 2023-09-22T21:21:20.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ead2cfdbc16c59a-00650e0550-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.824 DEBUG:boto:b'ffzdqu-47999falseobj1nulltrue2023-09-22T21:20:38.912Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:20:38.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:20:38.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:20:38.999Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:20.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:21:20.825 DEBUG:boto:path=/ffzdqu-47/obj1 2023-09-22T21:21:20.825 DEBUG:boto:auth_path=/ffzdqu-47/obj1 2023-09-22T21:21:20.825 DEBUG:boto:path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.826 DEBUG:boto:auth_path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.826 DEBUG:boto:Method: GET 2023-09-22T21:21:20.826 DEBUG:boto:Path: /ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.826 DEBUG:boto:Data: 2023-09-22T21:21:20.826 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.827 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.827 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.827 DEBUG:boto:Params: {} 2023-09-22T21:21:20.827 DEBUG:boto:Token: None 2023-09-22T21:21:20.827 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.828 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:28279LZBIN6XElJwLDWuHWDP+II= 2023-09-22T21:21:20.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:28279LZBIN6XElJwLDWuHWDP+II=', 'Content-Length': '0'} 2023-09-22T21:21:20.829 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b3065415fce8971-00650e0550-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.830 DEBUG:boto:path=/ffzdqu-47/obj1 2023-09-22T21:21:20.830 DEBUG:boto:auth_path=/ffzdqu-47/obj1 2023-09-22T21:21:20.830 DEBUG:boto:path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.830 DEBUG:boto:auth_path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.830 DEBUG:boto:Method: GET 2023-09-22T21:21:20.831 DEBUG:boto:Path: /ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.831 DEBUG:boto:Data: 2023-09-22T21:21:20.831 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.831 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.831 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.831 DEBUG:boto:Params: {} 2023-09-22T21:21:20.832 DEBUG:boto:Token: None 2023-09-22T21:21:20.832 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.832 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:28279LZBIN6XElJwLDWuHWDP+II= 2023-09-22T21:21:20.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:28279LZBIN6XElJwLDWuHWDP+II=', 'Content-Length': '0'} 2023-09-22T21:21:20.834 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007efb1c8a3432ea07-00650e0550-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.882 DEBUG:boto:path=/ffzdqu-47/obj1 2023-09-22T21:21:20.883 DEBUG:boto:auth_path=/ffzdqu-47/obj1 2023-09-22T21:21:20.883 DEBUG:boto:path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.883 DEBUG:boto:auth_path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.883 DEBUG:boto:Method: HEAD 2023-09-22T21:21:20.883 DEBUG:boto:Path: /ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.884 DEBUG:boto:Data: 2023-09-22T21:21:20.884 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.884 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.884 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.884 DEBUG:boto:Params: {} 2023-09-22T21:21:20.884 DEBUG:boto:Token: None 2023-09-22T21:21:20.885 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.885 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:izv0sYeARqzZsv5qre0AiyQw9/M= 2023-09-22T21:21:20.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:izv0sYeARqzZsv5qre0AiyQw9/M=', 'Content-Length': '0'} 2023-09-22T21:21:20.887 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c929f3189d91a3c-00650e0550-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.888 DEBUG:boto:path=/ffzdqu-47/obj1 2023-09-22T21:21:20.888 DEBUG:boto:auth_path=/ffzdqu-47/obj1 2023-09-22T21:21:20.888 DEBUG:boto:path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.888 DEBUG:boto:auth_path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.889 DEBUG:boto:Method: HEAD 2023-09-22T21:21:20.889 DEBUG:boto:Path: /ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.889 DEBUG:boto:Data: 2023-09-22T21:21:20.889 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.889 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.889 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.890 DEBUG:boto:Params: {} 2023-09-22T21:21:20.890 DEBUG:boto:Token: None 2023-09-22T21:21:20.890 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.890 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:izv0sYeARqzZsv5qre0AiyQw9/M= 2023-09-22T21:21:20.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:izv0sYeARqzZsv5qre0AiyQw9/M=', 'Content-Length': '0'} 2023-09-22T21:21:20.893 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dbdc359f3d1b909d-00650e0550-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:21:20.893 DEBUG:boto:path=/ffzdqu-47/obj1 2023-09-22T21:21:20.893 DEBUG:boto:auth_path=/ffzdqu-47/obj1 2023-09-22T21:21:20.894 DEBUG:boto:path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.894 DEBUG:boto:auth_path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.894 DEBUG:boto:Method: GET 2023-09-22T21:21:20.894 DEBUG:boto:Path: /ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.894 DEBUG:boto:Data: 2023-09-22T21:21:20.895 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.895 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.895 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.895 DEBUG:boto:Params: {} 2023-09-22T21:21:20.895 DEBUG:boto:Token: None 2023-09-22T21:21:20.895 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.896 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:28279LZBIN6XElJwLDWuHWDP+II= 2023-09-22T21:21:20.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:28279LZBIN6XElJwLDWuHWDP+II=', 'Content-Length': '0'} 2023-09-22T21:21:20.897 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082292a3fb08705e8-00650e0550-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.942 DEBUG:boto:path=/ffzdqu-47/obj1 2023-09-22T21:21:20.943 DEBUG:boto:auth_path=/ffzdqu-47/obj1 2023-09-22T21:21:20.943 DEBUG:boto:path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.943 DEBUG:boto:auth_path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.943 DEBUG:boto:Method: GET 2023-09-22T21:21:20.943 DEBUG:boto:Path: /ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.943 DEBUG:boto:Data: 2023-09-22T21:21:20.943 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.943 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.943 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.944 DEBUG:boto:Params: {} 2023-09-22T21:21:20.944 DEBUG:boto:Token: None 2023-09-22T21:21:20.944 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-47/obj1?versionId=null 2023-09-22T21:21:20.944 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:28279LZBIN6XElJwLDWuHWDP+II= 2023-09-22T21:21:20.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:28279LZBIN6XElJwLDWuHWDP+II=', 'Content-Length': '0'} 2023-09-22T21:21:20.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6406915a4682f70-00650e0550-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.947 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:20.947 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:20.947 DEBUG:boto:path=/ffzdqu-47/?prefix=obj1 2023-09-22T21:21:20.948 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj1 2023-09-22T21:21:20.948 DEBUG:boto:Method: GET 2023-09-22T21:21:20.948 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj1 2023-09-22T21:21:20.948 DEBUG:boto:Data: 2023-09-22T21:21:20.948 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.948 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.948 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.948 DEBUG:boto:Params: {} 2023-09-22T21:21:20.948 DEBUG:boto:Token: None 2023-09-22T21:21:20.949 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-47/ 2023-09-22T21:21:20.949 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/28eq6DSjskApvh/TgwCKfoNldg= 2023-09-22T21:21:20.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/28eq6DSjskApvh/TgwCKfoNldg=', 'Content-Length': '0'} 2023-09-22T21:21:20.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d94e7b62f9d6c301-00650e0550-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.953 DEBUG:boto:b'ffzdqu-47obj11000falseobj12023-09-22T21:20:38.912Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:20.954 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:20.954 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:20.954 DEBUG:boto:path=/ffzdqu-47/?prefix=obj1 2023-09-22T21:21:20.954 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj1 2023-09-22T21:21:20.954 DEBUG:boto:Method: GET 2023-09-22T21:21:20.954 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj1 2023-09-22T21:21:20.954 DEBUG:boto:Data: 2023-09-22T21:21:20.954 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.954 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.955 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.955 DEBUG:boto:Params: {} 2023-09-22T21:21:20.955 DEBUG:boto:Token: None 2023-09-22T21:21:20.955 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-47/ 2023-09-22T21:21:20.955 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/28eq6DSjskApvh/TgwCKfoNldg= 2023-09-22T21:21:20.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/28eq6DSjskApvh/TgwCKfoNldg=', 'Content-Length': '0'} 2023-09-22T21:21:20.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003183269c6cad3d1-00650e0550-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.959 DEBUG:boto:b'ffzdqu-47obj11000falseobj12023-09-22T21:20:38.912Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:20.962 DEBUG:boto:path=/ffzdqu-47/obj1 2023-09-22T21:21:20.962 DEBUG:boto:auth_path=/ffzdqu-47/obj1 2023-09-22T21:21:20.962 DEBUG:boto:Method: HEAD 2023-09-22T21:21:20.962 DEBUG:boto:Path: /ffzdqu-47/obj1 2023-09-22T21:21:20.962 DEBUG:boto:Data: 2023-09-22T21:21:20.962 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.963 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.963 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.963 DEBUG:boto:Params: {} 2023-09-22T21:21:20.963 DEBUG:boto:Token: None 2023-09-22T21:21:20.963 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-47/obj1 2023-09-22T21:21:20.963 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xHnkUODrODVJUTC2sOrqqpnbMgQ= 2023-09-22T21:21:20.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xHnkUODrODVJUTC2sOrqqpnbMgQ=', 'Content-Length': '0'} 2023-09-22T21:21:20.966 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000208f5557c21b167c-00650e0550-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.966 DEBUG:boto:path=/ffzdqu-47/obj1 2023-09-22T21:21:20.967 DEBUG:boto:auth_path=/ffzdqu-47/obj1 2023-09-22T21:21:20.967 DEBUG:boto:Method: HEAD 2023-09-22T21:21:20.967 DEBUG:boto:Path: /ffzdqu-47/obj1 2023-09-22T21:21:20.967 DEBUG:boto:Data: 2023-09-22T21:21:20.967 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.968 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.968 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.968 DEBUG:boto:Params: {} 2023-09-22T21:21:20.968 DEBUG:boto:Token: None 2023-09-22T21:21:20.968 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-47/obj1 2023-09-22T21:21:20.969 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xHnkUODrODVJUTC2sOrqqpnbMgQ= 2023-09-22T21:21:20.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xHnkUODrODVJUTC2sOrqqpnbMgQ=', 'Content-Length': '0'} 2023-09-22T21:21:20.971 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007df99c4aa5235824-00650e0550-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:21:20.972 DEBUG:boto:path=/ffzdqu-47/obj1 2023-09-22T21:21:20.972 DEBUG:boto:auth_path=/ffzdqu-47/obj1 2023-09-22T21:21:20.972 DEBUG:boto:Method: GET 2023-09-22T21:21:20.972 DEBUG:boto:Path: /ffzdqu-47/obj1 2023-09-22T21:21:20.972 DEBUG:boto:Data: 2023-09-22T21:21:20.973 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.973 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:20.973 DEBUG:boto:Port: 8001 2023-09-22T21:21:20.973 DEBUG:boto:Params: {} 2023-09-22T21:21:20.973 DEBUG:boto:Token: None 2023-09-22T21:21:20.973 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-47/obj1 2023-09-22T21:21:20.973 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LGzz2zf7R1NZfy1kf+XIi2xApr4= 2023-09-22T21:21:20.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LGzz2zf7R1NZfy1kf+XIi2xApr4=', 'Content-Length': '0'} 2023-09-22T21:21:20.976 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e4f492c155daf93-00650e0550-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:20.976 DEBUG:boto:path=/ffzdqu-47/obj1 2023-09-22T21:21:20.976 DEBUG:boto:auth_path=/ffzdqu-47/obj1 2023-09-22T21:21:20.976 DEBUG:boto:Method: GET 2023-09-22T21:21:20.977 DEBUG:boto:Path: /ffzdqu-47/obj1 2023-09-22T21:21:20.977 DEBUG:boto:Data: 2023-09-22T21:21:20.977 DEBUG:boto:Headers: {} 2023-09-22T21:21:20.977 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:20.977 DEBUG:boto:Port: 8000 2023-09-22T21:21:20.977 DEBUG:boto:Params: {} 2023-09-22T21:21:20.977 DEBUG:boto:Token: None 2023-09-22T21:21:20.977 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:20 GMT /ffzdqu-47/obj1 2023-09-22T21:21:20.977 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LGzz2zf7R1NZfy1kf+XIi2xApr4= 2023-09-22T21:21:20.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LGzz2zf7R1NZfy1kf+XIi2xApr4=', 'Content-Length': '0'} 2023-09-22T21:21:20.980 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037006f501f16d8d6-00650e0550-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.022 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:21.023 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:21.023 DEBUG:boto:path=/ffzdqu-47/?prefix=obj1 2023-09-22T21:21:21.023 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj1 2023-09-22T21:21:21.023 DEBUG:boto:Method: GET 2023-09-22T21:21:21.023 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj1 2023-09-22T21:21:21.023 DEBUG:boto:Data: 2023-09-22T21:21:21.023 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.023 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.024 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.024 DEBUG:boto:Params: {} 2023-09-22T21:21:21.024 DEBUG:boto:Token: None 2023-09-22T21:21:21.024 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/ 2023-09-22T21:21:21.024 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI= 2023-09-22T21:21:21.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI=', 'Content-Length': '0'} 2023-09-22T21:21:21.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb7ab4789bd53589-00650e0551-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.029 DEBUG:boto:b'ffzdqu-47obj11000falseobj12023-09-22T21:20:38.912Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:21.029 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:21.029 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:21.029 DEBUG:boto:path=/ffzdqu-47/?prefix=obj1 2023-09-22T21:21:21.030 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj1 2023-09-22T21:21:21.030 DEBUG:boto:Method: GET 2023-09-22T21:21:21.030 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj1 2023-09-22T21:21:21.030 DEBUG:boto:Data: 2023-09-22T21:21:21.030 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.030 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.030 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.030 DEBUG:boto:Params: {} 2023-09-22T21:21:21.031 DEBUG:boto:Token: None 2023-09-22T21:21:21.031 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/ 2023-09-22T21:21:21.031 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI= 2023-09-22T21:21:21.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI=', 'Content-Length': '0'} 2023-09-22T21:21:21.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091b8dc47382a6289-00650e0551-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.036 DEBUG:boto:b'ffzdqu-47obj11000falseobj12023-09-22T21:20:38.912Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:21.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:21:21.037 DEBUG:boto:path=/ffzdqu-47/obj2 2023-09-22T21:21:21.037 DEBUG:boto:auth_path=/ffzdqu-47/obj2 2023-09-22T21:21:21.037 DEBUG:boto:path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.037 DEBUG:boto:auth_path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.037 DEBUG:boto:Method: GET 2023-09-22T21:21:21.037 DEBUG:boto:Path: /ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.037 DEBUG:boto:Data: 2023-09-22T21:21:21.037 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.037 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.038 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.038 DEBUG:boto:Params: {} 2023-09-22T21:21:21.038 DEBUG:boto:Token: None 2023-09-22T21:21:21.038 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.038 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:U2eGmDdg82uQb/42cMJxkBo9tx0= 2023-09-22T21:21:21.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:U2eGmDdg82uQb/42cMJxkBo9tx0=', 'Content-Length': '0'} 2023-09-22T21:21:21.041 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000891e22b88e7ccb05-00650e0551-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.086 DEBUG:boto:path=/ffzdqu-47/obj2 2023-09-22T21:21:21.086 DEBUG:boto:auth_path=/ffzdqu-47/obj2 2023-09-22T21:21:21.087 DEBUG:boto:path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.087 DEBUG:boto:auth_path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.087 DEBUG:boto:Method: GET 2023-09-22T21:21:21.087 DEBUG:boto:Path: /ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.087 DEBUG:boto:Data: 2023-09-22T21:21:21.087 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.087 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.087 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.087 DEBUG:boto:Params: {} 2023-09-22T21:21:21.087 DEBUG:boto:Token: None 2023-09-22T21:21:21.088 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.088 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:U2eGmDdg82uQb/42cMJxkBo9tx0= 2023-09-22T21:21:21.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:U2eGmDdg82uQb/42cMJxkBo9tx0=', 'Content-Length': '0'} 2023-09-22T21:21:21.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a096e7ffa472602-00650e0551-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.091 DEBUG:boto:path=/ffzdqu-47/obj2 2023-09-22T21:21:21.091 DEBUG:boto:auth_path=/ffzdqu-47/obj2 2023-09-22T21:21:21.091 DEBUG:boto:path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.091 DEBUG:boto:auth_path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.091 DEBUG:boto:Method: HEAD 2023-09-22T21:21:21.091 DEBUG:boto:Path: /ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.091 DEBUG:boto:Data: 2023-09-22T21:21:21.092 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.092 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.092 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.092 DEBUG:boto:Params: {} 2023-09-22T21:21:21.092 DEBUG:boto:Token: None 2023-09-22T21:21:21.092 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.092 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:96b7lJvpmzwD9uVQWp9nuaJ47os= 2023-09-22T21:21:21.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:96b7lJvpmzwD9uVQWp9nuaJ47os=', 'Content-Length': '0'} 2023-09-22T21:21:21.095 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bad90a1812e9ffc8-00650e0551-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.095 DEBUG:boto:path=/ffzdqu-47/obj2 2023-09-22T21:21:21.095 DEBUG:boto:auth_path=/ffzdqu-47/obj2 2023-09-22T21:21:21.096 DEBUG:boto:path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.096 DEBUG:boto:auth_path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.096 DEBUG:boto:Method: HEAD 2023-09-22T21:21:21.096 DEBUG:boto:Path: /ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.096 DEBUG:boto:Data: 2023-09-22T21:21:21.096 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.096 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.096 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.096 DEBUG:boto:Params: {} 2023-09-22T21:21:21.097 DEBUG:boto:Token: None 2023-09-22T21:21:21.097 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.097 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:96b7lJvpmzwD9uVQWp9nuaJ47os= 2023-09-22T21:21:21.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:96b7lJvpmzwD9uVQWp9nuaJ47os=', 'Content-Length': '0'} 2023-09-22T21:21:21.099 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bec6c808ddd53e7a-00650e0551-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:21:21.100 DEBUG:boto:path=/ffzdqu-47/obj2 2023-09-22T21:21:21.100 DEBUG:boto:auth_path=/ffzdqu-47/obj2 2023-09-22T21:21:21.100 DEBUG:boto:path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.100 DEBUG:boto:auth_path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.100 DEBUG:boto:Method: GET 2023-09-22T21:21:21.100 DEBUG:boto:Path: /ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.101 DEBUG:boto:Data: 2023-09-22T21:21:21.101 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.101 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.101 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.101 DEBUG:boto:Params: {} 2023-09-22T21:21:21.101 DEBUG:boto:Token: None 2023-09-22T21:21:21.101 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.101 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:U2eGmDdg82uQb/42cMJxkBo9tx0= 2023-09-22T21:21:21.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:U2eGmDdg82uQb/42cMJxkBo9tx0=', 'Content-Length': '0'} 2023-09-22T21:21:21.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000145fc87e6749069d-00650e0551-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.105 DEBUG:boto:path=/ffzdqu-47/obj2 2023-09-22T21:21:21.105 DEBUG:boto:auth_path=/ffzdqu-47/obj2 2023-09-22T21:21:21.105 DEBUG:boto:path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.105 DEBUG:boto:auth_path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.105 DEBUG:boto:Method: GET 2023-09-22T21:21:21.105 DEBUG:boto:Path: /ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.105 DEBUG:boto:Data: 2023-09-22T21:21:21.106 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.106 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.106 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.106 DEBUG:boto:Params: {} 2023-09-22T21:21:21.106 DEBUG:boto:Token: None 2023-09-22T21:21:21.106 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj2?versionId=null 2023-09-22T21:21:21.106 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:U2eGmDdg82uQb/42cMJxkBo9tx0= 2023-09-22T21:21:21.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:U2eGmDdg82uQb/42cMJxkBo9tx0=', 'Content-Length': '0'} 2023-09-22T21:21:21.109 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b435c95d9dbf029-00650e0551-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.150 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:21.150 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:21.151 DEBUG:boto:path=/ffzdqu-47/?prefix=obj2 2023-09-22T21:21:21.151 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj2 2023-09-22T21:21:21.151 DEBUG:boto:Method: GET 2023-09-22T21:21:21.151 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj2 2023-09-22T21:21:21.151 DEBUG:boto:Data: 2023-09-22T21:21:21.151 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.151 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.151 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.151 DEBUG:boto:Params: {} 2023-09-22T21:21:21.152 DEBUG:boto:Token: None 2023-09-22T21:21:21.152 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/ 2023-09-22T21:21:21.152 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI= 2023-09-22T21:21:21.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI=', 'Content-Length': '0'} 2023-09-22T21:21:21.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dff34f8a3e1cfee-00650e0551-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.156 DEBUG:boto:b'ffzdqu-47obj21000falseobj22023-09-22T21:20:38.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:21.157 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:21.157 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:21.157 DEBUG:boto:path=/ffzdqu-47/?prefix=obj2 2023-09-22T21:21:21.157 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj2 2023-09-22T21:21:21.157 DEBUG:boto:Method: GET 2023-09-22T21:21:21.157 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj2 2023-09-22T21:21:21.158 DEBUG:boto:Data: 2023-09-22T21:21:21.158 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.158 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.158 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.158 DEBUG:boto:Params: {} 2023-09-22T21:21:21.158 DEBUG:boto:Token: None 2023-09-22T21:21:21.158 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/ 2023-09-22T21:21:21.158 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI= 2023-09-22T21:21:21.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI=', 'Content-Length': '0'} 2023-09-22T21:21:21.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a717a69ae71a3263-00650e0551-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.163 DEBUG:boto:b'ffzdqu-47obj21000falseobj22023-09-22T21:20:38.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:21.166 DEBUG:boto:path=/ffzdqu-47/obj2 2023-09-22T21:21:21.166 DEBUG:boto:auth_path=/ffzdqu-47/obj2 2023-09-22T21:21:21.166 DEBUG:boto:Method: HEAD 2023-09-22T21:21:21.166 DEBUG:boto:Path: /ffzdqu-47/obj2 2023-09-22T21:21:21.166 DEBUG:boto:Data: 2023-09-22T21:21:21.166 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.166 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.166 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.166 DEBUG:boto:Params: {} 2023-09-22T21:21:21.167 DEBUG:boto:Token: None 2023-09-22T21:21:21.167 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj2 2023-09-22T21:21:21.167 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pPqFxtnJlR1HfTo4C47NuY4k0lo= 2023-09-22T21:21:21.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pPqFxtnJlR1HfTo4C47NuY4k0lo=', 'Content-Length': '0'} 2023-09-22T21:21:21.170 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3a16126015f4c6d-00650e0551-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.170 DEBUG:boto:path=/ffzdqu-47/obj2 2023-09-22T21:21:21.170 DEBUG:boto:auth_path=/ffzdqu-47/obj2 2023-09-22T21:21:21.170 DEBUG:boto:Method: HEAD 2023-09-22T21:21:21.171 DEBUG:boto:Path: /ffzdqu-47/obj2 2023-09-22T21:21:21.171 DEBUG:boto:Data: 2023-09-22T21:21:21.171 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.171 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.171 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.171 DEBUG:boto:Params: {} 2023-09-22T21:21:21.171 DEBUG:boto:Token: None 2023-09-22T21:21:21.171 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj2 2023-09-22T21:21:21.171 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pPqFxtnJlR1HfTo4C47NuY4k0lo= 2023-09-22T21:21:21.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pPqFxtnJlR1HfTo4C47NuY4k0lo=', 'Content-Length': '0'} 2023-09-22T21:21:21.174 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db9600de3bd18d8b-00650e0551-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:21:21.175 DEBUG:boto:path=/ffzdqu-47/obj2 2023-09-22T21:21:21.175 DEBUG:boto:auth_path=/ffzdqu-47/obj2 2023-09-22T21:21:21.175 DEBUG:boto:Method: GET 2023-09-22T21:21:21.175 DEBUG:boto:Path: /ffzdqu-47/obj2 2023-09-22T21:21:21.175 DEBUG:boto:Data: 2023-09-22T21:21:21.175 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.175 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.175 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.175 DEBUG:boto:Params: {} 2023-09-22T21:21:21.175 DEBUG:boto:Token: None 2023-09-22T21:21:21.176 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj2 2023-09-22T21:21:21.176 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i6aC/6gSQ7rDP1UgOo76xIIGjx8= 2023-09-22T21:21:21.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i6aC/6gSQ7rDP1UgOo76xIIGjx8=', 'Content-Length': '0'} 2023-09-22T21:21:21.178 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac569f0e2470e202-00650e0551-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.222 DEBUG:boto:path=/ffzdqu-47/obj2 2023-09-22T21:21:21.223 DEBUG:boto:auth_path=/ffzdqu-47/obj2 2023-09-22T21:21:21.223 DEBUG:boto:Method: GET 2023-09-22T21:21:21.223 DEBUG:boto:Path: /ffzdqu-47/obj2 2023-09-22T21:21:21.223 DEBUG:boto:Data: 2023-09-22T21:21:21.223 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.223 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.223 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.223 DEBUG:boto:Params: {} 2023-09-22T21:21:21.223 DEBUG:boto:Token: None 2023-09-22T21:21:21.224 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj2 2023-09-22T21:21:21.224 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i6aC/6gSQ7rDP1UgOo76xIIGjx8= 2023-09-22T21:21:21.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i6aC/6gSQ7rDP1UgOo76xIIGjx8=', 'Content-Length': '0'} 2023-09-22T21:21:21.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f57816b5e8598088-00650e0551-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.227 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:21.227 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:21.227 DEBUG:boto:path=/ffzdqu-47/?prefix=obj2 2023-09-22T21:21:21.227 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj2 2023-09-22T21:21:21.227 DEBUG:boto:Method: GET 2023-09-22T21:21:21.228 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj2 2023-09-22T21:21:21.228 DEBUG:boto:Data: 2023-09-22T21:21:21.228 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.228 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.228 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.228 DEBUG:boto:Params: {} 2023-09-22T21:21:21.228 DEBUG:boto:Token: None 2023-09-22T21:21:21.228 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/ 2023-09-22T21:21:21.229 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI= 2023-09-22T21:21:21.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI=', 'Content-Length': '0'} 2023-09-22T21:21:21.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000929099845c5d8fae-00650e0551-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.234 DEBUG:boto:b'ffzdqu-47obj21000falseobj22023-09-22T21:20:38.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:21.234 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:21.234 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:21.235 DEBUG:boto:path=/ffzdqu-47/?prefix=obj2 2023-09-22T21:21:21.235 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj2 2023-09-22T21:21:21.235 DEBUG:boto:Method: GET 2023-09-22T21:21:21.235 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj2 2023-09-22T21:21:21.235 DEBUG:boto:Data: 2023-09-22T21:21:21.235 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.235 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.235 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.235 DEBUG:boto:Params: {} 2023-09-22T21:21:21.235 DEBUG:boto:Token: None 2023-09-22T21:21:21.236 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/ 2023-09-22T21:21:21.236 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI= 2023-09-22T21:21:21.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI=', 'Content-Length': '0'} 2023-09-22T21:21:21.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e523e68e1b5f0c38-00650e0551-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.282 DEBUG:boto:b'ffzdqu-47obj21000falseobj22023-09-22T21:20:38.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:21.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:21:21.283 DEBUG:boto:path=/ffzdqu-47/obj3 2023-09-22T21:21:21.283 DEBUG:boto:auth_path=/ffzdqu-47/obj3 2023-09-22T21:21:21.283 DEBUG:boto:path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.283 DEBUG:boto:auth_path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.283 DEBUG:boto:Method: GET 2023-09-22T21:21:21.283 DEBUG:boto:Path: /ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.284 DEBUG:boto:Data: 2023-09-22T21:21:21.284 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.284 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.284 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.284 DEBUG:boto:Params: {} 2023-09-22T21:21:21.284 DEBUG:boto:Token: None 2023-09-22T21:21:21.284 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.284 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PUqaozVVsj3sjqFRUvRUkFsb8tY= 2023-09-22T21:21:21.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PUqaozVVsj3sjqFRUvRUkFsb8tY=', 'Content-Length': '0'} 2023-09-22T21:21:21.287 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051f8dd2c0b1ecd65-00650e0551-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.288 DEBUG:boto:path=/ffzdqu-47/obj3 2023-09-22T21:21:21.288 DEBUG:boto:auth_path=/ffzdqu-47/obj3 2023-09-22T21:21:21.288 DEBUG:boto:path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.288 DEBUG:boto:auth_path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.288 DEBUG:boto:Method: GET 2023-09-22T21:21:21.288 DEBUG:boto:Path: /ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.288 DEBUG:boto:Data: 2023-09-22T21:21:21.289 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.289 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.289 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.289 DEBUG:boto:Params: {} 2023-09-22T21:21:21.289 DEBUG:boto:Token: None 2023-09-22T21:21:21.289 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.289 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PUqaozVVsj3sjqFRUvRUkFsb8tY= 2023-09-22T21:21:21.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PUqaozVVsj3sjqFRUvRUkFsb8tY=', 'Content-Length': '0'} 2023-09-22T21:21:21.292 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b359a1bab5bf230-00650e0551-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.292 DEBUG:boto:path=/ffzdqu-47/obj3 2023-09-22T21:21:21.292 DEBUG:boto:auth_path=/ffzdqu-47/obj3 2023-09-22T21:21:21.293 DEBUG:boto:path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.293 DEBUG:boto:auth_path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.293 DEBUG:boto:Method: HEAD 2023-09-22T21:21:21.293 DEBUG:boto:Path: /ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.293 DEBUG:boto:Data: 2023-09-22T21:21:21.293 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.293 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.293 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.293 DEBUG:boto:Params: {} 2023-09-22T21:21:21.293 DEBUG:boto:Token: None 2023-09-22T21:21:21.294 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.294 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1MX/dBU1+3M/d14Pyxf5344s7r4= 2023-09-22T21:21:21.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1MX/dBU1+3M/d14Pyxf5344s7r4=', 'Content-Length': '0'} 2023-09-22T21:21:21.296 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac419921dc5567b5-00650e0551-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.297 DEBUG:boto:path=/ffzdqu-47/obj3 2023-09-22T21:21:21.297 DEBUG:boto:auth_path=/ffzdqu-47/obj3 2023-09-22T21:21:21.297 DEBUG:boto:path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.297 DEBUG:boto:auth_path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.297 DEBUG:boto:Method: HEAD 2023-09-22T21:21:21.297 DEBUG:boto:Path: /ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.297 DEBUG:boto:Data: 2023-09-22T21:21:21.298 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.298 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.298 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.298 DEBUG:boto:Params: {} 2023-09-22T21:21:21.298 DEBUG:boto:Token: None 2023-09-22T21:21:21.298 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.298 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1MX/dBU1+3M/d14Pyxf5344s7r4= 2023-09-22T21:21:21.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1MX/dBU1+3M/d14Pyxf5344s7r4=', 'Content-Length': '0'} 2023-09-22T21:21:21.301 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000248adadc9e22c204-00650e0551-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:21:21.302 DEBUG:boto:path=/ffzdqu-47/obj3 2023-09-22T21:21:21.302 DEBUG:boto:auth_path=/ffzdqu-47/obj3 2023-09-22T21:21:21.302 DEBUG:boto:path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.302 DEBUG:boto:auth_path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.302 DEBUG:boto:Method: GET 2023-09-22T21:21:21.302 DEBUG:boto:Path: /ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.302 DEBUG:boto:Data: 2023-09-22T21:21:21.303 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.303 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.303 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.303 DEBUG:boto:Params: {} 2023-09-22T21:21:21.303 DEBUG:boto:Token: None 2023-09-22T21:21:21.303 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.303 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PUqaozVVsj3sjqFRUvRUkFsb8tY= 2023-09-22T21:21:21.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PUqaozVVsj3sjqFRUvRUkFsb8tY=', 'Content-Length': '0'} 2023-09-22T21:21:21.306 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003457c44c0ad25af4-00650e0551-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.306 DEBUG:boto:path=/ffzdqu-47/obj3 2023-09-22T21:21:21.307 DEBUG:boto:auth_path=/ffzdqu-47/obj3 2023-09-22T21:21:21.307 DEBUG:boto:path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.307 DEBUG:boto:auth_path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.307 DEBUG:boto:Method: GET 2023-09-22T21:21:21.307 DEBUG:boto:Path: /ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.307 DEBUG:boto:Data: 2023-09-22T21:21:21.307 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.307 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.307 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.307 DEBUG:boto:Params: {} 2023-09-22T21:21:21.307 DEBUG:boto:Token: None 2023-09-22T21:21:21.308 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj3?versionId=null 2023-09-22T21:21:21.308 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PUqaozVVsj3sjqFRUvRUkFsb8tY= 2023-09-22T21:21:21.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PUqaozVVsj3sjqFRUvRUkFsb8tY=', 'Content-Length': '0'} 2023-09-22T21:21:21.310 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000015f1c3690ee4e4f-00650e0551-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.311 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:21.311 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:21.311 DEBUG:boto:path=/ffzdqu-47/?prefix=obj3 2023-09-22T21:21:21.311 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj3 2023-09-22T21:21:21.311 DEBUG:boto:Method: GET 2023-09-22T21:21:21.311 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj3 2023-09-22T21:21:21.311 DEBUG:boto:Data: 2023-09-22T21:21:21.311 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.311 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.311 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.312 DEBUG:boto:Params: {} 2023-09-22T21:21:21.312 DEBUG:boto:Token: None 2023-09-22T21:21:21.312 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/ 2023-09-22T21:21:21.312 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI= 2023-09-22T21:21:21.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI=', 'Content-Length': '0'} 2023-09-22T21:21:21.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000761611896d5c2f9d-00650e0551-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.358 DEBUG:boto:b'ffzdqu-47obj31000falseobj32023-09-22T21:20:38.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:21.359 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:21.359 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:21.359 DEBUG:boto:path=/ffzdqu-47/?prefix=obj3 2023-09-22T21:21:21.359 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj3 2023-09-22T21:21:21.359 DEBUG:boto:Method: GET 2023-09-22T21:21:21.359 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj3 2023-09-22T21:21:21.360 DEBUG:boto:Data: 2023-09-22T21:21:21.360 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.360 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.360 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.360 DEBUG:boto:Params: {} 2023-09-22T21:21:21.360 DEBUG:boto:Token: None 2023-09-22T21:21:21.360 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/ 2023-09-22T21:21:21.360 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI= 2023-09-22T21:21:21.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI=', 'Content-Length': '0'} 2023-09-22T21:21:21.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e791ebbc1bd31802-00650e0551-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.365 DEBUG:boto:b'ffzdqu-47obj31000falseobj32023-09-22T21:20:38.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:21.366 DEBUG:boto:path=/ffzdqu-47/obj3 2023-09-22T21:21:21.366 DEBUG:boto:auth_path=/ffzdqu-47/obj3 2023-09-22T21:21:21.366 DEBUG:boto:Method: HEAD 2023-09-22T21:21:21.366 DEBUG:boto:Path: /ffzdqu-47/obj3 2023-09-22T21:21:21.366 DEBUG:boto:Data: 2023-09-22T21:21:21.366 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.366 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.366 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.366 DEBUG:boto:Params: {} 2023-09-22T21:21:21.367 DEBUG:boto:Token: None 2023-09-22T21:21:21.367 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj3 2023-09-22T21:21:21.367 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:43oL9yOKDvfrBfJHcKRUkk+IN1M= 2023-09-22T21:21:21.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:43oL9yOKDvfrBfJHcKRUkk+IN1M=', 'Content-Length': '0'} 2023-09-22T21:21:21.369 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f99c9b004a99e1d1-00650e0551-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.370 DEBUG:boto:path=/ffzdqu-47/obj3 2023-09-22T21:21:21.370 DEBUG:boto:auth_path=/ffzdqu-47/obj3 2023-09-22T21:21:21.370 DEBUG:boto:Method: HEAD 2023-09-22T21:21:21.370 DEBUG:boto:Path: /ffzdqu-47/obj3 2023-09-22T21:21:21.370 DEBUG:boto:Data: 2023-09-22T21:21:21.370 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.370 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.370 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.370 DEBUG:boto:Params: {} 2023-09-22T21:21:21.371 DEBUG:boto:Token: None 2023-09-22T21:21:21.371 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj3 2023-09-22T21:21:21.371 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:43oL9yOKDvfrBfJHcKRUkk+IN1M= 2023-09-22T21:21:21.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:43oL9yOKDvfrBfJHcKRUkk+IN1M=', 'Content-Length': '0'} 2023-09-22T21:21:21.373 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7108419de634d13-00650e0551-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:21:21.374 DEBUG:boto:path=/ffzdqu-47/obj3 2023-09-22T21:21:21.374 DEBUG:boto:auth_path=/ffzdqu-47/obj3 2023-09-22T21:21:21.374 DEBUG:boto:Method: GET 2023-09-22T21:21:21.374 DEBUG:boto:Path: /ffzdqu-47/obj3 2023-09-22T21:21:21.374 DEBUG:boto:Data: 2023-09-22T21:21:21.374 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.374 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.375 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.375 DEBUG:boto:Params: {} 2023-09-22T21:21:21.375 DEBUG:boto:Token: None 2023-09-22T21:21:21.375 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj3 2023-09-22T21:21:21.375 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZUCcwPBHQN4pMqRz8LU7KrFtMb4= 2023-09-22T21:21:21.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZUCcwPBHQN4pMqRz8LU7KrFtMb4=', 'Content-Length': '0'} 2023-09-22T21:21:21.377 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093b4ecd4c77fd79f-00650e0551-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.378 DEBUG:boto:path=/ffzdqu-47/obj3 2023-09-22T21:21:21.378 DEBUG:boto:auth_path=/ffzdqu-47/obj3 2023-09-22T21:21:21.378 DEBUG:boto:Method: GET 2023-09-22T21:21:21.378 DEBUG:boto:Path: /ffzdqu-47/obj3 2023-09-22T21:21:21.378 DEBUG:boto:Data: 2023-09-22T21:21:21.378 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.378 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.379 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.379 DEBUG:boto:Params: {} 2023-09-22T21:21:21.379 DEBUG:boto:Token: None 2023-09-22T21:21:21.379 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj3 2023-09-22T21:21:21.379 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZUCcwPBHQN4pMqRz8LU7KrFtMb4= 2023-09-22T21:21:21.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZUCcwPBHQN4pMqRz8LU7KrFtMb4=', 'Content-Length': '0'} 2023-09-22T21:21:21.382 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008a1fde263be957a-00650e0551-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.422 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:21.423 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:21.423 DEBUG:boto:path=/ffzdqu-47/?prefix=obj3 2023-09-22T21:21:21.423 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj3 2023-09-22T21:21:21.423 DEBUG:boto:Method: GET 2023-09-22T21:21:21.423 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj3 2023-09-22T21:21:21.423 DEBUG:boto:Data: 2023-09-22T21:21:21.423 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.423 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.423 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.423 DEBUG:boto:Params: {} 2023-09-22T21:21:21.424 DEBUG:boto:Token: None 2023-09-22T21:21:21.424 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/ 2023-09-22T21:21:21.424 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI= 2023-09-22T21:21:21.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI=', 'Content-Length': '0'} 2023-09-22T21:21:21.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007198afda7821616c-00650e0551-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.428 DEBUG:boto:b'ffzdqu-47obj31000falseobj32023-09-22T21:20:38.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:21.429 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:21.429 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:21.429 DEBUG:boto:path=/ffzdqu-47/?prefix=obj3 2023-09-22T21:21:21.429 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj3 2023-09-22T21:21:21.429 DEBUG:boto:Method: GET 2023-09-22T21:21:21.429 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj3 2023-09-22T21:21:21.430 DEBUG:boto:Data: 2023-09-22T21:21:21.430 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.430 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.430 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.430 DEBUG:boto:Params: {} 2023-09-22T21:21:21.430 DEBUG:boto:Token: None 2023-09-22T21:21:21.430 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/ 2023-09-22T21:21:21.430 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI= 2023-09-22T21:21:21.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI=', 'Content-Length': '0'} 2023-09-22T21:21:21.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdf3e1fcb527a0c2-00650e0551-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.435 DEBUG:boto:b'ffzdqu-47obj31000falseobj32023-09-22T21:20:38.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:21.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:21:21.436 DEBUG:boto:path=/ffzdqu-47/obj4 2023-09-22T21:21:21.436 DEBUG:boto:auth_path=/ffzdqu-47/obj4 2023-09-22T21:21:21.436 DEBUG:boto:path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.436 DEBUG:boto:auth_path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.436 DEBUG:boto:Method: GET 2023-09-22T21:21:21.436 DEBUG:boto:Path: /ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.436 DEBUG:boto:Data: 2023-09-22T21:21:21.437 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.437 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.437 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.437 DEBUG:boto:Params: {} 2023-09-22T21:21:21.437 DEBUG:boto:Token: None 2023-09-22T21:21:21.437 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.437 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DZEx4HtTT00LV11h/YB3MWX32Js= 2023-09-22T21:21:21.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DZEx4HtTT00LV11h/YB3MWX32Js=', 'Content-Length': '0'} 2023-09-22T21:21:21.440 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000492358dcbab1d6da-00650e0551-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.440 DEBUG:boto:path=/ffzdqu-47/obj4 2023-09-22T21:21:21.441 DEBUG:boto:auth_path=/ffzdqu-47/obj4 2023-09-22T21:21:21.441 DEBUG:boto:path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.441 DEBUG:boto:auth_path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.441 DEBUG:boto:Method: GET 2023-09-22T21:21:21.441 DEBUG:boto:Path: /ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.441 DEBUG:boto:Data: 2023-09-22T21:21:21.441 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.441 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.441 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.441 DEBUG:boto:Params: {} 2023-09-22T21:21:21.442 DEBUG:boto:Token: None 2023-09-22T21:21:21.442 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.442 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DZEx4HtTT00LV11h/YB3MWX32Js= 2023-09-22T21:21:21.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DZEx4HtTT00LV11h/YB3MWX32Js=', 'Content-Length': '0'} 2023-09-22T21:21:21.445 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d68899d96a3a31ea-00650e0551-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.445 DEBUG:boto:path=/ffzdqu-47/obj4 2023-09-22T21:21:21.445 DEBUG:boto:auth_path=/ffzdqu-47/obj4 2023-09-22T21:21:21.445 DEBUG:boto:path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.445 DEBUG:boto:auth_path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.445 DEBUG:boto:Method: HEAD 2023-09-22T21:21:21.446 DEBUG:boto:Path: /ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.446 DEBUG:boto:Data: 2023-09-22T21:21:21.446 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.446 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.446 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.446 DEBUG:boto:Params: {} 2023-09-22T21:21:21.446 DEBUG:boto:Token: None 2023-09-22T21:21:21.446 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.447 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ct2yvkOGefI6ceiezr9R1wML338= 2023-09-22T21:21:21.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ct2yvkOGefI6ceiezr9R1wML338=', 'Content-Length': '0'} 2023-09-22T21:21:21.449 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006de7bd77daf2e5aa-00650e0551-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.449 DEBUG:boto:path=/ffzdqu-47/obj4 2023-09-22T21:21:21.449 DEBUG:boto:auth_path=/ffzdqu-47/obj4 2023-09-22T21:21:21.450 DEBUG:boto:path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.450 DEBUG:boto:auth_path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.450 DEBUG:boto:Method: HEAD 2023-09-22T21:21:21.450 DEBUG:boto:Path: /ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.450 DEBUG:boto:Data: 2023-09-22T21:21:21.450 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.450 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.450 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.450 DEBUG:boto:Params: {} 2023-09-22T21:21:21.451 DEBUG:boto:Token: None 2023-09-22T21:21:21.451 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.451 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ct2yvkOGefI6ceiezr9R1wML338= 2023-09-22T21:21:21.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ct2yvkOGefI6ceiezr9R1wML338=', 'Content-Length': '0'} 2023-09-22T21:21:21.454 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f0510eeaf21a3f7-00650e0551-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:21:21.454 DEBUG:boto:path=/ffzdqu-47/obj4 2023-09-22T21:21:21.454 DEBUG:boto:auth_path=/ffzdqu-47/obj4 2023-09-22T21:21:21.454 DEBUG:boto:path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.454 DEBUG:boto:auth_path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.455 DEBUG:boto:Method: GET 2023-09-22T21:21:21.455 DEBUG:boto:Path: /ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.455 DEBUG:boto:Data: 2023-09-22T21:21:21.455 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.455 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.455 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.455 DEBUG:boto:Params: {} 2023-09-22T21:21:21.455 DEBUG:boto:Token: None 2023-09-22T21:21:21.456 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.456 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DZEx4HtTT00LV11h/YB3MWX32Js= 2023-09-22T21:21:21.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DZEx4HtTT00LV11h/YB3MWX32Js=', 'Content-Length': '0'} 2023-09-22T21:21:21.459 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec792e53ddc024fa-00650e0551-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.502 DEBUG:boto:path=/ffzdqu-47/obj4 2023-09-22T21:21:21.503 DEBUG:boto:auth_path=/ffzdqu-47/obj4 2023-09-22T21:21:21.503 DEBUG:boto:path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.503 DEBUG:boto:auth_path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.503 DEBUG:boto:Method: GET 2023-09-22T21:21:21.503 DEBUG:boto:Path: /ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.503 DEBUG:boto:Data: 2023-09-22T21:21:21.503 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.503 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.504 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.504 DEBUG:boto:Params: {} 2023-09-22T21:21:21.504 DEBUG:boto:Token: None 2023-09-22T21:21:21.504 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj4?versionId=null 2023-09-22T21:21:21.504 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DZEx4HtTT00LV11h/YB3MWX32Js= 2023-09-22T21:21:21.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DZEx4HtTT00LV11h/YB3MWX32Js=', 'Content-Length': '0'} 2023-09-22T21:21:21.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092a763a40c621b60-00650e0551-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.507 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:21.508 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:21.508 DEBUG:boto:path=/ffzdqu-47/?prefix=obj4 2023-09-22T21:21:21.508 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj4 2023-09-22T21:21:21.508 DEBUG:boto:Method: GET 2023-09-22T21:21:21.508 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj4 2023-09-22T21:21:21.508 DEBUG:boto:Data: 2023-09-22T21:21:21.508 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.508 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.508 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.509 DEBUG:boto:Params: {} 2023-09-22T21:21:21.509 DEBUG:boto:Token: None 2023-09-22T21:21:21.509 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/ 2023-09-22T21:21:21.509 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI= 2023-09-22T21:21:21.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI=', 'Content-Length': '0'} 2023-09-22T21:21:21.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042699d26bdf36b58-00650e0551-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.514 DEBUG:boto:b'ffzdqu-47obj41000falseobj42023-09-22T21:20:38.999Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:21.515 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:21.515 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:21.515 DEBUG:boto:path=/ffzdqu-47/?prefix=obj4 2023-09-22T21:21:21.515 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj4 2023-09-22T21:21:21.515 DEBUG:boto:Method: GET 2023-09-22T21:21:21.515 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj4 2023-09-22T21:21:21.516 DEBUG:boto:Data: 2023-09-22T21:21:21.516 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.516 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.516 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.516 DEBUG:boto:Params: {} 2023-09-22T21:21:21.516 DEBUG:boto:Token: None 2023-09-22T21:21:21.516 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/ 2023-09-22T21:21:21.516 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI= 2023-09-22T21:21:21.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI=', 'Content-Length': '0'} 2023-09-22T21:21:21.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000165b0c038d779c17-00650e0551-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.562 DEBUG:boto:b'ffzdqu-47obj41000falseobj42023-09-22T21:20:38.999Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:21.563 DEBUG:boto:path=/ffzdqu-47/obj4 2023-09-22T21:21:21.563 DEBUG:boto:auth_path=/ffzdqu-47/obj4 2023-09-22T21:21:21.563 DEBUG:boto:Method: HEAD 2023-09-22T21:21:21.563 DEBUG:boto:Path: /ffzdqu-47/obj4 2023-09-22T21:21:21.564 DEBUG:boto:Data: 2023-09-22T21:21:21.564 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.564 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.564 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.564 DEBUG:boto:Params: {} 2023-09-22T21:21:21.564 DEBUG:boto:Token: None 2023-09-22T21:21:21.564 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj4 2023-09-22T21:21:21.565 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cHqr93FHTv5h0FSMkx9Fy4qOj+Y= 2023-09-22T21:21:21.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cHqr93FHTv5h0FSMkx9Fy4qOj+Y=', 'Content-Length': '0'} 2023-09-22T21:21:21.567 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f42bd36a81de68e0-00650e0551-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.568 DEBUG:boto:path=/ffzdqu-47/obj4 2023-09-22T21:21:21.568 DEBUG:boto:auth_path=/ffzdqu-47/obj4 2023-09-22T21:21:21.568 DEBUG:boto:Method: HEAD 2023-09-22T21:21:21.568 DEBUG:boto:Path: /ffzdqu-47/obj4 2023-09-22T21:21:21.568 DEBUG:boto:Data: 2023-09-22T21:21:21.568 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.568 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.568 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.569 DEBUG:boto:Params: {} 2023-09-22T21:21:21.569 DEBUG:boto:Token: None 2023-09-22T21:21:21.569 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj4 2023-09-22T21:21:21.569 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cHqr93FHTv5h0FSMkx9Fy4qOj+Y= 2023-09-22T21:21:21.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cHqr93FHTv5h0FSMkx9Fy4qOj+Y=', 'Content-Length': '0'} 2023-09-22T21:21:21.573 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aca0cb8faea8043d-00650e0551-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:21:21.573 DEBUG:boto:path=/ffzdqu-47/obj4 2023-09-22T21:21:21.574 DEBUG:boto:auth_path=/ffzdqu-47/obj4 2023-09-22T21:21:21.574 DEBUG:boto:Method: GET 2023-09-22T21:21:21.574 DEBUG:boto:Path: /ffzdqu-47/obj4 2023-09-22T21:21:21.574 DEBUG:boto:Data: 2023-09-22T21:21:21.574 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.574 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.574 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.574 DEBUG:boto:Params: {} 2023-09-22T21:21:21.575 DEBUG:boto:Token: None 2023-09-22T21:21:21.575 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj4 2023-09-22T21:21:21.575 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GGQss4sJ7lDLj3CjFiIngFnDOm4= 2023-09-22T21:21:21.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GGQss4sJ7lDLj3CjFiIngFnDOm4=', 'Content-Length': '0'} 2023-09-22T21:21:21.578 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1773229cb22c15d-00650e0551-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.578 DEBUG:boto:path=/ffzdqu-47/obj4 2023-09-22T21:21:21.578 DEBUG:boto:auth_path=/ffzdqu-47/obj4 2023-09-22T21:21:21.578 DEBUG:boto:Method: GET 2023-09-22T21:21:21.578 DEBUG:boto:Path: /ffzdqu-47/obj4 2023-09-22T21:21:21.578 DEBUG:boto:Data: 2023-09-22T21:21:21.578 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.579 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.579 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.579 DEBUG:boto:Params: {} 2023-09-22T21:21:21.579 DEBUG:boto:Token: None 2023-09-22T21:21:21.579 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/obj4 2023-09-22T21:21:21.579 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GGQss4sJ7lDLj3CjFiIngFnDOm4= 2023-09-22T21:21:21.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GGQss4sJ7lDLj3CjFiIngFnDOm4=', 'Content-Length': '0'} 2023-09-22T21:21:21.582 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001585f05dc2b7ece6-00650e0551-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.583 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:21.583 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:21.583 DEBUG:boto:path=/ffzdqu-47/?prefix=obj4 2023-09-22T21:21:21.583 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj4 2023-09-22T21:21:21.583 DEBUG:boto:Method: GET 2023-09-22T21:21:21.583 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj4 2023-09-22T21:21:21.583 DEBUG:boto:Data: 2023-09-22T21:21:21.583 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.583 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:21.584 DEBUG:boto:Port: 8001 2023-09-22T21:21:21.584 DEBUG:boto:Params: {} 2023-09-22T21:21:21.584 DEBUG:boto:Token: None 2023-09-22T21:21:21.584 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/ 2023-09-22T21:21:21.584 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI= 2023-09-22T21:21:21.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI=', 'Content-Length': '0'} 2023-09-22T21:21:21.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca6d2ba160e5a14f-00650e0551-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.589 DEBUG:boto:b'ffzdqu-47obj41000falseobj42023-09-22T21:20:38.999Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:21.589 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:21.589 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:21.590 DEBUG:boto:path=/ffzdqu-47/?prefix=obj4 2023-09-22T21:21:21.590 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj4 2023-09-22T21:21:21.590 DEBUG:boto:Method: GET 2023-09-22T21:21:21.590 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj4 2023-09-22T21:21:21.590 DEBUG:boto:Data: 2023-09-22T21:21:21.590 DEBUG:boto:Headers: {} 2023-09-22T21:21:21.590 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:21.590 DEBUG:boto:Port: 8000 2023-09-22T21:21:21.590 DEBUG:boto:Params: {} 2023-09-22T21:21:21.591 DEBUG:boto:Token: None 2023-09-22T21:21:21.591 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:21 GMT /ffzdqu-47/ 2023-09-22T21:21:21.591 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI= 2023-09-22T21:21:21.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3HNJUxvvgvugn2WWZE20tOKpNLI=', 'Content-Length': '0'} 2023-09-22T21:21:21.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000799016c0df4d1bb9-00650e0551-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:21.642 DEBUG:boto:b'ffzdqu-47obj41000falseobj42023-09-22T21:20:38.999Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:21.643 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-47 zones={a2, a1} 2023-09-22T21:21:21.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'ffzdqu-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:21:21.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'ffzdqu-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:21:21.644 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket ffzdqu-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:21:21.687 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:21:21.688 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:21:21.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'ffzdqu-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:21:21.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'ffzdqu-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:21:21.778 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket ffzdqu-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:21:21.857 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:21:21.857 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:21:21.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'ffzdqu-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:21:21.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'ffzdqu-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:21:21.943 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket ffzdqu-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:21:22.022 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:21:22.023 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:21:22.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'ffzdqu-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:21:22.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'ffzdqu-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:21:22.106 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket ffzdqu-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:21:22.186 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:21:22.186 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:21:22.269 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:22.269 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:22.269 DEBUG:boto:Method: HEAD 2023-09-22T21:21:22.269 DEBUG:boto:Path: /ffzdqu-46/ 2023-09-22T21:21:22.269 DEBUG:boto:Data: 2023-09-22T21:21:22.269 DEBUG:boto:Headers: {} 2023-09-22T21:21:22.270 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:22.270 DEBUG:boto:Port: 8000 2023-09-22T21:21:22.270 DEBUG:boto:Params: {} 2023-09-22T21:21:22.270 DEBUG:boto:Token: None 2023-09-22T21:21:22.270 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:22 GMT /ffzdqu-46/ 2023-09-22T21:21:22.270 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qvDa6bJrRnNSbU19SwSj6NTXpzU= 2023-09-22T21:21:22.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qvDa6bJrRnNSbU19SwSj6NTXpzU=', 'Content-Length': '0'} 2023-09-22T21:21:22.272 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b513e22b8489f42-00650e0552-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:22.273 DEBUG:boto:path=/ffzdqu-46/obj5 2023-09-22T21:21:22.273 DEBUG:boto:auth_path=/ffzdqu-46/obj5 2023-09-22T21:21:22.273 DEBUG:boto:Method: PUT 2023-09-22T21:21:22.273 DEBUG:boto:Path: /ffzdqu-46/obj5 2023-09-22T21:21:22.273 DEBUG:boto:Data: 2023-09-22T21:21:22.273 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:21:22.274 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:22.274 DEBUG:boto:Port: 8000 2023-09-22T21:21:22.274 DEBUG:boto:Params: {} 2023-09-22T21:21:22.274 DEBUG:boto:Token: None 2023-09-22T21:21:22.274 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:21:22 GMT /ffzdqu-46/obj5 2023-09-22T21:21:22.274 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GqOPXU2Um1CKrh60/aTUK8x1R8E= 2023-09-22T21:21:22.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:21:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GqOPXU2Um1CKrh60/aTUK8x1R8E='} 2023-09-22T21:21:22.280 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d265f13e7bee1fde-00650e0552-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:22.281 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:22.281 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:22.281 DEBUG:boto:Method: HEAD 2023-09-22T21:21:22.281 DEBUG:boto:Path: /ffzdqu-46/ 2023-09-22T21:21:22.281 DEBUG:boto:Data: 2023-09-22T21:21:22.281 DEBUG:boto:Headers: {} 2023-09-22T21:21:22.281 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:22.281 DEBUG:boto:Port: 8000 2023-09-22T21:21:22.281 DEBUG:boto:Params: {} 2023-09-22T21:21:22.282 DEBUG:boto:Token: None 2023-09-22T21:21:22.282 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:22 GMT /ffzdqu-46/ 2023-09-22T21:21:22.282 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qvDa6bJrRnNSbU19SwSj6NTXpzU= 2023-09-22T21:21:22.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qvDa6bJrRnNSbU19SwSj6NTXpzU=', 'Content-Length': '0'} 2023-09-22T21:21:22.284 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c502cf1f62e9bb5-00650e0552-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:22.284 DEBUG:boto:path=/ffzdqu-46/obj6 2023-09-22T21:21:22.284 DEBUG:boto:auth_path=/ffzdqu-46/obj6 2023-09-22T21:21:22.284 DEBUG:boto:Method: PUT 2023-09-22T21:21:22.284 DEBUG:boto:Path: /ffzdqu-46/obj6 2023-09-22T21:21:22.285 DEBUG:boto:Data: 2023-09-22T21:21:22.285 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:21:22.285 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:22.285 DEBUG:boto:Port: 8000 2023-09-22T21:21:22.285 DEBUG:boto:Params: {} 2023-09-22T21:21:22.285 DEBUG:boto:Token: None 2023-09-22T21:21:22.285 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:21:22 GMT /ffzdqu-46/obj6 2023-09-22T21:21:22.285 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LWP2ENv6Zt55HxU15CW8NnPsy84= 2023-09-22T21:21:22.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:21:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LWP2ENv6Zt55HxU15CW8NnPsy84='} 2023-09-22T21:21:22.291 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087677d3a990178d7-00650e0552-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:22.291 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:22.291 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:22.291 DEBUG:boto:Method: HEAD 2023-09-22T21:21:22.291 DEBUG:boto:Path: /ffzdqu-46/ 2023-09-22T21:21:22.291 DEBUG:boto:Data: 2023-09-22T21:21:22.292 DEBUG:boto:Headers: {} 2023-09-22T21:21:22.292 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:22.292 DEBUG:boto:Port: 8000 2023-09-22T21:21:22.292 DEBUG:boto:Params: {} 2023-09-22T21:21:22.292 DEBUG:boto:Token: None 2023-09-22T21:21:22.292 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:22 GMT /ffzdqu-46/ 2023-09-22T21:21:22.292 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qvDa6bJrRnNSbU19SwSj6NTXpzU= 2023-09-22T21:21:22.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qvDa6bJrRnNSbU19SwSj6NTXpzU=', 'Content-Length': '0'} 2023-09-22T21:21:22.295 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e4f3dc54f865984-00650e0552-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:22.295 DEBUG:boto:path=/ffzdqu-46/obj7 2023-09-22T21:21:22.295 DEBUG:boto:auth_path=/ffzdqu-46/obj7 2023-09-22T21:21:22.295 DEBUG:boto:Method: PUT 2023-09-22T21:21:22.295 DEBUG:boto:Path: /ffzdqu-46/obj7 2023-09-22T21:21:22.295 DEBUG:boto:Data: 2023-09-22T21:21:22.295 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:21:22.296 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:22.296 DEBUG:boto:Port: 8000 2023-09-22T21:21:22.296 DEBUG:boto:Params: {} 2023-09-22T21:21:22.296 DEBUG:boto:Token: None 2023-09-22T21:21:22.296 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:21:22 GMT /ffzdqu-46/obj7 2023-09-22T21:21:22.296 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vSt50K129xHr2BZoXj3432/OpGU= 2023-09-22T21:21:22.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:21:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vSt50K129xHr2BZoXj3432/OpGU='} 2023-09-22T21:21:22.342 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009880034d16b3c339-00650e0552-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:22.343 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:22.343 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:22.343 DEBUG:boto:Method: HEAD 2023-09-22T21:21:22.343 DEBUG:boto:Path: /ffzdqu-46/ 2023-09-22T21:21:22.343 DEBUG:boto:Data: 2023-09-22T21:21:22.343 DEBUG:boto:Headers: {} 2023-09-22T21:21:22.343 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:22.343 DEBUG:boto:Port: 8000 2023-09-22T21:21:22.344 DEBUG:boto:Params: {} 2023-09-22T21:21:22.344 DEBUG:boto:Token: None 2023-09-22T21:21:22.344 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:22 GMT /ffzdqu-46/ 2023-09-22T21:21:22.344 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qvDa6bJrRnNSbU19SwSj6NTXpzU= 2023-09-22T21:21:22.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qvDa6bJrRnNSbU19SwSj6NTXpzU=', 'Content-Length': '0'} 2023-09-22T21:21:22.348 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008bc369c965a10ba6-00650e0552-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:22.348 DEBUG:boto:path=/ffzdqu-46/obj8 2023-09-22T21:21:22.348 DEBUG:boto:auth_path=/ffzdqu-46/obj8 2023-09-22T21:21:22.348 DEBUG:boto:Method: PUT 2023-09-22T21:21:22.348 DEBUG:boto:Path: /ffzdqu-46/obj8 2023-09-22T21:21:22.348 DEBUG:boto:Data: 2023-09-22T21:21:22.349 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:21:22.349 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:22.349 DEBUG:boto:Port: 8000 2023-09-22T21:21:22.349 DEBUG:boto:Params: {} 2023-09-22T21:21:22.349 DEBUG:boto:Token: None 2023-09-22T21:21:22.349 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:21:22 GMT /ffzdqu-46/obj8 2023-09-22T21:21:22.349 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:00/d3yxtPYJz37gb/AWsOo7p9S8= 2023-09-22T21:21:22.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:21:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:00/d3yxtPYJz37gb/AWsOo7p9S8='} 2023-09-22T21:21:22.359 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c3a570b83697105-00650e0552-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:22.359 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:22.359 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:22.359 DEBUG:boto:Method: HEAD 2023-09-22T21:21:22.359 DEBUG:boto:Path: /ffzdqu-47/ 2023-09-22T21:21:22.360 DEBUG:boto:Data: 2023-09-22T21:21:22.360 DEBUG:boto:Headers: {} 2023-09-22T21:21:22.360 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:22.360 DEBUG:boto:Port: 8001 2023-09-22T21:21:22.360 DEBUG:boto:Params: {} 2023-09-22T21:21:22.360 DEBUG:boto:Token: None 2023-09-22T21:21:22.360 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:22 GMT /ffzdqu-47/ 2023-09-22T21:21:22.360 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7rRmsY04f+alxKqtTLfYKa2d4C0= 2023-09-22T21:21:22.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7rRmsY04f+alxKqtTLfYKa2d4C0=', 'Content-Length': '0'} 2023-09-22T21:21:22.364 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004aaa6f20cc231413-00650e0552-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:22.364 DEBUG:boto:path=/ffzdqu-47/obj5 2023-09-22T21:21:22.364 DEBUG:boto:auth_path=/ffzdqu-47/obj5 2023-09-22T21:21:22.364 DEBUG:boto:Method: PUT 2023-09-22T21:21:22.365 DEBUG:boto:Path: /ffzdqu-47/obj5 2023-09-22T21:21:22.365 DEBUG:boto:Data: 2023-09-22T21:21:22.365 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:21:22.365 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:22.365 DEBUG:boto:Port: 8001 2023-09-22T21:21:22.365 DEBUG:boto:Params: {} 2023-09-22T21:21:22.365 DEBUG:boto:Token: None 2023-09-22T21:21:22.365 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:21:22 GMT /ffzdqu-47/obj5 2023-09-22T21:21:22.365 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FYrJ73vGIo5Ddjv/60qp8L/yYYk= 2023-09-22T21:21:22.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:21:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FYrJ73vGIo5Ddjv/60qp8L/yYYk='} 2023-09-22T21:21:22.410 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003c994ba3f60714c-00650e0552-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:22.411 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:22.411 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:22.411 DEBUG:boto:Method: HEAD 2023-09-22T21:21:22.411 DEBUG:boto:Path: /ffzdqu-47/ 2023-09-22T21:21:22.411 DEBUG:boto:Data: 2023-09-22T21:21:22.411 DEBUG:boto:Headers: {} 2023-09-22T21:21:22.412 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:22.412 DEBUG:boto:Port: 8001 2023-09-22T21:21:22.412 DEBUG:boto:Params: {} 2023-09-22T21:21:22.412 DEBUG:boto:Token: None 2023-09-22T21:21:22.412 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:22 GMT /ffzdqu-47/ 2023-09-22T21:21:22.412 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7rRmsY04f+alxKqtTLfYKa2d4C0= 2023-09-22T21:21:22.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7rRmsY04f+alxKqtTLfYKa2d4C0=', 'Content-Length': '0'} 2023-09-22T21:21:22.416 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007cf4128384086cec-00650e0552-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:22.416 DEBUG:boto:path=/ffzdqu-47/obj6 2023-09-22T21:21:22.416 DEBUG:boto:auth_path=/ffzdqu-47/obj6 2023-09-22T21:21:22.417 DEBUG:boto:Method: PUT 2023-09-22T21:21:22.417 DEBUG:boto:Path: /ffzdqu-47/obj6 2023-09-22T21:21:22.417 DEBUG:boto:Data: 2023-09-22T21:21:22.417 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:21:22.417 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:22.417 DEBUG:boto:Port: 8001 2023-09-22T21:21:22.417 DEBUG:boto:Params: {} 2023-09-22T21:21:22.417 DEBUG:boto:Token: None 2023-09-22T21:21:22.418 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:21:22 GMT /ffzdqu-47/obj6 2023-09-22T21:21:22.418 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ykSKQYfq7gRmFdghbaC4Nqvma3Q= 2023-09-22T21:21:22.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:21:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ykSKQYfq7gRmFdghbaC4Nqvma3Q='} 2023-09-22T21:21:22.430 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033f17462925ac48c-00650e0552-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:22.430 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:22.430 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:22.430 DEBUG:boto:Method: HEAD 2023-09-22T21:21:22.430 DEBUG:boto:Path: /ffzdqu-47/ 2023-09-22T21:21:22.430 DEBUG:boto:Data: 2023-09-22T21:21:22.431 DEBUG:boto:Headers: {} 2023-09-22T21:21:22.431 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:22.431 DEBUG:boto:Port: 8001 2023-09-22T21:21:22.431 DEBUG:boto:Params: {} 2023-09-22T21:21:22.431 DEBUG:boto:Token: None 2023-09-22T21:21:22.431 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:22 GMT /ffzdqu-47/ 2023-09-22T21:21:22.431 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7rRmsY04f+alxKqtTLfYKa2d4C0= 2023-09-22T21:21:22.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7rRmsY04f+alxKqtTLfYKa2d4C0=', 'Content-Length': '0'} 2023-09-22T21:21:22.435 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010c35ef65211e999-00650e0552-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:22.435 DEBUG:boto:path=/ffzdqu-47/obj7 2023-09-22T21:21:22.436 DEBUG:boto:auth_path=/ffzdqu-47/obj7 2023-09-22T21:21:22.436 DEBUG:boto:Method: PUT 2023-09-22T21:21:22.436 DEBUG:boto:Path: /ffzdqu-47/obj7 2023-09-22T21:21:22.436 DEBUG:boto:Data: 2023-09-22T21:21:22.436 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:21:22.436 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:22.436 DEBUG:boto:Port: 8001 2023-09-22T21:21:22.436 DEBUG:boto:Params: {} 2023-09-22T21:21:22.436 DEBUG:boto:Token: None 2023-09-22T21:21:22.437 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:21:22 GMT /ffzdqu-47/obj7 2023-09-22T21:21:22.437 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aMFErAMZ6Jt3at/GToJNVlSzoKw= 2023-09-22T21:21:22.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:21:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aMFErAMZ6Jt3at/GToJNVlSzoKw='} 2023-09-22T21:21:22.478 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000bdfc33eef97fb0a-00650e0552-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:22.479 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:21:22.479 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:21:22.479 DEBUG:boto:Method: HEAD 2023-09-22T21:21:22.479 DEBUG:boto:Path: /ffzdqu-47/ 2023-09-22T21:21:22.479 DEBUG:boto:Data: 2023-09-22T21:21:22.479 DEBUG:boto:Headers: {} 2023-09-22T21:21:22.479 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:22.479 DEBUG:boto:Port: 8001 2023-09-22T21:21:22.480 DEBUG:boto:Params: {} 2023-09-22T21:21:22.480 DEBUG:boto:Token: None 2023-09-22T21:21:22.480 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:22 GMT /ffzdqu-47/ 2023-09-22T21:21:22.480 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7rRmsY04f+alxKqtTLfYKa2d4C0= 2023-09-22T21:21:22.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7rRmsY04f+alxKqtTLfYKa2d4C0=', 'Content-Length': '0'} 2023-09-22T21:21:22.484 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066ddb53eafd71152-00650e0552-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:22.484 DEBUG:boto:path=/ffzdqu-47/obj8 2023-09-22T21:21:22.484 DEBUG:boto:auth_path=/ffzdqu-47/obj8 2023-09-22T21:21:22.485 DEBUG:boto:Method: PUT 2023-09-22T21:21:22.485 DEBUG:boto:Path: /ffzdqu-47/obj8 2023-09-22T21:21:22.485 DEBUG:boto:Data: 2023-09-22T21:21:22.485 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:21:22.485 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:22.485 DEBUG:boto:Port: 8001 2023-09-22T21:21:22.485 DEBUG:boto:Params: {} 2023-09-22T21:21:22.485 DEBUG:boto:Token: None 2023-09-22T21:21:22.485 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:21:22 GMT /ffzdqu-47/obj8 2023-09-22T21:21:22.486 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8UQC7GhQt8kSamwA/NcnXq5vx/g= 2023-09-22T21:21:22.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:21:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8UQC7GhQt8kSamwA/NcnXq5vx/g='} 2023-09-22T21:21:22.498 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8d5ba664c2d1fe9-00650e0552-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:22.498 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-46 2023-09-22T21:21:22.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:21:22.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:21:22.498 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-46 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:21:22.549 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:21:22.549 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:21:22.592 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:21:22.592+0000 7f2f4cc9aa40 1 waiting for incremental sync to catch up: 2023-09-22T21:21:22.593 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.24.6, , 00000000001.26.6, , 00000000001.22.6, , , 00000000001.22.6, , , ] 2023-09-22T21:21:22.593 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000004.30.6, 00000000002.22.1, 00000000004.36.6, 00000000002.11.1, 00000000004.26.6, 00000000002.22.1, 00000000002.19.1, 00000000004.26.6, 00000000002.25.1, 00000000002.32.1, 00000000002.25.1] 2023-09-22T21:21:27.595 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:21:27.595+0000 7f2f4cc9aa40 1 waiting for incremental sync to catch up: 2023-09-22T21:21:27.595 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.24.6, , 00000000001.26.6, , 00000000001.22.6, , , 00000000001.22.6, , , ] 2023-09-22T21:21:27.595 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000004.30.6, 00000000002.22.1, 00000000004.36.6, 00000000002.11.1, 00000000004.26.6, 00000000002.22.1, 00000000002.19.1, 00000000004.26.6, 00000000002.25.1, 00000000002.32.1, 00000000002.25.1] 2023-09-22T21:21:32.597 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:21:32.595+0000 7f2f4cc9aa40 1 waiting for incremental sync to catch up: 2023-09-22T21:21:32.598 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.24.6, , 00000000001.26.6, , 00000000001.22.6, , , 00000000001.22.6, , , ] 2023-09-22T21:21:32.598 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000004.30.6, 00000000002.22.1, 00000000004.36.6, 00000000002.11.1, 00000000004.26.6, 00000000002.22.1, 00000000002.19.1, 00000000004.26.6, 00000000002.25.1, 00000000002.32.1, 00000000002.25.1] 2023-09-22T21:21:37.599 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:21:37.599+0000 7f2f4cc9aa40 1 waiting for incremental sync to catch up: 2023-09-22T21:21:37.599 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.24.6, , 00000000001.26.6, , 00000000001.22.6, , , 00000000001.22.6, , , ] 2023-09-22T21:21:37.599 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000004.30.6, 00000000002.22.1, 00000000004.36.6, 00000000002.11.1, 00000000004.26.6, 00000000002.22.1, 00000000002.19.1, 00000000004.26.6, 00000000002.25.1, 00000000002.32.1, 00000000002.25.1] 2023-09-22T21:21:42.601 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:21:42.603+0000 7f2f4cc9aa40 1 bucket sync caught up with source: 2023-09-22T21:21:42.601 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000004.30.6, 00000000002.22.1, 00000000004.36.6, 00000000002.11.1, 00000000004.26.6, 00000000002.22.1, 00000000002.19.1, 00000000004.26.6, 00000000002.25.1, 00000000002.32.1, 00000000002.25.1] 2023-09-22T21:21:42.602 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000004.30.6, 00000000002.22.1, 00000000004.36.6, 00000000002.11.1, 00000000004.26.6, 00000000002.22.1, 00000000002.19.1, 00000000004.26.6, 00000000002.25.1, 00000000002.32.1, 00000000002.25.1] 2023-09-22T21:21:42.602 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:21:42.603+0000 7f2f4cc9aa40 0 bucket checkpoint complete 2023-09-22T21:21:42.633 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-46 2023-09-22T21:21:42.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:21:42.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:21:42.634 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-46 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:21:42.682 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:21:42.683 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:21:42.728 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:21:42.727+0000 7f4d660e0a40 1 bucket sync caught up with source: 2023-09-22T21:21:42.728 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.131.6, , 00000000002.217.6, , 00000000002.162.6, , , 00000000002.193.6, , , ] 2023-09-22T21:21:42.728 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.131.6, , 00000000002.217.6, , 00000000002.162.6, , , 00000000002.193.6, , , ] 2023-09-22T21:21:42.728 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:21:42.727+0000 7f4d660e0a40 0 bucket checkpoint complete 2023-09-22T21:21:42.744 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-46 zones={a2, a1} 2023-09-22T21:21:42.745 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:42.745 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:42.745 DEBUG:boto:Method: HEAD 2023-09-22T21:21:42.745 DEBUG:boto:Path: /ffzdqu-46/ 2023-09-22T21:21:42.745 DEBUG:boto:Data: 2023-09-22T21:21:42.745 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.745 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:42.746 DEBUG:boto:Port: 8001 2023-09-22T21:21:42.746 DEBUG:boto:Params: {} 2023-09-22T21:21:42.746 DEBUG:boto:Token: None 2023-09-22T21:21:42.746 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/ 2023-09-22T21:21:42.746 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HnpIjLbhGQfAwJNFDJqRop7ylpI= 2023-09-22T21:21:42.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HnpIjLbhGQfAwJNFDJqRop7ylpI=', 'Content-Length': '0'} 2023-09-22T21:21:42.748 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce336e9eec7cbaf5-00650e0566-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.749 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:42.749 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:42.749 DEBUG:boto:Method: HEAD 2023-09-22T21:21:42.749 DEBUG:boto:Path: /ffzdqu-46/ 2023-09-22T21:21:42.749 DEBUG:boto:Data: 2023-09-22T21:21:42.749 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.749 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:42.750 DEBUG:boto:Port: 8000 2023-09-22T21:21:42.750 DEBUG:boto:Params: {} 2023-09-22T21:21:42.750 DEBUG:boto:Token: None 2023-09-22T21:21:42.750 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/ 2023-09-22T21:21:42.750 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HnpIjLbhGQfAwJNFDJqRop7ylpI= 2023-09-22T21:21:42.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HnpIjLbhGQfAwJNFDJqRop7ylpI=', 'Content-Length': '0'} 2023-09-22T21:21:42.752 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b901e36ef1f0560b-00650e0566-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.753 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:21:42.753 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:42.753 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:42.753 DEBUG:boto:path=/ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:42.753 DEBUG:boto:auth_path=/ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:42.753 DEBUG:boto:Method: GET 2023-09-22T21:21:42.753 DEBUG:boto:Path: /ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:42.753 DEBUG:boto:Data: 2023-09-22T21:21:42.753 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.754 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:42.754 DEBUG:boto:Port: 8001 2023-09-22T21:21:42.754 DEBUG:boto:Params: {} 2023-09-22T21:21:42.754 DEBUG:boto:Token: None 2023-09-22T21:21:42.754 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/?versions 2023-09-22T21:21:42.754 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AcPpF1TPtH5oOGuKjsd0k3/vSvQ= 2023-09-22T21:21:42.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AcPpF1TPtH5oOGuKjsd0k3/vSvQ=', 'Content-Length': '0'} 2023-09-22T21:21:42.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d596fc0a00a1e7c-00650e0566-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.757 DEBUG:boto:b'ffzdqu-46999falseobj1nulltrue2023-09-22T21:20:38.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:20:38.787Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:20:38.842Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:20:38.862Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-22T21:21:22.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-22T21:21:22.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-22T21:21:22.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-22T21:21:22.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:42.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-22T21:21:42.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-22T21:21:42.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-22T21:21:42.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-22T21:21:42.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-09-22T21:21:42.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-09-22T21:21:42.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-09-22T21:21:42.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-09-22T21:21:42.758 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:21:42.758 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:42.759 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:42.759 DEBUG:boto:path=/ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:42.759 DEBUG:boto:auth_path=/ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:42.759 DEBUG:boto:Method: GET 2023-09-22T21:21:42.759 DEBUG:boto:Path: /ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:42.759 DEBUG:boto:Data: 2023-09-22T21:21:42.759 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.759 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:42.759 DEBUG:boto:Port: 8000 2023-09-22T21:21:42.759 DEBUG:boto:Params: {} 2023-09-22T21:21:42.760 DEBUG:boto:Token: None 2023-09-22T21:21:42.760 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/?versions 2023-09-22T21:21:42.760 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AcPpF1TPtH5oOGuKjsd0k3/vSvQ= 2023-09-22T21:21:42.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AcPpF1TPtH5oOGuKjsd0k3/vSvQ=', 'Content-Length': '0'} 2023-09-22T21:21:42.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5351f9f908c89aa-00650e0566-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.762 DEBUG:boto:b'ffzdqu-46999falseobj1nulltrue2023-09-22T21:20:38.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:20:38.787Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:20:38.842Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:20:38.862Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-22T21:21:22.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-22T21:21:22.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-22T21:21:22.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-22T21:21:22.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:42.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-22T21:21:42.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-22T21:21:42.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-22T21:21:42.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-22T21:21:42.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-09-22T21:21:42.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-09-22T21:21:42.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-09-22T21:21:42.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-09-22T21:21:42.764 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:42.764 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:42.764 DEBUG:boto:path=/ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:42.764 DEBUG:boto:auth_path=/ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:42.764 DEBUG:boto:Method: GET 2023-09-22T21:21:42.764 DEBUG:boto:Path: /ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:42.764 DEBUG:boto:Data: 2023-09-22T21:21:42.765 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.765 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:42.765 DEBUG:boto:Port: 8001 2023-09-22T21:21:42.765 DEBUG:boto:Params: {} 2023-09-22T21:21:42.765 DEBUG:boto:Token: None 2023-09-22T21:21:42.765 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/?versions 2023-09-22T21:21:42.765 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AcPpF1TPtH5oOGuKjsd0k3/vSvQ= 2023-09-22T21:21:42.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AcPpF1TPtH5oOGuKjsd0k3/vSvQ=', 'Content-Length': '0'} 2023-09-22T21:21:42.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9dbca6d122105ed-00650e0566-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.768 DEBUG:boto:b'ffzdqu-46999falseobj1nulltrue2023-09-22T21:20:38.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:20:38.787Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:20:38.842Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:20:38.862Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-22T21:21:22.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-22T21:21:22.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-22T21:21:22.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-22T21:21:22.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:42.768 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:42.768 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:42.768 DEBUG:boto:path=/ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:42.769 DEBUG:boto:auth_path=/ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:42.769 DEBUG:boto:Method: GET 2023-09-22T21:21:42.769 DEBUG:boto:Path: /ffzdqu-46/?versions&max-keys=999 2023-09-22T21:21:42.769 DEBUG:boto:Data: 2023-09-22T21:21:42.769 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.769 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:42.769 DEBUG:boto:Port: 8000 2023-09-22T21:21:42.769 DEBUG:boto:Params: {} 2023-09-22T21:21:42.769 DEBUG:boto:Token: None 2023-09-22T21:21:42.770 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/?versions 2023-09-22T21:21:42.770 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AcPpF1TPtH5oOGuKjsd0k3/vSvQ= 2023-09-22T21:21:42.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AcPpF1TPtH5oOGuKjsd0k3/vSvQ=', 'Content-Length': '0'} 2023-09-22T21:21:42.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090935a53b1d33679-00650e0566-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.772 DEBUG:boto:b'ffzdqu-46999falseobj1nulltrue2023-09-22T21:20:38.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:20:38.787Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:20:38.842Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:20:38.862Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-22T21:21:22.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-22T21:21:22.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-22T21:21:22.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-22T21:21:22.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:42.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:21:42.773 DEBUG:boto:path=/ffzdqu-46/obj1 2023-09-22T21:21:42.773 DEBUG:boto:auth_path=/ffzdqu-46/obj1 2023-09-22T21:21:42.773 DEBUG:boto:path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.774 DEBUG:boto:auth_path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.774 DEBUG:boto:Method: GET 2023-09-22T21:21:42.774 DEBUG:boto:Path: /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.774 DEBUG:boto:Data: 2023-09-22T21:21:42.774 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.774 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:42.774 DEBUG:boto:Port: 8001 2023-09-22T21:21:42.774 DEBUG:boto:Params: {} 2023-09-22T21:21:42.774 DEBUG:boto:Token: None 2023-09-22T21:21:42.774 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.775 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/kl+8DwspTAK+uP4dAEcytl6NO8= 2023-09-22T21:21:42.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/kl+8DwspTAK+uP4dAEcytl6NO8=', 'Content-Length': '0'} 2023-09-22T21:21:42.776 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006cf3b66034ca61e-00650e0566-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.818 DEBUG:boto:path=/ffzdqu-46/obj1 2023-09-22T21:21:42.818 DEBUG:boto:auth_path=/ffzdqu-46/obj1 2023-09-22T21:21:42.819 DEBUG:boto:path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.819 DEBUG:boto:auth_path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.819 DEBUG:boto:Method: GET 2023-09-22T21:21:42.819 DEBUG:boto:Path: /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.819 DEBUG:boto:Data: 2023-09-22T21:21:42.819 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.819 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:42.819 DEBUG:boto:Port: 8000 2023-09-22T21:21:42.819 DEBUG:boto:Params: {} 2023-09-22T21:21:42.820 DEBUG:boto:Token: None 2023-09-22T21:21:42.820 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.820 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/kl+8DwspTAK+uP4dAEcytl6NO8= 2023-09-22T21:21:42.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/kl+8DwspTAK+uP4dAEcytl6NO8=', 'Content-Length': '0'} 2023-09-22T21:21:42.822 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f412654df73fbdf-00650e0566-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.822 DEBUG:boto:path=/ffzdqu-46/obj1 2023-09-22T21:21:42.822 DEBUG:boto:auth_path=/ffzdqu-46/obj1 2023-09-22T21:21:42.822 DEBUG:boto:path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.823 DEBUG:boto:auth_path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.823 DEBUG:boto:Method: HEAD 2023-09-22T21:21:42.823 DEBUG:boto:Path: /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.823 DEBUG:boto:Data: 2023-09-22T21:21:42.823 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.823 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:42.823 DEBUG:boto:Port: 8001 2023-09-22T21:21:42.823 DEBUG:boto:Params: {} 2023-09-22T21:21:42.823 DEBUG:boto:Token: None 2023-09-22T21:21:42.824 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.824 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vS7UrrJ/zWVJRgNsTwOT2cYdHLU= 2023-09-22T21:21:42.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vS7UrrJ/zWVJRgNsTwOT2cYdHLU=', 'Content-Length': '0'} 2023-09-22T21:21:42.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029f0bef1d14d749f-00650e0566-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.826 DEBUG:boto:path=/ffzdqu-46/obj1 2023-09-22T21:21:42.826 DEBUG:boto:auth_path=/ffzdqu-46/obj1 2023-09-22T21:21:42.826 DEBUG:boto:path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.826 DEBUG:boto:auth_path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.826 DEBUG:boto:Method: HEAD 2023-09-22T21:21:42.826 DEBUG:boto:Path: /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.826 DEBUG:boto:Data: 2023-09-22T21:21:42.826 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.826 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:42.827 DEBUG:boto:Port: 8000 2023-09-22T21:21:42.827 DEBUG:boto:Params: {} 2023-09-22T21:21:42.827 DEBUG:boto:Token: None 2023-09-22T21:21:42.827 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.827 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vS7UrrJ/zWVJRgNsTwOT2cYdHLU= 2023-09-22T21:21:42.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vS7UrrJ/zWVJRgNsTwOT2cYdHLU=', 'Content-Length': '0'} 2023-09-22T21:21:42.829 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031f9c310c104cc79-00650e0566-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:21:42.829 DEBUG:boto:path=/ffzdqu-46/obj1 2023-09-22T21:21:42.829 DEBUG:boto:auth_path=/ffzdqu-46/obj1 2023-09-22T21:21:42.829 DEBUG:boto:path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.829 DEBUG:boto:auth_path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.830 DEBUG:boto:Method: GET 2023-09-22T21:21:42.830 DEBUG:boto:Path: /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.830 DEBUG:boto:Data: 2023-09-22T21:21:42.830 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.830 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:42.830 DEBUG:boto:Port: 8001 2023-09-22T21:21:42.830 DEBUG:boto:Params: {} 2023-09-22T21:21:42.830 DEBUG:boto:Token: None 2023-09-22T21:21:42.830 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.831 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/kl+8DwspTAK+uP4dAEcytl6NO8= 2023-09-22T21:21:42.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/kl+8DwspTAK+uP4dAEcytl6NO8=', 'Content-Length': '0'} 2023-09-22T21:21:42.833 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cd7b84d59aba032-00650e0566-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.833 DEBUG:boto:path=/ffzdqu-46/obj1 2023-09-22T21:21:42.833 DEBUG:boto:auth_path=/ffzdqu-46/obj1 2023-09-22T21:21:42.833 DEBUG:boto:path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.833 DEBUG:boto:auth_path=/ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.834 DEBUG:boto:Method: GET 2023-09-22T21:21:42.834 DEBUG:boto:Path: /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.834 DEBUG:boto:Data: 2023-09-22T21:21:42.834 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.834 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:42.834 DEBUG:boto:Port: 8000 2023-09-22T21:21:42.834 DEBUG:boto:Params: {} 2023-09-22T21:21:42.834 DEBUG:boto:Token: None 2023-09-22T21:21:42.834 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/obj1?versionId=null 2023-09-22T21:21:42.835 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/kl+8DwspTAK+uP4dAEcytl6NO8= 2023-09-22T21:21:42.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/kl+8DwspTAK+uP4dAEcytl6NO8=', 'Content-Length': '0'} 2023-09-22T21:21:42.836 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a43d066ab410a7e-00650e0566-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.837 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:42.837 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:42.837 DEBUG:boto:path=/ffzdqu-46/?prefix=obj1 2023-09-22T21:21:42.837 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj1 2023-09-22T21:21:42.837 DEBUG:boto:Method: GET 2023-09-22T21:21:42.837 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj1 2023-09-22T21:21:42.837 DEBUG:boto:Data: 2023-09-22T21:21:42.837 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.837 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:42.838 DEBUG:boto:Port: 8001 2023-09-22T21:21:42.838 DEBUG:boto:Params: {} 2023-09-22T21:21:42.838 DEBUG:boto:Token: None 2023-09-22T21:21:42.838 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/ 2023-09-22T21:21:42.838 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:o4J8Rydd1DCOW9b8vHk8NdRpscg= 2023-09-22T21:21:42.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:o4J8Rydd1DCOW9b8vHk8NdRpscg=', 'Content-Length': '0'} 2023-09-22T21:21:42.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000416f090a3bba049a-00650e0566-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.842 DEBUG:boto:b'ffzdqu-46obj11000falseobj12023-09-22T21:20:38.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:42.843 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:42.843 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:42.843 DEBUG:boto:path=/ffzdqu-46/?prefix=obj1 2023-09-22T21:21:42.843 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj1 2023-09-22T21:21:42.843 DEBUG:boto:Method: GET 2023-09-22T21:21:42.843 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj1 2023-09-22T21:21:42.843 DEBUG:boto:Data: 2023-09-22T21:21:42.843 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.844 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:42.844 DEBUG:boto:Port: 8000 2023-09-22T21:21:42.844 DEBUG:boto:Params: {} 2023-09-22T21:21:42.844 DEBUG:boto:Token: None 2023-09-22T21:21:42.844 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/ 2023-09-22T21:21:42.844 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:o4J8Rydd1DCOW9b8vHk8NdRpscg= 2023-09-22T21:21:42.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:o4J8Rydd1DCOW9b8vHk8NdRpscg=', 'Content-Length': '0'} 2023-09-22T21:21:42.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b07a56949dadc6a2-00650e0566-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.847 DEBUG:boto:b'ffzdqu-46obj11000falseobj12023-09-22T21:20:38.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:42.847 DEBUG:boto:path=/ffzdqu-46/obj1 2023-09-22T21:21:42.848 DEBUG:boto:auth_path=/ffzdqu-46/obj1 2023-09-22T21:21:42.848 DEBUG:boto:Method: HEAD 2023-09-22T21:21:42.848 DEBUG:boto:Path: /ffzdqu-46/obj1 2023-09-22T21:21:42.848 DEBUG:boto:Data: 2023-09-22T21:21:42.848 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.848 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:42.848 DEBUG:boto:Port: 8001 2023-09-22T21:21:42.848 DEBUG:boto:Params: {} 2023-09-22T21:21:42.848 DEBUG:boto:Token: None 2023-09-22T21:21:42.849 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/obj1 2023-09-22T21:21:42.849 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wwqbTk/JpOyx6UFVB5A1U66FAtI= 2023-09-22T21:21:42.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wwqbTk/JpOyx6UFVB5A1U66FAtI=', 'Content-Length': '0'} 2023-09-22T21:21:42.851 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000702cca28d8672628-00650e0566-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.851 DEBUG:boto:path=/ffzdqu-46/obj1 2023-09-22T21:21:42.851 DEBUG:boto:auth_path=/ffzdqu-46/obj1 2023-09-22T21:21:42.851 DEBUG:boto:Method: HEAD 2023-09-22T21:21:42.851 DEBUG:boto:Path: /ffzdqu-46/obj1 2023-09-22T21:21:42.851 DEBUG:boto:Data: 2023-09-22T21:21:42.852 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.852 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:42.852 DEBUG:boto:Port: 8000 2023-09-22T21:21:42.852 DEBUG:boto:Params: {} 2023-09-22T21:21:42.852 DEBUG:boto:Token: None 2023-09-22T21:21:42.852 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/obj1 2023-09-22T21:21:42.853 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wwqbTk/JpOyx6UFVB5A1U66FAtI= 2023-09-22T21:21:42.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wwqbTk/JpOyx6UFVB5A1U66FAtI=', 'Content-Length': '0'} 2023-09-22T21:21:42.855 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009821103064d9e71b-00650e0566-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:21:42.855 DEBUG:boto:path=/ffzdqu-46/obj1 2023-09-22T21:21:42.855 DEBUG:boto:auth_path=/ffzdqu-46/obj1 2023-09-22T21:21:42.855 DEBUG:boto:Method: GET 2023-09-22T21:21:42.856 DEBUG:boto:Path: /ffzdqu-46/obj1 2023-09-22T21:21:42.856 DEBUG:boto:Data: 2023-09-22T21:21:42.856 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.856 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:42.856 DEBUG:boto:Port: 8001 2023-09-22T21:21:42.856 DEBUG:boto:Params: {} 2023-09-22T21:21:42.856 DEBUG:boto:Token: None 2023-09-22T21:21:42.856 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/obj1 2023-09-22T21:21:42.856 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YGVlZN/6HSQXom/vE3N3hrBhsAg= 2023-09-22T21:21:42.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YGVlZN/6HSQXom/vE3N3hrBhsAg=', 'Content-Length': '0'} 2023-09-22T21:21:42.859 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000463e91ecd2c38282-00650e0566-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.903 DEBUG:boto:path=/ffzdqu-46/obj1 2023-09-22T21:21:42.903 DEBUG:boto:auth_path=/ffzdqu-46/obj1 2023-09-22T21:21:42.903 DEBUG:boto:Method: GET 2023-09-22T21:21:42.903 DEBUG:boto:Path: /ffzdqu-46/obj1 2023-09-22T21:21:42.903 DEBUG:boto:Data: 2023-09-22T21:21:42.903 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.903 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:42.904 DEBUG:boto:Port: 8000 2023-09-22T21:21:42.904 DEBUG:boto:Params: {} 2023-09-22T21:21:42.904 DEBUG:boto:Token: None 2023-09-22T21:21:42.904 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/obj1 2023-09-22T21:21:42.904 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YGVlZN/6HSQXom/vE3N3hrBhsAg= 2023-09-22T21:21:42.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YGVlZN/6HSQXom/vE3N3hrBhsAg=', 'Content-Length': '0'} 2023-09-22T21:21:42.907 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000024f098f1cbd345a-00650e0566-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.907 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:42.907 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:42.907 DEBUG:boto:path=/ffzdqu-46/?prefix=obj1 2023-09-22T21:21:42.907 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj1 2023-09-22T21:21:42.908 DEBUG:boto:Method: GET 2023-09-22T21:21:42.908 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj1 2023-09-22T21:21:42.908 DEBUG:boto:Data: 2023-09-22T21:21:42.908 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.908 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:42.908 DEBUG:boto:Port: 8001 2023-09-22T21:21:42.908 DEBUG:boto:Params: {} 2023-09-22T21:21:42.908 DEBUG:boto:Token: None 2023-09-22T21:21:42.908 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/ 2023-09-22T21:21:42.909 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:o4J8Rydd1DCOW9b8vHk8NdRpscg= 2023-09-22T21:21:42.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:o4J8Rydd1DCOW9b8vHk8NdRpscg=', 'Content-Length': '0'} 2023-09-22T21:21:42.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068e65021ad960206-00650e0566-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.913 DEBUG:boto:b'ffzdqu-46obj11000falseobj12023-09-22T21:20:38.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:42.914 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:42.914 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:42.914 DEBUG:boto:path=/ffzdqu-46/?prefix=obj1 2023-09-22T21:21:42.914 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj1 2023-09-22T21:21:42.914 DEBUG:boto:Method: GET 2023-09-22T21:21:42.914 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj1 2023-09-22T21:21:42.914 DEBUG:boto:Data: 2023-09-22T21:21:42.915 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.915 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:42.915 DEBUG:boto:Port: 8000 2023-09-22T21:21:42.915 DEBUG:boto:Params: {} 2023-09-22T21:21:42.915 DEBUG:boto:Token: None 2023-09-22T21:21:42.915 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/ 2023-09-22T21:21:42.915 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:o4J8Rydd1DCOW9b8vHk8NdRpscg= 2023-09-22T21:21:42.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:o4J8Rydd1DCOW9b8vHk8NdRpscg=', 'Content-Length': '0'} 2023-09-22T21:21:42.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfc8b1b842f82e74-00650e0566-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.920 DEBUG:boto:b'ffzdqu-46obj11000falseobj12023-09-22T21:20:38.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:42.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:21:42.921 DEBUG:boto:path=/ffzdqu-46/obj2 2023-09-22T21:21:42.921 DEBUG:boto:auth_path=/ffzdqu-46/obj2 2023-09-22T21:21:42.921 DEBUG:boto:path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:42.921 DEBUG:boto:auth_path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:42.921 DEBUG:boto:Method: GET 2023-09-22T21:21:42.921 DEBUG:boto:Path: /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:42.921 DEBUG:boto:Data: 2023-09-22T21:21:42.921 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.922 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:42.922 DEBUG:boto:Port: 8001 2023-09-22T21:21:42.922 DEBUG:boto:Params: {} 2023-09-22T21:21:42.922 DEBUG:boto:Token: None 2023-09-22T21:21:42.922 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:42.922 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+oToilnvrShv/aN1+0B7J7gUGSw= 2023-09-22T21:21:42.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+oToilnvrShv/aN1+0B7J7gUGSw=', 'Content-Length': '0'} 2023-09-22T21:21:42.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000931e9229aba3def1-00650e0566-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.925 DEBUG:boto:path=/ffzdqu-46/obj2 2023-09-22T21:21:42.925 DEBUG:boto:auth_path=/ffzdqu-46/obj2 2023-09-22T21:21:42.925 DEBUG:boto:path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:42.926 DEBUG:boto:auth_path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:42.926 DEBUG:boto:Method: GET 2023-09-22T21:21:42.926 DEBUG:boto:Path: /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:42.926 DEBUG:boto:Data: 2023-09-22T21:21:42.926 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.926 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:42.926 DEBUG:boto:Port: 8000 2023-09-22T21:21:42.926 DEBUG:boto:Params: {} 2023-09-22T21:21:42.926 DEBUG:boto:Token: None 2023-09-22T21:21:42.927 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:42.927 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+oToilnvrShv/aN1+0B7J7gUGSw= 2023-09-22T21:21:42.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+oToilnvrShv/aN1+0B7J7gUGSw=', 'Content-Length': '0'} 2023-09-22T21:21:42.930 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081ea0e4ed2fc9efd-00650e0566-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.970 DEBUG:boto:path=/ffzdqu-46/obj2 2023-09-22T21:21:42.970 DEBUG:boto:auth_path=/ffzdqu-46/obj2 2023-09-22T21:21:42.971 DEBUG:boto:path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:42.971 DEBUG:boto:auth_path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:42.971 DEBUG:boto:Method: HEAD 2023-09-22T21:21:42.971 DEBUG:boto:Path: /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:42.971 DEBUG:boto:Data: 2023-09-22T21:21:42.971 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.971 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:42.971 DEBUG:boto:Port: 8001 2023-09-22T21:21:42.971 DEBUG:boto:Params: {} 2023-09-22T21:21:42.972 DEBUG:boto:Token: None 2023-09-22T21:21:42.972 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:42.972 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g7EopVY8PFGGFJPsf+QliFQrMks= 2023-09-22T21:21:42.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g7EopVY8PFGGFJPsf+QliFQrMks=', 'Content-Length': '0'} 2023-09-22T21:21:42.975 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fa1c0f46a815375-00650e0566-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.975 DEBUG:boto:path=/ffzdqu-46/obj2 2023-09-22T21:21:42.975 DEBUG:boto:auth_path=/ffzdqu-46/obj2 2023-09-22T21:21:42.975 DEBUG:boto:path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:42.975 DEBUG:boto:auth_path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:42.975 DEBUG:boto:Method: HEAD 2023-09-22T21:21:42.975 DEBUG:boto:Path: /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:42.976 DEBUG:boto:Data: 2023-09-22T21:21:42.976 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.976 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:42.976 DEBUG:boto:Port: 8000 2023-09-22T21:21:42.976 DEBUG:boto:Params: {} 2023-09-22T21:21:42.976 DEBUG:boto:Token: None 2023-09-22T21:21:42.976 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:42.976 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g7EopVY8PFGGFJPsf+QliFQrMks= 2023-09-22T21:21:42.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g7EopVY8PFGGFJPsf+QliFQrMks=', 'Content-Length': '0'} 2023-09-22T21:21:42.979 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084399c08b381c720-00650e0566-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:42.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:21:42.980 DEBUG:boto:path=/ffzdqu-46/obj2 2023-09-22T21:21:42.980 DEBUG:boto:auth_path=/ffzdqu-46/obj2 2023-09-22T21:21:42.980 DEBUG:boto:path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:42.980 DEBUG:boto:auth_path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:42.980 DEBUG:boto:Method: GET 2023-09-22T21:21:42.980 DEBUG:boto:Path: /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:42.980 DEBUG:boto:Data: 2023-09-22T21:21:42.980 DEBUG:boto:Headers: {} 2023-09-22T21:21:42.981 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:42.981 DEBUG:boto:Port: 8001 2023-09-22T21:21:42.981 DEBUG:boto:Params: {} 2023-09-22T21:21:42.981 DEBUG:boto:Token: None 2023-09-22T21:21:42.981 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:42 GMT /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:42.981 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+oToilnvrShv/aN1+0B7J7gUGSw= 2023-09-22T21:21:42.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+oToilnvrShv/aN1+0B7J7gUGSw=', 'Content-Length': '0'} 2023-09-22T21:21:42.984 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000432adb00d875db22-00650e0566-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.030 DEBUG:boto:path=/ffzdqu-46/obj2 2023-09-22T21:21:43.030 DEBUG:boto:auth_path=/ffzdqu-46/obj2 2023-09-22T21:21:43.031 DEBUG:boto:path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:43.031 DEBUG:boto:auth_path=/ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:43.031 DEBUG:boto:Method: GET 2023-09-22T21:21:43.031 DEBUG:boto:Path: /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:43.031 DEBUG:boto:Data: 2023-09-22T21:21:43.031 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.031 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.031 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.031 DEBUG:boto:Params: {} 2023-09-22T21:21:43.031 DEBUG:boto:Token: None 2023-09-22T21:21:43.032 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj2?versionId=null 2023-09-22T21:21:43.032 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4eSo2djKvGGEzmqDUoPGwvRFiUc= 2023-09-22T21:21:43.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4eSo2djKvGGEzmqDUoPGwvRFiUc=', 'Content-Length': '0'} 2023-09-22T21:21:43.035 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f3ab9b414016d5c-00650e0567-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.035 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:43.035 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:43.035 DEBUG:boto:path=/ffzdqu-46/?prefix=obj2 2023-09-22T21:21:43.035 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj2 2023-09-22T21:21:43.035 DEBUG:boto:Method: GET 2023-09-22T21:21:43.036 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj2 2023-09-22T21:21:43.036 DEBUG:boto:Data: 2023-09-22T21:21:43.036 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.036 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.036 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.036 DEBUG:boto:Params: {} 2023-09-22T21:21:43.036 DEBUG:boto:Token: None 2023-09-22T21:21:43.036 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/ 2023-09-22T21:21:43.036 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U= 2023-09-22T21:21:43.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U=', 'Content-Length': '0'} 2023-09-22T21:21:43.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d82e83424ab739a-00650e0567-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.041 DEBUG:boto:b'ffzdqu-46obj21000falseobj22023-09-22T21:20:38.787Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:43.042 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:43.042 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:43.042 DEBUG:boto:path=/ffzdqu-46/?prefix=obj2 2023-09-22T21:21:43.042 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj2 2023-09-22T21:21:43.042 DEBUG:boto:Method: GET 2023-09-22T21:21:43.042 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj2 2023-09-22T21:21:43.042 DEBUG:boto:Data: 2023-09-22T21:21:43.042 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.042 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.043 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.043 DEBUG:boto:Params: {} 2023-09-22T21:21:43.043 DEBUG:boto:Token: None 2023-09-22T21:21:43.043 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/ 2023-09-22T21:21:43.043 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U= 2023-09-22T21:21:43.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U=', 'Content-Length': '0'} 2023-09-22T21:21:43.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f75dadc47106abf-00650e0567-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.048 DEBUG:boto:b'ffzdqu-46obj21000falseobj22023-09-22T21:20:38.787Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:43.048 DEBUG:boto:path=/ffzdqu-46/obj2 2023-09-22T21:21:43.049 DEBUG:boto:auth_path=/ffzdqu-46/obj2 2023-09-22T21:21:43.049 DEBUG:boto:Method: HEAD 2023-09-22T21:21:43.049 DEBUG:boto:Path: /ffzdqu-46/obj2 2023-09-22T21:21:43.049 DEBUG:boto:Data: 2023-09-22T21:21:43.049 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.049 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.049 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.049 DEBUG:boto:Params: {} 2023-09-22T21:21:43.049 DEBUG:boto:Token: None 2023-09-22T21:21:43.050 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj2 2023-09-22T21:21:43.050 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rycsUZorjOYiBW2U6J7qz/HeoB0= 2023-09-22T21:21:43.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rycsUZorjOYiBW2U6J7qz/HeoB0=', 'Content-Length': '0'} 2023-09-22T21:21:43.052 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e3e1c2fa6622d8d-00650e0567-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.053 DEBUG:boto:path=/ffzdqu-46/obj2 2023-09-22T21:21:43.053 DEBUG:boto:auth_path=/ffzdqu-46/obj2 2023-09-22T21:21:43.053 DEBUG:boto:Method: HEAD 2023-09-22T21:21:43.053 DEBUG:boto:Path: /ffzdqu-46/obj2 2023-09-22T21:21:43.053 DEBUG:boto:Data: 2023-09-22T21:21:43.053 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.053 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.053 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.054 DEBUG:boto:Params: {} 2023-09-22T21:21:43.054 DEBUG:boto:Token: None 2023-09-22T21:21:43.054 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj2 2023-09-22T21:21:43.054 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rycsUZorjOYiBW2U6J7qz/HeoB0= 2023-09-22T21:21:43.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rycsUZorjOYiBW2U6J7qz/HeoB0=', 'Content-Length': '0'} 2023-09-22T21:21:43.056 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006baf22943f53717a-00650e0567-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:21:43.057 DEBUG:boto:path=/ffzdqu-46/obj2 2023-09-22T21:21:43.057 DEBUG:boto:auth_path=/ffzdqu-46/obj2 2023-09-22T21:21:43.057 DEBUG:boto:Method: GET 2023-09-22T21:21:43.057 DEBUG:boto:Path: /ffzdqu-46/obj2 2023-09-22T21:21:43.057 DEBUG:boto:Data: 2023-09-22T21:21:43.057 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.057 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.058 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.058 DEBUG:boto:Params: {} 2023-09-22T21:21:43.058 DEBUG:boto:Token: None 2023-09-22T21:21:43.058 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj2 2023-09-22T21:21:43.058 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Yxp0zrgJ7AZQ3jVgU7vH9jtkntY= 2023-09-22T21:21:43.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Yxp0zrgJ7AZQ3jVgU7vH9jtkntY=', 'Content-Length': '0'} 2023-09-22T21:21:43.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008bbd9a85eae3979d-00650e0567-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.102 DEBUG:boto:path=/ffzdqu-46/obj2 2023-09-22T21:21:43.102 DEBUG:boto:auth_path=/ffzdqu-46/obj2 2023-09-22T21:21:43.103 DEBUG:boto:Method: GET 2023-09-22T21:21:43.103 DEBUG:boto:Path: /ffzdqu-46/obj2 2023-09-22T21:21:43.103 DEBUG:boto:Data: 2023-09-22T21:21:43.103 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.103 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.103 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.103 DEBUG:boto:Params: {} 2023-09-22T21:21:43.103 DEBUG:boto:Token: None 2023-09-22T21:21:43.104 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj2 2023-09-22T21:21:43.104 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Yxp0zrgJ7AZQ3jVgU7vH9jtkntY= 2023-09-22T21:21:43.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Yxp0zrgJ7AZQ3jVgU7vH9jtkntY=', 'Content-Length': '0'} 2023-09-22T21:21:43.107 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082ae683685c054d7-00650e0567-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.107 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:43.107 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:43.107 DEBUG:boto:path=/ffzdqu-46/?prefix=obj2 2023-09-22T21:21:43.107 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj2 2023-09-22T21:21:43.107 DEBUG:boto:Method: GET 2023-09-22T21:21:43.107 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj2 2023-09-22T21:21:43.108 DEBUG:boto:Data: 2023-09-22T21:21:43.108 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.108 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.108 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.108 DEBUG:boto:Params: {} 2023-09-22T21:21:43.108 DEBUG:boto:Token: None 2023-09-22T21:21:43.108 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/ 2023-09-22T21:21:43.108 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U= 2023-09-22T21:21:43.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U=', 'Content-Length': '0'} 2023-09-22T21:21:43.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051d443e9c61dba2c-00650e0567-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.154 DEBUG:boto:b'ffzdqu-46obj21000falseobj22023-09-22T21:20:38.787Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:43.155 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:43.155 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:43.155 DEBUG:boto:path=/ffzdqu-46/?prefix=obj2 2023-09-22T21:21:43.155 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj2 2023-09-22T21:21:43.155 DEBUG:boto:Method: GET 2023-09-22T21:21:43.155 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj2 2023-09-22T21:21:43.156 DEBUG:boto:Data: 2023-09-22T21:21:43.156 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.156 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.156 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.156 DEBUG:boto:Params: {} 2023-09-22T21:21:43.156 DEBUG:boto:Token: None 2023-09-22T21:21:43.156 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/ 2023-09-22T21:21:43.157 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U= 2023-09-22T21:21:43.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U=', 'Content-Length': '0'} 2023-09-22T21:21:43.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea0af143c0a30a36-00650e0567-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.161 DEBUG:boto:b'ffzdqu-46obj21000falseobj22023-09-22T21:20:38.787Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:43.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:21:43.162 DEBUG:boto:path=/ffzdqu-46/obj3 2023-09-22T21:21:43.162 DEBUG:boto:auth_path=/ffzdqu-46/obj3 2023-09-22T21:21:43.162 DEBUG:boto:path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.162 DEBUG:boto:auth_path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.162 DEBUG:boto:Method: GET 2023-09-22T21:21:43.162 DEBUG:boto:Path: /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.163 DEBUG:boto:Data: 2023-09-22T21:21:43.163 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.163 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.163 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.163 DEBUG:boto:Params: {} 2023-09-22T21:21:43.163 DEBUG:boto:Token: None 2023-09-22T21:21:43.164 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.164 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pfQ6Bwc8AmTgMbmdgkX36DBuy4Q= 2023-09-22T21:21:43.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pfQ6Bwc8AmTgMbmdgkX36DBuy4Q=', 'Content-Length': '0'} 2023-09-22T21:21:43.167 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1c679d673babf0b-00650e0567-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.168 DEBUG:boto:path=/ffzdqu-46/obj3 2023-09-22T21:21:43.168 DEBUG:boto:auth_path=/ffzdqu-46/obj3 2023-09-22T21:21:43.168 DEBUG:boto:path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.168 DEBUG:boto:auth_path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.168 DEBUG:boto:Method: GET 2023-09-22T21:21:43.168 DEBUG:boto:Path: /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.169 DEBUG:boto:Data: 2023-09-22T21:21:43.169 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.169 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.169 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.169 DEBUG:boto:Params: {} 2023-09-22T21:21:43.169 DEBUG:boto:Token: None 2023-09-22T21:21:43.169 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.169 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pfQ6Bwc8AmTgMbmdgkX36DBuy4Q= 2023-09-22T21:21:43.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pfQ6Bwc8AmTgMbmdgkX36DBuy4Q=', 'Content-Length': '0'} 2023-09-22T21:21:43.172 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a730c9eb4b70579c-00650e0567-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.173 DEBUG:boto:path=/ffzdqu-46/obj3 2023-09-22T21:21:43.173 DEBUG:boto:auth_path=/ffzdqu-46/obj3 2023-09-22T21:21:43.173 DEBUG:boto:path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.173 DEBUG:boto:auth_path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.173 DEBUG:boto:Method: HEAD 2023-09-22T21:21:43.173 DEBUG:boto:Path: /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.173 DEBUG:boto:Data: 2023-09-22T21:21:43.174 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.174 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.174 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.174 DEBUG:boto:Params: {} 2023-09-22T21:21:43.174 DEBUG:boto:Token: None 2023-09-22T21:21:43.174 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.174 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4yWEyyna/bfubI+aOLCS/WlPd14= 2023-09-22T21:21:43.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4yWEyyna/bfubI+aOLCS/WlPd14=', 'Content-Length': '0'} 2023-09-22T21:21:43.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebdd5d07749b7ebd-00650e0567-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.177 DEBUG:boto:path=/ffzdqu-46/obj3 2023-09-22T21:21:43.177 DEBUG:boto:auth_path=/ffzdqu-46/obj3 2023-09-22T21:21:43.178 DEBUG:boto:path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.178 DEBUG:boto:auth_path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.178 DEBUG:boto:Method: HEAD 2023-09-22T21:21:43.178 DEBUG:boto:Path: /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.178 DEBUG:boto:Data: 2023-09-22T21:21:43.178 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.178 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.178 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.178 DEBUG:boto:Params: {} 2023-09-22T21:21:43.179 DEBUG:boto:Token: None 2023-09-22T21:21:43.179 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.179 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4yWEyyna/bfubI+aOLCS/WlPd14= 2023-09-22T21:21:43.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4yWEyyna/bfubI+aOLCS/WlPd14=', 'Content-Length': '0'} 2023-09-22T21:21:43.182 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0a0c1ec11b738b4-00650e0567-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:21:43.182 DEBUG:boto:path=/ffzdqu-46/obj3 2023-09-22T21:21:43.182 DEBUG:boto:auth_path=/ffzdqu-46/obj3 2023-09-22T21:21:43.182 DEBUG:boto:path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.182 DEBUG:boto:auth_path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.183 DEBUG:boto:Method: GET 2023-09-22T21:21:43.183 DEBUG:boto:Path: /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.183 DEBUG:boto:Data: 2023-09-22T21:21:43.183 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.183 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.183 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.183 DEBUG:boto:Params: {} 2023-09-22T21:21:43.183 DEBUG:boto:Token: None 2023-09-22T21:21:43.183 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.184 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pfQ6Bwc8AmTgMbmdgkX36DBuy4Q= 2023-09-22T21:21:43.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pfQ6Bwc8AmTgMbmdgkX36DBuy4Q=', 'Content-Length': '0'} 2023-09-22T21:21:43.186 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000767e44bdddaedab5-00650e0567-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.234 DEBUG:boto:path=/ffzdqu-46/obj3 2023-09-22T21:21:43.235 DEBUG:boto:auth_path=/ffzdqu-46/obj3 2023-09-22T21:21:43.235 DEBUG:boto:path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.235 DEBUG:boto:auth_path=/ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.235 DEBUG:boto:Method: GET 2023-09-22T21:21:43.235 DEBUG:boto:Path: /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.235 DEBUG:boto:Data: 2023-09-22T21:21:43.235 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.235 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.235 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.235 DEBUG:boto:Params: {} 2023-09-22T21:21:43.236 DEBUG:boto:Token: None 2023-09-22T21:21:43.236 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj3?versionId=null 2023-09-22T21:21:43.236 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pfQ6Bwc8AmTgMbmdgkX36DBuy4Q= 2023-09-22T21:21:43.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pfQ6Bwc8AmTgMbmdgkX36DBuy4Q=', 'Content-Length': '0'} 2023-09-22T21:21:43.239 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa1fdbc19aad9ec7-00650e0567-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.239 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:43.239 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:43.239 DEBUG:boto:path=/ffzdqu-46/?prefix=obj3 2023-09-22T21:21:43.240 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj3 2023-09-22T21:21:43.240 DEBUG:boto:Method: GET 2023-09-22T21:21:43.240 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj3 2023-09-22T21:21:43.240 DEBUG:boto:Data: 2023-09-22T21:21:43.240 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.240 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.240 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.240 DEBUG:boto:Params: {} 2023-09-22T21:21:43.241 DEBUG:boto:Token: None 2023-09-22T21:21:43.241 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/ 2023-09-22T21:21:43.241 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U= 2023-09-22T21:21:43.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U=', 'Content-Length': '0'} 2023-09-22T21:21:43.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065ccb91a2fe7802d-00650e0567-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.290 DEBUG:boto:b'ffzdqu-46obj31000falseobj32023-09-22T21:20:38.842Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:43.291 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:43.291 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:43.291 DEBUG:boto:path=/ffzdqu-46/?prefix=obj3 2023-09-22T21:21:43.291 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj3 2023-09-22T21:21:43.291 DEBUG:boto:Method: GET 2023-09-22T21:21:43.291 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj3 2023-09-22T21:21:43.292 DEBUG:boto:Data: 2023-09-22T21:21:43.292 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.292 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.292 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.292 DEBUG:boto:Params: {} 2023-09-22T21:21:43.292 DEBUG:boto:Token: None 2023-09-22T21:21:43.292 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/ 2023-09-22T21:21:43.292 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U= 2023-09-22T21:21:43.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U=', 'Content-Length': '0'} 2023-09-22T21:21:43.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bdb33cb38df8024-00650e0567-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.298 DEBUG:boto:b'ffzdqu-46obj31000falseobj32023-09-22T21:20:38.842Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:43.298 DEBUG:boto:path=/ffzdqu-46/obj3 2023-09-22T21:21:43.298 DEBUG:boto:auth_path=/ffzdqu-46/obj3 2023-09-22T21:21:43.298 DEBUG:boto:Method: HEAD 2023-09-22T21:21:43.298 DEBUG:boto:Path: /ffzdqu-46/obj3 2023-09-22T21:21:43.299 DEBUG:boto:Data: 2023-09-22T21:21:43.299 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.299 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.299 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.299 DEBUG:boto:Params: {} 2023-09-22T21:21:43.299 DEBUG:boto:Token: None 2023-09-22T21:21:43.299 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj3 2023-09-22T21:21:43.299 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/tXMCtHiE9WLVGmd3jkOyMLtY60= 2023-09-22T21:21:43.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/tXMCtHiE9WLVGmd3jkOyMLtY60=', 'Content-Length': '0'} 2023-09-22T21:21:43.302 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020f96543a6e97521-00650e0567-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.302 DEBUG:boto:path=/ffzdqu-46/obj3 2023-09-22T21:21:43.302 DEBUG:boto:auth_path=/ffzdqu-46/obj3 2023-09-22T21:21:43.302 DEBUG:boto:Method: HEAD 2023-09-22T21:21:43.303 DEBUG:boto:Path: /ffzdqu-46/obj3 2023-09-22T21:21:43.303 DEBUG:boto:Data: 2023-09-22T21:21:43.303 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.303 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.303 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.303 DEBUG:boto:Params: {} 2023-09-22T21:21:43.303 DEBUG:boto:Token: None 2023-09-22T21:21:43.303 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj3 2023-09-22T21:21:43.303 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/tXMCtHiE9WLVGmd3jkOyMLtY60= 2023-09-22T21:21:43.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/tXMCtHiE9WLVGmd3jkOyMLtY60=', 'Content-Length': '0'} 2023-09-22T21:21:43.306 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe859e0f72857630-00650e0567-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:21:43.309 DEBUG:boto:path=/ffzdqu-46/obj3 2023-09-22T21:21:43.309 DEBUG:boto:auth_path=/ffzdqu-46/obj3 2023-09-22T21:21:43.309 DEBUG:boto:Method: GET 2023-09-22T21:21:43.309 DEBUG:boto:Path: /ffzdqu-46/obj3 2023-09-22T21:21:43.309 DEBUG:boto:Data: 2023-09-22T21:21:43.309 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.310 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.310 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.310 DEBUG:boto:Params: {} 2023-09-22T21:21:43.310 DEBUG:boto:Token: None 2023-09-22T21:21:43.310 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj3 2023-09-22T21:21:43.310 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AIk1HwapUaTjn2CkSDnWRfUvvPE= 2023-09-22T21:21:43.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AIk1HwapUaTjn2CkSDnWRfUvvPE=', 'Content-Length': '0'} 2023-09-22T21:21:43.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022ed9731ad0cafee-00650e0567-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.358 DEBUG:boto:path=/ffzdqu-46/obj3 2023-09-22T21:21:43.358 DEBUG:boto:auth_path=/ffzdqu-46/obj3 2023-09-22T21:21:43.359 DEBUG:boto:Method: GET 2023-09-22T21:21:43.359 DEBUG:boto:Path: /ffzdqu-46/obj3 2023-09-22T21:21:43.359 DEBUG:boto:Data: 2023-09-22T21:21:43.359 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.359 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.359 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.359 DEBUG:boto:Params: {} 2023-09-22T21:21:43.359 DEBUG:boto:Token: None 2023-09-22T21:21:43.360 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj3 2023-09-22T21:21:43.360 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AIk1HwapUaTjn2CkSDnWRfUvvPE= 2023-09-22T21:21:43.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AIk1HwapUaTjn2CkSDnWRfUvvPE=', 'Content-Length': '0'} 2023-09-22T21:21:43.362 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9ba1a486b417e27-00650e0567-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.363 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:43.363 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:43.363 DEBUG:boto:path=/ffzdqu-46/?prefix=obj3 2023-09-22T21:21:43.363 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj3 2023-09-22T21:21:43.363 DEBUG:boto:Method: GET 2023-09-22T21:21:43.363 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj3 2023-09-22T21:21:43.363 DEBUG:boto:Data: 2023-09-22T21:21:43.363 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.363 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.364 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.364 DEBUG:boto:Params: {} 2023-09-22T21:21:43.364 DEBUG:boto:Token: None 2023-09-22T21:21:43.364 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/ 2023-09-22T21:21:43.364 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U= 2023-09-22T21:21:43.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U=', 'Content-Length': '0'} 2023-09-22T21:21:43.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005962237f8e89e3a2-00650e0567-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.369 DEBUG:boto:b'ffzdqu-46obj31000falseobj32023-09-22T21:20:38.842Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:43.369 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:43.370 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:43.370 DEBUG:boto:path=/ffzdqu-46/?prefix=obj3 2023-09-22T21:21:43.370 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj3 2023-09-22T21:21:43.370 DEBUG:boto:Method: GET 2023-09-22T21:21:43.370 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj3 2023-09-22T21:21:43.370 DEBUG:boto:Data: 2023-09-22T21:21:43.370 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.370 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.370 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.371 DEBUG:boto:Params: {} 2023-09-22T21:21:43.371 DEBUG:boto:Token: None 2023-09-22T21:21:43.371 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/ 2023-09-22T21:21:43.371 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U= 2023-09-22T21:21:43.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U=', 'Content-Length': '0'} 2023-09-22T21:21:43.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000900aef7b3bb789d8-00650e0567-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.376 DEBUG:boto:b'ffzdqu-46obj31000falseobj32023-09-22T21:20:38.842Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:43.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:21:43.377 DEBUG:boto:path=/ffzdqu-46/obj4 2023-09-22T21:21:43.377 DEBUG:boto:auth_path=/ffzdqu-46/obj4 2023-09-22T21:21:43.377 DEBUG:boto:path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.377 DEBUG:boto:auth_path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.377 DEBUG:boto:Method: GET 2023-09-22T21:21:43.377 DEBUG:boto:Path: /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.377 DEBUG:boto:Data: 2023-09-22T21:21:43.377 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.378 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.378 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.378 DEBUG:boto:Params: {} 2023-09-22T21:21:43.378 DEBUG:boto:Token: None 2023-09-22T21:21:43.378 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.378 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nlI1uRugr9YspVeaAsAv02U3twI= 2023-09-22T21:21:43.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nlI1uRugr9YspVeaAsAv02U3twI=', 'Content-Length': '0'} 2023-09-22T21:21:43.381 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068969620ca5bba6a-00650e0567-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.426 DEBUG:boto:path=/ffzdqu-46/obj4 2023-09-22T21:21:43.427 DEBUG:boto:auth_path=/ffzdqu-46/obj4 2023-09-22T21:21:43.427 DEBUG:boto:path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.427 DEBUG:boto:auth_path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.427 DEBUG:boto:Method: GET 2023-09-22T21:21:43.427 DEBUG:boto:Path: /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.427 DEBUG:boto:Data: 2023-09-22T21:21:43.427 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.427 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.428 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.428 DEBUG:boto:Params: {} 2023-09-22T21:21:43.428 DEBUG:boto:Token: None 2023-09-22T21:21:43.428 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.428 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nlI1uRugr9YspVeaAsAv02U3twI= 2023-09-22T21:21:43.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nlI1uRugr9YspVeaAsAv02U3twI=', 'Content-Length': '0'} 2023-09-22T21:21:43.431 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d605076cf0445886-00650e0567-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.431 DEBUG:boto:path=/ffzdqu-46/obj4 2023-09-22T21:21:43.431 DEBUG:boto:auth_path=/ffzdqu-46/obj4 2023-09-22T21:21:43.432 DEBUG:boto:path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.432 DEBUG:boto:auth_path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.432 DEBUG:boto:Method: HEAD 2023-09-22T21:21:43.432 DEBUG:boto:Path: /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.432 DEBUG:boto:Data: 2023-09-22T21:21:43.432 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.432 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.432 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.432 DEBUG:boto:Params: {} 2023-09-22T21:21:43.433 DEBUG:boto:Token: None 2023-09-22T21:21:43.433 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.433 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dxTcAII9sqElL09WtXyRm4HEpcY= 2023-09-22T21:21:43.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dxTcAII9sqElL09WtXyRm4HEpcY=', 'Content-Length': '0'} 2023-09-22T21:21:43.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9592712b2c7d665-00650e0567-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.436 DEBUG:boto:path=/ffzdqu-46/obj4 2023-09-22T21:21:43.436 DEBUG:boto:auth_path=/ffzdqu-46/obj4 2023-09-22T21:21:43.436 DEBUG:boto:path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.436 DEBUG:boto:auth_path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.436 DEBUG:boto:Method: HEAD 2023-09-22T21:21:43.436 DEBUG:boto:Path: /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.437 DEBUG:boto:Data: 2023-09-22T21:21:43.437 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.437 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.437 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.437 DEBUG:boto:Params: {} 2023-09-22T21:21:43.437 DEBUG:boto:Token: None 2023-09-22T21:21:43.437 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.437 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dxTcAII9sqElL09WtXyRm4HEpcY= 2023-09-22T21:21:43.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dxTcAII9sqElL09WtXyRm4HEpcY=', 'Content-Length': '0'} 2023-09-22T21:21:43.440 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008882e1676f2f75fc-00650e0567-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:21:43.440 DEBUG:boto:path=/ffzdqu-46/obj4 2023-09-22T21:21:43.440 DEBUG:boto:auth_path=/ffzdqu-46/obj4 2023-09-22T21:21:43.441 DEBUG:boto:path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.441 DEBUG:boto:auth_path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.441 DEBUG:boto:Method: GET 2023-09-22T21:21:43.441 DEBUG:boto:Path: /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.441 DEBUG:boto:Data: 2023-09-22T21:21:43.441 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.441 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.441 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.441 DEBUG:boto:Params: {} 2023-09-22T21:21:43.441 DEBUG:boto:Token: None 2023-09-22T21:21:43.442 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.442 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nlI1uRugr9YspVeaAsAv02U3twI= 2023-09-22T21:21:43.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nlI1uRugr9YspVeaAsAv02U3twI=', 'Content-Length': '0'} 2023-09-22T21:21:43.444 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005427dc62394d78e4-00650e0567-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.490 DEBUG:boto:path=/ffzdqu-46/obj4 2023-09-22T21:21:43.491 DEBUG:boto:auth_path=/ffzdqu-46/obj4 2023-09-22T21:21:43.491 DEBUG:boto:path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.491 DEBUG:boto:auth_path=/ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.491 DEBUG:boto:Method: GET 2023-09-22T21:21:43.491 DEBUG:boto:Path: /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.491 DEBUG:boto:Data: 2023-09-22T21:21:43.491 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.491 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.492 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.492 DEBUG:boto:Params: {} 2023-09-22T21:21:43.492 DEBUG:boto:Token: None 2023-09-22T21:21:43.492 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj4?versionId=null 2023-09-22T21:21:43.492 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nlI1uRugr9YspVeaAsAv02U3twI= 2023-09-22T21:21:43.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nlI1uRugr9YspVeaAsAv02U3twI=', 'Content-Length': '0'} 2023-09-22T21:21:43.495 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005719bc44d8c91ae2-00650e0567-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.496 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:43.496 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:43.496 DEBUG:boto:path=/ffzdqu-46/?prefix=obj4 2023-09-22T21:21:43.496 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj4 2023-09-22T21:21:43.496 DEBUG:boto:Method: GET 2023-09-22T21:21:43.496 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj4 2023-09-22T21:21:43.497 DEBUG:boto:Data: 2023-09-22T21:21:43.497 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.497 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.497 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.497 DEBUG:boto:Params: {} 2023-09-22T21:21:43.497 DEBUG:boto:Token: None 2023-09-22T21:21:43.497 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/ 2023-09-22T21:21:43.497 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U= 2023-09-22T21:21:43.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U=', 'Content-Length': '0'} 2023-09-22T21:21:43.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000def87b8e9b115989-00650e0567-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.542 DEBUG:boto:b'ffzdqu-46obj41000falseobj42023-09-22T21:20:38.862Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:43.543 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:43.543 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:43.543 DEBUG:boto:path=/ffzdqu-46/?prefix=obj4 2023-09-22T21:21:43.543 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj4 2023-09-22T21:21:43.543 DEBUG:boto:Method: GET 2023-09-22T21:21:43.544 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj4 2023-09-22T21:21:43.544 DEBUG:boto:Data: 2023-09-22T21:21:43.544 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.544 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.544 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.544 DEBUG:boto:Params: {} 2023-09-22T21:21:43.544 DEBUG:boto:Token: None 2023-09-22T21:21:43.544 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/ 2023-09-22T21:21:43.545 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U= 2023-09-22T21:21:43.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U=', 'Content-Length': '0'} 2023-09-22T21:21:43.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccc88979d0548dcf-00650e0567-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.549 DEBUG:boto:b'ffzdqu-46obj41000falseobj42023-09-22T21:20:38.862Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:43.550 DEBUG:boto:path=/ffzdqu-46/obj4 2023-09-22T21:21:43.550 DEBUG:boto:auth_path=/ffzdqu-46/obj4 2023-09-22T21:21:43.550 DEBUG:boto:Method: HEAD 2023-09-22T21:21:43.550 DEBUG:boto:Path: /ffzdqu-46/obj4 2023-09-22T21:21:43.550 DEBUG:boto:Data: 2023-09-22T21:21:43.550 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.550 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.551 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.551 DEBUG:boto:Params: {} 2023-09-22T21:21:43.551 DEBUG:boto:Token: None 2023-09-22T21:21:43.551 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj4 2023-09-22T21:21:43.551 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tRQO8hZaDXeWIPo3qoNYw1BhIoI= 2023-09-22T21:21:43.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tRQO8hZaDXeWIPo3qoNYw1BhIoI=', 'Content-Length': '0'} 2023-09-22T21:21:43.554 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a3a082626f4264f-00650e0567-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.554 DEBUG:boto:path=/ffzdqu-46/obj4 2023-09-22T21:21:43.554 DEBUG:boto:auth_path=/ffzdqu-46/obj4 2023-09-22T21:21:43.554 DEBUG:boto:Method: HEAD 2023-09-22T21:21:43.554 DEBUG:boto:Path: /ffzdqu-46/obj4 2023-09-22T21:21:43.554 DEBUG:boto:Data: 2023-09-22T21:21:43.555 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.555 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.555 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.555 DEBUG:boto:Params: {} 2023-09-22T21:21:43.555 DEBUG:boto:Token: None 2023-09-22T21:21:43.555 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj4 2023-09-22T21:21:43.555 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tRQO8hZaDXeWIPo3qoNYw1BhIoI= 2023-09-22T21:21:43.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tRQO8hZaDXeWIPo3qoNYw1BhIoI=', 'Content-Length': '0'} 2023-09-22T21:21:43.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043dda8bc0b6c468d-00650e0567-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:21:43.559 DEBUG:boto:path=/ffzdqu-46/obj4 2023-09-22T21:21:43.559 DEBUG:boto:auth_path=/ffzdqu-46/obj4 2023-09-22T21:21:43.559 DEBUG:boto:Method: GET 2023-09-22T21:21:43.559 DEBUG:boto:Path: /ffzdqu-46/obj4 2023-09-22T21:21:43.559 DEBUG:boto:Data: 2023-09-22T21:21:43.559 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.559 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.560 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.560 DEBUG:boto:Params: {} 2023-09-22T21:21:43.560 DEBUG:boto:Token: None 2023-09-22T21:21:43.560 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj4 2023-09-22T21:21:43.560 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OCQkoD/0LvdSZhHJVYfsRMc13sM= 2023-09-22T21:21:43.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OCQkoD/0LvdSZhHJVYfsRMc13sM=', 'Content-Length': '0'} 2023-09-22T21:21:43.562 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000433c0641a68e9efe-00650e0567-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.606 DEBUG:boto:path=/ffzdqu-46/obj4 2023-09-22T21:21:43.606 DEBUG:boto:auth_path=/ffzdqu-46/obj4 2023-09-22T21:21:43.607 DEBUG:boto:Method: GET 2023-09-22T21:21:43.607 DEBUG:boto:Path: /ffzdqu-46/obj4 2023-09-22T21:21:43.607 DEBUG:boto:Data: 2023-09-22T21:21:43.607 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.607 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.607 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.607 DEBUG:boto:Params: {} 2023-09-22T21:21:43.607 DEBUG:boto:Token: None 2023-09-22T21:21:43.608 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj4 2023-09-22T21:21:43.608 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OCQkoD/0LvdSZhHJVYfsRMc13sM= 2023-09-22T21:21:43.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OCQkoD/0LvdSZhHJVYfsRMc13sM=', 'Content-Length': '0'} 2023-09-22T21:21:43.610 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7c99df76c20ad76-00650e0567-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.611 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:43.611 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:43.611 DEBUG:boto:path=/ffzdqu-46/?prefix=obj4 2023-09-22T21:21:43.611 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj4 2023-09-22T21:21:43.611 DEBUG:boto:Method: GET 2023-09-22T21:21:43.611 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj4 2023-09-22T21:21:43.612 DEBUG:boto:Data: 2023-09-22T21:21:43.612 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.612 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.612 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.612 DEBUG:boto:Params: {} 2023-09-22T21:21:43.612 DEBUG:boto:Token: None 2023-09-22T21:21:43.612 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/ 2023-09-22T21:21:43.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U= 2023-09-22T21:21:43.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U=', 'Content-Length': '0'} 2023-09-22T21:21:43.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1f85859e70209b2-00650e0567-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.617 DEBUG:boto:b'ffzdqu-46obj41000falseobj42023-09-22T21:20:38.862Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:43.617 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:43.618 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:43.618 DEBUG:boto:path=/ffzdqu-46/?prefix=obj4 2023-09-22T21:21:43.618 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj4 2023-09-22T21:21:43.618 DEBUG:boto:Method: GET 2023-09-22T21:21:43.618 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj4 2023-09-22T21:21:43.618 DEBUG:boto:Data: 2023-09-22T21:21:43.618 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.618 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.618 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.619 DEBUG:boto:Params: {} 2023-09-22T21:21:43.619 DEBUG:boto:Token: None 2023-09-22T21:21:43.619 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/ 2023-09-22T21:21:43.619 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U= 2023-09-22T21:21:43.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U=', 'Content-Length': '0'} 2023-09-22T21:21:43.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e86b57541ccfc05-00650e0567-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.624 DEBUG:boto:b'ffzdqu-46obj41000falseobj42023-09-22T21:20:38.862Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:43.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-22T21:21:43.624 DEBUG:boto:path=/ffzdqu-46/obj5 2023-09-22T21:21:43.624 DEBUG:boto:auth_path=/ffzdqu-46/obj5 2023-09-22T21:21:43.624 DEBUG:boto:path=/ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.624 DEBUG:boto:auth_path=/ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.625 DEBUG:boto:Method: GET 2023-09-22T21:21:43.625 DEBUG:boto:Path: /ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.625 DEBUG:boto:Data: 2023-09-22T21:21:43.625 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.625 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.625 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.625 DEBUG:boto:Params: {} 2023-09-22T21:21:43.625 DEBUG:boto:Token: None 2023-09-22T21:21:43.625 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.626 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/qny+nC0q0W012mtiJiS/xP41l0= 2023-09-22T21:21:43.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/qny+nC0q0W012mtiJiS/xP41l0=', 'Content-Length': '0'} 2023-09-22T21:21:43.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017c893c603996347-00650e0567-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.670 DEBUG:boto:path=/ffzdqu-46/obj5 2023-09-22T21:21:43.670 DEBUG:boto:auth_path=/ffzdqu-46/obj5 2023-09-22T21:21:43.671 DEBUG:boto:path=/ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.671 DEBUG:boto:auth_path=/ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.671 DEBUG:boto:Method: GET 2023-09-22T21:21:43.671 DEBUG:boto:Path: /ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.671 DEBUG:boto:Data: 2023-09-22T21:21:43.671 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.671 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.671 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.671 DEBUG:boto:Params: {} 2023-09-22T21:21:43.672 DEBUG:boto:Token: None 2023-09-22T21:21:43.672 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.672 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/qny+nC0q0W012mtiJiS/xP41l0= 2023-09-22T21:21:43.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/qny+nC0q0W012mtiJiS/xP41l0=', 'Content-Length': '0'} 2023-09-22T21:21:43.675 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011bd556c83718ab5-00650e0567-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.675 DEBUG:boto:path=/ffzdqu-46/obj5 2023-09-22T21:21:43.675 DEBUG:boto:auth_path=/ffzdqu-46/obj5 2023-09-22T21:21:43.675 DEBUG:boto:path=/ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.676 DEBUG:boto:auth_path=/ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.676 DEBUG:boto:Method: HEAD 2023-09-22T21:21:43.676 DEBUG:boto:Path: /ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.676 DEBUG:boto:Data: 2023-09-22T21:21:43.676 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.676 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.676 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.676 DEBUG:boto:Params: {} 2023-09-22T21:21:43.676 DEBUG:boto:Token: None 2023-09-22T21:21:43.677 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.677 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:a3mbCVTJ6IrckhzhW+ypC458RDU= 2023-09-22T21:21:43.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:a3mbCVTJ6IrckhzhW+ypC458RDU=', 'Content-Length': '0'} 2023-09-22T21:21:43.679 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043c2e7d5d2fa9050-00650e0567-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.680 DEBUG:boto:path=/ffzdqu-46/obj5 2023-09-22T21:21:43.680 DEBUG:boto:auth_path=/ffzdqu-46/obj5 2023-09-22T21:21:43.680 DEBUG:boto:path=/ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.680 DEBUG:boto:auth_path=/ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.680 DEBUG:boto:Method: HEAD 2023-09-22T21:21:43.680 DEBUG:boto:Path: /ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.680 DEBUG:boto:Data: 2023-09-22T21:21:43.680 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.681 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.681 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.681 DEBUG:boto:Params: {} 2023-09-22T21:21:43.681 DEBUG:boto:Token: None 2023-09-22T21:21:43.681 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.681 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:a3mbCVTJ6IrckhzhW+ypC458RDU= 2023-09-22T21:21:43.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:a3mbCVTJ6IrckhzhW+ypC458RDU=', 'Content-Length': '0'} 2023-09-22T21:21:43.684 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d0948a424dc9314-00650e0567-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-22T21:21:43.684 DEBUG:boto:path=/ffzdqu-46/obj5 2023-09-22T21:21:43.684 DEBUG:boto:auth_path=/ffzdqu-46/obj5 2023-09-22T21:21:43.684 DEBUG:boto:path=/ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.685 DEBUG:boto:auth_path=/ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.685 DEBUG:boto:Method: GET 2023-09-22T21:21:43.685 DEBUG:boto:Path: /ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.685 DEBUG:boto:Data: 2023-09-22T21:21:43.685 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.685 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.685 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.685 DEBUG:boto:Params: {} 2023-09-22T21:21:43.685 DEBUG:boto:Token: None 2023-09-22T21:21:43.686 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.686 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/qny+nC0q0W012mtiJiS/xP41l0= 2023-09-22T21:21:43.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/qny+nC0q0W012mtiJiS/xP41l0=', 'Content-Length': '0'} 2023-09-22T21:21:43.688 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8439b361bfaeded-00650e0567-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.734 DEBUG:boto:path=/ffzdqu-46/obj5 2023-09-22T21:21:43.735 DEBUG:boto:auth_path=/ffzdqu-46/obj5 2023-09-22T21:21:43.735 DEBUG:boto:path=/ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.735 DEBUG:boto:auth_path=/ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.735 DEBUG:boto:Method: GET 2023-09-22T21:21:43.735 DEBUG:boto:Path: /ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.735 DEBUG:boto:Data: 2023-09-22T21:21:43.735 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.735 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.735 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.735 DEBUG:boto:Params: {} 2023-09-22T21:21:43.736 DEBUG:boto:Token: None 2023-09-22T21:21:43.736 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj5?versionId=null 2023-09-22T21:21:43.736 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/qny+nC0q0W012mtiJiS/xP41l0= 2023-09-22T21:21:43.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/qny+nC0q0W012mtiJiS/xP41l0=', 'Content-Length': '0'} 2023-09-22T21:21:43.739 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000307f05f784c61515-00650e0567-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.739 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:43.739 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:43.739 DEBUG:boto:path=/ffzdqu-46/?prefix=obj5 2023-09-22T21:21:43.740 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj5 2023-09-22T21:21:43.740 DEBUG:boto:Method: GET 2023-09-22T21:21:43.740 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj5 2023-09-22T21:21:43.740 DEBUG:boto:Data: 2023-09-22T21:21:43.740 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.740 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.740 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.740 DEBUG:boto:Params: {} 2023-09-22T21:21:43.740 DEBUG:boto:Token: None 2023-09-22T21:21:43.740 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/ 2023-09-22T21:21:43.741 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U= 2023-09-22T21:21:43.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U=', 'Content-Length': '0'} 2023-09-22T21:21:43.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000803d62247af9197c-00650e0567-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.786 DEBUG:boto:b'ffzdqu-46obj51000falseobj52023-09-22T21:21:22.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:43.787 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:43.787 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:43.787 DEBUG:boto:path=/ffzdqu-46/?prefix=obj5 2023-09-22T21:21:43.787 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj5 2023-09-22T21:21:43.787 DEBUG:boto:Method: GET 2023-09-22T21:21:43.787 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj5 2023-09-22T21:21:43.788 DEBUG:boto:Data: 2023-09-22T21:21:43.788 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.788 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.788 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.788 DEBUG:boto:Params: {} 2023-09-22T21:21:43.788 DEBUG:boto:Token: None 2023-09-22T21:21:43.788 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/ 2023-09-22T21:21:43.788 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U= 2023-09-22T21:21:43.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U=', 'Content-Length': '0'} 2023-09-22T21:21:43.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000638bb06041a3ca19-00650e0567-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.793 DEBUG:boto:b'ffzdqu-46obj51000falseobj52023-09-22T21:21:22.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:43.794 DEBUG:boto:path=/ffzdqu-46/obj5 2023-09-22T21:21:43.794 DEBUG:boto:auth_path=/ffzdqu-46/obj5 2023-09-22T21:21:43.794 DEBUG:boto:Method: HEAD 2023-09-22T21:21:43.794 DEBUG:boto:Path: /ffzdqu-46/obj5 2023-09-22T21:21:43.794 DEBUG:boto:Data: 2023-09-22T21:21:43.794 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.794 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.795 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.795 DEBUG:boto:Params: {} 2023-09-22T21:21:43.795 DEBUG:boto:Token: None 2023-09-22T21:21:43.795 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj5 2023-09-22T21:21:43.795 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DDak95HUbUvi1LtK5o7GXRRkgJ0= 2023-09-22T21:21:43.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DDak95HUbUvi1LtK5o7GXRRkgJ0=', 'Content-Length': '0'} 2023-09-22T21:21:43.798 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3c57e34e6ae0dd9-00650e0567-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.799 DEBUG:boto:path=/ffzdqu-46/obj5 2023-09-22T21:21:43.799 DEBUG:boto:auth_path=/ffzdqu-46/obj5 2023-09-22T21:21:43.799 DEBUG:boto:Method: HEAD 2023-09-22T21:21:43.799 DEBUG:boto:Path: /ffzdqu-46/obj5 2023-09-22T21:21:43.799 DEBUG:boto:Data: 2023-09-22T21:21:43.799 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.799 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.799 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.800 DEBUG:boto:Params: {} 2023-09-22T21:21:43.800 DEBUG:boto:Token: None 2023-09-22T21:21:43.800 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj5 2023-09-22T21:21:43.800 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DDak95HUbUvi1LtK5o7GXRRkgJ0= 2023-09-22T21:21:43.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DDak95HUbUvi1LtK5o7GXRRkgJ0=', 'Content-Length': '0'} 2023-09-22T21:21:43.803 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f8f85e39100c1ee-00650e0567-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-22T21:21:43.803 DEBUG:boto:path=/ffzdqu-46/obj5 2023-09-22T21:21:43.803 DEBUG:boto:auth_path=/ffzdqu-46/obj5 2023-09-22T21:21:43.803 DEBUG:boto:Method: GET 2023-09-22T21:21:43.804 DEBUG:boto:Path: /ffzdqu-46/obj5 2023-09-22T21:21:43.804 DEBUG:boto:Data: 2023-09-22T21:21:43.804 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.804 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.804 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.804 DEBUG:boto:Params: {} 2023-09-22T21:21:43.804 DEBUG:boto:Token: None 2023-09-22T21:21:43.804 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj5 2023-09-22T21:21:43.804 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+rdx771i7gQjS71mdb0YpRxlzl4= 2023-09-22T21:21:43.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+rdx771i7gQjS71mdb0YpRxlzl4=', 'Content-Length': '0'} 2023-09-22T21:21:43.807 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d486500c29e7e0a-00650e0567-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.850 DEBUG:boto:path=/ffzdqu-46/obj5 2023-09-22T21:21:43.851 DEBUG:boto:auth_path=/ffzdqu-46/obj5 2023-09-22T21:21:43.851 DEBUG:boto:Method: GET 2023-09-22T21:21:43.851 DEBUG:boto:Path: /ffzdqu-46/obj5 2023-09-22T21:21:43.851 DEBUG:boto:Data: 2023-09-22T21:21:43.851 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.851 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.851 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.851 DEBUG:boto:Params: {} 2023-09-22T21:21:43.852 DEBUG:boto:Token: None 2023-09-22T21:21:43.852 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj5 2023-09-22T21:21:43.852 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+rdx771i7gQjS71mdb0YpRxlzl4= 2023-09-22T21:21:43.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+rdx771i7gQjS71mdb0YpRxlzl4=', 'Content-Length': '0'} 2023-09-22T21:21:43.855 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f03a988b6d02c01-00650e0567-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.855 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:43.855 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:43.855 DEBUG:boto:path=/ffzdqu-46/?prefix=obj5 2023-09-22T21:21:43.855 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj5 2023-09-22T21:21:43.856 DEBUG:boto:Method: GET 2023-09-22T21:21:43.856 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj5 2023-09-22T21:21:43.856 DEBUG:boto:Data: 2023-09-22T21:21:43.856 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.856 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.856 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.856 DEBUG:boto:Params: {} 2023-09-22T21:21:43.856 DEBUG:boto:Token: None 2023-09-22T21:21:43.856 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/ 2023-09-22T21:21:43.857 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U= 2023-09-22T21:21:43.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U=', 'Content-Length': '0'} 2023-09-22T21:21:43.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000057bc20fea1868a-00650e0567-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.902 DEBUG:boto:b'ffzdqu-46obj51000falseobj52023-09-22T21:21:22.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:43.903 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:43.903 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:43.903 DEBUG:boto:path=/ffzdqu-46/?prefix=obj5 2023-09-22T21:21:43.903 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj5 2023-09-22T21:21:43.903 DEBUG:boto:Method: GET 2023-09-22T21:21:43.904 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj5 2023-09-22T21:21:43.904 DEBUG:boto:Data: 2023-09-22T21:21:43.904 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.904 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.904 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.904 DEBUG:boto:Params: {} 2023-09-22T21:21:43.904 DEBUG:boto:Token: None 2023-09-22T21:21:43.904 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/ 2023-09-22T21:21:43.904 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U= 2023-09-22T21:21:43.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U=', 'Content-Length': '0'} 2023-09-22T21:21:43.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008379b8eecad3d2d7-00650e0567-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.912 DEBUG:boto:b'ffzdqu-46obj51000falseobj52023-09-22T21:21:22.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:43.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-22T21:21:43.912 DEBUG:boto:path=/ffzdqu-46/obj6 2023-09-22T21:21:43.913 DEBUG:boto:auth_path=/ffzdqu-46/obj6 2023-09-22T21:21:43.913 DEBUG:boto:path=/ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.913 DEBUG:boto:auth_path=/ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.913 DEBUG:boto:Method: GET 2023-09-22T21:21:43.913 DEBUG:boto:Path: /ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.913 DEBUG:boto:Data: 2023-09-22T21:21:43.913 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.913 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.913 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.913 DEBUG:boto:Params: {} 2023-09-22T21:21:43.914 DEBUG:boto:Token: None 2023-09-22T21:21:43.914 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.914 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tlvFuep9wLxa74ZdpsxLn+x0AaA= 2023-09-22T21:21:43.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tlvFuep9wLxa74ZdpsxLn+x0AaA=', 'Content-Length': '0'} 2023-09-22T21:21:43.917 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042adde873f22b303-00650e0567-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.917 DEBUG:boto:path=/ffzdqu-46/obj6 2023-09-22T21:21:43.918 DEBUG:boto:auth_path=/ffzdqu-46/obj6 2023-09-22T21:21:43.918 DEBUG:boto:path=/ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.918 DEBUG:boto:auth_path=/ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.918 DEBUG:boto:Method: GET 2023-09-22T21:21:43.918 DEBUG:boto:Path: /ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.918 DEBUG:boto:Data: 2023-09-22T21:21:43.918 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.918 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.919 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.919 DEBUG:boto:Params: {} 2023-09-22T21:21:43.919 DEBUG:boto:Token: None 2023-09-22T21:21:43.919 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.919 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tlvFuep9wLxa74ZdpsxLn+x0AaA= 2023-09-22T21:21:43.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tlvFuep9wLxa74ZdpsxLn+x0AaA=', 'Content-Length': '0'} 2023-09-22T21:21:43.922 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009907dba141ab89b2-00650e0567-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.922 DEBUG:boto:path=/ffzdqu-46/obj6 2023-09-22T21:21:43.922 DEBUG:boto:auth_path=/ffzdqu-46/obj6 2023-09-22T21:21:43.922 DEBUG:boto:path=/ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.922 DEBUG:boto:auth_path=/ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.922 DEBUG:boto:Method: HEAD 2023-09-22T21:21:43.922 DEBUG:boto:Path: /ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.923 DEBUG:boto:Data: 2023-09-22T21:21:43.923 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.923 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.923 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.923 DEBUG:boto:Params: {} 2023-09-22T21:21:43.923 DEBUG:boto:Token: None 2023-09-22T21:21:43.923 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.923 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GoN+39VNXdnmgfhRS+Yz7QgsfBs= 2023-09-22T21:21:43.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GoN+39VNXdnmgfhRS+Yz7QgsfBs=', 'Content-Length': '0'} 2023-09-22T21:21:43.926 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ccf245efb7d52870-00650e0567-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.926 DEBUG:boto:path=/ffzdqu-46/obj6 2023-09-22T21:21:43.927 DEBUG:boto:auth_path=/ffzdqu-46/obj6 2023-09-22T21:21:43.927 DEBUG:boto:path=/ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.927 DEBUG:boto:auth_path=/ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.927 DEBUG:boto:Method: HEAD 2023-09-22T21:21:43.927 DEBUG:boto:Path: /ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.927 DEBUG:boto:Data: 2023-09-22T21:21:43.927 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.927 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.927 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.928 DEBUG:boto:Params: {} 2023-09-22T21:21:43.928 DEBUG:boto:Token: None 2023-09-22T21:21:43.928 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.928 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GoN+39VNXdnmgfhRS+Yz7QgsfBs= 2023-09-22T21:21:43.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GoN+39VNXdnmgfhRS+Yz7QgsfBs=', 'Content-Length': '0'} 2023-09-22T21:21:43.931 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f88b955fc5cd1ed4-00650e0567-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-22T21:21:43.931 DEBUG:boto:path=/ffzdqu-46/obj6 2023-09-22T21:21:43.931 DEBUG:boto:auth_path=/ffzdqu-46/obj6 2023-09-22T21:21:43.931 DEBUG:boto:path=/ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.931 DEBUG:boto:auth_path=/ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.931 DEBUG:boto:Method: GET 2023-09-22T21:21:43.932 DEBUG:boto:Path: /ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.932 DEBUG:boto:Data: 2023-09-22T21:21:43.932 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.932 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.932 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.932 DEBUG:boto:Params: {} 2023-09-22T21:21:43.932 DEBUG:boto:Token: None 2023-09-22T21:21:43.932 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.932 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tlvFuep9wLxa74ZdpsxLn+x0AaA= 2023-09-22T21:21:43.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tlvFuep9wLxa74ZdpsxLn+x0AaA=', 'Content-Length': '0'} 2023-09-22T21:21:43.935 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c62ca275a00c723a-00650e0567-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.978 DEBUG:boto:path=/ffzdqu-46/obj6 2023-09-22T21:21:43.979 DEBUG:boto:auth_path=/ffzdqu-46/obj6 2023-09-22T21:21:43.979 DEBUG:boto:path=/ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.979 DEBUG:boto:auth_path=/ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.979 DEBUG:boto:Method: GET 2023-09-22T21:21:43.979 DEBUG:boto:Path: /ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.979 DEBUG:boto:Data: 2023-09-22T21:21:43.979 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.979 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:43.980 DEBUG:boto:Port: 8000 2023-09-22T21:21:43.980 DEBUG:boto:Params: {} 2023-09-22T21:21:43.980 DEBUG:boto:Token: None 2023-09-22T21:21:43.980 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/obj6?versionId=null 2023-09-22T21:21:43.980 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tlvFuep9wLxa74ZdpsxLn+x0AaA= 2023-09-22T21:21:43.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tlvFuep9wLxa74ZdpsxLn+x0AaA=', 'Content-Length': '0'} 2023-09-22T21:21:43.983 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000391395a0853b1fbc-00650e0567-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:43.983 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:43.983 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:43.983 DEBUG:boto:path=/ffzdqu-46/?prefix=obj6 2023-09-22T21:21:43.984 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj6 2023-09-22T21:21:43.984 DEBUG:boto:Method: GET 2023-09-22T21:21:43.984 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj6 2023-09-22T21:21:43.984 DEBUG:boto:Data: 2023-09-22T21:21:43.984 DEBUG:boto:Headers: {} 2023-09-22T21:21:43.984 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:43.984 DEBUG:boto:Port: 8001 2023-09-22T21:21:43.984 DEBUG:boto:Params: {} 2023-09-22T21:21:43.985 DEBUG:boto:Token: None 2023-09-22T21:21:43.985 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:43 GMT /ffzdqu-46/ 2023-09-22T21:21:43.985 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U= 2023-09-22T21:21:43.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WzzXYWlmuizO4mZGc2F5VnCXo1U=', 'Content-Length': '0'} 2023-09-22T21:21:43.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000690a7b65ba2e4fd3-00650e0567-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.030 DEBUG:boto:b'ffzdqu-46obj61000falseobj62023-09-22T21:21:22.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:44.031 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:44.031 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:44.031 DEBUG:boto:path=/ffzdqu-46/?prefix=obj6 2023-09-22T21:21:44.031 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj6 2023-09-22T21:21:44.031 DEBUG:boto:Method: GET 2023-09-22T21:21:44.032 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj6 2023-09-22T21:21:44.032 DEBUG:boto:Data: 2023-09-22T21:21:44.032 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.032 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:44.032 DEBUG:boto:Port: 8000 2023-09-22T21:21:44.032 DEBUG:boto:Params: {} 2023-09-22T21:21:44.032 DEBUG:boto:Token: None 2023-09-22T21:21:44.032 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/ 2023-09-22T21:21:44.033 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI= 2023-09-22T21:21:44.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI=', 'Content-Length': '0'} 2023-09-22T21:21:44.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043c96ceb2af7090e-00650e0568-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.037 DEBUG:boto:b'ffzdqu-46obj61000falseobj62023-09-22T21:21:22.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:44.037 DEBUG:boto:path=/ffzdqu-46/obj6 2023-09-22T21:21:44.037 DEBUG:boto:auth_path=/ffzdqu-46/obj6 2023-09-22T21:21:44.037 DEBUG:boto:Method: HEAD 2023-09-22T21:21:44.038 DEBUG:boto:Path: /ffzdqu-46/obj6 2023-09-22T21:21:44.038 DEBUG:boto:Data: 2023-09-22T21:21:44.038 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.038 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:44.038 DEBUG:boto:Port: 8001 2023-09-22T21:21:44.038 DEBUG:boto:Params: {} 2023-09-22T21:21:44.038 DEBUG:boto:Token: None 2023-09-22T21:21:44.038 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj6 2023-09-22T21:21:44.039 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MzAeC3LYbMylnbch6htlqju3ZyM= 2023-09-22T21:21:44.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MzAeC3LYbMylnbch6htlqju3ZyM=', 'Content-Length': '0'} 2023-09-22T21:21:44.041 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000557e0bea3689cf93-00650e0568-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.041 DEBUG:boto:path=/ffzdqu-46/obj6 2023-09-22T21:21:44.041 DEBUG:boto:auth_path=/ffzdqu-46/obj6 2023-09-22T21:21:44.042 DEBUG:boto:Method: HEAD 2023-09-22T21:21:44.042 DEBUG:boto:Path: /ffzdqu-46/obj6 2023-09-22T21:21:44.042 DEBUG:boto:Data: 2023-09-22T21:21:44.042 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.042 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:44.042 DEBUG:boto:Port: 8000 2023-09-22T21:21:44.042 DEBUG:boto:Params: {} 2023-09-22T21:21:44.042 DEBUG:boto:Token: None 2023-09-22T21:21:44.042 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj6 2023-09-22T21:21:44.043 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MzAeC3LYbMylnbch6htlqju3ZyM= 2023-09-22T21:21:44.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MzAeC3LYbMylnbch6htlqju3ZyM=', 'Content-Length': '0'} 2023-09-22T21:21:44.045 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001173181cf09c22ee-00650e0568-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-22T21:21:44.045 DEBUG:boto:path=/ffzdqu-46/obj6 2023-09-22T21:21:44.046 DEBUG:boto:auth_path=/ffzdqu-46/obj6 2023-09-22T21:21:44.046 DEBUG:boto:Method: GET 2023-09-22T21:21:44.046 DEBUG:boto:Path: /ffzdqu-46/obj6 2023-09-22T21:21:44.046 DEBUG:boto:Data: 2023-09-22T21:21:44.046 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.046 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:44.046 DEBUG:boto:Port: 8001 2023-09-22T21:21:44.046 DEBUG:boto:Params: {} 2023-09-22T21:21:44.046 DEBUG:boto:Token: None 2023-09-22T21:21:44.047 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj6 2023-09-22T21:21:44.047 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H6JUtterQpOdaEoRc7GapxNKBBA= 2023-09-22T21:21:44.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H6JUtterQpOdaEoRc7GapxNKBBA=', 'Content-Length': '0'} 2023-09-22T21:21:44.049 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009b07dff9d7e75f1-00650e0568-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.094 DEBUG:boto:path=/ffzdqu-46/obj6 2023-09-22T21:21:44.095 DEBUG:boto:auth_path=/ffzdqu-46/obj6 2023-09-22T21:21:44.095 DEBUG:boto:Method: GET 2023-09-22T21:21:44.095 DEBUG:boto:Path: /ffzdqu-46/obj6 2023-09-22T21:21:44.095 DEBUG:boto:Data: 2023-09-22T21:21:44.095 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:44.095 DEBUG:boto:Port: 8000 2023-09-22T21:21:44.095 DEBUG:boto:Params: {} 2023-09-22T21:21:44.096 DEBUG:boto:Token: None 2023-09-22T21:21:44.096 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj6 2023-09-22T21:21:44.096 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H6JUtterQpOdaEoRc7GapxNKBBA= 2023-09-22T21:21:44.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H6JUtterQpOdaEoRc7GapxNKBBA=', 'Content-Length': '0'} 2023-09-22T21:21:44.099 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fe0bb3716c394d5-00650e0568-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.099 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:44.099 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:44.100 DEBUG:boto:path=/ffzdqu-46/?prefix=obj6 2023-09-22T21:21:44.100 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj6 2023-09-22T21:21:44.100 DEBUG:boto:Method: GET 2023-09-22T21:21:44.100 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj6 2023-09-22T21:21:44.100 DEBUG:boto:Data: 2023-09-22T21:21:44.100 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.100 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:44.101 DEBUG:boto:Port: 8001 2023-09-22T21:21:44.101 DEBUG:boto:Params: {} 2023-09-22T21:21:44.101 DEBUG:boto:Token: None 2023-09-22T21:21:44.101 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/ 2023-09-22T21:21:44.102 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI= 2023-09-22T21:21:44.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI=', 'Content-Length': '0'} 2023-09-22T21:21:44.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fd2f3a185a66310-00650e0568-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.107 DEBUG:boto:b'ffzdqu-46obj61000falseobj62023-09-22T21:21:22.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:44.107 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:44.107 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:44.108 DEBUG:boto:path=/ffzdqu-46/?prefix=obj6 2023-09-22T21:21:44.108 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj6 2023-09-22T21:21:44.108 DEBUG:boto:Method: GET 2023-09-22T21:21:44.108 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj6 2023-09-22T21:21:44.108 DEBUG:boto:Data: 2023-09-22T21:21:44.108 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.108 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:44.108 DEBUG:boto:Port: 8000 2023-09-22T21:21:44.108 DEBUG:boto:Params: {} 2023-09-22T21:21:44.108 DEBUG:boto:Token: None 2023-09-22T21:21:44.109 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/ 2023-09-22T21:21:44.109 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI= 2023-09-22T21:21:44.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI=', 'Content-Length': '0'} 2023-09-22T21:21:44.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bd489f62e0a177c-00650e0568-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.154 DEBUG:boto:b'ffzdqu-46obj61000falseobj62023-09-22T21:21:22.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:44.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-22T21:21:44.155 DEBUG:boto:path=/ffzdqu-46/obj7 2023-09-22T21:21:44.155 DEBUG:boto:auth_path=/ffzdqu-46/obj7 2023-09-22T21:21:44.155 DEBUG:boto:path=/ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.155 DEBUG:boto:auth_path=/ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.156 DEBUG:boto:Method: GET 2023-09-22T21:21:44.156 DEBUG:boto:Path: /ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.156 DEBUG:boto:Data: 2023-09-22T21:21:44.156 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.156 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:44.156 DEBUG:boto:Port: 8001 2023-09-22T21:21:44.156 DEBUG:boto:Params: {} 2023-09-22T21:21:44.156 DEBUG:boto:Token: None 2023-09-22T21:21:44.156 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.157 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zYbk2xe8KdkAtOxEi8kVskLzcwc= 2023-09-22T21:21:44.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zYbk2xe8KdkAtOxEi8kVskLzcwc=', 'Content-Length': '0'} 2023-09-22T21:21:44.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000754ba5b91dad5308-00650e0568-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.206 DEBUG:boto:path=/ffzdqu-46/obj7 2023-09-22T21:21:44.207 DEBUG:boto:auth_path=/ffzdqu-46/obj7 2023-09-22T21:21:44.207 DEBUG:boto:path=/ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.207 DEBUG:boto:auth_path=/ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.207 DEBUG:boto:Method: GET 2023-09-22T21:21:44.207 DEBUG:boto:Path: /ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.207 DEBUG:boto:Data: 2023-09-22T21:21:44.207 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.207 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:44.208 DEBUG:boto:Port: 8000 2023-09-22T21:21:44.208 DEBUG:boto:Params: {} 2023-09-22T21:21:44.208 DEBUG:boto:Token: None 2023-09-22T21:21:44.208 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.208 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zYbk2xe8KdkAtOxEi8kVskLzcwc= 2023-09-22T21:21:44.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zYbk2xe8KdkAtOxEi8kVskLzcwc=', 'Content-Length': '0'} 2023-09-22T21:21:44.211 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026fc7e4cf99e948b-00650e0568-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.211 DEBUG:boto:path=/ffzdqu-46/obj7 2023-09-22T21:21:44.212 DEBUG:boto:auth_path=/ffzdqu-46/obj7 2023-09-22T21:21:44.212 DEBUG:boto:path=/ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.212 DEBUG:boto:auth_path=/ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.212 DEBUG:boto:Method: HEAD 2023-09-22T21:21:44.212 DEBUG:boto:Path: /ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.212 DEBUG:boto:Data: 2023-09-22T21:21:44.212 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.212 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:44.212 DEBUG:boto:Port: 8001 2023-09-22T21:21:44.213 DEBUG:boto:Params: {} 2023-09-22T21:21:44.213 DEBUG:boto:Token: None 2023-09-22T21:21:44.213 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.213 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7aXouCsmSq2+9/swOIOXpSGcFoA= 2023-09-22T21:21:44.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7aXouCsmSq2+9/swOIOXpSGcFoA=', 'Content-Length': '0'} 2023-09-22T21:21:44.216 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007e3fe2372f3ea81-00650e0568-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.216 DEBUG:boto:path=/ffzdqu-46/obj7 2023-09-22T21:21:44.216 DEBUG:boto:auth_path=/ffzdqu-46/obj7 2023-09-22T21:21:44.216 DEBUG:boto:path=/ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.216 DEBUG:boto:auth_path=/ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.217 DEBUG:boto:Method: HEAD 2023-09-22T21:21:44.217 DEBUG:boto:Path: /ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.217 DEBUG:boto:Data: 2023-09-22T21:21:44.217 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.217 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:44.217 DEBUG:boto:Port: 8000 2023-09-22T21:21:44.217 DEBUG:boto:Params: {} 2023-09-22T21:21:44.217 DEBUG:boto:Token: None 2023-09-22T21:21:44.217 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.218 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7aXouCsmSq2+9/swOIOXpSGcFoA= 2023-09-22T21:21:44.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7aXouCsmSq2+9/swOIOXpSGcFoA=', 'Content-Length': '0'} 2023-09-22T21:21:44.220 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063b822bf8d03b862-00650e0568-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-22T21:21:44.221 DEBUG:boto:path=/ffzdqu-46/obj7 2023-09-22T21:21:44.221 DEBUG:boto:auth_path=/ffzdqu-46/obj7 2023-09-22T21:21:44.221 DEBUG:boto:path=/ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.221 DEBUG:boto:auth_path=/ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.221 DEBUG:boto:Method: GET 2023-09-22T21:21:44.221 DEBUG:boto:Path: /ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.221 DEBUG:boto:Data: 2023-09-22T21:21:44.221 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.222 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:44.222 DEBUG:boto:Port: 8001 2023-09-22T21:21:44.222 DEBUG:boto:Params: {} 2023-09-22T21:21:44.222 DEBUG:boto:Token: None 2023-09-22T21:21:44.222 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.222 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zYbk2xe8KdkAtOxEi8kVskLzcwc= 2023-09-22T21:21:44.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zYbk2xe8KdkAtOxEi8kVskLzcwc=', 'Content-Length': '0'} 2023-09-22T21:21:44.225 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ecfce9e0eccd8343-00650e0568-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.266 DEBUG:boto:path=/ffzdqu-46/obj7 2023-09-22T21:21:44.267 DEBUG:boto:auth_path=/ffzdqu-46/obj7 2023-09-22T21:21:44.267 DEBUG:boto:path=/ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.267 DEBUG:boto:auth_path=/ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.267 DEBUG:boto:Method: GET 2023-09-22T21:21:44.268 DEBUG:boto:Path: /ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.268 DEBUG:boto:Data: 2023-09-22T21:21:44.268 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.268 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:44.269 DEBUG:boto:Port: 8000 2023-09-22T21:21:44.269 DEBUG:boto:Params: {} 2023-09-22T21:21:44.269 DEBUG:boto:Token: None 2023-09-22T21:21:44.269 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj7?versionId=null 2023-09-22T21:21:44.270 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zYbk2xe8KdkAtOxEi8kVskLzcwc= 2023-09-22T21:21:44.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zYbk2xe8KdkAtOxEi8kVskLzcwc=', 'Content-Length': '0'} 2023-09-22T21:21:44.273 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ce46cfaddcca2e6-00650e0568-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.273 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:44.273 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:44.273 DEBUG:boto:path=/ffzdqu-46/?prefix=obj7 2023-09-22T21:21:44.273 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj7 2023-09-22T21:21:44.274 DEBUG:boto:Method: GET 2023-09-22T21:21:44.274 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj7 2023-09-22T21:21:44.274 DEBUG:boto:Data: 2023-09-22T21:21:44.274 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.274 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:44.274 DEBUG:boto:Port: 8001 2023-09-22T21:21:44.274 DEBUG:boto:Params: {} 2023-09-22T21:21:44.274 DEBUG:boto:Token: None 2023-09-22T21:21:44.275 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/ 2023-09-22T21:21:44.275 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI= 2023-09-22T21:21:44.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI=', 'Content-Length': '0'} 2023-09-22T21:21:44.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000656a75304abf2024-00650e0568-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.326 DEBUG:boto:b'ffzdqu-46obj71000falseobj72023-09-22T21:21:22.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:44.327 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:44.327 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:44.327 DEBUG:boto:path=/ffzdqu-46/?prefix=obj7 2023-09-22T21:21:44.328 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj7 2023-09-22T21:21:44.328 DEBUG:boto:Method: GET 2023-09-22T21:21:44.328 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj7 2023-09-22T21:21:44.328 DEBUG:boto:Data: 2023-09-22T21:21:44.329 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.329 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:44.329 DEBUG:boto:Port: 8000 2023-09-22T21:21:44.329 DEBUG:boto:Params: {} 2023-09-22T21:21:44.329 DEBUG:boto:Token: None 2023-09-22T21:21:44.330 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/ 2023-09-22T21:21:44.330 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI= 2023-09-22T21:21:44.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI=', 'Content-Length': '0'} 2023-09-22T21:21:44.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c97f11e4450165c-00650e0568-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.335 DEBUG:boto:b'ffzdqu-46obj71000falseobj72023-09-22T21:21:22.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:44.336 DEBUG:boto:path=/ffzdqu-46/obj7 2023-09-22T21:21:44.336 DEBUG:boto:auth_path=/ffzdqu-46/obj7 2023-09-22T21:21:44.336 DEBUG:boto:Method: HEAD 2023-09-22T21:21:44.336 DEBUG:boto:Path: /ffzdqu-46/obj7 2023-09-22T21:21:44.336 DEBUG:boto:Data: 2023-09-22T21:21:44.337 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.337 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:44.337 DEBUG:boto:Port: 8001 2023-09-22T21:21:44.337 DEBUG:boto:Params: {} 2023-09-22T21:21:44.337 DEBUG:boto:Token: None 2023-09-22T21:21:44.338 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj7 2023-09-22T21:21:44.338 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZRohWjVQxcT4jyNG0SNk+AapM40= 2023-09-22T21:21:44.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZRohWjVQxcT4jyNG0SNk+AapM40=', 'Content-Length': '0'} 2023-09-22T21:21:44.341 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ceb731567f36eb07-00650e0568-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.341 DEBUG:boto:path=/ffzdqu-46/obj7 2023-09-22T21:21:44.341 DEBUG:boto:auth_path=/ffzdqu-46/obj7 2023-09-22T21:21:44.342 DEBUG:boto:Method: HEAD 2023-09-22T21:21:44.342 DEBUG:boto:Path: /ffzdqu-46/obj7 2023-09-22T21:21:44.342 DEBUG:boto:Data: 2023-09-22T21:21:44.342 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.343 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:44.343 DEBUG:boto:Port: 8000 2023-09-22T21:21:44.343 DEBUG:boto:Params: {} 2023-09-22T21:21:44.343 DEBUG:boto:Token: None 2023-09-22T21:21:44.343 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj7 2023-09-22T21:21:44.344 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZRohWjVQxcT4jyNG0SNk+AapM40= 2023-09-22T21:21:44.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZRohWjVQxcT4jyNG0SNk+AapM40=', 'Content-Length': '0'} 2023-09-22T21:21:44.346 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f09c39977ec858b-00650e0568-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-22T21:21:44.347 DEBUG:boto:path=/ffzdqu-46/obj7 2023-09-22T21:21:44.347 DEBUG:boto:auth_path=/ffzdqu-46/obj7 2023-09-22T21:21:44.348 DEBUG:boto:Method: GET 2023-09-22T21:21:44.348 DEBUG:boto:Path: /ffzdqu-46/obj7 2023-09-22T21:21:44.348 DEBUG:boto:Data: 2023-09-22T21:21:44.349 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.349 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:44.349 DEBUG:boto:Port: 8001 2023-09-22T21:21:44.349 DEBUG:boto:Params: {} 2023-09-22T21:21:44.350 DEBUG:boto:Token: None 2023-09-22T21:21:44.350 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj7 2023-09-22T21:21:44.350 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BqnFKh4YZAh2YkfasohlVPnCcfo= 2023-09-22T21:21:44.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BqnFKh4YZAh2YkfasohlVPnCcfo=', 'Content-Length': '0'} 2023-09-22T21:21:44.353 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000729364480aed9294-00650e0568-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.394 DEBUG:boto:path=/ffzdqu-46/obj7 2023-09-22T21:21:44.395 DEBUG:boto:auth_path=/ffzdqu-46/obj7 2023-09-22T21:21:44.395 DEBUG:boto:Method: GET 2023-09-22T21:21:44.396 DEBUG:boto:Path: /ffzdqu-46/obj7 2023-09-22T21:21:44.396 DEBUG:boto:Data: 2023-09-22T21:21:44.396 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.396 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:44.397 DEBUG:boto:Port: 8000 2023-09-22T21:21:44.397 DEBUG:boto:Params: {} 2023-09-22T21:21:44.397 DEBUG:boto:Token: None 2023-09-22T21:21:44.397 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj7 2023-09-22T21:21:44.398 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BqnFKh4YZAh2YkfasohlVPnCcfo= 2023-09-22T21:21:44.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BqnFKh4YZAh2YkfasohlVPnCcfo=', 'Content-Length': '0'} 2023-09-22T21:21:44.401 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000184cfbc86b18bc08-00650e0568-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.401 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:44.401 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:44.402 DEBUG:boto:path=/ffzdqu-46/?prefix=obj7 2023-09-22T21:21:44.402 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj7 2023-09-22T21:21:44.402 DEBUG:boto:Method: GET 2023-09-22T21:21:44.402 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj7 2023-09-22T21:21:44.403 DEBUG:boto:Data: 2023-09-22T21:21:44.403 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.403 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:44.403 DEBUG:boto:Port: 8001 2023-09-22T21:21:44.403 DEBUG:boto:Params: {} 2023-09-22T21:21:44.404 DEBUG:boto:Token: None 2023-09-22T21:21:44.404 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/ 2023-09-22T21:21:44.404 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI= 2023-09-22T21:21:44.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI=', 'Content-Length': '0'} 2023-09-22T21:21:44.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b7bfcb444277575-00650e0568-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.409 DEBUG:boto:b'ffzdqu-46obj71000falseobj72023-09-22T21:21:22.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:44.410 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:44.410 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:44.410 DEBUG:boto:path=/ffzdqu-46/?prefix=obj7 2023-09-22T21:21:44.411 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj7 2023-09-22T21:21:44.411 DEBUG:boto:Method: GET 2023-09-22T21:21:44.411 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj7 2023-09-22T21:21:44.411 DEBUG:boto:Data: 2023-09-22T21:21:44.411 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.412 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:44.412 DEBUG:boto:Port: 8000 2023-09-22T21:21:44.412 DEBUG:boto:Params: {} 2023-09-22T21:21:44.412 DEBUG:boto:Token: None 2023-09-22T21:21:44.413 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/ 2023-09-22T21:21:44.413 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI= 2023-09-22T21:21:44.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI=', 'Content-Length': '0'} 2023-09-22T21:21:44.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b2d7f9a03740439-00650e0568-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.418 DEBUG:boto:b'ffzdqu-46obj71000falseobj72023-09-22T21:21:22.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:44.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-22T21:21:44.419 DEBUG:boto:path=/ffzdqu-46/obj8 2023-09-22T21:21:44.419 DEBUG:boto:auth_path=/ffzdqu-46/obj8 2023-09-22T21:21:44.419 DEBUG:boto:path=/ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.419 DEBUG:boto:auth_path=/ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.420 DEBUG:boto:Method: GET 2023-09-22T21:21:44.420 DEBUG:boto:Path: /ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.420 DEBUG:boto:Data: 2023-09-22T21:21:44.420 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.421 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:44.421 DEBUG:boto:Port: 8001 2023-09-22T21:21:44.421 DEBUG:boto:Params: {} 2023-09-22T21:21:44.421 DEBUG:boto:Token: None 2023-09-22T21:21:44.421 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.422 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GcALftbkACxD0pG2bnrNHTMAPKY= 2023-09-22T21:21:44.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GcALftbkACxD0pG2bnrNHTMAPKY=', 'Content-Length': '0'} 2023-09-22T21:21:44.425 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007020dce53a8ac473-00650e0568-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.466 DEBUG:boto:path=/ffzdqu-46/obj8 2023-09-22T21:21:44.467 DEBUG:boto:auth_path=/ffzdqu-46/obj8 2023-09-22T21:21:44.467 DEBUG:boto:path=/ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.467 DEBUG:boto:auth_path=/ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.467 DEBUG:boto:Method: GET 2023-09-22T21:21:44.467 DEBUG:boto:Path: /ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.467 DEBUG:boto:Data: 2023-09-22T21:21:44.467 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.467 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:44.467 DEBUG:boto:Port: 8000 2023-09-22T21:21:44.468 DEBUG:boto:Params: {} 2023-09-22T21:21:44.468 DEBUG:boto:Token: None 2023-09-22T21:21:44.468 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.468 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GcALftbkACxD0pG2bnrNHTMAPKY= 2023-09-22T21:21:44.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GcALftbkACxD0pG2bnrNHTMAPKY=', 'Content-Length': '0'} 2023-09-22T21:21:44.471 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7577301df245fe4-00650e0568-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.472 DEBUG:boto:path=/ffzdqu-46/obj8 2023-09-22T21:21:44.472 DEBUG:boto:auth_path=/ffzdqu-46/obj8 2023-09-22T21:21:44.472 DEBUG:boto:path=/ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.472 DEBUG:boto:auth_path=/ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.472 DEBUG:boto:Method: HEAD 2023-09-22T21:21:44.472 DEBUG:boto:Path: /ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.472 DEBUG:boto:Data: 2023-09-22T21:21:44.472 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.472 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:44.473 DEBUG:boto:Port: 8001 2023-09-22T21:21:44.473 DEBUG:boto:Params: {} 2023-09-22T21:21:44.473 DEBUG:boto:Token: None 2023-09-22T21:21:44.473 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.473 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vNxibDSuYTBuGabJz+PGCuMQ7TA= 2023-09-22T21:21:44.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vNxibDSuYTBuGabJz+PGCuMQ7TA=', 'Content-Length': '0'} 2023-09-22T21:21:44.476 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035590cdefd548f7a-00650e0568-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.476 DEBUG:boto:path=/ffzdqu-46/obj8 2023-09-22T21:21:44.477 DEBUG:boto:auth_path=/ffzdqu-46/obj8 2023-09-22T21:21:44.477 DEBUG:boto:path=/ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.477 DEBUG:boto:auth_path=/ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.477 DEBUG:boto:Method: HEAD 2023-09-22T21:21:44.477 DEBUG:boto:Path: /ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.477 DEBUG:boto:Data: 2023-09-22T21:21:44.477 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.477 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:44.477 DEBUG:boto:Port: 8000 2023-09-22T21:21:44.477 DEBUG:boto:Params: {} 2023-09-22T21:21:44.477 DEBUG:boto:Token: None 2023-09-22T21:21:44.478 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.478 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vNxibDSuYTBuGabJz+PGCuMQ7TA= 2023-09-22T21:21:44.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vNxibDSuYTBuGabJz+PGCuMQ7TA=', 'Content-Length': '0'} 2023-09-22T21:21:44.480 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062eac8d63ca53d90-00650e0568-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-22T21:21:44.481 DEBUG:boto:path=/ffzdqu-46/obj8 2023-09-22T21:21:44.481 DEBUG:boto:auth_path=/ffzdqu-46/obj8 2023-09-22T21:21:44.481 DEBUG:boto:path=/ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.481 DEBUG:boto:auth_path=/ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.481 DEBUG:boto:Method: GET 2023-09-22T21:21:44.481 DEBUG:boto:Path: /ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.482 DEBUG:boto:Data: 2023-09-22T21:21:44.482 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.482 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:44.482 DEBUG:boto:Port: 8001 2023-09-22T21:21:44.482 DEBUG:boto:Params: {} 2023-09-22T21:21:44.482 DEBUG:boto:Token: None 2023-09-22T21:21:44.482 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.482 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GcALftbkACxD0pG2bnrNHTMAPKY= 2023-09-22T21:21:44.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GcALftbkACxD0pG2bnrNHTMAPKY=', 'Content-Length': '0'} 2023-09-22T21:21:44.485 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d184c2b480060d13-00650e0568-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.526 DEBUG:boto:path=/ffzdqu-46/obj8 2023-09-22T21:21:44.526 DEBUG:boto:auth_path=/ffzdqu-46/obj8 2023-09-22T21:21:44.526 DEBUG:boto:path=/ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.527 DEBUG:boto:auth_path=/ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.527 DEBUG:boto:Method: GET 2023-09-22T21:21:44.527 DEBUG:boto:Path: /ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.527 DEBUG:boto:Data: 2023-09-22T21:21:44.527 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.527 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:44.527 DEBUG:boto:Port: 8000 2023-09-22T21:21:44.527 DEBUG:boto:Params: {} 2023-09-22T21:21:44.527 DEBUG:boto:Token: None 2023-09-22T21:21:44.528 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj8?versionId=null 2023-09-22T21:21:44.528 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GcALftbkACxD0pG2bnrNHTMAPKY= 2023-09-22T21:21:44.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GcALftbkACxD0pG2bnrNHTMAPKY=', 'Content-Length': '0'} 2023-09-22T21:21:44.531 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a6af598de98a5f9-00650e0568-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.531 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:44.531 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:44.531 DEBUG:boto:path=/ffzdqu-46/?prefix=obj8 2023-09-22T21:21:44.531 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj8 2023-09-22T21:21:44.531 DEBUG:boto:Method: GET 2023-09-22T21:21:44.531 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj8 2023-09-22T21:21:44.532 DEBUG:boto:Data: 2023-09-22T21:21:44.532 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.532 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:44.532 DEBUG:boto:Port: 8001 2023-09-22T21:21:44.532 DEBUG:boto:Params: {} 2023-09-22T21:21:44.532 DEBUG:boto:Token: None 2023-09-22T21:21:44.532 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/ 2023-09-22T21:21:44.532 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI= 2023-09-22T21:21:44.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI=', 'Content-Length': '0'} 2023-09-22T21:21:44.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff1233730b9ad1a9-00650e0568-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.582 DEBUG:boto:b'ffzdqu-46obj81000falseobj82023-09-22T21:21:22.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:44.583 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:44.583 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:44.583 DEBUG:boto:path=/ffzdqu-46/?prefix=obj8 2023-09-22T21:21:44.584 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj8 2023-09-22T21:21:44.584 DEBUG:boto:Method: GET 2023-09-22T21:21:44.584 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj8 2023-09-22T21:21:44.584 DEBUG:boto:Data: 2023-09-22T21:21:44.584 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.585 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:44.585 DEBUG:boto:Port: 8000 2023-09-22T21:21:44.585 DEBUG:boto:Params: {} 2023-09-22T21:21:44.585 DEBUG:boto:Token: None 2023-09-22T21:21:44.586 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/ 2023-09-22T21:21:44.586 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI= 2023-09-22T21:21:44.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI=', 'Content-Length': '0'} 2023-09-22T21:21:44.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002adcaece513ea0ce-00650e0568-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.591 DEBUG:boto:b'ffzdqu-46obj81000falseobj82023-09-22T21:21:22.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:44.592 DEBUG:boto:path=/ffzdqu-46/obj8 2023-09-22T21:21:44.592 DEBUG:boto:auth_path=/ffzdqu-46/obj8 2023-09-22T21:21:44.592 DEBUG:boto:Method: HEAD 2023-09-22T21:21:44.592 DEBUG:boto:Path: /ffzdqu-46/obj8 2023-09-22T21:21:44.592 DEBUG:boto:Data: 2023-09-22T21:21:44.593 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.593 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:44.593 DEBUG:boto:Port: 8001 2023-09-22T21:21:44.593 DEBUG:boto:Params: {} 2023-09-22T21:21:44.593 DEBUG:boto:Token: None 2023-09-22T21:21:44.594 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj8 2023-09-22T21:21:44.594 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tFu0D2Jo2/pAZDwgfNw5xfGwRg8= 2023-09-22T21:21:44.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tFu0D2Jo2/pAZDwgfNw5xfGwRg8=', 'Content-Length': '0'} 2023-09-22T21:21:44.597 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000493c844ca332b554-00650e0568-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.597 DEBUG:boto:path=/ffzdqu-46/obj8 2023-09-22T21:21:44.597 DEBUG:boto:auth_path=/ffzdqu-46/obj8 2023-09-22T21:21:44.597 DEBUG:boto:Method: HEAD 2023-09-22T21:21:44.597 DEBUG:boto:Path: /ffzdqu-46/obj8 2023-09-22T21:21:44.597 DEBUG:boto:Data: 2023-09-22T21:21:44.598 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.598 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:44.598 DEBUG:boto:Port: 8000 2023-09-22T21:21:44.598 DEBUG:boto:Params: {} 2023-09-22T21:21:44.598 DEBUG:boto:Token: None 2023-09-22T21:21:44.598 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj8 2023-09-22T21:21:44.598 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tFu0D2Jo2/pAZDwgfNw5xfGwRg8= 2023-09-22T21:21:44.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tFu0D2Jo2/pAZDwgfNw5xfGwRg8=', 'Content-Length': '0'} 2023-09-22T21:21:44.601 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022a8e70af30dcf61-00650e0568-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-22T21:21:44.602 DEBUG:boto:path=/ffzdqu-46/obj8 2023-09-22T21:21:44.602 DEBUG:boto:auth_path=/ffzdqu-46/obj8 2023-09-22T21:21:44.602 DEBUG:boto:Method: GET 2023-09-22T21:21:44.602 DEBUG:boto:Path: /ffzdqu-46/obj8 2023-09-22T21:21:44.602 DEBUG:boto:Data: 2023-09-22T21:21:44.603 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.603 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:44.603 DEBUG:boto:Port: 8001 2023-09-22T21:21:44.603 DEBUG:boto:Params: {} 2023-09-22T21:21:44.604 DEBUG:boto:Token: None 2023-09-22T21:21:44.604 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj8 2023-09-22T21:21:44.604 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4NqDQlGP+ZQj+srgXaF3+z0o9XM= 2023-09-22T21:21:44.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4NqDQlGP+ZQj+srgXaF3+z0o9XM=', 'Content-Length': '0'} 2023-09-22T21:21:44.607 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-46:f132bdb9-20a7-4073-a932-8f99da20691e.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ce527133d3cb5d6-00650e0568-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.654 DEBUG:boto:path=/ffzdqu-46/obj8 2023-09-22T21:21:44.655 DEBUG:boto:auth_path=/ffzdqu-46/obj8 2023-09-22T21:21:44.655 DEBUG:boto:Method: GET 2023-09-22T21:21:44.655 DEBUG:boto:Path: /ffzdqu-46/obj8 2023-09-22T21:21:44.655 DEBUG:boto:Data: 2023-09-22T21:21:44.655 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.655 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:44.655 DEBUG:boto:Port: 8000 2023-09-22T21:21:44.655 DEBUG:boto:Params: {} 2023-09-22T21:21:44.656 DEBUG:boto:Token: None 2023-09-22T21:21:44.656 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/obj8 2023-09-22T21:21:44.656 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4NqDQlGP+ZQj+srgXaF3+z0o9XM= 2023-09-22T21:21:44.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4NqDQlGP+ZQj+srgXaF3+z0o9XM=', 'Content-Length': '0'} 2023-09-22T21:21:44.659 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006541d4ce74eedf4c-00650e0568-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.659 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:44.659 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:44.660 DEBUG:boto:path=/ffzdqu-46/?prefix=obj8 2023-09-22T21:21:44.660 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj8 2023-09-22T21:21:44.660 DEBUG:boto:Method: GET 2023-09-22T21:21:44.660 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj8 2023-09-22T21:21:44.660 DEBUG:boto:Data: 2023-09-22T21:21:44.660 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.660 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:21:44.660 DEBUG:boto:Port: 8001 2023-09-22T21:21:44.660 DEBUG:boto:Params: {} 2023-09-22T21:21:44.660 DEBUG:boto:Token: None 2023-09-22T21:21:44.661 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/ 2023-09-22T21:21:44.661 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI= 2023-09-22T21:21:44.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI=', 'Content-Length': '0'} 2023-09-22T21:21:44.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b8863160021a6b8-00650e0568-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.666 DEBUG:boto:b'ffzdqu-46obj81000falseobj82023-09-22T21:21:22.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:44.666 DEBUG:boto:path=/ffzdqu-46/ 2023-09-22T21:21:44.667 DEBUG:boto:auth_path=/ffzdqu-46/ 2023-09-22T21:21:44.667 DEBUG:boto:path=/ffzdqu-46/?prefix=obj8 2023-09-22T21:21:44.667 DEBUG:boto:auth_path=/ffzdqu-46/?prefix=obj8 2023-09-22T21:21:44.667 DEBUG:boto:Method: GET 2023-09-22T21:21:44.668 DEBUG:boto:Path: /ffzdqu-46/?prefix=obj8 2023-09-22T21:21:44.668 DEBUG:boto:Data: 2023-09-22T21:21:44.668 DEBUG:boto:Headers: {} 2023-09-22T21:21:44.668 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:21:44.668 DEBUG:boto:Port: 8000 2023-09-22T21:21:44.668 DEBUG:boto:Params: {} 2023-09-22T21:21:44.669 DEBUG:boto:Token: None 2023-09-22T21:21:44.669 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:21:44 GMT /ffzdqu-46/ 2023-09-22T21:21:44.669 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI= 2023-09-22T21:21:44.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:21:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h80k3WK/imG1I4a+Sz1ClaziFYI=', 'Content-Length': '0'} 2023-09-22T21:21:44.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbb56f2c1a58b618-00650e0568-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:21:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:21:44.674 DEBUG:boto:b'ffzdqu-46obj81000falseobj82023-09-22T21:21:22.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:21:44.674 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-46 zones={a2, a1} 2023-09-22T21:21:44.675 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-47 2023-09-22T21:21:44.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:21:44.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:21:44.675 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-47 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:21:44.726 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:21:44.726 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:21:44.769 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:21:44.771+0000 7f96c0080a40 1 waiting for incremental sync to catch up: 2023-09-22T21:21:44.769 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000003.34.1, 00000000002.41.1, 00000000004.33.6, 00000000002.28.1, 00000000004.31.6, 00000000002.14.1, 00000000002.21.1, 00000000003.24.1, 00000000002.10.1, 00000000002.18.1, 00000000002.29.1] 2023-09-22T21:21:44.770 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000004.38.6, 00000000002.41.1, 00000000004.33.6, 00000000002.28.1, 00000000004.31.6, 00000000002.14.1, 00000000002.21.1, 00000000004.26.6, 00000000002.10.1, 00000000002.18.1, 00000000002.29.1] 2023-09-22T21:21:49.771 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:21:49.771+0000 7f96c0080a40 1 waiting for incremental sync to catch up: 2023-09-22T21:21:49.772 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000003.34.1, 00000000002.41.1, 00000000004.33.6, 00000000002.28.1, 00000000004.31.6, 00000000002.14.1, 00000000002.21.1, 00000000003.24.1, 00000000002.10.1, 00000000002.18.1, 00000000002.29.1] 2023-09-22T21:21:49.772 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000004.38.6, 00000000002.41.1, 00000000004.33.6, 00000000002.28.1, 00000000004.31.6, 00000000002.14.1, 00000000002.21.1, 00000000004.26.6, 00000000002.10.1, 00000000002.18.1, 00000000002.29.1] 2023-09-22T21:21:54.774 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:21:54.775+0000 7f96c0080a40 1 waiting for incremental sync to catch up: 2023-09-22T21:21:54.774 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000003.34.1, 00000000002.41.1, 00000000004.33.6, 00000000002.28.1, 00000000004.31.6, 00000000002.14.1, 00000000002.21.1, 00000000003.24.1, 00000000002.10.1, 00000000002.18.1, 00000000002.29.1] 2023-09-22T21:21:54.774 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000004.38.6, 00000000002.41.1, 00000000004.33.6, 00000000002.28.1, 00000000004.31.6, 00000000002.14.1, 00000000002.21.1, 00000000004.26.6, 00000000002.10.1, 00000000002.18.1, 00000000002.29.1] 2023-09-22T21:21:59.776 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:21:59.775+0000 7f96c0080a40 1 waiting for incremental sync to catch up: 2023-09-22T21:21:59.776 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000003.34.1, 00000000002.41.1, 00000000004.33.6, 00000000002.28.1, 00000000004.31.6, 00000000002.14.1, 00000000002.21.1, 00000000004.26.6, 00000000002.10.1, 00000000002.18.1, 00000000002.29.1] 2023-09-22T21:21:59.777 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000004.38.6, 00000000002.41.1, 00000000004.33.6, 00000000002.28.1, 00000000004.31.6, 00000000002.14.1, 00000000002.21.1, 00000000004.26.6, 00000000002.10.1, 00000000002.18.1, 00000000002.29.1] 2023-09-22T21:22:04.778 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:04.779+0000 7f96c0080a40 1 bucket sync caught up with source: 2023-09-22T21:22:04.779 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000004.38.6, 00000000002.41.1, 00000000004.33.6, 00000000002.28.1, 00000000004.31.6, 00000000002.14.1, 00000000002.21.1, 00000000004.26.6, 00000000002.10.1, 00000000002.18.1, 00000000002.29.1] 2023-09-22T21:22:04.779 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000004.38.6, 00000000002.41.1, 00000000004.33.6, 00000000002.28.1, 00000000004.31.6, 00000000002.14.1, 00000000002.21.1, 00000000004.26.6, 00000000002.10.1, 00000000002.18.1, 00000000002.29.1] 2023-09-22T21:22:04.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:04.779+0000 7f96c0080a40 0 bucket checkpoint complete 2023-09-22T21:22:04.818 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-47 2023-09-22T21:22:04.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:22:04.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:22:04.819 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-47 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:22:04.868 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:04.868 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:04.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:04.911+0000 7f6c5f5bca40 1 bucket sync caught up with source: 2023-09-22T21:22:04.911 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.215.6, , 00000000002.309.6, , 00000000002.307.6, , , 00000000002.217.6, , , ] 2023-09-22T21:22:04.911 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.215.6, , 00000000002.309.6, , 00000000002.307.6, , , 00000000002.217.6, , , ] 2023-09-22T21:22:04.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:04.911+0000 7f6c5f5bca40 0 bucket checkpoint complete 2023-09-22T21:22:04.929 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-47 zones={a2, a1} 2023-09-22T21:22:04.929 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:04.930 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:04.930 DEBUG:boto:Method: HEAD 2023-09-22T21:22:04.930 DEBUG:boto:Path: /ffzdqu-47/ 2023-09-22T21:22:04.930 DEBUG:boto:Data: 2023-09-22T21:22:04.930 DEBUG:boto:Headers: {} 2023-09-22T21:22:04.930 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:04.930 DEBUG:boto:Port: 8001 2023-09-22T21:22:04.930 DEBUG:boto:Params: {} 2023-09-22T21:22:04.930 DEBUG:boto:Token: None 2023-09-22T21:22:04.931 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:04 GMT /ffzdqu-47/ 2023-09-22T21:22:04.931 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lIzlfuF/kRBl0ILwTg5rB/5/qaQ= 2023-09-22T21:22:04.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lIzlfuF/kRBl0ILwTg5rB/5/qaQ=', 'Content-Length': '0'} 2023-09-22T21:22:04.933 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042b557a71447164b-00650e057c-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:04.934 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:04.934 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:04.934 DEBUG:boto:Method: HEAD 2023-09-22T21:22:04.934 DEBUG:boto:Path: /ffzdqu-47/ 2023-09-22T21:22:04.935 DEBUG:boto:Data: 2023-09-22T21:22:04.935 DEBUG:boto:Headers: {} 2023-09-22T21:22:04.935 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:04.935 DEBUG:boto:Port: 8000 2023-09-22T21:22:04.935 DEBUG:boto:Params: {} 2023-09-22T21:22:04.935 DEBUG:boto:Token: None 2023-09-22T21:22:04.935 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:04 GMT /ffzdqu-47/ 2023-09-22T21:22:04.935 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lIzlfuF/kRBl0ILwTg5rB/5/qaQ= 2023-09-22T21:22:04.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lIzlfuF/kRBl0ILwTg5rB/5/qaQ=', 'Content-Length': '0'} 2023-09-22T21:22:04.938 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b97ea90e6f221526-00650e057c-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:04.938 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:22:04.938 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:04.938 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:04.938 DEBUG:boto:path=/ffzdqu-47/?versions&max-keys=999 2023-09-22T21:22:04.938 DEBUG:boto:auth_path=/ffzdqu-47/?versions&max-keys=999 2023-09-22T21:22:04.938 DEBUG:boto:Method: GET 2023-09-22T21:22:04.939 DEBUG:boto:Path: /ffzdqu-47/?versions&max-keys=999 2023-09-22T21:22:04.939 DEBUG:boto:Data: 2023-09-22T21:22:04.939 DEBUG:boto:Headers: {} 2023-09-22T21:22:04.939 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:04.939 DEBUG:boto:Port: 8001 2023-09-22T21:22:04.939 DEBUG:boto:Params: {} 2023-09-22T21:22:04.939 DEBUG:boto:Token: None 2023-09-22T21:22:04.939 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:04 GMT /ffzdqu-47/?versions 2023-09-22T21:22:04.939 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Fxz30ttNhgT6gbbW0swrvSufmnc= 2023-09-22T21:22:04.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Fxz30ttNhgT6gbbW0swrvSufmnc=', 'Content-Length': '0'} 2023-09-22T21:22:04.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000668e7e870ff15976-00650e057c-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:04.942 DEBUG:boto:b'ffzdqu-47999falseobj1nulltrue2023-09-22T21:20:38.912Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:20:38.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:20:38.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:20:38.999Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-22T21:21:22.372Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-22T21:21:22.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-22T21:21:22.443Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-22T21:21:22.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:04.942 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-22T21:22:04.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-22T21:22:04.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-22T21:22:04.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-22T21:22:04.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-09-22T21:22:04.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-09-22T21:22:04.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-09-22T21:22:04.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-09-22T21:22:04.943 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:22:04.943 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:04.944 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:04.944 DEBUG:boto:path=/ffzdqu-47/?versions&max-keys=999 2023-09-22T21:22:04.944 DEBUG:boto:auth_path=/ffzdqu-47/?versions&max-keys=999 2023-09-22T21:22:04.944 DEBUG:boto:Method: GET 2023-09-22T21:22:04.944 DEBUG:boto:Path: /ffzdqu-47/?versions&max-keys=999 2023-09-22T21:22:04.944 DEBUG:boto:Data: 2023-09-22T21:22:04.944 DEBUG:boto:Headers: {} 2023-09-22T21:22:04.944 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:04.944 DEBUG:boto:Port: 8000 2023-09-22T21:22:04.944 DEBUG:boto:Params: {} 2023-09-22T21:22:04.945 DEBUG:boto:Token: None 2023-09-22T21:22:04.945 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:04 GMT /ffzdqu-47/?versions 2023-09-22T21:22:04.945 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Fxz30ttNhgT6gbbW0swrvSufmnc= 2023-09-22T21:22:04.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Fxz30ttNhgT6gbbW0swrvSufmnc=', 'Content-Length': '0'} 2023-09-22T21:22:04.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000276113590ef6ce6a-00650e057c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:04.948 DEBUG:boto:b'ffzdqu-47999falseobj1nulltrue2023-09-22T21:20:38.912Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:20:38.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:20:38.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:20:38.999Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-22T21:21:22.372Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-22T21:21:22.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-22T21:21:22.443Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-22T21:21:22.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:04.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-22T21:22:04.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-22T21:22:04.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-22T21:22:04.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-22T21:22:04.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-09-22T21:22:04.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-09-22T21:22:04.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-09-22T21:22:04.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-09-22T21:22:04.949 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:04.949 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:04.949 DEBUG:boto:path=/ffzdqu-47/?versions&max-keys=999 2023-09-22T21:22:04.949 DEBUG:boto:auth_path=/ffzdqu-47/?versions&max-keys=999 2023-09-22T21:22:04.949 DEBUG:boto:Method: GET 2023-09-22T21:22:04.950 DEBUG:boto:Path: /ffzdqu-47/?versions&max-keys=999 2023-09-22T21:22:04.950 DEBUG:boto:Data: 2023-09-22T21:22:04.950 DEBUG:boto:Headers: {} 2023-09-22T21:22:04.950 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:04.950 DEBUG:boto:Port: 8001 2023-09-22T21:22:04.950 DEBUG:boto:Params: {} 2023-09-22T21:22:04.950 DEBUG:boto:Token: None 2023-09-22T21:22:04.950 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:04 GMT /ffzdqu-47/?versions 2023-09-22T21:22:04.950 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Fxz30ttNhgT6gbbW0swrvSufmnc= 2023-09-22T21:22:04.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Fxz30ttNhgT6gbbW0swrvSufmnc=', 'Content-Length': '0'} 2023-09-22T21:22:04.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e810f75cc37f9669-00650e057c-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:04.953 DEBUG:boto:b'ffzdqu-47999falseobj1nulltrue2023-09-22T21:20:38.912Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:20:38.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:20:38.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:20:38.999Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-22T21:21:22.372Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-22T21:21:22.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-22T21:21:22.443Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-22T21:21:22.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:04.954 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:04.954 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:04.954 DEBUG:boto:path=/ffzdqu-47/?versions&max-keys=999 2023-09-22T21:22:04.955 DEBUG:boto:auth_path=/ffzdqu-47/?versions&max-keys=999 2023-09-22T21:22:04.955 DEBUG:boto:Method: GET 2023-09-22T21:22:04.955 DEBUG:boto:Path: /ffzdqu-47/?versions&max-keys=999 2023-09-22T21:22:04.955 DEBUG:boto:Data: 2023-09-22T21:22:04.955 DEBUG:boto:Headers: {} 2023-09-22T21:22:04.955 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:04.956 DEBUG:boto:Port: 8000 2023-09-22T21:22:04.956 DEBUG:boto:Params: {} 2023-09-22T21:22:04.956 DEBUG:boto:Token: None 2023-09-22T21:22:04.956 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:04 GMT /ffzdqu-47/?versions 2023-09-22T21:22:04.956 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Fxz30ttNhgT6gbbW0swrvSufmnc= 2023-09-22T21:22:04.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Fxz30ttNhgT6gbbW0swrvSufmnc=', 'Content-Length': '0'} 2023-09-22T21:22:04.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb8f20a75518eaad-00650e057c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:04.959 DEBUG:boto:b'ffzdqu-47999falseobj1nulltrue2023-09-22T21:20:38.912Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:20:38.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:20:38.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:20:38.999Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-22T21:21:22.372Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-22T21:21:22.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-22T21:21:22.443Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-22T21:21:22.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:04.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:22:04.960 DEBUG:boto:path=/ffzdqu-47/obj1 2023-09-22T21:22:04.960 DEBUG:boto:auth_path=/ffzdqu-47/obj1 2023-09-22T21:22:04.960 DEBUG:boto:path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:04.960 DEBUG:boto:auth_path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:04.960 DEBUG:boto:Method: GET 2023-09-22T21:22:04.960 DEBUG:boto:Path: /ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:04.960 DEBUG:boto:Data: 2023-09-22T21:22:04.960 DEBUG:boto:Headers: {} 2023-09-22T21:22:04.961 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:04.961 DEBUG:boto:Port: 8001 2023-09-22T21:22:04.961 DEBUG:boto:Params: {} 2023-09-22T21:22:04.961 DEBUG:boto:Token: None 2023-09-22T21:22:04.961 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:04 GMT /ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:04.961 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BHu5gJDoRk28TZdD2w5GOfpj/iQ= 2023-09-22T21:22:04.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BHu5gJDoRk28TZdD2w5GOfpj/iQ=', 'Content-Length': '0'} 2023-09-22T21:22:04.963 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b7c21ed5740b1c5-00650e057c-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:04.963 DEBUG:boto:path=/ffzdqu-47/obj1 2023-09-22T21:22:04.963 DEBUG:boto:auth_path=/ffzdqu-47/obj1 2023-09-22T21:22:04.963 DEBUG:boto:path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:04.963 DEBUG:boto:auth_path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:04.964 DEBUG:boto:Method: GET 2023-09-22T21:22:04.964 DEBUG:boto:Path: /ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:04.964 DEBUG:boto:Data: 2023-09-22T21:22:04.964 DEBUG:boto:Headers: {} 2023-09-22T21:22:04.964 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:04.964 DEBUG:boto:Port: 8000 2023-09-22T21:22:04.964 DEBUG:boto:Params: {} 2023-09-22T21:22:04.964 DEBUG:boto:Token: None 2023-09-22T21:22:04.965 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:04 GMT /ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:04.965 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BHu5gJDoRk28TZdD2w5GOfpj/iQ= 2023-09-22T21:22:04.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BHu5gJDoRk28TZdD2w5GOfpj/iQ=', 'Content-Length': '0'} 2023-09-22T21:22:04.966 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000636258838dba8eb2-00650e057c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.014 DEBUG:boto:path=/ffzdqu-47/obj1 2023-09-22T21:22:05.015 DEBUG:boto:auth_path=/ffzdqu-47/obj1 2023-09-22T21:22:05.015 DEBUG:boto:path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:05.015 DEBUG:boto:auth_path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:05.015 DEBUG:boto:Method: HEAD 2023-09-22T21:22:05.015 DEBUG:boto:Path: /ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:05.015 DEBUG:boto:Data: 2023-09-22T21:22:05.015 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.015 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.015 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.016 DEBUG:boto:Params: {} 2023-09-22T21:22:05.016 DEBUG:boto:Token: None 2023-09-22T21:22:05.016 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:05.016 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/g20JqpgRypUvJ4lYCPipHnhOxo= 2023-09-22T21:22:05.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/g20JqpgRypUvJ4lYCPipHnhOxo=', 'Content-Length': '0'} 2023-09-22T21:22:05.019 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015225a1930096f33-00650e057d-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.019 DEBUG:boto:path=/ffzdqu-47/obj1 2023-09-22T21:22:05.019 DEBUG:boto:auth_path=/ffzdqu-47/obj1 2023-09-22T21:22:05.019 DEBUG:boto:path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:05.020 DEBUG:boto:auth_path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:05.020 DEBUG:boto:Method: HEAD 2023-09-22T21:22:05.020 DEBUG:boto:Path: /ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:05.020 DEBUG:boto:Data: 2023-09-22T21:22:05.020 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.020 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.020 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.020 DEBUG:boto:Params: {} 2023-09-22T21:22:05.020 DEBUG:boto:Token: None 2023-09-22T21:22:05.020 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:05.021 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/g20JqpgRypUvJ4lYCPipHnhOxo= 2023-09-22T21:22:05.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/g20JqpgRypUvJ4lYCPipHnhOxo=', 'Content-Length': '0'} 2023-09-22T21:22:05.023 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089b42560fd35c9d4-00650e057d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:22:05.024 DEBUG:boto:path=/ffzdqu-47/obj1 2023-09-22T21:22:05.024 DEBUG:boto:auth_path=/ffzdqu-47/obj1 2023-09-22T21:22:05.024 DEBUG:boto:path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:05.024 DEBUG:boto:auth_path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:05.025 DEBUG:boto:Method: GET 2023-09-22T21:22:05.025 DEBUG:boto:Path: /ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:05.025 DEBUG:boto:Data: 2023-09-22T21:22:05.025 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.026 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.026 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.026 DEBUG:boto:Params: {} 2023-09-22T21:22:05.026 DEBUG:boto:Token: None 2023-09-22T21:22:05.026 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:05.027 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H87VrHTwGJs3Y52++YndIXi/yrY= 2023-09-22T21:22:05.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H87VrHTwGJs3Y52++YndIXi/yrY=', 'Content-Length': '0'} 2023-09-22T21:22:05.029 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5b554c090cc6cd7-00650e057d-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.074 DEBUG:boto:path=/ffzdqu-47/obj1 2023-09-22T21:22:05.074 DEBUG:boto:auth_path=/ffzdqu-47/obj1 2023-09-22T21:22:05.075 DEBUG:boto:path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:05.075 DEBUG:boto:auth_path=/ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:05.075 DEBUG:boto:Method: GET 2023-09-22T21:22:05.075 DEBUG:boto:Path: /ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:05.075 DEBUG:boto:Data: 2023-09-22T21:22:05.075 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.075 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.075 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.075 DEBUG:boto:Params: {} 2023-09-22T21:22:05.076 DEBUG:boto:Token: None 2023-09-22T21:22:05.076 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj1?versionId=null 2023-09-22T21:22:05.076 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H87VrHTwGJs3Y52++YndIXi/yrY= 2023-09-22T21:22:05.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H87VrHTwGJs3Y52++YndIXi/yrY=', 'Content-Length': '0'} 2023-09-22T21:22:05.079 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000696587d06bdaa81f-00650e057d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.079 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:05.080 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:05.080 DEBUG:boto:path=/ffzdqu-47/?prefix=obj1 2023-09-22T21:22:05.080 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj1 2023-09-22T21:22:05.080 DEBUG:boto:Method: GET 2023-09-22T21:22:05.080 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj1 2023-09-22T21:22:05.081 DEBUG:boto:Data: 2023-09-22T21:22:05.081 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.081 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.081 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.081 DEBUG:boto:Params: {} 2023-09-22T21:22:05.082 DEBUG:boto:Token: None 2023-09-22T21:22:05.082 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/ 2023-09-22T21:22:05.082 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk= 2023-09-22T21:22:05.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk=', 'Content-Length': '0'} 2023-09-22T21:22:05.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021c2c543ed3f35ef-00650e057d-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.087 DEBUG:boto:b'ffzdqu-47obj11000falseobj12023-09-22T21:20:38.912Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:05.087 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:05.088 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:05.088 DEBUG:boto:path=/ffzdqu-47/?prefix=obj1 2023-09-22T21:22:05.088 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj1 2023-09-22T21:22:05.088 DEBUG:boto:Method: GET 2023-09-22T21:22:05.088 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj1 2023-09-22T21:22:05.089 DEBUG:boto:Data: 2023-09-22T21:22:05.089 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.089 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.089 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.089 DEBUG:boto:Params: {} 2023-09-22T21:22:05.090 DEBUG:boto:Token: None 2023-09-22T21:22:05.090 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/ 2023-09-22T21:22:05.090 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk= 2023-09-22T21:22:05.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk=', 'Content-Length': '0'} 2023-09-22T21:22:05.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5facab2d03aa689-00650e057d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.095 DEBUG:boto:b'ffzdqu-47obj11000falseobj12023-09-22T21:20:38.912Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:05.096 DEBUG:boto:path=/ffzdqu-47/obj1 2023-09-22T21:22:05.096 DEBUG:boto:auth_path=/ffzdqu-47/obj1 2023-09-22T21:22:05.096 DEBUG:boto:Method: HEAD 2023-09-22T21:22:05.096 DEBUG:boto:Path: /ffzdqu-47/obj1 2023-09-22T21:22:05.096 DEBUG:boto:Data: 2023-09-22T21:22:05.096 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.097 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.097 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.097 DEBUG:boto:Params: {} 2023-09-22T21:22:05.097 DEBUG:boto:Token: None 2023-09-22T21:22:05.097 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj1 2023-09-22T21:22:05.098 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BvWAbq6rmGZXtNR04KJffMWpfzM= 2023-09-22T21:22:05.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BvWAbq6rmGZXtNR04KJffMWpfzM=', 'Content-Length': '0'} 2023-09-22T21:22:05.100 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cd00327c5bec0c4-00650e057d-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.101 DEBUG:boto:path=/ffzdqu-47/obj1 2023-09-22T21:22:05.101 DEBUG:boto:auth_path=/ffzdqu-47/obj1 2023-09-22T21:22:05.101 DEBUG:boto:Method: HEAD 2023-09-22T21:22:05.101 DEBUG:boto:Path: /ffzdqu-47/obj1 2023-09-22T21:22:05.101 DEBUG:boto:Data: 2023-09-22T21:22:05.102 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.102 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.102 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.102 DEBUG:boto:Params: {} 2023-09-22T21:22:05.103 DEBUG:boto:Token: None 2023-09-22T21:22:05.103 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj1 2023-09-22T21:22:05.103 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BvWAbq6rmGZXtNR04KJffMWpfzM= 2023-09-22T21:22:05.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BvWAbq6rmGZXtNR04KJffMWpfzM=', 'Content-Length': '0'} 2023-09-22T21:22:05.106 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001918e8a9804258e3-00650e057d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:22:05.107 DEBUG:boto:path=/ffzdqu-47/obj1 2023-09-22T21:22:05.107 DEBUG:boto:auth_path=/ffzdqu-47/obj1 2023-09-22T21:22:05.107 DEBUG:boto:Method: GET 2023-09-22T21:22:05.107 DEBUG:boto:Path: /ffzdqu-47/obj1 2023-09-22T21:22:05.107 DEBUG:boto:Data: 2023-09-22T21:22:05.108 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.108 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.108 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.108 DEBUG:boto:Params: {} 2023-09-22T21:22:05.109 DEBUG:boto:Token: None 2023-09-22T21:22:05.109 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj1 2023-09-22T21:22:05.109 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RlyoIVKGjjFfshzhqKGiaux8PzU= 2023-09-22T21:22:05.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RlyoIVKGjjFfshzhqKGiaux8PzU=', 'Content-Length': '0'} 2023-09-22T21:22:05.112 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2145a73442eb846-00650e057d-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.112 DEBUG:boto:path=/ffzdqu-47/obj1 2023-09-22T21:22:05.112 DEBUG:boto:auth_path=/ffzdqu-47/obj1 2023-09-22T21:22:05.113 DEBUG:boto:Method: GET 2023-09-22T21:22:05.113 DEBUG:boto:Path: /ffzdqu-47/obj1 2023-09-22T21:22:05.113 DEBUG:boto:Data: 2023-09-22T21:22:05.113 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.114 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.114 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.114 DEBUG:boto:Params: {} 2023-09-22T21:22:05.114 DEBUG:boto:Token: None 2023-09-22T21:22:05.115 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj1 2023-09-22T21:22:05.115 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RlyoIVKGjjFfshzhqKGiaux8PzU= 2023-09-22T21:22:05.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RlyoIVKGjjFfshzhqKGiaux8PzU=', 'Content-Length': '0'} 2023-09-22T21:22:05.118 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000982372591dc50052-00650e057d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.158 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:05.158 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:05.159 DEBUG:boto:path=/ffzdqu-47/?prefix=obj1 2023-09-22T21:22:05.159 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj1 2023-09-22T21:22:05.159 DEBUG:boto:Method: GET 2023-09-22T21:22:05.159 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj1 2023-09-22T21:22:05.159 DEBUG:boto:Data: 2023-09-22T21:22:05.159 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.159 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.159 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.159 DEBUG:boto:Params: {} 2023-09-22T21:22:05.160 DEBUG:boto:Token: None 2023-09-22T21:22:05.160 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/ 2023-09-22T21:22:05.160 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk= 2023-09-22T21:22:05.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk=', 'Content-Length': '0'} 2023-09-22T21:22:05.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d4d8421c30607f2-00650e057d-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.165 DEBUG:boto:b'ffzdqu-47obj11000falseobj12023-09-22T21:20:38.912Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:05.166 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:05.166 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:05.166 DEBUG:boto:path=/ffzdqu-47/?prefix=obj1 2023-09-22T21:22:05.166 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj1 2023-09-22T21:22:05.166 DEBUG:boto:Method: GET 2023-09-22T21:22:05.167 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj1 2023-09-22T21:22:05.167 DEBUG:boto:Data: 2023-09-22T21:22:05.167 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.167 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.167 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.167 DEBUG:boto:Params: {} 2023-09-22T21:22:05.167 DEBUG:boto:Token: None 2023-09-22T21:22:05.167 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/ 2023-09-22T21:22:05.168 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk= 2023-09-22T21:22:05.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk=', 'Content-Length': '0'} 2023-09-22T21:22:05.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a18b0356d4e6ed64-00650e057d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.173 DEBUG:boto:b'ffzdqu-47obj11000falseobj12023-09-22T21:20:38.912Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:05.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:22:05.173 DEBUG:boto:path=/ffzdqu-47/obj2 2023-09-22T21:22:05.173 DEBUG:boto:auth_path=/ffzdqu-47/obj2 2023-09-22T21:22:05.173 DEBUG:boto:path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.173 DEBUG:boto:auth_path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.174 DEBUG:boto:Method: GET 2023-09-22T21:22:05.174 DEBUG:boto:Path: /ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.174 DEBUG:boto:Data: 2023-09-22T21:22:05.174 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.174 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.174 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.174 DEBUG:boto:Params: {} 2023-09-22T21:22:05.174 DEBUG:boto:Token: None 2023-09-22T21:22:05.174 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.175 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5KS+LhYC8GtfToAGNwnUmDDGfqk= 2023-09-22T21:22:05.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5KS+LhYC8GtfToAGNwnUmDDGfqk=', 'Content-Length': '0'} 2023-09-22T21:22:05.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000add460bf6ce8b559-00650e057d-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.218 DEBUG:boto:path=/ffzdqu-47/obj2 2023-09-22T21:22:05.219 DEBUG:boto:auth_path=/ffzdqu-47/obj2 2023-09-22T21:22:05.219 DEBUG:boto:path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.219 DEBUG:boto:auth_path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.219 DEBUG:boto:Method: GET 2023-09-22T21:22:05.219 DEBUG:boto:Path: /ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.219 DEBUG:boto:Data: 2023-09-22T21:22:05.219 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.220 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.220 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.220 DEBUG:boto:Params: {} 2023-09-22T21:22:05.220 DEBUG:boto:Token: None 2023-09-22T21:22:05.220 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.220 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5KS+LhYC8GtfToAGNwnUmDDGfqk= 2023-09-22T21:22:05.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5KS+LhYC8GtfToAGNwnUmDDGfqk=', 'Content-Length': '0'} 2023-09-22T21:22:05.224 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000feeaa1e940b1219c-00650e057d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.224 DEBUG:boto:path=/ffzdqu-47/obj2 2023-09-22T21:22:05.224 DEBUG:boto:auth_path=/ffzdqu-47/obj2 2023-09-22T21:22:05.224 DEBUG:boto:path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.225 DEBUG:boto:auth_path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.225 DEBUG:boto:Method: HEAD 2023-09-22T21:22:05.225 DEBUG:boto:Path: /ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.225 DEBUG:boto:Data: 2023-09-22T21:22:05.225 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.225 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.225 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.225 DEBUG:boto:Params: {} 2023-09-22T21:22:05.226 DEBUG:boto:Token: None 2023-09-22T21:22:05.226 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.226 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vswGxuRxFqEMTg92KW2pFyVPwl8= 2023-09-22T21:22:05.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vswGxuRxFqEMTg92KW2pFyVPwl8=', 'Content-Length': '0'} 2023-09-22T21:22:05.228 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a7327b1f29ae26a-00650e057d-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.229 DEBUG:boto:path=/ffzdqu-47/obj2 2023-09-22T21:22:05.229 DEBUG:boto:auth_path=/ffzdqu-47/obj2 2023-09-22T21:22:05.229 DEBUG:boto:path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.229 DEBUG:boto:auth_path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.229 DEBUG:boto:Method: HEAD 2023-09-22T21:22:05.230 DEBUG:boto:Path: /ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.230 DEBUG:boto:Data: 2023-09-22T21:22:05.230 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.230 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.230 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.230 DEBUG:boto:Params: {} 2023-09-22T21:22:05.230 DEBUG:boto:Token: None 2023-09-22T21:22:05.231 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.231 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vswGxuRxFqEMTg92KW2pFyVPwl8= 2023-09-22T21:22:05.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vswGxuRxFqEMTg92KW2pFyVPwl8=', 'Content-Length': '0'} 2023-09-22T21:22:05.233 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066d266dde376c63c-00650e057d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:22:05.234 DEBUG:boto:path=/ffzdqu-47/obj2 2023-09-22T21:22:05.234 DEBUG:boto:auth_path=/ffzdqu-47/obj2 2023-09-22T21:22:05.234 DEBUG:boto:path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.234 DEBUG:boto:auth_path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.234 DEBUG:boto:Method: GET 2023-09-22T21:22:05.234 DEBUG:boto:Path: /ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.234 DEBUG:boto:Data: 2023-09-22T21:22:05.234 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.235 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.235 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.235 DEBUG:boto:Params: {} 2023-09-22T21:22:05.235 DEBUG:boto:Token: None 2023-09-22T21:22:05.235 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.235 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5KS+LhYC8GtfToAGNwnUmDDGfqk= 2023-09-22T21:22:05.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5KS+LhYC8GtfToAGNwnUmDDGfqk=', 'Content-Length': '0'} 2023-09-22T21:22:05.238 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afd5708e0c4df498-00650e057d-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.238 DEBUG:boto:path=/ffzdqu-47/obj2 2023-09-22T21:22:05.239 DEBUG:boto:auth_path=/ffzdqu-47/obj2 2023-09-22T21:22:05.239 DEBUG:boto:path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.239 DEBUG:boto:auth_path=/ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.239 DEBUG:boto:Method: GET 2023-09-22T21:22:05.239 DEBUG:boto:Path: /ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.239 DEBUG:boto:Data: 2023-09-22T21:22:05.239 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.239 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.239 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.240 DEBUG:boto:Params: {} 2023-09-22T21:22:05.240 DEBUG:boto:Token: None 2023-09-22T21:22:05.240 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj2?versionId=null 2023-09-22T21:22:05.240 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5KS+LhYC8GtfToAGNwnUmDDGfqk= 2023-09-22T21:22:05.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5KS+LhYC8GtfToAGNwnUmDDGfqk=', 'Content-Length': '0'} 2023-09-22T21:22:05.243 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c3171a0f371a0b3-00650e057d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.286 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:05.287 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:05.287 DEBUG:boto:path=/ffzdqu-47/?prefix=obj2 2023-09-22T21:22:05.287 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj2 2023-09-22T21:22:05.287 DEBUG:boto:Method: GET 2023-09-22T21:22:05.287 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj2 2023-09-22T21:22:05.287 DEBUG:boto:Data: 2023-09-22T21:22:05.287 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.287 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.287 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.288 DEBUG:boto:Params: {} 2023-09-22T21:22:05.288 DEBUG:boto:Token: None 2023-09-22T21:22:05.288 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/ 2023-09-22T21:22:05.288 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk= 2023-09-22T21:22:05.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk=', 'Content-Length': '0'} 2023-09-22T21:22:05.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2b6ce9294906888-00650e057d-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.293 DEBUG:boto:b'ffzdqu-47obj21000falseobj22023-09-22T21:20:38.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:05.294 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:05.294 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:05.294 DEBUG:boto:path=/ffzdqu-47/?prefix=obj2 2023-09-22T21:22:05.294 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj2 2023-09-22T21:22:05.294 DEBUG:boto:Method: GET 2023-09-22T21:22:05.294 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj2 2023-09-22T21:22:05.294 DEBUG:boto:Data: 2023-09-22T21:22:05.295 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.295 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.295 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.295 DEBUG:boto:Params: {} 2023-09-22T21:22:05.295 DEBUG:boto:Token: None 2023-09-22T21:22:05.295 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/ 2023-09-22T21:22:05.295 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk= 2023-09-22T21:22:05.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk=', 'Content-Length': '0'} 2023-09-22T21:22:05.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088145d9241c71953-00650e057d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.301 DEBUG:boto:b'ffzdqu-47obj21000falseobj22023-09-22T21:20:38.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:05.302 DEBUG:boto:path=/ffzdqu-47/obj2 2023-09-22T21:22:05.302 DEBUG:boto:auth_path=/ffzdqu-47/obj2 2023-09-22T21:22:05.302 DEBUG:boto:Method: HEAD 2023-09-22T21:22:05.302 DEBUG:boto:Path: /ffzdqu-47/obj2 2023-09-22T21:22:05.303 DEBUG:boto:Data: 2023-09-22T21:22:05.303 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.303 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.303 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.303 DEBUG:boto:Params: {} 2023-09-22T21:22:05.304 DEBUG:boto:Token: None 2023-09-22T21:22:05.304 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj2 2023-09-22T21:22:05.304 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GAEcP5y82Qz7f3URD4DCXOxcweY= 2023-09-22T21:22:05.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GAEcP5y82Qz7f3URD4DCXOxcweY=', 'Content-Length': '0'} 2023-09-22T21:22:05.307 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004bae78d42844bd81-00650e057d-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.307 DEBUG:boto:path=/ffzdqu-47/obj2 2023-09-22T21:22:05.308 DEBUG:boto:auth_path=/ffzdqu-47/obj2 2023-09-22T21:22:05.308 DEBUG:boto:Method: HEAD 2023-09-22T21:22:05.308 DEBUG:boto:Path: /ffzdqu-47/obj2 2023-09-22T21:22:05.308 DEBUG:boto:Data: 2023-09-22T21:22:05.308 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.308 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.308 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.308 DEBUG:boto:Params: {} 2023-09-22T21:22:05.309 DEBUG:boto:Token: None 2023-09-22T21:22:05.309 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj2 2023-09-22T21:22:05.309 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GAEcP5y82Qz7f3URD4DCXOxcweY= 2023-09-22T21:22:05.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GAEcP5y82Qz7f3URD4DCXOxcweY=', 'Content-Length': '0'} 2023-09-22T21:22:05.312 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003638217f655ea844-00650e057d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:22:05.312 DEBUG:boto:path=/ffzdqu-47/obj2 2023-09-22T21:22:05.312 DEBUG:boto:auth_path=/ffzdqu-47/obj2 2023-09-22T21:22:05.312 DEBUG:boto:Method: GET 2023-09-22T21:22:05.312 DEBUG:boto:Path: /ffzdqu-47/obj2 2023-09-22T21:22:05.313 DEBUG:boto:Data: 2023-09-22T21:22:05.313 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.313 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.313 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.313 DEBUG:boto:Params: {} 2023-09-22T21:22:05.313 DEBUG:boto:Token: None 2023-09-22T21:22:05.313 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj2 2023-09-22T21:22:05.313 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mto43wnvxx/Jys3+CmRbvxcS2K4= 2023-09-22T21:22:05.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mto43wnvxx/Jys3+CmRbvxcS2K4=', 'Content-Length': '0'} 2023-09-22T21:22:05.316 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e6f31b1929388db-00650e057d-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.358 DEBUG:boto:path=/ffzdqu-47/obj2 2023-09-22T21:22:05.359 DEBUG:boto:auth_path=/ffzdqu-47/obj2 2023-09-22T21:22:05.359 DEBUG:boto:Method: GET 2023-09-22T21:22:05.359 DEBUG:boto:Path: /ffzdqu-47/obj2 2023-09-22T21:22:05.359 DEBUG:boto:Data: 2023-09-22T21:22:05.359 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.359 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.359 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.360 DEBUG:boto:Params: {} 2023-09-22T21:22:05.360 DEBUG:boto:Token: None 2023-09-22T21:22:05.360 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj2 2023-09-22T21:22:05.360 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mto43wnvxx/Jys3+CmRbvxcS2K4= 2023-09-22T21:22:05.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mto43wnvxx/Jys3+CmRbvxcS2K4=', 'Content-Length': '0'} 2023-09-22T21:22:05.363 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079a017de947f86d0-00650e057d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.407 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:05.407 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:05.407 DEBUG:boto:path=/ffzdqu-47/?prefix=obj2 2023-09-22T21:22:05.407 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj2 2023-09-22T21:22:05.407 DEBUG:boto:Method: GET 2023-09-22T21:22:05.407 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj2 2023-09-22T21:22:05.408 DEBUG:boto:Data: 2023-09-22T21:22:05.408 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.408 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.408 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.408 DEBUG:boto:Params: {} 2023-09-22T21:22:05.408 DEBUG:boto:Token: None 2023-09-22T21:22:05.408 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/ 2023-09-22T21:22:05.409 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk= 2023-09-22T21:22:05.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk=', 'Content-Length': '0'} 2023-09-22T21:22:05.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077220b7d5bba4d38-00650e057d-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.414 DEBUG:boto:b'ffzdqu-47obj21000falseobj22023-09-22T21:20:38.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:05.415 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:05.415 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:05.415 DEBUG:boto:path=/ffzdqu-47/?prefix=obj2 2023-09-22T21:22:05.415 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj2 2023-09-22T21:22:05.415 DEBUG:boto:Method: GET 2023-09-22T21:22:05.416 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj2 2023-09-22T21:22:05.416 DEBUG:boto:Data: 2023-09-22T21:22:05.416 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.416 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.416 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.416 DEBUG:boto:Params: {} 2023-09-22T21:22:05.416 DEBUG:boto:Token: None 2023-09-22T21:22:05.417 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/ 2023-09-22T21:22:05.417 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk= 2023-09-22T21:22:05.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk=', 'Content-Length': '0'} 2023-09-22T21:22:05.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f76647f4da66fbb9-00650e057d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.466 DEBUG:boto:b'ffzdqu-47obj21000falseobj22023-09-22T21:20:38.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:05.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:22:05.467 DEBUG:boto:path=/ffzdqu-47/obj3 2023-09-22T21:22:05.467 DEBUG:boto:auth_path=/ffzdqu-47/obj3 2023-09-22T21:22:05.467 DEBUG:boto:path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.467 DEBUG:boto:auth_path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.467 DEBUG:boto:Method: GET 2023-09-22T21:22:05.467 DEBUG:boto:Path: /ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.468 DEBUG:boto:Data: 2023-09-22T21:22:05.468 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.468 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.468 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.468 DEBUG:boto:Params: {} 2023-09-22T21:22:05.468 DEBUG:boto:Token: None 2023-09-22T21:22:05.468 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.468 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qh6gMzJon52o6DgJu3NX4mV/VnE= 2023-09-22T21:22:05.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qh6gMzJon52o6DgJu3NX4mV/VnE=', 'Content-Length': '0'} 2023-09-22T21:22:05.471 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a477d78e5df4aeef-00650e057d-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.472 DEBUG:boto:path=/ffzdqu-47/obj3 2023-09-22T21:22:05.472 DEBUG:boto:auth_path=/ffzdqu-47/obj3 2023-09-22T21:22:05.472 DEBUG:boto:path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.472 DEBUG:boto:auth_path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.472 DEBUG:boto:Method: GET 2023-09-22T21:22:05.472 DEBUG:boto:Path: /ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.472 DEBUG:boto:Data: 2023-09-22T21:22:05.472 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.472 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.473 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.473 DEBUG:boto:Params: {} 2023-09-22T21:22:05.473 DEBUG:boto:Token: None 2023-09-22T21:22:05.473 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.473 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qh6gMzJon52o6DgJu3NX4mV/VnE= 2023-09-22T21:22:05.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qh6gMzJon52o6DgJu3NX4mV/VnE=', 'Content-Length': '0'} 2023-09-22T21:22:05.477 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aecfa4f249708619-00650e057d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.478 DEBUG:boto:path=/ffzdqu-47/obj3 2023-09-22T21:22:05.478 DEBUG:boto:auth_path=/ffzdqu-47/obj3 2023-09-22T21:22:05.478 DEBUG:boto:path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.478 DEBUG:boto:auth_path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.478 DEBUG:boto:Method: HEAD 2023-09-22T21:22:05.478 DEBUG:boto:Path: /ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.478 DEBUG:boto:Data: 2023-09-22T21:22:05.479 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.479 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.479 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.479 DEBUG:boto:Params: {} 2023-09-22T21:22:05.479 DEBUG:boto:Token: None 2023-09-22T21:22:05.479 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.479 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+UQk2whQ4TkzNeESnJzq7KLFdNE= 2023-09-22T21:22:05.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+UQk2whQ4TkzNeESnJzq7KLFdNE=', 'Content-Length': '0'} 2023-09-22T21:22:05.482 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a4ff91e879a018c-00650e057d-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.482 DEBUG:boto:path=/ffzdqu-47/obj3 2023-09-22T21:22:05.482 DEBUG:boto:auth_path=/ffzdqu-47/obj3 2023-09-22T21:22:05.482 DEBUG:boto:path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.482 DEBUG:boto:auth_path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.483 DEBUG:boto:Method: HEAD 2023-09-22T21:22:05.483 DEBUG:boto:Path: /ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.483 DEBUG:boto:Data: 2023-09-22T21:22:05.483 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.483 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.483 DEBUG:boto:Params: {} 2023-09-22T21:22:05.483 DEBUG:boto:Token: None 2023-09-22T21:22:05.483 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.484 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+UQk2whQ4TkzNeESnJzq7KLFdNE= 2023-09-22T21:22:05.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+UQk2whQ4TkzNeESnJzq7KLFdNE=', 'Content-Length': '0'} 2023-09-22T21:22:05.486 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9b286d2e89cb3d7-00650e057d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:22:05.487 DEBUG:boto:path=/ffzdqu-47/obj3 2023-09-22T21:22:05.487 DEBUG:boto:auth_path=/ffzdqu-47/obj3 2023-09-22T21:22:05.487 DEBUG:boto:path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.487 DEBUG:boto:auth_path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.487 DEBUG:boto:Method: GET 2023-09-22T21:22:05.487 DEBUG:boto:Path: /ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.487 DEBUG:boto:Data: 2023-09-22T21:22:05.487 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.487 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.488 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.488 DEBUG:boto:Params: {} 2023-09-22T21:22:05.488 DEBUG:boto:Token: None 2023-09-22T21:22:05.488 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.488 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qh6gMzJon52o6DgJu3NX4mV/VnE= 2023-09-22T21:22:05.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qh6gMzJon52o6DgJu3NX4mV/VnE=', 'Content-Length': '0'} 2023-09-22T21:22:05.491 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000547bcbe5c520ffbd-00650e057d-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.491 DEBUG:boto:path=/ffzdqu-47/obj3 2023-09-22T21:22:05.492 DEBUG:boto:auth_path=/ffzdqu-47/obj3 2023-09-22T21:22:05.492 DEBUG:boto:path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.492 DEBUG:boto:auth_path=/ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.492 DEBUG:boto:Method: GET 2023-09-22T21:22:05.492 DEBUG:boto:Path: /ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.492 DEBUG:boto:Data: 2023-09-22T21:22:05.492 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.492 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.492 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.493 DEBUG:boto:Params: {} 2023-09-22T21:22:05.493 DEBUG:boto:Token: None 2023-09-22T21:22:05.493 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj3?versionId=null 2023-09-22T21:22:05.493 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qh6gMzJon52o6DgJu3NX4mV/VnE= 2023-09-22T21:22:05.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qh6gMzJon52o6DgJu3NX4mV/VnE=', 'Content-Length': '0'} 2023-09-22T21:22:05.496 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b7366115c98d024-00650e057d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.538 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:05.539 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:05.539 DEBUG:boto:path=/ffzdqu-47/?prefix=obj3 2023-09-22T21:22:05.539 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj3 2023-09-22T21:22:05.539 DEBUG:boto:Method: GET 2023-09-22T21:22:05.539 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj3 2023-09-22T21:22:05.539 DEBUG:boto:Data: 2023-09-22T21:22:05.539 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.539 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.539 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.540 DEBUG:boto:Params: {} 2023-09-22T21:22:05.540 DEBUG:boto:Token: None 2023-09-22T21:22:05.540 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/ 2023-09-22T21:22:05.540 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk= 2023-09-22T21:22:05.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk=', 'Content-Length': '0'} 2023-09-22T21:22:05.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1a5e853dfdb39c1-00650e057d-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.546 DEBUG:boto:b'ffzdqu-47obj31000falseobj32023-09-22T21:20:38.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:05.546 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:05.547 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:05.547 DEBUG:boto:path=/ffzdqu-47/?prefix=obj3 2023-09-22T21:22:05.547 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj3 2023-09-22T21:22:05.547 DEBUG:boto:Method: GET 2023-09-22T21:22:05.547 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj3 2023-09-22T21:22:05.547 DEBUG:boto:Data: 2023-09-22T21:22:05.547 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.547 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.547 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.547 DEBUG:boto:Params: {} 2023-09-22T21:22:05.548 DEBUG:boto:Token: None 2023-09-22T21:22:05.548 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/ 2023-09-22T21:22:05.548 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk= 2023-09-22T21:22:05.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk=', 'Content-Length': '0'} 2023-09-22T21:22:05.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f280915e63912bb2-00650e057d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.553 DEBUG:boto:b'ffzdqu-47obj31000falseobj32023-09-22T21:20:38.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:05.553 DEBUG:boto:path=/ffzdqu-47/obj3 2023-09-22T21:22:05.554 DEBUG:boto:auth_path=/ffzdqu-47/obj3 2023-09-22T21:22:05.554 DEBUG:boto:Method: HEAD 2023-09-22T21:22:05.554 DEBUG:boto:Path: /ffzdqu-47/obj3 2023-09-22T21:22:05.554 DEBUG:boto:Data: 2023-09-22T21:22:05.554 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.554 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.554 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.554 DEBUG:boto:Params: {} 2023-09-22T21:22:05.554 DEBUG:boto:Token: None 2023-09-22T21:22:05.555 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj3 2023-09-22T21:22:05.555 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Rp92ByMRUhw1p+NsZCkOAKK/whY= 2023-09-22T21:22:05.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rp92ByMRUhw1p+NsZCkOAKK/whY=', 'Content-Length': '0'} 2023-09-22T21:22:05.557 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f44dd282714bd77-00650e057d-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.557 DEBUG:boto:path=/ffzdqu-47/obj3 2023-09-22T21:22:05.558 DEBUG:boto:auth_path=/ffzdqu-47/obj3 2023-09-22T21:22:05.558 DEBUG:boto:Method: HEAD 2023-09-22T21:22:05.558 DEBUG:boto:Path: /ffzdqu-47/obj3 2023-09-22T21:22:05.558 DEBUG:boto:Data: 2023-09-22T21:22:05.558 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.558 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.558 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.558 DEBUG:boto:Params: {} 2023-09-22T21:22:05.558 DEBUG:boto:Token: None 2023-09-22T21:22:05.559 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj3 2023-09-22T21:22:05.559 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Rp92ByMRUhw1p+NsZCkOAKK/whY= 2023-09-22T21:22:05.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rp92ByMRUhw1p+NsZCkOAKK/whY=', 'Content-Length': '0'} 2023-09-22T21:22:05.561 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014d7c90ff243a5a7-00650e057d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:22:05.561 DEBUG:boto:path=/ffzdqu-47/obj3 2023-09-22T21:22:05.562 DEBUG:boto:auth_path=/ffzdqu-47/obj3 2023-09-22T21:22:05.562 DEBUG:boto:Method: GET 2023-09-22T21:22:05.562 DEBUG:boto:Path: /ffzdqu-47/obj3 2023-09-22T21:22:05.562 DEBUG:boto:Data: 2023-09-22T21:22:05.562 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.562 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.562 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.562 DEBUG:boto:Params: {} 2023-09-22T21:22:05.562 DEBUG:boto:Token: None 2023-09-22T21:22:05.563 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj3 2023-09-22T21:22:05.563 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y50WD6obYBNOKjoVy+1ikmdc87I= 2023-09-22T21:22:05.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y50WD6obYBNOKjoVy+1ikmdc87I=', 'Content-Length': '0'} 2023-09-22T21:22:05.565 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e35995028f4f098-00650e057d-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.566 DEBUG:boto:path=/ffzdqu-47/obj3 2023-09-22T21:22:05.566 DEBUG:boto:auth_path=/ffzdqu-47/obj3 2023-09-22T21:22:05.566 DEBUG:boto:Method: GET 2023-09-22T21:22:05.566 DEBUG:boto:Path: /ffzdqu-47/obj3 2023-09-22T21:22:05.566 DEBUG:boto:Data: 2023-09-22T21:22:05.566 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.566 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.566 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.566 DEBUG:boto:Params: {} 2023-09-22T21:22:05.567 DEBUG:boto:Token: None 2023-09-22T21:22:05.567 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj3 2023-09-22T21:22:05.567 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y50WD6obYBNOKjoVy+1ikmdc87I= 2023-09-22T21:22:05.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y50WD6obYBNOKjoVy+1ikmdc87I=', 'Content-Length': '0'} 2023-09-22T21:22:05.570 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f0e60a5d59e4a15-00650e057d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.610 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:05.611 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:05.611 DEBUG:boto:path=/ffzdqu-47/?prefix=obj3 2023-09-22T21:22:05.611 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj3 2023-09-22T21:22:05.611 DEBUG:boto:Method: GET 2023-09-22T21:22:05.612 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj3 2023-09-22T21:22:05.612 DEBUG:boto:Data: 2023-09-22T21:22:05.612 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.612 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.613 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.613 DEBUG:boto:Params: {} 2023-09-22T21:22:05.613 DEBUG:boto:Token: None 2023-09-22T21:22:05.613 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/ 2023-09-22T21:22:05.614 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk= 2023-09-22T21:22:05.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk=', 'Content-Length': '0'} 2023-09-22T21:22:05.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f37d7e849a082bb1-00650e057d-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.619 DEBUG:boto:b'ffzdqu-47obj31000falseobj32023-09-22T21:20:38.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:05.620 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:05.620 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:05.620 DEBUG:boto:path=/ffzdqu-47/?prefix=obj3 2023-09-22T21:22:05.620 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj3 2023-09-22T21:22:05.620 DEBUG:boto:Method: GET 2023-09-22T21:22:05.621 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj3 2023-09-22T21:22:05.621 DEBUG:boto:Data: 2023-09-22T21:22:05.621 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.621 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.621 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.622 DEBUG:boto:Params: {} 2023-09-22T21:22:05.622 DEBUG:boto:Token: None 2023-09-22T21:22:05.622 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/ 2023-09-22T21:22:05.623 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk= 2023-09-22T21:22:05.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk=', 'Content-Length': '0'} 2023-09-22T21:22:05.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab50b3b8d14e19be-00650e057d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.674 DEBUG:boto:b'ffzdqu-47obj31000falseobj32023-09-22T21:20:38.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:05.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:22:05.675 DEBUG:boto:path=/ffzdqu-47/obj4 2023-09-22T21:22:05.676 DEBUG:boto:auth_path=/ffzdqu-47/obj4 2023-09-22T21:22:05.676 DEBUG:boto:path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.676 DEBUG:boto:auth_path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.676 DEBUG:boto:Method: GET 2023-09-22T21:22:05.677 DEBUG:boto:Path: /ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.677 DEBUG:boto:Data: 2023-09-22T21:22:05.677 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.677 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.677 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.678 DEBUG:boto:Params: {} 2023-09-22T21:22:05.678 DEBUG:boto:Token: None 2023-09-22T21:22:05.681 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.682 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SAp7C26VWZCxzFPaBG4/gpv1hoQ= 2023-09-22T21:22:05.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SAp7C26VWZCxzFPaBG4/gpv1hoQ=', 'Content-Length': '0'} 2023-09-22T21:22:05.685 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037d9a1d002b6f2b5-00650e057d-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.686 DEBUG:boto:path=/ffzdqu-47/obj4 2023-09-22T21:22:05.686 DEBUG:boto:auth_path=/ffzdqu-47/obj4 2023-09-22T21:22:05.687 DEBUG:boto:path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.687 DEBUG:boto:auth_path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.687 DEBUG:boto:Method: GET 2023-09-22T21:22:05.687 DEBUG:boto:Path: /ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.687 DEBUG:boto:Data: 2023-09-22T21:22:05.688 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.688 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.688 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.688 DEBUG:boto:Params: {} 2023-09-22T21:22:05.689 DEBUG:boto:Token: None 2023-09-22T21:22:05.689 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.689 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SAp7C26VWZCxzFPaBG4/gpv1hoQ= 2023-09-22T21:22:05.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SAp7C26VWZCxzFPaBG4/gpv1hoQ=', 'Content-Length': '0'} 2023-09-22T21:22:05.692 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f3f09d72a433507-00650e057d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.693 DEBUG:boto:path=/ffzdqu-47/obj4 2023-09-22T21:22:05.693 DEBUG:boto:auth_path=/ffzdqu-47/obj4 2023-09-22T21:22:05.693 DEBUG:boto:path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.694 DEBUG:boto:auth_path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.694 DEBUG:boto:Method: HEAD 2023-09-22T21:22:05.694 DEBUG:boto:Path: /ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.694 DEBUG:boto:Data: 2023-09-22T21:22:05.694 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.695 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.695 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.695 DEBUG:boto:Params: {} 2023-09-22T21:22:05.695 DEBUG:boto:Token: None 2023-09-22T21:22:05.696 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.696 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZxrEBq9tZQqZckpCrVQQCbQAAMU= 2023-09-22T21:22:05.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZxrEBq9tZQqZckpCrVQQCbQAAMU=', 'Content-Length': '0'} 2023-09-22T21:22:05.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4c09e0c70916f6b-00650e057d-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.700 DEBUG:boto:path=/ffzdqu-47/obj4 2023-09-22T21:22:05.700 DEBUG:boto:auth_path=/ffzdqu-47/obj4 2023-09-22T21:22:05.700 DEBUG:boto:path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.700 DEBUG:boto:auth_path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.700 DEBUG:boto:Method: HEAD 2023-09-22T21:22:05.701 DEBUG:boto:Path: /ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.701 DEBUG:boto:Data: 2023-09-22T21:22:05.701 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.701 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.702 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.702 DEBUG:boto:Params: {} 2023-09-22T21:22:05.702 DEBUG:boto:Token: None 2023-09-22T21:22:05.702 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.703 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZxrEBq9tZQqZckpCrVQQCbQAAMU= 2023-09-22T21:22:05.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZxrEBq9tZQqZckpCrVQQCbQAAMU=', 'Content-Length': '0'} 2023-09-22T21:22:05.705 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004dfef71804260b89-00650e057d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:22:05.706 DEBUG:boto:path=/ffzdqu-47/obj4 2023-09-22T21:22:05.706 DEBUG:boto:auth_path=/ffzdqu-47/obj4 2023-09-22T21:22:05.706 DEBUG:boto:path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.706 DEBUG:boto:auth_path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.706 DEBUG:boto:Method: GET 2023-09-22T21:22:05.706 DEBUG:boto:Path: /ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.707 DEBUG:boto:Data: 2023-09-22T21:22:05.707 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.707 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.707 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.707 DEBUG:boto:Params: {} 2023-09-22T21:22:05.707 DEBUG:boto:Token: None 2023-09-22T21:22:05.707 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.707 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SAp7C26VWZCxzFPaBG4/gpv1hoQ= 2023-09-22T21:22:05.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SAp7C26VWZCxzFPaBG4/gpv1hoQ=', 'Content-Length': '0'} 2023-09-22T21:22:05.710 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea420e7984841e0b-00650e057d-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.754 DEBUG:boto:path=/ffzdqu-47/obj4 2023-09-22T21:22:05.755 DEBUG:boto:auth_path=/ffzdqu-47/obj4 2023-09-22T21:22:05.755 DEBUG:boto:path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.755 DEBUG:boto:auth_path=/ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.755 DEBUG:boto:Method: GET 2023-09-22T21:22:05.755 DEBUG:boto:Path: /ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.755 DEBUG:boto:Data: 2023-09-22T21:22:05.755 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.755 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.755 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.755 DEBUG:boto:Params: {} 2023-09-22T21:22:05.756 DEBUG:boto:Token: None 2023-09-22T21:22:05.756 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj4?versionId=null 2023-09-22T21:22:05.756 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SAp7C26VWZCxzFPaBG4/gpv1hoQ= 2023-09-22T21:22:05.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SAp7C26VWZCxzFPaBG4/gpv1hoQ=', 'Content-Length': '0'} 2023-09-22T21:22:05.759 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0823a4c897db809-00650e057d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.802 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:05.803 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:05.803 DEBUG:boto:path=/ffzdqu-47/?prefix=obj4 2023-09-22T21:22:05.803 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj4 2023-09-22T21:22:05.803 DEBUG:boto:Method: GET 2023-09-22T21:22:05.803 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj4 2023-09-22T21:22:05.804 DEBUG:boto:Data: 2023-09-22T21:22:05.804 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.804 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.804 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.805 DEBUG:boto:Params: {} 2023-09-22T21:22:05.805 DEBUG:boto:Token: None 2023-09-22T21:22:05.805 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/ 2023-09-22T21:22:05.805 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk= 2023-09-22T21:22:05.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk=', 'Content-Length': '0'} 2023-09-22T21:22:05.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f2a95d4acfb5e21-00650e057d-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.811 DEBUG:boto:b'ffzdqu-47obj41000falseobj42023-09-22T21:20:38.999Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:05.811 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:05.811 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:05.812 DEBUG:boto:path=/ffzdqu-47/?prefix=obj4 2023-09-22T21:22:05.812 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj4 2023-09-22T21:22:05.812 DEBUG:boto:Method: GET 2023-09-22T21:22:05.812 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj4 2023-09-22T21:22:05.812 DEBUG:boto:Data: 2023-09-22T21:22:05.813 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.813 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.813 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.813 DEBUG:boto:Params: {} 2023-09-22T21:22:05.813 DEBUG:boto:Token: None 2023-09-22T21:22:05.814 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/ 2023-09-22T21:22:05.814 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk= 2023-09-22T21:22:05.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk=', 'Content-Length': '0'} 2023-09-22T21:22:05.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ec99ad2d0817db1-00650e057d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.862 DEBUG:boto:b'ffzdqu-47obj41000falseobj42023-09-22T21:20:38.999Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:05.863 DEBUG:boto:path=/ffzdqu-47/obj4 2023-09-22T21:22:05.863 DEBUG:boto:auth_path=/ffzdqu-47/obj4 2023-09-22T21:22:05.863 DEBUG:boto:Method: HEAD 2023-09-22T21:22:05.863 DEBUG:boto:Path: /ffzdqu-47/obj4 2023-09-22T21:22:05.863 DEBUG:boto:Data: 2023-09-22T21:22:05.863 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.864 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.864 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.864 DEBUG:boto:Params: {} 2023-09-22T21:22:05.864 DEBUG:boto:Token: None 2023-09-22T21:22:05.864 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj4 2023-09-22T21:22:05.864 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mu6t1KxEbQOk28mqVusdeDKdNW8= 2023-09-22T21:22:05.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mu6t1KxEbQOk28mqVusdeDKdNW8=', 'Content-Length': '0'} 2023-09-22T21:22:05.867 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b027c03342daab44-00650e057d-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.867 DEBUG:boto:path=/ffzdqu-47/obj4 2023-09-22T21:22:05.868 DEBUG:boto:auth_path=/ffzdqu-47/obj4 2023-09-22T21:22:05.868 DEBUG:boto:Method: HEAD 2023-09-22T21:22:05.868 DEBUG:boto:Path: /ffzdqu-47/obj4 2023-09-22T21:22:05.868 DEBUG:boto:Data: 2023-09-22T21:22:05.868 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.868 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.868 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.868 DEBUG:boto:Params: {} 2023-09-22T21:22:05.869 DEBUG:boto:Token: None 2023-09-22T21:22:05.869 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj4 2023-09-22T21:22:05.869 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mu6t1KxEbQOk28mqVusdeDKdNW8= 2023-09-22T21:22:05.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mu6t1KxEbQOk28mqVusdeDKdNW8=', 'Content-Length': '0'} 2023-09-22T21:22:05.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd68cd04e137a1eb-00650e057d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:22:05.872 DEBUG:boto:path=/ffzdqu-47/obj4 2023-09-22T21:22:05.873 DEBUG:boto:auth_path=/ffzdqu-47/obj4 2023-09-22T21:22:05.873 DEBUG:boto:Method: GET 2023-09-22T21:22:05.873 DEBUG:boto:Path: /ffzdqu-47/obj4 2023-09-22T21:22:05.873 DEBUG:boto:Data: 2023-09-22T21:22:05.873 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.873 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.873 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.873 DEBUG:boto:Params: {} 2023-09-22T21:22:05.874 DEBUG:boto:Token: None 2023-09-22T21:22:05.874 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj4 2023-09-22T21:22:05.874 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2kU1X2RoWM3AGHq4mm7HDiMX5IM= 2023-09-22T21:22:05.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2kU1X2RoWM3AGHq4mm7HDiMX5IM=', 'Content-Length': '0'} 2023-09-22T21:22:05.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c2a8cfcca734b35-00650e057d-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.877 DEBUG:boto:path=/ffzdqu-47/obj4 2023-09-22T21:22:05.877 DEBUG:boto:auth_path=/ffzdqu-47/obj4 2023-09-22T21:22:05.877 DEBUG:boto:Method: GET 2023-09-22T21:22:05.877 DEBUG:boto:Path: /ffzdqu-47/obj4 2023-09-22T21:22:05.877 DEBUG:boto:Data: 2023-09-22T21:22:05.877 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.878 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.878 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.878 DEBUG:boto:Params: {} 2023-09-22T21:22:05.878 DEBUG:boto:Token: None 2023-09-22T21:22:05.878 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj4 2023-09-22T21:22:05.878 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2kU1X2RoWM3AGHq4mm7HDiMX5IM= 2023-09-22T21:22:05.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2kU1X2RoWM3AGHq4mm7HDiMX5IM=', 'Content-Length': '0'} 2023-09-22T21:22:05.881 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:20:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002725cd62a907716c-00650e057d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.926 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:05.927 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:05.927 DEBUG:boto:path=/ffzdqu-47/?prefix=obj4 2023-09-22T21:22:05.927 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj4 2023-09-22T21:22:05.927 DEBUG:boto:Method: GET 2023-09-22T21:22:05.927 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj4 2023-09-22T21:22:05.927 DEBUG:boto:Data: 2023-09-22T21:22:05.927 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.927 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.928 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.928 DEBUG:boto:Params: {} 2023-09-22T21:22:05.928 DEBUG:boto:Token: None 2023-09-22T21:22:05.928 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/ 2023-09-22T21:22:05.928 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk= 2023-09-22T21:22:05.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk=', 'Content-Length': '0'} 2023-09-22T21:22:05.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017a7a8032170b56d-00650e057d-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.933 DEBUG:boto:b'ffzdqu-47obj41000falseobj42023-09-22T21:20:38.999Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:05.934 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:05.934 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:05.934 DEBUG:boto:path=/ffzdqu-47/?prefix=obj4 2023-09-22T21:22:05.934 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj4 2023-09-22T21:22:05.934 DEBUG:boto:Method: GET 2023-09-22T21:22:05.935 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj4 2023-09-22T21:22:05.935 DEBUG:boto:Data: 2023-09-22T21:22:05.935 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.935 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.935 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.935 DEBUG:boto:Params: {} 2023-09-22T21:22:05.935 DEBUG:boto:Token: None 2023-09-22T21:22:05.935 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/ 2023-09-22T21:22:05.935 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk= 2023-09-22T21:22:05.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5DQhCQqJC96dRe4kFjZKDqhKcmk=', 'Content-Length': '0'} 2023-09-22T21:22:05.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b624671fc74df745-00650e057d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.982 DEBUG:boto:b'ffzdqu-47obj41000falseobj42023-09-22T21:20:38.999Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:05.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-22T21:22:05.983 DEBUG:boto:path=/ffzdqu-47/obj5 2023-09-22T21:22:05.983 DEBUG:boto:auth_path=/ffzdqu-47/obj5 2023-09-22T21:22:05.984 DEBUG:boto:path=/ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:05.984 DEBUG:boto:auth_path=/ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:05.984 DEBUG:boto:Method: GET 2023-09-22T21:22:05.984 DEBUG:boto:Path: /ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:05.984 DEBUG:boto:Data: 2023-09-22T21:22:05.984 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.985 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.985 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.985 DEBUG:boto:Params: {} 2023-09-22T21:22:05.985 DEBUG:boto:Token: None 2023-09-22T21:22:05.986 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:05.986 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QXL6/3aFFIhkNH27yny/oteee9M= 2023-09-22T21:22:05.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QXL6/3aFFIhkNH27yny/oteee9M=', 'Content-Length': '0'} 2023-09-22T21:22:05.989 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005badf18d02e45e64-00650e057d-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.989 DEBUG:boto:path=/ffzdqu-47/obj5 2023-09-22T21:22:05.989 DEBUG:boto:auth_path=/ffzdqu-47/obj5 2023-09-22T21:22:05.990 DEBUG:boto:path=/ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:05.990 DEBUG:boto:auth_path=/ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:05.990 DEBUG:boto:Method: GET 2023-09-22T21:22:05.990 DEBUG:boto:Path: /ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:05.990 DEBUG:boto:Data: 2023-09-22T21:22:05.991 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.991 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:05.991 DEBUG:boto:Port: 8000 2023-09-22T21:22:05.991 DEBUG:boto:Params: {} 2023-09-22T21:22:05.991 DEBUG:boto:Token: None 2023-09-22T21:22:05.992 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:05.992 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QXL6/3aFFIhkNH27yny/oteee9M= 2023-09-22T21:22:05.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QXL6/3aFFIhkNH27yny/oteee9M=', 'Content-Length': '0'} 2023-09-22T21:22:05.995 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005662d6158c2041eb-00650e057d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:05.995 DEBUG:boto:path=/ffzdqu-47/obj5 2023-09-22T21:22:05.995 DEBUG:boto:auth_path=/ffzdqu-47/obj5 2023-09-22T21:22:05.995 DEBUG:boto:path=/ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:05.996 DEBUG:boto:auth_path=/ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:05.996 DEBUG:boto:Method: HEAD 2023-09-22T21:22:05.996 DEBUG:boto:Path: /ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:05.996 DEBUG:boto:Data: 2023-09-22T21:22:05.996 DEBUG:boto:Headers: {} 2023-09-22T21:22:05.997 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:05.997 DEBUG:boto:Port: 8001 2023-09-22T21:22:05.997 DEBUG:boto:Params: {} 2023-09-22T21:22:05.997 DEBUG:boto:Token: None 2023-09-22T21:22:05.997 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:05 GMT /ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:05.997 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FaYDhhI3ahQV0ohZ8t+defY/L5w= 2023-09-22T21:22:05.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FaYDhhI3ahQV0ohZ8t+defY/L5w=', 'Content-Length': '0'} 2023-09-22T21:22:06.000 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017f673f38a93791d-00650e057d-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.001 DEBUG:boto:path=/ffzdqu-47/obj5 2023-09-22T21:22:06.001 DEBUG:boto:auth_path=/ffzdqu-47/obj5 2023-09-22T21:22:06.001 DEBUG:boto:path=/ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:06.001 DEBUG:boto:auth_path=/ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:06.001 DEBUG:boto:Method: HEAD 2023-09-22T21:22:06.001 DEBUG:boto:Path: /ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:06.002 DEBUG:boto:Data: 2023-09-22T21:22:06.002 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.002 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.002 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.002 DEBUG:boto:Params: {} 2023-09-22T21:22:06.003 DEBUG:boto:Token: None 2023-09-22T21:22:06.003 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:06.003 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CuGmEUPXbxOCSp17eV1RSoKGMJ4= 2023-09-22T21:22:06.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CuGmEUPXbxOCSp17eV1RSoKGMJ4=', 'Content-Length': '0'} 2023-09-22T21:22:06.006 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001107ac5a596a74c7-00650e057e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-22T21:22:06.006 DEBUG:boto:path=/ffzdqu-47/obj5 2023-09-22T21:22:06.007 DEBUG:boto:auth_path=/ffzdqu-47/obj5 2023-09-22T21:22:06.007 DEBUG:boto:path=/ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:06.007 DEBUG:boto:auth_path=/ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:06.007 DEBUG:boto:Method: GET 2023-09-22T21:22:06.007 DEBUG:boto:Path: /ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:06.008 DEBUG:boto:Data: 2023-09-22T21:22:06.008 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.008 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.008 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.008 DEBUG:boto:Params: {} 2023-09-22T21:22:06.008 DEBUG:boto:Token: None 2023-09-22T21:22:06.009 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:06.009 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g7ZOsKjXkT9r6hw1Wvyj4pLugl4= 2023-09-22T21:22:06.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g7ZOsKjXkT9r6hw1Wvyj4pLugl4=', 'Content-Length': '0'} 2023-09-22T21:22:06.012 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000197b0905d32f0985-00650e057e-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.012 DEBUG:boto:path=/ffzdqu-47/obj5 2023-09-22T21:22:06.012 DEBUG:boto:auth_path=/ffzdqu-47/obj5 2023-09-22T21:22:06.012 DEBUG:boto:path=/ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:06.013 DEBUG:boto:auth_path=/ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:06.013 DEBUG:boto:Method: GET 2023-09-22T21:22:06.013 DEBUG:boto:Path: /ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:06.013 DEBUG:boto:Data: 2023-09-22T21:22:06.013 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.013 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.014 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.014 DEBUG:boto:Params: {} 2023-09-22T21:22:06.014 DEBUG:boto:Token: None 2023-09-22T21:22:06.014 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj5?versionId=null 2023-09-22T21:22:06.015 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g7ZOsKjXkT9r6hw1Wvyj4pLugl4= 2023-09-22T21:22:06.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g7ZOsKjXkT9r6hw1Wvyj4pLugl4=', 'Content-Length': '0'} 2023-09-22T21:22:06.018 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ecf14ee7f9254c3d-00650e057e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.062 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:06.063 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:06.063 DEBUG:boto:path=/ffzdqu-47/?prefix=obj5 2023-09-22T21:22:06.063 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj5 2023-09-22T21:22:06.063 DEBUG:boto:Method: GET 2023-09-22T21:22:06.063 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj5 2023-09-22T21:22:06.064 DEBUG:boto:Data: 2023-09-22T21:22:06.064 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.064 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.064 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.064 DEBUG:boto:Params: {} 2023-09-22T21:22:06.065 DEBUG:boto:Token: None 2023-09-22T21:22:06.065 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/ 2023-09-22T21:22:06.065 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40= 2023-09-22T21:22:06.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40=', 'Content-Length': '0'} 2023-09-22T21:22:06.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee950866ba64a7fc-00650e057e-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.070 DEBUG:boto:b'ffzdqu-47obj51000falseobj52023-09-22T21:21:22.372Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:06.071 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:06.071 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:06.071 DEBUG:boto:path=/ffzdqu-47/?prefix=obj5 2023-09-22T21:22:06.071 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj5 2023-09-22T21:22:06.071 DEBUG:boto:Method: GET 2023-09-22T21:22:06.072 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj5 2023-09-22T21:22:06.072 DEBUG:boto:Data: 2023-09-22T21:22:06.072 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.072 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.072 DEBUG:boto:Params: {} 2023-09-22T21:22:06.073 DEBUG:boto:Token: None 2023-09-22T21:22:06.073 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/ 2023-09-22T21:22:06.073 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40= 2023-09-22T21:22:06.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40=', 'Content-Length': '0'} 2023-09-22T21:22:06.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001edd55a32ecdcd12-00650e057e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.118 DEBUG:boto:b'ffzdqu-47obj51000falseobj52023-09-22T21:21:22.372Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:06.119 DEBUG:boto:path=/ffzdqu-47/obj5 2023-09-22T21:22:06.119 DEBUG:boto:auth_path=/ffzdqu-47/obj5 2023-09-22T21:22:06.119 DEBUG:boto:Method: HEAD 2023-09-22T21:22:06.119 DEBUG:boto:Path: /ffzdqu-47/obj5 2023-09-22T21:22:06.119 DEBUG:boto:Data: 2023-09-22T21:22:06.120 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.120 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.120 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.120 DEBUG:boto:Params: {} 2023-09-22T21:22:06.120 DEBUG:boto:Token: None 2023-09-22T21:22:06.120 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj5 2023-09-22T21:22:06.120 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PTEt4Yj7dz9E3oQjl7nw/MIRYek= 2023-09-22T21:22:06.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PTEt4Yj7dz9E3oQjl7nw/MIRYek=', 'Content-Length': '0'} 2023-09-22T21:22:06.124 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e147137bfd87cf0e-00650e057e-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.124 DEBUG:boto:path=/ffzdqu-47/obj5 2023-09-22T21:22:06.124 DEBUG:boto:auth_path=/ffzdqu-47/obj5 2023-09-22T21:22:06.124 DEBUG:boto:Method: HEAD 2023-09-22T21:22:06.125 DEBUG:boto:Path: /ffzdqu-47/obj5 2023-09-22T21:22:06.125 DEBUG:boto:Data: 2023-09-22T21:22:06.125 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.125 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.125 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.125 DEBUG:boto:Params: {} 2023-09-22T21:22:06.125 DEBUG:boto:Token: None 2023-09-22T21:22:06.125 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj5 2023-09-22T21:22:06.126 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PTEt4Yj7dz9E3oQjl7nw/MIRYek= 2023-09-22T21:22:06.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PTEt4Yj7dz9E3oQjl7nw/MIRYek=', 'Content-Length': '0'} 2023-09-22T21:22:06.129 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb3795b7afb18e7a-00650e057e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-22T21:22:06.130 DEBUG:boto:path=/ffzdqu-47/obj5 2023-09-22T21:22:06.130 DEBUG:boto:auth_path=/ffzdqu-47/obj5 2023-09-22T21:22:06.130 DEBUG:boto:Method: GET 2023-09-22T21:22:06.130 DEBUG:boto:Path: /ffzdqu-47/obj5 2023-09-22T21:22:06.130 DEBUG:boto:Data: 2023-09-22T21:22:06.130 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.131 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.131 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.131 DEBUG:boto:Params: {} 2023-09-22T21:22:06.131 DEBUG:boto:Token: None 2023-09-22T21:22:06.131 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj5 2023-09-22T21:22:06.131 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:K8n7rhpTgYD4r+L/gRjIH54gNvQ= 2023-09-22T21:22:06.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:K8n7rhpTgYD4r+L/gRjIH54gNvQ=', 'Content-Length': '0'} 2023-09-22T21:22:06.134 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eedc5475caec9341-00650e057e-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.134 DEBUG:boto:path=/ffzdqu-47/obj5 2023-09-22T21:22:06.135 DEBUG:boto:auth_path=/ffzdqu-47/obj5 2023-09-22T21:22:06.135 DEBUG:boto:Method: GET 2023-09-22T21:22:06.135 DEBUG:boto:Path: /ffzdqu-47/obj5 2023-09-22T21:22:06.135 DEBUG:boto:Data: 2023-09-22T21:22:06.135 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.135 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.135 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.135 DEBUG:boto:Params: {} 2023-09-22T21:22:06.136 DEBUG:boto:Token: None 2023-09-22T21:22:06.136 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj5 2023-09-22T21:22:06.136 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:K8n7rhpTgYD4r+L/gRjIH54gNvQ= 2023-09-22T21:22:06.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:K8n7rhpTgYD4r+L/gRjIH54gNvQ=', 'Content-Length': '0'} 2023-09-22T21:22:06.139 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000243ee133a76b1bdc-00650e057e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.186 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:06.187 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:06.187 DEBUG:boto:path=/ffzdqu-47/?prefix=obj5 2023-09-22T21:22:06.187 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj5 2023-09-22T21:22:06.187 DEBUG:boto:Method: GET 2023-09-22T21:22:06.187 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj5 2023-09-22T21:22:06.187 DEBUG:boto:Data: 2023-09-22T21:22:06.188 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.188 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.188 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.188 DEBUG:boto:Params: {} 2023-09-22T21:22:06.188 DEBUG:boto:Token: None 2023-09-22T21:22:06.188 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/ 2023-09-22T21:22:06.188 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40= 2023-09-22T21:22:06.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40=', 'Content-Length': '0'} 2023-09-22T21:22:06.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043a5af8156b50a45-00650e057e-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.193 DEBUG:boto:b'ffzdqu-47obj51000falseobj52023-09-22T21:21:22.372Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:06.194 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:06.194 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:06.194 DEBUG:boto:path=/ffzdqu-47/?prefix=obj5 2023-09-22T21:22:06.194 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj5 2023-09-22T21:22:06.194 DEBUG:boto:Method: GET 2023-09-22T21:22:06.194 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj5 2023-09-22T21:22:06.194 DEBUG:boto:Data: 2023-09-22T21:22:06.194 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.194 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.195 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.195 DEBUG:boto:Params: {} 2023-09-22T21:22:06.195 DEBUG:boto:Token: None 2023-09-22T21:22:06.195 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/ 2023-09-22T21:22:06.195 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40= 2023-09-22T21:22:06.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40=', 'Content-Length': '0'} 2023-09-22T21:22:06.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006aaf9cf134146656-00650e057e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.200 DEBUG:boto:b'ffzdqu-47obj51000falseobj52023-09-22T21:21:22.372Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:06.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-22T21:22:06.201 DEBUG:boto:path=/ffzdqu-47/obj6 2023-09-22T21:22:06.201 DEBUG:boto:auth_path=/ffzdqu-47/obj6 2023-09-22T21:22:06.201 DEBUG:boto:path=/ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.201 DEBUG:boto:auth_path=/ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.201 DEBUG:boto:Method: GET 2023-09-22T21:22:06.201 DEBUG:boto:Path: /ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.202 DEBUG:boto:Data: 2023-09-22T21:22:06.202 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.202 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.202 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.202 DEBUG:boto:Params: {} 2023-09-22T21:22:06.202 DEBUG:boto:Token: None 2023-09-22T21:22:06.202 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.202 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uPbCLiKivOoMwRAUJWURyte29k8= 2023-09-22T21:22:06.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uPbCLiKivOoMwRAUJWURyte29k8=', 'Content-Length': '0'} 2023-09-22T21:22:06.205 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000580998c25bf3109f-00650e057e-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.246 DEBUG:boto:path=/ffzdqu-47/obj6 2023-09-22T21:22:06.247 DEBUG:boto:auth_path=/ffzdqu-47/obj6 2023-09-22T21:22:06.247 DEBUG:boto:path=/ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.247 DEBUG:boto:auth_path=/ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.247 DEBUG:boto:Method: GET 2023-09-22T21:22:06.247 DEBUG:boto:Path: /ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.247 DEBUG:boto:Data: 2023-09-22T21:22:06.247 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.247 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.247 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.247 DEBUG:boto:Params: {} 2023-09-22T21:22:06.248 DEBUG:boto:Token: None 2023-09-22T21:22:06.248 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.248 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uPbCLiKivOoMwRAUJWURyte29k8= 2023-09-22T21:22:06.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uPbCLiKivOoMwRAUJWURyte29k8=', 'Content-Length': '0'} 2023-09-22T21:22:06.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e29a7b740dd7c61a-00650e057e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.294 DEBUG:boto:path=/ffzdqu-47/obj6 2023-09-22T21:22:06.295 DEBUG:boto:auth_path=/ffzdqu-47/obj6 2023-09-22T21:22:06.295 DEBUG:boto:path=/ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.295 DEBUG:boto:auth_path=/ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.295 DEBUG:boto:Method: HEAD 2023-09-22T21:22:06.295 DEBUG:boto:Path: /ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.295 DEBUG:boto:Data: 2023-09-22T21:22:06.295 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.295 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.295 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.296 DEBUG:boto:Params: {} 2023-09-22T21:22:06.296 DEBUG:boto:Token: None 2023-09-22T21:22:06.296 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.296 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GwsZctI4p2+YelUJPll+MPIjjbc= 2023-09-22T21:22:06.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GwsZctI4p2+YelUJPll+MPIjjbc=', 'Content-Length': '0'} 2023-09-22T21:22:06.299 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061ffebe049af7b83-00650e057e-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.299 DEBUG:boto:path=/ffzdqu-47/obj6 2023-09-22T21:22:06.299 DEBUG:boto:auth_path=/ffzdqu-47/obj6 2023-09-22T21:22:06.299 DEBUG:boto:path=/ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.299 DEBUG:boto:auth_path=/ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.299 DEBUG:boto:Method: HEAD 2023-09-22T21:22:06.299 DEBUG:boto:Path: /ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.299 DEBUG:boto:Data: 2023-09-22T21:22:06.300 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.300 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.300 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.300 DEBUG:boto:Params: {} 2023-09-22T21:22:06.300 DEBUG:boto:Token: None 2023-09-22T21:22:06.300 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.300 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GwsZctI4p2+YelUJPll+MPIjjbc= 2023-09-22T21:22:06.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GwsZctI4p2+YelUJPll+MPIjjbc=', 'Content-Length': '0'} 2023-09-22T21:22:06.303 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5f1c8e3f1b4196f-00650e057e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-22T21:22:06.303 DEBUG:boto:path=/ffzdqu-47/obj6 2023-09-22T21:22:06.303 DEBUG:boto:auth_path=/ffzdqu-47/obj6 2023-09-22T21:22:06.303 DEBUG:boto:path=/ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.304 DEBUG:boto:auth_path=/ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.304 DEBUG:boto:Method: GET 2023-09-22T21:22:06.304 DEBUG:boto:Path: /ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.304 DEBUG:boto:Data: 2023-09-22T21:22:06.304 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.304 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.304 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.304 DEBUG:boto:Params: {} 2023-09-22T21:22:06.304 DEBUG:boto:Token: None 2023-09-22T21:22:06.305 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.305 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uPbCLiKivOoMwRAUJWURyte29k8= 2023-09-22T21:22:06.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uPbCLiKivOoMwRAUJWURyte29k8=', 'Content-Length': '0'} 2023-09-22T21:22:06.307 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fea9ed2a917f66c9-00650e057e-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.307 DEBUG:boto:path=/ffzdqu-47/obj6 2023-09-22T21:22:06.308 DEBUG:boto:auth_path=/ffzdqu-47/obj6 2023-09-22T21:22:06.308 DEBUG:boto:path=/ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.308 DEBUG:boto:auth_path=/ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.308 DEBUG:boto:Method: GET 2023-09-22T21:22:06.308 DEBUG:boto:Path: /ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.308 DEBUG:boto:Data: 2023-09-22T21:22:06.308 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.308 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.308 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.309 DEBUG:boto:Params: {} 2023-09-22T21:22:06.309 DEBUG:boto:Token: None 2023-09-22T21:22:06.309 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj6?versionId=null 2023-09-22T21:22:06.309 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uPbCLiKivOoMwRAUJWURyte29k8= 2023-09-22T21:22:06.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uPbCLiKivOoMwRAUJWURyte29k8=', 'Content-Length': '0'} 2023-09-22T21:22:06.312 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a827848cd463812-00650e057e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.354 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:06.355 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:06.355 DEBUG:boto:path=/ffzdqu-47/?prefix=obj6 2023-09-22T21:22:06.355 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj6 2023-09-22T21:22:06.355 DEBUG:boto:Method: GET 2023-09-22T21:22:06.355 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj6 2023-09-22T21:22:06.355 DEBUG:boto:Data: 2023-09-22T21:22:06.355 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.355 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.355 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.355 DEBUG:boto:Params: {} 2023-09-22T21:22:06.356 DEBUG:boto:Token: None 2023-09-22T21:22:06.356 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/ 2023-09-22T21:22:06.356 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40= 2023-09-22T21:22:06.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40=', 'Content-Length': '0'} 2023-09-22T21:22:06.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f87e40f46fbfb533-00650e057e-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.361 DEBUG:boto:b'ffzdqu-47obj61000falseobj62023-09-22T21:21:22.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:06.361 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:06.361 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:06.361 DEBUG:boto:path=/ffzdqu-47/?prefix=obj6 2023-09-22T21:22:06.362 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj6 2023-09-22T21:22:06.362 DEBUG:boto:Method: GET 2023-09-22T21:22:06.362 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj6 2023-09-22T21:22:06.362 DEBUG:boto:Data: 2023-09-22T21:22:06.362 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.362 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.362 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.362 DEBUG:boto:Params: {} 2023-09-22T21:22:06.362 DEBUG:boto:Token: None 2023-09-22T21:22:06.363 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/ 2023-09-22T21:22:06.363 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40= 2023-09-22T21:22:06.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40=', 'Content-Length': '0'} 2023-09-22T21:22:06.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9a3ed4debd43dbe-00650e057e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.410 DEBUG:boto:b'ffzdqu-47obj61000falseobj62023-09-22T21:21:22.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:06.411 DEBUG:boto:path=/ffzdqu-47/obj6 2023-09-22T21:22:06.411 DEBUG:boto:auth_path=/ffzdqu-47/obj6 2023-09-22T21:22:06.411 DEBUG:boto:Method: HEAD 2023-09-22T21:22:06.411 DEBUG:boto:Path: /ffzdqu-47/obj6 2023-09-22T21:22:06.411 DEBUG:boto:Data: 2023-09-22T21:22:06.411 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.411 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.411 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.412 DEBUG:boto:Params: {} 2023-09-22T21:22:06.412 DEBUG:boto:Token: None 2023-09-22T21:22:06.412 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj6 2023-09-22T21:22:06.412 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:73mo5OndJIjf518ui30hIn6pVOE= 2023-09-22T21:22:06.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:73mo5OndJIjf518ui30hIn6pVOE=', 'Content-Length': '0'} 2023-09-22T21:22:06.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd2d86ed7b385efb-00650e057e-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.415 DEBUG:boto:path=/ffzdqu-47/obj6 2023-09-22T21:22:06.415 DEBUG:boto:auth_path=/ffzdqu-47/obj6 2023-09-22T21:22:06.415 DEBUG:boto:Method: HEAD 2023-09-22T21:22:06.415 DEBUG:boto:Path: /ffzdqu-47/obj6 2023-09-22T21:22:06.415 DEBUG:boto:Data: 2023-09-22T21:22:06.416 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.416 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.416 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.416 DEBUG:boto:Params: {} 2023-09-22T21:22:06.416 DEBUG:boto:Token: None 2023-09-22T21:22:06.416 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj6 2023-09-22T21:22:06.416 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:73mo5OndJIjf518ui30hIn6pVOE= 2023-09-22T21:22:06.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:73mo5OndJIjf518ui30hIn6pVOE=', 'Content-Length': '0'} 2023-09-22T21:22:06.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010d7f2e4dfb7b81f-00650e057e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-22T21:22:06.419 DEBUG:boto:path=/ffzdqu-47/obj6 2023-09-22T21:22:06.419 DEBUG:boto:auth_path=/ffzdqu-47/obj6 2023-09-22T21:22:06.420 DEBUG:boto:Method: GET 2023-09-22T21:22:06.420 DEBUG:boto:Path: /ffzdqu-47/obj6 2023-09-22T21:22:06.420 DEBUG:boto:Data: 2023-09-22T21:22:06.420 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.420 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.420 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.420 DEBUG:boto:Params: {} 2023-09-22T21:22:06.420 DEBUG:boto:Token: None 2023-09-22T21:22:06.420 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj6 2023-09-22T21:22:06.421 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9VUAqbvTH+2Kcn+eKpyWwJcTPAQ= 2023-09-22T21:22:06.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9VUAqbvTH+2Kcn+eKpyWwJcTPAQ=', 'Content-Length': '0'} 2023-09-22T21:22:06.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000061d429fbd8d2886-00650e057e-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.424 DEBUG:boto:path=/ffzdqu-47/obj6 2023-09-22T21:22:06.424 DEBUG:boto:auth_path=/ffzdqu-47/obj6 2023-09-22T21:22:06.424 DEBUG:boto:Method: GET 2023-09-22T21:22:06.424 DEBUG:boto:Path: /ffzdqu-47/obj6 2023-09-22T21:22:06.424 DEBUG:boto:Data: 2023-09-22T21:22:06.424 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.425 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.425 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.425 DEBUG:boto:Params: {} 2023-09-22T21:22:06.425 DEBUG:boto:Token: None 2023-09-22T21:22:06.425 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj6 2023-09-22T21:22:06.425 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9VUAqbvTH+2Kcn+eKpyWwJcTPAQ= 2023-09-22T21:22:06.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9VUAqbvTH+2Kcn+eKpyWwJcTPAQ=', 'Content-Length': '0'} 2023-09-22T21:22:06.428 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000462e3adef17e700b-00650e057e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.470 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:06.470 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:06.471 DEBUG:boto:path=/ffzdqu-47/?prefix=obj6 2023-09-22T21:22:06.471 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj6 2023-09-22T21:22:06.471 DEBUG:boto:Method: GET 2023-09-22T21:22:06.471 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj6 2023-09-22T21:22:06.471 DEBUG:boto:Data: 2023-09-22T21:22:06.471 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.471 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.471 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.471 DEBUG:boto:Params: {} 2023-09-22T21:22:06.472 DEBUG:boto:Token: None 2023-09-22T21:22:06.472 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/ 2023-09-22T21:22:06.472 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40= 2023-09-22T21:22:06.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40=', 'Content-Length': '0'} 2023-09-22T21:22:06.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff78237dd2e7949a-00650e057e-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.477 DEBUG:boto:b'ffzdqu-47obj61000falseobj62023-09-22T21:21:22.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:06.478 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:06.478 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:06.478 DEBUG:boto:path=/ffzdqu-47/?prefix=obj6 2023-09-22T21:22:06.478 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj6 2023-09-22T21:22:06.478 DEBUG:boto:Method: GET 2023-09-22T21:22:06.478 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj6 2023-09-22T21:22:06.478 DEBUG:boto:Data: 2023-09-22T21:22:06.479 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.479 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.479 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.479 DEBUG:boto:Params: {} 2023-09-22T21:22:06.479 DEBUG:boto:Token: None 2023-09-22T21:22:06.479 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/ 2023-09-22T21:22:06.480 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40= 2023-09-22T21:22:06.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40=', 'Content-Length': '0'} 2023-09-22T21:22:06.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c00aa6577635ef9e-00650e057e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.526 DEBUG:boto:b'ffzdqu-47obj61000falseobj62023-09-22T21:21:22.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:06.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-22T21:22:06.527 DEBUG:boto:path=/ffzdqu-47/obj7 2023-09-22T21:22:06.527 DEBUG:boto:auth_path=/ffzdqu-47/obj7 2023-09-22T21:22:06.527 DEBUG:boto:path=/ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.527 DEBUG:boto:auth_path=/ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.528 DEBUG:boto:Method: GET 2023-09-22T21:22:06.528 DEBUG:boto:Path: /ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.528 DEBUG:boto:Data: 2023-09-22T21:22:06.528 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.528 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.528 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.528 DEBUG:boto:Params: {} 2023-09-22T21:22:06.528 DEBUG:boto:Token: None 2023-09-22T21:22:06.529 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.529 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kblxAByXLJlueTI9/Cw/Z+RED0c= 2023-09-22T21:22:06.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kblxAByXLJlueTI9/Cw/Z+RED0c=', 'Content-Length': '0'} 2023-09-22T21:22:06.532 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000936a92b1a5e19e8f-00650e057e-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.532 DEBUG:boto:path=/ffzdqu-47/obj7 2023-09-22T21:22:06.532 DEBUG:boto:auth_path=/ffzdqu-47/obj7 2023-09-22T21:22:06.533 DEBUG:boto:path=/ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.533 DEBUG:boto:auth_path=/ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.533 DEBUG:boto:Method: GET 2023-09-22T21:22:06.533 DEBUG:boto:Path: /ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.533 DEBUG:boto:Data: 2023-09-22T21:22:06.533 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.533 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.533 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.533 DEBUG:boto:Params: {} 2023-09-22T21:22:06.534 DEBUG:boto:Token: None 2023-09-22T21:22:06.534 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.534 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kblxAByXLJlueTI9/Cw/Z+RED0c= 2023-09-22T21:22:06.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kblxAByXLJlueTI9/Cw/Z+RED0c=', 'Content-Length': '0'} 2023-09-22T21:22:06.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081e4b3fc94388d44-00650e057e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.537 DEBUG:boto:path=/ffzdqu-47/obj7 2023-09-22T21:22:06.538 DEBUG:boto:auth_path=/ffzdqu-47/obj7 2023-09-22T21:22:06.538 DEBUG:boto:path=/ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.538 DEBUG:boto:auth_path=/ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.538 DEBUG:boto:Method: HEAD 2023-09-22T21:22:06.538 DEBUG:boto:Path: /ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.538 DEBUG:boto:Data: 2023-09-22T21:22:06.538 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.538 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.538 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.539 DEBUG:boto:Params: {} 2023-09-22T21:22:06.539 DEBUG:boto:Token: None 2023-09-22T21:22:06.539 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.539 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CaKThJoglyiUoZoYwEe2zhK5IrM= 2023-09-22T21:22:06.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CaKThJoglyiUoZoYwEe2zhK5IrM=', 'Content-Length': '0'} 2023-09-22T21:22:06.542 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089fb2b58e9cb6bcd-00650e057e-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.543 DEBUG:boto:path=/ffzdqu-47/obj7 2023-09-22T21:22:06.543 DEBUG:boto:auth_path=/ffzdqu-47/obj7 2023-09-22T21:22:06.543 DEBUG:boto:path=/ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.543 DEBUG:boto:auth_path=/ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.543 DEBUG:boto:Method: HEAD 2023-09-22T21:22:06.543 DEBUG:boto:Path: /ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.543 DEBUG:boto:Data: 2023-09-22T21:22:06.544 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.544 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.544 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.544 DEBUG:boto:Params: {} 2023-09-22T21:22:06.544 DEBUG:boto:Token: None 2023-09-22T21:22:06.544 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.544 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CaKThJoglyiUoZoYwEe2zhK5IrM= 2023-09-22T21:22:06.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CaKThJoglyiUoZoYwEe2zhK5IrM=', 'Content-Length': '0'} 2023-09-22T21:22:06.547 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1c86a5092594407-00650e057e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-22T21:22:06.548 DEBUG:boto:path=/ffzdqu-47/obj7 2023-09-22T21:22:06.548 DEBUG:boto:auth_path=/ffzdqu-47/obj7 2023-09-22T21:22:06.548 DEBUG:boto:path=/ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.548 DEBUG:boto:auth_path=/ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.548 DEBUG:boto:Method: GET 2023-09-22T21:22:06.548 DEBUG:boto:Path: /ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.548 DEBUG:boto:Data: 2023-09-22T21:22:06.548 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.549 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.549 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.549 DEBUG:boto:Params: {} 2023-09-22T21:22:06.549 DEBUG:boto:Token: None 2023-09-22T21:22:06.549 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.549 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kblxAByXLJlueTI9/Cw/Z+RED0c= 2023-09-22T21:22:06.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kblxAByXLJlueTI9/Cw/Z+RED0c=', 'Content-Length': '0'} 2023-09-22T21:22:06.552 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9d584bf7aaa3041-00650e057e-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.594 DEBUG:boto:path=/ffzdqu-47/obj7 2023-09-22T21:22:06.595 DEBUG:boto:auth_path=/ffzdqu-47/obj7 2023-09-22T21:22:06.595 DEBUG:boto:path=/ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.595 DEBUG:boto:auth_path=/ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.595 DEBUG:boto:Method: GET 2023-09-22T21:22:06.595 DEBUG:boto:Path: /ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.595 DEBUG:boto:Data: 2023-09-22T21:22:06.595 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.595 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.596 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.596 DEBUG:boto:Params: {} 2023-09-22T21:22:06.596 DEBUG:boto:Token: None 2023-09-22T21:22:06.596 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj7?versionId=null 2023-09-22T21:22:06.596 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kblxAByXLJlueTI9/Cw/Z+RED0c= 2023-09-22T21:22:06.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kblxAByXLJlueTI9/Cw/Z+RED0c=', 'Content-Length': '0'} 2023-09-22T21:22:06.599 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af4ef767156ff9c8-00650e057e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.646 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:06.647 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:06.647 DEBUG:boto:path=/ffzdqu-47/?prefix=obj7 2023-09-22T21:22:06.647 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj7 2023-09-22T21:22:06.647 DEBUG:boto:Method: GET 2023-09-22T21:22:06.647 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj7 2023-09-22T21:22:06.647 DEBUG:boto:Data: 2023-09-22T21:22:06.647 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.647 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.647 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.648 DEBUG:boto:Params: {} 2023-09-22T21:22:06.648 DEBUG:boto:Token: None 2023-09-22T21:22:06.648 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/ 2023-09-22T21:22:06.648 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40= 2023-09-22T21:22:06.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40=', 'Content-Length': '0'} 2023-09-22T21:22:06.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbabd8ea2faf80fa-00650e057e-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.653 DEBUG:boto:b'ffzdqu-47obj71000falseobj72023-09-22T21:21:22.443Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:06.654 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:06.654 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:06.654 DEBUG:boto:path=/ffzdqu-47/?prefix=obj7 2023-09-22T21:22:06.654 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj7 2023-09-22T21:22:06.654 DEBUG:boto:Method: GET 2023-09-22T21:22:06.654 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj7 2023-09-22T21:22:06.654 DEBUG:boto:Data: 2023-09-22T21:22:06.654 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.655 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.655 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.655 DEBUG:boto:Params: {} 2023-09-22T21:22:06.655 DEBUG:boto:Token: None 2023-09-22T21:22:06.655 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/ 2023-09-22T21:22:06.655 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40= 2023-09-22T21:22:06.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40=', 'Content-Length': '0'} 2023-09-22T21:22:06.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068b882b8303ba4d8-00650e057e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.660 DEBUG:boto:b'ffzdqu-47obj71000falseobj72023-09-22T21:21:22.443Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:06.660 DEBUG:boto:path=/ffzdqu-47/obj7 2023-09-22T21:22:06.660 DEBUG:boto:auth_path=/ffzdqu-47/obj7 2023-09-22T21:22:06.661 DEBUG:boto:Method: HEAD 2023-09-22T21:22:06.661 DEBUG:boto:Path: /ffzdqu-47/obj7 2023-09-22T21:22:06.661 DEBUG:boto:Data: 2023-09-22T21:22:06.661 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.661 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.661 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.661 DEBUG:boto:Params: {} 2023-09-22T21:22:06.661 DEBUG:boto:Token: None 2023-09-22T21:22:06.662 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj7 2023-09-22T21:22:06.662 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PtYyjxuKlhmg53y/zgU9GzZIW+Y= 2023-09-22T21:22:06.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PtYyjxuKlhmg53y/zgU9GzZIW+Y=', 'Content-Length': '0'} 2023-09-22T21:22:06.664 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000881d430cd1c71e45-00650e057e-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.665 DEBUG:boto:path=/ffzdqu-47/obj7 2023-09-22T21:22:06.665 DEBUG:boto:auth_path=/ffzdqu-47/obj7 2023-09-22T21:22:06.665 DEBUG:boto:Method: HEAD 2023-09-22T21:22:06.665 DEBUG:boto:Path: /ffzdqu-47/obj7 2023-09-22T21:22:06.665 DEBUG:boto:Data: 2023-09-22T21:22:06.665 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.665 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.665 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.665 DEBUG:boto:Params: {} 2023-09-22T21:22:06.666 DEBUG:boto:Token: None 2023-09-22T21:22:06.666 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj7 2023-09-22T21:22:06.666 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PtYyjxuKlhmg53y/zgU9GzZIW+Y= 2023-09-22T21:22:06.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PtYyjxuKlhmg53y/zgU9GzZIW+Y=', 'Content-Length': '0'} 2023-09-22T21:22:06.669 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e4717d001beaf9b-00650e057e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-22T21:22:06.669 DEBUG:boto:path=/ffzdqu-47/obj7 2023-09-22T21:22:06.669 DEBUG:boto:auth_path=/ffzdqu-47/obj7 2023-09-22T21:22:06.669 DEBUG:boto:Method: GET 2023-09-22T21:22:06.669 DEBUG:boto:Path: /ffzdqu-47/obj7 2023-09-22T21:22:06.669 DEBUG:boto:Data: 2023-09-22T21:22:06.669 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.670 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.670 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.670 DEBUG:boto:Params: {} 2023-09-22T21:22:06.670 DEBUG:boto:Token: None 2023-09-22T21:22:06.670 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj7 2023-09-22T21:22:06.670 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Pku8ISuTHjJEfKblLEzPpL5Dfgg= 2023-09-22T21:22:06.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Pku8ISuTHjJEfKblLEzPpL5Dfgg=', 'Content-Length': '0'} 2023-09-22T21:22:06.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ab9aa1e3bbb3291-00650e057e-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.673 DEBUG:boto:path=/ffzdqu-47/obj7 2023-09-22T21:22:06.673 DEBUG:boto:auth_path=/ffzdqu-47/obj7 2023-09-22T21:22:06.673 DEBUG:boto:Method: GET 2023-09-22T21:22:06.673 DEBUG:boto:Path: /ffzdqu-47/obj7 2023-09-22T21:22:06.674 DEBUG:boto:Data: 2023-09-22T21:22:06.674 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.674 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.674 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.674 DEBUG:boto:Params: {} 2023-09-22T21:22:06.674 DEBUG:boto:Token: None 2023-09-22T21:22:06.674 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj7 2023-09-22T21:22:06.674 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Pku8ISuTHjJEfKblLEzPpL5Dfgg= 2023-09-22T21:22:06.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Pku8ISuTHjJEfKblLEzPpL5Dfgg=', 'Content-Length': '0'} 2023-09-22T21:22:06.677 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051794bf61ef03146-00650e057e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.722 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:06.723 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:06.723 DEBUG:boto:path=/ffzdqu-47/?prefix=obj7 2023-09-22T21:22:06.723 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj7 2023-09-22T21:22:06.723 DEBUG:boto:Method: GET 2023-09-22T21:22:06.723 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj7 2023-09-22T21:22:06.723 DEBUG:boto:Data: 2023-09-22T21:22:06.723 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.723 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.724 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.724 DEBUG:boto:Params: {} 2023-09-22T21:22:06.724 DEBUG:boto:Token: None 2023-09-22T21:22:06.724 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/ 2023-09-22T21:22:06.724 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40= 2023-09-22T21:22:06.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40=', 'Content-Length': '0'} 2023-09-22T21:22:06.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f24a881a2d5584d-00650e057e-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.729 DEBUG:boto:b'ffzdqu-47obj71000falseobj72023-09-22T21:21:22.443Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:06.730 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:06.730 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:06.730 DEBUG:boto:path=/ffzdqu-47/?prefix=obj7 2023-09-22T21:22:06.730 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj7 2023-09-22T21:22:06.730 DEBUG:boto:Method: GET 2023-09-22T21:22:06.730 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj7 2023-09-22T21:22:06.731 DEBUG:boto:Data: 2023-09-22T21:22:06.731 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.731 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.731 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.731 DEBUG:boto:Params: {} 2023-09-22T21:22:06.731 DEBUG:boto:Token: None 2023-09-22T21:22:06.731 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/ 2023-09-22T21:22:06.731 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40= 2023-09-22T21:22:06.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40=', 'Content-Length': '0'} 2023-09-22T21:22:06.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000272d28f5ef046497-00650e057e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.736 DEBUG:boto:b'ffzdqu-47obj71000falseobj72023-09-22T21:21:22.443Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:06.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-22T21:22:06.737 DEBUG:boto:path=/ffzdqu-47/obj8 2023-09-22T21:22:06.737 DEBUG:boto:auth_path=/ffzdqu-47/obj8 2023-09-22T21:22:06.737 DEBUG:boto:path=/ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.737 DEBUG:boto:auth_path=/ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.737 DEBUG:boto:Method: GET 2023-09-22T21:22:06.738 DEBUG:boto:Path: /ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.738 DEBUG:boto:Data: 2023-09-22T21:22:06.738 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.738 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.738 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.738 DEBUG:boto:Params: {} 2023-09-22T21:22:06.738 DEBUG:boto:Token: None 2023-09-22T21:22:06.738 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.738 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:d3dDPPvfeHUycv6faFCHyEFO+RM= 2023-09-22T21:22:06.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:d3dDPPvfeHUycv6faFCHyEFO+RM=', 'Content-Length': '0'} 2023-09-22T21:22:06.741 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b2a2244a9b8fd86-00650e057e-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.741 DEBUG:boto:path=/ffzdqu-47/obj8 2023-09-22T21:22:06.742 DEBUG:boto:auth_path=/ffzdqu-47/obj8 2023-09-22T21:22:06.742 DEBUG:boto:path=/ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.742 DEBUG:boto:auth_path=/ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.742 DEBUG:boto:Method: GET 2023-09-22T21:22:06.742 DEBUG:boto:Path: /ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.742 DEBUG:boto:Data: 2023-09-22T21:22:06.742 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.742 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.742 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.742 DEBUG:boto:Params: {} 2023-09-22T21:22:06.743 DEBUG:boto:Token: None 2023-09-22T21:22:06.743 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.743 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:d3dDPPvfeHUycv6faFCHyEFO+RM= 2023-09-22T21:22:06.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:d3dDPPvfeHUycv6faFCHyEFO+RM=', 'Content-Length': '0'} 2023-09-22T21:22:06.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f5902c0b423a6bd8-00650e057e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.746 DEBUG:boto:path=/ffzdqu-47/obj8 2023-09-22T21:22:06.746 DEBUG:boto:auth_path=/ffzdqu-47/obj8 2023-09-22T21:22:06.746 DEBUG:boto:path=/ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.746 DEBUG:boto:auth_path=/ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.746 DEBUG:boto:Method: HEAD 2023-09-22T21:22:06.747 DEBUG:boto:Path: /ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.747 DEBUG:boto:Data: 2023-09-22T21:22:06.747 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.747 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.747 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.747 DEBUG:boto:Params: {} 2023-09-22T21:22:06.747 DEBUG:boto:Token: None 2023-09-22T21:22:06.747 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.748 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qmTAKKuR9GGzjZMbo3qkNUKP2lc= 2023-09-22T21:22:06.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qmTAKKuR9GGzjZMbo3qkNUKP2lc=', 'Content-Length': '0'} 2023-09-22T21:22:06.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002134d66922fe5ab5-00650e057e-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.750 DEBUG:boto:path=/ffzdqu-47/obj8 2023-09-22T21:22:06.750 DEBUG:boto:auth_path=/ffzdqu-47/obj8 2023-09-22T21:22:06.750 DEBUG:boto:path=/ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.750 DEBUG:boto:auth_path=/ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.751 DEBUG:boto:Method: HEAD 2023-09-22T21:22:06.751 DEBUG:boto:Path: /ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.751 DEBUG:boto:Data: 2023-09-22T21:22:06.751 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.751 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.751 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.751 DEBUG:boto:Params: {} 2023-09-22T21:22:06.751 DEBUG:boto:Token: None 2023-09-22T21:22:06.751 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.752 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qmTAKKuR9GGzjZMbo3qkNUKP2lc= 2023-09-22T21:22:06.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qmTAKKuR9GGzjZMbo3qkNUKP2lc=', 'Content-Length': '0'} 2023-09-22T21:22:06.754 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081bdb0551212c831-00650e057e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-22T21:22:06.754 DEBUG:boto:path=/ffzdqu-47/obj8 2023-09-22T21:22:06.755 DEBUG:boto:auth_path=/ffzdqu-47/obj8 2023-09-22T21:22:06.755 DEBUG:boto:path=/ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.755 DEBUG:boto:auth_path=/ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.755 DEBUG:boto:Method: GET 2023-09-22T21:22:06.755 DEBUG:boto:Path: /ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.755 DEBUG:boto:Data: 2023-09-22T21:22:06.755 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.755 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.755 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.755 DEBUG:boto:Params: {} 2023-09-22T21:22:06.756 DEBUG:boto:Token: None 2023-09-22T21:22:06.756 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.756 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:d3dDPPvfeHUycv6faFCHyEFO+RM= 2023-09-22T21:22:06.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:d3dDPPvfeHUycv6faFCHyEFO+RM=', 'Content-Length': '0'} 2023-09-22T21:22:06.758 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000584eaf6d4a851f54-00650e057e-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.759 DEBUG:boto:path=/ffzdqu-47/obj8 2023-09-22T21:22:06.759 DEBUG:boto:auth_path=/ffzdqu-47/obj8 2023-09-22T21:22:06.759 DEBUG:boto:path=/ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.759 DEBUG:boto:auth_path=/ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.759 DEBUG:boto:Method: GET 2023-09-22T21:22:06.759 DEBUG:boto:Path: /ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.759 DEBUG:boto:Data: 2023-09-22T21:22:06.759 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.759 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.760 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.760 DEBUG:boto:Params: {} 2023-09-22T21:22:06.760 DEBUG:boto:Token: None 2023-09-22T21:22:06.760 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj8?versionId=null 2023-09-22T21:22:06.760 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:d3dDPPvfeHUycv6faFCHyEFO+RM= 2023-09-22T21:22:06.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:d3dDPPvfeHUycv6faFCHyEFO+RM=', 'Content-Length': '0'} 2023-09-22T21:22:06.763 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000370118ad75f36b5-00650e057e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.806 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:06.807 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:06.807 DEBUG:boto:path=/ffzdqu-47/?prefix=obj8 2023-09-22T21:22:06.807 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj8 2023-09-22T21:22:06.807 DEBUG:boto:Method: GET 2023-09-22T21:22:06.807 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj8 2023-09-22T21:22:06.807 DEBUG:boto:Data: 2023-09-22T21:22:06.807 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.807 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.807 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.808 DEBUG:boto:Params: {} 2023-09-22T21:22:06.808 DEBUG:boto:Token: None 2023-09-22T21:22:06.808 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/ 2023-09-22T21:22:06.808 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40= 2023-09-22T21:22:06.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40=', 'Content-Length': '0'} 2023-09-22T21:22:06.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000223b39b4b3a6b215-00650e057e-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.815 DEBUG:boto:b'ffzdqu-47obj81000falseobj82023-09-22T21:21:22.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:06.816 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:06.816 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:06.816 DEBUG:boto:path=/ffzdqu-47/?prefix=obj8 2023-09-22T21:22:06.816 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj8 2023-09-22T21:22:06.816 DEBUG:boto:Method: GET 2023-09-22T21:22:06.816 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj8 2023-09-22T21:22:06.816 DEBUG:boto:Data: 2023-09-22T21:22:06.816 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.816 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.817 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.817 DEBUG:boto:Params: {} 2023-09-22T21:22:06.817 DEBUG:boto:Token: None 2023-09-22T21:22:06.817 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/ 2023-09-22T21:22:06.817 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40= 2023-09-22T21:22:06.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40=', 'Content-Length': '0'} 2023-09-22T21:22:06.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e2ee98c3b13cf51-00650e057e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.870 DEBUG:boto:b'ffzdqu-47obj81000falseobj82023-09-22T21:21:22.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:06.871 DEBUG:boto:path=/ffzdqu-47/obj8 2023-09-22T21:22:06.871 DEBUG:boto:auth_path=/ffzdqu-47/obj8 2023-09-22T21:22:06.871 DEBUG:boto:Method: HEAD 2023-09-22T21:22:06.871 DEBUG:boto:Path: /ffzdqu-47/obj8 2023-09-22T21:22:06.871 DEBUG:boto:Data: 2023-09-22T21:22:06.871 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.872 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.872 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.872 DEBUG:boto:Params: {} 2023-09-22T21:22:06.872 DEBUG:boto:Token: None 2023-09-22T21:22:06.872 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj8 2023-09-22T21:22:06.872 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i56HBVIdwm5OOe3Xv85e5S0IvvI= 2023-09-22T21:22:06.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i56HBVIdwm5OOe3Xv85e5S0IvvI=', 'Content-Length': '0'} 2023-09-22T21:22:06.875 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094a77bb5acba06fc-00650e057e-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.875 DEBUG:boto:path=/ffzdqu-47/obj8 2023-09-22T21:22:06.875 DEBUG:boto:auth_path=/ffzdqu-47/obj8 2023-09-22T21:22:06.875 DEBUG:boto:Method: HEAD 2023-09-22T21:22:06.875 DEBUG:boto:Path: /ffzdqu-47/obj8 2023-09-22T21:22:06.876 DEBUG:boto:Data: 2023-09-22T21:22:06.876 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.876 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.876 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.876 DEBUG:boto:Params: {} 2023-09-22T21:22:06.876 DEBUG:boto:Token: None 2023-09-22T21:22:06.876 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj8 2023-09-22T21:22:06.876 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i56HBVIdwm5OOe3Xv85e5S0IvvI= 2023-09-22T21:22:06.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i56HBVIdwm5OOe3Xv85e5S0IvvI=', 'Content-Length': '0'} 2023-09-22T21:22:06.879 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3ef4bda16ffd47b-00650e057e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-22T21:22:06.880 DEBUG:boto:path=/ffzdqu-47/obj8 2023-09-22T21:22:06.880 DEBUG:boto:auth_path=/ffzdqu-47/obj8 2023-09-22T21:22:06.880 DEBUG:boto:Method: GET 2023-09-22T21:22:06.880 DEBUG:boto:Path: /ffzdqu-47/obj8 2023-09-22T21:22:06.880 DEBUG:boto:Data: 2023-09-22T21:22:06.880 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.880 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.880 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.881 DEBUG:boto:Params: {} 2023-09-22T21:22:06.881 DEBUG:boto:Token: None 2023-09-22T21:22:06.881 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj8 2023-09-22T21:22:06.881 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GgVs5CdMZPDRJ0fUC3RCnklnojw= 2023-09-22T21:22:06.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GgVs5CdMZPDRJ0fUC3RCnklnojw=', 'Content-Length': '0'} 2023-09-22T21:22:06.884 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9b9ad45f35a580d-00650e057e-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.926 DEBUG:boto:path=/ffzdqu-47/obj8 2023-09-22T21:22:06.927 DEBUG:boto:auth_path=/ffzdqu-47/obj8 2023-09-22T21:22:06.927 DEBUG:boto:Method: GET 2023-09-22T21:22:06.927 DEBUG:boto:Path: /ffzdqu-47/obj8 2023-09-22T21:22:06.927 DEBUG:boto:Data: 2023-09-22T21:22:06.927 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.927 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.927 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.927 DEBUG:boto:Params: {} 2023-09-22T21:22:06.928 DEBUG:boto:Token: None 2023-09-22T21:22:06.928 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/obj8 2023-09-22T21:22:06.928 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GgVs5CdMZPDRJ0fUC3RCnklnojw= 2023-09-22T21:22:06.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GgVs5CdMZPDRJ0fUC3RCnklnojw=', 'Content-Length': '0'} 2023-09-22T21:22:06.931 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:21:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-47:f132bdb9-20a7-4073-a932-8f99da20691e.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000af5ca66f19f83aa-00650e057e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.974 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:06.975 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:06.975 DEBUG:boto:path=/ffzdqu-47/?prefix=obj8 2023-09-22T21:22:06.975 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj8 2023-09-22T21:22:06.975 DEBUG:boto:Method: GET 2023-09-22T21:22:06.975 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj8 2023-09-22T21:22:06.975 DEBUG:boto:Data: 2023-09-22T21:22:06.975 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.975 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:06.976 DEBUG:boto:Port: 8001 2023-09-22T21:22:06.976 DEBUG:boto:Params: {} 2023-09-22T21:22:06.976 DEBUG:boto:Token: None 2023-09-22T21:22:06.976 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/ 2023-09-22T21:22:06.976 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40= 2023-09-22T21:22:06.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40=', 'Content-Length': '0'} 2023-09-22T21:22:06.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cdabc122577e209-00650e057e-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:06.981 DEBUG:boto:b'ffzdqu-47obj81000falseobj82023-09-22T21:21:22.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:06.982 DEBUG:boto:path=/ffzdqu-47/ 2023-09-22T21:22:06.982 DEBUG:boto:auth_path=/ffzdqu-47/ 2023-09-22T21:22:06.982 DEBUG:boto:path=/ffzdqu-47/?prefix=obj8 2023-09-22T21:22:06.982 DEBUG:boto:auth_path=/ffzdqu-47/?prefix=obj8 2023-09-22T21:22:06.983 DEBUG:boto:Method: GET 2023-09-22T21:22:06.983 DEBUG:boto:Path: /ffzdqu-47/?prefix=obj8 2023-09-22T21:22:06.983 DEBUG:boto:Data: 2023-09-22T21:22:06.983 DEBUG:boto:Headers: {} 2023-09-22T21:22:06.983 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:06.983 DEBUG:boto:Port: 8000 2023-09-22T21:22:06.983 DEBUG:boto:Params: {} 2023-09-22T21:22:06.983 DEBUG:boto:Token: None 2023-09-22T21:22:06.983 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:06 GMT /ffzdqu-47/ 2023-09-22T21:22:06.984 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40= 2023-09-22T21:22:06.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bhp9DW51Nkc9UmNv07XxUfQpl40=', 'Content-Length': '0'} 2023-09-22T21:22:06.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005661a1506dc631f8-00650e057e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:07.030 DEBUG:boto:b'ffzdqu-47obj81000falseobj82023-09-22T21:21:22.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:07.031 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-47 zones={a2, a1} 2023-09-22T21:22:07.031 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-09-22T21:22:07.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:07.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:07.032 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:22:07.075 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:07.075 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:07.138 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"00000000000000000000:00000000000000002903","last_update":"2023-09-22T21:21:39.044729Z"},{"marker":"00000000000000000000:00000000000000006987","last_update":"2023-09-22T21:21:41.167401Z"},{"marker":"00000000000000000000:00000000000000005159","last_update":"2023-09-22T21:21:22.250572Z"},{"marker":"00000000000000000000:00000000000000004316","last_update":"2023-09-22T21:21:39.097454Z"}] 2023-09-22T21:22:07.139 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000002903', 1: '00000000000000000000:00000000000000006987', 2: '00000000000000000000:00000000000000005159', 3: '00000000000000000000:00000000000000004316'} 2023-09-22T21:22:07.139 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-22T21:22:07.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:07.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:07.139 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:22:07.217 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:07.218 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:07.273 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002903","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:21:39.044729Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006987","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:21:41.167401Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005159","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:21:22.250572Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004316","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:21:39.097454Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:22:07Z"} 2023-09-22T21:22:07.274 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002903","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:21:39.044729Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006987","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:21:41.167401Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005159","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:21:22.250572Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004316","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:21:39.097454Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:22:07Z"} 2023-09-22T21:22:07.274 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002903', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:21:39.044729Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006987', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:21:41.167401Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005159', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:21:22.250572Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004316', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:21:39.097454Z'}}] 2023-09-22T21:22:07.274 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002903', 1: '00000000000000000000:00000000000000006987', 2: '00000000000000000000:00000000000000005159', 3: '00000000000000000000:00000000000000004316'} 2023-09-22T21:22:07.274 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002903', 1: '00000000000000000000:00000000000000006987', 2: '00000000000000000000:00000000000000005159', 3: '00000000000000000000:00000000000000004316'} 2023-09-22T21:22:07.274 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-22T21:22:07.274 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-09-22T21:22:07.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:07.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:07.275 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:22:07.351 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:07.351 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:07.412 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"00000000000000000000:00000000000000001775","last_update":"2023-09-22T21:21:37.936328Z"},{"marker":"00000000000000000000:00000000000000006000","last_update":"2023-09-22T21:21:38.977193Z"},{"marker":"00000000000000000000:00000000000000002483","last_update":"2023-09-22T21:20:38.891745Z"},{"marker":"00000000000000000000:00000000000000002620","last_update":"2023-09-22T21:21:37.881495Z"}] 2023-09-22T21:22:07.412 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000001775', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000002483', 3: '00000000000000000000:00000000000000002620'} 2023-09-22T21:22:07.412 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-09-22T21:22:07.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:07.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:07.413 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:22:07.491 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:07.491 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:07.544 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10476880264182873081},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001775","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:21:37.936328Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006000","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:21:38.977193Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002483","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:20:38.891745Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002620","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:21:37.881495Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:22:07Z"} 2023-09-22T21:22:07.544 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10476880264182873081},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001775","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:21:37.936328Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006000","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:21:38.977193Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002483","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:20:38.891745Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002620","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:21:37.881495Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:22:07Z"} 2023-09-22T21:22:07.544 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001775', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:21:37.936328Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006000', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:21:38.977193Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002483', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:20:38.891745Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002620', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:21:37.881495Z'}}] 2023-09-22T21:22:07.545 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001775', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000002483', 3: '00000000000000000000:00000000000000002620'} 2023-09-22T21:22:07.545 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001775', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000002483', 3: '00000000000000000000:00000000000000002620'} 2023-09-22T21:22:07.545 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-09-22T21:22:07.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2023-09-22T21:22:07.546 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-48 2023-09-22T21:22:07.546 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:07.547 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:07.547 DEBUG:boto:Method: PUT 2023-09-22T21:22:07.547 DEBUG:boto:Path: /ffzdqu-48/ 2023-09-22T21:22:07.547 DEBUG:boto:Data: 2023-09-22T21:22:07.547 DEBUG:boto:Headers: {} 2023-09-22T21:22:07.547 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:07.547 DEBUG:boto:Port: 8000 2023-09-22T21:22:07.547 DEBUG:boto:Params: {} 2023-09-22T21:22:07.547 DEBUG:boto:Token: None 2023-09-22T21:22:07.548 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:22:07 GMT /ffzdqu-48/ 2023-09-22T21:22:07.548 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ExdeJdGiqYYiKJEAwY5apMGBd9c= 2023-09-22T21:22:07.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ExdeJdGiqYYiKJEAwY5apMGBd9c=', 'Content-Length': '0'} 2023-09-22T21:22:07.565 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008469627b33ddbbea-00650e057f-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:07.566 INFO:rgw_multi.tests:create bucket zone=a2 name=ffzdqu-49 2023-09-22T21:22:07.566 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:07.566 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:07.566 DEBUG:boto:Method: PUT 2023-09-22T21:22:07.566 DEBUG:boto:Path: /ffzdqu-49/ 2023-09-22T21:22:07.566 DEBUG:boto:Data: 2023-09-22T21:22:07.566 DEBUG:boto:Headers: {} 2023-09-22T21:22:07.566 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:07.567 DEBUG:boto:Port: 8001 2023-09-22T21:22:07.567 DEBUG:boto:Params: {} 2023-09-22T21:22:07.567 DEBUG:boto:Token: None 2023-09-22T21:22:07.567 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:22:07 GMT /ffzdqu-49/ 2023-09-22T21:22:07.567 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZOSReTO1GqJxhsiejp8zbOqBtKw= 2023-09-22T21:22:07.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZOSReTO1GqJxhsiejp8zbOqBtKw=', 'Content-Length': '0'} 2023-09-22T21:22:07.596 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002e1a3babc01eb4a8-00650e057f-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:07.596 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:07.597 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:07.597 DEBUG:boto:Method: HEAD 2023-09-22T21:22:07.597 DEBUG:boto:Path: /ffzdqu-48/ 2023-09-22T21:22:07.597 DEBUG:boto:Data: 2023-09-22T21:22:07.597 DEBUG:boto:Headers: {} 2023-09-22T21:22:07.597 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:07.597 DEBUG:boto:Port: 8000 2023-09-22T21:22:07.597 DEBUG:boto:Params: {} 2023-09-22T21:22:07.598 DEBUG:boto:Token: None 2023-09-22T21:22:07.598 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:07 GMT /ffzdqu-48/ 2023-09-22T21:22:07.598 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MZvR+leK7O7XvXLWNcIQ6zHpmNs= 2023-09-22T21:22:07.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MZvR+leK7O7XvXLWNcIQ6zHpmNs=', 'Content-Length': '0'} 2023-09-22T21:22:07.600 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008963419c0146238f-00650e057f-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:07.601 DEBUG:boto:path=/ffzdqu-48/obj1 2023-09-22T21:22:07.601 DEBUG:boto:auth_path=/ffzdqu-48/obj1 2023-09-22T21:22:07.601 DEBUG:boto:Method: PUT 2023-09-22T21:22:07.601 DEBUG:boto:Path: /ffzdqu-48/obj1 2023-09-22T21:22:07.601 DEBUG:boto:Data: 2023-09-22T21:22:07.601 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:22:07.601 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:07.601 DEBUG:boto:Port: 8000 2023-09-22T21:22:07.602 DEBUG:boto:Params: {} 2023-09-22T21:22:07.602 DEBUG:boto:Token: None 2023-09-22T21:22:07.602 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:22:07 GMT /ffzdqu-48/obj1 2023-09-22T21:22:07.602 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PF2DA645xvrBAgfLMJpQo9QvREI= 2023-09-22T21:22:07.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:22:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PF2DA645xvrBAgfLMJpQo9QvREI='} 2023-09-22T21:22:07.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000076e678cece729b0a-00650e057f-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:07.651 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:07.651 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:07.651 DEBUG:boto:Method: HEAD 2023-09-22T21:22:07.651 DEBUG:boto:Path: /ffzdqu-48/ 2023-09-22T21:22:07.651 DEBUG:boto:Data: 2023-09-22T21:22:07.651 DEBUG:boto:Headers: {} 2023-09-22T21:22:07.651 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:07.651 DEBUG:boto:Port: 8000 2023-09-22T21:22:07.651 DEBUG:boto:Params: {} 2023-09-22T21:22:07.652 DEBUG:boto:Token: None 2023-09-22T21:22:07.652 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:07 GMT /ffzdqu-48/ 2023-09-22T21:22:07.652 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MZvR+leK7O7XvXLWNcIQ6zHpmNs= 2023-09-22T21:22:07.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MZvR+leK7O7XvXLWNcIQ6zHpmNs=', 'Content-Length': '0'} 2023-09-22T21:22:07.656 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c053c111f4051952-00650e057f-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:07.656 DEBUG:boto:path=/ffzdqu-48/obj2 2023-09-22T21:22:07.657 DEBUG:boto:auth_path=/ffzdqu-48/obj2 2023-09-22T21:22:07.657 DEBUG:boto:Method: PUT 2023-09-22T21:22:07.657 DEBUG:boto:Path: /ffzdqu-48/obj2 2023-09-22T21:22:07.657 DEBUG:boto:Data: 2023-09-22T21:22:07.657 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:22:07.657 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:07.657 DEBUG:boto:Port: 8000 2023-09-22T21:22:07.657 DEBUG:boto:Params: {} 2023-09-22T21:22:07.658 DEBUG:boto:Token: None 2023-09-22T21:22:07.658 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:22:07 GMT /ffzdqu-48/obj2 2023-09-22T21:22:07.658 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SisYSBbPRInQ0fTBDN4TBxuvs/M= 2023-09-22T21:22:07.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:22:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SisYSBbPRInQ0fTBDN4TBxuvs/M='} 2023-09-22T21:22:07.670 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e84472b82aa4336-00650e057f-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:07.670 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:07.670 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:07.670 DEBUG:boto:Method: HEAD 2023-09-22T21:22:07.670 DEBUG:boto:Path: /ffzdqu-48/ 2023-09-22T21:22:07.670 DEBUG:boto:Data: 2023-09-22T21:22:07.671 DEBUG:boto:Headers: {} 2023-09-22T21:22:07.671 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:07.671 DEBUG:boto:Port: 8000 2023-09-22T21:22:07.671 DEBUG:boto:Params: {} 2023-09-22T21:22:07.671 DEBUG:boto:Token: None 2023-09-22T21:22:07.671 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:07 GMT /ffzdqu-48/ 2023-09-22T21:22:07.671 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MZvR+leK7O7XvXLWNcIQ6zHpmNs= 2023-09-22T21:22:07.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MZvR+leK7O7XvXLWNcIQ6zHpmNs=', 'Content-Length': '0'} 2023-09-22T21:22:07.677 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025e94002d12e7267-00650e057f-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:07.677 DEBUG:boto:path=/ffzdqu-48/obj3 2023-09-22T21:22:07.677 DEBUG:boto:auth_path=/ffzdqu-48/obj3 2023-09-22T21:22:07.677 DEBUG:boto:Method: PUT 2023-09-22T21:22:07.677 DEBUG:boto:Path: /ffzdqu-48/obj3 2023-09-22T21:22:07.677 DEBUG:boto:Data: 2023-09-22T21:22:07.678 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:22:07.678 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:07.678 DEBUG:boto:Port: 8000 2023-09-22T21:22:07.678 DEBUG:boto:Params: {} 2023-09-22T21:22:07.678 DEBUG:boto:Token: None 2023-09-22T21:22:07.678 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:22:07 GMT /ffzdqu-48/obj3 2023-09-22T21:22:07.679 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Oahi/ed+XG+YCooEXN2DKue/O+Q= 2023-09-22T21:22:07.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:22:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Oahi/ed+XG+YCooEXN2DKue/O+Q='} 2023-09-22T21:22:07.722 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000430d1a65164c9b91-00650e057f-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:07.723 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:07.723 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:07.723 DEBUG:boto:Method: HEAD 2023-09-22T21:22:07.723 DEBUG:boto:Path: /ffzdqu-48/ 2023-09-22T21:22:07.723 DEBUG:boto:Data: 2023-09-22T21:22:07.723 DEBUG:boto:Headers: {} 2023-09-22T21:22:07.723 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:07.724 DEBUG:boto:Port: 8000 2023-09-22T21:22:07.724 DEBUG:boto:Params: {} 2023-09-22T21:22:07.724 DEBUG:boto:Token: None 2023-09-22T21:22:07.724 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:07 GMT /ffzdqu-48/ 2023-09-22T21:22:07.724 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MZvR+leK7O7XvXLWNcIQ6zHpmNs= 2023-09-22T21:22:07.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MZvR+leK7O7XvXLWNcIQ6zHpmNs=', 'Content-Length': '0'} 2023-09-22T21:22:07.729 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051a9963662e4a99d-00650e057f-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:07.729 DEBUG:boto:path=/ffzdqu-48/obj4 2023-09-22T21:22:07.729 DEBUG:boto:auth_path=/ffzdqu-48/obj4 2023-09-22T21:22:07.729 DEBUG:boto:Method: PUT 2023-09-22T21:22:07.730 DEBUG:boto:Path: /ffzdqu-48/obj4 2023-09-22T21:22:07.730 DEBUG:boto:Data: 2023-09-22T21:22:07.730 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:22:07.730 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:07.730 DEBUG:boto:Port: 8000 2023-09-22T21:22:07.730 DEBUG:boto:Params: {} 2023-09-22T21:22:07.730 DEBUG:boto:Token: None 2023-09-22T21:22:07.731 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:22:07 GMT /ffzdqu-48/obj4 2023-09-22T21:22:07.731 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PD/GXoTjmsrY2ex8vgrmMid+CB4= 2023-09-22T21:22:07.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:22:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PD/GXoTjmsrY2ex8vgrmMid+CB4='} 2023-09-22T21:22:07.742 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f9e126f0722d200-00650e057f-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:07.742 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:07.743 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:07.743 DEBUG:boto:Method: HEAD 2023-09-22T21:22:07.743 DEBUG:boto:Path: /ffzdqu-49/ 2023-09-22T21:22:07.743 DEBUG:boto:Data: 2023-09-22T21:22:07.743 DEBUG:boto:Headers: {} 2023-09-22T21:22:07.743 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:07.744 DEBUG:boto:Port: 8001 2023-09-22T21:22:07.744 DEBUG:boto:Params: {} 2023-09-22T21:22:07.744 DEBUG:boto:Token: None 2023-09-22T21:22:07.744 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:07 GMT /ffzdqu-49/ 2023-09-22T21:22:07.744 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IOIifQOdzEtzdacp7odb584U0Wk= 2023-09-22T21:22:07.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IOIifQOdzEtzdacp7odb584U0Wk=', 'Content-Length': '0'} 2023-09-22T21:22:07.748 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b48c65fd08c8b884-00650e057f-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:07.748 DEBUG:boto:path=/ffzdqu-49/obj1 2023-09-22T21:22:07.748 DEBUG:boto:auth_path=/ffzdqu-49/obj1 2023-09-22T21:22:07.748 DEBUG:boto:Method: PUT 2023-09-22T21:22:07.748 DEBUG:boto:Path: /ffzdqu-49/obj1 2023-09-22T21:22:07.748 DEBUG:boto:Data: 2023-09-22T21:22:07.749 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:22:07.749 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:07.749 DEBUG:boto:Port: 8001 2023-09-22T21:22:07.749 DEBUG:boto:Params: {} 2023-09-22T21:22:07.749 DEBUG:boto:Token: None 2023-09-22T21:22:07.749 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:22:07 GMT /ffzdqu-49/obj1 2023-09-22T21:22:07.749 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:j7czrVCp+3l0vzELmmshBqGSR8U= 2023-09-22T21:22:07.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:22:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:j7czrVCp+3l0vzELmmshBqGSR8U='} 2023-09-22T21:22:07.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000196f65775b1fd6c8-00650e057f-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:07.762 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:07.762 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:07.763 DEBUG:boto:Method: HEAD 2023-09-22T21:22:07.763 DEBUG:boto:Path: /ffzdqu-49/ 2023-09-22T21:22:07.763 DEBUG:boto:Data: 2023-09-22T21:22:07.763 DEBUG:boto:Headers: {} 2023-09-22T21:22:07.763 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:07.763 DEBUG:boto:Port: 8001 2023-09-22T21:22:07.763 DEBUG:boto:Params: {} 2023-09-22T21:22:07.763 DEBUG:boto:Token: None 2023-09-22T21:22:07.764 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:07 GMT /ffzdqu-49/ 2023-09-22T21:22:07.764 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IOIifQOdzEtzdacp7odb584U0Wk= 2023-09-22T21:22:07.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IOIifQOdzEtzdacp7odb584U0Wk=', 'Content-Length': '0'} 2023-09-22T21:22:07.767 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ca48b2c5ac84efb-00650e057f-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:07.768 DEBUG:boto:path=/ffzdqu-49/obj2 2023-09-22T21:22:07.768 DEBUG:boto:auth_path=/ffzdqu-49/obj2 2023-09-22T21:22:07.768 DEBUG:boto:Method: PUT 2023-09-22T21:22:07.768 DEBUG:boto:Path: /ffzdqu-49/obj2 2023-09-22T21:22:07.768 DEBUG:boto:Data: 2023-09-22T21:22:07.768 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:22:07.768 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:07.769 DEBUG:boto:Port: 8001 2023-09-22T21:22:07.769 DEBUG:boto:Params: {} 2023-09-22T21:22:07.769 DEBUG:boto:Token: None 2023-09-22T21:22:07.769 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:22:07 GMT /ffzdqu-49/obj2 2023-09-22T21:22:07.769 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+EjT1p54wOnoZltpI7CGv6elMPI= 2023-09-22T21:22:07.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:22:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+EjT1p54wOnoZltpI7CGv6elMPI='} 2023-09-22T21:22:07.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000710aa1761db3ca4d-00650e057f-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:07.815 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:07.815 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:07.815 DEBUG:boto:Method: HEAD 2023-09-22T21:22:07.815 DEBUG:boto:Path: /ffzdqu-49/ 2023-09-22T21:22:07.815 DEBUG:boto:Data: 2023-09-22T21:22:07.815 DEBUG:boto:Headers: {} 2023-09-22T21:22:07.816 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:07.816 DEBUG:boto:Port: 8001 2023-09-22T21:22:07.816 DEBUG:boto:Params: {} 2023-09-22T21:22:07.816 DEBUG:boto:Token: None 2023-09-22T21:22:07.816 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:07 GMT /ffzdqu-49/ 2023-09-22T21:22:07.816 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IOIifQOdzEtzdacp7odb584U0Wk= 2023-09-22T21:22:07.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IOIifQOdzEtzdacp7odb584U0Wk=', 'Content-Length': '0'} 2023-09-22T21:22:07.820 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000525aeeaad66bd980-00650e057f-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:07.821 DEBUG:boto:path=/ffzdqu-49/obj3 2023-09-22T21:22:07.821 DEBUG:boto:auth_path=/ffzdqu-49/obj3 2023-09-22T21:22:07.821 DEBUG:boto:Method: PUT 2023-09-22T21:22:07.821 DEBUG:boto:Path: /ffzdqu-49/obj3 2023-09-22T21:22:07.821 DEBUG:boto:Data: 2023-09-22T21:22:07.821 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:22:07.821 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:07.822 DEBUG:boto:Port: 8001 2023-09-22T21:22:07.822 DEBUG:boto:Params: {} 2023-09-22T21:22:07.822 DEBUG:boto:Token: None 2023-09-22T21:22:07.822 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:22:07 GMT /ffzdqu-49/obj3 2023-09-22T21:22:07.822 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0Of/QajpOv8M/E8TDaVu8cOOq70= 2023-09-22T21:22:07.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:22:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0Of/QajpOv8M/E8TDaVu8cOOq70='} 2023-09-22T21:22:07.835 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b70965cc37d1db8d-00650e057f-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:07.835 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:07.835 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:07.835 DEBUG:boto:Method: HEAD 2023-09-22T21:22:07.836 DEBUG:boto:Path: /ffzdqu-49/ 2023-09-22T21:22:07.836 DEBUG:boto:Data: 2023-09-22T21:22:07.836 DEBUG:boto:Headers: {} 2023-09-22T21:22:07.836 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:07.836 DEBUG:boto:Port: 8001 2023-09-22T21:22:07.836 DEBUG:boto:Params: {} 2023-09-22T21:22:07.836 DEBUG:boto:Token: None 2023-09-22T21:22:07.836 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:07 GMT /ffzdqu-49/ 2023-09-22T21:22:07.837 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IOIifQOdzEtzdacp7odb584U0Wk= 2023-09-22T21:22:07.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IOIifQOdzEtzdacp7odb584U0Wk=', 'Content-Length': '0'} 2023-09-22T21:22:07.840 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b094169bd2631f68-00650e057f-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:07.841 DEBUG:boto:path=/ffzdqu-49/obj4 2023-09-22T21:22:07.841 DEBUG:boto:auth_path=/ffzdqu-49/obj4 2023-09-22T21:22:07.841 DEBUG:boto:Method: PUT 2023-09-22T21:22:07.841 DEBUG:boto:Path: /ffzdqu-49/obj4 2023-09-22T21:22:07.841 DEBUG:boto:Data: 2023-09-22T21:22:07.841 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:22:07.841 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:07.841 DEBUG:boto:Port: 8001 2023-09-22T21:22:07.842 DEBUG:boto:Params: {} 2023-09-22T21:22:07.842 DEBUG:boto:Token: None 2023-09-22T21:22:07.842 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:22:07 GMT /ffzdqu-49/obj4 2023-09-22T21:22:07.842 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bA6X7i93KQrLuuCN4Y28VRdgl0w= 2023-09-22T21:22:07.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:22:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bA6X7i93KQrLuuCN4Y28VRdgl0w='} 2023-09-22T21:22:07.887 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005d92d4102ef724d9-00650e057f-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:07.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:07.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:07.888 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:22:07.929 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:07.930 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:07.967 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:22:07.984 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695417727.589558_369.1","last_update":"2023-09-22T21:22:07.589558Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:22:07.985 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417727.589558_369.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:22:07.985 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:22:07.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:07.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:07.985 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:22:08.068 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:08.068 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:08.104 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:08.102+0000 7f189b3d6a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:22:08.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:08.106+0000 7f189b3d6a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:22:08.123 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417727.560648_361.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:07.560648Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:22:08Z"} 2023-09-22T21:22:08.123 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417727.560648_361.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:07.560648Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:22:08Z"} 2023-09-22T21:22:08.123 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417727.560648_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:22:07.560648Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:22:08.123 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417727.589558_369.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:22:08.123 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417727.560648_361.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:22:08.124 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417727.589558_369.1 target=1_1695417727.560648_361.1 2023-09-22T21:22:13.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:13.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:13.125 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:22:13.170 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:13.171 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:13.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:13.206+0000 7fb7fe315a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:22:13.210 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:13.210+0000 7fb7fe315a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:22:13.227 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417727.560648_361.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:07.560648Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:22:13Z"} 2023-09-22T21:22:13.228 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417727.560648_361.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:07.560648Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:22:13Z"} 2023-09-22T21:22:13.228 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417727.560648_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:22:07.560648Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:22:13.228 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417727.589558_369.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:22:13.228 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417727.560648_361.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:22:13.228 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417727.589558_369.1 target=1_1695417727.560648_361.1 2023-09-22T21:22:18.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:18.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:18.229 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:22:18.272 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:18.273 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:18.309 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:18.306+0000 7fc67a8e2a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:22:18.309 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:18.310+0000 7fc67a8e2a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:22:18.328 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417727.589558_369.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:07.589558Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:22:18Z"} 2023-09-22T21:22:18.328 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417727.589558_369.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:07.589558Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:22:18Z"} 2023-09-22T21:22:18.328 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417727.589558_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:22:07.589558Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:22:18.328 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417727.589558_369.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:22:18.329 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417727.589558_369.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:22:18.329 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:22:18.329 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-48 2023-09-22T21:22:18.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:22:18.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:22:18.329 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:22:18.413 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:18.414 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:18.455 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:18.454+0000 7fc3af5e4a40 1 waiting to reach incremental sync.. 2023-09-22T21:22:23.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:23.458+0000 7fc3af5e4a40 1 bucket sync caught up with source: 2023-09-22T21:22:23.458 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.23.6, , 00000000001.20.6, , 00000000001.18.6, , , 00000000001.33.6, , , ] 2023-09-22T21:22:23.458 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.23.6, , 00000000001.20.6, , 00000000001.18.6, , , 00000000001.33.6, , , ] 2023-09-22T21:22:23.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:23.458+0000 7fc3af5e4a40 0 bucket checkpoint complete 2023-09-22T21:22:23.491 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-48 2023-09-22T21:22:23.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:22:23.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:22:23.492 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:22:23.540 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:23.541 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:23.583 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:23.582+0000 7fcf9f050a40 1 bucket sync caught up with source: 2023-09-22T21:22:23.584 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.130.6, , 00000000001.172.6, , 00000000001.173.6, , , 00000000001.148.6, , , ] 2023-09-22T21:22:23.584 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.130.6, , 00000000001.172.6, , 00000000001.173.6, , , 00000000001.148.6, , , ] 2023-09-22T21:22:23.584 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:23.582+0000 7fcf9f050a40 0 bucket checkpoint complete 2023-09-22T21:22:23.601 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-48 zones={a2, a1} 2023-09-22T21:22:23.601 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:23.601 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:23.601 DEBUG:boto:Method: HEAD 2023-09-22T21:22:23.601 DEBUG:boto:Path: /ffzdqu-48/ 2023-09-22T21:22:23.601 DEBUG:boto:Data: 2023-09-22T21:22:23.601 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.602 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:23.602 DEBUG:boto:Port: 8001 2023-09-22T21:22:23.602 DEBUG:boto:Params: {} 2023-09-22T21:22:23.602 DEBUG:boto:Token: None 2023-09-22T21:22:23.602 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/ 2023-09-22T21:22:23.602 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g/8gKi+Ia/NcDZc5lzn/bJGLa/0= 2023-09-22T21:22:23.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g/8gKi+Ia/NcDZc5lzn/bJGLa/0=', 'Content-Length': '0'} 2023-09-22T21:22:23.605 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079a375064e23ec64-00650e058f-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.605 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:23.605 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:23.605 DEBUG:boto:Method: HEAD 2023-09-22T21:22:23.605 DEBUG:boto:Path: /ffzdqu-48/ 2023-09-22T21:22:23.605 DEBUG:boto:Data: 2023-09-22T21:22:23.606 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.606 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:23.606 DEBUG:boto:Port: 8000 2023-09-22T21:22:23.606 DEBUG:boto:Params: {} 2023-09-22T21:22:23.606 DEBUG:boto:Token: None 2023-09-22T21:22:23.606 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/ 2023-09-22T21:22:23.606 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g/8gKi+Ia/NcDZc5lzn/bJGLa/0= 2023-09-22T21:22:23.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g/8gKi+Ia/NcDZc5lzn/bJGLa/0=', 'Content-Length': '0'} 2023-09-22T21:22:23.609 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf8a1fc38c314c51-00650e058f-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.609 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:22:23.609 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:23.609 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:23.609 DEBUG:boto:path=/ffzdqu-48/?versions&max-keys=999 2023-09-22T21:22:23.609 DEBUG:boto:auth_path=/ffzdqu-48/?versions&max-keys=999 2023-09-22T21:22:23.609 DEBUG:boto:Method: GET 2023-09-22T21:22:23.610 DEBUG:boto:Path: /ffzdqu-48/?versions&max-keys=999 2023-09-22T21:22:23.610 DEBUG:boto:Data: 2023-09-22T21:22:23.610 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.610 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:23.610 DEBUG:boto:Port: 8001 2023-09-22T21:22:23.610 DEBUG:boto:Params: {} 2023-09-22T21:22:23.610 DEBUG:boto:Token: None 2023-09-22T21:22:23.610 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/?versions 2023-09-22T21:22:23.611 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E5XJV5+Eo6FOrFw94h9LKn7SsEw= 2023-09-22T21:22:23.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E5XJV5+Eo6FOrFw94h9LKn7SsEw=', 'Content-Length': '0'} 2023-09-22T21:22:23.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7b2bd7310a06c73-00650e058f-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.613 DEBUG:boto:b'ffzdqu-48999falseobj1nulltrue2023-09-22T21:22:07.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:22:07.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:22:07.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:22:07.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:23.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-22T21:22:23.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-22T21:22:23.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-22T21:22:23.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-22T21:22:23.614 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:22:23.614 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:23.615 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:23.615 DEBUG:boto:path=/ffzdqu-48/?versions&max-keys=999 2023-09-22T21:22:23.615 DEBUG:boto:auth_path=/ffzdqu-48/?versions&max-keys=999 2023-09-22T21:22:23.615 DEBUG:boto:Method: GET 2023-09-22T21:22:23.615 DEBUG:boto:Path: /ffzdqu-48/?versions&max-keys=999 2023-09-22T21:22:23.615 DEBUG:boto:Data: 2023-09-22T21:22:23.615 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.615 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:23.615 DEBUG:boto:Port: 8000 2023-09-22T21:22:23.615 DEBUG:boto:Params: {} 2023-09-22T21:22:23.616 DEBUG:boto:Token: None 2023-09-22T21:22:23.616 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/?versions 2023-09-22T21:22:23.616 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E5XJV5+Eo6FOrFw94h9LKn7SsEw= 2023-09-22T21:22:23.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E5XJV5+Eo6FOrFw94h9LKn7SsEw=', 'Content-Length': '0'} 2023-09-22T21:22:23.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cc5e455f50e5ecc-00650e058f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.618 DEBUG:boto:b'ffzdqu-48999falseobj1nulltrue2023-09-22T21:22:07.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:22:07.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:22:07.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:22:07.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:23.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-22T21:22:23.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-22T21:22:23.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-22T21:22:23.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-22T21:22:23.619 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:23.619 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:23.619 DEBUG:boto:path=/ffzdqu-48/?versions&max-keys=999 2023-09-22T21:22:23.620 DEBUG:boto:auth_path=/ffzdqu-48/?versions&max-keys=999 2023-09-22T21:22:23.620 DEBUG:boto:Method: GET 2023-09-22T21:22:23.620 DEBUG:boto:Path: /ffzdqu-48/?versions&max-keys=999 2023-09-22T21:22:23.620 DEBUG:boto:Data: 2023-09-22T21:22:23.620 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.620 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:23.620 DEBUG:boto:Port: 8001 2023-09-22T21:22:23.620 DEBUG:boto:Params: {} 2023-09-22T21:22:23.620 DEBUG:boto:Token: None 2023-09-22T21:22:23.621 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/?versions 2023-09-22T21:22:23.621 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E5XJV5+Eo6FOrFw94h9LKn7SsEw= 2023-09-22T21:22:23.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E5XJV5+Eo6FOrFw94h9LKn7SsEw=', 'Content-Length': '0'} 2023-09-22T21:22:23.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbf747e95f4a13fd-00650e058f-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.624 DEBUG:boto:b'ffzdqu-48999falseobj1nulltrue2023-09-22T21:22:07.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:22:07.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:22:07.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:22:07.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:23.624 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:23.624 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:23.624 DEBUG:boto:path=/ffzdqu-48/?versions&max-keys=999 2023-09-22T21:22:23.625 DEBUG:boto:auth_path=/ffzdqu-48/?versions&max-keys=999 2023-09-22T21:22:23.625 DEBUG:boto:Method: GET 2023-09-22T21:22:23.625 DEBUG:boto:Path: /ffzdqu-48/?versions&max-keys=999 2023-09-22T21:22:23.625 DEBUG:boto:Data: 2023-09-22T21:22:23.625 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.625 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:23.625 DEBUG:boto:Port: 8000 2023-09-22T21:22:23.625 DEBUG:boto:Params: {} 2023-09-22T21:22:23.625 DEBUG:boto:Token: None 2023-09-22T21:22:23.626 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/?versions 2023-09-22T21:22:23.626 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E5XJV5+Eo6FOrFw94h9LKn7SsEw= 2023-09-22T21:22:23.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E5XJV5+Eo6FOrFw94h9LKn7SsEw=', 'Content-Length': '0'} 2023-09-22T21:22:23.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb33abdc0de40ecd-00650e058f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.628 DEBUG:boto:b'ffzdqu-48999falseobj1nulltrue2023-09-22T21:22:07.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:22:07.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:22:07.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:22:07.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:23.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:22:23.629 DEBUG:boto:path=/ffzdqu-48/obj1 2023-09-22T21:22:23.629 DEBUG:boto:auth_path=/ffzdqu-48/obj1 2023-09-22T21:22:23.629 DEBUG:boto:path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.629 DEBUG:boto:auth_path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.629 DEBUG:boto:Method: GET 2023-09-22T21:22:23.629 DEBUG:boto:Path: /ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.629 DEBUG:boto:Data: 2023-09-22T21:22:23.630 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.630 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:23.630 DEBUG:boto:Port: 8001 2023-09-22T21:22:23.630 DEBUG:boto:Params: {} 2023-09-22T21:22:23.630 DEBUG:boto:Token: None 2023-09-22T21:22:23.630 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.630 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fU6jmKPTSqBDL0g746KAp8nc594= 2023-09-22T21:22:23.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fU6jmKPTSqBDL0g746KAp8nc594=', 'Content-Length': '0'} 2023-09-22T21:22:23.632 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007edf97f81767e1b7-00650e058f-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.632 DEBUG:boto:path=/ffzdqu-48/obj1 2023-09-22T21:22:23.632 DEBUG:boto:auth_path=/ffzdqu-48/obj1 2023-09-22T21:22:23.632 DEBUG:boto:path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.632 DEBUG:boto:auth_path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.633 DEBUG:boto:Method: GET 2023-09-22T21:22:23.633 DEBUG:boto:Path: /ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.633 DEBUG:boto:Data: 2023-09-22T21:22:23.633 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.633 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:23.633 DEBUG:boto:Port: 8000 2023-09-22T21:22:23.633 DEBUG:boto:Params: {} 2023-09-22T21:22:23.633 DEBUG:boto:Token: None 2023-09-22T21:22:23.633 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.634 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fU6jmKPTSqBDL0g746KAp8nc594= 2023-09-22T21:22:23.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fU6jmKPTSqBDL0g746KAp8nc594=', 'Content-Length': '0'} 2023-09-22T21:22:23.635 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fd807b04063fa05-00650e058f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.682 DEBUG:boto:path=/ffzdqu-48/obj1 2023-09-22T21:22:23.683 DEBUG:boto:auth_path=/ffzdqu-48/obj1 2023-09-22T21:22:23.683 DEBUG:boto:path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.683 DEBUG:boto:auth_path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.683 DEBUG:boto:Method: HEAD 2023-09-22T21:22:23.683 DEBUG:boto:Path: /ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.683 DEBUG:boto:Data: 2023-09-22T21:22:23.683 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.684 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:23.684 DEBUG:boto:Port: 8001 2023-09-22T21:22:23.684 DEBUG:boto:Params: {} 2023-09-22T21:22:23.684 DEBUG:boto:Token: None 2023-09-22T21:22:23.684 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.684 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:U1x/oFw1ow6vYBNUeeBXVF+mibM= 2023-09-22T21:22:23.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:U1x/oFw1ow6vYBNUeeBXVF+mibM=', 'Content-Length': '0'} 2023-09-22T21:22:23.687 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014913276becbd225-00650e058f-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.687 DEBUG:boto:path=/ffzdqu-48/obj1 2023-09-22T21:22:23.687 DEBUG:boto:auth_path=/ffzdqu-48/obj1 2023-09-22T21:22:23.688 DEBUG:boto:path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.688 DEBUG:boto:auth_path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.688 DEBUG:boto:Method: HEAD 2023-09-22T21:22:23.688 DEBUG:boto:Path: /ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.688 DEBUG:boto:Data: 2023-09-22T21:22:23.688 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.688 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:23.688 DEBUG:boto:Port: 8000 2023-09-22T21:22:23.689 DEBUG:boto:Params: {} 2023-09-22T21:22:23.689 DEBUG:boto:Token: None 2023-09-22T21:22:23.689 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.689 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:U1x/oFw1ow6vYBNUeeBXVF+mibM= 2023-09-22T21:22:23.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:U1x/oFw1ow6vYBNUeeBXVF+mibM=', 'Content-Length': '0'} 2023-09-22T21:22:23.692 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b929f353fdb265e-00650e058f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:22:23.692 DEBUG:boto:path=/ffzdqu-48/obj1 2023-09-22T21:22:23.692 DEBUG:boto:auth_path=/ffzdqu-48/obj1 2023-09-22T21:22:23.692 DEBUG:boto:path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.693 DEBUG:boto:auth_path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.693 DEBUG:boto:Method: GET 2023-09-22T21:22:23.693 DEBUG:boto:Path: /ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.693 DEBUG:boto:Data: 2023-09-22T21:22:23.693 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.693 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:23.693 DEBUG:boto:Port: 8001 2023-09-22T21:22:23.693 DEBUG:boto:Params: {} 2023-09-22T21:22:23.694 DEBUG:boto:Token: None 2023-09-22T21:22:23.694 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.694 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fU6jmKPTSqBDL0g746KAp8nc594= 2023-09-22T21:22:23.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fU6jmKPTSqBDL0g746KAp8nc594=', 'Content-Length': '0'} 2023-09-22T21:22:23.697 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c0cd9332edc81cf-00650e058f-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.697 DEBUG:boto:path=/ffzdqu-48/obj1 2023-09-22T21:22:23.697 DEBUG:boto:auth_path=/ffzdqu-48/obj1 2023-09-22T21:22:23.697 DEBUG:boto:path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.698 DEBUG:boto:auth_path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.698 DEBUG:boto:Method: GET 2023-09-22T21:22:23.698 DEBUG:boto:Path: /ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.698 DEBUG:boto:Data: 2023-09-22T21:22:23.698 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.698 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:23.698 DEBUG:boto:Port: 8000 2023-09-22T21:22:23.698 DEBUG:boto:Params: {} 2023-09-22T21:22:23.699 DEBUG:boto:Token: None 2023-09-22T21:22:23.699 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/obj1?versionId=null 2023-09-22T21:22:23.699 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fU6jmKPTSqBDL0g746KAp8nc594= 2023-09-22T21:22:23.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fU6jmKPTSqBDL0g746KAp8nc594=', 'Content-Length': '0'} 2023-09-22T21:22:23.702 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ab3769e8d95cf8a-00650e058f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.702 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:23.702 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:23.702 DEBUG:boto:path=/ffzdqu-48/?prefix=obj1 2023-09-22T21:22:23.702 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj1 2023-09-22T21:22:23.703 DEBUG:boto:Method: GET 2023-09-22T21:22:23.703 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj1 2023-09-22T21:22:23.703 DEBUG:boto:Data: 2023-09-22T21:22:23.703 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.703 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:23.703 DEBUG:boto:Port: 8001 2023-09-22T21:22:23.703 DEBUG:boto:Params: {} 2023-09-22T21:22:23.703 DEBUG:boto:Token: None 2023-09-22T21:22:23.703 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/ 2023-09-22T21:22:23.704 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nU5ExkcOQ/x+5h1reA/3RU95gI8= 2023-09-22T21:22:23.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nU5ExkcOQ/x+5h1reA/3RU95gI8=', 'Content-Length': '0'} 2023-09-22T21:22:23.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2b0480e9bd7a84a-00650e058f-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.750 DEBUG:boto:b'ffzdqu-48obj11000falseobj12023-09-22T21:22:07.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:23.751 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:23.751 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:23.751 DEBUG:boto:path=/ffzdqu-48/?prefix=obj1 2023-09-22T21:22:23.751 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj1 2023-09-22T21:22:23.751 DEBUG:boto:Method: GET 2023-09-22T21:22:23.752 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj1 2023-09-22T21:22:23.752 DEBUG:boto:Data: 2023-09-22T21:22:23.752 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.752 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:23.752 DEBUG:boto:Port: 8000 2023-09-22T21:22:23.752 DEBUG:boto:Params: {} 2023-09-22T21:22:23.752 DEBUG:boto:Token: None 2023-09-22T21:22:23.753 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/ 2023-09-22T21:22:23.753 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nU5ExkcOQ/x+5h1reA/3RU95gI8= 2023-09-22T21:22:23.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nU5ExkcOQ/x+5h1reA/3RU95gI8=', 'Content-Length': '0'} 2023-09-22T21:22:23.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc3d6a69773820fb-00650e058f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.757 DEBUG:boto:b'ffzdqu-48obj11000falseobj12023-09-22T21:22:07.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:23.757 DEBUG:boto:path=/ffzdqu-48/obj1 2023-09-22T21:22:23.758 DEBUG:boto:auth_path=/ffzdqu-48/obj1 2023-09-22T21:22:23.758 DEBUG:boto:Method: HEAD 2023-09-22T21:22:23.758 DEBUG:boto:Path: /ffzdqu-48/obj1 2023-09-22T21:22:23.758 DEBUG:boto:Data: 2023-09-22T21:22:23.758 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.758 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:23.758 DEBUG:boto:Port: 8001 2023-09-22T21:22:23.759 DEBUG:boto:Params: {} 2023-09-22T21:22:23.759 DEBUG:boto:Token: None 2023-09-22T21:22:23.759 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/obj1 2023-09-22T21:22:23.759 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bT6m9BDOmJeXouffvZxjOU4uZVQ= 2023-09-22T21:22:23.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bT6m9BDOmJeXouffvZxjOU4uZVQ=', 'Content-Length': '0'} 2023-09-22T21:22:23.761 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3ffdad074f0b797-00650e058f-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.762 DEBUG:boto:path=/ffzdqu-48/obj1 2023-09-22T21:22:23.762 DEBUG:boto:auth_path=/ffzdqu-48/obj1 2023-09-22T21:22:23.762 DEBUG:boto:Method: HEAD 2023-09-22T21:22:23.762 DEBUG:boto:Path: /ffzdqu-48/obj1 2023-09-22T21:22:23.762 DEBUG:boto:Data: 2023-09-22T21:22:23.762 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.763 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:23.763 DEBUG:boto:Port: 8000 2023-09-22T21:22:23.763 DEBUG:boto:Params: {} 2023-09-22T21:22:23.763 DEBUG:boto:Token: None 2023-09-22T21:22:23.763 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/obj1 2023-09-22T21:22:23.763 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bT6m9BDOmJeXouffvZxjOU4uZVQ= 2023-09-22T21:22:23.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bT6m9BDOmJeXouffvZxjOU4uZVQ=', 'Content-Length': '0'} 2023-09-22T21:22:23.766 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007a8376c4afc92b8-00650e058f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:22:23.767 DEBUG:boto:path=/ffzdqu-48/obj1 2023-09-22T21:22:23.767 DEBUG:boto:auth_path=/ffzdqu-48/obj1 2023-09-22T21:22:23.767 DEBUG:boto:Method: GET 2023-09-22T21:22:23.767 DEBUG:boto:Path: /ffzdqu-48/obj1 2023-09-22T21:22:23.767 DEBUG:boto:Data: 2023-09-22T21:22:23.767 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.768 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:23.768 DEBUG:boto:Port: 8001 2023-09-22T21:22:23.768 DEBUG:boto:Params: {} 2023-09-22T21:22:23.768 DEBUG:boto:Token: None 2023-09-22T21:22:23.768 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/obj1 2023-09-22T21:22:23.768 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:j+bYI6eFJx7P1YHpE43M8JG3eow= 2023-09-22T21:22:23.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:j+bYI6eFJx7P1YHpE43M8JG3eow=', 'Content-Length': '0'} 2023-09-22T21:22:23.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003797c4d6bb99a778-00650e058f-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.814 DEBUG:boto:path=/ffzdqu-48/obj1 2023-09-22T21:22:23.815 DEBUG:boto:auth_path=/ffzdqu-48/obj1 2023-09-22T21:22:23.815 DEBUG:boto:Method: GET 2023-09-22T21:22:23.815 DEBUG:boto:Path: /ffzdqu-48/obj1 2023-09-22T21:22:23.815 DEBUG:boto:Data: 2023-09-22T21:22:23.815 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.815 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:23.815 DEBUG:boto:Port: 8000 2023-09-22T21:22:23.815 DEBUG:boto:Params: {} 2023-09-22T21:22:23.816 DEBUG:boto:Token: None 2023-09-22T21:22:23.816 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/obj1 2023-09-22T21:22:23.816 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:j+bYI6eFJx7P1YHpE43M8JG3eow= 2023-09-22T21:22:23.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:j+bYI6eFJx7P1YHpE43M8JG3eow=', 'Content-Length': '0'} 2023-09-22T21:22:23.819 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7b24efb17f8b34a-00650e058f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.819 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:23.819 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:23.819 DEBUG:boto:path=/ffzdqu-48/?prefix=obj1 2023-09-22T21:22:23.820 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj1 2023-09-22T21:22:23.820 DEBUG:boto:Method: GET 2023-09-22T21:22:23.820 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj1 2023-09-22T21:22:23.820 DEBUG:boto:Data: 2023-09-22T21:22:23.820 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.820 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:23.820 DEBUG:boto:Port: 8001 2023-09-22T21:22:23.820 DEBUG:boto:Params: {} 2023-09-22T21:22:23.820 DEBUG:boto:Token: None 2023-09-22T21:22:23.821 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/ 2023-09-22T21:22:23.821 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nU5ExkcOQ/x+5h1reA/3RU95gI8= 2023-09-22T21:22:23.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nU5ExkcOQ/x+5h1reA/3RU95gI8=', 'Content-Length': '0'} 2023-09-22T21:22:23.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063303f845c749623-00650e058f-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.825 DEBUG:boto:b'ffzdqu-48obj11000falseobj12023-09-22T21:22:07.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:23.826 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:23.826 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:23.826 DEBUG:boto:path=/ffzdqu-48/?prefix=obj1 2023-09-22T21:22:23.826 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj1 2023-09-22T21:22:23.826 DEBUG:boto:Method: GET 2023-09-22T21:22:23.826 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj1 2023-09-22T21:22:23.826 DEBUG:boto:Data: 2023-09-22T21:22:23.827 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.827 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:23.827 DEBUG:boto:Port: 8000 2023-09-22T21:22:23.827 DEBUG:boto:Params: {} 2023-09-22T21:22:23.827 DEBUG:boto:Token: None 2023-09-22T21:22:23.827 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/ 2023-09-22T21:22:23.827 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nU5ExkcOQ/x+5h1reA/3RU95gI8= 2023-09-22T21:22:23.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nU5ExkcOQ/x+5h1reA/3RU95gI8=', 'Content-Length': '0'} 2023-09-22T21:22:23.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069e2f089bcb19919-00650e058f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.832 DEBUG:boto:b'ffzdqu-48obj11000falseobj12023-09-22T21:22:07.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:23.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:22:23.833 DEBUG:boto:path=/ffzdqu-48/obj2 2023-09-22T21:22:23.833 DEBUG:boto:auth_path=/ffzdqu-48/obj2 2023-09-22T21:22:23.833 DEBUG:boto:path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.833 DEBUG:boto:auth_path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.833 DEBUG:boto:Method: GET 2023-09-22T21:22:23.833 DEBUG:boto:Path: /ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.833 DEBUG:boto:Data: 2023-09-22T21:22:23.833 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.833 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:23.834 DEBUG:boto:Port: 8001 2023-09-22T21:22:23.834 DEBUG:boto:Params: {} 2023-09-22T21:22:23.834 DEBUG:boto:Token: None 2023-09-22T21:22:23.834 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.834 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t1C/yrAF0YR1OAUcDHOOEkVhII0= 2023-09-22T21:22:23.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t1C/yrAF0YR1OAUcDHOOEkVhII0=', 'Content-Length': '0'} 2023-09-22T21:22:23.837 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3f2f7fd59263424-00650e058f-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.837 DEBUG:boto:path=/ffzdqu-48/obj2 2023-09-22T21:22:23.837 DEBUG:boto:auth_path=/ffzdqu-48/obj2 2023-09-22T21:22:23.838 DEBUG:boto:path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.838 DEBUG:boto:auth_path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.838 DEBUG:boto:Method: GET 2023-09-22T21:22:23.838 DEBUG:boto:Path: /ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.838 DEBUG:boto:Data: 2023-09-22T21:22:23.838 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.838 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:23.838 DEBUG:boto:Port: 8000 2023-09-22T21:22:23.838 DEBUG:boto:Params: {} 2023-09-22T21:22:23.839 DEBUG:boto:Token: None 2023-09-22T21:22:23.839 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.839 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t1C/yrAF0YR1OAUcDHOOEkVhII0= 2023-09-22T21:22:23.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t1C/yrAF0YR1OAUcDHOOEkVhII0=', 'Content-Length': '0'} 2023-09-22T21:22:23.842 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000073080f6e603f56c-00650e058f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.842 DEBUG:boto:path=/ffzdqu-48/obj2 2023-09-22T21:22:23.842 DEBUG:boto:auth_path=/ffzdqu-48/obj2 2023-09-22T21:22:23.842 DEBUG:boto:path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.842 DEBUG:boto:auth_path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.842 DEBUG:boto:Method: HEAD 2023-09-22T21:22:23.842 DEBUG:boto:Path: /ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.843 DEBUG:boto:Data: 2023-09-22T21:22:23.843 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.843 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:23.843 DEBUG:boto:Port: 8001 2023-09-22T21:22:23.843 DEBUG:boto:Params: {} 2023-09-22T21:22:23.843 DEBUG:boto:Token: None 2023-09-22T21:22:23.843 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.843 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bQpnx+qlafiO0z6KN+umI66KbmQ= 2023-09-22T21:22:23.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bQpnx+qlafiO0z6KN+umI66KbmQ=', 'Content-Length': '0'} 2023-09-22T21:22:23.846 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b58bd20586e64d7f-00650e058f-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.846 DEBUG:boto:path=/ffzdqu-48/obj2 2023-09-22T21:22:23.846 DEBUG:boto:auth_path=/ffzdqu-48/obj2 2023-09-22T21:22:23.847 DEBUG:boto:path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.847 DEBUG:boto:auth_path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.847 DEBUG:boto:Method: HEAD 2023-09-22T21:22:23.847 DEBUG:boto:Path: /ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.847 DEBUG:boto:Data: 2023-09-22T21:22:23.847 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.847 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:23.847 DEBUG:boto:Port: 8000 2023-09-22T21:22:23.847 DEBUG:boto:Params: {} 2023-09-22T21:22:23.847 DEBUG:boto:Token: None 2023-09-22T21:22:23.848 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.848 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bQpnx+qlafiO0z6KN+umI66KbmQ= 2023-09-22T21:22:23.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bQpnx+qlafiO0z6KN+umI66KbmQ=', 'Content-Length': '0'} 2023-09-22T21:22:23.850 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c35c2c1fa33c387b-00650e058f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:22:23.851 DEBUG:boto:path=/ffzdqu-48/obj2 2023-09-22T21:22:23.851 DEBUG:boto:auth_path=/ffzdqu-48/obj2 2023-09-22T21:22:23.851 DEBUG:boto:path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.851 DEBUG:boto:auth_path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.851 DEBUG:boto:Method: GET 2023-09-22T21:22:23.851 DEBUG:boto:Path: /ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.851 DEBUG:boto:Data: 2023-09-22T21:22:23.852 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.852 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:23.852 DEBUG:boto:Port: 8001 2023-09-22T21:22:23.852 DEBUG:boto:Params: {} 2023-09-22T21:22:23.852 DEBUG:boto:Token: None 2023-09-22T21:22:23.852 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.852 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t1C/yrAF0YR1OAUcDHOOEkVhII0= 2023-09-22T21:22:23.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t1C/yrAF0YR1OAUcDHOOEkVhII0=', 'Content-Length': '0'} 2023-09-22T21:22:23.855 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000010f526325990b3-00650e058f-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.898 DEBUG:boto:path=/ffzdqu-48/obj2 2023-09-22T21:22:23.898 DEBUG:boto:auth_path=/ffzdqu-48/obj2 2023-09-22T21:22:23.899 DEBUG:boto:path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.899 DEBUG:boto:auth_path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.899 DEBUG:boto:Method: GET 2023-09-22T21:22:23.899 DEBUG:boto:Path: /ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.899 DEBUG:boto:Data: 2023-09-22T21:22:23.899 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.899 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:23.899 DEBUG:boto:Port: 8000 2023-09-22T21:22:23.899 DEBUG:boto:Params: {} 2023-09-22T21:22:23.900 DEBUG:boto:Token: None 2023-09-22T21:22:23.900 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/obj2?versionId=null 2023-09-22T21:22:23.900 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t1C/yrAF0YR1OAUcDHOOEkVhII0= 2023-09-22T21:22:23.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t1C/yrAF0YR1OAUcDHOOEkVhII0=', 'Content-Length': '0'} 2023-09-22T21:22:23.903 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016eae10fd11097c8-00650e058f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.903 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:23.903 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:23.903 DEBUG:boto:path=/ffzdqu-48/?prefix=obj2 2023-09-22T21:22:23.904 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj2 2023-09-22T21:22:23.904 DEBUG:boto:Method: GET 2023-09-22T21:22:23.904 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj2 2023-09-22T21:22:23.904 DEBUG:boto:Data: 2023-09-22T21:22:23.904 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.904 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:23.904 DEBUG:boto:Port: 8001 2023-09-22T21:22:23.904 DEBUG:boto:Params: {} 2023-09-22T21:22:23.904 DEBUG:boto:Token: None 2023-09-22T21:22:23.905 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/ 2023-09-22T21:22:23.905 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nU5ExkcOQ/x+5h1reA/3RU95gI8= 2023-09-22T21:22:23.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nU5ExkcOQ/x+5h1reA/3RU95gI8=', 'Content-Length': '0'} 2023-09-22T21:22:23.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f192518db40e5c6-00650e058f-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.950 DEBUG:boto:b'ffzdqu-48obj21000falseobj22023-09-22T21:22:07.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:23.951 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:23.951 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:23.951 DEBUG:boto:path=/ffzdqu-48/?prefix=obj2 2023-09-22T21:22:23.951 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj2 2023-09-22T21:22:23.951 DEBUG:boto:Method: GET 2023-09-22T21:22:23.952 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj2 2023-09-22T21:22:23.952 DEBUG:boto:Data: 2023-09-22T21:22:23.952 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.952 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:23.952 DEBUG:boto:Port: 8000 2023-09-22T21:22:23.952 DEBUG:boto:Params: {} 2023-09-22T21:22:23.952 DEBUG:boto:Token: None 2023-09-22T21:22:23.952 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/ 2023-09-22T21:22:23.953 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nU5ExkcOQ/x+5h1reA/3RU95gI8= 2023-09-22T21:22:23.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nU5ExkcOQ/x+5h1reA/3RU95gI8=', 'Content-Length': '0'} 2023-09-22T21:22:23.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008950e58816d949b4-00650e058f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.957 DEBUG:boto:b'ffzdqu-48obj21000falseobj22023-09-22T21:22:07.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:23.958 DEBUG:boto:path=/ffzdqu-48/obj2 2023-09-22T21:22:23.958 DEBUG:boto:auth_path=/ffzdqu-48/obj2 2023-09-22T21:22:23.958 DEBUG:boto:Method: HEAD 2023-09-22T21:22:23.958 DEBUG:boto:Path: /ffzdqu-48/obj2 2023-09-22T21:22:23.958 DEBUG:boto:Data: 2023-09-22T21:22:23.958 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.959 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:23.959 DEBUG:boto:Port: 8001 2023-09-22T21:22:23.959 DEBUG:boto:Params: {} 2023-09-22T21:22:23.959 DEBUG:boto:Token: None 2023-09-22T21:22:23.959 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/obj2 2023-09-22T21:22:23.959 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Adb0Y9uyE4IU5M0a9H7NDhdcQdo= 2023-09-22T21:22:23.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Adb0Y9uyE4IU5M0a9H7NDhdcQdo=', 'Content-Length': '0'} 2023-09-22T21:22:23.962 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b300e6c4b45c48bf-00650e058f-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.963 DEBUG:boto:path=/ffzdqu-48/obj2 2023-09-22T21:22:23.963 DEBUG:boto:auth_path=/ffzdqu-48/obj2 2023-09-22T21:22:23.963 DEBUG:boto:Method: HEAD 2023-09-22T21:22:23.963 DEBUG:boto:Path: /ffzdqu-48/obj2 2023-09-22T21:22:23.963 DEBUG:boto:Data: 2023-09-22T21:22:23.963 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.963 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:23.963 DEBUG:boto:Port: 8000 2023-09-22T21:22:23.964 DEBUG:boto:Params: {} 2023-09-22T21:22:23.964 DEBUG:boto:Token: None 2023-09-22T21:22:23.964 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/obj2 2023-09-22T21:22:23.964 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Adb0Y9uyE4IU5M0a9H7NDhdcQdo= 2023-09-22T21:22:23.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Adb0Y9uyE4IU5M0a9H7NDhdcQdo=', 'Content-Length': '0'} 2023-09-22T21:22:23.966 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020f23fdbfcb4aeaf-00650e058f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:23.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:22:23.967 DEBUG:boto:path=/ffzdqu-48/obj2 2023-09-22T21:22:23.967 DEBUG:boto:auth_path=/ffzdqu-48/obj2 2023-09-22T21:22:23.967 DEBUG:boto:Method: GET 2023-09-22T21:22:23.967 DEBUG:boto:Path: /ffzdqu-48/obj2 2023-09-22T21:22:23.967 DEBUG:boto:Data: 2023-09-22T21:22:23.967 DEBUG:boto:Headers: {} 2023-09-22T21:22:23.968 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:23.968 DEBUG:boto:Port: 8001 2023-09-22T21:22:23.968 DEBUG:boto:Params: {} 2023-09-22T21:22:23.968 DEBUG:boto:Token: None 2023-09-22T21:22:23.968 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:23 GMT /ffzdqu-48/obj2 2023-09-22T21:22:23.968 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Vu78xiqxG5u7ZHQ0vEsMJeRFqkw= 2023-09-22T21:22:23.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Vu78xiqxG5u7ZHQ0vEsMJeRFqkw=', 'Content-Length': '0'} 2023-09-22T21:22:23.971 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035f4059b7768ec16-00650e058f-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.014 DEBUG:boto:path=/ffzdqu-48/obj2 2023-09-22T21:22:24.015 DEBUG:boto:auth_path=/ffzdqu-48/obj2 2023-09-22T21:22:24.015 DEBUG:boto:Method: GET 2023-09-22T21:22:24.015 DEBUG:boto:Path: /ffzdqu-48/obj2 2023-09-22T21:22:24.015 DEBUG:boto:Data: 2023-09-22T21:22:24.015 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.015 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.015 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.015 DEBUG:boto:Params: {} 2023-09-22T21:22:24.016 DEBUG:boto:Token: None 2023-09-22T21:22:24.016 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/obj2 2023-09-22T21:22:24.016 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9VOqybhtsxs54lQn5kLot+OetsE= 2023-09-22T21:22:24.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9VOqybhtsxs54lQn5kLot+OetsE=', 'Content-Length': '0'} 2023-09-22T21:22:24.019 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000147f431d8c21d5b3-00650e0590-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.019 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:24.019 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:24.019 DEBUG:boto:path=/ffzdqu-48/?prefix=obj2 2023-09-22T21:22:24.020 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj2 2023-09-22T21:22:24.020 DEBUG:boto:Method: GET 2023-09-22T21:22:24.020 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj2 2023-09-22T21:22:24.020 DEBUG:boto:Data: 2023-09-22T21:22:24.020 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.020 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.020 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.020 DEBUG:boto:Params: {} 2023-09-22T21:22:24.020 DEBUG:boto:Token: None 2023-09-22T21:22:24.021 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/ 2023-09-22T21:22:24.021 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dj5GzxHKkx9mI04cRlp2hqB24jg= 2023-09-22T21:22:24.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dj5GzxHKkx9mI04cRlp2hqB24jg=', 'Content-Length': '0'} 2023-09-22T21:22:24.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090b3d1215800f827-00650e0590-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.026 DEBUG:boto:b'ffzdqu-48obj21000falseobj22023-09-22T21:22:07.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:24.026 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:24.026 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:24.026 DEBUG:boto:path=/ffzdqu-48/?prefix=obj2 2023-09-22T21:22:24.026 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj2 2023-09-22T21:22:24.027 DEBUG:boto:Method: GET 2023-09-22T21:22:24.027 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj2 2023-09-22T21:22:24.027 DEBUG:boto:Data: 2023-09-22T21:22:24.027 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.027 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.027 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.027 DEBUG:boto:Params: {} 2023-09-22T21:22:24.027 DEBUG:boto:Token: None 2023-09-22T21:22:24.027 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/ 2023-09-22T21:22:24.028 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dj5GzxHKkx9mI04cRlp2hqB24jg= 2023-09-22T21:22:24.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dj5GzxHKkx9mI04cRlp2hqB24jg=', 'Content-Length': '0'} 2023-09-22T21:22:24.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c92e9bd38139a47c-00650e0590-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.032 DEBUG:boto:b'ffzdqu-48obj21000falseobj22023-09-22T21:22:07.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:24.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:22:24.033 DEBUG:boto:path=/ffzdqu-48/obj3 2023-09-22T21:22:24.033 DEBUG:boto:auth_path=/ffzdqu-48/obj3 2023-09-22T21:22:24.033 DEBUG:boto:path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.033 DEBUG:boto:auth_path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.033 DEBUG:boto:Method: GET 2023-09-22T21:22:24.034 DEBUG:boto:Path: /ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.034 DEBUG:boto:Data: 2023-09-22T21:22:24.034 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.034 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.034 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.034 DEBUG:boto:Params: {} 2023-09-22T21:22:24.034 DEBUG:boto:Token: None 2023-09-22T21:22:24.034 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.035 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jbnzCpJtWac9SpbZvrLrZ3ItCX4= 2023-09-22T21:22:24.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jbnzCpJtWac9SpbZvrLrZ3ItCX4=', 'Content-Length': '0'} 2023-09-22T21:22:24.037 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f13275408a99afaf-00650e0590-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.082 DEBUG:boto:path=/ffzdqu-48/obj3 2023-09-22T21:22:24.083 DEBUG:boto:auth_path=/ffzdqu-48/obj3 2023-09-22T21:22:24.083 DEBUG:boto:path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.083 DEBUG:boto:auth_path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.083 DEBUG:boto:Method: GET 2023-09-22T21:22:24.083 DEBUG:boto:Path: /ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.083 DEBUG:boto:Data: 2023-09-22T21:22:24.083 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.084 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.084 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.084 DEBUG:boto:Params: {} 2023-09-22T21:22:24.084 DEBUG:boto:Token: None 2023-09-22T21:22:24.084 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.084 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jbnzCpJtWac9SpbZvrLrZ3ItCX4= 2023-09-22T21:22:24.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jbnzCpJtWac9SpbZvrLrZ3ItCX4=', 'Content-Length': '0'} 2023-09-22T21:22:24.087 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a71a33f4b641239-00650e0590-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.088 DEBUG:boto:path=/ffzdqu-48/obj3 2023-09-22T21:22:24.088 DEBUG:boto:auth_path=/ffzdqu-48/obj3 2023-09-22T21:22:24.088 DEBUG:boto:path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.088 DEBUG:boto:auth_path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.088 DEBUG:boto:Method: HEAD 2023-09-22T21:22:24.088 DEBUG:boto:Path: /ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.088 DEBUG:boto:Data: 2023-09-22T21:22:24.089 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.089 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.089 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.089 DEBUG:boto:Params: {} 2023-09-22T21:22:24.089 DEBUG:boto:Token: None 2023-09-22T21:22:24.089 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.089 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:83N24AGlBzuhFR9bW9YC9prH97g= 2023-09-22T21:22:24.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:83N24AGlBzuhFR9bW9YC9prH97g=', 'Content-Length': '0'} 2023-09-22T21:22:24.092 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb3f6fda96963390-00650e0590-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.092 DEBUG:boto:path=/ffzdqu-48/obj3 2023-09-22T21:22:24.093 DEBUG:boto:auth_path=/ffzdqu-48/obj3 2023-09-22T21:22:24.093 DEBUG:boto:path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.093 DEBUG:boto:auth_path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.093 DEBUG:boto:Method: HEAD 2023-09-22T21:22:24.093 DEBUG:boto:Path: /ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.093 DEBUG:boto:Data: 2023-09-22T21:22:24.093 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.093 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.093 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.094 DEBUG:boto:Params: {} 2023-09-22T21:22:24.094 DEBUG:boto:Token: None 2023-09-22T21:22:24.094 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.094 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:83N24AGlBzuhFR9bW9YC9prH97g= 2023-09-22T21:22:24.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:83N24AGlBzuhFR9bW9YC9prH97g=', 'Content-Length': '0'} 2023-09-22T21:22:24.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000edca2a9a3040e48f-00650e0590-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:22:24.097 DEBUG:boto:path=/ffzdqu-48/obj3 2023-09-22T21:22:24.097 DEBUG:boto:auth_path=/ffzdqu-48/obj3 2023-09-22T21:22:24.097 DEBUG:boto:path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.098 DEBUG:boto:auth_path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.098 DEBUG:boto:Method: GET 2023-09-22T21:22:24.098 DEBUG:boto:Path: /ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.098 DEBUG:boto:Data: 2023-09-22T21:22:24.098 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.098 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.098 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.098 DEBUG:boto:Params: {} 2023-09-22T21:22:24.098 DEBUG:boto:Token: None 2023-09-22T21:22:24.099 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.099 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jbnzCpJtWac9SpbZvrLrZ3ItCX4= 2023-09-22T21:22:24.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jbnzCpJtWac9SpbZvrLrZ3ItCX4=', 'Content-Length': '0'} 2023-09-22T21:22:24.102 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8685e5a36766001-00650e0590-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.146 DEBUG:boto:path=/ffzdqu-48/obj3 2023-09-22T21:22:24.147 DEBUG:boto:auth_path=/ffzdqu-48/obj3 2023-09-22T21:22:24.147 DEBUG:boto:path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.147 DEBUG:boto:auth_path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.147 DEBUG:boto:Method: GET 2023-09-22T21:22:24.147 DEBUG:boto:Path: /ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.147 DEBUG:boto:Data: 2023-09-22T21:22:24.147 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.147 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.148 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.148 DEBUG:boto:Params: {} 2023-09-22T21:22:24.148 DEBUG:boto:Token: None 2023-09-22T21:22:24.148 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/obj3?versionId=null 2023-09-22T21:22:24.148 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jbnzCpJtWac9SpbZvrLrZ3ItCX4= 2023-09-22T21:22:24.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jbnzCpJtWac9SpbZvrLrZ3ItCX4=', 'Content-Length': '0'} 2023-09-22T21:22:24.151 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e63d32d35eb6684a-00650e0590-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.152 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:24.152 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:24.152 DEBUG:boto:path=/ffzdqu-48/?prefix=obj3 2023-09-22T21:22:24.152 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj3 2023-09-22T21:22:24.152 DEBUG:boto:Method: GET 2023-09-22T21:22:24.152 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj3 2023-09-22T21:22:24.152 DEBUG:boto:Data: 2023-09-22T21:22:24.152 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.152 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.153 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.153 DEBUG:boto:Params: {} 2023-09-22T21:22:24.153 DEBUG:boto:Token: None 2023-09-22T21:22:24.153 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/ 2023-09-22T21:22:24.153 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dj5GzxHKkx9mI04cRlp2hqB24jg= 2023-09-22T21:22:24.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dj5GzxHKkx9mI04cRlp2hqB24jg=', 'Content-Length': '0'} 2023-09-22T21:22:24.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e91816bab49bc2a3-00650e0590-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.198 DEBUG:boto:b'ffzdqu-48obj31000falseobj32023-09-22T21:22:07.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:24.199 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:24.199 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:24.199 DEBUG:boto:path=/ffzdqu-48/?prefix=obj3 2023-09-22T21:22:24.199 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj3 2023-09-22T21:22:24.200 DEBUG:boto:Method: GET 2023-09-22T21:22:24.200 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj3 2023-09-22T21:22:24.200 DEBUG:boto:Data: 2023-09-22T21:22:24.200 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.200 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.200 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.200 DEBUG:boto:Params: {} 2023-09-22T21:22:24.200 DEBUG:boto:Token: None 2023-09-22T21:22:24.200 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/ 2023-09-22T21:22:24.201 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dj5GzxHKkx9mI04cRlp2hqB24jg= 2023-09-22T21:22:24.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dj5GzxHKkx9mI04cRlp2hqB24jg=', 'Content-Length': '0'} 2023-09-22T21:22:24.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015e9853ed4b01956-00650e0590-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.205 DEBUG:boto:b'ffzdqu-48obj31000falseobj32023-09-22T21:22:07.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:24.206 DEBUG:boto:path=/ffzdqu-48/obj3 2023-09-22T21:22:24.206 DEBUG:boto:auth_path=/ffzdqu-48/obj3 2023-09-22T21:22:24.206 DEBUG:boto:Method: HEAD 2023-09-22T21:22:24.206 DEBUG:boto:Path: /ffzdqu-48/obj3 2023-09-22T21:22:24.206 DEBUG:boto:Data: 2023-09-22T21:22:24.207 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.207 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.207 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.207 DEBUG:boto:Params: {} 2023-09-22T21:22:24.208 DEBUG:boto:Token: None 2023-09-22T21:22:24.208 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/obj3 2023-09-22T21:22:24.208 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sMaCioiGVBR9A9Yg+Og+1+IodYs= 2023-09-22T21:22:24.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sMaCioiGVBR9A9Yg+Og+1+IodYs=', 'Content-Length': '0'} 2023-09-22T21:22:24.211 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015e0337b0d272aa4-00650e0590-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.211 DEBUG:boto:path=/ffzdqu-48/obj3 2023-09-22T21:22:24.212 DEBUG:boto:auth_path=/ffzdqu-48/obj3 2023-09-22T21:22:24.212 DEBUG:boto:Method: HEAD 2023-09-22T21:22:24.212 DEBUG:boto:Path: /ffzdqu-48/obj3 2023-09-22T21:22:24.212 DEBUG:boto:Data: 2023-09-22T21:22:24.212 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.212 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.212 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.213 DEBUG:boto:Params: {} 2023-09-22T21:22:24.213 DEBUG:boto:Token: None 2023-09-22T21:22:24.213 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/obj3 2023-09-22T21:22:24.213 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sMaCioiGVBR9A9Yg+Og+1+IodYs= 2023-09-22T21:22:24.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sMaCioiGVBR9A9Yg+Og+1+IodYs=', 'Content-Length': '0'} 2023-09-22T21:22:24.216 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000350fa98a5a499b42-00650e0590-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:22:24.217 DEBUG:boto:path=/ffzdqu-48/obj3 2023-09-22T21:22:24.217 DEBUG:boto:auth_path=/ffzdqu-48/obj3 2023-09-22T21:22:24.217 DEBUG:boto:Method: GET 2023-09-22T21:22:24.217 DEBUG:boto:Path: /ffzdqu-48/obj3 2023-09-22T21:22:24.217 DEBUG:boto:Data: 2023-09-22T21:22:24.217 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.217 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.217 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.218 DEBUG:boto:Params: {} 2023-09-22T21:22:24.218 DEBUG:boto:Token: None 2023-09-22T21:22:24.218 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/obj3 2023-09-22T21:22:24.218 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:s3mBrXJ9S2sMtTz8PuDlJ2t02f0= 2023-09-22T21:22:24.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:s3mBrXJ9S2sMtTz8PuDlJ2t02f0=', 'Content-Length': '0'} 2023-09-22T21:22:24.221 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f38a037605b07ce-00650e0590-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.262 DEBUG:boto:path=/ffzdqu-48/obj3 2023-09-22T21:22:24.263 DEBUG:boto:auth_path=/ffzdqu-48/obj3 2023-09-22T21:22:24.263 DEBUG:boto:Method: GET 2023-09-22T21:22:24.263 DEBUG:boto:Path: /ffzdqu-48/obj3 2023-09-22T21:22:24.263 DEBUG:boto:Data: 2023-09-22T21:22:24.263 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.263 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.263 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.263 DEBUG:boto:Params: {} 2023-09-22T21:22:24.264 DEBUG:boto:Token: None 2023-09-22T21:22:24.264 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/obj3 2023-09-22T21:22:24.264 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:s3mBrXJ9S2sMtTz8PuDlJ2t02f0= 2023-09-22T21:22:24.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:s3mBrXJ9S2sMtTz8PuDlJ2t02f0=', 'Content-Length': '0'} 2023-09-22T21:22:24.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f88d3c4f1eb3ee91-00650e0590-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.267 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:24.267 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:24.267 DEBUG:boto:path=/ffzdqu-48/?prefix=obj3 2023-09-22T21:22:24.267 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj3 2023-09-22T21:22:24.268 DEBUG:boto:Method: GET 2023-09-22T21:22:24.268 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj3 2023-09-22T21:22:24.268 DEBUG:boto:Data: 2023-09-22T21:22:24.268 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.268 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.268 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.268 DEBUG:boto:Params: {} 2023-09-22T21:22:24.268 DEBUG:boto:Token: None 2023-09-22T21:22:24.268 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/ 2023-09-22T21:22:24.269 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dj5GzxHKkx9mI04cRlp2hqB24jg= 2023-09-22T21:22:24.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dj5GzxHKkx9mI04cRlp2hqB24jg=', 'Content-Length': '0'} 2023-09-22T21:22:24.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4ac985a4bddc44c-00650e0590-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.314 DEBUG:boto:b'ffzdqu-48obj31000falseobj32023-09-22T21:22:07.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:24.315 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:24.315 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:24.315 DEBUG:boto:path=/ffzdqu-48/?prefix=obj3 2023-09-22T21:22:24.315 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj3 2023-09-22T21:22:24.316 DEBUG:boto:Method: GET 2023-09-22T21:22:24.316 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj3 2023-09-22T21:22:24.316 DEBUG:boto:Data: 2023-09-22T21:22:24.316 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.316 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.316 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.316 DEBUG:boto:Params: {} 2023-09-22T21:22:24.316 DEBUG:boto:Token: None 2023-09-22T21:22:24.317 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/ 2023-09-22T21:22:24.317 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dj5GzxHKkx9mI04cRlp2hqB24jg= 2023-09-22T21:22:24.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dj5GzxHKkx9mI04cRlp2hqB24jg=', 'Content-Length': '0'} 2023-09-22T21:22:24.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003788a59b7a3f2db5-00650e0590-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.322 DEBUG:boto:b'ffzdqu-48obj31000falseobj32023-09-22T21:22:07.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:24.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:22:24.322 DEBUG:boto:path=/ffzdqu-48/obj4 2023-09-22T21:22:24.323 DEBUG:boto:auth_path=/ffzdqu-48/obj4 2023-09-22T21:22:24.323 DEBUG:boto:path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.323 DEBUG:boto:auth_path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.323 DEBUG:boto:Method: GET 2023-09-22T21:22:24.323 DEBUG:boto:Path: /ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.323 DEBUG:boto:Data: 2023-09-22T21:22:24.323 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.323 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.323 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.324 DEBUG:boto:Params: {} 2023-09-22T21:22:24.324 DEBUG:boto:Token: None 2023-09-22T21:22:24.324 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.324 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:c2HREQk2MNA1xrl9ZT5zao4nxOw= 2023-09-22T21:22:24.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:c2HREQk2MNA1xrl9ZT5zao4nxOw=', 'Content-Length': '0'} 2023-09-22T21:22:24.327 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000253e81bac07498e8-00650e0590-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.327 DEBUG:boto:path=/ffzdqu-48/obj4 2023-09-22T21:22:24.327 DEBUG:boto:auth_path=/ffzdqu-48/obj4 2023-09-22T21:22:24.328 DEBUG:boto:path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.328 DEBUG:boto:auth_path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.328 DEBUG:boto:Method: GET 2023-09-22T21:22:24.328 DEBUG:boto:Path: /ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.328 DEBUG:boto:Data: 2023-09-22T21:22:24.329 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.329 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.329 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.329 DEBUG:boto:Params: {} 2023-09-22T21:22:24.329 DEBUG:boto:Token: None 2023-09-22T21:22:24.329 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.330 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:c2HREQk2MNA1xrl9ZT5zao4nxOw= 2023-09-22T21:22:24.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:c2HREQk2MNA1xrl9ZT5zao4nxOw=', 'Content-Length': '0'} 2023-09-22T21:22:24.332 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd84d835030d1207-00650e0590-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.333 DEBUG:boto:path=/ffzdqu-48/obj4 2023-09-22T21:22:24.333 DEBUG:boto:auth_path=/ffzdqu-48/obj4 2023-09-22T21:22:24.333 DEBUG:boto:path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.333 DEBUG:boto:auth_path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.333 DEBUG:boto:Method: HEAD 2023-09-22T21:22:24.333 DEBUG:boto:Path: /ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.333 DEBUG:boto:Data: 2023-09-22T21:22:24.334 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.334 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.334 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.334 DEBUG:boto:Params: {} 2023-09-22T21:22:24.336 DEBUG:boto:Token: None 2023-09-22T21:22:24.336 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.336 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zcDMFFWnPtBPU3GR0rcu7LOmYXs= 2023-09-22T21:22:24.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zcDMFFWnPtBPU3GR0rcu7LOmYXs=', 'Content-Length': '0'} 2023-09-22T21:22:24.339 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088a508887314a8a9-00650e0590-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.339 DEBUG:boto:path=/ffzdqu-48/obj4 2023-09-22T21:22:24.339 DEBUG:boto:auth_path=/ffzdqu-48/obj4 2023-09-22T21:22:24.339 DEBUG:boto:path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.340 DEBUG:boto:auth_path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.340 DEBUG:boto:Method: HEAD 2023-09-22T21:22:24.340 DEBUG:boto:Path: /ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.340 DEBUG:boto:Data: 2023-09-22T21:22:24.340 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.340 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.340 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.340 DEBUG:boto:Params: {} 2023-09-22T21:22:24.340 DEBUG:boto:Token: None 2023-09-22T21:22:24.341 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.341 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zcDMFFWnPtBPU3GR0rcu7LOmYXs= 2023-09-22T21:22:24.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zcDMFFWnPtBPU3GR0rcu7LOmYXs=', 'Content-Length': '0'} 2023-09-22T21:22:24.343 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d3a2d991828d5a1-00650e0590-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:22:24.344 DEBUG:boto:path=/ffzdqu-48/obj4 2023-09-22T21:22:24.344 DEBUG:boto:auth_path=/ffzdqu-48/obj4 2023-09-22T21:22:24.344 DEBUG:boto:path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.344 DEBUG:boto:auth_path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.344 DEBUG:boto:Method: GET 2023-09-22T21:22:24.345 DEBUG:boto:Path: /ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.345 DEBUG:boto:Data: 2023-09-22T21:22:24.345 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.345 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.345 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.345 DEBUG:boto:Params: {} 2023-09-22T21:22:24.345 DEBUG:boto:Token: None 2023-09-22T21:22:24.345 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.346 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:c2HREQk2MNA1xrl9ZT5zao4nxOw= 2023-09-22T21:22:24.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:c2HREQk2MNA1xrl9ZT5zao4nxOw=', 'Content-Length': '0'} 2023-09-22T21:22:24.348 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017d03e2071fcf0f8-00650e0590-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.395 DEBUG:boto:path=/ffzdqu-48/obj4 2023-09-22T21:22:24.395 DEBUG:boto:auth_path=/ffzdqu-48/obj4 2023-09-22T21:22:24.395 DEBUG:boto:path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.395 DEBUG:boto:auth_path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.395 DEBUG:boto:Method: GET 2023-09-22T21:22:24.396 DEBUG:boto:Path: /ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.396 DEBUG:boto:Data: 2023-09-22T21:22:24.396 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.396 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.396 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.396 DEBUG:boto:Params: {} 2023-09-22T21:22:24.396 DEBUG:boto:Token: None 2023-09-22T21:22:24.396 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/obj4?versionId=null 2023-09-22T21:22:24.396 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:c2HREQk2MNA1xrl9ZT5zao4nxOw= 2023-09-22T21:22:24.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:c2HREQk2MNA1xrl9ZT5zao4nxOw=', 'Content-Length': '0'} 2023-09-22T21:22:24.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000098b702904b391775-00650e0590-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.399 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:24.400 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:24.400 DEBUG:boto:path=/ffzdqu-48/?prefix=obj4 2023-09-22T21:22:24.400 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj4 2023-09-22T21:22:24.400 DEBUG:boto:Method: GET 2023-09-22T21:22:24.400 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj4 2023-09-22T21:22:24.400 DEBUG:boto:Data: 2023-09-22T21:22:24.400 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.400 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.400 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.401 DEBUG:boto:Params: {} 2023-09-22T21:22:24.401 DEBUG:boto:Token: None 2023-09-22T21:22:24.401 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/ 2023-09-22T21:22:24.401 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dj5GzxHKkx9mI04cRlp2hqB24jg= 2023-09-22T21:22:24.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dj5GzxHKkx9mI04cRlp2hqB24jg=', 'Content-Length': '0'} 2023-09-22T21:22:24.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014f06776126fa13c-00650e0590-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.406 DEBUG:boto:b'ffzdqu-48obj41000falseobj42023-09-22T21:22:07.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:24.406 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:24.406 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:24.406 DEBUG:boto:path=/ffzdqu-48/?prefix=obj4 2023-09-22T21:22:24.406 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj4 2023-09-22T21:22:24.407 DEBUG:boto:Method: GET 2023-09-22T21:22:24.407 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj4 2023-09-22T21:22:24.407 DEBUG:boto:Data: 2023-09-22T21:22:24.407 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.407 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.407 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.407 DEBUG:boto:Params: {} 2023-09-22T21:22:24.407 DEBUG:boto:Token: None 2023-09-22T21:22:24.407 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/ 2023-09-22T21:22:24.408 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dj5GzxHKkx9mI04cRlp2hqB24jg= 2023-09-22T21:22:24.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dj5GzxHKkx9mI04cRlp2hqB24jg=', 'Content-Length': '0'} 2023-09-22T21:22:24.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007205af72de6e4d71-00650e0590-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.413 DEBUG:boto:b'ffzdqu-48obj41000falseobj42023-09-22T21:22:07.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:24.413 DEBUG:boto:path=/ffzdqu-48/obj4 2023-09-22T21:22:24.413 DEBUG:boto:auth_path=/ffzdqu-48/obj4 2023-09-22T21:22:24.413 DEBUG:boto:Method: HEAD 2023-09-22T21:22:24.414 DEBUG:boto:Path: /ffzdqu-48/obj4 2023-09-22T21:22:24.414 DEBUG:boto:Data: 2023-09-22T21:22:24.414 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.414 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.414 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.414 DEBUG:boto:Params: {} 2023-09-22T21:22:24.414 DEBUG:boto:Token: None 2023-09-22T21:22:24.414 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/obj4 2023-09-22T21:22:24.415 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NiMBFNzvRId4QrK5jk900ilIXdY= 2023-09-22T21:22:24.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NiMBFNzvRId4QrK5jk900ilIXdY=', 'Content-Length': '0'} 2023-09-22T21:22:24.417 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e991c0db99d5d3aa-00650e0590-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.417 DEBUG:boto:path=/ffzdqu-48/obj4 2023-09-22T21:22:24.418 DEBUG:boto:auth_path=/ffzdqu-48/obj4 2023-09-22T21:22:24.418 DEBUG:boto:Method: HEAD 2023-09-22T21:22:24.418 DEBUG:boto:Path: /ffzdqu-48/obj4 2023-09-22T21:22:24.418 DEBUG:boto:Data: 2023-09-22T21:22:24.418 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.418 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.418 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.418 DEBUG:boto:Params: {} 2023-09-22T21:22:24.418 DEBUG:boto:Token: None 2023-09-22T21:22:24.419 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/obj4 2023-09-22T21:22:24.419 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NiMBFNzvRId4QrK5jk900ilIXdY= 2023-09-22T21:22:24.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NiMBFNzvRId4QrK5jk900ilIXdY=', 'Content-Length': '0'} 2023-09-22T21:22:24.421 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fee7a934ba90dfec-00650e0590-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:22:24.422 DEBUG:boto:path=/ffzdqu-48/obj4 2023-09-22T21:22:24.422 DEBUG:boto:auth_path=/ffzdqu-48/obj4 2023-09-22T21:22:24.422 DEBUG:boto:Method: GET 2023-09-22T21:22:24.422 DEBUG:boto:Path: /ffzdqu-48/obj4 2023-09-22T21:22:24.422 DEBUG:boto:Data: 2023-09-22T21:22:24.422 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.422 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.422 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.422 DEBUG:boto:Params: {} 2023-09-22T21:22:24.423 DEBUG:boto:Token: None 2023-09-22T21:22:24.423 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/obj4 2023-09-22T21:22:24.423 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jM6GiZCjAMDEF06K4dF+IM8tVq0= 2023-09-22T21:22:24.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jM6GiZCjAMDEF06K4dF+IM8tVq0=', 'Content-Length': '0'} 2023-09-22T21:22:24.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000949dba88ac0e28f0-00650e0590-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.470 DEBUG:boto:path=/ffzdqu-48/obj4 2023-09-22T21:22:24.471 DEBUG:boto:auth_path=/ffzdqu-48/obj4 2023-09-22T21:22:24.471 DEBUG:boto:Method: GET 2023-09-22T21:22:24.471 DEBUG:boto:Path: /ffzdqu-48/obj4 2023-09-22T21:22:24.471 DEBUG:boto:Data: 2023-09-22T21:22:24.471 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.471 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.471 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.472 DEBUG:boto:Params: {} 2023-09-22T21:22:24.472 DEBUG:boto:Token: None 2023-09-22T21:22:24.472 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/obj4 2023-09-22T21:22:24.472 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jM6GiZCjAMDEF06K4dF+IM8tVq0= 2023-09-22T21:22:24.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jM6GiZCjAMDEF06K4dF+IM8tVq0=', 'Content-Length': '0'} 2023-09-22T21:22:24.475 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd8bd17c16a0b73a-00650e0590-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.476 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:24.476 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:24.476 DEBUG:boto:path=/ffzdqu-48/?prefix=obj4 2023-09-22T21:22:24.476 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj4 2023-09-22T21:22:24.476 DEBUG:boto:Method: GET 2023-09-22T21:22:24.476 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj4 2023-09-22T21:22:24.476 DEBUG:boto:Data: 2023-09-22T21:22:24.476 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.477 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.477 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.477 DEBUG:boto:Params: {} 2023-09-22T21:22:24.477 DEBUG:boto:Token: None 2023-09-22T21:22:24.477 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/ 2023-09-22T21:22:24.478 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dj5GzxHKkx9mI04cRlp2hqB24jg= 2023-09-22T21:22:24.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dj5GzxHKkx9mI04cRlp2hqB24jg=', 'Content-Length': '0'} 2023-09-22T21:22:24.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049b04763017d9dcc-00650e0590-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.530 DEBUG:boto:b'ffzdqu-48obj41000falseobj42023-09-22T21:22:07.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:24.531 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:24.531 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:24.531 DEBUG:boto:path=/ffzdqu-48/?prefix=obj4 2023-09-22T21:22:24.531 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj4 2023-09-22T21:22:24.531 DEBUG:boto:Method: GET 2023-09-22T21:22:24.531 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj4 2023-09-22T21:22:24.531 DEBUG:boto:Data: 2023-09-22T21:22:24.531 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.532 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.532 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.532 DEBUG:boto:Params: {} 2023-09-22T21:22:24.532 DEBUG:boto:Token: None 2023-09-22T21:22:24.532 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-48/ 2023-09-22T21:22:24.532 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dj5GzxHKkx9mI04cRlp2hqB24jg= 2023-09-22T21:22:24.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dj5GzxHKkx9mI04cRlp2hqB24jg=', 'Content-Length': '0'} 2023-09-22T21:22:24.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090834f4e778c77f7-00650e0590-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.538 DEBUG:boto:b'ffzdqu-48obj41000falseobj42023-09-22T21:22:07.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:24.538 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-48 zones={a2, a1} 2023-09-22T21:22:24.538 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-49 2023-09-22T21:22:24.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:22:24.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:22:24.539 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:22:24.589 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:24.589 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:24.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:24.634+0000 7fd12bfb0a40 1 bucket sync caught up with source: 2023-09-22T21:22:24.635 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.40.6, , 00000000001.45.6, , 00000000001.36.6, , , 00000000001.29.6, , , ] 2023-09-22T21:22:24.635 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.40.6, , 00000000001.45.6, , 00000000001.36.6, , , 00000000001.29.6, , , ] 2023-09-22T21:22:24.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:24.634+0000 7fd12bfb0a40 0 bucket checkpoint complete 2023-09-22T21:22:24.653 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-49 2023-09-22T21:22:24.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:22:24.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:22:24.654 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:22:24.695 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:24.695 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:24.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:24.734+0000 7f575ce42a40 1 bucket sync caught up with source: 2023-09-22T21:22:24.736 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.125.6, , 00000000001.189.6, , 00000000001.321.6, , , 00000000001.167.6, , , ] 2023-09-22T21:22:24.736 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.125.6, , 00000000001.189.6, , 00000000001.321.6, , , 00000000001.167.6, , , ] 2023-09-22T21:22:24.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:24.734+0000 7f575ce42a40 0 bucket checkpoint complete 2023-09-22T21:22:24.753 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-49 zones={a2, a1} 2023-09-22T21:22:24.753 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:24.753 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:24.753 DEBUG:boto:Method: HEAD 2023-09-22T21:22:24.753 DEBUG:boto:Path: /ffzdqu-49/ 2023-09-22T21:22:24.753 DEBUG:boto:Data: 2023-09-22T21:22:24.753 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.754 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.754 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.754 DEBUG:boto:Params: {} 2023-09-22T21:22:24.754 DEBUG:boto:Token: None 2023-09-22T21:22:24.754 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-49/ 2023-09-22T21:22:24.754 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FHQUBwSACXpE91iWunJ0N+pQAS0= 2023-09-22T21:22:24.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FHQUBwSACXpE91iWunJ0N+pQAS0=', 'Content-Length': '0'} 2023-09-22T21:22:24.756 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f74e2f9ffefc740e-00650e0590-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.757 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:24.757 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:24.757 DEBUG:boto:Method: HEAD 2023-09-22T21:22:24.757 DEBUG:boto:Path: /ffzdqu-49/ 2023-09-22T21:22:24.757 DEBUG:boto:Data: 2023-09-22T21:22:24.757 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.757 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.757 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.758 DEBUG:boto:Params: {} 2023-09-22T21:22:24.758 DEBUG:boto:Token: None 2023-09-22T21:22:24.758 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-49/ 2023-09-22T21:22:24.758 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FHQUBwSACXpE91iWunJ0N+pQAS0= 2023-09-22T21:22:24.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FHQUBwSACXpE91iWunJ0N+pQAS0=', 'Content-Length': '0'} 2023-09-22T21:22:24.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ea92f4f301f25a5-00650e0590-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.760 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:22:24.760 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:24.760 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:24.760 DEBUG:boto:path=/ffzdqu-49/?versions&max-keys=999 2023-09-22T21:22:24.761 DEBUG:boto:auth_path=/ffzdqu-49/?versions&max-keys=999 2023-09-22T21:22:24.761 DEBUG:boto:Method: GET 2023-09-22T21:22:24.761 DEBUG:boto:Path: /ffzdqu-49/?versions&max-keys=999 2023-09-22T21:22:24.761 DEBUG:boto:Data: 2023-09-22T21:22:24.761 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.761 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.761 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.761 DEBUG:boto:Params: {} 2023-09-22T21:22:24.761 DEBUG:boto:Token: None 2023-09-22T21:22:24.762 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-49/?versions 2023-09-22T21:22:24.762 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SvZ7SHs2Yx+vHzKM1ePA0bUEnLM= 2023-09-22T21:22:24.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SvZ7SHs2Yx+vHzKM1ePA0bUEnLM=', 'Content-Length': '0'} 2023-09-22T21:22:24.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d6792507fcfdc94-00650e0590-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.764 DEBUG:boto:b'ffzdqu-49999falseobj1nulltrue2023-09-22T21:22:07.756Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:22:07.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:22:07.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:22:07.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:24.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-22T21:22:24.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-22T21:22:24.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-22T21:22:24.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-22T21:22:24.765 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:22:24.765 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:24.766 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:24.766 DEBUG:boto:path=/ffzdqu-49/?versions&max-keys=999 2023-09-22T21:22:24.766 DEBUG:boto:auth_path=/ffzdqu-49/?versions&max-keys=999 2023-09-22T21:22:24.766 DEBUG:boto:Method: GET 2023-09-22T21:22:24.766 DEBUG:boto:Path: /ffzdqu-49/?versions&max-keys=999 2023-09-22T21:22:24.766 DEBUG:boto:Data: 2023-09-22T21:22:24.766 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.766 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.766 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.766 DEBUG:boto:Params: {} 2023-09-22T21:22:24.767 DEBUG:boto:Token: None 2023-09-22T21:22:24.767 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-49/?versions 2023-09-22T21:22:24.767 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SvZ7SHs2Yx+vHzKM1ePA0bUEnLM= 2023-09-22T21:22:24.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SvZ7SHs2Yx+vHzKM1ePA0bUEnLM=', 'Content-Length': '0'} 2023-09-22T21:22:24.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017f52e44e38d092b-00650e0590-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.769 DEBUG:boto:b'ffzdqu-49999falseobj1nulltrue2023-09-22T21:22:07.756Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:22:07.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:22:07.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:22:07.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:24.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-22T21:22:24.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-22T21:22:24.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-22T21:22:24.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-22T21:22:24.770 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:24.770 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:24.770 DEBUG:boto:path=/ffzdqu-49/?versions&max-keys=999 2023-09-22T21:22:24.771 DEBUG:boto:auth_path=/ffzdqu-49/?versions&max-keys=999 2023-09-22T21:22:24.771 DEBUG:boto:Method: GET 2023-09-22T21:22:24.771 DEBUG:boto:Path: /ffzdqu-49/?versions&max-keys=999 2023-09-22T21:22:24.771 DEBUG:boto:Data: 2023-09-22T21:22:24.771 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.771 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.771 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.771 DEBUG:boto:Params: {} 2023-09-22T21:22:24.771 DEBUG:boto:Token: None 2023-09-22T21:22:24.771 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-49/?versions 2023-09-22T21:22:24.772 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SvZ7SHs2Yx+vHzKM1ePA0bUEnLM= 2023-09-22T21:22:24.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SvZ7SHs2Yx+vHzKM1ePA0bUEnLM=', 'Content-Length': '0'} 2023-09-22T21:22:24.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5dd6ba3c0187f15-00650e0590-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.774 DEBUG:boto:b'ffzdqu-49999falseobj1nulltrue2023-09-22T21:22:07.756Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:22:07.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:22:07.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:22:07.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:24.774 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:24.774 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:24.775 DEBUG:boto:path=/ffzdqu-49/?versions&max-keys=999 2023-09-22T21:22:24.775 DEBUG:boto:auth_path=/ffzdqu-49/?versions&max-keys=999 2023-09-22T21:22:24.775 DEBUG:boto:Method: GET 2023-09-22T21:22:24.775 DEBUG:boto:Path: /ffzdqu-49/?versions&max-keys=999 2023-09-22T21:22:24.775 DEBUG:boto:Data: 2023-09-22T21:22:24.775 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.775 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.775 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.775 DEBUG:boto:Params: {} 2023-09-22T21:22:24.776 DEBUG:boto:Token: None 2023-09-22T21:22:24.776 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-49/?versions 2023-09-22T21:22:24.776 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SvZ7SHs2Yx+vHzKM1ePA0bUEnLM= 2023-09-22T21:22:24.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SvZ7SHs2Yx+vHzKM1ePA0bUEnLM=', 'Content-Length': '0'} 2023-09-22T21:22:24.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046514556e9fadedf-00650e0590-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.778 DEBUG:boto:b'ffzdqu-49999falseobj1nulltrue2023-09-22T21:22:07.756Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:22:07.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:22:07.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:22:07.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:24.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:22:24.781 DEBUG:boto:path=/ffzdqu-49/obj1 2023-09-22T21:22:24.781 DEBUG:boto:auth_path=/ffzdqu-49/obj1 2023-09-22T21:22:24.781 DEBUG:boto:path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.781 DEBUG:boto:auth_path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.781 DEBUG:boto:Method: GET 2023-09-22T21:22:24.782 DEBUG:boto:Path: /ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.782 DEBUG:boto:Data: 2023-09-22T21:22:24.782 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.782 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.782 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.782 DEBUG:boto:Params: {} 2023-09-22T21:22:24.782 DEBUG:boto:Token: None 2023-09-22T21:22:24.782 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.782 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RsP8V4/JAfaE2l6iShyh8KgpRKw= 2023-09-22T21:22:24.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RsP8V4/JAfaE2l6iShyh8KgpRKw=', 'Content-Length': '0'} 2023-09-22T21:22:24.784 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f91e4e8c84f86bc7-00650e0590-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.784 DEBUG:boto:path=/ffzdqu-49/obj1 2023-09-22T21:22:24.785 DEBUG:boto:auth_path=/ffzdqu-49/obj1 2023-09-22T21:22:24.785 DEBUG:boto:path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.785 DEBUG:boto:auth_path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.785 DEBUG:boto:Method: GET 2023-09-22T21:22:24.785 DEBUG:boto:Path: /ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.785 DEBUG:boto:Data: 2023-09-22T21:22:24.785 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.785 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.785 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.785 DEBUG:boto:Params: {} 2023-09-22T21:22:24.786 DEBUG:boto:Token: None 2023-09-22T21:22:24.786 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.786 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RsP8V4/JAfaE2l6iShyh8KgpRKw= 2023-09-22T21:22:24.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RsP8V4/JAfaE2l6iShyh8KgpRKw=', 'Content-Length': '0'} 2023-09-22T21:22:24.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0b2a34b5d3546b3-00650e0590-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.788 DEBUG:boto:path=/ffzdqu-49/obj1 2023-09-22T21:22:24.788 DEBUG:boto:auth_path=/ffzdqu-49/obj1 2023-09-22T21:22:24.788 DEBUG:boto:path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.789 DEBUG:boto:auth_path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.789 DEBUG:boto:Method: HEAD 2023-09-22T21:22:24.789 DEBUG:boto:Path: /ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.789 DEBUG:boto:Data: 2023-09-22T21:22:24.789 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.789 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.789 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.789 DEBUG:boto:Params: {} 2023-09-22T21:22:24.789 DEBUG:boto:Token: None 2023-09-22T21:22:24.790 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.790 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uY9XNLtXdwSC7pgOalkbzB7qlHc= 2023-09-22T21:22:24.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uY9XNLtXdwSC7pgOalkbzB7qlHc=', 'Content-Length': '0'} 2023-09-22T21:22:24.791 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077ca37481abf95ed-00650e0590-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.791 DEBUG:boto:path=/ffzdqu-49/obj1 2023-09-22T21:22:24.792 DEBUG:boto:auth_path=/ffzdqu-49/obj1 2023-09-22T21:22:24.792 DEBUG:boto:path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.792 DEBUG:boto:auth_path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.792 DEBUG:boto:Method: HEAD 2023-09-22T21:22:24.792 DEBUG:boto:Path: /ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.792 DEBUG:boto:Data: 2023-09-22T21:22:24.792 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.792 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.792 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.792 DEBUG:boto:Params: {} 2023-09-22T21:22:24.793 DEBUG:boto:Token: None 2023-09-22T21:22:24.793 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.793 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uY9XNLtXdwSC7pgOalkbzB7qlHc= 2023-09-22T21:22:24.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uY9XNLtXdwSC7pgOalkbzB7qlHc=', 'Content-Length': '0'} 2023-09-22T21:22:24.795 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ec5580da11b61bb-00650e0590-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:22:24.795 DEBUG:boto:path=/ffzdqu-49/obj1 2023-09-22T21:22:24.795 DEBUG:boto:auth_path=/ffzdqu-49/obj1 2023-09-22T21:22:24.795 DEBUG:boto:path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.796 DEBUG:boto:auth_path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.796 DEBUG:boto:Method: GET 2023-09-22T21:22:24.796 DEBUG:boto:Path: /ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.796 DEBUG:boto:Data: 2023-09-22T21:22:24.796 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.796 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.796 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.796 DEBUG:boto:Params: {} 2023-09-22T21:22:24.796 DEBUG:boto:Token: None 2023-09-22T21:22:24.797 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.797 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RsP8V4/JAfaE2l6iShyh8KgpRKw= 2023-09-22T21:22:24.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RsP8V4/JAfaE2l6iShyh8KgpRKw=', 'Content-Length': '0'} 2023-09-22T21:22:24.798 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a050b2fe22ee690d-00650e0590-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.798 DEBUG:boto:path=/ffzdqu-49/obj1 2023-09-22T21:22:24.799 DEBUG:boto:auth_path=/ffzdqu-49/obj1 2023-09-22T21:22:24.799 DEBUG:boto:path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.799 DEBUG:boto:auth_path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.799 DEBUG:boto:Method: GET 2023-09-22T21:22:24.799 DEBUG:boto:Path: /ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.799 DEBUG:boto:Data: 2023-09-22T21:22:24.799 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.799 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.799 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.799 DEBUG:boto:Params: {} 2023-09-22T21:22:24.800 DEBUG:boto:Token: None 2023-09-22T21:22:24.800 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-49/obj1?versionId=null 2023-09-22T21:22:24.800 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RsP8V4/JAfaE2l6iShyh8KgpRKw= 2023-09-22T21:22:24.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RsP8V4/JAfaE2l6iShyh8KgpRKw=', 'Content-Length': '0'} 2023-09-22T21:22:24.801 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020763746607570d3-00650e0590-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.842 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:24.843 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:24.843 DEBUG:boto:path=/ffzdqu-49/?prefix=obj1 2023-09-22T21:22:24.843 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj1 2023-09-22T21:22:24.843 DEBUG:boto:Method: GET 2023-09-22T21:22:24.843 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj1 2023-09-22T21:22:24.843 DEBUG:boto:Data: 2023-09-22T21:22:24.843 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.843 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.843 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.843 DEBUG:boto:Params: {} 2023-09-22T21:22:24.844 DEBUG:boto:Token: None 2023-09-22T21:22:24.844 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-49/ 2023-09-22T21:22:24.844 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hbdPZ4tLCRJn5UlNe/vmZJbJQig= 2023-09-22T21:22:24.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hbdPZ4tLCRJn5UlNe/vmZJbJQig=', 'Content-Length': '0'} 2023-09-22T21:22:24.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061929d2cbad0e0b0-00650e0590-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.848 DEBUG:boto:b'ffzdqu-49obj11000falseobj12023-09-22T21:22:07.756Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:24.848 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:24.848 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:24.848 DEBUG:boto:path=/ffzdqu-49/?prefix=obj1 2023-09-22T21:22:24.848 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj1 2023-09-22T21:22:24.849 DEBUG:boto:Method: GET 2023-09-22T21:22:24.849 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj1 2023-09-22T21:22:24.849 DEBUG:boto:Data: 2023-09-22T21:22:24.849 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.849 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.849 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.849 DEBUG:boto:Params: {} 2023-09-22T21:22:24.849 DEBUG:boto:Token: None 2023-09-22T21:22:24.849 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-49/ 2023-09-22T21:22:24.850 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hbdPZ4tLCRJn5UlNe/vmZJbJQig= 2023-09-22T21:22:24.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hbdPZ4tLCRJn5UlNe/vmZJbJQig=', 'Content-Length': '0'} 2023-09-22T21:22:24.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be6e61d187f98d3a-00650e0590-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.898 DEBUG:boto:b'ffzdqu-49obj11000falseobj12023-09-22T21:22:07.756Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:24.899 DEBUG:boto:path=/ffzdqu-49/obj1 2023-09-22T21:22:24.899 DEBUG:boto:auth_path=/ffzdqu-49/obj1 2023-09-22T21:22:24.899 DEBUG:boto:Method: HEAD 2023-09-22T21:22:24.899 DEBUG:boto:Path: /ffzdqu-49/obj1 2023-09-22T21:22:24.899 DEBUG:boto:Data: 2023-09-22T21:22:24.899 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.899 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.900 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.900 DEBUG:boto:Params: {} 2023-09-22T21:22:24.900 DEBUG:boto:Token: None 2023-09-22T21:22:24.900 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-49/obj1 2023-09-22T21:22:24.900 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:75Mx6mnZ7HFNAa7LJYP7pdeKO7c= 2023-09-22T21:22:24.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:75Mx6mnZ7HFNAa7LJYP7pdeKO7c=', 'Content-Length': '0'} 2023-09-22T21:22:24.903 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000106c37809faf1f23-00650e0590-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.903 DEBUG:boto:path=/ffzdqu-49/obj1 2023-09-22T21:22:24.903 DEBUG:boto:auth_path=/ffzdqu-49/obj1 2023-09-22T21:22:24.903 DEBUG:boto:Method: HEAD 2023-09-22T21:22:24.903 DEBUG:boto:Path: /ffzdqu-49/obj1 2023-09-22T21:22:24.904 DEBUG:boto:Data: 2023-09-22T21:22:24.904 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.904 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.904 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.904 DEBUG:boto:Params: {} 2023-09-22T21:22:24.904 DEBUG:boto:Token: None 2023-09-22T21:22:24.904 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-49/obj1 2023-09-22T21:22:24.904 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:75Mx6mnZ7HFNAa7LJYP7pdeKO7c= 2023-09-22T21:22:24.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:75Mx6mnZ7HFNAa7LJYP7pdeKO7c=', 'Content-Length': '0'} 2023-09-22T21:22:24.907 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c01f68961b9c52ea-00650e0590-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:22:24.908 DEBUG:boto:path=/ffzdqu-49/obj1 2023-09-22T21:22:24.908 DEBUG:boto:auth_path=/ffzdqu-49/obj1 2023-09-22T21:22:24.908 DEBUG:boto:Method: GET 2023-09-22T21:22:24.908 DEBUG:boto:Path: /ffzdqu-49/obj1 2023-09-22T21:22:24.908 DEBUG:boto:Data: 2023-09-22T21:22:24.908 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.908 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.908 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.908 DEBUG:boto:Params: {} 2023-09-22T21:22:24.909 DEBUG:boto:Token: None 2023-09-22T21:22:24.909 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-49/obj1 2023-09-22T21:22:24.909 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aI8+2tRLpo7el5bKMSuFN8KgLVw= 2023-09-22T21:22:24.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aI8+2tRLpo7el5bKMSuFN8KgLVw=', 'Content-Length': '0'} 2023-09-22T21:22:24.912 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002af076557611698-00650e0590-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.912 DEBUG:boto:path=/ffzdqu-49/obj1 2023-09-22T21:22:24.912 DEBUG:boto:auth_path=/ffzdqu-49/obj1 2023-09-22T21:22:24.912 DEBUG:boto:Method: GET 2023-09-22T21:22:24.912 DEBUG:boto:Path: /ffzdqu-49/obj1 2023-09-22T21:22:24.912 DEBUG:boto:Data: 2023-09-22T21:22:24.912 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.913 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.913 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.913 DEBUG:boto:Params: {} 2023-09-22T21:22:24.913 DEBUG:boto:Token: None 2023-09-22T21:22:24.913 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-49/obj1 2023-09-22T21:22:24.913 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aI8+2tRLpo7el5bKMSuFN8KgLVw= 2023-09-22T21:22:24.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aI8+2tRLpo7el5bKMSuFN8KgLVw=', 'Content-Length': '0'} 2023-09-22T21:22:24.916 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000199608762bb371a6-00650e0590-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.962 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:24.963 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:24.963 DEBUG:boto:path=/ffzdqu-49/?prefix=obj1 2023-09-22T21:22:24.963 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj1 2023-09-22T21:22:24.963 DEBUG:boto:Method: GET 2023-09-22T21:22:24.963 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj1 2023-09-22T21:22:24.963 DEBUG:boto:Data: 2023-09-22T21:22:24.963 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.964 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:24.964 DEBUG:boto:Port: 8001 2023-09-22T21:22:24.964 DEBUG:boto:Params: {} 2023-09-22T21:22:24.965 DEBUG:boto:Token: None 2023-09-22T21:22:24.965 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-49/ 2023-09-22T21:22:24.965 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hbdPZ4tLCRJn5UlNe/vmZJbJQig= 2023-09-22T21:22:24.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hbdPZ4tLCRJn5UlNe/vmZJbJQig=', 'Content-Length': '0'} 2023-09-22T21:22:24.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ef8991a6a369ba4-00650e0590-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:24.970 DEBUG:boto:b'ffzdqu-49obj11000falseobj12023-09-22T21:22:07.756Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:24.970 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:24.971 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:24.971 DEBUG:boto:path=/ffzdqu-49/?prefix=obj1 2023-09-22T21:22:24.971 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj1 2023-09-22T21:22:24.971 DEBUG:boto:Method: GET 2023-09-22T21:22:24.971 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj1 2023-09-22T21:22:24.971 DEBUG:boto:Data: 2023-09-22T21:22:24.971 DEBUG:boto:Headers: {} 2023-09-22T21:22:24.971 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:24.971 DEBUG:boto:Port: 8000 2023-09-22T21:22:24.971 DEBUG:boto:Params: {} 2023-09-22T21:22:24.972 DEBUG:boto:Token: None 2023-09-22T21:22:24.972 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:24 GMT /ffzdqu-49/ 2023-09-22T21:22:24.972 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hbdPZ4tLCRJn5UlNe/vmZJbJQig= 2023-09-22T21:22:24.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hbdPZ4tLCRJn5UlNe/vmZJbJQig=', 'Content-Length': '0'} 2023-09-22T21:22:24.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047e65f732b70c096-00650e0590-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.022 DEBUG:boto:b'ffzdqu-49obj11000falseobj12023-09-22T21:22:07.756Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:25.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:22:25.023 DEBUG:boto:path=/ffzdqu-49/obj2 2023-09-22T21:22:25.023 DEBUG:boto:auth_path=/ffzdqu-49/obj2 2023-09-22T21:22:25.023 DEBUG:boto:path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.023 DEBUG:boto:auth_path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.024 DEBUG:boto:Method: GET 2023-09-22T21:22:25.024 DEBUG:boto:Path: /ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.024 DEBUG:boto:Data: 2023-09-22T21:22:25.024 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.024 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:25.024 DEBUG:boto:Port: 8001 2023-09-22T21:22:25.024 DEBUG:boto:Params: {} 2023-09-22T21:22:25.024 DEBUG:boto:Token: None 2023-09-22T21:22:25.025 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.025 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QXkS2kbTAxqQKhtDgSNvYD8eBPM= 2023-09-22T21:22:25.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QXkS2kbTAxqQKhtDgSNvYD8eBPM=', 'Content-Length': '0'} 2023-09-22T21:22:25.028 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047a460e9d31cd013-00650e0591-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.028 DEBUG:boto:path=/ffzdqu-49/obj2 2023-09-22T21:22:25.029 DEBUG:boto:auth_path=/ffzdqu-49/obj2 2023-09-22T21:22:25.029 DEBUG:boto:path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.029 DEBUG:boto:auth_path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.029 DEBUG:boto:Method: GET 2023-09-22T21:22:25.029 DEBUG:boto:Path: /ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.029 DEBUG:boto:Data: 2023-09-22T21:22:25.029 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.030 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:25.030 DEBUG:boto:Port: 8000 2023-09-22T21:22:25.030 DEBUG:boto:Params: {} 2023-09-22T21:22:25.030 DEBUG:boto:Token: None 2023-09-22T21:22:25.030 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.030 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QXkS2kbTAxqQKhtDgSNvYD8eBPM= 2023-09-22T21:22:25.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QXkS2kbTAxqQKhtDgSNvYD8eBPM=', 'Content-Length': '0'} 2023-09-22T21:22:25.033 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000481bb55e05d2d7bb-00650e0591-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.033 DEBUG:boto:path=/ffzdqu-49/obj2 2023-09-22T21:22:25.034 DEBUG:boto:auth_path=/ffzdqu-49/obj2 2023-09-22T21:22:25.034 DEBUG:boto:path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.034 DEBUG:boto:auth_path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.034 DEBUG:boto:Method: HEAD 2023-09-22T21:22:25.034 DEBUG:boto:Path: /ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.034 DEBUG:boto:Data: 2023-09-22T21:22:25.034 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.034 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:25.034 DEBUG:boto:Port: 8001 2023-09-22T21:22:25.035 DEBUG:boto:Params: {} 2023-09-22T21:22:25.035 DEBUG:boto:Token: None 2023-09-22T21:22:25.035 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.035 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LFy7wmWftC8bHYXcGOVxqnn6nm4= 2023-09-22T21:22:25.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LFy7wmWftC8bHYXcGOVxqnn6nm4=', 'Content-Length': '0'} 2023-09-22T21:22:25.038 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086db1fc01c98c35e-00650e0591-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.039 DEBUG:boto:path=/ffzdqu-49/obj2 2023-09-22T21:22:25.039 DEBUG:boto:auth_path=/ffzdqu-49/obj2 2023-09-22T21:22:25.039 DEBUG:boto:path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.039 DEBUG:boto:auth_path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.039 DEBUG:boto:Method: HEAD 2023-09-22T21:22:25.039 DEBUG:boto:Path: /ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.039 DEBUG:boto:Data: 2023-09-22T21:22:25.039 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.040 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:25.040 DEBUG:boto:Port: 8000 2023-09-22T21:22:25.040 DEBUG:boto:Params: {} 2023-09-22T21:22:25.040 DEBUG:boto:Token: None 2023-09-22T21:22:25.040 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.040 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LFy7wmWftC8bHYXcGOVxqnn6nm4= 2023-09-22T21:22:25.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LFy7wmWftC8bHYXcGOVxqnn6nm4=', 'Content-Length': '0'} 2023-09-22T21:22:25.043 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f77a5851249ed7d-00650e0591-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:22:25.044 DEBUG:boto:path=/ffzdqu-49/obj2 2023-09-22T21:22:25.044 DEBUG:boto:auth_path=/ffzdqu-49/obj2 2023-09-22T21:22:25.044 DEBUG:boto:path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.044 DEBUG:boto:auth_path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.044 DEBUG:boto:Method: GET 2023-09-22T21:22:25.044 DEBUG:boto:Path: /ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.044 DEBUG:boto:Data: 2023-09-22T21:22:25.044 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.045 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:25.045 DEBUG:boto:Port: 8001 2023-09-22T21:22:25.045 DEBUG:boto:Params: {} 2023-09-22T21:22:25.045 DEBUG:boto:Token: None 2023-09-22T21:22:25.045 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.045 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QXkS2kbTAxqQKhtDgSNvYD8eBPM= 2023-09-22T21:22:25.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QXkS2kbTAxqQKhtDgSNvYD8eBPM=', 'Content-Length': '0'} 2023-09-22T21:22:25.048 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003fc31a0144b50da2-00650e0591-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.094 DEBUG:boto:path=/ffzdqu-49/obj2 2023-09-22T21:22:25.095 DEBUG:boto:auth_path=/ffzdqu-49/obj2 2023-09-22T21:22:25.095 DEBUG:boto:path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.095 DEBUG:boto:auth_path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.095 DEBUG:boto:Method: GET 2023-09-22T21:22:25.095 DEBUG:boto:Path: /ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.095 DEBUG:boto:Data: 2023-09-22T21:22:25.095 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:25.096 DEBUG:boto:Port: 8000 2023-09-22T21:22:25.096 DEBUG:boto:Params: {} 2023-09-22T21:22:25.096 DEBUG:boto:Token: None 2023-09-22T21:22:25.096 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj2?versionId=null 2023-09-22T21:22:25.096 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QXkS2kbTAxqQKhtDgSNvYD8eBPM= 2023-09-22T21:22:25.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QXkS2kbTAxqQKhtDgSNvYD8eBPM=', 'Content-Length': '0'} 2023-09-22T21:22:25.099 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b64cdb788688e283-00650e0591-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.142 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:25.143 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:25.143 DEBUG:boto:path=/ffzdqu-49/?prefix=obj2 2023-09-22T21:22:25.143 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj2 2023-09-22T21:22:25.143 DEBUG:boto:Method: GET 2023-09-22T21:22:25.143 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj2 2023-09-22T21:22:25.143 DEBUG:boto:Data: 2023-09-22T21:22:25.143 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.143 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:25.143 DEBUG:boto:Port: 8001 2023-09-22T21:22:25.144 DEBUG:boto:Params: {} 2023-09-22T21:22:25.144 DEBUG:boto:Token: None 2023-09-22T21:22:25.144 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/ 2023-09-22T21:22:25.144 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0= 2023-09-22T21:22:25.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0=', 'Content-Length': '0'} 2023-09-22T21:22:25.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036c92cebc8247bb6-00650e0591-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.149 DEBUG:boto:b'ffzdqu-49obj21000falseobj22023-09-22T21:22:07.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:25.149 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:25.149 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:25.149 DEBUG:boto:path=/ffzdqu-49/?prefix=obj2 2023-09-22T21:22:25.150 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj2 2023-09-22T21:22:25.150 DEBUG:boto:Method: GET 2023-09-22T21:22:25.150 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj2 2023-09-22T21:22:25.150 DEBUG:boto:Data: 2023-09-22T21:22:25.150 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.150 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:25.150 DEBUG:boto:Port: 8000 2023-09-22T21:22:25.150 DEBUG:boto:Params: {} 2023-09-22T21:22:25.150 DEBUG:boto:Token: None 2023-09-22T21:22:25.151 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/ 2023-09-22T21:22:25.151 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0= 2023-09-22T21:22:25.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0=', 'Content-Length': '0'} 2023-09-22T21:22:25.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3f79583971637e3-00650e0591-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.156 DEBUG:boto:b'ffzdqu-49obj21000falseobj22023-09-22T21:22:07.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:25.156 DEBUG:boto:path=/ffzdqu-49/obj2 2023-09-22T21:22:25.156 DEBUG:boto:auth_path=/ffzdqu-49/obj2 2023-09-22T21:22:25.157 DEBUG:boto:Method: HEAD 2023-09-22T21:22:25.157 DEBUG:boto:Path: /ffzdqu-49/obj2 2023-09-22T21:22:25.157 DEBUG:boto:Data: 2023-09-22T21:22:25.157 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.157 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:25.157 DEBUG:boto:Port: 8001 2023-09-22T21:22:25.157 DEBUG:boto:Params: {} 2023-09-22T21:22:25.157 DEBUG:boto:Token: None 2023-09-22T21:22:25.157 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj2 2023-09-22T21:22:25.158 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:umpQm0uzOP+rSoHVB4HBJngFW0s= 2023-09-22T21:22:25.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:umpQm0uzOP+rSoHVB4HBJngFW0s=', 'Content-Length': '0'} 2023-09-22T21:22:25.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc23d247b6f97594-00650e0591-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.160 DEBUG:boto:path=/ffzdqu-49/obj2 2023-09-22T21:22:25.161 DEBUG:boto:auth_path=/ffzdqu-49/obj2 2023-09-22T21:22:25.161 DEBUG:boto:Method: HEAD 2023-09-22T21:22:25.161 DEBUG:boto:Path: /ffzdqu-49/obj2 2023-09-22T21:22:25.161 DEBUG:boto:Data: 2023-09-22T21:22:25.161 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.161 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:25.161 DEBUG:boto:Port: 8000 2023-09-22T21:22:25.161 DEBUG:boto:Params: {} 2023-09-22T21:22:25.161 DEBUG:boto:Token: None 2023-09-22T21:22:25.162 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj2 2023-09-22T21:22:25.162 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:umpQm0uzOP+rSoHVB4HBJngFW0s= 2023-09-22T21:22:25.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:umpQm0uzOP+rSoHVB4HBJngFW0s=', 'Content-Length': '0'} 2023-09-22T21:22:25.164 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086052e7ae44a73eb-00650e0591-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:22:25.165 DEBUG:boto:path=/ffzdqu-49/obj2 2023-09-22T21:22:25.165 DEBUG:boto:auth_path=/ffzdqu-49/obj2 2023-09-22T21:22:25.165 DEBUG:boto:Method: GET 2023-09-22T21:22:25.165 DEBUG:boto:Path: /ffzdqu-49/obj2 2023-09-22T21:22:25.165 DEBUG:boto:Data: 2023-09-22T21:22:25.165 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.165 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:25.165 DEBUG:boto:Port: 8001 2023-09-22T21:22:25.166 DEBUG:boto:Params: {} 2023-09-22T21:22:25.166 DEBUG:boto:Token: None 2023-09-22T21:22:25.166 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj2 2023-09-22T21:22:25.166 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EdiPmDAItZN2joByBt87FyXQ+XY= 2023-09-22T21:22:25.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EdiPmDAItZN2joByBt87FyXQ+XY=', 'Content-Length': '0'} 2023-09-22T21:22:25.169 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a80aa7ad06e4b890-00650e0591-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.169 DEBUG:boto:path=/ffzdqu-49/obj2 2023-09-22T21:22:25.169 DEBUG:boto:auth_path=/ffzdqu-49/obj2 2023-09-22T21:22:25.169 DEBUG:boto:Method: GET 2023-09-22T21:22:25.169 DEBUG:boto:Path: /ffzdqu-49/obj2 2023-09-22T21:22:25.169 DEBUG:boto:Data: 2023-09-22T21:22:25.169 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.170 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:25.170 DEBUG:boto:Port: 8000 2023-09-22T21:22:25.170 DEBUG:boto:Params: {} 2023-09-22T21:22:25.170 DEBUG:boto:Token: None 2023-09-22T21:22:25.170 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj2 2023-09-22T21:22:25.170 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EdiPmDAItZN2joByBt87FyXQ+XY= 2023-09-22T21:22:25.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EdiPmDAItZN2joByBt87FyXQ+XY=', 'Content-Length': '0'} 2023-09-22T21:22:25.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4fbf3aa3e887c36-00650e0591-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.214 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:25.215 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:25.215 DEBUG:boto:path=/ffzdqu-49/?prefix=obj2 2023-09-22T21:22:25.215 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj2 2023-09-22T21:22:25.215 DEBUG:boto:Method: GET 2023-09-22T21:22:25.215 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj2 2023-09-22T21:22:25.215 DEBUG:boto:Data: 2023-09-22T21:22:25.215 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.216 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:25.216 DEBUG:boto:Port: 8001 2023-09-22T21:22:25.216 DEBUG:boto:Params: {} 2023-09-22T21:22:25.216 DEBUG:boto:Token: None 2023-09-22T21:22:25.216 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/ 2023-09-22T21:22:25.216 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0= 2023-09-22T21:22:25.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0=', 'Content-Length': '0'} 2023-09-22T21:22:25.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000575eee64d56ff869-00650e0591-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.221 DEBUG:boto:b'ffzdqu-49obj21000falseobj22023-09-22T21:22:07.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:25.222 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:25.222 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:25.222 DEBUG:boto:path=/ffzdqu-49/?prefix=obj2 2023-09-22T21:22:25.222 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj2 2023-09-22T21:22:25.222 DEBUG:boto:Method: GET 2023-09-22T21:22:25.222 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj2 2023-09-22T21:22:25.222 DEBUG:boto:Data: 2023-09-22T21:22:25.222 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.223 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:25.223 DEBUG:boto:Port: 8000 2023-09-22T21:22:25.223 DEBUG:boto:Params: {} 2023-09-22T21:22:25.223 DEBUG:boto:Token: None 2023-09-22T21:22:25.223 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/ 2023-09-22T21:22:25.223 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0= 2023-09-22T21:22:25.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0=', 'Content-Length': '0'} 2023-09-22T21:22:25.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ea6eb8746c7d35a-00650e0591-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.228 DEBUG:boto:b'ffzdqu-49obj21000falseobj22023-09-22T21:22:07.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:25.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:22:25.228 DEBUG:boto:path=/ffzdqu-49/obj3 2023-09-22T21:22:25.229 DEBUG:boto:auth_path=/ffzdqu-49/obj3 2023-09-22T21:22:25.229 DEBUG:boto:path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.229 DEBUG:boto:auth_path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.229 DEBUG:boto:Method: GET 2023-09-22T21:22:25.229 DEBUG:boto:Path: /ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.229 DEBUG:boto:Data: 2023-09-22T21:22:25.229 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.229 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:25.229 DEBUG:boto:Port: 8001 2023-09-22T21:22:25.229 DEBUG:boto:Params: {} 2023-09-22T21:22:25.230 DEBUG:boto:Token: None 2023-09-22T21:22:25.230 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.230 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PUF78ix6dVn72rCh6ySyfx+QUIg= 2023-09-22T21:22:25.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PUF78ix6dVn72rCh6ySyfx+QUIg=', 'Content-Length': '0'} 2023-09-22T21:22:25.233 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097482af928dc7f57-00650e0591-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.233 DEBUG:boto:path=/ffzdqu-49/obj3 2023-09-22T21:22:25.233 DEBUG:boto:auth_path=/ffzdqu-49/obj3 2023-09-22T21:22:25.234 DEBUG:boto:path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.234 DEBUG:boto:auth_path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.234 DEBUG:boto:Method: GET 2023-09-22T21:22:25.234 DEBUG:boto:Path: /ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.234 DEBUG:boto:Data: 2023-09-22T21:22:25.234 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.234 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:25.234 DEBUG:boto:Port: 8000 2023-09-22T21:22:25.234 DEBUG:boto:Params: {} 2023-09-22T21:22:25.235 DEBUG:boto:Token: None 2023-09-22T21:22:25.235 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.235 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PUF78ix6dVn72rCh6ySyfx+QUIg= 2023-09-22T21:22:25.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PUF78ix6dVn72rCh6ySyfx+QUIg=', 'Content-Length': '0'} 2023-09-22T21:22:25.238 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b190022004b9c092-00650e0591-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.238 DEBUG:boto:path=/ffzdqu-49/obj3 2023-09-22T21:22:25.238 DEBUG:boto:auth_path=/ffzdqu-49/obj3 2023-09-22T21:22:25.238 DEBUG:boto:path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.238 DEBUG:boto:auth_path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.238 DEBUG:boto:Method: HEAD 2023-09-22T21:22:25.239 DEBUG:boto:Path: /ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.239 DEBUG:boto:Data: 2023-09-22T21:22:25.239 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.239 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:25.239 DEBUG:boto:Port: 8001 2023-09-22T21:22:25.239 DEBUG:boto:Params: {} 2023-09-22T21:22:25.239 DEBUG:boto:Token: None 2023-09-22T21:22:25.239 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.239 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nTkkHt+EVqvQngrRq36JCvIwuI8= 2023-09-22T21:22:25.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nTkkHt+EVqvQngrRq36JCvIwuI8=', 'Content-Length': '0'} 2023-09-22T21:22:25.242 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b207fdc00d2b5d4d-00650e0591-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.242 DEBUG:boto:path=/ffzdqu-49/obj3 2023-09-22T21:22:25.242 DEBUG:boto:auth_path=/ffzdqu-49/obj3 2023-09-22T21:22:25.242 DEBUG:boto:path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.243 DEBUG:boto:auth_path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.243 DEBUG:boto:Method: HEAD 2023-09-22T21:22:25.243 DEBUG:boto:Path: /ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.243 DEBUG:boto:Data: 2023-09-22T21:22:25.243 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.243 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:25.243 DEBUG:boto:Port: 8000 2023-09-22T21:22:25.243 DEBUG:boto:Params: {} 2023-09-22T21:22:25.243 DEBUG:boto:Token: None 2023-09-22T21:22:25.244 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.244 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nTkkHt+EVqvQngrRq36JCvIwuI8= 2023-09-22T21:22:25.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nTkkHt+EVqvQngrRq36JCvIwuI8=', 'Content-Length': '0'} 2023-09-22T21:22:25.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f9fd37ded6cfe82-00650e0591-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:22:25.247 DEBUG:boto:path=/ffzdqu-49/obj3 2023-09-22T21:22:25.247 DEBUG:boto:auth_path=/ffzdqu-49/obj3 2023-09-22T21:22:25.247 DEBUG:boto:path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.247 DEBUG:boto:auth_path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.247 DEBUG:boto:Method: GET 2023-09-22T21:22:25.247 DEBUG:boto:Path: /ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.248 DEBUG:boto:Data: 2023-09-22T21:22:25.248 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.248 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:25.248 DEBUG:boto:Port: 8001 2023-09-22T21:22:25.248 DEBUG:boto:Params: {} 2023-09-22T21:22:25.248 DEBUG:boto:Token: None 2023-09-22T21:22:25.248 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.248 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PUF78ix6dVn72rCh6ySyfx+QUIg= 2023-09-22T21:22:25.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PUF78ix6dVn72rCh6ySyfx+QUIg=', 'Content-Length': '0'} 2023-09-22T21:22:25.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ec51ff724590062-00650e0591-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.252 DEBUG:boto:path=/ffzdqu-49/obj3 2023-09-22T21:22:25.252 DEBUG:boto:auth_path=/ffzdqu-49/obj3 2023-09-22T21:22:25.252 DEBUG:boto:path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.252 DEBUG:boto:auth_path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.252 DEBUG:boto:Method: GET 2023-09-22T21:22:25.252 DEBUG:boto:Path: /ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.252 DEBUG:boto:Data: 2023-09-22T21:22:25.252 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.252 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:25.253 DEBUG:boto:Port: 8000 2023-09-22T21:22:25.253 DEBUG:boto:Params: {} 2023-09-22T21:22:25.253 DEBUG:boto:Token: None 2023-09-22T21:22:25.253 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj3?versionId=null 2023-09-22T21:22:25.253 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PUF78ix6dVn72rCh6ySyfx+QUIg= 2023-09-22T21:22:25.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PUF78ix6dVn72rCh6ySyfx+QUIg=', 'Content-Length': '0'} 2023-09-22T21:22:25.256 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1ee43105b349e90-00650e0591-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.298 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:25.299 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:25.299 DEBUG:boto:path=/ffzdqu-49/?prefix=obj3 2023-09-22T21:22:25.299 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj3 2023-09-22T21:22:25.299 DEBUG:boto:Method: GET 2023-09-22T21:22:25.299 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj3 2023-09-22T21:22:25.299 DEBUG:boto:Data: 2023-09-22T21:22:25.299 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.299 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:25.299 DEBUG:boto:Port: 8001 2023-09-22T21:22:25.299 DEBUG:boto:Params: {} 2023-09-22T21:22:25.300 DEBUG:boto:Token: None 2023-09-22T21:22:25.300 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/ 2023-09-22T21:22:25.300 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0= 2023-09-22T21:22:25.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0=', 'Content-Length': '0'} 2023-09-22T21:22:25.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e130b36503f77ef-00650e0591-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.305 DEBUG:boto:b'ffzdqu-49obj31000falseobj32023-09-22T21:22:07.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:25.305 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:25.306 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:25.306 DEBUG:boto:path=/ffzdqu-49/?prefix=obj3 2023-09-22T21:22:25.306 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj3 2023-09-22T21:22:25.306 DEBUG:boto:Method: GET 2023-09-22T21:22:25.306 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj3 2023-09-22T21:22:25.306 DEBUG:boto:Data: 2023-09-22T21:22:25.306 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.306 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:25.306 DEBUG:boto:Port: 8000 2023-09-22T21:22:25.307 DEBUG:boto:Params: {} 2023-09-22T21:22:25.307 DEBUG:boto:Token: None 2023-09-22T21:22:25.307 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/ 2023-09-22T21:22:25.307 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0= 2023-09-22T21:22:25.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0=', 'Content-Length': '0'} 2023-09-22T21:22:25.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff4d4e1ddb55c6ae-00650e0591-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.311 DEBUG:boto:b'ffzdqu-49obj31000falseobj32023-09-22T21:22:07.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:25.312 DEBUG:boto:path=/ffzdqu-49/obj3 2023-09-22T21:22:25.312 DEBUG:boto:auth_path=/ffzdqu-49/obj3 2023-09-22T21:22:25.312 DEBUG:boto:Method: HEAD 2023-09-22T21:22:25.312 DEBUG:boto:Path: /ffzdqu-49/obj3 2023-09-22T21:22:25.312 DEBUG:boto:Data: 2023-09-22T21:22:25.312 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.312 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:25.313 DEBUG:boto:Port: 8001 2023-09-22T21:22:25.313 DEBUG:boto:Params: {} 2023-09-22T21:22:25.313 DEBUG:boto:Token: None 2023-09-22T21:22:25.313 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj3 2023-09-22T21:22:25.313 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:432JOWWuCPIGsINvNv3KCjmQi24= 2023-09-22T21:22:25.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:432JOWWuCPIGsINvNv3KCjmQi24=', 'Content-Length': '0'} 2023-09-22T21:22:25.316 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002220f7463c10f289-00650e0591-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.316 DEBUG:boto:path=/ffzdqu-49/obj3 2023-09-22T21:22:25.316 DEBUG:boto:auth_path=/ffzdqu-49/obj3 2023-09-22T21:22:25.316 DEBUG:boto:Method: HEAD 2023-09-22T21:22:25.316 DEBUG:boto:Path: /ffzdqu-49/obj3 2023-09-22T21:22:25.317 DEBUG:boto:Data: 2023-09-22T21:22:25.317 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.317 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:25.317 DEBUG:boto:Port: 8000 2023-09-22T21:22:25.317 DEBUG:boto:Params: {} 2023-09-22T21:22:25.317 DEBUG:boto:Token: None 2023-09-22T21:22:25.317 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj3 2023-09-22T21:22:25.317 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:432JOWWuCPIGsINvNv3KCjmQi24= 2023-09-22T21:22:25.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:432JOWWuCPIGsINvNv3KCjmQi24=', 'Content-Length': '0'} 2023-09-22T21:22:25.320 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000bb9a67ae3db5e3b-00650e0591-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:22:25.321 DEBUG:boto:path=/ffzdqu-49/obj3 2023-09-22T21:22:25.321 DEBUG:boto:auth_path=/ffzdqu-49/obj3 2023-09-22T21:22:25.321 DEBUG:boto:Method: GET 2023-09-22T21:22:25.321 DEBUG:boto:Path: /ffzdqu-49/obj3 2023-09-22T21:22:25.321 DEBUG:boto:Data: 2023-09-22T21:22:25.321 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.321 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:25.321 DEBUG:boto:Port: 8001 2023-09-22T21:22:25.321 DEBUG:boto:Params: {} 2023-09-22T21:22:25.321 DEBUG:boto:Token: None 2023-09-22T21:22:25.322 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj3 2023-09-22T21:22:25.322 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Nbl5GDJUv9Z7v/FpGMp5Er02NeI= 2023-09-22T21:22:25.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Nbl5GDJUv9Z7v/FpGMp5Er02NeI=', 'Content-Length': '0'} 2023-09-22T21:22:25.324 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ac2ac231a5cac55-00650e0591-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.366 DEBUG:boto:path=/ffzdqu-49/obj3 2023-09-22T21:22:25.366 DEBUG:boto:auth_path=/ffzdqu-49/obj3 2023-09-22T21:22:25.367 DEBUG:boto:Method: GET 2023-09-22T21:22:25.367 DEBUG:boto:Path: /ffzdqu-49/obj3 2023-09-22T21:22:25.367 DEBUG:boto:Data: 2023-09-22T21:22:25.367 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.367 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:25.367 DEBUG:boto:Port: 8000 2023-09-22T21:22:25.367 DEBUG:boto:Params: {} 2023-09-22T21:22:25.367 DEBUG:boto:Token: None 2023-09-22T21:22:25.368 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj3 2023-09-22T21:22:25.368 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Nbl5GDJUv9Z7v/FpGMp5Er02NeI= 2023-09-22T21:22:25.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Nbl5GDJUv9Z7v/FpGMp5Er02NeI=', 'Content-Length': '0'} 2023-09-22T21:22:25.370 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005787369a7e083889-00650e0591-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.414 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:25.415 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:25.415 DEBUG:boto:path=/ffzdqu-49/?prefix=obj3 2023-09-22T21:22:25.415 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj3 2023-09-22T21:22:25.415 DEBUG:boto:Method: GET 2023-09-22T21:22:25.415 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj3 2023-09-22T21:22:25.415 DEBUG:boto:Data: 2023-09-22T21:22:25.415 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.415 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:25.416 DEBUG:boto:Port: 8001 2023-09-22T21:22:25.416 DEBUG:boto:Params: {} 2023-09-22T21:22:25.416 DEBUG:boto:Token: None 2023-09-22T21:22:25.416 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/ 2023-09-22T21:22:25.416 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0= 2023-09-22T21:22:25.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0=', 'Content-Length': '0'} 2023-09-22T21:22:25.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048f9c56168354422-00650e0591-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.421 DEBUG:boto:b'ffzdqu-49obj31000falseobj32023-09-22T21:22:07.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:25.421 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:25.421 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:25.422 DEBUG:boto:path=/ffzdqu-49/?prefix=obj3 2023-09-22T21:22:25.422 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj3 2023-09-22T21:22:25.422 DEBUG:boto:Method: GET 2023-09-22T21:22:25.422 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj3 2023-09-22T21:22:25.422 DEBUG:boto:Data: 2023-09-22T21:22:25.422 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.422 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:25.422 DEBUG:boto:Port: 8000 2023-09-22T21:22:25.422 DEBUG:boto:Params: {} 2023-09-22T21:22:25.423 DEBUG:boto:Token: None 2023-09-22T21:22:25.423 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/ 2023-09-22T21:22:25.423 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0= 2023-09-22T21:22:25.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0=', 'Content-Length': '0'} 2023-09-22T21:22:25.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000766161b40d271bb3-00650e0591-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.474 DEBUG:boto:b'ffzdqu-49obj31000falseobj32023-09-22T21:22:07.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:25.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:22:25.475 DEBUG:boto:path=/ffzdqu-49/obj4 2023-09-22T21:22:25.475 DEBUG:boto:auth_path=/ffzdqu-49/obj4 2023-09-22T21:22:25.475 DEBUG:boto:path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.475 DEBUG:boto:auth_path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.475 DEBUG:boto:Method: GET 2023-09-22T21:22:25.476 DEBUG:boto:Path: /ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.476 DEBUG:boto:Data: 2023-09-22T21:22:25.476 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.476 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:25.476 DEBUG:boto:Port: 8001 2023-09-22T21:22:25.476 DEBUG:boto:Params: {} 2023-09-22T21:22:25.476 DEBUG:boto:Token: None 2023-09-22T21:22:25.476 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.477 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+b2LYwBZZ1E0laC0kj9FPfrWRPY= 2023-09-22T21:22:25.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+b2LYwBZZ1E0laC0kj9FPfrWRPY=', 'Content-Length': '0'} 2023-09-22T21:22:25.479 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063f3fff1c0823f13-00650e0591-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.480 DEBUG:boto:path=/ffzdqu-49/obj4 2023-09-22T21:22:25.480 DEBUG:boto:auth_path=/ffzdqu-49/obj4 2023-09-22T21:22:25.480 DEBUG:boto:path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.480 DEBUG:boto:auth_path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.480 DEBUG:boto:Method: GET 2023-09-22T21:22:25.480 DEBUG:boto:Path: /ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.481 DEBUG:boto:Data: 2023-09-22T21:22:25.481 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.481 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:25.481 DEBUG:boto:Port: 8000 2023-09-22T21:22:25.481 DEBUG:boto:Params: {} 2023-09-22T21:22:25.481 DEBUG:boto:Token: None 2023-09-22T21:22:25.481 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.481 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+b2LYwBZZ1E0laC0kj9FPfrWRPY= 2023-09-22T21:22:25.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+b2LYwBZZ1E0laC0kj9FPfrWRPY=', 'Content-Length': '0'} 2023-09-22T21:22:25.485 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051925619f65d8bc8-00650e0591-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.485 DEBUG:boto:path=/ffzdqu-49/obj4 2023-09-22T21:22:25.485 DEBUG:boto:auth_path=/ffzdqu-49/obj4 2023-09-22T21:22:25.485 DEBUG:boto:path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.486 DEBUG:boto:auth_path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.486 DEBUG:boto:Method: HEAD 2023-09-22T21:22:25.486 DEBUG:boto:Path: /ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.486 DEBUG:boto:Data: 2023-09-22T21:22:25.486 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.486 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:25.486 DEBUG:boto:Port: 8001 2023-09-22T21:22:25.486 DEBUG:boto:Params: {} 2023-09-22T21:22:25.487 DEBUG:boto:Token: None 2023-09-22T21:22:25.487 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.487 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jPcM00Ir6kfn43/d1omby7gf84g= 2023-09-22T21:22:25.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jPcM00Ir6kfn43/d1omby7gf84g=', 'Content-Length': '0'} 2023-09-22T21:22:25.490 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd6613e61bb3d5a8-00650e0591-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.490 DEBUG:boto:path=/ffzdqu-49/obj4 2023-09-22T21:22:25.490 DEBUG:boto:auth_path=/ffzdqu-49/obj4 2023-09-22T21:22:25.490 DEBUG:boto:path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.490 DEBUG:boto:auth_path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.491 DEBUG:boto:Method: HEAD 2023-09-22T21:22:25.491 DEBUG:boto:Path: /ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.491 DEBUG:boto:Data: 2023-09-22T21:22:25.491 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.491 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:25.491 DEBUG:boto:Port: 8000 2023-09-22T21:22:25.491 DEBUG:boto:Params: {} 2023-09-22T21:22:25.491 DEBUG:boto:Token: None 2023-09-22T21:22:25.491 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.492 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jPcM00Ir6kfn43/d1omby7gf84g= 2023-09-22T21:22:25.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jPcM00Ir6kfn43/d1omby7gf84g=', 'Content-Length': '0'} 2023-09-22T21:22:25.494 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dee407689db85478-00650e0591-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:22:25.495 DEBUG:boto:path=/ffzdqu-49/obj4 2023-09-22T21:22:25.495 DEBUG:boto:auth_path=/ffzdqu-49/obj4 2023-09-22T21:22:25.495 DEBUG:boto:path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.495 DEBUG:boto:auth_path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.495 DEBUG:boto:Method: GET 2023-09-22T21:22:25.495 DEBUG:boto:Path: /ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.495 DEBUG:boto:Data: 2023-09-22T21:22:25.495 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.496 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:25.496 DEBUG:boto:Port: 8001 2023-09-22T21:22:25.496 DEBUG:boto:Params: {} 2023-09-22T21:22:25.496 DEBUG:boto:Token: None 2023-09-22T21:22:25.496 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.496 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+b2LYwBZZ1E0laC0kj9FPfrWRPY= 2023-09-22T21:22:25.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+b2LYwBZZ1E0laC0kj9FPfrWRPY=', 'Content-Length': '0'} 2023-09-22T21:22:25.499 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6983a44add4ef6e-00650e0591-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.499 DEBUG:boto:path=/ffzdqu-49/obj4 2023-09-22T21:22:25.499 DEBUG:boto:auth_path=/ffzdqu-49/obj4 2023-09-22T21:22:25.499 DEBUG:boto:path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.499 DEBUG:boto:auth_path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.500 DEBUG:boto:Method: GET 2023-09-22T21:22:25.500 DEBUG:boto:Path: /ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.500 DEBUG:boto:Data: 2023-09-22T21:22:25.500 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.500 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:25.500 DEBUG:boto:Port: 8000 2023-09-22T21:22:25.500 DEBUG:boto:Params: {} 2023-09-22T21:22:25.500 DEBUG:boto:Token: None 2023-09-22T21:22:25.500 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj4?versionId=null 2023-09-22T21:22:25.501 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+b2LYwBZZ1E0laC0kj9FPfrWRPY= 2023-09-22T21:22:25.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+b2LYwBZZ1E0laC0kj9FPfrWRPY=', 'Content-Length': '0'} 2023-09-22T21:22:25.503 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061a1e56c91348340-00650e0591-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.546 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:25.546 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:25.547 DEBUG:boto:path=/ffzdqu-49/?prefix=obj4 2023-09-22T21:22:25.547 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj4 2023-09-22T21:22:25.547 DEBUG:boto:Method: GET 2023-09-22T21:22:25.547 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj4 2023-09-22T21:22:25.547 DEBUG:boto:Data: 2023-09-22T21:22:25.547 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.547 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:25.547 DEBUG:boto:Port: 8001 2023-09-22T21:22:25.547 DEBUG:boto:Params: {} 2023-09-22T21:22:25.548 DEBUG:boto:Token: None 2023-09-22T21:22:25.548 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/ 2023-09-22T21:22:25.548 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0= 2023-09-22T21:22:25.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0=', 'Content-Length': '0'} 2023-09-22T21:22:25.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c714998716910e0-00650e0591-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.554 DEBUG:boto:b'ffzdqu-49obj41000falseobj42023-09-22T21:22:07.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:25.554 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:25.554 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:25.555 DEBUG:boto:path=/ffzdqu-49/?prefix=obj4 2023-09-22T21:22:25.555 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj4 2023-09-22T21:22:25.555 DEBUG:boto:Method: GET 2023-09-22T21:22:25.555 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj4 2023-09-22T21:22:25.555 DEBUG:boto:Data: 2023-09-22T21:22:25.555 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.555 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:25.555 DEBUG:boto:Port: 8000 2023-09-22T21:22:25.555 DEBUG:boto:Params: {} 2023-09-22T21:22:25.556 DEBUG:boto:Token: None 2023-09-22T21:22:25.556 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/ 2023-09-22T21:22:25.556 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0= 2023-09-22T21:22:25.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0=', 'Content-Length': '0'} 2023-09-22T21:22:25.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c6312ae4734cbd6-00650e0591-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.602 DEBUG:boto:b'ffzdqu-49obj41000falseobj42023-09-22T21:22:07.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:25.603 DEBUG:boto:path=/ffzdqu-49/obj4 2023-09-22T21:22:25.603 DEBUG:boto:auth_path=/ffzdqu-49/obj4 2023-09-22T21:22:25.603 DEBUG:boto:Method: HEAD 2023-09-22T21:22:25.603 DEBUG:boto:Path: /ffzdqu-49/obj4 2023-09-22T21:22:25.603 DEBUG:boto:Data: 2023-09-22T21:22:25.603 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.604 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:25.604 DEBUG:boto:Port: 8001 2023-09-22T21:22:25.604 DEBUG:boto:Params: {} 2023-09-22T21:22:25.604 DEBUG:boto:Token: None 2023-09-22T21:22:25.604 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj4 2023-09-22T21:22:25.604 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ELQSy9w4yJ06h1lHicyqVTVaU5I= 2023-09-22T21:22:25.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ELQSy9w4yJ06h1lHicyqVTVaU5I=', 'Content-Length': '0'} 2023-09-22T21:22:25.607 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006989233faeb2772-00650e0591-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.607 DEBUG:boto:path=/ffzdqu-49/obj4 2023-09-22T21:22:25.607 DEBUG:boto:auth_path=/ffzdqu-49/obj4 2023-09-22T21:22:25.607 DEBUG:boto:Method: HEAD 2023-09-22T21:22:25.608 DEBUG:boto:Path: /ffzdqu-49/obj4 2023-09-22T21:22:25.608 DEBUG:boto:Data: 2023-09-22T21:22:25.608 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.608 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:25.608 DEBUG:boto:Port: 8000 2023-09-22T21:22:25.608 DEBUG:boto:Params: {} 2023-09-22T21:22:25.608 DEBUG:boto:Token: None 2023-09-22T21:22:25.608 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj4 2023-09-22T21:22:25.609 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ELQSy9w4yJ06h1lHicyqVTVaU5I= 2023-09-22T21:22:25.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ELQSy9w4yJ06h1lHicyqVTVaU5I=', 'Content-Length': '0'} 2023-09-22T21:22:25.611 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5e3e223109b050f-00650e0591-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:22:25.612 DEBUG:boto:path=/ffzdqu-49/obj4 2023-09-22T21:22:25.612 DEBUG:boto:auth_path=/ffzdqu-49/obj4 2023-09-22T21:22:25.612 DEBUG:boto:Method: GET 2023-09-22T21:22:25.612 DEBUG:boto:Path: /ffzdqu-49/obj4 2023-09-22T21:22:25.612 DEBUG:boto:Data: 2023-09-22T21:22:25.612 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.612 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:25.613 DEBUG:boto:Port: 8001 2023-09-22T21:22:25.613 DEBUG:boto:Params: {} 2023-09-22T21:22:25.613 DEBUG:boto:Token: None 2023-09-22T21:22:25.613 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj4 2023-09-22T21:22:25.613 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6mZgcgq/tV4Y/GN+h/+B/Cal1KU= 2023-09-22T21:22:25.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6mZgcgq/tV4Y/GN+h/+B/Cal1KU=', 'Content-Length': '0'} 2023-09-22T21:22:25.616 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c634fcc7e8aec59c-00650e0591-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.616 DEBUG:boto:path=/ffzdqu-49/obj4 2023-09-22T21:22:25.617 DEBUG:boto:auth_path=/ffzdqu-49/obj4 2023-09-22T21:22:25.617 DEBUG:boto:Method: GET 2023-09-22T21:22:25.617 DEBUG:boto:Path: /ffzdqu-49/obj4 2023-09-22T21:22:25.617 DEBUG:boto:Data: 2023-09-22T21:22:25.617 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.617 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:25.617 DEBUG:boto:Port: 8000 2023-09-22T21:22:25.617 DEBUG:boto:Params: {} 2023-09-22T21:22:25.617 DEBUG:boto:Token: None 2023-09-22T21:22:25.618 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/obj4 2023-09-22T21:22:25.618 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6mZgcgq/tV4Y/GN+h/+B/Cal1KU= 2023-09-22T21:22:25.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6mZgcgq/tV4Y/GN+h/+B/Cal1KU=', 'Content-Length': '0'} 2023-09-22T21:22:25.621 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ecde15e51d6a484b-00650e0591-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.666 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:25.667 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:25.667 DEBUG:boto:path=/ffzdqu-49/?prefix=obj4 2023-09-22T21:22:25.667 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj4 2023-09-22T21:22:25.667 DEBUG:boto:Method: GET 2023-09-22T21:22:25.667 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj4 2023-09-22T21:22:25.667 DEBUG:boto:Data: 2023-09-22T21:22:25.667 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.667 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:25.668 DEBUG:boto:Port: 8001 2023-09-22T21:22:25.668 DEBUG:boto:Params: {} 2023-09-22T21:22:25.668 DEBUG:boto:Token: None 2023-09-22T21:22:25.668 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/ 2023-09-22T21:22:25.668 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0= 2023-09-22T21:22:25.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0=', 'Content-Length': '0'} 2023-09-22T21:22:25.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025da6423325f853d-00650e0591-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.673 DEBUG:boto:b'ffzdqu-49obj41000falseobj42023-09-22T21:22:07.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:25.674 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:25.674 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:25.674 DEBUG:boto:path=/ffzdqu-49/?prefix=obj4 2023-09-22T21:22:25.674 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj4 2023-09-22T21:22:25.674 DEBUG:boto:Method: GET 2023-09-22T21:22:25.674 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj4 2023-09-22T21:22:25.674 DEBUG:boto:Data: 2023-09-22T21:22:25.675 DEBUG:boto:Headers: {} 2023-09-22T21:22:25.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:25.675 DEBUG:boto:Port: 8000 2023-09-22T21:22:25.675 DEBUG:boto:Params: {} 2023-09-22T21:22:25.675 DEBUG:boto:Token: None 2023-09-22T21:22:25.675 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:22:25 GMT /ffzdqu-49/ 2023-09-22T21:22:25.675 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0= 2023-09-22T21:22:25.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b9F4ENxn922zwLNXuKqXq348ee0=', 'Content-Length': '0'} 2023-09-22T21:22:25.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094bc73056d577432-00650e0591-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:25.722 DEBUG:boto:b'ffzdqu-49obj41000falseobj42023-09-22T21:22:07.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:22:25.723 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-49 zones={a2, a1} 2023-09-22T21:22:25.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'ffzdqu-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:22:25.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'ffzdqu-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:22:25.723 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket ffzdqu-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:22:25.769 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:25.769 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:25.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'ffzdqu-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:22:25.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'ffzdqu-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:22:25.857 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket ffzdqu-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:22:25.933 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:25.933 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:26.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:26.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:26.017 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:22:26.094 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:26.094 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:26.133 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:22:26.151 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695417746.003240_375.1","last_update":"2023-09-22T21:22:26.003240Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:22:26.152 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417746.003240_375.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:22:26.152 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:22:26.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:26.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:26.152 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:22:26.228 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:26.229 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:26.265 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:26.262+0000 7f153734ca40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:22:26.266 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:26.266+0000 7f153734ca40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:22:26.283 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417727.589558_369.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:07.589558Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:22:26Z"} 2023-09-22T21:22:26.284 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417727.589558_369.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:07.589558Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:22:26Z"} 2023-09-22T21:22:26.284 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417727.589558_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:22:07.589558Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:22:26.285 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417746.003240_375.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:22:26.285 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417727.589558_369.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:22:26.285 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417746.003240_375.1 target=1_1695417727.589558_369.1 2023-09-22T21:22:31.286 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:31.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:31.287 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:22:31.330 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:31.330 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:31.368 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:31.366+0000 7fdd443ffa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:22:31.369 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:31.370+0000 7fdd443ffa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:22:31.388 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417727.589558_369.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:07.589558Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:22:31Z"} 2023-09-22T21:22:31.388 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417727.589558_369.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:07.589558Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:22:31Z"} 2023-09-22T21:22:31.388 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417727.589558_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:22:07.589558Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:22:31.388 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417746.003240_375.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:22:31.389 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417727.589558_369.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:22:31.389 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417746.003240_375.1 target=1_1695417727.589558_369.1 2023-09-22T21:22:36.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:36.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:36.390 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:22:36.437 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:36.438 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:36.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:36.474+0000 7efcebb43a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:22:36.476 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:36.474+0000 7efcebb43a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:22:36.494 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417745.839927_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:25.839927Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:22:36Z"} 2023-09-22T21:22:36.494 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417745.839927_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:25.839927Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:22:36Z"} 2023-09-22T21:22:36.494 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417745.839927_370.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:22:25.839927Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:22:36.494 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417746.003240_375.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:22:36.494 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417745.839927_370.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:22:36.495 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417746.003240_375.1 target=1_1695417745.839927_370.1 2023-09-22T21:22:41.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:41.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:22:41.496 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:22:41.538 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:41.539 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:41.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:41.573+0000 7ffb5647aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:22:41.578 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:41.577+0000 7ffb5647aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:22:41.596 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417746.003240_375.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:26.003240Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:22:41Z"} 2023-09-22T21:22:41.596 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417746.003240_375.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:26.003240Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:22:41Z"} 2023-09-22T21:22:41.596 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417746.003240_375.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:22:26.003240Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:22:41.596 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417746.003240_375.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:22:41.597 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417746.003240_375.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:22:41.597 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:22:41.597 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:41.597 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:41.597 DEBUG:boto:Method: HEAD 2023-09-22T21:22:41.597 DEBUG:boto:Path: /ffzdqu-48/ 2023-09-22T21:22:41.597 DEBUG:boto:Data: 2023-09-22T21:22:41.597 DEBUG:boto:Headers: {} 2023-09-22T21:22:41.598 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:41.598 DEBUG:boto:Port: 8000 2023-09-22T21:22:41.598 DEBUG:boto:Params: {} 2023-09-22T21:22:41.598 DEBUG:boto:Token: None 2023-09-22T21:22:41.598 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:41 GMT /ffzdqu-48/ 2023-09-22T21:22:41.598 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BM6e7Mf5D8V8+1S3dkGpFctOAZE= 2023-09-22T21:22:41.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BM6e7Mf5D8V8+1S3dkGpFctOAZE=', 'Content-Length': '0'} 2023-09-22T21:22:41.601 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000608b1783153fc86-00650e05a1-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:41.601 DEBUG:boto:path=/ffzdqu-48/obj5 2023-09-22T21:22:41.601 DEBUG:boto:auth_path=/ffzdqu-48/obj5 2023-09-22T21:22:41.601 DEBUG:boto:Method: PUT 2023-09-22T21:22:41.601 DEBUG:boto:Path: /ffzdqu-48/obj5 2023-09-22T21:22:41.602 DEBUG:boto:Data: 2023-09-22T21:22:41.602 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:22:41.602 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:41.602 DEBUG:boto:Port: 8000 2023-09-22T21:22:41.602 DEBUG:boto:Params: {} 2023-09-22T21:22:41.602 DEBUG:boto:Token: None 2023-09-22T21:22:41.602 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:22:41 GMT /ffzdqu-48/obj5 2023-09-22T21:22:41.602 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FJZeycPc9Bx3wZcKPWxIGnn9OV0= 2023-09-22T21:22:41.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:22:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FJZeycPc9Bx3wZcKPWxIGnn9OV0='} 2023-09-22T21:22:41.608 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005acc5b09d1a9d29f-00650e05a1-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:41.609 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:41.609 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:41.609 DEBUG:boto:Method: HEAD 2023-09-22T21:22:41.609 DEBUG:boto:Path: /ffzdqu-48/ 2023-09-22T21:22:41.609 DEBUG:boto:Data: 2023-09-22T21:22:41.609 DEBUG:boto:Headers: {} 2023-09-22T21:22:41.609 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:41.609 DEBUG:boto:Port: 8000 2023-09-22T21:22:41.609 DEBUG:boto:Params: {} 2023-09-22T21:22:41.610 DEBUG:boto:Token: None 2023-09-22T21:22:41.610 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:41 GMT /ffzdqu-48/ 2023-09-22T21:22:41.610 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BM6e7Mf5D8V8+1S3dkGpFctOAZE= 2023-09-22T21:22:41.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BM6e7Mf5D8V8+1S3dkGpFctOAZE=', 'Content-Length': '0'} 2023-09-22T21:22:41.612 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002249f9ef4448328d-00650e05a1-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:41.612 DEBUG:boto:path=/ffzdqu-48/obj6 2023-09-22T21:22:41.612 DEBUG:boto:auth_path=/ffzdqu-48/obj6 2023-09-22T21:22:41.613 DEBUG:boto:Method: PUT 2023-09-22T21:22:41.613 DEBUG:boto:Path: /ffzdqu-48/obj6 2023-09-22T21:22:41.613 DEBUG:boto:Data: 2023-09-22T21:22:41.613 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:22:41.613 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:41.613 DEBUG:boto:Port: 8000 2023-09-22T21:22:41.613 DEBUG:boto:Params: {} 2023-09-22T21:22:41.613 DEBUG:boto:Token: None 2023-09-22T21:22:41.613 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:22:41 GMT /ffzdqu-48/obj6 2023-09-22T21:22:41.614 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vMWQexOcHqlF68YAUbORyGdCV9g= 2023-09-22T21:22:41.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:22:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vMWQexOcHqlF68YAUbORyGdCV9g='} 2023-09-22T21:22:41.619 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e66320ffd28e06ce-00650e05a1-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:41.619 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:41.619 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:41.620 DEBUG:boto:Method: HEAD 2023-09-22T21:22:41.620 DEBUG:boto:Path: /ffzdqu-48/ 2023-09-22T21:22:41.620 DEBUG:boto:Data: 2023-09-22T21:22:41.620 DEBUG:boto:Headers: {} 2023-09-22T21:22:41.620 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:41.620 DEBUG:boto:Port: 8000 2023-09-22T21:22:41.620 DEBUG:boto:Params: {} 2023-09-22T21:22:41.620 DEBUG:boto:Token: None 2023-09-22T21:22:41.621 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:41 GMT /ffzdqu-48/ 2023-09-22T21:22:41.621 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BM6e7Mf5D8V8+1S3dkGpFctOAZE= 2023-09-22T21:22:41.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BM6e7Mf5D8V8+1S3dkGpFctOAZE=', 'Content-Length': '0'} 2023-09-22T21:22:41.623 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a27a1e6575029cb-00650e05a1-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:41.623 DEBUG:boto:path=/ffzdqu-48/obj7 2023-09-22T21:22:41.623 DEBUG:boto:auth_path=/ffzdqu-48/obj7 2023-09-22T21:22:41.623 DEBUG:boto:Method: PUT 2023-09-22T21:22:41.623 DEBUG:boto:Path: /ffzdqu-48/obj7 2023-09-22T21:22:41.624 DEBUG:boto:Data: 2023-09-22T21:22:41.624 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:22:41.624 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:41.624 DEBUG:boto:Port: 8000 2023-09-22T21:22:41.624 DEBUG:boto:Params: {} 2023-09-22T21:22:41.624 DEBUG:boto:Token: None 2023-09-22T21:22:41.624 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:22:41 GMT /ffzdqu-48/obj7 2023-09-22T21:22:41.624 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NNOYnEY9n+vvY4bByeWBzGbd88M= 2023-09-22T21:22:41.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:22:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NNOYnEY9n+vvY4bByeWBzGbd88M='} 2023-09-22T21:22:41.667 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bec3a9a94463985c-00650e05a1-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:41.667 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:22:41.667 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:22:41.667 DEBUG:boto:Method: HEAD 2023-09-22T21:22:41.667 DEBUG:boto:Path: /ffzdqu-48/ 2023-09-22T21:22:41.667 DEBUG:boto:Data: 2023-09-22T21:22:41.667 DEBUG:boto:Headers: {} 2023-09-22T21:22:41.668 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:41.668 DEBUG:boto:Port: 8000 2023-09-22T21:22:41.668 DEBUG:boto:Params: {} 2023-09-22T21:22:41.668 DEBUG:boto:Token: None 2023-09-22T21:22:41.668 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:41 GMT /ffzdqu-48/ 2023-09-22T21:22:41.668 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BM6e7Mf5D8V8+1S3dkGpFctOAZE= 2023-09-22T21:22:41.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BM6e7Mf5D8V8+1S3dkGpFctOAZE=', 'Content-Length': '0'} 2023-09-22T21:22:41.672 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fff323bbe7956104-00650e05a1-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:41.672 DEBUG:boto:path=/ffzdqu-48/obj8 2023-09-22T21:22:41.673 DEBUG:boto:auth_path=/ffzdqu-48/obj8 2023-09-22T21:22:41.673 DEBUG:boto:Method: PUT 2023-09-22T21:22:41.673 DEBUG:boto:Path: /ffzdqu-48/obj8 2023-09-22T21:22:41.673 DEBUG:boto:Data: 2023-09-22T21:22:41.673 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:22:41.673 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:22:41.673 DEBUG:boto:Port: 8000 2023-09-22T21:22:41.673 DEBUG:boto:Params: {} 2023-09-22T21:22:41.674 DEBUG:boto:Token: None 2023-09-22T21:22:41.674 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:22:41 GMT /ffzdqu-48/obj8 2023-09-22T21:22:41.674 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5imoA6koy6xlROC4E7pEfX8NxYc= 2023-09-22T21:22:41.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:22:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5imoA6koy6xlROC4E7pEfX8NxYc='} 2023-09-22T21:22:41.686 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fef04b1c27e92360-00650e05a1-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:41.686 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:41.686 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:41.686 DEBUG:boto:Method: HEAD 2023-09-22T21:22:41.686 DEBUG:boto:Path: /ffzdqu-49/ 2023-09-22T21:22:41.687 DEBUG:boto:Data: 2023-09-22T21:22:41.687 DEBUG:boto:Headers: {} 2023-09-22T21:22:41.687 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:41.687 DEBUG:boto:Port: 8001 2023-09-22T21:22:41.687 DEBUG:boto:Params: {} 2023-09-22T21:22:41.687 DEBUG:boto:Token: None 2023-09-22T21:22:41.687 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:41 GMT /ffzdqu-49/ 2023-09-22T21:22:41.687 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zxWlCMpnSkEqhpy/UMJaXftXsR0= 2023-09-22T21:22:41.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zxWlCMpnSkEqhpy/UMJaXftXsR0=', 'Content-Length': '0'} 2023-09-22T21:22:41.691 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027332e25856932d8-00650e05a1-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:41.691 DEBUG:boto:path=/ffzdqu-49/obj5 2023-09-22T21:22:41.691 DEBUG:boto:auth_path=/ffzdqu-49/obj5 2023-09-22T21:22:41.691 DEBUG:boto:Method: PUT 2023-09-22T21:22:41.692 DEBUG:boto:Path: /ffzdqu-49/obj5 2023-09-22T21:22:41.692 DEBUG:boto:Data: 2023-09-22T21:22:41.692 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:22:41.692 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:41.692 DEBUG:boto:Port: 8001 2023-09-22T21:22:41.692 DEBUG:boto:Params: {} 2023-09-22T21:22:41.692 DEBUG:boto:Token: None 2023-09-22T21:22:41.692 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:22:41 GMT /ffzdqu-49/obj5 2023-09-22T21:22:41.692 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NlzW6wnsdJfMs4HDa4xAc9WUFEo= 2023-09-22T21:22:41.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:22:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NlzW6wnsdJfMs4HDa4xAc9WUFEo='} 2023-09-22T21:22:41.739 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000038bba5d76c1bf448-00650e05a1-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:41.739 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:41.739 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:41.739 DEBUG:boto:Method: HEAD 2023-09-22T21:22:41.739 DEBUG:boto:Path: /ffzdqu-49/ 2023-09-22T21:22:41.739 DEBUG:boto:Data: 2023-09-22T21:22:41.739 DEBUG:boto:Headers: {} 2023-09-22T21:22:41.740 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:41.740 DEBUG:boto:Port: 8001 2023-09-22T21:22:41.740 DEBUG:boto:Params: {} 2023-09-22T21:22:41.740 DEBUG:boto:Token: None 2023-09-22T21:22:41.740 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:41 GMT /ffzdqu-49/ 2023-09-22T21:22:41.740 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zxWlCMpnSkEqhpy/UMJaXftXsR0= 2023-09-22T21:22:41.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zxWlCMpnSkEqhpy/UMJaXftXsR0=', 'Content-Length': '0'} 2023-09-22T21:22:41.744 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de7d5ad775bbfd7f-00650e05a1-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:41.745 DEBUG:boto:path=/ffzdqu-49/obj6 2023-09-22T21:22:41.745 DEBUG:boto:auth_path=/ffzdqu-49/obj6 2023-09-22T21:22:41.745 DEBUG:boto:Method: PUT 2023-09-22T21:22:41.745 DEBUG:boto:Path: /ffzdqu-49/obj6 2023-09-22T21:22:41.745 DEBUG:boto:Data: 2023-09-22T21:22:41.745 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:22:41.745 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:41.745 DEBUG:boto:Port: 8001 2023-09-22T21:22:41.745 DEBUG:boto:Params: {} 2023-09-22T21:22:41.746 DEBUG:boto:Token: None 2023-09-22T21:22:41.746 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:22:41 GMT /ffzdqu-49/obj6 2023-09-22T21:22:41.746 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WoGbLj61oubZBLV1yMde2TzrGdE= 2023-09-22T21:22:41.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:22:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WoGbLj61oubZBLV1yMde2TzrGdE='} 2023-09-22T21:22:41.755 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000060cb345f768b6bfc-00650e05a1-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:41.755 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:41.755 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:41.755 DEBUG:boto:Method: HEAD 2023-09-22T21:22:41.755 DEBUG:boto:Path: /ffzdqu-49/ 2023-09-22T21:22:41.755 DEBUG:boto:Data: 2023-09-22T21:22:41.756 DEBUG:boto:Headers: {} 2023-09-22T21:22:41.756 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:41.756 DEBUG:boto:Port: 8001 2023-09-22T21:22:41.756 DEBUG:boto:Params: {} 2023-09-22T21:22:41.756 DEBUG:boto:Token: None 2023-09-22T21:22:41.756 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:41 GMT /ffzdqu-49/ 2023-09-22T21:22:41.756 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zxWlCMpnSkEqhpy/UMJaXftXsR0= 2023-09-22T21:22:41.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zxWlCMpnSkEqhpy/UMJaXftXsR0=', 'Content-Length': '0'} 2023-09-22T21:22:41.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002137a4019302d24d-00650e05a1-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:41.760 DEBUG:boto:path=/ffzdqu-49/obj7 2023-09-22T21:22:41.760 DEBUG:boto:auth_path=/ffzdqu-49/obj7 2023-09-22T21:22:41.761 DEBUG:boto:Method: PUT 2023-09-22T21:22:41.761 DEBUG:boto:Path: /ffzdqu-49/obj7 2023-09-22T21:22:41.761 DEBUG:boto:Data: 2023-09-22T21:22:41.761 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:22:41.761 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:41.761 DEBUG:boto:Port: 8001 2023-09-22T21:22:41.761 DEBUG:boto:Params: {} 2023-09-22T21:22:41.761 DEBUG:boto:Token: None 2023-09-22T21:22:41.761 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:22:41 GMT /ffzdqu-49/obj7 2023-09-22T21:22:41.762 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bN5mnopgIdbJHqbVe9tSss9NEzI= 2023-09-22T21:22:41.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:22:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bN5mnopgIdbJHqbVe9tSss9NEzI='} 2023-09-22T21:22:41.806 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c263d71f47f463d7-00650e05a1-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:41.807 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:22:41.807 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:22:41.807 DEBUG:boto:Method: HEAD 2023-09-22T21:22:41.807 DEBUG:boto:Path: /ffzdqu-49/ 2023-09-22T21:22:41.807 DEBUG:boto:Data: 2023-09-22T21:22:41.807 DEBUG:boto:Headers: {} 2023-09-22T21:22:41.807 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:41.808 DEBUG:boto:Port: 8001 2023-09-22T21:22:41.808 DEBUG:boto:Params: {} 2023-09-22T21:22:41.808 DEBUG:boto:Token: None 2023-09-22T21:22:41.808 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:22:41 GMT /ffzdqu-49/ 2023-09-22T21:22:41.808 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zxWlCMpnSkEqhpy/UMJaXftXsR0= 2023-09-22T21:22:41.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:22:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zxWlCMpnSkEqhpy/UMJaXftXsR0=', 'Content-Length': '0'} 2023-09-22T21:22:41.813 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4f1eb8819787ab2-00650e05a1-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:41.813 DEBUG:boto:path=/ffzdqu-49/obj8 2023-09-22T21:22:41.813 DEBUG:boto:auth_path=/ffzdqu-49/obj8 2023-09-22T21:22:41.813 DEBUG:boto:Method: PUT 2023-09-22T21:22:41.814 DEBUG:boto:Path: /ffzdqu-49/obj8 2023-09-22T21:22:41.814 DEBUG:boto:Data: 2023-09-22T21:22:41.814 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-22T21:22:41.814 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:22:41.814 DEBUG:boto:Port: 8001 2023-09-22T21:22:41.814 DEBUG:boto:Params: {} 2023-09-22T21:22:41.814 DEBUG:boto:Token: None 2023-09-22T21:22:41.814 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 22 Sep 2023 21:22:41 GMT /ffzdqu-49/obj8 2023-09-22T21:22:41.814 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OHTs7pHzi+r2MpkDElFvIY8EpEo= 2023-09-22T21:22:41.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:22:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OHTs7pHzi+r2MpkDElFvIY8EpEo='} 2023-09-22T21:22:41.823 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001388645af0c7e8fd-00650e05a1-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:22:41.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'ffzdqu-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:22:41.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'ffzdqu-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:22:41.824 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket ffzdqu-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:22:41.868 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:41.869 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:41.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'ffzdqu-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:22:41.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'ffzdqu-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:22:41.957 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket ffzdqu-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:22:42.035 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:42.035 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:42.121 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-48 2023-09-22T21:22:42.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:22:42.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:22:42.122 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:22:42.206 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:42.207 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:42.256 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:42.253+0000 7f02c83b5a40 1 waiting to reach incremental sync.. 2023-09-22T21:22:47.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:47.257+0000 7f02c83b5a40 1 bucket sync caught up with source: 2023-09-22T21:22:47.260 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000004.28.1, 00000000002.29.1, 00000000004.27.1, 00000000002.20.1, 00000000004.28.1, 00000000002.23.1, 00000000002.41.1, 00000000004.37.1, 00000000002.17.1, 00000000002.48.1, 00000000002.32.1] 2023-09-22T21:22:47.260 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000004.28.1, 00000000002.29.1, 00000000004.27.1, 00000000002.20.1, 00000000004.28.1, 00000000002.23.1, 00000000002.41.1, 00000000004.37.1, 00000000002.17.1, 00000000002.48.1, 00000000002.32.1] 2023-09-22T21:22:47.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:47.257+0000 7f02c83b5a40 0 bucket checkpoint complete 2023-09-22T21:22:47.289 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-48 2023-09-22T21:22:47.290 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:22:47.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:22:47.290 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:22:47.338 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:22:47.338 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:22:47.381 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:47.377+0000 7fea93b9ca40 1 waiting to reach incremental sync.. 2023-09-22T21:22:52.382 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:52.385+0000 7fea93b9ca40 1 waiting to reach incremental sync.. 2023-09-22T21:22:57.383 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:22:57.385+0000 7fea93b9ca40 1 waiting to reach incremental sync.. 2023-09-22T21:23:02.384 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:23:02.385+0000 7fea93b9ca40 1 waiting to reach incremental sync.. 2023-09-22T21:23:07.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:23:07.389+0000 7fea93b9ca40 1 bucket sync caught up with source: 2023-09-22T21:23:07.388 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000004.139.1, 00000000002.140.1, 00000000004.184.1, 00000000002.220.1, 00000000004.185.1, 00000000002.209.1, 00000000002.238.1, 00000000004.154.1, 00000000002.125.1, 00000000002.211.1, 00000000002.224.1] 2023-09-22T21:23:07.388 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.131.1, 00000000001.132.1, 00000000002.174.1, 00000000001.218.1, 00000000002.175.1, 00000000001.207.1, 00000000001.231.1, 00000000002.149.1, 00000000001.118.1, 00000000001.204.1, 00000000001.217.1] 2023-09-22T21:23:07.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:23:07.389+0000 7fea93b9ca40 0 bucket checkpoint complete 2023-09-22T21:23:07.422 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-48 zones={a2, a1} 2023-09-22T21:23:07.422 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:07.422 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:07.423 DEBUG:boto:Method: HEAD 2023-09-22T21:23:07.423 DEBUG:boto:Path: /ffzdqu-48/ 2023-09-22T21:23:07.423 DEBUG:boto:Data: 2023-09-22T21:23:07.423 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.423 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:07.423 DEBUG:boto:Port: 8001 2023-09-22T21:23:07.423 DEBUG:boto:Params: {} 2023-09-22T21:23:07.423 DEBUG:boto:Token: None 2023-09-22T21:23:07.424 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/ 2023-09-22T21:23:07.424 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:B1+dtUtBE7wexTVR4HsJ9QcIT3o= 2023-09-22T21:23:07.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:B1+dtUtBE7wexTVR4HsJ9QcIT3o=', 'Content-Length': '0'} 2023-09-22T21:23:07.427 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbc8fcb8a16e8649-00650e05bb-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.427 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:07.427 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:07.427 DEBUG:boto:Method: HEAD 2023-09-22T21:23:07.428 DEBUG:boto:Path: /ffzdqu-48/ 2023-09-22T21:23:07.428 DEBUG:boto:Data: 2023-09-22T21:23:07.428 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.428 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:07.428 DEBUG:boto:Port: 8000 2023-09-22T21:23:07.428 DEBUG:boto:Params: {} 2023-09-22T21:23:07.428 DEBUG:boto:Token: None 2023-09-22T21:23:07.428 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/ 2023-09-22T21:23:07.428 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:B1+dtUtBE7wexTVR4HsJ9QcIT3o= 2023-09-22T21:23:07.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:B1+dtUtBE7wexTVR4HsJ9QcIT3o=', 'Content-Length': '0'} 2023-09-22T21:23:07.432 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a046ede1a03dd946-00650e05bb-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.432 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:23:07.432 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:07.432 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:07.432 DEBUG:boto:path=/ffzdqu-48/?versions&max-keys=999 2023-09-22T21:23:07.433 DEBUG:boto:auth_path=/ffzdqu-48/?versions&max-keys=999 2023-09-22T21:23:07.433 DEBUG:boto:Method: GET 2023-09-22T21:23:07.433 DEBUG:boto:Path: /ffzdqu-48/?versions&max-keys=999 2023-09-22T21:23:07.433 DEBUG:boto:Data: 2023-09-22T21:23:07.433 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.433 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:07.433 DEBUG:boto:Port: 8001 2023-09-22T21:23:07.433 DEBUG:boto:Params: {} 2023-09-22T21:23:07.433 DEBUG:boto:Token: None 2023-09-22T21:23:07.434 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/?versions 2023-09-22T21:23:07.434 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:W7njZXSSdL6yyfcLOz7HTv9Lle4= 2023-09-22T21:23:07.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:W7njZXSSdL6yyfcLOz7HTv9Lle4=', 'Content-Length': '0'} 2023-09-22T21:23:07.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000805fa8680ed63548-00650e05bb-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.438 DEBUG:boto:b'ffzdqu-48999falseobj1nulltrue2023-09-22T21:22:07.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:22:07.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:22:07.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:22:07.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-22T21:22:41.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-22T21:22:41.619Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-22T21:22:41.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-22T21:22:41.680Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:07.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-22T21:23:07.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-22T21:23:07.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-22T21:23:07.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-22T21:23:07.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-09-22T21:23:07.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-09-22T21:23:07.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-09-22T21:23:07.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-09-22T21:23:07.439 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:23:07.439 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:07.440 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:07.440 DEBUG:boto:path=/ffzdqu-48/?versions&max-keys=999 2023-09-22T21:23:07.440 DEBUG:boto:auth_path=/ffzdqu-48/?versions&max-keys=999 2023-09-22T21:23:07.440 DEBUG:boto:Method: GET 2023-09-22T21:23:07.440 DEBUG:boto:Path: /ffzdqu-48/?versions&max-keys=999 2023-09-22T21:23:07.440 DEBUG:boto:Data: 2023-09-22T21:23:07.440 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.440 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:07.440 DEBUG:boto:Port: 8000 2023-09-22T21:23:07.440 DEBUG:boto:Params: {} 2023-09-22T21:23:07.441 DEBUG:boto:Token: None 2023-09-22T21:23:07.441 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/?versions 2023-09-22T21:23:07.441 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:W7njZXSSdL6yyfcLOz7HTv9Lle4= 2023-09-22T21:23:07.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:W7njZXSSdL6yyfcLOz7HTv9Lle4=', 'Content-Length': '0'} 2023-09-22T21:23:07.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b8a81831651817a-00650e05bb-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.445 DEBUG:boto:b'ffzdqu-48999falseobj1nulltrue2023-09-22T21:22:07.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:22:07.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:22:07.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:22:07.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-22T21:22:41.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-22T21:22:41.619Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-22T21:22:41.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-22T21:22:41.680Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:07.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-22T21:23:07.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-22T21:23:07.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-22T21:23:07.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-22T21:23:07.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-09-22T21:23:07.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-09-22T21:23:07.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-09-22T21:23:07.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-09-22T21:23:07.446 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:07.446 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:07.446 DEBUG:boto:path=/ffzdqu-48/?versions&max-keys=999 2023-09-22T21:23:07.447 DEBUG:boto:auth_path=/ffzdqu-48/?versions&max-keys=999 2023-09-22T21:23:07.447 DEBUG:boto:Method: GET 2023-09-22T21:23:07.447 DEBUG:boto:Path: /ffzdqu-48/?versions&max-keys=999 2023-09-22T21:23:07.447 DEBUG:boto:Data: 2023-09-22T21:23:07.447 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.447 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:07.447 DEBUG:boto:Port: 8001 2023-09-22T21:23:07.447 DEBUG:boto:Params: {} 2023-09-22T21:23:07.447 DEBUG:boto:Token: None 2023-09-22T21:23:07.447 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/?versions 2023-09-22T21:23:07.448 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:W7njZXSSdL6yyfcLOz7HTv9Lle4= 2023-09-22T21:23:07.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:W7njZXSSdL6yyfcLOz7HTv9Lle4=', 'Content-Length': '0'} 2023-09-22T21:23:07.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003814d503df075bfa-00650e05bb-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.452 DEBUG:boto:b'ffzdqu-48999falseobj1nulltrue2023-09-22T21:22:07.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:22:07.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:22:07.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:22:07.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-22T21:22:41.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-22T21:22:41.619Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-22T21:22:41.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-22T21:22:41.680Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:07.453 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:07.453 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:07.453 DEBUG:boto:path=/ffzdqu-48/?versions&max-keys=999 2023-09-22T21:23:07.453 DEBUG:boto:auth_path=/ffzdqu-48/?versions&max-keys=999 2023-09-22T21:23:07.453 DEBUG:boto:Method: GET 2023-09-22T21:23:07.453 DEBUG:boto:Path: /ffzdqu-48/?versions&max-keys=999 2023-09-22T21:23:07.453 DEBUG:boto:Data: 2023-09-22T21:23:07.453 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.454 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:07.454 DEBUG:boto:Port: 8000 2023-09-22T21:23:07.454 DEBUG:boto:Params: {} 2023-09-22T21:23:07.454 DEBUG:boto:Token: None 2023-09-22T21:23:07.454 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/?versions 2023-09-22T21:23:07.454 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:W7njZXSSdL6yyfcLOz7HTv9Lle4= 2023-09-22T21:23:07.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:W7njZXSSdL6yyfcLOz7HTv9Lle4=', 'Content-Length': '0'} 2023-09-22T21:23:07.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d758754083a7054-00650e05bb-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.459 DEBUG:boto:b'ffzdqu-48999falseobj1nulltrue2023-09-22T21:22:07.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:22:07.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:22:07.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:22:07.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-22T21:22:41.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-22T21:22:41.619Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-22T21:22:41.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-22T21:22:41.680Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:07.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:23:07.459 DEBUG:boto:path=/ffzdqu-48/obj1 2023-09-22T21:23:07.460 DEBUG:boto:auth_path=/ffzdqu-48/obj1 2023-09-22T21:23:07.460 DEBUG:boto:path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.460 DEBUG:boto:auth_path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.460 DEBUG:boto:Method: GET 2023-09-22T21:23:07.460 DEBUG:boto:Path: /ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.460 DEBUG:boto:Data: 2023-09-22T21:23:07.460 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.460 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:07.460 DEBUG:boto:Port: 8001 2023-09-22T21:23:07.460 DEBUG:boto:Params: {} 2023-09-22T21:23:07.461 DEBUG:boto:Token: None 2023-09-22T21:23:07.461 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.461 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vBnQmx+69zeN9H1LiWbhnhll9K8= 2023-09-22T21:23:07.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vBnQmx+69zeN9H1LiWbhnhll9K8=', 'Content-Length': '0'} 2023-09-22T21:23:07.463 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009670d6df3a97717-00650e05bb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.506 DEBUG:boto:path=/ffzdqu-48/obj1 2023-09-22T21:23:07.507 DEBUG:boto:auth_path=/ffzdqu-48/obj1 2023-09-22T21:23:07.507 DEBUG:boto:path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.507 DEBUG:boto:auth_path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.507 DEBUG:boto:Method: GET 2023-09-22T21:23:07.507 DEBUG:boto:Path: /ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.507 DEBUG:boto:Data: 2023-09-22T21:23:07.507 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.507 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:07.507 DEBUG:boto:Port: 8000 2023-09-22T21:23:07.508 DEBUG:boto:Params: {} 2023-09-22T21:23:07.508 DEBUG:boto:Token: None 2023-09-22T21:23:07.508 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.508 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vBnQmx+69zeN9H1LiWbhnhll9K8= 2023-09-22T21:23:07.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vBnQmx+69zeN9H1LiWbhnhll9K8=', 'Content-Length': '0'} 2023-09-22T21:23:07.511 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033f301795e005234-00650e05bb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.511 DEBUG:boto:path=/ffzdqu-48/obj1 2023-09-22T21:23:07.511 DEBUG:boto:auth_path=/ffzdqu-48/obj1 2023-09-22T21:23:07.511 DEBUG:boto:path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.511 DEBUG:boto:auth_path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.512 DEBUG:boto:Method: HEAD 2023-09-22T21:23:07.512 DEBUG:boto:Path: /ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.512 DEBUG:boto:Data: 2023-09-22T21:23:07.512 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.512 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:07.512 DEBUG:boto:Port: 8001 2023-09-22T21:23:07.512 DEBUG:boto:Params: {} 2023-09-22T21:23:07.512 DEBUG:boto:Token: None 2023-09-22T21:23:07.512 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.513 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1wz78D57tSKoSbbm/cWNQAAcj48= 2023-09-22T21:23:07.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1wz78D57tSKoSbbm/cWNQAAcj48=', 'Content-Length': '0'} 2023-09-22T21:23:07.515 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d246c79100984b17-00650e05bb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.516 DEBUG:boto:path=/ffzdqu-48/obj1 2023-09-22T21:23:07.516 DEBUG:boto:auth_path=/ffzdqu-48/obj1 2023-09-22T21:23:07.516 DEBUG:boto:path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.516 DEBUG:boto:auth_path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.516 DEBUG:boto:Method: HEAD 2023-09-22T21:23:07.516 DEBUG:boto:Path: /ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.516 DEBUG:boto:Data: 2023-09-22T21:23:07.517 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.517 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:07.517 DEBUG:boto:Port: 8000 2023-09-22T21:23:07.517 DEBUG:boto:Params: {} 2023-09-22T21:23:07.517 DEBUG:boto:Token: None 2023-09-22T21:23:07.517 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.517 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1wz78D57tSKoSbbm/cWNQAAcj48= 2023-09-22T21:23:07.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1wz78D57tSKoSbbm/cWNQAAcj48=', 'Content-Length': '0'} 2023-09-22T21:23:07.520 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dcbc53aa2d43e1b1-00650e05bb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:23:07.520 DEBUG:boto:path=/ffzdqu-48/obj1 2023-09-22T21:23:07.520 DEBUG:boto:auth_path=/ffzdqu-48/obj1 2023-09-22T21:23:07.521 DEBUG:boto:path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.521 DEBUG:boto:auth_path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.521 DEBUG:boto:Method: GET 2023-09-22T21:23:07.521 DEBUG:boto:Path: /ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.521 DEBUG:boto:Data: 2023-09-22T21:23:07.521 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.521 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:07.521 DEBUG:boto:Port: 8001 2023-09-22T21:23:07.521 DEBUG:boto:Params: {} 2023-09-22T21:23:07.521 DEBUG:boto:Token: None 2023-09-22T21:23:07.522 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.522 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vBnQmx+69zeN9H1LiWbhnhll9K8= 2023-09-22T21:23:07.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vBnQmx+69zeN9H1LiWbhnhll9K8=', 'Content-Length': '0'} 2023-09-22T21:23:07.524 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000faf3a07c358fed09-00650e05bb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.525 DEBUG:boto:path=/ffzdqu-48/obj1 2023-09-22T21:23:07.525 DEBUG:boto:auth_path=/ffzdqu-48/obj1 2023-09-22T21:23:07.525 DEBUG:boto:path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.525 DEBUG:boto:auth_path=/ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.525 DEBUG:boto:Method: GET 2023-09-22T21:23:07.525 DEBUG:boto:Path: /ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.525 DEBUG:boto:Data: 2023-09-22T21:23:07.525 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.526 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:07.526 DEBUG:boto:Port: 8000 2023-09-22T21:23:07.526 DEBUG:boto:Params: {} 2023-09-22T21:23:07.526 DEBUG:boto:Token: None 2023-09-22T21:23:07.526 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj1?versionId=null 2023-09-22T21:23:07.526 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vBnQmx+69zeN9H1LiWbhnhll9K8= 2023-09-22T21:23:07.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vBnQmx+69zeN9H1LiWbhnhll9K8=', 'Content-Length': '0'} 2023-09-22T21:23:07.529 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003828029bb4e751fe-00650e05bb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.529 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:07.530 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:07.530 DEBUG:boto:path=/ffzdqu-48/?prefix=obj1 2023-09-22T21:23:07.530 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj1 2023-09-22T21:23:07.530 DEBUG:boto:Method: GET 2023-09-22T21:23:07.530 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj1 2023-09-22T21:23:07.530 DEBUG:boto:Data: 2023-09-22T21:23:07.530 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.530 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:07.530 DEBUG:boto:Port: 8001 2023-09-22T21:23:07.530 DEBUG:boto:Params: {} 2023-09-22T21:23:07.531 DEBUG:boto:Token: None 2023-09-22T21:23:07.531 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/ 2023-09-22T21:23:07.531 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bTRUUAryCSnjcOamVsir2LXHCWw= 2023-09-22T21:23:07.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bTRUUAryCSnjcOamVsir2LXHCWw=', 'Content-Length': '0'} 2023-09-22T21:23:07.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a2a5810bb4f8aff-00650e05bb-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.535 DEBUG:boto:b'ffzdqu-48obj11000falseobj12023-09-22T21:22:07.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:07.536 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:07.536 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:07.536 DEBUG:boto:path=/ffzdqu-48/?prefix=obj1 2023-09-22T21:23:07.536 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj1 2023-09-22T21:23:07.536 DEBUG:boto:Method: GET 2023-09-22T21:23:07.536 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj1 2023-09-22T21:23:07.536 DEBUG:boto:Data: 2023-09-22T21:23:07.536 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.537 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:07.537 DEBUG:boto:Port: 8000 2023-09-22T21:23:07.537 DEBUG:boto:Params: {} 2023-09-22T21:23:07.537 DEBUG:boto:Token: None 2023-09-22T21:23:07.537 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/ 2023-09-22T21:23:07.537 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bTRUUAryCSnjcOamVsir2LXHCWw= 2023-09-22T21:23:07.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bTRUUAryCSnjcOamVsir2LXHCWw=', 'Content-Length': '0'} 2023-09-22T21:23:07.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d51375328cb2f5d-00650e05bb-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.542 DEBUG:boto:b'ffzdqu-48obj11000falseobj12023-09-22T21:22:07.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:07.543 DEBUG:boto:path=/ffzdqu-48/obj1 2023-09-22T21:23:07.543 DEBUG:boto:auth_path=/ffzdqu-48/obj1 2023-09-22T21:23:07.543 DEBUG:boto:Method: HEAD 2023-09-22T21:23:07.543 DEBUG:boto:Path: /ffzdqu-48/obj1 2023-09-22T21:23:07.543 DEBUG:boto:Data: 2023-09-22T21:23:07.543 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.543 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:07.543 DEBUG:boto:Port: 8001 2023-09-22T21:23:07.544 DEBUG:boto:Params: {} 2023-09-22T21:23:07.544 DEBUG:boto:Token: None 2023-09-22T21:23:07.544 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj1 2023-09-22T21:23:07.544 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PlS7vG9XR5VyWI7KIA40MBGvIkE= 2023-09-22T21:23:07.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PlS7vG9XR5VyWI7KIA40MBGvIkE=', 'Content-Length': '0'} 2023-09-22T21:23:07.547 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0e677b391e23707-00650e05bb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.547 DEBUG:boto:path=/ffzdqu-48/obj1 2023-09-22T21:23:07.547 DEBUG:boto:auth_path=/ffzdqu-48/obj1 2023-09-22T21:23:07.547 DEBUG:boto:Method: HEAD 2023-09-22T21:23:07.547 DEBUG:boto:Path: /ffzdqu-48/obj1 2023-09-22T21:23:07.547 DEBUG:boto:Data: 2023-09-22T21:23:07.547 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.548 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:07.548 DEBUG:boto:Port: 8000 2023-09-22T21:23:07.548 DEBUG:boto:Params: {} 2023-09-22T21:23:07.548 DEBUG:boto:Token: None 2023-09-22T21:23:07.548 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj1 2023-09-22T21:23:07.548 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PlS7vG9XR5VyWI7KIA40MBGvIkE= 2023-09-22T21:23:07.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PlS7vG9XR5VyWI7KIA40MBGvIkE=', 'Content-Length': '0'} 2023-09-22T21:23:07.551 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007677802be523527a-00650e05bb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:23:07.551 DEBUG:boto:path=/ffzdqu-48/obj1 2023-09-22T21:23:07.551 DEBUG:boto:auth_path=/ffzdqu-48/obj1 2023-09-22T21:23:07.552 DEBUG:boto:Method: GET 2023-09-22T21:23:07.552 DEBUG:boto:Path: /ffzdqu-48/obj1 2023-09-22T21:23:07.552 DEBUG:boto:Data: 2023-09-22T21:23:07.552 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.552 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:07.552 DEBUG:boto:Port: 8001 2023-09-22T21:23:07.552 DEBUG:boto:Params: {} 2023-09-22T21:23:07.552 DEBUG:boto:Token: None 2023-09-22T21:23:07.552 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj1 2023-09-22T21:23:07.553 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Rwrap4ZVIwmjAfR7uLtRYqKPwIA= 2023-09-22T21:23:07.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rwrap4ZVIwmjAfR7uLtRYqKPwIA=', 'Content-Length': '0'} 2023-09-22T21:23:07.555 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee7cd64851c0d138-00650e05bb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.598 DEBUG:boto:path=/ffzdqu-48/obj1 2023-09-22T21:23:07.598 DEBUG:boto:auth_path=/ffzdqu-48/obj1 2023-09-22T21:23:07.599 DEBUG:boto:Method: GET 2023-09-22T21:23:07.599 DEBUG:boto:Path: /ffzdqu-48/obj1 2023-09-22T21:23:07.599 DEBUG:boto:Data: 2023-09-22T21:23:07.599 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:07.599 DEBUG:boto:Port: 8000 2023-09-22T21:23:07.599 DEBUG:boto:Params: {} 2023-09-22T21:23:07.599 DEBUG:boto:Token: None 2023-09-22T21:23:07.600 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj1 2023-09-22T21:23:07.600 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Rwrap4ZVIwmjAfR7uLtRYqKPwIA= 2023-09-22T21:23:07.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rwrap4ZVIwmjAfR7uLtRYqKPwIA=', 'Content-Length': '0'} 2023-09-22T21:23:07.603 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055327ce8f684b456-00650e05bb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.603 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:07.603 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:07.603 DEBUG:boto:path=/ffzdqu-48/?prefix=obj1 2023-09-22T21:23:07.604 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj1 2023-09-22T21:23:07.604 DEBUG:boto:Method: GET 2023-09-22T21:23:07.604 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj1 2023-09-22T21:23:07.604 DEBUG:boto:Data: 2023-09-22T21:23:07.604 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.604 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:07.604 DEBUG:boto:Port: 8001 2023-09-22T21:23:07.604 DEBUG:boto:Params: {} 2023-09-22T21:23:07.604 DEBUG:boto:Token: None 2023-09-22T21:23:07.605 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/ 2023-09-22T21:23:07.605 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bTRUUAryCSnjcOamVsir2LXHCWw= 2023-09-22T21:23:07.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bTRUUAryCSnjcOamVsir2LXHCWw=', 'Content-Length': '0'} 2023-09-22T21:23:07.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fac77cbe9296db3-00650e05bb-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.609 DEBUG:boto:b'ffzdqu-48obj11000falseobj12023-09-22T21:22:07.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:07.610 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:07.610 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:07.610 DEBUG:boto:path=/ffzdqu-48/?prefix=obj1 2023-09-22T21:23:07.610 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj1 2023-09-22T21:23:07.610 DEBUG:boto:Method: GET 2023-09-22T21:23:07.610 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj1 2023-09-22T21:23:07.610 DEBUG:boto:Data: 2023-09-22T21:23:07.610 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:07.611 DEBUG:boto:Port: 8000 2023-09-22T21:23:07.611 DEBUG:boto:Params: {} 2023-09-22T21:23:07.611 DEBUG:boto:Token: None 2023-09-22T21:23:07.611 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/ 2023-09-22T21:23:07.611 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bTRUUAryCSnjcOamVsir2LXHCWw= 2023-09-22T21:23:07.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bTRUUAryCSnjcOamVsir2LXHCWw=', 'Content-Length': '0'} 2023-09-22T21:23:07.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fbea5c31bf1125a-00650e05bb-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.616 DEBUG:boto:b'ffzdqu-48obj11000falseobj12023-09-22T21:22:07.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:07.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:23:07.616 DEBUG:boto:path=/ffzdqu-48/obj2 2023-09-22T21:23:07.616 DEBUG:boto:auth_path=/ffzdqu-48/obj2 2023-09-22T21:23:07.617 DEBUG:boto:path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.617 DEBUG:boto:auth_path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.617 DEBUG:boto:Method: GET 2023-09-22T21:23:07.617 DEBUG:boto:Path: /ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.617 DEBUG:boto:Data: 2023-09-22T21:23:07.617 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.617 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:07.617 DEBUG:boto:Port: 8001 2023-09-22T21:23:07.617 DEBUG:boto:Params: {} 2023-09-22T21:23:07.617 DEBUG:boto:Token: None 2023-09-22T21:23:07.618 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.618 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h3gKuZVbSlEnG00Pp6H8306BvYo= 2023-09-22T21:23:07.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h3gKuZVbSlEnG00Pp6H8306BvYo=', 'Content-Length': '0'} 2023-09-22T21:23:07.621 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033c4d4c3afc64368-00650e05bb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.621 DEBUG:boto:path=/ffzdqu-48/obj2 2023-09-22T21:23:07.621 DEBUG:boto:auth_path=/ffzdqu-48/obj2 2023-09-22T21:23:07.621 DEBUG:boto:path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.621 DEBUG:boto:auth_path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.622 DEBUG:boto:Method: GET 2023-09-22T21:23:07.622 DEBUG:boto:Path: /ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.622 DEBUG:boto:Data: 2023-09-22T21:23:07.622 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.622 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:07.622 DEBUG:boto:Port: 8000 2023-09-22T21:23:07.622 DEBUG:boto:Params: {} 2023-09-22T21:23:07.622 DEBUG:boto:Token: None 2023-09-22T21:23:07.622 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.623 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h3gKuZVbSlEnG00Pp6H8306BvYo= 2023-09-22T21:23:07.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h3gKuZVbSlEnG00Pp6H8306BvYo=', 'Content-Length': '0'} 2023-09-22T21:23:07.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f949ef5ee4e999f-00650e05bb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.626 DEBUG:boto:path=/ffzdqu-48/obj2 2023-09-22T21:23:07.626 DEBUG:boto:auth_path=/ffzdqu-48/obj2 2023-09-22T21:23:07.626 DEBUG:boto:path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.626 DEBUG:boto:auth_path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.626 DEBUG:boto:Method: HEAD 2023-09-22T21:23:07.626 DEBUG:boto:Path: /ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.626 DEBUG:boto:Data: 2023-09-22T21:23:07.627 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.627 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:07.627 DEBUG:boto:Port: 8001 2023-09-22T21:23:07.627 DEBUG:boto:Params: {} 2023-09-22T21:23:07.627 DEBUG:boto:Token: None 2023-09-22T21:23:07.627 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.627 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cxAhwDge9zE9TzNTRuUGhjWazcw= 2023-09-22T21:23:07.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cxAhwDge9zE9TzNTRuUGhjWazcw=', 'Content-Length': '0'} 2023-09-22T21:23:07.630 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8a2452dc85101d0-00650e05bb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.630 DEBUG:boto:path=/ffzdqu-48/obj2 2023-09-22T21:23:07.630 DEBUG:boto:auth_path=/ffzdqu-48/obj2 2023-09-22T21:23:07.631 DEBUG:boto:path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.631 DEBUG:boto:auth_path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.631 DEBUG:boto:Method: HEAD 2023-09-22T21:23:07.631 DEBUG:boto:Path: /ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.631 DEBUG:boto:Data: 2023-09-22T21:23:07.631 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.631 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:07.631 DEBUG:boto:Port: 8000 2023-09-22T21:23:07.631 DEBUG:boto:Params: {} 2023-09-22T21:23:07.631 DEBUG:boto:Token: None 2023-09-22T21:23:07.632 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.632 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cxAhwDge9zE9TzNTRuUGhjWazcw= 2023-09-22T21:23:07.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cxAhwDge9zE9TzNTRuUGhjWazcw=', 'Content-Length': '0'} 2023-09-22T21:23:07.634 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d79d307c43432724-00650e05bb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:23:07.635 DEBUG:boto:path=/ffzdqu-48/obj2 2023-09-22T21:23:07.635 DEBUG:boto:auth_path=/ffzdqu-48/obj2 2023-09-22T21:23:07.635 DEBUG:boto:path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.635 DEBUG:boto:auth_path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.635 DEBUG:boto:Method: GET 2023-09-22T21:23:07.635 DEBUG:boto:Path: /ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.636 DEBUG:boto:Data: 2023-09-22T21:23:07.636 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.636 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:07.636 DEBUG:boto:Port: 8001 2023-09-22T21:23:07.636 DEBUG:boto:Params: {} 2023-09-22T21:23:07.636 DEBUG:boto:Token: None 2023-09-22T21:23:07.636 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.636 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h3gKuZVbSlEnG00Pp6H8306BvYo= 2023-09-22T21:23:07.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h3gKuZVbSlEnG00Pp6H8306BvYo=', 'Content-Length': '0'} 2023-09-22T21:23:07.639 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e279ad713af0b01a-00650e05bb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.682 DEBUG:boto:path=/ffzdqu-48/obj2 2023-09-22T21:23:07.682 DEBUG:boto:auth_path=/ffzdqu-48/obj2 2023-09-22T21:23:07.683 DEBUG:boto:path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.683 DEBUG:boto:auth_path=/ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.683 DEBUG:boto:Method: GET 2023-09-22T21:23:07.683 DEBUG:boto:Path: /ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.683 DEBUG:boto:Data: 2023-09-22T21:23:07.683 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.683 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:07.683 DEBUG:boto:Port: 8000 2023-09-22T21:23:07.683 DEBUG:boto:Params: {} 2023-09-22T21:23:07.683 DEBUG:boto:Token: None 2023-09-22T21:23:07.684 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj2?versionId=null 2023-09-22T21:23:07.684 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h3gKuZVbSlEnG00Pp6H8306BvYo= 2023-09-22T21:23:07.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h3gKuZVbSlEnG00Pp6H8306BvYo=', 'Content-Length': '0'} 2023-09-22T21:23:07.687 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0261a066a579b40-00650e05bb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.687 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:07.687 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:07.687 DEBUG:boto:path=/ffzdqu-48/?prefix=obj2 2023-09-22T21:23:07.687 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj2 2023-09-22T21:23:07.688 DEBUG:boto:Method: GET 2023-09-22T21:23:07.688 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj2 2023-09-22T21:23:07.688 DEBUG:boto:Data: 2023-09-22T21:23:07.688 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.688 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:07.688 DEBUG:boto:Port: 8001 2023-09-22T21:23:07.688 DEBUG:boto:Params: {} 2023-09-22T21:23:07.688 DEBUG:boto:Token: None 2023-09-22T21:23:07.688 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/ 2023-09-22T21:23:07.689 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bTRUUAryCSnjcOamVsir2LXHCWw= 2023-09-22T21:23:07.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bTRUUAryCSnjcOamVsir2LXHCWw=', 'Content-Length': '0'} 2023-09-22T21:23:07.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003333141af00e589e-00650e05bb-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.734 DEBUG:boto:b'ffzdqu-48obj21000falseobj22023-09-22T21:22:07.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:07.735 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:07.735 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:07.735 DEBUG:boto:path=/ffzdqu-48/?prefix=obj2 2023-09-22T21:23:07.735 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj2 2023-09-22T21:23:07.735 DEBUG:boto:Method: GET 2023-09-22T21:23:07.735 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj2 2023-09-22T21:23:07.736 DEBUG:boto:Data: 2023-09-22T21:23:07.736 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:07.736 DEBUG:boto:Port: 8000 2023-09-22T21:23:07.736 DEBUG:boto:Params: {} 2023-09-22T21:23:07.736 DEBUG:boto:Token: None 2023-09-22T21:23:07.736 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/ 2023-09-22T21:23:07.736 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bTRUUAryCSnjcOamVsir2LXHCWw= 2023-09-22T21:23:07.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bTRUUAryCSnjcOamVsir2LXHCWw=', 'Content-Length': '0'} 2023-09-22T21:23:07.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fc3666f0be8c817-00650e05bb-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.741 DEBUG:boto:b'ffzdqu-48obj21000falseobj22023-09-22T21:22:07.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:07.741 DEBUG:boto:path=/ffzdqu-48/obj2 2023-09-22T21:23:07.741 DEBUG:boto:auth_path=/ffzdqu-48/obj2 2023-09-22T21:23:07.742 DEBUG:boto:Method: HEAD 2023-09-22T21:23:07.742 DEBUG:boto:Path: /ffzdqu-48/obj2 2023-09-22T21:23:07.742 DEBUG:boto:Data: 2023-09-22T21:23:07.742 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.742 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:07.742 DEBUG:boto:Port: 8001 2023-09-22T21:23:07.742 DEBUG:boto:Params: {} 2023-09-22T21:23:07.742 DEBUG:boto:Token: None 2023-09-22T21:23:07.742 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj2 2023-09-22T21:23:07.743 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HQV7mE22TFNPIEEMHYRwKicoShE= 2023-09-22T21:23:07.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HQV7mE22TFNPIEEMHYRwKicoShE=', 'Content-Length': '0'} 2023-09-22T21:23:07.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9d7a93f93f536ab-00650e05bb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.746 DEBUG:boto:path=/ffzdqu-48/obj2 2023-09-22T21:23:07.746 DEBUG:boto:auth_path=/ffzdqu-48/obj2 2023-09-22T21:23:07.746 DEBUG:boto:Method: HEAD 2023-09-22T21:23:07.746 DEBUG:boto:Path: /ffzdqu-48/obj2 2023-09-22T21:23:07.746 DEBUG:boto:Data: 2023-09-22T21:23:07.746 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.746 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:07.746 DEBUG:boto:Port: 8000 2023-09-22T21:23:07.746 DEBUG:boto:Params: {} 2023-09-22T21:23:07.747 DEBUG:boto:Token: None 2023-09-22T21:23:07.747 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj2 2023-09-22T21:23:07.747 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HQV7mE22TFNPIEEMHYRwKicoShE= 2023-09-22T21:23:07.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HQV7mE22TFNPIEEMHYRwKicoShE=', 'Content-Length': '0'} 2023-09-22T21:23:07.749 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072cfbde8fe3cde5c-00650e05bb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:23:07.750 DEBUG:boto:path=/ffzdqu-48/obj2 2023-09-22T21:23:07.750 DEBUG:boto:auth_path=/ffzdqu-48/obj2 2023-09-22T21:23:07.750 DEBUG:boto:Method: GET 2023-09-22T21:23:07.750 DEBUG:boto:Path: /ffzdqu-48/obj2 2023-09-22T21:23:07.750 DEBUG:boto:Data: 2023-09-22T21:23:07.750 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.750 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:07.750 DEBUG:boto:Port: 8001 2023-09-22T21:23:07.750 DEBUG:boto:Params: {} 2023-09-22T21:23:07.751 DEBUG:boto:Token: None 2023-09-22T21:23:07.751 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj2 2023-09-22T21:23:07.751 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Yu0HhsxBCJxgQ8yAr3GWJrQYfTY= 2023-09-22T21:23:07.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Yu0HhsxBCJxgQ8yAr3GWJrQYfTY=', 'Content-Length': '0'} 2023-09-22T21:23:07.754 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d59f35f9b8a7f3e0-00650e05bb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.794 DEBUG:boto:path=/ffzdqu-48/obj2 2023-09-22T21:23:07.794 DEBUG:boto:auth_path=/ffzdqu-48/obj2 2023-09-22T21:23:07.795 DEBUG:boto:Method: GET 2023-09-22T21:23:07.795 DEBUG:boto:Path: /ffzdqu-48/obj2 2023-09-22T21:23:07.795 DEBUG:boto:Data: 2023-09-22T21:23:07.795 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.795 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:07.795 DEBUG:boto:Port: 8000 2023-09-22T21:23:07.795 DEBUG:boto:Params: {} 2023-09-22T21:23:07.795 DEBUG:boto:Token: None 2023-09-22T21:23:07.795 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj2 2023-09-22T21:23:07.796 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Yu0HhsxBCJxgQ8yAr3GWJrQYfTY= 2023-09-22T21:23:07.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Yu0HhsxBCJxgQ8yAr3GWJrQYfTY=', 'Content-Length': '0'} 2023-09-22T21:23:07.799 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000624e1cace3423c70-00650e05bb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.800 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:07.800 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:07.800 DEBUG:boto:path=/ffzdqu-48/?prefix=obj2 2023-09-22T21:23:07.800 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj2 2023-09-22T21:23:07.800 DEBUG:boto:Method: GET 2023-09-22T21:23:07.800 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj2 2023-09-22T21:23:07.800 DEBUG:boto:Data: 2023-09-22T21:23:07.800 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.800 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:07.801 DEBUG:boto:Port: 8001 2023-09-22T21:23:07.801 DEBUG:boto:Params: {} 2023-09-22T21:23:07.801 DEBUG:boto:Token: None 2023-09-22T21:23:07.801 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/ 2023-09-22T21:23:07.801 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bTRUUAryCSnjcOamVsir2LXHCWw= 2023-09-22T21:23:07.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bTRUUAryCSnjcOamVsir2LXHCWw=', 'Content-Length': '0'} 2023-09-22T21:23:07.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001138522d4c555cf7-00650e05bb-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.850 DEBUG:boto:b'ffzdqu-48obj21000falseobj22023-09-22T21:22:07.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:07.851 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:07.851 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:07.851 DEBUG:boto:path=/ffzdqu-48/?prefix=obj2 2023-09-22T21:23:07.851 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj2 2023-09-22T21:23:07.851 DEBUG:boto:Method: GET 2023-09-22T21:23:07.851 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj2 2023-09-22T21:23:07.851 DEBUG:boto:Data: 2023-09-22T21:23:07.852 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.852 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:07.852 DEBUG:boto:Port: 8000 2023-09-22T21:23:07.852 DEBUG:boto:Params: {} 2023-09-22T21:23:07.852 DEBUG:boto:Token: None 2023-09-22T21:23:07.852 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/ 2023-09-22T21:23:07.852 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bTRUUAryCSnjcOamVsir2LXHCWw= 2023-09-22T21:23:07.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bTRUUAryCSnjcOamVsir2LXHCWw=', 'Content-Length': '0'} 2023-09-22T21:23:07.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d7804192ecc04d0-00650e05bb-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.857 DEBUG:boto:b'ffzdqu-48obj21000falseobj22023-09-22T21:22:07.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:07.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:23:07.858 DEBUG:boto:path=/ffzdqu-48/obj3 2023-09-22T21:23:07.858 DEBUG:boto:auth_path=/ffzdqu-48/obj3 2023-09-22T21:23:07.858 DEBUG:boto:path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.858 DEBUG:boto:auth_path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.858 DEBUG:boto:Method: GET 2023-09-22T21:23:07.858 DEBUG:boto:Path: /ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.858 DEBUG:boto:Data: 2023-09-22T21:23:07.858 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.858 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:07.859 DEBUG:boto:Port: 8001 2023-09-22T21:23:07.859 DEBUG:boto:Params: {} 2023-09-22T21:23:07.859 DEBUG:boto:Token: None 2023-09-22T21:23:07.859 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.859 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JvtwUzG9sC5PozQyBgnGkiBLLiI= 2023-09-22T21:23:07.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JvtwUzG9sC5PozQyBgnGkiBLLiI=', 'Content-Length': '0'} 2023-09-22T21:23:07.862 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086aa7d8b2b88c0da-00650e05bb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.906 DEBUG:boto:path=/ffzdqu-48/obj3 2023-09-22T21:23:07.907 DEBUG:boto:auth_path=/ffzdqu-48/obj3 2023-09-22T21:23:07.907 DEBUG:boto:path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.907 DEBUG:boto:auth_path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.907 DEBUG:boto:Method: GET 2023-09-22T21:23:07.907 DEBUG:boto:Path: /ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.907 DEBUG:boto:Data: 2023-09-22T21:23:07.907 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.907 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:07.907 DEBUG:boto:Port: 8000 2023-09-22T21:23:07.907 DEBUG:boto:Params: {} 2023-09-22T21:23:07.908 DEBUG:boto:Token: None 2023-09-22T21:23:07.908 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.908 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JvtwUzG9sC5PozQyBgnGkiBLLiI= 2023-09-22T21:23:07.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JvtwUzG9sC5PozQyBgnGkiBLLiI=', 'Content-Length': '0'} 2023-09-22T21:23:07.911 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094604149a53ddf7b-00650e05bb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.911 DEBUG:boto:path=/ffzdqu-48/obj3 2023-09-22T21:23:07.911 DEBUG:boto:auth_path=/ffzdqu-48/obj3 2023-09-22T21:23:07.911 DEBUG:boto:path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.912 DEBUG:boto:auth_path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.912 DEBUG:boto:Method: HEAD 2023-09-22T21:23:07.912 DEBUG:boto:Path: /ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.912 DEBUG:boto:Data: 2023-09-22T21:23:07.912 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.912 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:07.912 DEBUG:boto:Port: 8001 2023-09-22T21:23:07.912 DEBUG:boto:Params: {} 2023-09-22T21:23:07.912 DEBUG:boto:Token: None 2023-09-22T21:23:07.913 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.913 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y1PlECdq5ObQwpF+yVCpxHLvbU4= 2023-09-22T21:23:07.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y1PlECdq5ObQwpF+yVCpxHLvbU4=', 'Content-Length': '0'} 2023-09-22T21:23:07.915 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003313d926ebdc71f6-00650e05bb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.916 DEBUG:boto:path=/ffzdqu-48/obj3 2023-09-22T21:23:07.916 DEBUG:boto:auth_path=/ffzdqu-48/obj3 2023-09-22T21:23:07.916 DEBUG:boto:path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.916 DEBUG:boto:auth_path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.916 DEBUG:boto:Method: HEAD 2023-09-22T21:23:07.916 DEBUG:boto:Path: /ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.916 DEBUG:boto:Data: 2023-09-22T21:23:07.916 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.916 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:07.917 DEBUG:boto:Port: 8000 2023-09-22T21:23:07.917 DEBUG:boto:Params: {} 2023-09-22T21:23:07.917 DEBUG:boto:Token: None 2023-09-22T21:23:07.917 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.917 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y1PlECdq5ObQwpF+yVCpxHLvbU4= 2023-09-22T21:23:07.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y1PlECdq5ObQwpF+yVCpxHLvbU4=', 'Content-Length': '0'} 2023-09-22T21:23:07.920 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8920068306cb448-00650e05bb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:23:07.920 DEBUG:boto:path=/ffzdqu-48/obj3 2023-09-22T21:23:07.921 DEBUG:boto:auth_path=/ffzdqu-48/obj3 2023-09-22T21:23:07.921 DEBUG:boto:path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.921 DEBUG:boto:auth_path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.921 DEBUG:boto:Method: GET 2023-09-22T21:23:07.921 DEBUG:boto:Path: /ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.921 DEBUG:boto:Data: 2023-09-22T21:23:07.921 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.921 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:07.921 DEBUG:boto:Port: 8001 2023-09-22T21:23:07.921 DEBUG:boto:Params: {} 2023-09-22T21:23:07.922 DEBUG:boto:Token: None 2023-09-22T21:23:07.922 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.922 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JvtwUzG9sC5PozQyBgnGkiBLLiI= 2023-09-22T21:23:07.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JvtwUzG9sC5PozQyBgnGkiBLLiI=', 'Content-Length': '0'} 2023-09-22T21:23:07.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031b5327e1737c8b4-00650e05bb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.966 DEBUG:boto:path=/ffzdqu-48/obj3 2023-09-22T21:23:07.966 DEBUG:boto:auth_path=/ffzdqu-48/obj3 2023-09-22T21:23:07.967 DEBUG:boto:path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.967 DEBUG:boto:auth_path=/ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.967 DEBUG:boto:Method: GET 2023-09-22T21:23:07.967 DEBUG:boto:Path: /ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.967 DEBUG:boto:Data: 2023-09-22T21:23:07.967 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.967 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:07.967 DEBUG:boto:Port: 8000 2023-09-22T21:23:07.967 DEBUG:boto:Params: {} 2023-09-22T21:23:07.967 DEBUG:boto:Token: None 2023-09-22T21:23:07.968 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/obj3?versionId=null 2023-09-22T21:23:07.968 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JvtwUzG9sC5PozQyBgnGkiBLLiI= 2023-09-22T21:23:07.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JvtwUzG9sC5PozQyBgnGkiBLLiI=', 'Content-Length': '0'} 2023-09-22T21:23:07.971 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2d672554e824661-00650e05bb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:07.971 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:07.971 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:07.971 DEBUG:boto:path=/ffzdqu-48/?prefix=obj3 2023-09-22T21:23:07.971 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj3 2023-09-22T21:23:07.971 DEBUG:boto:Method: GET 2023-09-22T21:23:07.972 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj3 2023-09-22T21:23:07.972 DEBUG:boto:Data: 2023-09-22T21:23:07.972 DEBUG:boto:Headers: {} 2023-09-22T21:23:07.972 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:07.972 DEBUG:boto:Port: 8001 2023-09-22T21:23:07.972 DEBUG:boto:Params: {} 2023-09-22T21:23:07.972 DEBUG:boto:Token: None 2023-09-22T21:23:07.972 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:07 GMT /ffzdqu-48/ 2023-09-22T21:23:07.972 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bTRUUAryCSnjcOamVsir2LXHCWw= 2023-09-22T21:23:07.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bTRUUAryCSnjcOamVsir2LXHCWw=', 'Content-Length': '0'} 2023-09-22T21:23:07.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031b486e93f393247-00650e05bb-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.018 DEBUG:boto:b'ffzdqu-48obj31000falseobj32023-09-22T21:22:07.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:08.019 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:08.019 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:08.019 DEBUG:boto:path=/ffzdqu-48/?prefix=obj3 2023-09-22T21:23:08.019 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj3 2023-09-22T21:23:08.019 DEBUG:boto:Method: GET 2023-09-22T21:23:08.019 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj3 2023-09-22T21:23:08.019 DEBUG:boto:Data: 2023-09-22T21:23:08.020 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.020 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.020 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.020 DEBUG:boto:Params: {} 2023-09-22T21:23:08.020 DEBUG:boto:Token: None 2023-09-22T21:23:08.020 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/ 2023-09-22T21:23:08.020 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA= 2023-09-22T21:23:08.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA=', 'Content-Length': '0'} 2023-09-22T21:23:08.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023d95326925f64b2-00650e05bc-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.025 DEBUG:boto:b'ffzdqu-48obj31000falseobj32023-09-22T21:22:07.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:08.025 DEBUG:boto:path=/ffzdqu-48/obj3 2023-09-22T21:23:08.026 DEBUG:boto:auth_path=/ffzdqu-48/obj3 2023-09-22T21:23:08.026 DEBUG:boto:Method: HEAD 2023-09-22T21:23:08.026 DEBUG:boto:Path: /ffzdqu-48/obj3 2023-09-22T21:23:08.026 DEBUG:boto:Data: 2023-09-22T21:23:08.026 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.026 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.026 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.026 DEBUG:boto:Params: {} 2023-09-22T21:23:08.026 DEBUG:boto:Token: None 2023-09-22T21:23:08.027 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj3 2023-09-22T21:23:08.027 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ct/IrV/Wg0bEgiAsJRVFcG+INzQ= 2023-09-22T21:23:08.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ct/IrV/Wg0bEgiAsJRVFcG+INzQ=', 'Content-Length': '0'} 2023-09-22T21:23:08.029 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000874a8df34232b78e-00650e05bc-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.030 DEBUG:boto:path=/ffzdqu-48/obj3 2023-09-22T21:23:08.030 DEBUG:boto:auth_path=/ffzdqu-48/obj3 2023-09-22T21:23:08.030 DEBUG:boto:Method: HEAD 2023-09-22T21:23:08.030 DEBUG:boto:Path: /ffzdqu-48/obj3 2023-09-22T21:23:08.030 DEBUG:boto:Data: 2023-09-22T21:23:08.030 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.030 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.030 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.031 DEBUG:boto:Params: {} 2023-09-22T21:23:08.031 DEBUG:boto:Token: None 2023-09-22T21:23:08.031 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj3 2023-09-22T21:23:08.031 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ct/IrV/Wg0bEgiAsJRVFcG+INzQ= 2023-09-22T21:23:08.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ct/IrV/Wg0bEgiAsJRVFcG+INzQ=', 'Content-Length': '0'} 2023-09-22T21:23:08.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f5796b781722b66-00650e05bc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:23:08.034 DEBUG:boto:path=/ffzdqu-48/obj3 2023-09-22T21:23:08.034 DEBUG:boto:auth_path=/ffzdqu-48/obj3 2023-09-22T21:23:08.034 DEBUG:boto:Method: GET 2023-09-22T21:23:08.034 DEBUG:boto:Path: /ffzdqu-48/obj3 2023-09-22T21:23:08.034 DEBUG:boto:Data: 2023-09-22T21:23:08.035 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.035 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.035 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.035 DEBUG:boto:Params: {} 2023-09-22T21:23:08.035 DEBUG:boto:Token: None 2023-09-22T21:23:08.035 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj3 2023-09-22T21:23:08.035 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SNAZ7rWLRIWiiwRjhbdXSvTn13Y= 2023-09-22T21:23:08.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNAZ7rWLRIWiiwRjhbdXSvTn13Y=', 'Content-Length': '0'} 2023-09-22T21:23:08.038 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b91d850a9d9fe140-00650e05bc-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.082 DEBUG:boto:path=/ffzdqu-48/obj3 2023-09-22T21:23:08.083 DEBUG:boto:auth_path=/ffzdqu-48/obj3 2023-09-22T21:23:08.083 DEBUG:boto:Method: GET 2023-09-22T21:23:08.083 DEBUG:boto:Path: /ffzdqu-48/obj3 2023-09-22T21:23:08.083 DEBUG:boto:Data: 2023-09-22T21:23:08.083 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.083 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.083 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.083 DEBUG:boto:Params: {} 2023-09-22T21:23:08.084 DEBUG:boto:Token: None 2023-09-22T21:23:08.084 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj3 2023-09-22T21:23:08.084 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SNAZ7rWLRIWiiwRjhbdXSvTn13Y= 2023-09-22T21:23:08.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNAZ7rWLRIWiiwRjhbdXSvTn13Y=', 'Content-Length': '0'} 2023-09-22T21:23:08.087 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b08010aa8dab49f-00650e05bc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.087 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:08.087 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:08.087 DEBUG:boto:path=/ffzdqu-48/?prefix=obj3 2023-09-22T21:23:08.088 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj3 2023-09-22T21:23:08.088 DEBUG:boto:Method: GET 2023-09-22T21:23:08.088 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj3 2023-09-22T21:23:08.088 DEBUG:boto:Data: 2023-09-22T21:23:08.088 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.088 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.088 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.088 DEBUG:boto:Params: {} 2023-09-22T21:23:08.088 DEBUG:boto:Token: None 2023-09-22T21:23:08.088 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/ 2023-09-22T21:23:08.089 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA= 2023-09-22T21:23:08.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA=', 'Content-Length': '0'} 2023-09-22T21:23:08.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f57824716f70370-00650e05bc-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.094 DEBUG:boto:b'ffzdqu-48obj31000falseobj32023-09-22T21:22:07.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:08.094 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:08.094 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:08.094 DEBUG:boto:path=/ffzdqu-48/?prefix=obj3 2023-09-22T21:23:08.094 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj3 2023-09-22T21:23:08.095 DEBUG:boto:Method: GET 2023-09-22T21:23:08.095 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj3 2023-09-22T21:23:08.095 DEBUG:boto:Data: 2023-09-22T21:23:08.095 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.095 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.095 DEBUG:boto:Params: {} 2023-09-22T21:23:08.095 DEBUG:boto:Token: None 2023-09-22T21:23:08.095 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/ 2023-09-22T21:23:08.096 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA= 2023-09-22T21:23:08.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA=', 'Content-Length': '0'} 2023-09-22T21:23:08.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a5b7782ec99f825-00650e05bc-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.142 DEBUG:boto:b'ffzdqu-48obj31000falseobj32023-09-22T21:22:07.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:08.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:23:08.143 DEBUG:boto:path=/ffzdqu-48/obj4 2023-09-22T21:23:08.143 DEBUG:boto:auth_path=/ffzdqu-48/obj4 2023-09-22T21:23:08.143 DEBUG:boto:path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.143 DEBUG:boto:auth_path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.143 DEBUG:boto:Method: GET 2023-09-22T21:23:08.144 DEBUG:boto:Path: /ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.144 DEBUG:boto:Data: 2023-09-22T21:23:08.144 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.144 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.144 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.144 DEBUG:boto:Params: {} 2023-09-22T21:23:08.144 DEBUG:boto:Token: None 2023-09-22T21:23:08.144 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.145 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F11aCJDcB40x22wHkZdju3lRr6g= 2023-09-22T21:23:08.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F11aCJDcB40x22wHkZdju3lRr6g=', 'Content-Length': '0'} 2023-09-22T21:23:08.147 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8e754c7d343a4aa-00650e05bc-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.190 DEBUG:boto:path=/ffzdqu-48/obj4 2023-09-22T21:23:08.190 DEBUG:boto:auth_path=/ffzdqu-48/obj4 2023-09-22T21:23:08.191 DEBUG:boto:path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.191 DEBUG:boto:auth_path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.191 DEBUG:boto:Method: GET 2023-09-22T21:23:08.191 DEBUG:boto:Path: /ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.191 DEBUG:boto:Data: 2023-09-22T21:23:08.191 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.191 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.191 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.191 DEBUG:boto:Params: {} 2023-09-22T21:23:08.192 DEBUG:boto:Token: None 2023-09-22T21:23:08.192 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.192 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F11aCJDcB40x22wHkZdju3lRr6g= 2023-09-22T21:23:08.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F11aCJDcB40x22wHkZdju3lRr6g=', 'Content-Length': '0'} 2023-09-22T21:23:08.195 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005876978841d4084b-00650e05bc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.195 DEBUG:boto:path=/ffzdqu-48/obj4 2023-09-22T21:23:08.195 DEBUG:boto:auth_path=/ffzdqu-48/obj4 2023-09-22T21:23:08.195 DEBUG:boto:path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.195 DEBUG:boto:auth_path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.196 DEBUG:boto:Method: HEAD 2023-09-22T21:23:08.196 DEBUG:boto:Path: /ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.196 DEBUG:boto:Data: 2023-09-22T21:23:08.196 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.196 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.196 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.196 DEBUG:boto:Params: {} 2023-09-22T21:23:08.196 DEBUG:boto:Token: None 2023-09-22T21:23:08.196 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.197 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nv0mBQMHqb8Yipei1Mz9sHnmuyg= 2023-09-22T21:23:08.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nv0mBQMHqb8Yipei1Mz9sHnmuyg=', 'Content-Length': '0'} 2023-09-22T21:23:08.199 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8fc599f5ad1f5aa-00650e05bc-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.199 DEBUG:boto:path=/ffzdqu-48/obj4 2023-09-22T21:23:08.200 DEBUG:boto:auth_path=/ffzdqu-48/obj4 2023-09-22T21:23:08.200 DEBUG:boto:path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.200 DEBUG:boto:auth_path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.200 DEBUG:boto:Method: HEAD 2023-09-22T21:23:08.200 DEBUG:boto:Path: /ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.200 DEBUG:boto:Data: 2023-09-22T21:23:08.200 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.200 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.200 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.200 DEBUG:boto:Params: {} 2023-09-22T21:23:08.201 DEBUG:boto:Token: None 2023-09-22T21:23:08.201 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.201 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nv0mBQMHqb8Yipei1Mz9sHnmuyg= 2023-09-22T21:23:08.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nv0mBQMHqb8Yipei1Mz9sHnmuyg=', 'Content-Length': '0'} 2023-09-22T21:23:08.203 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000aed47fc946d5158-00650e05bc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:23:08.204 DEBUG:boto:path=/ffzdqu-48/obj4 2023-09-22T21:23:08.204 DEBUG:boto:auth_path=/ffzdqu-48/obj4 2023-09-22T21:23:08.204 DEBUG:boto:path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.204 DEBUG:boto:auth_path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.204 DEBUG:boto:Method: GET 2023-09-22T21:23:08.204 DEBUG:boto:Path: /ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.205 DEBUG:boto:Data: 2023-09-22T21:23:08.205 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.205 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.205 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.205 DEBUG:boto:Params: {} 2023-09-22T21:23:08.205 DEBUG:boto:Token: None 2023-09-22T21:23:08.205 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.205 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F11aCJDcB40x22wHkZdju3lRr6g= 2023-09-22T21:23:08.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F11aCJDcB40x22wHkZdju3lRr6g=', 'Content-Length': '0'} 2023-09-22T21:23:08.208 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a37fd0412a862aaf-00650e05bc-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.254 DEBUG:boto:path=/ffzdqu-48/obj4 2023-09-22T21:23:08.254 DEBUG:boto:auth_path=/ffzdqu-48/obj4 2023-09-22T21:23:08.255 DEBUG:boto:path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.255 DEBUG:boto:auth_path=/ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.255 DEBUG:boto:Method: GET 2023-09-22T21:23:08.255 DEBUG:boto:Path: /ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.255 DEBUG:boto:Data: 2023-09-22T21:23:08.255 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.255 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.255 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.255 DEBUG:boto:Params: {} 2023-09-22T21:23:08.255 DEBUG:boto:Token: None 2023-09-22T21:23:08.256 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj4?versionId=null 2023-09-22T21:23:08.256 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F11aCJDcB40x22wHkZdju3lRr6g= 2023-09-22T21:23:08.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F11aCJDcB40x22wHkZdju3lRr6g=', 'Content-Length': '0'} 2023-09-22T21:23:08.259 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000eb4ffe3df43f4a4-00650e05bc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.259 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:08.259 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:08.259 DEBUG:boto:path=/ffzdqu-48/?prefix=obj4 2023-09-22T21:23:08.259 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj4 2023-09-22T21:23:08.259 DEBUG:boto:Method: GET 2023-09-22T21:23:08.260 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj4 2023-09-22T21:23:08.260 DEBUG:boto:Data: 2023-09-22T21:23:08.260 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.260 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.260 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.260 DEBUG:boto:Params: {} 2023-09-22T21:23:08.260 DEBUG:boto:Token: None 2023-09-22T21:23:08.260 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/ 2023-09-22T21:23:08.260 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA= 2023-09-22T21:23:08.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA=', 'Content-Length': '0'} 2023-09-22T21:23:08.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b900aa07d1360944-00650e05bc-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.306 DEBUG:boto:b'ffzdqu-48obj41000falseobj42023-09-22T21:22:07.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:08.307 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:08.307 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:08.307 DEBUG:boto:path=/ffzdqu-48/?prefix=obj4 2023-09-22T21:23:08.307 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj4 2023-09-22T21:23:08.307 DEBUG:boto:Method: GET 2023-09-22T21:23:08.307 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj4 2023-09-22T21:23:08.307 DEBUG:boto:Data: 2023-09-22T21:23:08.307 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.308 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.308 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.308 DEBUG:boto:Params: {} 2023-09-22T21:23:08.308 DEBUG:boto:Token: None 2023-09-22T21:23:08.308 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/ 2023-09-22T21:23:08.308 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA= 2023-09-22T21:23:08.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA=', 'Content-Length': '0'} 2023-09-22T21:23:08.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bbfaf3bbc14c9de-00650e05bc-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.313 DEBUG:boto:b'ffzdqu-48obj41000falseobj42023-09-22T21:22:07.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:08.313 DEBUG:boto:path=/ffzdqu-48/obj4 2023-09-22T21:23:08.313 DEBUG:boto:auth_path=/ffzdqu-48/obj4 2023-09-22T21:23:08.313 DEBUG:boto:Method: HEAD 2023-09-22T21:23:08.314 DEBUG:boto:Path: /ffzdqu-48/obj4 2023-09-22T21:23:08.314 DEBUG:boto:Data: 2023-09-22T21:23:08.314 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.314 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.314 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.314 DEBUG:boto:Params: {} 2023-09-22T21:23:08.314 DEBUG:boto:Token: None 2023-09-22T21:23:08.314 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj4 2023-09-22T21:23:08.315 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5Mp566cbsDgrgH5RCXX1vjYU8Qo= 2023-09-22T21:23:08.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5Mp566cbsDgrgH5RCXX1vjYU8Qo=', 'Content-Length': '0'} 2023-09-22T21:23:08.317 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3ebeed91d6fd1a6-00650e05bc-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.317 DEBUG:boto:path=/ffzdqu-48/obj4 2023-09-22T21:23:08.318 DEBUG:boto:auth_path=/ffzdqu-48/obj4 2023-09-22T21:23:08.318 DEBUG:boto:Method: HEAD 2023-09-22T21:23:08.318 DEBUG:boto:Path: /ffzdqu-48/obj4 2023-09-22T21:23:08.318 DEBUG:boto:Data: 2023-09-22T21:23:08.318 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.318 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.318 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.318 DEBUG:boto:Params: {} 2023-09-22T21:23:08.318 DEBUG:boto:Token: None 2023-09-22T21:23:08.319 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj4 2023-09-22T21:23:08.319 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5Mp566cbsDgrgH5RCXX1vjYU8Qo= 2023-09-22T21:23:08.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5Mp566cbsDgrgH5RCXX1vjYU8Qo=', 'Content-Length': '0'} 2023-09-22T21:23:08.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031b0bf8c4f1cfa61-00650e05bc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:23:08.322 DEBUG:boto:path=/ffzdqu-48/obj4 2023-09-22T21:23:08.322 DEBUG:boto:auth_path=/ffzdqu-48/obj4 2023-09-22T21:23:08.322 DEBUG:boto:Method: GET 2023-09-22T21:23:08.323 DEBUG:boto:Path: /ffzdqu-48/obj4 2023-09-22T21:23:08.323 DEBUG:boto:Data: 2023-09-22T21:23:08.323 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.323 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.323 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.323 DEBUG:boto:Params: {} 2023-09-22T21:23:08.323 DEBUG:boto:Token: None 2023-09-22T21:23:08.323 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj4 2023-09-22T21:23:08.323 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ja8f5h4Ta7bjt8NgI45xglEgAAM= 2023-09-22T21:23:08.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ja8f5h4Ta7bjt8NgI45xglEgAAM=', 'Content-Length': '0'} 2023-09-22T21:23:08.326 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030ee80ef30ec4792-00650e05bc-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.370 DEBUG:boto:path=/ffzdqu-48/obj4 2023-09-22T21:23:08.370 DEBUG:boto:auth_path=/ffzdqu-48/obj4 2023-09-22T21:23:08.371 DEBUG:boto:Method: GET 2023-09-22T21:23:08.371 DEBUG:boto:Path: /ffzdqu-48/obj4 2023-09-22T21:23:08.371 DEBUG:boto:Data: 2023-09-22T21:23:08.371 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.371 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.371 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.371 DEBUG:boto:Params: {} 2023-09-22T21:23:08.371 DEBUG:boto:Token: None 2023-09-22T21:23:08.371 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj4 2023-09-22T21:23:08.372 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ja8f5h4Ta7bjt8NgI45xglEgAAM= 2023-09-22T21:23:08.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ja8f5h4Ta7bjt8NgI45xglEgAAM=', 'Content-Length': '0'} 2023-09-22T21:23:08.374 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082d7fa99d7ad4e9f-00650e05bc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.375 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:08.375 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:08.375 DEBUG:boto:path=/ffzdqu-48/?prefix=obj4 2023-09-22T21:23:08.375 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj4 2023-09-22T21:23:08.375 DEBUG:boto:Method: GET 2023-09-22T21:23:08.375 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj4 2023-09-22T21:23:08.375 DEBUG:boto:Data: 2023-09-22T21:23:08.375 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.376 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.376 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.376 DEBUG:boto:Params: {} 2023-09-22T21:23:08.376 DEBUG:boto:Token: None 2023-09-22T21:23:08.376 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/ 2023-09-22T21:23:08.376 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA= 2023-09-22T21:23:08.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA=', 'Content-Length': '0'} 2023-09-22T21:23:08.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000438b27853b73c752-00650e05bc-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.381 DEBUG:boto:b'ffzdqu-48obj41000falseobj42023-09-22T21:22:07.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:08.381 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:08.381 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:08.381 DEBUG:boto:path=/ffzdqu-48/?prefix=obj4 2023-09-22T21:23:08.381 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj4 2023-09-22T21:23:08.381 DEBUG:boto:Method: GET 2023-09-22T21:23:08.382 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj4 2023-09-22T21:23:08.382 DEBUG:boto:Data: 2023-09-22T21:23:08.382 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.382 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.382 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.382 DEBUG:boto:Params: {} 2023-09-22T21:23:08.382 DEBUG:boto:Token: None 2023-09-22T21:23:08.382 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/ 2023-09-22T21:23:08.383 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA= 2023-09-22T21:23:08.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA=', 'Content-Length': '0'} 2023-09-22T21:23:08.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce89515a78563800-00650e05bc-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.430 DEBUG:boto:b'ffzdqu-48obj41000falseobj42023-09-22T21:22:07.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:08.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-22T21:23:08.431 DEBUG:boto:path=/ffzdqu-48/obj5 2023-09-22T21:23:08.431 DEBUG:boto:auth_path=/ffzdqu-48/obj5 2023-09-22T21:23:08.431 DEBUG:boto:path=/ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.431 DEBUG:boto:auth_path=/ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.431 DEBUG:boto:Method: GET 2023-09-22T21:23:08.431 DEBUG:boto:Path: /ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.432 DEBUG:boto:Data: 2023-09-22T21:23:08.432 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.432 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.432 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.432 DEBUG:boto:Params: {} 2023-09-22T21:23:08.432 DEBUG:boto:Token: None 2023-09-22T21:23:08.432 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.432 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bqMW4F8K3f44P4Ijxb/LkaOVi1A= 2023-09-22T21:23:08.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bqMW4F8K3f44P4Ijxb/LkaOVi1A=', 'Content-Length': '0'} 2023-09-22T21:23:08.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f40aeafda16b723c-00650e05bc-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.478 DEBUG:boto:path=/ffzdqu-48/obj5 2023-09-22T21:23:08.478 DEBUG:boto:auth_path=/ffzdqu-48/obj5 2023-09-22T21:23:08.479 DEBUG:boto:path=/ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.479 DEBUG:boto:auth_path=/ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.479 DEBUG:boto:Method: GET 2023-09-22T21:23:08.479 DEBUG:boto:Path: /ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.479 DEBUG:boto:Data: 2023-09-22T21:23:08.479 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.479 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.479 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.479 DEBUG:boto:Params: {} 2023-09-22T21:23:08.479 DEBUG:boto:Token: None 2023-09-22T21:23:08.480 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.480 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bqMW4F8K3f44P4Ijxb/LkaOVi1A= 2023-09-22T21:23:08.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bqMW4F8K3f44P4Ijxb/LkaOVi1A=', 'Content-Length': '0'} 2023-09-22T21:23:08.483 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe9e99bcc287fef1-00650e05bc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.483 DEBUG:boto:path=/ffzdqu-48/obj5 2023-09-22T21:23:08.483 DEBUG:boto:auth_path=/ffzdqu-48/obj5 2023-09-22T21:23:08.483 DEBUG:boto:path=/ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.483 DEBUG:boto:auth_path=/ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.483 DEBUG:boto:Method: HEAD 2023-09-22T21:23:08.484 DEBUG:boto:Path: /ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.484 DEBUG:boto:Data: 2023-09-22T21:23:08.484 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.484 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.484 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.484 DEBUG:boto:Params: {} 2023-09-22T21:23:08.484 DEBUG:boto:Token: None 2023-09-22T21:23:08.484 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.484 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kP1vvg5sJR52ZKPJPwgWnK4S0Vk= 2023-09-22T21:23:08.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kP1vvg5sJR52ZKPJPwgWnK4S0Vk=', 'Content-Length': '0'} 2023-09-22T21:23:08.487 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb632ef6ebb6c5cc-00650e05bc-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.487 DEBUG:boto:path=/ffzdqu-48/obj5 2023-09-22T21:23:08.487 DEBUG:boto:auth_path=/ffzdqu-48/obj5 2023-09-22T21:23:08.488 DEBUG:boto:path=/ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.488 DEBUG:boto:auth_path=/ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.488 DEBUG:boto:Method: HEAD 2023-09-22T21:23:08.488 DEBUG:boto:Path: /ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.488 DEBUG:boto:Data: 2023-09-22T21:23:08.488 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.488 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.488 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.488 DEBUG:boto:Params: {} 2023-09-22T21:23:08.489 DEBUG:boto:Token: None 2023-09-22T21:23:08.489 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.489 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kP1vvg5sJR52ZKPJPwgWnK4S0Vk= 2023-09-22T21:23:08.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kP1vvg5sJR52ZKPJPwgWnK4S0Vk=', 'Content-Length': '0'} 2023-09-22T21:23:08.492 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095d2657ef8c3b8ee-00650e05bc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-22T21:23:08.492 DEBUG:boto:path=/ffzdqu-48/obj5 2023-09-22T21:23:08.492 DEBUG:boto:auth_path=/ffzdqu-48/obj5 2023-09-22T21:23:08.492 DEBUG:boto:path=/ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.492 DEBUG:boto:auth_path=/ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.492 DEBUG:boto:Method: GET 2023-09-22T21:23:08.493 DEBUG:boto:Path: /ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.493 DEBUG:boto:Data: 2023-09-22T21:23:08.493 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.493 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.493 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.493 DEBUG:boto:Params: {} 2023-09-22T21:23:08.493 DEBUG:boto:Token: None 2023-09-22T21:23:08.493 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.493 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bqMW4F8K3f44P4Ijxb/LkaOVi1A= 2023-09-22T21:23:08.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bqMW4F8K3f44P4Ijxb/LkaOVi1A=', 'Content-Length': '0'} 2023-09-22T21:23:08.496 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8a01d7ccd6247b3-00650e05bc-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.538 DEBUG:boto:path=/ffzdqu-48/obj5 2023-09-22T21:23:08.539 DEBUG:boto:auth_path=/ffzdqu-48/obj5 2023-09-22T21:23:08.539 DEBUG:boto:path=/ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.539 DEBUG:boto:auth_path=/ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.539 DEBUG:boto:Method: GET 2023-09-22T21:23:08.539 DEBUG:boto:Path: /ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.539 DEBUG:boto:Data: 2023-09-22T21:23:08.539 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.539 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.539 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.540 DEBUG:boto:Params: {} 2023-09-22T21:23:08.540 DEBUG:boto:Token: None 2023-09-22T21:23:08.540 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj5?versionId=null 2023-09-22T21:23:08.540 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bqMW4F8K3f44P4Ijxb/LkaOVi1A= 2023-09-22T21:23:08.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bqMW4F8K3f44P4Ijxb/LkaOVi1A=', 'Content-Length': '0'} 2023-09-22T21:23:08.543 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004580d02f01c58879-00650e05bc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.544 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:08.544 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:08.544 DEBUG:boto:path=/ffzdqu-48/?prefix=obj5 2023-09-22T21:23:08.544 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj5 2023-09-22T21:23:08.544 DEBUG:boto:Method: GET 2023-09-22T21:23:08.544 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj5 2023-09-22T21:23:08.544 DEBUG:boto:Data: 2023-09-22T21:23:08.544 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.545 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.545 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.545 DEBUG:boto:Params: {} 2023-09-22T21:23:08.545 DEBUG:boto:Token: None 2023-09-22T21:23:08.545 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/ 2023-09-22T21:23:08.545 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA= 2023-09-22T21:23:08.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA=', 'Content-Length': '0'} 2023-09-22T21:23:08.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e428bdb2ccfdbc4-00650e05bc-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.594 DEBUG:boto:b'ffzdqu-48obj51000falseobj52023-09-22T21:22:41.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:08.595 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:08.595 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:08.595 DEBUG:boto:path=/ffzdqu-48/?prefix=obj5 2023-09-22T21:23:08.595 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj5 2023-09-22T21:23:08.595 DEBUG:boto:Method: GET 2023-09-22T21:23:08.595 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj5 2023-09-22T21:23:08.595 DEBUG:boto:Data: 2023-09-22T21:23:08.596 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.596 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.596 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.596 DEBUG:boto:Params: {} 2023-09-22T21:23:08.596 DEBUG:boto:Token: None 2023-09-22T21:23:08.596 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/ 2023-09-22T21:23:08.596 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA= 2023-09-22T21:23:08.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA=', 'Content-Length': '0'} 2023-09-22T21:23:08.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d4e210b937f2197-00650e05bc-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.601 DEBUG:boto:b'ffzdqu-48obj51000falseobj52023-09-22T21:22:41.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:08.602 DEBUG:boto:path=/ffzdqu-48/obj5 2023-09-22T21:23:08.602 DEBUG:boto:auth_path=/ffzdqu-48/obj5 2023-09-22T21:23:08.602 DEBUG:boto:Method: HEAD 2023-09-22T21:23:08.602 DEBUG:boto:Path: /ffzdqu-48/obj5 2023-09-22T21:23:08.602 DEBUG:boto:Data: 2023-09-22T21:23:08.602 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.602 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.602 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.603 DEBUG:boto:Params: {} 2023-09-22T21:23:08.603 DEBUG:boto:Token: None 2023-09-22T21:23:08.603 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj5 2023-09-22T21:23:08.603 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4CLdxbs85end+w3hvtNvDvNzVXQ= 2023-09-22T21:23:08.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4CLdxbs85end+w3hvtNvDvNzVXQ=', 'Content-Length': '0'} 2023-09-22T21:23:08.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000378a35fec795d4fa-00650e05bc-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.606 DEBUG:boto:path=/ffzdqu-48/obj5 2023-09-22T21:23:08.606 DEBUG:boto:auth_path=/ffzdqu-48/obj5 2023-09-22T21:23:08.606 DEBUG:boto:Method: HEAD 2023-09-22T21:23:08.607 DEBUG:boto:Path: /ffzdqu-48/obj5 2023-09-22T21:23:08.607 DEBUG:boto:Data: 2023-09-22T21:23:08.607 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.607 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.607 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.607 DEBUG:boto:Params: {} 2023-09-22T21:23:08.607 DEBUG:boto:Token: None 2023-09-22T21:23:08.607 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj5 2023-09-22T21:23:08.607 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4CLdxbs85end+w3hvtNvDvNzVXQ= 2023-09-22T21:23:08.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4CLdxbs85end+w3hvtNvDvNzVXQ=', 'Content-Length': '0'} 2023-09-22T21:23:08.610 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef2b5b325236fb8f-00650e05bc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-22T21:23:08.611 DEBUG:boto:path=/ffzdqu-48/obj5 2023-09-22T21:23:08.611 DEBUG:boto:auth_path=/ffzdqu-48/obj5 2023-09-22T21:23:08.611 DEBUG:boto:Method: GET 2023-09-22T21:23:08.611 DEBUG:boto:Path: /ffzdqu-48/obj5 2023-09-22T21:23:08.611 DEBUG:boto:Data: 2023-09-22T21:23:08.611 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.611 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.611 DEBUG:boto:Params: {} 2023-09-22T21:23:08.611 DEBUG:boto:Token: None 2023-09-22T21:23:08.612 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj5 2023-09-22T21:23:08.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YnnVO/jX3JtAwtHWq4vHAnWpj6E= 2023-09-22T21:23:08.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YnnVO/jX3JtAwtHWq4vHAnWpj6E=', 'Content-Length': '0'} 2023-09-22T21:23:08.615 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fda91f71ddfe4347-00650e05bc-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.659 DEBUG:boto:path=/ffzdqu-48/obj5 2023-09-22T21:23:08.659 DEBUG:boto:auth_path=/ffzdqu-48/obj5 2023-09-22T21:23:08.660 DEBUG:boto:Method: GET 2023-09-22T21:23:08.660 DEBUG:boto:Path: /ffzdqu-48/obj5 2023-09-22T21:23:08.660 DEBUG:boto:Data: 2023-09-22T21:23:08.660 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.660 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.660 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.660 DEBUG:boto:Params: {} 2023-09-22T21:23:08.660 DEBUG:boto:Token: None 2023-09-22T21:23:08.661 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj5 2023-09-22T21:23:08.661 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YnnVO/jX3JtAwtHWq4vHAnWpj6E= 2023-09-22T21:23:08.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YnnVO/jX3JtAwtHWq4vHAnWpj6E=', 'Content-Length': '0'} 2023-09-22T21:23:08.663 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b242e07ac0369898-00650e05bc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.664 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:08.664 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:08.664 DEBUG:boto:path=/ffzdqu-48/?prefix=obj5 2023-09-22T21:23:08.664 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj5 2023-09-22T21:23:08.664 DEBUG:boto:Method: GET 2023-09-22T21:23:08.664 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj5 2023-09-22T21:23:08.664 DEBUG:boto:Data: 2023-09-22T21:23:08.665 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.665 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.665 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.665 DEBUG:boto:Params: {} 2023-09-22T21:23:08.665 DEBUG:boto:Token: None 2023-09-22T21:23:08.665 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/ 2023-09-22T21:23:08.665 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA= 2023-09-22T21:23:08.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA=', 'Content-Length': '0'} 2023-09-22T21:23:08.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f0f9fca7949318b-00650e05bc-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.670 DEBUG:boto:b'ffzdqu-48obj51000falseobj52023-09-22T21:22:41.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:08.671 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:08.671 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:08.671 DEBUG:boto:path=/ffzdqu-48/?prefix=obj5 2023-09-22T21:23:08.671 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj5 2023-09-22T21:23:08.671 DEBUG:boto:Method: GET 2023-09-22T21:23:08.671 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj5 2023-09-22T21:23:08.671 DEBUG:boto:Data: 2023-09-22T21:23:08.672 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.672 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.672 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.672 DEBUG:boto:Params: {} 2023-09-22T21:23:08.672 DEBUG:boto:Token: None 2023-09-22T21:23:08.672 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/ 2023-09-22T21:23:08.672 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA= 2023-09-22T21:23:08.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA=', 'Content-Length': '0'} 2023-09-22T21:23:08.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009296b82cffd2a5e9-00650e05bc-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.722 DEBUG:boto:b'ffzdqu-48obj51000falseobj52023-09-22T21:22:41.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:08.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-22T21:23:08.723 DEBUG:boto:path=/ffzdqu-48/obj6 2023-09-22T21:23:08.723 DEBUG:boto:auth_path=/ffzdqu-48/obj6 2023-09-22T21:23:08.723 DEBUG:boto:path=/ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.723 DEBUG:boto:auth_path=/ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.723 DEBUG:boto:Method: GET 2023-09-22T21:23:08.724 DEBUG:boto:Path: /ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.724 DEBUG:boto:Data: 2023-09-22T21:23:08.724 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.724 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.724 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.724 DEBUG:boto:Params: {} 2023-09-22T21:23:08.724 DEBUG:boto:Token: None 2023-09-22T21:23:08.724 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.725 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G9sJCHTWdwjXjWGCvVuY1Es8/2w= 2023-09-22T21:23:08.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G9sJCHTWdwjXjWGCvVuY1Es8/2w=', 'Content-Length': '0'} 2023-09-22T21:23:08.728 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000440498eb839602f0-00650e05bc-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.770 DEBUG:boto:path=/ffzdqu-48/obj6 2023-09-22T21:23:08.771 DEBUG:boto:auth_path=/ffzdqu-48/obj6 2023-09-22T21:23:08.771 DEBUG:boto:path=/ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.771 DEBUG:boto:auth_path=/ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.771 DEBUG:boto:Method: GET 2023-09-22T21:23:08.771 DEBUG:boto:Path: /ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.771 DEBUG:boto:Data: 2023-09-22T21:23:08.771 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.771 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.771 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.771 DEBUG:boto:Params: {} 2023-09-22T21:23:08.772 DEBUG:boto:Token: None 2023-09-22T21:23:08.772 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.772 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G9sJCHTWdwjXjWGCvVuY1Es8/2w= 2023-09-22T21:23:08.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G9sJCHTWdwjXjWGCvVuY1Es8/2w=', 'Content-Length': '0'} 2023-09-22T21:23:08.775 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1c5f656b04b42a2-00650e05bc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.776 DEBUG:boto:path=/ffzdqu-48/obj6 2023-09-22T21:23:08.776 DEBUG:boto:auth_path=/ffzdqu-48/obj6 2023-09-22T21:23:08.776 DEBUG:boto:path=/ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.776 DEBUG:boto:auth_path=/ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.776 DEBUG:boto:Method: HEAD 2023-09-22T21:23:08.776 DEBUG:boto:Path: /ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.776 DEBUG:boto:Data: 2023-09-22T21:23:08.776 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.776 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.777 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.777 DEBUG:boto:Params: {} 2023-09-22T21:23:08.777 DEBUG:boto:Token: None 2023-09-22T21:23:08.777 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.777 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r/Ugcsjo23LJvV6BkotmVmFkVko= 2023-09-22T21:23:08.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r/Ugcsjo23LJvV6BkotmVmFkVko=', 'Content-Length': '0'} 2023-09-22T21:23:08.780 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0e3b8e529746a6b-00650e05bc-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.780 DEBUG:boto:path=/ffzdqu-48/obj6 2023-09-22T21:23:08.780 DEBUG:boto:auth_path=/ffzdqu-48/obj6 2023-09-22T21:23:08.780 DEBUG:boto:path=/ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.780 DEBUG:boto:auth_path=/ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.780 DEBUG:boto:Method: HEAD 2023-09-22T21:23:08.781 DEBUG:boto:Path: /ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.781 DEBUG:boto:Data: 2023-09-22T21:23:08.781 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.781 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.781 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.781 DEBUG:boto:Params: {} 2023-09-22T21:23:08.781 DEBUG:boto:Token: None 2023-09-22T21:23:08.781 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.781 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r/Ugcsjo23LJvV6BkotmVmFkVko= 2023-09-22T21:23:08.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r/Ugcsjo23LJvV6BkotmVmFkVko=', 'Content-Length': '0'} 2023-09-22T21:23:08.784 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032214c7807c23967-00650e05bc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-22T21:23:08.785 DEBUG:boto:path=/ffzdqu-48/obj6 2023-09-22T21:23:08.785 DEBUG:boto:auth_path=/ffzdqu-48/obj6 2023-09-22T21:23:08.785 DEBUG:boto:path=/ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.785 DEBUG:boto:auth_path=/ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.785 DEBUG:boto:Method: GET 2023-09-22T21:23:08.785 DEBUG:boto:Path: /ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.785 DEBUG:boto:Data: 2023-09-22T21:23:08.785 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.785 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.785 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.786 DEBUG:boto:Params: {} 2023-09-22T21:23:08.786 DEBUG:boto:Token: None 2023-09-22T21:23:08.786 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.786 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G9sJCHTWdwjXjWGCvVuY1Es8/2w= 2023-09-22T21:23:08.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G9sJCHTWdwjXjWGCvVuY1Es8/2w=', 'Content-Length': '0'} 2023-09-22T21:23:08.789 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000845530a1d8850676-00650e05bc-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.834 DEBUG:boto:path=/ffzdqu-48/obj6 2023-09-22T21:23:08.834 DEBUG:boto:auth_path=/ffzdqu-48/obj6 2023-09-22T21:23:08.835 DEBUG:boto:path=/ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.835 DEBUG:boto:auth_path=/ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.835 DEBUG:boto:Method: GET 2023-09-22T21:23:08.835 DEBUG:boto:Path: /ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.835 DEBUG:boto:Data: 2023-09-22T21:23:08.835 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.835 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.835 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.835 DEBUG:boto:Params: {} 2023-09-22T21:23:08.836 DEBUG:boto:Token: None 2023-09-22T21:23:08.836 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj6?versionId=null 2023-09-22T21:23:08.836 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G9sJCHTWdwjXjWGCvVuY1Es8/2w= 2023-09-22T21:23:08.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G9sJCHTWdwjXjWGCvVuY1Es8/2w=', 'Content-Length': '0'} 2023-09-22T21:23:08.839 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f4fc3863e1d6eae-00650e05bc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.839 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:08.839 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:08.839 DEBUG:boto:path=/ffzdqu-48/?prefix=obj6 2023-09-22T21:23:08.839 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj6 2023-09-22T21:23:08.840 DEBUG:boto:Method: GET 2023-09-22T21:23:08.840 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj6 2023-09-22T21:23:08.840 DEBUG:boto:Data: 2023-09-22T21:23:08.840 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.840 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.840 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.840 DEBUG:boto:Params: {} 2023-09-22T21:23:08.841 DEBUG:boto:Token: None 2023-09-22T21:23:08.841 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/ 2023-09-22T21:23:08.841 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA= 2023-09-22T21:23:08.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA=', 'Content-Length': '0'} 2023-09-22T21:23:08.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038b15a45a476a7af-00650e05bc-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.886 DEBUG:boto:b'ffzdqu-48obj61000falseobj62023-09-22T21:22:41.619Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:08.887 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:08.887 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:08.887 DEBUG:boto:path=/ffzdqu-48/?prefix=obj6 2023-09-22T21:23:08.887 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj6 2023-09-22T21:23:08.888 DEBUG:boto:Method: GET 2023-09-22T21:23:08.888 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj6 2023-09-22T21:23:08.888 DEBUG:boto:Data: 2023-09-22T21:23:08.888 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.888 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.889 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.889 DEBUG:boto:Params: {} 2023-09-22T21:23:08.889 DEBUG:boto:Token: None 2023-09-22T21:23:08.889 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/ 2023-09-22T21:23:08.889 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA= 2023-09-22T21:23:08.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA=', 'Content-Length': '0'} 2023-09-22T21:23:08.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2c8c16f6906bd4e-00650e05bc-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.895 DEBUG:boto:b'ffzdqu-48obj61000falseobj62023-09-22T21:22:41.619Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:08.895 DEBUG:boto:path=/ffzdqu-48/obj6 2023-09-22T21:23:08.896 DEBUG:boto:auth_path=/ffzdqu-48/obj6 2023-09-22T21:23:08.896 DEBUG:boto:Method: HEAD 2023-09-22T21:23:08.896 DEBUG:boto:Path: /ffzdqu-48/obj6 2023-09-22T21:23:08.896 DEBUG:boto:Data: 2023-09-22T21:23:08.896 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.896 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.896 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.896 DEBUG:boto:Params: {} 2023-09-22T21:23:08.896 DEBUG:boto:Token: None 2023-09-22T21:23:08.897 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj6 2023-09-22T21:23:08.897 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PGkVkEJKdnzSxviA1E/O1cmqALc= 2023-09-22T21:23:08.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PGkVkEJKdnzSxviA1E/O1cmqALc=', 'Content-Length': '0'} 2023-09-22T21:23:08.899 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043bd2ef5e3b87f8a-00650e05bc-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.900 DEBUG:boto:path=/ffzdqu-48/obj6 2023-09-22T21:23:08.900 DEBUG:boto:auth_path=/ffzdqu-48/obj6 2023-09-22T21:23:08.900 DEBUG:boto:Method: HEAD 2023-09-22T21:23:08.900 DEBUG:boto:Path: /ffzdqu-48/obj6 2023-09-22T21:23:08.900 DEBUG:boto:Data: 2023-09-22T21:23:08.901 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.901 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.901 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.901 DEBUG:boto:Params: {} 2023-09-22T21:23:08.901 DEBUG:boto:Token: None 2023-09-22T21:23:08.901 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj6 2023-09-22T21:23:08.901 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PGkVkEJKdnzSxviA1E/O1cmqALc= 2023-09-22T21:23:08.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PGkVkEJKdnzSxviA1E/O1cmqALc=', 'Content-Length': '0'} 2023-09-22T21:23:08.904 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004db86040508e2a76-00650e05bc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-22T21:23:08.905 DEBUG:boto:path=/ffzdqu-48/obj6 2023-09-22T21:23:08.905 DEBUG:boto:auth_path=/ffzdqu-48/obj6 2023-09-22T21:23:08.905 DEBUG:boto:Method: GET 2023-09-22T21:23:08.905 DEBUG:boto:Path: /ffzdqu-48/obj6 2023-09-22T21:23:08.905 DEBUG:boto:Data: 2023-09-22T21:23:08.906 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.906 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.906 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.906 DEBUG:boto:Params: {} 2023-09-22T21:23:08.906 DEBUG:boto:Token: None 2023-09-22T21:23:08.906 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj6 2023-09-22T21:23:08.906 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q+ZeyVvYnjsLLmM4z5mF3i9EL4k= 2023-09-22T21:23:08.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q+ZeyVvYnjsLLmM4z5mF3i9EL4k=', 'Content-Length': '0'} 2023-09-22T21:23:08.909 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000cb24d9a6940ccd5-00650e05bc-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.954 DEBUG:boto:path=/ffzdqu-48/obj6 2023-09-22T21:23:08.955 DEBUG:boto:auth_path=/ffzdqu-48/obj6 2023-09-22T21:23:08.955 DEBUG:boto:Method: GET 2023-09-22T21:23:08.955 DEBUG:boto:Path: /ffzdqu-48/obj6 2023-09-22T21:23:08.955 DEBUG:boto:Data: 2023-09-22T21:23:08.955 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.955 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:08.955 DEBUG:boto:Port: 8000 2023-09-22T21:23:08.956 DEBUG:boto:Params: {} 2023-09-22T21:23:08.956 DEBUG:boto:Token: None 2023-09-22T21:23:08.956 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/obj6 2023-09-22T21:23:08.956 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q+ZeyVvYnjsLLmM4z5mF3i9EL4k= 2023-09-22T21:23:08.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q+ZeyVvYnjsLLmM4z5mF3i9EL4k=', 'Content-Length': '0'} 2023-09-22T21:23:08.959 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b36654db3929d4c-00650e05bc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:08.959 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:08.960 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:08.960 DEBUG:boto:path=/ffzdqu-48/?prefix=obj6 2023-09-22T21:23:08.960 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj6 2023-09-22T21:23:08.960 DEBUG:boto:Method: GET 2023-09-22T21:23:08.960 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj6 2023-09-22T21:23:08.960 DEBUG:boto:Data: 2023-09-22T21:23:08.960 DEBUG:boto:Headers: {} 2023-09-22T21:23:08.961 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:08.961 DEBUG:boto:Port: 8001 2023-09-22T21:23:08.961 DEBUG:boto:Params: {} 2023-09-22T21:23:08.961 DEBUG:boto:Token: None 2023-09-22T21:23:08.961 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:08 GMT /ffzdqu-48/ 2023-09-22T21:23:08.961 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA= 2023-09-22T21:23:08.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYwwQ0hS1fDbr6R4EeAISlWj5lA=', 'Content-Length': '0'} 2023-09-22T21:23:08.965 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010b4edbb69f3fd7c-00650e05bc-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.006 DEBUG:boto:b'ffzdqu-48obj61000falseobj62023-09-22T21:22:41.619Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:09.007 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:09.007 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:09.007 DEBUG:boto:path=/ffzdqu-48/?prefix=obj6 2023-09-22T21:23:09.007 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj6 2023-09-22T21:23:09.007 DEBUG:boto:Method: GET 2023-09-22T21:23:09.007 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj6 2023-09-22T21:23:09.007 DEBUG:boto:Data: 2023-09-22T21:23:09.008 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.008 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.008 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.008 DEBUG:boto:Params: {} 2023-09-22T21:23:09.008 DEBUG:boto:Token: None 2023-09-22T21:23:09.008 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/ 2023-09-22T21:23:09.008 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2+4C2giYRjwTYW7B1TrkPRU7DHA= 2023-09-22T21:23:09.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2+4C2giYRjwTYW7B1TrkPRU7DHA=', 'Content-Length': '0'} 2023-09-22T21:23:09.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096a22445837072a4-00650e05bd-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.013 DEBUG:boto:b'ffzdqu-48obj61000falseobj62023-09-22T21:22:41.619Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:09.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-22T21:23:09.014 DEBUG:boto:path=/ffzdqu-48/obj7 2023-09-22T21:23:09.014 DEBUG:boto:auth_path=/ffzdqu-48/obj7 2023-09-22T21:23:09.014 DEBUG:boto:path=/ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.014 DEBUG:boto:auth_path=/ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.014 DEBUG:boto:Method: GET 2023-09-22T21:23:09.015 DEBUG:boto:Path: /ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.015 DEBUG:boto:Data: 2023-09-22T21:23:09.015 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.015 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.015 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.015 DEBUG:boto:Params: {} 2023-09-22T21:23:09.015 DEBUG:boto:Token: None 2023-09-22T21:23:09.015 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.015 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cgiiVHSf7CF/UfZ/0fZmf02BrOE= 2023-09-22T21:23:09.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cgiiVHSf7CF/UfZ/0fZmf02BrOE=', 'Content-Length': '0'} 2023-09-22T21:23:09.018 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9bb067da465ca3f-00650e05bd-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.019 DEBUG:boto:path=/ffzdqu-48/obj7 2023-09-22T21:23:09.019 DEBUG:boto:auth_path=/ffzdqu-48/obj7 2023-09-22T21:23:09.019 DEBUG:boto:path=/ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.019 DEBUG:boto:auth_path=/ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.019 DEBUG:boto:Method: GET 2023-09-22T21:23:09.019 DEBUG:boto:Path: /ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.019 DEBUG:boto:Data: 2023-09-22T21:23:09.019 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.020 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.020 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.020 DEBUG:boto:Params: {} 2023-09-22T21:23:09.020 DEBUG:boto:Token: None 2023-09-22T21:23:09.020 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.020 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cgiiVHSf7CF/UfZ/0fZmf02BrOE= 2023-09-22T21:23:09.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cgiiVHSf7CF/UfZ/0fZmf02BrOE=', 'Content-Length': '0'} 2023-09-22T21:23:09.023 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007176eb6fa6dbd95b-00650e05bd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.066 DEBUG:boto:path=/ffzdqu-48/obj7 2023-09-22T21:23:09.066 DEBUG:boto:auth_path=/ffzdqu-48/obj7 2023-09-22T21:23:09.067 DEBUG:boto:path=/ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.067 DEBUG:boto:auth_path=/ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.067 DEBUG:boto:Method: HEAD 2023-09-22T21:23:09.067 DEBUG:boto:Path: /ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.067 DEBUG:boto:Data: 2023-09-22T21:23:09.067 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.067 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.067 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.067 DEBUG:boto:Params: {} 2023-09-22T21:23:09.067 DEBUG:boto:Token: None 2023-09-22T21:23:09.068 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.068 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lWlVopPzG3rtZyCj3++kbyzHXOo= 2023-09-22T21:23:09.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lWlVopPzG3rtZyCj3++kbyzHXOo=', 'Content-Length': '0'} 2023-09-22T21:23:09.070 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005309e16a8f36040e-00650e05bd-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.071 DEBUG:boto:path=/ffzdqu-48/obj7 2023-09-22T21:23:09.071 DEBUG:boto:auth_path=/ffzdqu-48/obj7 2023-09-22T21:23:09.071 DEBUG:boto:path=/ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.071 DEBUG:boto:auth_path=/ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.071 DEBUG:boto:Method: HEAD 2023-09-22T21:23:09.071 DEBUG:boto:Path: /ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.072 DEBUG:boto:Data: 2023-09-22T21:23:09.072 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.072 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.072 DEBUG:boto:Params: {} 2023-09-22T21:23:09.072 DEBUG:boto:Token: None 2023-09-22T21:23:09.072 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.072 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lWlVopPzG3rtZyCj3++kbyzHXOo= 2023-09-22T21:23:09.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lWlVopPzG3rtZyCj3++kbyzHXOo=', 'Content-Length': '0'} 2023-09-22T21:23:09.075 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2177e2e405facee-00650e05bd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-22T21:23:09.075 DEBUG:boto:path=/ffzdqu-48/obj7 2023-09-22T21:23:09.076 DEBUG:boto:auth_path=/ffzdqu-48/obj7 2023-09-22T21:23:09.076 DEBUG:boto:path=/ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.076 DEBUG:boto:auth_path=/ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.076 DEBUG:boto:Method: GET 2023-09-22T21:23:09.076 DEBUG:boto:Path: /ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.076 DEBUG:boto:Data: 2023-09-22T21:23:09.076 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.076 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.076 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.076 DEBUG:boto:Params: {} 2023-09-22T21:23:09.077 DEBUG:boto:Token: None 2023-09-22T21:23:09.077 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.077 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cgiiVHSf7CF/UfZ/0fZmf02BrOE= 2023-09-22T21:23:09.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cgiiVHSf7CF/UfZ/0fZmf02BrOE=', 'Content-Length': '0'} 2023-09-22T21:23:09.079 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ff308eff9d5c492-00650e05bd-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.122 DEBUG:boto:path=/ffzdqu-48/obj7 2023-09-22T21:23:09.123 DEBUG:boto:auth_path=/ffzdqu-48/obj7 2023-09-22T21:23:09.123 DEBUG:boto:path=/ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.123 DEBUG:boto:auth_path=/ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.123 DEBUG:boto:Method: GET 2023-09-22T21:23:09.123 DEBUG:boto:Path: /ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.123 DEBUG:boto:Data: 2023-09-22T21:23:09.123 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.123 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.124 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.124 DEBUG:boto:Params: {} 2023-09-22T21:23:09.124 DEBUG:boto:Token: None 2023-09-22T21:23:09.124 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/obj7?versionId=null 2023-09-22T21:23:09.124 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cgiiVHSf7CF/UfZ/0fZmf02BrOE= 2023-09-22T21:23:09.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cgiiVHSf7CF/UfZ/0fZmf02BrOE=', 'Content-Length': '0'} 2023-09-22T21:23:09.127 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000554ca0d3de74575f-00650e05bd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.127 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:09.127 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:09.127 DEBUG:boto:path=/ffzdqu-48/?prefix=obj7 2023-09-22T21:23:09.127 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj7 2023-09-22T21:23:09.128 DEBUG:boto:Method: GET 2023-09-22T21:23:09.128 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj7 2023-09-22T21:23:09.128 DEBUG:boto:Data: 2023-09-22T21:23:09.128 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.128 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.128 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.128 DEBUG:boto:Params: {} 2023-09-22T21:23:09.128 DEBUG:boto:Token: None 2023-09-22T21:23:09.128 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/ 2023-09-22T21:23:09.129 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2+4C2giYRjwTYW7B1TrkPRU7DHA= 2023-09-22T21:23:09.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2+4C2giYRjwTYW7B1TrkPRU7DHA=', 'Content-Length': '0'} 2023-09-22T21:23:09.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2e739c06966a5c8-00650e05bd-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.174 DEBUG:boto:b'ffzdqu-48obj71000falseobj72023-09-22T21:22:41.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:09.175 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:09.175 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:09.175 DEBUG:boto:path=/ffzdqu-48/?prefix=obj7 2023-09-22T21:23:09.175 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj7 2023-09-22T21:23:09.175 DEBUG:boto:Method: GET 2023-09-22T21:23:09.175 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj7 2023-09-22T21:23:09.175 DEBUG:boto:Data: 2023-09-22T21:23:09.176 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.176 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.176 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.176 DEBUG:boto:Params: {} 2023-09-22T21:23:09.176 DEBUG:boto:Token: None 2023-09-22T21:23:09.176 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/ 2023-09-22T21:23:09.176 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2+4C2giYRjwTYW7B1TrkPRU7DHA= 2023-09-22T21:23:09.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2+4C2giYRjwTYW7B1TrkPRU7DHA=', 'Content-Length': '0'} 2023-09-22T21:23:09.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000352e1c20f7b9be62-00650e05bd-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.181 DEBUG:boto:b'ffzdqu-48obj71000falseobj72023-09-22T21:22:41.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:09.181 DEBUG:boto:path=/ffzdqu-48/obj7 2023-09-22T21:23:09.181 DEBUG:boto:auth_path=/ffzdqu-48/obj7 2023-09-22T21:23:09.181 DEBUG:boto:Method: HEAD 2023-09-22T21:23:09.182 DEBUG:boto:Path: /ffzdqu-48/obj7 2023-09-22T21:23:09.182 DEBUG:boto:Data: 2023-09-22T21:23:09.182 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.182 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.182 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.182 DEBUG:boto:Params: {} 2023-09-22T21:23:09.182 DEBUG:boto:Token: None 2023-09-22T21:23:09.182 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/obj7 2023-09-22T21:23:09.183 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h+t1O2jQ32uvdJr6fiDGlbl5iOk= 2023-09-22T21:23:09.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h+t1O2jQ32uvdJr6fiDGlbl5iOk=', 'Content-Length': '0'} 2023-09-22T21:23:09.185 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e8e7e4b9fd27b7f-00650e05bd-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.185 DEBUG:boto:path=/ffzdqu-48/obj7 2023-09-22T21:23:09.186 DEBUG:boto:auth_path=/ffzdqu-48/obj7 2023-09-22T21:23:09.186 DEBUG:boto:Method: HEAD 2023-09-22T21:23:09.186 DEBUG:boto:Path: /ffzdqu-48/obj7 2023-09-22T21:23:09.186 DEBUG:boto:Data: 2023-09-22T21:23:09.186 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.186 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.187 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.187 DEBUG:boto:Params: {} 2023-09-22T21:23:09.187 DEBUG:boto:Token: None 2023-09-22T21:23:09.187 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/obj7 2023-09-22T21:23:09.187 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h+t1O2jQ32uvdJr6fiDGlbl5iOk= 2023-09-22T21:23:09.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h+t1O2jQ32uvdJr6fiDGlbl5iOk=', 'Content-Length': '0'} 2023-09-22T21:23:09.190 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000880f19e38e9f3d5d-00650e05bd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-22T21:23:09.190 DEBUG:boto:path=/ffzdqu-48/obj7 2023-09-22T21:23:09.190 DEBUG:boto:auth_path=/ffzdqu-48/obj7 2023-09-22T21:23:09.190 DEBUG:boto:Method: GET 2023-09-22T21:23:09.191 DEBUG:boto:Path: /ffzdqu-48/obj7 2023-09-22T21:23:09.191 DEBUG:boto:Data: 2023-09-22T21:23:09.191 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.191 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.191 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.191 DEBUG:boto:Params: {} 2023-09-22T21:23:09.191 DEBUG:boto:Token: None 2023-09-22T21:23:09.191 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/obj7 2023-09-22T21:23:09.192 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:N04/xxLMnru4gwQYXd7Yp2NI4OE= 2023-09-22T21:23:09.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:N04/xxLMnru4gwQYXd7Yp2NI4OE=', 'Content-Length': '0'} 2023-09-22T21:23:09.194 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067a993c1bda26d01-00650e05bd-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.238 DEBUG:boto:path=/ffzdqu-48/obj7 2023-09-22T21:23:09.239 DEBUG:boto:auth_path=/ffzdqu-48/obj7 2023-09-22T21:23:09.239 DEBUG:boto:Method: GET 2023-09-22T21:23:09.239 DEBUG:boto:Path: /ffzdqu-48/obj7 2023-09-22T21:23:09.240 DEBUG:boto:Data: 2023-09-22T21:23:09.240 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.240 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.240 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.240 DEBUG:boto:Params: {} 2023-09-22T21:23:09.241 DEBUG:boto:Token: None 2023-09-22T21:23:09.241 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/obj7 2023-09-22T21:23:09.241 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:N04/xxLMnru4gwQYXd7Yp2NI4OE= 2023-09-22T21:23:09.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:N04/xxLMnru4gwQYXd7Yp2NI4OE=', 'Content-Length': '0'} 2023-09-22T21:23:09.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb68dbb354bf29cf-00650e05bd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.245 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:09.246 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:09.246 DEBUG:boto:path=/ffzdqu-48/?prefix=obj7 2023-09-22T21:23:09.246 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj7 2023-09-22T21:23:09.246 DEBUG:boto:Method: GET 2023-09-22T21:23:09.246 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj7 2023-09-22T21:23:09.247 DEBUG:boto:Data: 2023-09-22T21:23:09.247 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.247 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.247 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.248 DEBUG:boto:Params: {} 2023-09-22T21:23:09.248 DEBUG:boto:Token: None 2023-09-22T21:23:09.248 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/ 2023-09-22T21:23:09.248 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2+4C2giYRjwTYW7B1TrkPRU7DHA= 2023-09-22T21:23:09.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2+4C2giYRjwTYW7B1TrkPRU7DHA=', 'Content-Length': '0'} 2023-09-22T21:23:09.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a75efab7fbe740d-00650e05bd-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.254 DEBUG:boto:b'ffzdqu-48obj71000falseobj72023-09-22T21:22:41.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:09.255 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:09.255 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:09.255 DEBUG:boto:path=/ffzdqu-48/?prefix=obj7 2023-09-22T21:23:09.255 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj7 2023-09-22T21:23:09.255 DEBUG:boto:Method: GET 2023-09-22T21:23:09.255 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj7 2023-09-22T21:23:09.256 DEBUG:boto:Data: 2023-09-22T21:23:09.256 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.256 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.256 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.256 DEBUG:boto:Params: {} 2023-09-22T21:23:09.256 DEBUG:boto:Token: None 2023-09-22T21:23:09.257 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/ 2023-09-22T21:23:09.257 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2+4C2giYRjwTYW7B1TrkPRU7DHA= 2023-09-22T21:23:09.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2+4C2giYRjwTYW7B1TrkPRU7DHA=', 'Content-Length': '0'} 2023-09-22T21:23:09.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057e31bff9efb3719-00650e05bd-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.262 DEBUG:boto:b'ffzdqu-48obj71000falseobj72023-09-22T21:22:41.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:09.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-22T21:23:09.265 DEBUG:boto:path=/ffzdqu-48/obj8 2023-09-22T21:23:09.265 DEBUG:boto:auth_path=/ffzdqu-48/obj8 2023-09-22T21:23:09.265 DEBUG:boto:path=/ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.265 DEBUG:boto:auth_path=/ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.265 DEBUG:boto:Method: GET 2023-09-22T21:23:09.266 DEBUG:boto:Path: /ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.266 DEBUG:boto:Data: 2023-09-22T21:23:09.266 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.266 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.266 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.266 DEBUG:boto:Params: {} 2023-09-22T21:23:09.266 DEBUG:boto:Token: None 2023-09-22T21:23:09.266 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.267 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jRTBQ5ehxg7rFaJs9D6Gl0+uPYg= 2023-09-22T21:23:09.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jRTBQ5ehxg7rFaJs9D6Gl0+uPYg=', 'Content-Length': '0'} 2023-09-22T21:23:09.269 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000efb92db40dee1a0f-00650e05bd-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.310 DEBUG:boto:path=/ffzdqu-48/obj8 2023-09-22T21:23:09.311 DEBUG:boto:auth_path=/ffzdqu-48/obj8 2023-09-22T21:23:09.311 DEBUG:boto:path=/ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.311 DEBUG:boto:auth_path=/ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.311 DEBUG:boto:Method: GET 2023-09-22T21:23:09.311 DEBUG:boto:Path: /ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.311 DEBUG:boto:Data: 2023-09-22T21:23:09.311 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.311 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.311 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.312 DEBUG:boto:Params: {} 2023-09-22T21:23:09.312 DEBUG:boto:Token: None 2023-09-22T21:23:09.312 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.312 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jRTBQ5ehxg7rFaJs9D6Gl0+uPYg= 2023-09-22T21:23:09.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jRTBQ5ehxg7rFaJs9D6Gl0+uPYg=', 'Content-Length': '0'} 2023-09-22T21:23:09.315 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b74e9f9938687816-00650e05bd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.315 DEBUG:boto:path=/ffzdqu-48/obj8 2023-09-22T21:23:09.315 DEBUG:boto:auth_path=/ffzdqu-48/obj8 2023-09-22T21:23:09.315 DEBUG:boto:path=/ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.316 DEBUG:boto:auth_path=/ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.316 DEBUG:boto:Method: HEAD 2023-09-22T21:23:09.316 DEBUG:boto:Path: /ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.316 DEBUG:boto:Data: 2023-09-22T21:23:09.316 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.316 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.316 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.316 DEBUG:boto:Params: {} 2023-09-22T21:23:09.316 DEBUG:boto:Token: None 2023-09-22T21:23:09.317 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.317 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:U+G1Lrbv9Mibh4UmaqK8MhYCui4= 2023-09-22T21:23:09.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:U+G1Lrbv9Mibh4UmaqK8MhYCui4=', 'Content-Length': '0'} 2023-09-22T21:23:09.319 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000474553ae93ddcba4-00650e05bd-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.320 DEBUG:boto:path=/ffzdqu-48/obj8 2023-09-22T21:23:09.320 DEBUG:boto:auth_path=/ffzdqu-48/obj8 2023-09-22T21:23:09.320 DEBUG:boto:path=/ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.320 DEBUG:boto:auth_path=/ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.320 DEBUG:boto:Method: HEAD 2023-09-22T21:23:09.321 DEBUG:boto:Path: /ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.321 DEBUG:boto:Data: 2023-09-22T21:23:09.321 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.321 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.321 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.321 DEBUG:boto:Params: {} 2023-09-22T21:23:09.322 DEBUG:boto:Token: None 2023-09-22T21:23:09.322 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.322 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:U+G1Lrbv9Mibh4UmaqK8MhYCui4= 2023-09-22T21:23:09.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:U+G1Lrbv9Mibh4UmaqK8MhYCui4=', 'Content-Length': '0'} 2023-09-22T21:23:09.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d2afa08cc613df2-00650e05bd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-22T21:23:09.326 DEBUG:boto:path=/ffzdqu-48/obj8 2023-09-22T21:23:09.326 DEBUG:boto:auth_path=/ffzdqu-48/obj8 2023-09-22T21:23:09.326 DEBUG:boto:path=/ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.326 DEBUG:boto:auth_path=/ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.327 DEBUG:boto:Method: GET 2023-09-22T21:23:09.327 DEBUG:boto:Path: /ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.327 DEBUG:boto:Data: 2023-09-22T21:23:09.327 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.327 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.327 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.327 DEBUG:boto:Params: {} 2023-09-22T21:23:09.328 DEBUG:boto:Token: None 2023-09-22T21:23:09.328 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.328 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jRTBQ5ehxg7rFaJs9D6Gl0+uPYg= 2023-09-22T21:23:09.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jRTBQ5ehxg7rFaJs9D6Gl0+uPYg=', 'Content-Length': '0'} 2023-09-22T21:23:09.331 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070c88a1db8a6c0f4-00650e05bd-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.374 DEBUG:boto:path=/ffzdqu-48/obj8 2023-09-22T21:23:09.375 DEBUG:boto:auth_path=/ffzdqu-48/obj8 2023-09-22T21:23:09.375 DEBUG:boto:path=/ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.375 DEBUG:boto:auth_path=/ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.375 DEBUG:boto:Method: GET 2023-09-22T21:23:09.375 DEBUG:boto:Path: /ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.375 DEBUG:boto:Data: 2023-09-22T21:23:09.375 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.375 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.375 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.376 DEBUG:boto:Params: {} 2023-09-22T21:23:09.376 DEBUG:boto:Token: None 2023-09-22T21:23:09.376 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/obj8?versionId=null 2023-09-22T21:23:09.376 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jRTBQ5ehxg7rFaJs9D6Gl0+uPYg= 2023-09-22T21:23:09.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jRTBQ5ehxg7rFaJs9D6Gl0+uPYg=', 'Content-Length': '0'} 2023-09-22T21:23:09.379 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000958a3400f291f038-00650e05bd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.379 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:09.379 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:09.379 DEBUG:boto:path=/ffzdqu-48/?prefix=obj8 2023-09-22T21:23:09.380 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj8 2023-09-22T21:23:09.380 DEBUG:boto:Method: GET 2023-09-22T21:23:09.380 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj8 2023-09-22T21:23:09.380 DEBUG:boto:Data: 2023-09-22T21:23:09.380 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.380 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.380 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.380 DEBUG:boto:Params: {} 2023-09-22T21:23:09.380 DEBUG:boto:Token: None 2023-09-22T21:23:09.381 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/ 2023-09-22T21:23:09.381 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2+4C2giYRjwTYW7B1TrkPRU7DHA= 2023-09-22T21:23:09.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2+4C2giYRjwTYW7B1TrkPRU7DHA=', 'Content-Length': '0'} 2023-09-22T21:23:09.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cca95f6954cbf36a-00650e05bd-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.426 DEBUG:boto:b'ffzdqu-48obj81000falseobj82023-09-22T21:22:41.680Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:09.427 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:09.427 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:09.427 DEBUG:boto:path=/ffzdqu-48/?prefix=obj8 2023-09-22T21:23:09.427 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj8 2023-09-22T21:23:09.427 DEBUG:boto:Method: GET 2023-09-22T21:23:09.427 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj8 2023-09-22T21:23:09.428 DEBUG:boto:Data: 2023-09-22T21:23:09.428 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.428 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.428 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.428 DEBUG:boto:Params: {} 2023-09-22T21:23:09.428 DEBUG:boto:Token: None 2023-09-22T21:23:09.428 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/ 2023-09-22T21:23:09.428 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2+4C2giYRjwTYW7B1TrkPRU7DHA= 2023-09-22T21:23:09.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2+4C2giYRjwTYW7B1TrkPRU7DHA=', 'Content-Length': '0'} 2023-09-22T21:23:09.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c00d5f525ba7d92f-00650e05bd-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.433 DEBUG:boto:b'ffzdqu-48obj81000falseobj82023-09-22T21:22:41.680Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:09.434 DEBUG:boto:path=/ffzdqu-48/obj8 2023-09-22T21:23:09.434 DEBUG:boto:auth_path=/ffzdqu-48/obj8 2023-09-22T21:23:09.434 DEBUG:boto:Method: HEAD 2023-09-22T21:23:09.434 DEBUG:boto:Path: /ffzdqu-48/obj8 2023-09-22T21:23:09.434 DEBUG:boto:Data: 2023-09-22T21:23:09.434 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.435 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.435 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.435 DEBUG:boto:Params: {} 2023-09-22T21:23:09.435 DEBUG:boto:Token: None 2023-09-22T21:23:09.435 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/obj8 2023-09-22T21:23:09.435 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sFsqZdAknYJw51AoJni5dOtlVNw= 2023-09-22T21:23:09.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sFsqZdAknYJw51AoJni5dOtlVNw=', 'Content-Length': '0'} 2023-09-22T21:23:09.438 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a0bc48f4cfe0569-00650e05bd-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.438 DEBUG:boto:path=/ffzdqu-48/obj8 2023-09-22T21:23:09.438 DEBUG:boto:auth_path=/ffzdqu-48/obj8 2023-09-22T21:23:09.438 DEBUG:boto:Method: HEAD 2023-09-22T21:23:09.439 DEBUG:boto:Path: /ffzdqu-48/obj8 2023-09-22T21:23:09.439 DEBUG:boto:Data: 2023-09-22T21:23:09.439 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.439 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.439 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.439 DEBUG:boto:Params: {} 2023-09-22T21:23:09.439 DEBUG:boto:Token: None 2023-09-22T21:23:09.439 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/obj8 2023-09-22T21:23:09.439 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sFsqZdAknYJw51AoJni5dOtlVNw= 2023-09-22T21:23:09.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sFsqZdAknYJw51AoJni5dOtlVNw=', 'Content-Length': '0'} 2023-09-22T21:23:09.442 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5348e8ca626d75a-00650e05bd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-22T21:23:09.442 DEBUG:boto:path=/ffzdqu-48/obj8 2023-09-22T21:23:09.443 DEBUG:boto:auth_path=/ffzdqu-48/obj8 2023-09-22T21:23:09.443 DEBUG:boto:Method: GET 2023-09-22T21:23:09.443 DEBUG:boto:Path: /ffzdqu-48/obj8 2023-09-22T21:23:09.443 DEBUG:boto:Data: 2023-09-22T21:23:09.443 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.443 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.443 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.443 DEBUG:boto:Params: {} 2023-09-22T21:23:09.443 DEBUG:boto:Token: None 2023-09-22T21:23:09.444 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/obj8 2023-09-22T21:23:09.444 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UhEyu4/hHjFCucz+PCubhu4i+Ho= 2023-09-22T21:23:09.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UhEyu4/hHjFCucz+PCubhu4i+Ho=', 'Content-Length': '0'} 2023-09-22T21:23:09.447 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-48:f132bdb9-20a7-4073-a932-8f99da20691e.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a0501f4f5744257-00650e05bd-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.494 DEBUG:boto:path=/ffzdqu-48/obj8 2023-09-22T21:23:09.495 DEBUG:boto:auth_path=/ffzdqu-48/obj8 2023-09-22T21:23:09.495 DEBUG:boto:Method: GET 2023-09-22T21:23:09.495 DEBUG:boto:Path: /ffzdqu-48/obj8 2023-09-22T21:23:09.495 DEBUG:boto:Data: 2023-09-22T21:23:09.495 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.495 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.495 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.496 DEBUG:boto:Params: {} 2023-09-22T21:23:09.496 DEBUG:boto:Token: None 2023-09-22T21:23:09.496 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/obj8 2023-09-22T21:23:09.496 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UhEyu4/hHjFCucz+PCubhu4i+Ho= 2023-09-22T21:23:09.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UhEyu4/hHjFCucz+PCubhu4i+Ho=', 'Content-Length': '0'} 2023-09-22T21:23:09.499 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b197ced304ba8558-00650e05bd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.499 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:09.499 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:09.500 DEBUG:boto:path=/ffzdqu-48/?prefix=obj8 2023-09-22T21:23:09.500 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj8 2023-09-22T21:23:09.500 DEBUG:boto:Method: GET 2023-09-22T21:23:09.500 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj8 2023-09-22T21:23:09.500 DEBUG:boto:Data: 2023-09-22T21:23:09.500 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.500 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.500 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.500 DEBUG:boto:Params: {} 2023-09-22T21:23:09.501 DEBUG:boto:Token: None 2023-09-22T21:23:09.501 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/ 2023-09-22T21:23:09.501 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2+4C2giYRjwTYW7B1TrkPRU7DHA= 2023-09-22T21:23:09.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2+4C2giYRjwTYW7B1TrkPRU7DHA=', 'Content-Length': '0'} 2023-09-22T21:23:09.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b94ce4bde94a242e-00650e05bd-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.506 DEBUG:boto:b'ffzdqu-48obj81000falseobj82023-09-22T21:22:41.680Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:09.506 DEBUG:boto:path=/ffzdqu-48/ 2023-09-22T21:23:09.506 DEBUG:boto:auth_path=/ffzdqu-48/ 2023-09-22T21:23:09.507 DEBUG:boto:path=/ffzdqu-48/?prefix=obj8 2023-09-22T21:23:09.507 DEBUG:boto:auth_path=/ffzdqu-48/?prefix=obj8 2023-09-22T21:23:09.507 DEBUG:boto:Method: GET 2023-09-22T21:23:09.507 DEBUG:boto:Path: /ffzdqu-48/?prefix=obj8 2023-09-22T21:23:09.507 DEBUG:boto:Data: 2023-09-22T21:23:09.507 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.507 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.507 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.507 DEBUG:boto:Params: {} 2023-09-22T21:23:09.508 DEBUG:boto:Token: None 2023-09-22T21:23:09.508 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-48/ 2023-09-22T21:23:09.508 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2+4C2giYRjwTYW7B1TrkPRU7DHA= 2023-09-22T21:23:09.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2+4C2giYRjwTYW7B1TrkPRU7DHA=', 'Content-Length': '0'} 2023-09-22T21:23:09.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1a5413bea2952c0-00650e05bd-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.513 DEBUG:boto:b'ffzdqu-48obj81000falseobj82023-09-22T21:22:41.680Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:09.513 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-48 zones={a2, a1} 2023-09-22T21:23:09.513 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-49 2023-09-22T21:23:09.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:23:09.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:23:09.514 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:23:09.563 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:23:09.563 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:23:09.606 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:23:09.609+0000 7fb620923a40 1 bucket sync caught up with source: 2023-09-22T21:23:09.606 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000004.44.6, 00000000002.24.1, 00000000004.51.6, 00000000002.21.1, 00000000004.40.6, 00000000002.28.1, 00000000002.21.1, 00000000004.33.6, 00000000002.17.1, 00000000002.28.1, 00000000002.96.1] 2023-09-22T21:23:09.607 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000004.44.6, 00000000002.24.1, 00000000004.51.6, 00000000002.21.1, 00000000004.40.6, 00000000002.28.1, 00000000002.21.1, 00000000004.33.6, 00000000002.17.1, 00000000002.28.1, 00000000002.96.1] 2023-09-22T21:23:09.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:23:09.609+0000 7fb620923a40 0 bucket checkpoint complete 2023-09-22T21:23:09.623 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-49 2023-09-22T21:23:09.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:23:09.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:23:09.624 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:23:09.664 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:23:09.664 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:23:09.705 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:23:09.705+0000 7ff6cf121a40 1 bucket sync caught up with source: 2023-09-22T21:23:09.706 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000004.138.1, 00000000002.213.1, 00000000004.212.1, 00000000002.237.1, 00000000004.334.1, 00000000002.266.1, 00000000002.224.1, 00000000004.180.1, 00000000002.190.1, 00000000002.262.1, 00000000002.267.1] 2023-09-22T21:23:09.706 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000004.138.1, 00000000002.213.1, 00000000004.212.1, 00000000002.237.1, 00000000004.334.1, 00000000002.266.1, 00000000002.224.1, 00000000004.180.1, 00000000002.190.1, 00000000002.262.1, 00000000002.267.1] 2023-09-22T21:23:09.706 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:23:09.705+0000 7ff6cf121a40 0 bucket checkpoint complete 2023-09-22T21:23:09.721 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-49 zones={a2, a1} 2023-09-22T21:23:09.722 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:09.722 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:09.722 DEBUG:boto:Method: HEAD 2023-09-22T21:23:09.722 DEBUG:boto:Path: /ffzdqu-49/ 2023-09-22T21:23:09.722 DEBUG:boto:Data: 2023-09-22T21:23:09.722 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.722 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.723 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.723 DEBUG:boto:Params: {} 2023-09-22T21:23:09.723 DEBUG:boto:Token: None 2023-09-22T21:23:09.723 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-49/ 2023-09-22T21:23:09.723 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:p8Tcv4KEGsds+YA7ZUBT3EkzGIQ= 2023-09-22T21:23:09.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:p8Tcv4KEGsds+YA7ZUBT3EkzGIQ=', 'Content-Length': '0'} 2023-09-22T21:23:09.725 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c78e80c19b1d7f8-00650e05bd-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.726 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:09.726 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:09.726 DEBUG:boto:Method: HEAD 2023-09-22T21:23:09.726 DEBUG:boto:Path: /ffzdqu-49/ 2023-09-22T21:23:09.726 DEBUG:boto:Data: 2023-09-22T21:23:09.726 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.727 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.727 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.727 DEBUG:boto:Params: {} 2023-09-22T21:23:09.727 DEBUG:boto:Token: None 2023-09-22T21:23:09.727 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-49/ 2023-09-22T21:23:09.727 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:p8Tcv4KEGsds+YA7ZUBT3EkzGIQ= 2023-09-22T21:23:09.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:p8Tcv4KEGsds+YA7ZUBT3EkzGIQ=', 'Content-Length': '0'} 2023-09-22T21:23:09.730 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d1b57da07dca494-00650e05bd-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.730 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:23:09.730 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:09.730 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:09.730 DEBUG:boto:path=/ffzdqu-49/?versions&max-keys=999 2023-09-22T21:23:09.730 DEBUG:boto:auth_path=/ffzdqu-49/?versions&max-keys=999 2023-09-22T21:23:09.731 DEBUG:boto:Method: GET 2023-09-22T21:23:09.731 DEBUG:boto:Path: /ffzdqu-49/?versions&max-keys=999 2023-09-22T21:23:09.731 DEBUG:boto:Data: 2023-09-22T21:23:09.731 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.731 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.731 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.731 DEBUG:boto:Params: {} 2023-09-22T21:23:09.731 DEBUG:boto:Token: None 2023-09-22T21:23:09.732 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-49/?versions 2023-09-22T21:23:09.732 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:feyn62uxlyzvoLr/wdE2+qnkgeE= 2023-09-22T21:23:09.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:feyn62uxlyzvoLr/wdE2+qnkgeE=', 'Content-Length': '0'} 2023-09-22T21:23:09.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a46cb579284c15d2-00650e05bd-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.734 DEBUG:boto:b'ffzdqu-49999falseobj1nulltrue2023-09-22T21:22:07.756Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:22:07.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:22:07.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:22:07.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-22T21:22:41.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-22T21:22:41.752Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-22T21:22:41.767Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-22T21:22:41.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:09.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-22T21:23:09.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-22T21:23:09.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-22T21:23:09.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-22T21:23:09.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-09-22T21:23:09.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-09-22T21:23:09.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-09-22T21:23:09.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-09-22T21:23:09.736 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:23:09.736 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:09.736 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:09.736 DEBUG:boto:path=/ffzdqu-49/?versions&max-keys=999 2023-09-22T21:23:09.736 DEBUG:boto:auth_path=/ffzdqu-49/?versions&max-keys=999 2023-09-22T21:23:09.737 DEBUG:boto:Method: GET 2023-09-22T21:23:09.737 DEBUG:boto:Path: /ffzdqu-49/?versions&max-keys=999 2023-09-22T21:23:09.737 DEBUG:boto:Data: 2023-09-22T21:23:09.737 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.737 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.737 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.737 DEBUG:boto:Params: {} 2023-09-22T21:23:09.737 DEBUG:boto:Token: None 2023-09-22T21:23:09.737 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-49/?versions 2023-09-22T21:23:09.738 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:feyn62uxlyzvoLr/wdE2+qnkgeE= 2023-09-22T21:23:09.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:feyn62uxlyzvoLr/wdE2+qnkgeE=', 'Content-Length': '0'} 2023-09-22T21:23:09.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf7bff944da3b007-00650e05bd-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.741 DEBUG:boto:b'ffzdqu-49999falseobj1nulltrue2023-09-22T21:22:07.756Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:22:07.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:22:07.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:22:07.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-22T21:22:41.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-22T21:22:41.752Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-22T21:22:41.767Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-22T21:22:41.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:09.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-22T21:23:09.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-22T21:23:09.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-22T21:23:09.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-22T21:23:09.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-09-22T21:23:09.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-09-22T21:23:09.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-09-22T21:23:09.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-09-22T21:23:09.742 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:09.742 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:09.742 DEBUG:boto:path=/ffzdqu-49/?versions&max-keys=999 2023-09-22T21:23:09.743 DEBUG:boto:auth_path=/ffzdqu-49/?versions&max-keys=999 2023-09-22T21:23:09.743 DEBUG:boto:Method: GET 2023-09-22T21:23:09.743 DEBUG:boto:Path: /ffzdqu-49/?versions&max-keys=999 2023-09-22T21:23:09.743 DEBUG:boto:Data: 2023-09-22T21:23:09.743 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.743 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.743 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.743 DEBUG:boto:Params: {} 2023-09-22T21:23:09.743 DEBUG:boto:Token: None 2023-09-22T21:23:09.744 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-49/?versions 2023-09-22T21:23:09.744 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:feyn62uxlyzvoLr/wdE2+qnkgeE= 2023-09-22T21:23:09.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:feyn62uxlyzvoLr/wdE2+qnkgeE=', 'Content-Length': '0'} 2023-09-22T21:23:09.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003df1027cbbf171f7-00650e05bd-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.747 DEBUG:boto:b'ffzdqu-49999falseobj1nulltrue2023-09-22T21:22:07.756Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:22:07.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:22:07.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:22:07.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-22T21:22:41.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-22T21:22:41.752Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-22T21:22:41.767Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-22T21:22:41.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:09.748 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:09.748 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:09.748 DEBUG:boto:path=/ffzdqu-49/?versions&max-keys=999 2023-09-22T21:23:09.749 DEBUG:boto:auth_path=/ffzdqu-49/?versions&max-keys=999 2023-09-22T21:23:09.749 DEBUG:boto:Method: GET 2023-09-22T21:23:09.749 DEBUG:boto:Path: /ffzdqu-49/?versions&max-keys=999 2023-09-22T21:23:09.749 DEBUG:boto:Data: 2023-09-22T21:23:09.749 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.749 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.750 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.750 DEBUG:boto:Params: {} 2023-09-22T21:23:09.750 DEBUG:boto:Token: None 2023-09-22T21:23:09.750 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-49/?versions 2023-09-22T21:23:09.751 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:feyn62uxlyzvoLr/wdE2+qnkgeE= 2023-09-22T21:23:09.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:feyn62uxlyzvoLr/wdE2+qnkgeE=', 'Content-Length': '0'} 2023-09-22T21:23:09.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002827c84676fca45d-00650e05bd-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.754 DEBUG:boto:b'ffzdqu-49999falseobj1nulltrue2023-09-22T21:22:07.756Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-22T21:22:07.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-22T21:22:07.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-22T21:22:07.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-22T21:22:41.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-22T21:22:41.752Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-22T21:22:41.767Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-22T21:22:41.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:09.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:23:09.755 DEBUG:boto:path=/ffzdqu-49/obj1 2023-09-22T21:23:09.755 DEBUG:boto:auth_path=/ffzdqu-49/obj1 2023-09-22T21:23:09.755 DEBUG:boto:path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.755 DEBUG:boto:auth_path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.755 DEBUG:boto:Method: GET 2023-09-22T21:23:09.755 DEBUG:boto:Path: /ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.755 DEBUG:boto:Data: 2023-09-22T21:23:09.755 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.755 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.756 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.756 DEBUG:boto:Params: {} 2023-09-22T21:23:09.756 DEBUG:boto:Token: None 2023-09-22T21:23:09.756 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.756 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t5GgHCYsZvXpqeBvhROUAcRtOxk= 2023-09-22T21:23:09.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t5GgHCYsZvXpqeBvhROUAcRtOxk=', 'Content-Length': '0'} 2023-09-22T21:23:09.758 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc8ac9cdc5663241-00650e05bd-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.798 DEBUG:boto:path=/ffzdqu-49/obj1 2023-09-22T21:23:09.798 DEBUG:boto:auth_path=/ffzdqu-49/obj1 2023-09-22T21:23:09.799 DEBUG:boto:path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.799 DEBUG:boto:auth_path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.799 DEBUG:boto:Method: GET 2023-09-22T21:23:09.799 DEBUG:boto:Path: /ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.799 DEBUG:boto:Data: 2023-09-22T21:23:09.799 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.799 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.799 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.799 DEBUG:boto:Params: {} 2023-09-22T21:23:09.800 DEBUG:boto:Token: None 2023-09-22T21:23:09.800 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.800 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t5GgHCYsZvXpqeBvhROUAcRtOxk= 2023-09-22T21:23:09.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t5GgHCYsZvXpqeBvhROUAcRtOxk=', 'Content-Length': '0'} 2023-09-22T21:23:09.803 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa570702e5c262c9-00650e05bd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.803 DEBUG:boto:path=/ffzdqu-49/obj1 2023-09-22T21:23:09.803 DEBUG:boto:auth_path=/ffzdqu-49/obj1 2023-09-22T21:23:09.803 DEBUG:boto:path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.803 DEBUG:boto:auth_path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.803 DEBUG:boto:Method: HEAD 2023-09-22T21:23:09.804 DEBUG:boto:Path: /ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.804 DEBUG:boto:Data: 2023-09-22T21:23:09.804 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.804 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.804 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.804 DEBUG:boto:Params: {} 2023-09-22T21:23:09.804 DEBUG:boto:Token: None 2023-09-22T21:23:09.804 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.805 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:64SMgDibfjBu6vqxDVaHD4n5wD8= 2023-09-22T21:23:09.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:64SMgDibfjBu6vqxDVaHD4n5wD8=', 'Content-Length': '0'} 2023-09-22T21:23:09.807 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db8ab1dc6f1eb0eb-00650e05bd-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.807 DEBUG:boto:path=/ffzdqu-49/obj1 2023-09-22T21:23:09.807 DEBUG:boto:auth_path=/ffzdqu-49/obj1 2023-09-22T21:23:09.807 DEBUG:boto:path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.807 DEBUG:boto:auth_path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.808 DEBUG:boto:Method: HEAD 2023-09-22T21:23:09.808 DEBUG:boto:Path: /ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.808 DEBUG:boto:Data: 2023-09-22T21:23:09.808 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.808 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.808 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.808 DEBUG:boto:Params: {} 2023-09-22T21:23:09.808 DEBUG:boto:Token: None 2023-09-22T21:23:09.808 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.809 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:64SMgDibfjBu6vqxDVaHD4n5wD8= 2023-09-22T21:23:09.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:64SMgDibfjBu6vqxDVaHD4n5wD8=', 'Content-Length': '0'} 2023-09-22T21:23:09.811 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f75454c4a5c51fc7-00650e05bd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:23:09.812 DEBUG:boto:path=/ffzdqu-49/obj1 2023-09-22T21:23:09.812 DEBUG:boto:auth_path=/ffzdqu-49/obj1 2023-09-22T21:23:09.812 DEBUG:boto:path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.812 DEBUG:boto:auth_path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.812 DEBUG:boto:Method: GET 2023-09-22T21:23:09.812 DEBUG:boto:Path: /ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.812 DEBUG:boto:Data: 2023-09-22T21:23:09.812 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.812 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.812 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.813 DEBUG:boto:Params: {} 2023-09-22T21:23:09.813 DEBUG:boto:Token: None 2023-09-22T21:23:09.813 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.813 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t5GgHCYsZvXpqeBvhROUAcRtOxk= 2023-09-22T21:23:09.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t5GgHCYsZvXpqeBvhROUAcRtOxk=', 'Content-Length': '0'} 2023-09-22T21:23:09.815 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c48331f4707fe83c-00650e05bd-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.816 DEBUG:boto:path=/ffzdqu-49/obj1 2023-09-22T21:23:09.816 DEBUG:boto:auth_path=/ffzdqu-49/obj1 2023-09-22T21:23:09.816 DEBUG:boto:path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.816 DEBUG:boto:auth_path=/ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.816 DEBUG:boto:Method: GET 2023-09-22T21:23:09.816 DEBUG:boto:Path: /ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.816 DEBUG:boto:Data: 2023-09-22T21:23:09.816 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.817 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.817 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.817 DEBUG:boto:Params: {} 2023-09-22T21:23:09.817 DEBUG:boto:Token: None 2023-09-22T21:23:09.817 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-49/obj1?versionId=null 2023-09-22T21:23:09.817 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t5GgHCYsZvXpqeBvhROUAcRtOxk= 2023-09-22T21:23:09.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t5GgHCYsZvXpqeBvhROUAcRtOxk=', 'Content-Length': '0'} 2023-09-22T21:23:09.820 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021f2d9c095a11943-00650e05bd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.862 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:09.862 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:09.863 DEBUG:boto:path=/ffzdqu-49/?prefix=obj1 2023-09-22T21:23:09.863 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj1 2023-09-22T21:23:09.863 DEBUG:boto:Method: GET 2023-09-22T21:23:09.863 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj1 2023-09-22T21:23:09.863 DEBUG:boto:Data: 2023-09-22T21:23:09.863 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.863 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.863 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.863 DEBUG:boto:Params: {} 2023-09-22T21:23:09.864 DEBUG:boto:Token: None 2023-09-22T21:23:09.864 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-49/ 2023-09-22T21:23:09.864 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tHuuA6HnvBRyzCn35mUS9J0pip8= 2023-09-22T21:23:09.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tHuuA6HnvBRyzCn35mUS9J0pip8=', 'Content-Length': '0'} 2023-09-22T21:23:09.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e935a0e7daf3dc4-00650e05bd-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.868 DEBUG:boto:b'ffzdqu-49obj11000falseobj12023-09-22T21:22:07.756Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:09.869 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:09.869 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:09.869 DEBUG:boto:path=/ffzdqu-49/?prefix=obj1 2023-09-22T21:23:09.869 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj1 2023-09-22T21:23:09.869 DEBUG:boto:Method: GET 2023-09-22T21:23:09.869 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj1 2023-09-22T21:23:09.869 DEBUG:boto:Data: 2023-09-22T21:23:09.869 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.870 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.870 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.870 DEBUG:boto:Params: {} 2023-09-22T21:23:09.870 DEBUG:boto:Token: None 2023-09-22T21:23:09.870 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-49/ 2023-09-22T21:23:09.870 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tHuuA6HnvBRyzCn35mUS9J0pip8= 2023-09-22T21:23:09.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tHuuA6HnvBRyzCn35mUS9J0pip8=', 'Content-Length': '0'} 2023-09-22T21:23:09.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f1b8467d2b748af-00650e05bd-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.878 DEBUG:boto:b'ffzdqu-49obj11000falseobj12023-09-22T21:22:07.756Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:09.879 DEBUG:boto:path=/ffzdqu-49/obj1 2023-09-22T21:23:09.879 DEBUG:boto:auth_path=/ffzdqu-49/obj1 2023-09-22T21:23:09.879 DEBUG:boto:Method: HEAD 2023-09-22T21:23:09.879 DEBUG:boto:Path: /ffzdqu-49/obj1 2023-09-22T21:23:09.879 DEBUG:boto:Data: 2023-09-22T21:23:09.879 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.879 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.880 DEBUG:boto:Params: {} 2023-09-22T21:23:09.880 DEBUG:boto:Token: None 2023-09-22T21:23:09.880 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-49/obj1 2023-09-22T21:23:09.880 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:itkcwTCEGjJG2JhcDUMR+mPXM3o= 2023-09-22T21:23:09.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:itkcwTCEGjJG2JhcDUMR+mPXM3o=', 'Content-Length': '0'} 2023-09-22T21:23:09.883 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000715d343ef6af13bc-00650e05bd-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.883 DEBUG:boto:path=/ffzdqu-49/obj1 2023-09-22T21:23:09.883 DEBUG:boto:auth_path=/ffzdqu-49/obj1 2023-09-22T21:23:09.883 DEBUG:boto:Method: HEAD 2023-09-22T21:23:09.883 DEBUG:boto:Path: /ffzdqu-49/obj1 2023-09-22T21:23:09.884 DEBUG:boto:Data: 2023-09-22T21:23:09.884 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.884 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.884 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.884 DEBUG:boto:Params: {} 2023-09-22T21:23:09.884 DEBUG:boto:Token: None 2023-09-22T21:23:09.884 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-49/obj1 2023-09-22T21:23:09.884 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:itkcwTCEGjJG2JhcDUMR+mPXM3o= 2023-09-22T21:23:09.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:itkcwTCEGjJG2JhcDUMR+mPXM3o=', 'Content-Length': '0'} 2023-09-22T21:23:09.887 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbfc5fcc9bfcb856-00650e05bd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-22T21:23:09.887 DEBUG:boto:path=/ffzdqu-49/obj1 2023-09-22T21:23:09.888 DEBUG:boto:auth_path=/ffzdqu-49/obj1 2023-09-22T21:23:09.888 DEBUG:boto:Method: GET 2023-09-22T21:23:09.888 DEBUG:boto:Path: /ffzdqu-49/obj1 2023-09-22T21:23:09.888 DEBUG:boto:Data: 2023-09-22T21:23:09.888 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.888 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.888 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.888 DEBUG:boto:Params: {} 2023-09-22T21:23:09.889 DEBUG:boto:Token: None 2023-09-22T21:23:09.889 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-49/obj1 2023-09-22T21:23:09.889 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NHu1CsAcy88AziBM+RFVmcS7PnE= 2023-09-22T21:23:09.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NHu1CsAcy88AziBM+RFVmcS7PnE=', 'Content-Length': '0'} 2023-09-22T21:23:09.891 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2ee8177453457c5-00650e05bd-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.934 DEBUG:boto:path=/ffzdqu-49/obj1 2023-09-22T21:23:09.934 DEBUG:boto:auth_path=/ffzdqu-49/obj1 2023-09-22T21:23:09.935 DEBUG:boto:Method: GET 2023-09-22T21:23:09.935 DEBUG:boto:Path: /ffzdqu-49/obj1 2023-09-22T21:23:09.935 DEBUG:boto:Data: 2023-09-22T21:23:09.935 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.935 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.935 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.935 DEBUG:boto:Params: {} 2023-09-22T21:23:09.935 DEBUG:boto:Token: None 2023-09-22T21:23:09.935 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-49/obj1 2023-09-22T21:23:09.936 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NHu1CsAcy88AziBM+RFVmcS7PnE= 2023-09-22T21:23:09.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NHu1CsAcy88AziBM+RFVmcS7PnE=', 'Content-Length': '0'} 2023-09-22T21:23:09.938 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb08124031af152e-00650e05bd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.982 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:09.982 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:09.983 DEBUG:boto:path=/ffzdqu-49/?prefix=obj1 2023-09-22T21:23:09.983 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj1 2023-09-22T21:23:09.983 DEBUG:boto:Method: GET 2023-09-22T21:23:09.983 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj1 2023-09-22T21:23:09.983 DEBUG:boto:Data: 2023-09-22T21:23:09.983 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.983 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:09.983 DEBUG:boto:Port: 8001 2023-09-22T21:23:09.984 DEBUG:boto:Params: {} 2023-09-22T21:23:09.984 DEBUG:boto:Token: None 2023-09-22T21:23:09.984 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-49/ 2023-09-22T21:23:09.984 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tHuuA6HnvBRyzCn35mUS9J0pip8= 2023-09-22T21:23:09.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tHuuA6HnvBRyzCn35mUS9J0pip8=', 'Content-Length': '0'} 2023-09-22T21:23:09.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010eeb28c0a5d937c-00650e05bd-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:09.989 DEBUG:boto:b'ffzdqu-49obj11000falseobj12023-09-22T21:22:07.756Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:09.989 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:09.989 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:09.989 DEBUG:boto:path=/ffzdqu-49/?prefix=obj1 2023-09-22T21:23:09.990 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj1 2023-09-22T21:23:09.990 DEBUG:boto:Method: GET 2023-09-22T21:23:09.990 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj1 2023-09-22T21:23:09.990 DEBUG:boto:Data: 2023-09-22T21:23:09.990 DEBUG:boto:Headers: {} 2023-09-22T21:23:09.990 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:09.990 DEBUG:boto:Port: 8000 2023-09-22T21:23:09.990 DEBUG:boto:Params: {} 2023-09-22T21:23:09.990 DEBUG:boto:Token: None 2023-09-22T21:23:09.991 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:09 GMT /ffzdqu-49/ 2023-09-22T21:23:09.991 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tHuuA6HnvBRyzCn35mUS9J0pip8= 2023-09-22T21:23:09.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tHuuA6HnvBRyzCn35mUS9J0pip8=', 'Content-Length': '0'} 2023-09-22T21:23:09.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6b4957d98b526f6-00650e05bd-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.038 DEBUG:boto:b'ffzdqu-49obj11000falseobj12023-09-22T21:22:07.756Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:10.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:23:10.039 DEBUG:boto:path=/ffzdqu-49/obj2 2023-09-22T21:23:10.039 DEBUG:boto:auth_path=/ffzdqu-49/obj2 2023-09-22T21:23:10.039 DEBUG:boto:path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.039 DEBUG:boto:auth_path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.039 DEBUG:boto:Method: GET 2023-09-22T21:23:10.039 DEBUG:boto:Path: /ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.039 DEBUG:boto:Data: 2023-09-22T21:23:10.040 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.040 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.040 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.040 DEBUG:boto:Params: {} 2023-09-22T21:23:10.040 DEBUG:boto:Token: None 2023-09-22T21:23:10.040 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.040 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2zbauzs5EBh7tpmyji14bcBX75E= 2023-09-22T21:23:10.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2zbauzs5EBh7tpmyji14bcBX75E=', 'Content-Length': '0'} 2023-09-22T21:23:10.043 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcfd3d1ab77b49d9-00650e05be-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.043 DEBUG:boto:path=/ffzdqu-49/obj2 2023-09-22T21:23:10.043 DEBUG:boto:auth_path=/ffzdqu-49/obj2 2023-09-22T21:23:10.043 DEBUG:boto:path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.044 DEBUG:boto:auth_path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.044 DEBUG:boto:Method: GET 2023-09-22T21:23:10.044 DEBUG:boto:Path: /ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.044 DEBUG:boto:Data: 2023-09-22T21:23:10.044 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.044 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.044 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.044 DEBUG:boto:Params: {} 2023-09-22T21:23:10.044 DEBUG:boto:Token: None 2023-09-22T21:23:10.045 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.045 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2zbauzs5EBh7tpmyji14bcBX75E= 2023-09-22T21:23:10.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2zbauzs5EBh7tpmyji14bcBX75E=', 'Content-Length': '0'} 2023-09-22T21:23:10.047 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1565a571333f8f3-00650e05be-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.047 DEBUG:boto:path=/ffzdqu-49/obj2 2023-09-22T21:23:10.048 DEBUG:boto:auth_path=/ffzdqu-49/obj2 2023-09-22T21:23:10.048 DEBUG:boto:path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.048 DEBUG:boto:auth_path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.048 DEBUG:boto:Method: HEAD 2023-09-22T21:23:10.048 DEBUG:boto:Path: /ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.048 DEBUG:boto:Data: 2023-09-22T21:23:10.048 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.048 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.048 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.048 DEBUG:boto:Params: {} 2023-09-22T21:23:10.049 DEBUG:boto:Token: None 2023-09-22T21:23:10.049 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.049 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yeYEN54P9V+MKmQ380woQ8WEbjc= 2023-09-22T21:23:10.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yeYEN54P9V+MKmQ380woQ8WEbjc=', 'Content-Length': '0'} 2023-09-22T21:23:10.052 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c955a32d8544d2a8-00650e05be-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.052 DEBUG:boto:path=/ffzdqu-49/obj2 2023-09-22T21:23:10.052 DEBUG:boto:auth_path=/ffzdqu-49/obj2 2023-09-22T21:23:10.052 DEBUG:boto:path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.052 DEBUG:boto:auth_path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.053 DEBUG:boto:Method: HEAD 2023-09-22T21:23:10.053 DEBUG:boto:Path: /ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.053 DEBUG:boto:Data: 2023-09-22T21:23:10.053 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.053 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.053 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.053 DEBUG:boto:Params: {} 2023-09-22T21:23:10.054 DEBUG:boto:Token: None 2023-09-22T21:23:10.054 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.054 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yeYEN54P9V+MKmQ380woQ8WEbjc= 2023-09-22T21:23:10.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yeYEN54P9V+MKmQ380woQ8WEbjc=', 'Content-Length': '0'} 2023-09-22T21:23:10.057 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ec86737b2c1487a-00650e05be-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:23:10.057 DEBUG:boto:path=/ffzdqu-49/obj2 2023-09-22T21:23:10.057 DEBUG:boto:auth_path=/ffzdqu-49/obj2 2023-09-22T21:23:10.057 DEBUG:boto:path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.057 DEBUG:boto:auth_path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.057 DEBUG:boto:Method: GET 2023-09-22T21:23:10.058 DEBUG:boto:Path: /ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.058 DEBUG:boto:Data: 2023-09-22T21:23:10.058 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.058 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.058 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.058 DEBUG:boto:Params: {} 2023-09-22T21:23:10.058 DEBUG:boto:Token: None 2023-09-22T21:23:10.058 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.059 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2zbauzs5EBh7tpmyji14bcBX75E= 2023-09-22T21:23:10.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2zbauzs5EBh7tpmyji14bcBX75E=', 'Content-Length': '0'} 2023-09-22T21:23:10.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d42a17fede98b70-00650e05be-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.062 DEBUG:boto:path=/ffzdqu-49/obj2 2023-09-22T21:23:10.062 DEBUG:boto:auth_path=/ffzdqu-49/obj2 2023-09-22T21:23:10.062 DEBUG:boto:path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.062 DEBUG:boto:auth_path=/ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.062 DEBUG:boto:Method: GET 2023-09-22T21:23:10.062 DEBUG:boto:Path: /ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.062 DEBUG:boto:Data: 2023-09-22T21:23:10.063 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.063 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.063 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.063 DEBUG:boto:Params: {} 2023-09-22T21:23:10.063 DEBUG:boto:Token: None 2023-09-22T21:23:10.063 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj2?versionId=null 2023-09-22T21:23:10.063 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2zbauzs5EBh7tpmyji14bcBX75E= 2023-09-22T21:23:10.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2zbauzs5EBh7tpmyji14bcBX75E=', 'Content-Length': '0'} 2023-09-22T21:23:10.066 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000917e6765bd78cbb-00650e05be-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.110 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:10.111 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:10.111 DEBUG:boto:path=/ffzdqu-49/?prefix=obj2 2023-09-22T21:23:10.111 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj2 2023-09-22T21:23:10.111 DEBUG:boto:Method: GET 2023-09-22T21:23:10.111 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj2 2023-09-22T21:23:10.111 DEBUG:boto:Data: 2023-09-22T21:23:10.111 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.111 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.112 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.112 DEBUG:boto:Params: {} 2023-09-22T21:23:10.112 DEBUG:boto:Token: None 2023-09-22T21:23:10.112 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/ 2023-09-22T21:23:10.112 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg= 2023-09-22T21:23:10.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg=', 'Content-Length': '0'} 2023-09-22T21:23:10.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcaf2202a90b3331-00650e05be-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.117 DEBUG:boto:b'ffzdqu-49obj21000falseobj22023-09-22T21:22:07.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:10.118 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:10.118 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:10.118 DEBUG:boto:path=/ffzdqu-49/?prefix=obj2 2023-09-22T21:23:10.118 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj2 2023-09-22T21:23:10.118 DEBUG:boto:Method: GET 2023-09-22T21:23:10.119 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj2 2023-09-22T21:23:10.119 DEBUG:boto:Data: 2023-09-22T21:23:10.119 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.119 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.119 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.119 DEBUG:boto:Params: {} 2023-09-22T21:23:10.119 DEBUG:boto:Token: None 2023-09-22T21:23:10.119 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/ 2023-09-22T21:23:10.120 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg= 2023-09-22T21:23:10.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg=', 'Content-Length': '0'} 2023-09-22T21:23:10.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b493ad0bc43fefe2-00650e05be-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.166 DEBUG:boto:b'ffzdqu-49obj21000falseobj22023-09-22T21:22:07.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:10.167 DEBUG:boto:path=/ffzdqu-49/obj2 2023-09-22T21:23:10.167 DEBUG:boto:auth_path=/ffzdqu-49/obj2 2023-09-22T21:23:10.167 DEBUG:boto:Method: HEAD 2023-09-22T21:23:10.167 DEBUG:boto:Path: /ffzdqu-49/obj2 2023-09-22T21:23:10.167 DEBUG:boto:Data: 2023-09-22T21:23:10.168 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.168 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.168 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.168 DEBUG:boto:Params: {} 2023-09-22T21:23:10.168 DEBUG:boto:Token: None 2023-09-22T21:23:10.168 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj2 2023-09-22T21:23:10.168 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EGlQvKK7xotn4Byqlxegk/6OO6w= 2023-09-22T21:23:10.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EGlQvKK7xotn4Byqlxegk/6OO6w=', 'Content-Length': '0'} 2023-09-22T21:23:10.171 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000556e2a41917919a8-00650e05be-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.171 DEBUG:boto:path=/ffzdqu-49/obj2 2023-09-22T21:23:10.172 DEBUG:boto:auth_path=/ffzdqu-49/obj2 2023-09-22T21:23:10.172 DEBUG:boto:Method: HEAD 2023-09-22T21:23:10.172 DEBUG:boto:Path: /ffzdqu-49/obj2 2023-09-22T21:23:10.172 DEBUG:boto:Data: 2023-09-22T21:23:10.172 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.172 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.172 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.172 DEBUG:boto:Params: {} 2023-09-22T21:23:10.172 DEBUG:boto:Token: None 2023-09-22T21:23:10.173 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj2 2023-09-22T21:23:10.173 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EGlQvKK7xotn4Byqlxegk/6OO6w= 2023-09-22T21:23:10.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EGlQvKK7xotn4Byqlxegk/6OO6w=', 'Content-Length': '0'} 2023-09-22T21:23:10.175 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f1b3d1e2e2810f0-00650e05be-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-22T21:23:10.176 DEBUG:boto:path=/ffzdqu-49/obj2 2023-09-22T21:23:10.176 DEBUG:boto:auth_path=/ffzdqu-49/obj2 2023-09-22T21:23:10.176 DEBUG:boto:Method: GET 2023-09-22T21:23:10.176 DEBUG:boto:Path: /ffzdqu-49/obj2 2023-09-22T21:23:10.176 DEBUG:boto:Data: 2023-09-22T21:23:10.176 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.176 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.177 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.177 DEBUG:boto:Params: {} 2023-09-22T21:23:10.177 DEBUG:boto:Token: None 2023-09-22T21:23:10.177 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj2 2023-09-22T21:23:10.177 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:63vKekUO+2a4N48AynzSw1suHfo= 2023-09-22T21:23:10.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:63vKekUO+2a4N48AynzSw1suHfo=', 'Content-Length': '0'} 2023-09-22T21:23:10.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f5214148e00eed9-00650e05be-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.222 DEBUG:boto:path=/ffzdqu-49/obj2 2023-09-22T21:23:10.223 DEBUG:boto:auth_path=/ffzdqu-49/obj2 2023-09-22T21:23:10.223 DEBUG:boto:Method: GET 2023-09-22T21:23:10.223 DEBUG:boto:Path: /ffzdqu-49/obj2 2023-09-22T21:23:10.223 DEBUG:boto:Data: 2023-09-22T21:23:10.223 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.223 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.223 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.223 DEBUG:boto:Params: {} 2023-09-22T21:23:10.224 DEBUG:boto:Token: None 2023-09-22T21:23:10.224 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj2 2023-09-22T21:23:10.224 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:63vKekUO+2a4N48AynzSw1suHfo= 2023-09-22T21:23:10.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:63vKekUO+2a4N48AynzSw1suHfo=', 'Content-Length': '0'} 2023-09-22T21:23:10.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c52a9a1dcb54a4e-00650e05be-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.270 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:10.271 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:10.271 DEBUG:boto:path=/ffzdqu-49/?prefix=obj2 2023-09-22T21:23:10.271 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj2 2023-09-22T21:23:10.271 DEBUG:boto:Method: GET 2023-09-22T21:23:10.271 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj2 2023-09-22T21:23:10.271 DEBUG:boto:Data: 2023-09-22T21:23:10.271 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.271 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.271 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.271 DEBUG:boto:Params: {} 2023-09-22T21:23:10.272 DEBUG:boto:Token: None 2023-09-22T21:23:10.272 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/ 2023-09-22T21:23:10.272 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg= 2023-09-22T21:23:10.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg=', 'Content-Length': '0'} 2023-09-22T21:23:10.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0f82886447b9f28-00650e05be-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.277 DEBUG:boto:b'ffzdqu-49obj21000falseobj22023-09-22T21:22:07.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:10.278 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:10.278 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:10.278 DEBUG:boto:path=/ffzdqu-49/?prefix=obj2 2023-09-22T21:23:10.278 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj2 2023-09-22T21:23:10.278 DEBUG:boto:Method: GET 2023-09-22T21:23:10.278 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj2 2023-09-22T21:23:10.278 DEBUG:boto:Data: 2023-09-22T21:23:10.278 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.279 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.279 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.279 DEBUG:boto:Params: {} 2023-09-22T21:23:10.279 DEBUG:boto:Token: None 2023-09-22T21:23:10.279 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/ 2023-09-22T21:23:10.279 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg= 2023-09-22T21:23:10.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg=', 'Content-Length': '0'} 2023-09-22T21:23:10.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ad6d88552e85b46-00650e05be-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.284 DEBUG:boto:b'ffzdqu-49obj21000falseobj22023-09-22T21:22:07.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:10.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:23:10.284 DEBUG:boto:path=/ffzdqu-49/obj3 2023-09-22T21:23:10.285 DEBUG:boto:auth_path=/ffzdqu-49/obj3 2023-09-22T21:23:10.285 DEBUG:boto:path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.285 DEBUG:boto:auth_path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.285 DEBUG:boto:Method: GET 2023-09-22T21:23:10.285 DEBUG:boto:Path: /ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.285 DEBUG:boto:Data: 2023-09-22T21:23:10.285 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.285 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.285 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.285 DEBUG:boto:Params: {} 2023-09-22T21:23:10.286 DEBUG:boto:Token: None 2023-09-22T21:23:10.286 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.286 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bjThnuxmM74u9jiSHGKsYTjojPw= 2023-09-22T21:23:10.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bjThnuxmM74u9jiSHGKsYTjojPw=', 'Content-Length': '0'} 2023-09-22T21:23:10.289 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000acbe539753cd6eac-00650e05be-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.289 DEBUG:boto:path=/ffzdqu-49/obj3 2023-09-22T21:23:10.289 DEBUG:boto:auth_path=/ffzdqu-49/obj3 2023-09-22T21:23:10.289 DEBUG:boto:path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.289 DEBUG:boto:auth_path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.289 DEBUG:boto:Method: GET 2023-09-22T21:23:10.290 DEBUG:boto:Path: /ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.290 DEBUG:boto:Data: 2023-09-22T21:23:10.290 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.290 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.290 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.290 DEBUG:boto:Params: {} 2023-09-22T21:23:10.290 DEBUG:boto:Token: None 2023-09-22T21:23:10.290 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.290 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bjThnuxmM74u9jiSHGKsYTjojPw= 2023-09-22T21:23:10.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bjThnuxmM74u9jiSHGKsYTjojPw=', 'Content-Length': '0'} 2023-09-22T21:23:10.293 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014dd4db146baa0dc-00650e05be-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.338 DEBUG:boto:path=/ffzdqu-49/obj3 2023-09-22T21:23:10.338 DEBUG:boto:auth_path=/ffzdqu-49/obj3 2023-09-22T21:23:10.339 DEBUG:boto:path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.339 DEBUG:boto:auth_path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.339 DEBUG:boto:Method: HEAD 2023-09-22T21:23:10.339 DEBUG:boto:Path: /ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.339 DEBUG:boto:Data: 2023-09-22T21:23:10.339 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.339 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.339 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.339 DEBUG:boto:Params: {} 2023-09-22T21:23:10.340 DEBUG:boto:Token: None 2023-09-22T21:23:10.340 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.340 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RVUolllchaqHWTPnPPatjWUbRYs= 2023-09-22T21:23:10.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RVUolllchaqHWTPnPPatjWUbRYs=', 'Content-Length': '0'} 2023-09-22T21:23:10.342 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000852a43b31ede7ee0-00650e05be-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.343 DEBUG:boto:path=/ffzdqu-49/obj3 2023-09-22T21:23:10.343 DEBUG:boto:auth_path=/ffzdqu-49/obj3 2023-09-22T21:23:10.343 DEBUG:boto:path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.343 DEBUG:boto:auth_path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.343 DEBUG:boto:Method: HEAD 2023-09-22T21:23:10.343 DEBUG:boto:Path: /ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.343 DEBUG:boto:Data: 2023-09-22T21:23:10.343 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.344 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.344 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.344 DEBUG:boto:Params: {} 2023-09-22T21:23:10.344 DEBUG:boto:Token: None 2023-09-22T21:23:10.344 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.344 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RVUolllchaqHWTPnPPatjWUbRYs= 2023-09-22T21:23:10.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RVUolllchaqHWTPnPPatjWUbRYs=', 'Content-Length': '0'} 2023-09-22T21:23:10.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066a253c8bbcd82f9-00650e05be-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:23:10.347 DEBUG:boto:path=/ffzdqu-49/obj3 2023-09-22T21:23:10.347 DEBUG:boto:auth_path=/ffzdqu-49/obj3 2023-09-22T21:23:10.347 DEBUG:boto:path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.348 DEBUG:boto:auth_path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.348 DEBUG:boto:Method: GET 2023-09-22T21:23:10.348 DEBUG:boto:Path: /ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.348 DEBUG:boto:Data: 2023-09-22T21:23:10.348 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.348 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.348 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.348 DEBUG:boto:Params: {} 2023-09-22T21:23:10.348 DEBUG:boto:Token: None 2023-09-22T21:23:10.348 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.349 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bjThnuxmM74u9jiSHGKsYTjojPw= 2023-09-22T21:23:10.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bjThnuxmM74u9jiSHGKsYTjojPw=', 'Content-Length': '0'} 2023-09-22T21:23:10.351 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000132cbf2370bec014-00650e05be-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.352 DEBUG:boto:path=/ffzdqu-49/obj3 2023-09-22T21:23:10.352 DEBUG:boto:auth_path=/ffzdqu-49/obj3 2023-09-22T21:23:10.352 DEBUG:boto:path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.352 DEBUG:boto:auth_path=/ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.352 DEBUG:boto:Method: GET 2023-09-22T21:23:10.352 DEBUG:boto:Path: /ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.352 DEBUG:boto:Data: 2023-09-22T21:23:10.353 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.353 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.353 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.353 DEBUG:boto:Params: {} 2023-09-22T21:23:10.353 DEBUG:boto:Token: None 2023-09-22T21:23:10.353 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj3?versionId=null 2023-09-22T21:23:10.353 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bjThnuxmM74u9jiSHGKsYTjojPw= 2023-09-22T21:23:10.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bjThnuxmM74u9jiSHGKsYTjojPw=', 'Content-Length': '0'} 2023-09-22T21:23:10.356 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000223278b8b65df534-00650e05be-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.398 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:10.398 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:10.399 DEBUG:boto:path=/ffzdqu-49/?prefix=obj3 2023-09-22T21:23:10.399 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj3 2023-09-22T21:23:10.399 DEBUG:boto:Method: GET 2023-09-22T21:23:10.399 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj3 2023-09-22T21:23:10.399 DEBUG:boto:Data: 2023-09-22T21:23:10.399 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.399 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.399 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.399 DEBUG:boto:Params: {} 2023-09-22T21:23:10.399 DEBUG:boto:Token: None 2023-09-22T21:23:10.400 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/ 2023-09-22T21:23:10.400 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg= 2023-09-22T21:23:10.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg=', 'Content-Length': '0'} 2023-09-22T21:23:10.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056ad9d71e5c23875-00650e05be-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.404 DEBUG:boto:b'ffzdqu-49obj31000falseobj32023-09-22T21:22:07.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:10.405 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:10.405 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:10.405 DEBUG:boto:path=/ffzdqu-49/?prefix=obj3 2023-09-22T21:23:10.405 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj3 2023-09-22T21:23:10.405 DEBUG:boto:Method: GET 2023-09-22T21:23:10.405 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj3 2023-09-22T21:23:10.405 DEBUG:boto:Data: 2023-09-22T21:23:10.406 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.406 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.406 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.406 DEBUG:boto:Params: {} 2023-09-22T21:23:10.406 DEBUG:boto:Token: None 2023-09-22T21:23:10.406 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/ 2023-09-22T21:23:10.406 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg= 2023-09-22T21:23:10.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg=', 'Content-Length': '0'} 2023-09-22T21:23:10.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3de7d7b3889f26c-00650e05be-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.454 DEBUG:boto:b'ffzdqu-49obj31000falseobj32023-09-22T21:22:07.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:10.455 DEBUG:boto:path=/ffzdqu-49/obj3 2023-09-22T21:23:10.455 DEBUG:boto:auth_path=/ffzdqu-49/obj3 2023-09-22T21:23:10.455 DEBUG:boto:Method: HEAD 2023-09-22T21:23:10.455 DEBUG:boto:Path: /ffzdqu-49/obj3 2023-09-22T21:23:10.455 DEBUG:boto:Data: 2023-09-22T21:23:10.455 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.455 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.455 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.456 DEBUG:boto:Params: {} 2023-09-22T21:23:10.456 DEBUG:boto:Token: None 2023-09-22T21:23:10.456 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj3 2023-09-22T21:23:10.456 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:usWlGsPtD1vhTIDx/7j5uA/iaPU= 2023-09-22T21:23:10.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:usWlGsPtD1vhTIDx/7j5uA/iaPU=', 'Content-Length': '0'} 2023-09-22T21:23:10.459 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0f921984437d45d-00650e05be-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.459 DEBUG:boto:path=/ffzdqu-49/obj3 2023-09-22T21:23:10.459 DEBUG:boto:auth_path=/ffzdqu-49/obj3 2023-09-22T21:23:10.459 DEBUG:boto:Method: HEAD 2023-09-22T21:23:10.459 DEBUG:boto:Path: /ffzdqu-49/obj3 2023-09-22T21:23:10.459 DEBUG:boto:Data: 2023-09-22T21:23:10.460 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.460 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.460 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.460 DEBUG:boto:Params: {} 2023-09-22T21:23:10.460 DEBUG:boto:Token: None 2023-09-22T21:23:10.460 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj3 2023-09-22T21:23:10.460 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:usWlGsPtD1vhTIDx/7j5uA/iaPU= 2023-09-22T21:23:10.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:usWlGsPtD1vhTIDx/7j5uA/iaPU=', 'Content-Length': '0'} 2023-09-22T21:23:10.463 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030e79a0626c44d31-00650e05be-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-22T21:23:10.463 DEBUG:boto:path=/ffzdqu-49/obj3 2023-09-22T21:23:10.463 DEBUG:boto:auth_path=/ffzdqu-49/obj3 2023-09-22T21:23:10.464 DEBUG:boto:Method: GET 2023-09-22T21:23:10.464 DEBUG:boto:Path: /ffzdqu-49/obj3 2023-09-22T21:23:10.464 DEBUG:boto:Data: 2023-09-22T21:23:10.464 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.464 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.464 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.464 DEBUG:boto:Params: {} 2023-09-22T21:23:10.464 DEBUG:boto:Token: None 2023-09-22T21:23:10.464 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj3 2023-09-22T21:23:10.465 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rbMab4d1YvSJUCbQgdR+U9qfaJw= 2023-09-22T21:23:10.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rbMab4d1YvSJUCbQgdR+U9qfaJw=', 'Content-Length': '0'} 2023-09-22T21:23:10.467 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec09fffa58e4f229-00650e05be-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.510 DEBUG:boto:path=/ffzdqu-49/obj3 2023-09-22T21:23:10.510 DEBUG:boto:auth_path=/ffzdqu-49/obj3 2023-09-22T21:23:10.511 DEBUG:boto:Method: GET 2023-09-22T21:23:10.511 DEBUG:boto:Path: /ffzdqu-49/obj3 2023-09-22T21:23:10.511 DEBUG:boto:Data: 2023-09-22T21:23:10.511 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.511 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.511 DEBUG:boto:Params: {} 2023-09-22T21:23:10.511 DEBUG:boto:Token: None 2023-09-22T21:23:10.512 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj3 2023-09-22T21:23:10.512 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rbMab4d1YvSJUCbQgdR+U9qfaJw= 2023-09-22T21:23:10.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rbMab4d1YvSJUCbQgdR+U9qfaJw=', 'Content-Length': '0'} 2023-09-22T21:23:10.515 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e061d844edfb63f-00650e05be-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.558 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:10.559 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:10.559 DEBUG:boto:path=/ffzdqu-49/?prefix=obj3 2023-09-22T21:23:10.559 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj3 2023-09-22T21:23:10.559 DEBUG:boto:Method: GET 2023-09-22T21:23:10.559 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj3 2023-09-22T21:23:10.559 DEBUG:boto:Data: 2023-09-22T21:23:10.559 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.559 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.559 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.560 DEBUG:boto:Params: {} 2023-09-22T21:23:10.560 DEBUG:boto:Token: None 2023-09-22T21:23:10.560 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/ 2023-09-22T21:23:10.560 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg= 2023-09-22T21:23:10.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg=', 'Content-Length': '0'} 2023-09-22T21:23:10.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074f2a24af7310ae1-00650e05be-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.565 DEBUG:boto:b'ffzdqu-49obj31000falseobj32023-09-22T21:22:07.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:10.565 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:10.565 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:10.566 DEBUG:boto:path=/ffzdqu-49/?prefix=obj3 2023-09-22T21:23:10.566 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj3 2023-09-22T21:23:10.566 DEBUG:boto:Method: GET 2023-09-22T21:23:10.566 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj3 2023-09-22T21:23:10.566 DEBUG:boto:Data: 2023-09-22T21:23:10.566 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.566 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.566 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.566 DEBUG:boto:Params: {} 2023-09-22T21:23:10.567 DEBUG:boto:Token: None 2023-09-22T21:23:10.567 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/ 2023-09-22T21:23:10.567 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg= 2023-09-22T21:23:10.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg=', 'Content-Length': '0'} 2023-09-22T21:23:10.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ab21ea95d2efd1f-00650e05be-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.572 DEBUG:boto:b'ffzdqu-49obj31000falseobj32023-09-22T21:22:07.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:10.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:23:10.572 DEBUG:boto:path=/ffzdqu-49/obj4 2023-09-22T21:23:10.572 DEBUG:boto:auth_path=/ffzdqu-49/obj4 2023-09-22T21:23:10.572 DEBUG:boto:path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.572 DEBUG:boto:auth_path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.572 DEBUG:boto:Method: GET 2023-09-22T21:23:10.572 DEBUG:boto:Path: /ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.573 DEBUG:boto:Data: 2023-09-22T21:23:10.573 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.573 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.573 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.573 DEBUG:boto:Params: {} 2023-09-22T21:23:10.573 DEBUG:boto:Token: None 2023-09-22T21:23:10.573 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.573 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PBk29nBgDlHkxlanQ8//uwz0aTA= 2023-09-22T21:23:10.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PBk29nBgDlHkxlanQ8//uwz0aTA=', 'Content-Length': '0'} 2023-09-22T21:23:10.576 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf7883830425201a-00650e05be-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.576 DEBUG:boto:path=/ffzdqu-49/obj4 2023-09-22T21:23:10.577 DEBUG:boto:auth_path=/ffzdqu-49/obj4 2023-09-22T21:23:10.577 DEBUG:boto:path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.577 DEBUG:boto:auth_path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.577 DEBUG:boto:Method: GET 2023-09-22T21:23:10.577 DEBUG:boto:Path: /ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.577 DEBUG:boto:Data: 2023-09-22T21:23:10.577 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.577 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.577 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.577 DEBUG:boto:Params: {} 2023-09-22T21:23:10.578 DEBUG:boto:Token: None 2023-09-22T21:23:10.578 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.578 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PBk29nBgDlHkxlanQ8//uwz0aTA= 2023-09-22T21:23:10.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PBk29nBgDlHkxlanQ8//uwz0aTA=', 'Content-Length': '0'} 2023-09-22T21:23:10.580 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b55e9bc0570f4e5f-00650e05be-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.626 DEBUG:boto:path=/ffzdqu-49/obj4 2023-09-22T21:23:10.626 DEBUG:boto:auth_path=/ffzdqu-49/obj4 2023-09-22T21:23:10.627 DEBUG:boto:path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.627 DEBUG:boto:auth_path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.627 DEBUG:boto:Method: HEAD 2023-09-22T21:23:10.627 DEBUG:boto:Path: /ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.627 DEBUG:boto:Data: 2023-09-22T21:23:10.627 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.627 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.627 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.627 DEBUG:boto:Params: {} 2023-09-22T21:23:10.627 DEBUG:boto:Token: None 2023-09-22T21:23:10.628 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.628 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MDWRvcw3+fU+kPfLKazYVp4xanc= 2023-09-22T21:23:10.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MDWRvcw3+fU+kPfLKazYVp4xanc=', 'Content-Length': '0'} 2023-09-22T21:23:10.630 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e45eb3067922aff-00650e05be-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.631 DEBUG:boto:path=/ffzdqu-49/obj4 2023-09-22T21:23:10.631 DEBUG:boto:auth_path=/ffzdqu-49/obj4 2023-09-22T21:23:10.631 DEBUG:boto:path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.631 DEBUG:boto:auth_path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.631 DEBUG:boto:Method: HEAD 2023-09-22T21:23:10.631 DEBUG:boto:Path: /ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.631 DEBUG:boto:Data: 2023-09-22T21:23:10.631 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.632 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.632 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.632 DEBUG:boto:Params: {} 2023-09-22T21:23:10.632 DEBUG:boto:Token: None 2023-09-22T21:23:10.632 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.632 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MDWRvcw3+fU+kPfLKazYVp4xanc= 2023-09-22T21:23:10.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MDWRvcw3+fU+kPfLKazYVp4xanc=', 'Content-Length': '0'} 2023-09-22T21:23:10.635 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000751eece32e2aa4a4-00650e05be-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:23:10.636 DEBUG:boto:path=/ffzdqu-49/obj4 2023-09-22T21:23:10.636 DEBUG:boto:auth_path=/ffzdqu-49/obj4 2023-09-22T21:23:10.636 DEBUG:boto:path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.636 DEBUG:boto:auth_path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.636 DEBUG:boto:Method: GET 2023-09-22T21:23:10.636 DEBUG:boto:Path: /ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.637 DEBUG:boto:Data: 2023-09-22T21:23:10.637 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.637 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.637 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.637 DEBUG:boto:Params: {} 2023-09-22T21:23:10.637 DEBUG:boto:Token: None 2023-09-22T21:23:10.637 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.637 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PBk29nBgDlHkxlanQ8//uwz0aTA= 2023-09-22T21:23:10.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PBk29nBgDlHkxlanQ8//uwz0aTA=', 'Content-Length': '0'} 2023-09-22T21:23:10.640 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9851d305e6da0b7-00650e05be-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.641 DEBUG:boto:path=/ffzdqu-49/obj4 2023-09-22T21:23:10.641 DEBUG:boto:auth_path=/ffzdqu-49/obj4 2023-09-22T21:23:10.641 DEBUG:boto:path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.641 DEBUG:boto:auth_path=/ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.641 DEBUG:boto:Method: GET 2023-09-22T21:23:10.641 DEBUG:boto:Path: /ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.641 DEBUG:boto:Data: 2023-09-22T21:23:10.641 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.642 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.642 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.642 DEBUG:boto:Params: {} 2023-09-22T21:23:10.642 DEBUG:boto:Token: None 2023-09-22T21:23:10.642 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj4?versionId=null 2023-09-22T21:23:10.642 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PBk29nBgDlHkxlanQ8//uwz0aTA= 2023-09-22T21:23:10.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PBk29nBgDlHkxlanQ8//uwz0aTA=', 'Content-Length': '0'} 2023-09-22T21:23:10.645 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b5acd4f3f5019b9-00650e05be-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.686 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:10.687 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:10.687 DEBUG:boto:path=/ffzdqu-49/?prefix=obj4 2023-09-22T21:23:10.687 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj4 2023-09-22T21:23:10.687 DEBUG:boto:Method: GET 2023-09-22T21:23:10.687 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj4 2023-09-22T21:23:10.687 DEBUG:boto:Data: 2023-09-22T21:23:10.687 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.687 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.687 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.688 DEBUG:boto:Params: {} 2023-09-22T21:23:10.688 DEBUG:boto:Token: None 2023-09-22T21:23:10.688 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/ 2023-09-22T21:23:10.688 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg= 2023-09-22T21:23:10.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg=', 'Content-Length': '0'} 2023-09-22T21:23:10.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e1218c313e7a3eb-00650e05be-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.693 DEBUG:boto:b'ffzdqu-49obj41000falseobj42023-09-22T21:22:07.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:10.694 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:10.694 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:10.694 DEBUG:boto:path=/ffzdqu-49/?prefix=obj4 2023-09-22T21:23:10.694 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj4 2023-09-22T21:23:10.694 DEBUG:boto:Method: GET 2023-09-22T21:23:10.694 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj4 2023-09-22T21:23:10.694 DEBUG:boto:Data: 2023-09-22T21:23:10.695 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.695 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.695 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.695 DEBUG:boto:Params: {} 2023-09-22T21:23:10.695 DEBUG:boto:Token: None 2023-09-22T21:23:10.695 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/ 2023-09-22T21:23:10.695 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg= 2023-09-22T21:23:10.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg=', 'Content-Length': '0'} 2023-09-22T21:23:10.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006001f82e4292f204-00650e05be-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.742 DEBUG:boto:b'ffzdqu-49obj41000falseobj42023-09-22T21:22:07.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:10.743 DEBUG:boto:path=/ffzdqu-49/obj4 2023-09-22T21:23:10.743 DEBUG:boto:auth_path=/ffzdqu-49/obj4 2023-09-22T21:23:10.743 DEBUG:boto:Method: HEAD 2023-09-22T21:23:10.743 DEBUG:boto:Path: /ffzdqu-49/obj4 2023-09-22T21:23:10.743 DEBUG:boto:Data: 2023-09-22T21:23:10.743 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.744 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.744 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.744 DEBUG:boto:Params: {} 2023-09-22T21:23:10.744 DEBUG:boto:Token: None 2023-09-22T21:23:10.744 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj4 2023-09-22T21:23:10.744 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J4PSnTmxF2pXGsfQNPPyXN/X020= 2023-09-22T21:23:10.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J4PSnTmxF2pXGsfQNPPyXN/X020=', 'Content-Length': '0'} 2023-09-22T21:23:10.747 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7977345f1c7e3ec-00650e05be-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.747 DEBUG:boto:path=/ffzdqu-49/obj4 2023-09-22T21:23:10.747 DEBUG:boto:auth_path=/ffzdqu-49/obj4 2023-09-22T21:23:10.747 DEBUG:boto:Method: HEAD 2023-09-22T21:23:10.748 DEBUG:boto:Path: /ffzdqu-49/obj4 2023-09-22T21:23:10.748 DEBUG:boto:Data: 2023-09-22T21:23:10.748 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.748 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.748 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.748 DEBUG:boto:Params: {} 2023-09-22T21:23:10.748 DEBUG:boto:Token: None 2023-09-22T21:23:10.748 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj4 2023-09-22T21:23:10.749 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J4PSnTmxF2pXGsfQNPPyXN/X020= 2023-09-22T21:23:10.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J4PSnTmxF2pXGsfQNPPyXN/X020=', 'Content-Length': '0'} 2023-09-22T21:23:10.751 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028097447693cdf88-00650e05be-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-22T21:23:10.752 DEBUG:boto:path=/ffzdqu-49/obj4 2023-09-22T21:23:10.752 DEBUG:boto:auth_path=/ffzdqu-49/obj4 2023-09-22T21:23:10.752 DEBUG:boto:Method: GET 2023-09-22T21:23:10.752 DEBUG:boto:Path: /ffzdqu-49/obj4 2023-09-22T21:23:10.752 DEBUG:boto:Data: 2023-09-22T21:23:10.752 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.752 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.752 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.752 DEBUG:boto:Params: {} 2023-09-22T21:23:10.752 DEBUG:boto:Token: None 2023-09-22T21:23:10.753 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj4 2023-09-22T21:23:10.753 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Q8y4reNkXgSkvflx1QlU8GkZaao= 2023-09-22T21:23:10.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Q8y4reNkXgSkvflx1QlU8GkZaao=', 'Content-Length': '0'} 2023-09-22T21:23:10.755 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa52c4dd36ae136e-00650e05be-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.798 DEBUG:boto:path=/ffzdqu-49/obj4 2023-09-22T21:23:10.799 DEBUG:boto:auth_path=/ffzdqu-49/obj4 2023-09-22T21:23:10.799 DEBUG:boto:Method: GET 2023-09-22T21:23:10.799 DEBUG:boto:Path: /ffzdqu-49/obj4 2023-09-22T21:23:10.799 DEBUG:boto:Data: 2023-09-22T21:23:10.799 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.799 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.799 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.799 DEBUG:boto:Params: {} 2023-09-22T21:23:10.800 DEBUG:boto:Token: None 2023-09-22T21:23:10.800 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj4 2023-09-22T21:23:10.800 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Q8y4reNkXgSkvflx1QlU8GkZaao= 2023-09-22T21:23:10.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Q8y4reNkXgSkvflx1QlU8GkZaao=', 'Content-Length': '0'} 2023-09-22T21:23:10.803 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000083631561ae98eee-00650e05be-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.846 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:10.847 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:10.847 DEBUG:boto:path=/ffzdqu-49/?prefix=obj4 2023-09-22T21:23:10.847 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj4 2023-09-22T21:23:10.847 DEBUG:boto:Method: GET 2023-09-22T21:23:10.847 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj4 2023-09-22T21:23:10.847 DEBUG:boto:Data: 2023-09-22T21:23:10.848 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.848 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.848 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.848 DEBUG:boto:Params: {} 2023-09-22T21:23:10.848 DEBUG:boto:Token: None 2023-09-22T21:23:10.848 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/ 2023-09-22T21:23:10.848 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg= 2023-09-22T21:23:10.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg=', 'Content-Length': '0'} 2023-09-22T21:23:10.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000005b47a6d00d2ba4-00650e05be-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.853 DEBUG:boto:b'ffzdqu-49obj41000falseobj42023-09-22T21:22:07.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:10.854 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:10.854 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:10.854 DEBUG:boto:path=/ffzdqu-49/?prefix=obj4 2023-09-22T21:23:10.854 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj4 2023-09-22T21:23:10.854 DEBUG:boto:Method: GET 2023-09-22T21:23:10.855 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj4 2023-09-22T21:23:10.855 DEBUG:boto:Data: 2023-09-22T21:23:10.855 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.855 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.855 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.855 DEBUG:boto:Params: {} 2023-09-22T21:23:10.855 DEBUG:boto:Token: None 2023-09-22T21:23:10.855 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/ 2023-09-22T21:23:10.856 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg= 2023-09-22T21:23:10.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg=', 'Content-Length': '0'} 2023-09-22T21:23:10.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003130e50f3d59dabc-00650e05be-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.861 DEBUG:boto:b'ffzdqu-49obj41000falseobj42023-09-22T21:22:07.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:10.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-22T21:23:10.862 DEBUG:boto:path=/ffzdqu-49/obj5 2023-09-22T21:23:10.862 DEBUG:boto:auth_path=/ffzdqu-49/obj5 2023-09-22T21:23:10.862 DEBUG:boto:path=/ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.862 DEBUG:boto:auth_path=/ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.862 DEBUG:boto:Method: GET 2023-09-22T21:23:10.862 DEBUG:boto:Path: /ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.862 DEBUG:boto:Data: 2023-09-22T21:23:10.862 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.863 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.863 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.863 DEBUG:boto:Params: {} 2023-09-22T21:23:10.863 DEBUG:boto:Token: None 2023-09-22T21:23:10.863 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.863 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SYV3+TJyd0HWr/Nv1V4uDipyqYc= 2023-09-22T21:23:10.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SYV3+TJyd0HWr/Nv1V4uDipyqYc=', 'Content-Length': '0'} 2023-09-22T21:23:10.866 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7a4876da6dee74c-00650e05be-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.866 DEBUG:boto:path=/ffzdqu-49/obj5 2023-09-22T21:23:10.866 DEBUG:boto:auth_path=/ffzdqu-49/obj5 2023-09-22T21:23:10.867 DEBUG:boto:path=/ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.867 DEBUG:boto:auth_path=/ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.867 DEBUG:boto:Method: GET 2023-09-22T21:23:10.867 DEBUG:boto:Path: /ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.867 DEBUG:boto:Data: 2023-09-22T21:23:10.867 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.867 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.867 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.867 DEBUG:boto:Params: {} 2023-09-22T21:23:10.868 DEBUG:boto:Token: None 2023-09-22T21:23:10.868 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.868 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SYV3+TJyd0HWr/Nv1V4uDipyqYc= 2023-09-22T21:23:10.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SYV3+TJyd0HWr/Nv1V4uDipyqYc=', 'Content-Length': '0'} 2023-09-22T21:23:10.871 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060d4c1555c700d22-00650e05be-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.914 DEBUG:boto:path=/ffzdqu-49/obj5 2023-09-22T21:23:10.915 DEBUG:boto:auth_path=/ffzdqu-49/obj5 2023-09-22T21:23:10.915 DEBUG:boto:path=/ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.915 DEBUG:boto:auth_path=/ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.915 DEBUG:boto:Method: HEAD 2023-09-22T21:23:10.915 DEBUG:boto:Path: /ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.915 DEBUG:boto:Data: 2023-09-22T21:23:10.915 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.916 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.916 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.916 DEBUG:boto:Params: {} 2023-09-22T21:23:10.916 DEBUG:boto:Token: None 2023-09-22T21:23:10.916 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.916 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IWqsdc4iP4lUyXqYRmnxJb0oGJ0= 2023-09-22T21:23:10.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IWqsdc4iP4lUyXqYRmnxJb0oGJ0=', 'Content-Length': '0'} 2023-09-22T21:23:10.919 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4d8306d6748ced7-00650e05be-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.919 DEBUG:boto:path=/ffzdqu-49/obj5 2023-09-22T21:23:10.919 DEBUG:boto:auth_path=/ffzdqu-49/obj5 2023-09-22T21:23:10.920 DEBUG:boto:path=/ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.920 DEBUG:boto:auth_path=/ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.920 DEBUG:boto:Method: HEAD 2023-09-22T21:23:10.920 DEBUG:boto:Path: /ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.920 DEBUG:boto:Data: 2023-09-22T21:23:10.920 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.920 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.920 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.920 DEBUG:boto:Params: {} 2023-09-22T21:23:10.921 DEBUG:boto:Token: None 2023-09-22T21:23:10.921 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.921 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IWqsdc4iP4lUyXqYRmnxJb0oGJ0= 2023-09-22T21:23:10.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IWqsdc4iP4lUyXqYRmnxJb0oGJ0=', 'Content-Length': '0'} 2023-09-22T21:23:10.924 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c8dc6a76da4da82-00650e05be-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-22T21:23:10.924 DEBUG:boto:path=/ffzdqu-49/obj5 2023-09-22T21:23:10.924 DEBUG:boto:auth_path=/ffzdqu-49/obj5 2023-09-22T21:23:10.924 DEBUG:boto:path=/ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.924 DEBUG:boto:auth_path=/ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.924 DEBUG:boto:Method: GET 2023-09-22T21:23:10.925 DEBUG:boto:Path: /ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.925 DEBUG:boto:Data: 2023-09-22T21:23:10.925 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.925 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.925 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.925 DEBUG:boto:Params: {} 2023-09-22T21:23:10.925 DEBUG:boto:Token: None 2023-09-22T21:23:10.925 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.926 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SYV3+TJyd0HWr/Nv1V4uDipyqYc= 2023-09-22T21:23:10.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SYV3+TJyd0HWr/Nv1V4uDipyqYc=', 'Content-Length': '0'} 2023-09-22T21:23:10.928 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b772529e876a08e-00650e05be-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.929 DEBUG:boto:path=/ffzdqu-49/obj5 2023-09-22T21:23:10.929 DEBUG:boto:auth_path=/ffzdqu-49/obj5 2023-09-22T21:23:10.929 DEBUG:boto:path=/ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.929 DEBUG:boto:auth_path=/ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.929 DEBUG:boto:Method: GET 2023-09-22T21:23:10.929 DEBUG:boto:Path: /ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.929 DEBUG:boto:Data: 2023-09-22T21:23:10.930 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.930 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.930 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.930 DEBUG:boto:Params: {} 2023-09-22T21:23:10.930 DEBUG:boto:Token: None 2023-09-22T21:23:10.930 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/obj5?versionId=null 2023-09-22T21:23:10.930 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SYV3+TJyd0HWr/Nv1V4uDipyqYc= 2023-09-22T21:23:10.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SYV3+TJyd0HWr/Nv1V4uDipyqYc=', 'Content-Length': '0'} 2023-09-22T21:23:10.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000725f22653e1b9f66-00650e05be-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.974 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:10.975 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:10.975 DEBUG:boto:path=/ffzdqu-49/?prefix=obj5 2023-09-22T21:23:10.975 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj5 2023-09-22T21:23:10.975 DEBUG:boto:Method: GET 2023-09-22T21:23:10.975 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj5 2023-09-22T21:23:10.975 DEBUG:boto:Data: 2023-09-22T21:23:10.975 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.976 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:10.976 DEBUG:boto:Port: 8001 2023-09-22T21:23:10.976 DEBUG:boto:Params: {} 2023-09-22T21:23:10.976 DEBUG:boto:Token: None 2023-09-22T21:23:10.976 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/ 2023-09-22T21:23:10.976 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg= 2023-09-22T21:23:10.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg=', 'Content-Length': '0'} 2023-09-22T21:23:10.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061c2f88c078d28fd-00650e05be-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:10.982 DEBUG:boto:b'ffzdqu-49obj51000falseobj52023-09-22T21:22:41.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:10.982 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:10.982 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:10.982 DEBUG:boto:path=/ffzdqu-49/?prefix=obj5 2023-09-22T21:23:10.983 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj5 2023-09-22T21:23:10.983 DEBUG:boto:Method: GET 2023-09-22T21:23:10.983 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj5 2023-09-22T21:23:10.983 DEBUG:boto:Data: 2023-09-22T21:23:10.983 DEBUG:boto:Headers: {} 2023-09-22T21:23:10.983 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:10.983 DEBUG:boto:Port: 8000 2023-09-22T21:23:10.983 DEBUG:boto:Params: {} 2023-09-22T21:23:10.984 DEBUG:boto:Token: None 2023-09-22T21:23:10.984 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:10 GMT /ffzdqu-49/ 2023-09-22T21:23:10.984 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg= 2023-09-22T21:23:10.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wMpi3hezU/Eb3VNvUcX81oiMMfg=', 'Content-Length': '0'} 2023-09-22T21:23:10.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cef86148ed97dee3-00650e05be-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.030 DEBUG:boto:b'ffzdqu-49obj51000falseobj52023-09-22T21:22:41.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:11.031 DEBUG:boto:path=/ffzdqu-49/obj5 2023-09-22T21:23:11.031 DEBUG:boto:auth_path=/ffzdqu-49/obj5 2023-09-22T21:23:11.031 DEBUG:boto:Method: HEAD 2023-09-22T21:23:11.031 DEBUG:boto:Path: /ffzdqu-49/obj5 2023-09-22T21:23:11.031 DEBUG:boto:Data: 2023-09-22T21:23:11.032 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.032 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.032 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.032 DEBUG:boto:Params: {} 2023-09-22T21:23:11.032 DEBUG:boto:Token: None 2023-09-22T21:23:11.033 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj5 2023-09-22T21:23:11.033 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Yr9Mas+37LcMjb4gIlYr+mGVpCo= 2023-09-22T21:23:11.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Yr9Mas+37LcMjb4gIlYr+mGVpCo=', 'Content-Length': '0'} 2023-09-22T21:23:11.036 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee019ba11f1e25ce-00650e05bf-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.036 DEBUG:boto:path=/ffzdqu-49/obj5 2023-09-22T21:23:11.036 DEBUG:boto:auth_path=/ffzdqu-49/obj5 2023-09-22T21:23:11.036 DEBUG:boto:Method: HEAD 2023-09-22T21:23:11.036 DEBUG:boto:Path: /ffzdqu-49/obj5 2023-09-22T21:23:11.036 DEBUG:boto:Data: 2023-09-22T21:23:11.037 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.037 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.037 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.037 DEBUG:boto:Params: {} 2023-09-22T21:23:11.037 DEBUG:boto:Token: None 2023-09-22T21:23:11.037 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj5 2023-09-22T21:23:11.037 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Yr9Mas+37LcMjb4gIlYr+mGVpCo= 2023-09-22T21:23:11.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Yr9Mas+37LcMjb4gIlYr+mGVpCo=', 'Content-Length': '0'} 2023-09-22T21:23:11.040 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004337ce64fcc0a3bd-00650e05bf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-22T21:23:11.041 DEBUG:boto:path=/ffzdqu-49/obj5 2023-09-22T21:23:11.041 DEBUG:boto:auth_path=/ffzdqu-49/obj5 2023-09-22T21:23:11.041 DEBUG:boto:Method: GET 2023-09-22T21:23:11.041 DEBUG:boto:Path: /ffzdqu-49/obj5 2023-09-22T21:23:11.041 DEBUG:boto:Data: 2023-09-22T21:23:11.041 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.041 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.042 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.042 DEBUG:boto:Params: {} 2023-09-22T21:23:11.042 DEBUG:boto:Token: None 2023-09-22T21:23:11.042 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj5 2023-09-22T21:23:11.042 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:V379sCI7zCTDrcHS396EVt0UfOo= 2023-09-22T21:23:11.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:V379sCI7zCTDrcHS396EVt0UfOo=', 'Content-Length': '0'} 2023-09-22T21:23:11.045 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6cf7ecca5e0a907-00650e05bf-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.090 DEBUG:boto:path=/ffzdqu-49/obj5 2023-09-22T21:23:11.091 DEBUG:boto:auth_path=/ffzdqu-49/obj5 2023-09-22T21:23:11.091 DEBUG:boto:Method: GET 2023-09-22T21:23:11.091 DEBUG:boto:Path: /ffzdqu-49/obj5 2023-09-22T21:23:11.091 DEBUG:boto:Data: 2023-09-22T21:23:11.091 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.091 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.091 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.091 DEBUG:boto:Params: {} 2023-09-22T21:23:11.092 DEBUG:boto:Token: None 2023-09-22T21:23:11.092 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj5 2023-09-22T21:23:11.092 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:V379sCI7zCTDrcHS396EVt0UfOo= 2023-09-22T21:23:11.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:V379sCI7zCTDrcHS396EVt0UfOo=', 'Content-Length': '0'} 2023-09-22T21:23:11.095 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb0094553a1622c1-00650e05bf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.138 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:11.139 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:11.139 DEBUG:boto:path=/ffzdqu-49/?prefix=obj5 2023-09-22T21:23:11.139 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj5 2023-09-22T21:23:11.139 DEBUG:boto:Method: GET 2023-09-22T21:23:11.139 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj5 2023-09-22T21:23:11.139 DEBUG:boto:Data: 2023-09-22T21:23:11.139 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.139 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.139 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.140 DEBUG:boto:Params: {} 2023-09-22T21:23:11.140 DEBUG:boto:Token: None 2023-09-22T21:23:11.140 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/ 2023-09-22T21:23:11.140 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w= 2023-09-22T21:23:11.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w=', 'Content-Length': '0'} 2023-09-22T21:23:11.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f91c085fda675d68-00650e05bf-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.146 DEBUG:boto:b'ffzdqu-49obj51000falseobj52023-09-22T21:22:41.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:11.146 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:11.146 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:11.146 DEBUG:boto:path=/ffzdqu-49/?prefix=obj5 2023-09-22T21:23:11.146 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj5 2023-09-22T21:23:11.147 DEBUG:boto:Method: GET 2023-09-22T21:23:11.147 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj5 2023-09-22T21:23:11.147 DEBUG:boto:Data: 2023-09-22T21:23:11.147 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.147 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.147 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.147 DEBUG:boto:Params: {} 2023-09-22T21:23:11.148 DEBUG:boto:Token: None 2023-09-22T21:23:11.148 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/ 2023-09-22T21:23:11.148 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w= 2023-09-22T21:23:11.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w=', 'Content-Length': '0'} 2023-09-22T21:23:11.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000555016fba4eab54d-00650e05bf-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.194 DEBUG:boto:b'ffzdqu-49obj51000falseobj52023-09-22T21:22:41.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:11.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-22T21:23:11.195 DEBUG:boto:path=/ffzdqu-49/obj6 2023-09-22T21:23:11.195 DEBUG:boto:auth_path=/ffzdqu-49/obj6 2023-09-22T21:23:11.195 DEBUG:boto:path=/ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.195 DEBUG:boto:auth_path=/ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.195 DEBUG:boto:Method: GET 2023-09-22T21:23:11.195 DEBUG:boto:Path: /ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.195 DEBUG:boto:Data: 2023-09-22T21:23:11.196 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.196 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.196 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.196 DEBUG:boto:Params: {} 2023-09-22T21:23:11.196 DEBUG:boto:Token: None 2023-09-22T21:23:11.196 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.196 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0IthDjkYQI6CyNM44lFPQyPlGg4= 2023-09-22T21:23:11.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0IthDjkYQI6CyNM44lFPQyPlGg4=', 'Content-Length': '0'} 2023-09-22T21:23:11.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c77f04d12a31b488-00650e05bf-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.200 DEBUG:boto:path=/ffzdqu-49/obj6 2023-09-22T21:23:11.200 DEBUG:boto:auth_path=/ffzdqu-49/obj6 2023-09-22T21:23:11.200 DEBUG:boto:path=/ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.200 DEBUG:boto:auth_path=/ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.200 DEBUG:boto:Method: GET 2023-09-22T21:23:11.200 DEBUG:boto:Path: /ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.201 DEBUG:boto:Data: 2023-09-22T21:23:11.201 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.201 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.201 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.201 DEBUG:boto:Params: {} 2023-09-22T21:23:11.201 DEBUG:boto:Token: None 2023-09-22T21:23:11.201 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.202 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0IthDjkYQI6CyNM44lFPQyPlGg4= 2023-09-22T21:23:11.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0IthDjkYQI6CyNM44lFPQyPlGg4=', 'Content-Length': '0'} 2023-09-22T21:23:11.205 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e08bb07e63cf1d5-00650e05bf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.246 DEBUG:boto:path=/ffzdqu-49/obj6 2023-09-22T21:23:11.247 DEBUG:boto:auth_path=/ffzdqu-49/obj6 2023-09-22T21:23:11.247 DEBUG:boto:path=/ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.247 DEBUG:boto:auth_path=/ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.247 DEBUG:boto:Method: HEAD 2023-09-22T21:23:11.247 DEBUG:boto:Path: /ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.248 DEBUG:boto:Data: 2023-09-22T21:23:11.248 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.248 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.248 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.248 DEBUG:boto:Params: {} 2023-09-22T21:23:11.249 DEBUG:boto:Token: None 2023-09-22T21:23:11.249 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.249 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bQDoYM74SNjx5cdOBiM+w++33WY= 2023-09-22T21:23:11.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bQDoYM74SNjx5cdOBiM+w++33WY=', 'Content-Length': '0'} 2023-09-22T21:23:11.252 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026a3ca087bbb6ae3-00650e05bf-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.252 DEBUG:boto:path=/ffzdqu-49/obj6 2023-09-22T21:23:11.252 DEBUG:boto:auth_path=/ffzdqu-49/obj6 2023-09-22T21:23:11.253 DEBUG:boto:path=/ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.253 DEBUG:boto:auth_path=/ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.253 DEBUG:boto:Method: HEAD 2023-09-22T21:23:11.253 DEBUG:boto:Path: /ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.253 DEBUG:boto:Data: 2023-09-22T21:23:11.253 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.253 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.253 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.253 DEBUG:boto:Params: {} 2023-09-22T21:23:11.254 DEBUG:boto:Token: None 2023-09-22T21:23:11.254 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.254 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bQDoYM74SNjx5cdOBiM+w++33WY= 2023-09-22T21:23:11.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bQDoYM74SNjx5cdOBiM+w++33WY=', 'Content-Length': '0'} 2023-09-22T21:23:11.258 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac66003007975be6-00650e05bf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-22T21:23:11.258 DEBUG:boto:path=/ffzdqu-49/obj6 2023-09-22T21:23:11.258 DEBUG:boto:auth_path=/ffzdqu-49/obj6 2023-09-22T21:23:11.258 DEBUG:boto:path=/ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.259 DEBUG:boto:auth_path=/ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.259 DEBUG:boto:Method: GET 2023-09-22T21:23:11.259 DEBUG:boto:Path: /ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.259 DEBUG:boto:Data: 2023-09-22T21:23:11.259 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.259 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.259 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.259 DEBUG:boto:Params: {} 2023-09-22T21:23:11.259 DEBUG:boto:Token: None 2023-09-22T21:23:11.259 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.260 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0IthDjkYQI6CyNM44lFPQyPlGg4= 2023-09-22T21:23:11.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0IthDjkYQI6CyNM44lFPQyPlGg4=', 'Content-Length': '0'} 2023-09-22T21:23:11.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c66bdba38c9583c7-00650e05bf-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.262 DEBUG:boto:path=/ffzdqu-49/obj6 2023-09-22T21:23:11.263 DEBUG:boto:auth_path=/ffzdqu-49/obj6 2023-09-22T21:23:11.263 DEBUG:boto:path=/ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.263 DEBUG:boto:auth_path=/ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.263 DEBUG:boto:Method: GET 2023-09-22T21:23:11.263 DEBUG:boto:Path: /ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.263 DEBUG:boto:Data: 2023-09-22T21:23:11.263 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.264 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.264 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.264 DEBUG:boto:Params: {} 2023-09-22T21:23:11.264 DEBUG:boto:Token: None 2023-09-22T21:23:11.264 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj6?versionId=null 2023-09-22T21:23:11.265 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0IthDjkYQI6CyNM44lFPQyPlGg4= 2023-09-22T21:23:11.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0IthDjkYQI6CyNM44lFPQyPlGg4=', 'Content-Length': '0'} 2023-09-22T21:23:11.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097bdc13876bc2b05-00650e05bf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.310 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:11.310 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:11.311 DEBUG:boto:path=/ffzdqu-49/?prefix=obj6 2023-09-22T21:23:11.311 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj6 2023-09-22T21:23:11.311 DEBUG:boto:Method: GET 2023-09-22T21:23:11.311 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj6 2023-09-22T21:23:11.311 DEBUG:boto:Data: 2023-09-22T21:23:11.311 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.311 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.311 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.311 DEBUG:boto:Params: {} 2023-09-22T21:23:11.312 DEBUG:boto:Token: None 2023-09-22T21:23:11.312 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/ 2023-09-22T21:23:11.312 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w= 2023-09-22T21:23:11.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w=', 'Content-Length': '0'} 2023-09-22T21:23:11.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ed395572e955310-00650e05bf-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.317 DEBUG:boto:b'ffzdqu-49obj61000falseobj62023-09-22T21:22:41.752Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:11.317 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:11.318 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:11.318 DEBUG:boto:path=/ffzdqu-49/?prefix=obj6 2023-09-22T21:23:11.318 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj6 2023-09-22T21:23:11.318 DEBUG:boto:Method: GET 2023-09-22T21:23:11.318 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj6 2023-09-22T21:23:11.318 DEBUG:boto:Data: 2023-09-22T21:23:11.318 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.318 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.318 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.319 DEBUG:boto:Params: {} 2023-09-22T21:23:11.319 DEBUG:boto:Token: None 2023-09-22T21:23:11.319 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/ 2023-09-22T21:23:11.319 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w= 2023-09-22T21:23:11.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w=', 'Content-Length': '0'} 2023-09-22T21:23:11.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007adfcf560bccb723-00650e05bf-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.324 DEBUG:boto:b'ffzdqu-49obj61000falseobj62023-09-22T21:22:41.752Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:11.324 DEBUG:boto:path=/ffzdqu-49/obj6 2023-09-22T21:23:11.324 DEBUG:boto:auth_path=/ffzdqu-49/obj6 2023-09-22T21:23:11.324 DEBUG:boto:Method: HEAD 2023-09-22T21:23:11.325 DEBUG:boto:Path: /ffzdqu-49/obj6 2023-09-22T21:23:11.325 DEBUG:boto:Data: 2023-09-22T21:23:11.325 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.325 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.325 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.325 DEBUG:boto:Params: {} 2023-09-22T21:23:11.325 DEBUG:boto:Token: None 2023-09-22T21:23:11.325 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj6 2023-09-22T21:23:11.326 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WUnVeCdA4dQYEe4mkmk4pa0WrrI= 2023-09-22T21:23:11.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WUnVeCdA4dQYEe4mkmk4pa0WrrI=', 'Content-Length': '0'} 2023-09-22T21:23:11.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bad7d0d3f241a61f-00650e05bf-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.328 DEBUG:boto:path=/ffzdqu-49/obj6 2023-09-22T21:23:11.328 DEBUG:boto:auth_path=/ffzdqu-49/obj6 2023-09-22T21:23:11.328 DEBUG:boto:Method: HEAD 2023-09-22T21:23:11.328 DEBUG:boto:Path: /ffzdqu-49/obj6 2023-09-22T21:23:11.329 DEBUG:boto:Data: 2023-09-22T21:23:11.329 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.329 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.329 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.329 DEBUG:boto:Params: {} 2023-09-22T21:23:11.329 DEBUG:boto:Token: None 2023-09-22T21:23:11.329 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj6 2023-09-22T21:23:11.329 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WUnVeCdA4dQYEe4mkmk4pa0WrrI= 2023-09-22T21:23:11.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WUnVeCdA4dQYEe4mkmk4pa0WrrI=', 'Content-Length': '0'} 2023-09-22T21:23:11.332 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d37995a158ba4e97-00650e05bf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-22T21:23:11.333 DEBUG:boto:path=/ffzdqu-49/obj6 2023-09-22T21:23:11.333 DEBUG:boto:auth_path=/ffzdqu-49/obj6 2023-09-22T21:23:11.333 DEBUG:boto:Method: GET 2023-09-22T21:23:11.333 DEBUG:boto:Path: /ffzdqu-49/obj6 2023-09-22T21:23:11.334 DEBUG:boto:Data: 2023-09-22T21:23:11.334 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.334 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.334 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.334 DEBUG:boto:Params: {} 2023-09-22T21:23:11.335 DEBUG:boto:Token: None 2023-09-22T21:23:11.335 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj6 2023-09-22T21:23:11.335 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZrY24zEdJ3lQ0ioejL1yp6OUFH4= 2023-09-22T21:23:11.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZrY24zEdJ3lQ0ioejL1yp6OUFH4=', 'Content-Length': '0'} 2023-09-22T21:23:11.337 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c57eeb0108f5d2f-00650e05bf-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.378 DEBUG:boto:path=/ffzdqu-49/obj6 2023-09-22T21:23:11.379 DEBUG:boto:auth_path=/ffzdqu-49/obj6 2023-09-22T21:23:11.379 DEBUG:boto:Method: GET 2023-09-22T21:23:11.379 DEBUG:boto:Path: /ffzdqu-49/obj6 2023-09-22T21:23:11.379 DEBUG:boto:Data: 2023-09-22T21:23:11.379 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.379 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.379 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.379 DEBUG:boto:Params: {} 2023-09-22T21:23:11.380 DEBUG:boto:Token: None 2023-09-22T21:23:11.380 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj6 2023-09-22T21:23:11.380 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZrY24zEdJ3lQ0ioejL1yp6OUFH4= 2023-09-22T21:23:11.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZrY24zEdJ3lQ0ioejL1yp6OUFH4=', 'Content-Length': '0'} 2023-09-22T21:23:11.383 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086f71389f26d99df-00650e05bf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.426 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:11.427 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:11.427 DEBUG:boto:path=/ffzdqu-49/?prefix=obj6 2023-09-22T21:23:11.427 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj6 2023-09-22T21:23:11.427 DEBUG:boto:Method: GET 2023-09-22T21:23:11.427 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj6 2023-09-22T21:23:11.427 DEBUG:boto:Data: 2023-09-22T21:23:11.427 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.427 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.427 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.428 DEBUG:boto:Params: {} 2023-09-22T21:23:11.428 DEBUG:boto:Token: None 2023-09-22T21:23:11.428 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/ 2023-09-22T21:23:11.428 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w= 2023-09-22T21:23:11.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w=', 'Content-Length': '0'} 2023-09-22T21:23:11.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc392e026b16069a-00650e05bf-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.433 DEBUG:boto:b'ffzdqu-49obj61000falseobj62023-09-22T21:22:41.752Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:11.433 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:11.433 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:11.433 DEBUG:boto:path=/ffzdqu-49/?prefix=obj6 2023-09-22T21:23:11.434 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj6 2023-09-22T21:23:11.434 DEBUG:boto:Method: GET 2023-09-22T21:23:11.434 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj6 2023-09-22T21:23:11.434 DEBUG:boto:Data: 2023-09-22T21:23:11.434 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.434 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.434 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.434 DEBUG:boto:Params: {} 2023-09-22T21:23:11.435 DEBUG:boto:Token: None 2023-09-22T21:23:11.435 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/ 2023-09-22T21:23:11.435 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w= 2023-09-22T21:23:11.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w=', 'Content-Length': '0'} 2023-09-22T21:23:11.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007282f4589ce5a8c8-00650e05bf-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.440 DEBUG:boto:b'ffzdqu-49obj61000falseobj62023-09-22T21:22:41.752Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:11.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-22T21:23:11.441 DEBUG:boto:path=/ffzdqu-49/obj7 2023-09-22T21:23:11.441 DEBUG:boto:auth_path=/ffzdqu-49/obj7 2023-09-22T21:23:11.441 DEBUG:boto:path=/ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.441 DEBUG:boto:auth_path=/ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.441 DEBUG:boto:Method: GET 2023-09-22T21:23:11.441 DEBUG:boto:Path: /ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.441 DEBUG:boto:Data: 2023-09-22T21:23:11.442 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.442 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.442 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.442 DEBUG:boto:Params: {} 2023-09-22T21:23:11.442 DEBUG:boto:Token: None 2023-09-22T21:23:11.442 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.442 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4aF5vK26zs2gn3qAVKyhVvCVGlI= 2023-09-22T21:23:11.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4aF5vK26zs2gn3qAVKyhVvCVGlI=', 'Content-Length': '0'} 2023-09-22T21:23:11.445 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002ae4c87778bba11-00650e05bf-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.445 DEBUG:boto:path=/ffzdqu-49/obj7 2023-09-22T21:23:11.445 DEBUG:boto:auth_path=/ffzdqu-49/obj7 2023-09-22T21:23:11.446 DEBUG:boto:path=/ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.446 DEBUG:boto:auth_path=/ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.446 DEBUG:boto:Method: GET 2023-09-22T21:23:11.446 DEBUG:boto:Path: /ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.446 DEBUG:boto:Data: 2023-09-22T21:23:11.446 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.446 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.446 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.447 DEBUG:boto:Params: {} 2023-09-22T21:23:11.447 DEBUG:boto:Token: None 2023-09-22T21:23:11.447 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.447 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4aF5vK26zs2gn3qAVKyhVvCVGlI= 2023-09-22T21:23:11.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4aF5vK26zs2gn3qAVKyhVvCVGlI=', 'Content-Length': '0'} 2023-09-22T21:23:11.450 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5adacf343b814f7-00650e05bf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.451 DEBUG:boto:path=/ffzdqu-49/obj7 2023-09-22T21:23:11.451 DEBUG:boto:auth_path=/ffzdqu-49/obj7 2023-09-22T21:23:11.451 DEBUG:boto:path=/ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.451 DEBUG:boto:auth_path=/ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.451 DEBUG:boto:Method: HEAD 2023-09-22T21:23:11.451 DEBUG:boto:Path: /ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.451 DEBUG:boto:Data: 2023-09-22T21:23:11.451 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.452 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.452 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.452 DEBUG:boto:Params: {} 2023-09-22T21:23:11.452 DEBUG:boto:Token: None 2023-09-22T21:23:11.452 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.452 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HpVSkIfEtJAqYg7MytDotPaOwoc= 2023-09-22T21:23:11.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HpVSkIfEtJAqYg7MytDotPaOwoc=', 'Content-Length': '0'} 2023-09-22T21:23:11.455 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c8984c1feb0fb2e-00650e05bf-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.455 DEBUG:boto:path=/ffzdqu-49/obj7 2023-09-22T21:23:11.455 DEBUG:boto:auth_path=/ffzdqu-49/obj7 2023-09-22T21:23:11.455 DEBUG:boto:path=/ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.455 DEBUG:boto:auth_path=/ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.455 DEBUG:boto:Method: HEAD 2023-09-22T21:23:11.456 DEBUG:boto:Path: /ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.456 DEBUG:boto:Data: 2023-09-22T21:23:11.456 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.456 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.456 DEBUG:boto:Params: {} 2023-09-22T21:23:11.456 DEBUG:boto:Token: None 2023-09-22T21:23:11.456 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.456 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HpVSkIfEtJAqYg7MytDotPaOwoc= 2023-09-22T21:23:11.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HpVSkIfEtJAqYg7MytDotPaOwoc=', 'Content-Length': '0'} 2023-09-22T21:23:11.459 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d89caa7d488aef74-00650e05bf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-22T21:23:11.460 DEBUG:boto:path=/ffzdqu-49/obj7 2023-09-22T21:23:11.460 DEBUG:boto:auth_path=/ffzdqu-49/obj7 2023-09-22T21:23:11.460 DEBUG:boto:path=/ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.460 DEBUG:boto:auth_path=/ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.460 DEBUG:boto:Method: GET 2023-09-22T21:23:11.460 DEBUG:boto:Path: /ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.460 DEBUG:boto:Data: 2023-09-22T21:23:11.460 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.461 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.461 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.461 DEBUG:boto:Params: {} 2023-09-22T21:23:11.461 DEBUG:boto:Token: None 2023-09-22T21:23:11.461 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.461 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4aF5vK26zs2gn3qAVKyhVvCVGlI= 2023-09-22T21:23:11.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4aF5vK26zs2gn3qAVKyhVvCVGlI=', 'Content-Length': '0'} 2023-09-22T21:23:11.464 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2d688d57a0c9970-00650e05bf-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.464 DEBUG:boto:path=/ffzdqu-49/obj7 2023-09-22T21:23:11.465 DEBUG:boto:auth_path=/ffzdqu-49/obj7 2023-09-22T21:23:11.465 DEBUG:boto:path=/ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.465 DEBUG:boto:auth_path=/ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.465 DEBUG:boto:Method: GET 2023-09-22T21:23:11.465 DEBUG:boto:Path: /ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.465 DEBUG:boto:Data: 2023-09-22T21:23:11.465 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.465 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.465 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.466 DEBUG:boto:Params: {} 2023-09-22T21:23:11.466 DEBUG:boto:Token: None 2023-09-22T21:23:11.466 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj7?versionId=null 2023-09-22T21:23:11.466 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4aF5vK26zs2gn3qAVKyhVvCVGlI= 2023-09-22T21:23:11.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4aF5vK26zs2gn3qAVKyhVvCVGlI=', 'Content-Length': '0'} 2023-09-22T21:23:11.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a603569db24c5980-00650e05bf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.510 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:11.510 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:11.511 DEBUG:boto:path=/ffzdqu-49/?prefix=obj7 2023-09-22T21:23:11.511 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj7 2023-09-22T21:23:11.511 DEBUG:boto:Method: GET 2023-09-22T21:23:11.511 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj7 2023-09-22T21:23:11.511 DEBUG:boto:Data: 2023-09-22T21:23:11.511 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.512 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.512 DEBUG:boto:Params: {} 2023-09-22T21:23:11.512 DEBUG:boto:Token: None 2023-09-22T21:23:11.512 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/ 2023-09-22T21:23:11.512 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w= 2023-09-22T21:23:11.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w=', 'Content-Length': '0'} 2023-09-22T21:23:11.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002629d84f71c9dfec-00650e05bf-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.517 DEBUG:boto:b'ffzdqu-49obj71000falseobj72023-09-22T21:22:41.767Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:11.517 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:11.518 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:11.518 DEBUG:boto:path=/ffzdqu-49/?prefix=obj7 2023-09-22T21:23:11.518 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj7 2023-09-22T21:23:11.518 DEBUG:boto:Method: GET 2023-09-22T21:23:11.518 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj7 2023-09-22T21:23:11.518 DEBUG:boto:Data: 2023-09-22T21:23:11.518 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.518 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.518 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.519 DEBUG:boto:Params: {} 2023-09-22T21:23:11.519 DEBUG:boto:Token: None 2023-09-22T21:23:11.519 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/ 2023-09-22T21:23:11.519 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w= 2023-09-22T21:23:11.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w=', 'Content-Length': '0'} 2023-09-22T21:23:11.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0e1d174ef78d191-00650e05bf-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.524 DEBUG:boto:b'ffzdqu-49obj71000falseobj72023-09-22T21:22:41.767Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:11.524 DEBUG:boto:path=/ffzdqu-49/obj7 2023-09-22T21:23:11.524 DEBUG:boto:auth_path=/ffzdqu-49/obj7 2023-09-22T21:23:11.524 DEBUG:boto:Method: HEAD 2023-09-22T21:23:11.524 DEBUG:boto:Path: /ffzdqu-49/obj7 2023-09-22T21:23:11.525 DEBUG:boto:Data: 2023-09-22T21:23:11.525 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.525 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.525 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.525 DEBUG:boto:Params: {} 2023-09-22T21:23:11.525 DEBUG:boto:Token: None 2023-09-22T21:23:11.525 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj7 2023-09-22T21:23:11.525 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:inMjatnALVlTnUTxyU0Q/5bWSd4= 2023-09-22T21:23:11.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:inMjatnALVlTnUTxyU0Q/5bWSd4=', 'Content-Length': '0'} 2023-09-22T21:23:11.528 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae359417c647952c-00650e05bf-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.528 DEBUG:boto:path=/ffzdqu-49/obj7 2023-09-22T21:23:11.528 DEBUG:boto:auth_path=/ffzdqu-49/obj7 2023-09-22T21:23:11.529 DEBUG:boto:Method: HEAD 2023-09-22T21:23:11.529 DEBUG:boto:Path: /ffzdqu-49/obj7 2023-09-22T21:23:11.529 DEBUG:boto:Data: 2023-09-22T21:23:11.529 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.529 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.529 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.529 DEBUG:boto:Params: {} 2023-09-22T21:23:11.529 DEBUG:boto:Token: None 2023-09-22T21:23:11.529 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj7 2023-09-22T21:23:11.530 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:inMjatnALVlTnUTxyU0Q/5bWSd4= 2023-09-22T21:23:11.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:inMjatnALVlTnUTxyU0Q/5bWSd4=', 'Content-Length': '0'} 2023-09-22T21:23:11.532 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a601a692d4ba9a4-00650e05bf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-22T21:23:11.533 DEBUG:boto:path=/ffzdqu-49/obj7 2023-09-22T21:23:11.533 DEBUG:boto:auth_path=/ffzdqu-49/obj7 2023-09-22T21:23:11.533 DEBUG:boto:Method: GET 2023-09-22T21:23:11.533 DEBUG:boto:Path: /ffzdqu-49/obj7 2023-09-22T21:23:11.533 DEBUG:boto:Data: 2023-09-22T21:23:11.534 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.534 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.534 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.534 DEBUG:boto:Params: {} 2023-09-22T21:23:11.534 DEBUG:boto:Token: None 2023-09-22T21:23:11.534 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj7 2023-09-22T21:23:11.534 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:O1DkBAELhhyoHjlTtEaUnCfruWM= 2023-09-22T21:23:11.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:O1DkBAELhhyoHjlTtEaUnCfruWM=', 'Content-Length': '0'} 2023-09-22T21:23:11.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f94831e4006f4c5-00650e05bf-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.578 DEBUG:boto:path=/ffzdqu-49/obj7 2023-09-22T21:23:11.579 DEBUG:boto:auth_path=/ffzdqu-49/obj7 2023-09-22T21:23:11.579 DEBUG:boto:Method: GET 2023-09-22T21:23:11.579 DEBUG:boto:Path: /ffzdqu-49/obj7 2023-09-22T21:23:11.579 DEBUG:boto:Data: 2023-09-22T21:23:11.579 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.579 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.579 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.579 DEBUG:boto:Params: {} 2023-09-22T21:23:11.579 DEBUG:boto:Token: None 2023-09-22T21:23:11.580 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj7 2023-09-22T21:23:11.580 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:O1DkBAELhhyoHjlTtEaUnCfruWM= 2023-09-22T21:23:11.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:O1DkBAELhhyoHjlTtEaUnCfruWM=', 'Content-Length': '0'} 2023-09-22T21:23:11.583 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a51cdee6806b18ae-00650e05bf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.626 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:11.627 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:11.627 DEBUG:boto:path=/ffzdqu-49/?prefix=obj7 2023-09-22T21:23:11.627 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj7 2023-09-22T21:23:11.627 DEBUG:boto:Method: GET 2023-09-22T21:23:11.627 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj7 2023-09-22T21:23:11.627 DEBUG:boto:Data: 2023-09-22T21:23:11.627 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.627 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.627 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.628 DEBUG:boto:Params: {} 2023-09-22T21:23:11.628 DEBUG:boto:Token: None 2023-09-22T21:23:11.628 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/ 2023-09-22T21:23:11.628 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w= 2023-09-22T21:23:11.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w=', 'Content-Length': '0'} 2023-09-22T21:23:11.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022aea3776421cc44-00650e05bf-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.633 DEBUG:boto:b'ffzdqu-49obj71000falseobj72023-09-22T21:22:41.767Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:11.633 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:11.634 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:11.634 DEBUG:boto:path=/ffzdqu-49/?prefix=obj7 2023-09-22T21:23:11.634 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj7 2023-09-22T21:23:11.634 DEBUG:boto:Method: GET 2023-09-22T21:23:11.634 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj7 2023-09-22T21:23:11.634 DEBUG:boto:Data: 2023-09-22T21:23:11.635 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.635 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.635 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.635 DEBUG:boto:Params: {} 2023-09-22T21:23:11.635 DEBUG:boto:Token: None 2023-09-22T21:23:11.636 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/ 2023-09-22T21:23:11.636 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w= 2023-09-22T21:23:11.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w=', 'Content-Length': '0'} 2023-09-22T21:23:11.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000132f86b05da995d9-00650e05bf-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.682 DEBUG:boto:b'ffzdqu-49obj71000falseobj72023-09-22T21:22:41.767Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:11.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-22T21:23:11.683 DEBUG:boto:path=/ffzdqu-49/obj8 2023-09-22T21:23:11.683 DEBUG:boto:auth_path=/ffzdqu-49/obj8 2023-09-22T21:23:11.683 DEBUG:boto:path=/ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.684 DEBUG:boto:auth_path=/ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.684 DEBUG:boto:Method: GET 2023-09-22T21:23:11.684 DEBUG:boto:Path: /ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.684 DEBUG:boto:Data: 2023-09-22T21:23:11.684 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.684 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.684 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.684 DEBUG:boto:Params: {} 2023-09-22T21:23:11.685 DEBUG:boto:Token: None 2023-09-22T21:23:11.685 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.685 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+i8z5PMZaFIm4E1u+kdbfWhzyqw= 2023-09-22T21:23:11.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+i8z5PMZaFIm4E1u+kdbfWhzyqw=', 'Content-Length': '0'} 2023-09-22T21:23:11.688 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eff2fa7a326c9d86-00650e05bf-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.688 DEBUG:boto:path=/ffzdqu-49/obj8 2023-09-22T21:23:11.688 DEBUG:boto:auth_path=/ffzdqu-49/obj8 2023-09-22T21:23:11.688 DEBUG:boto:path=/ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.689 DEBUG:boto:auth_path=/ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.689 DEBUG:boto:Method: GET 2023-09-22T21:23:11.689 DEBUG:boto:Path: /ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.689 DEBUG:boto:Data: 2023-09-22T21:23:11.689 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.689 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.689 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.689 DEBUG:boto:Params: {} 2023-09-22T21:23:11.689 DEBUG:boto:Token: None 2023-09-22T21:23:11.690 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.690 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+i8z5PMZaFIm4E1u+kdbfWhzyqw= 2023-09-22T21:23:11.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+i8z5PMZaFIm4E1u+kdbfWhzyqw=', 'Content-Length': '0'} 2023-09-22T21:23:11.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a244cf501e1f70f-00650e05bf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.693 DEBUG:boto:path=/ffzdqu-49/obj8 2023-09-22T21:23:11.693 DEBUG:boto:auth_path=/ffzdqu-49/obj8 2023-09-22T21:23:11.693 DEBUG:boto:path=/ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.694 DEBUG:boto:auth_path=/ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.694 DEBUG:boto:Method: HEAD 2023-09-22T21:23:11.694 DEBUG:boto:Path: /ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.694 DEBUG:boto:Data: 2023-09-22T21:23:11.694 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.694 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.694 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.695 DEBUG:boto:Params: {} 2023-09-22T21:23:11.695 DEBUG:boto:Token: None 2023-09-22T21:23:11.695 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.695 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oD8ktzdzYGtGhnlyRJr65Mop0KM= 2023-09-22T21:23:11.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oD8ktzdzYGtGhnlyRJr65Mop0KM=', 'Content-Length': '0'} 2023-09-22T21:23:11.698 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b870841ee437d25-00650e05bf-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.698 DEBUG:boto:path=/ffzdqu-49/obj8 2023-09-22T21:23:11.698 DEBUG:boto:auth_path=/ffzdqu-49/obj8 2023-09-22T21:23:11.698 DEBUG:boto:path=/ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.698 DEBUG:boto:auth_path=/ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.699 DEBUG:boto:Method: HEAD 2023-09-22T21:23:11.699 DEBUG:boto:Path: /ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.699 DEBUG:boto:Data: 2023-09-22T21:23:11.699 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.699 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.699 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.699 DEBUG:boto:Params: {} 2023-09-22T21:23:11.699 DEBUG:boto:Token: None 2023-09-22T21:23:11.699 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.700 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oD8ktzdzYGtGhnlyRJr65Mop0KM= 2023-09-22T21:23:11.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oD8ktzdzYGtGhnlyRJr65Mop0KM=', 'Content-Length': '0'} 2023-09-22T21:23:11.702 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0500316c8b92039-00650e05bf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-22T21:23:11.703 DEBUG:boto:path=/ffzdqu-49/obj8 2023-09-22T21:23:11.703 DEBUG:boto:auth_path=/ffzdqu-49/obj8 2023-09-22T21:23:11.703 DEBUG:boto:path=/ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.703 DEBUG:boto:auth_path=/ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.703 DEBUG:boto:Method: GET 2023-09-22T21:23:11.704 DEBUG:boto:Path: /ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.704 DEBUG:boto:Data: 2023-09-22T21:23:11.704 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.704 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.704 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.704 DEBUG:boto:Params: {} 2023-09-22T21:23:11.704 DEBUG:boto:Token: None 2023-09-22T21:23:11.704 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.705 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+i8z5PMZaFIm4E1u+kdbfWhzyqw= 2023-09-22T21:23:11.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+i8z5PMZaFIm4E1u+kdbfWhzyqw=', 'Content-Length': '0'} 2023-09-22T21:23:11.707 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001176e524e3a1c78-00650e05bf-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.708 DEBUG:boto:path=/ffzdqu-49/obj8 2023-09-22T21:23:11.708 DEBUG:boto:auth_path=/ffzdqu-49/obj8 2023-09-22T21:23:11.708 DEBUG:boto:path=/ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.708 DEBUG:boto:auth_path=/ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.709 DEBUG:boto:Method: GET 2023-09-22T21:23:11.709 DEBUG:boto:Path: /ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.709 DEBUG:boto:Data: 2023-09-22T21:23:11.709 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.709 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.709 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.709 DEBUG:boto:Params: {} 2023-09-22T21:23:11.709 DEBUG:boto:Token: None 2023-09-22T21:23:11.709 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj8?versionId=null 2023-09-22T21:23:11.710 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+i8z5PMZaFIm4E1u+kdbfWhzyqw= 2023-09-22T21:23:11.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+i8z5PMZaFIm4E1u+kdbfWhzyqw=', 'Content-Length': '0'} 2023-09-22T21:23:11.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f3f7994e390b55c-00650e05bf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.754 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:11.755 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:11.755 DEBUG:boto:path=/ffzdqu-49/?prefix=obj8 2023-09-22T21:23:11.755 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj8 2023-09-22T21:23:11.755 DEBUG:boto:Method: GET 2023-09-22T21:23:11.755 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj8 2023-09-22T21:23:11.755 DEBUG:boto:Data: 2023-09-22T21:23:11.755 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.755 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.756 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.756 DEBUG:boto:Params: {} 2023-09-22T21:23:11.756 DEBUG:boto:Token: None 2023-09-22T21:23:11.756 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/ 2023-09-22T21:23:11.756 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w= 2023-09-22T21:23:11.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w=', 'Content-Length': '0'} 2023-09-22T21:23:11.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e7846b41af497a5-00650e05bf-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.761 DEBUG:boto:b'ffzdqu-49obj81000falseobj82023-09-22T21:22:41.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:11.762 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:11.762 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:11.762 DEBUG:boto:path=/ffzdqu-49/?prefix=obj8 2023-09-22T21:23:11.762 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj8 2023-09-22T21:23:11.763 DEBUG:boto:Method: GET 2023-09-22T21:23:11.763 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj8 2023-09-22T21:23:11.763 DEBUG:boto:Data: 2023-09-22T21:23:11.763 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.763 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.763 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.764 DEBUG:boto:Params: {} 2023-09-22T21:23:11.764 DEBUG:boto:Token: None 2023-09-22T21:23:11.764 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/ 2023-09-22T21:23:11.764 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w= 2023-09-22T21:23:11.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w=', 'Content-Length': '0'} 2023-09-22T21:23:11.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3c69fdd9b3b541e-00650e05bf-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.814 DEBUG:boto:b'ffzdqu-49obj81000falseobj82023-09-22T21:22:41.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:11.815 DEBUG:boto:path=/ffzdqu-49/obj8 2023-09-22T21:23:11.815 DEBUG:boto:auth_path=/ffzdqu-49/obj8 2023-09-22T21:23:11.815 DEBUG:boto:Method: HEAD 2023-09-22T21:23:11.815 DEBUG:boto:Path: /ffzdqu-49/obj8 2023-09-22T21:23:11.815 DEBUG:boto:Data: 2023-09-22T21:23:11.815 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.815 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.816 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.816 DEBUG:boto:Params: {} 2023-09-22T21:23:11.816 DEBUG:boto:Token: None 2023-09-22T21:23:11.816 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj8 2023-09-22T21:23:11.816 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VUegp5vn4flBlsIhNkUcieXt0ng= 2023-09-22T21:23:11.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VUegp5vn4flBlsIhNkUcieXt0ng=', 'Content-Length': '0'} 2023-09-22T21:23:11.819 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000385094c6836d51f0-00650e05bf-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.819 DEBUG:boto:path=/ffzdqu-49/obj8 2023-09-22T21:23:11.819 DEBUG:boto:auth_path=/ffzdqu-49/obj8 2023-09-22T21:23:11.820 DEBUG:boto:Method: HEAD 2023-09-22T21:23:11.820 DEBUG:boto:Path: /ffzdqu-49/obj8 2023-09-22T21:23:11.820 DEBUG:boto:Data: 2023-09-22T21:23:11.820 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.820 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.820 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.820 DEBUG:boto:Params: {} 2023-09-22T21:23:11.820 DEBUG:boto:Token: None 2023-09-22T21:23:11.821 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj8 2023-09-22T21:23:11.821 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VUegp5vn4flBlsIhNkUcieXt0ng= 2023-09-22T21:23:11.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VUegp5vn4flBlsIhNkUcieXt0ng=', 'Content-Length': '0'} 2023-09-22T21:23:11.823 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000142fa7a9b892cb7f-00650e05bf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-22T21:23:11.824 DEBUG:boto:path=/ffzdqu-49/obj8 2023-09-22T21:23:11.824 DEBUG:boto:auth_path=/ffzdqu-49/obj8 2023-09-22T21:23:11.824 DEBUG:boto:Method: GET 2023-09-22T21:23:11.824 DEBUG:boto:Path: /ffzdqu-49/obj8 2023-09-22T21:23:11.824 DEBUG:boto:Data: 2023-09-22T21:23:11.824 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.825 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.825 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.825 DEBUG:boto:Params: {} 2023-09-22T21:23:11.825 DEBUG:boto:Token: None 2023-09-22T21:23:11.825 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj8 2023-09-22T21:23:11.825 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:j9QkUTtyFoIGCyUxJYF1j9ra9kc= 2023-09-22T21:23:11.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:j9QkUTtyFoIGCyUxJYF1j9ra9kc=', 'Content-Length': '0'} 2023-09-22T21:23:11.828 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009aff6b3301b28045-00650e05bf-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.828 DEBUG:boto:path=/ffzdqu-49/obj8 2023-09-22T21:23:11.828 DEBUG:boto:auth_path=/ffzdqu-49/obj8 2023-09-22T21:23:11.829 DEBUG:boto:Method: GET 2023-09-22T21:23:11.829 DEBUG:boto:Path: /ffzdqu-49/obj8 2023-09-22T21:23:11.829 DEBUG:boto:Data: 2023-09-22T21:23:11.829 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.829 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.829 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.829 DEBUG:boto:Params: {} 2023-09-22T21:23:11.829 DEBUG:boto:Token: None 2023-09-22T21:23:11.829 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/obj8 2023-09-22T21:23:11.830 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:j9QkUTtyFoIGCyUxJYF1j9ra9kc= 2023-09-22T21:23:11.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:j9QkUTtyFoIGCyUxJYF1j9ra9kc=', 'Content-Length': '0'} 2023-09-22T21:23:11.832 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:22:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2b721fee-a149-45da-925b-502d5a3d6853:ffzdqu-49:f132bdb9-20a7-4073-a932-8f99da20691e.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a81ae4d7dae32669-00650e05bf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.874 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:11.875 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:11.875 DEBUG:boto:path=/ffzdqu-49/?prefix=obj8 2023-09-22T21:23:11.875 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj8 2023-09-22T21:23:11.875 DEBUG:boto:Method: GET 2023-09-22T21:23:11.875 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj8 2023-09-22T21:23:11.875 DEBUG:boto:Data: 2023-09-22T21:23:11.875 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.876 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:11.876 DEBUG:boto:Port: 8001 2023-09-22T21:23:11.876 DEBUG:boto:Params: {} 2023-09-22T21:23:11.876 DEBUG:boto:Token: None 2023-09-22T21:23:11.876 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/ 2023-09-22T21:23:11.876 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w= 2023-09-22T21:23:11.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w=', 'Content-Length': '0'} 2023-09-22T21:23:11.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6fdacb4d4b7a646-00650e05bf-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.881 DEBUG:boto:b'ffzdqu-49obj81000falseobj82023-09-22T21:22:41.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:11.882 DEBUG:boto:path=/ffzdqu-49/ 2023-09-22T21:23:11.882 DEBUG:boto:auth_path=/ffzdqu-49/ 2023-09-22T21:23:11.882 DEBUG:boto:path=/ffzdqu-49/?prefix=obj8 2023-09-22T21:23:11.882 DEBUG:boto:auth_path=/ffzdqu-49/?prefix=obj8 2023-09-22T21:23:11.882 DEBUG:boto:Method: GET 2023-09-22T21:23:11.882 DEBUG:boto:Path: /ffzdqu-49/?prefix=obj8 2023-09-22T21:23:11.882 DEBUG:boto:Data: 2023-09-22T21:23:11.883 DEBUG:boto:Headers: {} 2023-09-22T21:23:11.883 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:11.883 DEBUG:boto:Port: 8000 2023-09-22T21:23:11.883 DEBUG:boto:Params: {} 2023-09-22T21:23:11.883 DEBUG:boto:Token: None 2023-09-22T21:23:11.883 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:11 GMT /ffzdqu-49/ 2023-09-22T21:23:11.883 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w= 2023-09-22T21:23:11.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qEPA93s/YD84pY6yZqgngR+t+8w=', 'Content-Length': '0'} 2023-09-22T21:23:11.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e0ad2b0f8f23d19-00650e05bf-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:11.934 DEBUG:boto:b'ffzdqu-49obj81000falseobj82023-09-22T21:22:41.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:23:11.935 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-49 zones={a2, a1} 2023-09-22T21:23:11.935 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-09-22T21:23:11.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:23:11.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:23:11.935 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:23:11.979 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:23:11.979 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:23:12.043 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"00000000000000000000:00000000000000004595","last_update":"2023-09-22T21:23:00.229110Z"},{"marker":"00000000000000000000:00000000000000009807","last_update":"2023-09-22T21:23:01.450862Z"},{"marker":"00000000000000000000:00000000000000006851","last_update":"2023-09-22T21:22:42.099350Z"},{"marker":"00000000000000000000:00000000000000006576","last_update":"2023-09-22T21:23:00.173277Z"}] 2023-09-22T21:23:12.043 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000004595', 1: '00000000000000000000:00000000000000009807', 2: '00000000000000000000:00000000000000006851', 3: '00000000000000000000:00000000000000006576'} 2023-09-22T21:23:12.044 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-22T21:23:12.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:23:12.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:23:12.044 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:23:12.128 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:23:12.128 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:23:12.183 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004595","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:23:00.229110Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009807","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:23:01.450862Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006851","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:42.099350Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006576","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:23:00.173277Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:23:12Z"} 2023-09-22T21:23:12.184 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17572445636104201039},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004595","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:23:00.229110Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009807","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:23:01.450862Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006851","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:42.099350Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006576","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:23:00.173277Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:23:12Z"} 2023-09-22T21:23:12.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004595', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:23:00.229110Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009807', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:23:01.450862Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006851', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:22:42.099350Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006576', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:23:00.173277Z'}}] 2023-09-22T21:23:12.184 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004595', 1: '00000000000000000000:00000000000000009807', 2: '00000000000000000000:00000000000000006851', 3: '00000000000000000000:00000000000000006576'} 2023-09-22T21:23:12.184 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004595', 1: '00000000000000000000:00000000000000009807', 2: '00000000000000000000:00000000000000006851', 3: '00000000000000000000:00000000000000006576'} 2023-09-22T21:23:12.184 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-22T21:23:12.184 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-09-22T21:23:12.184 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:23:12.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:23:12.185 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:23:12.264 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:23:12.265 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:23:12.326 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"00000000000000000000:00000000000000003185","last_update":"2023-09-22T21:23:03.423368Z"},{"marker":"00000000000000000000:00000000000000008256","last_update":"2023-09-22T21:23:03.426846Z"},{"marker":"00000000000000000000:00000000000000004174","last_update":"2023-09-22T21:23:03.430116Z"},{"marker":"00000000000000000000:00000000000000004597","last_update":"2023-09-22T21:23:03.433810Z"}] 2023-09-22T21:23:12.326 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000003185', 1: '00000000000000000000:00000000000000008256', 2: '00000000000000000000:00000000000000004174', 3: '00000000000000000000:00000000000000004597'} 2023-09-22T21:23:12.326 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-09-22T21:23:12.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:23:12.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:23:12.329 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:23:12.406 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:23:12.406 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:23:12.460 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10476880264182873081},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002621","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:41.413053Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008256","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:23:03.426846Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003328","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:37.498449Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003750","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:41.419805Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:23:12Z"} 2023-09-22T21:23:12.460 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10476880264182873081},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002621","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:41.413053Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008256","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:23:03.426846Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003328","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:37.498449Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003750","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:41.419805Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:23:12Z"} 2023-09-22T21:23:12.461 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002621', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:22:41.413053Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008256', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:23:03.426846Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003328', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:22:37.498449Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003750', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:22:41.419805Z'}}] 2023-09-22T21:23:12.461 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003185', 1: '00000000000000000000:00000000000000008256', 2: '00000000000000000000:00000000000000004174', 3: '00000000000000000000:00000000000000004597'} 2023-09-22T21:23:12.461 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002621', 1: '00000000000000000000:00000000000000008256', 2: '00000000000000000000:00000000000000003328', 3: '00000000000000000000:00000000000000003750'} 2023-09-22T21:23:12.461 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000003185 target=00000000000000000000:00000000000000002621, shard=2 master=00000000000000000000:00000000000000004174 target=00000000000000000000:00000000000000003328, shard=3 master=00000000000000000000:00000000000000004597 target=00000000000000000000:00000000000000003750 2023-09-22T21:23:17.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:23:17.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:23:17.462 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:23:17.510 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:23:17.510 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:23:17.565 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10476880264182873081},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002621","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:41.413053Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008256","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:23:03.426846Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003328","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:37.498449Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003750","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:41.419805Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:23:17Z"} 2023-09-22T21:23:17.566 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10476880264182873081},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002621","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:41.413053Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008256","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:23:03.426846Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003328","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:37.498449Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003750","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:22:41.419805Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:23:17Z"} 2023-09-22T21:23:17.566 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002621', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:22:41.413053Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008256', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:23:03.426846Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003328', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:22:37.498449Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003750', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:22:41.419805Z'}}] 2023-09-22T21:23:17.566 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003185', 1: '00000000000000000000:00000000000000008256', 2: '00000000000000000000:00000000000000004174', 3: '00000000000000000000:00000000000000004597'} 2023-09-22T21:23:17.566 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002621', 1: '00000000000000000000:00000000000000008256', 2: '00000000000000000000:00000000000000003328', 3: '00000000000000000000:00000000000000003750'} 2023-09-22T21:23:17.567 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000003185 target=00000000000000000000:00000000000000002621, shard=2 master=00000000000000000000:00000000000000004174 target=00000000000000000000:00000000000000003328, shard=3 master=00000000000000000000:00000000000000004597 target=00000000000000000000:00000000000000003750 2023-09-22T21:23:22.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:23:22.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:23:22.568 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:23:22.613 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:23:22.613 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:23:22.669 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10476880264182873081},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003185","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:23:03.423368Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008256","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:23:03.426846Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004174","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:23:03.430116Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004597","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:23:03.433810Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:23:22Z"} 2023-09-22T21:23:22.669 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10476880264182873081},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003185","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:23:03.423368Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008256","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:23:03.426846Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004174","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:23:03.430116Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004597","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:23:03.433810Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:23:22Z"} 2023-09-22T21:23:22.669 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003185', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:23:03.423368Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008256', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:23:03.426846Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004174', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:23:03.430116Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004597', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:23:03.433810Z'}}] 2023-09-22T21:23:22.670 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003185', 1: '00000000000000000000:00000000000000008256', 2: '00000000000000000000:00000000000000004174', 3: '00000000000000000000:00000000000000004597'} 2023-09-22T21:23:22.670 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003185', 1: '00000000000000000000:00000000000000008256', 2: '00000000000000000000:00000000000000004174', 3: '00000000000000000000:00000000000000004597'} 2023-09-22T21:23:22.670 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-09-22T21:23:22.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2023-09-22T21:23:22.671 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-50 2023-09-22T21:23:22.671 DEBUG:boto:path=/ffzdqu-50/ 2023-09-22T21:23:22.672 DEBUG:boto:auth_path=/ffzdqu-50/ 2023-09-22T21:23:22.672 DEBUG:boto:Method: PUT 2023-09-22T21:23:22.672 DEBUG:boto:Path: /ffzdqu-50/ 2023-09-22T21:23:22.672 DEBUG:boto:Data: 2023-09-22T21:23:22.672 DEBUG:boto:Headers: {} 2023-09-22T21:23:22.672 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:22.672 DEBUG:boto:Port: 8000 2023-09-22T21:23:22.672 DEBUG:boto:Params: {} 2023-09-22T21:23:22.673 DEBUG:boto:Token: None 2023-09-22T21:23:22.673 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:23:22 GMT /ffzdqu-50/ 2023-09-22T21:23:22.673 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7ti9i7q7l7XZYHWDsUK0kfCIY1s= 2023-09-22T21:23:22.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7ti9i7q7l7XZYHWDsUK0kfCIY1s=', 'Content-Length': '0'} 2023-09-22T21:23:22.691 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ce8d5a1e89fd0cac-00650e05ca-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:23:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:22.691 INFO:rgw_multi.tests:create bucket zone=a2 name=ffzdqu-51 2023-09-22T21:23:22.691 DEBUG:boto:path=/ffzdqu-51/ 2023-09-22T21:23:22.692 DEBUG:boto:auth_path=/ffzdqu-51/ 2023-09-22T21:23:22.692 DEBUG:boto:Method: PUT 2023-09-22T21:23:22.692 DEBUG:boto:Path: /ffzdqu-51/ 2023-09-22T21:23:22.692 DEBUG:boto:Data: 2023-09-22T21:23:22.692 DEBUG:boto:Headers: {} 2023-09-22T21:23:22.692 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:23:22.692 DEBUG:boto:Port: 8001 2023-09-22T21:23:22.692 DEBUG:boto:Params: {} 2023-09-22T21:23:22.692 DEBUG:boto:Token: None 2023-09-22T21:23:22.693 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:23:22 GMT /ffzdqu-51/ 2023-09-22T21:23:22.693 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uG9kFxt7B8ZQp5dHdhJWotwKUDg= 2023-09-22T21:23:22.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uG9kFxt7B8ZQp5dHdhJWotwKUDg=', 'Content-Length': '0'} 2023-09-22T21:23:22.728 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000588ba0cf8ba80be6-00650e05ca-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:23:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:22.728 DEBUG:boto:path=/ffzdqu-50/MULTIPART 2023-09-22T21:23:22.728 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART 2023-09-22T21:23:22.728 DEBUG:boto:path=/ffzdqu-50/MULTIPART?uploads 2023-09-22T21:23:22.728 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART?uploads 2023-09-22T21:23:22.728 DEBUG:boto:Method: POST 2023-09-22T21:23:22.729 DEBUG:boto:Path: /ffzdqu-50/MULTIPART?uploads 2023-09-22T21:23:22.729 DEBUG:boto:Data: 2023-09-22T21:23:22.729 DEBUG:boto:Headers: {} 2023-09-22T21:23:22.729 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:22.729 DEBUG:boto:Port: 8000 2023-09-22T21:23:22.729 DEBUG:boto:Params: {} 2023-09-22T21:23:22.729 DEBUG:boto:Token: None 2023-09-22T21:23:22.729 DEBUG:boto:StringToSign: POST Fri, 22 Sep 2023 21:23:22 GMT /ffzdqu-50/MULTIPART?uploads 2023-09-22T21:23:22.730 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IZZhO99SUscARwkAhuCEJdiZLgQ= 2023-09-22T21:23:22.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IZZhO99SUscARwkAhuCEJdiZLgQ=', 'Content-Length': '0'} 2023-09-22T21:23:23.925 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000071a4efe44071025d-00650e05ca-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '249'), ('Date', 'Fri, 22 Sep 2023 21:23:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:23.925 DEBUG:boto:b'ffzdqu-50MULTIPART2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF' 2023-09-22T21:23:23.954 DEBUG:boto:path=/ffzdqu-50/MULTIPART 2023-09-22T21:23:23.955 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART 2023-09-22T21:23:23.955 DEBUG:boto:path=/ffzdqu-50/MULTIPART?uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF&partNumber=1 2023-09-22T21:23:23.955 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART?uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF&partNumber=1 2023-09-22T21:23:23.955 DEBUG:boto:Method: PUT 2023-09-22T21:23:23.955 DEBUG:boto:Path: /ffzdqu-50/MULTIPART?uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF&partNumber=1 2023-09-22T21:23:23.955 DEBUG:boto:Data: 2023-09-22T21:23:23.955 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-09-22T21:23:23.955 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:23.956 DEBUG:boto:Port: 8000 2023-09-22T21:23:23.956 DEBUG:boto:Params: {} 2023-09-22T21:23:23.956 DEBUG:boto:Token: None 2023-09-22T21:23:23.956 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Fri, 22 Sep 2023 21:23:23 GMT /ffzdqu-50/MULTIPART?partNumber=1&uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF 2023-09-22T21:23:23.956 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+cdRSainaJVZ4913iJoYY+1O+iU= 2023-09-22T21:23:23.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:23:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+cdRSainaJVZ4913iJoYY+1O+iU='} 2023-09-22T21:23:24.014 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005321b6dbc852cb76-00650e05cb-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:24.046 DEBUG:boto:path=/ffzdqu-50/MULTIPART 2023-09-22T21:23:24.046 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART 2023-09-22T21:23:24.046 DEBUG:boto:path=/ffzdqu-50/MULTIPART?uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF&partNumber=2 2023-09-22T21:23:24.046 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART?uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF&partNumber=2 2023-09-22T21:23:24.046 DEBUG:boto:Method: PUT 2023-09-22T21:23:24.046 DEBUG:boto:Path: /ffzdqu-50/MULTIPART?uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF&partNumber=2 2023-09-22T21:23:24.047 DEBUG:boto:Data: 2023-09-22T21:23:24.047 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-09-22T21:23:24.047 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:24.047 DEBUG:boto:Port: 8000 2023-09-22T21:23:24.047 DEBUG:boto:Params: {} 2023-09-22T21:23:24.047 DEBUG:boto:Token: None 2023-09-22T21:23:24.047 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Fri, 22 Sep 2023 21:23:24 GMT /ffzdqu-50/MULTIPART?partNumber=2&uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF 2023-09-22T21:23:24.048 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AMp93QzmMqBxSb91h5FsBQlfMU8= 2023-09-22T21:23:24.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:23:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AMp93QzmMqBxSb91h5FsBQlfMU8='} 2023-09-22T21:23:24.104 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c466b0e152359dfb-00650e05cc-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:24.124 DEBUG:boto:path=/ffzdqu-50/MULTIPART 2023-09-22T21:23:24.124 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART 2023-09-22T21:23:24.124 DEBUG:boto:path=/ffzdqu-50/MULTIPART?uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF&partNumber=3 2023-09-22T21:23:24.124 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART?uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF&partNumber=3 2023-09-22T21:23:24.125 DEBUG:boto:Method: PUT 2023-09-22T21:23:24.125 DEBUG:boto:Path: /ffzdqu-50/MULTIPART?uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF&partNumber=3 2023-09-22T21:23:24.125 DEBUG:boto:Data: 2023-09-22T21:23:24.125 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-09-22T21:23:24.125 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:24.125 DEBUG:boto:Port: 8000 2023-09-22T21:23:24.125 DEBUG:boto:Params: {} 2023-09-22T21:23:24.125 DEBUG:boto:Token: None 2023-09-22T21:23:24.125 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Fri, 22 Sep 2023 21:23:24 GMT /ffzdqu-50/MULTIPART?partNumber=3&uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF 2023-09-22T21:23:24.126 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PJ5jtQqT9pCEXxXyP2Xp8luXKvg= 2023-09-22T21:23:24.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:23:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PJ5jtQqT9pCEXxXyP2Xp8luXKvg='} 2023-09-22T21:23:24.179 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f9c5048f044c50ab-00650e05cc-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:24.198 DEBUG:boto:path=/ffzdqu-50/MULTIPART 2023-09-22T21:23:24.198 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART 2023-09-22T21:23:24.198 DEBUG:boto:path=/ffzdqu-50/MULTIPART?uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF&partNumber=4 2023-09-22T21:23:24.198 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART?uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF&partNumber=4 2023-09-22T21:23:24.198 DEBUG:boto:Method: PUT 2023-09-22T21:23:24.199 DEBUG:boto:Path: /ffzdqu-50/MULTIPART?uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF&partNumber=4 2023-09-22T21:23:24.199 DEBUG:boto:Data: 2023-09-22T21:23:24.199 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-09-22T21:23:24.199 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:24.199 DEBUG:boto:Port: 8000 2023-09-22T21:23:24.199 DEBUG:boto:Params: {} 2023-09-22T21:23:24.199 DEBUG:boto:Token: None 2023-09-22T21:23:24.199 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Fri, 22 Sep 2023 21:23:24 GMT /ffzdqu-50/MULTIPART?partNumber=4&uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF 2023-09-22T21:23:24.200 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qFViAP7EKaz18Hs+7CjB7R7aqjo= 2023-09-22T21:23:24.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:23:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qFViAP7EKaz18Hs+7CjB7R7aqjo='} 2023-09-22T21:23:24.253 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004ecd326dc8203539-00650e05cc-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:24.254 DEBUG:boto:path=/ffzdqu-50/MULTIPART 2023-09-22T21:23:24.254 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART 2023-09-22T21:23:24.254 DEBUG:boto:path=/ffzdqu-50/MULTIPART?uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF 2023-09-22T21:23:24.254 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART?uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF 2023-09-22T21:23:24.254 DEBUG:boto:Method: GET 2023-09-22T21:23:24.254 DEBUG:boto:Path: /ffzdqu-50/MULTIPART?uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF 2023-09-22T21:23:24.254 DEBUG:boto:Data: 2023-09-22T21:23:24.255 DEBUG:boto:Headers: {} 2023-09-22T21:23:24.255 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:24.255 DEBUG:boto:Port: 8000 2023-09-22T21:23:24.255 DEBUG:boto:Params: {} 2023-09-22T21:23:24.255 DEBUG:boto:Token: None 2023-09-22T21:23:24.255 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:23:24 GMT /ffzdqu-50/MULTIPART?uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF 2023-09-22T21:23:24.255 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:duv9EsGHup18iAPVPBVP81Z0Z0E= 2023-09-22T21:23:24.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:duv9EsGHup18iAPVPBVP81Z0Z0E=', 'Content-Length': '0'} 2023-09-22T21:23:24.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051c242fe8a70f386-00650e05cc-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:24.307 DEBUG:boto:path=/ffzdqu-50/MULTIPART 2023-09-22T21:23:24.307 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART 2023-09-22T21:23:24.307 DEBUG:boto:path=/ffzdqu-50/MULTIPART?uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF 2023-09-22T21:23:24.307 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART?uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF 2023-09-22T21:23:24.307 DEBUG:boto:Method: POST 2023-09-22T21:23:24.307 DEBUG:boto:Path: /ffzdqu-50/MULTIPART?uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF 2023-09-22T21:23:24.307 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2023-09-22T21:23:24.308 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2023-09-22T21:23:24.308 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:23:24.308 DEBUG:boto:Port: 8000 2023-09-22T21:23:24.308 DEBUG:boto:Params: {} 2023-09-22T21:23:24.308 DEBUG:boto:Token: None 2023-09-22T21:23:24.308 DEBUG:boto:StringToSign: POST text/xml Fri, 22 Sep 2023 21:23:24 GMT /ffzdqu-50/MULTIPART?uploadId=2~9R7Vew-zKi6bL5hSDuVxEXYImzMKDeF 2023-09-22T21:23:24.308 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Cq0bJvVlPNubA4b52v5JRGq+rfg= 2023-09-22T21:23:24.309 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:23:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Cq0bJvVlPNubA4b52v5JRGq+rfg=', 'Content-Length': '460'} 2023-09-22T21:23:24.332 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e209bc11d2110b8c-00650e05cc-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '291'), ('Date', 'Fri, 22 Sep 2023 21:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:23:24.332 DEBUG:boto:http://smithi160.front.sepia.ceph.com:8000/ffzdqu-50/MULTIPARTffzdqu-50MULTIPART 2023-09-22T21:23:24.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:23:24.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:23:24.333 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:23:24.374 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:23:24.374 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:23:24.414 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:23:24.432 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695417802.718299_390.1","last_update":"2023-09-22T21:23:22.718299Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:23:24.433 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417802.718299_390.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:23:24.433 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:23:24.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:23:24.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:23:24.434 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:23:24.510 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:23:24.510 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:23:24.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:23:24.552+0000 7f708bc35a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:23:24.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:23:24.552+0000 7f708bc35a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:23:24.570 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417802.718299_390.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:23:22.718299Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:23:24Z"} 2023-09-22T21:23:24.570 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417802.718299_390.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:23:22.718299Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:23:24Z"} 2023-09-22T21:23:24.570 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417802.718299_390.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:23:22.718299Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:23:24.570 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417802.718299_390.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:23:24.570 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417802.718299_390.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:23:24.571 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:23:24.571 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-50 2023-09-22T21:23:24.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:23:24.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:23:24.571 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-50 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:23:24.664 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:23:24.665 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:23:24.706 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:23:24.708+0000 7fd556445a40 1 waiting to reach incremental sync.. 2023-09-22T21:23:29.707 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:23:29.708+0000 7fd556445a40 1 waiting to reach incremental sync.. 2023-09-22T21:23:34.709 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:23:34.708+0000 7fd556445a40 1 waiting to reach incremental sync.. 2023-09-22T21:23:39.710 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:23:39.708+0000 7fd556445a40 1 waiting to reach incremental sync.. 2023-09-22T21:23:44.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:23:44.712+0000 7fd556445a40 1 bucket sync caught up with source: 2023-09-22T21:23:44.713 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000011.53.6, , , , , ] 2023-09-22T21:23:44.714 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000011.53.6, , , , , ] 2023-09-22T21:23:44.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:23:44.712+0000 7fd556445a40 0 bucket checkpoint complete 2023-09-22T21:23:44.745 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-50 2023-09-22T21:23:44.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:23:44.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:23:44.746 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-50 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:23:44.836 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:23:44.836 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:23:44.880 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:23:44.880+0000 7f58b32a9a40 1 waiting to reach incremental sync.. 2023-09-22T21:23:49.881 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:23:49.880+0000 7f58b32a9a40 1 waiting to reach incremental sync.. 2023-09-22T21:23:54.882 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:23:54.880+0000 7f58b32a9a40 1 waiting to reach incremental sync.. 2023-09-22T21:23:59.883 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:23:59.884+0000 7f58b32a9a40 1 waiting to reach incremental sync.. 2023-09-22T21:24:04.885 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:04.884+0000 7f58b32a9a40 1 bucket sync caught up with source: 2023-09-22T21:24:04.886 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.163.6, , , , , ] 2023-09-22T21:24:04.886 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.163.6, , , , , ] 2023-09-22T21:24:04.886 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:04.884+0000 7f58b32a9a40 0 bucket checkpoint complete 2023-09-22T21:24:04.921 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-50 zones={a2, a1} 2023-09-22T21:24:04.922 DEBUG:boto:path=/ffzdqu-50/ 2023-09-22T21:24:04.922 DEBUG:boto:auth_path=/ffzdqu-50/ 2023-09-22T21:24:04.922 DEBUG:boto:Method: HEAD 2023-09-22T21:24:04.922 DEBUG:boto:Path: /ffzdqu-50/ 2023-09-22T21:24:04.922 DEBUG:boto:Data: 2023-09-22T21:24:04.922 DEBUG:boto:Headers: {} 2023-09-22T21:24:04.922 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:04.923 DEBUG:boto:Port: 8001 2023-09-22T21:24:04.923 DEBUG:boto:Params: {} 2023-09-22T21:24:04.923 DEBUG:boto:Token: None 2023-09-22T21:24:04.923 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:04 GMT /ffzdqu-50/ 2023-09-22T21:24:04.923 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ojGUEmXuMhm7DZx03V5HAP5r5I4= 2023-09-22T21:24:04.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ojGUEmXuMhm7DZx03V5HAP5r5I4=', 'Content-Length': '0'} 2023-09-22T21:24:04.928 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000837b9f5f444c00de-00650e05f4-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:04.928 DEBUG:boto:path=/ffzdqu-50/ 2023-09-22T21:24:04.928 DEBUG:boto:auth_path=/ffzdqu-50/ 2023-09-22T21:24:04.928 DEBUG:boto:Method: HEAD 2023-09-22T21:24:04.928 DEBUG:boto:Path: /ffzdqu-50/ 2023-09-22T21:24:04.928 DEBUG:boto:Data: 2023-09-22T21:24:04.928 DEBUG:boto:Headers: {} 2023-09-22T21:24:04.929 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:04.929 DEBUG:boto:Port: 8000 2023-09-22T21:24:04.929 DEBUG:boto:Params: {} 2023-09-22T21:24:04.929 DEBUG:boto:Token: None 2023-09-22T21:24:04.929 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:04 GMT /ffzdqu-50/ 2023-09-22T21:24:04.929 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ojGUEmXuMhm7DZx03V5HAP5r5I4= 2023-09-22T21:24:04.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ojGUEmXuMhm7DZx03V5HAP5r5I4=', 'Content-Length': '0'} 2023-09-22T21:24:04.933 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008281bf611e724672-00650e05f4-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:04.933 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:24:04.933 DEBUG:boto:path=/ffzdqu-50/ 2023-09-22T21:24:04.933 DEBUG:boto:auth_path=/ffzdqu-50/ 2023-09-22T21:24:04.933 DEBUG:boto:path=/ffzdqu-50/?versions&max-keys=999 2023-09-22T21:24:04.933 DEBUG:boto:auth_path=/ffzdqu-50/?versions&max-keys=999 2023-09-22T21:24:04.934 DEBUG:boto:Method: GET 2023-09-22T21:24:04.934 DEBUG:boto:Path: /ffzdqu-50/?versions&max-keys=999 2023-09-22T21:24:04.934 DEBUG:boto:Data: 2023-09-22T21:24:04.934 DEBUG:boto:Headers: {} 2023-09-22T21:24:04.934 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:04.934 DEBUG:boto:Port: 8001 2023-09-22T21:24:04.934 DEBUG:boto:Params: {} 2023-09-22T21:24:04.934 DEBUG:boto:Token: None 2023-09-22T21:24:04.934 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:04 GMT /ffzdqu-50/?versions 2023-09-22T21:24:04.935 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T81kJiEMtcPDoE7h6eV6Pc8XfIg= 2023-09-22T21:24:04.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T81kJiEMtcPDoE7h6eV6Pc8XfIg=', 'Content-Length': '0'} 2023-09-22T21:24:04.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079b26d95fba9346e-00650e05f4-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:04.982 DEBUG:boto:b'ffzdqu-50999falseMULTIPARTnulltrue2023-09-22T21:23:24.320Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:04.983 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2023-09-22T21:24:04.983 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:24:04.983 DEBUG:boto:path=/ffzdqu-50/ 2023-09-22T21:24:04.983 DEBUG:boto:auth_path=/ffzdqu-50/ 2023-09-22T21:24:04.983 DEBUG:boto:path=/ffzdqu-50/?versions&max-keys=999 2023-09-22T21:24:04.983 DEBUG:boto:auth_path=/ffzdqu-50/?versions&max-keys=999 2023-09-22T21:24:04.984 DEBUG:boto:Method: GET 2023-09-22T21:24:04.984 DEBUG:boto:Path: /ffzdqu-50/?versions&max-keys=999 2023-09-22T21:24:04.984 DEBUG:boto:Data: 2023-09-22T21:24:04.984 DEBUG:boto:Headers: {} 2023-09-22T21:24:04.984 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:04.984 DEBUG:boto:Port: 8000 2023-09-22T21:24:04.984 DEBUG:boto:Params: {} 2023-09-22T21:24:04.984 DEBUG:boto:Token: None 2023-09-22T21:24:04.984 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:04 GMT /ffzdqu-50/?versions 2023-09-22T21:24:04.985 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T81kJiEMtcPDoE7h6eV6Pc8XfIg= 2023-09-22T21:24:04.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T81kJiEMtcPDoE7h6eV6Pc8XfIg=', 'Content-Length': '0'} 2023-09-22T21:24:04.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053d15e91ddeb68c6-00650e05f4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:04.989 DEBUG:boto:b'ffzdqu-50999falseMULTIPARTnulltrue2023-09-22T21:23:24.320Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:04.990 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2023-09-22T21:24:04.990 DEBUG:boto:path=/ffzdqu-50/ 2023-09-22T21:24:04.990 DEBUG:boto:auth_path=/ffzdqu-50/ 2023-09-22T21:24:04.990 DEBUG:boto:path=/ffzdqu-50/?versions&max-keys=999 2023-09-22T21:24:04.990 DEBUG:boto:auth_path=/ffzdqu-50/?versions&max-keys=999 2023-09-22T21:24:04.990 DEBUG:boto:Method: GET 2023-09-22T21:24:04.991 DEBUG:boto:Path: /ffzdqu-50/?versions&max-keys=999 2023-09-22T21:24:04.991 DEBUG:boto:Data: 2023-09-22T21:24:04.991 DEBUG:boto:Headers: {} 2023-09-22T21:24:04.991 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:04.991 DEBUG:boto:Port: 8001 2023-09-22T21:24:04.991 DEBUG:boto:Params: {} 2023-09-22T21:24:04.991 DEBUG:boto:Token: None 2023-09-22T21:24:04.991 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:04 GMT /ffzdqu-50/?versions 2023-09-22T21:24:04.991 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T81kJiEMtcPDoE7h6eV6Pc8XfIg= 2023-09-22T21:24:04.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T81kJiEMtcPDoE7h6eV6Pc8XfIg=', 'Content-Length': '0'} 2023-09-22T21:24:04.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de53004fbd2a7ba9-00650e05f4-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:04.996 DEBUG:boto:b'ffzdqu-50999falseMULTIPARTnulltrue2023-09-22T21:23:24.320Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:04.996 DEBUG:boto:path=/ffzdqu-50/ 2023-09-22T21:24:04.996 DEBUG:boto:auth_path=/ffzdqu-50/ 2023-09-22T21:24:04.996 DEBUG:boto:path=/ffzdqu-50/?versions&max-keys=999 2023-09-22T21:24:04.997 DEBUG:boto:auth_path=/ffzdqu-50/?versions&max-keys=999 2023-09-22T21:24:04.997 DEBUG:boto:Method: GET 2023-09-22T21:24:04.997 DEBUG:boto:Path: /ffzdqu-50/?versions&max-keys=999 2023-09-22T21:24:04.997 DEBUG:boto:Data: 2023-09-22T21:24:04.997 DEBUG:boto:Headers: {} 2023-09-22T21:24:04.997 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:04.997 DEBUG:boto:Port: 8000 2023-09-22T21:24:04.997 DEBUG:boto:Params: {} 2023-09-22T21:24:04.997 DEBUG:boto:Token: None 2023-09-22T21:24:04.998 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:04 GMT /ffzdqu-50/?versions 2023-09-22T21:24:04.998 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T81kJiEMtcPDoE7h6eV6Pc8XfIg= 2023-09-22T21:24:04.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T81kJiEMtcPDoE7h6eV6Pc8XfIg=', 'Content-Length': '0'} 2023-09-22T21:24:05.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b640b896cd2d68c8-00650e05f5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:05.003 DEBUG:boto:b'ffzdqu-50999falseMULTIPARTnulltrue2023-09-22T21:23:24.320Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:05.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-09-22T21:24:05.003 DEBUG:boto:path=/ffzdqu-50/MULTIPART 2023-09-22T21:24:05.003 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART 2023-09-22T21:24:05.004 DEBUG:boto:path=/ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.004 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.004 DEBUG:boto:Method: GET 2023-09-22T21:24:05.004 DEBUG:boto:Path: /ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.004 DEBUG:boto:Data: 2023-09-22T21:24:05.004 DEBUG:boto:Headers: {} 2023-09-22T21:24:05.004 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:05.004 DEBUG:boto:Port: 8001 2023-09-22T21:24:05.004 DEBUG:boto:Params: {} 2023-09-22T21:24:05.004 DEBUG:boto:Token: None 2023-09-22T21:24:05.005 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:05 GMT /ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.005 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZIHJ65C+GvjCqgprEzyk+sRzAPY= 2023-09-22T21:24:05.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZIHJ65C+GvjCqgprEzyk+sRzAPY=', 'Content-Length': '0'} 2023-09-22T21:24:05.011 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:23:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-50:f132bdb9-20a7-4073-a932-8f99da20691e.5729.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000002f79cfbb8e83b2b1-00650e05f5-165b-a2'), ('Content-Type', 'binary/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:05.086 DEBUG:boto:path=/ffzdqu-50/MULTIPART 2023-09-22T21:24:05.086 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART 2023-09-22T21:24:05.086 DEBUG:boto:path=/ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.086 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.086 DEBUG:boto:Method: GET 2023-09-22T21:24:05.087 DEBUG:boto:Path: /ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.087 DEBUG:boto:Data: 2023-09-22T21:24:05.087 DEBUG:boto:Headers: {} 2023-09-22T21:24:05.087 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:05.087 DEBUG:boto:Port: 8000 2023-09-22T21:24:05.087 DEBUG:boto:Params: {} 2023-09-22T21:24:05.087 DEBUG:boto:Token: None 2023-09-22T21:24:05.087 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:05 GMT /ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.087 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZIHJ65C+GvjCqgprEzyk+sRzAPY= 2023-09-22T21:24:05.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZIHJ65C+GvjCqgprEzyk+sRzAPY=', 'Content-Length': '0'} 2023-09-22T21:24:05.094 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:23:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000e82a3ee73a6a240c-00650e05f5-165f-a1'), ('Content-Type', 'binary/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:05.182 DEBUG:boto:path=/ffzdqu-50/MULTIPART 2023-09-22T21:24:05.182 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART 2023-09-22T21:24:05.182 DEBUG:boto:path=/ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.183 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.183 DEBUG:boto:Method: HEAD 2023-09-22T21:24:05.183 DEBUG:boto:Path: /ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.183 DEBUG:boto:Data: 2023-09-22T21:24:05.183 DEBUG:boto:Headers: {} 2023-09-22T21:24:05.183 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:05.183 DEBUG:boto:Port: 8001 2023-09-22T21:24:05.183 DEBUG:boto:Params: {} 2023-09-22T21:24:05.183 DEBUG:boto:Token: None 2023-09-22T21:24:05.184 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:05 GMT /ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.184 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J2R+2e84PpFKiMJ4aNi14wh+5U4= 2023-09-22T21:24:05.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J2R+2e84PpFKiMJ4aNi14wh+5U4=', 'Content-Length': '0'} 2023-09-22T21:24:05.187 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:23:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-50:f132bdb9-20a7-4073-a932-8f99da20691e.5729.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000049846ae702fc6dc7-00650e05f5-165b-a2'), ('Content-Type', 'binary/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:05.187 DEBUG:boto:path=/ffzdqu-50/MULTIPART 2023-09-22T21:24:05.187 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART 2023-09-22T21:24:05.188 DEBUG:boto:path=/ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.188 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.188 DEBUG:boto:Method: HEAD 2023-09-22T21:24:05.188 DEBUG:boto:Path: /ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.188 DEBUG:boto:Data: 2023-09-22T21:24:05.188 DEBUG:boto:Headers: {} 2023-09-22T21:24:05.188 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:05.188 DEBUG:boto:Port: 8000 2023-09-22T21:24:05.188 DEBUG:boto:Params: {} 2023-09-22T21:24:05.189 DEBUG:boto:Token: None 2023-09-22T21:24:05.189 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:05 GMT /ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.189 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J2R+2e84PpFKiMJ4aNi14wh+5U4= 2023-09-22T21:24:05.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J2R+2e84PpFKiMJ4aNi14wh+5U4=', 'Content-Length': '0'} 2023-09-22T21:24:05.192 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:23:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000027a9796fb021d4b5-00650e05f5-165f-a1'), ('Content-Type', 'binary/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:05.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-09-22T21:24:05.192 DEBUG:boto:path=/ffzdqu-50/MULTIPART 2023-09-22T21:24:05.192 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART 2023-09-22T21:24:05.192 DEBUG:boto:path=/ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.193 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.193 DEBUG:boto:Method: GET 2023-09-22T21:24:05.193 DEBUG:boto:Path: /ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.193 DEBUG:boto:Data: 2023-09-22T21:24:05.193 DEBUG:boto:Headers: {} 2023-09-22T21:24:05.193 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:05.193 DEBUG:boto:Port: 8001 2023-09-22T21:24:05.193 DEBUG:boto:Params: {} 2023-09-22T21:24:05.193 DEBUG:boto:Token: None 2023-09-22T21:24:05.193 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:05 GMT /ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.194 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZIHJ65C+GvjCqgprEzyk+sRzAPY= 2023-09-22T21:24:05.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZIHJ65C+GvjCqgprEzyk+sRzAPY=', 'Content-Length': '0'} 2023-09-22T21:24:05.200 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:23:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-50:f132bdb9-20a7-4073-a932-8f99da20691e.5729.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000007ed43507407bf501-00650e05f5-165b-a2'), ('Content-Type', 'binary/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:05.281 DEBUG:boto:path=/ffzdqu-50/MULTIPART 2023-09-22T21:24:05.281 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART 2023-09-22T21:24:05.282 DEBUG:boto:path=/ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.282 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.282 DEBUG:boto:Method: GET 2023-09-22T21:24:05.282 DEBUG:boto:Path: /ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.282 DEBUG:boto:Data: 2023-09-22T21:24:05.282 DEBUG:boto:Headers: {} 2023-09-22T21:24:05.282 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:05.282 DEBUG:boto:Port: 8000 2023-09-22T21:24:05.282 DEBUG:boto:Params: {} 2023-09-22T21:24:05.283 DEBUG:boto:Token: None 2023-09-22T21:24:05.283 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:05 GMT /ffzdqu-50/MULTIPART?versionId=null 2023-09-22T21:24:05.283 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZIHJ65C+GvjCqgprEzyk+sRzAPY= 2023-09-22T21:24:05.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZIHJ65C+GvjCqgprEzyk+sRzAPY=', 'Content-Length': '0'} 2023-09-22T21:24:05.290 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:23:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000003fa573bb100d2b48-00650e05f5-165f-a1'), ('Content-Type', 'binary/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:05.378 DEBUG:boto:path=/ffzdqu-50/ 2023-09-22T21:24:05.378 DEBUG:boto:auth_path=/ffzdqu-50/ 2023-09-22T21:24:05.378 DEBUG:boto:path=/ffzdqu-50/?prefix=MULTIPART 2023-09-22T21:24:05.378 DEBUG:boto:auth_path=/ffzdqu-50/?prefix=MULTIPART 2023-09-22T21:24:05.378 DEBUG:boto:Method: GET 2023-09-22T21:24:05.378 DEBUG:boto:Path: /ffzdqu-50/?prefix=MULTIPART 2023-09-22T21:24:05.379 DEBUG:boto:Data: 2023-09-22T21:24:05.379 DEBUG:boto:Headers: {} 2023-09-22T21:24:05.379 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:05.379 DEBUG:boto:Port: 8001 2023-09-22T21:24:05.379 DEBUG:boto:Params: {} 2023-09-22T21:24:05.379 DEBUG:boto:Token: None 2023-09-22T21:24:05.379 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:05 GMT /ffzdqu-50/ 2023-09-22T21:24:05.379 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:061noAFwVx2lMRnB0svS8UmxGmc= 2023-09-22T21:24:05.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:061noAFwVx2lMRnB0svS8UmxGmc=', 'Content-Length': '0'} 2023-09-22T21:24:05.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6149106a82c4f93-00650e05f5-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:05.385 DEBUG:boto:b'ffzdqu-50MULTIPART1000falseMULTIPART2023-09-22T21:23:24.320Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:05.385 DEBUG:boto:path=/ffzdqu-50/ 2023-09-22T21:24:05.385 DEBUG:boto:auth_path=/ffzdqu-50/ 2023-09-22T21:24:05.386 DEBUG:boto:path=/ffzdqu-50/?prefix=MULTIPART 2023-09-22T21:24:05.386 DEBUG:boto:auth_path=/ffzdqu-50/?prefix=MULTIPART 2023-09-22T21:24:05.386 DEBUG:boto:Method: GET 2023-09-22T21:24:05.386 DEBUG:boto:Path: /ffzdqu-50/?prefix=MULTIPART 2023-09-22T21:24:05.386 DEBUG:boto:Data: 2023-09-22T21:24:05.386 DEBUG:boto:Headers: {} 2023-09-22T21:24:05.386 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:05.386 DEBUG:boto:Port: 8000 2023-09-22T21:24:05.386 DEBUG:boto:Params: {} 2023-09-22T21:24:05.387 DEBUG:boto:Token: None 2023-09-22T21:24:05.387 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:05 GMT /ffzdqu-50/ 2023-09-22T21:24:05.387 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:061noAFwVx2lMRnB0svS8UmxGmc= 2023-09-22T21:24:05.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:061noAFwVx2lMRnB0svS8UmxGmc=', 'Content-Length': '0'} 2023-09-22T21:24:05.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ee74b70aa77b78c-00650e05f5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:05.392 DEBUG:boto:b'ffzdqu-50MULTIPART1000falseMULTIPART2023-09-22T21:23:24.320Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:05.392 DEBUG:boto:path=/ffzdqu-50/MULTIPART 2023-09-22T21:24:05.392 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART 2023-09-22T21:24:05.392 DEBUG:boto:Method: HEAD 2023-09-22T21:24:05.393 DEBUG:boto:Path: /ffzdqu-50/MULTIPART 2023-09-22T21:24:05.393 DEBUG:boto:Data: 2023-09-22T21:24:05.393 DEBUG:boto:Headers: {} 2023-09-22T21:24:05.393 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:05.393 DEBUG:boto:Port: 8001 2023-09-22T21:24:05.393 DEBUG:boto:Params: {} 2023-09-22T21:24:05.393 DEBUG:boto:Token: None 2023-09-22T21:24:05.393 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:05 GMT /ffzdqu-50/MULTIPART 2023-09-22T21:24:05.393 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ya0g8+CEXZAd/b7d4vSpjuyVVEs= 2023-09-22T21:24:05.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ya0g8+CEXZAd/b7d4vSpjuyVVEs=', 'Content-Length': '0'} 2023-09-22T21:24:05.396 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:23:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-50:f132bdb9-20a7-4073-a932-8f99da20691e.5729.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000e2663c37c26188e7-00650e05f5-165b-a2'), ('Content-Type', 'binary/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:05.396 DEBUG:boto:path=/ffzdqu-50/MULTIPART 2023-09-22T21:24:05.397 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART 2023-09-22T21:24:05.397 DEBUG:boto:Method: HEAD 2023-09-22T21:24:05.397 DEBUG:boto:Path: /ffzdqu-50/MULTIPART 2023-09-22T21:24:05.397 DEBUG:boto:Data: 2023-09-22T21:24:05.397 DEBUG:boto:Headers: {} 2023-09-22T21:24:05.397 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:05.397 DEBUG:boto:Port: 8000 2023-09-22T21:24:05.397 DEBUG:boto:Params: {} 2023-09-22T21:24:05.397 DEBUG:boto:Token: None 2023-09-22T21:24:05.398 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:05 GMT /ffzdqu-50/MULTIPART 2023-09-22T21:24:05.398 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ya0g8+CEXZAd/b7d4vSpjuyVVEs= 2023-09-22T21:24:05.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ya0g8+CEXZAd/b7d4vSpjuyVVEs=', 'Content-Length': '0'} 2023-09-22T21:24:05.401 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:23:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000000ba96e799282693b-00650e05f5-165f-a1'), ('Content-Type', 'binary/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:05.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-09-22T21:24:05.401 DEBUG:boto:path=/ffzdqu-50/MULTIPART 2023-09-22T21:24:05.401 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART 2023-09-22T21:24:05.401 DEBUG:boto:Method: GET 2023-09-22T21:24:05.401 DEBUG:boto:Path: /ffzdqu-50/MULTIPART 2023-09-22T21:24:05.401 DEBUG:boto:Data: 2023-09-22T21:24:05.402 DEBUG:boto:Headers: {} 2023-09-22T21:24:05.402 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:05.402 DEBUG:boto:Port: 8001 2023-09-22T21:24:05.402 DEBUG:boto:Params: {} 2023-09-22T21:24:05.402 DEBUG:boto:Token: None 2023-09-22T21:24:05.402 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:05 GMT /ffzdqu-50/MULTIPART 2023-09-22T21:24:05.402 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4A07n6SL3HUr51xXEneqA1YZ9Ms= 2023-09-22T21:24:05.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4A07n6SL3HUr51xXEneqA1YZ9Ms=', 'Content-Length': '0'} 2023-09-22T21:24:05.408 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:23:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-50:f132bdb9-20a7-4073-a932-8f99da20691e.5729.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000711b9d62d386e3e1-00650e05f5-165b-a2'), ('Content-Type', 'binary/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:05.489 DEBUG:boto:path=/ffzdqu-50/MULTIPART 2023-09-22T21:24:05.490 DEBUG:boto:auth_path=/ffzdqu-50/MULTIPART 2023-09-22T21:24:05.490 DEBUG:boto:Method: GET 2023-09-22T21:24:05.490 DEBUG:boto:Path: /ffzdqu-50/MULTIPART 2023-09-22T21:24:05.490 DEBUG:boto:Data: 2023-09-22T21:24:05.490 DEBUG:boto:Headers: {} 2023-09-22T21:24:05.490 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:05.490 DEBUG:boto:Port: 8000 2023-09-22T21:24:05.490 DEBUG:boto:Params: {} 2023-09-22T21:24:05.490 DEBUG:boto:Token: None 2023-09-22T21:24:05.491 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:05 GMT /ffzdqu-50/MULTIPART 2023-09-22T21:24:05.491 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4A07n6SL3HUr51xXEneqA1YZ9Ms= 2023-09-22T21:24:05.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4A07n6SL3HUr51xXEneqA1YZ9Ms=', 'Content-Length': '0'} 2023-09-22T21:24:05.498 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:23:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000001e88088a6fb1b880-00650e05f5-165f-a1'), ('Content-Type', 'binary/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:05.587 DEBUG:boto:path=/ffzdqu-50/ 2023-09-22T21:24:05.587 DEBUG:boto:auth_path=/ffzdqu-50/ 2023-09-22T21:24:05.587 DEBUG:boto:path=/ffzdqu-50/?prefix=MULTIPART 2023-09-22T21:24:05.587 DEBUG:boto:auth_path=/ffzdqu-50/?prefix=MULTIPART 2023-09-22T21:24:05.587 DEBUG:boto:Method: GET 2023-09-22T21:24:05.588 DEBUG:boto:Path: /ffzdqu-50/?prefix=MULTIPART 2023-09-22T21:24:05.588 DEBUG:boto:Data: 2023-09-22T21:24:05.588 DEBUG:boto:Headers: {} 2023-09-22T21:24:05.588 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:05.588 DEBUG:boto:Port: 8001 2023-09-22T21:24:05.588 DEBUG:boto:Params: {} 2023-09-22T21:24:05.588 DEBUG:boto:Token: None 2023-09-22T21:24:05.588 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:05 GMT /ffzdqu-50/ 2023-09-22T21:24:05.589 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:061noAFwVx2lMRnB0svS8UmxGmc= 2023-09-22T21:24:05.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:061noAFwVx2lMRnB0svS8UmxGmc=', 'Content-Length': '0'} 2023-09-22T21:24:05.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009addfed25d93cabe-00650e05f5-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:05.594 DEBUG:boto:b'ffzdqu-50MULTIPART1000falseMULTIPART2023-09-22T21:23:24.320Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:05.594 DEBUG:boto:path=/ffzdqu-50/ 2023-09-22T21:24:05.595 DEBUG:boto:auth_path=/ffzdqu-50/ 2023-09-22T21:24:05.595 DEBUG:boto:path=/ffzdqu-50/?prefix=MULTIPART 2023-09-22T21:24:05.595 DEBUG:boto:auth_path=/ffzdqu-50/?prefix=MULTIPART 2023-09-22T21:24:05.595 DEBUG:boto:Method: GET 2023-09-22T21:24:05.595 DEBUG:boto:Path: /ffzdqu-50/?prefix=MULTIPART 2023-09-22T21:24:05.595 DEBUG:boto:Data: 2023-09-22T21:24:05.595 DEBUG:boto:Headers: {} 2023-09-22T21:24:05.595 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:05.595 DEBUG:boto:Port: 8000 2023-09-22T21:24:05.595 DEBUG:boto:Params: {} 2023-09-22T21:24:05.596 DEBUG:boto:Token: None 2023-09-22T21:24:05.596 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:05 GMT /ffzdqu-50/ 2023-09-22T21:24:05.596 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:061noAFwVx2lMRnB0svS8UmxGmc= 2023-09-22T21:24:05.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:061noAFwVx2lMRnB0svS8UmxGmc=', 'Content-Length': '0'} 2023-09-22T21:24:05.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb109c5f92d4df2c-00650e05f5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:05.646 DEBUG:boto:b'ffzdqu-50MULTIPART1000falseMULTIPART2023-09-22T21:23:24.320Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:05.647 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-50 zones={a2, a1} 2023-09-22T21:24:05.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ok 2023-09-22T21:24:05.648 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-52 2023-09-22T21:24:05.648 DEBUG:boto:path=/ffzdqu-52/ 2023-09-22T21:24:05.648 DEBUG:boto:auth_path=/ffzdqu-52/ 2023-09-22T21:24:05.648 DEBUG:boto:Method: PUT 2023-09-22T21:24:05.648 DEBUG:boto:Path: /ffzdqu-52/ 2023-09-22T21:24:05.649 DEBUG:boto:Data: 2023-09-22T21:24:05.649 DEBUG:boto:Headers: {} 2023-09-22T21:24:05.649 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:05.649 DEBUG:boto:Port: 8000 2023-09-22T21:24:05.649 DEBUG:boto:Params: {} 2023-09-22T21:24:05.649 DEBUG:boto:Token: None 2023-09-22T21:24:05.649 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:24:05 GMT /ffzdqu-52/ 2023-09-22T21:24:05.649 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Bh6hGF09afwyLc5s7kGoH9BFlkA= 2023-09-22T21:24:05.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Bh6hGF09afwyLc5s7kGoH9BFlkA=', 'Content-Length': '0'} 2023-09-22T21:24:05.694 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000687aea431969b3c7-00650e05f5-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:05.695 DEBUG:boto:path=/ffzdqu-52/testobj-sse-c 2023-09-22T21:24:05.695 DEBUG:boto:auth_path=/ffzdqu-52/testobj-sse-c 2023-09-22T21:24:05.695 DEBUG:boto:Method: PUT 2023-09-22T21:24:05.695 DEBUG:boto:Path: /ffzdqu-52/testobj-sse-c 2023-09-22T21:24:05.695 DEBUG:boto:Data: 2023-09-22T21:24:05.695 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-09-22T21:24:05.695 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:05.695 DEBUG:boto:Port: 8000 2023-09-22T21:24:05.695 DEBUG:boto:Params: {} 2023-09-22T21:24:05.696 DEBUG:boto:Token: None 2023-09-22T21:24:05.696 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Fri, 22 Sep 2023 21:24:05 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /ffzdqu-52/testobj-sse-c 2023-09-22T21:24:05.696 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Qla3pzEojBKgjOLojvr/u3gM6Uw= 2023-09-22T21:24:05.696 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:24:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Qla3pzEojBKgjOLojvr/u3gM6Uw='} 2023-09-22T21:24:05.711 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx000008ebc3ccc95889690-00650e05f5-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:05.712 DEBUG:boto:path=/ffzdqu-52/testobj-sse-kms 2023-09-22T21:24:05.712 DEBUG:boto:auth_path=/ffzdqu-52/testobj-sse-kms 2023-09-22T21:24:05.712 DEBUG:boto:Method: PUT 2023-09-22T21:24:05.712 DEBUG:boto:Path: /ffzdqu-52/testobj-sse-kms 2023-09-22T21:24:05.712 DEBUG:boto:Data: 2023-09-22T21:24:05.712 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-09-22T21:24:05.712 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:05.713 DEBUG:boto:Port: 8000 2023-09-22T21:24:05.713 DEBUG:boto:Params: {} 2023-09-22T21:24:05.713 DEBUG:boto:Token: None 2023-09-22T21:24:05.713 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Fri, 22 Sep 2023 21:24:05 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /ffzdqu-52/testobj-sse-kms 2023-09-22T21:24:05.713 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2ew3CMwTK2cie5kcLlNV8mIOpPU= 2023-09-22T21:24:05.713 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:24:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2ew3CMwTK2cie5kcLlNV8mIOpPU='} 2023-09-22T21:24:05.758 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6ZmZ6ZHF1LTUyL3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx0000022241f91da993adf-00650e05f5-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:05.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:05.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:05.759 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:24:05.803 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:24:05.803 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:24:05.844 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:24:05.861 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695417845.691663_394.1","last_update":"2023-09-22T21:24:05.691663Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:24:05.862 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417845.691663_394.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:24:05.862 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:24:05.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:05.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:05.862 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:24:05.942 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:24:05.942 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:24:05.979 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:05.980+0000 7f259cf91a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:24:05.980 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:05.980+0000 7f259cf91a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:24:05.999 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417845.691663_394.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:24:05.691663Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:24:05Z"} 2023-09-22T21:24:05.999 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417845.691663_394.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:24:05.691663Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:24:05Z"} 2023-09-22T21:24:05.999 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417845.691663_394.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:24:05.691663Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:24:05.999 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417845.691663_394.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:24:05.999 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417845.691663_394.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:24:05.999 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:24:06.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:24:06.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:24:06.000 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-52 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:24:06.084 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:24:06.085 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:24:06.126 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:06.124+0000 7fe62073da40 1 waiting to reach incremental sync.. 2023-09-22T21:24:11.127 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:11.127+0000 7fe62073da40 1 waiting to reach incremental sync.. 2023-09-22T21:24:16.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:16.127+0000 7fe62073da40 1 waiting to reach incremental sync.. 2023-09-22T21:24:21.131 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:21.131+0000 7fe62073da40 1 bucket sync caught up with source: 2023-09-22T21:24:21.131 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.14.6, , , , 00000000001.36.6, , , , , ] 2023-09-22T21:24:21.132 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.14.6, , , , 00000000001.36.6, , , , , ] 2023-09-22T21:24:21.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:21.131+0000 7fe62073da40 0 bucket checkpoint complete 2023-09-22T21:24:21.169 DEBUG:boto:path=/ffzdqu-52/ 2023-09-22T21:24:21.169 DEBUG:boto:auth_path=/ffzdqu-52/ 2023-09-22T21:24:21.169 DEBUG:boto:Method: HEAD 2023-09-22T21:24:21.169 DEBUG:boto:Path: /ffzdqu-52/ 2023-09-22T21:24:21.169 DEBUG:boto:Data: 2023-09-22T21:24:21.169 DEBUG:boto:Headers: {} 2023-09-22T21:24:21.170 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:21.170 DEBUG:boto:Port: 8001 2023-09-22T21:24:21.170 DEBUG:boto:Params: {} 2023-09-22T21:24:21.170 DEBUG:boto:Token: None 2023-09-22T21:24:21.170 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:21 GMT /ffzdqu-52/ 2023-09-22T21:24:21.170 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cp9Fn316R2hmRqBQyURcj+KqVIM= 2023-09-22T21:24:21.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cp9Fn316R2hmRqBQyURcj+KqVIM=', 'Content-Length': '0'} 2023-09-22T21:24:21.174 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '1024'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000456899ef170319e0-00650e0605-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:21.174 DEBUG:boto:path=/ffzdqu-52/testobj-sse-c 2023-09-22T21:24:21.175 DEBUG:boto:auth_path=/ffzdqu-52/testobj-sse-c 2023-09-22T21:24:21.175 DEBUG:boto:Method: HEAD 2023-09-22T21:24:21.175 DEBUG:boto:Path: /ffzdqu-52/testobj-sse-c 2023-09-22T21:24:21.175 DEBUG:boto:Data: 2023-09-22T21:24:21.175 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2023-09-22T21:24:21.175 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:21.175 DEBUG:boto:Port: 8001 2023-09-22T21:24:21.175 DEBUG:boto:Params: {} 2023-09-22T21:24:21.176 DEBUG:boto:Token: None 2023-09-22T21:24:21.176 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:21 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /ffzdqu-52/testobj-sse-c 2023-09-22T21:24:21.176 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jRaZuk1Tr36hgxCDjkGociQFszs= 2023-09-22T21:24:21.176 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jRaZuk1Tr36hgxCDjkGociQFszs=', 'Content-Length': '0'} 2023-09-22T21:24:21.178 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-52:f132bdb9-20a7-4073-a932-8f99da20691e.5729.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000eba72eb9ab4031f9-00650e0605-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:21.179 DEBUG:boto:path=/ffzdqu-52/testobj-sse-c 2023-09-22T21:24:21.179 DEBUG:boto:auth_path=/ffzdqu-52/testobj-sse-c 2023-09-22T21:24:21.179 DEBUG:boto:Method: GET 2023-09-22T21:24:21.179 DEBUG:boto:Path: /ffzdqu-52/testobj-sse-c 2023-09-22T21:24:21.179 DEBUG:boto:Data: 2023-09-22T21:24:21.179 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2023-09-22T21:24:21.179 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:21.179 DEBUG:boto:Port: 8001 2023-09-22T21:24:21.179 DEBUG:boto:Params: {} 2023-09-22T21:24:21.180 DEBUG:boto:Token: None 2023-09-22T21:24:21.180 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:21 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /ffzdqu-52/testobj-sse-c 2023-09-22T21:24:21.180 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OxQmaxZxlbT4fgTRi+1YLQrU4yY= 2023-09-22T21:24:21.180 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OxQmaxZxlbT4fgTRi+1YLQrU4yY=', 'Content-Length': '0'} 2023-09-22T21:24:21.183 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-52:f132bdb9-20a7-4073-a932-8f99da20691e.5729.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000005fa3b02c68791b0f-00650e0605-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:21.230 DEBUG:boto:path=/ffzdqu-52/testobj-sse-kms 2023-09-22T21:24:21.231 DEBUG:boto:auth_path=/ffzdqu-52/testobj-sse-kms 2023-09-22T21:24:21.231 DEBUG:boto:Method: HEAD 2023-09-22T21:24:21.231 DEBUG:boto:Path: /ffzdqu-52/testobj-sse-kms 2023-09-22T21:24:21.231 DEBUG:boto:Data: 2023-09-22T21:24:21.231 DEBUG:boto:Headers: {} 2023-09-22T21:24:21.231 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:21.231 DEBUG:boto:Port: 8001 2023-09-22T21:24:21.231 DEBUG:boto:Params: {} 2023-09-22T21:24:21.231 DEBUG:boto:Token: None 2023-09-22T21:24:21.232 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:21 GMT /ffzdqu-52/testobj-sse-kms 2023-09-22T21:24:21.232 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GNSelannslmSkWDcbV/LEI6Jqos= 2023-09-22T21:24:21.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GNSelannslmSkWDcbV/LEI6Jqos=', 'Content-Length': '0'} 2023-09-22T21:24:21.235 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-52:f132bdb9-20a7-4073-a932-8f99da20691e.5729.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000009f155e3feb6a6d38-00650e0605-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:21.235 DEBUG:boto:path=/ffzdqu-52/testobj-sse-kms 2023-09-22T21:24:21.235 DEBUG:boto:auth_path=/ffzdqu-52/testobj-sse-kms 2023-09-22T21:24:21.235 DEBUG:boto:Method: GET 2023-09-22T21:24:21.235 DEBUG:boto:Path: /ffzdqu-52/testobj-sse-kms 2023-09-22T21:24:21.235 DEBUG:boto:Data: 2023-09-22T21:24:21.236 DEBUG:boto:Headers: {} 2023-09-22T21:24:21.236 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:21.236 DEBUG:boto:Port: 8001 2023-09-22T21:24:21.236 DEBUG:boto:Params: {} 2023-09-22T21:24:21.236 DEBUG:boto:Token: None 2023-09-22T21:24:21.236 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:21 GMT /ffzdqu-52/testobj-sse-kms 2023-09-22T21:24:21.236 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uMpWIgKJoMoI3TXlOrPXseOnuJs= 2023-09-22T21:24:21.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uMpWIgKJoMoI3TXlOrPXseOnuJs=', 'Content-Length': '0'} 2023-09-22T21:24:21.239 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-52:f132bdb9-20a7-4073-a932-8f99da20691e.5729.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000009d61321d3ac00200-00650e0605-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:21.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ok 2023-09-22T21:24:21.241 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-53 2023-09-22T21:24:21.241 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:21.241 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:21.241 DEBUG:boto:Method: PUT 2023-09-22T21:24:21.241 DEBUG:boto:Path: /ffzdqu-53/ 2023-09-22T21:24:21.241 DEBUG:boto:Data: 2023-09-22T21:24:21.241 DEBUG:boto:Headers: {} 2023-09-22T21:24:21.241 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:21.241 DEBUG:boto:Port: 8000 2023-09-22T21:24:21.241 DEBUG:boto:Params: {} 2023-09-22T21:24:21.242 DEBUG:boto:Token: None 2023-09-22T21:24:21.242 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:24:21 GMT /ffzdqu-53/ 2023-09-22T21:24:21.242 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rNtqrsdrNpE/7d7tb0WfrBH6UFw= 2023-09-22T21:24:21.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rNtqrsdrNpE/7d7tb0WfrBH6UFw=', 'Content-Length': '0'} 2023-09-22T21:24:21.285 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002ec47b078fc119ef-00650e0605-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:21.285 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:21.285 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:21.286 DEBUG:boto:Method: HEAD 2023-09-22T21:24:21.286 DEBUG:boto:Path: /ffzdqu-53/ 2023-09-22T21:24:21.286 DEBUG:boto:Data: 2023-09-22T21:24:21.286 DEBUG:boto:Headers: {} 2023-09-22T21:24:21.286 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:21.286 DEBUG:boto:Port: 8000 2023-09-22T21:24:21.286 DEBUG:boto:Params: {} 2023-09-22T21:24:21.286 DEBUG:boto:Token: None 2023-09-22T21:24:21.287 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:21 GMT /ffzdqu-53/ 2023-09-22T21:24:21.287 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2ZXzQ6itPUNLVTNnwBifBs9TJ/g= 2023-09-22T21:24:21.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2ZXzQ6itPUNLVTNnwBifBs9TJ/g=', 'Content-Length': '0'} 2023-09-22T21:24:21.290 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb1536d10f322c36-00650e0605-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:21.291 DEBUG:boto:path=/ffzdqu-53/a 2023-09-22T21:24:21.291 DEBUG:boto:auth_path=/ffzdqu-53/a 2023-09-22T21:24:21.291 DEBUG:boto:Method: PUT 2023-09-22T21:24:21.291 DEBUG:boto:Path: /ffzdqu-53/a 2023-09-22T21:24:21.291 DEBUG:boto:Data: 2023-09-22T21:24:21.291 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:24:21.291 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:21.291 DEBUG:boto:Port: 8000 2023-09-22T21:24:21.292 DEBUG:boto:Params: {} 2023-09-22T21:24:21.292 DEBUG:boto:Token: None 2023-09-22T21:24:21.292 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:24:21 GMT /ffzdqu-53/a 2023-09-22T21:24:21.292 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dhv7/3l0xkoYGCwNnhK24SAqwlc= 2023-09-22T21:24:21.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:24:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dhv7/3l0xkoYGCwNnhK24SAqwlc='} 2023-09-22T21:24:21.306 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f61594032cfe6211-00650e0605-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:21.307 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:21.307 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:21.307 DEBUG:boto:Method: HEAD 2023-09-22T21:24:21.307 DEBUG:boto:Path: /ffzdqu-53/ 2023-09-22T21:24:21.307 DEBUG:boto:Data: 2023-09-22T21:24:21.307 DEBUG:boto:Headers: {} 2023-09-22T21:24:21.307 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:21.307 DEBUG:boto:Port: 8000 2023-09-22T21:24:21.307 DEBUG:boto:Params: {} 2023-09-22T21:24:21.308 DEBUG:boto:Token: None 2023-09-22T21:24:21.308 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:21 GMT /ffzdqu-53/ 2023-09-22T21:24:21.308 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2ZXzQ6itPUNLVTNnwBifBs9TJ/g= 2023-09-22T21:24:21.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2ZXzQ6itPUNLVTNnwBifBs9TJ/g=', 'Content-Length': '0'} 2023-09-22T21:24:21.312 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002474e82adc998479-00650e0605-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:21.312 DEBUG:boto:path=/ffzdqu-53/b 2023-09-22T21:24:21.312 DEBUG:boto:auth_path=/ffzdqu-53/b 2023-09-22T21:24:21.312 DEBUG:boto:Method: PUT 2023-09-22T21:24:21.313 DEBUG:boto:Path: /ffzdqu-53/b 2023-09-22T21:24:21.313 DEBUG:boto:Data: 2023-09-22T21:24:21.313 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:24:21.313 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:21.313 DEBUG:boto:Port: 8000 2023-09-22T21:24:21.313 DEBUG:boto:Params: {} 2023-09-22T21:24:21.313 DEBUG:boto:Token: None 2023-09-22T21:24:21.313 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:24:21 GMT /ffzdqu-53/b 2023-09-22T21:24:21.313 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:j35/THx89OrluYct+b8lWJ+Onag= 2023-09-22T21:24:21.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:24:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:j35/THx89OrluYct+b8lWJ+Onag='} 2023-09-22T21:24:21.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e60e7409e5ae5aa-00650e0605-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:21.363 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:21.363 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:21.363 DEBUG:boto:Method: HEAD 2023-09-22T21:24:21.363 DEBUG:boto:Path: /ffzdqu-53/ 2023-09-22T21:24:21.363 DEBUG:boto:Data: 2023-09-22T21:24:21.363 DEBUG:boto:Headers: {} 2023-09-22T21:24:21.363 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:21.363 DEBUG:boto:Port: 8000 2023-09-22T21:24:21.363 DEBUG:boto:Params: {} 2023-09-22T21:24:21.364 DEBUG:boto:Token: None 2023-09-22T21:24:21.364 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:21 GMT /ffzdqu-53/ 2023-09-22T21:24:21.364 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2ZXzQ6itPUNLVTNnwBifBs9TJ/g= 2023-09-22T21:24:21.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2ZXzQ6itPUNLVTNnwBifBs9TJ/g=', 'Content-Length': '0'} 2023-09-22T21:24:21.368 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006bf6538141f0dd45-00650e0605-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:21.368 DEBUG:boto:path=/ffzdqu-53/c 2023-09-22T21:24:21.368 DEBUG:boto:auth_path=/ffzdqu-53/c 2023-09-22T21:24:21.368 DEBUG:boto:Method: PUT 2023-09-22T21:24:21.368 DEBUG:boto:Path: /ffzdqu-53/c 2023-09-22T21:24:21.368 DEBUG:boto:Data: 2023-09-22T21:24:21.368 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:24:21.368 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:21.369 DEBUG:boto:Port: 8000 2023-09-22T21:24:21.369 DEBUG:boto:Params: {} 2023-09-22T21:24:21.369 DEBUG:boto:Token: None 2023-09-22T21:24:21.369 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:24:21 GMT /ffzdqu-53/c 2023-09-22T21:24:21.369 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cgUwzfiN9r0DZj323r0MIaDqJVE= 2023-09-22T21:24:21.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:24:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cgUwzfiN9r0DZj323r0MIaDqJVE='} 2023-09-22T21:24:21.382 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de79b96399113799-00650e0605-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:21.382 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:21.383 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:21.383 DEBUG:boto:Method: HEAD 2023-09-22T21:24:21.383 DEBUG:boto:Path: /ffzdqu-53/ 2023-09-22T21:24:21.383 DEBUG:boto:Data: 2023-09-22T21:24:21.383 DEBUG:boto:Headers: {} 2023-09-22T21:24:21.383 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:21.383 DEBUG:boto:Port: 8000 2023-09-22T21:24:21.383 DEBUG:boto:Params: {} 2023-09-22T21:24:21.383 DEBUG:boto:Token: None 2023-09-22T21:24:21.384 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:21 GMT /ffzdqu-53/ 2023-09-22T21:24:21.384 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2ZXzQ6itPUNLVTNnwBifBs9TJ/g= 2023-09-22T21:24:21.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2ZXzQ6itPUNLVTNnwBifBs9TJ/g=', 'Content-Length': '0'} 2023-09-22T21:24:21.387 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007bbe75c3bb531dd4-00650e0605-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:21.388 DEBUG:boto:path=/ffzdqu-53/d 2023-09-22T21:24:21.388 DEBUG:boto:auth_path=/ffzdqu-53/d 2023-09-22T21:24:21.388 DEBUG:boto:Method: PUT 2023-09-22T21:24:21.388 DEBUG:boto:Path: /ffzdqu-53/d 2023-09-22T21:24:21.388 DEBUG:boto:Data: 2023-09-22T21:24:21.388 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:24:21.388 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:21.388 DEBUG:boto:Port: 8000 2023-09-22T21:24:21.389 DEBUG:boto:Params: {} 2023-09-22T21:24:21.389 DEBUG:boto:Token: None 2023-09-22T21:24:21.389 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:24:21 GMT /ffzdqu-53/d 2023-09-22T21:24:21.389 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XbHdy6l5inJHHhlvRjIUxkaoHW0= 2023-09-22T21:24:21.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:24:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XbHdy6l5inJHHhlvRjIUxkaoHW0='} 2023-09-22T21:24:21.434 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bfb237201bbc62b7-00650e0605-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:21.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:21.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:21.435 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:24:21.478 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:24:21.478 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:24:21.518 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:24:21.537 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695417861.283162_398.1","last_update":"2023-09-22T21:24:21.283162Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:24:21.537 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417861.283162_398.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:24:21.538 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:24:21.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:21.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:21.538 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:24:21.620 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:24:21.620 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:24:21.662 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:21.659+0000 7fc25765fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:24:21.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:21.663+0000 7fc25765fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:24:21.680 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417861.283162_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:24:21.283162Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:24:21Z"} 2023-09-22T21:24:21.680 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417861.283162_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:24:21.283162Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:24:21Z"} 2023-09-22T21:24:21.681 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417861.283162_398.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:24:21.283162Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:24:21.681 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417861.283162_398.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:24:21.681 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417861.283162_398.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:24:21.681 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:24:21.681 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-53 2023-09-22T21:24:21.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-53', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:24:21.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-53', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:24:21.682 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-53 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:24:21.764 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:24:21.765 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:24:21.806 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:21.803+0000 7fc6e8c66a40 1 bucket sync caught up with source: 2023-09-22T21:24:21.806 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.18.6, , , , 00000000001.47.6, , , 00000000001.31.6, , 00000000001.33.6] 2023-09-22T21:24:21.807 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.18.6, , , , 00000000001.47.6, , , 00000000001.31.6, , 00000000001.33.6] 2023-09-22T21:24:21.807 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:21.803+0000 7fc6e8c66a40 0 bucket checkpoint complete 2023-09-22T21:24:21.823 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-53 2023-09-22T21:24:21.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-53', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:24:21.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-53', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:24:21.824 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-53 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:24:21.913 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:24:21.913 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:24:21.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:21.951+0000 7f121f7e0a40 1 waiting to reach incremental sync.. 2023-09-22T21:24:26.957 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:26.955+0000 7f121f7e0a40 1 bucket sync caught up with source: 2023-09-22T21:24:26.958 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.93.6, , , , 00000000001.142.6, , , 00000000001.268.6, , 00000000001.328.6] 2023-09-22T21:24:26.958 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.93.6, , , , 00000000001.142.6, , , 00000000001.268.6, , 00000000001.328.6] 2023-09-22T21:24:26.958 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:26.955+0000 7f121f7e0a40 0 bucket checkpoint complete 2023-09-22T21:24:26.984 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-53 zones={a2, a1} 2023-09-22T21:24:26.984 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:26.984 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:26.985 DEBUG:boto:Method: HEAD 2023-09-22T21:24:26.985 DEBUG:boto:Path: /ffzdqu-53/ 2023-09-22T21:24:26.985 DEBUG:boto:Data: 2023-09-22T21:24:26.985 DEBUG:boto:Headers: {} 2023-09-22T21:24:26.985 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:26.985 DEBUG:boto:Port: 8001 2023-09-22T21:24:26.985 DEBUG:boto:Params: {} 2023-09-22T21:24:26.985 DEBUG:boto:Token: None 2023-09-22T21:24:26.986 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:26 GMT /ffzdqu-53/ 2023-09-22T21:24:26.986 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xJ9+eHgrsidHVcPO3UNK3Y4P7fI= 2023-09-22T21:24:26.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xJ9+eHgrsidHVcPO3UNK3Y4P7fI=', 'Content-Length': '0'} 2023-09-22T21:24:26.989 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7a3dde44fed8fa4-00650e060a-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:26.990 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:26.990 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:26.990 DEBUG:boto:Method: HEAD 2023-09-22T21:24:26.990 DEBUG:boto:Path: /ffzdqu-53/ 2023-09-22T21:24:26.990 DEBUG:boto:Data: 2023-09-22T21:24:26.990 DEBUG:boto:Headers: {} 2023-09-22T21:24:26.990 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:26.990 DEBUG:boto:Port: 8000 2023-09-22T21:24:26.991 DEBUG:boto:Params: {} 2023-09-22T21:24:26.991 DEBUG:boto:Token: None 2023-09-22T21:24:26.991 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:26 GMT /ffzdqu-53/ 2023-09-22T21:24:26.991 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xJ9+eHgrsidHVcPO3UNK3Y4P7fI= 2023-09-22T21:24:26.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xJ9+eHgrsidHVcPO3UNK3Y4P7fI=', 'Content-Length': '0'} 2023-09-22T21:24:26.994 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd1351630a3dfebf-00650e060a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:26.994 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:24:26.995 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:26.995 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:26.995 DEBUG:boto:path=/ffzdqu-53/?versions&max-keys=999 2023-09-22T21:24:26.995 DEBUG:boto:auth_path=/ffzdqu-53/?versions&max-keys=999 2023-09-22T21:24:26.995 DEBUG:boto:Method: GET 2023-09-22T21:24:26.995 DEBUG:boto:Path: /ffzdqu-53/?versions&max-keys=999 2023-09-22T21:24:26.995 DEBUG:boto:Data: 2023-09-22T21:24:26.995 DEBUG:boto:Headers: {} 2023-09-22T21:24:26.995 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:26.995 DEBUG:boto:Port: 8001 2023-09-22T21:24:26.996 DEBUG:boto:Params: {} 2023-09-22T21:24:26.996 DEBUG:boto:Token: None 2023-09-22T21:24:26.996 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:26 GMT /ffzdqu-53/?versions 2023-09-22T21:24:26.996 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CMWZ8koCKudVj19TQ0ndu9DNvBE= 2023-09-22T21:24:26.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CMWZ8koCKudVj19TQ0ndu9DNvBE=', 'Content-Length': '0'} 2023-09-22T21:24:26.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c1ae74e642466ab-00650e060a-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.000 DEBUG:boto:b'ffzdqu-53999falseanulltrue2023-09-22T21:24:21.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:21.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:21.375Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:21.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:27.000 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-22T21:24:27.000 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-22T21:24:27.000 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-22T21:24:27.001 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-22T21:24:27.001 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:24:27.001 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:27.001 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:27.001 DEBUG:boto:path=/ffzdqu-53/?versions&max-keys=999 2023-09-22T21:24:27.001 DEBUG:boto:auth_path=/ffzdqu-53/?versions&max-keys=999 2023-09-22T21:24:27.001 DEBUG:boto:Method: GET 2023-09-22T21:24:27.001 DEBUG:boto:Path: /ffzdqu-53/?versions&max-keys=999 2023-09-22T21:24:27.001 DEBUG:boto:Data: 2023-09-22T21:24:27.001 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.002 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.002 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.002 DEBUG:boto:Params: {} 2023-09-22T21:24:27.002 DEBUG:boto:Token: None 2023-09-22T21:24:27.002 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/?versions 2023-09-22T21:24:27.002 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BG/T7bfd4QjtHGjsBS1+JEyNzrE= 2023-09-22T21:24:27.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BG/T7bfd4QjtHGjsBS1+JEyNzrE=', 'Content-Length': '0'} 2023-09-22T21:24:27.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007829eb55af8ccc5c-00650e060b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.006 DEBUG:boto:b'ffzdqu-53999falseanulltrue2023-09-22T21:24:21.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:21.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:21.375Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:21.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:27.007 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-22T21:24:27.007 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-22T21:24:27.007 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-22T21:24:27.007 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-22T21:24:27.007 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:27.007 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:27.007 DEBUG:boto:path=/ffzdqu-53/?versions&max-keys=999 2023-09-22T21:24:27.007 DEBUG:boto:auth_path=/ffzdqu-53/?versions&max-keys=999 2023-09-22T21:24:27.008 DEBUG:boto:Method: GET 2023-09-22T21:24:27.008 DEBUG:boto:Path: /ffzdqu-53/?versions&max-keys=999 2023-09-22T21:24:27.008 DEBUG:boto:Data: 2023-09-22T21:24:27.008 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.008 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.008 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.008 DEBUG:boto:Params: {} 2023-09-22T21:24:27.008 DEBUG:boto:Token: None 2023-09-22T21:24:27.008 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/?versions 2023-09-22T21:24:27.009 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BG/T7bfd4QjtHGjsBS1+JEyNzrE= 2023-09-22T21:24:27.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BG/T7bfd4QjtHGjsBS1+JEyNzrE=', 'Content-Length': '0'} 2023-09-22T21:24:27.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ceeac4e2f2a07759-00650e060b-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.012 DEBUG:boto:b'ffzdqu-53999falseanulltrue2023-09-22T21:24:21.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:21.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:21.375Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:21.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:27.013 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:27.013 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:27.013 DEBUG:boto:path=/ffzdqu-53/?versions&max-keys=999 2023-09-22T21:24:27.013 DEBUG:boto:auth_path=/ffzdqu-53/?versions&max-keys=999 2023-09-22T21:24:27.013 DEBUG:boto:Method: GET 2023-09-22T21:24:27.013 DEBUG:boto:Path: /ffzdqu-53/?versions&max-keys=999 2023-09-22T21:24:27.013 DEBUG:boto:Data: 2023-09-22T21:24:27.013 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.013 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.014 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.014 DEBUG:boto:Params: {} 2023-09-22T21:24:27.014 DEBUG:boto:Token: None 2023-09-22T21:24:27.014 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/?versions 2023-09-22T21:24:27.014 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BG/T7bfd4QjtHGjsBS1+JEyNzrE= 2023-09-22T21:24:27.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BG/T7bfd4QjtHGjsBS1+JEyNzrE=', 'Content-Length': '0'} 2023-09-22T21:24:27.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb0df4bdd64959b1-00650e060b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.018 DEBUG:boto:b'ffzdqu-53999falseanulltrue2023-09-22T21:24:21.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:21.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:21.375Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:21.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:27.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:24:27.019 DEBUG:boto:path=/ffzdqu-53/a 2023-09-22T21:24:27.019 DEBUG:boto:auth_path=/ffzdqu-53/a 2023-09-22T21:24:27.019 DEBUG:boto:path=/ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.019 DEBUG:boto:auth_path=/ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.019 DEBUG:boto:Method: GET 2023-09-22T21:24:27.019 DEBUG:boto:Path: /ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.019 DEBUG:boto:Data: 2023-09-22T21:24:27.019 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.019 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.020 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.020 DEBUG:boto:Params: {} 2023-09-22T21:24:27.020 DEBUG:boto:Token: None 2023-09-22T21:24:27.020 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.020 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VfNN9+4twVDTMLZWcb8HmClHlxs= 2023-09-22T21:24:27.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VfNN9+4twVDTMLZWcb8HmClHlxs=', 'Content-Length': '0'} 2023-09-22T21:24:27.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000940d7f37b402d198-00650e060b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.070 DEBUG:boto:path=/ffzdqu-53/a 2023-09-22T21:24:27.071 DEBUG:boto:auth_path=/ffzdqu-53/a 2023-09-22T21:24:27.071 DEBUG:boto:path=/ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.071 DEBUG:boto:auth_path=/ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.071 DEBUG:boto:Method: GET 2023-09-22T21:24:27.071 DEBUG:boto:Path: /ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.071 DEBUG:boto:Data: 2023-09-22T21:24:27.071 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.071 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.071 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.071 DEBUG:boto:Params: {} 2023-09-22T21:24:27.072 DEBUG:boto:Token: None 2023-09-22T21:24:27.072 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.072 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VfNN9+4twVDTMLZWcb8HmClHlxs= 2023-09-22T21:24:27.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VfNN9+4twVDTMLZWcb8HmClHlxs=', 'Content-Length': '0'} 2023-09-22T21:24:27.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000018fd7ec0f127178-00650e060b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.075 DEBUG:boto:path=/ffzdqu-53/a 2023-09-22T21:24:27.075 DEBUG:boto:auth_path=/ffzdqu-53/a 2023-09-22T21:24:27.076 DEBUG:boto:path=/ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.076 DEBUG:boto:auth_path=/ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.077 DEBUG:boto:Method: HEAD 2023-09-22T21:24:27.077 DEBUG:boto:Path: /ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.077 DEBUG:boto:Data: 2023-09-22T21:24:27.077 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.077 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.077 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.077 DEBUG:boto:Params: {} 2023-09-22T21:24:27.077 DEBUG:boto:Token: None 2023-09-22T21:24:27.077 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.078 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:L4U4pAg5V9Z6rTVb/UW8BQsmXKA= 2023-09-22T21:24:27.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:L4U4pAg5V9Z6rTVb/UW8BQsmXKA=', 'Content-Length': '0'} 2023-09-22T21:24:27.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000959d9800645896a7-00650e060b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.080 DEBUG:boto:path=/ffzdqu-53/a 2023-09-22T21:24:27.081 DEBUG:boto:auth_path=/ffzdqu-53/a 2023-09-22T21:24:27.081 DEBUG:boto:path=/ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.081 DEBUG:boto:auth_path=/ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.081 DEBUG:boto:Method: HEAD 2023-09-22T21:24:27.081 DEBUG:boto:Path: /ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.081 DEBUG:boto:Data: 2023-09-22T21:24:27.081 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.081 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.081 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.082 DEBUG:boto:Params: {} 2023-09-22T21:24:27.082 DEBUG:boto:Token: None 2023-09-22T21:24:27.082 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.082 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:L4U4pAg5V9Z6rTVb/UW8BQsmXKA= 2023-09-22T21:24:27.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:L4U4pAg5V9Z6rTVb/UW8BQsmXKA=', 'Content-Length': '0'} 2023-09-22T21:24:27.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b35f12d0801ee6f2-00650e060b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:24:27.085 DEBUG:boto:path=/ffzdqu-53/a 2023-09-22T21:24:27.085 DEBUG:boto:auth_path=/ffzdqu-53/a 2023-09-22T21:24:27.085 DEBUG:boto:path=/ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.085 DEBUG:boto:auth_path=/ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.085 DEBUG:boto:Method: GET 2023-09-22T21:24:27.086 DEBUG:boto:Path: /ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.086 DEBUG:boto:Data: 2023-09-22T21:24:27.086 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.086 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.086 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.086 DEBUG:boto:Params: {} 2023-09-22T21:24:27.086 DEBUG:boto:Token: None 2023-09-22T21:24:27.086 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.087 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VfNN9+4twVDTMLZWcb8HmClHlxs= 2023-09-22T21:24:27.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VfNN9+4twVDTMLZWcb8HmClHlxs=', 'Content-Length': '0'} 2023-09-22T21:24:27.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cba7ced7bd5d722d-00650e060b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.090 DEBUG:boto:path=/ffzdqu-53/a 2023-09-22T21:24:27.090 DEBUG:boto:auth_path=/ffzdqu-53/a 2023-09-22T21:24:27.090 DEBUG:boto:path=/ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.090 DEBUG:boto:auth_path=/ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.090 DEBUG:boto:Method: GET 2023-09-22T21:24:27.091 DEBUG:boto:Path: /ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.091 DEBUG:boto:Data: 2023-09-22T21:24:27.091 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.091 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.091 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.091 DEBUG:boto:Params: {} 2023-09-22T21:24:27.091 DEBUG:boto:Token: None 2023-09-22T21:24:27.091 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/a?versionId=null 2023-09-22T21:24:27.091 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VfNN9+4twVDTMLZWcb8HmClHlxs= 2023-09-22T21:24:27.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VfNN9+4twVDTMLZWcb8HmClHlxs=', 'Content-Length': '0'} 2023-09-22T21:24:27.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000532fbe54fa33df7e-00650e060b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.094 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:27.094 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:27.094 DEBUG:boto:path=/ffzdqu-53/?prefix=a 2023-09-22T21:24:27.095 DEBUG:boto:auth_path=/ffzdqu-53/?prefix=a 2023-09-22T21:24:27.095 DEBUG:boto:Method: GET 2023-09-22T21:24:27.095 DEBUG:boto:Path: /ffzdqu-53/?prefix=a 2023-09-22T21:24:27.095 DEBUG:boto:Data: 2023-09-22T21:24:27.095 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.095 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.095 DEBUG:boto:Params: {} 2023-09-22T21:24:27.095 DEBUG:boto:Token: None 2023-09-22T21:24:27.096 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/ 2023-09-22T21:24:27.096 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18= 2023-09-22T21:24:27.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18=', 'Content-Length': '0'} 2023-09-22T21:24:27.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4aa47667bc57158-00650e060b-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.100 DEBUG:boto:b'ffzdqu-53a1000falsea2023-09-22T21:24:21.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:27.101 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:27.101 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:27.101 DEBUG:boto:path=/ffzdqu-53/?prefix=a 2023-09-22T21:24:27.101 DEBUG:boto:auth_path=/ffzdqu-53/?prefix=a 2023-09-22T21:24:27.101 DEBUG:boto:Method: GET 2023-09-22T21:24:27.101 DEBUG:boto:Path: /ffzdqu-53/?prefix=a 2023-09-22T21:24:27.101 DEBUG:boto:Data: 2023-09-22T21:24:27.101 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.101 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.102 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.102 DEBUG:boto:Params: {} 2023-09-22T21:24:27.102 DEBUG:boto:Token: None 2023-09-22T21:24:27.102 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/ 2023-09-22T21:24:27.102 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18= 2023-09-22T21:24:27.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18=', 'Content-Length': '0'} 2023-09-22T21:24:27.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdd78152a77cb181-00650e060b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.107 DEBUG:boto:b'ffzdqu-53a1000falsea2023-09-22T21:24:21.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:27.107 DEBUG:boto:path=/ffzdqu-53/a 2023-09-22T21:24:27.108 DEBUG:boto:auth_path=/ffzdqu-53/a 2023-09-22T21:24:27.108 DEBUG:boto:Method: HEAD 2023-09-22T21:24:27.108 DEBUG:boto:Path: /ffzdqu-53/a 2023-09-22T21:24:27.108 DEBUG:boto:Data: 2023-09-22T21:24:27.108 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.108 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.108 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.108 DEBUG:boto:Params: {} 2023-09-22T21:24:27.108 DEBUG:boto:Token: None 2023-09-22T21:24:27.109 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/a 2023-09-22T21:24:27.109 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cSINa0PuF+lGiVr5PlwBl/3ZShI= 2023-09-22T21:24:27.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cSINa0PuF+lGiVr5PlwBl/3ZShI=', 'Content-Length': '0'} 2023-09-22T21:24:27.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f85e65f41979256d-00650e060b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.111 DEBUG:boto:path=/ffzdqu-53/a 2023-09-22T21:24:27.112 DEBUG:boto:auth_path=/ffzdqu-53/a 2023-09-22T21:24:27.112 DEBUG:boto:Method: HEAD 2023-09-22T21:24:27.112 DEBUG:boto:Path: /ffzdqu-53/a 2023-09-22T21:24:27.112 DEBUG:boto:Data: 2023-09-22T21:24:27.112 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.112 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.112 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.112 DEBUG:boto:Params: {} 2023-09-22T21:24:27.112 DEBUG:boto:Token: None 2023-09-22T21:24:27.113 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/a 2023-09-22T21:24:27.113 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cSINa0PuF+lGiVr5PlwBl/3ZShI= 2023-09-22T21:24:27.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cSINa0PuF+lGiVr5PlwBl/3ZShI=', 'Content-Length': '0'} 2023-09-22T21:24:27.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000522d49a4edf5b9-00650e060b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:24:27.116 DEBUG:boto:path=/ffzdqu-53/a 2023-09-22T21:24:27.116 DEBUG:boto:auth_path=/ffzdqu-53/a 2023-09-22T21:24:27.116 DEBUG:boto:Method: GET 2023-09-22T21:24:27.116 DEBUG:boto:Path: /ffzdqu-53/a 2023-09-22T21:24:27.116 DEBUG:boto:Data: 2023-09-22T21:24:27.116 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.116 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.116 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.116 DEBUG:boto:Params: {} 2023-09-22T21:24:27.117 DEBUG:boto:Token: None 2023-09-22T21:24:27.117 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/a 2023-09-22T21:24:27.117 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wO1ZXVUNPfcjOkB+G+EFQeXLKVE= 2023-09-22T21:24:27.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wO1ZXVUNPfcjOkB+G+EFQeXLKVE=', 'Content-Length': '0'} 2023-09-22T21:24:27.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cfcc88adbb2d137-00650e060b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.162 DEBUG:boto:path=/ffzdqu-53/a 2023-09-22T21:24:27.163 DEBUG:boto:auth_path=/ffzdqu-53/a 2023-09-22T21:24:27.163 DEBUG:boto:Method: GET 2023-09-22T21:24:27.163 DEBUG:boto:Path: /ffzdqu-53/a 2023-09-22T21:24:27.163 DEBUG:boto:Data: 2023-09-22T21:24:27.163 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.163 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.163 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.163 DEBUG:boto:Params: {} 2023-09-22T21:24:27.163 DEBUG:boto:Token: None 2023-09-22T21:24:27.164 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/a 2023-09-22T21:24:27.164 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wO1ZXVUNPfcjOkB+G+EFQeXLKVE= 2023-09-22T21:24:27.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wO1ZXVUNPfcjOkB+G+EFQeXLKVE=', 'Content-Length': '0'} 2023-09-22T21:24:27.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004843e268a07c3c6d-00650e060b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.167 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:27.167 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:27.167 DEBUG:boto:path=/ffzdqu-53/?prefix=a 2023-09-22T21:24:27.167 DEBUG:boto:auth_path=/ffzdqu-53/?prefix=a 2023-09-22T21:24:27.167 DEBUG:boto:Method: GET 2023-09-22T21:24:27.167 DEBUG:boto:Path: /ffzdqu-53/?prefix=a 2023-09-22T21:24:27.168 DEBUG:boto:Data: 2023-09-22T21:24:27.168 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.168 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.168 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.168 DEBUG:boto:Params: {} 2023-09-22T21:24:27.168 DEBUG:boto:Token: None 2023-09-22T21:24:27.168 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/ 2023-09-22T21:24:27.168 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18= 2023-09-22T21:24:27.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18=', 'Content-Length': '0'} 2023-09-22T21:24:27.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b7d312062542254-00650e060b-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.173 DEBUG:boto:b'ffzdqu-53a1000falsea2023-09-22T21:24:21.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:27.173 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:27.173 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:27.173 DEBUG:boto:path=/ffzdqu-53/?prefix=a 2023-09-22T21:24:27.173 DEBUG:boto:auth_path=/ffzdqu-53/?prefix=a 2023-09-22T21:24:27.174 DEBUG:boto:Method: GET 2023-09-22T21:24:27.174 DEBUG:boto:Path: /ffzdqu-53/?prefix=a 2023-09-22T21:24:27.174 DEBUG:boto:Data: 2023-09-22T21:24:27.174 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.174 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.174 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.174 DEBUG:boto:Params: {} 2023-09-22T21:24:27.174 DEBUG:boto:Token: None 2023-09-22T21:24:27.174 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/ 2023-09-22T21:24:27.175 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18= 2023-09-22T21:24:27.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18=', 'Content-Length': '0'} 2023-09-22T21:24:27.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007a2389037d6531a-00650e060b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.226 DEBUG:boto:b'ffzdqu-53a1000falsea2023-09-22T21:24:21.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:27.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:24:27.227 DEBUG:boto:path=/ffzdqu-53/b 2023-09-22T21:24:27.227 DEBUG:boto:auth_path=/ffzdqu-53/b 2023-09-22T21:24:27.227 DEBUG:boto:path=/ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.227 DEBUG:boto:auth_path=/ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.227 DEBUG:boto:Method: GET 2023-09-22T21:24:27.227 DEBUG:boto:Path: /ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.227 DEBUG:boto:Data: 2023-09-22T21:24:27.228 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.228 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.228 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.228 DEBUG:boto:Params: {} 2023-09-22T21:24:27.228 DEBUG:boto:Token: None 2023-09-22T21:24:27.228 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.228 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fuF04Po4LhGk6dM7z0tdjvCBIkY= 2023-09-22T21:24:27.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fuF04Po4LhGk6dM7z0tdjvCBIkY=', 'Content-Length': '0'} 2023-09-22T21:24:27.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e5efeab83c91987-00650e060b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.231 DEBUG:boto:path=/ffzdqu-53/b 2023-09-22T21:24:27.231 DEBUG:boto:auth_path=/ffzdqu-53/b 2023-09-22T21:24:27.232 DEBUG:boto:path=/ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.232 DEBUG:boto:auth_path=/ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.232 DEBUG:boto:Method: GET 2023-09-22T21:24:27.232 DEBUG:boto:Path: /ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.232 DEBUG:boto:Data: 2023-09-22T21:24:27.232 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.232 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.232 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.232 DEBUG:boto:Params: {} 2023-09-22T21:24:27.232 DEBUG:boto:Token: None 2023-09-22T21:24:27.233 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.233 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fuF04Po4LhGk6dM7z0tdjvCBIkY= 2023-09-22T21:24:27.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fuF04Po4LhGk6dM7z0tdjvCBIkY=', 'Content-Length': '0'} 2023-09-22T21:24:27.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3908f0c61008b15-00650e060b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.236 DEBUG:boto:path=/ffzdqu-53/b 2023-09-22T21:24:27.236 DEBUG:boto:auth_path=/ffzdqu-53/b 2023-09-22T21:24:27.236 DEBUG:boto:path=/ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.236 DEBUG:boto:auth_path=/ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.236 DEBUG:boto:Method: HEAD 2023-09-22T21:24:27.236 DEBUG:boto:Path: /ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.236 DEBUG:boto:Data: 2023-09-22T21:24:27.236 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.237 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.237 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.237 DEBUG:boto:Params: {} 2023-09-22T21:24:27.237 DEBUG:boto:Token: None 2023-09-22T21:24:27.237 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.237 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wmeOvVARX+bsFaATcCyUQn42Vwo= 2023-09-22T21:24:27.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wmeOvVARX+bsFaATcCyUQn42Vwo=', 'Content-Length': '0'} 2023-09-22T21:24:27.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea4e3e38447866a8-00650e060b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.240 DEBUG:boto:path=/ffzdqu-53/b 2023-09-22T21:24:27.240 DEBUG:boto:auth_path=/ffzdqu-53/b 2023-09-22T21:24:27.240 DEBUG:boto:path=/ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.240 DEBUG:boto:auth_path=/ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.240 DEBUG:boto:Method: HEAD 2023-09-22T21:24:27.240 DEBUG:boto:Path: /ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.241 DEBUG:boto:Data: 2023-09-22T21:24:27.241 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.241 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.241 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.241 DEBUG:boto:Params: {} 2023-09-22T21:24:27.241 DEBUG:boto:Token: None 2023-09-22T21:24:27.241 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.241 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wmeOvVARX+bsFaATcCyUQn42Vwo= 2023-09-22T21:24:27.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wmeOvVARX+bsFaATcCyUQn42Vwo=', 'Content-Length': '0'} 2023-09-22T21:24:27.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000431eed033a3a574f-00650e060b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:24:27.244 DEBUG:boto:path=/ffzdqu-53/b 2023-09-22T21:24:27.244 DEBUG:boto:auth_path=/ffzdqu-53/b 2023-09-22T21:24:27.245 DEBUG:boto:path=/ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.245 DEBUG:boto:auth_path=/ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.245 DEBUG:boto:Method: GET 2023-09-22T21:24:27.245 DEBUG:boto:Path: /ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.245 DEBUG:boto:Data: 2023-09-22T21:24:27.245 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.245 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.245 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.245 DEBUG:boto:Params: {} 2023-09-22T21:24:27.245 DEBUG:boto:Token: None 2023-09-22T21:24:27.246 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.246 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fuF04Po4LhGk6dM7z0tdjvCBIkY= 2023-09-22T21:24:27.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fuF04Po4LhGk6dM7z0tdjvCBIkY=', 'Content-Length': '0'} 2023-09-22T21:24:27.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aecdbc0a4eb47b07-00650e060b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.290 DEBUG:boto:path=/ffzdqu-53/b 2023-09-22T21:24:27.290 DEBUG:boto:auth_path=/ffzdqu-53/b 2023-09-22T21:24:27.291 DEBUG:boto:path=/ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.291 DEBUG:boto:auth_path=/ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.291 DEBUG:boto:Method: GET 2023-09-22T21:24:27.291 DEBUG:boto:Path: /ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.291 DEBUG:boto:Data: 2023-09-22T21:24:27.291 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.291 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.291 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.291 DEBUG:boto:Params: {} 2023-09-22T21:24:27.291 DEBUG:boto:Token: None 2023-09-22T21:24:27.293 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/b?versionId=null 2023-09-22T21:24:27.293 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fuF04Po4LhGk6dM7z0tdjvCBIkY= 2023-09-22T21:24:27.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fuF04Po4LhGk6dM7z0tdjvCBIkY=', 'Content-Length': '0'} 2023-09-22T21:24:27.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071836bf8903f1c02-00650e060b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.296 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:27.296 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:27.296 DEBUG:boto:path=/ffzdqu-53/?prefix=b 2023-09-22T21:24:27.297 DEBUG:boto:auth_path=/ffzdqu-53/?prefix=b 2023-09-22T21:24:27.297 DEBUG:boto:Method: GET 2023-09-22T21:24:27.298 DEBUG:boto:Path: /ffzdqu-53/?prefix=b 2023-09-22T21:24:27.298 DEBUG:boto:Data: 2023-09-22T21:24:27.298 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.298 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.298 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.298 DEBUG:boto:Params: {} 2023-09-22T21:24:27.298 DEBUG:boto:Token: None 2023-09-22T21:24:27.299 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/ 2023-09-22T21:24:27.299 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18= 2023-09-22T21:24:27.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18=', 'Content-Length': '0'} 2023-09-22T21:24:27.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d20a630787b0de9a-00650e060b-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.303 DEBUG:boto:b'ffzdqu-53b1000falseb2023-09-22T21:24:21.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:27.304 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:27.304 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:27.304 DEBUG:boto:path=/ffzdqu-53/?prefix=b 2023-09-22T21:24:27.304 DEBUG:boto:auth_path=/ffzdqu-53/?prefix=b 2023-09-22T21:24:27.304 DEBUG:boto:Method: GET 2023-09-22T21:24:27.305 DEBUG:boto:Path: /ffzdqu-53/?prefix=b 2023-09-22T21:24:27.305 DEBUG:boto:Data: 2023-09-22T21:24:27.305 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.305 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.305 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.305 DEBUG:boto:Params: {} 2023-09-22T21:24:27.305 DEBUG:boto:Token: None 2023-09-22T21:24:27.305 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/ 2023-09-22T21:24:27.306 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18= 2023-09-22T21:24:27.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18=', 'Content-Length': '0'} 2023-09-22T21:24:27.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aae8ede299126778-00650e060b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.354 DEBUG:boto:b'ffzdqu-53b1000falseb2023-09-22T21:24:21.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:27.355 DEBUG:boto:path=/ffzdqu-53/b 2023-09-22T21:24:27.355 DEBUG:boto:auth_path=/ffzdqu-53/b 2023-09-22T21:24:27.355 DEBUG:boto:Method: HEAD 2023-09-22T21:24:27.355 DEBUG:boto:Path: /ffzdqu-53/b 2023-09-22T21:24:27.355 DEBUG:boto:Data: 2023-09-22T21:24:27.355 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.356 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.356 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.356 DEBUG:boto:Params: {} 2023-09-22T21:24:27.356 DEBUG:boto:Token: None 2023-09-22T21:24:27.356 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/b 2023-09-22T21:24:27.356 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oBJgw0zcHURC2Ct+lXC7GeHrEwE= 2023-09-22T21:24:27.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oBJgw0zcHURC2Ct+lXC7GeHrEwE=', 'Content-Length': '0'} 2023-09-22T21:24:27.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b4adbf072316bba-00650e060b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.359 DEBUG:boto:path=/ffzdqu-53/b 2023-09-22T21:24:27.360 DEBUG:boto:auth_path=/ffzdqu-53/b 2023-09-22T21:24:27.360 DEBUG:boto:Method: HEAD 2023-09-22T21:24:27.360 DEBUG:boto:Path: /ffzdqu-53/b 2023-09-22T21:24:27.360 DEBUG:boto:Data: 2023-09-22T21:24:27.360 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.360 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.360 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.360 DEBUG:boto:Params: {} 2023-09-22T21:24:27.360 DEBUG:boto:Token: None 2023-09-22T21:24:27.361 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/b 2023-09-22T21:24:27.361 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oBJgw0zcHURC2Ct+lXC7GeHrEwE= 2023-09-22T21:24:27.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oBJgw0zcHURC2Ct+lXC7GeHrEwE=', 'Content-Length': '0'} 2023-09-22T21:24:27.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f89ae1b49e48ae76-00650e060b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:24:27.364 DEBUG:boto:path=/ffzdqu-53/b 2023-09-22T21:24:27.364 DEBUG:boto:auth_path=/ffzdqu-53/b 2023-09-22T21:24:27.364 DEBUG:boto:Method: GET 2023-09-22T21:24:27.364 DEBUG:boto:Path: /ffzdqu-53/b 2023-09-22T21:24:27.364 DEBUG:boto:Data: 2023-09-22T21:24:27.364 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.364 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.365 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.365 DEBUG:boto:Params: {} 2023-09-22T21:24:27.365 DEBUG:boto:Token: None 2023-09-22T21:24:27.365 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/b 2023-09-22T21:24:27.365 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xbeWSLtwR9WrhIjC1xt07Pob36A= 2023-09-22T21:24:27.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xbeWSLtwR9WrhIjC1xt07Pob36A=', 'Content-Length': '0'} 2023-09-22T21:24:27.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000102c6cb0e8765067-00650e060b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.410 DEBUG:boto:path=/ffzdqu-53/b 2023-09-22T21:24:27.411 DEBUG:boto:auth_path=/ffzdqu-53/b 2023-09-22T21:24:27.411 DEBUG:boto:Method: GET 2023-09-22T21:24:27.411 DEBUG:boto:Path: /ffzdqu-53/b 2023-09-22T21:24:27.411 DEBUG:boto:Data: 2023-09-22T21:24:27.411 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.411 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.411 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.411 DEBUG:boto:Params: {} 2023-09-22T21:24:27.411 DEBUG:boto:Token: None 2023-09-22T21:24:27.412 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/b 2023-09-22T21:24:27.412 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xbeWSLtwR9WrhIjC1xt07Pob36A= 2023-09-22T21:24:27.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xbeWSLtwR9WrhIjC1xt07Pob36A=', 'Content-Length': '0'} 2023-09-22T21:24:27.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e89c6d9f4953cdb1-00650e060b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.415 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:27.416 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:27.416 DEBUG:boto:path=/ffzdqu-53/?prefix=b 2023-09-22T21:24:27.416 DEBUG:boto:auth_path=/ffzdqu-53/?prefix=b 2023-09-22T21:24:27.416 DEBUG:boto:Method: GET 2023-09-22T21:24:27.416 DEBUG:boto:Path: /ffzdqu-53/?prefix=b 2023-09-22T21:24:27.416 DEBUG:boto:Data: 2023-09-22T21:24:27.416 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.416 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.417 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.417 DEBUG:boto:Params: {} 2023-09-22T21:24:27.417 DEBUG:boto:Token: None 2023-09-22T21:24:27.417 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/ 2023-09-22T21:24:27.417 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18= 2023-09-22T21:24:27.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18=', 'Content-Length': '0'} 2023-09-22T21:24:27.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000329c8f5abe9db884-00650e060b-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.422 DEBUG:boto:b'ffzdqu-53b1000falseb2023-09-22T21:24:21.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:27.423 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:27.423 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:27.423 DEBUG:boto:path=/ffzdqu-53/?prefix=b 2023-09-22T21:24:27.423 DEBUG:boto:auth_path=/ffzdqu-53/?prefix=b 2023-09-22T21:24:27.423 DEBUG:boto:Method: GET 2023-09-22T21:24:27.423 DEBUG:boto:Path: /ffzdqu-53/?prefix=b 2023-09-22T21:24:27.423 DEBUG:boto:Data: 2023-09-22T21:24:27.423 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.424 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.424 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.424 DEBUG:boto:Params: {} 2023-09-22T21:24:27.424 DEBUG:boto:Token: None 2023-09-22T21:24:27.424 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/ 2023-09-22T21:24:27.424 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18= 2023-09-22T21:24:27.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18=', 'Content-Length': '0'} 2023-09-22T21:24:27.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000508ca6c8c2026a52-00650e060b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.430 DEBUG:boto:b'ffzdqu-53b1000falseb2023-09-22T21:24:21.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:27.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:24:27.430 DEBUG:boto:path=/ffzdqu-53/c 2023-09-22T21:24:27.430 DEBUG:boto:auth_path=/ffzdqu-53/c 2023-09-22T21:24:27.430 DEBUG:boto:path=/ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.431 DEBUG:boto:auth_path=/ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.431 DEBUG:boto:Method: GET 2023-09-22T21:24:27.431 DEBUG:boto:Path: /ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.431 DEBUG:boto:Data: 2023-09-22T21:24:27.431 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.431 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.431 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.431 DEBUG:boto:Params: {} 2023-09-22T21:24:27.431 DEBUG:boto:Token: None 2023-09-22T21:24:27.432 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.432 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BMtHanZIfqYqVtHJuV4JaTBC6bg= 2023-09-22T21:24:27.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BMtHanZIfqYqVtHJuV4JaTBC6bg=', 'Content-Length': '0'} 2023-09-22T21:24:27.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b550fa4e59814d5d-00650e060b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.435 DEBUG:boto:path=/ffzdqu-53/c 2023-09-22T21:24:27.435 DEBUG:boto:auth_path=/ffzdqu-53/c 2023-09-22T21:24:27.435 DEBUG:boto:path=/ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.435 DEBUG:boto:auth_path=/ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.435 DEBUG:boto:Method: GET 2023-09-22T21:24:27.435 DEBUG:boto:Path: /ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.436 DEBUG:boto:Data: 2023-09-22T21:24:27.436 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.436 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.436 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.436 DEBUG:boto:Params: {} 2023-09-22T21:24:27.436 DEBUG:boto:Token: None 2023-09-22T21:24:27.436 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.436 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BMtHanZIfqYqVtHJuV4JaTBC6bg= 2023-09-22T21:24:27.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BMtHanZIfqYqVtHJuV4JaTBC6bg=', 'Content-Length': '0'} 2023-09-22T21:24:27.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006118ecf1d275be25-00650e060b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.482 DEBUG:boto:path=/ffzdqu-53/c 2023-09-22T21:24:27.483 DEBUG:boto:auth_path=/ffzdqu-53/c 2023-09-22T21:24:27.483 DEBUG:boto:path=/ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.483 DEBUG:boto:auth_path=/ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.483 DEBUG:boto:Method: HEAD 2023-09-22T21:24:27.483 DEBUG:boto:Path: /ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.483 DEBUG:boto:Data: 2023-09-22T21:24:27.483 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.483 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.484 DEBUG:boto:Params: {} 2023-09-22T21:24:27.484 DEBUG:boto:Token: None 2023-09-22T21:24:27.484 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.484 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ccNXBUHsBAezBQ+ae1gQ9Fn8ZZc= 2023-09-22T21:24:27.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ccNXBUHsBAezBQ+ae1gQ9Fn8ZZc=', 'Content-Length': '0'} 2023-09-22T21:24:27.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040399661889aa528-00650e060b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.487 DEBUG:boto:path=/ffzdqu-53/c 2023-09-22T21:24:27.487 DEBUG:boto:auth_path=/ffzdqu-53/c 2023-09-22T21:24:27.487 DEBUG:boto:path=/ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.487 DEBUG:boto:auth_path=/ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.488 DEBUG:boto:Method: HEAD 2023-09-22T21:24:27.488 DEBUG:boto:Path: /ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.488 DEBUG:boto:Data: 2023-09-22T21:24:27.488 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.488 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.488 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.488 DEBUG:boto:Params: {} 2023-09-22T21:24:27.488 DEBUG:boto:Token: None 2023-09-22T21:24:27.488 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.489 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ccNXBUHsBAezBQ+ae1gQ9Fn8ZZc= 2023-09-22T21:24:27.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ccNXBUHsBAezBQ+ae1gQ9Fn8ZZc=', 'Content-Length': '0'} 2023-09-22T21:24:27.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000087f1193faba1b25-00650e060b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:24:27.492 DEBUG:boto:path=/ffzdqu-53/c 2023-09-22T21:24:27.492 DEBUG:boto:auth_path=/ffzdqu-53/c 2023-09-22T21:24:27.492 DEBUG:boto:path=/ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.492 DEBUG:boto:auth_path=/ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.492 DEBUG:boto:Method: GET 2023-09-22T21:24:27.492 DEBUG:boto:Path: /ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.492 DEBUG:boto:Data: 2023-09-22T21:24:27.493 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.493 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.493 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.493 DEBUG:boto:Params: {} 2023-09-22T21:24:27.493 DEBUG:boto:Token: None 2023-09-22T21:24:27.493 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.493 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BMtHanZIfqYqVtHJuV4JaTBC6bg= 2023-09-22T21:24:27.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BMtHanZIfqYqVtHJuV4JaTBC6bg=', 'Content-Length': '0'} 2023-09-22T21:24:27.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087566bb9d81d667b-00650e060b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.538 DEBUG:boto:path=/ffzdqu-53/c 2023-09-22T21:24:27.539 DEBUG:boto:auth_path=/ffzdqu-53/c 2023-09-22T21:24:27.539 DEBUG:boto:path=/ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.539 DEBUG:boto:auth_path=/ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.539 DEBUG:boto:Method: GET 2023-09-22T21:24:27.539 DEBUG:boto:Path: /ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.539 DEBUG:boto:Data: 2023-09-22T21:24:27.539 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.539 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.540 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.540 DEBUG:boto:Params: {} 2023-09-22T21:24:27.540 DEBUG:boto:Token: None 2023-09-22T21:24:27.540 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/c?versionId=null 2023-09-22T21:24:27.540 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BMtHanZIfqYqVtHJuV4JaTBC6bg= 2023-09-22T21:24:27.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BMtHanZIfqYqVtHJuV4JaTBC6bg=', 'Content-Length': '0'} 2023-09-22T21:24:27.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006467bbd61b1c3b7-00650e060b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.544 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:27.544 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:27.544 DEBUG:boto:path=/ffzdqu-53/?prefix=c 2023-09-22T21:24:27.544 DEBUG:boto:auth_path=/ffzdqu-53/?prefix=c 2023-09-22T21:24:27.544 DEBUG:boto:Method: GET 2023-09-22T21:24:27.544 DEBUG:boto:Path: /ffzdqu-53/?prefix=c 2023-09-22T21:24:27.544 DEBUG:boto:Data: 2023-09-22T21:24:27.544 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.545 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.545 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.545 DEBUG:boto:Params: {} 2023-09-22T21:24:27.545 DEBUG:boto:Token: None 2023-09-22T21:24:27.545 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/ 2023-09-22T21:24:27.545 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18= 2023-09-22T21:24:27.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18=', 'Content-Length': '0'} 2023-09-22T21:24:27.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016c1d6d6c4f87824-00650e060b-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.550 DEBUG:boto:b'ffzdqu-53c1000falsec2023-09-22T21:24:21.375Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:27.551 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:27.551 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:27.551 DEBUG:boto:path=/ffzdqu-53/?prefix=c 2023-09-22T21:24:27.551 DEBUG:boto:auth_path=/ffzdqu-53/?prefix=c 2023-09-22T21:24:27.551 DEBUG:boto:Method: GET 2023-09-22T21:24:27.551 DEBUG:boto:Path: /ffzdqu-53/?prefix=c 2023-09-22T21:24:27.551 DEBUG:boto:Data: 2023-09-22T21:24:27.551 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.552 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.552 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.552 DEBUG:boto:Params: {} 2023-09-22T21:24:27.552 DEBUG:boto:Token: None 2023-09-22T21:24:27.552 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/ 2023-09-22T21:24:27.552 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18= 2023-09-22T21:24:27.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18=', 'Content-Length': '0'} 2023-09-22T21:24:27.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083cc6d56350cef97-00650e060b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.557 DEBUG:boto:b'ffzdqu-53c1000falsec2023-09-22T21:24:21.375Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:27.557 DEBUG:boto:path=/ffzdqu-53/c 2023-09-22T21:24:27.557 DEBUG:boto:auth_path=/ffzdqu-53/c 2023-09-22T21:24:27.557 DEBUG:boto:Method: HEAD 2023-09-22T21:24:27.557 DEBUG:boto:Path: /ffzdqu-53/c 2023-09-22T21:24:27.558 DEBUG:boto:Data: 2023-09-22T21:24:27.558 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.558 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.558 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.558 DEBUG:boto:Params: {} 2023-09-22T21:24:27.558 DEBUG:boto:Token: None 2023-09-22T21:24:27.558 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/c 2023-09-22T21:24:27.558 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:a0fbeytUDdGDPFfZD0b4vUmNnLk= 2023-09-22T21:24:27.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:a0fbeytUDdGDPFfZD0b4vUmNnLk=', 'Content-Length': '0'} 2023-09-22T21:24:27.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005664c1da885437df-00650e060b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.561 DEBUG:boto:path=/ffzdqu-53/c 2023-09-22T21:24:27.561 DEBUG:boto:auth_path=/ffzdqu-53/c 2023-09-22T21:24:27.561 DEBUG:boto:Method: HEAD 2023-09-22T21:24:27.562 DEBUG:boto:Path: /ffzdqu-53/c 2023-09-22T21:24:27.562 DEBUG:boto:Data: 2023-09-22T21:24:27.562 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.562 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.562 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.562 DEBUG:boto:Params: {} 2023-09-22T21:24:27.562 DEBUG:boto:Token: None 2023-09-22T21:24:27.562 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/c 2023-09-22T21:24:27.563 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:a0fbeytUDdGDPFfZD0b4vUmNnLk= 2023-09-22T21:24:27.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:a0fbeytUDdGDPFfZD0b4vUmNnLk=', 'Content-Length': '0'} 2023-09-22T21:24:27.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000053b8b2bfb6e05bc-00650e060b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:24:27.566 DEBUG:boto:path=/ffzdqu-53/c 2023-09-22T21:24:27.566 DEBUG:boto:auth_path=/ffzdqu-53/c 2023-09-22T21:24:27.566 DEBUG:boto:Method: GET 2023-09-22T21:24:27.566 DEBUG:boto:Path: /ffzdqu-53/c 2023-09-22T21:24:27.567 DEBUG:boto:Data: 2023-09-22T21:24:27.567 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.567 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.567 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.567 DEBUG:boto:Params: {} 2023-09-22T21:24:27.567 DEBUG:boto:Token: None 2023-09-22T21:24:27.567 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/c 2023-09-22T21:24:27.567 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rAUt/wkp4EQRxVv/iAi9cAg1wTI= 2023-09-22T21:24:27.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rAUt/wkp4EQRxVv/iAi9cAg1wTI=', 'Content-Length': '0'} 2023-09-22T21:24:27.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000eb726cbf9cbebb6-00650e060b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.610 DEBUG:boto:path=/ffzdqu-53/c 2023-09-22T21:24:27.611 DEBUG:boto:auth_path=/ffzdqu-53/c 2023-09-22T21:24:27.611 DEBUG:boto:Method: GET 2023-09-22T21:24:27.611 DEBUG:boto:Path: /ffzdqu-53/c 2023-09-22T21:24:27.611 DEBUG:boto:Data: 2023-09-22T21:24:27.611 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.611 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.611 DEBUG:boto:Params: {} 2023-09-22T21:24:27.611 DEBUG:boto:Token: None 2023-09-22T21:24:27.612 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/c 2023-09-22T21:24:27.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rAUt/wkp4EQRxVv/iAi9cAg1wTI= 2023-09-22T21:24:27.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rAUt/wkp4EQRxVv/iAi9cAg1wTI=', 'Content-Length': '0'} 2023-09-22T21:24:27.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000131b3d8df65fca4b-00650e060b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.615 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:27.615 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:27.615 DEBUG:boto:path=/ffzdqu-53/?prefix=c 2023-09-22T21:24:27.615 DEBUG:boto:auth_path=/ffzdqu-53/?prefix=c 2023-09-22T21:24:27.615 DEBUG:boto:Method: GET 2023-09-22T21:24:27.616 DEBUG:boto:Path: /ffzdqu-53/?prefix=c 2023-09-22T21:24:27.616 DEBUG:boto:Data: 2023-09-22T21:24:27.616 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.616 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.616 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.616 DEBUG:boto:Params: {} 2023-09-22T21:24:27.616 DEBUG:boto:Token: None 2023-09-22T21:24:27.616 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/ 2023-09-22T21:24:27.616 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18= 2023-09-22T21:24:27.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18=', 'Content-Length': '0'} 2023-09-22T21:24:27.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd68bebed9e6b647-00650e060b-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.666 DEBUG:boto:b'ffzdqu-53c1000falsec2023-09-22T21:24:21.375Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:27.667 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:27.667 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:27.667 DEBUG:boto:path=/ffzdqu-53/?prefix=c 2023-09-22T21:24:27.667 DEBUG:boto:auth_path=/ffzdqu-53/?prefix=c 2023-09-22T21:24:27.667 DEBUG:boto:Method: GET 2023-09-22T21:24:27.668 DEBUG:boto:Path: /ffzdqu-53/?prefix=c 2023-09-22T21:24:27.668 DEBUG:boto:Data: 2023-09-22T21:24:27.668 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.668 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.668 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.668 DEBUG:boto:Params: {} 2023-09-22T21:24:27.668 DEBUG:boto:Token: None 2023-09-22T21:24:27.668 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/ 2023-09-22T21:24:27.668 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18= 2023-09-22T21:24:27.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18=', 'Content-Length': '0'} 2023-09-22T21:24:27.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b52b9671ac9899fa-00650e060b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.674 DEBUG:boto:b'ffzdqu-53c1000falsec2023-09-22T21:24:21.375Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:27.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:24:27.674 DEBUG:boto:path=/ffzdqu-53/d 2023-09-22T21:24:27.674 DEBUG:boto:auth_path=/ffzdqu-53/d 2023-09-22T21:24:27.674 DEBUG:boto:path=/ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.675 DEBUG:boto:auth_path=/ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.675 DEBUG:boto:Method: GET 2023-09-22T21:24:27.675 DEBUG:boto:Path: /ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.675 DEBUG:boto:Data: 2023-09-22T21:24:27.675 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.675 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.675 DEBUG:boto:Params: {} 2023-09-22T21:24:27.675 DEBUG:boto:Token: None 2023-09-22T21:24:27.676 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.676 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9vrkt1mbWNFlM2PvawptbbgbsWk= 2023-09-22T21:24:27.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9vrkt1mbWNFlM2PvawptbbgbsWk=', 'Content-Length': '0'} 2023-09-22T21:24:27.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cd1fc07a8ac060f-00650e060b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.679 DEBUG:boto:path=/ffzdqu-53/d 2023-09-22T21:24:27.679 DEBUG:boto:auth_path=/ffzdqu-53/d 2023-09-22T21:24:27.679 DEBUG:boto:path=/ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.679 DEBUG:boto:auth_path=/ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.679 DEBUG:boto:Method: GET 2023-09-22T21:24:27.679 DEBUG:boto:Path: /ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.679 DEBUG:boto:Data: 2023-09-22T21:24:27.679 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.680 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.680 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.680 DEBUG:boto:Params: {} 2023-09-22T21:24:27.680 DEBUG:boto:Token: None 2023-09-22T21:24:27.680 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.680 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9vrkt1mbWNFlM2PvawptbbgbsWk= 2023-09-22T21:24:27.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9vrkt1mbWNFlM2PvawptbbgbsWk=', 'Content-Length': '0'} 2023-09-22T21:24:27.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057247ea6dd5d7141-00650e060b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.683 DEBUG:boto:path=/ffzdqu-53/d 2023-09-22T21:24:27.683 DEBUG:boto:auth_path=/ffzdqu-53/d 2023-09-22T21:24:27.684 DEBUG:boto:path=/ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.684 DEBUG:boto:auth_path=/ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.684 DEBUG:boto:Method: HEAD 2023-09-22T21:24:27.684 DEBUG:boto:Path: /ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.684 DEBUG:boto:Data: 2023-09-22T21:24:27.684 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.684 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.684 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.684 DEBUG:boto:Params: {} 2023-09-22T21:24:27.684 DEBUG:boto:Token: None 2023-09-22T21:24:27.685 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.685 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qaC2IdhQZYFWIMMHzV/tpAwKMoc= 2023-09-22T21:24:27.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qaC2IdhQZYFWIMMHzV/tpAwKMoc=', 'Content-Length': '0'} 2023-09-22T21:24:27.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000437598c12d2a3694-00650e060b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.688 DEBUG:boto:path=/ffzdqu-53/d 2023-09-22T21:24:27.688 DEBUG:boto:auth_path=/ffzdqu-53/d 2023-09-22T21:24:27.688 DEBUG:boto:path=/ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.688 DEBUG:boto:auth_path=/ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.688 DEBUG:boto:Method: HEAD 2023-09-22T21:24:27.689 DEBUG:boto:Path: /ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.689 DEBUG:boto:Data: 2023-09-22T21:24:27.689 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.689 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.689 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.690 DEBUG:boto:Params: {} 2023-09-22T21:24:27.690 DEBUG:boto:Token: None 2023-09-22T21:24:27.691 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.691 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qaC2IdhQZYFWIMMHzV/tpAwKMoc= 2023-09-22T21:24:27.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qaC2IdhQZYFWIMMHzV/tpAwKMoc=', 'Content-Length': '0'} 2023-09-22T21:24:27.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3dd871eba31662e-00650e060b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:24:27.694 DEBUG:boto:path=/ffzdqu-53/d 2023-09-22T21:24:27.694 DEBUG:boto:auth_path=/ffzdqu-53/d 2023-09-22T21:24:27.694 DEBUG:boto:path=/ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.694 DEBUG:boto:auth_path=/ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.694 DEBUG:boto:Method: GET 2023-09-22T21:24:27.694 DEBUG:boto:Path: /ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.694 DEBUG:boto:Data: 2023-09-22T21:24:27.695 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.695 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.695 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.695 DEBUG:boto:Params: {} 2023-09-22T21:24:27.695 DEBUG:boto:Token: None 2023-09-22T21:24:27.695 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.695 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9vrkt1mbWNFlM2PvawptbbgbsWk= 2023-09-22T21:24:27.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9vrkt1mbWNFlM2PvawptbbgbsWk=', 'Content-Length': '0'} 2023-09-22T21:24:27.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090b77e303a9f69db-00650e060b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.738 DEBUG:boto:path=/ffzdqu-53/d 2023-09-22T21:24:27.739 DEBUG:boto:auth_path=/ffzdqu-53/d 2023-09-22T21:24:27.739 DEBUG:boto:path=/ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.739 DEBUG:boto:auth_path=/ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.739 DEBUG:boto:Method: GET 2023-09-22T21:24:27.739 DEBUG:boto:Path: /ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.739 DEBUG:boto:Data: 2023-09-22T21:24:27.739 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.739 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.739 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.740 DEBUG:boto:Params: {} 2023-09-22T21:24:27.740 DEBUG:boto:Token: None 2023-09-22T21:24:27.740 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/d?versionId=null 2023-09-22T21:24:27.740 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9vrkt1mbWNFlM2PvawptbbgbsWk= 2023-09-22T21:24:27.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9vrkt1mbWNFlM2PvawptbbgbsWk=', 'Content-Length': '0'} 2023-09-22T21:24:27.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000289ba34b618f4ed-00650e060b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.743 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:27.743 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:27.744 DEBUG:boto:path=/ffzdqu-53/?prefix=d 2023-09-22T21:24:27.744 DEBUG:boto:auth_path=/ffzdqu-53/?prefix=d 2023-09-22T21:24:27.744 DEBUG:boto:Method: GET 2023-09-22T21:24:27.744 DEBUG:boto:Path: /ffzdqu-53/?prefix=d 2023-09-22T21:24:27.744 DEBUG:boto:Data: 2023-09-22T21:24:27.744 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.744 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.744 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.744 DEBUG:boto:Params: {} 2023-09-22T21:24:27.745 DEBUG:boto:Token: None 2023-09-22T21:24:27.745 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/ 2023-09-22T21:24:27.745 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18= 2023-09-22T21:24:27.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18=', 'Content-Length': '0'} 2023-09-22T21:24:27.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6ef0bb9941c088a-00650e060b-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.750 DEBUG:boto:b'ffzdqu-53d1000falsed2023-09-22T21:24:21.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:27.751 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:27.751 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:27.751 DEBUG:boto:path=/ffzdqu-53/?prefix=d 2023-09-22T21:24:27.751 DEBUG:boto:auth_path=/ffzdqu-53/?prefix=d 2023-09-22T21:24:27.751 DEBUG:boto:Method: GET 2023-09-22T21:24:27.751 DEBUG:boto:Path: /ffzdqu-53/?prefix=d 2023-09-22T21:24:27.751 DEBUG:boto:Data: 2023-09-22T21:24:27.751 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.752 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.752 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.752 DEBUG:boto:Params: {} 2023-09-22T21:24:27.752 DEBUG:boto:Token: None 2023-09-22T21:24:27.752 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/ 2023-09-22T21:24:27.752 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18= 2023-09-22T21:24:27.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18=', 'Content-Length': '0'} 2023-09-22T21:24:27.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba3e4b32367f6053-00650e060b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.798 DEBUG:boto:b'ffzdqu-53d1000falsed2023-09-22T21:24:21.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:27.799 DEBUG:boto:path=/ffzdqu-53/d 2023-09-22T21:24:27.799 DEBUG:boto:auth_path=/ffzdqu-53/d 2023-09-22T21:24:27.799 DEBUG:boto:Method: HEAD 2023-09-22T21:24:27.799 DEBUG:boto:Path: /ffzdqu-53/d 2023-09-22T21:24:27.800 DEBUG:boto:Data: 2023-09-22T21:24:27.800 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.800 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.800 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.800 DEBUG:boto:Params: {} 2023-09-22T21:24:27.800 DEBUG:boto:Token: None 2023-09-22T21:24:27.800 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/d 2023-09-22T21:24:27.800 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HHSBQsF1ScC6uHCd+eKdsOyiL3A= 2023-09-22T21:24:27.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HHSBQsF1ScC6uHCd+eKdsOyiL3A=', 'Content-Length': '0'} 2023-09-22T21:24:27.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008934c06fa7bb0702-00650e060b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.806 DEBUG:boto:path=/ffzdqu-53/d 2023-09-22T21:24:27.806 DEBUG:boto:auth_path=/ffzdqu-53/d 2023-09-22T21:24:27.807 DEBUG:boto:Method: HEAD 2023-09-22T21:24:27.807 DEBUG:boto:Path: /ffzdqu-53/d 2023-09-22T21:24:27.807 DEBUG:boto:Data: 2023-09-22T21:24:27.807 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.807 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.807 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.807 DEBUG:boto:Params: {} 2023-09-22T21:24:27.807 DEBUG:boto:Token: None 2023-09-22T21:24:27.807 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/d 2023-09-22T21:24:27.807 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HHSBQsF1ScC6uHCd+eKdsOyiL3A= 2023-09-22T21:24:27.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HHSBQsF1ScC6uHCd+eKdsOyiL3A=', 'Content-Length': '0'} 2023-09-22T21:24:27.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4d7a7bfe131284f-00650e060b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:24:27.812 DEBUG:boto:path=/ffzdqu-53/d 2023-09-22T21:24:27.812 DEBUG:boto:auth_path=/ffzdqu-53/d 2023-09-22T21:24:27.812 DEBUG:boto:Method: GET 2023-09-22T21:24:27.812 DEBUG:boto:Path: /ffzdqu-53/d 2023-09-22T21:24:27.812 DEBUG:boto:Data: 2023-09-22T21:24:27.812 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.812 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.812 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.812 DEBUG:boto:Params: {} 2023-09-22T21:24:27.812 DEBUG:boto:Token: None 2023-09-22T21:24:27.813 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/d 2023-09-22T21:24:27.813 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6FZWkXVrbRwtn+pHwe2LfoMiGlU= 2023-09-22T21:24:27.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6FZWkXVrbRwtn+pHwe2LfoMiGlU=', 'Content-Length': '0'} 2023-09-22T21:24:27.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063c12a572e8270b6-00650e060b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.862 DEBUG:boto:path=/ffzdqu-53/d 2023-09-22T21:24:27.863 DEBUG:boto:auth_path=/ffzdqu-53/d 2023-09-22T21:24:27.863 DEBUG:boto:Method: GET 2023-09-22T21:24:27.863 DEBUG:boto:Path: /ffzdqu-53/d 2023-09-22T21:24:27.863 DEBUG:boto:Data: 2023-09-22T21:24:27.863 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.863 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.863 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.863 DEBUG:boto:Params: {} 2023-09-22T21:24:27.863 DEBUG:boto:Token: None 2023-09-22T21:24:27.864 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/d 2023-09-22T21:24:27.864 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6FZWkXVrbRwtn+pHwe2LfoMiGlU= 2023-09-22T21:24:27.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6FZWkXVrbRwtn+pHwe2LfoMiGlU=', 'Content-Length': '0'} 2023-09-22T21:24:27.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbc011c18e9df434-00650e060b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.869 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:27.869 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:27.869 DEBUG:boto:path=/ffzdqu-53/?prefix=d 2023-09-22T21:24:27.869 DEBUG:boto:auth_path=/ffzdqu-53/?prefix=d 2023-09-22T21:24:27.869 DEBUG:boto:Method: GET 2023-09-22T21:24:27.869 DEBUG:boto:Path: /ffzdqu-53/?prefix=d 2023-09-22T21:24:27.869 DEBUG:boto:Data: 2023-09-22T21:24:27.869 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.869 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:27.869 DEBUG:boto:Port: 8001 2023-09-22T21:24:27.870 DEBUG:boto:Params: {} 2023-09-22T21:24:27.870 DEBUG:boto:Token: None 2023-09-22T21:24:27.870 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/ 2023-09-22T21:24:27.870 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18= 2023-09-22T21:24:27.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18=', 'Content-Length': '0'} 2023-09-22T21:24:27.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000348dae77313c4e3f-00650e060b-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.875 DEBUG:boto:b'ffzdqu-53d1000falsed2023-09-22T21:24:21.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:27.876 DEBUG:boto:path=/ffzdqu-53/ 2023-09-22T21:24:27.876 DEBUG:boto:auth_path=/ffzdqu-53/ 2023-09-22T21:24:27.876 DEBUG:boto:path=/ffzdqu-53/?prefix=d 2023-09-22T21:24:27.876 DEBUG:boto:auth_path=/ffzdqu-53/?prefix=d 2023-09-22T21:24:27.876 DEBUG:boto:Method: GET 2023-09-22T21:24:27.876 DEBUG:boto:Path: /ffzdqu-53/?prefix=d 2023-09-22T21:24:27.876 DEBUG:boto:Data: 2023-09-22T21:24:27.876 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.877 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.877 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.877 DEBUG:boto:Params: {} 2023-09-22T21:24:27.877 DEBUG:boto:Token: None 2023-09-22T21:24:27.877 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-53/ 2023-09-22T21:24:27.877 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18= 2023-09-22T21:24:27.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IatmGJddbKJ9B3tRniJL0TQ6j18=', 'Content-Length': '0'} 2023-09-22T21:24:27.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029d996d1f113bced-00650e060b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:27.882 DEBUG:boto:b'ffzdqu-53d1000falsed2023-09-22T21:24:21.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:27.883 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-53 zones={a2, a1} 2023-09-22T21:24:27.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:24:27.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:24:27.883 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:24:27.923 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:24:27.923 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:24:27.981 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-54 2023-09-22T21:24:27.982 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:27.982 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:27.982 DEBUG:boto:Method: PUT 2023-09-22T21:24:27.982 DEBUG:boto:Path: /ffzdqu-54/ 2023-09-22T21:24:27.982 DEBUG:boto:Data: 2023-09-22T21:24:27.982 DEBUG:boto:Headers: {} 2023-09-22T21:24:27.982 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:27.983 DEBUG:boto:Port: 8000 2023-09-22T21:24:27.983 DEBUG:boto:Params: {} 2023-09-22T21:24:27.983 DEBUG:boto:Token: None 2023-09-22T21:24:27.983 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:24:27 GMT /ffzdqu-54/ 2023-09-22T21:24:27.983 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mgzXyxe+cEsAskceIlg5kz6pVXE= 2023-09-22T21:24:27.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mgzXyxe+cEsAskceIlg5kz6pVXE=', 'Content-Length': '0'} 2023-09-22T21:24:28.002 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004a211fa8e0c7f47c-00650e060b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:28.002 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:28.003 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:28.003 DEBUG:boto:Method: HEAD 2023-09-22T21:24:28.003 DEBUG:boto:Path: /ffzdqu-54/ 2023-09-22T21:24:28.003 DEBUG:boto:Data: 2023-09-22T21:24:28.003 DEBUG:boto:Headers: {} 2023-09-22T21:24:28.003 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:28.003 DEBUG:boto:Port: 8000 2023-09-22T21:24:28.003 DEBUG:boto:Params: {} 2023-09-22T21:24:28.003 DEBUG:boto:Token: None 2023-09-22T21:24:28.004 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:28 GMT /ffzdqu-54/ 2023-09-22T21:24:28.004 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YZDWnAXkkti3FzjmVEgLzCVri8U= 2023-09-22T21:24:28.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:28 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YZDWnAXkkti3FzjmVEgLzCVri8U=', 'Content-Length': '0'} 2023-09-22T21:24:28.006 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000187d4f0f33ae23c0-00650e060c-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:28.006 DEBUG:boto:path=/ffzdqu-54/a 2023-09-22T21:24:28.006 DEBUG:boto:auth_path=/ffzdqu-54/a 2023-09-22T21:24:28.006 DEBUG:boto:Method: PUT 2023-09-22T21:24:28.006 DEBUG:boto:Path: /ffzdqu-54/a 2023-09-22T21:24:28.006 DEBUG:boto:Data: 2023-09-22T21:24:28.006 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:24:28.007 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:28.007 DEBUG:boto:Port: 8000 2023-09-22T21:24:28.007 DEBUG:boto:Params: {} 2023-09-22T21:24:28.007 DEBUG:boto:Token: None 2023-09-22T21:24:28.007 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:24:28 GMT /ffzdqu-54/a 2023-09-22T21:24:28.007 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GalMs17T8wyn2e0MCIgTj9GTPKM= 2023-09-22T21:24:28.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:24:28 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GalMs17T8wyn2e0MCIgTj9GTPKM='} 2023-09-22T21:24:28.051 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001645171bcace262-00650e060c-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:28.051 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:28.051 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:28.051 DEBUG:boto:Method: HEAD 2023-09-22T21:24:28.051 DEBUG:boto:Path: /ffzdqu-54/ 2023-09-22T21:24:28.051 DEBUG:boto:Data: 2023-09-22T21:24:28.051 DEBUG:boto:Headers: {} 2023-09-22T21:24:28.052 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:28.052 DEBUG:boto:Port: 8000 2023-09-22T21:24:28.052 DEBUG:boto:Params: {} 2023-09-22T21:24:28.052 DEBUG:boto:Token: None 2023-09-22T21:24:28.052 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:28 GMT /ffzdqu-54/ 2023-09-22T21:24:28.052 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YZDWnAXkkti3FzjmVEgLzCVri8U= 2023-09-22T21:24:28.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:28 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YZDWnAXkkti3FzjmVEgLzCVri8U=', 'Content-Length': '0'} 2023-09-22T21:24:28.054 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b0b5ba0812df80f-00650e060c-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:28.055 DEBUG:boto:path=/ffzdqu-54/b 2023-09-22T21:24:28.055 DEBUG:boto:auth_path=/ffzdqu-54/b 2023-09-22T21:24:28.055 DEBUG:boto:Method: PUT 2023-09-22T21:24:28.055 DEBUG:boto:Path: /ffzdqu-54/b 2023-09-22T21:24:28.055 DEBUG:boto:Data: 2023-09-22T21:24:28.055 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:24:28.055 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:28.056 DEBUG:boto:Port: 8000 2023-09-22T21:24:28.056 DEBUG:boto:Params: {} 2023-09-22T21:24:28.056 DEBUG:boto:Token: None 2023-09-22T21:24:28.056 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:24:28 GMT /ffzdqu-54/b 2023-09-22T21:24:28.056 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RaMjxSSzPsYJ4d8Yi7TUgjcEmzM= 2023-09-22T21:24:28.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:24:28 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RaMjxSSzPsYJ4d8Yi7TUgjcEmzM='} 2023-09-22T21:24:28.062 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2615af341b156fd-00650e060c-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:28.063 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:28.063 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:28.063 DEBUG:boto:Method: HEAD 2023-09-22T21:24:28.063 DEBUG:boto:Path: /ffzdqu-54/ 2023-09-22T21:24:28.063 DEBUG:boto:Data: 2023-09-22T21:24:28.063 DEBUG:boto:Headers: {} 2023-09-22T21:24:28.063 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:28.063 DEBUG:boto:Port: 8000 2023-09-22T21:24:28.063 DEBUG:boto:Params: {} 2023-09-22T21:24:28.063 DEBUG:boto:Token: None 2023-09-22T21:24:28.064 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:28 GMT /ffzdqu-54/ 2023-09-22T21:24:28.064 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YZDWnAXkkti3FzjmVEgLzCVri8U= 2023-09-22T21:24:28.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:28 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YZDWnAXkkti3FzjmVEgLzCVri8U=', 'Content-Length': '0'} 2023-09-22T21:24:28.067 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2e43c6160e1cb4f-00650e060c-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:28.067 DEBUG:boto:path=/ffzdqu-54/c 2023-09-22T21:24:28.068 DEBUG:boto:auth_path=/ffzdqu-54/c 2023-09-22T21:24:28.068 DEBUG:boto:Method: PUT 2023-09-22T21:24:28.068 DEBUG:boto:Path: /ffzdqu-54/c 2023-09-22T21:24:28.068 DEBUG:boto:Data: 2023-09-22T21:24:28.068 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:24:28.068 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:28.068 DEBUG:boto:Port: 8000 2023-09-22T21:24:28.068 DEBUG:boto:Params: {} 2023-09-22T21:24:28.068 DEBUG:boto:Token: None 2023-09-22T21:24:28.069 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:24:28 GMT /ffzdqu-54/c 2023-09-22T21:24:28.069 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wmanTUiR7WIh/QwnZe/t3+UhAi4= 2023-09-22T21:24:28.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:24:28 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wmanTUiR7WIh/QwnZe/t3+UhAi4='} 2023-09-22T21:24:28.114 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cdefbd1435114781-00650e060c-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:28.115 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:28.115 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:28.115 DEBUG:boto:Method: HEAD 2023-09-22T21:24:28.115 DEBUG:boto:Path: /ffzdqu-54/ 2023-09-22T21:24:28.115 DEBUG:boto:Data: 2023-09-22T21:24:28.115 DEBUG:boto:Headers: {} 2023-09-22T21:24:28.115 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:28.115 DEBUG:boto:Port: 8000 2023-09-22T21:24:28.116 DEBUG:boto:Params: {} 2023-09-22T21:24:28.116 DEBUG:boto:Token: None 2023-09-22T21:24:28.116 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:28 GMT /ffzdqu-54/ 2023-09-22T21:24:28.116 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YZDWnAXkkti3FzjmVEgLzCVri8U= 2023-09-22T21:24:28.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:28 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YZDWnAXkkti3FzjmVEgLzCVri8U=', 'Content-Length': '0'} 2023-09-22T21:24:28.119 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005372f2a2f164e84f-00650e060c-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:28.120 DEBUG:boto:path=/ffzdqu-54/d 2023-09-22T21:24:28.120 DEBUG:boto:auth_path=/ffzdqu-54/d 2023-09-22T21:24:28.120 DEBUG:boto:Method: PUT 2023-09-22T21:24:28.120 DEBUG:boto:Path: /ffzdqu-54/d 2023-09-22T21:24:28.120 DEBUG:boto:Data: 2023-09-22T21:24:28.120 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:24:28.120 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:28.121 DEBUG:boto:Port: 8000 2023-09-22T21:24:28.121 DEBUG:boto:Params: {} 2023-09-22T21:24:28.121 DEBUG:boto:Token: None 2023-09-22T21:24:28.121 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:24:28 GMT /ffzdqu-54/d 2023-09-22T21:24:28.121 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kqSH7IqJBEdD9dm3DcdCcb5nJjg= 2023-09-22T21:24:28.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:24:28 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kqSH7IqJBEdD9dm3DcdCcb5nJjg='} 2023-09-22T21:24:28.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000012b29df74e77a4df-00650e060c-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:28.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:28.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:28.135 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:24:28.179 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:24:28.179 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:24:28.222 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:24:28.240 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695417868.003203_404.1","last_update":"2023-09-22T21:24:28.003203Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:24:28.240 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417868.003203_404.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:24:28.240 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:24:28.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:28.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:28.241 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:24:28.274 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:24:28.275 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:24:28.310 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:28.307+0000 7fcb1ae1ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:24:28.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:28.311+0000 7fcb1ae1ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:24:28.328 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417868.003203_404.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:24:28.003203Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:24:28Z"} 2023-09-22T21:24:28.329 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417868.003203_404.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:24:28.003203Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:24:28Z"} 2023-09-22T21:24:28.329 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417868.003203_404.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:24:28.003203Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:24:28.329 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417868.003203_404.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:24:28.329 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417868.003203_404.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:24:28.330 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:24:28.330 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-54 2023-09-22T21:24:28.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:24:28.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:24:28.330 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-54 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:24:28.417 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:24:28.417 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:24:28.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:28.459+0000 7f63664b8a40 1 waiting to reach incremental sync.. 2023-09-22T21:24:33.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:33.459+0000 7f63664b8a40 1 waiting to reach incremental sync.. 2023-09-22T21:24:38.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:38.459+0000 7f63664b8a40 1 waiting to reach incremental sync.. 2023-09-22T21:24:43.466 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:43.463+0000 7f63664b8a40 1 bucket sync caught up with source: 2023-09-22T21:24:43.466 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.19.6, , , , 00000000001.28.6, , , 00000000001.32.6, , 00000000001.21.6] 2023-09-22T21:24:43.466 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.19.6, , , , 00000000001.28.6, , , 00000000001.32.6, , 00000000001.21.6] 2023-09-22T21:24:43.466 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:43.463+0000 7f63664b8a40 0 bucket checkpoint complete 2023-09-22T21:24:43.514 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-54 2023-09-22T21:24:43.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-54', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:24:43.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-54', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:24:43.514 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-54 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:24:43.601 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:24:43.601 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:24:43.647 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:43.647+0000 7f02e2c82a40 1 bucket sync caught up with source: 2023-09-22T21:24:43.647 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.150.6, , , , 00000000001.149.6, , , 00000000001.147.6, , 00000000001.96.6] 2023-09-22T21:24:43.647 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.150.6, , , , 00000000001.149.6, , , 00000000001.147.6, , 00000000001.96.6] 2023-09-22T21:24:43.647 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:43.647+0000 7f02e2c82a40 0 bucket checkpoint complete 2023-09-22T21:24:43.664 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-54 zones={a2, a1} 2023-09-22T21:24:43.664 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:43.664 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:43.665 DEBUG:boto:Method: HEAD 2023-09-22T21:24:43.665 DEBUG:boto:Path: /ffzdqu-54/ 2023-09-22T21:24:43.665 DEBUG:boto:Data: 2023-09-22T21:24:43.665 DEBUG:boto:Headers: {} 2023-09-22T21:24:43.665 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:43.665 DEBUG:boto:Port: 8001 2023-09-22T21:24:43.665 DEBUG:boto:Params: {} 2023-09-22T21:24:43.665 DEBUG:boto:Token: None 2023-09-22T21:24:43.665 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:43 GMT /ffzdqu-54/ 2023-09-22T21:24:43.666 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DNw6pHWC80bC3GWb9lqn6r5p8tw= 2023-09-22T21:24:43.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DNw6pHWC80bC3GWb9lqn6r5p8tw=', 'Content-Length': '0'} 2023-09-22T21:24:43.668 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000337d2798ed717556-00650e061b-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:43.668 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:43.668 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:43.668 DEBUG:boto:Method: HEAD 2023-09-22T21:24:43.668 DEBUG:boto:Path: /ffzdqu-54/ 2023-09-22T21:24:43.669 DEBUG:boto:Data: 2023-09-22T21:24:43.669 DEBUG:boto:Headers: {} 2023-09-22T21:24:43.669 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:43.669 DEBUG:boto:Port: 8000 2023-09-22T21:24:43.669 DEBUG:boto:Params: {} 2023-09-22T21:24:43.669 DEBUG:boto:Token: None 2023-09-22T21:24:43.669 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:43 GMT /ffzdqu-54/ 2023-09-22T21:24:43.669 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DNw6pHWC80bC3GWb9lqn6r5p8tw= 2023-09-22T21:24:43.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DNw6pHWC80bC3GWb9lqn6r5p8tw=', 'Content-Length': '0'} 2023-09-22T21:24:43.672 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029504cd30886dbaa-00650e061b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:43.672 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:24:43.672 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:43.672 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:43.672 DEBUG:boto:path=/ffzdqu-54/?versions&max-keys=999 2023-09-22T21:24:43.672 DEBUG:boto:auth_path=/ffzdqu-54/?versions&max-keys=999 2023-09-22T21:24:43.673 DEBUG:boto:Method: GET 2023-09-22T21:24:43.673 DEBUG:boto:Path: /ffzdqu-54/?versions&max-keys=999 2023-09-22T21:24:43.673 DEBUG:boto:Data: 2023-09-22T21:24:43.673 DEBUG:boto:Headers: {} 2023-09-22T21:24:43.673 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:43.673 DEBUG:boto:Port: 8001 2023-09-22T21:24:43.673 DEBUG:boto:Params: {} 2023-09-22T21:24:43.673 DEBUG:boto:Token: None 2023-09-22T21:24:43.673 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:43 GMT /ffzdqu-54/?versions 2023-09-22T21:24:43.674 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5OVZaApihO16h1L4igj4oTdhBNw= 2023-09-22T21:24:43.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5OVZaApihO16h1L4igj4oTdhBNw=', 'Content-Length': '0'} 2023-09-22T21:24:43.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e096c2d0114878a1-00650e061b-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:43.676 DEBUG:boto:b'ffzdqu-54999falseanulltrue2023-09-22T21:24:28.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:28.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:28.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:28.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:43.676 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-22T21:24:43.676 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-22T21:24:43.677 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-22T21:24:43.677 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-22T21:24:43.677 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:24:43.677 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:43.677 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:43.677 DEBUG:boto:path=/ffzdqu-54/?versions&max-keys=999 2023-09-22T21:24:43.677 DEBUG:boto:auth_path=/ffzdqu-54/?versions&max-keys=999 2023-09-22T21:24:43.677 DEBUG:boto:Method: GET 2023-09-22T21:24:43.677 DEBUG:boto:Path: /ffzdqu-54/?versions&max-keys=999 2023-09-22T21:24:43.678 DEBUG:boto:Data: 2023-09-22T21:24:43.678 DEBUG:boto:Headers: {} 2023-09-22T21:24:43.678 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:43.678 DEBUG:boto:Port: 8000 2023-09-22T21:24:43.678 DEBUG:boto:Params: {} 2023-09-22T21:24:43.678 DEBUG:boto:Token: None 2023-09-22T21:24:43.678 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:43 GMT /ffzdqu-54/?versions 2023-09-22T21:24:43.678 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5OVZaApihO16h1L4igj4oTdhBNw= 2023-09-22T21:24:43.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5OVZaApihO16h1L4igj4oTdhBNw=', 'Content-Length': '0'} 2023-09-22T21:24:43.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0285d3793611a6c-00650e061b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:43.681 DEBUG:boto:b'ffzdqu-54999falseanulltrue2023-09-22T21:24:28.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:28.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:28.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:28.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:43.681 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-22T21:24:43.681 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-22T21:24:43.681 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-22T21:24:43.681 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-22T21:24:43.682 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:43.682 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:43.682 DEBUG:boto:path=/ffzdqu-54/?versions&max-keys=999 2023-09-22T21:24:43.682 DEBUG:boto:auth_path=/ffzdqu-54/?versions&max-keys=999 2023-09-22T21:24:43.682 DEBUG:boto:Method: GET 2023-09-22T21:24:43.682 DEBUG:boto:Path: /ffzdqu-54/?versions&max-keys=999 2023-09-22T21:24:43.682 DEBUG:boto:Data: 2023-09-22T21:24:43.682 DEBUG:boto:Headers: {} 2023-09-22T21:24:43.682 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:43.682 DEBUG:boto:Port: 8001 2023-09-22T21:24:43.683 DEBUG:boto:Params: {} 2023-09-22T21:24:43.683 DEBUG:boto:Token: None 2023-09-22T21:24:43.683 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:43 GMT /ffzdqu-54/?versions 2023-09-22T21:24:43.683 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5OVZaApihO16h1L4igj4oTdhBNw= 2023-09-22T21:24:43.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5OVZaApihO16h1L4igj4oTdhBNw=', 'Content-Length': '0'} 2023-09-22T21:24:43.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faccd6abd113e51c-00650e061b-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:43.685 DEBUG:boto:b'ffzdqu-54999falseanulltrue2023-09-22T21:24:28.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:28.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:28.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:28.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:43.686 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:43.686 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:43.686 DEBUG:boto:path=/ffzdqu-54/?versions&max-keys=999 2023-09-22T21:24:43.686 DEBUG:boto:auth_path=/ffzdqu-54/?versions&max-keys=999 2023-09-22T21:24:43.686 DEBUG:boto:Method: GET 2023-09-22T21:24:43.687 DEBUG:boto:Path: /ffzdqu-54/?versions&max-keys=999 2023-09-22T21:24:43.687 DEBUG:boto:Data: 2023-09-22T21:24:43.687 DEBUG:boto:Headers: {} 2023-09-22T21:24:43.687 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:43.687 DEBUG:boto:Port: 8000 2023-09-22T21:24:43.687 DEBUG:boto:Params: {} 2023-09-22T21:24:43.687 DEBUG:boto:Token: None 2023-09-22T21:24:43.687 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:43 GMT /ffzdqu-54/?versions 2023-09-22T21:24:43.687 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5OVZaApihO16h1L4igj4oTdhBNw= 2023-09-22T21:24:43.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5OVZaApihO16h1L4igj4oTdhBNw=', 'Content-Length': '0'} 2023-09-22T21:24:43.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca3f1495bdc7d3a7-00650e061b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:43.690 DEBUG:boto:b'ffzdqu-54999falseanulltrue2023-09-22T21:24:28.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:28.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:28.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:28.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:43.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:24:43.691 DEBUG:boto:path=/ffzdqu-54/a 2023-09-22T21:24:43.691 DEBUG:boto:auth_path=/ffzdqu-54/a 2023-09-22T21:24:43.691 DEBUG:boto:path=/ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.691 DEBUG:boto:auth_path=/ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.691 DEBUG:boto:Method: GET 2023-09-22T21:24:43.691 DEBUG:boto:Path: /ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.693 DEBUG:boto:Data: 2023-09-22T21:24:43.693 DEBUG:boto:Headers: {} 2023-09-22T21:24:43.693 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:43.693 DEBUG:boto:Port: 8001 2023-09-22T21:24:43.693 DEBUG:boto:Params: {} 2023-09-22T21:24:43.693 DEBUG:boto:Token: None 2023-09-22T21:24:43.693 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:43 GMT /ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.694 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VLBrbi3jYRy5fzTjP/uNxc350Qo= 2023-09-22T21:24:43.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VLBrbi3jYRy5fzTjP/uNxc350Qo=', 'Content-Length': '0'} 2023-09-22T21:24:43.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-54:f132bdb9-20a7-4073-a932-8f99da20691e.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008af96ff6d067dad8-00650e061b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:43.695 DEBUG:boto:path=/ffzdqu-54/a 2023-09-22T21:24:43.696 DEBUG:boto:auth_path=/ffzdqu-54/a 2023-09-22T21:24:43.696 DEBUG:boto:path=/ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.696 DEBUG:boto:auth_path=/ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.696 DEBUG:boto:Method: GET 2023-09-22T21:24:43.696 DEBUG:boto:Path: /ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.696 DEBUG:boto:Data: 2023-09-22T21:24:43.696 DEBUG:boto:Headers: {} 2023-09-22T21:24:43.696 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:43.696 DEBUG:boto:Port: 8000 2023-09-22T21:24:43.696 DEBUG:boto:Params: {} 2023-09-22T21:24:43.697 DEBUG:boto:Token: None 2023-09-22T21:24:43.697 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:43 GMT /ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.697 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VLBrbi3jYRy5fzTjP/uNxc350Qo= 2023-09-22T21:24:43.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VLBrbi3jYRy5fzTjP/uNxc350Qo=', 'Content-Length': '0'} 2023-09-22T21:24:43.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bfb86a3cc9b35c2-00650e061b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:43.742 DEBUG:boto:path=/ffzdqu-54/a 2023-09-22T21:24:43.878 DEBUG:boto:auth_path=/ffzdqu-54/a 2023-09-22T21:24:43.878 DEBUG:boto:path=/ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.878 DEBUG:boto:auth_path=/ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.878 DEBUG:boto:Method: HEAD 2023-09-22T21:24:43.878 DEBUG:boto:Path: /ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.878 DEBUG:boto:Data: 2023-09-22T21:24:43.879 DEBUG:boto:Headers: {} 2023-09-22T21:24:43.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:43.879 DEBUG:boto:Port: 8001 2023-09-22T21:24:43.879 DEBUG:boto:Params: {} 2023-09-22T21:24:43.879 DEBUG:boto:Token: None 2023-09-22T21:24:43.879 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:43 GMT /ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.879 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xlC5kwNFEWpETIX3eICWvaHdVEU= 2023-09-22T21:24:43.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xlC5kwNFEWpETIX3eICWvaHdVEU=', 'Content-Length': '0'} 2023-09-22T21:24:43.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-54:f132bdb9-20a7-4073-a932-8f99da20691e.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad6972cf963e1d8b-00650e061b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:43.883 DEBUG:boto:path=/ffzdqu-54/a 2023-09-22T21:24:43.883 DEBUG:boto:auth_path=/ffzdqu-54/a 2023-09-22T21:24:43.883 DEBUG:boto:path=/ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.883 DEBUG:boto:auth_path=/ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.883 DEBUG:boto:Method: HEAD 2023-09-22T21:24:43.883 DEBUG:boto:Path: /ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.883 DEBUG:boto:Data: 2023-09-22T21:24:43.883 DEBUG:boto:Headers: {} 2023-09-22T21:24:43.883 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:43.884 DEBUG:boto:Port: 8000 2023-09-22T21:24:43.884 DEBUG:boto:Params: {} 2023-09-22T21:24:43.884 DEBUG:boto:Token: None 2023-09-22T21:24:43.884 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:43 GMT /ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.884 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xlC5kwNFEWpETIX3eICWvaHdVEU= 2023-09-22T21:24:43.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xlC5kwNFEWpETIX3eICWvaHdVEU=', 'Content-Length': '0'} 2023-09-22T21:24:43.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec9613a12f70d459-00650e061b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:43.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:24:43.887 DEBUG:boto:path=/ffzdqu-54/a 2023-09-22T21:24:43.887 DEBUG:boto:auth_path=/ffzdqu-54/a 2023-09-22T21:24:43.887 DEBUG:boto:path=/ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.887 DEBUG:boto:auth_path=/ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.888 DEBUG:boto:Method: GET 2023-09-22T21:24:43.888 DEBUG:boto:Path: /ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.888 DEBUG:boto:Data: 2023-09-22T21:24:43.888 DEBUG:boto:Headers: {} 2023-09-22T21:24:43.888 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:43.888 DEBUG:boto:Port: 8001 2023-09-22T21:24:43.888 DEBUG:boto:Params: {} 2023-09-22T21:24:43.888 DEBUG:boto:Token: None 2023-09-22T21:24:43.888 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:43 GMT /ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.889 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VLBrbi3jYRy5fzTjP/uNxc350Qo= 2023-09-22T21:24:43.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VLBrbi3jYRy5fzTjP/uNxc350Qo=', 'Content-Length': '0'} 2023-09-22T21:24:43.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-54:f132bdb9-20a7-4073-a932-8f99da20691e.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054419a2c6cdcb362-00650e061b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:43.892 DEBUG:boto:path=/ffzdqu-54/a 2023-09-22T21:24:43.892 DEBUG:boto:auth_path=/ffzdqu-54/a 2023-09-22T21:24:43.892 DEBUG:boto:path=/ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.892 DEBUG:boto:auth_path=/ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.892 DEBUG:boto:Method: GET 2023-09-22T21:24:43.892 DEBUG:boto:Path: /ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.892 DEBUG:boto:Data: 2023-09-22T21:24:43.892 DEBUG:boto:Headers: {} 2023-09-22T21:24:43.892 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:43.892 DEBUG:boto:Port: 8000 2023-09-22T21:24:43.893 DEBUG:boto:Params: {} 2023-09-22T21:24:43.893 DEBUG:boto:Token: None 2023-09-22T21:24:43.893 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:43 GMT /ffzdqu-54/a?versionId=null 2023-09-22T21:24:43.893 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VLBrbi3jYRy5fzTjP/uNxc350Qo= 2023-09-22T21:24:43.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VLBrbi3jYRy5fzTjP/uNxc350Qo=', 'Content-Length': '0'} 2023-09-22T21:24:43.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e6f4c97a06007ef-00650e061b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:43.898 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:43.898 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:43.898 DEBUG:boto:path=/ffzdqu-54/?prefix=a 2023-09-22T21:24:43.899 DEBUG:boto:auth_path=/ffzdqu-54/?prefix=a 2023-09-22T21:24:43.899 DEBUG:boto:Method: GET 2023-09-22T21:24:43.899 DEBUG:boto:Path: /ffzdqu-54/?prefix=a 2023-09-22T21:24:43.899 DEBUG:boto:Data: 2023-09-22T21:24:43.899 DEBUG:boto:Headers: {} 2023-09-22T21:24:43.899 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:43.899 DEBUG:boto:Port: 8001 2023-09-22T21:24:43.899 DEBUG:boto:Params: {} 2023-09-22T21:24:43.899 DEBUG:boto:Token: None 2023-09-22T21:24:43.900 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:43 GMT /ffzdqu-54/ 2023-09-22T21:24:43.900 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P9wVgMorVur+j1lOX6BPd4G2VQ4= 2023-09-22T21:24:43.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P9wVgMorVur+j1lOX6BPd4G2VQ4=', 'Content-Length': '0'} 2023-09-22T21:24:43.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007488711781517711-00650e061b-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:43.984 DEBUG:boto:b'ffzdqu-54a1000falsea2023-09-22T21:24:28.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:43.984 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:43.984 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:43.985 DEBUG:boto:path=/ffzdqu-54/?prefix=a 2023-09-22T21:24:43.985 DEBUG:boto:auth_path=/ffzdqu-54/?prefix=a 2023-09-22T21:24:43.985 DEBUG:boto:Method: GET 2023-09-22T21:24:43.985 DEBUG:boto:Path: /ffzdqu-54/?prefix=a 2023-09-22T21:24:43.985 DEBUG:boto:Data: 2023-09-22T21:24:43.985 DEBUG:boto:Headers: {} 2023-09-22T21:24:43.985 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:43.985 DEBUG:boto:Port: 8000 2023-09-22T21:24:43.985 DEBUG:boto:Params: {} 2023-09-22T21:24:43.986 DEBUG:boto:Token: None 2023-09-22T21:24:43.986 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:43 GMT /ffzdqu-54/ 2023-09-22T21:24:43.986 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P9wVgMorVur+j1lOX6BPd4G2VQ4= 2023-09-22T21:24:43.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P9wVgMorVur+j1lOX6BPd4G2VQ4=', 'Content-Length': '0'} 2023-09-22T21:24:43.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000912b5e70a7df3e77-00650e061b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:43.991 DEBUG:boto:b'ffzdqu-54a1000falsea2023-09-22T21:24:28.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:43.991 DEBUG:boto:path=/ffzdqu-54/a 2023-09-22T21:24:43.991 DEBUG:boto:auth_path=/ffzdqu-54/a 2023-09-22T21:24:43.991 DEBUG:boto:Method: HEAD 2023-09-22T21:24:43.991 DEBUG:boto:Path: /ffzdqu-54/a 2023-09-22T21:24:43.991 DEBUG:boto:Data: 2023-09-22T21:24:43.991 DEBUG:boto:Headers: {} 2023-09-22T21:24:43.991 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:43.992 DEBUG:boto:Port: 8001 2023-09-22T21:24:43.992 DEBUG:boto:Params: {} 2023-09-22T21:24:43.992 DEBUG:boto:Token: None 2023-09-22T21:24:43.992 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:43 GMT /ffzdqu-54/a 2023-09-22T21:24:43.992 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RLToG7emdNXEcL1+P/9iL+x/8ng= 2023-09-22T21:24:43.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RLToG7emdNXEcL1+P/9iL+x/8ng=', 'Content-Length': '0'} 2023-09-22T21:24:43.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-54:f132bdb9-20a7-4073-a932-8f99da20691e.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045bf67fb9b8b0f56-00650e061b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:43.995 DEBUG:boto:path=/ffzdqu-54/a 2023-09-22T21:24:43.995 DEBUG:boto:auth_path=/ffzdqu-54/a 2023-09-22T21:24:43.995 DEBUG:boto:Method: HEAD 2023-09-22T21:24:43.995 DEBUG:boto:Path: /ffzdqu-54/a 2023-09-22T21:24:43.995 DEBUG:boto:Data: 2023-09-22T21:24:43.995 DEBUG:boto:Headers: {} 2023-09-22T21:24:43.996 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:43.996 DEBUG:boto:Port: 8000 2023-09-22T21:24:43.996 DEBUG:boto:Params: {} 2023-09-22T21:24:43.996 DEBUG:boto:Token: None 2023-09-22T21:24:43.996 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:43 GMT /ffzdqu-54/a 2023-09-22T21:24:43.996 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RLToG7emdNXEcL1+P/9iL+x/8ng= 2023-09-22T21:24:43.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RLToG7emdNXEcL1+P/9iL+x/8ng=', 'Content-Length': '0'} 2023-09-22T21:24:43.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dc2d46eed06e752-00650e061b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:43.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:24:44.000 DEBUG:boto:path=/ffzdqu-54/a 2023-09-22T21:24:44.000 DEBUG:boto:auth_path=/ffzdqu-54/a 2023-09-22T21:24:44.000 DEBUG:boto:Method: GET 2023-09-22T21:24:44.000 DEBUG:boto:Path: /ffzdqu-54/a 2023-09-22T21:24:44.000 DEBUG:boto:Data: 2023-09-22T21:24:44.000 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.000 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.000 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.000 DEBUG:boto:Params: {} 2023-09-22T21:24:44.000 DEBUG:boto:Token: None 2023-09-22T21:24:44.001 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/a 2023-09-22T21:24:44.001 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qmlJC58nxPMpK1pOKnvcaYoHK14= 2023-09-22T21:24:44.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qmlJC58nxPMpK1pOKnvcaYoHK14=', 'Content-Length': '0'} 2023-09-22T21:24:44.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-54:f132bdb9-20a7-4073-a932-8f99da20691e.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001b624db3382fbb5-00650e061b-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.046 DEBUG:boto:path=/ffzdqu-54/a 2023-09-22T21:24:44.046 DEBUG:boto:auth_path=/ffzdqu-54/a 2023-09-22T21:24:44.046 DEBUG:boto:Method: GET 2023-09-22T21:24:44.047 DEBUG:boto:Path: /ffzdqu-54/a 2023-09-22T21:24:44.047 DEBUG:boto:Data: 2023-09-22T21:24:44.047 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.047 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.047 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.047 DEBUG:boto:Params: {} 2023-09-22T21:24:44.047 DEBUG:boto:Token: None 2023-09-22T21:24:44.047 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/a 2023-09-22T21:24:44.047 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qmlJC58nxPMpK1pOKnvcaYoHK14= 2023-09-22T21:24:44.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qmlJC58nxPMpK1pOKnvcaYoHK14=', 'Content-Length': '0'} 2023-09-22T21:24:44.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030d656454c1b35fb-00650e061c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.051 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:44.051 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:44.051 DEBUG:boto:path=/ffzdqu-54/?prefix=a 2023-09-22T21:24:44.051 DEBUG:boto:auth_path=/ffzdqu-54/?prefix=a 2023-09-22T21:24:44.051 DEBUG:boto:Method: GET 2023-09-22T21:24:44.051 DEBUG:boto:Path: /ffzdqu-54/?prefix=a 2023-09-22T21:24:44.051 DEBUG:boto:Data: 2023-09-22T21:24:44.051 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.051 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.051 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.052 DEBUG:boto:Params: {} 2023-09-22T21:24:44.052 DEBUG:boto:Token: None 2023-09-22T21:24:44.052 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/ 2023-09-22T21:24:44.052 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk= 2023-09-22T21:24:44.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk=', 'Content-Length': '0'} 2023-09-22T21:24:44.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbb4b8f3dce5b3f8-00650e061c-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.102 DEBUG:boto:b'ffzdqu-54a1000falsea2023-09-22T21:24:28.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:44.103 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:44.103 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:44.103 DEBUG:boto:path=/ffzdqu-54/?prefix=a 2023-09-22T21:24:44.103 DEBUG:boto:auth_path=/ffzdqu-54/?prefix=a 2023-09-22T21:24:44.103 DEBUG:boto:Method: GET 2023-09-22T21:24:44.103 DEBUG:boto:Path: /ffzdqu-54/?prefix=a 2023-09-22T21:24:44.103 DEBUG:boto:Data: 2023-09-22T21:24:44.103 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.103 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.103 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.104 DEBUG:boto:Params: {} 2023-09-22T21:24:44.104 DEBUG:boto:Token: None 2023-09-22T21:24:44.104 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/ 2023-09-22T21:24:44.104 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk= 2023-09-22T21:24:44.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk=', 'Content-Length': '0'} 2023-09-22T21:24:44.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066c0658127900bed-00650e061c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.110 DEBUG:boto:b'ffzdqu-54a1000falsea2023-09-22T21:24:28.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:44.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:24:44.110 DEBUG:boto:path=/ffzdqu-54/b 2023-09-22T21:24:44.110 DEBUG:boto:auth_path=/ffzdqu-54/b 2023-09-22T21:24:44.111 DEBUG:boto:path=/ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.111 DEBUG:boto:auth_path=/ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.111 DEBUG:boto:Method: GET 2023-09-22T21:24:44.111 DEBUG:boto:Path: /ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.111 DEBUG:boto:Data: 2023-09-22T21:24:44.111 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.111 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.111 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.111 DEBUG:boto:Params: {} 2023-09-22T21:24:44.112 DEBUG:boto:Token: None 2023-09-22T21:24:44.112 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.112 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KzXov/bXqREmTyGPqTV43fKu6co= 2023-09-22T21:24:44.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KzXov/bXqREmTyGPqTV43fKu6co=', 'Content-Length': '0'} 2023-09-22T21:24:44.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-54:f132bdb9-20a7-4073-a932-8f99da20691e.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2a0998d7102e1e1-00650e061c-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.115 DEBUG:boto:path=/ffzdqu-54/b 2023-09-22T21:24:44.115 DEBUG:boto:auth_path=/ffzdqu-54/b 2023-09-22T21:24:44.115 DEBUG:boto:path=/ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.115 DEBUG:boto:auth_path=/ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.115 DEBUG:boto:Method: GET 2023-09-22T21:24:44.116 DEBUG:boto:Path: /ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.116 DEBUG:boto:Data: 2023-09-22T21:24:44.116 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.116 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.116 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.116 DEBUG:boto:Params: {} 2023-09-22T21:24:44.116 DEBUG:boto:Token: None 2023-09-22T21:24:44.116 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.116 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KzXov/bXqREmTyGPqTV43fKu6co= 2023-09-22T21:24:44.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KzXov/bXqREmTyGPqTV43fKu6co=', 'Content-Length': '0'} 2023-09-22T21:24:44.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dcdc9506b80c2fd-00650e061c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.162 DEBUG:boto:path=/ffzdqu-54/b 2023-09-22T21:24:44.162 DEBUG:boto:auth_path=/ffzdqu-54/b 2023-09-22T21:24:44.162 DEBUG:boto:path=/ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.163 DEBUG:boto:auth_path=/ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.163 DEBUG:boto:Method: HEAD 2023-09-22T21:24:44.163 DEBUG:boto:Path: /ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.163 DEBUG:boto:Data: 2023-09-22T21:24:44.163 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.163 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.163 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.163 DEBUG:boto:Params: {} 2023-09-22T21:24:44.163 DEBUG:boto:Token: None 2023-09-22T21:24:44.164 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.164 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OvDkbRNQInML15o0RO8PnljsRlM= 2023-09-22T21:24:44.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OvDkbRNQInML15o0RO8PnljsRlM=', 'Content-Length': '0'} 2023-09-22T21:24:44.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-54:f132bdb9-20a7-4073-a932-8f99da20691e.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a02bcf6fb7d2b636-00650e061c-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.167 DEBUG:boto:path=/ffzdqu-54/b 2023-09-22T21:24:44.167 DEBUG:boto:auth_path=/ffzdqu-54/b 2023-09-22T21:24:44.167 DEBUG:boto:path=/ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.167 DEBUG:boto:auth_path=/ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.167 DEBUG:boto:Method: HEAD 2023-09-22T21:24:44.168 DEBUG:boto:Path: /ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.168 DEBUG:boto:Data: 2023-09-22T21:24:44.168 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.168 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.168 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.168 DEBUG:boto:Params: {} 2023-09-22T21:24:44.168 DEBUG:boto:Token: None 2023-09-22T21:24:44.168 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.168 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OvDkbRNQInML15o0RO8PnljsRlM= 2023-09-22T21:24:44.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OvDkbRNQInML15o0RO8PnljsRlM=', 'Content-Length': '0'} 2023-09-22T21:24:44.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d4a60a426f593c0-00650e061c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:24:44.171 DEBUG:boto:path=/ffzdqu-54/b 2023-09-22T21:24:44.172 DEBUG:boto:auth_path=/ffzdqu-54/b 2023-09-22T21:24:44.172 DEBUG:boto:path=/ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.172 DEBUG:boto:auth_path=/ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.172 DEBUG:boto:Method: GET 2023-09-22T21:24:44.172 DEBUG:boto:Path: /ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.172 DEBUG:boto:Data: 2023-09-22T21:24:44.172 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.172 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.172 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.172 DEBUG:boto:Params: {} 2023-09-22T21:24:44.173 DEBUG:boto:Token: None 2023-09-22T21:24:44.173 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.173 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KzXov/bXqREmTyGPqTV43fKu6co= 2023-09-22T21:24:44.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KzXov/bXqREmTyGPqTV43fKu6co=', 'Content-Length': '0'} 2023-09-22T21:24:44.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-54:f132bdb9-20a7-4073-a932-8f99da20691e.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006960fef298817dee-00650e061c-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.218 DEBUG:boto:path=/ffzdqu-54/b 2023-09-22T21:24:44.218 DEBUG:boto:auth_path=/ffzdqu-54/b 2023-09-22T21:24:44.219 DEBUG:boto:path=/ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.219 DEBUG:boto:auth_path=/ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.219 DEBUG:boto:Method: GET 2023-09-22T21:24:44.219 DEBUG:boto:Path: /ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.219 DEBUG:boto:Data: 2023-09-22T21:24:44.219 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.219 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.219 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.219 DEBUG:boto:Params: {} 2023-09-22T21:24:44.220 DEBUG:boto:Token: None 2023-09-22T21:24:44.220 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/b?versionId=null 2023-09-22T21:24:44.220 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KzXov/bXqREmTyGPqTV43fKu6co= 2023-09-22T21:24:44.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KzXov/bXqREmTyGPqTV43fKu6co=', 'Content-Length': '0'} 2023-09-22T21:24:44.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcb9c54b1dab7919-00650e061c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.223 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:44.223 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:44.223 DEBUG:boto:path=/ffzdqu-54/?prefix=b 2023-09-22T21:24:44.223 DEBUG:boto:auth_path=/ffzdqu-54/?prefix=b 2023-09-22T21:24:44.223 DEBUG:boto:Method: GET 2023-09-22T21:24:44.224 DEBUG:boto:Path: /ffzdqu-54/?prefix=b 2023-09-22T21:24:44.224 DEBUG:boto:Data: 2023-09-22T21:24:44.224 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.224 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.224 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.224 DEBUG:boto:Params: {} 2023-09-22T21:24:44.224 DEBUG:boto:Token: None 2023-09-22T21:24:44.224 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/ 2023-09-22T21:24:44.224 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk= 2023-09-22T21:24:44.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk=', 'Content-Length': '0'} 2023-09-22T21:24:44.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ac790719c9bdd23-00650e061c-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.270 DEBUG:boto:b'ffzdqu-54b1000falseb2023-09-22T21:24:28.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:44.271 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:44.271 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:44.271 DEBUG:boto:path=/ffzdqu-54/?prefix=b 2023-09-22T21:24:44.271 DEBUG:boto:auth_path=/ffzdqu-54/?prefix=b 2023-09-22T21:24:44.271 DEBUG:boto:Method: GET 2023-09-22T21:24:44.271 DEBUG:boto:Path: /ffzdqu-54/?prefix=b 2023-09-22T21:24:44.271 DEBUG:boto:Data: 2023-09-22T21:24:44.271 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.271 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.272 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.272 DEBUG:boto:Params: {} 2023-09-22T21:24:44.272 DEBUG:boto:Token: None 2023-09-22T21:24:44.272 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/ 2023-09-22T21:24:44.272 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk= 2023-09-22T21:24:44.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk=', 'Content-Length': '0'} 2023-09-22T21:24:44.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d934410686784d9-00650e061c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.276 DEBUG:boto:b'ffzdqu-54b1000falseb2023-09-22T21:24:28.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:44.277 DEBUG:boto:path=/ffzdqu-54/b 2023-09-22T21:24:44.277 DEBUG:boto:auth_path=/ffzdqu-54/b 2023-09-22T21:24:44.277 DEBUG:boto:Method: HEAD 2023-09-22T21:24:44.277 DEBUG:boto:Path: /ffzdqu-54/b 2023-09-22T21:24:44.277 DEBUG:boto:Data: 2023-09-22T21:24:44.277 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.277 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.277 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.278 DEBUG:boto:Params: {} 2023-09-22T21:24:44.278 DEBUG:boto:Token: None 2023-09-22T21:24:44.278 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/b 2023-09-22T21:24:44.278 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Aj4qTUDpT3TZ1RyKDZC8Q2NuO1k= 2023-09-22T21:24:44.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Aj4qTUDpT3TZ1RyKDZC8Q2NuO1k=', 'Content-Length': '0'} 2023-09-22T21:24:44.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-54:f132bdb9-20a7-4073-a932-8f99da20691e.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000263f2c3d61e7b1c-00650e061c-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.281 DEBUG:boto:path=/ffzdqu-54/b 2023-09-22T21:24:44.281 DEBUG:boto:auth_path=/ffzdqu-54/b 2023-09-22T21:24:44.281 DEBUG:boto:Method: HEAD 2023-09-22T21:24:44.281 DEBUG:boto:Path: /ffzdqu-54/b 2023-09-22T21:24:44.281 DEBUG:boto:Data: 2023-09-22T21:24:44.281 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.281 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.282 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.282 DEBUG:boto:Params: {} 2023-09-22T21:24:44.282 DEBUG:boto:Token: None 2023-09-22T21:24:44.282 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/b 2023-09-22T21:24:44.282 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Aj4qTUDpT3TZ1RyKDZC8Q2NuO1k= 2023-09-22T21:24:44.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Aj4qTUDpT3TZ1RyKDZC8Q2NuO1k=', 'Content-Length': '0'} 2023-09-22T21:24:44.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b5eec5c1c641660-00650e061c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:24:44.285 DEBUG:boto:path=/ffzdqu-54/b 2023-09-22T21:24:44.285 DEBUG:boto:auth_path=/ffzdqu-54/b 2023-09-22T21:24:44.285 DEBUG:boto:Method: GET 2023-09-22T21:24:44.285 DEBUG:boto:Path: /ffzdqu-54/b 2023-09-22T21:24:44.285 DEBUG:boto:Data: 2023-09-22T21:24:44.285 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.286 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.286 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.286 DEBUG:boto:Params: {} 2023-09-22T21:24:44.286 DEBUG:boto:Token: None 2023-09-22T21:24:44.286 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/b 2023-09-22T21:24:44.286 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AJFbRq5r41gJsH90OdPUrA8PvxU= 2023-09-22T21:24:44.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AJFbRq5r41gJsH90OdPUrA8PvxU=', 'Content-Length': '0'} 2023-09-22T21:24:44.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-54:f132bdb9-20a7-4073-a932-8f99da20691e.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082c4b13ff3910c87-00650e061c-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.334 DEBUG:boto:path=/ffzdqu-54/b 2023-09-22T21:24:44.334 DEBUG:boto:auth_path=/ffzdqu-54/b 2023-09-22T21:24:44.335 DEBUG:boto:Method: GET 2023-09-22T21:24:44.335 DEBUG:boto:Path: /ffzdqu-54/b 2023-09-22T21:24:44.335 DEBUG:boto:Data: 2023-09-22T21:24:44.335 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.335 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.335 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.335 DEBUG:boto:Params: {} 2023-09-22T21:24:44.335 DEBUG:boto:Token: None 2023-09-22T21:24:44.335 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/b 2023-09-22T21:24:44.336 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AJFbRq5r41gJsH90OdPUrA8PvxU= 2023-09-22T21:24:44.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AJFbRq5r41gJsH90OdPUrA8PvxU=', 'Content-Length': '0'} 2023-09-22T21:24:44.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d825c943d0f2058-00650e061c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.339 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:44.339 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:44.339 DEBUG:boto:path=/ffzdqu-54/?prefix=b 2023-09-22T21:24:44.339 DEBUG:boto:auth_path=/ffzdqu-54/?prefix=b 2023-09-22T21:24:44.339 DEBUG:boto:Method: GET 2023-09-22T21:24:44.339 DEBUG:boto:Path: /ffzdqu-54/?prefix=b 2023-09-22T21:24:44.339 DEBUG:boto:Data: 2023-09-22T21:24:44.340 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.340 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.340 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.340 DEBUG:boto:Params: {} 2023-09-22T21:24:44.340 DEBUG:boto:Token: None 2023-09-22T21:24:44.340 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/ 2023-09-22T21:24:44.340 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk= 2023-09-22T21:24:44.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk=', 'Content-Length': '0'} 2023-09-22T21:24:44.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d55401d3a7517c5-00650e061c-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.345 DEBUG:boto:b'ffzdqu-54b1000falseb2023-09-22T21:24:28.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:44.345 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:44.345 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:44.345 DEBUG:boto:path=/ffzdqu-54/?prefix=b 2023-09-22T21:24:44.345 DEBUG:boto:auth_path=/ffzdqu-54/?prefix=b 2023-09-22T21:24:44.346 DEBUG:boto:Method: GET 2023-09-22T21:24:44.346 DEBUG:boto:Path: /ffzdqu-54/?prefix=b 2023-09-22T21:24:44.346 DEBUG:boto:Data: 2023-09-22T21:24:44.346 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.346 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.346 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.346 DEBUG:boto:Params: {} 2023-09-22T21:24:44.346 DEBUG:boto:Token: None 2023-09-22T21:24:44.346 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/ 2023-09-22T21:24:44.347 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk= 2023-09-22T21:24:44.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk=', 'Content-Length': '0'} 2023-09-22T21:24:44.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daefd41f82e2e0aa-00650e061c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.394 DEBUG:boto:b'ffzdqu-54b1000falseb2023-09-22T21:24:28.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:44.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:24:44.395 DEBUG:boto:path=/ffzdqu-54/c 2023-09-22T21:24:44.395 DEBUG:boto:auth_path=/ffzdqu-54/c 2023-09-22T21:24:44.395 DEBUG:boto:path=/ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.395 DEBUG:boto:auth_path=/ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.395 DEBUG:boto:Method: GET 2023-09-22T21:24:44.395 DEBUG:boto:Path: /ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.395 DEBUG:boto:Data: 2023-09-22T21:24:44.396 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.396 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.396 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.396 DEBUG:boto:Params: {} 2023-09-22T21:24:44.396 DEBUG:boto:Token: None 2023-09-22T21:24:44.396 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.396 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PhRLCCQCph3MgJkfVjxwgOqMizE= 2023-09-22T21:24:44.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PhRLCCQCph3MgJkfVjxwgOqMizE=', 'Content-Length': '0'} 2023-09-22T21:24:44.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-54:f132bdb9-20a7-4073-a932-8f99da20691e.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c52c82dd45d8459e-00650e061c-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.442 DEBUG:boto:path=/ffzdqu-54/c 2023-09-22T21:24:44.442 DEBUG:boto:auth_path=/ffzdqu-54/c 2023-09-22T21:24:44.442 DEBUG:boto:path=/ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.443 DEBUG:boto:auth_path=/ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.443 DEBUG:boto:Method: GET 2023-09-22T21:24:44.443 DEBUG:boto:Path: /ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.443 DEBUG:boto:Data: 2023-09-22T21:24:44.443 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.443 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.443 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.443 DEBUG:boto:Params: {} 2023-09-22T21:24:44.443 DEBUG:boto:Token: None 2023-09-22T21:24:44.444 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.444 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PhRLCCQCph3MgJkfVjxwgOqMizE= 2023-09-22T21:24:44.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PhRLCCQCph3MgJkfVjxwgOqMizE=', 'Content-Length': '0'} 2023-09-22T21:24:44.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034fefd3f4d4d6c83-00650e061c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.447 DEBUG:boto:path=/ffzdqu-54/c 2023-09-22T21:24:44.447 DEBUG:boto:auth_path=/ffzdqu-54/c 2023-09-22T21:24:44.447 DEBUG:boto:path=/ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.447 DEBUG:boto:auth_path=/ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.447 DEBUG:boto:Method: HEAD 2023-09-22T21:24:44.448 DEBUG:boto:Path: /ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.448 DEBUG:boto:Data: 2023-09-22T21:24:44.448 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.448 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.448 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.448 DEBUG:boto:Params: {} 2023-09-22T21:24:44.448 DEBUG:boto:Token: None 2023-09-22T21:24:44.448 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.449 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VzuXh7Kg8D+XdBspdoZxP6QxPTQ= 2023-09-22T21:24:44.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VzuXh7Kg8D+XdBspdoZxP6QxPTQ=', 'Content-Length': '0'} 2023-09-22T21:24:44.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-54:f132bdb9-20a7-4073-a932-8f99da20691e.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032fdd362a08b3992-00650e061c-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.451 DEBUG:boto:path=/ffzdqu-54/c 2023-09-22T21:24:44.451 DEBUG:boto:auth_path=/ffzdqu-54/c 2023-09-22T21:24:44.452 DEBUG:boto:path=/ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.452 DEBUG:boto:auth_path=/ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.452 DEBUG:boto:Method: HEAD 2023-09-22T21:24:44.452 DEBUG:boto:Path: /ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.452 DEBUG:boto:Data: 2023-09-22T21:24:44.452 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.452 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.452 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.452 DEBUG:boto:Params: {} 2023-09-22T21:24:44.453 DEBUG:boto:Token: None 2023-09-22T21:24:44.453 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.453 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VzuXh7Kg8D+XdBspdoZxP6QxPTQ= 2023-09-22T21:24:44.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VzuXh7Kg8D+XdBspdoZxP6QxPTQ=', 'Content-Length': '0'} 2023-09-22T21:24:44.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d57d41724eb5ee12-00650e061c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:24:44.456 DEBUG:boto:path=/ffzdqu-54/c 2023-09-22T21:24:44.456 DEBUG:boto:auth_path=/ffzdqu-54/c 2023-09-22T21:24:44.456 DEBUG:boto:path=/ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.456 DEBUG:boto:auth_path=/ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.456 DEBUG:boto:Method: GET 2023-09-22T21:24:44.457 DEBUG:boto:Path: /ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.457 DEBUG:boto:Data: 2023-09-22T21:24:44.457 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.457 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.457 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.457 DEBUG:boto:Params: {} 2023-09-22T21:24:44.457 DEBUG:boto:Token: None 2023-09-22T21:24:44.457 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.457 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PhRLCCQCph3MgJkfVjxwgOqMizE= 2023-09-22T21:24:44.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PhRLCCQCph3MgJkfVjxwgOqMizE=', 'Content-Length': '0'} 2023-09-22T21:24:44.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-54:f132bdb9-20a7-4073-a932-8f99da20691e.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079c38bbb43091bd7-00650e061c-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.502 DEBUG:boto:path=/ffzdqu-54/c 2023-09-22T21:24:44.502 DEBUG:boto:auth_path=/ffzdqu-54/c 2023-09-22T21:24:44.502 DEBUG:boto:path=/ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.503 DEBUG:boto:auth_path=/ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.503 DEBUG:boto:Method: GET 2023-09-22T21:24:44.503 DEBUG:boto:Path: /ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.503 DEBUG:boto:Data: 2023-09-22T21:24:44.503 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.503 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.503 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.503 DEBUG:boto:Params: {} 2023-09-22T21:24:44.503 DEBUG:boto:Token: None 2023-09-22T21:24:44.504 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/c?versionId=null 2023-09-22T21:24:44.504 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PhRLCCQCph3MgJkfVjxwgOqMizE= 2023-09-22T21:24:44.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PhRLCCQCph3MgJkfVjxwgOqMizE=', 'Content-Length': '0'} 2023-09-22T21:24:44.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a2758d631d7db2b-00650e061c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.507 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:44.507 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:44.507 DEBUG:boto:path=/ffzdqu-54/?prefix=c 2023-09-22T21:24:44.507 DEBUG:boto:auth_path=/ffzdqu-54/?prefix=c 2023-09-22T21:24:44.507 DEBUG:boto:Method: GET 2023-09-22T21:24:44.507 DEBUG:boto:Path: /ffzdqu-54/?prefix=c 2023-09-22T21:24:44.508 DEBUG:boto:Data: 2023-09-22T21:24:44.508 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.508 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.508 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.508 DEBUG:boto:Params: {} 2023-09-22T21:24:44.508 DEBUG:boto:Token: None 2023-09-22T21:24:44.508 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/ 2023-09-22T21:24:44.508 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk= 2023-09-22T21:24:44.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk=', 'Content-Length': '0'} 2023-09-22T21:24:44.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a7208545f3bff7a-00650e061c-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.554 DEBUG:boto:b'ffzdqu-54c1000falsec2023-09-22T21:24:28.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:44.555 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:44.555 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:44.555 DEBUG:boto:path=/ffzdqu-54/?prefix=c 2023-09-22T21:24:44.555 DEBUG:boto:auth_path=/ffzdqu-54/?prefix=c 2023-09-22T21:24:44.555 DEBUG:boto:Method: GET 2023-09-22T21:24:44.555 DEBUG:boto:Path: /ffzdqu-54/?prefix=c 2023-09-22T21:24:44.555 DEBUG:boto:Data: 2023-09-22T21:24:44.556 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.556 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.556 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.556 DEBUG:boto:Params: {} 2023-09-22T21:24:44.556 DEBUG:boto:Token: None 2023-09-22T21:24:44.556 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/ 2023-09-22T21:24:44.556 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk= 2023-09-22T21:24:44.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk=', 'Content-Length': '0'} 2023-09-22T21:24:44.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9ccd36345647528-00650e061c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.561 DEBUG:boto:b'ffzdqu-54c1000falsec2023-09-22T21:24:28.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:44.561 DEBUG:boto:path=/ffzdqu-54/c 2023-09-22T21:24:44.561 DEBUG:boto:auth_path=/ffzdqu-54/c 2023-09-22T21:24:44.561 DEBUG:boto:Method: HEAD 2023-09-22T21:24:44.562 DEBUG:boto:Path: /ffzdqu-54/c 2023-09-22T21:24:44.562 DEBUG:boto:Data: 2023-09-22T21:24:44.562 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.562 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.562 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.562 DEBUG:boto:Params: {} 2023-09-22T21:24:44.562 DEBUG:boto:Token: None 2023-09-22T21:24:44.562 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/c 2023-09-22T21:24:44.562 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KdZ7GvggB8l38Z4Fi/c+MIEr+ws= 2023-09-22T21:24:44.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KdZ7GvggB8l38Z4Fi/c+MIEr+ws=', 'Content-Length': '0'} 2023-09-22T21:24:44.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-54:f132bdb9-20a7-4073-a932-8f99da20691e.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9e52aaf41199539-00650e061c-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.565 DEBUG:boto:path=/ffzdqu-54/c 2023-09-22T21:24:44.565 DEBUG:boto:auth_path=/ffzdqu-54/c 2023-09-22T21:24:44.565 DEBUG:boto:Method: HEAD 2023-09-22T21:24:44.566 DEBUG:boto:Path: /ffzdqu-54/c 2023-09-22T21:24:44.566 DEBUG:boto:Data: 2023-09-22T21:24:44.566 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.566 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.566 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.566 DEBUG:boto:Params: {} 2023-09-22T21:24:44.566 DEBUG:boto:Token: None 2023-09-22T21:24:44.566 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/c 2023-09-22T21:24:44.567 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KdZ7GvggB8l38Z4Fi/c+MIEr+ws= 2023-09-22T21:24:44.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KdZ7GvggB8l38Z4Fi/c+MIEr+ws=', 'Content-Length': '0'} 2023-09-22T21:24:44.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000103ad2b34ad314af-00650e061c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:24:44.570 DEBUG:boto:path=/ffzdqu-54/c 2023-09-22T21:24:44.570 DEBUG:boto:auth_path=/ffzdqu-54/c 2023-09-22T21:24:44.570 DEBUG:boto:Method: GET 2023-09-22T21:24:44.570 DEBUG:boto:Path: /ffzdqu-54/c 2023-09-22T21:24:44.570 DEBUG:boto:Data: 2023-09-22T21:24:44.570 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.570 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.570 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.571 DEBUG:boto:Params: {} 2023-09-22T21:24:44.571 DEBUG:boto:Token: None 2023-09-22T21:24:44.571 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/c 2023-09-22T21:24:44.571 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GhyWUJQnhGyO1lK3Vvt+0/6AROs= 2023-09-22T21:24:44.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GhyWUJQnhGyO1lK3Vvt+0/6AROs=', 'Content-Length': '0'} 2023-09-22T21:24:44.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-54:f132bdb9-20a7-4073-a932-8f99da20691e.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009389382fdd6d30c0-00650e061c-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.618 DEBUG:boto:path=/ffzdqu-54/c 2023-09-22T21:24:44.618 DEBUG:boto:auth_path=/ffzdqu-54/c 2023-09-22T21:24:44.619 DEBUG:boto:Method: GET 2023-09-22T21:24:44.619 DEBUG:boto:Path: /ffzdqu-54/c 2023-09-22T21:24:44.619 DEBUG:boto:Data: 2023-09-22T21:24:44.619 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.619 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.619 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.619 DEBUG:boto:Params: {} 2023-09-22T21:24:44.619 DEBUG:boto:Token: None 2023-09-22T21:24:44.620 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/c 2023-09-22T21:24:44.620 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GhyWUJQnhGyO1lK3Vvt+0/6AROs= 2023-09-22T21:24:44.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GhyWUJQnhGyO1lK3Vvt+0/6AROs=', 'Content-Length': '0'} 2023-09-22T21:24:44.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8d6c6ee95e4fb59-00650e061c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.623 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:44.623 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:44.623 DEBUG:boto:path=/ffzdqu-54/?prefix=c 2023-09-22T21:24:44.623 DEBUG:boto:auth_path=/ffzdqu-54/?prefix=c 2023-09-22T21:24:44.623 DEBUG:boto:Method: GET 2023-09-22T21:24:44.624 DEBUG:boto:Path: /ffzdqu-54/?prefix=c 2023-09-22T21:24:44.624 DEBUG:boto:Data: 2023-09-22T21:24:44.624 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.624 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.624 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.624 DEBUG:boto:Params: {} 2023-09-22T21:24:44.624 DEBUG:boto:Token: None 2023-09-22T21:24:44.624 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/ 2023-09-22T21:24:44.624 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk= 2023-09-22T21:24:44.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk=', 'Content-Length': '0'} 2023-09-22T21:24:44.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2b6cd5eb3f17d51-00650e061c-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.629 DEBUG:boto:b'ffzdqu-54c1000falsec2023-09-22T21:24:28.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:44.629 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:44.630 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:44.630 DEBUG:boto:path=/ffzdqu-54/?prefix=c 2023-09-22T21:24:44.630 DEBUG:boto:auth_path=/ffzdqu-54/?prefix=c 2023-09-22T21:24:44.630 DEBUG:boto:Method: GET 2023-09-22T21:24:44.630 DEBUG:boto:Path: /ffzdqu-54/?prefix=c 2023-09-22T21:24:44.630 DEBUG:boto:Data: 2023-09-22T21:24:44.630 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.630 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.630 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.630 DEBUG:boto:Params: {} 2023-09-22T21:24:44.631 DEBUG:boto:Token: None 2023-09-22T21:24:44.631 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/ 2023-09-22T21:24:44.631 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk= 2023-09-22T21:24:44.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk=', 'Content-Length': '0'} 2023-09-22T21:24:44.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006df8da3ba19d40be-00650e061c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.678 DEBUG:boto:b'ffzdqu-54c1000falsec2023-09-22T21:24:28.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:44.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:24:44.679 DEBUG:boto:path=/ffzdqu-54/d 2023-09-22T21:24:44.679 DEBUG:boto:auth_path=/ffzdqu-54/d 2023-09-22T21:24:44.679 DEBUG:boto:path=/ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.679 DEBUG:boto:auth_path=/ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.679 DEBUG:boto:Method: GET 2023-09-22T21:24:44.679 DEBUG:boto:Path: /ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.679 DEBUG:boto:Data: 2023-09-22T21:24:44.680 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.680 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.680 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.680 DEBUG:boto:Params: {} 2023-09-22T21:24:44.680 DEBUG:boto:Token: None 2023-09-22T21:24:44.680 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.680 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bDxlTTz5nYP6+0qWjxJkSwKH+Zc= 2023-09-22T21:24:44.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bDxlTTz5nYP6+0qWjxJkSwKH+Zc=', 'Content-Length': '0'} 2023-09-22T21:24:44.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-54:f132bdb9-20a7-4073-a932-8f99da20691e.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003533f85bf91d9876-00650e061c-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.726 DEBUG:boto:path=/ffzdqu-54/d 2023-09-22T21:24:44.726 DEBUG:boto:auth_path=/ffzdqu-54/d 2023-09-22T21:24:44.727 DEBUG:boto:path=/ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.727 DEBUG:boto:auth_path=/ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.727 DEBUG:boto:Method: GET 2023-09-22T21:24:44.727 DEBUG:boto:Path: /ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.727 DEBUG:boto:Data: 2023-09-22T21:24:44.727 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.727 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.727 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.727 DEBUG:boto:Params: {} 2023-09-22T21:24:44.728 DEBUG:boto:Token: None 2023-09-22T21:24:44.728 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.728 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bDxlTTz5nYP6+0qWjxJkSwKH+Zc= 2023-09-22T21:24:44.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bDxlTTz5nYP6+0qWjxJkSwKH+Zc=', 'Content-Length': '0'} 2023-09-22T21:24:44.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076455040281bf5b4-00650e061c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.731 DEBUG:boto:path=/ffzdqu-54/d 2023-09-22T21:24:44.731 DEBUG:boto:auth_path=/ffzdqu-54/d 2023-09-22T21:24:44.731 DEBUG:boto:path=/ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.732 DEBUG:boto:auth_path=/ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.732 DEBUG:boto:Method: HEAD 2023-09-22T21:24:44.732 DEBUG:boto:Path: /ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.732 DEBUG:boto:Data: 2023-09-22T21:24:44.732 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.732 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.732 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.732 DEBUG:boto:Params: {} 2023-09-22T21:24:44.732 DEBUG:boto:Token: None 2023-09-22T21:24:44.733 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.733 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0H7duFQekHAD4PHP+hpI00GsPMk= 2023-09-22T21:24:44.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0H7duFQekHAD4PHP+hpI00GsPMk=', 'Content-Length': '0'} 2023-09-22T21:24:44.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-54:f132bdb9-20a7-4073-a932-8f99da20691e.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022a5426014bcf695-00650e061c-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.736 DEBUG:boto:path=/ffzdqu-54/d 2023-09-22T21:24:44.736 DEBUG:boto:auth_path=/ffzdqu-54/d 2023-09-22T21:24:44.736 DEBUG:boto:path=/ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.736 DEBUG:boto:auth_path=/ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.736 DEBUG:boto:Method: HEAD 2023-09-22T21:24:44.736 DEBUG:boto:Path: /ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.736 DEBUG:boto:Data: 2023-09-22T21:24:44.736 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.737 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.737 DEBUG:boto:Params: {} 2023-09-22T21:24:44.737 DEBUG:boto:Token: None 2023-09-22T21:24:44.737 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.737 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0H7duFQekHAD4PHP+hpI00GsPMk= 2023-09-22T21:24:44.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0H7duFQekHAD4PHP+hpI00GsPMk=', 'Content-Length': '0'} 2023-09-22T21:24:44.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eb2b850d3fd28bd-00650e061c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:24:44.740 DEBUG:boto:path=/ffzdqu-54/d 2023-09-22T21:24:44.740 DEBUG:boto:auth_path=/ffzdqu-54/d 2023-09-22T21:24:44.740 DEBUG:boto:path=/ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.740 DEBUG:boto:auth_path=/ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.740 DEBUG:boto:Method: GET 2023-09-22T21:24:44.741 DEBUG:boto:Path: /ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.741 DEBUG:boto:Data: 2023-09-22T21:24:44.741 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.741 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.741 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.741 DEBUG:boto:Params: {} 2023-09-22T21:24:44.741 DEBUG:boto:Token: None 2023-09-22T21:24:44.741 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.741 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bDxlTTz5nYP6+0qWjxJkSwKH+Zc= 2023-09-22T21:24:44.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bDxlTTz5nYP6+0qWjxJkSwKH+Zc=', 'Content-Length': '0'} 2023-09-22T21:24:44.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-54:f132bdb9-20a7-4073-a932-8f99da20691e.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007199dffc941ab141-00650e061c-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.786 DEBUG:boto:path=/ffzdqu-54/d 2023-09-22T21:24:44.786 DEBUG:boto:auth_path=/ffzdqu-54/d 2023-09-22T21:24:44.786 DEBUG:boto:path=/ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.787 DEBUG:boto:auth_path=/ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.787 DEBUG:boto:Method: GET 2023-09-22T21:24:44.787 DEBUG:boto:Path: /ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.787 DEBUG:boto:Data: 2023-09-22T21:24:44.787 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.787 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.787 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.787 DEBUG:boto:Params: {} 2023-09-22T21:24:44.787 DEBUG:boto:Token: None 2023-09-22T21:24:44.788 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/d?versionId=null 2023-09-22T21:24:44.788 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bDxlTTz5nYP6+0qWjxJkSwKH+Zc= 2023-09-22T21:24:44.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bDxlTTz5nYP6+0qWjxJkSwKH+Zc=', 'Content-Length': '0'} 2023-09-22T21:24:44.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009374b33ed7777001-00650e061c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.792 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:44.792 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:44.792 DEBUG:boto:path=/ffzdqu-54/?prefix=d 2023-09-22T21:24:44.792 DEBUG:boto:auth_path=/ffzdqu-54/?prefix=d 2023-09-22T21:24:44.792 DEBUG:boto:Method: GET 2023-09-22T21:24:44.792 DEBUG:boto:Path: /ffzdqu-54/?prefix=d 2023-09-22T21:24:44.792 DEBUG:boto:Data: 2023-09-22T21:24:44.792 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.792 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.793 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.793 DEBUG:boto:Params: {} 2023-09-22T21:24:44.793 DEBUG:boto:Token: None 2023-09-22T21:24:44.793 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/ 2023-09-22T21:24:44.793 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk= 2023-09-22T21:24:44.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk=', 'Content-Length': '0'} 2023-09-22T21:24:44.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026aa5263d5656637-00650e061c-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.838 DEBUG:boto:b'ffzdqu-54d1000falsed2023-09-22T21:24:28.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:44.839 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:44.839 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:44.839 DEBUG:boto:path=/ffzdqu-54/?prefix=d 2023-09-22T21:24:44.839 DEBUG:boto:auth_path=/ffzdqu-54/?prefix=d 2023-09-22T21:24:44.839 DEBUG:boto:Method: GET 2023-09-22T21:24:44.839 DEBUG:boto:Path: /ffzdqu-54/?prefix=d 2023-09-22T21:24:44.839 DEBUG:boto:Data: 2023-09-22T21:24:44.839 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.839 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.840 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.840 DEBUG:boto:Params: {} 2023-09-22T21:24:44.840 DEBUG:boto:Token: None 2023-09-22T21:24:44.840 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/ 2023-09-22T21:24:44.840 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk= 2023-09-22T21:24:44.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk=', 'Content-Length': '0'} 2023-09-22T21:24:44.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed5cedde2a9921c7-00650e061c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.845 DEBUG:boto:b'ffzdqu-54d1000falsed2023-09-22T21:24:28.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:44.845 DEBUG:boto:path=/ffzdqu-54/d 2023-09-22T21:24:44.846 DEBUG:boto:auth_path=/ffzdqu-54/d 2023-09-22T21:24:44.846 DEBUG:boto:Method: HEAD 2023-09-22T21:24:44.846 DEBUG:boto:Path: /ffzdqu-54/d 2023-09-22T21:24:44.846 DEBUG:boto:Data: 2023-09-22T21:24:44.846 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.846 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.846 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.846 DEBUG:boto:Params: {} 2023-09-22T21:24:44.847 DEBUG:boto:Token: None 2023-09-22T21:24:44.847 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/d 2023-09-22T21:24:44.847 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:s3uokOjRVbSZZqN5fLGl28mPyhw= 2023-09-22T21:24:44.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:s3uokOjRVbSZZqN5fLGl28mPyhw=', 'Content-Length': '0'} 2023-09-22T21:24:44.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-54:f132bdb9-20a7-4073-a932-8f99da20691e.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064be84549ccb2d81-00650e061c-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.850 DEBUG:boto:path=/ffzdqu-54/d 2023-09-22T21:24:44.850 DEBUG:boto:auth_path=/ffzdqu-54/d 2023-09-22T21:24:44.850 DEBUG:boto:Method: HEAD 2023-09-22T21:24:44.850 DEBUG:boto:Path: /ffzdqu-54/d 2023-09-22T21:24:44.851 DEBUG:boto:Data: 2023-09-22T21:24:44.851 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.851 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.851 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.851 DEBUG:boto:Params: {} 2023-09-22T21:24:44.851 DEBUG:boto:Token: None 2023-09-22T21:24:44.851 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/d 2023-09-22T21:24:44.851 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:s3uokOjRVbSZZqN5fLGl28mPyhw= 2023-09-22T21:24:44.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:s3uokOjRVbSZZqN5fLGl28mPyhw=', 'Content-Length': '0'} 2023-09-22T21:24:44.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a34f5385ccae29ca-00650e061c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:24:44.854 DEBUG:boto:path=/ffzdqu-54/d 2023-09-22T21:24:44.855 DEBUG:boto:auth_path=/ffzdqu-54/d 2023-09-22T21:24:44.855 DEBUG:boto:Method: GET 2023-09-22T21:24:44.855 DEBUG:boto:Path: /ffzdqu-54/d 2023-09-22T21:24:44.855 DEBUG:boto:Data: 2023-09-22T21:24:44.855 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.855 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.855 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.855 DEBUG:boto:Params: {} 2023-09-22T21:24:44.855 DEBUG:boto:Token: None 2023-09-22T21:24:44.856 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/d 2023-09-22T21:24:44.856 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Yltcw4DIQVQCHi2e8CHQ9dXDSag= 2023-09-22T21:24:44.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Yltcw4DIQVQCHi2e8CHQ9dXDSag=', 'Content-Length': '0'} 2023-09-22T21:24:44.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-54:f132bdb9-20a7-4073-a932-8f99da20691e.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc82f86e82ff29fe-00650e061c-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.902 DEBUG:boto:path=/ffzdqu-54/d 2023-09-22T21:24:44.902 DEBUG:boto:auth_path=/ffzdqu-54/d 2023-09-22T21:24:44.902 DEBUG:boto:Method: GET 2023-09-22T21:24:44.903 DEBUG:boto:Path: /ffzdqu-54/d 2023-09-22T21:24:44.903 DEBUG:boto:Data: 2023-09-22T21:24:44.903 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.903 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.903 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.903 DEBUG:boto:Params: {} 2023-09-22T21:24:44.903 DEBUG:boto:Token: None 2023-09-22T21:24:44.903 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/d 2023-09-22T21:24:44.903 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Yltcw4DIQVQCHi2e8CHQ9dXDSag= 2023-09-22T21:24:44.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Yltcw4DIQVQCHi2e8CHQ9dXDSag=', 'Content-Length': '0'} 2023-09-22T21:24:44.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3659b4759d844f7-00650e061c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.907 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:44.907 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:44.907 DEBUG:boto:path=/ffzdqu-54/?prefix=d 2023-09-22T21:24:44.907 DEBUG:boto:auth_path=/ffzdqu-54/?prefix=d 2023-09-22T21:24:44.907 DEBUG:boto:Method: GET 2023-09-22T21:24:44.907 DEBUG:boto:Path: /ffzdqu-54/?prefix=d 2023-09-22T21:24:44.907 DEBUG:boto:Data: 2023-09-22T21:24:44.907 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.907 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:24:44.907 DEBUG:boto:Port: 8001 2023-09-22T21:24:44.908 DEBUG:boto:Params: {} 2023-09-22T21:24:44.908 DEBUG:boto:Token: None 2023-09-22T21:24:44.908 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/ 2023-09-22T21:24:44.908 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk= 2023-09-22T21:24:44.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk=', 'Content-Length': '0'} 2023-09-22T21:24:44.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ce47ff047856c4e-00650e061c-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.913 DEBUG:boto:b'ffzdqu-54d1000falsed2023-09-22T21:24:28.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:44.913 DEBUG:boto:path=/ffzdqu-54/ 2023-09-22T21:24:44.914 DEBUG:boto:auth_path=/ffzdqu-54/ 2023-09-22T21:24:44.914 DEBUG:boto:path=/ffzdqu-54/?prefix=d 2023-09-22T21:24:44.914 DEBUG:boto:auth_path=/ffzdqu-54/?prefix=d 2023-09-22T21:24:44.914 DEBUG:boto:Method: GET 2023-09-22T21:24:44.914 DEBUG:boto:Path: /ffzdqu-54/?prefix=d 2023-09-22T21:24:44.914 DEBUG:boto:Data: 2023-09-22T21:24:44.914 DEBUG:boto:Headers: {} 2023-09-22T21:24:44.914 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:44.914 DEBUG:boto:Port: 8000 2023-09-22T21:24:44.915 DEBUG:boto:Params: {} 2023-09-22T21:24:44.915 DEBUG:boto:Token: None 2023-09-22T21:24:44.915 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:24:44 GMT /ffzdqu-54/ 2023-09-22T21:24:44.915 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk= 2023-09-22T21:24:44.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:44 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VWuUDLtb9pXCp6TSJgWnqx4mcFk=', 'Content-Length': '0'} 2023-09-22T21:24:44.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a843bfdd4eebb75-00650e061c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:44.962 DEBUG:boto:b'ffzdqu-54d1000falsed2023-09-22T21:24:28.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:24:44.963 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-54 zones={a2, a1} 2023-09-22T21:24:44.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:24:44.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:24:44.963 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:24:45.002 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:24:45.003 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:24:45.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'ffzdqu-54', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:45.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'ffzdqu-54', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:45.068 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket ffzdqu-54 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:24:45.143 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:24:45.143 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:24:45.199 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2023-09-22T21:24:45.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'ffzdqu-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:45.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'ffzdqu-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:45.200 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket ffzdqu-53 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:24:45.281 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:24:45.282 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:24:45.338 INFO:teuthology.orchestra.run.smithi160.stdout:[{"op_id":"1#00000000001.18.6","op_tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.17732241849201877521","op":"write","object":"a","instance":"","state":"complete","index_ver":1,"timestamp":"2023-09-22T21:24:21.298300351Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18"}]},{"op_id":"5#00000000001.47.6","op_tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.13813163618142544567","op":"write","object":"d","instance":"","state":"complete","index_ver":1,"timestamp":"2023-09-22T21:24:21.394394608Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18"}]},{"op_id":"8#00000000001.31.6","op_tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.16031048186022934425","op":"write","object":"c","instance":"","state":"complete","index_ver":1,"timestamp":"2023-09-22T21:24:21.375061632Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18"}]},{"op_id":"10#00000000001.33.6","op_tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.9106532711263823274","op":"write","object":"b","instance":"","state":"complete","index_ver":1,"timestamp":"2023-09-22T21:24:21.319367511Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-53:f132bdb9-20a7-4073-a932-8f99da20691e.5729.18"}]}] 2023-09-22T21:24:45.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:24:45.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:24:45.339 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:24:45.419 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:24:45.419 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:24:45.473 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.471+0000 7ff453b6d640 -1 req 4913076503514611621 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-22:f132bdb9-20a7-4073-a932-8f99da20691e.4382.22,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-22:f132bdb9-20a7-4073-a932-8f99da20691e.4382.22,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:45.474 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.471+0000 7ff446b53640 -1 req 10442566947887621457 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-24:f132bdb9-20a7-4073-a932-8f99da20691e.4382.24,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-24:f132bdb9-20a7-4073-a932-8f99da20691e.4382.24,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:45.478 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.475+0000 7ff441348640 -1 req 4916291762668049423 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-2:f132bdb9-20a7-4073-a932-8f99da20691e.4382.2,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-2:f132bdb9-20a7-4073-a932-8f99da20691e.4382.2,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:45.478 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.479+0000 7ff42c31e640 -1 req 112979225739090992 0.003999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-42:f132bdb9-20a7-4073-a932-8f99da20691e.5729.7,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-42:f132bdb9-20a7-4073-a932-8f99da20691e.5729.7,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:45.482 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.479+0000 7ff435b31640 -1 req 4284001834147215772 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-20:f132bdb9-20a7-4073-a932-8f99da20691e.4382.20,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-20:f132bdb9-20a7-4073-a932-8f99da20691e.4382.20,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:45.482 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.479+0000 7ff43d340640 -1 req 370860825508184346 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-27:f132bdb9-20a7-4073-a932-8f99da20691e.4382.27,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-27:f132bdb9-20a7-4073-a932-8f99da20691e.4382.27,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:45.578 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.575+0000 7ff419af9640 -1 req 1427852667212858548 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-21:f132bdb9-20a7-4073-a932-8f99da20691e.4382.21,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-21:f132bdb9-20a7-4073-a932-8f99da20691e.4382.21,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:45.622 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.619+0000 7ff41db01640 -1 req 5078064502970261146 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-1:f132bdb9-20a7-4073-a932-8f99da20691e.4382.1,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-1:f132bdb9-20a7-4073-a932-8f99da20691e.4382.1,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:45.636 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.635+0000 7ff40cadf640 -1 req 892540938554854948 0.003999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-39:f132bdb9-20a7-4073-a932-8f99da20691e.5729.4,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-39:f132bdb9-20a7-4073-a932-8f99da20691e.5729.4,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:45.641 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.639+0000 7ff3eba9d640 -1 req 122311715095784881 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-40:f132bdb9-20a7-4073-a932-8f99da20691e.5729.5,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-40:f132bdb9-20a7-4073-a932-8f99da20691e.5729.5,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:45.683 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.683+0000 7ff3eeaa3640 -1 req 6916649399644330982 0.003999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-28:f132bdb9-20a7-4073-a932-8f99da20691e.4382.28,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-28:f132bdb9-20a7-4073-a932-8f99da20691e.4382.28,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:45.715 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.715+0000 7ff3c6a53640 -1 req 3108011821153426942 0.003999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-3:f132bdb9-20a7-4073-a932-8f99da20691e.4382.3,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-3:f132bdb9-20a7-4073-a932-8f99da20691e.4382.3,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:45.748 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.747+0000 7ff3cf264640 -1 req 8013341478778425761 0.003999924s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-4:f132bdb9-20a7-4073-a932-8f99da20691e.4382.4,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-4:f132bdb9-20a7-4073-a932-8f99da20691e.4382.4,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:45.763 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.763+0000 7ff3b7a35640 -1 req 13664600767312655284 0.003999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-29:f132bdb9-20a7-4073-a932-8f99da20691e.4382.29,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-29:f132bdb9-20a7-4073-a932-8f99da20691e.4382.29,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:45.786 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.787+0000 7ff3b8a37640 -1 req 9772394361895907288 0.007999847s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-5:f132bdb9-20a7-4073-a932-8f99da20691e.4382.5,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-5:f132bdb9-20a7-4073-a932-8f99da20691e.4382.5,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:45.836 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.835+0000 7ff39f204640 -1 req 15117802336599248021 0.003999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-23:f132bdb9-20a7-4073-a932-8f99da20691e.4382.23,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-23:f132bdb9-20a7-4073-a932-8f99da20691e.4382.23,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:45.851 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.851+0000 7ff38e1e2640 -1 req 2920618827062741310 0.003999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-30:f132bdb9-20a7-4073-a932-8f99da20691e.4382.30,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-30:f132bdb9-20a7-4073-a932-8f99da20691e.4382.30,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:45.869 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.867+0000 7ff46c39e640 -1 req 5548125551355096384 0.003999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-26:f132bdb9-20a7-4073-a932-8f99da20691e.4382.26,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-26:f132bdb9-20a7-4073-a932-8f99da20691e.4382.26,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:45.891 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.891+0000 7ff3751b0640 -1 req 7577557812563745292 0.003999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-38:f132bdb9-20a7-4073-a932-8f99da20691e.5729.3,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-38:f132bdb9-20a7-4073-a932-8f99da20691e.5729.3,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:45.939 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.939+0000 7ff46cb9f640 -1 req 7390485238425666784 0.003999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-36:f132bdb9-20a7-4073-a932-8f99da20691e.5729.1,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-36:f132bdb9-20a7-4073-a932-8f99da20691e.5729.1,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:45.971 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.971+0000 7ff461b89640 -1 req 9971156878373135743 0.003999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-43:f132bdb9-20a7-4073-a932-8f99da20691e.5729.8,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-43:f132bdb9-20a7-4073-a932-8f99da20691e.5729.8,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:45.997 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:45.995+0000 7ff465390640 -1 req 15614302680412895421 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-19:f132bdb9-20a7-4073-a932-8f99da20691e.4382.19,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-19:f132bdb9-20a7-4073-a932-8f99da20691e.4382.19,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:46.038 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:46.035+0000 7ff42cb1f640 -1 req 13485591136098384813 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-37:f132bdb9-20a7-4073-a932-8f99da20691e.5729.2,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-37:f132bdb9-20a7-4073-a932-8f99da20691e.5729.2,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:46.043 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:46.043+0000 7ff43db41640 -1 req 9031031362782727284 0.003999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-41:f132bdb9-20a7-4073-a932-8f99da20691e.5729.6,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-41:f132bdb9-20a7-4073-a932-8f99da20691e.5729.6,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:46.073 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:46.071+0000 7ff42bb1d640 -1 req 2708406422905030882 0.003999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-6:f132bdb9-20a7-4073-a932-8f99da20691e.4382.6,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-6:f132bdb9-20a7-4073-a932-8f99da20691e.4382.6,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:46.075 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:46.075+0000 7ff44a35a640 -1 req 6939551570699276543 0.003999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-51:f132bdb9-20a7-4073-a932-8f99da20691e.5729.16,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-51:f132bdb9-20a7-4073-a932-8f99da20691e.5729.16,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:46.086 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:24:46.083+0000 7ff43eb43640 -1 req 3254621174573622150 0.003999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-25:f132bdb9-20a7-4073-a932-8f99da20691e.4382.25,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-25:f132bdb9-20a7-4073-a932-8f99da20691e.4382.25,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:24:46.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'ffzdqu-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:46.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'ffzdqu-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:46.174 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket ffzdqu-53 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:24:46.255 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:24:46.255 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:24:46.312 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2023-09-22T21:24:46.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ok 2023-09-22T21:24:46.313 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-55 2023-09-22T21:24:46.313 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:24:46.314 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:24:46.314 DEBUG:boto:Method: PUT 2023-09-22T21:24:46.314 DEBUG:boto:Path: /ffzdqu-55/ 2023-09-22T21:24:46.314 DEBUG:boto:Data: 2023-09-22T21:24:46.314 DEBUG:boto:Headers: {} 2023-09-22T21:24:46.314 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:46.314 DEBUG:boto:Port: 8000 2023-09-22T21:24:46.314 DEBUG:boto:Params: {} 2023-09-22T21:24:46.315 DEBUG:boto:Token: None 2023-09-22T21:24:46.315 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:24:46 GMT /ffzdqu-55/ 2023-09-22T21:24:46.315 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rHtILjz7GI5o3mwpZlYTo+sfXHw= 2023-09-22T21:24:46.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rHtILjz7GI5o3mwpZlYTo+sfXHw=', 'Content-Length': '0'} 2023-09-22T21:24:46.334 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ba2a4305f0863de1-00650e061e-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:46.334 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:24:46.334 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:24:46.334 DEBUG:boto:Method: HEAD 2023-09-22T21:24:46.334 DEBUG:boto:Path: /ffzdqu-55/ 2023-09-22T21:24:46.334 DEBUG:boto:Data: 2023-09-22T21:24:46.335 DEBUG:boto:Headers: {} 2023-09-22T21:24:46.335 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:46.335 DEBUG:boto:Port: 8000 2023-09-22T21:24:46.335 DEBUG:boto:Params: {} 2023-09-22T21:24:46.335 DEBUG:boto:Token: None 2023-09-22T21:24:46.335 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:46 GMT /ffzdqu-55/ 2023-09-22T21:24:46.335 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22h+P18CQEWtMU3DNTAGeL6t294= 2023-09-22T21:24:46.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22h+P18CQEWtMU3DNTAGeL6t294=', 'Content-Length': '0'} 2023-09-22T21:24:46.338 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000337ef3e4d9816750-00650e061e-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:46.338 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:24:46.339 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:24:46.339 DEBUG:boto:Method: PUT 2023-09-22T21:24:46.339 DEBUG:boto:Path: /ffzdqu-55/a 2023-09-22T21:24:46.339 DEBUG:boto:Data: 2023-09-22T21:24:46.339 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:24:46.339 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:46.339 DEBUG:boto:Port: 8000 2023-09-22T21:24:46.339 DEBUG:boto:Params: {} 2023-09-22T21:24:46.340 DEBUG:boto:Token: None 2023-09-22T21:24:46.340 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:24:46 GMT /ffzdqu-55/a 2023-09-22T21:24:46.340 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dsxFOaR8/aJ8ChMFxr0/MjRE8cQ= 2023-09-22T21:24:46.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:24:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dsxFOaR8/aJ8ChMFxr0/MjRE8cQ='} 2023-09-22T21:24:46.348 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e96a96a903f2b5e-00650e061e-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:46.348 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:24:46.348 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:24:46.348 DEBUG:boto:Method: HEAD 2023-09-22T21:24:46.348 DEBUG:boto:Path: /ffzdqu-55/ 2023-09-22T21:24:46.348 DEBUG:boto:Data: 2023-09-22T21:24:46.349 DEBUG:boto:Headers: {} 2023-09-22T21:24:46.349 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:46.349 DEBUG:boto:Port: 8000 2023-09-22T21:24:46.349 DEBUG:boto:Params: {} 2023-09-22T21:24:46.349 DEBUG:boto:Token: None 2023-09-22T21:24:46.349 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:46 GMT /ffzdqu-55/ 2023-09-22T21:24:46.349 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22h+P18CQEWtMU3DNTAGeL6t294= 2023-09-22T21:24:46.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22h+P18CQEWtMU3DNTAGeL6t294=', 'Content-Length': '0'} 2023-09-22T21:24:46.353 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070ef11137ad9a638-00650e061e-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:46.353 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:24:46.353 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:24:46.354 DEBUG:boto:Method: PUT 2023-09-22T21:24:46.354 DEBUG:boto:Path: /ffzdqu-55/b 2023-09-22T21:24:46.354 DEBUG:boto:Data: 2023-09-22T21:24:46.354 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:24:46.354 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:46.354 DEBUG:boto:Port: 8000 2023-09-22T21:24:46.354 DEBUG:boto:Params: {} 2023-09-22T21:24:46.354 DEBUG:boto:Token: None 2023-09-22T21:24:46.355 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:24:46 GMT /ffzdqu-55/b 2023-09-22T21:24:46.355 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x8GSsmXYu/ONBWi0ew2KmW9d0fE= 2023-09-22T21:24:46.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:24:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x8GSsmXYu/ONBWi0ew2KmW9d0fE='} 2023-09-22T21:24:46.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f893217924c8e00-00650e061e-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:46.399 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:24:46.399 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:24:46.399 DEBUG:boto:Method: HEAD 2023-09-22T21:24:46.399 DEBUG:boto:Path: /ffzdqu-55/ 2023-09-22T21:24:46.399 DEBUG:boto:Data: 2023-09-22T21:24:46.399 DEBUG:boto:Headers: {} 2023-09-22T21:24:46.399 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:46.399 DEBUG:boto:Port: 8000 2023-09-22T21:24:46.400 DEBUG:boto:Params: {} 2023-09-22T21:24:46.400 DEBUG:boto:Token: None 2023-09-22T21:24:46.400 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:46 GMT /ffzdqu-55/ 2023-09-22T21:24:46.400 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22h+P18CQEWtMU3DNTAGeL6t294= 2023-09-22T21:24:46.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22h+P18CQEWtMU3DNTAGeL6t294=', 'Content-Length': '0'} 2023-09-22T21:24:46.404 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d90cb4f63e535ed-00650e061e-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:46.404 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:24:46.404 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:24:46.404 DEBUG:boto:Method: PUT 2023-09-22T21:24:46.404 DEBUG:boto:Path: /ffzdqu-55/c 2023-09-22T21:24:46.404 DEBUG:boto:Data: 2023-09-22T21:24:46.405 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:24:46.405 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:46.405 DEBUG:boto:Port: 8000 2023-09-22T21:24:46.405 DEBUG:boto:Params: {} 2023-09-22T21:24:46.405 DEBUG:boto:Token: None 2023-09-22T21:24:46.405 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:24:46 GMT /ffzdqu-55/c 2023-09-22T21:24:46.405 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2TQoyQcC4IOpnAMOd9mmdFwA3CE= 2023-09-22T21:24:46.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:24:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2TQoyQcC4IOpnAMOd9mmdFwA3CE='} 2023-09-22T21:24:46.415 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000068f2d4d6991cfd65-00650e061e-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:46.416 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:24:46.416 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:24:46.416 DEBUG:boto:Method: HEAD 2023-09-22T21:24:46.416 DEBUG:boto:Path: /ffzdqu-55/ 2023-09-22T21:24:46.416 DEBUG:boto:Data: 2023-09-22T21:24:46.416 DEBUG:boto:Headers: {} 2023-09-22T21:24:46.416 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:46.416 DEBUG:boto:Port: 8000 2023-09-22T21:24:46.416 DEBUG:boto:Params: {} 2023-09-22T21:24:46.417 DEBUG:boto:Token: None 2023-09-22T21:24:46.417 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:24:46 GMT /ffzdqu-55/ 2023-09-22T21:24:46.417 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22h+P18CQEWtMU3DNTAGeL6t294= 2023-09-22T21:24:46.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:24:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22h+P18CQEWtMU3DNTAGeL6t294=', 'Content-Length': '0'} 2023-09-22T21:24:46.421 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e177f1200d12cca8-00650e061e-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:46.421 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:24:46.421 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:24:46.422 DEBUG:boto:Method: PUT 2023-09-22T21:24:46.422 DEBUG:boto:Path: /ffzdqu-55/d 2023-09-22T21:24:46.422 DEBUG:boto:Data: 2023-09-22T21:24:46.422 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:24:46.422 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:24:46.422 DEBUG:boto:Port: 8000 2023-09-22T21:24:46.422 DEBUG:boto:Params: {} 2023-09-22T21:24:46.422 DEBUG:boto:Token: None 2023-09-22T21:24:46.423 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:24:46 GMT /ffzdqu-55/d 2023-09-22T21:24:46.423 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/hf9CP5I4AQabDTmWfINotnn3Hs= 2023-09-22T21:24:46.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:24:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/hf9CP5I4AQabDTmWfINotnn3Hs='} 2023-09-22T21:24:46.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a80211569f7b0006-00650e061e-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:24:46.467 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:46.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:46.467 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:24:46.509 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:24:46.509 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:24:46.548 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:24:46.567 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695417886.334778_413.1","last_update":"2023-09-22T21:24:46.334778Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:24:46.567 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417886.334778_413.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:24:46.567 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:24:46.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:46.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:46.568 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:24:46.649 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:24:46.649 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:24:46.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:46.683+0000 7f69c1682a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:24:46.687 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:46.687+0000 7f69c1682a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:24:46.705 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417886.327221_410.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:24:46.327221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:24:46Z"} 2023-09-22T21:24:46.705 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417886.327221_410.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:24:46.327221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:24:46Z"} 2023-09-22T21:24:46.706 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417886.327221_410.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:24:46.327221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:24:46.706 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417886.334778_413.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:24:46.706 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417886.327221_410.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:24:46.706 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695417886.334778_413.1 target=1_1695417886.327221_410.1 2023-09-22T21:24:51.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:51.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:24:51.708 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:24:51.751 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:24:51.751 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:24:51.789 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:51.787+0000 7f6527f81a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:24:51.790 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:51.787+0000 7f6527f81a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:24:51.813 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417886.334778_413.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:24:46.334778Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:24:51Z"} 2023-09-22T21:24:51.813 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695417886.334778_413.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:24:46.334778Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:24:51Z"} 2023-09-22T21:24:51.813 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695417886.334778_413.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:24:46.334778Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:24:51.814 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417886.334778_413.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:24:51.814 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695417886.334778_413.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:24:51.814 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:24:51.814 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-55 2023-09-22T21:24:51.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:24:51.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:24:51.815 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:24:51.896 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:24:51.896 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:24:51.939 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:51.939+0000 7f5b94bf0a40 1 waiting to reach incremental sync.. 2023-09-22T21:24:56.939 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:24:56.939+0000 7f5b94bf0a40 1 waiting to reach incremental sync.. 2023-09-22T21:25:01.940 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:25:01.938+0000 7f5b94bf0a40 1 waiting to reach incremental sync.. 2023-09-22T21:25:06.944 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:25:06.942+0000 7f5b94bf0a40 1 bucket sync caught up with source: 2023-09-22T21:25:06.944 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.103.6, , , , 00000000001.26.6, , , 00000000001.29.6, , 00000000001.27.6] 2023-09-22T21:25:06.944 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.103.6, , , , 00000000001.26.6, , , 00000000001.29.6, , 00000000001.27.6] 2023-09-22T21:25:06.945 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:25:06.942+0000 7f5b94bf0a40 0 bucket checkpoint complete 2023-09-22T21:25:06.975 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-55 2023-09-22T21:25:06.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:25:06.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:25:06.975 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:25:07.067 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:25:07.067 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:25:07.110 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:25:07.106+0000 7fe65495ba40 1 bucket sync caught up with source: 2023-09-22T21:25:07.110 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.274.6, , , , 00000000001.197.6, , , 00000000001.157.6, , 00000000001.155.6] 2023-09-22T21:25:07.111 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.274.6, , , , 00000000001.197.6, , , 00000000001.157.6, , 00000000001.155.6] 2023-09-22T21:25:07.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:25:07.106+0000 7fe65495ba40 0 bucket checkpoint complete 2023-09-22T21:25:07.127 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-55 zones={a2, a1} 2023-09-22T21:25:07.127 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:07.127 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:07.127 DEBUG:boto:Method: HEAD 2023-09-22T21:25:07.128 DEBUG:boto:Path: /ffzdqu-55/ 2023-09-22T21:25:07.128 DEBUG:boto:Data: 2023-09-22T21:25:07.128 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.128 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.128 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.128 DEBUG:boto:Params: {} 2023-09-22T21:25:07.128 DEBUG:boto:Token: None 2023-09-22T21:25:07.128 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/ 2023-09-22T21:25:07.128 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kKN2pDTM1QW11XoFQeSrUowLqgc= 2023-09-22T21:25:07.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kKN2pDTM1QW11XoFQeSrUowLqgc=', 'Content-Length': '0'} 2023-09-22T21:25:07.131 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fbe76b90bcf41f70-00650e0633-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.131 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:07.131 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:07.132 DEBUG:boto:Method: HEAD 2023-09-22T21:25:07.132 DEBUG:boto:Path: /ffzdqu-55/ 2023-09-22T21:25:07.132 DEBUG:boto:Data: 2023-09-22T21:25:07.132 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.132 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.132 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.132 DEBUG:boto:Params: {} 2023-09-22T21:25:07.132 DEBUG:boto:Token: None 2023-09-22T21:25:07.132 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/ 2023-09-22T21:25:07.133 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kKN2pDTM1QW11XoFQeSrUowLqgc= 2023-09-22T21:25:07.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kKN2pDTM1QW11XoFQeSrUowLqgc=', 'Content-Length': '0'} 2023-09-22T21:25:07.135 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4fa19978b67573d-00650e0633-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.135 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:25:07.135 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:07.135 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:07.136 DEBUG:boto:path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:25:07.136 DEBUG:boto:auth_path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:25:07.136 DEBUG:boto:Method: GET 2023-09-22T21:25:07.136 DEBUG:boto:Path: /ffzdqu-55/?versions&max-keys=999 2023-09-22T21:25:07.136 DEBUG:boto:Data: 2023-09-22T21:25:07.136 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.136 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.136 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.136 DEBUG:boto:Params: {} 2023-09-22T21:25:07.136 DEBUG:boto:Token: None 2023-09-22T21:25:07.137 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/?versions 2023-09-22T21:25:07.137 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P23E/B7rB60fQ9nPXZGa8pF78ZQ= 2023-09-22T21:25:07.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P23E/B7rB60fQ9nPXZGa8pF78ZQ=', 'Content-Length': '0'} 2023-09-22T21:25:07.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014856555316a61ea-00650e0633-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.139 DEBUG:boto:b'ffzdqu-55999falseanulltrue2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:25:07.140 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-22T21:25:07.140 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-22T21:25:07.140 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-22T21:25:07.140 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-22T21:25:07.140 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:25:07.140 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:07.141 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:07.141 DEBUG:boto:path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:25:07.141 DEBUG:boto:auth_path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:25:07.141 DEBUG:boto:Method: GET 2023-09-22T21:25:07.141 DEBUG:boto:Path: /ffzdqu-55/?versions&max-keys=999 2023-09-22T21:25:07.141 DEBUG:boto:Data: 2023-09-22T21:25:07.141 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.141 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.141 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.141 DEBUG:boto:Params: {} 2023-09-22T21:25:07.142 DEBUG:boto:Token: None 2023-09-22T21:25:07.142 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/?versions 2023-09-22T21:25:07.142 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P23E/B7rB60fQ9nPXZGa8pF78ZQ= 2023-09-22T21:25:07.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P23E/B7rB60fQ9nPXZGa8pF78ZQ=', 'Content-Length': '0'} 2023-09-22T21:25:07.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba84921941181461-00650e0633-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.145 DEBUG:boto:b'ffzdqu-55999falseanulltrue2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:25:07.145 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-22T21:25:07.145 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-22T21:25:07.146 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-22T21:25:07.146 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-22T21:25:07.146 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:07.146 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:07.146 DEBUG:boto:path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:25:07.146 DEBUG:boto:auth_path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:25:07.146 DEBUG:boto:Method: GET 2023-09-22T21:25:07.146 DEBUG:boto:Path: /ffzdqu-55/?versions&max-keys=999 2023-09-22T21:25:07.146 DEBUG:boto:Data: 2023-09-22T21:25:07.146 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.147 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.147 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.147 DEBUG:boto:Params: {} 2023-09-22T21:25:07.147 DEBUG:boto:Token: None 2023-09-22T21:25:07.147 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/?versions 2023-09-22T21:25:07.147 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P23E/B7rB60fQ9nPXZGa8pF78ZQ= 2023-09-22T21:25:07.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P23E/B7rB60fQ9nPXZGa8pF78ZQ=', 'Content-Length': '0'} 2023-09-22T21:25:07.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd12a9cd344c0d08-00650e0633-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.150 DEBUG:boto:b'ffzdqu-55999falseanulltrue2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:25:07.150 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:07.151 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:07.151 DEBUG:boto:path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:25:07.151 DEBUG:boto:auth_path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:25:07.151 DEBUG:boto:Method: GET 2023-09-22T21:25:07.151 DEBUG:boto:Path: /ffzdqu-55/?versions&max-keys=999 2023-09-22T21:25:07.151 DEBUG:boto:Data: 2023-09-22T21:25:07.151 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.151 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.151 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.152 DEBUG:boto:Params: {} 2023-09-22T21:25:07.152 DEBUG:boto:Token: None 2023-09-22T21:25:07.152 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/?versions 2023-09-22T21:25:07.152 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P23E/B7rB60fQ9nPXZGa8pF78ZQ= 2023-09-22T21:25:07.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P23E/B7rB60fQ9nPXZGa8pF78ZQ=', 'Content-Length': '0'} 2023-09-22T21:25:07.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000669eb013ddfa91a3-00650e0633-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.155 DEBUG:boto:b'ffzdqu-55999falseanulltrue2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:25:07.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:25:07.155 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:25:07.155 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:25:07.156 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.156 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.156 DEBUG:boto:Method: GET 2023-09-22T21:25:07.156 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.156 DEBUG:boto:Data: 2023-09-22T21:25:07.156 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.156 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.156 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.156 DEBUG:boto:Params: {} 2023-09-22T21:25:07.156 DEBUG:boto:Token: None 2023-09-22T21:25:07.157 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.157 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+Loc9vD1vZphyXdXmOZu/kkJH+Q= 2023-09-22T21:25:07.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+Loc9vD1vZphyXdXmOZu/kkJH+Q=', 'Content-Length': '0'} 2023-09-22T21:25:07.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a69eaae8db5518cd-00650e0633-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.202 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:25:07.203 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:25:07.203 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.203 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.203 DEBUG:boto:Method: GET 2023-09-22T21:25:07.203 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.204 DEBUG:boto:Data: 2023-09-22T21:25:07.204 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.204 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.204 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.204 DEBUG:boto:Params: {} 2023-09-22T21:25:07.204 DEBUG:boto:Token: None 2023-09-22T21:25:07.204 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.204 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+Loc9vD1vZphyXdXmOZu/kkJH+Q= 2023-09-22T21:25:07.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+Loc9vD1vZphyXdXmOZu/kkJH+Q=', 'Content-Length': '0'} 2023-09-22T21:25:07.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000428a9ec2ab50b692-00650e0633-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.207 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:25:07.208 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:25:07.208 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.208 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.208 DEBUG:boto:Method: HEAD 2023-09-22T21:25:07.208 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.208 DEBUG:boto:Data: 2023-09-22T21:25:07.208 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.208 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.208 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.209 DEBUG:boto:Params: {} 2023-09-22T21:25:07.209 DEBUG:boto:Token: None 2023-09-22T21:25:07.209 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.209 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XhvWvueih/XZk6auN8+AL747wKE= 2023-09-22T21:25:07.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XhvWvueih/XZk6auN8+AL747wKE=', 'Content-Length': '0'} 2023-09-22T21:25:07.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039248434b436143b-00650e0633-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.212 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:25:07.212 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:25:07.212 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.212 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.212 DEBUG:boto:Method: HEAD 2023-09-22T21:25:07.212 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.212 DEBUG:boto:Data: 2023-09-22T21:25:07.212 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.212 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.212 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.213 DEBUG:boto:Params: {} 2023-09-22T21:25:07.213 DEBUG:boto:Token: None 2023-09-22T21:25:07.213 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.213 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XhvWvueih/XZk6auN8+AL747wKE= 2023-09-22T21:25:07.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XhvWvueih/XZk6auN8+AL747wKE=', 'Content-Length': '0'} 2023-09-22T21:25:07.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fe7610a3f8adf52-00650e0633-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:25:07.216 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:25:07.216 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:25:07.216 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.216 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.216 DEBUG:boto:Method: GET 2023-09-22T21:25:07.216 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.217 DEBUG:boto:Data: 2023-09-22T21:25:07.217 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.217 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.217 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.217 DEBUG:boto:Params: {} 2023-09-22T21:25:07.217 DEBUG:boto:Token: None 2023-09-22T21:25:07.217 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.217 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+Loc9vD1vZphyXdXmOZu/kkJH+Q= 2023-09-22T21:25:07.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+Loc9vD1vZphyXdXmOZu/kkJH+Q=', 'Content-Length': '0'} 2023-09-22T21:25:07.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab2af886779b6724-00650e0633-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.220 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:25:07.220 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:25:07.220 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.221 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.221 DEBUG:boto:Method: GET 2023-09-22T21:25:07.221 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.221 DEBUG:boto:Data: 2023-09-22T21:25:07.221 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.221 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.221 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.221 DEBUG:boto:Params: {} 2023-09-22T21:25:07.221 DEBUG:boto:Token: None 2023-09-22T21:25:07.222 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:25:07.222 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+Loc9vD1vZphyXdXmOZu/kkJH+Q= 2023-09-22T21:25:07.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+Loc9vD1vZphyXdXmOZu/kkJH+Q=', 'Content-Length': '0'} 2023-09-22T21:25:07.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006a36e560cb42f51-00650e0633-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.266 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:07.267 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:07.267 DEBUG:boto:path=/ffzdqu-55/?prefix=a 2023-09-22T21:25:07.267 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=a 2023-09-22T21:25:07.267 DEBUG:boto:Method: GET 2023-09-22T21:25:07.267 DEBUG:boto:Path: /ffzdqu-55/?prefix=a 2023-09-22T21:25:07.267 DEBUG:boto:Data: 2023-09-22T21:25:07.267 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.267 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.267 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.267 DEBUG:boto:Params: {} 2023-09-22T21:25:07.268 DEBUG:boto:Token: None 2023-09-22T21:25:07.268 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/ 2023-09-22T21:25:07.268 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0= 2023-09-22T21:25:07.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0=', 'Content-Length': '0'} 2023-09-22T21:25:07.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c045245af59a690-00650e0633-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.273 DEBUG:boto:b'ffzdqu-55a1000falsea2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:25:07.274 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:07.274 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:07.274 DEBUG:boto:path=/ffzdqu-55/?prefix=a 2023-09-22T21:25:07.274 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=a 2023-09-22T21:25:07.274 DEBUG:boto:Method: GET 2023-09-22T21:25:07.274 DEBUG:boto:Path: /ffzdqu-55/?prefix=a 2023-09-22T21:25:07.275 DEBUG:boto:Data: 2023-09-22T21:25:07.275 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.275 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.275 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.275 DEBUG:boto:Params: {} 2023-09-22T21:25:07.275 DEBUG:boto:Token: None 2023-09-22T21:25:07.275 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/ 2023-09-22T21:25:07.275 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0= 2023-09-22T21:25:07.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0=', 'Content-Length': '0'} 2023-09-22T21:25:07.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2811053a4ad3d84-00650e0633-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.281 DEBUG:boto:b'ffzdqu-55a1000falsea2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:25:07.281 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:25:07.282 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:25:07.282 DEBUG:boto:Method: HEAD 2023-09-22T21:25:07.282 DEBUG:boto:Path: /ffzdqu-55/a 2023-09-22T21:25:07.282 DEBUG:boto:Data: 2023-09-22T21:25:07.282 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.282 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.282 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.282 DEBUG:boto:Params: {} 2023-09-22T21:25:07.283 DEBUG:boto:Token: None 2023-09-22T21:25:07.283 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/a 2023-09-22T21:25:07.283 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vP2oo04TGutUGZ39lh8LzIOlHHM= 2023-09-22T21:25:07.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vP2oo04TGutUGZ39lh8LzIOlHHM=', 'Content-Length': '0'} 2023-09-22T21:25:07.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097b72c02167a25bd-00650e0633-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.286 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:25:07.286 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:25:07.286 DEBUG:boto:Method: HEAD 2023-09-22T21:25:07.286 DEBUG:boto:Path: /ffzdqu-55/a 2023-09-22T21:25:07.286 DEBUG:boto:Data: 2023-09-22T21:25:07.286 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.286 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.287 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.287 DEBUG:boto:Params: {} 2023-09-22T21:25:07.287 DEBUG:boto:Token: None 2023-09-22T21:25:07.287 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/a 2023-09-22T21:25:07.287 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vP2oo04TGutUGZ39lh8LzIOlHHM= 2023-09-22T21:25:07.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vP2oo04TGutUGZ39lh8LzIOlHHM=', 'Content-Length': '0'} 2023-09-22T21:25:07.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adf45ef0553b42a4-00650e0633-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:25:07.290 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:25:07.290 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:25:07.291 DEBUG:boto:Method: GET 2023-09-22T21:25:07.291 DEBUG:boto:Path: /ffzdqu-55/a 2023-09-22T21:25:07.291 DEBUG:boto:Data: 2023-09-22T21:25:07.291 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.291 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.291 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.291 DEBUG:boto:Params: {} 2023-09-22T21:25:07.291 DEBUG:boto:Token: None 2023-09-22T21:25:07.291 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/a 2023-09-22T21:25:07.291 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8tJOewt6tTfrjrBtRK85vBldobU= 2023-09-22T21:25:07.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8tJOewt6tTfrjrBtRK85vBldobU=', 'Content-Length': '0'} 2023-09-22T21:25:07.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000576553351cdf8c8f-00650e0633-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.338 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:25:07.339 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:25:07.339 DEBUG:boto:Method: GET 2023-09-22T21:25:07.339 DEBUG:boto:Path: /ffzdqu-55/a 2023-09-22T21:25:07.339 DEBUG:boto:Data: 2023-09-22T21:25:07.339 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.339 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.339 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.339 DEBUG:boto:Params: {} 2023-09-22T21:25:07.340 DEBUG:boto:Token: None 2023-09-22T21:25:07.340 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/a 2023-09-22T21:25:07.340 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8tJOewt6tTfrjrBtRK85vBldobU= 2023-09-22T21:25:07.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8tJOewt6tTfrjrBtRK85vBldobU=', 'Content-Length': '0'} 2023-09-22T21:25:07.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f64af8980ee572b4-00650e0633-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.343 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:07.343 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:07.343 DEBUG:boto:path=/ffzdqu-55/?prefix=a 2023-09-22T21:25:07.344 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=a 2023-09-22T21:25:07.344 DEBUG:boto:Method: GET 2023-09-22T21:25:07.344 DEBUG:boto:Path: /ffzdqu-55/?prefix=a 2023-09-22T21:25:07.344 DEBUG:boto:Data: 2023-09-22T21:25:07.344 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.344 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.344 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.344 DEBUG:boto:Params: {} 2023-09-22T21:25:07.344 DEBUG:boto:Token: None 2023-09-22T21:25:07.345 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/ 2023-09-22T21:25:07.345 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0= 2023-09-22T21:25:07.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0=', 'Content-Length': '0'} 2023-09-22T21:25:07.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073313fe97ff2b465-00650e0633-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.350 DEBUG:boto:b'ffzdqu-55a1000falsea2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:25:07.351 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:07.351 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:07.351 DEBUG:boto:path=/ffzdqu-55/?prefix=a 2023-09-22T21:25:07.351 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=a 2023-09-22T21:25:07.351 DEBUG:boto:Method: GET 2023-09-22T21:25:07.351 DEBUG:boto:Path: /ffzdqu-55/?prefix=a 2023-09-22T21:25:07.352 DEBUG:boto:Data: 2023-09-22T21:25:07.352 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.352 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.352 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.352 DEBUG:boto:Params: {} 2023-09-22T21:25:07.352 DEBUG:boto:Token: None 2023-09-22T21:25:07.352 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/ 2023-09-22T21:25:07.352 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0= 2023-09-22T21:25:07.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0=', 'Content-Length': '0'} 2023-09-22T21:25:07.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5a27aea85e03e5f-00650e0633-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.398 DEBUG:boto:b'ffzdqu-55a1000falsea2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:25:07.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:25:07.399 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:25:07.399 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:25:07.399 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.399 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.400 DEBUG:boto:Method: GET 2023-09-22T21:25:07.400 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.400 DEBUG:boto:Data: 2023-09-22T21:25:07.400 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.400 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.400 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.400 DEBUG:boto:Params: {} 2023-09-22T21:25:07.400 DEBUG:boto:Token: None 2023-09-22T21:25:07.400 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.401 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:35i7qITKSOi5IijhYX/iwHCrx4k= 2023-09-22T21:25:07.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:35i7qITKSOi5IijhYX/iwHCrx4k=', 'Content-Length': '0'} 2023-09-22T21:25:07.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f87fa5c526c684a-00650e0633-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.404 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:25:07.404 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:25:07.404 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.404 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.405 DEBUG:boto:Method: GET 2023-09-22T21:25:07.405 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.405 DEBUG:boto:Data: 2023-09-22T21:25:07.405 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.405 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.405 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.405 DEBUG:boto:Params: {} 2023-09-22T21:25:07.405 DEBUG:boto:Token: None 2023-09-22T21:25:07.405 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.406 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:35i7qITKSOi5IijhYX/iwHCrx4k= 2023-09-22T21:25:07.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:35i7qITKSOi5IijhYX/iwHCrx4k=', 'Content-Length': '0'} 2023-09-22T21:25:07.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8d83c5ad5677238-00650e0633-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.409 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:25:07.409 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:25:07.409 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.409 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.409 DEBUG:boto:Method: HEAD 2023-09-22T21:25:07.409 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.409 DEBUG:boto:Data: 2023-09-22T21:25:07.410 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.410 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.410 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.410 DEBUG:boto:Params: {} 2023-09-22T21:25:07.410 DEBUG:boto:Token: None 2023-09-22T21:25:07.410 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.410 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:O1yLEPCgAfo1wwr7QsjXM1NbTcA= 2023-09-22T21:25:07.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:O1yLEPCgAfo1wwr7QsjXM1NbTcA=', 'Content-Length': '0'} 2023-09-22T21:25:07.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c325b77735cc460e-00650e0633-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.413 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:25:07.413 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:25:07.413 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.413 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.413 DEBUG:boto:Method: HEAD 2023-09-22T21:25:07.414 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.414 DEBUG:boto:Data: 2023-09-22T21:25:07.414 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.414 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.414 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.414 DEBUG:boto:Params: {} 2023-09-22T21:25:07.414 DEBUG:boto:Token: None 2023-09-22T21:25:07.414 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.415 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:O1yLEPCgAfo1wwr7QsjXM1NbTcA= 2023-09-22T21:25:07.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:O1yLEPCgAfo1wwr7QsjXM1NbTcA=', 'Content-Length': '0'} 2023-09-22T21:25:07.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000498ca3eaba89b352-00650e0633-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:25:07.418 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:25:07.418 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:25:07.418 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.418 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.418 DEBUG:boto:Method: GET 2023-09-22T21:25:07.418 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.418 DEBUG:boto:Data: 2023-09-22T21:25:07.419 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.419 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.419 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.419 DEBUG:boto:Params: {} 2023-09-22T21:25:07.419 DEBUG:boto:Token: None 2023-09-22T21:25:07.419 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.419 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:35i7qITKSOi5IijhYX/iwHCrx4k= 2023-09-22T21:25:07.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:35i7qITKSOi5IijhYX/iwHCrx4k=', 'Content-Length': '0'} 2023-09-22T21:25:07.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce085068cf905d2d-00650e0633-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.462 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:25:07.462 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:25:07.463 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.463 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.463 DEBUG:boto:Method: GET 2023-09-22T21:25:07.463 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.463 DEBUG:boto:Data: 2023-09-22T21:25:07.463 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.463 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.463 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.463 DEBUG:boto:Params: {} 2023-09-22T21:25:07.464 DEBUG:boto:Token: None 2023-09-22T21:25:07.464 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:25:07.464 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:35i7qITKSOi5IijhYX/iwHCrx4k= 2023-09-22T21:25:07.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:35i7qITKSOi5IijhYX/iwHCrx4k=', 'Content-Length': '0'} 2023-09-22T21:25:07.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004be6bc0302e93651-00650e0633-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.467 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:07.467 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:07.467 DEBUG:boto:path=/ffzdqu-55/?prefix=b 2023-09-22T21:25:07.468 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=b 2023-09-22T21:25:07.468 DEBUG:boto:Method: GET 2023-09-22T21:25:07.468 DEBUG:boto:Path: /ffzdqu-55/?prefix=b 2023-09-22T21:25:07.468 DEBUG:boto:Data: 2023-09-22T21:25:07.468 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.468 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.468 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.468 DEBUG:boto:Params: {} 2023-09-22T21:25:07.468 DEBUG:boto:Token: None 2023-09-22T21:25:07.469 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/ 2023-09-22T21:25:07.469 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0= 2023-09-22T21:25:07.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0=', 'Content-Length': '0'} 2023-09-22T21:25:07.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7d782e54c4ac523-00650e0633-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.475 DEBUG:boto:b'ffzdqu-55b1000falseb2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:25:07.476 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:07.476 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:07.476 DEBUG:boto:path=/ffzdqu-55/?prefix=b 2023-09-22T21:25:07.476 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=b 2023-09-22T21:25:07.476 DEBUG:boto:Method: GET 2023-09-22T21:25:07.476 DEBUG:boto:Path: /ffzdqu-55/?prefix=b 2023-09-22T21:25:07.476 DEBUG:boto:Data: 2023-09-22T21:25:07.477 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.477 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.477 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.477 DEBUG:boto:Params: {} 2023-09-22T21:25:07.477 DEBUG:boto:Token: None 2023-09-22T21:25:07.477 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/ 2023-09-22T21:25:07.477 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0= 2023-09-22T21:25:07.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0=', 'Content-Length': '0'} 2023-09-22T21:25:07.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2a4b52d4c551a06-00650e0633-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.484 DEBUG:boto:b'ffzdqu-55b1000falseb2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:25:07.484 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:25:07.484 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:25:07.484 DEBUG:boto:Method: HEAD 2023-09-22T21:25:07.485 DEBUG:boto:Path: /ffzdqu-55/b 2023-09-22T21:25:07.485 DEBUG:boto:Data: 2023-09-22T21:25:07.485 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.485 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.485 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.485 DEBUG:boto:Params: {} 2023-09-22T21:25:07.485 DEBUG:boto:Token: None 2023-09-22T21:25:07.485 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/b 2023-09-22T21:25:07.485 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1gE41XLK2M7Buh/pzMmPgRRXzHI= 2023-09-22T21:25:07.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1gE41XLK2M7Buh/pzMmPgRRXzHI=', 'Content-Length': '0'} 2023-09-22T21:25:07.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f723d138c46e751d-00650e0633-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.488 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:25:07.488 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:25:07.488 DEBUG:boto:Method: HEAD 2023-09-22T21:25:07.489 DEBUG:boto:Path: /ffzdqu-55/b 2023-09-22T21:25:07.489 DEBUG:boto:Data: 2023-09-22T21:25:07.489 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.489 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.489 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.489 DEBUG:boto:Params: {} 2023-09-22T21:25:07.489 DEBUG:boto:Token: None 2023-09-22T21:25:07.489 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/b 2023-09-22T21:25:07.489 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1gE41XLK2M7Buh/pzMmPgRRXzHI= 2023-09-22T21:25:07.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1gE41XLK2M7Buh/pzMmPgRRXzHI=', 'Content-Length': '0'} 2023-09-22T21:25:07.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d708f5f53e772686-00650e0633-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:25:07.492 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:25:07.492 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:25:07.492 DEBUG:boto:Method: GET 2023-09-22T21:25:07.493 DEBUG:boto:Path: /ffzdqu-55/b 2023-09-22T21:25:07.493 DEBUG:boto:Data: 2023-09-22T21:25:07.493 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.493 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.493 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.493 DEBUG:boto:Params: {} 2023-09-22T21:25:07.493 DEBUG:boto:Token: None 2023-09-22T21:25:07.493 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/b 2023-09-22T21:25:07.494 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:a3mjJEQEbDwHO8uXT4dkg2CsZ0s= 2023-09-22T21:25:07.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:a3mjJEQEbDwHO8uXT4dkg2CsZ0s=', 'Content-Length': '0'} 2023-09-22T21:25:07.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab95757714529696-00650e0633-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.538 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:25:07.538 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:25:07.539 DEBUG:boto:Method: GET 2023-09-22T21:25:07.539 DEBUG:boto:Path: /ffzdqu-55/b 2023-09-22T21:25:07.539 DEBUG:boto:Data: 2023-09-22T21:25:07.539 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.539 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.539 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.539 DEBUG:boto:Params: {} 2023-09-22T21:25:07.539 DEBUG:boto:Token: None 2023-09-22T21:25:07.540 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/b 2023-09-22T21:25:07.540 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:a3mjJEQEbDwHO8uXT4dkg2CsZ0s= 2023-09-22T21:25:07.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:a3mjJEQEbDwHO8uXT4dkg2CsZ0s=', 'Content-Length': '0'} 2023-09-22T21:25:07.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a3135b620b6fb4e-00650e0633-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.543 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:07.543 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:07.543 DEBUG:boto:path=/ffzdqu-55/?prefix=b 2023-09-22T21:25:07.543 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=b 2023-09-22T21:25:07.543 DEBUG:boto:Method: GET 2023-09-22T21:25:07.543 DEBUG:boto:Path: /ffzdqu-55/?prefix=b 2023-09-22T21:25:07.544 DEBUG:boto:Data: 2023-09-22T21:25:07.544 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.544 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.544 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.544 DEBUG:boto:Params: {} 2023-09-22T21:25:07.544 DEBUG:boto:Token: None 2023-09-22T21:25:07.544 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/ 2023-09-22T21:25:07.544 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0= 2023-09-22T21:25:07.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0=', 'Content-Length': '0'} 2023-09-22T21:25:07.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acdc211c1c95a497-00650e0633-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.550 DEBUG:boto:b'ffzdqu-55b1000falseb2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:25:07.551 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:07.551 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:07.551 DEBUG:boto:path=/ffzdqu-55/?prefix=b 2023-09-22T21:25:07.551 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=b 2023-09-22T21:25:07.551 DEBUG:boto:Method: GET 2023-09-22T21:25:07.551 DEBUG:boto:Path: /ffzdqu-55/?prefix=b 2023-09-22T21:25:07.551 DEBUG:boto:Data: 2023-09-22T21:25:07.552 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.552 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.552 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.552 DEBUG:boto:Params: {} 2023-09-22T21:25:07.552 DEBUG:boto:Token: None 2023-09-22T21:25:07.552 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/ 2023-09-22T21:25:07.552 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0= 2023-09-22T21:25:07.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0=', 'Content-Length': '0'} 2023-09-22T21:25:07.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000914350396f84fdaa-00650e0633-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.560 DEBUG:boto:b'ffzdqu-55b1000falseb2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:25:07.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:25:07.560 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:25:07.560 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:25:07.561 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.561 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.561 DEBUG:boto:Method: GET 2023-09-22T21:25:07.561 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.561 DEBUG:boto:Data: 2023-09-22T21:25:07.561 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.561 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.561 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.561 DEBUG:boto:Params: {} 2023-09-22T21:25:07.561 DEBUG:boto:Token: None 2023-09-22T21:25:07.562 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.562 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pj43EmZiGHRYcrcauoZAHJLuDRo= 2023-09-22T21:25:07.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pj43EmZiGHRYcrcauoZAHJLuDRo=', 'Content-Length': '0'} 2023-09-22T21:25:07.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8d6bdf9d14ca30d-00650e0633-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.565 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:25:07.565 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:25:07.565 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.565 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.566 DEBUG:boto:Method: GET 2023-09-22T21:25:07.566 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.566 DEBUG:boto:Data: 2023-09-22T21:25:07.566 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.566 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.566 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.566 DEBUG:boto:Params: {} 2023-09-22T21:25:07.566 DEBUG:boto:Token: None 2023-09-22T21:25:07.566 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.567 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pj43EmZiGHRYcrcauoZAHJLuDRo= 2023-09-22T21:25:07.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pj43EmZiGHRYcrcauoZAHJLuDRo=', 'Content-Length': '0'} 2023-09-22T21:25:07.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008529ef2c73853c7f-00650e0633-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.570 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:25:07.570 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:25:07.570 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.570 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.570 DEBUG:boto:Method: HEAD 2023-09-22T21:25:07.571 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.571 DEBUG:boto:Data: 2023-09-22T21:25:07.571 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.571 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.571 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.571 DEBUG:boto:Params: {} 2023-09-22T21:25:07.571 DEBUG:boto:Token: None 2023-09-22T21:25:07.571 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.571 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GLXDlV7MI1KJyhrZZH4YVK7pExw= 2023-09-22T21:25:07.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GLXDlV7MI1KJyhrZZH4YVK7pExw=', 'Content-Length': '0'} 2023-09-22T21:25:07.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e2e2c78da1c2d2d-00650e0633-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.574 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:25:07.575 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:25:07.575 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.575 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.575 DEBUG:boto:Method: HEAD 2023-09-22T21:25:07.575 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.575 DEBUG:boto:Data: 2023-09-22T21:25:07.575 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.575 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.575 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.576 DEBUG:boto:Params: {} 2023-09-22T21:25:07.576 DEBUG:boto:Token: None 2023-09-22T21:25:07.576 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.576 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GLXDlV7MI1KJyhrZZH4YVK7pExw= 2023-09-22T21:25:07.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GLXDlV7MI1KJyhrZZH4YVK7pExw=', 'Content-Length': '0'} 2023-09-22T21:25:07.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d85ffd81aa81e04-00650e0633-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:25:07.579 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:25:07.579 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:25:07.579 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.579 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.579 DEBUG:boto:Method: GET 2023-09-22T21:25:07.580 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.580 DEBUG:boto:Data: 2023-09-22T21:25:07.580 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.580 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.580 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.580 DEBUG:boto:Params: {} 2023-09-22T21:25:07.580 DEBUG:boto:Token: None 2023-09-22T21:25:07.580 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.580 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pj43EmZiGHRYcrcauoZAHJLuDRo= 2023-09-22T21:25:07.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pj43EmZiGHRYcrcauoZAHJLuDRo=', 'Content-Length': '0'} 2023-09-22T21:25:07.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0b158ddb3bfba8b-00650e0633-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.630 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:25:07.631 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:25:07.631 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.631 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.631 DEBUG:boto:Method: GET 2023-09-22T21:25:07.631 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.631 DEBUG:boto:Data: 2023-09-22T21:25:07.631 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.631 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.631 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.632 DEBUG:boto:Params: {} 2023-09-22T21:25:07.632 DEBUG:boto:Token: None 2023-09-22T21:25:07.632 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:25:07.632 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pj43EmZiGHRYcrcauoZAHJLuDRo= 2023-09-22T21:25:07.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pj43EmZiGHRYcrcauoZAHJLuDRo=', 'Content-Length': '0'} 2023-09-22T21:25:07.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe76c556d554674c-00650e0633-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.636 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:07.636 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:07.636 DEBUG:boto:path=/ffzdqu-55/?prefix=c 2023-09-22T21:25:07.636 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=c 2023-09-22T21:25:07.636 DEBUG:boto:Method: GET 2023-09-22T21:25:07.636 DEBUG:boto:Path: /ffzdqu-55/?prefix=c 2023-09-22T21:25:07.636 DEBUG:boto:Data: 2023-09-22T21:25:07.637 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.637 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.637 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.637 DEBUG:boto:Params: {} 2023-09-22T21:25:07.637 DEBUG:boto:Token: None 2023-09-22T21:25:07.637 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/ 2023-09-22T21:25:07.637 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0= 2023-09-22T21:25:07.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0=', 'Content-Length': '0'} 2023-09-22T21:25:07.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070e110a4cc94fab9-00650e0633-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.686 DEBUG:boto:b'ffzdqu-55c1000falsec2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:25:07.687 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:07.687 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:07.687 DEBUG:boto:path=/ffzdqu-55/?prefix=c 2023-09-22T21:25:07.687 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=c 2023-09-22T21:25:07.687 DEBUG:boto:Method: GET 2023-09-22T21:25:07.687 DEBUG:boto:Path: /ffzdqu-55/?prefix=c 2023-09-22T21:25:07.688 DEBUG:boto:Data: 2023-09-22T21:25:07.688 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.688 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.688 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.688 DEBUG:boto:Params: {} 2023-09-22T21:25:07.688 DEBUG:boto:Token: None 2023-09-22T21:25:07.688 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/ 2023-09-22T21:25:07.688 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0= 2023-09-22T21:25:07.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0=', 'Content-Length': '0'} 2023-09-22T21:25:07.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000989b3df38d966e4-00650e0633-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.694 DEBUG:boto:b'ffzdqu-55c1000falsec2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:25:07.695 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:25:07.695 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:25:07.695 DEBUG:boto:Method: HEAD 2023-09-22T21:25:07.695 DEBUG:boto:Path: /ffzdqu-55/c 2023-09-22T21:25:07.695 DEBUG:boto:Data: 2023-09-22T21:25:07.695 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.695 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.696 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.696 DEBUG:boto:Params: {} 2023-09-22T21:25:07.696 DEBUG:boto:Token: None 2023-09-22T21:25:07.696 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/c 2023-09-22T21:25:07.696 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zh1NXaUFAR3npNkevS75xTcoFv0= 2023-09-22T21:25:07.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zh1NXaUFAR3npNkevS75xTcoFv0=', 'Content-Length': '0'} 2023-09-22T21:25:07.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbebc385b73788ce-00650e0633-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.699 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:25:07.699 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:25:07.699 DEBUG:boto:Method: HEAD 2023-09-22T21:25:07.699 DEBUG:boto:Path: /ffzdqu-55/c 2023-09-22T21:25:07.700 DEBUG:boto:Data: 2023-09-22T21:25:07.700 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.700 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.700 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.700 DEBUG:boto:Params: {} 2023-09-22T21:25:07.700 DEBUG:boto:Token: None 2023-09-22T21:25:07.700 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/c 2023-09-22T21:25:07.700 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zh1NXaUFAR3npNkevS75xTcoFv0= 2023-09-22T21:25:07.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zh1NXaUFAR3npNkevS75xTcoFv0=', 'Content-Length': '0'} 2023-09-22T21:25:07.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dafb80c47f9b3768-00650e0633-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:25:07.703 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:25:07.703 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:25:07.704 DEBUG:boto:Method: GET 2023-09-22T21:25:07.704 DEBUG:boto:Path: /ffzdqu-55/c 2023-09-22T21:25:07.704 DEBUG:boto:Data: 2023-09-22T21:25:07.704 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.704 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.704 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.704 DEBUG:boto:Params: {} 2023-09-22T21:25:07.704 DEBUG:boto:Token: None 2023-09-22T21:25:07.705 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/c 2023-09-22T21:25:07.705 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YhFj1OOVrcC92nOQ0yfxWweDWeM= 2023-09-22T21:25:07.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YhFj1OOVrcC92nOQ0yfxWweDWeM=', 'Content-Length': '0'} 2023-09-22T21:25:07.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055fa39ee65fb293a-00650e0633-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.750 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:25:07.750 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:25:07.751 DEBUG:boto:Method: GET 2023-09-22T21:25:07.751 DEBUG:boto:Path: /ffzdqu-55/c 2023-09-22T21:25:07.751 DEBUG:boto:Data: 2023-09-22T21:25:07.751 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.751 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.751 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.751 DEBUG:boto:Params: {} 2023-09-22T21:25:07.751 DEBUG:boto:Token: None 2023-09-22T21:25:07.752 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/c 2023-09-22T21:25:07.752 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YhFj1OOVrcC92nOQ0yfxWweDWeM= 2023-09-22T21:25:07.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YhFj1OOVrcC92nOQ0yfxWweDWeM=', 'Content-Length': '0'} 2023-09-22T21:25:07.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b794b113a79b3547-00650e0633-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.755 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:07.755 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:07.756 DEBUG:boto:path=/ffzdqu-55/?prefix=c 2023-09-22T21:25:07.756 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=c 2023-09-22T21:25:07.756 DEBUG:boto:Method: GET 2023-09-22T21:25:07.756 DEBUG:boto:Path: /ffzdqu-55/?prefix=c 2023-09-22T21:25:07.756 DEBUG:boto:Data: 2023-09-22T21:25:07.756 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.756 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.756 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.756 DEBUG:boto:Params: {} 2023-09-22T21:25:07.756 DEBUG:boto:Token: None 2023-09-22T21:25:07.757 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/ 2023-09-22T21:25:07.757 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0= 2023-09-22T21:25:07.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0=', 'Content-Length': '0'} 2023-09-22T21:25:07.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b56de5ec2e6f4327-00650e0633-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.810 DEBUG:boto:b'ffzdqu-55c1000falsec2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:25:07.811 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:07.811 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:07.811 DEBUG:boto:path=/ffzdqu-55/?prefix=c 2023-09-22T21:25:07.811 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=c 2023-09-22T21:25:07.811 DEBUG:boto:Method: GET 2023-09-22T21:25:07.811 DEBUG:boto:Path: /ffzdqu-55/?prefix=c 2023-09-22T21:25:07.811 DEBUG:boto:Data: 2023-09-22T21:25:07.812 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.812 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.812 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.812 DEBUG:boto:Params: {} 2023-09-22T21:25:07.812 DEBUG:boto:Token: None 2023-09-22T21:25:07.812 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/ 2023-09-22T21:25:07.812 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0= 2023-09-22T21:25:07.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0=', 'Content-Length': '0'} 2023-09-22T21:25:07.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d05d850c16207c6-00650e0633-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.819 DEBUG:boto:b'ffzdqu-55c1000falsec2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:25:07.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:25:07.819 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:25:07.819 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:25:07.819 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.820 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.820 DEBUG:boto:Method: GET 2023-09-22T21:25:07.820 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.820 DEBUG:boto:Data: 2023-09-22T21:25:07.820 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.820 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.820 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.820 DEBUG:boto:Params: {} 2023-09-22T21:25:07.820 DEBUG:boto:Token: None 2023-09-22T21:25:07.821 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.821 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CkCMuhMCJIXk857DOX8xcMMPy2Y= 2023-09-22T21:25:07.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CkCMuhMCJIXk857DOX8xcMMPy2Y=', 'Content-Length': '0'} 2023-09-22T21:25:07.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d493355ac6de67d7-00650e0633-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.824 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:25:07.824 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:25:07.824 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.824 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.825 DEBUG:boto:Method: GET 2023-09-22T21:25:07.825 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.825 DEBUG:boto:Data: 2023-09-22T21:25:07.825 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.825 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.825 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.825 DEBUG:boto:Params: {} 2023-09-22T21:25:07.825 DEBUG:boto:Token: None 2023-09-22T21:25:07.825 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.826 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CkCMuhMCJIXk857DOX8xcMMPy2Y= 2023-09-22T21:25:07.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CkCMuhMCJIXk857DOX8xcMMPy2Y=', 'Content-Length': '0'} 2023-09-22T21:25:07.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c0d15f60e05fc00-00650e0633-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.829 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:25:07.829 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:25:07.829 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.829 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.829 DEBUG:boto:Method: HEAD 2023-09-22T21:25:07.829 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.829 DEBUG:boto:Data: 2023-09-22T21:25:07.830 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.830 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.830 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.830 DEBUG:boto:Params: {} 2023-09-22T21:25:07.830 DEBUG:boto:Token: None 2023-09-22T21:25:07.830 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.830 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6Ldv/eWwrktEftOH9MvoHjcKEng= 2023-09-22T21:25:07.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6Ldv/eWwrktEftOH9MvoHjcKEng=', 'Content-Length': '0'} 2023-09-22T21:25:07.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071bce02b8750cbfb-00650e0633-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.833 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:25:07.833 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:25:07.834 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.834 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.834 DEBUG:boto:Method: HEAD 2023-09-22T21:25:07.834 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.834 DEBUG:boto:Data: 2023-09-22T21:25:07.834 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.834 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.834 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.834 DEBUG:boto:Params: {} 2023-09-22T21:25:07.835 DEBUG:boto:Token: None 2023-09-22T21:25:07.835 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.835 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6Ldv/eWwrktEftOH9MvoHjcKEng= 2023-09-22T21:25:07.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6Ldv/eWwrktEftOH9MvoHjcKEng=', 'Content-Length': '0'} 2023-09-22T21:25:07.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ed3027283410684-00650e0633-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:25:07.838 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:25:07.838 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:25:07.838 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.838 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.838 DEBUG:boto:Method: GET 2023-09-22T21:25:07.839 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.839 DEBUG:boto:Data: 2023-09-22T21:25:07.839 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.839 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.839 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.839 DEBUG:boto:Params: {} 2023-09-22T21:25:07.839 DEBUG:boto:Token: None 2023-09-22T21:25:07.839 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.839 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CkCMuhMCJIXk857DOX8xcMMPy2Y= 2023-09-22T21:25:07.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CkCMuhMCJIXk857DOX8xcMMPy2Y=', 'Content-Length': '0'} 2023-09-22T21:25:07.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002664cbb0ad6232a-00650e0633-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.886 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:25:07.886 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:25:07.887 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.887 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.887 DEBUG:boto:Method: GET 2023-09-22T21:25:07.887 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.887 DEBUG:boto:Data: 2023-09-22T21:25:07.887 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.887 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.887 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.887 DEBUG:boto:Params: {} 2023-09-22T21:25:07.888 DEBUG:boto:Token: None 2023-09-22T21:25:07.888 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:25:07.888 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CkCMuhMCJIXk857DOX8xcMMPy2Y= 2023-09-22T21:25:07.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CkCMuhMCJIXk857DOX8xcMMPy2Y=', 'Content-Length': '0'} 2023-09-22T21:25:07.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1afa2c4dd12f175-00650e0633-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.892 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:07.892 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:07.892 DEBUG:boto:path=/ffzdqu-55/?prefix=d 2023-09-22T21:25:07.892 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=d 2023-09-22T21:25:07.892 DEBUG:boto:Method: GET 2023-09-22T21:25:07.892 DEBUG:boto:Path: /ffzdqu-55/?prefix=d 2023-09-22T21:25:07.892 DEBUG:boto:Data: 2023-09-22T21:25:07.892 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.892 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.893 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.893 DEBUG:boto:Params: {} 2023-09-22T21:25:07.893 DEBUG:boto:Token: None 2023-09-22T21:25:07.893 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/ 2023-09-22T21:25:07.893 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0= 2023-09-22T21:25:07.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0=', 'Content-Length': '0'} 2023-09-22T21:25:07.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027b73a973df2e0ae-00650e0633-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.942 DEBUG:boto:b'ffzdqu-55d1000falsed2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:25:07.943 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:07.943 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:07.943 DEBUG:boto:path=/ffzdqu-55/?prefix=d 2023-09-22T21:25:07.943 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=d 2023-09-22T21:25:07.943 DEBUG:boto:Method: GET 2023-09-22T21:25:07.943 DEBUG:boto:Path: /ffzdqu-55/?prefix=d 2023-09-22T21:25:07.943 DEBUG:boto:Data: 2023-09-22T21:25:07.944 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.944 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.944 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.944 DEBUG:boto:Params: {} 2023-09-22T21:25:07.944 DEBUG:boto:Token: None 2023-09-22T21:25:07.944 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/ 2023-09-22T21:25:07.944 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0= 2023-09-22T21:25:07.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKsFwx35KyYVWznptnANkOs4Ln0=', 'Content-Length': '0'} 2023-09-22T21:25:07.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000af4f5ad8487198a-00650e0633-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.950 DEBUG:boto:b'ffzdqu-55d1000falsed2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:25:07.950 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:25:07.950 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:25:07.951 DEBUG:boto:Method: HEAD 2023-09-22T21:25:07.951 DEBUG:boto:Path: /ffzdqu-55/d 2023-09-22T21:25:07.951 DEBUG:boto:Data: 2023-09-22T21:25:07.951 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.951 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.951 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.951 DEBUG:boto:Params: {} 2023-09-22T21:25:07.951 DEBUG:boto:Token: None 2023-09-22T21:25:07.952 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/d 2023-09-22T21:25:07.952 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5bWAdCAFbBp1yfVIjN49LpCoC3U= 2023-09-22T21:25:07.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5bWAdCAFbBp1yfVIjN49LpCoC3U=', 'Content-Length': '0'} 2023-09-22T21:25:07.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033974c3955be2d56-00650e0633-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.955 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:25:07.955 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:25:07.955 DEBUG:boto:Method: HEAD 2023-09-22T21:25:07.956 DEBUG:boto:Path: /ffzdqu-55/d 2023-09-22T21:25:07.956 DEBUG:boto:Data: 2023-09-22T21:25:07.956 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.956 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:07.956 DEBUG:boto:Port: 8000 2023-09-22T21:25:07.956 DEBUG:boto:Params: {} 2023-09-22T21:25:07.956 DEBUG:boto:Token: None 2023-09-22T21:25:07.956 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/d 2023-09-22T21:25:07.956 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5bWAdCAFbBp1yfVIjN49LpCoC3U= 2023-09-22T21:25:07.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5bWAdCAFbBp1yfVIjN49LpCoC3U=', 'Content-Length': '0'} 2023-09-22T21:25:07.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b46998fe2fd33b50-00650e0633-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:07.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:25:07.960 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:25:07.960 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:25:07.960 DEBUG:boto:Method: GET 2023-09-22T21:25:07.960 DEBUG:boto:Path: /ffzdqu-55/d 2023-09-22T21:25:07.960 DEBUG:boto:Data: 2023-09-22T21:25:07.960 DEBUG:boto:Headers: {} 2023-09-22T21:25:07.960 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:07.960 DEBUG:boto:Port: 8001 2023-09-22T21:25:07.960 DEBUG:boto:Params: {} 2023-09-22T21:25:07.961 DEBUG:boto:Token: None 2023-09-22T21:25:07.961 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:07 GMT /ffzdqu-55/d 2023-09-22T21:25:07.961 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wb2BP9rOQrjjjhV68sr85QMiM1Y= 2023-09-22T21:25:07.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wb2BP9rOQrjjjhV68sr85QMiM1Y=', 'Content-Length': '0'} 2023-09-22T21:25:07.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ae315f859e2d026-00650e0633-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:08.010 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:25:08.010 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:25:08.011 DEBUG:boto:Method: GET 2023-09-22T21:25:08.011 DEBUG:boto:Path: /ffzdqu-55/d 2023-09-22T21:25:08.011 DEBUG:boto:Data: 2023-09-22T21:25:08.011 DEBUG:boto:Headers: {} 2023-09-22T21:25:08.011 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:08.011 DEBUG:boto:Port: 8000 2023-09-22T21:25:08.011 DEBUG:boto:Params: {} 2023-09-22T21:25:08.011 DEBUG:boto:Token: None 2023-09-22T21:25:08.011 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:08 GMT /ffzdqu-55/d 2023-09-22T21:25:08.012 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hYhoX3bgelDSCs58EQJuMVYpwH4= 2023-09-22T21:25:08.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hYhoX3bgelDSCs58EQJuMVYpwH4=', 'Content-Length': '0'} 2023-09-22T21:25:08.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081fea272e0bddd7a-00650e0634-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:08.015 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:08.015 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:08.015 DEBUG:boto:path=/ffzdqu-55/?prefix=d 2023-09-22T21:25:08.015 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=d 2023-09-22T21:25:08.015 DEBUG:boto:Method: GET 2023-09-22T21:25:08.015 DEBUG:boto:Path: /ffzdqu-55/?prefix=d 2023-09-22T21:25:08.015 DEBUG:boto:Data: 2023-09-22T21:25:08.016 DEBUG:boto:Headers: {} 2023-09-22T21:25:08.016 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:25:08.016 DEBUG:boto:Port: 8001 2023-09-22T21:25:08.016 DEBUG:boto:Params: {} 2023-09-22T21:25:08.016 DEBUG:boto:Token: None 2023-09-22T21:25:08.016 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:08 GMT /ffzdqu-55/ 2023-09-22T21:25:08.016 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qM84zwdkmyuaBP8RZXBmlGXpkTw= 2023-09-22T21:25:08.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qM84zwdkmyuaBP8RZXBmlGXpkTw=', 'Content-Length': '0'} 2023-09-22T21:25:08.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000185d6fdcdb40c786-00650e0634-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:08.070 DEBUG:boto:b'ffzdqu-55d1000falsed2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:25:08.071 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:08.071 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:08.071 DEBUG:boto:path=/ffzdqu-55/?prefix=d 2023-09-22T21:25:08.071 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=d 2023-09-22T21:25:08.071 DEBUG:boto:Method: GET 2023-09-22T21:25:08.071 DEBUG:boto:Path: /ffzdqu-55/?prefix=d 2023-09-22T21:25:08.071 DEBUG:boto:Data: 2023-09-22T21:25:08.072 DEBUG:boto:Headers: {} 2023-09-22T21:25:08.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:08.072 DEBUG:boto:Port: 8000 2023-09-22T21:25:08.072 DEBUG:boto:Params: {} 2023-09-22T21:25:08.072 DEBUG:boto:Token: None 2023-09-22T21:25:08.072 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:25:08 GMT /ffzdqu-55/ 2023-09-22T21:25:08.072 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qM84zwdkmyuaBP8RZXBmlGXpkTw= 2023-09-22T21:25:08.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qM84zwdkmyuaBP8RZXBmlGXpkTw=', 'Content-Length': '0'} 2023-09-22T21:25:08.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005201d93a26b2d37a-00650e0634-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:08.078 DEBUG:boto:b'ffzdqu-55d1000falsed2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:25:08.079 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-55 zones={a2, a1} 2023-09-22T21:25:08.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'ffzdqu-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:25:08.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'ffzdqu-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:25:08.079 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket ffzdqu-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:25:08.119 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:25:08.120 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:25:08.175 INFO:teuthology.orchestra.run.smithi160.stdout:{"layout":{"resharding":"None","current_index":{"gen":0,"layout":{"type":"Normal","normal":{"num_shards":11,"hash_type":"Mod"}}},"logs":[{"gen":0,"layout":{"type":"InIndex","in_index":{"gen":0,"layout":{"num_shards":11,"hash_type":"Mod"}}}}]}} 2023-09-22T21:25:08.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'ffzdqu-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:25:08.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'ffzdqu-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:25:08.175 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket ffzdqu-55 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:25:08.255 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:25:08.255 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:25:08.310 INFO:teuthology.orchestra.run.smithi160.stdout:[{"op_id":"1#00000000001.103.6","op_tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.3357056847402969950","op":"write","object":"a","instance":"","state":"complete","index_ver":1,"timestamp":"2023-09-22T21:24:46.345308474Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20"}]},{"op_id":"5#00000000001.26.6","op_tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.12106257812065812486","op":"write","object":"d","instance":"","state":"complete","index_ver":1,"timestamp":"2023-09-22T21:24:46.428378516Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20"}]},{"op_id":"8#00000000001.29.6","op_tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.7562340742465387877","op":"write","object":"c","instance":"","state":"complete","index_ver":1,"timestamp":"2023-09-22T21:24:46.410382934Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20"}]},{"op_id":"10#00000000001.27.6","op_tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.9189931591446334976","op":"write","object":"b","instance":"","state":"complete","index_ver":1,"timestamp":"2023-09-22T21:24:46.360139813Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20"}]}] 2023-09-22T21:25:08.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'ffzdqu-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:25:08.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'ffzdqu-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:25:08.311 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket ffzdqu-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:25:08.388 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:25:08.389 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:25:08.437 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:25:08.438 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-55 2023-09-22T21:25:08.444 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 4 2023-09-22T21:25:08.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'ffzdqu-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:25:08.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'ffzdqu-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:25:08.490 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket ffzdqu-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:25:08.562 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:25:08.562 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:25:08.617 INFO:teuthology.orchestra.run.smithi160.stdout:{"layout":{"resharding":"None","current_index":{"gen":1,"layout":{"type":"Normal","normal":{"num_shards":3,"hash_type":"Mod"}}},"logs":[{"gen":0,"layout":{"type":"InIndex","in_index":{"gen":0,"layout":{"num_shards":11,"hash_type":"Mod"}}}},{"gen":1,"layout":{"type":"InIndex","in_index":{"gen":1,"layout":{"num_shards":3,"hash_type":"Mod"}}}}]}} 2023-09-22T21:25:08.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'ffzdqu-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:25:08.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'ffzdqu-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:25:08.618 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket ffzdqu-55 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:25:08.693 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:25:08.693 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:25:08.748 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2023-09-22T21:25:08.749 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:08.749 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:08.749 DEBUG:boto:Method: HEAD 2023-09-22T21:25:08.749 DEBUG:boto:Path: /ffzdqu-55/ 2023-09-22T21:25:08.749 DEBUG:boto:Data: 2023-09-22T21:25:08.749 DEBUG:boto:Headers: {} 2023-09-22T21:25:08.749 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:08.750 DEBUG:boto:Port: 8000 2023-09-22T21:25:08.750 DEBUG:boto:Params: {} 2023-09-22T21:25:08.750 DEBUG:boto:Token: None 2023-09-22T21:25:08.750 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:08 GMT /ffzdqu-55/ 2023-09-22T21:25:08.750 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FxacCLDpY96qypcH7f6JIEs56eY= 2023-09-22T21:25:08.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FxacCLDpY96qypcH7f6JIEs56eY=', 'Content-Length': '0'} 2023-09-22T21:25:08.752 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ccacbefa8facc657-00650e0634-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:08.752 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:25:08.753 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:25:08.753 DEBUG:boto:Method: PUT 2023-09-22T21:25:08.753 DEBUG:boto:Path: /ffzdqu-55/e 2023-09-22T21:25:08.753 DEBUG:boto:Data: 2023-09-22T21:25:08.753 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:25:08.753 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:08.753 DEBUG:boto:Port: 8000 2023-09-22T21:25:08.753 DEBUG:boto:Params: {} 2023-09-22T21:25:08.754 DEBUG:boto:Token: None 2023-09-22T21:25:08.754 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:25:08 GMT /ffzdqu-55/e 2023-09-22T21:25:08.754 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J/kYoxaDNsvh4eKxZMk6ZQ17jxE= 2023-09-22T21:25:08.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:25:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J/kYoxaDNsvh4eKxZMk6ZQ17jxE='} 2023-09-22T21:25:08.760 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000971eb709e68d6d71-00650e0634-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:08.760 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:08.760 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:08.760 DEBUG:boto:Method: HEAD 2023-09-22T21:25:08.760 DEBUG:boto:Path: /ffzdqu-55/ 2023-09-22T21:25:08.760 DEBUG:boto:Data: 2023-09-22T21:25:08.761 DEBUG:boto:Headers: {} 2023-09-22T21:25:08.761 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:08.761 DEBUG:boto:Port: 8000 2023-09-22T21:25:08.761 DEBUG:boto:Params: {} 2023-09-22T21:25:08.761 DEBUG:boto:Token: None 2023-09-22T21:25:08.761 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:08 GMT /ffzdqu-55/ 2023-09-22T21:25:08.761 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FxacCLDpY96qypcH7f6JIEs56eY= 2023-09-22T21:25:08.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FxacCLDpY96qypcH7f6JIEs56eY=', 'Content-Length': '0'} 2023-09-22T21:25:08.763 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4a2717cb70a398c-00650e0634-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:08.763 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:25:08.763 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:25:08.763 DEBUG:boto:Method: PUT 2023-09-22T21:25:08.763 DEBUG:boto:Path: /ffzdqu-55/f 2023-09-22T21:25:08.763 DEBUG:boto:Data: 2023-09-22T21:25:08.764 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:25:08.764 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:08.764 DEBUG:boto:Port: 8000 2023-09-22T21:25:08.764 DEBUG:boto:Params: {} 2023-09-22T21:25:08.764 DEBUG:boto:Token: None 2023-09-22T21:25:08.764 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:25:08 GMT /ffzdqu-55/f 2023-09-22T21:25:08.764 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wLjXdCEorIlMEE5k4lllDEWmq8Q= 2023-09-22T21:25:08.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:25:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wLjXdCEorIlMEE5k4lllDEWmq8Q='} 2023-09-22T21:25:08.806 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000086422fe2ada5143a-00650e0634-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:08.807 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:08.807 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:08.807 DEBUG:boto:Method: HEAD 2023-09-22T21:25:08.807 DEBUG:boto:Path: /ffzdqu-55/ 2023-09-22T21:25:08.807 DEBUG:boto:Data: 2023-09-22T21:25:08.807 DEBUG:boto:Headers: {} 2023-09-22T21:25:08.807 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:08.808 DEBUG:boto:Port: 8000 2023-09-22T21:25:08.808 DEBUG:boto:Params: {} 2023-09-22T21:25:08.808 DEBUG:boto:Token: None 2023-09-22T21:25:08.808 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:08 GMT /ffzdqu-55/ 2023-09-22T21:25:08.808 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FxacCLDpY96qypcH7f6JIEs56eY= 2023-09-22T21:25:08.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FxacCLDpY96qypcH7f6JIEs56eY=', 'Content-Length': '0'} 2023-09-22T21:25:08.810 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d0373b63735e7b53-00650e0634-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:08.811 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:25:08.811 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:25:08.811 DEBUG:boto:Method: PUT 2023-09-22T21:25:08.811 DEBUG:boto:Path: /ffzdqu-55/g 2023-09-22T21:25:08.811 DEBUG:boto:Data: 2023-09-22T21:25:08.811 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:25:08.811 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:08.811 DEBUG:boto:Port: 8000 2023-09-22T21:25:08.812 DEBUG:boto:Params: {} 2023-09-22T21:25:08.812 DEBUG:boto:Token: None 2023-09-22T21:25:08.812 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:25:08 GMT /ffzdqu-55/g 2023-09-22T21:25:08.812 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LkzR33hqtDxYCM7tF13x2zRvFlo= 2023-09-22T21:25:08.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:25:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LkzR33hqtDxYCM7tF13x2zRvFlo='} 2023-09-22T21:25:08.822 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000be4f2c0e0670b671-00650e0634-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:08.822 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:25:08.822 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:25:08.822 DEBUG:boto:Method: HEAD 2023-09-22T21:25:08.823 DEBUG:boto:Path: /ffzdqu-55/ 2023-09-22T21:25:08.823 DEBUG:boto:Data: 2023-09-22T21:25:08.823 DEBUG:boto:Headers: {} 2023-09-22T21:25:08.823 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:08.823 DEBUG:boto:Port: 8000 2023-09-22T21:25:08.823 DEBUG:boto:Params: {} 2023-09-22T21:25:08.823 DEBUG:boto:Token: None 2023-09-22T21:25:08.823 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:25:08 GMT /ffzdqu-55/ 2023-09-22T21:25:08.823 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FxacCLDpY96qypcH7f6JIEs56eY= 2023-09-22T21:25:08.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:25:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FxacCLDpY96qypcH7f6JIEs56eY=', 'Content-Length': '0'} 2023-09-22T21:25:08.826 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008701b7556752153d-00650e0634-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:08.826 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:25:08.826 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:25:08.826 DEBUG:boto:Method: PUT 2023-09-22T21:25:08.826 DEBUG:boto:Path: /ffzdqu-55/h 2023-09-22T21:25:08.826 DEBUG:boto:Data: 2023-09-22T21:25:08.827 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:25:08.827 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:25:08.827 DEBUG:boto:Port: 8000 2023-09-22T21:25:08.827 DEBUG:boto:Params: {} 2023-09-22T21:25:08.827 DEBUG:boto:Token: None 2023-09-22T21:25:08.827 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:25:08 GMT /ffzdqu-55/h 2023-09-22T21:25:08.827 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ejrtcaQL4DXvWLD2qXET6CEllS0= 2023-09-22T21:25:08.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:25:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ejrtcaQL4DXvWLD2qXET6CEllS0='} 2023-09-22T21:25:08.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a68ac0c924c6d70-00650e0634-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:25:08.871 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-55 2023-09-22T21:25:08.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:25:08.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:25:08.871 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:25:08.921 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:25:09.066 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:25:09.067 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:25:08.962+0000 7f1e623d9a40 1 waiting to reach latest gen 1, on 0.. 2023-09-22T21:25:13.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:25:13.962+0000 7f1e623d9a40 1 waiting to reach latest gen 1, on 0.. 2023-09-22T21:25:18.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:25:18.962+0000 7f1e623d9a40 1 waiting to reach latest gen 1, on 0.. 2023-09-22T21:25:23.969 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:25:23.966+0000 7f1e623d9a40 1 waiting for incremental sync to catch up: 2023-09-22T21:25:23.969 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.22.6, ] 2023-09-22T21:25:23.969 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.31.6, 00000000002.22.6, 00000000002.59.6] 2023-09-22T21:25:28.970 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:25:28.966+0000 7f1e623d9a40 1 waiting for incremental sync to catch up: 2023-09-22T21:25:28.971 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.22.6, ] 2023-09-22T21:25:28.971 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.31.6, 00000000002.22.6, 00000000002.59.6] 2023-09-22T21:25:33.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:25:33.970+0000 7f1e623d9a40 1 waiting for incremental sync to catch up: 2023-09-22T21:25:33.972 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.22.6, ] 2023-09-22T21:25:33.972 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.31.6, 00000000002.22.6, 00000000002.59.6] 2023-09-22T21:25:38.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:25:38.970+0000 7f1e623d9a40 1 waiting for incremental sync to catch up: 2023-09-22T21:25:38.973 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.22.6, ] 2023-09-22T21:25:38.973 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.31.6, 00000000002.22.6, 00000000002.59.6] 2023-09-22T21:25:43.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:25:43.970+0000 7f1e623d9a40 1 waiting for incremental sync to catch up: 2023-09-22T21:25:43.974 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.22.6, ] 2023-09-22T21:25:43.975 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.31.6, 00000000002.22.6, 00000000002.59.6] 2023-09-22T21:25:48.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:25:48.974+0000 7f1e623d9a40 1 waiting for incremental sync to catch up: 2023-09-22T21:25:48.976 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.22.6, ] 2023-09-22T21:25:48.976 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.31.6, 00000000002.22.6, 00000000002.59.6] 2023-09-22T21:25:53.977 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:25:53.973+0000 7f1e623d9a40 1 waiting for incremental sync to catch up: 2023-09-22T21:25:53.977 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.22.6, ] 2023-09-22T21:25:53.977 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.31.6, 00000000002.22.6, 00000000002.59.6] 2023-09-22T21:25:57.788 INFO:tasks.rgw.c2.client.1.smithi204.stdout:2023-09-22T21:25:57.789+0000 7fb92d9f1640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:25:58.978 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:25:58.973+0000 7f1e623d9a40 1 waiting for incremental sync to catch up: 2023-09-22T21:25:58.978 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.22.6, ] 2023-09-22T21:25:58.979 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.31.6, 00000000002.22.6, 00000000002.59.6] 2023-09-22T21:26:03.979 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:26:03.977+0000 7f1e623d9a40 1 waiting for incremental sync to catch up: 2023-09-22T21:26:03.980 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.22.6, ] 2023-09-22T21:26:03.980 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.31.6, 00000000002.22.6, 00000000002.59.6] 2023-09-22T21:26:08.981 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:26:08.981+0000 7f1e623d9a40 1 waiting for incremental sync to catch up: 2023-09-22T21:26:08.981 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.22.6, ] 2023-09-22T21:26:08.981 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.31.6, 00000000002.22.6, 00000000002.59.6] 2023-09-22T21:26:13.982 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:26:13.981+0000 7f1e623d9a40 1 waiting for incremental sync to catch up: 2023-09-22T21:26:13.982 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.22.6, ] 2023-09-22T21:26:13.982 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.31.6, 00000000002.22.6, 00000000002.59.6] 2023-09-22T21:26:18.984 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:26:18.985+0000 7f1e623d9a40 1 waiting for incremental sync to catch up: 2023-09-22T21:26:18.984 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.22.6, ] 2023-09-22T21:26:18.984 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.31.6, 00000000002.22.6, 00000000002.59.6] 2023-09-22T21:26:23.592 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:23.593+0000 7ff4815cb640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:26:23.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:26:23.985+0000 7f1e623d9a40 1 waiting for incremental sync to catch up: 2023-09-22T21:26:23.985 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000003.31.6, 00000000002.22.6, ] 2023-09-22T21:26:23.986 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.31.6, 00000000002.22.6, 00000000002.59.6] 2023-09-22T21:26:24.575 INFO:tasks.rgw.c1.client.0.smithi160.stdout:2023-09-22T21:26:24.577+0000 7fa9c1675640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:26:28.987 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:26:28.985+0000 7f1e623d9a40 1 bucket sync caught up with source: 2023-09-22T21:26:28.987 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000003.31.6, 00000000002.22.6, 00000000002.59.6] 2023-09-22T21:26:28.987 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.31.6, 00000000002.22.6, 00000000002.59.6] 2023-09-22T21:26:28.987 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:26:28.985+0000 7f1e623d9a40 0 bucket checkpoint complete 2023-09-22T21:26:29.023 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-55 2023-09-22T21:26:29.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:26:29.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:26:29.024 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:26:29.074 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:26:29.074 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:26:29.116 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:26:29.117+0000 7efd91c0aa40 1 bucket sync caught up with source: 2023-09-22T21:26:29.116 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.278.6, , , 00000000001.231.6, 00000000001.197.6, , , 00000000002.167.6, , 00000000002.165.6] 2023-09-22T21:26:29.116 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.278.6, , , 00000000001.231.6, 00000000001.197.6, , , 00000000002.167.6, , 00000000002.165.6] 2023-09-22T21:26:29.116 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:26:29.117+0000 7efd91c0aa40 0 bucket checkpoint complete 2023-09-22T21:26:29.133 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-55 zones={a2, a1} 2023-09-22T21:26:29.133 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.133 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.134 DEBUG:boto:Method: HEAD 2023-09-22T21:26:29.134 DEBUG:boto:Path: /ffzdqu-55/ 2023-09-22T21:26:29.134 DEBUG:boto:Data: 2023-09-22T21:26:29.134 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.134 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.134 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.134 DEBUG:boto:Params: {} 2023-09-22T21:26:29.134 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:26:29.135 DEBUG:boto:Token: None 2023-09-22T21:26:29.135 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/ 2023-09-22T21:26:29.135 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3j/x+U8tmmegN1DuSel1DO+KKzk= 2023-09-22T21:26:29.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3j/x+U8tmmegN1DuSel1DO+KKzk=', 'Content-Length': '0'} 2023-09-22T21:26:29.140 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088c5228e69ad4086-00650e0685-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.140 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.140 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.140 DEBUG:boto:Method: HEAD 2023-09-22T21:26:29.140 DEBUG:boto:Path: /ffzdqu-55/ 2023-09-22T21:26:29.141 DEBUG:boto:Data: 2023-09-22T21:26:29.141 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.141 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.141 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.141 DEBUG:boto:Params: {} 2023-09-22T21:26:29.141 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:26:29.141 DEBUG:boto:Token: None 2023-09-22T21:26:29.141 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/ 2023-09-22T21:26:29.142 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3j/x+U8tmmegN1DuSel1DO+KKzk= 2023-09-22T21:26:29.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3j/x+U8tmmegN1DuSel1DO+KKzk=', 'Content-Length': '0'} 2023-09-22T21:26:29.144 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000700aee64fb28535d-00650e0685-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.145 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:26:29.145 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.145 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.145 DEBUG:boto:path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:29.145 DEBUG:boto:auth_path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:29.145 DEBUG:boto:Method: GET 2023-09-22T21:26:29.145 DEBUG:boto:Path: /ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:29.145 DEBUG:boto:Data: 2023-09-22T21:26:29.145 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.146 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.146 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.146 DEBUG:boto:Params: {} 2023-09-22T21:26:29.146 DEBUG:boto:Token: None 2023-09-22T21:26:29.146 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/?versions 2023-09-22T21:26:29.146 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4Wboi8Gjg/HhkXky1xhdspMsPAg= 2023-09-22T21:26:29.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4Wboi8Gjg/HhkXky1xhdspMsPAg=', 'Content-Length': '0'} 2023-09-22T21:26:29.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005eab0bf864930708-00650e0685-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.149 DEBUG:boto:b'ffzdqu-55999falseanulltrue2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:29.150 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-22T21:26:29.150 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-22T21:26:29.150 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-22T21:26:29.150 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-22T21:26:29.150 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-09-22T21:26:29.150 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-09-22T21:26:29.150 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-09-22T21:26:29.151 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-09-22T21:26:29.151 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:26:29.151 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.151 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.151 DEBUG:boto:path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:29.151 DEBUG:boto:auth_path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:29.151 DEBUG:boto:Method: GET 2023-09-22T21:26:29.151 DEBUG:boto:Path: /ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:29.151 DEBUG:boto:Data: 2023-09-22T21:26:29.151 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.151 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.152 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.152 DEBUG:boto:Params: {} 2023-09-22T21:26:29.152 DEBUG:boto:Token: None 2023-09-22T21:26:29.152 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/?versions 2023-09-22T21:26:29.152 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4Wboi8Gjg/HhkXky1xhdspMsPAg= 2023-09-22T21:26:29.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4Wboi8Gjg/HhkXky1xhdspMsPAg=', 'Content-Length': '0'} 2023-09-22T21:26:29.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000640eb7fff46adee0-00650e0685-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.154 DEBUG:boto:b'ffzdqu-55999falseanulltrue2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:29.155 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-22T21:26:29.155 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-22T21:26:29.155 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-22T21:26:29.155 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-22T21:26:29.155 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-09-22T21:26:29.155 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-09-22T21:26:29.156 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-09-22T21:26:29.156 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-09-22T21:26:29.156 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.156 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.156 DEBUG:boto:path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:29.156 DEBUG:boto:auth_path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:29.156 DEBUG:boto:Method: GET 2023-09-22T21:26:29.156 DEBUG:boto:Path: /ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:29.156 DEBUG:boto:Data: 2023-09-22T21:26:29.156 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.157 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.157 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.157 DEBUG:boto:Params: {} 2023-09-22T21:26:29.157 DEBUG:boto:Token: None 2023-09-22T21:26:29.157 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/?versions 2023-09-22T21:26:29.157 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4Wboi8Gjg/HhkXky1xhdspMsPAg= 2023-09-22T21:26:29.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4Wboi8Gjg/HhkXky1xhdspMsPAg=', 'Content-Length': '0'} 2023-09-22T21:26:29.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbbdd3eda8557eaf-00650e0685-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.160 DEBUG:boto:b'ffzdqu-55999falseanulltrue2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:29.161 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.161 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.161 DEBUG:boto:path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:29.161 DEBUG:boto:auth_path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:29.161 DEBUG:boto:Method: GET 2023-09-22T21:26:29.161 DEBUG:boto:Path: /ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:29.161 DEBUG:boto:Data: 2023-09-22T21:26:29.162 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.162 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.162 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.162 DEBUG:boto:Params: {} 2023-09-22T21:26:29.162 DEBUG:boto:Token: None 2023-09-22T21:26:29.162 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/?versions 2023-09-22T21:26:29.162 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4Wboi8Gjg/HhkXky1xhdspMsPAg= 2023-09-22T21:26:29.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4Wboi8Gjg/HhkXky1xhdspMsPAg=', 'Content-Length': '0'} 2023-09-22T21:26:29.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c45ffcdde23b061-00650e0685-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.164 DEBUG:boto:b'ffzdqu-55999falseanulltrue2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:29.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:26:29.165 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:26:29.165 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:26:29.165 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.166 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.166 DEBUG:boto:Method: GET 2023-09-22T21:26:29.166 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.166 DEBUG:boto:Data: 2023-09-22T21:26:29.166 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.166 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.166 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.166 DEBUG:boto:Params: {} 2023-09-22T21:26:29.166 DEBUG:boto:Token: None 2023-09-22T21:26:29.167 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.167 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PT/+JdvuMX1ozD0Pa6oLFy0Yw8I= 2023-09-22T21:26:29.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PT/+JdvuMX1ozD0Pa6oLFy0Yw8I=', 'Content-Length': '0'} 2023-09-22T21:26:29.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076e2c02265def3a9-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.169 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:26:29.169 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:26:29.169 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.169 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.169 DEBUG:boto:Method: GET 2023-09-22T21:26:29.169 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.169 DEBUG:boto:Data: 2023-09-22T21:26:29.169 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.169 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.169 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.170 DEBUG:boto:Params: {} 2023-09-22T21:26:29.170 DEBUG:boto:Token: None 2023-09-22T21:26:29.170 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.170 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PT/+JdvuMX1ozD0Pa6oLFy0Yw8I= 2023-09-22T21:26:29.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PT/+JdvuMX1ozD0Pa6oLFy0Yw8I=', 'Content-Length': '0'} 2023-09-22T21:26:29.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1adfbf9043e9821-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.172 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:26:29.172 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:26:29.173 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.173 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.173 DEBUG:boto:Method: HEAD 2023-09-22T21:26:29.173 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.173 DEBUG:boto:Data: 2023-09-22T21:26:29.173 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.173 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.173 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.173 DEBUG:boto:Params: {} 2023-09-22T21:26:29.173 DEBUG:boto:Token: None 2023-09-22T21:26:29.174 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.174 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/014CULJODusL10C8PcbTqwccG4= 2023-09-22T21:26:29.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/014CULJODusL10C8PcbTqwccG4=', 'Content-Length': '0'} 2023-09-22T21:26:29.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc44087c5c058b80-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.176 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:26:29.176 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:26:29.176 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.176 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.176 DEBUG:boto:Method: HEAD 2023-09-22T21:26:29.176 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.176 DEBUG:boto:Data: 2023-09-22T21:26:29.176 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.176 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.177 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.177 DEBUG:boto:Params: {} 2023-09-22T21:26:29.177 DEBUG:boto:Token: None 2023-09-22T21:26:29.177 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.177 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/014CULJODusL10C8PcbTqwccG4= 2023-09-22T21:26:29.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/014CULJODusL10C8PcbTqwccG4=', 'Content-Length': '0'} 2023-09-22T21:26:29.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000219b148b1257b6f2-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:26:29.179 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:26:29.179 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:26:29.179 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.179 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.180 DEBUG:boto:Method: GET 2023-09-22T21:26:29.180 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.180 DEBUG:boto:Data: 2023-09-22T21:26:29.180 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.180 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.180 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.180 DEBUG:boto:Params: {} 2023-09-22T21:26:29.180 DEBUG:boto:Token: None 2023-09-22T21:26:29.180 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.181 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PT/+JdvuMX1ozD0Pa6oLFy0Yw8I= 2023-09-22T21:26:29.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PT/+JdvuMX1ozD0Pa6oLFy0Yw8I=', 'Content-Length': '0'} 2023-09-22T21:26:29.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b93288bc6496c724-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.182 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:26:29.183 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:26:29.183 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.183 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.183 DEBUG:boto:Method: GET 2023-09-22T21:26:29.183 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.183 DEBUG:boto:Data: 2023-09-22T21:26:29.183 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.183 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.183 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.183 DEBUG:boto:Params: {} 2023-09-22T21:26:29.184 DEBUG:boto:Token: None 2023-09-22T21:26:29.184 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:26:29.184 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PT/+JdvuMX1ozD0Pa6oLFy0Yw8I= 2023-09-22T21:26:29.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PT/+JdvuMX1ozD0Pa6oLFy0Yw8I=', 'Content-Length': '0'} 2023-09-22T21:26:29.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6a3b70f520345a2-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.186 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.186 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.186 DEBUG:boto:path=/ffzdqu-55/?prefix=a 2023-09-22T21:26:29.186 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=a 2023-09-22T21:26:29.187 DEBUG:boto:Method: GET 2023-09-22T21:26:29.187 DEBUG:boto:Path: /ffzdqu-55/?prefix=a 2023-09-22T21:26:29.187 DEBUG:boto:Data: 2023-09-22T21:26:29.187 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.187 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.187 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.187 DEBUG:boto:Params: {} 2023-09-22T21:26:29.187 DEBUG:boto:Token: None 2023-09-22T21:26:29.187 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/ 2023-09-22T21:26:29.188 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4= 2023-09-22T21:26:29.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4=', 'Content-Length': '0'} 2023-09-22T21:26:29.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c18a2313a1e9547-00650e0685-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.234 DEBUG:boto:b'ffzdqu-55a1000falsea2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:29.235 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.235 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.235 DEBUG:boto:path=/ffzdqu-55/?prefix=a 2023-09-22T21:26:29.235 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=a 2023-09-22T21:26:29.235 DEBUG:boto:Method: GET 2023-09-22T21:26:29.235 DEBUG:boto:Path: /ffzdqu-55/?prefix=a 2023-09-22T21:26:29.235 DEBUG:boto:Data: 2023-09-22T21:26:29.235 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.236 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.236 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.236 DEBUG:boto:Params: {} 2023-09-22T21:26:29.236 DEBUG:boto:Token: None 2023-09-22T21:26:29.236 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/ 2023-09-22T21:26:29.236 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4= 2023-09-22T21:26:29.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4=', 'Content-Length': '0'} 2023-09-22T21:26:29.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008202f21516bd09f8-00650e0685-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.239 DEBUG:boto:b'ffzdqu-55a1000falsea2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:29.240 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:26:29.240 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:26:29.240 DEBUG:boto:Method: HEAD 2023-09-22T21:26:29.240 DEBUG:boto:Path: /ffzdqu-55/a 2023-09-22T21:26:29.240 DEBUG:boto:Data: 2023-09-22T21:26:29.240 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.240 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.241 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.241 DEBUG:boto:Params: {} 2023-09-22T21:26:29.241 DEBUG:boto:Token: None 2023-09-22T21:26:29.241 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/a 2023-09-22T21:26:29.241 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xNJbETpMzvrl9cRMmY9c0AkRQ+w= 2023-09-22T21:26:29.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xNJbETpMzvrl9cRMmY9c0AkRQ+w=', 'Content-Length': '0'} 2023-09-22T21:26:29.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f49c77a8c685ec5-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.244 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:26:29.244 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:26:29.244 DEBUG:boto:Method: HEAD 2023-09-22T21:26:29.244 DEBUG:boto:Path: /ffzdqu-55/a 2023-09-22T21:26:29.245 DEBUG:boto:Data: 2023-09-22T21:26:29.245 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.245 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.245 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.245 DEBUG:boto:Params: {} 2023-09-22T21:26:29.245 DEBUG:boto:Token: None 2023-09-22T21:26:29.245 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/a 2023-09-22T21:26:29.245 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xNJbETpMzvrl9cRMmY9c0AkRQ+w= 2023-09-22T21:26:29.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xNJbETpMzvrl9cRMmY9c0AkRQ+w=', 'Content-Length': '0'} 2023-09-22T21:26:29.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079ead605e968b508-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:26:29.248 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:26:29.248 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:26:29.249 DEBUG:boto:Method: GET 2023-09-22T21:26:29.249 DEBUG:boto:Path: /ffzdqu-55/a 2023-09-22T21:26:29.249 DEBUG:boto:Data: 2023-09-22T21:26:29.249 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.249 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.249 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.249 DEBUG:boto:Params: {} 2023-09-22T21:26:29.249 DEBUG:boto:Token: None 2023-09-22T21:26:29.249 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/a 2023-09-22T21:26:29.250 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:43Rqx9frHVD+LwGsOWV0YWkN1tw= 2023-09-22T21:26:29.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:43Rqx9frHVD+LwGsOWV0YWkN1tw=', 'Content-Length': '0'} 2023-09-22T21:26:29.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a61648f7e9cab73d-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.298 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:26:29.299 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:26:29.299 DEBUG:boto:Method: GET 2023-09-22T21:26:29.299 DEBUG:boto:Path: /ffzdqu-55/a 2023-09-22T21:26:29.299 DEBUG:boto:Data: 2023-09-22T21:26:29.299 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.299 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.299 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.299 DEBUG:boto:Params: {} 2023-09-22T21:26:29.299 DEBUG:boto:Token: None 2023-09-22T21:26:29.300 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/a 2023-09-22T21:26:29.300 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:43Rqx9frHVD+LwGsOWV0YWkN1tw= 2023-09-22T21:26:29.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:43Rqx9frHVD+LwGsOWV0YWkN1tw=', 'Content-Length': '0'} 2023-09-22T21:26:29.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cfcd9184430b727-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.303 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.303 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.303 DEBUG:boto:path=/ffzdqu-55/?prefix=a 2023-09-22T21:26:29.303 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=a 2023-09-22T21:26:29.303 DEBUG:boto:Method: GET 2023-09-22T21:26:29.303 DEBUG:boto:Path: /ffzdqu-55/?prefix=a 2023-09-22T21:26:29.304 DEBUG:boto:Data: 2023-09-22T21:26:29.304 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.304 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.304 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.304 DEBUG:boto:Params: {} 2023-09-22T21:26:29.304 DEBUG:boto:Token: None 2023-09-22T21:26:29.304 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/ 2023-09-22T21:26:29.304 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4= 2023-09-22T21:26:29.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4=', 'Content-Length': '0'} 2023-09-22T21:26:29.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df1ba289a22d0f3b-00650e0685-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.350 DEBUG:boto:b'ffzdqu-55a1000falsea2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:29.351 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.351 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.351 DEBUG:boto:path=/ffzdqu-55/?prefix=a 2023-09-22T21:26:29.351 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=a 2023-09-22T21:26:29.351 DEBUG:boto:Method: GET 2023-09-22T21:26:29.352 DEBUG:boto:Path: /ffzdqu-55/?prefix=a 2023-09-22T21:26:29.352 DEBUG:boto:Data: 2023-09-22T21:26:29.352 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.352 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.352 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.352 DEBUG:boto:Params: {} 2023-09-22T21:26:29.352 DEBUG:boto:Token: None 2023-09-22T21:26:29.352 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/ 2023-09-22T21:26:29.352 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4= 2023-09-22T21:26:29.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4=', 'Content-Length': '0'} 2023-09-22T21:26:29.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ef08105238b8de3-00650e0685-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.356 DEBUG:boto:b'ffzdqu-55a1000falsea2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:29.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:26:29.356 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:26:29.356 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:26:29.356 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.357 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.357 DEBUG:boto:Method: GET 2023-09-22T21:26:29.357 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.357 DEBUG:boto:Data: 2023-09-22T21:26:29.357 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.357 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.357 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.357 DEBUG:boto:Params: {} 2023-09-22T21:26:29.357 DEBUG:boto:Token: None 2023-09-22T21:26:29.358 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.358 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y0T/QC5+MXOklrJREt2EmvvRtYM= 2023-09-22T21:26:29.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y0T/QC5+MXOklrJREt2EmvvRtYM=', 'Content-Length': '0'} 2023-09-22T21:26:29.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bf690d4616451e9-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.361 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:26:29.361 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:26:29.361 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.361 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.362 DEBUG:boto:Method: GET 2023-09-22T21:26:29.362 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.362 DEBUG:boto:Data: 2023-09-22T21:26:29.362 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.362 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.362 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.362 DEBUG:boto:Params: {} 2023-09-22T21:26:29.362 DEBUG:boto:Token: None 2023-09-22T21:26:29.362 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.363 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y0T/QC5+MXOklrJREt2EmvvRtYM= 2023-09-22T21:26:29.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y0T/QC5+MXOklrJREt2EmvvRtYM=', 'Content-Length': '0'} 2023-09-22T21:26:29.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f1ae4fe20f58328-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.365 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:26:29.366 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:26:29.366 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.366 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.366 DEBUG:boto:Method: HEAD 2023-09-22T21:26:29.366 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.366 DEBUG:boto:Data: 2023-09-22T21:26:29.366 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.366 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.366 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.367 DEBUG:boto:Params: {} 2023-09-22T21:26:29.367 DEBUG:boto:Token: None 2023-09-22T21:26:29.367 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.367 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aTckdCFQftHjQwDyqQ+TJ3BY5Yo= 2023-09-22T21:26:29.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aTckdCFQftHjQwDyqQ+TJ3BY5Yo=', 'Content-Length': '0'} 2023-09-22T21:26:29.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a7df3dc1c34bd40-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.370 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:26:29.370 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:26:29.370 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.370 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.370 DEBUG:boto:Method: HEAD 2023-09-22T21:26:29.371 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.371 DEBUG:boto:Data: 2023-09-22T21:26:29.371 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.371 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.371 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.371 DEBUG:boto:Params: {} 2023-09-22T21:26:29.371 DEBUG:boto:Token: None 2023-09-22T21:26:29.371 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.371 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aTckdCFQftHjQwDyqQ+TJ3BY5Yo= 2023-09-22T21:26:29.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aTckdCFQftHjQwDyqQ+TJ3BY5Yo=', 'Content-Length': '0'} 2023-09-22T21:26:29.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080cb965668c9c682-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:26:29.375 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:26:29.375 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:26:29.375 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.375 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.375 DEBUG:boto:Method: GET 2023-09-22T21:26:29.375 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.375 DEBUG:boto:Data: 2023-09-22T21:26:29.375 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.375 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.376 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.376 DEBUG:boto:Params: {} 2023-09-22T21:26:29.376 DEBUG:boto:Token: None 2023-09-22T21:26:29.376 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.376 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y0T/QC5+MXOklrJREt2EmvvRtYM= 2023-09-22T21:26:29.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y0T/QC5+MXOklrJREt2EmvvRtYM=', 'Content-Length': '0'} 2023-09-22T21:26:29.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004453e613aefdc8d1-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.422 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:26:29.422 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:26:29.423 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.423 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.423 DEBUG:boto:Method: GET 2023-09-22T21:26:29.423 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.423 DEBUG:boto:Data: 2023-09-22T21:26:29.423 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.423 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.423 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.423 DEBUG:boto:Params: {} 2023-09-22T21:26:29.423 DEBUG:boto:Token: None 2023-09-22T21:26:29.424 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:26:29.424 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y0T/QC5+MXOklrJREt2EmvvRtYM= 2023-09-22T21:26:29.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y0T/QC5+MXOklrJREt2EmvvRtYM=', 'Content-Length': '0'} 2023-09-22T21:26:29.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007213811274b1841a-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.427 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.427 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.427 DEBUG:boto:path=/ffzdqu-55/?prefix=b 2023-09-22T21:26:29.427 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=b 2023-09-22T21:26:29.428 DEBUG:boto:Method: GET 2023-09-22T21:26:29.428 DEBUG:boto:Path: /ffzdqu-55/?prefix=b 2023-09-22T21:26:29.428 DEBUG:boto:Data: 2023-09-22T21:26:29.428 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.428 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.428 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.428 DEBUG:boto:Params: {} 2023-09-22T21:26:29.428 DEBUG:boto:Token: None 2023-09-22T21:26:29.428 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/ 2023-09-22T21:26:29.428 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4= 2023-09-22T21:26:29.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4=', 'Content-Length': '0'} 2023-09-22T21:26:29.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4a6e3c0beccb158-00650e0685-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.435 DEBUG:boto:b'ffzdqu-55b1000falseb2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:29.435 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.435 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.435 DEBUG:boto:path=/ffzdqu-55/?prefix=b 2023-09-22T21:26:29.435 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=b 2023-09-22T21:26:29.436 DEBUG:boto:Method: GET 2023-09-22T21:26:29.436 DEBUG:boto:Path: /ffzdqu-55/?prefix=b 2023-09-22T21:26:29.436 DEBUG:boto:Data: 2023-09-22T21:26:29.436 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.436 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.436 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.436 DEBUG:boto:Params: {} 2023-09-22T21:26:29.436 DEBUG:boto:Token: None 2023-09-22T21:26:29.436 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/ 2023-09-22T21:26:29.437 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4= 2023-09-22T21:26:29.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4=', 'Content-Length': '0'} 2023-09-22T21:26:29.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d69feb73dfbe75d-00650e0685-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.440 DEBUG:boto:b'ffzdqu-55b1000falseb2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:29.440 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:26:29.441 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:26:29.441 DEBUG:boto:Method: HEAD 2023-09-22T21:26:29.441 DEBUG:boto:Path: /ffzdqu-55/b 2023-09-22T21:26:29.441 DEBUG:boto:Data: 2023-09-22T21:26:29.441 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.441 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.441 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.441 DEBUG:boto:Params: {} 2023-09-22T21:26:29.441 DEBUG:boto:Token: None 2023-09-22T21:26:29.442 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/b 2023-09-22T21:26:29.442 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BA5Vv9BG8uJ6ap1ccD18byPTnu4= 2023-09-22T21:26:29.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BA5Vv9BG8uJ6ap1ccD18byPTnu4=', 'Content-Length': '0'} 2023-09-22T21:26:29.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c9dcba6e4d4a875-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.444 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:26:29.445 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:26:29.445 DEBUG:boto:Method: HEAD 2023-09-22T21:26:29.445 DEBUG:boto:Path: /ffzdqu-55/b 2023-09-22T21:26:29.445 DEBUG:boto:Data: 2023-09-22T21:26:29.445 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.445 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.445 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.445 DEBUG:boto:Params: {} 2023-09-22T21:26:29.446 DEBUG:boto:Token: None 2023-09-22T21:26:29.446 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/b 2023-09-22T21:26:29.446 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BA5Vv9BG8uJ6ap1ccD18byPTnu4= 2023-09-22T21:26:29.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BA5Vv9BG8uJ6ap1ccD18byPTnu4=', 'Content-Length': '0'} 2023-09-22T21:26:29.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000318b9523bfab3e86-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:26:29.449 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:26:29.449 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:26:29.449 DEBUG:boto:Method: GET 2023-09-22T21:26:29.449 DEBUG:boto:Path: /ffzdqu-55/b 2023-09-22T21:26:29.449 DEBUG:boto:Data: 2023-09-22T21:26:29.450 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.450 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.450 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.450 DEBUG:boto:Params: {} 2023-09-22T21:26:29.450 DEBUG:boto:Token: None 2023-09-22T21:26:29.450 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/b 2023-09-22T21:26:29.450 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3zLXj+CPvL2LdDM9Jp2HJj7/LLQ= 2023-09-22T21:26:29.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3zLXj+CPvL2LdDM9Jp2HJj7/LLQ=', 'Content-Length': '0'} 2023-09-22T21:26:29.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000435f7c33d6298d16-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.494 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:26:29.494 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:26:29.495 DEBUG:boto:Method: GET 2023-09-22T21:26:29.495 DEBUG:boto:Path: /ffzdqu-55/b 2023-09-22T21:26:29.495 DEBUG:boto:Data: 2023-09-22T21:26:29.495 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.495 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.495 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.495 DEBUG:boto:Params: {} 2023-09-22T21:26:29.495 DEBUG:boto:Token: None 2023-09-22T21:26:29.495 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/b 2023-09-22T21:26:29.496 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3zLXj+CPvL2LdDM9Jp2HJj7/LLQ= 2023-09-22T21:26:29.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3zLXj+CPvL2LdDM9Jp2HJj7/LLQ=', 'Content-Length': '0'} 2023-09-22T21:26:29.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2eec34a66f75e5b-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.498 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.499 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.499 DEBUG:boto:path=/ffzdqu-55/?prefix=b 2023-09-22T21:26:29.499 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=b 2023-09-22T21:26:29.499 DEBUG:boto:Method: GET 2023-09-22T21:26:29.499 DEBUG:boto:Path: /ffzdqu-55/?prefix=b 2023-09-22T21:26:29.499 DEBUG:boto:Data: 2023-09-22T21:26:29.499 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.499 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.499 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.500 DEBUG:boto:Params: {} 2023-09-22T21:26:29.500 DEBUG:boto:Token: None 2023-09-22T21:26:29.500 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/ 2023-09-22T21:26:29.500 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4= 2023-09-22T21:26:29.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4=', 'Content-Length': '0'} 2023-09-22T21:26:29.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051a6c6c371ecd95c-00650e0685-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.506 DEBUG:boto:b'ffzdqu-55b1000falseb2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:29.507 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.507 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.507 DEBUG:boto:path=/ffzdqu-55/?prefix=b 2023-09-22T21:26:29.507 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=b 2023-09-22T21:26:29.507 DEBUG:boto:Method: GET 2023-09-22T21:26:29.507 DEBUG:boto:Path: /ffzdqu-55/?prefix=b 2023-09-22T21:26:29.507 DEBUG:boto:Data: 2023-09-22T21:26:29.507 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.508 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.508 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.508 DEBUG:boto:Params: {} 2023-09-22T21:26:29.508 DEBUG:boto:Token: None 2023-09-22T21:26:29.508 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/ 2023-09-22T21:26:29.508 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4= 2023-09-22T21:26:29.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4=', 'Content-Length': '0'} 2023-09-22T21:26:29.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abb276a77450c039-00650e0685-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.511 DEBUG:boto:b'ffzdqu-55b1000falseb2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:29.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:26:29.512 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:26:29.512 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:26:29.512 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.512 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.512 DEBUG:boto:Method: GET 2023-09-22T21:26:29.512 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.512 DEBUG:boto:Data: 2023-09-22T21:26:29.513 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.513 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.513 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.513 DEBUG:boto:Params: {} 2023-09-22T21:26:29.513 DEBUG:boto:Token: None 2023-09-22T21:26:29.513 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.513 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:whia4VKJtv9MxrrArAZFQDybaRw= 2023-09-22T21:26:29.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:whia4VKJtv9MxrrArAZFQDybaRw=', 'Content-Length': '0'} 2023-09-22T21:26:29.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d6c8af4cc7d52b7-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.516 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:26:29.517 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:26:29.517 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.517 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.517 DEBUG:boto:Method: GET 2023-09-22T21:26:29.517 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.517 DEBUG:boto:Data: 2023-09-22T21:26:29.517 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.517 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.517 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.518 DEBUG:boto:Params: {} 2023-09-22T21:26:29.518 DEBUG:boto:Token: None 2023-09-22T21:26:29.518 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.518 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:whia4VKJtv9MxrrArAZFQDybaRw= 2023-09-22T21:26:29.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:whia4VKJtv9MxrrArAZFQDybaRw=', 'Content-Length': '0'} 2023-09-22T21:26:29.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088531719ecd952e4-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.521 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:26:29.521 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:26:29.521 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.521 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.521 DEBUG:boto:Method: HEAD 2023-09-22T21:26:29.521 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.522 DEBUG:boto:Data: 2023-09-22T21:26:29.522 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.522 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.522 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.522 DEBUG:boto:Params: {} 2023-09-22T21:26:29.522 DEBUG:boto:Token: None 2023-09-22T21:26:29.522 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.522 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IYoECtuuBCfHn7yK4Y9A0ORYsiA= 2023-09-22T21:26:29.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IYoECtuuBCfHn7yK4Y9A0ORYsiA=', 'Content-Length': '0'} 2023-09-22T21:26:29.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f258f51d9328bdbf-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.525 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:26:29.525 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:26:29.525 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.525 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.526 DEBUG:boto:Method: HEAD 2023-09-22T21:26:29.526 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.526 DEBUG:boto:Data: 2023-09-22T21:26:29.526 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.526 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.526 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.526 DEBUG:boto:Params: {} 2023-09-22T21:26:29.526 DEBUG:boto:Token: None 2023-09-22T21:26:29.526 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.527 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IYoECtuuBCfHn7yK4Y9A0ORYsiA= 2023-09-22T21:26:29.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IYoECtuuBCfHn7yK4Y9A0ORYsiA=', 'Content-Length': '0'} 2023-09-22T21:26:29.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000383751d48edf6a2-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:26:29.530 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:26:29.530 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:26:29.530 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.530 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.530 DEBUG:boto:Method: GET 2023-09-22T21:26:29.530 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.530 DEBUG:boto:Data: 2023-09-22T21:26:29.530 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.530 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.531 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.531 DEBUG:boto:Params: {} 2023-09-22T21:26:29.531 DEBUG:boto:Token: None 2023-09-22T21:26:29.531 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.531 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:whia4VKJtv9MxrrArAZFQDybaRw= 2023-09-22T21:26:29.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:whia4VKJtv9MxrrArAZFQDybaRw=', 'Content-Length': '0'} 2023-09-22T21:26:29.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000222f98b63ce7ecf8-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.578 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:26:29.578 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:26:29.578 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.579 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.579 DEBUG:boto:Method: GET 2023-09-22T21:26:29.579 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.579 DEBUG:boto:Data: 2023-09-22T21:26:29.579 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.579 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.579 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.579 DEBUG:boto:Params: {} 2023-09-22T21:26:29.579 DEBUG:boto:Token: None 2023-09-22T21:26:29.580 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:26:29.580 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:whia4VKJtv9MxrrArAZFQDybaRw= 2023-09-22T21:26:29.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:whia4VKJtv9MxrrArAZFQDybaRw=', 'Content-Length': '0'} 2023-09-22T21:26:29.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aed317cb8e34c3d6-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.583 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.583 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.583 DEBUG:boto:path=/ffzdqu-55/?prefix=c 2023-09-22T21:26:29.583 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=c 2023-09-22T21:26:29.583 DEBUG:boto:Method: GET 2023-09-22T21:26:29.583 DEBUG:boto:Path: /ffzdqu-55/?prefix=c 2023-09-22T21:26:29.583 DEBUG:boto:Data: 2023-09-22T21:26:29.584 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.584 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.584 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.584 DEBUG:boto:Params: {} 2023-09-22T21:26:29.584 DEBUG:boto:Token: None 2023-09-22T21:26:29.584 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/ 2023-09-22T21:26:29.584 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4= 2023-09-22T21:26:29.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4=', 'Content-Length': '0'} 2023-09-22T21:26:29.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093e39cca39095546-00650e0685-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.590 DEBUG:boto:b'ffzdqu-55c1000falsec2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:29.590 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.590 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.591 DEBUG:boto:path=/ffzdqu-55/?prefix=c 2023-09-22T21:26:29.591 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=c 2023-09-22T21:26:29.591 DEBUG:boto:Method: GET 2023-09-22T21:26:29.591 DEBUG:boto:Path: /ffzdqu-55/?prefix=c 2023-09-22T21:26:29.591 DEBUG:boto:Data: 2023-09-22T21:26:29.591 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.591 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.591 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.591 DEBUG:boto:Params: {} 2023-09-22T21:26:29.591 DEBUG:boto:Token: None 2023-09-22T21:26:29.592 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/ 2023-09-22T21:26:29.592 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4= 2023-09-22T21:26:29.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4=', 'Content-Length': '0'} 2023-09-22T21:26:29.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b4e6dfc66f55454-00650e0685-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.638 DEBUG:boto:b'ffzdqu-55c1000falsec2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:29.639 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:26:29.639 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:26:29.639 DEBUG:boto:Method: HEAD 2023-09-22T21:26:29.639 DEBUG:boto:Path: /ffzdqu-55/c 2023-09-22T21:26:29.639 DEBUG:boto:Data: 2023-09-22T21:26:29.639 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.639 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.639 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.640 DEBUG:boto:Params: {} 2023-09-22T21:26:29.640 DEBUG:boto:Token: None 2023-09-22T21:26:29.640 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/c 2023-09-22T21:26:29.640 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:So4K3fa3oF98l5dQOdPolNgUYlU= 2023-09-22T21:26:29.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:So4K3fa3oF98l5dQOdPolNgUYlU=', 'Content-Length': '0'} 2023-09-22T21:26:29.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abed49fe1f771af6-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.643 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:26:29.643 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:26:29.643 DEBUG:boto:Method: HEAD 2023-09-22T21:26:29.643 DEBUG:boto:Path: /ffzdqu-55/c 2023-09-22T21:26:29.644 DEBUG:boto:Data: 2023-09-22T21:26:29.644 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.644 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.644 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.644 DEBUG:boto:Params: {} 2023-09-22T21:26:29.644 DEBUG:boto:Token: None 2023-09-22T21:26:29.644 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/c 2023-09-22T21:26:29.644 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:So4K3fa3oF98l5dQOdPolNgUYlU= 2023-09-22T21:26:29.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:So4K3fa3oF98l5dQOdPolNgUYlU=', 'Content-Length': '0'} 2023-09-22T21:26:29.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecd67c7437b31a99-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:26:29.647 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:26:29.648 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:26:29.648 DEBUG:boto:Method: GET 2023-09-22T21:26:29.648 DEBUG:boto:Path: /ffzdqu-55/c 2023-09-22T21:26:29.648 DEBUG:boto:Data: 2023-09-22T21:26:29.648 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.648 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.648 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.648 DEBUG:boto:Params: {} 2023-09-22T21:26:29.649 DEBUG:boto:Token: None 2023-09-22T21:26:29.649 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/c 2023-09-22T21:26:29.649 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:U1te/5ZC760atK9UGWivP9JqLQE= 2023-09-22T21:26:29.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:U1te/5ZC760atK9UGWivP9JqLQE=', 'Content-Length': '0'} 2023-09-22T21:26:29.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000947bf037d7c27c09-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.694 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:26:29.694 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:26:29.695 DEBUG:boto:Method: GET 2023-09-22T21:26:29.695 DEBUG:boto:Path: /ffzdqu-55/c 2023-09-22T21:26:29.695 DEBUG:boto:Data: 2023-09-22T21:26:29.695 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.695 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.695 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.695 DEBUG:boto:Params: {} 2023-09-22T21:26:29.695 DEBUG:boto:Token: None 2023-09-22T21:26:29.696 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/c 2023-09-22T21:26:29.696 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:U1te/5ZC760atK9UGWivP9JqLQE= 2023-09-22T21:26:29.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:U1te/5ZC760atK9UGWivP9JqLQE=', 'Content-Length': '0'} 2023-09-22T21:26:29.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afc1f5b70ed06d5e-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.699 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.699 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.699 DEBUG:boto:path=/ffzdqu-55/?prefix=c 2023-09-22T21:26:29.699 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=c 2023-09-22T21:26:29.699 DEBUG:boto:Method: GET 2023-09-22T21:26:29.700 DEBUG:boto:Path: /ffzdqu-55/?prefix=c 2023-09-22T21:26:29.700 DEBUG:boto:Data: 2023-09-22T21:26:29.700 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.700 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.700 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.700 DEBUG:boto:Params: {} 2023-09-22T21:26:29.700 DEBUG:boto:Token: None 2023-09-22T21:26:29.700 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/ 2023-09-22T21:26:29.700 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4= 2023-09-22T21:26:29.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4=', 'Content-Length': '0'} 2023-09-22T21:26:29.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000265883aee0f90691-00650e0685-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.706 DEBUG:boto:b'ffzdqu-55c1000falsec2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:29.707 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.707 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.707 DEBUG:boto:path=/ffzdqu-55/?prefix=c 2023-09-22T21:26:29.707 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=c 2023-09-22T21:26:29.707 DEBUG:boto:Method: GET 2023-09-22T21:26:29.707 DEBUG:boto:Path: /ffzdqu-55/?prefix=c 2023-09-22T21:26:29.707 DEBUG:boto:Data: 2023-09-22T21:26:29.707 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.708 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.708 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.708 DEBUG:boto:Params: {} 2023-09-22T21:26:29.708 DEBUG:boto:Token: None 2023-09-22T21:26:29.708 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/ 2023-09-22T21:26:29.708 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4= 2023-09-22T21:26:29.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4=', 'Content-Length': '0'} 2023-09-22T21:26:29.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb6d1daca3bd3a83-00650e0685-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.712 DEBUG:boto:b'ffzdqu-55c1000falsec2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:29.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:26:29.712 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:26:29.712 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:26:29.712 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.713 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.713 DEBUG:boto:Method: GET 2023-09-22T21:26:29.713 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.713 DEBUG:boto:Data: 2023-09-22T21:26:29.713 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.713 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.713 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.713 DEBUG:boto:Params: {} 2023-09-22T21:26:29.713 DEBUG:boto:Token: None 2023-09-22T21:26:29.714 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.714 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9/7umAdPXTWM6QjvIVCuvk9gPxM= 2023-09-22T21:26:29.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9/7umAdPXTWM6QjvIVCuvk9gPxM=', 'Content-Length': '0'} 2023-09-22T21:26:29.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042a85f5fed7abb12-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.717 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:26:29.717 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:26:29.717 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.717 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.717 DEBUG:boto:Method: GET 2023-09-22T21:26:29.717 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.717 DEBUG:boto:Data: 2023-09-22T21:26:29.718 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.718 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.718 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.718 DEBUG:boto:Params: {} 2023-09-22T21:26:29.718 DEBUG:boto:Token: None 2023-09-22T21:26:29.718 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.718 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9/7umAdPXTWM6QjvIVCuvk9gPxM= 2023-09-22T21:26:29.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9/7umAdPXTWM6QjvIVCuvk9gPxM=', 'Content-Length': '0'} 2023-09-22T21:26:29.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6b26397d88afea6-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.762 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:26:29.762 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:26:29.763 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.763 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.763 DEBUG:boto:Method: HEAD 2023-09-22T21:26:29.763 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.763 DEBUG:boto:Data: 2023-09-22T21:26:29.763 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.763 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.763 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.763 DEBUG:boto:Params: {} 2023-09-22T21:26:29.764 DEBUG:boto:Token: None 2023-09-22T21:26:29.764 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.764 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SOFI5kdKQfOZMaualeLo3I9sLl0= 2023-09-22T21:26:29.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SOFI5kdKQfOZMaualeLo3I9sLl0=', 'Content-Length': '0'} 2023-09-22T21:26:29.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b43f6c5d0ecbaff5-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.767 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:26:29.767 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:26:29.767 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.768 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.768 DEBUG:boto:Method: HEAD 2023-09-22T21:26:29.768 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.768 DEBUG:boto:Data: 2023-09-22T21:26:29.768 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.768 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.768 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.768 DEBUG:boto:Params: {} 2023-09-22T21:26:29.768 DEBUG:boto:Token: None 2023-09-22T21:26:29.768 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.769 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SOFI5kdKQfOZMaualeLo3I9sLl0= 2023-09-22T21:26:29.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SOFI5kdKQfOZMaualeLo3I9sLl0=', 'Content-Length': '0'} 2023-09-22T21:26:29.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000651de73707f7aadc-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:26:29.772 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:26:29.772 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:26:29.772 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.772 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.772 DEBUG:boto:Method: GET 2023-09-22T21:26:29.772 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.772 DEBUG:boto:Data: 2023-09-22T21:26:29.772 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.773 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.773 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.773 DEBUG:boto:Params: {} 2023-09-22T21:26:29.773 DEBUG:boto:Token: None 2023-09-22T21:26:29.773 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.773 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9/7umAdPXTWM6QjvIVCuvk9gPxM= 2023-09-22T21:26:29.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9/7umAdPXTWM6QjvIVCuvk9gPxM=', 'Content-Length': '0'} 2023-09-22T21:26:29.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008924fda1127d0bc1-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.818 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:26:29.818 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:26:29.818 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.819 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.819 DEBUG:boto:Method: GET 2023-09-22T21:26:29.819 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.819 DEBUG:boto:Data: 2023-09-22T21:26:29.819 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.819 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.819 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.819 DEBUG:boto:Params: {} 2023-09-22T21:26:29.819 DEBUG:boto:Token: None 2023-09-22T21:26:29.820 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:26:29.820 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9/7umAdPXTWM6QjvIVCuvk9gPxM= 2023-09-22T21:26:29.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9/7umAdPXTWM6QjvIVCuvk9gPxM=', 'Content-Length': '0'} 2023-09-22T21:26:29.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b660a8dcae5caf25-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.823 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.823 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.823 DEBUG:boto:path=/ffzdqu-55/?prefix=d 2023-09-22T21:26:29.823 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=d 2023-09-22T21:26:29.823 DEBUG:boto:Method: GET 2023-09-22T21:26:29.823 DEBUG:boto:Path: /ffzdqu-55/?prefix=d 2023-09-22T21:26:29.823 DEBUG:boto:Data: 2023-09-22T21:26:29.824 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.824 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.824 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.824 DEBUG:boto:Params: {} 2023-09-22T21:26:29.824 DEBUG:boto:Token: None 2023-09-22T21:26:29.824 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/ 2023-09-22T21:26:29.824 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4= 2023-09-22T21:26:29.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4=', 'Content-Length': '0'} 2023-09-22T21:26:29.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f7df7e8f0473c7c-00650e0685-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.831 DEBUG:boto:b'ffzdqu-55d1000falsed2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:29.831 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.831 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.831 DEBUG:boto:path=/ffzdqu-55/?prefix=d 2023-09-22T21:26:29.832 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=d 2023-09-22T21:26:29.832 DEBUG:boto:Method: GET 2023-09-22T21:26:29.832 DEBUG:boto:Path: /ffzdqu-55/?prefix=d 2023-09-22T21:26:29.832 DEBUG:boto:Data: 2023-09-22T21:26:29.832 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.832 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.832 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.832 DEBUG:boto:Params: {} 2023-09-22T21:26:29.832 DEBUG:boto:Token: None 2023-09-22T21:26:29.833 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/ 2023-09-22T21:26:29.833 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4= 2023-09-22T21:26:29.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4=', 'Content-Length': '0'} 2023-09-22T21:26:29.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6707380679f80ae-00650e0685-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.836 DEBUG:boto:b'ffzdqu-55d1000falsed2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:29.836 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:26:29.837 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:26:29.837 DEBUG:boto:Method: HEAD 2023-09-22T21:26:29.837 DEBUG:boto:Path: /ffzdqu-55/d 2023-09-22T21:26:29.837 DEBUG:boto:Data: 2023-09-22T21:26:29.837 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.837 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.837 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.837 DEBUG:boto:Params: {} 2023-09-22T21:26:29.838 DEBUG:boto:Token: None 2023-09-22T21:26:29.838 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/d 2023-09-22T21:26:29.838 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6b9yelzZon+fsjOu1pxdCVQVLx0= 2023-09-22T21:26:29.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6b9yelzZon+fsjOu1pxdCVQVLx0=', 'Content-Length': '0'} 2023-09-22T21:26:29.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027776a6903f92921-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.841 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:26:29.841 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:26:29.841 DEBUG:boto:Method: HEAD 2023-09-22T21:26:29.841 DEBUG:boto:Path: /ffzdqu-55/d 2023-09-22T21:26:29.841 DEBUG:boto:Data: 2023-09-22T21:26:29.841 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.841 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.841 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.841 DEBUG:boto:Params: {} 2023-09-22T21:26:29.842 DEBUG:boto:Token: None 2023-09-22T21:26:29.842 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/d 2023-09-22T21:26:29.842 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6b9yelzZon+fsjOu1pxdCVQVLx0= 2023-09-22T21:26:29.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6b9yelzZon+fsjOu1pxdCVQVLx0=', 'Content-Length': '0'} 2023-09-22T21:26:29.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3ad2a4bacb63586-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:26:29.845 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:26:29.845 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:26:29.845 DEBUG:boto:Method: GET 2023-09-22T21:26:29.845 DEBUG:boto:Path: /ffzdqu-55/d 2023-09-22T21:26:29.845 DEBUG:boto:Data: 2023-09-22T21:26:29.845 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.846 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.846 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.846 DEBUG:boto:Params: {} 2023-09-22T21:26:29.846 DEBUG:boto:Token: None 2023-09-22T21:26:29.846 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/d 2023-09-22T21:26:29.846 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Q0CZfSQ9VbxwiUlKkOhEiFBIpxY= 2023-09-22T21:26:29.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Q0CZfSQ9VbxwiUlKkOhEiFBIpxY=', 'Content-Length': '0'} 2023-09-22T21:26:29.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092a9966bad00df8d-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.890 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:26:29.891 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:26:29.891 DEBUG:boto:Method: GET 2023-09-22T21:26:29.891 DEBUG:boto:Path: /ffzdqu-55/d 2023-09-22T21:26:29.891 DEBUG:boto:Data: 2023-09-22T21:26:29.891 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.891 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.891 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.891 DEBUG:boto:Params: {} 2023-09-22T21:26:29.892 DEBUG:boto:Token: None 2023-09-22T21:26:29.892 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/d 2023-09-22T21:26:29.892 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Q0CZfSQ9VbxwiUlKkOhEiFBIpxY= 2023-09-22T21:26:29.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Q0CZfSQ9VbxwiUlKkOhEiFBIpxY=', 'Content-Length': '0'} 2023-09-22T21:26:29.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099afbb26599335d0-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.895 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.895 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.895 DEBUG:boto:path=/ffzdqu-55/?prefix=d 2023-09-22T21:26:29.895 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=d 2023-09-22T21:26:29.896 DEBUG:boto:Method: GET 2023-09-22T21:26:29.896 DEBUG:boto:Path: /ffzdqu-55/?prefix=d 2023-09-22T21:26:29.896 DEBUG:boto:Data: 2023-09-22T21:26:29.896 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.896 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.896 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.896 DEBUG:boto:Params: {} 2023-09-22T21:26:29.896 DEBUG:boto:Token: None 2023-09-22T21:26:29.896 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/ 2023-09-22T21:26:29.897 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4= 2023-09-22T21:26:29.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4=', 'Content-Length': '0'} 2023-09-22T21:26:29.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068d1ba529ef9c71b-00650e0685-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.942 DEBUG:boto:b'ffzdqu-55d1000falsed2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:29.943 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:29.943 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:29.943 DEBUG:boto:path=/ffzdqu-55/?prefix=d 2023-09-22T21:26:29.943 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=d 2023-09-22T21:26:29.943 DEBUG:boto:Method: GET 2023-09-22T21:26:29.943 DEBUG:boto:Path: /ffzdqu-55/?prefix=d 2023-09-22T21:26:29.943 DEBUG:boto:Data: 2023-09-22T21:26:29.943 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.944 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.944 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.944 DEBUG:boto:Params: {} 2023-09-22T21:26:29.944 DEBUG:boto:Token: None 2023-09-22T21:26:29.944 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/ 2023-09-22T21:26:29.944 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4= 2023-09-22T21:26:29.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:frP+VWVRmUfywELya/fUDj3GIC4=', 'Content-Length': '0'} 2023-09-22T21:26:29.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9926a8e10dc4295-00650e0685-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.948 DEBUG:boto:b'ffzdqu-55d1000falsed2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:29.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-22T21:26:29.948 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:26:29.948 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:26:29.948 DEBUG:boto:path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:29.949 DEBUG:boto:auth_path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:29.949 DEBUG:boto:Method: GET 2023-09-22T21:26:29.949 DEBUG:boto:Path: /ffzdqu-55/e?versionId=null 2023-09-22T21:26:29.949 DEBUG:boto:Data: 2023-09-22T21:26:29.949 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.949 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.949 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.949 DEBUG:boto:Params: {} 2023-09-22T21:26:29.949 DEBUG:boto:Token: None 2023-09-22T21:26:29.950 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/e?versionId=null 2023-09-22T21:26:29.950 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JPirE8S80Eh9iVIfDkoAsIezlHk= 2023-09-22T21:26:29.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JPirE8S80Eh9iVIfDkoAsIezlHk=', 'Content-Length': '0'} 2023-09-22T21:26:29.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000789fe16dae9437c5-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.953 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:26:29.953 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:26:29.953 DEBUG:boto:path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:29.953 DEBUG:boto:auth_path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:29.953 DEBUG:boto:Method: GET 2023-09-22T21:26:29.954 DEBUG:boto:Path: /ffzdqu-55/e?versionId=null 2023-09-22T21:26:29.954 DEBUG:boto:Data: 2023-09-22T21:26:29.954 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.954 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.954 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.954 DEBUG:boto:Params: {} 2023-09-22T21:26:29.955 DEBUG:boto:Token: None 2023-09-22T21:26:29.955 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/e?versionId=null 2023-09-22T21:26:29.955 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JPirE8S80Eh9iVIfDkoAsIezlHk= 2023-09-22T21:26:29.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JPirE8S80Eh9iVIfDkoAsIezlHk=', 'Content-Length': '0'} 2023-09-22T21:26:29.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006183c4c81dcae614-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.958 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:26:29.958 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:26:29.958 DEBUG:boto:path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:29.958 DEBUG:boto:auth_path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:29.959 DEBUG:boto:Method: HEAD 2023-09-22T21:26:29.959 DEBUG:boto:Path: /ffzdqu-55/e?versionId=null 2023-09-22T21:26:29.959 DEBUG:boto:Data: 2023-09-22T21:26:29.959 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.959 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.959 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.959 DEBUG:boto:Params: {} 2023-09-22T21:26:29.959 DEBUG:boto:Token: None 2023-09-22T21:26:29.959 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/e?versionId=null 2023-09-22T21:26:29.960 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ql7aUWvmkrE+FiJSV4GSHlz73QM= 2023-09-22T21:26:29.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ql7aUWvmkrE+FiJSV4GSHlz73QM=', 'Content-Length': '0'} 2023-09-22T21:26:29.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a67c32f7adfe1a00-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.963 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:26:29.963 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:26:29.963 DEBUG:boto:path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:29.963 DEBUG:boto:auth_path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:29.963 DEBUG:boto:Method: HEAD 2023-09-22T21:26:29.963 DEBUG:boto:Path: /ffzdqu-55/e?versionId=null 2023-09-22T21:26:29.963 DEBUG:boto:Data: 2023-09-22T21:26:29.963 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.963 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:29.964 DEBUG:boto:Port: 8000 2023-09-22T21:26:29.964 DEBUG:boto:Params: {} 2023-09-22T21:26:29.964 DEBUG:boto:Token: None 2023-09-22T21:26:29.964 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/e?versionId=null 2023-09-22T21:26:29.964 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ql7aUWvmkrE+FiJSV4GSHlz73QM= 2023-09-22T21:26:29.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ql7aUWvmkrE+FiJSV4GSHlz73QM=', 'Content-Length': '0'} 2023-09-22T21:26:29.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043a3155c01138c91-00650e0685-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:29.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-22T21:26:29.967 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:26:29.968 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:26:29.968 DEBUG:boto:path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:29.968 DEBUG:boto:auth_path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:29.968 DEBUG:boto:Method: GET 2023-09-22T21:26:29.968 DEBUG:boto:Path: /ffzdqu-55/e?versionId=null 2023-09-22T21:26:29.968 DEBUG:boto:Data: 2023-09-22T21:26:29.968 DEBUG:boto:Headers: {} 2023-09-22T21:26:29.968 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:29.968 DEBUG:boto:Port: 8001 2023-09-22T21:26:29.968 DEBUG:boto:Params: {} 2023-09-22T21:26:29.969 DEBUG:boto:Token: None 2023-09-22T21:26:29.969 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:29 GMT /ffzdqu-55/e?versionId=null 2023-09-22T21:26:29.969 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JPirE8S80Eh9iVIfDkoAsIezlHk= 2023-09-22T21:26:29.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JPirE8S80Eh9iVIfDkoAsIezlHk=', 'Content-Length': '0'} 2023-09-22T21:26:29.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082b164960dc226f5-00650e0685-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.014 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:26:30.014 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:26:30.015 DEBUG:boto:path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:30.015 DEBUG:boto:auth_path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:30.015 DEBUG:boto:Method: GET 2023-09-22T21:26:30.015 DEBUG:boto:Path: /ffzdqu-55/e?versionId=null 2023-09-22T21:26:30.015 DEBUG:boto:Data: 2023-09-22T21:26:30.015 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.015 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.015 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.015 DEBUG:boto:Params: {} 2023-09-22T21:26:30.016 DEBUG:boto:Token: None 2023-09-22T21:26:30.016 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/e?versionId=null 2023-09-22T21:26:30.016 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DKSKh66CaniPV+SnciUxxFSZyRA= 2023-09-22T21:26:30.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DKSKh66CaniPV+SnciUxxFSZyRA=', 'Content-Length': '0'} 2023-09-22T21:26:30.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036ab3c226d45c4d8-00650e0686-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.019 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:30.019 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:30.019 DEBUG:boto:path=/ffzdqu-55/?prefix=e 2023-09-22T21:26:30.020 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=e 2023-09-22T21:26:30.020 DEBUG:boto:Method: GET 2023-09-22T21:26:30.020 DEBUG:boto:Path: /ffzdqu-55/?prefix=e 2023-09-22T21:26:30.020 DEBUG:boto:Data: 2023-09-22T21:26:30.020 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.020 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.020 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.020 DEBUG:boto:Params: {} 2023-09-22T21:26:30.020 DEBUG:boto:Token: None 2023-09-22T21:26:30.021 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/ 2023-09-22T21:26:30.021 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ= 2023-09-22T21:26:30.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ=', 'Content-Length': '0'} 2023-09-22T21:26:30.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009aa77f98eab89767-00650e0686-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.027 DEBUG:boto:b'ffzdqu-55e1000falsee2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:30.027 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:30.027 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:30.028 DEBUG:boto:path=/ffzdqu-55/?prefix=e 2023-09-22T21:26:30.028 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=e 2023-09-22T21:26:30.028 DEBUG:boto:Method: GET 2023-09-22T21:26:30.028 DEBUG:boto:Path: /ffzdqu-55/?prefix=e 2023-09-22T21:26:30.028 DEBUG:boto:Data: 2023-09-22T21:26:30.028 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.028 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.028 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.028 DEBUG:boto:Params: {} 2023-09-22T21:26:30.029 DEBUG:boto:Token: None 2023-09-22T21:26:30.029 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/ 2023-09-22T21:26:30.029 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ= 2023-09-22T21:26:30.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ=', 'Content-Length': '0'} 2023-09-22T21:26:30.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8abea9cdac2f2ff-00650e0686-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.032 DEBUG:boto:b'ffzdqu-55e1000falsee2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:30.033 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:26:30.033 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:26:30.033 DEBUG:boto:Method: HEAD 2023-09-22T21:26:30.033 DEBUG:boto:Path: /ffzdqu-55/e 2023-09-22T21:26:30.033 DEBUG:boto:Data: 2023-09-22T21:26:30.033 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.034 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.034 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.034 DEBUG:boto:Params: {} 2023-09-22T21:26:30.034 DEBUG:boto:Token: None 2023-09-22T21:26:30.034 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/e 2023-09-22T21:26:30.034 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mTZAffn4UpmOKRtqViAQ630t97c= 2023-09-22T21:26:30.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mTZAffn4UpmOKRtqViAQ630t97c=', 'Content-Length': '0'} 2023-09-22T21:26:30.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050bb826449ee77f5-00650e0686-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.037 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:26:30.037 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:26:30.037 DEBUG:boto:Method: HEAD 2023-09-22T21:26:30.038 DEBUG:boto:Path: /ffzdqu-55/e 2023-09-22T21:26:30.038 DEBUG:boto:Data: 2023-09-22T21:26:30.038 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.038 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.038 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.038 DEBUG:boto:Params: {} 2023-09-22T21:26:30.038 DEBUG:boto:Token: None 2023-09-22T21:26:30.038 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/e 2023-09-22T21:26:30.038 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mTZAffn4UpmOKRtqViAQ630t97c= 2023-09-22T21:26:30.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mTZAffn4UpmOKRtqViAQ630t97c=', 'Content-Length': '0'} 2023-09-22T21:26:30.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003119ff1e7f1151a-00650e0686-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-22T21:26:30.042 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:26:30.042 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:26:30.042 DEBUG:boto:Method: GET 2023-09-22T21:26:30.043 DEBUG:boto:Path: /ffzdqu-55/e 2023-09-22T21:26:30.043 DEBUG:boto:Data: 2023-09-22T21:26:30.043 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.043 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.043 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.043 DEBUG:boto:Params: {} 2023-09-22T21:26:30.043 DEBUG:boto:Token: None 2023-09-22T21:26:30.043 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/e 2023-09-22T21:26:30.043 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3cBmvVo53mObnR7fe0orqZPjHMQ= 2023-09-22T21:26:30.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3cBmvVo53mObnR7fe0orqZPjHMQ=', 'Content-Length': '0'} 2023-09-22T21:26:30.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042b356758d16697a-00650e0686-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.090 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:26:30.091 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:26:30.091 DEBUG:boto:Method: GET 2023-09-22T21:26:30.091 DEBUG:boto:Path: /ffzdqu-55/e 2023-09-22T21:26:30.091 DEBUG:boto:Data: 2023-09-22T21:26:30.091 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.091 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.091 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.091 DEBUG:boto:Params: {} 2023-09-22T21:26:30.091 DEBUG:boto:Token: None 2023-09-22T21:26:30.092 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/e 2023-09-22T21:26:30.092 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3cBmvVo53mObnR7fe0orqZPjHMQ= 2023-09-22T21:26:30.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3cBmvVo53mObnR7fe0orqZPjHMQ=', 'Content-Length': '0'} 2023-09-22T21:26:30.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc9833c5667fd63a-00650e0686-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.095 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:30.095 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:30.095 DEBUG:boto:path=/ffzdqu-55/?prefix=e 2023-09-22T21:26:30.096 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=e 2023-09-22T21:26:30.096 DEBUG:boto:Method: GET 2023-09-22T21:26:30.096 DEBUG:boto:Path: /ffzdqu-55/?prefix=e 2023-09-22T21:26:30.096 DEBUG:boto:Data: 2023-09-22T21:26:30.096 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.096 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.096 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.096 DEBUG:boto:Params: {} 2023-09-22T21:26:30.096 DEBUG:boto:Token: None 2023-09-22T21:26:30.097 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/ 2023-09-22T21:26:30.097 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ= 2023-09-22T21:26:30.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ=', 'Content-Length': '0'} 2023-09-22T21:26:30.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030a50c3284f4134a-00650e0686-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.102 DEBUG:boto:b'ffzdqu-55e1000falsee2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:30.102 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:30.103 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:30.103 DEBUG:boto:path=/ffzdqu-55/?prefix=e 2023-09-22T21:26:30.103 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=e 2023-09-22T21:26:30.103 DEBUG:boto:Method: GET 2023-09-22T21:26:30.103 DEBUG:boto:Path: /ffzdqu-55/?prefix=e 2023-09-22T21:26:30.103 DEBUG:boto:Data: 2023-09-22T21:26:30.103 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.103 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.103 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.104 DEBUG:boto:Params: {} 2023-09-22T21:26:30.104 DEBUG:boto:Token: None 2023-09-22T21:26:30.104 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/ 2023-09-22T21:26:30.104 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ= 2023-09-22T21:26:30.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ=', 'Content-Length': '0'} 2023-09-22T21:26:30.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ba02ab352267ff6-00650e0686-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.108 DEBUG:boto:b'ffzdqu-55e1000falsee2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:30.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-22T21:26:30.108 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:26:30.108 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:26:30.108 DEBUG:boto:path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.108 DEBUG:boto:auth_path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.109 DEBUG:boto:Method: GET 2023-09-22T21:26:30.109 DEBUG:boto:Path: /ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.109 DEBUG:boto:Data: 2023-09-22T21:26:30.109 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.109 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.109 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.109 DEBUG:boto:Params: {} 2023-09-22T21:26:30.109 DEBUG:boto:Token: None 2023-09-22T21:26:30.109 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.110 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dzK0drpq6vSC9L3CKkymv2q3prk= 2023-09-22T21:26:30.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dzK0drpq6vSC9L3CKkymv2q3prk=', 'Content-Length': '0'} 2023-09-22T21:26:30.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018e5d883d03c7d58-00650e0686-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.113 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:26:30.113 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:26:30.113 DEBUG:boto:path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.113 DEBUG:boto:auth_path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.113 DEBUG:boto:Method: GET 2023-09-22T21:26:30.113 DEBUG:boto:Path: /ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.113 DEBUG:boto:Data: 2023-09-22T21:26:30.113 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.114 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.114 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.114 DEBUG:boto:Params: {} 2023-09-22T21:26:30.114 DEBUG:boto:Token: None 2023-09-22T21:26:30.114 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.114 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dzK0drpq6vSC9L3CKkymv2q3prk= 2023-09-22T21:26:30.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dzK0drpq6vSC9L3CKkymv2q3prk=', 'Content-Length': '0'} 2023-09-22T21:26:30.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037d63926c32c36d8-00650e0686-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.117 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:26:30.118 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:26:30.118 DEBUG:boto:path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.118 DEBUG:boto:auth_path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.118 DEBUG:boto:Method: HEAD 2023-09-22T21:26:30.118 DEBUG:boto:Path: /ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.118 DEBUG:boto:Data: 2023-09-22T21:26:30.118 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.118 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.118 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.118 DEBUG:boto:Params: {} 2023-09-22T21:26:30.119 DEBUG:boto:Token: None 2023-09-22T21:26:30.119 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.119 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cfHlsAPkRcVE9WfRX/L0T6TEc7E= 2023-09-22T21:26:30.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cfHlsAPkRcVE9WfRX/L0T6TEc7E=', 'Content-Length': '0'} 2023-09-22T21:26:30.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000711b29b39832a4a5-00650e0686-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.122 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:26:30.122 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:26:30.122 DEBUG:boto:path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.122 DEBUG:boto:auth_path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.122 DEBUG:boto:Method: HEAD 2023-09-22T21:26:30.122 DEBUG:boto:Path: /ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.122 DEBUG:boto:Data: 2023-09-22T21:26:30.123 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.123 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.123 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.123 DEBUG:boto:Params: {} 2023-09-22T21:26:30.123 DEBUG:boto:Token: None 2023-09-22T21:26:30.123 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.123 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cfHlsAPkRcVE9WfRX/L0T6TEc7E= 2023-09-22T21:26:30.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cfHlsAPkRcVE9WfRX/L0T6TEc7E=', 'Content-Length': '0'} 2023-09-22T21:26:30.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bca4d23cdf57fa0-00650e0686-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-22T21:26:30.127 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:26:30.127 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:26:30.127 DEBUG:boto:path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.127 DEBUG:boto:auth_path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.127 DEBUG:boto:Method: GET 2023-09-22T21:26:30.127 DEBUG:boto:Path: /ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.127 DEBUG:boto:Data: 2023-09-22T21:26:30.127 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.127 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.128 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.128 DEBUG:boto:Params: {} 2023-09-22T21:26:30.128 DEBUG:boto:Token: None 2023-09-22T21:26:30.128 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.128 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dzK0drpq6vSC9L3CKkymv2q3prk= 2023-09-22T21:26:30.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dzK0drpq6vSC9L3CKkymv2q3prk=', 'Content-Length': '0'} 2023-09-22T21:26:30.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067fefcf38b276bcd-00650e0686-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.174 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:26:30.175 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:26:30.175 DEBUG:boto:path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.175 DEBUG:boto:auth_path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.175 DEBUG:boto:Method: GET 2023-09-22T21:26:30.175 DEBUG:boto:Path: /ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.175 DEBUG:boto:Data: 2023-09-22T21:26:30.175 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.175 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.175 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.176 DEBUG:boto:Params: {} 2023-09-22T21:26:30.176 DEBUG:boto:Token: None 2023-09-22T21:26:30.176 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/f?versionId=null 2023-09-22T21:26:30.176 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dzK0drpq6vSC9L3CKkymv2q3prk= 2023-09-22T21:26:30.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dzK0drpq6vSC9L3CKkymv2q3prk=', 'Content-Length': '0'} 2023-09-22T21:26:30.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c8bceeacb4d7674-00650e0686-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.179 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:30.179 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:30.180 DEBUG:boto:path=/ffzdqu-55/?prefix=f 2023-09-22T21:26:30.180 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=f 2023-09-22T21:26:30.180 DEBUG:boto:Method: GET 2023-09-22T21:26:30.180 DEBUG:boto:Path: /ffzdqu-55/?prefix=f 2023-09-22T21:26:30.180 DEBUG:boto:Data: 2023-09-22T21:26:30.180 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.180 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.180 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.180 DEBUG:boto:Params: {} 2023-09-22T21:26:30.181 DEBUG:boto:Token: None 2023-09-22T21:26:30.181 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/ 2023-09-22T21:26:30.181 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ= 2023-09-22T21:26:30.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ=', 'Content-Length': '0'} 2023-09-22T21:26:30.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f5b11abaa608eb8-00650e0686-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.226 DEBUG:boto:b'ffzdqu-55f1000falsef2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:30.227 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:30.227 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:30.227 DEBUG:boto:path=/ffzdqu-55/?prefix=f 2023-09-22T21:26:30.227 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=f 2023-09-22T21:26:30.227 DEBUG:boto:Method: GET 2023-09-22T21:26:30.227 DEBUG:boto:Path: /ffzdqu-55/?prefix=f 2023-09-22T21:26:30.227 DEBUG:boto:Data: 2023-09-22T21:26:30.228 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.228 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.228 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.228 DEBUG:boto:Params: {} 2023-09-22T21:26:30.228 DEBUG:boto:Token: None 2023-09-22T21:26:30.228 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/ 2023-09-22T21:26:30.228 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ= 2023-09-22T21:26:30.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ=', 'Content-Length': '0'} 2023-09-22T21:26:30.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e412beb9851dd7f-00650e0686-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.232 DEBUG:boto:b'ffzdqu-55f1000falsef2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:30.232 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:26:30.232 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:26:30.232 DEBUG:boto:Method: HEAD 2023-09-22T21:26:30.233 DEBUG:boto:Path: /ffzdqu-55/f 2023-09-22T21:26:30.233 DEBUG:boto:Data: 2023-09-22T21:26:30.233 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.233 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.233 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.233 DEBUG:boto:Params: {} 2023-09-22T21:26:30.233 DEBUG:boto:Token: None 2023-09-22T21:26:30.233 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/f 2023-09-22T21:26:30.234 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8xVIolk8NcsFxLSrvE8sDMHZQE4= 2023-09-22T21:26:30.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8xVIolk8NcsFxLSrvE8sDMHZQE4=', 'Content-Length': '0'} 2023-09-22T21:26:30.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c4ac46c30221499-00650e0686-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.237 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:26:30.237 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:26:30.237 DEBUG:boto:Method: HEAD 2023-09-22T21:26:30.237 DEBUG:boto:Path: /ffzdqu-55/f 2023-09-22T21:26:30.237 DEBUG:boto:Data: 2023-09-22T21:26:30.237 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.237 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.237 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.238 DEBUG:boto:Params: {} 2023-09-22T21:26:30.238 DEBUG:boto:Token: None 2023-09-22T21:26:30.238 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/f 2023-09-22T21:26:30.238 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8xVIolk8NcsFxLSrvE8sDMHZQE4= 2023-09-22T21:26:30.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8xVIolk8NcsFxLSrvE8sDMHZQE4=', 'Content-Length': '0'} 2023-09-22T21:26:30.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcecf9c560531291-00650e0686-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-22T21:26:30.241 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:26:30.241 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:26:30.241 DEBUG:boto:Method: GET 2023-09-22T21:26:30.241 DEBUG:boto:Path: /ffzdqu-55/f 2023-09-22T21:26:30.242 DEBUG:boto:Data: 2023-09-22T21:26:30.242 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.242 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.242 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.242 DEBUG:boto:Params: {} 2023-09-22T21:26:30.242 DEBUG:boto:Token: None 2023-09-22T21:26:30.242 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/f 2023-09-22T21:26:30.242 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:epcTRjTBbUWRyQpWtbDQ6026ocI= 2023-09-22T21:26:30.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:epcTRjTBbUWRyQpWtbDQ6026ocI=', 'Content-Length': '0'} 2023-09-22T21:26:30.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a9a83c1c218c634-00650e0686-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.286 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:26:30.286 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:26:30.287 DEBUG:boto:Method: GET 2023-09-22T21:26:30.287 DEBUG:boto:Path: /ffzdqu-55/f 2023-09-22T21:26:30.287 DEBUG:boto:Data: 2023-09-22T21:26:30.287 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.287 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.287 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.287 DEBUG:boto:Params: {} 2023-09-22T21:26:30.287 DEBUG:boto:Token: None 2023-09-22T21:26:30.288 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/f 2023-09-22T21:26:30.288 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:epcTRjTBbUWRyQpWtbDQ6026ocI= 2023-09-22T21:26:30.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:epcTRjTBbUWRyQpWtbDQ6026ocI=', 'Content-Length': '0'} 2023-09-22T21:26:30.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000940b21a6fd7292ab-00650e0686-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.291 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:30.291 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:30.291 DEBUG:boto:path=/ffzdqu-55/?prefix=f 2023-09-22T21:26:30.291 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=f 2023-09-22T21:26:30.291 DEBUG:boto:Method: GET 2023-09-22T21:26:30.291 DEBUG:boto:Path: /ffzdqu-55/?prefix=f 2023-09-22T21:26:30.291 DEBUG:boto:Data: 2023-09-22T21:26:30.291 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.291 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.292 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.292 DEBUG:boto:Params: {} 2023-09-22T21:26:30.292 DEBUG:boto:Token: None 2023-09-22T21:26:30.292 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/ 2023-09-22T21:26:30.292 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ= 2023-09-22T21:26:30.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ=', 'Content-Length': '0'} 2023-09-22T21:26:30.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000632428632920038d-00650e0686-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.338 DEBUG:boto:b'ffzdqu-55f1000falsef2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:30.339 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:30.339 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:30.339 DEBUG:boto:path=/ffzdqu-55/?prefix=f 2023-09-22T21:26:30.339 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=f 2023-09-22T21:26:30.339 DEBUG:boto:Method: GET 2023-09-22T21:26:30.339 DEBUG:boto:Path: /ffzdqu-55/?prefix=f 2023-09-22T21:26:30.340 DEBUG:boto:Data: 2023-09-22T21:26:30.340 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.340 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.340 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.340 DEBUG:boto:Params: {} 2023-09-22T21:26:30.340 DEBUG:boto:Token: None 2023-09-22T21:26:30.340 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/ 2023-09-22T21:26:30.340 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ= 2023-09-22T21:26:30.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ=', 'Content-Length': '0'} 2023-09-22T21:26:30.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d2996b0b6248eea-00650e0686-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.344 DEBUG:boto:b'ffzdqu-55f1000falsef2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:30.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-22T21:26:30.345 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:26:30.345 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:26:30.345 DEBUG:boto:path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.345 DEBUG:boto:auth_path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.345 DEBUG:boto:Method: GET 2023-09-22T21:26:30.345 DEBUG:boto:Path: /ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.345 DEBUG:boto:Data: 2023-09-22T21:26:30.345 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.345 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.346 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.346 DEBUG:boto:Params: {} 2023-09-22T21:26:30.346 DEBUG:boto:Token: None 2023-09-22T21:26:30.346 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.346 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mFwxfS3hm2eyVE8u9vaiRunetAk= 2023-09-22T21:26:30.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mFwxfS3hm2eyVE8u9vaiRunetAk=', 'Content-Length': '0'} 2023-09-22T21:26:30.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b895f4419758e9f5-00650e0686-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.391 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:26:30.391 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:26:30.391 DEBUG:boto:path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.391 DEBUG:boto:auth_path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.391 DEBUG:boto:Method: GET 2023-09-22T21:26:30.391 DEBUG:boto:Path: /ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.392 DEBUG:boto:Data: 2023-09-22T21:26:30.392 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.392 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.392 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.392 DEBUG:boto:Params: {} 2023-09-22T21:26:30.392 DEBUG:boto:Token: None 2023-09-22T21:26:30.392 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.393 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mFwxfS3hm2eyVE8u9vaiRunetAk= 2023-09-22T21:26:30.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mFwxfS3hm2eyVE8u9vaiRunetAk=', 'Content-Length': '0'} 2023-09-22T21:26:30.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d48f4460c16ad18f-00650e0686-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.396 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:26:30.396 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:26:30.396 DEBUG:boto:path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.397 DEBUG:boto:auth_path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.397 DEBUG:boto:Method: HEAD 2023-09-22T21:26:30.397 DEBUG:boto:Path: /ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.397 DEBUG:boto:Data: 2023-09-22T21:26:30.397 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.397 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.397 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.397 DEBUG:boto:Params: {} 2023-09-22T21:26:30.398 DEBUG:boto:Token: None 2023-09-22T21:26:30.398 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.398 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Qz277t+tyqfSt6AcXF7xULF2f7s= 2023-09-22T21:26:30.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Qz277t+tyqfSt6AcXF7xULF2f7s=', 'Content-Length': '0'} 2023-09-22T21:26:30.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000030d9e8c5a11bd3f-00650e0686-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.401 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:26:30.401 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:26:30.401 DEBUG:boto:path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.401 DEBUG:boto:auth_path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.401 DEBUG:boto:Method: HEAD 2023-09-22T21:26:30.401 DEBUG:boto:Path: /ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.401 DEBUG:boto:Data: 2023-09-22T21:26:30.402 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.402 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.402 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.402 DEBUG:boto:Params: {} 2023-09-22T21:26:30.402 DEBUG:boto:Token: None 2023-09-22T21:26:30.402 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.402 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Qz277t+tyqfSt6AcXF7xULF2f7s= 2023-09-22T21:26:30.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Qz277t+tyqfSt6AcXF7xULF2f7s=', 'Content-Length': '0'} 2023-09-22T21:26:30.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c09ebf0ae9cf243-00650e0686-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-22T21:26:30.406 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:26:30.406 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:26:30.406 DEBUG:boto:path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.406 DEBUG:boto:auth_path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.407 DEBUG:boto:Method: GET 2023-09-22T21:26:30.407 DEBUG:boto:Path: /ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.407 DEBUG:boto:Data: 2023-09-22T21:26:30.407 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.407 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.407 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.407 DEBUG:boto:Params: {} 2023-09-22T21:26:30.407 DEBUG:boto:Token: None 2023-09-22T21:26:30.408 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.408 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mFwxfS3hm2eyVE8u9vaiRunetAk= 2023-09-22T21:26:30.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mFwxfS3hm2eyVE8u9vaiRunetAk=', 'Content-Length': '0'} 2023-09-22T21:26:30.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c22cc2f888b8dcfe-00650e0686-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.454 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:26:30.454 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:26:30.455 DEBUG:boto:path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.455 DEBUG:boto:auth_path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.455 DEBUG:boto:Method: GET 2023-09-22T21:26:30.455 DEBUG:boto:Path: /ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.455 DEBUG:boto:Data: 2023-09-22T21:26:30.455 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.455 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.455 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.456 DEBUG:boto:Params: {} 2023-09-22T21:26:30.456 DEBUG:boto:Token: None 2023-09-22T21:26:30.456 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/g?versionId=null 2023-09-22T21:26:30.456 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mFwxfS3hm2eyVE8u9vaiRunetAk= 2023-09-22T21:26:30.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mFwxfS3hm2eyVE8u9vaiRunetAk=', 'Content-Length': '0'} 2023-09-22T21:26:30.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000203f07d6133b6a3b-00650e0686-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.459 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:30.460 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:30.460 DEBUG:boto:path=/ffzdqu-55/?prefix=g 2023-09-22T21:26:30.460 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=g 2023-09-22T21:26:30.460 DEBUG:boto:Method: GET 2023-09-22T21:26:30.460 DEBUG:boto:Path: /ffzdqu-55/?prefix=g 2023-09-22T21:26:30.460 DEBUG:boto:Data: 2023-09-22T21:26:30.460 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.460 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.460 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.461 DEBUG:boto:Params: {} 2023-09-22T21:26:30.461 DEBUG:boto:Token: None 2023-09-22T21:26:30.461 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/ 2023-09-22T21:26:30.461 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ= 2023-09-22T21:26:30.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ=', 'Content-Length': '0'} 2023-09-22T21:26:30.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f16c1c579bc4c190-00650e0686-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.467 DEBUG:boto:b'ffzdqu-55g1000falseg2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:30.467 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:30.467 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:30.468 DEBUG:boto:path=/ffzdqu-55/?prefix=g 2023-09-22T21:26:30.468 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=g 2023-09-22T21:26:30.468 DEBUG:boto:Method: GET 2023-09-22T21:26:30.468 DEBUG:boto:Path: /ffzdqu-55/?prefix=g 2023-09-22T21:26:30.468 DEBUG:boto:Data: 2023-09-22T21:26:30.468 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.468 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.468 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.468 DEBUG:boto:Params: {} 2023-09-22T21:26:30.469 DEBUG:boto:Token: None 2023-09-22T21:26:30.469 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/ 2023-09-22T21:26:30.469 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ= 2023-09-22T21:26:30.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ=', 'Content-Length': '0'} 2023-09-22T21:26:30.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ffacc30f4fc6b34-00650e0686-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.473 DEBUG:boto:b'ffzdqu-55g1000falseg2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:30.474 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:26:30.474 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:26:30.474 DEBUG:boto:Method: HEAD 2023-09-22T21:26:30.474 DEBUG:boto:Path: /ffzdqu-55/g 2023-09-22T21:26:30.474 DEBUG:boto:Data: 2023-09-22T21:26:30.475 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.475 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.475 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.475 DEBUG:boto:Params: {} 2023-09-22T21:26:30.475 DEBUG:boto:Token: None 2023-09-22T21:26:30.475 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/g 2023-09-22T21:26:30.475 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZUyNlcIhm/r9L8c/A7g296iKQhQ= 2023-09-22T21:26:30.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZUyNlcIhm/r9L8c/A7g296iKQhQ=', 'Content-Length': '0'} 2023-09-22T21:26:30.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca1110a92478ab9b-00650e0686-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.478 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:26:30.478 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:26:30.478 DEBUG:boto:Method: HEAD 2023-09-22T21:26:30.479 DEBUG:boto:Path: /ffzdqu-55/g 2023-09-22T21:26:30.479 DEBUG:boto:Data: 2023-09-22T21:26:30.479 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.479 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.479 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.479 DEBUG:boto:Params: {} 2023-09-22T21:26:30.479 DEBUG:boto:Token: None 2023-09-22T21:26:30.479 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/g 2023-09-22T21:26:30.480 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZUyNlcIhm/r9L8c/A7g296iKQhQ= 2023-09-22T21:26:30.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZUyNlcIhm/r9L8c/A7g296iKQhQ=', 'Content-Length': '0'} 2023-09-22T21:26:30.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a962dc088d31810-00650e0686-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-22T21:26:30.483 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:26:30.483 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:26:30.483 DEBUG:boto:Method: GET 2023-09-22T21:26:30.483 DEBUG:boto:Path: /ffzdqu-55/g 2023-09-22T21:26:30.483 DEBUG:boto:Data: 2023-09-22T21:26:30.483 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.484 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.484 DEBUG:boto:Params: {} 2023-09-22T21:26:30.484 DEBUG:boto:Token: None 2023-09-22T21:26:30.484 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/g 2023-09-22T21:26:30.484 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ciy50/a2Pwqkj9I7wi/9giRGu0E= 2023-09-22T21:26:30.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ciy50/a2Pwqkj9I7wi/9giRGu0E=', 'Content-Length': '0'} 2023-09-22T21:26:30.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbceeb542c40185a-00650e0686-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.530 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:26:30.531 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:26:30.531 DEBUG:boto:Method: GET 2023-09-22T21:26:30.531 DEBUG:boto:Path: /ffzdqu-55/g 2023-09-22T21:26:30.531 DEBUG:boto:Data: 2023-09-22T21:26:30.531 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.531 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.531 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.531 DEBUG:boto:Params: {} 2023-09-22T21:26:30.531 DEBUG:boto:Token: None 2023-09-22T21:26:30.532 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/g 2023-09-22T21:26:30.532 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ciy50/a2Pwqkj9I7wi/9giRGu0E= 2023-09-22T21:26:30.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ciy50/a2Pwqkj9I7wi/9giRGu0E=', 'Content-Length': '0'} 2023-09-22T21:26:30.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051c53ca4110ccde0-00650e0686-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.535 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:30.535 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:30.535 DEBUG:boto:path=/ffzdqu-55/?prefix=g 2023-09-22T21:26:30.535 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=g 2023-09-22T21:26:30.535 DEBUG:boto:Method: GET 2023-09-22T21:26:30.535 DEBUG:boto:Path: /ffzdqu-55/?prefix=g 2023-09-22T21:26:30.536 DEBUG:boto:Data: 2023-09-22T21:26:30.536 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.536 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.536 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.536 DEBUG:boto:Params: {} 2023-09-22T21:26:30.536 DEBUG:boto:Token: None 2023-09-22T21:26:30.536 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/ 2023-09-22T21:26:30.536 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ= 2023-09-22T21:26:30.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ=', 'Content-Length': '0'} 2023-09-22T21:26:30.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d56b9185bcb7783-00650e0686-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.542 DEBUG:boto:b'ffzdqu-55g1000falseg2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:30.543 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:30.543 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:30.543 DEBUG:boto:path=/ffzdqu-55/?prefix=g 2023-09-22T21:26:30.543 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=g 2023-09-22T21:26:30.543 DEBUG:boto:Method: GET 2023-09-22T21:26:30.543 DEBUG:boto:Path: /ffzdqu-55/?prefix=g 2023-09-22T21:26:30.544 DEBUG:boto:Data: 2023-09-22T21:26:30.544 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.544 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.544 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.544 DEBUG:boto:Params: {} 2023-09-22T21:26:30.544 DEBUG:boto:Token: None 2023-09-22T21:26:30.544 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/ 2023-09-22T21:26:30.545 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ= 2023-09-22T21:26:30.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ=', 'Content-Length': '0'} 2023-09-22T21:26:30.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b80b03b0d62e8fa-00650e0686-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.548 DEBUG:boto:b'ffzdqu-55g1000falseg2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:30.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-22T21:26:30.549 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:26:30.549 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:26:30.549 DEBUG:boto:path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.549 DEBUG:boto:auth_path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.549 DEBUG:boto:Method: GET 2023-09-22T21:26:30.549 DEBUG:boto:Path: /ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.549 DEBUG:boto:Data: 2023-09-22T21:26:30.550 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.550 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.550 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.550 DEBUG:boto:Params: {} 2023-09-22T21:26:30.550 DEBUG:boto:Token: None 2023-09-22T21:26:30.550 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.550 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qzhZdTNoRvof+hmfCUJ8jbUzlpk= 2023-09-22T21:26:30.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qzhZdTNoRvof+hmfCUJ8jbUzlpk=', 'Content-Length': '0'} 2023-09-22T21:26:30.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbb70d546ebfec41-00650e0686-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.554 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:26:30.554 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:26:30.554 DEBUG:boto:path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.554 DEBUG:boto:auth_path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.554 DEBUG:boto:Method: GET 2023-09-22T21:26:30.554 DEBUG:boto:Path: /ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.555 DEBUG:boto:Data: 2023-09-22T21:26:30.555 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.555 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.555 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.555 DEBUG:boto:Params: {} 2023-09-22T21:26:30.555 DEBUG:boto:Token: None 2023-09-22T21:26:30.555 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.556 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qzhZdTNoRvof+hmfCUJ8jbUzlpk= 2023-09-22T21:26:30.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qzhZdTNoRvof+hmfCUJ8jbUzlpk=', 'Content-Length': '0'} 2023-09-22T21:26:30.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086b3ab91e0bd6244-00650e0686-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.559 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:26:30.559 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:26:30.559 DEBUG:boto:path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.559 DEBUG:boto:auth_path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.559 DEBUG:boto:Method: HEAD 2023-09-22T21:26:30.559 DEBUG:boto:Path: /ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.560 DEBUG:boto:Data: 2023-09-22T21:26:30.560 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.560 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.560 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.560 DEBUG:boto:Params: {} 2023-09-22T21:26:30.560 DEBUG:boto:Token: None 2023-09-22T21:26:30.560 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.561 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b54IIqLrDqJJxebm0+pA/N2608A= 2023-09-22T21:26:30.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b54IIqLrDqJJxebm0+pA/N2608A=', 'Content-Length': '0'} 2023-09-22T21:26:30.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a1049f5fe8ef18d-00650e0686-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.563 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:26:30.564 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:26:30.564 DEBUG:boto:path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.564 DEBUG:boto:auth_path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.564 DEBUG:boto:Method: HEAD 2023-09-22T21:26:30.564 DEBUG:boto:Path: /ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.564 DEBUG:boto:Data: 2023-09-22T21:26:30.564 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.565 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.565 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.565 DEBUG:boto:Params: {} 2023-09-22T21:26:30.565 DEBUG:boto:Token: None 2023-09-22T21:26:30.565 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.565 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b54IIqLrDqJJxebm0+pA/N2608A= 2023-09-22T21:26:30.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b54IIqLrDqJJxebm0+pA/N2608A=', 'Content-Length': '0'} 2023-09-22T21:26:30.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000110864c837ffccee-00650e0686-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-22T21:26:30.569 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:26:30.569 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:26:30.569 DEBUG:boto:path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.569 DEBUG:boto:auth_path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.569 DEBUG:boto:Method: GET 2023-09-22T21:26:30.569 DEBUG:boto:Path: /ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.569 DEBUG:boto:Data: 2023-09-22T21:26:30.569 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.570 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.570 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.570 DEBUG:boto:Params: {} 2023-09-22T21:26:30.570 DEBUG:boto:Token: None 2023-09-22T21:26:30.570 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.570 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qzhZdTNoRvof+hmfCUJ8jbUzlpk= 2023-09-22T21:26:30.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qzhZdTNoRvof+hmfCUJ8jbUzlpk=', 'Content-Length': '0'} 2023-09-22T21:26:30.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcc0f5804d6b8e0a-00650e0686-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.614 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:26:30.615 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:26:30.615 DEBUG:boto:path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.615 DEBUG:boto:auth_path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.615 DEBUG:boto:Method: GET 2023-09-22T21:26:30.615 DEBUG:boto:Path: /ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.615 DEBUG:boto:Data: 2023-09-22T21:26:30.615 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.615 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.615 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.616 DEBUG:boto:Params: {} 2023-09-22T21:26:30.616 DEBUG:boto:Token: None 2023-09-22T21:26:30.616 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/h?versionId=null 2023-09-22T21:26:30.616 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qzhZdTNoRvof+hmfCUJ8jbUzlpk= 2023-09-22T21:26:30.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qzhZdTNoRvof+hmfCUJ8jbUzlpk=', 'Content-Length': '0'} 2023-09-22T21:26:30.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d14c9a5b3324967a-00650e0686-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.620 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:30.620 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:30.620 DEBUG:boto:path=/ffzdqu-55/?prefix=h 2023-09-22T21:26:30.620 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=h 2023-09-22T21:26:30.620 DEBUG:boto:Method: GET 2023-09-22T21:26:30.620 DEBUG:boto:Path: /ffzdqu-55/?prefix=h 2023-09-22T21:26:30.620 DEBUG:boto:Data: 2023-09-22T21:26:30.620 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.621 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.621 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.621 DEBUG:boto:Params: {} 2023-09-22T21:26:30.621 DEBUG:boto:Token: None 2023-09-22T21:26:30.621 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/ 2023-09-22T21:26:30.621 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ= 2023-09-22T21:26:30.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ=', 'Content-Length': '0'} 2023-09-22T21:26:30.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e84b79eec53a0ad9-00650e0686-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.627 DEBUG:boto:b'ffzdqu-55h1000falseh2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:30.628 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:30.628 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:30.628 DEBUG:boto:path=/ffzdqu-55/?prefix=h 2023-09-22T21:26:30.628 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=h 2023-09-22T21:26:30.628 DEBUG:boto:Method: GET 2023-09-22T21:26:30.628 DEBUG:boto:Path: /ffzdqu-55/?prefix=h 2023-09-22T21:26:30.628 DEBUG:boto:Data: 2023-09-22T21:26:30.628 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.629 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.629 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.629 DEBUG:boto:Params: {} 2023-09-22T21:26:30.629 DEBUG:boto:Token: None 2023-09-22T21:26:30.629 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/ 2023-09-22T21:26:30.629 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ= 2023-09-22T21:26:30.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ=', 'Content-Length': '0'} 2023-09-22T21:26:30.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb3154923a8ceb95-00650e0686-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.633 DEBUG:boto:b'ffzdqu-55h1000falseh2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:30.633 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:26:30.633 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:26:30.633 DEBUG:boto:Method: HEAD 2023-09-22T21:26:30.634 DEBUG:boto:Path: /ffzdqu-55/h 2023-09-22T21:26:30.634 DEBUG:boto:Data: 2023-09-22T21:26:30.634 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.634 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.634 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.634 DEBUG:boto:Params: {} 2023-09-22T21:26:30.634 DEBUG:boto:Token: None 2023-09-22T21:26:30.635 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/h 2023-09-22T21:26:30.635 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3lIOeUE6NUvYKmDeMNBDdQFnj9c= 2023-09-22T21:26:30.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3lIOeUE6NUvYKmDeMNBDdQFnj9c=', 'Content-Length': '0'} 2023-09-22T21:26:30.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1cadd7113f88f67-00650e0686-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.638 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:26:30.638 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:26:30.638 DEBUG:boto:Method: HEAD 2023-09-22T21:26:30.638 DEBUG:boto:Path: /ffzdqu-55/h 2023-09-22T21:26:30.638 DEBUG:boto:Data: 2023-09-22T21:26:30.639 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.639 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.639 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.639 DEBUG:boto:Params: {} 2023-09-22T21:26:30.639 DEBUG:boto:Token: None 2023-09-22T21:26:30.639 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/h 2023-09-22T21:26:30.639 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3lIOeUE6NUvYKmDeMNBDdQFnj9c= 2023-09-22T21:26:30.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3lIOeUE6NUvYKmDeMNBDdQFnj9c=', 'Content-Length': '0'} 2023-09-22T21:26:30.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8428226ebd6f22d-00650e0686-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-22T21:26:30.642 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:26:30.643 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:26:30.643 DEBUG:boto:Method: GET 2023-09-22T21:26:30.643 DEBUG:boto:Path: /ffzdqu-55/h 2023-09-22T21:26:30.643 DEBUG:boto:Data: 2023-09-22T21:26:30.643 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.643 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.643 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.643 DEBUG:boto:Params: {} 2023-09-22T21:26:30.643 DEBUG:boto:Token: None 2023-09-22T21:26:30.644 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/h 2023-09-22T21:26:30.644 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ml0RdlQsyocxIW+pOXSlLOCmhIs= 2023-09-22T21:26:30.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ml0RdlQsyocxIW+pOXSlLOCmhIs=', 'Content-Length': '0'} 2023-09-22T21:26:30.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000441b0af986702513-00650e0686-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.690 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:26:30.690 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:26:30.691 DEBUG:boto:Method: GET 2023-09-22T21:26:30.691 DEBUG:boto:Path: /ffzdqu-55/h 2023-09-22T21:26:30.691 DEBUG:boto:Data: 2023-09-22T21:26:30.691 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.691 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.691 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.691 DEBUG:boto:Params: {} 2023-09-22T21:26:30.691 DEBUG:boto:Token: None 2023-09-22T21:26:30.692 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/h 2023-09-22T21:26:30.692 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ml0RdlQsyocxIW+pOXSlLOCmhIs= 2023-09-22T21:26:30.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ml0RdlQsyocxIW+pOXSlLOCmhIs=', 'Content-Length': '0'} 2023-09-22T21:26:30.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013f57108ddca9db9-00650e0686-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.695 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:30.695 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:30.695 DEBUG:boto:path=/ffzdqu-55/?prefix=h 2023-09-22T21:26:30.696 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=h 2023-09-22T21:26:30.696 DEBUG:boto:Method: GET 2023-09-22T21:26:30.696 DEBUG:boto:Path: /ffzdqu-55/?prefix=h 2023-09-22T21:26:30.696 DEBUG:boto:Data: 2023-09-22T21:26:30.696 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.696 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:30.696 DEBUG:boto:Port: 8001 2023-09-22T21:26:30.696 DEBUG:boto:Params: {} 2023-09-22T21:26:30.696 DEBUG:boto:Token: None 2023-09-22T21:26:30.697 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/ 2023-09-22T21:26:30.697 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ= 2023-09-22T21:26:30.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ=', 'Content-Length': '0'} 2023-09-22T21:26:30.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a38f73efafcdc6b0-00650e0686-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.746 DEBUG:boto:b'ffzdqu-55h1000falseh2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:30.747 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:30.747 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:30.747 DEBUG:boto:path=/ffzdqu-55/?prefix=h 2023-09-22T21:26:30.747 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=h 2023-09-22T21:26:30.747 DEBUG:boto:Method: GET 2023-09-22T21:26:30.747 DEBUG:boto:Path: /ffzdqu-55/?prefix=h 2023-09-22T21:26:30.748 DEBUG:boto:Data: 2023-09-22T21:26:30.748 DEBUG:boto:Headers: {} 2023-09-22T21:26:30.748 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:30.748 DEBUG:boto:Port: 8000 2023-09-22T21:26:30.748 DEBUG:boto:Params: {} 2023-09-22T21:26:30.748 DEBUG:boto:Token: None 2023-09-22T21:26:30.748 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:30 GMT /ffzdqu-55/ 2023-09-22T21:26:30.748 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ= 2023-09-22T21:26:30.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bF9db+tk2qYsA3ucKle18lKcbvQ=', 'Content-Length': '0'} 2023-09-22T21:26:30.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000975a1f4bde1356c2-00650e0686-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:30.752 DEBUG:boto:b'ffzdqu-55h1000falseh2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:30.752 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-55 zones={a2, a1} 2023-09-22T21:26:30.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'ffzdqu-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:26:30.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'ffzdqu-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:26:30.753 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket ffzdqu-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:26:30.797 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:26:30.797 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:26:30.855 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:26:30.855 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-55 2023-09-22T21:26:30.857 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 8 2023-09-22T21:26:30.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'ffzdqu-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:26:30.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'ffzdqu-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:26:30.892 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket ffzdqu-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:26:30.968 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:26:30.968 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:26:31.024 INFO:teuthology.orchestra.run.smithi160.stdout:{"layout":{"resharding":"None","current_index":{"gen":2,"layout":{"type":"Normal","normal":{"num_shards":3,"hash_type":"Mod"}}},"logs":[{"gen":0,"layout":{"type":"InIndex","in_index":{"gen":0,"layout":{"num_shards":11,"hash_type":"Mod"}}}},{"gen":1,"layout":{"type":"InIndex","in_index":{"gen":1,"layout":{"num_shards":3,"hash_type":"Mod"}}}},{"gen":2,"layout":{"type":"InIndex","in_index":{"gen":2,"layout":{"num_shards":3,"hash_type":"Mod"}}}}]}} 2023-09-22T21:26:31.024 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-55 2023-09-22T21:26:31.025 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:26:31.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:26:31.025 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:26:31.113 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:26:31.113 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:26:31.155 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:26:31.157+0000 7f19f3842a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:26:36.156 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:26:36.157+0000 7f19f3842a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:26:41.157 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:26:41.157+0000 7f19f3842a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:26:46.158 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:26:46.157+0000 7f19f3842a40 1 bucket sync caught up with empty source 2023-09-22T21:26:46.158 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:26:46.157+0000 7f19f3842a40 0 bucket checkpoint complete 2023-09-22T21:26:46.193 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-55 2023-09-22T21:26:46.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:26:46.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:26:46.194 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:26:46.284 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:26:46.284 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:26:46.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:26:46.329+0000 7ff9014d9a40 1 bucket sync caught up with source: 2023-09-22T21:26:46.328 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.278.6, , , 00000000001.231.6, 00000000001.197.6, , , 00000000002.167.6, , 00000000002.165.6] 2023-09-22T21:26:46.328 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.278.6, , , 00000000001.231.6, 00000000001.197.6, , , 00000000002.167.6, , 00000000002.165.6] 2023-09-22T21:26:46.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:26:46.329+0000 7ff9014d9a40 0 bucket checkpoint complete 2023-09-22T21:26:46.345 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-55 zones={a2, a1} 2023-09-22T21:26:46.345 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:46.346 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:46.346 DEBUG:boto:Method: HEAD 2023-09-22T21:26:46.346 DEBUG:boto:Path: /ffzdqu-55/ 2023-09-22T21:26:46.346 DEBUG:boto:Data: 2023-09-22T21:26:46.346 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.346 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.346 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.346 DEBUG:boto:Params: {} 2023-09-22T21:26:46.347 DEBUG:boto:Token: None 2023-09-22T21:26:46.347 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/ 2023-09-22T21:26:46.347 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JwizBIxKTg+V03ip8wkg3K38maY= 2023-09-22T21:26:46.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JwizBIxKTg+V03ip8wkg3K38maY=', 'Content-Length': '0'} 2023-09-22T21:26:46.349 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd2efa40d9a6fcbe-00650e0696-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.350 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:46.350 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:46.350 DEBUG:boto:Method: HEAD 2023-09-22T21:26:46.350 DEBUG:boto:Path: /ffzdqu-55/ 2023-09-22T21:26:46.350 DEBUG:boto:Data: 2023-09-22T21:26:46.350 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.350 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.350 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.350 DEBUG:boto:Params: {} 2023-09-22T21:26:46.351 DEBUG:boto:Token: None 2023-09-22T21:26:46.351 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/ 2023-09-22T21:26:46.351 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JwizBIxKTg+V03ip8wkg3K38maY= 2023-09-22T21:26:46.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JwizBIxKTg+V03ip8wkg3K38maY=', 'Content-Length': '0'} 2023-09-22T21:26:46.353 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c55c318d3a0f169-00650e0696-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.353 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:26:46.353 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:46.354 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:46.354 DEBUG:boto:path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:46.354 DEBUG:boto:auth_path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:46.354 DEBUG:boto:Method: GET 2023-09-22T21:26:46.354 DEBUG:boto:Path: /ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:46.354 DEBUG:boto:Data: 2023-09-22T21:26:46.354 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.354 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.354 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.354 DEBUG:boto:Params: {} 2023-09-22T21:26:46.355 DEBUG:boto:Token: None 2023-09-22T21:26:46.355 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/?versions 2023-09-22T21:26:46.355 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8P0PRvTJ9jbFRPVD4Aam1vlCbg4= 2023-09-22T21:26:46.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8P0PRvTJ9jbFRPVD4Aam1vlCbg4=', 'Content-Length': '0'} 2023-09-22T21:26:46.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffc0dc4d1e7f2f60-00650e0696-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.358 DEBUG:boto:b'ffzdqu-55999falseanulltrue2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:46.359 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-22T21:26:46.359 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-22T21:26:46.359 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-22T21:26:46.359 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-22T21:26:46.359 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-09-22T21:26:46.359 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-09-22T21:26:46.359 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-09-22T21:26:46.359 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-09-22T21:26:46.359 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:26:46.359 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:46.360 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:46.360 DEBUG:boto:path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:46.360 DEBUG:boto:auth_path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:46.360 DEBUG:boto:Method: GET 2023-09-22T21:26:46.360 DEBUG:boto:Path: /ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:46.360 DEBUG:boto:Data: 2023-09-22T21:26:46.360 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.360 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.360 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.360 DEBUG:boto:Params: {} 2023-09-22T21:26:46.361 DEBUG:boto:Token: None 2023-09-22T21:26:46.361 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/?versions 2023-09-22T21:26:46.361 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8P0PRvTJ9jbFRPVD4Aam1vlCbg4= 2023-09-22T21:26:46.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8P0PRvTJ9jbFRPVD4Aam1vlCbg4=', 'Content-Length': '0'} 2023-09-22T21:26:46.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000270c05cdf072c2e5-00650e0696-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.363 DEBUG:boto:b'ffzdqu-55999falseanulltrue2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:46.364 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-22T21:26:46.364 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-22T21:26:46.364 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-22T21:26:46.364 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-22T21:26:46.364 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-09-22T21:26:46.364 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-09-22T21:26:46.364 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-09-22T21:26:46.365 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-09-22T21:26:46.365 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:46.365 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:46.365 DEBUG:boto:path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:46.365 DEBUG:boto:auth_path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:46.365 DEBUG:boto:Method: GET 2023-09-22T21:26:46.365 DEBUG:boto:Path: /ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:46.365 DEBUG:boto:Data: 2023-09-22T21:26:46.365 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.365 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.366 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.366 DEBUG:boto:Params: {} 2023-09-22T21:26:46.366 DEBUG:boto:Token: None 2023-09-22T21:26:46.366 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/?versions 2023-09-22T21:26:46.366 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8P0PRvTJ9jbFRPVD4Aam1vlCbg4= 2023-09-22T21:26:46.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8P0PRvTJ9jbFRPVD4Aam1vlCbg4=', 'Content-Length': '0'} 2023-09-22T21:26:46.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d257b133ad0fe18-00650e0696-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.369 DEBUG:boto:b'ffzdqu-55999falseanulltrue2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:46.370 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:46.370 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:46.370 DEBUG:boto:path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:46.370 DEBUG:boto:auth_path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:46.370 DEBUG:boto:Method: GET 2023-09-22T21:26:46.370 DEBUG:boto:Path: /ffzdqu-55/?versions&max-keys=999 2023-09-22T21:26:46.370 DEBUG:boto:Data: 2023-09-22T21:26:46.370 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.370 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.370 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.371 DEBUG:boto:Params: {} 2023-09-22T21:26:46.371 DEBUG:boto:Token: None 2023-09-22T21:26:46.371 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/?versions 2023-09-22T21:26:46.371 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8P0PRvTJ9jbFRPVD4Aam1vlCbg4= 2023-09-22T21:26:46.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8P0PRvTJ9jbFRPVD4Aam1vlCbg4=', 'Content-Length': '0'} 2023-09-22T21:26:46.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dd566bed345dda6-00650e0696-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.373 DEBUG:boto:b'ffzdqu-55999falseanulltrue2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:46.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:26:46.374 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:26:46.374 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:26:46.374 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.374 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.374 DEBUG:boto:Method: GET 2023-09-22T21:26:46.375 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.375 DEBUG:boto:Data: 2023-09-22T21:26:46.375 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.375 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.375 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.375 DEBUG:boto:Params: {} 2023-09-22T21:26:46.375 DEBUG:boto:Token: None 2023-09-22T21:26:46.375 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.375 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UhvlOprQLxXAeATqw7aemVnGFWg= 2023-09-22T21:26:46.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UhvlOprQLxXAeATqw7aemVnGFWg=', 'Content-Length': '0'} 2023-09-22T21:26:46.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077a3a55552ea4e2d-00650e0696-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.378 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:26:46.378 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:26:46.378 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.378 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.378 DEBUG:boto:Method: GET 2023-09-22T21:26:46.378 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.378 DEBUG:boto:Data: 2023-09-22T21:26:46.379 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.379 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.379 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.379 DEBUG:boto:Params: {} 2023-09-22T21:26:46.379 DEBUG:boto:Token: None 2023-09-22T21:26:46.379 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.379 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UhvlOprQLxXAeATqw7aemVnGFWg= 2023-09-22T21:26:46.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UhvlOprQLxXAeATqw7aemVnGFWg=', 'Content-Length': '0'} 2023-09-22T21:26:46.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd7aceda5d5d0e54-00650e0696-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.422 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:26:46.422 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:26:46.423 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.423 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.423 DEBUG:boto:Method: HEAD 2023-09-22T21:26:46.423 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.423 DEBUG:boto:Data: 2023-09-22T21:26:46.423 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.423 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.423 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.423 DEBUG:boto:Params: {} 2023-09-22T21:26:46.424 DEBUG:boto:Token: None 2023-09-22T21:26:46.424 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.424 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M7dhSEnAkUSvkDwsvuQb03L0ffQ= 2023-09-22T21:26:46.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M7dhSEnAkUSvkDwsvuQb03L0ffQ=', 'Content-Length': '0'} 2023-09-22T21:26:46.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054079bb3aee1392c-00650e0696-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.426 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:26:46.427 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:26:46.427 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.427 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.427 DEBUG:boto:Method: HEAD 2023-09-22T21:26:46.427 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.427 DEBUG:boto:Data: 2023-09-22T21:26:46.427 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.427 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.427 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.428 DEBUG:boto:Params: {} 2023-09-22T21:26:46.428 DEBUG:boto:Token: None 2023-09-22T21:26:46.428 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.428 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M7dhSEnAkUSvkDwsvuQb03L0ffQ= 2023-09-22T21:26:46.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M7dhSEnAkUSvkDwsvuQb03L0ffQ=', 'Content-Length': '0'} 2023-09-22T21:26:46.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fce9cd4e3fa859c0-00650e0696-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:26:46.431 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:26:46.431 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:26:46.431 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.431 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.431 DEBUG:boto:Method: GET 2023-09-22T21:26:46.431 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.431 DEBUG:boto:Data: 2023-09-22T21:26:46.431 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.432 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.432 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.432 DEBUG:boto:Params: {} 2023-09-22T21:26:46.432 DEBUG:boto:Token: None 2023-09-22T21:26:46.432 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.432 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UhvlOprQLxXAeATqw7aemVnGFWg= 2023-09-22T21:26:46.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UhvlOprQLxXAeATqw7aemVnGFWg=', 'Content-Length': '0'} 2023-09-22T21:26:46.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072cc35a92e7ebbd8-00650e0696-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.435 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:26:46.435 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:26:46.435 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.435 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.435 DEBUG:boto:Method: GET 2023-09-22T21:26:46.436 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.436 DEBUG:boto:Data: 2023-09-22T21:26:46.436 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.436 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.436 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.436 DEBUG:boto:Params: {} 2023-09-22T21:26:46.436 DEBUG:boto:Token: None 2023-09-22T21:26:46.436 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:26:46.437 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UhvlOprQLxXAeATqw7aemVnGFWg= 2023-09-22T21:26:46.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UhvlOprQLxXAeATqw7aemVnGFWg=', 'Content-Length': '0'} 2023-09-22T21:26:46.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a495d2babe270f6b-00650e0696-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.439 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:46.440 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:46.440 DEBUG:boto:path=/ffzdqu-55/?prefix=a 2023-09-22T21:26:46.440 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=a 2023-09-22T21:26:46.440 DEBUG:boto:Method: GET 2023-09-22T21:26:46.440 DEBUG:boto:Path: /ffzdqu-55/?prefix=a 2023-09-22T21:26:46.440 DEBUG:boto:Data: 2023-09-22T21:26:46.440 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.440 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.440 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.441 DEBUG:boto:Params: {} 2023-09-22T21:26:46.441 DEBUG:boto:Token: None 2023-09-22T21:26:46.441 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/ 2023-09-22T21:26:46.441 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw= 2023-09-22T21:26:46.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw=', 'Content-Length': '0'} 2023-09-22T21:26:46.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000050adbe103a7f8dc-00650e0696-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.445 DEBUG:boto:b'ffzdqu-55a1000falsea2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:46.445 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:46.446 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:46.446 DEBUG:boto:path=/ffzdqu-55/?prefix=a 2023-09-22T21:26:46.446 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=a 2023-09-22T21:26:46.446 DEBUG:boto:Method: GET 2023-09-22T21:26:46.446 DEBUG:boto:Path: /ffzdqu-55/?prefix=a 2023-09-22T21:26:46.446 DEBUG:boto:Data: 2023-09-22T21:26:46.446 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.446 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.447 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.447 DEBUG:boto:Params: {} 2023-09-22T21:26:46.447 DEBUG:boto:Token: None 2023-09-22T21:26:46.447 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/ 2023-09-22T21:26:46.447 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw= 2023-09-22T21:26:46.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw=', 'Content-Length': '0'} 2023-09-22T21:26:46.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fddd29bd0e4ddfd0-00650e0696-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.494 DEBUG:boto:b'ffzdqu-55a1000falsea2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:46.495 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:26:46.495 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:26:46.495 DEBUG:boto:Method: HEAD 2023-09-22T21:26:46.495 DEBUG:boto:Path: /ffzdqu-55/a 2023-09-22T21:26:46.495 DEBUG:boto:Data: 2023-09-22T21:26:46.495 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.495 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.496 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.496 DEBUG:boto:Params: {} 2023-09-22T21:26:46.496 DEBUG:boto:Token: None 2023-09-22T21:26:46.496 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/a 2023-09-22T21:26:46.496 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yl185vsoD8byzP+u6QSatyYze6Q= 2023-09-22T21:26:46.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yl185vsoD8byzP+u6QSatyYze6Q=', 'Content-Length': '0'} 2023-09-22T21:26:46.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebc191fa33aae439-00650e0696-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.498 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:26:46.499 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:26:46.499 DEBUG:boto:Method: HEAD 2023-09-22T21:26:46.499 DEBUG:boto:Path: /ffzdqu-55/a 2023-09-22T21:26:46.499 DEBUG:boto:Data: 2023-09-22T21:26:46.499 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.499 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.499 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.499 DEBUG:boto:Params: {} 2023-09-22T21:26:46.499 DEBUG:boto:Token: None 2023-09-22T21:26:46.500 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/a 2023-09-22T21:26:46.500 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yl185vsoD8byzP+u6QSatyYze6Q= 2023-09-22T21:26:46.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yl185vsoD8byzP+u6QSatyYze6Q=', 'Content-Length': '0'} 2023-09-22T21:26:46.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f72d6217ac5be26-00650e0696-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:26:46.502 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:26:46.503 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:26:46.503 DEBUG:boto:Method: GET 2023-09-22T21:26:46.503 DEBUG:boto:Path: /ffzdqu-55/a 2023-09-22T21:26:46.503 DEBUG:boto:Data: 2023-09-22T21:26:46.503 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.503 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.503 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.503 DEBUG:boto:Params: {} 2023-09-22T21:26:46.503 DEBUG:boto:Token: None 2023-09-22T21:26:46.504 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/a 2023-09-22T21:26:46.504 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4WytLZvB/I9I0ag9TsymexFBCt0= 2023-09-22T21:26:46.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4WytLZvB/I9I0ag9TsymexFBCt0=', 'Content-Length': '0'} 2023-09-22T21:26:46.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a5325ef2874ef92-00650e0696-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.546 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:26:46.546 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:26:46.547 DEBUG:boto:Method: GET 2023-09-22T21:26:46.547 DEBUG:boto:Path: /ffzdqu-55/a 2023-09-22T21:26:46.547 DEBUG:boto:Data: 2023-09-22T21:26:46.547 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.547 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.547 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.547 DEBUG:boto:Params: {} 2023-09-22T21:26:46.547 DEBUG:boto:Token: None 2023-09-22T21:26:46.548 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/a 2023-09-22T21:26:46.548 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4WytLZvB/I9I0ag9TsymexFBCt0= 2023-09-22T21:26:46.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4WytLZvB/I9I0ag9TsymexFBCt0=', 'Content-Length': '0'} 2023-09-22T21:26:46.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2e24aec81c3728a-00650e0696-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.551 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:46.551 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:46.551 DEBUG:boto:path=/ffzdqu-55/?prefix=a 2023-09-22T21:26:46.551 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=a 2023-09-22T21:26:46.551 DEBUG:boto:Method: GET 2023-09-22T21:26:46.551 DEBUG:boto:Path: /ffzdqu-55/?prefix=a 2023-09-22T21:26:46.552 DEBUG:boto:Data: 2023-09-22T21:26:46.552 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.552 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.552 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.552 DEBUG:boto:Params: {} 2023-09-22T21:26:46.552 DEBUG:boto:Token: None 2023-09-22T21:26:46.552 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/ 2023-09-22T21:26:46.552 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw= 2023-09-22T21:26:46.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw=', 'Content-Length': '0'} 2023-09-22T21:26:46.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcbf39c4e9da5755-00650e0696-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.558 DEBUG:boto:b'ffzdqu-55a1000falsea2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:46.559 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:46.559 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:46.559 DEBUG:boto:path=/ffzdqu-55/?prefix=a 2023-09-22T21:26:46.559 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=a 2023-09-22T21:26:46.559 DEBUG:boto:Method: GET 2023-09-22T21:26:46.559 DEBUG:boto:Path: /ffzdqu-55/?prefix=a 2023-09-22T21:26:46.559 DEBUG:boto:Data: 2023-09-22T21:26:46.559 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.560 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.560 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.560 DEBUG:boto:Params: {} 2023-09-22T21:26:46.560 DEBUG:boto:Token: None 2023-09-22T21:26:46.560 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/ 2023-09-22T21:26:46.560 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw= 2023-09-22T21:26:46.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw=', 'Content-Length': '0'} 2023-09-22T21:26:46.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000252871ae6fddd15a-00650e0696-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.564 DEBUG:boto:b'ffzdqu-55a1000falsea2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:46.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:26:46.564 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:26:46.565 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:26:46.565 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.565 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.565 DEBUG:boto:Method: GET 2023-09-22T21:26:46.565 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.565 DEBUG:boto:Data: 2023-09-22T21:26:46.565 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.565 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.565 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.565 DEBUG:boto:Params: {} 2023-09-22T21:26:46.566 DEBUG:boto:Token: None 2023-09-22T21:26:46.566 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.566 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cxGUAAjBx7Ixas/YYLJiyzzWgUg= 2023-09-22T21:26:46.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cxGUAAjBx7Ixas/YYLJiyzzWgUg=', 'Content-Length': '0'} 2023-09-22T21:26:46.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006af1eb70b56a55f5-00650e0696-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.569 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:26:46.569 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:26:46.569 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.569 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.569 DEBUG:boto:Method: GET 2023-09-22T21:26:46.569 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.569 DEBUG:boto:Data: 2023-09-22T21:26:46.570 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.570 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.570 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.570 DEBUG:boto:Params: {} 2023-09-22T21:26:46.570 DEBUG:boto:Token: None 2023-09-22T21:26:46.570 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.570 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cxGUAAjBx7Ixas/YYLJiyzzWgUg= 2023-09-22T21:26:46.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cxGUAAjBx7Ixas/YYLJiyzzWgUg=', 'Content-Length': '0'} 2023-09-22T21:26:46.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d09755a2bc944eeb-00650e0696-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.618 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:26:46.618 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:26:46.619 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.619 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.619 DEBUG:boto:Method: HEAD 2023-09-22T21:26:46.619 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.619 DEBUG:boto:Data: 2023-09-22T21:26:46.619 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.619 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.619 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.619 DEBUG:boto:Params: {} 2023-09-22T21:26:46.620 DEBUG:boto:Token: None 2023-09-22T21:26:46.620 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.620 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q2fhePcfVA1t6KgMDaTOdn7i2+A= 2023-09-22T21:26:46.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q2fhePcfVA1t6KgMDaTOdn7i2+A=', 'Content-Length': '0'} 2023-09-22T21:26:46.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e9f442699eb7266-00650e0696-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.623 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:26:46.623 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:26:46.623 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.623 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.623 DEBUG:boto:Method: HEAD 2023-09-22T21:26:46.623 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.623 DEBUG:boto:Data: 2023-09-22T21:26:46.624 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.624 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.624 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.624 DEBUG:boto:Params: {} 2023-09-22T21:26:46.624 DEBUG:boto:Token: None 2023-09-22T21:26:46.624 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.624 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q2fhePcfVA1t6KgMDaTOdn7i2+A= 2023-09-22T21:26:46.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q2fhePcfVA1t6KgMDaTOdn7i2+A=', 'Content-Length': '0'} 2023-09-22T21:26:46.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004ac499b83781fe4-00650e0696-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:26:46.627 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:26:46.627 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:26:46.627 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.628 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.628 DEBUG:boto:Method: GET 2023-09-22T21:26:46.628 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.628 DEBUG:boto:Data: 2023-09-22T21:26:46.628 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.628 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.628 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.628 DEBUG:boto:Params: {} 2023-09-22T21:26:46.628 DEBUG:boto:Token: None 2023-09-22T21:26:46.629 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.629 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cxGUAAjBx7Ixas/YYLJiyzzWgUg= 2023-09-22T21:26:46.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cxGUAAjBx7Ixas/YYLJiyzzWgUg=', 'Content-Length': '0'} 2023-09-22T21:26:46.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003510b990e5cb90a4-00650e0696-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.674 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:26:46.675 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:26:46.675 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.675 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.675 DEBUG:boto:Method: GET 2023-09-22T21:26:46.675 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.675 DEBUG:boto:Data: 2023-09-22T21:26:46.675 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.675 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.676 DEBUG:boto:Params: {} 2023-09-22T21:26:46.676 DEBUG:boto:Token: None 2023-09-22T21:26:46.676 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:26:46.676 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cxGUAAjBx7Ixas/YYLJiyzzWgUg= 2023-09-22T21:26:46.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cxGUAAjBx7Ixas/YYLJiyzzWgUg=', 'Content-Length': '0'} 2023-09-22T21:26:46.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000150add9aa48a7564-00650e0696-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.679 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:46.680 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:46.680 DEBUG:boto:path=/ffzdqu-55/?prefix=b 2023-09-22T21:26:46.680 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=b 2023-09-22T21:26:46.680 DEBUG:boto:Method: GET 2023-09-22T21:26:46.680 DEBUG:boto:Path: /ffzdqu-55/?prefix=b 2023-09-22T21:26:46.680 DEBUG:boto:Data: 2023-09-22T21:26:46.680 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.680 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.680 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.681 DEBUG:boto:Params: {} 2023-09-22T21:26:46.681 DEBUG:boto:Token: None 2023-09-22T21:26:46.681 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/ 2023-09-22T21:26:46.681 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw= 2023-09-22T21:26:46.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw=', 'Content-Length': '0'} 2023-09-22T21:26:46.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bcd0a8f35ae4336-00650e0696-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.687 DEBUG:boto:b'ffzdqu-55b1000falseb2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:46.687 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:46.687 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:46.687 DEBUG:boto:path=/ffzdqu-55/?prefix=b 2023-09-22T21:26:46.687 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=b 2023-09-22T21:26:46.688 DEBUG:boto:Method: GET 2023-09-22T21:26:46.688 DEBUG:boto:Path: /ffzdqu-55/?prefix=b 2023-09-22T21:26:46.688 DEBUG:boto:Data: 2023-09-22T21:26:46.688 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.688 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.688 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.688 DEBUG:boto:Params: {} 2023-09-22T21:26:46.688 DEBUG:boto:Token: None 2023-09-22T21:26:46.688 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/ 2023-09-22T21:26:46.689 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw= 2023-09-22T21:26:46.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw=', 'Content-Length': '0'} 2023-09-22T21:26:46.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028fa1aabd3f22566-00650e0696-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.693 DEBUG:boto:b'ffzdqu-55b1000falseb2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:46.693 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:26:46.693 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:26:46.693 DEBUG:boto:Method: HEAD 2023-09-22T21:26:46.694 DEBUG:boto:Path: /ffzdqu-55/b 2023-09-22T21:26:46.694 DEBUG:boto:Data: 2023-09-22T21:26:46.694 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.694 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.694 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.694 DEBUG:boto:Params: {} 2023-09-22T21:26:46.694 DEBUG:boto:Token: None 2023-09-22T21:26:46.694 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/b 2023-09-22T21:26:46.695 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:genLVOgb1qJPbuC87CoONM1NCXY= 2023-09-22T21:26:46.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:genLVOgb1qJPbuC87CoONM1NCXY=', 'Content-Length': '0'} 2023-09-22T21:26:46.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009581e263ae6701ce-00650e0696-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.697 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:26:46.697 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:26:46.698 DEBUG:boto:Method: HEAD 2023-09-22T21:26:46.698 DEBUG:boto:Path: /ffzdqu-55/b 2023-09-22T21:26:46.698 DEBUG:boto:Data: 2023-09-22T21:26:46.698 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.698 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.698 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.698 DEBUG:boto:Params: {} 2023-09-22T21:26:46.698 DEBUG:boto:Token: None 2023-09-22T21:26:46.698 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/b 2023-09-22T21:26:46.699 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:genLVOgb1qJPbuC87CoONM1NCXY= 2023-09-22T21:26:46.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:genLVOgb1qJPbuC87CoONM1NCXY=', 'Content-Length': '0'} 2023-09-22T21:26:46.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efb4436f08a11318-00650e0696-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:26:46.702 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:26:46.702 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:26:46.702 DEBUG:boto:Method: GET 2023-09-22T21:26:46.702 DEBUG:boto:Path: /ffzdqu-55/b 2023-09-22T21:26:46.702 DEBUG:boto:Data: 2023-09-22T21:26:46.703 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.703 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.703 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.703 DEBUG:boto:Params: {} 2023-09-22T21:26:46.703 DEBUG:boto:Token: None 2023-09-22T21:26:46.703 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/b 2023-09-22T21:26:46.703 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Z2qzzd3uhVyIG1SXLaaqmk94LIE= 2023-09-22T21:26:46.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Z2qzzd3uhVyIG1SXLaaqmk94LIE=', 'Content-Length': '0'} 2023-09-22T21:26:46.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d2dbefc37f68532-00650e0696-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.746 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:26:46.746 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:26:46.747 DEBUG:boto:Method: GET 2023-09-22T21:26:46.747 DEBUG:boto:Path: /ffzdqu-55/b 2023-09-22T21:26:46.747 DEBUG:boto:Data: 2023-09-22T21:26:46.747 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.747 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.747 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.747 DEBUG:boto:Params: {} 2023-09-22T21:26:46.747 DEBUG:boto:Token: None 2023-09-22T21:26:46.747 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/b 2023-09-22T21:26:46.748 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Z2qzzd3uhVyIG1SXLaaqmk94LIE= 2023-09-22T21:26:46.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Z2qzzd3uhVyIG1SXLaaqmk94LIE=', 'Content-Length': '0'} 2023-09-22T21:26:46.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e6caeae08cf41ce-00650e0696-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.751 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:46.751 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:46.751 DEBUG:boto:path=/ffzdqu-55/?prefix=b 2023-09-22T21:26:46.751 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=b 2023-09-22T21:26:46.751 DEBUG:boto:Method: GET 2023-09-22T21:26:46.751 DEBUG:boto:Path: /ffzdqu-55/?prefix=b 2023-09-22T21:26:46.751 DEBUG:boto:Data: 2023-09-22T21:26:46.751 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.752 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.752 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.752 DEBUG:boto:Params: {} 2023-09-22T21:26:46.752 DEBUG:boto:Token: None 2023-09-22T21:26:46.752 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/ 2023-09-22T21:26:46.752 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw= 2023-09-22T21:26:46.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw=', 'Content-Length': '0'} 2023-09-22T21:26:46.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f57de87e6463cb3-00650e0696-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.798 DEBUG:boto:b'ffzdqu-55b1000falseb2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:46.799 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:46.799 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:46.799 DEBUG:boto:path=/ffzdqu-55/?prefix=b 2023-09-22T21:26:46.799 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=b 2023-09-22T21:26:46.799 DEBUG:boto:Method: GET 2023-09-22T21:26:46.799 DEBUG:boto:Path: /ffzdqu-55/?prefix=b 2023-09-22T21:26:46.799 DEBUG:boto:Data: 2023-09-22T21:26:46.799 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.799 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.800 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.800 DEBUG:boto:Params: {} 2023-09-22T21:26:46.800 DEBUG:boto:Token: None 2023-09-22T21:26:46.800 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/ 2023-09-22T21:26:46.800 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw= 2023-09-22T21:26:46.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw=', 'Content-Length': '0'} 2023-09-22T21:26:46.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c89bafcda7fa5829-00650e0696-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.804 DEBUG:boto:b'ffzdqu-55b1000falseb2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:46.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:26:46.805 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:26:46.805 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:26:46.805 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.805 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.805 DEBUG:boto:Method: GET 2023-09-22T21:26:46.805 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.805 DEBUG:boto:Data: 2023-09-22T21:26:46.805 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.805 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.806 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.806 DEBUG:boto:Params: {} 2023-09-22T21:26:46.806 DEBUG:boto:Token: None 2023-09-22T21:26:46.806 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.806 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NkkDYrgJOh+MUgsxukurwm1JE2U= 2023-09-22T21:26:46.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NkkDYrgJOh+MUgsxukurwm1JE2U=', 'Content-Length': '0'} 2023-09-22T21:26:46.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000565acf376e4aabb7-00650e0696-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.810 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:26:46.810 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:26:46.810 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.810 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.810 DEBUG:boto:Method: GET 2023-09-22T21:26:46.810 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.810 DEBUG:boto:Data: 2023-09-22T21:26:46.810 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.810 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.810 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.811 DEBUG:boto:Params: {} 2023-09-22T21:26:46.811 DEBUG:boto:Token: None 2023-09-22T21:26:46.811 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.811 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NkkDYrgJOh+MUgsxukurwm1JE2U= 2023-09-22T21:26:46.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NkkDYrgJOh+MUgsxukurwm1JE2U=', 'Content-Length': '0'} 2023-09-22T21:26:46.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000285fab93a074bfd8-00650e0696-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.814 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:26:46.814 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:26:46.814 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.814 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.815 DEBUG:boto:Method: HEAD 2023-09-22T21:26:46.815 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.815 DEBUG:boto:Data: 2023-09-22T21:26:46.815 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.815 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.815 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.815 DEBUG:boto:Params: {} 2023-09-22T21:26:46.815 DEBUG:boto:Token: None 2023-09-22T21:26:46.815 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.816 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TOTbLr6X1Y9Pvjk05cmPi1Co/vA= 2023-09-22T21:26:46.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TOTbLr6X1Y9Pvjk05cmPi1Co/vA=', 'Content-Length': '0'} 2023-09-22T21:26:46.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009451a2b8a5e1e3b-00650e0696-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.818 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:26:46.819 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:26:46.819 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.819 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.819 DEBUG:boto:Method: HEAD 2023-09-22T21:26:46.819 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.819 DEBUG:boto:Data: 2023-09-22T21:26:46.819 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.819 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.819 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.819 DEBUG:boto:Params: {} 2023-09-22T21:26:46.820 DEBUG:boto:Token: None 2023-09-22T21:26:46.820 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.820 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TOTbLr6X1Y9Pvjk05cmPi1Co/vA= 2023-09-22T21:26:46.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TOTbLr6X1Y9Pvjk05cmPi1Co/vA=', 'Content-Length': '0'} 2023-09-22T21:26:46.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aca582a299b46db-00650e0696-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:26:46.823 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:26:46.824 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:26:46.824 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.824 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.824 DEBUG:boto:Method: GET 2023-09-22T21:26:46.824 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.824 DEBUG:boto:Data: 2023-09-22T21:26:46.824 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.824 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.825 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.825 DEBUG:boto:Params: {} 2023-09-22T21:26:46.825 DEBUG:boto:Token: None 2023-09-22T21:26:46.825 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.825 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NkkDYrgJOh+MUgsxukurwm1JE2U= 2023-09-22T21:26:46.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NkkDYrgJOh+MUgsxukurwm1JE2U=', 'Content-Length': '0'} 2023-09-22T21:26:46.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000828ed7ab55a929e3-00650e0696-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.870 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:26:46.871 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:26:46.871 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.871 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.871 DEBUG:boto:Method: GET 2023-09-22T21:26:46.871 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.871 DEBUG:boto:Data: 2023-09-22T21:26:46.871 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.871 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.871 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.871 DEBUG:boto:Params: {} 2023-09-22T21:26:46.872 DEBUG:boto:Token: None 2023-09-22T21:26:46.872 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:26:46.872 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NkkDYrgJOh+MUgsxukurwm1JE2U= 2023-09-22T21:26:46.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NkkDYrgJOh+MUgsxukurwm1JE2U=', 'Content-Length': '0'} 2023-09-22T21:26:46.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000739e5b13f4634e9e-00650e0696-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.875 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:46.875 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:46.875 DEBUG:boto:path=/ffzdqu-55/?prefix=c 2023-09-22T21:26:46.876 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=c 2023-09-22T21:26:46.876 DEBUG:boto:Method: GET 2023-09-22T21:26:46.876 DEBUG:boto:Path: /ffzdqu-55/?prefix=c 2023-09-22T21:26:46.876 DEBUG:boto:Data: 2023-09-22T21:26:46.876 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.876 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.876 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.876 DEBUG:boto:Params: {} 2023-09-22T21:26:46.876 DEBUG:boto:Token: None 2023-09-22T21:26:46.877 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/ 2023-09-22T21:26:46.877 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw= 2023-09-22T21:26:46.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw=', 'Content-Length': '0'} 2023-09-22T21:26:46.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003edd5bb2f1e9de85-00650e0696-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.883 DEBUG:boto:b'ffzdqu-55c1000falsec2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:46.883 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:46.884 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:46.884 DEBUG:boto:path=/ffzdqu-55/?prefix=c 2023-09-22T21:26:46.884 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=c 2023-09-22T21:26:46.884 DEBUG:boto:Method: GET 2023-09-22T21:26:46.884 DEBUG:boto:Path: /ffzdqu-55/?prefix=c 2023-09-22T21:26:46.884 DEBUG:boto:Data: 2023-09-22T21:26:46.884 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.884 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.884 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.884 DEBUG:boto:Params: {} 2023-09-22T21:26:46.885 DEBUG:boto:Token: None 2023-09-22T21:26:46.885 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/ 2023-09-22T21:26:46.885 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw= 2023-09-22T21:26:46.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw=', 'Content-Length': '0'} 2023-09-22T21:26:46.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002153c3a3ff541ee3-00650e0696-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.888 DEBUG:boto:b'ffzdqu-55c1000falsec2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:46.889 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:26:46.889 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:26:46.889 DEBUG:boto:Method: HEAD 2023-09-22T21:26:46.889 DEBUG:boto:Path: /ffzdqu-55/c 2023-09-22T21:26:46.889 DEBUG:boto:Data: 2023-09-22T21:26:46.889 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.889 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.890 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.890 DEBUG:boto:Params: {} 2023-09-22T21:26:46.890 DEBUG:boto:Token: None 2023-09-22T21:26:46.890 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/c 2023-09-22T21:26:46.890 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n9Gl5HnCg5Vrt1kWRqbr3n8Hock= 2023-09-22T21:26:46.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n9Gl5HnCg5Vrt1kWRqbr3n8Hock=', 'Content-Length': '0'} 2023-09-22T21:26:46.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d02d12d63dae59e1-00650e0696-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.893 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:26:46.893 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:26:46.893 DEBUG:boto:Method: HEAD 2023-09-22T21:26:46.894 DEBUG:boto:Path: /ffzdqu-55/c 2023-09-22T21:26:46.894 DEBUG:boto:Data: 2023-09-22T21:26:46.894 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.894 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.894 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.894 DEBUG:boto:Params: {} 2023-09-22T21:26:46.894 DEBUG:boto:Token: None 2023-09-22T21:26:46.894 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/c 2023-09-22T21:26:46.894 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n9Gl5HnCg5Vrt1kWRqbr3n8Hock= 2023-09-22T21:26:46.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n9Gl5HnCg5Vrt1kWRqbr3n8Hock=', 'Content-Length': '0'} 2023-09-22T21:26:46.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000982a528b247ba673-00650e0696-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:26:46.898 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:26:46.898 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:26:46.898 DEBUG:boto:Method: GET 2023-09-22T21:26:46.898 DEBUG:boto:Path: /ffzdqu-55/c 2023-09-22T21:26:46.898 DEBUG:boto:Data: 2023-09-22T21:26:46.898 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.898 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.899 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.899 DEBUG:boto:Params: {} 2023-09-22T21:26:46.899 DEBUG:boto:Token: None 2023-09-22T21:26:46.899 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/c 2023-09-22T21:26:46.899 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C0ZssC1+tLpGlGIF+XRZAjLNJ0g= 2023-09-22T21:26:46.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C0ZssC1+tLpGlGIF+XRZAjLNJ0g=', 'Content-Length': '0'} 2023-09-22T21:26:46.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a8fe9583203d51b-00650e0696-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.942 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:26:46.943 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:26:46.943 DEBUG:boto:Method: GET 2023-09-22T21:26:46.943 DEBUG:boto:Path: /ffzdqu-55/c 2023-09-22T21:26:46.943 DEBUG:boto:Data: 2023-09-22T21:26:46.943 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.943 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.943 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.943 DEBUG:boto:Params: {} 2023-09-22T21:26:46.943 DEBUG:boto:Token: None 2023-09-22T21:26:46.944 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/c 2023-09-22T21:26:46.944 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C0ZssC1+tLpGlGIF+XRZAjLNJ0g= 2023-09-22T21:26:46.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C0ZssC1+tLpGlGIF+XRZAjLNJ0g=', 'Content-Length': '0'} 2023-09-22T21:26:46.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006646dd8a53db590a-00650e0696-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.947 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:46.948 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:46.948 DEBUG:boto:path=/ffzdqu-55/?prefix=c 2023-09-22T21:26:46.948 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=c 2023-09-22T21:26:46.948 DEBUG:boto:Method: GET 2023-09-22T21:26:46.948 DEBUG:boto:Path: /ffzdqu-55/?prefix=c 2023-09-22T21:26:46.948 DEBUG:boto:Data: 2023-09-22T21:26:46.948 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.948 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.948 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.948 DEBUG:boto:Params: {} 2023-09-22T21:26:46.949 DEBUG:boto:Token: None 2023-09-22T21:26:46.949 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/ 2023-09-22T21:26:46.949 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw= 2023-09-22T21:26:46.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw=', 'Content-Length': '0'} 2023-09-22T21:26:46.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010db35708bb656f5-00650e0696-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.954 DEBUG:boto:b'ffzdqu-55c1000falsec2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:46.955 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:46.955 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:46.955 DEBUG:boto:path=/ffzdqu-55/?prefix=c 2023-09-22T21:26:46.955 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=c 2023-09-22T21:26:46.955 DEBUG:boto:Method: GET 2023-09-22T21:26:46.955 DEBUG:boto:Path: /ffzdqu-55/?prefix=c 2023-09-22T21:26:46.956 DEBUG:boto:Data: 2023-09-22T21:26:46.956 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.956 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.956 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.956 DEBUG:boto:Params: {} 2023-09-22T21:26:46.956 DEBUG:boto:Token: None 2023-09-22T21:26:46.956 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/ 2023-09-22T21:26:46.956 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw= 2023-09-22T21:26:46.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Mmw3iJlyPz3VIDJYqsrlzMYn/Cw=', 'Content-Length': '0'} 2023-09-22T21:26:46.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e04b85b57b43fd85-00650e0696-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.960 DEBUG:boto:b'ffzdqu-55c1000falsec2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:46.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:26:46.960 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:26:46.960 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:26:46.961 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:46.961 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:46.961 DEBUG:boto:Method: GET 2023-09-22T21:26:46.961 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:26:46.961 DEBUG:boto:Data: 2023-09-22T21:26:46.961 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.961 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.961 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.961 DEBUG:boto:Params: {} 2023-09-22T21:26:46.962 DEBUG:boto:Token: None 2023-09-22T21:26:46.962 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:26:46.962 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+aSMDJ6/Vr0xXgTZdbumKFdj/rY= 2023-09-22T21:26:46.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+aSMDJ6/Vr0xXgTZdbumKFdj/rY=', 'Content-Length': '0'} 2023-09-22T21:26:46.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f81ee0c785f86ab-00650e0696-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.965 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:26:46.965 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:26:46.965 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:46.965 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:46.965 DEBUG:boto:Method: GET 2023-09-22T21:26:46.965 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:26:46.966 DEBUG:boto:Data: 2023-09-22T21:26:46.966 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.966 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.966 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.966 DEBUG:boto:Params: {} 2023-09-22T21:26:46.966 DEBUG:boto:Token: None 2023-09-22T21:26:46.966 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:26:46.966 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+aSMDJ6/Vr0xXgTZdbumKFdj/rY= 2023-09-22T21:26:46.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+aSMDJ6/Vr0xXgTZdbumKFdj/rY=', 'Content-Length': '0'} 2023-09-22T21:26:46.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051369d096c269b25-00650e0696-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.970 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:26:46.970 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:26:46.970 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:46.970 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:46.970 DEBUG:boto:Method: HEAD 2023-09-22T21:26:46.971 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:26:46.971 DEBUG:boto:Data: 2023-09-22T21:26:46.971 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.971 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.971 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.971 DEBUG:boto:Params: {} 2023-09-22T21:26:46.971 DEBUG:boto:Token: None 2023-09-22T21:26:46.971 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:26:46.971 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oToiDRXS2k+mmXgfaN6gp/PaP+I= 2023-09-22T21:26:46.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oToiDRXS2k+mmXgfaN6gp/PaP+I=', 'Content-Length': '0'} 2023-09-22T21:26:46.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005628db0d4fe89960-00650e0696-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.975 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:26:46.975 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:26:46.975 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:46.975 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:46.975 DEBUG:boto:Method: HEAD 2023-09-22T21:26:46.975 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:26:46.975 DEBUG:boto:Data: 2023-09-22T21:26:46.975 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.975 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:46.976 DEBUG:boto:Port: 8000 2023-09-22T21:26:46.976 DEBUG:boto:Params: {} 2023-09-22T21:26:46.976 DEBUG:boto:Token: None 2023-09-22T21:26:46.976 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:26:46.976 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oToiDRXS2k+mmXgfaN6gp/PaP+I= 2023-09-22T21:26:46.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oToiDRXS2k+mmXgfaN6gp/PaP+I=', 'Content-Length': '0'} 2023-09-22T21:26:46.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bae2f9782b479338-00650e0696-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:46.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:26:46.980 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:26:46.980 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:26:46.980 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:46.980 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:46.980 DEBUG:boto:Method: GET 2023-09-22T21:26:46.980 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:26:46.980 DEBUG:boto:Data: 2023-09-22T21:26:46.980 DEBUG:boto:Headers: {} 2023-09-22T21:26:46.981 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:46.981 DEBUG:boto:Port: 8001 2023-09-22T21:26:46.981 DEBUG:boto:Params: {} 2023-09-22T21:26:46.981 DEBUG:boto:Token: None 2023-09-22T21:26:46.981 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:46 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:26:46.981 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+aSMDJ6/Vr0xXgTZdbumKFdj/rY= 2023-09-22T21:26:46.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+aSMDJ6/Vr0xXgTZdbumKFdj/rY=', 'Content-Length': '0'} 2023-09-22T21:26:46.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfaf468e19c82280-00650e0696-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.026 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:26:47.026 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:26:47.027 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:47.027 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:26:47.027 DEBUG:boto:Method: GET 2023-09-22T21:26:47.027 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:26:47.027 DEBUG:boto:Data: 2023-09-22T21:26:47.027 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.027 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.027 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.028 DEBUG:boto:Params: {} 2023-09-22T21:26:47.028 DEBUG:boto:Token: None 2023-09-22T21:26:47.028 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:26:47.028 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PuC4S39QuwkzKk9hC3E77dVO9II= 2023-09-22T21:26:47.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PuC4S39QuwkzKk9hC3E77dVO9II=', 'Content-Length': '0'} 2023-09-22T21:26:47.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca70c8323b0ed497-00650e0697-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.032 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:47.032 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:47.032 DEBUG:boto:path=/ffzdqu-55/?prefix=d 2023-09-22T21:26:47.032 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=d 2023-09-22T21:26:47.032 DEBUG:boto:Method: GET 2023-09-22T21:26:47.032 DEBUG:boto:Path: /ffzdqu-55/?prefix=d 2023-09-22T21:26:47.032 DEBUG:boto:Data: 2023-09-22T21:26:47.033 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.033 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.033 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.033 DEBUG:boto:Params: {} 2023-09-22T21:26:47.033 DEBUG:boto:Token: None 2023-09-22T21:26:47.033 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/ 2023-09-22T21:26:47.033 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU= 2023-09-22T21:26:47.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU=', 'Content-Length': '0'} 2023-09-22T21:26:47.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f40c4794d431bc89-00650e0697-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.040 DEBUG:boto:b'ffzdqu-55d1000falsed2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:47.040 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:47.040 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:47.040 DEBUG:boto:path=/ffzdqu-55/?prefix=d 2023-09-22T21:26:47.041 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=d 2023-09-22T21:26:47.041 DEBUG:boto:Method: GET 2023-09-22T21:26:47.041 DEBUG:boto:Path: /ffzdqu-55/?prefix=d 2023-09-22T21:26:47.041 DEBUG:boto:Data: 2023-09-22T21:26:47.041 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.041 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.041 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.041 DEBUG:boto:Params: {} 2023-09-22T21:26:47.042 DEBUG:boto:Token: None 2023-09-22T21:26:47.042 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/ 2023-09-22T21:26:47.042 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU= 2023-09-22T21:26:47.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU=', 'Content-Length': '0'} 2023-09-22T21:26:47.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7998cf5e5880ebf-00650e0697-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.086 DEBUG:boto:b'ffzdqu-55d1000falsed2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:47.087 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:26:47.087 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:26:47.087 DEBUG:boto:Method: HEAD 2023-09-22T21:26:47.087 DEBUG:boto:Path: /ffzdqu-55/d 2023-09-22T21:26:47.087 DEBUG:boto:Data: 2023-09-22T21:26:47.087 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.087 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.087 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.088 DEBUG:boto:Params: {} 2023-09-22T21:26:47.088 DEBUG:boto:Token: None 2023-09-22T21:26:47.088 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/d 2023-09-22T21:26:47.088 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mpQEK4JaoWFp26rTc52jwXHEP+E= 2023-09-22T21:26:47.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mpQEK4JaoWFp26rTc52jwXHEP+E=', 'Content-Length': '0'} 2023-09-22T21:26:47.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000112cb0f7d57a62f4-00650e0697-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.091 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:26:47.091 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:26:47.091 DEBUG:boto:Method: HEAD 2023-09-22T21:26:47.091 DEBUG:boto:Path: /ffzdqu-55/d 2023-09-22T21:26:47.091 DEBUG:boto:Data: 2023-09-22T21:26:47.091 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.092 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.092 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.092 DEBUG:boto:Params: {} 2023-09-22T21:26:47.092 DEBUG:boto:Token: None 2023-09-22T21:26:47.092 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/d 2023-09-22T21:26:47.092 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mpQEK4JaoWFp26rTc52jwXHEP+E= 2023-09-22T21:26:47.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mpQEK4JaoWFp26rTc52jwXHEP+E=', 'Content-Length': '0'} 2023-09-22T21:26:47.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000587d0f1c92e35519-00650e0697-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:26:47.095 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:26:47.095 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:26:47.096 DEBUG:boto:Method: GET 2023-09-22T21:26:47.096 DEBUG:boto:Path: /ffzdqu-55/d 2023-09-22T21:26:47.096 DEBUG:boto:Data: 2023-09-22T21:26:47.096 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.096 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.096 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.096 DEBUG:boto:Params: {} 2023-09-22T21:26:47.096 DEBUG:boto:Token: None 2023-09-22T21:26:47.096 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/d 2023-09-22T21:26:47.097 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Zv5kuBlO0BTFb+9YCRG3/p2umpA= 2023-09-22T21:26:47.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Zv5kuBlO0BTFb+9YCRG3/p2umpA=', 'Content-Length': '0'} 2023-09-22T21:26:47.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acbc1498aceabbf3-00650e0697-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.143 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:26:47.143 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:26:47.143 DEBUG:boto:Method: GET 2023-09-22T21:26:47.143 DEBUG:boto:Path: /ffzdqu-55/d 2023-09-22T21:26:47.143 DEBUG:boto:Data: 2023-09-22T21:26:47.143 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.143 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.143 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.144 DEBUG:boto:Params: {} 2023-09-22T21:26:47.144 DEBUG:boto:Token: None 2023-09-22T21:26:47.144 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/d 2023-09-22T21:26:47.144 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Zv5kuBlO0BTFb+9YCRG3/p2umpA= 2023-09-22T21:26:47.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Zv5kuBlO0BTFb+9YCRG3/p2umpA=', 'Content-Length': '0'} 2023-09-22T21:26:47.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e84ff3d210e4b121-00650e0697-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.147 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:47.148 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:47.148 DEBUG:boto:path=/ffzdqu-55/?prefix=d 2023-09-22T21:26:47.148 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=d 2023-09-22T21:26:47.148 DEBUG:boto:Method: GET 2023-09-22T21:26:47.148 DEBUG:boto:Path: /ffzdqu-55/?prefix=d 2023-09-22T21:26:47.148 DEBUG:boto:Data: 2023-09-22T21:26:47.148 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.148 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.149 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.149 DEBUG:boto:Params: {} 2023-09-22T21:26:47.149 DEBUG:boto:Token: None 2023-09-22T21:26:47.149 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/ 2023-09-22T21:26:47.149 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU= 2023-09-22T21:26:47.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU=', 'Content-Length': '0'} 2023-09-22T21:26:47.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041078ea8aa30bf81-00650e0697-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.156 DEBUG:boto:b'ffzdqu-55d1000falsed2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:47.156 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:47.156 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:47.156 DEBUG:boto:path=/ffzdqu-55/?prefix=d 2023-09-22T21:26:47.157 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=d 2023-09-22T21:26:47.157 DEBUG:boto:Method: GET 2023-09-22T21:26:47.157 DEBUG:boto:Path: /ffzdqu-55/?prefix=d 2023-09-22T21:26:47.157 DEBUG:boto:Data: 2023-09-22T21:26:47.157 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.157 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.157 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.157 DEBUG:boto:Params: {} 2023-09-22T21:26:47.157 DEBUG:boto:Token: None 2023-09-22T21:26:47.157 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/ 2023-09-22T21:26:47.158 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU= 2023-09-22T21:26:47.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU=', 'Content-Length': '0'} 2023-09-22T21:26:47.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021110f54c9c75123-00650e0697-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.162 DEBUG:boto:b'ffzdqu-55d1000falsed2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:47.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-22T21:26:47.162 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:26:47.162 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:26:47.162 DEBUG:boto:path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.163 DEBUG:boto:auth_path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.163 DEBUG:boto:Method: GET 2023-09-22T21:26:47.163 DEBUG:boto:Path: /ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.163 DEBUG:boto:Data: 2023-09-22T21:26:47.163 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.163 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.163 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.163 DEBUG:boto:Params: {} 2023-09-22T21:26:47.164 DEBUG:boto:Token: None 2023-09-22T21:26:47.164 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.164 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9JqbMODVXHUKxw2XZfpms/uKQ1Y= 2023-09-22T21:26:47.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9JqbMODVXHUKxw2XZfpms/uKQ1Y=', 'Content-Length': '0'} 2023-09-22T21:26:47.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005986911fbd2ab5ad-00650e0697-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.167 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:26:47.167 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:26:47.167 DEBUG:boto:path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.168 DEBUG:boto:auth_path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.168 DEBUG:boto:Method: GET 2023-09-22T21:26:47.168 DEBUG:boto:Path: /ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.168 DEBUG:boto:Data: 2023-09-22T21:26:47.168 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.168 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.168 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.168 DEBUG:boto:Params: {} 2023-09-22T21:26:47.169 DEBUG:boto:Token: None 2023-09-22T21:26:47.169 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.169 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9JqbMODVXHUKxw2XZfpms/uKQ1Y= 2023-09-22T21:26:47.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9JqbMODVXHUKxw2XZfpms/uKQ1Y=', 'Content-Length': '0'} 2023-09-22T21:26:47.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9e3697b7aba723e-00650e0697-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.172 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:26:47.172 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:26:47.172 DEBUG:boto:path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.173 DEBUG:boto:auth_path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.173 DEBUG:boto:Method: HEAD 2023-09-22T21:26:47.173 DEBUG:boto:Path: /ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.173 DEBUG:boto:Data: 2023-09-22T21:26:47.173 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.173 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.173 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.173 DEBUG:boto:Params: {} 2023-09-22T21:26:47.174 DEBUG:boto:Token: None 2023-09-22T21:26:47.174 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.174 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VeGX+DO+tf2KFI8DqgCiQmy88Dk= 2023-09-22T21:26:47.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VeGX+DO+tf2KFI8DqgCiQmy88Dk=', 'Content-Length': '0'} 2023-09-22T21:26:47.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027b07565c5158bd8-00650e0697-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.177 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:26:47.177 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:26:47.177 DEBUG:boto:path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.177 DEBUG:boto:auth_path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.177 DEBUG:boto:Method: HEAD 2023-09-22T21:26:47.178 DEBUG:boto:Path: /ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.178 DEBUG:boto:Data: 2023-09-22T21:26:47.178 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.178 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.178 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.178 DEBUG:boto:Params: {} 2023-09-22T21:26:47.178 DEBUG:boto:Token: None 2023-09-22T21:26:47.178 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.179 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VeGX+DO+tf2KFI8DqgCiQmy88Dk= 2023-09-22T21:26:47.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VeGX+DO+tf2KFI8DqgCiQmy88Dk=', 'Content-Length': '0'} 2023-09-22T21:26:47.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b1571dced77b05f-00650e0697-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-22T21:26:47.182 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:26:47.182 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:26:47.182 DEBUG:boto:path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.182 DEBUG:boto:auth_path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.182 DEBUG:boto:Method: GET 2023-09-22T21:26:47.182 DEBUG:boto:Path: /ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.182 DEBUG:boto:Data: 2023-09-22T21:26:47.182 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.182 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.182 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.183 DEBUG:boto:Params: {} 2023-09-22T21:26:47.183 DEBUG:boto:Token: None 2023-09-22T21:26:47.183 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.183 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9JqbMODVXHUKxw2XZfpms/uKQ1Y= 2023-09-22T21:26:47.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9JqbMODVXHUKxw2XZfpms/uKQ1Y=', 'Content-Length': '0'} 2023-09-22T21:26:47.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac5e10f1f5df79f2-00650e0697-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.226 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:26:47.226 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:26:47.227 DEBUG:boto:path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.227 DEBUG:boto:auth_path=/ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.227 DEBUG:boto:Method: GET 2023-09-22T21:26:47.227 DEBUG:boto:Path: /ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.227 DEBUG:boto:Data: 2023-09-22T21:26:47.227 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.227 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.227 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.227 DEBUG:boto:Params: {} 2023-09-22T21:26:47.228 DEBUG:boto:Token: None 2023-09-22T21:26:47.228 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/e?versionId=null 2023-09-22T21:26:47.228 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9JqbMODVXHUKxw2XZfpms/uKQ1Y= 2023-09-22T21:26:47.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9JqbMODVXHUKxw2XZfpms/uKQ1Y=', 'Content-Length': '0'} 2023-09-22T21:26:47.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b8be69aed74547c-00650e0697-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.231 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:47.232 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:47.232 DEBUG:boto:path=/ffzdqu-55/?prefix=e 2023-09-22T21:26:47.232 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=e 2023-09-22T21:26:47.232 DEBUG:boto:Method: GET 2023-09-22T21:26:47.232 DEBUG:boto:Path: /ffzdqu-55/?prefix=e 2023-09-22T21:26:47.232 DEBUG:boto:Data: 2023-09-22T21:26:47.232 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.232 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.232 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.233 DEBUG:boto:Params: {} 2023-09-22T21:26:47.233 DEBUG:boto:Token: None 2023-09-22T21:26:47.233 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/ 2023-09-22T21:26:47.233 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU= 2023-09-22T21:26:47.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU=', 'Content-Length': '0'} 2023-09-22T21:26:47.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000404a38cf1087a160-00650e0697-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.239 DEBUG:boto:b'ffzdqu-55e1000falsee2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:47.239 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:47.239 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:47.240 DEBUG:boto:path=/ffzdqu-55/?prefix=e 2023-09-22T21:26:47.240 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=e 2023-09-22T21:26:47.240 DEBUG:boto:Method: GET 2023-09-22T21:26:47.240 DEBUG:boto:Path: /ffzdqu-55/?prefix=e 2023-09-22T21:26:47.240 DEBUG:boto:Data: 2023-09-22T21:26:47.240 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.240 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.240 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.240 DEBUG:boto:Params: {} 2023-09-22T21:26:47.241 DEBUG:boto:Token: None 2023-09-22T21:26:47.241 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/ 2023-09-22T21:26:47.241 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU= 2023-09-22T21:26:47.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU=', 'Content-Length': '0'} 2023-09-22T21:26:47.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a86613afe9977acc-00650e0697-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.246 DEBUG:boto:b'ffzdqu-55e1000falsee2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:47.246 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:26:47.246 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:26:47.246 DEBUG:boto:Method: HEAD 2023-09-22T21:26:47.246 DEBUG:boto:Path: /ffzdqu-55/e 2023-09-22T21:26:47.246 DEBUG:boto:Data: 2023-09-22T21:26:47.247 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.247 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.247 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.247 DEBUG:boto:Params: {} 2023-09-22T21:26:47.247 DEBUG:boto:Token: None 2023-09-22T21:26:47.247 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/e 2023-09-22T21:26:47.247 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:X11eouXtpiK7mfidj+ZweadLKug= 2023-09-22T21:26:47.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:X11eouXtpiK7mfidj+ZweadLKug=', 'Content-Length': '0'} 2023-09-22T21:26:47.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be84d53e8b87b1d3-00650e0697-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.250 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:26:47.250 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:26:47.250 DEBUG:boto:Method: HEAD 2023-09-22T21:26:47.251 DEBUG:boto:Path: /ffzdqu-55/e 2023-09-22T21:26:47.251 DEBUG:boto:Data: 2023-09-22T21:26:47.251 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.251 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.251 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.251 DEBUG:boto:Params: {} 2023-09-22T21:26:47.251 DEBUG:boto:Token: None 2023-09-22T21:26:47.251 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/e 2023-09-22T21:26:47.251 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:X11eouXtpiK7mfidj+ZweadLKug= 2023-09-22T21:26:47.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:X11eouXtpiK7mfidj+ZweadLKug=', 'Content-Length': '0'} 2023-09-22T21:26:47.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e78e75ccd582ad9-00650e0697-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-22T21:26:47.255 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:26:47.255 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:26:47.255 DEBUG:boto:Method: GET 2023-09-22T21:26:47.255 DEBUG:boto:Path: /ffzdqu-55/e 2023-09-22T21:26:47.255 DEBUG:boto:Data: 2023-09-22T21:26:47.255 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.255 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.255 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.255 DEBUG:boto:Params: {} 2023-09-22T21:26:47.256 DEBUG:boto:Token: None 2023-09-22T21:26:47.256 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/e 2023-09-22T21:26:47.256 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lMNU4+3CjBYqjlWqYjTB/1RS2Zo= 2023-09-22T21:26:47.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lMNU4+3CjBYqjlWqYjTB/1RS2Zo=', 'Content-Length': '0'} 2023-09-22T21:26:47.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c2fbfb974e9a645-00650e0697-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.302 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:26:47.302 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:26:47.303 DEBUG:boto:Method: GET 2023-09-22T21:26:47.303 DEBUG:boto:Path: /ffzdqu-55/e 2023-09-22T21:26:47.303 DEBUG:boto:Data: 2023-09-22T21:26:47.303 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.303 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.303 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.303 DEBUG:boto:Params: {} 2023-09-22T21:26:47.303 DEBUG:boto:Token: None 2023-09-22T21:26:47.303 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/e 2023-09-22T21:26:47.304 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lMNU4+3CjBYqjlWqYjTB/1RS2Zo= 2023-09-22T21:26:47.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lMNU4+3CjBYqjlWqYjTB/1RS2Zo=', 'Content-Length': '0'} 2023-09-22T21:26:47.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000021e77d2341f5c86-00650e0697-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.307 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:47.307 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:47.307 DEBUG:boto:path=/ffzdqu-55/?prefix=e 2023-09-22T21:26:47.307 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=e 2023-09-22T21:26:47.307 DEBUG:boto:Method: GET 2023-09-22T21:26:47.307 DEBUG:boto:Path: /ffzdqu-55/?prefix=e 2023-09-22T21:26:47.307 DEBUG:boto:Data: 2023-09-22T21:26:47.307 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.308 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.308 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.308 DEBUG:boto:Params: {} 2023-09-22T21:26:47.308 DEBUG:boto:Token: None 2023-09-22T21:26:47.308 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/ 2023-09-22T21:26:47.308 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU= 2023-09-22T21:26:47.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU=', 'Content-Length': '0'} 2023-09-22T21:26:47.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ac3c7972bbbfe76-00650e0697-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.358 DEBUG:boto:b'ffzdqu-55e1000falsee2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:47.359 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:47.359 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:47.359 DEBUG:boto:path=/ffzdqu-55/?prefix=e 2023-09-22T21:26:47.359 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=e 2023-09-22T21:26:47.359 DEBUG:boto:Method: GET 2023-09-22T21:26:47.359 DEBUG:boto:Path: /ffzdqu-55/?prefix=e 2023-09-22T21:26:47.359 DEBUG:boto:Data: 2023-09-22T21:26:47.359 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.360 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.360 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.360 DEBUG:boto:Params: {} 2023-09-22T21:26:47.360 DEBUG:boto:Token: None 2023-09-22T21:26:47.360 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/ 2023-09-22T21:26:47.360 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU= 2023-09-22T21:26:47.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU=', 'Content-Length': '0'} 2023-09-22T21:26:47.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084f73a40f2486db8-00650e0697-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.364 DEBUG:boto:b'ffzdqu-55e1000falsee2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:47.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-22T21:26:47.365 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:26:47.365 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:26:47.365 DEBUG:boto:path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.365 DEBUG:boto:auth_path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.365 DEBUG:boto:Method: GET 2023-09-22T21:26:47.365 DEBUG:boto:Path: /ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.365 DEBUG:boto:Data: 2023-09-22T21:26:47.365 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.365 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.366 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.366 DEBUG:boto:Params: {} 2023-09-22T21:26:47.366 DEBUG:boto:Token: None 2023-09-22T21:26:47.366 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.366 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P0V4eO8rzNxcvG/BFes1v8u7GEQ= 2023-09-22T21:26:47.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P0V4eO8rzNxcvG/BFes1v8u7GEQ=', 'Content-Length': '0'} 2023-09-22T21:26:47.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e079d24939d5dc7d-00650e0697-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.369 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:26:47.370 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:26:47.370 DEBUG:boto:path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.370 DEBUG:boto:auth_path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.370 DEBUG:boto:Method: GET 2023-09-22T21:26:47.370 DEBUG:boto:Path: /ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.370 DEBUG:boto:Data: 2023-09-22T21:26:47.370 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.370 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.371 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.371 DEBUG:boto:Params: {} 2023-09-22T21:26:47.371 DEBUG:boto:Token: None 2023-09-22T21:26:47.371 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.371 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P0V4eO8rzNxcvG/BFes1v8u7GEQ= 2023-09-22T21:26:47.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P0V4eO8rzNxcvG/BFes1v8u7GEQ=', 'Content-Length': '0'} 2023-09-22T21:26:47.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e86cc4b0d837a5a-00650e0697-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.414 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:26:47.414 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:26:47.415 DEBUG:boto:path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.415 DEBUG:boto:auth_path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.415 DEBUG:boto:Method: HEAD 2023-09-22T21:26:47.415 DEBUG:boto:Path: /ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.415 DEBUG:boto:Data: 2023-09-22T21:26:47.415 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.415 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.415 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.415 DEBUG:boto:Params: {} 2023-09-22T21:26:47.416 DEBUG:boto:Token: None 2023-09-22T21:26:47.416 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.416 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:l66hrb6m/gOBZpmHB4GMjqc/HZI= 2023-09-22T21:26:47.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:l66hrb6m/gOBZpmHB4GMjqc/HZI=', 'Content-Length': '0'} 2023-09-22T21:26:47.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eda924e940cbac43-00650e0697-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.419 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:26:47.419 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:26:47.419 DEBUG:boto:path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.420 DEBUG:boto:auth_path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.420 DEBUG:boto:Method: HEAD 2023-09-22T21:26:47.420 DEBUG:boto:Path: /ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.420 DEBUG:boto:Data: 2023-09-22T21:26:47.420 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.420 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.420 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.420 DEBUG:boto:Params: {} 2023-09-22T21:26:47.420 DEBUG:boto:Token: None 2023-09-22T21:26:47.421 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.421 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:l66hrb6m/gOBZpmHB4GMjqc/HZI= 2023-09-22T21:26:47.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:l66hrb6m/gOBZpmHB4GMjqc/HZI=', 'Content-Length': '0'} 2023-09-22T21:26:47.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007661e66f51396c67-00650e0697-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-22T21:26:47.424 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:26:47.424 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:26:47.424 DEBUG:boto:path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.424 DEBUG:boto:auth_path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.424 DEBUG:boto:Method: GET 2023-09-22T21:26:47.425 DEBUG:boto:Path: /ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.425 DEBUG:boto:Data: 2023-09-22T21:26:47.425 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.425 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.425 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.425 DEBUG:boto:Params: {} 2023-09-22T21:26:47.425 DEBUG:boto:Token: None 2023-09-22T21:26:47.426 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.426 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P0V4eO8rzNxcvG/BFes1v8u7GEQ= 2023-09-22T21:26:47.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P0V4eO8rzNxcvG/BFes1v8u7GEQ=', 'Content-Length': '0'} 2023-09-22T21:26:47.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006166324ef34a6256-00650e0697-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.470 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:26:47.470 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:26:47.470 DEBUG:boto:path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.471 DEBUG:boto:auth_path=/ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.471 DEBUG:boto:Method: GET 2023-09-22T21:26:47.471 DEBUG:boto:Path: /ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.471 DEBUG:boto:Data: 2023-09-22T21:26:47.471 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.471 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.471 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.471 DEBUG:boto:Params: {} 2023-09-22T21:26:47.471 DEBUG:boto:Token: None 2023-09-22T21:26:47.472 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/f?versionId=null 2023-09-22T21:26:47.472 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P0V4eO8rzNxcvG/BFes1v8u7GEQ= 2023-09-22T21:26:47.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P0V4eO8rzNxcvG/BFes1v8u7GEQ=', 'Content-Length': '0'} 2023-09-22T21:26:47.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b04eb1b13bc8d3b-00650e0697-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.475 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:47.475 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:47.475 DEBUG:boto:path=/ffzdqu-55/?prefix=f 2023-09-22T21:26:47.475 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=f 2023-09-22T21:26:47.475 DEBUG:boto:Method: GET 2023-09-22T21:26:47.476 DEBUG:boto:Path: /ffzdqu-55/?prefix=f 2023-09-22T21:26:47.476 DEBUG:boto:Data: 2023-09-22T21:26:47.476 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.476 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.476 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.476 DEBUG:boto:Params: {} 2023-09-22T21:26:47.476 DEBUG:boto:Token: None 2023-09-22T21:26:47.476 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/ 2023-09-22T21:26:47.477 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU= 2023-09-22T21:26:47.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU=', 'Content-Length': '0'} 2023-09-22T21:26:47.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008411f88edf6517b3-00650e0697-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.522 DEBUG:boto:b'ffzdqu-55f1000falsef2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:47.523 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:47.523 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:47.523 DEBUG:boto:path=/ffzdqu-55/?prefix=f 2023-09-22T21:26:47.523 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=f 2023-09-22T21:26:47.523 DEBUG:boto:Method: GET 2023-09-22T21:26:47.523 DEBUG:boto:Path: /ffzdqu-55/?prefix=f 2023-09-22T21:26:47.523 DEBUG:boto:Data: 2023-09-22T21:26:47.524 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.524 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.524 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.524 DEBUG:boto:Params: {} 2023-09-22T21:26:47.524 DEBUG:boto:Token: None 2023-09-22T21:26:47.524 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/ 2023-09-22T21:26:47.524 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU= 2023-09-22T21:26:47.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU=', 'Content-Length': '0'} 2023-09-22T21:26:47.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0acbe3f774c3f44-00650e0697-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.529 DEBUG:boto:b'ffzdqu-55f1000falsef2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:47.530 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:26:47.530 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:26:47.530 DEBUG:boto:Method: HEAD 2023-09-22T21:26:47.530 DEBUG:boto:Path: /ffzdqu-55/f 2023-09-22T21:26:47.530 DEBUG:boto:Data: 2023-09-22T21:26:47.530 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.530 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.530 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.531 DEBUG:boto:Params: {} 2023-09-22T21:26:47.531 DEBUG:boto:Token: None 2023-09-22T21:26:47.531 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/f 2023-09-22T21:26:47.531 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7roJ6WrCpYzqnR7mM5rc1OdvYJs= 2023-09-22T21:26:47.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7roJ6WrCpYzqnR7mM5rc1OdvYJs=', 'Content-Length': '0'} 2023-09-22T21:26:47.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d936ffadeebf6197-00650e0697-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.534 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:26:47.534 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:26:47.534 DEBUG:boto:Method: HEAD 2023-09-22T21:26:47.534 DEBUG:boto:Path: /ffzdqu-55/f 2023-09-22T21:26:47.535 DEBUG:boto:Data: 2023-09-22T21:26:47.535 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.535 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.535 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.535 DEBUG:boto:Params: {} 2023-09-22T21:26:47.535 DEBUG:boto:Token: None 2023-09-22T21:26:47.535 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/f 2023-09-22T21:26:47.535 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7roJ6WrCpYzqnR7mM5rc1OdvYJs= 2023-09-22T21:26:47.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7roJ6WrCpYzqnR7mM5rc1OdvYJs=', 'Content-Length': '0'} 2023-09-22T21:26:47.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000528c49c5a64f008-00650e0697-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-22T21:26:47.539 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:26:47.539 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:26:47.539 DEBUG:boto:Method: GET 2023-09-22T21:26:47.539 DEBUG:boto:Path: /ffzdqu-55/f 2023-09-22T21:26:47.539 DEBUG:boto:Data: 2023-09-22T21:26:47.540 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.540 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.540 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.540 DEBUG:boto:Params: {} 2023-09-22T21:26:47.540 DEBUG:boto:Token: None 2023-09-22T21:26:47.540 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/f 2023-09-22T21:26:47.540 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:U37NR2e3vvBmPuNUOMyb3cdIKzA= 2023-09-22T21:26:47.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:U37NR2e3vvBmPuNUOMyb3cdIKzA=', 'Content-Length': '0'} 2023-09-22T21:26:47.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009708d2dcac767159-00650e0697-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.586 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:26:47.587 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:26:47.587 DEBUG:boto:Method: GET 2023-09-22T21:26:47.587 DEBUG:boto:Path: /ffzdqu-55/f 2023-09-22T21:26:47.587 DEBUG:boto:Data: 2023-09-22T21:26:47.587 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.587 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.587 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.587 DEBUG:boto:Params: {} 2023-09-22T21:26:47.587 DEBUG:boto:Token: None 2023-09-22T21:26:47.588 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/f 2023-09-22T21:26:47.588 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:U37NR2e3vvBmPuNUOMyb3cdIKzA= 2023-09-22T21:26:47.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:U37NR2e3vvBmPuNUOMyb3cdIKzA=', 'Content-Length': '0'} 2023-09-22T21:26:47.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094917c8d64cccd56-00650e0697-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.591 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:47.591 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:47.591 DEBUG:boto:path=/ffzdqu-55/?prefix=f 2023-09-22T21:26:47.592 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=f 2023-09-22T21:26:47.592 DEBUG:boto:Method: GET 2023-09-22T21:26:47.592 DEBUG:boto:Path: /ffzdqu-55/?prefix=f 2023-09-22T21:26:47.592 DEBUG:boto:Data: 2023-09-22T21:26:47.592 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.592 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.592 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.592 DEBUG:boto:Params: {} 2023-09-22T21:26:47.592 DEBUG:boto:Token: None 2023-09-22T21:26:47.592 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/ 2023-09-22T21:26:47.593 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU= 2023-09-22T21:26:47.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU=', 'Content-Length': '0'} 2023-09-22T21:26:47.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d06b0f2d07debda-00650e0697-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.598 DEBUG:boto:b'ffzdqu-55f1000falsef2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:47.599 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:47.599 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:47.599 DEBUG:boto:path=/ffzdqu-55/?prefix=f 2023-09-22T21:26:47.599 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=f 2023-09-22T21:26:47.599 DEBUG:boto:Method: GET 2023-09-22T21:26:47.599 DEBUG:boto:Path: /ffzdqu-55/?prefix=f 2023-09-22T21:26:47.599 DEBUG:boto:Data: 2023-09-22T21:26:47.599 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.600 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.600 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.600 DEBUG:boto:Params: {} 2023-09-22T21:26:47.600 DEBUG:boto:Token: None 2023-09-22T21:26:47.600 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/ 2023-09-22T21:26:47.600 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU= 2023-09-22T21:26:47.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU=', 'Content-Length': '0'} 2023-09-22T21:26:47.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000953a2df5e8588fb8-00650e0697-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.646 DEBUG:boto:b'ffzdqu-55f1000falsef2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:47.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-22T21:26:47.647 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:26:47.647 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:26:47.647 DEBUG:boto:path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.647 DEBUG:boto:auth_path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.647 DEBUG:boto:Method: GET 2023-09-22T21:26:47.647 DEBUG:boto:Path: /ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.648 DEBUG:boto:Data: 2023-09-22T21:26:47.648 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.648 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.648 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.648 DEBUG:boto:Params: {} 2023-09-22T21:26:47.648 DEBUG:boto:Token: None 2023-09-22T21:26:47.648 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.648 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HcL1ntmAQmZ3pIxVR16SNuAeV88= 2023-09-22T21:26:47.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HcL1ntmAQmZ3pIxVR16SNuAeV88=', 'Content-Length': '0'} 2023-09-22T21:26:47.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008619c571737e7c67-00650e0697-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.694 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:26:47.694 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:26:47.695 DEBUG:boto:path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.695 DEBUG:boto:auth_path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.695 DEBUG:boto:Method: GET 2023-09-22T21:26:47.695 DEBUG:boto:Path: /ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.695 DEBUG:boto:Data: 2023-09-22T21:26:47.695 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.695 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.695 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.696 DEBUG:boto:Params: {} 2023-09-22T21:26:47.696 DEBUG:boto:Token: None 2023-09-22T21:26:47.696 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.696 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HcL1ntmAQmZ3pIxVR16SNuAeV88= 2023-09-22T21:26:47.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HcL1ntmAQmZ3pIxVR16SNuAeV88=', 'Content-Length': '0'} 2023-09-22T21:26:47.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fedadd25aa85f57-00650e0697-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.700 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:26:47.700 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:26:47.700 DEBUG:boto:path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.700 DEBUG:boto:auth_path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.700 DEBUG:boto:Method: HEAD 2023-09-22T21:26:47.700 DEBUG:boto:Path: /ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.700 DEBUG:boto:Data: 2023-09-22T21:26:47.700 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.700 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.701 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.701 DEBUG:boto:Params: {} 2023-09-22T21:26:47.701 DEBUG:boto:Token: None 2023-09-22T21:26:47.701 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.701 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Fv7T/wqwsOlsun6sXYoNBaYafYg= 2023-09-22T21:26:47.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Fv7T/wqwsOlsun6sXYoNBaYafYg=', 'Content-Length': '0'} 2023-09-22T21:26:47.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0dfa8c954094d7c-00650e0697-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.704 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:26:47.704 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:26:47.704 DEBUG:boto:path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.704 DEBUG:boto:auth_path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.704 DEBUG:boto:Method: HEAD 2023-09-22T21:26:47.705 DEBUG:boto:Path: /ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.705 DEBUG:boto:Data: 2023-09-22T21:26:47.705 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.705 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.705 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.705 DEBUG:boto:Params: {} 2023-09-22T21:26:47.705 DEBUG:boto:Token: None 2023-09-22T21:26:47.705 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.705 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Fv7T/wqwsOlsun6sXYoNBaYafYg= 2023-09-22T21:26:47.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Fv7T/wqwsOlsun6sXYoNBaYafYg=', 'Content-Length': '0'} 2023-09-22T21:26:47.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b0bfef2f4ee68a2-00650e0697-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-22T21:26:47.709 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:26:47.709 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:26:47.709 DEBUG:boto:path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.709 DEBUG:boto:auth_path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.709 DEBUG:boto:Method: GET 2023-09-22T21:26:47.709 DEBUG:boto:Path: /ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.709 DEBUG:boto:Data: 2023-09-22T21:26:47.709 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.709 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.710 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.710 DEBUG:boto:Params: {} 2023-09-22T21:26:47.710 DEBUG:boto:Token: None 2023-09-22T21:26:47.710 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.710 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HcL1ntmAQmZ3pIxVR16SNuAeV88= 2023-09-22T21:26:47.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HcL1ntmAQmZ3pIxVR16SNuAeV88=', 'Content-Length': '0'} 2023-09-22T21:26:47.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffcbd7784032cad9-00650e0697-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.754 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:26:47.754 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:26:47.755 DEBUG:boto:path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.755 DEBUG:boto:auth_path=/ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.755 DEBUG:boto:Method: GET 2023-09-22T21:26:47.755 DEBUG:boto:Path: /ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.755 DEBUG:boto:Data: 2023-09-22T21:26:47.755 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.755 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.755 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.755 DEBUG:boto:Params: {} 2023-09-22T21:26:47.755 DEBUG:boto:Token: None 2023-09-22T21:26:47.756 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/g?versionId=null 2023-09-22T21:26:47.756 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HcL1ntmAQmZ3pIxVR16SNuAeV88= 2023-09-22T21:26:47.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HcL1ntmAQmZ3pIxVR16SNuAeV88=', 'Content-Length': '0'} 2023-09-22T21:26:47.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ce0f74bd793aed4-00650e0697-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.759 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:47.760 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:47.760 DEBUG:boto:path=/ffzdqu-55/?prefix=g 2023-09-22T21:26:47.760 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=g 2023-09-22T21:26:47.760 DEBUG:boto:Method: GET 2023-09-22T21:26:47.760 DEBUG:boto:Path: /ffzdqu-55/?prefix=g 2023-09-22T21:26:47.760 DEBUG:boto:Data: 2023-09-22T21:26:47.760 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.760 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.760 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.760 DEBUG:boto:Params: {} 2023-09-22T21:26:47.761 DEBUG:boto:Token: None 2023-09-22T21:26:47.761 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/ 2023-09-22T21:26:47.761 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU= 2023-09-22T21:26:47.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU=', 'Content-Length': '0'} 2023-09-22T21:26:47.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064123d5342a8e1e1-00650e0697-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.810 DEBUG:boto:b'ffzdqu-55g1000falseg2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:47.811 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:47.811 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:47.811 DEBUG:boto:path=/ffzdqu-55/?prefix=g 2023-09-22T21:26:47.811 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=g 2023-09-22T21:26:47.811 DEBUG:boto:Method: GET 2023-09-22T21:26:47.811 DEBUG:boto:Path: /ffzdqu-55/?prefix=g 2023-09-22T21:26:47.811 DEBUG:boto:Data: 2023-09-22T21:26:47.812 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.812 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.812 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.812 DEBUG:boto:Params: {} 2023-09-22T21:26:47.812 DEBUG:boto:Token: None 2023-09-22T21:26:47.812 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/ 2023-09-22T21:26:47.812 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU= 2023-09-22T21:26:47.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU=', 'Content-Length': '0'} 2023-09-22T21:26:47.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9f26247485dcb17-00650e0697-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.816 DEBUG:boto:b'ffzdqu-55g1000falseg2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:47.817 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:26:47.817 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:26:47.817 DEBUG:boto:Method: HEAD 2023-09-22T21:26:47.817 DEBUG:boto:Path: /ffzdqu-55/g 2023-09-22T21:26:47.817 DEBUG:boto:Data: 2023-09-22T21:26:47.817 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.818 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.818 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.818 DEBUG:boto:Params: {} 2023-09-22T21:26:47.818 DEBUG:boto:Token: None 2023-09-22T21:26:47.818 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/g 2023-09-22T21:26:47.818 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BowlOt2PblMSHjnWXtddUWCIDkQ= 2023-09-22T21:26:47.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BowlOt2PblMSHjnWXtddUWCIDkQ=', 'Content-Length': '0'} 2023-09-22T21:26:47.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b0aca8f76c7b205-00650e0697-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.821 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:26:47.821 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:26:47.821 DEBUG:boto:Method: HEAD 2023-09-22T21:26:47.821 DEBUG:boto:Path: /ffzdqu-55/g 2023-09-22T21:26:47.822 DEBUG:boto:Data: 2023-09-22T21:26:47.822 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.822 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.822 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.822 DEBUG:boto:Params: {} 2023-09-22T21:26:47.822 DEBUG:boto:Token: None 2023-09-22T21:26:47.822 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/g 2023-09-22T21:26:47.822 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BowlOt2PblMSHjnWXtddUWCIDkQ= 2023-09-22T21:26:47.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BowlOt2PblMSHjnWXtddUWCIDkQ=', 'Content-Length': '0'} 2023-09-22T21:26:47.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccb6a5462361fab4-00650e0697-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-22T21:26:47.826 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:26:47.826 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:26:47.826 DEBUG:boto:Method: GET 2023-09-22T21:26:47.826 DEBUG:boto:Path: /ffzdqu-55/g 2023-09-22T21:26:47.826 DEBUG:boto:Data: 2023-09-22T21:26:47.826 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.826 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.826 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.826 DEBUG:boto:Params: {} 2023-09-22T21:26:47.827 DEBUG:boto:Token: None 2023-09-22T21:26:47.827 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/g 2023-09-22T21:26:47.827 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xRbtGe42xHmcuGYHMOOWHlpMd0I= 2023-09-22T21:26:47.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xRbtGe42xHmcuGYHMOOWHlpMd0I=', 'Content-Length': '0'} 2023-09-22T21:26:47.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba29e942427d40de-00650e0697-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.870 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:26:47.870 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:26:47.871 DEBUG:boto:Method: GET 2023-09-22T21:26:47.871 DEBUG:boto:Path: /ffzdqu-55/g 2023-09-22T21:26:47.871 DEBUG:boto:Data: 2023-09-22T21:26:47.871 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.871 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.871 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.871 DEBUG:boto:Params: {} 2023-09-22T21:26:47.871 DEBUG:boto:Token: None 2023-09-22T21:26:47.872 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/g 2023-09-22T21:26:47.872 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xRbtGe42xHmcuGYHMOOWHlpMd0I= 2023-09-22T21:26:47.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xRbtGe42xHmcuGYHMOOWHlpMd0I=', 'Content-Length': '0'} 2023-09-22T21:26:47.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb6e4da29fa0ae28-00650e0697-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.875 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:47.875 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:47.875 DEBUG:boto:path=/ffzdqu-55/?prefix=g 2023-09-22T21:26:47.875 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=g 2023-09-22T21:26:47.875 DEBUG:boto:Method: GET 2023-09-22T21:26:47.875 DEBUG:boto:Path: /ffzdqu-55/?prefix=g 2023-09-22T21:26:47.876 DEBUG:boto:Data: 2023-09-22T21:26:47.876 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.876 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.876 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.876 DEBUG:boto:Params: {} 2023-09-22T21:26:47.876 DEBUG:boto:Token: None 2023-09-22T21:26:47.876 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/ 2023-09-22T21:26:47.876 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU= 2023-09-22T21:26:47.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU=', 'Content-Length': '0'} 2023-09-22T21:26:47.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d302544cafd2ec8a-00650e0697-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.882 DEBUG:boto:b'ffzdqu-55g1000falseg2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:47.882 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:47.882 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:47.883 DEBUG:boto:path=/ffzdqu-55/?prefix=g 2023-09-22T21:26:47.883 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=g 2023-09-22T21:26:47.883 DEBUG:boto:Method: GET 2023-09-22T21:26:47.883 DEBUG:boto:Path: /ffzdqu-55/?prefix=g 2023-09-22T21:26:47.883 DEBUG:boto:Data: 2023-09-22T21:26:47.883 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.883 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.883 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.883 DEBUG:boto:Params: {} 2023-09-22T21:26:47.884 DEBUG:boto:Token: None 2023-09-22T21:26:47.884 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/ 2023-09-22T21:26:47.884 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU= 2023-09-22T21:26:47.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:acsbzKbyINYzILIF5OfAx94TbkU=', 'Content-Length': '0'} 2023-09-22T21:26:47.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000310985428bf24251-00650e0697-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.930 DEBUG:boto:b'ffzdqu-55g1000falseg2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:47.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-22T21:26:47.931 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:26:47.931 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:26:47.931 DEBUG:boto:path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:47.931 DEBUG:boto:auth_path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:47.931 DEBUG:boto:Method: GET 2023-09-22T21:26:47.931 DEBUG:boto:Path: /ffzdqu-55/h?versionId=null 2023-09-22T21:26:47.931 DEBUG:boto:Data: 2023-09-22T21:26:47.932 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.932 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.932 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.932 DEBUG:boto:Params: {} 2023-09-22T21:26:47.932 DEBUG:boto:Token: None 2023-09-22T21:26:47.932 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/h?versionId=null 2023-09-22T21:26:47.932 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3liCgoLsbFTO72l7ozisB2dWVYU= 2023-09-22T21:26:47.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3liCgoLsbFTO72l7ozisB2dWVYU=', 'Content-Length': '0'} 2023-09-22T21:26:47.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7a7a4ac5fbca8ef-00650e0697-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.978 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:26:47.978 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:26:47.979 DEBUG:boto:path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:47.979 DEBUG:boto:auth_path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:47.979 DEBUG:boto:Method: GET 2023-09-22T21:26:47.979 DEBUG:boto:Path: /ffzdqu-55/h?versionId=null 2023-09-22T21:26:47.979 DEBUG:boto:Data: 2023-09-22T21:26:47.979 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.979 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.979 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.980 DEBUG:boto:Params: {} 2023-09-22T21:26:47.980 DEBUG:boto:Token: None 2023-09-22T21:26:47.980 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/h?versionId=null 2023-09-22T21:26:47.980 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3liCgoLsbFTO72l7ozisB2dWVYU= 2023-09-22T21:26:47.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3liCgoLsbFTO72l7ozisB2dWVYU=', 'Content-Length': '0'} 2023-09-22T21:26:47.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000a16df749ef70b1-00650e0697-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.983 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:26:47.983 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:26:47.984 DEBUG:boto:path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:47.984 DEBUG:boto:auth_path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:47.984 DEBUG:boto:Method: HEAD 2023-09-22T21:26:47.984 DEBUG:boto:Path: /ffzdqu-55/h?versionId=null 2023-09-22T21:26:47.984 DEBUG:boto:Data: 2023-09-22T21:26:47.984 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.984 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.984 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.984 DEBUG:boto:Params: {} 2023-09-22T21:26:47.984 DEBUG:boto:Token: None 2023-09-22T21:26:47.985 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/h?versionId=null 2023-09-22T21:26:47.985 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BfZ5SY1h8Sl57rBFmhQv5ay5Aik= 2023-09-22T21:26:47.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BfZ5SY1h8Sl57rBFmhQv5ay5Aik=', 'Content-Length': '0'} 2023-09-22T21:26:47.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc7cd9197f32301c-00650e0697-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.988 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:26:47.988 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:26:47.988 DEBUG:boto:path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:47.988 DEBUG:boto:auth_path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:47.988 DEBUG:boto:Method: HEAD 2023-09-22T21:26:47.988 DEBUG:boto:Path: /ffzdqu-55/h?versionId=null 2023-09-22T21:26:47.988 DEBUG:boto:Data: 2023-09-22T21:26:47.988 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.988 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:47.989 DEBUG:boto:Port: 8000 2023-09-22T21:26:47.989 DEBUG:boto:Params: {} 2023-09-22T21:26:47.989 DEBUG:boto:Token: None 2023-09-22T21:26:47.989 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/h?versionId=null 2023-09-22T21:26:47.989 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BfZ5SY1h8Sl57rBFmhQv5ay5Aik= 2023-09-22T21:26:47.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BfZ5SY1h8Sl57rBFmhQv5ay5Aik=', 'Content-Length': '0'} 2023-09-22T21:26:47.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000846d7df94f1f285a-00650e0697-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:47.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-22T21:26:47.992 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:26:47.992 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:26:47.992 DEBUG:boto:path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:47.993 DEBUG:boto:auth_path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:47.993 DEBUG:boto:Method: GET 2023-09-22T21:26:47.993 DEBUG:boto:Path: /ffzdqu-55/h?versionId=null 2023-09-22T21:26:47.993 DEBUG:boto:Data: 2023-09-22T21:26:47.993 DEBUG:boto:Headers: {} 2023-09-22T21:26:47.993 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:47.993 DEBUG:boto:Port: 8001 2023-09-22T21:26:47.993 DEBUG:boto:Params: {} 2023-09-22T21:26:47.993 DEBUG:boto:Token: None 2023-09-22T21:26:47.993 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:47 GMT /ffzdqu-55/h?versionId=null 2023-09-22T21:26:47.994 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3liCgoLsbFTO72l7ozisB2dWVYU= 2023-09-22T21:26:47.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3liCgoLsbFTO72l7ozisB2dWVYU=', 'Content-Length': '0'} 2023-09-22T21:26:47.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016a405d8fa681499-00650e0697-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:48.038 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:26:48.038 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:26:48.039 DEBUG:boto:path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:48.039 DEBUG:boto:auth_path=/ffzdqu-55/h?versionId=null 2023-09-22T21:26:48.039 DEBUG:boto:Method: GET 2023-09-22T21:26:48.039 DEBUG:boto:Path: /ffzdqu-55/h?versionId=null 2023-09-22T21:26:48.039 DEBUG:boto:Data: 2023-09-22T21:26:48.039 DEBUG:boto:Headers: {} 2023-09-22T21:26:48.039 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:48.039 DEBUG:boto:Port: 8000 2023-09-22T21:26:48.040 DEBUG:boto:Params: {} 2023-09-22T21:26:48.040 DEBUG:boto:Token: None 2023-09-22T21:26:48.040 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:48 GMT /ffzdqu-55/h?versionId=null 2023-09-22T21:26:48.040 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bU7e2mc+tbKg9Hr9SQqCz8nfdd4= 2023-09-22T21:26:48.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bU7e2mc+tbKg9Hr9SQqCz8nfdd4=', 'Content-Length': '0'} 2023-09-22T21:26:48.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014deddc993ea8fcd-00650e0698-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:48.043 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:48.043 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:48.044 DEBUG:boto:path=/ffzdqu-55/?prefix=h 2023-09-22T21:26:48.044 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=h 2023-09-22T21:26:48.044 DEBUG:boto:Method: GET 2023-09-22T21:26:48.044 DEBUG:boto:Path: /ffzdqu-55/?prefix=h 2023-09-22T21:26:48.044 DEBUG:boto:Data: 2023-09-22T21:26:48.044 DEBUG:boto:Headers: {} 2023-09-22T21:26:48.044 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:48.044 DEBUG:boto:Port: 8001 2023-09-22T21:26:48.044 DEBUG:boto:Params: {} 2023-09-22T21:26:48.045 DEBUG:boto:Token: None 2023-09-22T21:26:48.045 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:48 GMT /ffzdqu-55/ 2023-09-22T21:26:48.045 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:O2/3jJFUEJ1EmhRkHXC5T1ieqlQ= 2023-09-22T21:26:48.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:O2/3jJFUEJ1EmhRkHXC5T1ieqlQ=', 'Content-Length': '0'} 2023-09-22T21:26:48.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095b7d97c5b71f89a-00650e0698-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:48.094 DEBUG:boto:b'ffzdqu-55h1000falseh2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:48.095 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:48.095 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:48.095 DEBUG:boto:path=/ffzdqu-55/?prefix=h 2023-09-22T21:26:48.095 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=h 2023-09-22T21:26:48.095 DEBUG:boto:Method: GET 2023-09-22T21:26:48.095 DEBUG:boto:Path: /ffzdqu-55/?prefix=h 2023-09-22T21:26:48.095 DEBUG:boto:Data: 2023-09-22T21:26:48.095 DEBUG:boto:Headers: {} 2023-09-22T21:26:48.096 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:48.096 DEBUG:boto:Port: 8000 2023-09-22T21:26:48.096 DEBUG:boto:Params: {} 2023-09-22T21:26:48.096 DEBUG:boto:Token: None 2023-09-22T21:26:48.096 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:48 GMT /ffzdqu-55/ 2023-09-22T21:26:48.096 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:O2/3jJFUEJ1EmhRkHXC5T1ieqlQ= 2023-09-22T21:26:48.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:O2/3jJFUEJ1EmhRkHXC5T1ieqlQ=', 'Content-Length': '0'} 2023-09-22T21:26:48.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035bb657eed1c0e1f-00650e0698-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:48.100 DEBUG:boto:b'ffzdqu-55h1000falseh2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:48.101 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:26:48.101 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:26:48.101 DEBUG:boto:Method: HEAD 2023-09-22T21:26:48.101 DEBUG:boto:Path: /ffzdqu-55/h 2023-09-22T21:26:48.101 DEBUG:boto:Data: 2023-09-22T21:26:48.101 DEBUG:boto:Headers: {} 2023-09-22T21:26:48.102 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:48.102 DEBUG:boto:Port: 8001 2023-09-22T21:26:48.102 DEBUG:boto:Params: {} 2023-09-22T21:26:48.102 DEBUG:boto:Token: None 2023-09-22T21:26:48.102 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:48 GMT /ffzdqu-55/h 2023-09-22T21:26:48.102 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t8rx4mG0AcqQpOw+n0w82/hdS5c= 2023-09-22T21:26:48.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t8rx4mG0AcqQpOw+n0w82/hdS5c=', 'Content-Length': '0'} 2023-09-22T21:26:48.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000075215811ebb8372-00650e0698-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:48.105 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:26:48.105 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:26:48.105 DEBUG:boto:Method: HEAD 2023-09-22T21:26:48.105 DEBUG:boto:Path: /ffzdqu-55/h 2023-09-22T21:26:48.106 DEBUG:boto:Data: 2023-09-22T21:26:48.106 DEBUG:boto:Headers: {} 2023-09-22T21:26:48.106 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:48.106 DEBUG:boto:Port: 8000 2023-09-22T21:26:48.106 DEBUG:boto:Params: {} 2023-09-22T21:26:48.106 DEBUG:boto:Token: None 2023-09-22T21:26:48.106 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:48 GMT /ffzdqu-55/h 2023-09-22T21:26:48.107 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t8rx4mG0AcqQpOw+n0w82/hdS5c= 2023-09-22T21:26:48.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t8rx4mG0AcqQpOw+n0w82/hdS5c=', 'Content-Length': '0'} 2023-09-22T21:26:48.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000747339149f85bd1d-00650e0698-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:48.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-22T21:26:48.110 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:26:48.110 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:26:48.110 DEBUG:boto:Method: GET 2023-09-22T21:26:48.110 DEBUG:boto:Path: /ffzdqu-55/h 2023-09-22T21:26:48.110 DEBUG:boto:Data: 2023-09-22T21:26:48.110 DEBUG:boto:Headers: {} 2023-09-22T21:26:48.111 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:48.111 DEBUG:boto:Port: 8001 2023-09-22T21:26:48.111 DEBUG:boto:Params: {} 2023-09-22T21:26:48.111 DEBUG:boto:Token: None 2023-09-22T21:26:48.111 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:48 GMT /ffzdqu-55/h 2023-09-22T21:26:48.111 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CtOTUKE1RcW2pmbZlULS/0cN6qk= 2023-09-22T21:26:48.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CtOTUKE1RcW2pmbZlULS/0cN6qk=', 'Content-Length': '0'} 2023-09-22T21:26:48.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000976d78444a32382b-00650e0698-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:48.154 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:26:48.154 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:26:48.155 DEBUG:boto:Method: GET 2023-09-22T21:26:48.155 DEBUG:boto:Path: /ffzdqu-55/h 2023-09-22T21:26:48.155 DEBUG:boto:Data: 2023-09-22T21:26:48.155 DEBUG:boto:Headers: {} 2023-09-22T21:26:48.155 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:48.155 DEBUG:boto:Port: 8000 2023-09-22T21:26:48.155 DEBUG:boto:Params: {} 2023-09-22T21:26:48.155 DEBUG:boto:Token: None 2023-09-22T21:26:48.155 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:48 GMT /ffzdqu-55/h 2023-09-22T21:26:48.156 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CtOTUKE1RcW2pmbZlULS/0cN6qk= 2023-09-22T21:26:48.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CtOTUKE1RcW2pmbZlULS/0cN6qk=', 'Content-Length': '0'} 2023-09-22T21:26:48.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c51d069154c37396-00650e0698-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:48.159 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:48.159 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:48.159 DEBUG:boto:path=/ffzdqu-55/?prefix=h 2023-09-22T21:26:48.159 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=h 2023-09-22T21:26:48.159 DEBUG:boto:Method: GET 2023-09-22T21:26:48.159 DEBUG:boto:Path: /ffzdqu-55/?prefix=h 2023-09-22T21:26:48.159 DEBUG:boto:Data: 2023-09-22T21:26:48.160 DEBUG:boto:Headers: {} 2023-09-22T21:26:48.160 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:26:48.160 DEBUG:boto:Port: 8001 2023-09-22T21:26:48.160 DEBUG:boto:Params: {} 2023-09-22T21:26:48.160 DEBUG:boto:Token: None 2023-09-22T21:26:48.160 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:48 GMT /ffzdqu-55/ 2023-09-22T21:26:48.160 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:O2/3jJFUEJ1EmhRkHXC5T1ieqlQ= 2023-09-22T21:26:48.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:O2/3jJFUEJ1EmhRkHXC5T1ieqlQ=', 'Content-Length': '0'} 2023-09-22T21:26:48.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b304cca73442c59-00650e0698-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:48.166 DEBUG:boto:b'ffzdqu-55h1000falseh2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:48.167 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:48.167 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:48.167 DEBUG:boto:path=/ffzdqu-55/?prefix=h 2023-09-22T21:26:48.167 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=h 2023-09-22T21:26:48.167 DEBUG:boto:Method: GET 2023-09-22T21:26:48.167 DEBUG:boto:Path: /ffzdqu-55/?prefix=h 2023-09-22T21:26:48.167 DEBUG:boto:Data: 2023-09-22T21:26:48.167 DEBUG:boto:Headers: {} 2023-09-22T21:26:48.168 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:48.168 DEBUG:boto:Port: 8000 2023-09-22T21:26:48.168 DEBUG:boto:Params: {} 2023-09-22T21:26:48.168 DEBUG:boto:Token: None 2023-09-22T21:26:48.168 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:26:48 GMT /ffzdqu-55/ 2023-09-22T21:26:48.168 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:O2/3jJFUEJ1EmhRkHXC5T1ieqlQ= 2023-09-22T21:26:48.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:O2/3jJFUEJ1EmhRkHXC5T1ieqlQ=', 'Content-Length': '0'} 2023-09-22T21:26:48.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006211897a526cd819-00650e0698-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:48.214 DEBUG:boto:b'ffzdqu-55h1000falseh2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:26:48.215 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-55 zones={a2, a1} 2023-09-22T21:26:48.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:26:48.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:26:48.215 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:26:48.254 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:26:48.254 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:26:48.309 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:48.309+0000 7ff3dfa85640 -1 req 4909970032439138699 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-24:f132bdb9-20a7-4073-a932-8f99da20691e.4382.24,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-24:f132bdb9-20a7-4073-a932-8f99da20691e.4382.24,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:26:48.313 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:48.313+0000 7ff3bca3f640 -1 req 660588480449426823 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-22:f132bdb9-20a7-4073-a932-8f99da20691e.4382.22,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-22:f132bdb9-20a7-4073-a932-8f99da20691e.4382.22,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:26:48.317 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:48.317+0000 7ff3a0a07640 -1 req 6595571375585534454 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-42:f132bdb9-20a7-4073-a932-8f99da20691e.5729.7,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-42:f132bdb9-20a7-4073-a932-8f99da20691e.5729.7,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:26:48.318 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:48.317+0000 7ff3b1a29640 -1 req 10701292581907308272 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-2:f132bdb9-20a7-4073-a932-8f99da20691e.4382.2,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-2:f132bdb9-20a7-4073-a932-8f99da20691e.4382.2,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:26:48.324 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:48.325+0000 7ff3919e9640 -1 req 16561821449500795260 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-20:f132bdb9-20a7-4073-a932-8f99da20691e.4382.20,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-20:f132bdb9-20a7-4073-a932-8f99da20691e.4382.20,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:26:48.357 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:48.357+0000 7ff3821ca640 -1 req 3882416955881144790 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-27:f132bdb9-20a7-4073-a932-8f99da20691e.4382.27,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-27:f132bdb9-20a7-4073-a932-8f99da20691e.4382.27,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:26:48.413 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:48.413+0000 7ff37c9bf640 -1 req 17738198957476340216 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-21:f132bdb9-20a7-4073-a932-8f99da20691e.4382.21,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-21:f132bdb9-20a7-4073-a932-8f99da20691e.4382.21,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:26:48.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'ffzdqu-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:26:48.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'ffzdqu-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:26:48.499 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket ffzdqu-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:26:48.580 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:26:48.580 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:26:48.634 INFO:teuthology.orchestra.run.smithi160.stdout:{"layout":{"resharding":"None","current_index":{"gen":2,"layout":{"type":"Normal","normal":{"num_shards":3,"hash_type":"Mod"}}},"logs":[{"gen":1,"layout":{"type":"InIndex","in_index":{"gen":1,"layout":{"num_shards":3,"hash_type":"Mod"}}}},{"gen":2,"layout":{"type":"InIndex","in_index":{"gen":2,"layout":{"num_shards":3,"hash_type":"Mod"}}}}]}} 2023-09-22T21:26:48.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:26:48.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:26:48.635 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:26:48.708 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:26:48.708 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:26:48.762 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:48.761+0000 7ff3881d6640 -1 req 10748955096029285252 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-39:f132bdb9-20a7-4073-a932-8f99da20691e.5729.4,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-39:f132bdb9-20a7-4073-a932-8f99da20691e.5729.4,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:26:48.762 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:48.761+0000 7ff461388640 -1 req 11172615132188315618 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-40:f132bdb9-20a7-4073-a932-8f99da20691e.5729.5,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-40:f132bdb9-20a7-4073-a932-8f99da20691e.5729.5,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:26:48.763 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:48.761+0000 7ff45d380640 -1 req 4684746605941055443 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-1:f132bdb9-20a7-4073-a932-8f99da20691e.4382.1,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-1:f132bdb9-20a7-4073-a932-8f99da20691e.4382.1,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:26:48.767 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:48.769+0000 7ff4703a6640 -1 req 11851194383583860451 0.003999928s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-28:f132bdb9-20a7-4073-a932-8f99da20691e.4382.28,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-28:f132bdb9-20a7-4073-a932-8f99da20691e.4382.28,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:26:48.813 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:48.813+0000 7ff43ab3b640 -1 req 4410927999431671917 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-3:f132bdb9-20a7-4073-a932-8f99da20691e.4382.3,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-3:f132bdb9-20a7-4073-a932-8f99da20691e.4382.3,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:26:48.856 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:48.857+0000 7ff43d340640 -1 req 7599547768376231901 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-4:f132bdb9-20a7-4073-a932-8f99da20691e.4382.4,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-4:f132bdb9-20a7-4073-a932-8f99da20691e.4382.4,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:26:48.857 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:48.857+0000 7ff43cb3f640 -1 req 5422852469641964144 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-29:f132bdb9-20a7-4073-a932-8f99da20691e.4382.29,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-29:f132bdb9-20a7-4073-a932-8f99da20691e.4382.29,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:26:48.870 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:48.869+0000 7ff4172f4640 -1 req 377960382548722541 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-5:f132bdb9-20a7-4073-a932-8f99da20691e.4382.5,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-5:f132bdb9-20a7-4073-a932-8f99da20691e.4382.5,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:26:48.952 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'ffzdqu-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:26:48.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'ffzdqu-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:26:48.952 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket ffzdqu-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:26:49.033 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:26:49.033 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:26:49.089 INFO:teuthology.orchestra.run.smithi160.stdout:{"layout":{"resharding":"None","current_index":{"gen":2,"layout":{"type":"Normal","normal":{"num_shards":3,"hash_type":"Mod"}}},"logs":[{"gen":2,"layout":{"type":"InIndex","in_index":{"gen":2,"layout":{"num_shards":3,"hash_type":"Mod"}}}}]}} 2023-09-22T21:26:49.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:26:49.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:26:49.090 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:26:49.161 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:26:49.162 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:26:49.210 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:49.208+0000 7ff4122ea640 -1 req 3709216382112606741 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-23:f132bdb9-20a7-4073-a932-8f99da20691e.4382.23,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-23:f132bdb9-20a7-4073-a932-8f99da20691e.4382.23,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:26:49.221 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:49.220+0000 7ff410ae7640 -1 req 3038452443216117256 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-30:f132bdb9-20a7-4073-a932-8f99da20691e.4382.30,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-30:f132bdb9-20a7-4073-a932-8f99da20691e.4382.30,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:26:49.221 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:49.220+0000 7ff4042ce640 -1 req 6283868354949899856 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-26:f132bdb9-20a7-4073-a932-8f99da20691e.4382.26,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-26:f132bdb9-20a7-4073-a932-8f99da20691e.4382.26,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:26:49.259 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:49.260+0000 7ff40cadf640 -1 req 15718420619197206137 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-38:f132bdb9-20a7-4073-a932-8f99da20691e.5729.3,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-38:f132bdb9-20a7-4073-a932-8f99da20691e.5729.3,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:26:49.306 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:49.304+0000 7ff3fcabf640 -1 req 1944918884125991566 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-36:f132bdb9-20a7-4073-a932-8f99da20691e.5729.1,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-36:f132bdb9-20a7-4073-a932-8f99da20691e.5729.1,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:26:49.327 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:26:49.328+0000 7ff3d5a71640 -1 req 2033980875475932551 0.003999928s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ffzdqu-43:f132bdb9-20a7-4073-a932-8f99da20691e.5729.8,z=f132bdb9-20a7-4073-a932-8f99da20691e,az=0},d={b=ffzdqu-43:f132bdb9-20a7-4073-a932-8f99da20691e.5729.8,z=2b721fee-a149-45da-925b-502d5a3d6853,az=0}} returned ret=-2 2023-09-22T21:26:49.422 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:49.422 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:49.422 DEBUG:boto:Method: HEAD 2023-09-22T21:26:49.422 DEBUG:boto:Path: /ffzdqu-55/ 2023-09-22T21:26:49.423 DEBUG:boto:Data: 2023-09-22T21:26:49.423 DEBUG:boto:Headers: {} 2023-09-22T21:26:49.423 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:49.423 DEBUG:boto:Port: 8000 2023-09-22T21:26:49.423 DEBUG:boto:Params: {} 2023-09-22T21:26:49.423 DEBUG:boto:Token: None 2023-09-22T21:26:49.423 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:49 GMT /ffzdqu-55/ 2023-09-22T21:26:49.423 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/claKuA1EN53TNanx6UsgZGcVDM= 2023-09-22T21:26:49.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/claKuA1EN53TNanx6UsgZGcVDM=', 'Content-Length': '0'} 2023-09-22T21:26:49.427 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d98b069e869674f2-00650e0699-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:49.428 DEBUG:boto:path=/ffzdqu-55/i 2023-09-22T21:26:49.428 DEBUG:boto:auth_path=/ffzdqu-55/i 2023-09-22T21:26:49.428 DEBUG:boto:Method: PUT 2023-09-22T21:26:49.428 DEBUG:boto:Path: /ffzdqu-55/i 2023-09-22T21:26:49.428 DEBUG:boto:Data: 2023-09-22T21:26:49.428 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:26:49.428 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:49.428 DEBUG:boto:Port: 8000 2023-09-22T21:26:49.429 DEBUG:boto:Params: {} 2023-09-22T21:26:49.429 DEBUG:boto:Token: None 2023-09-22T21:26:49.429 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:26:49 GMT /ffzdqu-55/i 2023-09-22T21:26:49.429 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Q22KW2WNbBtIVGGqwH31mqDXu4g= 2023-09-22T21:26:49.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:26:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Q22KW2WNbBtIVGGqwH31mqDXu4g='} 2023-09-22T21:26:49.439 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000086ba681555272ace-00650e0699-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:49.439 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:49.439 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:49.439 DEBUG:boto:Method: HEAD 2023-09-22T21:26:49.439 DEBUG:boto:Path: /ffzdqu-55/ 2023-09-22T21:26:49.439 DEBUG:boto:Data: 2023-09-22T21:26:49.439 DEBUG:boto:Headers: {} 2023-09-22T21:26:49.440 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:49.440 DEBUG:boto:Port: 8000 2023-09-22T21:26:49.440 DEBUG:boto:Params: {} 2023-09-22T21:26:49.440 DEBUG:boto:Token: None 2023-09-22T21:26:49.440 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:49 GMT /ffzdqu-55/ 2023-09-22T21:26:49.440 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/claKuA1EN53TNanx6UsgZGcVDM= 2023-09-22T21:26:49.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/claKuA1EN53TNanx6UsgZGcVDM=', 'Content-Length': '0'} 2023-09-22T21:26:49.443 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088e4f4afbb66e130-00650e0699-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:49.444 DEBUG:boto:path=/ffzdqu-55/j 2023-09-22T21:26:49.444 DEBUG:boto:auth_path=/ffzdqu-55/j 2023-09-22T21:26:49.444 DEBUG:boto:Method: PUT 2023-09-22T21:26:49.444 DEBUG:boto:Path: /ffzdqu-55/j 2023-09-22T21:26:49.444 DEBUG:boto:Data: 2023-09-22T21:26:49.444 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:26:49.444 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:49.444 DEBUG:boto:Port: 8000 2023-09-22T21:26:49.445 DEBUG:boto:Params: {} 2023-09-22T21:26:49.445 DEBUG:boto:Token: None 2023-09-22T21:26:49.445 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:26:49 GMT /ffzdqu-55/j 2023-09-22T21:26:49.445 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+YSYZiWvxHTpGt+F9yBxAggsjck= 2023-09-22T21:26:49.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:26:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+YSYZiWvxHTpGt+F9yBxAggsjck='} 2023-09-22T21:26:49.491 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000421412f2df80dbf8-00650e0699-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:49.491 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:49.491 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:49.491 DEBUG:boto:Method: HEAD 2023-09-22T21:26:49.491 DEBUG:boto:Path: /ffzdqu-55/ 2023-09-22T21:26:49.491 DEBUG:boto:Data: 2023-09-22T21:26:49.491 DEBUG:boto:Headers: {} 2023-09-22T21:26:49.491 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:49.492 DEBUG:boto:Port: 8000 2023-09-22T21:26:49.492 DEBUG:boto:Params: {} 2023-09-22T21:26:49.492 DEBUG:boto:Token: None 2023-09-22T21:26:49.492 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:49 GMT /ffzdqu-55/ 2023-09-22T21:26:49.492 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/claKuA1EN53TNanx6UsgZGcVDM= 2023-09-22T21:26:49.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/claKuA1EN53TNanx6UsgZGcVDM=', 'Content-Length': '0'} 2023-09-22T21:26:49.495 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000911abf72a1b54efa-00650e0699-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:49.496 DEBUG:boto:path=/ffzdqu-55/k 2023-09-22T21:26:49.496 DEBUG:boto:auth_path=/ffzdqu-55/k 2023-09-22T21:26:49.496 DEBUG:boto:Method: PUT 2023-09-22T21:26:49.496 DEBUG:boto:Path: /ffzdqu-55/k 2023-09-22T21:26:49.496 DEBUG:boto:Data: 2023-09-22T21:26:49.496 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:26:49.496 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:49.496 DEBUG:boto:Port: 8000 2023-09-22T21:26:49.496 DEBUG:boto:Params: {} 2023-09-22T21:26:49.497 DEBUG:boto:Token: None 2023-09-22T21:26:49.497 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:26:49 GMT /ffzdqu-55/k 2023-09-22T21:26:49.497 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JCsD54gpFfC4FATQK4vpx1zHsvI= 2023-09-22T21:26:49.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:26:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JCsD54gpFfC4FATQK4vpx1zHsvI='} 2023-09-22T21:26:49.508 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002837b6d5a2aac6c3-00650e0699-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:49.508 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:26:49.509 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:26:49.509 DEBUG:boto:Method: HEAD 2023-09-22T21:26:49.509 DEBUG:boto:Path: /ffzdqu-55/ 2023-09-22T21:26:49.509 DEBUG:boto:Data: 2023-09-22T21:26:49.509 DEBUG:boto:Headers: {} 2023-09-22T21:26:49.509 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:49.509 DEBUG:boto:Port: 8000 2023-09-22T21:26:49.509 DEBUG:boto:Params: {} 2023-09-22T21:26:49.509 DEBUG:boto:Token: None 2023-09-22T21:26:49.510 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:26:49 GMT /ffzdqu-55/ 2023-09-22T21:26:49.510 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/claKuA1EN53TNanx6UsgZGcVDM= 2023-09-22T21:26:49.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:26:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/claKuA1EN53TNanx6UsgZGcVDM=', 'Content-Length': '0'} 2023-09-22T21:26:49.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee7b4e15089d17d3-00650e0699-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:49.513 DEBUG:boto:path=/ffzdqu-55/l 2023-09-22T21:26:49.514 DEBUG:boto:auth_path=/ffzdqu-55/l 2023-09-22T21:26:49.514 DEBUG:boto:Method: PUT 2023-09-22T21:26:49.514 DEBUG:boto:Path: /ffzdqu-55/l 2023-09-22T21:26:49.514 DEBUG:boto:Data: 2023-09-22T21:26:49.514 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:26:49.514 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:26:49.514 DEBUG:boto:Port: 8000 2023-09-22T21:26:49.514 DEBUG:boto:Params: {} 2023-09-22T21:26:49.515 DEBUG:boto:Token: None 2023-09-22T21:26:49.515 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:26:49 GMT /ffzdqu-55/l 2023-09-22T21:26:49.515 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RscDJTvwfTU9FvZPGN94M6gE/5U= 2023-09-22T21:26:49.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:26:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RscDJTvwfTU9FvZPGN94M6gE/5U='} 2023-09-22T21:26:49.558 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000677dcb8649841921-00650e0699-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:26:49.559 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-55 2023-09-22T21:26:49.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:26:49.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:26:49.559 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:26:49.609 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:26:49.609 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:26:49.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:26:49.652+0000 7f7372493a40 1 waiting for incremental sync to catch up: 2023-09-22T21:26:49.652 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , ] 2023-09-22T21:26:49.652 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.35.6, 00000000002.35.6, 00000000003.37.6] 2023-09-22T21:26:54.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:26:54.652+0000 7f7372493a40 1 waiting for incremental sync to catch up: 2023-09-22T21:26:54.654 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , ] 2023-09-22T21:26:54.654 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.35.6, 00000000002.35.6, 00000000003.37.6] 2023-09-22T21:26:59.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:26:59.656+0000 7f7372493a40 1 waiting for incremental sync to catch up: 2023-09-22T21:26:59.655 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , ] 2023-09-22T21:26:59.655 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.35.6, 00000000002.35.6, 00000000003.37.6] 2023-09-22T21:27:04.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:27:04.656+0000 7f7372493a40 1 waiting for incremental sync to catch up: 2023-09-22T21:27:04.657 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.35.6, ] 2023-09-22T21:27:04.657 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.35.6, 00000000002.35.6, 00000000003.37.6] 2023-09-22T21:27:09.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:27:09.656+0000 7f7372493a40 1 waiting for incremental sync to catch up: 2023-09-22T21:27:09.658 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.35.6, 00000000002.35.6, ] 2023-09-22T21:27:09.658 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.35.6, 00000000002.35.6, 00000000003.37.6] 2023-09-22T21:27:14.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:27:14.660+0000 7f7372493a40 1 waiting for incremental sync to catch up: 2023-09-22T21:27:14.660 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.35.6, 00000000002.35.6, ] 2023-09-22T21:27:14.660 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.35.6, 00000000002.35.6, 00000000003.37.6] 2023-09-22T21:27:19.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:27:19.660+0000 7f7372493a40 1 waiting for incremental sync to catch up: 2023-09-22T21:27:19.661 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.35.6, 00000000002.35.6, ] 2023-09-22T21:27:19.661 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.35.6, 00000000002.35.6, 00000000003.37.6] 2023-09-22T21:27:24.662 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:27:24.660+0000 7f7372493a40 1 waiting for incremental sync to catch up: 2023-09-22T21:27:24.662 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.35.6, 00000000002.35.6, ] 2023-09-22T21:27:24.662 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.35.6, 00000000002.35.6, 00000000003.37.6] 2023-09-22T21:27:29.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:27:29.664+0000 7f7372493a40 1 waiting for incremental sync to catch up: 2023-09-22T21:27:29.664 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.35.6, 00000000002.35.6, ] 2023-09-22T21:27:29.664 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.35.6, 00000000002.35.6, 00000000003.37.6] 2023-09-22T21:27:34.665 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:27:34.664+0000 7f7372493a40 1 waiting for incremental sync to catch up: 2023-09-22T21:27:34.665 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.35.6, 00000000002.35.6, ] 2023-09-22T21:27:34.665 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.35.6, 00000000002.35.6, 00000000003.37.6] 2023-09-22T21:27:39.666 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:27:39.664+0000 7f7372493a40 1 waiting for incremental sync to catch up: 2023-09-22T21:27:39.666 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.35.6, 00000000002.35.6, ] 2023-09-22T21:27:39.666 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.35.6, 00000000002.35.6, 00000000003.37.6] 2023-09-22T21:27:44.667 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:27:44.667+0000 7f7372493a40 1 waiting for incremental sync to catch up: 2023-09-22T21:27:44.668 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.35.6, 00000000002.35.6, ] 2023-09-22T21:27:44.668 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.35.6, 00000000002.35.6, 00000000003.37.6] 2023-09-22T21:27:49.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:27:49.667+0000 7f7372493a40 1 waiting for incremental sync to catch up: 2023-09-22T21:27:49.669 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.35.6, 00000000002.35.6, ] 2023-09-22T21:27:49.670 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.35.6, 00000000002.35.6, 00000000003.37.6] 2023-09-22T21:27:54.670 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:27:54.667+0000 7f7372493a40 1 waiting for incremental sync to catch up: 2023-09-22T21:27:54.671 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.35.6, 00000000002.35.6, ] 2023-09-22T21:27:54.671 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.35.6, 00000000002.35.6, 00000000003.37.6] 2023-09-22T21:27:59.672 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:27:59.671+0000 7f7372493a40 1 waiting for incremental sync to catch up: 2023-09-22T21:27:59.672 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.35.6, 00000000002.35.6, ] 2023-09-22T21:27:59.672 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.35.6, 00000000002.35.6, 00000000003.37.6] 2023-09-22T21:28:04.673 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:28:04.671+0000 7f7372493a40 1 waiting for incremental sync to catch up: 2023-09-22T21:28:04.674 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.35.6, 00000000002.35.6, ] 2023-09-22T21:28:04.674 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.35.6, 00000000002.35.6, 00000000003.37.6] 2023-09-22T21:28:09.674 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:28:09.671+0000 7f7372493a40 1 bucket sync caught up with source: 2023-09-22T21:28:09.779 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.35.6, 00000000002.35.6, 00000000003.37.6] 2023-09-22T21:28:09.780 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.35.6, 00000000002.35.6, 00000000003.37.6] 2023-09-22T21:28:09.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:28:09.671+0000 7f7372493a40 0 bucket checkpoint complete 2023-09-22T21:28:09.781 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-55 2023-09-22T21:28:09.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:28:09.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:28:09.782 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:28:09.834 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:28:09.860 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:28:09.879 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:28:09.875+0000 7f90118aca40 1 waiting for incremental sync to catch up: 2023-09-22T21:28:09.879 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.282.6, 00000000002.278.6, , , 00000000002.242.6, 00000000001.197.6, 00000000001.211.6, , 00000000002.167.6, , 00000000002.165.6] 2023-09-22T21:28:09.879 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.282.6, 00000000002.278.6, 00000000001.177.6, , 00000000002.242.6, 00000000001.197.6, 00000000001.211.6, , 00000000002.167.6, , 00000000002.165.6] 2023-09-22T21:28:14.881 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:28:14.879+0000 7f90118aca40 1 waiting for incremental sync to catch up: 2023-09-22T21:28:14.882 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.282.6, 00000000002.278.6, , , 00000000002.242.6, 00000000001.197.6, 00000000001.211.6, , 00000000002.167.6, , 00000000002.165.6] 2023-09-22T21:28:14.882 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.282.6, 00000000002.278.6, 00000000001.177.6, , 00000000002.242.6, 00000000001.197.6, 00000000001.211.6, , 00000000002.167.6, , 00000000002.165.6] 2023-09-22T21:28:19.884 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:28:19.883+0000 7f90118aca40 1 waiting for incremental sync to catch up: 2023-09-22T21:28:19.884 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.282.6, 00000000002.278.6, , , 00000000002.242.6, 00000000001.197.6, 00000000001.211.6, , 00000000002.167.6, , 00000000002.165.6] 2023-09-22T21:28:19.884 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.282.6, 00000000002.278.6, 00000000001.177.6, , 00000000002.242.6, 00000000001.197.6, 00000000001.211.6, , 00000000002.167.6, , 00000000002.165.6] 2023-09-22T21:28:24.886 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:28:24.883+0000 7f90118aca40 1 bucket sync caught up with source: 2023-09-22T21:28:24.886 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.282.6, 00000000002.278.6, 00000000001.177.6, , 00000000002.242.6, 00000000001.197.6, 00000000001.211.6, , 00000000002.167.6, , 00000000002.165.6] 2023-09-22T21:28:24.886 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.282.6, 00000000002.278.6, 00000000001.177.6, , 00000000002.242.6, 00000000001.197.6, 00000000001.211.6, , 00000000002.167.6, , 00000000002.165.6] 2023-09-22T21:28:24.886 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:28:24.883+0000 7f90118aca40 0 bucket checkpoint complete 2023-09-22T21:28:24.919 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-55 zones={a2, a1} 2023-09-22T21:28:24.919 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:24.919 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:24.919 DEBUG:boto:Method: HEAD 2023-09-22T21:28:24.919 DEBUG:boto:Path: /ffzdqu-55/ 2023-09-22T21:28:24.919 DEBUG:boto:Data: 2023-09-22T21:28:24.919 DEBUG:boto:Headers: {} 2023-09-22T21:28:24.920 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:24.920 DEBUG:boto:Port: 8001 2023-09-22T21:28:24.920 DEBUG:boto:Params: {} 2023-09-22T21:28:24.920 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:28:24.920 DEBUG:boto:Token: None 2023-09-22T21:28:24.920 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:24 GMT /ffzdqu-55/ 2023-09-22T21:28:24.921 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gfSBNBGb+/7c4FSJ4dZTbUZYUIc= 2023-09-22T21:28:24.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gfSBNBGb+/7c4FSJ4dZTbUZYUIc=', 'Content-Length': '0'} 2023-09-22T21:28:24.926 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2d6d66421d1e989-00650e06f8-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:24.926 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:24.926 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:24.926 DEBUG:boto:Method: HEAD 2023-09-22T21:28:24.926 DEBUG:boto:Path: /ffzdqu-55/ 2023-09-22T21:28:24.926 DEBUG:boto:Data: 2023-09-22T21:28:24.927 DEBUG:boto:Headers: {} 2023-09-22T21:28:24.927 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:24.927 DEBUG:boto:Port: 8000 2023-09-22T21:28:24.927 DEBUG:boto:Params: {} 2023-09-22T21:28:24.927 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:28:24.927 DEBUG:boto:Token: None 2023-09-22T21:28:24.927 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:24 GMT /ffzdqu-55/ 2023-09-22T21:28:24.927 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gfSBNBGb+/7c4FSJ4dZTbUZYUIc= 2023-09-22T21:28:24.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gfSBNBGb+/7c4FSJ4dZTbUZYUIc=', 'Content-Length': '0'} 2023-09-22T21:28:24.931 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082d626cd67c47000-00650e06f8-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:24.931 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:28:24.932 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:24.932 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:24.932 DEBUG:boto:path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:28:24.932 DEBUG:boto:auth_path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:28:24.932 DEBUG:boto:Method: GET 2023-09-22T21:28:24.932 DEBUG:boto:Path: /ffzdqu-55/?versions&max-keys=999 2023-09-22T21:28:24.932 DEBUG:boto:Data: 2023-09-22T21:28:24.932 DEBUG:boto:Headers: {} 2023-09-22T21:28:24.932 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:24.932 DEBUG:boto:Port: 8001 2023-09-22T21:28:24.933 DEBUG:boto:Params: {} 2023-09-22T21:28:24.933 DEBUG:boto:Token: None 2023-09-22T21:28:24.933 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:24 GMT /ffzdqu-55/?versions 2023-09-22T21:28:24.933 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J6aa/ErLpf2vUiWJx2pUlhiv9Ic= 2023-09-22T21:28:24.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J6aa/ErLpf2vUiWJx2pUlhiv9Ic=', 'Content-Length': '0'} 2023-09-22T21:28:24.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000840fb5da369b7204-00650e06f8-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:24.939 DEBUG:boto:b'ffzdqu-55999falseanulltrue2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalinulltrue2023-09-22T21:26:49.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaljnulltrue2023-09-22T21:26:49.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalknulltrue2023-09-22T21:26:49.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormallnulltrue2023-09-22T21:26:49.519Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:24.940 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-22T21:28:24.940 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-22T21:28:24.940 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-22T21:28:24.940 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-22T21:28:24.940 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-09-22T21:28:24.940 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-09-22T21:28:24.940 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-09-22T21:28:24.940 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-09-22T21:28:24.940 DEBUG:tasks.rgw_multi.zone_rados:o=i 2023-09-22T21:28:24.940 DEBUG:tasks.rgw_multi.zone_rados:o=j 2023-09-22T21:28:24.941 DEBUG:tasks.rgw_multi.zone_rados:o=k 2023-09-22T21:28:24.941 DEBUG:tasks.rgw_multi.zone_rados:o=l 2023-09-22T21:28:24.941 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:28:24.941 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:24.941 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:24.941 DEBUG:boto:path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:28:24.941 DEBUG:boto:auth_path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:28:24.941 DEBUG:boto:Method: GET 2023-09-22T21:28:24.941 DEBUG:boto:Path: /ffzdqu-55/?versions&max-keys=999 2023-09-22T21:28:24.941 DEBUG:boto:Data: 2023-09-22T21:28:24.942 DEBUG:boto:Headers: {} 2023-09-22T21:28:24.942 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:24.942 DEBUG:boto:Port: 8000 2023-09-22T21:28:24.942 DEBUG:boto:Params: {} 2023-09-22T21:28:24.942 DEBUG:boto:Token: None 2023-09-22T21:28:24.942 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:24 GMT /ffzdqu-55/?versions 2023-09-22T21:28:24.942 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J6aa/ErLpf2vUiWJx2pUlhiv9Ic= 2023-09-22T21:28:24.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J6aa/ErLpf2vUiWJx2pUlhiv9Ic=', 'Content-Length': '0'} 2023-09-22T21:28:24.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dd78f524ea7fbca-00650e06f8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:24.945 DEBUG:boto:b'ffzdqu-55999falseanulltrue2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalinulltrue2023-09-22T21:26:49.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaljnulltrue2023-09-22T21:26:49.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalknulltrue2023-09-22T21:26:49.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormallnulltrue2023-09-22T21:26:49.519Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:24.946 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-22T21:28:24.946 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-22T21:28:24.946 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-22T21:28:24.946 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-22T21:28:24.946 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-09-22T21:28:24.946 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-09-22T21:28:24.947 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-09-22T21:28:24.947 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-09-22T21:28:24.947 DEBUG:tasks.rgw_multi.zone_rados:o=i 2023-09-22T21:28:24.947 DEBUG:tasks.rgw_multi.zone_rados:o=j 2023-09-22T21:28:24.947 DEBUG:tasks.rgw_multi.zone_rados:o=k 2023-09-22T21:28:24.947 DEBUG:tasks.rgw_multi.zone_rados:o=l 2023-09-22T21:28:24.947 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:24.947 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:24.947 DEBUG:boto:path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:28:24.947 DEBUG:boto:auth_path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:28:24.948 DEBUG:boto:Method: GET 2023-09-22T21:28:24.948 DEBUG:boto:Path: /ffzdqu-55/?versions&max-keys=999 2023-09-22T21:28:24.948 DEBUG:boto:Data: 2023-09-22T21:28:24.948 DEBUG:boto:Headers: {} 2023-09-22T21:28:24.948 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:24.948 DEBUG:boto:Port: 8001 2023-09-22T21:28:24.948 DEBUG:boto:Params: {} 2023-09-22T21:28:24.948 DEBUG:boto:Token: None 2023-09-22T21:28:24.948 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:24 GMT /ffzdqu-55/?versions 2023-09-22T21:28:24.948 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J6aa/ErLpf2vUiWJx2pUlhiv9Ic= 2023-09-22T21:28:24.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J6aa/ErLpf2vUiWJx2pUlhiv9Ic=', 'Content-Length': '0'} 2023-09-22T21:28:24.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc0a5827261c7ffd-00650e06f8-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:24.954 DEBUG:boto:b'ffzdqu-55999falseanulltrue2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalinulltrue2023-09-22T21:26:49.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaljnulltrue2023-09-22T21:26:49.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalknulltrue2023-09-22T21:26:49.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormallnulltrue2023-09-22T21:26:49.519Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:24.955 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:24.955 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:24.955 DEBUG:boto:path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:28:24.955 DEBUG:boto:auth_path=/ffzdqu-55/?versions&max-keys=999 2023-09-22T21:28:24.955 DEBUG:boto:Method: GET 2023-09-22T21:28:24.955 DEBUG:boto:Path: /ffzdqu-55/?versions&max-keys=999 2023-09-22T21:28:24.955 DEBUG:boto:Data: 2023-09-22T21:28:24.955 DEBUG:boto:Headers: {} 2023-09-22T21:28:24.956 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:24.956 DEBUG:boto:Port: 8000 2023-09-22T21:28:24.956 DEBUG:boto:Params: {} 2023-09-22T21:28:24.956 DEBUG:boto:Token: None 2023-09-22T21:28:24.956 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:24 GMT /ffzdqu-55/?versions 2023-09-22T21:28:24.956 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J6aa/ErLpf2vUiWJx2pUlhiv9Ic= 2023-09-22T21:28:24.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J6aa/ErLpf2vUiWJx2pUlhiv9Ic=', 'Content-Length': '0'} 2023-09-22T21:28:24.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009abeafdf4f209f60-00650e06f8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:24.959 DEBUG:boto:b'ffzdqu-55999falseanulltrue2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalinulltrue2023-09-22T21:26:49.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaljnulltrue2023-09-22T21:26:49.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalknulltrue2023-09-22T21:26:49.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormallnulltrue2023-09-22T21:26:49.519Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:24.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:28:24.960 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:28:24.960 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:28:24.960 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:28:24.961 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:28:24.961 DEBUG:boto:Method: GET 2023-09-22T21:28:24.961 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:28:24.961 DEBUG:boto:Data: 2023-09-22T21:28:24.961 DEBUG:boto:Headers: {} 2023-09-22T21:28:24.961 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:24.961 DEBUG:boto:Port: 8001 2023-09-22T21:28:24.961 DEBUG:boto:Params: {} 2023-09-22T21:28:24.961 DEBUG:boto:Token: None 2023-09-22T21:28:24.962 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:24 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:28:24.962 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5T6hQMnTVeRrJ98gVtOaHVmK6Qw= 2023-09-22T21:28:24.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5T6hQMnTVeRrJ98gVtOaHVmK6Qw=', 'Content-Length': '0'} 2023-09-22T21:28:24.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000761bff7777d3c06f-00650e06f8-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:24.964 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:28:24.964 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:28:24.964 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:28:24.964 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:28:24.964 DEBUG:boto:Method: GET 2023-09-22T21:28:24.964 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:28:24.965 DEBUG:boto:Data: 2023-09-22T21:28:24.965 DEBUG:boto:Headers: {} 2023-09-22T21:28:24.965 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:24.965 DEBUG:boto:Port: 8000 2023-09-22T21:28:24.965 DEBUG:boto:Params: {} 2023-09-22T21:28:24.965 DEBUG:boto:Token: None 2023-09-22T21:28:24.965 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:24 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:28:24.965 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5T6hQMnTVeRrJ98gVtOaHVmK6Qw= 2023-09-22T21:28:24.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5T6hQMnTVeRrJ98gVtOaHVmK6Qw=', 'Content-Length': '0'} 2023-09-22T21:28:24.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001232474e7d061c81-00650e06f8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:24.968 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:28:24.968 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:28:24.968 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:28:24.969 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:28:24.969 DEBUG:boto:Method: HEAD 2023-09-22T21:28:24.969 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:28:24.969 DEBUG:boto:Data: 2023-09-22T21:28:24.969 DEBUG:boto:Headers: {} 2023-09-22T21:28:24.969 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:24.969 DEBUG:boto:Port: 8001 2023-09-22T21:28:24.969 DEBUG:boto:Params: {} 2023-09-22T21:28:24.969 DEBUG:boto:Token: None 2023-09-22T21:28:24.969 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:24 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:28:24.970 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dZxzsA4Ut0fmcOxzS2jcXBtLXh4= 2023-09-22T21:28:24.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dZxzsA4Ut0fmcOxzS2jcXBtLXh4=', 'Content-Length': '0'} 2023-09-22T21:28:24.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5d151850b3c5d50-00650e06f8-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:24.972 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:28:24.973 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:28:24.973 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:28:24.973 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:28:24.973 DEBUG:boto:Method: HEAD 2023-09-22T21:28:24.973 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:28:24.973 DEBUG:boto:Data: 2023-09-22T21:28:24.973 DEBUG:boto:Headers: {} 2023-09-22T21:28:24.973 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:24.973 DEBUG:boto:Port: 8000 2023-09-22T21:28:24.973 DEBUG:boto:Params: {} 2023-09-22T21:28:24.973 DEBUG:boto:Token: None 2023-09-22T21:28:24.974 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:24 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:28:24.974 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dZxzsA4Ut0fmcOxzS2jcXBtLXh4= 2023-09-22T21:28:24.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dZxzsA4Ut0fmcOxzS2jcXBtLXh4=', 'Content-Length': '0'} 2023-09-22T21:28:24.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb12d2c2354e4765-00650e06f8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:24.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:28:24.976 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:28:24.976 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:28:24.976 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:28:24.976 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:28:24.977 DEBUG:boto:Method: GET 2023-09-22T21:28:24.977 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:28:24.977 DEBUG:boto:Data: 2023-09-22T21:28:24.977 DEBUG:boto:Headers: {} 2023-09-22T21:28:24.977 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:24.977 DEBUG:boto:Port: 8001 2023-09-22T21:28:24.977 DEBUG:boto:Params: {} 2023-09-22T21:28:24.978 DEBUG:boto:Token: None 2023-09-22T21:28:24.978 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:24 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:28:24.978 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5T6hQMnTVeRrJ98gVtOaHVmK6Qw= 2023-09-22T21:28:24.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5T6hQMnTVeRrJ98gVtOaHVmK6Qw=', 'Content-Length': '0'} 2023-09-22T21:28:24.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000490882da776cf42b-00650e06f8-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.022 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:28:25.022 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:28:25.023 DEBUG:boto:path=/ffzdqu-55/a?versionId=null 2023-09-22T21:28:25.023 DEBUG:boto:auth_path=/ffzdqu-55/a?versionId=null 2023-09-22T21:28:25.023 DEBUG:boto:Method: GET 2023-09-22T21:28:25.023 DEBUG:boto:Path: /ffzdqu-55/a?versionId=null 2023-09-22T21:28:25.023 DEBUG:boto:Data: 2023-09-22T21:28:25.023 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.023 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.023 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.023 DEBUG:boto:Params: {} 2023-09-22T21:28:25.023 DEBUG:boto:Token: None 2023-09-22T21:28:25.024 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/a?versionId=null 2023-09-22T21:28:25.024 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FyMfOkY6dY8J1djEjrQbHudz3yE= 2023-09-22T21:28:25.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FyMfOkY6dY8J1djEjrQbHudz3yE=', 'Content-Length': '0'} 2023-09-22T21:28:25.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae5fb9e0e3cc9e17-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.027 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:25.027 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:25.027 DEBUG:boto:path=/ffzdqu-55/?prefix=a 2023-09-22T21:28:25.027 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=a 2023-09-22T21:28:25.027 DEBUG:boto:Method: GET 2023-09-22T21:28:25.027 DEBUG:boto:Path: /ffzdqu-55/?prefix=a 2023-09-22T21:28:25.028 DEBUG:boto:Data: 2023-09-22T21:28:25.028 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.028 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.028 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.028 DEBUG:boto:Params: {} 2023-09-22T21:28:25.028 DEBUG:boto:Token: None 2023-09-22T21:28:25.028 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/ 2023-09-22T21:28:25.028 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg= 2023-09-22T21:28:25.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg=', 'Content-Length': '0'} 2023-09-22T21:28:25.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001aee7c07d2684f96-00650e06f9-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.074 DEBUG:boto:b'ffzdqu-55a1000falsea2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:25.075 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:25.075 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:25.075 DEBUG:boto:path=/ffzdqu-55/?prefix=a 2023-09-22T21:28:25.075 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=a 2023-09-22T21:28:25.075 DEBUG:boto:Method: GET 2023-09-22T21:28:25.075 DEBUG:boto:Path: /ffzdqu-55/?prefix=a 2023-09-22T21:28:25.075 DEBUG:boto:Data: 2023-09-22T21:28:25.075 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.075 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.075 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.076 DEBUG:boto:Params: {} 2023-09-22T21:28:25.076 DEBUG:boto:Token: None 2023-09-22T21:28:25.076 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/ 2023-09-22T21:28:25.076 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg= 2023-09-22T21:28:25.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg=', 'Content-Length': '0'} 2023-09-22T21:28:25.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec0f3d2588b5a19c-00650e06f9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.080 DEBUG:boto:b'ffzdqu-55a1000falsea2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:25.080 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:28:25.080 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:28:25.081 DEBUG:boto:Method: HEAD 2023-09-22T21:28:25.081 DEBUG:boto:Path: /ffzdqu-55/a 2023-09-22T21:28:25.081 DEBUG:boto:Data: 2023-09-22T21:28:25.081 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.081 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.081 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.081 DEBUG:boto:Params: {} 2023-09-22T21:28:25.081 DEBUG:boto:Token: None 2023-09-22T21:28:25.081 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/a 2023-09-22T21:28:25.081 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0g1sB8gi8o/Os24FOfeev7hfOpk= 2023-09-22T21:28:25.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0g1sB8gi8o/Os24FOfeev7hfOpk=', 'Content-Length': '0'} 2023-09-22T21:28:25.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec229b5af7843da7-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.084 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:28:25.084 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:28:25.085 DEBUG:boto:Method: HEAD 2023-09-22T21:28:25.085 DEBUG:boto:Path: /ffzdqu-55/a 2023-09-22T21:28:25.085 DEBUG:boto:Data: 2023-09-22T21:28:25.085 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.085 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.085 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.085 DEBUG:boto:Params: {} 2023-09-22T21:28:25.085 DEBUG:boto:Token: None 2023-09-22T21:28:25.085 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/a 2023-09-22T21:28:25.086 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0g1sB8gi8o/Os24FOfeev7hfOpk= 2023-09-22T21:28:25.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0g1sB8gi8o/Os24FOfeev7hfOpk=', 'Content-Length': '0'} 2023-09-22T21:28:25.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f04b501bffe752c-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:28:25.089 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:28:25.089 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:28:25.089 DEBUG:boto:Method: GET 2023-09-22T21:28:25.089 DEBUG:boto:Path: /ffzdqu-55/a 2023-09-22T21:28:25.089 DEBUG:boto:Data: 2023-09-22T21:28:25.089 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.089 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.089 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.090 DEBUG:boto:Params: {} 2023-09-22T21:28:25.090 DEBUG:boto:Token: None 2023-09-22T21:28:25.090 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/a 2023-09-22T21:28:25.090 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5UBhXJOUWakSBBI7HFcm8QV0Hhk= 2023-09-22T21:28:25.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5UBhXJOUWakSBBI7HFcm8QV0Hhk=', 'Content-Length': '0'} 2023-09-22T21:28:25.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003191014c0794fa1b-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.134 DEBUG:boto:path=/ffzdqu-55/a 2023-09-22T21:28:25.134 DEBUG:boto:auth_path=/ffzdqu-55/a 2023-09-22T21:28:25.134 DEBUG:boto:Method: GET 2023-09-22T21:28:25.135 DEBUG:boto:Path: /ffzdqu-55/a 2023-09-22T21:28:25.135 DEBUG:boto:Data: 2023-09-22T21:28:25.135 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.135 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.135 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.135 DEBUG:boto:Params: {} 2023-09-22T21:28:25.135 DEBUG:boto:Token: None 2023-09-22T21:28:25.135 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/a 2023-09-22T21:28:25.135 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5UBhXJOUWakSBBI7HFcm8QV0Hhk= 2023-09-22T21:28:25.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5UBhXJOUWakSBBI7HFcm8QV0Hhk=', 'Content-Length': '0'} 2023-09-22T21:28:25.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000491c3b0b146103c4-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.138 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:25.139 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:25.139 DEBUG:boto:path=/ffzdqu-55/?prefix=a 2023-09-22T21:28:25.139 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=a 2023-09-22T21:28:25.139 DEBUG:boto:Method: GET 2023-09-22T21:28:25.139 DEBUG:boto:Path: /ffzdqu-55/?prefix=a 2023-09-22T21:28:25.139 DEBUG:boto:Data: 2023-09-22T21:28:25.139 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.139 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.139 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.139 DEBUG:boto:Params: {} 2023-09-22T21:28:25.141 DEBUG:boto:Token: None 2023-09-22T21:28:25.141 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/ 2023-09-22T21:28:25.142 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg= 2023-09-22T21:28:25.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg=', 'Content-Length': '0'} 2023-09-22T21:28:25.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006548e44993e549e1-00650e06f9-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.190 DEBUG:boto:b'ffzdqu-55a1000falsea2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:25.191 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:25.191 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:25.191 DEBUG:boto:path=/ffzdqu-55/?prefix=a 2023-09-22T21:28:25.191 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=a 2023-09-22T21:28:25.191 DEBUG:boto:Method: GET 2023-09-22T21:28:25.191 DEBUG:boto:Path: /ffzdqu-55/?prefix=a 2023-09-22T21:28:25.191 DEBUG:boto:Data: 2023-09-22T21:28:25.191 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.192 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.192 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.192 DEBUG:boto:Params: {} 2023-09-22T21:28:25.192 DEBUG:boto:Token: None 2023-09-22T21:28:25.192 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/ 2023-09-22T21:28:25.192 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg= 2023-09-22T21:28:25.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg=', 'Content-Length': '0'} 2023-09-22T21:28:25.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008df117ebe6f10b5b-00650e06f9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.196 DEBUG:boto:b'ffzdqu-55a1000falsea2023-09-22T21:24:46.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:25.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:28:25.197 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:28:25.197 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:28:25.197 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.197 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.197 DEBUG:boto:Method: GET 2023-09-22T21:28:25.197 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.197 DEBUG:boto:Data: 2023-09-22T21:28:25.197 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.197 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.197 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.197 DEBUG:boto:Params: {} 2023-09-22T21:28:25.198 DEBUG:boto:Token: None 2023-09-22T21:28:25.198 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.198 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tqc0TwHEft6ThldNp5IHuZtQy20= 2023-09-22T21:28:25.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tqc0TwHEft6ThldNp5IHuZtQy20=', 'Content-Length': '0'} 2023-09-22T21:28:25.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db72a8591b4de7f5-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.201 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:28:25.201 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:28:25.201 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.201 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.201 DEBUG:boto:Method: GET 2023-09-22T21:28:25.201 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.201 DEBUG:boto:Data: 2023-09-22T21:28:25.201 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.201 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.202 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.202 DEBUG:boto:Params: {} 2023-09-22T21:28:25.202 DEBUG:boto:Token: None 2023-09-22T21:28:25.202 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.202 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tqc0TwHEft6ThldNp5IHuZtQy20= 2023-09-22T21:28:25.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tqc0TwHEft6ThldNp5IHuZtQy20=', 'Content-Length': '0'} 2023-09-22T21:28:25.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000267a327ba72a408-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.205 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:28:25.205 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:28:25.205 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.206 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.206 DEBUG:boto:Method: HEAD 2023-09-22T21:28:25.206 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.206 DEBUG:boto:Data: 2023-09-22T21:28:25.206 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.206 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.206 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.206 DEBUG:boto:Params: {} 2023-09-22T21:28:25.206 DEBUG:boto:Token: None 2023-09-22T21:28:25.206 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.207 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EYm23/HRYV+Vr0sYRThpdt1miEg= 2023-09-22T21:28:25.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EYm23/HRYV+Vr0sYRThpdt1miEg=', 'Content-Length': '0'} 2023-09-22T21:28:25.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9cea6bfb513e367-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.210 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:28:25.210 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:28:25.210 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.210 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.210 DEBUG:boto:Method: HEAD 2023-09-22T21:28:25.210 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.210 DEBUG:boto:Data: 2023-09-22T21:28:25.210 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.210 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.211 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.211 DEBUG:boto:Params: {} 2023-09-22T21:28:25.211 DEBUG:boto:Token: None 2023-09-22T21:28:25.211 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.211 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EYm23/HRYV+Vr0sYRThpdt1miEg= 2023-09-22T21:28:25.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EYm23/HRYV+Vr0sYRThpdt1miEg=', 'Content-Length': '0'} 2023-09-22T21:28:25.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b5a1d9b4534e838-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:28:25.214 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:28:25.214 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:28:25.214 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.214 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.214 DEBUG:boto:Method: GET 2023-09-22T21:28:25.214 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.215 DEBUG:boto:Data: 2023-09-22T21:28:25.215 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.215 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.215 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.215 DEBUG:boto:Params: {} 2023-09-22T21:28:25.215 DEBUG:boto:Token: None 2023-09-22T21:28:25.215 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.215 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tqc0TwHEft6ThldNp5IHuZtQy20= 2023-09-22T21:28:25.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tqc0TwHEft6ThldNp5IHuZtQy20=', 'Content-Length': '0'} 2023-09-22T21:28:25.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe5c4864bfbc733d-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.262 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:28:25.262 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:28:25.263 DEBUG:boto:path=/ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.263 DEBUG:boto:auth_path=/ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.263 DEBUG:boto:Method: GET 2023-09-22T21:28:25.263 DEBUG:boto:Path: /ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.263 DEBUG:boto:Data: 2023-09-22T21:28:25.263 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.263 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.263 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.263 DEBUG:boto:Params: {} 2023-09-22T21:28:25.263 DEBUG:boto:Token: None 2023-09-22T21:28:25.264 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/b?versionId=null 2023-09-22T21:28:25.264 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tqc0TwHEft6ThldNp5IHuZtQy20= 2023-09-22T21:28:25.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tqc0TwHEft6ThldNp5IHuZtQy20=', 'Content-Length': '0'} 2023-09-22T21:28:25.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048d0c367e0580559-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.267 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:25.267 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:25.267 DEBUG:boto:path=/ffzdqu-55/?prefix=b 2023-09-22T21:28:25.267 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=b 2023-09-22T21:28:25.268 DEBUG:boto:Method: GET 2023-09-22T21:28:25.268 DEBUG:boto:Path: /ffzdqu-55/?prefix=b 2023-09-22T21:28:25.268 DEBUG:boto:Data: 2023-09-22T21:28:25.268 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.268 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.268 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.268 DEBUG:boto:Params: {} 2023-09-22T21:28:25.268 DEBUG:boto:Token: None 2023-09-22T21:28:25.268 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/ 2023-09-22T21:28:25.268 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg= 2023-09-22T21:28:25.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg=', 'Content-Length': '0'} 2023-09-22T21:28:25.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000086736afbed1e77f-00650e06f9-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.274 DEBUG:boto:b'ffzdqu-55b1000falseb2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:25.275 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:25.275 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:25.275 DEBUG:boto:path=/ffzdqu-55/?prefix=b 2023-09-22T21:28:25.275 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=b 2023-09-22T21:28:25.275 DEBUG:boto:Method: GET 2023-09-22T21:28:25.275 DEBUG:boto:Path: /ffzdqu-55/?prefix=b 2023-09-22T21:28:25.275 DEBUG:boto:Data: 2023-09-22T21:28:25.275 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.275 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.275 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.276 DEBUG:boto:Params: {} 2023-09-22T21:28:25.276 DEBUG:boto:Token: None 2023-09-22T21:28:25.276 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/ 2023-09-22T21:28:25.276 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg= 2023-09-22T21:28:25.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg=', 'Content-Length': '0'} 2023-09-22T21:28:25.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000309b89d9a480da86-00650e06f9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.322 DEBUG:boto:b'ffzdqu-55b1000falseb2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:25.323 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:28:25.323 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:28:25.323 DEBUG:boto:Method: HEAD 2023-09-22T21:28:25.323 DEBUG:boto:Path: /ffzdqu-55/b 2023-09-22T21:28:25.323 DEBUG:boto:Data: 2023-09-22T21:28:25.323 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.323 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.323 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.323 DEBUG:boto:Params: {} 2023-09-22T21:28:25.324 DEBUG:boto:Token: None 2023-09-22T21:28:25.324 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/b 2023-09-22T21:28:25.324 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JtLQFTXRVEG5LcWLO1tpvMh6JgU= 2023-09-22T21:28:25.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JtLQFTXRVEG5LcWLO1tpvMh6JgU=', 'Content-Length': '0'} 2023-09-22T21:28:25.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007acddeb8d87cd2af-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.327 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:28:25.327 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:28:25.327 DEBUG:boto:Method: HEAD 2023-09-22T21:28:25.327 DEBUG:boto:Path: /ffzdqu-55/b 2023-09-22T21:28:25.327 DEBUG:boto:Data: 2023-09-22T21:28:25.327 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.327 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.327 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.327 DEBUG:boto:Params: {} 2023-09-22T21:28:25.328 DEBUG:boto:Token: None 2023-09-22T21:28:25.328 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/b 2023-09-22T21:28:25.328 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JtLQFTXRVEG5LcWLO1tpvMh6JgU= 2023-09-22T21:28:25.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JtLQFTXRVEG5LcWLO1tpvMh6JgU=', 'Content-Length': '0'} 2023-09-22T21:28:25.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b28c1a374df63bd-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:28:25.331 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:28:25.331 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:28:25.331 DEBUG:boto:Method: GET 2023-09-22T21:28:25.331 DEBUG:boto:Path: /ffzdqu-55/b 2023-09-22T21:28:25.331 DEBUG:boto:Data: 2023-09-22T21:28:25.331 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.331 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.331 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.331 DEBUG:boto:Params: {} 2023-09-22T21:28:25.332 DEBUG:boto:Token: None 2023-09-22T21:28:25.332 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/b 2023-09-22T21:28:25.332 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AynzNNdAEvd/FnWcY9mRv0Z/qq0= 2023-09-22T21:28:25.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AynzNNdAEvd/FnWcY9mRv0Z/qq0=', 'Content-Length': '0'} 2023-09-22T21:28:25.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f368cb42fc51d62d-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.378 DEBUG:boto:path=/ffzdqu-55/b 2023-09-22T21:28:25.378 DEBUG:boto:auth_path=/ffzdqu-55/b 2023-09-22T21:28:25.379 DEBUG:boto:Method: GET 2023-09-22T21:28:25.379 DEBUG:boto:Path: /ffzdqu-55/b 2023-09-22T21:28:25.379 DEBUG:boto:Data: 2023-09-22T21:28:25.379 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.379 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.379 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.379 DEBUG:boto:Params: {} 2023-09-22T21:28:25.379 DEBUG:boto:Token: None 2023-09-22T21:28:25.379 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/b 2023-09-22T21:28:25.379 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AynzNNdAEvd/FnWcY9mRv0Z/qq0= 2023-09-22T21:28:25.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AynzNNdAEvd/FnWcY9mRv0Z/qq0=', 'Content-Length': '0'} 2023-09-22T21:28:25.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000318535f97b20fcb6-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.382 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:25.383 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:25.383 DEBUG:boto:path=/ffzdqu-55/?prefix=b 2023-09-22T21:28:25.383 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=b 2023-09-22T21:28:25.383 DEBUG:boto:Method: GET 2023-09-22T21:28:25.383 DEBUG:boto:Path: /ffzdqu-55/?prefix=b 2023-09-22T21:28:25.383 DEBUG:boto:Data: 2023-09-22T21:28:25.383 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.383 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.383 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.383 DEBUG:boto:Params: {} 2023-09-22T21:28:25.384 DEBUG:boto:Token: None 2023-09-22T21:28:25.384 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/ 2023-09-22T21:28:25.384 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg= 2023-09-22T21:28:25.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg=', 'Content-Length': '0'} 2023-09-22T21:28:25.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a34b3eb2b4d82bc8-00650e06f9-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.390 DEBUG:boto:b'ffzdqu-55b1000falseb2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:25.390 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:25.391 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:25.391 DEBUG:boto:path=/ffzdqu-55/?prefix=b 2023-09-22T21:28:25.391 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=b 2023-09-22T21:28:25.391 DEBUG:boto:Method: GET 2023-09-22T21:28:25.391 DEBUG:boto:Path: /ffzdqu-55/?prefix=b 2023-09-22T21:28:25.391 DEBUG:boto:Data: 2023-09-22T21:28:25.391 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.391 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.391 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.391 DEBUG:boto:Params: {} 2023-09-22T21:28:25.392 DEBUG:boto:Token: None 2023-09-22T21:28:25.392 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/ 2023-09-22T21:28:25.392 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg= 2023-09-22T21:28:25.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg=', 'Content-Length': '0'} 2023-09-22T21:28:25.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b3192b3fa684aa3-00650e06f9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.396 DEBUG:boto:b'ffzdqu-55b1000falseb2023-09-22T21:24:46.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:25.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:28:25.396 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:28:25.396 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:28:25.396 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.396 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.396 DEBUG:boto:Method: GET 2023-09-22T21:28:25.397 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.397 DEBUG:boto:Data: 2023-09-22T21:28:25.397 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.397 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.397 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.397 DEBUG:boto:Params: {} 2023-09-22T21:28:25.397 DEBUG:boto:Token: None 2023-09-22T21:28:25.397 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.397 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZVTUZDqx9jfdr95RqTH7z0KPJRQ= 2023-09-22T21:28:25.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZVTUZDqx9jfdr95RqTH7z0KPJRQ=', 'Content-Length': '0'} 2023-09-22T21:28:25.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c752e26907d590e-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.400 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:28:25.401 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:28:25.401 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.401 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.401 DEBUG:boto:Method: GET 2023-09-22T21:28:25.401 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.401 DEBUG:boto:Data: 2023-09-22T21:28:25.401 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.401 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.401 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.401 DEBUG:boto:Params: {} 2023-09-22T21:28:25.402 DEBUG:boto:Token: None 2023-09-22T21:28:25.402 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.402 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZVTUZDqx9jfdr95RqTH7z0KPJRQ= 2023-09-22T21:28:25.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZVTUZDqx9jfdr95RqTH7z0KPJRQ=', 'Content-Length': '0'} 2023-09-22T21:28:25.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e568cbe9f0a76d17-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.446 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:28:25.446 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:28:25.447 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.447 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.447 DEBUG:boto:Method: HEAD 2023-09-22T21:28:25.447 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.447 DEBUG:boto:Data: 2023-09-22T21:28:25.447 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.447 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.447 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.447 DEBUG:boto:Params: {} 2023-09-22T21:28:25.447 DEBUG:boto:Token: None 2023-09-22T21:28:25.448 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.448 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NGmn40Fws2XPguM+a1efSrJQUXE= 2023-09-22T21:28:25.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NGmn40Fws2XPguM+a1efSrJQUXE=', 'Content-Length': '0'} 2023-09-22T21:28:25.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c59577cf99872ba3-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.451 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:28:25.451 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:28:25.451 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.451 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.451 DEBUG:boto:Method: HEAD 2023-09-22T21:28:25.451 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.451 DEBUG:boto:Data: 2023-09-22T21:28:25.451 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.451 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.452 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.452 DEBUG:boto:Params: {} 2023-09-22T21:28:25.452 DEBUG:boto:Token: None 2023-09-22T21:28:25.452 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.452 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NGmn40Fws2XPguM+a1efSrJQUXE= 2023-09-22T21:28:25.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NGmn40Fws2XPguM+a1efSrJQUXE=', 'Content-Length': '0'} 2023-09-22T21:28:25.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023e30e488ff1e290-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:28:25.455 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:28:25.455 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:28:25.455 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.455 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.455 DEBUG:boto:Method: GET 2023-09-22T21:28:25.455 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.455 DEBUG:boto:Data: 2023-09-22T21:28:25.456 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.456 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.456 DEBUG:boto:Params: {} 2023-09-22T21:28:25.456 DEBUG:boto:Token: None 2023-09-22T21:28:25.456 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.456 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZVTUZDqx9jfdr95RqTH7z0KPJRQ= 2023-09-22T21:28:25.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZVTUZDqx9jfdr95RqTH7z0KPJRQ=', 'Content-Length': '0'} 2023-09-22T21:28:25.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f9078aa6d847f96-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.502 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:28:25.502 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:28:25.503 DEBUG:boto:path=/ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.503 DEBUG:boto:auth_path=/ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.503 DEBUG:boto:Method: GET 2023-09-22T21:28:25.503 DEBUG:boto:Path: /ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.503 DEBUG:boto:Data: 2023-09-22T21:28:25.503 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.503 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.503 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.503 DEBUG:boto:Params: {} 2023-09-22T21:28:25.503 DEBUG:boto:Token: None 2023-09-22T21:28:25.504 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/c?versionId=null 2023-09-22T21:28:25.504 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZVTUZDqx9jfdr95RqTH7z0KPJRQ= 2023-09-22T21:28:25.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZVTUZDqx9jfdr95RqTH7z0KPJRQ=', 'Content-Length': '0'} 2023-09-22T21:28:25.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000390a7627d77d5d5e-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.507 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:25.507 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:25.507 DEBUG:boto:path=/ffzdqu-55/?prefix=c 2023-09-22T21:28:25.507 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=c 2023-09-22T21:28:25.507 DEBUG:boto:Method: GET 2023-09-22T21:28:25.507 DEBUG:boto:Path: /ffzdqu-55/?prefix=c 2023-09-22T21:28:25.507 DEBUG:boto:Data: 2023-09-22T21:28:25.508 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.508 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.508 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.508 DEBUG:boto:Params: {} 2023-09-22T21:28:25.508 DEBUG:boto:Token: None 2023-09-22T21:28:25.508 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/ 2023-09-22T21:28:25.508 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg= 2023-09-22T21:28:25.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg=', 'Content-Length': '0'} 2023-09-22T21:28:25.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053c8278b2418ebb6-00650e06f9-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.515 DEBUG:boto:b'ffzdqu-55c1000falsec2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:25.515 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:25.515 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:25.515 DEBUG:boto:path=/ffzdqu-55/?prefix=c 2023-09-22T21:28:25.515 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=c 2023-09-22T21:28:25.516 DEBUG:boto:Method: GET 2023-09-22T21:28:25.516 DEBUG:boto:Path: /ffzdqu-55/?prefix=c 2023-09-22T21:28:25.516 DEBUG:boto:Data: 2023-09-22T21:28:25.516 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.516 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.516 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.516 DEBUG:boto:Params: {} 2023-09-22T21:28:25.516 DEBUG:boto:Token: None 2023-09-22T21:28:25.516 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/ 2023-09-22T21:28:25.516 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg= 2023-09-22T21:28:25.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg=', 'Content-Length': '0'} 2023-09-22T21:28:25.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9270d2f610cdf85-00650e06f9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.520 DEBUG:boto:b'ffzdqu-55c1000falsec2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:25.521 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:28:25.521 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:28:25.521 DEBUG:boto:Method: HEAD 2023-09-22T21:28:25.521 DEBUG:boto:Path: /ffzdqu-55/c 2023-09-22T21:28:25.521 DEBUG:boto:Data: 2023-09-22T21:28:25.521 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.521 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.522 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.522 DEBUG:boto:Params: {} 2023-09-22T21:28:25.522 DEBUG:boto:Token: None 2023-09-22T21:28:25.522 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/c 2023-09-22T21:28:25.522 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0eMtOQqUI/hY7oB6d8Yui1odDg4= 2023-09-22T21:28:25.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0eMtOQqUI/hY7oB6d8Yui1odDg4=', 'Content-Length': '0'} 2023-09-22T21:28:25.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a91ed9072bfc0baf-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.525 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:28:25.525 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:28:25.525 DEBUG:boto:Method: HEAD 2023-09-22T21:28:25.525 DEBUG:boto:Path: /ffzdqu-55/c 2023-09-22T21:28:25.525 DEBUG:boto:Data: 2023-09-22T21:28:25.526 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.526 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.526 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.526 DEBUG:boto:Params: {} 2023-09-22T21:28:25.526 DEBUG:boto:Token: None 2023-09-22T21:28:25.526 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/c 2023-09-22T21:28:25.526 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0eMtOQqUI/hY7oB6d8Yui1odDg4= 2023-09-22T21:28:25.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0eMtOQqUI/hY7oB6d8Yui1odDg4=', 'Content-Length': '0'} 2023-09-22T21:28:25.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e102a2cee4167420-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:28:25.529 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:28:25.530 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:28:25.530 DEBUG:boto:Method: GET 2023-09-22T21:28:25.530 DEBUG:boto:Path: /ffzdqu-55/c 2023-09-22T21:28:25.530 DEBUG:boto:Data: 2023-09-22T21:28:25.530 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.530 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.530 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.530 DEBUG:boto:Params: {} 2023-09-22T21:28:25.530 DEBUG:boto:Token: None 2023-09-22T21:28:25.531 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/c 2023-09-22T21:28:25.531 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uX1PhUAm43QoEy+VI7O1BAhG0IA= 2023-09-22T21:28:25.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uX1PhUAm43QoEy+VI7O1BAhG0IA=', 'Content-Length': '0'} 2023-09-22T21:28:25.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fd17c01e50340dd-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.574 DEBUG:boto:path=/ffzdqu-55/c 2023-09-22T21:28:25.574 DEBUG:boto:auth_path=/ffzdqu-55/c 2023-09-22T21:28:25.575 DEBUG:boto:Method: GET 2023-09-22T21:28:25.575 DEBUG:boto:Path: /ffzdqu-55/c 2023-09-22T21:28:25.575 DEBUG:boto:Data: 2023-09-22T21:28:25.575 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.575 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.575 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.575 DEBUG:boto:Params: {} 2023-09-22T21:28:25.575 DEBUG:boto:Token: None 2023-09-22T21:28:25.575 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/c 2023-09-22T21:28:25.576 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uX1PhUAm43QoEy+VI7O1BAhG0IA= 2023-09-22T21:28:25.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uX1PhUAm43QoEy+VI7O1BAhG0IA=', 'Content-Length': '0'} 2023-09-22T21:28:25.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000558583958aeee808-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.579 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:25.579 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:25.579 DEBUG:boto:path=/ffzdqu-55/?prefix=c 2023-09-22T21:28:25.579 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=c 2023-09-22T21:28:25.579 DEBUG:boto:Method: GET 2023-09-22T21:28:25.579 DEBUG:boto:Path: /ffzdqu-55/?prefix=c 2023-09-22T21:28:25.579 DEBUG:boto:Data: 2023-09-22T21:28:25.579 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.580 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.580 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.580 DEBUG:boto:Params: {} 2023-09-22T21:28:25.580 DEBUG:boto:Token: None 2023-09-22T21:28:25.580 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/ 2023-09-22T21:28:25.580 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg= 2023-09-22T21:28:25.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg=', 'Content-Length': '0'} 2023-09-22T21:28:25.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014b0a312de82335d-00650e06f9-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.630 DEBUG:boto:b'ffzdqu-55c1000falsec2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:25.631 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:25.631 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:25.631 DEBUG:boto:path=/ffzdqu-55/?prefix=c 2023-09-22T21:28:25.631 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=c 2023-09-22T21:28:25.631 DEBUG:boto:Method: GET 2023-09-22T21:28:25.631 DEBUG:boto:Path: /ffzdqu-55/?prefix=c 2023-09-22T21:28:25.631 DEBUG:boto:Data: 2023-09-22T21:28:25.631 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.631 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.631 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.632 DEBUG:boto:Params: {} 2023-09-22T21:28:25.632 DEBUG:boto:Token: None 2023-09-22T21:28:25.632 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/ 2023-09-22T21:28:25.632 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg= 2023-09-22T21:28:25.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg=', 'Content-Length': '0'} 2023-09-22T21:28:25.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ccc324ab0c2f7cf-00650e06f9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.636 DEBUG:boto:b'ffzdqu-55c1000falsec2023-09-22T21:24:46.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:25.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:28:25.636 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:28:25.636 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:28:25.636 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.637 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.637 DEBUG:boto:Method: GET 2023-09-22T21:28:25.637 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.637 DEBUG:boto:Data: 2023-09-22T21:28:25.637 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.637 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.637 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.637 DEBUG:boto:Params: {} 2023-09-22T21:28:25.637 DEBUG:boto:Token: None 2023-09-22T21:28:25.637 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.638 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IKaFaLei821QWxPeHIlSIRv337w= 2023-09-22T21:28:25.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IKaFaLei821QWxPeHIlSIRv337w=', 'Content-Length': '0'} 2023-09-22T21:28:25.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053a45965c74265e0-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.641 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:28:25.641 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:28:25.641 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.641 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.641 DEBUG:boto:Method: GET 2023-09-22T21:28:25.641 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.641 DEBUG:boto:Data: 2023-09-22T21:28:25.642 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.642 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.642 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.642 DEBUG:boto:Params: {} 2023-09-22T21:28:25.642 DEBUG:boto:Token: None 2023-09-22T21:28:25.642 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.642 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IKaFaLei821QWxPeHIlSIRv337w= 2023-09-22T21:28:25.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IKaFaLei821QWxPeHIlSIRv337w=', 'Content-Length': '0'} 2023-09-22T21:28:25.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afcd57786520b3d9-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.645 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:28:25.646 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:28:25.646 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.646 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.646 DEBUG:boto:Method: HEAD 2023-09-22T21:28:25.646 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.646 DEBUG:boto:Data: 2023-09-22T21:28:25.646 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.646 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.646 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.646 DEBUG:boto:Params: {} 2023-09-22T21:28:25.647 DEBUG:boto:Token: None 2023-09-22T21:28:25.647 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.647 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bv9qfrlWxIbrriyLXAqBeBIJ0Zw= 2023-09-22T21:28:25.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bv9qfrlWxIbrriyLXAqBeBIJ0Zw=', 'Content-Length': '0'} 2023-09-22T21:28:25.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013aea5a20f3e1caa-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.650 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:28:25.650 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:28:25.650 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.650 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.650 DEBUG:boto:Method: HEAD 2023-09-22T21:28:25.650 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.650 DEBUG:boto:Data: 2023-09-22T21:28:25.650 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.650 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.651 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.651 DEBUG:boto:Params: {} 2023-09-22T21:28:25.651 DEBUG:boto:Token: None 2023-09-22T21:28:25.651 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.651 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bv9qfrlWxIbrriyLXAqBeBIJ0Zw= 2023-09-22T21:28:25.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bv9qfrlWxIbrriyLXAqBeBIJ0Zw=', 'Content-Length': '0'} 2023-09-22T21:28:25.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062283ab7f5ae8c77-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:28:25.654 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:28:25.654 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:28:25.654 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.654 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.654 DEBUG:boto:Method: GET 2023-09-22T21:28:25.654 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.654 DEBUG:boto:Data: 2023-09-22T21:28:25.655 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.655 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.655 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.655 DEBUG:boto:Params: {} 2023-09-22T21:28:25.655 DEBUG:boto:Token: None 2023-09-22T21:28:25.655 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.655 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IKaFaLei821QWxPeHIlSIRv337w= 2023-09-22T21:28:25.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IKaFaLei821QWxPeHIlSIRv337w=', 'Content-Length': '0'} 2023-09-22T21:28:25.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed514c10336ff22f-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.698 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:28:25.698 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:28:25.699 DEBUG:boto:path=/ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.699 DEBUG:boto:auth_path=/ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.699 DEBUG:boto:Method: GET 2023-09-22T21:28:25.699 DEBUG:boto:Path: /ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.699 DEBUG:boto:Data: 2023-09-22T21:28:25.699 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.699 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.699 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.699 DEBUG:boto:Params: {} 2023-09-22T21:28:25.699 DEBUG:boto:Token: None 2023-09-22T21:28:25.700 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/d?versionId=null 2023-09-22T21:28:25.700 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IKaFaLei821QWxPeHIlSIRv337w= 2023-09-22T21:28:25.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IKaFaLei821QWxPeHIlSIRv337w=', 'Content-Length': '0'} 2023-09-22T21:28:25.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e0d2969ebff971d-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.703 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:25.703 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:25.703 DEBUG:boto:path=/ffzdqu-55/?prefix=d 2023-09-22T21:28:25.703 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=d 2023-09-22T21:28:25.703 DEBUG:boto:Method: GET 2023-09-22T21:28:25.703 DEBUG:boto:Path: /ffzdqu-55/?prefix=d 2023-09-22T21:28:25.703 DEBUG:boto:Data: 2023-09-22T21:28:25.704 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.704 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.704 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.704 DEBUG:boto:Params: {} 2023-09-22T21:28:25.704 DEBUG:boto:Token: None 2023-09-22T21:28:25.704 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/ 2023-09-22T21:28:25.704 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg= 2023-09-22T21:28:25.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg=', 'Content-Length': '0'} 2023-09-22T21:28:25.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000059de03ce9620e65-00650e06f9-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.710 DEBUG:boto:b'ffzdqu-55d1000falsed2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:25.710 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:25.710 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:25.710 DEBUG:boto:path=/ffzdqu-55/?prefix=d 2023-09-22T21:28:25.710 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=d 2023-09-22T21:28:25.711 DEBUG:boto:Method: GET 2023-09-22T21:28:25.711 DEBUG:boto:Path: /ffzdqu-55/?prefix=d 2023-09-22T21:28:25.711 DEBUG:boto:Data: 2023-09-22T21:28:25.711 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.711 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.711 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.711 DEBUG:boto:Params: {} 2023-09-22T21:28:25.711 DEBUG:boto:Token: None 2023-09-22T21:28:25.711 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/ 2023-09-22T21:28:25.711 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg= 2023-09-22T21:28:25.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg=', 'Content-Length': '0'} 2023-09-22T21:28:25.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021b392aea1f857ce-00650e06f9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.715 DEBUG:boto:b'ffzdqu-55d1000falsed2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:25.715 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:28:25.715 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:28:25.716 DEBUG:boto:Method: HEAD 2023-09-22T21:28:25.716 DEBUG:boto:Path: /ffzdqu-55/d 2023-09-22T21:28:25.716 DEBUG:boto:Data: 2023-09-22T21:28:25.716 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.716 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.716 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.716 DEBUG:boto:Params: {} 2023-09-22T21:28:25.716 DEBUG:boto:Token: None 2023-09-22T21:28:25.716 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/d 2023-09-22T21:28:25.717 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FL/SPhe2Fva9OdLw4k21qNPsa30= 2023-09-22T21:28:25.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FL/SPhe2Fva9OdLw4k21qNPsa30=', 'Content-Length': '0'} 2023-09-22T21:28:25.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f28753815aee1fa-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.719 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:28:25.719 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:28:25.720 DEBUG:boto:Method: HEAD 2023-09-22T21:28:25.720 DEBUG:boto:Path: /ffzdqu-55/d 2023-09-22T21:28:25.720 DEBUG:boto:Data: 2023-09-22T21:28:25.720 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.720 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.720 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.720 DEBUG:boto:Params: {} 2023-09-22T21:28:25.720 DEBUG:boto:Token: None 2023-09-22T21:28:25.720 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/d 2023-09-22T21:28:25.720 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FL/SPhe2Fva9OdLw4k21qNPsa30= 2023-09-22T21:28:25.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FL/SPhe2Fva9OdLw4k21qNPsa30=', 'Content-Length': '0'} 2023-09-22T21:28:25.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d4067b4a9b1620a-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:28:25.724 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:28:25.724 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:28:25.724 DEBUG:boto:Method: GET 2023-09-22T21:28:25.724 DEBUG:boto:Path: /ffzdqu-55/d 2023-09-22T21:28:25.724 DEBUG:boto:Data: 2023-09-22T21:28:25.724 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.724 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.724 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.725 DEBUG:boto:Params: {} 2023-09-22T21:28:25.725 DEBUG:boto:Token: None 2023-09-22T21:28:25.725 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/d 2023-09-22T21:28:25.725 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Syzfy1sbHasIez9Dm4Vr4rnDg1c= 2023-09-22T21:28:25.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Syzfy1sbHasIez9Dm4Vr4rnDg1c=', 'Content-Length': '0'} 2023-09-22T21:28:25.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d08ccaf5d491f384-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.770 DEBUG:boto:path=/ffzdqu-55/d 2023-09-22T21:28:25.770 DEBUG:boto:auth_path=/ffzdqu-55/d 2023-09-22T21:28:25.771 DEBUG:boto:Method: GET 2023-09-22T21:28:25.771 DEBUG:boto:Path: /ffzdqu-55/d 2023-09-22T21:28:25.771 DEBUG:boto:Data: 2023-09-22T21:28:25.771 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.771 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.771 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.771 DEBUG:boto:Params: {} 2023-09-22T21:28:25.771 DEBUG:boto:Token: None 2023-09-22T21:28:25.771 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/d 2023-09-22T21:28:25.772 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Syzfy1sbHasIez9Dm4Vr4rnDg1c= 2023-09-22T21:28:25.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Syzfy1sbHasIez9Dm4Vr4rnDg1c=', 'Content-Length': '0'} 2023-09-22T21:28:25.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:24:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3350d0ff308c23a-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.775 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:25.775 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:25.775 DEBUG:boto:path=/ffzdqu-55/?prefix=d 2023-09-22T21:28:25.775 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=d 2023-09-22T21:28:25.775 DEBUG:boto:Method: GET 2023-09-22T21:28:25.775 DEBUG:boto:Path: /ffzdqu-55/?prefix=d 2023-09-22T21:28:25.775 DEBUG:boto:Data: 2023-09-22T21:28:25.775 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.776 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.776 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.776 DEBUG:boto:Params: {} 2023-09-22T21:28:25.776 DEBUG:boto:Token: None 2023-09-22T21:28:25.776 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/ 2023-09-22T21:28:25.776 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg= 2023-09-22T21:28:25.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg=', 'Content-Length': '0'} 2023-09-22T21:28:25.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000569a553d823c7bb4-00650e06f9-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.782 DEBUG:boto:b'ffzdqu-55d1000falsed2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:25.782 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:25.783 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:25.783 DEBUG:boto:path=/ffzdqu-55/?prefix=d 2023-09-22T21:28:25.783 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=d 2023-09-22T21:28:25.783 DEBUG:boto:Method: GET 2023-09-22T21:28:25.783 DEBUG:boto:Path: /ffzdqu-55/?prefix=d 2023-09-22T21:28:25.783 DEBUG:boto:Data: 2023-09-22T21:28:25.783 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.783 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.783 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.783 DEBUG:boto:Params: {} 2023-09-22T21:28:25.784 DEBUG:boto:Token: None 2023-09-22T21:28:25.784 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/ 2023-09-22T21:28:25.784 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg= 2023-09-22T21:28:25.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg=', 'Content-Length': '0'} 2023-09-22T21:28:25.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b907552c3cb72f58-00650e06f9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.789 DEBUG:boto:b'ffzdqu-55d1000falsed2023-09-22T21:24:46.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:25.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-22T21:28:25.789 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:28:25.789 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:28:25.789 DEBUG:boto:path=/ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.789 DEBUG:boto:auth_path=/ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.789 DEBUG:boto:Method: GET 2023-09-22T21:28:25.790 DEBUG:boto:Path: /ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.790 DEBUG:boto:Data: 2023-09-22T21:28:25.790 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.790 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.790 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.790 DEBUG:boto:Params: {} 2023-09-22T21:28:25.790 DEBUG:boto:Token: None 2023-09-22T21:28:25.790 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.790 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pQYpuPjY8kPN5713qMTGJQb/Qvs= 2023-09-22T21:28:25.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pQYpuPjY8kPN5713qMTGJQb/Qvs=', 'Content-Length': '0'} 2023-09-22T21:28:25.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e13d15c15dea3ce-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.795 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:28:25.795 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:28:25.796 DEBUG:boto:path=/ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.796 DEBUG:boto:auth_path=/ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.796 DEBUG:boto:Method: GET 2023-09-22T21:28:25.796 DEBUG:boto:Path: /ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.796 DEBUG:boto:Data: 2023-09-22T21:28:25.796 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.796 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.796 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.796 DEBUG:boto:Params: {} 2023-09-22T21:28:25.796 DEBUG:boto:Token: None 2023-09-22T21:28:25.797 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.797 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pQYpuPjY8kPN5713qMTGJQb/Qvs= 2023-09-22T21:28:25.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pQYpuPjY8kPN5713qMTGJQb/Qvs=', 'Content-Length': '0'} 2023-09-22T21:28:25.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f4e5a5284dfab4a-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.800 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:28:25.800 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:28:25.800 DEBUG:boto:path=/ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.800 DEBUG:boto:auth_path=/ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.801 DEBUG:boto:Method: HEAD 2023-09-22T21:28:25.801 DEBUG:boto:Path: /ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.801 DEBUG:boto:Data: 2023-09-22T21:28:25.801 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.801 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.801 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.801 DEBUG:boto:Params: {} 2023-09-22T21:28:25.801 DEBUG:boto:Token: None 2023-09-22T21:28:25.801 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.801 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AvotGvrVJ/Ow4DDDtrHdal5K4/Q= 2023-09-22T21:28:25.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AvotGvrVJ/Ow4DDDtrHdal5K4/Q=', 'Content-Length': '0'} 2023-09-22T21:28:25.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030f798c45abf6bd3-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.806 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:28:25.806 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:28:25.806 DEBUG:boto:path=/ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.806 DEBUG:boto:auth_path=/ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.807 DEBUG:boto:Method: HEAD 2023-09-22T21:28:25.807 DEBUG:boto:Path: /ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.807 DEBUG:boto:Data: 2023-09-22T21:28:25.807 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.807 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.807 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.807 DEBUG:boto:Params: {} 2023-09-22T21:28:25.807 DEBUG:boto:Token: None 2023-09-22T21:28:25.807 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.808 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AvotGvrVJ/Ow4DDDtrHdal5K4/Q= 2023-09-22T21:28:25.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AvotGvrVJ/Ow4DDDtrHdal5K4/Q=', 'Content-Length': '0'} 2023-09-22T21:28:25.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4129aa1d28502d9-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-22T21:28:25.811 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:28:25.811 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:28:25.811 DEBUG:boto:path=/ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.811 DEBUG:boto:auth_path=/ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.811 DEBUG:boto:Method: GET 2023-09-22T21:28:25.811 DEBUG:boto:Path: /ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.811 DEBUG:boto:Data: 2023-09-22T21:28:25.811 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.811 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.812 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.812 DEBUG:boto:Params: {} 2023-09-22T21:28:25.812 DEBUG:boto:Token: None 2023-09-22T21:28:25.812 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.812 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pQYpuPjY8kPN5713qMTGJQb/Qvs= 2023-09-22T21:28:25.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pQYpuPjY8kPN5713qMTGJQb/Qvs=', 'Content-Length': '0'} 2023-09-22T21:28:25.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004556368da8d77b1e-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.862 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:28:25.862 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:28:25.863 DEBUG:boto:path=/ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.863 DEBUG:boto:auth_path=/ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.863 DEBUG:boto:Method: GET 2023-09-22T21:28:25.863 DEBUG:boto:Path: /ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.863 DEBUG:boto:Data: 2023-09-22T21:28:25.863 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.863 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.863 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.863 DEBUG:boto:Params: {} 2023-09-22T21:28:25.864 DEBUG:boto:Token: None 2023-09-22T21:28:25.864 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/e?versionId=null 2023-09-22T21:28:25.864 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pQYpuPjY8kPN5713qMTGJQb/Qvs= 2023-09-22T21:28:25.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pQYpuPjY8kPN5713qMTGJQb/Qvs=', 'Content-Length': '0'} 2023-09-22T21:28:25.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f584046f867224b3-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.867 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:25.867 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:25.867 DEBUG:boto:path=/ffzdqu-55/?prefix=e 2023-09-22T21:28:25.867 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=e 2023-09-22T21:28:25.867 DEBUG:boto:Method: GET 2023-09-22T21:28:25.867 DEBUG:boto:Path: /ffzdqu-55/?prefix=e 2023-09-22T21:28:25.867 DEBUG:boto:Data: 2023-09-22T21:28:25.868 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.868 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.868 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.868 DEBUG:boto:Params: {} 2023-09-22T21:28:25.868 DEBUG:boto:Token: None 2023-09-22T21:28:25.868 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/ 2023-09-22T21:28:25.868 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg= 2023-09-22T21:28:25.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg=', 'Content-Length': '0'} 2023-09-22T21:28:25.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0e6555460425f1a-00650e06f9-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.874 DEBUG:boto:b'ffzdqu-55e1000falsee2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:25.874 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:25.875 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:25.875 DEBUG:boto:path=/ffzdqu-55/?prefix=e 2023-09-22T21:28:25.875 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=e 2023-09-22T21:28:25.875 DEBUG:boto:Method: GET 2023-09-22T21:28:25.875 DEBUG:boto:Path: /ffzdqu-55/?prefix=e 2023-09-22T21:28:25.875 DEBUG:boto:Data: 2023-09-22T21:28:25.875 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.875 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.875 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.875 DEBUG:boto:Params: {} 2023-09-22T21:28:25.876 DEBUG:boto:Token: None 2023-09-22T21:28:25.876 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/ 2023-09-22T21:28:25.876 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg= 2023-09-22T21:28:25.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg=', 'Content-Length': '0'} 2023-09-22T21:28:25.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f298b8bc46d6945-00650e06f9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.881 DEBUG:boto:b'ffzdqu-55e1000falsee2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:25.881 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:28:25.881 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:28:25.881 DEBUG:boto:Method: HEAD 2023-09-22T21:28:25.881 DEBUG:boto:Path: /ffzdqu-55/e 2023-09-22T21:28:25.881 DEBUG:boto:Data: 2023-09-22T21:28:25.881 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.881 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.881 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.882 DEBUG:boto:Params: {} 2023-09-22T21:28:25.882 DEBUG:boto:Token: None 2023-09-22T21:28:25.882 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/e 2023-09-22T21:28:25.882 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NHF4zLo1bCV6Fl0gtBCwKdeKBB4= 2023-09-22T21:28:25.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NHF4zLo1bCV6Fl0gtBCwKdeKBB4=', 'Content-Length': '0'} 2023-09-22T21:28:25.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0d9efd024b984b7-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.885 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:28:25.885 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:28:25.885 DEBUG:boto:Method: HEAD 2023-09-22T21:28:25.885 DEBUG:boto:Path: /ffzdqu-55/e 2023-09-22T21:28:25.885 DEBUG:boto:Data: 2023-09-22T21:28:25.886 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.886 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.886 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.886 DEBUG:boto:Params: {} 2023-09-22T21:28:25.886 DEBUG:boto:Token: None 2023-09-22T21:28:25.886 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/e 2023-09-22T21:28:25.886 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NHF4zLo1bCV6Fl0gtBCwKdeKBB4= 2023-09-22T21:28:25.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NHF4zLo1bCV6Fl0gtBCwKdeKBB4=', 'Content-Length': '0'} 2023-09-22T21:28:25.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad62eae4bccd9296-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-22T21:28:25.889 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:28:25.889 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:28:25.889 DEBUG:boto:Method: GET 2023-09-22T21:28:25.889 DEBUG:boto:Path: /ffzdqu-55/e 2023-09-22T21:28:25.889 DEBUG:boto:Data: 2023-09-22T21:28:25.890 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.890 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.890 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.890 DEBUG:boto:Params: {} 2023-09-22T21:28:25.890 DEBUG:boto:Token: None 2023-09-22T21:28:25.890 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/e 2023-09-22T21:28:25.890 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:98f5NlAq3g3jrgL7mq4K9x63GQo= 2023-09-22T21:28:25.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:98f5NlAq3g3jrgL7mq4K9x63GQo=', 'Content-Length': '0'} 2023-09-22T21:28:25.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2ddddff7cbf5323-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.934 DEBUG:boto:path=/ffzdqu-55/e 2023-09-22T21:28:25.934 DEBUG:boto:auth_path=/ffzdqu-55/e 2023-09-22T21:28:25.935 DEBUG:boto:Method: GET 2023-09-22T21:28:25.935 DEBUG:boto:Path: /ffzdqu-55/e 2023-09-22T21:28:25.935 DEBUG:boto:Data: 2023-09-22T21:28:25.935 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.935 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.935 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.935 DEBUG:boto:Params: {} 2023-09-22T21:28:25.935 DEBUG:boto:Token: None 2023-09-22T21:28:25.935 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/e 2023-09-22T21:28:25.936 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:98f5NlAq3g3jrgL7mq4K9x63GQo= 2023-09-22T21:28:25.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:98f5NlAq3g3jrgL7mq4K9x63GQo=', 'Content-Length': '0'} 2023-09-22T21:28:25.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039e1a2aa31770374-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.939 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:25.939 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:25.939 DEBUG:boto:path=/ffzdqu-55/?prefix=e 2023-09-22T21:28:25.939 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=e 2023-09-22T21:28:25.939 DEBUG:boto:Method: GET 2023-09-22T21:28:25.939 DEBUG:boto:Path: /ffzdqu-55/?prefix=e 2023-09-22T21:28:25.939 DEBUG:boto:Data: 2023-09-22T21:28:25.939 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.939 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.939 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.940 DEBUG:boto:Params: {} 2023-09-22T21:28:25.940 DEBUG:boto:Token: None 2023-09-22T21:28:25.940 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/ 2023-09-22T21:28:25.940 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg= 2023-09-22T21:28:25.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg=', 'Content-Length': '0'} 2023-09-22T21:28:25.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009de22e38e10af4f7-00650e06f9-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.986 DEBUG:boto:b'ffzdqu-55e1000falsee2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:25.987 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:25.987 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:25.987 DEBUG:boto:path=/ffzdqu-55/?prefix=e 2023-09-22T21:28:25.987 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=e 2023-09-22T21:28:25.987 DEBUG:boto:Method: GET 2023-09-22T21:28:25.987 DEBUG:boto:Path: /ffzdqu-55/?prefix=e 2023-09-22T21:28:25.987 DEBUG:boto:Data: 2023-09-22T21:28:25.987 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.988 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.988 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.988 DEBUG:boto:Params: {} 2023-09-22T21:28:25.988 DEBUG:boto:Token: None 2023-09-22T21:28:25.988 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/ 2023-09-22T21:28:25.988 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg= 2023-09-22T21:28:25.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C+BL+EecGUKybry1xsujauKCWbg=', 'Content-Length': '0'} 2023-09-22T21:28:25.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de1ec640745bc478-00650e06f9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.992 DEBUG:boto:b'ffzdqu-55e1000falsee2023-09-22T21:25:08.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:25.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-22T21:28:25.993 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:28:25.993 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:28:25.993 DEBUG:boto:path=/ffzdqu-55/f?versionId=null 2023-09-22T21:28:25.993 DEBUG:boto:auth_path=/ffzdqu-55/f?versionId=null 2023-09-22T21:28:25.993 DEBUG:boto:Method: GET 2023-09-22T21:28:25.993 DEBUG:boto:Path: /ffzdqu-55/f?versionId=null 2023-09-22T21:28:25.993 DEBUG:boto:Data: 2023-09-22T21:28:25.993 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.993 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:25.993 DEBUG:boto:Port: 8001 2023-09-22T21:28:25.994 DEBUG:boto:Params: {} 2023-09-22T21:28:25.994 DEBUG:boto:Token: None 2023-09-22T21:28:25.994 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/f?versionId=null 2023-09-22T21:28:25.994 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oovF+mRnOgUnSH7LZNbn0g2N6iU= 2023-09-22T21:28:25.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oovF+mRnOgUnSH7LZNbn0g2N6iU=', 'Content-Length': '0'} 2023-09-22T21:28:25.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014e0d16b9426e2a3-00650e06f9-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:25.997 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:28:25.997 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:28:25.997 DEBUG:boto:path=/ffzdqu-55/f?versionId=null 2023-09-22T21:28:25.998 DEBUG:boto:auth_path=/ffzdqu-55/f?versionId=null 2023-09-22T21:28:25.998 DEBUG:boto:Method: GET 2023-09-22T21:28:25.998 DEBUG:boto:Path: /ffzdqu-55/f?versionId=null 2023-09-22T21:28:25.998 DEBUG:boto:Data: 2023-09-22T21:28:25.998 DEBUG:boto:Headers: {} 2023-09-22T21:28:25.998 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:25.998 DEBUG:boto:Port: 8000 2023-09-22T21:28:25.998 DEBUG:boto:Params: {} 2023-09-22T21:28:25.998 DEBUG:boto:Token: None 2023-09-22T21:28:25.999 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:25 GMT /ffzdqu-55/f?versionId=null 2023-09-22T21:28:25.999 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oovF+mRnOgUnSH7LZNbn0g2N6iU= 2023-09-22T21:28:25.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oovF+mRnOgUnSH7LZNbn0g2N6iU=', 'Content-Length': '0'} 2023-09-22T21:28:26.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033ee9379986e862f-00650e06f9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.042 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:28:26.042 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:28:26.043 DEBUG:boto:path=/ffzdqu-55/f?versionId=null 2023-09-22T21:28:26.043 DEBUG:boto:auth_path=/ffzdqu-55/f?versionId=null 2023-09-22T21:28:26.043 DEBUG:boto:Method: HEAD 2023-09-22T21:28:26.043 DEBUG:boto:Path: /ffzdqu-55/f?versionId=null 2023-09-22T21:28:26.043 DEBUG:boto:Data: 2023-09-22T21:28:26.043 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.043 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.043 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.043 DEBUG:boto:Params: {} 2023-09-22T21:28:26.043 DEBUG:boto:Token: None 2023-09-22T21:28:26.044 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/f?versionId=null 2023-09-22T21:28:26.044 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y+3Gt3CXxn5mqFLOl0mEcPU+SN0= 2023-09-22T21:28:26.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y+3Gt3CXxn5mqFLOl0mEcPU+SN0=', 'Content-Length': '0'} 2023-09-22T21:28:26.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d38cfcdd4c29da03-00650e06fa-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.047 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:28:26.047 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:28:26.047 DEBUG:boto:path=/ffzdqu-55/f?versionId=null 2023-09-22T21:28:26.047 DEBUG:boto:auth_path=/ffzdqu-55/f?versionId=null 2023-09-22T21:28:26.047 DEBUG:boto:Method: HEAD 2023-09-22T21:28:26.047 DEBUG:boto:Path: /ffzdqu-55/f?versionId=null 2023-09-22T21:28:26.047 DEBUG:boto:Data: 2023-09-22T21:28:26.048 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.048 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.048 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.048 DEBUG:boto:Params: {} 2023-09-22T21:28:26.048 DEBUG:boto:Token: None 2023-09-22T21:28:26.048 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/f?versionId=null 2023-09-22T21:28:26.048 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y+3Gt3CXxn5mqFLOl0mEcPU+SN0= 2023-09-22T21:28:26.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y+3Gt3CXxn5mqFLOl0mEcPU+SN0=', 'Content-Length': '0'} 2023-09-22T21:28:26.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1a7da373f323df6-00650e06fa-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-22T21:28:26.052 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:28:26.052 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:28:26.052 DEBUG:boto:path=/ffzdqu-55/f?versionId=null 2023-09-22T21:28:26.052 DEBUG:boto:auth_path=/ffzdqu-55/f?versionId=null 2023-09-22T21:28:26.052 DEBUG:boto:Method: GET 2023-09-22T21:28:26.052 DEBUG:boto:Path: /ffzdqu-55/f?versionId=null 2023-09-22T21:28:26.052 DEBUG:boto:Data: 2023-09-22T21:28:26.052 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.052 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.053 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.053 DEBUG:boto:Params: {} 2023-09-22T21:28:26.053 DEBUG:boto:Token: None 2023-09-22T21:28:26.053 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/f?versionId=null 2023-09-22T21:28:26.053 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QjWJviJOdmrcqDluKahq682gmrY= 2023-09-22T21:28:26.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QjWJviJOdmrcqDluKahq682gmrY=', 'Content-Length': '0'} 2023-09-22T21:28:26.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000376f76480fbfc4f8-00650e06fa-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.098 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:28:26.098 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:28:26.099 DEBUG:boto:path=/ffzdqu-55/f?versionId=null 2023-09-22T21:28:26.099 DEBUG:boto:auth_path=/ffzdqu-55/f?versionId=null 2023-09-22T21:28:26.099 DEBUG:boto:Method: GET 2023-09-22T21:28:26.099 DEBUG:boto:Path: /ffzdqu-55/f?versionId=null 2023-09-22T21:28:26.099 DEBUG:boto:Data: 2023-09-22T21:28:26.099 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.099 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.099 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.099 DEBUG:boto:Params: {} 2023-09-22T21:28:26.100 DEBUG:boto:Token: None 2023-09-22T21:28:26.100 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/f?versionId=null 2023-09-22T21:28:26.100 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QjWJviJOdmrcqDluKahq682gmrY= 2023-09-22T21:28:26.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QjWJviJOdmrcqDluKahq682gmrY=', 'Content-Length': '0'} 2023-09-22T21:28:26.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000158223dfcffe2f3-00650e06fa-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.103 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:26.103 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:26.103 DEBUG:boto:path=/ffzdqu-55/?prefix=f 2023-09-22T21:28:26.103 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=f 2023-09-22T21:28:26.103 DEBUG:boto:Method: GET 2023-09-22T21:28:26.103 DEBUG:boto:Path: /ffzdqu-55/?prefix=f 2023-09-22T21:28:26.103 DEBUG:boto:Data: 2023-09-22T21:28:26.103 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.103 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.104 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.104 DEBUG:boto:Params: {} 2023-09-22T21:28:26.104 DEBUG:boto:Token: None 2023-09-22T21:28:26.104 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/ 2023-09-22T21:28:26.104 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k= 2023-09-22T21:28:26.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k=', 'Content-Length': '0'} 2023-09-22T21:28:26.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdd5d4f9cbce4e2d-00650e06fa-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.150 DEBUG:boto:b'ffzdqu-55f1000falsef2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:26.151 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:26.151 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:26.151 DEBUG:boto:path=/ffzdqu-55/?prefix=f 2023-09-22T21:28:26.151 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=f 2023-09-22T21:28:26.151 DEBUG:boto:Method: GET 2023-09-22T21:28:26.151 DEBUG:boto:Path: /ffzdqu-55/?prefix=f 2023-09-22T21:28:26.151 DEBUG:boto:Data: 2023-09-22T21:28:26.151 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.151 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.152 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.152 DEBUG:boto:Params: {} 2023-09-22T21:28:26.152 DEBUG:boto:Token: None 2023-09-22T21:28:26.152 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/ 2023-09-22T21:28:26.152 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k= 2023-09-22T21:28:26.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k=', 'Content-Length': '0'} 2023-09-22T21:28:26.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cb281cf840809bf-00650e06fa-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.155 DEBUG:boto:b'ffzdqu-55f1000falsef2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:26.156 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:28:26.156 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:28:26.156 DEBUG:boto:Method: HEAD 2023-09-22T21:28:26.156 DEBUG:boto:Path: /ffzdqu-55/f 2023-09-22T21:28:26.156 DEBUG:boto:Data: 2023-09-22T21:28:26.156 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.156 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.156 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.157 DEBUG:boto:Params: {} 2023-09-22T21:28:26.157 DEBUG:boto:Token: None 2023-09-22T21:28:26.157 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/f 2023-09-22T21:28:26.157 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lyzFe+bBL7iXxEasNMuEgniFmfU= 2023-09-22T21:28:26.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lyzFe+bBL7iXxEasNMuEgniFmfU=', 'Content-Length': '0'} 2023-09-22T21:28:26.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c809471f92170b47-00650e06fa-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.160 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:28:26.160 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:28:26.160 DEBUG:boto:Method: HEAD 2023-09-22T21:28:26.160 DEBUG:boto:Path: /ffzdqu-55/f 2023-09-22T21:28:26.160 DEBUG:boto:Data: 2023-09-22T21:28:26.160 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.161 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.161 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.161 DEBUG:boto:Params: {} 2023-09-22T21:28:26.161 DEBUG:boto:Token: None 2023-09-22T21:28:26.161 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/f 2023-09-22T21:28:26.161 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lyzFe+bBL7iXxEasNMuEgniFmfU= 2023-09-22T21:28:26.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lyzFe+bBL7iXxEasNMuEgniFmfU=', 'Content-Length': '0'} 2023-09-22T21:28:26.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c9b410a37298b87-00650e06fa-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-22T21:28:26.166 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:28:26.166 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:28:26.166 DEBUG:boto:Method: GET 2023-09-22T21:28:26.166 DEBUG:boto:Path: /ffzdqu-55/f 2023-09-22T21:28:26.166 DEBUG:boto:Data: 2023-09-22T21:28:26.166 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.166 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.167 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.167 DEBUG:boto:Params: {} 2023-09-22T21:28:26.167 DEBUG:boto:Token: None 2023-09-22T21:28:26.167 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/f 2023-09-22T21:28:26.167 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ejXJ8UErQ55XCcEnpU5ucGKIM9Q= 2023-09-22T21:28:26.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ejXJ8UErQ55XCcEnpU5ucGKIM9Q=', 'Content-Length': '0'} 2023-09-22T21:28:26.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038a7c36e3980c6da-00650e06fa-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.210 DEBUG:boto:path=/ffzdqu-55/f 2023-09-22T21:28:26.210 DEBUG:boto:auth_path=/ffzdqu-55/f 2023-09-22T21:28:26.211 DEBUG:boto:Method: GET 2023-09-22T21:28:26.211 DEBUG:boto:Path: /ffzdqu-55/f 2023-09-22T21:28:26.211 DEBUG:boto:Data: 2023-09-22T21:28:26.211 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.211 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.211 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.211 DEBUG:boto:Params: {} 2023-09-22T21:28:26.211 DEBUG:boto:Token: None 2023-09-22T21:28:26.211 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/f 2023-09-22T21:28:26.211 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ejXJ8UErQ55XCcEnpU5ucGKIM9Q= 2023-09-22T21:28:26.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ejXJ8UErQ55XCcEnpU5ucGKIM9Q=', 'Content-Length': '0'} 2023-09-22T21:28:26.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043827f074aba98dd-00650e06fa-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.214 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:26.215 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:26.215 DEBUG:boto:path=/ffzdqu-55/?prefix=f 2023-09-22T21:28:26.215 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=f 2023-09-22T21:28:26.215 DEBUG:boto:Method: GET 2023-09-22T21:28:26.215 DEBUG:boto:Path: /ffzdqu-55/?prefix=f 2023-09-22T21:28:26.215 DEBUG:boto:Data: 2023-09-22T21:28:26.215 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.215 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.215 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.215 DEBUG:boto:Params: {} 2023-09-22T21:28:26.216 DEBUG:boto:Token: None 2023-09-22T21:28:26.216 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/ 2023-09-22T21:28:26.216 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k= 2023-09-22T21:28:26.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k=', 'Content-Length': '0'} 2023-09-22T21:28:26.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e68bccb92e2f9717-00650e06fa-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.262 DEBUG:boto:b'ffzdqu-55f1000falsef2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:26.263 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:26.263 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:26.263 DEBUG:boto:path=/ffzdqu-55/?prefix=f 2023-09-22T21:28:26.263 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=f 2023-09-22T21:28:26.263 DEBUG:boto:Method: GET 2023-09-22T21:28:26.263 DEBUG:boto:Path: /ffzdqu-55/?prefix=f 2023-09-22T21:28:26.263 DEBUG:boto:Data: 2023-09-22T21:28:26.263 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.263 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.263 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.264 DEBUG:boto:Params: {} 2023-09-22T21:28:26.264 DEBUG:boto:Token: None 2023-09-22T21:28:26.264 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/ 2023-09-22T21:28:26.264 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k= 2023-09-22T21:28:26.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k=', 'Content-Length': '0'} 2023-09-22T21:28:26.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efb8545d1e77c1e8-00650e06fa-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.268 DEBUG:boto:b'ffzdqu-55f1000falsef2023-09-22T21:25:08.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:26.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-22T21:28:26.269 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:28:26.269 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:28:26.269 DEBUG:boto:path=/ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.269 DEBUG:boto:auth_path=/ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.269 DEBUG:boto:Method: GET 2023-09-22T21:28:26.269 DEBUG:boto:Path: /ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.269 DEBUG:boto:Data: 2023-09-22T21:28:26.269 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.270 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.270 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.270 DEBUG:boto:Params: {} 2023-09-22T21:28:26.270 DEBUG:boto:Token: None 2023-09-22T21:28:26.270 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.270 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zx6d34cJ1xs2cUOms7aAqXACUPY= 2023-09-22T21:28:26.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zx6d34cJ1xs2cUOms7aAqXACUPY=', 'Content-Length': '0'} 2023-09-22T21:28:26.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e58dc54ce6f4eb35-00650e06fa-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.273 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:28:26.273 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:28:26.274 DEBUG:boto:path=/ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.274 DEBUG:boto:auth_path=/ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.274 DEBUG:boto:Method: GET 2023-09-22T21:28:26.274 DEBUG:boto:Path: /ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.274 DEBUG:boto:Data: 2023-09-22T21:28:26.274 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.274 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.274 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.274 DEBUG:boto:Params: {} 2023-09-22T21:28:26.275 DEBUG:boto:Token: None 2023-09-22T21:28:26.275 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.275 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zx6d34cJ1xs2cUOms7aAqXACUPY= 2023-09-22T21:28:26.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zx6d34cJ1xs2cUOms7aAqXACUPY=', 'Content-Length': '0'} 2023-09-22T21:28:26.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bba81aedc4626079-00650e06fa-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.322 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:28:26.322 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:28:26.322 DEBUG:boto:path=/ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.323 DEBUG:boto:auth_path=/ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.323 DEBUG:boto:Method: HEAD 2023-09-22T21:28:26.323 DEBUG:boto:Path: /ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.323 DEBUG:boto:Data: 2023-09-22T21:28:26.323 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.323 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.323 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.323 DEBUG:boto:Params: {} 2023-09-22T21:28:26.323 DEBUG:boto:Token: None 2023-09-22T21:28:26.324 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.324 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0nrpFOCg0U3OajKmK0ZXmmJ4XPQ= 2023-09-22T21:28:26.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0nrpFOCg0U3OajKmK0ZXmmJ4XPQ=', 'Content-Length': '0'} 2023-09-22T21:28:26.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068fc232ddee4693a-00650e06fa-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.327 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:28:26.327 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:28:26.327 DEBUG:boto:path=/ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.327 DEBUG:boto:auth_path=/ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.327 DEBUG:boto:Method: HEAD 2023-09-22T21:28:26.327 DEBUG:boto:Path: /ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.327 DEBUG:boto:Data: 2023-09-22T21:28:26.327 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.328 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.328 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.328 DEBUG:boto:Params: {} 2023-09-22T21:28:26.328 DEBUG:boto:Token: None 2023-09-22T21:28:26.328 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.328 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0nrpFOCg0U3OajKmK0ZXmmJ4XPQ= 2023-09-22T21:28:26.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0nrpFOCg0U3OajKmK0ZXmmJ4XPQ=', 'Content-Length': '0'} 2023-09-22T21:28:26.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bbf58790f962e25-00650e06fa-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-22T21:28:26.331 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:28:26.331 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:28:26.331 DEBUG:boto:path=/ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.332 DEBUG:boto:auth_path=/ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.332 DEBUG:boto:Method: GET 2023-09-22T21:28:26.332 DEBUG:boto:Path: /ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.332 DEBUG:boto:Data: 2023-09-22T21:28:26.332 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.332 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.332 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.332 DEBUG:boto:Params: {} 2023-09-22T21:28:26.332 DEBUG:boto:Token: None 2023-09-22T21:28:26.333 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.333 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zx6d34cJ1xs2cUOms7aAqXACUPY= 2023-09-22T21:28:26.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zx6d34cJ1xs2cUOms7aAqXACUPY=', 'Content-Length': '0'} 2023-09-22T21:28:26.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008714b33f23b7634f-00650e06fa-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.378 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:28:26.378 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:28:26.379 DEBUG:boto:path=/ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.379 DEBUG:boto:auth_path=/ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.379 DEBUG:boto:Method: GET 2023-09-22T21:28:26.379 DEBUG:boto:Path: /ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.379 DEBUG:boto:Data: 2023-09-22T21:28:26.379 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.379 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.379 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.379 DEBUG:boto:Params: {} 2023-09-22T21:28:26.379 DEBUG:boto:Token: None 2023-09-22T21:28:26.380 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/g?versionId=null 2023-09-22T21:28:26.380 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zx6d34cJ1xs2cUOms7aAqXACUPY= 2023-09-22T21:28:26.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zx6d34cJ1xs2cUOms7aAqXACUPY=', 'Content-Length': '0'} 2023-09-22T21:28:26.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eed7b19aea3f6eee-00650e06fa-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.383 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:26.383 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:26.383 DEBUG:boto:path=/ffzdqu-55/?prefix=g 2023-09-22T21:28:26.383 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=g 2023-09-22T21:28:26.384 DEBUG:boto:Method: GET 2023-09-22T21:28:26.384 DEBUG:boto:Path: /ffzdqu-55/?prefix=g 2023-09-22T21:28:26.384 DEBUG:boto:Data: 2023-09-22T21:28:26.384 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.384 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.384 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.384 DEBUG:boto:Params: {} 2023-09-22T21:28:26.384 DEBUG:boto:Token: None 2023-09-22T21:28:26.384 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/ 2023-09-22T21:28:26.385 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k= 2023-09-22T21:28:26.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k=', 'Content-Length': '0'} 2023-09-22T21:28:26.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d534f0c43885917c-00650e06fa-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.434 DEBUG:boto:b'ffzdqu-55g1000falseg2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:26.435 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:26.435 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:26.435 DEBUG:boto:path=/ffzdqu-55/?prefix=g 2023-09-22T21:28:26.435 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=g 2023-09-22T21:28:26.435 DEBUG:boto:Method: GET 2023-09-22T21:28:26.435 DEBUG:boto:Path: /ffzdqu-55/?prefix=g 2023-09-22T21:28:26.435 DEBUG:boto:Data: 2023-09-22T21:28:26.436 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.436 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.436 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.436 DEBUG:boto:Params: {} 2023-09-22T21:28:26.436 DEBUG:boto:Token: None 2023-09-22T21:28:26.436 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/ 2023-09-22T21:28:26.436 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k= 2023-09-22T21:28:26.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k=', 'Content-Length': '0'} 2023-09-22T21:28:26.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ec93413fbd103a9-00650e06fa-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.440 DEBUG:boto:b'ffzdqu-55g1000falseg2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:26.440 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:28:26.441 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:28:26.441 DEBUG:boto:Method: HEAD 2023-09-22T21:28:26.441 DEBUG:boto:Path: /ffzdqu-55/g 2023-09-22T21:28:26.441 DEBUG:boto:Data: 2023-09-22T21:28:26.441 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.441 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.441 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.441 DEBUG:boto:Params: {} 2023-09-22T21:28:26.441 DEBUG:boto:Token: None 2023-09-22T21:28:26.441 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/g 2023-09-22T21:28:26.442 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rSRKKMilswsniq3U+hyNxdS1KTE= 2023-09-22T21:28:26.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rSRKKMilswsniq3U+hyNxdS1KTE=', 'Content-Length': '0'} 2023-09-22T21:28:26.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa79dbe3a2c1b650-00650e06fa-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.445 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:28:26.445 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:28:26.445 DEBUG:boto:Method: HEAD 2023-09-22T21:28:26.445 DEBUG:boto:Path: /ffzdqu-55/g 2023-09-22T21:28:26.445 DEBUG:boto:Data: 2023-09-22T21:28:26.445 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.446 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.446 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.446 DEBUG:boto:Params: {} 2023-09-22T21:28:26.446 DEBUG:boto:Token: None 2023-09-22T21:28:26.446 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/g 2023-09-22T21:28:26.446 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rSRKKMilswsniq3U+hyNxdS1KTE= 2023-09-22T21:28:26.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rSRKKMilswsniq3U+hyNxdS1KTE=', 'Content-Length': '0'} 2023-09-22T21:28:26.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca081e088dba67ad-00650e06fa-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-22T21:28:26.450 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:28:26.450 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:28:26.450 DEBUG:boto:Method: GET 2023-09-22T21:28:26.450 DEBUG:boto:Path: /ffzdqu-55/g 2023-09-22T21:28:26.450 DEBUG:boto:Data: 2023-09-22T21:28:26.450 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.450 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.450 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.451 DEBUG:boto:Params: {} 2023-09-22T21:28:26.451 DEBUG:boto:Token: None 2023-09-22T21:28:26.451 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/g 2023-09-22T21:28:26.451 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ih/CdpCzK18AVHuMQgjcYTe62/U= 2023-09-22T21:28:26.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ih/CdpCzK18AVHuMQgjcYTe62/U=', 'Content-Length': '0'} 2023-09-22T21:28:26.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072b46974fa1f18b8-00650e06fa-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.494 DEBUG:boto:path=/ffzdqu-55/g 2023-09-22T21:28:26.494 DEBUG:boto:auth_path=/ffzdqu-55/g 2023-09-22T21:28:26.495 DEBUG:boto:Method: GET 2023-09-22T21:28:26.495 DEBUG:boto:Path: /ffzdqu-55/g 2023-09-22T21:28:26.495 DEBUG:boto:Data: 2023-09-22T21:28:26.495 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.495 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.495 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.495 DEBUG:boto:Params: {} 2023-09-22T21:28:26.495 DEBUG:boto:Token: None 2023-09-22T21:28:26.496 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/g 2023-09-22T21:28:26.496 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ih/CdpCzK18AVHuMQgjcYTe62/U= 2023-09-22T21:28:26.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ih/CdpCzK18AVHuMQgjcYTe62/U=', 'Content-Length': '0'} 2023-09-22T21:28:26.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1b9cd5953cc99cd-00650e06fa-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.499 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:26.499 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:26.499 DEBUG:boto:path=/ffzdqu-55/?prefix=g 2023-09-22T21:28:26.499 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=g 2023-09-22T21:28:26.500 DEBUG:boto:Method: GET 2023-09-22T21:28:26.500 DEBUG:boto:Path: /ffzdqu-55/?prefix=g 2023-09-22T21:28:26.500 DEBUG:boto:Data: 2023-09-22T21:28:26.500 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.500 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.500 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.500 DEBUG:boto:Params: {} 2023-09-22T21:28:26.500 DEBUG:boto:Token: None 2023-09-22T21:28:26.501 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/ 2023-09-22T21:28:26.501 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k= 2023-09-22T21:28:26.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k=', 'Content-Length': '0'} 2023-09-22T21:28:26.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6cabf14966688b4-00650e06fa-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.507 DEBUG:boto:b'ffzdqu-55g1000falseg2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:26.508 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:26.508 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:26.508 DEBUG:boto:path=/ffzdqu-55/?prefix=g 2023-09-22T21:28:26.508 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=g 2023-09-22T21:28:26.508 DEBUG:boto:Method: GET 2023-09-22T21:28:26.508 DEBUG:boto:Path: /ffzdqu-55/?prefix=g 2023-09-22T21:28:26.508 DEBUG:boto:Data: 2023-09-22T21:28:26.508 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.509 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.509 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.509 DEBUG:boto:Params: {} 2023-09-22T21:28:26.509 DEBUG:boto:Token: None 2023-09-22T21:28:26.509 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/ 2023-09-22T21:28:26.509 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k= 2023-09-22T21:28:26.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k=', 'Content-Length': '0'} 2023-09-22T21:28:26.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000752b2b7875bafac2-00650e06fa-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.554 DEBUG:boto:b'ffzdqu-55g1000falseg2023-09-22T21:25:08.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:26.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-22T21:28:26.555 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:28:26.555 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:28:26.555 DEBUG:boto:path=/ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.555 DEBUG:boto:auth_path=/ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.555 DEBUG:boto:Method: GET 2023-09-22T21:28:26.556 DEBUG:boto:Path: /ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.556 DEBUG:boto:Data: 2023-09-22T21:28:26.556 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.556 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.556 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.556 DEBUG:boto:Params: {} 2023-09-22T21:28:26.556 DEBUG:boto:Token: None 2023-09-22T21:28:26.556 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.557 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:X7JLcPWVJfpDc8en0W2Iom0JC0U= 2023-09-22T21:28:26.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:X7JLcPWVJfpDc8en0W2Iom0JC0U=', 'Content-Length': '0'} 2023-09-22T21:28:26.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d9402af0a2957b1-00650e06fa-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.602 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:28:26.603 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:28:26.603 DEBUG:boto:path=/ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.603 DEBUG:boto:auth_path=/ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.603 DEBUG:boto:Method: GET 2023-09-22T21:28:26.603 DEBUG:boto:Path: /ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.603 DEBUG:boto:Data: 2023-09-22T21:28:26.603 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.604 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.604 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.604 DEBUG:boto:Params: {} 2023-09-22T21:28:26.604 DEBUG:boto:Token: None 2023-09-22T21:28:26.604 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.604 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:X7JLcPWVJfpDc8en0W2Iom0JC0U= 2023-09-22T21:28:26.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:X7JLcPWVJfpDc8en0W2Iom0JC0U=', 'Content-Length': '0'} 2023-09-22T21:28:26.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d67d0de7fe69867-00650e06fa-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.608 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:28:26.608 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:28:26.608 DEBUG:boto:path=/ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.608 DEBUG:boto:auth_path=/ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.608 DEBUG:boto:Method: HEAD 2023-09-22T21:28:26.608 DEBUG:boto:Path: /ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.609 DEBUG:boto:Data: 2023-09-22T21:28:26.609 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.609 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.609 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.609 DEBUG:boto:Params: {} 2023-09-22T21:28:26.609 DEBUG:boto:Token: None 2023-09-22T21:28:26.609 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.610 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cFFSdno3Pev+7W78agTXbR1DFIo= 2023-09-22T21:28:26.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cFFSdno3Pev+7W78agTXbR1DFIo=', 'Content-Length': '0'} 2023-09-22T21:28:26.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000838a17252fc4f229-00650e06fa-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.612 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:28:26.613 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:28:26.613 DEBUG:boto:path=/ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.613 DEBUG:boto:auth_path=/ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.613 DEBUG:boto:Method: HEAD 2023-09-22T21:28:26.613 DEBUG:boto:Path: /ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.613 DEBUG:boto:Data: 2023-09-22T21:28:26.613 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.613 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.614 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.614 DEBUG:boto:Params: {} 2023-09-22T21:28:26.614 DEBUG:boto:Token: None 2023-09-22T21:28:26.614 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.614 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cFFSdno3Pev+7W78agTXbR1DFIo= 2023-09-22T21:28:26.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cFFSdno3Pev+7W78agTXbR1DFIo=', 'Content-Length': '0'} 2023-09-22T21:28:26.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002535e11b53c48c44-00650e06fa-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-22T21:28:26.617 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:28:26.617 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:28:26.618 DEBUG:boto:path=/ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.618 DEBUG:boto:auth_path=/ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.618 DEBUG:boto:Method: GET 2023-09-22T21:28:26.618 DEBUG:boto:Path: /ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.618 DEBUG:boto:Data: 2023-09-22T21:28:26.618 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.618 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.618 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.619 DEBUG:boto:Params: {} 2023-09-22T21:28:26.619 DEBUG:boto:Token: None 2023-09-22T21:28:26.619 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.619 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:X7JLcPWVJfpDc8en0W2Iom0JC0U= 2023-09-22T21:28:26.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:X7JLcPWVJfpDc8en0W2Iom0JC0U=', 'Content-Length': '0'} 2023-09-22T21:28:26.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081e146add24b876a-00650e06fa-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.662 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:28:26.663 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:28:26.663 DEBUG:boto:path=/ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.663 DEBUG:boto:auth_path=/ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.663 DEBUG:boto:Method: GET 2023-09-22T21:28:26.663 DEBUG:boto:Path: /ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.663 DEBUG:boto:Data: 2023-09-22T21:28:26.663 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.663 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.664 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.664 DEBUG:boto:Params: {} 2023-09-22T21:28:26.664 DEBUG:boto:Token: None 2023-09-22T21:28:26.664 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/h?versionId=null 2023-09-22T21:28:26.664 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:X7JLcPWVJfpDc8en0W2Iom0JC0U= 2023-09-22T21:28:26.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:X7JLcPWVJfpDc8en0W2Iom0JC0U=', 'Content-Length': '0'} 2023-09-22T21:28:26.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071294b4ad778e0ff-00650e06fa-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.667 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:26.668 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:26.668 DEBUG:boto:path=/ffzdqu-55/?prefix=h 2023-09-22T21:28:26.668 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=h 2023-09-22T21:28:26.668 DEBUG:boto:Method: GET 2023-09-22T21:28:26.668 DEBUG:boto:Path: /ffzdqu-55/?prefix=h 2023-09-22T21:28:26.668 DEBUG:boto:Data: 2023-09-22T21:28:26.668 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.668 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.669 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.669 DEBUG:boto:Params: {} 2023-09-22T21:28:26.669 DEBUG:boto:Token: None 2023-09-22T21:28:26.669 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/ 2023-09-22T21:28:26.669 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k= 2023-09-22T21:28:26.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k=', 'Content-Length': '0'} 2023-09-22T21:28:26.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cce65cc69c13c78a-00650e06fa-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.675 DEBUG:boto:b'ffzdqu-55h1000falseh2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:26.675 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:26.675 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:26.675 DEBUG:boto:path=/ffzdqu-55/?prefix=h 2023-09-22T21:28:26.675 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=h 2023-09-22T21:28:26.676 DEBUG:boto:Method: GET 2023-09-22T21:28:26.676 DEBUG:boto:Path: /ffzdqu-55/?prefix=h 2023-09-22T21:28:26.676 DEBUG:boto:Data: 2023-09-22T21:28:26.676 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.676 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.676 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.676 DEBUG:boto:Params: {} 2023-09-22T21:28:26.676 DEBUG:boto:Token: None 2023-09-22T21:28:26.677 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/ 2023-09-22T21:28:26.677 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k= 2023-09-22T21:28:26.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k=', 'Content-Length': '0'} 2023-09-22T21:28:26.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000465c3ca289b1fd3d-00650e06fa-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.681 DEBUG:boto:b'ffzdqu-55h1000falseh2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:26.681 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:28:26.681 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:28:26.681 DEBUG:boto:Method: HEAD 2023-09-22T21:28:26.681 DEBUG:boto:Path: /ffzdqu-55/h 2023-09-22T21:28:26.681 DEBUG:boto:Data: 2023-09-22T21:28:26.682 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.682 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.682 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.682 DEBUG:boto:Params: {} 2023-09-22T21:28:26.682 DEBUG:boto:Token: None 2023-09-22T21:28:26.682 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/h 2023-09-22T21:28:26.682 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zCRKFaUcznFMq2wMZOg8iouqXXc= 2023-09-22T21:28:26.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zCRKFaUcznFMq2wMZOg8iouqXXc=', 'Content-Length': '0'} 2023-09-22T21:28:26.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007643bc1ed8d9ff83-00650e06fa-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.685 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:28:26.685 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:28:26.685 DEBUG:boto:Method: HEAD 2023-09-22T21:28:26.686 DEBUG:boto:Path: /ffzdqu-55/h 2023-09-22T21:28:26.686 DEBUG:boto:Data: 2023-09-22T21:28:26.686 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.686 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.686 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.686 DEBUG:boto:Params: {} 2023-09-22T21:28:26.686 DEBUG:boto:Token: None 2023-09-22T21:28:26.686 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/h 2023-09-22T21:28:26.687 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zCRKFaUcznFMq2wMZOg8iouqXXc= 2023-09-22T21:28:26.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zCRKFaUcznFMq2wMZOg8iouqXXc=', 'Content-Length': '0'} 2023-09-22T21:28:26.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003974769ab45c5809-00650e06fa-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-22T21:28:26.689 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:28:26.690 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:28:26.690 DEBUG:boto:Method: GET 2023-09-22T21:28:26.690 DEBUG:boto:Path: /ffzdqu-55/h 2023-09-22T21:28:26.690 DEBUG:boto:Data: 2023-09-22T21:28:26.690 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.690 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.690 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.690 DEBUG:boto:Params: {} 2023-09-22T21:28:26.690 DEBUG:boto:Token: None 2023-09-22T21:28:26.691 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/h 2023-09-22T21:28:26.691 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Chb/NtPFDR0I8iZMy/CevTapRCA= 2023-09-22T21:28:26.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Chb/NtPFDR0I8iZMy/CevTapRCA=', 'Content-Length': '0'} 2023-09-22T21:28:26.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001665131c31ff745c-00650e06fa-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.738 DEBUG:boto:path=/ffzdqu-55/h 2023-09-22T21:28:26.738 DEBUG:boto:auth_path=/ffzdqu-55/h 2023-09-22T21:28:26.739 DEBUG:boto:Method: GET 2023-09-22T21:28:26.739 DEBUG:boto:Path: /ffzdqu-55/h 2023-09-22T21:28:26.739 DEBUG:boto:Data: 2023-09-22T21:28:26.739 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.739 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.739 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.739 DEBUG:boto:Params: {} 2023-09-22T21:28:26.739 DEBUG:boto:Token: None 2023-09-22T21:28:26.739 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/h 2023-09-22T21:28:26.740 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Chb/NtPFDR0I8iZMy/CevTapRCA= 2023-09-22T21:28:26.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Chb/NtPFDR0I8iZMy/CevTapRCA=', 'Content-Length': '0'} 2023-09-22T21:28:26.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:25:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000226eca12a56e8776-00650e06fa-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.743 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:26.743 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:26.743 DEBUG:boto:path=/ffzdqu-55/?prefix=h 2023-09-22T21:28:26.743 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=h 2023-09-22T21:28:26.743 DEBUG:boto:Method: GET 2023-09-22T21:28:26.743 DEBUG:boto:Path: /ffzdqu-55/?prefix=h 2023-09-22T21:28:26.744 DEBUG:boto:Data: 2023-09-22T21:28:26.744 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.744 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.744 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.744 DEBUG:boto:Params: {} 2023-09-22T21:28:26.744 DEBUG:boto:Token: None 2023-09-22T21:28:26.744 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/ 2023-09-22T21:28:26.744 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k= 2023-09-22T21:28:26.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k=', 'Content-Length': '0'} 2023-09-22T21:28:26.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4f466287eef9bb8-00650e06fa-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.750 DEBUG:boto:b'ffzdqu-55h1000falseh2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:26.750 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:26.750 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:26.750 DEBUG:boto:path=/ffzdqu-55/?prefix=h 2023-09-22T21:28:26.750 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=h 2023-09-22T21:28:26.751 DEBUG:boto:Method: GET 2023-09-22T21:28:26.751 DEBUG:boto:Path: /ffzdqu-55/?prefix=h 2023-09-22T21:28:26.751 DEBUG:boto:Data: 2023-09-22T21:28:26.751 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.751 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.751 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.751 DEBUG:boto:Params: {} 2023-09-22T21:28:26.751 DEBUG:boto:Token: None 2023-09-22T21:28:26.751 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/ 2023-09-22T21:28:26.752 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k= 2023-09-22T21:28:26.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k=', 'Content-Length': '0'} 2023-09-22T21:28:26.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fde57e18d3e114cb-00650e06fa-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.756 DEBUG:boto:b'ffzdqu-55h1000falseh2023-09-22T21:25:08.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:26.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=i 2023-09-22T21:28:26.756 DEBUG:boto:path=/ffzdqu-55/i 2023-09-22T21:28:26.756 DEBUG:boto:auth_path=/ffzdqu-55/i 2023-09-22T21:28:26.757 DEBUG:boto:path=/ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.757 DEBUG:boto:auth_path=/ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.757 DEBUG:boto:Method: GET 2023-09-22T21:28:26.757 DEBUG:boto:Path: /ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.757 DEBUG:boto:Data: 2023-09-22T21:28:26.757 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.757 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.757 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.757 DEBUG:boto:Params: {} 2023-09-22T21:28:26.757 DEBUG:boto:Token: None 2023-09-22T21:28:26.757 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.758 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:p4qUlaJwUOS332Ef9GaqyC8XaXo= 2023-09-22T21:28:26.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:p4qUlaJwUOS332Ef9GaqyC8XaXo=', 'Content-Length': '0'} 2023-09-22T21:28:26.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f89c4a9dd769b9c-00650e06fa-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.761 DEBUG:boto:path=/ffzdqu-55/i 2023-09-22T21:28:26.761 DEBUG:boto:auth_path=/ffzdqu-55/i 2023-09-22T21:28:26.761 DEBUG:boto:path=/ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.761 DEBUG:boto:auth_path=/ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.761 DEBUG:boto:Method: GET 2023-09-22T21:28:26.762 DEBUG:boto:Path: /ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.762 DEBUG:boto:Data: 2023-09-22T21:28:26.762 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.762 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.762 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.762 DEBUG:boto:Params: {} 2023-09-22T21:28:26.762 DEBUG:boto:Token: None 2023-09-22T21:28:26.762 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.762 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:p4qUlaJwUOS332Ef9GaqyC8XaXo= 2023-09-22T21:28:26.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:p4qUlaJwUOS332Ef9GaqyC8XaXo=', 'Content-Length': '0'} 2023-09-22T21:28:26.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b2151a5246a0a4a-00650e06fa-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.765 DEBUG:boto:path=/ffzdqu-55/i 2023-09-22T21:28:26.766 DEBUG:boto:auth_path=/ffzdqu-55/i 2023-09-22T21:28:26.766 DEBUG:boto:path=/ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.766 DEBUG:boto:auth_path=/ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.766 DEBUG:boto:Method: HEAD 2023-09-22T21:28:26.766 DEBUG:boto:Path: /ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.766 DEBUG:boto:Data: 2023-09-22T21:28:26.766 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.766 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.766 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.767 DEBUG:boto:Params: {} 2023-09-22T21:28:26.767 DEBUG:boto:Token: None 2023-09-22T21:28:26.767 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.767 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:L/VNZl4h6iiBK1nbeI2Sgw6Dioo= 2023-09-22T21:28:26.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:L/VNZl4h6iiBK1nbeI2Sgw6Dioo=', 'Content-Length': '0'} 2023-09-22T21:28:26.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f509dac4dfe41ba-00650e06fa-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.770 DEBUG:boto:path=/ffzdqu-55/i 2023-09-22T21:28:26.770 DEBUG:boto:auth_path=/ffzdqu-55/i 2023-09-22T21:28:26.770 DEBUG:boto:path=/ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.770 DEBUG:boto:auth_path=/ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.770 DEBUG:boto:Method: HEAD 2023-09-22T21:28:26.771 DEBUG:boto:Path: /ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.771 DEBUG:boto:Data: 2023-09-22T21:28:26.771 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.771 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.771 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.771 DEBUG:boto:Params: {} 2023-09-22T21:28:26.771 DEBUG:boto:Token: None 2023-09-22T21:28:26.771 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.772 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:L/VNZl4h6iiBK1nbeI2Sgw6Dioo= 2023-09-22T21:28:26.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:L/VNZl4h6iiBK1nbeI2Sgw6Dioo=', 'Content-Length': '0'} 2023-09-22T21:28:26.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c900b77f781ea805-00650e06fa-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=i 2023-09-22T21:28:26.775 DEBUG:boto:path=/ffzdqu-55/i 2023-09-22T21:28:26.775 DEBUG:boto:auth_path=/ffzdqu-55/i 2023-09-22T21:28:26.775 DEBUG:boto:path=/ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.775 DEBUG:boto:auth_path=/ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.775 DEBUG:boto:Method: GET 2023-09-22T21:28:26.775 DEBUG:boto:Path: /ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.776 DEBUG:boto:Data: 2023-09-22T21:28:26.776 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.776 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.776 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.776 DEBUG:boto:Params: {} 2023-09-22T21:28:26.776 DEBUG:boto:Token: None 2023-09-22T21:28:26.776 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.776 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:p4qUlaJwUOS332Ef9GaqyC8XaXo= 2023-09-22T21:28:26.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:p4qUlaJwUOS332Ef9GaqyC8XaXo=', 'Content-Length': '0'} 2023-09-22T21:28:26.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015d0d84b2d0d805a-00650e06fa-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.822 DEBUG:boto:path=/ffzdqu-55/i 2023-09-22T21:28:26.822 DEBUG:boto:auth_path=/ffzdqu-55/i 2023-09-22T21:28:26.823 DEBUG:boto:path=/ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.823 DEBUG:boto:auth_path=/ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.823 DEBUG:boto:Method: GET 2023-09-22T21:28:26.823 DEBUG:boto:Path: /ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.823 DEBUG:boto:Data: 2023-09-22T21:28:26.823 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.823 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.823 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.824 DEBUG:boto:Params: {} 2023-09-22T21:28:26.824 DEBUG:boto:Token: None 2023-09-22T21:28:26.824 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/i?versionId=null 2023-09-22T21:28:26.824 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:p4qUlaJwUOS332Ef9GaqyC8XaXo= 2023-09-22T21:28:26.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:p4qUlaJwUOS332Ef9GaqyC8XaXo=', 'Content-Length': '0'} 2023-09-22T21:28:26.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abe2a5402a1641ff-00650e06fa-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.827 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:26.827 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:26.828 DEBUG:boto:path=/ffzdqu-55/?prefix=i 2023-09-22T21:28:26.828 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=i 2023-09-22T21:28:26.828 DEBUG:boto:Method: GET 2023-09-22T21:28:26.828 DEBUG:boto:Path: /ffzdqu-55/?prefix=i 2023-09-22T21:28:26.828 DEBUG:boto:Data: 2023-09-22T21:28:26.828 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.828 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.828 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.828 DEBUG:boto:Params: {} 2023-09-22T21:28:26.829 DEBUG:boto:Token: None 2023-09-22T21:28:26.829 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/ 2023-09-22T21:28:26.829 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k= 2023-09-22T21:28:26.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k=', 'Content-Length': '0'} 2023-09-22T21:28:26.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e940bad10c52b4ce-00650e06fa-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.874 DEBUG:boto:b'ffzdqu-55i1000falsei2023-09-22T21:26:49.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:26.875 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:26.875 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:26.875 DEBUG:boto:path=/ffzdqu-55/?prefix=i 2023-09-22T21:28:26.875 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=i 2023-09-22T21:28:26.875 DEBUG:boto:Method: GET 2023-09-22T21:28:26.875 DEBUG:boto:Path: /ffzdqu-55/?prefix=i 2023-09-22T21:28:26.875 DEBUG:boto:Data: 2023-09-22T21:28:26.875 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.875 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.876 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.876 DEBUG:boto:Params: {} 2023-09-22T21:28:26.876 DEBUG:boto:Token: None 2023-09-22T21:28:26.876 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/ 2023-09-22T21:28:26.876 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k= 2023-09-22T21:28:26.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k=', 'Content-Length': '0'} 2023-09-22T21:28:26.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054905d6151854005-00650e06fa-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.880 DEBUG:boto:b'ffzdqu-55i1000falsei2023-09-22T21:26:49.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:26.881 DEBUG:boto:path=/ffzdqu-55/i 2023-09-22T21:28:26.881 DEBUG:boto:auth_path=/ffzdqu-55/i 2023-09-22T21:28:26.881 DEBUG:boto:Method: HEAD 2023-09-22T21:28:26.881 DEBUG:boto:Path: /ffzdqu-55/i 2023-09-22T21:28:26.881 DEBUG:boto:Data: 2023-09-22T21:28:26.881 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.881 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.882 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.882 DEBUG:boto:Params: {} 2023-09-22T21:28:26.882 DEBUG:boto:Token: None 2023-09-22T21:28:26.882 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/i 2023-09-22T21:28:26.882 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qtqtoaO4kQgD9jiELXrvmY/QP5g= 2023-09-22T21:28:26.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qtqtoaO4kQgD9jiELXrvmY/QP5g=', 'Content-Length': '0'} 2023-09-22T21:28:26.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4559975ce483237-00650e06fa-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.885 DEBUG:boto:path=/ffzdqu-55/i 2023-09-22T21:28:26.885 DEBUG:boto:auth_path=/ffzdqu-55/i 2023-09-22T21:28:26.885 DEBUG:boto:Method: HEAD 2023-09-22T21:28:26.885 DEBUG:boto:Path: /ffzdqu-55/i 2023-09-22T21:28:26.885 DEBUG:boto:Data: 2023-09-22T21:28:26.885 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.885 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.886 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.886 DEBUG:boto:Params: {} 2023-09-22T21:28:26.886 DEBUG:boto:Token: None 2023-09-22T21:28:26.886 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/i 2023-09-22T21:28:26.886 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qtqtoaO4kQgD9jiELXrvmY/QP5g= 2023-09-22T21:28:26.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qtqtoaO4kQgD9jiELXrvmY/QP5g=', 'Content-Length': '0'} 2023-09-22T21:28:26.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001146b9612f4e8c68-00650e06fa-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=i 2023-09-22T21:28:26.889 DEBUG:boto:path=/ffzdqu-55/i 2023-09-22T21:28:26.889 DEBUG:boto:auth_path=/ffzdqu-55/i 2023-09-22T21:28:26.889 DEBUG:boto:Method: GET 2023-09-22T21:28:26.889 DEBUG:boto:Path: /ffzdqu-55/i 2023-09-22T21:28:26.890 DEBUG:boto:Data: 2023-09-22T21:28:26.890 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.890 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.890 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.890 DEBUG:boto:Params: {} 2023-09-22T21:28:26.890 DEBUG:boto:Token: None 2023-09-22T21:28:26.890 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/i 2023-09-22T21:28:26.890 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Iurao4Ezeqkrxrvr4hvALZckNmU= 2023-09-22T21:28:26.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Iurao4Ezeqkrxrvr4hvALZckNmU=', 'Content-Length': '0'} 2023-09-22T21:28:26.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba01512670eaf49c-00650e06fa-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.934 DEBUG:boto:path=/ffzdqu-55/i 2023-09-22T21:28:26.935 DEBUG:boto:auth_path=/ffzdqu-55/i 2023-09-22T21:28:26.935 DEBUG:boto:Method: GET 2023-09-22T21:28:26.935 DEBUG:boto:Path: /ffzdqu-55/i 2023-09-22T21:28:26.935 DEBUG:boto:Data: 2023-09-22T21:28:26.935 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.935 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.935 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.935 DEBUG:boto:Params: {} 2023-09-22T21:28:26.935 DEBUG:boto:Token: None 2023-09-22T21:28:26.936 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/i 2023-09-22T21:28:26.936 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Iurao4Ezeqkrxrvr4hvALZckNmU= 2023-09-22T21:28:26.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Iurao4Ezeqkrxrvr4hvALZckNmU=', 'Content-Length': '0'} 2023-09-22T21:28:26.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000097329422966fe6b-00650e06fa-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.939 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:26.939 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:26.939 DEBUG:boto:path=/ffzdqu-55/?prefix=i 2023-09-22T21:28:26.939 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=i 2023-09-22T21:28:26.939 DEBUG:boto:Method: GET 2023-09-22T21:28:26.939 DEBUG:boto:Path: /ffzdqu-55/?prefix=i 2023-09-22T21:28:26.939 DEBUG:boto:Data: 2023-09-22T21:28:26.940 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.940 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.940 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.940 DEBUG:boto:Params: {} 2023-09-22T21:28:26.940 DEBUG:boto:Token: None 2023-09-22T21:28:26.940 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/ 2023-09-22T21:28:26.940 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k= 2023-09-22T21:28:26.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k=', 'Content-Length': '0'} 2023-09-22T21:28:26.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007079efe5e903c619-00650e06fa-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.946 DEBUG:boto:b'ffzdqu-55i1000falsei2023-09-22T21:26:49.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:26.946 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:26.946 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:26.946 DEBUG:boto:path=/ffzdqu-55/?prefix=i 2023-09-22T21:28:26.946 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=i 2023-09-22T21:28:26.947 DEBUG:boto:Method: GET 2023-09-22T21:28:26.947 DEBUG:boto:Path: /ffzdqu-55/?prefix=i 2023-09-22T21:28:26.947 DEBUG:boto:Data: 2023-09-22T21:28:26.947 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.947 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.947 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.947 DEBUG:boto:Params: {} 2023-09-22T21:28:26.947 DEBUG:boto:Token: None 2023-09-22T21:28:26.947 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/ 2023-09-22T21:28:26.948 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k= 2023-09-22T21:28:26.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sgupsh5hZinTpM8sVyTDq4ZPH8k=', 'Content-Length': '0'} 2023-09-22T21:28:26.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000127e5a3080ad3823-00650e06fa-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.952 DEBUG:boto:b'ffzdqu-55i1000falsei2023-09-22T21:26:49.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:26.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=j 2023-09-22T21:28:26.952 DEBUG:boto:path=/ffzdqu-55/j 2023-09-22T21:28:26.952 DEBUG:boto:auth_path=/ffzdqu-55/j 2023-09-22T21:28:26.952 DEBUG:boto:path=/ffzdqu-55/j?versionId=null 2023-09-22T21:28:26.952 DEBUG:boto:auth_path=/ffzdqu-55/j?versionId=null 2023-09-22T21:28:26.953 DEBUG:boto:Method: GET 2023-09-22T21:28:26.953 DEBUG:boto:Path: /ffzdqu-55/j?versionId=null 2023-09-22T21:28:26.953 DEBUG:boto:Data: 2023-09-22T21:28:26.953 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.953 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:26.953 DEBUG:boto:Port: 8001 2023-09-22T21:28:26.953 DEBUG:boto:Params: {} 2023-09-22T21:28:26.953 DEBUG:boto:Token: None 2023-09-22T21:28:26.954 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:26 GMT /ffzdqu-55/j?versionId=null 2023-09-22T21:28:26.954 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fCH/Vlr1eh3QSXRrdSJ0gT9UtFY= 2023-09-22T21:28:26.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fCH/Vlr1eh3QSXRrdSJ0gT9UtFY=', 'Content-Length': '0'} 2023-09-22T21:28:26.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c04a035ff5222dd4-00650e06fa-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:26.998 DEBUG:boto:path=/ffzdqu-55/j 2023-09-22T21:28:26.998 DEBUG:boto:auth_path=/ffzdqu-55/j 2023-09-22T21:28:26.999 DEBUG:boto:path=/ffzdqu-55/j?versionId=null 2023-09-22T21:28:26.999 DEBUG:boto:auth_path=/ffzdqu-55/j?versionId=null 2023-09-22T21:28:26.999 DEBUG:boto:Method: GET 2023-09-22T21:28:26.999 DEBUG:boto:Path: /ffzdqu-55/j?versionId=null 2023-09-22T21:28:26.999 DEBUG:boto:Data: 2023-09-22T21:28:26.999 DEBUG:boto:Headers: {} 2023-09-22T21:28:26.999 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:26.999 DEBUG:boto:Port: 8000 2023-09-22T21:28:26.999 DEBUG:boto:Params: {} 2023-09-22T21:28:26.999 DEBUG:boto:Token: None 2023-09-22T21:28:27.000 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/j?versionId=null 2023-09-22T21:28:27.000 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HOTWyoplCYlWsbUuJK4xc9WNGN0= 2023-09-22T21:28:27.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HOTWyoplCYlWsbUuJK4xc9WNGN0=', 'Content-Length': '0'} 2023-09-22T21:28:27.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053d92dd1df3b114e-00650e06fa-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.003 DEBUG:boto:path=/ffzdqu-55/j 2023-09-22T21:28:27.003 DEBUG:boto:auth_path=/ffzdqu-55/j 2023-09-22T21:28:27.003 DEBUG:boto:path=/ffzdqu-55/j?versionId=null 2023-09-22T21:28:27.003 DEBUG:boto:auth_path=/ffzdqu-55/j?versionId=null 2023-09-22T21:28:27.003 DEBUG:boto:Method: HEAD 2023-09-22T21:28:27.003 DEBUG:boto:Path: /ffzdqu-55/j?versionId=null 2023-09-22T21:28:27.003 DEBUG:boto:Data: 2023-09-22T21:28:27.004 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.004 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:27.004 DEBUG:boto:Port: 8001 2023-09-22T21:28:27.004 DEBUG:boto:Params: {} 2023-09-22T21:28:27.004 DEBUG:boto:Token: None 2023-09-22T21:28:27.004 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/j?versionId=null 2023-09-22T21:28:27.004 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UqNwwXW2XJMPk7obn6JJ4v9QbJc= 2023-09-22T21:28:27.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UqNwwXW2XJMPk7obn6JJ4v9QbJc=', 'Content-Length': '0'} 2023-09-22T21:28:27.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000965453b91b3a4437-00650e06fb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.007 DEBUG:boto:path=/ffzdqu-55/j 2023-09-22T21:28:27.007 DEBUG:boto:auth_path=/ffzdqu-55/j 2023-09-22T21:28:27.007 DEBUG:boto:path=/ffzdqu-55/j?versionId=null 2023-09-22T21:28:27.008 DEBUG:boto:auth_path=/ffzdqu-55/j?versionId=null 2023-09-22T21:28:27.008 DEBUG:boto:Method: HEAD 2023-09-22T21:28:27.008 DEBUG:boto:Path: /ffzdqu-55/j?versionId=null 2023-09-22T21:28:27.008 DEBUG:boto:Data: 2023-09-22T21:28:27.008 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.008 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:27.008 DEBUG:boto:Port: 8000 2023-09-22T21:28:27.008 DEBUG:boto:Params: {} 2023-09-22T21:28:27.008 DEBUG:boto:Token: None 2023-09-22T21:28:27.009 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/j?versionId=null 2023-09-22T21:28:27.009 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UqNwwXW2XJMPk7obn6JJ4v9QbJc= 2023-09-22T21:28:27.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UqNwwXW2XJMPk7obn6JJ4v9QbJc=', 'Content-Length': '0'} 2023-09-22T21:28:27.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f042b85497a04e81-00650e06fb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=j 2023-09-22T21:28:27.012 DEBUG:boto:path=/ffzdqu-55/j 2023-09-22T21:28:27.012 DEBUG:boto:auth_path=/ffzdqu-55/j 2023-09-22T21:28:27.012 DEBUG:boto:path=/ffzdqu-55/j?versionId=null 2023-09-22T21:28:27.012 DEBUG:boto:auth_path=/ffzdqu-55/j?versionId=null 2023-09-22T21:28:27.012 DEBUG:boto:Method: GET 2023-09-22T21:28:27.012 DEBUG:boto:Path: /ffzdqu-55/j?versionId=null 2023-09-22T21:28:27.012 DEBUG:boto:Data: 2023-09-22T21:28:27.012 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.012 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:27.013 DEBUG:boto:Port: 8001 2023-09-22T21:28:27.013 DEBUG:boto:Params: {} 2023-09-22T21:28:27.013 DEBUG:boto:Token: None 2023-09-22T21:28:27.013 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/j?versionId=null 2023-09-22T21:28:27.013 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HOTWyoplCYlWsbUuJK4xc9WNGN0= 2023-09-22T21:28:27.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HOTWyoplCYlWsbUuJK4xc9WNGN0=', 'Content-Length': '0'} 2023-09-22T21:28:27.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004701f190052dd30e-00650e06fb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.058 DEBUG:boto:path=/ffzdqu-55/j 2023-09-22T21:28:27.058 DEBUG:boto:auth_path=/ffzdqu-55/j 2023-09-22T21:28:27.059 DEBUG:boto:path=/ffzdqu-55/j?versionId=null 2023-09-22T21:28:27.059 DEBUG:boto:auth_path=/ffzdqu-55/j?versionId=null 2023-09-22T21:28:27.059 DEBUG:boto:Method: GET 2023-09-22T21:28:27.059 DEBUG:boto:Path: /ffzdqu-55/j?versionId=null 2023-09-22T21:28:27.059 DEBUG:boto:Data: 2023-09-22T21:28:27.059 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.059 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:27.059 DEBUG:boto:Port: 8000 2023-09-22T21:28:27.059 DEBUG:boto:Params: {} 2023-09-22T21:28:27.059 DEBUG:boto:Token: None 2023-09-22T21:28:27.060 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/j?versionId=null 2023-09-22T21:28:27.060 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HOTWyoplCYlWsbUuJK4xc9WNGN0= 2023-09-22T21:28:27.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HOTWyoplCYlWsbUuJK4xc9WNGN0=', 'Content-Length': '0'} 2023-09-22T21:28:27.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1ccd280f4f043fa-00650e06fb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.063 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:27.063 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:27.063 DEBUG:boto:path=/ffzdqu-55/?prefix=j 2023-09-22T21:28:27.063 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=j 2023-09-22T21:28:27.063 DEBUG:boto:Method: GET 2023-09-22T21:28:27.063 DEBUG:boto:Path: /ffzdqu-55/?prefix=j 2023-09-22T21:28:27.063 DEBUG:boto:Data: 2023-09-22T21:28:27.063 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.063 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:27.064 DEBUG:boto:Port: 8001 2023-09-22T21:28:27.064 DEBUG:boto:Params: {} 2023-09-22T21:28:27.064 DEBUG:boto:Token: None 2023-09-22T21:28:27.064 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/ 2023-09-22T21:28:27.064 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg= 2023-09-22T21:28:27.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg=', 'Content-Length': '0'} 2023-09-22T21:28:27.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039c715661380200b-00650e06fb-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.110 DEBUG:boto:b'ffzdqu-55j1000falsej2023-09-22T21:26:49.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:27.111 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:27.111 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:27.111 DEBUG:boto:path=/ffzdqu-55/?prefix=j 2023-09-22T21:28:27.111 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=j 2023-09-22T21:28:27.111 DEBUG:boto:Method: GET 2023-09-22T21:28:27.111 DEBUG:boto:Path: /ffzdqu-55/?prefix=j 2023-09-22T21:28:27.111 DEBUG:boto:Data: 2023-09-22T21:28:27.111 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.112 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:27.112 DEBUG:boto:Port: 8000 2023-09-22T21:28:27.112 DEBUG:boto:Params: {} 2023-09-22T21:28:27.112 DEBUG:boto:Token: None 2023-09-22T21:28:27.112 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/ 2023-09-22T21:28:27.112 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg= 2023-09-22T21:28:27.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg=', 'Content-Length': '0'} 2023-09-22T21:28:27.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e0d301d8f6fb38f-00650e06fb-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.116 DEBUG:boto:b'ffzdqu-55j1000falsej2023-09-22T21:26:49.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:27.116 DEBUG:boto:path=/ffzdqu-55/j 2023-09-22T21:28:27.116 DEBUG:boto:auth_path=/ffzdqu-55/j 2023-09-22T21:28:27.117 DEBUG:boto:Method: HEAD 2023-09-22T21:28:27.117 DEBUG:boto:Path: /ffzdqu-55/j 2023-09-22T21:28:27.117 DEBUG:boto:Data: 2023-09-22T21:28:27.117 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.117 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:27.117 DEBUG:boto:Port: 8001 2023-09-22T21:28:27.117 DEBUG:boto:Params: {} 2023-09-22T21:28:27.117 DEBUG:boto:Token: None 2023-09-22T21:28:27.117 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/j 2023-09-22T21:28:27.118 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3KDmRucSvBnQR/sVDxdaFDjvUv4= 2023-09-22T21:28:27.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3KDmRucSvBnQR/sVDxdaFDjvUv4=', 'Content-Length': '0'} 2023-09-22T21:28:27.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008789c9342fcc1b99-00650e06fb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.120 DEBUG:boto:path=/ffzdqu-55/j 2023-09-22T21:28:27.120 DEBUG:boto:auth_path=/ffzdqu-55/j 2023-09-22T21:28:27.121 DEBUG:boto:Method: HEAD 2023-09-22T21:28:27.121 DEBUG:boto:Path: /ffzdqu-55/j 2023-09-22T21:28:27.121 DEBUG:boto:Data: 2023-09-22T21:28:27.121 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.121 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:27.121 DEBUG:boto:Port: 8000 2023-09-22T21:28:27.121 DEBUG:boto:Params: {} 2023-09-22T21:28:27.121 DEBUG:boto:Token: None 2023-09-22T21:28:27.121 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/j 2023-09-22T21:28:27.121 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3KDmRucSvBnQR/sVDxdaFDjvUv4= 2023-09-22T21:28:27.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3KDmRucSvBnQR/sVDxdaFDjvUv4=', 'Content-Length': '0'} 2023-09-22T21:28:27.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009053091db5ad1a9b-00650e06fb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=j 2023-09-22T21:28:27.124 DEBUG:boto:path=/ffzdqu-55/j 2023-09-22T21:28:27.125 DEBUG:boto:auth_path=/ffzdqu-55/j 2023-09-22T21:28:27.125 DEBUG:boto:Method: GET 2023-09-22T21:28:27.125 DEBUG:boto:Path: /ffzdqu-55/j 2023-09-22T21:28:27.125 DEBUG:boto:Data: 2023-09-22T21:28:27.125 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.125 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:27.125 DEBUG:boto:Port: 8001 2023-09-22T21:28:27.125 DEBUG:boto:Params: {} 2023-09-22T21:28:27.125 DEBUG:boto:Token: None 2023-09-22T21:28:27.126 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/j 2023-09-22T21:28:27.126 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EYniwG/OX4YZ4ti6ovmOMwdOLWI= 2023-09-22T21:28:27.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EYniwG/OX4YZ4ti6ovmOMwdOLWI=', 'Content-Length': '0'} 2023-09-22T21:28:27.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005f66fcfd473c83e-00650e06fb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.170 DEBUG:boto:path=/ffzdqu-55/j 2023-09-22T21:28:27.171 DEBUG:boto:auth_path=/ffzdqu-55/j 2023-09-22T21:28:27.171 DEBUG:boto:Method: GET 2023-09-22T21:28:27.171 DEBUG:boto:Path: /ffzdqu-55/j 2023-09-22T21:28:27.171 DEBUG:boto:Data: 2023-09-22T21:28:27.171 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.171 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:27.171 DEBUG:boto:Port: 8000 2023-09-22T21:28:27.171 DEBUG:boto:Params: {} 2023-09-22T21:28:27.171 DEBUG:boto:Token: None 2023-09-22T21:28:27.172 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/j 2023-09-22T21:28:27.172 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EYniwG/OX4YZ4ti6ovmOMwdOLWI= 2023-09-22T21:28:27.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EYniwG/OX4YZ4ti6ovmOMwdOLWI=', 'Content-Length': '0'} 2023-09-22T21:28:27.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000354db1e12a7c4802-00650e06fb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.175 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:27.175 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:27.175 DEBUG:boto:path=/ffzdqu-55/?prefix=j 2023-09-22T21:28:27.175 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=j 2023-09-22T21:28:27.175 DEBUG:boto:Method: GET 2023-09-22T21:28:27.175 DEBUG:boto:Path: /ffzdqu-55/?prefix=j 2023-09-22T21:28:27.175 DEBUG:boto:Data: 2023-09-22T21:28:27.176 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.176 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:27.176 DEBUG:boto:Port: 8001 2023-09-22T21:28:27.176 DEBUG:boto:Params: {} 2023-09-22T21:28:27.176 DEBUG:boto:Token: None 2023-09-22T21:28:27.176 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/ 2023-09-22T21:28:27.176 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg= 2023-09-22T21:28:27.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg=', 'Content-Length': '0'} 2023-09-22T21:28:27.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000475073f0c896e4c3-00650e06fb-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.222 DEBUG:boto:b'ffzdqu-55j1000falsej2023-09-22T21:26:49.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:27.223 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:27.223 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:27.223 DEBUG:boto:path=/ffzdqu-55/?prefix=j 2023-09-22T21:28:27.223 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=j 2023-09-22T21:28:27.223 DEBUG:boto:Method: GET 2023-09-22T21:28:27.223 DEBUG:boto:Path: /ffzdqu-55/?prefix=j 2023-09-22T21:28:27.224 DEBUG:boto:Data: 2023-09-22T21:28:27.224 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.224 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:27.224 DEBUG:boto:Port: 8000 2023-09-22T21:28:27.224 DEBUG:boto:Params: {} 2023-09-22T21:28:27.224 DEBUG:boto:Token: None 2023-09-22T21:28:27.224 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/ 2023-09-22T21:28:27.225 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg= 2023-09-22T21:28:27.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg=', 'Content-Length': '0'} 2023-09-22T21:28:27.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fb1e2ffd09658d0-00650e06fb-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.229 DEBUG:boto:b'ffzdqu-55j1000falsej2023-09-22T21:26:49.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:27.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=k 2023-09-22T21:28:27.230 DEBUG:boto:path=/ffzdqu-55/k 2023-09-22T21:28:27.230 DEBUG:boto:auth_path=/ffzdqu-55/k 2023-09-22T21:28:27.230 DEBUG:boto:path=/ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.230 DEBUG:boto:auth_path=/ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.230 DEBUG:boto:Method: GET 2023-09-22T21:28:27.230 DEBUG:boto:Path: /ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.231 DEBUG:boto:Data: 2023-09-22T21:28:27.231 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.231 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:27.231 DEBUG:boto:Port: 8001 2023-09-22T21:28:27.231 DEBUG:boto:Params: {} 2023-09-22T21:28:27.231 DEBUG:boto:Token: None 2023-09-22T21:28:27.231 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.231 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sDkM9g3okkJpJ7LTDaN/Ynkemfo= 2023-09-22T21:28:27.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sDkM9g3okkJpJ7LTDaN/Ynkemfo=', 'Content-Length': '0'} 2023-09-22T21:28:27.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009433190afc9841bc-00650e06fb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.235 DEBUG:boto:path=/ffzdqu-55/k 2023-09-22T21:28:27.235 DEBUG:boto:auth_path=/ffzdqu-55/k 2023-09-22T21:28:27.235 DEBUG:boto:path=/ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.235 DEBUG:boto:auth_path=/ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.235 DEBUG:boto:Method: GET 2023-09-22T21:28:27.236 DEBUG:boto:Path: /ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.236 DEBUG:boto:Data: 2023-09-22T21:28:27.236 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.236 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:27.236 DEBUG:boto:Port: 8000 2023-09-22T21:28:27.236 DEBUG:boto:Params: {} 2023-09-22T21:28:27.236 DEBUG:boto:Token: None 2023-09-22T21:28:27.236 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.236 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sDkM9g3okkJpJ7LTDaN/Ynkemfo= 2023-09-22T21:28:27.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sDkM9g3okkJpJ7LTDaN/Ynkemfo=', 'Content-Length': '0'} 2023-09-22T21:28:27.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a7568e69aa09fb1-00650e06fb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.239 DEBUG:boto:path=/ffzdqu-55/k 2023-09-22T21:28:27.239 DEBUG:boto:auth_path=/ffzdqu-55/k 2023-09-22T21:28:27.240 DEBUG:boto:path=/ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.240 DEBUG:boto:auth_path=/ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.240 DEBUG:boto:Method: HEAD 2023-09-22T21:28:27.240 DEBUG:boto:Path: /ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.240 DEBUG:boto:Data: 2023-09-22T21:28:27.240 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.240 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:27.240 DEBUG:boto:Port: 8001 2023-09-22T21:28:27.240 DEBUG:boto:Params: {} 2023-09-22T21:28:27.240 DEBUG:boto:Token: None 2023-09-22T21:28:27.241 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.241 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jgHnZ4iu869nzj1Lqq6m4zKX6wU= 2023-09-22T21:28:27.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jgHnZ4iu869nzj1Lqq6m4zKX6wU=', 'Content-Length': '0'} 2023-09-22T21:28:27.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b8786718efab842-00650e06fb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.244 DEBUG:boto:path=/ffzdqu-55/k 2023-09-22T21:28:27.244 DEBUG:boto:auth_path=/ffzdqu-55/k 2023-09-22T21:28:27.244 DEBUG:boto:path=/ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.244 DEBUG:boto:auth_path=/ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.244 DEBUG:boto:Method: HEAD 2023-09-22T21:28:27.244 DEBUG:boto:Path: /ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.244 DEBUG:boto:Data: 2023-09-22T21:28:27.244 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.245 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:27.245 DEBUG:boto:Port: 8000 2023-09-22T21:28:27.245 DEBUG:boto:Params: {} 2023-09-22T21:28:27.245 DEBUG:boto:Token: None 2023-09-22T21:28:27.245 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.245 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jgHnZ4iu869nzj1Lqq6m4zKX6wU= 2023-09-22T21:28:27.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jgHnZ4iu869nzj1Lqq6m4zKX6wU=', 'Content-Length': '0'} 2023-09-22T21:28:27.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001119c1f49fb2815d-00650e06fb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=k 2023-09-22T21:28:27.248 DEBUG:boto:path=/ffzdqu-55/k 2023-09-22T21:28:27.248 DEBUG:boto:auth_path=/ffzdqu-55/k 2023-09-22T21:28:27.248 DEBUG:boto:path=/ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.249 DEBUG:boto:auth_path=/ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.249 DEBUG:boto:Method: GET 2023-09-22T21:28:27.249 DEBUG:boto:Path: /ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.249 DEBUG:boto:Data: 2023-09-22T21:28:27.249 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.249 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:27.249 DEBUG:boto:Port: 8001 2023-09-22T21:28:27.249 DEBUG:boto:Params: {} 2023-09-22T21:28:27.249 DEBUG:boto:Token: None 2023-09-22T21:28:27.249 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.250 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sDkM9g3okkJpJ7LTDaN/Ynkemfo= 2023-09-22T21:28:27.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sDkM9g3okkJpJ7LTDaN/Ynkemfo=', 'Content-Length': '0'} 2023-09-22T21:28:27.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004652b6d1a7775bc9-00650e06fb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.294 DEBUG:boto:path=/ffzdqu-55/k 2023-09-22T21:28:27.294 DEBUG:boto:auth_path=/ffzdqu-55/k 2023-09-22T21:28:27.295 DEBUG:boto:path=/ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.295 DEBUG:boto:auth_path=/ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.295 DEBUG:boto:Method: GET 2023-09-22T21:28:27.295 DEBUG:boto:Path: /ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.295 DEBUG:boto:Data: 2023-09-22T21:28:27.295 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.295 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:27.295 DEBUG:boto:Port: 8000 2023-09-22T21:28:27.295 DEBUG:boto:Params: {} 2023-09-22T21:28:27.295 DEBUG:boto:Token: None 2023-09-22T21:28:27.296 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/k?versionId=null 2023-09-22T21:28:27.296 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sDkM9g3okkJpJ7LTDaN/Ynkemfo= 2023-09-22T21:28:27.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sDkM9g3okkJpJ7LTDaN/Ynkemfo=', 'Content-Length': '0'} 2023-09-22T21:28:27.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023617bbe1c80a682-00650e06fb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.299 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:27.299 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:27.299 DEBUG:boto:path=/ffzdqu-55/?prefix=k 2023-09-22T21:28:27.299 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=k 2023-09-22T21:28:27.299 DEBUG:boto:Method: GET 2023-09-22T21:28:27.299 DEBUG:boto:Path: /ffzdqu-55/?prefix=k 2023-09-22T21:28:27.299 DEBUG:boto:Data: 2023-09-22T21:28:27.300 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.300 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:27.300 DEBUG:boto:Port: 8001 2023-09-22T21:28:27.300 DEBUG:boto:Params: {} 2023-09-22T21:28:27.300 DEBUG:boto:Token: None 2023-09-22T21:28:27.300 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/ 2023-09-22T21:28:27.300 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg= 2023-09-22T21:28:27.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg=', 'Content-Length': '0'} 2023-09-22T21:28:27.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f28d448566e9ea05-00650e06fb-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.306 DEBUG:boto:b'ffzdqu-55k1000falsek2023-09-22T21:26:49.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:27.307 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:27.307 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:27.307 DEBUG:boto:path=/ffzdqu-55/?prefix=k 2023-09-22T21:28:27.307 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=k 2023-09-22T21:28:27.307 DEBUG:boto:Method: GET 2023-09-22T21:28:27.307 DEBUG:boto:Path: /ffzdqu-55/?prefix=k 2023-09-22T21:28:27.307 DEBUG:boto:Data: 2023-09-22T21:28:27.307 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.307 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:27.308 DEBUG:boto:Port: 8000 2023-09-22T21:28:27.308 DEBUG:boto:Params: {} 2023-09-22T21:28:27.308 DEBUG:boto:Token: None 2023-09-22T21:28:27.308 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/ 2023-09-22T21:28:27.308 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg= 2023-09-22T21:28:27.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg=', 'Content-Length': '0'} 2023-09-22T21:28:27.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031baaccc66634eeb-00650e06fb-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.354 DEBUG:boto:b'ffzdqu-55k1000falsek2023-09-22T21:26:49.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:27.355 DEBUG:boto:path=/ffzdqu-55/k 2023-09-22T21:28:27.355 DEBUG:boto:auth_path=/ffzdqu-55/k 2023-09-22T21:28:27.355 DEBUG:boto:Method: HEAD 2023-09-22T21:28:27.355 DEBUG:boto:Path: /ffzdqu-55/k 2023-09-22T21:28:27.355 DEBUG:boto:Data: 2023-09-22T21:28:27.355 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.355 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:27.355 DEBUG:boto:Port: 8001 2023-09-22T21:28:27.355 DEBUG:boto:Params: {} 2023-09-22T21:28:27.355 DEBUG:boto:Token: None 2023-09-22T21:28:27.356 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/k 2023-09-22T21:28:27.356 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PMWka6XmjDuC8/1vbCTH4T8aPOU= 2023-09-22T21:28:27.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PMWka6XmjDuC8/1vbCTH4T8aPOU=', 'Content-Length': '0'} 2023-09-22T21:28:27.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027cf96583bf12991-00650e06fb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.359 DEBUG:boto:path=/ffzdqu-55/k 2023-09-22T21:28:27.359 DEBUG:boto:auth_path=/ffzdqu-55/k 2023-09-22T21:28:27.359 DEBUG:boto:Method: HEAD 2023-09-22T21:28:27.359 DEBUG:boto:Path: /ffzdqu-55/k 2023-09-22T21:28:27.359 DEBUG:boto:Data: 2023-09-22T21:28:27.359 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.359 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:27.359 DEBUG:boto:Port: 8000 2023-09-22T21:28:27.360 DEBUG:boto:Params: {} 2023-09-22T21:28:27.360 DEBUG:boto:Token: None 2023-09-22T21:28:27.360 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/k 2023-09-22T21:28:27.360 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PMWka6XmjDuC8/1vbCTH4T8aPOU= 2023-09-22T21:28:27.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PMWka6XmjDuC8/1vbCTH4T8aPOU=', 'Content-Length': '0'} 2023-09-22T21:28:27.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a55723a7628a9b9-00650e06fb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=k 2023-09-22T21:28:27.363 DEBUG:boto:path=/ffzdqu-55/k 2023-09-22T21:28:27.363 DEBUG:boto:auth_path=/ffzdqu-55/k 2023-09-22T21:28:27.363 DEBUG:boto:Method: GET 2023-09-22T21:28:27.363 DEBUG:boto:Path: /ffzdqu-55/k 2023-09-22T21:28:27.363 DEBUG:boto:Data: 2023-09-22T21:28:27.364 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.364 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:27.364 DEBUG:boto:Port: 8001 2023-09-22T21:28:27.364 DEBUG:boto:Params: {} 2023-09-22T21:28:27.364 DEBUG:boto:Token: None 2023-09-22T21:28:27.364 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/k 2023-09-22T21:28:27.364 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mnmlde5RWxuN1F4Fw6aevt3V8G0= 2023-09-22T21:28:27.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mnmlde5RWxuN1F4Fw6aevt3V8G0=', 'Content-Length': '0'} 2023-09-22T21:28:27.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001da5d6ddeee4163b-00650e06fb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.410 DEBUG:boto:path=/ffzdqu-55/k 2023-09-22T21:28:27.411 DEBUG:boto:auth_path=/ffzdqu-55/k 2023-09-22T21:28:27.411 DEBUG:boto:Method: GET 2023-09-22T21:28:27.411 DEBUG:boto:Path: /ffzdqu-55/k 2023-09-22T21:28:27.411 DEBUG:boto:Data: 2023-09-22T21:28:27.411 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.411 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:27.411 DEBUG:boto:Port: 8000 2023-09-22T21:28:27.411 DEBUG:boto:Params: {} 2023-09-22T21:28:27.412 DEBUG:boto:Token: None 2023-09-22T21:28:27.412 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/k 2023-09-22T21:28:27.412 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mnmlde5RWxuN1F4Fw6aevt3V8G0= 2023-09-22T21:28:27.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mnmlde5RWxuN1F4Fw6aevt3V8G0=', 'Content-Length': '0'} 2023-09-22T21:28:27.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000949aaf911c9056e2-00650e06fb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.415 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:27.415 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:27.416 DEBUG:boto:path=/ffzdqu-55/?prefix=k 2023-09-22T21:28:27.416 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=k 2023-09-22T21:28:27.416 DEBUG:boto:Method: GET 2023-09-22T21:28:27.416 DEBUG:boto:Path: /ffzdqu-55/?prefix=k 2023-09-22T21:28:27.416 DEBUG:boto:Data: 2023-09-22T21:28:27.416 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.416 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:27.416 DEBUG:boto:Port: 8001 2023-09-22T21:28:27.416 DEBUG:boto:Params: {} 2023-09-22T21:28:27.416 DEBUG:boto:Token: None 2023-09-22T21:28:27.417 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/ 2023-09-22T21:28:27.417 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg= 2023-09-22T21:28:27.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg=', 'Content-Length': '0'} 2023-09-22T21:28:27.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b04b160461fb6cd2-00650e06fb-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.470 DEBUG:boto:b'ffzdqu-55k1000falsek2023-09-22T21:26:49.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:27.471 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:27.471 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:27.471 DEBUG:boto:path=/ffzdqu-55/?prefix=k 2023-09-22T21:28:27.471 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=k 2023-09-22T21:28:27.471 DEBUG:boto:Method: GET 2023-09-22T21:28:27.472 DEBUG:boto:Path: /ffzdqu-55/?prefix=k 2023-09-22T21:28:27.472 DEBUG:boto:Data: 2023-09-22T21:28:27.472 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.472 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:27.472 DEBUG:boto:Port: 8000 2023-09-22T21:28:27.472 DEBUG:boto:Params: {} 2023-09-22T21:28:27.472 DEBUG:boto:Token: None 2023-09-22T21:28:27.473 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/ 2023-09-22T21:28:27.473 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg= 2023-09-22T21:28:27.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg=', 'Content-Length': '0'} 2023-09-22T21:28:27.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000433dbafb0687c91f-00650e06fb-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.477 DEBUG:boto:b'ffzdqu-55k1000falsek2023-09-22T21:26:49.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:27.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=l 2023-09-22T21:28:27.477 DEBUG:boto:path=/ffzdqu-55/l 2023-09-22T21:28:27.478 DEBUG:boto:auth_path=/ffzdqu-55/l 2023-09-22T21:28:27.478 DEBUG:boto:path=/ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.478 DEBUG:boto:auth_path=/ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.478 DEBUG:boto:Method: GET 2023-09-22T21:28:27.478 DEBUG:boto:Path: /ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.478 DEBUG:boto:Data: 2023-09-22T21:28:27.478 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.478 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:27.478 DEBUG:boto:Port: 8001 2023-09-22T21:28:27.478 DEBUG:boto:Params: {} 2023-09-22T21:28:27.479 DEBUG:boto:Token: None 2023-09-22T21:28:27.479 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.479 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f52aEvDwgce1nRZbfuzmI/FvwWM= 2023-09-22T21:28:27.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f52aEvDwgce1nRZbfuzmI/FvwWM=', 'Content-Length': '0'} 2023-09-22T21:28:27.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cc42551efcd87ec-00650e06fb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.522 DEBUG:boto:path=/ffzdqu-55/l 2023-09-22T21:28:27.522 DEBUG:boto:auth_path=/ffzdqu-55/l 2023-09-22T21:28:27.523 DEBUG:boto:path=/ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.523 DEBUG:boto:auth_path=/ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.523 DEBUG:boto:Method: GET 2023-09-22T21:28:27.523 DEBUG:boto:Path: /ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.523 DEBUG:boto:Data: 2023-09-22T21:28:27.523 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.523 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:27.523 DEBUG:boto:Port: 8000 2023-09-22T21:28:27.523 DEBUG:boto:Params: {} 2023-09-22T21:28:27.523 DEBUG:boto:Token: None 2023-09-22T21:28:27.524 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.524 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f52aEvDwgce1nRZbfuzmI/FvwWM= 2023-09-22T21:28:27.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f52aEvDwgce1nRZbfuzmI/FvwWM=', 'Content-Length': '0'} 2023-09-22T21:28:27.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4f74d015be7d457-00650e06fb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.527 DEBUG:boto:path=/ffzdqu-55/l 2023-09-22T21:28:27.528 DEBUG:boto:auth_path=/ffzdqu-55/l 2023-09-22T21:28:27.528 DEBUG:boto:path=/ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.528 DEBUG:boto:auth_path=/ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.528 DEBUG:boto:Method: HEAD 2023-09-22T21:28:27.528 DEBUG:boto:Path: /ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.528 DEBUG:boto:Data: 2023-09-22T21:28:27.528 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.528 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:27.528 DEBUG:boto:Port: 8001 2023-09-22T21:28:27.529 DEBUG:boto:Params: {} 2023-09-22T21:28:27.529 DEBUG:boto:Token: None 2023-09-22T21:28:27.529 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.529 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bXk6TR5+f6Vv/UzFUUM//pzGQ4w= 2023-09-22T21:28:27.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bXk6TR5+f6Vv/UzFUUM//pzGQ4w=', 'Content-Length': '0'} 2023-09-22T21:28:27.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c230f0342d1362b5-00650e06fb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.532 DEBUG:boto:path=/ffzdqu-55/l 2023-09-22T21:28:27.532 DEBUG:boto:auth_path=/ffzdqu-55/l 2023-09-22T21:28:27.532 DEBUG:boto:path=/ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.532 DEBUG:boto:auth_path=/ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.533 DEBUG:boto:Method: HEAD 2023-09-22T21:28:27.533 DEBUG:boto:Path: /ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.533 DEBUG:boto:Data: 2023-09-22T21:28:27.533 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.533 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:27.533 DEBUG:boto:Port: 8000 2023-09-22T21:28:27.533 DEBUG:boto:Params: {} 2023-09-22T21:28:27.533 DEBUG:boto:Token: None 2023-09-22T21:28:27.533 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.534 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bXk6TR5+f6Vv/UzFUUM//pzGQ4w= 2023-09-22T21:28:27.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bXk6TR5+f6Vv/UzFUUM//pzGQ4w=', 'Content-Length': '0'} 2023-09-22T21:28:27.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008404d8cea45aaa26-00650e06fb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=l 2023-09-22T21:28:27.537 DEBUG:boto:path=/ffzdqu-55/l 2023-09-22T21:28:27.537 DEBUG:boto:auth_path=/ffzdqu-55/l 2023-09-22T21:28:27.537 DEBUG:boto:path=/ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.537 DEBUG:boto:auth_path=/ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.537 DEBUG:boto:Method: GET 2023-09-22T21:28:27.537 DEBUG:boto:Path: /ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.538 DEBUG:boto:Data: 2023-09-22T21:28:27.538 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.538 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:27.538 DEBUG:boto:Port: 8001 2023-09-22T21:28:27.538 DEBUG:boto:Params: {} 2023-09-22T21:28:27.538 DEBUG:boto:Token: None 2023-09-22T21:28:27.538 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.538 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f52aEvDwgce1nRZbfuzmI/FvwWM= 2023-09-22T21:28:27.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f52aEvDwgce1nRZbfuzmI/FvwWM=', 'Content-Length': '0'} 2023-09-22T21:28:27.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de0aac692553473e-00650e06fb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.582 DEBUG:boto:path=/ffzdqu-55/l 2023-09-22T21:28:27.582 DEBUG:boto:auth_path=/ffzdqu-55/l 2023-09-22T21:28:27.582 DEBUG:boto:path=/ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.583 DEBUG:boto:auth_path=/ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.583 DEBUG:boto:Method: GET 2023-09-22T21:28:27.583 DEBUG:boto:Path: /ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.583 DEBUG:boto:Data: 2023-09-22T21:28:27.583 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.583 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:27.583 DEBUG:boto:Port: 8000 2023-09-22T21:28:27.583 DEBUG:boto:Params: {} 2023-09-22T21:28:27.583 DEBUG:boto:Token: None 2023-09-22T21:28:27.584 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/l?versionId=null 2023-09-22T21:28:27.584 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f52aEvDwgce1nRZbfuzmI/FvwWM= 2023-09-22T21:28:27.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f52aEvDwgce1nRZbfuzmI/FvwWM=', 'Content-Length': '0'} 2023-09-22T21:28:27.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3aa67548225d796-00650e06fb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.587 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:27.587 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:27.587 DEBUG:boto:path=/ffzdqu-55/?prefix=l 2023-09-22T21:28:27.587 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=l 2023-09-22T21:28:27.587 DEBUG:boto:Method: GET 2023-09-22T21:28:27.587 DEBUG:boto:Path: /ffzdqu-55/?prefix=l 2023-09-22T21:28:27.587 DEBUG:boto:Data: 2023-09-22T21:28:27.587 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.587 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:27.588 DEBUG:boto:Port: 8001 2023-09-22T21:28:27.588 DEBUG:boto:Params: {} 2023-09-22T21:28:27.588 DEBUG:boto:Token: None 2023-09-22T21:28:27.588 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/ 2023-09-22T21:28:27.588 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg= 2023-09-22T21:28:27.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg=', 'Content-Length': '0'} 2023-09-22T21:28:27.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e64b8fe3bc7cc09e-00650e06fb-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.594 DEBUG:boto:b'ffzdqu-55l1000falsel2023-09-22T21:26:49.519Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:27.594 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:27.595 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:27.595 DEBUG:boto:path=/ffzdqu-55/?prefix=l 2023-09-22T21:28:27.595 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=l 2023-09-22T21:28:27.595 DEBUG:boto:Method: GET 2023-09-22T21:28:27.595 DEBUG:boto:Path: /ffzdqu-55/?prefix=l 2023-09-22T21:28:27.595 DEBUG:boto:Data: 2023-09-22T21:28:27.595 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.595 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:27.595 DEBUG:boto:Port: 8000 2023-09-22T21:28:27.595 DEBUG:boto:Params: {} 2023-09-22T21:28:27.596 DEBUG:boto:Token: None 2023-09-22T21:28:27.596 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/ 2023-09-22T21:28:27.596 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg= 2023-09-22T21:28:27.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg=', 'Content-Length': '0'} 2023-09-22T21:28:27.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006008a87fb0bb9e1a-00650e06fb-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.642 DEBUG:boto:b'ffzdqu-55l1000falsel2023-09-22T21:26:49.519Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:27.642 DEBUG:boto:path=/ffzdqu-55/l 2023-09-22T21:28:27.643 DEBUG:boto:auth_path=/ffzdqu-55/l 2023-09-22T21:28:27.643 DEBUG:boto:Method: HEAD 2023-09-22T21:28:27.643 DEBUG:boto:Path: /ffzdqu-55/l 2023-09-22T21:28:27.643 DEBUG:boto:Data: 2023-09-22T21:28:27.643 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.643 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:27.643 DEBUG:boto:Port: 8001 2023-09-22T21:28:27.643 DEBUG:boto:Params: {} 2023-09-22T21:28:27.644 DEBUG:boto:Token: None 2023-09-22T21:28:27.644 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/l 2023-09-22T21:28:27.644 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:U6/grIkFqnTRXBXMldZUNbolZ0Y= 2023-09-22T21:28:27.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:U6/grIkFqnTRXBXMldZUNbolZ0Y=', 'Content-Length': '0'} 2023-09-22T21:28:27.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a0f87561abfd5e4-00650e06fb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.647 DEBUG:boto:path=/ffzdqu-55/l 2023-09-22T21:28:27.647 DEBUG:boto:auth_path=/ffzdqu-55/l 2023-09-22T21:28:27.647 DEBUG:boto:Method: HEAD 2023-09-22T21:28:27.647 DEBUG:boto:Path: /ffzdqu-55/l 2023-09-22T21:28:27.647 DEBUG:boto:Data: 2023-09-22T21:28:27.647 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.647 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:27.647 DEBUG:boto:Port: 8000 2023-09-22T21:28:27.648 DEBUG:boto:Params: {} 2023-09-22T21:28:27.648 DEBUG:boto:Token: None 2023-09-22T21:28:27.648 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/l 2023-09-22T21:28:27.648 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:U6/grIkFqnTRXBXMldZUNbolZ0Y= 2023-09-22T21:28:27.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:U6/grIkFqnTRXBXMldZUNbolZ0Y=', 'Content-Length': '0'} 2023-09-22T21:28:27.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000382790e47a96d4b-00650e06fb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=l 2023-09-22T21:28:27.651 DEBUG:boto:path=/ffzdqu-55/l 2023-09-22T21:28:27.651 DEBUG:boto:auth_path=/ffzdqu-55/l 2023-09-22T21:28:27.651 DEBUG:boto:Method: GET 2023-09-22T21:28:27.651 DEBUG:boto:Path: /ffzdqu-55/l 2023-09-22T21:28:27.652 DEBUG:boto:Data: 2023-09-22T21:28:27.652 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.652 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:27.652 DEBUG:boto:Port: 8001 2023-09-22T21:28:27.652 DEBUG:boto:Params: {} 2023-09-22T21:28:27.652 DEBUG:boto:Token: None 2023-09-22T21:28:27.652 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/l 2023-09-22T21:28:27.652 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tztzIc8myY2xzrDjfB73RXPWjbo= 2023-09-22T21:28:27.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tztzIc8myY2xzrDjfB73RXPWjbo=', 'Content-Length': '0'} 2023-09-22T21:28:27.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd4a68384adadfc7-00650e06fb-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.702 DEBUG:boto:path=/ffzdqu-55/l 2023-09-22T21:28:27.702 DEBUG:boto:auth_path=/ffzdqu-55/l 2023-09-22T21:28:27.703 DEBUG:boto:Method: GET 2023-09-22T21:28:27.703 DEBUG:boto:Path: /ffzdqu-55/l 2023-09-22T21:28:27.703 DEBUG:boto:Data: 2023-09-22T21:28:27.703 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.703 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:27.703 DEBUG:boto:Port: 8000 2023-09-22T21:28:27.703 DEBUG:boto:Params: {} 2023-09-22T21:28:27.703 DEBUG:boto:Token: None 2023-09-22T21:28:27.703 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/l 2023-09-22T21:28:27.704 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tztzIc8myY2xzrDjfB73RXPWjbo= 2023-09-22T21:28:27.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tztzIc8myY2xzrDjfB73RXPWjbo=', 'Content-Length': '0'} 2023-09-22T21:28:27.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:26:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000303e236bbe16f45a-00650e06fb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.707 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:27.707 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:27.707 DEBUG:boto:path=/ffzdqu-55/?prefix=l 2023-09-22T21:28:27.707 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=l 2023-09-22T21:28:27.707 DEBUG:boto:Method: GET 2023-09-22T21:28:27.707 DEBUG:boto:Path: /ffzdqu-55/?prefix=l 2023-09-22T21:28:27.707 DEBUG:boto:Data: 2023-09-22T21:28:27.707 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.708 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:27.708 DEBUG:boto:Port: 8001 2023-09-22T21:28:27.708 DEBUG:boto:Params: {} 2023-09-22T21:28:27.708 DEBUG:boto:Token: None 2023-09-22T21:28:27.708 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/ 2023-09-22T21:28:27.708 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg= 2023-09-22T21:28:27.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg=', 'Content-Length': '0'} 2023-09-22T21:28:27.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a2be39b8629c089-00650e06fb-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.714 DEBUG:boto:b'ffzdqu-55l1000falsel2023-09-22T21:26:49.519Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:27.714 DEBUG:boto:path=/ffzdqu-55/ 2023-09-22T21:28:27.715 DEBUG:boto:auth_path=/ffzdqu-55/ 2023-09-22T21:28:27.715 DEBUG:boto:path=/ffzdqu-55/?prefix=l 2023-09-22T21:28:27.715 DEBUG:boto:auth_path=/ffzdqu-55/?prefix=l 2023-09-22T21:28:27.715 DEBUG:boto:Method: GET 2023-09-22T21:28:27.715 DEBUG:boto:Path: /ffzdqu-55/?prefix=l 2023-09-22T21:28:27.715 DEBUG:boto:Data: 2023-09-22T21:28:27.715 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.715 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:27.715 DEBUG:boto:Port: 8000 2023-09-22T21:28:27.716 DEBUG:boto:Params: {} 2023-09-22T21:28:27.716 DEBUG:boto:Token: None 2023-09-22T21:28:27.716 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-55/ 2023-09-22T21:28:27.716 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg= 2023-09-22T21:28:27.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pKwypaRZnPCmW3jChblY8B8cnvg=', 'Content-Length': '0'} 2023-09-22T21:28:27.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b24786aea29afdff-00650e06fb-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.719 DEBUG:boto:b'ffzdqu-55l1000falsel2023-09-22T21:26:49.519Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:27.720 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-55 zones={a2, a1} 2023-09-22T21:28:27.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'ffzdqu-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:28:27.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'ffzdqu-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:28:27.720 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket ffzdqu-55 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:28:27.761 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:28:27.761 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:28:27.822 INFO:teuthology.orchestra.run.smithi160.stdout:[{"op_id":"0#00000000002.35.6","op_tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.2897985913909593795","op":"write","object":"k","instance":"","state":"complete","index_ver":2,"timestamp":"2023-09-22T21:26:49.501795270Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20"}]},{"op_id":"1#00000000002.35.6","op_tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.4761451540378606584","op":"write","object":"j","instance":"","state":"complete","index_ver":2,"timestamp":"2023-09-22T21:26:49.449704329Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20"}]},{"op_id":"2#00000000002.33.6","op_tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.9708186387582757582","op":"write","object":"i","instance":"","state":"complete","index_ver":2,"timestamp":"2023-09-22T21:26:49.433693161Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20"}]},{"op_id":"2#00000000003.37.6","op_tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.7457340335614859553","op":"write","object":"l","instance":"","state":"complete","index_ver":3,"timestamp":"2023-09-22T21:26:49.519809512Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-55:f132bdb9-20a7-4073-a932-8f99da20691e.5729.20"}]}] 2023-09-22T21:28:27.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... ok 2023-09-22T21:28:27.824 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:27.824 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:27.825 DEBUG:boto:Method: PUT 2023-09-22T21:28:27.825 DEBUG:boto:Path: /ffzdqu-56/ 2023-09-22T21:28:27.825 DEBUG:boto:Data: 2023-09-22T21:28:27.825 DEBUG:boto:Headers: {} 2023-09-22T21:28:27.825 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:27.825 DEBUG:boto:Port: 8000 2023-09-22T21:28:27.825 DEBUG:boto:Params: {} 2023-09-22T21:28:27.826 DEBUG:boto:Token: None 2023-09-22T21:28:27.826 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:28:27 GMT /ffzdqu-56/ 2023-09-22T21:28:27.826 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7TBrIxnmJYYgYfJ/7ev0u/XsXII= 2023-09-22T21:28:27.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:27 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7TBrIxnmJYYgYfJ/7ev0u/XsXII=', 'Content-Length': '0'} 2023-09-22T21:28:27.844 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f1ccbee7aae36f93-00650e06fb-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:27.844 DEBUG:rgw_multi.tests:created bucket=ffzdqu-56 2023-09-22T21:28:27.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:28:27.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:28:27.845 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:28:27.896 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:28:27.896 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:28:27.938 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:28:27.956 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695418107.844535_439.1","last_update":"2023-09-22T21:28:27.844535Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:28:27.956 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695418107.844535_439.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:28:27.956 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:28:27.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:28:27.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:28:27.957 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:28:28.034 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:28:28.034 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:28:28.069 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:28:28.067+0000 7fee77c44a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:28:28.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:28:28.067+0000 7fee77c44a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:28:28.088 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695418107.844535_439.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:28:27.844535Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:28:28Z"} 2023-09-22T21:28:28.089 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695418107.844535_439.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:28:27.844535Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:28:28Z"} 2023-09-22T21:28:28.089 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695418107.844535_439.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:28:27.844535Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:28:28.089 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695418107.844535_439.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:28:28.089 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695418107.844535_439.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:28:28.089 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:28:28.089 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:28.089 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:28.090 DEBUG:boto:Method: HEAD 2023-09-22T21:28:28.090 DEBUG:boto:Path: /ffzdqu-56/ 2023-09-22T21:28:28.090 DEBUG:boto:Data: 2023-09-22T21:28:28.090 DEBUG:boto:Headers: {} 2023-09-22T21:28:28.091 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:28.091 DEBUG:boto:Port: 8000 2023-09-22T21:28:28.091 DEBUG:boto:Params: {} 2023-09-22T21:28:28.091 DEBUG:boto:Token: None 2023-09-22T21:28:28.091 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:28 GMT /ffzdqu-56/ 2023-09-22T21:28:28.091 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:k0Sg4LdJhRh1KGkBDiYd3hLbJnw= 2023-09-22T21:28:28.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:28 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:k0Sg4LdJhRh1KGkBDiYd3hLbJnw=', 'Content-Length': '0'} 2023-09-22T21:28:28.094 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f589ccff7dbfce59-00650e06fc-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:28.094 DEBUG:boto:path=/ffzdqu-56/a 2023-09-22T21:28:28.094 DEBUG:boto:auth_path=/ffzdqu-56/a 2023-09-22T21:28:28.094 DEBUG:boto:Method: PUT 2023-09-22T21:28:28.094 DEBUG:boto:Path: /ffzdqu-56/a 2023-09-22T21:28:28.095 DEBUG:boto:Data: 2023-09-22T21:28:28.095 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:28:28.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:28.095 DEBUG:boto:Port: 8000 2023-09-22T21:28:28.095 DEBUG:boto:Params: {} 2023-09-22T21:28:28.095 DEBUG:boto:Token: None 2023-09-22T21:28:28.095 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:28:28 GMT /ffzdqu-56/a 2023-09-22T21:28:28.095 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4OfOUF1RpgsD0/rGH2NCyXcl418= 2023-09-22T21:28:28.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:28:28 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4OfOUF1RpgsD0/rGH2NCyXcl418='} 2023-09-22T21:28:28.102 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000273be924ff9fff3c-00650e06fc-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:28.102 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:28.103 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:28.103 DEBUG:boto:Method: HEAD 2023-09-22T21:28:28.103 DEBUG:boto:Path: /ffzdqu-56/ 2023-09-22T21:28:28.103 DEBUG:boto:Data: 2023-09-22T21:28:28.103 DEBUG:boto:Headers: {} 2023-09-22T21:28:28.103 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:28.103 DEBUG:boto:Port: 8000 2023-09-22T21:28:28.103 DEBUG:boto:Params: {} 2023-09-22T21:28:28.103 DEBUG:boto:Token: None 2023-09-22T21:28:28.104 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:28 GMT /ffzdqu-56/ 2023-09-22T21:28:28.104 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:k0Sg4LdJhRh1KGkBDiYd3hLbJnw= 2023-09-22T21:28:28.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:28 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:k0Sg4LdJhRh1KGkBDiYd3hLbJnw=', 'Content-Length': '0'} 2023-09-22T21:28:28.106 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008de20de530769045-00650e06fc-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:28.106 DEBUG:boto:path=/ffzdqu-56/b 2023-09-22T21:28:28.106 DEBUG:boto:auth_path=/ffzdqu-56/b 2023-09-22T21:28:28.106 DEBUG:boto:Method: PUT 2023-09-22T21:28:28.107 DEBUG:boto:Path: /ffzdqu-56/b 2023-09-22T21:28:28.107 DEBUG:boto:Data: 2023-09-22T21:28:28.107 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:28:28.107 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:28.107 DEBUG:boto:Port: 8000 2023-09-22T21:28:28.107 DEBUG:boto:Params: {} 2023-09-22T21:28:28.107 DEBUG:boto:Token: None 2023-09-22T21:28:28.107 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:28:28 GMT /ffzdqu-56/b 2023-09-22T21:28:28.107 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:crMAkbO4zCINviiuTULdo1EP8yE= 2023-09-22T21:28:28.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:28:28 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:crMAkbO4zCINviiuTULdo1EP8yE='} 2023-09-22T21:28:28.150 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6bad65d6db69826-00650e06fc-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:28.151 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:28.151 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:28.151 DEBUG:boto:Method: HEAD 2023-09-22T21:28:28.151 DEBUG:boto:Path: /ffzdqu-56/ 2023-09-22T21:28:28.151 DEBUG:boto:Data: 2023-09-22T21:28:28.151 DEBUG:boto:Headers: {} 2023-09-22T21:28:28.151 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:28.151 DEBUG:boto:Port: 8000 2023-09-22T21:28:28.151 DEBUG:boto:Params: {} 2023-09-22T21:28:28.152 DEBUG:boto:Token: None 2023-09-22T21:28:28.152 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:28 GMT /ffzdqu-56/ 2023-09-22T21:28:28.152 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:k0Sg4LdJhRh1KGkBDiYd3hLbJnw= 2023-09-22T21:28:28.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:28 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:k0Sg4LdJhRh1KGkBDiYd3hLbJnw=', 'Content-Length': '0'} 2023-09-22T21:28:28.155 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a73d0cbe9531bc0-00650e06fc-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:28.155 DEBUG:boto:path=/ffzdqu-56/c 2023-09-22T21:28:28.155 DEBUG:boto:auth_path=/ffzdqu-56/c 2023-09-22T21:28:28.155 DEBUG:boto:Method: PUT 2023-09-22T21:28:28.156 DEBUG:boto:Path: /ffzdqu-56/c 2023-09-22T21:28:28.156 DEBUG:boto:Data: 2023-09-22T21:28:28.156 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:28:28.156 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:28.156 DEBUG:boto:Port: 8000 2023-09-22T21:28:28.156 DEBUG:boto:Params: {} 2023-09-22T21:28:28.156 DEBUG:boto:Token: None 2023-09-22T21:28:28.156 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:28:28 GMT /ffzdqu-56/c 2023-09-22T21:28:28.156 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GTb6vPvcmTGeEcqlM5UrLsBxdbs= 2023-09-22T21:28:28.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:28:28 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GTb6vPvcmTGeEcqlM5UrLsBxdbs='} 2023-09-22T21:28:28.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000344a4de523576c20-00650e06fc-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:28.166 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:28.166 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:28.166 DEBUG:boto:Method: HEAD 2023-09-22T21:28:28.166 DEBUG:boto:Path: /ffzdqu-56/ 2023-09-22T21:28:28.166 DEBUG:boto:Data: 2023-09-22T21:28:28.167 DEBUG:boto:Headers: {} 2023-09-22T21:28:28.167 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:28.167 DEBUG:boto:Port: 8000 2023-09-22T21:28:28.167 DEBUG:boto:Params: {} 2023-09-22T21:28:28.167 DEBUG:boto:Token: None 2023-09-22T21:28:28.167 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:28 GMT /ffzdqu-56/ 2023-09-22T21:28:28.167 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:k0Sg4LdJhRh1KGkBDiYd3hLbJnw= 2023-09-22T21:28:28.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:28 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:k0Sg4LdJhRh1KGkBDiYd3hLbJnw=', 'Content-Length': '0'} 2023-09-22T21:28:28.170 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001d605000c6cb588-00650e06fc-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:28.171 DEBUG:boto:path=/ffzdqu-56/d 2023-09-22T21:28:28.171 DEBUG:boto:auth_path=/ffzdqu-56/d 2023-09-22T21:28:28.171 DEBUG:boto:Method: PUT 2023-09-22T21:28:28.171 DEBUG:boto:Path: /ffzdqu-56/d 2023-09-22T21:28:28.171 DEBUG:boto:Data: 2023-09-22T21:28:28.171 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:28:28.171 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:28.171 DEBUG:boto:Port: 8000 2023-09-22T21:28:28.172 DEBUG:boto:Params: {} 2023-09-22T21:28:28.172 DEBUG:boto:Token: None 2023-09-22T21:28:28.172 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:28:28 GMT /ffzdqu-56/d 2023-09-22T21:28:28.172 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H6oqVZRKj+k7LN65XKdAz0DAHK4= 2023-09-22T21:28:28.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:28:28 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H6oqVZRKj+k7LN65XKdAz0DAHK4='} 2023-09-22T21:28:28.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b500ee47cc3eb46-00650e06fc-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:28.215 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-56 2023-09-22T21:28:28.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:28:28.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:28:28.215 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:28:28.264 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:28:28.264 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:28:28.308 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:28:28.307+0000 7f63ebc75a40 1 waiting to reach incremental sync.. 2023-09-22T21:28:33.309 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:28:33.307+0000 7f63ebc75a40 1 waiting to reach incremental sync.. 2023-09-22T21:28:38.310 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:28:38.306+0000 7f63ebc75a40 1 waiting to reach incremental sync.. 2023-09-22T21:28:43.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:28:43.306+0000 7f63ebc75a40 1 waiting to reach incremental sync.. 2023-09-22T21:28:48.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:28:48.310+0000 7f63ebc75a40 1 bucket sync caught up with source: 2023-09-22T21:28:48.314 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.67.6, , , , 00000000001.15.6, , , 00000000001.54.6, , 00000000001.26.6] 2023-09-22T21:28:48.315 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.67.6, , , , 00000000001.15.6, , , 00000000001.54.6, , 00000000001.26.6] 2023-09-22T21:28:48.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:28:48.310+0000 7f63ebc75a40 0 bucket checkpoint complete 2023-09-22T21:28:48.352 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-56 2023-09-22T21:28:48.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:28:48.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:28:48.352 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:28:48.401 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:28:48.401 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:28:48.447 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:28:48.442+0000 7f11b3a59a40 1 bucket sync caught up with source: 2023-09-22T21:28:48.447 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.180.6, , , , 00000000001.113.6, , , 00000000001.247.6, , 00000000001.225.6] 2023-09-22T21:28:48.447 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.180.6, , , , 00000000001.113.6, , , 00000000001.247.6, , 00000000001.225.6] 2023-09-22T21:28:48.447 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:28:48.442+0000 7f11b3a59a40 0 bucket checkpoint complete 2023-09-22T21:28:48.464 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-56 zones={a2, a1} 2023-09-22T21:28:48.464 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:48.464 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:48.464 DEBUG:boto:Method: HEAD 2023-09-22T21:28:48.464 DEBUG:boto:Path: /ffzdqu-56/ 2023-09-22T21:28:48.464 DEBUG:boto:Data: 2023-09-22T21:28:48.464 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.464 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:48.465 DEBUG:boto:Port: 8001 2023-09-22T21:28:48.465 DEBUG:boto:Params: {} 2023-09-22T21:28:48.465 DEBUG:boto:Token: None 2023-09-22T21:28:48.465 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/ 2023-09-22T21:28:48.465 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KRid0RqOYp5YHN7sEIV5oZtI540= 2023-09-22T21:28:48.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KRid0RqOYp5YHN7sEIV5oZtI540=', 'Content-Length': '0'} 2023-09-22T21:28:48.468 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082fd90bd1e5edf89-00650e0710-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.468 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:48.468 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:48.468 DEBUG:boto:Method: HEAD 2023-09-22T21:28:48.468 DEBUG:boto:Path: /ffzdqu-56/ 2023-09-22T21:28:48.468 DEBUG:boto:Data: 2023-09-22T21:28:48.468 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.468 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:48.469 DEBUG:boto:Port: 8000 2023-09-22T21:28:48.469 DEBUG:boto:Params: {} 2023-09-22T21:28:48.469 DEBUG:boto:Token: None 2023-09-22T21:28:48.469 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/ 2023-09-22T21:28:48.469 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KRid0RqOYp5YHN7sEIV5oZtI540= 2023-09-22T21:28:48.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KRid0RqOYp5YHN7sEIV5oZtI540=', 'Content-Length': '0'} 2023-09-22T21:28:48.471 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001731a9c31e3a34ce-00650e0710-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.472 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:28:48.472 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:48.472 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:48.472 DEBUG:boto:path=/ffzdqu-56/?versions&max-keys=999 2023-09-22T21:28:48.472 DEBUG:boto:auth_path=/ffzdqu-56/?versions&max-keys=999 2023-09-22T21:28:48.472 DEBUG:boto:Method: GET 2023-09-22T21:28:48.472 DEBUG:boto:Path: /ffzdqu-56/?versions&max-keys=999 2023-09-22T21:28:48.472 DEBUG:boto:Data: 2023-09-22T21:28:48.473 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.473 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:48.473 DEBUG:boto:Port: 8001 2023-09-22T21:28:48.473 DEBUG:boto:Params: {} 2023-09-22T21:28:48.473 DEBUG:boto:Token: None 2023-09-22T21:28:48.473 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/?versions 2023-09-22T21:28:48.473 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PT/3Da6vFHabUZjNwPTsGj8US2s= 2023-09-22T21:28:48.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PT/3Da6vFHabUZjNwPTsGj8US2s=', 'Content-Length': '0'} 2023-09-22T21:28:48.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088745eb1fe1accf6-00650e0710-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.477 DEBUG:boto:b'ffzdqu-56999falseanulltrue2023-09-22T21:28:28.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:28:28.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:28:28.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:28:28.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:48.478 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-22T21:28:48.478 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-22T21:28:48.478 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-22T21:28:48.478 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-22T21:28:48.479 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:28:48.479 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:48.479 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:48.479 DEBUG:boto:path=/ffzdqu-56/?versions&max-keys=999 2023-09-22T21:28:48.479 DEBUG:boto:auth_path=/ffzdqu-56/?versions&max-keys=999 2023-09-22T21:28:48.479 DEBUG:boto:Method: GET 2023-09-22T21:28:48.479 DEBUG:boto:Path: /ffzdqu-56/?versions&max-keys=999 2023-09-22T21:28:48.479 DEBUG:boto:Data: 2023-09-22T21:28:48.479 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.480 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:48.480 DEBUG:boto:Port: 8000 2023-09-22T21:28:48.480 DEBUG:boto:Params: {} 2023-09-22T21:28:48.480 DEBUG:boto:Token: None 2023-09-22T21:28:48.480 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/?versions 2023-09-22T21:28:48.480 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PT/3Da6vFHabUZjNwPTsGj8US2s= 2023-09-22T21:28:48.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PT/3Da6vFHabUZjNwPTsGj8US2s=', 'Content-Length': '0'} 2023-09-22T21:28:48.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ead59ac4aff50d50-00650e0710-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.483 DEBUG:boto:b'ffzdqu-56999falseanulltrue2023-09-22T21:28:28.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:28:28.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:28:28.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:28:28.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:48.484 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-22T21:28:48.484 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-22T21:28:48.484 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-22T21:28:48.484 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-22T21:28:48.484 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:48.484 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:48.484 DEBUG:boto:path=/ffzdqu-56/?versions&max-keys=999 2023-09-22T21:28:48.484 DEBUG:boto:auth_path=/ffzdqu-56/?versions&max-keys=999 2023-09-22T21:28:48.485 DEBUG:boto:Method: GET 2023-09-22T21:28:48.485 DEBUG:boto:Path: /ffzdqu-56/?versions&max-keys=999 2023-09-22T21:28:48.485 DEBUG:boto:Data: 2023-09-22T21:28:48.485 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.485 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:48.485 DEBUG:boto:Port: 8001 2023-09-22T21:28:48.485 DEBUG:boto:Params: {} 2023-09-22T21:28:48.485 DEBUG:boto:Token: None 2023-09-22T21:28:48.485 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/?versions 2023-09-22T21:28:48.486 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PT/3Da6vFHabUZjNwPTsGj8US2s= 2023-09-22T21:28:48.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PT/3Da6vFHabUZjNwPTsGj8US2s=', 'Content-Length': '0'} 2023-09-22T21:28:48.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053b06ab8779c9f45-00650e0710-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.488 DEBUG:boto:b'ffzdqu-56999falseanulltrue2023-09-22T21:28:28.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:28:28.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:28:28.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:28:28.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:48.489 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:48.489 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:48.489 DEBUG:boto:path=/ffzdqu-56/?versions&max-keys=999 2023-09-22T21:28:48.489 DEBUG:boto:auth_path=/ffzdqu-56/?versions&max-keys=999 2023-09-22T21:28:48.489 DEBUG:boto:Method: GET 2023-09-22T21:28:48.489 DEBUG:boto:Path: /ffzdqu-56/?versions&max-keys=999 2023-09-22T21:28:48.489 DEBUG:boto:Data: 2023-09-22T21:28:48.489 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.490 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:48.490 DEBUG:boto:Port: 8000 2023-09-22T21:28:48.490 DEBUG:boto:Params: {} 2023-09-22T21:28:48.490 DEBUG:boto:Token: None 2023-09-22T21:28:48.490 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/?versions 2023-09-22T21:28:48.490 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PT/3Da6vFHabUZjNwPTsGj8US2s= 2023-09-22T21:28:48.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PT/3Da6vFHabUZjNwPTsGj8US2s=', 'Content-Length': '0'} 2023-09-22T21:28:48.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001df10279fbef85a0-00650e0710-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.493 DEBUG:boto:b'ffzdqu-56999falseanulltrue2023-09-22T21:28:28.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:28:28.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:28:28.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:28:28.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:48.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:28:48.493 DEBUG:boto:path=/ffzdqu-56/a 2023-09-22T21:28:48.494 DEBUG:boto:auth_path=/ffzdqu-56/a 2023-09-22T21:28:48.494 DEBUG:boto:path=/ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.494 DEBUG:boto:auth_path=/ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.494 DEBUG:boto:Method: GET 2023-09-22T21:28:48.494 DEBUG:boto:Path: /ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.494 DEBUG:boto:Data: 2023-09-22T21:28:48.494 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.494 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:48.494 DEBUG:boto:Port: 8001 2023-09-22T21:28:48.495 DEBUG:boto:Params: {} 2023-09-22T21:28:48.495 DEBUG:boto:Token: None 2023-09-22T21:28:48.495 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.495 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ejMm9/4amjXMbQxC/7q3riJId3Q= 2023-09-22T21:28:48.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ejMm9/4amjXMbQxC/7q3riJId3Q=', 'Content-Length': '0'} 2023-09-22T21:28:48.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3cfa3ef3aebb75c-00650e0710-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.538 DEBUG:boto:path=/ffzdqu-56/a 2023-09-22T21:28:48.539 DEBUG:boto:auth_path=/ffzdqu-56/a 2023-09-22T21:28:48.539 DEBUG:boto:path=/ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.539 DEBUG:boto:auth_path=/ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.539 DEBUG:boto:Method: GET 2023-09-22T21:28:48.539 DEBUG:boto:Path: /ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.539 DEBUG:boto:Data: 2023-09-22T21:28:48.539 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.539 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:48.539 DEBUG:boto:Port: 8000 2023-09-22T21:28:48.539 DEBUG:boto:Params: {} 2023-09-22T21:28:48.540 DEBUG:boto:Token: None 2023-09-22T21:28:48.540 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.540 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ejMm9/4amjXMbQxC/7q3riJId3Q= 2023-09-22T21:28:48.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ejMm9/4amjXMbQxC/7q3riJId3Q=', 'Content-Length': '0'} 2023-09-22T21:28:48.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3495945dbe56553-00650e0710-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.542 DEBUG:boto:path=/ffzdqu-56/a 2023-09-22T21:28:48.542 DEBUG:boto:auth_path=/ffzdqu-56/a 2023-09-22T21:28:48.543 DEBUG:boto:path=/ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.543 DEBUG:boto:auth_path=/ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.543 DEBUG:boto:Method: HEAD 2023-09-22T21:28:48.543 DEBUG:boto:Path: /ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.543 DEBUG:boto:Data: 2023-09-22T21:28:48.543 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.543 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:48.543 DEBUG:boto:Port: 8001 2023-09-22T21:28:48.543 DEBUG:boto:Params: {} 2023-09-22T21:28:48.544 DEBUG:boto:Token: None 2023-09-22T21:28:48.544 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.544 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Rsn5KHQYvlQGG99TB1gGlsuAb98= 2023-09-22T21:28:48.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rsn5KHQYvlQGG99TB1gGlsuAb98=', 'Content-Length': '0'} 2023-09-22T21:28:48.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8c307ca3560b156-00650e0710-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.546 DEBUG:boto:path=/ffzdqu-56/a 2023-09-22T21:28:48.546 DEBUG:boto:auth_path=/ffzdqu-56/a 2023-09-22T21:28:48.546 DEBUG:boto:path=/ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.546 DEBUG:boto:auth_path=/ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.546 DEBUG:boto:Method: HEAD 2023-09-22T21:28:48.546 DEBUG:boto:Path: /ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.546 DEBUG:boto:Data: 2023-09-22T21:28:48.547 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.547 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:48.547 DEBUG:boto:Port: 8000 2023-09-22T21:28:48.547 DEBUG:boto:Params: {} 2023-09-22T21:28:48.547 DEBUG:boto:Token: None 2023-09-22T21:28:48.547 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.547 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Rsn5KHQYvlQGG99TB1gGlsuAb98= 2023-09-22T21:28:48.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rsn5KHQYvlQGG99TB1gGlsuAb98=', 'Content-Length': '0'} 2023-09-22T21:28:48.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000873a6b927bf16619-00650e0710-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:28:48.549 DEBUG:boto:path=/ffzdqu-56/a 2023-09-22T21:28:48.550 DEBUG:boto:auth_path=/ffzdqu-56/a 2023-09-22T21:28:48.550 DEBUG:boto:path=/ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.550 DEBUG:boto:auth_path=/ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.550 DEBUG:boto:Method: GET 2023-09-22T21:28:48.550 DEBUG:boto:Path: /ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.550 DEBUG:boto:Data: 2023-09-22T21:28:48.550 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.550 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:48.550 DEBUG:boto:Port: 8001 2023-09-22T21:28:48.551 DEBUG:boto:Params: {} 2023-09-22T21:28:48.551 DEBUG:boto:Token: None 2023-09-22T21:28:48.551 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.551 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ejMm9/4amjXMbQxC/7q3riJId3Q= 2023-09-22T21:28:48.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ejMm9/4amjXMbQxC/7q3riJId3Q=', 'Content-Length': '0'} 2023-09-22T21:28:48.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007145bf576f14d239-00650e0710-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.553 DEBUG:boto:path=/ffzdqu-56/a 2023-09-22T21:28:48.553 DEBUG:boto:auth_path=/ffzdqu-56/a 2023-09-22T21:28:48.553 DEBUG:boto:path=/ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.553 DEBUG:boto:auth_path=/ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.554 DEBUG:boto:Method: GET 2023-09-22T21:28:48.554 DEBUG:boto:Path: /ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.554 DEBUG:boto:Data: 2023-09-22T21:28:48.554 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.554 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:48.554 DEBUG:boto:Port: 8000 2023-09-22T21:28:48.554 DEBUG:boto:Params: {} 2023-09-22T21:28:48.554 DEBUG:boto:Token: None 2023-09-22T21:28:48.554 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/a?versionId=null 2023-09-22T21:28:48.555 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ejMm9/4amjXMbQxC/7q3riJId3Q= 2023-09-22T21:28:48.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ejMm9/4amjXMbQxC/7q3riJId3Q=', 'Content-Length': '0'} 2023-09-22T21:28:48.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f7cf9db2dfd04e7-00650e0710-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.558 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:48.558 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:48.558 DEBUG:boto:path=/ffzdqu-56/?prefix=a 2023-09-22T21:28:48.558 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=a 2023-09-22T21:28:48.558 DEBUG:boto:Method: GET 2023-09-22T21:28:48.558 DEBUG:boto:Path: /ffzdqu-56/?prefix=a 2023-09-22T21:28:48.558 DEBUG:boto:Data: 2023-09-22T21:28:48.558 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.559 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:48.559 DEBUG:boto:Port: 8001 2023-09-22T21:28:48.559 DEBUG:boto:Params: {} 2023-09-22T21:28:48.559 DEBUG:boto:Token: None 2023-09-22T21:28:48.559 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/ 2023-09-22T21:28:48.559 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cApLcjU+kBL/4Q0RCVQqm6FKjWM= 2023-09-22T21:28:48.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cApLcjU+kBL/4Q0RCVQqm6FKjWM=', 'Content-Length': '0'} 2023-09-22T21:28:48.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b02ae148ddb03bfe-00650e0710-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.563 DEBUG:boto:b'ffzdqu-56a1000falsea2023-09-22T21:28:28.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:48.563 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:48.563 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:48.564 DEBUG:boto:path=/ffzdqu-56/?prefix=a 2023-09-22T21:28:48.564 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=a 2023-09-22T21:28:48.564 DEBUG:boto:Method: GET 2023-09-22T21:28:48.564 DEBUG:boto:Path: /ffzdqu-56/?prefix=a 2023-09-22T21:28:48.564 DEBUG:boto:Data: 2023-09-22T21:28:48.564 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.564 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:48.564 DEBUG:boto:Port: 8000 2023-09-22T21:28:48.564 DEBUG:boto:Params: {} 2023-09-22T21:28:48.564 DEBUG:boto:Token: None 2023-09-22T21:28:48.565 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/ 2023-09-22T21:28:48.565 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cApLcjU+kBL/4Q0RCVQqm6FKjWM= 2023-09-22T21:28:48.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cApLcjU+kBL/4Q0RCVQqm6FKjWM=', 'Content-Length': '0'} 2023-09-22T21:28:48.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c99eba7881533e7-00650e0710-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.569 DEBUG:boto:b'ffzdqu-56a1000falsea2023-09-22T21:28:28.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:48.569 DEBUG:boto:path=/ffzdqu-56/a 2023-09-22T21:28:48.569 DEBUG:boto:auth_path=/ffzdqu-56/a 2023-09-22T21:28:48.569 DEBUG:boto:Method: HEAD 2023-09-22T21:28:48.569 DEBUG:boto:Path: /ffzdqu-56/a 2023-09-22T21:28:48.570 DEBUG:boto:Data: 2023-09-22T21:28:48.570 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.570 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:48.570 DEBUG:boto:Port: 8001 2023-09-22T21:28:48.570 DEBUG:boto:Params: {} 2023-09-22T21:28:48.570 DEBUG:boto:Token: None 2023-09-22T21:28:48.570 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/a 2023-09-22T21:28:48.570 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zOUIg1+UqIO64aqA1OccIVrwh0I= 2023-09-22T21:28:48.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zOUIg1+UqIO64aqA1OccIVrwh0I=', 'Content-Length': '0'} 2023-09-22T21:28:48.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d668797b7060bc9d-00650e0710-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.572 DEBUG:boto:path=/ffzdqu-56/a 2023-09-22T21:28:48.572 DEBUG:boto:auth_path=/ffzdqu-56/a 2023-09-22T21:28:48.572 DEBUG:boto:Method: HEAD 2023-09-22T21:28:48.573 DEBUG:boto:Path: /ffzdqu-56/a 2023-09-22T21:28:48.573 DEBUG:boto:Data: 2023-09-22T21:28:48.573 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.573 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:48.573 DEBUG:boto:Port: 8000 2023-09-22T21:28:48.573 DEBUG:boto:Params: {} 2023-09-22T21:28:48.573 DEBUG:boto:Token: None 2023-09-22T21:28:48.573 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/a 2023-09-22T21:28:48.573 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zOUIg1+UqIO64aqA1OccIVrwh0I= 2023-09-22T21:28:48.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zOUIg1+UqIO64aqA1OccIVrwh0I=', 'Content-Length': '0'} 2023-09-22T21:28:48.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dc54c527118a44f-00650e0710-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:28:48.576 DEBUG:boto:path=/ffzdqu-56/a 2023-09-22T21:28:48.576 DEBUG:boto:auth_path=/ffzdqu-56/a 2023-09-22T21:28:48.576 DEBUG:boto:Method: GET 2023-09-22T21:28:48.576 DEBUG:boto:Path: /ffzdqu-56/a 2023-09-22T21:28:48.576 DEBUG:boto:Data: 2023-09-22T21:28:48.577 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.577 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:48.577 DEBUG:boto:Port: 8001 2023-09-22T21:28:48.577 DEBUG:boto:Params: {} 2023-09-22T21:28:48.577 DEBUG:boto:Token: None 2023-09-22T21:28:48.577 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/a 2023-09-22T21:28:48.577 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cx2xU4izmgyts1RmBQaIA1CogEs= 2023-09-22T21:28:48.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cx2xU4izmgyts1RmBQaIA1CogEs=', 'Content-Length': '0'} 2023-09-22T21:28:48.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051b33d61d63a3eba-00650e0710-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.622 DEBUG:boto:path=/ffzdqu-56/a 2023-09-22T21:28:48.622 DEBUG:boto:auth_path=/ffzdqu-56/a 2023-09-22T21:28:48.623 DEBUG:boto:Method: GET 2023-09-22T21:28:48.623 DEBUG:boto:Path: /ffzdqu-56/a 2023-09-22T21:28:48.623 DEBUG:boto:Data: 2023-09-22T21:28:48.623 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.623 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:48.623 DEBUG:boto:Port: 8000 2023-09-22T21:28:48.623 DEBUG:boto:Params: {} 2023-09-22T21:28:48.623 DEBUG:boto:Token: None 2023-09-22T21:28:48.623 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/a 2023-09-22T21:28:48.624 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cx2xU4izmgyts1RmBQaIA1CogEs= 2023-09-22T21:28:48.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cx2xU4izmgyts1RmBQaIA1CogEs=', 'Content-Length': '0'} 2023-09-22T21:28:48.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6ad773672281169-00650e0710-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.627 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:48.627 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:48.627 DEBUG:boto:path=/ffzdqu-56/?prefix=a 2023-09-22T21:28:48.627 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=a 2023-09-22T21:28:48.627 DEBUG:boto:Method: GET 2023-09-22T21:28:48.627 DEBUG:boto:Path: /ffzdqu-56/?prefix=a 2023-09-22T21:28:48.627 DEBUG:boto:Data: 2023-09-22T21:28:48.627 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.627 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:48.628 DEBUG:boto:Port: 8001 2023-09-22T21:28:48.628 DEBUG:boto:Params: {} 2023-09-22T21:28:48.628 DEBUG:boto:Token: None 2023-09-22T21:28:48.628 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/ 2023-09-22T21:28:48.628 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cApLcjU+kBL/4Q0RCVQqm6FKjWM= 2023-09-22T21:28:48.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cApLcjU+kBL/4Q0RCVQqm6FKjWM=', 'Content-Length': '0'} 2023-09-22T21:28:48.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffac16c4f7693c55-00650e0710-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.674 DEBUG:boto:b'ffzdqu-56a1000falsea2023-09-22T21:28:28.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:48.675 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:48.675 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:48.675 DEBUG:boto:path=/ffzdqu-56/?prefix=a 2023-09-22T21:28:48.675 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=a 2023-09-22T21:28:48.675 DEBUG:boto:Method: GET 2023-09-22T21:28:48.675 DEBUG:boto:Path: /ffzdqu-56/?prefix=a 2023-09-22T21:28:48.675 DEBUG:boto:Data: 2023-09-22T21:28:48.675 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:48.676 DEBUG:boto:Port: 8000 2023-09-22T21:28:48.676 DEBUG:boto:Params: {} 2023-09-22T21:28:48.676 DEBUG:boto:Token: None 2023-09-22T21:28:48.676 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/ 2023-09-22T21:28:48.676 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cApLcjU+kBL/4Q0RCVQqm6FKjWM= 2023-09-22T21:28:48.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cApLcjU+kBL/4Q0RCVQqm6FKjWM=', 'Content-Length': '0'} 2023-09-22T21:28:48.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b6a9ebe3c0f2567-00650e0710-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.682 DEBUG:boto:b'ffzdqu-56a1000falsea2023-09-22T21:28:28.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:48.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:28:48.682 DEBUG:boto:path=/ffzdqu-56/b 2023-09-22T21:28:48.682 DEBUG:boto:auth_path=/ffzdqu-56/b 2023-09-22T21:28:48.682 DEBUG:boto:path=/ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.683 DEBUG:boto:auth_path=/ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.683 DEBUG:boto:Method: GET 2023-09-22T21:28:48.683 DEBUG:boto:Path: /ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.683 DEBUG:boto:Data: 2023-09-22T21:28:48.683 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.683 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:48.683 DEBUG:boto:Port: 8001 2023-09-22T21:28:48.683 DEBUG:boto:Params: {} 2023-09-22T21:28:48.683 DEBUG:boto:Token: None 2023-09-22T21:28:48.684 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.684 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VjysSHDMMhhli97u4yl11cP8Up0= 2023-09-22T21:28:48.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VjysSHDMMhhli97u4yl11cP8Up0=', 'Content-Length': '0'} 2023-09-22T21:28:48.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e843875fb1cec07b-00650e0710-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.687 DEBUG:boto:path=/ffzdqu-56/b 2023-09-22T21:28:48.687 DEBUG:boto:auth_path=/ffzdqu-56/b 2023-09-22T21:28:48.687 DEBUG:boto:path=/ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.687 DEBUG:boto:auth_path=/ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.688 DEBUG:boto:Method: GET 2023-09-22T21:28:48.688 DEBUG:boto:Path: /ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.688 DEBUG:boto:Data: 2023-09-22T21:28:48.688 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.688 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:48.688 DEBUG:boto:Port: 8000 2023-09-22T21:28:48.688 DEBUG:boto:Params: {} 2023-09-22T21:28:48.688 DEBUG:boto:Token: None 2023-09-22T21:28:48.688 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.689 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VjysSHDMMhhli97u4yl11cP8Up0= 2023-09-22T21:28:48.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VjysSHDMMhhli97u4yl11cP8Up0=', 'Content-Length': '0'} 2023-09-22T21:28:48.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a5a614f7ded84d0-00650e0710-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.692 DEBUG:boto:path=/ffzdqu-56/b 2023-09-22T21:28:48.692 DEBUG:boto:auth_path=/ffzdqu-56/b 2023-09-22T21:28:48.692 DEBUG:boto:path=/ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.692 DEBUG:boto:auth_path=/ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.692 DEBUG:boto:Method: HEAD 2023-09-22T21:28:48.692 DEBUG:boto:Path: /ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.692 DEBUG:boto:Data: 2023-09-22T21:28:48.692 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.692 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:48.693 DEBUG:boto:Port: 8001 2023-09-22T21:28:48.693 DEBUG:boto:Params: {} 2023-09-22T21:28:48.693 DEBUG:boto:Token: None 2023-09-22T21:28:48.693 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.693 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uVYWi2t0P1QkvgkK6L1px/6OzuU= 2023-09-22T21:28:48.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uVYWi2t0P1QkvgkK6L1px/6OzuU=', 'Content-Length': '0'} 2023-09-22T21:28:48.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000504e58ebce27d130-00650e0710-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.696 DEBUG:boto:path=/ffzdqu-56/b 2023-09-22T21:28:48.696 DEBUG:boto:auth_path=/ffzdqu-56/b 2023-09-22T21:28:48.696 DEBUG:boto:path=/ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.696 DEBUG:boto:auth_path=/ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.697 DEBUG:boto:Method: HEAD 2023-09-22T21:28:48.697 DEBUG:boto:Path: /ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.697 DEBUG:boto:Data: 2023-09-22T21:28:48.697 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.697 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:48.697 DEBUG:boto:Port: 8000 2023-09-22T21:28:48.697 DEBUG:boto:Params: {} 2023-09-22T21:28:48.697 DEBUG:boto:Token: None 2023-09-22T21:28:48.697 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.698 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uVYWi2t0P1QkvgkK6L1px/6OzuU= 2023-09-22T21:28:48.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uVYWi2t0P1QkvgkK6L1px/6OzuU=', 'Content-Length': '0'} 2023-09-22T21:28:48.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001536bf23a5a2c22b-00650e0710-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:28:48.701 DEBUG:boto:path=/ffzdqu-56/b 2023-09-22T21:28:48.701 DEBUG:boto:auth_path=/ffzdqu-56/b 2023-09-22T21:28:48.701 DEBUG:boto:path=/ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.701 DEBUG:boto:auth_path=/ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.701 DEBUG:boto:Method: GET 2023-09-22T21:28:48.701 DEBUG:boto:Path: /ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.701 DEBUG:boto:Data: 2023-09-22T21:28:48.701 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.702 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:48.702 DEBUG:boto:Port: 8001 2023-09-22T21:28:48.702 DEBUG:boto:Params: {} 2023-09-22T21:28:48.702 DEBUG:boto:Token: None 2023-09-22T21:28:48.702 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.702 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VjysSHDMMhhli97u4yl11cP8Up0= 2023-09-22T21:28:48.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VjysSHDMMhhli97u4yl11cP8Up0=', 'Content-Length': '0'} 2023-09-22T21:28:48.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ac7ea0a6a8f606e-00650e0710-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.750 DEBUG:boto:path=/ffzdqu-56/b 2023-09-22T21:28:48.751 DEBUG:boto:auth_path=/ffzdqu-56/b 2023-09-22T21:28:48.751 DEBUG:boto:path=/ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.751 DEBUG:boto:auth_path=/ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.751 DEBUG:boto:Method: GET 2023-09-22T21:28:48.751 DEBUG:boto:Path: /ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.751 DEBUG:boto:Data: 2023-09-22T21:28:48.751 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.751 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:48.752 DEBUG:boto:Port: 8000 2023-09-22T21:28:48.752 DEBUG:boto:Params: {} 2023-09-22T21:28:48.752 DEBUG:boto:Token: None 2023-09-22T21:28:48.752 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/b?versionId=null 2023-09-22T21:28:48.752 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VjysSHDMMhhli97u4yl11cP8Up0= 2023-09-22T21:28:48.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VjysSHDMMhhli97u4yl11cP8Up0=', 'Content-Length': '0'} 2023-09-22T21:28:48.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c144fbe82aeed9c-00650e0710-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.755 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:48.755 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:48.755 DEBUG:boto:path=/ffzdqu-56/?prefix=b 2023-09-22T21:28:48.756 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=b 2023-09-22T21:28:48.756 DEBUG:boto:Method: GET 2023-09-22T21:28:48.756 DEBUG:boto:Path: /ffzdqu-56/?prefix=b 2023-09-22T21:28:48.756 DEBUG:boto:Data: 2023-09-22T21:28:48.756 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.756 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:48.756 DEBUG:boto:Port: 8001 2023-09-22T21:28:48.756 DEBUG:boto:Params: {} 2023-09-22T21:28:48.756 DEBUG:boto:Token: None 2023-09-22T21:28:48.757 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/ 2023-09-22T21:28:48.757 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cApLcjU+kBL/4Q0RCVQqm6FKjWM= 2023-09-22T21:28:48.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cApLcjU+kBL/4Q0RCVQqm6FKjWM=', 'Content-Length': '0'} 2023-09-22T21:28:48.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003aecf21f4864db58-00650e0710-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.762 DEBUG:boto:b'ffzdqu-56b1000falseb2023-09-22T21:28:28.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:48.762 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:48.762 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:48.762 DEBUG:boto:path=/ffzdqu-56/?prefix=b 2023-09-22T21:28:48.762 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=b 2023-09-22T21:28:48.762 DEBUG:boto:Method: GET 2023-09-22T21:28:48.763 DEBUG:boto:Path: /ffzdqu-56/?prefix=b 2023-09-22T21:28:48.763 DEBUG:boto:Data: 2023-09-22T21:28:48.763 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.763 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:48.763 DEBUG:boto:Port: 8000 2023-09-22T21:28:48.763 DEBUG:boto:Params: {} 2023-09-22T21:28:48.763 DEBUG:boto:Token: None 2023-09-22T21:28:48.763 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/ 2023-09-22T21:28:48.764 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cApLcjU+kBL/4Q0RCVQqm6FKjWM= 2023-09-22T21:28:48.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cApLcjU+kBL/4Q0RCVQqm6FKjWM=', 'Content-Length': '0'} 2023-09-22T21:28:48.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002812f46d488f1ae8-00650e0710-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.810 DEBUG:boto:b'ffzdqu-56b1000falseb2023-09-22T21:28:28.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:48.811 DEBUG:boto:path=/ffzdqu-56/b 2023-09-22T21:28:48.811 DEBUG:boto:auth_path=/ffzdqu-56/b 2023-09-22T21:28:48.811 DEBUG:boto:Method: HEAD 2023-09-22T21:28:48.811 DEBUG:boto:Path: /ffzdqu-56/b 2023-09-22T21:28:48.811 DEBUG:boto:Data: 2023-09-22T21:28:48.811 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.812 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:48.812 DEBUG:boto:Port: 8001 2023-09-22T21:28:48.812 DEBUG:boto:Params: {} 2023-09-22T21:28:48.812 DEBUG:boto:Token: None 2023-09-22T21:28:48.812 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/b 2023-09-22T21:28:48.812 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5Av2NcGvzBOKFKC+hdZQgR5WFOk= 2023-09-22T21:28:48.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5Av2NcGvzBOKFKC+hdZQgR5WFOk=', 'Content-Length': '0'} 2023-09-22T21:28:48.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003092ae211981eda8-00650e0710-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.816 DEBUG:boto:path=/ffzdqu-56/b 2023-09-22T21:28:48.816 DEBUG:boto:auth_path=/ffzdqu-56/b 2023-09-22T21:28:48.816 DEBUG:boto:Method: HEAD 2023-09-22T21:28:48.816 DEBUG:boto:Path: /ffzdqu-56/b 2023-09-22T21:28:48.816 DEBUG:boto:Data: 2023-09-22T21:28:48.816 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.816 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:48.816 DEBUG:boto:Port: 8000 2023-09-22T21:28:48.816 DEBUG:boto:Params: {} 2023-09-22T21:28:48.817 DEBUG:boto:Token: None 2023-09-22T21:28:48.817 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/b 2023-09-22T21:28:48.817 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5Av2NcGvzBOKFKC+hdZQgR5WFOk= 2023-09-22T21:28:48.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5Av2NcGvzBOKFKC+hdZQgR5WFOk=', 'Content-Length': '0'} 2023-09-22T21:28:48.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc700d89cf54565e-00650e0710-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:28:48.820 DEBUG:boto:path=/ffzdqu-56/b 2023-09-22T21:28:48.820 DEBUG:boto:auth_path=/ffzdqu-56/b 2023-09-22T21:28:48.820 DEBUG:boto:Method: GET 2023-09-22T21:28:48.820 DEBUG:boto:Path: /ffzdqu-56/b 2023-09-22T21:28:48.820 DEBUG:boto:Data: 2023-09-22T21:28:48.820 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.821 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:48.821 DEBUG:boto:Port: 8001 2023-09-22T21:28:48.821 DEBUG:boto:Params: {} 2023-09-22T21:28:48.821 DEBUG:boto:Token: None 2023-09-22T21:28:48.821 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/b 2023-09-22T21:28:48.821 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lC25T4rGtuq2LPDJvuoj9regqyI= 2023-09-22T21:28:48.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lC25T4rGtuq2LPDJvuoj9regqyI=', 'Content-Length': '0'} 2023-09-22T21:28:48.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000946b90fdf24c4734-00650e0710-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.866 DEBUG:boto:path=/ffzdqu-56/b 2023-09-22T21:28:48.866 DEBUG:boto:auth_path=/ffzdqu-56/b 2023-09-22T21:28:48.866 DEBUG:boto:Method: GET 2023-09-22T21:28:48.867 DEBUG:boto:Path: /ffzdqu-56/b 2023-09-22T21:28:48.867 DEBUG:boto:Data: 2023-09-22T21:28:48.867 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.867 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:48.867 DEBUG:boto:Port: 8000 2023-09-22T21:28:48.867 DEBUG:boto:Params: {} 2023-09-22T21:28:48.867 DEBUG:boto:Token: None 2023-09-22T21:28:48.867 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/b 2023-09-22T21:28:48.868 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lC25T4rGtuq2LPDJvuoj9regqyI= 2023-09-22T21:28:48.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lC25T4rGtuq2LPDJvuoj9regqyI=', 'Content-Length': '0'} 2023-09-22T21:28:48.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a00f3c049020582b-00650e0710-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.870 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:48.871 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:48.871 DEBUG:boto:path=/ffzdqu-56/?prefix=b 2023-09-22T21:28:48.871 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=b 2023-09-22T21:28:48.871 DEBUG:boto:Method: GET 2023-09-22T21:28:48.871 DEBUG:boto:Path: /ffzdqu-56/?prefix=b 2023-09-22T21:28:48.871 DEBUG:boto:Data: 2023-09-22T21:28:48.871 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.871 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:48.871 DEBUG:boto:Port: 8001 2023-09-22T21:28:48.872 DEBUG:boto:Params: {} 2023-09-22T21:28:48.872 DEBUG:boto:Token: None 2023-09-22T21:28:48.872 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/ 2023-09-22T21:28:48.872 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cApLcjU+kBL/4Q0RCVQqm6FKjWM= 2023-09-22T21:28:48.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cApLcjU+kBL/4Q0RCVQqm6FKjWM=', 'Content-Length': '0'} 2023-09-22T21:28:48.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ff06edb843c6b12-00650e0710-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.877 DEBUG:boto:b'ffzdqu-56b1000falseb2023-09-22T21:28:28.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:48.877 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:48.877 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:48.878 DEBUG:boto:path=/ffzdqu-56/?prefix=b 2023-09-22T21:28:48.878 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=b 2023-09-22T21:28:48.878 DEBUG:boto:Method: GET 2023-09-22T21:28:48.878 DEBUG:boto:Path: /ffzdqu-56/?prefix=b 2023-09-22T21:28:48.878 DEBUG:boto:Data: 2023-09-22T21:28:48.878 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.878 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:48.878 DEBUG:boto:Port: 8000 2023-09-22T21:28:48.878 DEBUG:boto:Params: {} 2023-09-22T21:28:48.879 DEBUG:boto:Token: None 2023-09-22T21:28:48.879 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/ 2023-09-22T21:28:48.879 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cApLcjU+kBL/4Q0RCVQqm6FKjWM= 2023-09-22T21:28:48.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cApLcjU+kBL/4Q0RCVQqm6FKjWM=', 'Content-Length': '0'} 2023-09-22T21:28:48.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bea4a2715ad1b35-00650e0710-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.886 DEBUG:boto:b'ffzdqu-56b1000falseb2023-09-22T21:28:28.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:48.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:28:48.886 DEBUG:boto:path=/ffzdqu-56/c 2023-09-22T21:28:48.887 DEBUG:boto:auth_path=/ffzdqu-56/c 2023-09-22T21:28:48.887 DEBUG:boto:path=/ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.887 DEBUG:boto:auth_path=/ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.887 DEBUG:boto:Method: GET 2023-09-22T21:28:48.887 DEBUG:boto:Path: /ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.887 DEBUG:boto:Data: 2023-09-22T21:28:48.887 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.887 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:48.887 DEBUG:boto:Port: 8001 2023-09-22T21:28:48.887 DEBUG:boto:Params: {} 2023-09-22T21:28:48.888 DEBUG:boto:Token: None 2023-09-22T21:28:48.888 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.888 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g6fA2WRGOTmLISon0i+/XWeqImE= 2023-09-22T21:28:48.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g6fA2WRGOTmLISon0i+/XWeqImE=', 'Content-Length': '0'} 2023-09-22T21:28:48.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d335e986a418bd83-00650e0710-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.891 DEBUG:boto:path=/ffzdqu-56/c 2023-09-22T21:28:48.891 DEBUG:boto:auth_path=/ffzdqu-56/c 2023-09-22T21:28:48.891 DEBUG:boto:path=/ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.891 DEBUG:boto:auth_path=/ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.891 DEBUG:boto:Method: GET 2023-09-22T21:28:48.892 DEBUG:boto:Path: /ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.892 DEBUG:boto:Data: 2023-09-22T21:28:48.892 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.892 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:48.892 DEBUG:boto:Port: 8000 2023-09-22T21:28:48.892 DEBUG:boto:Params: {} 2023-09-22T21:28:48.892 DEBUG:boto:Token: None 2023-09-22T21:28:48.892 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.893 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g6fA2WRGOTmLISon0i+/XWeqImE= 2023-09-22T21:28:48.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g6fA2WRGOTmLISon0i+/XWeqImE=', 'Content-Length': '0'} 2023-09-22T21:28:48.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b43fda39c72388d8-00650e0710-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.938 DEBUG:boto:path=/ffzdqu-56/c 2023-09-22T21:28:48.939 DEBUG:boto:auth_path=/ffzdqu-56/c 2023-09-22T21:28:48.939 DEBUG:boto:path=/ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.939 DEBUG:boto:auth_path=/ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.939 DEBUG:boto:Method: HEAD 2023-09-22T21:28:48.939 DEBUG:boto:Path: /ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.939 DEBUG:boto:Data: 2023-09-22T21:28:48.939 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.939 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:48.939 DEBUG:boto:Port: 8001 2023-09-22T21:28:48.940 DEBUG:boto:Params: {} 2023-09-22T21:28:48.940 DEBUG:boto:Token: None 2023-09-22T21:28:48.940 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.940 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zlkCM7jr/JTUCzgGohwE2hB4lWM= 2023-09-22T21:28:48.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zlkCM7jr/JTUCzgGohwE2hB4lWM=', 'Content-Length': '0'} 2023-09-22T21:28:48.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ffbd2c87ba9e6b0-00650e0710-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.943 DEBUG:boto:path=/ffzdqu-56/c 2023-09-22T21:28:48.943 DEBUG:boto:auth_path=/ffzdqu-56/c 2023-09-22T21:28:48.943 DEBUG:boto:path=/ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.944 DEBUG:boto:auth_path=/ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.944 DEBUG:boto:Method: HEAD 2023-09-22T21:28:48.944 DEBUG:boto:Path: /ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.944 DEBUG:boto:Data: 2023-09-22T21:28:48.944 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.944 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:48.944 DEBUG:boto:Port: 8000 2023-09-22T21:28:48.944 DEBUG:boto:Params: {} 2023-09-22T21:28:48.944 DEBUG:boto:Token: None 2023-09-22T21:28:48.945 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.945 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zlkCM7jr/JTUCzgGohwE2hB4lWM= 2023-09-22T21:28:48.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zlkCM7jr/JTUCzgGohwE2hB4lWM=', 'Content-Length': '0'} 2023-09-22T21:28:48.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009de52f146b5986ed-00650e0710-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:28:48.948 DEBUG:boto:path=/ffzdqu-56/c 2023-09-22T21:28:48.949 DEBUG:boto:auth_path=/ffzdqu-56/c 2023-09-22T21:28:48.949 DEBUG:boto:path=/ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.949 DEBUG:boto:auth_path=/ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.949 DEBUG:boto:Method: GET 2023-09-22T21:28:48.949 DEBUG:boto:Path: /ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.950 DEBUG:boto:Data: 2023-09-22T21:28:48.950 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.950 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:48.950 DEBUG:boto:Port: 8001 2023-09-22T21:28:48.950 DEBUG:boto:Params: {} 2023-09-22T21:28:48.951 DEBUG:boto:Token: None 2023-09-22T21:28:48.951 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.951 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g6fA2WRGOTmLISon0i+/XWeqImE= 2023-09-22T21:28:48.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g6fA2WRGOTmLISon0i+/XWeqImE=', 'Content-Length': '0'} 2023-09-22T21:28:48.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fedbae0e15a9108-00650e0710-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.994 DEBUG:boto:path=/ffzdqu-56/c 2023-09-22T21:28:48.995 DEBUG:boto:auth_path=/ffzdqu-56/c 2023-09-22T21:28:48.995 DEBUG:boto:path=/ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.995 DEBUG:boto:auth_path=/ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.995 DEBUG:boto:Method: GET 2023-09-22T21:28:48.995 DEBUG:boto:Path: /ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.995 DEBUG:boto:Data: 2023-09-22T21:28:48.995 DEBUG:boto:Headers: {} 2023-09-22T21:28:48.995 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:48.995 DEBUG:boto:Port: 8000 2023-09-22T21:28:48.995 DEBUG:boto:Params: {} 2023-09-22T21:28:48.996 DEBUG:boto:Token: None 2023-09-22T21:28:48.996 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:48 GMT /ffzdqu-56/c?versionId=null 2023-09-22T21:28:48.996 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g6fA2WRGOTmLISon0i+/XWeqImE= 2023-09-22T21:28:48.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g6fA2WRGOTmLISon0i+/XWeqImE=', 'Content-Length': '0'} 2023-09-22T21:28:48.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006013e2637299c468-00650e0710-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:48.999 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:48.999 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:48.999 DEBUG:boto:path=/ffzdqu-56/?prefix=c 2023-09-22T21:28:48.999 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=c 2023-09-22T21:28:49.000 DEBUG:boto:Method: GET 2023-09-22T21:28:49.000 DEBUG:boto:Path: /ffzdqu-56/?prefix=c 2023-09-22T21:28:49.000 DEBUG:boto:Data: 2023-09-22T21:28:49.000 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.000 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:49.000 DEBUG:boto:Port: 8001 2023-09-22T21:28:49.000 DEBUG:boto:Params: {} 2023-09-22T21:28:49.000 DEBUG:boto:Token: None 2023-09-22T21:28:49.000 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/ 2023-09-22T21:28:49.000 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JIV3ruOyAexUHRs+OsfWU3B4PZc= 2023-09-22T21:28:49.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JIV3ruOyAexUHRs+OsfWU3B4PZc=', 'Content-Length': '0'} 2023-09-22T21:28:49.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a06e2921bee6b9ee-00650e0710-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.005 DEBUG:boto:b'ffzdqu-56c1000falsec2023-09-22T21:28:28.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:49.006 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:49.006 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:49.006 DEBUG:boto:path=/ffzdqu-56/?prefix=c 2023-09-22T21:28:49.006 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=c 2023-09-22T21:28:49.006 DEBUG:boto:Method: GET 2023-09-22T21:28:49.006 DEBUG:boto:Path: /ffzdqu-56/?prefix=c 2023-09-22T21:28:49.006 DEBUG:boto:Data: 2023-09-22T21:28:49.006 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.006 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:49.007 DEBUG:boto:Port: 8000 2023-09-22T21:28:49.007 DEBUG:boto:Params: {} 2023-09-22T21:28:49.007 DEBUG:boto:Token: None 2023-09-22T21:28:49.007 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/ 2023-09-22T21:28:49.007 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JIV3ruOyAexUHRs+OsfWU3B4PZc= 2023-09-22T21:28:49.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JIV3ruOyAexUHRs+OsfWU3B4PZc=', 'Content-Length': '0'} 2023-09-22T21:28:49.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003b15fb0d53d3ff4-00650e0711-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.012 DEBUG:boto:b'ffzdqu-56c1000falsec2023-09-22T21:28:28.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:49.012 DEBUG:boto:path=/ffzdqu-56/c 2023-09-22T21:28:49.013 DEBUG:boto:auth_path=/ffzdqu-56/c 2023-09-22T21:28:49.013 DEBUG:boto:Method: HEAD 2023-09-22T21:28:49.013 DEBUG:boto:Path: /ffzdqu-56/c 2023-09-22T21:28:49.013 DEBUG:boto:Data: 2023-09-22T21:28:49.013 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.013 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:49.013 DEBUG:boto:Port: 8001 2023-09-22T21:28:49.013 DEBUG:boto:Params: {} 2023-09-22T21:28:49.013 DEBUG:boto:Token: None 2023-09-22T21:28:49.014 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/c 2023-09-22T21:28:49.014 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J6m77890F3lKOP8bEHWMMu/5LcQ= 2023-09-22T21:28:49.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J6m77890F3lKOP8bEHWMMu/5LcQ=', 'Content-Length': '0'} 2023-09-22T21:28:49.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a4cdb40ee6a006a-00650e0711-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.017 DEBUG:boto:path=/ffzdqu-56/c 2023-09-22T21:28:49.017 DEBUG:boto:auth_path=/ffzdqu-56/c 2023-09-22T21:28:49.017 DEBUG:boto:Method: HEAD 2023-09-22T21:28:49.017 DEBUG:boto:Path: /ffzdqu-56/c 2023-09-22T21:28:49.017 DEBUG:boto:Data: 2023-09-22T21:28:49.017 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.017 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:49.017 DEBUG:boto:Port: 8000 2023-09-22T21:28:49.017 DEBUG:boto:Params: {} 2023-09-22T21:28:49.018 DEBUG:boto:Token: None 2023-09-22T21:28:49.018 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/c 2023-09-22T21:28:49.018 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J6m77890F3lKOP8bEHWMMu/5LcQ= 2023-09-22T21:28:49.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J6m77890F3lKOP8bEHWMMu/5LcQ=', 'Content-Length': '0'} 2023-09-22T21:28:49.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f49d6a2542d9437-00650e0711-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:28:49.021 DEBUG:boto:path=/ffzdqu-56/c 2023-09-22T21:28:49.021 DEBUG:boto:auth_path=/ffzdqu-56/c 2023-09-22T21:28:49.021 DEBUG:boto:Method: GET 2023-09-22T21:28:49.021 DEBUG:boto:Path: /ffzdqu-56/c 2023-09-22T21:28:49.021 DEBUG:boto:Data: 2023-09-22T21:28:49.021 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.022 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:49.022 DEBUG:boto:Port: 8001 2023-09-22T21:28:49.022 DEBUG:boto:Params: {} 2023-09-22T21:28:49.022 DEBUG:boto:Token: None 2023-09-22T21:28:49.022 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/c 2023-09-22T21:28:49.022 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xPImKZixCS5h0KyWpuJ5ThLeCOs= 2023-09-22T21:28:49.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xPImKZixCS5h0KyWpuJ5ThLeCOs=', 'Content-Length': '0'} 2023-09-22T21:28:49.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a881ad311137feb-00650e0711-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.070 DEBUG:boto:path=/ffzdqu-56/c 2023-09-22T21:28:49.071 DEBUG:boto:auth_path=/ffzdqu-56/c 2023-09-22T21:28:49.071 DEBUG:boto:Method: GET 2023-09-22T21:28:49.071 DEBUG:boto:Path: /ffzdqu-56/c 2023-09-22T21:28:49.071 DEBUG:boto:Data: 2023-09-22T21:28:49.071 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.071 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:49.071 DEBUG:boto:Port: 8000 2023-09-22T21:28:49.071 DEBUG:boto:Params: {} 2023-09-22T21:28:49.072 DEBUG:boto:Token: None 2023-09-22T21:28:49.072 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/c 2023-09-22T21:28:49.072 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xPImKZixCS5h0KyWpuJ5ThLeCOs= 2023-09-22T21:28:49.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xPImKZixCS5h0KyWpuJ5ThLeCOs=', 'Content-Length': '0'} 2023-09-22T21:28:49.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e638bf1f1637da3e-00650e0711-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.075 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:49.075 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:49.075 DEBUG:boto:path=/ffzdqu-56/?prefix=c 2023-09-22T21:28:49.075 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=c 2023-09-22T21:28:49.075 DEBUG:boto:Method: GET 2023-09-22T21:28:49.075 DEBUG:boto:Path: /ffzdqu-56/?prefix=c 2023-09-22T21:28:49.076 DEBUG:boto:Data: 2023-09-22T21:28:49.076 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.076 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:49.076 DEBUG:boto:Port: 8001 2023-09-22T21:28:49.076 DEBUG:boto:Params: {} 2023-09-22T21:28:49.076 DEBUG:boto:Token: None 2023-09-22T21:28:49.076 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/ 2023-09-22T21:28:49.076 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JIV3ruOyAexUHRs+OsfWU3B4PZc= 2023-09-22T21:28:49.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JIV3ruOyAexUHRs+OsfWU3B4PZc=', 'Content-Length': '0'} 2023-09-22T21:28:49.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cee54bdd244256f-00650e0711-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.126 DEBUG:boto:b'ffzdqu-56c1000falsec2023-09-22T21:28:28.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:49.127 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:49.127 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:49.127 DEBUG:boto:path=/ffzdqu-56/?prefix=c 2023-09-22T21:28:49.127 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=c 2023-09-22T21:28:49.127 DEBUG:boto:Method: GET 2023-09-22T21:28:49.127 DEBUG:boto:Path: /ffzdqu-56/?prefix=c 2023-09-22T21:28:49.127 DEBUG:boto:Data: 2023-09-22T21:28:49.128 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.128 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:49.128 DEBUG:boto:Port: 8000 2023-09-22T21:28:49.128 DEBUG:boto:Params: {} 2023-09-22T21:28:49.128 DEBUG:boto:Token: None 2023-09-22T21:28:49.128 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/ 2023-09-22T21:28:49.128 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JIV3ruOyAexUHRs+OsfWU3B4PZc= 2023-09-22T21:28:49.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JIV3ruOyAexUHRs+OsfWU3B4PZc=', 'Content-Length': '0'} 2023-09-22T21:28:49.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087a48d02e9bba28b-00650e0711-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.133 DEBUG:boto:b'ffzdqu-56c1000falsec2023-09-22T21:28:28.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:49.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:28:49.133 DEBUG:boto:path=/ffzdqu-56/d 2023-09-22T21:28:49.134 DEBUG:boto:auth_path=/ffzdqu-56/d 2023-09-22T21:28:49.134 DEBUG:boto:path=/ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.134 DEBUG:boto:auth_path=/ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.134 DEBUG:boto:Method: GET 2023-09-22T21:28:49.134 DEBUG:boto:Path: /ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.134 DEBUG:boto:Data: 2023-09-22T21:28:49.134 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.134 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:49.134 DEBUG:boto:Port: 8001 2023-09-22T21:28:49.135 DEBUG:boto:Params: {} 2023-09-22T21:28:49.135 DEBUG:boto:Token: None 2023-09-22T21:28:49.135 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.135 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VNVq+r6Z7deMz97e829TCIfMtGU= 2023-09-22T21:28:49.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VNVq+r6Z7deMz97e829TCIfMtGU=', 'Content-Length': '0'} 2023-09-22T21:28:49.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047ffaa7a242505b6-00650e0711-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.138 DEBUG:boto:path=/ffzdqu-56/d 2023-09-22T21:28:49.138 DEBUG:boto:auth_path=/ffzdqu-56/d 2023-09-22T21:28:49.139 DEBUG:boto:path=/ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.139 DEBUG:boto:auth_path=/ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.139 DEBUG:boto:Method: GET 2023-09-22T21:28:49.139 DEBUG:boto:Path: /ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.139 DEBUG:boto:Data: 2023-09-22T21:28:49.139 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.139 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:49.139 DEBUG:boto:Port: 8000 2023-09-22T21:28:49.139 DEBUG:boto:Params: {} 2023-09-22T21:28:49.139 DEBUG:boto:Token: None 2023-09-22T21:28:49.140 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.140 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VNVq+r6Z7deMz97e829TCIfMtGU= 2023-09-22T21:28:49.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VNVq+r6Z7deMz97e829TCIfMtGU=', 'Content-Length': '0'} 2023-09-22T21:28:49.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095204d0722c33d87-00650e0711-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.143 DEBUG:boto:path=/ffzdqu-56/d 2023-09-22T21:28:49.143 DEBUG:boto:auth_path=/ffzdqu-56/d 2023-09-22T21:28:49.143 DEBUG:boto:path=/ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.144 DEBUG:boto:auth_path=/ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.144 DEBUG:boto:Method: HEAD 2023-09-22T21:28:49.144 DEBUG:boto:Path: /ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.144 DEBUG:boto:Data: 2023-09-22T21:28:49.144 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.144 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:49.144 DEBUG:boto:Port: 8001 2023-09-22T21:28:49.144 DEBUG:boto:Params: {} 2023-09-22T21:28:49.144 DEBUG:boto:Token: None 2023-09-22T21:28:49.145 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.145 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:txky3TISNt3gBzHC54x8qzyguIQ= 2023-09-22T21:28:49.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:txky3TISNt3gBzHC54x8qzyguIQ=', 'Content-Length': '0'} 2023-09-22T21:28:49.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013c8b4476c45091c-00650e0711-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.148 DEBUG:boto:path=/ffzdqu-56/d 2023-09-22T21:28:49.148 DEBUG:boto:auth_path=/ffzdqu-56/d 2023-09-22T21:28:49.148 DEBUG:boto:path=/ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.148 DEBUG:boto:auth_path=/ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.148 DEBUG:boto:Method: HEAD 2023-09-22T21:28:49.148 DEBUG:boto:Path: /ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.148 DEBUG:boto:Data: 2023-09-22T21:28:49.148 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.148 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:49.149 DEBUG:boto:Port: 8000 2023-09-22T21:28:49.149 DEBUG:boto:Params: {} 2023-09-22T21:28:49.149 DEBUG:boto:Token: None 2023-09-22T21:28:49.149 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.149 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:txky3TISNt3gBzHC54x8qzyguIQ= 2023-09-22T21:28:49.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:txky3TISNt3gBzHC54x8qzyguIQ=', 'Content-Length': '0'} 2023-09-22T21:28:49.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cdeec01cb354bc8-00650e0711-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:28:49.152 DEBUG:boto:path=/ffzdqu-56/d 2023-09-22T21:28:49.153 DEBUG:boto:auth_path=/ffzdqu-56/d 2023-09-22T21:28:49.153 DEBUG:boto:path=/ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.153 DEBUG:boto:auth_path=/ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.153 DEBUG:boto:Method: GET 2023-09-22T21:28:49.153 DEBUG:boto:Path: /ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.153 DEBUG:boto:Data: 2023-09-22T21:28:49.153 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.153 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:49.153 DEBUG:boto:Port: 8001 2023-09-22T21:28:49.153 DEBUG:boto:Params: {} 2023-09-22T21:28:49.154 DEBUG:boto:Token: None 2023-09-22T21:28:49.154 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.154 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VNVq+r6Z7deMz97e829TCIfMtGU= 2023-09-22T21:28:49.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VNVq+r6Z7deMz97e829TCIfMtGU=', 'Content-Length': '0'} 2023-09-22T21:28:49.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a6ef98ecdc961a2-00650e0711-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.198 DEBUG:boto:path=/ffzdqu-56/d 2023-09-22T21:28:49.198 DEBUG:boto:auth_path=/ffzdqu-56/d 2023-09-22T21:28:49.199 DEBUG:boto:path=/ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.199 DEBUG:boto:auth_path=/ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.199 DEBUG:boto:Method: GET 2023-09-22T21:28:49.199 DEBUG:boto:Path: /ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.199 DEBUG:boto:Data: 2023-09-22T21:28:49.199 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.199 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:49.199 DEBUG:boto:Port: 8000 2023-09-22T21:28:49.199 DEBUG:boto:Params: {} 2023-09-22T21:28:49.200 DEBUG:boto:Token: None 2023-09-22T21:28:49.200 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/d?versionId=null 2023-09-22T21:28:49.200 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VNVq+r6Z7deMz97e829TCIfMtGU= 2023-09-22T21:28:49.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VNVq+r6Z7deMz97e829TCIfMtGU=', 'Content-Length': '0'} 2023-09-22T21:28:49.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014a32b157636cb88-00650e0711-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.203 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:49.204 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:49.204 DEBUG:boto:path=/ffzdqu-56/?prefix=d 2023-09-22T21:28:49.204 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=d 2023-09-22T21:28:49.204 DEBUG:boto:Method: GET 2023-09-22T21:28:49.204 DEBUG:boto:Path: /ffzdqu-56/?prefix=d 2023-09-22T21:28:49.204 DEBUG:boto:Data: 2023-09-22T21:28:49.204 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.204 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:49.204 DEBUG:boto:Port: 8001 2023-09-22T21:28:49.204 DEBUG:boto:Params: {} 2023-09-22T21:28:49.205 DEBUG:boto:Token: None 2023-09-22T21:28:49.205 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/ 2023-09-22T21:28:49.205 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JIV3ruOyAexUHRs+OsfWU3B4PZc= 2023-09-22T21:28:49.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JIV3ruOyAexUHRs+OsfWU3B4PZc=', 'Content-Length': '0'} 2023-09-22T21:28:49.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000805588845d975155-00650e0711-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.250 DEBUG:boto:b'ffzdqu-56d1000falsed2023-09-22T21:28:28.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:49.251 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:49.251 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:49.251 DEBUG:boto:path=/ffzdqu-56/?prefix=d 2023-09-22T21:28:49.251 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=d 2023-09-22T21:28:49.251 DEBUG:boto:Method: GET 2023-09-22T21:28:49.251 DEBUG:boto:Path: /ffzdqu-56/?prefix=d 2023-09-22T21:28:49.251 DEBUG:boto:Data: 2023-09-22T21:28:49.251 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.252 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:49.252 DEBUG:boto:Port: 8000 2023-09-22T21:28:49.252 DEBUG:boto:Params: {} 2023-09-22T21:28:49.252 DEBUG:boto:Token: None 2023-09-22T21:28:49.252 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/ 2023-09-22T21:28:49.252 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JIV3ruOyAexUHRs+OsfWU3B4PZc= 2023-09-22T21:28:49.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JIV3ruOyAexUHRs+OsfWU3B4PZc=', 'Content-Length': '0'} 2023-09-22T21:28:49.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b758d215f330c59d-00650e0711-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.257 DEBUG:boto:b'ffzdqu-56d1000falsed2023-09-22T21:28:28.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:49.257 DEBUG:boto:path=/ffzdqu-56/d 2023-09-22T21:28:49.257 DEBUG:boto:auth_path=/ffzdqu-56/d 2023-09-22T21:28:49.258 DEBUG:boto:Method: HEAD 2023-09-22T21:28:49.258 DEBUG:boto:Path: /ffzdqu-56/d 2023-09-22T21:28:49.258 DEBUG:boto:Data: 2023-09-22T21:28:49.258 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.258 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:49.258 DEBUG:boto:Port: 8001 2023-09-22T21:28:49.258 DEBUG:boto:Params: {} 2023-09-22T21:28:49.258 DEBUG:boto:Token: None 2023-09-22T21:28:49.258 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/d 2023-09-22T21:28:49.259 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vpo4Qv2d98C+4BUNGgJVCuUKlK0= 2023-09-22T21:28:49.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vpo4Qv2d98C+4BUNGgJVCuUKlK0=', 'Content-Length': '0'} 2023-09-22T21:28:49.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a85c48a9e3deb906-00650e0711-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.262 DEBUG:boto:path=/ffzdqu-56/d 2023-09-22T21:28:49.262 DEBUG:boto:auth_path=/ffzdqu-56/d 2023-09-22T21:28:49.262 DEBUG:boto:Method: HEAD 2023-09-22T21:28:49.262 DEBUG:boto:Path: /ffzdqu-56/d 2023-09-22T21:28:49.262 DEBUG:boto:Data: 2023-09-22T21:28:49.262 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.262 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:49.262 DEBUG:boto:Port: 8000 2023-09-22T21:28:49.263 DEBUG:boto:Params: {} 2023-09-22T21:28:49.263 DEBUG:boto:Token: None 2023-09-22T21:28:49.263 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/d 2023-09-22T21:28:49.263 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vpo4Qv2d98C+4BUNGgJVCuUKlK0= 2023-09-22T21:28:49.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vpo4Qv2d98C+4BUNGgJVCuUKlK0=', 'Content-Length': '0'} 2023-09-22T21:28:49.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b5b48e0deaef438-00650e0711-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:28:49.266 DEBUG:boto:path=/ffzdqu-56/d 2023-09-22T21:28:49.266 DEBUG:boto:auth_path=/ffzdqu-56/d 2023-09-22T21:28:49.266 DEBUG:boto:Method: GET 2023-09-22T21:28:49.266 DEBUG:boto:Path: /ffzdqu-56/d 2023-09-22T21:28:49.266 DEBUG:boto:Data: 2023-09-22T21:28:49.266 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.267 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:49.267 DEBUG:boto:Port: 8001 2023-09-22T21:28:49.267 DEBUG:boto:Params: {} 2023-09-22T21:28:49.267 DEBUG:boto:Token: None 2023-09-22T21:28:49.267 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/d 2023-09-22T21:28:49.267 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2yt4I5W++gMspmwI5f9Zdn3z51g= 2023-09-22T21:28:49.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2yt4I5W++gMspmwI5f9Zdn3z51g=', 'Content-Length': '0'} 2023-09-22T21:28:49.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb1180ada7eb5b6f-00650e0711-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.310 DEBUG:boto:path=/ffzdqu-56/d 2023-09-22T21:28:49.310 DEBUG:boto:auth_path=/ffzdqu-56/d 2023-09-22T21:28:49.311 DEBUG:boto:Method: GET 2023-09-22T21:28:49.311 DEBUG:boto:Path: /ffzdqu-56/d 2023-09-22T21:28:49.311 DEBUG:boto:Data: 2023-09-22T21:28:49.311 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.311 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:49.311 DEBUG:boto:Port: 8000 2023-09-22T21:28:49.311 DEBUG:boto:Params: {} 2023-09-22T21:28:49.311 DEBUG:boto:Token: None 2023-09-22T21:28:49.312 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/d 2023-09-22T21:28:49.312 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2yt4I5W++gMspmwI5f9Zdn3z51g= 2023-09-22T21:28:49.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2yt4I5W++gMspmwI5f9Zdn3z51g=', 'Content-Length': '0'} 2023-09-22T21:28:49.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000771a5bdeeaa36ddf-00650e0711-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.315 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:49.315 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:49.315 DEBUG:boto:path=/ffzdqu-56/?prefix=d 2023-09-22T21:28:49.315 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=d 2023-09-22T21:28:49.315 DEBUG:boto:Method: GET 2023-09-22T21:28:49.315 DEBUG:boto:Path: /ffzdqu-56/?prefix=d 2023-09-22T21:28:49.316 DEBUG:boto:Data: 2023-09-22T21:28:49.316 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.316 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:28:49.316 DEBUG:boto:Port: 8001 2023-09-22T21:28:49.316 DEBUG:boto:Params: {} 2023-09-22T21:28:49.316 DEBUG:boto:Token: None 2023-09-22T21:28:49.316 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/ 2023-09-22T21:28:49.316 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JIV3ruOyAexUHRs+OsfWU3B4PZc= 2023-09-22T21:28:49.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JIV3ruOyAexUHRs+OsfWU3B4PZc=', 'Content-Length': '0'} 2023-09-22T21:28:49.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f78e6047ea707ce0-00650e0711-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.321 DEBUG:boto:b'ffzdqu-56d1000falsed2023-09-22T21:28:28.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:49.322 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:49.322 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:49.322 DEBUG:boto:path=/ffzdqu-56/?prefix=d 2023-09-22T21:28:49.322 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=d 2023-09-22T21:28:49.322 DEBUG:boto:Method: GET 2023-09-22T21:28:49.322 DEBUG:boto:Path: /ffzdqu-56/?prefix=d 2023-09-22T21:28:49.322 DEBUG:boto:Data: 2023-09-22T21:28:49.323 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.323 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:49.323 DEBUG:boto:Port: 8000 2023-09-22T21:28:49.323 DEBUG:boto:Params: {} 2023-09-22T21:28:49.323 DEBUG:boto:Token: None 2023-09-22T21:28:49.323 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/ 2023-09-22T21:28:49.323 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JIV3ruOyAexUHRs+OsfWU3B4PZc= 2023-09-22T21:28:49.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JIV3ruOyAexUHRs+OsfWU3B4PZc=', 'Content-Length': '0'} 2023-09-22T21:28:49.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f687a76dade77ecf-00650e0711-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.328 DEBUG:boto:b'ffzdqu-56d1000falsed2023-09-22T21:28:28.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:28:49.329 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-56 zones={a2, a1} 2023-09-22T21:28:49.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'ffzdqu-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:28:49.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'ffzdqu-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:28:49.329 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket ffzdqu-56 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:28:49.374 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:28:49.374 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:28:49.428 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:28:49.428 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-56 2023-09-22T21:28:49.435 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 4 2023-09-22T21:28:49.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'ffzdqu-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:28:49.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'ffzdqu-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:28:49.481 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket ffzdqu-56 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:28:49.561 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:28:49.562 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:28:49.608 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:28:49.608 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-56 2023-09-22T21:28:49.609 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 4 2023-09-22T21:28:49.644 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:49.644 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:49.644 DEBUG:boto:Method: HEAD 2023-09-22T21:28:49.644 DEBUG:boto:Path: /ffzdqu-56/ 2023-09-22T21:28:49.644 DEBUG:boto:Data: 2023-09-22T21:28:49.644 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.645 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:49.645 DEBUG:boto:Port: 8000 2023-09-22T21:28:49.645 DEBUG:boto:Params: {} 2023-09-22T21:28:49.645 DEBUG:boto:Token: None 2023-09-22T21:28:49.645 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/ 2023-09-22T21:28:49.645 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HqiYeX1eCwLPcAykZLLzoNwv1CM= 2023-09-22T21:28:49.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HqiYeX1eCwLPcAykZLLzoNwv1CM=', 'Content-Length': '0'} 2023-09-22T21:28:49.648 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005228d22da35b6485-00650e0711-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.648 DEBUG:boto:path=/ffzdqu-56/e 2023-09-22T21:28:49.648 DEBUG:boto:auth_path=/ffzdqu-56/e 2023-09-22T21:28:49.648 DEBUG:boto:Method: PUT 2023-09-22T21:28:49.648 DEBUG:boto:Path: /ffzdqu-56/e 2023-09-22T21:28:49.648 DEBUG:boto:Data: 2023-09-22T21:28:49.648 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:28:49.648 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:49.649 DEBUG:boto:Port: 8000 2023-09-22T21:28:49.649 DEBUG:boto:Params: {} 2023-09-22T21:28:49.649 DEBUG:boto:Token: None 2023-09-22T21:28:49.649 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/e 2023-09-22T21:28:49.649 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tSwmPXPLZ/WNWE/Dxatc9aVzS4o= 2023-09-22T21:28:49.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tSwmPXPLZ/WNWE/Dxatc9aVzS4o='} 2023-09-22T21:28:49.655 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8dead724322335b-00650e0711-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.656 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:49.656 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:49.656 DEBUG:boto:Method: HEAD 2023-09-22T21:28:49.656 DEBUG:boto:Path: /ffzdqu-56/ 2023-09-22T21:28:49.656 DEBUG:boto:Data: 2023-09-22T21:28:49.656 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.656 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:49.656 DEBUG:boto:Port: 8000 2023-09-22T21:28:49.657 DEBUG:boto:Params: {} 2023-09-22T21:28:49.657 DEBUG:boto:Token: None 2023-09-22T21:28:49.657 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/ 2023-09-22T21:28:49.657 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HqiYeX1eCwLPcAykZLLzoNwv1CM= 2023-09-22T21:28:49.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HqiYeX1eCwLPcAykZLLzoNwv1CM=', 'Content-Length': '0'} 2023-09-22T21:28:49.659 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e531e3de480a3566-00650e0711-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.659 DEBUG:boto:path=/ffzdqu-56/f 2023-09-22T21:28:49.659 DEBUG:boto:auth_path=/ffzdqu-56/f 2023-09-22T21:28:49.659 DEBUG:boto:Method: PUT 2023-09-22T21:28:49.659 DEBUG:boto:Path: /ffzdqu-56/f 2023-09-22T21:28:49.659 DEBUG:boto:Data: 2023-09-22T21:28:49.659 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:28:49.659 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:49.659 DEBUG:boto:Port: 8000 2023-09-22T21:28:49.660 DEBUG:boto:Params: {} 2023-09-22T21:28:49.660 DEBUG:boto:Token: None 2023-09-22T21:28:49.660 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/f 2023-09-22T21:28:49.660 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pPR55ViZH+wTGX7LdK0oOzR5KlM= 2023-09-22T21:28:49.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pPR55ViZH+wTGX7LdK0oOzR5KlM='} 2023-09-22T21:28:49.702 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0b8f1c8b9334682-00650e0711-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.703 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:49.703 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:49.703 DEBUG:boto:Method: HEAD 2023-09-22T21:28:49.703 DEBUG:boto:Path: /ffzdqu-56/ 2023-09-22T21:28:49.703 DEBUG:boto:Data: 2023-09-22T21:28:49.703 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.703 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:49.703 DEBUG:boto:Port: 8000 2023-09-22T21:28:49.704 DEBUG:boto:Params: {} 2023-09-22T21:28:49.704 DEBUG:boto:Token: None 2023-09-22T21:28:49.704 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/ 2023-09-22T21:28:49.704 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HqiYeX1eCwLPcAykZLLzoNwv1CM= 2023-09-22T21:28:49.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HqiYeX1eCwLPcAykZLLzoNwv1CM=', 'Content-Length': '0'} 2023-09-22T21:28:49.706 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b98db71b27d3019-00650e0711-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.706 DEBUG:boto:path=/ffzdqu-56/g 2023-09-22T21:28:49.706 DEBUG:boto:auth_path=/ffzdqu-56/g 2023-09-22T21:28:49.706 DEBUG:boto:Method: PUT 2023-09-22T21:28:49.706 DEBUG:boto:Path: /ffzdqu-56/g 2023-09-22T21:28:49.707 DEBUG:boto:Data: 2023-09-22T21:28:49.707 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:28:49.707 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:49.707 DEBUG:boto:Port: 8000 2023-09-22T21:28:49.707 DEBUG:boto:Params: {} 2023-09-22T21:28:49.707 DEBUG:boto:Token: None 2023-09-22T21:28:49.707 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/g 2023-09-22T21:28:49.708 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3m4WlaQ6h/EaSVCfsAtMhrDming= 2023-09-22T21:28:49.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3m4WlaQ6h/EaSVCfsAtMhrDming='} 2023-09-22T21:28:49.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000270e7202ae84e794-00650e0711-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.714 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:28:49.715 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:28:49.715 DEBUG:boto:Method: HEAD 2023-09-22T21:28:49.715 DEBUG:boto:Path: /ffzdqu-56/ 2023-09-22T21:28:49.715 DEBUG:boto:Data: 2023-09-22T21:28:49.715 DEBUG:boto:Headers: {} 2023-09-22T21:28:49.715 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:49.715 DEBUG:boto:Port: 8000 2023-09-22T21:28:49.715 DEBUG:boto:Params: {} 2023-09-22T21:28:49.716 DEBUG:boto:Token: None 2023-09-22T21:28:49.716 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/ 2023-09-22T21:28:49.716 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HqiYeX1eCwLPcAykZLLzoNwv1CM= 2023-09-22T21:28:49.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HqiYeX1eCwLPcAykZLLzoNwv1CM=', 'Content-Length': '0'} 2023-09-22T21:28:49.718 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009bc4230ae4a32d7f-00650e0711-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.718 DEBUG:boto:path=/ffzdqu-56/h 2023-09-22T21:28:49.718 DEBUG:boto:auth_path=/ffzdqu-56/h 2023-09-22T21:28:49.718 DEBUG:boto:Method: PUT 2023-09-22T21:28:49.719 DEBUG:boto:Path: /ffzdqu-56/h 2023-09-22T21:28:49.719 DEBUG:boto:Data: 2023-09-22T21:28:49.719 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:28:49.719 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:28:49.719 DEBUG:boto:Port: 8000 2023-09-22T21:28:49.719 DEBUG:boto:Params: {} 2023-09-22T21:28:49.719 DEBUG:boto:Token: None 2023-09-22T21:28:49.719 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:28:49 GMT /ffzdqu-56/h 2023-09-22T21:28:49.720 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1EJcO2oGLgzHZdCStS41LGYsSy0= 2023-09-22T21:28:49.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:28:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1EJcO2oGLgzHZdCStS41LGYsSy0='} 2023-09-22T21:28:49.725 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f25cd3608f034621-00650e0711-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:28:49.725 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-56 2023-09-22T21:28:49.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:28:49.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:28:49.725 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:28:49.770 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:28:49.770 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:28:49.815 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:28:49.814+0000 7f1165f92a40 1 waiting to reach latest gen 2, on 0.. 2023-09-22T21:28:54.817 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:28:54.814+0000 7f1165f92a40 1 waiting to reach latest gen 2, on 0.. 2023-09-22T21:28:59.818 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:28:59.814+0000 7f1165f92a40 1 waiting to reach latest gen 2, on 0.. 2023-09-22T21:29:04.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:29:04.814+0000 7f1165f92a40 1 waiting to reach latest gen 2, on 0.. 2023-09-22T21:29:09.820 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:29:09.818+0000 7f1165f92a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:29:14.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:29:14.818+0000 7f1165f92a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:29:19.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:29:19.818+0000 7f1165f92a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:29:24.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:29:24.818+0000 7f1165f92a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:29:29.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:29:29.822+0000 7f1165f92a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:29:34.826 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:29:34.821+0000 7f1165f92a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:29:39.827 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:29:39.821+0000 7f1165f92a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:29:44.828 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:29:44.825+0000 7f1165f92a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:29:49.829 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:29:49.829+0000 7f1165f92a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:29:54.830 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:29:54.829+0000 7f1165f92a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:29:59.831 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:29:59.829+0000 7f1165f92a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:30:04.832 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:30:04.833+0000 7f1165f92a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:30:09.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:30:09.833+0000 7f1165f92a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:30:14.835 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:30:14.833+0000 7f1165f92a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:30:19.836 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:30:19.833+0000 7f1165f92a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:30:24.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:30:24.837+0000 7f1165f92a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:30:29.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:30:29.836+0000 7f1165f92a40 1 waiting for incremental sync to catch up: 2023-09-22T21:30:29.839 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.41.6] 2023-09-22T21:30:29.840 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.27.6, 00000000002.26.6, 00000000002.41.6] 2023-09-22T21:30:34.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:30:34.840+0000 7f1165f92a40 1 waiting for incremental sync to catch up: 2023-09-22T21:30:34.841 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.41.6] 2023-09-22T21:30:34.841 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.27.6, 00000000002.26.6, 00000000002.41.6] 2023-09-22T21:30:39.842 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:30:39.840+0000 7f1165f92a40 1 waiting for incremental sync to catch up: 2023-09-22T21:30:39.842 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.41.6] 2023-09-22T21:30:39.842 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.27.6, 00000000002.26.6, 00000000002.41.6] 2023-09-22T21:30:44.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:30:44.840+0000 7f1165f92a40 1 waiting for incremental sync to catch up: 2023-09-22T21:30:44.844 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.41.6] 2023-09-22T21:30:44.844 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.27.6, 00000000002.26.6, 00000000002.41.6] 2023-09-22T21:30:49.845 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:30:49.844+0000 7f1165f92a40 1 waiting for incremental sync to catch up: 2023-09-22T21:30:49.845 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.41.6] 2023-09-22T21:30:49.845 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.27.6, 00000000002.26.6, 00000000002.41.6] 2023-09-22T21:30:54.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:30:54.844+0000 7f1165f92a40 1 waiting for incremental sync to catch up: 2023-09-22T21:30:54.846 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.41.6] 2023-09-22T21:30:54.846 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.27.6, 00000000002.26.6, 00000000002.41.6] 2023-09-22T21:30:59.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:30:59.844+0000 7f1165f92a40 1 waiting for incremental sync to catch up: 2023-09-22T21:30:59.848 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.41.6] 2023-09-22T21:30:59.848 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.27.6, 00000000002.26.6, 00000000002.41.6] 2023-09-22T21:31:04.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:31:04.848+0000 7f1165f92a40 1 waiting for incremental sync to catch up: 2023-09-22T21:31:04.849 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.41.6] 2023-09-22T21:31:04.849 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.27.6, 00000000002.26.6, 00000000002.41.6] 2023-09-22T21:31:09.850 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:31:09.848+0000 7f1165f92a40 1 waiting for incremental sync to catch up: 2023-09-22T21:31:09.850 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.41.6] 2023-09-22T21:31:09.850 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.27.6, 00000000002.26.6, 00000000002.41.6] 2023-09-22T21:31:14.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:31:14.848+0000 7f1165f92a40 1 waiting for incremental sync to catch up: 2023-09-22T21:31:14.852 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.41.6] 2023-09-22T21:31:14.852 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.27.6, 00000000002.26.6, 00000000002.41.6] 2023-09-22T21:31:19.853 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:31:19.852+0000 7f1165f92a40 1 waiting for incremental sync to catch up: 2023-09-22T21:31:19.853 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.41.6] 2023-09-22T21:31:19.853 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.27.6, 00000000002.26.6, 00000000002.41.6] 2023-09-22T21:31:24.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:31:24.852+0000 7f1165f92a40 1 waiting for incremental sync to catch up: 2023-09-22T21:31:24.854 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.41.6] 2023-09-22T21:31:24.854 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.27.6, 00000000002.26.6, 00000000002.41.6] 2023-09-22T21:31:29.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:31:29.851+0000 7f1165f92a40 1 waiting for incremental sync to catch up: 2023-09-22T21:31:29.856 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.41.6] 2023-09-22T21:31:29.856 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.27.6, 00000000002.26.6, 00000000002.41.6] 2023-09-22T21:31:34.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:31:34.855+0000 7f1165f92a40 1 waiting for incremental sync to catch up: 2023-09-22T21:31:34.857 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.41.6] 2023-09-22T21:31:34.857 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.27.6, 00000000002.26.6, 00000000002.41.6] 2023-09-22T21:31:39.858 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:31:39.855+0000 7f1165f92a40 1 waiting for incremental sync to catch up: 2023-09-22T21:31:39.858 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.41.6] 2023-09-22T21:31:39.859 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.27.6, 00000000002.26.6, 00000000002.41.6] 2023-09-22T21:31:44.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:31:44.855+0000 7f1165f92a40 1 waiting for incremental sync to catch up: 2023-09-22T21:31:44.859 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.41.6] 2023-09-22T21:31:44.860 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.27.6, 00000000002.26.6, 00000000002.41.6] 2023-09-22T21:31:45.781 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:31:45.779+0000 7ff4815cb640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:31:49.861 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:31:49.859+0000 7f1165f92a40 1 bucket sync caught up with source: 2023-09-22T21:31:49.861 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000003.27.6, 00000000002.26.6, 00000000002.41.6] 2023-09-22T21:31:49.861 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.27.6, 00000000002.26.6, 00000000002.41.6] 2023-09-22T21:31:49.861 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:31:49.859+0000 7f1165f92a40 0 bucket checkpoint complete 2023-09-22T21:31:49.894 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-56 2023-09-22T21:31:49.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:31:49.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:31:49.895 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:31:49.985 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:31:49.985 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:31:50.028 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:31:50.027+0000 7f8d8a930a40 1 waiting for incremental sync to catch up: 2023-09-22T21:31:50.028 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.180.6, , , , 00000000001.113.6, , , 00000000002.265.6, , 00000000001.225.6] 2023-09-22T21:31:50.028 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.198.6, , , 00000000001.212.6, 00000000001.113.6, , , 00000000002.265.6, , 00000000002.243.6] 2023-09-22T21:31:55.030 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:31:55.027+0000 7f8d8a930a40 1 waiting for incremental sync to catch up: 2023-09-22T21:31:55.031 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.180.6, , , , 00000000001.113.6, , , 00000000002.265.6, , 00000000001.225.6] 2023-09-22T21:31:55.031 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.198.6, , , 00000000001.212.6, 00000000001.113.6, , , 00000000002.265.6, , 00000000002.243.6] 2023-09-22T21:32:00.033 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:32:00.031+0000 7f8d8a930a40 1 waiting for incremental sync to catch up: 2023-09-22T21:32:00.033 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.180.6, , , , 00000000001.113.6, , , 00000000002.265.6, , 00000000001.225.6] 2023-09-22T21:32:00.034 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.198.6, , , 00000000001.212.6, 00000000001.113.6, , , 00000000002.265.6, , 00000000002.243.6] 2023-09-22T21:32:02.401 INFO:tasks.rgw.c1.client.0.smithi160.stdout:2023-09-22T21:32:02.399+0000 7fa9c1675640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:32:05.036 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:32:05.031+0000 7f8d8a930a40 1 waiting for incremental sync to catch up: 2023-09-22T21:32:05.036 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.198.6, , , , 00000000001.113.6, , , 00000000002.265.6, , 00000000002.243.6] 2023-09-22T21:32:05.036 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.198.6, , , 00000000001.212.6, 00000000001.113.6, , , 00000000002.265.6, , 00000000002.243.6] 2023-09-22T21:32:10.038 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:32:10.035+0000 7f8d8a930a40 1 bucket sync caught up with source: 2023-09-22T21:32:10.038 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.198.6, , , 00000000001.212.6, 00000000001.113.6, , , 00000000002.265.6, , 00000000002.243.6] 2023-09-22T21:32:10.038 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.198.6, , , 00000000001.212.6, 00000000001.113.6, , , 00000000002.265.6, , 00000000002.243.6] 2023-09-22T21:32:10.038 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:32:10.035+0000 7f8d8a930a40 0 bucket checkpoint complete 2023-09-22T21:32:10.071 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-56 zones={a2, a1} 2023-09-22T21:32:10.072 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:10.072 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:10.072 DEBUG:boto:Method: HEAD 2023-09-22T21:32:10.072 DEBUG:boto:Path: /ffzdqu-56/ 2023-09-22T21:32:10.072 DEBUG:boto:Data: 2023-09-22T21:32:10.072 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.073 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.073 DEBUG:boto:Params: {} 2023-09-22T21:32:10.073 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:32:10.073 DEBUG:boto:Token: None 2023-09-22T21:32:10.074 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/ 2023-09-22T21:32:10.074 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LIkCe3eJqEsl3/3RT3Yv4wonGrM= 2023-09-22T21:32:10.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LIkCe3eJqEsl3/3RT3Yv4wonGrM=', 'Content-Length': '0'} 2023-09-22T21:32:10.079 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a98b5c7d52c4dee-00650e07da-165b-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.079 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:10.079 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:10.080 DEBUG:boto:Method: HEAD 2023-09-22T21:32:10.080 DEBUG:boto:Path: /ffzdqu-56/ 2023-09-22T21:32:10.080 DEBUG:boto:Data: 2023-09-22T21:32:10.080 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.080 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.080 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.080 DEBUG:boto:Params: {} 2023-09-22T21:32:10.080 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:32:10.081 DEBUG:boto:Token: None 2023-09-22T21:32:10.081 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/ 2023-09-22T21:32:10.081 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LIkCe3eJqEsl3/3RT3Yv4wonGrM= 2023-09-22T21:32:10.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LIkCe3eJqEsl3/3RT3Yv4wonGrM=', 'Content-Length': '0'} 2023-09-22T21:32:10.085 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2c3008ef5ab217b-00650e07da-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.085 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:32:10.085 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:10.085 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:10.085 DEBUG:boto:path=/ffzdqu-56/?versions&max-keys=999 2023-09-22T21:32:10.086 DEBUG:boto:auth_path=/ffzdqu-56/?versions&max-keys=999 2023-09-22T21:32:10.086 DEBUG:boto:Method: GET 2023-09-22T21:32:10.086 DEBUG:boto:Path: /ffzdqu-56/?versions&max-keys=999 2023-09-22T21:32:10.086 DEBUG:boto:Data: 2023-09-22T21:32:10.086 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.086 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.086 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.086 DEBUG:boto:Params: {} 2023-09-22T21:32:10.089 DEBUG:boto:Token: None 2023-09-22T21:32:10.089 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/?versions 2023-09-22T21:32:10.089 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:v+wdEPaS4N/EtYP+oQBOVMe/FR4= 2023-09-22T21:32:10.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:v+wdEPaS4N/EtYP+oQBOVMe/FR4=', 'Content-Length': '0'} 2023-09-22T21:32:10.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000117b2f2816bd6d24-00650e07da-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.094 DEBUG:boto:b'ffzdqu-56999falseanulltrue2023-09-22T21:28:28.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:28:28.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:28:28.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:28:28.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-22T21:28:49.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-22T21:28:49.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-22T21:28:49.711Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-22T21:28:49.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:10.094 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-22T21:32:10.095 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-22T21:32:10.095 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-22T21:32:10.095 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-22T21:32:10.095 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-09-22T21:32:10.095 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-09-22T21:32:10.095 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-09-22T21:32:10.095 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-09-22T21:32:10.095 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:32:10.096 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:10.096 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:10.096 DEBUG:boto:path=/ffzdqu-56/?versions&max-keys=999 2023-09-22T21:32:10.096 DEBUG:boto:auth_path=/ffzdqu-56/?versions&max-keys=999 2023-09-22T21:32:10.096 DEBUG:boto:Method: GET 2023-09-22T21:32:10.096 DEBUG:boto:Path: /ffzdqu-56/?versions&max-keys=999 2023-09-22T21:32:10.096 DEBUG:boto:Data: 2023-09-22T21:32:10.096 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.097 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.097 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.097 DEBUG:boto:Params: {} 2023-09-22T21:32:10.097 DEBUG:boto:Token: None 2023-09-22T21:32:10.097 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/?versions 2023-09-22T21:32:10.097 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:v+wdEPaS4N/EtYP+oQBOVMe/FR4= 2023-09-22T21:32:10.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:v+wdEPaS4N/EtYP+oQBOVMe/FR4=', 'Content-Length': '0'} 2023-09-22T21:32:10.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029161ee93236d7ca-00650e07da-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.100 DEBUG:boto:b'ffzdqu-56999falseanulltrue2023-09-22T21:28:28.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:28:28.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:28:28.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:28:28.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-22T21:28:49.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-22T21:28:49.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-22T21:28:49.711Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-22T21:28:49.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:10.101 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-22T21:32:10.101 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-22T21:32:10.101 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-22T21:32:10.101 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-22T21:32:10.101 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-09-22T21:32:10.102 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-09-22T21:32:10.102 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-09-22T21:32:10.102 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-09-22T21:32:10.102 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:10.102 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:10.102 DEBUG:boto:path=/ffzdqu-56/?versions&max-keys=999 2023-09-22T21:32:10.102 DEBUG:boto:auth_path=/ffzdqu-56/?versions&max-keys=999 2023-09-22T21:32:10.102 DEBUG:boto:Method: GET 2023-09-22T21:32:10.103 DEBUG:boto:Path: /ffzdqu-56/?versions&max-keys=999 2023-09-22T21:32:10.103 DEBUG:boto:Data: 2023-09-22T21:32:10.103 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.103 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.103 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.103 DEBUG:boto:Params: {} 2023-09-22T21:32:10.103 DEBUG:boto:Token: None 2023-09-22T21:32:10.103 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/?versions 2023-09-22T21:32:10.104 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:v+wdEPaS4N/EtYP+oQBOVMe/FR4= 2023-09-22T21:32:10.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:v+wdEPaS4N/EtYP+oQBOVMe/FR4=', 'Content-Length': '0'} 2023-09-22T21:32:10.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f8b7ae8be388d6c-00650e07da-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.107 DEBUG:boto:b'ffzdqu-56999falseanulltrue2023-09-22T21:28:28.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:28:28.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:28:28.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:28:28.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-22T21:28:49.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-22T21:28:49.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-22T21:28:49.711Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-22T21:28:49.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:10.108 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:10.108 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:10.108 DEBUG:boto:path=/ffzdqu-56/?versions&max-keys=999 2023-09-22T21:32:10.109 DEBUG:boto:auth_path=/ffzdqu-56/?versions&max-keys=999 2023-09-22T21:32:10.109 DEBUG:boto:Method: GET 2023-09-22T21:32:10.109 DEBUG:boto:Path: /ffzdqu-56/?versions&max-keys=999 2023-09-22T21:32:10.109 DEBUG:boto:Data: 2023-09-22T21:32:10.109 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.109 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.109 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.109 DEBUG:boto:Params: {} 2023-09-22T21:32:10.109 DEBUG:boto:Token: None 2023-09-22T21:32:10.110 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/?versions 2023-09-22T21:32:10.110 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:v+wdEPaS4N/EtYP+oQBOVMe/FR4= 2023-09-22T21:32:10.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:v+wdEPaS4N/EtYP+oQBOVMe/FR4=', 'Content-Length': '0'} 2023-09-22T21:32:10.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008264b52fb9d5ff85-00650e07da-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.113 DEBUG:boto:b'ffzdqu-56999falseanulltrue2023-09-22T21:28:28.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:28:28.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:28:28.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:28:28.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-22T21:28:49.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-22T21:28:49.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-22T21:28:49.711Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-22T21:28:49.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:10.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:32:10.114 DEBUG:boto:path=/ffzdqu-56/a 2023-09-22T21:32:10.114 DEBUG:boto:auth_path=/ffzdqu-56/a 2023-09-22T21:32:10.114 DEBUG:boto:path=/ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.114 DEBUG:boto:auth_path=/ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.114 DEBUG:boto:Method: GET 2023-09-22T21:32:10.114 DEBUG:boto:Path: /ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.114 DEBUG:boto:Data: 2023-09-22T21:32:10.115 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.115 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.115 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.115 DEBUG:boto:Params: {} 2023-09-22T21:32:10.115 DEBUG:boto:Token: None 2023-09-22T21:32:10.115 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.115 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fqHfjnV7eGJbnL/ar43uMYuEH3Y= 2023-09-22T21:32:10.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fqHfjnV7eGJbnL/ar43uMYuEH3Y=', 'Content-Length': '0'} 2023-09-22T21:32:10.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000795649adbaa71b00-00650e07da-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.162 DEBUG:boto:path=/ffzdqu-56/a 2023-09-22T21:32:10.163 DEBUG:boto:auth_path=/ffzdqu-56/a 2023-09-22T21:32:10.163 DEBUG:boto:path=/ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.163 DEBUG:boto:auth_path=/ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.163 DEBUG:boto:Method: GET 2023-09-22T21:32:10.163 DEBUG:boto:Path: /ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.163 DEBUG:boto:Data: 2023-09-22T21:32:10.163 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.163 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.163 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.164 DEBUG:boto:Params: {} 2023-09-22T21:32:10.164 DEBUG:boto:Token: None 2023-09-22T21:32:10.164 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.164 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fqHfjnV7eGJbnL/ar43uMYuEH3Y= 2023-09-22T21:32:10.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fqHfjnV7eGJbnL/ar43uMYuEH3Y=', 'Content-Length': '0'} 2023-09-22T21:32:10.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029b64a2c56a6a2cc-00650e07da-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.167 DEBUG:boto:path=/ffzdqu-56/a 2023-09-22T21:32:10.167 DEBUG:boto:auth_path=/ffzdqu-56/a 2023-09-22T21:32:10.168 DEBUG:boto:path=/ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.168 DEBUG:boto:auth_path=/ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.168 DEBUG:boto:Method: HEAD 2023-09-22T21:32:10.168 DEBUG:boto:Path: /ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.168 DEBUG:boto:Data: 2023-09-22T21:32:10.168 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.168 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.168 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.168 DEBUG:boto:Params: {} 2023-09-22T21:32:10.169 DEBUG:boto:Token: None 2023-09-22T21:32:10.169 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.169 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6+uvcMeM6auk94bi28YqV+gzsFg= 2023-09-22T21:32:10.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6+uvcMeM6auk94bi28YqV+gzsFg=', 'Content-Length': '0'} 2023-09-22T21:32:10.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003743ecbabbc6db0a-00650e07da-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.172 DEBUG:boto:path=/ffzdqu-56/a 2023-09-22T21:32:10.172 DEBUG:boto:auth_path=/ffzdqu-56/a 2023-09-22T21:32:10.172 DEBUG:boto:path=/ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.172 DEBUG:boto:auth_path=/ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.172 DEBUG:boto:Method: HEAD 2023-09-22T21:32:10.172 DEBUG:boto:Path: /ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.172 DEBUG:boto:Data: 2023-09-22T21:32:10.173 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.173 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.173 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.173 DEBUG:boto:Params: {} 2023-09-22T21:32:10.173 DEBUG:boto:Token: None 2023-09-22T21:32:10.173 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.173 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6+uvcMeM6auk94bi28YqV+gzsFg= 2023-09-22T21:32:10.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6+uvcMeM6auk94bi28YqV+gzsFg=', 'Content-Length': '0'} 2023-09-22T21:32:10.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025c8ced4ade330fa-00650e07da-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:32:10.176 DEBUG:boto:path=/ffzdqu-56/a 2023-09-22T21:32:10.176 DEBUG:boto:auth_path=/ffzdqu-56/a 2023-09-22T21:32:10.176 DEBUG:boto:path=/ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.176 DEBUG:boto:auth_path=/ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.177 DEBUG:boto:Method: GET 2023-09-22T21:32:10.177 DEBUG:boto:Path: /ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.177 DEBUG:boto:Data: 2023-09-22T21:32:10.177 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.177 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.177 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.177 DEBUG:boto:Params: {} 2023-09-22T21:32:10.177 DEBUG:boto:Token: None 2023-09-22T21:32:10.177 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.177 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fqHfjnV7eGJbnL/ar43uMYuEH3Y= 2023-09-22T21:32:10.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fqHfjnV7eGJbnL/ar43uMYuEH3Y=', 'Content-Length': '0'} 2023-09-22T21:32:10.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000036a695bf0ac8c19-00650e07da-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.222 DEBUG:boto:path=/ffzdqu-56/a 2023-09-22T21:32:10.223 DEBUG:boto:auth_path=/ffzdqu-56/a 2023-09-22T21:32:10.223 DEBUG:boto:path=/ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.223 DEBUG:boto:auth_path=/ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.223 DEBUG:boto:Method: GET 2023-09-22T21:32:10.223 DEBUG:boto:Path: /ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.223 DEBUG:boto:Data: 2023-09-22T21:32:10.223 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.223 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.223 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.223 DEBUG:boto:Params: {} 2023-09-22T21:32:10.224 DEBUG:boto:Token: None 2023-09-22T21:32:10.224 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/a?versionId=null 2023-09-22T21:32:10.224 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fqHfjnV7eGJbnL/ar43uMYuEH3Y= 2023-09-22T21:32:10.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fqHfjnV7eGJbnL/ar43uMYuEH3Y=', 'Content-Length': '0'} 2023-09-22T21:32:10.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000957da40c814635fe-00650e07da-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.227 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:10.227 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:10.228 DEBUG:boto:path=/ffzdqu-56/?prefix=a 2023-09-22T21:32:10.228 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=a 2023-09-22T21:32:10.228 DEBUG:boto:Method: GET 2023-09-22T21:32:10.228 DEBUG:boto:Path: /ffzdqu-56/?prefix=a 2023-09-22T21:32:10.228 DEBUG:boto:Data: 2023-09-22T21:32:10.228 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.228 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.228 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.228 DEBUG:boto:Params: {} 2023-09-22T21:32:10.228 DEBUG:boto:Token: None 2023-09-22T21:32:10.229 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/ 2023-09-22T21:32:10.229 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY= 2023-09-22T21:32:10.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY=', 'Content-Length': '0'} 2023-09-22T21:32:10.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021ff0c5dd70e11b1-00650e07da-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.274 DEBUG:boto:b'ffzdqu-56a1000falsea2023-09-22T21:28:28.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:10.275 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:10.275 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:10.275 DEBUG:boto:path=/ffzdqu-56/?prefix=a 2023-09-22T21:32:10.275 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=a 2023-09-22T21:32:10.275 DEBUG:boto:Method: GET 2023-09-22T21:32:10.275 DEBUG:boto:Path: /ffzdqu-56/?prefix=a 2023-09-22T21:32:10.275 DEBUG:boto:Data: 2023-09-22T21:32:10.275 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.276 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.276 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.276 DEBUG:boto:Params: {} 2023-09-22T21:32:10.276 DEBUG:boto:Token: None 2023-09-22T21:32:10.276 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/ 2023-09-22T21:32:10.276 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY= 2023-09-22T21:32:10.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY=', 'Content-Length': '0'} 2023-09-22T21:32:10.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082b63c75099faa70-00650e07da-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.279 DEBUG:boto:b'ffzdqu-56a1000falsea2023-09-22T21:28:28.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:10.280 DEBUG:boto:path=/ffzdqu-56/a 2023-09-22T21:32:10.280 DEBUG:boto:auth_path=/ffzdqu-56/a 2023-09-22T21:32:10.280 DEBUG:boto:Method: HEAD 2023-09-22T21:32:10.280 DEBUG:boto:Path: /ffzdqu-56/a 2023-09-22T21:32:10.280 DEBUG:boto:Data: 2023-09-22T21:32:10.280 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.280 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.280 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.280 DEBUG:boto:Params: {} 2023-09-22T21:32:10.281 DEBUG:boto:Token: None 2023-09-22T21:32:10.281 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/a 2023-09-22T21:32:10.281 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8h9OzZySPegUYru89N+k0cQ4soY= 2023-09-22T21:32:10.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8h9OzZySPegUYru89N+k0cQ4soY=', 'Content-Length': '0'} 2023-09-22T21:32:10.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de22d7a80a71048d-00650e07da-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.284 DEBUG:boto:path=/ffzdqu-56/a 2023-09-22T21:32:10.284 DEBUG:boto:auth_path=/ffzdqu-56/a 2023-09-22T21:32:10.284 DEBUG:boto:Method: HEAD 2023-09-22T21:32:10.284 DEBUG:boto:Path: /ffzdqu-56/a 2023-09-22T21:32:10.284 DEBUG:boto:Data: 2023-09-22T21:32:10.284 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.284 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.285 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.285 DEBUG:boto:Params: {} 2023-09-22T21:32:10.285 DEBUG:boto:Token: None 2023-09-22T21:32:10.285 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/a 2023-09-22T21:32:10.285 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8h9OzZySPegUYru89N+k0cQ4soY= 2023-09-22T21:32:10.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8h9OzZySPegUYru89N+k0cQ4soY=', 'Content-Length': '0'} 2023-09-22T21:32:10.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d418a9536dca16ae-00650e07da-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:32:10.288 DEBUG:boto:path=/ffzdqu-56/a 2023-09-22T21:32:10.288 DEBUG:boto:auth_path=/ffzdqu-56/a 2023-09-22T21:32:10.288 DEBUG:boto:Method: GET 2023-09-22T21:32:10.289 DEBUG:boto:Path: /ffzdqu-56/a 2023-09-22T21:32:10.289 DEBUG:boto:Data: 2023-09-22T21:32:10.289 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.289 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.289 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.289 DEBUG:boto:Params: {} 2023-09-22T21:32:10.289 DEBUG:boto:Token: None 2023-09-22T21:32:10.289 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/a 2023-09-22T21:32:10.289 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dfWSz+kw+61c6IomGoNXZsSgozk= 2023-09-22T21:32:10.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dfWSz+kw+61c6IomGoNXZsSgozk=', 'Content-Length': '0'} 2023-09-22T21:32:10.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a28d6ad85a901e5c-00650e07da-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.334 DEBUG:boto:path=/ffzdqu-56/a 2023-09-22T21:32:10.334 DEBUG:boto:auth_path=/ffzdqu-56/a 2023-09-22T21:32:10.335 DEBUG:boto:Method: GET 2023-09-22T21:32:10.335 DEBUG:boto:Path: /ffzdqu-56/a 2023-09-22T21:32:10.335 DEBUG:boto:Data: 2023-09-22T21:32:10.335 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.335 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.335 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.335 DEBUG:boto:Params: {} 2023-09-22T21:32:10.335 DEBUG:boto:Token: None 2023-09-22T21:32:10.335 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/a 2023-09-22T21:32:10.336 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dfWSz+kw+61c6IomGoNXZsSgozk= 2023-09-22T21:32:10.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dfWSz+kw+61c6IomGoNXZsSgozk=', 'Content-Length': '0'} 2023-09-22T21:32:10.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000173f6cc32cfdd482-00650e07da-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.339 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:10.340 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:10.340 DEBUG:boto:path=/ffzdqu-56/?prefix=a 2023-09-22T21:32:10.340 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=a 2023-09-22T21:32:10.340 DEBUG:boto:Method: GET 2023-09-22T21:32:10.340 DEBUG:boto:Path: /ffzdqu-56/?prefix=a 2023-09-22T21:32:10.340 DEBUG:boto:Data: 2023-09-22T21:32:10.340 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.340 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.340 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.340 DEBUG:boto:Params: {} 2023-09-22T21:32:10.341 DEBUG:boto:Token: None 2023-09-22T21:32:10.341 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/ 2023-09-22T21:32:10.341 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY= 2023-09-22T21:32:10.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY=', 'Content-Length': '0'} 2023-09-22T21:32:10.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c200091c98af8f38-00650e07da-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.386 DEBUG:boto:b'ffzdqu-56a1000falsea2023-09-22T21:28:28.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:10.387 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:10.387 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:10.387 DEBUG:boto:path=/ffzdqu-56/?prefix=a 2023-09-22T21:32:10.387 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=a 2023-09-22T21:32:10.387 DEBUG:boto:Method: GET 2023-09-22T21:32:10.387 DEBUG:boto:Path: /ffzdqu-56/?prefix=a 2023-09-22T21:32:10.387 DEBUG:boto:Data: 2023-09-22T21:32:10.388 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.388 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.388 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.388 DEBUG:boto:Params: {} 2023-09-22T21:32:10.388 DEBUG:boto:Token: None 2023-09-22T21:32:10.388 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/ 2023-09-22T21:32:10.388 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY= 2023-09-22T21:32:10.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY=', 'Content-Length': '0'} 2023-09-22T21:32:10.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f52406934bcb2eb9-00650e07da-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.392 DEBUG:boto:b'ffzdqu-56a1000falsea2023-09-22T21:28:28.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:10.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:32:10.392 DEBUG:boto:path=/ffzdqu-56/b 2023-09-22T21:32:10.393 DEBUG:boto:auth_path=/ffzdqu-56/b 2023-09-22T21:32:10.393 DEBUG:boto:path=/ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.393 DEBUG:boto:auth_path=/ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.393 DEBUG:boto:Method: GET 2023-09-22T21:32:10.393 DEBUG:boto:Path: /ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.393 DEBUG:boto:Data: 2023-09-22T21:32:10.393 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.393 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.393 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.393 DEBUG:boto:Params: {} 2023-09-22T21:32:10.394 DEBUG:boto:Token: None 2023-09-22T21:32:10.394 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.394 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qhWzotOC/IqG08fibpvBoTx4ZwA= 2023-09-22T21:32:10.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qhWzotOC/IqG08fibpvBoTx4ZwA=', 'Content-Length': '0'} 2023-09-22T21:32:10.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b54db7f5ad9cd645-00650e07da-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.397 DEBUG:boto:path=/ffzdqu-56/b 2023-09-22T21:32:10.397 DEBUG:boto:auth_path=/ffzdqu-56/b 2023-09-22T21:32:10.397 DEBUG:boto:path=/ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.398 DEBUG:boto:auth_path=/ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.398 DEBUG:boto:Method: GET 2023-09-22T21:32:10.398 DEBUG:boto:Path: /ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.398 DEBUG:boto:Data: 2023-09-22T21:32:10.398 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.398 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.398 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.398 DEBUG:boto:Params: {} 2023-09-22T21:32:10.398 DEBUG:boto:Token: None 2023-09-22T21:32:10.399 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.399 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qhWzotOC/IqG08fibpvBoTx4ZwA= 2023-09-22T21:32:10.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qhWzotOC/IqG08fibpvBoTx4ZwA=', 'Content-Length': '0'} 2023-09-22T21:32:10.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a066d0d6fab2366-00650e07da-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.402 DEBUG:boto:path=/ffzdqu-56/b 2023-09-22T21:32:10.402 DEBUG:boto:auth_path=/ffzdqu-56/b 2023-09-22T21:32:10.402 DEBUG:boto:path=/ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.402 DEBUG:boto:auth_path=/ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.402 DEBUG:boto:Method: HEAD 2023-09-22T21:32:10.402 DEBUG:boto:Path: /ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.402 DEBUG:boto:Data: 2023-09-22T21:32:10.403 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.403 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.403 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.403 DEBUG:boto:Params: {} 2023-09-22T21:32:10.403 DEBUG:boto:Token: None 2023-09-22T21:32:10.403 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.404 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nkVvAjUArBYbR8mIT9duB0Oe2wo= 2023-09-22T21:32:10.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nkVvAjUArBYbR8mIT9duB0Oe2wo=', 'Content-Length': '0'} 2023-09-22T21:32:10.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029daa13479b124e7-00650e07da-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.407 DEBUG:boto:path=/ffzdqu-56/b 2023-09-22T21:32:10.407 DEBUG:boto:auth_path=/ffzdqu-56/b 2023-09-22T21:32:10.407 DEBUG:boto:path=/ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.407 DEBUG:boto:auth_path=/ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.407 DEBUG:boto:Method: HEAD 2023-09-22T21:32:10.407 DEBUG:boto:Path: /ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.407 DEBUG:boto:Data: 2023-09-22T21:32:10.407 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.408 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.408 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.408 DEBUG:boto:Params: {} 2023-09-22T21:32:10.408 DEBUG:boto:Token: None 2023-09-22T21:32:10.408 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.408 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nkVvAjUArBYbR8mIT9duB0Oe2wo= 2023-09-22T21:32:10.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nkVvAjUArBYbR8mIT9duB0Oe2wo=', 'Content-Length': '0'} 2023-09-22T21:32:10.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e22624cbe3de7f6d-00650e07da-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:32:10.411 DEBUG:boto:path=/ffzdqu-56/b 2023-09-22T21:32:10.412 DEBUG:boto:auth_path=/ffzdqu-56/b 2023-09-22T21:32:10.412 DEBUG:boto:path=/ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.412 DEBUG:boto:auth_path=/ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.412 DEBUG:boto:Method: GET 2023-09-22T21:32:10.412 DEBUG:boto:Path: /ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.412 DEBUG:boto:Data: 2023-09-22T21:32:10.412 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.412 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.412 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.412 DEBUG:boto:Params: {} 2023-09-22T21:32:10.413 DEBUG:boto:Token: None 2023-09-22T21:32:10.413 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.413 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qhWzotOC/IqG08fibpvBoTx4ZwA= 2023-09-22T21:32:10.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qhWzotOC/IqG08fibpvBoTx4ZwA=', 'Content-Length': '0'} 2023-09-22T21:32:10.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0ffdda638655a70-00650e07da-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.458 DEBUG:boto:path=/ffzdqu-56/b 2023-09-22T21:32:10.458 DEBUG:boto:auth_path=/ffzdqu-56/b 2023-09-22T21:32:10.459 DEBUG:boto:path=/ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.459 DEBUG:boto:auth_path=/ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.459 DEBUG:boto:Method: GET 2023-09-22T21:32:10.459 DEBUG:boto:Path: /ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.459 DEBUG:boto:Data: 2023-09-22T21:32:10.459 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.459 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.459 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.460 DEBUG:boto:Params: {} 2023-09-22T21:32:10.460 DEBUG:boto:Token: None 2023-09-22T21:32:10.460 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/b?versionId=null 2023-09-22T21:32:10.460 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qhWzotOC/IqG08fibpvBoTx4ZwA= 2023-09-22T21:32:10.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qhWzotOC/IqG08fibpvBoTx4ZwA=', 'Content-Length': '0'} 2023-09-22T21:32:10.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088631ea825592c00-00650e07da-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.463 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:10.463 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:10.464 DEBUG:boto:path=/ffzdqu-56/?prefix=b 2023-09-22T21:32:10.464 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=b 2023-09-22T21:32:10.464 DEBUG:boto:Method: GET 2023-09-22T21:32:10.464 DEBUG:boto:Path: /ffzdqu-56/?prefix=b 2023-09-22T21:32:10.464 DEBUG:boto:Data: 2023-09-22T21:32:10.464 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.464 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.464 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.464 DEBUG:boto:Params: {} 2023-09-22T21:32:10.465 DEBUG:boto:Token: None 2023-09-22T21:32:10.465 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/ 2023-09-22T21:32:10.465 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY= 2023-09-22T21:32:10.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY=', 'Content-Length': '0'} 2023-09-22T21:32:10.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d8f7d27d888024f-00650e07da-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.514 DEBUG:boto:b'ffzdqu-56b1000falseb2023-09-22T21:28:28.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:10.515 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:10.515 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:10.515 DEBUG:boto:path=/ffzdqu-56/?prefix=b 2023-09-22T21:32:10.515 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=b 2023-09-22T21:32:10.516 DEBUG:boto:Method: GET 2023-09-22T21:32:10.516 DEBUG:boto:Path: /ffzdqu-56/?prefix=b 2023-09-22T21:32:10.516 DEBUG:boto:Data: 2023-09-22T21:32:10.516 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.516 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.516 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.516 DEBUG:boto:Params: {} 2023-09-22T21:32:10.516 DEBUG:boto:Token: None 2023-09-22T21:32:10.517 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/ 2023-09-22T21:32:10.517 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY= 2023-09-22T21:32:10.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY=', 'Content-Length': '0'} 2023-09-22T21:32:10.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002585ae2bc0c544db-00650e07da-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.520 DEBUG:boto:b'ffzdqu-56b1000falseb2023-09-22T21:28:28.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:10.521 DEBUG:boto:path=/ffzdqu-56/b 2023-09-22T21:32:10.521 DEBUG:boto:auth_path=/ffzdqu-56/b 2023-09-22T21:32:10.521 DEBUG:boto:Method: HEAD 2023-09-22T21:32:10.521 DEBUG:boto:Path: /ffzdqu-56/b 2023-09-22T21:32:10.521 DEBUG:boto:Data: 2023-09-22T21:32:10.521 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.521 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.522 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.522 DEBUG:boto:Params: {} 2023-09-22T21:32:10.522 DEBUG:boto:Token: None 2023-09-22T21:32:10.522 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/b 2023-09-22T21:32:10.522 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jvyZ0WzpwFqI6c3Es0ZI/94BinY= 2023-09-22T21:32:10.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jvyZ0WzpwFqI6c3Es0ZI/94BinY=', 'Content-Length': '0'} 2023-09-22T21:32:10.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e564e45ef00d300-00650e07da-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.525 DEBUG:boto:path=/ffzdqu-56/b 2023-09-22T21:32:10.525 DEBUG:boto:auth_path=/ffzdqu-56/b 2023-09-22T21:32:10.525 DEBUG:boto:Method: HEAD 2023-09-22T21:32:10.525 DEBUG:boto:Path: /ffzdqu-56/b 2023-09-22T21:32:10.525 DEBUG:boto:Data: 2023-09-22T21:32:10.526 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.526 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.526 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.526 DEBUG:boto:Params: {} 2023-09-22T21:32:10.526 DEBUG:boto:Token: None 2023-09-22T21:32:10.526 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/b 2023-09-22T21:32:10.526 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jvyZ0WzpwFqI6c3Es0ZI/94BinY= 2023-09-22T21:32:10.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jvyZ0WzpwFqI6c3Es0ZI/94BinY=', 'Content-Length': '0'} 2023-09-22T21:32:10.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006141bf1a41cd3784-00650e07da-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:32:10.529 DEBUG:boto:path=/ffzdqu-56/b 2023-09-22T21:32:10.530 DEBUG:boto:auth_path=/ffzdqu-56/b 2023-09-22T21:32:10.530 DEBUG:boto:Method: GET 2023-09-22T21:32:10.530 DEBUG:boto:Path: /ffzdqu-56/b 2023-09-22T21:32:10.530 DEBUG:boto:Data: 2023-09-22T21:32:10.530 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.530 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.530 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.530 DEBUG:boto:Params: {} 2023-09-22T21:32:10.530 DEBUG:boto:Token: None 2023-09-22T21:32:10.531 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/b 2023-09-22T21:32:10.531 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6R+SpqQkafkQ3LK1/9p6a5muWjE= 2023-09-22T21:32:10.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6R+SpqQkafkQ3LK1/9p6a5muWjE=', 'Content-Length': '0'} 2023-09-22T21:32:10.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079bedc52dbebee1d-00650e07da-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.574 DEBUG:boto:path=/ffzdqu-56/b 2023-09-22T21:32:10.575 DEBUG:boto:auth_path=/ffzdqu-56/b 2023-09-22T21:32:10.575 DEBUG:boto:Method: GET 2023-09-22T21:32:10.575 DEBUG:boto:Path: /ffzdqu-56/b 2023-09-22T21:32:10.575 DEBUG:boto:Data: 2023-09-22T21:32:10.575 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.575 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.575 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.575 DEBUG:boto:Params: {} 2023-09-22T21:32:10.576 DEBUG:boto:Token: None 2023-09-22T21:32:10.576 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/b 2023-09-22T21:32:10.576 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6R+SpqQkafkQ3LK1/9p6a5muWjE= 2023-09-22T21:32:10.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6R+SpqQkafkQ3LK1/9p6a5muWjE=', 'Content-Length': '0'} 2023-09-22T21:32:10.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001583e0c5b424a2f4-00650e07da-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.579 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:10.580 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:10.580 DEBUG:boto:path=/ffzdqu-56/?prefix=b 2023-09-22T21:32:10.580 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=b 2023-09-22T21:32:10.580 DEBUG:boto:Method: GET 2023-09-22T21:32:10.580 DEBUG:boto:Path: /ffzdqu-56/?prefix=b 2023-09-22T21:32:10.580 DEBUG:boto:Data: 2023-09-22T21:32:10.580 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.580 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.580 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.581 DEBUG:boto:Params: {} 2023-09-22T21:32:10.581 DEBUG:boto:Token: None 2023-09-22T21:32:10.581 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/ 2023-09-22T21:32:10.581 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY= 2023-09-22T21:32:10.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY=', 'Content-Length': '0'} 2023-09-22T21:32:10.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098894acf58fa367c-00650e07da-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.626 DEBUG:boto:b'ffzdqu-56b1000falseb2023-09-22T21:28:28.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:10.627 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:10.627 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:10.627 DEBUG:boto:path=/ffzdqu-56/?prefix=b 2023-09-22T21:32:10.627 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=b 2023-09-22T21:32:10.627 DEBUG:boto:Method: GET 2023-09-22T21:32:10.628 DEBUG:boto:Path: /ffzdqu-56/?prefix=b 2023-09-22T21:32:10.628 DEBUG:boto:Data: 2023-09-22T21:32:10.628 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.628 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.628 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.628 DEBUG:boto:Params: {} 2023-09-22T21:32:10.628 DEBUG:boto:Token: None 2023-09-22T21:32:10.628 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/ 2023-09-22T21:32:10.629 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY= 2023-09-22T21:32:10.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY=', 'Content-Length': '0'} 2023-09-22T21:32:10.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bbea203d204a854-00650e07da-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.632 DEBUG:boto:b'ffzdqu-56b1000falseb2023-09-22T21:28:28.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:10.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:32:10.632 DEBUG:boto:path=/ffzdqu-56/c 2023-09-22T21:32:10.632 DEBUG:boto:auth_path=/ffzdqu-56/c 2023-09-22T21:32:10.633 DEBUG:boto:path=/ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.633 DEBUG:boto:auth_path=/ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.633 DEBUG:boto:Method: GET 2023-09-22T21:32:10.633 DEBUG:boto:Path: /ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.633 DEBUG:boto:Data: 2023-09-22T21:32:10.633 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.633 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.633 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.633 DEBUG:boto:Params: {} 2023-09-22T21:32:10.634 DEBUG:boto:Token: None 2023-09-22T21:32:10.634 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.634 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:htwkisQMcrolBnXxI2Aq87zqKeM= 2023-09-22T21:32:10.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:htwkisQMcrolBnXxI2Aq87zqKeM=', 'Content-Length': '0'} 2023-09-22T21:32:10.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3534494522c437f-00650e07da-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.678 DEBUG:boto:path=/ffzdqu-56/c 2023-09-22T21:32:10.678 DEBUG:boto:auth_path=/ffzdqu-56/c 2023-09-22T21:32:10.679 DEBUG:boto:path=/ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.679 DEBUG:boto:auth_path=/ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.679 DEBUG:boto:Method: GET 2023-09-22T21:32:10.679 DEBUG:boto:Path: /ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.679 DEBUG:boto:Data: 2023-09-22T21:32:10.679 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.679 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.679 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.679 DEBUG:boto:Params: {} 2023-09-22T21:32:10.680 DEBUG:boto:Token: None 2023-09-22T21:32:10.680 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.680 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:htwkisQMcrolBnXxI2Aq87zqKeM= 2023-09-22T21:32:10.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:htwkisQMcrolBnXxI2Aq87zqKeM=', 'Content-Length': '0'} 2023-09-22T21:32:10.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004060b14516aad593-00650e07da-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.683 DEBUG:boto:path=/ffzdqu-56/c 2023-09-22T21:32:10.683 DEBUG:boto:auth_path=/ffzdqu-56/c 2023-09-22T21:32:10.683 DEBUG:boto:path=/ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.683 DEBUG:boto:auth_path=/ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.683 DEBUG:boto:Method: HEAD 2023-09-22T21:32:10.683 DEBUG:boto:Path: /ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.684 DEBUG:boto:Data: 2023-09-22T21:32:10.684 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.684 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.684 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.684 DEBUG:boto:Params: {} 2023-09-22T21:32:10.684 DEBUG:boto:Token: None 2023-09-22T21:32:10.684 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.684 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Wfav26zSA2u3uIY/GKpadCJ/3mc= 2023-09-22T21:32:10.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Wfav26zSA2u3uIY/GKpadCJ/3mc=', 'Content-Length': '0'} 2023-09-22T21:32:10.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077044ab9997c4871-00650e07da-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.687 DEBUG:boto:path=/ffzdqu-56/c 2023-09-22T21:32:10.687 DEBUG:boto:auth_path=/ffzdqu-56/c 2023-09-22T21:32:10.687 DEBUG:boto:path=/ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.688 DEBUG:boto:auth_path=/ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.688 DEBUG:boto:Method: HEAD 2023-09-22T21:32:10.688 DEBUG:boto:Path: /ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.688 DEBUG:boto:Data: 2023-09-22T21:32:10.688 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.688 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.688 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.688 DEBUG:boto:Params: {} 2023-09-22T21:32:10.688 DEBUG:boto:Token: None 2023-09-22T21:32:10.689 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.689 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Wfav26zSA2u3uIY/GKpadCJ/3mc= 2023-09-22T21:32:10.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Wfav26zSA2u3uIY/GKpadCJ/3mc=', 'Content-Length': '0'} 2023-09-22T21:32:10.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f39244a2416aca73-00650e07da-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:32:10.692 DEBUG:boto:path=/ffzdqu-56/c 2023-09-22T21:32:10.692 DEBUG:boto:auth_path=/ffzdqu-56/c 2023-09-22T21:32:10.692 DEBUG:boto:path=/ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.692 DEBUG:boto:auth_path=/ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.692 DEBUG:boto:Method: GET 2023-09-22T21:32:10.692 DEBUG:boto:Path: /ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.693 DEBUG:boto:Data: 2023-09-22T21:32:10.693 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.693 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.693 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.693 DEBUG:boto:Params: {} 2023-09-22T21:32:10.693 DEBUG:boto:Token: None 2023-09-22T21:32:10.693 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.693 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:htwkisQMcrolBnXxI2Aq87zqKeM= 2023-09-22T21:32:10.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:htwkisQMcrolBnXxI2Aq87zqKeM=', 'Content-Length': '0'} 2023-09-22T21:32:10.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c50071f2ac736228-00650e07da-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.738 DEBUG:boto:path=/ffzdqu-56/c 2023-09-22T21:32:10.738 DEBUG:boto:auth_path=/ffzdqu-56/c 2023-09-22T21:32:10.739 DEBUG:boto:path=/ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.739 DEBUG:boto:auth_path=/ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.739 DEBUG:boto:Method: GET 2023-09-22T21:32:10.739 DEBUG:boto:Path: /ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.739 DEBUG:boto:Data: 2023-09-22T21:32:10.739 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.739 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.739 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.739 DEBUG:boto:Params: {} 2023-09-22T21:32:10.739 DEBUG:boto:Token: None 2023-09-22T21:32:10.740 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/c?versionId=null 2023-09-22T21:32:10.740 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:htwkisQMcrolBnXxI2Aq87zqKeM= 2023-09-22T21:32:10.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:htwkisQMcrolBnXxI2Aq87zqKeM=', 'Content-Length': '0'} 2023-09-22T21:32:10.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6dfc3601dac06a4-00650e07da-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.743 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:10.743 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:10.743 DEBUG:boto:path=/ffzdqu-56/?prefix=c 2023-09-22T21:32:10.743 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=c 2023-09-22T21:32:10.744 DEBUG:boto:Method: GET 2023-09-22T21:32:10.744 DEBUG:boto:Path: /ffzdqu-56/?prefix=c 2023-09-22T21:32:10.744 DEBUG:boto:Data: 2023-09-22T21:32:10.744 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.744 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.744 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.744 DEBUG:boto:Params: {} 2023-09-22T21:32:10.744 DEBUG:boto:Token: None 2023-09-22T21:32:10.745 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/ 2023-09-22T21:32:10.745 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY= 2023-09-22T21:32:10.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY=', 'Content-Length': '0'} 2023-09-22T21:32:10.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000687e5c0e5da37978-00650e07da-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.790 DEBUG:boto:b'ffzdqu-56c1000falsec2023-09-22T21:28:28.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:10.791 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:10.791 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:10.791 DEBUG:boto:path=/ffzdqu-56/?prefix=c 2023-09-22T21:32:10.791 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=c 2023-09-22T21:32:10.791 DEBUG:boto:Method: GET 2023-09-22T21:32:10.791 DEBUG:boto:Path: /ffzdqu-56/?prefix=c 2023-09-22T21:32:10.791 DEBUG:boto:Data: 2023-09-22T21:32:10.791 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.792 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.792 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.792 DEBUG:boto:Params: {} 2023-09-22T21:32:10.792 DEBUG:boto:Token: None 2023-09-22T21:32:10.792 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/ 2023-09-22T21:32:10.792 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY= 2023-09-22T21:32:10.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY=', 'Content-Length': '0'} 2023-09-22T21:32:10.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b33306f4e95226d2-00650e07da-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.796 DEBUG:boto:b'ffzdqu-56c1000falsec2023-09-22T21:28:28.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:10.797 DEBUG:boto:path=/ffzdqu-56/c 2023-09-22T21:32:10.797 DEBUG:boto:auth_path=/ffzdqu-56/c 2023-09-22T21:32:10.797 DEBUG:boto:Method: HEAD 2023-09-22T21:32:10.797 DEBUG:boto:Path: /ffzdqu-56/c 2023-09-22T21:32:10.797 DEBUG:boto:Data: 2023-09-22T21:32:10.797 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.797 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.797 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.798 DEBUG:boto:Params: {} 2023-09-22T21:32:10.798 DEBUG:boto:Token: None 2023-09-22T21:32:10.798 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/c 2023-09-22T21:32:10.798 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:erKwTLX48vtZT1U6oJxavwyRZNg= 2023-09-22T21:32:10.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:erKwTLX48vtZT1U6oJxavwyRZNg=', 'Content-Length': '0'} 2023-09-22T21:32:10.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ced461c2776e2e5d-00650e07da-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.802 DEBUG:boto:path=/ffzdqu-56/c 2023-09-22T21:32:10.802 DEBUG:boto:auth_path=/ffzdqu-56/c 2023-09-22T21:32:10.802 DEBUG:boto:Method: HEAD 2023-09-22T21:32:10.802 DEBUG:boto:Path: /ffzdqu-56/c 2023-09-22T21:32:10.802 DEBUG:boto:Data: 2023-09-22T21:32:10.802 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.802 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.803 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.803 DEBUG:boto:Params: {} 2023-09-22T21:32:10.803 DEBUG:boto:Token: None 2023-09-22T21:32:10.803 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/c 2023-09-22T21:32:10.803 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:erKwTLX48vtZT1U6oJxavwyRZNg= 2023-09-22T21:32:10.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:erKwTLX48vtZT1U6oJxavwyRZNg=', 'Content-Length': '0'} 2023-09-22T21:32:10.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4fc53d14fd6e3d5-00650e07da-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:32:10.807 DEBUG:boto:path=/ffzdqu-56/c 2023-09-22T21:32:10.807 DEBUG:boto:auth_path=/ffzdqu-56/c 2023-09-22T21:32:10.807 DEBUG:boto:Method: GET 2023-09-22T21:32:10.807 DEBUG:boto:Path: /ffzdqu-56/c 2023-09-22T21:32:10.807 DEBUG:boto:Data: 2023-09-22T21:32:10.808 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.808 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.808 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.808 DEBUG:boto:Params: {} 2023-09-22T21:32:10.808 DEBUG:boto:Token: None 2023-09-22T21:32:10.808 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/c 2023-09-22T21:32:10.808 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h/gC3KKC2Bi9W6hBw3RoVzzVr3Q= 2023-09-22T21:32:10.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h/gC3KKC2Bi9W6hBw3RoVzzVr3Q=', 'Content-Length': '0'} 2023-09-22T21:32:10.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051d52aebbc9686c1-00650e07da-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.854 DEBUG:boto:path=/ffzdqu-56/c 2023-09-22T21:32:10.854 DEBUG:boto:auth_path=/ffzdqu-56/c 2023-09-22T21:32:10.855 DEBUG:boto:Method: GET 2023-09-22T21:32:10.855 DEBUG:boto:Path: /ffzdqu-56/c 2023-09-22T21:32:10.855 DEBUG:boto:Data: 2023-09-22T21:32:10.855 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.855 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.856 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.856 DEBUG:boto:Params: {} 2023-09-22T21:32:10.856 DEBUG:boto:Token: None 2023-09-22T21:32:10.856 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/c 2023-09-22T21:32:10.856 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h/gC3KKC2Bi9W6hBw3RoVzzVr3Q= 2023-09-22T21:32:10.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h/gC3KKC2Bi9W6hBw3RoVzzVr3Q=', 'Content-Length': '0'} 2023-09-22T21:32:10.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dfdc04a04c0da65-00650e07da-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.859 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:10.859 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:10.859 DEBUG:boto:path=/ffzdqu-56/?prefix=c 2023-09-22T21:32:10.860 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=c 2023-09-22T21:32:10.860 DEBUG:boto:Method: GET 2023-09-22T21:32:10.860 DEBUG:boto:Path: /ffzdqu-56/?prefix=c 2023-09-22T21:32:10.860 DEBUG:boto:Data: 2023-09-22T21:32:10.860 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.860 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.860 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.860 DEBUG:boto:Params: {} 2023-09-22T21:32:10.860 DEBUG:boto:Token: None 2023-09-22T21:32:10.860 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/ 2023-09-22T21:32:10.861 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY= 2023-09-22T21:32:10.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY=', 'Content-Length': '0'} 2023-09-22T21:32:10.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec380bf6f6569ed5-00650e07da-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.906 DEBUG:boto:b'ffzdqu-56c1000falsec2023-09-22T21:28:28.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:10.907 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:10.907 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:10.907 DEBUG:boto:path=/ffzdqu-56/?prefix=c 2023-09-22T21:32:10.907 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=c 2023-09-22T21:32:10.907 DEBUG:boto:Method: GET 2023-09-22T21:32:10.907 DEBUG:boto:Path: /ffzdqu-56/?prefix=c 2023-09-22T21:32:10.907 DEBUG:boto:Data: 2023-09-22T21:32:10.907 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.908 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.908 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.908 DEBUG:boto:Params: {} 2023-09-22T21:32:10.908 DEBUG:boto:Token: None 2023-09-22T21:32:10.908 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/ 2023-09-22T21:32:10.908 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY= 2023-09-22T21:32:10.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ty2t9eCQ8joGzvHGoIlcPVUO2qY=', 'Content-Length': '0'} 2023-09-22T21:32:10.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000083e80ea5e094bbe-00650e07da-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.911 DEBUG:boto:b'ffzdqu-56c1000falsec2023-09-22T21:28:28.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:10.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:32:10.912 DEBUG:boto:path=/ffzdqu-56/d 2023-09-22T21:32:10.912 DEBUG:boto:auth_path=/ffzdqu-56/d 2023-09-22T21:32:10.912 DEBUG:boto:path=/ffzdqu-56/d?versionId=null 2023-09-22T21:32:10.912 DEBUG:boto:auth_path=/ffzdqu-56/d?versionId=null 2023-09-22T21:32:10.912 DEBUG:boto:Method: GET 2023-09-22T21:32:10.912 DEBUG:boto:Path: /ffzdqu-56/d?versionId=null 2023-09-22T21:32:10.913 DEBUG:boto:Data: 2023-09-22T21:32:10.913 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.913 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.913 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.913 DEBUG:boto:Params: {} 2023-09-22T21:32:10.913 DEBUG:boto:Token: None 2023-09-22T21:32:10.913 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/d?versionId=null 2023-09-22T21:32:10.913 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WN1goaZClkohWzAeCruULpHkU2U= 2023-09-22T21:32:10.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WN1goaZClkohWzAeCruULpHkU2U=', 'Content-Length': '0'} 2023-09-22T21:32:10.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000167eb4abd7149671-00650e07da-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.958 DEBUG:boto:path=/ffzdqu-56/d 2023-09-22T21:32:10.958 DEBUG:boto:auth_path=/ffzdqu-56/d 2023-09-22T21:32:10.959 DEBUG:boto:path=/ffzdqu-56/d?versionId=null 2023-09-22T21:32:10.959 DEBUG:boto:auth_path=/ffzdqu-56/d?versionId=null 2023-09-22T21:32:10.959 DEBUG:boto:Method: GET 2023-09-22T21:32:10.959 DEBUG:boto:Path: /ffzdqu-56/d?versionId=null 2023-09-22T21:32:10.959 DEBUG:boto:Data: 2023-09-22T21:32:10.959 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.959 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.959 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.959 DEBUG:boto:Params: {} 2023-09-22T21:32:10.960 DEBUG:boto:Token: None 2023-09-22T21:32:10.960 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/d?versionId=null 2023-09-22T21:32:10.960 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WN1goaZClkohWzAeCruULpHkU2U= 2023-09-22T21:32:10.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WN1goaZClkohWzAeCruULpHkU2U=', 'Content-Length': '0'} 2023-09-22T21:32:10.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000398c200c56e86770-00650e07da-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.963 DEBUG:boto:path=/ffzdqu-56/d 2023-09-22T21:32:10.963 DEBUG:boto:auth_path=/ffzdqu-56/d 2023-09-22T21:32:10.963 DEBUG:boto:path=/ffzdqu-56/d?versionId=null 2023-09-22T21:32:10.964 DEBUG:boto:auth_path=/ffzdqu-56/d?versionId=null 2023-09-22T21:32:10.964 DEBUG:boto:Method: HEAD 2023-09-22T21:32:10.964 DEBUG:boto:Path: /ffzdqu-56/d?versionId=null 2023-09-22T21:32:10.964 DEBUG:boto:Data: 2023-09-22T21:32:10.964 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.964 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.964 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.964 DEBUG:boto:Params: {} 2023-09-22T21:32:10.964 DEBUG:boto:Token: None 2023-09-22T21:32:10.965 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/d?versionId=null 2023-09-22T21:32:10.965 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xZT6RieoXN4V9G18nNTPRILe5V8= 2023-09-22T21:32:10.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xZT6RieoXN4V9G18nNTPRILe5V8=', 'Content-Length': '0'} 2023-09-22T21:32:10.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b00e8bd855e05b0-00650e07da-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.967 DEBUG:boto:path=/ffzdqu-56/d 2023-09-22T21:32:10.968 DEBUG:boto:auth_path=/ffzdqu-56/d 2023-09-22T21:32:10.968 DEBUG:boto:path=/ffzdqu-56/d?versionId=null 2023-09-22T21:32:10.968 DEBUG:boto:auth_path=/ffzdqu-56/d?versionId=null 2023-09-22T21:32:10.968 DEBUG:boto:Method: HEAD 2023-09-22T21:32:10.968 DEBUG:boto:Path: /ffzdqu-56/d?versionId=null 2023-09-22T21:32:10.968 DEBUG:boto:Data: 2023-09-22T21:32:10.968 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.968 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:10.968 DEBUG:boto:Port: 8000 2023-09-22T21:32:10.969 DEBUG:boto:Params: {} 2023-09-22T21:32:10.969 DEBUG:boto:Token: None 2023-09-22T21:32:10.969 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/d?versionId=null 2023-09-22T21:32:10.969 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xZT6RieoXN4V9G18nNTPRILe5V8= 2023-09-22T21:32:10.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xZT6RieoXN4V9G18nNTPRILe5V8=', 'Content-Length': '0'} 2023-09-22T21:32:10.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088b07e16d38c4dc0-00650e07da-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:10.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:32:10.972 DEBUG:boto:path=/ffzdqu-56/d 2023-09-22T21:32:10.972 DEBUG:boto:auth_path=/ffzdqu-56/d 2023-09-22T21:32:10.972 DEBUG:boto:path=/ffzdqu-56/d?versionId=null 2023-09-22T21:32:10.972 DEBUG:boto:auth_path=/ffzdqu-56/d?versionId=null 2023-09-22T21:32:10.972 DEBUG:boto:Method: GET 2023-09-22T21:32:10.973 DEBUG:boto:Path: /ffzdqu-56/d?versionId=null 2023-09-22T21:32:10.973 DEBUG:boto:Data: 2023-09-22T21:32:10.973 DEBUG:boto:Headers: {} 2023-09-22T21:32:10.973 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:10.973 DEBUG:boto:Port: 8001 2023-09-22T21:32:10.973 DEBUG:boto:Params: {} 2023-09-22T21:32:10.973 DEBUG:boto:Token: None 2023-09-22T21:32:10.973 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:10 GMT /ffzdqu-56/d?versionId=null 2023-09-22T21:32:10.973 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WN1goaZClkohWzAeCruULpHkU2U= 2023-09-22T21:32:10.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WN1goaZClkohWzAeCruULpHkU2U=', 'Content-Length': '0'} 2023-09-22T21:32:10.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8f0f1ef6176dc55-00650e07da-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.018 DEBUG:boto:path=/ffzdqu-56/d 2023-09-22T21:32:11.019 DEBUG:boto:auth_path=/ffzdqu-56/d 2023-09-22T21:32:11.019 DEBUG:boto:path=/ffzdqu-56/d?versionId=null 2023-09-22T21:32:11.019 DEBUG:boto:auth_path=/ffzdqu-56/d?versionId=null 2023-09-22T21:32:11.019 DEBUG:boto:Method: GET 2023-09-22T21:32:11.019 DEBUG:boto:Path: /ffzdqu-56/d?versionId=null 2023-09-22T21:32:11.019 DEBUG:boto:Data: 2023-09-22T21:32:11.019 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.019 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.019 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.020 DEBUG:boto:Params: {} 2023-09-22T21:32:11.020 DEBUG:boto:Token: None 2023-09-22T21:32:11.020 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/d?versionId=null 2023-09-22T21:32:11.020 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5mWLronx4cUacbdRy+/L16BZYSQ= 2023-09-22T21:32:11.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5mWLronx4cUacbdRy+/L16BZYSQ=', 'Content-Length': '0'} 2023-09-22T21:32:11.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa02773907bc740e-00650e07db-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.023 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:11.023 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:11.023 DEBUG:boto:path=/ffzdqu-56/?prefix=d 2023-09-22T21:32:11.024 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=d 2023-09-22T21:32:11.024 DEBUG:boto:Method: GET 2023-09-22T21:32:11.024 DEBUG:boto:Path: /ffzdqu-56/?prefix=d 2023-09-22T21:32:11.024 DEBUG:boto:Data: 2023-09-22T21:32:11.024 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.024 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.024 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.024 DEBUG:boto:Params: {} 2023-09-22T21:32:11.024 DEBUG:boto:Token: None 2023-09-22T21:32:11.024 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/ 2023-09-22T21:32:11.025 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4= 2023-09-22T21:32:11.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4=', 'Content-Length': '0'} 2023-09-22T21:32:11.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024620b32b012bcb6-00650e07db-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.070 DEBUG:boto:b'ffzdqu-56d1000falsed2023-09-22T21:28:28.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:11.071 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:11.071 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:11.071 DEBUG:boto:path=/ffzdqu-56/?prefix=d 2023-09-22T21:32:11.071 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=d 2023-09-22T21:32:11.072 DEBUG:boto:Method: GET 2023-09-22T21:32:11.072 DEBUG:boto:Path: /ffzdqu-56/?prefix=d 2023-09-22T21:32:11.072 DEBUG:boto:Data: 2023-09-22T21:32:11.072 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.072 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.072 DEBUG:boto:Params: {} 2023-09-22T21:32:11.072 DEBUG:boto:Token: None 2023-09-22T21:32:11.073 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/ 2023-09-22T21:32:11.073 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4= 2023-09-22T21:32:11.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4=', 'Content-Length': '0'} 2023-09-22T21:32:11.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c95698878321c65-00650e07db-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.076 DEBUG:boto:b'ffzdqu-56d1000falsed2023-09-22T21:28:28.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:11.077 DEBUG:boto:path=/ffzdqu-56/d 2023-09-22T21:32:11.077 DEBUG:boto:auth_path=/ffzdqu-56/d 2023-09-22T21:32:11.077 DEBUG:boto:Method: HEAD 2023-09-22T21:32:11.077 DEBUG:boto:Path: /ffzdqu-56/d 2023-09-22T21:32:11.077 DEBUG:boto:Data: 2023-09-22T21:32:11.077 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.077 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.078 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.078 DEBUG:boto:Params: {} 2023-09-22T21:32:11.078 DEBUG:boto:Token: None 2023-09-22T21:32:11.078 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/d 2023-09-22T21:32:11.078 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t5x46Zq85W/aZI6YOWbMBP+RcRw= 2023-09-22T21:32:11.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t5x46Zq85W/aZI6YOWbMBP+RcRw=', 'Content-Length': '0'} 2023-09-22T21:32:11.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c09e8420cb7228f5-00650e07db-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.081 DEBUG:boto:path=/ffzdqu-56/d 2023-09-22T21:32:11.081 DEBUG:boto:auth_path=/ffzdqu-56/d 2023-09-22T21:32:11.081 DEBUG:boto:Method: HEAD 2023-09-22T21:32:11.081 DEBUG:boto:Path: /ffzdqu-56/d 2023-09-22T21:32:11.081 DEBUG:boto:Data: 2023-09-22T21:32:11.081 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.082 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.082 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.082 DEBUG:boto:Params: {} 2023-09-22T21:32:11.082 DEBUG:boto:Token: None 2023-09-22T21:32:11.082 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/d 2023-09-22T21:32:11.082 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t5x46Zq85W/aZI6YOWbMBP+RcRw= 2023-09-22T21:32:11.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t5x46Zq85W/aZI6YOWbMBP+RcRw=', 'Content-Length': '0'} 2023-09-22T21:32:11.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dc231ee59b0d0cc-00650e07db-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:32:11.085 DEBUG:boto:path=/ffzdqu-56/d 2023-09-22T21:32:11.085 DEBUG:boto:auth_path=/ffzdqu-56/d 2023-09-22T21:32:11.085 DEBUG:boto:Method: GET 2023-09-22T21:32:11.085 DEBUG:boto:Path: /ffzdqu-56/d 2023-09-22T21:32:11.086 DEBUG:boto:Data: 2023-09-22T21:32:11.086 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.086 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.086 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.086 DEBUG:boto:Params: {} 2023-09-22T21:32:11.086 DEBUG:boto:Token: None 2023-09-22T21:32:11.086 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/d 2023-09-22T21:32:11.086 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RuIEu/YtUveDJ14OCYd7ZV2RGWQ= 2023-09-22T21:32:11.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RuIEu/YtUveDJ14OCYd7ZV2RGWQ=', 'Content-Length': '0'} 2023-09-22T21:32:11.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e8b4f365dce8cb4-00650e07db-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.130 DEBUG:boto:path=/ffzdqu-56/d 2023-09-22T21:32:11.130 DEBUG:boto:auth_path=/ffzdqu-56/d 2023-09-22T21:32:11.131 DEBUG:boto:Method: GET 2023-09-22T21:32:11.131 DEBUG:boto:Path: /ffzdqu-56/d 2023-09-22T21:32:11.131 DEBUG:boto:Data: 2023-09-22T21:32:11.131 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.131 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.131 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.131 DEBUG:boto:Params: {} 2023-09-22T21:32:11.131 DEBUG:boto:Token: None 2023-09-22T21:32:11.131 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/d 2023-09-22T21:32:11.132 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RuIEu/YtUveDJ14OCYd7ZV2RGWQ= 2023-09-22T21:32:11.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RuIEu/YtUveDJ14OCYd7ZV2RGWQ=', 'Content-Length': '0'} 2023-09-22T21:32:11.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002792ed8ba009844-00650e07db-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.135 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:11.135 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:11.135 DEBUG:boto:path=/ffzdqu-56/?prefix=d 2023-09-22T21:32:11.135 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=d 2023-09-22T21:32:11.135 DEBUG:boto:Method: GET 2023-09-22T21:32:11.135 DEBUG:boto:Path: /ffzdqu-56/?prefix=d 2023-09-22T21:32:11.135 DEBUG:boto:Data: 2023-09-22T21:32:11.135 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.135 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.135 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.136 DEBUG:boto:Params: {} 2023-09-22T21:32:11.136 DEBUG:boto:Token: None 2023-09-22T21:32:11.136 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/ 2023-09-22T21:32:11.136 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4= 2023-09-22T21:32:11.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4=', 'Content-Length': '0'} 2023-09-22T21:32:11.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ddc09453a599ede-00650e07db-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.141 DEBUG:boto:b'ffzdqu-56d1000falsed2023-09-22T21:28:28.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:11.142 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:11.142 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:11.142 DEBUG:boto:path=/ffzdqu-56/?prefix=d 2023-09-22T21:32:11.142 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=d 2023-09-22T21:32:11.142 DEBUG:boto:Method: GET 2023-09-22T21:32:11.142 DEBUG:boto:Path: /ffzdqu-56/?prefix=d 2023-09-22T21:32:11.142 DEBUG:boto:Data: 2023-09-22T21:32:11.142 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.142 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.143 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.143 DEBUG:boto:Params: {} 2023-09-22T21:32:11.143 DEBUG:boto:Token: None 2023-09-22T21:32:11.143 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/ 2023-09-22T21:32:11.143 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4= 2023-09-22T21:32:11.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4=', 'Content-Length': '0'} 2023-09-22T21:32:11.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000260449b92ef7a4c4-00650e07db-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.146 DEBUG:boto:b'ffzdqu-56d1000falsed2023-09-22T21:28:28.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:11.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-22T21:32:11.147 DEBUG:boto:path=/ffzdqu-56/e 2023-09-22T21:32:11.147 DEBUG:boto:auth_path=/ffzdqu-56/e 2023-09-22T21:32:11.147 DEBUG:boto:path=/ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.147 DEBUG:boto:auth_path=/ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.147 DEBUG:boto:Method: GET 2023-09-22T21:32:11.147 DEBUG:boto:Path: /ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.147 DEBUG:boto:Data: 2023-09-22T21:32:11.148 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.148 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.148 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.148 DEBUG:boto:Params: {} 2023-09-22T21:32:11.148 DEBUG:boto:Token: None 2023-09-22T21:32:11.148 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.148 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WRVs40GDSkAE9vi66zZfa/laGp4= 2023-09-22T21:32:11.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WRVs40GDSkAE9vi66zZfa/laGp4=', 'Content-Length': '0'} 2023-09-22T21:32:11.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2ad5c0f3c3afd63-00650e07db-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.194 DEBUG:boto:path=/ffzdqu-56/e 2023-09-22T21:32:11.194 DEBUG:boto:auth_path=/ffzdqu-56/e 2023-09-22T21:32:11.195 DEBUG:boto:path=/ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.195 DEBUG:boto:auth_path=/ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.195 DEBUG:boto:Method: GET 2023-09-22T21:32:11.195 DEBUG:boto:Path: /ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.195 DEBUG:boto:Data: 2023-09-22T21:32:11.195 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.195 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.195 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.195 DEBUG:boto:Params: {} 2023-09-22T21:32:11.195 DEBUG:boto:Token: None 2023-09-22T21:32:11.196 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.196 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WRVs40GDSkAE9vi66zZfa/laGp4= 2023-09-22T21:32:11.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WRVs40GDSkAE9vi66zZfa/laGp4=', 'Content-Length': '0'} 2023-09-22T21:32:11.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005292c2a37807490e-00650e07db-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.199 DEBUG:boto:path=/ffzdqu-56/e 2023-09-22T21:32:11.199 DEBUG:boto:auth_path=/ffzdqu-56/e 2023-09-22T21:32:11.199 DEBUG:boto:path=/ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.199 DEBUG:boto:auth_path=/ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.199 DEBUG:boto:Method: HEAD 2023-09-22T21:32:11.199 DEBUG:boto:Path: /ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.200 DEBUG:boto:Data: 2023-09-22T21:32:11.200 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.200 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.200 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.200 DEBUG:boto:Params: {} 2023-09-22T21:32:11.200 DEBUG:boto:Token: None 2023-09-22T21:32:11.200 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.200 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pZbHt38KBqld05RuSX4W2v9b8R0= 2023-09-22T21:32:11.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pZbHt38KBqld05RuSX4W2v9b8R0=', 'Content-Length': '0'} 2023-09-22T21:32:11.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b06297bef0295c53-00650e07db-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.203 DEBUG:boto:path=/ffzdqu-56/e 2023-09-22T21:32:11.203 DEBUG:boto:auth_path=/ffzdqu-56/e 2023-09-22T21:32:11.203 DEBUG:boto:path=/ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.204 DEBUG:boto:auth_path=/ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.204 DEBUG:boto:Method: HEAD 2023-09-22T21:32:11.204 DEBUG:boto:Path: /ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.204 DEBUG:boto:Data: 2023-09-22T21:32:11.204 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.204 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.204 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.204 DEBUG:boto:Params: {} 2023-09-22T21:32:11.204 DEBUG:boto:Token: None 2023-09-22T21:32:11.205 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.205 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pZbHt38KBqld05RuSX4W2v9b8R0= 2023-09-22T21:32:11.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pZbHt38KBqld05RuSX4W2v9b8R0=', 'Content-Length': '0'} 2023-09-22T21:32:11.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee362de9952b8879-00650e07db-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-22T21:32:11.208 DEBUG:boto:path=/ffzdqu-56/e 2023-09-22T21:32:11.208 DEBUG:boto:auth_path=/ffzdqu-56/e 2023-09-22T21:32:11.208 DEBUG:boto:path=/ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.208 DEBUG:boto:auth_path=/ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.209 DEBUG:boto:Method: GET 2023-09-22T21:32:11.209 DEBUG:boto:Path: /ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.209 DEBUG:boto:Data: 2023-09-22T21:32:11.209 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.209 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.209 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.209 DEBUG:boto:Params: {} 2023-09-22T21:32:11.209 DEBUG:boto:Token: None 2023-09-22T21:32:11.209 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.209 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WRVs40GDSkAE9vi66zZfa/laGp4= 2023-09-22T21:32:11.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WRVs40GDSkAE9vi66zZfa/laGp4=', 'Content-Length': '0'} 2023-09-22T21:32:11.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000067358276373b889-00650e07db-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.254 DEBUG:boto:path=/ffzdqu-56/e 2023-09-22T21:32:11.255 DEBUG:boto:auth_path=/ffzdqu-56/e 2023-09-22T21:32:11.255 DEBUG:boto:path=/ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.255 DEBUG:boto:auth_path=/ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.255 DEBUG:boto:Method: GET 2023-09-22T21:32:11.255 DEBUG:boto:Path: /ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.255 DEBUG:boto:Data: 2023-09-22T21:32:11.255 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.255 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.255 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.256 DEBUG:boto:Params: {} 2023-09-22T21:32:11.256 DEBUG:boto:Token: None 2023-09-22T21:32:11.256 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/e?versionId=null 2023-09-22T21:32:11.256 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WRVs40GDSkAE9vi66zZfa/laGp4= 2023-09-22T21:32:11.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WRVs40GDSkAE9vi66zZfa/laGp4=', 'Content-Length': '0'} 2023-09-22T21:32:11.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed7a0562160a7a0d-00650e07db-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.259 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:11.259 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:11.260 DEBUG:boto:path=/ffzdqu-56/?prefix=e 2023-09-22T21:32:11.260 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=e 2023-09-22T21:32:11.260 DEBUG:boto:Method: GET 2023-09-22T21:32:11.260 DEBUG:boto:Path: /ffzdqu-56/?prefix=e 2023-09-22T21:32:11.260 DEBUG:boto:Data: 2023-09-22T21:32:11.260 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.260 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.260 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.260 DEBUG:boto:Params: {} 2023-09-22T21:32:11.261 DEBUG:boto:Token: None 2023-09-22T21:32:11.261 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/ 2023-09-22T21:32:11.261 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4= 2023-09-22T21:32:11.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4=', 'Content-Length': '0'} 2023-09-22T21:32:11.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fab96c275c08473-00650e07db-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.306 DEBUG:boto:b'ffzdqu-56e1000falsee2023-09-22T21:28:49.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:11.307 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:11.307 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:11.307 DEBUG:boto:path=/ffzdqu-56/?prefix=e 2023-09-22T21:32:11.307 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=e 2023-09-22T21:32:11.307 DEBUG:boto:Method: GET 2023-09-22T21:32:11.307 DEBUG:boto:Path: /ffzdqu-56/?prefix=e 2023-09-22T21:32:11.308 DEBUG:boto:Data: 2023-09-22T21:32:11.308 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.308 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.308 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.308 DEBUG:boto:Params: {} 2023-09-22T21:32:11.308 DEBUG:boto:Token: None 2023-09-22T21:32:11.308 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/ 2023-09-22T21:32:11.308 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4= 2023-09-22T21:32:11.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4=', 'Content-Length': '0'} 2023-09-22T21:32:11.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c53bec26ad4009c-00650e07db-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.312 DEBUG:boto:b'ffzdqu-56e1000falsee2023-09-22T21:28:49.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:11.312 DEBUG:boto:path=/ffzdqu-56/e 2023-09-22T21:32:11.312 DEBUG:boto:auth_path=/ffzdqu-56/e 2023-09-22T21:32:11.312 DEBUG:boto:Method: HEAD 2023-09-22T21:32:11.313 DEBUG:boto:Path: /ffzdqu-56/e 2023-09-22T21:32:11.313 DEBUG:boto:Data: 2023-09-22T21:32:11.313 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.313 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.313 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.313 DEBUG:boto:Params: {} 2023-09-22T21:32:11.313 DEBUG:boto:Token: None 2023-09-22T21:32:11.313 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/e 2023-09-22T21:32:11.314 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F51cBX/Eofrf/C41JiTA+ZKKevc= 2023-09-22T21:32:11.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F51cBX/Eofrf/C41JiTA+ZKKevc=', 'Content-Length': '0'} 2023-09-22T21:32:11.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b42360312c75a720-00650e07db-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.316 DEBUG:boto:path=/ffzdqu-56/e 2023-09-22T21:32:11.316 DEBUG:boto:auth_path=/ffzdqu-56/e 2023-09-22T21:32:11.317 DEBUG:boto:Method: HEAD 2023-09-22T21:32:11.317 DEBUG:boto:Path: /ffzdqu-56/e 2023-09-22T21:32:11.317 DEBUG:boto:Data: 2023-09-22T21:32:11.317 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.317 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.317 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.317 DEBUG:boto:Params: {} 2023-09-22T21:32:11.317 DEBUG:boto:Token: None 2023-09-22T21:32:11.318 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/e 2023-09-22T21:32:11.318 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F51cBX/Eofrf/C41JiTA+ZKKevc= 2023-09-22T21:32:11.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F51cBX/Eofrf/C41JiTA+ZKKevc=', 'Content-Length': '0'} 2023-09-22T21:32:11.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f54a4ec66dce269e-00650e07db-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-22T21:32:11.321 DEBUG:boto:path=/ffzdqu-56/e 2023-09-22T21:32:11.321 DEBUG:boto:auth_path=/ffzdqu-56/e 2023-09-22T21:32:11.321 DEBUG:boto:Method: GET 2023-09-22T21:32:11.321 DEBUG:boto:Path: /ffzdqu-56/e 2023-09-22T21:32:11.321 DEBUG:boto:Data: 2023-09-22T21:32:11.321 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.321 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.321 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.322 DEBUG:boto:Params: {} 2023-09-22T21:32:11.322 DEBUG:boto:Token: None 2023-09-22T21:32:11.322 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/e 2023-09-22T21:32:11.322 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kRZfnliQsnUNVY5E874kiFLXdqc= 2023-09-22T21:32:11.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kRZfnliQsnUNVY5E874kiFLXdqc=', 'Content-Length': '0'} 2023-09-22T21:32:11.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e00d1d6179ed286-00650e07db-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.366 DEBUG:boto:path=/ffzdqu-56/e 2023-09-22T21:32:11.366 DEBUG:boto:auth_path=/ffzdqu-56/e 2023-09-22T21:32:11.367 DEBUG:boto:Method: GET 2023-09-22T21:32:11.367 DEBUG:boto:Path: /ffzdqu-56/e 2023-09-22T21:32:11.367 DEBUG:boto:Data: 2023-09-22T21:32:11.367 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.367 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.367 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.367 DEBUG:boto:Params: {} 2023-09-22T21:32:11.367 DEBUG:boto:Token: None 2023-09-22T21:32:11.368 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/e 2023-09-22T21:32:11.368 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kRZfnliQsnUNVY5E874kiFLXdqc= 2023-09-22T21:32:11.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kRZfnliQsnUNVY5E874kiFLXdqc=', 'Content-Length': '0'} 2023-09-22T21:32:11.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df8418aeb4d42558-00650e07db-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.371 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:11.371 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:11.372 DEBUG:boto:path=/ffzdqu-56/?prefix=e 2023-09-22T21:32:11.372 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=e 2023-09-22T21:32:11.372 DEBUG:boto:Method: GET 2023-09-22T21:32:11.372 DEBUG:boto:Path: /ffzdqu-56/?prefix=e 2023-09-22T21:32:11.372 DEBUG:boto:Data: 2023-09-22T21:32:11.372 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.372 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.372 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.372 DEBUG:boto:Params: {} 2023-09-22T21:32:11.373 DEBUG:boto:Token: None 2023-09-22T21:32:11.373 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/ 2023-09-22T21:32:11.373 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4= 2023-09-22T21:32:11.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4=', 'Content-Length': '0'} 2023-09-22T21:32:11.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f62f0a8dbc0385b4-00650e07db-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.377 DEBUG:boto:b'ffzdqu-56e1000falsee2023-09-22T21:28:49.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:11.378 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:11.378 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:11.378 DEBUG:boto:path=/ffzdqu-56/?prefix=e 2023-09-22T21:32:11.378 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=e 2023-09-22T21:32:11.378 DEBUG:boto:Method: GET 2023-09-22T21:32:11.379 DEBUG:boto:Path: /ffzdqu-56/?prefix=e 2023-09-22T21:32:11.379 DEBUG:boto:Data: 2023-09-22T21:32:11.379 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.379 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.379 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.379 DEBUG:boto:Params: {} 2023-09-22T21:32:11.379 DEBUG:boto:Token: None 2023-09-22T21:32:11.379 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/ 2023-09-22T21:32:11.379 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4= 2023-09-22T21:32:11.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4=', 'Content-Length': '0'} 2023-09-22T21:32:11.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ef86948b1af3bee-00650e07db-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.383 DEBUG:boto:b'ffzdqu-56e1000falsee2023-09-22T21:28:49.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:11.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-22T21:32:11.384 DEBUG:boto:path=/ffzdqu-56/f 2023-09-22T21:32:11.384 DEBUG:boto:auth_path=/ffzdqu-56/f 2023-09-22T21:32:11.384 DEBUG:boto:path=/ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.384 DEBUG:boto:auth_path=/ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.384 DEBUG:boto:Method: GET 2023-09-22T21:32:11.384 DEBUG:boto:Path: /ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.384 DEBUG:boto:Data: 2023-09-22T21:32:11.384 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.384 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.385 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.385 DEBUG:boto:Params: {} 2023-09-22T21:32:11.385 DEBUG:boto:Token: None 2023-09-22T21:32:11.385 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.385 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xDCTFgvJVHkIbWced14oiWGDOwU= 2023-09-22T21:32:11.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xDCTFgvJVHkIbWced14oiWGDOwU=', 'Content-Length': '0'} 2023-09-22T21:32:11.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078da39ec1aba466c-00650e07db-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.430 DEBUG:boto:path=/ffzdqu-56/f 2023-09-22T21:32:11.431 DEBUG:boto:auth_path=/ffzdqu-56/f 2023-09-22T21:32:11.431 DEBUG:boto:path=/ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.431 DEBUG:boto:auth_path=/ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.431 DEBUG:boto:Method: GET 2023-09-22T21:32:11.431 DEBUG:boto:Path: /ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.431 DEBUG:boto:Data: 2023-09-22T21:32:11.431 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.432 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.432 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.432 DEBUG:boto:Params: {} 2023-09-22T21:32:11.432 DEBUG:boto:Token: None 2023-09-22T21:32:11.432 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.432 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xDCTFgvJVHkIbWced14oiWGDOwU= 2023-09-22T21:32:11.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xDCTFgvJVHkIbWced14oiWGDOwU=', 'Content-Length': '0'} 2023-09-22T21:32:11.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000687ce7e99ec9396a-00650e07db-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.436 DEBUG:boto:path=/ffzdqu-56/f 2023-09-22T21:32:11.436 DEBUG:boto:auth_path=/ffzdqu-56/f 2023-09-22T21:32:11.436 DEBUG:boto:path=/ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.436 DEBUG:boto:auth_path=/ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.436 DEBUG:boto:Method: HEAD 2023-09-22T21:32:11.436 DEBUG:boto:Path: /ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.436 DEBUG:boto:Data: 2023-09-22T21:32:11.437 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.437 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.437 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.437 DEBUG:boto:Params: {} 2023-09-22T21:32:11.437 DEBUG:boto:Token: None 2023-09-22T21:32:11.437 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.437 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NXKq2BqrRgUevfUjAJhcOe5h4lI= 2023-09-22T21:32:11.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NXKq2BqrRgUevfUjAJhcOe5h4lI=', 'Content-Length': '0'} 2023-09-22T21:32:11.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000239b4da1c39ebd00-00650e07db-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.440 DEBUG:boto:path=/ffzdqu-56/f 2023-09-22T21:32:11.441 DEBUG:boto:auth_path=/ffzdqu-56/f 2023-09-22T21:32:11.441 DEBUG:boto:path=/ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.441 DEBUG:boto:auth_path=/ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.441 DEBUG:boto:Method: HEAD 2023-09-22T21:32:11.441 DEBUG:boto:Path: /ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.441 DEBUG:boto:Data: 2023-09-22T21:32:11.441 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.441 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.442 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.442 DEBUG:boto:Params: {} 2023-09-22T21:32:11.442 DEBUG:boto:Token: None 2023-09-22T21:32:11.442 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.442 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NXKq2BqrRgUevfUjAJhcOe5h4lI= 2023-09-22T21:32:11.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NXKq2BqrRgUevfUjAJhcOe5h4lI=', 'Content-Length': '0'} 2023-09-22T21:32:11.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002df9f305b19cc28b-00650e07db-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-22T21:32:11.445 DEBUG:boto:path=/ffzdqu-56/f 2023-09-22T21:32:11.445 DEBUG:boto:auth_path=/ffzdqu-56/f 2023-09-22T21:32:11.445 DEBUG:boto:path=/ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.446 DEBUG:boto:auth_path=/ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.446 DEBUG:boto:Method: GET 2023-09-22T21:32:11.446 DEBUG:boto:Path: /ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.446 DEBUG:boto:Data: 2023-09-22T21:32:11.446 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.446 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.446 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.446 DEBUG:boto:Params: {} 2023-09-22T21:32:11.447 DEBUG:boto:Token: None 2023-09-22T21:32:11.447 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.447 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xDCTFgvJVHkIbWced14oiWGDOwU= 2023-09-22T21:32:11.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xDCTFgvJVHkIbWced14oiWGDOwU=', 'Content-Length': '0'} 2023-09-22T21:32:11.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9060f82b5b3a1e8-00650e07db-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.494 DEBUG:boto:path=/ffzdqu-56/f 2023-09-22T21:32:11.495 DEBUG:boto:auth_path=/ffzdqu-56/f 2023-09-22T21:32:11.495 DEBUG:boto:path=/ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.495 DEBUG:boto:auth_path=/ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.495 DEBUG:boto:Method: GET 2023-09-22T21:32:11.495 DEBUG:boto:Path: /ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.495 DEBUG:boto:Data: 2023-09-22T21:32:11.495 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.495 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.495 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.495 DEBUG:boto:Params: {} 2023-09-22T21:32:11.496 DEBUG:boto:Token: None 2023-09-22T21:32:11.496 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/f?versionId=null 2023-09-22T21:32:11.496 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xDCTFgvJVHkIbWced14oiWGDOwU= 2023-09-22T21:32:11.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xDCTFgvJVHkIbWced14oiWGDOwU=', 'Content-Length': '0'} 2023-09-22T21:32:11.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca818bad53ef4736-00650e07db-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.499 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:11.499 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:11.500 DEBUG:boto:path=/ffzdqu-56/?prefix=f 2023-09-22T21:32:11.500 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=f 2023-09-22T21:32:11.500 DEBUG:boto:Method: GET 2023-09-22T21:32:11.500 DEBUG:boto:Path: /ffzdqu-56/?prefix=f 2023-09-22T21:32:11.500 DEBUG:boto:Data: 2023-09-22T21:32:11.500 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.500 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.500 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.500 DEBUG:boto:Params: {} 2023-09-22T21:32:11.500 DEBUG:boto:Token: None 2023-09-22T21:32:11.501 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/ 2023-09-22T21:32:11.501 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4= 2023-09-22T21:32:11.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4=', 'Content-Length': '0'} 2023-09-22T21:32:11.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e44beeeb59390d3c-00650e07db-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.550 DEBUG:boto:b'ffzdqu-56f1000falsef2023-09-22T21:28:49.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:11.551 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:11.551 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:11.551 DEBUG:boto:path=/ffzdqu-56/?prefix=f 2023-09-22T21:32:11.551 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=f 2023-09-22T21:32:11.551 DEBUG:boto:Method: GET 2023-09-22T21:32:11.551 DEBUG:boto:Path: /ffzdqu-56/?prefix=f 2023-09-22T21:32:11.552 DEBUG:boto:Data: 2023-09-22T21:32:11.552 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.552 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.552 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.552 DEBUG:boto:Params: {} 2023-09-22T21:32:11.552 DEBUG:boto:Token: None 2023-09-22T21:32:11.552 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/ 2023-09-22T21:32:11.552 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4= 2023-09-22T21:32:11.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4=', 'Content-Length': '0'} 2023-09-22T21:32:11.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1581ce25a430412-00650e07db-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.556 DEBUG:boto:b'ffzdqu-56f1000falsef2023-09-22T21:28:49.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:11.556 DEBUG:boto:path=/ffzdqu-56/f 2023-09-22T21:32:11.556 DEBUG:boto:auth_path=/ffzdqu-56/f 2023-09-22T21:32:11.557 DEBUG:boto:Method: HEAD 2023-09-22T21:32:11.557 DEBUG:boto:Path: /ffzdqu-56/f 2023-09-22T21:32:11.557 DEBUG:boto:Data: 2023-09-22T21:32:11.557 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.557 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.557 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.557 DEBUG:boto:Params: {} 2023-09-22T21:32:11.557 DEBUG:boto:Token: None 2023-09-22T21:32:11.557 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/f 2023-09-22T21:32:11.558 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sCMVnEx+VfI9HSjpGKzUnWYKaZY= 2023-09-22T21:32:11.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sCMVnEx+VfI9HSjpGKzUnWYKaZY=', 'Content-Length': '0'} 2023-09-22T21:32:11.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000568086f0c635b3fd-00650e07db-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.560 DEBUG:boto:path=/ffzdqu-56/f 2023-09-22T21:32:11.561 DEBUG:boto:auth_path=/ffzdqu-56/f 2023-09-22T21:32:11.561 DEBUG:boto:Method: HEAD 2023-09-22T21:32:11.561 DEBUG:boto:Path: /ffzdqu-56/f 2023-09-22T21:32:11.561 DEBUG:boto:Data: 2023-09-22T21:32:11.561 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.561 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.561 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.561 DEBUG:boto:Params: {} 2023-09-22T21:32:11.561 DEBUG:boto:Token: None 2023-09-22T21:32:11.562 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/f 2023-09-22T21:32:11.562 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sCMVnEx+VfI9HSjpGKzUnWYKaZY= 2023-09-22T21:32:11.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sCMVnEx+VfI9HSjpGKzUnWYKaZY=', 'Content-Length': '0'} 2023-09-22T21:32:11.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008add20404b6e6c7a-00650e07db-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-22T21:32:11.565 DEBUG:boto:path=/ffzdqu-56/f 2023-09-22T21:32:11.565 DEBUG:boto:auth_path=/ffzdqu-56/f 2023-09-22T21:32:11.565 DEBUG:boto:Method: GET 2023-09-22T21:32:11.565 DEBUG:boto:Path: /ffzdqu-56/f 2023-09-22T21:32:11.565 DEBUG:boto:Data: 2023-09-22T21:32:11.565 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.565 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.566 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.566 DEBUG:boto:Params: {} 2023-09-22T21:32:11.566 DEBUG:boto:Token: None 2023-09-22T21:32:11.566 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/f 2023-09-22T21:32:11.566 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zwKvQX3u+qBsEXaXDPdZGZysQ6Y= 2023-09-22T21:32:11.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zwKvQX3u+qBsEXaXDPdZGZysQ6Y=', 'Content-Length': '0'} 2023-09-22T21:32:11.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d748cffdfd7e86bc-00650e07db-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.610 DEBUG:boto:path=/ffzdqu-56/f 2023-09-22T21:32:11.611 DEBUG:boto:auth_path=/ffzdqu-56/f 2023-09-22T21:32:11.611 DEBUG:boto:Method: GET 2023-09-22T21:32:11.611 DEBUG:boto:Path: /ffzdqu-56/f 2023-09-22T21:32:11.611 DEBUG:boto:Data: 2023-09-22T21:32:11.611 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.611 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.611 DEBUG:boto:Params: {} 2023-09-22T21:32:11.612 DEBUG:boto:Token: None 2023-09-22T21:32:11.612 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/f 2023-09-22T21:32:11.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zwKvQX3u+qBsEXaXDPdZGZysQ6Y= 2023-09-22T21:32:11.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zwKvQX3u+qBsEXaXDPdZGZysQ6Y=', 'Content-Length': '0'} 2023-09-22T21:32:11.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052ad9b223c92dae0-00650e07db-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.615 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:11.615 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:11.615 DEBUG:boto:path=/ffzdqu-56/?prefix=f 2023-09-22T21:32:11.616 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=f 2023-09-22T21:32:11.616 DEBUG:boto:Method: GET 2023-09-22T21:32:11.616 DEBUG:boto:Path: /ffzdqu-56/?prefix=f 2023-09-22T21:32:11.616 DEBUG:boto:Data: 2023-09-22T21:32:11.616 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.616 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.616 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.616 DEBUG:boto:Params: {} 2023-09-22T21:32:11.617 DEBUG:boto:Token: None 2023-09-22T21:32:11.617 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/ 2023-09-22T21:32:11.617 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4= 2023-09-22T21:32:11.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4=', 'Content-Length': '0'} 2023-09-22T21:32:11.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004aa27e3ab991bf8f-00650e07db-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.622 DEBUG:boto:b'ffzdqu-56f1000falsef2023-09-22T21:28:49.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:11.622 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:11.622 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:11.622 DEBUG:boto:path=/ffzdqu-56/?prefix=f 2023-09-22T21:32:11.623 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=f 2023-09-22T21:32:11.623 DEBUG:boto:Method: GET 2023-09-22T21:32:11.623 DEBUG:boto:Path: /ffzdqu-56/?prefix=f 2023-09-22T21:32:11.623 DEBUG:boto:Data: 2023-09-22T21:32:11.623 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.623 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.623 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.623 DEBUG:boto:Params: {} 2023-09-22T21:32:11.623 DEBUG:boto:Token: None 2023-09-22T21:32:11.624 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/ 2023-09-22T21:32:11.624 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4= 2023-09-22T21:32:11.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4=', 'Content-Length': '0'} 2023-09-22T21:32:11.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000592d91261873c90d-00650e07db-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.627 DEBUG:boto:b'ffzdqu-56f1000falsef2023-09-22T21:28:49.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:11.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-22T21:32:11.628 DEBUG:boto:path=/ffzdqu-56/g 2023-09-22T21:32:11.628 DEBUG:boto:auth_path=/ffzdqu-56/g 2023-09-22T21:32:11.628 DEBUG:boto:path=/ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.628 DEBUG:boto:auth_path=/ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.628 DEBUG:boto:Method: GET 2023-09-22T21:32:11.628 DEBUG:boto:Path: /ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.628 DEBUG:boto:Data: 2023-09-22T21:32:11.629 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.629 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.629 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.629 DEBUG:boto:Params: {} 2023-09-22T21:32:11.629 DEBUG:boto:Token: None 2023-09-22T21:32:11.629 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.629 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aJjlun+GXIf0sHCsZvfNvYUEEfU= 2023-09-22T21:32:11.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aJjlun+GXIf0sHCsZvfNvYUEEfU=', 'Content-Length': '0'} 2023-09-22T21:32:11.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd300db246e91c58-00650e07db-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.674 DEBUG:boto:path=/ffzdqu-56/g 2023-09-22T21:32:11.675 DEBUG:boto:auth_path=/ffzdqu-56/g 2023-09-22T21:32:11.675 DEBUG:boto:path=/ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.675 DEBUG:boto:auth_path=/ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.675 DEBUG:boto:Method: GET 2023-09-22T21:32:11.675 DEBUG:boto:Path: /ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.675 DEBUG:boto:Data: 2023-09-22T21:32:11.675 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.676 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.676 DEBUG:boto:Params: {} 2023-09-22T21:32:11.676 DEBUG:boto:Token: None 2023-09-22T21:32:11.676 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.676 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aJjlun+GXIf0sHCsZvfNvYUEEfU= 2023-09-22T21:32:11.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aJjlun+GXIf0sHCsZvfNvYUEEfU=', 'Content-Length': '0'} 2023-09-22T21:32:11.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc9bb03dd0340e49-00650e07db-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.679 DEBUG:boto:path=/ffzdqu-56/g 2023-09-22T21:32:11.680 DEBUG:boto:auth_path=/ffzdqu-56/g 2023-09-22T21:32:11.680 DEBUG:boto:path=/ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.680 DEBUG:boto:auth_path=/ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.680 DEBUG:boto:Method: HEAD 2023-09-22T21:32:11.680 DEBUG:boto:Path: /ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.680 DEBUG:boto:Data: 2023-09-22T21:32:11.680 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.680 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.681 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.681 DEBUG:boto:Params: {} 2023-09-22T21:32:11.681 DEBUG:boto:Token: None 2023-09-22T21:32:11.681 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.681 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6gjZnV3d8zz3iDxXbUQoyfQuEYo= 2023-09-22T21:32:11.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6gjZnV3d8zz3iDxXbUQoyfQuEYo=', 'Content-Length': '0'} 2023-09-22T21:32:11.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ee6ec22d997f587-00650e07db-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.684 DEBUG:boto:path=/ffzdqu-56/g 2023-09-22T21:32:11.684 DEBUG:boto:auth_path=/ffzdqu-56/g 2023-09-22T21:32:11.684 DEBUG:boto:path=/ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.684 DEBUG:boto:auth_path=/ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.685 DEBUG:boto:Method: HEAD 2023-09-22T21:32:11.685 DEBUG:boto:Path: /ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.685 DEBUG:boto:Data: 2023-09-22T21:32:11.685 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.685 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.685 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.685 DEBUG:boto:Params: {} 2023-09-22T21:32:11.685 DEBUG:boto:Token: None 2023-09-22T21:32:11.686 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.686 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6gjZnV3d8zz3iDxXbUQoyfQuEYo= 2023-09-22T21:32:11.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6gjZnV3d8zz3iDxXbUQoyfQuEYo=', 'Content-Length': '0'} 2023-09-22T21:32:11.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2c5c07b3eab48b4-00650e07db-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-22T21:32:11.689 DEBUG:boto:path=/ffzdqu-56/g 2023-09-22T21:32:11.689 DEBUG:boto:auth_path=/ffzdqu-56/g 2023-09-22T21:32:11.689 DEBUG:boto:path=/ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.689 DEBUG:boto:auth_path=/ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.690 DEBUG:boto:Method: GET 2023-09-22T21:32:11.690 DEBUG:boto:Path: /ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.690 DEBUG:boto:Data: 2023-09-22T21:32:11.690 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.690 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.690 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.690 DEBUG:boto:Params: {} 2023-09-22T21:32:11.690 DEBUG:boto:Token: None 2023-09-22T21:32:11.690 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.691 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aJjlun+GXIf0sHCsZvfNvYUEEfU= 2023-09-22T21:32:11.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aJjlun+GXIf0sHCsZvfNvYUEEfU=', 'Content-Length': '0'} 2023-09-22T21:32:11.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bff56fff95453217-00650e07db-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.734 DEBUG:boto:path=/ffzdqu-56/g 2023-09-22T21:32:11.735 DEBUG:boto:auth_path=/ffzdqu-56/g 2023-09-22T21:32:11.735 DEBUG:boto:path=/ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.735 DEBUG:boto:auth_path=/ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.735 DEBUG:boto:Method: GET 2023-09-22T21:32:11.735 DEBUG:boto:Path: /ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.735 DEBUG:boto:Data: 2023-09-22T21:32:11.735 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.735 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.735 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.736 DEBUG:boto:Params: {} 2023-09-22T21:32:11.736 DEBUG:boto:Token: None 2023-09-22T21:32:11.736 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/g?versionId=null 2023-09-22T21:32:11.736 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aJjlun+GXIf0sHCsZvfNvYUEEfU= 2023-09-22T21:32:11.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aJjlun+GXIf0sHCsZvfNvYUEEfU=', 'Content-Length': '0'} 2023-09-22T21:32:11.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009efe416110f63950-00650e07db-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.739 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:11.739 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:11.739 DEBUG:boto:path=/ffzdqu-56/?prefix=g 2023-09-22T21:32:11.740 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=g 2023-09-22T21:32:11.740 DEBUG:boto:Method: GET 2023-09-22T21:32:11.740 DEBUG:boto:Path: /ffzdqu-56/?prefix=g 2023-09-22T21:32:11.740 DEBUG:boto:Data: 2023-09-22T21:32:11.740 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.740 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.740 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.740 DEBUG:boto:Params: {} 2023-09-22T21:32:11.741 DEBUG:boto:Token: None 2023-09-22T21:32:11.741 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/ 2023-09-22T21:32:11.741 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4= 2023-09-22T21:32:11.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4=', 'Content-Length': '0'} 2023-09-22T21:32:11.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001453f51356ae28a3-00650e07db-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.786 DEBUG:boto:b'ffzdqu-56g1000falseg2023-09-22T21:28:49.711Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:11.787 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:11.787 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:11.787 DEBUG:boto:path=/ffzdqu-56/?prefix=g 2023-09-22T21:32:11.787 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=g 2023-09-22T21:32:11.787 DEBUG:boto:Method: GET 2023-09-22T21:32:11.787 DEBUG:boto:Path: /ffzdqu-56/?prefix=g 2023-09-22T21:32:11.788 DEBUG:boto:Data: 2023-09-22T21:32:11.788 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.788 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.788 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.788 DEBUG:boto:Params: {} 2023-09-22T21:32:11.788 DEBUG:boto:Token: None 2023-09-22T21:32:11.788 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/ 2023-09-22T21:32:11.788 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4= 2023-09-22T21:32:11.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4=', 'Content-Length': '0'} 2023-09-22T21:32:11.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b255b78a69b5b1a6-00650e07db-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.792 DEBUG:boto:b'ffzdqu-56g1000falseg2023-09-22T21:28:49.711Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:11.793 DEBUG:boto:path=/ffzdqu-56/g 2023-09-22T21:32:11.793 DEBUG:boto:auth_path=/ffzdqu-56/g 2023-09-22T21:32:11.793 DEBUG:boto:Method: HEAD 2023-09-22T21:32:11.793 DEBUG:boto:Path: /ffzdqu-56/g 2023-09-22T21:32:11.793 DEBUG:boto:Data: 2023-09-22T21:32:11.793 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.793 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.794 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.794 DEBUG:boto:Params: {} 2023-09-22T21:32:11.794 DEBUG:boto:Token: None 2023-09-22T21:32:11.794 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/g 2023-09-22T21:32:11.794 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cR5rPY3dG5PVvHGMhWVyTVcHz6g= 2023-09-22T21:32:11.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cR5rPY3dG5PVvHGMhWVyTVcHz6g=', 'Content-Length': '0'} 2023-09-22T21:32:11.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004530213120e6f8ba-00650e07db-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.797 DEBUG:boto:path=/ffzdqu-56/g 2023-09-22T21:32:11.797 DEBUG:boto:auth_path=/ffzdqu-56/g 2023-09-22T21:32:11.797 DEBUG:boto:Method: HEAD 2023-09-22T21:32:11.798 DEBUG:boto:Path: /ffzdqu-56/g 2023-09-22T21:32:11.798 DEBUG:boto:Data: 2023-09-22T21:32:11.798 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.798 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.798 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.798 DEBUG:boto:Params: {} 2023-09-22T21:32:11.798 DEBUG:boto:Token: None 2023-09-22T21:32:11.799 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/g 2023-09-22T21:32:11.799 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cR5rPY3dG5PVvHGMhWVyTVcHz6g= 2023-09-22T21:32:11.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cR5rPY3dG5PVvHGMhWVyTVcHz6g=', 'Content-Length': '0'} 2023-09-22T21:32:11.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b97b52ff392187cd-00650e07db-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-22T21:32:11.802 DEBUG:boto:path=/ffzdqu-56/g 2023-09-22T21:32:11.802 DEBUG:boto:auth_path=/ffzdqu-56/g 2023-09-22T21:32:11.802 DEBUG:boto:Method: GET 2023-09-22T21:32:11.802 DEBUG:boto:Path: /ffzdqu-56/g 2023-09-22T21:32:11.802 DEBUG:boto:Data: 2023-09-22T21:32:11.802 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.802 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.802 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.803 DEBUG:boto:Params: {} 2023-09-22T21:32:11.803 DEBUG:boto:Token: None 2023-09-22T21:32:11.803 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/g 2023-09-22T21:32:11.803 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTZ/+A72IvahX3wuQScyIOm4V+E= 2023-09-22T21:32:11.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTZ/+A72IvahX3wuQScyIOm4V+E=', 'Content-Length': '0'} 2023-09-22T21:32:11.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f03d1d80b7ce3f5-00650e07db-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.847 DEBUG:boto:path=/ffzdqu-56/g 2023-09-22T21:32:11.848 DEBUG:boto:auth_path=/ffzdqu-56/g 2023-09-22T21:32:11.848 DEBUG:boto:Method: GET 2023-09-22T21:32:11.848 DEBUG:boto:Path: /ffzdqu-56/g 2023-09-22T21:32:11.848 DEBUG:boto:Data: 2023-09-22T21:32:11.848 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.848 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.848 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.848 DEBUG:boto:Params: {} 2023-09-22T21:32:11.849 DEBUG:boto:Token: None 2023-09-22T21:32:11.849 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/g 2023-09-22T21:32:11.849 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTZ/+A72IvahX3wuQScyIOm4V+E= 2023-09-22T21:32:11.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTZ/+A72IvahX3wuQScyIOm4V+E=', 'Content-Length': '0'} 2023-09-22T21:32:11.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000078617c2caebaf16-00650e07db-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.852 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:11.853 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:11.853 DEBUG:boto:path=/ffzdqu-56/?prefix=g 2023-09-22T21:32:11.853 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=g 2023-09-22T21:32:11.853 DEBUG:boto:Method: GET 2023-09-22T21:32:11.853 DEBUG:boto:Path: /ffzdqu-56/?prefix=g 2023-09-22T21:32:11.853 DEBUG:boto:Data: 2023-09-22T21:32:11.853 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.853 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.853 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.853 DEBUG:boto:Params: {} 2023-09-22T21:32:11.854 DEBUG:boto:Token: None 2023-09-22T21:32:11.854 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/ 2023-09-22T21:32:11.854 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4= 2023-09-22T21:32:11.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4=', 'Content-Length': '0'} 2023-09-22T21:32:11.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094608c09cb93fca1-00650e07db-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.859 DEBUG:boto:b'ffzdqu-56g1000falseg2023-09-22T21:28:49.711Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:11.859 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:11.859 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:11.859 DEBUG:boto:path=/ffzdqu-56/?prefix=g 2023-09-22T21:32:11.859 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=g 2023-09-22T21:32:11.860 DEBUG:boto:Method: GET 2023-09-22T21:32:11.860 DEBUG:boto:Path: /ffzdqu-56/?prefix=g 2023-09-22T21:32:11.860 DEBUG:boto:Data: 2023-09-22T21:32:11.860 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.860 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.860 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.860 DEBUG:boto:Params: {} 2023-09-22T21:32:11.860 DEBUG:boto:Token: None 2023-09-22T21:32:11.860 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/ 2023-09-22T21:32:11.861 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4= 2023-09-22T21:32:11.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4=', 'Content-Length': '0'} 2023-09-22T21:32:11.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042d08c78882b8f76-00650e07db-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.864 DEBUG:boto:b'ffzdqu-56g1000falseg2023-09-22T21:28:49.711Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:11.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-22T21:32:11.865 DEBUG:boto:path=/ffzdqu-56/h 2023-09-22T21:32:11.865 DEBUG:boto:auth_path=/ffzdqu-56/h 2023-09-22T21:32:11.865 DEBUG:boto:path=/ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.865 DEBUG:boto:auth_path=/ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.865 DEBUG:boto:Method: GET 2023-09-22T21:32:11.865 DEBUG:boto:Path: /ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.865 DEBUG:boto:Data: 2023-09-22T21:32:11.865 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.865 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.865 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.866 DEBUG:boto:Params: {} 2023-09-22T21:32:11.866 DEBUG:boto:Token: None 2023-09-22T21:32:11.866 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.866 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0psRJGJU7G+/MBtJ43mPTQTann8= 2023-09-22T21:32:11.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0psRJGJU7G+/MBtJ43mPTQTann8=', 'Content-Length': '0'} 2023-09-22T21:32:11.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e1b8f6ece5eaa36-00650e07db-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.910 DEBUG:boto:path=/ffzdqu-56/h 2023-09-22T21:32:11.910 DEBUG:boto:auth_path=/ffzdqu-56/h 2023-09-22T21:32:11.911 DEBUG:boto:path=/ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.911 DEBUG:boto:auth_path=/ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.911 DEBUG:boto:Method: GET 2023-09-22T21:32:11.911 DEBUG:boto:Path: /ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.911 DEBUG:boto:Data: 2023-09-22T21:32:11.911 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.911 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.911 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.911 DEBUG:boto:Params: {} 2023-09-22T21:32:11.912 DEBUG:boto:Token: None 2023-09-22T21:32:11.912 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.912 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0psRJGJU7G+/MBtJ43mPTQTann8= 2023-09-22T21:32:11.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0psRJGJU7G+/MBtJ43mPTQTann8=', 'Content-Length': '0'} 2023-09-22T21:32:11.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efd2301661613222-00650e07db-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.915 DEBUG:boto:path=/ffzdqu-56/h 2023-09-22T21:32:11.915 DEBUG:boto:auth_path=/ffzdqu-56/h 2023-09-22T21:32:11.915 DEBUG:boto:path=/ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.915 DEBUG:boto:auth_path=/ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.916 DEBUG:boto:Method: HEAD 2023-09-22T21:32:11.916 DEBUG:boto:Path: /ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.916 DEBUG:boto:Data: 2023-09-22T21:32:11.916 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.916 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.916 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.916 DEBUG:boto:Params: {} 2023-09-22T21:32:11.916 DEBUG:boto:Token: None 2023-09-22T21:32:11.916 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.916 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aKb+FDR1PGy2aXP+qWcn1ab+gsE= 2023-09-22T21:32:11.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aKb+FDR1PGy2aXP+qWcn1ab+gsE=', 'Content-Length': '0'} 2023-09-22T21:32:11.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c062b6fe2030353f-00650e07db-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.919 DEBUG:boto:path=/ffzdqu-56/h 2023-09-22T21:32:11.919 DEBUG:boto:auth_path=/ffzdqu-56/h 2023-09-22T21:32:11.919 DEBUG:boto:path=/ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.920 DEBUG:boto:auth_path=/ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.920 DEBUG:boto:Method: HEAD 2023-09-22T21:32:11.920 DEBUG:boto:Path: /ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.920 DEBUG:boto:Data: 2023-09-22T21:32:11.920 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.920 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.920 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.920 DEBUG:boto:Params: {} 2023-09-22T21:32:11.920 DEBUG:boto:Token: None 2023-09-22T21:32:11.920 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.921 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aKb+FDR1PGy2aXP+qWcn1ab+gsE= 2023-09-22T21:32:11.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aKb+FDR1PGy2aXP+qWcn1ab+gsE=', 'Content-Length': '0'} 2023-09-22T21:32:11.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0bd0f36b57301e9-00650e07db-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-22T21:32:11.923 DEBUG:boto:path=/ffzdqu-56/h 2023-09-22T21:32:11.924 DEBUG:boto:auth_path=/ffzdqu-56/h 2023-09-22T21:32:11.924 DEBUG:boto:path=/ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.924 DEBUG:boto:auth_path=/ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.924 DEBUG:boto:Method: GET 2023-09-22T21:32:11.924 DEBUG:boto:Path: /ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.924 DEBUG:boto:Data: 2023-09-22T21:32:11.924 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.924 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.924 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.925 DEBUG:boto:Params: {} 2023-09-22T21:32:11.925 DEBUG:boto:Token: None 2023-09-22T21:32:11.925 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.925 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0psRJGJU7G+/MBtJ43mPTQTann8= 2023-09-22T21:32:11.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0psRJGJU7G+/MBtJ43mPTQTann8=', 'Content-Length': '0'} 2023-09-22T21:32:11.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033315ed60fcda3aa-00650e07db-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.970 DEBUG:boto:path=/ffzdqu-56/h 2023-09-22T21:32:11.971 DEBUG:boto:auth_path=/ffzdqu-56/h 2023-09-22T21:32:11.971 DEBUG:boto:path=/ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.971 DEBUG:boto:auth_path=/ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.971 DEBUG:boto:Method: GET 2023-09-22T21:32:11.971 DEBUG:boto:Path: /ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.971 DEBUG:boto:Data: 2023-09-22T21:32:11.971 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.971 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:11.971 DEBUG:boto:Port: 8000 2023-09-22T21:32:11.972 DEBUG:boto:Params: {} 2023-09-22T21:32:11.972 DEBUG:boto:Token: None 2023-09-22T21:32:11.972 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/h?versionId=null 2023-09-22T21:32:11.972 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0psRJGJU7G+/MBtJ43mPTQTann8= 2023-09-22T21:32:11.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0psRJGJU7G+/MBtJ43mPTQTann8=', 'Content-Length': '0'} 2023-09-22T21:32:11.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d455ee8f5a11736-00650e07db-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:11.975 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:11.975 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:11.976 DEBUG:boto:path=/ffzdqu-56/?prefix=h 2023-09-22T21:32:11.976 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=h 2023-09-22T21:32:11.976 DEBUG:boto:Method: GET 2023-09-22T21:32:11.976 DEBUG:boto:Path: /ffzdqu-56/?prefix=h 2023-09-22T21:32:11.976 DEBUG:boto:Data: 2023-09-22T21:32:11.976 DEBUG:boto:Headers: {} 2023-09-22T21:32:11.976 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:11.976 DEBUG:boto:Port: 8001 2023-09-22T21:32:11.976 DEBUG:boto:Params: {} 2023-09-22T21:32:11.976 DEBUG:boto:Token: None 2023-09-22T21:32:11.977 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:11 GMT /ffzdqu-56/ 2023-09-22T21:32:11.977 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4= 2023-09-22T21:32:11.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F4k3C/T/8QYt8ZmMyHSVNXy/au4=', 'Content-Length': '0'} 2023-09-22T21:32:11.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090c7ebb6a76ae64d-00650e07db-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:12.022 DEBUG:boto:b'ffzdqu-56h1000falseh2023-09-22T21:28:49.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:12.023 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:12.023 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:12.023 DEBUG:boto:path=/ffzdqu-56/?prefix=h 2023-09-22T21:32:12.023 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=h 2023-09-22T21:32:12.023 DEBUG:boto:Method: GET 2023-09-22T21:32:12.024 DEBUG:boto:Path: /ffzdqu-56/?prefix=h 2023-09-22T21:32:12.024 DEBUG:boto:Data: 2023-09-22T21:32:12.024 DEBUG:boto:Headers: {} 2023-09-22T21:32:12.024 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:12.024 DEBUG:boto:Port: 8000 2023-09-22T21:32:12.024 DEBUG:boto:Params: {} 2023-09-22T21:32:12.024 DEBUG:boto:Token: None 2023-09-22T21:32:12.024 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:12 GMT /ffzdqu-56/ 2023-09-22T21:32:12.024 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lkU0LlXLp4IP71InQ9CLyK64vH0= 2023-09-22T21:32:12.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lkU0LlXLp4IP71InQ9CLyK64vH0=', 'Content-Length': '0'} 2023-09-22T21:32:12.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000b1c1652958d88e-00650e07dc-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:12.028 DEBUG:boto:b'ffzdqu-56h1000falseh2023-09-22T21:28:49.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:12.028 DEBUG:boto:path=/ffzdqu-56/h 2023-09-22T21:32:12.028 DEBUG:boto:auth_path=/ffzdqu-56/h 2023-09-22T21:32:12.028 DEBUG:boto:Method: HEAD 2023-09-22T21:32:12.029 DEBUG:boto:Path: /ffzdqu-56/h 2023-09-22T21:32:12.029 DEBUG:boto:Data: 2023-09-22T21:32:12.029 DEBUG:boto:Headers: {} 2023-09-22T21:32:12.029 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:12.029 DEBUG:boto:Port: 8001 2023-09-22T21:32:12.029 DEBUG:boto:Params: {} 2023-09-22T21:32:12.029 DEBUG:boto:Token: None 2023-09-22T21:32:12.029 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:12 GMT /ffzdqu-56/h 2023-09-22T21:32:12.029 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6Gy/kDRhVZSwFdVrMRtdJn/eyvI= 2023-09-22T21:32:12.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6Gy/kDRhVZSwFdVrMRtdJn/eyvI=', 'Content-Length': '0'} 2023-09-22T21:32:12.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093f1c55f7be54f66-00650e07dc-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:12.032 DEBUG:boto:path=/ffzdqu-56/h 2023-09-22T21:32:12.032 DEBUG:boto:auth_path=/ffzdqu-56/h 2023-09-22T21:32:12.033 DEBUG:boto:Method: HEAD 2023-09-22T21:32:12.033 DEBUG:boto:Path: /ffzdqu-56/h 2023-09-22T21:32:12.033 DEBUG:boto:Data: 2023-09-22T21:32:12.033 DEBUG:boto:Headers: {} 2023-09-22T21:32:12.033 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:12.033 DEBUG:boto:Port: 8000 2023-09-22T21:32:12.033 DEBUG:boto:Params: {} 2023-09-22T21:32:12.033 DEBUG:boto:Token: None 2023-09-22T21:32:12.033 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:12 GMT /ffzdqu-56/h 2023-09-22T21:32:12.033 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6Gy/kDRhVZSwFdVrMRtdJn/eyvI= 2023-09-22T21:32:12.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6Gy/kDRhVZSwFdVrMRtdJn/eyvI=', 'Content-Length': '0'} 2023-09-22T21:32:12.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e096ef002814f05-00650e07dc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:12.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-22T21:32:12.036 DEBUG:boto:path=/ffzdqu-56/h 2023-09-22T21:32:12.036 DEBUG:boto:auth_path=/ffzdqu-56/h 2023-09-22T21:32:12.037 DEBUG:boto:Method: GET 2023-09-22T21:32:12.037 DEBUG:boto:Path: /ffzdqu-56/h 2023-09-22T21:32:12.037 DEBUG:boto:Data: 2023-09-22T21:32:12.037 DEBUG:boto:Headers: {} 2023-09-22T21:32:12.037 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:12.037 DEBUG:boto:Port: 8001 2023-09-22T21:32:12.037 DEBUG:boto:Params: {} 2023-09-22T21:32:12.037 DEBUG:boto:Token: None 2023-09-22T21:32:12.037 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:12 GMT /ffzdqu-56/h 2023-09-22T21:32:12.038 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XbKAPXcIgiQMXG/qa4t3RTkCp+Q= 2023-09-22T21:32:12.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XbKAPXcIgiQMXG/qa4t3RTkCp+Q=', 'Content-Length': '0'} 2023-09-22T21:32:12.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-56:f132bdb9-20a7-4073-a932-8f99da20691e.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006105736755a2bc88-00650e07dc-165b-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:12.082 DEBUG:boto:path=/ffzdqu-56/h 2023-09-22T21:32:12.082 DEBUG:boto:auth_path=/ffzdqu-56/h 2023-09-22T21:32:12.082 DEBUG:boto:Method: GET 2023-09-22T21:32:12.083 DEBUG:boto:Path: /ffzdqu-56/h 2023-09-22T21:32:12.083 DEBUG:boto:Data: 2023-09-22T21:32:12.083 DEBUG:boto:Headers: {} 2023-09-22T21:32:12.083 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:12.083 DEBUG:boto:Port: 8000 2023-09-22T21:32:12.083 DEBUG:boto:Params: {} 2023-09-22T21:32:12.083 DEBUG:boto:Token: None 2023-09-22T21:32:12.083 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:12 GMT /ffzdqu-56/h 2023-09-22T21:32:12.083 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XbKAPXcIgiQMXG/qa4t3RTkCp+Q= 2023-09-22T21:32:12.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XbKAPXcIgiQMXG/qa4t3RTkCp+Q=', 'Content-Length': '0'} 2023-09-22T21:32:12.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:28:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adcd27e038b7e636-00650e07dc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:12.087 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:12.087 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:12.087 DEBUG:boto:path=/ffzdqu-56/?prefix=h 2023-09-22T21:32:12.087 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=h 2023-09-22T21:32:12.087 DEBUG:boto:Method: GET 2023-09-22T21:32:12.087 DEBUG:boto:Path: /ffzdqu-56/?prefix=h 2023-09-22T21:32:12.087 DEBUG:boto:Data: 2023-09-22T21:32:12.087 DEBUG:boto:Headers: {} 2023-09-22T21:32:12.087 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:12.088 DEBUG:boto:Port: 8001 2023-09-22T21:32:12.088 DEBUG:boto:Params: {} 2023-09-22T21:32:12.088 DEBUG:boto:Token: None 2023-09-22T21:32:12.088 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:12 GMT /ffzdqu-56/ 2023-09-22T21:32:12.088 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lkU0LlXLp4IP71InQ9CLyK64vH0= 2023-09-22T21:32:12.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lkU0LlXLp4IP71InQ9CLyK64vH0=', 'Content-Length': '0'} 2023-09-22T21:32:12.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2259d341e54c722-00650e07dc-165b-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:12.135 DEBUG:boto:b'ffzdqu-56h1000falseh2023-09-22T21:28:49.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:12.135 DEBUG:boto:path=/ffzdqu-56/ 2023-09-22T21:32:12.135 DEBUG:boto:auth_path=/ffzdqu-56/ 2023-09-22T21:32:12.135 DEBUG:boto:path=/ffzdqu-56/?prefix=h 2023-09-22T21:32:12.136 DEBUG:boto:auth_path=/ffzdqu-56/?prefix=h 2023-09-22T21:32:12.136 DEBUG:boto:Method: GET 2023-09-22T21:32:12.136 DEBUG:boto:Path: /ffzdqu-56/?prefix=h 2023-09-22T21:32:12.136 DEBUG:boto:Data: 2023-09-22T21:32:12.136 DEBUG:boto:Headers: {} 2023-09-22T21:32:12.136 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:12.136 DEBUG:boto:Port: 8000 2023-09-22T21:32:12.136 DEBUG:boto:Params: {} 2023-09-22T21:32:12.136 DEBUG:boto:Token: None 2023-09-22T21:32:12.137 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:32:12 GMT /ffzdqu-56/ 2023-09-22T21:32:12.137 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lkU0LlXLp4IP71InQ9CLyK64vH0= 2023-09-22T21:32:12.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lkU0LlXLp4IP71InQ9CLyK64vH0=', 'Content-Length': '0'} 2023-09-22T21:32:12.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e51140c80b2f454-00650e07dc-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:12.140 DEBUG:boto:b'ffzdqu-56h1000falseh2023-09-22T21:28:49.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:32:12.141 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-56 zones={a2, a1} 2023-09-22T21:32:12.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... ok 2023-09-22T21:32:12.142 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:32:12.142 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:32:12.142 DEBUG:boto:Method: PUT 2023-09-22T21:32:12.142 DEBUG:boto:Path: /ffzdqu-57/ 2023-09-22T21:32:12.142 DEBUG:boto:Data: 2023-09-22T21:32:12.142 DEBUG:boto:Headers: {} 2023-09-22T21:32:12.143 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:12.143 DEBUG:boto:Port: 8000 2023-09-22T21:32:12.143 DEBUG:boto:Params: {} 2023-09-22T21:32:12.143 DEBUG:boto:Token: None 2023-09-22T21:32:12.143 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:32:12 GMT /ffzdqu-57/ 2023-09-22T21:32:12.143 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xH+U9fwA4j9MVFJ4ASST1nYWB8Q= 2023-09-22T21:32:12.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xH+U9fwA4j9MVFJ4ASST1nYWB8Q=', 'Content-Length': '0'} 2023-09-22T21:32:12.186 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c12817ac59b73255-00650e07dc-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:32:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:12.186 DEBUG:rgw_multi.tests:created bucket=ffzdqu-57 2023-09-22T21:32:12.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:32:12.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:32:12.187 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:32:12.226 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:32:12.226 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:32:12.266 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:32:12.284 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695418332.182904_453.1","last_update":"2023-09-22T21:32:12.182904Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:32:12.285 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695418332.182904_453.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:32:12.285 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:32:12.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:32:12.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:32:12.285 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:32:12.363 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:32:12.363 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:32:12.400 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:32:12.399+0000 7f0680670a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:32:12.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:32:12.399+0000 7f0680670a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:32:12.422 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695418332.166738_449.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:32:12.166738Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:32:12Z"} 2023-09-22T21:32:12.422 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695418332.166738_449.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:32:12.166738Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:32:12Z"} 2023-09-22T21:32:12.422 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695418332.166738_449.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:32:12.166738Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:32:12.422 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695418332.182904_453.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:32:12.423 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695418332.166738_449.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:32:12.423 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695418332.182904_453.1 target=1_1695418332.166738_449.1 2023-09-22T21:32:17.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:32:17.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:32:17.424 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:32:17.467 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:32:17.468 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:32:17.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:32:17.503+0000 7f8e54031a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:32:17.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:32:17.503+0000 7f8e54031a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:32:17.526 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695418332.182904_453.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:32:12.182904Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:32:17Z"} 2023-09-22T21:32:17.526 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695418332.182904_453.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:32:12.182904Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:32:17Z"} 2023-09-22T21:32:17.526 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695418332.182904_453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:32:12.182904Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:32:17.526 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695418332.182904_453.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:32:17.527 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695418332.182904_453.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:32:17.527 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:32:17.527 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-09-22T21:32:17.527 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T21:32:17.568 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:32:17.567+0000 7ff4a260d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 40554) UID: 0 2023-09-22T21:32:17.568 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:32:17.567+0000 7ff4a5e54a40 -1 shutting down 2023-09-22T21:32:22.965 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-09-22T21:32:23.629 INFO:tasks.rgw.c1.client.1:Stopped 2023-09-22T21:32:23.629 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:32:23.629 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:32:23.630 DEBUG:boto:Method: HEAD 2023-09-22T21:32:23.630 DEBUG:boto:Path: /ffzdqu-57/ 2023-09-22T21:32:23.630 DEBUG:boto:Data: 2023-09-22T21:32:23.630 DEBUG:boto:Headers: {} 2023-09-22T21:32:23.630 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:23.630 DEBUG:boto:Port: 8000 2023-09-22T21:32:23.630 DEBUG:boto:Params: {} 2023-09-22T21:32:23.631 DEBUG:boto:Token: None 2023-09-22T21:32:23.631 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:23 GMT /ffzdqu-57/ 2023-09-22T21:32:23.631 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1eowoTmeU6crWzWggvZC+Hnvq0= 2023-09-22T21:32:23.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1eowoTmeU6crWzWggvZC+Hnvq0=', 'Content-Length': '0'} 2023-09-22T21:32:23.635 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8ed76e0b87cef0a-00650e07e7-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:23.636 DEBUG:boto:path=/ffzdqu-57/a 2023-09-22T21:32:23.636 DEBUG:boto:auth_path=/ffzdqu-57/a 2023-09-22T21:32:23.636 DEBUG:boto:Method: PUT 2023-09-22T21:32:23.636 DEBUG:boto:Path: /ffzdqu-57/a 2023-09-22T21:32:23.636 DEBUG:boto:Data: 2023-09-22T21:32:23.636 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:32:23.636 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:23.637 DEBUG:boto:Port: 8000 2023-09-22T21:32:23.637 DEBUG:boto:Params: {} 2023-09-22T21:32:23.637 DEBUG:boto:Token: None 2023-09-22T21:32:23.637 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:32:23 GMT /ffzdqu-57/a 2023-09-22T21:32:23.637 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3rIwa44VqMI9IsRBtrs3xzUpTlo= 2023-09-22T21:32:23.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:32:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3rIwa44VqMI9IsRBtrs3xzUpTlo='} 2023-09-22T21:32:23.652 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e382f944496d9528-00650e07e7-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:23.653 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:32:23.653 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:32:23.653 DEBUG:boto:Method: HEAD 2023-09-22T21:32:23.653 DEBUG:boto:Path: /ffzdqu-57/ 2023-09-22T21:32:23.653 DEBUG:boto:Data: 2023-09-22T21:32:23.653 DEBUG:boto:Headers: {} 2023-09-22T21:32:23.654 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:23.654 DEBUG:boto:Port: 8000 2023-09-22T21:32:23.654 DEBUG:boto:Params: {} 2023-09-22T21:32:23.654 DEBUG:boto:Token: None 2023-09-22T21:32:23.654 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:23 GMT /ffzdqu-57/ 2023-09-22T21:32:23.654 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1eowoTmeU6crWzWggvZC+Hnvq0= 2023-09-22T21:32:23.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1eowoTmeU6crWzWggvZC+Hnvq0=', 'Content-Length': '0'} 2023-09-22T21:32:23.658 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6cc10dbc3a03c49-00650e07e7-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:23.659 DEBUG:boto:path=/ffzdqu-57/b 2023-09-22T21:32:23.659 DEBUG:boto:auth_path=/ffzdqu-57/b 2023-09-22T21:32:23.659 DEBUG:boto:Method: PUT 2023-09-22T21:32:23.659 DEBUG:boto:Path: /ffzdqu-57/b 2023-09-22T21:32:23.659 DEBUG:boto:Data: 2023-09-22T21:32:23.659 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:32:23.659 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:23.659 DEBUG:boto:Port: 8000 2023-09-22T21:32:23.660 DEBUG:boto:Params: {} 2023-09-22T21:32:23.660 DEBUG:boto:Token: None 2023-09-22T21:32:23.660 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:32:23 GMT /ffzdqu-57/b 2023-09-22T21:32:23.660 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GSfhqW2ipgAs023Hf4vVLeO80Ok= 2023-09-22T21:32:23.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:32:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GSfhqW2ipgAs023Hf4vVLeO80Ok='} 2023-09-22T21:32:23.702 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003873600edb822e89-00650e07e7-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:23.703 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:32:23.703 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:32:23.703 DEBUG:boto:Method: HEAD 2023-09-22T21:32:23.704 DEBUG:boto:Path: /ffzdqu-57/ 2023-09-22T21:32:23.704 DEBUG:boto:Data: 2023-09-22T21:32:23.704 DEBUG:boto:Headers: {} 2023-09-22T21:32:23.704 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:23.704 DEBUG:boto:Port: 8000 2023-09-22T21:32:23.705 DEBUG:boto:Params: {} 2023-09-22T21:32:23.705 DEBUG:boto:Token: None 2023-09-22T21:32:23.705 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:23 GMT /ffzdqu-57/ 2023-09-22T21:32:23.705 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1eowoTmeU6crWzWggvZC+Hnvq0= 2023-09-22T21:32:23.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1eowoTmeU6crWzWggvZC+Hnvq0=', 'Content-Length': '0'} 2023-09-22T21:32:23.710 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a1674d3e84793df-00650e07e7-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:23.710 DEBUG:boto:path=/ffzdqu-57/c 2023-09-22T21:32:23.710 DEBUG:boto:auth_path=/ffzdqu-57/c 2023-09-22T21:32:23.711 DEBUG:boto:Method: PUT 2023-09-22T21:32:23.711 DEBUG:boto:Path: /ffzdqu-57/c 2023-09-22T21:32:23.711 DEBUG:boto:Data: 2023-09-22T21:32:23.711 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:32:23.711 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:23.712 DEBUG:boto:Port: 8000 2023-09-22T21:32:23.712 DEBUG:boto:Params: {} 2023-09-22T21:32:23.712 DEBUG:boto:Token: None 2023-09-22T21:32:23.712 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:32:23 GMT /ffzdqu-57/c 2023-09-22T21:32:23.712 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HcVcXGcwzw3uq+fSMDhFHmVfxKk= 2023-09-22T21:32:23.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:32:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HcVcXGcwzw3uq+fSMDhFHmVfxKk='} 2023-09-22T21:32:23.726 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f97ce2ae061c6257-00650e07e7-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:23.726 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:32:23.726 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:32:23.726 DEBUG:boto:Method: HEAD 2023-09-22T21:32:23.727 DEBUG:boto:Path: /ffzdqu-57/ 2023-09-22T21:32:23.727 DEBUG:boto:Data: 2023-09-22T21:32:23.727 DEBUG:boto:Headers: {} 2023-09-22T21:32:23.727 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:23.727 DEBUG:boto:Port: 8000 2023-09-22T21:32:23.728 DEBUG:boto:Params: {} 2023-09-22T21:32:23.728 DEBUG:boto:Token: None 2023-09-22T21:32:23.728 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:23 GMT /ffzdqu-57/ 2023-09-22T21:32:23.728 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1eowoTmeU6crWzWggvZC+Hnvq0= 2023-09-22T21:32:23.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1eowoTmeU6crWzWggvZC+Hnvq0=', 'Content-Length': '0'} 2023-09-22T21:32:23.732 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fac9a4972d86edac-00650e07e7-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:23.733 DEBUG:boto:path=/ffzdqu-57/d 2023-09-22T21:32:23.733 DEBUG:boto:auth_path=/ffzdqu-57/d 2023-09-22T21:32:23.733 DEBUG:boto:Method: PUT 2023-09-22T21:32:23.734 DEBUG:boto:Path: /ffzdqu-57/d 2023-09-22T21:32:23.734 DEBUG:boto:Data: 2023-09-22T21:32:23.734 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:32:23.734 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:23.734 DEBUG:boto:Port: 8000 2023-09-22T21:32:23.735 DEBUG:boto:Params: {} 2023-09-22T21:32:23.735 DEBUG:boto:Token: None 2023-09-22T21:32:23.735 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:32:23 GMT /ffzdqu-57/d 2023-09-22T21:32:23.735 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aHde7QDKd8qa3u9VTS9uN5s8BmU= 2023-09-22T21:32:23.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:32:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aHde7QDKd8qa3u9VTS9uN5s8BmU='} 2023-09-22T21:32:23.778 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000035f1531895484935-00650e07e7-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:23.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'ffzdqu-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:32:23.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'ffzdqu-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:32:23.779 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket ffzdqu-57 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:32:23.824 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:32:23.825 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:32:23.874 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:32:23.874 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-57 2023-09-22T21:32:23.880 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 4 2023-09-22T21:32:23.926 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:32:23.927 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:32:23.927 DEBUG:boto:Method: HEAD 2023-09-22T21:32:23.927 DEBUG:boto:Path: /ffzdqu-57/ 2023-09-22T21:32:23.927 DEBUG:boto:Data: 2023-09-22T21:32:23.927 DEBUG:boto:Headers: {} 2023-09-22T21:32:23.928 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:23.928 DEBUG:boto:Port: 8000 2023-09-22T21:32:23.928 DEBUG:boto:Params: {} 2023-09-22T21:32:23.928 DEBUG:boto:Token: None 2023-09-22T21:32:23.928 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:23 GMT /ffzdqu-57/ 2023-09-22T21:32:23.929 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1eowoTmeU6crWzWggvZC+Hnvq0= 2023-09-22T21:32:23.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1eowoTmeU6crWzWggvZC+Hnvq0=', 'Content-Length': '0'} 2023-09-22T21:32:23.931 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b4d34efc6692f90-00650e07e7-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:23.931 DEBUG:boto:path=/ffzdqu-57/e 2023-09-22T21:32:23.931 DEBUG:boto:auth_path=/ffzdqu-57/e 2023-09-22T21:32:23.932 DEBUG:boto:Method: PUT 2023-09-22T21:32:23.932 DEBUG:boto:Path: /ffzdqu-57/e 2023-09-22T21:32:23.932 DEBUG:boto:Data: 2023-09-22T21:32:23.932 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:32:23.932 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:23.933 DEBUG:boto:Port: 8000 2023-09-22T21:32:23.933 DEBUG:boto:Params: {} 2023-09-22T21:32:23.933 DEBUG:boto:Token: None 2023-09-22T21:32:23.933 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:32:23 GMT /ffzdqu-57/e 2023-09-22T21:32:23.933 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WsXgBZ0AQVP0BEzInt33SH1jhRQ= 2023-09-22T21:32:23.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:32:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WsXgBZ0AQVP0BEzInt33SH1jhRQ='} 2023-09-22T21:32:23.940 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a4df18be52291739-00650e07e7-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:23.940 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:32:23.940 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:32:23.941 DEBUG:boto:Method: HEAD 2023-09-22T21:32:23.941 DEBUG:boto:Path: /ffzdqu-57/ 2023-09-22T21:32:23.941 DEBUG:boto:Data: 2023-09-22T21:32:23.941 DEBUG:boto:Headers: {} 2023-09-22T21:32:23.941 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:23.941 DEBUG:boto:Port: 8000 2023-09-22T21:32:23.942 DEBUG:boto:Params: {} 2023-09-22T21:32:23.942 DEBUG:boto:Token: None 2023-09-22T21:32:23.942 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:23 GMT /ffzdqu-57/ 2023-09-22T21:32:23.942 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1eowoTmeU6crWzWggvZC+Hnvq0= 2023-09-22T21:32:23.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1eowoTmeU6crWzWggvZC+Hnvq0=', 'Content-Length': '0'} 2023-09-22T21:32:23.944 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b17802f9b645dc9-00650e07e7-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:23.944 DEBUG:boto:path=/ffzdqu-57/f 2023-09-22T21:32:23.945 DEBUG:boto:auth_path=/ffzdqu-57/f 2023-09-22T21:32:23.945 DEBUG:boto:Method: PUT 2023-09-22T21:32:23.945 DEBUG:boto:Path: /ffzdqu-57/f 2023-09-22T21:32:23.945 DEBUG:boto:Data: 2023-09-22T21:32:23.945 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:32:23.946 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:23.946 DEBUG:boto:Port: 8000 2023-09-22T21:32:23.946 DEBUG:boto:Params: {} 2023-09-22T21:32:23.946 DEBUG:boto:Token: None 2023-09-22T21:32:23.946 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:32:23 GMT /ffzdqu-57/f 2023-09-22T21:32:23.947 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UDguU2LJqyPO6fM3DUmkcFPzWDU= 2023-09-22T21:32:23.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:32:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UDguU2LJqyPO6fM3DUmkcFPzWDU='} 2023-09-22T21:32:23.952 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eb804854b81be99f-00650e07e7-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:23.953 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:32:23.953 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:32:23.953 DEBUG:boto:Method: HEAD 2023-09-22T21:32:23.953 DEBUG:boto:Path: /ffzdqu-57/ 2023-09-22T21:32:23.953 DEBUG:boto:Data: 2023-09-22T21:32:23.953 DEBUG:boto:Headers: {} 2023-09-22T21:32:23.953 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:23.953 DEBUG:boto:Port: 8000 2023-09-22T21:32:23.953 DEBUG:boto:Params: {} 2023-09-22T21:32:23.954 DEBUG:boto:Token: None 2023-09-22T21:32:23.954 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:23 GMT /ffzdqu-57/ 2023-09-22T21:32:23.954 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1eowoTmeU6crWzWggvZC+Hnvq0= 2023-09-22T21:32:23.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1eowoTmeU6crWzWggvZC+Hnvq0=', 'Content-Length': '0'} 2023-09-22T21:32:23.955 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b44b57112b57a409-00650e07e7-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:23.956 DEBUG:boto:path=/ffzdqu-57/g 2023-09-22T21:32:23.956 DEBUG:boto:auth_path=/ffzdqu-57/g 2023-09-22T21:32:23.956 DEBUG:boto:Method: PUT 2023-09-22T21:32:23.956 DEBUG:boto:Path: /ffzdqu-57/g 2023-09-22T21:32:23.956 DEBUG:boto:Data: 2023-09-22T21:32:23.956 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:32:23.956 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:23.956 DEBUG:boto:Port: 8000 2023-09-22T21:32:23.957 DEBUG:boto:Params: {} 2023-09-22T21:32:23.957 DEBUG:boto:Token: None 2023-09-22T21:32:23.957 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:32:23 GMT /ffzdqu-57/g 2023-09-22T21:32:23.957 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wBbptqwokGz1vqvaPuxNN6yam+M= 2023-09-22T21:32:23.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:32:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wBbptqwokGz1vqvaPuxNN6yam+M='} 2023-09-22T21:32:23.999 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000045d8d377a4e49548-00650e07e7-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:23.999 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:32:23.999 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:32:23.999 DEBUG:boto:Method: HEAD 2023-09-22T21:32:24.000 DEBUG:boto:Path: /ffzdqu-57/ 2023-09-22T21:32:24.000 DEBUG:boto:Data: 2023-09-22T21:32:24.000 DEBUG:boto:Headers: {} 2023-09-22T21:32:24.000 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:24.000 DEBUG:boto:Port: 8000 2023-09-22T21:32:24.000 DEBUG:boto:Params: {} 2023-09-22T21:32:24.001 DEBUG:boto:Token: None 2023-09-22T21:32:24.001 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:32:24 GMT /ffzdqu-57/ 2023-09-22T21:32:24.001 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1gy1pem3Swj2HY9ighhFsiPjX1c= 2023-09-22T21:32:24.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:32:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1gy1pem3Swj2HY9ighhFsiPjX1c=', 'Content-Length': '0'} 2023-09-22T21:32:24.004 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006aa0a42d0674319f-00650e07e7-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:32:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:24.005 DEBUG:boto:path=/ffzdqu-57/h 2023-09-22T21:32:24.005 DEBUG:boto:auth_path=/ffzdqu-57/h 2023-09-22T21:32:24.005 DEBUG:boto:Method: PUT 2023-09-22T21:32:24.005 DEBUG:boto:Path: /ffzdqu-57/h 2023-09-22T21:32:24.006 DEBUG:boto:Data: 2023-09-22T21:32:24.006 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:32:24.006 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:32:24.006 DEBUG:boto:Port: 8000 2023-09-22T21:32:24.006 DEBUG:boto:Params: {} 2023-09-22T21:32:24.007 DEBUG:boto:Token: None 2023-09-22T21:32:24.007 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:32:24 GMT /ffzdqu-57/h 2023-09-22T21:32:24.007 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XhSD2wP2vRq7dBnwLEDZyrysn8o= 2023-09-22T21:32:24.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:32:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XhSD2wP2vRq7dBnwLEDZyrysn8o='} 2023-09-22T21:32:24.015 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000701b841dfc9145e8-00650e07e8-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:32:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:32:24.015 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-22T21:32:24.015 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-22T21:32:24.018 INFO:tasks.rgw.c1.client.1:Started 2023-09-22T21:32:24.018 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:24.027 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:32:24.027 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:32:24.028 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T21:32:24.028 INFO:teuthology.orchestra.run.smithi160.stderr:curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T21:32:24.028 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T21:32:25.029 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T21:32:25.041 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:32:25.041 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:32:25.042 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 157k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-22T21:32:25.043 INFO:teuthology.orchestra.run.smithi160.stdout:anonymous 2023-09-22T21:32:25.043 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-57 2023-09-22T21:32:25.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:32:25.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:32:25.044 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:32:25.133 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:32:25.133 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:32:25.174 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:32:25.170+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:32:28.672 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-09-22T21:32:30.175 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:32:30.170+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:32:35.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:32:35.174+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:32:40.178 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:32:40.174+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:32:45.179 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:32:45.174+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:32:50.179 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:32:50.174+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:32:55.180 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:32:55.178+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:33:00.181 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:33:00.178+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:33:05.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:33:05.178+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:33:10.183 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:33:10.178+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:33:15.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:33:15.182+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:33:20.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:33:20.182+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:33:25.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:33:25.182+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:33:30.188 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:33:30.181+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:33:35.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:33:35.185+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:33:40.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:33:40.189+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:33:45.191 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:33:45.189+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:33:50.192 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:33:50.189+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:33:55.193 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:33:55.193+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:33:55.263 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:33:55.261+0000 7f415c9d9640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:34:00.194 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:34:00.193+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:34:05.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:34:05.193+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:34:10.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:34:10.197+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:34:15.198 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:34:15.197+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:34:20.200 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:34:20.197+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:34:25.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:34:25.201+0000 7f062a27fa40 1 waiting to reach incremental sync.. 2023-09-22T21:34:30.203 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:34:30.200+0000 7f062a27fa40 1 bucket sync caught up with source: 2023-09-22T21:34:30.203 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000003.43.6, 00000000002.48.6, 00000000002.39.6] 2023-09-22T21:34:30.204 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.43.6, 00000000002.48.6, 00000000002.39.6] 2023-09-22T21:34:30.204 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:34:30.200+0000 7f062a27fa40 0 bucket checkpoint complete 2023-09-22T21:34:30.238 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-57 2023-09-22T21:34:30.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:34:30.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:34:30.238 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:34:30.288 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:34:30.288 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:34:30.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:34:30.328+0000 7fc0d5979a40 1 bucket sync caught up with source: 2023-09-22T21:34:30.331 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.222.6, , , 00000000001.202.6, 00000000001.215.6, , , 00000000002.265.6, , 00000000002.288.6] 2023-09-22T21:34:30.332 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.222.6, , , 00000000001.202.6, 00000000001.215.6, , , 00000000002.265.6, , 00000000002.288.6] 2023-09-22T21:34:30.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:34:30.328+0000 7fc0d5979a40 0 bucket checkpoint complete 2023-09-22T21:34:30.348 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-57 zones={a2, a1} 2023-09-22T21:34:30.348 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:30.349 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:30.349 DEBUG:boto:Method: HEAD 2023-09-22T21:34:30.349 DEBUG:boto:Path: /ffzdqu-57/ 2023-09-22T21:34:30.349 DEBUG:boto:Data: 2023-09-22T21:34:30.349 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.349 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.349 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.349 DEBUG:boto:Params: {} 2023-09-22T21:34:30.350 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:34:30.350 DEBUG:boto:Token: None 2023-09-22T21:34:30.350 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/ 2023-09-22T21:34:30.350 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:svdsjtiWBe9HF0yTAY+HWI+Wb4c= 2023-09-22T21:34:30.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:svdsjtiWBe9HF0yTAY+HWI+Wb4c=', 'Content-Length': '0'} 2023-09-22T21:34:30.356 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb2ae7068ea1e88d-00650e0866-1ada-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.356 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:30.357 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:30.357 DEBUG:boto:Method: HEAD 2023-09-22T21:34:30.357 DEBUG:boto:Path: /ffzdqu-57/ 2023-09-22T21:34:30.357 DEBUG:boto:Data: 2023-09-22T21:34:30.357 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.357 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.357 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.357 DEBUG:boto:Params: {} 2023-09-22T21:34:30.358 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:34:30.358 DEBUG:boto:Token: None 2023-09-22T21:34:30.358 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/ 2023-09-22T21:34:30.358 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:svdsjtiWBe9HF0yTAY+HWI+Wb4c= 2023-09-22T21:34:30.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:svdsjtiWBe9HF0yTAY+HWI+Wb4c=', 'Content-Length': '0'} 2023-09-22T21:34:30.362 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003cb391fe6f9e30a0-00650e0866-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.362 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:34:30.362 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:30.363 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:30.363 DEBUG:boto:path=/ffzdqu-57/?versions&max-keys=999 2023-09-22T21:34:30.363 DEBUG:boto:auth_path=/ffzdqu-57/?versions&max-keys=999 2023-09-22T21:34:30.363 DEBUG:boto:Method: GET 2023-09-22T21:34:30.363 DEBUG:boto:Path: /ffzdqu-57/?versions&max-keys=999 2023-09-22T21:34:30.363 DEBUG:boto:Data: 2023-09-22T21:34:30.363 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.363 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.363 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.363 DEBUG:boto:Params: {} 2023-09-22T21:34:30.364 DEBUG:boto:Token: None 2023-09-22T21:34:30.364 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/?versions 2023-09-22T21:34:30.364 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zfrk+nU5rBKbgFiaoO9moLRGsnE= 2023-09-22T21:34:30.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zfrk+nU5rBKbgFiaoO9moLRGsnE=', 'Content-Length': '0'} 2023-09-22T21:34:30.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dfb18cec848d1e4-00650e0866-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.366 DEBUG:boto:b'ffzdqu-57999falseanulltrue2023-09-22T21:32:23.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:32:23.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:32:23.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:32:23.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-22T21:32:23.937Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-22T21:32:23.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-22T21:32:23.960Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-22T21:32:24.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:30.367 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-22T21:34:30.367 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-22T21:34:30.368 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-22T21:34:30.368 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-22T21:34:30.368 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-09-22T21:34:30.368 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-09-22T21:34:30.368 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-09-22T21:34:30.368 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-09-22T21:34:30.368 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:34:30.368 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:30.368 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:30.368 DEBUG:boto:path=/ffzdqu-57/?versions&max-keys=999 2023-09-22T21:34:30.369 DEBUG:boto:auth_path=/ffzdqu-57/?versions&max-keys=999 2023-09-22T21:34:30.369 DEBUG:boto:Method: GET 2023-09-22T21:34:30.369 DEBUG:boto:Path: /ffzdqu-57/?versions&max-keys=999 2023-09-22T21:34:30.369 DEBUG:boto:Data: 2023-09-22T21:34:30.369 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.369 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.369 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.369 DEBUG:boto:Params: {} 2023-09-22T21:34:30.369 DEBUG:boto:Token: None 2023-09-22T21:34:30.369 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/?versions 2023-09-22T21:34:30.370 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zfrk+nU5rBKbgFiaoO9moLRGsnE= 2023-09-22T21:34:30.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zfrk+nU5rBKbgFiaoO9moLRGsnE=', 'Content-Length': '0'} 2023-09-22T21:34:30.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d07ebfc920e2491-00650e0866-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.372 DEBUG:boto:b'ffzdqu-57999falseanulltrue2023-09-22T21:32:23.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:32:23.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:32:23.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:32:23.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-22T21:32:23.937Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-22T21:32:23.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-22T21:32:23.960Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-22T21:32:24.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:30.372 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-22T21:34:30.372 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-22T21:34:30.372 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-22T21:34:30.372 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-22T21:34:30.373 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-09-22T21:34:30.373 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-09-22T21:34:30.373 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-09-22T21:34:30.373 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-09-22T21:34:30.373 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:30.373 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:30.373 DEBUG:boto:path=/ffzdqu-57/?versions&max-keys=999 2023-09-22T21:34:30.373 DEBUG:boto:auth_path=/ffzdqu-57/?versions&max-keys=999 2023-09-22T21:34:30.373 DEBUG:boto:Method: GET 2023-09-22T21:34:30.374 DEBUG:boto:Path: /ffzdqu-57/?versions&max-keys=999 2023-09-22T21:34:30.374 DEBUG:boto:Data: 2023-09-22T21:34:30.374 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.374 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.374 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.374 DEBUG:boto:Params: {} 2023-09-22T21:34:30.374 DEBUG:boto:Token: None 2023-09-22T21:34:30.374 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/?versions 2023-09-22T21:34:30.374 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zfrk+nU5rBKbgFiaoO9moLRGsnE= 2023-09-22T21:34:30.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zfrk+nU5rBKbgFiaoO9moLRGsnE=', 'Content-Length': '0'} 2023-09-22T21:34:30.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdd7df2cf79deda3-00650e0866-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.377 DEBUG:boto:b'ffzdqu-57999falseanulltrue2023-09-22T21:32:23.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:32:23.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:32:23.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:32:23.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-22T21:32:23.937Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-22T21:32:23.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-22T21:32:23.960Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-22T21:32:24.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:30.378 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:30.378 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:30.378 DEBUG:boto:path=/ffzdqu-57/?versions&max-keys=999 2023-09-22T21:34:30.378 DEBUG:boto:auth_path=/ffzdqu-57/?versions&max-keys=999 2023-09-22T21:34:30.378 DEBUG:boto:Method: GET 2023-09-22T21:34:30.378 DEBUG:boto:Path: /ffzdqu-57/?versions&max-keys=999 2023-09-22T21:34:30.378 DEBUG:boto:Data: 2023-09-22T21:34:30.378 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.378 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.379 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.379 DEBUG:boto:Params: {} 2023-09-22T21:34:30.379 DEBUG:boto:Token: None 2023-09-22T21:34:30.379 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/?versions 2023-09-22T21:34:30.379 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zfrk+nU5rBKbgFiaoO9moLRGsnE= 2023-09-22T21:34:30.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zfrk+nU5rBKbgFiaoO9moLRGsnE=', 'Content-Length': '0'} 2023-09-22T21:34:30.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005976698c743eca79-00650e0866-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.381 DEBUG:boto:b'ffzdqu-57999falseanulltrue2023-09-22T21:32:23.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-22T21:32:23.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-22T21:32:23.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-22T21:32:23.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-22T21:32:23.937Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-22T21:32:23.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-22T21:32:23.960Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-22T21:32:24.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:30.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:34:30.382 DEBUG:boto:path=/ffzdqu-57/a 2023-09-22T21:34:30.382 DEBUG:boto:auth_path=/ffzdqu-57/a 2023-09-22T21:34:30.382 DEBUG:boto:path=/ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.382 DEBUG:boto:auth_path=/ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.382 DEBUG:boto:Method: GET 2023-09-22T21:34:30.382 DEBUG:boto:Path: /ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.382 DEBUG:boto:Data: 2023-09-22T21:34:30.383 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.383 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.383 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.383 DEBUG:boto:Params: {} 2023-09-22T21:34:30.383 DEBUG:boto:Token: None 2023-09-22T21:34:30.383 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.383 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4yWsmb7ijy/ajGQJ+m0c9HZn6tk= 2023-09-22T21:34:30.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4yWsmb7ijy/ajGQJ+m0c9HZn6tk=', 'Content-Length': '0'} 2023-09-22T21:34:30.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c78d86219a8f475c-00650e0866-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.386 DEBUG:boto:path=/ffzdqu-57/a 2023-09-22T21:34:30.386 DEBUG:boto:auth_path=/ffzdqu-57/a 2023-09-22T21:34:30.386 DEBUG:boto:path=/ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.386 DEBUG:boto:auth_path=/ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.386 DEBUG:boto:Method: GET 2023-09-22T21:34:30.386 DEBUG:boto:Path: /ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.386 DEBUG:boto:Data: 2023-09-22T21:34:30.386 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.387 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.387 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.387 DEBUG:boto:Params: {} 2023-09-22T21:34:30.387 DEBUG:boto:Token: None 2023-09-22T21:34:30.387 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.387 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4yWsmb7ijy/ajGQJ+m0c9HZn6tk= 2023-09-22T21:34:30.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4yWsmb7ijy/ajGQJ+m0c9HZn6tk=', 'Content-Length': '0'} 2023-09-22T21:34:30.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000268f24afe5271413-00650e0866-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.389 DEBUG:boto:path=/ffzdqu-57/a 2023-09-22T21:34:30.389 DEBUG:boto:auth_path=/ffzdqu-57/a 2023-09-22T21:34:30.389 DEBUG:boto:path=/ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.389 DEBUG:boto:auth_path=/ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.389 DEBUG:boto:Method: HEAD 2023-09-22T21:34:30.390 DEBUG:boto:Path: /ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.390 DEBUG:boto:Data: 2023-09-22T21:34:30.390 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.390 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.390 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.390 DEBUG:boto:Params: {} 2023-09-22T21:34:30.390 DEBUG:boto:Token: None 2023-09-22T21:34:30.390 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.390 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dud9yllAG/Z5mFEFDoU45nvCDa0= 2023-09-22T21:34:30.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dud9yllAG/Z5mFEFDoU45nvCDa0=', 'Content-Length': '0'} 2023-09-22T21:34:30.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021110d6a48cbc3da-00650e0866-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.392 DEBUG:boto:path=/ffzdqu-57/a 2023-09-22T21:34:30.393 DEBUG:boto:auth_path=/ffzdqu-57/a 2023-09-22T21:34:30.393 DEBUG:boto:path=/ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.393 DEBUG:boto:auth_path=/ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.393 DEBUG:boto:Method: HEAD 2023-09-22T21:34:30.393 DEBUG:boto:Path: /ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.393 DEBUG:boto:Data: 2023-09-22T21:34:30.393 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.393 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.393 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.394 DEBUG:boto:Params: {} 2023-09-22T21:34:30.394 DEBUG:boto:Token: None 2023-09-22T21:34:30.394 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.394 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dud9yllAG/Z5mFEFDoU45nvCDa0= 2023-09-22T21:34:30.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dud9yllAG/Z5mFEFDoU45nvCDa0=', 'Content-Length': '0'} 2023-09-22T21:34:30.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000770a7b0f46024354-00650e0866-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:34:30.396 DEBUG:boto:path=/ffzdqu-57/a 2023-09-22T21:34:30.396 DEBUG:boto:auth_path=/ffzdqu-57/a 2023-09-22T21:34:30.396 DEBUG:boto:path=/ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.396 DEBUG:boto:auth_path=/ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.396 DEBUG:boto:Method: GET 2023-09-22T21:34:30.396 DEBUG:boto:Path: /ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.397 DEBUG:boto:Data: 2023-09-22T21:34:30.397 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.397 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.397 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.397 DEBUG:boto:Params: {} 2023-09-22T21:34:30.397 DEBUG:boto:Token: None 2023-09-22T21:34:30.397 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.397 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4yWsmb7ijy/ajGQJ+m0c9HZn6tk= 2023-09-22T21:34:30.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4yWsmb7ijy/ajGQJ+m0c9HZn6tk=', 'Content-Length': '0'} 2023-09-22T21:34:30.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007e4c9f7db7c51c9-00650e0866-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.442 DEBUG:boto:path=/ffzdqu-57/a 2023-09-22T21:34:30.443 DEBUG:boto:auth_path=/ffzdqu-57/a 2023-09-22T21:34:30.443 DEBUG:boto:path=/ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.443 DEBUG:boto:auth_path=/ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.443 DEBUG:boto:Method: GET 2023-09-22T21:34:30.443 DEBUG:boto:Path: /ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.443 DEBUG:boto:Data: 2023-09-22T21:34:30.443 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.443 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.443 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.443 DEBUG:boto:Params: {} 2023-09-22T21:34:30.444 DEBUG:boto:Token: None 2023-09-22T21:34:30.444 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/a?versionId=null 2023-09-22T21:34:30.444 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4yWsmb7ijy/ajGQJ+m0c9HZn6tk= 2023-09-22T21:34:30.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4yWsmb7ijy/ajGQJ+m0c9HZn6tk=', 'Content-Length': '0'} 2023-09-22T21:34:30.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f588cb8379b2ec80-00650e0866-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.447 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:30.447 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:30.447 DEBUG:boto:path=/ffzdqu-57/?prefix=a 2023-09-22T21:34:30.448 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=a 2023-09-22T21:34:30.448 DEBUG:boto:Method: GET 2023-09-22T21:34:30.448 DEBUG:boto:Path: /ffzdqu-57/?prefix=a 2023-09-22T21:34:30.448 DEBUG:boto:Data: 2023-09-22T21:34:30.448 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.448 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.448 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.448 DEBUG:boto:Params: {} 2023-09-22T21:34:30.448 DEBUG:boto:Token: None 2023-09-22T21:34:30.448 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/ 2023-09-22T21:34:30.449 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jd+zcamXazs3Fhw6B65N9P48JhM= 2023-09-22T21:34:30.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jd+zcamXazs3Fhw6B65N9P48JhM=', 'Content-Length': '0'} 2023-09-22T21:34:30.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ad520887141f030-00650e0866-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.494 DEBUG:boto:b'ffzdqu-57a1000falsea2023-09-22T21:32:23.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:30.495 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:30.495 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:30.495 DEBUG:boto:path=/ffzdqu-57/?prefix=a 2023-09-22T21:34:30.495 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=a 2023-09-22T21:34:30.495 DEBUG:boto:Method: GET 2023-09-22T21:34:30.495 DEBUG:boto:Path: /ffzdqu-57/?prefix=a 2023-09-22T21:34:30.495 DEBUG:boto:Data: 2023-09-22T21:34:30.495 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.495 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.496 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.496 DEBUG:boto:Params: {} 2023-09-22T21:34:30.496 DEBUG:boto:Token: None 2023-09-22T21:34:30.496 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/ 2023-09-22T21:34:30.496 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jd+zcamXazs3Fhw6B65N9P48JhM= 2023-09-22T21:34:30.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jd+zcamXazs3Fhw6B65N9P48JhM=', 'Content-Length': '0'} 2023-09-22T21:34:30.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c11e6face5f4e3f0-00650e0866-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.499 DEBUG:boto:b'ffzdqu-57a1000falsea2023-09-22T21:32:23.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:30.500 DEBUG:boto:path=/ffzdqu-57/a 2023-09-22T21:34:30.500 DEBUG:boto:auth_path=/ffzdqu-57/a 2023-09-22T21:34:30.500 DEBUG:boto:Method: HEAD 2023-09-22T21:34:30.500 DEBUG:boto:Path: /ffzdqu-57/a 2023-09-22T21:34:30.500 DEBUG:boto:Data: 2023-09-22T21:34:30.500 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.500 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.500 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.501 DEBUG:boto:Params: {} 2023-09-22T21:34:30.501 DEBUG:boto:Token: None 2023-09-22T21:34:30.501 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/a 2023-09-22T21:34:30.501 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0TurEceoJ657x2HvSoJcNLGWxG4= 2023-09-22T21:34:30.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0TurEceoJ657x2HvSoJcNLGWxG4=', 'Content-Length': '0'} 2023-09-22T21:34:30.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e46de4f343f043af-00650e0866-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.504 DEBUG:boto:path=/ffzdqu-57/a 2023-09-22T21:34:30.504 DEBUG:boto:auth_path=/ffzdqu-57/a 2023-09-22T21:34:30.504 DEBUG:boto:Method: HEAD 2023-09-22T21:34:30.505 DEBUG:boto:Path: /ffzdqu-57/a 2023-09-22T21:34:30.505 DEBUG:boto:Data: 2023-09-22T21:34:30.505 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.505 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.505 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.505 DEBUG:boto:Params: {} 2023-09-22T21:34:30.505 DEBUG:boto:Token: None 2023-09-22T21:34:30.505 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/a 2023-09-22T21:34:30.505 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0TurEceoJ657x2HvSoJcNLGWxG4= 2023-09-22T21:34:30.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0TurEceoJ657x2HvSoJcNLGWxG4=', 'Content-Length': '0'} 2023-09-22T21:34:30.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020fe89b94a0e1f9d-00650e0866-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-22T21:34:30.509 DEBUG:boto:path=/ffzdqu-57/a 2023-09-22T21:34:30.509 DEBUG:boto:auth_path=/ffzdqu-57/a 2023-09-22T21:34:30.509 DEBUG:boto:Method: GET 2023-09-22T21:34:30.509 DEBUG:boto:Path: /ffzdqu-57/a 2023-09-22T21:34:30.509 DEBUG:boto:Data: 2023-09-22T21:34:30.509 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.509 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.509 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.510 DEBUG:boto:Params: {} 2023-09-22T21:34:30.510 DEBUG:boto:Token: None 2023-09-22T21:34:30.510 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/a 2023-09-22T21:34:30.510 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gbAHOCJIvXyv/M5V/DoI4uMSaaA= 2023-09-22T21:34:30.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gbAHOCJIvXyv/M5V/DoI4uMSaaA=', 'Content-Length': '0'} 2023-09-22T21:34:30.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe48987b27071d2a-00650e0866-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.554 DEBUG:boto:path=/ffzdqu-57/a 2023-09-22T21:34:30.554 DEBUG:boto:auth_path=/ffzdqu-57/a 2023-09-22T21:34:30.555 DEBUG:boto:Method: GET 2023-09-22T21:34:30.555 DEBUG:boto:Path: /ffzdqu-57/a 2023-09-22T21:34:30.555 DEBUG:boto:Data: 2023-09-22T21:34:30.555 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.555 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.555 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.555 DEBUG:boto:Params: {} 2023-09-22T21:34:30.555 DEBUG:boto:Token: None 2023-09-22T21:34:30.555 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/a 2023-09-22T21:34:30.556 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gbAHOCJIvXyv/M5V/DoI4uMSaaA= 2023-09-22T21:34:30.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gbAHOCJIvXyv/M5V/DoI4uMSaaA=', 'Content-Length': '0'} 2023-09-22T21:34:30.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec1934ba25483013-00650e0866-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.559 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:30.559 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:30.559 DEBUG:boto:path=/ffzdqu-57/?prefix=a 2023-09-22T21:34:30.559 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=a 2023-09-22T21:34:30.559 DEBUG:boto:Method: GET 2023-09-22T21:34:30.559 DEBUG:boto:Path: /ffzdqu-57/?prefix=a 2023-09-22T21:34:30.559 DEBUG:boto:Data: 2023-09-22T21:34:30.560 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.560 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.560 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.560 DEBUG:boto:Params: {} 2023-09-22T21:34:30.560 DEBUG:boto:Token: None 2023-09-22T21:34:30.560 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/ 2023-09-22T21:34:30.560 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jd+zcamXazs3Fhw6B65N9P48JhM= 2023-09-22T21:34:30.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jd+zcamXazs3Fhw6B65N9P48JhM=', 'Content-Length': '0'} 2023-09-22T21:34:30.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d07f43ff6498ccb5-00650e0866-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.606 DEBUG:boto:b'ffzdqu-57a1000falsea2023-09-22T21:32:23.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:30.607 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:30.607 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:30.607 DEBUG:boto:path=/ffzdqu-57/?prefix=a 2023-09-22T21:34:30.607 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=a 2023-09-22T21:34:30.607 DEBUG:boto:Method: GET 2023-09-22T21:34:30.607 DEBUG:boto:Path: /ffzdqu-57/?prefix=a 2023-09-22T21:34:30.607 DEBUG:boto:Data: 2023-09-22T21:34:30.607 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.608 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.608 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.608 DEBUG:boto:Params: {} 2023-09-22T21:34:30.608 DEBUG:boto:Token: None 2023-09-22T21:34:30.608 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/ 2023-09-22T21:34:30.608 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jd+zcamXazs3Fhw6B65N9P48JhM= 2023-09-22T21:34:30.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jd+zcamXazs3Fhw6B65N9P48JhM=', 'Content-Length': '0'} 2023-09-22T21:34:30.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004507a5745de17cee-00650e0866-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.612 DEBUG:boto:b'ffzdqu-57a1000falsea2023-09-22T21:32:23.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:30.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:34:30.612 DEBUG:boto:path=/ffzdqu-57/b 2023-09-22T21:34:30.612 DEBUG:boto:auth_path=/ffzdqu-57/b 2023-09-22T21:34:30.613 DEBUG:boto:path=/ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.613 DEBUG:boto:auth_path=/ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.613 DEBUG:boto:Method: GET 2023-09-22T21:34:30.613 DEBUG:boto:Path: /ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.613 DEBUG:boto:Data: 2023-09-22T21:34:30.613 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.613 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.613 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.613 DEBUG:boto:Params: {} 2023-09-22T21:34:30.613 DEBUG:boto:Token: None 2023-09-22T21:34:30.614 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.614 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CRanq4K6x9+x+Ai52c9XSAJL1rM= 2023-09-22T21:34:30.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRanq4K6x9+x+Ai52c9XSAJL1rM=', 'Content-Length': '0'} 2023-09-22T21:34:30.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006766841dd93efb54-00650e0866-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.617 DEBUG:boto:path=/ffzdqu-57/b 2023-09-22T21:34:30.618 DEBUG:boto:auth_path=/ffzdqu-57/b 2023-09-22T21:34:30.618 DEBUG:boto:path=/ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.618 DEBUG:boto:auth_path=/ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.618 DEBUG:boto:Method: GET 2023-09-22T21:34:30.618 DEBUG:boto:Path: /ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.618 DEBUG:boto:Data: 2023-09-22T21:34:30.618 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.618 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.618 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.618 DEBUG:boto:Params: {} 2023-09-22T21:34:30.619 DEBUG:boto:Token: None 2023-09-22T21:34:30.619 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.619 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CRanq4K6x9+x+Ai52c9XSAJL1rM= 2023-09-22T21:34:30.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRanq4K6x9+x+Ai52c9XSAJL1rM=', 'Content-Length': '0'} 2023-09-22T21:34:30.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093b6e5b84a6e33e4-00650e0866-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.622 DEBUG:boto:path=/ffzdqu-57/b 2023-09-22T21:34:30.622 DEBUG:boto:auth_path=/ffzdqu-57/b 2023-09-22T21:34:30.622 DEBUG:boto:path=/ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.622 DEBUG:boto:auth_path=/ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.622 DEBUG:boto:Method: HEAD 2023-09-22T21:34:30.623 DEBUG:boto:Path: /ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.623 DEBUG:boto:Data: 2023-09-22T21:34:30.623 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.623 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.623 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.623 DEBUG:boto:Params: {} 2023-09-22T21:34:30.623 DEBUG:boto:Token: None 2023-09-22T21:34:30.623 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.623 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t/q82RSiL4GCBI4zhy/iJLmDBF0= 2023-09-22T21:34:30.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t/q82RSiL4GCBI4zhy/iJLmDBF0=', 'Content-Length': '0'} 2023-09-22T21:34:30.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db721c815395f384-00650e0866-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.626 DEBUG:boto:path=/ffzdqu-57/b 2023-09-22T21:34:30.627 DEBUG:boto:auth_path=/ffzdqu-57/b 2023-09-22T21:34:30.627 DEBUG:boto:path=/ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.627 DEBUG:boto:auth_path=/ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.627 DEBUG:boto:Method: HEAD 2023-09-22T21:34:30.627 DEBUG:boto:Path: /ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.627 DEBUG:boto:Data: 2023-09-22T21:34:30.627 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.627 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.627 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.627 DEBUG:boto:Params: {} 2023-09-22T21:34:30.628 DEBUG:boto:Token: None 2023-09-22T21:34:30.628 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.628 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t/q82RSiL4GCBI4zhy/iJLmDBF0= 2023-09-22T21:34:30.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t/q82RSiL4GCBI4zhy/iJLmDBF0=', 'Content-Length': '0'} 2023-09-22T21:34:30.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e01411e40af88fe3-00650e0866-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:34:30.631 DEBUG:boto:path=/ffzdqu-57/b 2023-09-22T21:34:30.631 DEBUG:boto:auth_path=/ffzdqu-57/b 2023-09-22T21:34:30.631 DEBUG:boto:path=/ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.631 DEBUG:boto:auth_path=/ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.631 DEBUG:boto:Method: GET 2023-09-22T21:34:30.632 DEBUG:boto:Path: /ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.632 DEBUG:boto:Data: 2023-09-22T21:34:30.632 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.632 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.632 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.632 DEBUG:boto:Params: {} 2023-09-22T21:34:30.632 DEBUG:boto:Token: None 2023-09-22T21:34:30.632 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.632 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CRanq4K6x9+x+Ai52c9XSAJL1rM= 2023-09-22T21:34:30.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRanq4K6x9+x+Ai52c9XSAJL1rM=', 'Content-Length': '0'} 2023-09-22T21:34:30.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000536a183667103f69-00650e0866-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.678 DEBUG:boto:path=/ffzdqu-57/b 2023-09-22T21:34:30.678 DEBUG:boto:auth_path=/ffzdqu-57/b 2023-09-22T21:34:30.679 DEBUG:boto:path=/ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.679 DEBUG:boto:auth_path=/ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.679 DEBUG:boto:Method: GET 2023-09-22T21:34:30.679 DEBUG:boto:Path: /ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.679 DEBUG:boto:Data: 2023-09-22T21:34:30.679 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.679 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.679 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.679 DEBUG:boto:Params: {} 2023-09-22T21:34:30.679 DEBUG:boto:Token: None 2023-09-22T21:34:30.680 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/b?versionId=null 2023-09-22T21:34:30.680 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CRanq4K6x9+x+Ai52c9XSAJL1rM= 2023-09-22T21:34:30.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRanq4K6x9+x+Ai52c9XSAJL1rM=', 'Content-Length': '0'} 2023-09-22T21:34:30.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015980ced0c915b4e-00650e0866-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.683 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:30.683 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:30.683 DEBUG:boto:path=/ffzdqu-57/?prefix=b 2023-09-22T21:34:30.683 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=b 2023-09-22T21:34:30.683 DEBUG:boto:Method: GET 2023-09-22T21:34:30.684 DEBUG:boto:Path: /ffzdqu-57/?prefix=b 2023-09-22T21:34:30.684 DEBUG:boto:Data: 2023-09-22T21:34:30.684 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.684 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.684 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.684 DEBUG:boto:Params: {} 2023-09-22T21:34:30.684 DEBUG:boto:Token: None 2023-09-22T21:34:30.684 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/ 2023-09-22T21:34:30.684 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jd+zcamXazs3Fhw6B65N9P48JhM= 2023-09-22T21:34:30.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jd+zcamXazs3Fhw6B65N9P48JhM=', 'Content-Length': '0'} 2023-09-22T21:34:30.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2840fbf8e3f4656-00650e0866-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.730 DEBUG:boto:b'ffzdqu-57b1000falseb2023-09-22T21:32:23.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:30.731 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:30.731 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:30.731 DEBUG:boto:path=/ffzdqu-57/?prefix=b 2023-09-22T21:34:30.731 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=b 2023-09-22T21:34:30.731 DEBUG:boto:Method: GET 2023-09-22T21:34:30.731 DEBUG:boto:Path: /ffzdqu-57/?prefix=b 2023-09-22T21:34:30.731 DEBUG:boto:Data: 2023-09-22T21:34:30.731 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.731 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.731 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.732 DEBUG:boto:Params: {} 2023-09-22T21:34:30.732 DEBUG:boto:Token: None 2023-09-22T21:34:30.732 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/ 2023-09-22T21:34:30.732 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jd+zcamXazs3Fhw6B65N9P48JhM= 2023-09-22T21:34:30.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jd+zcamXazs3Fhw6B65N9P48JhM=', 'Content-Length': '0'} 2023-09-22T21:34:30.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080a9d57099529389-00650e0866-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.735 DEBUG:boto:b'ffzdqu-57b1000falseb2023-09-22T21:32:23.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:30.735 DEBUG:boto:path=/ffzdqu-57/b 2023-09-22T21:34:30.735 DEBUG:boto:auth_path=/ffzdqu-57/b 2023-09-22T21:34:30.736 DEBUG:boto:Method: HEAD 2023-09-22T21:34:30.736 DEBUG:boto:Path: /ffzdqu-57/b 2023-09-22T21:34:30.736 DEBUG:boto:Data: 2023-09-22T21:34:30.736 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.736 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.736 DEBUG:boto:Params: {} 2023-09-22T21:34:30.736 DEBUG:boto:Token: None 2023-09-22T21:34:30.736 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/b 2023-09-22T21:34:30.737 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vgva4+ZOtJbBlqlqqOWZR+8hO8w= 2023-09-22T21:34:30.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vgva4+ZOtJbBlqlqqOWZR+8hO8w=', 'Content-Length': '0'} 2023-09-22T21:34:30.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de791b8048182955-00650e0866-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.740 DEBUG:boto:path=/ffzdqu-57/b 2023-09-22T21:34:30.740 DEBUG:boto:auth_path=/ffzdqu-57/b 2023-09-22T21:34:30.740 DEBUG:boto:Method: HEAD 2023-09-22T21:34:30.740 DEBUG:boto:Path: /ffzdqu-57/b 2023-09-22T21:34:30.740 DEBUG:boto:Data: 2023-09-22T21:34:30.740 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.740 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.740 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.740 DEBUG:boto:Params: {} 2023-09-22T21:34:30.741 DEBUG:boto:Token: None 2023-09-22T21:34:30.741 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/b 2023-09-22T21:34:30.741 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vgva4+ZOtJbBlqlqqOWZR+8hO8w= 2023-09-22T21:34:30.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vgva4+ZOtJbBlqlqqOWZR+8hO8w=', 'Content-Length': '0'} 2023-09-22T21:34:30.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f9d4c18216af7d9-00650e0866-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-22T21:34:30.744 DEBUG:boto:path=/ffzdqu-57/b 2023-09-22T21:34:30.744 DEBUG:boto:auth_path=/ffzdqu-57/b 2023-09-22T21:34:30.744 DEBUG:boto:Method: GET 2023-09-22T21:34:30.744 DEBUG:boto:Path: /ffzdqu-57/b 2023-09-22T21:34:30.744 DEBUG:boto:Data: 2023-09-22T21:34:30.744 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.744 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.744 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.744 DEBUG:boto:Params: {} 2023-09-22T21:34:30.745 DEBUG:boto:Token: None 2023-09-22T21:34:30.745 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/b 2023-09-22T21:34:30.745 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XGpuvoThz2fDhYDkEfwmMOc6Vxg= 2023-09-22T21:34:30.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XGpuvoThz2fDhYDkEfwmMOc6Vxg=', 'Content-Length': '0'} 2023-09-22T21:34:30.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a996580ef4a2049-00650e0866-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.790 DEBUG:boto:path=/ffzdqu-57/b 2023-09-22T21:34:30.790 DEBUG:boto:auth_path=/ffzdqu-57/b 2023-09-22T21:34:30.791 DEBUG:boto:Method: GET 2023-09-22T21:34:30.791 DEBUG:boto:Path: /ffzdqu-57/b 2023-09-22T21:34:30.791 DEBUG:boto:Data: 2023-09-22T21:34:30.791 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.791 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.791 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.791 DEBUG:boto:Params: {} 2023-09-22T21:34:30.791 DEBUG:boto:Token: None 2023-09-22T21:34:30.791 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/b 2023-09-22T21:34:30.792 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XGpuvoThz2fDhYDkEfwmMOc6Vxg= 2023-09-22T21:34:30.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XGpuvoThz2fDhYDkEfwmMOc6Vxg=', 'Content-Length': '0'} 2023-09-22T21:34:30.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030be455ef4805fc1-00650e0866-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.795 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:30.795 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:30.795 DEBUG:boto:path=/ffzdqu-57/?prefix=b 2023-09-22T21:34:30.795 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=b 2023-09-22T21:34:30.795 DEBUG:boto:Method: GET 2023-09-22T21:34:30.795 DEBUG:boto:Path: /ffzdqu-57/?prefix=b 2023-09-22T21:34:30.795 DEBUG:boto:Data: 2023-09-22T21:34:30.795 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.795 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.796 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.796 DEBUG:boto:Params: {} 2023-09-22T21:34:30.796 DEBUG:boto:Token: None 2023-09-22T21:34:30.796 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/ 2023-09-22T21:34:30.796 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jd+zcamXazs3Fhw6B65N9P48JhM= 2023-09-22T21:34:30.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jd+zcamXazs3Fhw6B65N9P48JhM=', 'Content-Length': '0'} 2023-09-22T21:34:30.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd675005bdd35544-00650e0866-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.842 DEBUG:boto:b'ffzdqu-57b1000falseb2023-09-22T21:32:23.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:30.843 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:30.843 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:30.843 DEBUG:boto:path=/ffzdqu-57/?prefix=b 2023-09-22T21:34:30.843 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=b 2023-09-22T21:34:30.843 DEBUG:boto:Method: GET 2023-09-22T21:34:30.843 DEBUG:boto:Path: /ffzdqu-57/?prefix=b 2023-09-22T21:34:30.843 DEBUG:boto:Data: 2023-09-22T21:34:30.843 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.843 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.844 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.844 DEBUG:boto:Params: {} 2023-09-22T21:34:30.844 DEBUG:boto:Token: None 2023-09-22T21:34:30.844 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/ 2023-09-22T21:34:30.844 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jd+zcamXazs3Fhw6B65N9P48JhM= 2023-09-22T21:34:30.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jd+zcamXazs3Fhw6B65N9P48JhM=', 'Content-Length': '0'} 2023-09-22T21:34:30.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000058ecd17ae3a96a4-00650e0866-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.848 DEBUG:boto:b'ffzdqu-57b1000falseb2023-09-22T21:32:23.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:30.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:34:30.848 DEBUG:boto:path=/ffzdqu-57/c 2023-09-22T21:34:30.849 DEBUG:boto:auth_path=/ffzdqu-57/c 2023-09-22T21:34:30.849 DEBUG:boto:path=/ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.849 DEBUG:boto:auth_path=/ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.849 DEBUG:boto:Method: GET 2023-09-22T21:34:30.849 DEBUG:boto:Path: /ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.849 DEBUG:boto:Data: 2023-09-22T21:34:30.849 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.849 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.849 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.849 DEBUG:boto:Params: {} 2023-09-22T21:34:30.850 DEBUG:boto:Token: None 2023-09-22T21:34:30.850 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.850 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b5b4t9dhHzc21STs1cAtpYiQweY= 2023-09-22T21:34:30.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b5b4t9dhHzc21STs1cAtpYiQweY=', 'Content-Length': '0'} 2023-09-22T21:34:30.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceecfd6e9634322e-00650e0866-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.853 DEBUG:boto:path=/ffzdqu-57/c 2023-09-22T21:34:30.854 DEBUG:boto:auth_path=/ffzdqu-57/c 2023-09-22T21:34:30.854 DEBUG:boto:path=/ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.854 DEBUG:boto:auth_path=/ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.854 DEBUG:boto:Method: GET 2023-09-22T21:34:30.854 DEBUG:boto:Path: /ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.854 DEBUG:boto:Data: 2023-09-22T21:34:30.854 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.854 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.854 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.854 DEBUG:boto:Params: {} 2023-09-22T21:34:30.855 DEBUG:boto:Token: None 2023-09-22T21:34:30.855 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.855 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b5b4t9dhHzc21STs1cAtpYiQweY= 2023-09-22T21:34:30.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b5b4t9dhHzc21STs1cAtpYiQweY=', 'Content-Length': '0'} 2023-09-22T21:34:30.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d2a85f290d82bf0-00650e0866-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.858 DEBUG:boto:path=/ffzdqu-57/c 2023-09-22T21:34:30.858 DEBUG:boto:auth_path=/ffzdqu-57/c 2023-09-22T21:34:30.858 DEBUG:boto:path=/ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.858 DEBUG:boto:auth_path=/ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.858 DEBUG:boto:Method: HEAD 2023-09-22T21:34:30.858 DEBUG:boto:Path: /ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.859 DEBUG:boto:Data: 2023-09-22T21:34:30.859 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.859 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.859 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.859 DEBUG:boto:Params: {} 2023-09-22T21:34:30.859 DEBUG:boto:Token: None 2023-09-22T21:34:30.859 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.859 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iRn5p4bkiuXgebGBfYVKICCILSw= 2023-09-22T21:34:30.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iRn5p4bkiuXgebGBfYVKICCILSw=', 'Content-Length': '0'} 2023-09-22T21:34:30.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000807b2f4eacd32785-00650e0866-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.863 DEBUG:boto:path=/ffzdqu-57/c 2023-09-22T21:34:30.863 DEBUG:boto:auth_path=/ffzdqu-57/c 2023-09-22T21:34:30.863 DEBUG:boto:path=/ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.863 DEBUG:boto:auth_path=/ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.863 DEBUG:boto:Method: HEAD 2023-09-22T21:34:30.863 DEBUG:boto:Path: /ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.863 DEBUG:boto:Data: 2023-09-22T21:34:30.863 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.864 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.864 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.864 DEBUG:boto:Params: {} 2023-09-22T21:34:30.864 DEBUG:boto:Token: None 2023-09-22T21:34:30.864 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.864 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iRn5p4bkiuXgebGBfYVKICCILSw= 2023-09-22T21:34:30.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iRn5p4bkiuXgebGBfYVKICCILSw=', 'Content-Length': '0'} 2023-09-22T21:34:30.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e41eca83b29655d-00650e0866-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:34:30.867 DEBUG:boto:path=/ffzdqu-57/c 2023-09-22T21:34:30.867 DEBUG:boto:auth_path=/ffzdqu-57/c 2023-09-22T21:34:30.867 DEBUG:boto:path=/ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.867 DEBUG:boto:auth_path=/ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.867 DEBUG:boto:Method: GET 2023-09-22T21:34:30.867 DEBUG:boto:Path: /ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.867 DEBUG:boto:Data: 2023-09-22T21:34:30.868 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.868 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.868 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.868 DEBUG:boto:Params: {} 2023-09-22T21:34:30.868 DEBUG:boto:Token: None 2023-09-22T21:34:30.868 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.868 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b5b4t9dhHzc21STs1cAtpYiQweY= 2023-09-22T21:34:30.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b5b4t9dhHzc21STs1cAtpYiQweY=', 'Content-Length': '0'} 2023-09-22T21:34:30.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031c5674e66edbe8a-00650e0866-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.914 DEBUG:boto:path=/ffzdqu-57/c 2023-09-22T21:34:30.914 DEBUG:boto:auth_path=/ffzdqu-57/c 2023-09-22T21:34:30.915 DEBUG:boto:path=/ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.915 DEBUG:boto:auth_path=/ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.915 DEBUG:boto:Method: GET 2023-09-22T21:34:30.915 DEBUG:boto:Path: /ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.915 DEBUG:boto:Data: 2023-09-22T21:34:30.915 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.915 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.915 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.915 DEBUG:boto:Params: {} 2023-09-22T21:34:30.915 DEBUG:boto:Token: None 2023-09-22T21:34:30.916 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/c?versionId=null 2023-09-22T21:34:30.916 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b5b4t9dhHzc21STs1cAtpYiQweY= 2023-09-22T21:34:30.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b5b4t9dhHzc21STs1cAtpYiQweY=', 'Content-Length': '0'} 2023-09-22T21:34:30.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a8ea0a62971071d-00650e0866-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.919 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:30.919 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:30.919 DEBUG:boto:path=/ffzdqu-57/?prefix=c 2023-09-22T21:34:30.920 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=c 2023-09-22T21:34:30.920 DEBUG:boto:Method: GET 2023-09-22T21:34:30.920 DEBUG:boto:Path: /ffzdqu-57/?prefix=c 2023-09-22T21:34:30.920 DEBUG:boto:Data: 2023-09-22T21:34:30.920 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.920 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.920 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.920 DEBUG:boto:Params: {} 2023-09-22T21:34:30.920 DEBUG:boto:Token: None 2023-09-22T21:34:30.920 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/ 2023-09-22T21:34:30.921 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jd+zcamXazs3Fhw6B65N9P48JhM= 2023-09-22T21:34:30.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jd+zcamXazs3Fhw6B65N9P48JhM=', 'Content-Length': '0'} 2023-09-22T21:34:30.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecc182e875f095b1-00650e0866-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.966 DEBUG:boto:b'ffzdqu-57c1000falsec2023-09-22T21:32:23.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:30.967 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:30.967 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:30.967 DEBUG:boto:path=/ffzdqu-57/?prefix=c 2023-09-22T21:34:30.967 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=c 2023-09-22T21:34:30.967 DEBUG:boto:Method: GET 2023-09-22T21:34:30.967 DEBUG:boto:Path: /ffzdqu-57/?prefix=c 2023-09-22T21:34:30.967 DEBUG:boto:Data: 2023-09-22T21:34:30.968 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.968 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.968 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.968 DEBUG:boto:Params: {} 2023-09-22T21:34:30.968 DEBUG:boto:Token: None 2023-09-22T21:34:30.968 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/ 2023-09-22T21:34:30.968 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jd+zcamXazs3Fhw6B65N9P48JhM= 2023-09-22T21:34:30.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jd+zcamXazs3Fhw6B65N9P48JhM=', 'Content-Length': '0'} 2023-09-22T21:34:30.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009159e40603ad5c5e-00650e0866-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.972 DEBUG:boto:b'ffzdqu-57c1000falsec2023-09-22T21:32:23.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:30.972 DEBUG:boto:path=/ffzdqu-57/c 2023-09-22T21:34:30.972 DEBUG:boto:auth_path=/ffzdqu-57/c 2023-09-22T21:34:30.973 DEBUG:boto:Method: HEAD 2023-09-22T21:34:30.973 DEBUG:boto:Path: /ffzdqu-57/c 2023-09-22T21:34:30.973 DEBUG:boto:Data: 2023-09-22T21:34:30.973 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.973 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.973 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.973 DEBUG:boto:Params: {} 2023-09-22T21:34:30.973 DEBUG:boto:Token: None 2023-09-22T21:34:30.973 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/c 2023-09-22T21:34:30.973 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cH1bL0r55OoEwP4VaGO15U8S6XI= 2023-09-22T21:34:30.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cH1bL0r55OoEwP4VaGO15U8S6XI=', 'Content-Length': '0'} 2023-09-22T21:34:30.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff68336e2674418f-00650e0866-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.976 DEBUG:boto:path=/ffzdqu-57/c 2023-09-22T21:34:30.977 DEBUG:boto:auth_path=/ffzdqu-57/c 2023-09-22T21:34:30.977 DEBUG:boto:Method: HEAD 2023-09-22T21:34:30.977 DEBUG:boto:Path: /ffzdqu-57/c 2023-09-22T21:34:30.977 DEBUG:boto:Data: 2023-09-22T21:34:30.977 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.977 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:30.977 DEBUG:boto:Port: 8000 2023-09-22T21:34:30.977 DEBUG:boto:Params: {} 2023-09-22T21:34:30.977 DEBUG:boto:Token: None 2023-09-22T21:34:30.978 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/c 2023-09-22T21:34:30.978 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cH1bL0r55OoEwP4VaGO15U8S6XI= 2023-09-22T21:34:30.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cH1bL0r55OoEwP4VaGO15U8S6XI=', 'Content-Length': '0'} 2023-09-22T21:34:30.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1da106f7f749259-00650e0866-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:30.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-22T21:34:30.981 DEBUG:boto:path=/ffzdqu-57/c 2023-09-22T21:34:30.981 DEBUG:boto:auth_path=/ffzdqu-57/c 2023-09-22T21:34:30.981 DEBUG:boto:Method: GET 2023-09-22T21:34:30.981 DEBUG:boto:Path: /ffzdqu-57/c 2023-09-22T21:34:30.981 DEBUG:boto:Data: 2023-09-22T21:34:30.981 DEBUG:boto:Headers: {} 2023-09-22T21:34:30.981 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:30.981 DEBUG:boto:Port: 8001 2023-09-22T21:34:30.981 DEBUG:boto:Params: {} 2023-09-22T21:34:30.981 DEBUG:boto:Token: None 2023-09-22T21:34:30.982 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:30 GMT /ffzdqu-57/c 2023-09-22T21:34:30.982 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hjVkrDWnOC+hIdq+/fyrpQQZMSE= 2023-09-22T21:34:30.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hjVkrDWnOC+hIdq+/fyrpQQZMSE=', 'Content-Length': '0'} 2023-09-22T21:34:30.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bcc4a70f327b29a-00650e0866-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.026 DEBUG:boto:path=/ffzdqu-57/c 2023-09-22T21:34:31.026 DEBUG:boto:auth_path=/ffzdqu-57/c 2023-09-22T21:34:31.027 DEBUG:boto:Method: GET 2023-09-22T21:34:31.027 DEBUG:boto:Path: /ffzdqu-57/c 2023-09-22T21:34:31.027 DEBUG:boto:Data: 2023-09-22T21:34:31.027 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.027 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.027 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.027 DEBUG:boto:Params: {} 2023-09-22T21:34:31.027 DEBUG:boto:Token: None 2023-09-22T21:34:31.028 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/c 2023-09-22T21:34:31.028 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1mbRoKgOJKTH5Kh2rTEsqMreOMY= 2023-09-22T21:34:31.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1mbRoKgOJKTH5Kh2rTEsqMreOMY=', 'Content-Length': '0'} 2023-09-22T21:34:31.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a6b7c0da38d62cc-00650e0867-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.031 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:31.031 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:31.031 DEBUG:boto:path=/ffzdqu-57/?prefix=c 2023-09-22T21:34:31.031 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=c 2023-09-22T21:34:31.031 DEBUG:boto:Method: GET 2023-09-22T21:34:31.031 DEBUG:boto:Path: /ffzdqu-57/?prefix=c 2023-09-22T21:34:31.031 DEBUG:boto:Data: 2023-09-22T21:34:31.031 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.032 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.032 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.032 DEBUG:boto:Params: {} 2023-09-22T21:34:31.032 DEBUG:boto:Token: None 2023-09-22T21:34:31.032 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/ 2023-09-22T21:34:31.032 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk= 2023-09-22T21:34:31.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk=', 'Content-Length': '0'} 2023-09-22T21:34:31.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044e6cf6406d96c2d-00650e0867-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.078 DEBUG:boto:b'ffzdqu-57c1000falsec2023-09-22T21:32:23.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:31.079 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:31.079 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:31.079 DEBUG:boto:path=/ffzdqu-57/?prefix=c 2023-09-22T21:34:31.079 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=c 2023-09-22T21:34:31.079 DEBUG:boto:Method: GET 2023-09-22T21:34:31.079 DEBUG:boto:Path: /ffzdqu-57/?prefix=c 2023-09-22T21:34:31.080 DEBUG:boto:Data: 2023-09-22T21:34:31.080 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.080 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.080 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.080 DEBUG:boto:Params: {} 2023-09-22T21:34:31.080 DEBUG:boto:Token: None 2023-09-22T21:34:31.080 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/ 2023-09-22T21:34:31.080 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk= 2023-09-22T21:34:31.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk=', 'Content-Length': '0'} 2023-09-22T21:34:31.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bcd2a8426108661-00650e0867-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.084 DEBUG:boto:b'ffzdqu-57c1000falsec2023-09-22T21:32:23.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:31.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:34:31.085 DEBUG:boto:path=/ffzdqu-57/d 2023-09-22T21:34:31.085 DEBUG:boto:auth_path=/ffzdqu-57/d 2023-09-22T21:34:31.085 DEBUG:boto:path=/ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.085 DEBUG:boto:auth_path=/ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.085 DEBUG:boto:Method: GET 2023-09-22T21:34:31.085 DEBUG:boto:Path: /ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.085 DEBUG:boto:Data: 2023-09-22T21:34:31.086 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.086 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.086 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.086 DEBUG:boto:Params: {} 2023-09-22T21:34:31.086 DEBUG:boto:Token: None 2023-09-22T21:34:31.086 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.086 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ktp94GotP6cb/+uCCodhf0Ez1yU= 2023-09-22T21:34:31.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ktp94GotP6cb/+uCCodhf0Ez1yU=', 'Content-Length': '0'} 2023-09-22T21:34:31.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032f529c9af40e84f-00650e0867-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.089 DEBUG:boto:path=/ffzdqu-57/d 2023-09-22T21:34:31.090 DEBUG:boto:auth_path=/ffzdqu-57/d 2023-09-22T21:34:31.090 DEBUG:boto:path=/ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.090 DEBUG:boto:auth_path=/ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.090 DEBUG:boto:Method: GET 2023-09-22T21:34:31.090 DEBUG:boto:Path: /ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.090 DEBUG:boto:Data: 2023-09-22T21:34:31.090 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.090 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.090 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.091 DEBUG:boto:Params: {} 2023-09-22T21:34:31.091 DEBUG:boto:Token: None 2023-09-22T21:34:31.091 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.091 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ktp94GotP6cb/+uCCodhf0Ez1yU= 2023-09-22T21:34:31.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ktp94GotP6cb/+uCCodhf0Ez1yU=', 'Content-Length': '0'} 2023-09-22T21:34:31.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e29dd4c5c5844c2c-00650e0867-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.134 DEBUG:boto:path=/ffzdqu-57/d 2023-09-22T21:34:31.134 DEBUG:boto:auth_path=/ffzdqu-57/d 2023-09-22T21:34:31.135 DEBUG:boto:path=/ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.135 DEBUG:boto:auth_path=/ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.135 DEBUG:boto:Method: HEAD 2023-09-22T21:34:31.135 DEBUG:boto:Path: /ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.135 DEBUG:boto:Data: 2023-09-22T21:34:31.135 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.135 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.135 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.135 DEBUG:boto:Params: {} 2023-09-22T21:34:31.136 DEBUG:boto:Token: None 2023-09-22T21:34:31.136 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.136 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M6HKSkL0bloSdZ1YZJjwQESkCpk= 2023-09-22T21:34:31.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M6HKSkL0bloSdZ1YZJjwQESkCpk=', 'Content-Length': '0'} 2023-09-22T21:34:31.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003130e260bbf9c8cb-00650e0867-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.139 DEBUG:boto:path=/ffzdqu-57/d 2023-09-22T21:34:31.139 DEBUG:boto:auth_path=/ffzdqu-57/d 2023-09-22T21:34:31.139 DEBUG:boto:path=/ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.139 DEBUG:boto:auth_path=/ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.139 DEBUG:boto:Method: HEAD 2023-09-22T21:34:31.139 DEBUG:boto:Path: /ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.140 DEBUG:boto:Data: 2023-09-22T21:34:31.140 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.140 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.140 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.140 DEBUG:boto:Params: {} 2023-09-22T21:34:31.140 DEBUG:boto:Token: None 2023-09-22T21:34:31.140 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.140 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M6HKSkL0bloSdZ1YZJjwQESkCpk= 2023-09-22T21:34:31.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M6HKSkL0bloSdZ1YZJjwQESkCpk=', 'Content-Length': '0'} 2023-09-22T21:34:31.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081f82f0211cded6f-00650e0867-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:34:31.144 DEBUG:boto:path=/ffzdqu-57/d 2023-09-22T21:34:31.144 DEBUG:boto:auth_path=/ffzdqu-57/d 2023-09-22T21:34:31.144 DEBUG:boto:path=/ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.144 DEBUG:boto:auth_path=/ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.144 DEBUG:boto:Method: GET 2023-09-22T21:34:31.144 DEBUG:boto:Path: /ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.144 DEBUG:boto:Data: 2023-09-22T21:34:31.145 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.145 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.145 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.145 DEBUG:boto:Params: {} 2023-09-22T21:34:31.145 DEBUG:boto:Token: None 2023-09-22T21:34:31.145 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.145 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ktp94GotP6cb/+uCCodhf0Ez1yU= 2023-09-22T21:34:31.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ktp94GotP6cb/+uCCodhf0Ez1yU=', 'Content-Length': '0'} 2023-09-22T21:34:31.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de0e36f3f63e32ca-00650e0867-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.190 DEBUG:boto:path=/ffzdqu-57/d 2023-09-22T21:34:31.190 DEBUG:boto:auth_path=/ffzdqu-57/d 2023-09-22T21:34:31.191 DEBUG:boto:path=/ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.191 DEBUG:boto:auth_path=/ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.191 DEBUG:boto:Method: GET 2023-09-22T21:34:31.191 DEBUG:boto:Path: /ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.191 DEBUG:boto:Data: 2023-09-22T21:34:31.191 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.191 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.191 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.191 DEBUG:boto:Params: {} 2023-09-22T21:34:31.191 DEBUG:boto:Token: None 2023-09-22T21:34:31.192 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/d?versionId=null 2023-09-22T21:34:31.192 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ktp94GotP6cb/+uCCodhf0Ez1yU= 2023-09-22T21:34:31.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ktp94GotP6cb/+uCCodhf0Ez1yU=', 'Content-Length': '0'} 2023-09-22T21:34:31.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000717de6713b526df7-00650e0867-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.195 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:31.195 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:31.195 DEBUG:boto:path=/ffzdqu-57/?prefix=d 2023-09-22T21:34:31.195 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=d 2023-09-22T21:34:31.195 DEBUG:boto:Method: GET 2023-09-22T21:34:31.195 DEBUG:boto:Path: /ffzdqu-57/?prefix=d 2023-09-22T21:34:31.196 DEBUG:boto:Data: 2023-09-22T21:34:31.196 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.196 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.196 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.196 DEBUG:boto:Params: {} 2023-09-22T21:34:31.196 DEBUG:boto:Token: None 2023-09-22T21:34:31.196 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/ 2023-09-22T21:34:31.196 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk= 2023-09-22T21:34:31.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk=', 'Content-Length': '0'} 2023-09-22T21:34:31.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015cf00653b0a65dc-00650e0867-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.242 DEBUG:boto:b'ffzdqu-57d1000falsed2023-09-22T21:32:23.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:31.243 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:31.243 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:31.243 DEBUG:boto:path=/ffzdqu-57/?prefix=d 2023-09-22T21:34:31.243 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=d 2023-09-22T21:34:31.243 DEBUG:boto:Method: GET 2023-09-22T21:34:31.243 DEBUG:boto:Path: /ffzdqu-57/?prefix=d 2023-09-22T21:34:31.243 DEBUG:boto:Data: 2023-09-22T21:34:31.243 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.244 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.244 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.244 DEBUG:boto:Params: {} 2023-09-22T21:34:31.244 DEBUG:boto:Token: None 2023-09-22T21:34:31.244 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/ 2023-09-22T21:34:31.244 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk= 2023-09-22T21:34:31.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk=', 'Content-Length': '0'} 2023-09-22T21:34:31.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d86d0b205c3a4761-00650e0867-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.248 DEBUG:boto:b'ffzdqu-57d1000falsed2023-09-22T21:32:23.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:31.248 DEBUG:boto:path=/ffzdqu-57/d 2023-09-22T21:34:31.248 DEBUG:boto:auth_path=/ffzdqu-57/d 2023-09-22T21:34:31.248 DEBUG:boto:Method: HEAD 2023-09-22T21:34:31.248 DEBUG:boto:Path: /ffzdqu-57/d 2023-09-22T21:34:31.248 DEBUG:boto:Data: 2023-09-22T21:34:31.249 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.249 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.249 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.249 DEBUG:boto:Params: {} 2023-09-22T21:34:31.249 DEBUG:boto:Token: None 2023-09-22T21:34:31.249 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/d 2023-09-22T21:34:31.249 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M74o0btlAHcvvlV8D18nVrEfQS8= 2023-09-22T21:34:31.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M74o0btlAHcvvlV8D18nVrEfQS8=', 'Content-Length': '0'} 2023-09-22T21:34:31.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3618c7a45181019-00650e0867-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.252 DEBUG:boto:path=/ffzdqu-57/d 2023-09-22T21:34:31.252 DEBUG:boto:auth_path=/ffzdqu-57/d 2023-09-22T21:34:31.252 DEBUG:boto:Method: HEAD 2023-09-22T21:34:31.252 DEBUG:boto:Path: /ffzdqu-57/d 2023-09-22T21:34:31.252 DEBUG:boto:Data: 2023-09-22T21:34:31.253 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.253 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.253 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.253 DEBUG:boto:Params: {} 2023-09-22T21:34:31.253 DEBUG:boto:Token: None 2023-09-22T21:34:31.253 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/d 2023-09-22T21:34:31.253 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M74o0btlAHcvvlV8D18nVrEfQS8= 2023-09-22T21:34:31.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M74o0btlAHcvvlV8D18nVrEfQS8=', 'Content-Length': '0'} 2023-09-22T21:34:31.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5e913733a573634-00650e0867-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-22T21:34:31.257 DEBUG:boto:path=/ffzdqu-57/d 2023-09-22T21:34:31.257 DEBUG:boto:auth_path=/ffzdqu-57/d 2023-09-22T21:34:31.257 DEBUG:boto:Method: GET 2023-09-22T21:34:31.257 DEBUG:boto:Path: /ffzdqu-57/d 2023-09-22T21:34:31.257 DEBUG:boto:Data: 2023-09-22T21:34:31.257 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.257 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.258 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.258 DEBUG:boto:Params: {} 2023-09-22T21:34:31.258 DEBUG:boto:Token: None 2023-09-22T21:34:31.258 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/d 2023-09-22T21:34:31.258 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oxtD4xMkgqhE2g4otZp+UIKPk8Q= 2023-09-22T21:34:31.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oxtD4xMkgqhE2g4otZp+UIKPk8Q=', 'Content-Length': '0'} 2023-09-22T21:34:31.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000510b1ce01d01664-00650e0867-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.302 DEBUG:boto:path=/ffzdqu-57/d 2023-09-22T21:34:31.302 DEBUG:boto:auth_path=/ffzdqu-57/d 2023-09-22T21:34:31.303 DEBUG:boto:Method: GET 2023-09-22T21:34:31.303 DEBUG:boto:Path: /ffzdqu-57/d 2023-09-22T21:34:31.303 DEBUG:boto:Data: 2023-09-22T21:34:31.303 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.303 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.303 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.303 DEBUG:boto:Params: {} 2023-09-22T21:34:31.303 DEBUG:boto:Token: None 2023-09-22T21:34:31.303 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/d 2023-09-22T21:34:31.304 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oxtD4xMkgqhE2g4otZp+UIKPk8Q= 2023-09-22T21:34:31.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oxtD4xMkgqhE2g4otZp+UIKPk8Q=', 'Content-Length': '0'} 2023-09-22T21:34:31.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1ac3678dbaa1b5a-00650e0867-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.307 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:31.307 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:31.307 DEBUG:boto:path=/ffzdqu-57/?prefix=d 2023-09-22T21:34:31.307 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=d 2023-09-22T21:34:31.307 DEBUG:boto:Method: GET 2023-09-22T21:34:31.307 DEBUG:boto:Path: /ffzdqu-57/?prefix=d 2023-09-22T21:34:31.307 DEBUG:boto:Data: 2023-09-22T21:34:31.307 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.308 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.308 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.308 DEBUG:boto:Params: {} 2023-09-22T21:34:31.308 DEBUG:boto:Token: None 2023-09-22T21:34:31.308 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/ 2023-09-22T21:34:31.308 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk= 2023-09-22T21:34:31.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk=', 'Content-Length': '0'} 2023-09-22T21:34:31.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000198abd70e10bf183-00650e0867-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.354 DEBUG:boto:b'ffzdqu-57d1000falsed2023-09-22T21:32:23.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:31.355 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:31.355 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:31.355 DEBUG:boto:path=/ffzdqu-57/?prefix=d 2023-09-22T21:34:31.355 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=d 2023-09-22T21:34:31.355 DEBUG:boto:Method: GET 2023-09-22T21:34:31.355 DEBUG:boto:Path: /ffzdqu-57/?prefix=d 2023-09-22T21:34:31.355 DEBUG:boto:Data: 2023-09-22T21:34:31.355 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.356 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.356 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.356 DEBUG:boto:Params: {} 2023-09-22T21:34:31.356 DEBUG:boto:Token: None 2023-09-22T21:34:31.356 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/ 2023-09-22T21:34:31.356 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk= 2023-09-22T21:34:31.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk=', 'Content-Length': '0'} 2023-09-22T21:34:31.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000773894176a2497d3-00650e0867-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.359 DEBUG:boto:b'ffzdqu-57d1000falsed2023-09-22T21:32:23.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:31.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-22T21:34:31.360 DEBUG:boto:path=/ffzdqu-57/e 2023-09-22T21:34:31.360 DEBUG:boto:auth_path=/ffzdqu-57/e 2023-09-22T21:34:31.360 DEBUG:boto:path=/ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.360 DEBUG:boto:auth_path=/ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.360 DEBUG:boto:Method: GET 2023-09-22T21:34:31.361 DEBUG:boto:Path: /ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.361 DEBUG:boto:Data: 2023-09-22T21:34:31.361 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.361 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.361 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.361 DEBUG:boto:Params: {} 2023-09-22T21:34:31.361 DEBUG:boto:Token: None 2023-09-22T21:34:31.361 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.361 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6Wis4ni5hTdmkjVqT984SEjxc4s= 2023-09-22T21:34:31.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6Wis4ni5hTdmkjVqT984SEjxc4s=', 'Content-Length': '0'} 2023-09-22T21:34:31.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2df1f7c419f48b0-00650e0867-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.365 DEBUG:boto:path=/ffzdqu-57/e 2023-09-22T21:34:31.365 DEBUG:boto:auth_path=/ffzdqu-57/e 2023-09-22T21:34:31.365 DEBUG:boto:path=/ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.365 DEBUG:boto:auth_path=/ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.365 DEBUG:boto:Method: GET 2023-09-22T21:34:31.365 DEBUG:boto:Path: /ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.365 DEBUG:boto:Data: 2023-09-22T21:34:31.365 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.365 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.365 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.366 DEBUG:boto:Params: {} 2023-09-22T21:34:31.366 DEBUG:boto:Token: None 2023-09-22T21:34:31.366 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.366 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6Wis4ni5hTdmkjVqT984SEjxc4s= 2023-09-22T21:34:31.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6Wis4ni5hTdmkjVqT984SEjxc4s=', 'Content-Length': '0'} 2023-09-22T21:34:31.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af5d63e2fb1ec9af-00650e0867-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.410 DEBUG:boto:path=/ffzdqu-57/e 2023-09-22T21:34:31.410 DEBUG:boto:auth_path=/ffzdqu-57/e 2023-09-22T21:34:31.411 DEBUG:boto:path=/ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.411 DEBUG:boto:auth_path=/ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.411 DEBUG:boto:Method: HEAD 2023-09-22T21:34:31.411 DEBUG:boto:Path: /ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.411 DEBUG:boto:Data: 2023-09-22T21:34:31.411 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.411 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.411 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.411 DEBUG:boto:Params: {} 2023-09-22T21:34:31.411 DEBUG:boto:Token: None 2023-09-22T21:34:31.412 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.412 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AU0YKBVrSj1r9aDZBMBf4984YA4= 2023-09-22T21:34:31.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AU0YKBVrSj1r9aDZBMBf4984YA4=', 'Content-Length': '0'} 2023-09-22T21:34:31.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf72e160408cab06-00650e0867-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.415 DEBUG:boto:path=/ffzdqu-57/e 2023-09-22T21:34:31.415 DEBUG:boto:auth_path=/ffzdqu-57/e 2023-09-22T21:34:31.415 DEBUG:boto:path=/ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.415 DEBUG:boto:auth_path=/ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.415 DEBUG:boto:Method: HEAD 2023-09-22T21:34:31.415 DEBUG:boto:Path: /ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.415 DEBUG:boto:Data: 2023-09-22T21:34:31.416 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.416 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.416 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.416 DEBUG:boto:Params: {} 2023-09-22T21:34:31.416 DEBUG:boto:Token: None 2023-09-22T21:34:31.416 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.416 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AU0YKBVrSj1r9aDZBMBf4984YA4= 2023-09-22T21:34:31.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AU0YKBVrSj1r9aDZBMBf4984YA4=', 'Content-Length': '0'} 2023-09-22T21:34:31.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4cd65555e3dd874-00650e0867-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-22T21:34:31.419 DEBUG:boto:path=/ffzdqu-57/e 2023-09-22T21:34:31.420 DEBUG:boto:auth_path=/ffzdqu-57/e 2023-09-22T21:34:31.420 DEBUG:boto:path=/ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.420 DEBUG:boto:auth_path=/ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.420 DEBUG:boto:Method: GET 2023-09-22T21:34:31.420 DEBUG:boto:Path: /ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.420 DEBUG:boto:Data: 2023-09-22T21:34:31.420 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.420 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.420 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.420 DEBUG:boto:Params: {} 2023-09-22T21:34:31.420 DEBUG:boto:Token: None 2023-09-22T21:34:31.421 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.421 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6Wis4ni5hTdmkjVqT984SEjxc4s= 2023-09-22T21:34:31.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6Wis4ni5hTdmkjVqT984SEjxc4s=', 'Content-Length': '0'} 2023-09-22T21:34:31.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056d1a94d623ef88d-00650e0867-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.466 DEBUG:boto:path=/ffzdqu-57/e 2023-09-22T21:34:31.466 DEBUG:boto:auth_path=/ffzdqu-57/e 2023-09-22T21:34:31.467 DEBUG:boto:path=/ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.467 DEBUG:boto:auth_path=/ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.467 DEBUG:boto:Method: GET 2023-09-22T21:34:31.467 DEBUG:boto:Path: /ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.467 DEBUG:boto:Data: 2023-09-22T21:34:31.467 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.467 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.467 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.467 DEBUG:boto:Params: {} 2023-09-22T21:34:31.468 DEBUG:boto:Token: None 2023-09-22T21:34:31.468 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/e?versionId=null 2023-09-22T21:34:31.468 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6Wis4ni5hTdmkjVqT984SEjxc4s= 2023-09-22T21:34:31.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6Wis4ni5hTdmkjVqT984SEjxc4s=', 'Content-Length': '0'} 2023-09-22T21:34:31.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fe8d5acfea5ff72-00650e0867-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.471 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:31.471 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:31.471 DEBUG:boto:path=/ffzdqu-57/?prefix=e 2023-09-22T21:34:31.471 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=e 2023-09-22T21:34:31.471 DEBUG:boto:Method: GET 2023-09-22T21:34:31.472 DEBUG:boto:Path: /ffzdqu-57/?prefix=e 2023-09-22T21:34:31.472 DEBUG:boto:Data: 2023-09-22T21:34:31.472 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.472 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.472 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.472 DEBUG:boto:Params: {} 2023-09-22T21:34:31.472 DEBUG:boto:Token: None 2023-09-22T21:34:31.472 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/ 2023-09-22T21:34:31.472 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk= 2023-09-22T21:34:31.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk=', 'Content-Length': '0'} 2023-09-22T21:34:31.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000822962dbcd8f3700-00650e0867-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.518 DEBUG:boto:b'ffzdqu-57e1000falsee2023-09-22T21:32:23.937Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:31.519 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:31.519 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:31.519 DEBUG:boto:path=/ffzdqu-57/?prefix=e 2023-09-22T21:34:31.519 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=e 2023-09-22T21:34:31.519 DEBUG:boto:Method: GET 2023-09-22T21:34:31.519 DEBUG:boto:Path: /ffzdqu-57/?prefix=e 2023-09-22T21:34:31.519 DEBUG:boto:Data: 2023-09-22T21:34:31.519 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.520 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.520 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.520 DEBUG:boto:Params: {} 2023-09-22T21:34:31.520 DEBUG:boto:Token: None 2023-09-22T21:34:31.520 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/ 2023-09-22T21:34:31.520 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk= 2023-09-22T21:34:31.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk=', 'Content-Length': '0'} 2023-09-22T21:34:31.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ec9ecab9a7b837c-00650e0867-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.524 DEBUG:boto:b'ffzdqu-57e1000falsee2023-09-22T21:32:23.937Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:31.524 DEBUG:boto:path=/ffzdqu-57/e 2023-09-22T21:34:31.524 DEBUG:boto:auth_path=/ffzdqu-57/e 2023-09-22T21:34:31.524 DEBUG:boto:Method: HEAD 2023-09-22T21:34:31.524 DEBUG:boto:Path: /ffzdqu-57/e 2023-09-22T21:34:31.524 DEBUG:boto:Data: 2023-09-22T21:34:31.525 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.525 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.525 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.525 DEBUG:boto:Params: {} 2023-09-22T21:34:31.525 DEBUG:boto:Token: None 2023-09-22T21:34:31.525 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/e 2023-09-22T21:34:31.525 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e9jAWpbDjUDCZBsWveyr+3FnkvI= 2023-09-22T21:34:31.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e9jAWpbDjUDCZBsWveyr+3FnkvI=', 'Content-Length': '0'} 2023-09-22T21:34:31.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008dc65e3d7d928ff-00650e0867-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.528 DEBUG:boto:path=/ffzdqu-57/e 2023-09-22T21:34:31.528 DEBUG:boto:auth_path=/ffzdqu-57/e 2023-09-22T21:34:31.528 DEBUG:boto:Method: HEAD 2023-09-22T21:34:31.529 DEBUG:boto:Path: /ffzdqu-57/e 2023-09-22T21:34:31.529 DEBUG:boto:Data: 2023-09-22T21:34:31.529 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.529 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.529 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.529 DEBUG:boto:Params: {} 2023-09-22T21:34:31.529 DEBUG:boto:Token: None 2023-09-22T21:34:31.529 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/e 2023-09-22T21:34:31.529 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e9jAWpbDjUDCZBsWveyr+3FnkvI= 2023-09-22T21:34:31.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e9jAWpbDjUDCZBsWveyr+3FnkvI=', 'Content-Length': '0'} 2023-09-22T21:34:31.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1ba38a0cad1d718-00650e0867-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-22T21:34:31.532 DEBUG:boto:path=/ffzdqu-57/e 2023-09-22T21:34:31.532 DEBUG:boto:auth_path=/ffzdqu-57/e 2023-09-22T21:34:31.533 DEBUG:boto:Method: GET 2023-09-22T21:34:31.533 DEBUG:boto:Path: /ffzdqu-57/e 2023-09-22T21:34:31.533 DEBUG:boto:Data: 2023-09-22T21:34:31.533 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.533 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.533 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.533 DEBUG:boto:Params: {} 2023-09-22T21:34:31.533 DEBUG:boto:Token: None 2023-09-22T21:34:31.533 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/e 2023-09-22T21:34:31.534 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vXv8xPk7ZaDXN6paAIO5hMXHvYs= 2023-09-22T21:34:31.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vXv8xPk7ZaDXN6paAIO5hMXHvYs=', 'Content-Length': '0'} 2023-09-22T21:34:31.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca9359f8f5a5cd41-00650e0867-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.578 DEBUG:boto:path=/ffzdqu-57/e 2023-09-22T21:34:31.578 DEBUG:boto:auth_path=/ffzdqu-57/e 2023-09-22T21:34:31.578 DEBUG:boto:Method: GET 2023-09-22T21:34:31.579 DEBUG:boto:Path: /ffzdqu-57/e 2023-09-22T21:34:31.579 DEBUG:boto:Data: 2023-09-22T21:34:31.579 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.579 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.579 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.579 DEBUG:boto:Params: {} 2023-09-22T21:34:31.579 DEBUG:boto:Token: None 2023-09-22T21:34:31.579 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/e 2023-09-22T21:34:31.579 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vXv8xPk7ZaDXN6paAIO5hMXHvYs= 2023-09-22T21:34:31.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vXv8xPk7ZaDXN6paAIO5hMXHvYs=', 'Content-Length': '0'} 2023-09-22T21:34:31.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ead855919655f395-00650e0867-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.582 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:31.583 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:31.583 DEBUG:boto:path=/ffzdqu-57/?prefix=e 2023-09-22T21:34:31.583 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=e 2023-09-22T21:34:31.583 DEBUG:boto:Method: GET 2023-09-22T21:34:31.583 DEBUG:boto:Path: /ffzdqu-57/?prefix=e 2023-09-22T21:34:31.583 DEBUG:boto:Data: 2023-09-22T21:34:31.583 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.583 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.583 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.584 DEBUG:boto:Params: {} 2023-09-22T21:34:31.584 DEBUG:boto:Token: None 2023-09-22T21:34:31.584 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/ 2023-09-22T21:34:31.584 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk= 2023-09-22T21:34:31.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk=', 'Content-Length': '0'} 2023-09-22T21:34:31.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051a547ec2d95e88f-00650e0867-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.630 DEBUG:boto:b'ffzdqu-57e1000falsee2023-09-22T21:32:23.937Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:31.631 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:31.631 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:31.631 DEBUG:boto:path=/ffzdqu-57/?prefix=e 2023-09-22T21:34:31.631 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=e 2023-09-22T21:34:31.631 DEBUG:boto:Method: GET 2023-09-22T21:34:31.631 DEBUG:boto:Path: /ffzdqu-57/?prefix=e 2023-09-22T21:34:31.631 DEBUG:boto:Data: 2023-09-22T21:34:31.631 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.632 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.632 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.632 DEBUG:boto:Params: {} 2023-09-22T21:34:31.632 DEBUG:boto:Token: None 2023-09-22T21:34:31.632 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/ 2023-09-22T21:34:31.632 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk= 2023-09-22T21:34:31.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk=', 'Content-Length': '0'} 2023-09-22T21:34:31.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b54a851209a57459-00650e0867-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.636 DEBUG:boto:b'ffzdqu-57e1000falsee2023-09-22T21:32:23.937Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:31.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-22T21:34:31.636 DEBUG:boto:path=/ffzdqu-57/f 2023-09-22T21:34:31.636 DEBUG:boto:auth_path=/ffzdqu-57/f 2023-09-22T21:34:31.636 DEBUG:boto:path=/ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.637 DEBUG:boto:auth_path=/ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.637 DEBUG:boto:Method: GET 2023-09-22T21:34:31.637 DEBUG:boto:Path: /ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.637 DEBUG:boto:Data: 2023-09-22T21:34:31.637 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.637 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.637 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.637 DEBUG:boto:Params: {} 2023-09-22T21:34:31.637 DEBUG:boto:Token: None 2023-09-22T21:34:31.637 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.638 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fiuHK2LTqJSVYuUWP2X+FIzquI0= 2023-09-22T21:34:31.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fiuHK2LTqJSVYuUWP2X+FIzquI0=', 'Content-Length': '0'} 2023-09-22T21:34:31.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012f5b6963776201a-00650e0867-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.641 DEBUG:boto:path=/ffzdqu-57/f 2023-09-22T21:34:31.641 DEBUG:boto:auth_path=/ffzdqu-57/f 2023-09-22T21:34:31.641 DEBUG:boto:path=/ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.641 DEBUG:boto:auth_path=/ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.641 DEBUG:boto:Method: GET 2023-09-22T21:34:31.641 DEBUG:boto:Path: /ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.641 DEBUG:boto:Data: 2023-09-22T21:34:31.642 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.642 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.642 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.642 DEBUG:boto:Params: {} 2023-09-22T21:34:31.642 DEBUG:boto:Token: None 2023-09-22T21:34:31.642 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.642 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fiuHK2LTqJSVYuUWP2X+FIzquI0= 2023-09-22T21:34:31.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fiuHK2LTqJSVYuUWP2X+FIzquI0=', 'Content-Length': '0'} 2023-09-22T21:34:31.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc57dbb0a6d486ab-00650e0867-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.686 DEBUG:boto:path=/ffzdqu-57/f 2023-09-22T21:34:31.687 DEBUG:boto:auth_path=/ffzdqu-57/f 2023-09-22T21:34:31.687 DEBUG:boto:path=/ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.687 DEBUG:boto:auth_path=/ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.687 DEBUG:boto:Method: HEAD 2023-09-22T21:34:31.687 DEBUG:boto:Path: /ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.687 DEBUG:boto:Data: 2023-09-22T21:34:31.687 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.687 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.687 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.688 DEBUG:boto:Params: {} 2023-09-22T21:34:31.688 DEBUG:boto:Token: None 2023-09-22T21:34:31.688 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.688 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bHYSjfS0+U4cbEpeL3urMZyXhQc= 2023-09-22T21:34:31.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bHYSjfS0+U4cbEpeL3urMZyXhQc=', 'Content-Length': '0'} 2023-09-22T21:34:31.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ef33610c6f8a4e6-00650e0867-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.691 DEBUG:boto:path=/ffzdqu-57/f 2023-09-22T21:34:31.691 DEBUG:boto:auth_path=/ffzdqu-57/f 2023-09-22T21:34:31.691 DEBUG:boto:path=/ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.692 DEBUG:boto:auth_path=/ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.692 DEBUG:boto:Method: HEAD 2023-09-22T21:34:31.692 DEBUG:boto:Path: /ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.692 DEBUG:boto:Data: 2023-09-22T21:34:31.692 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.692 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.692 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.692 DEBUG:boto:Params: {} 2023-09-22T21:34:31.692 DEBUG:boto:Token: None 2023-09-22T21:34:31.693 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.693 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bHYSjfS0+U4cbEpeL3urMZyXhQc= 2023-09-22T21:34:31.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bHYSjfS0+U4cbEpeL3urMZyXhQc=', 'Content-Length': '0'} 2023-09-22T21:34:31.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000939b05e67ed70b13-00650e0867-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-22T21:34:31.696 DEBUG:boto:path=/ffzdqu-57/f 2023-09-22T21:34:31.696 DEBUG:boto:auth_path=/ffzdqu-57/f 2023-09-22T21:34:31.696 DEBUG:boto:path=/ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.696 DEBUG:boto:auth_path=/ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.696 DEBUG:boto:Method: GET 2023-09-22T21:34:31.696 DEBUG:boto:Path: /ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.697 DEBUG:boto:Data: 2023-09-22T21:34:31.697 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.697 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.697 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.697 DEBUG:boto:Params: {} 2023-09-22T21:34:31.697 DEBUG:boto:Token: None 2023-09-22T21:34:31.697 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.697 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fiuHK2LTqJSVYuUWP2X+FIzquI0= 2023-09-22T21:34:31.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fiuHK2LTqJSVYuUWP2X+FIzquI0=', 'Content-Length': '0'} 2023-09-22T21:34:31.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edb13400e2d9513c-00650e0867-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.742 DEBUG:boto:path=/ffzdqu-57/f 2023-09-22T21:34:31.742 DEBUG:boto:auth_path=/ffzdqu-57/f 2023-09-22T21:34:31.743 DEBUG:boto:path=/ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.743 DEBUG:boto:auth_path=/ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.743 DEBUG:boto:Method: GET 2023-09-22T21:34:31.743 DEBUG:boto:Path: /ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.743 DEBUG:boto:Data: 2023-09-22T21:34:31.743 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.743 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.743 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.743 DEBUG:boto:Params: {} 2023-09-22T21:34:31.743 DEBUG:boto:Token: None 2023-09-22T21:34:31.744 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/f?versionId=null 2023-09-22T21:34:31.744 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fiuHK2LTqJSVYuUWP2X+FIzquI0= 2023-09-22T21:34:31.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fiuHK2LTqJSVYuUWP2X+FIzquI0=', 'Content-Length': '0'} 2023-09-22T21:34:31.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5c851b7cdbc32f6-00650e0867-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.747 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:31.747 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:31.747 DEBUG:boto:path=/ffzdqu-57/?prefix=f 2023-09-22T21:34:31.747 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=f 2023-09-22T21:34:31.747 DEBUG:boto:Method: GET 2023-09-22T21:34:31.748 DEBUG:boto:Path: /ffzdqu-57/?prefix=f 2023-09-22T21:34:31.748 DEBUG:boto:Data: 2023-09-22T21:34:31.748 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.748 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.748 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.748 DEBUG:boto:Params: {} 2023-09-22T21:34:31.748 DEBUG:boto:Token: None 2023-09-22T21:34:31.748 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/ 2023-09-22T21:34:31.748 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk= 2023-09-22T21:34:31.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk=', 'Content-Length': '0'} 2023-09-22T21:34:31.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000459d7897343b6ea6-00650e0867-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.794 DEBUG:boto:b'ffzdqu-57f1000falsef2023-09-22T21:32:23.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:31.795 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:31.795 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:31.795 DEBUG:boto:path=/ffzdqu-57/?prefix=f 2023-09-22T21:34:31.795 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=f 2023-09-22T21:34:31.795 DEBUG:boto:Method: GET 2023-09-22T21:34:31.795 DEBUG:boto:Path: /ffzdqu-57/?prefix=f 2023-09-22T21:34:31.795 DEBUG:boto:Data: 2023-09-22T21:34:31.795 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.796 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.796 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.796 DEBUG:boto:Params: {} 2023-09-22T21:34:31.796 DEBUG:boto:Token: None 2023-09-22T21:34:31.796 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/ 2023-09-22T21:34:31.796 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk= 2023-09-22T21:34:31.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk=', 'Content-Length': '0'} 2023-09-22T21:34:31.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006265f6901ea6a2cc-00650e0867-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.801 DEBUG:boto:b'ffzdqu-57f1000falsef2023-09-22T21:32:23.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:31.801 DEBUG:boto:path=/ffzdqu-57/f 2023-09-22T21:34:31.801 DEBUG:boto:auth_path=/ffzdqu-57/f 2023-09-22T21:34:31.801 DEBUG:boto:Method: HEAD 2023-09-22T21:34:31.802 DEBUG:boto:Path: /ffzdqu-57/f 2023-09-22T21:34:31.802 DEBUG:boto:Data: 2023-09-22T21:34:31.802 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.802 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.802 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.802 DEBUG:boto:Params: {} 2023-09-22T21:34:31.802 DEBUG:boto:Token: None 2023-09-22T21:34:31.802 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/f 2023-09-22T21:34:31.803 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/7J0rIqR6HRwySTNI/KSRYWpxMY= 2023-09-22T21:34:31.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/7J0rIqR6HRwySTNI/KSRYWpxMY=', 'Content-Length': '0'} 2023-09-22T21:34:31.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000999a776b71c4f0ac-00650e0867-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.805 DEBUG:boto:path=/ffzdqu-57/f 2023-09-22T21:34:31.805 DEBUG:boto:auth_path=/ffzdqu-57/f 2023-09-22T21:34:31.806 DEBUG:boto:Method: HEAD 2023-09-22T21:34:31.806 DEBUG:boto:Path: /ffzdqu-57/f 2023-09-22T21:34:31.806 DEBUG:boto:Data: 2023-09-22T21:34:31.806 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.806 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.806 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.806 DEBUG:boto:Params: {} 2023-09-22T21:34:31.806 DEBUG:boto:Token: None 2023-09-22T21:34:31.806 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/f 2023-09-22T21:34:31.807 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/7J0rIqR6HRwySTNI/KSRYWpxMY= 2023-09-22T21:34:31.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/7J0rIqR6HRwySTNI/KSRYWpxMY=', 'Content-Length': '0'} 2023-09-22T21:34:31.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ffca4636728ce7f-00650e0867-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-22T21:34:31.809 DEBUG:boto:path=/ffzdqu-57/f 2023-09-22T21:34:31.809 DEBUG:boto:auth_path=/ffzdqu-57/f 2023-09-22T21:34:31.810 DEBUG:boto:Method: GET 2023-09-22T21:34:31.810 DEBUG:boto:Path: /ffzdqu-57/f 2023-09-22T21:34:31.810 DEBUG:boto:Data: 2023-09-22T21:34:31.810 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.810 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.810 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.810 DEBUG:boto:Params: {} 2023-09-22T21:34:31.810 DEBUG:boto:Token: None 2023-09-22T21:34:31.811 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/f 2023-09-22T21:34:31.811 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KwfjLbe4phUBLAll8Bk20tYEliY= 2023-09-22T21:34:31.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KwfjLbe4phUBLAll8Bk20tYEliY=', 'Content-Length': '0'} 2023-09-22T21:34:31.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010c16365cdc44052-00650e0867-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.854 DEBUG:boto:path=/ffzdqu-57/f 2023-09-22T21:34:31.854 DEBUG:boto:auth_path=/ffzdqu-57/f 2023-09-22T21:34:31.855 DEBUG:boto:Method: GET 2023-09-22T21:34:31.855 DEBUG:boto:Path: /ffzdqu-57/f 2023-09-22T21:34:31.855 DEBUG:boto:Data: 2023-09-22T21:34:31.855 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.855 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.855 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.855 DEBUG:boto:Params: {} 2023-09-22T21:34:31.855 DEBUG:boto:Token: None 2023-09-22T21:34:31.855 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/f 2023-09-22T21:34:31.856 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KwfjLbe4phUBLAll8Bk20tYEliY= 2023-09-22T21:34:31.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KwfjLbe4phUBLAll8Bk20tYEliY=', 'Content-Length': '0'} 2023-09-22T21:34:31.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1c5689427975e70-00650e0867-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.859 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:31.859 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:31.859 DEBUG:boto:path=/ffzdqu-57/?prefix=f 2023-09-22T21:34:31.859 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=f 2023-09-22T21:34:31.859 DEBUG:boto:Method: GET 2023-09-22T21:34:31.859 DEBUG:boto:Path: /ffzdqu-57/?prefix=f 2023-09-22T21:34:31.860 DEBUG:boto:Data: 2023-09-22T21:34:31.860 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.860 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.860 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.860 DEBUG:boto:Params: {} 2023-09-22T21:34:31.860 DEBUG:boto:Token: None 2023-09-22T21:34:31.860 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/ 2023-09-22T21:34:31.860 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk= 2023-09-22T21:34:31.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk=', 'Content-Length': '0'} 2023-09-22T21:34:31.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a03ca84d5baabece-00650e0867-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.865 DEBUG:boto:b'ffzdqu-57f1000falsef2023-09-22T21:32:23.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:31.866 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:31.866 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:31.866 DEBUG:boto:path=/ffzdqu-57/?prefix=f 2023-09-22T21:34:31.866 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=f 2023-09-22T21:34:31.866 DEBUG:boto:Method: GET 2023-09-22T21:34:31.866 DEBUG:boto:Path: /ffzdqu-57/?prefix=f 2023-09-22T21:34:31.866 DEBUG:boto:Data: 2023-09-22T21:34:31.867 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.867 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.867 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.867 DEBUG:boto:Params: {} 2023-09-22T21:34:31.867 DEBUG:boto:Token: None 2023-09-22T21:34:31.867 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/ 2023-09-22T21:34:31.867 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk= 2023-09-22T21:34:31.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nxzgqhXeCxZabfeP6N7VA4lwLHk=', 'Content-Length': '0'} 2023-09-22T21:34:31.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8f648c4fe71a973-00650e0867-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.914 DEBUG:boto:b'ffzdqu-57f1000falsef2023-09-22T21:32:23.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:31.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-22T21:34:31.915 DEBUG:boto:path=/ffzdqu-57/g 2023-09-22T21:34:31.915 DEBUG:boto:auth_path=/ffzdqu-57/g 2023-09-22T21:34:31.915 DEBUG:boto:path=/ffzdqu-57/g?versionId=null 2023-09-22T21:34:31.915 DEBUG:boto:auth_path=/ffzdqu-57/g?versionId=null 2023-09-22T21:34:31.915 DEBUG:boto:Method: GET 2023-09-22T21:34:31.916 DEBUG:boto:Path: /ffzdqu-57/g?versionId=null 2023-09-22T21:34:31.916 DEBUG:boto:Data: 2023-09-22T21:34:31.916 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.916 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.916 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.916 DEBUG:boto:Params: {} 2023-09-22T21:34:31.916 DEBUG:boto:Token: None 2023-09-22T21:34:31.916 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/g?versionId=null 2023-09-22T21:34:31.917 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h7JP8VS5uP+y49AIVr6+55rw8Uk= 2023-09-22T21:34:31.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h7JP8VS5uP+y49AIVr6+55rw8Uk=', 'Content-Length': '0'} 2023-09-22T21:34:31.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021e27aa9303c95e7-00650e0867-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.962 DEBUG:boto:path=/ffzdqu-57/g 2023-09-22T21:34:31.963 DEBUG:boto:auth_path=/ffzdqu-57/g 2023-09-22T21:34:31.963 DEBUG:boto:path=/ffzdqu-57/g?versionId=null 2023-09-22T21:34:31.963 DEBUG:boto:auth_path=/ffzdqu-57/g?versionId=null 2023-09-22T21:34:31.963 DEBUG:boto:Method: GET 2023-09-22T21:34:31.963 DEBUG:boto:Path: /ffzdqu-57/g?versionId=null 2023-09-22T21:34:31.963 DEBUG:boto:Data: 2023-09-22T21:34:31.963 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.963 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.963 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.964 DEBUG:boto:Params: {} 2023-09-22T21:34:31.964 DEBUG:boto:Token: None 2023-09-22T21:34:31.964 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/g?versionId=null 2023-09-22T21:34:31.964 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h7JP8VS5uP+y49AIVr6+55rw8Uk= 2023-09-22T21:34:31.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h7JP8VS5uP+y49AIVr6+55rw8Uk=', 'Content-Length': '0'} 2023-09-22T21:34:31.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003346286db8f8b1b0-00650e0867-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.968 DEBUG:boto:path=/ffzdqu-57/g 2023-09-22T21:34:31.968 DEBUG:boto:auth_path=/ffzdqu-57/g 2023-09-22T21:34:31.968 DEBUG:boto:path=/ffzdqu-57/g?versionId=null 2023-09-22T21:34:31.968 DEBUG:boto:auth_path=/ffzdqu-57/g?versionId=null 2023-09-22T21:34:31.968 DEBUG:boto:Method: HEAD 2023-09-22T21:34:31.968 DEBUG:boto:Path: /ffzdqu-57/g?versionId=null 2023-09-22T21:34:31.968 DEBUG:boto:Data: 2023-09-22T21:34:31.968 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.968 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.969 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.969 DEBUG:boto:Params: {} 2023-09-22T21:34:31.969 DEBUG:boto:Token: None 2023-09-22T21:34:31.969 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/g?versionId=null 2023-09-22T21:34:31.969 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NmPUSC+5wId6dBiHkiaubDGnlP4= 2023-09-22T21:34:31.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NmPUSC+5wId6dBiHkiaubDGnlP4=', 'Content-Length': '0'} 2023-09-22T21:34:31.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8bee343357c58c4-00650e0867-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.972 DEBUG:boto:path=/ffzdqu-57/g 2023-09-22T21:34:31.973 DEBUG:boto:auth_path=/ffzdqu-57/g 2023-09-22T21:34:31.973 DEBUG:boto:path=/ffzdqu-57/g?versionId=null 2023-09-22T21:34:31.973 DEBUG:boto:auth_path=/ffzdqu-57/g?versionId=null 2023-09-22T21:34:31.973 DEBUG:boto:Method: HEAD 2023-09-22T21:34:31.973 DEBUG:boto:Path: /ffzdqu-57/g?versionId=null 2023-09-22T21:34:31.973 DEBUG:boto:Data: 2023-09-22T21:34:31.973 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.973 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:31.973 DEBUG:boto:Port: 8000 2023-09-22T21:34:31.973 DEBUG:boto:Params: {} 2023-09-22T21:34:31.974 DEBUG:boto:Token: None 2023-09-22T21:34:31.974 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/g?versionId=null 2023-09-22T21:34:31.974 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NmPUSC+5wId6dBiHkiaubDGnlP4= 2023-09-22T21:34:31.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NmPUSC+5wId6dBiHkiaubDGnlP4=', 'Content-Length': '0'} 2023-09-22T21:34:31.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013b05533ec9292e0-00650e0867-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:31.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-22T21:34:31.977 DEBUG:boto:path=/ffzdqu-57/g 2023-09-22T21:34:31.977 DEBUG:boto:auth_path=/ffzdqu-57/g 2023-09-22T21:34:31.977 DEBUG:boto:path=/ffzdqu-57/g?versionId=null 2023-09-22T21:34:31.977 DEBUG:boto:auth_path=/ffzdqu-57/g?versionId=null 2023-09-22T21:34:31.977 DEBUG:boto:Method: GET 2023-09-22T21:34:31.977 DEBUG:boto:Path: /ffzdqu-57/g?versionId=null 2023-09-22T21:34:31.977 DEBUG:boto:Data: 2023-09-22T21:34:31.977 DEBUG:boto:Headers: {} 2023-09-22T21:34:31.977 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:31.978 DEBUG:boto:Port: 8001 2023-09-22T21:34:31.978 DEBUG:boto:Params: {} 2023-09-22T21:34:31.978 DEBUG:boto:Token: None 2023-09-22T21:34:31.978 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:31 GMT /ffzdqu-57/g?versionId=null 2023-09-22T21:34:31.978 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h7JP8VS5uP+y49AIVr6+55rw8Uk= 2023-09-22T21:34:31.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h7JP8VS5uP+y49AIVr6+55rw8Uk=', 'Content-Length': '0'} 2023-09-22T21:34:31.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071e8b8c283d2acce-00650e0867-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.022 DEBUG:boto:path=/ffzdqu-57/g 2023-09-22T21:34:32.023 DEBUG:boto:auth_path=/ffzdqu-57/g 2023-09-22T21:34:32.023 DEBUG:boto:path=/ffzdqu-57/g?versionId=null 2023-09-22T21:34:32.023 DEBUG:boto:auth_path=/ffzdqu-57/g?versionId=null 2023-09-22T21:34:32.023 DEBUG:boto:Method: GET 2023-09-22T21:34:32.023 DEBUG:boto:Path: /ffzdqu-57/g?versionId=null 2023-09-22T21:34:32.023 DEBUG:boto:Data: 2023-09-22T21:34:32.023 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.023 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:32.023 DEBUG:boto:Port: 8000 2023-09-22T21:34:32.024 DEBUG:boto:Params: {} 2023-09-22T21:34:32.024 DEBUG:boto:Token: None 2023-09-22T21:34:32.024 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/g?versionId=null 2023-09-22T21:34:32.024 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Cy8WRxyB6BJ9iYsv0ykNPT0Cq4g= 2023-09-22T21:34:32.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Cy8WRxyB6BJ9iYsv0ykNPT0Cq4g=', 'Content-Length': '0'} 2023-09-22T21:34:32.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c222627eb65a8baf-00650e0868-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.027 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:32.027 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:32.028 DEBUG:boto:path=/ffzdqu-57/?prefix=g 2023-09-22T21:34:32.028 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=g 2023-09-22T21:34:32.028 DEBUG:boto:Method: GET 2023-09-22T21:34:32.028 DEBUG:boto:Path: /ffzdqu-57/?prefix=g 2023-09-22T21:34:32.028 DEBUG:boto:Data: 2023-09-22T21:34:32.028 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.028 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:32.028 DEBUG:boto:Port: 8001 2023-09-22T21:34:32.028 DEBUG:boto:Params: {} 2023-09-22T21:34:32.028 DEBUG:boto:Token: None 2023-09-22T21:34:32.029 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/ 2023-09-22T21:34:32.029 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m58F7QgUjd5bfWI9Fg41jE0e3NA= 2023-09-22T21:34:32.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m58F7QgUjd5bfWI9Fg41jE0e3NA=', 'Content-Length': '0'} 2023-09-22T21:34:32.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc33a3b993b8f420-00650e0868-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.074 DEBUG:boto:b'ffzdqu-57g1000falseg2023-09-22T21:32:23.960Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:32.075 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:32.075 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:32.075 DEBUG:boto:path=/ffzdqu-57/?prefix=g 2023-09-22T21:34:32.075 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=g 2023-09-22T21:34:32.075 DEBUG:boto:Method: GET 2023-09-22T21:34:32.075 DEBUG:boto:Path: /ffzdqu-57/?prefix=g 2023-09-22T21:34:32.075 DEBUG:boto:Data: 2023-09-22T21:34:32.076 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.076 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:32.076 DEBUG:boto:Port: 8000 2023-09-22T21:34:32.076 DEBUG:boto:Params: {} 2023-09-22T21:34:32.076 DEBUG:boto:Token: None 2023-09-22T21:34:32.076 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/ 2023-09-22T21:34:32.076 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m58F7QgUjd5bfWI9Fg41jE0e3NA= 2023-09-22T21:34:32.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m58F7QgUjd5bfWI9Fg41jE0e3NA=', 'Content-Length': '0'} 2023-09-22T21:34:32.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000718485a2a0b28527-00650e0868-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.079 DEBUG:boto:b'ffzdqu-57g1000falseg2023-09-22T21:32:23.960Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:32.080 DEBUG:boto:path=/ffzdqu-57/g 2023-09-22T21:34:32.080 DEBUG:boto:auth_path=/ffzdqu-57/g 2023-09-22T21:34:32.080 DEBUG:boto:Method: HEAD 2023-09-22T21:34:32.080 DEBUG:boto:Path: /ffzdqu-57/g 2023-09-22T21:34:32.080 DEBUG:boto:Data: 2023-09-22T21:34:32.080 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.081 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:32.081 DEBUG:boto:Port: 8001 2023-09-22T21:34:32.081 DEBUG:boto:Params: {} 2023-09-22T21:34:32.081 DEBUG:boto:Token: None 2023-09-22T21:34:32.081 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/g 2023-09-22T21:34:32.081 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RkSL102sCtbMwC4IF95QzdhHku4= 2023-09-22T21:34:32.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RkSL102sCtbMwC4IF95QzdhHku4=', 'Content-Length': '0'} 2023-09-22T21:34:32.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bea54fb7b39703ee-00650e0868-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.084 DEBUG:boto:path=/ffzdqu-57/g 2023-09-22T21:34:32.084 DEBUG:boto:auth_path=/ffzdqu-57/g 2023-09-22T21:34:32.085 DEBUG:boto:Method: HEAD 2023-09-22T21:34:32.085 DEBUG:boto:Path: /ffzdqu-57/g 2023-09-22T21:34:32.085 DEBUG:boto:Data: 2023-09-22T21:34:32.085 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.085 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:32.085 DEBUG:boto:Port: 8000 2023-09-22T21:34:32.085 DEBUG:boto:Params: {} 2023-09-22T21:34:32.085 DEBUG:boto:Token: None 2023-09-22T21:34:32.085 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/g 2023-09-22T21:34:32.086 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RkSL102sCtbMwC4IF95QzdhHku4= 2023-09-22T21:34:32.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RkSL102sCtbMwC4IF95QzdhHku4=', 'Content-Length': '0'} 2023-09-22T21:34:32.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000286e8d35c237debb-00650e0868-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-22T21:34:32.089 DEBUG:boto:path=/ffzdqu-57/g 2023-09-22T21:34:32.089 DEBUG:boto:auth_path=/ffzdqu-57/g 2023-09-22T21:34:32.089 DEBUG:boto:Method: GET 2023-09-22T21:34:32.089 DEBUG:boto:Path: /ffzdqu-57/g 2023-09-22T21:34:32.089 DEBUG:boto:Data: 2023-09-22T21:34:32.089 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.089 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:32.089 DEBUG:boto:Port: 8001 2023-09-22T21:34:32.090 DEBUG:boto:Params: {} 2023-09-22T21:34:32.090 DEBUG:boto:Token: None 2023-09-22T21:34:32.090 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/g 2023-09-22T21:34:32.090 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DRHkKpoPh6rIj+ifLhjYIjh9WGY= 2023-09-22T21:34:32.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DRHkKpoPh6rIj+ifLhjYIjh9WGY=', 'Content-Length': '0'} 2023-09-22T21:34:32.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b28a81027c10568c-00650e0868-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.138 DEBUG:boto:path=/ffzdqu-57/g 2023-09-22T21:34:32.139 DEBUG:boto:auth_path=/ffzdqu-57/g 2023-09-22T21:34:32.139 DEBUG:boto:Method: GET 2023-09-22T21:34:32.139 DEBUG:boto:Path: /ffzdqu-57/g 2023-09-22T21:34:32.139 DEBUG:boto:Data: 2023-09-22T21:34:32.139 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.139 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:32.139 DEBUG:boto:Port: 8000 2023-09-22T21:34:32.139 DEBUG:boto:Params: {} 2023-09-22T21:34:32.139 DEBUG:boto:Token: None 2023-09-22T21:34:32.140 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/g 2023-09-22T21:34:32.140 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DRHkKpoPh6rIj+ifLhjYIjh9WGY= 2023-09-22T21:34:32.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DRHkKpoPh6rIj+ifLhjYIjh9WGY=', 'Content-Length': '0'} 2023-09-22T21:34:32.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fd4d4813c4424a5-00650e0868-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.143 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:32.143 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:32.143 DEBUG:boto:path=/ffzdqu-57/?prefix=g 2023-09-22T21:34:32.143 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=g 2023-09-22T21:34:32.143 DEBUG:boto:Method: GET 2023-09-22T21:34:32.143 DEBUG:boto:Path: /ffzdqu-57/?prefix=g 2023-09-22T21:34:32.143 DEBUG:boto:Data: 2023-09-22T21:34:32.144 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.144 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:32.144 DEBUG:boto:Port: 8001 2023-09-22T21:34:32.144 DEBUG:boto:Params: {} 2023-09-22T21:34:32.144 DEBUG:boto:Token: None 2023-09-22T21:34:32.144 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/ 2023-09-22T21:34:32.144 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m58F7QgUjd5bfWI9Fg41jE0e3NA= 2023-09-22T21:34:32.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m58F7QgUjd5bfWI9Fg41jE0e3NA=', 'Content-Length': '0'} 2023-09-22T21:34:32.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da26c6c166ede8b0-00650e0868-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.190 DEBUG:boto:b'ffzdqu-57g1000falseg2023-09-22T21:32:23.960Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:32.191 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:32.191 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:32.191 DEBUG:boto:path=/ffzdqu-57/?prefix=g 2023-09-22T21:34:32.191 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=g 2023-09-22T21:34:32.191 DEBUG:boto:Method: GET 2023-09-22T21:34:32.191 DEBUG:boto:Path: /ffzdqu-57/?prefix=g 2023-09-22T21:34:32.191 DEBUG:boto:Data: 2023-09-22T21:34:32.192 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.192 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:32.192 DEBUG:boto:Port: 8000 2023-09-22T21:34:32.192 DEBUG:boto:Params: {} 2023-09-22T21:34:32.192 DEBUG:boto:Token: None 2023-09-22T21:34:32.192 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/ 2023-09-22T21:34:32.192 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m58F7QgUjd5bfWI9Fg41jE0e3NA= 2023-09-22T21:34:32.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m58F7QgUjd5bfWI9Fg41jE0e3NA=', 'Content-Length': '0'} 2023-09-22T21:34:32.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fdf2652c71ddeec-00650e0868-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.196 DEBUG:boto:b'ffzdqu-57g1000falseg2023-09-22T21:32:23.960Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:32.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-22T21:34:32.196 DEBUG:boto:path=/ffzdqu-57/h 2023-09-22T21:34:32.196 DEBUG:boto:auth_path=/ffzdqu-57/h 2023-09-22T21:34:32.197 DEBUG:boto:path=/ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.197 DEBUG:boto:auth_path=/ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.197 DEBUG:boto:Method: GET 2023-09-22T21:34:32.197 DEBUG:boto:Path: /ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.197 DEBUG:boto:Data: 2023-09-22T21:34:32.197 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.197 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:32.197 DEBUG:boto:Port: 8001 2023-09-22T21:34:32.197 DEBUG:boto:Params: {} 2023-09-22T21:34:32.197 DEBUG:boto:Token: None 2023-09-22T21:34:32.198 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.198 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qHCe3fnIZ6UiaInQ1RldjcOlod4= 2023-09-22T21:34:32.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qHCe3fnIZ6UiaInQ1RldjcOlod4=', 'Content-Length': '0'} 2023-09-22T21:34:32.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a20578722239da55-00650e0868-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.201 DEBUG:boto:path=/ffzdqu-57/h 2023-09-22T21:34:32.201 DEBUG:boto:auth_path=/ffzdqu-57/h 2023-09-22T21:34:32.201 DEBUG:boto:path=/ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.201 DEBUG:boto:auth_path=/ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.201 DEBUG:boto:Method: GET 2023-09-22T21:34:32.201 DEBUG:boto:Path: /ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.201 DEBUG:boto:Data: 2023-09-22T21:34:32.202 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.202 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:32.202 DEBUG:boto:Port: 8000 2023-09-22T21:34:32.202 DEBUG:boto:Params: {} 2023-09-22T21:34:32.202 DEBUG:boto:Token: None 2023-09-22T21:34:32.202 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.202 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qHCe3fnIZ6UiaInQ1RldjcOlod4= 2023-09-22T21:34:32.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qHCe3fnIZ6UiaInQ1RldjcOlod4=', 'Content-Length': '0'} 2023-09-22T21:34:32.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035287dae1605181e-00650e0868-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.250 DEBUG:boto:path=/ffzdqu-57/h 2023-09-22T21:34:32.250 DEBUG:boto:auth_path=/ffzdqu-57/h 2023-09-22T21:34:32.251 DEBUG:boto:path=/ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.251 DEBUG:boto:auth_path=/ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.251 DEBUG:boto:Method: HEAD 2023-09-22T21:34:32.251 DEBUG:boto:Path: /ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.251 DEBUG:boto:Data: 2023-09-22T21:34:32.251 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.251 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:32.251 DEBUG:boto:Port: 8001 2023-09-22T21:34:32.251 DEBUG:boto:Params: {} 2023-09-22T21:34:32.252 DEBUG:boto:Token: None 2023-09-22T21:34:32.252 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.252 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jjpbeEwkyQU0YMQ2ywcBrGeZplk= 2023-09-22T21:34:32.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jjpbeEwkyQU0YMQ2ywcBrGeZplk=', 'Content-Length': '0'} 2023-09-22T21:34:32.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6f2e6aa0e426393-00650e0868-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.255 DEBUG:boto:path=/ffzdqu-57/h 2023-09-22T21:34:32.255 DEBUG:boto:auth_path=/ffzdqu-57/h 2023-09-22T21:34:32.255 DEBUG:boto:path=/ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.255 DEBUG:boto:auth_path=/ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.255 DEBUG:boto:Method: HEAD 2023-09-22T21:34:32.255 DEBUG:boto:Path: /ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.255 DEBUG:boto:Data: 2023-09-22T21:34:32.256 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.256 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:32.256 DEBUG:boto:Port: 8000 2023-09-22T21:34:32.256 DEBUG:boto:Params: {} 2023-09-22T21:34:32.256 DEBUG:boto:Token: None 2023-09-22T21:34:32.256 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.256 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jjpbeEwkyQU0YMQ2ywcBrGeZplk= 2023-09-22T21:34:32.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jjpbeEwkyQU0YMQ2ywcBrGeZplk=', 'Content-Length': '0'} 2023-09-22T21:34:32.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0d5bcb45d5c60c9-00650e0868-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-22T21:34:32.259 DEBUG:boto:path=/ffzdqu-57/h 2023-09-22T21:34:32.259 DEBUG:boto:auth_path=/ffzdqu-57/h 2023-09-22T21:34:32.259 DEBUG:boto:path=/ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.260 DEBUG:boto:auth_path=/ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.260 DEBUG:boto:Method: GET 2023-09-22T21:34:32.260 DEBUG:boto:Path: /ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.260 DEBUG:boto:Data: 2023-09-22T21:34:32.260 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.260 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:32.260 DEBUG:boto:Port: 8001 2023-09-22T21:34:32.260 DEBUG:boto:Params: {} 2023-09-22T21:34:32.260 DEBUG:boto:Token: None 2023-09-22T21:34:32.260 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.261 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qHCe3fnIZ6UiaInQ1RldjcOlod4= 2023-09-22T21:34:32.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qHCe3fnIZ6UiaInQ1RldjcOlod4=', 'Content-Length': '0'} 2023-09-22T21:34:32.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098d4d48f964d4849-00650e0868-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.306 DEBUG:boto:path=/ffzdqu-57/h 2023-09-22T21:34:32.306 DEBUG:boto:auth_path=/ffzdqu-57/h 2023-09-22T21:34:32.307 DEBUG:boto:path=/ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.307 DEBUG:boto:auth_path=/ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.307 DEBUG:boto:Method: GET 2023-09-22T21:34:32.307 DEBUG:boto:Path: /ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.307 DEBUG:boto:Data: 2023-09-22T21:34:32.307 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.307 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:32.307 DEBUG:boto:Port: 8000 2023-09-22T21:34:32.307 DEBUG:boto:Params: {} 2023-09-22T21:34:32.308 DEBUG:boto:Token: None 2023-09-22T21:34:32.308 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/h?versionId=null 2023-09-22T21:34:32.308 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qHCe3fnIZ6UiaInQ1RldjcOlod4= 2023-09-22T21:34:32.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qHCe3fnIZ6UiaInQ1RldjcOlod4=', 'Content-Length': '0'} 2023-09-22T21:34:32.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009434119a7c1189b8-00650e0868-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.311 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:32.311 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:32.311 DEBUG:boto:path=/ffzdqu-57/?prefix=h 2023-09-22T21:34:32.311 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=h 2023-09-22T21:34:32.311 DEBUG:boto:Method: GET 2023-09-22T21:34:32.312 DEBUG:boto:Path: /ffzdqu-57/?prefix=h 2023-09-22T21:34:32.312 DEBUG:boto:Data: 2023-09-22T21:34:32.312 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.312 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:32.312 DEBUG:boto:Port: 8001 2023-09-22T21:34:32.312 DEBUG:boto:Params: {} 2023-09-22T21:34:32.312 DEBUG:boto:Token: None 2023-09-22T21:34:32.312 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/ 2023-09-22T21:34:32.312 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m58F7QgUjd5bfWI9Fg41jE0e3NA= 2023-09-22T21:34:32.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m58F7QgUjd5bfWI9Fg41jE0e3NA=', 'Content-Length': '0'} 2023-09-22T21:34:32.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004213598a006aca43-00650e0868-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.358 DEBUG:boto:b'ffzdqu-57h1000falseh2023-09-22T21:32:24.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:32.359 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:32.359 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:32.359 DEBUG:boto:path=/ffzdqu-57/?prefix=h 2023-09-22T21:34:32.359 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=h 2023-09-22T21:34:32.359 DEBUG:boto:Method: GET 2023-09-22T21:34:32.359 DEBUG:boto:Path: /ffzdqu-57/?prefix=h 2023-09-22T21:34:32.359 DEBUG:boto:Data: 2023-09-22T21:34:32.359 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.360 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:32.360 DEBUG:boto:Port: 8000 2023-09-22T21:34:32.360 DEBUG:boto:Params: {} 2023-09-22T21:34:32.360 DEBUG:boto:Token: None 2023-09-22T21:34:32.360 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/ 2023-09-22T21:34:32.360 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m58F7QgUjd5bfWI9Fg41jE0e3NA= 2023-09-22T21:34:32.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m58F7QgUjd5bfWI9Fg41jE0e3NA=', 'Content-Length': '0'} 2023-09-22T21:34:32.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf09d9d743550ff0-00650e0868-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.364 DEBUG:boto:b'ffzdqu-57h1000falseh2023-09-22T21:32:24.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:32.364 DEBUG:boto:path=/ffzdqu-57/h 2023-09-22T21:34:32.364 DEBUG:boto:auth_path=/ffzdqu-57/h 2023-09-22T21:34:32.364 DEBUG:boto:Method: HEAD 2023-09-22T21:34:32.364 DEBUG:boto:Path: /ffzdqu-57/h 2023-09-22T21:34:32.364 DEBUG:boto:Data: 2023-09-22T21:34:32.365 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.365 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:32.365 DEBUG:boto:Port: 8001 2023-09-22T21:34:32.365 DEBUG:boto:Params: {} 2023-09-22T21:34:32.365 DEBUG:boto:Token: None 2023-09-22T21:34:32.365 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/h 2023-09-22T21:34:32.365 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FFppu3JJsxTtRvWsEJfL6jljuNs= 2023-09-22T21:34:32.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FFppu3JJsxTtRvWsEJfL6jljuNs=', 'Content-Length': '0'} 2023-09-22T21:34:32.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000208ff4f013f44212-00650e0868-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.368 DEBUG:boto:path=/ffzdqu-57/h 2023-09-22T21:34:32.368 DEBUG:boto:auth_path=/ffzdqu-57/h 2023-09-22T21:34:32.369 DEBUG:boto:Method: HEAD 2023-09-22T21:34:32.369 DEBUG:boto:Path: /ffzdqu-57/h 2023-09-22T21:34:32.369 DEBUG:boto:Data: 2023-09-22T21:34:32.369 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.369 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:32.369 DEBUG:boto:Port: 8000 2023-09-22T21:34:32.369 DEBUG:boto:Params: {} 2023-09-22T21:34:32.369 DEBUG:boto:Token: None 2023-09-22T21:34:32.369 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/h 2023-09-22T21:34:32.370 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FFppu3JJsxTtRvWsEJfL6jljuNs= 2023-09-22T21:34:32.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FFppu3JJsxTtRvWsEJfL6jljuNs=', 'Content-Length': '0'} 2023-09-22T21:34:32.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7e9e9d93c4b63be-00650e0868-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-22T21:34:32.372 DEBUG:boto:path=/ffzdqu-57/h 2023-09-22T21:34:32.373 DEBUG:boto:auth_path=/ffzdqu-57/h 2023-09-22T21:34:32.373 DEBUG:boto:Method: GET 2023-09-22T21:34:32.373 DEBUG:boto:Path: /ffzdqu-57/h 2023-09-22T21:34:32.373 DEBUG:boto:Data: 2023-09-22T21:34:32.373 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.373 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:32.373 DEBUG:boto:Port: 8001 2023-09-22T21:34:32.373 DEBUG:boto:Params: {} 2023-09-22T21:34:32.373 DEBUG:boto:Token: None 2023-09-22T21:34:32.373 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/h 2023-09-22T21:34:32.374 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WMso4lWv6vuPPxv9eoDoICmbQ9Q= 2023-09-22T21:34:32.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WMso4lWv6vuPPxv9eoDoICmbQ9Q=', 'Content-Length': '0'} 2023-09-22T21:34:32.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-57:f132bdb9-20a7-4073-a932-8f99da20691e.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e42c83aebc472be-00650e0868-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.418 DEBUG:boto:path=/ffzdqu-57/h 2023-09-22T21:34:32.419 DEBUG:boto:auth_path=/ffzdqu-57/h 2023-09-22T21:34:32.419 DEBUG:boto:Method: GET 2023-09-22T21:34:32.419 DEBUG:boto:Path: /ffzdqu-57/h 2023-09-22T21:34:32.419 DEBUG:boto:Data: 2023-09-22T21:34:32.419 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.419 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:32.419 DEBUG:boto:Port: 8000 2023-09-22T21:34:32.419 DEBUG:boto:Params: {} 2023-09-22T21:34:32.419 DEBUG:boto:Token: None 2023-09-22T21:34:32.420 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/h 2023-09-22T21:34:32.420 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WMso4lWv6vuPPxv9eoDoICmbQ9Q= 2023-09-22T21:34:32.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WMso4lWv6vuPPxv9eoDoICmbQ9Q=', 'Content-Length': '0'} 2023-09-22T21:34:32.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:32:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000778c8e2c7b63a13b-00650e0868-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.423 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:32.423 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:32.423 DEBUG:boto:path=/ffzdqu-57/?prefix=h 2023-09-22T21:34:32.423 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=h 2023-09-22T21:34:32.423 DEBUG:boto:Method: GET 2023-09-22T21:34:32.423 DEBUG:boto:Path: /ffzdqu-57/?prefix=h 2023-09-22T21:34:32.424 DEBUG:boto:Data: 2023-09-22T21:34:32.424 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.424 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:32.424 DEBUG:boto:Port: 8001 2023-09-22T21:34:32.424 DEBUG:boto:Params: {} 2023-09-22T21:34:32.424 DEBUG:boto:Token: None 2023-09-22T21:34:32.424 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/ 2023-09-22T21:34:32.424 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m58F7QgUjd5bfWI9Fg41jE0e3NA= 2023-09-22T21:34:32.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m58F7QgUjd5bfWI9Fg41jE0e3NA=', 'Content-Length': '0'} 2023-09-22T21:34:32.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000982f17bb16c3f4b2-00650e0868-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.470 DEBUG:boto:b'ffzdqu-57h1000falseh2023-09-22T21:32:24.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:32.471 DEBUG:boto:path=/ffzdqu-57/ 2023-09-22T21:34:32.471 DEBUG:boto:auth_path=/ffzdqu-57/ 2023-09-22T21:34:32.471 DEBUG:boto:path=/ffzdqu-57/?prefix=h 2023-09-22T21:34:32.471 DEBUG:boto:auth_path=/ffzdqu-57/?prefix=h 2023-09-22T21:34:32.471 DEBUG:boto:Method: GET 2023-09-22T21:34:32.471 DEBUG:boto:Path: /ffzdqu-57/?prefix=h 2023-09-22T21:34:32.471 DEBUG:boto:Data: 2023-09-22T21:34:32.471 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.472 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:32.472 DEBUG:boto:Port: 8000 2023-09-22T21:34:32.472 DEBUG:boto:Params: {} 2023-09-22T21:34:32.472 DEBUG:boto:Token: None 2023-09-22T21:34:32.472 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-57/ 2023-09-22T21:34:32.472 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m58F7QgUjd5bfWI9Fg41jE0e3NA= 2023-09-22T21:34:32.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m58F7QgUjd5bfWI9Fg41jE0e3NA=', 'Content-Length': '0'} 2023-09-22T21:34:32.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039214dcb8e705d38-00650e0868-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.475 DEBUG:boto:b'ffzdqu-57h1000falseh2023-09-22T21:32:24.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:32.476 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-57 zones={a2, a1} 2023-09-22T21:34:32.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... ok 2023-09-22T21:34:32.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:34:32.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:34:32.477 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:34:32.519 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:34:32.519 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:34:32.561 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:34:32.581 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695418343.887390_457.1","last_update":"2023-09-22T21:32:23.887390Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:34:32.581 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695418343.887390_457.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:34:32.581 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:34:32.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:34:32.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:34:32.582 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:34:32.663 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:34:32.664 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:34:32.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:34:32.700+0000 7f48c69aca40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:34:32.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:34:32.700+0000 7f48c69aca40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:34:32.720 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695418343.887390_457.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:32:23.887390Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:34:32Z"} 2023-09-22T21:34:32.720 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695418343.887390_457.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:32:23.887390Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:34:32Z"} 2023-09-22T21:34:32.720 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695418343.887390_457.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:32:23.887390Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:34:32.720 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695418343.887390_457.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:34:32.721 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695418343.887390_457.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:34:32.721 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:34:32.721 DEBUG:boto:path=/ 2023-09-22T21:34:32.721 DEBUG:boto:auth_path=/ 2023-09-22T21:34:32.721 DEBUG:boto:Method: GET 2023-09-22T21:34:32.721 DEBUG:boto:Path: / 2023-09-22T21:34:32.721 DEBUG:boto:Data: 2023-09-22T21:34:32.721 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.721 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:32.722 DEBUG:boto:Port: 8000 2023-09-22T21:34:32.722 DEBUG:boto:Params: {} 2023-09-22T21:34:32.722 DEBUG:boto:Token: None 2023-09-22T21:34:32.722 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:32 GMT / 2023-09-22T21:34:32.722 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pHNjRzhhZ0tM04ek5MsDWN/EXgI= 2023-09-22T21:34:32.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pHNjRzhhZ0tM04ek5MsDWN/EXgI=', 'Content-Length': '0'} 2023-09-22T21:34:32.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000767c87a1f968dc9b-00650e0868-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.728 DEBUG:boto:path=/ 2023-09-22T21:34:32.728 DEBUG:boto:auth_path=/ 2023-09-22T21:34:32.728 DEBUG:boto:Method: GET 2023-09-22T21:34:32.728 DEBUG:boto:Path: / 2023-09-22T21:34:32.728 DEBUG:boto:Data: 2023-09-22T21:34:32.728 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.728 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:32.729 DEBUG:boto:Port: 8001 2023-09-22T21:34:32.729 DEBUG:boto:Params: {} 2023-09-22T21:34:32.729 DEBUG:boto:Token: None 2023-09-22T21:34:32.729 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:32 GMT / 2023-09-22T21:34:32.729 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pHNjRzhhZ0tM04ek5MsDWN/EXgI= 2023-09-22T21:34:32.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pHNjRzhhZ0tM04ek5MsDWN/EXgI=', 'Content-Length': '0'} 2023-09-22T21:34:32.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cd0772c213fae93-00650e0868-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ok 2023-09-22T21:34:32.733 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:32.733 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:32.733 DEBUG:boto:Method: PUT 2023-09-22T21:34:32.734 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:32.734 DEBUG:boto:Data: 2023-09-22T21:34:32.734 DEBUG:boto:Headers: {} 2023-09-22T21:34:32.734 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:32.734 DEBUG:boto:Port: 8000 2023-09-22T21:34:32.734 DEBUG:boto:Params: {} 2023-09-22T21:34:32.734 DEBUG:boto:Token: None 2023-09-22T21:34:32.734 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:34:32 GMT /ffzdqu-58/ 2023-09-22T21:34:32.735 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IU7kJtuDNGW9CpT/w+N7G4pq8BY= 2023-09-22T21:34:32.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:32 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IU7kJtuDNGW9CpT/w+N7G4pq8BY=', 'Content-Length': '0'} 2023-09-22T21:34:32.753 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c36d1167cd72575c-00650e0868-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:32.753 DEBUG:rgw_multi.tests:created bucket=ffzdqu-58 2023-09-22T21:34:32.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:34:32.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:34:32.754 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:34:32.797 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:34:32.797 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:34:32.834 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:34:32.854 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695418472.752550_466.1","last_update":"2023-09-22T21:34:32.752550Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:34:32.855 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695418472.752550_466.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:34:32.855 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:34:32.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:34:32.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:34:32.855 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:34:32.930 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:34:32.930 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:34:32.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:34:32.964+0000 7f1d081b1a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:34:32.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:34:32.964+0000 7f1d081b1a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:34:32.983 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695418472.752550_466.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:34:32.752550Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:34:32Z"} 2023-09-22T21:34:32.983 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695418472.752550_466.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:34:32.752550Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:34:32Z"} 2023-09-22T21:34:32.984 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695418472.752550_466.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:34:32.752550Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:34:32.984 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695418472.752550_466.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:34:32.984 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695418472.752550_466.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:34:32.984 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:34:32.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:34:32.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:34:32.984 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:34:33.060 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:34:33.060 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:34:33.112 INFO:teuthology.orchestra.run.smithi160.stdout:{"objs":["obj00000000000000000001","obj00000000000000000005","obj00000000000000000012","obj00000000000000000004","obj00000000000000000011","obj00000000000000000000","obj00000000000000000010","obj00000000000000000003","obj00000000000000000013","obj00000000000000000002","obj00000000000000000006"]} 2023-09-22T21:34:33.113 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:33.113 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:33.113 DEBUG:boto:Method: HEAD 2023-09-22T21:34:33.113 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:33.113 DEBUG:boto:Data: 2023-09-22T21:34:33.113 DEBUG:boto:Headers: {} 2023-09-22T21:34:33.114 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:33.114 DEBUG:boto:Port: 8000 2023-09-22T21:34:33.114 DEBUG:boto:Params: {} 2023-09-22T21:34:33.114 DEBUG:boto:Token: None 2023-09-22T21:34:33.114 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:33 GMT /ffzdqu-58/ 2023-09-22T21:34:33.114 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KFBQzGvGmQToUTcjGKfnSpO3iGQ= 2023-09-22T21:34:33.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:33 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KFBQzGvGmQToUTcjGKfnSpO3iGQ=', 'Content-Length': '0'} 2023-09-22T21:34:33.117 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8721dd4d643bb04-00650e0869-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:33.117 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:33.117 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:33.117 DEBUG:boto:Method: PUT 2023-09-22T21:34:33.117 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:33.117 DEBUG:boto:Data: 2023-09-22T21:34:33.117 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:33.118 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:33.118 DEBUG:boto:Port: 8000 2023-09-22T21:34:33.118 DEBUG:boto:Params: {} 2023-09-22T21:34:33.118 DEBUG:boto:Token: None 2023-09-22T21:34:33.118 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:33 GMT /ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:33.118 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AIyh7hvSe4vo5qEQ2Usu6VbJD1Q= 2023-09-22T21:34:33.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:33 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AIyh7hvSe4vo5qEQ2Usu6VbJD1Q='} 2023-09-22T21:34:33.125 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000090b189c12cd1f4bf-00650e0869-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:33.125 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:33.125 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:33.125 DEBUG:boto:Method: HEAD 2023-09-22T21:34:33.126 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:33.126 DEBUG:boto:Data: 2023-09-22T21:34:33.126 DEBUG:boto:Headers: {} 2023-09-22T21:34:33.126 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:33.126 DEBUG:boto:Port: 8000 2023-09-22T21:34:33.126 DEBUG:boto:Params: {} 2023-09-22T21:34:33.126 DEBUG:boto:Token: None 2023-09-22T21:34:33.126 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:33 GMT /ffzdqu-58/ 2023-09-22T21:34:33.126 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KFBQzGvGmQToUTcjGKfnSpO3iGQ= 2023-09-22T21:34:33.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:33 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KFBQzGvGmQToUTcjGKfnSpO3iGQ=', 'Content-Length': '0'} 2023-09-22T21:34:33.128 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000424973ac3f6dbe4e-00650e0869-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:33.129 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:33.129 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:33.129 DEBUG:boto:Method: PUT 2023-09-22T21:34:33.129 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:33.129 DEBUG:boto:Data: 2023-09-22T21:34:33.129 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:33.129 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:33.130 DEBUG:boto:Port: 8000 2023-09-22T21:34:33.130 DEBUG:boto:Params: {} 2023-09-22T21:34:33.130 DEBUG:boto:Token: None 2023-09-22T21:34:33.130 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:33 GMT /ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:33.130 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kiOs0m41jeEnfEXJFgvTmAqsDcc= 2023-09-22T21:34:33.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:33 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kiOs0m41jeEnfEXJFgvTmAqsDcc='} 2023-09-22T21:34:33.136 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed7ae50c88bf60d7-00650e0869-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:33.136 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:33.136 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:33.136 DEBUG:boto:Method: HEAD 2023-09-22T21:34:33.137 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:33.137 DEBUG:boto:Data: 2023-09-22T21:34:33.137 DEBUG:boto:Headers: {} 2023-09-22T21:34:33.137 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:33.137 DEBUG:boto:Port: 8000 2023-09-22T21:34:33.137 DEBUG:boto:Params: {} 2023-09-22T21:34:33.137 DEBUG:boto:Token: None 2023-09-22T21:34:33.137 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:33 GMT /ffzdqu-58/ 2023-09-22T21:34:33.137 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KFBQzGvGmQToUTcjGKfnSpO3iGQ= 2023-09-22T21:34:33.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:33 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KFBQzGvGmQToUTcjGKfnSpO3iGQ=', 'Content-Length': '0'} 2023-09-22T21:34:33.141 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000336524951cf58a15-00650e0869-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:33.141 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:33.141 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:33.141 DEBUG:boto:Method: PUT 2023-09-22T21:34:33.141 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:33.142 DEBUG:boto:Data: 2023-09-22T21:34:33.142 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:33.142 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:33.142 DEBUG:boto:Port: 8000 2023-09-22T21:34:33.142 DEBUG:boto:Params: {} 2023-09-22T21:34:33.142 DEBUG:boto:Token: None 2023-09-22T21:34:33.142 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:33 GMT /ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:33.142 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ACC3stcOEhDvLkMv+gFLzuTDnDo= 2023-09-22T21:34:33.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:33 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ACC3stcOEhDvLkMv+gFLzuTDnDo='} 2023-09-22T21:34:33.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000565e683846a06905-00650e0869-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:33.191 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:33.191 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:33.191 DEBUG:boto:Method: HEAD 2023-09-22T21:34:33.191 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:33.191 DEBUG:boto:Data: 2023-09-22T21:34:33.191 DEBUG:boto:Headers: {} 2023-09-22T21:34:33.191 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:33.192 DEBUG:boto:Port: 8000 2023-09-22T21:34:33.192 DEBUG:boto:Params: {} 2023-09-22T21:34:33.192 DEBUG:boto:Token: None 2023-09-22T21:34:33.192 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:33 GMT /ffzdqu-58/ 2023-09-22T21:34:33.192 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KFBQzGvGmQToUTcjGKfnSpO3iGQ= 2023-09-22T21:34:33.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:33 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KFBQzGvGmQToUTcjGKfnSpO3iGQ=', 'Content-Length': '0'} 2023-09-22T21:34:33.196 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000880b8e213d144fa0-00650e0869-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:33.196 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:33.196 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:33.196 DEBUG:boto:Method: PUT 2023-09-22T21:34:33.197 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:33.197 DEBUG:boto:Data: 2023-09-22T21:34:33.197 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:33.197 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:33.197 DEBUG:boto:Port: 8000 2023-09-22T21:34:33.197 DEBUG:boto:Params: {} 2023-09-22T21:34:33.197 DEBUG:boto:Token: None 2023-09-22T21:34:33.197 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:33 GMT /ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:33.197 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NHV7VpYmtfRFttNZijOyF6B7DHY= 2023-09-22T21:34:33.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:33 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NHV7VpYmtfRFttNZijOyF6B7DHY='} 2023-09-22T21:34:33.209 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d7535dc8e71a6475-00650e0869-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:33.210 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:33.210 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:33.210 DEBUG:boto:Method: HEAD 2023-09-22T21:34:33.210 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:33.210 DEBUG:boto:Data: 2023-09-22T21:34:33.210 DEBUG:boto:Headers: {} 2023-09-22T21:34:33.210 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:33.210 DEBUG:boto:Port: 8000 2023-09-22T21:34:33.210 DEBUG:boto:Params: {} 2023-09-22T21:34:33.211 DEBUG:boto:Token: None 2023-09-22T21:34:33.211 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:33 GMT /ffzdqu-58/ 2023-09-22T21:34:33.211 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KFBQzGvGmQToUTcjGKfnSpO3iGQ= 2023-09-22T21:34:33.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:33 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KFBQzGvGmQToUTcjGKfnSpO3iGQ=', 'Content-Length': '0'} 2023-09-22T21:34:33.214 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000918caddb8fa3f4c9-00650e0869-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:33.215 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:33.215 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:33.215 DEBUG:boto:Method: PUT 2023-09-22T21:34:33.215 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:33.215 DEBUG:boto:Data: 2023-09-22T21:34:33.215 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:33.215 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:33.215 DEBUG:boto:Port: 8000 2023-09-22T21:34:33.215 DEBUG:boto:Params: {} 2023-09-22T21:34:33.216 DEBUG:boto:Token: None 2023-09-22T21:34:33.216 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:33 GMT /ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:33.216 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xQ4CmzwFXWbN4km5gSoJIR7xXMQ= 2023-09-22T21:34:33.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:33 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xQ4CmzwFXWbN4km5gSoJIR7xXMQ='} 2023-09-22T21:34:33.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d42fafe13ccca5d2-00650e0869-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:33.259 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:33.259 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:33.259 DEBUG:boto:Method: HEAD 2023-09-22T21:34:33.259 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:33.259 DEBUG:boto:Data: 2023-09-22T21:34:33.259 DEBUG:boto:Headers: {} 2023-09-22T21:34:33.259 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:33.259 DEBUG:boto:Port: 8000 2023-09-22T21:34:33.259 DEBUG:boto:Params: {} 2023-09-22T21:34:33.260 DEBUG:boto:Token: None 2023-09-22T21:34:33.260 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:33 GMT /ffzdqu-58/ 2023-09-22T21:34:33.260 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KFBQzGvGmQToUTcjGKfnSpO3iGQ= 2023-09-22T21:34:33.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:33 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KFBQzGvGmQToUTcjGKfnSpO3iGQ=', 'Content-Length': '0'} 2023-09-22T21:34:33.264 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099e449954da72f1c-00650e0869-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:33.264 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:33.265 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:33.265 DEBUG:boto:Method: PUT 2023-09-22T21:34:33.265 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:33.265 DEBUG:boto:Data: 2023-09-22T21:34:33.265 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:33.265 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:33.265 DEBUG:boto:Port: 8000 2023-09-22T21:34:33.265 DEBUG:boto:Params: {} 2023-09-22T21:34:33.265 DEBUG:boto:Token: None 2023-09-22T21:34:33.266 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:33 GMT /ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:33.266 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y4c09owLpV8AEpZLRuNedvEo1Ks= 2023-09-22T21:34:33.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:33 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y4c09owLpV8AEpZLRuNedvEo1Ks='} 2023-09-22T21:34:33.278 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ae26e53a3ea441a-00650e0869-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:33.279 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:33.279 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:33.279 DEBUG:boto:Method: HEAD 2023-09-22T21:34:33.279 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:33.279 DEBUG:boto:Data: 2023-09-22T21:34:33.279 DEBUG:boto:Headers: {} 2023-09-22T21:34:33.279 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:33.279 DEBUG:boto:Port: 8000 2023-09-22T21:34:33.279 DEBUG:boto:Params: {} 2023-09-22T21:34:33.279 DEBUG:boto:Token: None 2023-09-22T21:34:33.280 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:33 GMT /ffzdqu-58/ 2023-09-22T21:34:33.280 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KFBQzGvGmQToUTcjGKfnSpO3iGQ= 2023-09-22T21:34:33.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:33 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KFBQzGvGmQToUTcjGKfnSpO3iGQ=', 'Content-Length': '0'} 2023-09-22T21:34:33.284 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064fad44b0b256e80-00650e0869-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:33.284 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:33.284 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:33.284 DEBUG:boto:Method: PUT 2023-09-22T21:34:33.284 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:33.284 DEBUG:boto:Data: 2023-09-22T21:34:33.284 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:33.285 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:33.285 DEBUG:boto:Port: 8000 2023-09-22T21:34:33.285 DEBUG:boto:Params: {} 2023-09-22T21:34:33.285 DEBUG:boto:Token: None 2023-09-22T21:34:33.285 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:33 GMT /ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:33.285 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XKs8+HgB9Lr17Ei1qzGA4MBRvxs= 2023-09-22T21:34:33.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:33 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XKs8+HgB9Lr17Ei1qzGA4MBRvxs='} 2023-09-22T21:34:33.330 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004ffebf877ba70c51-00650e0869-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:33.331 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:33.331 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:33.331 DEBUG:boto:Method: HEAD 2023-09-22T21:34:33.331 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:33.331 DEBUG:boto:Data: 2023-09-22T21:34:33.331 DEBUG:boto:Headers: {} 2023-09-22T21:34:33.331 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:33.331 DEBUG:boto:Port: 8000 2023-09-22T21:34:33.331 DEBUG:boto:Params: {} 2023-09-22T21:34:33.332 DEBUG:boto:Token: None 2023-09-22T21:34:33.332 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:33 GMT /ffzdqu-58/ 2023-09-22T21:34:33.332 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KFBQzGvGmQToUTcjGKfnSpO3iGQ= 2023-09-22T21:34:33.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:33 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KFBQzGvGmQToUTcjGKfnSpO3iGQ=', 'Content-Length': '0'} 2023-09-22T21:34:33.336 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047f18015e05b98c0-00650e0869-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:33.336 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:33.337 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:33.337 DEBUG:boto:Method: PUT 2023-09-22T21:34:33.337 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:33.337 DEBUG:boto:Data: 2023-09-22T21:34:33.337 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:33.337 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:33.337 DEBUG:boto:Port: 8000 2023-09-22T21:34:33.337 DEBUG:boto:Params: {} 2023-09-22T21:34:33.337 DEBUG:boto:Token: None 2023-09-22T21:34:33.338 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:33 GMT /ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:33.338 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BIr+CnTDQBUcHe5ry7fexp4mQX8= 2023-09-22T21:34:33.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:33 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BIr+CnTDQBUcHe5ry7fexp4mQX8='} 2023-09-22T21:34:33.350 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006e9473cdee2d9d37-00650e0869-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:33.350 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:33.351 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:33.351 DEBUG:boto:Method: HEAD 2023-09-22T21:34:33.351 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:33.351 DEBUG:boto:Data: 2023-09-22T21:34:33.351 DEBUG:boto:Headers: {} 2023-09-22T21:34:33.351 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:33.351 DEBUG:boto:Port: 8000 2023-09-22T21:34:33.351 DEBUG:boto:Params: {} 2023-09-22T21:34:33.351 DEBUG:boto:Token: None 2023-09-22T21:34:33.352 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:33 GMT /ffzdqu-58/ 2023-09-22T21:34:33.352 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KFBQzGvGmQToUTcjGKfnSpO3iGQ= 2023-09-22T21:34:33.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:33 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KFBQzGvGmQToUTcjGKfnSpO3iGQ=', 'Content-Length': '0'} 2023-09-22T21:34:33.356 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005277456d5f027556-00650e0869-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:33.356 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:33.356 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:33.356 DEBUG:boto:Method: PUT 2023-09-22T21:34:33.357 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:33.357 DEBUG:boto:Data: 2023-09-22T21:34:33.357 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:33.357 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:33.357 DEBUG:boto:Port: 8000 2023-09-22T21:34:33.357 DEBUG:boto:Params: {} 2023-09-22T21:34:33.357 DEBUG:boto:Token: None 2023-09-22T21:34:33.357 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:33 GMT /ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:33.357 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CNM63Ae0+xVc7H2c4O9p9ndfdBQ= 2023-09-22T21:34:33.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:33 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CNM63Ae0+xVc7H2c4O9p9ndfdBQ='} 2023-09-22T21:34:33.403 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b00e4494313f718-00650e0869-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:33.403 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:33.403 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:33.403 DEBUG:boto:Method: HEAD 2023-09-22T21:34:33.403 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:33.403 DEBUG:boto:Data: 2023-09-22T21:34:33.403 DEBUG:boto:Headers: {} 2023-09-22T21:34:33.404 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:33.404 DEBUG:boto:Port: 8000 2023-09-22T21:34:33.404 DEBUG:boto:Params: {} 2023-09-22T21:34:33.404 DEBUG:boto:Token: None 2023-09-22T21:34:33.404 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:33 GMT /ffzdqu-58/ 2023-09-22T21:34:33.404 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KFBQzGvGmQToUTcjGKfnSpO3iGQ= 2023-09-22T21:34:33.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:33 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KFBQzGvGmQToUTcjGKfnSpO3iGQ=', 'Content-Length': '0'} 2023-09-22T21:34:33.408 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007889568d9eff72ee-00650e0869-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:33.408 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:33.409 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:33.409 DEBUG:boto:Method: PUT 2023-09-22T21:34:33.409 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:33.409 DEBUG:boto:Data: 2023-09-22T21:34:33.409 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:33.409 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:33.409 DEBUG:boto:Port: 8000 2023-09-22T21:34:33.409 DEBUG:boto:Params: {} 2023-09-22T21:34:33.409 DEBUG:boto:Token: None 2023-09-22T21:34:33.410 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:33 GMT /ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:33.410 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TpzozLu94xde3g0SvK7bR1ZsqZg= 2023-09-22T21:34:33.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:33 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TpzozLu94xde3g0SvK7bR1ZsqZg='} 2023-09-22T21:34:33.423 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e301dbce1207ae88-00650e0869-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:33.423 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-58 2023-09-22T21:34:33.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:34:33.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:34:33.423 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:34:33.475 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:34:33.475 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:34:33.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:34:33.516+0000 7f2bd78dda40 1 waiting to reach incremental sync.. 2023-09-22T21:34:38.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:34:38.516+0000 7f2bd78dda40 1 bucket sync caught up with source: 2023-09-22T21:34:38.520 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.71.6, 00000000001.31.6, 00000000001.42.6, , 00000000001.30.6, 00000000001.21.6, 00000000001.43.6, 00000000001.41.6, 00000000001.28.6, 00000000001.45.6, 00000000001.53.6] 2023-09-22T21:34:38.520 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.71.6, 00000000001.31.6, 00000000001.42.6, , 00000000001.30.6, 00000000001.21.6, 00000000001.43.6, 00000000001.41.6, 00000000001.28.6, 00000000001.45.6, 00000000001.53.6] 2023-09-22T21:34:38.521 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:34:38.516+0000 7f2bd78dda40 0 bucket checkpoint complete 2023-09-22T21:34:38.547 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-58 2023-09-22T21:34:38.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:34:38.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:34:38.548 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:34:38.595 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:34:38.595 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:34:38.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:34:38.636+0000 7f6c172c8a40 1 waiting to reach incremental sync.. 2023-09-22T21:34:43.638 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:34:43.636+0000 7f6c172c8a40 1 waiting to reach incremental sync.. 2023-09-22T21:34:48.642 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:34:48.640+0000 7f6c172c8a40 1 bucket sync caught up with source: 2023-09-22T21:34:48.642 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.201.6, 00000000001.159.6, 00000000001.119.6, , 00000000001.246.6, 00000000001.138.6, 00000000001.225.6, 00000000001.223.6, 00000000001.203.6, 00000000001.182.6, 00000000001.345.6] 2023-09-22T21:34:48.642 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.201.6, 00000000001.159.6, 00000000001.119.6, , 00000000001.246.6, 00000000001.138.6, 00000000001.225.6, 00000000001.223.6, 00000000001.203.6, 00000000001.182.6, 00000000001.345.6] 2023-09-22T21:34:48.642 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:34:48.640+0000 7f6c172c8a40 0 bucket checkpoint complete 2023-09-22T21:34:48.674 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-58 zones={a2, a1} 2023-09-22T21:34:48.674 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:48.674 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:48.675 DEBUG:boto:Method: HEAD 2023-09-22T21:34:48.675 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:48.675 DEBUG:boto:Data: 2023-09-22T21:34:48.675 DEBUG:boto:Headers: {} 2023-09-22T21:34:48.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:48.675 DEBUG:boto:Port: 8001 2023-09-22T21:34:48.675 DEBUG:boto:Params: {} 2023-09-22T21:34:48.675 DEBUG:boto:Token: None 2023-09-22T21:34:48.676 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:48 GMT /ffzdqu-58/ 2023-09-22T21:34:48.676 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BN00pIk1mcdtrvJiazdRB0zHftA= 2023-09-22T21:34:48.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BN00pIk1mcdtrvJiazdRB0zHftA=', 'Content-Length': '0'} 2023-09-22T21:34:48.679 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010bd7d68cdf34e2c-00650e0878-1ada-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:48.679 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:48.679 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:48.679 DEBUG:boto:Method: HEAD 2023-09-22T21:34:48.679 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:48.680 DEBUG:boto:Data: 2023-09-22T21:34:48.680 DEBUG:boto:Headers: {} 2023-09-22T21:34:48.680 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:48.680 DEBUG:boto:Port: 8000 2023-09-22T21:34:48.680 DEBUG:boto:Params: {} 2023-09-22T21:34:48.680 DEBUG:boto:Token: None 2023-09-22T21:34:48.680 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:48 GMT /ffzdqu-58/ 2023-09-22T21:34:48.680 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BN00pIk1mcdtrvJiazdRB0zHftA= 2023-09-22T21:34:48.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BN00pIk1mcdtrvJiazdRB0zHftA=', 'Content-Length': '0'} 2023-09-22T21:34:48.683 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c1eb3bcd6da0a20-00650e0878-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:48.684 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:34:48.684 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:48.684 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:48.684 DEBUG:boto:path=/ffzdqu-58/?versions&max-keys=999 2023-09-22T21:34:48.684 DEBUG:boto:auth_path=/ffzdqu-58/?versions&max-keys=999 2023-09-22T21:34:48.684 DEBUG:boto:Method: GET 2023-09-22T21:34:48.684 DEBUG:boto:Path: /ffzdqu-58/?versions&max-keys=999 2023-09-22T21:34:48.685 DEBUG:boto:Data: 2023-09-22T21:34:48.685 DEBUG:boto:Headers: {} 2023-09-22T21:34:48.685 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:48.685 DEBUG:boto:Port: 8001 2023-09-22T21:34:48.685 DEBUG:boto:Params: {} 2023-09-22T21:34:48.685 DEBUG:boto:Token: None 2023-09-22T21:34:48.685 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:48 GMT /ffzdqu-58/?versions 2023-09-22T21:34:48.685 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D7gL+QPwB9cOgE/tyJns2EYSEEM= 2023-09-22T21:34:48.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D7gL+QPwB9cOgE/tyJns2EYSEEM=', 'Content-Length': '0'} 2023-09-22T21:34:48.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035c73d76000be091-00650e0878-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:48.689 DEBUG:boto:b'ffzdqu-58999falseobj00000000000000000000nulltrue2023-09-22T21:34:33.288Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-09-22T21:34:33.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-09-22T21:34:33.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-09-22T21:34:33.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-09-22T21:34:33.361Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-09-22T21:34:33.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-09-22T21:34:33.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-09-22T21:34:33.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-09-22T21:34:33.269Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-09-22T21:34:33.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:48.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-09-22T21:34:48.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-09-22T21:34:48.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-09-22T21:34:48.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-09-22T21:34:48.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-09-22T21:34:48.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-09-22T21:34:48.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-09-22T21:34:48.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-09-22T21:34:48.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-09-22T21:34:48.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-09-22T21:34:48.691 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:34:48.691 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:48.691 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:48.691 DEBUG:boto:path=/ffzdqu-58/?versions&max-keys=999 2023-09-22T21:34:48.692 DEBUG:boto:auth_path=/ffzdqu-58/?versions&max-keys=999 2023-09-22T21:34:48.692 DEBUG:boto:Method: GET 2023-09-22T21:34:48.692 DEBUG:boto:Path: /ffzdqu-58/?versions&max-keys=999 2023-09-22T21:34:48.692 DEBUG:boto:Data: 2023-09-22T21:34:48.692 DEBUG:boto:Headers: {} 2023-09-22T21:34:48.692 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:48.692 DEBUG:boto:Port: 8000 2023-09-22T21:34:48.692 DEBUG:boto:Params: {} 2023-09-22T21:34:48.692 DEBUG:boto:Token: None 2023-09-22T21:34:48.693 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:48 GMT /ffzdqu-58/?versions 2023-09-22T21:34:48.693 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D7gL+QPwB9cOgE/tyJns2EYSEEM= 2023-09-22T21:34:48.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D7gL+QPwB9cOgE/tyJns2EYSEEM=', 'Content-Length': '0'} 2023-09-22T21:34:48.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fe0d36a232db32b-00650e0878-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:48.697 DEBUG:boto:b'ffzdqu-58999falseobj00000000000000000000nulltrue2023-09-22T21:34:33.288Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-09-22T21:34:33.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-09-22T21:34:33.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-09-22T21:34:33.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-09-22T21:34:33.361Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-09-22T21:34:33.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-09-22T21:34:33.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-09-22T21:34:33.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-09-22T21:34:33.269Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-09-22T21:34:33.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:48.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-09-22T21:34:48.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-09-22T21:34:48.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-09-22T21:34:48.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-09-22T21:34:48.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-09-22T21:34:48.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-09-22T21:34:48.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-09-22T21:34:48.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-09-22T21:34:48.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-09-22T21:34:48.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-09-22T21:34:48.699 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:48.699 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:48.699 DEBUG:boto:path=/ffzdqu-58/?versions&max-keys=999 2023-09-22T21:34:48.699 DEBUG:boto:auth_path=/ffzdqu-58/?versions&max-keys=999 2023-09-22T21:34:48.700 DEBUG:boto:Method: GET 2023-09-22T21:34:48.700 DEBUG:boto:Path: /ffzdqu-58/?versions&max-keys=999 2023-09-22T21:34:48.700 DEBUG:boto:Data: 2023-09-22T21:34:48.700 DEBUG:boto:Headers: {} 2023-09-22T21:34:48.700 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:48.700 DEBUG:boto:Port: 8001 2023-09-22T21:34:48.700 DEBUG:boto:Params: {} 2023-09-22T21:34:48.700 DEBUG:boto:Token: None 2023-09-22T21:34:48.700 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:48 GMT /ffzdqu-58/?versions 2023-09-22T21:34:48.701 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D7gL+QPwB9cOgE/tyJns2EYSEEM= 2023-09-22T21:34:48.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D7gL+QPwB9cOgE/tyJns2EYSEEM=', 'Content-Length': '0'} 2023-09-22T21:34:48.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5d6e2e2063a0309-00650e0878-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:48.705 DEBUG:boto:b'ffzdqu-58999falseobj00000000000000000000nulltrue2023-09-22T21:34:33.288Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-09-22T21:34:33.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-09-22T21:34:33.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-09-22T21:34:33.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-09-22T21:34:33.361Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-09-22T21:34:33.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-09-22T21:34:33.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-09-22T21:34:33.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-09-22T21:34:33.269Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-09-22T21:34:33.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:48.705 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:48.706 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:48.706 DEBUG:boto:path=/ffzdqu-58/?versions&max-keys=999 2023-09-22T21:34:48.706 DEBUG:boto:auth_path=/ffzdqu-58/?versions&max-keys=999 2023-09-22T21:34:48.706 DEBUG:boto:Method: GET 2023-09-22T21:34:48.706 DEBUG:boto:Path: /ffzdqu-58/?versions&max-keys=999 2023-09-22T21:34:48.706 DEBUG:boto:Data: 2023-09-22T21:34:48.706 DEBUG:boto:Headers: {} 2023-09-22T21:34:48.706 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:48.706 DEBUG:boto:Port: 8000 2023-09-22T21:34:48.707 DEBUG:boto:Params: {} 2023-09-22T21:34:48.707 DEBUG:boto:Token: None 2023-09-22T21:34:48.707 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:48 GMT /ffzdqu-58/?versions 2023-09-22T21:34:48.707 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D7gL+QPwB9cOgE/tyJns2EYSEEM= 2023-09-22T21:34:48.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D7gL+QPwB9cOgE/tyJns2EYSEEM=', 'Content-Length': '0'} 2023-09-22T21:34:48.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004580f08b925e38b6-00650e0878-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:48.711 DEBUG:boto:b'ffzdqu-58999falseobj00000000000000000000nulltrue2023-09-22T21:34:33.288Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-09-22T21:34:33.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-09-22T21:34:33.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-09-22T21:34:33.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-09-22T21:34:33.361Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-09-22T21:34:33.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-09-22T21:34:33.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-09-22T21:34:33.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-09-22T21:34:33.269Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-09-22T21:34:33.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:48.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-09-22T21:34:48.712 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.712 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.712 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.712 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.712 DEBUG:boto:Method: GET 2023-09-22T21:34:48.712 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.712 DEBUG:boto:Data: 2023-09-22T21:34:48.712 DEBUG:boto:Headers: {} 2023-09-22T21:34:48.712 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:48.713 DEBUG:boto:Port: 8001 2023-09-22T21:34:48.713 DEBUG:boto:Params: {} 2023-09-22T21:34:48.713 DEBUG:boto:Token: None 2023-09-22T21:34:48.713 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:48 GMT /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.713 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kx6jWVSOKXRqXpjs5hH2uSGmLVU= 2023-09-22T21:34:48.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kx6jWVSOKXRqXpjs5hH2uSGmLVU=', 'Content-Length': '0'} 2023-09-22T21:34:48.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f58fc8dd0f5cf4e6-00650e0878-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:48.762 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.763 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.763 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.763 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.763 DEBUG:boto:Method: GET 2023-09-22T21:34:48.763 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.763 DEBUG:boto:Data: 2023-09-22T21:34:48.763 DEBUG:boto:Headers: {} 2023-09-22T21:34:48.763 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:48.763 DEBUG:boto:Port: 8000 2023-09-22T21:34:48.763 DEBUG:boto:Params: {} 2023-09-22T21:34:48.764 DEBUG:boto:Token: None 2023-09-22T21:34:48.764 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:48 GMT /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.764 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kx6jWVSOKXRqXpjs5hH2uSGmLVU= 2023-09-22T21:34:48.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kx6jWVSOKXRqXpjs5hH2uSGmLVU=', 'Content-Length': '0'} 2023-09-22T21:34:48.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cec4c26a7460588-00650e0878-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:48.767 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.767 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.767 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.767 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.767 DEBUG:boto:Method: HEAD 2023-09-22T21:34:48.767 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.767 DEBUG:boto:Data: 2023-09-22T21:34:48.767 DEBUG:boto:Headers: {} 2023-09-22T21:34:48.768 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:48.768 DEBUG:boto:Port: 8001 2023-09-22T21:34:48.768 DEBUG:boto:Params: {} 2023-09-22T21:34:48.768 DEBUG:boto:Token: None 2023-09-22T21:34:48.768 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:48 GMT /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.768 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0fd/3RxwOd+CG1lfzlGpN0nZ2o8= 2023-09-22T21:34:48.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0fd/3RxwOd+CG1lfzlGpN0nZ2o8=', 'Content-Length': '0'} 2023-09-22T21:34:48.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000431db5981a849160-00650e0878-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:48.771 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.771 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.772 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.772 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.772 DEBUG:boto:Method: HEAD 2023-09-22T21:34:48.772 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.772 DEBUG:boto:Data: 2023-09-22T21:34:48.772 DEBUG:boto:Headers: {} 2023-09-22T21:34:48.772 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:48.772 DEBUG:boto:Port: 8000 2023-09-22T21:34:48.772 DEBUG:boto:Params: {} 2023-09-22T21:34:48.773 DEBUG:boto:Token: None 2023-09-22T21:34:48.773 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:48 GMT /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.773 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0fd/3RxwOd+CG1lfzlGpN0nZ2o8= 2023-09-22T21:34:48.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0fd/3RxwOd+CG1lfzlGpN0nZ2o8=', 'Content-Length': '0'} 2023-09-22T21:34:48.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a8528002df7e9a0-00650e0878-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:48.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-09-22T21:34:48.776 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.776 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.776 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.776 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.776 DEBUG:boto:Method: GET 2023-09-22T21:34:48.776 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.776 DEBUG:boto:Data: 2023-09-22T21:34:48.776 DEBUG:boto:Headers: {} 2023-09-22T21:34:48.777 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:48.777 DEBUG:boto:Port: 8001 2023-09-22T21:34:48.777 DEBUG:boto:Params: {} 2023-09-22T21:34:48.777 DEBUG:boto:Token: None 2023-09-22T21:34:48.777 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:48 GMT /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.777 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kx6jWVSOKXRqXpjs5hH2uSGmLVU= 2023-09-22T21:34:48.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kx6jWVSOKXRqXpjs5hH2uSGmLVU=', 'Content-Length': '0'} 2023-09-22T21:34:48.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008aceb812e7f5486a-00650e0878-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:48.822 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.822 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.823 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.823 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.823 DEBUG:boto:Method: GET 2023-09-22T21:34:48.823 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.823 DEBUG:boto:Data: 2023-09-22T21:34:48.823 DEBUG:boto:Headers: {} 2023-09-22T21:34:48.823 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:48.823 DEBUG:boto:Port: 8000 2023-09-22T21:34:48.823 DEBUG:boto:Params: {} 2023-09-22T21:34:48.824 DEBUG:boto:Token: None 2023-09-22T21:34:48.824 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:48 GMT /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:34:48.824 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kx6jWVSOKXRqXpjs5hH2uSGmLVU= 2023-09-22T21:34:48.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kx6jWVSOKXRqXpjs5hH2uSGmLVU=', 'Content-Length': '0'} 2023-09-22T21:34:48.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000355b55362901fda2-00650e0878-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:48.827 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:48.827 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:48.827 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:34:48.827 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:34:48.828 DEBUG:boto:Method: GET 2023-09-22T21:34:48.828 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:34:48.828 DEBUG:boto:Data: 2023-09-22T21:34:48.828 DEBUG:boto:Headers: {} 2023-09-22T21:34:48.828 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:48.828 DEBUG:boto:Port: 8001 2023-09-22T21:34:48.828 DEBUG:boto:Params: {} 2023-09-22T21:34:48.828 DEBUG:boto:Token: None 2023-09-22T21:34:48.828 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:48 GMT /ffzdqu-58/ 2023-09-22T21:34:48.829 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kznLgGbz5ljljq/K/3pwVRgADEI= 2023-09-22T21:34:48.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kznLgGbz5ljljq/K/3pwVRgADEI=', 'Content-Length': '0'} 2023-09-22T21:34:48.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bcc09f97217f9f8-00650e0878-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:48.874 DEBUG:boto:b'ffzdqu-58obj000000000000000000001000falseobj000000000000000000002023-09-22T21:34:33.288Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:48.875 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:48.875 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:48.875 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:34:48.875 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:34:48.875 DEBUG:boto:Method: GET 2023-09-22T21:34:48.875 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:34:48.876 DEBUG:boto:Data: 2023-09-22T21:34:48.876 DEBUG:boto:Headers: {} 2023-09-22T21:34:48.876 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:48.876 DEBUG:boto:Port: 8000 2023-09-22T21:34:48.876 DEBUG:boto:Params: {} 2023-09-22T21:34:48.876 DEBUG:boto:Token: None 2023-09-22T21:34:48.876 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:48 GMT /ffzdqu-58/ 2023-09-22T21:34:48.876 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kznLgGbz5ljljq/K/3pwVRgADEI= 2023-09-22T21:34:48.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kznLgGbz5ljljq/K/3pwVRgADEI=', 'Content-Length': '0'} 2023-09-22T21:34:48.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef1db7b4539c3f15-00650e0878-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:48.882 DEBUG:boto:b'ffzdqu-58obj000000000000000000001000falseobj000000000000000000002023-09-22T21:34:33.288Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:48.883 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.883 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.883 DEBUG:boto:Method: HEAD 2023-09-22T21:34:48.883 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.883 DEBUG:boto:Data: 2023-09-22T21:34:48.883 DEBUG:boto:Headers: {} 2023-09-22T21:34:48.884 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:48.884 DEBUG:boto:Port: 8001 2023-09-22T21:34:48.884 DEBUG:boto:Params: {} 2023-09-22T21:34:48.884 DEBUG:boto:Token: None 2023-09-22T21:34:48.884 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:48 GMT /ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.884 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wkf6ajKL19xU4gldGMz2XvR5ERE= 2023-09-22T21:34:48.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wkf6ajKL19xU4gldGMz2XvR5ERE=', 'Content-Length': '0'} 2023-09-22T21:34:48.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008458720203ea4ddf-00650e0878-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:48.887 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.887 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.887 DEBUG:boto:Method: HEAD 2023-09-22T21:34:48.888 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.888 DEBUG:boto:Data: 2023-09-22T21:34:48.888 DEBUG:boto:Headers: {} 2023-09-22T21:34:48.888 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:48.888 DEBUG:boto:Port: 8000 2023-09-22T21:34:48.888 DEBUG:boto:Params: {} 2023-09-22T21:34:48.888 DEBUG:boto:Token: None 2023-09-22T21:34:48.888 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:48 GMT /ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.888 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wkf6ajKL19xU4gldGMz2XvR5ERE= 2023-09-22T21:34:48.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wkf6ajKL19xU4gldGMz2XvR5ERE=', 'Content-Length': '0'} 2023-09-22T21:34:48.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041aa2cc3901a44f6-00650e0878-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:48.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-09-22T21:34:48.891 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.892 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.892 DEBUG:boto:Method: GET 2023-09-22T21:34:48.892 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.892 DEBUG:boto:Data: 2023-09-22T21:34:48.892 DEBUG:boto:Headers: {} 2023-09-22T21:34:48.892 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:48.892 DEBUG:boto:Port: 8001 2023-09-22T21:34:48.892 DEBUG:boto:Params: {} 2023-09-22T21:34:48.892 DEBUG:boto:Token: None 2023-09-22T21:34:48.893 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:48 GMT /ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.893 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VROtNpkTO85H/MfQ9Ntrjfg2/8I= 2023-09-22T21:34:48.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VROtNpkTO85H/MfQ9Ntrjfg2/8I=', 'Content-Length': '0'} 2023-09-22T21:34:48.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac84ada1c3369354-00650e0878-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:48.938 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.939 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.939 DEBUG:boto:Method: GET 2023-09-22T21:34:48.939 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.939 DEBUG:boto:Data: 2023-09-22T21:34:48.939 DEBUG:boto:Headers: {} 2023-09-22T21:34:48.939 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:48.939 DEBUG:boto:Port: 8000 2023-09-22T21:34:48.939 DEBUG:boto:Params: {} 2023-09-22T21:34:48.939 DEBUG:boto:Token: None 2023-09-22T21:34:48.940 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:48 GMT /ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:48.940 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VROtNpkTO85H/MfQ9Ntrjfg2/8I= 2023-09-22T21:34:48.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VROtNpkTO85H/MfQ9Ntrjfg2/8I=', 'Content-Length': '0'} 2023-09-22T21:34:48.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce345fc9e82b7264-00650e0878-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:48.943 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:48.943 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:48.943 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:34:48.943 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:34:48.943 DEBUG:boto:Method: GET 2023-09-22T21:34:48.944 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:34:48.944 DEBUG:boto:Data: 2023-09-22T21:34:48.944 DEBUG:boto:Headers: {} 2023-09-22T21:34:48.944 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:48.944 DEBUG:boto:Port: 8001 2023-09-22T21:34:48.944 DEBUG:boto:Params: {} 2023-09-22T21:34:48.944 DEBUG:boto:Token: None 2023-09-22T21:34:48.944 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:48 GMT /ffzdqu-58/ 2023-09-22T21:34:48.945 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kznLgGbz5ljljq/K/3pwVRgADEI= 2023-09-22T21:34:48.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kznLgGbz5ljljq/K/3pwVRgADEI=', 'Content-Length': '0'} 2023-09-22T21:34:48.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f17d82a9082809b1-00650e0878-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:48.990 DEBUG:boto:b'ffzdqu-58obj000000000000000000001000falseobj000000000000000000002023-09-22T21:34:33.288Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:48.991 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:48.991 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:48.991 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:34:48.991 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:34:48.991 DEBUG:boto:Method: GET 2023-09-22T21:34:48.991 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:34:48.991 DEBUG:boto:Data: 2023-09-22T21:34:48.992 DEBUG:boto:Headers: {} 2023-09-22T21:34:48.992 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:48.992 DEBUG:boto:Port: 8000 2023-09-22T21:34:48.992 DEBUG:boto:Params: {} 2023-09-22T21:34:48.992 DEBUG:boto:Token: None 2023-09-22T21:34:48.992 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:48 GMT /ffzdqu-58/ 2023-09-22T21:34:48.992 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kznLgGbz5ljljq/K/3pwVRgADEI= 2023-09-22T21:34:48.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kznLgGbz5ljljq/K/3pwVRgADEI=', 'Content-Length': '0'} 2023-09-22T21:34:48.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064e0a70f1e6ea70d-00650e0878-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:48.998 DEBUG:boto:b'ffzdqu-58obj000000000000000000001000falseobj000000000000000000002023-09-22T21:34:33.288Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:48.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-09-22T21:34:48.999 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:48.999 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:48.999 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:48.999 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:48.999 DEBUG:boto:Method: GET 2023-09-22T21:34:48.999 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.000 DEBUG:boto:Data: 2023-09-22T21:34:49.000 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.000 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.000 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.000 DEBUG:boto:Params: {} 2023-09-22T21:34:49.000 DEBUG:boto:Token: None 2023-09-22T21:34:49.000 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.000 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5C/H+k1MyAyoc/CmydQgIlv1HDY= 2023-09-22T21:34:49.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5C/H+k1MyAyoc/CmydQgIlv1HDY=', 'Content-Length': '0'} 2023-09-22T21:34:49.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006966930b74e7182e-00650e0879-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.004 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.004 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.004 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.004 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.004 DEBUG:boto:Method: GET 2023-09-22T21:34:49.004 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.004 DEBUG:boto:Data: 2023-09-22T21:34:49.005 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.005 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.005 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.005 DEBUG:boto:Params: {} 2023-09-22T21:34:49.005 DEBUG:boto:Token: None 2023-09-22T21:34:49.005 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.005 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5C/H+k1MyAyoc/CmydQgIlv1HDY= 2023-09-22T21:34:49.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5C/H+k1MyAyoc/CmydQgIlv1HDY=', 'Content-Length': '0'} 2023-09-22T21:34:49.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c098965e4110e736-00650e0879-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.050 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.051 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.051 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.051 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.051 DEBUG:boto:Method: HEAD 2023-09-22T21:34:49.051 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.051 DEBUG:boto:Data: 2023-09-22T21:34:49.051 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.051 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.051 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.052 DEBUG:boto:Params: {} 2023-09-22T21:34:49.052 DEBUG:boto:Token: None 2023-09-22T21:34:49.052 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.052 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:d48BhlzPs8XOS4NUIJh5zLioDa0= 2023-09-22T21:34:49.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:d48BhlzPs8XOS4NUIJh5zLioDa0=', 'Content-Length': '0'} 2023-09-22T21:34:49.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de7ff6efcc5da617-00650e0879-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.055 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.055 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.056 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.056 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.056 DEBUG:boto:Method: HEAD 2023-09-22T21:34:49.056 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.056 DEBUG:boto:Data: 2023-09-22T21:34:49.056 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.056 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.056 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.056 DEBUG:boto:Params: {} 2023-09-22T21:34:49.056 DEBUG:boto:Token: None 2023-09-22T21:34:49.057 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.057 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:d48BhlzPs8XOS4NUIJh5zLioDa0= 2023-09-22T21:34:49.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:d48BhlzPs8XOS4NUIJh5zLioDa0=', 'Content-Length': '0'} 2023-09-22T21:34:49.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3feda76d1787f28-00650e0879-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-09-22T21:34:49.060 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.060 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.060 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.060 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.060 DEBUG:boto:Method: GET 2023-09-22T21:34:49.060 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.061 DEBUG:boto:Data: 2023-09-22T21:34:49.061 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.061 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.061 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.061 DEBUG:boto:Params: {} 2023-09-22T21:34:49.061 DEBUG:boto:Token: None 2023-09-22T21:34:49.061 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.061 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5C/H+k1MyAyoc/CmydQgIlv1HDY= 2023-09-22T21:34:49.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5C/H+k1MyAyoc/CmydQgIlv1HDY=', 'Content-Length': '0'} 2023-09-22T21:34:49.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a13b0de6d331aab2-00650e0879-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.106 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.107 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.107 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.107 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.107 DEBUG:boto:Method: GET 2023-09-22T21:34:49.107 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.107 DEBUG:boto:Data: 2023-09-22T21:34:49.107 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.107 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.107 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.108 DEBUG:boto:Params: {} 2023-09-22T21:34:49.108 DEBUG:boto:Token: None 2023-09-22T21:34:49.108 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:34:49.108 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5C/H+k1MyAyoc/CmydQgIlv1HDY= 2023-09-22T21:34:49.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5C/H+k1MyAyoc/CmydQgIlv1HDY=', 'Content-Length': '0'} 2023-09-22T21:34:49.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000027dc5321443daa8-00650e0879-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.111 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:49.111 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:49.112 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:34:49.112 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:34:49.112 DEBUG:boto:Method: GET 2023-09-22T21:34:49.112 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:34:49.112 DEBUG:boto:Data: 2023-09-22T21:34:49.112 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.112 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.112 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.112 DEBUG:boto:Params: {} 2023-09-22T21:34:49.112 DEBUG:boto:Token: None 2023-09-22T21:34:49.113 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/ 2023-09-22T21:34:49.113 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU= 2023-09-22T21:34:49.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU=', 'Content-Length': '0'} 2023-09-22T21:34:49.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a6854fd25a6b6ed-00650e0879-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.158 DEBUG:boto:b'ffzdqu-58obj000000000000000000011000falseobj000000000000000000012023-09-22T21:34:33.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:49.159 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:49.159 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:49.159 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:34:49.160 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:34:49.160 DEBUG:boto:Method: GET 2023-09-22T21:34:49.160 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:34:49.160 DEBUG:boto:Data: 2023-09-22T21:34:49.160 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.160 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.160 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.160 DEBUG:boto:Params: {} 2023-09-22T21:34:49.160 DEBUG:boto:Token: None 2023-09-22T21:34:49.161 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/ 2023-09-22T21:34:49.161 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU= 2023-09-22T21:34:49.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU=', 'Content-Length': '0'} 2023-09-22T21:34:49.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089dd9b8840a7e644-00650e0879-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.166 DEBUG:boto:b'ffzdqu-58obj000000000000000000011000falseobj000000000000000000012023-09-22T21:34:33.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:49.166 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.166 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.166 DEBUG:boto:Method: HEAD 2023-09-22T21:34:49.166 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.167 DEBUG:boto:Data: 2023-09-22T21:34:49.167 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.167 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.167 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.167 DEBUG:boto:Params: {} 2023-09-22T21:34:49.167 DEBUG:boto:Token: None 2023-09-22T21:34:49.167 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.167 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vVdQzaQHccdRmktcxlS/1qQtS68= 2023-09-22T21:34:49.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vVdQzaQHccdRmktcxlS/1qQtS68=', 'Content-Length': '0'} 2023-09-22T21:34:49.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b48924985d85091-00650e0879-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.170 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.170 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.171 DEBUG:boto:Method: HEAD 2023-09-22T21:34:49.171 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.171 DEBUG:boto:Data: 2023-09-22T21:34:49.171 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.171 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.171 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.171 DEBUG:boto:Params: {} 2023-09-22T21:34:49.171 DEBUG:boto:Token: None 2023-09-22T21:34:49.171 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.171 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vVdQzaQHccdRmktcxlS/1qQtS68= 2023-09-22T21:34:49.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vVdQzaQHccdRmktcxlS/1qQtS68=', 'Content-Length': '0'} 2023-09-22T21:34:49.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008686b521d1a2d5b6-00650e0879-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-09-22T21:34:49.174 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.175 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.175 DEBUG:boto:Method: GET 2023-09-22T21:34:49.175 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.175 DEBUG:boto:Data: 2023-09-22T21:34:49.175 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.175 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.175 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.175 DEBUG:boto:Params: {} 2023-09-22T21:34:49.175 DEBUG:boto:Token: None 2023-09-22T21:34:49.175 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.176 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WbQMPq0rGS2xnb8mP6/otNrcFx8= 2023-09-22T21:34:49.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WbQMPq0rGS2xnb8mP6/otNrcFx8=', 'Content-Length': '0'} 2023-09-22T21:34:49.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000279a315b59d48138-00650e0879-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.222 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.222 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.223 DEBUG:boto:Method: GET 2023-09-22T21:34:49.223 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.223 DEBUG:boto:Data: 2023-09-22T21:34:49.223 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.223 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.223 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.223 DEBUG:boto:Params: {} 2023-09-22T21:34:49.223 DEBUG:boto:Token: None 2023-09-22T21:34:49.224 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000001 2023-09-22T21:34:49.224 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WbQMPq0rGS2xnb8mP6/otNrcFx8= 2023-09-22T21:34:49.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WbQMPq0rGS2xnb8mP6/otNrcFx8=', 'Content-Length': '0'} 2023-09-22T21:34:49.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c44a67ffd0bd282f-00650e0879-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.227 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:49.227 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:49.227 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:34:49.227 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:34:49.227 DEBUG:boto:Method: GET 2023-09-22T21:34:49.227 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:34:49.227 DEBUG:boto:Data: 2023-09-22T21:34:49.227 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.228 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.228 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.228 DEBUG:boto:Params: {} 2023-09-22T21:34:49.228 DEBUG:boto:Token: None 2023-09-22T21:34:49.228 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/ 2023-09-22T21:34:49.228 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU= 2023-09-22T21:34:49.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU=', 'Content-Length': '0'} 2023-09-22T21:34:49.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c333d77df4451d23-00650e0879-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.233 DEBUG:boto:b'ffzdqu-58obj000000000000000000011000falseobj000000000000000000012023-09-22T21:34:33.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:49.233 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:49.233 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:49.233 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:34:49.233 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:34:49.234 DEBUG:boto:Method: GET 2023-09-22T21:34:49.234 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:34:49.234 DEBUG:boto:Data: 2023-09-22T21:34:49.234 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.234 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.234 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.234 DEBUG:boto:Params: {} 2023-09-22T21:34:49.234 DEBUG:boto:Token: None 2023-09-22T21:34:49.234 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/ 2023-09-22T21:34:49.235 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU= 2023-09-22T21:34:49.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU=', 'Content-Length': '0'} 2023-09-22T21:34:49.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9a56e873163c339-00650e0879-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.282 DEBUG:boto:b'ffzdqu-58obj000000000000000000011000falseobj000000000000000000012023-09-22T21:34:33.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:49.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-09-22T21:34:49.283 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.283 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.283 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.283 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.283 DEBUG:boto:Method: GET 2023-09-22T21:34:49.284 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.284 DEBUG:boto:Data: 2023-09-22T21:34:49.284 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.284 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.284 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.284 DEBUG:boto:Params: {} 2023-09-22T21:34:49.284 DEBUG:boto:Token: None 2023-09-22T21:34:49.284 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.284 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YeJxcHR91vyQc9Qq7p5k4RBZ9U= 2023-09-22T21:34:49.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YeJxcHR91vyQc9Qq7p5k4RBZ9U=', 'Content-Length': '0'} 2023-09-22T21:34:49.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099d67ec8bae3ea71-00650e0879-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.334 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.335 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.335 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.335 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.335 DEBUG:boto:Method: GET 2023-09-22T21:34:49.335 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.335 DEBUG:boto:Data: 2023-09-22T21:34:49.335 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.335 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.335 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.336 DEBUG:boto:Params: {} 2023-09-22T21:34:49.336 DEBUG:boto:Token: None 2023-09-22T21:34:49.336 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.336 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YeJxcHR91vyQc9Qq7p5k4RBZ9U= 2023-09-22T21:34:49.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YeJxcHR91vyQc9Qq7p5k4RBZ9U=', 'Content-Length': '0'} 2023-09-22T21:34:49.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075b60f96b09c87a2-00650e0879-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.339 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.339 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.340 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.340 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.340 DEBUG:boto:Method: HEAD 2023-09-22T21:34:49.340 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.340 DEBUG:boto:Data: 2023-09-22T21:34:49.340 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.340 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.340 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.340 DEBUG:boto:Params: {} 2023-09-22T21:34:49.341 DEBUG:boto:Token: None 2023-09-22T21:34:49.341 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.341 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DFihlg0ijWh80qRz6jCZoEPZw4k= 2023-09-22T21:34:49.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DFihlg0ijWh80qRz6jCZoEPZw4k=', 'Content-Length': '0'} 2023-09-22T21:34:49.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc9f425c850152e7-00650e0879-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.344 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.344 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.344 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.344 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.344 DEBUG:boto:Method: HEAD 2023-09-22T21:34:49.344 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.344 DEBUG:boto:Data: 2023-09-22T21:34:49.344 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.345 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.345 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.345 DEBUG:boto:Params: {} 2023-09-22T21:34:49.345 DEBUG:boto:Token: None 2023-09-22T21:34:49.345 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.345 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DFihlg0ijWh80qRz6jCZoEPZw4k= 2023-09-22T21:34:49.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DFihlg0ijWh80qRz6jCZoEPZw4k=', 'Content-Length': '0'} 2023-09-22T21:34:49.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c349b83a2c984791-00650e0879-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-09-22T21:34:49.348 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.348 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.348 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.349 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.349 DEBUG:boto:Method: GET 2023-09-22T21:34:49.349 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.349 DEBUG:boto:Data: 2023-09-22T21:34:49.349 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.349 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.349 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.349 DEBUG:boto:Params: {} 2023-09-22T21:34:49.349 DEBUG:boto:Token: None 2023-09-22T21:34:49.350 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.350 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YeJxcHR91vyQc9Qq7p5k4RBZ9U= 2023-09-22T21:34:49.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YeJxcHR91vyQc9Qq7p5k4RBZ9U=', 'Content-Length': '0'} 2023-09-22T21:34:49.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c40f9f56ab14018-00650e0879-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.394 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.394 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.395 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.395 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.395 DEBUG:boto:Method: GET 2023-09-22T21:34:49.395 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.395 DEBUG:boto:Data: 2023-09-22T21:34:49.395 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.395 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.395 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.395 DEBUG:boto:Params: {} 2023-09-22T21:34:49.396 DEBUG:boto:Token: None 2023-09-22T21:34:49.396 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:34:49.396 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YeJxcHR91vyQc9Qq7p5k4RBZ9U= 2023-09-22T21:34:49.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YeJxcHR91vyQc9Qq7p5k4RBZ9U=', 'Content-Length': '0'} 2023-09-22T21:34:49.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7bc1fdfe0b3a067-00650e0879-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.399 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:49.399 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:49.399 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:34:49.399 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:34:49.400 DEBUG:boto:Method: GET 2023-09-22T21:34:49.400 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:34:49.400 DEBUG:boto:Data: 2023-09-22T21:34:49.400 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.400 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.400 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.400 DEBUG:boto:Params: {} 2023-09-22T21:34:49.400 DEBUG:boto:Token: None 2023-09-22T21:34:49.400 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/ 2023-09-22T21:34:49.400 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU= 2023-09-22T21:34:49.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU=', 'Content-Length': '0'} 2023-09-22T21:34:49.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e3ca7eb277721c6-00650e0879-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.446 DEBUG:boto:b'ffzdqu-58obj000000000000000000021000falseobj000000000000000000022023-09-22T21:34:33.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:49.447 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:49.447 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:49.447 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:34:49.447 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:34:49.447 DEBUG:boto:Method: GET 2023-09-22T21:34:49.447 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:34:49.447 DEBUG:boto:Data: 2023-09-22T21:34:49.447 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.447 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.448 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.448 DEBUG:boto:Params: {} 2023-09-22T21:34:49.448 DEBUG:boto:Token: None 2023-09-22T21:34:49.448 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/ 2023-09-22T21:34:49.448 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU= 2023-09-22T21:34:49.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU=', 'Content-Length': '0'} 2023-09-22T21:34:49.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8b4fe82a291b795-00650e0879-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.453 DEBUG:boto:b'ffzdqu-58obj000000000000000000021000falseobj000000000000000000022023-09-22T21:34:33.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:49.453 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.454 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.454 DEBUG:boto:Method: HEAD 2023-09-22T21:34:49.454 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.454 DEBUG:boto:Data: 2023-09-22T21:34:49.454 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.454 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.454 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.454 DEBUG:boto:Params: {} 2023-09-22T21:34:49.454 DEBUG:boto:Token: None 2023-09-22T21:34:49.455 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.455 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kGSXZ4i8QzSNWu7T4XDsBmFWEoQ= 2023-09-22T21:34:49.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kGSXZ4i8QzSNWu7T4XDsBmFWEoQ=', 'Content-Length': '0'} 2023-09-22T21:34:49.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006af4c4d6ab7c5f74-00650e0879-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.457 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.458 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.458 DEBUG:boto:Method: HEAD 2023-09-22T21:34:49.458 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.458 DEBUG:boto:Data: 2023-09-22T21:34:49.458 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.458 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.458 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.458 DEBUG:boto:Params: {} 2023-09-22T21:34:49.458 DEBUG:boto:Token: None 2023-09-22T21:34:49.459 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.459 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kGSXZ4i8QzSNWu7T4XDsBmFWEoQ= 2023-09-22T21:34:49.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kGSXZ4i8QzSNWu7T4XDsBmFWEoQ=', 'Content-Length': '0'} 2023-09-22T21:34:49.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004311d7f884ade2b9-00650e0879-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-09-22T21:34:49.462 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.462 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.462 DEBUG:boto:Method: GET 2023-09-22T21:34:49.462 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.462 DEBUG:boto:Data: 2023-09-22T21:34:49.462 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.463 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.463 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.463 DEBUG:boto:Params: {} 2023-09-22T21:34:49.463 DEBUG:boto:Token: None 2023-09-22T21:34:49.463 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.463 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4QTnao3lyB+Gu44xxp7/l765ZZM= 2023-09-22T21:34:49.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4QTnao3lyB+Gu44xxp7/l765ZZM=', 'Content-Length': '0'} 2023-09-22T21:34:49.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de91b06a6366f312-00650e0879-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.506 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.506 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.507 DEBUG:boto:Method: GET 2023-09-22T21:34:49.507 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.507 DEBUG:boto:Data: 2023-09-22T21:34:49.507 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.507 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.507 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.507 DEBUG:boto:Params: {} 2023-09-22T21:34:49.507 DEBUG:boto:Token: None 2023-09-22T21:34:49.508 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000002 2023-09-22T21:34:49.508 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4QTnao3lyB+Gu44xxp7/l765ZZM= 2023-09-22T21:34:49.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4QTnao3lyB+Gu44xxp7/l765ZZM=', 'Content-Length': '0'} 2023-09-22T21:34:49.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b2001bd9826d386-00650e0879-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.511 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:49.511 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:49.511 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:34:49.511 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:34:49.511 DEBUG:boto:Method: GET 2023-09-22T21:34:49.511 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:34:49.511 DEBUG:boto:Data: 2023-09-22T21:34:49.511 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.512 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.512 DEBUG:boto:Params: {} 2023-09-22T21:34:49.512 DEBUG:boto:Token: None 2023-09-22T21:34:49.512 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/ 2023-09-22T21:34:49.512 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU= 2023-09-22T21:34:49.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU=', 'Content-Length': '0'} 2023-09-22T21:34:49.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4160a70f5ba3149-00650e0879-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.517 DEBUG:boto:b'ffzdqu-58obj000000000000000000021000falseobj000000000000000000022023-09-22T21:34:33.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:49.518 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:49.518 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:49.518 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:34:49.518 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:34:49.518 DEBUG:boto:Method: GET 2023-09-22T21:34:49.518 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:34:49.518 DEBUG:boto:Data: 2023-09-22T21:34:49.519 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.519 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.519 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.519 DEBUG:boto:Params: {} 2023-09-22T21:34:49.519 DEBUG:boto:Token: None 2023-09-22T21:34:49.519 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/ 2023-09-22T21:34:49.519 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU= 2023-09-22T21:34:49.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU=', 'Content-Length': '0'} 2023-09-22T21:34:49.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009420c161946f2e2a-00650e0879-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.566 DEBUG:boto:b'ffzdqu-58obj000000000000000000021000falseobj000000000000000000022023-09-22T21:34:33.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:49.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-09-22T21:34:49.567 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.567 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.567 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.567 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.567 DEBUG:boto:Method: GET 2023-09-22T21:34:49.568 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.568 DEBUG:boto:Data: 2023-09-22T21:34:49.568 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.568 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.568 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.568 DEBUG:boto:Params: {} 2023-09-22T21:34:49.568 DEBUG:boto:Token: None 2023-09-22T21:34:49.568 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.568 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IfTMhypcrCyhM8MHO6Q8hxqJg+k= 2023-09-22T21:34:49.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IfTMhypcrCyhM8MHO6Q8hxqJg+k=', 'Content-Length': '0'} 2023-09-22T21:34:49.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c714e0523ad62a65-00650e0879-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.614 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.615 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.615 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.615 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.615 DEBUG:boto:Method: GET 2023-09-22T21:34:49.615 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.615 DEBUG:boto:Data: 2023-09-22T21:34:49.615 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.615 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.615 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.615 DEBUG:boto:Params: {} 2023-09-22T21:34:49.616 DEBUG:boto:Token: None 2023-09-22T21:34:49.616 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.616 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IfTMhypcrCyhM8MHO6Q8hxqJg+k= 2023-09-22T21:34:49.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IfTMhypcrCyhM8MHO6Q8hxqJg+k=', 'Content-Length': '0'} 2023-09-22T21:34:49.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e40bb1ba675749d0-00650e0879-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.619 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.619 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.619 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.619 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.620 DEBUG:boto:Method: HEAD 2023-09-22T21:34:49.620 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.620 DEBUG:boto:Data: 2023-09-22T21:34:49.620 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.620 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.620 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.620 DEBUG:boto:Params: {} 2023-09-22T21:34:49.620 DEBUG:boto:Token: None 2023-09-22T21:34:49.620 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.621 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WHoQp/MIC5XajD0iC1ykvEQEWek= 2023-09-22T21:34:49.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WHoQp/MIC5XajD0iC1ykvEQEWek=', 'Content-Length': '0'} 2023-09-22T21:34:49.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dd1be4172fdea43-00650e0879-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.623 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.623 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.624 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.624 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.624 DEBUG:boto:Method: HEAD 2023-09-22T21:34:49.624 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.624 DEBUG:boto:Data: 2023-09-22T21:34:49.624 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.624 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.624 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.624 DEBUG:boto:Params: {} 2023-09-22T21:34:49.624 DEBUG:boto:Token: None 2023-09-22T21:34:49.625 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.625 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WHoQp/MIC5XajD0iC1ykvEQEWek= 2023-09-22T21:34:49.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WHoQp/MIC5XajD0iC1ykvEQEWek=', 'Content-Length': '0'} 2023-09-22T21:34:49.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b071baff33b7b6f-00650e0879-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-09-22T21:34:49.628 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.628 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.628 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.628 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.628 DEBUG:boto:Method: GET 2023-09-22T21:34:49.628 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.628 DEBUG:boto:Data: 2023-09-22T21:34:49.628 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.629 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.629 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.629 DEBUG:boto:Params: {} 2023-09-22T21:34:49.629 DEBUG:boto:Token: None 2023-09-22T21:34:49.629 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.629 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IfTMhypcrCyhM8MHO6Q8hxqJg+k= 2023-09-22T21:34:49.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IfTMhypcrCyhM8MHO6Q8hxqJg+k=', 'Content-Length': '0'} 2023-09-22T21:34:49.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbe6027883c32e41-00650e0879-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.674 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.675 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.675 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.675 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.675 DEBUG:boto:Method: GET 2023-09-22T21:34:49.675 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.675 DEBUG:boto:Data: 2023-09-22T21:34:49.675 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.675 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.676 DEBUG:boto:Params: {} 2023-09-22T21:34:49.676 DEBUG:boto:Token: None 2023-09-22T21:34:49.676 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:34:49.676 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IfTMhypcrCyhM8MHO6Q8hxqJg+k= 2023-09-22T21:34:49.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IfTMhypcrCyhM8MHO6Q8hxqJg+k=', 'Content-Length': '0'} 2023-09-22T21:34:49.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003de2413b2ac91079-00650e0879-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.679 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:49.679 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:49.680 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:34:49.680 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:34:49.680 DEBUG:boto:Method: GET 2023-09-22T21:34:49.680 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:34:49.680 DEBUG:boto:Data: 2023-09-22T21:34:49.680 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.680 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.680 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.680 DEBUG:boto:Params: {} 2023-09-22T21:34:49.680 DEBUG:boto:Token: None 2023-09-22T21:34:49.681 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/ 2023-09-22T21:34:49.681 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU= 2023-09-22T21:34:49.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU=', 'Content-Length': '0'} 2023-09-22T21:34:49.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033b25c54b6cee7e5-00650e0879-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.726 DEBUG:boto:b'ffzdqu-58obj000000000000000000031000falseobj000000000000000000032023-09-22T21:34:33.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:49.727 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:49.727 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:49.727 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:34:49.727 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:34:49.728 DEBUG:boto:Method: GET 2023-09-22T21:34:49.728 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:34:49.728 DEBUG:boto:Data: 2023-09-22T21:34:49.728 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.728 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.728 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.728 DEBUG:boto:Params: {} 2023-09-22T21:34:49.728 DEBUG:boto:Token: None 2023-09-22T21:34:49.728 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/ 2023-09-22T21:34:49.729 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU= 2023-09-22T21:34:49.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU=', 'Content-Length': '0'} 2023-09-22T21:34:49.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066846a72e4c10dfe-00650e0879-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.733 DEBUG:boto:b'ffzdqu-58obj000000000000000000031000falseobj000000000000000000032023-09-22T21:34:33.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:49.734 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.734 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.734 DEBUG:boto:Method: HEAD 2023-09-22T21:34:49.734 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.734 DEBUG:boto:Data: 2023-09-22T21:34:49.734 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.734 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.734 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.734 DEBUG:boto:Params: {} 2023-09-22T21:34:49.735 DEBUG:boto:Token: None 2023-09-22T21:34:49.735 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.735 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BEskNmIlTCwmDYDYjHMlnymW4e8= 2023-09-22T21:34:49.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BEskNmIlTCwmDYDYjHMlnymW4e8=', 'Content-Length': '0'} 2023-09-22T21:34:49.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d5dad4e2cc4066f-00650e0879-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.738 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.738 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.738 DEBUG:boto:Method: HEAD 2023-09-22T21:34:49.738 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.738 DEBUG:boto:Data: 2023-09-22T21:34:49.738 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.739 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.739 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.739 DEBUG:boto:Params: {} 2023-09-22T21:34:49.739 DEBUG:boto:Token: None 2023-09-22T21:34:49.739 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.739 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BEskNmIlTCwmDYDYjHMlnymW4e8= 2023-09-22T21:34:49.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BEskNmIlTCwmDYDYjHMlnymW4e8=', 'Content-Length': '0'} 2023-09-22T21:34:49.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b17299c06fd0b659-00650e0879-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-09-22T21:34:49.742 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.742 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.742 DEBUG:boto:Method: GET 2023-09-22T21:34:49.743 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.743 DEBUG:boto:Data: 2023-09-22T21:34:49.743 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.743 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.743 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.743 DEBUG:boto:Params: {} 2023-09-22T21:34:49.743 DEBUG:boto:Token: None 2023-09-22T21:34:49.743 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.743 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:og/M4ZC0Osd4UF0GcYLHr0b0mqw= 2023-09-22T21:34:49.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:og/M4ZC0Osd4UF0GcYLHr0b0mqw=', 'Content-Length': '0'} 2023-09-22T21:34:49.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea392ece777ef594-00650e0879-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.786 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.787 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.787 DEBUG:boto:Method: GET 2023-09-22T21:34:49.787 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.787 DEBUG:boto:Data: 2023-09-22T21:34:49.787 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.787 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.787 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.787 DEBUG:boto:Params: {} 2023-09-22T21:34:49.788 DEBUG:boto:Token: None 2023-09-22T21:34:49.788 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000003 2023-09-22T21:34:49.788 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:og/M4ZC0Osd4UF0GcYLHr0b0mqw= 2023-09-22T21:34:49.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:og/M4ZC0Osd4UF0GcYLHr0b0mqw=', 'Content-Length': '0'} 2023-09-22T21:34:49.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058d21a58b0388a6e-00650e0879-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.791 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:49.791 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:49.791 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:34:49.791 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:34:49.791 DEBUG:boto:Method: GET 2023-09-22T21:34:49.792 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:34:49.792 DEBUG:boto:Data: 2023-09-22T21:34:49.792 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.792 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.792 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.792 DEBUG:boto:Params: {} 2023-09-22T21:34:49.792 DEBUG:boto:Token: None 2023-09-22T21:34:49.792 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/ 2023-09-22T21:34:49.792 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU= 2023-09-22T21:34:49.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU=', 'Content-Length': '0'} 2023-09-22T21:34:49.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000253dff55fd957a5b-00650e0879-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.797 DEBUG:boto:b'ffzdqu-58obj000000000000000000031000falseobj000000000000000000032023-09-22T21:34:33.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:49.798 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:49.798 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:49.798 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:34:49.798 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:34:49.798 DEBUG:boto:Method: GET 2023-09-22T21:34:49.798 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:34:49.798 DEBUG:boto:Data: 2023-09-22T21:34:49.799 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.799 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.799 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.799 DEBUG:boto:Params: {} 2023-09-22T21:34:49.799 DEBUG:boto:Token: None 2023-09-22T21:34:49.799 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/ 2023-09-22T21:34:49.799 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU= 2023-09-22T21:34:49.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU=', 'Content-Length': '0'} 2023-09-22T21:34:49.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003963c47ba462fe02-00650e0879-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.846 DEBUG:boto:b'ffzdqu-58obj000000000000000000031000falseobj000000000000000000032023-09-22T21:34:33.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:49.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-09-22T21:34:49.847 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:49.847 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:49.847 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.847 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.847 DEBUG:boto:Method: GET 2023-09-22T21:34:49.848 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.848 DEBUG:boto:Data: 2023-09-22T21:34:49.848 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.848 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.848 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.848 DEBUG:boto:Params: {} 2023-09-22T21:34:49.848 DEBUG:boto:Token: None 2023-09-22T21:34:49.848 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.848 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Fzk3Dt4JNrEp9z/hPyAe7nP82sk= 2023-09-22T21:34:49.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Fzk3Dt4JNrEp9z/hPyAe7nP82sk=', 'Content-Length': '0'} 2023-09-22T21:34:49.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045cfe2da8f2eb0dd-00650e0879-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.894 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:49.895 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:49.895 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.895 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.895 DEBUG:boto:Method: GET 2023-09-22T21:34:49.895 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.895 DEBUG:boto:Data: 2023-09-22T21:34:49.895 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.895 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.895 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.895 DEBUG:boto:Params: {} 2023-09-22T21:34:49.896 DEBUG:boto:Token: None 2023-09-22T21:34:49.896 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.896 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Fzk3Dt4JNrEp9z/hPyAe7nP82sk= 2023-09-22T21:34:49.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Fzk3Dt4JNrEp9z/hPyAe7nP82sk=', 'Content-Length': '0'} 2023-09-22T21:34:49.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a88c4ab402ca6e1b-00650e0879-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.899 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:49.899 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:49.900 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.900 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.900 DEBUG:boto:Method: HEAD 2023-09-22T21:34:49.900 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.900 DEBUG:boto:Data: 2023-09-22T21:34:49.900 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.900 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.900 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.900 DEBUG:boto:Params: {} 2023-09-22T21:34:49.900 DEBUG:boto:Token: None 2023-09-22T21:34:49.901 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.901 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dPo2vQXoFeAsUg3sFxA6PKNB42A= 2023-09-22T21:34:49.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dPo2vQXoFeAsUg3sFxA6PKNB42A=', 'Content-Length': '0'} 2023-09-22T21:34:49.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac17c490e877c2a0-00650e0879-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.904 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:49.904 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:49.904 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.904 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.904 DEBUG:boto:Method: HEAD 2023-09-22T21:34:49.904 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.904 DEBUG:boto:Data: 2023-09-22T21:34:49.904 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.904 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.904 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.905 DEBUG:boto:Params: {} 2023-09-22T21:34:49.905 DEBUG:boto:Token: None 2023-09-22T21:34:49.905 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.905 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dPo2vQXoFeAsUg3sFxA6PKNB42A= 2023-09-22T21:34:49.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dPo2vQXoFeAsUg3sFxA6PKNB42A=', 'Content-Length': '0'} 2023-09-22T21:34:49.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b77f880d4d93cb5-00650e0879-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-09-22T21:34:49.908 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:49.908 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:49.908 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.908 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.908 DEBUG:boto:Method: GET 2023-09-22T21:34:49.908 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.909 DEBUG:boto:Data: 2023-09-22T21:34:49.909 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.909 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.909 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.909 DEBUG:boto:Params: {} 2023-09-22T21:34:49.909 DEBUG:boto:Token: None 2023-09-22T21:34:49.909 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.909 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Fzk3Dt4JNrEp9z/hPyAe7nP82sk= 2023-09-22T21:34:49.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Fzk3Dt4JNrEp9z/hPyAe7nP82sk=', 'Content-Length': '0'} 2023-09-22T21:34:49.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001171fbccdecd03cd-00650e0879-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.958 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:49.959 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:49.959 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.959 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.959 DEBUG:boto:Method: GET 2023-09-22T21:34:49.959 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.959 DEBUG:boto:Data: 2023-09-22T21:34:49.959 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.959 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:49.959 DEBUG:boto:Port: 8000 2023-09-22T21:34:49.960 DEBUG:boto:Params: {} 2023-09-22T21:34:49.960 DEBUG:boto:Token: None 2023-09-22T21:34:49.960 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:34:49.960 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Fzk3Dt4JNrEp9z/hPyAe7nP82sk= 2023-09-22T21:34:49.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Fzk3Dt4JNrEp9z/hPyAe7nP82sk=', 'Content-Length': '0'} 2023-09-22T21:34:49.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d4610ec30eafe13-00650e0879-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:49.963 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:49.964 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:49.964 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:34:49.964 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:34:49.964 DEBUG:boto:Method: GET 2023-09-22T21:34:49.964 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:34:49.964 DEBUG:boto:Data: 2023-09-22T21:34:49.964 DEBUG:boto:Headers: {} 2023-09-22T21:34:49.964 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:49.964 DEBUG:boto:Port: 8001 2023-09-22T21:34:49.964 DEBUG:boto:Params: {} 2023-09-22T21:34:49.965 DEBUG:boto:Token: None 2023-09-22T21:34:49.965 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:49 GMT /ffzdqu-58/ 2023-09-22T21:34:49.965 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU= 2023-09-22T21:34:49.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:emUDEYXEOMGCnk9it68HY3iKmrU=', 'Content-Length': '0'} 2023-09-22T21:34:49.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099711551b89b88a3-00650e0879-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.018 DEBUG:boto:b'ffzdqu-58obj000000000000000000051000falseobj000000000000000000052023-09-22T21:34:33.361Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:50.019 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:50.019 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:50.019 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:34:50.019 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:34:50.019 DEBUG:boto:Method: GET 2023-09-22T21:34:50.019 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:34:50.020 DEBUG:boto:Data: 2023-09-22T21:34:50.020 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.020 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.020 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.020 DEBUG:boto:Params: {} 2023-09-22T21:34:50.020 DEBUG:boto:Token: None 2023-09-22T21:34:50.020 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/ 2023-09-22T21:34:50.020 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg= 2023-09-22T21:34:50.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg=', 'Content-Length': '0'} 2023-09-22T21:34:50.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adf9d5d2a2a545fd-00650e087a-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.025 DEBUG:boto:b'ffzdqu-58obj000000000000000000051000falseobj000000000000000000052023-09-22T21:34:33.361Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:50.026 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:50.026 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:50.026 DEBUG:boto:Method: HEAD 2023-09-22T21:34:50.026 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:50.026 DEBUG:boto:Data: 2023-09-22T21:34:50.026 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.027 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.027 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.027 DEBUG:boto:Params: {} 2023-09-22T21:34:50.027 DEBUG:boto:Token: None 2023-09-22T21:34:50.027 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:50.027 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zgxwUMEr1GHCw6vcMecAJsCLJec= 2023-09-22T21:34:50.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zgxwUMEr1GHCw6vcMecAJsCLJec=', 'Content-Length': '0'} 2023-09-22T21:34:50.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e8016c65cae89d0-00650e087a-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.030 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:50.031 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:50.031 DEBUG:boto:Method: HEAD 2023-09-22T21:34:50.031 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:50.031 DEBUG:boto:Data: 2023-09-22T21:34:50.031 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.031 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.031 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.031 DEBUG:boto:Params: {} 2023-09-22T21:34:50.031 DEBUG:boto:Token: None 2023-09-22T21:34:50.031 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:50.032 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zgxwUMEr1GHCw6vcMecAJsCLJec= 2023-09-22T21:34:50.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zgxwUMEr1GHCw6vcMecAJsCLJec=', 'Content-Length': '0'} 2023-09-22T21:34:50.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069c1ba8f3478cf0e-00650e087a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-09-22T21:34:50.035 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:50.035 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:50.035 DEBUG:boto:Method: GET 2023-09-22T21:34:50.035 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:50.035 DEBUG:boto:Data: 2023-09-22T21:34:50.035 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.036 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.036 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.036 DEBUG:boto:Params: {} 2023-09-22T21:34:50.036 DEBUG:boto:Token: None 2023-09-22T21:34:50.036 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:50.036 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RrKDk7hIWtNs1MO/QTVSRJNskR4= 2023-09-22T21:34:50.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RrKDk7hIWtNs1MO/QTVSRJNskR4=', 'Content-Length': '0'} 2023-09-22T21:34:50.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e8f87d828032a35-00650e087a-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.082 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:50.083 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:50.083 DEBUG:boto:Method: GET 2023-09-22T21:34:50.083 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:50.083 DEBUG:boto:Data: 2023-09-22T21:34:50.083 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.083 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.083 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.083 DEBUG:boto:Params: {} 2023-09-22T21:34:50.084 DEBUG:boto:Token: None 2023-09-22T21:34:50.084 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000005 2023-09-22T21:34:50.084 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RrKDk7hIWtNs1MO/QTVSRJNskR4= 2023-09-22T21:34:50.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RrKDk7hIWtNs1MO/QTVSRJNskR4=', 'Content-Length': '0'} 2023-09-22T21:34:50.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6dad43755dc874b-00650e087a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.087 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:50.087 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:50.088 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:34:50.088 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:34:50.088 DEBUG:boto:Method: GET 2023-09-22T21:34:50.088 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:34:50.088 DEBUG:boto:Data: 2023-09-22T21:34:50.088 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.088 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.088 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.088 DEBUG:boto:Params: {} 2023-09-22T21:34:50.088 DEBUG:boto:Token: None 2023-09-22T21:34:50.089 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/ 2023-09-22T21:34:50.089 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg= 2023-09-22T21:34:50.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg=', 'Content-Length': '0'} 2023-09-22T21:34:50.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c4ef54236545bd2-00650e087a-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.094 DEBUG:boto:b'ffzdqu-58obj000000000000000000051000falseobj000000000000000000052023-09-22T21:34:33.361Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:50.094 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:50.094 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:50.094 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:34:50.094 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:34:50.095 DEBUG:boto:Method: GET 2023-09-22T21:34:50.095 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:34:50.095 DEBUG:boto:Data: 2023-09-22T21:34:50.095 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.095 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.095 DEBUG:boto:Params: {} 2023-09-22T21:34:50.095 DEBUG:boto:Token: None 2023-09-22T21:34:50.095 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/ 2023-09-22T21:34:50.096 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg= 2023-09-22T21:34:50.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg=', 'Content-Length': '0'} 2023-09-22T21:34:50.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000505423f346b957e1-00650e087a-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.142 DEBUG:boto:b'ffzdqu-58obj000000000000000000051000falseobj000000000000000000052023-09-22T21:34:33.361Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:50.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-09-22T21:34:50.143 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.143 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.143 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.143 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.143 DEBUG:boto:Method: GET 2023-09-22T21:34:50.143 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.143 DEBUG:boto:Data: 2023-09-22T21:34:50.144 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.144 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.144 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.144 DEBUG:boto:Params: {} 2023-09-22T21:34:50.144 DEBUG:boto:Token: None 2023-09-22T21:34:50.144 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.144 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jusGb2WOJRig+VOqdtxsehSdjbw= 2023-09-22T21:34:50.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jusGb2WOJRig+VOqdtxsehSdjbw=', 'Content-Length': '0'} 2023-09-22T21:34:50.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000612414dd8b1d0c69-00650e087a-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.190 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.190 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.191 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.191 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.191 DEBUG:boto:Method: GET 2023-09-22T21:34:50.191 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.191 DEBUG:boto:Data: 2023-09-22T21:34:50.191 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.191 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.191 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.191 DEBUG:boto:Params: {} 2023-09-22T21:34:50.192 DEBUG:boto:Token: None 2023-09-22T21:34:50.192 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.192 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jusGb2WOJRig+VOqdtxsehSdjbw= 2023-09-22T21:34:50.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jusGb2WOJRig+VOqdtxsehSdjbw=', 'Content-Length': '0'} 2023-09-22T21:34:50.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7c9bb29bb190fca-00650e087a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.195 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.195 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.195 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.195 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.195 DEBUG:boto:Method: HEAD 2023-09-22T21:34:50.195 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.196 DEBUG:boto:Data: 2023-09-22T21:34:50.196 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.196 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.196 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.196 DEBUG:boto:Params: {} 2023-09-22T21:34:50.196 DEBUG:boto:Token: None 2023-09-22T21:34:50.196 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.196 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bkyLj0eSJnSsEp1xzCNpxUZILlc= 2023-09-22T21:34:50.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bkyLj0eSJnSsEp1xzCNpxUZILlc=', 'Content-Length': '0'} 2023-09-22T21:34:50.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8c5e6736e2526ba-00650e087a-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.200 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.200 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.200 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.200 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.200 DEBUG:boto:Method: HEAD 2023-09-22T21:34:50.200 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.200 DEBUG:boto:Data: 2023-09-22T21:34:50.200 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.200 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.201 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.201 DEBUG:boto:Params: {} 2023-09-22T21:34:50.201 DEBUG:boto:Token: None 2023-09-22T21:34:50.201 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.201 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bkyLj0eSJnSsEp1xzCNpxUZILlc= 2023-09-22T21:34:50.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bkyLj0eSJnSsEp1xzCNpxUZILlc=', 'Content-Length': '0'} 2023-09-22T21:34:50.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000584430eb68c007b5-00650e087a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-09-22T21:34:50.204 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.204 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.204 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.204 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.204 DEBUG:boto:Method: GET 2023-09-22T21:34:50.205 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.205 DEBUG:boto:Data: 2023-09-22T21:34:50.205 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.205 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.205 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.205 DEBUG:boto:Params: {} 2023-09-22T21:34:50.205 DEBUG:boto:Token: None 2023-09-22T21:34:50.205 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.205 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jusGb2WOJRig+VOqdtxsehSdjbw= 2023-09-22T21:34:50.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jusGb2WOJRig+VOqdtxsehSdjbw=', 'Content-Length': '0'} 2023-09-22T21:34:50.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8c77c999e6fef4e-00650e087a-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.254 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.254 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.255 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.255 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.255 DEBUG:boto:Method: GET 2023-09-22T21:34:50.255 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.255 DEBUG:boto:Data: 2023-09-22T21:34:50.255 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.255 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.255 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.255 DEBUG:boto:Params: {} 2023-09-22T21:34:50.256 DEBUG:boto:Token: None 2023-09-22T21:34:50.256 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:34:50.256 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jusGb2WOJRig+VOqdtxsehSdjbw= 2023-09-22T21:34:50.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jusGb2WOJRig+VOqdtxsehSdjbw=', 'Content-Length': '0'} 2023-09-22T21:34:50.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067c415315587e5d7-00650e087a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.259 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:50.259 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:50.259 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:34:50.259 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:34:50.260 DEBUG:boto:Method: GET 2023-09-22T21:34:50.260 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:34:50.260 DEBUG:boto:Data: 2023-09-22T21:34:50.260 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.260 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.260 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.260 DEBUG:boto:Params: {} 2023-09-22T21:34:50.260 DEBUG:boto:Token: None 2023-09-22T21:34:50.260 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/ 2023-09-22T21:34:50.261 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg= 2023-09-22T21:34:50.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg=', 'Content-Length': '0'} 2023-09-22T21:34:50.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000406ba308338b80fc-00650e087a-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.310 DEBUG:boto:b'ffzdqu-58obj000000000000000000061000falseobj000000000000000000062023-09-22T21:34:33.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:50.311 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:50.311 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:50.311 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:34:50.311 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:34:50.311 DEBUG:boto:Method: GET 2023-09-22T21:34:50.311 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:34:50.311 DEBUG:boto:Data: 2023-09-22T21:34:50.312 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.312 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.312 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.312 DEBUG:boto:Params: {} 2023-09-22T21:34:50.312 DEBUG:boto:Token: None 2023-09-22T21:34:50.312 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/ 2023-09-22T21:34:50.312 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg= 2023-09-22T21:34:50.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg=', 'Content-Length': '0'} 2023-09-22T21:34:50.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac8db204e310c2a1-00650e087a-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.317 DEBUG:boto:b'ffzdqu-58obj000000000000000000061000falseobj000000000000000000062023-09-22T21:34:33.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:50.317 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.318 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.318 DEBUG:boto:Method: HEAD 2023-09-22T21:34:50.318 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.318 DEBUG:boto:Data: 2023-09-22T21:34:50.318 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.318 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.318 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.318 DEBUG:boto:Params: {} 2023-09-22T21:34:50.319 DEBUG:boto:Token: None 2023-09-22T21:34:50.319 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.319 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YypVVL62oA2YmVs1ENmyKMR/m/g= 2023-09-22T21:34:50.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YypVVL62oA2YmVs1ENmyKMR/m/g=', 'Content-Length': '0'} 2023-09-22T21:34:50.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fc9773b8841f6aa-00650e087a-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.322 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.322 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.322 DEBUG:boto:Method: HEAD 2023-09-22T21:34:50.322 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.322 DEBUG:boto:Data: 2023-09-22T21:34:50.322 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.323 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.323 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.323 DEBUG:boto:Params: {} 2023-09-22T21:34:50.323 DEBUG:boto:Token: None 2023-09-22T21:34:50.323 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.323 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YypVVL62oA2YmVs1ENmyKMR/m/g= 2023-09-22T21:34:50.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YypVVL62oA2YmVs1ENmyKMR/m/g=', 'Content-Length': '0'} 2023-09-22T21:34:50.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085bcd460c9451a1d-00650e087a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-09-22T21:34:50.326 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.326 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.326 DEBUG:boto:Method: GET 2023-09-22T21:34:50.327 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.327 DEBUG:boto:Data: 2023-09-22T21:34:50.327 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.327 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.327 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.327 DEBUG:boto:Params: {} 2023-09-22T21:34:50.327 DEBUG:boto:Token: None 2023-09-22T21:34:50.327 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.327 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D+oltS+F53Xu41ibSULiXqITTks= 2023-09-22T21:34:50.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D+oltS+F53Xu41ibSULiXqITTks=', 'Content-Length': '0'} 2023-09-22T21:34:50.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb86cfea11609b5d-00650e087a-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.370 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.370 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.371 DEBUG:boto:Method: GET 2023-09-22T21:34:50.371 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.371 DEBUG:boto:Data: 2023-09-22T21:34:50.371 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.371 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.371 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.371 DEBUG:boto:Params: {} 2023-09-22T21:34:50.371 DEBUG:boto:Token: None 2023-09-22T21:34:50.371 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000006 2023-09-22T21:34:50.372 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D+oltS+F53Xu41ibSULiXqITTks= 2023-09-22T21:34:50.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D+oltS+F53Xu41ibSULiXqITTks=', 'Content-Length': '0'} 2023-09-22T21:34:50.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d200b07e0e86e651-00650e087a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.375 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:50.375 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:50.375 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:34:50.375 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:34:50.375 DEBUG:boto:Method: GET 2023-09-22T21:34:50.375 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:34:50.376 DEBUG:boto:Data: 2023-09-22T21:34:50.376 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.376 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.376 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.376 DEBUG:boto:Params: {} 2023-09-22T21:34:50.376 DEBUG:boto:Token: None 2023-09-22T21:34:50.376 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/ 2023-09-22T21:34:50.376 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg= 2023-09-22T21:34:50.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg=', 'Content-Length': '0'} 2023-09-22T21:34:50.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083c35a4d8db01599-00650e087a-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.381 DEBUG:boto:b'ffzdqu-58obj000000000000000000061000falseobj000000000000000000062023-09-22T21:34:33.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:50.381 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:50.382 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:50.382 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:34:50.382 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:34:50.382 DEBUG:boto:Method: GET 2023-09-22T21:34:50.382 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:34:50.382 DEBUG:boto:Data: 2023-09-22T21:34:50.382 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.382 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.382 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.383 DEBUG:boto:Params: {} 2023-09-22T21:34:50.383 DEBUG:boto:Token: None 2023-09-22T21:34:50.383 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/ 2023-09-22T21:34:50.383 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg= 2023-09-22T21:34:50.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg=', 'Content-Length': '0'} 2023-09-22T21:34:50.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000080d8329be067067-00650e087a-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.430 DEBUG:boto:b'ffzdqu-58obj000000000000000000061000falseobj000000000000000000062023-09-22T21:34:33.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:50.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-09-22T21:34:50.431 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.431 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.431 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.431 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.431 DEBUG:boto:Method: GET 2023-09-22T21:34:50.432 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.432 DEBUG:boto:Data: 2023-09-22T21:34:50.432 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.432 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.432 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.432 DEBUG:boto:Params: {} 2023-09-22T21:34:50.432 DEBUG:boto:Token: None 2023-09-22T21:34:50.432 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.432 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PdjXp+jdLyrxeFoi2EHf3p8mU0c= 2023-09-22T21:34:50.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PdjXp+jdLyrxeFoi2EHf3p8mU0c=', 'Content-Length': '0'} 2023-09-22T21:34:50.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eba83724b2db61e8-00650e087a-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.482 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.483 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.483 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.483 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.483 DEBUG:boto:Method: GET 2023-09-22T21:34:50.483 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.483 DEBUG:boto:Data: 2023-09-22T21:34:50.483 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.483 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.484 DEBUG:boto:Params: {} 2023-09-22T21:34:50.484 DEBUG:boto:Token: None 2023-09-22T21:34:50.484 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.484 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PdjXp+jdLyrxeFoi2EHf3p8mU0c= 2023-09-22T21:34:50.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PdjXp+jdLyrxeFoi2EHf3p8mU0c=', 'Content-Length': '0'} 2023-09-22T21:34:50.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8a14b264816e524-00650e087a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.487 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.487 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.488 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.488 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.488 DEBUG:boto:Method: HEAD 2023-09-22T21:34:50.488 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.488 DEBUG:boto:Data: 2023-09-22T21:34:50.488 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.488 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.488 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.488 DEBUG:boto:Params: {} 2023-09-22T21:34:50.489 DEBUG:boto:Token: None 2023-09-22T21:34:50.489 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.489 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:45em8iFJ0ZGqCnMWJslss2e2zl0= 2023-09-22T21:34:50.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:45em8iFJ0ZGqCnMWJslss2e2zl0=', 'Content-Length': '0'} 2023-09-22T21:34:50.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fa1e1c59b550e56-00650e087a-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.492 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.492 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.492 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.492 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.492 DEBUG:boto:Method: HEAD 2023-09-22T21:34:50.492 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.492 DEBUG:boto:Data: 2023-09-22T21:34:50.492 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.492 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.492 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.493 DEBUG:boto:Params: {} 2023-09-22T21:34:50.493 DEBUG:boto:Token: None 2023-09-22T21:34:50.493 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.493 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:45em8iFJ0ZGqCnMWJslss2e2zl0= 2023-09-22T21:34:50.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:45em8iFJ0ZGqCnMWJslss2e2zl0=', 'Content-Length': '0'} 2023-09-22T21:34:50.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cba9a58caec87d8-00650e087a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-09-22T21:34:50.496 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.496 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.496 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.496 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.496 DEBUG:boto:Method: GET 2023-09-22T21:34:50.496 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.497 DEBUG:boto:Data: 2023-09-22T21:34:50.497 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.497 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.497 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.497 DEBUG:boto:Params: {} 2023-09-22T21:34:50.497 DEBUG:boto:Token: None 2023-09-22T21:34:50.497 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.497 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PdjXp+jdLyrxeFoi2EHf3p8mU0c= 2023-09-22T21:34:50.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PdjXp+jdLyrxeFoi2EHf3p8mU0c=', 'Content-Length': '0'} 2023-09-22T21:34:50.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3511d0717506244-00650e087a-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.542 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.543 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.543 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.543 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.543 DEBUG:boto:Method: GET 2023-09-22T21:34:50.543 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.543 DEBUG:boto:Data: 2023-09-22T21:34:50.543 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.543 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.543 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.544 DEBUG:boto:Params: {} 2023-09-22T21:34:50.544 DEBUG:boto:Token: None 2023-09-22T21:34:50.544 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:34:50.544 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PdjXp+jdLyrxeFoi2EHf3p8mU0c= 2023-09-22T21:34:50.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PdjXp+jdLyrxeFoi2EHf3p8mU0c=', 'Content-Length': '0'} 2023-09-22T21:34:50.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000affe950a25ddf49a-00650e087a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.547 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:50.547 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:50.548 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:34:50.548 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:34:50.548 DEBUG:boto:Method: GET 2023-09-22T21:34:50.548 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:34:50.548 DEBUG:boto:Data: 2023-09-22T21:34:50.548 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.548 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.548 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.548 DEBUG:boto:Params: {} 2023-09-22T21:34:50.548 DEBUG:boto:Token: None 2023-09-22T21:34:50.549 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/ 2023-09-22T21:34:50.549 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg= 2023-09-22T21:34:50.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg=', 'Content-Length': '0'} 2023-09-22T21:34:50.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b48b60fb82f3eac-00650e087a-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.598 DEBUG:boto:b'ffzdqu-58obj000000000000000000101000falseobj000000000000000000102023-09-22T21:34:33.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:50.599 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:50.599 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:50.599 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:34:50.599 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:34:50.599 DEBUG:boto:Method: GET 2023-09-22T21:34:50.600 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:34:50.600 DEBUG:boto:Data: 2023-09-22T21:34:50.600 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.600 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.600 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.600 DEBUG:boto:Params: {} 2023-09-22T21:34:50.600 DEBUG:boto:Token: None 2023-09-22T21:34:50.600 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/ 2023-09-22T21:34:50.601 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg= 2023-09-22T21:34:50.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg=', 'Content-Length': '0'} 2023-09-22T21:34:50.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002002bd19a78f5a00-00650e087a-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.606 DEBUG:boto:b'ffzdqu-58obj000000000000000000101000falseobj000000000000000000102023-09-22T21:34:33.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:50.606 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.606 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.606 DEBUG:boto:Method: HEAD 2023-09-22T21:34:50.606 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.607 DEBUG:boto:Data: 2023-09-22T21:34:50.607 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.607 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.607 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.607 DEBUG:boto:Params: {} 2023-09-22T21:34:50.607 DEBUG:boto:Token: None 2023-09-22T21:34:50.607 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.607 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8UquGnCyCCU1Pr5qwCoGwNQTJO4= 2023-09-22T21:34:50.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8UquGnCyCCU1Pr5qwCoGwNQTJO4=', 'Content-Length': '0'} 2023-09-22T21:34:50.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de68f2d37e9be042-00650e087a-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.610 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.610 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.611 DEBUG:boto:Method: HEAD 2023-09-22T21:34:50.611 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.611 DEBUG:boto:Data: 2023-09-22T21:34:50.611 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.611 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.611 DEBUG:boto:Params: {} 2023-09-22T21:34:50.611 DEBUG:boto:Token: None 2023-09-22T21:34:50.612 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8UquGnCyCCU1Pr5qwCoGwNQTJO4= 2023-09-22T21:34:50.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8UquGnCyCCU1Pr5qwCoGwNQTJO4=', 'Content-Length': '0'} 2023-09-22T21:34:50.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd5d1ba5e213c601-00650e087a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-09-22T21:34:50.615 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.615 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.615 DEBUG:boto:Method: GET 2023-09-22T21:34:50.615 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.615 DEBUG:boto:Data: 2023-09-22T21:34:50.616 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.616 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.616 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.616 DEBUG:boto:Params: {} 2023-09-22T21:34:50.616 DEBUG:boto:Token: None 2023-09-22T21:34:50.616 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.616 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9gW0XytEqgozUJDRkqi/DAEK/oc= 2023-09-22T21:34:50.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9gW0XytEqgozUJDRkqi/DAEK/oc=', 'Content-Length': '0'} 2023-09-22T21:34:50.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005de937f37d0d2dcf-00650e087a-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.666 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.667 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.667 DEBUG:boto:Method: GET 2023-09-22T21:34:50.667 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.667 DEBUG:boto:Data: 2023-09-22T21:34:50.668 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.668 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.668 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.668 DEBUG:boto:Params: {} 2023-09-22T21:34:50.669 DEBUG:boto:Token: None 2023-09-22T21:34:50.669 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:50.669 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9gW0XytEqgozUJDRkqi/DAEK/oc= 2023-09-22T21:34:50.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9gW0XytEqgozUJDRkqi/DAEK/oc=', 'Content-Length': '0'} 2023-09-22T21:34:50.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef50f5999b18527b-00650e087a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.673 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:50.673 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:50.673 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:34:50.673 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:34:50.674 DEBUG:boto:Method: GET 2023-09-22T21:34:50.674 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:34:50.674 DEBUG:boto:Data: 2023-09-22T21:34:50.674 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.674 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.674 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.675 DEBUG:boto:Params: {} 2023-09-22T21:34:50.675 DEBUG:boto:Token: None 2023-09-22T21:34:50.675 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/ 2023-09-22T21:34:50.675 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg= 2023-09-22T21:34:50.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg=', 'Content-Length': '0'} 2023-09-22T21:34:50.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025ee6b5477d7ee21-00650e087a-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.726 DEBUG:boto:b'ffzdqu-58obj000000000000000000101000falseobj000000000000000000102023-09-22T21:34:33.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:50.727 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:50.727 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:50.727 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:34:50.727 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:34:50.727 DEBUG:boto:Method: GET 2023-09-22T21:34:50.727 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:34:50.728 DEBUG:boto:Data: 2023-09-22T21:34:50.728 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.728 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.728 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.728 DEBUG:boto:Params: {} 2023-09-22T21:34:50.728 DEBUG:boto:Token: None 2023-09-22T21:34:50.728 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/ 2023-09-22T21:34:50.728 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg= 2023-09-22T21:34:50.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg=', 'Content-Length': '0'} 2023-09-22T21:34:50.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a4e40250573d31a-00650e087a-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.735 DEBUG:boto:b'ffzdqu-58obj000000000000000000101000falseobj000000000000000000102023-09-22T21:34:33.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:50.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-09-22T21:34:50.735 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.735 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.735 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.736 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.736 DEBUG:boto:Method: GET 2023-09-22T21:34:50.736 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.736 DEBUG:boto:Data: 2023-09-22T21:34:50.736 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.736 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.736 DEBUG:boto:Params: {} 2023-09-22T21:34:50.737 DEBUG:boto:Token: None 2023-09-22T21:34:50.737 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.737 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3SWM7TmGpVLiyHcRaU/9HW3J77s= 2023-09-22T21:34:50.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3SWM7TmGpVLiyHcRaU/9HW3J77s=', 'Content-Length': '0'} 2023-09-22T21:34:50.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a03e0b0711967694-00650e087a-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.782 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.783 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.783 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.783 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.783 DEBUG:boto:Method: GET 2023-09-22T21:34:50.783 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.783 DEBUG:boto:Data: 2023-09-22T21:34:50.783 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.783 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.784 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.784 DEBUG:boto:Params: {} 2023-09-22T21:34:50.784 DEBUG:boto:Token: None 2023-09-22T21:34:50.784 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.784 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3SWM7TmGpVLiyHcRaU/9HW3J77s= 2023-09-22T21:34:50.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3SWM7TmGpVLiyHcRaU/9HW3J77s=', 'Content-Length': '0'} 2023-09-22T21:34:50.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba15906b53258da3-00650e087a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.787 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.788 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.788 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.788 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.788 DEBUG:boto:Method: HEAD 2023-09-22T21:34:50.788 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.788 DEBUG:boto:Data: 2023-09-22T21:34:50.788 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.788 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.789 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.789 DEBUG:boto:Params: {} 2023-09-22T21:34:50.789 DEBUG:boto:Token: None 2023-09-22T21:34:50.789 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.789 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WvS/suMDUJvq+w/KaEPwT7Lo4D8= 2023-09-22T21:34:50.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WvS/suMDUJvq+w/KaEPwT7Lo4D8=', 'Content-Length': '0'} 2023-09-22T21:34:50.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a13870827bfd55c-00650e087a-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.793 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.793 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.793 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.793 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.794 DEBUG:boto:Method: HEAD 2023-09-22T21:34:50.794 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.794 DEBUG:boto:Data: 2023-09-22T21:34:50.794 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.794 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.794 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.794 DEBUG:boto:Params: {} 2023-09-22T21:34:50.794 DEBUG:boto:Token: None 2023-09-22T21:34:50.795 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.795 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WvS/suMDUJvq+w/KaEPwT7Lo4D8= 2023-09-22T21:34:50.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WvS/suMDUJvq+w/KaEPwT7Lo4D8=', 'Content-Length': '0'} 2023-09-22T21:34:50.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c006d801029297d-00650e087a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-09-22T21:34:50.799 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.799 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.799 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.799 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.799 DEBUG:boto:Method: GET 2023-09-22T21:34:50.799 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.799 DEBUG:boto:Data: 2023-09-22T21:34:50.800 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.800 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.800 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.800 DEBUG:boto:Params: {} 2023-09-22T21:34:50.800 DEBUG:boto:Token: None 2023-09-22T21:34:50.800 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.800 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3SWM7TmGpVLiyHcRaU/9HW3J77s= 2023-09-22T21:34:50.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3SWM7TmGpVLiyHcRaU/9HW3J77s=', 'Content-Length': '0'} 2023-09-22T21:34:50.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f616fb1ce53b779-00650e087a-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.850 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.851 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.851 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.851 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.851 DEBUG:boto:Method: GET 2023-09-22T21:34:50.851 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.851 DEBUG:boto:Data: 2023-09-22T21:34:50.851 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.851 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.851 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.851 DEBUG:boto:Params: {} 2023-09-22T21:34:50.852 DEBUG:boto:Token: None 2023-09-22T21:34:50.852 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:34:50.852 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3SWM7TmGpVLiyHcRaU/9HW3J77s= 2023-09-22T21:34:50.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3SWM7TmGpVLiyHcRaU/9HW3J77s=', 'Content-Length': '0'} 2023-09-22T21:34:50.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003304d527b879cd31-00650e087a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.855 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:50.855 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:50.855 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:34:50.855 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:34:50.856 DEBUG:boto:Method: GET 2023-09-22T21:34:50.856 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:34:50.856 DEBUG:boto:Data: 2023-09-22T21:34:50.856 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.856 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.856 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.856 DEBUG:boto:Params: {} 2023-09-22T21:34:50.856 DEBUG:boto:Token: None 2023-09-22T21:34:50.856 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/ 2023-09-22T21:34:50.857 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg= 2023-09-22T21:34:50.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg=', 'Content-Length': '0'} 2023-09-22T21:34:50.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079906a5e0a2e9e1b-00650e087a-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.902 DEBUG:boto:b'ffzdqu-58obj000000000000000000111000falseobj000000000000000000112023-09-22T21:34:33.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:50.903 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:50.903 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:50.903 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:34:50.903 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:34:50.903 DEBUG:boto:Method: GET 2023-09-22T21:34:50.903 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:34:50.904 DEBUG:boto:Data: 2023-09-22T21:34:50.904 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.904 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.904 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.904 DEBUG:boto:Params: {} 2023-09-22T21:34:50.904 DEBUG:boto:Token: None 2023-09-22T21:34:50.904 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/ 2023-09-22T21:34:50.904 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg= 2023-09-22T21:34:50.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg=', 'Content-Length': '0'} 2023-09-22T21:34:50.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd2bc4d794643dab-00650e087a-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.910 DEBUG:boto:b'ffzdqu-58obj000000000000000000111000falseobj000000000000000000112023-09-22T21:34:33.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:50.910 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.911 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.911 DEBUG:boto:Method: HEAD 2023-09-22T21:34:50.911 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.911 DEBUG:boto:Data: 2023-09-22T21:34:50.911 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.911 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.911 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.911 DEBUG:boto:Params: {} 2023-09-22T21:34:50.911 DEBUG:boto:Token: None 2023-09-22T21:34:50.912 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.912 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ObAzyQET8U7xBMwhUo3QQxjL1us= 2023-09-22T21:34:50.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ObAzyQET8U7xBMwhUo3QQxjL1us=', 'Content-Length': '0'} 2023-09-22T21:34:50.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064c5f5eb6e12482f-00650e087a-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.915 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.915 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.915 DEBUG:boto:Method: HEAD 2023-09-22T21:34:50.915 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.915 DEBUG:boto:Data: 2023-09-22T21:34:50.915 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.915 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.916 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.916 DEBUG:boto:Params: {} 2023-09-22T21:34:50.916 DEBUG:boto:Token: None 2023-09-22T21:34:50.916 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.916 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ObAzyQET8U7xBMwhUo3QQxjL1us= 2023-09-22T21:34:50.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ObAzyQET8U7xBMwhUo3QQxjL1us=', 'Content-Length': '0'} 2023-09-22T21:34:50.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f18b5efd23e8b607-00650e087a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-09-22T21:34:50.919 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.919 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.919 DEBUG:boto:Method: GET 2023-09-22T21:34:50.919 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.920 DEBUG:boto:Data: 2023-09-22T21:34:50.920 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.920 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.920 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.920 DEBUG:boto:Params: {} 2023-09-22T21:34:50.920 DEBUG:boto:Token: None 2023-09-22T21:34:50.920 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.920 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Sv6nsN+PRepjqmRvfcsVwf4zl5M= 2023-09-22T21:34:50.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Sv6nsN+PRepjqmRvfcsVwf4zl5M=', 'Content-Length': '0'} 2023-09-22T21:34:50.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034dd805ca0b729c2-00650e087a-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.966 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.967 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.967 DEBUG:boto:Method: GET 2023-09-22T21:34:50.967 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.967 DEBUG:boto:Data: 2023-09-22T21:34:50.967 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.967 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:50.967 DEBUG:boto:Port: 8000 2023-09-22T21:34:50.967 DEBUG:boto:Params: {} 2023-09-22T21:34:50.968 DEBUG:boto:Token: None 2023-09-22T21:34:50.968 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/obj00000000000000000011 2023-09-22T21:34:50.968 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Sv6nsN+PRepjqmRvfcsVwf4zl5M= 2023-09-22T21:34:50.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Sv6nsN+PRepjqmRvfcsVwf4zl5M=', 'Content-Length': '0'} 2023-09-22T21:34:50.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f581251a0d6aa19-00650e087a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:50.971 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:50.972 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:50.972 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:34:50.972 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:34:50.972 DEBUG:boto:Method: GET 2023-09-22T21:34:50.972 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:34:50.972 DEBUG:boto:Data: 2023-09-22T21:34:50.972 DEBUG:boto:Headers: {} 2023-09-22T21:34:50.972 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:50.972 DEBUG:boto:Port: 8001 2023-09-22T21:34:50.972 DEBUG:boto:Params: {} 2023-09-22T21:34:50.973 DEBUG:boto:Token: None 2023-09-22T21:34:50.973 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:50 GMT /ffzdqu-58/ 2023-09-22T21:34:50.973 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg= 2023-09-22T21:34:50.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:50 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+769crdODERpLsd9A6EZtptt6hg=', 'Content-Length': '0'} 2023-09-22T21:34:50.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037465e297c87307d-00650e087a-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.022 DEBUG:boto:b'ffzdqu-58obj000000000000000000111000falseobj000000000000000000112023-09-22T21:34:33.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:51.023 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:51.023 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:51.023 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:34:51.023 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:34:51.024 DEBUG:boto:Method: GET 2023-09-22T21:34:51.024 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:34:51.024 DEBUG:boto:Data: 2023-09-22T21:34:51.024 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.024 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:51.024 DEBUG:boto:Port: 8000 2023-09-22T21:34:51.024 DEBUG:boto:Params: {} 2023-09-22T21:34:51.024 DEBUG:boto:Token: None 2023-09-22T21:34:51.024 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/ 2023-09-22T21:34:51.025 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MMgZkjVARknygwB1Atke4l4WHTU= 2023-09-22T21:34:51.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MMgZkjVARknygwB1Atke4l4WHTU=', 'Content-Length': '0'} 2023-09-22T21:34:51.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003093f551414e532f-00650e087b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.030 DEBUG:boto:b'ffzdqu-58obj000000000000000000111000falseobj000000000000000000112023-09-22T21:34:33.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:51.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-09-22T21:34:51.030 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.031 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.031 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.031 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.031 DEBUG:boto:Method: GET 2023-09-22T21:34:51.031 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.031 DEBUG:boto:Data: 2023-09-22T21:34:51.031 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.031 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:51.031 DEBUG:boto:Port: 8001 2023-09-22T21:34:51.031 DEBUG:boto:Params: {} 2023-09-22T21:34:51.032 DEBUG:boto:Token: None 2023-09-22T21:34:51.032 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.032 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fEls0xDemIIXdz/YXCsFGASgBVs= 2023-09-22T21:34:51.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fEls0xDemIIXdz/YXCsFGASgBVs=', 'Content-Length': '0'} 2023-09-22T21:34:51.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb16189942fd2a7f-00650e087b-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.035 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.035 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.035 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.036 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.036 DEBUG:boto:Method: GET 2023-09-22T21:34:51.036 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.036 DEBUG:boto:Data: 2023-09-22T21:34:51.036 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.036 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:51.036 DEBUG:boto:Port: 8000 2023-09-22T21:34:51.036 DEBUG:boto:Params: {} 2023-09-22T21:34:51.036 DEBUG:boto:Token: None 2023-09-22T21:34:51.037 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.037 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fEls0xDemIIXdz/YXCsFGASgBVs= 2023-09-22T21:34:51.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fEls0xDemIIXdz/YXCsFGASgBVs=', 'Content-Length': '0'} 2023-09-22T21:34:51.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061bf284112dc924b-00650e087b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.040 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.040 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.040 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.040 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.040 DEBUG:boto:Method: HEAD 2023-09-22T21:34:51.040 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.041 DEBUG:boto:Data: 2023-09-22T21:34:51.041 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.041 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:51.041 DEBUG:boto:Port: 8001 2023-09-22T21:34:51.041 DEBUG:boto:Params: {} 2023-09-22T21:34:51.041 DEBUG:boto:Token: None 2023-09-22T21:34:51.041 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.041 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zyxbFRvxB0jFQr+1ArViJisg0Lo= 2023-09-22T21:34:51.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zyxbFRvxB0jFQr+1ArViJisg0Lo=', 'Content-Length': '0'} 2023-09-22T21:34:51.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d4b20be49790034-00650e087b-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.044 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.044 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.044 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.045 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.045 DEBUG:boto:Method: HEAD 2023-09-22T21:34:51.045 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.045 DEBUG:boto:Data: 2023-09-22T21:34:51.045 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.045 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:51.045 DEBUG:boto:Port: 8000 2023-09-22T21:34:51.045 DEBUG:boto:Params: {} 2023-09-22T21:34:51.045 DEBUG:boto:Token: None 2023-09-22T21:34:51.046 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.046 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zyxbFRvxB0jFQr+1ArViJisg0Lo= 2023-09-22T21:34:51.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zyxbFRvxB0jFQr+1ArViJisg0Lo=', 'Content-Length': '0'} 2023-09-22T21:34:51.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000446b57cbb4f5d019-00650e087b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-09-22T21:34:51.049 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.049 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.049 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.049 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.049 DEBUG:boto:Method: GET 2023-09-22T21:34:51.049 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.049 DEBUG:boto:Data: 2023-09-22T21:34:51.049 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.049 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:51.050 DEBUG:boto:Port: 8001 2023-09-22T21:34:51.050 DEBUG:boto:Params: {} 2023-09-22T21:34:51.050 DEBUG:boto:Token: None 2023-09-22T21:34:51.050 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.050 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fEls0xDemIIXdz/YXCsFGASgBVs= 2023-09-22T21:34:51.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fEls0xDemIIXdz/YXCsFGASgBVs=', 'Content-Length': '0'} 2023-09-22T21:34:51.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018e48bfd25b38af6-00650e087b-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.094 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.095 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.095 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.095 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.095 DEBUG:boto:Method: GET 2023-09-22T21:34:51.095 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.095 DEBUG:boto:Data: 2023-09-22T21:34:51.095 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:51.095 DEBUG:boto:Port: 8000 2023-09-22T21:34:51.096 DEBUG:boto:Params: {} 2023-09-22T21:34:51.096 DEBUG:boto:Token: None 2023-09-22T21:34:51.096 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:34:51.096 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fEls0xDemIIXdz/YXCsFGASgBVs= 2023-09-22T21:34:51.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fEls0xDemIIXdz/YXCsFGASgBVs=', 'Content-Length': '0'} 2023-09-22T21:34:51.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029026e4d1925f69e-00650e087b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.099 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:51.099 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:51.100 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:34:51.100 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:34:51.100 DEBUG:boto:Method: GET 2023-09-22T21:34:51.100 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:34:51.100 DEBUG:boto:Data: 2023-09-22T21:34:51.100 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.100 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:51.100 DEBUG:boto:Port: 8001 2023-09-22T21:34:51.100 DEBUG:boto:Params: {} 2023-09-22T21:34:51.101 DEBUG:boto:Token: None 2023-09-22T21:34:51.101 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/ 2023-09-22T21:34:51.101 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MMgZkjVARknygwB1Atke4l4WHTU= 2023-09-22T21:34:51.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MMgZkjVARknygwB1Atke4l4WHTU=', 'Content-Length': '0'} 2023-09-22T21:34:51.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a6f2746653bc0d3-00650e087b-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.106 DEBUG:boto:b'ffzdqu-58obj000000000000000000121000falseobj000000000000000000122023-09-22T21:34:33.269Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:51.106 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:51.107 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:51.107 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:34:51.107 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:34:51.107 DEBUG:boto:Method: GET 2023-09-22T21:34:51.107 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:34:51.107 DEBUG:boto:Data: 2023-09-22T21:34:51.107 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.107 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:51.107 DEBUG:boto:Port: 8000 2023-09-22T21:34:51.107 DEBUG:boto:Params: {} 2023-09-22T21:34:51.108 DEBUG:boto:Token: None 2023-09-22T21:34:51.108 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/ 2023-09-22T21:34:51.108 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MMgZkjVARknygwB1Atke4l4WHTU= 2023-09-22T21:34:51.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MMgZkjVARknygwB1Atke4l4WHTU=', 'Content-Length': '0'} 2023-09-22T21:34:51.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e7b856ea4fcd06a-00650e087b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.113 DEBUG:boto:b'ffzdqu-58obj000000000000000000121000falseobj000000000000000000122023-09-22T21:34:33.269Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:51.113 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.113 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.113 DEBUG:boto:Method: HEAD 2023-09-22T21:34:51.114 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.114 DEBUG:boto:Data: 2023-09-22T21:34:51.114 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.114 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:51.114 DEBUG:boto:Port: 8001 2023-09-22T21:34:51.114 DEBUG:boto:Params: {} 2023-09-22T21:34:51.114 DEBUG:boto:Token: None 2023-09-22T21:34:51.114 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.115 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MsXZLu9pOpTWNkV+ZqbgUEhi+0w= 2023-09-22T21:34:51.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MsXZLu9pOpTWNkV+ZqbgUEhi+0w=', 'Content-Length': '0'} 2023-09-22T21:34:51.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e562f338f11436e7-00650e087b-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.118 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.118 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.118 DEBUG:boto:Method: HEAD 2023-09-22T21:34:51.118 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.118 DEBUG:boto:Data: 2023-09-22T21:34:51.118 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.118 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:51.118 DEBUG:boto:Port: 8000 2023-09-22T21:34:51.119 DEBUG:boto:Params: {} 2023-09-22T21:34:51.119 DEBUG:boto:Token: None 2023-09-22T21:34:51.119 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.119 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MsXZLu9pOpTWNkV+ZqbgUEhi+0w= 2023-09-22T21:34:51.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MsXZLu9pOpTWNkV+ZqbgUEhi+0w=', 'Content-Length': '0'} 2023-09-22T21:34:51.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0451d4a31fb0d91-00650e087b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-09-22T21:34:51.122 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.122 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.122 DEBUG:boto:Method: GET 2023-09-22T21:34:51.122 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.122 DEBUG:boto:Data: 2023-09-22T21:34:51.123 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.123 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:51.123 DEBUG:boto:Port: 8001 2023-09-22T21:34:51.123 DEBUG:boto:Params: {} 2023-09-22T21:34:51.123 DEBUG:boto:Token: None 2023-09-22T21:34:51.123 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.123 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fGiJwBxoLKHr/b7HMeKQ6MWnvUQ= 2023-09-22T21:34:51.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fGiJwBxoLKHr/b7HMeKQ6MWnvUQ=', 'Content-Length': '0'} 2023-09-22T21:34:51.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000605b7c82ab2a704d-00650e087b-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.174 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.175 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.175 DEBUG:boto:Method: GET 2023-09-22T21:34:51.175 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.175 DEBUG:boto:Data: 2023-09-22T21:34:51.175 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.175 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:51.176 DEBUG:boto:Port: 8000 2023-09-22T21:34:51.176 DEBUG:boto:Params: {} 2023-09-22T21:34:51.176 DEBUG:boto:Token: None 2023-09-22T21:34:51.176 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/obj00000000000000000012 2023-09-22T21:34:51.176 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fGiJwBxoLKHr/b7HMeKQ6MWnvUQ= 2023-09-22T21:34:51.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fGiJwBxoLKHr/b7HMeKQ6MWnvUQ=', 'Content-Length': '0'} 2023-09-22T21:34:51.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000253e409a9f0e65a8-00650e087b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.180 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:51.180 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:51.180 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:34:51.180 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:34:51.180 DEBUG:boto:Method: GET 2023-09-22T21:34:51.180 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:34:51.180 DEBUG:boto:Data: 2023-09-22T21:34:51.181 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.181 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:51.181 DEBUG:boto:Port: 8001 2023-09-22T21:34:51.181 DEBUG:boto:Params: {} 2023-09-22T21:34:51.181 DEBUG:boto:Token: None 2023-09-22T21:34:51.181 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/ 2023-09-22T21:34:51.181 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MMgZkjVARknygwB1Atke4l4WHTU= 2023-09-22T21:34:51.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MMgZkjVARknygwB1Atke4l4WHTU=', 'Content-Length': '0'} 2023-09-22T21:34:51.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2fa45f64e282187-00650e087b-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.230 DEBUG:boto:b'ffzdqu-58obj000000000000000000121000falseobj000000000000000000122023-09-22T21:34:33.269Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:51.231 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:51.231 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:51.231 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:34:51.231 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:34:51.231 DEBUG:boto:Method: GET 2023-09-22T21:34:51.231 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:34:51.231 DEBUG:boto:Data: 2023-09-22T21:34:51.232 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.232 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:51.232 DEBUG:boto:Port: 8000 2023-09-22T21:34:51.232 DEBUG:boto:Params: {} 2023-09-22T21:34:51.232 DEBUG:boto:Token: None 2023-09-22T21:34:51.232 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/ 2023-09-22T21:34:51.233 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MMgZkjVARknygwB1Atke4l4WHTU= 2023-09-22T21:34:51.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MMgZkjVARknygwB1Atke4l4WHTU=', 'Content-Length': '0'} 2023-09-22T21:34:51.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc691e834103f18e-00650e087b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.238 DEBUG:boto:b'ffzdqu-58obj000000000000000000121000falseobj000000000000000000122023-09-22T21:34:33.269Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:51.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-09-22T21:34:51.239 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.239 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.239 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.239 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.239 DEBUG:boto:Method: GET 2023-09-22T21:34:51.239 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.239 DEBUG:boto:Data: 2023-09-22T21:34:51.239 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.240 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:51.240 DEBUG:boto:Port: 8001 2023-09-22T21:34:51.240 DEBUG:boto:Params: {} 2023-09-22T21:34:51.240 DEBUG:boto:Token: None 2023-09-22T21:34:51.240 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.240 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EWYJHtwltZBXM7FQ2t3IBYa+bnI= 2023-09-22T21:34:51.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EWYJHtwltZBXM7FQ2t3IBYa+bnI=', 'Content-Length': '0'} 2023-09-22T21:34:51.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f596efc13a7155c-00650e087b-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.243 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.243 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.244 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.244 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.244 DEBUG:boto:Method: GET 2023-09-22T21:34:51.244 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.244 DEBUG:boto:Data: 2023-09-22T21:34:51.244 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.244 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:51.244 DEBUG:boto:Port: 8000 2023-09-22T21:34:51.244 DEBUG:boto:Params: {} 2023-09-22T21:34:51.244 DEBUG:boto:Token: None 2023-09-22T21:34:51.245 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.245 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EWYJHtwltZBXM7FQ2t3IBYa+bnI= 2023-09-22T21:34:51.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EWYJHtwltZBXM7FQ2t3IBYa+bnI=', 'Content-Length': '0'} 2023-09-22T21:34:51.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025171abe8b5ddfe8-00650e087b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.290 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.291 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.291 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.291 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.291 DEBUG:boto:Method: HEAD 2023-09-22T21:34:51.291 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.291 DEBUG:boto:Data: 2023-09-22T21:34:51.291 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.291 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:51.291 DEBUG:boto:Port: 8001 2023-09-22T21:34:51.292 DEBUG:boto:Params: {} 2023-09-22T21:34:51.292 DEBUG:boto:Token: None 2023-09-22T21:34:51.292 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.292 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YGAK5fcxkWtHhNQKyaW/kEaj+UU= 2023-09-22T21:34:51.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YGAK5fcxkWtHhNQKyaW/kEaj+UU=', 'Content-Length': '0'} 2023-09-22T21:34:51.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099b3b056e12fe759-00650e087b-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.296 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.296 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.296 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.296 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.296 DEBUG:boto:Method: HEAD 2023-09-22T21:34:51.296 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.296 DEBUG:boto:Data: 2023-09-22T21:34:51.296 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.296 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:51.297 DEBUG:boto:Port: 8000 2023-09-22T21:34:51.297 DEBUG:boto:Params: {} 2023-09-22T21:34:51.297 DEBUG:boto:Token: None 2023-09-22T21:34:51.297 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.297 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YGAK5fcxkWtHhNQKyaW/kEaj+UU= 2023-09-22T21:34:51.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YGAK5fcxkWtHhNQKyaW/kEaj+UU=', 'Content-Length': '0'} 2023-09-22T21:34:51.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000019b46756bf0801d-00650e087b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-09-22T21:34:51.300 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.300 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.300 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.301 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.301 DEBUG:boto:Method: GET 2023-09-22T21:34:51.301 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.301 DEBUG:boto:Data: 2023-09-22T21:34:51.301 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.301 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:51.301 DEBUG:boto:Port: 8001 2023-09-22T21:34:51.301 DEBUG:boto:Params: {} 2023-09-22T21:34:51.301 DEBUG:boto:Token: None 2023-09-22T21:34:51.302 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.302 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EWYJHtwltZBXM7FQ2t3IBYa+bnI= 2023-09-22T21:34:51.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EWYJHtwltZBXM7FQ2t3IBYa+bnI=', 'Content-Length': '0'} 2023-09-22T21:34:51.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000543a0e2a0e08f321-00650e087b-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.350 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.351 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.351 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.351 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.351 DEBUG:boto:Method: GET 2023-09-22T21:34:51.351 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.351 DEBUG:boto:Data: 2023-09-22T21:34:51.351 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.351 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:51.351 DEBUG:boto:Port: 8000 2023-09-22T21:34:51.352 DEBUG:boto:Params: {} 2023-09-22T21:34:51.352 DEBUG:boto:Token: None 2023-09-22T21:34:51.352 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:34:51.352 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EWYJHtwltZBXM7FQ2t3IBYa+bnI= 2023-09-22T21:34:51.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EWYJHtwltZBXM7FQ2t3IBYa+bnI=', 'Content-Length': '0'} 2023-09-22T21:34:51.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000882f65c39ec8cc3e-00650e087b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.355 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:51.355 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:51.355 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:34:51.356 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:34:51.356 DEBUG:boto:Method: GET 2023-09-22T21:34:51.356 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:34:51.356 DEBUG:boto:Data: 2023-09-22T21:34:51.356 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.356 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:51.356 DEBUG:boto:Port: 8001 2023-09-22T21:34:51.356 DEBUG:boto:Params: {} 2023-09-22T21:34:51.356 DEBUG:boto:Token: None 2023-09-22T21:34:51.357 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/ 2023-09-22T21:34:51.357 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MMgZkjVARknygwB1Atke4l4WHTU= 2023-09-22T21:34:51.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MMgZkjVARknygwB1Atke4l4WHTU=', 'Content-Length': '0'} 2023-09-22T21:34:51.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071c2750564196105-00650e087b-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.402 DEBUG:boto:b'ffzdqu-58obj000000000000000000131000falseobj000000000000000000132023-09-22T21:34:33.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:51.403 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:51.403 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:51.403 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:34:51.403 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:34:51.403 DEBUG:boto:Method: GET 2023-09-22T21:34:51.404 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:34:51.404 DEBUG:boto:Data: 2023-09-22T21:34:51.404 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.404 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:51.404 DEBUG:boto:Port: 8000 2023-09-22T21:34:51.404 DEBUG:boto:Params: {} 2023-09-22T21:34:51.404 DEBUG:boto:Token: None 2023-09-22T21:34:51.404 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/ 2023-09-22T21:34:51.404 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MMgZkjVARknygwB1Atke4l4WHTU= 2023-09-22T21:34:51.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MMgZkjVARknygwB1Atke4l4WHTU=', 'Content-Length': '0'} 2023-09-22T21:34:51.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029372db3a656bdf5-00650e087b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.409 DEBUG:boto:b'ffzdqu-58obj000000000000000000131000falseobj000000000000000000132023-09-22T21:34:33.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:51.410 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.410 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.410 DEBUG:boto:Method: HEAD 2023-09-22T21:34:51.410 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.410 DEBUG:boto:Data: 2023-09-22T21:34:51.410 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.410 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:51.411 DEBUG:boto:Port: 8001 2023-09-22T21:34:51.411 DEBUG:boto:Params: {} 2023-09-22T21:34:51.411 DEBUG:boto:Token: None 2023-09-22T21:34:51.411 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.411 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9IaUo+DZTizV+f5+WP/21RE/6VY= 2023-09-22T21:34:51.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9IaUo+DZTizV+f5+WP/21RE/6VY=', 'Content-Length': '0'} 2023-09-22T21:34:51.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d0c7855472ee8a9-00650e087b-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.414 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.414 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.414 DEBUG:boto:Method: HEAD 2023-09-22T21:34:51.414 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.415 DEBUG:boto:Data: 2023-09-22T21:34:51.415 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.415 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:51.415 DEBUG:boto:Port: 8000 2023-09-22T21:34:51.415 DEBUG:boto:Params: {} 2023-09-22T21:34:51.415 DEBUG:boto:Token: None 2023-09-22T21:34:51.415 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.415 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9IaUo+DZTizV+f5+WP/21RE/6VY= 2023-09-22T21:34:51.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9IaUo+DZTizV+f5+WP/21RE/6VY=', 'Content-Length': '0'} 2023-09-22T21:34:51.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7005208f01610af-00650e087b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-09-22T21:34:51.418 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.418 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.419 DEBUG:boto:Method: GET 2023-09-22T21:34:51.419 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.419 DEBUG:boto:Data: 2023-09-22T21:34:51.419 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.419 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:51.419 DEBUG:boto:Port: 8001 2023-09-22T21:34:51.419 DEBUG:boto:Params: {} 2023-09-22T21:34:51.419 DEBUG:boto:Token: None 2023-09-22T21:34:51.420 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.420 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BIdh8PmV9EeBbxzyV42UDZPeL4g= 2023-09-22T21:34:51.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BIdh8PmV9EeBbxzyV42UDZPeL4g=', 'Content-Length': '0'} 2023-09-22T21:34:51.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1862877f77fddf8-00650e087b-1ada-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.466 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.467 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.467 DEBUG:boto:Method: GET 2023-09-22T21:34:51.467 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.467 DEBUG:boto:Data: 2023-09-22T21:34:51.467 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.467 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:51.467 DEBUG:boto:Port: 8000 2023-09-22T21:34:51.467 DEBUG:boto:Params: {} 2023-09-22T21:34:51.468 DEBUG:boto:Token: None 2023-09-22T21:34:51.468 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:51.468 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BIdh8PmV9EeBbxzyV42UDZPeL4g= 2023-09-22T21:34:51.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BIdh8PmV9EeBbxzyV42UDZPeL4g=', 'Content-Length': '0'} 2023-09-22T21:34:51.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001703a16602a432bd-00650e087b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.471 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:51.471 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:51.471 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:34:51.471 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:34:51.471 DEBUG:boto:Method: GET 2023-09-22T21:34:51.472 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:34:51.472 DEBUG:boto:Data: 2023-09-22T21:34:51.472 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.472 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:34:51.472 DEBUG:boto:Port: 8001 2023-09-22T21:34:51.472 DEBUG:boto:Params: {} 2023-09-22T21:34:51.472 DEBUG:boto:Token: None 2023-09-22T21:34:51.472 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/ 2023-09-22T21:34:51.472 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MMgZkjVARknygwB1Atke4l4WHTU= 2023-09-22T21:34:51.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MMgZkjVARknygwB1Atke4l4WHTU=', 'Content-Length': '0'} 2023-09-22T21:34:51.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cace33c4a41be4d0-00650e087b-1ada-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.478 DEBUG:boto:b'ffzdqu-58obj000000000000000000131000falseobj000000000000000000132023-09-22T21:34:33.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:51.478 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:51.478 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:51.478 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:34:51.478 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:34:51.478 DEBUG:boto:Method: GET 2023-09-22T21:34:51.479 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:34:51.479 DEBUG:boto:Data: 2023-09-22T21:34:51.479 DEBUG:boto:Headers: {} 2023-09-22T21:34:51.479 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:51.479 DEBUG:boto:Port: 8000 2023-09-22T21:34:51.479 DEBUG:boto:Params: {} 2023-09-22T21:34:51.479 DEBUG:boto:Token: None 2023-09-22T21:34:51.479 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:34:51 GMT /ffzdqu-58/ 2023-09-22T21:34:51.479 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MMgZkjVARknygwB1Atke4l4WHTU= 2023-09-22T21:34:51.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MMgZkjVARknygwB1Atke4l4WHTU=', 'Content-Length': '0'} 2023-09-22T21:34:51.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1447c04c9452b77-00650e087b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:51.530 DEBUG:boto:b'ffzdqu-58obj000000000000000000131000falseobj000000000000000000132023-09-22T21:34:33.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:34:51.531 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-58 zones={a2, a1} 2023-09-22T21:34:51.531 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-09-22T21:34:51.532 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T21:34:51.533 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:34:51.532+0000 7f417e21c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 68775) UID: 0 2023-09-22T21:34:51.533 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:34:51.532+0000 7f4181a63a40 -1 shutting down 2023-09-22T21:34:57.332 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-09-22T21:34:57.632 INFO:tasks.rgw.c1.client.1:Stopped 2023-09-22T21:34:58.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-58', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:34:58.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-58', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:34:58.896 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-58 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:34:58.943 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:34:58.943 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:34:58.997 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:34:58.998 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-58 2023-09-22T21:34:59.005 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 10 2023-09-22T21:34:59.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:34:59.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:34:59.052 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:34:59.088 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:34:59.088 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:34:59.140 INFO:teuthology.orchestra.run.smithi160.stdout:{"objs":["obj00000000000000000007","obj00000000000000000016","obj00000000000000000024","obj00000000000000000020","obj00000000000000000028","obj00000000000000000000","obj00000000000000000032","obj00000000000000000001","obj00000000000000000010","obj00000000000000000019","obj00000000000000000013","obj00000000000000000021","obj00000000000000000030","obj00000000000000000026","obj00000000000000000033","obj00000000000000000004","obj00000000000000000076"]} 2023-09-22T21:34:59.141 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:59.141 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:59.141 DEBUG:boto:Method: HEAD 2023-09-22T21:34:59.141 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:59.141 DEBUG:boto:Data: 2023-09-22T21:34:59.141 DEBUG:boto:Headers: {} 2023-09-22T21:34:59.141 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.142 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.142 DEBUG:boto:Params: {} 2023-09-22T21:34:59.142 DEBUG:boto:Token: None 2023-09-22T21:34:59.142 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/ 2023-09-22T21:34:59.142 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8= 2023-09-22T21:34:59.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8=', 'Content-Length': '0'} 2023-09-22T21:34:59.145 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebc58332b93e8812-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.145 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:34:59.146 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:34:59.146 DEBUG:boto:Method: PUT 2023-09-22T21:34:59.146 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000021 2023-09-22T21:34:59.146 DEBUG:boto:Data: 2023-09-22T21:34:59.146 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:59.146 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.146 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.146 DEBUG:boto:Params: {} 2023-09-22T21:34:59.146 DEBUG:boto:Token: None 2023-09-22T21:34:59.147 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/obj00000000000000000021 2023-09-22T21:34:59.147 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UfuajkVRAUSbsN2QNXpFdNBSfk8= 2023-09-22T21:34:59.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UfuajkVRAUSbsN2QNXpFdNBSfk8='} 2023-09-22T21:34:59.152 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab3a726de7761a9f-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.153 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:59.153 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:59.153 DEBUG:boto:Method: HEAD 2023-09-22T21:34:59.153 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:59.153 DEBUG:boto:Data: 2023-09-22T21:34:59.153 DEBUG:boto:Headers: {} 2023-09-22T21:34:59.153 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.153 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.153 DEBUG:boto:Params: {} 2023-09-22T21:34:59.154 DEBUG:boto:Token: None 2023-09-22T21:34:59.154 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/ 2023-09-22T21:34:59.154 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8= 2023-09-22T21:34:59.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8=', 'Content-Length': '0'} 2023-09-22T21:34:59.156 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000378679aa5d222709-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.157 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:34:59.157 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:34:59.157 DEBUG:boto:Method: PUT 2023-09-22T21:34:59.157 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000019 2023-09-22T21:34:59.157 DEBUG:boto:Data: 2023-09-22T21:34:59.157 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:59.157 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.157 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.157 DEBUG:boto:Params: {} 2023-09-22T21:34:59.157 DEBUG:boto:Token: None 2023-09-22T21:34:59.158 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/obj00000000000000000019 2023-09-22T21:34:59.158 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mg31Htl/4Cq89+8cP3wJCZRjqqw= 2023-09-22T21:34:59.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mg31Htl/4Cq89+8cP3wJCZRjqqw='} 2023-09-22T21:34:59.164 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c7c75094bdec4a6-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.164 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:59.164 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:59.164 DEBUG:boto:Method: HEAD 2023-09-22T21:34:59.164 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:59.164 DEBUG:boto:Data: 2023-09-22T21:34:59.164 DEBUG:boto:Headers: {} 2023-09-22T21:34:59.164 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.165 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.165 DEBUG:boto:Params: {} 2023-09-22T21:34:59.165 DEBUG:boto:Token: None 2023-09-22T21:34:59.165 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/ 2023-09-22T21:34:59.165 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8= 2023-09-22T21:34:59.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8=', 'Content-Length': '0'} 2023-09-22T21:34:59.168 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c408ab251a32661-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.168 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:59.168 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:59.168 DEBUG:boto:Method: PUT 2023-09-22T21:34:59.168 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:59.168 DEBUG:boto:Data: 2023-09-22T21:34:59.169 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:59.169 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.169 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.169 DEBUG:boto:Params: {} 2023-09-22T21:34:59.169 DEBUG:boto:Token: None 2023-09-22T21:34:59.169 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/obj00000000000000000000 2023-09-22T21:34:59.169 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1w+J/3dha7B0iaOjDEJscDAuzV8= 2023-09-22T21:34:59.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1w+J/3dha7B0iaOjDEJscDAuzV8='} 2023-09-22T21:34:59.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e48562094fbcae23-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.215 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:59.215 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:59.215 DEBUG:boto:Method: HEAD 2023-09-22T21:34:59.215 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:59.215 DEBUG:boto:Data: 2023-09-22T21:34:59.215 DEBUG:boto:Headers: {} 2023-09-22T21:34:59.215 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.215 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.215 DEBUG:boto:Params: {} 2023-09-22T21:34:59.216 DEBUG:boto:Token: None 2023-09-22T21:34:59.216 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/ 2023-09-22T21:34:59.216 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8= 2023-09-22T21:34:59.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8=', 'Content-Length': '0'} 2023-09-22T21:34:59.219 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010c22716321fe132-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.219 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:34:59.220 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:34:59.220 DEBUG:boto:Method: PUT 2023-09-22T21:34:59.220 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000026 2023-09-22T21:34:59.220 DEBUG:boto:Data: 2023-09-22T21:34:59.220 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:59.220 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.220 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.220 DEBUG:boto:Params: {} 2023-09-22T21:34:59.220 DEBUG:boto:Token: None 2023-09-22T21:34:59.221 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/obj00000000000000000026 2023-09-22T21:34:59.221 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:z+eSCHZItEUbHZpGBSNeuXFvaiE= 2023-09-22T21:34:59.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:z+eSCHZItEUbHZpGBSNeuXFvaiE='} 2023-09-22T21:34:59.227 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004fd504ab14f832e9-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.228 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:59.228 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:59.228 DEBUG:boto:Method: HEAD 2023-09-22T21:34:59.228 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:59.228 DEBUG:boto:Data: 2023-09-22T21:34:59.228 DEBUG:boto:Headers: {} 2023-09-22T21:34:59.228 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.228 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.228 DEBUG:boto:Params: {} 2023-09-22T21:34:59.229 DEBUG:boto:Token: None 2023-09-22T21:34:59.229 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/ 2023-09-22T21:34:59.229 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8= 2023-09-22T21:34:59.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8=', 'Content-Length': '0'} 2023-09-22T21:34:59.232 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc0e93c3f1d76474-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.232 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:34:59.232 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:34:59.232 DEBUG:boto:Method: PUT 2023-09-22T21:34:59.233 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000032 2023-09-22T21:34:59.233 DEBUG:boto:Data: 2023-09-22T21:34:59.233 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:59.233 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.233 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.233 DEBUG:boto:Params: {} 2023-09-22T21:34:59.233 DEBUG:boto:Token: None 2023-09-22T21:34:59.233 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/obj00000000000000000032 2023-09-22T21:34:59.234 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TZwvSTpbL/qDQDfOdSmM+UbzDTk= 2023-09-22T21:34:59.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TZwvSTpbL/qDQDfOdSmM+UbzDTk='} 2023-09-22T21:34:59.278 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e593980b637b51cc-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.279 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:59.279 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:59.279 DEBUG:boto:Method: HEAD 2023-09-22T21:34:59.279 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:59.279 DEBUG:boto:Data: 2023-09-22T21:34:59.279 DEBUG:boto:Headers: {} 2023-09-22T21:34:59.280 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.280 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.280 DEBUG:boto:Params: {} 2023-09-22T21:34:59.280 DEBUG:boto:Token: None 2023-09-22T21:34:59.280 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/ 2023-09-22T21:34:59.280 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8= 2023-09-22T21:34:59.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8=', 'Content-Length': '0'} 2023-09-22T21:34:59.285 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029956a22fa814511-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.286 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:59.286 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:59.286 DEBUG:boto:Method: PUT 2023-09-22T21:34:59.286 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:59.286 DEBUG:boto:Data: 2023-09-22T21:34:59.286 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:59.286 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.287 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.287 DEBUG:boto:Params: {} 2023-09-22T21:34:59.287 DEBUG:boto:Token: None 2023-09-22T21:34:59.287 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/obj00000000000000000010 2023-09-22T21:34:59.287 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LjIkdOQdInPBSssxuLLiX6BJhkc= 2023-09-22T21:34:59.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LjIkdOQdInPBSssxuLLiX6BJhkc='} 2023-09-22T21:34:59.301 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8b688ceacc7f16f-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.301 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:59.301 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:59.301 DEBUG:boto:Method: HEAD 2023-09-22T21:34:59.302 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:59.302 DEBUG:boto:Data: 2023-09-22T21:34:59.302 DEBUG:boto:Headers: {} 2023-09-22T21:34:59.302 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.302 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.302 DEBUG:boto:Params: {} 2023-09-22T21:34:59.302 DEBUG:boto:Token: None 2023-09-22T21:34:59.302 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/ 2023-09-22T21:34:59.303 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8= 2023-09-22T21:34:59.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8=', 'Content-Length': '0'} 2023-09-22T21:34:59.307 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b5f1df3d5d44b52-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.307 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:34:59.307 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:34:59.307 DEBUG:boto:Method: PUT 2023-09-22T21:34:59.308 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000030 2023-09-22T21:34:59.308 DEBUG:boto:Data: 2023-09-22T21:34:59.308 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:59.308 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.308 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.308 DEBUG:boto:Params: {} 2023-09-22T21:34:59.308 DEBUG:boto:Token: None 2023-09-22T21:34:59.309 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/obj00000000000000000030 2023-09-22T21:34:59.309 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S9qGT92s0Nnz5hGhmN2FGwMS9VA= 2023-09-22T21:34:59.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9qGT92s0Nnz5hGhmN2FGwMS9VA='} 2023-09-22T21:34:59.354 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e59e7a8af8884ad8-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.355 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:59.355 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:59.355 DEBUG:boto:Method: HEAD 2023-09-22T21:34:59.355 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:59.355 DEBUG:boto:Data: 2023-09-22T21:34:59.356 DEBUG:boto:Headers: {} 2023-09-22T21:34:59.356 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.356 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.356 DEBUG:boto:Params: {} 2023-09-22T21:34:59.356 DEBUG:boto:Token: None 2023-09-22T21:34:59.356 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/ 2023-09-22T21:34:59.356 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8= 2023-09-22T21:34:59.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8=', 'Content-Length': '0'} 2023-09-22T21:34:59.361 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000120ad4d8c20ce5e2-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.361 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:59.361 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:59.361 DEBUG:boto:Method: PUT 2023-09-22T21:34:59.361 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:59.362 DEBUG:boto:Data: 2023-09-22T21:34:59.362 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:59.362 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.362 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.362 DEBUG:boto:Params: {} 2023-09-22T21:34:59.362 DEBUG:boto:Token: None 2023-09-22T21:34:59.362 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/obj00000000000000000013 2023-09-22T21:34:59.363 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+uUJVSpkS3cyigOe9tdtPc7XNHY= 2023-09-22T21:34:59.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+uUJVSpkS3cyigOe9tdtPc7XNHY='} 2023-09-22T21:34:59.377 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000005b1da13a80358d-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.378 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:59.378 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:59.378 DEBUG:boto:Method: HEAD 2023-09-22T21:34:59.378 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:59.378 DEBUG:boto:Data: 2023-09-22T21:34:59.378 DEBUG:boto:Headers: {} 2023-09-22T21:34:59.378 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.378 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.378 DEBUG:boto:Params: {} 2023-09-22T21:34:59.379 DEBUG:boto:Token: None 2023-09-22T21:34:59.379 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/ 2023-09-22T21:34:59.379 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8= 2023-09-22T21:34:59.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8=', 'Content-Length': '0'} 2023-09-22T21:34:59.384 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019676d2cc4c3f1b7-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.384 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:34:59.385 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:34:59.385 DEBUG:boto:Method: PUT 2023-09-22T21:34:59.385 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000007 2023-09-22T21:34:59.385 DEBUG:boto:Data: 2023-09-22T21:34:59.385 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:59.385 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.385 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.386 DEBUG:boto:Params: {} 2023-09-22T21:34:59.386 DEBUG:boto:Token: None 2023-09-22T21:34:59.386 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/obj00000000000000000007 2023-09-22T21:34:59.386 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CWhG17DP8SxvqrQiiTCnvQh2+tc= 2023-09-22T21:34:59.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CWhG17DP8SxvqrQiiTCnvQh2+tc='} 2023-09-22T21:34:59.430 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052d88af3c1b4621b-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.431 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:59.431 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:59.431 DEBUG:boto:Method: HEAD 2023-09-22T21:34:59.431 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:59.431 DEBUG:boto:Data: 2023-09-22T21:34:59.431 DEBUG:boto:Headers: {} 2023-09-22T21:34:59.431 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.431 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.431 DEBUG:boto:Params: {} 2023-09-22T21:34:59.432 DEBUG:boto:Token: None 2023-09-22T21:34:59.432 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/ 2023-09-22T21:34:59.432 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8= 2023-09-22T21:34:59.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8=', 'Content-Length': '0'} 2023-09-22T21:34:59.437 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007a1184e24013a05-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.437 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:34:59.437 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:34:59.437 DEBUG:boto:Method: PUT 2023-09-22T21:34:59.437 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000033 2023-09-22T21:34:59.438 DEBUG:boto:Data: 2023-09-22T21:34:59.438 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:59.438 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.438 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.438 DEBUG:boto:Params: {} 2023-09-22T21:34:59.438 DEBUG:boto:Token: None 2023-09-22T21:34:59.438 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/obj00000000000000000033 2023-09-22T21:34:59.438 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CnYt0LxJJOScAjDOrTk1FzUpiHQ= 2023-09-22T21:34:59.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CnYt0LxJJOScAjDOrTk1FzUpiHQ='} 2023-09-22T21:34:59.451 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a68c4829ed8e824-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.451 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:59.452 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:59.452 DEBUG:boto:Method: HEAD 2023-09-22T21:34:59.452 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:59.452 DEBUG:boto:Data: 2023-09-22T21:34:59.452 DEBUG:boto:Headers: {} 2023-09-22T21:34:59.452 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.452 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.452 DEBUG:boto:Params: {} 2023-09-22T21:34:59.452 DEBUG:boto:Token: None 2023-09-22T21:34:59.453 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/ 2023-09-22T21:34:59.453 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8= 2023-09-22T21:34:59.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8=', 'Content-Length': '0'} 2023-09-22T21:34:59.458 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f5073141825d7bb-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.458 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:34:59.458 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:34:59.458 DEBUG:boto:Method: PUT 2023-09-22T21:34:59.458 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000004 2023-09-22T21:34:59.458 DEBUG:boto:Data: 2023-09-22T21:34:59.458 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:59.459 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.459 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.459 DEBUG:boto:Params: {} 2023-09-22T21:34:59.459 DEBUG:boto:Token: None 2023-09-22T21:34:59.459 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/obj00000000000000000004 2023-09-22T21:34:59.459 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:a9aKRzaTjQ+3J4cXB/O4JNz/R4c= 2023-09-22T21:34:59.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:a9aKRzaTjQ+3J4cXB/O4JNz/R4c='} 2023-09-22T21:34:59.502 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca724bfc81ef8ca2-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.503 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:59.503 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:59.503 DEBUG:boto:Method: HEAD 2023-09-22T21:34:59.503 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:59.503 DEBUG:boto:Data: 2023-09-22T21:34:59.503 DEBUG:boto:Headers: {} 2023-09-22T21:34:59.503 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.503 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.504 DEBUG:boto:Params: {} 2023-09-22T21:34:59.504 DEBUG:boto:Token: None 2023-09-22T21:34:59.504 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/ 2023-09-22T21:34:59.504 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8= 2023-09-22T21:34:59.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8=', 'Content-Length': '0'} 2023-09-22T21:34:59.510 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afeb69f35146fc86-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.510 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:34:59.510 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:34:59.510 DEBUG:boto:Method: PUT 2023-09-22T21:34:59.511 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000020 2023-09-22T21:34:59.511 DEBUG:boto:Data: 2023-09-22T21:34:59.511 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:59.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.511 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.511 DEBUG:boto:Params: {} 2023-09-22T21:34:59.511 DEBUG:boto:Token: None 2023-09-22T21:34:59.511 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/obj00000000000000000020 2023-09-22T21:34:59.511 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vPFq5wgjL28ZTvgLbLLW2aFLmy4= 2023-09-22T21:34:59.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vPFq5wgjL28ZTvgLbLLW2aFLmy4='} 2023-09-22T21:34:59.524 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5bf85478a92e4c8-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.524 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:59.524 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:59.525 DEBUG:boto:Method: HEAD 2023-09-22T21:34:59.525 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:59.525 DEBUG:boto:Data: 2023-09-22T21:34:59.525 DEBUG:boto:Headers: {} 2023-09-22T21:34:59.525 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.525 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.525 DEBUG:boto:Params: {} 2023-09-22T21:34:59.525 DEBUG:boto:Token: None 2023-09-22T21:34:59.525 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/ 2023-09-22T21:34:59.526 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8= 2023-09-22T21:34:59.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8=', 'Content-Length': '0'} 2023-09-22T21:34:59.531 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ff0a5202011285f-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.531 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:34:59.531 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:34:59.531 DEBUG:boto:Method: PUT 2023-09-22T21:34:59.531 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000076 2023-09-22T21:34:59.532 DEBUG:boto:Data: 2023-09-22T21:34:59.532 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:59.532 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.532 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.532 DEBUG:boto:Params: {} 2023-09-22T21:34:59.532 DEBUG:boto:Token: None 2023-09-22T21:34:59.532 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/obj00000000000000000076 2023-09-22T21:34:59.532 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SXZKxK3XfgYKs6iaM6J/lVGDHmk= 2023-09-22T21:34:59.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SXZKxK3XfgYKs6iaM6J/lVGDHmk='} 2023-09-22T21:34:59.575 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077930a054ea378fc-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.575 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:59.575 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:59.575 DEBUG:boto:Method: HEAD 2023-09-22T21:34:59.575 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:59.575 DEBUG:boto:Data: 2023-09-22T21:34:59.575 DEBUG:boto:Headers: {} 2023-09-22T21:34:59.576 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.576 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.576 DEBUG:boto:Params: {} 2023-09-22T21:34:59.576 DEBUG:boto:Token: None 2023-09-22T21:34:59.576 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/ 2023-09-22T21:34:59.576 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8= 2023-09-22T21:34:59.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8=', 'Content-Length': '0'} 2023-09-22T21:34:59.581 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037ac02389cdc4a3b-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.582 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:34:59.582 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:34:59.582 DEBUG:boto:Method: PUT 2023-09-22T21:34:59.582 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000024 2023-09-22T21:34:59.582 DEBUG:boto:Data: 2023-09-22T21:34:59.582 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:59.583 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.583 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.583 DEBUG:boto:Params: {} 2023-09-22T21:34:59.583 DEBUG:boto:Token: None 2023-09-22T21:34:59.583 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/obj00000000000000000024 2023-09-22T21:34:59.583 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QMJE3NiPQTpNIyaPfJlXssK+wyM= 2023-09-22T21:34:59.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QMJE3NiPQTpNIyaPfJlXssK+wyM='} 2023-09-22T21:34:59.595 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000464c12bc3f94ed0b-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.595 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:59.596 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:59.596 DEBUG:boto:Method: HEAD 2023-09-22T21:34:59.596 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:59.596 DEBUG:boto:Data: 2023-09-22T21:34:59.596 DEBUG:boto:Headers: {} 2023-09-22T21:34:59.596 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.596 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.596 DEBUG:boto:Params: {} 2023-09-22T21:34:59.596 DEBUG:boto:Token: None 2023-09-22T21:34:59.597 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/ 2023-09-22T21:34:59.597 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8= 2023-09-22T21:34:59.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8=', 'Content-Length': '0'} 2023-09-22T21:34:59.603 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef9328958e520397-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.603 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:34:59.603 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:34:59.603 DEBUG:boto:Method: PUT 2023-09-22T21:34:59.603 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000016 2023-09-22T21:34:59.603 DEBUG:boto:Data: 2023-09-22T21:34:59.604 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:59.604 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.604 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.604 DEBUG:boto:Params: {} 2023-09-22T21:34:59.604 DEBUG:boto:Token: None 2023-09-22T21:34:59.604 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/obj00000000000000000016 2023-09-22T21:34:59.604 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YDgI56qy8dKoDtxGjKVSRVk2qTM= 2023-09-22T21:34:59.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YDgI56qy8dKoDtxGjKVSRVk2qTM='} 2023-09-22T21:34:59.646 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000432bde6a5d7fd422-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.647 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:59.647 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:59.647 DEBUG:boto:Method: HEAD 2023-09-22T21:34:59.647 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:59.647 DEBUG:boto:Data: 2023-09-22T21:34:59.647 DEBUG:boto:Headers: {} 2023-09-22T21:34:59.647 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.647 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.647 DEBUG:boto:Params: {} 2023-09-22T21:34:59.648 DEBUG:boto:Token: None 2023-09-22T21:34:59.648 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/ 2023-09-22T21:34:59.648 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8= 2023-09-22T21:34:59.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8=', 'Content-Length': '0'} 2023-09-22T21:34:59.653 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083062432844ea009-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.653 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:34:59.653 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:34:59.653 DEBUG:boto:Method: PUT 2023-09-22T21:34:59.654 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000028 2023-09-22T21:34:59.654 DEBUG:boto:Data: 2023-09-22T21:34:59.654 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:59.654 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.654 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.654 DEBUG:boto:Params: {} 2023-09-22T21:34:59.654 DEBUG:boto:Token: None 2023-09-22T21:34:59.654 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/obj00000000000000000028 2023-09-22T21:34:59.655 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HLuhK7ZCwK7m9Lj1/1eAHyJOHgk= 2023-09-22T21:34:59.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HLuhK7ZCwK7m9Lj1/1eAHyJOHgk='} 2023-09-22T21:34:59.667 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000007f46c7c602ce53-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-58', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:34:59.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-58', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:34:59.667 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-58 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:34:59.712 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:34:59.712 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:34:59.770 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:34:59.770 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-58 2023-09-22T21:34:59.781 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 23 2023-09-22T21:34:59.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:34:59.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:34:59.844 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:34:59.880 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:34:59.880 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:34:59.933 INFO:teuthology.orchestra.run.smithi160.stdout:{"objs":["obj00000000000000000005","obj00000000000000000010","obj00000000000000000016","obj00000000000000000021","obj00000000000000000004","obj00000000000000000025","obj00000000000000000030","obj00000000000000000015","obj00000000000000000019","obj00000000000000000003","obj00000000000000000009","obj00000000000000000014","obj00000000000000000012","obj00000000000000000002","obj00000000000000000000","obj00000000000000000006","obj00000000000000000011","obj00000000000000000001","obj00000000000000000007"]} 2023-09-22T21:34:59.933 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:59.933 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:59.934 DEBUG:boto:Method: HEAD 2023-09-22T21:34:59.934 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:59.934 DEBUG:boto:Data: 2023-09-22T21:34:59.934 DEBUG:boto:Headers: {} 2023-09-22T21:34:59.934 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.934 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.934 DEBUG:boto:Params: {} 2023-09-22T21:34:59.934 DEBUG:boto:Token: None 2023-09-22T21:34:59.935 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/ 2023-09-22T21:34:59.935 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8= 2023-09-22T21:34:59.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8=', 'Content-Length': '0'} 2023-09-22T21:34:59.938 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e28fbcee8b3e62f8-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.938 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:34:59.938 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:34:59.938 DEBUG:boto:Method: PUT 2023-09-22T21:34:59.939 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000007 2023-09-22T21:34:59.939 DEBUG:boto:Data: 2023-09-22T21:34:59.939 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:59.939 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.939 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.939 DEBUG:boto:Params: {} 2023-09-22T21:34:59.939 DEBUG:boto:Token: None 2023-09-22T21:34:59.939 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/obj00000000000000000007 2023-09-22T21:34:59.939 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CWhG17DP8SxvqrQiiTCnvQh2+tc= 2023-09-22T21:34:59.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CWhG17DP8SxvqrQiiTCnvQh2+tc='} 2023-09-22T21:34:59.947 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000071e9b5bb058ef624-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.947 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:59.948 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:59.948 DEBUG:boto:Method: HEAD 2023-09-22T21:34:59.948 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:59.948 DEBUG:boto:Data: 2023-09-22T21:34:59.948 DEBUG:boto:Headers: {} 2023-09-22T21:34:59.948 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.948 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.948 DEBUG:boto:Params: {} 2023-09-22T21:34:59.948 DEBUG:boto:Token: None 2023-09-22T21:34:59.949 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/ 2023-09-22T21:34:59.949 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8= 2023-09-22T21:34:59.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8=', 'Content-Length': '0'} 2023-09-22T21:34:59.951 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048941473c14654ed-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.951 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:34:59.951 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:34:59.952 DEBUG:boto:Method: PUT 2023-09-22T21:34:59.952 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000004 2023-09-22T21:34:59.952 DEBUG:boto:Data: 2023-09-22T21:34:59.952 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:34:59.952 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.952 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.952 DEBUG:boto:Params: {} 2023-09-22T21:34:59.952 DEBUG:boto:Token: None 2023-09-22T21:34:59.952 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/obj00000000000000000004 2023-09-22T21:34:59.953 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:a9aKRzaTjQ+3J4cXB/O4JNz/R4c= 2023-09-22T21:34:59.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:a9aKRzaTjQ+3J4cXB/O4JNz/R4c='} 2023-09-22T21:34:59.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000feffb95111c41dc2-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:34:59.995 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:34:59.995 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:34:59.995 DEBUG:boto:Method: HEAD 2023-09-22T21:34:59.995 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:34:59.995 DEBUG:boto:Data: 2023-09-22T21:34:59.995 DEBUG:boto:Headers: {} 2023-09-22T21:34:59.995 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:34:59.995 DEBUG:boto:Port: 8000 2023-09-22T21:34:59.995 DEBUG:boto:Params: {} 2023-09-22T21:34:59.996 DEBUG:boto:Token: None 2023-09-22T21:34:59.996 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:34:59 GMT /ffzdqu-58/ 2023-09-22T21:34:59.996 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8= 2023-09-22T21:34:59.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:34:59 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LVanDSqyOH/kaCqrYJMfwA0f2E8=', 'Content-Length': '0'} 2023-09-22T21:35:00.000 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018d39cc4f6647b0f-00650e0883-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.000 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:35:00.000 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:35:00.000 DEBUG:boto:Method: PUT 2023-09-22T21:35:00.000 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000030 2023-09-22T21:35:00.001 DEBUG:boto:Data: 2023-09-22T21:35:00.001 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:00.001 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.001 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.001 DEBUG:boto:Params: {} 2023-09-22T21:35:00.001 DEBUG:boto:Token: None 2023-09-22T21:35:00.001 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/obj00000000000000000030 2023-09-22T21:35:00.001 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8025sGRl8iXyFMrDqcgXoPfXky8= 2023-09-22T21:35:00.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8025sGRl8iXyFMrDqcgXoPfXky8='} 2023-09-22T21:35:00.013 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033bdb32b8785c571-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.013 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:00.013 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:00.013 DEBUG:boto:Method: HEAD 2023-09-22T21:35:00.013 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:00.013 DEBUG:boto:Data: 2023-09-22T21:35:00.013 DEBUG:boto:Headers: {} 2023-09-22T21:35:00.013 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.014 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.014 DEBUG:boto:Params: {} 2023-09-22T21:35:00.014 DEBUG:boto:Token: None 2023-09-22T21:35:00.014 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/ 2023-09-22T21:35:00.014 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50= 2023-09-22T21:35:00.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50=', 'Content-Length': '0'} 2023-09-22T21:35:00.019 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8187da81f650ac4-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.019 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:35:00.019 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:35:00.019 DEBUG:boto:Method: PUT 2023-09-22T21:35:00.019 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000021 2023-09-22T21:35:00.019 DEBUG:boto:Data: 2023-09-22T21:35:00.019 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:00.019 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.020 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.020 DEBUG:boto:Params: {} 2023-09-22T21:35:00.020 DEBUG:boto:Token: None 2023-09-22T21:35:00.020 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/obj00000000000000000021 2023-09-22T21:35:00.020 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HVrgHj44UWkPEdjXs4EG00URElw= 2023-09-22T21:35:00.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HVrgHj44UWkPEdjXs4EG00URElw='} 2023-09-22T21:35:00.066 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e9ef83ec2fc1a84f-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.067 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:00.067 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:00.067 DEBUG:boto:Method: HEAD 2023-09-22T21:35:00.067 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:00.067 DEBUG:boto:Data: 2023-09-22T21:35:00.067 DEBUG:boto:Headers: {} 2023-09-22T21:35:00.067 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.067 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.067 DEBUG:boto:Params: {} 2023-09-22T21:35:00.068 DEBUG:boto:Token: None 2023-09-22T21:35:00.068 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/ 2023-09-22T21:35:00.068 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50= 2023-09-22T21:35:00.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50=', 'Content-Length': '0'} 2023-09-22T21:35:00.072 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cdbe8678f7e62d81-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.073 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:35:00.073 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:35:00.073 DEBUG:boto:Method: PUT 2023-09-22T21:35:00.073 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011 2023-09-22T21:35:00.073 DEBUG:boto:Data: 2023-09-22T21:35:00.073 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:00.073 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.073 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.074 DEBUG:boto:Params: {} 2023-09-22T21:35:00.074 DEBUG:boto:Token: None 2023-09-22T21:35:00.074 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/obj00000000000000000011 2023-09-22T21:35:00.074 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q1zZ/mCoufxoG3OuvBf5MxzG3+s= 2023-09-22T21:35:00.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q1zZ/mCoufxoG3OuvBf5MxzG3+s='} 2023-09-22T21:35:00.087 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f9d23160c631745-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.088 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:00.088 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:00.088 DEBUG:boto:Method: HEAD 2023-09-22T21:35:00.088 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:00.088 DEBUG:boto:Data: 2023-09-22T21:35:00.088 DEBUG:boto:Headers: {} 2023-09-22T21:35:00.088 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.088 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.088 DEBUG:boto:Params: {} 2023-09-22T21:35:00.089 DEBUG:boto:Token: None 2023-09-22T21:35:00.089 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/ 2023-09-22T21:35:00.089 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50= 2023-09-22T21:35:00.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50=', 'Content-Length': '0'} 2023-09-22T21:35:00.093 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000edfe43b2e827f0ff-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.094 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:35:00.094 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:35:00.094 DEBUG:boto:Method: PUT 2023-09-22T21:35:00.094 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000009 2023-09-22T21:35:00.094 DEBUG:boto:Data: 2023-09-22T21:35:00.094 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:00.094 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.094 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.095 DEBUG:boto:Params: {} 2023-09-22T21:35:00.095 DEBUG:boto:Token: None 2023-09-22T21:35:00.095 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/obj00000000000000000009 2023-09-22T21:35:00.095 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Rv3Ag4XFOf1We1uJqIkV9N2x410= 2023-09-22T21:35:00.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rv3Ag4XFOf1We1uJqIkV9N2x410='} 2023-09-22T21:35:00.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004516f123b6082c39-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.139 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:00.139 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:00.139 DEBUG:boto:Method: HEAD 2023-09-22T21:35:00.139 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:00.139 DEBUG:boto:Data: 2023-09-22T21:35:00.139 DEBUG:boto:Headers: {} 2023-09-22T21:35:00.139 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.139 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.140 DEBUG:boto:Params: {} 2023-09-22T21:35:00.140 DEBUG:boto:Token: None 2023-09-22T21:35:00.140 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/ 2023-09-22T21:35:00.140 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50= 2023-09-22T21:35:00.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50=', 'Content-Length': '0'} 2023-09-22T21:35:00.145 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b37d37fdf5dbeeb8-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.145 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:35:00.145 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:35:00.146 DEBUG:boto:Method: PUT 2023-09-22T21:35:00.146 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000 2023-09-22T21:35:00.146 DEBUG:boto:Data: 2023-09-22T21:35:00.146 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:00.146 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.146 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.146 DEBUG:boto:Params: {} 2023-09-22T21:35:00.147 DEBUG:boto:Token: None 2023-09-22T21:35:00.147 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/obj00000000000000000000 2023-09-22T21:35:00.147 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:k4AWPWuSW8sUYF7xKfbDlUsoJGI= 2023-09-22T21:35:00.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:k4AWPWuSW8sUYF7xKfbDlUsoJGI='} 2023-09-22T21:35:00.162 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a1e0c2e5ceb67a12-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.162 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:00.162 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:00.163 DEBUG:boto:Method: HEAD 2023-09-22T21:35:00.163 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:00.163 DEBUG:boto:Data: 2023-09-22T21:35:00.163 DEBUG:boto:Headers: {} 2023-09-22T21:35:00.163 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.163 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.163 DEBUG:boto:Params: {} 2023-09-22T21:35:00.163 DEBUG:boto:Token: None 2023-09-22T21:35:00.164 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/ 2023-09-22T21:35:00.164 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50= 2023-09-22T21:35:00.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50=', 'Content-Length': '0'} 2023-09-22T21:35:00.169 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cde1e74e1cad2a37-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.169 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:35:00.169 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:35:00.170 DEBUG:boto:Method: PUT 2023-09-22T21:35:00.170 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000019 2023-09-22T21:35:00.170 DEBUG:boto:Data: 2023-09-22T21:35:00.170 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:00.170 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.170 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.170 DEBUG:boto:Params: {} 2023-09-22T21:35:00.170 DEBUG:boto:Token: None 2023-09-22T21:35:00.171 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/obj00000000000000000019 2023-09-22T21:35:00.171 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+4859kQrhMx8126FqS6ZEGWDjE0= 2023-09-22T21:35:00.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+4859kQrhMx8126FqS6ZEGWDjE0='} 2023-09-22T21:35:00.218 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033ca5d518640667a-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.219 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:00.219 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:00.219 DEBUG:boto:Method: HEAD 2023-09-22T21:35:00.219 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:00.219 DEBUG:boto:Data: 2023-09-22T21:35:00.220 DEBUG:boto:Headers: {} 2023-09-22T21:35:00.220 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.220 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.220 DEBUG:boto:Params: {} 2023-09-22T21:35:00.220 DEBUG:boto:Token: None 2023-09-22T21:35:00.220 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/ 2023-09-22T21:35:00.221 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50= 2023-09-22T21:35:00.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50=', 'Content-Length': '0'} 2023-09-22T21:35:00.226 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e83e676d10fa6cdd-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.226 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:35:00.226 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:35:00.227 DEBUG:boto:Method: PUT 2023-09-22T21:35:00.227 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002 2023-09-22T21:35:00.227 DEBUG:boto:Data: 2023-09-22T21:35:00.227 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:00.227 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.227 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.227 DEBUG:boto:Params: {} 2023-09-22T21:35:00.227 DEBUG:boto:Token: None 2023-09-22T21:35:00.228 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/obj00000000000000000002 2023-09-22T21:35:00.228 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/7LUvMabR5L2YhBRdvjw9luNhYA= 2023-09-22T21:35:00.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/7LUvMabR5L2YhBRdvjw9luNhYA='} 2023-09-22T21:35:00.244 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae66d7e80a38e124-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.245 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:00.245 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:00.245 DEBUG:boto:Method: HEAD 2023-09-22T21:35:00.245 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:00.245 DEBUG:boto:Data: 2023-09-22T21:35:00.245 DEBUG:boto:Headers: {} 2023-09-22T21:35:00.245 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.245 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.245 DEBUG:boto:Params: {} 2023-09-22T21:35:00.246 DEBUG:boto:Token: None 2023-09-22T21:35:00.246 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/ 2023-09-22T21:35:00.246 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50= 2023-09-22T21:35:00.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50=', 'Content-Length': '0'} 2023-09-22T21:35:00.252 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f746aa6623a384e8-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.252 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:35:00.252 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:35:00.252 DEBUG:boto:Method: PUT 2023-09-22T21:35:00.252 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012 2023-09-22T21:35:00.252 DEBUG:boto:Data: 2023-09-22T21:35:00.252 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:00.253 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.253 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.253 DEBUG:boto:Params: {} 2023-09-22T21:35:00.253 DEBUG:boto:Token: None 2023-09-22T21:35:00.253 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/obj00000000000000000012 2023-09-22T21:35:00.253 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Z34Kjgt89scFC0zbgPiVvcZ2+mM= 2023-09-22T21:35:00.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Z34Kjgt89scFC0zbgPiVvcZ2+mM='} 2023-09-22T21:35:00.299 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077484ae6f93b9e5e-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.299 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:00.299 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:00.299 DEBUG:boto:Method: HEAD 2023-09-22T21:35:00.299 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:00.299 DEBUG:boto:Data: 2023-09-22T21:35:00.299 DEBUG:boto:Headers: {} 2023-09-22T21:35:00.300 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.300 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.300 DEBUG:boto:Params: {} 2023-09-22T21:35:00.300 DEBUG:boto:Token: None 2023-09-22T21:35:00.300 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/ 2023-09-22T21:35:00.300 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50= 2023-09-22T21:35:00.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50=', 'Content-Length': '0'} 2023-09-22T21:35:00.604 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f77734cb88a2a5f-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.604 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:35:00.604 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:35:00.604 DEBUG:boto:Method: PUT 2023-09-22T21:35:00.605 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006 2023-09-22T21:35:00.605 DEBUG:boto:Data: 2023-09-22T21:35:00.605 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:00.605 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.605 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.605 DEBUG:boto:Params: {} 2023-09-22T21:35:00.605 DEBUG:boto:Token: None 2023-09-22T21:35:00.605 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/obj00000000000000000006 2023-09-22T21:35:00.605 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9iRoA5Rg3/A2c4sEE+/9kDkwPWk= 2023-09-22T21:35:00.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9iRoA5Rg3/A2c4sEE+/9kDkwPWk='} 2023-09-22T21:35:00.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d5fd20ace2e764fe-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.622 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:00.622 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:00.622 DEBUG:boto:Method: HEAD 2023-09-22T21:35:00.622 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:00.622 DEBUG:boto:Data: 2023-09-22T21:35:00.622 DEBUG:boto:Headers: {} 2023-09-22T21:35:00.623 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.623 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.623 DEBUG:boto:Params: {} 2023-09-22T21:35:00.623 DEBUG:boto:Token: None 2023-09-22T21:35:00.623 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/ 2023-09-22T21:35:00.623 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50= 2023-09-22T21:35:00.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50=', 'Content-Length': '0'} 2023-09-22T21:35:00.628 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e060685931476d03-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.629 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:35:00.629 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:35:00.629 DEBUG:boto:Method: PUT 2023-09-22T21:35:00.629 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000014 2023-09-22T21:35:00.629 DEBUG:boto:Data: 2023-09-22T21:35:00.629 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:00.629 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.629 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.629 DEBUG:boto:Params: {} 2023-09-22T21:35:00.629 DEBUG:boto:Token: None 2023-09-22T21:35:00.630 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/obj00000000000000000014 2023-09-22T21:35:00.630 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ihkV7cDHxVNRWnUiBT26lCGRq0s= 2023-09-22T21:35:00.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ihkV7cDHxVNRWnUiBT26lCGRq0s='} 2023-09-22T21:35:00.674 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aa4bfc5d1c1d45c0-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.675 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:00.675 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:00.675 DEBUG:boto:Method: HEAD 2023-09-22T21:35:00.675 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:00.675 DEBUG:boto:Data: 2023-09-22T21:35:00.675 DEBUG:boto:Headers: {} 2023-09-22T21:35:00.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.675 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.675 DEBUG:boto:Params: {} 2023-09-22T21:35:00.676 DEBUG:boto:Token: None 2023-09-22T21:35:00.676 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/ 2023-09-22T21:35:00.676 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50= 2023-09-22T21:35:00.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50=', 'Content-Length': '0'} 2023-09-22T21:35:00.681 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000286de1726a216f5d-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.681 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:35:00.682 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:35:00.682 DEBUG:boto:Method: PUT 2023-09-22T21:35:00.682 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000025 2023-09-22T21:35:00.682 DEBUG:boto:Data: 2023-09-22T21:35:00.682 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:00.682 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.682 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.682 DEBUG:boto:Params: {} 2023-09-22T21:35:00.682 DEBUG:boto:Token: None 2023-09-22T21:35:00.683 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/obj00000000000000000025 2023-09-22T21:35:00.683 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3ZQu9PbuFdFK8LFvuD7U5dEOvFU= 2023-09-22T21:35:00.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3ZQu9PbuFdFK8LFvuD7U5dEOvFU='} 2023-09-22T21:35:00.695 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000002a40be93583f0b5-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.695 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:00.695 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:00.695 DEBUG:boto:Method: HEAD 2023-09-22T21:35:00.696 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:00.696 DEBUG:boto:Data: 2023-09-22T21:35:00.696 DEBUG:boto:Headers: {} 2023-09-22T21:35:00.696 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.696 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.696 DEBUG:boto:Params: {} 2023-09-22T21:35:00.696 DEBUG:boto:Token: None 2023-09-22T21:35:00.696 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/ 2023-09-22T21:35:00.696 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50= 2023-09-22T21:35:00.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50=', 'Content-Length': '0'} 2023-09-22T21:35:00.702 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7a75cfadaf86d30-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.702 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:35:00.702 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:35:00.702 DEBUG:boto:Method: PUT 2023-09-22T21:35:00.702 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000015 2023-09-22T21:35:00.702 DEBUG:boto:Data: 2023-09-22T21:35:00.702 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:00.703 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.703 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.703 DEBUG:boto:Params: {} 2023-09-22T21:35:00.703 DEBUG:boto:Token: None 2023-09-22T21:35:00.703 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/obj00000000000000000015 2023-09-22T21:35:00.703 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:17BZk4IvZd3KnE2vTs3xzl4vITk= 2023-09-22T21:35:00.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:17BZk4IvZd3KnE2vTs3xzl4vITk='} 2023-09-22T21:35:00.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a020a2f6fcd8c25-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.751 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:00.751 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:00.751 DEBUG:boto:Method: HEAD 2023-09-22T21:35:00.751 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:00.751 DEBUG:boto:Data: 2023-09-22T21:35:00.751 DEBUG:boto:Headers: {} 2023-09-22T21:35:00.751 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.751 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.751 DEBUG:boto:Params: {} 2023-09-22T21:35:00.752 DEBUG:boto:Token: None 2023-09-22T21:35:00.752 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/ 2023-09-22T21:35:00.752 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50= 2023-09-22T21:35:00.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50=', 'Content-Length': '0'} 2023-09-22T21:35:00.758 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac1d2e51af25d5ca-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.758 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:35:00.758 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:35:00.758 DEBUG:boto:Method: PUT 2023-09-22T21:35:00.758 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010 2023-09-22T21:35:00.758 DEBUG:boto:Data: 2023-09-22T21:35:00.758 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:00.759 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.759 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.759 DEBUG:boto:Params: {} 2023-09-22T21:35:00.759 DEBUG:boto:Token: None 2023-09-22T21:35:00.759 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/obj00000000000000000010 2023-09-22T21:35:00.759 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Dqt5zvX1qYJP9tgleVHX/GoG+0M= 2023-09-22T21:35:00.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Dqt5zvX1qYJP9tgleVHX/GoG+0M='} 2023-09-22T21:35:00.776 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba37a66ece43b549-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.776 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:00.776 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:00.776 DEBUG:boto:Method: HEAD 2023-09-22T21:35:00.776 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:00.776 DEBUG:boto:Data: 2023-09-22T21:35:00.776 DEBUG:boto:Headers: {} 2023-09-22T21:35:00.776 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.777 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.777 DEBUG:boto:Params: {} 2023-09-22T21:35:00.777 DEBUG:boto:Token: None 2023-09-22T21:35:00.777 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/ 2023-09-22T21:35:00.777 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50= 2023-09-22T21:35:00.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50=', 'Content-Length': '0'} 2023-09-22T21:35:00.782 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b900d7fe68c87605-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.783 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:35:00.783 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:35:00.783 DEBUG:boto:Method: PUT 2023-09-22T21:35:00.783 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001 2023-09-22T21:35:00.783 DEBUG:boto:Data: 2023-09-22T21:35:00.783 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:00.783 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.783 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.784 DEBUG:boto:Params: {} 2023-09-22T21:35:00.784 DEBUG:boto:Token: None 2023-09-22T21:35:00.784 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/obj00000000000000000001 2023-09-22T21:35:00.784 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wukKLq3ZcWTBJQbeFutum6nrdbE= 2023-09-22T21:35:00.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wukKLq3ZcWTBJQbeFutum6nrdbE='} 2023-09-22T21:35:00.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036b988d4c2a74800-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.827 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:00.827 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:00.827 DEBUG:boto:Method: HEAD 2023-09-22T21:35:00.827 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:00.827 DEBUG:boto:Data: 2023-09-22T21:35:00.827 DEBUG:boto:Headers: {} 2023-09-22T21:35:00.827 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.827 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.827 DEBUG:boto:Params: {} 2023-09-22T21:35:00.827 DEBUG:boto:Token: None 2023-09-22T21:35:00.828 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/ 2023-09-22T21:35:00.828 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50= 2023-09-22T21:35:00.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50=', 'Content-Length': '0'} 2023-09-22T21:35:00.836 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009377e97cfd26d7f0-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.836 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:35:00.836 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:35:00.836 DEBUG:boto:Method: PUT 2023-09-22T21:35:00.836 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003 2023-09-22T21:35:00.836 DEBUG:boto:Data: 2023-09-22T21:35:00.836 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:00.836 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.837 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.837 DEBUG:boto:Params: {} 2023-09-22T21:35:00.837 DEBUG:boto:Token: None 2023-09-22T21:35:00.837 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/obj00000000000000000003 2023-09-22T21:35:00.837 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:R/AOd9593Ah9HwztHi1c4oRhRhU= 2023-09-22T21:35:00.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:R/AOd9593Ah9HwztHi1c4oRhRhU='} 2023-09-22T21:35:00.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9e507d9cf3b4cd3-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.854 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:00.854 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:00.854 DEBUG:boto:Method: HEAD 2023-09-22T21:35:00.855 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:00.855 DEBUG:boto:Data: 2023-09-22T21:35:00.855 DEBUG:boto:Headers: {} 2023-09-22T21:35:00.855 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.855 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.855 DEBUG:boto:Params: {} 2023-09-22T21:35:00.855 DEBUG:boto:Token: None 2023-09-22T21:35:00.855 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/ 2023-09-22T21:35:00.855 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50= 2023-09-22T21:35:00.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qli4i0LgeH9C4I2SvTOONsAVp50=', 'Content-Length': '0'} 2023-09-22T21:35:00.860 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000614e42b42cc31015-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.861 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:35:00.861 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:35:00.861 DEBUG:boto:Method: PUT 2023-09-22T21:35:00.861 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005 2023-09-22T21:35:00.861 DEBUG:boto:Data: 2023-09-22T21:35:00.861 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:00.861 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:00.861 DEBUG:boto:Port: 8000 2023-09-22T21:35:00.862 DEBUG:boto:Params: {} 2023-09-22T21:35:00.862 DEBUG:boto:Token: None 2023-09-22T21:35:00.862 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:00 GMT /ffzdqu-58/obj00000000000000000005 2023-09-22T21:35:00.862 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FzuKi7KSeyu585muyq3/TPhMsn4= 2023-09-22T21:35:00.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FzuKi7KSeyu585muyq3/TPhMsn4='} 2023-09-22T21:35:00.906 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f19d7ee36f8de96-00650e0884-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:00.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-58', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:35:00.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-58', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:35:00.907 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-58 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:35:00.954 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:35:00.954 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:35:01.017 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:35:01.017 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-58 2023-09-22T21:35:01.030 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 27 2023-09-22T21:35:01.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:35:01.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:35:01.090 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:35:01.126 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:35:01.126 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:35:01.177 INFO:teuthology.orchestra.run.smithi160.stdout:{"objs":["obj00000000000000000058","obj00000000000000000026","obj00000000000000000011","obj00000000000000000046","obj00000000000000000014","obj00000000000000000050","obj00000000000000000032","obj00000000000000000002","obj00000000000000000039","obj00000000000000000007","obj00000000000000000006","obj00000000000000000040","obj00000000000000000009","obj00000000000000000045","obj00000000000000000013","obj00000000000000000012","obj00000000000000000033","obj00000000000000000001","obj00000000000000000000","obj00000000000000000021","obj00000000000000000003","obj00000000000000000055","obj00000000000000000008"]} 2023-09-22T21:35:01.178 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:01.178 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:01.178 DEBUG:boto:Method: HEAD 2023-09-22T21:35:01.178 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:01.178 DEBUG:boto:Data: 2023-09-22T21:35:01.178 DEBUG:boto:Headers: {} 2023-09-22T21:35:01.179 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.179 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.179 DEBUG:boto:Params: {} 2023-09-22T21:35:01.179 DEBUG:boto:Token: None 2023-09-22T21:35:01.179 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/ 2023-09-22T21:35:01.179 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk= 2023-09-22T21:35:01.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk=', 'Content-Length': '0'} 2023-09-22T21:35:01.182 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094ac5c4a2d0eef6e-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.182 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:35:01.183 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:35:01.183 DEBUG:boto:Method: PUT 2023-09-22T21:35:01.183 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000045 2023-09-22T21:35:01.183 DEBUG:boto:Data: 2023-09-22T21:35:01.183 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:01.183 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.183 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.183 DEBUG:boto:Params: {} 2023-09-22T21:35:01.183 DEBUG:boto:Token: None 2023-09-22T21:35:01.184 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/obj00000000000000000045 2023-09-22T21:35:01.184 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:X8rF5wfSHQ2lz7rCAFx/8DWwVEs= 2023-09-22T21:35:01.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:X8rF5wfSHQ2lz7rCAFx/8DWwVEs='} 2023-09-22T21:35:01.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008bcebd5334a68df0-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.190 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:01.190 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:01.190 DEBUG:boto:Method: HEAD 2023-09-22T21:35:01.190 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:01.190 DEBUG:boto:Data: 2023-09-22T21:35:01.191 DEBUG:boto:Headers: {} 2023-09-22T21:35:01.191 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.191 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.191 DEBUG:boto:Params: {} 2023-09-22T21:35:01.191 DEBUG:boto:Token: None 2023-09-22T21:35:01.191 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/ 2023-09-22T21:35:01.191 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk= 2023-09-22T21:35:01.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk=', 'Content-Length': '0'} 2023-09-22T21:35:01.194 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a5904506af12fea-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.194 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:35:01.195 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:35:01.195 DEBUG:boto:Method: PUT 2023-09-22T21:35:01.195 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002 2023-09-22T21:35:01.195 DEBUG:boto:Data: 2023-09-22T21:35:01.195 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:01.195 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.195 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.195 DEBUG:boto:Params: {} 2023-09-22T21:35:01.195 DEBUG:boto:Token: None 2023-09-22T21:35:01.196 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/obj00000000000000000002 2023-09-22T21:35:01.196 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2FixdCUJlGNRUBalLv0JbQbD1sc= 2023-09-22T21:35:01.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2FixdCUJlGNRUBalLv0JbQbD1sc='} 2023-09-22T21:35:01.203 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000061289ecb5f017394-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.203 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:01.203 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:01.203 DEBUG:boto:Method: HEAD 2023-09-22T21:35:01.203 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:01.203 DEBUG:boto:Data: 2023-09-22T21:35:01.203 DEBUG:boto:Headers: {} 2023-09-22T21:35:01.203 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.204 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.204 DEBUG:boto:Params: {} 2023-09-22T21:35:01.204 DEBUG:boto:Token: None 2023-09-22T21:35:01.204 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/ 2023-09-22T21:35:01.204 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk= 2023-09-22T21:35:01.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk=', 'Content-Length': '0'} 2023-09-22T21:35:01.207 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000619a7514c09101c8-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.207 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:35:01.208 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:35:01.208 DEBUG:boto:Method: PUT 2023-09-22T21:35:01.208 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000026 2023-09-22T21:35:01.208 DEBUG:boto:Data: 2023-09-22T21:35:01.208 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:01.208 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.208 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.208 DEBUG:boto:Params: {} 2023-09-22T21:35:01.208 DEBUG:boto:Token: None 2023-09-22T21:35:01.209 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/obj00000000000000000026 2023-09-22T21:35:01.209 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:irCfHwi1J+a0DPX2ATJugjfKfXk= 2023-09-22T21:35:01.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:irCfHwi1J+a0DPX2ATJugjfKfXk='} 2023-09-22T21:35:01.250 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006cefbd32ab1f4014-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.251 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:01.251 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:01.251 DEBUG:boto:Method: HEAD 2023-09-22T21:35:01.251 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:01.251 DEBUG:boto:Data: 2023-09-22T21:35:01.251 DEBUG:boto:Headers: {} 2023-09-22T21:35:01.251 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.252 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.252 DEBUG:boto:Params: {} 2023-09-22T21:35:01.252 DEBUG:boto:Token: None 2023-09-22T21:35:01.252 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/ 2023-09-22T21:35:01.252 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk= 2023-09-22T21:35:01.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk=', 'Content-Length': '0'} 2023-09-22T21:35:01.256 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c76e3bd90e80b4c7-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.256 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:35:01.256 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:35:01.256 DEBUG:boto:Method: PUT 2023-09-22T21:35:01.256 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011 2023-09-22T21:35:01.256 DEBUG:boto:Data: 2023-09-22T21:35:01.256 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:01.257 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.257 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.257 DEBUG:boto:Params: {} 2023-09-22T21:35:01.257 DEBUG:boto:Token: None 2023-09-22T21:35:01.257 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/obj00000000000000000011 2023-09-22T21:35:01.257 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5Jy2QrCQUfBDe+luOHaQukUAElc= 2023-09-22T21:35:01.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5Jy2QrCQUfBDe+luOHaQukUAElc='} 2023-09-22T21:35:01.269 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ee0fe95e75089724-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.269 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:01.269 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:01.269 DEBUG:boto:Method: HEAD 2023-09-22T21:35:01.269 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:01.270 DEBUG:boto:Data: 2023-09-22T21:35:01.270 DEBUG:boto:Headers: {} 2023-09-22T21:35:01.270 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.270 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.270 DEBUG:boto:Params: {} 2023-09-22T21:35:01.270 DEBUG:boto:Token: None 2023-09-22T21:35:01.270 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/ 2023-09-22T21:35:01.270 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk= 2023-09-22T21:35:01.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk=', 'Content-Length': '0'} 2023-09-22T21:35:01.276 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ee840bb4845173d-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.276 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:35:01.276 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:35:01.276 DEBUG:boto:Method: PUT 2023-09-22T21:35:01.276 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000009 2023-09-22T21:35:01.276 DEBUG:boto:Data: 2023-09-22T21:35:01.276 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:01.277 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.277 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.277 DEBUG:boto:Params: {} 2023-09-22T21:35:01.277 DEBUG:boto:Token: None 2023-09-22T21:35:01.277 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/obj00000000000000000009 2023-09-22T21:35:01.277 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x4d/L7chTK8OPxj2H+CmUCacTNs= 2023-09-22T21:35:01.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x4d/L7chTK8OPxj2H+CmUCacTNs='} 2023-09-22T21:35:01.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008bb5f7a480551de6-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.323 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:01.323 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:01.323 DEBUG:boto:Method: HEAD 2023-09-22T21:35:01.323 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:01.323 DEBUG:boto:Data: 2023-09-22T21:35:01.323 DEBUG:boto:Headers: {} 2023-09-22T21:35:01.323 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.324 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.324 DEBUG:boto:Params: {} 2023-09-22T21:35:01.324 DEBUG:boto:Token: None 2023-09-22T21:35:01.324 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/ 2023-09-22T21:35:01.324 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk= 2023-09-22T21:35:01.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk=', 'Content-Length': '0'} 2023-09-22T21:35:01.330 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009451441275558886-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.330 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:35:01.330 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:35:01.331 DEBUG:boto:Method: PUT 2023-09-22T21:35:01.331 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013 2023-09-22T21:35:01.331 DEBUG:boto:Data: 2023-09-22T21:35:01.331 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:01.331 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.331 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.331 DEBUG:boto:Params: {} 2023-09-22T21:35:01.331 DEBUG:boto:Token: None 2023-09-22T21:35:01.332 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/obj00000000000000000013 2023-09-22T21:35:01.332 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IZl6rH50rSn9dchnt1htGzIE708= 2023-09-22T21:35:01.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IZl6rH50rSn9dchnt1htGzIE708='} 2023-09-22T21:35:01.347 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba1ccccf0923eced-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.348 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:01.348 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:01.348 DEBUG:boto:Method: HEAD 2023-09-22T21:35:01.348 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:01.348 DEBUG:boto:Data: 2023-09-22T21:35:01.348 DEBUG:boto:Headers: {} 2023-09-22T21:35:01.348 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.348 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.348 DEBUG:boto:Params: {} 2023-09-22T21:35:01.349 DEBUG:boto:Token: None 2023-09-22T21:35:01.349 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/ 2023-09-22T21:35:01.349 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk= 2023-09-22T21:35:01.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk=', 'Content-Length': '0'} 2023-09-22T21:35:01.355 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015e7523bb3f2d176-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.355 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:35:01.356 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:35:01.356 DEBUG:boto:Method: PUT 2023-09-22T21:35:01.356 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000007 2023-09-22T21:35:01.356 DEBUG:boto:Data: 2023-09-22T21:35:01.356 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:01.356 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.356 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.356 DEBUG:boto:Params: {} 2023-09-22T21:35:01.356 DEBUG:boto:Token: None 2023-09-22T21:35:01.357 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/obj00000000000000000007 2023-09-22T21:35:01.357 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rQaMMgK0F7K9KA7HH0R20p5p6+o= 2023-09-22T21:35:01.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rQaMMgK0F7K9KA7HH0R20p5p6+o='} 2023-09-22T21:35:01.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000abacea431bdf40f-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.403 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:01.403 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:01.403 DEBUG:boto:Method: HEAD 2023-09-22T21:35:01.403 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:01.403 DEBUG:boto:Data: 2023-09-22T21:35:01.403 DEBUG:boto:Headers: {} 2023-09-22T21:35:01.403 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.403 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.403 DEBUG:boto:Params: {} 2023-09-22T21:35:01.404 DEBUG:boto:Token: None 2023-09-22T21:35:01.404 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/ 2023-09-22T21:35:01.404 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk= 2023-09-22T21:35:01.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk=', 'Content-Length': '0'} 2023-09-22T21:35:01.410 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbf84789a263be03-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.410 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:35:01.410 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:35:01.411 DEBUG:boto:Method: PUT 2023-09-22T21:35:01.411 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012 2023-09-22T21:35:01.411 DEBUG:boto:Data: 2023-09-22T21:35:01.411 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:01.411 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.411 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.411 DEBUG:boto:Params: {} 2023-09-22T21:35:01.411 DEBUG:boto:Token: None 2023-09-22T21:35:01.412 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/obj00000000000000000012 2023-09-22T21:35:01.412 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1ubApY4Zuu1prF9iUgjYLMqlB+U= 2023-09-22T21:35:01.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1ubApY4Zuu1prF9iUgjYLMqlB+U='} 2023-09-22T21:35:01.429 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003bd224add384e84-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.429 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:01.429 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:01.429 DEBUG:boto:Method: HEAD 2023-09-22T21:35:01.429 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:01.429 DEBUG:boto:Data: 2023-09-22T21:35:01.429 DEBUG:boto:Headers: {} 2023-09-22T21:35:01.429 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.430 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.430 DEBUG:boto:Params: {} 2023-09-22T21:35:01.430 DEBUG:boto:Token: None 2023-09-22T21:35:01.430 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/ 2023-09-22T21:35:01.430 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk= 2023-09-22T21:35:01.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk=', 'Content-Length': '0'} 2023-09-22T21:35:01.436 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a1cf9593ff8eabf-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.436 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:35:01.436 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:35:01.436 DEBUG:boto:Method: PUT 2023-09-22T21:35:01.436 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000021 2023-09-22T21:35:01.437 DEBUG:boto:Data: 2023-09-22T21:35:01.437 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:01.437 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.437 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.437 DEBUG:boto:Params: {} 2023-09-22T21:35:01.437 DEBUG:boto:Token: None 2023-09-22T21:35:01.437 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/obj00000000000000000021 2023-09-22T21:35:01.437 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3bAp8KqaK5fiHBPraqR9UCZJkMI= 2023-09-22T21:35:01.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3bAp8KqaK5fiHBPraqR9UCZJkMI='} 2023-09-22T21:35:01.483 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003e3d35db504372a-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.483 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:01.483 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:01.483 DEBUG:boto:Method: HEAD 2023-09-22T21:35:01.483 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:01.483 DEBUG:boto:Data: 2023-09-22T21:35:01.483 DEBUG:boto:Headers: {} 2023-09-22T21:35:01.484 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.484 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.484 DEBUG:boto:Params: {} 2023-09-22T21:35:01.484 DEBUG:boto:Token: None 2023-09-22T21:35:01.484 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/ 2023-09-22T21:35:01.484 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk= 2023-09-22T21:35:01.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk=', 'Content-Length': '0'} 2023-09-22T21:35:01.490 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ea0915524c934b1-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.490 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:35:01.491 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:35:01.491 DEBUG:boto:Method: PUT 2023-09-22T21:35:01.491 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000033 2023-09-22T21:35:01.491 DEBUG:boto:Data: 2023-09-22T21:35:01.491 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:01.491 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.491 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.492 DEBUG:boto:Params: {} 2023-09-22T21:35:01.492 DEBUG:boto:Token: None 2023-09-22T21:35:01.492 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/obj00000000000000000033 2023-09-22T21:35:01.492 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+/BwaI7slLSG4KdLRUGkIUYQj9E= 2023-09-22T21:35:01.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+/BwaI7slLSG4KdLRUGkIUYQj9E='} 2023-09-22T21:35:01.508 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000669d82aec368c370-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.508 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:01.508 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:01.508 DEBUG:boto:Method: HEAD 2023-09-22T21:35:01.508 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:01.508 DEBUG:boto:Data: 2023-09-22T21:35:01.508 DEBUG:boto:Headers: {} 2023-09-22T21:35:01.509 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.509 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.509 DEBUG:boto:Params: {} 2023-09-22T21:35:01.509 DEBUG:boto:Token: None 2023-09-22T21:35:01.509 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/ 2023-09-22T21:35:01.509 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk= 2023-09-22T21:35:01.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk=', 'Content-Length': '0'} 2023-09-22T21:35:01.515 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009049fa7a109134cf-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.516 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:35:01.516 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:35:01.516 DEBUG:boto:Method: PUT 2023-09-22T21:35:01.516 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000050 2023-09-22T21:35:01.516 DEBUG:boto:Data: 2023-09-22T21:35:01.516 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:01.516 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.517 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.517 DEBUG:boto:Params: {} 2023-09-22T21:35:01.517 DEBUG:boto:Token: None 2023-09-22T21:35:01.517 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/obj00000000000000000050 2023-09-22T21:35:01.517 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:loKTvmSdak9gGHal4iJJNVnv88U= 2023-09-22T21:35:01.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:loKTvmSdak9gGHal4iJJNVnv88U='} 2023-09-22T21:35:01.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054bdc16318d0ed22-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.563 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:01.563 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:01.563 DEBUG:boto:Method: HEAD 2023-09-22T21:35:01.563 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:01.563 DEBUG:boto:Data: 2023-09-22T21:35:01.563 DEBUG:boto:Headers: {} 2023-09-22T21:35:01.563 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.564 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.564 DEBUG:boto:Params: {} 2023-09-22T21:35:01.564 DEBUG:boto:Token: None 2023-09-22T21:35:01.564 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/ 2023-09-22T21:35:01.564 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk= 2023-09-22T21:35:01.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk=', 'Content-Length': '0'} 2023-09-22T21:35:01.570 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fec007cbe78dfff-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.570 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:35:01.571 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:35:01.571 DEBUG:boto:Method: PUT 2023-09-22T21:35:01.571 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000032 2023-09-22T21:35:01.571 DEBUG:boto:Data: 2023-09-22T21:35:01.571 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:01.571 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.571 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.571 DEBUG:boto:Params: {} 2023-09-22T21:35:01.571 DEBUG:boto:Token: None 2023-09-22T21:35:01.572 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/obj00000000000000000032 2023-09-22T21:35:01.572 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4q5rxpRIpgNeu1aDTmUNIY3kAnE= 2023-09-22T21:35:01.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4q5rxpRIpgNeu1aDTmUNIY3kAnE='} 2023-09-22T21:35:01.590 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed479c253b9dd737-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.590 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:01.590 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:01.590 DEBUG:boto:Method: HEAD 2023-09-22T21:35:01.590 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:01.590 DEBUG:boto:Data: 2023-09-22T21:35:01.590 DEBUG:boto:Headers: {} 2023-09-22T21:35:01.591 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.591 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.591 DEBUG:boto:Params: {} 2023-09-22T21:35:01.591 DEBUG:boto:Token: None 2023-09-22T21:35:01.591 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/ 2023-09-22T21:35:01.591 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk= 2023-09-22T21:35:01.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk=', 'Content-Length': '0'} 2023-09-22T21:35:01.597 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005694a59597b4bd1e-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.598 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:35:01.598 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:35:01.598 DEBUG:boto:Method: PUT 2023-09-22T21:35:01.598 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000039 2023-09-22T21:35:01.598 DEBUG:boto:Data: 2023-09-22T21:35:01.598 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:01.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.599 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.599 DEBUG:boto:Params: {} 2023-09-22T21:35:01.599 DEBUG:boto:Token: None 2023-09-22T21:35:01.599 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/obj00000000000000000039 2023-09-22T21:35:01.599 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:v7O8rLY+SDnNKX5pSi5U7bUNRnI= 2023-09-22T21:35:01.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:v7O8rLY+SDnNKX5pSi5U7bUNRnI='} 2023-09-22T21:35:01.646 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000012975ab9d32dbc7c-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.647 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:01.647 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:01.647 DEBUG:boto:Method: HEAD 2023-09-22T21:35:01.647 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:01.647 DEBUG:boto:Data: 2023-09-22T21:35:01.647 DEBUG:boto:Headers: {} 2023-09-22T21:35:01.647 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.647 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.647 DEBUG:boto:Params: {} 2023-09-22T21:35:01.648 DEBUG:boto:Token: None 2023-09-22T21:35:01.648 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/ 2023-09-22T21:35:01.648 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk= 2023-09-22T21:35:01.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk=', 'Content-Length': '0'} 2023-09-22T21:35:01.654 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1e4d39e0471c955-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.654 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:35:01.654 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:35:01.655 DEBUG:boto:Method: PUT 2023-09-22T21:35:01.655 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000055 2023-09-22T21:35:01.655 DEBUG:boto:Data: 2023-09-22T21:35:01.655 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:01.655 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.655 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.655 DEBUG:boto:Params: {} 2023-09-22T21:35:01.655 DEBUG:boto:Token: None 2023-09-22T21:35:01.655 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/obj00000000000000000055 2023-09-22T21:35:01.656 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jkusUfPi4mqtcrtVWjcarq2Hdus= 2023-09-22T21:35:01.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jkusUfPi4mqtcrtVWjcarq2Hdus='} 2023-09-22T21:35:01.668 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bbac6f0c7cae4cce-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.668 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:01.669 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:01.669 DEBUG:boto:Method: HEAD 2023-09-22T21:35:01.669 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:01.669 DEBUG:boto:Data: 2023-09-22T21:35:01.669 DEBUG:boto:Headers: {} 2023-09-22T21:35:01.669 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.669 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.669 DEBUG:boto:Params: {} 2023-09-22T21:35:01.670 DEBUG:boto:Token: None 2023-09-22T21:35:01.670 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/ 2023-09-22T21:35:01.670 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk= 2023-09-22T21:35:01.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk=', 'Content-Length': '0'} 2023-09-22T21:35:01.676 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e9358e71d16385f-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.676 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:35:01.676 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:35:01.676 DEBUG:boto:Method: PUT 2023-09-22T21:35:01.677 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000008 2023-09-22T21:35:01.677 DEBUG:boto:Data: 2023-09-22T21:35:01.677 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:01.677 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.677 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.677 DEBUG:boto:Params: {} 2023-09-22T21:35:01.677 DEBUG:boto:Token: None 2023-09-22T21:35:01.677 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/obj00000000000000000008 2023-09-22T21:35:01.677 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T9cKi/Le97xAi9R1UPjRGlqHqOo= 2023-09-22T21:35:01.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T9cKi/Le97xAi9R1UPjRGlqHqOo='} 2023-09-22T21:35:01.722 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1887e596a1fb9c9-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.723 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:01.723 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:01.723 DEBUG:boto:Method: HEAD 2023-09-22T21:35:01.723 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:01.723 DEBUG:boto:Data: 2023-09-22T21:35:01.723 DEBUG:boto:Headers: {} 2023-09-22T21:35:01.723 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.723 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.724 DEBUG:boto:Params: {} 2023-09-22T21:35:01.724 DEBUG:boto:Token: None 2023-09-22T21:35:01.724 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/ 2023-09-22T21:35:01.724 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk= 2023-09-22T21:35:01.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk=', 'Content-Length': '0'} 2023-09-22T21:35:01.731 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2e9310b0b84997a-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.731 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:35:01.731 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:35:01.731 DEBUG:boto:Method: PUT 2023-09-22T21:35:01.731 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000058 2023-09-22T21:35:01.731 DEBUG:boto:Data: 2023-09-22T21:35:01.731 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:01.732 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.732 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.732 DEBUG:boto:Params: {} 2023-09-22T21:35:01.732 DEBUG:boto:Token: None 2023-09-22T21:35:01.732 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/obj00000000000000000058 2023-09-22T21:35:01.732 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2Z/vfgNxAfRO29GdXSJQoD44LSk= 2023-09-22T21:35:01.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2Z/vfgNxAfRO29GdXSJQoD44LSk='} 2023-09-22T21:35:01.745 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a04cb3619600e01-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.745 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:01.745 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:01.745 DEBUG:boto:Method: HEAD 2023-09-22T21:35:01.745 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:01.745 DEBUG:boto:Data: 2023-09-22T21:35:01.745 DEBUG:boto:Headers: {} 2023-09-22T21:35:01.745 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.745 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.746 DEBUG:boto:Params: {} 2023-09-22T21:35:01.746 DEBUG:boto:Token: None 2023-09-22T21:35:01.746 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/ 2023-09-22T21:35:01.746 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk= 2023-09-22T21:35:01.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk=', 'Content-Length': '0'} 2023-09-22T21:35:01.752 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8003bfbaec6c94e-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.752 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:35:01.752 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:35:01.752 DEBUG:boto:Method: PUT 2023-09-22T21:35:01.753 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001 2023-09-22T21:35:01.753 DEBUG:boto:Data: 2023-09-22T21:35:01.753 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:01.753 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.753 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.753 DEBUG:boto:Params: {} 2023-09-22T21:35:01.753 DEBUG:boto:Token: None 2023-09-22T21:35:01.753 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/obj00000000000000000001 2023-09-22T21:35:01.754 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WZ3tC8If7lCAr4ty7nGWf63VcxE= 2023-09-22T21:35:01.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WZ3tC8If7lCAr4ty7nGWf63VcxE='} 2023-09-22T21:35:01.802 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1f55a359e18cb67-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.803 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:01.803 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:01.803 DEBUG:boto:Method: HEAD 2023-09-22T21:35:01.803 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:01.804 DEBUG:boto:Data: 2023-09-22T21:35:01.804 DEBUG:boto:Headers: {} 2023-09-22T21:35:01.804 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.804 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.804 DEBUG:boto:Params: {} 2023-09-22T21:35:01.804 DEBUG:boto:Token: None 2023-09-22T21:35:01.805 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/ 2023-09-22T21:35:01.805 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk= 2023-09-22T21:35:01.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk=', 'Content-Length': '0'} 2023-09-22T21:35:01.811 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098a59b19de3f8ba2-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.811 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:35:01.812 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:35:01.812 DEBUG:boto:Method: PUT 2023-09-22T21:35:01.812 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000040 2023-09-22T21:35:01.812 DEBUG:boto:Data: 2023-09-22T21:35:01.812 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:01.812 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.813 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.813 DEBUG:boto:Params: {} 2023-09-22T21:35:01.813 DEBUG:boto:Token: None 2023-09-22T21:35:01.813 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/obj00000000000000000040 2023-09-22T21:35:01.813 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0uZsrZuAcsrJUMU4Z5E7V94oV6Y= 2023-09-22T21:35:01.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0uZsrZuAcsrJUMU4Z5E7V94oV6Y='} 2023-09-22T21:35:01.825 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000090e7766c5484210a-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.826 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:01.826 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:01.826 DEBUG:boto:Method: HEAD 2023-09-22T21:35:01.826 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:01.826 DEBUG:boto:Data: 2023-09-22T21:35:01.827 DEBUG:boto:Headers: {} 2023-09-22T21:35:01.827 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.827 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.827 DEBUG:boto:Params: {} 2023-09-22T21:35:01.827 DEBUG:boto:Token: None 2023-09-22T21:35:01.827 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/ 2023-09-22T21:35:01.827 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk= 2023-09-22T21:35:01.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk=', 'Content-Length': '0'} 2023-09-22T21:35:01.833 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4de44c06570d5cc-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.834 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:35:01.834 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:35:01.834 DEBUG:boto:Method: PUT 2023-09-22T21:35:01.834 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003 2023-09-22T21:35:01.834 DEBUG:boto:Data: 2023-09-22T21:35:01.835 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:01.835 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.835 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.835 DEBUG:boto:Params: {} 2023-09-22T21:35:01.835 DEBUG:boto:Token: None 2023-09-22T21:35:01.835 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/obj00000000000000000003 2023-09-22T21:35:01.836 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BxwXs8MaFW1U4d6n3BwhO2lbzAE= 2023-09-22T21:35:01.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BxwXs8MaFW1U4d6n3BwhO2lbzAE='} 2023-09-22T21:35:01.882 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000099a405c1d85e5937-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.883 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:01.883 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:01.883 DEBUG:boto:Method: HEAD 2023-09-22T21:35:01.883 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:01.883 DEBUG:boto:Data: 2023-09-22T21:35:01.884 DEBUG:boto:Headers: {} 2023-09-22T21:35:01.884 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.884 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.884 DEBUG:boto:Params: {} 2023-09-22T21:35:01.884 DEBUG:boto:Token: None 2023-09-22T21:35:01.884 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/ 2023-09-22T21:35:01.885 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk= 2023-09-22T21:35:01.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk=', 'Content-Length': '0'} 2023-09-22T21:35:01.891 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050f9a563a7cd12ea-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.891 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:35:01.891 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:35:01.891 DEBUG:boto:Method: PUT 2023-09-22T21:35:01.891 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000 2023-09-22T21:35:01.892 DEBUG:boto:Data: 2023-09-22T21:35:01.892 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:01.892 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.892 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.892 DEBUG:boto:Params: {} 2023-09-22T21:35:01.892 DEBUG:boto:Token: None 2023-09-22T21:35:01.893 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/obj00000000000000000000 2023-09-22T21:35:01.893 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:N8KWLJXScXeTvIoEdJILSYrXulk= 2023-09-22T21:35:01.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:N8KWLJXScXeTvIoEdJILSYrXulk='} 2023-09-22T21:35:01.910 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ddd80e0e2fa8efbc-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.910 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:01.910 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:01.910 DEBUG:boto:Method: HEAD 2023-09-22T21:35:01.910 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:01.911 DEBUG:boto:Data: 2023-09-22T21:35:01.911 DEBUG:boto:Headers: {} 2023-09-22T21:35:01.911 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.911 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.911 DEBUG:boto:Params: {} 2023-09-22T21:35:01.911 DEBUG:boto:Token: None 2023-09-22T21:35:01.911 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/ 2023-09-22T21:35:01.911 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk= 2023-09-22T21:35:01.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mPxKURFol3sw3tMjr9HPG0GoZkk=', 'Content-Length': '0'} 2023-09-22T21:35:01.917 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089c10674181b925f-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.918 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:35:01.918 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:35:01.918 DEBUG:boto:Method: PUT 2023-09-22T21:35:01.918 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000014 2023-09-22T21:35:01.918 DEBUG:boto:Data: 2023-09-22T21:35:01.918 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:01.918 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:01.918 DEBUG:boto:Port: 8000 2023-09-22T21:35:01.919 DEBUG:boto:Params: {} 2023-09-22T21:35:01.919 DEBUG:boto:Token: None 2023-09-22T21:35:01.919 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:01 GMT /ffzdqu-58/obj00000000000000000014 2023-09-22T21:35:01.919 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:35q/3kwu8aWXxpEqi9NOCV9Vojk= 2023-09-22T21:35:01.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:35q/3kwu8aWXxpEqi9NOCV9Vojk='} 2023-09-22T21:35:01.962 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d2bddce312ed6b45-00650e0885-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:01.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-58', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:35:01.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-58', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:35:01.963 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-58 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:35:02.008 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:35:02.009 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:35:02.070 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:35:02.071 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-58 2023-09-22T21:35:02.086 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 34 2023-09-22T21:35:02.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:35:02.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:35:02.154 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 29 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:35:02.192 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:35:02.192 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:35:02.243 INFO:teuthology.orchestra.run.smithi160.stdout:{"objs":["obj00000000000000000006","obj00000000000000000059","obj00000000000000000027","obj00000000000000000004","obj00000000000000000011","obj00000000000000000002","obj00000000000000000019","obj00000000000000000049","obj00000000000000000017","obj00000000000000000008","obj00000000000000000015","obj00000000000000000031","obj00000000000000000022","obj00000000000000000013","obj00000000000000000020","obj00000000000000000037","obj00000000000000000005","obj00000000000000000012","obj00000000000000000003","obj00000000000000000042","obj00000000000000000010","obj00000000000000000001","obj00000000000000000009","obj00000000000000000025","obj00000000000000000016","obj00000000000000000000","obj00000000000000000014","obj00000000000000000030","obj00000000000000000021"]} 2023-09-22T21:35:02.244 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:02.244 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:02.244 DEBUG:boto:Method: HEAD 2023-09-22T21:35:02.244 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:02.244 DEBUG:boto:Data: 2023-09-22T21:35:02.245 DEBUG:boto:Headers: {} 2023-09-22T21:35:02.245 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.245 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.245 DEBUG:boto:Params: {} 2023-09-22T21:35:02.245 DEBUG:boto:Token: None 2023-09-22T21:35:02.245 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/ 2023-09-22T21:35:02.245 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM= 2023-09-22T21:35:02.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM=', 'Content-Length': '0'} 2023-09-22T21:35:02.248 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034e5d40865f05d83-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.249 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:35:02.249 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:35:02.249 DEBUG:boto:Method: PUT 2023-09-22T21:35:02.249 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000 2023-09-22T21:35:02.249 DEBUG:boto:Data: 2023-09-22T21:35:02.249 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:02.249 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.249 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.250 DEBUG:boto:Params: {} 2023-09-22T21:35:02.250 DEBUG:boto:Token: None 2023-09-22T21:35:02.250 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/obj00000000000000000000 2023-09-22T21:35:02.250 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2tRL2jzskAexZGwwrW73W41XRt8= 2023-09-22T21:35:02.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2tRL2jzskAexZGwwrW73W41XRt8='} 2023-09-22T21:35:02.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000090ef0a90b7835f79-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.258 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:02.258 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:02.258 DEBUG:boto:Method: HEAD 2023-09-22T21:35:02.258 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:02.258 DEBUG:boto:Data: 2023-09-22T21:35:02.259 DEBUG:boto:Headers: {} 2023-09-22T21:35:02.259 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.259 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.259 DEBUG:boto:Params: {} 2023-09-22T21:35:02.259 DEBUG:boto:Token: None 2023-09-22T21:35:02.259 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/ 2023-09-22T21:35:02.259 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM= 2023-09-22T21:35:02.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM=', 'Content-Length': '0'} 2023-09-22T21:35:02.262 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004fdb6c195dbfa461-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.262 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:35:02.262 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:35:02.263 DEBUG:boto:Method: PUT 2023-09-22T21:35:02.263 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000009 2023-09-22T21:35:02.263 DEBUG:boto:Data: 2023-09-22T21:35:02.263 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:02.263 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.263 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.263 DEBUG:boto:Params: {} 2023-09-22T21:35:02.263 DEBUG:boto:Token: None 2023-09-22T21:35:02.263 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/obj00000000000000000009 2023-09-22T21:35:02.264 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h6tIon8ALs/g2b/HCF1ZNjx9C0c= 2023-09-22T21:35:02.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h6tIon8ALs/g2b/HCF1ZNjx9C0c='} 2023-09-22T21:35:02.271 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003461e24dbb1a0d93-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.271 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:02.271 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:02.272 DEBUG:boto:Method: HEAD 2023-09-22T21:35:02.272 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:02.272 DEBUG:boto:Data: 2023-09-22T21:35:02.272 DEBUG:boto:Headers: {} 2023-09-22T21:35:02.272 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.272 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.272 DEBUG:boto:Params: {} 2023-09-22T21:35:02.272 DEBUG:boto:Token: None 2023-09-22T21:35:02.272 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/ 2023-09-22T21:35:02.273 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM= 2023-09-22T21:35:02.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM=', 'Content-Length': '0'} 2023-09-22T21:35:02.276 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d8c20c99f4af37a-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.276 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:35:02.277 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:35:02.277 DEBUG:boto:Method: PUT 2023-09-22T21:35:02.277 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000008 2023-09-22T21:35:02.277 DEBUG:boto:Data: 2023-09-22T21:35:02.277 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:02.277 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.277 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.277 DEBUG:boto:Params: {} 2023-09-22T21:35:02.277 DEBUG:boto:Token: None 2023-09-22T21:35:02.278 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/obj00000000000000000008 2023-09-22T21:35:02.278 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kfxWfS4LG1XDRqTpSu7U6ovTjAI= 2023-09-22T21:35:02.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kfxWfS4LG1XDRqTpSu7U6ovTjAI='} 2023-09-22T21:35:02.326 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000004798d2cfe611c3f-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.327 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:02.327 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:02.327 DEBUG:boto:Method: HEAD 2023-09-22T21:35:02.327 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:02.328 DEBUG:boto:Data: 2023-09-22T21:35:02.328 DEBUG:boto:Headers: {} 2023-09-22T21:35:02.328 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.328 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.328 DEBUG:boto:Params: {} 2023-09-22T21:35:02.328 DEBUG:boto:Token: None 2023-09-22T21:35:02.329 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/ 2023-09-22T21:35:02.329 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM= 2023-09-22T21:35:02.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM=', 'Content-Length': '0'} 2023-09-22T21:35:02.333 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c4f940667c28941-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.333 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:35:02.333 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:35:02.334 DEBUG:boto:Method: PUT 2023-09-22T21:35:02.334 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000025 2023-09-22T21:35:02.334 DEBUG:boto:Data: 2023-09-22T21:35:02.334 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:02.334 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.334 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.334 DEBUG:boto:Params: {} 2023-09-22T21:35:02.335 DEBUG:boto:Token: None 2023-09-22T21:35:02.335 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/obj00000000000000000025 2023-09-22T21:35:02.335 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/gvh4vkqGtpavKi7fVYr5IdT4l4= 2023-09-22T21:35:02.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/gvh4vkqGtpavKi7fVYr5IdT4l4='} 2023-09-22T21:35:02.344 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c399352d7082bf20-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.344 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:02.344 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:02.344 DEBUG:boto:Method: HEAD 2023-09-22T21:35:02.344 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:02.344 DEBUG:boto:Data: 2023-09-22T21:35:02.344 DEBUG:boto:Headers: {} 2023-09-22T21:35:02.345 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.345 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.345 DEBUG:boto:Params: {} 2023-09-22T21:35:02.345 DEBUG:boto:Token: None 2023-09-22T21:35:02.345 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/ 2023-09-22T21:35:02.345 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM= 2023-09-22T21:35:02.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM=', 'Content-Length': '0'} 2023-09-22T21:35:02.350 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc6803f6f01bfcf7-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.350 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:35:02.350 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:35:02.350 DEBUG:boto:Method: PUT 2023-09-22T21:35:02.350 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005 2023-09-22T21:35:02.351 DEBUG:boto:Data: 2023-09-22T21:35:02.351 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:02.351 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.351 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.351 DEBUG:boto:Params: {} 2023-09-22T21:35:02.351 DEBUG:boto:Token: None 2023-09-22T21:35:02.351 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/obj00000000000000000005 2023-09-22T21:35:02.351 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cIOHPtrZ8hER81VyNDGtJs9w48k= 2023-09-22T21:35:02.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cIOHPtrZ8hER81VyNDGtJs9w48k='} 2023-09-22T21:35:02.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2be33dc7f889e5d-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.399 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:02.399 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:02.399 DEBUG:boto:Method: HEAD 2023-09-22T21:35:02.399 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:02.399 DEBUG:boto:Data: 2023-09-22T21:35:02.399 DEBUG:boto:Headers: {} 2023-09-22T21:35:02.399 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.399 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.399 DEBUG:boto:Params: {} 2023-09-22T21:35:02.400 DEBUG:boto:Token: None 2023-09-22T21:35:02.400 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/ 2023-09-22T21:35:02.400 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM= 2023-09-22T21:35:02.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM=', 'Content-Length': '0'} 2023-09-22T21:35:02.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfdb56b332c7bc5f-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.406 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:35:02.406 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:35:02.406 DEBUG:boto:Method: PUT 2023-09-22T21:35:02.406 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013 2023-09-22T21:35:02.406 DEBUG:boto:Data: 2023-09-22T21:35:02.406 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:02.406 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.406 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.407 DEBUG:boto:Params: {} 2023-09-22T21:35:02.407 DEBUG:boto:Token: None 2023-09-22T21:35:02.407 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/obj00000000000000000013 2023-09-22T21:35:02.407 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+qTrDr0LLyW0I/chfY8kLkyCmXM= 2023-09-22T21:35:02.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+qTrDr0LLyW0I/chfY8kLkyCmXM='} 2023-09-22T21:35:02.421 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000032c1712da8d5b702-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.422 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:02.422 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:02.422 DEBUG:boto:Method: HEAD 2023-09-22T21:35:02.422 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:02.422 DEBUG:boto:Data: 2023-09-22T21:35:02.422 DEBUG:boto:Headers: {} 2023-09-22T21:35:02.422 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.422 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.422 DEBUG:boto:Params: {} 2023-09-22T21:35:02.423 DEBUG:boto:Token: None 2023-09-22T21:35:02.423 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/ 2023-09-22T21:35:02.423 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM= 2023-09-22T21:35:02.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM=', 'Content-Length': '0'} 2023-09-22T21:35:02.429 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003040f565b05a94a1-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.429 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:35:02.429 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:35:02.429 DEBUG:boto:Method: PUT 2023-09-22T21:35:02.429 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000004 2023-09-22T21:35:02.429 DEBUG:boto:Data: 2023-09-22T21:35:02.429 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:02.429 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.430 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.430 DEBUG:boto:Params: {} 2023-09-22T21:35:02.430 DEBUG:boto:Token: None 2023-09-22T21:35:02.430 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/obj00000000000000000004 2023-09-22T21:35:02.430 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IcRvUvP/28FqsKpa4tuIypmM2G8= 2023-09-22T21:35:02.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IcRvUvP/28FqsKpa4tuIypmM2G8='} 2023-09-22T21:35:02.474 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3775b61f446559a-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.474 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:02.475 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:02.475 DEBUG:boto:Method: HEAD 2023-09-22T21:35:02.475 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:02.475 DEBUG:boto:Data: 2023-09-22T21:35:02.475 DEBUG:boto:Headers: {} 2023-09-22T21:35:02.475 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.475 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.475 DEBUG:boto:Params: {} 2023-09-22T21:35:02.475 DEBUG:boto:Token: None 2023-09-22T21:35:02.476 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/ 2023-09-22T21:35:02.476 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM= 2023-09-22T21:35:02.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM=', 'Content-Length': '0'} 2023-09-22T21:35:02.482 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007da3bf979c61f30e-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.483 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:35:02.483 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:35:02.483 DEBUG:boto:Method: PUT 2023-09-22T21:35:02.483 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012 2023-09-22T21:35:02.483 DEBUG:boto:Data: 2023-09-22T21:35:02.484 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:02.484 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.484 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.484 DEBUG:boto:Params: {} 2023-09-22T21:35:02.484 DEBUG:boto:Token: None 2023-09-22T21:35:02.484 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/obj00000000000000000012 2023-09-22T21:35:02.484 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GRofJzkpyM1kwZXPv4+1TvX+Op0= 2023-09-22T21:35:02.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GRofJzkpyM1kwZXPv4+1TvX+Op0='} 2023-09-22T21:35:02.501 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000076b7eb7d6ead0ccb-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.502 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:02.502 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:02.502 DEBUG:boto:Method: HEAD 2023-09-22T21:35:02.502 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:02.502 DEBUG:boto:Data: 2023-09-22T21:35:02.502 DEBUG:boto:Headers: {} 2023-09-22T21:35:02.502 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.502 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.503 DEBUG:boto:Params: {} 2023-09-22T21:35:02.503 DEBUG:boto:Token: None 2023-09-22T21:35:02.503 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/ 2023-09-22T21:35:02.503 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM= 2023-09-22T21:35:02.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM=', 'Content-Length': '0'} 2023-09-22T21:35:02.509 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed41bd31b8d4ca4e-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.510 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:35:02.510 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:35:02.510 DEBUG:boto:Method: PUT 2023-09-22T21:35:02.510 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000016 2023-09-22T21:35:02.510 DEBUG:boto:Data: 2023-09-22T21:35:02.510 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:02.510 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.510 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.510 DEBUG:boto:Params: {} 2023-09-22T21:35:02.511 DEBUG:boto:Token: None 2023-09-22T21:35:02.511 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/obj00000000000000000016 2023-09-22T21:35:02.511 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XtWX1354N3VTr0+jAjeX4bt59qA= 2023-09-22T21:35:02.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XtWX1354N3VTr0+jAjeX4bt59qA='} 2023-09-22T21:35:02.554 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006407b1ba5a2d3c9c-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.555 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:02.555 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:02.555 DEBUG:boto:Method: HEAD 2023-09-22T21:35:02.555 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:02.555 DEBUG:boto:Data: 2023-09-22T21:35:02.555 DEBUG:boto:Headers: {} 2023-09-22T21:35:02.555 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.555 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.555 DEBUG:boto:Params: {} 2023-09-22T21:35:02.556 DEBUG:boto:Token: None 2023-09-22T21:35:02.556 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/ 2023-09-22T21:35:02.556 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM= 2023-09-22T21:35:02.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM=', 'Content-Length': '0'} 2023-09-22T21:35:02.563 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087f95865e6521680-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.563 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:35:02.563 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:35:02.563 DEBUG:boto:Method: PUT 2023-09-22T21:35:02.563 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000059 2023-09-22T21:35:02.563 DEBUG:boto:Data: 2023-09-22T21:35:02.563 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:02.564 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.564 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.564 DEBUG:boto:Params: {} 2023-09-22T21:35:02.564 DEBUG:boto:Token: None 2023-09-22T21:35:02.564 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/obj00000000000000000059 2023-09-22T21:35:02.564 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OwRI0oBfEsw05YYHHFvKSXYTqNk= 2023-09-22T21:35:02.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OwRI0oBfEsw05YYHHFvKSXYTqNk='} 2023-09-22T21:35:02.577 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000066c7c0366d1b24ec-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.577 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:02.577 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:02.577 DEBUG:boto:Method: HEAD 2023-09-22T21:35:02.577 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:02.577 DEBUG:boto:Data: 2023-09-22T21:35:02.577 DEBUG:boto:Headers: {} 2023-09-22T21:35:02.578 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.578 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.578 DEBUG:boto:Params: {} 2023-09-22T21:35:02.578 DEBUG:boto:Token: None 2023-09-22T21:35:02.578 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/ 2023-09-22T21:35:02.578 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM= 2023-09-22T21:35:02.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM=', 'Content-Length': '0'} 2023-09-22T21:35:02.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e801ebcc0c3b4d15-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.585 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:35:02.586 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:35:02.586 DEBUG:boto:Method: PUT 2023-09-22T21:35:02.586 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000020 2023-09-22T21:35:02.586 DEBUG:boto:Data: 2023-09-22T21:35:02.586 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:02.586 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.586 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.587 DEBUG:boto:Params: {} 2023-09-22T21:35:02.587 DEBUG:boto:Token: None 2023-09-22T21:35:02.587 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/obj00000000000000000020 2023-09-22T21:35:02.587 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Bbkot6y5A1iCQ45+weBSiiliyVI= 2023-09-22T21:35:02.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Bbkot6y5A1iCQ45+weBSiiliyVI='} 2023-09-22T21:35:02.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000709339e0612e581c-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.635 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:02.635 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:02.635 DEBUG:boto:Method: HEAD 2023-09-22T21:35:02.635 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:02.635 DEBUG:boto:Data: 2023-09-22T21:35:02.635 DEBUG:boto:Headers: {} 2023-09-22T21:35:02.635 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.635 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.636 DEBUG:boto:Params: {} 2023-09-22T21:35:02.636 DEBUG:boto:Token: None 2023-09-22T21:35:02.636 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/ 2023-09-22T21:35:02.636 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM= 2023-09-22T21:35:02.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM=', 'Content-Length': '0'} 2023-09-22T21:35:02.643 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf33686c134e862d-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.643 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:35:02.643 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:35:02.643 DEBUG:boto:Method: PUT 2023-09-22T21:35:02.644 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000014 2023-09-22T21:35:02.644 DEBUG:boto:Data: 2023-09-22T21:35:02.644 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:02.644 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.644 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.644 DEBUG:boto:Params: {} 2023-09-22T21:35:02.644 DEBUG:boto:Token: None 2023-09-22T21:35:02.644 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/obj00000000000000000014 2023-09-22T21:35:02.645 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f/sw2hb3eNACKSQLgXGGfZ2krvs= 2023-09-22T21:35:02.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f/sw2hb3eNACKSQLgXGGfZ2krvs='} 2023-09-22T21:35:02.660 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ee56b31cc2bb2258-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.660 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:02.660 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:02.660 DEBUG:boto:Method: HEAD 2023-09-22T21:35:02.661 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:02.661 DEBUG:boto:Data: 2023-09-22T21:35:02.661 DEBUG:boto:Headers: {} 2023-09-22T21:35:02.661 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.661 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.661 DEBUG:boto:Params: {} 2023-09-22T21:35:02.661 DEBUG:boto:Token: None 2023-09-22T21:35:02.661 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/ 2023-09-22T21:35:02.661 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM= 2023-09-22T21:35:02.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM=', 'Content-Length': '0'} 2023-09-22T21:35:02.668 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a8b5def7d5c1933-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.668 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:35:02.669 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:35:02.669 DEBUG:boto:Method: PUT 2023-09-22T21:35:02.669 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001 2023-09-22T21:35:02.669 DEBUG:boto:Data: 2023-09-22T21:35:02.669 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:02.669 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.669 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.669 DEBUG:boto:Params: {} 2023-09-22T21:35:02.669 DEBUG:boto:Token: None 2023-09-22T21:35:02.670 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/obj00000000000000000001 2023-09-22T21:35:02.670 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JXS80yckV2KSh5IKLgbuvBdQ1gE= 2023-09-22T21:35:02.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JXS80yckV2KSh5IKLgbuvBdQ1gE='} 2023-09-22T21:35:02.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fae5ea21ad48e752-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.715 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:02.715 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:02.715 DEBUG:boto:Method: HEAD 2023-09-22T21:35:02.715 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:02.715 DEBUG:boto:Data: 2023-09-22T21:35:02.715 DEBUG:boto:Headers: {} 2023-09-22T21:35:02.715 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.715 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.716 DEBUG:boto:Params: {} 2023-09-22T21:35:02.716 DEBUG:boto:Token: None 2023-09-22T21:35:02.716 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/ 2023-09-22T21:35:02.716 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM= 2023-09-22T21:35:02.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM=', 'Content-Length': '0'} 2023-09-22T21:35:02.722 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4e43f10b64306b7-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.723 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:35:02.723 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:35:02.723 DEBUG:boto:Method: PUT 2023-09-22T21:35:02.723 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002 2023-09-22T21:35:02.723 DEBUG:boto:Data: 2023-09-22T21:35:02.723 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:02.723 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.723 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.724 DEBUG:boto:Params: {} 2023-09-22T21:35:02.724 DEBUG:boto:Token: None 2023-09-22T21:35:02.724 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/obj00000000000000000002 2023-09-22T21:35:02.724 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Db0j+WBmMbIG7XWzlZ83FaznAOs= 2023-09-22T21:35:02.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Db0j+WBmMbIG7XWzlZ83FaznAOs='} 2023-09-22T21:35:02.740 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000051eeb15ed3c43d19-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.741 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:02.741 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:02.741 DEBUG:boto:Method: HEAD 2023-09-22T21:35:02.741 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:02.741 DEBUG:boto:Data: 2023-09-22T21:35:02.741 DEBUG:boto:Headers: {} 2023-09-22T21:35:02.741 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.741 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.741 DEBUG:boto:Params: {} 2023-09-22T21:35:02.742 DEBUG:boto:Token: None 2023-09-22T21:35:02.742 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/ 2023-09-22T21:35:02.742 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM= 2023-09-22T21:35:02.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM=', 'Content-Length': '0'} 2023-09-22T21:35:02.749 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003cf6f6ae7f9f5bc5-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.749 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:35:02.749 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:35:02.749 DEBUG:boto:Method: PUT 2023-09-22T21:35:02.749 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000015 2023-09-22T21:35:02.750 DEBUG:boto:Data: 2023-09-22T21:35:02.750 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:02.750 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.750 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.750 DEBUG:boto:Params: {} 2023-09-22T21:35:02.750 DEBUG:boto:Token: None 2023-09-22T21:35:02.750 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/obj00000000000000000015 2023-09-22T21:35:02.750 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:30P2UDVAfXV/w8XCvMV7le35XxU= 2023-09-22T21:35:02.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:30P2UDVAfXV/w8XCvMV7le35XxU='} 2023-09-22T21:35:02.794 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002f7506e71a5ce7bf-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.795 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:02.795 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:02.795 DEBUG:boto:Method: HEAD 2023-09-22T21:35:02.795 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:02.795 DEBUG:boto:Data: 2023-09-22T21:35:02.795 DEBUG:boto:Headers: {} 2023-09-22T21:35:02.795 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.795 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.796 DEBUG:boto:Params: {} 2023-09-22T21:35:02.796 DEBUG:boto:Token: None 2023-09-22T21:35:02.796 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/ 2023-09-22T21:35:02.796 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM= 2023-09-22T21:35:02.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM=', 'Content-Length': '0'} 2023-09-22T21:35:02.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004621c73f0eba23be-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.804 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:35:02.804 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:35:02.804 DEBUG:boto:Method: PUT 2023-09-22T21:35:02.805 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000021 2023-09-22T21:35:02.805 DEBUG:boto:Data: 2023-09-22T21:35:02.805 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:02.805 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.805 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.805 DEBUG:boto:Params: {} 2023-09-22T21:35:02.805 DEBUG:boto:Token: None 2023-09-22T21:35:02.805 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/obj00000000000000000021 2023-09-22T21:35:02.806 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:c31iyyg88zBTGBFYvCSAu55YzoI= 2023-09-22T21:35:02.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:c31iyyg88zBTGBFYvCSAu55YzoI='} 2023-09-22T21:35:02.821 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cde5ff1b3609f27d-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.822 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:02.822 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:02.822 DEBUG:boto:Method: HEAD 2023-09-22T21:35:02.822 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:02.822 DEBUG:boto:Data: 2023-09-22T21:35:02.822 DEBUG:boto:Headers: {} 2023-09-22T21:35:02.822 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.823 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.823 DEBUG:boto:Params: {} 2023-09-22T21:35:02.823 DEBUG:boto:Token: None 2023-09-22T21:35:02.823 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/ 2023-09-22T21:35:02.823 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM= 2023-09-22T21:35:02.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM=', 'Content-Length': '0'} 2023-09-22T21:35:02.830 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000109e1893b446e7d4-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.830 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:35:02.831 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:35:02.831 DEBUG:boto:Method: PUT 2023-09-22T21:35:02.831 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000022 2023-09-22T21:35:02.831 DEBUG:boto:Data: 2023-09-22T21:35:02.831 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:02.831 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.831 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.831 DEBUG:boto:Params: {} 2023-09-22T21:35:02.831 DEBUG:boto:Token: None 2023-09-22T21:35:02.831 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/obj00000000000000000022 2023-09-22T21:35:02.832 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b+kBlnB3heSFWcHoXtHq+R+21UQ= 2023-09-22T21:35:02.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b+kBlnB3heSFWcHoXtHq+R+21UQ='} 2023-09-22T21:35:02.874 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d389f498be0e68b3-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.875 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:02.875 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:02.875 DEBUG:boto:Method: HEAD 2023-09-22T21:35:02.875 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:02.875 DEBUG:boto:Data: 2023-09-22T21:35:02.875 DEBUG:boto:Headers: {} 2023-09-22T21:35:02.875 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.875 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.876 DEBUG:boto:Params: {} 2023-09-22T21:35:02.876 DEBUG:boto:Token: None 2023-09-22T21:35:02.876 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/ 2023-09-22T21:35:02.876 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM= 2023-09-22T21:35:02.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM=', 'Content-Length': '0'} 2023-09-22T21:35:02.883 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000694ff3df5ae9915f-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.883 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:35:02.883 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:35:02.883 DEBUG:boto:Method: PUT 2023-09-22T21:35:02.883 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000049 2023-09-22T21:35:02.883 DEBUG:boto:Data: 2023-09-22T21:35:02.884 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:02.884 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.884 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.884 DEBUG:boto:Params: {} 2023-09-22T21:35:02.884 DEBUG:boto:Token: None 2023-09-22T21:35:02.884 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/obj00000000000000000049 2023-09-22T21:35:02.884 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:N02jmEOrg36r7h/VXglAu7otNuI= 2023-09-22T21:35:02.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:N02jmEOrg36r7h/VXglAu7otNuI='} 2023-09-22T21:35:02.896 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c35df9d210040596-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.897 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:02.897 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:02.897 DEBUG:boto:Method: HEAD 2023-09-22T21:35:02.897 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:02.897 DEBUG:boto:Data: 2023-09-22T21:35:02.897 DEBUG:boto:Headers: {} 2023-09-22T21:35:02.897 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.897 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.897 DEBUG:boto:Params: {} 2023-09-22T21:35:02.898 DEBUG:boto:Token: None 2023-09-22T21:35:02.898 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/ 2023-09-22T21:35:02.898 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM= 2023-09-22T21:35:02.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM=', 'Content-Length': '0'} 2023-09-22T21:35:02.905 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb2371039b1738d6-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.905 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:35:02.905 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:35:02.906 DEBUG:boto:Method: PUT 2023-09-22T21:35:02.906 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006 2023-09-22T21:35:02.906 DEBUG:boto:Data: 2023-09-22T21:35:02.906 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:02.906 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.906 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.906 DEBUG:boto:Params: {} 2023-09-22T21:35:02.906 DEBUG:boto:Token: None 2023-09-22T21:35:02.907 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/obj00000000000000000006 2023-09-22T21:35:02.907 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:k0Zl1Y0+ElEX8V1tIHhiZTs6GzM= 2023-09-22T21:35:02.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:k0Zl1Y0+ElEX8V1tIHhiZTs6GzM='} 2023-09-22T21:35:02.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-09-22T21:35:02.950 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054f5ad8dc6d532ca-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.951 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:02.951 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:02.951 DEBUG:boto:Method: HEAD 2023-09-22T21:35:02.951 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:02.951 DEBUG:boto:Data: 2023-09-22T21:35:02.951 DEBUG:boto:Headers: {} 2023-09-22T21:35:02.951 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.951 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.951 DEBUG:boto:Params: {} 2023-09-22T21:35:02.952 DEBUG:boto:Token: None 2023-09-22T21:35:02.952 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/ 2023-09-22T21:35:02.952 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM= 2023-09-22T21:35:02.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM=', 'Content-Length': '0'} 2023-09-22T21:35:02.959 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f23a743a6e635302-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.959 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:35:02.960 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:35:02.960 DEBUG:boto:Method: PUT 2023-09-22T21:35:02.960 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011 2023-09-22T21:35:02.960 DEBUG:boto:Data: 2023-09-22T21:35:02.960 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:02.960 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.960 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.960 DEBUG:boto:Params: {} 2023-09-22T21:35:02.961 DEBUG:boto:Token: None 2023-09-22T21:35:02.961 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/obj00000000000000000011 2023-09-22T21:35:02.961 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oQih3YOOhny7BHrsDKI/en2jTwk= 2023-09-22T21:35:02.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oQih3YOOhny7BHrsDKI/en2jTwk='} 2023-09-22T21:35:02.977 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000949eec7948f4eb9c-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.978 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:02.978 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:02.978 DEBUG:boto:Method: HEAD 2023-09-22T21:35:02.978 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:02.978 DEBUG:boto:Data: 2023-09-22T21:35:02.978 DEBUG:boto:Headers: {} 2023-09-22T21:35:02.978 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.978 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.978 DEBUG:boto:Params: {} 2023-09-22T21:35:02.979 DEBUG:boto:Token: None 2023-09-22T21:35:02.979 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/ 2023-09-22T21:35:02.979 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM= 2023-09-22T21:35:02.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TUzRMo+rXaU22V2V7Zv61qahUGM=', 'Content-Length': '0'} 2023-09-22T21:35:02.986 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071b499adc8da332c-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:02.986 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:35:02.986 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:35:02.986 DEBUG:boto:Method: PUT 2023-09-22T21:35:02.986 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000037 2023-09-22T21:35:02.986 DEBUG:boto:Data: 2023-09-22T21:35:02.987 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:02.987 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:02.987 DEBUG:boto:Port: 8000 2023-09-22T21:35:02.987 DEBUG:boto:Params: {} 2023-09-22T21:35:02.987 DEBUG:boto:Token: None 2023-09-22T21:35:02.987 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:02 GMT /ffzdqu-58/obj00000000000000000037 2023-09-22T21:35:02.987 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ulf77sCawdWL2amAvfYjMwEFA8g= 2023-09-22T21:35:02.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ulf77sCawdWL2amAvfYjMwEFA8g='} 2023-09-22T21:35:03.034 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000419e2a33675195a4-00650e0886-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.035 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:03.035 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:03.035 DEBUG:boto:Method: HEAD 2023-09-22T21:35:03.035 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:03.035 DEBUG:boto:Data: 2023-09-22T21:35:03.035 DEBUG:boto:Headers: {} 2023-09-22T21:35:03.035 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.035 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.036 DEBUG:boto:Params: {} 2023-09-22T21:35:03.036 DEBUG:boto:Token: None 2023-09-22T21:35:03.036 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/ 2023-09-22T21:35:03.036 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k= 2023-09-22T21:35:03.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k=', 'Content-Length': '0'} 2023-09-22T21:35:03.043 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed475fce1cabe111-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.043 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:35:03.044 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:35:03.044 DEBUG:boto:Method: PUT 2023-09-22T21:35:03.044 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003 2023-09-22T21:35:03.044 DEBUG:boto:Data: 2023-09-22T21:35:03.044 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:03.044 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.044 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.044 DEBUG:boto:Params: {} 2023-09-22T21:35:03.044 DEBUG:boto:Token: None 2023-09-22T21:35:03.045 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/obj00000000000000000003 2023-09-22T21:35:03.045 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sLHyfZHyo89/eNS/FeM61OVWE7E= 2023-09-22T21:35:03.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sLHyfZHyo89/eNS/FeM61OVWE7E='} 2023-09-22T21:35:03.061 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e5262598c247a4c3-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.061 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:03.061 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:03.061 DEBUG:boto:Method: HEAD 2023-09-22T21:35:03.061 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:03.061 DEBUG:boto:Data: 2023-09-22T21:35:03.061 DEBUG:boto:Headers: {} 2023-09-22T21:35:03.062 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.062 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.062 DEBUG:boto:Params: {} 2023-09-22T21:35:03.062 DEBUG:boto:Token: None 2023-09-22T21:35:03.062 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/ 2023-09-22T21:35:03.062 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k= 2023-09-22T21:35:03.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k=', 'Content-Length': '0'} 2023-09-22T21:35:03.069 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4e2d3015b8e9042-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.069 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:35:03.069 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:35:03.070 DEBUG:boto:Method: PUT 2023-09-22T21:35:03.070 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000042 2023-09-22T21:35:03.070 DEBUG:boto:Data: 2023-09-22T21:35:03.070 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:03.070 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.070 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.070 DEBUG:boto:Params: {} 2023-09-22T21:35:03.070 DEBUG:boto:Token: None 2023-09-22T21:35:03.071 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/obj00000000000000000042 2023-09-22T21:35:03.071 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M/GET7glGT7oez11O+yFhG/Uc3w= 2023-09-22T21:35:03.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M/GET7glGT7oez11O+yFhG/Uc3w='} 2023-09-22T21:35:03.114 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095eadd20850df236-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.115 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:03.115 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:03.115 DEBUG:boto:Method: HEAD 2023-09-22T21:35:03.115 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:03.115 DEBUG:boto:Data: 2023-09-22T21:35:03.115 DEBUG:boto:Headers: {} 2023-09-22T21:35:03.115 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.115 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.116 DEBUG:boto:Params: {} 2023-09-22T21:35:03.116 DEBUG:boto:Token: None 2023-09-22T21:35:03.116 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/ 2023-09-22T21:35:03.116 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k= 2023-09-22T21:35:03.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k=', 'Content-Length': '0'} 2023-09-22T21:35:03.122 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031fdbe054b9674aa-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.123 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:35:03.123 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:35:03.123 DEBUG:boto:Method: PUT 2023-09-22T21:35:03.123 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000019 2023-09-22T21:35:03.123 DEBUG:boto:Data: 2023-09-22T21:35:03.123 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:03.124 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.124 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.124 DEBUG:boto:Params: {} 2023-09-22T21:35:03.124 DEBUG:boto:Token: None 2023-09-22T21:35:03.124 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/obj00000000000000000019 2023-09-22T21:35:03.124 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mFtPDAd/Z1E+4VnT0sWR+4MvorM= 2023-09-22T21:35:03.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mFtPDAd/Z1E+4VnT0sWR+4MvorM='} 2023-09-22T21:35:03.141 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007fc7fa1dc4cd71fc-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.141 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:03.141 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:03.141 DEBUG:boto:Method: HEAD 2023-09-22T21:35:03.141 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:03.141 DEBUG:boto:Data: 2023-09-22T21:35:03.142 DEBUG:boto:Headers: {} 2023-09-22T21:35:03.142 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.142 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.142 DEBUG:boto:Params: {} 2023-09-22T21:35:03.142 DEBUG:boto:Token: None 2023-09-22T21:35:03.142 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/ 2023-09-22T21:35:03.142 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k= 2023-09-22T21:35:03.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k=', 'Content-Length': '0'} 2023-09-22T21:35:03.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085d1a4f4760520fb-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.150 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:35:03.150 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:35:03.150 DEBUG:boto:Method: PUT 2023-09-22T21:35:03.150 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000031 2023-09-22T21:35:03.150 DEBUG:boto:Data: 2023-09-22T21:35:03.151 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:03.151 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.151 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.151 DEBUG:boto:Params: {} 2023-09-22T21:35:03.151 DEBUG:boto:Token: None 2023-09-22T21:35:03.151 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/obj00000000000000000031 2023-09-22T21:35:03.151 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IMiJLlKS7BivqcKVSDBCgN7/zDw= 2023-09-22T21:35:03.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IMiJLlKS7BivqcKVSDBCgN7/zDw='} 2023-09-22T21:35:03.198 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002f5a82d3011c7ddc-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.199 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:03.199 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:03.199 DEBUG:boto:Method: HEAD 2023-09-22T21:35:03.199 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:03.199 DEBUG:boto:Data: 2023-09-22T21:35:03.199 DEBUG:boto:Headers: {} 2023-09-22T21:35:03.199 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.200 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.200 DEBUG:boto:Params: {} 2023-09-22T21:35:03.200 DEBUG:boto:Token: None 2023-09-22T21:35:03.200 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/ 2023-09-22T21:35:03.200 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k= 2023-09-22T21:35:03.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k=', 'Content-Length': '0'} 2023-09-22T21:35:03.206 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ac69e844087aadb-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.207 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:35:03.207 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:35:03.207 DEBUG:boto:Method: PUT 2023-09-22T21:35:03.207 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000027 2023-09-22T21:35:03.207 DEBUG:boto:Data: 2023-09-22T21:35:03.207 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:03.207 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.207 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.207 DEBUG:boto:Params: {} 2023-09-22T21:35:03.208 DEBUG:boto:Token: None 2023-09-22T21:35:03.208 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/obj00000000000000000027 2023-09-22T21:35:03.208 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7vg+GFFsDWnowVAQ1PX81TYcxFc= 2023-09-22T21:35:03.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7vg+GFFsDWnowVAQ1PX81TYcxFc='} 2023-09-22T21:35:03.221 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dafb6324f06a08cd-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.221 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:03.221 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:03.221 DEBUG:boto:Method: HEAD 2023-09-22T21:35:03.221 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:03.221 DEBUG:boto:Data: 2023-09-22T21:35:03.221 DEBUG:boto:Headers: {} 2023-09-22T21:35:03.222 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.222 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.222 DEBUG:boto:Params: {} 2023-09-22T21:35:03.222 DEBUG:boto:Token: None 2023-09-22T21:35:03.222 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/ 2023-09-22T21:35:03.222 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k= 2023-09-22T21:35:03.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k=', 'Content-Length': '0'} 2023-09-22T21:35:03.228 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c18976fd49ac6e7-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.229 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:35:03.229 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:35:03.229 DEBUG:boto:Method: PUT 2023-09-22T21:35:03.229 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010 2023-09-22T21:35:03.229 DEBUG:boto:Data: 2023-09-22T21:35:03.229 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:03.229 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.229 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.229 DEBUG:boto:Params: {} 2023-09-22T21:35:03.230 DEBUG:boto:Token: None 2023-09-22T21:35:03.230 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/obj00000000000000000010 2023-09-22T21:35:03.230 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EurCcCXfwuP3kt1KniuNm7RxGWQ= 2023-09-22T21:35:03.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EurCcCXfwuP3kt1KniuNm7RxGWQ='} 2023-09-22T21:35:03.274 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000378103bc90354637-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-58', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:35:03.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-58', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:35:03.275 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-58 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:35:03.321 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:35:03.321 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:35:03.386 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:35:03.386 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-58 2023-09-22T21:35:03.406 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 41 2023-09-22T21:35:03.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:35:03.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:35:03.482 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 31 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:35:03.518 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:35:03.518 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:35:03.575 INFO:teuthology.orchestra.run.smithi160.stdout:{"objs":["obj00000000000000000019","obj00000000000000000006","obj00000000000000000467","obj00000000000000000005","obj00000000000000000056","obj00000000000000000004","obj00000000000000000023","obj00000000000000000317","obj00000000000000000022","obj00000000000000000000","obj00000000000000000021","obj00000000000000000009","obj00000000000000000271","obj00000000000000000008","obj00000000000000000018","obj00000000000000000007","obj00000000000000000017","obj00000000000000000162","obj00000000000000000016","obj00000000000000000003","obj00000000000000000423","obj00000000000000000002","obj00000000000000000012","obj00000000000000000001","obj00000000000000000011","obj00000000000000000283","obj00000000000000000010","obj00000000000000000061","obj00000000000000000050","obj00000000000000000029","obj00000000000000000133"]} 2023-09-22T21:35:03.575 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:03.575 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:03.575 DEBUG:boto:Method: HEAD 2023-09-22T21:35:03.576 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:03.576 DEBUG:boto:Data: 2023-09-22T21:35:03.576 DEBUG:boto:Headers: {} 2023-09-22T21:35:03.576 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.576 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.576 DEBUG:boto:Params: {} 2023-09-22T21:35:03.576 DEBUG:boto:Token: None 2023-09-22T21:35:03.577 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/ 2023-09-22T21:35:03.577 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k= 2023-09-22T21:35:03.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k=', 'Content-Length': '0'} 2023-09-22T21:35:03.580 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7b44ce93d7c2f8f-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.581 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:35:03.581 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:35:03.581 DEBUG:boto:Method: PUT 2023-09-22T21:35:03.581 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000023 2023-09-22T21:35:03.581 DEBUG:boto:Data: 2023-09-22T21:35:03.581 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:03.581 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.581 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.581 DEBUG:boto:Params: {} 2023-09-22T21:35:03.582 DEBUG:boto:Token: None 2023-09-22T21:35:03.582 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/obj00000000000000000023 2023-09-22T21:35:03.582 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1Q9rPL/Xel6YXaOJ/6A6Aqfq/fc= 2023-09-22T21:35:03.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1Q9rPL/Xel6YXaOJ/6A6Aqfq/fc='} 2023-09-22T21:35:03.588 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c329cb3acd1d704-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.588 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:03.588 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:03.588 DEBUG:boto:Method: HEAD 2023-09-22T21:35:03.588 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:03.589 DEBUG:boto:Data: 2023-09-22T21:35:03.589 DEBUG:boto:Headers: {} 2023-09-22T21:35:03.589 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.589 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.589 DEBUG:boto:Params: {} 2023-09-22T21:35:03.589 DEBUG:boto:Token: None 2023-09-22T21:35:03.589 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/ 2023-09-22T21:35:03.589 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k= 2023-09-22T21:35:03.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k=', 'Content-Length': '0'} 2023-09-22T21:35:03.593 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098bcc71b407708a9-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.593 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:35:03.593 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:35:03.593 DEBUG:boto:Method: PUT 2023-09-22T21:35:03.593 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000019 2023-09-22T21:35:03.594 DEBUG:boto:Data: 2023-09-22T21:35:03.594 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:03.594 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.594 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.594 DEBUG:boto:Params: {} 2023-09-22T21:35:03.594 DEBUG:boto:Token: None 2023-09-22T21:35:03.594 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/obj00000000000000000019 2023-09-22T21:35:03.595 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mFtPDAd/Z1E+4VnT0sWR+4MvorM= 2023-09-22T21:35:03.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mFtPDAd/Z1E+4VnT0sWR+4MvorM='} 2023-09-22T21:35:03.638 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b92e4ff03a602d3-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.639 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:03.639 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:03.639 DEBUG:boto:Method: HEAD 2023-09-22T21:35:03.639 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:03.639 DEBUG:boto:Data: 2023-09-22T21:35:03.639 DEBUG:boto:Headers: {} 2023-09-22T21:35:03.639 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.639 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.640 DEBUG:boto:Params: {} 2023-09-22T21:35:03.640 DEBUG:boto:Token: None 2023-09-22T21:35:03.640 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/ 2023-09-22T21:35:03.640 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k= 2023-09-22T21:35:03.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k=', 'Content-Length': '0'} 2023-09-22T21:35:03.646 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c6a283a494651ad-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.646 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:35:03.646 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:35:03.647 DEBUG:boto:Method: PUT 2023-09-22T21:35:03.647 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000467 2023-09-22T21:35:03.647 DEBUG:boto:Data: 2023-09-22T21:35:03.647 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:03.647 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.647 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.647 DEBUG:boto:Params: {} 2023-09-22T21:35:03.648 DEBUG:boto:Token: None 2023-09-22T21:35:03.648 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/obj00000000000000000467 2023-09-22T21:35:03.648 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hQJNM7oSKcnaq7/7Mx4pUm4y47w= 2023-09-22T21:35:03.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hQJNM7oSKcnaq7/7Mx4pUm4y47w='} 2023-09-22T21:35:03.655 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ddb951b0ec62a415-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.656 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:03.656 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:03.656 DEBUG:boto:Method: HEAD 2023-09-22T21:35:03.656 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:03.656 DEBUG:boto:Data: 2023-09-22T21:35:03.656 DEBUG:boto:Headers: {} 2023-09-22T21:35:03.656 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.656 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.657 DEBUG:boto:Params: {} 2023-09-22T21:35:03.657 DEBUG:boto:Token: None 2023-09-22T21:35:03.657 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/ 2023-09-22T21:35:03.657 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k= 2023-09-22T21:35:03.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k=', 'Content-Length': '0'} 2023-09-22T21:35:03.662 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002066a0ea6202e62b-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.662 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:35:03.662 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:35:03.662 DEBUG:boto:Method: PUT 2023-09-22T21:35:03.663 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000271 2023-09-22T21:35:03.663 DEBUG:boto:Data: 2023-09-22T21:35:03.663 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:03.663 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.663 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.663 DEBUG:boto:Params: {} 2023-09-22T21:35:03.663 DEBUG:boto:Token: None 2023-09-22T21:35:03.663 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/obj00000000000000000271 2023-09-22T21:35:03.664 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iltkGGiv+GxtfJxBBL2pGUTGuKs= 2023-09-22T21:35:03.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iltkGGiv+GxtfJxBBL2pGUTGuKs='} 2023-09-22T21:35:03.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9a7747dfb0785c1-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.707 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:03.707 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:03.707 DEBUG:boto:Method: HEAD 2023-09-22T21:35:03.707 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:03.707 DEBUG:boto:Data: 2023-09-22T21:35:03.707 DEBUG:boto:Headers: {} 2023-09-22T21:35:03.707 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.708 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.708 DEBUG:boto:Params: {} 2023-09-22T21:35:03.708 DEBUG:boto:Token: None 2023-09-22T21:35:03.708 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/ 2023-09-22T21:35:03.708 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k= 2023-09-22T21:35:03.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k=', 'Content-Length': '0'} 2023-09-22T21:35:03.713 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ae9d16fcdc2e592-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.714 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:35:03.714 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:35:03.714 DEBUG:boto:Method: PUT 2023-09-22T21:35:03.714 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012 2023-09-22T21:35:03.714 DEBUG:boto:Data: 2023-09-22T21:35:03.714 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:03.714 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.714 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.715 DEBUG:boto:Params: {} 2023-09-22T21:35:03.715 DEBUG:boto:Token: None 2023-09-22T21:35:03.715 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/obj00000000000000000012 2023-09-22T21:35:03.715 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8izsI7imJAg5H+3GAk4g1C37aJ0= 2023-09-22T21:35:03.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8izsI7imJAg5H+3GAk4g1C37aJ0='} 2023-09-22T21:35:03.727 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e0cd17b5ee6e73af-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.727 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:03.727 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:03.727 DEBUG:boto:Method: HEAD 2023-09-22T21:35:03.727 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:03.728 DEBUG:boto:Data: 2023-09-22T21:35:03.728 DEBUG:boto:Headers: {} 2023-09-22T21:35:03.728 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.728 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.728 DEBUG:boto:Params: {} 2023-09-22T21:35:03.728 DEBUG:boto:Token: None 2023-09-22T21:35:03.728 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/ 2023-09-22T21:35:03.728 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k= 2023-09-22T21:35:03.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k=', 'Content-Length': '0'} 2023-09-22T21:35:03.734 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007cf3a707cce928b9-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.735 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:35:03.735 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:35:03.735 DEBUG:boto:Method: PUT 2023-09-22T21:35:03.735 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000021 2023-09-22T21:35:03.735 DEBUG:boto:Data: 2023-09-22T21:35:03.735 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:03.735 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.736 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.736 DEBUG:boto:Params: {} 2023-09-22T21:35:03.736 DEBUG:boto:Token: None 2023-09-22T21:35:03.736 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/obj00000000000000000021 2023-09-22T21:35:03.736 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Sa0mUmYgdYodQCxHZf47sWcTnDc= 2023-09-22T21:35:03.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Sa0mUmYgdYodQCxHZf47sWcTnDc='} 2023-09-22T21:35:03.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009efa6d141a37740a-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.783 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:03.783 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:03.783 DEBUG:boto:Method: HEAD 2023-09-22T21:35:03.783 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:03.783 DEBUG:boto:Data: 2023-09-22T21:35:03.783 DEBUG:boto:Headers: {} 2023-09-22T21:35:03.783 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.783 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.783 DEBUG:boto:Params: {} 2023-09-22T21:35:03.784 DEBUG:boto:Token: None 2023-09-22T21:35:03.784 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/ 2023-09-22T21:35:03.784 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k= 2023-09-22T21:35:03.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k=', 'Content-Length': '0'} 2023-09-22T21:35:03.791 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001743c03664f1ccc4-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.792 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:35:03.792 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:35:03.792 DEBUG:boto:Method: PUT 2023-09-22T21:35:03.792 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001 2023-09-22T21:35:03.792 DEBUG:boto:Data: 2023-09-22T21:35:03.793 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:03.793 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.793 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.793 DEBUG:boto:Params: {} 2023-09-22T21:35:03.793 DEBUG:boto:Token: None 2023-09-22T21:35:03.793 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/obj00000000000000000001 2023-09-22T21:35:03.793 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ttCWvpBZiGF2Cnwk5ZplHxqrLyA= 2023-09-22T21:35:03.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ttCWvpBZiGF2Cnwk5ZplHxqrLyA='} 2023-09-22T21:35:03.812 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc20f4f23ee2ba5a-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.812 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:03.812 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:03.812 DEBUG:boto:Method: HEAD 2023-09-22T21:35:03.813 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:03.813 DEBUG:boto:Data: 2023-09-22T21:35:03.813 DEBUG:boto:Headers: {} 2023-09-22T21:35:03.813 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.813 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.813 DEBUG:boto:Params: {} 2023-09-22T21:35:03.813 DEBUG:boto:Token: None 2023-09-22T21:35:03.813 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/ 2023-09-22T21:35:03.814 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k= 2023-09-22T21:35:03.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k=', 'Content-Length': '0'} 2023-09-22T21:35:03.821 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f63982b21a84ac26-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.821 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:35:03.821 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:35:03.821 DEBUG:boto:Method: PUT 2023-09-22T21:35:03.822 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011 2023-09-22T21:35:03.822 DEBUG:boto:Data: 2023-09-22T21:35:03.822 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:03.822 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.822 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.822 DEBUG:boto:Params: {} 2023-09-22T21:35:03.822 DEBUG:boto:Token: None 2023-09-22T21:35:03.822 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/obj00000000000000000011 2023-09-22T21:35:03.823 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Km2ETdOU8fE+2ZMYr/Z/haPZbAo= 2023-09-22T21:35:03.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Km2ETdOU8fE+2ZMYr/Z/haPZbAo='} 2023-09-22T21:35:03.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dbf0b85a41ef0bc0-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.867 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:03.867 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:03.867 DEBUG:boto:Method: HEAD 2023-09-22T21:35:03.867 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:03.867 DEBUG:boto:Data: 2023-09-22T21:35:03.867 DEBUG:boto:Headers: {} 2023-09-22T21:35:03.867 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.867 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.868 DEBUG:boto:Params: {} 2023-09-22T21:35:03.868 DEBUG:boto:Token: None 2023-09-22T21:35:03.868 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/ 2023-09-22T21:35:03.868 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k= 2023-09-22T21:35:03.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k=', 'Content-Length': '0'} 2023-09-22T21:35:03.875 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000400987bee33900d8-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.876 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:35:03.876 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:35:03.876 DEBUG:boto:Method: PUT 2023-09-22T21:35:03.876 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000029 2023-09-22T21:35:03.876 DEBUG:boto:Data: 2023-09-22T21:35:03.876 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:03.877 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.877 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.877 DEBUG:boto:Params: {} 2023-09-22T21:35:03.877 DEBUG:boto:Token: None 2023-09-22T21:35:03.877 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/obj00000000000000000029 2023-09-22T21:35:03.877 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:41RW5cUlzAGoTI78hBRGyFzJ9Vc= 2023-09-22T21:35:03.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:41RW5cUlzAGoTI78hBRGyFzJ9Vc='} 2023-09-22T21:35:03.890 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000523406e96294d6e9-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.890 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:03.891 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:03.891 DEBUG:boto:Method: HEAD 2023-09-22T21:35:03.891 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:03.891 DEBUG:boto:Data: 2023-09-22T21:35:03.891 DEBUG:boto:Headers: {} 2023-09-22T21:35:03.891 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.891 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.891 DEBUG:boto:Params: {} 2023-09-22T21:35:03.891 DEBUG:boto:Token: None 2023-09-22T21:35:03.892 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/ 2023-09-22T21:35:03.892 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k= 2023-09-22T21:35:03.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k=', 'Content-Length': '0'} 2023-09-22T21:35:03.898 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e98e48215223cda-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.899 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:35:03.899 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:35:03.899 DEBUG:boto:Method: PUT 2023-09-22T21:35:03.899 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000133 2023-09-22T21:35:03.899 DEBUG:boto:Data: 2023-09-22T21:35:03.899 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:03.899 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.899 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.899 DEBUG:boto:Params: {} 2023-09-22T21:35:03.900 DEBUG:boto:Token: None 2023-09-22T21:35:03.900 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/obj00000000000000000133 2023-09-22T21:35:03.900 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mJFDG0kjQVX3sUsa6TNSnzF7fqw= 2023-09-22T21:35:03.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mJFDG0kjQVX3sUsa6TNSnzF7fqw='} 2023-09-22T21:35:03.943 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073d3ccc55d0400a4-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.943 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:03.943 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:03.943 DEBUG:boto:Method: HEAD 2023-09-22T21:35:03.943 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:03.943 DEBUG:boto:Data: 2023-09-22T21:35:03.944 DEBUG:boto:Headers: {} 2023-09-22T21:35:03.944 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.944 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.944 DEBUG:boto:Params: {} 2023-09-22T21:35:03.944 DEBUG:boto:Token: None 2023-09-22T21:35:03.944 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/ 2023-09-22T21:35:03.944 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k= 2023-09-22T21:35:03.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k=', 'Content-Length': '0'} 2023-09-22T21:35:03.952 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000081a513d4f2dfc97-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.952 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:35:03.952 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:35:03.953 DEBUG:boto:Method: PUT 2023-09-22T21:35:03.953 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000317 2023-09-22T21:35:03.953 DEBUG:boto:Data: 2023-09-22T21:35:03.953 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:03.953 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.953 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.953 DEBUG:boto:Params: {} 2023-09-22T21:35:03.953 DEBUG:boto:Token: None 2023-09-22T21:35:03.953 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/obj00000000000000000317 2023-09-22T21:35:03.954 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zU081XgOaLqsdPinvXvyn5vVAE8= 2023-09-22T21:35:03.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zU081XgOaLqsdPinvXvyn5vVAE8='} 2023-09-22T21:35:03.967 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002266260db7008efb-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.967 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:03.967 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:03.967 DEBUG:boto:Method: HEAD 2023-09-22T21:35:03.967 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:03.967 DEBUG:boto:Data: 2023-09-22T21:35:03.967 DEBUG:boto:Headers: {} 2023-09-22T21:35:03.967 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.968 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.968 DEBUG:boto:Params: {} 2023-09-22T21:35:03.968 DEBUG:boto:Token: None 2023-09-22T21:35:03.968 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/ 2023-09-22T21:35:03.968 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k= 2023-09-22T21:35:03.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e+PPc0DRKcAa8XEyQ36agxsij8k=', 'Content-Length': '0'} 2023-09-22T21:35:03.975 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8bdf790bc9bdfaa-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:03.976 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:35:03.976 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:35:03.976 DEBUG:boto:Method: PUT 2023-09-22T21:35:03.976 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000022 2023-09-22T21:35:03.976 DEBUG:boto:Data: 2023-09-22T21:35:03.976 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:03.976 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:03.976 DEBUG:boto:Port: 8000 2023-09-22T21:35:03.976 DEBUG:boto:Params: {} 2023-09-22T21:35:03.977 DEBUG:boto:Token: None 2023-09-22T21:35:03.977 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:03 GMT /ffzdqu-58/obj00000000000000000022 2023-09-22T21:35:03.977 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yDdMDpzUmYcgBIzSTED06a7Q2Lk= 2023-09-22T21:35:03.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:03 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yDdMDpzUmYcgBIzSTED06a7Q2Lk='} 2023-09-22T21:35:04.022 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc68688d63b27671-00650e0887-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.023 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:04.023 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:04.023 DEBUG:boto:Method: HEAD 2023-09-22T21:35:04.023 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:04.023 DEBUG:boto:Data: 2023-09-22T21:35:04.023 DEBUG:boto:Headers: {} 2023-09-22T21:35:04.023 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.023 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.024 DEBUG:boto:Params: {} 2023-09-22T21:35:04.024 DEBUG:boto:Token: None 2023-09-22T21:35:04.024 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/ 2023-09-22T21:35:04.024 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY= 2023-09-22T21:35:04.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY=', 'Content-Length': '0'} 2023-09-22T21:35:04.031 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000155e2ffb5fc7e8c4-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.032 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:35:04.032 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:35:04.032 DEBUG:boto:Method: PUT 2023-09-22T21:35:04.032 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005 2023-09-22T21:35:04.032 DEBUG:boto:Data: 2023-09-22T21:35:04.032 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:04.033 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.033 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.033 DEBUG:boto:Params: {} 2023-09-22T21:35:04.033 DEBUG:boto:Token: None 2023-09-22T21:35:04.033 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/obj00000000000000000005 2023-09-22T21:35:04.033 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fKV/dP0oAtiZawvwLzo2qKd77tI= 2023-09-22T21:35:04.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fKV/dP0oAtiZawvwLzo2qKd77tI='} 2023-09-22T21:35:04.049 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c99c2abdb106b290-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.049 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:04.049 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:04.049 DEBUG:boto:Method: HEAD 2023-09-22T21:35:04.049 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:04.050 DEBUG:boto:Data: 2023-09-22T21:35:04.050 DEBUG:boto:Headers: {} 2023-09-22T21:35:04.050 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.050 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.050 DEBUG:boto:Params: {} 2023-09-22T21:35:04.050 DEBUG:boto:Token: None 2023-09-22T21:35:04.050 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/ 2023-09-22T21:35:04.050 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY= 2023-09-22T21:35:04.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY=', 'Content-Length': '0'} 2023-09-22T21:35:04.058 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085bc4278fe7176b2-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.058 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:35:04.058 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:35:04.058 DEBUG:boto:Method: PUT 2023-09-22T21:35:04.058 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000004 2023-09-22T21:35:04.059 DEBUG:boto:Data: 2023-09-22T21:35:04.059 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:04.059 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.059 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.059 DEBUG:boto:Params: {} 2023-09-22T21:35:04.059 DEBUG:boto:Token: None 2023-09-22T21:35:04.059 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/obj00000000000000000004 2023-09-22T21:35:04.059 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:p8xt6NsxV52okL0TAz9bZ/jGFH4= 2023-09-22T21:35:04.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:p8xt6NsxV52okL0TAz9bZ/jGFH4='} 2023-09-22T21:35:04.103 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b9b91fcfeacd6dd1-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.103 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:04.103 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:04.103 DEBUG:boto:Method: HEAD 2023-09-22T21:35:04.103 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:04.103 DEBUG:boto:Data: 2023-09-22T21:35:04.103 DEBUG:boto:Headers: {} 2023-09-22T21:35:04.104 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.104 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.104 DEBUG:boto:Params: {} 2023-09-22T21:35:04.104 DEBUG:boto:Token: None 2023-09-22T21:35:04.104 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/ 2023-09-22T21:35:04.104 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY= 2023-09-22T21:35:04.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY=', 'Content-Length': '0'} 2023-09-22T21:35:04.111 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8a7dc21262e4551-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.111 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:35:04.112 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:35:04.112 DEBUG:boto:Method: PUT 2023-09-22T21:35:04.112 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000423 2023-09-22T21:35:04.112 DEBUG:boto:Data: 2023-09-22T21:35:04.112 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:04.112 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.112 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.112 DEBUG:boto:Params: {} 2023-09-22T21:35:04.113 DEBUG:boto:Token: None 2023-09-22T21:35:04.113 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/obj00000000000000000423 2023-09-22T21:35:04.113 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nbX/PtbSk/PjO6B6ETjCFDpO33k= 2023-09-22T21:35:04.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nbX/PtbSk/PjO6B6ETjCFDpO33k='} 2023-09-22T21:35:04.125 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c332b9586184f425-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.125 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:04.125 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:04.126 DEBUG:boto:Method: HEAD 2023-09-22T21:35:04.126 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:04.126 DEBUG:boto:Data: 2023-09-22T21:35:04.126 DEBUG:boto:Headers: {} 2023-09-22T21:35:04.126 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.126 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.126 DEBUG:boto:Params: {} 2023-09-22T21:35:04.126 DEBUG:boto:Token: None 2023-09-22T21:35:04.127 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/ 2023-09-22T21:35:04.127 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY= 2023-09-22T21:35:04.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY=', 'Content-Length': '0'} 2023-09-22T21:35:04.134 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054eef2c6534f1e98-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.134 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:35:04.135 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:35:04.135 DEBUG:boto:Method: PUT 2023-09-22T21:35:04.135 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010 2023-09-22T21:35:04.135 DEBUG:boto:Data: 2023-09-22T21:35:04.135 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:04.135 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.135 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.135 DEBUG:boto:Params: {} 2023-09-22T21:35:04.136 DEBUG:boto:Token: None 2023-09-22T21:35:04.136 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/obj00000000000000000010 2023-09-22T21:35:04.136 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:X06sCi6yjN+Hileuwq6kjx9VP8I= 2023-09-22T21:35:04.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:X06sCi6yjN+Hileuwq6kjx9VP8I='} 2023-09-22T21:35:04.178 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb93404192762d44-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.179 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:04.179 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:04.179 DEBUG:boto:Method: HEAD 2023-09-22T21:35:04.179 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:04.179 DEBUG:boto:Data: 2023-09-22T21:35:04.179 DEBUG:boto:Headers: {} 2023-09-22T21:35:04.180 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.180 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.180 DEBUG:boto:Params: {} 2023-09-22T21:35:04.180 DEBUG:boto:Token: None 2023-09-22T21:35:04.180 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/ 2023-09-22T21:35:04.180 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY= 2023-09-22T21:35:04.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY=', 'Content-Length': '0'} 2023-09-22T21:35:04.195 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000094326dc8a4c5c63-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.196 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:35:04.196 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:35:04.196 DEBUG:boto:Method: PUT 2023-09-22T21:35:04.196 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000017 2023-09-22T21:35:04.196 DEBUG:boto:Data: 2023-09-22T21:35:04.196 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:04.196 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.196 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.197 DEBUG:boto:Params: {} 2023-09-22T21:35:04.197 DEBUG:boto:Token: None 2023-09-22T21:35:04.197 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/obj00000000000000000017 2023-09-22T21:35:04.197 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UyUWII+yseeE+nczHA3lEvOS2fU= 2023-09-22T21:35:04.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UyUWII+yseeE+nczHA3lEvOS2fU='} 2023-09-22T21:35:04.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bbf97df081934705-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.210 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:04.210 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:04.211 DEBUG:boto:Method: HEAD 2023-09-22T21:35:04.211 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:04.211 DEBUG:boto:Data: 2023-09-22T21:35:04.211 DEBUG:boto:Headers: {} 2023-09-22T21:35:04.211 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.211 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.211 DEBUG:boto:Params: {} 2023-09-22T21:35:04.211 DEBUG:boto:Token: None 2023-09-22T21:35:04.211 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/ 2023-09-22T21:35:04.212 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY= 2023-09-22T21:35:04.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY=', 'Content-Length': '0'} 2023-09-22T21:35:04.218 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f50bfd7df8464cd-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.219 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:35:04.219 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:35:04.219 DEBUG:boto:Method: PUT 2023-09-22T21:35:04.219 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000061 2023-09-22T21:35:04.219 DEBUG:boto:Data: 2023-09-22T21:35:04.219 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:04.219 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.220 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.220 DEBUG:boto:Params: {} 2023-09-22T21:35:04.220 DEBUG:boto:Token: None 2023-09-22T21:35:04.220 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/obj00000000000000000061 2023-09-22T21:35:04.220 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FWwRNGlI6wkIMLU4H9eGwrxdWeg= 2023-09-22T21:35:04.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FWwRNGlI6wkIMLU4H9eGwrxdWeg='} 2023-09-22T21:35:04.263 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000002cdaf4c90291623-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.263 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:04.263 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:04.263 DEBUG:boto:Method: HEAD 2023-09-22T21:35:04.263 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:04.263 DEBUG:boto:Data: 2023-09-22T21:35:04.263 DEBUG:boto:Headers: {} 2023-09-22T21:35:04.264 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.264 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.264 DEBUG:boto:Params: {} 2023-09-22T21:35:04.264 DEBUG:boto:Token: None 2023-09-22T21:35:04.264 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/ 2023-09-22T21:35:04.264 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY= 2023-09-22T21:35:04.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY=', 'Content-Length': '0'} 2023-09-22T21:35:04.272 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a44e9055b2942ef0-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.272 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:35:04.272 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:35:04.272 DEBUG:boto:Method: PUT 2023-09-22T21:35:04.272 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000283 2023-09-22T21:35:04.272 DEBUG:boto:Data: 2023-09-22T21:35:04.273 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:04.273 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.273 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.273 DEBUG:boto:Params: {} 2023-09-22T21:35:04.273 DEBUG:boto:Token: None 2023-09-22T21:35:04.273 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/obj00000000000000000283 2023-09-22T21:35:04.273 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IgTfZ4cpiyJyZ/WlVjm5dhgCNYo= 2023-09-22T21:35:04.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IgTfZ4cpiyJyZ/WlVjm5dhgCNYo='} 2023-09-22T21:35:04.286 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e89047850e64b7c-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.286 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:04.286 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:04.287 DEBUG:boto:Method: HEAD 2023-09-22T21:35:04.287 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:04.287 DEBUG:boto:Data: 2023-09-22T21:35:04.287 DEBUG:boto:Headers: {} 2023-09-22T21:35:04.287 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.287 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.287 DEBUG:boto:Params: {} 2023-09-22T21:35:04.287 DEBUG:boto:Token: None 2023-09-22T21:35:04.287 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/ 2023-09-22T21:35:04.288 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY= 2023-09-22T21:35:04.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY=', 'Content-Length': '0'} 2023-09-22T21:35:04.294 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000385f5ec3190fea7c-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.295 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:35:04.295 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:35:04.295 DEBUG:boto:Method: PUT 2023-09-22T21:35:04.295 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003 2023-09-22T21:35:04.295 DEBUG:boto:Data: 2023-09-22T21:35:04.295 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:04.295 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.295 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.295 DEBUG:boto:Params: {} 2023-09-22T21:35:04.296 DEBUG:boto:Token: None 2023-09-22T21:35:04.296 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/obj00000000000000000003 2023-09-22T21:35:04.296 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:U/8YZRqMpcDNqwv4BXJJ1n4gGco= 2023-09-22T21:35:04.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:U/8YZRqMpcDNqwv4BXJJ1n4gGco='} 2023-09-22T21:35:04.339 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c18af1a10bd2928d-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.339 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:04.339 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:04.339 DEBUG:boto:Method: HEAD 2023-09-22T21:35:04.339 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:04.339 DEBUG:boto:Data: 2023-09-22T21:35:04.339 DEBUG:boto:Headers: {} 2023-09-22T21:35:04.339 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.340 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.340 DEBUG:boto:Params: {} 2023-09-22T21:35:04.340 DEBUG:boto:Token: None 2023-09-22T21:35:04.340 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/ 2023-09-22T21:35:04.340 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY= 2023-09-22T21:35:04.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY=', 'Content-Length': '0'} 2023-09-22T21:35:04.349 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000882385ce07bf6ebb-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.349 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:35:04.349 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:35:04.349 DEBUG:boto:Method: PUT 2023-09-22T21:35:04.350 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000050 2023-09-22T21:35:04.350 DEBUG:boto:Data: 2023-09-22T21:35:04.350 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:04.350 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.350 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.350 DEBUG:boto:Params: {} 2023-09-22T21:35:04.350 DEBUG:boto:Token: None 2023-09-22T21:35:04.350 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/obj00000000000000000050 2023-09-22T21:35:04.351 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UUm6dxO3y42rJasDP0W+bm2TaVs= 2023-09-22T21:35:04.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UUm6dxO3y42rJasDP0W+bm2TaVs='} 2023-09-22T21:35:04.367 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002470081bdc89131e-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.367 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:04.368 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:04.368 DEBUG:boto:Method: HEAD 2023-09-22T21:35:04.368 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:04.368 DEBUG:boto:Data: 2023-09-22T21:35:04.368 DEBUG:boto:Headers: {} 2023-09-22T21:35:04.368 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.368 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.368 DEBUG:boto:Params: {} 2023-09-22T21:35:04.368 DEBUG:boto:Token: None 2023-09-22T21:35:04.368 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/ 2023-09-22T21:35:04.369 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY= 2023-09-22T21:35:04.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY=', 'Content-Length': '0'} 2023-09-22T21:35:04.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b915ede439157c97-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.376 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:35:04.376 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:35:04.376 DEBUG:boto:Method: PUT 2023-09-22T21:35:04.376 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000 2023-09-22T21:35:04.376 DEBUG:boto:Data: 2023-09-22T21:35:04.376 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:04.376 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.377 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.377 DEBUG:boto:Params: {} 2023-09-22T21:35:04.377 DEBUG:boto:Token: None 2023-09-22T21:35:04.377 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/obj00000000000000000000 2023-09-22T21:35:04.377 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JlltJdZEvQiIyJusAUsA/XSzbcc= 2023-09-22T21:35:04.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JlltJdZEvQiIyJusAUsA/XSzbcc='} 2023-09-22T21:35:04.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007145e18e80a0db1f-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.423 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:04.423 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:04.423 DEBUG:boto:Method: HEAD 2023-09-22T21:35:04.423 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:04.423 DEBUG:boto:Data: 2023-09-22T21:35:04.423 DEBUG:boto:Headers: {} 2023-09-22T21:35:04.423 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.424 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.424 DEBUG:boto:Params: {} 2023-09-22T21:35:04.424 DEBUG:boto:Token: None 2023-09-22T21:35:04.424 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/ 2023-09-22T21:35:04.424 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY= 2023-09-22T21:35:04.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY=', 'Content-Length': '0'} 2023-09-22T21:35:04.431 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af44a2c6492561d9-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.432 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:35:04.432 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:35:04.432 DEBUG:boto:Method: PUT 2023-09-22T21:35:04.432 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000008 2023-09-22T21:35:04.432 DEBUG:boto:Data: 2023-09-22T21:35:04.432 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:04.432 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.432 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.432 DEBUG:boto:Params: {} 2023-09-22T21:35:04.433 DEBUG:boto:Token: None 2023-09-22T21:35:04.433 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/obj00000000000000000008 2023-09-22T21:35:04.433 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lOReYhOPijOQ9uasBCsVvoD7Jlg= 2023-09-22T21:35:04.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lOReYhOPijOQ9uasBCsVvoD7Jlg='} 2023-09-22T21:35:04.450 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dcebb482f984a9f6-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.450 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:04.450 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:04.450 DEBUG:boto:Method: HEAD 2023-09-22T21:35:04.450 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:04.451 DEBUG:boto:Data: 2023-09-22T21:35:04.451 DEBUG:boto:Headers: {} 2023-09-22T21:35:04.451 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.451 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.451 DEBUG:boto:Params: {} 2023-09-22T21:35:04.451 DEBUG:boto:Token: None 2023-09-22T21:35:04.451 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/ 2023-09-22T21:35:04.451 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY= 2023-09-22T21:35:04.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY=', 'Content-Length': '0'} 2023-09-22T21:35:04.458 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b748e616c8d310d7-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.459 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:35:04.459 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:35:04.459 DEBUG:boto:Method: PUT 2023-09-22T21:35:04.459 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000009 2023-09-22T21:35:04.459 DEBUG:boto:Data: 2023-09-22T21:35:04.459 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:04.459 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.459 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.460 DEBUG:boto:Params: {} 2023-09-22T21:35:04.460 DEBUG:boto:Token: None 2023-09-22T21:35:04.460 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/obj00000000000000000009 2023-09-22T21:35:04.460 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bXjdpgB1nXh23dQ3K5oNJYDdy4Q= 2023-09-22T21:35:04.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bXjdpgB1nXh23dQ3K5oNJYDdy4Q='} 2023-09-22T21:35:04.503 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c5e863892035cd2-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.754 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:04.754 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:04.754 DEBUG:boto:Method: HEAD 2023-09-22T21:35:04.755 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:04.755 DEBUG:boto:Data: 2023-09-22T21:35:04.755 DEBUG:boto:Headers: {} 2023-09-22T21:35:04.755 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.755 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.755 DEBUG:boto:Params: {} 2023-09-22T21:35:04.755 DEBUG:boto:Token: None 2023-09-22T21:35:04.755 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/ 2023-09-22T21:35:04.756 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY= 2023-09-22T21:35:04.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY=', 'Content-Length': '0'} 2023-09-22T21:35:04.763 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002458e9f4016daf0-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.763 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:35:04.764 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:35:04.764 DEBUG:boto:Method: PUT 2023-09-22T21:35:04.764 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000018 2023-09-22T21:35:04.764 DEBUG:boto:Data: 2023-09-22T21:35:04.764 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:04.764 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.764 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.764 DEBUG:boto:Params: {} 2023-09-22T21:35:04.764 DEBUG:boto:Token: None 2023-09-22T21:35:04.765 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/obj00000000000000000018 2023-09-22T21:35:04.765 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:j3peKsTabCh/TgCGAPjbyJfw4QQ= 2023-09-22T21:35:04.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:j3peKsTabCh/TgCGAPjbyJfw4QQ='} 2023-09-22T21:35:04.778 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f48377527413824a-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.779 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:04.779 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:04.779 DEBUG:boto:Method: HEAD 2023-09-22T21:35:04.779 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:04.779 DEBUG:boto:Data: 2023-09-22T21:35:04.779 DEBUG:boto:Headers: {} 2023-09-22T21:35:04.779 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.779 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.780 DEBUG:boto:Params: {} 2023-09-22T21:35:04.780 DEBUG:boto:Token: None 2023-09-22T21:35:04.780 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/ 2023-09-22T21:35:04.780 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY= 2023-09-22T21:35:04.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY=', 'Content-Length': '0'} 2023-09-22T21:35:04.787 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f44fa0215e15703-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.787 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:35:04.788 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:35:04.788 DEBUG:boto:Method: PUT 2023-09-22T21:35:04.788 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006 2023-09-22T21:35:04.788 DEBUG:boto:Data: 2023-09-22T21:35:04.788 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:04.788 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.788 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.788 DEBUG:boto:Params: {} 2023-09-22T21:35:04.789 DEBUG:boto:Token: None 2023-09-22T21:35:04.789 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/obj00000000000000000006 2023-09-22T21:35:04.789 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uXPeosKUY2t1Ehl0KlLA8PSZTRc= 2023-09-22T21:35:04.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uXPeosKUY2t1Ehl0KlLA8PSZTRc='} 2023-09-22T21:35:04.837 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c3f44aec26f7c5b4-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.837 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:04.837 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:04.837 DEBUG:boto:Method: HEAD 2023-09-22T21:35:04.838 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:04.838 DEBUG:boto:Data: 2023-09-22T21:35:04.838 DEBUG:boto:Headers: {} 2023-09-22T21:35:04.838 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.838 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.838 DEBUG:boto:Params: {} 2023-09-22T21:35:04.838 DEBUG:boto:Token: None 2023-09-22T21:35:04.838 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/ 2023-09-22T21:35:04.839 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY= 2023-09-22T21:35:04.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY=', 'Content-Length': '0'} 2023-09-22T21:35:04.846 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000373e22eb43d55689-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.846 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:35:04.846 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:35:04.846 DEBUG:boto:Method: PUT 2023-09-22T21:35:04.846 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000007 2023-09-22T21:35:04.846 DEBUG:boto:Data: 2023-09-22T21:35:04.847 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:04.847 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.847 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.847 DEBUG:boto:Params: {} 2023-09-22T21:35:04.847 DEBUG:boto:Token: None 2023-09-22T21:35:04.847 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/obj00000000000000000007 2023-09-22T21:35:04.847 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MFLdrY2EB7qKuKT0bUS9bhUfaD8= 2023-09-22T21:35:04.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MFLdrY2EB7qKuKT0bUS9bhUfaD8='} 2023-09-22T21:35:04.864 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000078da96a54a48008c-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.864 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:04.864 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:04.864 DEBUG:boto:Method: HEAD 2023-09-22T21:35:04.864 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:04.864 DEBUG:boto:Data: 2023-09-22T21:35:04.865 DEBUG:boto:Headers: {} 2023-09-22T21:35:04.865 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.865 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.865 DEBUG:boto:Params: {} 2023-09-22T21:35:04.865 DEBUG:boto:Token: None 2023-09-22T21:35:04.865 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/ 2023-09-22T21:35:04.865 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY= 2023-09-22T21:35:04.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tSUQqWFOTtid35QnfJsH93rpELY=', 'Content-Length': '0'} 2023-09-22T21:35:04.876 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043ed1ccaf8477ff3-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.876 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:35:04.877 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:35:04.877 DEBUG:boto:Method: PUT 2023-09-22T21:35:04.877 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002 2023-09-22T21:35:04.877 DEBUG:boto:Data: 2023-09-22T21:35:04.877 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:04.877 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:04.877 DEBUG:boto:Port: 8000 2023-09-22T21:35:04.877 DEBUG:boto:Params: {} 2023-09-22T21:35:04.877 DEBUG:boto:Token: None 2023-09-22T21:35:04.878 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:04 GMT /ffzdqu-58/obj00000000000000000002 2023-09-22T21:35:04.878 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bGri9aDDeg/v2I3c/EI0OI6IjCY= 2023-09-22T21:35:04.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:04 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bGri9aDDeg/v2I3c/EI0OI6IjCY='} 2023-09-22T21:35:04.923 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000066f9a4da83872887-00650e0888-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:04.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-58', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:35:04.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-58', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:35:04.923 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-58 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:35:04.968 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:35:04.968 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:35:05.032 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:35:05.033 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-58 2023-09-22T21:35:05.051 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 52 2023-09-22T21:35:05.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:35:05.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:35:05.134 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 37 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:35:05.218 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:35:05.219 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:35:05.274 INFO:teuthology.orchestra.run.smithi160.stdout:{"objs":["obj00000000000000000060","obj00000000000000000000","obj00000000000000000011","obj00000000000000000001","obj00000000000000000033","obj00000000000000000065","obj00000000000000000005","obj00000000000000000016","obj00000000000000000048","obj00000000000000000015","obj00000000000000000026","obj00000000000000000030","obj00000000000000000062","obj00000000000000000009","obj00000000000000000040","obj00000000000000000024","obj00000000000000000012","obj00000000000000000002","obj00000000000000000013","obj00000000000000000045","obj00000000000000000006","obj00000000000000000017","obj00000000000000000007","obj00000000000000000039","obj00000000000000000027","obj00000000000000000010","obj00000000000000000021","obj00000000000000000053","obj00000000000000000020","obj00000000000000000004","obj00000000000000000036","obj00000000000000000003","obj00000000000000000014","obj00000000000000000019","obj00000000000000000057","obj00000000000000000018","obj00000000000000000008"]} 2023-09-22T21:35:05.274 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:05.275 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:05.275 DEBUG:boto:Method: HEAD 2023-09-22T21:35:05.275 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:05.275 DEBUG:boto:Data: 2023-09-22T21:35:05.275 DEBUG:boto:Headers: {} 2023-09-22T21:35:05.275 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.275 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.275 DEBUG:boto:Params: {} 2023-09-22T21:35:05.276 DEBUG:boto:Token: None 2023-09-22T21:35:05.276 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/ 2023-09-22T21:35:05.276 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU= 2023-09-22T21:35:05.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU=', 'Content-Length': '0'} 2023-09-22T21:35:05.280 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a285109891f78979-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.280 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:35:05.281 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:35:05.281 DEBUG:boto:Method: PUT 2023-09-22T21:35:05.281 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000062 2023-09-22T21:35:05.281 DEBUG:boto:Data: 2023-09-22T21:35:05.281 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:05.281 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.281 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.281 DEBUG:boto:Params: {} 2023-09-22T21:35:05.282 DEBUG:boto:Token: None 2023-09-22T21:35:05.282 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/obj00000000000000000062 2023-09-22T21:35:05.282 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Du4ngtE549ouRl1QlS5dMrhcN08= 2023-09-22T21:35:05.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Du4ngtE549ouRl1QlS5dMrhcN08='} 2023-09-22T21:35:05.288 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003548078e42b0d4a4-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.288 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:05.289 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:05.289 DEBUG:boto:Method: HEAD 2023-09-22T21:35:05.289 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:05.289 DEBUG:boto:Data: 2023-09-22T21:35:05.289 DEBUG:boto:Headers: {} 2023-09-22T21:35:05.289 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.289 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.289 DEBUG:boto:Params: {} 2023-09-22T21:35:05.290 DEBUG:boto:Token: None 2023-09-22T21:35:05.290 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/ 2023-09-22T21:35:05.290 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU= 2023-09-22T21:35:05.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU=', 'Content-Length': '0'} 2023-09-22T21:35:05.294 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004bb5b0515bc00050-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.294 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:35:05.294 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:35:05.294 DEBUG:boto:Method: PUT 2023-09-22T21:35:05.294 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000009 2023-09-22T21:35:05.294 DEBUG:boto:Data: 2023-09-22T21:35:05.295 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:05.295 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.295 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.295 DEBUG:boto:Params: {} 2023-09-22T21:35:05.295 DEBUG:boto:Token: None 2023-09-22T21:35:05.295 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/obj00000000000000000009 2023-09-22T21:35:05.295 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8vhcx1aq1GI02XQwUAMXebJMfwg= 2023-09-22T21:35:05.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8vhcx1aq1GI02XQwUAMXebJMfwg='} 2023-09-22T21:35:05.303 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006aa929f4655ff3d3-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.303 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:05.303 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:05.303 DEBUG:boto:Method: HEAD 2023-09-22T21:35:05.303 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:05.304 DEBUG:boto:Data: 2023-09-22T21:35:05.304 DEBUG:boto:Headers: {} 2023-09-22T21:35:05.304 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.304 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.304 DEBUG:boto:Params: {} 2023-09-22T21:35:05.304 DEBUG:boto:Token: None 2023-09-22T21:35:05.304 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/ 2023-09-22T21:35:05.304 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU= 2023-09-22T21:35:05.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU=', 'Content-Length': '0'} 2023-09-22T21:35:05.308 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005253f6be438a299c-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.308 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:35:05.308 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:35:05.309 DEBUG:boto:Method: PUT 2023-09-22T21:35:05.309 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000060 2023-09-22T21:35:05.309 DEBUG:boto:Data: 2023-09-22T21:35:05.309 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:05.309 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.309 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.309 DEBUG:boto:Params: {} 2023-09-22T21:35:05.309 DEBUG:boto:Token: None 2023-09-22T21:35:05.310 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/obj00000000000000000060 2023-09-22T21:35:05.310 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IMNAZ8bpc8N/4L9q01Y1WCQzZ8k= 2023-09-22T21:35:05.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IMNAZ8bpc8N/4L9q01Y1WCQzZ8k='} 2023-09-22T21:35:05.385 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002332e145b5fc55fd-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.385 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:05.386 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:05.386 DEBUG:boto:Method: HEAD 2023-09-22T21:35:05.386 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:05.386 DEBUG:boto:Data: 2023-09-22T21:35:05.386 DEBUG:boto:Headers: {} 2023-09-22T21:35:05.386 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.386 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.386 DEBUG:boto:Params: {} 2023-09-22T21:35:05.387 DEBUG:boto:Token: None 2023-09-22T21:35:05.387 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/ 2023-09-22T21:35:05.387 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU= 2023-09-22T21:35:05.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU=', 'Content-Length': '0'} 2023-09-22T21:35:05.395 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff2fdf2a430d19ad-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.395 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:35:05.395 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:35:05.395 DEBUG:boto:Method: PUT 2023-09-22T21:35:05.395 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000016 2023-09-22T21:35:05.396 DEBUG:boto:Data: 2023-09-22T21:35:05.396 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:05.396 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.396 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.396 DEBUG:boto:Params: {} 2023-09-22T21:35:05.396 DEBUG:boto:Token: None 2023-09-22T21:35:05.396 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/obj00000000000000000016 2023-09-22T21:35:05.396 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T73uZF4gZYWYnWiaPSejYolK2z0= 2023-09-22T21:35:05.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T73uZF4gZYWYnWiaPSejYolK2z0='} 2023-09-22T21:35:05.413 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000891b0d5f79ac1448-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.413 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:05.413 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:05.413 DEBUG:boto:Method: HEAD 2023-09-22T21:35:05.413 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:05.414 DEBUG:boto:Data: 2023-09-22T21:35:05.414 DEBUG:boto:Headers: {} 2023-09-22T21:35:05.414 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.414 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.414 DEBUG:boto:Params: {} 2023-09-22T21:35:05.414 DEBUG:boto:Token: None 2023-09-22T21:35:05.414 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/ 2023-09-22T21:35:05.414 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU= 2023-09-22T21:35:05.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU=', 'Content-Length': '0'} 2023-09-22T21:35:05.423 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ab60036daba9f22-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.423 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:35:05.423 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:35:05.423 DEBUG:boto:Method: PUT 2023-09-22T21:35:05.423 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000036 2023-09-22T21:35:05.423 DEBUG:boto:Data: 2023-09-22T21:35:05.424 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:05.424 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.424 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.424 DEBUG:boto:Params: {} 2023-09-22T21:35:05.424 DEBUG:boto:Token: None 2023-09-22T21:35:05.424 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/obj00000000000000000036 2023-09-22T21:35:05.424 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VzslItDfyLj0h485Zs/FwTrP/UI= 2023-09-22T21:35:05.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VzslItDfyLj0h485Zs/FwTrP/UI='} 2023-09-22T21:35:05.467 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6098c15817446a0-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.467 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:05.467 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:05.467 DEBUG:boto:Method: HEAD 2023-09-22T21:35:05.467 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:05.468 DEBUG:boto:Data: 2023-09-22T21:35:05.468 DEBUG:boto:Headers: {} 2023-09-22T21:35:05.468 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.468 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.468 DEBUG:boto:Params: {} 2023-09-22T21:35:05.468 DEBUG:boto:Token: None 2023-09-22T21:35:05.469 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/ 2023-09-22T21:35:05.469 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU= 2023-09-22T21:35:05.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU=', 'Content-Length': '0'} 2023-09-22T21:35:05.477 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d377bd336671332-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.477 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:35:05.477 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:35:05.478 DEBUG:boto:Method: PUT 2023-09-22T21:35:05.478 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000015 2023-09-22T21:35:05.478 DEBUG:boto:Data: 2023-09-22T21:35:05.478 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:05.478 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.478 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.479 DEBUG:boto:Params: {} 2023-09-22T21:35:05.479 DEBUG:boto:Token: None 2023-09-22T21:35:05.479 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/obj00000000000000000015 2023-09-22T21:35:05.479 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SiKR6qjT0CZC1+Q7DjiDzt3sUEw= 2023-09-22T21:35:05.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SiKR6qjT0CZC1+Q7DjiDzt3sUEw='} 2023-09-22T21:35:05.497 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000627be88e5fd3904e-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.498 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:05.498 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:05.498 DEBUG:boto:Method: HEAD 2023-09-22T21:35:05.498 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:05.498 DEBUG:boto:Data: 2023-09-22T21:35:05.498 DEBUG:boto:Headers: {} 2023-09-22T21:35:05.499 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.499 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.499 DEBUG:boto:Params: {} 2023-09-22T21:35:05.499 DEBUG:boto:Token: None 2023-09-22T21:35:05.499 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/ 2023-09-22T21:35:05.499 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU= 2023-09-22T21:35:05.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU=', 'Content-Length': '0'} 2023-09-22T21:35:05.509 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b596133249372b8c-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.509 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:35:05.509 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:35:05.509 DEBUG:boto:Method: PUT 2023-09-22T21:35:05.510 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010 2023-09-22T21:35:05.510 DEBUG:boto:Data: 2023-09-22T21:35:05.510 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:05.510 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.510 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.511 DEBUG:boto:Params: {} 2023-09-22T21:35:05.511 DEBUG:boto:Token: None 2023-09-22T21:35:05.511 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/obj00000000000000000010 2023-09-22T21:35:05.511 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:19+yrsxXhk7n0Y3xqsO1Isqr/p4= 2023-09-22T21:35:05.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:19+yrsxXhk7n0Y3xqsO1Isqr/p4='} 2023-09-22T21:35:05.554 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006bbcdfa24b0a4194-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.555 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:05.555 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:05.555 DEBUG:boto:Method: HEAD 2023-09-22T21:35:05.555 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:05.555 DEBUG:boto:Data: 2023-09-22T21:35:05.556 DEBUG:boto:Headers: {} 2023-09-22T21:35:05.556 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.556 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.556 DEBUG:boto:Params: {} 2023-09-22T21:35:05.556 DEBUG:boto:Token: None 2023-09-22T21:35:05.556 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/ 2023-09-22T21:35:05.557 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU= 2023-09-22T21:35:05.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU=', 'Content-Length': '0'} 2023-09-22T21:35:05.565 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007392976a6bdd0313-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.565 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:35:05.566 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:35:05.566 DEBUG:boto:Method: PUT 2023-09-22T21:35:05.566 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000 2023-09-22T21:35:05.566 DEBUG:boto:Data: 2023-09-22T21:35:05.566 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:05.566 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.567 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.567 DEBUG:boto:Params: {} 2023-09-22T21:35:05.567 DEBUG:boto:Token: None 2023-09-22T21:35:05.567 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/obj00000000000000000000 2023-09-22T21:35:05.567 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:p3SPSoVqgjAgNBfIjdnpKgvUNws= 2023-09-22T21:35:05.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:p3SPSoVqgjAgNBfIjdnpKgvUNws='} 2023-09-22T21:35:05.585 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d357415abdbe481-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.586 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:05.586 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:05.586 DEBUG:boto:Method: HEAD 2023-09-22T21:35:05.586 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:05.587 DEBUG:boto:Data: 2023-09-22T21:35:05.587 DEBUG:boto:Headers: {} 2023-09-22T21:35:05.587 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.587 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.587 DEBUG:boto:Params: {} 2023-09-22T21:35:05.587 DEBUG:boto:Token: None 2023-09-22T21:35:05.588 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/ 2023-09-22T21:35:05.588 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU= 2023-09-22T21:35:05.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU=', 'Content-Length': '0'} 2023-09-22T21:35:05.596 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a69f69cb3d3a2dc7-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.596 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:35:05.596 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:35:05.596 DEBUG:boto:Method: PUT 2023-09-22T21:35:05.597 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000021 2023-09-22T21:35:05.597 DEBUG:boto:Data: 2023-09-22T21:35:05.597 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:05.597 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.597 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.597 DEBUG:boto:Params: {} 2023-09-22T21:35:05.597 DEBUG:boto:Token: None 2023-09-22T21:35:05.598 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/obj00000000000000000021 2023-09-22T21:35:05.598 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fvaDtNOJjqkzEtemOuKYVPZH7tA= 2023-09-22T21:35:05.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fvaDtNOJjqkzEtemOuKYVPZH7tA='} 2023-09-22T21:35:05.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb2df1ba3dfd6ea9-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.643 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:05.643 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:05.643 DEBUG:boto:Method: HEAD 2023-09-22T21:35:05.643 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:05.643 DEBUG:boto:Data: 2023-09-22T21:35:05.643 DEBUG:boto:Headers: {} 2023-09-22T21:35:05.644 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.644 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.644 DEBUG:boto:Params: {} 2023-09-22T21:35:05.644 DEBUG:boto:Token: None 2023-09-22T21:35:05.644 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/ 2023-09-22T21:35:05.644 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU= 2023-09-22T21:35:05.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU=', 'Content-Length': '0'} 2023-09-22T21:35:05.653 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f0d2c37176f3dfc-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.653 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:35:05.653 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:35:05.653 DEBUG:boto:Method: PUT 2023-09-22T21:35:05.654 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000057 2023-09-22T21:35:05.654 DEBUG:boto:Data: 2023-09-22T21:35:05.654 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:05.654 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.654 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.654 DEBUG:boto:Params: {} 2023-09-22T21:35:05.655 DEBUG:boto:Token: None 2023-09-22T21:35:05.655 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/obj00000000000000000057 2023-09-22T21:35:05.655 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XvXgTNN9rCAZTaLwNK03DBstM0Q= 2023-09-22T21:35:05.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XvXgTNN9rCAZTaLwNK03DBstM0Q='} 2023-09-22T21:35:05.667 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000ddfa201f28c2a75-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.667 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:05.667 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:05.668 DEBUG:boto:Method: HEAD 2023-09-22T21:35:05.668 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:05.668 DEBUG:boto:Data: 2023-09-22T21:35:05.668 DEBUG:boto:Headers: {} 2023-09-22T21:35:05.668 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.668 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.668 DEBUG:boto:Params: {} 2023-09-22T21:35:05.668 DEBUG:boto:Token: None 2023-09-22T21:35:05.668 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/ 2023-09-22T21:35:05.668 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU= 2023-09-22T21:35:05.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU=', 'Content-Length': '0'} 2023-09-22T21:35:05.677 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e6f9135038fd1d7-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.677 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:35:05.677 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:35:05.677 DEBUG:boto:Method: PUT 2023-09-22T21:35:05.677 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000019 2023-09-22T21:35:05.677 DEBUG:boto:Data: 2023-09-22T21:35:05.678 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:05.678 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.678 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.678 DEBUG:boto:Params: {} 2023-09-22T21:35:05.678 DEBUG:boto:Token: None 2023-09-22T21:35:05.678 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/obj00000000000000000019 2023-09-22T21:35:05.678 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SDrqfYQiiQPpOtAeIPM/OPFbMSY= 2023-09-22T21:35:05.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SDrqfYQiiQPpOtAeIPM/OPFbMSY='} 2023-09-22T21:35:05.722 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000062ae47c3b51b7679-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.723 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:05.723 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:05.723 DEBUG:boto:Method: HEAD 2023-09-22T21:35:05.723 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:05.723 DEBUG:boto:Data: 2023-09-22T21:35:05.723 DEBUG:boto:Headers: {} 2023-09-22T21:35:05.723 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.723 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.724 DEBUG:boto:Params: {} 2023-09-22T21:35:05.724 DEBUG:boto:Token: None 2023-09-22T21:35:05.724 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/ 2023-09-22T21:35:05.724 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU= 2023-09-22T21:35:05.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU=', 'Content-Length': '0'} 2023-09-22T21:35:05.733 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3c668b23ef104d0-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.733 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:35:05.733 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:35:05.733 DEBUG:boto:Method: PUT 2023-09-22T21:35:05.733 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000017 2023-09-22T21:35:05.733 DEBUG:boto:Data: 2023-09-22T21:35:05.734 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:05.734 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.734 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.734 DEBUG:boto:Params: {} 2023-09-22T21:35:05.734 DEBUG:boto:Token: None 2023-09-22T21:35:05.734 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/obj00000000000000000017 2023-09-22T21:35:05.734 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tKVHkVd21VSOJaOmy1LYHCEgnAA= 2023-09-22T21:35:05.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tKVHkVd21VSOJaOmy1LYHCEgnAA='} 2023-09-22T21:35:05.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c27d6dc7e7722a26-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.751 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:05.751 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:05.751 DEBUG:boto:Method: HEAD 2023-09-22T21:35:05.751 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:05.751 DEBUG:boto:Data: 2023-09-22T21:35:05.751 DEBUG:boto:Headers: {} 2023-09-22T21:35:05.752 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.752 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.752 DEBUG:boto:Params: {} 2023-09-22T21:35:05.752 DEBUG:boto:Token: None 2023-09-22T21:35:05.752 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/ 2023-09-22T21:35:05.752 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU= 2023-09-22T21:35:05.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ag3GLfkHaJYcT7f2PmbNHqQOuHU=', 'Content-Length': '0'} 2023-09-22T21:35:05.761 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d541b3b8441be9e-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:05.761 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:35:05.761 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:35:05.762 DEBUG:boto:Method: PUT 2023-09-22T21:35:05.762 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000024 2023-09-22T21:35:05.762 DEBUG:boto:Data: 2023-09-22T21:35:05.762 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:05.762 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:05.762 DEBUG:boto:Port: 8000 2023-09-22T21:35:05.763 DEBUG:boto:Params: {} 2023-09-22T21:35:05.763 DEBUG:boto:Token: None 2023-09-22T21:35:05.763 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:05 GMT /ffzdqu-58/obj00000000000000000024 2023-09-22T21:35:05.763 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y3JCyv9S66HYlclYpvlXiNWj73w= 2023-09-22T21:35:05.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:05 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y3JCyv9S66HYlclYpvlXiNWj73w='} 2023-09-22T21:35:05.806 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c370b0d3293341f-00650e0889-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:09.650 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:09.651 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:09.652 DEBUG:boto:Method: HEAD 2023-09-22T21:35:09.652 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:09.652 DEBUG:boto:Data: 2023-09-22T21:35:09.652 DEBUG:boto:Headers: {} 2023-09-22T21:35:09.652 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:09.652 DEBUG:boto:Port: 8000 2023-09-22T21:35:09.652 DEBUG:boto:Params: {} 2023-09-22T21:35:09.653 DEBUG:boto:Token: None 2023-09-22T21:35:09.653 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:09 GMT /ffzdqu-58/ 2023-09-22T21:35:09.653 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BjWo+AkwlZq9oN4QuCuRWSJeC+k= 2023-09-22T21:35:09.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BjWo+AkwlZq9oN4QuCuRWSJeC+k=', 'Content-Length': '0'} 2023-09-22T21:35:09.662 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044879f7835cfcd93-00650e088d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:09.663 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:35:09.663 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:35:09.663 DEBUG:boto:Method: PUT 2023-09-22T21:35:09.663 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000014 2023-09-22T21:35:09.663 DEBUG:boto:Data: 2023-09-22T21:35:09.663 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:09.663 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:09.663 DEBUG:boto:Port: 8000 2023-09-22T21:35:09.663 DEBUG:boto:Params: {} 2023-09-22T21:35:09.664 DEBUG:boto:Token: None 2023-09-22T21:35:09.664 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:09 GMT /ffzdqu-58/obj00000000000000000014 2023-09-22T21:35:09.664 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EB2SrXKt0zJCuF3tRoypnsqyTsE= 2023-09-22T21:35:09.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EB2SrXKt0zJCuF3tRoypnsqyTsE='} 2023-09-22T21:35:09.681 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba41aae67172265e-00650e088d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:09.681 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:09.681 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:09.681 DEBUG:boto:Method: HEAD 2023-09-22T21:35:09.681 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:09.681 DEBUG:boto:Data: 2023-09-22T21:35:09.682 DEBUG:boto:Headers: {} 2023-09-22T21:35:09.682 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:09.682 DEBUG:boto:Port: 8000 2023-09-22T21:35:09.682 DEBUG:boto:Params: {} 2023-09-22T21:35:09.682 DEBUG:boto:Token: None 2023-09-22T21:35:09.682 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:09 GMT /ffzdqu-58/ 2023-09-22T21:35:09.682 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BjWo+AkwlZq9oN4QuCuRWSJeC+k= 2023-09-22T21:35:09.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BjWo+AkwlZq9oN4QuCuRWSJeC+k=', 'Content-Length': '0'} 2023-09-22T21:35:09.691 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d82563d39d0d362-00650e088d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:09.691 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:35:09.778 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:35:09.778 DEBUG:boto:Method: PUT 2023-09-22T21:35:09.778 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003 2023-09-22T21:35:09.778 DEBUG:boto:Data: 2023-09-22T21:35:09.779 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:09.779 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:09.779 DEBUG:boto:Port: 8000 2023-09-22T21:35:09.779 DEBUG:boto:Params: {} 2023-09-22T21:35:09.779 DEBUG:boto:Token: None 2023-09-22T21:35:09.779 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:09 GMT /ffzdqu-58/obj00000000000000000003 2023-09-22T21:35:09.779 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nSB/7aRqACN/y/l0ompdfn7OZKo= 2023-09-22T21:35:09.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nSB/7aRqACN/y/l0ompdfn7OZKo='} 2023-09-22T21:35:09.796 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000023e8cf370355b307-00650e088d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:09.796 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:09.796 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:09.796 DEBUG:boto:Method: HEAD 2023-09-22T21:35:09.796 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:09.796 DEBUG:boto:Data: 2023-09-22T21:35:09.796 DEBUG:boto:Headers: {} 2023-09-22T21:35:09.797 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:09.797 DEBUG:boto:Port: 8000 2023-09-22T21:35:09.797 DEBUG:boto:Params: {} 2023-09-22T21:35:09.797 DEBUG:boto:Token: None 2023-09-22T21:35:09.797 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:09 GMT /ffzdqu-58/ 2023-09-22T21:35:09.797 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BjWo+AkwlZq9oN4QuCuRWSJeC+k= 2023-09-22T21:35:09.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BjWo+AkwlZq9oN4QuCuRWSJeC+k=', 'Content-Length': '0'} 2023-09-22T21:35:09.805 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097e432a8ade58732-00650e088d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:09.965 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:35:09.965 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:35:09.965 DEBUG:boto:Method: PUT 2023-09-22T21:35:09.965 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000027 2023-09-22T21:35:09.965 DEBUG:boto:Data: 2023-09-22T21:35:09.965 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:09.966 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:09.966 DEBUG:boto:Port: 8000 2023-09-22T21:35:09.966 DEBUG:boto:Params: {} 2023-09-22T21:35:09.966 DEBUG:boto:Token: None 2023-09-22T21:35:09.966 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:09 GMT /ffzdqu-58/obj00000000000000000027 2023-09-22T21:35:09.966 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TIkGsI8duAd81FeielDyiCSg2XU= 2023-09-22T21:35:09.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TIkGsI8duAd81FeielDyiCSg2XU='} 2023-09-22T21:35:09.983 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4d279c3b929e972-00650e088d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:09.984 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:09.984 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:09.984 DEBUG:boto:Method: HEAD 2023-09-22T21:35:09.984 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:09.984 DEBUG:boto:Data: 2023-09-22T21:35:09.984 DEBUG:boto:Headers: {} 2023-09-22T21:35:09.984 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:09.985 DEBUG:boto:Port: 8000 2023-09-22T21:35:09.985 DEBUG:boto:Params: {} 2023-09-22T21:35:09.985 DEBUG:boto:Token: None 2023-09-22T21:35:09.985 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:09 GMT /ffzdqu-58/ 2023-09-22T21:35:09.985 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BjWo+AkwlZq9oN4QuCuRWSJeC+k= 2023-09-22T21:35:09.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BjWo+AkwlZq9oN4QuCuRWSJeC+k=', 'Content-Length': '0'} 2023-09-22T21:35:09.993 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c0ebbb02e4d9ac98-00650e088d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:11.716 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:35:11.717 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:35:11.717 DEBUG:boto:Method: PUT 2023-09-22T21:35:11.717 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000007 2023-09-22T21:35:11.717 DEBUG:boto:Data: 2023-09-22T21:35:11.718 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:11.718 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:11.718 DEBUG:boto:Port: 8000 2023-09-22T21:35:11.718 DEBUG:boto:Params: {} 2023-09-22T21:35:11.718 DEBUG:boto:Token: None 2023-09-22T21:35:11.718 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:11 GMT /ffzdqu-58/obj00000000000000000007 2023-09-22T21:35:11.719 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:O73gVJlTPxGdJAp7i++3VqVZrN0= 2023-09-22T21:35:11.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:O73gVJlTPxGdJAp7i++3VqVZrN0='} 2023-09-22T21:35:11.736 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aab1f9616a11b6f6-00650e088f-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:11.868 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:11.868 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:11.868 DEBUG:boto:Method: HEAD 2023-09-22T21:35:11.868 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:11.868 DEBUG:boto:Data: 2023-09-22T21:35:11.868 DEBUG:boto:Headers: {} 2023-09-22T21:35:11.868 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:11.869 DEBUG:boto:Port: 8000 2023-09-22T21:35:11.869 DEBUG:boto:Params: {} 2023-09-22T21:35:11.869 DEBUG:boto:Token: None 2023-09-22T21:35:11.869 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:11 GMT /ffzdqu-58/ 2023-09-22T21:35:11.869 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:d01LK+MVO0Ld/HOJXKh+apjs+cI= 2023-09-22T21:35:11.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:d01LK+MVO0Ld/HOJXKh+apjs+cI=', 'Content-Length': '0'} 2023-09-22T21:35:11.878 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3ceb5d0a0cdfb93-00650e088f-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:11.878 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:35:11.878 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:35:11.879 DEBUG:boto:Method: PUT 2023-09-22T21:35:11.879 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012 2023-09-22T21:35:11.879 DEBUG:boto:Data: 2023-09-22T21:35:11.879 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:11.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:11.879 DEBUG:boto:Port: 8000 2023-09-22T21:35:11.879 DEBUG:boto:Params: {} 2023-09-22T21:35:11.879 DEBUG:boto:Token: None 2023-09-22T21:35:11.880 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:11 GMT /ffzdqu-58/obj00000000000000000012 2023-09-22T21:35:11.880 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/693PripC9ywFQOjcbqUdIYUA44= 2023-09-22T21:35:11.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/693PripC9ywFQOjcbqUdIYUA44='} 2023-09-22T21:35:11.896 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007cd34c20350a6b2d-00650e088f-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:11.918 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:11.919 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:11.919 DEBUG:boto:Method: HEAD 2023-09-22T21:35:11.919 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:11.919 DEBUG:boto:Data: 2023-09-22T21:35:11.919 DEBUG:boto:Headers: {} 2023-09-22T21:35:11.919 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:11.919 DEBUG:boto:Port: 8000 2023-09-22T21:35:11.920 DEBUG:boto:Params: {} 2023-09-22T21:35:11.920 DEBUG:boto:Token: None 2023-09-22T21:35:11.920 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:11 GMT /ffzdqu-58/ 2023-09-22T21:35:11.920 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:d01LK+MVO0Ld/HOJXKh+apjs+cI= 2023-09-22T21:35:11.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:d01LK+MVO0Ld/HOJXKh+apjs+cI=', 'Content-Length': '0'} 2023-09-22T21:35:11.929 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000872d6e6c52a5856a-00650e088f-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:11.929 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:35:11.929 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:35:11.929 DEBUG:boto:Method: PUT 2023-09-22T21:35:11.930 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000030 2023-09-22T21:35:11.930 DEBUG:boto:Data: 2023-09-22T21:35:11.930 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:11.930 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:11.930 DEBUG:boto:Port: 8000 2023-09-22T21:35:11.930 DEBUG:boto:Params: {} 2023-09-22T21:35:11.930 DEBUG:boto:Token: None 2023-09-22T21:35:11.930 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:11 GMT /ffzdqu-58/obj00000000000000000030 2023-09-22T21:35:11.931 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kpIhU7RnQXXVgIgsnupv54vIAxk= 2023-09-22T21:35:11.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kpIhU7RnQXXVgIgsnupv54vIAxk='} 2023-09-22T21:35:11.974 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091bd0bd0ddb3be72-00650e088f-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:11.975 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:11.975 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:11.975 DEBUG:boto:Method: HEAD 2023-09-22T21:35:11.975 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:11.975 DEBUG:boto:Data: 2023-09-22T21:35:11.975 DEBUG:boto:Headers: {} 2023-09-22T21:35:11.976 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:11.976 DEBUG:boto:Port: 8000 2023-09-22T21:35:11.976 DEBUG:boto:Params: {} 2023-09-22T21:35:11.976 DEBUG:boto:Token: None 2023-09-22T21:35:11.976 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:11 GMT /ffzdqu-58/ 2023-09-22T21:35:11.976 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:d01LK+MVO0Ld/HOJXKh+apjs+cI= 2023-09-22T21:35:11.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:d01LK+MVO0Ld/HOJXKh+apjs+cI=', 'Content-Length': '0'} 2023-09-22T21:35:11.984 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c07cb7d9764b803-00650e088f-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:11.984 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:35:11.984 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:35:11.984 DEBUG:boto:Method: PUT 2023-09-22T21:35:11.984 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005 2023-09-22T21:35:11.985 DEBUG:boto:Data: 2023-09-22T21:35:11.985 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:11.985 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:11.985 DEBUG:boto:Port: 8000 2023-09-22T21:35:11.985 DEBUG:boto:Params: {} 2023-09-22T21:35:11.985 DEBUG:boto:Token: None 2023-09-22T21:35:11.985 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:11 GMT /ffzdqu-58/obj00000000000000000005 2023-09-22T21:35:11.985 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S/d3pQGkxdMxt37MzEt/2N9NqiA= 2023-09-22T21:35:11.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S/d3pQGkxdMxt37MzEt/2N9NqiA='} 2023-09-22T21:35:12.002 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d943e3b98b2b2f77-00650e088f-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.003 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:12.003 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:12.003 DEBUG:boto:Method: HEAD 2023-09-22T21:35:12.003 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:12.003 DEBUG:boto:Data: 2023-09-22T21:35:12.003 DEBUG:boto:Headers: {} 2023-09-22T21:35:12.003 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.003 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.004 DEBUG:boto:Params: {} 2023-09-22T21:35:12.004 DEBUG:boto:Token: None 2023-09-22T21:35:12.004 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/ 2023-09-22T21:35:12.004 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8= 2023-09-22T21:35:12.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8=', 'Content-Length': '0'} 2023-09-22T21:35:12.011 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dcced5b4ab28d84d-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.012 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:35:12.012 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:35:12.012 DEBUG:boto:Method: PUT 2023-09-22T21:35:12.012 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000039 2023-09-22T21:35:12.012 DEBUG:boto:Data: 2023-09-22T21:35:12.012 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:12.012 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.013 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.013 DEBUG:boto:Params: {} 2023-09-22T21:35:12.013 DEBUG:boto:Token: None 2023-09-22T21:35:12.013 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/obj00000000000000000039 2023-09-22T21:35:12.013 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E2fPlTvaRr1DJLZFO5Dp9/RvsbY= 2023-09-22T21:35:12.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E2fPlTvaRr1DJLZFO5Dp9/RvsbY='} 2023-09-22T21:35:12.059 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b675f03d180e0ea-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.059 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:12.059 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:12.059 DEBUG:boto:Method: HEAD 2023-09-22T21:35:12.059 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:12.059 DEBUG:boto:Data: 2023-09-22T21:35:12.060 DEBUG:boto:Headers: {} 2023-09-22T21:35:12.060 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.060 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.060 DEBUG:boto:Params: {} 2023-09-22T21:35:12.060 DEBUG:boto:Token: None 2023-09-22T21:35:12.060 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/ 2023-09-22T21:35:12.060 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8= 2023-09-22T21:35:12.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8=', 'Content-Length': '0'} 2023-09-22T21:35:12.069 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019bf7c8a6e153b98-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.069 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:35:12.069 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:35:12.069 DEBUG:boto:Method: PUT 2023-09-22T21:35:12.069 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011 2023-09-22T21:35:12.069 DEBUG:boto:Data: 2023-09-22T21:35:12.070 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:12.070 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.070 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.070 DEBUG:boto:Params: {} 2023-09-22T21:35:12.070 DEBUG:boto:Token: None 2023-09-22T21:35:12.070 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/obj00000000000000000011 2023-09-22T21:35:12.070 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:utc69eNObV8v6ILJTosY9GU7vyA= 2023-09-22T21:35:12.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:utc69eNObV8v6ILJTosY9GU7vyA='} 2023-09-22T21:35:12.086 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d42460cf70119492-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.087 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:12.087 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:12.087 DEBUG:boto:Method: HEAD 2023-09-22T21:35:12.087 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:12.087 DEBUG:boto:Data: 2023-09-22T21:35:12.087 DEBUG:boto:Headers: {} 2023-09-22T21:35:12.087 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.088 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.088 DEBUG:boto:Params: {} 2023-09-22T21:35:12.088 DEBUG:boto:Token: None 2023-09-22T21:35:12.088 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/ 2023-09-22T21:35:12.088 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8= 2023-09-22T21:35:12.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8=', 'Content-Length': '0'} 2023-09-22T21:35:12.096 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c411d9cbcfaa77c-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.097 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:35:12.097 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:35:12.097 DEBUG:boto:Method: PUT 2023-09-22T21:35:12.097 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000065 2023-09-22T21:35:12.097 DEBUG:boto:Data: 2023-09-22T21:35:12.097 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:12.098 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.098 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.098 DEBUG:boto:Params: {} 2023-09-22T21:35:12.098 DEBUG:boto:Token: None 2023-09-22T21:35:12.098 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/obj00000000000000000065 2023-09-22T21:35:12.098 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GAGIwu/ohgIJW+LOf537dL2pD5Y= 2023-09-22T21:35:12.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GAGIwu/ohgIJW+LOf537dL2pD5Y='} 2023-09-22T21:35:12.143 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca0727938f554ada-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.143 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:12.143 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:12.143 DEBUG:boto:Method: HEAD 2023-09-22T21:35:12.143 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:12.143 DEBUG:boto:Data: 2023-09-22T21:35:12.143 DEBUG:boto:Headers: {} 2023-09-22T21:35:12.144 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.144 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.144 DEBUG:boto:Params: {} 2023-09-22T21:35:12.144 DEBUG:boto:Token: None 2023-09-22T21:35:12.144 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/ 2023-09-22T21:35:12.144 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8= 2023-09-22T21:35:12.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8=', 'Content-Length': '0'} 2023-09-22T21:35:12.152 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002a8e322b10c93d9-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.153 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:35:12.153 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:35:12.153 DEBUG:boto:Method: PUT 2023-09-22T21:35:12.153 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000040 2023-09-22T21:35:12.153 DEBUG:boto:Data: 2023-09-22T21:35:12.153 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:12.153 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.153 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.154 DEBUG:boto:Params: {} 2023-09-22T21:35:12.154 DEBUG:boto:Token: None 2023-09-22T21:35:12.154 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/obj00000000000000000040 2023-09-22T21:35:12.154 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F6mt4HMcDGLOHcYsipCVSa/mkfo= 2023-09-22T21:35:12.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F6mt4HMcDGLOHcYsipCVSa/mkfo='} 2023-09-22T21:35:12.170 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2b8353d8c81347f-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.171 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:12.171 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:12.171 DEBUG:boto:Method: HEAD 2023-09-22T21:35:12.171 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:12.171 DEBUG:boto:Data: 2023-09-22T21:35:12.171 DEBUG:boto:Headers: {} 2023-09-22T21:35:12.171 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.171 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.171 DEBUG:boto:Params: {} 2023-09-22T21:35:12.172 DEBUG:boto:Token: None 2023-09-22T21:35:12.172 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/ 2023-09-22T21:35:12.172 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8= 2023-09-22T21:35:12.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8=', 'Content-Length': '0'} 2023-09-22T21:35:12.180 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5af303c6a5652a9-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.180 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:35:12.180 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:35:12.180 DEBUG:boto:Method: PUT 2023-09-22T21:35:12.180 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000053 2023-09-22T21:35:12.180 DEBUG:boto:Data: 2023-09-22T21:35:12.181 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:12.181 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.181 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.181 DEBUG:boto:Params: {} 2023-09-22T21:35:12.181 DEBUG:boto:Token: None 2023-09-22T21:35:12.181 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/obj00000000000000000053 2023-09-22T21:35:12.181 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3+ISkza7J0IwyrlFyoUC7lkJMDs= 2023-09-22T21:35:12.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3+ISkza7J0IwyrlFyoUC7lkJMDs='} 2023-09-22T21:35:12.231 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f90aabe5f3103412-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.231 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:12.231 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:12.231 DEBUG:boto:Method: HEAD 2023-09-22T21:35:12.231 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:12.231 DEBUG:boto:Data: 2023-09-22T21:35:12.231 DEBUG:boto:Headers: {} 2023-09-22T21:35:12.232 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.232 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.232 DEBUG:boto:Params: {} 2023-09-22T21:35:12.232 DEBUG:boto:Token: None 2023-09-22T21:35:12.232 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/ 2023-09-22T21:35:12.232 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8= 2023-09-22T21:35:12.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8=', 'Content-Length': '0'} 2023-09-22T21:35:12.241 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008377ce81cf504853-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.241 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:35:12.241 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:35:12.241 DEBUG:boto:Method: PUT 2023-09-22T21:35:12.241 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006 2023-09-22T21:35:12.241 DEBUG:boto:Data: 2023-09-22T21:35:12.241 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:12.241 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.242 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.242 DEBUG:boto:Params: {} 2023-09-22T21:35:12.242 DEBUG:boto:Token: None 2023-09-22T21:35:12.242 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/obj00000000000000000006 2023-09-22T21:35:12.242 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BcCc4rEaclTnnQ94ubbVGc8wTzQ= 2023-09-22T21:35:12.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BcCc4rEaclTnnQ94ubbVGc8wTzQ='} 2023-09-22T21:35:12.259 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000019a31a4fdf9ce712-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.259 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:12.259 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:12.260 DEBUG:boto:Method: HEAD 2023-09-22T21:35:12.260 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:12.260 DEBUG:boto:Data: 2023-09-22T21:35:12.260 DEBUG:boto:Headers: {} 2023-09-22T21:35:12.260 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.260 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.260 DEBUG:boto:Params: {} 2023-09-22T21:35:12.260 DEBUG:boto:Token: None 2023-09-22T21:35:12.260 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/ 2023-09-22T21:35:12.261 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8= 2023-09-22T21:35:12.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8=', 'Content-Length': '0'} 2023-09-22T21:35:12.268 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007872a4727b4be463-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.268 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:35:12.269 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:35:12.269 DEBUG:boto:Method: PUT 2023-09-22T21:35:12.269 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000018 2023-09-22T21:35:12.269 DEBUG:boto:Data: 2023-09-22T21:35:12.269 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:12.269 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.269 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.269 DEBUG:boto:Params: {} 2023-09-22T21:35:12.269 DEBUG:boto:Token: None 2023-09-22T21:35:12.270 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/obj00000000000000000018 2023-09-22T21:35:12.270 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D/OXUIW1Tb3gClHbrzJyu01n1+Y= 2023-09-22T21:35:12.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D/OXUIW1Tb3gClHbrzJyu01n1+Y='} 2023-09-22T21:35:12.315 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d62301d950144182-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.315 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:12.315 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:12.315 DEBUG:boto:Method: HEAD 2023-09-22T21:35:12.315 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:12.315 DEBUG:boto:Data: 2023-09-22T21:35:12.315 DEBUG:boto:Headers: {} 2023-09-22T21:35:12.316 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.316 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.316 DEBUG:boto:Params: {} 2023-09-22T21:35:12.316 DEBUG:boto:Token: None 2023-09-22T21:35:12.316 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/ 2023-09-22T21:35:12.316 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8= 2023-09-22T21:35:12.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8=', 'Content-Length': '0'} 2023-09-22T21:35:12.324 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c45f23c1f2cff5fe-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.324 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:35:12.325 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:35:12.325 DEBUG:boto:Method: PUT 2023-09-22T21:35:12.325 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000008 2023-09-22T21:35:12.325 DEBUG:boto:Data: 2023-09-22T21:35:12.325 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:12.325 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.325 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.325 DEBUG:boto:Params: {} 2023-09-22T21:35:12.325 DEBUG:boto:Token: None 2023-09-22T21:35:12.326 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/obj00000000000000000008 2023-09-22T21:35:12.326 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gLdw9BSwCx8BvkZumuVrrOcsW5s= 2023-09-22T21:35:12.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gLdw9BSwCx8BvkZumuVrrOcsW5s='} 2023-09-22T21:35:12.342 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c6b09db0cfc242cf-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.343 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:12.343 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:12.343 DEBUG:boto:Method: HEAD 2023-09-22T21:35:12.343 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:12.343 DEBUG:boto:Data: 2023-09-22T21:35:12.343 DEBUG:boto:Headers: {} 2023-09-22T21:35:12.343 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.343 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.343 DEBUG:boto:Params: {} 2023-09-22T21:35:12.344 DEBUG:boto:Token: None 2023-09-22T21:35:12.344 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/ 2023-09-22T21:35:12.344 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8= 2023-09-22T21:35:12.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8=', 'Content-Length': '0'} 2023-09-22T21:35:12.351 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eeb6e4149798326b-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.352 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:35:12.352 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:35:12.352 DEBUG:boto:Method: PUT 2023-09-22T21:35:12.352 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000045 2023-09-22T21:35:12.352 DEBUG:boto:Data: 2023-09-22T21:35:12.352 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:12.352 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.352 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.352 DEBUG:boto:Params: {} 2023-09-22T21:35:12.353 DEBUG:boto:Token: None 2023-09-22T21:35:12.353 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/obj00000000000000000045 2023-09-22T21:35:12.353 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HXVSirwgc2nKzmadpnhhKTEgdAs= 2023-09-22T21:35:12.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HXVSirwgc2nKzmadpnhhKTEgdAs='} 2023-09-22T21:35:12.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c7977426323df94-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.403 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:12.403 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:12.403 DEBUG:boto:Method: HEAD 2023-09-22T21:35:12.403 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:12.403 DEBUG:boto:Data: 2023-09-22T21:35:12.403 DEBUG:boto:Headers: {} 2023-09-22T21:35:12.403 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.404 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.404 DEBUG:boto:Params: {} 2023-09-22T21:35:12.404 DEBUG:boto:Token: None 2023-09-22T21:35:12.404 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/ 2023-09-22T21:35:12.404 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8= 2023-09-22T21:35:12.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8=', 'Content-Length': '0'} 2023-09-22T21:35:12.414 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff1bb3836ae20652-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.414 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:35:12.414 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:35:12.414 DEBUG:boto:Method: PUT 2023-09-22T21:35:12.414 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000004 2023-09-22T21:35:12.415 DEBUG:boto:Data: 2023-09-22T21:35:12.415 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:12.415 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.415 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.415 DEBUG:boto:Params: {} 2023-09-22T21:35:12.415 DEBUG:boto:Token: None 2023-09-22T21:35:12.415 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/obj00000000000000000004 2023-09-22T21:35:12.415 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Nvy9Gbaj5/629Nqn51HM/JzL0RI= 2023-09-22T21:35:12.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Nvy9Gbaj5/629Nqn51HM/JzL0RI='} 2023-09-22T21:35:12.432 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ba31990733cc074-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.432 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:12.433 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:12.433 DEBUG:boto:Method: HEAD 2023-09-22T21:35:12.433 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:12.433 DEBUG:boto:Data: 2023-09-22T21:35:12.433 DEBUG:boto:Headers: {} 2023-09-22T21:35:12.433 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.433 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.433 DEBUG:boto:Params: {} 2023-09-22T21:35:12.433 DEBUG:boto:Token: None 2023-09-22T21:35:12.434 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/ 2023-09-22T21:35:12.434 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8= 2023-09-22T21:35:12.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8=', 'Content-Length': '0'} 2023-09-22T21:35:12.442 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000709a392bf7d7c676-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.442 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:35:12.442 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:35:12.442 DEBUG:boto:Method: PUT 2023-09-22T21:35:12.443 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002 2023-09-22T21:35:12.443 DEBUG:boto:Data: 2023-09-22T21:35:12.443 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:12.443 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.443 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.443 DEBUG:boto:Params: {} 2023-09-22T21:35:12.443 DEBUG:boto:Token: None 2023-09-22T21:35:12.443 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/obj00000000000000000002 2023-09-22T21:35:12.443 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FVMJofZKIfXAxyb4Q55Ys7PJdzc= 2023-09-22T21:35:12.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FVMJofZKIfXAxyb4Q55Ys7PJdzc='} 2023-09-22T21:35:12.487 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac2df8804b7612cc-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.487 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:12.487 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:12.487 DEBUG:boto:Method: HEAD 2023-09-22T21:35:12.487 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:12.487 DEBUG:boto:Data: 2023-09-22T21:35:12.487 DEBUG:boto:Headers: {} 2023-09-22T21:35:12.488 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.488 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.488 DEBUG:boto:Params: {} 2023-09-22T21:35:12.488 DEBUG:boto:Token: None 2023-09-22T21:35:12.488 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/ 2023-09-22T21:35:12.488 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8= 2023-09-22T21:35:12.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8=', 'Content-Length': '0'} 2023-09-22T21:35:12.496 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a7df3229bd38df1-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.497 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:35:12.497 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:35:12.497 DEBUG:boto:Method: PUT 2023-09-22T21:35:12.497 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000020 2023-09-22T21:35:12.497 DEBUG:boto:Data: 2023-09-22T21:35:12.497 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:12.497 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.498 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.498 DEBUG:boto:Params: {} 2023-09-22T21:35:12.498 DEBUG:boto:Token: None 2023-09-22T21:35:12.498 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/obj00000000000000000020 2023-09-22T21:35:12.498 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Cdwq6UxyyWywd9O1bJW11SfgZPo= 2023-09-22T21:35:12.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Cdwq6UxyyWywd9O1bJW11SfgZPo='} 2023-09-22T21:35:12.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d44830894db7ce7b-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.515 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:12.516 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:12.516 DEBUG:boto:Method: HEAD 2023-09-22T21:35:12.516 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:12.516 DEBUG:boto:Data: 2023-09-22T21:35:12.516 DEBUG:boto:Headers: {} 2023-09-22T21:35:12.516 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.516 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.516 DEBUG:boto:Params: {} 2023-09-22T21:35:12.516 DEBUG:boto:Token: None 2023-09-22T21:35:12.517 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/ 2023-09-22T21:35:12.517 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8= 2023-09-22T21:35:12.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8=', 'Content-Length': '0'} 2023-09-22T21:35:12.526 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6c5dfc0d6b5dd94-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.526 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:35:12.527 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:35:12.527 DEBUG:boto:Method: PUT 2023-09-22T21:35:12.527 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001 2023-09-22T21:35:12.527 DEBUG:boto:Data: 2023-09-22T21:35:12.527 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:12.527 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.527 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.527 DEBUG:boto:Params: {} 2023-09-22T21:35:12.528 DEBUG:boto:Token: None 2023-09-22T21:35:12.528 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/obj00000000000000000001 2023-09-22T21:35:12.528 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dMWn5QuFxgqq+m+RT4nDSdGRGKs= 2023-09-22T21:35:12.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dMWn5QuFxgqq+m+RT4nDSdGRGKs='} 2023-09-22T21:35:12.570 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b03df04c23e11bef-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.571 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:35:12.571 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:35:12.571 DEBUG:boto:Method: HEAD 2023-09-22T21:35:12.571 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:35:12.571 DEBUG:boto:Data: 2023-09-22T21:35:12.571 DEBUG:boto:Headers: {} 2023-09-22T21:35:12.571 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.571 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.572 DEBUG:boto:Params: {} 2023-09-22T21:35:12.572 DEBUG:boto:Token: None 2023-09-22T21:35:12.572 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/ 2023-09-22T21:35:12.572 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8= 2023-09-22T21:35:12.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y6hvUhW/g+gKV5WS2kGT3Hc00F8=', 'Content-Length': '0'} 2023-09-22T21:35:12.580 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000828fc3a1ceac82a5-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.580 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:35:12.581 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:35:12.581 DEBUG:boto:Method: PUT 2023-09-22T21:35:12.581 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000026 2023-09-22T21:35:12.581 DEBUG:boto:Data: 2023-09-22T21:35:12.581 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:35:12.581 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:35:12.581 DEBUG:boto:Port: 8000 2023-09-22T21:35:12.581 DEBUG:boto:Params: {} 2023-09-22T21:35:12.581 DEBUG:boto:Token: None 2023-09-22T21:35:12.582 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:35:12 GMT /ffzdqu-58/obj00000000000000000026 2023-09-22T21:35:12.582 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+5hgVV+OpquljkmSJhQHea1r+1k= 2023-09-22T21:35:12.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:35:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+5hgVV+OpquljkmSJhQHea1r+1k='} 2023-09-22T21:35:12.599 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007697d982fc34aac1-00650e0890-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:35:12.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-58', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:35:12.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-58', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:35:12.600 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-58 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:35:12.645 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:35:12.646 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:35:12.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.728+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: sync status for source bucket shard: 2 2023-09-22T21:35:12.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: listing bilog for incremental sync; position=00000000001.71.6 2023-09-22T21:35:12.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: sync status for source bucket shard: 2 2023-09-22T21:35:12.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: listing bilog for incremental sync; position=00000000001.31.6 2023-09-22T21:35:12.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: sync status for source bucket shard: 2 2023-09-22T21:35:12.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: sync status for source bucket shard: 2 2023-09-22T21:35:12.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: listing bilog for incremental sync; position=00000000001.42.6 2023-09-22T21:35:12.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: sync status for source bucket shard: 2 2023-09-22T21:35:12.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: listing bilog for incremental sync; position=00000000001.30.6 2023-09-22T21:35:12.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: sync status for source bucket shard: 2 2023-09-22T21:35:12.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: listing bilog for incremental sync; position=00000000001.21.6 2023-09-22T21:35:12.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: sync status for source bucket shard: 2 2023-09-22T21:35:12.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: listing bilog for incremental sync; position=00000000001.43.6 2023-09-22T21:35:12.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: sync status for source bucket shard: 2 2023-09-22T21:35:12.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: sync status for source bucket shard: 2 2023-09-22T21:35:12.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: listing bilog for incremental sync; position=00000000001.53.6 2023-09-22T21:35:12.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: listing bilog for incremental sync; position=00000000001.28.6 2023-09-22T21:35:12.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: sync status for source bucket shard: 2 2023-09-22T21:35:12.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: listing bilog for incremental sync; position=00000000001.41.6 2023-09-22T21:35:12.738 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: sync status for source bucket shard: 2 2023-09-22T21:35:12.738 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.732+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: listing bilog for incremental sync; position=00000000001.45.6 2023-09-22T21:35:12.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.776+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.776+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.776+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.776+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: sync status for source bucket shard: 0 2023-09-22T21:35:12.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.776+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.781 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: sync status for source bucket shard: 0 2023-09-22T21:35:12.781 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.781 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: sync status for source bucket shard: 0 2023-09-22T21:35:12.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: sync status for source bucket shard: 0 2023-09-22T21:35:12.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: sync status for source bucket shard: 0 2023-09-22T21:35:12.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: sync status for source bucket shard: 0 2023-09-22T21:35:12.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: parsed entry: id=00000000001.47.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-09-22T21:35:12.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2/obj00000000000000000024 2023-09-22T21:35:12.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: entry->timestamp=2023-09-22T21:34:59.586837+0000 2023-09-22T21:35:12.785 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.47.6 op=0 op_state=1 2023-09-22T21:35:12.785 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:entry[obj00000000000000000024]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000024[0] 2023-09-22T21:35:12.785 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: listing bilog for incremental sync; position=00000000001.47.6 2023-09-22T21:35:12.786 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: parsed entry: id=00000000001.44.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-09-22T21:35:12.786 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0/obj00000000000000000007 2023-09-22T21:35:12.786 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: entry->timestamp=2023-09-22T21:34:59.389735+0000 2023-09-22T21:35:12.786 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.44.6 op=0 op_state=1 2023-09-22T21:35:12.786 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: parsed entry: id=00000000001.42.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-09-22T21:35:12.786 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1/obj00000000000000000016 2023-09-22T21:35:12.786 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: entry->timestamp=2023-09-22T21:34:59.607930+0000 2023-09-22T21:35:12.786 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.42.6 op=0 op_state=1 2023-09-22T21:35:12.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000007]: : injecting data sync error on key=obj00000000000000000007 2023-09-22T21:35:12.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: listing bilog for incremental sync; position=00000000001.44.6 2023-09-22T21:35:12.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000016]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000016[0] 2023-09-22T21:35:12.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: listing bilog for incremental sync; position=00000000001.42.6 2023-09-22T21:35:12.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000007]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:12.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.780+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000007]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0/obj00000000000000000007 2023-09-22T21:35:12.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.784+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: sync status for source bucket shard: 0 2023-09-22T21:35:12.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.784+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: sync status for source bucket shard: 0 2023-09-22T21:35:12.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.784+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: sync status for source bucket shard: 0 2023-09-22T21:35:12.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.784+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: sync status for source bucket shard: 0 2023-09-22T21:35:12.788 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.784+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: sync status for source bucket shard: 0 2023-09-22T21:35:12.788 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.784+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.788 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.784+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.788 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.784+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.788 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.784+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.788 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.784+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.788 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.784+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.788 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.784+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.788 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.784+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.789 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.784+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.789 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.784+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.789 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.784+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.790 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.784+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: sync status for source bucket shard: 0 2023-09-22T21:35:12.790 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.784+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: sync status for source bucket shard: 0 2023-09-22T21:35:12.790 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.784+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.790 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.784+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.790 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: sync status for source bucket shard: 0 2023-09-22T21:35:12.790 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: sync status for source bucket shard: 0 2023-09-22T21:35:12.790 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: sync status for source bucket shard: 0 2023-09-22T21:35:12.791 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: sync status for source bucket shard: 0 2023-09-22T21:35:12.791 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.791 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.791 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.791 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.791 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2023-09-22T21:35:12.791 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6/obj00000000000000000032 2023-09-22T21:35:12.791 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: entry->timestamp=2023-09-22T21:34:59.237138+0000 2023-09-22T21:35:12.791 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.43.6 op=0 op_state=1 2023-09-22T21:35:12.791 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: parsed entry: id=00000000001.31.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-09-22T21:35:12.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3/obj00000000000000000020 2023-09-22T21:35:12.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: entry->timestamp=2023-09-22T21:34:59.515114+0000 2023-09-22T21:35:12.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.31.6 op=0 op_state=1 2023-09-22T21:35:12.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: parsed entry: id=00000000001.34.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-09-22T21:35:12.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9/obj00000000000000000019 2023-09-22T21:35:12.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: entry->timestamp=2023-09-22T21:34:59.160915+0000 2023-09-22T21:35:12.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.34.6 op=0 op_state=1 2023-09-22T21:35:12.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000032]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000032[0] 2023-09-22T21:35:12.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: listing bilog for incremental sync; position=00000000001.43.6 2023-09-22T21:35:12.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:entry[obj00000000000000000020]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000020[0] 2023-09-22T21:35:12.793 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: listing bilog for incremental sync; position=00000000001.31.6 2023-09-22T21:35:12.793 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:entry[obj00000000000000000019]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000019[0] 2023-09-22T21:35:12.793 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: listing bilog for incremental sync; position=00000000001.34.6 2023-09-22T21:35:12.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-09-22T21:35:12.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10/obj00000000000000000013 2023-09-22T21:35:12.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: entry->timestamp=2023-09-22T21:34:59.366386+0000 2023-09-22T21:35:12.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-09-22T21:35:12.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:entry[obj00000000000000000013]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000013[0] 2023-09-22T21:35:12.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: listing bilog for incremental sync; position=00000000002.46.6 2023-09-22T21:35:12.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: parsed entry: id=00000000001.57.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-09-22T21:35:12.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12/obj00000000000000000030 2023-09-22T21:35:12.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: entry->timestamp=2023-09-22T21:34:59.312079+0000 2023-09-22T21:35:12.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.57.6 op=0 op_state=1 2023-09-22T21:35:12.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000030]: : injecting data sync error on key=obj00000000000000000030 2023-09-22T21:35:12.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: listing bilog for incremental sync; position=00000000001.57.6 2023-09-22T21:35:12.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000030]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:12.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.788+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000030]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12/obj00000000000000000030 2023-09-22T21:35:12.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.796+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: parsed entry: id=00000000001.47.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-09-22T21:35:12.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.796+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13/obj00000000000000000026 2023-09-22T21:35:12.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.796+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: entry->timestamp=2023-09-22T21:34:59.223961+0000 2023-09-22T21:35:12.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.796+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.47.6 op=0 op_state=1 2023-09-22T21:35:12.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.796+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:entry[obj00000000000000000026]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000026[0] 2023-09-22T21:35:12.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.796+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: listing bilog for incremental sync; position=00000000001.47.6 2023-09-22T21:35:12.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.796+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000016]: success 2023-09-22T21:35:12.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.796+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1:1 marker=00000000001.42.6 timestamp=2023-09-22T21:34:59.607930+0000 2023-09-22T21:35:12.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.796+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:entry[obj00000000000000000024]: success 2023-09-22T21:35:12.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.796+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2:1 marker=00000000001.47.6 timestamp=2023-09-22T21:34:59.586837+0000 2023-09-22T21:35:12.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.800+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:entry[obj00000000000000000013]: success 2023-09-22T21:35:12.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.800+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10:1 marker=00000000002.46.6 timestamp=2023-09-22T21:34:59.366386+0000 2023-09-22T21:35:12.805 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.804+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:entry[obj00000000000000000019]: success 2023-09-22T21:35:12.805 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.804+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9:1 marker=00000000001.34.6 timestamp=2023-09-22T21:34:59.160915+0000 2023-09-22T21:35:12.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.824+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: parsed entry: id=00000000002.36.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-09-22T21:35:12.825 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.824+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8/obj00000000000000000010 2023-09-22T21:35:12.825 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.824+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: entry->timestamp=2023-09-22T21:34:59.290391+0000 2023-09-22T21:35:12.825 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.824+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.36.6 op=0 op_state=1 2023-09-22T21:35:12.825 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.824+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:entry[obj00000000000000000010]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000010[0] 2023-09-22T21:35:12.825 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.824+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: listing bilog for incremental sync; position=00000000002.36.6 2023-09-22T21:35:12.832 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.832+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: parsed entry: id=00000000002.34.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-09-22T21:35:12.833 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.832+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5/obj00000000000000000000 2023-09-22T21:35:12.833 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.832+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: entry->timestamp=2023-09-22T21:34:59.172457+0000 2023-09-22T21:35:12.833 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.832+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.34.6 op=0 op_state=1 2023-09-22T21:35:12.833 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.832+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000000]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000000[0] 2023-09-22T21:35:12.833 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.832+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: listing bilog for incremental sync; position=00000000002.34.6 2023-09-22T21:35:12.833 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.832+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: a sync operation returned error 2023-09-22T21:35:12.833 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.832+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: backing out with sync_status=-5 2023-09-22T21:35:12.833 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.832+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:12.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.832+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:12.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.832+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.832+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: sync status for source bucket shard: 0 2023-09-22T21:35:12.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.832+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: parsed entry: id=00000000001.25.6 iter->object=obj00000000000000000076 iter->instance= name=obj00000000000000000076 instance= ns= 2023-09-22T21:35:12.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16/obj00000000000000000076 2023-09-22T21:35:12.838 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: entry->timestamp=2023-09-22T21:34:59.536192+0000 2023-09-22T21:35:12.838 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000076]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.25.6 op=0 op_state=1 2023-09-22T21:35:12.838 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000076]: : injecting data sync error on key=obj00000000000000000076 2023-09-22T21:35:12.838 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: listing bilog for incremental sync; position=00000000001.25.6 2023-09-22T21:35:12.838 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: parsed entry: id=00000000001.32.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-09-22T21:35:12.838 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11/obj00000000000000000021 2023-09-22T21:35:12.838 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: entry->timestamp=2023-09-22T21:34:59.149983+0000 2023-09-22T21:35:12.838 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.32.6 op=0 op_state=1 2023-09-22T21:35:12.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: parsed entry: id=00000000001.49.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-09-22T21:35:12.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14/obj00000000000000000033 2023-09-22T21:35:12.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: entry->timestamp=2023-09-22T21:34:59.442731+0000 2023-09-22T21:35:12.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.49.6 op=0 op_state=1 2023-09-22T21:35:12.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-09-22T21:35:12.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15/obj00000000000000000004 2023-09-22T21:35:12.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: entry->timestamp=2023-09-22T21:34:59.462844+0000 2023-09-22T21:35:12.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-09-22T21:35:12.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: parsed entry: id=00000000001.45.6 iter->object=obj00000000000000000028 iter->instance= name=obj00000000000000000028 instance= ns= 2023-09-22T21:35:12.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4/obj00000000000000000028 2023-09-22T21:35:12.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: entry->timestamp=2023-09-22T21:34:59.658464+0000 2023-09-22T21:35:12.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:entry[obj00000000000000000028]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.45.6 op=0 op_state=1 2023-09-22T21:35:12.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000076]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:12.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000076]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16/obj00000000000000000076 2023-09-22T21:35:12.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000021]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000021[0] 2023-09-22T21:35:12.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: listing bilog for incremental sync; position=00000000001.32.6 2023-09-22T21:35:12.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000033]: : injecting data sync error on key=obj00000000000000000033 2023-09-22T21:35:12.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: listing bilog for incremental sync; position=00000000001.49.6 2023-09-22T21:35:12.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:entry[obj00000000000000000004]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000004[0] 2023-09-22T21:35:12.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: listing bilog for incremental sync; position=00000000002.69.6 2023-09-22T21:35:12.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:entry[obj00000000000000000028]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000028[0] 2023-09-22T21:35:12.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: listing bilog for incremental sync; position=00000000001.45.6 2023-09-22T21:35:12.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000033]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:12.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.836+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000033]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14/obj00000000000000000033 2023-09-22T21:35:12.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.840+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: a sync operation returned error 2023-09-22T21:35:12.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.840+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: backing out with sync_status=-5 2023-09-22T21:35:12.842 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.840+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:12.842 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.840+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:12.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.840+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.845 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.844+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: sync status for source bucket shard: 0 2023-09-22T21:35:12.845 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.844+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.844+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:entry[obj00000000000000000026]: success 2023-09-22T21:35:12.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.844+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13:1 marker=00000000001.47.6 timestamp=2023-09-22T21:34:59.223961+0000 2023-09-22T21:35:12.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.848+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000000]: success 2023-09-22T21:35:12.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.848+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5:1 marker=00000000002.34.6 timestamp=2023-09-22T21:34:59.172457+0000 2023-09-22T21:35:12.850 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.848+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:entry[obj00000000000000000020]: success 2023-09-22T21:35:12.850 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.848+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3:1 marker=00000000001.31.6 timestamp=2023-09-22T21:34:59.515114+0000 2023-09-22T21:35:12.850 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.848+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000032]: success 2023-09-22T21:35:12.850 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.848+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6:1 marker=00000000001.43.6 timestamp=2023-09-22T21:34:59.237138+0000 2023-09-22T21:35:12.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.868+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:entry[obj00000000000000000010]: success 2023-09-22T21:35:12.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.868+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8:1 marker=00000000002.36.6 timestamp=2023-09-22T21:34:59.290391+0000 2023-09-22T21:35:12.881 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: parsed entry: id=00000000001.44.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-09-22T21:35:12.881 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0/obj00000000000000000007 2023-09-22T21:35:12.881 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: entry->timestamp=2023-09-22T21:34:59.389735+0000 2023-09-22T21:35:12.881 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.44.6 op=0 op_state=1 2023-09-22T21:35:12.881 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000007]: : injecting data sync error on key=obj00000000000000000007 2023-09-22T21:35:12.881 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: listing bilog for incremental sync; position=00000000001.44.6 2023-09-22T21:35:12.881 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000007]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:12.881 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000007]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0/obj00000000000000000007 2023-09-22T21:35:12.882 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: a sync operation returned error 2023-09-22T21:35:12.882 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: a sync operation returned error 2023-09-22T21:35:12.882 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: backing out with sync_status=-5 2023-09-22T21:35:12.882 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: backing out with sync_status=-5 2023-09-22T21:35:12.882 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:12.882 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:12.882 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:12.882 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:12.883 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.883 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: sync status for source bucket shard: 0 2023-09-22T21:35:12.883 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.883 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.883 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: sync status for source bucket shard: 0 2023-09-22T21:35:12.884 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.880+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.887 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.884+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000021]: success 2023-09-22T21:35:12.888 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.884+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11:1 marker=00000000001.32.6 timestamp=2023-09-22T21:34:59.149983+0000 2023-09-22T21:35:12.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.888+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: parsed entry: id=00000000001.57.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-09-22T21:35:12.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.888+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12/obj00000000000000000030 2023-09-22T21:35:12.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.888+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: entry->timestamp=2023-09-22T21:34:59.312079+0000 2023-09-22T21:35:12.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.888+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.57.6 op=0 op_state=1 2023-09-22T21:35:12.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.888+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000030]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000030[0] 2023-09-22T21:35:12.890 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.888+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: listing bilog for incremental sync; position=00000000001.57.6 2023-09-22T21:35:12.890 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.888+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:entry[obj00000000000000000028]: success 2023-09-22T21:35:12.890 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.888+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4:1 marker=00000000001.45.6 timestamp=2023-09-22T21:34:59.658464+0000 2023-09-22T21:35:12.890 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.888+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:entry[obj00000000000000000004]: success 2023-09-22T21:35:12.890 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.888+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15:1 marker=00000000002.69.6 timestamp=2023-09-22T21:34:59.462844+0000 2023-09-22T21:35:12.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.928+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: a sync operation returned error 2023-09-22T21:35:12.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.928+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: parsed entry: id=00000000001.49.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-09-22T21:35:12.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.928+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14/obj00000000000000000033 2023-09-22T21:35:12.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.928+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: entry->timestamp=2023-09-22T21:34:59.442731+0000 2023-09-22T21:35:12.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.928+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.49.6 op=0 op_state=1 2023-09-22T21:35:12.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.928+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: parsed entry: id=00000000001.25.6 iter->object=obj00000000000000000076 iter->instance= name=obj00000000000000000076 instance= ns= 2023-09-22T21:35:12.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.928+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16/obj00000000000000000076 2023-09-22T21:35:12.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.928+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: entry->timestamp=2023-09-22T21:34:59.536192+0000 2023-09-22T21:35:12.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.928+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000076]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.25.6 op=0 op_state=1 2023-09-22T21:35:12.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.928+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000033]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000033[0] 2023-09-22T21:35:12.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.928+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: listing bilog for incremental sync; position=00000000001.49.6 2023-09-22T21:35:12.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.928+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000076]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000076[0] 2023-09-22T21:35:12.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.928+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: listing bilog for incremental sync; position=00000000001.25.6 2023-09-22T21:35:12.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.928+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: backing out with sync_status=-5 2023-09-22T21:35:12.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.928+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:12.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.928+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:12.931 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.928+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:12.931 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.928+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: sync status for source bucket shard: 0 2023-09-22T21:35:12.931 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.928+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: listing bilog for incremental sync; position= 2023-09-22T21:35:12.936 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.932+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000030]: success 2023-09-22T21:35:12.936 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.932+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12:1 marker=00000000001.57.6 timestamp=2023-09-22T21:34:59.312079+0000 2023-09-22T21:35:12.937 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.936+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000076]: success 2023-09-22T21:35:12.937 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.936+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16:1 marker=00000000001.25.6 timestamp=2023-09-22T21:34:59.536192+0000 2023-09-22T21:35:12.938 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.936+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000033]: success 2023-09-22T21:35:12.938 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.936+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14:1 marker=00000000001.49.6 timestamp=2023-09-22T21:34:59.442731+0000 2023-09-22T21:35:12.977 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.976+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: parsed entry: id=00000000001.44.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-09-22T21:35:12.977 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.976+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0/obj00000000000000000007 2023-09-22T21:35:12.977 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.976+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: entry->timestamp=2023-09-22T21:34:59.389735+0000 2023-09-22T21:35:12.977 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.976+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.44.6 op=0 op_state=1 2023-09-22T21:35:12.977 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.976+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000007]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000007[0] 2023-09-22T21:35:12.977 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.976+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: listing bilog for incremental sync; position=00000000001.44.6 2023-09-22T21:35:12.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.984+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000007]: success 2023-09-22T21:35:12.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:12.984+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0:1 marker=00000000001.44.6 timestamp=2023-09-22T21:34:59.389735+0000 2023-09-22T21:35:13.032 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.028+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.032 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.032+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.032 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.032+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.033 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.032+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.033 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.032+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.033 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.032+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: sync status for source bucket shard: 0 2023-09-22T21:35:13.033 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.032+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.034 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.032+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: sync status for source bucket shard: 0 2023-09-22T21:35:13.034 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.032+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: sync status for source bucket shard: 0 2023-09-22T21:35:13.034 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.032+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: sync status for source bucket shard: 0 2023-09-22T21:35:13.034 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.032+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.034 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.032+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.034 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.032+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.035 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.032+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: sync status for source bucket shard: 0 2023-09-22T21:35:13.035 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.032+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.038 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.039 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.039 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.039 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: sync status for source bucket shard: 0 2023-09-22T21:35:13.039 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: sync status for source bucket shard: 0 2023-09-22T21:35:13.040 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-09-22T21:35:13.040 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1/obj00000000000000000010 2023-09-22T21:35:13.040 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: entry->timestamp=2023-09-22T21:35:00.762967+0000 2023-09-22T21:35:13.040 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.39.6 op=0 op_state=1 2023-09-22T21:35:13.040 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.040 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.041 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000010]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000010[0] 2023-09-22T21:35:13.041 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: listing bilog for incremental sync; position=00000000002.39.6 2023-09-22T21:35:13.041 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: parsed entry: id=00000000002.35.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-09-22T21:35:13.041 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3/obj00000000000000000021 2023-09-22T21:35:13.041 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: entry->timestamp=2023-09-22T21:35:00.023429+0000 2023-09-22T21:35:13.041 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.35.6 op=0 op_state=1 2023-09-22T21:35:13.041 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:entry[obj00000000000000000021]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000021[0] 2023-09-22T21:35:13.041 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.036+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: listing bilog for incremental sync; position=00000000002.35.6 2023-09-22T21:35:13.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.040+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: sync status for source bucket shard: 0 2023-09-22T21:35:13.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.040+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: sync status for source bucket shard: 0 2023-09-22T21:35:13.043 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.040+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.043 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.040+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: sync status for source bucket shard: 0 2023-09-22T21:35:13.043 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.040+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: sync status for source bucket shard: 0 2023-09-22T21:35:13.043 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.040+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-09-22T21:35:13.043 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.040+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4/obj00000000000000000004 2023-09-22T21:35:13.043 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.040+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: entry->timestamp=2023-09-22T21:34:59.955844+0000 2023-09-22T21:35:13.043 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.040+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-09-22T21:35:13.043 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.040+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.043 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.040+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.040+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.040+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.040+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: sync status for source bucket shard: 0 2023-09-22T21:35:13.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.040+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:entry[obj00000000000000000004]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000004[0] 2023-09-22T21:35:13.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.040+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: listing bilog for incremental sync; position=00000000002.49.6 2023-09-22T21:35:13.045 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.040+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.045 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.040+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.045 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.040+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.045 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.044+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.046 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.044+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.047 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.044+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: sync status for source bucket shard: 0 2023-09-22T21:35:13.051 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.051 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.051 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: sync status for source bucket shard: 0 2023-09-22T21:35:13.051 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.051 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000010]: success 2023-09-22T21:35:13.051 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1:2 marker=00000000002.39.6 timestamp=2023-09-22T21:35:00.762967+0000 2023-09-22T21:35:13.051 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: parsed entry: id=00000000001.46.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-09-22T21:35:13.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5/obj00000000000000000025 2023-09-22T21:35:13.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: entry->timestamp=2023-09-22T21:35:00.686575+0000 2023-09-22T21:35:13.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.46.6 op=0 op_state=1 2023-09-22T21:35:13.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: parsed entry: id=00000000001.50.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-09-22T21:35:13.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11/obj00000000000000000014 2023-09-22T21:35:13.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: entry->timestamp=2023-09-22T21:35:00.633518+0000 2023-09-22T21:35:13.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.50.6 op=0 op_state=1 2023-09-22T21:35:13.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-09-22T21:35:13.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8/obj00000000000000000019 2023-09-22T21:35:13.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: entry->timestamp=2023-09-22T21:35:00.174568+0000 2023-09-22T21:35:13.053 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-09-22T21:35:13.053 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000025]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000025[0] 2023-09-22T21:35:13.053 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: listing bilog for incremental sync; position=00000000001.46.6 2023-09-22T21:35:13.053 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000014]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000014[0] 2023-09-22T21:35:13.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: listing bilog for incremental sync; position=00000000001.50.6 2023-09-22T21:35:13.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:entry[obj00000000000000000019]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000019[0] 2023-09-22T21:35:13.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.048+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: listing bilog for incremental sync; position=00000000002.37.6 2023-09-22T21:35:13.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.052+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: sync status for source bucket shard: 0 2023-09-22T21:35:13.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.052+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.052+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: sync status for source bucket shard: 0 2023-09-22T21:35:13.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.052+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: sync status for source bucket shard: 0 2023-09-22T21:35:13.056 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.052+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: sync status for source bucket shard: 0 2023-09-22T21:35:13.056 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.052+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.056 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.052+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.056 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.052+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.052+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: sync status for source bucket shard: 0 2023-09-22T21:35:13.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.056+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:entry[obj00000000000000000021]: success 2023-09-22T21:35:13.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.056+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3:2 marker=00000000002.35.6 timestamp=2023-09-22T21:35:00.023429+0000 2023-09-22T21:35:13.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.056+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: parsed entry: id=00000000002.78.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-09-22T21:35:13.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.056+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9/obj00000000000000000003 2023-09-22T21:35:13.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.056+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: entry->timestamp=2023-09-22T21:35:00.840895+0000 2023-09-22T21:35:13.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.056+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.78.6 op=0 op_state=1 2023-09-22T21:35:13.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.056+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.056+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:entry[obj00000000000000000003]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000003[0] 2023-09-22T21:35:13.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.056+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: listing bilog for incremental sync; position=00000000002.78.6 2023-09-22T21:35:13.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.056+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:entry[obj00000000000000000004]: success 2023-09-22T21:35:13.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.056+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4:2 marker=00000000002.49.6 timestamp=2023-09-22T21:34:59.955844+0000 2023-09-22T21:35:13.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.060+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-09-22T21:35:13.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.060+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15/obj00000000000000000006 2023-09-22T21:35:13.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.060+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: entry->timestamp=2023-09-22T21:35:00.609377+0000 2023-09-22T21:35:13.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.060+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-09-22T21:35:13.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.060+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:entry[obj00000000000000000006]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000006[0] 2023-09-22T21:35:13.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.060+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: listing bilog for incremental sync; position=00000000002.42.6 2023-09-22T21:35:13.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.064+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:entry[obj00000000000000000019]: success 2023-09-22T21:35:13.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.064+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8:2 marker=00000000002.37.6 timestamp=2023-09-22T21:35:00.174568+0000 2023-09-22T21:35:13.069 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.068+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: parsed entry: id=00000000002.36.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-09-22T21:35:13.069 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.068+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18/obj00000000000000000007 2023-09-22T21:35:13.069 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.068+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: entry->timestamp=2023-09-22T21:34:59.942969+0000 2023-09-22T21:35:13.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.068+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.36.6 op=0 op_state=1 2023-09-22T21:35:13.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.068+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:entry[obj00000000000000000007]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000007[0] 2023-09-22T21:35:13.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.068+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: listing bilog for incremental sync; position=00000000002.36.6 2023-09-22T21:35:13.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.068+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: parsed entry: id=00000000002.114.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-09-22T21:35:13.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.068+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16/obj00000000000000000011 2023-09-22T21:35:13.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.068+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: entry->timestamp=2023-09-22T21:35:00.077991+0000 2023-09-22T21:35:13.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.068+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.114.6 op=0 op_state=1 2023-09-22T21:35:13.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.068+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000011]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000011[0] 2023-09-22T21:35:13.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.068+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: listing bilog for incremental sync; position=00000000002.114.6 2023-09-22T21:35:13.075 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.072+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:entry[obj00000000000000000007]: success 2023-09-22T21:35:13.075 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.072+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18:2 marker=00000000002.36.6 timestamp=2023-09-22T21:34:59.942969+0000 2023-09-22T21:35:13.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.080+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-09-22T21:35:13.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.080+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0/obj00000000000000000005 2023-09-22T21:35:13.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.080+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: entry->timestamp=2023-09-22T21:35:00.866068+0000 2023-09-22T21:35:13.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.080+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.39.6 op=0 op_state=1 2023-09-22T21:35:13.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.080+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000005]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000005[0] 2023-09-22T21:35:13.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.080+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: listing bilog for incremental sync; position=00000000002.39.6 2023-09-22T21:35:13.085 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.084+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: parsed entry: id=00000000002.76.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-09-22T21:35:13.086 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.084+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6/obj00000000000000000030 2023-09-22T21:35:13.086 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.084+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: entry->timestamp=2023-09-22T21:35:00.004578+0000 2023-09-22T21:35:13.086 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.084+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.76.6 op=0 op_state=1 2023-09-22T21:35:13.086 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.084+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000030]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000030[0] 2023-09-22T21:35:13.086 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.084+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: listing bilog for incremental sync; position=00000000002.76.6 2023-09-22T21:35:13.086 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.084+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-09-22T21:35:13.086 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.084+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7/obj00000000000000000015 2023-09-22T21:35:13.086 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.084+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: entry->timestamp=2023-09-22T21:35:00.707040+0000 2023-09-22T21:35:13.086 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.084+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-09-22T21:35:13.087 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.084+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:entry[obj00000000000000000015]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000015[0] 2023-09-22T21:35:13.087 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.084+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: listing bilog for incremental sync; position=00000000002.47.6 2023-09-22T21:35:13.087 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.084+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:entry[obj00000000000000000006]: success 2023-09-22T21:35:13.087 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.084+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15:2 marker=00000000002.42.6 timestamp=2023-09-22T21:35:00.609377+0000 2023-09-22T21:35:13.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.088+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000030]: success 2023-09-22T21:35:13.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.088+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6:2 marker=00000000002.76.6 timestamp=2023-09-22T21:35:00.004578+0000 2023-09-22T21:35:13.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.088+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:entry[obj00000000000000000003]: success 2023-09-22T21:35:13.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.088+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9:2 marker=00000000002.78.6 timestamp=2023-09-22T21:35:00.840895+0000 2023-09-22T21:35:13.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.096+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: parsed entry: id=00000000001.56.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-09-22T21:35:13.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.096+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10/obj00000000000000000009 2023-09-22T21:35:13.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.096+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: entry->timestamp=2023-09-22T21:35:00.098781+0000 2023-09-22T21:35:13.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.096+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.56.6 op=0 op_state=1 2023-09-22T21:35:13.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.096+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:entry[obj00000000000000000009]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000009[0] 2023-09-22T21:35:13.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.096+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: listing bilog for incremental sync; position=00000000001.56.6 2023-09-22T21:35:13.102 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.100+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: parsed entry: id=00000000002.116.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-09-22T21:35:13.102 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.100+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13/obj00000000000000000002 2023-09-22T21:35:13.102 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.100+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: entry->timestamp=2023-09-22T21:35:00.231506+0000 2023-09-22T21:35:13.102 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.100+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.116.6 op=0 op_state=1 2023-09-22T21:35:13.102 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.100+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:entry[obj00000000000000000002]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000002[0] 2023-09-22T21:35:13.102 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.100+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: listing bilog for incremental sync; position=00000000002.116.6 2023-09-22T21:35:13.103 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.100+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:entry[obj00000000000000000015]: success 2023-09-22T21:35:13.103 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.100+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7:2 marker=00000000002.47.6 timestamp=2023-09-22T21:35:00.707040+0000 2023-09-22T21:35:13.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.104+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-09-22T21:35:13.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.104+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12/obj00000000000000000012 2023-09-22T21:35:13.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.104+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: entry->timestamp=2023-09-22T21:35:00.257125+0000 2023-09-22T21:35:13.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.104+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.58.6 op=0 op_state=1 2023-09-22T21:35:13.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.104+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000012]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000012[0] 2023-09-22T21:35:13.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.104+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: listing bilog for incremental sync; position=00000000002.58.6 2023-09-22T21:35:13.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.108+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000025]: success 2023-09-22T21:35:13.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.108+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5:2 marker=00000000001.46.6 timestamp=2023-09-22T21:35:00.686575+0000 2023-09-22T21:35:13.113 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.112+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-09-22T21:35:13.113 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.112+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14/obj00000000000000000000 2023-09-22T21:35:13.113 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.112+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: entry->timestamp=2023-09-22T21:35:00.150852+0000 2023-09-22T21:35:13.113 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.112+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-09-22T21:35:13.113 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.112+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000000]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000000[0] 2023-09-22T21:35:13.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.112+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: listing bilog for incremental sync; position=00000000002.37.6 2023-09-22T21:35:13.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.112+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-09-22T21:35:13.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.112+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17/obj00000000000000000001 2023-09-22T21:35:13.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.112+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: entry->timestamp=2023-09-22T21:35:00.787871+0000 2023-09-22T21:35:13.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.112+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.58.6 op=0 op_state=1 2023-09-22T21:35:13.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.112+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:entry[obj00000000000000000001]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000001[0] 2023-09-22T21:35:13.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.112+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: listing bilog for incremental sync; position=00000000002.58.6 2023-09-22T21:35:13.115 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.112+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000014]: success 2023-09-22T21:35:13.115 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.112+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11:2 marker=00000000001.50.6 timestamp=2023-09-22T21:35:00.633518+0000 2023-09-22T21:35:13.117 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.112+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000000]: success 2023-09-22T21:35:13.117 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.116+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14:2 marker=00000000002.37.6 timestamp=2023-09-22T21:35:00.150852+0000 2023-09-22T21:35:13.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.120+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000011]: success 2023-09-22T21:35:13.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.120+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16:2 marker=00000000002.114.6 timestamp=2023-09-22T21:35:00.077991+0000 2023-09-22T21:35:13.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.120+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:entry[obj00000000000000000001]: success 2023-09-22T21:35:13.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.120+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17:2 marker=00000000002.58.6 timestamp=2023-09-22T21:35:00.787871+0000 2023-09-22T21:35:13.138 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.136+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000005]: success 2023-09-22T21:35:13.138 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.136+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0:2 marker=00000000002.39.6 timestamp=2023-09-22T21:35:00.866068+0000 2023-09-22T21:35:13.147 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.144+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:entry[obj00000000000000000009]: success 2023-09-22T21:35:13.147 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.144+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10:2 marker=00000000001.56.6 timestamp=2023-09-22T21:35:00.098781+0000 2023-09-22T21:35:13.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.152+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:entry[obj00000000000000000002]: success 2023-09-22T21:35:13.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.152+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13:2 marker=00000000002.116.6 timestamp=2023-09-22T21:35:00.231506+0000 2023-09-22T21:35:13.160 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.156+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000012]: success 2023-09-22T21:35:13.160 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.156+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12:2 marker=00000000002.58.6 timestamp=2023-09-22T21:35:00.257125+0000 2023-09-22T21:35:13.168 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.164+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.168 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.164+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.169 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.164+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.169 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.164+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.169 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.164+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.169 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.164+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.169 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: sync status for source bucket shard: 0 2023-09-22T21:35:13.169 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.170 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: sync status for source bucket shard: 0 2023-09-22T21:35:13.170 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: sync status for source bucket shard: 0 2023-09-22T21:35:13.170 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: sync status for source bucket shard: 0 2023-09-22T21:35:13.170 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.170 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.171 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.171 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.171 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.171 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.171 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: sync status for source bucket shard: 0 2023-09-22T21:35:13.171 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.171 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.172 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.172 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.172 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: sync status for source bucket shard: 0 2023-09-22T21:35:13.172 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: sync status for source bucket shard: 0 2023-09-22T21:35:13.172 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: sync status for source bucket shard: 0 2023-09-22T21:35:13.173 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.173 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.173 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.173 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.173 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.173 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.174 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.174 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: sync status for source bucket shard: 0 2023-09-22T21:35:13.174 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.168+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: sync status for source bucket shard: 0 2023-09-22T21:35:13.174 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.174 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.174 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.175 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.175 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: sync status for source bucket shard: 0 2023-09-22T21:35:13.175 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.175 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: sync status for source bucket shard: 0 2023-09-22T21:35:13.175 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: sync status for source bucket shard: 0 2023-09-22T21:35:13.175 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.175 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: sync status for source bucket shard: 0 2023-09-22T21:35:13.175 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.175 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.175 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.178 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: sync status for source bucket shard: 0 2023-09-22T21:35:13.178 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: sync status for source bucket shard: 0 2023-09-22T21:35:13.178 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: sync status for source bucket shard: 0 2023-09-22T21:35:13.178 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: sync status for source bucket shard: 0 2023-09-22T21:35:13.178 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.178 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.178 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.179 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.179 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.172+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.179 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: sync status for source bucket shard: 0 2023-09-22T21:35:13.179 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: sync status for source bucket shard: 0 2023-09-22T21:35:13.179 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: sync status for source bucket shard: 0 2023-09-22T21:35:13.179 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: sync status for source bucket shard: 0 2023-09-22T21:35:13.179 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.179 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.179 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.180 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.180 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: parsed entry: id=00000000002.28.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-09-22T21:35:13.180 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12/obj00000000000000000009 2023-09-22T21:35:13.180 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: entry->timestamp=2023-09-22T21:35:01.281101+0000 2023-09-22T21:35:13.180 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.28.6 op=0 op_state=1 2023-09-22T21:35:13.180 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-09-22T21:35:13.180 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9/obj00000000000000000007 2023-09-22T21:35:13.180 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: entry->timestamp=2023-09-22T21:35:01.360569+0000 2023-09-22T21:35:13.180 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.54.6 op=0 op_state=1 2023-09-22T21:35:13.181 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: parsed entry: id=00000000001.44.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2023-09-22T21:35:13.181 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11/obj00000000000000000040 2023-09-22T21:35:13.181 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: entry->timestamp=2023-09-22T21:35:01.817448+0000 2023-09-22T21:35:13.181 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.44.6 op=0 op_state=1 2023-09-22T21:35:13.181 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000009]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000009[0] 2023-09-22T21:35:13.181 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: listing bilog for incremental sync; position=00000000002.28.6 2023-09-22T21:35:13.181 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:entry[obj00000000000000000007]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000007[0] 2023-09-22T21:35:13.181 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: listing bilog for incremental sync; position=00000000002.54.6 2023-09-22T21:35:13.181 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: parsed entry: id=00000000002.76.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-09-22T21:35:13.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14/obj00000000000000000013 2023-09-22T21:35:13.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: entry->timestamp=2023-09-22T21:35:01.335423+0000 2023-09-22T21:35:13.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.76.6 op=0 op_state=1 2023-09-22T21:35:13.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: sync status for source bucket shard: 0 2023-09-22T21:35:13.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000040]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000040[0] 2023-09-22T21:35:13.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: listing bilog for incremental sync; position=00000000001.44.6 2023-09-22T21:35:13.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000013]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000013[0] 2023-09-22T21:35:13.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: listing bilog for incremental sync; position=00000000002.76.6 2023-09-22T21:35:13.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: parsed entry: id=00000000001.62.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2023-09-22T21:35:13.183 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13/obj00000000000000000045 2023-09-22T21:35:13.183 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: entry->timestamp=2023-09-22T21:35:01.186971+0000 2023-09-22T21:35:13.183 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.62.6 op=0 op_state=1 2023-09-22T21:35:13.183 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:entry[obj00000000000000000045]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000045[0] 2023-09-22T21:35:13.183 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: listing bilog for incremental sync; position=00000000001.62.6 2023-09-22T21:35:13.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: parsed entry: id=00000000002.36.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-09-22T21:35:13.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16/obj00000000000000000033 2023-09-22T21:35:13.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: entry->timestamp=2023-09-22T21:35:01.495725+0000 2023-09-22T21:35:13.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.36.6 op=0 op_state=1 2023-09-22T21:35:13.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000033]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000033[0] 2023-09-22T21:35:13.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.176+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: listing bilog for incremental sync; position=00000000002.36.6 2023-09-22T21:35:13.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.180+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:entry[obj00000000000000000007]: success 2023-09-22T21:35:13.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.180+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9:3 marker=00000000002.54.6 timestamp=2023-09-22T21:35:01.360569+0000 2023-09-22T21:35:13.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.180+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000009]: success 2023-09-22T21:35:13.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.180+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12:3 marker=00000000002.28.6 timestamp=2023-09-22T21:35:01.281101+0000 2023-09-22T21:35:13.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.180+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000033]: success 2023-09-22T21:35:13.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.180+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16:3 marker=00000000002.36.6 timestamp=2023-09-22T21:35:01.495725+0000 2023-09-22T21:35:13.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.180+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000013]: success 2023-09-22T21:35:13.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.180+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14:3 marker=00000000002.76.6 timestamp=2023-09-22T21:35:01.335423+0000 2023-09-22T21:35:13.214 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-09-22T21:35:13.214 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1/obj00000000000000000026 2023-09-22T21:35:13.214 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: entry->timestamp=2023-09-22T21:35:01.212121+0000 2023-09-22T21:35:13.214 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.54.6 op=0 op_state=1 2023-09-22T21:35:13.214 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-09-22T21:35:13.214 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2/obj00000000000000000011 2023-09-22T21:35:13.214 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: entry->timestamp=2023-09-22T21:35:01.260519+0000 2023-09-22T21:35:13.215 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.54.6 op=0 op_state=1 2023-09-22T21:35:13.215 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: parsed entry: id=00000000001.49.6 iter->object=obj00000000000000000058 iter->instance= name=obj00000000000000000058 instance= ns= 2023-09-22T21:35:13.215 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0/obj00000000000000000058 2023-09-22T21:35:13.215 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: entry->timestamp=2023-09-22T21:35:01.736432+0000 2023-09-22T21:35:13.215 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000058]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.49.6 op=0 op_state=1 2023-09-22T21:35:13.215 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: parsed entry: id=00000000002.126.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-09-22T21:35:13.215 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22/obj00000000000000000008 2023-09-22T21:35:13.215 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: entry->timestamp=2023-09-22T21:35:01.681297+0000 2023-09-22T21:35:13.215 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.126.6 op=0 op_state=1 2023-09-22T21:35:13.216 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000026]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000026[0] 2023-09-22T21:35:13.216 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: listing bilog for incremental sync; position=00000000002.54.6 2023-09-22T21:35:13.216 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:entry[obj00000000000000000011]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000011[0] 2023-09-22T21:35:13.216 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: listing bilog for incremental sync; position=00000000002.54.6 2023-09-22T21:35:13.216 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000058]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000058[0] 2023-09-22T21:35:13.216 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: listing bilog for incremental sync; position=00000000001.49.6 2023-09-22T21:35:13.216 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:entry[obj00000000000000000008]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000008[0] 2023-09-22T21:35:13.216 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.212+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: listing bilog for incremental sync; position=00000000002.126.6 2023-09-22T21:35:13.218 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: parsed entry: id=00000000001.58.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2023-09-22T21:35:13.218 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5/obj00000000000000000050 2023-09-22T21:35:13.218 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: entry->timestamp=2023-09-22T21:35:01.520971+0000 2023-09-22T21:35:13.218 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.58.6 op=0 op_state=1 2023-09-22T21:35:13.218 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000050]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000050[0] 2023-09-22T21:35:13.218 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: listing bilog for incremental sync; position=00000000001.58.6 2023-09-22T21:35:13.218 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: parsed entry: id=00000000002.124.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2023-09-22T21:35:13.218 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6/obj00000000000000000032 2023-09-22T21:35:13.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: entry->timestamp=2023-09-22T21:35:01.575757+0000 2023-09-22T21:35:13.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.124.6 op=0 op_state=1 2023-09-22T21:35:13.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2023-09-22T21:35:13.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8/obj00000000000000000039 2023-09-22T21:35:13.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: entry->timestamp=2023-09-22T21:35:01.602964+0000 2023-09-22T21:35:13.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.43.6 op=0 op_state=1 2023-09-22T21:35:13.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-09-22T21:35:13.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7/obj00000000000000000002 2023-09-22T21:35:13.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: entry->timestamp=2023-09-22T21:35:01.198836+0000 2023-09-22T21:35:13.220 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.41.6 op=0 op_state=1 2023-09-22T21:35:13.220 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000032]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000032[0] 2023-09-22T21:35:13.220 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: listing bilog for incremental sync; position=00000000002.124.6 2023-09-22T21:35:13.220 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:entry[obj00000000000000000039]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000039[0] 2023-09-22T21:35:13.220 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: listing bilog for incremental sync; position=00000000001.43.6 2023-09-22T21:35:13.220 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:entry[obj00000000000000000002]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000002[0] 2023-09-22T21:35:13.220 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: listing bilog for incremental sync; position=00000000002.41.6 2023-09-22T21:35:13.221 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000026]: success 2023-09-22T21:35:13.221 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1:3 marker=00000000002.54.6 timestamp=2023-09-22T21:35:01.212121+0000 2023-09-22T21:35:13.221 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:entry[obj00000000000000000011]: success 2023-09-22T21:35:13.221 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.216+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2:3 marker=00000000002.54.6 timestamp=2023-09-22T21:35:01.260519+0000 2023-09-22T21:35:13.222 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-09-22T21:35:13.222 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15/obj00000000000000000012 2023-09-22T21:35:13.222 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: entry->timestamp=2023-09-22T21:35:01.415807+0000 2023-09-22T21:35:13.222 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-09-22T21:35:13.222 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:entry[obj00000000000000000012]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000012[0] 2023-09-22T21:35:13.222 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: listing bilog for incremental sync; position=00000000002.42.6 2023-09-22T21:35:13.223 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-09-22T21:35:13.223 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18/obj00000000000000000000 2023-09-22T21:35:13.223 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: entry->timestamp=2023-09-22T21:35:01.896829+0000 2023-09-22T21:35:13.223 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.48.6 op=0 op_state=1 2023-09-22T21:35:13.223 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-09-22T21:35:13.223 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19/obj00000000000000000021 2023-09-22T21:35:13.223 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: entry->timestamp=2023-09-22T21:35:01.441221+0000 2023-09-22T21:35:13.223 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-09-22T21:35:13.223 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: parsed entry: id=00000000001.63.6 iter->object=obj00000000000000000055 iter->instance= name=obj00000000000000000055 instance= ns= 2023-09-22T21:35:13.224 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21/obj00000000000000000055 2023-09-22T21:35:13.224 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: entry->timestamp=2023-09-22T21:35:01.659463+0000 2023-09-22T21:35:13.224 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:entry[obj00000000000000000055]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.63.6 op=0 op_state=1 2023-09-22T21:35:13.224 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:entry[obj00000000000000000000]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000000[0] 2023-09-22T21:35:13.224 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: listing bilog for incremental sync; position=00000000002.48.6 2023-09-22T21:35:13.224 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:entry[obj00000000000000000021]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000021[0] 2023-09-22T21:35:13.224 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: listing bilog for incremental sync; position=00000000002.56.6 2023-09-22T21:35:13.224 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:entry[obj00000000000000000055]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000055[0] 2023-09-22T21:35:13.224 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: listing bilog for incremental sync; position=00000000001.63.6 2023-09-22T21:35:13.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000032]: success 2023-09-22T21:35:13.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6:3 marker=00000000002.124.6 timestamp=2023-09-22T21:35:01.575757+0000 2023-09-22T21:35:13.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:entry[obj00000000000000000002]: success 2023-09-22T21:35:13.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.220+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7:3 marker=00000000002.41.6 timestamp=2023-09-22T21:35:01.198836+0000 2023-09-22T21:35:13.228 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.224+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-09-22T21:35:13.229 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.224+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4/obj00000000000000000014 2023-09-22T21:35:13.229 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.224+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: entry->timestamp=2023-09-22T21:35:01.922738+0000 2023-09-22T21:35:13.229 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.224+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-09-22T21:35:13.229 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.224+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-09-22T21:35:13.229 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.224+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17/obj00000000000000000001 2023-09-22T21:35:13.229 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.224+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: entry->timestamp=2023-09-22T21:35:01.757591+0000 2023-09-22T21:35:13.229 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.224+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-09-22T21:35:13.229 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.224+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-09-22T21:35:13.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.224+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20/obj00000000000000000003 2023-09-22T21:35:13.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.224+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: entry->timestamp=2023-09-22T21:35:01.839402+0000 2023-09-22T21:35:13.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.224+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.62.6 op=0 op_state=1 2023-09-22T21:35:13.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.224+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:entry[obj00000000000000000014]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000014[0] 2023-09-22T21:35:13.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.224+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: listing bilog for incremental sync; position=00000000002.56.6 2023-09-22T21:35:13.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.224+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:entry[obj00000000000000000001]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000001[0] 2023-09-22T21:35:13.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.224+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: listing bilog for incremental sync; position=00000000002.52.6 2023-09-22T21:35:13.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.224+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:entry[obj00000000000000000021]: success 2023-09-22T21:35:13.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.224+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19:3 marker=00000000002.56.6 timestamp=2023-09-22T21:35:01.441221+0000 2023-09-22T21:35:13.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.228+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:entry[obj00000000000000000003]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000003[0] 2023-09-22T21:35:13.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.228+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: listing bilog for incremental sync; position=00000000002.62.6 2023-09-22T21:35:13.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.228+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000058]: success 2023-09-22T21:35:13.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.228+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0:3 marker=00000000001.49.6 timestamp=2023-09-22T21:35:01.736432+0000 2023-09-22T21:35:13.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.228+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:entry[obj00000000000000000012]: success 2023-09-22T21:35:13.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.228+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15:3 marker=00000000002.42.6 timestamp=2023-09-22T21:35:01.415807+0000 2023-09-22T21:35:13.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.228+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:entry[obj00000000000000000000]: success 2023-09-22T21:35:13.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.228+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18:3 marker=00000000002.48.6 timestamp=2023-09-22T21:35:01.896829+0000 2023-09-22T21:35:13.232 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.228+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:entry[obj00000000000000000003]: success 2023-09-22T21:35:13.232 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.228+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20:3 marker=00000000002.62.6 timestamp=2023-09-22T21:35:01.839402+0000 2023-09-22T21:35:13.234 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.232+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000040]: success 2023-09-22T21:35:13.234 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.232+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11:3 marker=00000000001.44.6 timestamp=2023-09-22T21:35:01.817448+0000 2023-09-22T21:35:13.235 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.232+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:entry[obj00000000000000000001]: success 2023-09-22T21:35:13.235 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.232+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17:3 marker=00000000002.52.6 timestamp=2023-09-22T21:35:01.757591+0000 2023-09-22T21:35:13.235 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.232+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:entry[obj00000000000000000014]: success 2023-09-22T21:35:13.235 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.232+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4:3 marker=00000000002.56.6 timestamp=2023-09-22T21:35:01.922738+0000 2023-09-22T21:35:13.238 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.236+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:entry[obj00000000000000000045]: success 2023-09-22T21:35:13.238 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.236+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13:3 marker=00000000001.62.6 timestamp=2023-09-22T21:35:01.186971+0000 2023-09-22T21:35:13.239 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.236+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:entry[obj00000000000000000055]: success 2023-09-22T21:35:13.240 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.236+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21:3 marker=00000000001.63.6 timestamp=2023-09-22T21:35:01.659463+0000 2023-09-22T21:35:13.264 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.264+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:entry[obj00000000000000000008]: success 2023-09-22T21:35:13.264 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.264+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22:3 marker=00000000002.126.6 timestamp=2023-09-22T21:35:01.681297+0000 2023-09-22T21:35:13.269 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.268+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000050]: success 2023-09-22T21:35:13.269 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.268+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5:3 marker=00000000001.58.6 timestamp=2023-09-22T21:35:01.520971+0000 2023-09-22T21:35:13.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.272+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:entry[obj00000000000000000039]: success 2023-09-22T21:35:13.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.272+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8:3 marker=00000000001.43.6 timestamp=2023-09-22T21:35:01.602964+0000 2023-09-22T21:35:13.317 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.317 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.317 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.317 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: sync status for source bucket shard: 0 2023-09-22T21:35:13.317 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: sync status for source bucket shard: 0 2023-09-22T21:35:13.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: sync status for source bucket shard: 0 2023-09-22T21:35:13.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.319 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: sync status for source bucket shard: 0 2023-09-22T21:35:13.319 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.319 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: sync status for source bucket shard: 0 2023-09-22T21:35:13.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: sync status for source bucket shard: 0 2023-09-22T21:35:13.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.322 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: sync status for source bucket shard: 0 2023-09-22T21:35:13.322 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: sync status for source bucket shard: 0 2023-09-22T21:35:13.322 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: sync status for source bucket shard: 0 2023-09-22T21:35:13.322 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.322 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.322 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-09-22T21:35:13.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0/obj00000000000000000006 2023-09-22T21:35:13.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: entry->timestamp=2023-09-22T21:35:02.910384+0000 2023-09-22T21:35:13.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-09-22T21:35:13.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-09-22T21:35:13.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5/obj00000000000000000002 2023-09-22T21:35:13.324 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: entry->timestamp=2023-09-22T21:35:02.727864+0000 2023-09-22T21:35:13.324 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.30.6 op=0 op_state=1 2023-09-22T21:35:13.324 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000006]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000006[0] 2023-09-22T21:35:13.324 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.316+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: listing bilog for incremental sync; position=00000000002.50.6 2023-09-22T21:35:13.324 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000002]: : injecting data sync error on key=obj00000000000000000002 2023-09-22T21:35:13.324 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: listing bilog for incremental sync; position=00000000002.30.6 2023-09-22T21:35:13.324 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000002]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:13.324 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000002]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5/obj00000000000000000002 2023-09-22T21:35:13.324 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: sync status for source bucket shard: 0 2023-09-22T21:35:13.324 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.325 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.325 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: parsed entry: id=00000000002.136.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-09-22T21:35:13.325 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3/obj00000000000000000004 2023-09-22T21:35:13.325 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: entry->timestamp=2023-09-22T21:35:02.434102+0000 2023-09-22T21:35:13.325 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.136.6 op=0 op_state=1 2023-09-22T21:35:13.325 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-09-22T21:35:13.325 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4/obj00000000000000000011 2023-09-22T21:35:13.325 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: entry->timestamp=2023-09-22T21:35:02.964679+0000 2023-09-22T21:35:13.325 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-09-22T21:35:13.326 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:entry[obj00000000000000000004]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000004[0] 2023-09-22T21:35:13.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: listing bilog for incremental sync; position=00000000002.136.6 2023-09-22T21:35:13.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:entry[obj00000000000000000011]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000011[0] 2023-09-22T21:35:13.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: listing bilog for incremental sync; position=00000000002.46.6 2023-09-22T21:35:13.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-09-22T21:35:13.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6/obj00000000000000000019 2023-09-22T21:35:13.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: entry->timestamp=2023-09-22T21:35:03.128117+0000 2023-09-22T21:35:13.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-09-22T21:35:13.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000019]: : injecting data sync error on key=obj00000000000000000019 2023-09-22T21:35:13.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: listing bilog for incremental sync; position=00000000002.61.6 2023-09-22T21:35:13.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: sync status for source bucket shard: 0 2023-09-22T21:35:13.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: sync status for source bucket shard: 0 2023-09-22T21:35:13.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000019]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:13.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000019]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6/obj00000000000000000019 2023-09-22T21:35:13.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.320+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: sync status for source bucket shard: 0 2023-09-22T21:35:13.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: sync status for source bucket shard: 0 2023-09-22T21:35:13.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: sync status for source bucket shard: 0 2023-09-22T21:35:13.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000006]: success 2023-09-22T21:35:13.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0:4 marker=00000000002.50.6 timestamp=2023-09-22T21:35:02.910384+0000 2023-09-22T21:35:13.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.333 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.333 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: sync status for source bucket shard: 0 2023-09-22T21:35:13.333 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: sync status for source bucket shard: 0 2023-09-22T21:35:13.333 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.333 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.333 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2023-09-22T21:35:13.333 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2/obj00000000000000000027 2023-09-22T21:35:13.333 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: entry->timestamp=2023-09-22T21:35:03.211837+0000 2023-09-22T21:35:13.333 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.43.6 op=0 op_state=1 2023-09-22T21:35:13.334 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:entry[obj00000000000000000027]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000027[0] 2023-09-22T21:35:13.334 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: listing bilog for incremental sync; position=00000000001.43.6 2023-09-22T21:35:13.334 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:entry[obj00000000000000000004]: success 2023-09-22T21:35:13.334 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3:4 marker=00000000002.136.6 timestamp=2023-09-22T21:35:02.434102+0000 2023-09-22T21:35:13.334 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: sync status for source bucket shard: 0 2023-09-22T21:35:13.334 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.324+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: sync status for source bucket shard: 0 2023-09-22T21:35:13.336 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.336 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.336 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: sync status for source bucket shard: 0 2023-09-22T21:35:13.336 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.336 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: sync status for source bucket shard: 0 2023-09-22T21:35:13.336 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: sync status for source bucket shard: 0 2023-09-22T21:35:13.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: sync status for source bucket shard: 0 2023-09-22T21:35:13.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: sync status for source bucket shard: 0 2023-09-22T21:35:13.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: sync status for source bucket shard: 0 2023-09-22T21:35:13.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: sync status for source bucket shard: 0 2023-09-22T21:35:13.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: sync status for source bucket shard: 0 2023-09-22T21:35:13.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.338 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.338 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.338 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.338 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: sync status for source bucket shard: 0 2023-09-22T21:35:13.338 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]: sync status for source bucket shard: 0 2023-09-22T21:35:13.338 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.338 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.338 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:entry[obj00000000000000000011]: success 2023-09-22T21:35:13.338 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.328+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4:4 marker=00000000002.46.6 timestamp=2023-09-22T21:35:02.964679+0000 2023-09-22T21:35:13.339 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-09-22T21:35:13.339 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10/obj00000000000000000015 2023-09-22T21:35:13.339 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: entry->timestamp=2023-09-22T21:35:02.754030+0000 2023-09-22T21:35:13.339 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.64.6 op=0 op_state=1 2023-09-22T21:35:13.339 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:entry[obj00000000000000000015]: : injecting data sync error on key=obj00000000000000000015 2023-09-22T21:35:13.339 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: listing bilog for incremental sync; position=00000000002.64.6 2023-09-22T21:35:13.339 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-09-22T21:35:13.339 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14/obj00000000000000000020 2023-09-22T21:35:13.339 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: entry->timestamp=2023-09-22T21:35:02.591196+0000 2023-09-22T21:35:13.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.55.6 op=0 op_state=1 2023-09-22T21:35:13.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:entry[obj00000000000000000015]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:13.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:entry[obj00000000000000000015]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10/obj00000000000000000015 2023-09-22T21:35:13.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000020]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000020[0] 2023-09-22T21:35:13.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: listing bilog for incremental sync; position=00000000002.55.6 2023-09-22T21:35:13.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-09-22T21:35:13.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16/obj00000000000000000005 2023-09-22T21:35:13.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: entry->timestamp=2023-09-22T21:35:02.354612+0000 2023-09-22T21:35:13.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.45.6 op=0 op_state=1 2023-09-22T21:35:13.341 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000005]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000005[0] 2023-09-22T21:35:13.341 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: listing bilog for incremental sync; position=00000000002.45.6 2023-09-22T21:35:13.341 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-09-22T21:35:13.341 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28/obj00000000000000000021 2023-09-22T21:35:13.341 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: entry->timestamp=2023-09-22T21:35:02.809391+0000 2023-09-22T21:35:13.341 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.59.6 op=0 op_state=1 2023-09-22T21:35:13.341 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-09-22T21:35:13.341 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21/obj00000000000000000001 2023-09-22T21:35:13.341 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: entry->timestamp=2023-09-22T21:35:02.674057+0000 2023-09-22T21:35:13.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-09-22T21:35:13.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-09-22T21:35:13.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18/obj00000000000000000003 2023-09-22T21:35:13.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: entry->timestamp=2023-09-22T21:35:03.048518+0000 2023-09-22T21:35:13.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.66.6 op=0 op_state=1 2023-09-22T21:35:13.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:entry[obj00000000000000000021]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000021[0] 2023-09-22T21:35:13.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: listing bilog for incremental sync; position=00000000002.59.6 2023-09-22T21:35:13.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:entry[obj00000000000000000001]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000001[0] 2023-09-22T21:35:13.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: listing bilog for incremental sync; position=00000000002.51.6 2023-09-22T21:35:13.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:entry[obj00000000000000000003]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000003[0] 2023-09-22T21:35:13.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: listing bilog for incremental sync; position=00000000002.66.6 2023-09-22T21:35:13.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: parsed entry: id=00000000002.67.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-09-22T21:35:13.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22/obj00000000000000000009 2023-09-22T21:35:13.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: entry->timestamp=2023-09-22T21:35:02.266827+0000 2023-09-22T21:35:13.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.67.6 op=0 op_state=1 2023-09-22T21:35:13.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:entry[obj00000000000000000009]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000009[0] 2023-09-22T21:35:13.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.332+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: listing bilog for incremental sync; position=00000000002.67.6 2023-09-22T21:35:13.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.336+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-09-22T21:35:13.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.336+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26/obj00000000000000000014 2023-09-22T21:35:13.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.336+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: entry->timestamp=2023-09-22T21:35:02.648222+0000 2023-09-22T21:35:13.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.336+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.62.6 op=0 op_state=1 2023-09-22T21:35:13.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.336+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:entry[obj00000000000000000014]: : injecting data sync error on key=obj00000000000000000014 2023-09-22T21:35:13.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.336+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: listing bilog for incremental sync; position=00000000002.62.6 2023-09-22T21:35:13.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.336+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:entry[obj00000000000000000014]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:13.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.336+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:entry[obj00000000000000000014]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26/obj00000000000000000014 2023-09-22T21:35:13.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.336+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000005]: success 2023-09-22T21:35:13.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.336+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16:4 marker=00000000002.45.6 timestamp=2023-09-22T21:35:02.354612+0000 2023-09-22T21:35:13.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.336+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000020]: success 2023-09-22T21:35:13.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.336+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14:4 marker=00000000002.55.6 timestamp=2023-09-22T21:35:02.591196+0000 2023-09-22T21:35:13.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.336+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:entry[obj00000000000000000001]: success 2023-09-22T21:35:13.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.336+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21:4 marker=00000000002.51.6 timestamp=2023-09-22T21:35:02.674057+0000 2023-09-22T21:35:13.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.340+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:entry[obj00000000000000000021]: success 2023-09-22T21:35:13.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.340+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28:4 marker=00000000002.59.6 timestamp=2023-09-22T21:35:02.809391+0000 2023-09-22T21:35:13.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.340+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:entry[obj00000000000000000003]: success 2023-09-22T21:35:13.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.340+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18:4 marker=00000000002.66.6 timestamp=2023-09-22T21:35:03.048518+0000 2023-09-22T21:35:13.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.340+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:entry[obj00000000000000000009]: success 2023-09-22T21:35:13.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.340+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22:4 marker=00000000002.67.6 timestamp=2023-09-22T21:35:02.266827+0000 2023-09-22T21:35:13.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.364+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: parsed entry: id=00000000001.53.6 iter->object=obj00000000000000000049 iter->instance= name=obj00000000000000000049 instance= ns= 2023-09-22T21:35:13.365 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.364+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7/obj00000000000000000049 2023-09-22T21:35:13.365 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.364+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: entry->timestamp=2023-09-22T21:35:02.887970+0000 2023-09-22T21:35:13.365 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.364+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:entry[obj00000000000000000049]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.53.6 op=0 op_state=1 2023-09-22T21:35:13.365 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.364+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:entry[obj00000000000000000049]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000049[0] 2023-09-22T21:35:13.365 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.364+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: listing bilog for incremental sync; position=00000000001.53.6 2023-09-22T21:35:13.365 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.364+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: parsed entry: id=00000000002.134.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-09-22T21:35:13.365 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.364+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9/obj00000000000000000008 2023-09-22T21:35:13.365 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.364+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: entry->timestamp=2023-09-22T21:35:02.281036+0000 2023-09-22T21:35:13.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.364+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.134.6 op=0 op_state=1 2023-09-22T21:35:13.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.364+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:entry[obj00000000000000000008]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000008[0] 2023-09-22T21:35:13.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.364+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: listing bilog for incremental sync; position=00000000002.134.6 2023-09-22T21:35:13.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.364+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:entry[obj00000000000000000008]: success 2023-09-22T21:35:13.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.364+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9:4 marker=00000000002.134.6 timestamp=2023-09-22T21:35:02.281036+0000 2023-09-22T21:35:13.369 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: parsed entry: id=00000000001.50.6 iter->object=obj00000000000000000059 iter->instance= name=obj00000000000000000059 instance= ns= 2023-09-22T21:35:13.369 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1/obj00000000000000000059 2023-09-22T21:35:13.369 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: entry->timestamp=2023-09-22T21:35:02.568352+0000 2023-09-22T21:35:13.370 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000059]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.50.6 op=0 op_state=1 2023-09-22T21:35:13.370 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000059]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000059[0] 2023-09-22T21:35:13.370 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: listing bilog for incremental sync; position=00000000001.50.6 2023-09-22T21:35:13.370 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: parsed entry: id=00000000001.59.6 iter->object=obj00000000000000000031 iter->instance= name=obj00000000000000000031 instance= ns= 2023-09-22T21:35:13.370 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11/obj00000000000000000031 2023-09-22T21:35:13.370 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: entry->timestamp=2023-09-22T21:35:03.155230+0000 2023-09-22T21:35:13.370 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000031]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.59.6 op=0 op_state=1 2023-09-22T21:35:13.370 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-09-22T21:35:13.370 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13/obj00000000000000000013 2023-09-22T21:35:13.371 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: entry->timestamp=2023-09-22T21:35:02.410768+0000 2023-09-22T21:35:13.371 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-09-22T21:35:13.371 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: a sync operation returned error 2023-09-22T21:35:13.371 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000031]: : injecting data sync error on key=obj00000000000000000031 2023-09-22T21:35:13.371 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: listing bilog for incremental sync; position=00000000001.59.6 2023-09-22T21:35:13.371 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:entry[obj00000000000000000013]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000013[0] 2023-09-22T21:35:13.371 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: listing bilog for incremental sync; position=00000000002.61.6 2023-09-22T21:35:13.371 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000031]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:13.371 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000031]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11/obj00000000000000000031 2023-09-22T21:35:13.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: backing out with sync_status=-5 2023-09-22T21:35:13.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:13.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:13.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: sync status for source bucket shard: 0 2023-09-22T21:35:13.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:entry[obj00000000000000000013]: success 2023-09-22T21:35:13.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.368+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13:4 marker=00000000002.61.6 timestamp=2023-09-22T21:35:02.410768+0000 2023-09-22T21:35:13.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: a sync operation returned error 2023-09-22T21:35:13.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-09-22T21:35:13.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12/obj00000000000000000022 2023-09-22T21:35:13.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: entry->timestamp=2023-09-22T21:35:02.835656+0000 2023-09-22T21:35:13.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-09-22T21:35:13.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: parsed entry: id=00000000001.66.6 iter->object=obj00000000000000000037 iter->instance= name=obj00000000000000000037 instance= ns= 2023-09-22T21:35:13.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15/obj00000000000000000037 2023-09-22T21:35:13.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: entry->timestamp=2023-09-22T21:35:02.991134+0000 2023-09-22T21:35:13.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:entry[obj00000000000000000037]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.66.6 op=0 op_state=1 2023-09-22T21:35:13.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: parsed entry: id=00000000002.60.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-09-22T21:35:13.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17/obj00000000000000000012 2023-09-22T21:35:13.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: entry->timestamp=2023-09-22T21:35:02.488314+0000 2023-09-22T21:35:13.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.60.6 op=0 op_state=1 2023-09-22T21:35:13.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: parsed entry: id=00000000001.57.6 iter->object=obj00000000000000000042 iter->instance= name=obj00000000000000000042 instance= ns= 2023-09-22T21:35:13.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19/obj00000000000000000042 2023-09-22T21:35:13.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: entry->timestamp=2023-09-22T21:35:03.074635+0000 2023-09-22T21:35:13.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:entry[obj00000000000000000042]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.57.6 op=0 op_state=1 2023-09-22T21:35:13.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000022]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000022[0] 2023-09-22T21:35:13.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: listing bilog for incremental sync; position=00000000002.52.6 2023-09-22T21:35:13.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:entry[obj00000000000000000037]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000037[0] 2023-09-22T21:35:13.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: listing bilog for incremental sync; position=00000000001.66.6 2023-09-22T21:35:13.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:entry[obj00000000000000000012]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000012[0] 2023-09-22T21:35:13.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: listing bilog for incremental sync; position=00000000002.60.6 2023-09-22T21:35:13.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:entry[obj00000000000000000042]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000042[0] 2023-09-22T21:35:13.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: listing bilog for incremental sync; position=00000000001.57.6 2023-09-22T21:35:13.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: backing out with sync_status=-5 2023-09-22T21:35:13.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:13.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:13.378 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.378 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: sync status for source bucket shard: 0 2023-09-22T21:35:13.378 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.378 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:entry[obj00000000000000000012]: success 2023-09-22T21:35:13.378 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.372+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17:4 marker=00000000002.60.6 timestamp=2023-09-22T21:35:02.488314+0000 2023-09-22T21:35:13.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-09-22T21:35:13.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24/obj00000000000000000016 2023-09-22T21:35:13.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: entry->timestamp=2023-09-22T21:35:02.514682+0000 2023-09-22T21:35:13.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-09-22T21:35:13.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:entry[obj00000000000000000016]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000016[0] 2023-09-22T21:35:13.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: listing bilog for incremental sync; position=00000000002.47.6 2023-09-22T21:35:13.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-09-22T21:35:13.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23/obj00000000000000000025 2023-09-22T21:35:13.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: entry->timestamp=2023-09-22T21:35:02.338233+0000 2023-09-22T21:35:13.380 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.64.6 op=0 op_state=1 2023-09-22T21:35:13.380 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-09-22T21:35:13.380 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25/obj00000000000000000000 2023-09-22T21:35:13.380 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: entry->timestamp=2023-09-22T21:35:02.253278+0000 2023-09-22T21:35:13.380 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.59.6 op=0 op_state=1 2023-09-22T21:35:13.380 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:entry[obj00000000000000000025]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000025[0] 2023-09-22T21:35:13.380 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: listing bilog for incremental sync; position=00000000002.64.6 2023-09-22T21:35:13.380 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:entry[obj00000000000000000000]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000000[0] 2023-09-22T21:35:13.380 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: listing bilog for incremental sync; position=00000000002.59.6 2023-09-22T21:35:13.381 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: a sync operation returned error 2023-09-22T21:35:13.381 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: backing out with sync_status=-5 2023-09-22T21:35:13.381 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:13.381 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:13.382 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.382 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: sync status for source bucket shard: 0 2023-09-22T21:35:13.382 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.382 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:entry[obj00000000000000000025]: success 2023-09-22T21:35:13.382 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.376+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23:4 marker=00000000002.64.6 timestamp=2023-09-22T21:35:02.338233+0000 2023-09-22T21:35:13.382 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.380+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:entry[obj00000000000000000000]: success 2023-09-22T21:35:13.383 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.380+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25:4 marker=00000000002.59.6 timestamp=2023-09-22T21:35:02.253278+0000 2023-09-22T21:35:13.383 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.380+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-09-22T21:35:13.383 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.380+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20/obj00000000000000000010 2023-09-22T21:35:13.383 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.380+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: entry->timestamp=2023-09-22T21:35:03.233800+0000 2023-09-22T21:35:13.383 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.380+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.63.6 op=0 op_state=1 2023-09-22T21:35:13.383 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.380+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:entry[obj00000000000000000016]: success 2023-09-22T21:35:13.383 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.380+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24:4 marker=00000000002.47.6 timestamp=2023-09-22T21:35:02.514682+0000 2023-09-22T21:35:13.383 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.380+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:entry[obj00000000000000000010]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000010[0] 2023-09-22T21:35:13.383 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.380+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: listing bilog for incremental sync; position=00000000002.63.6 2023-09-22T21:35:13.384 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.380+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:entry[obj00000000000000000027]: success 2023-09-22T21:35:13.384 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.380+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2:4 marker=00000000001.43.6 timestamp=2023-09-22T21:35:03.211837+0000 2023-09-22T21:35:13.384 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.380+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:entry[obj00000000000000000010]: success 2023-09-22T21:35:13.384 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.380+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20:4 marker=00000000002.63.6 timestamp=2023-09-22T21:35:03.233800+0000 2023-09-22T21:35:13.385 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.384+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: a sync operation returned error 2023-09-22T21:35:13.385 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.384+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: backing out with sync_status=-5 2023-09-22T21:35:13.385 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.384+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:13.385 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.384+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:13.386 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.384+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.384+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: sync status for source bucket shard: 0 2023-09-22T21:35:13.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.384+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.412 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.408+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:entry[obj00000000000000000049]: success 2023-09-22T21:35:13.412 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.408+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7:4 marker=00000000001.53.6 timestamp=2023-09-22T21:35:02.887970+0000 2023-09-22T21:35:13.412 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.412+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: a sync operation returned error 2023-09-22T21:35:13.412 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.412+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: backing out with sync_status=-5 2023-09-22T21:35:13.413 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.412+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:13.413 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.412+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:13.413 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.412+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.413 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.412+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: sync status for source bucket shard: 0 2023-09-22T21:35:13.414 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.412+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.412+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000059]: success 2023-09-22T21:35:13.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.412+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1:4 marker=00000000001.50.6 timestamp=2023-09-22T21:35:02.568352+0000 2023-09-22T21:35:13.420 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.416+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000022]: success 2023-09-22T21:35:13.420 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.416+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12:4 marker=00000000002.52.6 timestamp=2023-09-22T21:35:02.835656+0000 2023-09-22T21:35:13.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.420+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-09-22T21:35:13.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.420+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5/obj00000000000000000002 2023-09-22T21:35:13.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.420+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: entry->timestamp=2023-09-22T21:35:02.727864+0000 2023-09-22T21:35:13.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.420+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.30.6 op=0 op_state=1 2023-09-22T21:35:13.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.420+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000002]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000002[0] 2023-09-22T21:35:13.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.420+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: listing bilog for incremental sync; position=00000000002.30.6 2023-09-22T21:35:13.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.420+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-09-22T21:35:13.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.420+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6/obj00000000000000000019 2023-09-22T21:35:13.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.420+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: entry->timestamp=2023-09-22T21:35:03.128117+0000 2023-09-22T21:35:13.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.420+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-09-22T21:35:13.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.420+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000019]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000019[0] 2023-09-22T21:35:13.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.420+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: listing bilog for incremental sync; position=00000000002.61.6 2023-09-22T21:35:13.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.424+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-09-22T21:35:13.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.424+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10/obj00000000000000000015 2023-09-22T21:35:13.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.424+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: entry->timestamp=2023-09-22T21:35:02.754030+0000 2023-09-22T21:35:13.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.424+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.64.6 op=0 op_state=1 2023-09-22T21:35:13.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.424+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:entry[obj00000000000000000015]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000015[0] 2023-09-22T21:35:13.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.424+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: listing bilog for incremental sync; position=00000000002.64.6 2023-09-22T21:35:13.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.424+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000002]: success 2023-09-22T21:35:13.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.424+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5:4 marker=00000000002.30.6 timestamp=2023-09-22T21:35:02.727864+0000 2023-09-22T21:35:13.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.424+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000019]: success 2023-09-22T21:35:13.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.424+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6:4 marker=00000000002.61.6 timestamp=2023-09-22T21:35:03.128117+0000 2023-09-22T21:35:13.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.424+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:entry[obj00000000000000000042]: success 2023-09-22T21:35:13.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.424+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19:4 marker=00000000001.57.6 timestamp=2023-09-22T21:35:03.074635+0000 2023-09-22T21:35:13.429 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.424+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:entry[obj00000000000000000037]: success 2023-09-22T21:35:13.429 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.424+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15:4 marker=00000000001.66.6 timestamp=2023-09-22T21:35:02.991134+0000 2023-09-22T21:35:13.429 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.428+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:entry[obj00000000000000000015]: success 2023-09-22T21:35:13.430 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.428+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10:4 marker=00000000002.64.6 timestamp=2023-09-22T21:35:02.754030+0000 2023-09-22T21:35:13.432 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.432+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-09-22T21:35:13.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.432+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26/obj00000000000000000014 2023-09-22T21:35:13.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.432+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: entry->timestamp=2023-09-22T21:35:02.648222+0000 2023-09-22T21:35:13.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.432+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.62.6 op=0 op_state=1 2023-09-22T21:35:13.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.432+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:entry[obj00000000000000000014]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000014[0] 2023-09-22T21:35:13.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.432+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: listing bilog for incremental sync; position=00000000002.62.6 2023-09-22T21:35:13.435 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.432+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:entry[obj00000000000000000014]: success 2023-09-22T21:35:13.435 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.432+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26:4 marker=00000000002.62.6 timestamp=2023-09-22T21:35:02.648222+0000 2023-09-22T21:35:13.456 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.456+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: parsed entry: id=00000000001.59.6 iter->object=obj00000000000000000031 iter->instance= name=obj00000000000000000031 instance= ns= 2023-09-22T21:35:13.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.456+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11/obj00000000000000000031 2023-09-22T21:35:13.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.456+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: entry->timestamp=2023-09-22T21:35:03.155230+0000 2023-09-22T21:35:13.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.456+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000031]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.59.6 op=0 op_state=1 2023-09-22T21:35:13.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.456+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000031]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000031[0] 2023-09-22T21:35:13.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.456+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: listing bilog for incremental sync; position=00000000001.59.6 2023-09-22T21:35:13.504 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.500+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000031]: success 2023-09-22T21:35:13.504 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.500+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11:4 marker=00000000001.59.6 timestamp=2023-09-22T21:35:03.155230+0000 2023-09-22T21:35:13.511 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.508+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.511 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.508+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.511 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.508+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.512 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.508+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.512 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.508+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: sync status for source bucket shard: 0 2023-09-22T21:35:13.513 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.508+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.513 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.508+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: sync status for source bucket shard: 0 2023-09-22T21:35:13.513 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.508+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: sync status for source bucket shard: 0 2023-09-22T21:35:13.513 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.508+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.513 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.508+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.513 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: sync status for source bucket shard: 0 2023-09-22T21:35:13.513 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.513 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.513 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.515 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.515 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: sync status for source bucket shard: 0 2023-09-22T21:35:13.515 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.515 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.516 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.516 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.516 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.516 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: sync status for source bucket shard: 0 2023-09-22T21:35:13.516 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.516 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: sync status for source bucket shard: 0 2023-09-22T21:35:13.516 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.516 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.516 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: sync status for source bucket shard: 0 2023-09-22T21:35:13.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: sync status for source bucket shard: 0 2023-09-22T21:35:13.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: sync status for source bucket shard: 0 2023-09-22T21:35:13.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: sync status for source bucket shard: 0 2023-09-22T21:35:13.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: sync status for source bucket shard: 0 2023-09-22T21:35:13.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.512+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: sync status for source bucket shard: 0 2023-09-22T21:35:13.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: sync status for source bucket shard: 0 2023-09-22T21:35:13.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: sync status for source bucket shard: 0 2023-09-22T21:35:13.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: parsed entry: id=00000000001.70.6 iter->object=obj00000000000000000467 iter->instance= name=obj00000000000000000467 instance= ns= 2023-09-22T21:35:13.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2/obj00000000000000000467 2023-09-22T21:35:13.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: entry->timestamp=2023-09-22T21:35:03.651153+0000 2023-09-22T21:35:13.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:entry[obj00000000000000000467]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.70.6 op=0 op_state=1 2023-09-22T21:35:13.521 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:entry[obj00000000000000000467]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000467[0] 2023-09-22T21:35:13.521 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: listing bilog for incremental sync; position=00000000001.70.6 2023-09-22T21:35:13.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: sync status for source bucket shard: 0 2023-09-22T21:35:13.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-09-22T21:35:13.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0/obj00000000000000000019 2023-09-22T21:35:13.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: entry->timestamp=2023-09-22T21:35:03.597754+0000 2023-09-22T21:35:13.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.45.6 op=0 op_state=1 2023-09-22T21:35:13.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: sync status for source bucket shard: 0 2023-09-22T21:35:13.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: sync status for source bucket shard: 0 2023-09-22T21:35:13.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: sync status for source bucket shard: 0 2023-09-22T21:35:13.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: sync status for source bucket shard: 0 2023-09-22T21:35:13.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000019]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000019[0] 2023-09-22T21:35:13.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: listing bilog for incremental sync; position=00000000002.45.6 2023-09-22T21:35:13.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: sync status for source bucket shard: 0 2023-09-22T21:35:13.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: sync status for source bucket shard: 0 2023-09-22T21:35:13.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: parsed entry: id=00000000001.66.6 iter->object=obj00000000000000000317 iter->instance= name=obj00000000000000000317 instance= ns= 2023-09-22T21:35:13.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7/obj00000000000000000317 2023-09-22T21:35:13.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: entry->timestamp=2023-09-22T21:35:03.957705+0000 2023-09-22T21:35:13.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:entry[obj00000000000000000317]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.66.6 op=0 op_state=1 2023-09-22T21:35:13.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: parsed entry: id=00000000002.35.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-09-22T21:35:13.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11/obj00000000000000000009 2023-09-22T21:35:13.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: entry->timestamp=2023-09-22T21:35:04.463737+0000 2023-09-22T21:35:13.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.35.6 op=0 op_state=1 2023-09-22T21:35:13.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.516+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.520+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.520+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.520+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.520+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.520+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.520+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.520+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:entry[obj00000000000000000317]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000317[0] 2023-09-22T21:35:13.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.520+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: listing bilog for incremental sync; position=00000000001.66.6 2023-09-22T21:35:13.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.520+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-09-22T21:35:13.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.520+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10/obj00000000000000000021 2023-09-22T21:35:13.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.520+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: entry->timestamp=2023-09-22T21:35:03.739831+0000 2023-09-22T21:35:13.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.520+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.62.6 op=0 op_state=1 2023-09-22T21:35:13.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.520+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.520+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.520+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.520+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000009]: : injecting data sync error on key=obj00000000000000000009 2023-09-22T21:35:13.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.520+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: listing bilog for incremental sync; position=00000000002.35.6 2023-09-22T21:35:13.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.520+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:entry[obj00000000000000000021]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000021[0] 2023-09-22T21:35:13.528 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.520+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: listing bilog for incremental sync; position=00000000002.62.6 2023-09-22T21:35:13.528 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.520+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000009]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:13.528 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.520+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000009]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11/obj00000000000000000009 2023-09-22T21:35:13.529 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: parsed entry: id=00000000001.66.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2023-09-22T21:35:13.529 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12/obj00000000000000000271 2023-09-22T21:35:13.529 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: entry->timestamp=2023-09-22T21:35:03.667222+0000 2023-09-22T21:35:13.529 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.66.6 op=0 op_state=1 2023-09-22T21:35:13.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000271]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000271[0] 2023-09-22T21:35:13.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: listing bilog for incremental sync; position=00000000001.66.6 2023-09-22T21:35:13.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000023 iter->instance= name=obj00000000000000000023 instance= ns= 2023-09-22T21:35:13.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6/obj00000000000000000023 2023-09-22T21:35:13.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: entry->timestamp=2023-09-22T21:35:03.585390+0000 2023-09-22T21:35:13.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000023]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-09-22T21:35:13.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000023]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000023[0] 2023-09-22T21:35:13.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: listing bilog for incremental sync; position=00000000002.61.6 2023-09-22T21:35:13.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: sync status for source bucket shard: 0 2023-09-22T21:35:13.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: sync status for source bucket shard: 0 2023-09-22T21:35:13.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: sync status for source bucket shard: 0 2023-09-22T21:35:13.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: sync status for source bucket shard: 0 2023-09-22T21:35:13.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: sync status for source bucket shard: 0 2023-09-22T21:35:13.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: sync status for source bucket shard: 0 2023-09-22T21:35:13.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]: sync status for source bucket shard: 0 2023-09-22T21:35:13.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-09-22T21:35:13.532 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15/obj00000000000000000007 2023-09-22T21:35:13.532 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: entry->timestamp=2023-09-22T21:35:04.850952+0000 2023-09-22T21:35:13.532 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.63.6 op=0 op_state=1 2023-09-22T21:35:13.532 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-09-22T21:35:13.532 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14/obj00000000000000000018 2023-09-22T21:35:13.532 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: entry->timestamp=2023-09-22T21:35:04.768771+0000 2023-09-22T21:35:13.532 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.68.6 op=0 op_state=1 2023-09-22T21:35:13.532 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.532 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.532 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: sync status for source bucket shard: 0 2023-09-22T21:35:13.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-09-22T21:35:13.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16/obj00000000000000000017 2023-09-22T21:35:13.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: entry->timestamp=2023-09-22T21:35:04.200882+0000 2023-09-22T21:35:13.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-09-22T21:35:13.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-09-22T21:35:13.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13/obj00000000000000000008 2023-09-22T21:35:13.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: entry->timestamp=2023-09-22T21:35:04.437129+0000 2023-09-22T21:35:13.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-09-22T21:35:13.534 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:entry[obj00000000000000000007]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000007[0] 2023-09-22T21:35:13.534 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: listing bilog for incremental sync; position=00000000002.63.6 2023-09-22T21:35:13.534 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000018]: : injecting data sync error on key=obj00000000000000000018 2023-09-22T21:35:13.534 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: listing bilog for incremental sync; position=00000000002.68.6 2023-09-22T21:35:13.534 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.534 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000017]: : injecting data sync error on key=obj00000000000000000017 2023-09-22T21:35:13.534 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: listing bilog for incremental sync; position=00000000002.56.6 2023-09-22T21:35:13.534 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:entry[obj00000000000000000008]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000008[0] 2023-09-22T21:35:13.534 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: listing bilog for incremental sync; position=00000000002.49.6 2023-09-22T21:35:13.534 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000018]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:13.535 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000018]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14/obj00000000000000000018 2023-09-22T21:35:13.535 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000017]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:13.535 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.524+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000017]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16/obj00000000000000000017 2023-09-22T21:35:13.536 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.528+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: sync status for source bucket shard: 0 2023-09-22T21:35:13.536 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.528+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.536 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.528+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000019]: success 2023-09-22T21:35:13.536 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.528+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0:5 marker=00000000002.45.6 timestamp=2023-09-22T21:35:03.597754+0000 2023-09-22T21:35:13.536 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.528+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:entry[obj00000000000000000021]: success 2023-09-22T21:35:13.536 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.528+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10:5 marker=00000000002.62.6 timestamp=2023-09-22T21:35:03.739831+0000 2023-09-22T21:35:13.536 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.532+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: parsed entry: id=00000000002.72.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-09-22T21:35:13.536 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.532+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24/obj00000000000000000011 2023-09-22T21:35:13.536 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.532+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: entry->timestamp=2023-09-22T21:35:03.826433+0000 2023-09-22T21:35:13.537 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.532+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.72.6 op=0 op_state=1 2023-09-22T21:35:13.537 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.532+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:entry[obj00000000000000000011]: : injecting data sync error on key=obj00000000000000000011 2023-09-22T21:35:13.537 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.532+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: listing bilog for incremental sync; position=00000000002.72.6 2023-09-22T21:35:13.537 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.532+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: parsed entry: id=00000000001.72.6 iter->object=obj00000000000000000283 iter->instance= name=obj00000000000000000283 instance= ns= 2023-09-22T21:35:13.537 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.532+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25/obj00000000000000000283 2023-09-22T21:35:13.537 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.532+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: entry->timestamp=2023-09-22T21:35:04.277065+0000 2023-09-22T21:35:13.537 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.532+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:entry[obj00000000000000000283]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.72.6 op=0 op_state=1 2023-09-22T21:35:13.537 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.532+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:entry[obj00000000000000000011]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:13.537 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.532+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:entry[obj00000000000000000011]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24/obj00000000000000000011 2023-09-22T21:35:13.537 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.532+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:entry[obj00000000000000000283]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000283[0] 2023-09-22T21:35:13.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.532+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: listing bilog for incremental sync; position=00000000001.72.6 2023-09-22T21:35:13.539 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:entry[obj00000000000000000007]: success 2023-09-22T21:35:13.539 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15:5 marker=00000000002.63.6 timestamp=2023-09-22T21:35:04.850952+0000 2023-09-22T21:35:13.539 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-09-22T21:35:13.539 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26/obj00000000000000000010 2023-09-22T21:35:13.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: entry->timestamp=2023-09-22T21:35:04.139836+0000 2023-09-22T21:35:13.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-09-22T21:35:13.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:entry[obj00000000000000000010]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000010[0] 2023-09-22T21:35:13.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: listing bilog for incremental sync; position=00000000002.47.6 2023-09-22T21:35:13.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:entry[obj00000000000000000008]: success 2023-09-22T21:35:13.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13:5 marker=00000000002.49.6 timestamp=2023-09-22T21:35:04.437129+0000 2023-09-22T21:35:13.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: parsed entry: id=00000000001.54.6 iter->object=obj00000000000000000133 iter->instance= name=obj00000000000000000133 instance= ns= 2023-09-22T21:35:13.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30/obj00000000000000000133 2023-09-22T21:35:13.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: entry->timestamp=2023-09-22T21:35:03.904168+0000 2023-09-22T21:35:13.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:entry[obj00000000000000000133]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.54.6 op=0 op_state=1 2023-09-22T21:35:13.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: parsed entry: id=00000000001.58.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2023-09-22T21:35:13.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20/obj00000000000000000423 2023-09-22T21:35:13.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: entry->timestamp=2023-09-22T21:35:04.116870+0000 2023-09-22T21:35:13.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.58.6 op=0 op_state=1 2023-09-22T21:35:13.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: parsed entry: id=00000000002.81.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-09-22T21:35:13.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22/obj00000000000000000012 2023-09-22T21:35:13.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: entry->timestamp=2023-09-22T21:35:03.718380+0000 2023-09-22T21:35:13.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.81.6 op=0 op_state=1 2023-09-22T21:35:13.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: parsed entry: id=00000000001.39.6 iter->object=obj00000000000000000029 iter->instance= name=obj00000000000000000029 instance= ns= 2023-09-22T21:35:13.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29/obj00000000000000000029 2023-09-22T21:35:13.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: entry->timestamp=2023-09-22T21:35:03.881350+0000 2023-09-22T21:35:13.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:entry[obj00000000000000000029]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.39.6 op=0 op_state=1 2023-09-22T21:35:13.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:entry[obj00000000000000000133]: : injecting data sync error on key=obj00000000000000000133 2023-09-22T21:35:13.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: listing bilog for incremental sync; position=00000000001.54.6 2023-09-22T21:35:13.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2023-09-22T21:35:13.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28/obj00000000000000000050 2023-09-22T21:35:13.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: entry->timestamp=2023-09-22T21:35:04.354482+0000 2023-09-22T21:35:13.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.41.6 op=0 op_state=1 2023-09-22T21:35:13.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:entry[obj00000000000000000423]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000423[0] 2023-09-22T21:35:13.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: listing bilog for incremental sync; position=00000000001.58.6 2023-09-22T21:35:13.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: a sync operation returned error 2023-09-22T21:35:13.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: a sync operation returned error 2023-09-22T21:35:13.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:entry[obj00000000000000000012]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000012[0] 2023-09-22T21:35:13.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: listing bilog for incremental sync; position=00000000002.81.6 2023-09-22T21:35:13.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:entry[obj00000000000000000029]: : injecting data sync error on key=obj00000000000000000029 2023-09-22T21:35:13.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: listing bilog for incremental sync; position=00000000001.39.6 2023-09-22T21:35:13.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:entry[obj00000000000000000133]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:13.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:entry[obj00000000000000000133]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30/obj00000000000000000133 2023-09-22T21:35:13.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:entry[obj00000000000000000050]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000050[0] 2023-09-22T21:35:13.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: listing bilog for incremental sync; position=00000000002.41.6 2023-09-22T21:35:13.544 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:entry[obj00000000000000000029]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:13.544 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:entry[obj00000000000000000029]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29/obj00000000000000000029 2023-09-22T21:35:13.544 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: backing out with sync_status=-5 2023-09-22T21:35:13.544 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: backing out with sync_status=-5 2023-09-22T21:35:13.544 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:13.544 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:13.544 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:13.544 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:13.544 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:entry[obj00000000000000000317]: success 2023-09-22T21:35:13.545 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.536+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7:5 marker=00000000001.66.6 timestamp=2023-09-22T21:35:03.957705+0000 2023-09-22T21:35:13.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.540+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:entry[obj00000000000000000010]: success 2023-09-22T21:35:13.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.540+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26:5 marker=00000000002.47.6 timestamp=2023-09-22T21:35:04.139836+0000 2023-09-22T21:35:13.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.540+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.540+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.540+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:entry[obj00000000000000000050]: success 2023-09-22T21:35:13.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.540+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28:5 marker=00000000002.41.6 timestamp=2023-09-22T21:35:04.354482+0000 2023-09-22T21:35:13.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.540+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: sync status for source bucket shard: 0 2023-09-22T21:35:13.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.540+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.547 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.540+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: sync status for source bucket shard: 0 2023-09-22T21:35:13.547 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.540+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.547 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.544+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:entry[obj00000000000000000012]: success 2023-09-22T21:35:13.547 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.544+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22:5 marker=00000000002.81.6 timestamp=2023-09-22T21:35:03.718380+0000 2023-09-22T21:35:13.548 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.544+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:entry[obj00000000000000000423]: success 2023-09-22T21:35:13.548 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.544+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20:5 marker=00000000001.58.6 timestamp=2023-09-22T21:35:04.116870+0000 2023-09-22T21:35:13.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.560+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-09-22T21:35:13.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.560+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1/obj00000000000000000006 2023-09-22T21:35:13.561 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.560+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: entry->timestamp=2023-09-22T21:35:04.793067+0000 2023-09-22T21:35:13.561 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.560+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.53.6 op=0 op_state=1 2023-09-22T21:35:13.561 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.560+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000006]: : injecting data sync error on key=obj00000000000000000006 2023-09-22T21:35:13.561 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.560+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: listing bilog for incremental sync; position=00000000002.53.6 2023-09-22T21:35:13.561 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.560+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000006]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:13.561 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.560+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000006]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1/obj00000000000000000006 2023-09-22T21:35:13.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.564+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-09-22T21:35:13.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.564+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5/obj00000000000000000004 2023-09-22T21:35:13.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.564+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: entry->timestamp=2023-09-22T21:35:04.063167+0000 2023-09-22T21:35:13.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.564+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-09-22T21:35:13.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.564+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000004]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000004[0] 2023-09-22T21:35:13.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.564+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: listing bilog for incremental sync; position=00000000002.69.6 2023-09-22T21:35:13.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.564+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000004]: success 2023-09-22T21:35:13.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.564+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5:5 marker=00000000002.69.6 timestamp=2023-09-22T21:35:04.063167+0000 2023-09-22T21:35:13.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.564+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:entry[obj00000000000000000467]: success 2023-09-22T21:35:13.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.564+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2:5 marker=00000000001.70.6 timestamp=2023-09-22T21:35:03.651153+0000 2023-09-22T21:35:13.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-09-22T21:35:13.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23/obj00000000000000000001 2023-09-22T21:35:13.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: entry->timestamp=2023-09-22T21:35:03.797539+0000 2023-09-22T21:35:13.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-09-22T21:35:13.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-09-22T21:35:13.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19/obj00000000000000000003 2023-09-22T21:35:13.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: entry->timestamp=2023-09-22T21:35:04.299787+0000 2023-09-22T21:35:13.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-09-22T21:35:13.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: parsed entry: id=00000000002.71.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-09-22T21:35:13.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21/obj00000000000000000002 2023-09-22T21:35:13.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: entry->timestamp=2023-09-22T21:35:04.881955+0000 2023-09-22T21:35:13.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.71.6 op=0 op_state=1 2023-09-22T21:35:13.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-09-22T21:35:13.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9/obj00000000000000000000 2023-09-22T21:35:13.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: entry->timestamp=2023-09-22T21:35:04.380987+0000 2023-09-22T21:35:13.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.58.6 op=0 op_state=1 2023-09-22T21:35:13.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-09-22T21:35:13.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8/obj00000000000000000022 2023-09-22T21:35:13.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: entry->timestamp=2023-09-22T21:35:03.980645+0000 2023-09-22T21:35:13.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.74.6 op=0 op_state=1 2023-09-22T21:35:13.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-09-22T21:35:13.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3/obj00000000000000000005 2023-09-22T21:35:13.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: entry->timestamp=2023-09-22T21:35:04.037073+0000 2023-09-22T21:35:13.573 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-09-22T21:35:13.573 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:entry[obj00000000000000000001]: : injecting data sync error on key=obj00000000000000000001 2023-09-22T21:35:13.573 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: listing bilog for incremental sync; position=00000000002.49.6 2023-09-22T21:35:13.573 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:entry[obj00000000000000000003]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000003[0] 2023-09-22T21:35:13.573 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: listing bilog for incremental sync; position=00000000002.47.6 2023-09-22T21:35:13.573 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:entry[obj00000000000000000002]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000002[0] 2023-09-22T21:35:13.573 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: listing bilog for incremental sync; position=00000000002.71.6 2023-09-22T21:35:13.573 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:entry[obj00000000000000000000]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000000[0] 2023-09-22T21:35:13.573 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: listing bilog for incremental sync; position=00000000002.58.6 2023-09-22T21:35:13.574 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:entry[obj00000000000000000022]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000022[0] 2023-09-22T21:35:13.574 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: listing bilog for incremental sync; position=00000000002.74.6 2023-09-22T21:35:13.574 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:entry[obj00000000000000000005]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000005[0] 2023-09-22T21:35:13.574 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: listing bilog for incremental sync; position=00000000002.50.6 2023-09-22T21:35:13.574 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:entry[obj00000000000000000001]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:13.574 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.568+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:entry[obj00000000000000000001]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23/obj00000000000000000001 2023-09-22T21:35:13.575 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.572+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:entry[obj00000000000000000003]: success 2023-09-22T21:35:13.575 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.572+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19:5 marker=00000000002.47.6 timestamp=2023-09-22T21:35:04.299787+0000 2023-09-22T21:35:13.575 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.572+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:entry[obj00000000000000000022]: success 2023-09-22T21:35:13.575 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.572+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8:5 marker=00000000002.74.6 timestamp=2023-09-22T21:35:03.980645+0000 2023-09-22T21:35:13.575 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.572+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: a sync operation returned error 2023-09-22T21:35:13.575 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.572+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: backing out with sync_status=-5 2023-09-22T21:35:13.575 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.572+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:13.575 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.572+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:13.575 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.572+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:entry[obj00000000000000000002]: success 2023-09-22T21:35:13.576 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.572+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21:5 marker=00000000002.71.6 timestamp=2023-09-22T21:35:04.881955+0000 2023-09-22T21:35:13.576 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.572+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:entry[obj00000000000000000005]: success 2023-09-22T21:35:13.576 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.572+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3:5 marker=00000000002.50.6 timestamp=2023-09-22T21:35:04.037073+0000 2023-09-22T21:35:13.576 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.572+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:entry[obj00000000000000000000]: success 2023-09-22T21:35:13.576 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.572+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9:5 marker=00000000002.58.6 timestamp=2023-09-22T21:35:04.380987+0000 2023-09-22T21:35:13.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.576+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]: parsed entry: id=00000000001.141.6 iter->object=obj00000000000000000061 iter->instance= name=obj00000000000000000061 instance= ns= 2023-09-22T21:35:13.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.576+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27/obj00000000000000000061 2023-09-22T21:35:13.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.576+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]: entry->timestamp=2023-09-22T21:35:04.223951+0000 2023-09-22T21:35:13.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.576+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:entry[obj00000000000000000061]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.141.6 op=0 op_state=1 2023-09-22T21:35:13.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.576+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:entry[obj00000000000000000061]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000061[0] 2023-09-22T21:35:13.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.576+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]: listing bilog for incremental sync; position=00000000001.141.6 2023-09-22T21:35:13.578 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.576+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: a sync operation returned error 2023-09-22T21:35:13.578 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.576+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: backing out with sync_status=-5 2023-09-22T21:35:13.578 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.576+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:13.578 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.576+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:13.579 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.576+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.579 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.576+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000271]: success 2023-09-22T21:35:13.579 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.576+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12:5 marker=00000000001.66.6 timestamp=2023-09-22T21:35:03.667222+0000 2023-09-22T21:35:13.580 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.576+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: sync status for source bucket shard: 0 2023-09-22T21:35:13.580 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.576+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.581 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.580+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000023]: success 2023-09-22T21:35:13.581 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.580+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6:5 marker=00000000002.61.6 timestamp=2023-09-22T21:35:03.585390+0000 2023-09-22T21:35:13.582 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.580+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.582 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.580+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: sync status for source bucket shard: 0 2023-09-22T21:35:13.582 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.580+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.584 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.584+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: a sync operation returned error 2023-09-22T21:35:13.584 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.584+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: backing out with sync_status=-5 2023-09-22T21:35:13.584 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.584+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:13.584 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.584+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:13.586 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.584+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:entry[obj00000000000000000283]: success 2023-09-22T21:35:13.586 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.584+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25:5 marker=00000000001.72.6 timestamp=2023-09-22T21:35:04.277065+0000 2023-09-22T21:35:13.588 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.584+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.588+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: sync status for source bucket shard: 0 2023-09-22T21:35:13.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.588+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.588+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: a sync operation returned error 2023-09-22T21:35:13.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.588+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-09-22T21:35:13.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.588+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16/obj00000000000000000017 2023-09-22T21:35:13.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.588+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: entry->timestamp=2023-09-22T21:35:04.200882+0000 2023-09-22T21:35:13.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.588+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-09-22T21:35:13.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.588+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-09-22T21:35:13.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.588+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14/obj00000000000000000018 2023-09-22T21:35:13.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.588+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: entry->timestamp=2023-09-22T21:35:04.768771+0000 2023-09-22T21:35:13.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.588+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.68.6 op=0 op_state=1 2023-09-22T21:35:13.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.588+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000017]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000017[0] 2023-09-22T21:35:13.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.588+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: listing bilog for incremental sync; position=00000000002.56.6 2023-09-22T21:35:13.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.588+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000018]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000018[0] 2023-09-22T21:35:13.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.588+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: listing bilog for incremental sync; position=00000000002.68.6 2023-09-22T21:35:13.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.588+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: backing out with sync_status=-5 2023-09-22T21:35:13.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.588+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:13.591 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.588+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:13.594 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.592+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.595 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.592+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: sync status for source bucket shard: 0 2023-09-22T21:35:13.595 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.592+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.604+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: a sync operation returned error 2023-09-22T21:35:13.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.604+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: backing out with sync_status=-5 2023-09-22T21:35:13.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.604+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:13.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.604+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:13.606 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.604+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.606 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.604+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: sync status for source bucket shard: 0 2023-09-22T21:35:13.606 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.604+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.612 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.612+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: a sync operation returned error 2023-09-22T21:35:13.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.612+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: backing out with sync_status=-5 2023-09-22T21:35:13.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.612+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:13.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.612+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:13.615 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.612+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.615 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.612+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: sync status for source bucket shard: 0 2023-09-22T21:35:13.615 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.612+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.624 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.620+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:entry[obj00000000000000000061]: success 2023-09-22T21:35:13.624 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.620+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27:5 marker=00000000001.141.6 timestamp=2023-09-22T21:35:04.223951+0000 2023-09-22T21:35:13.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.624+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: parsed entry: id=00000000002.35.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-09-22T21:35:13.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.624+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11/obj00000000000000000009 2023-09-22T21:35:13.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.624+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: entry->timestamp=2023-09-22T21:35:04.463737+0000 2023-09-22T21:35:13.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.624+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.35.6 op=0 op_state=1 2023-09-22T21:35:13.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.624+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: parsed entry: id=00000000002.72.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-09-22T21:35:13.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.624+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24/obj00000000000000000011 2023-09-22T21:35:13.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.624+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: entry->timestamp=2023-09-22T21:35:03.826433+0000 2023-09-22T21:35:13.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.624+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.72.6 op=0 op_state=1 2023-09-22T21:35:13.626 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.624+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000009]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000009[0] 2023-09-22T21:35:13.626 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.624+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: listing bilog for incremental sync; position=00000000002.35.6 2023-09-22T21:35:13.626 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.624+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:entry[obj00000000000000000011]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000011[0] 2023-09-22T21:35:13.626 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.624+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: listing bilog for incremental sync; position=00000000002.72.6 2023-09-22T21:35:13.627 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.624+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:entry[obj00000000000000000011]: success 2023-09-22T21:35:13.627 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.624+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24:5 marker=00000000002.72.6 timestamp=2023-09-22T21:35:03.826433+0000 2023-09-22T21:35:13.631 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.628+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000009]: success 2023-09-22T21:35:13.631 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.628+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11:5 marker=00000000002.35.6 timestamp=2023-09-22T21:35:04.463737+0000 2023-09-22T21:35:13.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.632+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: parsed entry: id=00000000001.39.6 iter->object=obj00000000000000000029 iter->instance= name=obj00000000000000000029 instance= ns= 2023-09-22T21:35:13.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.632+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29/obj00000000000000000029 2023-09-22T21:35:13.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.632+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: entry->timestamp=2023-09-22T21:35:03.881350+0000 2023-09-22T21:35:13.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.632+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:entry[obj00000000000000000029]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.39.6 op=0 op_state=1 2023-09-22T21:35:13.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.632+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:entry[obj00000000000000000029]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000029[0] 2023-09-22T21:35:13.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.632+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: listing bilog for incremental sync; position=00000000001.39.6 2023-09-22T21:35:13.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.636+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: parsed entry: id=00000000001.54.6 iter->object=obj00000000000000000133 iter->instance= name=obj00000000000000000133 instance= ns= 2023-09-22T21:35:13.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.636+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30/obj00000000000000000133 2023-09-22T21:35:13.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.636+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: entry->timestamp=2023-09-22T21:35:03.904168+0000 2023-09-22T21:35:13.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.636+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:entry[obj00000000000000000133]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.54.6 op=0 op_state=1 2023-09-22T21:35:13.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.636+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:entry[obj00000000000000000133]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000133[0] 2023-09-22T21:35:13.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.636+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: listing bilog for incremental sync; position=00000000001.54.6 2023-09-22T21:35:13.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.636+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000018]: success 2023-09-22T21:35:13.638 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.636+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14:5 marker=00000000002.68.6 timestamp=2023-09-22T21:35:04.768771+0000 2023-09-22T21:35:13.639 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.636+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000017]: success 2023-09-22T21:35:13.639 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.636+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16:5 marker=00000000002.56.6 timestamp=2023-09-22T21:35:04.200882+0000 2023-09-22T21:35:13.648 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.648+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-09-22T21:35:13.648 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.648+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1/obj00000000000000000006 2023-09-22T21:35:13.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.648+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: entry->timestamp=2023-09-22T21:35:04.793067+0000 2023-09-22T21:35:13.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.648+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.53.6 op=0 op_state=1 2023-09-22T21:35:13.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.648+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000006]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000006[0] 2023-09-22T21:35:13.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.648+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: listing bilog for incremental sync; position=00000000002.53.6 2023-09-22T21:35:13.650 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.648+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000006]: success 2023-09-22T21:35:13.651 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.648+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1:5 marker=00000000002.53.6 timestamp=2023-09-22T21:35:04.793067+0000 2023-09-22T21:35:13.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.664+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-09-22T21:35:13.665 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.664+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23/obj00000000000000000001 2023-09-22T21:35:13.665 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.664+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: entry->timestamp=2023-09-22T21:35:03.797539+0000 2023-09-22T21:35:13.665 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.664+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-09-22T21:35:13.665 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.664+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:entry[obj00000000000000000001]: : injecting data sync error on key=obj00000000000000000001 2023-09-22T21:35:13.665 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.664+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: listing bilog for incremental sync; position=00000000002.49.6 2023-09-22T21:35:13.665 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.664+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:entry[obj00000000000000000001]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:13.665 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.664+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:entry[obj00000000000000000001]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23/obj00000000000000000001 2023-09-22T21:35:13.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.684+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:entry[obj00000000000000000133]: success 2023-09-22T21:35:13.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.684+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30:5 marker=00000000001.54.6 timestamp=2023-09-22T21:35:03.904168+0000 2023-09-22T21:35:13.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.684+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:entry[obj00000000000000000029]: success 2023-09-22T21:35:13.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.684+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29:5 marker=00000000001.39.6 timestamp=2023-09-22T21:35:03.881350+0000 2023-09-22T21:35:13.708 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.708+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: a sync operation returned error 2023-09-22T21:35:13.708 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.708+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: backing out with sync_status=-5 2023-09-22T21:35:13.709 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.708+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:13.709 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.708+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:13.709 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.708+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.709 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.708+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: sync status for source bucket shard: 0 2023-09-22T21:35:13.710 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.708+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.752+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-09-22T21:35:13.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.752+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23/obj00000000000000000001 2023-09-22T21:35:13.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.752+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: entry->timestamp=2023-09-22T21:35:03.797539+0000 2023-09-22T21:35:13.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.752+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-09-22T21:35:13.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.752+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:entry[obj00000000000000000001]: : injecting data sync error on key=obj00000000000000000001 2023-09-22T21:35:13.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.752+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: listing bilog for incremental sync; position=00000000002.49.6 2023-09-22T21:35:13.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.752+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:entry[obj00000000000000000001]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:13.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.752+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:entry[obj00000000000000000001]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23/obj00000000000000000001 2023-09-22T21:35:13.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.796+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: a sync operation returned error 2023-09-22T21:35:13.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.796+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: backing out with sync_status=-5 2023-09-22T21:35:13.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.796+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:13.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.796+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:13.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.796+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.796+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: sync status for source bucket shard: 0 2023-09-22T21:35:13.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.796+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.840+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-09-22T21:35:13.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.840+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23/obj00000000000000000001 2023-09-22T21:35:13.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.840+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: entry->timestamp=2023-09-22T21:35:03.797539+0000 2023-09-22T21:35:13.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.840+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-09-22T21:35:13.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.840+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:entry[obj00000000000000000001]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000001[0] 2023-09-22T21:35:13.842 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.840+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: listing bilog for incremental sync; position=00000000002.49.6 2023-09-22T21:35:13.845 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.844+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:entry[obj00000000000000000001]: success 2023-09-22T21:35:13.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.844+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23:5 marker=00000000002.49.6 timestamp=2023-09-22T21:35:03.797539+0000 2023-09-22T21:35:13.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.896+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.896+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.896+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.896+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.900 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.896+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.900 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.896+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: sync status for source bucket shard: 0 2023-09-22T21:35:13.900 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.896+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.900 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.896+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: sync status for source bucket shard: 0 2023-09-22T21:35:13.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.896+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: sync status for source bucket shard: 0 2023-09-22T21:35:13.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.896+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.900+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.900+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.900+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: sync status for source bucket shard: 0 2023-09-22T21:35:13.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.900+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:4]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.900+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.903 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.900+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: sync status for source bucket shard: 0 2023-09-22T21:35:13.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.900+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.900+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: sync status for source bucket shard: 0 2023-09-22T21:35:13.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.900+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.904+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.904+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.904+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.904+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.904+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.904+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.904+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: parsed entry: id=00000000002.85.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-09-22T21:35:13.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.904+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2/obj00000000000000000011 2023-09-22T21:35:13.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.904+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: entry->timestamp=2023-09-22T21:35:12.074000+0000 2023-09-22T21:35:13.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.904+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.85.6 op=0 op_state=1 2023-09-22T21:35:13.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.904+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: parsed entry: id=00000000002.85.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-09-22T21:35:13.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.904+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1/obj00000000000000000000 2023-09-22T21:35:13.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.904+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: entry->timestamp=2023-09-22T21:35:05.571412+0000 2023-09-22T21:35:13.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.904+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.85.6 op=0 op_state=1 2023-09-22T21:35:13.909 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.904+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:entry[obj00000000000000000011]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000011[0] 2023-09-22T21:35:13.909 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.904+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: listing bilog for incremental sync; position=00000000002.85.6 2023-09-22T21:35:13.909 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.904+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000000]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000000[0] 2023-09-22T21:35:13.909 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.904+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: listing bilog for incremental sync; position=00000000002.85.6 2023-09-22T21:35:13.909 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.908+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: sync status for source bucket shard: 0 2023-09-22T21:35:13.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.908+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: sync status for source bucket shard: 0 2023-09-22T21:35:13.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.908+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.908+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: parsed entry: id=00000000001.54.6 iter->object=obj00000000000000000065 iter->instance= name=obj00000000000000000065 instance= ns= 2023-09-22T21:35:13.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.908+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5/obj00000000000000000065 2023-09-22T21:35:13.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.908+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: entry->timestamp=2023-09-22T21:35:12.101872+0000 2023-09-22T21:35:13.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.908+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000065]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.54.6 op=0 op_state=1 2023-09-22T21:35:13.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.908+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.908+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:8]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.908+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000065]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000065[0] 2023-09-22T21:35:13.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.908+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: listing bilog for incremental sync; position=00000000001.54.6 2023-09-22T21:35:13.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.908+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.908+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: parsed entry: id=00000000002.84.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-09-22T21:35:13.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.908+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6/obj00000000000000000005 2023-09-22T21:35:13.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.908+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: entry->timestamp=2023-09-22T21:35:11.989247+0000 2023-09-22T21:35:13.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.908+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.84.6 op=0 op_state=1 2023-09-22T21:35:13.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.908+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000005]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000005[0] 2023-09-22T21:35:13.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.908+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: listing bilog for incremental sync; position=00000000002.84.6 2023-09-22T21:35:13.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.908+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.912+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: sync status for source bucket shard: 0 2023-09-22T21:35:13.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.912+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: sync status for source bucket shard: 0 2023-09-22T21:35:13.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.916+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: sync status for source bucket shard: 0 2023-09-22T21:35:13.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.916+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: sync status for source bucket shard: 0 2023-09-22T21:35:13.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.916+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.916+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.916+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.916+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.916+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.916+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.916+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.916+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: sync status for source bucket shard: 0 2023-09-22T21:35:13.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.916+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: sync status for source bucket shard: 0 2023-09-22T21:35:13.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.924+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.924+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.924+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.926 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.924+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: sync status for source bucket shard: 0 2023-09-22T21:35:13.927 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.924+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.927 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.924+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.927 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.924+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.927 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.924+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.927 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.924+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.924+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.924+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.924+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.924+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.924+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:entry[obj00000000000000000000]: success 2023-09-22T21:35:13.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.924+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:1:6 marker=00000000002.85.6 timestamp=2023-09-22T21:35:05.571412+0000 2023-09-22T21:35:13.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.924+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: sync status for source bucket shard: 0 2023-09-22T21:35:13.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.924+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.928+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: sync status for source bucket shard: 0 2023-09-22T21:35:13.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.928+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: sync status for source bucket shard: 0 2023-09-22T21:35:13.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.928+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.928+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.928+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:18]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.928+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: sync status for source bucket shard: 0 2023-09-22T21:35:13.933 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.932+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.933 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.932+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.934 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.932+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: sync status for source bucket shard: 0 2023-09-22T21:35:13.934 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.932+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: sync status for source bucket shard: 0 2023-09-22T21:35:13.934 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.932+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.935 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.932+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.935 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.932+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.935 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.932+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.937 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.936+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: sync status for source bucket shard: 0 2023-09-22T21:35:13.937 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.936+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: sync status for source bucket shard: 0 2023-09-22T21:35:13.938 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.936+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: sync status for source bucket shard: 0 2023-09-22T21:35:13.938 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.936+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: sync status for source bucket shard: 0 2023-09-22T21:35:13.938 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.936+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]: sync status for source bucket shard: 0 2023-09-22T21:35:13.938 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.936+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: sync status for source bucket shard: 0 2023-09-22T21:35:13.939 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.936+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.939 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.936+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.939 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.936+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.939 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.936+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.939 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.936+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.939 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.936+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.939 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.936+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:entry[obj00000000000000000011]: success 2023-09-22T21:35:13.940 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.936+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:2:6 marker=00000000002.85.6 timestamp=2023-09-22T21:35:12.074000+0000 2023-09-22T21:35:13.944 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.944 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: sync status for source bucket shard: 0 2023-09-22T21:35:13.946 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.946 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.946 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: sync status for source bucket shard: 0 2023-09-22T21:35:13.946 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: sync status for source bucket shard: 0 2023-09-22T21:35:13.946 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: sync status for source bucket shard: 0 2023-09-22T21:35:13.946 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31]: sync status for source bucket shard: 0 2023-09-22T21:35:13.946 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.947 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.947 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.947 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.947 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:13.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: parsed entry: id=00000000001.54.6 iter->object=obj00000000000000000062 iter->instance= name=obj00000000000000000062 instance= ns= 2023-09-22T21:35:13.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12/obj00000000000000000062 2023-09-22T21:35:13.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: entry->timestamp=2023-09-22T21:35:05.285275+0000 2023-09-22T21:35:13.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000062]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.54.6 op=0 op_state=1 2023-09-22T21:35:13.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: parsed entry: id=00000000002.81.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-09-22T21:35:13.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13/obj00000000000000000009 2023-09-22T21:35:13.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: entry->timestamp=2023-09-22T21:35:05.298620+0000 2023-09-22T21:35:13.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.81.6 op=0 op_state=1 2023-09-22T21:35:13.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2023-09-22T21:35:13.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14/obj00000000000000000040 2023-09-22T21:35:13.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: entry->timestamp=2023-09-22T21:35:12.157791+0000 2023-09-22T21:35:13.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.55.6 op=0 op_state=1 2023-09-22T21:35:13.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:entry[obj00000000000000000005]: success 2023-09-22T21:35:13.953 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.944+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:6:6 marker=00000000002.84.6 timestamp=2023-09-22T21:35:11.989247+0000 2023-09-22T21:35:13.953 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.948+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000062]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000062[0] 2023-09-22T21:35:13.953 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.948+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: listing bilog for incremental sync; position=00000000001.54.6 2023-09-22T21:35:13.953 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.948+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:entry[obj00000000000000000009]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000009[0] 2023-09-22T21:35:13.953 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.948+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: listing bilog for incremental sync; position=00000000002.81.6 2023-09-22T21:35:13.953 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.948+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:entry[obj00000000000000000065]: success 2023-09-22T21:35:13.953 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.948+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:5:6 marker=00000000001.54.6 timestamp=2023-09-22T21:35:12.101872+0000 2023-09-22T21:35:13.953 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.948+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000040]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000040[0] 2023-09-22T21:35:13.953 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.948+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: listing bilog for incremental sync; position=00000000002.55.6 2023-09-22T21:35:13.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.952+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]: sync status for source bucket shard: 0 2023-09-22T21:35:13.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.952+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.962 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.952+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32]: sync status for source bucket shard: 0 2023-09-22T21:35:13.962 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.952+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.962 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-09-22T21:35:13.962 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3/obj00000000000000000001 2023-09-22T21:35:13.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: entry->timestamp=2023-09-22T21:35:12.531706+0000 2023-09-22T21:35:13.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.64.6 op=0 op_state=1 2023-09-22T21:35:13.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: parsed entry: id=00000000002.81.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-09-22T21:35:13.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10/obj00000000000000000026 2023-09-22T21:35:13.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: entry->timestamp=2023-09-22T21:35:12.586212+0000 2023-09-22T21:35:13.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.81.6 op=0 op_state=1 2023-09-22T21:35:13.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-09-22T21:35:13.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11/obj00000000000000000030 2023-09-22T21:35:13.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: entry->timestamp=2023-09-22T21:35:11.934521+0000 2023-09-22T21:35:13.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-09-22T21:35:13.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-09-22T21:35:13.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20/obj00000000000000000006 2023-09-22T21:35:13.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: entry->timestamp=2023-09-22T21:35:12.246181+0000 2023-09-22T21:35:13.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.62.6 op=0 op_state=1 2023-09-22T21:35:13.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-09-22T21:35:13.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9/obj00000000000000000015 2023-09-22T21:35:13.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: entry->timestamp=2023-09-22T21:35:05.483352+0000 2023-09-22T21:35:13.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.48.6 op=0 op_state=1 2023-09-22T21:35:13.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35]: sync status for source bucket shard: 0 2023-09-22T21:35:13.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36]: sync status for source bucket shard: 0 2023-09-22T21:35:13.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34]: sync status for source bucket shard: 0 2023-09-22T21:35:13.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-09-22T21:35:13.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21/obj00000000000000000017 2023-09-22T21:35:13.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: entry->timestamp=2023-09-22T21:35:05.738308+0000 2023-09-22T21:35:13.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.58.6 op=0 op_state=1 2023-09-22T21:35:13.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: parsed entry: id=00000000002.89.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-09-22T21:35:13.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22/obj00000000000000000007 2023-09-22T21:35:13.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: entry->timestamp=2023-09-22T21:35:11.723056+0000 2023-09-22T21:35:13.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.89.6 op=0 op_state=1 2023-09-22T21:35:13.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:entry[obj00000000000000000001]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000001[0] 2023-09-22T21:35:13.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.956+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: listing bilog for incremental sync; position=00000000002.64.6 2023-09-22T21:35:13.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.960+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:entry[obj00000000000000000026]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000026[0] 2023-09-22T21:35:13.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.960+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: listing bilog for incremental sync; position=00000000002.81.6 2023-09-22T21:35:13.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.960+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000030]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000030[0] 2023-09-22T21:35:13.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.960+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: listing bilog for incremental sync; position=00000000002.52.6 2023-09-22T21:35:13.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.960+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:entry[obj00000000000000000006]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000006[0] 2023-09-22T21:35:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.960+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: listing bilog for incremental sync; position=00000000002.62.6 2023-09-22T21:35:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.960+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:entry[obj00000000000000000015]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000015[0] 2023-09-22T21:35:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.960+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: listing bilog for incremental sync; position=00000000002.48.6 2023-09-22T21:35:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.960+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.960+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.960+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34]: listing bilog for incremental sync; position= 2023-09-22T21:35:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.960+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:entry[obj00000000000000000017]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000017[0] 2023-09-22T21:35:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.960+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: listing bilog for incremental sync; position=00000000002.58.6 2023-09-22T21:35:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.960+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:entry[obj00000000000000000007]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000007[0] 2023-09-22T21:35:13.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.960+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: listing bilog for incremental sync; position=00000000002.89.6 2023-09-22T21:35:13.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: parsed entry: id=00000000002.79.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-09-22T21:35:13.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15/obj00000000000000000024 2023-09-22T21:35:13.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: entry->timestamp=2023-09-22T21:35:05.767101+0000 2023-09-22T21:35:13.969 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.79.6 op=0 op_state=1 2023-09-22T21:35:13.969 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: parsed entry: id=00000000002.73.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2023-09-22T21:35:13.969 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19/obj00000000000000000045 2023-09-22T21:35:13.969 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: entry->timestamp=2023-09-22T21:35:12.356751+0000 2023-09-22T21:35:13.969 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.73.6 op=0 op_state=1 2023-09-22T21:35:13.969 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2023-09-22T21:35:13.969 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24/obj00000000000000000027 2023-09-22T21:35:13.969 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: entry->timestamp=2023-09-22T21:35:09.970418+0000 2023-09-22T21:35:13.969 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.65.6 op=0 op_state=1 2023-09-22T21:35:13.970 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:entry[obj00000000000000000024]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000024[0] 2023-09-22T21:35:13.970 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: listing bilog for incremental sync; position=00000000002.79.6 2023-09-22T21:35:13.970 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: parsed entry: id=00000000002.79.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-09-22T21:35:13.970 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29/obj00000000000000000004 2023-09-22T21:35:13.970 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: entry->timestamp=2023-09-22T21:35:12.419375+0000 2023-09-22T21:35:13.970 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.79.6 op=0 op_state=1 2023-09-22T21:35:13.970 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:entry[obj00000000000000000045]: : injecting data sync error on key=obj00000000000000000045 2023-09-22T21:35:13.970 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: listing bilog for incremental sync; position=00000000002.73.6 2023-09-22T21:35:13.970 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:entry[obj00000000000000000027]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000027[0] 2023-09-22T21:35:13.971 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: listing bilog for incremental sync; position=00000000002.65.6 2023-09-22T21:35:13.971 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:entry[obj00000000000000000004]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000004[0] 2023-09-22T21:35:13.971 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: listing bilog for incremental sync; position=00000000002.79.6 2023-09-22T21:35:13.971 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:entry[obj00000000000000000045]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:13.971 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.964+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:entry[obj00000000000000000045]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19/obj00000000000000000045 2023-09-22T21:35:13.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: parsed entry: id=00000000002.67.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-09-22T21:35:13.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16/obj00000000000000000012 2023-09-22T21:35:13.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: entry->timestamp=2023-09-22T21:35:11.883712+0000 2023-09-22T21:35:13.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.67.6 op=0 op_state=1 2023-09-22T21:35:13.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: parsed entry: id=00000000002.89.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-09-22T21:35:13.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28/obj00000000000000000020 2023-09-22T21:35:13.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: entry->timestamp=2023-09-22T21:35:12.502061+0000 2023-09-22T21:35:13.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.89.6 op=0 op_state=1 2023-09-22T21:35:13.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:entry[obj00000000000000000009]: success 2023-09-22T21:35:13.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:13:6 marker=00000000002.81.6 timestamp=2023-09-22T21:35:05.298620+0000 2023-09-22T21:35:13.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000012]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000012[0] 2023-09-22T21:35:13.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: listing bilog for incremental sync; position=00000000002.67.6 2023-09-22T21:35:13.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: parsed entry: id=00000000002.77.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-09-22T21:35:13.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7/obj00000000000000000016 2023-09-22T21:35:13.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: entry->timestamp=2023-09-22T21:35:05.400454+0000 2023-09-22T21:35:13.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.77.6 op=0 op_state=1 2023-09-22T21:35:13.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:entry[obj00000000000000000020]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000020[0] 2023-09-22T21:35:13.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: listing bilog for incremental sync; position=00000000002.89.6 2023-09-22T21:35:13.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:entry[obj00000000000000000040]: success 2023-09-22T21:35:13.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:14:6 marker=00000000002.55.6 timestamp=2023-09-22T21:35:12.157791+0000 2023-09-22T21:35:13.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:entry[obj00000000000000000016]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000016[0] 2023-09-22T21:35:13.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.968+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: listing bilog for incremental sync; position=00000000002.77.6 2023-09-22T21:35:13.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.972+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:entry[obj00000000000000000001]: success 2023-09-22T21:35:13.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.972+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:3:6 marker=00000000002.64.6 timestamp=2023-09-22T21:35:12.531706+0000 2023-09-22T21:35:13.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.972+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:entry[obj00000000000000000030]: success 2023-09-22T21:35:13.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.972+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:11:6 marker=00000000002.52.6 timestamp=2023-09-22T21:35:11.934521+0000 2023-09-22T21:35:13.980 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.976+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:entry[obj00000000000000000015]: success 2023-09-22T21:35:13.980 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.976+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:9:6 marker=00000000002.48.6 timestamp=2023-09-22T21:35:05.483352+0000 2023-09-22T21:35:13.980 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.976+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:entry[obj00000000000000000006]: success 2023-09-22T21:35:13.980 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.976+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:20:6 marker=00000000002.62.6 timestamp=2023-09-22T21:35:12.246181+0000 2023-09-22T21:35:13.981 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.976+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:entry[obj00000000000000000026]: success 2023-09-22T21:35:13.981 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.976+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:10:6 marker=00000000002.81.6 timestamp=2023-09-22T21:35:12.586212+0000 2023-09-22T21:35:13.981 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.976+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:entry[obj00000000000000000017]: success 2023-09-22T21:35:13.981 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.976+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:21:6 marker=00000000002.58.6 timestamp=2023-09-22T21:35:05.738308+0000 2023-09-22T21:35:13.981 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.976+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:entry[obj00000000000000000012]: success 2023-09-22T21:35:13.981 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.976+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:16:6 marker=00000000002.67.6 timestamp=2023-09-22T21:35:11.883712+0000 2023-09-22T21:35:13.981 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.976+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:entry[obj00000000000000000027]: success 2023-09-22T21:35:13.982 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.976+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:24:6 marker=00000000002.65.6 timestamp=2023-09-22T21:35:09.970418+0000 2023-09-22T21:35:13.984 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.980+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:entry[obj00000000000000000016]: success 2023-09-22T21:35:13.984 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.980+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:7:6 marker=00000000002.77.6 timestamp=2023-09-22T21:35:05.400454+0000 2023-09-22T21:35:13.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.984+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:entry[obj00000000000000000004]: success 2023-09-22T21:35:13.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.984+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:29:6 marker=00000000002.79.6 timestamp=2023-09-22T21:35:12.419375+0000 2023-09-22T21:35:13.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.984+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:entry[obj00000000000000000020]: success 2023-09-22T21:35:13.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.984+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:28:6 marker=00000000002.89.6 timestamp=2023-09-22T21:35:12.502061+0000 2023-09-22T21:35:13.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.984+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:entry[obj00000000000000000007]: success 2023-09-22T21:35:13.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.984+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:22:6 marker=00000000002.89.6 timestamp=2023-09-22T21:35:11.723056+0000 2023-09-22T21:35:13.987 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.984+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:entry[obj00000000000000000024]: success 2023-09-22T21:35:13.987 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.984+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:15:6 marker=00000000002.79.6 timestamp=2023-09-22T21:35:05.767101+0000 2023-09-22T21:35:13.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.992+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-09-22T21:35:13.993 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.992+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17/obj00000000000000000002 2023-09-22T21:35:13.993 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.992+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: entry->timestamp=2023-09-22T21:35:12.447117+0000 2023-09-22T21:35:13.993 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.992+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-09-22T21:35:13.993 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.992+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:entry[obj00000000000000000002]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000002[0] 2023-09-22T21:35:13.993 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.992+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: listing bilog for incremental sync; position=00000000002.61.6 2023-09-22T21:35:13.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.992+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:entry[obj00000000000000000002]: success 2023-09-22T21:35:13.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:13.992+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:17:6 marker=00000000002.61.6 timestamp=2023-09-22T21:35:12.447117+0000 2023-09-22T21:35:14.001 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.000+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: parsed entry: id=00000000002.60.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-09-22T21:35:14.001 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.000+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26/obj00000000000000000021 2023-09-22T21:35:14.001 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.000+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: entry->timestamp=2023-09-22T21:35:05.601809+0000 2023-09-22T21:35:14.001 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.000+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.60.6 op=0 op_state=1 2023-09-22T21:35:14.001 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.000+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:entry[obj00000000000000000021]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000021[0] 2023-09-22T21:35:14.001 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.000+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: listing bilog for incremental sync; position=00000000002.60.6 2023-09-22T21:35:14.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.000+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-09-22T21:35:14.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.000+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33/obj00000000000000000019 2023-09-22T21:35:14.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.000+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]: entry->timestamp=2023-09-22T21:35:05.682247+0000 2023-09-22T21:35:14.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.000+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.83.6 op=0 op_state=1 2023-09-22T21:35:14.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.000+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:entry[obj00000000000000000019]: : injecting data sync error on key=obj00000000000000000019 2023-09-22T21:35:14.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.000+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]: listing bilog for incremental sync; position=00000000002.83.6 2023-09-22T21:35:14.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.000+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:entry[obj00000000000000000019]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:35:14.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.000+0000 7fc2bb0c6a40 0 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:entry[obj00000000000000000019]: ERROR: failed to sync object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33/obj00000000000000000019 2023-09-22T21:35:14.003 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.000+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:entry[obj00000000000000000021]: success 2023-09-22T21:35:14.004 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.000+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:26:6 marker=00000000002.60.6 timestamp=2023-09-22T21:35:05.601809+0000 2023-09-22T21:35:14.005 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: parsed entry: id=00000000002.77.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-09-22T21:35:14.006 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25/obj00000000000000000010 2023-09-22T21:35:14.006 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: entry->timestamp=2023-09-22T21:35:05.514851+0000 2023-09-22T21:35:14.006 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.77.6 op=0 op_state=1 2023-09-22T21:35:14.006 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:entry[obj00000000000000000010]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000010[0] 2023-09-22T21:35:14.006 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: listing bilog for incremental sync; position=00000000002.77.6 2023-09-22T21:35:14.006 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: parsed entry: id=00000000002.71.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2023-09-22T21:35:14.006 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23/obj00000000000000000039 2023-09-22T21:35:14.006 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: entry->timestamp=2023-09-22T21:35:12.017023+0000 2023-09-22T21:35:14.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.71.6 op=0 op_state=1 2023-09-22T21:35:14.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]: parsed entry: id=00000000002.91.6 iter->object=obj00000000000000000053 iter->instance= name=obj00000000000000000053 instance= ns= 2023-09-22T21:35:14.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27/obj00000000000000000053 2023-09-22T21:35:14.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]: entry->timestamp=2023-09-22T21:35:12.185165+0000 2023-09-22T21:35:14.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:entry[obj00000000000000000053]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.91.6 op=0 op_state=1 2023-09-22T21:35:14.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: parsed entry: id=00000000001.83.6 iter->object=obj00000000000000000036 iter->instance= name=obj00000000000000000036 instance= ns= 2023-09-22T21:35:14.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30/obj00000000000000000036 2023-09-22T21:35:14.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: entry->timestamp=2023-09-22T21:35:05.428220+0000 2023-09-22T21:35:14.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:entry[obj00000000000000000036]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.83.6 op=0 op_state=1 2023-09-22T21:35:14.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: parsed entry: id=00000000001.60.6 iter->object=obj00000000000000000060 iter->instance= name=obj00000000000000000060 instance= ns= 2023-09-22T21:35:14.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0/obj00000000000000000060 2023-09-22T21:35:14.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: entry->timestamp=2023-09-22T21:35:05.376620+0000 2023-09-22T21:35:14.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000060]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.60.6 op=0 op_state=1 2023-09-22T21:35:14.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31]: parsed entry: id=00000000002.87.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-09-22T21:35:14.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31/obj00000000000000000003 2023-09-22T21:35:14.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31]: entry->timestamp=2023-09-22T21:35:09.783299+0000 2023-09-22T21:35:14.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.87.6 op=0 op_state=1 2023-09-22T21:35:14.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:entry[obj00000000000000000039]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000039[0] 2023-09-22T21:35:14.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: listing bilog for incremental sync; position=00000000002.71.6 2023-09-22T21:35:14.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:entry[obj00000000000000000053]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000053[0] 2023-09-22T21:35:14.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]: listing bilog for incremental sync; position=00000000002.91.6 2023-09-22T21:35:14.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:entry[obj00000000000000000036]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000036[0] 2023-09-22T21:35:14.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: listing bilog for incremental sync; position=00000000001.83.6 2023-09-22T21:35:14.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000060]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000060[0] 2023-09-22T21:35:14.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: listing bilog for incremental sync; position=00000000001.60.6 2023-09-22T21:35:14.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31]:entry[obj00000000000000000003]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000003[0] 2023-09-22T21:35:14.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.004+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31]: listing bilog for incremental sync; position=00000000002.87.6 2023-09-22T21:35:14.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.008+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:entry[obj00000000000000000010]: success 2023-09-22T21:35:14.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.008+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:25:6 marker=00000000002.77.6 timestamp=2023-09-22T21:35:05.514851+0000 2023-09-22T21:35:14.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.008+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31]:entry[obj00000000000000000003]: success 2023-09-22T21:35:14.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.008+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:31:6 marker=00000000002.87.6 timestamp=2023-09-22T21:35:09.783299+0000 2023-09-22T21:35:14.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.008+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:entry[obj00000000000000000039]: success 2023-09-22T21:35:14.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.008+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:23:6 marker=00000000002.71.6 timestamp=2023-09-22T21:35:12.017023+0000 2023-09-22T21:35:14.014 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.012+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-09-22T21:35:14.014 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.012+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32/obj00000000000000000014 2023-09-22T21:35:14.014 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.012+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32]: entry->timestamp=2023-09-22T21:35:09.667678+0000 2023-09-22T21:35:14.014 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.012+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.62.6 op=0 op_state=1 2023-09-22T21:35:14.014 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.012+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32]:entry[obj00000000000000000014]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000014[0] 2023-09-22T21:35:14.014 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.012+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32]: listing bilog for incremental sync; position=00000000002.62.6 2023-09-22T21:35:14.014 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.012+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34]: parsed entry: id=00000000001.53.6 iter->object=obj00000000000000000057 iter->instance= name=obj00000000000000000057 instance= ns= 2023-09-22T21:35:14.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.012+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34/obj00000000000000000057 2023-09-22T21:35:14.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.012+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34]: entry->timestamp=2023-09-22T21:35:05.658707+0000 2023-09-22T21:35:14.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.012+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34]:entry[obj00000000000000000057]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000001.53.6 op=0 op_state=1 2023-09-22T21:35:14.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.012+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34]:entry[obj00000000000000000057]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000057[0] 2023-09-22T21:35:14.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.012+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34]: listing bilog for incremental sync; position=00000000001.53.6 2023-09-22T21:35:14.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.012+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:entry[obj00000000000000000062]: success 2023-09-22T21:35:14.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.012+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:12:6 marker=00000000001.54.6 timestamp=2023-09-22T21:35:05.285275+0000 2023-09-22T21:35:14.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.016+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-09-22T21:35:14.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.016+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36/obj00000000000000000008 2023-09-22T21:35:14.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.016+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36]: entry->timestamp=2023-09-22T21:35:12.329960+0000 2023-09-22T21:35:14.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.016+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.64.6 op=0 op_state=1 2023-09-22T21:35:14.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.016+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36]:entry[obj00000000000000000008]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000008[0] 2023-09-22T21:35:14.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.016+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36]: listing bilog for incremental sync; position=00000000002.64.6 2023-09-22T21:35:14.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.016+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: a sync operation returned error 2023-09-22T21:35:14.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.016+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: backing out with sync_status=-5 2023-09-22T21:35:14.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.016+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:14.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.016+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:14.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.016+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32]:entry[obj00000000000000000014]: success 2023-09-22T21:35:14.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.016+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:32:6 marker=00000000002.62.6 timestamp=2023-09-22T21:35:09.667678+0000 2023-09-22T21:35:14.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.016+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:14.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.016+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: sync status for source bucket shard: 0 2023-09-22T21:35:14.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.016+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: listing bilog for incremental sync; position= 2023-09-22T21:35:14.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.016+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36]:entry[obj00000000000000000008]: success 2023-09-22T21:35:14.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.016+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:36:6 marker=00000000002.64.6 timestamp=2023-09-22T21:35:12.329960+0000 2023-09-22T21:35:14.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.020+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35]: parsed entry: id=00000000002.87.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-09-22T21:35:14.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.020+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35/obj00000000000000000018 2023-09-22T21:35:14.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.020+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35]: entry->timestamp=2023-09-22T21:35:12.273568+0000 2023-09-22T21:35:14.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.020+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.87.6 op=0 op_state=1 2023-09-22T21:35:14.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.020+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35]:entry[obj00000000000000000018]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000018[0] 2023-09-22T21:35:14.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.020+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35]: listing bilog for incremental sync; position=00000000002.87.6 2023-09-22T21:35:14.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.020+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35]:entry[obj00000000000000000018]: success 2023-09-22T21:35:14.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.020+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:35:6 marker=00000000002.87.6 timestamp=2023-09-22T21:35:12.273568+0000 2023-09-22T21:35:14.045 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.044+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]: a sync operation returned error 2023-09-22T21:35:14.046 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.044+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]: backing out with sync_status=-5 2023-09-22T21:35:14.046 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.044+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]: incremental sync on bucket failed, retcode=-5 2023-09-22T21:35:14.046 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.044+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]: ERROR: incremental sync failed. error: -5 2023-09-22T21:35:14.046 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.044+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:35:14.047 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.044+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]: sync status for source bucket shard: 0 2023-09-22T21:35:14.047 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.044+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]: listing bilog for incremental sync; position= 2023-09-22T21:35:14.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.056+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:entry[obj00000000000000000060]: success 2023-09-22T21:35:14.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.056+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:0:6 marker=00000000001.60.6 timestamp=2023-09-22T21:35:05.376620+0000 2023-09-22T21:35:14.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.056+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:entry[obj00000000000000000053]: success 2023-09-22T21:35:14.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.056+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:27:6 marker=00000000002.91.6 timestamp=2023-09-22T21:35:12.185165+0000 2023-09-22T21:35:14.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.056+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:entry[obj00000000000000000036]: success 2023-09-22T21:35:14.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.056+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:30:6 marker=00000000001.83.6 timestamp=2023-09-22T21:35:05.428220+0000 2023-09-22T21:35:14.065 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.064+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: parsed entry: id=00000000002.73.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2023-09-22T21:35:14.065 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.064+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19/obj00000000000000000045 2023-09-22T21:35:14.065 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.064+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: entry->timestamp=2023-09-22T21:35:12.356751+0000 2023-09-22T21:35:14.065 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.064+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.73.6 op=0 op_state=1 2023-09-22T21:35:14.065 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.064+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:entry[obj00000000000000000045]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000045[0] 2023-09-22T21:35:14.065 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.064+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: listing bilog for incremental sync; position=00000000002.73.6 2023-09-22T21:35:14.065 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.064+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34]:entry[obj00000000000000000057]: success 2023-09-22T21:35:14.065 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.064+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:34:6 marker=00000000001.53.6 timestamp=2023-09-22T21:35:05.658707+0000 2023-09-22T21:35:14.067 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.064+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:entry[obj00000000000000000045]: success 2023-09-22T21:35:14.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.064+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:19:6 marker=00000000002.73.6 timestamp=2023-09-22T21:35:12.356751+0000 2023-09-22T21:35:14.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.088+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-09-22T21:35:14.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.088+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]: syncing object: ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33/obj00000000000000000019 2023-09-22T21:35:14.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.088+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]: entry->timestamp=2023-09-22T21:35:05.682247+0000 2023-09-22T21:35:14.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.088+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=00000000002.83.6 op=0 op_state=1 2023-09-22T21:35:14.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.088+0000 7fc2bb0c6a40 5 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:entry[obj00000000000000000019]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-58[f132bdb9-20a7-4073-a932-8f99da20691e.5729.23])/obj00000000000000000019[0] 2023-09-22T21:35:14.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.088+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]: listing bilog for incremental sync; position=00000000002.83.6 2023-09-22T21:35:14.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.088+0000 7fc2bb0c6a40 10 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:entry[obj00000000000000000019]: success 2023-09-22T21:35:14.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:14.088+0000 7fc2bb0c6a40 20 RGW-SYNC:bucket[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23<-ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]:inc_sync[ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33]: updating marker marker_oid=bucket.sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23:33:6 marker=00000000002.83.6 timestamp=2023-09-22T21:35:05.682247+0000 2023-09-22T21:35:14.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:35:14.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:35:14.184 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-58 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:35:14.268 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:35:14.268 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:35:14.328 INFO:teuthology.orchestra.run.smithi160.stdout:[{"name":"obj00000000000000000000","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:05.571412Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.5563480551112565889","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000001","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:12.531706Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.12699570734064802799","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000002","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:12.447117Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.12406845778381116108","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000003","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:09.783299Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.2587545821110448903","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000004","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:12.419375Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.11214835605282013300","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000005","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:11.989247Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.15655607115713294199","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000006","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:35:12.246181Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.1847349202508375826","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000007","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:11.723056Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.12299886254112225014","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000008","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:12.329960Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.14317116598136226511","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000009","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:05.298620Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.7685720368734008275","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000010","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:05.514851Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.7763325745816551828","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000011","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:12.074000Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.15286449479257003154","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000012","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:11.883712Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.8994616582001552173","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000013","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:02.410768Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.3657328813321598722","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000014","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:09.667678Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.13421196271263295070","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000015","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:05.483352Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.7096521336043049038","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000016","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:35:05.400454Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.9879504911281689672","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000017","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:05.738308Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.14014478320795789862","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000018","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:12.273568Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.15430178781164487042","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000019","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:35:05.682247Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.7110699767547262585","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000020","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:12.502061Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.15296529500636434043","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000021","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:05.601809Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.14640623745857580713","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000022","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:03.980645Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.18187902051608065649","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000023","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:35:03.585390Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.3184780182016874244","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000024","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:05.767101Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.10103556440339985439","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000025","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:02.338233Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.14094354978110095136","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000026","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:35:12.586212Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.8545537974559091393","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000027","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:09.970418Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.15335453562682075506","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000028","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:34:59.658464Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.35825145876762195","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000029","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:03.881350Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.5923367009350113001","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000030","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:11.934521Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.10501562897798512242","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000031","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:03.155230Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.3412183510454926812","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000032","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:01.575757Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.17097806194063300407","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000033","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:01.495725Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.7394209850326565744","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000036","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:05.428220Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.13117169413687363232","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000037","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:02.991134Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.4728263059050698148","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000039","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:12.017023Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.1974651431629676778","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000040","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:12.157791Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.14031023177539335295","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000042","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:03.074635Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.10802689787942531638","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000045","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:12.356751Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.2051802232258813844","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000049","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:02.887970Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.14077682690837185942","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000050","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:04.354482Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.2625607498514109214","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000053","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:12.185165Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.17945344669324948498","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000055","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:01.659463Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.13523305880508845262","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000057","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:05.658707Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.999695771547478645","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000058","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:01.736432Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.3027768282714213889","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000059","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:02.568352Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.7406099452225332460","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000060","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:05.376620Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.2536337229685216765","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000061","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:04.223951Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.202010301672527395","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000062","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:05.285275Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.3839326989918983332","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000065","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:12.101872Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.14557647835216825050","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000076","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:34:59.536192Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.8616241530000341244","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000133","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:03.904168Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.8346239682488828068","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000271","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:03.667222Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.14530710807165502913","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000283","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:04.277065Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.5659059321567595388","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000317","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:03.957705Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.2478710485260734203","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000423","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:35:04.116870Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.14065508375475844133","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000467","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:03.651153Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.15976890973395526677","flags":0,"pending_map":[],"versioned_epoch":0}] 2023-09-22T21:35:14.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:35:14.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:35:14.330 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-58 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:35:14.415 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:35:14.415 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:35:14.471 INFO:teuthology.orchestra.run.smithi160.stdout:[{"name":"obj00000000000000000000","instance":"","ver":{"pool":13,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:05.571412Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_yzxWEWsNffVoXFmBbmkh5nC8d-Ph0te","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000001","instance":"","ver":{"pool":13,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:12.531706Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_kgRGafo6kn9-CgM1Jen6d0IHXmUCGNd","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000002","instance":"","ver":{"pool":13,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:12.447117Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_GpbcLTiwqDVtONEeVIvhPR1V7M1JhNR","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000003","instance":"","ver":{"pool":13,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:09.783299Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_4goT7uvNGazbMBT9X6IO0LpgHhIw538","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000004","instance":"","ver":{"pool":13,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:35:12.419375Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_9Dj-l-9D4ek-rM7nKllGp9jh9H5jnnm","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000005","instance":"","ver":{"pool":13,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:11.989247Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_LuKWYGANFsBn0NkOVjZRooRPNb8U5OJ","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000006","instance":"","ver":{"pool":13,"epoch":2},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:35:12.246181Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_tP6DGUbN6NAQ8ua0Nfk4rj7pJjX9tAq","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000007","instance":"","ver":{"pool":13,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:11.723056Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_trpolCRYWKayrIq_P3FYbP_54nSBx_H","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000008","instance":"","ver":{"pool":13,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:12.329960Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_jtHq-UvM15lkbL3nirO8LFJmp2G3LnB","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000009","instance":"","ver":{"pool":13,"epoch":2},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:05.298620Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_Li-QpZg1L5JoAk42faZRXE44lrZ0MEk","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000010","instance":"","ver":{"pool":13,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:05.514851Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_TQoSmP5c--buvEdMOOpFi1WY4Nq-1yk","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000011","instance":"","ver":{"pool":13,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:12.074000Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_NEYCOQuUfKbDtcd37opY6f80DRCMfFi","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000012","instance":"","ver":{"pool":13,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:11.883712Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_7oPEhcJOz4Y8pwxjr0YbX064H4Xlkm3","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000013","instance":"","ver":{"pool":13,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:02.410768Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_zCnndaTjWZrHv5N2feAXYhbkZiAUZHQ","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000014","instance":"","ver":{"pool":13,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:09.667678Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_c1dvO4_n9K572Dl8Iefi1rBzwXsgV5v","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000015","instance":"","ver":{"pool":13,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:05.483352Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_gfr5MBvvaceiCUD-VxPGEldu832OmTj","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000016","instance":"","ver":{"pool":13,"epoch":2},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:05.400454Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_oe7GTmbhjok6ToRK6I1P53ZELPYcnh-","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000017","instance":"","ver":{"pool":13,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:05.738308Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_Fgm37fx6-APOOZFHF9UJ-a-PvUiWHW6","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000018","instance":"","ver":{"pool":13,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:12.273568Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_TtNlWLi1nWbIz2cm4N95PlBKuwTXhqC","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000019","instance":"","ver":{"pool":13,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:05.682247Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_DYIIcTqAbUbzzoFsU-IeO5f3JPbk6kE","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000020","instance":"","ver":{"pool":13,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:35:12.502061Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_CjTfHxyZPoCZzs3NRf943cr3Phr0rfs","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000021","instance":"","ver":{"pool":13,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:05.601809Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_J0Tr6XjMtMRVwhB-u_jL96t7AoSmvfk","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000022","instance":"","ver":{"pool":13,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:35:03.980645Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_lkxtb7HMUsObPylO3rqB3-6ruCwg1ut","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000023","instance":"","ver":{"pool":13,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:35:03.585390Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_uqGQ3RK4pT3ry7iYIysgp61xA_8gtNo","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000024","instance":"","ver":{"pool":13,"epoch":14},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:05.767101Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_AwvG4xyWPT53ymD8iXZN6OIhgmh3h-Q","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000025","instance":"","ver":{"pool":13,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:02.338233Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_qj0VI8iQ0L-2-NDuDfnkc8n_cKjczHm","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000026","instance":"","ver":{"pool":13,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:12.586212Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_gFhF2qveN65z_sjcgxC9WdOpMx2YPfM","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000027","instance":"","ver":{"pool":13,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:09.970418Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_5gKWlH2PueLJJCloSjGJGPE6atml0-U","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000028","instance":"","ver":{"pool":13,"epoch":2},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:34:59.658464Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_neCLOjIRtTbW1tUnmbIAiUB9yrlqNmw","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000029","instance":"","ver":{"pool":13,"epoch":1},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:03.881350Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_W9ruh2DjV0TKwFvkDE0vAgopAMNX6CJ","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000030","instance":"","ver":{"pool":13,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:11.934521Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_fLHS1H5pnRxYr0EY3b6yfUiHDRv8ezA","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000031","instance":"","ver":{"pool":13,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:03.155230Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_Ix-10NYSE8FbXHfGKdXhRzvpyykVvo_","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000032","instance":"","ver":{"pool":13,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:01.575757Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_cq535vmin0qjX8IzhyFkJps8KbrEzj9","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000033","instance":"","ver":{"pool":13,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:01.495725Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_dKWtX23CU-CLjZ-vgJTO24SCTSOCZX9","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000036","instance":"","ver":{"pool":13,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:05.428220Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_CCl_1-nYbZoqWKifbJucdxyR4BuwWoL","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000037","instance":"","ver":{"pool":13,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:02.991134Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_4piAo0CD1GlUJI1PffnoNHV4AnfjBc7","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000039","instance":"","ver":{"pool":13,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:35:12.017023Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_pNKK2SuD_T3dtW3iEMpfbbPJNV26Vrn","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000040","instance":"","ver":{"pool":13,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:35:12.157791Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_yXuofMDL0wvo-sEyrB3oPGgIOnd3KUv","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000042","instance":"","ver":{"pool":13,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:03.074635Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_0oQE-aspxGCuY_bW-kpS2IlGhRCVuSx","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000045","instance":"","ver":{"pool":13,"epoch":1},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:12.356751Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_Xt7d3_xZnbiPW2YD6d5h63WFI0rNThx","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000049","instance":"","ver":{"pool":13,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:02.887970Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_-Mg1Y_ox2aW5uy-SbCMOixuXXMRlrTF","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000050","instance":"","ver":{"pool":13,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:04.354482Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_Q51ZVrDZoPGh7HsBT1W1iQ8ZniOgmuh","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000053","instance":"","ver":{"pool":13,"epoch":8},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:35:12.185165Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_bD6YUYlVUxnpCpTctMKqU3SsXtBSNbc","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000055","instance":"","ver":{"pool":13,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:01.659463Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_egyh-coI2wQcd0voZZL-f77P5apFh9I","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000057","instance":"","ver":{"pool":13,"epoch":15},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:35:05.658707Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_HM1sXRjHstxB0E7F6kPj4pL429g43iV","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000058","instance":"","ver":{"pool":13,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:01.736432Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_XJEHpMUK23KL9FSE1MlpOs7hVZYaE7W","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000059","instance":"","ver":{"pool":13,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:02.568352Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_PbI7SMp4xq3yQ_HrYNgfSc02KkNd4Pg","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000060","instance":"","ver":{"pool":13,"epoch":36},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:05.376620Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_TmdzJTqwmyob0lkWb-Id4_X2dEXVHf4","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000061","instance":"","ver":{"pool":13,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:04.223951Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_191Ox5Oun6tsEGZs9VBM6YXgVP6VaNL","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000062","instance":"","ver":{"pool":13,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:35:05.285275Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_dAiggOU6Poz4y584HIvi4Zxiwg2H5DJ","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000065","instance":"","ver":{"pool":13,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:12.101872Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_dHI9-H2ZYMKCbQo371vBP8GvaKu9RMt","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000076","instance":"","ver":{"pool":13,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:34:59.536192Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_je-TlFHFLrPc5MfbrzDLruwkh6qQugv","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000133","instance":"","ver":{"pool":13,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:03.904168Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_UJHuwF8oxrEQ92akwYQHYMn6f4RbUZe","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000271","instance":"","ver":{"pool":13,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:03.667222Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"__a1YYctWHFXnNBxr8DsbCIBJsN5ulHj","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000283","instance":"","ver":{"pool":13,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:04.277065Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_SbmaqyeS76U8qsTr6VdOOkGPUpYpTBR","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000317","instance":"","ver":{"pool":13,"epoch":11},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:03.957705Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_TACLV3uCNVEUOiF2bl4EliJv7WycOnt","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000423","instance":"","ver":{"pool":13,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:35:04.116870Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_1fNFHrc0L61mnLJsNFYglkjsorwKzx_","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000467","instance":"","ver":{"pool":13,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:35:03.651153Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_S1kZ7pZ4Y0KuKB19X9xvt8iJcAaSTs-","flags":0,"pending_map":[],"versioned_epoch":0}] 2023-09-22T21:35:14.472 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-22T21:35:14.472 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-22T21:35:14.513 INFO:tasks.rgw.c1.client.1:Started 2023-09-22T21:35:14.513 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T21:35:14.522 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T21:35:14.523 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:35:14.523 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:35:14.523 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T21:35:14.523 INFO:teuthology.orchestra.run.smithi160.stderr:curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T21:35:15.523 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T21:35:15.535 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:35:15.535 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:35:15.536 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 174k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-22T21:35:15.536 INFO:teuthology.orchestra.run.smithi160.stdout:anonymous 2023-09-22T21:35:15.537 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-58 2023-09-22T21:35:15.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:35:15.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:35:15.537 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:35:15.630 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:35:15.630 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:35:15.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:15.680+0000 7fb7df774a40 1 bucket sync caught up with source: 2023-09-22T21:35:15.683 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.60.6, 00000000002.85.6, 00000000002.85.6, 00000000002.64.6, , 00000000001.54.6, 00000000002.84.6, 00000000002.77.6, , 00000000002.48.6, 00000000002.81.6, 00000000002.52.6, 00000000001.54.6, 00000000002.81.6, 00000000002.55.6, 00000000002.79.6, 00000000002.67.6, 00000000002.61.6, , 00000000002.73.6, 00000000002.62.6, 00000000002.58.6, 00000000002.89.6, 00000000002.71.6, 00000000002.65.6, 00000000002.77.6, 00000000002.60.6, 00000000002.91.6, 00000000002.89.6, 00000000002.79.6, 00000000001.83.6, 00000000002.87.6, 00000000002.62.6, 00000000002.83.6, 00000000001.53.6, 00000000002.87.6, 00000000002.64.6] 2023-09-22T21:35:15.683 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.60.6, 00000000002.85.6, 00000000002.85.6, 00000000002.64.6, , 00000000001.54.6, 00000000002.84.6, 00000000002.77.6, , 00000000002.48.6, 00000000002.81.6, 00000000002.52.6, 00000000001.54.6, 00000000002.81.6, 00000000002.55.6, 00000000002.79.6, 00000000002.67.6, 00000000002.61.6, , 00000000002.73.6, 00000000002.62.6, 00000000002.58.6, 00000000002.89.6, 00000000002.71.6, 00000000002.65.6, 00000000002.77.6, 00000000002.60.6, 00000000002.91.6, 00000000002.89.6, 00000000002.79.6, 00000000001.83.6, 00000000002.87.6, 00000000002.62.6, 00000000002.83.6, 00000000001.53.6, 00000000002.87.6, 00000000002.64.6] 2023-09-22T21:35:15.683 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:15.680+0000 7fb7df774a40 0 bucket checkpoint complete 2023-09-22T21:35:15.699 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-58 2023-09-22T21:35:15.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:35:15.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:35:15.700 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:35:15.741 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:35:15.742 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:35:15.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:15.780+0000 7fdc3c775a40 1 waiting for incremental sync to catch up: 2023-09-22T21:35:15.782 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000005.209.6, 00000000001.159.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000001.138.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000001.182.6, 00000000004.351.6] 2023-09-22T21:35:15.782 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000005.209.6, 00000000006.169.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000008.152.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000006.192.6, 00000000004.351.6] 2023-09-22T21:35:20.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:20.784+0000 7fdc3c775a40 1 waiting for incremental sync to catch up: 2023-09-22T21:35:20.785 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000005.209.6, 00000000001.159.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000001.138.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000001.182.6, 00000000004.351.6] 2023-09-22T21:35:20.785 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000005.209.6, 00000000006.169.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000008.152.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000006.192.6, 00000000004.351.6] 2023-09-22T21:35:25.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:25.784+0000 7fdc3c775a40 1 waiting for incremental sync to catch up: 2023-09-22T21:35:25.787 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000005.209.6, 00000000001.159.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000001.138.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000001.182.6, 00000000004.351.6] 2023-09-22T21:35:25.787 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000005.209.6, 00000000006.169.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000008.152.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000006.192.6, 00000000004.351.6] 2023-09-22T21:35:30.789 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:30.788+0000 7fdc3c775a40 1 waiting for incremental sync to catch up: 2023-09-22T21:35:30.789 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000005.209.6, 00000000001.159.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000001.138.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000001.182.6, 00000000004.351.6] 2023-09-22T21:35:30.789 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000005.209.6, 00000000006.169.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000008.152.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000006.192.6, 00000000004.351.6] 2023-09-22T21:35:35.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:35.787+0000 7fdc3c775a40 1 waiting for incremental sync to catch up: 2023-09-22T21:35:35.792 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000005.209.6, 00000000001.159.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000001.138.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000001.182.6, 00000000004.351.6] 2023-09-22T21:35:35.792 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000005.209.6, 00000000006.169.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000008.152.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000006.192.6, 00000000004.351.6] 2023-09-22T21:35:40.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:40.791+0000 7fdc3c775a40 1 waiting for incremental sync to catch up: 2023-09-22T21:35:40.794 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000005.209.6, 00000000001.159.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000001.138.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000001.182.6, 00000000004.351.6] 2023-09-22T21:35:40.794 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000005.209.6, 00000000006.169.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000008.152.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000006.192.6, 00000000004.351.6] 2023-09-22T21:35:45.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:45.795+0000 7fdc3c775a40 1 waiting for incremental sync to catch up: 2023-09-22T21:35:45.797 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000005.209.6, 00000000001.159.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000001.138.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000001.182.6, 00000000004.351.6] 2023-09-22T21:35:45.797 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000005.209.6, 00000000006.169.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000008.152.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000006.192.6, 00000000004.351.6] 2023-09-22T21:35:50.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:50.795+0000 7fdc3c775a40 1 waiting for incremental sync to catch up: 2023-09-22T21:35:50.799 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000005.209.6, 00000000001.159.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000001.138.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000001.182.6, 00000000004.351.6] 2023-09-22T21:35:50.799 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000005.209.6, 00000000006.169.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000008.152.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000006.192.6, 00000000004.351.6] 2023-09-22T21:35:55.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:35:55.799+0000 7fdc3c775a40 1 waiting for incremental sync to catch up: 2023-09-22T21:35:55.801 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000005.209.6, 00000000001.159.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000001.138.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000001.182.6, 00000000004.351.6] 2023-09-22T21:35:55.801 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000005.209.6, 00000000006.169.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000008.152.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000006.192.6, 00000000004.351.6] 2023-09-22T21:36:00.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:36:00.799+0000 7fdc3c775a40 1 waiting for incremental sync to catch up: 2023-09-22T21:36:00.803 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000005.209.6, 00000000001.159.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000001.138.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000001.182.6, 00000000004.351.6] 2023-09-22T21:36:00.803 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000005.209.6, 00000000006.169.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000008.152.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000006.192.6, 00000000004.351.6] 2023-09-22T21:36:05.805 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:36:05.803+0000 7fdc3c775a40 1 waiting for incremental sync to catch up: 2023-09-22T21:36:05.806 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000005.209.6, 00000000001.159.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000001.138.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000001.182.6, 00000000004.351.6] 2023-09-22T21:36:05.806 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000005.209.6, 00000000006.169.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000008.152.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000006.192.6, 00000000004.351.6] 2023-09-22T21:36:10.810 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:36:10.807+0000 7fdc3c775a40 1 bucket sync caught up with source: 2023-09-22T21:36:10.810 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000005.209.6, 00000000006.169.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000008.152.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000006.192.6, 00000000004.351.6] 2023-09-22T21:36:10.810 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000005.209.6, 00000000006.169.6, 00000000005.127.6, 00000000009.245.6, 00000000005.254.6, 00000000008.152.6, 00000000008.245.6, 00000000006.249.6, 00000000006.213.6, 00000000006.192.6, 00000000004.351.6] 2023-09-22T21:36:10.810 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:36:10.807+0000 7fdc3c775a40 0 bucket checkpoint complete 2023-09-22T21:36:10.841 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-58 zones={a2, a1} 2023-09-22T21:36:10.841 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:10.841 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:10.842 DEBUG:boto:Method: HEAD 2023-09-22T21:36:10.842 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:36:10.842 DEBUG:boto:Data: 2023-09-22T21:36:10.842 DEBUG:boto:Headers: {} 2023-09-22T21:36:10.842 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:10.842 DEBUG:boto:Port: 8001 2023-09-22T21:36:10.842 DEBUG:boto:Params: {} 2023-09-22T21:36:10.845 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:36:10.845 DEBUG:boto:Token: None 2023-09-22T21:36:10.845 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:10 GMT /ffzdqu-58/ 2023-09-22T21:36:10.845 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KSj3W2A4OJPr7hytjFvZWVpsDVo= 2023-09-22T21:36:10.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KSj3W2A4OJPr7hytjFvZWVpsDVo=', 'Content-Length': '0'} 2023-09-22T21:36:10.854 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031646bc32a2ca553-00650e08ca-1b9c-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:36:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:10.854 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:10.854 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:10.854 DEBUG:boto:Method: HEAD 2023-09-22T21:36:10.854 DEBUG:boto:Path: /ffzdqu-58/ 2023-09-22T21:36:10.854 DEBUG:boto:Data: 2023-09-22T21:36:10.854 DEBUG:boto:Headers: {} 2023-09-22T21:36:10.854 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:10.855 DEBUG:boto:Port: 8000 2023-09-22T21:36:10.855 DEBUG:boto:Params: {} 2023-09-22T21:36:10.855 DEBUG:boto:Token: None 2023-09-22T21:36:10.855 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:10 GMT /ffzdqu-58/ 2023-09-22T21:36:10.855 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KSj3W2A4OJPr7hytjFvZWVpsDVo= 2023-09-22T21:36:10.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KSj3W2A4OJPr7hytjFvZWVpsDVo=', 'Content-Length': '0'} 2023-09-22T21:36:10.861 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e564bd5adc13119-00650e08ca-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:36:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:10.861 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:36:10.861 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:10.861 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:10.862 DEBUG:boto:path=/ffzdqu-58/?versions&max-keys=999 2023-09-22T21:36:10.862 DEBUG:boto:auth_path=/ffzdqu-58/?versions&max-keys=999 2023-09-22T21:36:10.862 DEBUG:boto:Method: GET 2023-09-22T21:36:10.862 DEBUG:boto:Path: /ffzdqu-58/?versions&max-keys=999 2023-09-22T21:36:10.862 DEBUG:boto:Data: 2023-09-22T21:36:10.862 DEBUG:boto:Headers: {} 2023-09-22T21:36:10.862 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:10.862 DEBUG:boto:Port: 8001 2023-09-22T21:36:10.862 DEBUG:boto:Params: {} 2023-09-22T21:36:10.863 DEBUG:boto:Token: None 2023-09-22T21:36:10.863 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:10 GMT /ffzdqu-58/?versions 2023-09-22T21:36:10.863 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HAtX1MwI0d3VpVzEyH5qBoOuQos= 2023-09-22T21:36:10.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HAtX1MwI0d3VpVzEyH5qBoOuQos=', 'Content-Length': '0'} 2023-09-22T21:36:10.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e153c83901d77da-00650e08ca-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:10.868 DEBUG:boto:b'ffzdqu-58999falseobj00000000000000000000nulltrue2023-09-22T21:35:05.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-09-22T21:35:12.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-09-22T21:35:12.447Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-09-22T21:35:09.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-09-22T21:35:12.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-09-22T21:35:11.989Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-09-22T21:35:12.246Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-09-22T21:35:11.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-09-22T21:35:12.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-09-22T21:35:05.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-09-22T21:35:05.514Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-09-22T21:35:12.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-09-22T21:35:11.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-09-22T21:35:02.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-09-22T21:35:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-09-22T21:35:05.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-09-22T21:35:05.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-09-22T21:35:05.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-09-22T21:35:12.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-09-22T21:35:05.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-09-22T21:35:12.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-09-22T21:35:05.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-09-22T21:35:03.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-09-22T21:35:03.585Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-09-22T21:35:05.767Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-09-22T21:35:02.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-09-22T21:35:12.586Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-09-22T21:35:09.970Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-09-22T21:34:59.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-09-22T21:35:03.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-09-22T21:35:11.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-09-22T21:35:03.155Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-09-22T21:35:01.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-09-22T21:35:01.495Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-09-22T21:35:05.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-09-22T21:35:02.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-09-22T21:35:12.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-09-22T21:35:12.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-09-22T21:35:03.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-09-22T21:35:12.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-09-22T21:35:02.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-09-22T21:35:04.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-09-22T21:35:12.185Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-09-22T21:35:01.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-09-22T21:35:05.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-09-22T21:35:01.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-09-22T21:35:02.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-09-22T21:35:05.376Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-09-22T21:35:04.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-09-22T21:35:05.285Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-09-22T21:35:12.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-09-22T21:34:59.536Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-09-22T21:35:03.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-09-22T21:35:03.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-09-22T21:35:04.277Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-09-22T21:35:03.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-09-22T21:35:04.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-09-22T21:35:03.651Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:10.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-09-22T21:36:10.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-09-22T21:36:10.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-09-22T21:36:10.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-09-22T21:36:10.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-09-22T21:36:10.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-09-22T21:36:10.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-09-22T21:36:10.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2023-09-22T21:36:10.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2023-09-22T21:36:10.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2023-09-22T21:36:10.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-09-22T21:36:10.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-09-22T21:36:10.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-09-22T21:36:10.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-09-22T21:36:10.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2023-09-22T21:36:10.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2023-09-22T21:36:10.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2023-09-22T21:36:10.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2023-09-22T21:36:10.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2023-09-22T21:36:10.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2023-09-22T21:36:10.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2023-09-22T21:36:10.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2023-09-22T21:36:10.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2023-09-22T21:36:10.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2023-09-22T21:36:10.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2023-09-22T21:36:10.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2023-09-22T21:36:10.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2023-09-22T21:36:10.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2023-09-22T21:36:10.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2023-09-22T21:36:10.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2023-09-22T21:36:10.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2023-09-22T21:36:10.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2023-09-22T21:36:10.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2023-09-22T21:36:10.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2023-09-22T21:36:10.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2023-09-22T21:36:10.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2023-09-22T21:36:10.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2023-09-22T21:36:10.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2023-09-22T21:36:10.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2023-09-22T21:36:10.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2023-09-22T21:36:10.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2023-09-22T21:36:10.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2023-09-22T21:36:10.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2023-09-22T21:36:10.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2023-09-22T21:36:10.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2023-09-22T21:36:10.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2023-09-22T21:36:10.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2023-09-22T21:36:10.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2023-09-22T21:36:10.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2023-09-22T21:36:10.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2023-09-22T21:36:10.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2023-09-22T21:36:10.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2023-09-22T21:36:10.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2023-09-22T21:36:10.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2023-09-22T21:36:10.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2023-09-22T21:36:10.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2023-09-22T21:36:10.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2023-09-22T21:36:10.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2023-09-22T21:36:10.878 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:36:10.878 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:10.878 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:10.878 DEBUG:boto:path=/ffzdqu-58/?versions&max-keys=999 2023-09-22T21:36:10.878 DEBUG:boto:auth_path=/ffzdqu-58/?versions&max-keys=999 2023-09-22T21:36:10.878 DEBUG:boto:Method: GET 2023-09-22T21:36:10.878 DEBUG:boto:Path: /ffzdqu-58/?versions&max-keys=999 2023-09-22T21:36:10.878 DEBUG:boto:Data: 2023-09-22T21:36:10.878 DEBUG:boto:Headers: {} 2023-09-22T21:36:10.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:10.879 DEBUG:boto:Port: 8000 2023-09-22T21:36:10.879 DEBUG:boto:Params: {} 2023-09-22T21:36:10.879 DEBUG:boto:Token: None 2023-09-22T21:36:10.879 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:10 GMT /ffzdqu-58/?versions 2023-09-22T21:36:10.879 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HAtX1MwI0d3VpVzEyH5qBoOuQos= 2023-09-22T21:36:10.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HAtX1MwI0d3VpVzEyH5qBoOuQos=', 'Content-Length': '0'} 2023-09-22T21:36:10.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fedd1c40d1634559-00650e08ca-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:10.888 DEBUG:boto:b'ffzdqu-58999falseobj00000000000000000000nulltrue2023-09-22T21:35:05.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-09-22T21:35:12.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-09-22T21:35:12.447Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-09-22T21:35:09.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-09-22T21:35:12.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-09-22T21:35:11.989Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-09-22T21:35:12.246Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-09-22T21:35:11.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-09-22T21:35:12.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-09-22T21:35:05.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-09-22T21:35:05.514Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-09-22T21:35:12.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-09-22T21:35:11.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-09-22T21:35:02.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-09-22T21:35:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-09-22T21:35:05.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-09-22T21:35:05.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-09-22T21:35:05.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-09-22T21:35:12.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-09-22T21:35:05.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-09-22T21:35:12.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-09-22T21:35:05.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-09-22T21:35:03.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-09-22T21:35:03.585Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-09-22T21:35:05.767Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-09-22T21:35:02.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-09-22T21:35:12.586Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-09-22T21:35:09.970Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-09-22T21:34:59.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-09-22T21:35:03.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-09-22T21:35:11.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-09-22T21:35:03.155Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-09-22T21:35:01.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-09-22T21:35:01.495Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-09-22T21:35:05.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-09-22T21:35:02.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-09-22T21:35:12.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-09-22T21:35:12.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-09-22T21:35:03.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-09-22T21:35:12.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-09-22T21:35:02.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-09-22T21:35:04.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-09-22T21:35:12.185Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-09-22T21:35:01.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-09-22T21:35:05.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-09-22T21:35:01.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-09-22T21:35:02.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-09-22T21:35:05.376Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-09-22T21:35:04.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-09-22T21:35:05.285Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-09-22T21:35:12.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-09-22T21:34:59.536Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-09-22T21:35:03.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-09-22T21:35:03.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-09-22T21:35:04.277Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-09-22T21:35:03.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-09-22T21:35:04.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-09-22T21:35:03.651Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:10.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-09-22T21:36:10.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-09-22T21:36:10.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-09-22T21:36:10.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-09-22T21:36:10.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-09-22T21:36:10.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-09-22T21:36:10.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-09-22T21:36:10.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2023-09-22T21:36:10.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2023-09-22T21:36:10.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2023-09-22T21:36:10.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-09-22T21:36:10.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-09-22T21:36:10.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-09-22T21:36:10.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-09-22T21:36:10.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2023-09-22T21:36:10.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2023-09-22T21:36:10.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2023-09-22T21:36:10.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2023-09-22T21:36:10.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2023-09-22T21:36:10.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2023-09-22T21:36:10.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2023-09-22T21:36:10.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2023-09-22T21:36:10.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2023-09-22T21:36:10.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2023-09-22T21:36:10.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2023-09-22T21:36:10.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2023-09-22T21:36:10.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2023-09-22T21:36:10.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2023-09-22T21:36:10.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2023-09-22T21:36:10.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2023-09-22T21:36:10.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2023-09-22T21:36:10.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2023-09-22T21:36:10.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2023-09-22T21:36:10.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2023-09-22T21:36:10.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2023-09-22T21:36:10.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2023-09-22T21:36:10.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2023-09-22T21:36:10.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2023-09-22T21:36:10.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2023-09-22T21:36:10.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2023-09-22T21:36:10.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2023-09-22T21:36:10.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2023-09-22T21:36:10.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2023-09-22T21:36:10.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2023-09-22T21:36:10.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2023-09-22T21:36:10.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2023-09-22T21:36:10.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2023-09-22T21:36:10.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2023-09-22T21:36:10.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2023-09-22T21:36:10.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2023-09-22T21:36:10.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2023-09-22T21:36:10.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2023-09-22T21:36:10.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2023-09-22T21:36:10.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2023-09-22T21:36:10.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2023-09-22T21:36:10.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2023-09-22T21:36:10.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2023-09-22T21:36:10.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2023-09-22T21:36:10.896 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:10.896 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:10.896 DEBUG:boto:path=/ffzdqu-58/?versions&max-keys=999 2023-09-22T21:36:10.896 DEBUG:boto:auth_path=/ffzdqu-58/?versions&max-keys=999 2023-09-22T21:36:10.896 DEBUG:boto:Method: GET 2023-09-22T21:36:10.897 DEBUG:boto:Path: /ffzdqu-58/?versions&max-keys=999 2023-09-22T21:36:10.897 DEBUG:boto:Data: 2023-09-22T21:36:10.897 DEBUG:boto:Headers: {} 2023-09-22T21:36:10.897 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:10.897 DEBUG:boto:Port: 8001 2023-09-22T21:36:10.897 DEBUG:boto:Params: {} 2023-09-22T21:36:10.897 DEBUG:boto:Token: None 2023-09-22T21:36:10.897 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:10 GMT /ffzdqu-58/?versions 2023-09-22T21:36:10.897 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HAtX1MwI0d3VpVzEyH5qBoOuQos= 2023-09-22T21:36:10.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HAtX1MwI0d3VpVzEyH5qBoOuQos=', 'Content-Length': '0'} 2023-09-22T21:36:10.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042682a56ffc80cdc-00650e08ca-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:10.903 DEBUG:boto:b'ffzdqu-58999falseobj00000000000000000000nulltrue2023-09-22T21:35:05.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-09-22T21:35:12.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-09-22T21:35:12.447Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-09-22T21:35:09.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-09-22T21:35:12.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-09-22T21:35:11.989Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-09-22T21:35:12.246Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-09-22T21:35:11.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-09-22T21:35:12.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-09-22T21:35:05.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-09-22T21:35:05.514Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-09-22T21:35:12.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-09-22T21:35:11.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-09-22T21:35:02.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-09-22T21:35:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-09-22T21:35:05.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-09-22T21:35:05.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-09-22T21:35:05.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-09-22T21:35:12.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-09-22T21:35:05.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-09-22T21:35:12.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-09-22T21:35:05.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-09-22T21:35:03.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-09-22T21:35:03.585Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-09-22T21:35:05.767Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-09-22T21:35:02.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-09-22T21:35:12.586Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-09-22T21:35:09.970Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-09-22T21:34:59.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-09-22T21:35:03.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-09-22T21:35:11.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-09-22T21:35:03.155Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-09-22T21:35:01.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-09-22T21:35:01.495Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-09-22T21:35:05.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-09-22T21:35:02.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-09-22T21:35:12.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-09-22T21:35:12.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-09-22T21:35:03.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-09-22T21:35:12.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-09-22T21:35:02.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-09-22T21:35:04.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-09-22T21:35:12.185Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-09-22T21:35:01.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-09-22T21:35:05.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-09-22T21:35:01.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-09-22T21:35:02.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-09-22T21:35:05.376Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-09-22T21:35:04.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-09-22T21:35:05.285Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-09-22T21:35:12.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-09-22T21:34:59.536Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-09-22T21:35:03.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-09-22T21:35:03.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-09-22T21:35:04.277Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-09-22T21:35:03.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-09-22T21:35:04.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-09-22T21:35:03.651Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:10.906 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:10.906 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:10.906 DEBUG:boto:path=/ffzdqu-58/?versions&max-keys=999 2023-09-22T21:36:10.906 DEBUG:boto:auth_path=/ffzdqu-58/?versions&max-keys=999 2023-09-22T21:36:10.906 DEBUG:boto:Method: GET 2023-09-22T21:36:10.906 DEBUG:boto:Path: /ffzdqu-58/?versions&max-keys=999 2023-09-22T21:36:10.907 DEBUG:boto:Data: 2023-09-22T21:36:10.907 DEBUG:boto:Headers: {} 2023-09-22T21:36:10.907 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:10.907 DEBUG:boto:Port: 8000 2023-09-22T21:36:10.907 DEBUG:boto:Params: {} 2023-09-22T21:36:10.907 DEBUG:boto:Token: None 2023-09-22T21:36:10.907 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:10 GMT /ffzdqu-58/?versions 2023-09-22T21:36:10.907 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HAtX1MwI0d3VpVzEyH5qBoOuQos= 2023-09-22T21:36:10.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HAtX1MwI0d3VpVzEyH5qBoOuQos=', 'Content-Length': '0'} 2023-09-22T21:36:10.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d0d17ff9912560b-00650e08ca-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:10.916 DEBUG:boto:b'ffzdqu-58999falseobj00000000000000000000nulltrue2023-09-22T21:35:05.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-09-22T21:35:12.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-09-22T21:35:12.447Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-09-22T21:35:09.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-09-22T21:35:12.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-09-22T21:35:11.989Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-09-22T21:35:12.246Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-09-22T21:35:11.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-09-22T21:35:12.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-09-22T21:35:05.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-09-22T21:35:05.514Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-09-22T21:35:12.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-09-22T21:35:11.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-09-22T21:35:02.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-09-22T21:35:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-09-22T21:35:05.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-09-22T21:35:05.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-09-22T21:35:05.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-09-22T21:35:12.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-09-22T21:35:05.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-09-22T21:35:12.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-09-22T21:35:05.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-09-22T21:35:03.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-09-22T21:35:03.585Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-09-22T21:35:05.767Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-09-22T21:35:02.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-09-22T21:35:12.586Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-09-22T21:35:09.970Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-09-22T21:34:59.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-09-22T21:35:03.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-09-22T21:35:11.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-09-22T21:35:03.155Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-09-22T21:35:01.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-09-22T21:35:01.495Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-09-22T21:35:05.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-09-22T21:35:02.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-09-22T21:35:12.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-09-22T21:35:12.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-09-22T21:35:03.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-09-22T21:35:12.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-09-22T21:35:02.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-09-22T21:35:04.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-09-22T21:35:12.185Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-09-22T21:35:01.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-09-22T21:35:05.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-09-22T21:35:01.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-09-22T21:35:02.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-09-22T21:35:05.376Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-09-22T21:35:04.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-09-22T21:35:05.285Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-09-22T21:35:12.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-09-22T21:34:59.536Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-09-22T21:35:03.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-09-22T21:35:03.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-09-22T21:35:04.277Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-09-22T21:35:03.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-09-22T21:35:04.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-09-22T21:35:03.651Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:10.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-09-22T21:36:10.920 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:10.920 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:10.920 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.920 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.920 DEBUG:boto:Method: GET 2023-09-22T21:36:10.920 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.920 DEBUG:boto:Data: 2023-09-22T21:36:10.920 DEBUG:boto:Headers: {} 2023-09-22T21:36:10.920 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:10.920 DEBUG:boto:Port: 8001 2023-09-22T21:36:10.921 DEBUG:boto:Params: {} 2023-09-22T21:36:10.921 DEBUG:boto:Token: None 2023-09-22T21:36:10.921 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:10 GMT /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.921 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nfROk3ZlQ+QCPZbEPmtzDepsGB0= 2023-09-22T21:36:10.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nfROk3ZlQ+QCPZbEPmtzDepsGB0=', 'Content-Length': '0'} 2023-09-22T21:36:10.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d23d4224203c2c-00650e08ca-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:10.970 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:10.970 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:10.971 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.971 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.971 DEBUG:boto:Method: GET 2023-09-22T21:36:10.971 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.971 DEBUG:boto:Data: 2023-09-22T21:36:10.971 DEBUG:boto:Headers: {} 2023-09-22T21:36:10.971 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:10.971 DEBUG:boto:Port: 8000 2023-09-22T21:36:10.971 DEBUG:boto:Params: {} 2023-09-22T21:36:10.971 DEBUG:boto:Token: None 2023-09-22T21:36:10.972 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:10 GMT /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.972 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nfROk3ZlQ+QCPZbEPmtzDepsGB0= 2023-09-22T21:36:10.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nfROk3ZlQ+QCPZbEPmtzDepsGB0=', 'Content-Length': '0'} 2023-09-22T21:36:10.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f309527205f1808-00650e08ca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:10.976 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:10.976 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:10.976 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.976 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.976 DEBUG:boto:Method: HEAD 2023-09-22T21:36:10.977 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.977 DEBUG:boto:Data: 2023-09-22T21:36:10.977 DEBUG:boto:Headers: {} 2023-09-22T21:36:10.977 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:10.977 DEBUG:boto:Port: 8001 2023-09-22T21:36:10.977 DEBUG:boto:Params: {} 2023-09-22T21:36:10.977 DEBUG:boto:Token: None 2023-09-22T21:36:10.977 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:10 GMT /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.977 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+J8BDE37xXCXAcjMffHitmzP3+o= 2023-09-22T21:36:10.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+J8BDE37xXCXAcjMffHitmzP3+o=', 'Content-Length': '0'} 2023-09-22T21:36:10.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000406952b103c3c2d4-00650e08ca-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:10.981 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:10.981 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:10.981 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.981 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.981 DEBUG:boto:Method: HEAD 2023-09-22T21:36:10.981 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.981 DEBUG:boto:Data: 2023-09-22T21:36:10.981 DEBUG:boto:Headers: {} 2023-09-22T21:36:10.981 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:10.981 DEBUG:boto:Port: 8000 2023-09-22T21:36:10.982 DEBUG:boto:Params: {} 2023-09-22T21:36:10.982 DEBUG:boto:Token: None 2023-09-22T21:36:10.982 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:10 GMT /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.982 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+J8BDE37xXCXAcjMffHitmzP3+o= 2023-09-22T21:36:10.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+J8BDE37xXCXAcjMffHitmzP3+o=', 'Content-Length': '0'} 2023-09-22T21:36:10.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5bd97358d30dd92-00650e08ca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:10.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-09-22T21:36:10.985 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:10.985 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:10.986 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.986 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.986 DEBUG:boto:Method: GET 2023-09-22T21:36:10.986 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.986 DEBUG:boto:Data: 2023-09-22T21:36:10.986 DEBUG:boto:Headers: {} 2023-09-22T21:36:10.986 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:10.986 DEBUG:boto:Port: 8001 2023-09-22T21:36:10.986 DEBUG:boto:Params: {} 2023-09-22T21:36:10.986 DEBUG:boto:Token: None 2023-09-22T21:36:10.987 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:10 GMT /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.987 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nfROk3ZlQ+QCPZbEPmtzDepsGB0= 2023-09-22T21:36:10.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nfROk3ZlQ+QCPZbEPmtzDepsGB0=', 'Content-Length': '0'} 2023-09-22T21:36:10.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023550cca3c21b7ed-00650e08ca-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:10.990 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:10.990 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:10.990 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.990 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.991 DEBUG:boto:Method: GET 2023-09-22T21:36:10.991 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.991 DEBUG:boto:Data: 2023-09-22T21:36:10.991 DEBUG:boto:Headers: {} 2023-09-22T21:36:10.991 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:10.991 DEBUG:boto:Port: 8000 2023-09-22T21:36:10.991 DEBUG:boto:Params: {} 2023-09-22T21:36:10.991 DEBUG:boto:Token: None 2023-09-22T21:36:10.991 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:10 GMT /ffzdqu-58/obj00000000000000000000?versionId=null 2023-09-22T21:36:10.992 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nfROk3ZlQ+QCPZbEPmtzDepsGB0= 2023-09-22T21:36:10.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nfROk3ZlQ+QCPZbEPmtzDepsGB0=', 'Content-Length': '0'} 2023-09-22T21:36:10.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000088917521e8ffe1b-00650e08ca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:10.994 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:10.995 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:10.995 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:36:10.995 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:36:10.995 DEBUG:boto:Method: GET 2023-09-22T21:36:10.995 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:36:10.995 DEBUG:boto:Data: 2023-09-22T21:36:10.995 DEBUG:boto:Headers: {} 2023-09-22T21:36:10.995 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:10.995 DEBUG:boto:Port: 8001 2023-09-22T21:36:10.995 DEBUG:boto:Params: {} 2023-09-22T21:36:10.996 DEBUG:boto:Token: None 2023-09-22T21:36:10.996 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:10 GMT /ffzdqu-58/ 2023-09-22T21:36:10.996 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ca2qDuDzKKVLNydJFl1zxg+s03M= 2023-09-22T21:36:10.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ca2qDuDzKKVLNydJFl1zxg+s03M=', 'Content-Length': '0'} 2023-09-22T21:36:11.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009368927b1f31ab4c-00650e08ca-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.000 DEBUG:boto:b'ffzdqu-58obj000000000000000000001000falseobj000000000000000000002023-09-22T21:35:05.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:11.001 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:11.001 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:11.001 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:36:11.001 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:36:11.001 DEBUG:boto:Method: GET 2023-09-22T21:36:11.001 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:36:11.001 DEBUG:boto:Data: 2023-09-22T21:36:11.002 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.002 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.002 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.002 DEBUG:boto:Params: {} 2023-09-22T21:36:11.002 DEBUG:boto:Token: None 2023-09-22T21:36:11.002 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/ 2023-09-22T21:36:11.002 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4= 2023-09-22T21:36:11.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4=', 'Content-Length': '0'} 2023-09-22T21:36:11.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000437b86b3b7a55e7-00650e08ca-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.013 DEBUG:boto:b'ffzdqu-58obj000000000000000000001000falseobj000000000000000000002023-09-22T21:35:05.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:11.013 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:11.014 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:11.014 DEBUG:boto:Method: HEAD 2023-09-22T21:36:11.014 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:11.014 DEBUG:boto:Data: 2023-09-22T21:36:11.014 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.014 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.014 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.014 DEBUG:boto:Params: {} 2023-09-22T21:36:11.014 DEBUG:boto:Token: None 2023-09-22T21:36:11.015 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:11.015 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bihPLtTYtX8MMOtdpgTlWd/Z1sQ= 2023-09-22T21:36:11.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bihPLtTYtX8MMOtdpgTlWd/Z1sQ=', 'Content-Length': '0'} 2023-09-22T21:36:11.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8d3fa7646738278-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.018 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:11.018 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:11.018 DEBUG:boto:Method: HEAD 2023-09-22T21:36:11.018 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:11.018 DEBUG:boto:Data: 2023-09-22T21:36:11.018 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.018 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.018 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.018 DEBUG:boto:Params: {} 2023-09-22T21:36:11.019 DEBUG:boto:Token: None 2023-09-22T21:36:11.019 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:11.019 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bihPLtTYtX8MMOtdpgTlWd/Z1sQ= 2023-09-22T21:36:11.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bihPLtTYtX8MMOtdpgTlWd/Z1sQ=', 'Content-Length': '0'} 2023-09-22T21:36:11.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb77832af8ae4c9f-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-09-22T21:36:11.022 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:11.022 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:11.022 DEBUG:boto:Method: GET 2023-09-22T21:36:11.022 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:11.022 DEBUG:boto:Data: 2023-09-22T21:36:11.022 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.023 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.023 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.023 DEBUG:boto:Params: {} 2023-09-22T21:36:11.023 DEBUG:boto:Token: None 2023-09-22T21:36:11.023 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:11.023 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CmdoEp9fNGpKsIZqY7m2G812veA= 2023-09-22T21:36:11.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CmdoEp9fNGpKsIZqY7m2G812veA=', 'Content-Length': '0'} 2023-09-22T21:36:11.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027bf7fdafbc8b25f-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.066 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:11.066 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:11.067 DEBUG:boto:Method: GET 2023-09-22T21:36:11.067 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:11.067 DEBUG:boto:Data: 2023-09-22T21:36:11.067 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.067 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.067 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.067 DEBUG:boto:Params: {} 2023-09-22T21:36:11.067 DEBUG:boto:Token: None 2023-09-22T21:36:11.068 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000000 2023-09-22T21:36:11.068 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CmdoEp9fNGpKsIZqY7m2G812veA= 2023-09-22T21:36:11.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CmdoEp9fNGpKsIZqY7m2G812veA=', 'Content-Length': '0'} 2023-09-22T21:36:11.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001526c10bf1fd48d0-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.071 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:11.071 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:11.071 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:36:11.071 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:36:11.071 DEBUG:boto:Method: GET 2023-09-22T21:36:11.071 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:36:11.071 DEBUG:boto:Data: 2023-09-22T21:36:11.072 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.072 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.072 DEBUG:boto:Params: {} 2023-09-22T21:36:11.072 DEBUG:boto:Token: None 2023-09-22T21:36:11.072 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/ 2023-09-22T21:36:11.072 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4= 2023-09-22T21:36:11.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4=', 'Content-Length': '0'} 2023-09-22T21:36:11.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000459da7874c4dd006-00650e08cb-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.077 DEBUG:boto:b'ffzdqu-58obj000000000000000000001000falseobj000000000000000000002023-09-22T21:35:05.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:11.077 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:11.077 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:11.078 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:36:11.078 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:36:11.078 DEBUG:boto:Method: GET 2023-09-22T21:36:11.078 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000000 2023-09-22T21:36:11.078 DEBUG:boto:Data: 2023-09-22T21:36:11.078 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.078 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.078 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.078 DEBUG:boto:Params: {} 2023-09-22T21:36:11.078 DEBUG:boto:Token: None 2023-09-22T21:36:11.079 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/ 2023-09-22T21:36:11.079 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4= 2023-09-22T21:36:11.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4=', 'Content-Length': '0'} 2023-09-22T21:36:11.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb6348e79904ea87-00650e08cb-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.090 DEBUG:boto:b'ffzdqu-58obj000000000000000000001000falseobj000000000000000000002023-09-22T21:35:05.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:11.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-09-22T21:36:11.090 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.090 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.090 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.090 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.091 DEBUG:boto:Method: GET 2023-09-22T21:36:11.091 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.091 DEBUG:boto:Data: 2023-09-22T21:36:11.091 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.091 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.091 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.091 DEBUG:boto:Params: {} 2023-09-22T21:36:11.091 DEBUG:boto:Token: None 2023-09-22T21:36:11.091 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.092 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vChE8LCmxchgwKNqUjUHB9fEhJU= 2023-09-22T21:36:11.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vChE8LCmxchgwKNqUjUHB9fEhJU=', 'Content-Length': '0'} 2023-09-22T21:36:11.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f11324a17b4b0499-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.096 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.096 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.096 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.096 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.096 DEBUG:boto:Method: GET 2023-09-22T21:36:11.096 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.096 DEBUG:boto:Data: 2023-09-22T21:36:11.096 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.097 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.097 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.097 DEBUG:boto:Params: {} 2023-09-22T21:36:11.097 DEBUG:boto:Token: None 2023-09-22T21:36:11.097 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.097 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vChE8LCmxchgwKNqUjUHB9fEhJU= 2023-09-22T21:36:11.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vChE8LCmxchgwKNqUjUHB9fEhJU=', 'Content-Length': '0'} 2023-09-22T21:36:11.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006eb8cee1934aeb7f-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.100 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.100 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.100 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.100 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.101 DEBUG:boto:Method: HEAD 2023-09-22T21:36:11.101 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.101 DEBUG:boto:Data: 2023-09-22T21:36:11.101 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.101 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.101 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.101 DEBUG:boto:Params: {} 2023-09-22T21:36:11.101 DEBUG:boto:Token: None 2023-09-22T21:36:11.101 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.102 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oRNmpx2fk52BfkaGs17DTJnMeqU= 2023-09-22T21:36:11.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oRNmpx2fk52BfkaGs17DTJnMeqU=', 'Content-Length': '0'} 2023-09-22T21:36:11.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec89925a45ac20c6-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.104 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.105 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.105 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.105 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.105 DEBUG:boto:Method: HEAD 2023-09-22T21:36:11.105 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.105 DEBUG:boto:Data: 2023-09-22T21:36:11.105 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.105 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.105 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.105 DEBUG:boto:Params: {} 2023-09-22T21:36:11.106 DEBUG:boto:Token: None 2023-09-22T21:36:11.106 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.106 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oRNmpx2fk52BfkaGs17DTJnMeqU= 2023-09-22T21:36:11.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oRNmpx2fk52BfkaGs17DTJnMeqU=', 'Content-Length': '0'} 2023-09-22T21:36:11.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000113634a1b69a2e7c-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-09-22T21:36:11.109 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.109 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.109 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.109 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.109 DEBUG:boto:Method: GET 2023-09-22T21:36:11.109 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.109 DEBUG:boto:Data: 2023-09-22T21:36:11.110 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.110 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.110 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.110 DEBUG:boto:Params: {} 2023-09-22T21:36:11.110 DEBUG:boto:Token: None 2023-09-22T21:36:11.110 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.110 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vChE8LCmxchgwKNqUjUHB9fEhJU= 2023-09-22T21:36:11.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vChE8LCmxchgwKNqUjUHB9fEhJU=', 'Content-Length': '0'} 2023-09-22T21:36:11.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012ac03a6eb97d813-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.158 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.158 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.159 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.159 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.159 DEBUG:boto:Method: GET 2023-09-22T21:36:11.159 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.159 DEBUG:boto:Data: 2023-09-22T21:36:11.159 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.159 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.159 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.159 DEBUG:boto:Params: {} 2023-09-22T21:36:11.159 DEBUG:boto:Token: None 2023-09-22T21:36:11.160 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000001?versionId=null 2023-09-22T21:36:11.160 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vChE8LCmxchgwKNqUjUHB9fEhJU= 2023-09-22T21:36:11.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vChE8LCmxchgwKNqUjUHB9fEhJU=', 'Content-Length': '0'} 2023-09-22T21:36:11.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070f1ac4d510932ff-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.163 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:11.163 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:11.163 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:36:11.163 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:36:11.163 DEBUG:boto:Method: GET 2023-09-22T21:36:11.164 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:36:11.164 DEBUG:boto:Data: 2023-09-22T21:36:11.164 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.164 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.164 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.164 DEBUG:boto:Params: {} 2023-09-22T21:36:11.164 DEBUG:boto:Token: None 2023-09-22T21:36:11.164 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/ 2023-09-22T21:36:11.164 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4= 2023-09-22T21:36:11.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4=', 'Content-Length': '0'} 2023-09-22T21:36:11.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d110e6c494fca93c-00650e08cb-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.169 DEBUG:boto:b'ffzdqu-58obj000000000000000000011000falseobj000000000000000000012023-09-22T21:35:12.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:11.169 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:11.170 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:11.170 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:36:11.170 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:36:11.170 DEBUG:boto:Method: GET 2023-09-22T21:36:11.170 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:36:11.170 DEBUG:boto:Data: 2023-09-22T21:36:11.170 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.170 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.170 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.170 DEBUG:boto:Params: {} 2023-09-22T21:36:11.171 DEBUG:boto:Token: None 2023-09-22T21:36:11.171 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/ 2023-09-22T21:36:11.171 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4= 2023-09-22T21:36:11.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4=', 'Content-Length': '0'} 2023-09-22T21:36:11.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b13808ad66a60d9f-00650e08cb-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.181 DEBUG:boto:b'ffzdqu-58obj000000000000000000011000falseobj000000000000000000012023-09-22T21:35:12.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:11.181 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.181 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.181 DEBUG:boto:Method: HEAD 2023-09-22T21:36:11.182 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.182 DEBUG:boto:Data: 2023-09-22T21:36:11.182 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.182 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.182 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.182 DEBUG:boto:Params: {} 2023-09-22T21:36:11.182 DEBUG:boto:Token: None 2023-09-22T21:36:11.182 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.182 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4KZFjWa1TeEKpIu+eO5OhL5bQ0E= 2023-09-22T21:36:11.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4KZFjWa1TeEKpIu+eO5OhL5bQ0E=', 'Content-Length': '0'} 2023-09-22T21:36:11.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d355d7ec0456352-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.185 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.185 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.186 DEBUG:boto:Method: HEAD 2023-09-22T21:36:11.186 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.186 DEBUG:boto:Data: 2023-09-22T21:36:11.186 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.186 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.186 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.186 DEBUG:boto:Params: {} 2023-09-22T21:36:11.186 DEBUG:boto:Token: None 2023-09-22T21:36:11.186 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.187 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4KZFjWa1TeEKpIu+eO5OhL5bQ0E= 2023-09-22T21:36:11.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4KZFjWa1TeEKpIu+eO5OhL5bQ0E=', 'Content-Length': '0'} 2023-09-22T21:36:11.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073261f5d4b13a7e2-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-09-22T21:36:11.189 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.190 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.190 DEBUG:boto:Method: GET 2023-09-22T21:36:11.190 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.190 DEBUG:boto:Data: 2023-09-22T21:36:11.190 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.190 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.190 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.190 DEBUG:boto:Params: {} 2023-09-22T21:36:11.190 DEBUG:boto:Token: None 2023-09-22T21:36:11.191 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.191 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VGSjbNSMVz2lktANEN/uJfJwXYk= 2023-09-22T21:36:11.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VGSjbNSMVz2lktANEN/uJfJwXYk=', 'Content-Length': '0'} 2023-09-22T21:36:11.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000983bb0238c657c74-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.194 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.194 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.194 DEBUG:boto:Method: GET 2023-09-22T21:36:11.194 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.194 DEBUG:boto:Data: 2023-09-22T21:36:11.194 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.194 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.195 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.195 DEBUG:boto:Params: {} 2023-09-22T21:36:11.195 DEBUG:boto:Token: None 2023-09-22T21:36:11.195 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000001 2023-09-22T21:36:11.195 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VGSjbNSMVz2lktANEN/uJfJwXYk= 2023-09-22T21:36:11.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VGSjbNSMVz2lktANEN/uJfJwXYk=', 'Content-Length': '0'} 2023-09-22T21:36:11.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1743a1945569b15-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.198 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:11.198 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:11.198 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:36:11.198 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:36:11.199 DEBUG:boto:Method: GET 2023-09-22T21:36:11.199 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:36:11.199 DEBUG:boto:Data: 2023-09-22T21:36:11.199 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.199 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.199 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.199 DEBUG:boto:Params: {} 2023-09-22T21:36:11.199 DEBUG:boto:Token: None 2023-09-22T21:36:11.199 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/ 2023-09-22T21:36:11.200 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4= 2023-09-22T21:36:11.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4=', 'Content-Length': '0'} 2023-09-22T21:36:11.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045944a37fed96a27-00650e08cb-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.204 DEBUG:boto:b'ffzdqu-58obj000000000000000000011000falseobj000000000000000000012023-09-22T21:35:12.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:11.205 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:11.205 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:11.205 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:36:11.205 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:36:11.205 DEBUG:boto:Method: GET 2023-09-22T21:36:11.205 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000001 2023-09-22T21:36:11.205 DEBUG:boto:Data: 2023-09-22T21:36:11.206 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.206 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.206 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.206 DEBUG:boto:Params: {} 2023-09-22T21:36:11.206 DEBUG:boto:Token: None 2023-09-22T21:36:11.206 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/ 2023-09-22T21:36:11.206 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4= 2023-09-22T21:36:11.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4=', 'Content-Length': '0'} 2023-09-22T21:36:11.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009499aae716e1ffbd-00650e08cb-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.217 DEBUG:boto:b'ffzdqu-58obj000000000000000000011000falseobj000000000000000000012023-09-22T21:35:12.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:11.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-09-22T21:36:11.217 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.217 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.217 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.217 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.217 DEBUG:boto:Method: GET 2023-09-22T21:36:11.218 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.218 DEBUG:boto:Data: 2023-09-22T21:36:11.218 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.218 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.218 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.218 DEBUG:boto:Params: {} 2023-09-22T21:36:11.218 DEBUG:boto:Token: None 2023-09-22T21:36:11.218 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.218 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mkFtL7yFT4Krt/KY+EvckvltQyU= 2023-09-22T21:36:11.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mkFtL7yFT4Krt/KY+EvckvltQyU=', 'Content-Length': '0'} 2023-09-22T21:36:11.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000277a0ed15edde733-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.262 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.262 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.263 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.263 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.263 DEBUG:boto:Method: GET 2023-09-22T21:36:11.263 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.263 DEBUG:boto:Data: 2023-09-22T21:36:11.263 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.263 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.263 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.263 DEBUG:boto:Params: {} 2023-09-22T21:36:11.263 DEBUG:boto:Token: None 2023-09-22T21:36:11.264 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.264 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mkFtL7yFT4Krt/KY+EvckvltQyU= 2023-09-22T21:36:11.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mkFtL7yFT4Krt/KY+EvckvltQyU=', 'Content-Length': '0'} 2023-09-22T21:36:11.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a3f743b4ed2cabe-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.267 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.267 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.267 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.267 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.267 DEBUG:boto:Method: HEAD 2023-09-22T21:36:11.268 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.268 DEBUG:boto:Data: 2023-09-22T21:36:11.268 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.268 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.268 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.268 DEBUG:boto:Params: {} 2023-09-22T21:36:11.268 DEBUG:boto:Token: None 2023-09-22T21:36:11.268 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.268 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Nl2eP98z49JLqiEULi5ZSJow7rw= 2023-09-22T21:36:11.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Nl2eP98z49JLqiEULi5ZSJow7rw=', 'Content-Length': '0'} 2023-09-22T21:36:11.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8ae04d58eca048c-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.271 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.271 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.272 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.272 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.272 DEBUG:boto:Method: HEAD 2023-09-22T21:36:11.272 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.272 DEBUG:boto:Data: 2023-09-22T21:36:11.272 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.272 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.272 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.272 DEBUG:boto:Params: {} 2023-09-22T21:36:11.272 DEBUG:boto:Token: None 2023-09-22T21:36:11.273 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.273 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Nl2eP98z49JLqiEULi5ZSJow7rw= 2023-09-22T21:36:11.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Nl2eP98z49JLqiEULi5ZSJow7rw=', 'Content-Length': '0'} 2023-09-22T21:36:11.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3eccffbd0e52aa7-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-09-22T21:36:11.276 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.276 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.277 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.277 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.277 DEBUG:boto:Method: GET 2023-09-22T21:36:11.277 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.277 DEBUG:boto:Data: 2023-09-22T21:36:11.277 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.277 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.277 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.277 DEBUG:boto:Params: {} 2023-09-22T21:36:11.277 DEBUG:boto:Token: None 2023-09-22T21:36:11.278 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.278 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mkFtL7yFT4Krt/KY+EvckvltQyU= 2023-09-22T21:36:11.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mkFtL7yFT4Krt/KY+EvckvltQyU=', 'Content-Length': '0'} 2023-09-22T21:36:11.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055ee3e7540d58d34-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.326 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.326 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.326 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.327 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.327 DEBUG:boto:Method: GET 2023-09-22T21:36:11.327 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.327 DEBUG:boto:Data: 2023-09-22T21:36:11.327 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.327 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.327 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.327 DEBUG:boto:Params: {} 2023-09-22T21:36:11.327 DEBUG:boto:Token: None 2023-09-22T21:36:11.328 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000002?versionId=null 2023-09-22T21:36:11.328 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mkFtL7yFT4Krt/KY+EvckvltQyU= 2023-09-22T21:36:11.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mkFtL7yFT4Krt/KY+EvckvltQyU=', 'Content-Length': '0'} 2023-09-22T21:36:11.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbe97b59cb9f83f4-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.331 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:11.331 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:11.331 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:36:11.331 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:36:11.331 DEBUG:boto:Method: GET 2023-09-22T21:36:11.332 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:36:11.332 DEBUG:boto:Data: 2023-09-22T21:36:11.332 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.332 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.332 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.332 DEBUG:boto:Params: {} 2023-09-22T21:36:11.332 DEBUG:boto:Token: None 2023-09-22T21:36:11.332 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/ 2023-09-22T21:36:11.332 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4= 2023-09-22T21:36:11.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4=', 'Content-Length': '0'} 2023-09-22T21:36:11.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6af1903486c7c07-00650e08cb-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.378 DEBUG:boto:b'ffzdqu-58obj000000000000000000021000falseobj000000000000000000022023-09-22T21:35:12.447Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:11.379 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:11.379 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:11.379 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:36:11.379 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:36:11.379 DEBUG:boto:Method: GET 2023-09-22T21:36:11.379 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:36:11.380 DEBUG:boto:Data: 2023-09-22T21:36:11.380 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.380 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.380 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.380 DEBUG:boto:Params: {} 2023-09-22T21:36:11.380 DEBUG:boto:Token: None 2023-09-22T21:36:11.380 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/ 2023-09-22T21:36:11.380 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4= 2023-09-22T21:36:11.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4=', 'Content-Length': '0'} 2023-09-22T21:36:11.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038d609e5f4f04c14-00650e08cb-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.390 DEBUG:boto:b'ffzdqu-58obj000000000000000000021000falseobj000000000000000000022023-09-22T21:35:12.447Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:11.391 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.391 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.391 DEBUG:boto:Method: HEAD 2023-09-22T21:36:11.391 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.391 DEBUG:boto:Data: 2023-09-22T21:36:11.391 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.391 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.391 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.391 DEBUG:boto:Params: {} 2023-09-22T21:36:11.392 DEBUG:boto:Token: None 2023-09-22T21:36:11.392 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.392 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x3AdPmVZqGwqvLG8jglyvrY6zCo= 2023-09-22T21:36:11.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x3AdPmVZqGwqvLG8jglyvrY6zCo=', 'Content-Length': '0'} 2023-09-22T21:36:11.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb7ef44a29b78a8c-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.395 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.395 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.395 DEBUG:boto:Method: HEAD 2023-09-22T21:36:11.395 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.395 DEBUG:boto:Data: 2023-09-22T21:36:11.395 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.396 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.396 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.396 DEBUG:boto:Params: {} 2023-09-22T21:36:11.396 DEBUG:boto:Token: None 2023-09-22T21:36:11.396 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.396 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x3AdPmVZqGwqvLG8jglyvrY6zCo= 2023-09-22T21:36:11.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x3AdPmVZqGwqvLG8jglyvrY6zCo=', 'Content-Length': '0'} 2023-09-22T21:36:11.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0449ca5f000a9eb-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-09-22T21:36:11.399 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.400 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.400 DEBUG:boto:Method: GET 2023-09-22T21:36:11.400 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.400 DEBUG:boto:Data: 2023-09-22T21:36:11.400 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.400 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.400 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.400 DEBUG:boto:Params: {} 2023-09-22T21:36:11.400 DEBUG:boto:Token: None 2023-09-22T21:36:11.401 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.401 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bU8Hi54NVWiU6e880ZckOp1m84k= 2023-09-22T21:36:11.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bU8Hi54NVWiU6e880ZckOp1m84k=', 'Content-Length': '0'} 2023-09-22T21:36:11.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071810c93d8bf1b73-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.450 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.450 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.451 DEBUG:boto:Method: GET 2023-09-22T21:36:11.451 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.451 DEBUG:boto:Data: 2023-09-22T21:36:11.451 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.451 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.451 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.451 DEBUG:boto:Params: {} 2023-09-22T21:36:11.451 DEBUG:boto:Token: None 2023-09-22T21:36:11.451 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000002 2023-09-22T21:36:11.452 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bU8Hi54NVWiU6e880ZckOp1m84k= 2023-09-22T21:36:11.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bU8Hi54NVWiU6e880ZckOp1m84k=', 'Content-Length': '0'} 2023-09-22T21:36:11.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b70065a99a25e865-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.455 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:11.455 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:11.455 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:36:11.455 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:36:11.456 DEBUG:boto:Method: GET 2023-09-22T21:36:11.456 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:36:11.456 DEBUG:boto:Data: 2023-09-22T21:36:11.456 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.456 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.456 DEBUG:boto:Params: {} 2023-09-22T21:36:11.456 DEBUG:boto:Token: None 2023-09-22T21:36:11.456 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/ 2023-09-22T21:36:11.456 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4= 2023-09-22T21:36:11.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4=', 'Content-Length': '0'} 2023-09-22T21:36:11.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000842bcf379fb59fa8-00650e08cb-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.502 DEBUG:boto:b'ffzdqu-58obj000000000000000000021000falseobj000000000000000000022023-09-22T21:35:12.447Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:11.503 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:11.503 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:11.503 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:36:11.503 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:36:11.503 DEBUG:boto:Method: GET 2023-09-22T21:36:11.503 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000002 2023-09-22T21:36:11.504 DEBUG:boto:Data: 2023-09-22T21:36:11.504 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.504 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.504 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.504 DEBUG:boto:Params: {} 2023-09-22T21:36:11.504 DEBUG:boto:Token: None 2023-09-22T21:36:11.504 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/ 2023-09-22T21:36:11.504 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4= 2023-09-22T21:36:11.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4=', 'Content-Length': '0'} 2023-09-22T21:36:11.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000accd8a8d2152f232-00650e08cb-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.515 DEBUG:boto:b'ffzdqu-58obj000000000000000000021000falseobj000000000000000000022023-09-22T21:35:12.447Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:11.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-09-22T21:36:11.516 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.516 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.516 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.516 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.516 DEBUG:boto:Method: GET 2023-09-22T21:36:11.517 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.517 DEBUG:boto:Data: 2023-09-22T21:36:11.517 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.517 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.517 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.517 DEBUG:boto:Params: {} 2023-09-22T21:36:11.517 DEBUG:boto:Token: None 2023-09-22T21:36:11.517 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.517 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lPYPgR1PXc4i1+CAG9mULDfC5Zo= 2023-09-22T21:36:11.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lPYPgR1PXc4i1+CAG9mULDfC5Zo=', 'Content-Length': '0'} 2023-09-22T21:36:11.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053d33f8fd4986c34-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.521 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.521 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.521 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.521 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.521 DEBUG:boto:Method: GET 2023-09-22T21:36:11.521 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.521 DEBUG:boto:Data: 2023-09-22T21:36:11.521 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.522 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.522 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.522 DEBUG:boto:Params: {} 2023-09-22T21:36:11.522 DEBUG:boto:Token: None 2023-09-22T21:36:11.522 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.522 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lPYPgR1PXc4i1+CAG9mULDfC5Zo= 2023-09-22T21:36:11.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lPYPgR1PXc4i1+CAG9mULDfC5Zo=', 'Content-Length': '0'} 2023-09-22T21:36:11.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4cf68aa4686fbf9-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.525 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.526 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.526 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.526 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.526 DEBUG:boto:Method: HEAD 2023-09-22T21:36:11.526 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.526 DEBUG:boto:Data: 2023-09-22T21:36:11.526 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.526 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.526 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.526 DEBUG:boto:Params: {} 2023-09-22T21:36:11.527 DEBUG:boto:Token: None 2023-09-22T21:36:11.527 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.527 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Rup8DYQmHFK5Q9NU2SHbqz4pIaY= 2023-09-22T21:36:11.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rup8DYQmHFK5Q9NU2SHbqz4pIaY=', 'Content-Length': '0'} 2023-09-22T21:36:11.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042bf0fad90419bb6-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.530 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.530 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.530 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.530 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.530 DEBUG:boto:Method: HEAD 2023-09-22T21:36:11.530 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.530 DEBUG:boto:Data: 2023-09-22T21:36:11.531 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.531 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.531 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.531 DEBUG:boto:Params: {} 2023-09-22T21:36:11.531 DEBUG:boto:Token: None 2023-09-22T21:36:11.531 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.531 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Rup8DYQmHFK5Q9NU2SHbqz4pIaY= 2023-09-22T21:36:11.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rup8DYQmHFK5Q9NU2SHbqz4pIaY=', 'Content-Length': '0'} 2023-09-22T21:36:11.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f0809da555e6c9c-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-09-22T21:36:11.534 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.534 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.534 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.534 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.535 DEBUG:boto:Method: GET 2023-09-22T21:36:11.535 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.535 DEBUG:boto:Data: 2023-09-22T21:36:11.535 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.535 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.535 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.535 DEBUG:boto:Params: {} 2023-09-22T21:36:11.535 DEBUG:boto:Token: None 2023-09-22T21:36:11.535 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.536 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lPYPgR1PXc4i1+CAG9mULDfC5Zo= 2023-09-22T21:36:11.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lPYPgR1PXc4i1+CAG9mULDfC5Zo=', 'Content-Length': '0'} 2023-09-22T21:36:11.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f428f3b38894bd3-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.582 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.583 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.583 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.583 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.583 DEBUG:boto:Method: GET 2023-09-22T21:36:11.583 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.583 DEBUG:boto:Data: 2023-09-22T21:36:11.583 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.583 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.583 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.583 DEBUG:boto:Params: {} 2023-09-22T21:36:11.584 DEBUG:boto:Token: None 2023-09-22T21:36:11.584 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000003?versionId=null 2023-09-22T21:36:11.584 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lPYPgR1PXc4i1+CAG9mULDfC5Zo= 2023-09-22T21:36:11.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lPYPgR1PXc4i1+CAG9mULDfC5Zo=', 'Content-Length': '0'} 2023-09-22T21:36:11.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8e743fb0aa68687-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.587 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:11.587 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:11.588 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:36:11.588 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:36:11.588 DEBUG:boto:Method: GET 2023-09-22T21:36:11.588 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:36:11.588 DEBUG:boto:Data: 2023-09-22T21:36:11.588 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.588 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.588 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.588 DEBUG:boto:Params: {} 2023-09-22T21:36:11.588 DEBUG:boto:Token: None 2023-09-22T21:36:11.589 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/ 2023-09-22T21:36:11.589 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4= 2023-09-22T21:36:11.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4=', 'Content-Length': '0'} 2023-09-22T21:36:11.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049d494031f31a431-00650e08cb-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.634 DEBUG:boto:b'ffzdqu-58obj000000000000000000031000falseobj000000000000000000032023-09-22T21:35:09.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:11.635 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:11.635 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:11.635 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:36:11.635 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:36:11.635 DEBUG:boto:Method: GET 2023-09-22T21:36:11.636 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:36:11.636 DEBUG:boto:Data: 2023-09-22T21:36:11.636 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.636 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.636 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.636 DEBUG:boto:Params: {} 2023-09-22T21:36:11.636 DEBUG:boto:Token: None 2023-09-22T21:36:11.636 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/ 2023-09-22T21:36:11.637 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4= 2023-09-22T21:36:11.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4=', 'Content-Length': '0'} 2023-09-22T21:36:11.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a5916f99483a37a-00650e08cb-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.646 DEBUG:boto:b'ffzdqu-58obj000000000000000000031000falseobj000000000000000000032023-09-22T21:35:09.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:11.647 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.647 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.647 DEBUG:boto:Method: HEAD 2023-09-22T21:36:11.647 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.647 DEBUG:boto:Data: 2023-09-22T21:36:11.647 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.647 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.647 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.647 DEBUG:boto:Params: {} 2023-09-22T21:36:11.648 DEBUG:boto:Token: None 2023-09-22T21:36:11.648 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.648 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2xpdMoCjBemTHdwj30RRb3q+1qc= 2023-09-22T21:36:11.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2xpdMoCjBemTHdwj30RRb3q+1qc=', 'Content-Length': '0'} 2023-09-22T21:36:11.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a195a50948b80288-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.651 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.651 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.651 DEBUG:boto:Method: HEAD 2023-09-22T21:36:11.651 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.651 DEBUG:boto:Data: 2023-09-22T21:36:11.652 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.652 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.652 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.652 DEBUG:boto:Params: {} 2023-09-22T21:36:11.652 DEBUG:boto:Token: None 2023-09-22T21:36:11.652 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.652 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2xpdMoCjBemTHdwj30RRb3q+1qc= 2023-09-22T21:36:11.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2xpdMoCjBemTHdwj30RRb3q+1qc=', 'Content-Length': '0'} 2023-09-22T21:36:11.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e616fa0ec49d71d6-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-09-22T21:36:11.655 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.655 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.655 DEBUG:boto:Method: GET 2023-09-22T21:36:11.656 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.656 DEBUG:boto:Data: 2023-09-22T21:36:11.656 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.656 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.656 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.656 DEBUG:boto:Params: {} 2023-09-22T21:36:11.656 DEBUG:boto:Token: None 2023-09-22T21:36:11.656 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.657 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:o6prk59RWjo88JgE8lU4JUif9KE= 2023-09-22T21:36:11.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:o6prk59RWjo88JgE8lU4JUif9KE=', 'Content-Length': '0'} 2023-09-22T21:36:11.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdbad138111385b3-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.706 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.707 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.707 DEBUG:boto:Method: GET 2023-09-22T21:36:11.707 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.707 DEBUG:boto:Data: 2023-09-22T21:36:11.707 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.707 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.707 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.707 DEBUG:boto:Params: {} 2023-09-22T21:36:11.707 DEBUG:boto:Token: None 2023-09-22T21:36:11.708 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000003 2023-09-22T21:36:11.708 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:o6prk59RWjo88JgE8lU4JUif9KE= 2023-09-22T21:36:11.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:o6prk59RWjo88JgE8lU4JUif9KE=', 'Content-Length': '0'} 2023-09-22T21:36:11.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb6a8e02d37bf702-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.711 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:11.712 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:11.712 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:36:11.712 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:36:11.712 DEBUG:boto:Method: GET 2023-09-22T21:36:11.712 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:36:11.712 DEBUG:boto:Data: 2023-09-22T21:36:11.712 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.712 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.712 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.712 DEBUG:boto:Params: {} 2023-09-22T21:36:11.713 DEBUG:boto:Token: None 2023-09-22T21:36:11.713 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/ 2023-09-22T21:36:11.713 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4= 2023-09-22T21:36:11.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4=', 'Content-Length': '0'} 2023-09-22T21:36:11.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004aa45b87ae5d744-00650e08cb-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.758 DEBUG:boto:b'ffzdqu-58obj000000000000000000031000falseobj000000000000000000032023-09-22T21:35:09.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:11.759 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:11.759 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:11.759 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:36:11.759 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:36:11.759 DEBUG:boto:Method: GET 2023-09-22T21:36:11.759 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000003 2023-09-22T21:36:11.759 DEBUG:boto:Data: 2023-09-22T21:36:11.760 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.760 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.760 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.760 DEBUG:boto:Params: {} 2023-09-22T21:36:11.760 DEBUG:boto:Token: None 2023-09-22T21:36:11.760 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/ 2023-09-22T21:36:11.760 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4= 2023-09-22T21:36:11.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4=', 'Content-Length': '0'} 2023-09-22T21:36:11.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001add6cfd12c75556-00650e08cb-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.771 DEBUG:boto:b'ffzdqu-58obj000000000000000000031000falseobj000000000000000000032023-09-22T21:35:09.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:11.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-09-22T21:36:11.771 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.771 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.772 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.772 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.772 DEBUG:boto:Method: GET 2023-09-22T21:36:11.772 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.772 DEBUG:boto:Data: 2023-09-22T21:36:11.772 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.772 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.772 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.772 DEBUG:boto:Params: {} 2023-09-22T21:36:11.772 DEBUG:boto:Token: None 2023-09-22T21:36:11.773 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.773 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+22oY8+PPhhnHAGSiIFDkx+RydM= 2023-09-22T21:36:11.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+22oY8+PPhhnHAGSiIFDkx+RydM=', 'Content-Length': '0'} 2023-09-22T21:36:11.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a30ab09019736e0d-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.777 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.777 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.777 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.777 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.777 DEBUG:boto:Method: GET 2023-09-22T21:36:11.777 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.777 DEBUG:boto:Data: 2023-09-22T21:36:11.777 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.778 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.778 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.778 DEBUG:boto:Params: {} 2023-09-22T21:36:11.778 DEBUG:boto:Token: None 2023-09-22T21:36:11.778 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.778 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+22oY8+PPhhnHAGSiIFDkx+RydM= 2023-09-22T21:36:11.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+22oY8+PPhhnHAGSiIFDkx+RydM=', 'Content-Length': '0'} 2023-09-22T21:36:11.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046988e3278de411b-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.781 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.782 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.782 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.782 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.782 DEBUG:boto:Method: HEAD 2023-09-22T21:36:11.782 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.782 DEBUG:boto:Data: 2023-09-22T21:36:11.782 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.782 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.782 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.783 DEBUG:boto:Params: {} 2023-09-22T21:36:11.783 DEBUG:boto:Token: None 2023-09-22T21:36:11.783 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.783 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sL2HxIgnfxgCNEsJvLjeltkVjjY= 2023-09-22T21:36:11.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sL2HxIgnfxgCNEsJvLjeltkVjjY=', 'Content-Length': '0'} 2023-09-22T21:36:11.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091678bae846b0ca1-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.786 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.786 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.786 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.786 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.787 DEBUG:boto:Method: HEAD 2023-09-22T21:36:11.787 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.787 DEBUG:boto:Data: 2023-09-22T21:36:11.787 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.787 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.787 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.787 DEBUG:boto:Params: {} 2023-09-22T21:36:11.787 DEBUG:boto:Token: None 2023-09-22T21:36:11.787 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.788 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sL2HxIgnfxgCNEsJvLjeltkVjjY= 2023-09-22T21:36:11.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sL2HxIgnfxgCNEsJvLjeltkVjjY=', 'Content-Length': '0'} 2023-09-22T21:36:11.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039a5cad400b7dbb5-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-09-22T21:36:11.791 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.791 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.791 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.791 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.791 DEBUG:boto:Method: GET 2023-09-22T21:36:11.791 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.792 DEBUG:boto:Data: 2023-09-22T21:36:11.792 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.792 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.792 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.792 DEBUG:boto:Params: {} 2023-09-22T21:36:11.792 DEBUG:boto:Token: None 2023-09-22T21:36:11.792 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.792 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+22oY8+PPhhnHAGSiIFDkx+RydM= 2023-09-22T21:36:11.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+22oY8+PPhhnHAGSiIFDkx+RydM=', 'Content-Length': '0'} 2023-09-22T21:36:11.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a2fa35d5260292a-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.838 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.838 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.839 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.839 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.839 DEBUG:boto:Method: GET 2023-09-22T21:36:11.839 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.839 DEBUG:boto:Data: 2023-09-22T21:36:11.839 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.839 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.839 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.839 DEBUG:boto:Params: {} 2023-09-22T21:36:11.840 DEBUG:boto:Token: None 2023-09-22T21:36:11.840 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000004?versionId=null 2023-09-22T21:36:11.840 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+22oY8+PPhhnHAGSiIFDkx+RydM= 2023-09-22T21:36:11.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+22oY8+PPhhnHAGSiIFDkx+RydM=', 'Content-Length': '0'} 2023-09-22T21:36:11.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096436f6bf184dee4-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.844 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:11.844 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:11.844 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000004 2023-09-22T21:36:11.844 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000004 2023-09-22T21:36:11.844 DEBUG:boto:Method: GET 2023-09-22T21:36:11.844 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000004 2023-09-22T21:36:11.845 DEBUG:boto:Data: 2023-09-22T21:36:11.845 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.845 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.845 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.845 DEBUG:boto:Params: {} 2023-09-22T21:36:11.845 DEBUG:boto:Token: None 2023-09-22T21:36:11.845 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/ 2023-09-22T21:36:11.845 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4= 2023-09-22T21:36:11.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4=', 'Content-Length': '0'} 2023-09-22T21:36:11.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c31ea5b615ab613d-00650e08cb-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.890 DEBUG:boto:b'ffzdqu-58obj000000000000000000041000falseobj000000000000000000042023-09-22T21:35:12.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:11.891 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:11.891 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:11.891 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000004 2023-09-22T21:36:11.891 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000004 2023-09-22T21:36:11.891 DEBUG:boto:Method: GET 2023-09-22T21:36:11.892 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000004 2023-09-22T21:36:11.892 DEBUG:boto:Data: 2023-09-22T21:36:11.892 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.892 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.892 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.892 DEBUG:boto:Params: {} 2023-09-22T21:36:11.892 DEBUG:boto:Token: None 2023-09-22T21:36:11.892 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/ 2023-09-22T21:36:11.892 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4= 2023-09-22T21:36:11.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4=', 'Content-Length': '0'} 2023-09-22T21:36:11.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000959b96bbdd1e2b58-00650e08cb-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.902 DEBUG:boto:b'ffzdqu-58obj000000000000000000041000falseobj000000000000000000042023-09-22T21:35:12.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:11.902 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.902 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.903 DEBUG:boto:Method: HEAD 2023-09-22T21:36:11.903 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.903 DEBUG:boto:Data: 2023-09-22T21:36:11.903 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.903 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.903 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.903 DEBUG:boto:Params: {} 2023-09-22T21:36:11.903 DEBUG:boto:Token: None 2023-09-22T21:36:11.903 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.904 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gkDdvDvttMO1kNPZshEIWk0wdmM= 2023-09-22T21:36:11.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gkDdvDvttMO1kNPZshEIWk0wdmM=', 'Content-Length': '0'} 2023-09-22T21:36:11.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bc5d5224341a834-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.907 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.907 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.907 DEBUG:boto:Method: HEAD 2023-09-22T21:36:11.907 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.907 DEBUG:boto:Data: 2023-09-22T21:36:11.907 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.907 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.907 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.907 DEBUG:boto:Params: {} 2023-09-22T21:36:11.908 DEBUG:boto:Token: None 2023-09-22T21:36:11.908 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.908 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gkDdvDvttMO1kNPZshEIWk0wdmM= 2023-09-22T21:36:11.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gkDdvDvttMO1kNPZshEIWk0wdmM=', 'Content-Length': '0'} 2023-09-22T21:36:11.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2d7807ad90c8721-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-09-22T21:36:11.911 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.911 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.911 DEBUG:boto:Method: GET 2023-09-22T21:36:11.911 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.911 DEBUG:boto:Data: 2023-09-22T21:36:11.911 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.912 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.912 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.912 DEBUG:boto:Params: {} 2023-09-22T21:36:11.912 DEBUG:boto:Token: None 2023-09-22T21:36:11.912 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.912 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ke7SRPKWHDah1un5yu2L8hMbqBI= 2023-09-22T21:36:11.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ke7SRPKWHDah1un5yu2L8hMbqBI=', 'Content-Length': '0'} 2023-09-22T21:36:11.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000160502666dad7d5d-00650e08cb-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.958 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.958 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.959 DEBUG:boto:Method: GET 2023-09-22T21:36:11.959 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.959 DEBUG:boto:Data: 2023-09-22T21:36:11.959 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.959 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:11.959 DEBUG:boto:Port: 8000 2023-09-22T21:36:11.959 DEBUG:boto:Params: {} 2023-09-22T21:36:11.959 DEBUG:boto:Token: None 2023-09-22T21:36:11.959 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/obj00000000000000000004 2023-09-22T21:36:11.959 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ke7SRPKWHDah1un5yu2L8hMbqBI= 2023-09-22T21:36:11.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ke7SRPKWHDah1un5yu2L8hMbqBI=', 'Content-Length': '0'} 2023-09-22T21:36:11.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc81b6e4de18dbbe-00650e08cb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:11.963 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:11.963 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:11.963 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000004 2023-09-22T21:36:11.963 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000004 2023-09-22T21:36:11.963 DEBUG:boto:Method: GET 2023-09-22T21:36:11.964 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000004 2023-09-22T21:36:11.964 DEBUG:boto:Data: 2023-09-22T21:36:11.964 DEBUG:boto:Headers: {} 2023-09-22T21:36:11.964 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:11.964 DEBUG:boto:Port: 8001 2023-09-22T21:36:11.964 DEBUG:boto:Params: {} 2023-09-22T21:36:11.964 DEBUG:boto:Token: None 2023-09-22T21:36:11.965 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:11 GMT /ffzdqu-58/ 2023-09-22T21:36:11.965 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4= 2023-09-22T21:36:11.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/7g+moEmFolKvEluNKNaaacWPu4=', 'Content-Length': '0'} 2023-09-22T21:36:11.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8bfe5fb5a2b9c04-00650e08cb-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.010 DEBUG:boto:b'ffzdqu-58obj000000000000000000041000falseobj000000000000000000042023-09-22T21:35:12.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:12.011 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:12.011 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:12.011 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000004 2023-09-22T21:36:12.011 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000004 2023-09-22T21:36:12.011 DEBUG:boto:Method: GET 2023-09-22T21:36:12.011 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000004 2023-09-22T21:36:12.011 DEBUG:boto:Data: 2023-09-22T21:36:12.011 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.011 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.012 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.012 DEBUG:boto:Params: {} 2023-09-22T21:36:12.012 DEBUG:boto:Token: None 2023-09-22T21:36:12.012 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/ 2023-09-22T21:36:12.012 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8= 2023-09-22T21:36:12.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8=', 'Content-Length': '0'} 2023-09-22T21:36:12.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1d0b2af11c5e6c1-00650e08cc-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.021 DEBUG:boto:b'ffzdqu-58obj000000000000000000041000falseobj000000000000000000042023-09-22T21:35:12.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:12.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-09-22T21:36:12.022 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.022 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.022 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.022 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.023 DEBUG:boto:Method: GET 2023-09-22T21:36:12.023 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.023 DEBUG:boto:Data: 2023-09-22T21:36:12.023 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.023 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.023 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.024 DEBUG:boto:Params: {} 2023-09-22T21:36:12.024 DEBUG:boto:Token: None 2023-09-22T21:36:12.024 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.024 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YRN1c5xoARd+ezNxNo67AExyzaE= 2023-09-22T21:36:12.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YRN1c5xoARd+ezNxNo67AExyzaE=', 'Content-Length': '0'} 2023-09-22T21:36:12.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a7f77c56325199b-00650e08cc-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.028 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.029 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.029 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.029 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.029 DEBUG:boto:Method: GET 2023-09-22T21:36:12.029 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.030 DEBUG:boto:Data: 2023-09-22T21:36:12.030 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.030 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.030 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.030 DEBUG:boto:Params: {} 2023-09-22T21:36:12.031 DEBUG:boto:Token: None 2023-09-22T21:36:12.031 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.031 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YRN1c5xoARd+ezNxNo67AExyzaE= 2023-09-22T21:36:12.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YRN1c5xoARd+ezNxNo67AExyzaE=', 'Content-Length': '0'} 2023-09-22T21:36:12.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006373e1ee8c1e08b2-00650e08cc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.082 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.083 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.083 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.083 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.083 DEBUG:boto:Method: HEAD 2023-09-22T21:36:12.083 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.083 DEBUG:boto:Data: 2023-09-22T21:36:12.083 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.083 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.083 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.084 DEBUG:boto:Params: {} 2023-09-22T21:36:12.084 DEBUG:boto:Token: None 2023-09-22T21:36:12.084 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.084 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qaYiNyQKn1FjUt6c5K0E9T81804= 2023-09-22T21:36:12.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qaYiNyQKn1FjUt6c5K0E9T81804=', 'Content-Length': '0'} 2023-09-22T21:36:12.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000915894f3d7d64b29-00650e08cc-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.087 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.087 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.087 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.088 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.088 DEBUG:boto:Method: HEAD 2023-09-22T21:36:12.088 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.088 DEBUG:boto:Data: 2023-09-22T21:36:12.088 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.088 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.088 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.088 DEBUG:boto:Params: {} 2023-09-22T21:36:12.088 DEBUG:boto:Token: None 2023-09-22T21:36:12.089 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.089 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qaYiNyQKn1FjUt6c5K0E9T81804= 2023-09-22T21:36:12.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qaYiNyQKn1FjUt6c5K0E9T81804=', 'Content-Length': '0'} 2023-09-22T21:36:12.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098d9add812e430c3-00650e08cc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-09-22T21:36:12.092 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.092 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.092 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.093 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.093 DEBUG:boto:Method: GET 2023-09-22T21:36:12.093 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.093 DEBUG:boto:Data: 2023-09-22T21:36:12.093 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.093 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.093 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.093 DEBUG:boto:Params: {} 2023-09-22T21:36:12.093 DEBUG:boto:Token: None 2023-09-22T21:36:12.094 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.094 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YRN1c5xoARd+ezNxNo67AExyzaE= 2023-09-22T21:36:12.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YRN1c5xoARd+ezNxNo67AExyzaE=', 'Content-Length': '0'} 2023-09-22T21:36:12.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a940eec52716de7-00650e08cc-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.138 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.139 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.139 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.139 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.139 DEBUG:boto:Method: GET 2023-09-22T21:36:12.139 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.139 DEBUG:boto:Data: 2023-09-22T21:36:12.139 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.139 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.140 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.140 DEBUG:boto:Params: {} 2023-09-22T21:36:12.140 DEBUG:boto:Token: None 2023-09-22T21:36:12.140 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000005?versionId=null 2023-09-22T21:36:12.140 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YRN1c5xoARd+ezNxNo67AExyzaE= 2023-09-22T21:36:12.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YRN1c5xoARd+ezNxNo67AExyzaE=', 'Content-Length': '0'} 2023-09-22T21:36:12.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000237c423057ca5991-00650e08cc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.143 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:12.144 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:12.144 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:36:12.144 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:36:12.144 DEBUG:boto:Method: GET 2023-09-22T21:36:12.144 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:36:12.144 DEBUG:boto:Data: 2023-09-22T21:36:12.144 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.144 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.144 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.144 DEBUG:boto:Params: {} 2023-09-22T21:36:12.145 DEBUG:boto:Token: None 2023-09-22T21:36:12.145 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/ 2023-09-22T21:36:12.145 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8= 2023-09-22T21:36:12.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8=', 'Content-Length': '0'} 2023-09-22T21:36:12.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9d8d5c9ee179a55-00650e08cc-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.190 DEBUG:boto:b'ffzdqu-58obj000000000000000000051000falseobj000000000000000000052023-09-22T21:35:11.989Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:12.191 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:12.191 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:12.191 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:36:12.191 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:36:12.191 DEBUG:boto:Method: GET 2023-09-22T21:36:12.191 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:36:12.192 DEBUG:boto:Data: 2023-09-22T21:36:12.192 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.192 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.192 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.192 DEBUG:boto:Params: {} 2023-09-22T21:36:12.192 DEBUG:boto:Token: None 2023-09-22T21:36:12.192 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/ 2023-09-22T21:36:12.192 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8= 2023-09-22T21:36:12.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8=', 'Content-Length': '0'} 2023-09-22T21:36:12.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093b5cd403a97b933-00650e08cc-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.203 DEBUG:boto:b'ffzdqu-58obj000000000000000000051000falseobj000000000000000000052023-09-22T21:35:11.989Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:12.204 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.204 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.204 DEBUG:boto:Method: HEAD 2023-09-22T21:36:12.204 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.204 DEBUG:boto:Data: 2023-09-22T21:36:12.204 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.204 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.205 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.205 DEBUG:boto:Params: {} 2023-09-22T21:36:12.205 DEBUG:boto:Token: None 2023-09-22T21:36:12.205 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.205 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fS9nMYibtjgqLrA4aLFZ2Zrdv8g= 2023-09-22T21:36:12.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fS9nMYibtjgqLrA4aLFZ2Zrdv8g=', 'Content-Length': '0'} 2023-09-22T21:36:12.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a852991a6c3b8b01-00650e08cc-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.208 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.208 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.208 DEBUG:boto:Method: HEAD 2023-09-22T21:36:12.209 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.209 DEBUG:boto:Data: 2023-09-22T21:36:12.209 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.209 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.209 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.209 DEBUG:boto:Params: {} 2023-09-22T21:36:12.209 DEBUG:boto:Token: None 2023-09-22T21:36:12.209 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.209 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fS9nMYibtjgqLrA4aLFZ2Zrdv8g= 2023-09-22T21:36:12.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fS9nMYibtjgqLrA4aLFZ2Zrdv8g=', 'Content-Length': '0'} 2023-09-22T21:36:12.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052b77fd44f760c5e-00650e08cc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-09-22T21:36:12.213 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.213 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.213 DEBUG:boto:Method: GET 2023-09-22T21:36:12.213 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.213 DEBUG:boto:Data: 2023-09-22T21:36:12.213 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.213 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.213 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.214 DEBUG:boto:Params: {} 2023-09-22T21:36:12.214 DEBUG:boto:Token: None 2023-09-22T21:36:12.214 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.214 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JHryR1t1KU4Lk7udWjGsqK2u6P8= 2023-09-22T21:36:12.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JHryR1t1KU4Lk7udWjGsqK2u6P8=', 'Content-Length': '0'} 2023-09-22T21:36:12.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089f49aeb8bfda39f-00650e08cc-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.262 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.263 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.263 DEBUG:boto:Method: GET 2023-09-22T21:36:12.263 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.263 DEBUG:boto:Data: 2023-09-22T21:36:12.263 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.263 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.263 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.263 DEBUG:boto:Params: {} 2023-09-22T21:36:12.263 DEBUG:boto:Token: None 2023-09-22T21:36:12.264 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000005 2023-09-22T21:36:12.264 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JHryR1t1KU4Lk7udWjGsqK2u6P8= 2023-09-22T21:36:12.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JHryR1t1KU4Lk7udWjGsqK2u6P8=', 'Content-Length': '0'} 2023-09-22T21:36:12.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c610b640ce1b0bcb-00650e08cc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.267 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:12.267 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:12.267 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:36:12.267 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:36:12.268 DEBUG:boto:Method: GET 2023-09-22T21:36:12.268 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:36:12.268 DEBUG:boto:Data: 2023-09-22T21:36:12.268 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.268 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.268 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.268 DEBUG:boto:Params: {} 2023-09-22T21:36:12.268 DEBUG:boto:Token: None 2023-09-22T21:36:12.268 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/ 2023-09-22T21:36:12.269 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8= 2023-09-22T21:36:12.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8=', 'Content-Length': '0'} 2023-09-22T21:36:12.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1abb02236c51342-00650e08cc-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.314 DEBUG:boto:b'ffzdqu-58obj000000000000000000051000falseobj000000000000000000052023-09-22T21:35:11.989Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:12.315 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:12.315 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:12.315 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:36:12.315 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:36:12.315 DEBUG:boto:Method: GET 2023-09-22T21:36:12.315 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000005 2023-09-22T21:36:12.315 DEBUG:boto:Data: 2023-09-22T21:36:12.315 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.316 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.316 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.316 DEBUG:boto:Params: {} 2023-09-22T21:36:12.316 DEBUG:boto:Token: None 2023-09-22T21:36:12.316 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/ 2023-09-22T21:36:12.316 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8= 2023-09-22T21:36:12.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8=', 'Content-Length': '0'} 2023-09-22T21:36:12.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000806f1c3a7b92260b-00650e08cc-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.327 DEBUG:boto:b'ffzdqu-58obj000000000000000000051000falseobj000000000000000000052023-09-22T21:35:11.989Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:12.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-09-22T21:36:12.327 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.327 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.327 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.328 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.328 DEBUG:boto:Method: GET 2023-09-22T21:36:12.328 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.328 DEBUG:boto:Data: 2023-09-22T21:36:12.328 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.328 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.328 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.328 DEBUG:boto:Params: {} 2023-09-22T21:36:12.328 DEBUG:boto:Token: None 2023-09-22T21:36:12.329 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.329 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y6HYpkE/WzWi6LdeSVWUHgZSn5w= 2023-09-22T21:36:12.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y6HYpkE/WzWi6LdeSVWUHgZSn5w=', 'Content-Length': '0'} 2023-09-22T21:36:12.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000985160d5ac15e7a4-00650e08cc-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.332 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.332 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.333 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.333 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.333 DEBUG:boto:Method: GET 2023-09-22T21:36:12.333 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.333 DEBUG:boto:Data: 2023-09-22T21:36:12.333 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.333 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.333 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.333 DEBUG:boto:Params: {} 2023-09-22T21:36:12.334 DEBUG:boto:Token: None 2023-09-22T21:36:12.334 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.334 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y6HYpkE/WzWi6LdeSVWUHgZSn5w= 2023-09-22T21:36:12.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y6HYpkE/WzWi6LdeSVWUHgZSn5w=', 'Content-Length': '0'} 2023-09-22T21:36:12.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc498b740fa1bb80-00650e08cc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.378 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.378 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.379 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.379 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.379 DEBUG:boto:Method: HEAD 2023-09-22T21:36:12.379 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.379 DEBUG:boto:Data: 2023-09-22T21:36:12.379 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.379 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.379 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.379 DEBUG:boto:Params: {} 2023-09-22T21:36:12.380 DEBUG:boto:Token: None 2023-09-22T21:36:12.380 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.380 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7eI5LZk99kv8fJ7Io5OqgwulPbw= 2023-09-22T21:36:12.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7eI5LZk99kv8fJ7Io5OqgwulPbw=', 'Content-Length': '0'} 2023-09-22T21:36:12.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000746219e6d79e7165-00650e08cc-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.383 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.383 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.383 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.383 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.383 DEBUG:boto:Method: HEAD 2023-09-22T21:36:12.384 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.384 DEBUG:boto:Data: 2023-09-22T21:36:12.384 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.384 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.384 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.384 DEBUG:boto:Params: {} 2023-09-22T21:36:12.384 DEBUG:boto:Token: None 2023-09-22T21:36:12.384 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.384 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7eI5LZk99kv8fJ7Io5OqgwulPbw= 2023-09-22T21:36:12.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7eI5LZk99kv8fJ7Io5OqgwulPbw=', 'Content-Length': '0'} 2023-09-22T21:36:12.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004523160df06ed78f-00650e08cc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-09-22T21:36:12.387 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.388 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.388 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.388 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.388 DEBUG:boto:Method: GET 2023-09-22T21:36:12.388 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.388 DEBUG:boto:Data: 2023-09-22T21:36:12.388 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.388 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.388 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.388 DEBUG:boto:Params: {} 2023-09-22T21:36:12.389 DEBUG:boto:Token: None 2023-09-22T21:36:12.389 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.389 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y6HYpkE/WzWi6LdeSVWUHgZSn5w= 2023-09-22T21:36:12.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y6HYpkE/WzWi6LdeSVWUHgZSn5w=', 'Content-Length': '0'} 2023-09-22T21:36:12.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afc8bfb48a23892c-00650e08cc-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.438 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.439 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.439 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.439 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.439 DEBUG:boto:Method: GET 2023-09-22T21:36:12.439 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.439 DEBUG:boto:Data: 2023-09-22T21:36:12.439 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.439 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.439 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.440 DEBUG:boto:Params: {} 2023-09-22T21:36:12.440 DEBUG:boto:Token: None 2023-09-22T21:36:12.440 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000006?versionId=null 2023-09-22T21:36:12.440 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y6HYpkE/WzWi6LdeSVWUHgZSn5w= 2023-09-22T21:36:12.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y6HYpkE/WzWi6LdeSVWUHgZSn5w=', 'Content-Length': '0'} 2023-09-22T21:36:12.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081acd3686c25a49d-00650e08cc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.443 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:12.443 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:12.443 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:36:12.443 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:36:12.443 DEBUG:boto:Method: GET 2023-09-22T21:36:12.444 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:36:12.444 DEBUG:boto:Data: 2023-09-22T21:36:12.444 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.444 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.444 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.444 DEBUG:boto:Params: {} 2023-09-22T21:36:12.444 DEBUG:boto:Token: None 2023-09-22T21:36:12.444 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/ 2023-09-22T21:36:12.444 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8= 2023-09-22T21:36:12.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8=', 'Content-Length': '0'} 2023-09-22T21:36:12.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010fa9e16a7578911-00650e08cc-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.490 DEBUG:boto:b'ffzdqu-58obj000000000000000000061000falseobj000000000000000000062023-09-22T21:35:12.246Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:12.491 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:12.491 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:12.491 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:36:12.491 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:36:12.491 DEBUG:boto:Method: GET 2023-09-22T21:36:12.492 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:36:12.492 DEBUG:boto:Data: 2023-09-22T21:36:12.492 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.492 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.492 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.492 DEBUG:boto:Params: {} 2023-09-22T21:36:12.492 DEBUG:boto:Token: None 2023-09-22T21:36:12.492 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/ 2023-09-22T21:36:12.493 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8= 2023-09-22T21:36:12.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8=', 'Content-Length': '0'} 2023-09-22T21:36:12.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f7d1adddeea2297-00650e08cc-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.502 DEBUG:boto:b'ffzdqu-58obj000000000000000000061000falseobj000000000000000000062023-09-22T21:35:12.246Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:12.503 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.503 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.503 DEBUG:boto:Method: HEAD 2023-09-22T21:36:12.503 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.503 DEBUG:boto:Data: 2023-09-22T21:36:12.503 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.503 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.503 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.504 DEBUG:boto:Params: {} 2023-09-22T21:36:12.504 DEBUG:boto:Token: None 2023-09-22T21:36:12.504 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.504 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cnUgqdXiwmcNx7ZTJCOhrOx7dnU= 2023-09-22T21:36:12.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cnUgqdXiwmcNx7ZTJCOhrOx7dnU=', 'Content-Length': '0'} 2023-09-22T21:36:12.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073140a67851fb6b3-00650e08cc-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.508 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.508 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.508 DEBUG:boto:Method: HEAD 2023-09-22T21:36:12.508 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.508 DEBUG:boto:Data: 2023-09-22T21:36:12.508 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.508 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.508 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.508 DEBUG:boto:Params: {} 2023-09-22T21:36:12.509 DEBUG:boto:Token: None 2023-09-22T21:36:12.509 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.509 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cnUgqdXiwmcNx7ZTJCOhrOx7dnU= 2023-09-22T21:36:12.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cnUgqdXiwmcNx7ZTJCOhrOx7dnU=', 'Content-Length': '0'} 2023-09-22T21:36:12.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ea2d4f68e0d0a02-00650e08cc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-09-22T21:36:12.512 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.512 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.512 DEBUG:boto:Method: GET 2023-09-22T21:36:12.512 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.512 DEBUG:boto:Data: 2023-09-22T21:36:12.513 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.513 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.513 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.513 DEBUG:boto:Params: {} 2023-09-22T21:36:12.513 DEBUG:boto:Token: None 2023-09-22T21:36:12.513 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.513 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JbImVr6rBwKSzMqlFGrEGj0lD+4= 2023-09-22T21:36:12.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JbImVr6rBwKSzMqlFGrEGj0lD+4=', 'Content-Length': '0'} 2023-09-22T21:36:12.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6bc8d2aaec14c98-00650e08cc-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.558 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.558 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.559 DEBUG:boto:Method: GET 2023-09-22T21:36:12.559 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.559 DEBUG:boto:Data: 2023-09-22T21:36:12.559 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.559 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.559 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.559 DEBUG:boto:Params: {} 2023-09-22T21:36:12.559 DEBUG:boto:Token: None 2023-09-22T21:36:12.560 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000006 2023-09-22T21:36:12.560 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JbImVr6rBwKSzMqlFGrEGj0lD+4= 2023-09-22T21:36:12.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JbImVr6rBwKSzMqlFGrEGj0lD+4=', 'Content-Length': '0'} 2023-09-22T21:36:12.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a61d507904cc427-00650e08cc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.563 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:12.563 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:12.563 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:36:12.563 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:36:12.563 DEBUG:boto:Method: GET 2023-09-22T21:36:12.563 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:36:12.563 DEBUG:boto:Data: 2023-09-22T21:36:12.564 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.564 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.564 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.564 DEBUG:boto:Params: {} 2023-09-22T21:36:12.564 DEBUG:boto:Token: None 2023-09-22T21:36:12.564 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/ 2023-09-22T21:36:12.564 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8= 2023-09-22T21:36:12.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8=', 'Content-Length': '0'} 2023-09-22T21:36:12.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd4ea2c6760a76d7-00650e08cc-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.610 DEBUG:boto:b'ffzdqu-58obj000000000000000000061000falseobj000000000000000000062023-09-22T21:35:12.246Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:12.611 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:12.611 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:12.611 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:36:12.611 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:36:12.611 DEBUG:boto:Method: GET 2023-09-22T21:36:12.611 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000006 2023-09-22T21:36:12.612 DEBUG:boto:Data: 2023-09-22T21:36:12.612 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.612 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.612 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.612 DEBUG:boto:Params: {} 2023-09-22T21:36:12.612 DEBUG:boto:Token: None 2023-09-22T21:36:12.612 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/ 2023-09-22T21:36:12.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8= 2023-09-22T21:36:12.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8=', 'Content-Length': '0'} 2023-09-22T21:36:12.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062b763fe612f8eeb-00650e08cc-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.623 DEBUG:boto:b'ffzdqu-58obj000000000000000000061000falseobj000000000000000000062023-09-22T21:35:12.246Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:12.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-09-22T21:36:12.624 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.624 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.624 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.624 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.624 DEBUG:boto:Method: GET 2023-09-22T21:36:12.624 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.624 DEBUG:boto:Data: 2023-09-22T21:36:12.625 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.625 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.625 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.625 DEBUG:boto:Params: {} 2023-09-22T21:36:12.625 DEBUG:boto:Token: None 2023-09-22T21:36:12.625 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.625 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zErzCFh8zvuOBHd6zV8UePl+t2Q= 2023-09-22T21:36:12.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zErzCFh8zvuOBHd6zV8UePl+t2Q=', 'Content-Length': '0'} 2023-09-22T21:36:12.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b77c94c1ca46e91-00650e08cc-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.629 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.629 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.629 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.629 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.629 DEBUG:boto:Method: GET 2023-09-22T21:36:12.629 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.629 DEBUG:boto:Data: 2023-09-22T21:36:12.629 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.629 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.630 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.630 DEBUG:boto:Params: {} 2023-09-22T21:36:12.630 DEBUG:boto:Token: None 2023-09-22T21:36:12.630 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.630 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zErzCFh8zvuOBHd6zV8UePl+t2Q= 2023-09-22T21:36:12.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zErzCFh8zvuOBHd6zV8UePl+t2Q=', 'Content-Length': '0'} 2023-09-22T21:36:12.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d1df97641542c75-00650e08cc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.678 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.679 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.679 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.679 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.679 DEBUG:boto:Method: HEAD 2023-09-22T21:36:12.679 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.679 DEBUG:boto:Data: 2023-09-22T21:36:12.679 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.679 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.679 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.679 DEBUG:boto:Params: {} 2023-09-22T21:36:12.680 DEBUG:boto:Token: None 2023-09-22T21:36:12.680 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.680 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3Oaf3zHyG7KmJ8sK16i9yPLdeAc= 2023-09-22T21:36:12.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3Oaf3zHyG7KmJ8sK16i9yPLdeAc=', 'Content-Length': '0'} 2023-09-22T21:36:12.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021b43eb47a31238b-00650e08cc-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.683 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.683 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.684 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.684 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.684 DEBUG:boto:Method: HEAD 2023-09-22T21:36:12.684 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.684 DEBUG:boto:Data: 2023-09-22T21:36:12.684 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.684 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.684 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.684 DEBUG:boto:Params: {} 2023-09-22T21:36:12.684 DEBUG:boto:Token: None 2023-09-22T21:36:12.685 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.685 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3Oaf3zHyG7KmJ8sK16i9yPLdeAc= 2023-09-22T21:36:12.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3Oaf3zHyG7KmJ8sK16i9yPLdeAc=', 'Content-Length': '0'} 2023-09-22T21:36:12.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012624c729d89051a-00650e08cc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-09-22T21:36:12.688 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.688 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.688 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.688 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.688 DEBUG:boto:Method: GET 2023-09-22T21:36:12.688 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.688 DEBUG:boto:Data: 2023-09-22T21:36:12.689 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.689 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.689 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.689 DEBUG:boto:Params: {} 2023-09-22T21:36:12.689 DEBUG:boto:Token: None 2023-09-22T21:36:12.689 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.689 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zErzCFh8zvuOBHd6zV8UePl+t2Q= 2023-09-22T21:36:12.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zErzCFh8zvuOBHd6zV8UePl+t2Q=', 'Content-Length': '0'} 2023-09-22T21:36:12.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097ac9440a7c284e9-00650e08cc-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.734 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.735 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.735 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.735 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.735 DEBUG:boto:Method: GET 2023-09-22T21:36:12.735 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.735 DEBUG:boto:Data: 2023-09-22T21:36:12.735 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.735 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.735 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.736 DEBUG:boto:Params: {} 2023-09-22T21:36:12.736 DEBUG:boto:Token: None 2023-09-22T21:36:12.736 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000007?versionId=null 2023-09-22T21:36:12.736 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zErzCFh8zvuOBHd6zV8UePl+t2Q= 2023-09-22T21:36:12.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zErzCFh8zvuOBHd6zV8UePl+t2Q=', 'Content-Length': '0'} 2023-09-22T21:36:12.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d87c54bcb350a42e-00650e08cc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.741 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:12.741 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:12.741 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000007 2023-09-22T21:36:12.741 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000007 2023-09-22T21:36:12.741 DEBUG:boto:Method: GET 2023-09-22T21:36:12.741 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000007 2023-09-22T21:36:12.741 DEBUG:boto:Data: 2023-09-22T21:36:12.741 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.741 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.741 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.742 DEBUG:boto:Params: {} 2023-09-22T21:36:12.742 DEBUG:boto:Token: None 2023-09-22T21:36:12.742 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/ 2023-09-22T21:36:12.742 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8= 2023-09-22T21:36:12.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8=', 'Content-Length': '0'} 2023-09-22T21:36:12.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6bc32294014d730-00650e08cc-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.790 DEBUG:boto:b'ffzdqu-58obj000000000000000000071000falseobj000000000000000000072023-09-22T21:35:11.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:12.791 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:12.791 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:12.791 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000007 2023-09-22T21:36:12.791 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000007 2023-09-22T21:36:12.792 DEBUG:boto:Method: GET 2023-09-22T21:36:12.792 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000007 2023-09-22T21:36:12.792 DEBUG:boto:Data: 2023-09-22T21:36:12.792 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.792 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.792 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.792 DEBUG:boto:Params: {} 2023-09-22T21:36:12.792 DEBUG:boto:Token: None 2023-09-22T21:36:12.793 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/ 2023-09-22T21:36:12.793 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8= 2023-09-22T21:36:12.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8=', 'Content-Length': '0'} 2023-09-22T21:36:12.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1603d0f8513581c-00650e08cc-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.803 DEBUG:boto:b'ffzdqu-58obj000000000000000000071000falseobj000000000000000000072023-09-22T21:35:11.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:12.803 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.803 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.803 DEBUG:boto:Method: HEAD 2023-09-22T21:36:12.804 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.804 DEBUG:boto:Data: 2023-09-22T21:36:12.804 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.804 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.804 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.804 DEBUG:boto:Params: {} 2023-09-22T21:36:12.804 DEBUG:boto:Token: None 2023-09-22T21:36:12.804 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.805 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8JPy9w2+QtuRmPQOrhrDluazX8M= 2023-09-22T21:36:12.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8JPy9w2+QtuRmPQOrhrDluazX8M=', 'Content-Length': '0'} 2023-09-22T21:36:12.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b363d5adf1513ee-00650e08cc-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.808 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.808 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.808 DEBUG:boto:Method: HEAD 2023-09-22T21:36:12.809 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.809 DEBUG:boto:Data: 2023-09-22T21:36:12.809 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.809 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.809 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.809 DEBUG:boto:Params: {} 2023-09-22T21:36:12.809 DEBUG:boto:Token: None 2023-09-22T21:36:12.809 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.810 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8JPy9w2+QtuRmPQOrhrDluazX8M= 2023-09-22T21:36:12.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8JPy9w2+QtuRmPQOrhrDluazX8M=', 'Content-Length': '0'} 2023-09-22T21:36:12.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9e0db8e4b5d3523-00650e08cc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-09-22T21:36:12.813 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.813 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.813 DEBUG:boto:Method: GET 2023-09-22T21:36:12.813 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.813 DEBUG:boto:Data: 2023-09-22T21:36:12.813 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.814 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.814 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.814 DEBUG:boto:Params: {} 2023-09-22T21:36:12.814 DEBUG:boto:Token: None 2023-09-22T21:36:12.814 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.814 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4NvCP9nnM4UMsykbcAojUhDJJEE= 2023-09-22T21:36:12.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4NvCP9nnM4UMsykbcAojUhDJJEE=', 'Content-Length': '0'} 2023-09-22T21:36:12.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca8b06203055857e-00650e08cc-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.858 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.858 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.859 DEBUG:boto:Method: GET 2023-09-22T21:36:12.859 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.859 DEBUG:boto:Data: 2023-09-22T21:36:12.859 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.859 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.859 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.859 DEBUG:boto:Params: {} 2023-09-22T21:36:12.859 DEBUG:boto:Token: None 2023-09-22T21:36:12.860 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000007 2023-09-22T21:36:12.860 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4NvCP9nnM4UMsykbcAojUhDJJEE= 2023-09-22T21:36:12.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4NvCP9nnM4UMsykbcAojUhDJJEE=', 'Content-Length': '0'} 2023-09-22T21:36:12.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000601244031e16dca7-00650e08cc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.863 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:12.863 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:12.863 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000007 2023-09-22T21:36:12.863 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000007 2023-09-22T21:36:12.864 DEBUG:boto:Method: GET 2023-09-22T21:36:12.864 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000007 2023-09-22T21:36:12.864 DEBUG:boto:Data: 2023-09-22T21:36:12.864 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.864 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.864 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.864 DEBUG:boto:Params: {} 2023-09-22T21:36:12.864 DEBUG:boto:Token: None 2023-09-22T21:36:12.864 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/ 2023-09-22T21:36:12.865 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8= 2023-09-22T21:36:12.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8=', 'Content-Length': '0'} 2023-09-22T21:36:12.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a66305f2c4b994e-00650e08cc-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.910 DEBUG:boto:b'ffzdqu-58obj000000000000000000071000falseobj000000000000000000072023-09-22T21:35:11.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:12.911 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:12.911 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:12.911 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000007 2023-09-22T21:36:12.911 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000007 2023-09-22T21:36:12.911 DEBUG:boto:Method: GET 2023-09-22T21:36:12.911 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000007 2023-09-22T21:36:12.911 DEBUG:boto:Data: 2023-09-22T21:36:12.912 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.912 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.912 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.912 DEBUG:boto:Params: {} 2023-09-22T21:36:12.912 DEBUG:boto:Token: None 2023-09-22T21:36:12.912 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/ 2023-09-22T21:36:12.912 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8= 2023-09-22T21:36:12.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iBMwlotqEgzzSecWgL7UTJ6I0F8=', 'Content-Length': '0'} 2023-09-22T21:36:12.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8a901bebcc2bd59-00650e08cc-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.923 DEBUG:boto:b'ffzdqu-58obj000000000000000000071000falseobj000000000000000000072023-09-22T21:35:11.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:12.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-09-22T21:36:12.923 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:12.924 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:12.924 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:12.924 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:12.924 DEBUG:boto:Method: GET 2023-09-22T21:36:12.924 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:12.924 DEBUG:boto:Data: 2023-09-22T21:36:12.924 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.924 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.924 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.924 DEBUG:boto:Params: {} 2023-09-22T21:36:12.925 DEBUG:boto:Token: None 2023-09-22T21:36:12.925 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:12.925 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:u7WOPnKpUI/k4F2Yn6PVXZMEMGY= 2023-09-22T21:36:12.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:u7WOPnKpUI/k4F2Yn6PVXZMEMGY=', 'Content-Length': '0'} 2023-09-22T21:36:12.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007286799e6b1d0108-00650e08cc-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.928 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:12.929 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:12.929 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:12.929 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:12.929 DEBUG:boto:Method: GET 2023-09-22T21:36:12.929 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:12.929 DEBUG:boto:Data: 2023-09-22T21:36:12.929 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.929 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.929 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.929 DEBUG:boto:Params: {} 2023-09-22T21:36:12.930 DEBUG:boto:Token: None 2023-09-22T21:36:12.930 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:12.930 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:u7WOPnKpUI/k4F2Yn6PVXZMEMGY= 2023-09-22T21:36:12.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:u7WOPnKpUI/k4F2Yn6PVXZMEMGY=', 'Content-Length': '0'} 2023-09-22T21:36:12.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055f33c6dd03d208e-00650e08cc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.974 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:12.974 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:12.975 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:12.975 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:12.975 DEBUG:boto:Method: HEAD 2023-09-22T21:36:12.975 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:12.975 DEBUG:boto:Data: 2023-09-22T21:36:12.975 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.975 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.975 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.975 DEBUG:boto:Params: {} 2023-09-22T21:36:12.976 DEBUG:boto:Token: None 2023-09-22T21:36:12.976 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:12.976 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CfUvB/Wp72zfbEaHrzKSAOKHKrI= 2023-09-22T21:36:12.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CfUvB/Wp72zfbEaHrzKSAOKHKrI=', 'Content-Length': '0'} 2023-09-22T21:36:12.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004de6d13621c8721c-00650e08cc-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.980 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:12.980 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:12.980 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:12.980 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:12.980 DEBUG:boto:Method: HEAD 2023-09-22T21:36:12.980 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:12.980 DEBUG:boto:Data: 2023-09-22T21:36:12.980 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.980 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:12.981 DEBUG:boto:Port: 8000 2023-09-22T21:36:12.981 DEBUG:boto:Params: {} 2023-09-22T21:36:12.981 DEBUG:boto:Token: None 2023-09-22T21:36:12.981 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:12.981 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CfUvB/Wp72zfbEaHrzKSAOKHKrI= 2023-09-22T21:36:12.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CfUvB/Wp72zfbEaHrzKSAOKHKrI=', 'Content-Length': '0'} 2023-09-22T21:36:12.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f524d9ceb81fb06d-00650e08cc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:12.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-09-22T21:36:12.984 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:12.984 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:12.984 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:12.984 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:12.985 DEBUG:boto:Method: GET 2023-09-22T21:36:12.985 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:12.985 DEBUG:boto:Data: 2023-09-22T21:36:12.985 DEBUG:boto:Headers: {} 2023-09-22T21:36:12.985 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:12.985 DEBUG:boto:Port: 8001 2023-09-22T21:36:12.985 DEBUG:boto:Params: {} 2023-09-22T21:36:12.985 DEBUG:boto:Token: None 2023-09-22T21:36:12.985 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:12 GMT /ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:12.986 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:u7WOPnKpUI/k4F2Yn6PVXZMEMGY= 2023-09-22T21:36:12.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:u7WOPnKpUI/k4F2Yn6PVXZMEMGY=', 'Content-Length': '0'} 2023-09-22T21:36:12.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dd7f863848a0db4-00650e08cc-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.030 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:13.031 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:13.031 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:13.031 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:13.031 DEBUG:boto:Method: GET 2023-09-22T21:36:13.031 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:13.031 DEBUG:boto:Data: 2023-09-22T21:36:13.031 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.031 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.031 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.032 DEBUG:boto:Params: {} 2023-09-22T21:36:13.032 DEBUG:boto:Token: None 2023-09-22T21:36:13.032 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000008?versionId=null 2023-09-22T21:36:13.032 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S8S4eOjVkIzhoM8Jam474Ya09NQ= 2023-09-22T21:36:13.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S8S4eOjVkIzhoM8Jam474Ya09NQ=', 'Content-Length': '0'} 2023-09-22T21:36:13.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005f3c67e770a63d2-00650e08cd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.035 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:13.035 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:13.036 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000008 2023-09-22T21:36:13.036 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000008 2023-09-22T21:36:13.036 DEBUG:boto:Method: GET 2023-09-22T21:36:13.036 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000008 2023-09-22T21:36:13.036 DEBUG:boto:Data: 2023-09-22T21:36:13.036 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.036 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.036 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.036 DEBUG:boto:Params: {} 2023-09-22T21:36:13.037 DEBUG:boto:Token: None 2023-09-22T21:36:13.037 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/ 2023-09-22T21:36:13.037 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E= 2023-09-22T21:36:13.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E=', 'Content-Length': '0'} 2023-09-22T21:36:13.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079b49cda9a7b2d63-00650e08cd-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.082 DEBUG:boto:b'ffzdqu-58obj000000000000000000081000falseobj000000000000000000082023-09-22T21:35:12.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:13.083 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:13.083 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:13.083 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000008 2023-09-22T21:36:13.083 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000008 2023-09-22T21:36:13.083 DEBUG:boto:Method: GET 2023-09-22T21:36:13.084 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000008 2023-09-22T21:36:13.084 DEBUG:boto:Data: 2023-09-22T21:36:13.084 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.084 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.084 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.084 DEBUG:boto:Params: {} 2023-09-22T21:36:13.084 DEBUG:boto:Token: None 2023-09-22T21:36:13.084 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/ 2023-09-22T21:36:13.085 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E= 2023-09-22T21:36:13.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E=', 'Content-Length': '0'} 2023-09-22T21:36:13.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ade6bf9358be9ab4-00650e08cd-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.096 DEBUG:boto:b'ffzdqu-58obj000000000000000000081000falseobj000000000000000000082023-09-22T21:35:12.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:13.096 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:13.096 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:13.096 DEBUG:boto:Method: HEAD 2023-09-22T21:36:13.097 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:13.097 DEBUG:boto:Data: 2023-09-22T21:36:13.097 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.097 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.097 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.097 DEBUG:boto:Params: {} 2023-09-22T21:36:13.097 DEBUG:boto:Token: None 2023-09-22T21:36:13.097 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:13.098 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eSKH++03wMYvqwiFH6wCxIXhrfM= 2023-09-22T21:36:13.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eSKH++03wMYvqwiFH6wCxIXhrfM=', 'Content-Length': '0'} 2023-09-22T21:36:13.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4d968972760db0d-00650e08cd-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.101 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:13.101 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:13.101 DEBUG:boto:Method: HEAD 2023-09-22T21:36:13.101 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:13.101 DEBUG:boto:Data: 2023-09-22T21:36:13.101 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.102 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.102 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.102 DEBUG:boto:Params: {} 2023-09-22T21:36:13.102 DEBUG:boto:Token: None 2023-09-22T21:36:13.102 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:13.102 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eSKH++03wMYvqwiFH6wCxIXhrfM= 2023-09-22T21:36:13.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eSKH++03wMYvqwiFH6wCxIXhrfM=', 'Content-Length': '0'} 2023-09-22T21:36:13.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093a817acb2b6c202-00650e08cd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-09-22T21:36:13.105 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:13.105 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:13.106 DEBUG:boto:Method: GET 2023-09-22T21:36:13.106 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:13.106 DEBUG:boto:Data: 2023-09-22T21:36:13.106 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.106 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.106 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.106 DEBUG:boto:Params: {} 2023-09-22T21:36:13.106 DEBUG:boto:Token: None 2023-09-22T21:36:13.107 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:13.107 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q33wjXNR7uxwIKp3A4GF53jrcEI= 2023-09-22T21:36:13.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q33wjXNR7uxwIKp3A4GF53jrcEI=', 'Content-Length': '0'} 2023-09-22T21:36:13.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005374b1695685a132-00650e08cd-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.154 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:13.155 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:13.155 DEBUG:boto:Method: GET 2023-09-22T21:36:13.155 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:13.155 DEBUG:boto:Data: 2023-09-22T21:36:13.155 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.155 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.155 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.155 DEBUG:boto:Params: {} 2023-09-22T21:36:13.156 DEBUG:boto:Token: None 2023-09-22T21:36:13.156 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000008 2023-09-22T21:36:13.156 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q33wjXNR7uxwIKp3A4GF53jrcEI= 2023-09-22T21:36:13.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q33wjXNR7uxwIKp3A4GF53jrcEI=', 'Content-Length': '0'} 2023-09-22T21:36:13.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c508892652cddba-00650e08cd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.159 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:13.159 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:13.159 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000008 2023-09-22T21:36:13.159 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000008 2023-09-22T21:36:13.159 DEBUG:boto:Method: GET 2023-09-22T21:36:13.160 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000008 2023-09-22T21:36:13.160 DEBUG:boto:Data: 2023-09-22T21:36:13.160 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.160 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.160 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.160 DEBUG:boto:Params: {} 2023-09-22T21:36:13.160 DEBUG:boto:Token: None 2023-09-22T21:36:13.160 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/ 2023-09-22T21:36:13.160 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E= 2023-09-22T21:36:13.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E=', 'Content-Length': '0'} 2023-09-22T21:36:13.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c1c819df69212a0-00650e08cd-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.206 DEBUG:boto:b'ffzdqu-58obj000000000000000000081000falseobj000000000000000000082023-09-22T21:35:12.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:13.207 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:13.207 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:13.207 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000008 2023-09-22T21:36:13.207 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000008 2023-09-22T21:36:13.207 DEBUG:boto:Method: GET 2023-09-22T21:36:13.207 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000008 2023-09-22T21:36:13.208 DEBUG:boto:Data: 2023-09-22T21:36:13.208 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.208 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.208 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.208 DEBUG:boto:Params: {} 2023-09-22T21:36:13.208 DEBUG:boto:Token: None 2023-09-22T21:36:13.208 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/ 2023-09-22T21:36:13.208 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E= 2023-09-22T21:36:13.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E=', 'Content-Length': '0'} 2023-09-22T21:36:13.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e88109cb3d53764d-00650e08cd-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.219 DEBUG:boto:b'ffzdqu-58obj000000000000000000081000falseobj000000000000000000082023-09-22T21:35:12.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:13.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-09-22T21:36:13.219 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.219 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.219 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.219 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.220 DEBUG:boto:Method: GET 2023-09-22T21:36:13.220 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.220 DEBUG:boto:Data: 2023-09-22T21:36:13.220 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.220 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.220 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.220 DEBUG:boto:Params: {} 2023-09-22T21:36:13.220 DEBUG:boto:Token: None 2023-09-22T21:36:13.221 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.221 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+FftO1hA9gahFSWUHkR4dVfvw2s= 2023-09-22T21:36:13.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+FftO1hA9gahFSWUHkR4dVfvw2s=', 'Content-Length': '0'} 2023-09-22T21:36:13.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c8b98772f5f97f9-00650e08cd-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.266 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.267 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.267 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.267 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.267 DEBUG:boto:Method: GET 2023-09-22T21:36:13.267 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.267 DEBUG:boto:Data: 2023-09-22T21:36:13.267 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.267 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.267 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.267 DEBUG:boto:Params: {} 2023-09-22T21:36:13.268 DEBUG:boto:Token: None 2023-09-22T21:36:13.268 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.268 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+FftO1hA9gahFSWUHkR4dVfvw2s= 2023-09-22T21:36:13.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+FftO1hA9gahFSWUHkR4dVfvw2s=', 'Content-Length': '0'} 2023-09-22T21:36:13.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000664114bedc6f2bb1-00650e08cd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.271 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.271 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.271 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.271 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.271 DEBUG:boto:Method: HEAD 2023-09-22T21:36:13.272 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.272 DEBUG:boto:Data: 2023-09-22T21:36:13.272 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.272 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.272 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.272 DEBUG:boto:Params: {} 2023-09-22T21:36:13.272 DEBUG:boto:Token: None 2023-09-22T21:36:13.272 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.272 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NapoWgTxUfa4tv7Jku9PTNAgCas= 2023-09-22T21:36:13.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NapoWgTxUfa4tv7Jku9PTNAgCas=', 'Content-Length': '0'} 2023-09-22T21:36:13.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfed2efb999dff06-00650e08cd-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.276 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.276 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.276 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.276 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.276 DEBUG:boto:Method: HEAD 2023-09-22T21:36:13.276 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.276 DEBUG:boto:Data: 2023-09-22T21:36:13.276 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.276 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.277 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.277 DEBUG:boto:Params: {} 2023-09-22T21:36:13.277 DEBUG:boto:Token: None 2023-09-22T21:36:13.277 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.277 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NapoWgTxUfa4tv7Jku9PTNAgCas= 2023-09-22T21:36:13.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NapoWgTxUfa4tv7Jku9PTNAgCas=', 'Content-Length': '0'} 2023-09-22T21:36:13.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fe330c0399f5df9-00650e08cd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-09-22T21:36:13.280 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.280 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.281 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.281 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.281 DEBUG:boto:Method: GET 2023-09-22T21:36:13.281 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.281 DEBUG:boto:Data: 2023-09-22T21:36:13.281 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.281 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.281 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.281 DEBUG:boto:Params: {} 2023-09-22T21:36:13.281 DEBUG:boto:Token: None 2023-09-22T21:36:13.282 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.282 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+FftO1hA9gahFSWUHkR4dVfvw2s= 2023-09-22T21:36:13.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+FftO1hA9gahFSWUHkR4dVfvw2s=', 'Content-Length': '0'} 2023-09-22T21:36:13.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002edcf0231ac0067b-00650e08cd-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.326 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.327 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.327 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.327 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.327 DEBUG:boto:Method: GET 2023-09-22T21:36:13.327 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.327 DEBUG:boto:Data: 2023-09-22T21:36:13.327 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.327 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.328 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.328 DEBUG:boto:Params: {} 2023-09-22T21:36:13.328 DEBUG:boto:Token: None 2023-09-22T21:36:13.328 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000009?versionId=null 2023-09-22T21:36:13.328 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+FftO1hA9gahFSWUHkR4dVfvw2s= 2023-09-22T21:36:13.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+FftO1hA9gahFSWUHkR4dVfvw2s=', 'Content-Length': '0'} 2023-09-22T21:36:13.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddbdf248cbe6d80a-00650e08cd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.331 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:13.332 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:13.332 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000009 2023-09-22T21:36:13.332 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000009 2023-09-22T21:36:13.332 DEBUG:boto:Method: GET 2023-09-22T21:36:13.332 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000009 2023-09-22T21:36:13.332 DEBUG:boto:Data: 2023-09-22T21:36:13.332 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.332 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.332 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.332 DEBUG:boto:Params: {} 2023-09-22T21:36:13.333 DEBUG:boto:Token: None 2023-09-22T21:36:13.333 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/ 2023-09-22T21:36:13.333 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E= 2023-09-22T21:36:13.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E=', 'Content-Length': '0'} 2023-09-22T21:36:13.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d7bb3a372b0504e-00650e08cd-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.378 DEBUG:boto:b'ffzdqu-58obj000000000000000000091000falseobj000000000000000000092023-09-22T21:35:05.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:13.379 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:13.379 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:13.379 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000009 2023-09-22T21:36:13.379 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000009 2023-09-22T21:36:13.379 DEBUG:boto:Method: GET 2023-09-22T21:36:13.380 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000009 2023-09-22T21:36:13.380 DEBUG:boto:Data: 2023-09-22T21:36:13.380 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.380 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.380 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.380 DEBUG:boto:Params: {} 2023-09-22T21:36:13.380 DEBUG:boto:Token: None 2023-09-22T21:36:13.380 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/ 2023-09-22T21:36:13.380 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E= 2023-09-22T21:36:13.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E=', 'Content-Length': '0'} 2023-09-22T21:36:13.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003119d3fbe8f1f081-00650e08cd-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.391 DEBUG:boto:b'ffzdqu-58obj000000000000000000091000falseobj000000000000000000092023-09-22T21:35:05.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:13.392 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.392 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.392 DEBUG:boto:Method: HEAD 2023-09-22T21:36:13.392 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.392 DEBUG:boto:Data: 2023-09-22T21:36:13.392 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.392 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.392 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.392 DEBUG:boto:Params: {} 2023-09-22T21:36:13.393 DEBUG:boto:Token: None 2023-09-22T21:36:13.393 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.393 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:az2sHE0Ee1kgNeVFMINDb+8B+Wk= 2023-09-22T21:36:13.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:az2sHE0Ee1kgNeVFMINDb+8B+Wk=', 'Content-Length': '0'} 2023-09-22T21:36:13.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d9424453b6427b5-00650e08cd-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.396 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.396 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.396 DEBUG:boto:Method: HEAD 2023-09-22T21:36:13.396 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.397 DEBUG:boto:Data: 2023-09-22T21:36:13.397 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.397 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.397 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.397 DEBUG:boto:Params: {} 2023-09-22T21:36:13.397 DEBUG:boto:Token: None 2023-09-22T21:36:13.397 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.398 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:az2sHE0Ee1kgNeVFMINDb+8B+Wk= 2023-09-22T21:36:13.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:az2sHE0Ee1kgNeVFMINDb+8B+Wk=', 'Content-Length': '0'} 2023-09-22T21:36:13.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071cf938c1e830773-00650e08cd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-09-22T21:36:13.401 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.401 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.401 DEBUG:boto:Method: GET 2023-09-22T21:36:13.401 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.401 DEBUG:boto:Data: 2023-09-22T21:36:13.401 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.401 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.401 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.402 DEBUG:boto:Params: {} 2023-09-22T21:36:13.402 DEBUG:boto:Token: None 2023-09-22T21:36:13.402 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.402 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mo2DQrvA300rDGxRx96nlajc7Os= 2023-09-22T21:36:13.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mo2DQrvA300rDGxRx96nlajc7Os=', 'Content-Length': '0'} 2023-09-22T21:36:13.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f69d18a3ad1da00-00650e08cd-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.446 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.447 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.447 DEBUG:boto:Method: GET 2023-09-22T21:36:13.447 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.447 DEBUG:boto:Data: 2023-09-22T21:36:13.447 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.447 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.447 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.447 DEBUG:boto:Params: {} 2023-09-22T21:36:13.448 DEBUG:boto:Token: None 2023-09-22T21:36:13.448 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000009 2023-09-22T21:36:13.448 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mo2DQrvA300rDGxRx96nlajc7Os= 2023-09-22T21:36:13.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mo2DQrvA300rDGxRx96nlajc7Os=', 'Content-Length': '0'} 2023-09-22T21:36:13.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3c272f9570d5c23-00650e08cd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.451 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:13.451 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:13.451 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000009 2023-09-22T21:36:13.452 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000009 2023-09-22T21:36:13.452 DEBUG:boto:Method: GET 2023-09-22T21:36:13.452 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000009 2023-09-22T21:36:13.452 DEBUG:boto:Data: 2023-09-22T21:36:13.452 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.452 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.452 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.452 DEBUG:boto:Params: {} 2023-09-22T21:36:13.453 DEBUG:boto:Token: None 2023-09-22T21:36:13.453 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/ 2023-09-22T21:36:13.453 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E= 2023-09-22T21:36:13.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E=', 'Content-Length': '0'} 2023-09-22T21:36:13.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae4b3f016300bae7-00650e08cd-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.502 DEBUG:boto:b'ffzdqu-58obj000000000000000000091000falseobj000000000000000000092023-09-22T21:35:05.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:13.503 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:13.503 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:13.503 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000009 2023-09-22T21:36:13.503 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000009 2023-09-22T21:36:13.503 DEBUG:boto:Method: GET 2023-09-22T21:36:13.504 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000009 2023-09-22T21:36:13.504 DEBUG:boto:Data: 2023-09-22T21:36:13.504 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.504 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.504 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.504 DEBUG:boto:Params: {} 2023-09-22T21:36:13.504 DEBUG:boto:Token: None 2023-09-22T21:36:13.504 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/ 2023-09-22T21:36:13.504 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E= 2023-09-22T21:36:13.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E=', 'Content-Length': '0'} 2023-09-22T21:36:13.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084f43206e4321d4b-00650e08cd-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.515 DEBUG:boto:b'ffzdqu-58obj000000000000000000091000falseobj000000000000000000092023-09-22T21:35:05.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:13.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-09-22T21:36:13.515 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.515 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.516 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.516 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.516 DEBUG:boto:Method: GET 2023-09-22T21:36:13.516 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.516 DEBUG:boto:Data: 2023-09-22T21:36:13.516 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.516 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.516 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.516 DEBUG:boto:Params: {} 2023-09-22T21:36:13.517 DEBUG:boto:Token: None 2023-09-22T21:36:13.517 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.517 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rCsax65lLgA/y6hoXzQ+GOohfdo= 2023-09-22T21:36:13.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rCsax65lLgA/y6hoXzQ+GOohfdo=', 'Content-Length': '0'} 2023-09-22T21:36:13.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000100134df3e67b786-00650e08cd-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.520 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.520 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.520 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.520 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.521 DEBUG:boto:Method: GET 2023-09-22T21:36:13.521 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.521 DEBUG:boto:Data: 2023-09-22T21:36:13.521 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.521 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.521 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.521 DEBUG:boto:Params: {} 2023-09-22T21:36:13.521 DEBUG:boto:Token: None 2023-09-22T21:36:13.521 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.522 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rCsax65lLgA/y6hoXzQ+GOohfdo= 2023-09-22T21:36:13.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rCsax65lLgA/y6hoXzQ+GOohfdo=', 'Content-Length': '0'} 2023-09-22T21:36:13.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ff3fcf496faed6f-00650e08cd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.524 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.524 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.525 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.525 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.525 DEBUG:boto:Method: HEAD 2023-09-22T21:36:13.525 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.525 DEBUG:boto:Data: 2023-09-22T21:36:13.525 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.525 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.525 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.525 DEBUG:boto:Params: {} 2023-09-22T21:36:13.526 DEBUG:boto:Token: None 2023-09-22T21:36:13.526 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.526 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qehMBkWJslIElco+Z/3hsX0uhJ4= 2023-09-22T21:36:13.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qehMBkWJslIElco+Z/3hsX0uhJ4=', 'Content-Length': '0'} 2023-09-22T21:36:13.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6714f9d5881b66b-00650e08cd-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.529 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.529 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.529 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.529 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.529 DEBUG:boto:Method: HEAD 2023-09-22T21:36:13.529 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.529 DEBUG:boto:Data: 2023-09-22T21:36:13.529 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.529 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.530 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.530 DEBUG:boto:Params: {} 2023-09-22T21:36:13.530 DEBUG:boto:Token: None 2023-09-22T21:36:13.530 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.530 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qehMBkWJslIElco+Z/3hsX0uhJ4= 2023-09-22T21:36:13.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qehMBkWJslIElco+Z/3hsX0uhJ4=', 'Content-Length': '0'} 2023-09-22T21:36:13.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051462af86646ef4b-00650e08cd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-09-22T21:36:13.533 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.533 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.533 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.533 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.533 DEBUG:boto:Method: GET 2023-09-22T21:36:13.534 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.534 DEBUG:boto:Data: 2023-09-22T21:36:13.534 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.534 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.534 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.534 DEBUG:boto:Params: {} 2023-09-22T21:36:13.534 DEBUG:boto:Token: None 2023-09-22T21:36:13.534 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.535 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rCsax65lLgA/y6hoXzQ+GOohfdo= 2023-09-22T21:36:13.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rCsax65lLgA/y6hoXzQ+GOohfdo=', 'Content-Length': '0'} 2023-09-22T21:36:13.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de0f06e460e25fd0-00650e08cd-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.582 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.583 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.583 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.583 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.583 DEBUG:boto:Method: GET 2023-09-22T21:36:13.583 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.583 DEBUG:boto:Data: 2023-09-22T21:36:13.583 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.583 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.583 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.584 DEBUG:boto:Params: {} 2023-09-22T21:36:13.584 DEBUG:boto:Token: None 2023-09-22T21:36:13.584 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000010?versionId=null 2023-09-22T21:36:13.584 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rCsax65lLgA/y6hoXzQ+GOohfdo= 2023-09-22T21:36:13.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rCsax65lLgA/y6hoXzQ+GOohfdo=', 'Content-Length': '0'} 2023-09-22T21:36:13.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009025b17974b8d905-00650e08cd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.588 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:13.588 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:13.588 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:36:13.588 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:36:13.588 DEBUG:boto:Method: GET 2023-09-22T21:36:13.588 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:36:13.588 DEBUG:boto:Data: 2023-09-22T21:36:13.588 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.588 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.589 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.589 DEBUG:boto:Params: {} 2023-09-22T21:36:13.589 DEBUG:boto:Token: None 2023-09-22T21:36:13.589 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/ 2023-09-22T21:36:13.589 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E= 2023-09-22T21:36:13.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E=', 'Content-Length': '0'} 2023-09-22T21:36:13.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c94b4a09b6a9a235-00650e08cd-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.638 DEBUG:boto:b'ffzdqu-58obj000000000000000000101000falseobj000000000000000000102023-09-22T21:35:05.514Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:13.639 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:13.639 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:13.639 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:36:13.639 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:36:13.639 DEBUG:boto:Method: GET 2023-09-22T21:36:13.639 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:36:13.640 DEBUG:boto:Data: 2023-09-22T21:36:13.640 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.640 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.640 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.640 DEBUG:boto:Params: {} 2023-09-22T21:36:13.640 DEBUG:boto:Token: None 2023-09-22T21:36:13.640 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/ 2023-09-22T21:36:13.640 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E= 2023-09-22T21:36:13.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E=', 'Content-Length': '0'} 2023-09-22T21:36:13.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bfbfb8b488fd9a6-00650e08cd-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.651 DEBUG:boto:b'ffzdqu-58obj000000000000000000101000falseobj000000000000000000102023-09-22T21:35:05.514Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:13.651 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.651 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.651 DEBUG:boto:Method: HEAD 2023-09-22T21:36:13.651 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.652 DEBUG:boto:Data: 2023-09-22T21:36:13.652 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.652 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.652 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.652 DEBUG:boto:Params: {} 2023-09-22T21:36:13.652 DEBUG:boto:Token: None 2023-09-22T21:36:13.652 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.652 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4Tdda0jIgCm/NZnmmy2qLD3cvCg= 2023-09-22T21:36:13.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4Tdda0jIgCm/NZnmmy2qLD3cvCg=', 'Content-Length': '0'} 2023-09-22T21:36:13.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df8926344c355e74-00650e08cd-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.656 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.656 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.656 DEBUG:boto:Method: HEAD 2023-09-22T21:36:13.656 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.656 DEBUG:boto:Data: 2023-09-22T21:36:13.656 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.657 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.657 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.657 DEBUG:boto:Params: {} 2023-09-22T21:36:13.657 DEBUG:boto:Token: None 2023-09-22T21:36:13.657 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.657 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4Tdda0jIgCm/NZnmmy2qLD3cvCg= 2023-09-22T21:36:13.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4Tdda0jIgCm/NZnmmy2qLD3cvCg=', 'Content-Length': '0'} 2023-09-22T21:36:13.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f55b37f41a03d656-00650e08cd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-09-22T21:36:13.660 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.660 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.661 DEBUG:boto:Method: GET 2023-09-22T21:36:13.661 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.661 DEBUG:boto:Data: 2023-09-22T21:36:13.661 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.661 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.661 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.661 DEBUG:boto:Params: {} 2023-09-22T21:36:13.661 DEBUG:boto:Token: None 2023-09-22T21:36:13.661 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.662 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+G2wKt/b3mKkWsSdxELh+yBgBUc= 2023-09-22T21:36:13.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+G2wKt/b3mKkWsSdxELh+yBgBUc=', 'Content-Length': '0'} 2023-09-22T21:36:13.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000468bd808f3fa5c8f-00650e08cd-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.706 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.707 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.707 DEBUG:boto:Method: GET 2023-09-22T21:36:13.707 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.707 DEBUG:boto:Data: 2023-09-22T21:36:13.707 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.707 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.707 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.707 DEBUG:boto:Params: {} 2023-09-22T21:36:13.708 DEBUG:boto:Token: None 2023-09-22T21:36:13.708 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000010 2023-09-22T21:36:13.708 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+G2wKt/b3mKkWsSdxELh+yBgBUc= 2023-09-22T21:36:13.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+G2wKt/b3mKkWsSdxELh+yBgBUc=', 'Content-Length': '0'} 2023-09-22T21:36:13.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a7407a325734cf5-00650e08cd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.711 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:13.712 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:13.712 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:36:13.712 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:36:13.712 DEBUG:boto:Method: GET 2023-09-22T21:36:13.712 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:36:13.712 DEBUG:boto:Data: 2023-09-22T21:36:13.713 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.713 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.713 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.713 DEBUG:boto:Params: {} 2023-09-22T21:36:13.713 DEBUG:boto:Token: None 2023-09-22T21:36:13.714 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/ 2023-09-22T21:36:13.714 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E= 2023-09-22T21:36:13.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E=', 'Content-Length': '0'} 2023-09-22T21:36:13.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d77817eebd02e0e-00650e08cd-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.766 DEBUG:boto:b'ffzdqu-58obj000000000000000000101000falseobj000000000000000000102023-09-22T21:35:05.514Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:13.767 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:13.767 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:13.768 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:36:13.768 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:36:13.768 DEBUG:boto:Method: GET 2023-09-22T21:36:13.768 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000010 2023-09-22T21:36:13.768 DEBUG:boto:Data: 2023-09-22T21:36:13.769 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.769 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.769 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.769 DEBUG:boto:Params: {} 2023-09-22T21:36:13.769 DEBUG:boto:Token: None 2023-09-22T21:36:13.770 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/ 2023-09-22T21:36:13.770 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E= 2023-09-22T21:36:13.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E=', 'Content-Length': '0'} 2023-09-22T21:36:13.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab10ce2c0c5ed960-00650e08cd-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.785 DEBUG:boto:b'ffzdqu-58obj000000000000000000101000falseobj000000000000000000102023-09-22T21:35:05.514Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:13.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-09-22T21:36:13.786 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.786 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.786 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.786 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.786 DEBUG:boto:Method: GET 2023-09-22T21:36:13.786 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.787 DEBUG:boto:Data: 2023-09-22T21:36:13.787 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.787 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.787 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.787 DEBUG:boto:Params: {} 2023-09-22T21:36:13.787 DEBUG:boto:Token: None 2023-09-22T21:36:13.787 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.787 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5FLov9Ad5AjibfkHFQiZd/2I6tE= 2023-09-22T21:36:13.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5FLov9Ad5AjibfkHFQiZd/2I6tE=', 'Content-Length': '0'} 2023-09-22T21:36:13.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb59c386370377ce-00650e08cd-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.791 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.791 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.791 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.791 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.792 DEBUG:boto:Method: GET 2023-09-22T21:36:13.792 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.792 DEBUG:boto:Data: 2023-09-22T21:36:13.792 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.792 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.792 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.792 DEBUG:boto:Params: {} 2023-09-22T21:36:13.793 DEBUG:boto:Token: None 2023-09-22T21:36:13.793 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.793 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5FLov9Ad5AjibfkHFQiZd/2I6tE= 2023-09-22T21:36:13.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5FLov9Ad5AjibfkHFQiZd/2I6tE=', 'Content-Length': '0'} 2023-09-22T21:36:13.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d280dfaacbc3d95-00650e08cd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.796 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.796 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.796 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.796 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.796 DEBUG:boto:Method: HEAD 2023-09-22T21:36:13.797 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.797 DEBUG:boto:Data: 2023-09-22T21:36:13.797 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.797 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.797 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.797 DEBUG:boto:Params: {} 2023-09-22T21:36:13.797 DEBUG:boto:Token: None 2023-09-22T21:36:13.797 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.797 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xLrNioyyxLQiavW9kNGfsL/A1fk= 2023-09-22T21:36:13.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xLrNioyyxLQiavW9kNGfsL/A1fk=', 'Content-Length': '0'} 2023-09-22T21:36:13.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a76b37a8429465b-00650e08cd-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.800 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.800 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.801 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.801 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.801 DEBUG:boto:Method: HEAD 2023-09-22T21:36:13.801 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.801 DEBUG:boto:Data: 2023-09-22T21:36:13.801 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.801 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.801 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.801 DEBUG:boto:Params: {} 2023-09-22T21:36:13.801 DEBUG:boto:Token: None 2023-09-22T21:36:13.802 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.802 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xLrNioyyxLQiavW9kNGfsL/A1fk= 2023-09-22T21:36:13.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xLrNioyyxLQiavW9kNGfsL/A1fk=', 'Content-Length': '0'} 2023-09-22T21:36:13.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d13fe238ad3db4f4-00650e08cd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-09-22T21:36:13.805 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.805 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.805 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.805 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.805 DEBUG:boto:Method: GET 2023-09-22T21:36:13.805 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.805 DEBUG:boto:Data: 2023-09-22T21:36:13.806 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.806 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.806 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.806 DEBUG:boto:Params: {} 2023-09-22T21:36:13.806 DEBUG:boto:Token: None 2023-09-22T21:36:13.806 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.806 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5FLov9Ad5AjibfkHFQiZd/2I6tE= 2023-09-22T21:36:13.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5FLov9Ad5AjibfkHFQiZd/2I6tE=', 'Content-Length': '0'} 2023-09-22T21:36:13.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d32192ae35c71aff-00650e08cd-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.850 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.851 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.851 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.851 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.851 DEBUG:boto:Method: GET 2023-09-22T21:36:13.851 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.851 DEBUG:boto:Data: 2023-09-22T21:36:13.851 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.851 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.852 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.852 DEBUG:boto:Params: {} 2023-09-22T21:36:13.852 DEBUG:boto:Token: None 2023-09-22T21:36:13.852 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000011?versionId=null 2023-09-22T21:36:13.852 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5FLov9Ad5AjibfkHFQiZd/2I6tE= 2023-09-22T21:36:13.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5FLov9Ad5AjibfkHFQiZd/2I6tE=', 'Content-Length': '0'} 2023-09-22T21:36:13.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043016b5198550562-00650e08cd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.856 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:13.856 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:13.856 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:36:13.856 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:36:13.856 DEBUG:boto:Method: GET 2023-09-22T21:36:13.856 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:36:13.856 DEBUG:boto:Data: 2023-09-22T21:36:13.856 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.856 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.857 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.857 DEBUG:boto:Params: {} 2023-09-22T21:36:13.857 DEBUG:boto:Token: None 2023-09-22T21:36:13.857 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/ 2023-09-22T21:36:13.857 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E= 2023-09-22T21:36:13.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E=', 'Content-Length': '0'} 2023-09-22T21:36:13.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e030628f906fa413-00650e08cd-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.862 DEBUG:boto:b'ffzdqu-58obj000000000000000000111000falseobj000000000000000000112023-09-22T21:35:12.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:13.862 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:13.862 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:13.863 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:36:13.863 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:36:13.863 DEBUG:boto:Method: GET 2023-09-22T21:36:13.863 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:36:13.863 DEBUG:boto:Data: 2023-09-22T21:36:13.863 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.863 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.863 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.863 DEBUG:boto:Params: {} 2023-09-22T21:36:13.863 DEBUG:boto:Token: None 2023-09-22T21:36:13.864 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/ 2023-09-22T21:36:13.864 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E= 2023-09-22T21:36:13.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E=', 'Content-Length': '0'} 2023-09-22T21:36:13.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066ad22484698725f-00650e08cd-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.918 DEBUG:boto:b'ffzdqu-58obj000000000000000000111000falseobj000000000000000000112023-09-22T21:35:12.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:13.921 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.921 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.921 DEBUG:boto:Method: HEAD 2023-09-22T21:36:13.921 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.921 DEBUG:boto:Data: 2023-09-22T21:36:13.921 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.921 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.922 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.922 DEBUG:boto:Params: {} 2023-09-22T21:36:13.922 DEBUG:boto:Token: None 2023-09-22T21:36:13.922 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.922 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ahU/S2cdb9zffmchjx/dXug8WbQ= 2023-09-22T21:36:13.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ahU/S2cdb9zffmchjx/dXug8WbQ=', 'Content-Length': '0'} 2023-09-22T21:36:13.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0479595164044ab-00650e08cd-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.925 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.926 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.926 DEBUG:boto:Method: HEAD 2023-09-22T21:36:13.926 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.926 DEBUG:boto:Data: 2023-09-22T21:36:13.926 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.926 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.926 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.926 DEBUG:boto:Params: {} 2023-09-22T21:36:13.926 DEBUG:boto:Token: None 2023-09-22T21:36:13.927 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.927 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ahU/S2cdb9zffmchjx/dXug8WbQ= 2023-09-22T21:36:13.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ahU/S2cdb9zffmchjx/dXug8WbQ=', 'Content-Length': '0'} 2023-09-22T21:36:13.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009396fb906c620c33-00650e08cd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-09-22T21:36:13.930 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.930 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.930 DEBUG:boto:Method: GET 2023-09-22T21:36:13.930 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.931 DEBUG:boto:Data: 2023-09-22T21:36:13.931 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.931 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.931 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.931 DEBUG:boto:Params: {} 2023-09-22T21:36:13.931 DEBUG:boto:Token: None 2023-09-22T21:36:13.931 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.931 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ANSs7mXWUnmnbGM8v2+MUG1/zKc= 2023-09-22T21:36:13.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ANSs7mXWUnmnbGM8v2+MUG1/zKc=', 'Content-Length': '0'} 2023-09-22T21:36:13.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000198cbb6fe698da3b-00650e08cd-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.974 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.975 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.975 DEBUG:boto:Method: GET 2023-09-22T21:36:13.975 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.975 DEBUG:boto:Data: 2023-09-22T21:36:13.975 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.975 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:13.975 DEBUG:boto:Port: 8000 2023-09-22T21:36:13.975 DEBUG:boto:Params: {} 2023-09-22T21:36:13.975 DEBUG:boto:Token: None 2023-09-22T21:36:13.976 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/obj00000000000000000011 2023-09-22T21:36:13.976 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ANSs7mXWUnmnbGM8v2+MUG1/zKc= 2023-09-22T21:36:13.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ANSs7mXWUnmnbGM8v2+MUG1/zKc=', 'Content-Length': '0'} 2023-09-22T21:36:13.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc874f1d3c9ccc31-00650e08cd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:13.979 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:13.979 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:13.979 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:36:13.979 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:36:13.979 DEBUG:boto:Method: GET 2023-09-22T21:36:13.980 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:36:13.980 DEBUG:boto:Data: 2023-09-22T21:36:13.980 DEBUG:boto:Headers: {} 2023-09-22T21:36:13.980 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:13.980 DEBUG:boto:Port: 8001 2023-09-22T21:36:13.980 DEBUG:boto:Params: {} 2023-09-22T21:36:13.980 DEBUG:boto:Token: None 2023-09-22T21:36:13.980 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:13 GMT /ffzdqu-58/ 2023-09-22T21:36:13.981 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E= 2023-09-22T21:36:13.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+JqQ8HEGchPgXE2R+SkHAl5tj6E=', 'Content-Length': '0'} 2023-09-22T21:36:13.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028ca9a277773730e-00650e08cd-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.026 DEBUG:boto:b'ffzdqu-58obj000000000000000000111000falseobj000000000000000000112023-09-22T21:35:12.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:14.027 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:14.027 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:14.027 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:36:14.027 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:36:14.027 DEBUG:boto:Method: GET 2023-09-22T21:36:14.027 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000011 2023-09-22T21:36:14.028 DEBUG:boto:Data: 2023-09-22T21:36:14.028 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.028 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:14.028 DEBUG:boto:Port: 8000 2023-09-22T21:36:14.028 DEBUG:boto:Params: {} 2023-09-22T21:36:14.028 DEBUG:boto:Token: None 2023-09-22T21:36:14.028 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/ 2023-09-22T21:36:14.028 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8= 2023-09-22T21:36:14.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8=', 'Content-Length': '0'} 2023-09-22T21:36:14.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcc42fcbb52ea7e8-00650e08ce-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.038 DEBUG:boto:b'ffzdqu-58obj000000000000000000111000falseobj000000000000000000112023-09-22T21:35:12.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:14.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-09-22T21:36:14.039 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.039 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.039 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.039 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.039 DEBUG:boto:Method: GET 2023-09-22T21:36:14.039 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.040 DEBUG:boto:Data: 2023-09-22T21:36:14.040 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.040 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:14.040 DEBUG:boto:Port: 8001 2023-09-22T21:36:14.040 DEBUG:boto:Params: {} 2023-09-22T21:36:14.040 DEBUG:boto:Token: None 2023-09-22T21:36:14.040 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.040 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:67PStSQNDsa+hpnDWelmhU69+XM= 2023-09-22T21:36:14.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:67PStSQNDsa+hpnDWelmhU69+XM=', 'Content-Length': '0'} 2023-09-22T21:36:14.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3a383a2b9a38da2-00650e08ce-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.044 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.044 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.044 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.044 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.044 DEBUG:boto:Method: GET 2023-09-22T21:36:14.045 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.045 DEBUG:boto:Data: 2023-09-22T21:36:14.045 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.045 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:14.045 DEBUG:boto:Port: 8000 2023-09-22T21:36:14.045 DEBUG:boto:Params: {} 2023-09-22T21:36:14.045 DEBUG:boto:Token: None 2023-09-22T21:36:14.045 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.045 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:67PStSQNDsa+hpnDWelmhU69+XM= 2023-09-22T21:36:14.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:67PStSQNDsa+hpnDWelmhU69+XM=', 'Content-Length': '0'} 2023-09-22T21:36:14.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023cddc1cb124ddc2-00650e08ce-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.049 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.049 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.049 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.049 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.049 DEBUG:boto:Method: HEAD 2023-09-22T21:36:14.049 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.049 DEBUG:boto:Data: 2023-09-22T21:36:14.050 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.050 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:14.050 DEBUG:boto:Port: 8001 2023-09-22T21:36:14.050 DEBUG:boto:Params: {} 2023-09-22T21:36:14.050 DEBUG:boto:Token: None 2023-09-22T21:36:14.050 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.051 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gV+vZSY5qXTQfPzVigGbCmiNRGQ= 2023-09-22T21:36:14.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gV+vZSY5qXTQfPzVigGbCmiNRGQ=', 'Content-Length': '0'} 2023-09-22T21:36:14.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e7e42962c940cff-00650e08ce-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.054 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.054 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.054 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.054 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.054 DEBUG:boto:Method: HEAD 2023-09-22T21:36:14.055 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.055 DEBUG:boto:Data: 2023-09-22T21:36:14.055 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.055 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:14.055 DEBUG:boto:Port: 8000 2023-09-22T21:36:14.055 DEBUG:boto:Params: {} 2023-09-22T21:36:14.055 DEBUG:boto:Token: None 2023-09-22T21:36:14.055 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.055 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gV+vZSY5qXTQfPzVigGbCmiNRGQ= 2023-09-22T21:36:14.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gV+vZSY5qXTQfPzVigGbCmiNRGQ=', 'Content-Length': '0'} 2023-09-22T21:36:14.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8b1a2f669c09d1a-00650e08ce-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-09-22T21:36:14.058 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.058 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.059 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.059 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.059 DEBUG:boto:Method: GET 2023-09-22T21:36:14.059 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.059 DEBUG:boto:Data: 2023-09-22T21:36:14.059 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.059 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:14.059 DEBUG:boto:Port: 8001 2023-09-22T21:36:14.059 DEBUG:boto:Params: {} 2023-09-22T21:36:14.060 DEBUG:boto:Token: None 2023-09-22T21:36:14.060 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.060 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:67PStSQNDsa+hpnDWelmhU69+XM= 2023-09-22T21:36:14.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:67PStSQNDsa+hpnDWelmhU69+XM=', 'Content-Length': '0'} 2023-09-22T21:36:14.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018bc12b363ef9d68-00650e08ce-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.106 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.107 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.107 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.107 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.107 DEBUG:boto:Method: GET 2023-09-22T21:36:14.107 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.107 DEBUG:boto:Data: 2023-09-22T21:36:14.107 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.107 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:14.107 DEBUG:boto:Port: 8000 2023-09-22T21:36:14.108 DEBUG:boto:Params: {} 2023-09-22T21:36:14.108 DEBUG:boto:Token: None 2023-09-22T21:36:14.108 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000012?versionId=null 2023-09-22T21:36:14.108 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:67PStSQNDsa+hpnDWelmhU69+XM= 2023-09-22T21:36:14.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:67PStSQNDsa+hpnDWelmhU69+XM=', 'Content-Length': '0'} 2023-09-22T21:36:14.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cbaf110776685fb-00650e08ce-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.111 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:14.112 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:14.112 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:36:14.112 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:36:14.112 DEBUG:boto:Method: GET 2023-09-22T21:36:14.112 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:36:14.112 DEBUG:boto:Data: 2023-09-22T21:36:14.112 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.112 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:14.112 DEBUG:boto:Port: 8001 2023-09-22T21:36:14.112 DEBUG:boto:Params: {} 2023-09-22T21:36:14.113 DEBUG:boto:Token: None 2023-09-22T21:36:14.113 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/ 2023-09-22T21:36:14.113 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8= 2023-09-22T21:36:14.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8=', 'Content-Length': '0'} 2023-09-22T21:36:14.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e08e9899e1634e2-00650e08ce-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.158 DEBUG:boto:b'ffzdqu-58obj000000000000000000121000falseobj000000000000000000122023-09-22T21:35:11.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:14.159 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:14.159 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:14.159 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:36:14.159 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:36:14.159 DEBUG:boto:Method: GET 2023-09-22T21:36:14.160 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:36:14.160 DEBUG:boto:Data: 2023-09-22T21:36:14.160 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.160 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:14.160 DEBUG:boto:Port: 8000 2023-09-22T21:36:14.160 DEBUG:boto:Params: {} 2023-09-22T21:36:14.160 DEBUG:boto:Token: None 2023-09-22T21:36:14.160 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/ 2023-09-22T21:36:14.161 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8= 2023-09-22T21:36:14.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8=', 'Content-Length': '0'} 2023-09-22T21:36:14.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ab36f7889eb4c8e-00650e08ce-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.171 DEBUG:boto:b'ffzdqu-58obj000000000000000000121000falseobj000000000000000000122023-09-22T21:35:11.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:14.171 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.172 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.172 DEBUG:boto:Method: HEAD 2023-09-22T21:36:14.172 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.172 DEBUG:boto:Data: 2023-09-22T21:36:14.172 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.172 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:14.172 DEBUG:boto:Port: 8001 2023-09-22T21:36:14.172 DEBUG:boto:Params: {} 2023-09-22T21:36:14.172 DEBUG:boto:Token: None 2023-09-22T21:36:14.173 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.173 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oWYpdiFJZa0YNbS2yz9lAGeJsaI= 2023-09-22T21:36:14.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oWYpdiFJZa0YNbS2yz9lAGeJsaI=', 'Content-Length': '0'} 2023-09-22T21:36:14.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4469f303fe3a238-00650e08ce-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.176 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.176 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.177 DEBUG:boto:Method: HEAD 2023-09-22T21:36:14.177 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.177 DEBUG:boto:Data: 2023-09-22T21:36:14.177 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.177 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:14.177 DEBUG:boto:Port: 8000 2023-09-22T21:36:14.177 DEBUG:boto:Params: {} 2023-09-22T21:36:14.177 DEBUG:boto:Token: None 2023-09-22T21:36:14.177 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.178 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oWYpdiFJZa0YNbS2yz9lAGeJsaI= 2023-09-22T21:36:14.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oWYpdiFJZa0YNbS2yz9lAGeJsaI=', 'Content-Length': '0'} 2023-09-22T21:36:14.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ac5d32e18404477-00650e08ce-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-09-22T21:36:14.181 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.181 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.181 DEBUG:boto:Method: GET 2023-09-22T21:36:14.181 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.181 DEBUG:boto:Data: 2023-09-22T21:36:14.181 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.181 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:14.182 DEBUG:boto:Port: 8001 2023-09-22T21:36:14.182 DEBUG:boto:Params: {} 2023-09-22T21:36:14.182 DEBUG:boto:Token: None 2023-09-22T21:36:14.182 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.182 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qqawNUhySDbqw9cBOabp0xAqlGE= 2023-09-22T21:36:14.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qqawNUhySDbqw9cBOabp0xAqlGE=', 'Content-Length': '0'} 2023-09-22T21:36:14.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e8c1bc806ba0fa1-00650e08ce-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.226 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.227 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.227 DEBUG:boto:Method: GET 2023-09-22T21:36:14.227 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.227 DEBUG:boto:Data: 2023-09-22T21:36:14.227 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.227 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:14.228 DEBUG:boto:Port: 8000 2023-09-22T21:36:14.228 DEBUG:boto:Params: {} 2023-09-22T21:36:14.228 DEBUG:boto:Token: None 2023-09-22T21:36:14.228 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000012 2023-09-22T21:36:14.228 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qqawNUhySDbqw9cBOabp0xAqlGE= 2023-09-22T21:36:14.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qqawNUhySDbqw9cBOabp0xAqlGE=', 'Content-Length': '0'} 2023-09-22T21:36:14.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2f550b6c33ac53f-00650e08ce-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.232 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:14.232 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:14.232 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:36:14.233 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:36:14.233 DEBUG:boto:Method: GET 2023-09-22T21:36:14.233 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:36:14.233 DEBUG:boto:Data: 2023-09-22T21:36:14.233 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.233 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:14.233 DEBUG:boto:Port: 8001 2023-09-22T21:36:14.234 DEBUG:boto:Params: {} 2023-09-22T21:36:14.234 DEBUG:boto:Token: None 2023-09-22T21:36:14.234 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/ 2023-09-22T21:36:14.234 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8= 2023-09-22T21:36:14.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8=', 'Content-Length': '0'} 2023-09-22T21:36:14.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c9255c7a433895d-00650e08ce-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.282 DEBUG:boto:b'ffzdqu-58obj000000000000000000121000falseobj000000000000000000122023-09-22T21:35:11.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:14.283 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:14.283 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:14.283 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:36:14.283 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:36:14.284 DEBUG:boto:Method: GET 2023-09-22T21:36:14.284 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000012 2023-09-22T21:36:14.284 DEBUG:boto:Data: 2023-09-22T21:36:14.284 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.284 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:14.284 DEBUG:boto:Port: 8000 2023-09-22T21:36:14.284 DEBUG:boto:Params: {} 2023-09-22T21:36:14.285 DEBUG:boto:Token: None 2023-09-22T21:36:14.285 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/ 2023-09-22T21:36:14.285 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8= 2023-09-22T21:36:14.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8=', 'Content-Length': '0'} 2023-09-22T21:36:14.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000754ceed6c676cfb6-00650e08ce-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.295 DEBUG:boto:b'ffzdqu-58obj000000000000000000121000falseobj000000000000000000122023-09-22T21:35:11.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:14.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-09-22T21:36:14.295 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.295 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.296 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.296 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.296 DEBUG:boto:Method: GET 2023-09-22T21:36:14.296 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.296 DEBUG:boto:Data: 2023-09-22T21:36:14.296 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.296 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:14.297 DEBUG:boto:Port: 8001 2023-09-22T21:36:14.297 DEBUG:boto:Params: {} 2023-09-22T21:36:14.297 DEBUG:boto:Token: None 2023-09-22T21:36:14.297 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.297 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7WJ7d1kEnmyKLDNfdWOjr0cOtdI= 2023-09-22T21:36:14.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7WJ7d1kEnmyKLDNfdWOjr0cOtdI=', 'Content-Length': '0'} 2023-09-22T21:36:14.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096bcd89cb5715a8b-00650e08ce-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.301 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.301 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.301 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.302 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.302 DEBUG:boto:Method: GET 2023-09-22T21:36:14.302 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.302 DEBUG:boto:Data: 2023-09-22T21:36:14.302 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.302 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:14.303 DEBUG:boto:Port: 8000 2023-09-22T21:36:14.303 DEBUG:boto:Params: {} 2023-09-22T21:36:14.303 DEBUG:boto:Token: None 2023-09-22T21:36:14.303 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.303 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7WJ7d1kEnmyKLDNfdWOjr0cOtdI= 2023-09-22T21:36:14.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7WJ7d1kEnmyKLDNfdWOjr0cOtdI=', 'Content-Length': '0'} 2023-09-22T21:36:14.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5780c44c2fe56cf-00650e08ce-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.306 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.307 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.307 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.307 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.307 DEBUG:boto:Method: HEAD 2023-09-22T21:36:14.307 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.307 DEBUG:boto:Data: 2023-09-22T21:36:14.307 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.308 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:14.308 DEBUG:boto:Port: 8001 2023-09-22T21:36:14.308 DEBUG:boto:Params: {} 2023-09-22T21:36:14.308 DEBUG:boto:Token: None 2023-09-22T21:36:14.308 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.308 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Q//VcrJ0uLl3qsw9w/b+qJK1b7A= 2023-09-22T21:36:14.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Q//VcrJ0uLl3qsw9w/b+qJK1b7A=', 'Content-Length': '0'} 2023-09-22T21:36:14.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f77fb8f266935d9a-00650e08ce-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.311 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.312 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.312 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.312 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.312 DEBUG:boto:Method: HEAD 2023-09-22T21:36:14.312 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.312 DEBUG:boto:Data: 2023-09-22T21:36:14.312 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.313 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:14.313 DEBUG:boto:Port: 8000 2023-09-22T21:36:14.313 DEBUG:boto:Params: {} 2023-09-22T21:36:14.313 DEBUG:boto:Token: None 2023-09-22T21:36:14.313 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.313 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Q//VcrJ0uLl3qsw9w/b+qJK1b7A= 2023-09-22T21:36:14.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Q//VcrJ0uLl3qsw9w/b+qJK1b7A=', 'Content-Length': '0'} 2023-09-22T21:36:14.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beea50f7172cdf10-00650e08ce-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-09-22T21:36:14.317 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.317 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.317 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.317 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.317 DEBUG:boto:Method: GET 2023-09-22T21:36:14.317 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.318 DEBUG:boto:Data: 2023-09-22T21:36:14.318 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.318 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:14.318 DEBUG:boto:Port: 8001 2023-09-22T21:36:14.318 DEBUG:boto:Params: {} 2023-09-22T21:36:14.318 DEBUG:boto:Token: None 2023-09-22T21:36:14.319 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.319 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7WJ7d1kEnmyKLDNfdWOjr0cOtdI= 2023-09-22T21:36:14.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7WJ7d1kEnmyKLDNfdWOjr0cOtdI=', 'Content-Length': '0'} 2023-09-22T21:36:14.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000208a5485313d221e-00650e08ce-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.362 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.362 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.363 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.363 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.363 DEBUG:boto:Method: GET 2023-09-22T21:36:14.363 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.363 DEBUG:boto:Data: 2023-09-22T21:36:14.364 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.364 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:14.364 DEBUG:boto:Port: 8000 2023-09-22T21:36:14.364 DEBUG:boto:Params: {} 2023-09-22T21:36:14.364 DEBUG:boto:Token: None 2023-09-22T21:36:14.364 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000013?versionId=null 2023-09-22T21:36:14.364 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7WJ7d1kEnmyKLDNfdWOjr0cOtdI= 2023-09-22T21:36:14.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7WJ7d1kEnmyKLDNfdWOjr0cOtdI=', 'Content-Length': '0'} 2023-09-22T21:36:14.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000246a36ff60d90b5b-00650e08ce-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.368 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:14.368 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:14.368 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:36:14.368 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:36:14.368 DEBUG:boto:Method: GET 2023-09-22T21:36:14.369 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:36:14.369 DEBUG:boto:Data: 2023-09-22T21:36:14.369 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.369 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:14.369 DEBUG:boto:Port: 8001 2023-09-22T21:36:14.369 DEBUG:boto:Params: {} 2023-09-22T21:36:14.369 DEBUG:boto:Token: None 2023-09-22T21:36:14.369 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/ 2023-09-22T21:36:14.370 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8= 2023-09-22T21:36:14.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8=', 'Content-Length': '0'} 2023-09-22T21:36:14.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036f2fc1e394137fa-00650e08ce-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.374 DEBUG:boto:b'ffzdqu-58obj000000000000000000131000falseobj000000000000000000132023-09-22T21:35:02.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:14.375 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:14.375 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:14.375 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:36:14.375 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:36:14.375 DEBUG:boto:Method: GET 2023-09-22T21:36:14.375 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:36:14.376 DEBUG:boto:Data: 2023-09-22T21:36:14.376 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.376 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:14.376 DEBUG:boto:Port: 8000 2023-09-22T21:36:14.376 DEBUG:boto:Params: {} 2023-09-22T21:36:14.376 DEBUG:boto:Token: None 2023-09-22T21:36:14.376 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/ 2023-09-22T21:36:14.376 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8= 2023-09-22T21:36:14.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8=', 'Content-Length': '0'} 2023-09-22T21:36:14.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000149432e2c8e13560-00650e08ce-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.387 DEBUG:boto:b'ffzdqu-58obj000000000000000000131000falseobj000000000000000000132023-09-22T21:35:02.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:14.388 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.388 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.388 DEBUG:boto:Method: HEAD 2023-09-22T21:36:14.388 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.388 DEBUG:boto:Data: 2023-09-22T21:36:14.388 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.388 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:14.389 DEBUG:boto:Port: 8001 2023-09-22T21:36:14.389 DEBUG:boto:Params: {} 2023-09-22T21:36:14.389 DEBUG:boto:Token: None 2023-09-22T21:36:14.389 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.389 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9QwAyAIUSp/sbWOF0IZkol0Zgz4= 2023-09-22T21:36:14.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9QwAyAIUSp/sbWOF0IZkol0Zgz4=', 'Content-Length': '0'} 2023-09-22T21:36:14.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2da96188e01b7d2-00650e08ce-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.392 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.392 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.392 DEBUG:boto:Method: HEAD 2023-09-22T21:36:14.392 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.393 DEBUG:boto:Data: 2023-09-22T21:36:14.393 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.393 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:14.393 DEBUG:boto:Port: 8000 2023-09-22T21:36:14.393 DEBUG:boto:Params: {} 2023-09-22T21:36:14.393 DEBUG:boto:Token: None 2023-09-22T21:36:14.393 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.393 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9QwAyAIUSp/sbWOF0IZkol0Zgz4= 2023-09-22T21:36:14.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9QwAyAIUSp/sbWOF0IZkol0Zgz4=', 'Content-Length': '0'} 2023-09-22T21:36:14.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c66adfee8b7f2274-00650e08ce-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-09-22T21:36:14.396 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.397 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.397 DEBUG:boto:Method: GET 2023-09-22T21:36:14.397 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.397 DEBUG:boto:Data: 2023-09-22T21:36:14.397 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.397 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:14.397 DEBUG:boto:Port: 8001 2023-09-22T21:36:14.397 DEBUG:boto:Params: {} 2023-09-22T21:36:14.397 DEBUG:boto:Token: None 2023-09-22T21:36:14.398 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.398 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QPOfy4vcFaNs7nGcVYseEZqIC7w= 2023-09-22T21:36:14.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QPOfy4vcFaNs7nGcVYseEZqIC7w=', 'Content-Length': '0'} 2023-09-22T21:36:14.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e5c83731e0487de-00650e08ce-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.442 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.443 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.443 DEBUG:boto:Method: GET 2023-09-22T21:36:14.443 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.443 DEBUG:boto:Data: 2023-09-22T21:36:14.443 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.443 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:14.443 DEBUG:boto:Port: 8000 2023-09-22T21:36:14.443 DEBUG:boto:Params: {} 2023-09-22T21:36:14.443 DEBUG:boto:Token: None 2023-09-22T21:36:14.444 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000013 2023-09-22T21:36:14.444 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QPOfy4vcFaNs7nGcVYseEZqIC7w= 2023-09-22T21:36:14.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QPOfy4vcFaNs7nGcVYseEZqIC7w=', 'Content-Length': '0'} 2023-09-22T21:36:14.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d44f76a488b1e350-00650e08ce-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.447 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:14.447 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:14.447 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:36:14.448 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:36:14.448 DEBUG:boto:Method: GET 2023-09-22T21:36:14.448 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:36:14.448 DEBUG:boto:Data: 2023-09-22T21:36:14.448 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.448 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:14.448 DEBUG:boto:Port: 8001 2023-09-22T21:36:14.448 DEBUG:boto:Params: {} 2023-09-22T21:36:14.448 DEBUG:boto:Token: None 2023-09-22T21:36:14.449 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/ 2023-09-22T21:36:14.449 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8= 2023-09-22T21:36:14.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8=', 'Content-Length': '0'} 2023-09-22T21:36:14.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1ebd9cf7ea4d7c2-00650e08ce-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.454 DEBUG:boto:b'ffzdqu-58obj000000000000000000131000falseobj000000000000000000132023-09-22T21:35:02.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:14.454 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:14.455 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:14.455 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:36:14.455 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:36:14.455 DEBUG:boto:Method: GET 2023-09-22T21:36:14.455 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000013 2023-09-22T21:36:14.455 DEBUG:boto:Data: 2023-09-22T21:36:14.455 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.455 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:14.455 DEBUG:boto:Port: 8000 2023-09-22T21:36:14.456 DEBUG:boto:Params: {} 2023-09-22T21:36:14.456 DEBUG:boto:Token: None 2023-09-22T21:36:14.456 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/ 2023-09-22T21:36:14.456 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8= 2023-09-22T21:36:14.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8=', 'Content-Length': '0'} 2023-09-22T21:36:14.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f78f8d8b84f11ce4-00650e08ce-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.510 DEBUG:boto:b'ffzdqu-58obj000000000000000000131000falseobj000000000000000000132023-09-22T21:35:02.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:14.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-09-22T21:36:14.511 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.511 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.511 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.511 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.511 DEBUG:boto:Method: GET 2023-09-22T21:36:14.511 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.512 DEBUG:boto:Data: 2023-09-22T21:36:14.512 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.512 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:14.512 DEBUG:boto:Port: 8001 2023-09-22T21:36:14.512 DEBUG:boto:Params: {} 2023-09-22T21:36:14.512 DEBUG:boto:Token: None 2023-09-22T21:36:14.512 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.512 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+NEFCkSP+NcwqBDkyeMX/to8Pdc= 2023-09-22T21:36:14.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+NEFCkSP+NcwqBDkyeMX/to8Pdc=', 'Content-Length': '0'} 2023-09-22T21:36:14.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000680997a87b62c32a-00650e08ce-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.516 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.516 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.517 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.517 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.517 DEBUG:boto:Method: GET 2023-09-22T21:36:14.517 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.517 DEBUG:boto:Data: 2023-09-22T21:36:14.517 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.517 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:14.517 DEBUG:boto:Port: 8000 2023-09-22T21:36:14.518 DEBUG:boto:Params: {} 2023-09-22T21:36:14.518 DEBUG:boto:Token: None 2023-09-22T21:36:14.518 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.518 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+NEFCkSP+NcwqBDkyeMX/to8Pdc= 2023-09-22T21:36:14.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+NEFCkSP+NcwqBDkyeMX/to8Pdc=', 'Content-Length': '0'} 2023-09-22T21:36:14.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a01d6af3563df4aa-00650e08ce-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.521 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.522 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.522 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.522 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.522 DEBUG:boto:Method: HEAD 2023-09-22T21:36:14.522 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.522 DEBUG:boto:Data: 2023-09-22T21:36:14.522 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.522 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:14.522 DEBUG:boto:Port: 8001 2023-09-22T21:36:14.523 DEBUG:boto:Params: {} 2023-09-22T21:36:14.523 DEBUG:boto:Token: None 2023-09-22T21:36:14.523 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.523 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:307aBnBMhoaxbWlT5WnJnbxtvCc= 2023-09-22T21:36:14.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:307aBnBMhoaxbWlT5WnJnbxtvCc=', 'Content-Length': '0'} 2023-09-22T21:36:14.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca8460dfc7d9bf52-00650e08ce-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.526 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.526 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.526 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.527 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.527 DEBUG:boto:Method: HEAD 2023-09-22T21:36:14.527 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.527 DEBUG:boto:Data: 2023-09-22T21:36:14.527 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.527 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:14.527 DEBUG:boto:Port: 8000 2023-09-22T21:36:14.527 DEBUG:boto:Params: {} 2023-09-22T21:36:14.527 DEBUG:boto:Token: None 2023-09-22T21:36:14.527 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.528 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:307aBnBMhoaxbWlT5WnJnbxtvCc= 2023-09-22T21:36:14.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:307aBnBMhoaxbWlT5WnJnbxtvCc=', 'Content-Length': '0'} 2023-09-22T21:36:14.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036c8b5500acd149d-00650e08ce-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-09-22T21:36:14.533 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.533 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.534 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.534 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.534 DEBUG:boto:Method: GET 2023-09-22T21:36:14.534 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.534 DEBUG:boto:Data: 2023-09-22T21:36:14.534 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.534 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:14.534 DEBUG:boto:Port: 8001 2023-09-22T21:36:14.535 DEBUG:boto:Params: {} 2023-09-22T21:36:14.535 DEBUG:boto:Token: None 2023-09-22T21:36:14.535 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.535 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+NEFCkSP+NcwqBDkyeMX/to8Pdc= 2023-09-22T21:36:14.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+NEFCkSP+NcwqBDkyeMX/to8Pdc=', 'Content-Length': '0'} 2023-09-22T21:36:14.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7a9e42bc6e1e66c-00650e08ce-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.578 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.579 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.579 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.579 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.579 DEBUG:boto:Method: GET 2023-09-22T21:36:14.579 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.579 DEBUG:boto:Data: 2023-09-22T21:36:14.579 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.579 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:14.579 DEBUG:boto:Port: 8000 2023-09-22T21:36:14.579 DEBUG:boto:Params: {} 2023-09-22T21:36:14.580 DEBUG:boto:Token: None 2023-09-22T21:36:14.580 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000014?versionId=null 2023-09-22T21:36:14.580 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+NEFCkSP+NcwqBDkyeMX/to8Pdc= 2023-09-22T21:36:14.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+NEFCkSP+NcwqBDkyeMX/to8Pdc=', 'Content-Length': '0'} 2023-09-22T21:36:14.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7fdc7d98ca2db49-00650e08ce-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.583 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:14.583 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:14.583 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000014 2023-09-22T21:36:14.583 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000014 2023-09-22T21:36:14.584 DEBUG:boto:Method: GET 2023-09-22T21:36:14.584 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000014 2023-09-22T21:36:14.584 DEBUG:boto:Data: 2023-09-22T21:36:14.584 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.584 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:14.584 DEBUG:boto:Port: 8001 2023-09-22T21:36:14.584 DEBUG:boto:Params: {} 2023-09-22T21:36:14.584 DEBUG:boto:Token: None 2023-09-22T21:36:14.584 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/ 2023-09-22T21:36:14.585 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8= 2023-09-22T21:36:14.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8=', 'Content-Length': '0'} 2023-09-22T21:36:14.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000590efcb369a64855-00650e08ce-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.630 DEBUG:boto:b'ffzdqu-58obj000000000000000000141000falseobj000000000000000000142023-09-22T21:35:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:14.631 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:14.631 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:14.631 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000014 2023-09-22T21:36:14.631 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000014 2023-09-22T21:36:14.631 DEBUG:boto:Method: GET 2023-09-22T21:36:14.631 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000014 2023-09-22T21:36:14.631 DEBUG:boto:Data: 2023-09-22T21:36:14.632 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.632 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:14.632 DEBUG:boto:Port: 8000 2023-09-22T21:36:14.632 DEBUG:boto:Params: {} 2023-09-22T21:36:14.632 DEBUG:boto:Token: None 2023-09-22T21:36:14.632 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/ 2023-09-22T21:36:14.632 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8= 2023-09-22T21:36:14.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yEG+itMZfwINoqrUdAfZzAJqQO8=', 'Content-Length': '0'} 2023-09-22T21:36:14.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000f4db7e931970b2-00650e08ce-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.644 DEBUG:boto:b'ffzdqu-58obj000000000000000000141000falseobj000000000000000000142023-09-22T21:35:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:14.645 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.645 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.645 DEBUG:boto:Method: HEAD 2023-09-22T21:36:14.645 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.645 DEBUG:boto:Data: 2023-09-22T21:36:14.645 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.645 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:14.645 DEBUG:boto:Port: 8001 2023-09-22T21:36:14.646 DEBUG:boto:Params: {} 2023-09-22T21:36:14.646 DEBUG:boto:Token: None 2023-09-22T21:36:14.646 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.646 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tmiJeFVCDrFZiG9IGYqvLutyWu4= 2023-09-22T21:36:14.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tmiJeFVCDrFZiG9IGYqvLutyWu4=', 'Content-Length': '0'} 2023-09-22T21:36:14.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006246744e75e2f3f-00650e08ce-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.649 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.649 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.649 DEBUG:boto:Method: HEAD 2023-09-22T21:36:14.649 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.650 DEBUG:boto:Data: 2023-09-22T21:36:14.650 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.650 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:14.650 DEBUG:boto:Port: 8000 2023-09-22T21:36:14.650 DEBUG:boto:Params: {} 2023-09-22T21:36:14.650 DEBUG:boto:Token: None 2023-09-22T21:36:14.650 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.650 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tmiJeFVCDrFZiG9IGYqvLutyWu4= 2023-09-22T21:36:14.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tmiJeFVCDrFZiG9IGYqvLutyWu4=', 'Content-Length': '0'} 2023-09-22T21:36:14.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027f05f968c665144-00650e08ce-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-09-22T21:36:14.654 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.654 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.654 DEBUG:boto:Method: GET 2023-09-22T21:36:14.654 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.654 DEBUG:boto:Data: 2023-09-22T21:36:14.654 DEBUG:boto:Headers: {} 2023-09-22T21:36:14.654 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:14.654 DEBUG:boto:Port: 8001 2023-09-22T21:36:14.654 DEBUG:boto:Params: {} 2023-09-22T21:36:14.655 DEBUG:boto:Token: None 2023-09-22T21:36:14.655 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:14 GMT /ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:14.655 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h8zbmAcpB5LKDvMGVoCP3S9vz0A= 2023-09-22T21:36:14.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h8zbmAcpB5LKDvMGVoCP3S9vz0A=', 'Content-Length': '0'} 2023-09-22T21:36:14.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000379ff642ef4234b8-00650e08ce-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:14.698 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:15.011 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:15.011 DEBUG:boto:Method: GET 2023-09-22T21:36:15.012 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:15.012 DEBUG:boto:Data: 2023-09-22T21:36:15.012 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.012 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.013 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.013 DEBUG:boto:Params: {} 2023-09-22T21:36:15.013 DEBUG:boto:Token: None 2023-09-22T21:36:15.013 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000014 2023-09-22T21:36:15.014 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:R5FyTk8DZc7f3D6JMD2Yj2nKdio= 2023-09-22T21:36:15.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:R5FyTk8DZc7f3D6JMD2Yj2nKdio=', 'Content-Length': '0'} 2023-09-22T21:36:15.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e490eeaebc0c0686-00650e08cf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.017 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:15.017 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:15.017 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000014 2023-09-22T21:36:15.017 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000014 2023-09-22T21:36:15.017 DEBUG:boto:Method: GET 2023-09-22T21:36:15.018 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000014 2023-09-22T21:36:15.018 DEBUG:boto:Data: 2023-09-22T21:36:15.018 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.018 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.018 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.018 DEBUG:boto:Params: {} 2023-09-22T21:36:15.018 DEBUG:boto:Token: None 2023-09-22T21:36:15.018 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/ 2023-09-22T21:36:15.019 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ= 2023-09-22T21:36:15.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ=', 'Content-Length': '0'} 2023-09-22T21:36:15.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f883bd9bc01f7424-00650e08cf-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.023 DEBUG:boto:b'ffzdqu-58obj000000000000000000141000falseobj000000000000000000142023-09-22T21:35:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:15.024 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:15.024 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:15.024 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000014 2023-09-22T21:36:15.024 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000014 2023-09-22T21:36:15.024 DEBUG:boto:Method: GET 2023-09-22T21:36:15.024 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000014 2023-09-22T21:36:15.024 DEBUG:boto:Data: 2023-09-22T21:36:15.025 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.025 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.025 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.025 DEBUG:boto:Params: {} 2023-09-22T21:36:15.025 DEBUG:boto:Token: None 2023-09-22T21:36:15.025 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/ 2023-09-22T21:36:15.025 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ= 2023-09-22T21:36:15.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ=', 'Content-Length': '0'} 2023-09-22T21:36:15.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e1c25497b38ba76-00650e08cf-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.036 DEBUG:boto:b'ffzdqu-58obj000000000000000000141000falseobj000000000000000000142023-09-22T21:35:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:15.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-09-22T21:36:15.036 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.036 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.037 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.037 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.037 DEBUG:boto:Method: GET 2023-09-22T21:36:15.037 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.037 DEBUG:boto:Data: 2023-09-22T21:36:15.037 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.037 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.037 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.037 DEBUG:boto:Params: {} 2023-09-22T21:36:15.037 DEBUG:boto:Token: None 2023-09-22T21:36:15.038 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.038 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MSc4ajdOj6YsYQL0+OepfXKlkkE= 2023-09-22T21:36:15.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MSc4ajdOj6YsYQL0+OepfXKlkkE=', 'Content-Length': '0'} 2023-09-22T21:36:15.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e37ed983abfc62b-00650e08cf-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.082 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.082 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.083 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.083 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.083 DEBUG:boto:Method: GET 2023-09-22T21:36:15.083 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.083 DEBUG:boto:Data: 2023-09-22T21:36:15.083 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.083 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.083 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.084 DEBUG:boto:Params: {} 2023-09-22T21:36:15.084 DEBUG:boto:Token: None 2023-09-22T21:36:15.084 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.084 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MSc4ajdOj6YsYQL0+OepfXKlkkE= 2023-09-22T21:36:15.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MSc4ajdOj6YsYQL0+OepfXKlkkE=', 'Content-Length': '0'} 2023-09-22T21:36:15.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000644c286df6b5a8eb-00650e08cf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.087 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.087 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.087 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.087 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.087 DEBUG:boto:Method: HEAD 2023-09-22T21:36:15.087 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.088 DEBUG:boto:Data: 2023-09-22T21:36:15.088 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.088 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.088 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.088 DEBUG:boto:Params: {} 2023-09-22T21:36:15.088 DEBUG:boto:Token: None 2023-09-22T21:36:15.088 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.088 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ygSGHQRZE4+C6CPRi02fN1vRRBw= 2023-09-22T21:36:15.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ygSGHQRZE4+C6CPRi02fN1vRRBw=', 'Content-Length': '0'} 2023-09-22T21:36:15.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003890a3125d44a70-00650e08cf-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.091 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.091 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.091 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.092 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.092 DEBUG:boto:Method: HEAD 2023-09-22T21:36:15.092 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.092 DEBUG:boto:Data: 2023-09-22T21:36:15.092 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.092 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.092 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.092 DEBUG:boto:Params: {} 2023-09-22T21:36:15.092 DEBUG:boto:Token: None 2023-09-22T21:36:15.093 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.093 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ygSGHQRZE4+C6CPRi02fN1vRRBw= 2023-09-22T21:36:15.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ygSGHQRZE4+C6CPRi02fN1vRRBw=', 'Content-Length': '0'} 2023-09-22T21:36:15.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db96cce524741c92-00650e08cf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-09-22T21:36:15.096 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.096 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.096 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.096 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.096 DEBUG:boto:Method: GET 2023-09-22T21:36:15.096 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.096 DEBUG:boto:Data: 2023-09-22T21:36:15.097 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.097 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.097 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.097 DEBUG:boto:Params: {} 2023-09-22T21:36:15.097 DEBUG:boto:Token: None 2023-09-22T21:36:15.097 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.097 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MSc4ajdOj6YsYQL0+OepfXKlkkE= 2023-09-22T21:36:15.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MSc4ajdOj6YsYQL0+OepfXKlkkE=', 'Content-Length': '0'} 2023-09-22T21:36:15.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093295752b1a4d7d6-00650e08cf-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.100 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.100 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.100 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.101 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.101 DEBUG:boto:Method: GET 2023-09-22T21:36:15.101 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.101 DEBUG:boto:Data: 2023-09-22T21:36:15.101 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.101 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.101 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.101 DEBUG:boto:Params: {} 2023-09-22T21:36:15.101 DEBUG:boto:Token: None 2023-09-22T21:36:15.102 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000015?versionId=null 2023-09-22T21:36:15.102 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MSc4ajdOj6YsYQL0+OepfXKlkkE= 2023-09-22T21:36:15.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MSc4ajdOj6YsYQL0+OepfXKlkkE=', 'Content-Length': '0'} 2023-09-22T21:36:15.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d031309b5a3675d-00650e08cf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.146 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:15.147 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:15.147 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000015 2023-09-22T21:36:15.147 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000015 2023-09-22T21:36:15.147 DEBUG:boto:Method: GET 2023-09-22T21:36:15.147 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000015 2023-09-22T21:36:15.147 DEBUG:boto:Data: 2023-09-22T21:36:15.147 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.147 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.148 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.148 DEBUG:boto:Params: {} 2023-09-22T21:36:15.148 DEBUG:boto:Token: None 2023-09-22T21:36:15.148 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/ 2023-09-22T21:36:15.148 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ= 2023-09-22T21:36:15.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ=', 'Content-Length': '0'} 2023-09-22T21:36:15.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b6407869ac49fc5-00650e08cf-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.153 DEBUG:boto:b'ffzdqu-58obj000000000000000000151000falseobj000000000000000000152023-09-22T21:35:05.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:15.154 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:15.154 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:15.154 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000015 2023-09-22T21:36:15.154 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000015 2023-09-22T21:36:15.154 DEBUG:boto:Method: GET 2023-09-22T21:36:15.154 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000015 2023-09-22T21:36:15.154 DEBUG:boto:Data: 2023-09-22T21:36:15.154 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.155 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.155 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.155 DEBUG:boto:Params: {} 2023-09-22T21:36:15.155 DEBUG:boto:Token: None 2023-09-22T21:36:15.155 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/ 2023-09-22T21:36:15.155 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ= 2023-09-22T21:36:15.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ=', 'Content-Length': '0'} 2023-09-22T21:36:15.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000557fce181d69393b-00650e08cf-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.165 DEBUG:boto:b'ffzdqu-58obj000000000000000000151000falseobj000000000000000000152023-09-22T21:35:05.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:15.165 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.165 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.165 DEBUG:boto:Method: HEAD 2023-09-22T21:36:15.165 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.166 DEBUG:boto:Data: 2023-09-22T21:36:15.166 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.166 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.166 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.166 DEBUG:boto:Params: {} 2023-09-22T21:36:15.166 DEBUG:boto:Token: None 2023-09-22T21:36:15.166 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.166 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GcstFoRvFumHo9LXrGUXjOdkDx0= 2023-09-22T21:36:15.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GcstFoRvFumHo9LXrGUXjOdkDx0=', 'Content-Length': '0'} 2023-09-22T21:36:15.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fefc5e8f595a9baa-00650e08cf-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.170 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.170 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.170 DEBUG:boto:Method: HEAD 2023-09-22T21:36:15.170 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.170 DEBUG:boto:Data: 2023-09-22T21:36:15.170 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.170 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.170 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.170 DEBUG:boto:Params: {} 2023-09-22T21:36:15.171 DEBUG:boto:Token: None 2023-09-22T21:36:15.171 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.171 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GcstFoRvFumHo9LXrGUXjOdkDx0= 2023-09-22T21:36:15.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GcstFoRvFumHo9LXrGUXjOdkDx0=', 'Content-Length': '0'} 2023-09-22T21:36:15.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdf1ac47930774d6-00650e08cf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-09-22T21:36:15.174 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.174 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.174 DEBUG:boto:Method: GET 2023-09-22T21:36:15.174 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.174 DEBUG:boto:Data: 2023-09-22T21:36:15.175 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.175 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.175 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.175 DEBUG:boto:Params: {} 2023-09-22T21:36:15.175 DEBUG:boto:Token: None 2023-09-22T21:36:15.175 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.175 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:l/eHg1yiK6ZWPgFX5ZDBsJPQ0nQ= 2023-09-22T21:36:15.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:l/eHg1yiK6ZWPgFX5ZDBsJPQ0nQ=', 'Content-Length': '0'} 2023-09-22T21:36:15.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b572209bd9cbba6-00650e08cf-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.222 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.223 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.223 DEBUG:boto:Method: GET 2023-09-22T21:36:15.223 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.223 DEBUG:boto:Data: 2023-09-22T21:36:15.223 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.224 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.224 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.224 DEBUG:boto:Params: {} 2023-09-22T21:36:15.224 DEBUG:boto:Token: None 2023-09-22T21:36:15.225 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000015 2023-09-22T21:36:15.225 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:l/eHg1yiK6ZWPgFX5ZDBsJPQ0nQ= 2023-09-22T21:36:15.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:l/eHg1yiK6ZWPgFX5ZDBsJPQ0nQ=', 'Content-Length': '0'} 2023-09-22T21:36:15.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045ca50e3b3e11a71-00650e08cf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.228 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:15.228 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:15.228 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000015 2023-09-22T21:36:15.229 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000015 2023-09-22T21:36:15.229 DEBUG:boto:Method: GET 2023-09-22T21:36:15.229 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000015 2023-09-22T21:36:15.229 DEBUG:boto:Data: 2023-09-22T21:36:15.229 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.229 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.230 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.230 DEBUG:boto:Params: {} 2023-09-22T21:36:15.230 DEBUG:boto:Token: None 2023-09-22T21:36:15.230 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/ 2023-09-22T21:36:15.230 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ= 2023-09-22T21:36:15.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ=', 'Content-Length': '0'} 2023-09-22T21:36:15.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a75a0f9e7609582a-00650e08cf-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.235 DEBUG:boto:b'ffzdqu-58obj000000000000000000151000falseobj000000000000000000152023-09-22T21:35:05.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:15.236 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:15.236 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:15.236 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000015 2023-09-22T21:36:15.236 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000015 2023-09-22T21:36:15.237 DEBUG:boto:Method: GET 2023-09-22T21:36:15.237 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000015 2023-09-22T21:36:15.237 DEBUG:boto:Data: 2023-09-22T21:36:15.237 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.237 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.237 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.238 DEBUG:boto:Params: {} 2023-09-22T21:36:15.238 DEBUG:boto:Token: None 2023-09-22T21:36:15.238 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/ 2023-09-22T21:36:15.238 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ= 2023-09-22T21:36:15.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ=', 'Content-Length': '0'} 2023-09-22T21:36:15.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000355f43ce2d3cff3c-00650e08cf-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.294 DEBUG:boto:b'ffzdqu-58obj000000000000000000151000falseobj000000000000000000152023-09-22T21:35:05.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:15.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-09-22T21:36:15.295 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.295 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.295 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.295 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.296 DEBUG:boto:Method: GET 2023-09-22T21:36:15.296 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.296 DEBUG:boto:Data: 2023-09-22T21:36:15.296 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.296 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.296 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.296 DEBUG:boto:Params: {} 2023-09-22T21:36:15.296 DEBUG:boto:Token: None 2023-09-22T21:36:15.296 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.297 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0YL7eW+POBEQBsRmqE9WppFGs2Q= 2023-09-22T21:36:15.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0YL7eW+POBEQBsRmqE9WppFGs2Q=', 'Content-Length': '0'} 2023-09-22T21:36:15.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8366469d51eb2e8-00650e08cf-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.300 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.300 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.300 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.300 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.301 DEBUG:boto:Method: GET 2023-09-22T21:36:15.301 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.301 DEBUG:boto:Data: 2023-09-22T21:36:15.301 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.301 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.301 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.301 DEBUG:boto:Params: {} 2023-09-22T21:36:15.301 DEBUG:boto:Token: None 2023-09-22T21:36:15.302 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.302 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0YL7eW+POBEQBsRmqE9WppFGs2Q= 2023-09-22T21:36:15.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0YL7eW+POBEQBsRmqE9WppFGs2Q=', 'Content-Length': '0'} 2023-09-22T21:36:15.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000254220f09bf94088-00650e08cf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.305 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.305 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.305 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.306 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.306 DEBUG:boto:Method: HEAD 2023-09-22T21:36:15.306 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.306 DEBUG:boto:Data: 2023-09-22T21:36:15.306 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.306 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.306 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.306 DEBUG:boto:Params: {} 2023-09-22T21:36:15.307 DEBUG:boto:Token: None 2023-09-22T21:36:15.307 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.307 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E/ODw2Mt/+YDA9sakxBD5znGkzE= 2023-09-22T21:36:15.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E/ODw2Mt/+YDA9sakxBD5znGkzE=', 'Content-Length': '0'} 2023-09-22T21:36:15.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024dab88693a6d37a-00650e08cf-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.310 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.310 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.310 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.310 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.310 DEBUG:boto:Method: HEAD 2023-09-22T21:36:15.310 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.310 DEBUG:boto:Data: 2023-09-22T21:36:15.311 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.311 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.311 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.311 DEBUG:boto:Params: {} 2023-09-22T21:36:15.311 DEBUG:boto:Token: None 2023-09-22T21:36:15.311 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.311 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E/ODw2Mt/+YDA9sakxBD5znGkzE= 2023-09-22T21:36:15.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E/ODw2Mt/+YDA9sakxBD5znGkzE=', 'Content-Length': '0'} 2023-09-22T21:36:15.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a49b21d0f7286f72-00650e08cf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-09-22T21:36:15.314 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.314 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.315 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.315 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.315 DEBUG:boto:Method: GET 2023-09-22T21:36:15.315 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.315 DEBUG:boto:Data: 2023-09-22T21:36:15.315 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.315 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.315 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.315 DEBUG:boto:Params: {} 2023-09-22T21:36:15.315 DEBUG:boto:Token: None 2023-09-22T21:36:15.316 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.316 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0YL7eW+POBEQBsRmqE9WppFGs2Q= 2023-09-22T21:36:15.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0YL7eW+POBEQBsRmqE9WppFGs2Q=', 'Content-Length': '0'} 2023-09-22T21:36:15.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef3b4fa280e8f3ff-00650e08cf-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.362 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.363 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.363 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.363 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.363 DEBUG:boto:Method: GET 2023-09-22T21:36:15.363 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.363 DEBUG:boto:Data: 2023-09-22T21:36:15.364 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.364 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.364 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.364 DEBUG:boto:Params: {} 2023-09-22T21:36:15.364 DEBUG:boto:Token: None 2023-09-22T21:36:15.365 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000016?versionId=null 2023-09-22T21:36:15.365 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0YL7eW+POBEQBsRmqE9WppFGs2Q= 2023-09-22T21:36:15.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0YL7eW+POBEQBsRmqE9WppFGs2Q=', 'Content-Length': '0'} 2023-09-22T21:36:15.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009982539b90772673-00650e08cf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.368 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:15.368 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:15.369 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000016 2023-09-22T21:36:15.369 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000016 2023-09-22T21:36:15.369 DEBUG:boto:Method: GET 2023-09-22T21:36:15.369 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000016 2023-09-22T21:36:15.369 DEBUG:boto:Data: 2023-09-22T21:36:15.369 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.370 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.370 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.370 DEBUG:boto:Params: {} 2023-09-22T21:36:15.370 DEBUG:boto:Token: None 2023-09-22T21:36:15.371 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/ 2023-09-22T21:36:15.371 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ= 2023-09-22T21:36:15.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ=', 'Content-Length': '0'} 2023-09-22T21:36:15.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6ceb654ff11f274-00650e08cf-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.376 DEBUG:boto:b'ffzdqu-58obj000000000000000000161000falseobj000000000000000000162023-09-22T21:35:05.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:15.376 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:15.377 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:15.377 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000016 2023-09-22T21:36:15.377 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000016 2023-09-22T21:36:15.377 DEBUG:boto:Method: GET 2023-09-22T21:36:15.377 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000016 2023-09-22T21:36:15.378 DEBUG:boto:Data: 2023-09-22T21:36:15.378 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.378 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.378 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.378 DEBUG:boto:Params: {} 2023-09-22T21:36:15.378 DEBUG:boto:Token: None 2023-09-22T21:36:15.379 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/ 2023-09-22T21:36:15.379 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ= 2023-09-22T21:36:15.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ=', 'Content-Length': '0'} 2023-09-22T21:36:15.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fb2dd48f0f21d54-00650e08cf-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.389 DEBUG:boto:b'ffzdqu-58obj000000000000000000161000falseobj000000000000000000162023-09-22T21:35:05.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:15.390 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.390 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.390 DEBUG:boto:Method: HEAD 2023-09-22T21:36:15.390 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.390 DEBUG:boto:Data: 2023-09-22T21:36:15.390 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.391 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.391 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.391 DEBUG:boto:Params: {} 2023-09-22T21:36:15.391 DEBUG:boto:Token: None 2023-09-22T21:36:15.391 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.391 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Tq9x/oNAjHXtAMZSOb1Ylalkgn4= 2023-09-22T21:36:15.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Tq9x/oNAjHXtAMZSOb1Ylalkgn4=', 'Content-Length': '0'} 2023-09-22T21:36:15.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c30a79a4bd649ca-00650e08cf-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.394 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.394 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.394 DEBUG:boto:Method: HEAD 2023-09-22T21:36:15.395 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.395 DEBUG:boto:Data: 2023-09-22T21:36:15.395 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.395 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.395 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.395 DEBUG:boto:Params: {} 2023-09-22T21:36:15.395 DEBUG:boto:Token: None 2023-09-22T21:36:15.395 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.395 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Tq9x/oNAjHXtAMZSOb1Ylalkgn4= 2023-09-22T21:36:15.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Tq9x/oNAjHXtAMZSOb1Ylalkgn4=', 'Content-Length': '0'} 2023-09-22T21:36:15.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dbdea2607b7e39c-00650e08cf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-09-22T21:36:15.399 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.399 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.399 DEBUG:boto:Method: GET 2023-09-22T21:36:15.399 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.399 DEBUG:boto:Data: 2023-09-22T21:36:15.400 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.400 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.400 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.400 DEBUG:boto:Params: {} 2023-09-22T21:36:15.400 DEBUG:boto:Token: None 2023-09-22T21:36:15.401 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.401 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iCeY0+O7ajJR4kbDUy8iIBVeDI0= 2023-09-22T21:36:15.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iCeY0+O7ajJR4kbDUy8iIBVeDI0=', 'Content-Length': '0'} 2023-09-22T21:36:15.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055a9fbafbeef2f57-00650e08cf-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.446 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.446 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.447 DEBUG:boto:Method: GET 2023-09-22T21:36:15.447 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.447 DEBUG:boto:Data: 2023-09-22T21:36:15.447 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.447 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.447 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.447 DEBUG:boto:Params: {} 2023-09-22T21:36:15.447 DEBUG:boto:Token: None 2023-09-22T21:36:15.448 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000016 2023-09-22T21:36:15.448 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iCeY0+O7ajJR4kbDUy8iIBVeDI0= 2023-09-22T21:36:15.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iCeY0+O7ajJR4kbDUy8iIBVeDI0=', 'Content-Length': '0'} 2023-09-22T21:36:15.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061476d002b6e2e06-00650e08cf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.451 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:15.452 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:15.452 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000016 2023-09-22T21:36:15.452 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000016 2023-09-22T21:36:15.452 DEBUG:boto:Method: GET 2023-09-22T21:36:15.452 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000016 2023-09-22T21:36:15.452 DEBUG:boto:Data: 2023-09-22T21:36:15.452 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.452 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.453 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.453 DEBUG:boto:Params: {} 2023-09-22T21:36:15.453 DEBUG:boto:Token: None 2023-09-22T21:36:15.453 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/ 2023-09-22T21:36:15.453 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ= 2023-09-22T21:36:15.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ=', 'Content-Length': '0'} 2023-09-22T21:36:15.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b75d5fd8490c475b-00650e08cf-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.498 DEBUG:boto:b'ffzdqu-58obj000000000000000000161000falseobj000000000000000000162023-09-22T21:35:05.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:15.499 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:15.499 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:15.499 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000016 2023-09-22T21:36:15.499 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000016 2023-09-22T21:36:15.499 DEBUG:boto:Method: GET 2023-09-22T21:36:15.499 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000016 2023-09-22T21:36:15.500 DEBUG:boto:Data: 2023-09-22T21:36:15.500 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.500 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.500 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.500 DEBUG:boto:Params: {} 2023-09-22T21:36:15.500 DEBUG:boto:Token: None 2023-09-22T21:36:15.500 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/ 2023-09-22T21:36:15.500 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ= 2023-09-22T21:36:15.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ=', 'Content-Length': '0'} 2023-09-22T21:36:15.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f101ea65a58c333-00650e08cf-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.510 DEBUG:boto:b'ffzdqu-58obj000000000000000000161000falseobj000000000000000000162023-09-22T21:35:05.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:15.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-09-22T21:36:15.511 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.511 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.511 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.511 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.511 DEBUG:boto:Method: GET 2023-09-22T21:36:15.511 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.512 DEBUG:boto:Data: 2023-09-22T21:36:15.512 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.512 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.512 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.512 DEBUG:boto:Params: {} 2023-09-22T21:36:15.512 DEBUG:boto:Token: None 2023-09-22T21:36:15.512 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.512 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t8pxCpF0+ta9zCqQ/Jrbrpd5TWM= 2023-09-22T21:36:15.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t8pxCpF0+ta9zCqQ/Jrbrpd5TWM=', 'Content-Length': '0'} 2023-09-22T21:36:15.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9e542b06d695a92-00650e08cf-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.516 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.516 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.516 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.516 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.517 DEBUG:boto:Method: GET 2023-09-22T21:36:15.517 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.517 DEBUG:boto:Data: 2023-09-22T21:36:15.517 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.517 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.517 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.517 DEBUG:boto:Params: {} 2023-09-22T21:36:15.517 DEBUG:boto:Token: None 2023-09-22T21:36:15.517 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.518 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t8pxCpF0+ta9zCqQ/Jrbrpd5TWM= 2023-09-22T21:36:15.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t8pxCpF0+ta9zCqQ/Jrbrpd5TWM=', 'Content-Length': '0'} 2023-09-22T21:36:15.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea1127fa706e8023-00650e08cf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.521 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.521 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.521 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.521 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.521 DEBUG:boto:Method: HEAD 2023-09-22T21:36:15.522 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.522 DEBUG:boto:Data: 2023-09-22T21:36:15.522 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.522 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.522 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.522 DEBUG:boto:Params: {} 2023-09-22T21:36:15.522 DEBUG:boto:Token: None 2023-09-22T21:36:15.522 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.523 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4mVukjnD9AZLLC8zY47pG1pI97Q= 2023-09-22T21:36:15.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4mVukjnD9AZLLC8zY47pG1pI97Q=', 'Content-Length': '0'} 2023-09-22T21:36:15.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebf1cdb2702b9140-00650e08cf-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.525 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.526 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.526 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.526 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.526 DEBUG:boto:Method: HEAD 2023-09-22T21:36:15.526 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.526 DEBUG:boto:Data: 2023-09-22T21:36:15.526 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.526 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.526 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.526 DEBUG:boto:Params: {} 2023-09-22T21:36:15.527 DEBUG:boto:Token: None 2023-09-22T21:36:15.527 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.527 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4mVukjnD9AZLLC8zY47pG1pI97Q= 2023-09-22T21:36:15.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4mVukjnD9AZLLC8zY47pG1pI97Q=', 'Content-Length': '0'} 2023-09-22T21:36:15.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab7a2dc916c1d42e-00650e08cf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-09-22T21:36:15.530 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.530 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.530 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.530 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.530 DEBUG:boto:Method: GET 2023-09-22T21:36:15.531 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.531 DEBUG:boto:Data: 2023-09-22T21:36:15.531 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.531 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.531 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.531 DEBUG:boto:Params: {} 2023-09-22T21:36:15.531 DEBUG:boto:Token: None 2023-09-22T21:36:15.531 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.531 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t8pxCpF0+ta9zCqQ/Jrbrpd5TWM= 2023-09-22T21:36:15.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t8pxCpF0+ta9zCqQ/Jrbrpd5TWM=', 'Content-Length': '0'} 2023-09-22T21:36:15.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006abacf4cf21608fa-00650e08cf-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.578 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.579 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.579 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.579 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.579 DEBUG:boto:Method: GET 2023-09-22T21:36:15.579 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.579 DEBUG:boto:Data: 2023-09-22T21:36:15.579 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.580 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.580 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.580 DEBUG:boto:Params: {} 2023-09-22T21:36:15.580 DEBUG:boto:Token: None 2023-09-22T21:36:15.580 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000017?versionId=null 2023-09-22T21:36:15.580 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t8pxCpF0+ta9zCqQ/Jrbrpd5TWM= 2023-09-22T21:36:15.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t8pxCpF0+ta9zCqQ/Jrbrpd5TWM=', 'Content-Length': '0'} 2023-09-22T21:36:15.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004509106818054378-00650e08cf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.584 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:15.584 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:15.584 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000017 2023-09-22T21:36:15.585 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000017 2023-09-22T21:36:15.585 DEBUG:boto:Method: GET 2023-09-22T21:36:15.585 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000017 2023-09-22T21:36:15.585 DEBUG:boto:Data: 2023-09-22T21:36:15.585 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.586 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.586 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.586 DEBUG:boto:Params: {} 2023-09-22T21:36:15.586 DEBUG:boto:Token: None 2023-09-22T21:36:15.586 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/ 2023-09-22T21:36:15.587 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ= 2023-09-22T21:36:15.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ=', 'Content-Length': '0'} 2023-09-22T21:36:15.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009976b7c11c5b8b64-00650e08cf-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.634 DEBUG:boto:b'ffzdqu-58obj000000000000000000171000falseobj000000000000000000172023-09-22T21:35:05.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:15.635 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:15.635 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:15.635 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000017 2023-09-22T21:36:15.635 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000017 2023-09-22T21:36:15.635 DEBUG:boto:Method: GET 2023-09-22T21:36:15.635 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000017 2023-09-22T21:36:15.636 DEBUG:boto:Data: 2023-09-22T21:36:15.636 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.636 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.636 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.636 DEBUG:boto:Params: {} 2023-09-22T21:36:15.636 DEBUG:boto:Token: None 2023-09-22T21:36:15.636 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/ 2023-09-22T21:36:15.636 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ= 2023-09-22T21:36:15.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ=', 'Content-Length': '0'} 2023-09-22T21:36:15.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d996a47efbded99-00650e08cf-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.647 DEBUG:boto:b'ffzdqu-58obj000000000000000000171000falseobj000000000000000000172023-09-22T21:35:05.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:15.648 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.648 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.648 DEBUG:boto:Method: HEAD 2023-09-22T21:36:15.648 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.648 DEBUG:boto:Data: 2023-09-22T21:36:15.648 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.648 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.648 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.649 DEBUG:boto:Params: {} 2023-09-22T21:36:15.649 DEBUG:boto:Token: None 2023-09-22T21:36:15.649 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.649 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nfNI/65WZMTJ7fNYftoflITJfKo= 2023-09-22T21:36:15.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nfNI/65WZMTJ7fNYftoflITJfKo=', 'Content-Length': '0'} 2023-09-22T21:36:15.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c332b27ede01acc6-00650e08cf-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.653 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.653 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.653 DEBUG:boto:Method: HEAD 2023-09-22T21:36:15.653 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.654 DEBUG:boto:Data: 2023-09-22T21:36:15.654 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.654 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.654 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.654 DEBUG:boto:Params: {} 2023-09-22T21:36:15.655 DEBUG:boto:Token: None 2023-09-22T21:36:15.655 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.655 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nfNI/65WZMTJ7fNYftoflITJfKo= 2023-09-22T21:36:15.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nfNI/65WZMTJ7fNYftoflITJfKo=', 'Content-Length': '0'} 2023-09-22T21:36:15.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072bc2570f18d6398-00650e08cf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-09-22T21:36:15.659 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.660 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.660 DEBUG:boto:Method: GET 2023-09-22T21:36:15.660 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.660 DEBUG:boto:Data: 2023-09-22T21:36:15.660 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.660 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.661 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.661 DEBUG:boto:Params: {} 2023-09-22T21:36:15.661 DEBUG:boto:Token: None 2023-09-22T21:36:15.661 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.662 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+BVlMMXV+bemBimtUoawAiTQGF4= 2023-09-22T21:36:15.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+BVlMMXV+bemBimtUoawAiTQGF4=', 'Content-Length': '0'} 2023-09-22T21:36:15.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de0a329366d78d61-00650e08cf-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.706 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.707 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.707 DEBUG:boto:Method: GET 2023-09-22T21:36:15.707 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.707 DEBUG:boto:Data: 2023-09-22T21:36:15.707 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.707 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.707 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.707 DEBUG:boto:Params: {} 2023-09-22T21:36:15.707 DEBUG:boto:Token: None 2023-09-22T21:36:15.708 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000017 2023-09-22T21:36:15.708 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+BVlMMXV+bemBimtUoawAiTQGF4= 2023-09-22T21:36:15.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+BVlMMXV+bemBimtUoawAiTQGF4=', 'Content-Length': '0'} 2023-09-22T21:36:15.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005154d973edce918d-00650e08cf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.711 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:15.711 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:15.712 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000017 2023-09-22T21:36:15.712 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000017 2023-09-22T21:36:15.712 DEBUG:boto:Method: GET 2023-09-22T21:36:15.712 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000017 2023-09-22T21:36:15.712 DEBUG:boto:Data: 2023-09-22T21:36:15.712 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.712 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.712 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.712 DEBUG:boto:Params: {} 2023-09-22T21:36:15.712 DEBUG:boto:Token: None 2023-09-22T21:36:15.713 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/ 2023-09-22T21:36:15.713 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ= 2023-09-22T21:36:15.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ=', 'Content-Length': '0'} 2023-09-22T21:36:15.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fddca40ac9bce2b6-00650e08cf-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.758 DEBUG:boto:b'ffzdqu-58obj000000000000000000171000falseobj000000000000000000172023-09-22T21:35:05.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:15.759 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:15.759 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:15.759 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000017 2023-09-22T21:36:15.759 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000017 2023-09-22T21:36:15.760 DEBUG:boto:Method: GET 2023-09-22T21:36:15.760 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000017 2023-09-22T21:36:15.760 DEBUG:boto:Data: 2023-09-22T21:36:15.760 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.760 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.760 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.760 DEBUG:boto:Params: {} 2023-09-22T21:36:15.760 DEBUG:boto:Token: None 2023-09-22T21:36:15.760 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/ 2023-09-22T21:36:15.761 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ= 2023-09-22T21:36:15.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ=', 'Content-Length': '0'} 2023-09-22T21:36:15.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000244a1546664da0c0-00650e08cf-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.774 DEBUG:boto:b'ffzdqu-58obj000000000000000000171000falseobj000000000000000000172023-09-22T21:35:05.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:15.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-09-22T21:36:15.775 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.775 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.775 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.775 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.775 DEBUG:boto:Method: GET 2023-09-22T21:36:15.775 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.776 DEBUG:boto:Data: 2023-09-22T21:36:15.776 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.776 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.776 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.776 DEBUG:boto:Params: {} 2023-09-22T21:36:15.776 DEBUG:boto:Token: None 2023-09-22T21:36:15.776 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.776 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GEIJEj0NqMpM8tSCcoiN2ddVUiw= 2023-09-22T21:36:15.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GEIJEj0NqMpM8tSCcoiN2ddVUiw=', 'Content-Length': '0'} 2023-09-22T21:36:15.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000099123880927018-00650e08cf-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.780 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.780 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.781 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.781 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.781 DEBUG:boto:Method: GET 2023-09-22T21:36:15.781 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.781 DEBUG:boto:Data: 2023-09-22T21:36:15.781 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.782 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.782 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.782 DEBUG:boto:Params: {} 2023-09-22T21:36:15.782 DEBUG:boto:Token: None 2023-09-22T21:36:15.782 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.783 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GEIJEj0NqMpM8tSCcoiN2ddVUiw= 2023-09-22T21:36:15.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GEIJEj0NqMpM8tSCcoiN2ddVUiw=', 'Content-Length': '0'} 2023-09-22T21:36:15.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005691a7bb0ef08c72-00650e08cf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.830 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.831 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.831 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.831 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.831 DEBUG:boto:Method: HEAD 2023-09-22T21:36:15.831 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.831 DEBUG:boto:Data: 2023-09-22T21:36:15.831 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.831 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.831 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.832 DEBUG:boto:Params: {} 2023-09-22T21:36:15.832 DEBUG:boto:Token: None 2023-09-22T21:36:15.832 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.832 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fRMWGBTZhB1UxE71VuSX5tieCNk= 2023-09-22T21:36:15.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fRMWGBTZhB1UxE71VuSX5tieCNk=', 'Content-Length': '0'} 2023-09-22T21:36:15.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f955dbe30260706-00650e08cf-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.835 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.835 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.836 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.836 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.836 DEBUG:boto:Method: HEAD 2023-09-22T21:36:15.836 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.836 DEBUG:boto:Data: 2023-09-22T21:36:15.836 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.836 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.836 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.836 DEBUG:boto:Params: {} 2023-09-22T21:36:15.836 DEBUG:boto:Token: None 2023-09-22T21:36:15.837 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.837 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fRMWGBTZhB1UxE71VuSX5tieCNk= 2023-09-22T21:36:15.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fRMWGBTZhB1UxE71VuSX5tieCNk=', 'Content-Length': '0'} 2023-09-22T21:36:15.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063c0afffefc44cea-00650e08cf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-09-22T21:36:15.840 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.840 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.840 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.840 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.840 DEBUG:boto:Method: GET 2023-09-22T21:36:15.841 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.841 DEBUG:boto:Data: 2023-09-22T21:36:15.841 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.841 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.841 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.841 DEBUG:boto:Params: {} 2023-09-22T21:36:15.841 DEBUG:boto:Token: None 2023-09-22T21:36:15.841 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.841 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GEIJEj0NqMpM8tSCcoiN2ddVUiw= 2023-09-22T21:36:15.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GEIJEj0NqMpM8tSCcoiN2ddVUiw=', 'Content-Length': '0'} 2023-09-22T21:36:15.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000729f01ac7fd37f2d-00650e08cf-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.886 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.886 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.887 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.887 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.887 DEBUG:boto:Method: GET 2023-09-22T21:36:15.887 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.887 DEBUG:boto:Data: 2023-09-22T21:36:15.887 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.887 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.887 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.887 DEBUG:boto:Params: {} 2023-09-22T21:36:15.887 DEBUG:boto:Token: None 2023-09-22T21:36:15.888 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000018?versionId=null 2023-09-22T21:36:15.888 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GEIJEj0NqMpM8tSCcoiN2ddVUiw= 2023-09-22T21:36:15.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GEIJEj0NqMpM8tSCcoiN2ddVUiw=', 'Content-Length': '0'} 2023-09-22T21:36:15.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085f8669cd65213b2-00650e08cf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.891 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:15.891 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:15.891 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000018 2023-09-22T21:36:15.891 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000018 2023-09-22T21:36:15.891 DEBUG:boto:Method: GET 2023-09-22T21:36:15.891 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000018 2023-09-22T21:36:15.891 DEBUG:boto:Data: 2023-09-22T21:36:15.892 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.892 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.892 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.892 DEBUG:boto:Params: {} 2023-09-22T21:36:15.892 DEBUG:boto:Token: None 2023-09-22T21:36:15.892 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/ 2023-09-22T21:36:15.892 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ= 2023-09-22T21:36:15.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ=', 'Content-Length': '0'} 2023-09-22T21:36:15.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f035bd40ef65854-00650e08cf-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.942 DEBUG:boto:b'ffzdqu-58obj000000000000000000181000falseobj000000000000000000182023-09-22T21:35:12.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:15.943 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:15.943 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:15.943 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000018 2023-09-22T21:36:15.943 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000018 2023-09-22T21:36:15.943 DEBUG:boto:Method: GET 2023-09-22T21:36:15.943 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000018 2023-09-22T21:36:15.943 DEBUG:boto:Data: 2023-09-22T21:36:15.944 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.944 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.944 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.944 DEBUG:boto:Params: {} 2023-09-22T21:36:15.944 DEBUG:boto:Token: None 2023-09-22T21:36:15.944 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/ 2023-09-22T21:36:15.944 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ= 2023-09-22T21:36:15.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2tM7EZ+141SYoEFlMuPi4uRe2cQ=', 'Content-Length': '0'} 2023-09-22T21:36:15.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d68445b831ecfd5c-00650e08cf-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.956 DEBUG:boto:b'ffzdqu-58obj000000000000000000181000falseobj000000000000000000182023-09-22T21:35:12.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:15.956 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.956 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.956 DEBUG:boto:Method: HEAD 2023-09-22T21:36:15.956 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.957 DEBUG:boto:Data: 2023-09-22T21:36:15.957 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.957 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.957 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.957 DEBUG:boto:Params: {} 2023-09-22T21:36:15.957 DEBUG:boto:Token: None 2023-09-22T21:36:15.957 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.957 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fl4hIt/ZvXX0XhfasYXJgIqEyWk= 2023-09-22T21:36:15.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fl4hIt/ZvXX0XhfasYXJgIqEyWk=', 'Content-Length': '0'} 2023-09-22T21:36:15.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9d20525af5bc15f-00650e08cf-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.961 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.961 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.962 DEBUG:boto:Method: HEAD 2023-09-22T21:36:15.962 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.962 DEBUG:boto:Data: 2023-09-22T21:36:15.962 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.962 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:15.962 DEBUG:boto:Port: 8000 2023-09-22T21:36:15.962 DEBUG:boto:Params: {} 2023-09-22T21:36:15.962 DEBUG:boto:Token: None 2023-09-22T21:36:15.962 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.963 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fl4hIt/ZvXX0XhfasYXJgIqEyWk= 2023-09-22T21:36:15.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fl4hIt/ZvXX0XhfasYXJgIqEyWk=', 'Content-Length': '0'} 2023-09-22T21:36:15.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbae548995aa2da8-00650e08cf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:15.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-09-22T21:36:15.966 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.966 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.966 DEBUG:boto:Method: GET 2023-09-22T21:36:15.966 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.966 DEBUG:boto:Data: 2023-09-22T21:36:15.966 DEBUG:boto:Headers: {} 2023-09-22T21:36:15.966 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:15.967 DEBUG:boto:Port: 8001 2023-09-22T21:36:15.967 DEBUG:boto:Params: {} 2023-09-22T21:36:15.967 DEBUG:boto:Token: None 2023-09-22T21:36:15.967 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:15 GMT /ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:15.967 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yGijRinf3su7/AhijFMabcgPiNo= 2023-09-22T21:36:15.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yGijRinf3su7/AhijFMabcgPiNo=', 'Content-Length': '0'} 2023-09-22T21:36:15.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000055956e4434a110a-00650e08cf-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.010 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:16.010 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:16.011 DEBUG:boto:Method: GET 2023-09-22T21:36:16.011 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:16.011 DEBUG:boto:Data: 2023-09-22T21:36:16.011 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.011 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.012 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.012 DEBUG:boto:Params: {} 2023-09-22T21:36:16.012 DEBUG:boto:Token: None 2023-09-22T21:36:16.012 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000018 2023-09-22T21:36:16.013 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GV0BtGAqWuMWekneMyuZgI8Uvts= 2023-09-22T21:36:16.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GV0BtGAqWuMWekneMyuZgI8Uvts=', 'Content-Length': '0'} 2023-09-22T21:36:16.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c4e6ca5b044669c-00650e08d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.016 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:16.016 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:16.016 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000018 2023-09-22T21:36:16.016 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000018 2023-09-22T21:36:16.016 DEBUG:boto:Method: GET 2023-09-22T21:36:16.017 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000018 2023-09-22T21:36:16.017 DEBUG:boto:Data: 2023-09-22T21:36:16.017 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.017 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.017 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.017 DEBUG:boto:Params: {} 2023-09-22T21:36:16.018 DEBUG:boto:Token: None 2023-09-22T21:36:16.018 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/ 2023-09-22T21:36:16.018 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k= 2023-09-22T21:36:16.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k=', 'Content-Length': '0'} 2023-09-22T21:36:16.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e11372c0036ddcd-00650e08d0-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.066 DEBUG:boto:b'ffzdqu-58obj000000000000000000181000falseobj000000000000000000182023-09-22T21:35:12.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:16.067 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:16.067 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:16.067 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000018 2023-09-22T21:36:16.067 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000018 2023-09-22T21:36:16.067 DEBUG:boto:Method: GET 2023-09-22T21:36:16.067 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000018 2023-09-22T21:36:16.068 DEBUG:boto:Data: 2023-09-22T21:36:16.068 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.068 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.068 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.068 DEBUG:boto:Params: {} 2023-09-22T21:36:16.068 DEBUG:boto:Token: None 2023-09-22T21:36:16.069 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/ 2023-09-22T21:36:16.069 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k= 2023-09-22T21:36:16.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k=', 'Content-Length': '0'} 2023-09-22T21:36:16.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5ee63c213f8ea79-00650e08d0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.079 DEBUG:boto:b'ffzdqu-58obj000000000000000000181000falseobj000000000000000000182023-09-22T21:35:12.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:16.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-09-22T21:36:16.080 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.080 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.080 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.081 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.081 DEBUG:boto:Method: GET 2023-09-22T21:36:16.081 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.081 DEBUG:boto:Data: 2023-09-22T21:36:16.081 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.082 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.082 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.082 DEBUG:boto:Params: {} 2023-09-22T21:36:16.082 DEBUG:boto:Token: None 2023-09-22T21:36:16.082 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.083 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:p0Jw8BENiNIggjwPzTGmV97k3KE= 2023-09-22T21:36:16.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:p0Jw8BENiNIggjwPzTGmV97k3KE=', 'Content-Length': '0'} 2023-09-22T21:36:16.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3e54a4055493ddc-00650e08d0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.087 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.087 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.087 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.087 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.087 DEBUG:boto:Method: GET 2023-09-22T21:36:16.087 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.088 DEBUG:boto:Data: 2023-09-22T21:36:16.088 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.088 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.088 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.088 DEBUG:boto:Params: {} 2023-09-22T21:36:16.088 DEBUG:boto:Token: None 2023-09-22T21:36:16.089 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.089 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:p0Jw8BENiNIggjwPzTGmV97k3KE= 2023-09-22T21:36:16.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:p0Jw8BENiNIggjwPzTGmV97k3KE=', 'Content-Length': '0'} 2023-09-22T21:36:16.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055016edfce280a8a-00650e08d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.134 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.135 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.135 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.135 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.135 DEBUG:boto:Method: HEAD 2023-09-22T21:36:16.135 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.135 DEBUG:boto:Data: 2023-09-22T21:36:16.135 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.135 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.135 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.135 DEBUG:boto:Params: {} 2023-09-22T21:36:16.136 DEBUG:boto:Token: None 2023-09-22T21:36:16.136 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.136 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9b59TcOwNJ2KMpz7I42mky39Y0E= 2023-09-22T21:36:16.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9b59TcOwNJ2KMpz7I42mky39Y0E=', 'Content-Length': '0'} 2023-09-22T21:36:16.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000357aea8edb8fb47d-00650e08d0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.139 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.140 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.140 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.140 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.140 DEBUG:boto:Method: HEAD 2023-09-22T21:36:16.140 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.140 DEBUG:boto:Data: 2023-09-22T21:36:16.141 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.141 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.141 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.141 DEBUG:boto:Params: {} 2023-09-22T21:36:16.141 DEBUG:boto:Token: None 2023-09-22T21:36:16.142 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.142 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9b59TcOwNJ2KMpz7I42mky39Y0E= 2023-09-22T21:36:16.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9b59TcOwNJ2KMpz7I42mky39Y0E=', 'Content-Length': '0'} 2023-09-22T21:36:16.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c8049cd02daf3cb-00650e08d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-09-22T21:36:16.145 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.146 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.146 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.146 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.146 DEBUG:boto:Method: GET 2023-09-22T21:36:16.146 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.147 DEBUG:boto:Data: 2023-09-22T21:36:16.147 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.147 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.147 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.147 DEBUG:boto:Params: {} 2023-09-22T21:36:16.148 DEBUG:boto:Token: None 2023-09-22T21:36:16.148 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.148 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:p0Jw8BENiNIggjwPzTGmV97k3KE= 2023-09-22T21:36:16.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:p0Jw8BENiNIggjwPzTGmV97k3KE=', 'Content-Length': '0'} 2023-09-22T21:36:16.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d45a302f88bf721-00650e08d0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.194 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.195 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.195 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.195 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.195 DEBUG:boto:Method: GET 2023-09-22T21:36:16.195 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.195 DEBUG:boto:Data: 2023-09-22T21:36:16.195 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.195 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.196 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.196 DEBUG:boto:Params: {} 2023-09-22T21:36:16.196 DEBUG:boto:Token: None 2023-09-22T21:36:16.196 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000019?versionId=null 2023-09-22T21:36:16.196 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:p0Jw8BENiNIggjwPzTGmV97k3KE= 2023-09-22T21:36:16.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:p0Jw8BENiNIggjwPzTGmV97k3KE=', 'Content-Length': '0'} 2023-09-22T21:36:16.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000318b35188e6cee2-00650e08d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.199 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:16.199 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:16.200 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000019 2023-09-22T21:36:16.200 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000019 2023-09-22T21:36:16.200 DEBUG:boto:Method: GET 2023-09-22T21:36:16.200 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000019 2023-09-22T21:36:16.200 DEBUG:boto:Data: 2023-09-22T21:36:16.200 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.200 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.200 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.200 DEBUG:boto:Params: {} 2023-09-22T21:36:16.200 DEBUG:boto:Token: None 2023-09-22T21:36:16.201 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/ 2023-09-22T21:36:16.201 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k= 2023-09-22T21:36:16.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k=', 'Content-Length': '0'} 2023-09-22T21:36:16.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f46a8a112b9800b-00650e08d0-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.246 DEBUG:boto:b'ffzdqu-58obj000000000000000000191000falseobj000000000000000000192023-09-22T21:35:05.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:16.247 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:16.247 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:16.247 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000019 2023-09-22T21:36:16.247 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000019 2023-09-22T21:36:16.247 DEBUG:boto:Method: GET 2023-09-22T21:36:16.247 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000019 2023-09-22T21:36:16.247 DEBUG:boto:Data: 2023-09-22T21:36:16.248 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.248 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.248 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.248 DEBUG:boto:Params: {} 2023-09-22T21:36:16.248 DEBUG:boto:Token: None 2023-09-22T21:36:16.248 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/ 2023-09-22T21:36:16.248 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k= 2023-09-22T21:36:16.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k=', 'Content-Length': '0'} 2023-09-22T21:36:16.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009df73413291df90-00650e08d0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.258 DEBUG:boto:b'ffzdqu-58obj000000000000000000191000falseobj000000000000000000192023-09-22T21:35:05.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:16.258 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.259 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.259 DEBUG:boto:Method: HEAD 2023-09-22T21:36:16.259 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.259 DEBUG:boto:Data: 2023-09-22T21:36:16.259 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.259 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.259 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.259 DEBUG:boto:Params: {} 2023-09-22T21:36:16.259 DEBUG:boto:Token: None 2023-09-22T21:36:16.260 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.260 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OWbT7ERwvQrmLNYg+mWBswn50b8= 2023-09-22T21:36:16.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OWbT7ERwvQrmLNYg+mWBswn50b8=', 'Content-Length': '0'} 2023-09-22T21:36:16.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000615b9fea8752e564-00650e08d0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.263 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.263 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.263 DEBUG:boto:Method: HEAD 2023-09-22T21:36:16.263 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.263 DEBUG:boto:Data: 2023-09-22T21:36:16.263 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.263 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.263 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.264 DEBUG:boto:Params: {} 2023-09-22T21:36:16.264 DEBUG:boto:Token: None 2023-09-22T21:36:16.264 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.264 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OWbT7ERwvQrmLNYg+mWBswn50b8= 2023-09-22T21:36:16.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OWbT7ERwvQrmLNYg+mWBswn50b8=', 'Content-Length': '0'} 2023-09-22T21:36:16.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046da5cc53898ef37-00650e08d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-09-22T21:36:16.268 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.268 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.268 DEBUG:boto:Method: GET 2023-09-22T21:36:16.268 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.268 DEBUG:boto:Data: 2023-09-22T21:36:16.268 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.268 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.268 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.268 DEBUG:boto:Params: {} 2023-09-22T21:36:16.269 DEBUG:boto:Token: None 2023-09-22T21:36:16.269 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.269 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E/z9M/WkmYAqbPWC5qel6VE3xsA= 2023-09-22T21:36:16.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E/z9M/WkmYAqbPWC5qel6VE3xsA=', 'Content-Length': '0'} 2023-09-22T21:36:16.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069212613da12b036-00650e08d0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.318 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.319 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.319 DEBUG:boto:Method: GET 2023-09-22T21:36:16.319 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.319 DEBUG:boto:Data: 2023-09-22T21:36:16.319 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.319 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.319 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.319 DEBUG:boto:Params: {} 2023-09-22T21:36:16.320 DEBUG:boto:Token: None 2023-09-22T21:36:16.320 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000019 2023-09-22T21:36:16.320 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E/z9M/WkmYAqbPWC5qel6VE3xsA= 2023-09-22T21:36:16.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E/z9M/WkmYAqbPWC5qel6VE3xsA=', 'Content-Length': '0'} 2023-09-22T21:36:16.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1c7343db3544c00-00650e08d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.323 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:16.323 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:16.323 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000019 2023-09-22T21:36:16.324 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000019 2023-09-22T21:36:16.324 DEBUG:boto:Method: GET 2023-09-22T21:36:16.324 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000019 2023-09-22T21:36:16.324 DEBUG:boto:Data: 2023-09-22T21:36:16.324 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.324 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.324 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.324 DEBUG:boto:Params: {} 2023-09-22T21:36:16.324 DEBUG:boto:Token: None 2023-09-22T21:36:16.325 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/ 2023-09-22T21:36:16.325 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k= 2023-09-22T21:36:16.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k=', 'Content-Length': '0'} 2023-09-22T21:36:16.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8c075a9a3db6ff6-00650e08d0-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.373 DEBUG:boto:b'ffzdqu-58obj000000000000000000191000falseobj000000000000000000192023-09-22T21:35:05.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:16.373 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:16.373 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:16.373 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000019 2023-09-22T21:36:16.374 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000019 2023-09-22T21:36:16.374 DEBUG:boto:Method: GET 2023-09-22T21:36:16.374 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000019 2023-09-22T21:36:16.374 DEBUG:boto:Data: 2023-09-22T21:36:16.374 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.374 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.374 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.374 DEBUG:boto:Params: {} 2023-09-22T21:36:16.374 DEBUG:boto:Token: None 2023-09-22T21:36:16.375 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/ 2023-09-22T21:36:16.375 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k= 2023-09-22T21:36:16.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k=', 'Content-Length': '0'} 2023-09-22T21:36:16.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096bcc6072be97ad4-00650e08d0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.385 DEBUG:boto:b'ffzdqu-58obj000000000000000000191000falseobj000000000000000000192023-09-22T21:35:05.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:16.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-09-22T21:36:16.385 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.385 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.386 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.386 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.386 DEBUG:boto:Method: GET 2023-09-22T21:36:16.386 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.386 DEBUG:boto:Data: 2023-09-22T21:36:16.386 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.386 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.386 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.386 DEBUG:boto:Params: {} 2023-09-22T21:36:16.387 DEBUG:boto:Token: None 2023-09-22T21:36:16.387 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.387 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rPKQ3TBxH6ZXaVgPF7UpJKz1jrA= 2023-09-22T21:36:16.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rPKQ3TBxH6ZXaVgPF7UpJKz1jrA=', 'Content-Length': '0'} 2023-09-22T21:36:16.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000271a7bb85af79697-00650e08d0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.390 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.390 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.391 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.391 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.391 DEBUG:boto:Method: GET 2023-09-22T21:36:16.391 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.391 DEBUG:boto:Data: 2023-09-22T21:36:16.391 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.391 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.391 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.392 DEBUG:boto:Params: {} 2023-09-22T21:36:16.392 DEBUG:boto:Token: None 2023-09-22T21:36:16.393 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.393 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rPKQ3TBxH6ZXaVgPF7UpJKz1jrA= 2023-09-22T21:36:16.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rPKQ3TBxH6ZXaVgPF7UpJKz1jrA=', 'Content-Length': '0'} 2023-09-22T21:36:16.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000974e6cefde6cb41e-00650e08d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.438 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.439 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.439 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.439 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.439 DEBUG:boto:Method: HEAD 2023-09-22T21:36:16.439 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.440 DEBUG:boto:Data: 2023-09-22T21:36:16.440 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.440 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.440 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.440 DEBUG:boto:Params: {} 2023-09-22T21:36:16.441 DEBUG:boto:Token: None 2023-09-22T21:36:16.441 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.441 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cBUdC86hlIzsxDNz/eC/aUtCs88= 2023-09-22T21:36:16.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cBUdC86hlIzsxDNz/eC/aUtCs88=', 'Content-Length': '0'} 2023-09-22T21:36:16.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037cf7b7e4e2c2a9c-00650e08d0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.444 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.445 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.445 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.445 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.445 DEBUG:boto:Method: HEAD 2023-09-22T21:36:16.445 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.445 DEBUG:boto:Data: 2023-09-22T21:36:16.446 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.446 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.446 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.446 DEBUG:boto:Params: {} 2023-09-22T21:36:16.446 DEBUG:boto:Token: None 2023-09-22T21:36:16.447 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.447 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cBUdC86hlIzsxDNz/eC/aUtCs88= 2023-09-22T21:36:16.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cBUdC86hlIzsxDNz/eC/aUtCs88=', 'Content-Length': '0'} 2023-09-22T21:36:16.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0ea9be8319b8bb8-00650e08d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-09-22T21:36:16.450 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.450 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.451 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.451 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.451 DEBUG:boto:Method: GET 2023-09-22T21:36:16.451 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.451 DEBUG:boto:Data: 2023-09-22T21:36:16.451 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.452 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.452 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.452 DEBUG:boto:Params: {} 2023-09-22T21:36:16.452 DEBUG:boto:Token: None 2023-09-22T21:36:16.452 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.452 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rPKQ3TBxH6ZXaVgPF7UpJKz1jrA= 2023-09-22T21:36:16.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rPKQ3TBxH6ZXaVgPF7UpJKz1jrA=', 'Content-Length': '0'} 2023-09-22T21:36:16.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbe1efe26d130ce1-00650e08d0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.498 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.498 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.499 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.499 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.499 DEBUG:boto:Method: GET 2023-09-22T21:36:16.499 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.499 DEBUG:boto:Data: 2023-09-22T21:36:16.499 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.499 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.499 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.499 DEBUG:boto:Params: {} 2023-09-22T21:36:16.500 DEBUG:boto:Token: None 2023-09-22T21:36:16.500 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000020?versionId=null 2023-09-22T21:36:16.500 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rPKQ3TBxH6ZXaVgPF7UpJKz1jrA= 2023-09-22T21:36:16.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rPKQ3TBxH6ZXaVgPF7UpJKz1jrA=', 'Content-Length': '0'} 2023-09-22T21:36:16.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000977510b2af64d3a4-00650e08d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.503 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:16.503 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:16.503 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000020 2023-09-22T21:36:16.504 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000020 2023-09-22T21:36:16.504 DEBUG:boto:Method: GET 2023-09-22T21:36:16.504 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000020 2023-09-22T21:36:16.504 DEBUG:boto:Data: 2023-09-22T21:36:16.504 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.504 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.504 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.504 DEBUG:boto:Params: {} 2023-09-22T21:36:16.504 DEBUG:boto:Token: None 2023-09-22T21:36:16.504 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/ 2023-09-22T21:36:16.505 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k= 2023-09-22T21:36:16.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k=', 'Content-Length': '0'} 2023-09-22T21:36:16.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca724a0380ec8fa8-00650e08d0-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.554 DEBUG:boto:b'ffzdqu-58obj000000000000000000201000falseobj000000000000000000202023-09-22T21:35:12.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:16.555 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:16.555 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:16.555 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000020 2023-09-22T21:36:16.555 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000020 2023-09-22T21:36:16.555 DEBUG:boto:Method: GET 2023-09-22T21:36:16.555 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000020 2023-09-22T21:36:16.556 DEBUG:boto:Data: 2023-09-22T21:36:16.556 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.556 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.556 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.556 DEBUG:boto:Params: {} 2023-09-22T21:36:16.556 DEBUG:boto:Token: None 2023-09-22T21:36:16.556 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/ 2023-09-22T21:36:16.556 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k= 2023-09-22T21:36:16.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k=', 'Content-Length': '0'} 2023-09-22T21:36:16.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4df0ce28c14fb66-00650e08d0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.571 DEBUG:boto:b'ffzdqu-58obj000000000000000000201000falseobj000000000000000000202023-09-22T21:35:12.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:16.571 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.571 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.571 DEBUG:boto:Method: HEAD 2023-09-22T21:36:16.571 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.571 DEBUG:boto:Data: 2023-09-22T21:36:16.571 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.572 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.572 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.572 DEBUG:boto:Params: {} 2023-09-22T21:36:16.572 DEBUG:boto:Token: None 2023-09-22T21:36:16.572 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.572 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZIjcvf2O461JwWTeFMnPupnI9wE= 2023-09-22T21:36:16.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZIjcvf2O461JwWTeFMnPupnI9wE=', 'Content-Length': '0'} 2023-09-22T21:36:16.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056dd4660d6fb5abc-00650e08d0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.577 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.577 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.577 DEBUG:boto:Method: HEAD 2023-09-22T21:36:16.577 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.578 DEBUG:boto:Data: 2023-09-22T21:36:16.578 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.578 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.578 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.578 DEBUG:boto:Params: {} 2023-09-22T21:36:16.578 DEBUG:boto:Token: None 2023-09-22T21:36:16.578 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.578 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZIjcvf2O461JwWTeFMnPupnI9wE= 2023-09-22T21:36:16.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZIjcvf2O461JwWTeFMnPupnI9wE=', 'Content-Length': '0'} 2023-09-22T21:36:16.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acc27f8c77808282-00650e08d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-09-22T21:36:16.581 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.581 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.581 DEBUG:boto:Method: GET 2023-09-22T21:36:16.582 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.582 DEBUG:boto:Data: 2023-09-22T21:36:16.582 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.582 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.582 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.582 DEBUG:boto:Params: {} 2023-09-22T21:36:16.582 DEBUG:boto:Token: None 2023-09-22T21:36:16.582 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.582 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GG+9JYkhqY5gUmtxKXD4sHt7UiY= 2023-09-22T21:36:16.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GG+9JYkhqY5gUmtxKXD4sHt7UiY=', 'Content-Length': '0'} 2023-09-22T21:36:16.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000282d883269dbe29b-00650e08d0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.626 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.626 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.627 DEBUG:boto:Method: GET 2023-09-22T21:36:16.627 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.627 DEBUG:boto:Data: 2023-09-22T21:36:16.627 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.627 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.627 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.627 DEBUG:boto:Params: {} 2023-09-22T21:36:16.627 DEBUG:boto:Token: None 2023-09-22T21:36:16.627 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000020 2023-09-22T21:36:16.628 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GG+9JYkhqY5gUmtxKXD4sHt7UiY= 2023-09-22T21:36:16.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GG+9JYkhqY5gUmtxKXD4sHt7UiY=', 'Content-Length': '0'} 2023-09-22T21:36:16.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca1c00a8c071ca01-00650e08d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.631 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:16.631 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:16.631 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000020 2023-09-22T21:36:16.631 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000020 2023-09-22T21:36:16.631 DEBUG:boto:Method: GET 2023-09-22T21:36:16.631 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000020 2023-09-22T21:36:16.631 DEBUG:boto:Data: 2023-09-22T21:36:16.631 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.631 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.631 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.632 DEBUG:boto:Params: {} 2023-09-22T21:36:16.632 DEBUG:boto:Token: None 2023-09-22T21:36:16.632 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/ 2023-09-22T21:36:16.632 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k= 2023-09-22T21:36:16.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k=', 'Content-Length': '0'} 2023-09-22T21:36:16.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5e3ee4736d43eee-00650e08d0-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.682 DEBUG:boto:b'ffzdqu-58obj000000000000000000201000falseobj000000000000000000202023-09-22T21:35:12.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:16.683 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:16.683 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:16.683 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000020 2023-09-22T21:36:16.683 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000020 2023-09-22T21:36:16.683 DEBUG:boto:Method: GET 2023-09-22T21:36:16.684 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000020 2023-09-22T21:36:16.684 DEBUG:boto:Data: 2023-09-22T21:36:16.684 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.684 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.684 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.684 DEBUG:boto:Params: {} 2023-09-22T21:36:16.684 DEBUG:boto:Token: None 2023-09-22T21:36:16.684 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/ 2023-09-22T21:36:16.684 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k= 2023-09-22T21:36:16.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k=', 'Content-Length': '0'} 2023-09-22T21:36:16.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c349da213e0170e-00650e08d0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.696 DEBUG:boto:b'ffzdqu-58obj000000000000000000201000falseobj000000000000000000202023-09-22T21:35:12.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:16.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-09-22T21:36:16.696 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.697 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.697 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.697 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.697 DEBUG:boto:Method: GET 2023-09-22T21:36:16.697 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.697 DEBUG:boto:Data: 2023-09-22T21:36:16.697 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.697 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.697 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.697 DEBUG:boto:Params: {} 2023-09-22T21:36:16.698 DEBUG:boto:Token: None 2023-09-22T21:36:16.698 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.698 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ec/C9woAlVn3fDPsV1Ay6ZXQjMo= 2023-09-22T21:36:16.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ec/C9woAlVn3fDPsV1Ay6ZXQjMo=', 'Content-Length': '0'} 2023-09-22T21:36:16.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043e60bde58935469-00650e08d0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.702 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.702 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.702 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.702 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.702 DEBUG:boto:Method: GET 2023-09-22T21:36:16.702 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.702 DEBUG:boto:Data: 2023-09-22T21:36:16.702 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.702 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.703 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.703 DEBUG:boto:Params: {} 2023-09-22T21:36:16.703 DEBUG:boto:Token: None 2023-09-22T21:36:16.703 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.703 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ec/C9woAlVn3fDPsV1Ay6ZXQjMo= 2023-09-22T21:36:16.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ec/C9woAlVn3fDPsV1Ay6ZXQjMo=', 'Content-Length': '0'} 2023-09-22T21:36:16.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038454cbdba4e25f0-00650e08d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.746 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.746 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.747 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.747 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.747 DEBUG:boto:Method: HEAD 2023-09-22T21:36:16.747 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.747 DEBUG:boto:Data: 2023-09-22T21:36:16.747 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.747 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.747 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.747 DEBUG:boto:Params: {} 2023-09-22T21:36:16.748 DEBUG:boto:Token: None 2023-09-22T21:36:16.748 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.748 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AcgNAIoZ23WMUdFhsyP+iaS1ar4= 2023-09-22T21:36:16.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AcgNAIoZ23WMUdFhsyP+iaS1ar4=', 'Content-Length': '0'} 2023-09-22T21:36:16.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc62fbb79b6eb99d-00650e08d0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.751 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.751 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.751 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.751 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.751 DEBUG:boto:Method: HEAD 2023-09-22T21:36:16.752 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.752 DEBUG:boto:Data: 2023-09-22T21:36:16.752 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.752 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.752 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.752 DEBUG:boto:Params: {} 2023-09-22T21:36:16.752 DEBUG:boto:Token: None 2023-09-22T21:36:16.752 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.752 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AcgNAIoZ23WMUdFhsyP+iaS1ar4= 2023-09-22T21:36:16.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AcgNAIoZ23WMUdFhsyP+iaS1ar4=', 'Content-Length': '0'} 2023-09-22T21:36:16.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068211eb90c4d3f87-00650e08d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-09-22T21:36:16.755 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.755 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.756 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.756 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.756 DEBUG:boto:Method: GET 2023-09-22T21:36:16.756 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.756 DEBUG:boto:Data: 2023-09-22T21:36:16.756 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.756 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.756 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.756 DEBUG:boto:Params: {} 2023-09-22T21:36:16.757 DEBUG:boto:Token: None 2023-09-22T21:36:16.757 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.757 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ec/C9woAlVn3fDPsV1Ay6ZXQjMo= 2023-09-22T21:36:16.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ec/C9woAlVn3fDPsV1Ay6ZXQjMo=', 'Content-Length': '0'} 2023-09-22T21:36:16.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a145bac7ccbec6c7-00650e08d0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.802 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.802 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.803 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.803 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.803 DEBUG:boto:Method: GET 2023-09-22T21:36:16.803 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.803 DEBUG:boto:Data: 2023-09-22T21:36:16.803 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.803 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.803 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.803 DEBUG:boto:Params: {} 2023-09-22T21:36:16.804 DEBUG:boto:Token: None 2023-09-22T21:36:16.804 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000021?versionId=null 2023-09-22T21:36:16.804 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ec/C9woAlVn3fDPsV1Ay6ZXQjMo= 2023-09-22T21:36:16.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ec/C9woAlVn3fDPsV1Ay6ZXQjMo=', 'Content-Length': '0'} 2023-09-22T21:36:16.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048c59f72a9b8ffd6-00650e08d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.807 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:16.807 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:16.807 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000021 2023-09-22T21:36:16.808 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000021 2023-09-22T21:36:16.808 DEBUG:boto:Method: GET 2023-09-22T21:36:16.808 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000021 2023-09-22T21:36:16.808 DEBUG:boto:Data: 2023-09-22T21:36:16.808 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.808 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.808 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.808 DEBUG:boto:Params: {} 2023-09-22T21:36:16.808 DEBUG:boto:Token: None 2023-09-22T21:36:16.809 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/ 2023-09-22T21:36:16.809 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k= 2023-09-22T21:36:16.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k=', 'Content-Length': '0'} 2023-09-22T21:36:16.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007da0e8902786f460-00650e08d0-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.854 DEBUG:boto:b'ffzdqu-58obj000000000000000000211000falseobj000000000000000000212023-09-22T21:35:05.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:16.855 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:16.855 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:16.855 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000021 2023-09-22T21:36:16.855 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000021 2023-09-22T21:36:16.855 DEBUG:boto:Method: GET 2023-09-22T21:36:16.855 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000021 2023-09-22T21:36:16.855 DEBUG:boto:Data: 2023-09-22T21:36:16.855 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.856 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.856 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.856 DEBUG:boto:Params: {} 2023-09-22T21:36:16.856 DEBUG:boto:Token: None 2023-09-22T21:36:16.856 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/ 2023-09-22T21:36:16.856 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k= 2023-09-22T21:36:16.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k=', 'Content-Length': '0'} 2023-09-22T21:36:16.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000034bd7f6436f9423-00650e08d0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.870 DEBUG:boto:b'ffzdqu-58obj000000000000000000211000falseobj000000000000000000212023-09-22T21:35:05.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:16.870 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.870 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.870 DEBUG:boto:Method: HEAD 2023-09-22T21:36:16.871 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.871 DEBUG:boto:Data: 2023-09-22T21:36:16.871 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.871 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.871 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.871 DEBUG:boto:Params: {} 2023-09-22T21:36:16.871 DEBUG:boto:Token: None 2023-09-22T21:36:16.871 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.871 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:V1COSFDXXQYMP+5V9XzELaUCO+A= 2023-09-22T21:36:16.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:V1COSFDXXQYMP+5V9XzELaUCO+A=', 'Content-Length': '0'} 2023-09-22T21:36:16.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0865ed1695fb161-00650e08d0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.875 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.875 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.875 DEBUG:boto:Method: HEAD 2023-09-22T21:36:16.875 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.875 DEBUG:boto:Data: 2023-09-22T21:36:16.876 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.876 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.876 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.876 DEBUG:boto:Params: {} 2023-09-22T21:36:16.876 DEBUG:boto:Token: None 2023-09-22T21:36:16.876 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.876 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:V1COSFDXXQYMP+5V9XzELaUCO+A= 2023-09-22T21:36:16.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:V1COSFDXXQYMP+5V9XzELaUCO+A=', 'Content-Length': '0'} 2023-09-22T21:36:16.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b8966a1c53ceeb9-00650e08d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-09-22T21:36:16.879 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.879 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.880 DEBUG:boto:Method: GET 2023-09-22T21:36:16.880 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.880 DEBUG:boto:Data: 2023-09-22T21:36:16.880 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.880 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.880 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.880 DEBUG:boto:Params: {} 2023-09-22T21:36:16.880 DEBUG:boto:Token: None 2023-09-22T21:36:16.881 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.881 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3HRLLRAbJVC8PmVEX3ofb3qpTUs= 2023-09-22T21:36:16.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3HRLLRAbJVC8PmVEX3ofb3qpTUs=', 'Content-Length': '0'} 2023-09-22T21:36:16.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038c71e1dcdf8cc35-00650e08d0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.926 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.926 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.926 DEBUG:boto:Method: GET 2023-09-22T21:36:16.927 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.927 DEBUG:boto:Data: 2023-09-22T21:36:16.927 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.927 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.927 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.927 DEBUG:boto:Params: {} 2023-09-22T21:36:16.927 DEBUG:boto:Token: None 2023-09-22T21:36:16.927 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000021 2023-09-22T21:36:16.928 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3HRLLRAbJVC8PmVEX3ofb3qpTUs= 2023-09-22T21:36:16.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3HRLLRAbJVC8PmVEX3ofb3qpTUs=', 'Content-Length': '0'} 2023-09-22T21:36:16.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c24598c2c54d077b-00650e08d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.931 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:16.931 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:16.932 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000021 2023-09-22T21:36:16.932 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000021 2023-09-22T21:36:16.932 DEBUG:boto:Method: GET 2023-09-22T21:36:16.932 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000021 2023-09-22T21:36:16.932 DEBUG:boto:Data: 2023-09-22T21:36:16.932 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.932 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.932 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.932 DEBUG:boto:Params: {} 2023-09-22T21:36:16.933 DEBUG:boto:Token: None 2023-09-22T21:36:16.933 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/ 2023-09-22T21:36:16.933 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k= 2023-09-22T21:36:16.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k=', 'Content-Length': '0'} 2023-09-22T21:36:16.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027b4f6dd94a3608d-00650e08d0-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.978 DEBUG:boto:b'ffzdqu-58obj000000000000000000211000falseobj000000000000000000212023-09-22T21:35:05.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:16.979 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:16.979 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:16.979 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000021 2023-09-22T21:36:16.979 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000021 2023-09-22T21:36:16.979 DEBUG:boto:Method: GET 2023-09-22T21:36:16.980 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000021 2023-09-22T21:36:16.980 DEBUG:boto:Data: 2023-09-22T21:36:16.980 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.980 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.980 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.980 DEBUG:boto:Params: {} 2023-09-22T21:36:16.980 DEBUG:boto:Token: None 2023-09-22T21:36:16.980 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/ 2023-09-22T21:36:16.981 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k= 2023-09-22T21:36:16.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xbGwiqfc06qBsnLdta5QrbwPk/k=', 'Content-Length': '0'} 2023-09-22T21:36:16.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e8e48d044642f2c-00650e08d0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.990 DEBUG:boto:b'ffzdqu-58obj000000000000000000211000falseobj000000000000000000212023-09-22T21:35:05.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:16.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-09-22T21:36:16.991 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:16.991 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:16.991 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:16.991 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:16.991 DEBUG:boto:Method: GET 2023-09-22T21:36:16.992 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:16.992 DEBUG:boto:Data: 2023-09-22T21:36:16.992 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.992 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:16.992 DEBUG:boto:Port: 8001 2023-09-22T21:36:16.992 DEBUG:boto:Params: {} 2023-09-22T21:36:16.992 DEBUG:boto:Token: None 2023-09-22T21:36:16.992 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:16.993 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LRsH0PLBg69zw5eQBl2wfdHETIo= 2023-09-22T21:36:16.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LRsH0PLBg69zw5eQBl2wfdHETIo=', 'Content-Length': '0'} 2023-09-22T21:36:16.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dec199a1759e6ea-00650e08d0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:16.996 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:16.996 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:16.996 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:16.996 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:16.997 DEBUG:boto:Method: GET 2023-09-22T21:36:16.997 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:16.997 DEBUG:boto:Data: 2023-09-22T21:36:16.997 DEBUG:boto:Headers: {} 2023-09-22T21:36:16.997 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:16.997 DEBUG:boto:Port: 8000 2023-09-22T21:36:16.997 DEBUG:boto:Params: {} 2023-09-22T21:36:16.997 DEBUG:boto:Token: None 2023-09-22T21:36:16.997 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:16 GMT /ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:16.998 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LRsH0PLBg69zw5eQBl2wfdHETIo= 2023-09-22T21:36:16.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LRsH0PLBg69zw5eQBl2wfdHETIo=', 'Content-Length': '0'} 2023-09-22T21:36:17.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017d31c026a5bba4f-00650e08d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.042 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.043 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.043 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:17.043 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:17.044 DEBUG:boto:Method: HEAD 2023-09-22T21:36:17.044 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:17.044 DEBUG:boto:Data: 2023-09-22T21:36:17.044 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.044 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.044 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.045 DEBUG:boto:Params: {} 2023-09-22T21:36:17.045 DEBUG:boto:Token: None 2023-09-22T21:36:17.045 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:17.045 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pMoXSyPW2qwpnG5AQdo1HQg2kpk= 2023-09-22T21:36:17.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pMoXSyPW2qwpnG5AQdo1HQg2kpk=', 'Content-Length': '0'} 2023-09-22T21:36:17.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000719cb2c906ca45e1-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.049 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.049 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.049 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:17.049 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:17.050 DEBUG:boto:Method: HEAD 2023-09-22T21:36:17.050 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:17.050 DEBUG:boto:Data: 2023-09-22T21:36:17.050 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.050 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.050 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.050 DEBUG:boto:Params: {} 2023-09-22T21:36:17.050 DEBUG:boto:Token: None 2023-09-22T21:36:17.051 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:17.051 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pMoXSyPW2qwpnG5AQdo1HQg2kpk= 2023-09-22T21:36:17.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pMoXSyPW2qwpnG5AQdo1HQg2kpk=', 'Content-Length': '0'} 2023-09-22T21:36:17.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000accaebaac9431d36-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-09-22T21:36:17.054 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.054 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.054 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:17.054 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:17.054 DEBUG:boto:Method: GET 2023-09-22T21:36:17.054 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:17.054 DEBUG:boto:Data: 2023-09-22T21:36:17.055 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.055 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.055 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.055 DEBUG:boto:Params: {} 2023-09-22T21:36:17.055 DEBUG:boto:Token: None 2023-09-22T21:36:17.055 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:17.055 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kRlGF760649bpE8cdSW5B6SkOZ0= 2023-09-22T21:36:17.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kRlGF760649bpE8cdSW5B6SkOZ0=', 'Content-Length': '0'} 2023-09-22T21:36:17.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006df9a4420938fa20-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.098 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.098 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.099 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:17.099 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:17.099 DEBUG:boto:Method: GET 2023-09-22T21:36:17.099 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:17.099 DEBUG:boto:Data: 2023-09-22T21:36:17.099 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.099 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.099 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.099 DEBUG:boto:Params: {} 2023-09-22T21:36:17.100 DEBUG:boto:Token: None 2023-09-22T21:36:17.100 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000022?versionId=null 2023-09-22T21:36:17.100 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kRlGF760649bpE8cdSW5B6SkOZ0= 2023-09-22T21:36:17.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kRlGF760649bpE8cdSW5B6SkOZ0=', 'Content-Length': '0'} 2023-09-22T21:36:17.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077b63d20dbf75193-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.103 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:17.103 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:17.103 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000022 2023-09-22T21:36:17.103 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000022 2023-09-22T21:36:17.104 DEBUG:boto:Method: GET 2023-09-22T21:36:17.104 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000022 2023-09-22T21:36:17.104 DEBUG:boto:Data: 2023-09-22T21:36:17.104 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.104 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.104 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.104 DEBUG:boto:Params: {} 2023-09-22T21:36:17.104 DEBUG:boto:Token: None 2023-09-22T21:36:17.104 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/ 2023-09-22T21:36:17.105 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc= 2023-09-22T21:36:17.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc=', 'Content-Length': '0'} 2023-09-22T21:36:17.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009920bdf004ab1ffc-00650e08d1-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.109 DEBUG:boto:b'ffzdqu-58obj000000000000000000221000falseobj000000000000000000222023-09-22T21:35:03.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:17.110 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:17.110 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:17.110 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000022 2023-09-22T21:36:17.110 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000022 2023-09-22T21:36:17.110 DEBUG:boto:Method: GET 2023-09-22T21:36:17.110 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000022 2023-09-22T21:36:17.110 DEBUG:boto:Data: 2023-09-22T21:36:17.111 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.111 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.111 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.111 DEBUG:boto:Params: {} 2023-09-22T21:36:17.111 DEBUG:boto:Token: None 2023-09-22T21:36:17.112 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/ 2023-09-22T21:36:17.112 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc= 2023-09-22T21:36:17.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc=', 'Content-Length': '0'} 2023-09-22T21:36:17.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053ee6800c223effa-00650e08d1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.122 DEBUG:boto:b'ffzdqu-58obj000000000000000000221000falseobj000000000000000000222023-09-22T21:35:03.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:17.123 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.123 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.123 DEBUG:boto:Method: HEAD 2023-09-22T21:36:17.123 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.124 DEBUG:boto:Data: 2023-09-22T21:36:17.124 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.124 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.124 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.124 DEBUG:boto:Params: {} 2023-09-22T21:36:17.124 DEBUG:boto:Token: None 2023-09-22T21:36:17.125 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.125 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3vBMOQ0T17ZhKpjst4HI2/P7KAM= 2023-09-22T21:36:17.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3vBMOQ0T17ZhKpjst4HI2/P7KAM=', 'Content-Length': '0'} 2023-09-22T21:36:17.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009979afeea236910e-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.128 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.128 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.129 DEBUG:boto:Method: HEAD 2023-09-22T21:36:17.129 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.129 DEBUG:boto:Data: 2023-09-22T21:36:17.129 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.129 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.129 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.130 DEBUG:boto:Params: {} 2023-09-22T21:36:17.130 DEBUG:boto:Token: None 2023-09-22T21:36:17.130 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.130 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3vBMOQ0T17ZhKpjst4HI2/P7KAM= 2023-09-22T21:36:17.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3vBMOQ0T17ZhKpjst4HI2/P7KAM=', 'Content-Length': '0'} 2023-09-22T21:36:17.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000116a2016d7195f29-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-09-22T21:36:17.133 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.133 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.133 DEBUG:boto:Method: GET 2023-09-22T21:36:17.133 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.134 DEBUG:boto:Data: 2023-09-22T21:36:17.134 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.134 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.134 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.134 DEBUG:boto:Params: {} 2023-09-22T21:36:17.134 DEBUG:boto:Token: None 2023-09-22T21:36:17.134 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.135 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CgNIufgvDtQ5ZbLrOQbcINleA4s= 2023-09-22T21:36:17.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CgNIufgvDtQ5ZbLrOQbcINleA4s=', 'Content-Length': '0'} 2023-09-22T21:36:17.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d12667ac1c2a62bd-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.178 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.178 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.179 DEBUG:boto:Method: GET 2023-09-22T21:36:17.179 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.179 DEBUG:boto:Data: 2023-09-22T21:36:17.179 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.179 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.179 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.179 DEBUG:boto:Params: {} 2023-09-22T21:36:17.179 DEBUG:boto:Token: None 2023-09-22T21:36:17.180 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000022 2023-09-22T21:36:17.180 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CgNIufgvDtQ5ZbLrOQbcINleA4s= 2023-09-22T21:36:17.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CgNIufgvDtQ5ZbLrOQbcINleA4s=', 'Content-Length': '0'} 2023-09-22T21:36:17.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000749e2495a270d393-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.183 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:17.183 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:17.183 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000022 2023-09-22T21:36:17.184 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000022 2023-09-22T21:36:17.184 DEBUG:boto:Method: GET 2023-09-22T21:36:17.184 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000022 2023-09-22T21:36:17.184 DEBUG:boto:Data: 2023-09-22T21:36:17.184 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.184 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.184 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.185 DEBUG:boto:Params: {} 2023-09-22T21:36:17.185 DEBUG:boto:Token: None 2023-09-22T21:36:17.185 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/ 2023-09-22T21:36:17.185 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc= 2023-09-22T21:36:17.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc=', 'Content-Length': '0'} 2023-09-22T21:36:17.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5e8fcc26eb3f337-00650e08d1-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.190 DEBUG:boto:b'ffzdqu-58obj000000000000000000221000falseobj000000000000000000222023-09-22T21:35:03.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:17.190 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:17.191 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:17.191 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000022 2023-09-22T21:36:17.191 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000022 2023-09-22T21:36:17.191 DEBUG:boto:Method: GET 2023-09-22T21:36:17.191 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000022 2023-09-22T21:36:17.191 DEBUG:boto:Data: 2023-09-22T21:36:17.191 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.191 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.191 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.192 DEBUG:boto:Params: {} 2023-09-22T21:36:17.192 DEBUG:boto:Token: None 2023-09-22T21:36:17.192 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/ 2023-09-22T21:36:17.192 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc= 2023-09-22T21:36:17.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc=', 'Content-Length': '0'} 2023-09-22T21:36:17.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a1e9e5037de57ce-00650e08d1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.202 DEBUG:boto:b'ffzdqu-58obj000000000000000000221000falseobj000000000000000000222023-09-22T21:35:03.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:17.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-09-22T21:36:17.203 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.203 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.203 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.203 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.203 DEBUG:boto:Method: GET 2023-09-22T21:36:17.203 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.203 DEBUG:boto:Data: 2023-09-22T21:36:17.203 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.203 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.204 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.204 DEBUG:boto:Params: {} 2023-09-22T21:36:17.204 DEBUG:boto:Token: None 2023-09-22T21:36:17.204 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.204 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+Hmip0b2z/LI7PMz0knen8VZ1qs= 2023-09-22T21:36:17.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+Hmip0b2z/LI7PMz0knen8VZ1qs=', 'Content-Length': '0'} 2023-09-22T21:36:17.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b2a32fdb24b7768-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.207 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.208 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.208 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.208 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.208 DEBUG:boto:Method: GET 2023-09-22T21:36:17.208 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.208 DEBUG:boto:Data: 2023-09-22T21:36:17.208 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.208 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.209 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.209 DEBUG:boto:Params: {} 2023-09-22T21:36:17.209 DEBUG:boto:Token: None 2023-09-22T21:36:17.209 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.209 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+Hmip0b2z/LI7PMz0knen8VZ1qs= 2023-09-22T21:36:17.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+Hmip0b2z/LI7PMz0knen8VZ1qs=', 'Content-Length': '0'} 2023-09-22T21:36:17.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e7fea19e5c6b31a-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.212 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.212 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.213 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.213 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.213 DEBUG:boto:Method: HEAD 2023-09-22T21:36:17.213 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.213 DEBUG:boto:Data: 2023-09-22T21:36:17.213 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.213 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.213 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.214 DEBUG:boto:Params: {} 2023-09-22T21:36:17.214 DEBUG:boto:Token: None 2023-09-22T21:36:17.214 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.214 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9X00h1AfXFkPkx3/9h5oiqz8nzY= 2023-09-22T21:36:17.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9X00h1AfXFkPkx3/9h5oiqz8nzY=', 'Content-Length': '0'} 2023-09-22T21:36:17.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bbb0ee8dcb43a81-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.217 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.217 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.217 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.217 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.218 DEBUG:boto:Method: HEAD 2023-09-22T21:36:17.218 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.218 DEBUG:boto:Data: 2023-09-22T21:36:17.218 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.218 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.218 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.218 DEBUG:boto:Params: {} 2023-09-22T21:36:17.218 DEBUG:boto:Token: None 2023-09-22T21:36:17.219 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.219 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9X00h1AfXFkPkx3/9h5oiqz8nzY= 2023-09-22T21:36:17.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9X00h1AfXFkPkx3/9h5oiqz8nzY=', 'Content-Length': '0'} 2023-09-22T21:36:17.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034e65d0c2c34c306-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-09-22T21:36:17.222 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.222 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.222 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.222 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.222 DEBUG:boto:Method: GET 2023-09-22T21:36:17.222 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.222 DEBUG:boto:Data: 2023-09-22T21:36:17.222 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.223 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.223 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.223 DEBUG:boto:Params: {} 2023-09-22T21:36:17.223 DEBUG:boto:Token: None 2023-09-22T21:36:17.223 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.223 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+Hmip0b2z/LI7PMz0knen8VZ1qs= 2023-09-22T21:36:17.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+Hmip0b2z/LI7PMz0knen8VZ1qs=', 'Content-Length': '0'} 2023-09-22T21:36:17.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000307c446a1aeeb9ab-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.266 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.267 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.267 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.267 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.267 DEBUG:boto:Method: GET 2023-09-22T21:36:17.267 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.267 DEBUG:boto:Data: 2023-09-22T21:36:17.267 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.268 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.268 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.268 DEBUG:boto:Params: {} 2023-09-22T21:36:17.268 DEBUG:boto:Token: None 2023-09-22T21:36:17.268 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000023?versionId=null 2023-09-22T21:36:17.268 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+Hmip0b2z/LI7PMz0knen8VZ1qs= 2023-09-22T21:36:17.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+Hmip0b2z/LI7PMz0knen8VZ1qs=', 'Content-Length': '0'} 2023-09-22T21:36:17.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecc49fc90e9dd019-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.273 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:17.273 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:17.273 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000023 2023-09-22T21:36:17.273 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000023 2023-09-22T21:36:17.273 DEBUG:boto:Method: GET 2023-09-22T21:36:17.274 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000023 2023-09-22T21:36:17.274 DEBUG:boto:Data: 2023-09-22T21:36:17.274 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.274 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.274 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.274 DEBUG:boto:Params: {} 2023-09-22T21:36:17.274 DEBUG:boto:Token: None 2023-09-22T21:36:17.274 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/ 2023-09-22T21:36:17.275 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc= 2023-09-22T21:36:17.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc=', 'Content-Length': '0'} 2023-09-22T21:36:17.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c765cd58bb9de1f-00650e08d1-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.281 DEBUG:boto:b'ffzdqu-58obj000000000000000000231000falseobj000000000000000000232023-09-22T21:35:03.585Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:17.281 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:17.281 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:17.281 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000023 2023-09-22T21:36:17.281 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000023 2023-09-22T21:36:17.282 DEBUG:boto:Method: GET 2023-09-22T21:36:17.282 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000023 2023-09-22T21:36:17.282 DEBUG:boto:Data: 2023-09-22T21:36:17.282 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.282 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.282 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.282 DEBUG:boto:Params: {} 2023-09-22T21:36:17.282 DEBUG:boto:Token: None 2023-09-22T21:36:17.282 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/ 2023-09-22T21:36:17.283 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc= 2023-09-22T21:36:17.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc=', 'Content-Length': '0'} 2023-09-22T21:36:17.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a16431deec7df88c-00650e08d1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.295 DEBUG:boto:b'ffzdqu-58obj000000000000000000231000falseobj000000000000000000232023-09-22T21:35:03.585Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:17.295 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.295 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.295 DEBUG:boto:Method: HEAD 2023-09-22T21:36:17.296 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.296 DEBUG:boto:Data: 2023-09-22T21:36:17.296 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.296 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.296 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.296 DEBUG:boto:Params: {} 2023-09-22T21:36:17.296 DEBUG:boto:Token: None 2023-09-22T21:36:17.296 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.297 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jMBBokZO3lwIp/zdq7HxdOTM0bE= 2023-09-22T21:36:17.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jMBBokZO3lwIp/zdq7HxdOTM0bE=', 'Content-Length': '0'} 2023-09-22T21:36:17.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaf0481c5e5e2a44-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.299 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.299 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.299 DEBUG:boto:Method: HEAD 2023-09-22T21:36:17.300 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.300 DEBUG:boto:Data: 2023-09-22T21:36:17.300 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.300 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.300 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.300 DEBUG:boto:Params: {} 2023-09-22T21:36:17.300 DEBUG:boto:Token: None 2023-09-22T21:36:17.300 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.301 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jMBBokZO3lwIp/zdq7HxdOTM0bE= 2023-09-22T21:36:17.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jMBBokZO3lwIp/zdq7HxdOTM0bE=', 'Content-Length': '0'} 2023-09-22T21:36:17.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6484d46f07517cf-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-09-22T21:36:17.303 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.304 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.304 DEBUG:boto:Method: GET 2023-09-22T21:36:17.304 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.304 DEBUG:boto:Data: 2023-09-22T21:36:17.304 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.304 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.304 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.304 DEBUG:boto:Params: {} 2023-09-22T21:36:17.304 DEBUG:boto:Token: None 2023-09-22T21:36:17.305 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.305 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1+0FcQlGfPD8HX8BI+pAJzigx84= 2023-09-22T21:36:17.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1+0FcQlGfPD8HX8BI+pAJzigx84=', 'Content-Length': '0'} 2023-09-22T21:36:17.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f52ab21fb7891aa-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.350 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.351 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.351 DEBUG:boto:Method: GET 2023-09-22T21:36:17.351 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.351 DEBUG:boto:Data: 2023-09-22T21:36:17.351 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.351 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.351 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.351 DEBUG:boto:Params: {} 2023-09-22T21:36:17.351 DEBUG:boto:Token: None 2023-09-22T21:36:17.352 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000023 2023-09-22T21:36:17.352 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1+0FcQlGfPD8HX8BI+pAJzigx84= 2023-09-22T21:36:17.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1+0FcQlGfPD8HX8BI+pAJzigx84=', 'Content-Length': '0'} 2023-09-22T21:36:17.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007576c17a39575da4-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.355 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:17.355 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:17.355 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000023 2023-09-22T21:36:17.355 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000023 2023-09-22T21:36:17.355 DEBUG:boto:Method: GET 2023-09-22T21:36:17.355 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000023 2023-09-22T21:36:17.355 DEBUG:boto:Data: 2023-09-22T21:36:17.355 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.356 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.356 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.356 DEBUG:boto:Params: {} 2023-09-22T21:36:17.356 DEBUG:boto:Token: None 2023-09-22T21:36:17.356 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/ 2023-09-22T21:36:17.356 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc= 2023-09-22T21:36:17.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc=', 'Content-Length': '0'} 2023-09-22T21:36:17.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b42c52d8c35b273-00650e08d1-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.402 DEBUG:boto:b'ffzdqu-58obj000000000000000000231000falseobj000000000000000000232023-09-22T21:35:03.585Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:17.403 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:17.403 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:17.403 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000023 2023-09-22T21:36:17.403 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000023 2023-09-22T21:36:17.403 DEBUG:boto:Method: GET 2023-09-22T21:36:17.403 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000023 2023-09-22T21:36:17.403 DEBUG:boto:Data: 2023-09-22T21:36:17.404 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.404 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.404 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.404 DEBUG:boto:Params: {} 2023-09-22T21:36:17.404 DEBUG:boto:Token: None 2023-09-22T21:36:17.404 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/ 2023-09-22T21:36:17.404 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc= 2023-09-22T21:36:17.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc=', 'Content-Length': '0'} 2023-09-22T21:36:17.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7fb15db4ac6bfeb-00650e08d1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.416 DEBUG:boto:b'ffzdqu-58obj000000000000000000231000falseobj000000000000000000232023-09-22T21:35:03.585Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:17.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-09-22T21:36:17.417 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.417 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.417 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.417 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.417 DEBUG:boto:Method: GET 2023-09-22T21:36:17.417 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.417 DEBUG:boto:Data: 2023-09-22T21:36:17.418 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.418 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.418 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.418 DEBUG:boto:Params: {} 2023-09-22T21:36:17.418 DEBUG:boto:Token: None 2023-09-22T21:36:17.418 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.418 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+dbgnbnwx87w2qgR4uWVLZNVX7g= 2023-09-22T21:36:17.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+dbgnbnwx87w2qgR4uWVLZNVX7g=', 'Content-Length': '0'} 2023-09-22T21:36:17.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f05fe5028dec42eb-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.422 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.422 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.422 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.422 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.422 DEBUG:boto:Method: GET 2023-09-22T21:36:17.422 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.422 DEBUG:boto:Data: 2023-09-22T21:36:17.423 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.423 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.423 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.423 DEBUG:boto:Params: {} 2023-09-22T21:36:17.423 DEBUG:boto:Token: None 2023-09-22T21:36:17.423 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.423 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+dbgnbnwx87w2qgR4uWVLZNVX7g= 2023-09-22T21:36:17.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+dbgnbnwx87w2qgR4uWVLZNVX7g=', 'Content-Length': '0'} 2023-09-22T21:36:17.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b5ed3b00fbdf740-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.466 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.466 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.466 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.467 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.467 DEBUG:boto:Method: HEAD 2023-09-22T21:36:17.467 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.467 DEBUG:boto:Data: 2023-09-22T21:36:17.467 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.467 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.467 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.467 DEBUG:boto:Params: {} 2023-09-22T21:36:17.467 DEBUG:boto:Token: None 2023-09-22T21:36:17.468 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.468 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zjm/+j6mRKjhJOFfK7jfTdVOK54= 2023-09-22T21:36:17.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zjm/+j6mRKjhJOFfK7jfTdVOK54=', 'Content-Length': '0'} 2023-09-22T21:36:17.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae366298b02f4977-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.471 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.471 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.471 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.471 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.471 DEBUG:boto:Method: HEAD 2023-09-22T21:36:17.471 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.471 DEBUG:boto:Data: 2023-09-22T21:36:17.471 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.472 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.472 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.472 DEBUG:boto:Params: {} 2023-09-22T21:36:17.472 DEBUG:boto:Token: None 2023-09-22T21:36:17.472 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.472 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zjm/+j6mRKjhJOFfK7jfTdVOK54= 2023-09-22T21:36:17.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zjm/+j6mRKjhJOFfK7jfTdVOK54=', 'Content-Length': '0'} 2023-09-22T21:36:17.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1ec15c99d671c06-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-09-22T21:36:17.475 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.475 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.476 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.476 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.476 DEBUG:boto:Method: GET 2023-09-22T21:36:17.476 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.476 DEBUG:boto:Data: 2023-09-22T21:36:17.476 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.476 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.476 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.476 DEBUG:boto:Params: {} 2023-09-22T21:36:17.476 DEBUG:boto:Token: None 2023-09-22T21:36:17.477 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.477 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+dbgnbnwx87w2qgR4uWVLZNVX7g= 2023-09-22T21:36:17.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+dbgnbnwx87w2qgR4uWVLZNVX7g=', 'Content-Length': '0'} 2023-09-22T21:36:17.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4be452b9c6807a8-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.522 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.523 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.523 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.523 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.523 DEBUG:boto:Method: GET 2023-09-22T21:36:17.523 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.523 DEBUG:boto:Data: 2023-09-22T21:36:17.523 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.523 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.523 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.524 DEBUG:boto:Params: {} 2023-09-22T21:36:17.524 DEBUG:boto:Token: None 2023-09-22T21:36:17.524 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000024?versionId=null 2023-09-22T21:36:17.524 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+dbgnbnwx87w2qgR4uWVLZNVX7g= 2023-09-22T21:36:17.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+dbgnbnwx87w2qgR4uWVLZNVX7g=', 'Content-Length': '0'} 2023-09-22T21:36:17.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6c3b58b7a7ba57b-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.527 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:17.527 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:17.527 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000024 2023-09-22T21:36:17.527 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000024 2023-09-22T21:36:17.528 DEBUG:boto:Method: GET 2023-09-22T21:36:17.528 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000024 2023-09-22T21:36:17.528 DEBUG:boto:Data: 2023-09-22T21:36:17.528 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.528 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.528 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.528 DEBUG:boto:Params: {} 2023-09-22T21:36:17.528 DEBUG:boto:Token: None 2023-09-22T21:36:17.528 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/ 2023-09-22T21:36:17.529 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc= 2023-09-22T21:36:17.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc=', 'Content-Length': '0'} 2023-09-22T21:36:17.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6c5f92a23be5366-00650e08d1-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.533 DEBUG:boto:b'ffzdqu-58obj000000000000000000241000falseobj000000000000000000242023-09-22T21:35:05.767Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:17.533 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:17.534 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:17.534 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000024 2023-09-22T21:36:17.534 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000024 2023-09-22T21:36:17.534 DEBUG:boto:Method: GET 2023-09-22T21:36:17.534 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000024 2023-09-22T21:36:17.534 DEBUG:boto:Data: 2023-09-22T21:36:17.534 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.534 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.534 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.535 DEBUG:boto:Params: {} 2023-09-22T21:36:17.535 DEBUG:boto:Token: None 2023-09-22T21:36:17.535 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/ 2023-09-22T21:36:17.535 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc= 2023-09-22T21:36:17.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc=', 'Content-Length': '0'} 2023-09-22T21:36:17.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000721d72d60e487127-00650e08d1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.544 DEBUG:boto:b'ffzdqu-58obj000000000000000000241000falseobj000000000000000000242023-09-22T21:35:05.767Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:17.545 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.545 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.545 DEBUG:boto:Method: HEAD 2023-09-22T21:36:17.545 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.545 DEBUG:boto:Data: 2023-09-22T21:36:17.545 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.545 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.545 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.546 DEBUG:boto:Params: {} 2023-09-22T21:36:17.546 DEBUG:boto:Token: None 2023-09-22T21:36:17.546 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.546 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Gi5fSJHNFyGlUekgTyz7c1GL7Sk= 2023-09-22T21:36:17.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Gi5fSJHNFyGlUekgTyz7c1GL7Sk=', 'Content-Length': '0'} 2023-09-22T21:36:17.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056eb57bb8dedae73-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.549 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.549 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.549 DEBUG:boto:Method: HEAD 2023-09-22T21:36:17.549 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.549 DEBUG:boto:Data: 2023-09-22T21:36:17.549 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.549 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.550 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.550 DEBUG:boto:Params: {} 2023-09-22T21:36:17.550 DEBUG:boto:Token: None 2023-09-22T21:36:17.550 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.550 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Gi5fSJHNFyGlUekgTyz7c1GL7Sk= 2023-09-22T21:36:17.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Gi5fSJHNFyGlUekgTyz7c1GL7Sk=', 'Content-Length': '0'} 2023-09-22T21:36:17.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003995eff9df250aa1-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-09-22T21:36:17.553 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.554 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.554 DEBUG:boto:Method: GET 2023-09-22T21:36:17.554 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.554 DEBUG:boto:Data: 2023-09-22T21:36:17.554 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.554 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.554 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.554 DEBUG:boto:Params: {} 2023-09-22T21:36:17.554 DEBUG:boto:Token: None 2023-09-22T21:36:17.555 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.555 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jgGCuey4kDtYMhhKuBNqi/6swYs= 2023-09-22T21:36:17.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jgGCuey4kDtYMhhKuBNqi/6swYs=', 'Content-Length': '0'} 2023-09-22T21:36:17.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066568dea5fa574da-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.598 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.599 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.599 DEBUG:boto:Method: GET 2023-09-22T21:36:17.599 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.599 DEBUG:boto:Data: 2023-09-22T21:36:17.599 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.599 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.599 DEBUG:boto:Params: {} 2023-09-22T21:36:17.599 DEBUG:boto:Token: None 2023-09-22T21:36:17.600 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000024 2023-09-22T21:36:17.600 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jgGCuey4kDtYMhhKuBNqi/6swYs= 2023-09-22T21:36:17.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jgGCuey4kDtYMhhKuBNqi/6swYs=', 'Content-Length': '0'} 2023-09-22T21:36:17.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b75871cfc1d11f61-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.603 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:17.603 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:17.603 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000024 2023-09-22T21:36:17.603 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000024 2023-09-22T21:36:17.603 DEBUG:boto:Method: GET 2023-09-22T21:36:17.603 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000024 2023-09-22T21:36:17.603 DEBUG:boto:Data: 2023-09-22T21:36:17.604 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.604 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.604 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.604 DEBUG:boto:Params: {} 2023-09-22T21:36:17.604 DEBUG:boto:Token: None 2023-09-22T21:36:17.604 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/ 2023-09-22T21:36:17.604 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc= 2023-09-22T21:36:17.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc=', 'Content-Length': '0'} 2023-09-22T21:36:17.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079a39a47613ef42e-00650e08d1-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.650 DEBUG:boto:b'ffzdqu-58obj000000000000000000241000falseobj000000000000000000242023-09-22T21:35:05.767Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:17.651 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:17.651 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:17.651 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000024 2023-09-22T21:36:17.651 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000024 2023-09-22T21:36:17.651 DEBUG:boto:Method: GET 2023-09-22T21:36:17.651 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000024 2023-09-22T21:36:17.652 DEBUG:boto:Data: 2023-09-22T21:36:17.652 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.652 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.652 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.652 DEBUG:boto:Params: {} 2023-09-22T21:36:17.652 DEBUG:boto:Token: None 2023-09-22T21:36:17.652 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/ 2023-09-22T21:36:17.652 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc= 2023-09-22T21:36:17.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc=', 'Content-Length': '0'} 2023-09-22T21:36:17.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d70fbee739596f1-00650e08d1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.663 DEBUG:boto:b'ffzdqu-58obj000000000000000000241000falseobj000000000000000000242023-09-22T21:35:05.767Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:17.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-09-22T21:36:17.664 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.664 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.664 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.665 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.665 DEBUG:boto:Method: GET 2023-09-22T21:36:17.665 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.665 DEBUG:boto:Data: 2023-09-22T21:36:17.665 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.665 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.666 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.666 DEBUG:boto:Params: {} 2023-09-22T21:36:17.666 DEBUG:boto:Token: None 2023-09-22T21:36:17.666 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.666 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hC1IGGYc0WyNO5Jl7MmJ3Vo+CRo= 2023-09-22T21:36:17.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hC1IGGYc0WyNO5Jl7MmJ3Vo+CRo=', 'Content-Length': '0'} 2023-09-22T21:36:17.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cfa3235c8053984-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.670 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.670 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.670 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.671 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.671 DEBUG:boto:Method: GET 2023-09-22T21:36:17.671 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.671 DEBUG:boto:Data: 2023-09-22T21:36:17.671 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.671 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.671 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.671 DEBUG:boto:Params: {} 2023-09-22T21:36:17.671 DEBUG:boto:Token: None 2023-09-22T21:36:17.672 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.672 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hC1IGGYc0WyNO5Jl7MmJ3Vo+CRo= 2023-09-22T21:36:17.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hC1IGGYc0WyNO5Jl7MmJ3Vo+CRo=', 'Content-Length': '0'} 2023-09-22T21:36:17.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094d1a0c0c9dd9efb-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.675 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.675 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.676 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.676 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.676 DEBUG:boto:Method: HEAD 2023-09-22T21:36:17.676 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.676 DEBUG:boto:Data: 2023-09-22T21:36:17.676 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.676 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.676 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.676 DEBUG:boto:Params: {} 2023-09-22T21:36:17.677 DEBUG:boto:Token: None 2023-09-22T21:36:17.677 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.677 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+mCkzqYSBomabMKWtRuOiAtwOw8= 2023-09-22T21:36:17.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+mCkzqYSBomabMKWtRuOiAtwOw8=', 'Content-Length': '0'} 2023-09-22T21:36:17.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b55e34ad2232883b-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.680 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.680 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.680 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.680 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.681 DEBUG:boto:Method: HEAD 2023-09-22T21:36:17.681 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.681 DEBUG:boto:Data: 2023-09-22T21:36:17.681 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.681 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.681 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.681 DEBUG:boto:Params: {} 2023-09-22T21:36:17.681 DEBUG:boto:Token: None 2023-09-22T21:36:17.681 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.682 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+mCkzqYSBomabMKWtRuOiAtwOw8= 2023-09-22T21:36:17.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+mCkzqYSBomabMKWtRuOiAtwOw8=', 'Content-Length': '0'} 2023-09-22T21:36:17.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4021fc07edf6490-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-09-22T21:36:17.685 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.685 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.685 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.685 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.685 DEBUG:boto:Method: GET 2023-09-22T21:36:17.685 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.685 DEBUG:boto:Data: 2023-09-22T21:36:17.686 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.686 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.686 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.686 DEBUG:boto:Params: {} 2023-09-22T21:36:17.686 DEBUG:boto:Token: None 2023-09-22T21:36:17.686 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.686 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hC1IGGYc0WyNO5Jl7MmJ3Vo+CRo= 2023-09-22T21:36:17.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hC1IGGYc0WyNO5Jl7MmJ3Vo+CRo=', 'Content-Length': '0'} 2023-09-22T21:36:17.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e181a0368479f9e-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.730 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.730 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.731 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.731 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.731 DEBUG:boto:Method: GET 2023-09-22T21:36:17.731 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.731 DEBUG:boto:Data: 2023-09-22T21:36:17.731 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.731 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.731 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.731 DEBUG:boto:Params: {} 2023-09-22T21:36:17.731 DEBUG:boto:Token: None 2023-09-22T21:36:17.732 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000025?versionId=null 2023-09-22T21:36:17.732 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hC1IGGYc0WyNO5Jl7MmJ3Vo+CRo= 2023-09-22T21:36:17.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hC1IGGYc0WyNO5Jl7MmJ3Vo+CRo=', 'Content-Length': '0'} 2023-09-22T21:36:17.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8b0027cf42979c3-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.735 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:17.735 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:17.735 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000025 2023-09-22T21:36:17.735 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000025 2023-09-22T21:36:17.735 DEBUG:boto:Method: GET 2023-09-22T21:36:17.735 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000025 2023-09-22T21:36:17.736 DEBUG:boto:Data: 2023-09-22T21:36:17.736 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.736 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.736 DEBUG:boto:Params: {} 2023-09-22T21:36:17.736 DEBUG:boto:Token: None 2023-09-22T21:36:17.736 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/ 2023-09-22T21:36:17.736 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc= 2023-09-22T21:36:17.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc=', 'Content-Length': '0'} 2023-09-22T21:36:17.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1e8071aa8b760d5-00650e08d1-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.782 DEBUG:boto:b'ffzdqu-58obj000000000000000000251000falseobj000000000000000000252023-09-22T21:35:02.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:17.783 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:17.783 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:17.783 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000025 2023-09-22T21:36:17.783 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000025 2023-09-22T21:36:17.783 DEBUG:boto:Method: GET 2023-09-22T21:36:17.784 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000025 2023-09-22T21:36:17.784 DEBUG:boto:Data: 2023-09-22T21:36:17.784 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.784 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.784 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.784 DEBUG:boto:Params: {} 2023-09-22T21:36:17.784 DEBUG:boto:Token: None 2023-09-22T21:36:17.784 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/ 2023-09-22T21:36:17.784 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc= 2023-09-22T21:36:17.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc=', 'Content-Length': '0'} 2023-09-22T21:36:17.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c25a88e88255bdd-00650e08d1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.795 DEBUG:boto:b'ffzdqu-58obj000000000000000000251000falseobj000000000000000000252023-09-22T21:35:02.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:17.795 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.795 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.795 DEBUG:boto:Method: HEAD 2023-09-22T21:36:17.795 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.796 DEBUG:boto:Data: 2023-09-22T21:36:17.796 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.796 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.796 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.796 DEBUG:boto:Params: {} 2023-09-22T21:36:17.796 DEBUG:boto:Token: None 2023-09-22T21:36:17.796 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.796 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MAuxmpIhREGjJ+LAnUiiceP9DCQ= 2023-09-22T21:36:17.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MAuxmpIhREGjJ+LAnUiiceP9DCQ=', 'Content-Length': '0'} 2023-09-22T21:36:17.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e30706c33881785f-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.799 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.800 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.800 DEBUG:boto:Method: HEAD 2023-09-22T21:36:17.800 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.800 DEBUG:boto:Data: 2023-09-22T21:36:17.800 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.800 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.800 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.800 DEBUG:boto:Params: {} 2023-09-22T21:36:17.801 DEBUG:boto:Token: None 2023-09-22T21:36:17.801 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.801 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MAuxmpIhREGjJ+LAnUiiceP9DCQ= 2023-09-22T21:36:17.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MAuxmpIhREGjJ+LAnUiiceP9DCQ=', 'Content-Length': '0'} 2023-09-22T21:36:17.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c294146e054589df-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-09-22T21:36:17.805 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.805 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.805 DEBUG:boto:Method: GET 2023-09-22T21:36:17.806 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.806 DEBUG:boto:Data: 2023-09-22T21:36:17.806 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.806 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.806 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.806 DEBUG:boto:Params: {} 2023-09-22T21:36:17.806 DEBUG:boto:Token: None 2023-09-22T21:36:17.806 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.807 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hy1pwimdA2k4EX++qo9EEniph3Y= 2023-09-22T21:36:17.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hy1pwimdA2k4EX++qo9EEniph3Y=', 'Content-Length': '0'} 2023-09-22T21:36:17.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000150186f65a817df3-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.854 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.855 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.855 DEBUG:boto:Method: GET 2023-09-22T21:36:17.855 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.855 DEBUG:boto:Data: 2023-09-22T21:36:17.855 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.855 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.855 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.856 DEBUG:boto:Params: {} 2023-09-22T21:36:17.856 DEBUG:boto:Token: None 2023-09-22T21:36:17.856 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000025 2023-09-22T21:36:17.856 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hy1pwimdA2k4EX++qo9EEniph3Y= 2023-09-22T21:36:17.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hy1pwimdA2k4EX++qo9EEniph3Y=', 'Content-Length': '0'} 2023-09-22T21:36:17.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d86ed371c63fe372-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.859 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:17.860 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:17.860 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000025 2023-09-22T21:36:17.860 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000025 2023-09-22T21:36:17.860 DEBUG:boto:Method: GET 2023-09-22T21:36:17.860 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000025 2023-09-22T21:36:17.860 DEBUG:boto:Data: 2023-09-22T21:36:17.861 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.861 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.861 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.861 DEBUG:boto:Params: {} 2023-09-22T21:36:17.861 DEBUG:boto:Token: None 2023-09-22T21:36:17.861 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/ 2023-09-22T21:36:17.861 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc= 2023-09-22T21:36:17.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc=', 'Content-Length': '0'} 2023-09-22T21:36:17.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb3cace7cc07d25f-00650e08d1-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.910 DEBUG:boto:b'ffzdqu-58obj000000000000000000251000falseobj000000000000000000252023-09-22T21:35:02.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:17.911 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:17.911 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:17.911 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000025 2023-09-22T21:36:17.911 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000025 2023-09-22T21:36:17.912 DEBUG:boto:Method: GET 2023-09-22T21:36:17.912 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000025 2023-09-22T21:36:17.912 DEBUG:boto:Data: 2023-09-22T21:36:17.912 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.912 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.912 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.912 DEBUG:boto:Params: {} 2023-09-22T21:36:17.912 DEBUG:boto:Token: None 2023-09-22T21:36:17.912 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/ 2023-09-22T21:36:17.913 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc= 2023-09-22T21:36:17.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dkHtq75iH4WZ3ALzP35zvwtxJzc=', 'Content-Length': '0'} 2023-09-22T21:36:17.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb20f5bcf5498cc4-00650e08d1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.925 DEBUG:boto:b'ffzdqu-58obj000000000000000000251000falseobj000000000000000000252023-09-22T21:35:02.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:17.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-09-22T21:36:17.925 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:17.926 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:17.926 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.926 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.926 DEBUG:boto:Method: GET 2023-09-22T21:36:17.926 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.926 DEBUG:boto:Data: 2023-09-22T21:36:17.927 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.927 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.927 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.927 DEBUG:boto:Params: {} 2023-09-22T21:36:17.927 DEBUG:boto:Token: None 2023-09-22T21:36:17.927 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.927 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gxC9qgONJwDU0Mlo1ls3BQRxpos= 2023-09-22T21:36:17.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gxC9qgONJwDU0Mlo1ls3BQRxpos=', 'Content-Length': '0'} 2023-09-22T21:36:17.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e15b5726e21c3c2-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.931 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:17.931 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:17.931 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.931 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.931 DEBUG:boto:Method: GET 2023-09-22T21:36:17.931 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.931 DEBUG:boto:Data: 2023-09-22T21:36:17.931 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.931 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.932 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.932 DEBUG:boto:Params: {} 2023-09-22T21:36:17.932 DEBUG:boto:Token: None 2023-09-22T21:36:17.932 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.932 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gxC9qgONJwDU0Mlo1ls3BQRxpos= 2023-09-22T21:36:17.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gxC9qgONJwDU0Mlo1ls3BQRxpos=', 'Content-Length': '0'} 2023-09-22T21:36:17.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5cf2a2410756215-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.935 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:17.936 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:17.936 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.936 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.936 DEBUG:boto:Method: HEAD 2023-09-22T21:36:17.936 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.936 DEBUG:boto:Data: 2023-09-22T21:36:17.936 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.936 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.936 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.937 DEBUG:boto:Params: {} 2023-09-22T21:36:17.937 DEBUG:boto:Token: None 2023-09-22T21:36:17.937 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.937 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pGUqn81wGT9Di5/hBzz2MsqCBgc= 2023-09-22T21:36:17.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pGUqn81wGT9Di5/hBzz2MsqCBgc=', 'Content-Length': '0'} 2023-09-22T21:36:17.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f5e2543be49fa13-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.940 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:17.940 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:17.941 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.941 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.941 DEBUG:boto:Method: HEAD 2023-09-22T21:36:17.941 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.941 DEBUG:boto:Data: 2023-09-22T21:36:17.941 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.941 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.941 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.941 DEBUG:boto:Params: {} 2023-09-22T21:36:17.941 DEBUG:boto:Token: None 2023-09-22T21:36:17.942 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.942 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pGUqn81wGT9Di5/hBzz2MsqCBgc= 2023-09-22T21:36:17.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pGUqn81wGT9Di5/hBzz2MsqCBgc=', 'Content-Length': '0'} 2023-09-22T21:36:17.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025026be66feacd63-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-09-22T21:36:17.945 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:17.945 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:17.946 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.946 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.946 DEBUG:boto:Method: GET 2023-09-22T21:36:17.946 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.946 DEBUG:boto:Data: 2023-09-22T21:36:17.946 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.946 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:17.946 DEBUG:boto:Port: 8001 2023-09-22T21:36:17.946 DEBUG:boto:Params: {} 2023-09-22T21:36:17.947 DEBUG:boto:Token: None 2023-09-22T21:36:17.947 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.947 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gxC9qgONJwDU0Mlo1ls3BQRxpos= 2023-09-22T21:36:17.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gxC9qgONJwDU0Mlo1ls3BQRxpos=', 'Content-Length': '0'} 2023-09-22T21:36:17.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000498f16147baa3179-00650e08d1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.994 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:17.995 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:17.995 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.995 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.995 DEBUG:boto:Method: GET 2023-09-22T21:36:17.995 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.995 DEBUG:boto:Data: 2023-09-22T21:36:17.995 DEBUG:boto:Headers: {} 2023-09-22T21:36:17.995 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:17.995 DEBUG:boto:Port: 8000 2023-09-22T21:36:17.995 DEBUG:boto:Params: {} 2023-09-22T21:36:17.996 DEBUG:boto:Token: None 2023-09-22T21:36:17.996 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:17 GMT /ffzdqu-58/obj00000000000000000026?versionId=null 2023-09-22T21:36:17.996 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gxC9qgONJwDU0Mlo1ls3BQRxpos= 2023-09-22T21:36:17.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gxC9qgONJwDU0Mlo1ls3BQRxpos=', 'Content-Length': '0'} 2023-09-22T21:36:17.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8b27276942a73fb-00650e08d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:17.999 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:18.000 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:18.000 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000026 2023-09-22T21:36:18.000 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000026 2023-09-22T21:36:18.000 DEBUG:boto:Method: GET 2023-09-22T21:36:18.000 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000026 2023-09-22T21:36:18.000 DEBUG:boto:Data: 2023-09-22T21:36:18.000 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.000 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.000 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.001 DEBUG:boto:Params: {} 2023-09-22T21:36:18.001 DEBUG:boto:Token: None 2023-09-22T21:36:18.001 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/ 2023-09-22T21:36:18.001 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk= 2023-09-22T21:36:18.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk=', 'Content-Length': '0'} 2023-09-22T21:36:18.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000611bf0da5e54d8dd-00650e08d1-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.006 DEBUG:boto:b'ffzdqu-58obj000000000000000000261000falseobj000000000000000000262023-09-22T21:35:12.586Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:18.007 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:18.007 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:18.007 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000026 2023-09-22T21:36:18.007 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000026 2023-09-22T21:36:18.007 DEBUG:boto:Method: GET 2023-09-22T21:36:18.008 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000026 2023-09-22T21:36:18.008 DEBUG:boto:Data: 2023-09-22T21:36:18.008 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.008 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.008 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.008 DEBUG:boto:Params: {} 2023-09-22T21:36:18.008 DEBUG:boto:Token: None 2023-09-22T21:36:18.008 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/ 2023-09-22T21:36:18.009 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk= 2023-09-22T21:36:18.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk=', 'Content-Length': '0'} 2023-09-22T21:36:18.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e588ebe6ea4f235a-00650e08d2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.066 DEBUG:boto:b'ffzdqu-58obj000000000000000000261000falseobj000000000000000000262023-09-22T21:35:12.586Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:18.067 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:18.067 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:18.067 DEBUG:boto:Method: HEAD 2023-09-22T21:36:18.067 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:18.067 DEBUG:boto:Data: 2023-09-22T21:36:18.067 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.068 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.068 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.068 DEBUG:boto:Params: {} 2023-09-22T21:36:18.068 DEBUG:boto:Token: None 2023-09-22T21:36:18.068 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:18.068 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PNwXfeaGKUWfeP9UlolXESXVY44= 2023-09-22T21:36:18.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PNwXfeaGKUWfeP9UlolXESXVY44=', 'Content-Length': '0'} 2023-09-22T21:36:18.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c3c7d501d9ed692-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.072 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:18.072 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:18.072 DEBUG:boto:Method: HEAD 2023-09-22T21:36:18.072 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:18.073 DEBUG:boto:Data: 2023-09-22T21:36:18.073 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.073 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.073 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.073 DEBUG:boto:Params: {} 2023-09-22T21:36:18.073 DEBUG:boto:Token: None 2023-09-22T21:36:18.073 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:18.074 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PNwXfeaGKUWfeP9UlolXESXVY44= 2023-09-22T21:36:18.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PNwXfeaGKUWfeP9UlolXESXVY44=', 'Content-Length': '0'} 2023-09-22T21:36:18.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050c65e84dbb6bae5-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-09-22T21:36:18.077 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:18.077 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:18.077 DEBUG:boto:Method: GET 2023-09-22T21:36:18.077 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:18.078 DEBUG:boto:Data: 2023-09-22T21:36:18.078 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.078 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.078 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.078 DEBUG:boto:Params: {} 2023-09-22T21:36:18.078 DEBUG:boto:Token: None 2023-09-22T21:36:18.078 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:18.078 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Lw4Zm+PCMu90/7JKclc8+K3h6cI= 2023-09-22T21:36:18.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Lw4Zm+PCMu90/7JKclc8+K3h6cI=', 'Content-Length': '0'} 2023-09-22T21:36:18.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002508bad42c212690-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.122 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:18.123 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:18.123 DEBUG:boto:Method: GET 2023-09-22T21:36:18.123 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:18.123 DEBUG:boto:Data: 2023-09-22T21:36:18.123 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.123 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.124 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.124 DEBUG:boto:Params: {} 2023-09-22T21:36:18.124 DEBUG:boto:Token: None 2023-09-22T21:36:18.124 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000026 2023-09-22T21:36:18.124 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Lw4Zm+PCMu90/7JKclc8+K3h6cI= 2023-09-22T21:36:18.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Lw4Zm+PCMu90/7JKclc8+K3h6cI=', 'Content-Length': '0'} 2023-09-22T21:36:18.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e73b807a49fbb9c-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.127 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:18.128 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:18.128 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000026 2023-09-22T21:36:18.128 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000026 2023-09-22T21:36:18.128 DEBUG:boto:Method: GET 2023-09-22T21:36:18.128 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000026 2023-09-22T21:36:18.128 DEBUG:boto:Data: 2023-09-22T21:36:18.128 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.128 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.128 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.128 DEBUG:boto:Params: {} 2023-09-22T21:36:18.129 DEBUG:boto:Token: None 2023-09-22T21:36:18.129 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/ 2023-09-22T21:36:18.129 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk= 2023-09-22T21:36:18.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk=', 'Content-Length': '0'} 2023-09-22T21:36:18.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca81f80d8b83fc9c-00650e08d2-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.174 DEBUG:boto:b'ffzdqu-58obj000000000000000000261000falseobj000000000000000000262023-09-22T21:35:12.586Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:18.175 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:18.175 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:18.175 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000026 2023-09-22T21:36:18.175 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000026 2023-09-22T21:36:18.175 DEBUG:boto:Method: GET 2023-09-22T21:36:18.175 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000026 2023-09-22T21:36:18.175 DEBUG:boto:Data: 2023-09-22T21:36:18.175 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.176 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.176 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.176 DEBUG:boto:Params: {} 2023-09-22T21:36:18.176 DEBUG:boto:Token: None 2023-09-22T21:36:18.176 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/ 2023-09-22T21:36:18.176 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk= 2023-09-22T21:36:18.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk=', 'Content-Length': '0'} 2023-09-22T21:36:18.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bf48e2e1973f7cd-00650e08d2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.186 DEBUG:boto:b'ffzdqu-58obj000000000000000000261000falseobj000000000000000000262023-09-22T21:35:12.586Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:18.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-09-22T21:36:18.187 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.187 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.187 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.187 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.188 DEBUG:boto:Method: GET 2023-09-22T21:36:18.188 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.188 DEBUG:boto:Data: 2023-09-22T21:36:18.188 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.188 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.188 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.188 DEBUG:boto:Params: {} 2023-09-22T21:36:18.188 DEBUG:boto:Token: None 2023-09-22T21:36:18.188 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.188 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6u+sRxlbk2jyLPBW+XRKZNKzkwY= 2023-09-22T21:36:18.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6u+sRxlbk2jyLPBW+XRKZNKzkwY=', 'Content-Length': '0'} 2023-09-22T21:36:18.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3a059c05a49e686-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.192 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.192 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.192 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.192 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.192 DEBUG:boto:Method: GET 2023-09-22T21:36:18.192 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.193 DEBUG:boto:Data: 2023-09-22T21:36:18.193 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.193 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.193 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.193 DEBUG:boto:Params: {} 2023-09-22T21:36:18.193 DEBUG:boto:Token: None 2023-09-22T21:36:18.193 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.193 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6u+sRxlbk2jyLPBW+XRKZNKzkwY= 2023-09-22T21:36:18.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6u+sRxlbk2jyLPBW+XRKZNKzkwY=', 'Content-Length': '0'} 2023-09-22T21:36:18.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000425e5407744529b1-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.197 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.197 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.197 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.197 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.197 DEBUG:boto:Method: HEAD 2023-09-22T21:36:18.197 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.197 DEBUG:boto:Data: 2023-09-22T21:36:18.197 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.197 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.198 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.198 DEBUG:boto:Params: {} 2023-09-22T21:36:18.198 DEBUG:boto:Token: None 2023-09-22T21:36:18.198 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.198 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MR4ZQDQbKtshiwBVFPWsAGtrua4= 2023-09-22T21:36:18.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MR4ZQDQbKtshiwBVFPWsAGtrua4=', 'Content-Length': '0'} 2023-09-22T21:36:18.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030d79881715d0f54-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.201 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.201 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.201 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.201 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.201 DEBUG:boto:Method: HEAD 2023-09-22T21:36:18.201 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.202 DEBUG:boto:Data: 2023-09-22T21:36:18.202 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.202 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.202 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.202 DEBUG:boto:Params: {} 2023-09-22T21:36:18.202 DEBUG:boto:Token: None 2023-09-22T21:36:18.202 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.202 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MR4ZQDQbKtshiwBVFPWsAGtrua4= 2023-09-22T21:36:18.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MR4ZQDQbKtshiwBVFPWsAGtrua4=', 'Content-Length': '0'} 2023-09-22T21:36:18.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fd0e74a78be4ce8-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-09-22T21:36:18.206 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.206 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.206 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.206 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.206 DEBUG:boto:Method: GET 2023-09-22T21:36:18.206 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.206 DEBUG:boto:Data: 2023-09-22T21:36:18.206 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.206 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.207 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.207 DEBUG:boto:Params: {} 2023-09-22T21:36:18.207 DEBUG:boto:Token: None 2023-09-22T21:36:18.207 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.207 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6u+sRxlbk2jyLPBW+XRKZNKzkwY= 2023-09-22T21:36:18.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6u+sRxlbk2jyLPBW+XRKZNKzkwY=', 'Content-Length': '0'} 2023-09-22T21:36:18.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e68885d27ff524f-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.254 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.255 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.255 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.255 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.255 DEBUG:boto:Method: GET 2023-09-22T21:36:18.255 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.255 DEBUG:boto:Data: 2023-09-22T21:36:18.255 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.255 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.255 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.256 DEBUG:boto:Params: {} 2023-09-22T21:36:18.256 DEBUG:boto:Token: None 2023-09-22T21:36:18.256 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000027?versionId=null 2023-09-22T21:36:18.256 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6u+sRxlbk2jyLPBW+XRKZNKzkwY= 2023-09-22T21:36:18.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6u+sRxlbk2jyLPBW+XRKZNKzkwY=', 'Content-Length': '0'} 2023-09-22T21:36:18.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083f594c65020a621-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.259 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:18.259 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:18.260 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000027 2023-09-22T21:36:18.260 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000027 2023-09-22T21:36:18.260 DEBUG:boto:Method: GET 2023-09-22T21:36:18.260 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000027 2023-09-22T21:36:18.260 DEBUG:boto:Data: 2023-09-22T21:36:18.260 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.261 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.261 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.261 DEBUG:boto:Params: {} 2023-09-22T21:36:18.261 DEBUG:boto:Token: None 2023-09-22T21:36:18.261 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/ 2023-09-22T21:36:18.261 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk= 2023-09-22T21:36:18.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk=', 'Content-Length': '0'} 2023-09-22T21:36:18.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bc49b9622735b23-00650e08d2-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.266 DEBUG:boto:b'ffzdqu-58obj000000000000000000271000falseobj000000000000000000272023-09-22T21:35:09.970Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:18.266 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:18.267 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:18.267 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000027 2023-09-22T21:36:18.267 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000027 2023-09-22T21:36:18.267 DEBUG:boto:Method: GET 2023-09-22T21:36:18.267 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000027 2023-09-22T21:36:18.267 DEBUG:boto:Data: 2023-09-22T21:36:18.267 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.267 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.267 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.268 DEBUG:boto:Params: {} 2023-09-22T21:36:18.268 DEBUG:boto:Token: None 2023-09-22T21:36:18.268 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/ 2023-09-22T21:36:18.268 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk= 2023-09-22T21:36:18.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk=', 'Content-Length': '0'} 2023-09-22T21:36:18.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aeab6690b30a579f-00650e08d2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.279 DEBUG:boto:b'ffzdqu-58obj000000000000000000271000falseobj000000000000000000272023-09-22T21:35:09.970Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:18.279 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.280 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.280 DEBUG:boto:Method: HEAD 2023-09-22T21:36:18.280 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.280 DEBUG:boto:Data: 2023-09-22T21:36:18.280 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.280 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.280 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.280 DEBUG:boto:Params: {} 2023-09-22T21:36:18.280 DEBUG:boto:Token: None 2023-09-22T21:36:18.281 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.281 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Co4SiSPeuFfN2eSuqd9AXanXJ0Q= 2023-09-22T21:36:18.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Co4SiSPeuFfN2eSuqd9AXanXJ0Q=', 'Content-Length': '0'} 2023-09-22T21:36:18.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052121d137e1ebcea-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.284 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.284 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.284 DEBUG:boto:Method: HEAD 2023-09-22T21:36:18.284 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.284 DEBUG:boto:Data: 2023-09-22T21:36:18.285 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.285 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.285 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.285 DEBUG:boto:Params: {} 2023-09-22T21:36:18.285 DEBUG:boto:Token: None 2023-09-22T21:36:18.285 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.285 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Co4SiSPeuFfN2eSuqd9AXanXJ0Q= 2023-09-22T21:36:18.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Co4SiSPeuFfN2eSuqd9AXanXJ0Q=', 'Content-Length': '0'} 2023-09-22T21:36:18.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004552c2cde39cc59a-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-09-22T21:36:18.288 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.288 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.288 DEBUG:boto:Method: GET 2023-09-22T21:36:18.288 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.289 DEBUG:boto:Data: 2023-09-22T21:36:18.289 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.289 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.289 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.289 DEBUG:boto:Params: {} 2023-09-22T21:36:18.289 DEBUG:boto:Token: None 2023-09-22T21:36:18.289 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.289 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SI8QNy97paK0U5Co9wJbJM+FVso= 2023-09-22T21:36:18.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SI8QNy97paK0U5Co9wJbJM+FVso=', 'Content-Length': '0'} 2023-09-22T21:36:18.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a374d3b2881c3df3-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.334 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.335 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.335 DEBUG:boto:Method: GET 2023-09-22T21:36:18.335 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.335 DEBUG:boto:Data: 2023-09-22T21:36:18.335 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.335 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.335 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.335 DEBUG:boto:Params: {} 2023-09-22T21:36:18.336 DEBUG:boto:Token: None 2023-09-22T21:36:18.336 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000027 2023-09-22T21:36:18.336 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SI8QNy97paK0U5Co9wJbJM+FVso= 2023-09-22T21:36:18.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SI8QNy97paK0U5Co9wJbJM+FVso=', 'Content-Length': '0'} 2023-09-22T21:36:18.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dcc102729425923-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.339 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:18.339 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:18.339 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000027 2023-09-22T21:36:18.339 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000027 2023-09-22T21:36:18.340 DEBUG:boto:Method: GET 2023-09-22T21:36:18.340 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000027 2023-09-22T21:36:18.340 DEBUG:boto:Data: 2023-09-22T21:36:18.340 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.340 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.340 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.340 DEBUG:boto:Params: {} 2023-09-22T21:36:18.340 DEBUG:boto:Token: None 2023-09-22T21:36:18.340 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/ 2023-09-22T21:36:18.341 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk= 2023-09-22T21:36:18.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk=', 'Content-Length': '0'} 2023-09-22T21:36:18.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd3c3c8dc6699e77-00650e08d2-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.345 DEBUG:boto:b'ffzdqu-58obj000000000000000000271000falseobj000000000000000000272023-09-22T21:35:09.970Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:18.346 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:18.346 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:18.346 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000027 2023-09-22T21:36:18.346 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000027 2023-09-22T21:36:18.346 DEBUG:boto:Method: GET 2023-09-22T21:36:18.347 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000027 2023-09-22T21:36:18.347 DEBUG:boto:Data: 2023-09-22T21:36:18.347 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.347 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.347 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.347 DEBUG:boto:Params: {} 2023-09-22T21:36:18.347 DEBUG:boto:Token: None 2023-09-22T21:36:18.348 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/ 2023-09-22T21:36:18.348 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk= 2023-09-22T21:36:18.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk=', 'Content-Length': '0'} 2023-09-22T21:36:18.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b94578641ac32da-00650e08d2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.398 DEBUG:boto:b'ffzdqu-58obj000000000000000000271000falseobj000000000000000000272023-09-22T21:35:09.970Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:18.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-09-22T21:36:18.399 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.399 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.399 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.399 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.399 DEBUG:boto:Method: GET 2023-09-22T21:36:18.400 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.400 DEBUG:boto:Data: 2023-09-22T21:36:18.400 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.400 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.400 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.400 DEBUG:boto:Params: {} 2023-09-22T21:36:18.400 DEBUG:boto:Token: None 2023-09-22T21:36:18.400 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.401 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+8GWHyw55TubwP5UunRbNpI99ng= 2023-09-22T21:36:18.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+8GWHyw55TubwP5UunRbNpI99ng=', 'Content-Length': '0'} 2023-09-22T21:36:18.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006baf314046d9b332-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.404 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.404 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.405 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.405 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.405 DEBUG:boto:Method: GET 2023-09-22T21:36:18.405 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.405 DEBUG:boto:Data: 2023-09-22T21:36:18.405 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.405 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.405 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.405 DEBUG:boto:Params: {} 2023-09-22T21:36:18.405 DEBUG:boto:Token: None 2023-09-22T21:36:18.406 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.406 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+8GWHyw55TubwP5UunRbNpI99ng= 2023-09-22T21:36:18.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+8GWHyw55TubwP5UunRbNpI99ng=', 'Content-Length': '0'} 2023-09-22T21:36:18.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006db569838b708349-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.409 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.409 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.409 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.409 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.409 DEBUG:boto:Method: HEAD 2023-09-22T21:36:18.410 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.410 DEBUG:boto:Data: 2023-09-22T21:36:18.410 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.410 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.410 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.410 DEBUG:boto:Params: {} 2023-09-22T21:36:18.410 DEBUG:boto:Token: None 2023-09-22T21:36:18.410 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.411 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Uge8YnQae2xQi8l3n9sgEOXolVA= 2023-09-22T21:36:18.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Uge8YnQae2xQi8l3n9sgEOXolVA=', 'Content-Length': '0'} 2023-09-22T21:36:18.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d53bf5ddd2291762-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.413 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.413 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.414 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.414 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.414 DEBUG:boto:Method: HEAD 2023-09-22T21:36:18.414 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.414 DEBUG:boto:Data: 2023-09-22T21:36:18.414 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.414 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.414 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.414 DEBUG:boto:Params: {} 2023-09-22T21:36:18.415 DEBUG:boto:Token: None 2023-09-22T21:36:18.415 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.415 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Uge8YnQae2xQi8l3n9sgEOXolVA= 2023-09-22T21:36:18.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Uge8YnQae2xQi8l3n9sgEOXolVA=', 'Content-Length': '0'} 2023-09-22T21:36:18.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000232bff08ae39b954-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-09-22T21:36:18.418 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.419 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.419 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.419 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.419 DEBUG:boto:Method: GET 2023-09-22T21:36:18.419 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.419 DEBUG:boto:Data: 2023-09-22T21:36:18.419 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.419 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.419 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.420 DEBUG:boto:Params: {} 2023-09-22T21:36:18.420 DEBUG:boto:Token: None 2023-09-22T21:36:18.420 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.420 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+8GWHyw55TubwP5UunRbNpI99ng= 2023-09-22T21:36:18.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+8GWHyw55TubwP5UunRbNpI99ng=', 'Content-Length': '0'} 2023-09-22T21:36:18.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c550f412764f5fbb-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.470 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.471 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.471 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.471 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.471 DEBUG:boto:Method: GET 2023-09-22T21:36:18.471 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.471 DEBUG:boto:Data: 2023-09-22T21:36:18.471 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.471 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.472 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.472 DEBUG:boto:Params: {} 2023-09-22T21:36:18.472 DEBUG:boto:Token: None 2023-09-22T21:36:18.472 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000028?versionId=null 2023-09-22T21:36:18.472 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+8GWHyw55TubwP5UunRbNpI99ng= 2023-09-22T21:36:18.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+8GWHyw55TubwP5UunRbNpI99ng=', 'Content-Length': '0'} 2023-09-22T21:36:18.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ccb1101fc67b23b-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.476 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:18.476 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:18.476 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000028 2023-09-22T21:36:18.476 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000028 2023-09-22T21:36:18.476 DEBUG:boto:Method: GET 2023-09-22T21:36:18.476 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000028 2023-09-22T21:36:18.477 DEBUG:boto:Data: 2023-09-22T21:36:18.477 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.477 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.477 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.477 DEBUG:boto:Params: {} 2023-09-22T21:36:18.477 DEBUG:boto:Token: None 2023-09-22T21:36:18.477 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/ 2023-09-22T21:36:18.477 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk= 2023-09-22T21:36:18.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk=', 'Content-Length': '0'} 2023-09-22T21:36:18.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015c13ba5ab6187f3-00650e08d2-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.482 DEBUG:boto:b'ffzdqu-58obj000000000000000000281000falseobj000000000000000000282023-09-22T21:34:59.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:18.483 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:18.483 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:18.483 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000028 2023-09-22T21:36:18.483 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000028 2023-09-22T21:36:18.483 DEBUG:boto:Method: GET 2023-09-22T21:36:18.483 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000028 2023-09-22T21:36:18.483 DEBUG:boto:Data: 2023-09-22T21:36:18.483 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.484 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.484 DEBUG:boto:Params: {} 2023-09-22T21:36:18.484 DEBUG:boto:Token: None 2023-09-22T21:36:18.484 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/ 2023-09-22T21:36:18.484 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk= 2023-09-22T21:36:18.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk=', 'Content-Length': '0'} 2023-09-22T21:36:18.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000823d2c5ff366033c-00650e08d2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.495 DEBUG:boto:b'ffzdqu-58obj000000000000000000281000falseobj000000000000000000282023-09-22T21:34:59.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:18.495 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.495 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.495 DEBUG:boto:Method: HEAD 2023-09-22T21:36:18.496 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.496 DEBUG:boto:Data: 2023-09-22T21:36:18.496 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.496 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.496 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.496 DEBUG:boto:Params: {} 2023-09-22T21:36:18.496 DEBUG:boto:Token: None 2023-09-22T21:36:18.496 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.497 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hT4AnrcIYA2L7qYS6Ft+w/wXn20= 2023-09-22T21:36:18.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hT4AnrcIYA2L7qYS6Ft+w/wXn20=', 'Content-Length': '0'} 2023-09-22T21:36:18.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b00fbfc4f3b69d6-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.500 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.500 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.500 DEBUG:boto:Method: HEAD 2023-09-22T21:36:18.500 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.500 DEBUG:boto:Data: 2023-09-22T21:36:18.501 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.501 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.501 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.501 DEBUG:boto:Params: {} 2023-09-22T21:36:18.501 DEBUG:boto:Token: None 2023-09-22T21:36:18.501 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.501 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hT4AnrcIYA2L7qYS6Ft+w/wXn20= 2023-09-22T21:36:18.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hT4AnrcIYA2L7qYS6Ft+w/wXn20=', 'Content-Length': '0'} 2023-09-22T21:36:18.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab5340baa032e255-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-09-22T21:36:18.504 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.505 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.505 DEBUG:boto:Method: GET 2023-09-22T21:36:18.505 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.505 DEBUG:boto:Data: 2023-09-22T21:36:18.505 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.505 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.505 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.505 DEBUG:boto:Params: {} 2023-09-22T21:36:18.505 DEBUG:boto:Token: None 2023-09-22T21:36:18.506 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.506 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1ndvtUX8ldQo7GzQ4moDz7EHXXg= 2023-09-22T21:36:18.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1ndvtUX8ldQo7GzQ4moDz7EHXXg=', 'Content-Length': '0'} 2023-09-22T21:36:18.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a04dc321a1573dcd-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.550 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.551 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.551 DEBUG:boto:Method: GET 2023-09-22T21:36:18.551 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.551 DEBUG:boto:Data: 2023-09-22T21:36:18.551 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.551 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.551 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.551 DEBUG:boto:Params: {} 2023-09-22T21:36:18.551 DEBUG:boto:Token: None 2023-09-22T21:36:18.552 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000028 2023-09-22T21:36:18.552 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1ndvtUX8ldQo7GzQ4moDz7EHXXg= 2023-09-22T21:36:18.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1ndvtUX8ldQo7GzQ4moDz7EHXXg=', 'Content-Length': '0'} 2023-09-22T21:36:18.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d66e770e02de007c-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.555 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:18.555 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:18.555 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000028 2023-09-22T21:36:18.555 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000028 2023-09-22T21:36:18.556 DEBUG:boto:Method: GET 2023-09-22T21:36:18.556 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000028 2023-09-22T21:36:18.556 DEBUG:boto:Data: 2023-09-22T21:36:18.556 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.556 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.556 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.556 DEBUG:boto:Params: {} 2023-09-22T21:36:18.556 DEBUG:boto:Token: None 2023-09-22T21:36:18.556 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/ 2023-09-22T21:36:18.557 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk= 2023-09-22T21:36:18.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk=', 'Content-Length': '0'} 2023-09-22T21:36:18.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c0d31cdbc4cdbe9-00650e08d2-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.561 DEBUG:boto:b'ffzdqu-58obj000000000000000000281000falseobj000000000000000000282023-09-22T21:34:59.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:18.561 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:18.562 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:18.562 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000028 2023-09-22T21:36:18.562 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000028 2023-09-22T21:36:18.562 DEBUG:boto:Method: GET 2023-09-22T21:36:18.562 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000028 2023-09-22T21:36:18.562 DEBUG:boto:Data: 2023-09-22T21:36:18.562 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.562 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.562 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.563 DEBUG:boto:Params: {} 2023-09-22T21:36:18.563 DEBUG:boto:Token: None 2023-09-22T21:36:18.563 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/ 2023-09-22T21:36:18.563 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk= 2023-09-22T21:36:18.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk=', 'Content-Length': '0'} 2023-09-22T21:36:18.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d8a1dbc17262911-00650e08d2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.575 DEBUG:boto:b'ffzdqu-58obj000000000000000000281000falseobj000000000000000000282023-09-22T21:34:59.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:18.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-09-22T21:36:18.575 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.575 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.576 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.576 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.576 DEBUG:boto:Method: GET 2023-09-22T21:36:18.576 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.576 DEBUG:boto:Data: 2023-09-22T21:36:18.576 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.576 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.576 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.576 DEBUG:boto:Params: {} 2023-09-22T21:36:18.576 DEBUG:boto:Token: None 2023-09-22T21:36:18.576 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.577 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Cz/v9oQU5YIurckBOmOh2RUuVhg= 2023-09-22T21:36:18.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Cz/v9oQU5YIurckBOmOh2RUuVhg=', 'Content-Length': '0'} 2023-09-22T21:36:18.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb35cf72a6b89d76-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.580 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.580 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.581 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.581 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.581 DEBUG:boto:Method: GET 2023-09-22T21:36:18.581 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.581 DEBUG:boto:Data: 2023-09-22T21:36:18.581 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.581 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.581 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.581 DEBUG:boto:Params: {} 2023-09-22T21:36:18.582 DEBUG:boto:Token: None 2023-09-22T21:36:18.582 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.582 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Cz/v9oQU5YIurckBOmOh2RUuVhg= 2023-09-22T21:36:18.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Cz/v9oQU5YIurckBOmOh2RUuVhg=', 'Content-Length': '0'} 2023-09-22T21:36:18.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058c0c0d8ca682b5c-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.585 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.585 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.585 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.585 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.585 DEBUG:boto:Method: HEAD 2023-09-22T21:36:18.586 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.586 DEBUG:boto:Data: 2023-09-22T21:36:18.586 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.586 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.586 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.586 DEBUG:boto:Params: {} 2023-09-22T21:36:18.586 DEBUG:boto:Token: None 2023-09-22T21:36:18.586 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.587 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t22rhU3eGrbX4mY1t/OnQpWcTOg= 2023-09-22T21:36:18.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t22rhU3eGrbX4mY1t/OnQpWcTOg=', 'Content-Length': '0'} 2023-09-22T21:36:18.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4c8be1dfb541be4-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.589 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.590 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.590 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.590 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.590 DEBUG:boto:Method: HEAD 2023-09-22T21:36:18.590 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.590 DEBUG:boto:Data: 2023-09-22T21:36:18.590 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.590 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.590 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.591 DEBUG:boto:Params: {} 2023-09-22T21:36:18.591 DEBUG:boto:Token: None 2023-09-22T21:36:18.591 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.591 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t22rhU3eGrbX4mY1t/OnQpWcTOg= 2023-09-22T21:36:18.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t22rhU3eGrbX4mY1t/OnQpWcTOg=', 'Content-Length': '0'} 2023-09-22T21:36:18.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b5693dcf4b1950b-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-09-22T21:36:18.594 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.594 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.594 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.594 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.595 DEBUG:boto:Method: GET 2023-09-22T21:36:18.595 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.595 DEBUG:boto:Data: 2023-09-22T21:36:18.595 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.595 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.595 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.595 DEBUG:boto:Params: {} 2023-09-22T21:36:18.595 DEBUG:boto:Token: None 2023-09-22T21:36:18.595 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.596 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Cz/v9oQU5YIurckBOmOh2RUuVhg= 2023-09-22T21:36:18.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Cz/v9oQU5YIurckBOmOh2RUuVhg=', 'Content-Length': '0'} 2023-09-22T21:36:18.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6dca287e2f5ae22-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.642 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.642 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.643 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.643 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.643 DEBUG:boto:Method: GET 2023-09-22T21:36:18.643 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.643 DEBUG:boto:Data: 2023-09-22T21:36:18.643 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.643 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.643 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.643 DEBUG:boto:Params: {} 2023-09-22T21:36:18.644 DEBUG:boto:Token: None 2023-09-22T21:36:18.644 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000029?versionId=null 2023-09-22T21:36:18.644 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Cz/v9oQU5YIurckBOmOh2RUuVhg= 2023-09-22T21:36:18.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Cz/v9oQU5YIurckBOmOh2RUuVhg=', 'Content-Length': '0'} 2023-09-22T21:36:18.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000731ab24deef732db-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.647 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:18.647 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:18.648 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000029 2023-09-22T21:36:18.648 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000029 2023-09-22T21:36:18.648 DEBUG:boto:Method: GET 2023-09-22T21:36:18.648 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000029 2023-09-22T21:36:18.648 DEBUG:boto:Data: 2023-09-22T21:36:18.648 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.648 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.648 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.648 DEBUG:boto:Params: {} 2023-09-22T21:36:18.649 DEBUG:boto:Token: None 2023-09-22T21:36:18.649 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/ 2023-09-22T21:36:18.649 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk= 2023-09-22T21:36:18.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk=', 'Content-Length': '0'} 2023-09-22T21:36:18.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000750b6f50af4660b7-00650e08d2-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.654 DEBUG:boto:b'ffzdqu-58obj000000000000000000291000falseobj000000000000000000292023-09-22T21:35:03.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:18.654 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:18.654 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:18.654 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000029 2023-09-22T21:36:18.654 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000029 2023-09-22T21:36:18.654 DEBUG:boto:Method: GET 2023-09-22T21:36:18.655 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000029 2023-09-22T21:36:18.655 DEBUG:boto:Data: 2023-09-22T21:36:18.655 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.655 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.655 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.655 DEBUG:boto:Params: {} 2023-09-22T21:36:18.655 DEBUG:boto:Token: None 2023-09-22T21:36:18.655 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/ 2023-09-22T21:36:18.655 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk= 2023-09-22T21:36:18.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk=', 'Content-Length': '0'} 2023-09-22T21:36:18.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ee4da0fc9bdfc6f-00650e08d2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.710 DEBUG:boto:b'ffzdqu-58obj000000000000000000291000falseobj000000000000000000292023-09-22T21:35:03.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:18.711 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.711 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.711 DEBUG:boto:Method: HEAD 2023-09-22T21:36:18.711 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.711 DEBUG:boto:Data: 2023-09-22T21:36:18.711 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.711 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.711 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.712 DEBUG:boto:Params: {} 2023-09-22T21:36:18.712 DEBUG:boto:Token: None 2023-09-22T21:36:18.712 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.712 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ihY9YNhnAu3tFJh0AgjZylnci+Y= 2023-09-22T21:36:18.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ihY9YNhnAu3tFJh0AgjZylnci+Y=', 'Content-Length': '0'} 2023-09-22T21:36:18.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082061a9c3563d601-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.715 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.715 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.716 DEBUG:boto:Method: HEAD 2023-09-22T21:36:18.716 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.716 DEBUG:boto:Data: 2023-09-22T21:36:18.716 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.716 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.716 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.716 DEBUG:boto:Params: {} 2023-09-22T21:36:18.716 DEBUG:boto:Token: None 2023-09-22T21:36:18.717 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.717 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ihY9YNhnAu3tFJh0AgjZylnci+Y= 2023-09-22T21:36:18.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ihY9YNhnAu3tFJh0AgjZylnci+Y=', 'Content-Length': '0'} 2023-09-22T21:36:18.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051c235de83b5c846-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-09-22T21:36:18.720 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.720 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.720 DEBUG:boto:Method: GET 2023-09-22T21:36:18.720 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.720 DEBUG:boto:Data: 2023-09-22T21:36:18.720 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.720 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.721 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.721 DEBUG:boto:Params: {} 2023-09-22T21:36:18.721 DEBUG:boto:Token: None 2023-09-22T21:36:18.721 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.721 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:apLC52bG9T9JiPEriDCEkUN/gEo= 2023-09-22T21:36:18.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:apLC52bG9T9JiPEriDCEkUN/gEo=', 'Content-Length': '0'} 2023-09-22T21:36:18.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca546e7730f2336c-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.766 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.766 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.767 DEBUG:boto:Method: GET 2023-09-22T21:36:18.767 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.767 DEBUG:boto:Data: 2023-09-22T21:36:18.767 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.767 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.767 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.767 DEBUG:boto:Params: {} 2023-09-22T21:36:18.767 DEBUG:boto:Token: None 2023-09-22T21:36:18.768 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000029 2023-09-22T21:36:18.768 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:apLC52bG9T9JiPEriDCEkUN/gEo= 2023-09-22T21:36:18.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:apLC52bG9T9JiPEriDCEkUN/gEo=', 'Content-Length': '0'} 2023-09-22T21:36:18.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbc739803a062e72-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.771 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:18.772 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:18.772 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000029 2023-09-22T21:36:18.772 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000029 2023-09-22T21:36:18.772 DEBUG:boto:Method: GET 2023-09-22T21:36:18.772 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000029 2023-09-22T21:36:18.772 DEBUG:boto:Data: 2023-09-22T21:36:18.772 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.772 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.772 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.772 DEBUG:boto:Params: {} 2023-09-22T21:36:18.773 DEBUG:boto:Token: None 2023-09-22T21:36:18.773 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/ 2023-09-22T21:36:18.773 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk= 2023-09-22T21:36:18.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk=', 'Content-Length': '0'} 2023-09-22T21:36:18.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006022c98d97335641-00650e08d2-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.778 DEBUG:boto:b'ffzdqu-58obj000000000000000000291000falseobj000000000000000000292023-09-22T21:35:03.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:18.778 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:18.779 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:18.779 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000029 2023-09-22T21:36:18.779 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000029 2023-09-22T21:36:18.779 DEBUG:boto:Method: GET 2023-09-22T21:36:18.779 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000029 2023-09-22T21:36:18.779 DEBUG:boto:Data: 2023-09-22T21:36:18.779 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.779 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.779 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.779 DEBUG:boto:Params: {} 2023-09-22T21:36:18.780 DEBUG:boto:Token: None 2023-09-22T21:36:18.780 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/ 2023-09-22T21:36:18.780 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk= 2023-09-22T21:36:18.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk=', 'Content-Length': '0'} 2023-09-22T21:36:18.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000724eb39550a72735-00650e08d2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.790 DEBUG:boto:b'ffzdqu-58obj000000000000000000291000falseobj000000000000000000292023-09-22T21:35:03.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:18.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-09-22T21:36:18.790 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.791 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.791 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.791 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.791 DEBUG:boto:Method: GET 2023-09-22T21:36:18.791 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.791 DEBUG:boto:Data: 2023-09-22T21:36:18.791 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.791 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.791 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.791 DEBUG:boto:Params: {} 2023-09-22T21:36:18.792 DEBUG:boto:Token: None 2023-09-22T21:36:18.792 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.792 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:acsAoCnSLdM2t+BfOmO87pz+C0Q= 2023-09-22T21:36:18.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:acsAoCnSLdM2t+BfOmO87pz+C0Q=', 'Content-Length': '0'} 2023-09-22T21:36:18.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4aea181b7234e2a-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.796 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.796 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.796 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.796 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.797 DEBUG:boto:Method: GET 2023-09-22T21:36:18.797 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.797 DEBUG:boto:Data: 2023-09-22T21:36:18.797 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.797 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.797 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.797 DEBUG:boto:Params: {} 2023-09-22T21:36:18.797 DEBUG:boto:Token: None 2023-09-22T21:36:18.797 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.798 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:acsAoCnSLdM2t+BfOmO87pz+C0Q= 2023-09-22T21:36:18.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:acsAoCnSLdM2t+BfOmO87pz+C0Q=', 'Content-Length': '0'} 2023-09-22T21:36:18.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be77039c7542f14b-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.842 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.843 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.843 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.843 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.843 DEBUG:boto:Method: HEAD 2023-09-22T21:36:18.843 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.843 DEBUG:boto:Data: 2023-09-22T21:36:18.843 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.843 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.843 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.843 DEBUG:boto:Params: {} 2023-09-22T21:36:18.844 DEBUG:boto:Token: None 2023-09-22T21:36:18.844 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.844 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:56ma9RD9aPw8Ycem6ncbrA9kpc8= 2023-09-22T21:36:18.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:56ma9RD9aPw8Ycem6ncbrA9kpc8=', 'Content-Length': '0'} 2023-09-22T21:36:18.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ea7ee3fab76b421-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.847 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.847 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.847 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.847 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.848 DEBUG:boto:Method: HEAD 2023-09-22T21:36:18.848 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.848 DEBUG:boto:Data: 2023-09-22T21:36:18.848 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.848 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.848 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.848 DEBUG:boto:Params: {} 2023-09-22T21:36:18.848 DEBUG:boto:Token: None 2023-09-22T21:36:18.848 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.849 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:56ma9RD9aPw8Ycem6ncbrA9kpc8= 2023-09-22T21:36:18.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:56ma9RD9aPw8Ycem6ncbrA9kpc8=', 'Content-Length': '0'} 2023-09-22T21:36:18.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a25cc5c36806a718-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-09-22T21:36:18.852 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.852 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.852 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.852 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.852 DEBUG:boto:Method: GET 2023-09-22T21:36:18.852 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.852 DEBUG:boto:Data: 2023-09-22T21:36:18.853 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.853 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.853 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.853 DEBUG:boto:Params: {} 2023-09-22T21:36:18.853 DEBUG:boto:Token: None 2023-09-22T21:36:18.853 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.853 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:acsAoCnSLdM2t+BfOmO87pz+C0Q= 2023-09-22T21:36:18.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:acsAoCnSLdM2t+BfOmO87pz+C0Q=', 'Content-Length': '0'} 2023-09-22T21:36:18.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b5b7441d4ef1771-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.902 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.903 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.903 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.903 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.903 DEBUG:boto:Method: GET 2023-09-22T21:36:18.903 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.904 DEBUG:boto:Data: 2023-09-22T21:36:18.904 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.904 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.904 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.904 DEBUG:boto:Params: {} 2023-09-22T21:36:18.904 DEBUG:boto:Token: None 2023-09-22T21:36:18.905 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000030?versionId=null 2023-09-22T21:36:18.905 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:acsAoCnSLdM2t+BfOmO87pz+C0Q= 2023-09-22T21:36:18.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:acsAoCnSLdM2t+BfOmO87pz+C0Q=', 'Content-Length': '0'} 2023-09-22T21:36:18.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006920b741a13d3182-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.908 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:18.909 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:18.909 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000030 2023-09-22T21:36:18.909 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000030 2023-09-22T21:36:18.909 DEBUG:boto:Method: GET 2023-09-22T21:36:18.909 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000030 2023-09-22T21:36:18.909 DEBUG:boto:Data: 2023-09-22T21:36:18.909 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.909 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.909 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.910 DEBUG:boto:Params: {} 2023-09-22T21:36:18.910 DEBUG:boto:Token: None 2023-09-22T21:36:18.910 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/ 2023-09-22T21:36:18.910 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk= 2023-09-22T21:36:18.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk=', 'Content-Length': '0'} 2023-09-22T21:36:18.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c16fe4b22c62885b-00650e08d2-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.915 DEBUG:boto:b'ffzdqu-58obj000000000000000000301000falseobj000000000000000000302023-09-22T21:35:11.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:18.916 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:18.916 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:18.916 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000030 2023-09-22T21:36:18.916 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000030 2023-09-22T21:36:18.916 DEBUG:boto:Method: GET 2023-09-22T21:36:18.916 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000030 2023-09-22T21:36:18.916 DEBUG:boto:Data: 2023-09-22T21:36:18.916 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.917 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.917 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.917 DEBUG:boto:Params: {} 2023-09-22T21:36:18.917 DEBUG:boto:Token: None 2023-09-22T21:36:18.917 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/ 2023-09-22T21:36:18.917 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk= 2023-09-22T21:36:18.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk=', 'Content-Length': '0'} 2023-09-22T21:36:18.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083d57073543fad02-00650e08d2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.927 DEBUG:boto:b'ffzdqu-58obj000000000000000000301000falseobj000000000000000000302023-09-22T21:35:11.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:18.928 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.928 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.928 DEBUG:boto:Method: HEAD 2023-09-22T21:36:18.928 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.928 DEBUG:boto:Data: 2023-09-22T21:36:18.928 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.928 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.929 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.929 DEBUG:boto:Params: {} 2023-09-22T21:36:18.929 DEBUG:boto:Token: None 2023-09-22T21:36:18.929 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.929 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jDmA3fxS9hR6FAl6RvtPu9TsMEQ= 2023-09-22T21:36:18.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jDmA3fxS9hR6FAl6RvtPu9TsMEQ=', 'Content-Length': '0'} 2023-09-22T21:36:18.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009286330e06ed4de2-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.933 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.933 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.933 DEBUG:boto:Method: HEAD 2023-09-22T21:36:18.933 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.933 DEBUG:boto:Data: 2023-09-22T21:36:18.933 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.933 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.933 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.933 DEBUG:boto:Params: {} 2023-09-22T21:36:18.934 DEBUG:boto:Token: None 2023-09-22T21:36:18.934 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.934 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jDmA3fxS9hR6FAl6RvtPu9TsMEQ= 2023-09-22T21:36:18.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jDmA3fxS9hR6FAl6RvtPu9TsMEQ=', 'Content-Length': '0'} 2023-09-22T21:36:18.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a60107ce83569e5-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-09-22T21:36:18.937 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.938 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.938 DEBUG:boto:Method: GET 2023-09-22T21:36:18.938 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.938 DEBUG:boto:Data: 2023-09-22T21:36:18.938 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.939 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.939 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.939 DEBUG:boto:Params: {} 2023-09-22T21:36:18.939 DEBUG:boto:Token: None 2023-09-22T21:36:18.941 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.941 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GaU8zm7EiBuUtpwDcdRfSY1JSF0= 2023-09-22T21:36:18.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GaU8zm7EiBuUtpwDcdRfSY1JSF0=', 'Content-Length': '0'} 2023-09-22T21:36:18.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed313a1140f54158-00650e08d2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.986 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.987 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.987 DEBUG:boto:Method: GET 2023-09-22T21:36:18.987 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.987 DEBUG:boto:Data: 2023-09-22T21:36:18.987 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.988 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:18.988 DEBUG:boto:Port: 8000 2023-09-22T21:36:18.988 DEBUG:boto:Params: {} 2023-09-22T21:36:18.988 DEBUG:boto:Token: None 2023-09-22T21:36:18.989 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/obj00000000000000000030 2023-09-22T21:36:18.989 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GaU8zm7EiBuUtpwDcdRfSY1JSF0= 2023-09-22T21:36:18.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GaU8zm7EiBuUtpwDcdRfSY1JSF0=', 'Content-Length': '0'} 2023-09-22T21:36:18.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015319cd70399b289-00650e08d2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.992 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:18.992 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:18.992 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000030 2023-09-22T21:36:18.992 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000030 2023-09-22T21:36:18.993 DEBUG:boto:Method: GET 2023-09-22T21:36:18.993 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000030 2023-09-22T21:36:18.993 DEBUG:boto:Data: 2023-09-22T21:36:18.993 DEBUG:boto:Headers: {} 2023-09-22T21:36:18.993 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:18.993 DEBUG:boto:Port: 8001 2023-09-22T21:36:18.993 DEBUG:boto:Params: {} 2023-09-22T21:36:18.993 DEBUG:boto:Token: None 2023-09-22T21:36:18.993 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:18 GMT /ffzdqu-58/ 2023-09-22T21:36:18.994 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk= 2023-09-22T21:36:18.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QoiAxm0API5BXmdBkkzUbfidNvk=', 'Content-Length': '0'} 2023-09-22T21:36:18.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3cea59a387cb3c9-00650e08d2-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:18.999 DEBUG:boto:b'ffzdqu-58obj000000000000000000301000falseobj000000000000000000302023-09-22T21:35:11.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:19.000 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:19.000 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:19.000 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000030 2023-09-22T21:36:19.000 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000030 2023-09-22T21:36:19.000 DEBUG:boto:Method: GET 2023-09-22T21:36:19.000 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000030 2023-09-22T21:36:19.000 DEBUG:boto:Data: 2023-09-22T21:36:19.000 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.001 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.001 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.001 DEBUG:boto:Params: {} 2023-09-22T21:36:19.001 DEBUG:boto:Token: None 2023-09-22T21:36:19.001 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/ 2023-09-22T21:36:19.001 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI= 2023-09-22T21:36:19.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI=', 'Content-Length': '0'} 2023-09-22T21:36:19.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078b9c8caf775afc1-00650e08d2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.011 DEBUG:boto:b'ffzdqu-58obj000000000000000000301000falseobj000000000000000000302023-09-22T21:35:11.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:19.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-09-22T21:36:19.011 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.011 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.011 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.011 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.012 DEBUG:boto:Method: GET 2023-09-22T21:36:19.012 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.012 DEBUG:boto:Data: 2023-09-22T21:36:19.012 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.012 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.012 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.012 DEBUG:boto:Params: {} 2023-09-22T21:36:19.012 DEBUG:boto:Token: None 2023-09-22T21:36:19.012 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.013 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ia4XIu8tXCBQmmy1DGrHVaSpDmw= 2023-09-22T21:36:19.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ia4XIu8tXCBQmmy1DGrHVaSpDmw=', 'Content-Length': '0'} 2023-09-22T21:36:19.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcd6f3ba14deb8e5-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.016 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.016 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.016 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.016 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.017 DEBUG:boto:Method: GET 2023-09-22T21:36:19.017 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.017 DEBUG:boto:Data: 2023-09-22T21:36:19.017 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.017 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.017 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.017 DEBUG:boto:Params: {} 2023-09-22T21:36:19.017 DEBUG:boto:Token: None 2023-09-22T21:36:19.017 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.018 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ia4XIu8tXCBQmmy1DGrHVaSpDmw= 2023-09-22T21:36:19.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ia4XIu8tXCBQmmy1DGrHVaSpDmw=', 'Content-Length': '0'} 2023-09-22T21:36:19.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007872b42283577d43-00650e08d3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.021 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.021 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.021 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.021 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.021 DEBUG:boto:Method: HEAD 2023-09-22T21:36:19.021 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.022 DEBUG:boto:Data: 2023-09-22T21:36:19.022 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.022 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.022 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.022 DEBUG:boto:Params: {} 2023-09-22T21:36:19.022 DEBUG:boto:Token: None 2023-09-22T21:36:19.022 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.022 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i+nIat11/xBr3tn2twifvxL4ZnU= 2023-09-22T21:36:19.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i+nIat11/xBr3tn2twifvxL4ZnU=', 'Content-Length': '0'} 2023-09-22T21:36:19.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049e219178e3399ac-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.025 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.025 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.025 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.026 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.026 DEBUG:boto:Method: HEAD 2023-09-22T21:36:19.026 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.026 DEBUG:boto:Data: 2023-09-22T21:36:19.026 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.026 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.026 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.026 DEBUG:boto:Params: {} 2023-09-22T21:36:19.026 DEBUG:boto:Token: None 2023-09-22T21:36:19.027 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.027 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i+nIat11/xBr3tn2twifvxL4ZnU= 2023-09-22T21:36:19.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i+nIat11/xBr3tn2twifvxL4ZnU=', 'Content-Length': '0'} 2023-09-22T21:36:19.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cf3ae5270a5f0e0-00650e08d3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-09-22T21:36:19.030 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.030 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.030 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.030 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.030 DEBUG:boto:Method: GET 2023-09-22T21:36:19.030 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.030 DEBUG:boto:Data: 2023-09-22T21:36:19.031 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.031 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.031 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.031 DEBUG:boto:Params: {} 2023-09-22T21:36:19.031 DEBUG:boto:Token: None 2023-09-22T21:36:19.031 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.031 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ia4XIu8tXCBQmmy1DGrHVaSpDmw= 2023-09-22T21:36:19.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ia4XIu8tXCBQmmy1DGrHVaSpDmw=', 'Content-Length': '0'} 2023-09-22T21:36:19.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079a52739acd1c9c9-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.074 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.074 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.075 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.075 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.075 DEBUG:boto:Method: GET 2023-09-22T21:36:19.075 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.075 DEBUG:boto:Data: 2023-09-22T21:36:19.075 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.075 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.075 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.075 DEBUG:boto:Params: {} 2023-09-22T21:36:19.075 DEBUG:boto:Token: None 2023-09-22T21:36:19.076 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000031?versionId=null 2023-09-22T21:36:19.076 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ia4XIu8tXCBQmmy1DGrHVaSpDmw= 2023-09-22T21:36:19.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ia4XIu8tXCBQmmy1DGrHVaSpDmw=', 'Content-Length': '0'} 2023-09-22T21:36:19.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc7c88fab61bdead-00650e08d3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.079 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:19.079 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:19.079 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000031 2023-09-22T21:36:19.080 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000031 2023-09-22T21:36:19.080 DEBUG:boto:Method: GET 2023-09-22T21:36:19.080 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000031 2023-09-22T21:36:19.080 DEBUG:boto:Data: 2023-09-22T21:36:19.080 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.080 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.080 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.080 DEBUG:boto:Params: {} 2023-09-22T21:36:19.080 DEBUG:boto:Token: None 2023-09-22T21:36:19.081 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/ 2023-09-22T21:36:19.081 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI= 2023-09-22T21:36:19.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI=', 'Content-Length': '0'} 2023-09-22T21:36:19.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014fda3295bcfbdad-00650e08d3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.085 DEBUG:boto:b'ffzdqu-58obj000000000000000000311000falseobj000000000000000000312023-09-22T21:35:03.155Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:19.086 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:19.086 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:19.086 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000031 2023-09-22T21:36:19.086 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000031 2023-09-22T21:36:19.086 DEBUG:boto:Method: GET 2023-09-22T21:36:19.086 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000031 2023-09-22T21:36:19.086 DEBUG:boto:Data: 2023-09-22T21:36:19.087 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.087 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.087 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.087 DEBUG:boto:Params: {} 2023-09-22T21:36:19.087 DEBUG:boto:Token: None 2023-09-22T21:36:19.087 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/ 2023-09-22T21:36:19.087 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI= 2023-09-22T21:36:19.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI=', 'Content-Length': '0'} 2023-09-22T21:36:19.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f1c6c76a9b9b7f7-00650e08d3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.142 DEBUG:boto:b'ffzdqu-58obj000000000000000000311000falseobj000000000000000000312023-09-22T21:35:03.155Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:19.143 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.143 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.143 DEBUG:boto:Method: HEAD 2023-09-22T21:36:19.144 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.144 DEBUG:boto:Data: 2023-09-22T21:36:19.144 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.144 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.144 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.145 DEBUG:boto:Params: {} 2023-09-22T21:36:19.145 DEBUG:boto:Token: None 2023-09-22T21:36:19.145 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.145 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:u2GMcMsziBASwQrA2sVbolOjJww= 2023-09-22T21:36:19.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:u2GMcMsziBASwQrA2sVbolOjJww=', 'Content-Length': '0'} 2023-09-22T21:36:19.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e6a4c57cfcbcc58-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.149 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.150 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.150 DEBUG:boto:Method: HEAD 2023-09-22T21:36:19.150 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.150 DEBUG:boto:Data: 2023-09-22T21:36:19.150 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.150 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.150 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.150 DEBUG:boto:Params: {} 2023-09-22T21:36:19.151 DEBUG:boto:Token: None 2023-09-22T21:36:19.151 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.151 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:u2GMcMsziBASwQrA2sVbolOjJww= 2023-09-22T21:36:19.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:u2GMcMsziBASwQrA2sVbolOjJww=', 'Content-Length': '0'} 2023-09-22T21:36:19.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdf56264fa4bf6d3-00650e08d3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-09-22T21:36:19.155 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.155 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.155 DEBUG:boto:Method: GET 2023-09-22T21:36:19.155 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.155 DEBUG:boto:Data: 2023-09-22T21:36:19.155 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.155 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.155 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.155 DEBUG:boto:Params: {} 2023-09-22T21:36:19.155 DEBUG:boto:Token: None 2023-09-22T21:36:19.156 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.156 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BYLjq8UW93pznc1vOEv/ZckYTa8= 2023-09-22T21:36:19.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BYLjq8UW93pznc1vOEv/ZckYTa8=', 'Content-Length': '0'} 2023-09-22T21:36:19.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a87b5dd4d4af96ee-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.202 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.203 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.203 DEBUG:boto:Method: GET 2023-09-22T21:36:19.203 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.203 DEBUG:boto:Data: 2023-09-22T21:36:19.203 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.203 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.203 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.203 DEBUG:boto:Params: {} 2023-09-22T21:36:19.203 DEBUG:boto:Token: None 2023-09-22T21:36:19.204 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000031 2023-09-22T21:36:19.204 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BYLjq8UW93pznc1vOEv/ZckYTa8= 2023-09-22T21:36:19.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BYLjq8UW93pznc1vOEv/ZckYTa8=', 'Content-Length': '0'} 2023-09-22T21:36:19.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001805b4795d00b1e9-00650e08d3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.207 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:19.207 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:19.207 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000031 2023-09-22T21:36:19.207 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000031 2023-09-22T21:36:19.207 DEBUG:boto:Method: GET 2023-09-22T21:36:19.208 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000031 2023-09-22T21:36:19.208 DEBUG:boto:Data: 2023-09-22T21:36:19.208 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.208 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.208 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.208 DEBUG:boto:Params: {} 2023-09-22T21:36:19.208 DEBUG:boto:Token: None 2023-09-22T21:36:19.208 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/ 2023-09-22T21:36:19.208 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI= 2023-09-22T21:36:19.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI=', 'Content-Length': '0'} 2023-09-22T21:36:19.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030cca486d3595960-00650e08d3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.258 DEBUG:boto:b'ffzdqu-58obj000000000000000000311000falseobj000000000000000000312023-09-22T21:35:03.155Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:19.259 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:19.259 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:19.259 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000031 2023-09-22T21:36:19.259 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000031 2023-09-22T21:36:19.259 DEBUG:boto:Method: GET 2023-09-22T21:36:19.259 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000031 2023-09-22T21:36:19.260 DEBUG:boto:Data: 2023-09-22T21:36:19.260 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.260 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.260 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.260 DEBUG:boto:Params: {} 2023-09-22T21:36:19.260 DEBUG:boto:Token: None 2023-09-22T21:36:19.260 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/ 2023-09-22T21:36:19.261 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI= 2023-09-22T21:36:19.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI=', 'Content-Length': '0'} 2023-09-22T21:36:19.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f7909dcc909b549-00650e08d3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.270 DEBUG:boto:b'ffzdqu-58obj000000000000000000311000falseobj000000000000000000312023-09-22T21:35:03.155Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:19.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-09-22T21:36:19.271 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.271 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.271 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.271 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.271 DEBUG:boto:Method: GET 2023-09-22T21:36:19.272 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.272 DEBUG:boto:Data: 2023-09-22T21:36:19.272 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.272 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.272 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.272 DEBUG:boto:Params: {} 2023-09-22T21:36:19.272 DEBUG:boto:Token: None 2023-09-22T21:36:19.272 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.273 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WT2LW/96tkmIjMkCknhA+w1MGsE= 2023-09-22T21:36:19.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WT2LW/96tkmIjMkCknhA+w1MGsE=', 'Content-Length': '0'} 2023-09-22T21:36:19.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfd55bbe41a2663d-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.276 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.276 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.276 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.277 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.277 DEBUG:boto:Method: GET 2023-09-22T21:36:19.277 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.277 DEBUG:boto:Data: 2023-09-22T21:36:19.277 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.277 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.277 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.277 DEBUG:boto:Params: {} 2023-09-22T21:36:19.277 DEBUG:boto:Token: None 2023-09-22T21:36:19.277 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.278 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WT2LW/96tkmIjMkCknhA+w1MGsE= 2023-09-22T21:36:19.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WT2LW/96tkmIjMkCknhA+w1MGsE=', 'Content-Length': '0'} 2023-09-22T21:36:19.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b9894582e049973-00650e08d3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.281 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.281 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.282 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.282 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.282 DEBUG:boto:Method: HEAD 2023-09-22T21:36:19.282 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.282 DEBUG:boto:Data: 2023-09-22T21:36:19.282 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.282 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.282 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.282 DEBUG:boto:Params: {} 2023-09-22T21:36:19.283 DEBUG:boto:Token: None 2023-09-22T21:36:19.283 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.283 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XvCjggBvcbS0YWbXQWwG5ETViDY= 2023-09-22T21:36:19.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XvCjggBvcbS0YWbXQWwG5ETViDY=', 'Content-Length': '0'} 2023-09-22T21:36:19.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001187f057214851c7-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.286 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.286 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.286 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.286 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.286 DEBUG:boto:Method: HEAD 2023-09-22T21:36:19.286 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.286 DEBUG:boto:Data: 2023-09-22T21:36:19.287 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.287 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.287 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.287 DEBUG:boto:Params: {} 2023-09-22T21:36:19.287 DEBUG:boto:Token: None 2023-09-22T21:36:19.287 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.287 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XvCjggBvcbS0YWbXQWwG5ETViDY= 2023-09-22T21:36:19.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XvCjggBvcbS0YWbXQWwG5ETViDY=', 'Content-Length': '0'} 2023-09-22T21:36:19.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afbb91d4c7dae804-00650e08d3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-09-22T21:36:19.291 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.291 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.291 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.291 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.291 DEBUG:boto:Method: GET 2023-09-22T21:36:19.291 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.291 DEBUG:boto:Data: 2023-09-22T21:36:19.291 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.291 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.291 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.292 DEBUG:boto:Params: {} 2023-09-22T21:36:19.292 DEBUG:boto:Token: None 2023-09-22T21:36:19.292 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.292 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WT2LW/96tkmIjMkCknhA+w1MGsE= 2023-09-22T21:36:19.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WT2LW/96tkmIjMkCknhA+w1MGsE=', 'Content-Length': '0'} 2023-09-22T21:36:19.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d67e07be7e71a20-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.338 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.339 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.339 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.339 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.339 DEBUG:boto:Method: GET 2023-09-22T21:36:19.339 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.339 DEBUG:boto:Data: 2023-09-22T21:36:19.339 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.339 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.339 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.339 DEBUG:boto:Params: {} 2023-09-22T21:36:19.340 DEBUG:boto:Token: None 2023-09-22T21:36:19.340 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000032?versionId=null 2023-09-22T21:36:19.340 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WT2LW/96tkmIjMkCknhA+w1MGsE= 2023-09-22T21:36:19.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WT2LW/96tkmIjMkCknhA+w1MGsE=', 'Content-Length': '0'} 2023-09-22T21:36:19.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ee5c99c44992725-00650e08d3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.343 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:19.343 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:19.343 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000032 2023-09-22T21:36:19.344 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000032 2023-09-22T21:36:19.344 DEBUG:boto:Method: GET 2023-09-22T21:36:19.344 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000032 2023-09-22T21:36:19.344 DEBUG:boto:Data: 2023-09-22T21:36:19.344 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.344 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.344 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.344 DEBUG:boto:Params: {} 2023-09-22T21:36:19.344 DEBUG:boto:Token: None 2023-09-22T21:36:19.345 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/ 2023-09-22T21:36:19.345 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI= 2023-09-22T21:36:19.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI=', 'Content-Length': '0'} 2023-09-22T21:36:19.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dac5f2a48865c1f1-00650e08d3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.390 DEBUG:boto:b'ffzdqu-58obj000000000000000000321000falseobj000000000000000000322023-09-22T21:35:01.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:19.391 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:19.391 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:19.391 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000032 2023-09-22T21:36:19.391 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000032 2023-09-22T21:36:19.391 DEBUG:boto:Method: GET 2023-09-22T21:36:19.392 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000032 2023-09-22T21:36:19.392 DEBUG:boto:Data: 2023-09-22T21:36:19.392 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.392 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.392 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.392 DEBUG:boto:Params: {} 2023-09-22T21:36:19.392 DEBUG:boto:Token: None 2023-09-22T21:36:19.393 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/ 2023-09-22T21:36:19.393 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI= 2023-09-22T21:36:19.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI=', 'Content-Length': '0'} 2023-09-22T21:36:19.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aba4e3837e18cbc3-00650e08d3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.404 DEBUG:boto:b'ffzdqu-58obj000000000000000000321000falseobj000000000000000000322023-09-22T21:35:01.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:19.405 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.405 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.405 DEBUG:boto:Method: HEAD 2023-09-22T21:36:19.405 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.405 DEBUG:boto:Data: 2023-09-22T21:36:19.405 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.405 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.405 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.406 DEBUG:boto:Params: {} 2023-09-22T21:36:19.406 DEBUG:boto:Token: None 2023-09-22T21:36:19.406 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.406 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:03y164dumr7fq1X2R9dUTf7IpS0= 2023-09-22T21:36:19.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:03y164dumr7fq1X2R9dUTf7IpS0=', 'Content-Length': '0'} 2023-09-22T21:36:19.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc49c4086ca67667-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.409 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.409 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.409 DEBUG:boto:Method: HEAD 2023-09-22T21:36:19.409 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.409 DEBUG:boto:Data: 2023-09-22T21:36:19.410 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.410 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.410 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.410 DEBUG:boto:Params: {} 2023-09-22T21:36:19.410 DEBUG:boto:Token: None 2023-09-22T21:36:19.410 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.410 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:03y164dumr7fq1X2R9dUTf7IpS0= 2023-09-22T21:36:19.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:03y164dumr7fq1X2R9dUTf7IpS0=', 'Content-Length': '0'} 2023-09-22T21:36:19.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4c7f305332b28c7-00650e08d3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-09-22T21:36:19.414 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.414 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.414 DEBUG:boto:Method: GET 2023-09-22T21:36:19.414 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.414 DEBUG:boto:Data: 2023-09-22T21:36:19.414 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.415 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.415 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.415 DEBUG:boto:Params: {} 2023-09-22T21:36:19.415 DEBUG:boto:Token: None 2023-09-22T21:36:19.415 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.415 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:188dDfGbvqbbMRnKzNdd8XAb7OA= 2023-09-22T21:36:19.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:188dDfGbvqbbMRnKzNdd8XAb7OA=', 'Content-Length': '0'} 2023-09-22T21:36:19.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9244dc6184d9854-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.462 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.463 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.463 DEBUG:boto:Method: GET 2023-09-22T21:36:19.463 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.463 DEBUG:boto:Data: 2023-09-22T21:36:19.463 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.463 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.463 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.464 DEBUG:boto:Params: {} 2023-09-22T21:36:19.464 DEBUG:boto:Token: None 2023-09-22T21:36:19.464 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000032 2023-09-22T21:36:19.464 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:188dDfGbvqbbMRnKzNdd8XAb7OA= 2023-09-22T21:36:19.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:188dDfGbvqbbMRnKzNdd8XAb7OA=', 'Content-Length': '0'} 2023-09-22T21:36:19.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079b0e66d330e6938-00650e08d3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.468 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:19.468 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:19.468 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000032 2023-09-22T21:36:19.468 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000032 2023-09-22T21:36:19.469 DEBUG:boto:Method: GET 2023-09-22T21:36:19.469 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000032 2023-09-22T21:36:19.469 DEBUG:boto:Data: 2023-09-22T21:36:19.469 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.469 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.469 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.470 DEBUG:boto:Params: {} 2023-09-22T21:36:19.470 DEBUG:boto:Token: None 2023-09-22T21:36:19.470 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/ 2023-09-22T21:36:19.470 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI= 2023-09-22T21:36:19.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI=', 'Content-Length': '0'} 2023-09-22T21:36:19.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c3816b7dbd924f8-00650e08d3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.476 DEBUG:boto:b'ffzdqu-58obj000000000000000000321000falseobj000000000000000000322023-09-22T21:35:01.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:19.477 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:19.477 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:19.477 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000032 2023-09-22T21:36:19.477 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000032 2023-09-22T21:36:19.478 DEBUG:boto:Method: GET 2023-09-22T21:36:19.478 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000032 2023-09-22T21:36:19.478 DEBUG:boto:Data: 2023-09-22T21:36:19.478 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.478 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.479 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.479 DEBUG:boto:Params: {} 2023-09-22T21:36:19.479 DEBUG:boto:Token: None 2023-09-22T21:36:19.479 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/ 2023-09-22T21:36:19.480 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI= 2023-09-22T21:36:19.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI=', 'Content-Length': '0'} 2023-09-22T21:36:19.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c75a50f8578daca8-00650e08d3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.490 DEBUG:boto:b'ffzdqu-58obj000000000000000000321000falseobj000000000000000000322023-09-22T21:35:01.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:19.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-09-22T21:36:19.491 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.491 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.491 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.492 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.492 DEBUG:boto:Method: GET 2023-09-22T21:36:19.492 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.492 DEBUG:boto:Data: 2023-09-22T21:36:19.492 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.492 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.493 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.493 DEBUG:boto:Params: {} 2023-09-22T21:36:19.493 DEBUG:boto:Token: None 2023-09-22T21:36:19.493 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.494 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ty/GhABbC/OJ3hiDkxRZAx6kUNM= 2023-09-22T21:36:19.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ty/GhABbC/OJ3hiDkxRZAx6kUNM=', 'Content-Length': '0'} 2023-09-22T21:36:19.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077ddf8503c3cbbf7-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.542 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.543 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.543 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.543 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.543 DEBUG:boto:Method: GET 2023-09-22T21:36:19.543 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.543 DEBUG:boto:Data: 2023-09-22T21:36:19.544 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.544 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.544 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.544 DEBUG:boto:Params: {} 2023-09-22T21:36:19.544 DEBUG:boto:Token: None 2023-09-22T21:36:19.544 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.545 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ty/GhABbC/OJ3hiDkxRZAx6kUNM= 2023-09-22T21:36:19.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ty/GhABbC/OJ3hiDkxRZAx6kUNM=', 'Content-Length': '0'} 2023-09-22T21:36:19.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045c068821f1a09b8-00650e08d3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.548 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.548 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.548 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.548 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.549 DEBUG:boto:Method: HEAD 2023-09-22T21:36:19.549 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.549 DEBUG:boto:Data: 2023-09-22T21:36:19.549 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.549 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.549 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.549 DEBUG:boto:Params: {} 2023-09-22T21:36:19.550 DEBUG:boto:Token: None 2023-09-22T21:36:19.550 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.550 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JLiKisKusKt6qZTggGczW8O15wU= 2023-09-22T21:36:19.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JLiKisKusKt6qZTggGczW8O15wU=', 'Content-Length': '0'} 2023-09-22T21:36:19.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017a15d11e7457717-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.553 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.553 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.553 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.554 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.554 DEBUG:boto:Method: HEAD 2023-09-22T21:36:19.554 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.554 DEBUG:boto:Data: 2023-09-22T21:36:19.554 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.554 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.554 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.555 DEBUG:boto:Params: {} 2023-09-22T21:36:19.555 DEBUG:boto:Token: None 2023-09-22T21:36:19.555 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.555 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JLiKisKusKt6qZTggGczW8O15wU= 2023-09-22T21:36:19.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JLiKisKusKt6qZTggGczW8O15wU=', 'Content-Length': '0'} 2023-09-22T21:36:19.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5c49f4f085f37a7-00650e08d3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-09-22T21:36:19.558 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.558 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.559 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.559 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.559 DEBUG:boto:Method: GET 2023-09-22T21:36:19.559 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.559 DEBUG:boto:Data: 2023-09-22T21:36:19.559 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.559 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.560 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.560 DEBUG:boto:Params: {} 2023-09-22T21:36:19.560 DEBUG:boto:Token: None 2023-09-22T21:36:19.560 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.560 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ty/GhABbC/OJ3hiDkxRZAx6kUNM= 2023-09-22T21:36:19.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ty/GhABbC/OJ3hiDkxRZAx6kUNM=', 'Content-Length': '0'} 2023-09-22T21:36:19.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1350340ba00ab48-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.606 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.607 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.607 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.607 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.607 DEBUG:boto:Method: GET 2023-09-22T21:36:19.608 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.608 DEBUG:boto:Data: 2023-09-22T21:36:19.608 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.608 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.609 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.609 DEBUG:boto:Params: {} 2023-09-22T21:36:19.609 DEBUG:boto:Token: None 2023-09-22T21:36:19.609 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000033?versionId=null 2023-09-22T21:36:19.610 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ty/GhABbC/OJ3hiDkxRZAx6kUNM= 2023-09-22T21:36:19.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ty/GhABbC/OJ3hiDkxRZAx6kUNM=', 'Content-Length': '0'} 2023-09-22T21:36:19.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079db6f0883413731-00650e08d3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.613 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:19.614 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:19.614 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000033 2023-09-22T21:36:19.614 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000033 2023-09-22T21:36:19.614 DEBUG:boto:Method: GET 2023-09-22T21:36:19.614 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000033 2023-09-22T21:36:19.615 DEBUG:boto:Data: 2023-09-22T21:36:19.615 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.615 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.615 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.615 DEBUG:boto:Params: {} 2023-09-22T21:36:19.615 DEBUG:boto:Token: None 2023-09-22T21:36:19.616 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/ 2023-09-22T21:36:19.616 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI= 2023-09-22T21:36:19.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI=', 'Content-Length': '0'} 2023-09-22T21:36:19.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3dabc104b7941e2-00650e08d3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.666 DEBUG:boto:b'ffzdqu-58obj000000000000000000331000falseobj000000000000000000332023-09-22T21:35:01.495Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:19.667 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:19.667 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:19.667 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000033 2023-09-22T21:36:19.667 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000033 2023-09-22T21:36:19.667 DEBUG:boto:Method: GET 2023-09-22T21:36:19.667 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000033 2023-09-22T21:36:19.668 DEBUG:boto:Data: 2023-09-22T21:36:19.668 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.668 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.668 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.668 DEBUG:boto:Params: {} 2023-09-22T21:36:19.669 DEBUG:boto:Token: None 2023-09-22T21:36:19.669 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/ 2023-09-22T21:36:19.669 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI= 2023-09-22T21:36:19.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI=', 'Content-Length': '0'} 2023-09-22T21:36:19.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060997dba77b63117-00650e08d3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.680 DEBUG:boto:b'ffzdqu-58obj000000000000000000331000falseobj000000000000000000332023-09-22T21:35:01.495Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:19.681 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.681 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.681 DEBUG:boto:Method: HEAD 2023-09-22T21:36:19.681 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.681 DEBUG:boto:Data: 2023-09-22T21:36:19.682 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.682 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.682 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.682 DEBUG:boto:Params: {} 2023-09-22T21:36:19.682 DEBUG:boto:Token: None 2023-09-22T21:36:19.683 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.683 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3TsxAae9u3z7FZxQMNmw7amtyss= 2023-09-22T21:36:19.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3TsxAae9u3z7FZxQMNmw7amtyss=', 'Content-Length': '0'} 2023-09-22T21:36:19.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003efaa841f88f2cf9-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.686 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.686 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.686 DEBUG:boto:Method: HEAD 2023-09-22T21:36:19.686 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.687 DEBUG:boto:Data: 2023-09-22T21:36:19.687 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.687 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.687 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.687 DEBUG:boto:Params: {} 2023-09-22T21:36:19.687 DEBUG:boto:Token: None 2023-09-22T21:36:19.688 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.688 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3TsxAae9u3z7FZxQMNmw7amtyss= 2023-09-22T21:36:19.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3TsxAae9u3z7FZxQMNmw7amtyss=', 'Content-Length': '0'} 2023-09-22T21:36:19.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f5897c345453a67-00650e08d3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-09-22T21:36:19.691 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.691 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.691 DEBUG:boto:Method: GET 2023-09-22T21:36:19.691 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.691 DEBUG:boto:Data: 2023-09-22T21:36:19.691 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.691 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.692 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.692 DEBUG:boto:Params: {} 2023-09-22T21:36:19.692 DEBUG:boto:Token: None 2023-09-22T21:36:19.692 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.692 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ifTAgVEkQwdBT9YO5q9veK+r+Hg= 2023-09-22T21:36:19.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ifTAgVEkQwdBT9YO5q9veK+r+Hg=', 'Content-Length': '0'} 2023-09-22T21:36:19.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a080bd53c61b2675-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.742 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.743 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.743 DEBUG:boto:Method: GET 2023-09-22T21:36:19.743 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.743 DEBUG:boto:Data: 2023-09-22T21:36:19.744 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.744 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.744 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.744 DEBUG:boto:Params: {} 2023-09-22T21:36:19.744 DEBUG:boto:Token: None 2023-09-22T21:36:19.745 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000033 2023-09-22T21:36:19.745 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ifTAgVEkQwdBT9YO5q9veK+r+Hg= 2023-09-22T21:36:19.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ifTAgVEkQwdBT9YO5q9veK+r+Hg=', 'Content-Length': '0'} 2023-09-22T21:36:19.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adda59ae7d04f775-00650e08d3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.749 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:19.749 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:19.749 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000033 2023-09-22T21:36:19.749 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000033 2023-09-22T21:36:19.749 DEBUG:boto:Method: GET 2023-09-22T21:36:19.749 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000033 2023-09-22T21:36:19.749 DEBUG:boto:Data: 2023-09-22T21:36:19.750 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.750 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.750 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.750 DEBUG:boto:Params: {} 2023-09-22T21:36:19.750 DEBUG:boto:Token: None 2023-09-22T21:36:19.750 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/ 2023-09-22T21:36:19.750 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI= 2023-09-22T21:36:19.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI=', 'Content-Length': '0'} 2023-09-22T21:36:19.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047f3fc6d26b40ee8-00650e08d3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.755 DEBUG:boto:b'ffzdqu-58obj000000000000000000331000falseobj000000000000000000332023-09-22T21:35:01.495Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:19.756 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:19.756 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:19.756 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000033 2023-09-22T21:36:19.756 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000033 2023-09-22T21:36:19.756 DEBUG:boto:Method: GET 2023-09-22T21:36:19.756 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000033 2023-09-22T21:36:19.756 DEBUG:boto:Data: 2023-09-22T21:36:19.757 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.757 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.757 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.757 DEBUG:boto:Params: {} 2023-09-22T21:36:19.757 DEBUG:boto:Token: None 2023-09-22T21:36:19.757 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/ 2023-09-22T21:36:19.758 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI= 2023-09-22T21:36:19.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI=', 'Content-Length': '0'} 2023-09-22T21:36:19.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8e5df3093ac26f5-00650e08d3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.767 DEBUG:boto:b'ffzdqu-58obj000000000000000000331000falseobj000000000000000000332023-09-22T21:35:01.495Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:19.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-09-22T21:36:19.768 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.768 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.768 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.768 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.769 DEBUG:boto:Method: GET 2023-09-22T21:36:19.769 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.769 DEBUG:boto:Data: 2023-09-22T21:36:19.769 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.769 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.769 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.769 DEBUG:boto:Params: {} 2023-09-22T21:36:19.770 DEBUG:boto:Token: None 2023-09-22T21:36:19.770 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.770 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FW21Qd8IGwxJlHn7xBEZk/UQrTM= 2023-09-22T21:36:19.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FW21Qd8IGwxJlHn7xBEZk/UQrTM=', 'Content-Length': '0'} 2023-09-22T21:36:19.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4b140396f1cc3a7-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.774 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.774 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.774 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.774 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.774 DEBUG:boto:Method: GET 2023-09-22T21:36:19.774 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.775 DEBUG:boto:Data: 2023-09-22T21:36:19.775 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.775 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.775 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.775 DEBUG:boto:Params: {} 2023-09-22T21:36:19.775 DEBUG:boto:Token: None 2023-09-22T21:36:19.776 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.776 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FW21Qd8IGwxJlHn7xBEZk/UQrTM= 2023-09-22T21:36:19.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FW21Qd8IGwxJlHn7xBEZk/UQrTM=', 'Content-Length': '0'} 2023-09-22T21:36:19.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021bacf8dd8360d4b-00650e08d3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.779 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.779 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.779 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.779 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.780 DEBUG:boto:Method: HEAD 2023-09-22T21:36:19.780 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.780 DEBUG:boto:Data: 2023-09-22T21:36:19.780 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.780 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.780 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.780 DEBUG:boto:Params: {} 2023-09-22T21:36:19.781 DEBUG:boto:Token: None 2023-09-22T21:36:19.781 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.781 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:io8vTlvsom1g7MH6OcYbJrC6HgU= 2023-09-22T21:36:19.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:io8vTlvsom1g7MH6OcYbJrC6HgU=', 'Content-Length': '0'} 2023-09-22T21:36:19.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f66bb2d457c1e2a-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.784 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.785 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.785 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.785 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.785 DEBUG:boto:Method: HEAD 2023-09-22T21:36:19.786 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.786 DEBUG:boto:Data: 2023-09-22T21:36:19.786 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.786 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.786 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.787 DEBUG:boto:Params: {} 2023-09-22T21:36:19.787 DEBUG:boto:Token: None 2023-09-22T21:36:19.787 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.787 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:io8vTlvsom1g7MH6OcYbJrC6HgU= 2023-09-22T21:36:19.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:io8vTlvsom1g7MH6OcYbJrC6HgU=', 'Content-Length': '0'} 2023-09-22T21:36:19.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e936c0996e65fa39-00650e08d3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-09-22T21:36:19.791 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.791 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.792 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.792 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.792 DEBUG:boto:Method: GET 2023-09-22T21:36:19.792 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.792 DEBUG:boto:Data: 2023-09-22T21:36:19.792 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.793 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.793 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.793 DEBUG:boto:Params: {} 2023-09-22T21:36:19.793 DEBUG:boto:Token: None 2023-09-22T21:36:19.793 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.793 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FW21Qd8IGwxJlHn7xBEZk/UQrTM= 2023-09-22T21:36:19.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FW21Qd8IGwxJlHn7xBEZk/UQrTM=', 'Content-Length': '0'} 2023-09-22T21:36:19.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099a9dcb6d70ddb0f-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.838 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.838 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.839 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.839 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.839 DEBUG:boto:Method: GET 2023-09-22T21:36:19.839 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.839 DEBUG:boto:Data: 2023-09-22T21:36:19.839 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.839 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.839 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.840 DEBUG:boto:Params: {} 2023-09-22T21:36:19.840 DEBUG:boto:Token: None 2023-09-22T21:36:19.840 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000036?versionId=null 2023-09-22T21:36:19.840 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FW21Qd8IGwxJlHn7xBEZk/UQrTM= 2023-09-22T21:36:19.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FW21Qd8IGwxJlHn7xBEZk/UQrTM=', 'Content-Length': '0'} 2023-09-22T21:36:19.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d04aa00f657d991f-00650e08d3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.844 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:19.845 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:19.845 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000036 2023-09-22T21:36:19.845 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000036 2023-09-22T21:36:19.845 DEBUG:boto:Method: GET 2023-09-22T21:36:19.845 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000036 2023-09-22T21:36:19.845 DEBUG:boto:Data: 2023-09-22T21:36:19.845 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.845 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.846 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.846 DEBUG:boto:Params: {} 2023-09-22T21:36:19.846 DEBUG:boto:Token: None 2023-09-22T21:36:19.846 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/ 2023-09-22T21:36:19.846 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI= 2023-09-22T21:36:19.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI=', 'Content-Length': '0'} 2023-09-22T21:36:19.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1b2d9fe63430287-00650e08d3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.851 DEBUG:boto:b'ffzdqu-58obj000000000000000000361000falseobj000000000000000000362023-09-22T21:35:05.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:19.852 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:19.852 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:19.852 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000036 2023-09-22T21:36:19.852 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000036 2023-09-22T21:36:19.852 DEBUG:boto:Method: GET 2023-09-22T21:36:19.852 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000036 2023-09-22T21:36:19.852 DEBUG:boto:Data: 2023-09-22T21:36:19.852 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.853 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.853 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.853 DEBUG:boto:Params: {} 2023-09-22T21:36:19.853 DEBUG:boto:Token: None 2023-09-22T21:36:19.853 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/ 2023-09-22T21:36:19.853 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI= 2023-09-22T21:36:19.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI=', 'Content-Length': '0'} 2023-09-22T21:36:19.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b91f23ed7265b63a-00650e08d3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.863 DEBUG:boto:b'ffzdqu-58obj000000000000000000361000falseobj000000000000000000362023-09-22T21:35:05.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:19.864 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.864 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.864 DEBUG:boto:Method: HEAD 2023-09-22T21:36:19.864 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.864 DEBUG:boto:Data: 2023-09-22T21:36:19.864 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.865 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.865 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.865 DEBUG:boto:Params: {} 2023-09-22T21:36:19.865 DEBUG:boto:Token: None 2023-09-22T21:36:19.865 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.865 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:28FGCW1BvV9hLQCm3zM9ACiHFYk= 2023-09-22T21:36:19.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:28FGCW1BvV9hLQCm3zM9ACiHFYk=', 'Content-Length': '0'} 2023-09-22T21:36:19.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e04a82f0f6a5344-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.868 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.868 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.868 DEBUG:boto:Method: HEAD 2023-09-22T21:36:19.868 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.869 DEBUG:boto:Data: 2023-09-22T21:36:19.869 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.869 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.869 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.869 DEBUG:boto:Params: {} 2023-09-22T21:36:19.869 DEBUG:boto:Token: None 2023-09-22T21:36:19.869 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.869 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:28FGCW1BvV9hLQCm3zM9ACiHFYk= 2023-09-22T21:36:19.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:28FGCW1BvV9hLQCm3zM9ACiHFYk=', 'Content-Length': '0'} 2023-09-22T21:36:19.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094d82528218099c7-00650e08d3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-09-22T21:36:19.872 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.873 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.873 DEBUG:boto:Method: GET 2023-09-22T21:36:19.873 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.873 DEBUG:boto:Data: 2023-09-22T21:36:19.873 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.873 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.873 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.873 DEBUG:boto:Params: {} 2023-09-22T21:36:19.873 DEBUG:boto:Token: None 2023-09-22T21:36:19.874 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.874 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PjUbYvvbTU6cPqCa4ylEH5g+njw= 2023-09-22T21:36:19.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PjUbYvvbTU6cPqCa4ylEH5g+njw=', 'Content-Length': '0'} 2023-09-22T21:36:19.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb6358c40126c20e-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.922 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.922 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.923 DEBUG:boto:Method: GET 2023-09-22T21:36:19.923 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.923 DEBUG:boto:Data: 2023-09-22T21:36:19.923 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.923 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.923 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.923 DEBUG:boto:Params: {} 2023-09-22T21:36:19.923 DEBUG:boto:Token: None 2023-09-22T21:36:19.924 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000036 2023-09-22T21:36:19.924 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PjUbYvvbTU6cPqCa4ylEH5g+njw= 2023-09-22T21:36:19.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PjUbYvvbTU6cPqCa4ylEH5g+njw=', 'Content-Length': '0'} 2023-09-22T21:36:19.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c41cf7af8329b3f-00650e08d3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.927 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:19.927 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:19.927 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000036 2023-09-22T21:36:19.927 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000036 2023-09-22T21:36:19.927 DEBUG:boto:Method: GET 2023-09-22T21:36:19.927 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000036 2023-09-22T21:36:19.927 DEBUG:boto:Data: 2023-09-22T21:36:19.927 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.928 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.928 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.928 DEBUG:boto:Params: {} 2023-09-22T21:36:19.928 DEBUG:boto:Token: None 2023-09-22T21:36:19.928 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/ 2023-09-22T21:36:19.928 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI= 2023-09-22T21:36:19.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI=', 'Content-Length': '0'} 2023-09-22T21:36:19.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be7f6de2c613bd35-00650e08d3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.934 DEBUG:boto:b'ffzdqu-58obj000000000000000000361000falseobj000000000000000000362023-09-22T21:35:05.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:19.934 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:19.935 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:19.935 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000036 2023-09-22T21:36:19.935 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000036 2023-09-22T21:36:19.935 DEBUG:boto:Method: GET 2023-09-22T21:36:19.935 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000036 2023-09-22T21:36:19.935 DEBUG:boto:Data: 2023-09-22T21:36:19.935 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.935 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.935 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.935 DEBUG:boto:Params: {} 2023-09-22T21:36:19.936 DEBUG:boto:Token: None 2023-09-22T21:36:19.936 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/ 2023-09-22T21:36:19.936 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI= 2023-09-22T21:36:19.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5WRSC30penLjPtf/cEza6XWATgI=', 'Content-Length': '0'} 2023-09-22T21:36:19.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c187f6b3848e8163-00650e08d3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.990 DEBUG:boto:b'ffzdqu-58obj000000000000000000361000falseobj000000000000000000362023-09-22T21:35:05.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:19.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2023-09-22T21:36:19.991 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:19.991 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:19.991 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:19.991 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:19.991 DEBUG:boto:Method: GET 2023-09-22T21:36:19.992 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:19.992 DEBUG:boto:Data: 2023-09-22T21:36:19.992 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.992 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:19.992 DEBUG:boto:Port: 8001 2023-09-22T21:36:19.992 DEBUG:boto:Params: {} 2023-09-22T21:36:19.992 DEBUG:boto:Token: None 2023-09-22T21:36:19.992 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:19.992 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IABm7OXwLrUi/PE+zv9SbbKk0K0= 2023-09-22T21:36:19.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IABm7OXwLrUi/PE+zv9SbbKk0K0=', 'Content-Length': '0'} 2023-09-22T21:36:19.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecb38a6d4bbf8f6f-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:19.996 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:19.996 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:19.996 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:19.996 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:19.996 DEBUG:boto:Method: GET 2023-09-22T21:36:19.996 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:19.997 DEBUG:boto:Data: 2023-09-22T21:36:19.997 DEBUG:boto:Headers: {} 2023-09-22T21:36:19.997 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:19.997 DEBUG:boto:Port: 8000 2023-09-22T21:36:19.997 DEBUG:boto:Params: {} 2023-09-22T21:36:19.997 DEBUG:boto:Token: None 2023-09-22T21:36:19.997 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:19 GMT /ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:19.997 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IABm7OXwLrUi/PE+zv9SbbKk0K0= 2023-09-22T21:36:19.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IABm7OXwLrUi/PE+zv9SbbKk0K0=', 'Content-Length': '0'} 2023-09-22T21:36:20.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7e63a1078470aaf-00650e08d3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.001 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.001 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.001 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:20.001 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:20.001 DEBUG:boto:Method: HEAD 2023-09-22T21:36:20.001 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:20.001 DEBUG:boto:Data: 2023-09-22T21:36:20.001 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.002 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.002 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.002 DEBUG:boto:Params: {} 2023-09-22T21:36:20.002 DEBUG:boto:Token: None 2023-09-22T21:36:20.002 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:20.002 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:W9a3Pwy1wjJbWdPztFzKqZuYR5g= 2023-09-22T21:36:20.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:W9a3Pwy1wjJbWdPztFzKqZuYR5g=', 'Content-Length': '0'} 2023-09-22T21:36:20.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000842287ae5d6b955f-00650e08d3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.005 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.005 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.005 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:20.006 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:20.006 DEBUG:boto:Method: HEAD 2023-09-22T21:36:20.006 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:20.006 DEBUG:boto:Data: 2023-09-22T21:36:20.006 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.006 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.006 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.006 DEBUG:boto:Params: {} 2023-09-22T21:36:20.006 DEBUG:boto:Token: None 2023-09-22T21:36:20.007 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:20.007 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:W9a3Pwy1wjJbWdPztFzKqZuYR5g= 2023-09-22T21:36:20.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:W9a3Pwy1wjJbWdPztFzKqZuYR5g=', 'Content-Length': '0'} 2023-09-22T21:36:20.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099ea9422879ee2e4-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2023-09-22T21:36:20.010 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.010 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.010 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:20.010 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:20.010 DEBUG:boto:Method: GET 2023-09-22T21:36:20.010 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:20.010 DEBUG:boto:Data: 2023-09-22T21:36:20.011 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.011 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.011 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.011 DEBUG:boto:Params: {} 2023-09-22T21:36:20.011 DEBUG:boto:Token: None 2023-09-22T21:36:20.011 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:20.011 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7jTS9f57D2XoZpwyDHVbYLTrM84= 2023-09-22T21:36:20.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7jTS9f57D2XoZpwyDHVbYLTrM84=', 'Content-Length': '0'} 2023-09-22T21:36:20.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097b1175278225c7c-00650e08d4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.057 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.057 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.057 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:20.058 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:20.058 DEBUG:boto:Method: GET 2023-09-22T21:36:20.058 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:20.058 DEBUG:boto:Data: 2023-09-22T21:36:20.058 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.058 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.058 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.058 DEBUG:boto:Params: {} 2023-09-22T21:36:20.058 DEBUG:boto:Token: None 2023-09-22T21:36:20.059 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000037?versionId=null 2023-09-22T21:36:20.059 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7jTS9f57D2XoZpwyDHVbYLTrM84= 2023-09-22T21:36:20.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7jTS9f57D2XoZpwyDHVbYLTrM84=', 'Content-Length': '0'} 2023-09-22T21:36:20.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002956f76ef3c200c9-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.062 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:20.062 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:20.062 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000037 2023-09-22T21:36:20.062 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000037 2023-09-22T21:36:20.063 DEBUG:boto:Method: GET 2023-09-22T21:36:20.063 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000037 2023-09-22T21:36:20.063 DEBUG:boto:Data: 2023-09-22T21:36:20.063 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.063 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.063 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.063 DEBUG:boto:Params: {} 2023-09-22T21:36:20.063 DEBUG:boto:Token: None 2023-09-22T21:36:20.063 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/ 2023-09-22T21:36:20.064 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0= 2023-09-22T21:36:20.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0=', 'Content-Length': '0'} 2023-09-22T21:36:20.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e623cdc6d8bc43c-00650e08d4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.068 DEBUG:boto:b'ffzdqu-58obj000000000000000000371000falseobj000000000000000000372023-09-22T21:35:02.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:20.069 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:20.069 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:20.069 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000037 2023-09-22T21:36:20.069 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000037 2023-09-22T21:36:20.069 DEBUG:boto:Method: GET 2023-09-22T21:36:20.069 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000037 2023-09-22T21:36:20.069 DEBUG:boto:Data: 2023-09-22T21:36:20.069 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.069 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.070 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.070 DEBUG:boto:Params: {} 2023-09-22T21:36:20.070 DEBUG:boto:Token: None 2023-09-22T21:36:20.070 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/ 2023-09-22T21:36:20.070 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0= 2023-09-22T21:36:20.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0=', 'Content-Length': '0'} 2023-09-22T21:36:20.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9ab53f3577fb11e-00650e08d4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.081 DEBUG:boto:b'ffzdqu-58obj000000000000000000371000falseobj000000000000000000372023-09-22T21:35:02.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:20.081 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.081 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.081 DEBUG:boto:Method: HEAD 2023-09-22T21:36:20.082 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.082 DEBUG:boto:Data: 2023-09-22T21:36:20.082 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.082 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.082 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.082 DEBUG:boto:Params: {} 2023-09-22T21:36:20.082 DEBUG:boto:Token: None 2023-09-22T21:36:20.082 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.082 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SeqYSm3fv3GPhEM4A8Jns5DqRF8= 2023-09-22T21:36:20.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SeqYSm3fv3GPhEM4A8Jns5DqRF8=', 'Content-Length': '0'} 2023-09-22T21:36:20.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecf229526928ee87-00650e08d4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.085 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.085 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.085 DEBUG:boto:Method: HEAD 2023-09-22T21:36:20.086 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.086 DEBUG:boto:Data: 2023-09-22T21:36:20.086 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.086 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.086 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.086 DEBUG:boto:Params: {} 2023-09-22T21:36:20.086 DEBUG:boto:Token: None 2023-09-22T21:36:20.086 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.087 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SeqYSm3fv3GPhEM4A8Jns5DqRF8= 2023-09-22T21:36:20.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SeqYSm3fv3GPhEM4A8Jns5DqRF8=', 'Content-Length': '0'} 2023-09-22T21:36:20.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d76af4fc004fcff-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2023-09-22T21:36:20.089 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.090 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.090 DEBUG:boto:Method: GET 2023-09-22T21:36:20.090 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.090 DEBUG:boto:Data: 2023-09-22T21:36:20.090 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.091 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.091 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.091 DEBUG:boto:Params: {} 2023-09-22T21:36:20.091 DEBUG:boto:Token: None 2023-09-22T21:36:20.091 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.092 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Xrl6Q1PTCsVfNQJcLVnQn5bdnFA= 2023-09-22T21:36:20.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Xrl6Q1PTCsVfNQJcLVnQn5bdnFA=', 'Content-Length': '0'} 2023-09-22T21:36:20.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df2c5784339d8f35-00650e08d4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.138 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.139 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.139 DEBUG:boto:Method: GET 2023-09-22T21:36:20.139 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.139 DEBUG:boto:Data: 2023-09-22T21:36:20.139 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.139 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.139 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.139 DEBUG:boto:Params: {} 2023-09-22T21:36:20.139 DEBUG:boto:Token: None 2023-09-22T21:36:20.140 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000037 2023-09-22T21:36:20.140 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Xrl6Q1PTCsVfNQJcLVnQn5bdnFA= 2023-09-22T21:36:20.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Xrl6Q1PTCsVfNQJcLVnQn5bdnFA=', 'Content-Length': '0'} 2023-09-22T21:36:20.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb9b3b99aa61d624-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.143 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:20.144 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:20.144 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000037 2023-09-22T21:36:20.144 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000037 2023-09-22T21:36:20.144 DEBUG:boto:Method: GET 2023-09-22T21:36:20.144 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000037 2023-09-22T21:36:20.145 DEBUG:boto:Data: 2023-09-22T21:36:20.145 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.145 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.145 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.145 DEBUG:boto:Params: {} 2023-09-22T21:36:20.146 DEBUG:boto:Token: None 2023-09-22T21:36:20.146 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/ 2023-09-22T21:36:20.146 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0= 2023-09-22T21:36:20.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0=', 'Content-Length': '0'} 2023-09-22T21:36:20.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca2036596922e230-00650e08d4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.194 DEBUG:boto:b'ffzdqu-58obj000000000000000000371000falseobj000000000000000000372023-09-22T21:35:02.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:20.195 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:20.195 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:20.195 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000037 2023-09-22T21:36:20.196 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000037 2023-09-22T21:36:20.196 DEBUG:boto:Method: GET 2023-09-22T21:36:20.196 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000037 2023-09-22T21:36:20.196 DEBUG:boto:Data: 2023-09-22T21:36:20.196 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.196 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.197 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.197 DEBUG:boto:Params: {} 2023-09-22T21:36:20.197 DEBUG:boto:Token: None 2023-09-22T21:36:20.197 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/ 2023-09-22T21:36:20.197 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0= 2023-09-22T21:36:20.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0=', 'Content-Length': '0'} 2023-09-22T21:36:20.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000196ee4fb193afd4a-00650e08d4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.208 DEBUG:boto:b'ffzdqu-58obj000000000000000000371000falseobj000000000000000000372023-09-22T21:35:02.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:20.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-09-22T21:36:20.209 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.209 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.209 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.209 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.209 DEBUG:boto:Method: GET 2023-09-22T21:36:20.209 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.209 DEBUG:boto:Data: 2023-09-22T21:36:20.209 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.210 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.210 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.210 DEBUG:boto:Params: {} 2023-09-22T21:36:20.210 DEBUG:boto:Token: None 2023-09-22T21:36:20.210 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.210 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G1laG8bBH2bZ1EHKJKj8l+0N8VI= 2023-09-22T21:36:20.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G1laG8bBH2bZ1EHKJKj8l+0N8VI=', 'Content-Length': '0'} 2023-09-22T21:36:20.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062d0c26363d28737-00650e08d4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.214 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.214 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.214 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.214 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.214 DEBUG:boto:Method: GET 2023-09-22T21:36:20.215 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.215 DEBUG:boto:Data: 2023-09-22T21:36:20.215 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.215 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.215 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.215 DEBUG:boto:Params: {} 2023-09-22T21:36:20.215 DEBUG:boto:Token: None 2023-09-22T21:36:20.215 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.215 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G1laG8bBH2bZ1EHKJKj8l+0N8VI= 2023-09-22T21:36:20.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G1laG8bBH2bZ1EHKJKj8l+0N8VI=', 'Content-Length': '0'} 2023-09-22T21:36:20.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d7a5fd9246c6142-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.219 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.219 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.219 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.219 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.220 DEBUG:boto:Method: HEAD 2023-09-22T21:36:20.220 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.220 DEBUG:boto:Data: 2023-09-22T21:36:20.220 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.220 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.221 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.221 DEBUG:boto:Params: {} 2023-09-22T21:36:20.221 DEBUG:boto:Token: None 2023-09-22T21:36:20.221 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.221 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:p9FfYktuhsYGwL/Fb4FSp7fIqDU= 2023-09-22T21:36:20.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:p9FfYktuhsYGwL/Fb4FSp7fIqDU=', 'Content-Length': '0'} 2023-09-22T21:36:20.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001687fa01183d200c-00650e08d4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.225 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.225 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.225 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.225 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.225 DEBUG:boto:Method: HEAD 2023-09-22T21:36:20.226 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.226 DEBUG:boto:Data: 2023-09-22T21:36:20.226 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.226 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.226 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.227 DEBUG:boto:Params: {} 2023-09-22T21:36:20.227 DEBUG:boto:Token: None 2023-09-22T21:36:20.227 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.227 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:p9FfYktuhsYGwL/Fb4FSp7fIqDU= 2023-09-22T21:36:20.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:p9FfYktuhsYGwL/Fb4FSp7fIqDU=', 'Content-Length': '0'} 2023-09-22T21:36:20.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002db61e15680a64e6-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-09-22T21:36:20.231 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.231 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.231 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.232 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.232 DEBUG:boto:Method: GET 2023-09-22T21:36:20.232 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.232 DEBUG:boto:Data: 2023-09-22T21:36:20.232 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.232 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.233 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.233 DEBUG:boto:Params: {} 2023-09-22T21:36:20.233 DEBUG:boto:Token: None 2023-09-22T21:36:20.233 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.234 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G1laG8bBH2bZ1EHKJKj8l+0N8VI= 2023-09-22T21:36:20.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G1laG8bBH2bZ1EHKJKj8l+0N8VI=', 'Content-Length': '0'} 2023-09-22T21:36:20.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fabe3680d0e3979f-00650e08d4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.278 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.278 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.279 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.279 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.279 DEBUG:boto:Method: GET 2023-09-22T21:36:20.279 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.279 DEBUG:boto:Data: 2023-09-22T21:36:20.279 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.279 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.279 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.279 DEBUG:boto:Params: {} 2023-09-22T21:36:20.280 DEBUG:boto:Token: None 2023-09-22T21:36:20.280 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000039?versionId=null 2023-09-22T21:36:20.280 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G1laG8bBH2bZ1EHKJKj8l+0N8VI= 2023-09-22T21:36:20.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G1laG8bBH2bZ1EHKJKj8l+0N8VI=', 'Content-Length': '0'} 2023-09-22T21:36:20.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056fda791f2138607-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.283 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:20.283 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:20.284 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000039 2023-09-22T21:36:20.284 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000039 2023-09-22T21:36:20.284 DEBUG:boto:Method: GET 2023-09-22T21:36:20.284 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000039 2023-09-22T21:36:20.284 DEBUG:boto:Data: 2023-09-22T21:36:20.284 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.285 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.285 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.285 DEBUG:boto:Params: {} 2023-09-22T21:36:20.285 DEBUG:boto:Token: None 2023-09-22T21:36:20.285 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/ 2023-09-22T21:36:20.286 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0= 2023-09-22T21:36:20.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0=', 'Content-Length': '0'} 2023-09-22T21:36:20.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b90ca96af8642fc-00650e08d4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.334 DEBUG:boto:b'ffzdqu-58obj000000000000000000391000falseobj000000000000000000392023-09-22T21:35:12.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:20.335 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:20.335 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:20.335 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000039 2023-09-22T21:36:20.335 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000039 2023-09-22T21:36:20.335 DEBUG:boto:Method: GET 2023-09-22T21:36:20.335 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000039 2023-09-22T21:36:20.335 DEBUG:boto:Data: 2023-09-22T21:36:20.336 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.336 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.336 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.336 DEBUG:boto:Params: {} 2023-09-22T21:36:20.336 DEBUG:boto:Token: None 2023-09-22T21:36:20.336 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/ 2023-09-22T21:36:20.336 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0= 2023-09-22T21:36:20.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0=', 'Content-Length': '0'} 2023-09-22T21:36:20.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cdae8869c461456-00650e08d4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.348 DEBUG:boto:b'ffzdqu-58obj000000000000000000391000falseobj000000000000000000392023-09-22T21:35:12.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:20.348 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.349 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.349 DEBUG:boto:Method: HEAD 2023-09-22T21:36:20.349 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.349 DEBUG:boto:Data: 2023-09-22T21:36:20.349 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.349 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.350 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.350 DEBUG:boto:Params: {} 2023-09-22T21:36:20.350 DEBUG:boto:Token: None 2023-09-22T21:36:20.350 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.350 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jQnaXuxpzs2ZCo6TJT3LlYuQ1rE= 2023-09-22T21:36:20.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jQnaXuxpzs2ZCo6TJT3LlYuQ1rE=', 'Content-Length': '0'} 2023-09-22T21:36:20.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c690646dc9246c50-00650e08d4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.353 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.353 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.353 DEBUG:boto:Method: HEAD 2023-09-22T21:36:20.354 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.354 DEBUG:boto:Data: 2023-09-22T21:36:20.354 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.354 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.354 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.354 DEBUG:boto:Params: {} 2023-09-22T21:36:20.354 DEBUG:boto:Token: None 2023-09-22T21:36:20.354 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.354 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jQnaXuxpzs2ZCo6TJT3LlYuQ1rE= 2023-09-22T21:36:20.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jQnaXuxpzs2ZCo6TJT3LlYuQ1rE=', 'Content-Length': '0'} 2023-09-22T21:36:20.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2bd4d9de972bc24-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-09-22T21:36:20.357 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.358 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.358 DEBUG:boto:Method: GET 2023-09-22T21:36:20.358 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.358 DEBUG:boto:Data: 2023-09-22T21:36:20.358 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.358 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.358 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.358 DEBUG:boto:Params: {} 2023-09-22T21:36:20.358 DEBUG:boto:Token: None 2023-09-22T21:36:20.359 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.359 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UX5+r+IXtj7LBqRqKIMcF5HguCo= 2023-09-22T21:36:20.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UX5+r+IXtj7LBqRqKIMcF5HguCo=', 'Content-Length': '0'} 2023-09-22T21:36:20.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bda25b02658365ed-00650e08d4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.406 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.407 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.407 DEBUG:boto:Method: GET 2023-09-22T21:36:20.407 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.407 DEBUG:boto:Data: 2023-09-22T21:36:20.407 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.407 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.407 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.407 DEBUG:boto:Params: {} 2023-09-22T21:36:20.407 DEBUG:boto:Token: None 2023-09-22T21:36:20.408 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000039 2023-09-22T21:36:20.408 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UX5+r+IXtj7LBqRqKIMcF5HguCo= 2023-09-22T21:36:20.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UX5+r+IXtj7LBqRqKIMcF5HguCo=', 'Content-Length': '0'} 2023-09-22T21:36:20.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000453952be57ea7285-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.411 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:20.412 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:20.412 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000039 2023-09-22T21:36:20.412 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000039 2023-09-22T21:36:20.412 DEBUG:boto:Method: GET 2023-09-22T21:36:20.412 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000039 2023-09-22T21:36:20.412 DEBUG:boto:Data: 2023-09-22T21:36:20.412 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.412 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.412 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.412 DEBUG:boto:Params: {} 2023-09-22T21:36:20.413 DEBUG:boto:Token: None 2023-09-22T21:36:20.413 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/ 2023-09-22T21:36:20.413 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0= 2023-09-22T21:36:20.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0=', 'Content-Length': '0'} 2023-09-22T21:36:20.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000981ada88b599ae2a-00650e08d4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.462 DEBUG:boto:b'ffzdqu-58obj000000000000000000391000falseobj000000000000000000392023-09-22T21:35:12.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:20.463 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:20.463 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:20.463 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000039 2023-09-22T21:36:20.463 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000039 2023-09-22T21:36:20.463 DEBUG:boto:Method: GET 2023-09-22T21:36:20.463 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000039 2023-09-22T21:36:20.463 DEBUG:boto:Data: 2023-09-22T21:36:20.463 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.465 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.465 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.465 DEBUG:boto:Params: {} 2023-09-22T21:36:20.465 DEBUG:boto:Token: None 2023-09-22T21:36:20.465 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/ 2023-09-22T21:36:20.465 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0= 2023-09-22T21:36:20.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0=', 'Content-Length': '0'} 2023-09-22T21:36:20.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcc0654e037807f0-00650e08d4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.475 DEBUG:boto:b'ffzdqu-58obj000000000000000000391000falseobj000000000000000000392023-09-22T21:35:12.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:20.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-09-22T21:36:20.476 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.476 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.476 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.476 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.476 DEBUG:boto:Method: GET 2023-09-22T21:36:20.476 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.476 DEBUG:boto:Data: 2023-09-22T21:36:20.476 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.477 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.477 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.477 DEBUG:boto:Params: {} 2023-09-22T21:36:20.477 DEBUG:boto:Token: None 2023-09-22T21:36:20.477 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.477 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KumB7a+v6QD9N87z3/nvD5yuC84= 2023-09-22T21:36:20.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KumB7a+v6QD9N87z3/nvD5yuC84=', 'Content-Length': '0'} 2023-09-22T21:36:20.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a46665bd6babd573-00650e08d4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.481 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.481 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.481 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.481 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.482 DEBUG:boto:Method: GET 2023-09-22T21:36:20.482 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.482 DEBUG:boto:Data: 2023-09-22T21:36:20.482 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.482 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.482 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.483 DEBUG:boto:Params: {} 2023-09-22T21:36:20.483 DEBUG:boto:Token: None 2023-09-22T21:36:20.483 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.483 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KumB7a+v6QD9N87z3/nvD5yuC84= 2023-09-22T21:36:20.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KumB7a+v6QD9N87z3/nvD5yuC84=', 'Content-Length': '0'} 2023-09-22T21:36:20.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddf1da8eeac3ce7b-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.487 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.487 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.487 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.488 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.488 DEBUG:boto:Method: HEAD 2023-09-22T21:36:20.488 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.488 DEBUG:boto:Data: 2023-09-22T21:36:20.489 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.489 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.489 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.489 DEBUG:boto:Params: {} 2023-09-22T21:36:20.489 DEBUG:boto:Token: None 2023-09-22T21:36:20.490 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.490 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:v50SdbFCd3zQ+g5AeMQPu+OfTCs= 2023-09-22T21:36:20.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:v50SdbFCd3zQ+g5AeMQPu+OfTCs=', 'Content-Length': '0'} 2023-09-22T21:36:20.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000632d2359442470b2-00650e08d4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.493 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.493 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.493 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.493 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.493 DEBUG:boto:Method: HEAD 2023-09-22T21:36:20.494 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.494 DEBUG:boto:Data: 2023-09-22T21:36:20.494 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.494 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.494 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.494 DEBUG:boto:Params: {} 2023-09-22T21:36:20.494 DEBUG:boto:Token: None 2023-09-22T21:36:20.495 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.495 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:v50SdbFCd3zQ+g5AeMQPu+OfTCs= 2023-09-22T21:36:20.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:v50SdbFCd3zQ+g5AeMQPu+OfTCs=', 'Content-Length': '0'} 2023-09-22T21:36:20.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9e6c899555f54bf-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-09-22T21:36:20.498 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.498 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.498 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.498 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.498 DEBUG:boto:Method: GET 2023-09-22T21:36:20.498 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.498 DEBUG:boto:Data: 2023-09-22T21:36:20.499 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.499 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.499 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.499 DEBUG:boto:Params: {} 2023-09-22T21:36:20.499 DEBUG:boto:Token: None 2023-09-22T21:36:20.499 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.499 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KumB7a+v6QD9N87z3/nvD5yuC84= 2023-09-22T21:36:20.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KumB7a+v6QD9N87z3/nvD5yuC84=', 'Content-Length': '0'} 2023-09-22T21:36:20.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d9b61d091f8996f-00650e08d4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.542 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.542 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.543 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.543 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.543 DEBUG:boto:Method: GET 2023-09-22T21:36:20.543 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.543 DEBUG:boto:Data: 2023-09-22T21:36:20.543 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.543 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.543 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.543 DEBUG:boto:Params: {} 2023-09-22T21:36:20.543 DEBUG:boto:Token: None 2023-09-22T21:36:20.544 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000040?versionId=null 2023-09-22T21:36:20.544 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KumB7a+v6QD9N87z3/nvD5yuC84= 2023-09-22T21:36:20.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KumB7a+v6QD9N87z3/nvD5yuC84=', 'Content-Length': '0'} 2023-09-22T21:36:20.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000866c1e0e2ea42f78-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.548 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:20.548 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:20.548 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000040 2023-09-22T21:36:20.548 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000040 2023-09-22T21:36:20.548 DEBUG:boto:Method: GET 2023-09-22T21:36:20.548 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000040 2023-09-22T21:36:20.548 DEBUG:boto:Data: 2023-09-22T21:36:20.548 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.548 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.549 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.549 DEBUG:boto:Params: {} 2023-09-22T21:36:20.549 DEBUG:boto:Token: None 2023-09-22T21:36:20.549 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/ 2023-09-22T21:36:20.549 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0= 2023-09-22T21:36:20.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0=', 'Content-Length': '0'} 2023-09-22T21:36:20.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbcb2d0c80cbdd0d-00650e08d4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.598 DEBUG:boto:b'ffzdqu-58obj000000000000000000401000falseobj000000000000000000402023-09-22T21:35:12.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:20.599 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:20.599 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:20.599 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000040 2023-09-22T21:36:20.599 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000040 2023-09-22T21:36:20.600 DEBUG:boto:Method: GET 2023-09-22T21:36:20.600 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000040 2023-09-22T21:36:20.600 DEBUG:boto:Data: 2023-09-22T21:36:20.600 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.600 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.601 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.601 DEBUG:boto:Params: {} 2023-09-22T21:36:20.601 DEBUG:boto:Token: None 2023-09-22T21:36:20.601 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/ 2023-09-22T21:36:20.601 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0= 2023-09-22T21:36:20.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0=', 'Content-Length': '0'} 2023-09-22T21:36:20.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc1b5f4061ab5ee0-00650e08d4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.612 DEBUG:boto:b'ffzdqu-58obj000000000000000000401000falseobj000000000000000000402023-09-22T21:35:12.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:20.612 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.612 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.612 DEBUG:boto:Method: HEAD 2023-09-22T21:36:20.613 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.613 DEBUG:boto:Data: 2023-09-22T21:36:20.613 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.613 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.613 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.613 DEBUG:boto:Params: {} 2023-09-22T21:36:20.613 DEBUG:boto:Token: None 2023-09-22T21:36:20.613 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.613 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F8gD6g5rZ9kUEtgj4G4YMISDAAs= 2023-09-22T21:36:20.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F8gD6g5rZ9kUEtgj4G4YMISDAAs=', 'Content-Length': '0'} 2023-09-22T21:36:20.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000078b4d25f6bf085f-00650e08d4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.616 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.617 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.617 DEBUG:boto:Method: HEAD 2023-09-22T21:36:20.617 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.617 DEBUG:boto:Data: 2023-09-22T21:36:20.617 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.617 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.617 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.617 DEBUG:boto:Params: {} 2023-09-22T21:36:20.617 DEBUG:boto:Token: None 2023-09-22T21:36:20.618 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.618 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F8gD6g5rZ9kUEtgj4G4YMISDAAs= 2023-09-22T21:36:20.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F8gD6g5rZ9kUEtgj4G4YMISDAAs=', 'Content-Length': '0'} 2023-09-22T21:36:20.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e845b6e8d0c9c0a-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-09-22T21:36:20.621 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.621 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.621 DEBUG:boto:Method: GET 2023-09-22T21:36:20.622 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.622 DEBUG:boto:Data: 2023-09-22T21:36:20.622 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.622 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.622 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.622 DEBUG:boto:Params: {} 2023-09-22T21:36:20.622 DEBUG:boto:Token: None 2023-09-22T21:36:20.622 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.623 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OrJH/GVZprLCDciD7S2KcVSQc2Y= 2023-09-22T21:36:20.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OrJH/GVZprLCDciD7S2KcVSQc2Y=', 'Content-Length': '0'} 2023-09-22T21:36:20.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000051bbbf0ab30abe8-00650e08d4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.666 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.667 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.667 DEBUG:boto:Method: GET 2023-09-22T21:36:20.667 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.667 DEBUG:boto:Data: 2023-09-22T21:36:20.667 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.667 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.667 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.667 DEBUG:boto:Params: {} 2023-09-22T21:36:20.668 DEBUG:boto:Token: None 2023-09-22T21:36:20.668 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000040 2023-09-22T21:36:20.668 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OrJH/GVZprLCDciD7S2KcVSQc2Y= 2023-09-22T21:36:20.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OrJH/GVZprLCDciD7S2KcVSQc2Y=', 'Content-Length': '0'} 2023-09-22T21:36:20.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd6769652ad6374f-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.671 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:20.672 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:20.672 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000040 2023-09-22T21:36:20.672 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000040 2023-09-22T21:36:20.672 DEBUG:boto:Method: GET 2023-09-22T21:36:20.672 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000040 2023-09-22T21:36:20.672 DEBUG:boto:Data: 2023-09-22T21:36:20.672 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.672 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.672 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.672 DEBUG:boto:Params: {} 2023-09-22T21:36:20.673 DEBUG:boto:Token: None 2023-09-22T21:36:20.673 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/ 2023-09-22T21:36:20.673 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0= 2023-09-22T21:36:20.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0=', 'Content-Length': '0'} 2023-09-22T21:36:20.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ae5f26eb188a764-00650e08d4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.722 DEBUG:boto:b'ffzdqu-58obj000000000000000000401000falseobj000000000000000000402023-09-22T21:35:12.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:20.723 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:20.723 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:20.723 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000040 2023-09-22T21:36:20.723 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000040 2023-09-22T21:36:20.723 DEBUG:boto:Method: GET 2023-09-22T21:36:20.723 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000040 2023-09-22T21:36:20.724 DEBUG:boto:Data: 2023-09-22T21:36:20.724 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.724 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.724 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.724 DEBUG:boto:Params: {} 2023-09-22T21:36:20.724 DEBUG:boto:Token: None 2023-09-22T21:36:20.724 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/ 2023-09-22T21:36:20.725 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0= 2023-09-22T21:36:20.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0=', 'Content-Length': '0'} 2023-09-22T21:36:20.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057656a2228a1b5b7-00650e08d4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.735 DEBUG:boto:b'ffzdqu-58obj000000000000000000401000falseobj000000000000000000402023-09-22T21:35:12.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:20.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-09-22T21:36:20.735 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.736 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.736 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.736 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.736 DEBUG:boto:Method: GET 2023-09-22T21:36:20.736 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.736 DEBUG:boto:Data: 2023-09-22T21:36:20.736 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.737 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.737 DEBUG:boto:Params: {} 2023-09-22T21:36:20.737 DEBUG:boto:Token: None 2023-09-22T21:36:20.737 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.737 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FskrO5A+VtzQb+0oMdCqOeqYi2w= 2023-09-22T21:36:20.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FskrO5A+VtzQb+0oMdCqOeqYi2w=', 'Content-Length': '0'} 2023-09-22T21:36:20.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbfc51cc74ce18a0-00650e08d4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.741 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.741 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.741 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.741 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.741 DEBUG:boto:Method: GET 2023-09-22T21:36:20.741 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.741 DEBUG:boto:Data: 2023-09-22T21:36:20.742 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.742 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.742 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.742 DEBUG:boto:Params: {} 2023-09-22T21:36:20.742 DEBUG:boto:Token: None 2023-09-22T21:36:20.742 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.742 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FskrO5A+VtzQb+0oMdCqOeqYi2w= 2023-09-22T21:36:20.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FskrO5A+VtzQb+0oMdCqOeqYi2w=', 'Content-Length': '0'} 2023-09-22T21:36:20.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093d0df311c98fa53-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.746 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.746 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.746 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.746 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.746 DEBUG:boto:Method: HEAD 2023-09-22T21:36:20.746 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.747 DEBUG:boto:Data: 2023-09-22T21:36:20.747 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.747 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.747 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.747 DEBUG:boto:Params: {} 2023-09-22T21:36:20.747 DEBUG:boto:Token: None 2023-09-22T21:36:20.747 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.747 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rCnLBcw4L1ENsXGtWhxz/eoDUE8= 2023-09-22T21:36:20.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rCnLBcw4L1ENsXGtWhxz/eoDUE8=', 'Content-Length': '0'} 2023-09-22T21:36:20.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039e73bf690873771-00650e08d4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.750 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.751 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.751 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.751 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.751 DEBUG:boto:Method: HEAD 2023-09-22T21:36:20.751 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.751 DEBUG:boto:Data: 2023-09-22T21:36:20.751 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.751 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.752 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.752 DEBUG:boto:Params: {} 2023-09-22T21:36:20.752 DEBUG:boto:Token: None 2023-09-22T21:36:20.752 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.752 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rCnLBcw4L1ENsXGtWhxz/eoDUE8= 2023-09-22T21:36:20.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rCnLBcw4L1ENsXGtWhxz/eoDUE8=', 'Content-Length': '0'} 2023-09-22T21:36:20.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026eee7ab019c23a9-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-09-22T21:36:20.755 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.756 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.756 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.756 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.756 DEBUG:boto:Method: GET 2023-09-22T21:36:20.756 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.756 DEBUG:boto:Data: 2023-09-22T21:36:20.756 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.756 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.756 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.756 DEBUG:boto:Params: {} 2023-09-22T21:36:20.757 DEBUG:boto:Token: None 2023-09-22T21:36:20.757 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.757 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FskrO5A+VtzQb+0oMdCqOeqYi2w= 2023-09-22T21:36:20.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FskrO5A+VtzQb+0oMdCqOeqYi2w=', 'Content-Length': '0'} 2023-09-22T21:36:20.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a6c1eafd6f53b37-00650e08d4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.802 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.803 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.803 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.803 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.803 DEBUG:boto:Method: GET 2023-09-22T21:36:20.803 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.803 DEBUG:boto:Data: 2023-09-22T21:36:20.803 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.803 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.803 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.804 DEBUG:boto:Params: {} 2023-09-22T21:36:20.804 DEBUG:boto:Token: None 2023-09-22T21:36:20.804 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000042?versionId=null 2023-09-22T21:36:20.804 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FskrO5A+VtzQb+0oMdCqOeqYi2w= 2023-09-22T21:36:20.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FskrO5A+VtzQb+0oMdCqOeqYi2w=', 'Content-Length': '0'} 2023-09-22T21:36:20.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000619ec0bdf4865ba6-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.807 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:20.807 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:20.808 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000042 2023-09-22T21:36:20.808 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000042 2023-09-22T21:36:20.808 DEBUG:boto:Method: GET 2023-09-22T21:36:20.808 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000042 2023-09-22T21:36:20.808 DEBUG:boto:Data: 2023-09-22T21:36:20.808 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.808 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.808 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.808 DEBUG:boto:Params: {} 2023-09-22T21:36:20.809 DEBUG:boto:Token: None 2023-09-22T21:36:20.809 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/ 2023-09-22T21:36:20.809 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0= 2023-09-22T21:36:20.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0=', 'Content-Length': '0'} 2023-09-22T21:36:20.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051ee5deda8f909c9-00650e08d4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.813 DEBUG:boto:b'ffzdqu-58obj000000000000000000421000falseobj000000000000000000422023-09-22T21:35:03.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:20.814 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:20.814 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:20.814 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000042 2023-09-22T21:36:20.814 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000042 2023-09-22T21:36:20.814 DEBUG:boto:Method: GET 2023-09-22T21:36:20.814 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000042 2023-09-22T21:36:20.814 DEBUG:boto:Data: 2023-09-22T21:36:20.814 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.815 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.815 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.815 DEBUG:boto:Params: {} 2023-09-22T21:36:20.815 DEBUG:boto:Token: None 2023-09-22T21:36:20.815 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/ 2023-09-22T21:36:20.815 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0= 2023-09-22T21:36:20.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0=', 'Content-Length': '0'} 2023-09-22T21:36:20.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067d307af7ff0b9f8-00650e08d4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.830 DEBUG:boto:b'ffzdqu-58obj000000000000000000421000falseobj000000000000000000422023-09-22T21:35:03.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:20.831 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.831 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.831 DEBUG:boto:Method: HEAD 2023-09-22T21:36:20.831 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.831 DEBUG:boto:Data: 2023-09-22T21:36:20.831 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.831 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.832 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.832 DEBUG:boto:Params: {} 2023-09-22T21:36:20.832 DEBUG:boto:Token: None 2023-09-22T21:36:20.832 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.832 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Qr6J/bhf4PrSRPSDrAuWrIQXEzI= 2023-09-22T21:36:20.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Qr6J/bhf4PrSRPSDrAuWrIQXEzI=', 'Content-Length': '0'} 2023-09-22T21:36:20.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f64c793854489c3-00650e08d4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.835 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.835 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.836 DEBUG:boto:Method: HEAD 2023-09-22T21:36:20.836 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.836 DEBUG:boto:Data: 2023-09-22T21:36:20.836 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.836 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.836 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.836 DEBUG:boto:Params: {} 2023-09-22T21:36:20.836 DEBUG:boto:Token: None 2023-09-22T21:36:20.837 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.837 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Qr6J/bhf4PrSRPSDrAuWrIQXEzI= 2023-09-22T21:36:20.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Qr6J/bhf4PrSRPSDrAuWrIQXEzI=', 'Content-Length': '0'} 2023-09-22T21:36:20.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038fa1fb2f686807b-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-09-22T21:36:20.839 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.840 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.840 DEBUG:boto:Method: GET 2023-09-22T21:36:20.840 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.840 DEBUG:boto:Data: 2023-09-22T21:36:20.840 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.840 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.840 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.840 DEBUG:boto:Params: {} 2023-09-22T21:36:20.840 DEBUG:boto:Token: None 2023-09-22T21:36:20.841 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.841 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VCgnxsEIxG5mQHP8b7FNlllJmj4= 2023-09-22T21:36:20.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VCgnxsEIxG5mQHP8b7FNlllJmj4=', 'Content-Length': '0'} 2023-09-22T21:36:20.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd0c72b4444ef8e7-00650e08d4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.886 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.887 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.887 DEBUG:boto:Method: GET 2023-09-22T21:36:20.887 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.887 DEBUG:boto:Data: 2023-09-22T21:36:20.887 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.887 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.887 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.887 DEBUG:boto:Params: {} 2023-09-22T21:36:20.888 DEBUG:boto:Token: None 2023-09-22T21:36:20.888 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000042 2023-09-22T21:36:20.888 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VCgnxsEIxG5mQHP8b7FNlllJmj4= 2023-09-22T21:36:20.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VCgnxsEIxG5mQHP8b7FNlllJmj4=', 'Content-Length': '0'} 2023-09-22T21:36:20.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aee00dbbb3cdab43-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.891 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:20.892 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:20.892 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000042 2023-09-22T21:36:20.892 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000042 2023-09-22T21:36:20.892 DEBUG:boto:Method: GET 2023-09-22T21:36:20.892 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000042 2023-09-22T21:36:20.892 DEBUG:boto:Data: 2023-09-22T21:36:20.892 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.892 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.892 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.893 DEBUG:boto:Params: {} 2023-09-22T21:36:20.893 DEBUG:boto:Token: None 2023-09-22T21:36:20.893 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/ 2023-09-22T21:36:20.893 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0= 2023-09-22T21:36:20.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0=', 'Content-Length': '0'} 2023-09-22T21:36:20.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040634438689ba8d3-00650e08d4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.898 DEBUG:boto:b'ffzdqu-58obj000000000000000000421000falseobj000000000000000000422023-09-22T21:35:03.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:20.899 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:20.899 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:20.899 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000042 2023-09-22T21:36:20.899 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000042 2023-09-22T21:36:20.899 DEBUG:boto:Method: GET 2023-09-22T21:36:20.899 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000042 2023-09-22T21:36:20.899 DEBUG:boto:Data: 2023-09-22T21:36:20.899 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.900 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.900 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.900 DEBUG:boto:Params: {} 2023-09-22T21:36:20.900 DEBUG:boto:Token: None 2023-09-22T21:36:20.900 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/ 2023-09-22T21:36:20.901 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0= 2023-09-22T21:36:20.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0=', 'Content-Length': '0'} 2023-09-22T21:36:20.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035cd6e512cd6528b-00650e08d4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.911 DEBUG:boto:b'ffzdqu-58obj000000000000000000421000falseobj000000000000000000422023-09-22T21:35:03.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:20.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-09-22T21:36:20.911 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:20.912 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:20.912 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.912 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.912 DEBUG:boto:Method: GET 2023-09-22T21:36:20.912 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.912 DEBUG:boto:Data: 2023-09-22T21:36:20.912 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.912 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.912 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.912 DEBUG:boto:Params: {} 2023-09-22T21:36:20.913 DEBUG:boto:Token: None 2023-09-22T21:36:20.913 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.913 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CSzNdzF7ImvQ1cvvYbVEfRVsDdw= 2023-09-22T21:36:20.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CSzNdzF7ImvQ1cvvYbVEfRVsDdw=', 'Content-Length': '0'} 2023-09-22T21:36:20.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e65a2f64bc55def7-00650e08d4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.916 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:20.916 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:20.917 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.917 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.917 DEBUG:boto:Method: GET 2023-09-22T21:36:20.917 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.917 DEBUG:boto:Data: 2023-09-22T21:36:20.917 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.917 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.917 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.917 DEBUG:boto:Params: {} 2023-09-22T21:36:20.918 DEBUG:boto:Token: None 2023-09-22T21:36:20.918 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.918 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CSzNdzF7ImvQ1cvvYbVEfRVsDdw= 2023-09-22T21:36:20.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CSzNdzF7ImvQ1cvvYbVEfRVsDdw=', 'Content-Length': '0'} 2023-09-22T21:36:20.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007549a8b818a2adbc-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.921 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:20.921 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:20.922 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.922 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.922 DEBUG:boto:Method: HEAD 2023-09-22T21:36:20.922 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.922 DEBUG:boto:Data: 2023-09-22T21:36:20.922 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.922 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.922 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.922 DEBUG:boto:Params: {} 2023-09-22T21:36:20.923 DEBUG:boto:Token: None 2023-09-22T21:36:20.923 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.923 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kcEVaL7e+6QPbT/ykzLR93+AWVw= 2023-09-22T21:36:20.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kcEVaL7e+6QPbT/ykzLR93+AWVw=', 'Content-Length': '0'} 2023-09-22T21:36:20.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068fb18a45632bf54-00650e08d4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.926 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:20.926 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:20.926 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.926 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.927 DEBUG:boto:Method: HEAD 2023-09-22T21:36:20.927 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.927 DEBUG:boto:Data: 2023-09-22T21:36:20.927 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.927 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.927 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.927 DEBUG:boto:Params: {} 2023-09-22T21:36:20.927 DEBUG:boto:Token: None 2023-09-22T21:36:20.928 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.928 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kcEVaL7e+6QPbT/ykzLR93+AWVw= 2023-09-22T21:36:20.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kcEVaL7e+6QPbT/ykzLR93+AWVw=', 'Content-Length': '0'} 2023-09-22T21:36:20.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8e86ccec2dfd9a2-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-09-22T21:36:20.931 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:20.931 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:20.931 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.931 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.931 DEBUG:boto:Method: GET 2023-09-22T21:36:20.931 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.932 DEBUG:boto:Data: 2023-09-22T21:36:20.932 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.932 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.932 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.933 DEBUG:boto:Params: {} 2023-09-22T21:36:20.933 DEBUG:boto:Token: None 2023-09-22T21:36:20.933 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.933 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CSzNdzF7ImvQ1cvvYbVEfRVsDdw= 2023-09-22T21:36:20.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CSzNdzF7ImvQ1cvvYbVEfRVsDdw=', 'Content-Length': '0'} 2023-09-22T21:36:20.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a3bc905ad28977b-00650e08d4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.978 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:20.978 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:20.979 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.979 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.979 DEBUG:boto:Method: GET 2023-09-22T21:36:20.979 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.979 DEBUG:boto:Data: 2023-09-22T21:36:20.979 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.979 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.979 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.980 DEBUG:boto:Params: {} 2023-09-22T21:36:20.980 DEBUG:boto:Token: None 2023-09-22T21:36:20.980 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/obj00000000000000000045?versionId=null 2023-09-22T21:36:20.980 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CSzNdzF7ImvQ1cvvYbVEfRVsDdw= 2023-09-22T21:36:20.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CSzNdzF7ImvQ1cvvYbVEfRVsDdw=', 'Content-Length': '0'} 2023-09-22T21:36:20.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2a06ec63b1363e3-00650e08d4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.983 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:20.984 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:20.984 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000045 2023-09-22T21:36:20.984 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000045 2023-09-22T21:36:20.984 DEBUG:boto:Method: GET 2023-09-22T21:36:20.984 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000045 2023-09-22T21:36:20.984 DEBUG:boto:Data: 2023-09-22T21:36:20.985 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.985 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:20.985 DEBUG:boto:Port: 8001 2023-09-22T21:36:20.985 DEBUG:boto:Params: {} 2023-09-22T21:36:20.985 DEBUG:boto:Token: None 2023-09-22T21:36:20.985 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/ 2023-09-22T21:36:20.985 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0= 2023-09-22T21:36:20.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0=', 'Content-Length': '0'} 2023-09-22T21:36:20.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019aa2f8201819d8d-00650e08d4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:20.991 DEBUG:boto:b'ffzdqu-58obj000000000000000000451000falseobj000000000000000000452023-09-22T21:35:12.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:20.991 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:20.991 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:20.991 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000045 2023-09-22T21:36:20.991 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000045 2023-09-22T21:36:20.992 DEBUG:boto:Method: GET 2023-09-22T21:36:20.992 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000045 2023-09-22T21:36:20.992 DEBUG:boto:Data: 2023-09-22T21:36:20.992 DEBUG:boto:Headers: {} 2023-09-22T21:36:20.992 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:20.992 DEBUG:boto:Port: 8000 2023-09-22T21:36:20.992 DEBUG:boto:Params: {} 2023-09-22T21:36:20.992 DEBUG:boto:Token: None 2023-09-22T21:36:20.993 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:20 GMT /ffzdqu-58/ 2023-09-22T21:36:20.993 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0= 2023-09-22T21:36:20.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hryJgMtGTInvLZ+GWpWuThJRiP0=', 'Content-Length': '0'} 2023-09-22T21:36:21.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000350488d02552b5cd-00650e08d4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.046 DEBUG:boto:b'ffzdqu-58obj000000000000000000451000falseobj000000000000000000452023-09-22T21:35:12.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:21.047 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:21.047 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:21.047 DEBUG:boto:Method: HEAD 2023-09-22T21:36:21.047 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:21.047 DEBUG:boto:Data: 2023-09-22T21:36:21.047 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.047 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.047 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.048 DEBUG:boto:Params: {} 2023-09-22T21:36:21.048 DEBUG:boto:Token: None 2023-09-22T21:36:21.048 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:21.048 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J8uYg8oitscxEYb0ldySD94emO0= 2023-09-22T21:36:21.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J8uYg8oitscxEYb0ldySD94emO0=', 'Content-Length': '0'} 2023-09-22T21:36:21.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac28e2852fbe4739-00650e08d5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.051 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:21.051 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:21.052 DEBUG:boto:Method: HEAD 2023-09-22T21:36:21.052 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:21.052 DEBUG:boto:Data: 2023-09-22T21:36:21.052 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.052 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.052 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.052 DEBUG:boto:Params: {} 2023-09-22T21:36:21.052 DEBUG:boto:Token: None 2023-09-22T21:36:21.052 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:21.053 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J8uYg8oitscxEYb0ldySD94emO0= 2023-09-22T21:36:21.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J8uYg8oitscxEYb0ldySD94emO0=', 'Content-Length': '0'} 2023-09-22T21:36:21.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078b491a3cabac976-00650e08d5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-09-22T21:36:21.056 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:21.056 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:21.056 DEBUG:boto:Method: GET 2023-09-22T21:36:21.056 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:21.057 DEBUG:boto:Data: 2023-09-22T21:36:21.057 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.057 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.057 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.057 DEBUG:boto:Params: {} 2023-09-22T21:36:21.057 DEBUG:boto:Token: None 2023-09-22T21:36:21.057 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:21.057 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:v2U7u6krQUUL6jiqdqJiLx5vMGc= 2023-09-22T21:36:21.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:v2U7u6krQUUL6jiqdqJiLx5vMGc=', 'Content-Length': '0'} 2023-09-22T21:36:21.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000645a8a6f5707d5b-00650e08d5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.102 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:21.103 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:21.103 DEBUG:boto:Method: GET 2023-09-22T21:36:21.103 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:21.103 DEBUG:boto:Data: 2023-09-22T21:36:21.103 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.104 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.104 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.104 DEBUG:boto:Params: {} 2023-09-22T21:36:21.104 DEBUG:boto:Token: None 2023-09-22T21:36:21.104 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000045 2023-09-22T21:36:21.105 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:v2U7u6krQUUL6jiqdqJiLx5vMGc= 2023-09-22T21:36:21.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:v2U7u6krQUUL6jiqdqJiLx5vMGc=', 'Content-Length': '0'} 2023-09-22T21:36:21.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018a33f47a8927dbc-00650e08d5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.109 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:21.109 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:21.109 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000045 2023-09-22T21:36:21.109 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000045 2023-09-22T21:36:21.109 DEBUG:boto:Method: GET 2023-09-22T21:36:21.110 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000045 2023-09-22T21:36:21.110 DEBUG:boto:Data: 2023-09-22T21:36:21.110 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.110 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.110 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.111 DEBUG:boto:Params: {} 2023-09-22T21:36:21.111 DEBUG:boto:Token: None 2023-09-22T21:36:21.111 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/ 2023-09-22T21:36:21.111 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg= 2023-09-22T21:36:21.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg=', 'Content-Length': '0'} 2023-09-22T21:36:21.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011ca63f935a9a4df-00650e08d5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.158 DEBUG:boto:b'ffzdqu-58obj000000000000000000451000falseobj000000000000000000452023-09-22T21:35:12.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:21.159 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:21.159 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:21.159 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000045 2023-09-22T21:36:21.159 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000045 2023-09-22T21:36:21.160 DEBUG:boto:Method: GET 2023-09-22T21:36:21.160 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000045 2023-09-22T21:36:21.160 DEBUG:boto:Data: 2023-09-22T21:36:21.160 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.160 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.161 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.161 DEBUG:boto:Params: {} 2023-09-22T21:36:21.161 DEBUG:boto:Token: None 2023-09-22T21:36:21.161 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/ 2023-09-22T21:36:21.161 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg= 2023-09-22T21:36:21.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg=', 'Content-Length': '0'} 2023-09-22T21:36:21.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecb6570d3b4495b2-00650e08d5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.173 DEBUG:boto:b'ffzdqu-58obj000000000000000000451000falseobj000000000000000000452023-09-22T21:35:12.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:21.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-09-22T21:36:21.174 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.174 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.174 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.174 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.174 DEBUG:boto:Method: GET 2023-09-22T21:36:21.174 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.175 DEBUG:boto:Data: 2023-09-22T21:36:21.175 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.175 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.175 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.175 DEBUG:boto:Params: {} 2023-09-22T21:36:21.175 DEBUG:boto:Token: None 2023-09-22T21:36:21.175 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.175 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9JPMYDXmmEKg/E1aRmqIyvPhK60= 2023-09-22T21:36:21.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9JPMYDXmmEKg/E1aRmqIyvPhK60=', 'Content-Length': '0'} 2023-09-22T21:36:21.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017c22f683c6487ba-00650e08d5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.179 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.180 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.180 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.180 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.180 DEBUG:boto:Method: GET 2023-09-22T21:36:21.180 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.180 DEBUG:boto:Data: 2023-09-22T21:36:21.180 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.180 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.180 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.181 DEBUG:boto:Params: {} 2023-09-22T21:36:21.181 DEBUG:boto:Token: None 2023-09-22T21:36:21.181 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.181 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9JPMYDXmmEKg/E1aRmqIyvPhK60= 2023-09-22T21:36:21.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9JPMYDXmmEKg/E1aRmqIyvPhK60=', 'Content-Length': '0'} 2023-09-22T21:36:21.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030c3524d8b942837-00650e08d5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.184 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.184 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.185 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.185 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.185 DEBUG:boto:Method: HEAD 2023-09-22T21:36:21.185 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.185 DEBUG:boto:Data: 2023-09-22T21:36:21.185 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.185 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.185 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.185 DEBUG:boto:Params: {} 2023-09-22T21:36:21.186 DEBUG:boto:Token: None 2023-09-22T21:36:21.186 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.186 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:o2zPj6y0BvuGNVux+umYd1onr+k= 2023-09-22T21:36:21.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:o2zPj6y0BvuGNVux+umYd1onr+k=', 'Content-Length': '0'} 2023-09-22T21:36:21.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003acb590f0d5b2af0-00650e08d5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.189 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.189 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.189 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.189 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.189 DEBUG:boto:Method: HEAD 2023-09-22T21:36:21.190 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.190 DEBUG:boto:Data: 2023-09-22T21:36:21.190 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.190 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.190 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.190 DEBUG:boto:Params: {} 2023-09-22T21:36:21.190 DEBUG:boto:Token: None 2023-09-22T21:36:21.190 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.190 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:o2zPj6y0BvuGNVux+umYd1onr+k= 2023-09-22T21:36:21.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:o2zPj6y0BvuGNVux+umYd1onr+k=', 'Content-Length': '0'} 2023-09-22T21:36:21.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fec9ceeceb51313-00650e08d5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-09-22T21:36:21.193 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.194 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.194 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.194 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.194 DEBUG:boto:Method: GET 2023-09-22T21:36:21.194 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.194 DEBUG:boto:Data: 2023-09-22T21:36:21.194 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.194 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.194 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.194 DEBUG:boto:Params: {} 2023-09-22T21:36:21.195 DEBUG:boto:Token: None 2023-09-22T21:36:21.195 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.195 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9JPMYDXmmEKg/E1aRmqIyvPhK60= 2023-09-22T21:36:21.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9JPMYDXmmEKg/E1aRmqIyvPhK60=', 'Content-Length': '0'} 2023-09-22T21:36:21.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c86e21424ade447-00650e08d5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.238 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.239 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.239 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.239 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.239 DEBUG:boto:Method: GET 2023-09-22T21:36:21.239 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.239 DEBUG:boto:Data: 2023-09-22T21:36:21.239 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.239 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.240 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.240 DEBUG:boto:Params: {} 2023-09-22T21:36:21.240 DEBUG:boto:Token: None 2023-09-22T21:36:21.240 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000049?versionId=null 2023-09-22T21:36:21.240 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9JPMYDXmmEKg/E1aRmqIyvPhK60= 2023-09-22T21:36:21.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9JPMYDXmmEKg/E1aRmqIyvPhK60=', 'Content-Length': '0'} 2023-09-22T21:36:21.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024f9e0c38e3fac3c-00650e08d5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.243 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:21.244 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:21.244 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000049 2023-09-22T21:36:21.244 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000049 2023-09-22T21:36:21.244 DEBUG:boto:Method: GET 2023-09-22T21:36:21.244 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000049 2023-09-22T21:36:21.244 DEBUG:boto:Data: 2023-09-22T21:36:21.244 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.244 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.244 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.244 DEBUG:boto:Params: {} 2023-09-22T21:36:21.245 DEBUG:boto:Token: None 2023-09-22T21:36:21.245 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/ 2023-09-22T21:36:21.245 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg= 2023-09-22T21:36:21.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg=', 'Content-Length': '0'} 2023-09-22T21:36:21.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be5ec4a88609e2aa-00650e08d5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.250 DEBUG:boto:b'ffzdqu-58obj000000000000000000491000falseobj000000000000000000492023-09-22T21:35:02.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:21.250 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:21.251 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:21.251 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000049 2023-09-22T21:36:21.251 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000049 2023-09-22T21:36:21.251 DEBUG:boto:Method: GET 2023-09-22T21:36:21.251 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000049 2023-09-22T21:36:21.251 DEBUG:boto:Data: 2023-09-22T21:36:21.251 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.251 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.251 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.251 DEBUG:boto:Params: {} 2023-09-22T21:36:21.252 DEBUG:boto:Token: None 2023-09-22T21:36:21.252 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/ 2023-09-22T21:36:21.252 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg= 2023-09-22T21:36:21.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg=', 'Content-Length': '0'} 2023-09-22T21:36:21.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000556b064675274217-00650e08d5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.263 DEBUG:boto:b'ffzdqu-58obj000000000000000000491000falseobj000000000000000000492023-09-22T21:35:02.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:21.263 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.263 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.264 DEBUG:boto:Method: HEAD 2023-09-22T21:36:21.264 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.264 DEBUG:boto:Data: 2023-09-22T21:36:21.264 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.264 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.264 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.264 DEBUG:boto:Params: {} 2023-09-22T21:36:21.264 DEBUG:boto:Token: None 2023-09-22T21:36:21.265 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.265 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fJZjtT3qaz5tCIgOzqMU/2zuNRg= 2023-09-22T21:36:21.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fJZjtT3qaz5tCIgOzqMU/2zuNRg=', 'Content-Length': '0'} 2023-09-22T21:36:21.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017093f64e41d5b79-00650e08d5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.268 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.268 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.268 DEBUG:boto:Method: HEAD 2023-09-22T21:36:21.268 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.268 DEBUG:boto:Data: 2023-09-22T21:36:21.268 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.268 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.268 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.268 DEBUG:boto:Params: {} 2023-09-22T21:36:21.269 DEBUG:boto:Token: None 2023-09-22T21:36:21.269 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.269 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fJZjtT3qaz5tCIgOzqMU/2zuNRg= 2023-09-22T21:36:21.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fJZjtT3qaz5tCIgOzqMU/2zuNRg=', 'Content-Length': '0'} 2023-09-22T21:36:21.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002139e1d4394b68a8-00650e08d5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-09-22T21:36:21.272 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.272 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.273 DEBUG:boto:Method: GET 2023-09-22T21:36:21.273 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.273 DEBUG:boto:Data: 2023-09-22T21:36:21.273 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.273 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.273 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.273 DEBUG:boto:Params: {} 2023-09-22T21:36:21.273 DEBUG:boto:Token: None 2023-09-22T21:36:21.273 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.274 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/jw+Da4jBCSOgZlwSj+QG1MWmHk= 2023-09-22T21:36:21.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/jw+Da4jBCSOgZlwSj+QG1MWmHk=', 'Content-Length': '0'} 2023-09-22T21:36:21.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000347fa6fe7f5b3a36-00650e08d5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.318 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.319 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.319 DEBUG:boto:Method: GET 2023-09-22T21:36:21.319 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.319 DEBUG:boto:Data: 2023-09-22T21:36:21.319 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.319 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.319 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.319 DEBUG:boto:Params: {} 2023-09-22T21:36:21.320 DEBUG:boto:Token: None 2023-09-22T21:36:21.320 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000049 2023-09-22T21:36:21.320 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/jw+Da4jBCSOgZlwSj+QG1MWmHk= 2023-09-22T21:36:21.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/jw+Da4jBCSOgZlwSj+QG1MWmHk=', 'Content-Length': '0'} 2023-09-22T21:36:21.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025a48698126f59d8-00650e08d5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.323 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:21.323 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:21.323 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000049 2023-09-22T21:36:21.324 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000049 2023-09-22T21:36:21.324 DEBUG:boto:Method: GET 2023-09-22T21:36:21.324 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000049 2023-09-22T21:36:21.324 DEBUG:boto:Data: 2023-09-22T21:36:21.324 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.324 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.324 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.324 DEBUG:boto:Params: {} 2023-09-22T21:36:21.324 DEBUG:boto:Token: None 2023-09-22T21:36:21.324 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/ 2023-09-22T21:36:21.325 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg= 2023-09-22T21:36:21.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg=', 'Content-Length': '0'} 2023-09-22T21:36:21.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbb149d19a08ec7b-00650e08d5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.330 DEBUG:boto:b'ffzdqu-58obj000000000000000000491000falseobj000000000000000000492023-09-22T21:35:02.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:21.330 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:21.330 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:21.330 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000049 2023-09-22T21:36:21.331 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000049 2023-09-22T21:36:21.331 DEBUG:boto:Method: GET 2023-09-22T21:36:21.331 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000049 2023-09-22T21:36:21.331 DEBUG:boto:Data: 2023-09-22T21:36:21.331 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.331 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.331 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.331 DEBUG:boto:Params: {} 2023-09-22T21:36:21.331 DEBUG:boto:Token: None 2023-09-22T21:36:21.332 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/ 2023-09-22T21:36:21.332 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg= 2023-09-22T21:36:21.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg=', 'Content-Length': '0'} 2023-09-22T21:36:21.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e006d9744354940-00650e08d5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.386 DEBUG:boto:b'ffzdqu-58obj000000000000000000491000falseobj000000000000000000492023-09-22T21:35:02.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:21.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-09-22T21:36:21.387 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.387 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.387 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.387 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.387 DEBUG:boto:Method: GET 2023-09-22T21:36:21.388 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.388 DEBUG:boto:Data: 2023-09-22T21:36:21.388 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.388 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.388 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.388 DEBUG:boto:Params: {} 2023-09-22T21:36:21.388 DEBUG:boto:Token: None 2023-09-22T21:36:21.388 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.388 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4PmtDRo6unKOshX8HQxMzb5ZSCU= 2023-09-22T21:36:21.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4PmtDRo6unKOshX8HQxMzb5ZSCU=', 'Content-Length': '0'} 2023-09-22T21:36:21.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c707afd5773670db-00650e08d5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.392 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.392 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.392 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.392 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.393 DEBUG:boto:Method: GET 2023-09-22T21:36:21.393 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.393 DEBUG:boto:Data: 2023-09-22T21:36:21.393 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.393 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.393 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.394 DEBUG:boto:Params: {} 2023-09-22T21:36:21.394 DEBUG:boto:Token: None 2023-09-22T21:36:21.394 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.394 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4PmtDRo6unKOshX8HQxMzb5ZSCU= 2023-09-22T21:36:21.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4PmtDRo6unKOshX8HQxMzb5ZSCU=', 'Content-Length': '0'} 2023-09-22T21:36:21.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5f981f987a5aa97-00650e08d5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.398 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.398 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.398 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.399 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.399 DEBUG:boto:Method: HEAD 2023-09-22T21:36:21.399 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.399 DEBUG:boto:Data: 2023-09-22T21:36:21.399 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.400 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.400 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.400 DEBUG:boto:Params: {} 2023-09-22T21:36:21.400 DEBUG:boto:Token: None 2023-09-22T21:36:21.400 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.401 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:k9kPWcJVs16+AkbobLkXJ6yzaPA= 2023-09-22T21:36:21.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:k9kPWcJVs16+AkbobLkXJ6yzaPA=', 'Content-Length': '0'} 2023-09-22T21:36:21.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000978f5454a7984693-00650e08d5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.404 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.404 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.405 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.405 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.405 DEBUG:boto:Method: HEAD 2023-09-22T21:36:21.405 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.405 DEBUG:boto:Data: 2023-09-22T21:36:21.405 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.406 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.406 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.406 DEBUG:boto:Params: {} 2023-09-22T21:36:21.406 DEBUG:boto:Token: None 2023-09-22T21:36:21.406 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.407 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:k9kPWcJVs16+AkbobLkXJ6yzaPA= 2023-09-22T21:36:21.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:k9kPWcJVs16+AkbobLkXJ6yzaPA=', 'Content-Length': '0'} 2023-09-22T21:36:21.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b8fc26fc80e44c4-00650e08d5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-09-22T21:36:21.410 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.410 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.411 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.411 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.411 DEBUG:boto:Method: GET 2023-09-22T21:36:21.411 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.411 DEBUG:boto:Data: 2023-09-22T21:36:21.412 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.412 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.412 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.412 DEBUG:boto:Params: {} 2023-09-22T21:36:21.412 DEBUG:boto:Token: None 2023-09-22T21:36:21.413 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.413 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4PmtDRo6unKOshX8HQxMzb5ZSCU= 2023-09-22T21:36:21.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4PmtDRo6unKOshX8HQxMzb5ZSCU=', 'Content-Length': '0'} 2023-09-22T21:36:21.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bc1e4997e08680e-00650e08d5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.458 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.459 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.459 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.459 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.459 DEBUG:boto:Method: GET 2023-09-22T21:36:21.459 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.459 DEBUG:boto:Data: 2023-09-22T21:36:21.459 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.459 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.459 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.460 DEBUG:boto:Params: {} 2023-09-22T21:36:21.460 DEBUG:boto:Token: None 2023-09-22T21:36:21.460 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000050?versionId=null 2023-09-22T21:36:21.460 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4PmtDRo6unKOshX8HQxMzb5ZSCU= 2023-09-22T21:36:21.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4PmtDRo6unKOshX8HQxMzb5ZSCU=', 'Content-Length': '0'} 2023-09-22T21:36:21.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ab4d2727ff62c1c-00650e08d5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.463 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:21.464 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:21.464 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000050 2023-09-22T21:36:21.464 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000050 2023-09-22T21:36:21.464 DEBUG:boto:Method: GET 2023-09-22T21:36:21.464 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000050 2023-09-22T21:36:21.464 DEBUG:boto:Data: 2023-09-22T21:36:21.464 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.464 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.464 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.465 DEBUG:boto:Params: {} 2023-09-22T21:36:21.465 DEBUG:boto:Token: None 2023-09-22T21:36:21.465 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/ 2023-09-22T21:36:21.465 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg= 2023-09-22T21:36:21.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg=', 'Content-Length': '0'} 2023-09-22T21:36:21.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017be1e9d269fcc08-00650e08d5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.470 DEBUG:boto:b'ffzdqu-58obj000000000000000000501000falseobj000000000000000000502023-09-22T21:35:04.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:21.471 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:21.471 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:21.471 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000050 2023-09-22T21:36:21.471 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000050 2023-09-22T21:36:21.471 DEBUG:boto:Method: GET 2023-09-22T21:36:21.471 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000050 2023-09-22T21:36:21.471 DEBUG:boto:Data: 2023-09-22T21:36:21.471 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.472 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.472 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.472 DEBUG:boto:Params: {} 2023-09-22T21:36:21.472 DEBUG:boto:Token: None 2023-09-22T21:36:21.472 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/ 2023-09-22T21:36:21.472 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg= 2023-09-22T21:36:21.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg=', 'Content-Length': '0'} 2023-09-22T21:36:21.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eed9d360022500e2-00650e08d5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.482 DEBUG:boto:b'ffzdqu-58obj000000000000000000501000falseobj000000000000000000502023-09-22T21:35:04.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:21.482 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.485 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.485 DEBUG:boto:Method: HEAD 2023-09-22T21:36:21.485 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.485 DEBUG:boto:Data: 2023-09-22T21:36:21.485 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.485 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.485 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.486 DEBUG:boto:Params: {} 2023-09-22T21:36:21.486 DEBUG:boto:Token: None 2023-09-22T21:36:21.486 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.486 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3ED7s8o6QWYL5p3Psp7o3p/qFmM= 2023-09-22T21:36:21.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3ED7s8o6QWYL5p3Psp7o3p/qFmM=', 'Content-Length': '0'} 2023-09-22T21:36:21.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007802d018214e73ed-00650e08d5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.489 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.489 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.489 DEBUG:boto:Method: HEAD 2023-09-22T21:36:21.489 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.490 DEBUG:boto:Data: 2023-09-22T21:36:21.490 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.490 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.490 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.490 DEBUG:boto:Params: {} 2023-09-22T21:36:21.490 DEBUG:boto:Token: None 2023-09-22T21:36:21.490 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.490 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3ED7s8o6QWYL5p3Psp7o3p/qFmM= 2023-09-22T21:36:21.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3ED7s8o6QWYL5p3Psp7o3p/qFmM=', 'Content-Length': '0'} 2023-09-22T21:36:21.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b800013a30a41456-00650e08d5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-09-22T21:36:21.493 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.494 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.494 DEBUG:boto:Method: GET 2023-09-22T21:36:21.494 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.494 DEBUG:boto:Data: 2023-09-22T21:36:21.494 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.494 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.494 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.494 DEBUG:boto:Params: {} 2023-09-22T21:36:21.494 DEBUG:boto:Token: None 2023-09-22T21:36:21.495 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.495 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZpPcLd6dPrDcpdMAeDOVbAFdI4M= 2023-09-22T21:36:21.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZpPcLd6dPrDcpdMAeDOVbAFdI4M=', 'Content-Length': '0'} 2023-09-22T21:36:21.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1da7bee5695f420-00650e08d5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.538 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.539 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.539 DEBUG:boto:Method: GET 2023-09-22T21:36:21.539 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.539 DEBUG:boto:Data: 2023-09-22T21:36:21.539 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.539 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.539 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.539 DEBUG:boto:Params: {} 2023-09-22T21:36:21.539 DEBUG:boto:Token: None 2023-09-22T21:36:21.540 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000050 2023-09-22T21:36:21.540 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZpPcLd6dPrDcpdMAeDOVbAFdI4M= 2023-09-22T21:36:21.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZpPcLd6dPrDcpdMAeDOVbAFdI4M=', 'Content-Length': '0'} 2023-09-22T21:36:21.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e090f9c199b9eeea-00650e08d5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.543 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:21.543 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:21.543 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000050 2023-09-22T21:36:21.543 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000050 2023-09-22T21:36:21.544 DEBUG:boto:Method: GET 2023-09-22T21:36:21.544 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000050 2023-09-22T21:36:21.544 DEBUG:boto:Data: 2023-09-22T21:36:21.544 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.544 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.544 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.544 DEBUG:boto:Params: {} 2023-09-22T21:36:21.544 DEBUG:boto:Token: None 2023-09-22T21:36:21.545 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/ 2023-09-22T21:36:21.545 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg= 2023-09-22T21:36:21.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg=', 'Content-Length': '0'} 2023-09-22T21:36:21.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a6cb703051535af-00650e08d5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.590 DEBUG:boto:b'ffzdqu-58obj000000000000000000501000falseobj000000000000000000502023-09-22T21:35:04.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:21.591 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:21.591 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:21.591 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000050 2023-09-22T21:36:21.591 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000050 2023-09-22T21:36:21.591 DEBUG:boto:Method: GET 2023-09-22T21:36:21.591 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000050 2023-09-22T21:36:21.591 DEBUG:boto:Data: 2023-09-22T21:36:21.591 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.592 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.592 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.592 DEBUG:boto:Params: {} 2023-09-22T21:36:21.592 DEBUG:boto:Token: None 2023-09-22T21:36:21.592 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/ 2023-09-22T21:36:21.592 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg= 2023-09-22T21:36:21.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg=', 'Content-Length': '0'} 2023-09-22T21:36:21.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048291701e2a7fd31-00650e08d5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.602 DEBUG:boto:b'ffzdqu-58obj000000000000000000501000falseobj000000000000000000502023-09-22T21:35:04.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:21.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-09-22T21:36:21.603 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.603 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.603 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.603 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.603 DEBUG:boto:Method: GET 2023-09-22T21:36:21.603 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.603 DEBUG:boto:Data: 2023-09-22T21:36:21.604 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.604 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.604 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.604 DEBUG:boto:Params: {} 2023-09-22T21:36:21.604 DEBUG:boto:Token: None 2023-09-22T21:36:21.604 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.604 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:c+IrRUcO2WeG52W9i2yYoYniyCU= 2023-09-22T21:36:21.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:c+IrRUcO2WeG52W9i2yYoYniyCU=', 'Content-Length': '0'} 2023-09-22T21:36:21.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a156e33b9967d461-00650e08d5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.608 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.608 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.608 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.608 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.608 DEBUG:boto:Method: GET 2023-09-22T21:36:21.608 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.608 DEBUG:boto:Data: 2023-09-22T21:36:21.608 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.608 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.609 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.609 DEBUG:boto:Params: {} 2023-09-22T21:36:21.609 DEBUG:boto:Token: None 2023-09-22T21:36:21.609 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.609 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:c+IrRUcO2WeG52W9i2yYoYniyCU= 2023-09-22T21:36:21.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:c+IrRUcO2WeG52W9i2yYoYniyCU=', 'Content-Length': '0'} 2023-09-22T21:36:21.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051e2d7eb4011472a-00650e08d5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.612 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.612 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.612 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.612 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.613 DEBUG:boto:Method: HEAD 2023-09-22T21:36:21.613 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.613 DEBUG:boto:Data: 2023-09-22T21:36:21.613 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.613 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.613 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.613 DEBUG:boto:Params: {} 2023-09-22T21:36:21.613 DEBUG:boto:Token: None 2023-09-22T21:36:21.613 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.614 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8lxC0mi1Sseox11wNw8BRIzq6c= 2023-09-22T21:36:21.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8lxC0mi1Sseox11wNw8BRIzq6c=', 'Content-Length': '0'} 2023-09-22T21:36:21.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c3c09d422645c9a-00650e08d5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.616 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.617 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.617 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.617 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.617 DEBUG:boto:Method: HEAD 2023-09-22T21:36:21.617 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.617 DEBUG:boto:Data: 2023-09-22T21:36:21.617 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.617 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.617 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.618 DEBUG:boto:Params: {} 2023-09-22T21:36:21.618 DEBUG:boto:Token: None 2023-09-22T21:36:21.618 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.618 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8lxC0mi1Sseox11wNw8BRIzq6c= 2023-09-22T21:36:21.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8lxC0mi1Sseox11wNw8BRIzq6c=', 'Content-Length': '0'} 2023-09-22T21:36:21.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0881e400ed736f2-00650e08d5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-09-22T21:36:21.621 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.621 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.621 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.622 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.622 DEBUG:boto:Method: GET 2023-09-22T21:36:21.622 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.622 DEBUG:boto:Data: 2023-09-22T21:36:21.622 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.622 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.622 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.622 DEBUG:boto:Params: {} 2023-09-22T21:36:21.622 DEBUG:boto:Token: None 2023-09-22T21:36:21.623 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.623 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:c+IrRUcO2WeG52W9i2yYoYniyCU= 2023-09-22T21:36:21.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:c+IrRUcO2WeG52W9i2yYoYniyCU=', 'Content-Length': '0'} 2023-09-22T21:36:21.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000201d47e1443f8e46-00650e08d5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.666 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.667 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.667 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.667 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.667 DEBUG:boto:Method: GET 2023-09-22T21:36:21.667 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.667 DEBUG:boto:Data: 2023-09-22T21:36:21.667 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.667 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.668 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.668 DEBUG:boto:Params: {} 2023-09-22T21:36:21.668 DEBUG:boto:Token: None 2023-09-22T21:36:21.668 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000053?versionId=null 2023-09-22T21:36:21.668 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:c+IrRUcO2WeG52W9i2yYoYniyCU= 2023-09-22T21:36:21.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:c+IrRUcO2WeG52W9i2yYoYniyCU=', 'Content-Length': '0'} 2023-09-22T21:36:21.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b13289d3152592f0-00650e08d5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.672 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:21.672 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:21.672 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000053 2023-09-22T21:36:21.672 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000053 2023-09-22T21:36:21.672 DEBUG:boto:Method: GET 2023-09-22T21:36:21.672 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000053 2023-09-22T21:36:21.672 DEBUG:boto:Data: 2023-09-22T21:36:21.673 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.673 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.673 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.673 DEBUG:boto:Params: {} 2023-09-22T21:36:21.673 DEBUG:boto:Token: None 2023-09-22T21:36:21.673 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/ 2023-09-22T21:36:21.673 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg= 2023-09-22T21:36:21.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg=', 'Content-Length': '0'} 2023-09-22T21:36:21.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c32931fefe814962-00650e08d5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.722 DEBUG:boto:b'ffzdqu-58obj000000000000000000531000falseobj000000000000000000532023-09-22T21:35:12.185Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:21.723 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:21.723 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:21.723 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000053 2023-09-22T21:36:21.723 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000053 2023-09-22T21:36:21.723 DEBUG:boto:Method: GET 2023-09-22T21:36:21.723 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000053 2023-09-22T21:36:21.724 DEBUG:boto:Data: 2023-09-22T21:36:21.724 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.724 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.724 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.724 DEBUG:boto:Params: {} 2023-09-22T21:36:21.724 DEBUG:boto:Token: None 2023-09-22T21:36:21.724 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/ 2023-09-22T21:36:21.725 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg= 2023-09-22T21:36:21.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg=', 'Content-Length': '0'} 2023-09-22T21:36:21.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008271dbdff86f8018-00650e08d5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.737 DEBUG:boto:b'ffzdqu-58obj000000000000000000531000falseobj000000000000000000532023-09-22T21:35:12.185Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:21.737 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.737 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.737 DEBUG:boto:Method: HEAD 2023-09-22T21:36:21.738 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.738 DEBUG:boto:Data: 2023-09-22T21:36:21.738 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.738 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.738 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.738 DEBUG:boto:Params: {} 2023-09-22T21:36:21.738 DEBUG:boto:Token: None 2023-09-22T21:36:21.738 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.739 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UidQAh31ymhJ+Wx4L0NlYS/VUsI= 2023-09-22T21:36:21.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UidQAh31ymhJ+Wx4L0NlYS/VUsI=', 'Content-Length': '0'} 2023-09-22T21:36:21.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb7da5c2fe288777-00650e08d5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.742 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.742 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.742 DEBUG:boto:Method: HEAD 2023-09-22T21:36:21.742 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.742 DEBUG:boto:Data: 2023-09-22T21:36:21.742 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.742 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.742 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.742 DEBUG:boto:Params: {} 2023-09-22T21:36:21.743 DEBUG:boto:Token: None 2023-09-22T21:36:21.743 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.743 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UidQAh31ymhJ+Wx4L0NlYS/VUsI= 2023-09-22T21:36:21.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UidQAh31ymhJ+Wx4L0NlYS/VUsI=', 'Content-Length': '0'} 2023-09-22T21:36:21.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d22a9e110f37cbfa-00650e08d5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-09-22T21:36:21.746 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.746 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.746 DEBUG:boto:Method: GET 2023-09-22T21:36:21.746 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.746 DEBUG:boto:Data: 2023-09-22T21:36:21.747 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.747 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.747 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.747 DEBUG:boto:Params: {} 2023-09-22T21:36:21.747 DEBUG:boto:Token: None 2023-09-22T21:36:21.747 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.747 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qcoSVPGDtMqf+mX+umTt/ywe3LE= 2023-09-22T21:36:21.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qcoSVPGDtMqf+mX+umTt/ywe3LE=', 'Content-Length': '0'} 2023-09-22T21:36:21.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4907bdd0129a177-00650e08d5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.790 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.791 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.791 DEBUG:boto:Method: GET 2023-09-22T21:36:21.791 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.791 DEBUG:boto:Data: 2023-09-22T21:36:21.791 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.791 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.792 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.792 DEBUG:boto:Params: {} 2023-09-22T21:36:21.792 DEBUG:boto:Token: None 2023-09-22T21:36:21.792 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000053 2023-09-22T21:36:21.792 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qcoSVPGDtMqf+mX+umTt/ywe3LE= 2023-09-22T21:36:21.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qcoSVPGDtMqf+mX+umTt/ywe3LE=', 'Content-Length': '0'} 2023-09-22T21:36:21.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d50ae9b5aac2d9fa-00650e08d5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.795 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:21.796 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:21.796 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000053 2023-09-22T21:36:21.796 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000053 2023-09-22T21:36:21.796 DEBUG:boto:Method: GET 2023-09-22T21:36:21.796 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000053 2023-09-22T21:36:21.796 DEBUG:boto:Data: 2023-09-22T21:36:21.796 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.796 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.797 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.797 DEBUG:boto:Params: {} 2023-09-22T21:36:21.797 DEBUG:boto:Token: None 2023-09-22T21:36:21.797 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/ 2023-09-22T21:36:21.797 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg= 2023-09-22T21:36:21.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg=', 'Content-Length': '0'} 2023-09-22T21:36:21.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003da270e1f0b909d8-00650e08d5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.802 DEBUG:boto:b'ffzdqu-58obj000000000000000000531000falseobj000000000000000000532023-09-22T21:35:12.185Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:21.803 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:21.803 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:21.803 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000053 2023-09-22T21:36:21.803 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000053 2023-09-22T21:36:21.804 DEBUG:boto:Method: GET 2023-09-22T21:36:21.804 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000053 2023-09-22T21:36:21.804 DEBUG:boto:Data: 2023-09-22T21:36:21.804 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.804 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.804 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.804 DEBUG:boto:Params: {} 2023-09-22T21:36:21.805 DEBUG:boto:Token: None 2023-09-22T21:36:21.805 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/ 2023-09-22T21:36:21.805 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg= 2023-09-22T21:36:21.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg=', 'Content-Length': '0'} 2023-09-22T21:36:21.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcdab8ce83ceecb0-00650e08d5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.815 DEBUG:boto:b'ffzdqu-58obj000000000000000000531000falseobj000000000000000000532023-09-22T21:35:12.185Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:21.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-09-22T21:36:21.815 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:21.816 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:21.816 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.816 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.816 DEBUG:boto:Method: GET 2023-09-22T21:36:21.816 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.816 DEBUG:boto:Data: 2023-09-22T21:36:21.816 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.816 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.817 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.817 DEBUG:boto:Params: {} 2023-09-22T21:36:21.817 DEBUG:boto:Token: None 2023-09-22T21:36:21.817 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.817 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yZzr7kdAYHliM3mJ4L6Rn9W/HJI= 2023-09-22T21:36:21.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yZzr7kdAYHliM3mJ4L6Rn9W/HJI=', 'Content-Length': '0'} 2023-09-22T21:36:21.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010ddf54a74ae59fc-00650e08d5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.862 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:21.862 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:21.863 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.863 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.863 DEBUG:boto:Method: GET 2023-09-22T21:36:21.863 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.863 DEBUG:boto:Data: 2023-09-22T21:36:21.863 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.863 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.863 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.864 DEBUG:boto:Params: {} 2023-09-22T21:36:21.864 DEBUG:boto:Token: None 2023-09-22T21:36:21.864 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.864 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yZzr7kdAYHliM3mJ4L6Rn9W/HJI= 2023-09-22T21:36:21.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yZzr7kdAYHliM3mJ4L6Rn9W/HJI=', 'Content-Length': '0'} 2023-09-22T21:36:21.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bec195dab3a73329-00650e08d5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.868 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:21.868 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:21.868 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.868 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.868 DEBUG:boto:Method: HEAD 2023-09-22T21:36:21.868 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.869 DEBUG:boto:Data: 2023-09-22T21:36:21.869 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.869 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.869 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.869 DEBUG:boto:Params: {} 2023-09-22T21:36:21.869 DEBUG:boto:Token: None 2023-09-22T21:36:21.869 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.870 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pAA2KDC5XkImqkDOQLOJLjR/DEw= 2023-09-22T21:36:21.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pAA2KDC5XkImqkDOQLOJLjR/DEw=', 'Content-Length': '0'} 2023-09-22T21:36:21.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071cc0f33d0c5be00-00650e08d5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.873 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:21.873 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:21.873 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.874 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.874 DEBUG:boto:Method: HEAD 2023-09-22T21:36:21.874 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.874 DEBUG:boto:Data: 2023-09-22T21:36:21.874 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.874 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.874 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.874 DEBUG:boto:Params: {} 2023-09-22T21:36:21.875 DEBUG:boto:Token: None 2023-09-22T21:36:21.875 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.875 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pAA2KDC5XkImqkDOQLOJLjR/DEw= 2023-09-22T21:36:21.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pAA2KDC5XkImqkDOQLOJLjR/DEw=', 'Content-Length': '0'} 2023-09-22T21:36:21.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6e27f5224a382aa-00650e08d5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-09-22T21:36:21.878 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:21.878 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:21.878 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.879 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.879 DEBUG:boto:Method: GET 2023-09-22T21:36:21.879 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.879 DEBUG:boto:Data: 2023-09-22T21:36:21.879 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.879 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.879 DEBUG:boto:Params: {} 2023-09-22T21:36:21.880 DEBUG:boto:Token: None 2023-09-22T21:36:21.880 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.880 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yZzr7kdAYHliM3mJ4L6Rn9W/HJI= 2023-09-22T21:36:21.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yZzr7kdAYHliM3mJ4L6Rn9W/HJI=', 'Content-Length': '0'} 2023-09-22T21:36:21.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc8a0244bf111797-00650e08d5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.926 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:21.927 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:21.927 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.927 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.927 DEBUG:boto:Method: GET 2023-09-22T21:36:21.927 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.927 DEBUG:boto:Data: 2023-09-22T21:36:21.928 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.928 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.928 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.928 DEBUG:boto:Params: {} 2023-09-22T21:36:21.928 DEBUG:boto:Token: None 2023-09-22T21:36:21.928 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000055?versionId=null 2023-09-22T21:36:21.928 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yZzr7kdAYHliM3mJ4L6Rn9W/HJI= 2023-09-22T21:36:21.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yZzr7kdAYHliM3mJ4L6Rn9W/HJI=', 'Content-Length': '0'} 2023-09-22T21:36:21.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5299793e699f7eb-00650e08d5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.932 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:21.932 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:21.932 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000055 2023-09-22T21:36:21.932 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000055 2023-09-22T21:36:21.932 DEBUG:boto:Method: GET 2023-09-22T21:36:21.932 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000055 2023-09-22T21:36:21.933 DEBUG:boto:Data: 2023-09-22T21:36:21.933 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.933 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.933 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.933 DEBUG:boto:Params: {} 2023-09-22T21:36:21.933 DEBUG:boto:Token: None 2023-09-22T21:36:21.933 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/ 2023-09-22T21:36:21.933 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg= 2023-09-22T21:36:21.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg=', 'Content-Length': '0'} 2023-09-22T21:36:21.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ca8c61a662994ad-00650e08d5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.982 DEBUG:boto:b'ffzdqu-58obj000000000000000000551000falseobj000000000000000000552023-09-22T21:35:01.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:21.983 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:21.983 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:21.983 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000055 2023-09-22T21:36:21.983 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000055 2023-09-22T21:36:21.983 DEBUG:boto:Method: GET 2023-09-22T21:36:21.984 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000055 2023-09-22T21:36:21.984 DEBUG:boto:Data: 2023-09-22T21:36:21.984 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.984 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:21.984 DEBUG:boto:Port: 8000 2023-09-22T21:36:21.984 DEBUG:boto:Params: {} 2023-09-22T21:36:21.984 DEBUG:boto:Token: None 2023-09-22T21:36:21.984 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/ 2023-09-22T21:36:21.985 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg= 2023-09-22T21:36:21.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:41/nfaIBU6Jf8Wls4LwT6AHVKTg=', 'Content-Length': '0'} 2023-09-22T21:36:21.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030f37ce53f4c4059-00650e08d5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:21.996 DEBUG:boto:b'ffzdqu-58obj000000000000000000551000falseobj000000000000000000552023-09-22T21:35:01.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:21.996 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:21.996 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:21.997 DEBUG:boto:Method: HEAD 2023-09-22T21:36:21.997 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:21.997 DEBUG:boto:Data: 2023-09-22T21:36:21.997 DEBUG:boto:Headers: {} 2023-09-22T21:36:21.997 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:21.997 DEBUG:boto:Port: 8001 2023-09-22T21:36:21.997 DEBUG:boto:Params: {} 2023-09-22T21:36:21.997 DEBUG:boto:Token: None 2023-09-22T21:36:21.998 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:21 GMT /ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:21.998 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9hdBiL9bwE8clugdUeb2aub4Iqg= 2023-09-22T21:36:21.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9hdBiL9bwE8clugdUeb2aub4Iqg=', 'Content-Length': '0'} 2023-09-22T21:36:22.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f2ca3f69ad09517-00650e08d5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.001 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:22.001 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:22.001 DEBUG:boto:Method: HEAD 2023-09-22T21:36:22.002 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:22.002 DEBUG:boto:Data: 2023-09-22T21:36:22.002 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.002 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.002 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.002 DEBUG:boto:Params: {} 2023-09-22T21:36:22.002 DEBUG:boto:Token: None 2023-09-22T21:36:22.002 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:22.003 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xTztPppbiTUuQ7QHf6+RzHNJ66A= 2023-09-22T21:36:22.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xTztPppbiTUuQ7QHf6+RzHNJ66A=', 'Content-Length': '0'} 2023-09-22T21:36:22.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000097983ca6b910367-00650e08d5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-09-22T21:36:22.006 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:22.006 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:22.006 DEBUG:boto:Method: GET 2023-09-22T21:36:22.007 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:22.007 DEBUG:boto:Data: 2023-09-22T21:36:22.007 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.007 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.007 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.007 DEBUG:boto:Params: {} 2023-09-22T21:36:22.007 DEBUG:boto:Token: None 2023-09-22T21:36:22.007 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:22.008 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rq6RQyNRE3MNc97Fbtz6Lf3iWBA= 2023-09-22T21:36:22.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rq6RQyNRE3MNc97Fbtz6Lf3iWBA=', 'Content-Length': '0'} 2023-09-22T21:36:22.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006abfdcaf89654ef8-00650e08d6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.054 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:22.055 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:22.055 DEBUG:boto:Method: GET 2023-09-22T21:36:22.055 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:22.055 DEBUG:boto:Data: 2023-09-22T21:36:22.055 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.055 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.055 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.055 DEBUG:boto:Params: {} 2023-09-22T21:36:22.056 DEBUG:boto:Token: None 2023-09-22T21:36:22.056 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000055 2023-09-22T21:36:22.056 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rq6RQyNRE3MNc97Fbtz6Lf3iWBA= 2023-09-22T21:36:22.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rq6RQyNRE3MNc97Fbtz6Lf3iWBA=', 'Content-Length': '0'} 2023-09-22T21:36:22.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a45cb83c7516ecc5-00650e08d6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.059 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:22.059 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:22.059 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000055 2023-09-22T21:36:22.060 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000055 2023-09-22T21:36:22.060 DEBUG:boto:Method: GET 2023-09-22T21:36:22.060 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000055 2023-09-22T21:36:22.060 DEBUG:boto:Data: 2023-09-22T21:36:22.060 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.060 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.060 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.060 DEBUG:boto:Params: {} 2023-09-22T21:36:22.060 DEBUG:boto:Token: None 2023-09-22T21:36:22.061 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/ 2023-09-22T21:36:22.061 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw= 2023-09-22T21:36:22.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw=', 'Content-Length': '0'} 2023-09-22T21:36:22.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5e8cae5b52226dc-00650e08d6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.065 DEBUG:boto:b'ffzdqu-58obj000000000000000000551000falseobj000000000000000000552023-09-22T21:35:01.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:22.066 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:22.066 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:22.066 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000055 2023-09-22T21:36:22.066 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000055 2023-09-22T21:36:22.066 DEBUG:boto:Method: GET 2023-09-22T21:36:22.066 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000055 2023-09-22T21:36:22.066 DEBUG:boto:Data: 2023-09-22T21:36:22.067 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.067 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.067 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.067 DEBUG:boto:Params: {} 2023-09-22T21:36:22.067 DEBUG:boto:Token: None 2023-09-22T21:36:22.067 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/ 2023-09-22T21:36:22.067 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw= 2023-09-22T21:36:22.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw=', 'Content-Length': '0'} 2023-09-22T21:36:22.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c63cfdf383985eb7-00650e08d6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.078 DEBUG:boto:b'ffzdqu-58obj000000000000000000551000falseobj000000000000000000552023-09-22T21:35:01.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:22.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-09-22T21:36:22.078 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.078 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.078 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.078 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.079 DEBUG:boto:Method: GET 2023-09-22T21:36:22.079 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.079 DEBUG:boto:Data: 2023-09-22T21:36:22.079 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.079 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.079 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.079 DEBUG:boto:Params: {} 2023-09-22T21:36:22.079 DEBUG:boto:Token: None 2023-09-22T21:36:22.080 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.080 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+s5GnfbMQfM9dCQmy+togMo8VfY= 2023-09-22T21:36:22.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+s5GnfbMQfM9dCQmy+togMo8VfY=', 'Content-Length': '0'} 2023-09-22T21:36:22.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da17d753be17aa7a-00650e08d6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.083 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.083 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.083 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.083 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.083 DEBUG:boto:Method: GET 2023-09-22T21:36:22.084 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.084 DEBUG:boto:Data: 2023-09-22T21:36:22.084 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.084 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.084 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.084 DEBUG:boto:Params: {} 2023-09-22T21:36:22.084 DEBUG:boto:Token: None 2023-09-22T21:36:22.084 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.085 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+s5GnfbMQfM9dCQmy+togMo8VfY= 2023-09-22T21:36:22.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+s5GnfbMQfM9dCQmy+togMo8VfY=', 'Content-Length': '0'} 2023-09-22T21:36:22.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008403a827081ce33a-00650e08d6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.088 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.088 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.088 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.088 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.088 DEBUG:boto:Method: HEAD 2023-09-22T21:36:22.088 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.088 DEBUG:boto:Data: 2023-09-22T21:36:22.089 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.089 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.089 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.089 DEBUG:boto:Params: {} 2023-09-22T21:36:22.089 DEBUG:boto:Token: None 2023-09-22T21:36:22.089 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.089 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:THb1ozuaGxfTYMKh4DHI5SxOJY0= 2023-09-22T21:36:22.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:THb1ozuaGxfTYMKh4DHI5SxOJY0=', 'Content-Length': '0'} 2023-09-22T21:36:22.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000402281ed32771de3-00650e08d6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.092 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.092 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.092 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.092 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.093 DEBUG:boto:Method: HEAD 2023-09-22T21:36:22.093 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.093 DEBUG:boto:Data: 2023-09-22T21:36:22.093 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.093 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.093 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.093 DEBUG:boto:Params: {} 2023-09-22T21:36:22.093 DEBUG:boto:Token: None 2023-09-22T21:36:22.093 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.094 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:THb1ozuaGxfTYMKh4DHI5SxOJY0= 2023-09-22T21:36:22.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:THb1ozuaGxfTYMKh4DHI5SxOJY0=', 'Content-Length': '0'} 2023-09-22T21:36:22.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079a16d1cc3812663-00650e08d6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-09-22T21:36:22.097 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.097 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.097 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.097 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.097 DEBUG:boto:Method: GET 2023-09-22T21:36:22.097 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.097 DEBUG:boto:Data: 2023-09-22T21:36:22.097 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.097 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.098 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.098 DEBUG:boto:Params: {} 2023-09-22T21:36:22.098 DEBUG:boto:Token: None 2023-09-22T21:36:22.098 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.098 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+s5GnfbMQfM9dCQmy+togMo8VfY= 2023-09-22T21:36:22.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+s5GnfbMQfM9dCQmy+togMo8VfY=', 'Content-Length': '0'} 2023-09-22T21:36:22.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000605870b89b40a9e6-00650e08d6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.146 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.147 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.147 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.147 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.147 DEBUG:boto:Method: GET 2023-09-22T21:36:22.147 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.147 DEBUG:boto:Data: 2023-09-22T21:36:22.147 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.147 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.147 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.148 DEBUG:boto:Params: {} 2023-09-22T21:36:22.148 DEBUG:boto:Token: None 2023-09-22T21:36:22.148 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000057?versionId=null 2023-09-22T21:36:22.148 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+s5GnfbMQfM9dCQmy+togMo8VfY= 2023-09-22T21:36:22.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+s5GnfbMQfM9dCQmy+togMo8VfY=', 'Content-Length': '0'} 2023-09-22T21:36:22.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb4d4044cc3b0252-00650e08d6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.151 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:22.152 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:22.152 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000057 2023-09-22T21:36:22.152 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000057 2023-09-22T21:36:22.152 DEBUG:boto:Method: GET 2023-09-22T21:36:22.152 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000057 2023-09-22T21:36:22.152 DEBUG:boto:Data: 2023-09-22T21:36:22.152 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.152 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.152 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.152 DEBUG:boto:Params: {} 2023-09-22T21:36:22.153 DEBUG:boto:Token: None 2023-09-22T21:36:22.153 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/ 2023-09-22T21:36:22.153 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw= 2023-09-22T21:36:22.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw=', 'Content-Length': '0'} 2023-09-22T21:36:22.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089d8a80d36a193b1-00650e08d6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.158 DEBUG:boto:b'ffzdqu-58obj000000000000000000571000falseobj000000000000000000572023-09-22T21:35:05.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:22.158 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:22.158 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:22.158 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000057 2023-09-22T21:36:22.159 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000057 2023-09-22T21:36:22.159 DEBUG:boto:Method: GET 2023-09-22T21:36:22.159 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000057 2023-09-22T21:36:22.159 DEBUG:boto:Data: 2023-09-22T21:36:22.159 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.159 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.159 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.159 DEBUG:boto:Params: {} 2023-09-22T21:36:22.159 DEBUG:boto:Token: None 2023-09-22T21:36:22.160 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/ 2023-09-22T21:36:22.160 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw= 2023-09-22T21:36:22.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw=', 'Content-Length': '0'} 2023-09-22T21:36:22.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bac8d7b0c489da7-00650e08d6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.170 DEBUG:boto:b'ffzdqu-58obj000000000000000000571000falseobj000000000000000000572023-09-22T21:35:05.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:22.170 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.170 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.170 DEBUG:boto:Method: HEAD 2023-09-22T21:36:22.170 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.170 DEBUG:boto:Data: 2023-09-22T21:36:22.170 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.171 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.171 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.171 DEBUG:boto:Params: {} 2023-09-22T21:36:22.171 DEBUG:boto:Token: None 2023-09-22T21:36:22.171 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.171 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dfw9WAbf/av8868bK1EEopv306o= 2023-09-22T21:36:22.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dfw9WAbf/av8868bK1EEopv306o=', 'Content-Length': '0'} 2023-09-22T21:36:22.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a40eb13af1040e5-00650e08d6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.174 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.174 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.174 DEBUG:boto:Method: HEAD 2023-09-22T21:36:22.175 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.175 DEBUG:boto:Data: 2023-09-22T21:36:22.175 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.175 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.175 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.175 DEBUG:boto:Params: {} 2023-09-22T21:36:22.175 DEBUG:boto:Token: None 2023-09-22T21:36:22.175 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.175 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dfw9WAbf/av8868bK1EEopv306o= 2023-09-22T21:36:22.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dfw9WAbf/av8868bK1EEopv306o=', 'Content-Length': '0'} 2023-09-22T21:36:22.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfc64d2128cde635-00650e08d6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-09-22T21:36:22.178 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.179 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.179 DEBUG:boto:Method: GET 2023-09-22T21:36:22.179 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.179 DEBUG:boto:Data: 2023-09-22T21:36:22.179 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.179 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.179 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.179 DEBUG:boto:Params: {} 2023-09-22T21:36:22.179 DEBUG:boto:Token: None 2023-09-22T21:36:22.180 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.180 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mrXMILQxbJV70YWrnk8v0eYtOFo= 2023-09-22T21:36:22.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mrXMILQxbJV70YWrnk8v0eYtOFo=', 'Content-Length': '0'} 2023-09-22T21:36:22.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a39ebadd859df80-00650e08d6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.230 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.230 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.231 DEBUG:boto:Method: GET 2023-09-22T21:36:22.231 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.231 DEBUG:boto:Data: 2023-09-22T21:36:22.231 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.231 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.231 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.231 DEBUG:boto:Params: {} 2023-09-22T21:36:22.231 DEBUG:boto:Token: None 2023-09-22T21:36:22.232 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000057 2023-09-22T21:36:22.232 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mrXMILQxbJV70YWrnk8v0eYtOFo= 2023-09-22T21:36:22.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mrXMILQxbJV70YWrnk8v0eYtOFo=', 'Content-Length': '0'} 2023-09-22T21:36:22.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b1a2427d60f9b12-00650e08d6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.235 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:22.235 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:22.235 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000057 2023-09-22T21:36:22.235 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000057 2023-09-22T21:36:22.235 DEBUG:boto:Method: GET 2023-09-22T21:36:22.236 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000057 2023-09-22T21:36:22.236 DEBUG:boto:Data: 2023-09-22T21:36:22.236 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.236 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.236 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.236 DEBUG:boto:Params: {} 2023-09-22T21:36:22.236 DEBUG:boto:Token: None 2023-09-22T21:36:22.236 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/ 2023-09-22T21:36:22.236 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw= 2023-09-22T21:36:22.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw=', 'Content-Length': '0'} 2023-09-22T21:36:22.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fa24bbeb1ae0ae7-00650e08d6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.282 DEBUG:boto:b'ffzdqu-58obj000000000000000000571000falseobj000000000000000000572023-09-22T21:35:05.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:22.283 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:22.283 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:22.283 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000057 2023-09-22T21:36:22.283 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000057 2023-09-22T21:36:22.283 DEBUG:boto:Method: GET 2023-09-22T21:36:22.283 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000057 2023-09-22T21:36:22.284 DEBUG:boto:Data: 2023-09-22T21:36:22.284 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.284 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.284 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.284 DEBUG:boto:Params: {} 2023-09-22T21:36:22.284 DEBUG:boto:Token: None 2023-09-22T21:36:22.284 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/ 2023-09-22T21:36:22.284 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw= 2023-09-22T21:36:22.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw=', 'Content-Length': '0'} 2023-09-22T21:36:22.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021ce6a42e60b64b1-00650e08d6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.294 DEBUG:boto:b'ffzdqu-58obj000000000000000000571000falseobj000000000000000000572023-09-22T21:35:05.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:22.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-09-22T21:36:22.295 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.295 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.295 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.295 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.295 DEBUG:boto:Method: GET 2023-09-22T21:36:22.295 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.296 DEBUG:boto:Data: 2023-09-22T21:36:22.296 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.296 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.296 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.296 DEBUG:boto:Params: {} 2023-09-22T21:36:22.296 DEBUG:boto:Token: None 2023-09-22T21:36:22.296 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.296 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hmPNE3SxuLJY70RcQlVjCEqciLc= 2023-09-22T21:36:22.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hmPNE3SxuLJY70RcQlVjCEqciLc=', 'Content-Length': '0'} 2023-09-22T21:36:22.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000313e1f5b0601571c-00650e08d6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.300 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.300 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.300 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.300 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.300 DEBUG:boto:Method: GET 2023-09-22T21:36:22.300 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.300 DEBUG:boto:Data: 2023-09-22T21:36:22.301 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.301 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.301 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.301 DEBUG:boto:Params: {} 2023-09-22T21:36:22.301 DEBUG:boto:Token: None 2023-09-22T21:36:22.301 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.301 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hmPNE3SxuLJY70RcQlVjCEqciLc= 2023-09-22T21:36:22.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hmPNE3SxuLJY70RcQlVjCEqciLc=', 'Content-Length': '0'} 2023-09-22T21:36:22.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d203a9c394f184eb-00650e08d6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.350 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.350 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.351 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.351 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.351 DEBUG:boto:Method: HEAD 2023-09-22T21:36:22.351 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.351 DEBUG:boto:Data: 2023-09-22T21:36:22.351 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.351 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.351 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.351 DEBUG:boto:Params: {} 2023-09-22T21:36:22.352 DEBUG:boto:Token: None 2023-09-22T21:36:22.352 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.352 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3mxyu9UEf4zLsqO3DHO1189AFNQ= 2023-09-22T21:36:22.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3mxyu9UEf4zLsqO3DHO1189AFNQ=', 'Content-Length': '0'} 2023-09-22T21:36:22.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a933577c2eba2a9-00650e08d6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.355 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.355 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.355 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.355 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.355 DEBUG:boto:Method: HEAD 2023-09-22T21:36:22.355 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.356 DEBUG:boto:Data: 2023-09-22T21:36:22.356 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.356 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.356 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.356 DEBUG:boto:Params: {} 2023-09-22T21:36:22.356 DEBUG:boto:Token: None 2023-09-22T21:36:22.356 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.356 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3mxyu9UEf4zLsqO3DHO1189AFNQ= 2023-09-22T21:36:22.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3mxyu9UEf4zLsqO3DHO1189AFNQ=', 'Content-Length': '0'} 2023-09-22T21:36:22.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066cba25feaf5c16f-00650e08d6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-09-22T21:36:22.359 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.360 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.360 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.360 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.360 DEBUG:boto:Method: GET 2023-09-22T21:36:22.360 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.360 DEBUG:boto:Data: 2023-09-22T21:36:22.360 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.360 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.360 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.360 DEBUG:boto:Params: {} 2023-09-22T21:36:22.361 DEBUG:boto:Token: None 2023-09-22T21:36:22.361 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.361 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hmPNE3SxuLJY70RcQlVjCEqciLc= 2023-09-22T21:36:22.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hmPNE3SxuLJY70RcQlVjCEqciLc=', 'Content-Length': '0'} 2023-09-22T21:36:22.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa13342dfd435af0-00650e08d6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.406 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.407 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.407 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.407 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.407 DEBUG:boto:Method: GET 2023-09-22T21:36:22.407 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.407 DEBUG:boto:Data: 2023-09-22T21:36:22.407 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.407 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.408 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.408 DEBUG:boto:Params: {} 2023-09-22T21:36:22.408 DEBUG:boto:Token: None 2023-09-22T21:36:22.408 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000058?versionId=null 2023-09-22T21:36:22.408 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hmPNE3SxuLJY70RcQlVjCEqciLc= 2023-09-22T21:36:22.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hmPNE3SxuLJY70RcQlVjCEqciLc=', 'Content-Length': '0'} 2023-09-22T21:36:22.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001deb1f6ff7119f7-00650e08d6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.411 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:22.411 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:22.412 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000058 2023-09-22T21:36:22.412 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000058 2023-09-22T21:36:22.412 DEBUG:boto:Method: GET 2023-09-22T21:36:22.412 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000058 2023-09-22T21:36:22.412 DEBUG:boto:Data: 2023-09-22T21:36:22.412 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.412 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.412 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.412 DEBUG:boto:Params: {} 2023-09-22T21:36:22.413 DEBUG:boto:Token: None 2023-09-22T21:36:22.413 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/ 2023-09-22T21:36:22.413 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw= 2023-09-22T21:36:22.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw=', 'Content-Length': '0'} 2023-09-22T21:36:22.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000defae9ddda37a961-00650e08d6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.458 DEBUG:boto:b'ffzdqu-58obj000000000000000000581000falseobj000000000000000000582023-09-22T21:35:01.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:22.459 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:22.459 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:22.459 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000058 2023-09-22T21:36:22.459 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000058 2023-09-22T21:36:22.459 DEBUG:boto:Method: GET 2023-09-22T21:36:22.460 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000058 2023-09-22T21:36:22.460 DEBUG:boto:Data: 2023-09-22T21:36:22.460 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.460 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.460 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.460 DEBUG:boto:Params: {} 2023-09-22T21:36:22.460 DEBUG:boto:Token: None 2023-09-22T21:36:22.460 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/ 2023-09-22T21:36:22.461 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw= 2023-09-22T21:36:22.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw=', 'Content-Length': '0'} 2023-09-22T21:36:22.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000689215fbfb7c0259-00650e08d6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.471 DEBUG:boto:b'ffzdqu-58obj000000000000000000581000falseobj000000000000000000582023-09-22T21:35:01.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:22.472 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.472 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.472 DEBUG:boto:Method: HEAD 2023-09-22T21:36:22.472 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.472 DEBUG:boto:Data: 2023-09-22T21:36:22.472 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.472 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.473 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.473 DEBUG:boto:Params: {} 2023-09-22T21:36:22.473 DEBUG:boto:Token: None 2023-09-22T21:36:22.473 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.473 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RrMadjfIEXunVjREYWu8cpthGyQ= 2023-09-22T21:36:22.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RrMadjfIEXunVjREYWu8cpthGyQ=', 'Content-Length': '0'} 2023-09-22T21:36:22.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092dbeb33fceffb36-00650e08d6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.476 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.476 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.476 DEBUG:boto:Method: HEAD 2023-09-22T21:36:22.476 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.477 DEBUG:boto:Data: 2023-09-22T21:36:22.477 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.477 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.477 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.477 DEBUG:boto:Params: {} 2023-09-22T21:36:22.477 DEBUG:boto:Token: None 2023-09-22T21:36:22.477 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.478 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RrMadjfIEXunVjREYWu8cpthGyQ= 2023-09-22T21:36:22.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RrMadjfIEXunVjREYWu8cpthGyQ=', 'Content-Length': '0'} 2023-09-22T21:36:22.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078e13b58b27b76f6-00650e08d6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-09-22T21:36:22.481 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.481 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.481 DEBUG:boto:Method: GET 2023-09-22T21:36:22.481 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.481 DEBUG:boto:Data: 2023-09-22T21:36:22.482 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.482 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.482 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.482 DEBUG:boto:Params: {} 2023-09-22T21:36:22.482 DEBUG:boto:Token: None 2023-09-22T21:36:22.482 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.482 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J9kqIEHOQpPdP12V0cVFRlh13gw= 2023-09-22T21:36:22.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J9kqIEHOQpPdP12V0cVFRlh13gw=', 'Content-Length': '0'} 2023-09-22T21:36:22.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb28873576ead6d7-00650e08d6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.530 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.531 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.531 DEBUG:boto:Method: GET 2023-09-22T21:36:22.531 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.531 DEBUG:boto:Data: 2023-09-22T21:36:22.531 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.531 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.531 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.531 DEBUG:boto:Params: {} 2023-09-22T21:36:22.532 DEBUG:boto:Token: None 2023-09-22T21:36:22.532 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000058 2023-09-22T21:36:22.532 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J9kqIEHOQpPdP12V0cVFRlh13gw= 2023-09-22T21:36:22.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J9kqIEHOQpPdP12V0cVFRlh13gw=', 'Content-Length': '0'} 2023-09-22T21:36:22.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000399697d4b53aa0f0-00650e08d6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.535 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:22.535 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:22.535 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000058 2023-09-22T21:36:22.536 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000058 2023-09-22T21:36:22.536 DEBUG:boto:Method: GET 2023-09-22T21:36:22.536 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000058 2023-09-22T21:36:22.536 DEBUG:boto:Data: 2023-09-22T21:36:22.536 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.536 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.536 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.536 DEBUG:boto:Params: {} 2023-09-22T21:36:22.536 DEBUG:boto:Token: None 2023-09-22T21:36:22.537 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/ 2023-09-22T21:36:22.537 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw= 2023-09-22T21:36:22.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw=', 'Content-Length': '0'} 2023-09-22T21:36:22.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b4170ffddbe079b-00650e08d6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.582 DEBUG:boto:b'ffzdqu-58obj000000000000000000581000falseobj000000000000000000582023-09-22T21:35:01.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:22.583 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:22.583 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:22.583 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000058 2023-09-22T21:36:22.583 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000058 2023-09-22T21:36:22.583 DEBUG:boto:Method: GET 2023-09-22T21:36:22.584 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000058 2023-09-22T21:36:22.584 DEBUG:boto:Data: 2023-09-22T21:36:22.584 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.584 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.584 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.584 DEBUG:boto:Params: {} 2023-09-22T21:36:22.584 DEBUG:boto:Token: None 2023-09-22T21:36:22.584 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/ 2023-09-22T21:36:22.585 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw= 2023-09-22T21:36:22.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw=', 'Content-Length': '0'} 2023-09-22T21:36:22.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090a63567dd6207fa-00650e08d6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.596 DEBUG:boto:b'ffzdqu-58obj000000000000000000581000falseobj000000000000000000582023-09-22T21:35:01.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:22.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-09-22T21:36:22.597 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.597 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.597 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.597 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.597 DEBUG:boto:Method: GET 2023-09-22T21:36:22.597 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.597 DEBUG:boto:Data: 2023-09-22T21:36:22.597 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.598 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.598 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.598 DEBUG:boto:Params: {} 2023-09-22T21:36:22.598 DEBUG:boto:Token: None 2023-09-22T21:36:22.598 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.598 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gn+Zoul4rP4djnxYUBGGCKK9sY0= 2023-09-22T21:36:22.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gn+Zoul4rP4djnxYUBGGCKK9sY0=', 'Content-Length': '0'} 2023-09-22T21:36:22.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae5dd8ea6d237354-00650e08d6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.602 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.602 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.602 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.602 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.602 DEBUG:boto:Method: GET 2023-09-22T21:36:22.602 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.603 DEBUG:boto:Data: 2023-09-22T21:36:22.603 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.603 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.603 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.603 DEBUG:boto:Params: {} 2023-09-22T21:36:22.603 DEBUG:boto:Token: None 2023-09-22T21:36:22.603 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.604 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gn+Zoul4rP4djnxYUBGGCKK9sY0= 2023-09-22T21:36:22.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gn+Zoul4rP4djnxYUBGGCKK9sY0=', 'Content-Length': '0'} 2023-09-22T21:36:22.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cc23321904d3aeb-00650e08d6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.650 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.651 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.651 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.651 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.651 DEBUG:boto:Method: HEAD 2023-09-22T21:36:22.651 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.651 DEBUG:boto:Data: 2023-09-22T21:36:22.651 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.651 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.651 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.652 DEBUG:boto:Params: {} 2023-09-22T21:36:22.652 DEBUG:boto:Token: None 2023-09-22T21:36:22.652 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.652 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vin+HvLKCmzX7jBJlI7aW9O4VQw= 2023-09-22T21:36:22.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vin+HvLKCmzX7jBJlI7aW9O4VQw=', 'Content-Length': '0'} 2023-09-22T21:36:22.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfcb09d1d6ba1e9f-00650e08d6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.655 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.656 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.656 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.656 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.656 DEBUG:boto:Method: HEAD 2023-09-22T21:36:22.656 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.656 DEBUG:boto:Data: 2023-09-22T21:36:22.656 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.656 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.656 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.656 DEBUG:boto:Params: {} 2023-09-22T21:36:22.657 DEBUG:boto:Token: None 2023-09-22T21:36:22.657 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.657 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vin+HvLKCmzX7jBJlI7aW9O4VQw= 2023-09-22T21:36:22.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vin+HvLKCmzX7jBJlI7aW9O4VQw=', 'Content-Length': '0'} 2023-09-22T21:36:22.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d14c858fa87a7127-00650e08d6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-09-22T21:36:22.660 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.660 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.660 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.661 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.661 DEBUG:boto:Method: GET 2023-09-22T21:36:22.661 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.661 DEBUG:boto:Data: 2023-09-22T21:36:22.661 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.661 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.661 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.661 DEBUG:boto:Params: {} 2023-09-22T21:36:22.661 DEBUG:boto:Token: None 2023-09-22T21:36:22.662 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.662 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gn+Zoul4rP4djnxYUBGGCKK9sY0= 2023-09-22T21:36:22.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gn+Zoul4rP4djnxYUBGGCKK9sY0=', 'Content-Length': '0'} 2023-09-22T21:36:22.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003522cb662a24b748-00650e08d6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.706 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.707 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.707 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.707 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.707 DEBUG:boto:Method: GET 2023-09-22T21:36:22.707 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.707 DEBUG:boto:Data: 2023-09-22T21:36:22.707 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.707 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.707 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.708 DEBUG:boto:Params: {} 2023-09-22T21:36:22.708 DEBUG:boto:Token: None 2023-09-22T21:36:22.708 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000059?versionId=null 2023-09-22T21:36:22.708 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gn+Zoul4rP4djnxYUBGGCKK9sY0= 2023-09-22T21:36:22.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gn+Zoul4rP4djnxYUBGGCKK9sY0=', 'Content-Length': '0'} 2023-09-22T21:36:22.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c248692c36c3427d-00650e08d6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.711 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:22.711 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:22.711 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000059 2023-09-22T21:36:22.711 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000059 2023-09-22T21:36:22.712 DEBUG:boto:Method: GET 2023-09-22T21:36:22.712 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000059 2023-09-22T21:36:22.712 DEBUG:boto:Data: 2023-09-22T21:36:22.712 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.712 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.712 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.712 DEBUG:boto:Params: {} 2023-09-22T21:36:22.712 DEBUG:boto:Token: None 2023-09-22T21:36:22.713 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/ 2023-09-22T21:36:22.713 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw= 2023-09-22T21:36:22.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw=', 'Content-Length': '0'} 2023-09-22T21:36:22.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad41e2c74c1a6313-00650e08d6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.758 DEBUG:boto:b'ffzdqu-58obj000000000000000000591000falseobj000000000000000000592023-09-22T21:35:02.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:22.759 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:22.759 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:22.759 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000059 2023-09-22T21:36:22.759 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000059 2023-09-22T21:36:22.759 DEBUG:boto:Method: GET 2023-09-22T21:36:22.759 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000059 2023-09-22T21:36:22.759 DEBUG:boto:Data: 2023-09-22T21:36:22.760 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.760 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.760 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.760 DEBUG:boto:Params: {} 2023-09-22T21:36:22.760 DEBUG:boto:Token: None 2023-09-22T21:36:22.760 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/ 2023-09-22T21:36:22.760 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw= 2023-09-22T21:36:22.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw=', 'Content-Length': '0'} 2023-09-22T21:36:22.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000961572985bdb9c4-00650e08d6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.771 DEBUG:boto:b'ffzdqu-58obj000000000000000000591000falseobj000000000000000000592023-09-22T21:35:02.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:22.772 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.772 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.772 DEBUG:boto:Method: HEAD 2023-09-22T21:36:22.772 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.772 DEBUG:boto:Data: 2023-09-22T21:36:22.772 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.772 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.773 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.773 DEBUG:boto:Params: {} 2023-09-22T21:36:22.773 DEBUG:boto:Token: None 2023-09-22T21:36:22.773 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.773 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:47Nb4x/viSRgQTvD3GFpMb8/UOs= 2023-09-22T21:36:22.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:47Nb4x/viSRgQTvD3GFpMb8/UOs=', 'Content-Length': '0'} 2023-09-22T21:36:22.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000945a0cacf68694b8-00650e08d6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.776 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.776 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.776 DEBUG:boto:Method: HEAD 2023-09-22T21:36:22.777 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.777 DEBUG:boto:Data: 2023-09-22T21:36:22.777 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.777 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.777 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.777 DEBUG:boto:Params: {} 2023-09-22T21:36:22.777 DEBUG:boto:Token: None 2023-09-22T21:36:22.777 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.777 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:47Nb4x/viSRgQTvD3GFpMb8/UOs= 2023-09-22T21:36:22.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:47Nb4x/viSRgQTvD3GFpMb8/UOs=', 'Content-Length': '0'} 2023-09-22T21:36:22.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064967c799533d70a-00650e08d6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-09-22T21:36:22.780 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.781 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.781 DEBUG:boto:Method: GET 2023-09-22T21:36:22.781 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.781 DEBUG:boto:Data: 2023-09-22T21:36:22.781 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.781 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.781 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.781 DEBUG:boto:Params: {} 2023-09-22T21:36:22.781 DEBUG:boto:Token: None 2023-09-22T21:36:22.782 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.782 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LBJ2x+GusAH42A35pHQPztC2atA= 2023-09-22T21:36:22.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LBJ2x+GusAH42A35pHQPztC2atA=', 'Content-Length': '0'} 2023-09-22T21:36:22.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005381d66c9442dd1d-00650e08d6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.826 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.826 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.826 DEBUG:boto:Method: GET 2023-09-22T21:36:22.827 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.827 DEBUG:boto:Data: 2023-09-22T21:36:22.827 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.827 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.827 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.827 DEBUG:boto:Params: {} 2023-09-22T21:36:22.827 DEBUG:boto:Token: None 2023-09-22T21:36:22.827 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000059 2023-09-22T21:36:22.827 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LBJ2x+GusAH42A35pHQPztC2atA= 2023-09-22T21:36:22.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LBJ2x+GusAH42A35pHQPztC2atA=', 'Content-Length': '0'} 2023-09-22T21:36:22.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000349f8fb2dca8dc04-00650e08d6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.831 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:22.831 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:22.831 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000059 2023-09-22T21:36:22.831 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000059 2023-09-22T21:36:22.831 DEBUG:boto:Method: GET 2023-09-22T21:36:22.831 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000059 2023-09-22T21:36:22.831 DEBUG:boto:Data: 2023-09-22T21:36:22.831 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.831 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.832 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.832 DEBUG:boto:Params: {} 2023-09-22T21:36:22.832 DEBUG:boto:Token: None 2023-09-22T21:36:22.832 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/ 2023-09-22T21:36:22.832 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw= 2023-09-22T21:36:22.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw=', 'Content-Length': '0'} 2023-09-22T21:36:22.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049d44a5bf96be4eb-00650e08d6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.878 DEBUG:boto:b'ffzdqu-58obj000000000000000000591000falseobj000000000000000000592023-09-22T21:35:02.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:22.879 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:22.879 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:22.879 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000059 2023-09-22T21:36:22.879 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000059 2023-09-22T21:36:22.879 DEBUG:boto:Method: GET 2023-09-22T21:36:22.880 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000059 2023-09-22T21:36:22.880 DEBUG:boto:Data: 2023-09-22T21:36:22.880 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.880 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.880 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.880 DEBUG:boto:Params: {} 2023-09-22T21:36:22.880 DEBUG:boto:Token: None 2023-09-22T21:36:22.880 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/ 2023-09-22T21:36:22.881 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw= 2023-09-22T21:36:22.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3y8KYL+uBmQliTiAlaA5orH0OHw=', 'Content-Length': '0'} 2023-09-22T21:36:22.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c432fe194ec797f6-00650e08d6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.892 DEBUG:boto:b'ffzdqu-58obj000000000000000000591000falseobj000000000000000000592023-09-22T21:35:02.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:22.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-09-22T21:36:22.893 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:22.893 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:22.893 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:22.893 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:22.893 DEBUG:boto:Method: GET 2023-09-22T21:36:22.893 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:22.893 DEBUG:boto:Data: 2023-09-22T21:36:22.893 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.893 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.893 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.893 DEBUG:boto:Params: {} 2023-09-22T21:36:22.894 DEBUG:boto:Token: None 2023-09-22T21:36:22.894 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:22.894 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:efcZJBCVGIQf3OALO+dwpLrQx0k= 2023-09-22T21:36:22.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:efcZJBCVGIQf3OALO+dwpLrQx0k=', 'Content-Length': '0'} 2023-09-22T21:36:22.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009974c7661dcd8014-00650e08d6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.898 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:22.898 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:22.898 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:22.898 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:22.898 DEBUG:boto:Method: GET 2023-09-22T21:36:22.898 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:22.898 DEBUG:boto:Data: 2023-09-22T21:36:22.898 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.898 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.899 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.899 DEBUG:boto:Params: {} 2023-09-22T21:36:22.899 DEBUG:boto:Token: None 2023-09-22T21:36:22.899 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:22.899 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:efcZJBCVGIQf3OALO+dwpLrQx0k= 2023-09-22T21:36:22.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:efcZJBCVGIQf3OALO+dwpLrQx0k=', 'Content-Length': '0'} 2023-09-22T21:36:22.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000903bde6ad4216b6b-00650e08d6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.946 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:22.947 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:22.947 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:22.947 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:22.947 DEBUG:boto:Method: HEAD 2023-09-22T21:36:22.947 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:22.947 DEBUG:boto:Data: 2023-09-22T21:36:22.947 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.947 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.947 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.948 DEBUG:boto:Params: {} 2023-09-22T21:36:22.948 DEBUG:boto:Token: None 2023-09-22T21:36:22.948 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:22.948 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aHWaXUZIBTGHcF+NwzwNjFxrea8= 2023-09-22T21:36:22.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aHWaXUZIBTGHcF+NwzwNjFxrea8=', 'Content-Length': '0'} 2023-09-22T21:36:22.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c442abdf593c2542-00650e08d6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.951 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:22.951 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:22.951 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:22.952 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:22.952 DEBUG:boto:Method: HEAD 2023-09-22T21:36:22.952 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:22.952 DEBUG:boto:Data: 2023-09-22T21:36:22.952 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.952 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:22.952 DEBUG:boto:Port: 8000 2023-09-22T21:36:22.952 DEBUG:boto:Params: {} 2023-09-22T21:36:22.952 DEBUG:boto:Token: None 2023-09-22T21:36:22.953 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:22.953 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aHWaXUZIBTGHcF+NwzwNjFxrea8= 2023-09-22T21:36:22.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aHWaXUZIBTGHcF+NwzwNjFxrea8=', 'Content-Length': '0'} 2023-09-22T21:36:22.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097c3a9057e0f134d-00650e08d6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:22.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-09-22T21:36:22.956 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:22.956 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:22.956 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:22.957 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:22.957 DEBUG:boto:Method: GET 2023-09-22T21:36:22.957 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:22.957 DEBUG:boto:Data: 2023-09-22T21:36:22.957 DEBUG:boto:Headers: {} 2023-09-22T21:36:22.957 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:22.957 DEBUG:boto:Port: 8001 2023-09-22T21:36:22.957 DEBUG:boto:Params: {} 2023-09-22T21:36:22.958 DEBUG:boto:Token: None 2023-09-22T21:36:22.958 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:22 GMT /ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:22.958 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:efcZJBCVGIQf3OALO+dwpLrQx0k= 2023-09-22T21:36:22.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:22 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:efcZJBCVGIQf3OALO+dwpLrQx0k=', 'Content-Length': '0'} 2023-09-22T21:36:22.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aae638c4359b7124-00650e08d6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.002 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:23.003 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:23.003 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:23.003 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:23.003 DEBUG:boto:Method: GET 2023-09-22T21:36:23.003 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:23.003 DEBUG:boto:Data: 2023-09-22T21:36:23.003 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.003 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.003 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.004 DEBUG:boto:Params: {} 2023-09-22T21:36:23.004 DEBUG:boto:Token: None 2023-09-22T21:36:23.004 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000060?versionId=null 2023-09-22T21:36:23.004 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cNU8r49zp1kGGoFs/2tDgiKzIQw= 2023-09-22T21:36:23.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cNU8r49zp1kGGoFs/2tDgiKzIQw=', 'Content-Length': '0'} 2023-09-22T21:36:23.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f9812f593a9cf94-00650e08d7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.008 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:23.008 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:23.008 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000060 2023-09-22T21:36:23.008 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000060 2023-09-22T21:36:23.008 DEBUG:boto:Method: GET 2023-09-22T21:36:23.008 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000060 2023-09-22T21:36:23.008 DEBUG:boto:Data: 2023-09-22T21:36:23.008 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.009 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.009 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.009 DEBUG:boto:Params: {} 2023-09-22T21:36:23.009 DEBUG:boto:Token: None 2023-09-22T21:36:23.009 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/ 2023-09-22T21:36:23.009 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU= 2023-09-22T21:36:23.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU=', 'Content-Length': '0'} 2023-09-22T21:36:23.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055f2cbd5967e8af8-00650e08d7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.058 DEBUG:boto:b'ffzdqu-58obj000000000000000000601000falseobj000000000000000000602023-09-22T21:35:05.376Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:23.059 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:23.059 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:23.059 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000060 2023-09-22T21:36:23.060 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000060 2023-09-22T21:36:23.060 DEBUG:boto:Method: GET 2023-09-22T21:36:23.060 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000060 2023-09-22T21:36:23.060 DEBUG:boto:Data: 2023-09-22T21:36:23.060 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.061 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.061 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.061 DEBUG:boto:Params: {} 2023-09-22T21:36:23.061 DEBUG:boto:Token: None 2023-09-22T21:36:23.062 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/ 2023-09-22T21:36:23.062 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU= 2023-09-22T21:36:23.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU=', 'Content-Length': '0'} 2023-09-22T21:36:23.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cb789a0387b4900-00650e08d7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.072 DEBUG:boto:b'ffzdqu-58obj000000000000000000601000falseobj000000000000000000602023-09-22T21:35:05.376Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:23.073 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:23.073 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:23.073 DEBUG:boto:Method: HEAD 2023-09-22T21:36:23.073 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:23.073 DEBUG:boto:Data: 2023-09-22T21:36:23.073 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.073 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.073 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.073 DEBUG:boto:Params: {} 2023-09-22T21:36:23.074 DEBUG:boto:Token: None 2023-09-22T21:36:23.074 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:23.074 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ItO7qRuiKyHsT9spek0VIJuM0so= 2023-09-22T21:36:23.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ItO7qRuiKyHsT9spek0VIJuM0so=', 'Content-Length': '0'} 2023-09-22T21:36:23.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fe91937014912ed-00650e08d7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.077 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:23.077 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:23.077 DEBUG:boto:Method: HEAD 2023-09-22T21:36:23.077 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:23.077 DEBUG:boto:Data: 2023-09-22T21:36:23.078 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.078 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.078 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.078 DEBUG:boto:Params: {} 2023-09-22T21:36:23.078 DEBUG:boto:Token: None 2023-09-22T21:36:23.078 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:23.078 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ItO7qRuiKyHsT9spek0VIJuM0so= 2023-09-22T21:36:23.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ItO7qRuiKyHsT9spek0VIJuM0so=', 'Content-Length': '0'} 2023-09-22T21:36:23.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c17f801b2d6c2fc5-00650e08d7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-09-22T21:36:23.081 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:23.082 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:23.082 DEBUG:boto:Method: GET 2023-09-22T21:36:23.082 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:23.082 DEBUG:boto:Data: 2023-09-22T21:36:23.082 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.082 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.082 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.082 DEBUG:boto:Params: {} 2023-09-22T21:36:23.082 DEBUG:boto:Token: None 2023-09-22T21:36:23.083 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:23.083 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t1X1wV0YpDtjEYdD3WAUMMozXTQ= 2023-09-22T21:36:23.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t1X1wV0YpDtjEYdD3WAUMMozXTQ=', 'Content-Length': '0'} 2023-09-22T21:36:23.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000393aa3db0ccd2c4d-00650e08d7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.126 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:23.127 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:23.127 DEBUG:boto:Method: GET 2023-09-22T21:36:23.127 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:23.127 DEBUG:boto:Data: 2023-09-22T21:36:23.127 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.127 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.127 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.127 DEBUG:boto:Params: {} 2023-09-22T21:36:23.128 DEBUG:boto:Token: None 2023-09-22T21:36:23.128 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000060 2023-09-22T21:36:23.128 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t1X1wV0YpDtjEYdD3WAUMMozXTQ= 2023-09-22T21:36:23.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t1X1wV0YpDtjEYdD3WAUMMozXTQ=', 'Content-Length': '0'} 2023-09-22T21:36:23.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3c2845fbffacd73-00650e08d7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.131 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:23.132 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:23.132 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000060 2023-09-22T21:36:23.132 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000060 2023-09-22T21:36:23.132 DEBUG:boto:Method: GET 2023-09-22T21:36:23.132 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000060 2023-09-22T21:36:23.132 DEBUG:boto:Data: 2023-09-22T21:36:23.132 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.132 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.132 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.133 DEBUG:boto:Params: {} 2023-09-22T21:36:23.133 DEBUG:boto:Token: None 2023-09-22T21:36:23.133 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/ 2023-09-22T21:36:23.133 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU= 2023-09-22T21:36:23.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU=', 'Content-Length': '0'} 2023-09-22T21:36:23.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dadde069cdfc8da1-00650e08d7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.138 DEBUG:boto:b'ffzdqu-58obj000000000000000000601000falseobj000000000000000000602023-09-22T21:35:05.376Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:23.138 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:23.138 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:23.138 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000060 2023-09-22T21:36:23.139 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000060 2023-09-22T21:36:23.139 DEBUG:boto:Method: GET 2023-09-22T21:36:23.139 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000060 2023-09-22T21:36:23.139 DEBUG:boto:Data: 2023-09-22T21:36:23.139 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.139 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.139 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.139 DEBUG:boto:Params: {} 2023-09-22T21:36:23.139 DEBUG:boto:Token: None 2023-09-22T21:36:23.140 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/ 2023-09-22T21:36:23.140 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU= 2023-09-22T21:36:23.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU=', 'Content-Length': '0'} 2023-09-22T21:36:23.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000265cea42a2bc695e-00650e08d7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.198 DEBUG:boto:b'ffzdqu-58obj000000000000000000601000falseobj000000000000000000602023-09-22T21:35:05.376Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:23.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-09-22T21:36:23.199 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.199 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.199 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.199 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.199 DEBUG:boto:Method: GET 2023-09-22T21:36:23.200 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.200 DEBUG:boto:Data: 2023-09-22T21:36:23.200 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.200 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.200 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.200 DEBUG:boto:Params: {} 2023-09-22T21:36:23.200 DEBUG:boto:Token: None 2023-09-22T21:36:23.200 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.201 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JI+QiJvviexNvxCtyht03hpY5js= 2023-09-22T21:36:23.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JI+QiJvviexNvxCtyht03hpY5js=', 'Content-Length': '0'} 2023-09-22T21:36:23.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfe923ccc6b9124c-00650e08d7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.246 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.246 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.247 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.247 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.247 DEBUG:boto:Method: GET 2023-09-22T21:36:23.247 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.247 DEBUG:boto:Data: 2023-09-22T21:36:23.247 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.247 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.247 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.247 DEBUG:boto:Params: {} 2023-09-22T21:36:23.248 DEBUG:boto:Token: None 2023-09-22T21:36:23.248 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.248 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JI+QiJvviexNvxCtyht03hpY5js= 2023-09-22T21:36:23.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JI+QiJvviexNvxCtyht03hpY5js=', 'Content-Length': '0'} 2023-09-22T21:36:23.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a09bbd0e1ec581fa-00650e08d7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.251 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.251 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.252 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.252 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.252 DEBUG:boto:Method: HEAD 2023-09-22T21:36:23.252 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.252 DEBUG:boto:Data: 2023-09-22T21:36:23.252 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.252 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.252 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.252 DEBUG:boto:Params: {} 2023-09-22T21:36:23.253 DEBUG:boto:Token: None 2023-09-22T21:36:23.253 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.253 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ygpZQAyGiTx87S246HvZNg3VgZE= 2023-09-22T21:36:23.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ygpZQAyGiTx87S246HvZNg3VgZE=', 'Content-Length': '0'} 2023-09-22T21:36:23.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acfd67e8ad6ec156-00650e08d7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.256 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.256 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.256 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.256 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.257 DEBUG:boto:Method: HEAD 2023-09-22T21:36:23.257 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.257 DEBUG:boto:Data: 2023-09-22T21:36:23.257 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.257 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.257 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.257 DEBUG:boto:Params: {} 2023-09-22T21:36:23.257 DEBUG:boto:Token: None 2023-09-22T21:36:23.257 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.258 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ygpZQAyGiTx87S246HvZNg3VgZE= 2023-09-22T21:36:23.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ygpZQAyGiTx87S246HvZNg3VgZE=', 'Content-Length': '0'} 2023-09-22T21:36:23.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5c9f1e4232b03af-00650e08d7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-09-22T21:36:23.261 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.261 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.261 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.261 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.261 DEBUG:boto:Method: GET 2023-09-22T21:36:23.261 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.262 DEBUG:boto:Data: 2023-09-22T21:36:23.262 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.262 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.262 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.262 DEBUG:boto:Params: {} 2023-09-22T21:36:23.262 DEBUG:boto:Token: None 2023-09-22T21:36:23.262 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.262 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JI+QiJvviexNvxCtyht03hpY5js= 2023-09-22T21:36:23.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JI+QiJvviexNvxCtyht03hpY5js=', 'Content-Length': '0'} 2023-09-22T21:36:23.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000400fd0f99c835ced-00650e08d7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.314 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.314 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.315 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.315 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.315 DEBUG:boto:Method: GET 2023-09-22T21:36:23.315 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.315 DEBUG:boto:Data: 2023-09-22T21:36:23.316 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.316 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.316 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.316 DEBUG:boto:Params: {} 2023-09-22T21:36:23.316 DEBUG:boto:Token: None 2023-09-22T21:36:23.317 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000061?versionId=null 2023-09-22T21:36:23.317 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JI+QiJvviexNvxCtyht03hpY5js= 2023-09-22T21:36:23.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JI+QiJvviexNvxCtyht03hpY5js=', 'Content-Length': '0'} 2023-09-22T21:36:23.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082867bc38c372270-00650e08d7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.320 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:23.321 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:23.321 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000061 2023-09-22T21:36:23.321 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000061 2023-09-22T21:36:23.321 DEBUG:boto:Method: GET 2023-09-22T21:36:23.322 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000061 2023-09-22T21:36:23.322 DEBUG:boto:Data: 2023-09-22T21:36:23.322 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.322 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.322 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.322 DEBUG:boto:Params: {} 2023-09-22T21:36:23.323 DEBUG:boto:Token: None 2023-09-22T21:36:23.323 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/ 2023-09-22T21:36:23.323 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU= 2023-09-22T21:36:23.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU=', 'Content-Length': '0'} 2023-09-22T21:36:23.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000911c770303235041-00650e08d7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.370 DEBUG:boto:b'ffzdqu-58obj000000000000000000611000falseobj000000000000000000612023-09-22T21:35:04.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:23.371 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:23.371 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:23.371 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000061 2023-09-22T21:36:23.371 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000061 2023-09-22T21:36:23.372 DEBUG:boto:Method: GET 2023-09-22T21:36:23.372 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000061 2023-09-22T21:36:23.372 DEBUG:boto:Data: 2023-09-22T21:36:23.372 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.372 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.373 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.373 DEBUG:boto:Params: {} 2023-09-22T21:36:23.373 DEBUG:boto:Token: None 2023-09-22T21:36:23.373 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/ 2023-09-22T21:36:23.374 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU= 2023-09-22T21:36:23.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU=', 'Content-Length': '0'} 2023-09-22T21:36:23.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f33916d085311563-00650e08d7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.384 DEBUG:boto:b'ffzdqu-58obj000000000000000000611000falseobj000000000000000000612023-09-22T21:35:04.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:23.385 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.385 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.385 DEBUG:boto:Method: HEAD 2023-09-22T21:36:23.385 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.385 DEBUG:boto:Data: 2023-09-22T21:36:23.385 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.385 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.385 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.386 DEBUG:boto:Params: {} 2023-09-22T21:36:23.386 DEBUG:boto:Token: None 2023-09-22T21:36:23.386 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.386 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hJOumtXRuQF3m/zz8+jIgwjTgsU= 2023-09-22T21:36:23.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hJOumtXRuQF3m/zz8+jIgwjTgsU=', 'Content-Length': '0'} 2023-09-22T21:36:23.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e02ed0a6c25a770-00650e08d7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.389 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.389 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.389 DEBUG:boto:Method: HEAD 2023-09-22T21:36:23.389 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.390 DEBUG:boto:Data: 2023-09-22T21:36:23.390 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.390 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.390 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.390 DEBUG:boto:Params: {} 2023-09-22T21:36:23.390 DEBUG:boto:Token: None 2023-09-22T21:36:23.390 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.390 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hJOumtXRuQF3m/zz8+jIgwjTgsU= 2023-09-22T21:36:23.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hJOumtXRuQF3m/zz8+jIgwjTgsU=', 'Content-Length': '0'} 2023-09-22T21:36:23.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000009b4eb3262a9b03-00650e08d7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-09-22T21:36:23.394 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.394 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.394 DEBUG:boto:Method: GET 2023-09-22T21:36:23.394 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.394 DEBUG:boto:Data: 2023-09-22T21:36:23.394 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.395 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.395 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.395 DEBUG:boto:Params: {} 2023-09-22T21:36:23.395 DEBUG:boto:Token: None 2023-09-22T21:36:23.395 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.395 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G95yhRlP1BKT5qCpfJ+Jr+sHbo8= 2023-09-22T21:36:23.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G95yhRlP1BKT5qCpfJ+Jr+sHbo8=', 'Content-Length': '0'} 2023-09-22T21:36:23.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010c334de3c7b3503-00650e08d7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.442 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.443 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.443 DEBUG:boto:Method: GET 2023-09-22T21:36:23.443 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.443 DEBUG:boto:Data: 2023-09-22T21:36:23.443 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.443 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.443 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.443 DEBUG:boto:Params: {} 2023-09-22T21:36:23.444 DEBUG:boto:Token: None 2023-09-22T21:36:23.444 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000061 2023-09-22T21:36:23.444 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G95yhRlP1BKT5qCpfJ+Jr+sHbo8= 2023-09-22T21:36:23.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G95yhRlP1BKT5qCpfJ+Jr+sHbo8=', 'Content-Length': '0'} 2023-09-22T21:36:23.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec09507f7ee6479a-00650e08d7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.448 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:23.448 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:23.448 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000061 2023-09-22T21:36:23.449 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000061 2023-09-22T21:36:23.449 DEBUG:boto:Method: GET 2023-09-22T21:36:23.449 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000061 2023-09-22T21:36:23.449 DEBUG:boto:Data: 2023-09-22T21:36:23.449 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.450 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.450 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.450 DEBUG:boto:Params: {} 2023-09-22T21:36:23.450 DEBUG:boto:Token: None 2023-09-22T21:36:23.451 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/ 2023-09-22T21:36:23.451 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU= 2023-09-22T21:36:23.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU=', 'Content-Length': '0'} 2023-09-22T21:36:23.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f7a8d718fcefd5d-00650e08d7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.456 DEBUG:boto:b'ffzdqu-58obj000000000000000000611000falseobj000000000000000000612023-09-22T21:35:04.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:23.456 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:23.457 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:23.457 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000061 2023-09-22T21:36:23.457 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000061 2023-09-22T21:36:23.457 DEBUG:boto:Method: GET 2023-09-22T21:36:23.457 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000061 2023-09-22T21:36:23.457 DEBUG:boto:Data: 2023-09-22T21:36:23.457 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.458 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.458 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.458 DEBUG:boto:Params: {} 2023-09-22T21:36:23.458 DEBUG:boto:Token: None 2023-09-22T21:36:23.458 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/ 2023-09-22T21:36:23.458 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU= 2023-09-22T21:36:23.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU=', 'Content-Length': '0'} 2023-09-22T21:36:23.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009218ca38eaa800e0-00650e08d7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.514 DEBUG:boto:b'ffzdqu-58obj000000000000000000611000falseobj000000000000000000612023-09-22T21:35:04.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:23.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-09-22T21:36:23.515 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.515 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.515 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.515 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.515 DEBUG:boto:Method: GET 2023-09-22T21:36:23.516 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.516 DEBUG:boto:Data: 2023-09-22T21:36:23.516 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.516 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.516 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.516 DEBUG:boto:Params: {} 2023-09-22T21:36:23.516 DEBUG:boto:Token: None 2023-09-22T21:36:23.516 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.517 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Te0nDuXfFr4MxVpvy5pEgsvyRGw= 2023-09-22T21:36:23.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Te0nDuXfFr4MxVpvy5pEgsvyRGw=', 'Content-Length': '0'} 2023-09-22T21:36:23.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8a59d493f5f3066-00650e08d7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.566 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.567 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.567 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.567 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.567 DEBUG:boto:Method: GET 2023-09-22T21:36:23.567 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.567 DEBUG:boto:Data: 2023-09-22T21:36:23.567 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.567 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.567 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.568 DEBUG:boto:Params: {} 2023-09-22T21:36:23.568 DEBUG:boto:Token: None 2023-09-22T21:36:23.568 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.568 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Te0nDuXfFr4MxVpvy5pEgsvyRGw= 2023-09-22T21:36:23.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Te0nDuXfFr4MxVpvy5pEgsvyRGw=', 'Content-Length': '0'} 2023-09-22T21:36:23.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f6def4f973ee814-00650e08d7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.571 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.571 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.572 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.572 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.572 DEBUG:boto:Method: HEAD 2023-09-22T21:36:23.572 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.572 DEBUG:boto:Data: 2023-09-22T21:36:23.572 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.572 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.572 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.572 DEBUG:boto:Params: {} 2023-09-22T21:36:23.573 DEBUG:boto:Token: None 2023-09-22T21:36:23.573 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.573 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2G/XdI6h+Cx4wSgteWsDWlnf6nU= 2023-09-22T21:36:23.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2G/XdI6h+Cx4wSgteWsDWlnf6nU=', 'Content-Length': '0'} 2023-09-22T21:36:23.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cae487ff97ce141-00650e08d7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.576 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.576 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.576 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.576 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.576 DEBUG:boto:Method: HEAD 2023-09-22T21:36:23.576 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.577 DEBUG:boto:Data: 2023-09-22T21:36:23.577 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.577 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.577 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.577 DEBUG:boto:Params: {} 2023-09-22T21:36:23.577 DEBUG:boto:Token: None 2023-09-22T21:36:23.577 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.577 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2G/XdI6h+Cx4wSgteWsDWlnf6nU= 2023-09-22T21:36:23.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2G/XdI6h+Cx4wSgteWsDWlnf6nU=', 'Content-Length': '0'} 2023-09-22T21:36:23.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c7ea882ccd986cf-00650e08d7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-09-22T21:36:23.582 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.582 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.582 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.582 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.582 DEBUG:boto:Method: GET 2023-09-22T21:36:23.582 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.582 DEBUG:boto:Data: 2023-09-22T21:36:23.582 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.582 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.583 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.583 DEBUG:boto:Params: {} 2023-09-22T21:36:23.583 DEBUG:boto:Token: None 2023-09-22T21:36:23.583 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.583 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Te0nDuXfFr4MxVpvy5pEgsvyRGw= 2023-09-22T21:36:23.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Te0nDuXfFr4MxVpvy5pEgsvyRGw=', 'Content-Length': '0'} 2023-09-22T21:36:23.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eac7e7eca97ad2d2-00650e08d7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.634 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.635 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.635 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.635 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.635 DEBUG:boto:Method: GET 2023-09-22T21:36:23.635 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.635 DEBUG:boto:Data: 2023-09-22T21:36:23.635 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.635 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.635 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.636 DEBUG:boto:Params: {} 2023-09-22T21:36:23.636 DEBUG:boto:Token: None 2023-09-22T21:36:23.636 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000062?versionId=null 2023-09-22T21:36:23.636 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Te0nDuXfFr4MxVpvy5pEgsvyRGw= 2023-09-22T21:36:23.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Te0nDuXfFr4MxVpvy5pEgsvyRGw=', 'Content-Length': '0'} 2023-09-22T21:36:23.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0b70086da2e508a-00650e08d7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.639 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:23.640 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:23.640 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000062 2023-09-22T21:36:23.640 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000062 2023-09-22T21:36:23.640 DEBUG:boto:Method: GET 2023-09-22T21:36:23.640 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000062 2023-09-22T21:36:23.640 DEBUG:boto:Data: 2023-09-22T21:36:23.640 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.640 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.640 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.641 DEBUG:boto:Params: {} 2023-09-22T21:36:23.641 DEBUG:boto:Token: None 2023-09-22T21:36:23.641 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/ 2023-09-22T21:36:23.641 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU= 2023-09-22T21:36:23.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU=', 'Content-Length': '0'} 2023-09-22T21:36:23.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a64b8fb03c6150a3-00650e08d7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.690 DEBUG:boto:b'ffzdqu-58obj000000000000000000621000falseobj000000000000000000622023-09-22T21:35:05.285Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:23.691 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:23.691 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:23.691 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000062 2023-09-22T21:36:23.691 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000062 2023-09-22T21:36:23.691 DEBUG:boto:Method: GET 2023-09-22T21:36:23.692 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000062 2023-09-22T21:36:23.692 DEBUG:boto:Data: 2023-09-22T21:36:23.692 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.692 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.692 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.692 DEBUG:boto:Params: {} 2023-09-22T21:36:23.692 DEBUG:boto:Token: None 2023-09-22T21:36:23.692 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/ 2023-09-22T21:36:23.692 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU= 2023-09-22T21:36:23.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU=', 'Content-Length': '0'} 2023-09-22T21:36:23.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f7dc04f73c3ae06-00650e08d7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.705 DEBUG:boto:b'ffzdqu-58obj000000000000000000621000falseobj000000000000000000622023-09-22T21:35:05.285Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:23.705 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.705 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.706 DEBUG:boto:Method: HEAD 2023-09-22T21:36:23.706 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.706 DEBUG:boto:Data: 2023-09-22T21:36:23.706 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.706 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.706 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.706 DEBUG:boto:Params: {} 2023-09-22T21:36:23.706 DEBUG:boto:Token: None 2023-09-22T21:36:23.707 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.707 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+P9u7viSGqap+3RPvZZMiDrmj/w= 2023-09-22T21:36:23.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+P9u7viSGqap+3RPvZZMiDrmj/w=', 'Content-Length': '0'} 2023-09-22T21:36:23.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e79cd582349afd49-00650e08d7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.710 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.710 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.710 DEBUG:boto:Method: HEAD 2023-09-22T21:36:23.710 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.710 DEBUG:boto:Data: 2023-09-22T21:36:23.710 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.710 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.711 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.711 DEBUG:boto:Params: {} 2023-09-22T21:36:23.711 DEBUG:boto:Token: None 2023-09-22T21:36:23.711 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.711 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+P9u7viSGqap+3RPvZZMiDrmj/w= 2023-09-22T21:36:23.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+P9u7viSGqap+3RPvZZMiDrmj/w=', 'Content-Length': '0'} 2023-09-22T21:36:23.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2678ce40f0a2a17-00650e08d7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-09-22T21:36:23.714 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.714 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.714 DEBUG:boto:Method: GET 2023-09-22T21:36:23.715 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.715 DEBUG:boto:Data: 2023-09-22T21:36:23.715 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.715 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.715 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.715 DEBUG:boto:Params: {} 2023-09-22T21:36:23.715 DEBUG:boto:Token: None 2023-09-22T21:36:23.715 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.715 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zrTkBeyRtkyW/QO1W1zNAD9M/Zg= 2023-09-22T21:36:23.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zrTkBeyRtkyW/QO1W1zNAD9M/Zg=', 'Content-Length': '0'} 2023-09-22T21:36:23.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fc8bd4728e311e4-00650e08d7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.762 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.763 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.763 DEBUG:boto:Method: GET 2023-09-22T21:36:23.763 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.763 DEBUG:boto:Data: 2023-09-22T21:36:23.763 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.763 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.763 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.763 DEBUG:boto:Params: {} 2023-09-22T21:36:23.763 DEBUG:boto:Token: None 2023-09-22T21:36:23.764 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000062 2023-09-22T21:36:23.764 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zrTkBeyRtkyW/QO1W1zNAD9M/Zg= 2023-09-22T21:36:23.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zrTkBeyRtkyW/QO1W1zNAD9M/Zg=', 'Content-Length': '0'} 2023-09-22T21:36:23.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066b560d700fdb4be-00650e08d7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.767 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:23.767 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:23.767 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000062 2023-09-22T21:36:23.767 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000062 2023-09-22T21:36:23.767 DEBUG:boto:Method: GET 2023-09-22T21:36:23.768 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000062 2023-09-22T21:36:23.768 DEBUG:boto:Data: 2023-09-22T21:36:23.768 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.768 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.768 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.768 DEBUG:boto:Params: {} 2023-09-22T21:36:23.768 DEBUG:boto:Token: None 2023-09-22T21:36:23.768 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/ 2023-09-22T21:36:23.768 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU= 2023-09-22T21:36:23.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU=', 'Content-Length': '0'} 2023-09-22T21:36:23.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c65d1d837d082e78-00650e08d7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.773 DEBUG:boto:b'ffzdqu-58obj000000000000000000621000falseobj000000000000000000622023-09-22T21:35:05.285Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:23.774 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:23.774 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:23.774 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000062 2023-09-22T21:36:23.774 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000062 2023-09-22T21:36:23.774 DEBUG:boto:Method: GET 2023-09-22T21:36:23.774 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000062 2023-09-22T21:36:23.774 DEBUG:boto:Data: 2023-09-22T21:36:23.775 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.775 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.775 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.775 DEBUG:boto:Params: {} 2023-09-22T21:36:23.775 DEBUG:boto:Token: None 2023-09-22T21:36:23.775 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/ 2023-09-22T21:36:23.775 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU= 2023-09-22T21:36:23.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU=', 'Content-Length': '0'} 2023-09-22T21:36:23.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cec3d3a6f9ed07d9-00650e08d7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.830 DEBUG:boto:b'ffzdqu-58obj000000000000000000621000falseobj000000000000000000622023-09-22T21:35:05.285Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:23.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-09-22T21:36:23.831 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:23.831 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:23.831 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.831 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.831 DEBUG:boto:Method: GET 2023-09-22T21:36:23.831 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.831 DEBUG:boto:Data: 2023-09-22T21:36:23.832 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.832 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.832 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.832 DEBUG:boto:Params: {} 2023-09-22T21:36:23.832 DEBUG:boto:Token: None 2023-09-22T21:36:23.832 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.832 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4sQUcLDsjqq5ReSlkFtTvH7m6Eo= 2023-09-22T21:36:23.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4sQUcLDsjqq5ReSlkFtTvH7m6Eo=', 'Content-Length': '0'} 2023-09-22T21:36:23.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ba6c374d43bca05-00650e08d7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.878 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:23.878 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:23.879 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.879 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.879 DEBUG:boto:Method: GET 2023-09-22T21:36:23.879 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.879 DEBUG:boto:Data: 2023-09-22T21:36:23.879 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.879 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.879 DEBUG:boto:Params: {} 2023-09-22T21:36:23.880 DEBUG:boto:Token: None 2023-09-22T21:36:23.880 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.880 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4sQUcLDsjqq5ReSlkFtTvH7m6Eo= 2023-09-22T21:36:23.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4sQUcLDsjqq5ReSlkFtTvH7m6Eo=', 'Content-Length': '0'} 2023-09-22T21:36:23.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faf5b5e5c71f1a0a-00650e08d7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.884 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:23.884 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:23.884 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.884 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.884 DEBUG:boto:Method: HEAD 2023-09-22T21:36:23.884 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.884 DEBUG:boto:Data: 2023-09-22T21:36:23.884 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.884 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.885 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.885 DEBUG:boto:Params: {} 2023-09-22T21:36:23.885 DEBUG:boto:Token: None 2023-09-22T21:36:23.885 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.885 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:heeo22459FpK9ddmxPYN7dSTWSI= 2023-09-22T21:36:23.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:heeo22459FpK9ddmxPYN7dSTWSI=', 'Content-Length': '0'} 2023-09-22T21:36:23.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fd796d46681af6b-00650e08d7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.888 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:23.889 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:23.889 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.889 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.889 DEBUG:boto:Method: HEAD 2023-09-22T21:36:23.889 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.889 DEBUG:boto:Data: 2023-09-22T21:36:23.889 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.889 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.889 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.889 DEBUG:boto:Params: {} 2023-09-22T21:36:23.890 DEBUG:boto:Token: None 2023-09-22T21:36:23.890 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.890 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:heeo22459FpK9ddmxPYN7dSTWSI= 2023-09-22T21:36:23.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:heeo22459FpK9ddmxPYN7dSTWSI=', 'Content-Length': '0'} 2023-09-22T21:36:23.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009934c7b2cb74bb6f-00650e08d7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-09-22T21:36:23.893 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:23.893 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:23.893 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.893 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.893 DEBUG:boto:Method: GET 2023-09-22T21:36:23.893 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.894 DEBUG:boto:Data: 2023-09-22T21:36:23.894 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.894 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.894 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.894 DEBUG:boto:Params: {} 2023-09-22T21:36:23.894 DEBUG:boto:Token: None 2023-09-22T21:36:23.894 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.894 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4sQUcLDsjqq5ReSlkFtTvH7m6Eo= 2023-09-22T21:36:23.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4sQUcLDsjqq5ReSlkFtTvH7m6Eo=', 'Content-Length': '0'} 2023-09-22T21:36:23.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fff2c3cf366b529-00650e08d7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.942 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:23.943 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:23.943 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.943 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.943 DEBUG:boto:Method: GET 2023-09-22T21:36:23.943 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.943 DEBUG:boto:Data: 2023-09-22T21:36:23.943 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.943 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:23.944 DEBUG:boto:Port: 8000 2023-09-22T21:36:23.944 DEBUG:boto:Params: {} 2023-09-22T21:36:23.944 DEBUG:boto:Token: None 2023-09-22T21:36:23.944 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/obj00000000000000000065?versionId=null 2023-09-22T21:36:23.944 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4sQUcLDsjqq5ReSlkFtTvH7m6Eo= 2023-09-22T21:36:23.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4sQUcLDsjqq5ReSlkFtTvH7m6Eo=', 'Content-Length': '0'} 2023-09-22T21:36:23.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d795c503f525110-00650e08d7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.948 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:23.948 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:23.948 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000065 2023-09-22T21:36:23.948 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000065 2023-09-22T21:36:23.948 DEBUG:boto:Method: GET 2023-09-22T21:36:23.948 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000065 2023-09-22T21:36:23.948 DEBUG:boto:Data: 2023-09-22T21:36:23.949 DEBUG:boto:Headers: {} 2023-09-22T21:36:23.949 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:23.949 DEBUG:boto:Port: 8001 2023-09-22T21:36:23.949 DEBUG:boto:Params: {} 2023-09-22T21:36:23.949 DEBUG:boto:Token: None 2023-09-22T21:36:23.949 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:23 GMT /ffzdqu-58/ 2023-09-22T21:36:23.949 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU= 2023-09-22T21:36:23.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:23 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H9R1xdv2vMTznAOnwuetQTYSYBU=', 'Content-Length': '0'} 2023-09-22T21:36:23.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d612cfa981b7f48-00650e08d7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:23.998 DEBUG:boto:b'ffzdqu-58obj000000000000000000651000falseobj000000000000000000652023-09-22T21:35:12.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:23.999 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:23.999 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:23.999 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000065 2023-09-22T21:36:23.999 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000065 2023-09-22T21:36:23.999 DEBUG:boto:Method: GET 2023-09-22T21:36:23.999 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000065 2023-09-22T21:36:24.000 DEBUG:boto:Data: 2023-09-22T21:36:24.000 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.000 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.000 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.000 DEBUG:boto:Params: {} 2023-09-22T21:36:24.000 DEBUG:boto:Token: None 2023-09-22T21:36:24.000 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/ 2023-09-22T21:36:24.000 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho= 2023-09-22T21:36:24.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho=', 'Content-Length': '0'} 2023-09-22T21:36:24.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084d629677d1a4237-00650e08d7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.010 DEBUG:boto:b'ffzdqu-58obj000000000000000000651000falseobj000000000000000000652023-09-22T21:35:12.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:24.011 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:24.011 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:24.011 DEBUG:boto:Method: HEAD 2023-09-22T21:36:24.011 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:24.011 DEBUG:boto:Data: 2023-09-22T21:36:24.011 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.011 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.011 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.012 DEBUG:boto:Params: {} 2023-09-22T21:36:24.012 DEBUG:boto:Token: None 2023-09-22T21:36:24.012 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:24.012 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+xJqR+a4ARwZKBmGnCo68Ilvrh0= 2023-09-22T21:36:24.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+xJqR+a4ARwZKBmGnCo68Ilvrh0=', 'Content-Length': '0'} 2023-09-22T21:36:24.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e58557e7948a9cfb-00650e08d8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.015 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:24.015 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:24.015 DEBUG:boto:Method: HEAD 2023-09-22T21:36:24.016 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:24.016 DEBUG:boto:Data: 2023-09-22T21:36:24.016 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.016 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.016 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.016 DEBUG:boto:Params: {} 2023-09-22T21:36:24.016 DEBUG:boto:Token: None 2023-09-22T21:36:24.016 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:24.017 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+xJqR+a4ARwZKBmGnCo68Ilvrh0= 2023-09-22T21:36:24.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+xJqR+a4ARwZKBmGnCo68Ilvrh0=', 'Content-Length': '0'} 2023-09-22T21:36:24.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8de53ee1539c733-00650e08d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-09-22T21:36:24.019 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:24.019 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:24.020 DEBUG:boto:Method: GET 2023-09-22T21:36:24.020 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:24.020 DEBUG:boto:Data: 2023-09-22T21:36:24.020 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.020 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.020 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.020 DEBUG:boto:Params: {} 2023-09-22T21:36:24.020 DEBUG:boto:Token: None 2023-09-22T21:36:24.020 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:24.021 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E7uGHFdILoeF8g1emSPogjzrFpE= 2023-09-22T21:36:24.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E7uGHFdILoeF8g1emSPogjzrFpE=', 'Content-Length': '0'} 2023-09-22T21:36:24.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c78c6165046e335e-00650e08d8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.066 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:24.067 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:24.067 DEBUG:boto:Method: GET 2023-09-22T21:36:24.067 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:24.067 DEBUG:boto:Data: 2023-09-22T21:36:24.067 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.067 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.067 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.067 DEBUG:boto:Params: {} 2023-09-22T21:36:24.067 DEBUG:boto:Token: None 2023-09-22T21:36:24.068 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000065 2023-09-22T21:36:24.068 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E7uGHFdILoeF8g1emSPogjzrFpE= 2023-09-22T21:36:24.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E7uGHFdILoeF8g1emSPogjzrFpE=', 'Content-Length': '0'} 2023-09-22T21:36:24.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b12e437e63ec00ff-00650e08d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.071 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:24.071 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:24.071 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000065 2023-09-22T21:36:24.071 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000065 2023-09-22T21:36:24.071 DEBUG:boto:Method: GET 2023-09-22T21:36:24.071 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000065 2023-09-22T21:36:24.072 DEBUG:boto:Data: 2023-09-22T21:36:24.072 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.072 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.072 DEBUG:boto:Params: {} 2023-09-22T21:36:24.072 DEBUG:boto:Token: None 2023-09-22T21:36:24.072 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/ 2023-09-22T21:36:24.072 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho= 2023-09-22T21:36:24.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho=', 'Content-Length': '0'} 2023-09-22T21:36:24.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000114ea516bf36d10d-00650e08d8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.118 DEBUG:boto:b'ffzdqu-58obj000000000000000000651000falseobj000000000000000000652023-09-22T21:35:12.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:24.119 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:24.119 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:24.119 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000065 2023-09-22T21:36:24.119 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000065 2023-09-22T21:36:24.119 DEBUG:boto:Method: GET 2023-09-22T21:36:24.119 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000065 2023-09-22T21:36:24.119 DEBUG:boto:Data: 2023-09-22T21:36:24.119 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.120 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.120 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.120 DEBUG:boto:Params: {} 2023-09-22T21:36:24.120 DEBUG:boto:Token: None 2023-09-22T21:36:24.120 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/ 2023-09-22T21:36:24.120 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho= 2023-09-22T21:36:24.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho=', 'Content-Length': '0'} 2023-09-22T21:36:24.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c4250327a365aea-00650e08d8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.131 DEBUG:boto:b'ffzdqu-58obj000000000000000000651000falseobj000000000000000000652023-09-22T21:35:12.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:24.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2023-09-22T21:36:24.131 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.131 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.131 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.132 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.132 DEBUG:boto:Method: GET 2023-09-22T21:36:24.132 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.132 DEBUG:boto:Data: 2023-09-22T21:36:24.132 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.132 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.132 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.132 DEBUG:boto:Params: {} 2023-09-22T21:36:24.132 DEBUG:boto:Token: None 2023-09-22T21:36:24.133 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.133 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/e/VMsg8jMjMtpkQAH4AXTsNJD4= 2023-09-22T21:36:24.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/e/VMsg8jMjMtpkQAH4AXTsNJD4=', 'Content-Length': '0'} 2023-09-22T21:36:24.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2d5218fe1cb7a50-00650e08d8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.136 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.136 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.136 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.137 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.137 DEBUG:boto:Method: GET 2023-09-22T21:36:24.137 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.137 DEBUG:boto:Data: 2023-09-22T21:36:24.137 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.137 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.137 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.137 DEBUG:boto:Params: {} 2023-09-22T21:36:24.137 DEBUG:boto:Token: None 2023-09-22T21:36:24.138 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.138 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/e/VMsg8jMjMtpkQAH4AXTsNJD4= 2023-09-22T21:36:24.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/e/VMsg8jMjMtpkQAH4AXTsNJD4=', 'Content-Length': '0'} 2023-09-22T21:36:24.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1dabb8217312152-00650e08d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.182 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.183 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.183 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.183 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.183 DEBUG:boto:Method: HEAD 2023-09-22T21:36:24.183 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.183 DEBUG:boto:Data: 2023-09-22T21:36:24.183 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.183 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.183 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.184 DEBUG:boto:Params: {} 2023-09-22T21:36:24.184 DEBUG:boto:Token: None 2023-09-22T21:36:24.184 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.184 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4GgVHcoZqS3T706uGHLCIsMlJd4= 2023-09-22T21:36:24.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4GgVHcoZqS3T706uGHLCIsMlJd4=', 'Content-Length': '0'} 2023-09-22T21:36:24.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009588a13aa401ef55-00650e08d8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.187 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.187 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.187 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.188 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.188 DEBUG:boto:Method: HEAD 2023-09-22T21:36:24.188 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.188 DEBUG:boto:Data: 2023-09-22T21:36:24.188 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.188 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.188 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.188 DEBUG:boto:Params: {} 2023-09-22T21:36:24.188 DEBUG:boto:Token: None 2023-09-22T21:36:24.189 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.189 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4GgVHcoZqS3T706uGHLCIsMlJd4= 2023-09-22T21:36:24.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4GgVHcoZqS3T706uGHLCIsMlJd4=', 'Content-Length': '0'} 2023-09-22T21:36:24.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f80336a355e7464b-00650e08d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2023-09-22T21:36:24.192 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.192 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.192 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.192 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.192 DEBUG:boto:Method: GET 2023-09-22T21:36:24.192 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.192 DEBUG:boto:Data: 2023-09-22T21:36:24.192 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.193 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.193 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.193 DEBUG:boto:Params: {} 2023-09-22T21:36:24.193 DEBUG:boto:Token: None 2023-09-22T21:36:24.193 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.193 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/e/VMsg8jMjMtpkQAH4AXTsNJD4= 2023-09-22T21:36:24.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/e/VMsg8jMjMtpkQAH4AXTsNJD4=', 'Content-Length': '0'} 2023-09-22T21:36:24.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c45d23676c0d4aad-00650e08d8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.242 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.243 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.243 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.243 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.243 DEBUG:boto:Method: GET 2023-09-22T21:36:24.243 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.243 DEBUG:boto:Data: 2023-09-22T21:36:24.243 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.243 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.244 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.244 DEBUG:boto:Params: {} 2023-09-22T21:36:24.244 DEBUG:boto:Token: None 2023-09-22T21:36:24.244 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000076?versionId=null 2023-09-22T21:36:24.244 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/e/VMsg8jMjMtpkQAH4AXTsNJD4= 2023-09-22T21:36:24.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/e/VMsg8jMjMtpkQAH4AXTsNJD4=', 'Content-Length': '0'} 2023-09-22T21:36:24.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029d0c304ea249ca2-00650e08d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.248 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:24.248 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:24.248 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000076 2023-09-22T21:36:24.248 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000076 2023-09-22T21:36:24.248 DEBUG:boto:Method: GET 2023-09-22T21:36:24.249 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000076 2023-09-22T21:36:24.249 DEBUG:boto:Data: 2023-09-22T21:36:24.249 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.249 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.249 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.249 DEBUG:boto:Params: {} 2023-09-22T21:36:24.249 DEBUG:boto:Token: None 2023-09-22T21:36:24.249 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/ 2023-09-22T21:36:24.250 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho= 2023-09-22T21:36:24.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho=', 'Content-Length': '0'} 2023-09-22T21:36:24.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2804170ba93888b-00650e08d8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.294 DEBUG:boto:b'ffzdqu-58obj000000000000000000761000falseobj000000000000000000762023-09-22T21:34:59.536Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:24.295 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:24.295 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:24.295 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000076 2023-09-22T21:36:24.295 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000076 2023-09-22T21:36:24.295 DEBUG:boto:Method: GET 2023-09-22T21:36:24.295 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000076 2023-09-22T21:36:24.296 DEBUG:boto:Data: 2023-09-22T21:36:24.296 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.296 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.296 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.296 DEBUG:boto:Params: {} 2023-09-22T21:36:24.296 DEBUG:boto:Token: None 2023-09-22T21:36:24.296 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/ 2023-09-22T21:36:24.296 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho= 2023-09-22T21:36:24.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho=', 'Content-Length': '0'} 2023-09-22T21:36:24.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d7a4aa5198ca43a-00650e08d8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.308 DEBUG:boto:b'ffzdqu-58obj000000000000000000761000falseobj000000000000000000762023-09-22T21:34:59.536Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:24.309 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.309 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.309 DEBUG:boto:Method: HEAD 2023-09-22T21:36:24.309 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.309 DEBUG:boto:Data: 2023-09-22T21:36:24.309 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.309 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.309 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.310 DEBUG:boto:Params: {} 2023-09-22T21:36:24.310 DEBUG:boto:Token: None 2023-09-22T21:36:24.310 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.310 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3j1GzhvPmLoQaMSJDrne/jMduVw= 2023-09-22T21:36:24.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3j1GzhvPmLoQaMSJDrne/jMduVw=', 'Content-Length': '0'} 2023-09-22T21:36:24.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b7c2977db1ebf4e-00650e08d8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.313 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.313 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.313 DEBUG:boto:Method: HEAD 2023-09-22T21:36:24.313 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.314 DEBUG:boto:Data: 2023-09-22T21:36:24.314 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.314 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.314 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.314 DEBUG:boto:Params: {} 2023-09-22T21:36:24.314 DEBUG:boto:Token: None 2023-09-22T21:36:24.314 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.314 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3j1GzhvPmLoQaMSJDrne/jMduVw= 2023-09-22T21:36:24.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3j1GzhvPmLoQaMSJDrne/jMduVw=', 'Content-Length': '0'} 2023-09-22T21:36:24.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000823d4212fe87e89-00650e08d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2023-09-22T21:36:24.318 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.318 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.318 DEBUG:boto:Method: GET 2023-09-22T21:36:24.318 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.318 DEBUG:boto:Data: 2023-09-22T21:36:24.318 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.318 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.318 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.318 DEBUG:boto:Params: {} 2023-09-22T21:36:24.319 DEBUG:boto:Token: None 2023-09-22T21:36:24.319 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.319 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fLS+TjisC47CTHYa6cck8u+bgBI= 2023-09-22T21:36:24.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fLS+TjisC47CTHYa6cck8u+bgBI=', 'Content-Length': '0'} 2023-09-22T21:36:24.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dbcbdb74da46fc1-00650e08d8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.366 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.367 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.367 DEBUG:boto:Method: GET 2023-09-22T21:36:24.367 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.367 DEBUG:boto:Data: 2023-09-22T21:36:24.367 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.367 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.367 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.367 DEBUG:boto:Params: {} 2023-09-22T21:36:24.368 DEBUG:boto:Token: None 2023-09-22T21:36:24.368 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000076 2023-09-22T21:36:24.368 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fLS+TjisC47CTHYa6cck8u+bgBI= 2023-09-22T21:36:24.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fLS+TjisC47CTHYa6cck8u+bgBI=', 'Content-Length': '0'} 2023-09-22T21:36:24.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:34:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ace3d9329205239-00650e08d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.371 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:24.371 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:24.371 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000076 2023-09-22T21:36:24.372 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000076 2023-09-22T21:36:24.372 DEBUG:boto:Method: GET 2023-09-22T21:36:24.372 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000076 2023-09-22T21:36:24.372 DEBUG:boto:Data: 2023-09-22T21:36:24.372 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.372 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.372 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.372 DEBUG:boto:Params: {} 2023-09-22T21:36:24.373 DEBUG:boto:Token: None 2023-09-22T21:36:24.373 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/ 2023-09-22T21:36:24.373 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho= 2023-09-22T21:36:24.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho=', 'Content-Length': '0'} 2023-09-22T21:36:24.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcaffb7f58946436-00650e08d8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.378 DEBUG:boto:b'ffzdqu-58obj000000000000000000761000falseobj000000000000000000762023-09-22T21:34:59.536Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:24.378 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:24.378 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:24.378 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000076 2023-09-22T21:36:24.379 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000076 2023-09-22T21:36:24.379 DEBUG:boto:Method: GET 2023-09-22T21:36:24.379 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000076 2023-09-22T21:36:24.379 DEBUG:boto:Data: 2023-09-22T21:36:24.379 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.379 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.379 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.379 DEBUG:boto:Params: {} 2023-09-22T21:36:24.380 DEBUG:boto:Token: None 2023-09-22T21:36:24.380 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/ 2023-09-22T21:36:24.380 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho= 2023-09-22T21:36:24.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho=', 'Content-Length': '0'} 2023-09-22T21:36:24.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062fa057e0d16be30-00650e08d8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.430 DEBUG:boto:b'ffzdqu-58obj000000000000000000761000falseobj000000000000000000762023-09-22T21:34:59.536Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:24.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-09-22T21:36:24.431 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.431 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.431 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.431 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.431 DEBUG:boto:Method: GET 2023-09-22T21:36:24.432 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.432 DEBUG:boto:Data: 2023-09-22T21:36:24.432 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.432 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.432 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.432 DEBUG:boto:Params: {} 2023-09-22T21:36:24.432 DEBUG:boto:Token: None 2023-09-22T21:36:24.432 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.433 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b9Q9WekTjMgJPMn6q3p4iPIrYxg= 2023-09-22T21:36:24.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b9Q9WekTjMgJPMn6q3p4iPIrYxg=', 'Content-Length': '0'} 2023-09-22T21:36:24.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce06b647fd3aeca1-00650e08d8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.482 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.483 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.483 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.483 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.483 DEBUG:boto:Method: GET 2023-09-22T21:36:24.483 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.483 DEBUG:boto:Data: 2023-09-22T21:36:24.483 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.484 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.484 DEBUG:boto:Params: {} 2023-09-22T21:36:24.484 DEBUG:boto:Token: None 2023-09-22T21:36:24.484 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.484 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b9Q9WekTjMgJPMn6q3p4iPIrYxg= 2023-09-22T21:36:24.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b9Q9WekTjMgJPMn6q3p4iPIrYxg=', 'Content-Length': '0'} 2023-09-22T21:36:24.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027c2606cf5885296-00650e08d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.487 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.488 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.488 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.488 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.488 DEBUG:boto:Method: HEAD 2023-09-22T21:36:24.488 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.488 DEBUG:boto:Data: 2023-09-22T21:36:24.488 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.488 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.488 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.489 DEBUG:boto:Params: {} 2023-09-22T21:36:24.489 DEBUG:boto:Token: None 2023-09-22T21:36:24.489 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.489 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ztG03lJQXI9l62r1mKOwD45su8Y= 2023-09-22T21:36:24.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ztG03lJQXI9l62r1mKOwD45su8Y=', 'Content-Length': '0'} 2023-09-22T21:36:24.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004909e9bfaf7be15e-00650e08d8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.492 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.492 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.492 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.492 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.493 DEBUG:boto:Method: HEAD 2023-09-22T21:36:24.493 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.493 DEBUG:boto:Data: 2023-09-22T21:36:24.493 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.493 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.493 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.493 DEBUG:boto:Params: {} 2023-09-22T21:36:24.493 DEBUG:boto:Token: None 2023-09-22T21:36:24.493 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.494 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ztG03lJQXI9l62r1mKOwD45su8Y= 2023-09-22T21:36:24.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ztG03lJQXI9l62r1mKOwD45su8Y=', 'Content-Length': '0'} 2023-09-22T21:36:24.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bc204c60e887a6b-00650e08d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-09-22T21:36:24.497 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.497 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.497 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.497 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.497 DEBUG:boto:Method: GET 2023-09-22T21:36:24.497 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.497 DEBUG:boto:Data: 2023-09-22T21:36:24.497 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.497 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.498 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.498 DEBUG:boto:Params: {} 2023-09-22T21:36:24.498 DEBUG:boto:Token: None 2023-09-22T21:36:24.498 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.498 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b9Q9WekTjMgJPMn6q3p4iPIrYxg= 2023-09-22T21:36:24.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b9Q9WekTjMgJPMn6q3p4iPIrYxg=', 'Content-Length': '0'} 2023-09-22T21:36:24.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008067ec8e4c195a4c-00650e08d8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.542 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.543 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.543 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.543 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.543 DEBUG:boto:Method: GET 2023-09-22T21:36:24.543 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.543 DEBUG:boto:Data: 2023-09-22T21:36:24.543 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.544 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.544 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.544 DEBUG:boto:Params: {} 2023-09-22T21:36:24.544 DEBUG:boto:Token: None 2023-09-22T21:36:24.544 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000133?versionId=null 2023-09-22T21:36:24.544 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b9Q9WekTjMgJPMn6q3p4iPIrYxg= 2023-09-22T21:36:24.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b9Q9WekTjMgJPMn6q3p4iPIrYxg=', 'Content-Length': '0'} 2023-09-22T21:36:24.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021e7bd4e62dc489e-00650e08d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.548 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:24.548 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:24.548 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000133 2023-09-22T21:36:24.548 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000133 2023-09-22T21:36:24.548 DEBUG:boto:Method: GET 2023-09-22T21:36:24.548 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000133 2023-09-22T21:36:24.548 DEBUG:boto:Data: 2023-09-22T21:36:24.548 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.549 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.549 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.549 DEBUG:boto:Params: {} 2023-09-22T21:36:24.549 DEBUG:boto:Token: None 2023-09-22T21:36:24.549 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/ 2023-09-22T21:36:24.549 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho= 2023-09-22T21:36:24.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho=', 'Content-Length': '0'} 2023-09-22T21:36:24.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3d3a883ea6d59dd-00650e08d8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.594 DEBUG:boto:b'ffzdqu-58obj000000000000000001331000falseobj000000000000000001332023-09-22T21:35:03.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:24.595 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:24.595 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:24.595 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000133 2023-09-22T21:36:24.595 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000133 2023-09-22T21:36:24.595 DEBUG:boto:Method: GET 2023-09-22T21:36:24.596 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000133 2023-09-22T21:36:24.596 DEBUG:boto:Data: 2023-09-22T21:36:24.596 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.596 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.596 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.596 DEBUG:boto:Params: {} 2023-09-22T21:36:24.596 DEBUG:boto:Token: None 2023-09-22T21:36:24.596 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/ 2023-09-22T21:36:24.596 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho= 2023-09-22T21:36:24.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho=', 'Content-Length': '0'} 2023-09-22T21:36:24.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001eac38fb7ef18c4f-00650e08d8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.608 DEBUG:boto:b'ffzdqu-58obj000000000000000001331000falseobj000000000000000001332023-09-22T21:35:03.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:24.608 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.608 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.608 DEBUG:boto:Method: HEAD 2023-09-22T21:36:24.608 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.608 DEBUG:boto:Data: 2023-09-22T21:36:24.609 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.609 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.609 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.609 DEBUG:boto:Params: {} 2023-09-22T21:36:24.609 DEBUG:boto:Token: None 2023-09-22T21:36:24.609 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.609 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hm+qj0FORD+z0VBUcINW1RGuE04= 2023-09-22T21:36:24.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hm+qj0FORD+z0VBUcINW1RGuE04=', 'Content-Length': '0'} 2023-09-22T21:36:24.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003be4bb3adbee10c9-00650e08d8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.612 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.612 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.613 DEBUG:boto:Method: HEAD 2023-09-22T21:36:24.613 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.613 DEBUG:boto:Data: 2023-09-22T21:36:24.613 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.613 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.613 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.613 DEBUG:boto:Params: {} 2023-09-22T21:36:24.613 DEBUG:boto:Token: None 2023-09-22T21:36:24.613 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.614 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hm+qj0FORD+z0VBUcINW1RGuE04= 2023-09-22T21:36:24.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hm+qj0FORD+z0VBUcINW1RGuE04=', 'Content-Length': '0'} 2023-09-22T21:36:24.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007effaa9e0a3837d4-00650e08d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-09-22T21:36:24.617 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.617 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.617 DEBUG:boto:Method: GET 2023-09-22T21:36:24.617 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.617 DEBUG:boto:Data: 2023-09-22T21:36:24.617 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.617 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.617 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.617 DEBUG:boto:Params: {} 2023-09-22T21:36:24.618 DEBUG:boto:Token: None 2023-09-22T21:36:24.618 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.618 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ru1tNPOc9/k0Nc3smgWyymNM+Cg= 2023-09-22T21:36:24.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ru1tNPOc9/k0Nc3smgWyymNM+Cg=', 'Content-Length': '0'} 2023-09-22T21:36:24.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a264fd361ec6875f-00650e08d8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.662 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.663 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.663 DEBUG:boto:Method: GET 2023-09-22T21:36:24.663 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.663 DEBUG:boto:Data: 2023-09-22T21:36:24.663 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.663 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.663 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.663 DEBUG:boto:Params: {} 2023-09-22T21:36:24.663 DEBUG:boto:Token: None 2023-09-22T21:36:24.664 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000133 2023-09-22T21:36:24.664 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ru1tNPOc9/k0Nc3smgWyymNM+Cg= 2023-09-22T21:36:24.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ru1tNPOc9/k0Nc3smgWyymNM+Cg=', 'Content-Length': '0'} 2023-09-22T21:36:24.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bac03a0959466ffe-00650e08d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.667 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:24.667 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:24.667 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000133 2023-09-22T21:36:24.667 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000133 2023-09-22T21:36:24.667 DEBUG:boto:Method: GET 2023-09-22T21:36:24.667 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000133 2023-09-22T21:36:24.668 DEBUG:boto:Data: 2023-09-22T21:36:24.668 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.668 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.668 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.668 DEBUG:boto:Params: {} 2023-09-22T21:36:24.668 DEBUG:boto:Token: None 2023-09-22T21:36:24.668 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/ 2023-09-22T21:36:24.668 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho= 2023-09-22T21:36:24.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho=', 'Content-Length': '0'} 2023-09-22T21:36:24.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000322d6fa641453065-00650e08d8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.718 DEBUG:boto:b'ffzdqu-58obj000000000000000001331000falseobj000000000000000001332023-09-22T21:35:03.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:24.719 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:24.719 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:24.719 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000133 2023-09-22T21:36:24.719 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000133 2023-09-22T21:36:24.719 DEBUG:boto:Method: GET 2023-09-22T21:36:24.719 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000133 2023-09-22T21:36:24.720 DEBUG:boto:Data: 2023-09-22T21:36:24.720 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.720 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.720 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.720 DEBUG:boto:Params: {} 2023-09-22T21:36:24.720 DEBUG:boto:Token: None 2023-09-22T21:36:24.720 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/ 2023-09-22T21:36:24.720 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho= 2023-09-22T21:36:24.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho=', 'Content-Length': '0'} 2023-09-22T21:36:24.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056cbfba6f8be92e6-00650e08d8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.731 DEBUG:boto:b'ffzdqu-58obj000000000000000001331000falseobj000000000000000001332023-09-22T21:35:03.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:24.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-09-22T21:36:24.731 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.732 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.732 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.732 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.732 DEBUG:boto:Method: GET 2023-09-22T21:36:24.732 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.732 DEBUG:boto:Data: 2023-09-22T21:36:24.732 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.732 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.732 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.733 DEBUG:boto:Params: {} 2023-09-22T21:36:24.733 DEBUG:boto:Token: None 2023-09-22T21:36:24.733 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.733 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t8XTmbegCZpd0n2LwQZ1DfBg2rc= 2023-09-22T21:36:24.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t8XTmbegCZpd0n2LwQZ1DfBg2rc=', 'Content-Length': '0'} 2023-09-22T21:36:24.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049497c7a49f1bb08-00650e08d8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.737 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.737 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.737 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.737 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.737 DEBUG:boto:Method: GET 2023-09-22T21:36:24.737 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.737 DEBUG:boto:Data: 2023-09-22T21:36:24.737 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.738 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.738 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.738 DEBUG:boto:Params: {} 2023-09-22T21:36:24.738 DEBUG:boto:Token: None 2023-09-22T21:36:24.738 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.738 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t8XTmbegCZpd0n2LwQZ1DfBg2rc= 2023-09-22T21:36:24.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t8XTmbegCZpd0n2LwQZ1DfBg2rc=', 'Content-Length': '0'} 2023-09-22T21:36:24.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa88992bb1e0452f-00650e08d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.786 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.787 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.787 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.787 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.787 DEBUG:boto:Method: HEAD 2023-09-22T21:36:24.787 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.787 DEBUG:boto:Data: 2023-09-22T21:36:24.787 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.787 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.787 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.787 DEBUG:boto:Params: {} 2023-09-22T21:36:24.788 DEBUG:boto:Token: None 2023-09-22T21:36:24.788 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.788 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ezyOdRDdwwrTyoi1apCeFZoN9Cg= 2023-09-22T21:36:24.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ezyOdRDdwwrTyoi1apCeFZoN9Cg=', 'Content-Length': '0'} 2023-09-22T21:36:24.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7e55aa2c5b10a0f-00650e08d8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.791 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.791 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.791 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.792 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.792 DEBUG:boto:Method: HEAD 2023-09-22T21:36:24.792 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.792 DEBUG:boto:Data: 2023-09-22T21:36:24.792 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.792 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.792 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.792 DEBUG:boto:Params: {} 2023-09-22T21:36:24.792 DEBUG:boto:Token: None 2023-09-22T21:36:24.793 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.793 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ezyOdRDdwwrTyoi1apCeFZoN9Cg= 2023-09-22T21:36:24.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ezyOdRDdwwrTyoi1apCeFZoN9Cg=', 'Content-Length': '0'} 2023-09-22T21:36:24.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ddd008dd92b8563-00650e08d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-09-22T21:36:24.796 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.796 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.796 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.796 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.796 DEBUG:boto:Method: GET 2023-09-22T21:36:24.797 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.797 DEBUG:boto:Data: 2023-09-22T21:36:24.797 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.797 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.797 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.797 DEBUG:boto:Params: {} 2023-09-22T21:36:24.797 DEBUG:boto:Token: None 2023-09-22T21:36:24.797 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.797 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t8XTmbegCZpd0n2LwQZ1DfBg2rc= 2023-09-22T21:36:24.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t8XTmbegCZpd0n2LwQZ1DfBg2rc=', 'Content-Length': '0'} 2023-09-22T21:36:24.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d08f456119ee1774-00650e08d8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.842 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.842 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.843 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.843 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.843 DEBUG:boto:Method: GET 2023-09-22T21:36:24.843 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.843 DEBUG:boto:Data: 2023-09-22T21:36:24.843 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.843 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.843 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.843 DEBUG:boto:Params: {} 2023-09-22T21:36:24.844 DEBUG:boto:Token: None 2023-09-22T21:36:24.844 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000271?versionId=null 2023-09-22T21:36:24.844 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t8XTmbegCZpd0n2LwQZ1DfBg2rc= 2023-09-22T21:36:24.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t8XTmbegCZpd0n2LwQZ1DfBg2rc=', 'Content-Length': '0'} 2023-09-22T21:36:24.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc902d54c8e36ec5-00650e08d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.847 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:24.847 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:24.848 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000271 2023-09-22T21:36:24.848 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000271 2023-09-22T21:36:24.848 DEBUG:boto:Method: GET 2023-09-22T21:36:24.848 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000271 2023-09-22T21:36:24.848 DEBUG:boto:Data: 2023-09-22T21:36:24.848 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.848 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.848 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.848 DEBUG:boto:Params: {} 2023-09-22T21:36:24.849 DEBUG:boto:Token: None 2023-09-22T21:36:24.849 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/ 2023-09-22T21:36:24.849 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho= 2023-09-22T21:36:24.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho=', 'Content-Length': '0'} 2023-09-22T21:36:24.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032f26fda4951dc37-00650e08d8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.894 DEBUG:boto:b'ffzdqu-58obj000000000000000002711000falseobj000000000000000002712023-09-22T21:35:03.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:24.895 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:24.895 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:24.895 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000271 2023-09-22T21:36:24.895 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000271 2023-09-22T21:36:24.895 DEBUG:boto:Method: GET 2023-09-22T21:36:24.895 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000271 2023-09-22T21:36:24.895 DEBUG:boto:Data: 2023-09-22T21:36:24.896 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.896 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.896 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.896 DEBUG:boto:Params: {} 2023-09-22T21:36:24.896 DEBUG:boto:Token: None 2023-09-22T21:36:24.896 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/ 2023-09-22T21:36:24.896 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho= 2023-09-22T21:36:24.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho=', 'Content-Length': '0'} 2023-09-22T21:36:24.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007210a776fe840aa4-00650e08d8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.914 DEBUG:boto:b'ffzdqu-58obj000000000000000002711000falseobj000000000000000002712023-09-22T21:35:03.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:24.914 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.914 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.915 DEBUG:boto:Method: HEAD 2023-09-22T21:36:24.915 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.915 DEBUG:boto:Data: 2023-09-22T21:36:24.915 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.915 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.915 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.915 DEBUG:boto:Params: {} 2023-09-22T21:36:24.915 DEBUG:boto:Token: None 2023-09-22T21:36:24.915 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.916 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1WNFIe1Rx2xHdgl7KrNfysbQCYg= 2023-09-22T21:36:24.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1WNFIe1Rx2xHdgl7KrNfysbQCYg=', 'Content-Length': '0'} 2023-09-22T21:36:24.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043c181ba9b533b81-00650e08d8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.918 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.919 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.919 DEBUG:boto:Method: HEAD 2023-09-22T21:36:24.919 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.919 DEBUG:boto:Data: 2023-09-22T21:36:24.919 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.919 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.919 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.919 DEBUG:boto:Params: {} 2023-09-22T21:36:24.919 DEBUG:boto:Token: None 2023-09-22T21:36:24.920 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.920 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1WNFIe1Rx2xHdgl7KrNfysbQCYg= 2023-09-22T21:36:24.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1WNFIe1Rx2xHdgl7KrNfysbQCYg=', 'Content-Length': '0'} 2023-09-22T21:36:24.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086eeaa85a241f109-00650e08d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-09-22T21:36:24.923 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.923 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.923 DEBUG:boto:Method: GET 2023-09-22T21:36:24.923 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.923 DEBUG:boto:Data: 2023-09-22T21:36:24.923 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.923 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.924 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.924 DEBUG:boto:Params: {} 2023-09-22T21:36:24.924 DEBUG:boto:Token: None 2023-09-22T21:36:24.924 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.924 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kLUN0+hIB6ri0VsiFMZHEV1/ZQg= 2023-09-22T21:36:24.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kLUN0+hIB6ri0VsiFMZHEV1/ZQg=', 'Content-Length': '0'} 2023-09-22T21:36:24.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b270ceab4e70ef7-00650e08d8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.970 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.970 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.971 DEBUG:boto:Method: GET 2023-09-22T21:36:24.971 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.971 DEBUG:boto:Data: 2023-09-22T21:36:24.971 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.971 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:24.971 DEBUG:boto:Port: 8000 2023-09-22T21:36:24.971 DEBUG:boto:Params: {} 2023-09-22T21:36:24.971 DEBUG:boto:Token: None 2023-09-22T21:36:24.972 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/obj00000000000000000271 2023-09-22T21:36:24.972 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kLUN0+hIB6ri0VsiFMZHEV1/ZQg= 2023-09-22T21:36:24.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kLUN0+hIB6ri0VsiFMZHEV1/ZQg=', 'Content-Length': '0'} 2023-09-22T21:36:24.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdb801067fb96f2b-00650e08d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:24.975 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:24.975 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:24.976 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000271 2023-09-22T21:36:24.976 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000271 2023-09-22T21:36:24.976 DEBUG:boto:Method: GET 2023-09-22T21:36:24.976 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000271 2023-09-22T21:36:24.976 DEBUG:boto:Data: 2023-09-22T21:36:24.976 DEBUG:boto:Headers: {} 2023-09-22T21:36:24.976 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:24.976 DEBUG:boto:Port: 8001 2023-09-22T21:36:24.976 DEBUG:boto:Params: {} 2023-09-22T21:36:24.976 DEBUG:boto:Token: None 2023-09-22T21:36:24.977 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:24 GMT /ffzdqu-58/ 2023-09-22T21:36:24.977 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho= 2023-09-22T21:36:24.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:24 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8fxbXVptYnAHx3sBqnjwisgz7Ho=', 'Content-Length': '0'} 2023-09-22T21:36:24.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f9b87bc196ac200-00650e08d8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.022 DEBUG:boto:b'ffzdqu-58obj000000000000000002711000falseobj000000000000000002712023-09-22T21:35:03.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:25.023 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:25.023 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:25.023 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000271 2023-09-22T21:36:25.023 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000271 2023-09-22T21:36:25.023 DEBUG:boto:Method: GET 2023-09-22T21:36:25.023 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000271 2023-09-22T21:36:25.024 DEBUG:boto:Data: 2023-09-22T21:36:25.024 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.024 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.024 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.024 DEBUG:boto:Params: {} 2023-09-22T21:36:25.024 DEBUG:boto:Token: None 2023-09-22T21:36:25.024 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/ 2023-09-22T21:36:25.024 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec= 2023-09-22T21:36:25.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec=', 'Content-Length': '0'} 2023-09-22T21:36:25.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b78cf9f432d0d9f-00650e08d9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.035 DEBUG:boto:b'ffzdqu-58obj000000000000000002711000falseobj000000000000000002712023-09-22T21:35:03.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:25.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-09-22T21:36:25.036 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.036 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.036 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.036 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.036 DEBUG:boto:Method: GET 2023-09-22T21:36:25.036 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.036 DEBUG:boto:Data: 2023-09-22T21:36:25.036 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.037 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.037 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.037 DEBUG:boto:Params: {} 2023-09-22T21:36:25.037 DEBUG:boto:Token: None 2023-09-22T21:36:25.037 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.037 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gMzAj1XrEDrPxH2YEvs/i4cGuRQ= 2023-09-22T21:36:25.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gMzAj1XrEDrPxH2YEvs/i4cGuRQ=', 'Content-Length': '0'} 2023-09-22T21:36:25.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002deb9a73d624b3ee-00650e08d9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.041 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.041 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.041 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.041 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.041 DEBUG:boto:Method: GET 2023-09-22T21:36:25.041 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.041 DEBUG:boto:Data: 2023-09-22T21:36:25.041 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.042 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.042 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.042 DEBUG:boto:Params: {} 2023-09-22T21:36:25.042 DEBUG:boto:Token: None 2023-09-22T21:36:25.042 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.042 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gMzAj1XrEDrPxH2YEvs/i4cGuRQ= 2023-09-22T21:36:25.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gMzAj1XrEDrPxH2YEvs/i4cGuRQ=', 'Content-Length': '0'} 2023-09-22T21:36:25.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030270efde7f7041c-00650e08d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.086 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.086 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.087 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.087 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.087 DEBUG:boto:Method: HEAD 2023-09-22T21:36:25.087 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.087 DEBUG:boto:Data: 2023-09-22T21:36:25.087 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.087 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.087 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.087 DEBUG:boto:Params: {} 2023-09-22T21:36:25.088 DEBUG:boto:Token: None 2023-09-22T21:36:25.088 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.088 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HlSJWmdDQ+FIbepuiO5rzbnblao= 2023-09-22T21:36:25.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HlSJWmdDQ+FIbepuiO5rzbnblao=', 'Content-Length': '0'} 2023-09-22T21:36:25.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dd2a93a4b80376d-00650e08d9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.091 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.091 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.091 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.091 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.091 DEBUG:boto:Method: HEAD 2023-09-22T21:36:25.092 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.092 DEBUG:boto:Data: 2023-09-22T21:36:25.092 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.092 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.092 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.092 DEBUG:boto:Params: {} 2023-09-22T21:36:25.092 DEBUG:boto:Token: None 2023-09-22T21:36:25.092 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.093 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HlSJWmdDQ+FIbepuiO5rzbnblao= 2023-09-22T21:36:25.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HlSJWmdDQ+FIbepuiO5rzbnblao=', 'Content-Length': '0'} 2023-09-22T21:36:25.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa2e437b5dea6ff3-00650e08d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-09-22T21:36:25.096 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.096 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.096 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.096 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.096 DEBUG:boto:Method: GET 2023-09-22T21:36:25.096 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.096 DEBUG:boto:Data: 2023-09-22T21:36:25.097 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.097 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.097 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.097 DEBUG:boto:Params: {} 2023-09-22T21:36:25.097 DEBUG:boto:Token: None 2023-09-22T21:36:25.097 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.097 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gMzAj1XrEDrPxH2YEvs/i4cGuRQ= 2023-09-22T21:36:25.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gMzAj1XrEDrPxH2YEvs/i4cGuRQ=', 'Content-Length': '0'} 2023-09-22T21:36:25.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025f5334ff2236f5b-00650e08d9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.146 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.146 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.147 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.147 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.147 DEBUG:boto:Method: GET 2023-09-22T21:36:25.147 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.147 DEBUG:boto:Data: 2023-09-22T21:36:25.147 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.147 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.147 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.148 DEBUG:boto:Params: {} 2023-09-22T21:36:25.148 DEBUG:boto:Token: None 2023-09-22T21:36:25.148 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000283?versionId=null 2023-09-22T21:36:25.148 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gMzAj1XrEDrPxH2YEvs/i4cGuRQ= 2023-09-22T21:36:25.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gMzAj1XrEDrPxH2YEvs/i4cGuRQ=', 'Content-Length': '0'} 2023-09-22T21:36:25.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009744f0b9811ee8d3-00650e08d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.151 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:25.151 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:25.151 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000283 2023-09-22T21:36:25.152 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000283 2023-09-22T21:36:25.152 DEBUG:boto:Method: GET 2023-09-22T21:36:25.152 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000283 2023-09-22T21:36:25.152 DEBUG:boto:Data: 2023-09-22T21:36:25.152 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.152 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.152 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.152 DEBUG:boto:Params: {} 2023-09-22T21:36:25.152 DEBUG:boto:Token: None 2023-09-22T21:36:25.153 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/ 2023-09-22T21:36:25.153 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec= 2023-09-22T21:36:25.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec=', 'Content-Length': '0'} 2023-09-22T21:36:25.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000899770b34474856a-00650e08d9-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.198 DEBUG:boto:b'ffzdqu-58obj000000000000000002831000falseobj000000000000000002832023-09-22T21:35:04.277Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:25.199 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:25.199 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:25.199 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000283 2023-09-22T21:36:25.199 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000283 2023-09-22T21:36:25.199 DEBUG:boto:Method: GET 2023-09-22T21:36:25.199 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000283 2023-09-22T21:36:25.199 DEBUG:boto:Data: 2023-09-22T21:36:25.199 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.200 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.200 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.200 DEBUG:boto:Params: {} 2023-09-22T21:36:25.200 DEBUG:boto:Token: None 2023-09-22T21:36:25.200 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/ 2023-09-22T21:36:25.200 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec= 2023-09-22T21:36:25.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec=', 'Content-Length': '0'} 2023-09-22T21:36:25.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a36eebef80ce768f-00650e08d9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.210 DEBUG:boto:b'ffzdqu-58obj000000000000000002831000falseobj000000000000000002832023-09-22T21:35:04.277Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:25.211 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.211 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.211 DEBUG:boto:Method: HEAD 2023-09-22T21:36:25.211 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.211 DEBUG:boto:Data: 2023-09-22T21:36:25.211 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.211 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.211 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.212 DEBUG:boto:Params: {} 2023-09-22T21:36:25.212 DEBUG:boto:Token: None 2023-09-22T21:36:25.212 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.212 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AToIaMAn0aNfvHbhxkMzhvakbHs= 2023-09-22T21:36:25.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AToIaMAn0aNfvHbhxkMzhvakbHs=', 'Content-Length': '0'} 2023-09-22T21:36:25.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ed0d2e2c4477fe0-00650e08d9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.215 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.215 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.216 DEBUG:boto:Method: HEAD 2023-09-22T21:36:25.216 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.216 DEBUG:boto:Data: 2023-09-22T21:36:25.216 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.216 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.216 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.216 DEBUG:boto:Params: {} 2023-09-22T21:36:25.216 DEBUG:boto:Token: None 2023-09-22T21:36:25.216 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.217 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AToIaMAn0aNfvHbhxkMzhvakbHs= 2023-09-22T21:36:25.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AToIaMAn0aNfvHbhxkMzhvakbHs=', 'Content-Length': '0'} 2023-09-22T21:36:25.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000697c9ded42d7a1f3-00650e08d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-09-22T21:36:25.220 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.220 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.220 DEBUG:boto:Method: GET 2023-09-22T21:36:25.220 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.220 DEBUG:boto:Data: 2023-09-22T21:36:25.220 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.221 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.221 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.221 DEBUG:boto:Params: {} 2023-09-22T21:36:25.221 DEBUG:boto:Token: None 2023-09-22T21:36:25.221 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.221 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P4kKjfRwm+bYGiY8B46DBLe9Nr0= 2023-09-22T21:36:25.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P4kKjfRwm+bYGiY8B46DBLe9Nr0=', 'Content-Length': '0'} 2023-09-22T21:36:25.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e754bff33cbc94a9-00650e08d9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.270 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.271 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.271 DEBUG:boto:Method: GET 2023-09-22T21:36:25.271 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.271 DEBUG:boto:Data: 2023-09-22T21:36:25.271 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.271 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.271 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.271 DEBUG:boto:Params: {} 2023-09-22T21:36:25.271 DEBUG:boto:Token: None 2023-09-22T21:36:25.272 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000283 2023-09-22T21:36:25.272 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P4kKjfRwm+bYGiY8B46DBLe9Nr0= 2023-09-22T21:36:25.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P4kKjfRwm+bYGiY8B46DBLe9Nr0=', 'Content-Length': '0'} 2023-09-22T21:36:25.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2428f7036a71e1f-00650e08d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.275 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:25.275 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:25.275 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000283 2023-09-22T21:36:25.275 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000283 2023-09-22T21:36:25.276 DEBUG:boto:Method: GET 2023-09-22T21:36:25.276 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000283 2023-09-22T21:36:25.276 DEBUG:boto:Data: 2023-09-22T21:36:25.276 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.276 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.276 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.276 DEBUG:boto:Params: {} 2023-09-22T21:36:25.276 DEBUG:boto:Token: None 2023-09-22T21:36:25.277 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/ 2023-09-22T21:36:25.277 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec= 2023-09-22T21:36:25.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec=', 'Content-Length': '0'} 2023-09-22T21:36:25.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fefd1b373a2e163-00650e08d9-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.326 DEBUG:boto:b'ffzdqu-58obj000000000000000002831000falseobj000000000000000002832023-09-22T21:35:04.277Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:25.327 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:25.327 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:25.327 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000283 2023-09-22T21:36:25.327 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000283 2023-09-22T21:36:25.327 DEBUG:boto:Method: GET 2023-09-22T21:36:25.327 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000283 2023-09-22T21:36:25.327 DEBUG:boto:Data: 2023-09-22T21:36:25.328 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.328 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.328 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.328 DEBUG:boto:Params: {} 2023-09-22T21:36:25.328 DEBUG:boto:Token: None 2023-09-22T21:36:25.328 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/ 2023-09-22T21:36:25.328 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec= 2023-09-22T21:36:25.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec=', 'Content-Length': '0'} 2023-09-22T21:36:25.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031820333d7393f1c-00650e08d9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.339 DEBUG:boto:b'ffzdqu-58obj000000000000000002831000falseobj000000000000000002832023-09-22T21:35:04.277Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:25.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-09-22T21:36:25.339 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.340 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.340 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.340 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.340 DEBUG:boto:Method: GET 2023-09-22T21:36:25.340 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.340 DEBUG:boto:Data: 2023-09-22T21:36:25.340 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.340 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.340 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.340 DEBUG:boto:Params: {} 2023-09-22T21:36:25.341 DEBUG:boto:Token: None 2023-09-22T21:36:25.341 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.341 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5Hb1A3k/l7vGhPIkl4pLfQYcLVM= 2023-09-22T21:36:25.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5Hb1A3k/l7vGhPIkl4pLfQYcLVM=', 'Content-Length': '0'} 2023-09-22T21:36:25.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000036df18d40633eec-00650e08d9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.344 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.344 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.345 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.345 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.345 DEBUG:boto:Method: GET 2023-09-22T21:36:25.345 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.345 DEBUG:boto:Data: 2023-09-22T21:36:25.345 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.345 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.345 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.345 DEBUG:boto:Params: {} 2023-09-22T21:36:25.346 DEBUG:boto:Token: None 2023-09-22T21:36:25.346 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.346 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5Hb1A3k/l7vGhPIkl4pLfQYcLVM= 2023-09-22T21:36:25.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5Hb1A3k/l7vGhPIkl4pLfQYcLVM=', 'Content-Length': '0'} 2023-09-22T21:36:25.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf2c44b869f9181d-00650e08d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.394 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.394 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.395 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.395 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.395 DEBUG:boto:Method: HEAD 2023-09-22T21:36:25.395 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.395 DEBUG:boto:Data: 2023-09-22T21:36:25.395 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.395 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.395 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.395 DEBUG:boto:Params: {} 2023-09-22T21:36:25.395 DEBUG:boto:Token: None 2023-09-22T21:36:25.396 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.396 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f8p6JEIDkZvwXPCm1F9PxrVL4rs= 2023-09-22T21:36:25.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f8p6JEIDkZvwXPCm1F9PxrVL4rs=', 'Content-Length': '0'} 2023-09-22T21:36:25.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000084ffd6b55cd4c1a-00650e08d9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.399 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.399 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.399 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.399 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.399 DEBUG:boto:Method: HEAD 2023-09-22T21:36:25.399 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.399 DEBUG:boto:Data: 2023-09-22T21:36:25.400 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.400 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.400 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.400 DEBUG:boto:Params: {} 2023-09-22T21:36:25.400 DEBUG:boto:Token: None 2023-09-22T21:36:25.400 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.400 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f8p6JEIDkZvwXPCm1F9PxrVL4rs= 2023-09-22T21:36:25.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f8p6JEIDkZvwXPCm1F9PxrVL4rs=', 'Content-Length': '0'} 2023-09-22T21:36:25.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ea6063bb12f96f7-00650e08d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-09-22T21:36:25.403 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.404 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.404 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.404 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.404 DEBUG:boto:Method: GET 2023-09-22T21:36:25.404 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.404 DEBUG:boto:Data: 2023-09-22T21:36:25.404 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.404 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.404 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.404 DEBUG:boto:Params: {} 2023-09-22T21:36:25.405 DEBUG:boto:Token: None 2023-09-22T21:36:25.405 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.405 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5Hb1A3k/l7vGhPIkl4pLfQYcLVM= 2023-09-22T21:36:25.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5Hb1A3k/l7vGhPIkl4pLfQYcLVM=', 'Content-Length': '0'} 2023-09-22T21:36:25.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b951cf89eedc672b-00650e08d9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.450 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.450 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.451 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.451 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.451 DEBUG:boto:Method: GET 2023-09-22T21:36:25.451 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.451 DEBUG:boto:Data: 2023-09-22T21:36:25.451 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.451 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.451 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.451 DEBUG:boto:Params: {} 2023-09-22T21:36:25.452 DEBUG:boto:Token: None 2023-09-22T21:36:25.452 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000317?versionId=null 2023-09-22T21:36:25.452 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5Hb1A3k/l7vGhPIkl4pLfQYcLVM= 2023-09-22T21:36:25.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5Hb1A3k/l7vGhPIkl4pLfQYcLVM=', 'Content-Length': '0'} 2023-09-22T21:36:25.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035cb7873ab74e802-00650e08d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.455 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:25.455 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:25.455 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000317 2023-09-22T21:36:25.455 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000317 2023-09-22T21:36:25.455 DEBUG:boto:Method: GET 2023-09-22T21:36:25.456 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000317 2023-09-22T21:36:25.456 DEBUG:boto:Data: 2023-09-22T21:36:25.456 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.456 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.456 DEBUG:boto:Params: {} 2023-09-22T21:36:25.456 DEBUG:boto:Token: None 2023-09-22T21:36:25.456 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/ 2023-09-22T21:36:25.456 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec= 2023-09-22T21:36:25.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec=', 'Content-Length': '0'} 2023-09-22T21:36:25.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa5721f7d39433f4-00650e08d9-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.502 DEBUG:boto:b'ffzdqu-58obj000000000000000003171000falseobj000000000000000003172023-09-22T21:35:03.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:25.537 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:25.537 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:25.537 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000317 2023-09-22T21:36:25.537 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000317 2023-09-22T21:36:25.538 DEBUG:boto:Method: GET 2023-09-22T21:36:25.538 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000317 2023-09-22T21:36:25.538 DEBUG:boto:Data: 2023-09-22T21:36:25.538 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.538 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.538 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.538 DEBUG:boto:Params: {} 2023-09-22T21:36:25.538 DEBUG:boto:Token: None 2023-09-22T21:36:25.539 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/ 2023-09-22T21:36:25.539 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec= 2023-09-22T21:36:25.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec=', 'Content-Length': '0'} 2023-09-22T21:36:25.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ef4fd3a1066e649-00650e08d9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.549 DEBUG:boto:b'ffzdqu-58obj000000000000000003171000falseobj000000000000000003172023-09-22T21:35:03.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:25.550 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.550 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.550 DEBUG:boto:Method: HEAD 2023-09-22T21:36:25.550 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.550 DEBUG:boto:Data: 2023-09-22T21:36:25.550 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.550 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.551 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.551 DEBUG:boto:Params: {} 2023-09-22T21:36:25.551 DEBUG:boto:Token: None 2023-09-22T21:36:25.551 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.551 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kH0COacwrX7MtMa2DcUYTY2XD78= 2023-09-22T21:36:25.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kH0COacwrX7MtMa2DcUYTY2XD78=', 'Content-Length': '0'} 2023-09-22T21:36:25.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084f53a37ebaf8e76-00650e08d9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.554 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.554 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.554 DEBUG:boto:Method: HEAD 2023-09-22T21:36:25.554 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.555 DEBUG:boto:Data: 2023-09-22T21:36:25.555 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.555 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.555 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.555 DEBUG:boto:Params: {} 2023-09-22T21:36:25.555 DEBUG:boto:Token: None 2023-09-22T21:36:25.555 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.555 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kH0COacwrX7MtMa2DcUYTY2XD78= 2023-09-22T21:36:25.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kH0COacwrX7MtMa2DcUYTY2XD78=', 'Content-Length': '0'} 2023-09-22T21:36:25.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051466ffd89ac71bd-00650e08d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-09-22T21:36:25.559 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.559 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.559 DEBUG:boto:Method: GET 2023-09-22T21:36:25.559 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.559 DEBUG:boto:Data: 2023-09-22T21:36:25.559 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.559 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.559 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.559 DEBUG:boto:Params: {} 2023-09-22T21:36:25.560 DEBUG:boto:Token: None 2023-09-22T21:36:25.560 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.560 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:W4kNA/N+Y0kuy/PXD9rmsxjXBns= 2023-09-22T21:36:25.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:W4kNA/N+Y0kuy/PXD9rmsxjXBns=', 'Content-Length': '0'} 2023-09-22T21:36:25.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ca73a6c5b3dce7f-00650e08d9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.610 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.610 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.611 DEBUG:boto:Method: GET 2023-09-22T21:36:25.611 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.611 DEBUG:boto:Data: 2023-09-22T21:36:25.611 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.611 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.611 DEBUG:boto:Params: {} 2023-09-22T21:36:25.611 DEBUG:boto:Token: None 2023-09-22T21:36:25.612 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000317 2023-09-22T21:36:25.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:W4kNA/N+Y0kuy/PXD9rmsxjXBns= 2023-09-22T21:36:25.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:W4kNA/N+Y0kuy/PXD9rmsxjXBns=', 'Content-Length': '0'} 2023-09-22T21:36:25.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0c3660455b3316d-00650e08d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.615 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:25.615 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:25.615 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000317 2023-09-22T21:36:25.616 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000317 2023-09-22T21:36:25.616 DEBUG:boto:Method: GET 2023-09-22T21:36:25.616 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000317 2023-09-22T21:36:25.616 DEBUG:boto:Data: 2023-09-22T21:36:25.616 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.616 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.616 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.616 DEBUG:boto:Params: {} 2023-09-22T21:36:25.616 DEBUG:boto:Token: None 2023-09-22T21:36:25.617 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/ 2023-09-22T21:36:25.617 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec= 2023-09-22T21:36:25.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec=', 'Content-Length': '0'} 2023-09-22T21:36:25.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d81c77d4898fcf16-00650e08d9-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.621 DEBUG:boto:b'ffzdqu-58obj000000000000000003171000falseobj000000000000000003172023-09-22T21:35:03.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:25.622 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:25.622 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:25.622 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000317 2023-09-22T21:36:25.622 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000317 2023-09-22T21:36:25.622 DEBUG:boto:Method: GET 2023-09-22T21:36:25.622 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000317 2023-09-22T21:36:25.623 DEBUG:boto:Data: 2023-09-22T21:36:25.623 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.623 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.623 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.623 DEBUG:boto:Params: {} 2023-09-22T21:36:25.623 DEBUG:boto:Token: None 2023-09-22T21:36:25.623 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/ 2023-09-22T21:36:25.623 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec= 2023-09-22T21:36:25.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec=', 'Content-Length': '0'} 2023-09-22T21:36:25.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000287364a53ddc337b-00650e08d9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.678 DEBUG:boto:b'ffzdqu-58obj000000000000000003171000falseobj000000000000000003172023-09-22T21:35:03.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:25.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-09-22T21:36:25.679 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.679 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.679 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.679 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.679 DEBUG:boto:Method: GET 2023-09-22T21:36:25.680 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.680 DEBUG:boto:Data: 2023-09-22T21:36:25.680 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.680 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.680 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.680 DEBUG:boto:Params: {} 2023-09-22T21:36:25.680 DEBUG:boto:Token: None 2023-09-22T21:36:25.680 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.681 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:X39xOx6aDf7+Kn6sSCriMRJxM6o= 2023-09-22T21:36:25.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:X39xOx6aDf7+Kn6sSCriMRJxM6o=', 'Content-Length': '0'} 2023-09-22T21:36:25.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be6bedc1c5dfd097-00650e08d9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.726 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.727 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.727 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.727 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.727 DEBUG:boto:Method: GET 2023-09-22T21:36:25.727 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.727 DEBUG:boto:Data: 2023-09-22T21:36:25.727 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.727 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.727 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.728 DEBUG:boto:Params: {} 2023-09-22T21:36:25.728 DEBUG:boto:Token: None 2023-09-22T21:36:25.728 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.728 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:X39xOx6aDf7+Kn6sSCriMRJxM6o= 2023-09-22T21:36:25.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:X39xOx6aDf7+Kn6sSCriMRJxM6o=', 'Content-Length': '0'} 2023-09-22T21:36:25.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ceae96af15988d4-00650e08d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.731 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.731 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.731 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.731 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.732 DEBUG:boto:Method: HEAD 2023-09-22T21:36:25.732 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.732 DEBUG:boto:Data: 2023-09-22T21:36:25.732 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.732 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.732 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.732 DEBUG:boto:Params: {} 2023-09-22T21:36:25.732 DEBUG:boto:Token: None 2023-09-22T21:36:25.732 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.733 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZjsMF7YF6EEC0FcIOIQncxD3lVE= 2023-09-22T21:36:25.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZjsMF7YF6EEC0FcIOIQncxD3lVE=', 'Content-Length': '0'} 2023-09-22T21:36:25.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d96abedd7889f407-00650e08d9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.736 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.736 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.736 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.736 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.736 DEBUG:boto:Method: HEAD 2023-09-22T21:36:25.736 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.736 DEBUG:boto:Data: 2023-09-22T21:36:25.736 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.736 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.737 DEBUG:boto:Params: {} 2023-09-22T21:36:25.737 DEBUG:boto:Token: None 2023-09-22T21:36:25.737 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.737 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZjsMF7YF6EEC0FcIOIQncxD3lVE= 2023-09-22T21:36:25.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZjsMF7YF6EEC0FcIOIQncxD3lVE=', 'Content-Length': '0'} 2023-09-22T21:36:25.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c376b04f3c8abcd1-00650e08d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-09-22T21:36:25.740 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.740 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.740 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.740 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.740 DEBUG:boto:Method: GET 2023-09-22T21:36:25.740 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.741 DEBUG:boto:Data: 2023-09-22T21:36:25.741 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.741 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.741 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.741 DEBUG:boto:Params: {} 2023-09-22T21:36:25.741 DEBUG:boto:Token: None 2023-09-22T21:36:25.741 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.741 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:X39xOx6aDf7+Kn6sSCriMRJxM6o= 2023-09-22T21:36:25.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:X39xOx6aDf7+Kn6sSCriMRJxM6o=', 'Content-Length': '0'} 2023-09-22T21:36:25.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cce9a7ba4c624e16-00650e08d9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.786 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.786 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.786 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.787 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.787 DEBUG:boto:Method: GET 2023-09-22T21:36:25.787 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.787 DEBUG:boto:Data: 2023-09-22T21:36:25.787 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.787 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.787 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.787 DEBUG:boto:Params: {} 2023-09-22T21:36:25.787 DEBUG:boto:Token: None 2023-09-22T21:36:25.788 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000423?versionId=null 2023-09-22T21:36:25.788 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:X39xOx6aDf7+Kn6sSCriMRJxM6o= 2023-09-22T21:36:25.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:X39xOx6aDf7+Kn6sSCriMRJxM6o=', 'Content-Length': '0'} 2023-09-22T21:36:25.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4bb4a6575392e98-00650e08d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.791 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:25.791 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:25.793 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000423 2023-09-22T21:36:25.793 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000423 2023-09-22T21:36:25.793 DEBUG:boto:Method: GET 2023-09-22T21:36:25.793 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000423 2023-09-22T21:36:25.793 DEBUG:boto:Data: 2023-09-22T21:36:25.793 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.793 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.793 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.794 DEBUG:boto:Params: {} 2023-09-22T21:36:25.794 DEBUG:boto:Token: None 2023-09-22T21:36:25.794 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/ 2023-09-22T21:36:25.794 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec= 2023-09-22T21:36:25.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec=', 'Content-Length': '0'} 2023-09-22T21:36:25.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c7094585475907f-00650e08d9-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.842 DEBUG:boto:b'ffzdqu-58obj000000000000000004231000falseobj000000000000000004232023-09-22T21:35:04.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:25.843 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:25.843 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:25.843 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000423 2023-09-22T21:36:25.843 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000423 2023-09-22T21:36:25.843 DEBUG:boto:Method: GET 2023-09-22T21:36:25.843 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000423 2023-09-22T21:36:25.844 DEBUG:boto:Data: 2023-09-22T21:36:25.844 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.844 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.844 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.844 DEBUG:boto:Params: {} 2023-09-22T21:36:25.844 DEBUG:boto:Token: None 2023-09-22T21:36:25.844 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/ 2023-09-22T21:36:25.844 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec= 2023-09-22T21:36:25.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec=', 'Content-Length': '0'} 2023-09-22T21:36:25.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003154233bd05a993d-00650e08d9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.855 DEBUG:boto:b'ffzdqu-58obj000000000000000004231000falseobj000000000000000004232023-09-22T21:35:04.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:25.855 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.856 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.856 DEBUG:boto:Method: HEAD 2023-09-22T21:36:25.856 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.856 DEBUG:boto:Data: 2023-09-22T21:36:25.856 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.856 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.856 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.856 DEBUG:boto:Params: {} 2023-09-22T21:36:25.856 DEBUG:boto:Token: None 2023-09-22T21:36:25.857 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.857 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wxAvulfXGxd/IlOMoFhElTmGjj0= 2023-09-22T21:36:25.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wxAvulfXGxd/IlOMoFhElTmGjj0=', 'Content-Length': '0'} 2023-09-22T21:36:25.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006c0bcfe146ce3ff-00650e08d9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.860 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.860 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.860 DEBUG:boto:Method: HEAD 2023-09-22T21:36:25.860 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.860 DEBUG:boto:Data: 2023-09-22T21:36:25.861 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.861 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.861 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.861 DEBUG:boto:Params: {} 2023-09-22T21:36:25.861 DEBUG:boto:Token: None 2023-09-22T21:36:25.861 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.861 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wxAvulfXGxd/IlOMoFhElTmGjj0= 2023-09-22T21:36:25.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wxAvulfXGxd/IlOMoFhElTmGjj0=', 'Content-Length': '0'} 2023-09-22T21:36:25.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000526ca1bd6a2c146b-00650e08d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-09-22T21:36:25.864 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.864 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.864 DEBUG:boto:Method: GET 2023-09-22T21:36:25.864 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.865 DEBUG:boto:Data: 2023-09-22T21:36:25.865 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.865 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.865 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.865 DEBUG:boto:Params: {} 2023-09-22T21:36:25.865 DEBUG:boto:Token: None 2023-09-22T21:36:25.865 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.865 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Kv8/iUc+5hJrPhG0Cdnkskj9Ykk= 2023-09-22T21:36:25.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Kv8/iUc+5hJrPhG0Cdnkskj9Ykk=', 'Content-Length': '0'} 2023-09-22T21:36:25.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8750e073806af10-00650e08d9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.910 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.911 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.911 DEBUG:boto:Method: GET 2023-09-22T21:36:25.911 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.911 DEBUG:boto:Data: 2023-09-22T21:36:25.911 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.911 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.911 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.911 DEBUG:boto:Params: {} 2023-09-22T21:36:25.911 DEBUG:boto:Token: None 2023-09-22T21:36:25.912 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000423 2023-09-22T21:36:25.912 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Kv8/iUc+5hJrPhG0Cdnkskj9Ykk= 2023-09-22T21:36:25.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Kv8/iUc+5hJrPhG0Cdnkskj9Ykk=', 'Content-Length': '0'} 2023-09-22T21:36:25.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085e724fefcc1839a-00650e08d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.915 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:25.915 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:25.915 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000423 2023-09-22T21:36:25.915 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000423 2023-09-22T21:36:25.915 DEBUG:boto:Method: GET 2023-09-22T21:36:25.916 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000423 2023-09-22T21:36:25.916 DEBUG:boto:Data: 2023-09-22T21:36:25.916 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.916 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.916 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.916 DEBUG:boto:Params: {} 2023-09-22T21:36:25.916 DEBUG:boto:Token: None 2023-09-22T21:36:25.916 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/ 2023-09-22T21:36:25.916 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec= 2023-09-22T21:36:25.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec=', 'Content-Length': '0'} 2023-09-22T21:36:25.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000af9228da8b2b24e-00650e08d9-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.962 DEBUG:boto:b'ffzdqu-58obj000000000000000004231000falseobj000000000000000004232023-09-22T21:35:04.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:25.963 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:25.963 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:25.963 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000423 2023-09-22T21:36:25.963 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000423 2023-09-22T21:36:25.963 DEBUG:boto:Method: GET 2023-09-22T21:36:25.963 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000423 2023-09-22T21:36:25.963 DEBUG:boto:Data: 2023-09-22T21:36:25.963 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.964 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.964 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.964 DEBUG:boto:Params: {} 2023-09-22T21:36:25.964 DEBUG:boto:Token: None 2023-09-22T21:36:25.964 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/ 2023-09-22T21:36:25.964 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec= 2023-09-22T21:36:25.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QbUk1wxeWlx9fDIX1+qvwQt8bec=', 'Content-Length': '0'} 2023-09-22T21:36:25.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000631264460ca27c19-00650e08d9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.974 DEBUG:boto:b'ffzdqu-58obj000000000000000004231000falseobj000000000000000004232023-09-22T21:35:04.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:25.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-09-22T21:36:25.975 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:25.975 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:25.975 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:25.975 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:25.975 DEBUG:boto:Method: GET 2023-09-22T21:36:25.975 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:25.975 DEBUG:boto:Data: 2023-09-22T21:36:25.975 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.975 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:25.975 DEBUG:boto:Port: 8001 2023-09-22T21:36:25.976 DEBUG:boto:Params: {} 2023-09-22T21:36:25.976 DEBUG:boto:Token: None 2023-09-22T21:36:25.976 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:25.976 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6pClznjuzyyxge0Vpo/ZYZQXmCI= 2023-09-22T21:36:25.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6pClznjuzyyxge0Vpo/ZYZQXmCI=', 'Content-Length': '0'} 2023-09-22T21:36:25.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf69a51f640bab22-00650e08d9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:25.980 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:25.980 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:25.980 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:25.980 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:25.980 DEBUG:boto:Method: GET 2023-09-22T21:36:25.980 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:25.980 DEBUG:boto:Data: 2023-09-22T21:36:25.980 DEBUG:boto:Headers: {} 2023-09-22T21:36:25.980 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:25.981 DEBUG:boto:Port: 8000 2023-09-22T21:36:25.981 DEBUG:boto:Params: {} 2023-09-22T21:36:25.981 DEBUG:boto:Token: None 2023-09-22T21:36:25.981 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:25 GMT /ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:25.981 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6pClznjuzyyxge0Vpo/ZYZQXmCI= 2023-09-22T21:36:25.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:25 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6pClznjuzyyxge0Vpo/ZYZQXmCI=', 'Content-Length': '0'} 2023-09-22T21:36:25.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006825c0306c441297-00650e08d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:26.026 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.027 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.027 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:26.027 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:26.027 DEBUG:boto:Method: HEAD 2023-09-22T21:36:26.027 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:26.027 DEBUG:boto:Data: 2023-09-22T21:36:26.027 DEBUG:boto:Headers: {} 2023-09-22T21:36:26.027 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:26.027 DEBUG:boto:Port: 8001 2023-09-22T21:36:26.028 DEBUG:boto:Params: {} 2023-09-22T21:36:26.028 DEBUG:boto:Token: None 2023-09-22T21:36:26.028 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:26 GMT /ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:26.028 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:trr6UAjo8L8iDpV41v8eFc8gsy4= 2023-09-22T21:36:26.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:trr6UAjo8L8iDpV41v8eFc8gsy4=', 'Content-Length': '0'} 2023-09-22T21:36:26.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b7c449469041ddb-00650e08da-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:26.031 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.031 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.032 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:26.032 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:26.032 DEBUG:boto:Method: HEAD 2023-09-22T21:36:26.032 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:26.032 DEBUG:boto:Data: 2023-09-22T21:36:26.032 DEBUG:boto:Headers: {} 2023-09-22T21:36:26.032 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:26.032 DEBUG:boto:Port: 8000 2023-09-22T21:36:26.032 DEBUG:boto:Params: {} 2023-09-22T21:36:26.032 DEBUG:boto:Token: None 2023-09-22T21:36:26.033 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:26 GMT /ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:26.033 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:trr6UAjo8L8iDpV41v8eFc8gsy4= 2023-09-22T21:36:26.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:trr6UAjo8L8iDpV41v8eFc8gsy4=', 'Content-Length': '0'} 2023-09-22T21:36:26.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb24d5cef21187fc-00650e08da-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:26.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-09-22T21:36:26.036 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.036 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.036 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:26.036 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:26.036 DEBUG:boto:Method: GET 2023-09-22T21:36:26.036 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:26.037 DEBUG:boto:Data: 2023-09-22T21:36:26.037 DEBUG:boto:Headers: {} 2023-09-22T21:36:26.037 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:26.037 DEBUG:boto:Port: 8001 2023-09-22T21:36:26.037 DEBUG:boto:Params: {} 2023-09-22T21:36:26.037 DEBUG:boto:Token: None 2023-09-22T21:36:26.037 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:26 GMT /ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:26.037 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:v23tKkuFw5NkptZ2BGcU/GFDOiU= 2023-09-22T21:36:26.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:v23tKkuFw5NkptZ2BGcU/GFDOiU=', 'Content-Length': '0'} 2023-09-22T21:36:26.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063760c89288b68dd-00650e08da-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:26.082 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.083 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.083 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:26.083 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:26.083 DEBUG:boto:Method: GET 2023-09-22T21:36:26.083 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:26.083 DEBUG:boto:Data: 2023-09-22T21:36:26.083 DEBUG:boto:Headers: {} 2023-09-22T21:36:26.083 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:26.083 DEBUG:boto:Port: 8000 2023-09-22T21:36:26.083 DEBUG:boto:Params: {} 2023-09-22T21:36:26.084 DEBUG:boto:Token: None 2023-09-22T21:36:26.084 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:26 GMT /ffzdqu-58/obj00000000000000000467?versionId=null 2023-09-22T21:36:26.084 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:v23tKkuFw5NkptZ2BGcU/GFDOiU= 2023-09-22T21:36:26.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:v23tKkuFw5NkptZ2BGcU/GFDOiU=', 'Content-Length': '0'} 2023-09-22T21:36:26.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018ba01195454b8e2-00650e08da-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:26.087 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:26.088 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:26.088 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000467 2023-09-22T21:36:26.088 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000467 2023-09-22T21:36:26.088 DEBUG:boto:Method: GET 2023-09-22T21:36:26.088 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000467 2023-09-22T21:36:26.088 DEBUG:boto:Data: 2023-09-22T21:36:26.088 DEBUG:boto:Headers: {} 2023-09-22T21:36:26.088 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:26.088 DEBUG:boto:Port: 8001 2023-09-22T21:36:26.089 DEBUG:boto:Params: {} 2023-09-22T21:36:26.089 DEBUG:boto:Token: None 2023-09-22T21:36:26.089 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:26 GMT /ffzdqu-58/ 2023-09-22T21:36:26.089 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SgWnqom38rIqyBzdOiIyffChR/M= 2023-09-22T21:36:26.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SgWnqom38rIqyBzdOiIyffChR/M=', 'Content-Length': '0'} 2023-09-22T21:36:26.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7320fb29283aa32-00650e08da-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:26.134 DEBUG:boto:b'ffzdqu-58obj000000000000000004671000falseobj000000000000000004672023-09-22T21:35:03.651Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:26.135 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:26.135 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:26.135 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000467 2023-09-22T21:36:26.135 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000467 2023-09-22T21:36:26.135 DEBUG:boto:Method: GET 2023-09-22T21:36:26.136 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000467 2023-09-22T21:36:26.136 DEBUG:boto:Data: 2023-09-22T21:36:26.136 DEBUG:boto:Headers: {} 2023-09-22T21:36:26.136 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:26.136 DEBUG:boto:Port: 8000 2023-09-22T21:36:26.136 DEBUG:boto:Params: {} 2023-09-22T21:36:26.136 DEBUG:boto:Token: None 2023-09-22T21:36:26.136 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:26 GMT /ffzdqu-58/ 2023-09-22T21:36:26.136 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SgWnqom38rIqyBzdOiIyffChR/M= 2023-09-22T21:36:26.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SgWnqom38rIqyBzdOiIyffChR/M=', 'Content-Length': '0'} 2023-09-22T21:36:26.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed6cce75cf1dd5c8-00650e08da-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:26.147 DEBUG:boto:b'ffzdqu-58obj000000000000000004671000falseobj000000000000000004672023-09-22T21:35:03.651Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:26.147 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.147 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.148 DEBUG:boto:Method: HEAD 2023-09-22T21:36:26.148 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.148 DEBUG:boto:Data: 2023-09-22T21:36:26.148 DEBUG:boto:Headers: {} 2023-09-22T21:36:26.148 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:26.148 DEBUG:boto:Port: 8001 2023-09-22T21:36:26.148 DEBUG:boto:Params: {} 2023-09-22T21:36:26.148 DEBUG:boto:Token: None 2023-09-22T21:36:26.149 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:26 GMT /ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.149 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jXaZ8CR4E3H/RD0yFAtFm7NWShk= 2023-09-22T21:36:26.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jXaZ8CR4E3H/RD0yFAtFm7NWShk=', 'Content-Length': '0'} 2023-09-22T21:36:26.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ecb3587032347e9-00650e08da-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:26.152 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.152 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.152 DEBUG:boto:Method: HEAD 2023-09-22T21:36:26.152 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.152 DEBUG:boto:Data: 2023-09-22T21:36:26.152 DEBUG:boto:Headers: {} 2023-09-22T21:36:26.152 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:26.152 DEBUG:boto:Port: 8000 2023-09-22T21:36:26.153 DEBUG:boto:Params: {} 2023-09-22T21:36:26.153 DEBUG:boto:Token: None 2023-09-22T21:36:26.153 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:26 GMT /ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.153 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jXaZ8CR4E3H/RD0yFAtFm7NWShk= 2023-09-22T21:36:26.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jXaZ8CR4E3H/RD0yFAtFm7NWShk=', 'Content-Length': '0'} 2023-09-22T21:36:26.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e163ae69292a2e1-00650e08da-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:26.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-09-22T21:36:26.156 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.156 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.156 DEBUG:boto:Method: GET 2023-09-22T21:36:26.157 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.157 DEBUG:boto:Data: 2023-09-22T21:36:26.157 DEBUG:boto:Headers: {} 2023-09-22T21:36:26.157 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:26.157 DEBUG:boto:Port: 8001 2023-09-22T21:36:26.157 DEBUG:boto:Params: {} 2023-09-22T21:36:26.157 DEBUG:boto:Token: None 2023-09-22T21:36:26.157 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:26 GMT /ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.157 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:c4XRKzaqANL+5ZEZmb94GHmPO10= 2023-09-22T21:36:26.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:c4XRKzaqANL+5ZEZmb94GHmPO10=', 'Content-Length': '0'} 2023-09-22T21:36:26.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-58:f132bdb9-20a7-4073-a932-8f99da20691e.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b18c5980a46163f3-00650e08da-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:26.202 DEBUG:boto:path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.202 DEBUG:boto:auth_path=/ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.203 DEBUG:boto:Method: GET 2023-09-22T21:36:26.203 DEBUG:boto:Path: /ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.203 DEBUG:boto:Data: 2023-09-22T21:36:26.203 DEBUG:boto:Headers: {} 2023-09-22T21:36:26.203 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:26.203 DEBUG:boto:Port: 8000 2023-09-22T21:36:26.203 DEBUG:boto:Params: {} 2023-09-22T21:36:26.203 DEBUG:boto:Token: None 2023-09-22T21:36:26.203 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:26 GMT /ffzdqu-58/obj00000000000000000467 2023-09-22T21:36:26.204 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:c4XRKzaqANL+5ZEZmb94GHmPO10= 2023-09-22T21:36:26.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:c4XRKzaqANL+5ZEZmb94GHmPO10=', 'Content-Length': '0'} 2023-09-22T21:36:26.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:35:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021e5b604588b4f38-00650e08da-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:26.207 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:26.207 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:26.207 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000467 2023-09-22T21:36:26.207 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000467 2023-09-22T21:36:26.207 DEBUG:boto:Method: GET 2023-09-22T21:36:26.207 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000467 2023-09-22T21:36:26.207 DEBUG:boto:Data: 2023-09-22T21:36:26.207 DEBUG:boto:Headers: {} 2023-09-22T21:36:26.208 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:26.208 DEBUG:boto:Port: 8001 2023-09-22T21:36:26.208 DEBUG:boto:Params: {} 2023-09-22T21:36:26.208 DEBUG:boto:Token: None 2023-09-22T21:36:26.208 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:26 GMT /ffzdqu-58/ 2023-09-22T21:36:26.208 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SgWnqom38rIqyBzdOiIyffChR/M= 2023-09-22T21:36:26.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SgWnqom38rIqyBzdOiIyffChR/M=', 'Content-Length': '0'} 2023-09-22T21:36:26.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060e3fb1eab68ddf1-00650e08da-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:26.254 DEBUG:boto:b'ffzdqu-58obj000000000000000004671000falseobj000000000000000004672023-09-22T21:35:03.651Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:26.255 DEBUG:boto:path=/ffzdqu-58/ 2023-09-22T21:36:26.255 DEBUG:boto:auth_path=/ffzdqu-58/ 2023-09-22T21:36:26.255 DEBUG:boto:path=/ffzdqu-58/?prefix=obj00000000000000000467 2023-09-22T21:36:26.255 DEBUG:boto:auth_path=/ffzdqu-58/?prefix=obj00000000000000000467 2023-09-22T21:36:26.255 DEBUG:boto:Method: GET 2023-09-22T21:36:26.255 DEBUG:boto:Path: /ffzdqu-58/?prefix=obj00000000000000000467 2023-09-22T21:36:26.255 DEBUG:boto:Data: 2023-09-22T21:36:26.256 DEBUG:boto:Headers: {} 2023-09-22T21:36:26.256 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:26.256 DEBUG:boto:Port: 8000 2023-09-22T21:36:26.256 DEBUG:boto:Params: {} 2023-09-22T21:36:26.256 DEBUG:boto:Token: None 2023-09-22T21:36:26.256 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:26 GMT /ffzdqu-58/ 2023-09-22T21:36:26.256 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SgWnqom38rIqyBzdOiIyffChR/M= 2023-09-22T21:36:26.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SgWnqom38rIqyBzdOiIyffChR/M=', 'Content-Length': '0'} 2023-09-22T21:36:26.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000519afe7267037ee6-00650e08da-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:26.267 DEBUG:boto:b'ffzdqu-58obj000000000000000004671000falseobj000000000000000004672023-09-22T21:35:03.651Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:26.267 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-58 zones={a2, a1} 2023-09-22T21:36:26.268 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... ok 2023-09-22T21:36:26.270 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:26.270 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:26.270 DEBUG:boto:Method: PUT 2023-09-22T21:36:26.270 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:36:26.270 DEBUG:boto:Data: 2023-09-22T21:36:26.270 DEBUG:boto:Headers: {} 2023-09-22T21:36:26.271 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:26.271 DEBUG:boto:Port: 8000 2023-09-22T21:36:26.271 DEBUG:boto:Params: {} 2023-09-22T21:36:26.271 DEBUG:boto:Token: None 2023-09-22T21:36:26.271 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:36:26 GMT /ffzdqu-59/ 2023-09-22T21:36:26.271 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vdmYCAGCKaFFZ/0+yIxapG0Wgh0= 2023-09-22T21:36:26.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:26 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vdmYCAGCKaFFZ/0+yIxapG0Wgh0=', 'Content-Length': '0'} 2023-09-22T21:36:26.314 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000006404b2a350c8033-00650e08da-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:36:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:26.314 DEBUG:rgw_multi.tests:created bucket=ffzdqu-59 2023-09-22T21:36:26.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:36:26.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:36:26.315 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:36:26.355 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:36:26.356 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:36:26.396 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:36:26.416 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1695418586.309673_512.1","last_update":"2023-09-22T21:36:26.309673Z"},{"marker":"1_1695416704.185257_96.1","last_update":"2023-09-22T21:05:04.185257Z"}] 2023-09-22T21:36:26.416 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695418586.309673_512.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:36:26.417 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:36:26.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:36:26.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:36:26.417 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:36:26.496 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:36:26.496 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:36:26.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:36:26.559+0000 7fe2c94efa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:36:26.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:36:26.563+0000 7fe2c94efa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:36:26.587 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695418498.991407_468.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:34:58.991407Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:36:26Z"} 2023-09-22T21:36:26.587 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695418498.991407_468.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:34:58.991407Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:36:26Z"} 2023-09-22T21:36:26.587 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695418498.991407_468.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:34:58.991407Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:36:26.587 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695418586.309673_512.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:36:26.587 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695418498.991407_468.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:36:26.588 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695418586.309673_512.1 target=1_1695418498.991407_468.1 2023-09-22T21:36:31.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:36:31.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:36:31.589 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:36:31.633 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:36:31.633 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:36:31.670 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:36:31.667+0000 7f6da5da3a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:36:31.671 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:36:31.667+0000 7f6da5da3a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:36:31.690 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695418586.309673_512.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:36:26.309673Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:36:31Z"} 2023-09-22T21:36:31.690 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695418586.309673_512.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:36:26.309673Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:36:31Z"} 2023-09-22T21:36:31.690 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695418586.309673_512.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:36:26.309673Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:36:31.690 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695418586.309673_512.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:36:31.690 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695418586.309673_512.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:36:31.691 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:36:31.691 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:31.691 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:31.691 DEBUG:boto:Method: HEAD 2023-09-22T21:36:31.691 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:36:31.691 DEBUG:boto:Data: 2023-09-22T21:36:31.691 DEBUG:boto:Headers: {} 2023-09-22T21:36:31.691 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:31.691 DEBUG:boto:Port: 8000 2023-09-22T21:36:31.692 DEBUG:boto:Params: {} 2023-09-22T21:36:31.692 DEBUG:boto:Token: None 2023-09-22T21:36:31.692 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:31 GMT /ffzdqu-59/ 2023-09-22T21:36:31.692 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UKghVhkggSWtnkkyB2xeA6G8qYc= 2023-09-22T21:36:31.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UKghVhkggSWtnkkyB2xeA6G8qYc=', 'Content-Length': '0'} 2023-09-22T21:36:31.694 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a195e4f2e97ef887-00650e08df-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:31.695 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:36:31.695 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:36:31.695 DEBUG:boto:Method: PUT 2023-09-22T21:36:31.695 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:36:31.695 DEBUG:boto:Data: 2023-09-22T21:36:31.696 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:36:31.696 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:31.696 DEBUG:boto:Port: 8000 2023-09-22T21:36:31.696 DEBUG:boto:Params: {} 2023-09-22T21:36:31.696 DEBUG:boto:Token: None 2023-09-22T21:36:31.696 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:36:31 GMT /ffzdqu-59/obj11 2023-09-22T21:36:31.696 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Gdp9fKCM9gKKqGvckszYOQhkZGI= 2023-09-22T21:36:31.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:36:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Gdp9fKCM9gKKqGvckszYOQhkZGI='} 2023-09-22T21:36:31.704 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab94c556b9ff0bb0-00650e08df-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:31.704 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:31.704 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:31.704 DEBUG:boto:Method: HEAD 2023-09-22T21:36:31.704 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:36:31.704 DEBUG:boto:Data: 2023-09-22T21:36:31.705 DEBUG:boto:Headers: {} 2023-09-22T21:36:31.705 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:31.705 DEBUG:boto:Port: 8000 2023-09-22T21:36:31.705 DEBUG:boto:Params: {} 2023-09-22T21:36:31.705 DEBUG:boto:Token: None 2023-09-22T21:36:31.705 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:31 GMT /ffzdqu-59/ 2023-09-22T21:36:31.705 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UKghVhkggSWtnkkyB2xeA6G8qYc= 2023-09-22T21:36:31.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UKghVhkggSWtnkkyB2xeA6G8qYc=', 'Content-Length': '0'} 2023-09-22T21:36:31.707 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006948899b8d640daf-00650e08df-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:31.708 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:36:31.708 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:36:31.708 DEBUG:boto:Method: PUT 2023-09-22T21:36:31.708 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:36:31.708 DEBUG:boto:Data: 2023-09-22T21:36:31.708 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:36:31.709 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:31.709 DEBUG:boto:Port: 8000 2023-09-22T21:36:31.709 DEBUG:boto:Params: {} 2023-09-22T21:36:31.709 DEBUG:boto:Token: None 2023-09-22T21:36:31.709 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:36:31 GMT /ffzdqu-59/obj22 2023-09-22T21:36:31.709 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Lu4AucSYtn++0KIaEs/h7UmP7K4= 2023-09-22T21:36:31.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:36:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Lu4AucSYtn++0KIaEs/h7UmP7K4='} 2023-09-22T21:36:31.759 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d8d7920cfe13617-00650e08df-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:31.759 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:31.759 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:31.759 DEBUG:boto:Method: HEAD 2023-09-22T21:36:31.759 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:36:31.759 DEBUG:boto:Data: 2023-09-22T21:36:31.760 DEBUG:boto:Headers: {} 2023-09-22T21:36:31.760 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:31.760 DEBUG:boto:Port: 8000 2023-09-22T21:36:31.760 DEBUG:boto:Params: {} 2023-09-22T21:36:31.760 DEBUG:boto:Token: None 2023-09-22T21:36:31.760 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:31 GMT /ffzdqu-59/ 2023-09-22T21:36:31.760 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UKghVhkggSWtnkkyB2xeA6G8qYc= 2023-09-22T21:36:31.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UKghVhkggSWtnkkyB2xeA6G8qYc=', 'Content-Length': '0'} 2023-09-22T21:36:31.764 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fea9df6c8a7333dd-00650e08df-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:31.764 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:36:31.765 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:36:31.765 DEBUG:boto:Method: PUT 2023-09-22T21:36:31.765 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:36:31.765 DEBUG:boto:Data: 2023-09-22T21:36:31.765 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:36:31.765 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:31.765 DEBUG:boto:Port: 8000 2023-09-22T21:36:31.765 DEBUG:boto:Params: {} 2023-09-22T21:36:31.766 DEBUG:boto:Token: None 2023-09-22T21:36:31.766 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:36:31 GMT /ffzdqu-59/obj33 2023-09-22T21:36:31.766 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5o32uam11m8amBCmzvUQYD4N//g= 2023-09-22T21:36:31.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:36:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5o32uam11m8amBCmzvUQYD4N//g='} 2023-09-22T21:36:31.776 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000089903b9b92342618-00650e08df-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:31.777 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:31.777 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:31.777 DEBUG:boto:Method: HEAD 2023-09-22T21:36:31.777 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:36:31.777 DEBUG:boto:Data: 2023-09-22T21:36:31.777 DEBUG:boto:Headers: {} 2023-09-22T21:36:31.777 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:31.777 DEBUG:boto:Port: 8000 2023-09-22T21:36:31.778 DEBUG:boto:Params: {} 2023-09-22T21:36:31.778 DEBUG:boto:Token: None 2023-09-22T21:36:31.778 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:31 GMT /ffzdqu-59/ 2023-09-22T21:36:31.778 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UKghVhkggSWtnkkyB2xeA6G8qYc= 2023-09-22T21:36:31.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UKghVhkggSWtnkkyB2xeA6G8qYc=', 'Content-Length': '0'} 2023-09-22T21:36:31.782 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1b7ff8a06f745f7-00650e08df-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:31.782 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:36:31.782 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:36:31.783 DEBUG:boto:Method: PUT 2023-09-22T21:36:31.783 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:36:31.783 DEBUG:boto:Data: 2023-09-22T21:36:31.783 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:36:31.783 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:31.783 DEBUG:boto:Port: 8000 2023-09-22T21:36:31.783 DEBUG:boto:Params: {} 2023-09-22T21:36:31.783 DEBUG:boto:Token: None 2023-09-22T21:36:31.784 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:36:31 GMT /ffzdqu-59/obj44 2023-09-22T21:36:31.784 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GOCJhBGRY7bAwyOs1CEAarP1M2Q= 2023-09-22T21:36:31.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:36:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GOCJhBGRY7bAwyOs1CEAarP1M2Q='} 2023-09-22T21:36:31.827 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b5a7e0396a484ce-00650e08df-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:31.827 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:31.827 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:31.827 DEBUG:boto:Method: HEAD 2023-09-22T21:36:31.827 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:36:31.827 DEBUG:boto:Data: 2023-09-22T21:36:31.828 DEBUG:boto:Headers: {} 2023-09-22T21:36:31.828 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:31.828 DEBUG:boto:Port: 8000 2023-09-22T21:36:31.828 DEBUG:boto:Params: {} 2023-09-22T21:36:31.828 DEBUG:boto:Token: None 2023-09-22T21:36:31.828 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:31 GMT /ffzdqu-59/ 2023-09-22T21:36:31.828 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UKghVhkggSWtnkkyB2xeA6G8qYc= 2023-09-22T21:36:31.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UKghVhkggSWtnkkyB2xeA6G8qYc=', 'Content-Length': '0'} 2023-09-22T21:36:31.833 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011b23a34d14e461f-00650e08df-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:31.833 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:36:31.833 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:36:31.834 DEBUG:boto:Method: PUT 2023-09-22T21:36:31.834 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:36:31.834 DEBUG:boto:Data: 2023-09-22T21:36:31.834 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:36:31.834 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:31.834 DEBUG:boto:Port: 8000 2023-09-22T21:36:31.834 DEBUG:boto:Params: {} 2023-09-22T21:36:31.834 DEBUG:boto:Token: None 2023-09-22T21:36:31.834 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:36:31 GMT /ffzdqu-59/obj55 2023-09-22T21:36:31.835 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Unaj0prKKiiL23/7/770i+oXEUc= 2023-09-22T21:36:31.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:36:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Unaj0prKKiiL23/7/770i+oXEUc='} 2023-09-22T21:36:31.843 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000350197da232b2cae-00650e08df-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:31.843 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-59 2023-09-22T21:36:31.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:36:31.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:36:31.844 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:36:31.892 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:36:31.892 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:36:31.935 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:36:31.931+0000 7fcc6dd6ca40 1 waiting to reach incremental sync.. 2023-09-22T21:36:36.936 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:36:36.935+0000 7fcc6dd6ca40 1 waiting to reach incremental sync.. 2023-09-22T21:36:41.937 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:36:41.934+0000 7fcc6dd6ca40 1 waiting to reach incremental sync.. 2023-09-22T21:36:45.756 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:36:45.750+0000 7f092e9f8640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:36:46.941 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:36:46.938+0000 7fcc6dd6ca40 1 bucket sync caught up with source: 2023-09-22T21:36:46.941 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.55.6, 00000000002.94.6, 00000000002.72.6] 2023-09-22T21:36:46.941 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.55.6, 00000000002.94.6, 00000000002.72.6] 2023-09-22T21:36:46.941 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:36:46.938+0000 7fcc6dd6ca40 0 bucket checkpoint complete 2023-09-22T21:36:46.973 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-59 2023-09-22T21:36:46.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:36:46.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:36:46.974 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:36:47.022 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:36:47.023 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:36:47.067 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:36:47.062+0000 7f6eb69caa40 1 bucket sync caught up with source: 2023-09-22T21:36:47.067 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.129.6, 00000000002.148.6, 00000000002.298.6] 2023-09-22T21:36:47.067 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.129.6, 00000000002.148.6, 00000000002.298.6] 2023-09-22T21:36:47.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:36:47.062+0000 7f6eb69caa40 0 bucket checkpoint complete 2023-09-22T21:36:47.084 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-59 zones={a2, a1} 2023-09-22T21:36:47.085 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:47.085 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:47.085 DEBUG:boto:Method: HEAD 2023-09-22T21:36:47.085 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:36:47.085 DEBUG:boto:Data: 2023-09-22T21:36:47.085 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.085 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.085 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.086 DEBUG:boto:Params: {} 2023-09-22T21:36:47.086 DEBUG:boto:Token: None 2023-09-22T21:36:47.086 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/ 2023-09-22T21:36:47.086 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qUJrOKCPThvD6J9KkyD2HNrHMcU= 2023-09-22T21:36:47.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qUJrOKCPThvD6J9KkyD2HNrHMcU=', 'Content-Length': '0'} 2023-09-22T21:36:47.089 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051c42b9b05064645-00650e08ef-1b9c-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.089 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:47.089 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:47.089 DEBUG:boto:Method: HEAD 2023-09-22T21:36:47.089 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:36:47.089 DEBUG:boto:Data: 2023-09-22T21:36:47.089 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.089 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.090 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.090 DEBUG:boto:Params: {} 2023-09-22T21:36:47.090 DEBUG:boto:Token: None 2023-09-22T21:36:47.090 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/ 2023-09-22T21:36:47.090 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qUJrOKCPThvD6J9KkyD2HNrHMcU= 2023-09-22T21:36:47.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qUJrOKCPThvD6J9KkyD2HNrHMcU=', 'Content-Length': '0'} 2023-09-22T21:36:47.092 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003480bfed0f6f814-00650e08ef-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.093 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:36:47.093 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:47.093 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:47.093 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:36:47.093 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:36:47.093 DEBUG:boto:Method: GET 2023-09-22T21:36:47.093 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:36:47.093 DEBUG:boto:Data: 2023-09-22T21:36:47.094 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.094 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.094 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.094 DEBUG:boto:Params: {} 2023-09-22T21:36:47.094 DEBUG:boto:Token: None 2023-09-22T21:36:47.094 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/?versions 2023-09-22T21:36:47.094 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:of+I5bbMCT+ScBceZh37DpbCy4g= 2023-09-22T21:36:47.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:of+I5bbMCT+ScBceZh37DpbCy4g=', 'Content-Length': '0'} 2023-09-22T21:36:47.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc16286eb51c51cc-00650e08ef-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.097 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:47.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T21:36:47.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T21:36:47.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T21:36:47.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T21:36:47.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T21:36:47.098 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:36:47.098 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:47.098 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:47.098 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:36:47.099 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:36:47.099 DEBUG:boto:Method: GET 2023-09-22T21:36:47.099 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:36:47.099 DEBUG:boto:Data: 2023-09-22T21:36:47.099 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.099 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.099 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.099 DEBUG:boto:Params: {} 2023-09-22T21:36:47.099 DEBUG:boto:Token: None 2023-09-22T21:36:47.100 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/?versions 2023-09-22T21:36:47.100 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:of+I5bbMCT+ScBceZh37DpbCy4g= 2023-09-22T21:36:47.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:of+I5bbMCT+ScBceZh37DpbCy4g=', 'Content-Length': '0'} 2023-09-22T21:36:47.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a97e5f14a931b478-00650e08ef-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.102 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:47.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T21:36:47.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T21:36:47.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T21:36:47.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T21:36:47.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T21:36:47.103 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:47.104 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:47.104 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:36:47.104 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:36:47.104 DEBUG:boto:Method: GET 2023-09-22T21:36:47.104 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:36:47.104 DEBUG:boto:Data: 2023-09-22T21:36:47.104 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.104 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.104 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.104 DEBUG:boto:Params: {} 2023-09-22T21:36:47.105 DEBUG:boto:Token: None 2023-09-22T21:36:47.105 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/?versions 2023-09-22T21:36:47.105 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:of+I5bbMCT+ScBceZh37DpbCy4g= 2023-09-22T21:36:47.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:of+I5bbMCT+ScBceZh37DpbCy4g=', 'Content-Length': '0'} 2023-09-22T21:36:47.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002aafc65f4952a38a-00650e08ef-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.107 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:47.108 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:47.108 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:47.108 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:36:47.108 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:36:47.108 DEBUG:boto:Method: GET 2023-09-22T21:36:47.108 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:36:47.108 DEBUG:boto:Data: 2023-09-22T21:36:47.108 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.109 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.109 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.109 DEBUG:boto:Params: {} 2023-09-22T21:36:47.109 DEBUG:boto:Token: None 2023-09-22T21:36:47.109 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/?versions 2023-09-22T21:36:47.109 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:of+I5bbMCT+ScBceZh37DpbCy4g= 2023-09-22T21:36:47.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:of+I5bbMCT+ScBceZh37DpbCy4g=', 'Content-Length': '0'} 2023-09-22T21:36:47.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022b3a869f71d9388-00650e08ef-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.111 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:47.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:36:47.112 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:36:47.112 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:36:47.112 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.112 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.113 DEBUG:boto:Method: GET 2023-09-22T21:36:47.113 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.113 DEBUG:boto:Data: 2023-09-22T21:36:47.113 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.113 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.113 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.113 DEBUG:boto:Params: {} 2023-09-22T21:36:47.113 DEBUG:boto:Token: None 2023-09-22T21:36:47.113 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.114 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1S3KmfNnQ9t1XnECNNmTcQzETq8= 2023-09-22T21:36:47.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1S3KmfNnQ9t1XnECNNmTcQzETq8=', 'Content-Length': '0'} 2023-09-22T21:36:47.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab8aca341b99cc2d-00650e08ef-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.116 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:36:47.116 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:36:47.116 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.116 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.116 DEBUG:boto:Method: GET 2023-09-22T21:36:47.117 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.117 DEBUG:boto:Data: 2023-09-22T21:36:47.117 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.117 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.117 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.117 DEBUG:boto:Params: {} 2023-09-22T21:36:47.117 DEBUG:boto:Token: None 2023-09-22T21:36:47.117 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.117 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1S3KmfNnQ9t1XnECNNmTcQzETq8= 2023-09-22T21:36:47.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1S3KmfNnQ9t1XnECNNmTcQzETq8=', 'Content-Length': '0'} 2023-09-22T21:36:47.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002574ee01a9983aa6-00650e08ef-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.162 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:36:47.163 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:36:47.163 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.163 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.163 DEBUG:boto:Method: HEAD 2023-09-22T21:36:47.163 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.163 DEBUG:boto:Data: 2023-09-22T21:36:47.163 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.163 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.164 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.164 DEBUG:boto:Params: {} 2023-09-22T21:36:47.164 DEBUG:boto:Token: None 2023-09-22T21:36:47.164 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.164 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4Db3PURHwUjxmo7zJUffhjuKLxg= 2023-09-22T21:36:47.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4Db3PURHwUjxmo7zJUffhjuKLxg=', 'Content-Length': '0'} 2023-09-22T21:36:47.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fb2d9a5af133e26-00650e08ef-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.167 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:36:47.167 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:36:47.167 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.167 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.167 DEBUG:boto:Method: HEAD 2023-09-22T21:36:47.167 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.167 DEBUG:boto:Data: 2023-09-22T21:36:47.168 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.168 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.168 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.168 DEBUG:boto:Params: {} 2023-09-22T21:36:47.168 DEBUG:boto:Token: None 2023-09-22T21:36:47.168 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.168 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4Db3PURHwUjxmo7zJUffhjuKLxg= 2023-09-22T21:36:47.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4Db3PURHwUjxmo7zJUffhjuKLxg=', 'Content-Length': '0'} 2023-09-22T21:36:47.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c627436d5ee7e22-00650e08ef-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:36:47.171 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:36:47.171 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:36:47.172 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.172 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.172 DEBUG:boto:Method: GET 2023-09-22T21:36:47.172 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.172 DEBUG:boto:Data: 2023-09-22T21:36:47.172 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.172 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.172 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.172 DEBUG:boto:Params: {} 2023-09-22T21:36:47.173 DEBUG:boto:Token: None 2023-09-22T21:36:47.173 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.173 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1S3KmfNnQ9t1XnECNNmTcQzETq8= 2023-09-22T21:36:47.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1S3KmfNnQ9t1XnECNNmTcQzETq8=', 'Content-Length': '0'} 2023-09-22T21:36:47.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000702097fe9734e652-00650e08ef-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.176 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:36:47.176 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:36:47.176 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.176 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.176 DEBUG:boto:Method: GET 2023-09-22T21:36:47.176 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.176 DEBUG:boto:Data: 2023-09-22T21:36:47.176 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.177 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.177 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.177 DEBUG:boto:Params: {} 2023-09-22T21:36:47.177 DEBUG:boto:Token: None 2023-09-22T21:36:47.177 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:36:47.177 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1S3KmfNnQ9t1XnECNNmTcQzETq8= 2023-09-22T21:36:47.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1S3KmfNnQ9t1XnECNNmTcQzETq8=', 'Content-Length': '0'} 2023-09-22T21:36:47.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003836c746c7cfac7b-00650e08ef-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.180 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:47.180 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:47.180 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:36:47.180 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:36:47.180 DEBUG:boto:Method: GET 2023-09-22T21:36:47.181 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:36:47.181 DEBUG:boto:Data: 2023-09-22T21:36:47.181 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.181 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.181 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.181 DEBUG:boto:Params: {} 2023-09-22T21:36:47.181 DEBUG:boto:Token: None 2023-09-22T21:36:47.181 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/ 2023-09-22T21:36:47.182 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk= 2023-09-22T21:36:47.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk=', 'Content-Length': '0'} 2023-09-22T21:36:47.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000739d9e8313ae99b3-00650e08ef-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.226 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:47.227 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:47.227 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:47.227 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:36:47.227 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:36:47.227 DEBUG:boto:Method: GET 2023-09-22T21:36:47.228 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:36:47.228 DEBUG:boto:Data: 2023-09-22T21:36:47.228 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.228 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.228 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.228 DEBUG:boto:Params: {} 2023-09-22T21:36:47.228 DEBUG:boto:Token: None 2023-09-22T21:36:47.228 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/ 2023-09-22T21:36:47.229 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk= 2023-09-22T21:36:47.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk=', 'Content-Length': '0'} 2023-09-22T21:36:47.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee2994601ef5ca2d-00650e08ef-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.232 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:47.232 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:36:47.233 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:36:47.233 DEBUG:boto:Method: HEAD 2023-09-22T21:36:47.233 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:36:47.233 DEBUG:boto:Data: 2023-09-22T21:36:47.234 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.234 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.234 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.234 DEBUG:boto:Params: {} 2023-09-22T21:36:47.234 DEBUG:boto:Token: None 2023-09-22T21:36:47.235 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj11 2023-09-22T21:36:47.235 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6wCOxS3cQVONAeYAYzwldLdqHs8= 2023-09-22T21:36:47.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6wCOxS3cQVONAeYAYzwldLdqHs8=', 'Content-Length': '0'} 2023-09-22T21:36:47.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b85ad8bbc72da4a-00650e08ef-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.238 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:36:47.238 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:36:47.238 DEBUG:boto:Method: HEAD 2023-09-22T21:36:47.238 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:36:47.238 DEBUG:boto:Data: 2023-09-22T21:36:47.239 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.239 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.239 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.239 DEBUG:boto:Params: {} 2023-09-22T21:36:47.239 DEBUG:boto:Token: None 2023-09-22T21:36:47.239 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj11 2023-09-22T21:36:47.239 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6wCOxS3cQVONAeYAYzwldLdqHs8= 2023-09-22T21:36:47.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6wCOxS3cQVONAeYAYzwldLdqHs8=', 'Content-Length': '0'} 2023-09-22T21:36:47.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079d3788e6dd389e5-00650e08ef-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:36:47.242 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:36:47.242 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:36:47.242 DEBUG:boto:Method: GET 2023-09-22T21:36:47.243 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:36:47.243 DEBUG:boto:Data: 2023-09-22T21:36:47.243 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.243 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.243 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.243 DEBUG:boto:Params: {} 2023-09-22T21:36:47.243 DEBUG:boto:Token: None 2023-09-22T21:36:47.243 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj11 2023-09-22T21:36:47.243 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wPunaOqstH4K6QfQmxinoyXXKEI= 2023-09-22T21:36:47.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wPunaOqstH4K6QfQmxinoyXXKEI=', 'Content-Length': '0'} 2023-09-22T21:36:47.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e23ad92182294d3-00650e08ef-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.286 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:36:47.287 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:36:47.287 DEBUG:boto:Method: GET 2023-09-22T21:36:47.287 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:36:47.287 DEBUG:boto:Data: 2023-09-22T21:36:47.287 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.287 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.288 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.288 DEBUG:boto:Params: {} 2023-09-22T21:36:47.288 DEBUG:boto:Token: None 2023-09-22T21:36:47.288 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj11 2023-09-22T21:36:47.289 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wPunaOqstH4K6QfQmxinoyXXKEI= 2023-09-22T21:36:47.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wPunaOqstH4K6QfQmxinoyXXKEI=', 'Content-Length': '0'} 2023-09-22T21:36:47.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000597c567dfe34dcce-00650e08ef-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.292 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:47.292 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:47.293 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:36:47.293 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:36:47.293 DEBUG:boto:Method: GET 2023-09-22T21:36:47.293 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:36:47.293 DEBUG:boto:Data: 2023-09-22T21:36:47.293 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.293 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.293 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.293 DEBUG:boto:Params: {} 2023-09-22T21:36:47.294 DEBUG:boto:Token: None 2023-09-22T21:36:47.294 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/ 2023-09-22T21:36:47.294 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk= 2023-09-22T21:36:47.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk=', 'Content-Length': '0'} 2023-09-22T21:36:47.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005684b6ceebedbb4-00650e08ef-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.346 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:47.347 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:47.347 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:47.347 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:36:47.348 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:36:47.348 DEBUG:boto:Method: GET 2023-09-22T21:36:47.348 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:36:47.348 DEBUG:boto:Data: 2023-09-22T21:36:47.348 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.349 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.349 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.349 DEBUG:boto:Params: {} 2023-09-22T21:36:47.349 DEBUG:boto:Token: None 2023-09-22T21:36:47.350 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/ 2023-09-22T21:36:47.350 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk= 2023-09-22T21:36:47.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk=', 'Content-Length': '0'} 2023-09-22T21:36:47.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063ad6b578e107c47-00650e08ef-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.355 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:47.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:36:47.355 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:36:47.355 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:36:47.355 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.355 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.355 DEBUG:boto:Method: GET 2023-09-22T21:36:47.356 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.356 DEBUG:boto:Data: 2023-09-22T21:36:47.356 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.356 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.356 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.356 DEBUG:boto:Params: {} 2023-09-22T21:36:47.356 DEBUG:boto:Token: None 2023-09-22T21:36:47.356 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.357 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zmm6C3TMD7Jk3SwKdyDnjz+S/3M= 2023-09-22T21:36:47.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zmm6C3TMD7Jk3SwKdyDnjz+S/3M=', 'Content-Length': '0'} 2023-09-22T21:36:47.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000746abc8945569be7-00650e08ef-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.402 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:36:47.402 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:36:47.403 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.403 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.403 DEBUG:boto:Method: GET 2023-09-22T21:36:47.403 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.403 DEBUG:boto:Data: 2023-09-22T21:36:47.403 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.403 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.403 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.403 DEBUG:boto:Params: {} 2023-09-22T21:36:47.403 DEBUG:boto:Token: None 2023-09-22T21:36:47.404 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.404 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zmm6C3TMD7Jk3SwKdyDnjz+S/3M= 2023-09-22T21:36:47.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zmm6C3TMD7Jk3SwKdyDnjz+S/3M=', 'Content-Length': '0'} 2023-09-22T21:36:47.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed086164ce469b77-00650e08ef-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.407 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:36:47.407 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:36:47.407 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.407 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.407 DEBUG:boto:Method: HEAD 2023-09-22T21:36:47.408 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.408 DEBUG:boto:Data: 2023-09-22T21:36:47.408 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.408 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.408 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.408 DEBUG:boto:Params: {} 2023-09-22T21:36:47.408 DEBUG:boto:Token: None 2023-09-22T21:36:47.408 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.408 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C4Anr0bFisyAQ7nrsH1EqXDYLdw= 2023-09-22T21:36:47.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C4Anr0bFisyAQ7nrsH1EqXDYLdw=', 'Content-Length': '0'} 2023-09-22T21:36:47.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bab3406831c8e467-00650e08ef-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.411 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:36:47.411 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:36:47.412 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.412 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.412 DEBUG:boto:Method: HEAD 2023-09-22T21:36:47.412 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.412 DEBUG:boto:Data: 2023-09-22T21:36:47.412 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.412 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.412 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.412 DEBUG:boto:Params: {} 2023-09-22T21:36:47.413 DEBUG:boto:Token: None 2023-09-22T21:36:47.413 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.413 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C4Anr0bFisyAQ7nrsH1EqXDYLdw= 2023-09-22T21:36:47.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C4Anr0bFisyAQ7nrsH1EqXDYLdw=', 'Content-Length': '0'} 2023-09-22T21:36:47.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b3842be686c8d33-00650e08ef-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:36:47.416 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:36:47.416 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:36:47.416 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.416 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.416 DEBUG:boto:Method: GET 2023-09-22T21:36:47.416 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.416 DEBUG:boto:Data: 2023-09-22T21:36:47.416 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.417 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.417 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.417 DEBUG:boto:Params: {} 2023-09-22T21:36:47.417 DEBUG:boto:Token: None 2023-09-22T21:36:47.417 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.417 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zmm6C3TMD7Jk3SwKdyDnjz+S/3M= 2023-09-22T21:36:47.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zmm6C3TMD7Jk3SwKdyDnjz+S/3M=', 'Content-Length': '0'} 2023-09-22T21:36:47.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d67e51d224658934-00650e08ef-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.462 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:36:47.462 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:36:47.463 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.463 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.463 DEBUG:boto:Method: GET 2023-09-22T21:36:47.463 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.463 DEBUG:boto:Data: 2023-09-22T21:36:47.463 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.463 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.463 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.463 DEBUG:boto:Params: {} 2023-09-22T21:36:47.464 DEBUG:boto:Token: None 2023-09-22T21:36:47.464 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:36:47.464 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zmm6C3TMD7Jk3SwKdyDnjz+S/3M= 2023-09-22T21:36:47.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zmm6C3TMD7Jk3SwKdyDnjz+S/3M=', 'Content-Length': '0'} 2023-09-22T21:36:47.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a0192e952a25052-00650e08ef-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.467 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:47.467 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:47.467 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:36:47.467 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:36:47.468 DEBUG:boto:Method: GET 2023-09-22T21:36:47.468 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:36:47.468 DEBUG:boto:Data: 2023-09-22T21:36:47.468 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.468 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.468 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.468 DEBUG:boto:Params: {} 2023-09-22T21:36:47.468 DEBUG:boto:Token: None 2023-09-22T21:36:47.468 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/ 2023-09-22T21:36:47.469 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk= 2023-09-22T21:36:47.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk=', 'Content-Length': '0'} 2023-09-22T21:36:47.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b10235174bd370e-00650e08ef-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.518 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:47.519 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:47.519 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:47.519 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:36:47.519 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:36:47.519 DEBUG:boto:Method: GET 2023-09-22T21:36:47.519 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:36:47.519 DEBUG:boto:Data: 2023-09-22T21:36:47.519 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.520 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.520 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.520 DEBUG:boto:Params: {} 2023-09-22T21:36:47.520 DEBUG:boto:Token: None 2023-09-22T21:36:47.520 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/ 2023-09-22T21:36:47.520 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk= 2023-09-22T21:36:47.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk=', 'Content-Length': '0'} 2023-09-22T21:36:47.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000484870297656f93c-00650e08ef-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.526 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:47.526 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:36:47.526 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:36:47.527 DEBUG:boto:Method: HEAD 2023-09-22T21:36:47.527 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:36:47.527 DEBUG:boto:Data: 2023-09-22T21:36:47.527 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.527 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.527 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.527 DEBUG:boto:Params: {} 2023-09-22T21:36:47.527 DEBUG:boto:Token: None 2023-09-22T21:36:47.527 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj22 2023-09-22T21:36:47.527 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H1LTYVYZUQxfG+nm7ZlDc7k1/c8= 2023-09-22T21:36:47.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H1LTYVYZUQxfG+nm7ZlDc7k1/c8=', 'Content-Length': '0'} 2023-09-22T21:36:47.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fbc804982240a21-00650e08ef-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.531 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:36:47.531 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:36:47.531 DEBUG:boto:Method: HEAD 2023-09-22T21:36:47.531 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:36:47.531 DEBUG:boto:Data: 2023-09-22T21:36:47.531 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.532 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.532 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.532 DEBUG:boto:Params: {} 2023-09-22T21:36:47.532 DEBUG:boto:Token: None 2023-09-22T21:36:47.532 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj22 2023-09-22T21:36:47.533 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H1LTYVYZUQxfG+nm7ZlDc7k1/c8= 2023-09-22T21:36:47.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H1LTYVYZUQxfG+nm7ZlDc7k1/c8=', 'Content-Length': '0'} 2023-09-22T21:36:47.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c1938b867532a14-00650e08ef-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:36:47.536 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:36:47.536 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:36:47.536 DEBUG:boto:Method: GET 2023-09-22T21:36:47.536 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:36:47.537 DEBUG:boto:Data: 2023-09-22T21:36:47.537 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.537 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.537 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.537 DEBUG:boto:Params: {} 2023-09-22T21:36:47.537 DEBUG:boto:Token: None 2023-09-22T21:36:47.538 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj22 2023-09-22T21:36:47.538 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bwREp1oKxYVRLtdsKnClx5tNcro= 2023-09-22T21:36:47.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bwREp1oKxYVRLtdsKnClx5tNcro=', 'Content-Length': '0'} 2023-09-22T21:36:47.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008986f66dd11f9cc2-00650e08ef-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.582 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:36:47.583 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:36:47.583 DEBUG:boto:Method: GET 2023-09-22T21:36:47.583 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:36:47.583 DEBUG:boto:Data: 2023-09-22T21:36:47.583 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.583 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.583 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.583 DEBUG:boto:Params: {} 2023-09-22T21:36:47.584 DEBUG:boto:Token: None 2023-09-22T21:36:47.584 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj22 2023-09-22T21:36:47.584 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bwREp1oKxYVRLtdsKnClx5tNcro= 2023-09-22T21:36:47.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bwREp1oKxYVRLtdsKnClx5tNcro=', 'Content-Length': '0'} 2023-09-22T21:36:47.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d3f7ffa32d9f6a0-00650e08ef-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.587 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:47.587 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:47.587 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:36:47.588 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:36:47.588 DEBUG:boto:Method: GET 2023-09-22T21:36:47.588 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:36:47.588 DEBUG:boto:Data: 2023-09-22T21:36:47.588 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.588 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.588 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.588 DEBUG:boto:Params: {} 2023-09-22T21:36:47.588 DEBUG:boto:Token: None 2023-09-22T21:36:47.588 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/ 2023-09-22T21:36:47.589 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk= 2023-09-22T21:36:47.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk=', 'Content-Length': '0'} 2023-09-22T21:36:47.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000689ebcb3f8745b96-00650e08ef-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.638 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:47.639 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:47.639 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:47.639 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:36:47.639 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:36:47.639 DEBUG:boto:Method: GET 2023-09-22T21:36:47.639 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:36:47.639 DEBUG:boto:Data: 2023-09-22T21:36:47.639 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.639 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.640 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.640 DEBUG:boto:Params: {} 2023-09-22T21:36:47.640 DEBUG:boto:Token: None 2023-09-22T21:36:47.640 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/ 2023-09-22T21:36:47.640 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk= 2023-09-22T21:36:47.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk=', 'Content-Length': '0'} 2023-09-22T21:36:47.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebb079f3ef6ee06c-00650e08ef-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.645 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:47.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:36:47.645 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:36:47.645 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:36:47.645 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.645 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.646 DEBUG:boto:Method: GET 2023-09-22T21:36:47.646 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.646 DEBUG:boto:Data: 2023-09-22T21:36:47.646 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.646 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.646 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.646 DEBUG:boto:Params: {} 2023-09-22T21:36:47.646 DEBUG:boto:Token: None 2023-09-22T21:36:47.646 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.647 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KI8l4dlRMPH0s2BaWCAHeWIOhk0= 2023-09-22T21:36:47.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KI8l4dlRMPH0s2BaWCAHeWIOhk0=', 'Content-Length': '0'} 2023-09-22T21:36:47.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004205dda3161dc46f-00650e08ef-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.650 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:36:47.650 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:36:47.650 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.650 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.650 DEBUG:boto:Method: GET 2023-09-22T21:36:47.650 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.651 DEBUG:boto:Data: 2023-09-22T21:36:47.651 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.651 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.651 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.651 DEBUG:boto:Params: {} 2023-09-22T21:36:47.651 DEBUG:boto:Token: None 2023-09-22T21:36:47.651 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.651 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KI8l4dlRMPH0s2BaWCAHeWIOhk0= 2023-09-22T21:36:47.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KI8l4dlRMPH0s2BaWCAHeWIOhk0=', 'Content-Length': '0'} 2023-09-22T21:36:47.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002250a304aebb134b-00650e08ef-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.698 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:36:47.698 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:36:47.699 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.699 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.699 DEBUG:boto:Method: HEAD 2023-09-22T21:36:47.699 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.699 DEBUG:boto:Data: 2023-09-22T21:36:47.699 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.699 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.699 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.699 DEBUG:boto:Params: {} 2023-09-22T21:36:47.699 DEBUG:boto:Token: None 2023-09-22T21:36:47.700 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.700 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:21zOafXsGh7jEokw+98Ybxv1J74= 2023-09-22T21:36:47.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:21zOafXsGh7jEokw+98Ybxv1J74=', 'Content-Length': '0'} 2023-09-22T21:36:47.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006a0930415a5d39a-00650e08ef-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.703 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:36:47.703 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:36:47.703 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.703 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.703 DEBUG:boto:Method: HEAD 2023-09-22T21:36:47.703 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.703 DEBUG:boto:Data: 2023-09-22T21:36:47.703 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.704 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.704 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.704 DEBUG:boto:Params: {} 2023-09-22T21:36:47.704 DEBUG:boto:Token: None 2023-09-22T21:36:47.704 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.704 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:21zOafXsGh7jEokw+98Ybxv1J74= 2023-09-22T21:36:47.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:21zOafXsGh7jEokw+98Ybxv1J74=', 'Content-Length': '0'} 2023-09-22T21:36:47.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be58f3570a3eee3b-00650e08ef-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:36:47.707 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:36:47.707 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:36:47.707 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.707 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.707 DEBUG:boto:Method: GET 2023-09-22T21:36:47.708 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.708 DEBUG:boto:Data: 2023-09-22T21:36:47.708 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.708 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.708 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.708 DEBUG:boto:Params: {} 2023-09-22T21:36:47.708 DEBUG:boto:Token: None 2023-09-22T21:36:47.708 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.708 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KI8l4dlRMPH0s2BaWCAHeWIOhk0= 2023-09-22T21:36:47.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KI8l4dlRMPH0s2BaWCAHeWIOhk0=', 'Content-Length': '0'} 2023-09-22T21:36:47.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023f7b3c02695ddb0-00650e08ef-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.754 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:36:47.754 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:36:47.755 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.755 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.755 DEBUG:boto:Method: GET 2023-09-22T21:36:47.755 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.755 DEBUG:boto:Data: 2023-09-22T21:36:47.755 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.755 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.755 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.755 DEBUG:boto:Params: {} 2023-09-22T21:36:47.755 DEBUG:boto:Token: None 2023-09-22T21:36:47.756 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:36:47.756 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KI8l4dlRMPH0s2BaWCAHeWIOhk0= 2023-09-22T21:36:47.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KI8l4dlRMPH0s2BaWCAHeWIOhk0=', 'Content-Length': '0'} 2023-09-22T21:36:47.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cad3b74277fd6142-00650e08ef-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.759 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:47.759 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:47.759 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:36:47.759 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:36:47.759 DEBUG:boto:Method: GET 2023-09-22T21:36:47.760 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:36:47.760 DEBUG:boto:Data: 2023-09-22T21:36:47.760 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.760 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.760 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.760 DEBUG:boto:Params: {} 2023-09-22T21:36:47.760 DEBUG:boto:Token: None 2023-09-22T21:36:47.760 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/ 2023-09-22T21:36:47.760 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk= 2023-09-22T21:36:47.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk=', 'Content-Length': '0'} 2023-09-22T21:36:47.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a8948a4cbdc8019-00650e08ef-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.806 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:47.807 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:47.807 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:47.807 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:36:47.807 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:36:47.807 DEBUG:boto:Method: GET 2023-09-22T21:36:47.807 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:36:47.807 DEBUG:boto:Data: 2023-09-22T21:36:47.807 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.807 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.807 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.808 DEBUG:boto:Params: {} 2023-09-22T21:36:47.808 DEBUG:boto:Token: None 2023-09-22T21:36:47.808 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/ 2023-09-22T21:36:47.808 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk= 2023-09-22T21:36:47.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk=', 'Content-Length': '0'} 2023-09-22T21:36:47.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4ede079e6a4cc93-00650e08ef-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.814 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:47.814 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:36:47.814 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:36:47.814 DEBUG:boto:Method: HEAD 2023-09-22T21:36:47.815 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:36:47.815 DEBUG:boto:Data: 2023-09-22T21:36:47.815 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.815 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.815 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.815 DEBUG:boto:Params: {} 2023-09-22T21:36:47.815 DEBUG:boto:Token: None 2023-09-22T21:36:47.815 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj33 2023-09-22T21:36:47.815 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dzlRqWaIMjFPDDuTdLwIcnQK7+c= 2023-09-22T21:36:47.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dzlRqWaIMjFPDDuTdLwIcnQK7+c=', 'Content-Length': '0'} 2023-09-22T21:36:47.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e641ab6d29b812d3-00650e08ef-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.818 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:36:47.819 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:36:47.819 DEBUG:boto:Method: HEAD 2023-09-22T21:36:47.819 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:36:47.819 DEBUG:boto:Data: 2023-09-22T21:36:47.819 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.819 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.819 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.819 DEBUG:boto:Params: {} 2023-09-22T21:36:47.819 DEBUG:boto:Token: None 2023-09-22T21:36:47.820 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj33 2023-09-22T21:36:47.820 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dzlRqWaIMjFPDDuTdLwIcnQK7+c= 2023-09-22T21:36:47.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dzlRqWaIMjFPDDuTdLwIcnQK7+c=', 'Content-Length': '0'} 2023-09-22T21:36:47.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002395e386fcc281d2-00650e08ef-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:36:47.822 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:36:47.823 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:36:47.823 DEBUG:boto:Method: GET 2023-09-22T21:36:47.823 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:36:47.823 DEBUG:boto:Data: 2023-09-22T21:36:47.823 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.823 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.823 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.823 DEBUG:boto:Params: {} 2023-09-22T21:36:47.823 DEBUG:boto:Token: None 2023-09-22T21:36:47.824 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj33 2023-09-22T21:36:47.824 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mHL0UqDneiCyjZYH382PtKyxFJQ= 2023-09-22T21:36:47.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mHL0UqDneiCyjZYH382PtKyxFJQ=', 'Content-Length': '0'} 2023-09-22T21:36:47.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dba5e6bf632fc8c-00650e08ef-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.874 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:36:47.874 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:36:47.875 DEBUG:boto:Method: GET 2023-09-22T21:36:47.875 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:36:47.875 DEBUG:boto:Data: 2023-09-22T21:36:47.875 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.875 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.875 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.875 DEBUG:boto:Params: {} 2023-09-22T21:36:47.875 DEBUG:boto:Token: None 2023-09-22T21:36:47.875 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj33 2023-09-22T21:36:47.876 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mHL0UqDneiCyjZYH382PtKyxFJQ= 2023-09-22T21:36:47.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mHL0UqDneiCyjZYH382PtKyxFJQ=', 'Content-Length': '0'} 2023-09-22T21:36:47.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3169add55cbe802-00650e08ef-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.879 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:47.879 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:47.879 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:36:47.879 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:36:47.879 DEBUG:boto:Method: GET 2023-09-22T21:36:47.879 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:36:47.879 DEBUG:boto:Data: 2023-09-22T21:36:47.879 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.880 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.880 DEBUG:boto:Params: {} 2023-09-22T21:36:47.880 DEBUG:boto:Token: None 2023-09-22T21:36:47.880 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/ 2023-09-22T21:36:47.880 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk= 2023-09-22T21:36:47.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk=', 'Content-Length': '0'} 2023-09-22T21:36:47.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c324b4803104b71-00650e08ef-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.926 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:47.927 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:47.927 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:47.927 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:36:47.927 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:36:47.927 DEBUG:boto:Method: GET 2023-09-22T21:36:47.927 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:36:47.927 DEBUG:boto:Data: 2023-09-22T21:36:47.927 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.927 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.927 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.928 DEBUG:boto:Params: {} 2023-09-22T21:36:47.928 DEBUG:boto:Token: None 2023-09-22T21:36:47.928 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/ 2023-09-22T21:36:47.928 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk= 2023-09-22T21:36:47.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:apBHwxVfaUxMxsGL/fCwqeuVgPk=', 'Content-Length': '0'} 2023-09-22T21:36:47.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1afe3839e477615-00650e08ef-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.933 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:47.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:36:47.934 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:36:47.934 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:36:47.934 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:47.934 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:47.934 DEBUG:boto:Method: GET 2023-09-22T21:36:47.934 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:47.934 DEBUG:boto:Data: 2023-09-22T21:36:47.934 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.935 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.935 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.935 DEBUG:boto:Params: {} 2023-09-22T21:36:47.935 DEBUG:boto:Token: None 2023-09-22T21:36:47.935 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:47.935 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:snLh1WQ1DxuVRjyk+hGoCeB24Mw= 2023-09-22T21:36:47.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:snLh1WQ1DxuVRjyk+hGoCeB24Mw=', 'Content-Length': '0'} 2023-09-22T21:36:47.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f26fca66c8e70196-00650e08ef-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.939 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:36:47.939 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:36:47.939 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:47.939 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:47.939 DEBUG:boto:Method: GET 2023-09-22T21:36:47.939 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:47.939 DEBUG:boto:Data: 2023-09-22T21:36:47.939 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.939 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.939 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.940 DEBUG:boto:Params: {} 2023-09-22T21:36:47.940 DEBUG:boto:Token: None 2023-09-22T21:36:47.940 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:47.940 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:snLh1WQ1DxuVRjyk+hGoCeB24Mw= 2023-09-22T21:36:47.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:snLh1WQ1DxuVRjyk+hGoCeB24Mw=', 'Content-Length': '0'} 2023-09-22T21:36:47.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ab449f259f6d16a-00650e08ef-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.986 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:36:47.986 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:36:47.987 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:47.987 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:47.987 DEBUG:boto:Method: HEAD 2023-09-22T21:36:47.987 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:47.987 DEBUG:boto:Data: 2023-09-22T21:36:47.987 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.987 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.987 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.987 DEBUG:boto:Params: {} 2023-09-22T21:36:47.987 DEBUG:boto:Token: None 2023-09-22T21:36:47.988 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:47.988 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VKo6XQnJoVTQdWug1BiVHOpnE/U= 2023-09-22T21:36:47.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VKo6XQnJoVTQdWug1BiVHOpnE/U=', 'Content-Length': '0'} 2023-09-22T21:36:47.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008747a97946723e47-00650e08ef-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.991 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:36:47.991 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:36:47.991 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:47.991 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:47.991 DEBUG:boto:Method: HEAD 2023-09-22T21:36:47.991 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:47.992 DEBUG:boto:Data: 2023-09-22T21:36:47.992 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.992 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:47.992 DEBUG:boto:Port: 8000 2023-09-22T21:36:47.992 DEBUG:boto:Params: {} 2023-09-22T21:36:47.992 DEBUG:boto:Token: None 2023-09-22T21:36:47.992 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:47.992 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VKo6XQnJoVTQdWug1BiVHOpnE/U= 2023-09-22T21:36:47.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VKo6XQnJoVTQdWug1BiVHOpnE/U=', 'Content-Length': '0'} 2023-09-22T21:36:47.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081d3c55ce537b2f2-00650e08ef-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:47.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:36:47.995 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:36:47.995 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:36:47.995 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:47.996 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:47.996 DEBUG:boto:Method: GET 2023-09-22T21:36:47.996 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:47.996 DEBUG:boto:Data: 2023-09-22T21:36:47.996 DEBUG:boto:Headers: {} 2023-09-22T21:36:47.996 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:47.996 DEBUG:boto:Port: 8001 2023-09-22T21:36:47.996 DEBUG:boto:Params: {} 2023-09-22T21:36:47.996 DEBUG:boto:Token: None 2023-09-22T21:36:47.996 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:47 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:47.997 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:snLh1WQ1DxuVRjyk+hGoCeB24Mw= 2023-09-22T21:36:47.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:47 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:snLh1WQ1DxuVRjyk+hGoCeB24Mw=', 'Content-Length': '0'} 2023-09-22T21:36:47.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b548e7aeb5ae0b5-00650e08ef-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.046 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:36:48.046 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:36:48.046 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:48.047 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:48.047 DEBUG:boto:Method: GET 2023-09-22T21:36:48.047 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:48.047 DEBUG:boto:Data: 2023-09-22T21:36:48.047 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.047 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.047 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.047 DEBUG:boto:Params: {} 2023-09-22T21:36:48.047 DEBUG:boto:Token: None 2023-09-22T21:36:48.048 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:36:48.048 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HzUfZEaz67GioKYqFTdLxgCnZjw= 2023-09-22T21:36:48.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HzUfZEaz67GioKYqFTdLxgCnZjw=', 'Content-Length': '0'} 2023-09-22T21:36:48.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd05e04f257f5b27-00650e08f0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.051 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:48.051 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:48.051 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:36:48.051 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:36:48.051 DEBUG:boto:Method: GET 2023-09-22T21:36:48.051 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:36:48.051 DEBUG:boto:Data: 2023-09-22T21:36:48.051 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.051 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:48.051 DEBUG:boto:Port: 8001 2023-09-22T21:36:48.052 DEBUG:boto:Params: {} 2023-09-22T21:36:48.052 DEBUG:boto:Token: None 2023-09-22T21:36:48.052 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/ 2023-09-22T21:36:48.052 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:poyIKuFM7iAeViGoK3zozOQH+Ig= 2023-09-22T21:36:48.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:poyIKuFM7iAeViGoK3zozOQH+Ig=', 'Content-Length': '0'} 2023-09-22T21:36:48.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd831e9b6094419f-00650e08f0-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.098 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:48.099 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:48.099 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:48.099 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:36:48.099 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:36:48.099 DEBUG:boto:Method: GET 2023-09-22T21:36:48.099 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:36:48.099 DEBUG:boto:Data: 2023-09-22T21:36:48.099 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.099 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.099 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.100 DEBUG:boto:Params: {} 2023-09-22T21:36:48.100 DEBUG:boto:Token: None 2023-09-22T21:36:48.100 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/ 2023-09-22T21:36:48.100 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:poyIKuFM7iAeViGoK3zozOQH+Ig= 2023-09-22T21:36:48.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:poyIKuFM7iAeViGoK3zozOQH+Ig=', 'Content-Length': '0'} 2023-09-22T21:36:48.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf3b421d39f9586f-00650e08f0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.105 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:48.105 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:36:48.105 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:36:48.105 DEBUG:boto:Method: HEAD 2023-09-22T21:36:48.105 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:36:48.106 DEBUG:boto:Data: 2023-09-22T21:36:48.106 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.106 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:48.106 DEBUG:boto:Port: 8001 2023-09-22T21:36:48.106 DEBUG:boto:Params: {} 2023-09-22T21:36:48.106 DEBUG:boto:Token: None 2023-09-22T21:36:48.106 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/obj44 2023-09-22T21:36:48.106 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TZZOZ2GoLhVuVTtPJIZc4LL4C+c= 2023-09-22T21:36:48.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TZZOZ2GoLhVuVTtPJIZc4LL4C+c=', 'Content-Length': '0'} 2023-09-22T21:36:48.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007b4adf7af403998-00650e08f0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.109 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:36:48.109 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:36:48.109 DEBUG:boto:Method: HEAD 2023-09-22T21:36:48.109 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:36:48.110 DEBUG:boto:Data: 2023-09-22T21:36:48.110 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.110 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.110 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.110 DEBUG:boto:Params: {} 2023-09-22T21:36:48.110 DEBUG:boto:Token: None 2023-09-22T21:36:48.110 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/obj44 2023-09-22T21:36:48.110 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TZZOZ2GoLhVuVTtPJIZc4LL4C+c= 2023-09-22T21:36:48.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TZZOZ2GoLhVuVTtPJIZc4LL4C+c=', 'Content-Length': '0'} 2023-09-22T21:36:48.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002506bea5a782759d-00650e08f0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:36:48.114 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:36:48.114 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:36:48.114 DEBUG:boto:Method: GET 2023-09-22T21:36:48.114 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:36:48.114 DEBUG:boto:Data: 2023-09-22T21:36:48.115 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.115 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:48.115 DEBUG:boto:Port: 8001 2023-09-22T21:36:48.115 DEBUG:boto:Params: {} 2023-09-22T21:36:48.115 DEBUG:boto:Token: None 2023-09-22T21:36:48.115 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/obj44 2023-09-22T21:36:48.115 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gWP//7Q65WOrlFtCOZe3ktsbRMk= 2023-09-22T21:36:48.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gWP//7Q65WOrlFtCOZe3ktsbRMk=', 'Content-Length': '0'} 2023-09-22T21:36:48.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c64c11703ef56e4-00650e08f0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.162 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:36:48.162 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:36:48.163 DEBUG:boto:Method: GET 2023-09-22T21:36:48.163 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:36:48.163 DEBUG:boto:Data: 2023-09-22T21:36:48.163 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.163 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.163 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.163 DEBUG:boto:Params: {} 2023-09-22T21:36:48.163 DEBUG:boto:Token: None 2023-09-22T21:36:48.163 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/obj44 2023-09-22T21:36:48.164 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gWP//7Q65WOrlFtCOZe3ktsbRMk= 2023-09-22T21:36:48.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gWP//7Q65WOrlFtCOZe3ktsbRMk=', 'Content-Length': '0'} 2023-09-22T21:36:48.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f4b31ee064ae2c9-00650e08f0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.167 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:48.167 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:48.167 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:36:48.167 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:36:48.167 DEBUG:boto:Method: GET 2023-09-22T21:36:48.167 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:36:48.167 DEBUG:boto:Data: 2023-09-22T21:36:48.167 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.167 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:48.168 DEBUG:boto:Port: 8001 2023-09-22T21:36:48.168 DEBUG:boto:Params: {} 2023-09-22T21:36:48.168 DEBUG:boto:Token: None 2023-09-22T21:36:48.168 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/ 2023-09-22T21:36:48.168 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:poyIKuFM7iAeViGoK3zozOQH+Ig= 2023-09-22T21:36:48.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:poyIKuFM7iAeViGoK3zozOQH+Ig=', 'Content-Length': '0'} 2023-09-22T21:36:48.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009025c3c97e1d1425-00650e08f0-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.173 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:48.174 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:48.174 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:48.174 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:36:48.174 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:36:48.174 DEBUG:boto:Method: GET 2023-09-22T21:36:48.174 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:36:48.174 DEBUG:boto:Data: 2023-09-22T21:36:48.175 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.175 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.175 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.175 DEBUG:boto:Params: {} 2023-09-22T21:36:48.175 DEBUG:boto:Token: None 2023-09-22T21:36:48.175 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/ 2023-09-22T21:36:48.175 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:poyIKuFM7iAeViGoK3zozOQH+Ig= 2023-09-22T21:36:48.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:poyIKuFM7iAeViGoK3zozOQH+Ig=', 'Content-Length': '0'} 2023-09-22T21:36:48.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000836aaf9d94b62b1-00650e08f0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.222 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:48.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:36:48.223 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:36:48.223 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:36:48.223 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.223 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.223 DEBUG:boto:Method: GET 2023-09-22T21:36:48.223 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.223 DEBUG:boto:Data: 2023-09-22T21:36:48.223 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.224 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:48.224 DEBUG:boto:Port: 8001 2023-09-22T21:36:48.224 DEBUG:boto:Params: {} 2023-09-22T21:36:48.224 DEBUG:boto:Token: None 2023-09-22T21:36:48.224 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.224 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eAHFXJsCHr0+650KmAHS7M0iM0k= 2023-09-22T21:36:48.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eAHFXJsCHr0+650KmAHS7M0iM0k=', 'Content-Length': '0'} 2023-09-22T21:36:48.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4e8c7fbc9a33e22-00650e08f0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.270 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:36:48.270 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:36:48.271 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.271 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.271 DEBUG:boto:Method: GET 2023-09-22T21:36:48.271 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.271 DEBUG:boto:Data: 2023-09-22T21:36:48.271 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.271 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.271 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.271 DEBUG:boto:Params: {} 2023-09-22T21:36:48.271 DEBUG:boto:Token: None 2023-09-22T21:36:48.272 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.272 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eAHFXJsCHr0+650KmAHS7M0iM0k= 2023-09-22T21:36:48.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eAHFXJsCHr0+650KmAHS7M0iM0k=', 'Content-Length': '0'} 2023-09-22T21:36:48.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d4c6cbbf3bbf1a2-00650e08f0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.275 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:36:48.275 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:36:48.275 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.275 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.276 DEBUG:boto:Method: HEAD 2023-09-22T21:36:48.276 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.276 DEBUG:boto:Data: 2023-09-22T21:36:48.276 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.276 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:48.276 DEBUG:boto:Port: 8001 2023-09-22T21:36:48.276 DEBUG:boto:Params: {} 2023-09-22T21:36:48.276 DEBUG:boto:Token: None 2023-09-22T21:36:48.276 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.276 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Xyh08WJxCsmos4E2ZERWh/mh+nw= 2023-09-22T21:36:48.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Xyh08WJxCsmos4E2ZERWh/mh+nw=', 'Content-Length': '0'} 2023-09-22T21:36:48.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032f7e52315e9dcb1-00650e08f0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.280 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:36:48.280 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:36:48.280 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.280 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.280 DEBUG:boto:Method: HEAD 2023-09-22T21:36:48.280 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.280 DEBUG:boto:Data: 2023-09-22T21:36:48.280 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.280 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.280 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.281 DEBUG:boto:Params: {} 2023-09-22T21:36:48.281 DEBUG:boto:Token: None 2023-09-22T21:36:48.281 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.281 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Xyh08WJxCsmos4E2ZERWh/mh+nw= 2023-09-22T21:36:48.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Xyh08WJxCsmos4E2ZERWh/mh+nw=', 'Content-Length': '0'} 2023-09-22T21:36:48.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ad8d9aa7a11ea71-00650e08f0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:36:48.284 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:36:48.284 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:36:48.284 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.284 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.284 DEBUG:boto:Method: GET 2023-09-22T21:36:48.284 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.284 DEBUG:boto:Data: 2023-09-22T21:36:48.285 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.285 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:48.285 DEBUG:boto:Port: 8001 2023-09-22T21:36:48.285 DEBUG:boto:Params: {} 2023-09-22T21:36:48.285 DEBUG:boto:Token: None 2023-09-22T21:36:48.285 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.285 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eAHFXJsCHr0+650KmAHS7M0iM0k= 2023-09-22T21:36:48.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eAHFXJsCHr0+650KmAHS7M0iM0k=', 'Content-Length': '0'} 2023-09-22T21:36:48.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe188a6101eb79cf-00650e08f0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.330 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:36:48.330 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:36:48.331 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.331 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.331 DEBUG:boto:Method: GET 2023-09-22T21:36:48.331 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.331 DEBUG:boto:Data: 2023-09-22T21:36:48.331 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.331 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.331 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.331 DEBUG:boto:Params: {} 2023-09-22T21:36:48.331 DEBUG:boto:Token: None 2023-09-22T21:36:48.332 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:36:48.332 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eAHFXJsCHr0+650KmAHS7M0iM0k= 2023-09-22T21:36:48.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eAHFXJsCHr0+650KmAHS7M0iM0k=', 'Content-Length': '0'} 2023-09-22T21:36:48.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cceb6ddbb55ec60-00650e08f0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.335 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:48.335 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:48.335 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:36:48.335 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:36:48.335 DEBUG:boto:Method: GET 2023-09-22T21:36:48.335 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:36:48.336 DEBUG:boto:Data: 2023-09-22T21:36:48.336 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.336 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:48.336 DEBUG:boto:Port: 8001 2023-09-22T21:36:48.336 DEBUG:boto:Params: {} 2023-09-22T21:36:48.336 DEBUG:boto:Token: None 2023-09-22T21:36:48.336 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/ 2023-09-22T21:36:48.336 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:poyIKuFM7iAeViGoK3zozOQH+Ig= 2023-09-22T21:36:48.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:poyIKuFM7iAeViGoK3zozOQH+Ig=', 'Content-Length': '0'} 2023-09-22T21:36:48.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e7c8c7dabe68e59-00650e08f0-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.382 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:48.383 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:48.383 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:48.383 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:36:48.383 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:36:48.384 DEBUG:boto:Method: GET 2023-09-22T21:36:48.384 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:36:48.384 DEBUG:boto:Data: 2023-09-22T21:36:48.384 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.384 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.384 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.384 DEBUG:boto:Params: {} 2023-09-22T21:36:48.384 DEBUG:boto:Token: None 2023-09-22T21:36:48.384 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/ 2023-09-22T21:36:48.385 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:poyIKuFM7iAeViGoK3zozOQH+Ig= 2023-09-22T21:36:48.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:poyIKuFM7iAeViGoK3zozOQH+Ig=', 'Content-Length': '0'} 2023-09-22T21:36:48.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009515a9cf2e2aa8f2-00650e08f0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.391 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:48.391 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:36:48.391 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:36:48.391 DEBUG:boto:Method: HEAD 2023-09-22T21:36:48.391 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:36:48.391 DEBUG:boto:Data: 2023-09-22T21:36:48.391 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.392 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:48.392 DEBUG:boto:Port: 8001 2023-09-22T21:36:48.392 DEBUG:boto:Params: {} 2023-09-22T21:36:48.392 DEBUG:boto:Token: None 2023-09-22T21:36:48.392 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/obj55 2023-09-22T21:36:48.392 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YH9JBG01Hdz5cGms4tp55edM6gM= 2023-09-22T21:36:48.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YH9JBG01Hdz5cGms4tp55edM6gM=', 'Content-Length': '0'} 2023-09-22T21:36:48.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009501c0e4783913ef-00650e08f0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.395 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:36:48.395 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:36:48.395 DEBUG:boto:Method: HEAD 2023-09-22T21:36:48.395 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:36:48.395 DEBUG:boto:Data: 2023-09-22T21:36:48.396 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.396 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.396 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.396 DEBUG:boto:Params: {} 2023-09-22T21:36:48.396 DEBUG:boto:Token: None 2023-09-22T21:36:48.396 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/obj55 2023-09-22T21:36:48.396 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YH9JBG01Hdz5cGms4tp55edM6gM= 2023-09-22T21:36:48.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YH9JBG01Hdz5cGms4tp55edM6gM=', 'Content-Length': '0'} 2023-09-22T21:36:48.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000825de8883d252093-00650e08f0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:36:48.399 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:36:48.399 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:36:48.399 DEBUG:boto:Method: GET 2023-09-22T21:36:48.399 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:36:48.399 DEBUG:boto:Data: 2023-09-22T21:36:48.399 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.400 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:48.400 DEBUG:boto:Port: 8001 2023-09-22T21:36:48.400 DEBUG:boto:Params: {} 2023-09-22T21:36:48.400 DEBUG:boto:Token: None 2023-09-22T21:36:48.400 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/obj55 2023-09-22T21:36:48.400 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r7ieid/mtl58lfbGIc6Ng0ihcYM= 2023-09-22T21:36:48.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r7ieid/mtl58lfbGIc6Ng0ihcYM=', 'Content-Length': '0'} 2023-09-22T21:36:48.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079a5cb9d011ff767-00650e08f0-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.446 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:36:48.447 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:36:48.447 DEBUG:boto:Method: GET 2023-09-22T21:36:48.447 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:36:48.447 DEBUG:boto:Data: 2023-09-22T21:36:48.447 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.447 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.447 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.447 DEBUG:boto:Params: {} 2023-09-22T21:36:48.447 DEBUG:boto:Token: None 2023-09-22T21:36:48.448 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/obj55 2023-09-22T21:36:48.448 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r7ieid/mtl58lfbGIc6Ng0ihcYM= 2023-09-22T21:36:48.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r7ieid/mtl58lfbGIc6Ng0ihcYM=', 'Content-Length': '0'} 2023-09-22T21:36:48.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018850dccac863f0f-00650e08f0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.451 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:48.451 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:48.451 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:36:48.451 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:36:48.451 DEBUG:boto:Method: GET 2023-09-22T21:36:48.451 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:36:48.451 DEBUG:boto:Data: 2023-09-22T21:36:48.451 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.452 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:36:48.452 DEBUG:boto:Port: 8001 2023-09-22T21:36:48.452 DEBUG:boto:Params: {} 2023-09-22T21:36:48.452 DEBUG:boto:Token: None 2023-09-22T21:36:48.452 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/ 2023-09-22T21:36:48.452 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:poyIKuFM7iAeViGoK3zozOQH+Ig= 2023-09-22T21:36:48.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:poyIKuFM7iAeViGoK3zozOQH+Ig=', 'Content-Length': '0'} 2023-09-22T21:36:48.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cee070a4a69dedd-00650e08f0-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.498 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:48.499 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:48.499 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:48.499 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:36:48.499 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:36:48.499 DEBUG:boto:Method: GET 2023-09-22T21:36:48.499 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:36:48.500 DEBUG:boto:Data: 2023-09-22T21:36:48.500 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.500 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.500 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.500 DEBUG:boto:Params: {} 2023-09-22T21:36:48.500 DEBUG:boto:Token: None 2023-09-22T21:36:48.500 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/ 2023-09-22T21:36:48.500 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:poyIKuFM7iAeViGoK3zozOQH+Ig= 2023-09-22T21:36:48.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:poyIKuFM7iAeViGoK3zozOQH+Ig=', 'Content-Length': '0'} 2023-09-22T21:36:48.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e5f860d69377245-00650e08f0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.506 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:36:48.506 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-59 zones={a2, a1} 2023-09-22T21:36:48.506 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:36:48.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:36:48.507 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:36:48.554 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:36:48.554 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:36:48.613 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:36:48.613 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-59 2023-09-22T21:36:48.621 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 5 2023-09-22T21:36:48.669 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:48.669 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:48.669 DEBUG:boto:Method: HEAD 2023-09-22T21:36:48.669 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:36:48.669 DEBUG:boto:Data: 2023-09-22T21:36:48.669 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.669 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.670 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.670 DEBUG:boto:Params: {} 2023-09-22T21:36:48.670 DEBUG:boto:Token: None 2023-09-22T21:36:48.670 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/ 2023-09-22T21:36:48.670 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4/PXEAHl5hC8gJky4ecBArgX1IM= 2023-09-22T21:36:48.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4/PXEAHl5hC8gJky4ecBArgX1IM=', 'Content-Length': '0'} 2023-09-22T21:36:48.673 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008526eee969849e78-00650e08f0-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.674 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:36:48.674 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:36:48.674 DEBUG:boto:Method: PUT 2023-09-22T21:36:48.674 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:36:48.674 DEBUG:boto:Data: 2023-09-22T21:36:48.674 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:36:48.674 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.674 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.674 DEBUG:boto:Params: {} 2023-09-22T21:36:48.675 DEBUG:boto:Token: None 2023-09-22T21:36:48.675 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/obj17 2023-09-22T21:36:48.675 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7c9lGKJfkdHYU87k8A6iBMgPSmo= 2023-09-22T21:36:48.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7c9lGKJfkdHYU87k8A6iBMgPSmo='} 2023-09-22T21:36:48.681 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d57a358d3ec2279-00650e08f0-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.681 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:48.681 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:48.681 DEBUG:boto:Method: HEAD 2023-09-22T21:36:48.681 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:36:48.681 DEBUG:boto:Data: 2023-09-22T21:36:48.681 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.681 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.682 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.682 DEBUG:boto:Params: {} 2023-09-22T21:36:48.682 DEBUG:boto:Token: None 2023-09-22T21:36:48.682 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/ 2023-09-22T21:36:48.682 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4/PXEAHl5hC8gJky4ecBArgX1IM= 2023-09-22T21:36:48.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4/PXEAHl5hC8gJky4ecBArgX1IM=', 'Content-Length': '0'} 2023-09-22T21:36:48.685 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9147d9ddc37eae3-00650e08f0-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.685 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:36:48.685 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:36:48.685 DEBUG:boto:Method: PUT 2023-09-22T21:36:48.685 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:36:48.685 DEBUG:boto:Data: 2023-09-22T21:36:48.685 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:36:48.685 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.686 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.686 DEBUG:boto:Params: {} 2023-09-22T21:36:48.686 DEBUG:boto:Token: None 2023-09-22T21:36:48.686 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/obj34 2023-09-22T21:36:48.686 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Liua6tYKa5vFRAa6kVCpG+q/acw= 2023-09-22T21:36:48.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Liua6tYKa5vFRAa6kVCpG+q/acw='} 2023-09-22T21:36:48.734 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c51ba9d821ebfb1b-00650e08f0-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.735 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:48.735 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:48.735 DEBUG:boto:Method: HEAD 2023-09-22T21:36:48.735 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:36:48.735 DEBUG:boto:Data: 2023-09-22T21:36:48.735 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.735 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.735 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.735 DEBUG:boto:Params: {} 2023-09-22T21:36:48.735 DEBUG:boto:Token: None 2023-09-22T21:36:48.736 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/ 2023-09-22T21:36:48.736 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4/PXEAHl5hC8gJky4ecBArgX1IM= 2023-09-22T21:36:48.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4/PXEAHl5hC8gJky4ecBArgX1IM=', 'Content-Length': '0'} 2023-09-22T21:36:48.739 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9022091b14b3d3a-00650e08f0-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.739 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:36:48.739 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:36:48.740 DEBUG:boto:Method: PUT 2023-09-22T21:36:48.740 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:36:48.740 DEBUG:boto:Data: 2023-09-22T21:36:48.740 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:36:48.740 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.740 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.740 DEBUG:boto:Params: {} 2023-09-22T21:36:48.740 DEBUG:boto:Token: None 2023-09-22T21:36:48.740 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/obj51 2023-09-22T21:36:48.741 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0jzlzZ1CJhv4vFNzfJkEYydjLVg= 2023-09-22T21:36:48.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0jzlzZ1CJhv4vFNzfJkEYydjLVg='} 2023-09-22T21:36:48.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c9eae0238ea55b3-00650e08f0-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.751 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:48.751 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:48.751 DEBUG:boto:Method: HEAD 2023-09-22T21:36:48.751 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:36:48.751 DEBUG:boto:Data: 2023-09-22T21:36:48.751 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.751 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.751 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.751 DEBUG:boto:Params: {} 2023-09-22T21:36:48.751 DEBUG:boto:Token: None 2023-09-22T21:36:48.752 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/ 2023-09-22T21:36:48.752 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4/PXEAHl5hC8gJky4ecBArgX1IM= 2023-09-22T21:36:48.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4/PXEAHl5hC8gJky4ecBArgX1IM=', 'Content-Length': '0'} 2023-09-22T21:36:48.756 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095634544fb26b9b8-00650e08f0-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.756 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:36:48.756 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:36:48.756 DEBUG:boto:Method: PUT 2023-09-22T21:36:48.756 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:36:48.756 DEBUG:boto:Data: 2023-09-22T21:36:48.757 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:36:48.757 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.757 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.757 DEBUG:boto:Params: {} 2023-09-22T21:36:48.757 DEBUG:boto:Token: None 2023-09-22T21:36:48.757 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/obj68 2023-09-22T21:36:48.757 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tPHVj02wvaK+n119R+xEcASCISA= 2023-09-22T21:36:48.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tPHVj02wvaK+n119R+xEcASCISA='} 2023-09-22T21:36:48.802 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d75c0c665af77685-00650e08f0-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.802 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:36:48.803 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:36:48.803 DEBUG:boto:Method: HEAD 2023-09-22T21:36:48.803 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:36:48.803 DEBUG:boto:Data: 2023-09-22T21:36:48.803 DEBUG:boto:Headers: {} 2023-09-22T21:36:48.803 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.803 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.803 DEBUG:boto:Params: {} 2023-09-22T21:36:48.803 DEBUG:boto:Token: None 2023-09-22T21:36:48.804 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/ 2023-09-22T21:36:48.804 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4/PXEAHl5hC8gJky4ecBArgX1IM= 2023-09-22T21:36:48.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4/PXEAHl5hC8gJky4ecBArgX1IM=', 'Content-Length': '0'} 2023-09-22T21:36:48.809 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077378e7de3fb6cf3-00650e08f0-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.809 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:36:48.809 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:36:48.809 DEBUG:boto:Method: PUT 2023-09-22T21:36:48.809 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:36:48.809 DEBUG:boto:Data: 2023-09-22T21:36:48.809 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:36:48.810 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:36:48.810 DEBUG:boto:Port: 8000 2023-09-22T21:36:48.810 DEBUG:boto:Params: {} 2023-09-22T21:36:48.810 DEBUG:boto:Token: None 2023-09-22T21:36:48.810 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:36:48 GMT /ffzdqu-59/obj85 2023-09-22T21:36:48.810 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HMZZrVBU7lH2qRfNjdTSWDHPXQQ= 2023-09-22T21:36:48.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:36:48 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HMZZrVBU7lH2qRfNjdTSWDHPXQQ='} 2023-09-22T21:36:48.823 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0187215e97604c5-00650e08f0-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:36:48.823 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-59 2023-09-22T21:36:48.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:36:48.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:36:48.823 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:36:48.873 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:36:48.874 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:36:48.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:36:48.914+0000 7fbbeada1a40 1 waiting to reach latest gen 1, on 0.. 2023-09-22T21:36:53.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:36:53.914+0000 7fbbeada1a40 1 waiting to reach latest gen 1, on 0.. 2023-09-22T21:36:58.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:36:58.918+0000 7fbbeada1a40 1 waiting to reach latest gen 1, on 0.. 2023-09-22T21:37:03.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:37:03.918+0000 7fbbeada1a40 1 waiting to reach latest gen 1, on 0.. 2023-09-22T21:37:06.594 INFO:tasks.rgw.c1.client.0.smithi160.stdout:2023-09-22T21:37:06.590+0000 7fa9c1675640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:37:08.926 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:37:08.922+0000 7fbbeada1a40 1 waiting for incremental sync to catch up: 2023-09-22T21:37:09.230 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.67.6, , , , , , 00000000001.35.6, , , , , , 00000000001.40.6, , , , ] 2023-09-22T21:37:09.230 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.67.6, , , , , , 00000000001.35.6, , 00000000001.39.6, , , , 00000000001.40.6, 00000000001.149.6, , , ] 2023-09-22T21:37:13.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:37:13.926+0000 7fbbeada1a40 1 waiting for incremental sync to catch up: 2023-09-22T21:37:13.929 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.67.6, , , , , , 00000000001.35.6, , , , , , 00000000001.40.6, , , , ] 2023-09-22T21:37:13.929 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.67.6, , , , , , 00000000001.35.6, , 00000000001.39.6, , , , 00000000001.40.6, 00000000001.149.6, , , ] 2023-09-22T21:37:18.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:37:18.926+0000 7fbbeada1a40 1 waiting for incremental sync to catch up: 2023-09-22T21:37:18.932 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.67.6, , , , , , 00000000001.35.6, , , , , , 00000000001.40.6, , , , ] 2023-09-22T21:37:18.932 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.67.6, , , , , , 00000000001.35.6, , 00000000001.39.6, , , , 00000000001.40.6, 00000000001.149.6, , , ] 2023-09-22T21:37:23.935 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:37:23.930+0000 7fbbeada1a40 1 waiting for incremental sync to catch up: 2023-09-22T21:37:23.935 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.67.6, , , , , , 00000000001.35.6, , , , , , 00000000001.40.6, , , , ] 2023-09-22T21:37:23.935 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.67.6, , , , , , 00000000001.35.6, , 00000000001.39.6, , , , 00000000001.40.6, 00000000001.149.6, , , ] 2023-09-22T21:37:28.938 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:37:28.934+0000 7fbbeada1a40 1 waiting for incremental sync to catch up: 2023-09-22T21:37:28.938 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.67.6, , , , , , 00000000001.35.6, , , , , , 00000000001.40.6, , , , ] 2023-09-22T21:37:28.938 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.67.6, , , , , , 00000000001.35.6, , 00000000001.39.6, , , , 00000000001.40.6, 00000000001.149.6, , , ] 2023-09-22T21:37:33.940 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:37:33.938+0000 7fbbeada1a40 1 waiting for incremental sync to catch up: 2023-09-22T21:37:33.941 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.67.6, , , , , , 00000000001.35.6, , , , , , 00000000001.40.6, , , , ] 2023-09-22T21:37:33.941 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.67.6, , , , , , 00000000001.35.6, , 00000000001.39.6, , , , 00000000001.40.6, 00000000001.149.6, , , ] 2023-09-22T21:37:38.943 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:37:38.938+0000 7fbbeada1a40 1 waiting for incremental sync to catch up: 2023-09-22T21:37:38.947 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.67.6, , , , , , 00000000001.35.6, , , , , , 00000000001.40.6, , , , ] 2023-09-22T21:37:38.947 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.67.6, , , , , , 00000000001.35.6, , 00000000001.39.6, , , , 00000000001.40.6, 00000000001.149.6, , , ] 2023-09-22T21:37:43.946 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:37:43.942+0000 7fbbeada1a40 1 waiting for incremental sync to catch up: 2023-09-22T21:37:43.946 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.67.6, , , , , , 00000000001.35.6, , , , , , 00000000001.40.6, , , , ] 2023-09-22T21:37:43.946 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.67.6, , , , , , 00000000001.35.6, , 00000000001.39.6, , , , 00000000001.40.6, 00000000001.149.6, , , ] 2023-09-22T21:37:48.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:37:48.941+0000 7fbbeada1a40 1 waiting for incremental sync to catch up: 2023-09-22T21:37:48.949 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.67.6, , , , , , 00000000001.35.6, , , , , , 00000000001.40.6, , , , ] 2023-09-22T21:37:48.949 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.67.6, , , , , , 00000000001.35.6, , 00000000001.39.6, , , , 00000000001.40.6, 00000000001.149.6, , , ] 2023-09-22T21:37:53.951 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:37:53.945+0000 7fbbeada1a40 1 waiting for incremental sync to catch up: 2023-09-22T21:37:53.951 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.67.6, , , , , , 00000000001.35.6, , , , , , 00000000001.40.6, , , , ] 2023-09-22T21:37:53.951 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.67.6, , , , , , 00000000001.35.6, , 00000000001.39.6, , , , 00000000001.40.6, 00000000001.149.6, , , ] 2023-09-22T21:37:58.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:37:58.949+0000 7fbbeada1a40 1 waiting for incremental sync to catch up: 2023-09-22T21:37:58.954 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.67.6, , , , , , 00000000001.35.6, , , , , , 00000000001.40.6, , , , ] 2023-09-22T21:37:58.954 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.67.6, , , , , , 00000000001.35.6, , 00000000001.39.6, , , , 00000000001.40.6, 00000000001.149.6, , , ] 2023-09-22T21:38:03.956 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:38:03.957+0000 7fbbeada1a40 1 waiting for incremental sync to catch up: 2023-09-22T21:38:03.957 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.67.6, , , , , , 00000000001.35.6, , , , , , 00000000001.40.6, , , , ] 2023-09-22T21:38:03.957 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.67.6, , , , , , 00000000001.35.6, , 00000000001.39.6, , , , 00000000001.40.6, 00000000001.149.6, , , ] 2023-09-22T21:38:08.959 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:38:08.957+0000 7fbbeada1a40 1 bucket sync caught up with source: 2023-09-22T21:38:08.959 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.67.6, , , , , , 00000000001.35.6, , 00000000001.39.6, , , , 00000000001.40.6, 00000000001.149.6, , , ] 2023-09-22T21:38:08.959 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.67.6, , , , , , 00000000001.35.6, , 00000000001.39.6, , , , 00000000001.40.6, 00000000001.149.6, , , ] 2023-09-22T21:38:08.959 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:38:08.957+0000 7fbbeada1a40 0 bucket checkpoint complete 2023-09-22T21:38:08.997 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-59 2023-09-22T21:38:08.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:38:08.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:38:08.998 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:38:09.046 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:38:09.046 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:38:09.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:38:09.089+0000 7f72a9469a40 1 waiting for incremental sync to catch up: 2023-09-22T21:38:09.091 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.297.6, , , , , , , 00000000001.129.6, 00000000003.154.6, 00000000003.304.6] 2023-09-22T21:38:09.091 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.297.6, 00000000001.259.6, , , 00000000001.385.6, , , 00000000001.129.6, 00000000003.154.6, 00000000003.304.6] 2023-09-22T21:38:14.093 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:38:14.093+0000 7f72a9469a40 1 waiting for incremental sync to catch up: 2023-09-22T21:38:14.093 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.297.6, , , , , , , 00000000001.129.6, 00000000003.154.6, 00000000003.304.6] 2023-09-22T21:38:14.093 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.297.6, 00000000001.259.6, , , 00000000001.385.6, , , 00000000001.129.6, 00000000003.154.6, 00000000003.304.6] 2023-09-22T21:38:19.095 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:38:19.093+0000 7f72a9469a40 1 waiting for incremental sync to catch up: 2023-09-22T21:38:19.095 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.297.6, 00000000001.259.6, , , , , , 00000000001.129.6, 00000000003.154.6, 00000000003.304.6] 2023-09-22T21:38:19.095 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.297.6, 00000000001.259.6, , , 00000000001.385.6, , , 00000000001.129.6, 00000000003.154.6, 00000000003.304.6] 2023-09-22T21:38:24.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:38:24.097+0000 7f72a9469a40 1 waiting for incremental sync to catch up: 2023-09-22T21:38:24.098 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.297.6, 00000000001.259.6, , , , , , 00000000001.129.6, 00000000003.154.6, 00000000003.304.6] 2023-09-22T21:38:24.098 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.297.6, 00000000001.259.6, , , 00000000001.385.6, , , 00000000001.129.6, 00000000003.154.6, 00000000003.304.6] 2023-09-22T21:38:29.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:38:29.097+0000 7f72a9469a40 1 bucket sync caught up with source: 2023-09-22T21:38:29.100 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.297.6, 00000000001.259.6, , , 00000000001.385.6, , , 00000000001.129.6, 00000000003.154.6, 00000000003.304.6] 2023-09-22T21:38:29.100 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.297.6, 00000000001.259.6, , , 00000000001.385.6, , , 00000000001.129.6, 00000000003.154.6, 00000000003.304.6] 2023-09-22T21:38:29.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:38:29.097+0000 7f72a9469a40 0 bucket checkpoint complete 2023-09-22T21:38:29.133 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-59 zones={a2, a1} 2023-09-22T21:38:29.133 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:29.133 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:29.133 DEBUG:boto:Method: HEAD 2023-09-22T21:38:29.133 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:38:29.134 DEBUG:boto:Data: 2023-09-22T21:38:29.134 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.134 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.134 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.134 DEBUG:boto:Params: {} 2023-09-22T21:38:29.134 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:38:29.135 DEBUG:boto:Token: None 2023-09-22T21:38:29.135 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/ 2023-09-22T21:38:29.135 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fuwmL/jIB+0L7OKz3N80//mpf3A= 2023-09-22T21:38:29.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fuwmL/jIB+0L7OKz3N80//mpf3A=', 'Content-Length': '0'} 2023-09-22T21:38:29.140 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f13611ec64d0fd6b-00650e0955-1b9c-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.141 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:29.141 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:29.141 DEBUG:boto:Method: HEAD 2023-09-22T21:38:29.141 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:38:29.141 DEBUG:boto:Data: 2023-09-22T21:38:29.141 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.141 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.141 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.142 DEBUG:boto:Params: {} 2023-09-22T21:38:29.142 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:38:29.142 DEBUG:boto:Token: None 2023-09-22T21:38:29.142 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/ 2023-09-22T21:38:29.142 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fuwmL/jIB+0L7OKz3N80//mpf3A= 2023-09-22T21:38:29.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fuwmL/jIB+0L7OKz3N80//mpf3A=', 'Content-Length': '0'} 2023-09-22T21:38:29.147 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000033dd5103be6f2fa7-00650e0955-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.148 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:38:29.148 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:29.148 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:29.148 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:38:29.148 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:38:29.148 DEBUG:boto:Method: GET 2023-09-22T21:38:29.148 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:38:29.148 DEBUG:boto:Data: 2023-09-22T21:38:29.149 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.149 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.149 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.149 DEBUG:boto:Params: {} 2023-09-22T21:38:29.149 DEBUG:boto:Token: None 2023-09-22T21:38:29.149 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/?versions 2023-09-22T21:38:29.149 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pPHJF295ngAJ86dfSYHuvVrNX9A= 2023-09-22T21:38:29.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pPHJF295ngAJ86dfSYHuvVrNX9A=', 'Content-Length': '0'} 2023-09-22T21:38:29.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000229e3039f46803ec-00650e0955-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.153 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:29.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T21:38:29.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-22T21:38:29.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T21:38:29.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T21:38:29.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-22T21:38:29.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T21:38:29.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-22T21:38:29.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T21:38:29.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-22T21:38:29.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-22T21:38:29.155 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:38:29.156 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:29.156 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:29.156 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:38:29.156 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:38:29.156 DEBUG:boto:Method: GET 2023-09-22T21:38:29.156 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:38:29.156 DEBUG:boto:Data: 2023-09-22T21:38:29.156 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.156 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.156 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.157 DEBUG:boto:Params: {} 2023-09-22T21:38:29.157 DEBUG:boto:Token: None 2023-09-22T21:38:29.157 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/?versions 2023-09-22T21:38:29.157 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pPHJF295ngAJ86dfSYHuvVrNX9A= 2023-09-22T21:38:29.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pPHJF295ngAJ86dfSYHuvVrNX9A=', 'Content-Length': '0'} 2023-09-22T21:38:29.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092f1474b713ca097-00650e0955-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.161 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:29.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T21:38:29.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-22T21:38:29.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T21:38:29.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T21:38:29.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-22T21:38:29.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T21:38:29.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-22T21:38:29.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T21:38:29.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-22T21:38:29.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-22T21:38:29.163 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:29.163 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:29.163 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:38:29.164 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:38:29.164 DEBUG:boto:Method: GET 2023-09-22T21:38:29.164 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:38:29.164 DEBUG:boto:Data: 2023-09-22T21:38:29.164 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.164 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.164 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.164 DEBUG:boto:Params: {} 2023-09-22T21:38:29.164 DEBUG:boto:Token: None 2023-09-22T21:38:29.164 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/?versions 2023-09-22T21:38:29.165 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pPHJF295ngAJ86dfSYHuvVrNX9A= 2023-09-22T21:38:29.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pPHJF295ngAJ86dfSYHuvVrNX9A=', 'Content-Length': '0'} 2023-09-22T21:38:29.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000194f61e2f00c1f1-00650e0955-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.169 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:29.169 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:29.170 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:29.170 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:38:29.170 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:38:29.170 DEBUG:boto:Method: GET 2023-09-22T21:38:29.170 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:38:29.170 DEBUG:boto:Data: 2023-09-22T21:38:29.170 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.170 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.170 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.171 DEBUG:boto:Params: {} 2023-09-22T21:38:29.171 DEBUG:boto:Token: None 2023-09-22T21:38:29.171 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/?versions 2023-09-22T21:38:29.171 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pPHJF295ngAJ86dfSYHuvVrNX9A= 2023-09-22T21:38:29.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pPHJF295ngAJ86dfSYHuvVrNX9A=', 'Content-Length': '0'} 2023-09-22T21:38:29.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d7fbf0c05b21607-00650e0955-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.178 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:29.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:38:29.178 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:38:29.179 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:38:29.179 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.179 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.179 DEBUG:boto:Method: GET 2023-09-22T21:38:29.179 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.179 DEBUG:boto:Data: 2023-09-22T21:38:29.179 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.179 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.179 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.180 DEBUG:boto:Params: {} 2023-09-22T21:38:29.180 DEBUG:boto:Token: None 2023-09-22T21:38:29.180 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.180 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qg+4iFqeHjfN7ku/RwsOoMvnHyI= 2023-09-22T21:38:29.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qg+4iFqeHjfN7ku/RwsOoMvnHyI=', 'Content-Length': '0'} 2023-09-22T21:38:29.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b6fc707012a7797-00650e0955-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.226 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:38:29.226 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:38:29.227 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.227 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.227 DEBUG:boto:Method: GET 2023-09-22T21:38:29.227 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.227 DEBUG:boto:Data: 2023-09-22T21:38:29.227 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.227 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.227 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.227 DEBUG:boto:Params: {} 2023-09-22T21:38:29.228 DEBUG:boto:Token: None 2023-09-22T21:38:29.228 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.228 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qg+4iFqeHjfN7ku/RwsOoMvnHyI= 2023-09-22T21:38:29.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qg+4iFqeHjfN7ku/RwsOoMvnHyI=', 'Content-Length': '0'} 2023-09-22T21:38:29.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b598e2653a0e294d-00650e0955-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.231 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:38:29.231 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:38:29.232 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.232 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.232 DEBUG:boto:Method: HEAD 2023-09-22T21:38:29.232 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.232 DEBUG:boto:Data: 2023-09-22T21:38:29.232 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.232 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.232 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.232 DEBUG:boto:Params: {} 2023-09-22T21:38:29.232 DEBUG:boto:Token: None 2023-09-22T21:38:29.233 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.233 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8MNsLr44HcgMxQe6iGMFXDFO67Y= 2023-09-22T21:38:29.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8MNsLr44HcgMxQe6iGMFXDFO67Y=', 'Content-Length': '0'} 2023-09-22T21:38:29.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbc7101224c7f13e-00650e0955-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.236 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:38:29.236 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:38:29.236 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.236 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.236 DEBUG:boto:Method: HEAD 2023-09-22T21:38:29.236 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.236 DEBUG:boto:Data: 2023-09-22T21:38:29.237 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.237 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.237 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.237 DEBUG:boto:Params: {} 2023-09-22T21:38:29.237 DEBUG:boto:Token: None 2023-09-22T21:38:29.237 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.237 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8MNsLr44HcgMxQe6iGMFXDFO67Y= 2023-09-22T21:38:29.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8MNsLr44HcgMxQe6iGMFXDFO67Y=', 'Content-Length': '0'} 2023-09-22T21:38:29.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5a30edcf1c130fa-00650e0955-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:38:29.241 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:38:29.241 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:38:29.241 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.241 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.241 DEBUG:boto:Method: GET 2023-09-22T21:38:29.241 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.241 DEBUG:boto:Data: 2023-09-22T21:38:29.241 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.242 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.242 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.242 DEBUG:boto:Params: {} 2023-09-22T21:38:29.242 DEBUG:boto:Token: None 2023-09-22T21:38:29.242 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.242 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qg+4iFqeHjfN7ku/RwsOoMvnHyI= 2023-09-22T21:38:29.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qg+4iFqeHjfN7ku/RwsOoMvnHyI=', 'Content-Length': '0'} 2023-09-22T21:38:29.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032a456723bbe4bd7-00650e0955-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.286 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:38:29.286 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:38:29.287 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.287 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.287 DEBUG:boto:Method: GET 2023-09-22T21:38:29.287 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.287 DEBUG:boto:Data: 2023-09-22T21:38:29.287 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.287 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.287 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.287 DEBUG:boto:Params: {} 2023-09-22T21:38:29.288 DEBUG:boto:Token: None 2023-09-22T21:38:29.288 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:38:29.288 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qg+4iFqeHjfN7ku/RwsOoMvnHyI= 2023-09-22T21:38:29.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qg+4iFqeHjfN7ku/RwsOoMvnHyI=', 'Content-Length': '0'} 2023-09-22T21:38:29.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf3d24e9b1938d0e-00650e0955-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.291 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:29.291 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:29.292 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:38:29.292 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:38:29.292 DEBUG:boto:Method: GET 2023-09-22T21:38:29.292 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:38:29.292 DEBUG:boto:Data: 2023-09-22T21:38:29.292 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.292 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.292 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.292 DEBUG:boto:Params: {} 2023-09-22T21:38:29.293 DEBUG:boto:Token: None 2023-09-22T21:38:29.293 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/ 2023-09-22T21:38:29.293 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo= 2023-09-22T21:38:29.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo=', 'Content-Length': '0'} 2023-09-22T21:38:29.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a38c34eb059e41e0-00650e0955-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.338 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:29.339 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:29.339 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:29.339 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:38:29.339 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:38:29.339 DEBUG:boto:Method: GET 2023-09-22T21:38:29.339 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:38:29.339 DEBUG:boto:Data: 2023-09-22T21:38:29.339 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.340 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.340 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.340 DEBUG:boto:Params: {} 2023-09-22T21:38:29.340 DEBUG:boto:Token: None 2023-09-22T21:38:29.340 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/ 2023-09-22T21:38:29.340 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo= 2023-09-22T21:38:29.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo=', 'Content-Length': '0'} 2023-09-22T21:38:29.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e303d55295489c57-00650e0955-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.347 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:29.347 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:38:29.347 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:38:29.347 DEBUG:boto:Method: HEAD 2023-09-22T21:38:29.348 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:38:29.348 DEBUG:boto:Data: 2023-09-22T21:38:29.348 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.348 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.348 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.348 DEBUG:boto:Params: {} 2023-09-22T21:38:29.348 DEBUG:boto:Token: None 2023-09-22T21:38:29.348 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj11 2023-09-22T21:38:29.349 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TGR6B+kPFUwO22OQVJQj7AKGaEc= 2023-09-22T21:38:29.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TGR6B+kPFUwO22OQVJQj7AKGaEc=', 'Content-Length': '0'} 2023-09-22T21:38:29.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8e0c6f1f6ca22c7-00650e0955-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.352 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:38:29.352 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:38:29.352 DEBUG:boto:Method: HEAD 2023-09-22T21:38:29.352 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:38:29.352 DEBUG:boto:Data: 2023-09-22T21:38:29.352 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.352 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.352 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.353 DEBUG:boto:Params: {} 2023-09-22T21:38:29.353 DEBUG:boto:Token: None 2023-09-22T21:38:29.353 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj11 2023-09-22T21:38:29.353 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TGR6B+kPFUwO22OQVJQj7AKGaEc= 2023-09-22T21:38:29.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TGR6B+kPFUwO22OQVJQj7AKGaEc=', 'Content-Length': '0'} 2023-09-22T21:38:29.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092c5a0ed0e7c2d99-00650e0955-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:38:29.356 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:38:29.356 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:38:29.356 DEBUG:boto:Method: GET 2023-09-22T21:38:29.356 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:38:29.357 DEBUG:boto:Data: 2023-09-22T21:38:29.357 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.357 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.357 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.357 DEBUG:boto:Params: {} 2023-09-22T21:38:29.357 DEBUG:boto:Token: None 2023-09-22T21:38:29.357 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj11 2023-09-22T21:38:29.357 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lvH7+B0TM2zZOjqOgsY4wM40MWU= 2023-09-22T21:38:29.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lvH7+B0TM2zZOjqOgsY4wM40MWU=', 'Content-Length': '0'} 2023-09-22T21:38:29.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000248632d8bad024d7-00650e0955-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.406 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:38:29.406 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:38:29.407 DEBUG:boto:Method: GET 2023-09-22T21:38:29.407 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:38:29.407 DEBUG:boto:Data: 2023-09-22T21:38:29.407 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.407 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.407 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.407 DEBUG:boto:Params: {} 2023-09-22T21:38:29.407 DEBUG:boto:Token: None 2023-09-22T21:38:29.407 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj11 2023-09-22T21:38:29.408 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lvH7+B0TM2zZOjqOgsY4wM40MWU= 2023-09-22T21:38:29.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lvH7+B0TM2zZOjqOgsY4wM40MWU=', 'Content-Length': '0'} 2023-09-22T21:38:29.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b250ddd7ece86c88-00650e0955-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.411 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:29.411 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:29.411 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:38:29.411 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:38:29.411 DEBUG:boto:Method: GET 2023-09-22T21:38:29.412 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:38:29.412 DEBUG:boto:Data: 2023-09-22T21:38:29.412 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.412 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.412 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.412 DEBUG:boto:Params: {} 2023-09-22T21:38:29.412 DEBUG:boto:Token: None 2023-09-22T21:38:29.412 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/ 2023-09-22T21:38:29.412 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo= 2023-09-22T21:38:29.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo=', 'Content-Length': '0'} 2023-09-22T21:38:29.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd121f740d58d65e-00650e0955-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.462 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:29.463 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:29.463 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:29.463 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:38:29.463 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:38:29.463 DEBUG:boto:Method: GET 2023-09-22T21:38:29.463 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:38:29.463 DEBUG:boto:Data: 2023-09-22T21:38:29.463 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.464 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.464 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.464 DEBUG:boto:Params: {} 2023-09-22T21:38:29.464 DEBUG:boto:Token: None 2023-09-22T21:38:29.464 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/ 2023-09-22T21:38:29.464 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo= 2023-09-22T21:38:29.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo=', 'Content-Length': '0'} 2023-09-22T21:38:29.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1adf88f627f1204-00650e0955-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.471 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:29.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:38:29.471 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:38:29.472 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:38:29.472 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.472 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.472 DEBUG:boto:Method: GET 2023-09-22T21:38:29.472 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.472 DEBUG:boto:Data: 2023-09-22T21:38:29.472 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.472 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.472 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.472 DEBUG:boto:Params: {} 2023-09-22T21:38:29.473 DEBUG:boto:Token: None 2023-09-22T21:38:29.473 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.473 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:a4Ut4Njp7R4l9b2/P8wbmnrt6Mc= 2023-09-22T21:38:29.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:a4Ut4Njp7R4l9b2/P8wbmnrt6Mc=', 'Content-Length': '0'} 2023-09-22T21:38:29.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae4db7eaa796e943-00650e0955-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.476 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:38:29.476 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:38:29.477 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.477 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.477 DEBUG:boto:Method: GET 2023-09-22T21:38:29.477 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.477 DEBUG:boto:Data: 2023-09-22T21:38:29.477 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.477 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.477 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.477 DEBUG:boto:Params: {} 2023-09-22T21:38:29.478 DEBUG:boto:Token: None 2023-09-22T21:38:29.478 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.478 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:a4Ut4Njp7R4l9b2/P8wbmnrt6Mc= 2023-09-22T21:38:29.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:a4Ut4Njp7R4l9b2/P8wbmnrt6Mc=', 'Content-Length': '0'} 2023-09-22T21:38:29.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0ea589def18e845-00650e0955-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.481 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:38:29.481 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:38:29.481 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.481 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.481 DEBUG:boto:Method: HEAD 2023-09-22T21:38:29.482 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.482 DEBUG:boto:Data: 2023-09-22T21:38:29.482 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.482 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.482 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.482 DEBUG:boto:Params: {} 2023-09-22T21:38:29.482 DEBUG:boto:Token: None 2023-09-22T21:38:29.482 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.482 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n7GI6VzLMvmLAifuhI3AskwGszk= 2023-09-22T21:38:29.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n7GI6VzLMvmLAifuhI3AskwGszk=', 'Content-Length': '0'} 2023-09-22T21:38:29.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6322364529d70a0-00650e0955-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.485 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:38:29.485 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:38:29.486 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.486 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.486 DEBUG:boto:Method: HEAD 2023-09-22T21:38:29.486 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.486 DEBUG:boto:Data: 2023-09-22T21:38:29.486 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.486 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.486 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.486 DEBUG:boto:Params: {} 2023-09-22T21:38:29.487 DEBUG:boto:Token: None 2023-09-22T21:38:29.487 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.487 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n7GI6VzLMvmLAifuhI3AskwGszk= 2023-09-22T21:38:29.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n7GI6VzLMvmLAifuhI3AskwGszk=', 'Content-Length': '0'} 2023-09-22T21:38:29.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072dc68b77d4de99a-00650e0955-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:38:29.490 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:38:29.490 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:38:29.490 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.490 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.490 DEBUG:boto:Method: GET 2023-09-22T21:38:29.491 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.491 DEBUG:boto:Data: 2023-09-22T21:38:29.491 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.491 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.491 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.491 DEBUG:boto:Params: {} 2023-09-22T21:38:29.491 DEBUG:boto:Token: None 2023-09-22T21:38:29.491 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.491 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:a4Ut4Njp7R4l9b2/P8wbmnrt6Mc= 2023-09-22T21:38:29.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:a4Ut4Njp7R4l9b2/P8wbmnrt6Mc=', 'Content-Length': '0'} 2023-09-22T21:38:29.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a22a7154984a92c-00650e0955-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.534 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:38:29.535 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:38:29.535 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.535 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.535 DEBUG:boto:Method: GET 2023-09-22T21:38:29.535 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.535 DEBUG:boto:Data: 2023-09-22T21:38:29.535 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.535 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.535 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.536 DEBUG:boto:Params: {} 2023-09-22T21:38:29.536 DEBUG:boto:Token: None 2023-09-22T21:38:29.536 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:38:29.536 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:a4Ut4Njp7R4l9b2/P8wbmnrt6Mc= 2023-09-22T21:38:29.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:a4Ut4Njp7R4l9b2/P8wbmnrt6Mc=', 'Content-Length': '0'} 2023-09-22T21:38:29.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fc0c02e901165df-00650e0955-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.539 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:29.539 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:29.540 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:38:29.540 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:38:29.540 DEBUG:boto:Method: GET 2023-09-22T21:38:29.540 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:38:29.540 DEBUG:boto:Data: 2023-09-22T21:38:29.540 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.540 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.540 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.540 DEBUG:boto:Params: {} 2023-09-22T21:38:29.540 DEBUG:boto:Token: None 2023-09-22T21:38:29.541 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/ 2023-09-22T21:38:29.541 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo= 2023-09-22T21:38:29.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo=', 'Content-Length': '0'} 2023-09-22T21:38:29.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e65b5191513292c-00650e0955-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.586 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:29.587 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:29.587 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:29.587 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:38:29.587 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:38:29.587 DEBUG:boto:Method: GET 2023-09-22T21:38:29.587 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:38:29.588 DEBUG:boto:Data: 2023-09-22T21:38:29.588 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.588 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.588 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.588 DEBUG:boto:Params: {} 2023-09-22T21:38:29.588 DEBUG:boto:Token: None 2023-09-22T21:38:29.588 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/ 2023-09-22T21:38:29.588 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo= 2023-09-22T21:38:29.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo=', 'Content-Length': '0'} 2023-09-22T21:38:29.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1ef65485009d21a-00650e0955-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.595 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:29.596 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:38:29.596 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:38:29.596 DEBUG:boto:Method: HEAD 2023-09-22T21:38:29.596 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:38:29.596 DEBUG:boto:Data: 2023-09-22T21:38:29.596 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.596 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.597 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.597 DEBUG:boto:Params: {} 2023-09-22T21:38:29.597 DEBUG:boto:Token: None 2023-09-22T21:38:29.597 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj17 2023-09-22T21:38:29.597 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M09NP8PemHwIQkc7GCIbO00Pfho= 2023-09-22T21:38:29.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M09NP8PemHwIQkc7GCIbO00Pfho=', 'Content-Length': '0'} 2023-09-22T21:38:29.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d92276b4e6b85e88-00650e0955-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.600 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:38:29.600 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:38:29.600 DEBUG:boto:Method: HEAD 2023-09-22T21:38:29.600 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:38:29.600 DEBUG:boto:Data: 2023-09-22T21:38:29.601 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.601 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.601 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.601 DEBUG:boto:Params: {} 2023-09-22T21:38:29.601 DEBUG:boto:Token: None 2023-09-22T21:38:29.601 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj17 2023-09-22T21:38:29.601 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M09NP8PemHwIQkc7GCIbO00Pfho= 2023-09-22T21:38:29.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M09NP8PemHwIQkc7GCIbO00Pfho=', 'Content-Length': '0'} 2023-09-22T21:38:29.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076f17a9bf38d0757-00650e0955-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:38:29.604 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:38:29.604 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:38:29.604 DEBUG:boto:Method: GET 2023-09-22T21:38:29.605 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:38:29.605 DEBUG:boto:Data: 2023-09-22T21:38:29.605 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.605 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.605 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.605 DEBUG:boto:Params: {} 2023-09-22T21:38:29.605 DEBUG:boto:Token: None 2023-09-22T21:38:29.605 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj17 2023-09-22T21:38:29.605 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TgFj+llxQ2EcYcei0g6IhpcENXo= 2023-09-22T21:38:29.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TgFj+llxQ2EcYcei0g6IhpcENXo=', 'Content-Length': '0'} 2023-09-22T21:38:29.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc802681d0c31a3e-00650e0955-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.654 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:38:29.654 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:38:29.655 DEBUG:boto:Method: GET 2023-09-22T21:38:29.655 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:38:29.655 DEBUG:boto:Data: 2023-09-22T21:38:29.655 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.655 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.655 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.655 DEBUG:boto:Params: {} 2023-09-22T21:38:29.655 DEBUG:boto:Token: None 2023-09-22T21:38:29.656 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj17 2023-09-22T21:38:29.656 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TgFj+llxQ2EcYcei0g6IhpcENXo= 2023-09-22T21:38:29.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TgFj+llxQ2EcYcei0g6IhpcENXo=', 'Content-Length': '0'} 2023-09-22T21:38:29.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064e00a93ce2e38d9-00650e0955-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.659 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:29.659 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:29.659 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:38:29.659 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:38:29.659 DEBUG:boto:Method: GET 2023-09-22T21:38:29.660 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:38:29.660 DEBUG:boto:Data: 2023-09-22T21:38:29.660 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.660 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.660 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.660 DEBUG:boto:Params: {} 2023-09-22T21:38:29.660 DEBUG:boto:Token: None 2023-09-22T21:38:29.660 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/ 2023-09-22T21:38:29.660 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo= 2023-09-22T21:38:29.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo=', 'Content-Length': '0'} 2023-09-22T21:38:29.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5268634f3d2a0e9-00650e0955-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.710 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:29.711 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:29.711 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:29.711 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:38:29.711 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:38:29.711 DEBUG:boto:Method: GET 2023-09-22T21:38:29.711 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:38:29.711 DEBUG:boto:Data: 2023-09-22T21:38:29.711 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.712 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.712 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.712 DEBUG:boto:Params: {} 2023-09-22T21:38:29.712 DEBUG:boto:Token: None 2023-09-22T21:38:29.712 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/ 2023-09-22T21:38:29.712 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo= 2023-09-22T21:38:29.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo=', 'Content-Length': '0'} 2023-09-22T21:38:29.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cee684bbc4b246c-00650e0955-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.719 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:29.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:38:29.720 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:38:29.720 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:38:29.720 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.720 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.720 DEBUG:boto:Method: GET 2023-09-22T21:38:29.720 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.720 DEBUG:boto:Data: 2023-09-22T21:38:29.720 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.721 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.721 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.721 DEBUG:boto:Params: {} 2023-09-22T21:38:29.721 DEBUG:boto:Token: None 2023-09-22T21:38:29.721 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.721 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nyfBHBMzauN2khys0PZ6z95PFl4= 2023-09-22T21:38:29.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nyfBHBMzauN2khys0PZ6z95PFl4=', 'Content-Length': '0'} 2023-09-22T21:38:29.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1e5897a81b6c449-00650e0955-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.724 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:38:29.724 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:38:29.725 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.725 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.725 DEBUG:boto:Method: GET 2023-09-22T21:38:29.725 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.725 DEBUG:boto:Data: 2023-09-22T21:38:29.725 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.725 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.725 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.725 DEBUG:boto:Params: {} 2023-09-22T21:38:29.725 DEBUG:boto:Token: None 2023-09-22T21:38:29.726 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.726 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nyfBHBMzauN2khys0PZ6z95PFl4= 2023-09-22T21:38:29.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nyfBHBMzauN2khys0PZ6z95PFl4=', 'Content-Length': '0'} 2023-09-22T21:38:29.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b39957c7e2417d39-00650e0955-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.729 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:38:29.729 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:38:29.729 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.729 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.729 DEBUG:boto:Method: HEAD 2023-09-22T21:38:29.729 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.729 DEBUG:boto:Data: 2023-09-22T21:38:29.730 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.730 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.730 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.730 DEBUG:boto:Params: {} 2023-09-22T21:38:29.730 DEBUG:boto:Token: None 2023-09-22T21:38:29.730 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.730 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jAMG+UUGhSZuT8ZBapAAEKk9tKQ= 2023-09-22T21:38:29.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jAMG+UUGhSZuT8ZBapAAEKk9tKQ=', 'Content-Length': '0'} 2023-09-22T21:38:29.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caed57989c71bc38-00650e0955-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.733 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:38:29.733 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:38:29.733 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.734 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.734 DEBUG:boto:Method: HEAD 2023-09-22T21:38:29.734 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.734 DEBUG:boto:Data: 2023-09-22T21:38:29.734 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.734 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.734 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.734 DEBUG:boto:Params: {} 2023-09-22T21:38:29.734 DEBUG:boto:Token: None 2023-09-22T21:38:29.735 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.735 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jAMG+UUGhSZuT8ZBapAAEKk9tKQ= 2023-09-22T21:38:29.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jAMG+UUGhSZuT8ZBapAAEKk9tKQ=', 'Content-Length': '0'} 2023-09-22T21:38:29.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016f745e82c4087ea-00650e0955-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:38:29.738 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:38:29.738 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:38:29.738 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.738 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.738 DEBUG:boto:Method: GET 2023-09-22T21:38:29.738 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.738 DEBUG:boto:Data: 2023-09-22T21:38:29.738 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.739 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.739 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.739 DEBUG:boto:Params: {} 2023-09-22T21:38:29.739 DEBUG:boto:Token: None 2023-09-22T21:38:29.739 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.739 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nyfBHBMzauN2khys0PZ6z95PFl4= 2023-09-22T21:38:29.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nyfBHBMzauN2khys0PZ6z95PFl4=', 'Content-Length': '0'} 2023-09-22T21:38:29.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d269d108f8a2e00-00650e0955-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.790 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:38:29.790 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:38:29.791 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.791 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.791 DEBUG:boto:Method: GET 2023-09-22T21:38:29.791 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.791 DEBUG:boto:Data: 2023-09-22T21:38:29.791 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.791 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.791 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.793 DEBUG:boto:Params: {} 2023-09-22T21:38:29.793 DEBUG:boto:Token: None 2023-09-22T21:38:29.793 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:38:29.793 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nyfBHBMzauN2khys0PZ6z95PFl4= 2023-09-22T21:38:29.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nyfBHBMzauN2khys0PZ6z95PFl4=', 'Content-Length': '0'} 2023-09-22T21:38:29.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5935fbe669af001-00650e0955-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.797 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:29.797 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:29.797 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:38:29.797 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:38:29.797 DEBUG:boto:Method: GET 2023-09-22T21:38:29.797 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:38:29.797 DEBUG:boto:Data: 2023-09-22T21:38:29.798 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.798 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.798 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.798 DEBUG:boto:Params: {} 2023-09-22T21:38:29.798 DEBUG:boto:Token: None 2023-09-22T21:38:29.798 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/ 2023-09-22T21:38:29.798 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo= 2023-09-22T21:38:29.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo=', 'Content-Length': '0'} 2023-09-22T21:38:29.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ee8c25e8abf2dd1-00650e0955-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.850 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:29.851 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:29.851 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:29.851 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:38:29.851 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:38:29.851 DEBUG:boto:Method: GET 2023-09-22T21:38:29.851 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:38:29.851 DEBUG:boto:Data: 2023-09-22T21:38:29.852 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.852 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.852 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.852 DEBUG:boto:Params: {} 2023-09-22T21:38:29.852 DEBUG:boto:Token: None 2023-09-22T21:38:29.852 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/ 2023-09-22T21:38:29.852 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo= 2023-09-22T21:38:29.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo=', 'Content-Length': '0'} 2023-09-22T21:38:29.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daca5f34cd3b153b-00650e0955-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.860 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:29.860 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:38:29.861 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:38:29.861 DEBUG:boto:Method: HEAD 2023-09-22T21:38:29.861 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:38:29.861 DEBUG:boto:Data: 2023-09-22T21:38:29.861 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.861 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.861 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.861 DEBUG:boto:Params: {} 2023-09-22T21:38:29.862 DEBUG:boto:Token: None 2023-09-22T21:38:29.862 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj22 2023-09-22T21:38:29.862 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3rDVRis/so3+NjZbukSagaOV4Ho= 2023-09-22T21:38:29.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3rDVRis/so3+NjZbukSagaOV4Ho=', 'Content-Length': '0'} 2023-09-22T21:38:29.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6a5908304d29dff-00650e0955-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.865 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:38:29.865 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:38:29.865 DEBUG:boto:Method: HEAD 2023-09-22T21:38:29.866 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:38:29.866 DEBUG:boto:Data: 2023-09-22T21:38:29.866 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.866 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.866 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.866 DEBUG:boto:Params: {} 2023-09-22T21:38:29.866 DEBUG:boto:Token: None 2023-09-22T21:38:29.866 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj22 2023-09-22T21:38:29.867 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3rDVRis/so3+NjZbukSagaOV4Ho= 2023-09-22T21:38:29.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3rDVRis/so3+NjZbukSagaOV4Ho=', 'Content-Length': '0'} 2023-09-22T21:38:29.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002579f44cbd446d59-00650e0955-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:38:29.870 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:38:29.870 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:38:29.870 DEBUG:boto:Method: GET 2023-09-22T21:38:29.870 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:38:29.870 DEBUG:boto:Data: 2023-09-22T21:38:29.871 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.871 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.871 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.871 DEBUG:boto:Params: {} 2023-09-22T21:38:29.871 DEBUG:boto:Token: None 2023-09-22T21:38:29.871 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj22 2023-09-22T21:38:29.871 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:N1y3UlUN+MKZ+xfQTLykXDZSkkY= 2023-09-22T21:38:29.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:N1y3UlUN+MKZ+xfQTLykXDZSkkY=', 'Content-Length': '0'} 2023-09-22T21:38:29.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016e849f67cfedde6-00650e0955-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.918 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:38:29.919 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:38:29.919 DEBUG:boto:Method: GET 2023-09-22T21:38:29.919 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:38:29.919 DEBUG:boto:Data: 2023-09-22T21:38:29.919 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.919 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.919 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.919 DEBUG:boto:Params: {} 2023-09-22T21:38:29.920 DEBUG:boto:Token: None 2023-09-22T21:38:29.920 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj22 2023-09-22T21:38:29.920 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:N1y3UlUN+MKZ+xfQTLykXDZSkkY= 2023-09-22T21:38:29.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:N1y3UlUN+MKZ+xfQTLykXDZSkkY=', 'Content-Length': '0'} 2023-09-22T21:38:29.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb3e27fe4e724cda-00650e0955-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.923 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:29.923 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:29.923 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:38:29.923 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:38:29.924 DEBUG:boto:Method: GET 2023-09-22T21:38:29.924 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:38:29.924 DEBUG:boto:Data: 2023-09-22T21:38:29.924 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.924 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.924 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.924 DEBUG:boto:Params: {} 2023-09-22T21:38:29.924 DEBUG:boto:Token: None 2023-09-22T21:38:29.925 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/ 2023-09-22T21:38:29.925 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo= 2023-09-22T21:38:29.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo=', 'Content-Length': '0'} 2023-09-22T21:38:29.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8b88ad5b4661c77-00650e0955-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.930 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:29.930 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:29.930 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:29.931 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:38:29.931 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:38:29.931 DEBUG:boto:Method: GET 2023-09-22T21:38:29.931 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:38:29.931 DEBUG:boto:Data: 2023-09-22T21:38:29.931 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.931 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:29.931 DEBUG:boto:Port: 8000 2023-09-22T21:38:29.931 DEBUG:boto:Params: {} 2023-09-22T21:38:29.932 DEBUG:boto:Token: None 2023-09-22T21:38:29.932 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/ 2023-09-22T21:38:29.932 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo= 2023-09-22T21:38:29.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZvFTDd7cTS4oOrWtWmUHjz/7GYo=', 'Content-Length': '0'} 2023-09-22T21:38:29.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c487081de832cb19-00650e0955-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:29.982 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:29.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:38:29.983 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:38:29.983 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:38:29.983 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:29.983 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:29.985 DEBUG:boto:Method: GET 2023-09-22T21:38:29.985 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:29.985 DEBUG:boto:Data: 2023-09-22T21:38:29.985 DEBUG:boto:Headers: {} 2023-09-22T21:38:29.985 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:29.985 DEBUG:boto:Port: 8001 2023-09-22T21:38:29.985 DEBUG:boto:Params: {} 2023-09-22T21:38:29.986 DEBUG:boto:Token: None 2023-09-22T21:38:29.986 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:29 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:29.986 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/tfwYfM5UOnbFFRKt4/4SVibFI0= 2023-09-22T21:38:29.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:29 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/tfwYfM5UOnbFFRKt4/4SVibFI0=', 'Content-Length': '0'} 2023-09-22T21:38:29.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b254c7c46ebe1069-00650e0955-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.030 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:38:30.031 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:38:30.031 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:30.031 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:30.031 DEBUG:boto:Method: GET 2023-09-22T21:38:30.031 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:30.031 DEBUG:boto:Data: 2023-09-22T21:38:30.031 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.032 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.032 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.032 DEBUG:boto:Params: {} 2023-09-22T21:38:30.032 DEBUG:boto:Token: None 2023-09-22T21:38:30.032 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:30.032 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gGl0nxfJ6nIZbxBbJjWmMmZ/aSg= 2023-09-22T21:38:30.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gGl0nxfJ6nIZbxBbJjWmMmZ/aSg=', 'Content-Length': '0'} 2023-09-22T21:38:30.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c212bb43af5752f-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.036 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:38:30.036 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:38:30.036 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:30.036 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:30.036 DEBUG:boto:Method: HEAD 2023-09-22T21:38:30.037 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:30.037 DEBUG:boto:Data: 2023-09-22T21:38:30.037 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.037 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.037 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.037 DEBUG:boto:Params: {} 2023-09-22T21:38:30.037 DEBUG:boto:Token: None 2023-09-22T21:38:30.037 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:30.038 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nAxUPT7sZeeLaA0jsuO3VshxCNU= 2023-09-22T21:38:30.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nAxUPT7sZeeLaA0jsuO3VshxCNU=', 'Content-Length': '0'} 2023-09-22T21:38:30.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045bae4a04b05445e-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.041 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:38:30.041 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:38:30.041 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:30.041 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:30.041 DEBUG:boto:Method: HEAD 2023-09-22T21:38:30.041 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:30.041 DEBUG:boto:Data: 2023-09-22T21:38:30.042 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.042 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.042 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.042 DEBUG:boto:Params: {} 2023-09-22T21:38:30.042 DEBUG:boto:Token: None 2023-09-22T21:38:30.042 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:30.042 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nAxUPT7sZeeLaA0jsuO3VshxCNU= 2023-09-22T21:38:30.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nAxUPT7sZeeLaA0jsuO3VshxCNU=', 'Content-Length': '0'} 2023-09-22T21:38:30.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000242bb4cb5226cc13-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:38:30.045 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:38:30.046 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:38:30.046 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:30.046 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:30.046 DEBUG:boto:Method: GET 2023-09-22T21:38:30.046 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:30.046 DEBUG:boto:Data: 2023-09-22T21:38:30.046 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.046 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.047 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.047 DEBUG:boto:Params: {} 2023-09-22T21:38:30.047 DEBUG:boto:Token: None 2023-09-22T21:38:30.047 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:30.047 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gGl0nxfJ6nIZbxBbJjWmMmZ/aSg= 2023-09-22T21:38:30.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gGl0nxfJ6nIZbxBbJjWmMmZ/aSg=', 'Content-Length': '0'} 2023-09-22T21:38:30.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ae262c37dfcf103-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.090 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:38:30.091 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:38:30.091 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:30.091 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:30.091 DEBUG:boto:Method: GET 2023-09-22T21:38:30.091 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:30.091 DEBUG:boto:Data: 2023-09-22T21:38:30.091 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.091 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.092 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.092 DEBUG:boto:Params: {} 2023-09-22T21:38:30.092 DEBUG:boto:Token: None 2023-09-22T21:38:30.092 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:38:30.092 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gGl0nxfJ6nIZbxBbJjWmMmZ/aSg= 2023-09-22T21:38:30.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gGl0nxfJ6nIZbxBbJjWmMmZ/aSg=', 'Content-Length': '0'} 2023-09-22T21:38:30.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040c8516d3b93998d-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.095 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:30.095 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:30.096 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:38:30.096 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:38:30.096 DEBUG:boto:Method: GET 2023-09-22T21:38:30.096 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:38:30.096 DEBUG:boto:Data: 2023-09-22T21:38:30.096 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.096 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.096 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.097 DEBUG:boto:Params: {} 2023-09-22T21:38:30.097 DEBUG:boto:Token: None 2023-09-22T21:38:30.097 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/ 2023-09-22T21:38:30.097 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y= 2023-09-22T21:38:30.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y=', 'Content-Length': '0'} 2023-09-22T21:38:30.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053de9947a6c941ca-00650e0956-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.102 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:30.102 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:30.102 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:30.103 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:38:30.103 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:38:30.103 DEBUG:boto:Method: GET 2023-09-22T21:38:30.103 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:38:30.103 DEBUG:boto:Data: 2023-09-22T21:38:30.103 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.103 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.103 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.103 DEBUG:boto:Params: {} 2023-09-22T21:38:30.104 DEBUG:boto:Token: None 2023-09-22T21:38:30.104 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/ 2023-09-22T21:38:30.104 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y= 2023-09-22T21:38:30.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y=', 'Content-Length': '0'} 2023-09-22T21:38:30.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbab8f812a09dfb0-00650e0956-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.111 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:30.111 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:38:30.111 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:38:30.112 DEBUG:boto:Method: HEAD 2023-09-22T21:38:30.112 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:38:30.112 DEBUG:boto:Data: 2023-09-22T21:38:30.112 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.112 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.112 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.112 DEBUG:boto:Params: {} 2023-09-22T21:38:30.112 DEBUG:boto:Token: None 2023-09-22T21:38:30.113 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj33 2023-09-22T21:38:30.113 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M6uojVpzexGfyiaNdVTAimLTCxE= 2023-09-22T21:38:30.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M6uojVpzexGfyiaNdVTAimLTCxE=', 'Content-Length': '0'} 2023-09-22T21:38:30.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1ba4f6d9682b13b-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.115 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:38:30.116 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:38:30.116 DEBUG:boto:Method: HEAD 2023-09-22T21:38:30.116 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:38:30.116 DEBUG:boto:Data: 2023-09-22T21:38:30.116 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.116 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.116 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.117 DEBUG:boto:Params: {} 2023-09-22T21:38:30.117 DEBUG:boto:Token: None 2023-09-22T21:38:30.117 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj33 2023-09-22T21:38:30.117 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M6uojVpzexGfyiaNdVTAimLTCxE= 2023-09-22T21:38:30.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M6uojVpzexGfyiaNdVTAimLTCxE=', 'Content-Length': '0'} 2023-09-22T21:38:30.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065b7a1b6ded6d2d4-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:38:30.120 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:38:30.120 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:38:30.120 DEBUG:boto:Method: GET 2023-09-22T21:38:30.120 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:38:30.120 DEBUG:boto:Data: 2023-09-22T21:38:30.121 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.121 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.121 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.121 DEBUG:boto:Params: {} 2023-09-22T21:38:30.121 DEBUG:boto:Token: None 2023-09-22T21:38:30.121 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj33 2023-09-22T21:38:30.121 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AF3OQZltaRTOdQKZOeHlsbUNA/U= 2023-09-22T21:38:30.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AF3OQZltaRTOdQKZOeHlsbUNA/U=', 'Content-Length': '0'} 2023-09-22T21:38:30.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d11125eeb5616960-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.166 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:38:30.166 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:38:30.167 DEBUG:boto:Method: GET 2023-09-22T21:38:30.167 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:38:30.167 DEBUG:boto:Data: 2023-09-22T21:38:30.167 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.167 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.167 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.167 DEBUG:boto:Params: {} 2023-09-22T21:38:30.167 DEBUG:boto:Token: None 2023-09-22T21:38:30.168 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj33 2023-09-22T21:38:30.168 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AF3OQZltaRTOdQKZOeHlsbUNA/U= 2023-09-22T21:38:30.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AF3OQZltaRTOdQKZOeHlsbUNA/U=', 'Content-Length': '0'} 2023-09-22T21:38:30.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f4df4e790dbf53c-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.171 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:30.171 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:30.171 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:38:30.171 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:38:30.171 DEBUG:boto:Method: GET 2023-09-22T21:38:30.172 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:38:30.172 DEBUG:boto:Data: 2023-09-22T21:38:30.172 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.172 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.172 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.172 DEBUG:boto:Params: {} 2023-09-22T21:38:30.172 DEBUG:boto:Token: None 2023-09-22T21:38:30.172 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/ 2023-09-22T21:38:30.173 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y= 2023-09-22T21:38:30.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y=', 'Content-Length': '0'} 2023-09-22T21:38:30.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000083275c5b0da65a5-00650e0956-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.178 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:30.179 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:30.179 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:30.179 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:38:30.179 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:38:30.179 DEBUG:boto:Method: GET 2023-09-22T21:38:30.179 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:38:30.179 DEBUG:boto:Data: 2023-09-22T21:38:30.180 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.180 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.180 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.180 DEBUG:boto:Params: {} 2023-09-22T21:38:30.180 DEBUG:boto:Token: None 2023-09-22T21:38:30.180 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/ 2023-09-22T21:38:30.180 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y= 2023-09-22T21:38:30.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y=', 'Content-Length': '0'} 2023-09-22T21:38:30.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efc1ee148ae7177d-00650e0956-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.187 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:30.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:38:30.188 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:38:30.188 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:38:30.188 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.188 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.188 DEBUG:boto:Method: GET 2023-09-22T21:38:30.188 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.188 DEBUG:boto:Data: 2023-09-22T21:38:30.189 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.189 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.189 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.189 DEBUG:boto:Params: {} 2023-09-22T21:38:30.189 DEBUG:boto:Token: None 2023-09-22T21:38:30.189 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.189 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fq1BnmQXVxLv/XHLfdHrYaxJtJ8= 2023-09-22T21:38:30.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fq1BnmQXVxLv/XHLfdHrYaxJtJ8=', 'Content-Length': '0'} 2023-09-22T21:38:30.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082f654c0e8955304-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.193 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:38:30.193 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:38:30.193 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.194 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.194 DEBUG:boto:Method: GET 2023-09-22T21:38:30.194 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.194 DEBUG:boto:Data: 2023-09-22T21:38:30.194 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.194 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.194 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.194 DEBUG:boto:Params: {} 2023-09-22T21:38:30.195 DEBUG:boto:Token: None 2023-09-22T21:38:30.195 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.195 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fq1BnmQXVxLv/XHLfdHrYaxJtJ8= 2023-09-22T21:38:30.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fq1BnmQXVxLv/XHLfdHrYaxJtJ8=', 'Content-Length': '0'} 2023-09-22T21:38:30.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b7f89b356565ee5-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.198 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:38:30.198 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:38:30.198 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.198 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.198 DEBUG:boto:Method: HEAD 2023-09-22T21:38:30.199 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.199 DEBUG:boto:Data: 2023-09-22T21:38:30.199 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.199 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.199 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.199 DEBUG:boto:Params: {} 2023-09-22T21:38:30.199 DEBUG:boto:Token: None 2023-09-22T21:38:30.199 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.200 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fsXTnpowtApV80bkKepG8MdYMuc= 2023-09-22T21:38:30.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fsXTnpowtApV80bkKepG8MdYMuc=', 'Content-Length': '0'} 2023-09-22T21:38:30.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eba88711dbd90510-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.203 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:38:30.203 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:38:30.203 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.203 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.205 DEBUG:boto:Method: HEAD 2023-09-22T21:38:30.205 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.205 DEBUG:boto:Data: 2023-09-22T21:38:30.205 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.205 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.206 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.206 DEBUG:boto:Params: {} 2023-09-22T21:38:30.206 DEBUG:boto:Token: None 2023-09-22T21:38:30.206 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.206 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fsXTnpowtApV80bkKepG8MdYMuc= 2023-09-22T21:38:30.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fsXTnpowtApV80bkKepG8MdYMuc=', 'Content-Length': '0'} 2023-09-22T21:38:30.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9624f1664e62b59-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:38:30.209 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:38:30.209 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:38:30.209 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.210 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.210 DEBUG:boto:Method: GET 2023-09-22T21:38:30.210 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.210 DEBUG:boto:Data: 2023-09-22T21:38:30.210 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.210 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.210 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.210 DEBUG:boto:Params: {} 2023-09-22T21:38:30.211 DEBUG:boto:Token: None 2023-09-22T21:38:30.211 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.211 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fq1BnmQXVxLv/XHLfdHrYaxJtJ8= 2023-09-22T21:38:30.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fq1BnmQXVxLv/XHLfdHrYaxJtJ8=', 'Content-Length': '0'} 2023-09-22T21:38:30.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007275190401a804cd-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.254 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:38:30.255 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:38:30.255 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.255 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.255 DEBUG:boto:Method: GET 2023-09-22T21:38:30.255 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.255 DEBUG:boto:Data: 2023-09-22T21:38:30.255 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.255 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.255 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.256 DEBUG:boto:Params: {} 2023-09-22T21:38:30.256 DEBUG:boto:Token: None 2023-09-22T21:38:30.256 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:38:30.256 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fq1BnmQXVxLv/XHLfdHrYaxJtJ8= 2023-09-22T21:38:30.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fq1BnmQXVxLv/XHLfdHrYaxJtJ8=', 'Content-Length': '0'} 2023-09-22T21:38:30.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c57c3772a64af9a7-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.259 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:30.259 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:30.260 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:38:30.260 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:38:30.260 DEBUG:boto:Method: GET 2023-09-22T21:38:30.260 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:38:30.260 DEBUG:boto:Data: 2023-09-22T21:38:30.260 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.260 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.260 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.260 DEBUG:boto:Params: {} 2023-09-22T21:38:30.261 DEBUG:boto:Token: None 2023-09-22T21:38:30.261 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/ 2023-09-22T21:38:30.261 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y= 2023-09-22T21:38:30.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y=', 'Content-Length': '0'} 2023-09-22T21:38:30.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e07e5bc80d819838-00650e0956-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.266 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:30.266 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:30.266 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:30.267 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:38:30.267 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:38:30.267 DEBUG:boto:Method: GET 2023-09-22T21:38:30.267 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:38:30.267 DEBUG:boto:Data: 2023-09-22T21:38:30.267 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.267 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.267 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.268 DEBUG:boto:Params: {} 2023-09-22T21:38:30.268 DEBUG:boto:Token: None 2023-09-22T21:38:30.268 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/ 2023-09-22T21:38:30.268 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y= 2023-09-22T21:38:30.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y=', 'Content-Length': '0'} 2023-09-22T21:38:30.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b7f6f9128f01f9a-00650e0956-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.318 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:30.319 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:38:30.319 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:38:30.319 DEBUG:boto:Method: HEAD 2023-09-22T21:38:30.319 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:38:30.319 DEBUG:boto:Data: 2023-09-22T21:38:30.320 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.320 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.320 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.320 DEBUG:boto:Params: {} 2023-09-22T21:38:30.320 DEBUG:boto:Token: None 2023-09-22T21:38:30.320 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj34 2023-09-22T21:38:30.320 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GSMBXiL1gfBz05P2hA3vZfCzJFQ= 2023-09-22T21:38:30.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GSMBXiL1gfBz05P2hA3vZfCzJFQ=', 'Content-Length': '0'} 2023-09-22T21:38:30.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004965331000634481-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.323 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:38:30.323 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:38:30.324 DEBUG:boto:Method: HEAD 2023-09-22T21:38:30.324 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:38:30.324 DEBUG:boto:Data: 2023-09-22T21:38:30.324 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.324 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.324 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.324 DEBUG:boto:Params: {} 2023-09-22T21:38:30.324 DEBUG:boto:Token: None 2023-09-22T21:38:30.325 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj34 2023-09-22T21:38:30.325 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GSMBXiL1gfBz05P2hA3vZfCzJFQ= 2023-09-22T21:38:30.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GSMBXiL1gfBz05P2hA3vZfCzJFQ=', 'Content-Length': '0'} 2023-09-22T21:38:30.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7cfce039880b4f4-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:38:30.328 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:38:30.328 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:38:30.328 DEBUG:boto:Method: GET 2023-09-22T21:38:30.328 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:38:30.329 DEBUG:boto:Data: 2023-09-22T21:38:30.329 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.329 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.329 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.329 DEBUG:boto:Params: {} 2023-09-22T21:38:30.329 DEBUG:boto:Token: None 2023-09-22T21:38:30.329 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj34 2023-09-22T21:38:30.329 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YHgM9YadrpNoEDJDg1ubHyyB13U= 2023-09-22T21:38:30.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YHgM9YadrpNoEDJDg1ubHyyB13U=', 'Content-Length': '0'} 2023-09-22T21:38:30.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aab9e09e77aec5a0-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.374 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:38:30.375 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:38:30.375 DEBUG:boto:Method: GET 2023-09-22T21:38:30.375 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:38:30.375 DEBUG:boto:Data: 2023-09-22T21:38:30.375 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.375 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.375 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.375 DEBUG:boto:Params: {} 2023-09-22T21:38:30.375 DEBUG:boto:Token: None 2023-09-22T21:38:30.376 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj34 2023-09-22T21:38:30.376 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YHgM9YadrpNoEDJDg1ubHyyB13U= 2023-09-22T21:38:30.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YHgM9YadrpNoEDJDg1ubHyyB13U=', 'Content-Length': '0'} 2023-09-22T21:38:30.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e082d769da2d2eb-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.379 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:30.379 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:30.379 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:38:30.379 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:38:30.379 DEBUG:boto:Method: GET 2023-09-22T21:38:30.380 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:38:30.380 DEBUG:boto:Data: 2023-09-22T21:38:30.380 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.380 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.380 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.380 DEBUG:boto:Params: {} 2023-09-22T21:38:30.380 DEBUG:boto:Token: None 2023-09-22T21:38:30.380 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/ 2023-09-22T21:38:30.380 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y= 2023-09-22T21:38:30.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y=', 'Content-Length': '0'} 2023-09-22T21:38:30.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c909d62ba57389c6-00650e0956-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.385 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:30.386 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:30.386 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:30.386 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:38:30.386 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:38:30.386 DEBUG:boto:Method: GET 2023-09-22T21:38:30.386 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:38:30.386 DEBUG:boto:Data: 2023-09-22T21:38:30.387 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.387 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.387 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.387 DEBUG:boto:Params: {} 2023-09-22T21:38:30.387 DEBUG:boto:Token: None 2023-09-22T21:38:30.387 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/ 2023-09-22T21:38:30.387 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y= 2023-09-22T21:38:30.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y=', 'Content-Length': '0'} 2023-09-22T21:38:30.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e979d87d57044cc0-00650e0956-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.394 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:30.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:38:30.394 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:38:30.395 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:38:30.395 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.395 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.395 DEBUG:boto:Method: GET 2023-09-22T21:38:30.395 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.395 DEBUG:boto:Data: 2023-09-22T21:38:30.395 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.395 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.395 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.395 DEBUG:boto:Params: {} 2023-09-22T21:38:30.396 DEBUG:boto:Token: None 2023-09-22T21:38:30.396 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.396 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YhiLWot3faMtfy8wHH5ZBeaEd78= 2023-09-22T21:38:30.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YhiLWot3faMtfy8wHH5ZBeaEd78=', 'Content-Length': '0'} 2023-09-22T21:38:30.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d46b6a29de639884-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.399 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:38:30.399 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:38:30.399 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.399 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.399 DEBUG:boto:Method: GET 2023-09-22T21:38:30.400 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.400 DEBUG:boto:Data: 2023-09-22T21:38:30.400 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.400 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.400 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.400 DEBUG:boto:Params: {} 2023-09-22T21:38:30.400 DEBUG:boto:Token: None 2023-09-22T21:38:30.400 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.400 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YhiLWot3faMtfy8wHH5ZBeaEd78= 2023-09-22T21:38:30.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YhiLWot3faMtfy8wHH5ZBeaEd78=', 'Content-Length': '0'} 2023-09-22T21:38:30.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c6a590f825201f5-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.446 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:38:30.447 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:38:30.447 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.447 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.447 DEBUG:boto:Method: HEAD 2023-09-22T21:38:30.447 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.447 DEBUG:boto:Data: 2023-09-22T21:38:30.447 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.447 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.447 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.448 DEBUG:boto:Params: {} 2023-09-22T21:38:30.448 DEBUG:boto:Token: None 2023-09-22T21:38:30.448 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.448 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SOZZ8UuOvlju6twWX1fJ1w1O2es= 2023-09-22T21:38:30.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SOZZ8UuOvlju6twWX1fJ1w1O2es=', 'Content-Length': '0'} 2023-09-22T21:38:30.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000315c138de2a006f9-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.451 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:38:30.451 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:38:30.451 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.452 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.452 DEBUG:boto:Method: HEAD 2023-09-22T21:38:30.452 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.452 DEBUG:boto:Data: 2023-09-22T21:38:30.452 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.452 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.452 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.452 DEBUG:boto:Params: {} 2023-09-22T21:38:30.452 DEBUG:boto:Token: None 2023-09-22T21:38:30.453 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.453 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SOZZ8UuOvlju6twWX1fJ1w1O2es= 2023-09-22T21:38:30.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SOZZ8UuOvlju6twWX1fJ1w1O2es=', 'Content-Length': '0'} 2023-09-22T21:38:30.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb2b7e10fc1884cd-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:38:30.456 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:38:30.456 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:38:30.456 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.456 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.456 DEBUG:boto:Method: GET 2023-09-22T21:38:30.456 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.457 DEBUG:boto:Data: 2023-09-22T21:38:30.457 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.457 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.457 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.457 DEBUG:boto:Params: {} 2023-09-22T21:38:30.457 DEBUG:boto:Token: None 2023-09-22T21:38:30.457 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.457 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YhiLWot3faMtfy8wHH5ZBeaEd78= 2023-09-22T21:38:30.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YhiLWot3faMtfy8wHH5ZBeaEd78=', 'Content-Length': '0'} 2023-09-22T21:38:30.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab6e7481138e6d52-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.502 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:38:30.503 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:38:30.503 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.503 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.503 DEBUG:boto:Method: GET 2023-09-22T21:38:30.503 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.503 DEBUG:boto:Data: 2023-09-22T21:38:30.503 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.503 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.504 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.504 DEBUG:boto:Params: {} 2023-09-22T21:38:30.504 DEBUG:boto:Token: None 2023-09-22T21:38:30.504 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:38:30.504 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YhiLWot3faMtfy8wHH5ZBeaEd78= 2023-09-22T21:38:30.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YhiLWot3faMtfy8wHH5ZBeaEd78=', 'Content-Length': '0'} 2023-09-22T21:38:30.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b91c2850eaeb48f-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.507 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:30.508 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:30.508 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:38:30.508 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:38:30.508 DEBUG:boto:Method: GET 2023-09-22T21:38:30.508 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:38:30.508 DEBUG:boto:Data: 2023-09-22T21:38:30.508 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.508 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.508 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.509 DEBUG:boto:Params: {} 2023-09-22T21:38:30.509 DEBUG:boto:Token: None 2023-09-22T21:38:30.509 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/ 2023-09-22T21:38:30.509 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y= 2023-09-22T21:38:30.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y=', 'Content-Length': '0'} 2023-09-22T21:38:30.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034b5258fee9644fc-00650e0956-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.514 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:30.514 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:30.514 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:30.514 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:38:30.514 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:38:30.515 DEBUG:boto:Method: GET 2023-09-22T21:38:30.515 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:38:30.515 DEBUG:boto:Data: 2023-09-22T21:38:30.515 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.515 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.515 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.515 DEBUG:boto:Params: {} 2023-09-22T21:38:30.515 DEBUG:boto:Token: None 2023-09-22T21:38:30.515 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/ 2023-09-22T21:38:30.516 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y= 2023-09-22T21:38:30.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y=', 'Content-Length': '0'} 2023-09-22T21:38:30.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059326be795cc0981-00650e0956-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.523 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:30.523 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:38:30.524 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:38:30.524 DEBUG:boto:Method: HEAD 2023-09-22T21:38:30.524 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:38:30.524 DEBUG:boto:Data: 2023-09-22T21:38:30.524 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.524 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.524 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.524 DEBUG:boto:Params: {} 2023-09-22T21:38:30.524 DEBUG:boto:Token: None 2023-09-22T21:38:30.525 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj44 2023-09-22T21:38:30.525 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OPnv6kws25EtikhcTmqbrsZ4qgI= 2023-09-22T21:38:30.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OPnv6kws25EtikhcTmqbrsZ4qgI=', 'Content-Length': '0'} 2023-09-22T21:38:30.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0bdef63e86a7373-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.528 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:38:30.528 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:38:30.528 DEBUG:boto:Method: HEAD 2023-09-22T21:38:30.528 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:38:30.528 DEBUG:boto:Data: 2023-09-22T21:38:30.528 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.528 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.528 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.528 DEBUG:boto:Params: {} 2023-09-22T21:38:30.529 DEBUG:boto:Token: None 2023-09-22T21:38:30.529 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj44 2023-09-22T21:38:30.529 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OPnv6kws25EtikhcTmqbrsZ4qgI= 2023-09-22T21:38:30.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OPnv6kws25EtikhcTmqbrsZ4qgI=', 'Content-Length': '0'} 2023-09-22T21:38:30.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016cbbe1159283e97-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:38:30.532 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:38:30.532 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:38:30.532 DEBUG:boto:Method: GET 2023-09-22T21:38:30.532 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:38:30.532 DEBUG:boto:Data: 2023-09-22T21:38:30.533 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.533 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.533 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.533 DEBUG:boto:Params: {} 2023-09-22T21:38:30.533 DEBUG:boto:Token: None 2023-09-22T21:38:30.533 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj44 2023-09-22T21:38:30.533 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8vgU++4Pfg0IlEqEZ49A7lGK7nA= 2023-09-22T21:38:30.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8vgU++4Pfg0IlEqEZ49A7lGK7nA=', 'Content-Length': '0'} 2023-09-22T21:38:30.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4f215a64f118e7c-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.578 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:38:30.579 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:38:30.579 DEBUG:boto:Method: GET 2023-09-22T21:38:30.579 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:38:30.579 DEBUG:boto:Data: 2023-09-22T21:38:30.579 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.579 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.579 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.579 DEBUG:boto:Params: {} 2023-09-22T21:38:30.579 DEBUG:boto:Token: None 2023-09-22T21:38:30.580 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj44 2023-09-22T21:38:30.580 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8vgU++4Pfg0IlEqEZ49A7lGK7nA= 2023-09-22T21:38:30.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8vgU++4Pfg0IlEqEZ49A7lGK7nA=', 'Content-Length': '0'} 2023-09-22T21:38:30.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002313beecbe68dd23-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.583 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:30.583 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:30.583 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:38:30.584 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:38:30.584 DEBUG:boto:Method: GET 2023-09-22T21:38:30.584 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:38:30.584 DEBUG:boto:Data: 2023-09-22T21:38:30.584 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.584 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.584 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.584 DEBUG:boto:Params: {} 2023-09-22T21:38:30.584 DEBUG:boto:Token: None 2023-09-22T21:38:30.585 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/ 2023-09-22T21:38:30.585 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y= 2023-09-22T21:38:30.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y=', 'Content-Length': '0'} 2023-09-22T21:38:30.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065073a9fe7184079-00650e0956-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.589 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:30.590 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:30.590 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:30.590 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:38:30.590 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:38:30.590 DEBUG:boto:Method: GET 2023-09-22T21:38:30.590 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:38:30.590 DEBUG:boto:Data: 2023-09-22T21:38:30.591 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.591 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.591 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.591 DEBUG:boto:Params: {} 2023-09-22T21:38:30.591 DEBUG:boto:Token: None 2023-09-22T21:38:30.591 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/ 2023-09-22T21:38:30.591 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y= 2023-09-22T21:38:30.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y=', 'Content-Length': '0'} 2023-09-22T21:38:30.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a03a5c9c2e661da-00650e0956-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.597 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:30.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:38:30.598 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:38:30.598 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:38:30.598 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.598 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.598 DEBUG:boto:Method: GET 2023-09-22T21:38:30.598 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.598 DEBUG:boto:Data: 2023-09-22T21:38:30.598 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.599 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.599 DEBUG:boto:Params: {} 2023-09-22T21:38:30.599 DEBUG:boto:Token: None 2023-09-22T21:38:30.599 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.599 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mO8g0ucgtj0FZdn5e8WHK0Vm/10= 2023-09-22T21:38:30.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mO8g0ucgtj0FZdn5e8WHK0Vm/10=', 'Content-Length': '0'} 2023-09-22T21:38:30.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e47800a37307963f-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.602 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:38:30.603 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:38:30.603 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.603 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.603 DEBUG:boto:Method: GET 2023-09-22T21:38:30.603 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.603 DEBUG:boto:Data: 2023-09-22T21:38:30.603 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.603 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.603 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.604 DEBUG:boto:Params: {} 2023-09-22T21:38:30.604 DEBUG:boto:Token: None 2023-09-22T21:38:30.604 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.604 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mO8g0ucgtj0FZdn5e8WHK0Vm/10= 2023-09-22T21:38:30.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mO8g0ucgtj0FZdn5e8WHK0Vm/10=', 'Content-Length': '0'} 2023-09-22T21:38:30.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009de7614a8badee31-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.607 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:38:30.607 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:38:30.607 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.607 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.608 DEBUG:boto:Method: HEAD 2023-09-22T21:38:30.608 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.608 DEBUG:boto:Data: 2023-09-22T21:38:30.608 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.608 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.608 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.608 DEBUG:boto:Params: {} 2023-09-22T21:38:30.608 DEBUG:boto:Token: None 2023-09-22T21:38:30.608 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.609 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bPGRXAl5LUzHvLefslVsagFenwg= 2023-09-22T21:38:30.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bPGRXAl5LUzHvLefslVsagFenwg=', 'Content-Length': '0'} 2023-09-22T21:38:30.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005655779830e8fabc-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.611 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:38:30.612 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:38:30.612 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.612 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.612 DEBUG:boto:Method: HEAD 2023-09-22T21:38:30.612 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.612 DEBUG:boto:Data: 2023-09-22T21:38:30.612 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.612 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.612 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.612 DEBUG:boto:Params: {} 2023-09-22T21:38:30.613 DEBUG:boto:Token: None 2023-09-22T21:38:30.613 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.613 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bPGRXAl5LUzHvLefslVsagFenwg= 2023-09-22T21:38:30.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bPGRXAl5LUzHvLefslVsagFenwg=', 'Content-Length': '0'} 2023-09-22T21:38:30.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000975cf360516d2f6-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:38:30.616 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:38:30.616 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:38:30.616 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.617 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.617 DEBUG:boto:Method: GET 2023-09-22T21:38:30.617 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.617 DEBUG:boto:Data: 2023-09-22T21:38:30.617 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.617 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.617 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.617 DEBUG:boto:Params: {} 2023-09-22T21:38:30.617 DEBUG:boto:Token: None 2023-09-22T21:38:30.618 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.618 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mO8g0ucgtj0FZdn5e8WHK0Vm/10= 2023-09-22T21:38:30.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mO8g0ucgtj0FZdn5e8WHK0Vm/10=', 'Content-Length': '0'} 2023-09-22T21:38:30.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009309b37777a66968-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.662 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:38:30.663 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:38:30.663 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.663 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.663 DEBUG:boto:Method: GET 2023-09-22T21:38:30.663 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.663 DEBUG:boto:Data: 2023-09-22T21:38:30.663 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.663 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.664 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.664 DEBUG:boto:Params: {} 2023-09-22T21:38:30.664 DEBUG:boto:Token: None 2023-09-22T21:38:30.664 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:38:30.664 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mO8g0ucgtj0FZdn5e8WHK0Vm/10= 2023-09-22T21:38:30.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mO8g0ucgtj0FZdn5e8WHK0Vm/10=', 'Content-Length': '0'} 2023-09-22T21:38:30.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d940d43179561a24-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.667 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:30.667 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:30.667 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:38:30.668 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:38:30.668 DEBUG:boto:Method: GET 2023-09-22T21:38:30.668 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:38:30.668 DEBUG:boto:Data: 2023-09-22T21:38:30.668 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.668 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.668 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.668 DEBUG:boto:Params: {} 2023-09-22T21:38:30.668 DEBUG:boto:Token: None 2023-09-22T21:38:30.669 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/ 2023-09-22T21:38:30.669 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y= 2023-09-22T21:38:30.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y=', 'Content-Length': '0'} 2023-09-22T21:38:30.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e252dfe154889452-00650e0956-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.674 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:30.674 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:30.675 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:30.675 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:38:30.675 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:38:30.675 DEBUG:boto:Method: GET 2023-09-22T21:38:30.675 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:38:30.675 DEBUG:boto:Data: 2023-09-22T21:38:30.675 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.675 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.676 DEBUG:boto:Params: {} 2023-09-22T21:38:30.676 DEBUG:boto:Token: None 2023-09-22T21:38:30.676 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/ 2023-09-22T21:38:30.676 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y= 2023-09-22T21:38:30.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y=', 'Content-Length': '0'} 2023-09-22T21:38:30.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000524064eb3a43098a-00650e0956-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.683 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:30.684 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:38:30.684 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:38:30.684 DEBUG:boto:Method: HEAD 2023-09-22T21:38:30.684 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:38:30.684 DEBUG:boto:Data: 2023-09-22T21:38:30.684 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.684 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.685 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.685 DEBUG:boto:Params: {} 2023-09-22T21:38:30.685 DEBUG:boto:Token: None 2023-09-22T21:38:30.685 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj51 2023-09-22T21:38:30.685 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yTUsom8cCKNS9kVVJJyC7tsh18g= 2023-09-22T21:38:30.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yTUsom8cCKNS9kVVJJyC7tsh18g=', 'Content-Length': '0'} 2023-09-22T21:38:30.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec107b08abbcae3b-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.688 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:38:30.688 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:38:30.688 DEBUG:boto:Method: HEAD 2023-09-22T21:38:30.688 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:38:30.688 DEBUG:boto:Data: 2023-09-22T21:38:30.688 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.688 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.689 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.689 DEBUG:boto:Params: {} 2023-09-22T21:38:30.689 DEBUG:boto:Token: None 2023-09-22T21:38:30.689 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj51 2023-09-22T21:38:30.689 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yTUsom8cCKNS9kVVJJyC7tsh18g= 2023-09-22T21:38:30.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yTUsom8cCKNS9kVVJJyC7tsh18g=', 'Content-Length': '0'} 2023-09-22T21:38:30.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f06fbd99ae5623a1-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:38:30.692 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:38:30.692 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:38:30.692 DEBUG:boto:Method: GET 2023-09-22T21:38:30.692 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:38:30.693 DEBUG:boto:Data: 2023-09-22T21:38:30.693 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.693 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.693 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.693 DEBUG:boto:Params: {} 2023-09-22T21:38:30.693 DEBUG:boto:Token: None 2023-09-22T21:38:30.693 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj51 2023-09-22T21:38:30.693 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6+I+bZhM3bSig//TvEJ9vfRYvic= 2023-09-22T21:38:30.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6+I+bZhM3bSig//TvEJ9vfRYvic=', 'Content-Length': '0'} 2023-09-22T21:38:30.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bf1f6a64529927a-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.738 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:38:30.739 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:38:30.739 DEBUG:boto:Method: GET 2023-09-22T21:38:30.739 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:38:30.739 DEBUG:boto:Data: 2023-09-22T21:38:30.739 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.739 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.739 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.739 DEBUG:boto:Params: {} 2023-09-22T21:38:30.739 DEBUG:boto:Token: None 2023-09-22T21:38:30.740 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj51 2023-09-22T21:38:30.740 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6+I+bZhM3bSig//TvEJ9vfRYvic= 2023-09-22T21:38:30.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6+I+bZhM3bSig//TvEJ9vfRYvic=', 'Content-Length': '0'} 2023-09-22T21:38:30.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc6cd288a733c511-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.743 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:30.743 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:30.743 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:38:30.743 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:38:30.744 DEBUG:boto:Method: GET 2023-09-22T21:38:30.744 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:38:30.744 DEBUG:boto:Data: 2023-09-22T21:38:30.744 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.744 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.744 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.744 DEBUG:boto:Params: {} 2023-09-22T21:38:30.744 DEBUG:boto:Token: None 2023-09-22T21:38:30.744 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/ 2023-09-22T21:38:30.745 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y= 2023-09-22T21:38:30.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y=', 'Content-Length': '0'} 2023-09-22T21:38:30.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007811fcb96987bd86-00650e0956-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.750 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:30.750 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:30.750 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:30.750 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:38:30.751 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:38:30.751 DEBUG:boto:Method: GET 2023-09-22T21:38:30.751 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:38:30.751 DEBUG:boto:Data: 2023-09-22T21:38:30.751 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.751 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.751 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.751 DEBUG:boto:Params: {} 2023-09-22T21:38:30.751 DEBUG:boto:Token: None 2023-09-22T21:38:30.752 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/ 2023-09-22T21:38:30.752 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y= 2023-09-22T21:38:30.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y=', 'Content-Length': '0'} 2023-09-22T21:38:30.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af6efe6a8248cb6c-00650e0956-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.806 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:30.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:38:30.807 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:38:30.807 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:38:30.807 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.807 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.807 DEBUG:boto:Method: GET 2023-09-22T21:38:30.807 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.807 DEBUG:boto:Data: 2023-09-22T21:38:30.808 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.808 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.808 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.808 DEBUG:boto:Params: {} 2023-09-22T21:38:30.808 DEBUG:boto:Token: None 2023-09-22T21:38:30.808 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.808 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VVW7O4PhI5XQqZIm5TpWTrSJZec= 2023-09-22T21:38:30.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VVW7O4PhI5XQqZIm5TpWTrSJZec=', 'Content-Length': '0'} 2023-09-22T21:38:30.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002184ad3014fa458d-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.812 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:38:30.812 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:38:30.812 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.812 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.812 DEBUG:boto:Method: GET 2023-09-22T21:38:30.812 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.812 DEBUG:boto:Data: 2023-09-22T21:38:30.812 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.813 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.813 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.813 DEBUG:boto:Params: {} 2023-09-22T21:38:30.813 DEBUG:boto:Token: None 2023-09-22T21:38:30.813 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.813 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VVW7O4PhI5XQqZIm5TpWTrSJZec= 2023-09-22T21:38:30.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VVW7O4PhI5XQqZIm5TpWTrSJZec=', 'Content-Length': '0'} 2023-09-22T21:38:30.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f2e514f76fff395-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.816 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:38:30.816 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:38:30.816 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.817 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.817 DEBUG:boto:Method: HEAD 2023-09-22T21:38:30.817 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.817 DEBUG:boto:Data: 2023-09-22T21:38:30.817 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.817 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.817 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.817 DEBUG:boto:Params: {} 2023-09-22T21:38:30.817 DEBUG:boto:Token: None 2023-09-22T21:38:30.817 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.818 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wdi5CjymzLzrWg8mb50Qj7ENbGI= 2023-09-22T21:38:30.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wdi5CjymzLzrWg8mb50Qj7ENbGI=', 'Content-Length': '0'} 2023-09-22T21:38:30.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000370266f421615a40-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.821 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:38:30.821 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:38:30.821 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.821 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.821 DEBUG:boto:Method: HEAD 2023-09-22T21:38:30.821 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.821 DEBUG:boto:Data: 2023-09-22T21:38:30.821 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.821 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.822 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.822 DEBUG:boto:Params: {} 2023-09-22T21:38:30.822 DEBUG:boto:Token: None 2023-09-22T21:38:30.822 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.822 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wdi5CjymzLzrWg8mb50Qj7ENbGI= 2023-09-22T21:38:30.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wdi5CjymzLzrWg8mb50Qj7ENbGI=', 'Content-Length': '0'} 2023-09-22T21:38:30.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002aa11f46c594cce6-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:38:30.825 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:38:30.825 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:38:30.825 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.825 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.825 DEBUG:boto:Method: GET 2023-09-22T21:38:30.826 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.826 DEBUG:boto:Data: 2023-09-22T21:38:30.826 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.826 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.826 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.826 DEBUG:boto:Params: {} 2023-09-22T21:38:30.826 DEBUG:boto:Token: None 2023-09-22T21:38:30.826 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.826 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VVW7O4PhI5XQqZIm5TpWTrSJZec= 2023-09-22T21:38:30.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VVW7O4PhI5XQqZIm5TpWTrSJZec=', 'Content-Length': '0'} 2023-09-22T21:38:30.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7a687b90d9a542a-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.874 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:38:30.875 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:38:30.875 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.875 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.875 DEBUG:boto:Method: GET 2023-09-22T21:38:30.875 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.875 DEBUG:boto:Data: 2023-09-22T21:38:30.875 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.875 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.875 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.876 DEBUG:boto:Params: {} 2023-09-22T21:38:30.876 DEBUG:boto:Token: None 2023-09-22T21:38:30.876 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:38:30.876 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VVW7O4PhI5XQqZIm5TpWTrSJZec= 2023-09-22T21:38:30.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VVW7O4PhI5XQqZIm5TpWTrSJZec=', 'Content-Length': '0'} 2023-09-22T21:38:30.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074d5405ee596f88b-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.879 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:30.880 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:30.880 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:38:30.880 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:38:30.880 DEBUG:boto:Method: GET 2023-09-22T21:38:30.880 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:38:30.880 DEBUG:boto:Data: 2023-09-22T21:38:30.880 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.880 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.880 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.881 DEBUG:boto:Params: {} 2023-09-22T21:38:30.881 DEBUG:boto:Token: None 2023-09-22T21:38:30.881 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/ 2023-09-22T21:38:30.881 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y= 2023-09-22T21:38:30.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y=', 'Content-Length': '0'} 2023-09-22T21:38:30.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7f1894f45192ff1-00650e0956-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.886 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:30.886 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:30.886 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:30.886 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:38:30.887 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:38:30.887 DEBUG:boto:Method: GET 2023-09-22T21:38:30.887 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:38:30.887 DEBUG:boto:Data: 2023-09-22T21:38:30.887 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.887 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.887 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.887 DEBUG:boto:Params: {} 2023-09-22T21:38:30.887 DEBUG:boto:Token: None 2023-09-22T21:38:30.887 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/ 2023-09-22T21:38:30.888 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y= 2023-09-22T21:38:30.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J4zfoNasxZNftD4Y8EeXQ07O74Y=', 'Content-Length': '0'} 2023-09-22T21:38:30.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005937c98bd6444894-00650e0956-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.934 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:30.935 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:38:30.935 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:38:30.935 DEBUG:boto:Method: HEAD 2023-09-22T21:38:30.935 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:38:30.935 DEBUG:boto:Data: 2023-09-22T21:38:30.935 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.935 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.935 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.936 DEBUG:boto:Params: {} 2023-09-22T21:38:30.936 DEBUG:boto:Token: None 2023-09-22T21:38:30.936 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj55 2023-09-22T21:38:30.936 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2TfHlIRjp/ENWlAEyn9pcIAdIUw= 2023-09-22T21:38:30.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2TfHlIRjp/ENWlAEyn9pcIAdIUw=', 'Content-Length': '0'} 2023-09-22T21:38:30.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9a8c5d568082b6e-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.939 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:38:30.939 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:38:30.939 DEBUG:boto:Method: HEAD 2023-09-22T21:38:30.939 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:38:30.940 DEBUG:boto:Data: 2023-09-22T21:38:30.940 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.940 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.940 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.940 DEBUG:boto:Params: {} 2023-09-22T21:38:30.940 DEBUG:boto:Token: None 2023-09-22T21:38:30.940 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj55 2023-09-22T21:38:30.940 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2TfHlIRjp/ENWlAEyn9pcIAdIUw= 2023-09-22T21:38:30.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2TfHlIRjp/ENWlAEyn9pcIAdIUw=', 'Content-Length': '0'} 2023-09-22T21:38:30.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffa9cd1fcb1083a1-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:38:30.944 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:38:30.944 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:38:30.944 DEBUG:boto:Method: GET 2023-09-22T21:38:30.944 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:38:30.944 DEBUG:boto:Data: 2023-09-22T21:38:30.944 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.945 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:30.945 DEBUG:boto:Port: 8001 2023-09-22T21:38:30.945 DEBUG:boto:Params: {} 2023-09-22T21:38:30.945 DEBUG:boto:Token: None 2023-09-22T21:38:30.945 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj55 2023-09-22T21:38:30.945 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LjsbBBSFok0QC2VPPIeYLK2YByE= 2023-09-22T21:38:30.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LjsbBBSFok0QC2VPPIeYLK2YByE=', 'Content-Length': '0'} 2023-09-22T21:38:30.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d04cd62187d87dc-00650e0956-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.994 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:38:30.995 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:38:30.995 DEBUG:boto:Method: GET 2023-09-22T21:38:30.995 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:38:30.995 DEBUG:boto:Data: 2023-09-22T21:38:30.995 DEBUG:boto:Headers: {} 2023-09-22T21:38:30.995 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:30.995 DEBUG:boto:Port: 8000 2023-09-22T21:38:30.995 DEBUG:boto:Params: {} 2023-09-22T21:38:30.995 DEBUG:boto:Token: None 2023-09-22T21:38:30.996 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:30 GMT /ffzdqu-59/obj55 2023-09-22T21:38:30.996 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LjsbBBSFok0QC2VPPIeYLK2YByE= 2023-09-22T21:38:30.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:30 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LjsbBBSFok0QC2VPPIeYLK2YByE=', 'Content-Length': '0'} 2023-09-22T21:38:30.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066f2514fc1a8e151-00650e0956-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:30.999 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:30.999 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:30.999 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:38:30.999 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:38:31.000 DEBUG:boto:Method: GET 2023-09-22T21:38:31.000 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:38:31.000 DEBUG:boto:Data: 2023-09-22T21:38:31.000 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.000 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:31.000 DEBUG:boto:Port: 8001 2023-09-22T21:38:31.000 DEBUG:boto:Params: {} 2023-09-22T21:38:31.000 DEBUG:boto:Token: None 2023-09-22T21:38:31.000 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/ 2023-09-22T21:38:31.001 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RqDFkadEor8euASt69C7AKkCNvs= 2023-09-22T21:38:31.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RqDFkadEor8euASt69C7AKkCNvs=', 'Content-Length': '0'} 2023-09-22T21:38:31.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029979ed81c1d2065-00650e0957-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.005 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:31.005 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:31.006 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:31.006 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:38:31.006 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:38:31.006 DEBUG:boto:Method: GET 2023-09-22T21:38:31.006 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:38:31.006 DEBUG:boto:Data: 2023-09-22T21:38:31.006 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.006 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.006 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.007 DEBUG:boto:Params: {} 2023-09-22T21:38:31.007 DEBUG:boto:Token: None 2023-09-22T21:38:31.007 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/ 2023-09-22T21:38:31.007 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RqDFkadEor8euASt69C7AKkCNvs= 2023-09-22T21:38:31.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RqDFkadEor8euASt69C7AKkCNvs=', 'Content-Length': '0'} 2023-09-22T21:38:31.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ea8179c60ac49d5-00650e0957-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.013 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:31.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:38:31.014 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:38:31.014 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:38:31.014 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.014 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.014 DEBUG:boto:Method: GET 2023-09-22T21:38:31.014 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.014 DEBUG:boto:Data: 2023-09-22T21:38:31.015 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.015 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:31.015 DEBUG:boto:Port: 8001 2023-09-22T21:38:31.015 DEBUG:boto:Params: {} 2023-09-22T21:38:31.015 DEBUG:boto:Token: None 2023-09-22T21:38:31.015 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.015 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0qSKnrP1e07AjcbaKjJ5+ru9wjk= 2023-09-22T21:38:31.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0qSKnrP1e07AjcbaKjJ5+ru9wjk=', 'Content-Length': '0'} 2023-09-22T21:38:31.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000476ef38cecb4dda0-00650e0957-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.019 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:38:31.019 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:38:31.019 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.019 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.019 DEBUG:boto:Method: GET 2023-09-22T21:38:31.019 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.019 DEBUG:boto:Data: 2023-09-22T21:38:31.019 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.019 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.019 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.020 DEBUG:boto:Params: {} 2023-09-22T21:38:31.020 DEBUG:boto:Token: None 2023-09-22T21:38:31.020 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.020 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0qSKnrP1e07AjcbaKjJ5+ru9wjk= 2023-09-22T21:38:31.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0qSKnrP1e07AjcbaKjJ5+ru9wjk=', 'Content-Length': '0'} 2023-09-22T21:38:31.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e727fd5a4843936-00650e0957-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.066 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:38:31.066 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:38:31.067 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.067 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.067 DEBUG:boto:Method: HEAD 2023-09-22T21:38:31.067 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.067 DEBUG:boto:Data: 2023-09-22T21:38:31.067 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.067 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:31.067 DEBUG:boto:Port: 8001 2023-09-22T21:38:31.067 DEBUG:boto:Params: {} 2023-09-22T21:38:31.068 DEBUG:boto:Token: None 2023-09-22T21:38:31.068 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.068 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kCgkEGVT4AUR+fSbAn0Dj/Fv5rM= 2023-09-22T21:38:31.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kCgkEGVT4AUR+fSbAn0Dj/Fv5rM=', 'Content-Length': '0'} 2023-09-22T21:38:31.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009aceb4df0650cfcb-00650e0957-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.071 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:38:31.071 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:38:31.071 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.071 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.071 DEBUG:boto:Method: HEAD 2023-09-22T21:38:31.071 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.071 DEBUG:boto:Data: 2023-09-22T21:38:31.072 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.072 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.072 DEBUG:boto:Params: {} 2023-09-22T21:38:31.072 DEBUG:boto:Token: None 2023-09-22T21:38:31.072 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.072 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kCgkEGVT4AUR+fSbAn0Dj/Fv5rM= 2023-09-22T21:38:31.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kCgkEGVT4AUR+fSbAn0Dj/Fv5rM=', 'Content-Length': '0'} 2023-09-22T21:38:31.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f349574bdae2eba7-00650e0957-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:38:31.075 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:38:31.075 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:38:31.076 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.076 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.076 DEBUG:boto:Method: GET 2023-09-22T21:38:31.076 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.076 DEBUG:boto:Data: 2023-09-22T21:38:31.076 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.076 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:31.076 DEBUG:boto:Port: 8001 2023-09-22T21:38:31.076 DEBUG:boto:Params: {} 2023-09-22T21:38:31.077 DEBUG:boto:Token: None 2023-09-22T21:38:31.077 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.077 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0qSKnrP1e07AjcbaKjJ5+ru9wjk= 2023-09-22T21:38:31.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0qSKnrP1e07AjcbaKjJ5+ru9wjk=', 'Content-Length': '0'} 2023-09-22T21:38:31.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf0fba56c6e93351-00650e0957-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.126 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:38:31.127 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:38:31.127 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.127 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.127 DEBUG:boto:Method: GET 2023-09-22T21:38:31.127 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.127 DEBUG:boto:Data: 2023-09-22T21:38:31.127 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.127 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.127 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.128 DEBUG:boto:Params: {} 2023-09-22T21:38:31.128 DEBUG:boto:Token: None 2023-09-22T21:38:31.128 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:38:31.128 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0qSKnrP1e07AjcbaKjJ5+ru9wjk= 2023-09-22T21:38:31.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0qSKnrP1e07AjcbaKjJ5+ru9wjk=', 'Content-Length': '0'} 2023-09-22T21:38:31.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ce55c7f58a2acb4-00650e0957-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.131 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:31.132 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:31.132 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:38:31.132 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:38:31.132 DEBUG:boto:Method: GET 2023-09-22T21:38:31.132 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:38:31.132 DEBUG:boto:Data: 2023-09-22T21:38:31.132 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.132 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:31.132 DEBUG:boto:Port: 8001 2023-09-22T21:38:31.132 DEBUG:boto:Params: {} 2023-09-22T21:38:31.133 DEBUG:boto:Token: None 2023-09-22T21:38:31.133 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/ 2023-09-22T21:38:31.133 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RqDFkadEor8euASt69C7AKkCNvs= 2023-09-22T21:38:31.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RqDFkadEor8euASt69C7AKkCNvs=', 'Content-Length': '0'} 2023-09-22T21:38:31.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4ed4308d5e61470-00650e0957-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.138 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:31.138 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:31.138 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:31.138 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:38:31.138 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:38:31.139 DEBUG:boto:Method: GET 2023-09-22T21:38:31.139 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:38:31.139 DEBUG:boto:Data: 2023-09-22T21:38:31.139 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.139 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.139 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.139 DEBUG:boto:Params: {} 2023-09-22T21:38:31.139 DEBUG:boto:Token: None 2023-09-22T21:38:31.140 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/ 2023-09-22T21:38:31.140 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RqDFkadEor8euASt69C7AKkCNvs= 2023-09-22T21:38:31.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RqDFkadEor8euASt69C7AKkCNvs=', 'Content-Length': '0'} 2023-09-22T21:38:31.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098da17c9c939a6b8-00650e0957-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.147 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:31.147 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:38:31.147 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:38:31.148 DEBUG:boto:Method: HEAD 2023-09-22T21:38:31.148 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:38:31.148 DEBUG:boto:Data: 2023-09-22T21:38:31.148 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.148 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:31.148 DEBUG:boto:Port: 8001 2023-09-22T21:38:31.148 DEBUG:boto:Params: {} 2023-09-22T21:38:31.148 DEBUG:boto:Token: None 2023-09-22T21:38:31.148 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj68 2023-09-22T21:38:31.149 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/rQaZt2CzCSOlUmp52xOLivfr9o= 2023-09-22T21:38:31.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/rQaZt2CzCSOlUmp52xOLivfr9o=', 'Content-Length': '0'} 2023-09-22T21:38:31.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007860fcfc6d801a35-00650e0957-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.151 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:38:31.152 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:38:31.152 DEBUG:boto:Method: HEAD 2023-09-22T21:38:31.152 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:38:31.152 DEBUG:boto:Data: 2023-09-22T21:38:31.152 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.152 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.152 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.152 DEBUG:boto:Params: {} 2023-09-22T21:38:31.153 DEBUG:boto:Token: None 2023-09-22T21:38:31.153 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj68 2023-09-22T21:38:31.153 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/rQaZt2CzCSOlUmp52xOLivfr9o= 2023-09-22T21:38:31.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/rQaZt2CzCSOlUmp52xOLivfr9o=', 'Content-Length': '0'} 2023-09-22T21:38:31.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a21d1665bee11d50-00650e0957-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:38:31.156 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:38:31.156 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:38:31.156 DEBUG:boto:Method: GET 2023-09-22T21:38:31.156 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:38:31.156 DEBUG:boto:Data: 2023-09-22T21:38:31.156 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.156 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:31.156 DEBUG:boto:Port: 8001 2023-09-22T21:38:31.157 DEBUG:boto:Params: {} 2023-09-22T21:38:31.157 DEBUG:boto:Token: None 2023-09-22T21:38:31.157 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj68 2023-09-22T21:38:31.157 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:io1Aa65SVlICQSw8oUzMWi1FHEg= 2023-09-22T21:38:31.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:io1Aa65SVlICQSw8oUzMWi1FHEg=', 'Content-Length': '0'} 2023-09-22T21:38:31.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007672edf60cff6cd9-00650e0957-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.206 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:38:31.207 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:38:31.207 DEBUG:boto:Method: GET 2023-09-22T21:38:31.207 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:38:31.207 DEBUG:boto:Data: 2023-09-22T21:38:31.207 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.207 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.207 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.207 DEBUG:boto:Params: {} 2023-09-22T21:38:31.207 DEBUG:boto:Token: None 2023-09-22T21:38:31.208 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj68 2023-09-22T21:38:31.208 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:io1Aa65SVlICQSw8oUzMWi1FHEg= 2023-09-22T21:38:31.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:io1Aa65SVlICQSw8oUzMWi1FHEg=', 'Content-Length': '0'} 2023-09-22T21:38:31.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042b7899db47aab0e-00650e0957-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.211 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:31.211 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:31.211 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:38:31.211 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:38:31.211 DEBUG:boto:Method: GET 2023-09-22T21:38:31.212 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:38:31.212 DEBUG:boto:Data: 2023-09-22T21:38:31.212 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.212 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:31.212 DEBUG:boto:Port: 8001 2023-09-22T21:38:31.212 DEBUG:boto:Params: {} 2023-09-22T21:38:31.212 DEBUG:boto:Token: None 2023-09-22T21:38:31.212 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/ 2023-09-22T21:38:31.212 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RqDFkadEor8euASt69C7AKkCNvs= 2023-09-22T21:38:31.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RqDFkadEor8euASt69C7AKkCNvs=', 'Content-Length': '0'} 2023-09-22T21:38:31.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039027449bc38392a-00650e0957-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.217 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:31.218 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:31.218 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:31.218 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:38:31.218 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:38:31.218 DEBUG:boto:Method: GET 2023-09-22T21:38:31.218 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:38:31.218 DEBUG:boto:Data: 2023-09-22T21:38:31.218 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.219 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.219 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.219 DEBUG:boto:Params: {} 2023-09-22T21:38:31.219 DEBUG:boto:Token: None 2023-09-22T21:38:31.219 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/ 2023-09-22T21:38:31.219 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RqDFkadEor8euASt69C7AKkCNvs= 2023-09-22T21:38:31.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RqDFkadEor8euASt69C7AKkCNvs=', 'Content-Length': '0'} 2023-09-22T21:38:31.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000885d6ca8b99df99f-00650e0957-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.226 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:31.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:38:31.227 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:38:31.227 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:38:31.227 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.227 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.227 DEBUG:boto:Method: GET 2023-09-22T21:38:31.227 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.227 DEBUG:boto:Data: 2023-09-22T21:38:31.227 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.228 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:31.228 DEBUG:boto:Port: 8001 2023-09-22T21:38:31.228 DEBUG:boto:Params: {} 2023-09-22T21:38:31.228 DEBUG:boto:Token: None 2023-09-22T21:38:31.228 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.228 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dt6K2duBg93Twb7eUg8+8x9wlBI= 2023-09-22T21:38:31.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dt6K2duBg93Twb7eUg8+8x9wlBI=', 'Content-Length': '0'} 2023-09-22T21:38:31.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002165c74e3c692c19-00650e0957-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.232 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:38:31.232 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:38:31.232 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.232 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.232 DEBUG:boto:Method: GET 2023-09-22T21:38:31.232 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.232 DEBUG:boto:Data: 2023-09-22T21:38:31.233 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.233 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.233 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.233 DEBUG:boto:Params: {} 2023-09-22T21:38:31.233 DEBUG:boto:Token: None 2023-09-22T21:38:31.233 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.233 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dt6K2duBg93Twb7eUg8+8x9wlBI= 2023-09-22T21:38:31.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dt6K2duBg93Twb7eUg8+8x9wlBI=', 'Content-Length': '0'} 2023-09-22T21:38:31.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006bea218000d6b81-00650e0957-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.236 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:38:31.237 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:38:31.237 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.237 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.237 DEBUG:boto:Method: HEAD 2023-09-22T21:38:31.237 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.237 DEBUG:boto:Data: 2023-09-22T21:38:31.237 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.237 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:31.238 DEBUG:boto:Port: 8001 2023-09-22T21:38:31.238 DEBUG:boto:Params: {} 2023-09-22T21:38:31.238 DEBUG:boto:Token: None 2023-09-22T21:38:31.238 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.238 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jp8/x6+q+awN2YT/IDcbQb6E1wc= 2023-09-22T21:38:31.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jp8/x6+q+awN2YT/IDcbQb6E1wc=', 'Content-Length': '0'} 2023-09-22T21:38:31.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c87473a188df5bc-00650e0957-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.241 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:38:31.241 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:38:31.241 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.242 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.242 DEBUG:boto:Method: HEAD 2023-09-22T21:38:31.242 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.242 DEBUG:boto:Data: 2023-09-22T21:38:31.242 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.242 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.242 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.242 DEBUG:boto:Params: {} 2023-09-22T21:38:31.242 DEBUG:boto:Token: None 2023-09-22T21:38:31.243 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.243 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jp8/x6+q+awN2YT/IDcbQb6E1wc= 2023-09-22T21:38:31.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jp8/x6+q+awN2YT/IDcbQb6E1wc=', 'Content-Length': '0'} 2023-09-22T21:38:31.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003e1f9b042178281-00650e0957-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:38:31.246 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:38:31.246 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:38:31.246 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.247 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.247 DEBUG:boto:Method: GET 2023-09-22T21:38:31.247 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.247 DEBUG:boto:Data: 2023-09-22T21:38:31.247 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.247 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:31.247 DEBUG:boto:Port: 8001 2023-09-22T21:38:31.247 DEBUG:boto:Params: {} 2023-09-22T21:38:31.247 DEBUG:boto:Token: None 2023-09-22T21:38:31.247 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.248 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dt6K2duBg93Twb7eUg8+8x9wlBI= 2023-09-22T21:38:31.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dt6K2duBg93Twb7eUg8+8x9wlBI=', 'Content-Length': '0'} 2023-09-22T21:38:31.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e0e703ef39f0c77-00650e0957-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.294 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:38:31.294 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:38:31.295 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.295 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.295 DEBUG:boto:Method: GET 2023-09-22T21:38:31.295 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.295 DEBUG:boto:Data: 2023-09-22T21:38:31.295 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.295 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.295 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.295 DEBUG:boto:Params: {} 2023-09-22T21:38:31.296 DEBUG:boto:Token: None 2023-09-22T21:38:31.296 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:38:31.296 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dt6K2duBg93Twb7eUg8+8x9wlBI= 2023-09-22T21:38:31.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dt6K2duBg93Twb7eUg8+8x9wlBI=', 'Content-Length': '0'} 2023-09-22T21:38:31.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c796cca1dc512865-00650e0957-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.299 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:31.299 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:31.299 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:38:31.299 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:38:31.299 DEBUG:boto:Method: GET 2023-09-22T21:38:31.300 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:38:31.300 DEBUG:boto:Data: 2023-09-22T21:38:31.300 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.300 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:31.300 DEBUG:boto:Port: 8001 2023-09-22T21:38:31.300 DEBUG:boto:Params: {} 2023-09-22T21:38:31.300 DEBUG:boto:Token: None 2023-09-22T21:38:31.300 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/ 2023-09-22T21:38:31.300 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RqDFkadEor8euASt69C7AKkCNvs= 2023-09-22T21:38:31.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RqDFkadEor8euASt69C7AKkCNvs=', 'Content-Length': '0'} 2023-09-22T21:38:31.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a140b327b863b282-00650e0957-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.306 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:31.307 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:31.307 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:31.307 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:38:31.307 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:38:31.307 DEBUG:boto:Method: GET 2023-09-22T21:38:31.308 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:38:31.308 DEBUG:boto:Data: 2023-09-22T21:38:31.308 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.308 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.308 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.308 DEBUG:boto:Params: {} 2023-09-22T21:38:31.308 DEBUG:boto:Token: None 2023-09-22T21:38:31.308 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/ 2023-09-22T21:38:31.308 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RqDFkadEor8euASt69C7AKkCNvs= 2023-09-22T21:38:31.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RqDFkadEor8euASt69C7AKkCNvs=', 'Content-Length': '0'} 2023-09-22T21:38:31.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6647c2e449acb48-00650e0957-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.316 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:31.316 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:38:31.317 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:38:31.317 DEBUG:boto:Method: HEAD 2023-09-22T21:38:31.317 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:38:31.317 DEBUG:boto:Data: 2023-09-22T21:38:31.317 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.317 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:31.317 DEBUG:boto:Port: 8001 2023-09-22T21:38:31.317 DEBUG:boto:Params: {} 2023-09-22T21:38:31.317 DEBUG:boto:Token: None 2023-09-22T21:38:31.318 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj85 2023-09-22T21:38:31.318 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jzFXv0alZ1VNFDkquh9/Unvq/qI= 2023-09-22T21:38:31.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jzFXv0alZ1VNFDkquh9/Unvq/qI=', 'Content-Length': '0'} 2023-09-22T21:38:31.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066547d6b509bb322-00650e0957-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.321 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:38:31.321 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:38:31.321 DEBUG:boto:Method: HEAD 2023-09-22T21:38:31.321 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:38:31.321 DEBUG:boto:Data: 2023-09-22T21:38:31.321 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.321 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.321 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.321 DEBUG:boto:Params: {} 2023-09-22T21:38:31.322 DEBUG:boto:Token: None 2023-09-22T21:38:31.322 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj85 2023-09-22T21:38:31.322 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jzFXv0alZ1VNFDkquh9/Unvq/qI= 2023-09-22T21:38:31.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jzFXv0alZ1VNFDkquh9/Unvq/qI=', 'Content-Length': '0'} 2023-09-22T21:38:31.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9ee00d7093e15a5-00650e0957-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:38:31.325 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:38:31.325 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:38:31.325 DEBUG:boto:Method: GET 2023-09-22T21:38:31.325 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:38:31.325 DEBUG:boto:Data: 2023-09-22T21:38:31.326 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.326 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:31.326 DEBUG:boto:Port: 8001 2023-09-22T21:38:31.326 DEBUG:boto:Params: {} 2023-09-22T21:38:31.326 DEBUG:boto:Token: None 2023-09-22T21:38:31.326 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj85 2023-09-22T21:38:31.326 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0xloEeXTmeVad7cw85tLcjxo/Bw= 2023-09-22T21:38:31.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0xloEeXTmeVad7cw85tLcjxo/Bw=', 'Content-Length': '0'} 2023-09-22T21:38:31.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000582dcc746cde1838-00650e0957-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.370 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:38:31.370 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:38:31.371 DEBUG:boto:Method: GET 2023-09-22T21:38:31.371 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:38:31.371 DEBUG:boto:Data: 2023-09-22T21:38:31.371 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.371 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.371 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.371 DEBUG:boto:Params: {} 2023-09-22T21:38:31.371 DEBUG:boto:Token: None 2023-09-22T21:38:31.372 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj85 2023-09-22T21:38:31.372 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0xloEeXTmeVad7cw85tLcjxo/Bw= 2023-09-22T21:38:31.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0xloEeXTmeVad7cw85tLcjxo/Bw=', 'Content-Length': '0'} 2023-09-22T21:38:31.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098d0b5fef9788100-00650e0957-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.375 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:31.375 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:31.375 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:38:31.375 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:38:31.375 DEBUG:boto:Method: GET 2023-09-22T21:38:31.376 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:38:31.376 DEBUG:boto:Data: 2023-09-22T21:38:31.376 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.376 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:38:31.376 DEBUG:boto:Port: 8001 2023-09-22T21:38:31.376 DEBUG:boto:Params: {} 2023-09-22T21:38:31.376 DEBUG:boto:Token: None 2023-09-22T21:38:31.376 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/ 2023-09-22T21:38:31.376 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RqDFkadEor8euASt69C7AKkCNvs= 2023-09-22T21:38:31.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RqDFkadEor8euASt69C7AKkCNvs=', 'Content-Length': '0'} 2023-09-22T21:38:31.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025df6b4ac67ccd2d-00650e0957-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.422 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:31.423 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:31.423 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:31.423 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:38:31.423 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:38:31.423 DEBUG:boto:Method: GET 2023-09-22T21:38:31.423 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:38:31.423 DEBUG:boto:Data: 2023-09-22T21:38:31.424 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.424 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.424 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.424 DEBUG:boto:Params: {} 2023-09-22T21:38:31.424 DEBUG:boto:Token: None 2023-09-22T21:38:31.424 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/ 2023-09-22T21:38:31.424 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RqDFkadEor8euASt69C7AKkCNvs= 2023-09-22T21:38:31.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RqDFkadEor8euASt69C7AKkCNvs=', 'Content-Length': '0'} 2023-09-22T21:38:31.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069db8b46f1bd4c59-00650e0957-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.431 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:38:31.432 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-59 zones={a2, a1} 2023-09-22T21:38:31.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:38:31.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:38:31.433 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:38:31.480 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:38:31.480 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:38:31.538 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:38:31.538 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-59 2023-09-22T21:38:31.548 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 10 2023-09-22T21:38:31.606 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:31.606 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:31.606 DEBUG:boto:Method: HEAD 2023-09-22T21:38:31.606 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:38:31.607 DEBUG:boto:Data: 2023-09-22T21:38:31.607 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.607 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.607 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.607 DEBUG:boto:Params: {} 2023-09-22T21:38:31.607 DEBUG:boto:Token: None 2023-09-22T21:38:31.607 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/ 2023-09-22T21:38:31.608 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SeJffqNK29avIiIMRVFb0VlEwpE= 2023-09-22T21:38:31.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SeJffqNK29avIiIMRVFb0VlEwpE=', 'Content-Length': '0'} 2023-09-22T21:38:31.610 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000eb9a131bf253d8f-00650e0957-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.611 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:38:31.611 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:38:31.611 DEBUG:boto:Method: PUT 2023-09-22T21:38:31.611 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:38:31.611 DEBUG:boto:Data: 2023-09-22T21:38:31.611 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:38:31.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.612 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.612 DEBUG:boto:Params: {} 2023-09-22T21:38:31.612 DEBUG:boto:Token: None 2023-09-22T21:38:31.612 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj19 2023-09-22T21:38:31.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uiYUoH3QzhtUH3hFQ/dVPBXMb8s= 2023-09-22T21:38:31.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uiYUoH3QzhtUH3hFQ/dVPBXMb8s='} 2023-09-22T21:38:31.658 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000225f2c63a33a65d0-00650e0957-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.659 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:31.659 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:31.659 DEBUG:boto:Method: HEAD 2023-09-22T21:38:31.659 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:38:31.659 DEBUG:boto:Data: 2023-09-22T21:38:31.659 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.659 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.659 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.660 DEBUG:boto:Params: {} 2023-09-22T21:38:31.660 DEBUG:boto:Token: None 2023-09-22T21:38:31.660 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/ 2023-09-22T21:38:31.660 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SeJffqNK29avIiIMRVFb0VlEwpE= 2023-09-22T21:38:31.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SeJffqNK29avIiIMRVFb0VlEwpE=', 'Content-Length': '0'} 2023-09-22T21:38:31.665 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d66b0cbfc27b5bed-00650e0957-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.666 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:38:31.666 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:38:31.666 DEBUG:boto:Method: PUT 2023-09-22T21:38:31.666 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:38:31.666 DEBUG:boto:Data: 2023-09-22T21:38:31.666 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:38:31.666 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.666 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.666 DEBUG:boto:Params: {} 2023-09-22T21:38:31.667 DEBUG:boto:Token: None 2023-09-22T21:38:31.667 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj38 2023-09-22T21:38:31.667 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4XmMD8Kg2jwlOq9gOH/GcntIikQ= 2023-09-22T21:38:31.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4XmMD8Kg2jwlOq9gOH/GcntIikQ='} 2023-09-22T21:38:31.675 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a093b7568186f467-00650e0957-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.675 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:31.675 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:31.675 DEBUG:boto:Method: HEAD 2023-09-22T21:38:31.675 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:38:31.675 DEBUG:boto:Data: 2023-09-22T21:38:31.676 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.676 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.676 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.676 DEBUG:boto:Params: {} 2023-09-22T21:38:31.676 DEBUG:boto:Token: None 2023-09-22T21:38:31.676 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/ 2023-09-22T21:38:31.676 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SeJffqNK29avIiIMRVFb0VlEwpE= 2023-09-22T21:38:31.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SeJffqNK29avIiIMRVFb0VlEwpE=', 'Content-Length': '0'} 2023-09-22T21:38:31.681 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000db2bc56ec0b45d9-00650e0957-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.681 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:38:31.681 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:38:31.682 DEBUG:boto:Method: PUT 2023-09-22T21:38:31.682 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:38:31.682 DEBUG:boto:Data: 2023-09-22T21:38:31.682 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:38:31.682 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.682 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.682 DEBUG:boto:Params: {} 2023-09-22T21:38:31.682 DEBUG:boto:Token: None 2023-09-22T21:38:31.683 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj57 2023-09-22T21:38:31.683 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:O6nG4vVl85WV3a0WddhRAvsw+g0= 2023-09-22T21:38:31.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:O6nG4vVl85WV3a0WddhRAvsw+g0='} 2023-09-22T21:38:31.726 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a013be996040149c-00650e0957-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.727 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:31.727 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:31.727 DEBUG:boto:Method: HEAD 2023-09-22T21:38:31.727 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:38:31.727 DEBUG:boto:Data: 2023-09-22T21:38:31.727 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.728 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.728 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.728 DEBUG:boto:Params: {} 2023-09-22T21:38:31.728 DEBUG:boto:Token: None 2023-09-22T21:38:31.728 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/ 2023-09-22T21:38:31.728 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SeJffqNK29avIiIMRVFb0VlEwpE= 2023-09-22T21:38:31.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SeJffqNK29avIiIMRVFb0VlEwpE=', 'Content-Length': '0'} 2023-09-22T21:38:31.734 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ab0d50df4247f8b-00650e0957-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.734 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:38:31.734 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:38:31.734 DEBUG:boto:Method: PUT 2023-09-22T21:38:31.734 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:38:31.734 DEBUG:boto:Data: 2023-09-22T21:38:31.735 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:38:31.735 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.735 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.735 DEBUG:boto:Params: {} 2023-09-22T21:38:31.735 DEBUG:boto:Token: None 2023-09-22T21:38:31.735 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj76 2023-09-22T21:38:31.735 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4AAPN7U7N3blVN62Vqh1MqZHl0o= 2023-09-22T21:38:31.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4AAPN7U7N3blVN62Vqh1MqZHl0o='} 2023-09-22T21:38:31.747 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000df93a8512fd496d-00650e0957-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.748 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:38:31.748 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:38:31.748 DEBUG:boto:Method: HEAD 2023-09-22T21:38:31.748 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:38:31.748 DEBUG:boto:Data: 2023-09-22T21:38:31.748 DEBUG:boto:Headers: {} 2023-09-22T21:38:31.748 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.748 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.748 DEBUG:boto:Params: {} 2023-09-22T21:38:31.749 DEBUG:boto:Token: None 2023-09-22T21:38:31.749 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/ 2023-09-22T21:38:31.749 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SeJffqNK29avIiIMRVFb0VlEwpE= 2023-09-22T21:38:31.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SeJffqNK29avIiIMRVFb0VlEwpE=', 'Content-Length': '0'} 2023-09-22T21:38:31.754 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af22abb05de3b35a-00650e0957-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.754 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:38:31.754 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:38:31.754 DEBUG:boto:Method: PUT 2023-09-22T21:38:31.754 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:38:31.755 DEBUG:boto:Data: 2023-09-22T21:38:31.755 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:38:31.755 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:38:31.755 DEBUG:boto:Port: 8000 2023-09-22T21:38:31.755 DEBUG:boto:Params: {} 2023-09-22T21:38:31.755 DEBUG:boto:Token: None 2023-09-22T21:38:31.755 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:38:31 GMT /ffzdqu-59/obj95 2023-09-22T21:38:31.755 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gSou2idsxDHQ6ts5mJRaldKUUlE= 2023-09-22T21:38:31.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:38:31 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gSou2idsxDHQ6ts5mJRaldKUUlE='} 2023-09-22T21:38:31.798 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ea8fbbc65aa69d9d-00650e0957-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:38:31.799 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-59 2023-09-22T21:38:31.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:38:31.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:38:31.799 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:38:31.850 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:38:31.851 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:38:31.896 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:38:31.893+0000 7f8a46534a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:38:36.897 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:38:36.897+0000 7f8a46534a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:38:41.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:38:41.897+0000 7f8a46534a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:38:46.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:38:46.897+0000 7f8a46534a40 1 waiting to reach latest gen 2, on 1.. 2023-09-22T21:38:51.903 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:38:51.900+0000 7f8a46534a40 1 waiting for incremental sync to catch up: 2023-09-22T21:38:51.903 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-09-22T21:38:51.903 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.62.6, 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:38:56.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:38:56.904+0000 7f8a46534a40 1 waiting for incremental sync to catch up: 2023-09-22T21:38:56.908 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-09-22T21:38:56.908 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.62.6, 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:01.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:39:01.908+0000 7f8a46534a40 1 waiting for incremental sync to catch up: 2023-09-22T21:39:01.910 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-09-22T21:39:01.910 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.62.6, 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:06.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:39:06.912+0000 7f8a46534a40 1 waiting for incremental sync to catch up: 2023-09-22T21:39:06.913 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-09-22T21:39:06.913 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.62.6, 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:11.916 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:39:11.912+0000 7f8a46534a40 1 waiting for incremental sync to catch up: 2023-09-22T21:39:11.916 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:11.916 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.62.6, 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:16.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:39:16.916+0000 7f8a46534a40 1 waiting for incremental sync to catch up: 2023-09-22T21:39:17.055 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:17.055 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.62.6, 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:21.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:39:21.920+0000 7f8a46534a40 1 waiting for incremental sync to catch up: 2023-09-22T21:39:21.922 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:21.922 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.62.6, 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:26.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:39:26.924+0000 7f8a46534a40 1 waiting for incremental sync to catch up: 2023-09-22T21:39:26.926 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:26.926 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.62.6, 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:31.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:39:31.928+0000 7f8a46534a40 1 waiting for incremental sync to catch up: 2023-09-22T21:39:31.929 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:31.929 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.62.6, 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:36.931 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:39:36.928+0000 7f8a46534a40 1 waiting for incremental sync to catch up: 2023-09-22T21:39:36.931 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:36.931 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.62.6, 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:41.935 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:39:41.932+0000 7f8a46534a40 1 waiting for incremental sync to catch up: 2023-09-22T21:39:41.935 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:41.935 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.62.6, 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:46.938 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:39:46.936+0000 7f8a46534a40 1 waiting for incremental sync to catch up: 2023-09-22T21:39:46.938 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:46.938 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.62.6, 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:51.941 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:39:51.940+0000 7f8a46534a40 1 waiting for incremental sync to catch up: 2023-09-22T21:39:51.941 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:51.942 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.62.6, 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:56.944 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:39:56.939+0000 7f8a46534a40 1 waiting for incremental sync to catch up: 2023-09-22T21:39:56.944 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:39:56.944 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.62.6, 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:40:01.947 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:40:01.943+0000 7f8a46534a40 1 waiting for incremental sync to catch up: 2023-09-22T21:40:01.947 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:40:01.948 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.62.6, 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:40:06.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:40:06.947+0000 7f8a46534a40 1 waiting for incremental sync to catch up: 2023-09-22T21:40:06.950 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:40:06.950 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.62.6, 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:40:11.953 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:40:11.951+0000 7f8a46534a40 1 bucket sync caught up with source: 2023-09-22T21:40:11.953 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.62.6, 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:40:11.953 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.62.6, 00000000004.57.6, , , , , , , , , , , ] 2023-09-22T21:40:11.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:40:11.951+0000 7f8a46534a40 0 bucket checkpoint complete 2023-09-22T21:40:11.985 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-59 2023-09-22T21:40:11.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:40:11.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:40:11.986 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:40:12.034 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:40:12.035 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:40:12.079 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:40:12.075+0000 7f19aa11aa40 1 waiting for incremental sync to catch up: 2023-09-22T21:40:12.079 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.301.6, 00000000001.259.6, 00000000001.144.6, , 00000000001.385.6, , , 00000000002.137.6, 00000000004.158.6, 00000000003.304.6] 2023-09-22T21:40:12.080 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.301.6, 00000000001.259.6, 00000000001.144.6, 00000000001.270.6, 00000000001.385.6, , , 00000000002.137.6, 00000000004.158.6, 00000000003.304.6] 2023-09-22T21:40:17.082 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:40:17.079+0000 7f19aa11aa40 1 bucket sync caught up with source: 2023-09-22T21:40:17.082 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.301.6, 00000000001.259.6, 00000000001.144.6, 00000000001.270.6, 00000000001.385.6, , , 00000000002.137.6, 00000000004.158.6, 00000000003.304.6] 2023-09-22T21:40:17.082 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.301.6, 00000000001.259.6, 00000000001.144.6, 00000000001.270.6, 00000000001.385.6, , , 00000000002.137.6, 00000000004.158.6, 00000000003.304.6] 2023-09-22T21:40:17.082 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:40:17.079+0000 7f19aa11aa40 0 bucket checkpoint complete 2023-09-22T21:40:17.111 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-59 zones={a2, a1} 2023-09-22T21:40:17.111 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:17.112 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:17.112 DEBUG:boto:Method: HEAD 2023-09-22T21:40:17.112 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:40:17.112 DEBUG:boto:Data: 2023-09-22T21:40:17.112 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.112 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.112 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.112 DEBUG:boto:Params: {} 2023-09-22T21:40:17.113 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:40:17.113 DEBUG:boto:Token: None 2023-09-22T21:40:17.113 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/ 2023-09-22T21:40:17.113 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AEE07spU4R/H3bs6uNdmpquKidI= 2023-09-22T21:40:17.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AEE07spU4R/H3bs6uNdmpquKidI=', 'Content-Length': '0'} 2023-09-22T21:40:17.118 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8fc9b9efcaac805-00650e09c1-1b9c-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.119 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:17.119 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:17.119 DEBUG:boto:Method: HEAD 2023-09-22T21:40:17.119 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:40:17.119 DEBUG:boto:Data: 2023-09-22T21:40:17.119 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.119 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.119 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.119 DEBUG:boto:Params: {} 2023-09-22T21:40:17.120 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:40:17.120 DEBUG:boto:Token: None 2023-09-22T21:40:17.120 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/ 2023-09-22T21:40:17.120 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AEE07spU4R/H3bs6uNdmpquKidI= 2023-09-22T21:40:17.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AEE07spU4R/H3bs6uNdmpquKidI=', 'Content-Length': '0'} 2023-09-22T21:40:17.125 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e924ada9299d071-00650e09c1-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.125 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:40:17.126 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:17.126 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:17.126 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:40:17.126 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:40:17.126 DEBUG:boto:Method: GET 2023-09-22T21:40:17.126 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:40:17.126 DEBUG:boto:Data: 2023-09-22T21:40:17.126 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.127 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.127 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.127 DEBUG:boto:Params: {} 2023-09-22T21:40:17.127 DEBUG:boto:Token: None 2023-09-22T21:40:17.127 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/?versions 2023-09-22T21:40:17.127 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3ygxxJKZh3F6m64XsiU0IcG7Snc= 2023-09-22T21:40:17.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3ygxxJKZh3F6m64XsiU0IcG7Snc=', 'Content-Length': '0'} 2023-09-22T21:40:17.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a366124cf6e1660d-00650e09c1-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.131 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:17.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T21:40:17.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-22T21:40:17.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-22T21:40:17.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T21:40:17.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T21:40:17.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-22T21:40:17.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-22T21:40:17.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T21:40:17.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-22T21:40:17.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T21:40:17.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-22T21:40:17.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-22T21:40:17.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-22T21:40:17.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-22T21:40:17.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-22T21:40:17.134 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:40:17.134 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:17.134 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:17.134 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:40:17.134 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:40:17.134 DEBUG:boto:Method: GET 2023-09-22T21:40:17.135 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:40:17.135 DEBUG:boto:Data: 2023-09-22T21:40:17.135 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.135 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.135 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.135 DEBUG:boto:Params: {} 2023-09-22T21:40:17.135 DEBUG:boto:Token: None 2023-09-22T21:40:17.135 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/?versions 2023-09-22T21:40:17.135 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3ygxxJKZh3F6m64XsiU0IcG7Snc= 2023-09-22T21:40:17.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3ygxxJKZh3F6m64XsiU0IcG7Snc=', 'Content-Length': '0'} 2023-09-22T21:40:17.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd4aaa366cb2ed33-00650e09c1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.140 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:17.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T21:40:17.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-22T21:40:17.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-22T21:40:17.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T21:40:17.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T21:40:17.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-22T21:40:17.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-22T21:40:17.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T21:40:17.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-22T21:40:17.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T21:40:17.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-22T21:40:17.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-22T21:40:17.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-22T21:40:17.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-22T21:40:17.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-22T21:40:17.142 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:17.143 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:17.143 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:40:17.143 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:40:17.143 DEBUG:boto:Method: GET 2023-09-22T21:40:17.143 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:40:17.143 DEBUG:boto:Data: 2023-09-22T21:40:17.143 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.143 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.144 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.144 DEBUG:boto:Params: {} 2023-09-22T21:40:17.144 DEBUG:boto:Token: None 2023-09-22T21:40:17.144 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/?versions 2023-09-22T21:40:17.144 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3ygxxJKZh3F6m64XsiU0IcG7Snc= 2023-09-22T21:40:17.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3ygxxJKZh3F6m64XsiU0IcG7Snc=', 'Content-Length': '0'} 2023-09-22T21:40:17.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e19fb5d7f23b905b-00650e09c1-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.148 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:17.149 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:17.149 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:17.149 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:40:17.149 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:40:17.149 DEBUG:boto:Method: GET 2023-09-22T21:40:17.149 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:40:17.149 DEBUG:boto:Data: 2023-09-22T21:40:17.149 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.150 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.150 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.150 DEBUG:boto:Params: {} 2023-09-22T21:40:17.150 DEBUG:boto:Token: None 2023-09-22T21:40:17.150 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/?versions 2023-09-22T21:40:17.150 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3ygxxJKZh3F6m64XsiU0IcG7Snc= 2023-09-22T21:40:17.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3ygxxJKZh3F6m64XsiU0IcG7Snc=', 'Content-Length': '0'} 2023-09-22T21:40:17.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010f464857615b221-00650e09c1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.155 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:17.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:40:17.156 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:40:17.157 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:40:17.157 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.157 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.157 DEBUG:boto:Method: GET 2023-09-22T21:40:17.157 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.157 DEBUG:boto:Data: 2023-09-22T21:40:17.157 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.157 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.157 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.158 DEBUG:boto:Params: {} 2023-09-22T21:40:17.158 DEBUG:boto:Token: None 2023-09-22T21:40:17.158 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.158 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Abi0Yq2J8jfPowACgfwnbS6kkKI= 2023-09-22T21:40:17.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Abi0Yq2J8jfPowACgfwnbS6kkKI=', 'Content-Length': '0'} 2023-09-22T21:40:17.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5fa99ff57410a35-00650e09c1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.206 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:40:17.207 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:40:17.207 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.207 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.207 DEBUG:boto:Method: GET 2023-09-22T21:40:17.207 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.207 DEBUG:boto:Data: 2023-09-22T21:40:17.208 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.208 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.208 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.208 DEBUG:boto:Params: {} 2023-09-22T21:40:17.208 DEBUG:boto:Token: None 2023-09-22T21:40:17.209 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.209 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Abi0Yq2J8jfPowACgfwnbS6kkKI= 2023-09-22T21:40:17.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Abi0Yq2J8jfPowACgfwnbS6kkKI=', 'Content-Length': '0'} 2023-09-22T21:40:17.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098ecd9f968041706-00650e09c1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.212 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:40:17.213 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:40:17.213 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.213 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.213 DEBUG:boto:Method: HEAD 2023-09-22T21:40:17.213 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.214 DEBUG:boto:Data: 2023-09-22T21:40:17.214 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.214 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.214 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.214 DEBUG:boto:Params: {} 2023-09-22T21:40:17.215 DEBUG:boto:Token: None 2023-09-22T21:40:17.215 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.215 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Op/63XPbuQSKEX4cF2Uo6ZwdSTE= 2023-09-22T21:40:17.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Op/63XPbuQSKEX4cF2Uo6ZwdSTE=', 'Content-Length': '0'} 2023-09-22T21:40:17.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000682a4904bf981be3-00650e09c1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.219 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:40:17.219 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:40:17.219 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.219 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.220 DEBUG:boto:Method: HEAD 2023-09-22T21:40:17.220 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.220 DEBUG:boto:Data: 2023-09-22T21:40:17.220 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.221 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.221 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.221 DEBUG:boto:Params: {} 2023-09-22T21:40:17.221 DEBUG:boto:Token: None 2023-09-22T21:40:17.221 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.222 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Op/63XPbuQSKEX4cF2Uo6ZwdSTE= 2023-09-22T21:40:17.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Op/63XPbuQSKEX4cF2Uo6ZwdSTE=', 'Content-Length': '0'} 2023-09-22T21:40:17.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004333866ecab0715-00650e09c1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:40:17.225 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:40:17.225 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:40:17.225 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.225 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.225 DEBUG:boto:Method: GET 2023-09-22T21:40:17.225 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.226 DEBUG:boto:Data: 2023-09-22T21:40:17.226 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.226 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.226 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.226 DEBUG:boto:Params: {} 2023-09-22T21:40:17.226 DEBUG:boto:Token: None 2023-09-22T21:40:17.226 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.226 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Abi0Yq2J8jfPowACgfwnbS6kkKI= 2023-09-22T21:40:17.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Abi0Yq2J8jfPowACgfwnbS6kkKI=', 'Content-Length': '0'} 2023-09-22T21:40:17.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069f676c8c008a463-00650e09c1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.270 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:40:17.271 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:40:17.271 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.271 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.271 DEBUG:boto:Method: GET 2023-09-22T21:40:17.272 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.272 DEBUG:boto:Data: 2023-09-22T21:40:17.272 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.272 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.272 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.272 DEBUG:boto:Params: {} 2023-09-22T21:40:17.273 DEBUG:boto:Token: None 2023-09-22T21:40:17.273 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:40:17.273 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Abi0Yq2J8jfPowACgfwnbS6kkKI= 2023-09-22T21:40:17.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Abi0Yq2J8jfPowACgfwnbS6kkKI=', 'Content-Length': '0'} 2023-09-22T21:40:17.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a15ebb399f333dd5-00650e09c1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.277 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:17.277 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:17.277 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:40:17.278 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:40:17.278 DEBUG:boto:Method: GET 2023-09-22T21:40:17.278 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:40:17.278 DEBUG:boto:Data: 2023-09-22T21:40:17.278 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.279 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.279 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.279 DEBUG:boto:Params: {} 2023-09-22T21:40:17.279 DEBUG:boto:Token: None 2023-09-22T21:40:17.279 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/ 2023-09-22T21:40:17.280 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ= 2023-09-22T21:40:17.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ=', 'Content-Length': '0'} 2023-09-22T21:40:17.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c2fdd09a3f3680c-00650e09c1-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.285 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:17.286 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:17.286 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:17.286 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:40:17.286 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:40:17.287 DEBUG:boto:Method: GET 2023-09-22T21:40:17.287 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:40:17.287 DEBUG:boto:Data: 2023-09-22T21:40:17.287 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.287 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.287 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.288 DEBUG:boto:Params: {} 2023-09-22T21:40:17.288 DEBUG:boto:Token: None 2023-09-22T21:40:17.288 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/ 2023-09-22T21:40:17.288 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ= 2023-09-22T21:40:17.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ=', 'Content-Length': '0'} 2023-09-22T21:40:17.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d40c1ce97e75fae-00650e09c1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.294 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:17.295 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:40:17.295 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:40:17.295 DEBUG:boto:Method: HEAD 2023-09-22T21:40:17.296 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:40:17.296 DEBUG:boto:Data: 2023-09-22T21:40:17.296 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.296 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.296 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.297 DEBUG:boto:Params: {} 2023-09-22T21:40:17.297 DEBUG:boto:Token: None 2023-09-22T21:40:17.297 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj11 2023-09-22T21:40:17.297 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FrJjwhXqeAcs3pzJ+xUhKLj795U= 2023-09-22T21:40:17.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FrJjwhXqeAcs3pzJ+xUhKLj795U=', 'Content-Length': '0'} 2023-09-22T21:40:17.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb64d3dae391af6c-00650e09c1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.300 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:40:17.301 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:40:17.301 DEBUG:boto:Method: HEAD 2023-09-22T21:40:17.301 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:40:17.301 DEBUG:boto:Data: 2023-09-22T21:40:17.301 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.301 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.301 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.301 DEBUG:boto:Params: {} 2023-09-22T21:40:17.302 DEBUG:boto:Token: None 2023-09-22T21:40:17.302 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj11 2023-09-22T21:40:17.302 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FrJjwhXqeAcs3pzJ+xUhKLj795U= 2023-09-22T21:40:17.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FrJjwhXqeAcs3pzJ+xUhKLj795U=', 'Content-Length': '0'} 2023-09-22T21:40:17.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a08034d8002ef6f-00650e09c1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:40:17.306 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:40:17.306 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:40:17.306 DEBUG:boto:Method: GET 2023-09-22T21:40:17.306 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:40:17.306 DEBUG:boto:Data: 2023-09-22T21:40:17.306 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.306 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.306 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.307 DEBUG:boto:Params: {} 2023-09-22T21:40:17.307 DEBUG:boto:Token: None 2023-09-22T21:40:17.307 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj11 2023-09-22T21:40:17.307 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dM6lzcfN3f8nb3iLo94psW/5NcA= 2023-09-22T21:40:17.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dM6lzcfN3f8nb3iLo94psW/5NcA=', 'Content-Length': '0'} 2023-09-22T21:40:17.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5e79d427a703aae-00650e09c1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.354 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:40:17.354 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:40:17.355 DEBUG:boto:Method: GET 2023-09-22T21:40:17.355 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:40:17.355 DEBUG:boto:Data: 2023-09-22T21:40:17.355 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.355 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.355 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.355 DEBUG:boto:Params: {} 2023-09-22T21:40:17.355 DEBUG:boto:Token: None 2023-09-22T21:40:17.356 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj11 2023-09-22T21:40:17.356 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dM6lzcfN3f8nb3iLo94psW/5NcA= 2023-09-22T21:40:17.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dM6lzcfN3f8nb3iLo94psW/5NcA=', 'Content-Length': '0'} 2023-09-22T21:40:17.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002807dae61749a4d7-00650e09c1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.359 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:17.359 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:17.359 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:40:17.359 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:40:17.359 DEBUG:boto:Method: GET 2023-09-22T21:40:17.359 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:40:17.359 DEBUG:boto:Data: 2023-09-22T21:40:17.360 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.360 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.360 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.360 DEBUG:boto:Params: {} 2023-09-22T21:40:17.360 DEBUG:boto:Token: None 2023-09-22T21:40:17.360 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/ 2023-09-22T21:40:17.360 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ= 2023-09-22T21:40:17.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ=', 'Content-Length': '0'} 2023-09-22T21:40:17.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063e4480bb0dc6487-00650e09c1-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.365 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:17.365 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:17.365 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:17.366 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:40:17.366 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:40:17.366 DEBUG:boto:Method: GET 2023-09-22T21:40:17.366 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:40:17.366 DEBUG:boto:Data: 2023-09-22T21:40:17.366 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.366 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.366 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.366 DEBUG:boto:Params: {} 2023-09-22T21:40:17.367 DEBUG:boto:Token: None 2023-09-22T21:40:17.367 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/ 2023-09-22T21:40:17.367 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ= 2023-09-22T21:40:17.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ=', 'Content-Length': '0'} 2023-09-22T21:40:17.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005820569485a8656a-00650e09c1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.414 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:17.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:40:17.415 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:40:17.415 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:40:17.415 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.415 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.415 DEBUG:boto:Method: GET 2023-09-22T21:40:17.416 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.416 DEBUG:boto:Data: 2023-09-22T21:40:17.416 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.416 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.416 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.416 DEBUG:boto:Params: {} 2023-09-22T21:40:17.416 DEBUG:boto:Token: None 2023-09-22T21:40:17.416 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.417 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ljqYgt/NjNj/DhehbUarUD+38Ng= 2023-09-22T21:40:17.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ljqYgt/NjNj/DhehbUarUD+38Ng=', 'Content-Length': '0'} 2023-09-22T21:40:17.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064d221db2c1289f6-00650e09c1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.420 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:40:17.420 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:40:17.420 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.420 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.420 DEBUG:boto:Method: GET 2023-09-22T21:40:17.420 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.420 DEBUG:boto:Data: 2023-09-22T21:40:17.421 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.421 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.421 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.421 DEBUG:boto:Params: {} 2023-09-22T21:40:17.421 DEBUG:boto:Token: None 2023-09-22T21:40:17.421 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.421 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ljqYgt/NjNj/DhehbUarUD+38Ng= 2023-09-22T21:40:17.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ljqYgt/NjNj/DhehbUarUD+38Ng=', 'Content-Length': '0'} 2023-09-22T21:40:17.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd0cad9a22830d44-00650e09c1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.424 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:40:17.424 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:40:17.425 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.425 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.425 DEBUG:boto:Method: HEAD 2023-09-22T21:40:17.425 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.425 DEBUG:boto:Data: 2023-09-22T21:40:17.425 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.425 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.425 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.425 DEBUG:boto:Params: {} 2023-09-22T21:40:17.425 DEBUG:boto:Token: None 2023-09-22T21:40:17.426 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.426 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uhnpArYkcxMyiv78Vl2WXAwJ+54= 2023-09-22T21:40:17.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uhnpArYkcxMyiv78Vl2WXAwJ+54=', 'Content-Length': '0'} 2023-09-22T21:40:17.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000740ddb51233d97b2-00650e09c1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.429 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:40:17.429 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:40:17.429 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.429 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.429 DEBUG:boto:Method: HEAD 2023-09-22T21:40:17.429 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.430 DEBUG:boto:Data: 2023-09-22T21:40:17.430 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.430 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.430 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.430 DEBUG:boto:Params: {} 2023-09-22T21:40:17.430 DEBUG:boto:Token: None 2023-09-22T21:40:17.430 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.430 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uhnpArYkcxMyiv78Vl2WXAwJ+54= 2023-09-22T21:40:17.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uhnpArYkcxMyiv78Vl2WXAwJ+54=', 'Content-Length': '0'} 2023-09-22T21:40:17.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eb4d3e7e8b76765-00650e09c1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:40:17.433 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:40:17.434 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:40:17.434 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.434 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.434 DEBUG:boto:Method: GET 2023-09-22T21:40:17.434 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.434 DEBUG:boto:Data: 2023-09-22T21:40:17.434 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.434 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.434 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.435 DEBUG:boto:Params: {} 2023-09-22T21:40:17.435 DEBUG:boto:Token: None 2023-09-22T21:40:17.435 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.435 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ljqYgt/NjNj/DhehbUarUD+38Ng= 2023-09-22T21:40:17.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ljqYgt/NjNj/DhehbUarUD+38Ng=', 'Content-Length': '0'} 2023-09-22T21:40:17.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f911202bc26da255-00650e09c1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.482 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:40:17.482 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:40:17.483 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.483 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.483 DEBUG:boto:Method: GET 2023-09-22T21:40:17.483 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.483 DEBUG:boto:Data: 2023-09-22T21:40:17.483 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.483 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.483 DEBUG:boto:Params: {} 2023-09-22T21:40:17.484 DEBUG:boto:Token: None 2023-09-22T21:40:17.484 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:40:17.484 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ljqYgt/NjNj/DhehbUarUD+38Ng= 2023-09-22T21:40:17.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ljqYgt/NjNj/DhehbUarUD+38Ng=', 'Content-Length': '0'} 2023-09-22T21:40:17.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032fdf8a3b3a6415c-00650e09c1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.487 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:17.487 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:17.487 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:40:17.488 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:40:17.488 DEBUG:boto:Method: GET 2023-09-22T21:40:17.488 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:40:17.488 DEBUG:boto:Data: 2023-09-22T21:40:17.488 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.488 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.488 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.488 DEBUG:boto:Params: {} 2023-09-22T21:40:17.488 DEBUG:boto:Token: None 2023-09-22T21:40:17.489 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/ 2023-09-22T21:40:17.489 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ= 2023-09-22T21:40:17.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ=', 'Content-Length': '0'} 2023-09-22T21:40:17.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9971dbf19eab57a-00650e09c1-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.493 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:17.494 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:17.494 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:17.494 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:40:17.495 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:40:17.495 DEBUG:boto:Method: GET 2023-09-22T21:40:17.495 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:40:17.495 DEBUG:boto:Data: 2023-09-22T21:40:17.495 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.495 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.495 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.495 DEBUG:boto:Params: {} 2023-09-22T21:40:17.495 DEBUG:boto:Token: None 2023-09-22T21:40:17.496 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/ 2023-09-22T21:40:17.496 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ= 2023-09-22T21:40:17.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ=', 'Content-Length': '0'} 2023-09-22T21:40:17.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0a207ae58c6f440-00650e09c1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.546 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:17.547 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:40:17.547 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:40:17.547 DEBUG:boto:Method: HEAD 2023-09-22T21:40:17.547 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:40:17.547 DEBUG:boto:Data: 2023-09-22T21:40:17.547 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.547 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.548 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.548 DEBUG:boto:Params: {} 2023-09-22T21:40:17.548 DEBUG:boto:Token: None 2023-09-22T21:40:17.549 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj17 2023-09-22T21:40:17.549 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8uQ/QgzsfaMRd5es/7Na8OorFeo= 2023-09-22T21:40:17.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8uQ/QgzsfaMRd5es/7Na8OorFeo=', 'Content-Length': '0'} 2023-09-22T21:40:17.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd07b3e8356015d-00650e09c1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.553 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:40:17.553 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:40:17.553 DEBUG:boto:Method: HEAD 2023-09-22T21:40:17.553 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:40:17.554 DEBUG:boto:Data: 2023-09-22T21:40:17.554 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.554 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.554 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.554 DEBUG:boto:Params: {} 2023-09-22T21:40:17.555 DEBUG:boto:Token: None 2023-09-22T21:40:17.555 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj17 2023-09-22T21:40:17.555 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8uQ/QgzsfaMRd5es/7Na8OorFeo= 2023-09-22T21:40:17.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8uQ/QgzsfaMRd5es/7Na8OorFeo=', 'Content-Length': '0'} 2023-09-22T21:40:17.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058175b3d6a72393b-00650e09c1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:40:17.558 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:40:17.559 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:40:17.559 DEBUG:boto:Method: GET 2023-09-22T21:40:17.559 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:40:17.559 DEBUG:boto:Data: 2023-09-22T21:40:17.559 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.559 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.559 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.560 DEBUG:boto:Params: {} 2023-09-22T21:40:17.560 DEBUG:boto:Token: None 2023-09-22T21:40:17.560 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj17 2023-09-22T21:40:17.560 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4q7EoFrxenG8xrZHSmz7LQBZE2k= 2023-09-22T21:40:17.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4q7EoFrxenG8xrZHSmz7LQBZE2k=', 'Content-Length': '0'} 2023-09-22T21:40:17.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e49135f9e09c0334-00650e09c1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.606 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:40:17.607 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:40:17.607 DEBUG:boto:Method: GET 2023-09-22T21:40:17.607 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:40:17.607 DEBUG:boto:Data: 2023-09-22T21:40:17.607 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.607 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.607 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.607 DEBUG:boto:Params: {} 2023-09-22T21:40:17.608 DEBUG:boto:Token: None 2023-09-22T21:40:17.608 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj17 2023-09-22T21:40:17.608 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4q7EoFrxenG8xrZHSmz7LQBZE2k= 2023-09-22T21:40:17.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4q7EoFrxenG8xrZHSmz7LQBZE2k=', 'Content-Length': '0'} 2023-09-22T21:40:17.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016535ee39da830cf-00650e09c1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.611 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:17.611 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:17.611 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:40:17.611 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:40:17.612 DEBUG:boto:Method: GET 2023-09-22T21:40:17.612 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:40:17.612 DEBUG:boto:Data: 2023-09-22T21:40:17.612 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.612 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.612 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.612 DEBUG:boto:Params: {} 2023-09-22T21:40:17.612 DEBUG:boto:Token: None 2023-09-22T21:40:17.612 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/ 2023-09-22T21:40:17.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ= 2023-09-22T21:40:17.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ=', 'Content-Length': '0'} 2023-09-22T21:40:17.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ab92b37daa68006-00650e09c1-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.658 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:17.659 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:17.659 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:17.659 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:40:17.659 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:40:17.659 DEBUG:boto:Method: GET 2023-09-22T21:40:17.659 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:40:17.659 DEBUG:boto:Data: 2023-09-22T21:40:17.659 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.660 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.660 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.660 DEBUG:boto:Params: {} 2023-09-22T21:40:17.660 DEBUG:boto:Token: None 2023-09-22T21:40:17.660 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/ 2023-09-22T21:40:17.660 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ= 2023-09-22T21:40:17.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ=', 'Content-Length': '0'} 2023-09-22T21:40:17.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dab0910011405ace-00650e09c1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.667 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:17.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T21:40:17.668 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:40:17.668 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:40:17.668 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.668 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.668 DEBUG:boto:Method: GET 2023-09-22T21:40:17.668 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.668 DEBUG:boto:Data: 2023-09-22T21:40:17.668 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.668 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.668 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.669 DEBUG:boto:Params: {} 2023-09-22T21:40:17.669 DEBUG:boto:Token: None 2023-09-22T21:40:17.669 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.669 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9qvp7RIExrbmRTrKDOMMOXIQwzA= 2023-09-22T21:40:17.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9qvp7RIExrbmRTrKDOMMOXIQwzA=', 'Content-Length': '0'} 2023-09-22T21:40:17.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4d7dc945c48d92d-00650e09c1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.672 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:40:17.673 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:40:17.673 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.673 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.673 DEBUG:boto:Method: GET 2023-09-22T21:40:17.673 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.673 DEBUG:boto:Data: 2023-09-22T21:40:17.673 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.673 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.673 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.673 DEBUG:boto:Params: {} 2023-09-22T21:40:17.674 DEBUG:boto:Token: None 2023-09-22T21:40:17.674 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.674 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9qvp7RIExrbmRTrKDOMMOXIQwzA= 2023-09-22T21:40:17.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9qvp7RIExrbmRTrKDOMMOXIQwzA=', 'Content-Length': '0'} 2023-09-22T21:40:17.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b96156386544a12c-00650e09c1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.677 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:40:17.677 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:40:17.677 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.677 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.677 DEBUG:boto:Method: HEAD 2023-09-22T21:40:17.677 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.678 DEBUG:boto:Data: 2023-09-22T21:40:17.678 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.678 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.678 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.678 DEBUG:boto:Params: {} 2023-09-22T21:40:17.678 DEBUG:boto:Token: None 2023-09-22T21:40:17.678 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.678 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t5ZdHP7kMOhA3g1ShUSQOnYocn0= 2023-09-22T21:40:17.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t5ZdHP7kMOhA3g1ShUSQOnYocn0=', 'Content-Length': '0'} 2023-09-22T21:40:17.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e49842e77e438a08-00650e09c1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.681 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:40:17.682 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:40:17.682 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.682 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.682 DEBUG:boto:Method: HEAD 2023-09-22T21:40:17.682 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.682 DEBUG:boto:Data: 2023-09-22T21:40:17.682 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.682 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.682 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.682 DEBUG:boto:Params: {} 2023-09-22T21:40:17.683 DEBUG:boto:Token: None 2023-09-22T21:40:17.683 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.683 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t5ZdHP7kMOhA3g1ShUSQOnYocn0= 2023-09-22T21:40:17.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t5ZdHP7kMOhA3g1ShUSQOnYocn0=', 'Content-Length': '0'} 2023-09-22T21:40:17.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083f3a4515185454d-00650e09c1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T21:40:17.686 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:40:17.686 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:40:17.686 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.686 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.686 DEBUG:boto:Method: GET 2023-09-22T21:40:17.686 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.686 DEBUG:boto:Data: 2023-09-22T21:40:17.686 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.687 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.687 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.687 DEBUG:boto:Params: {} 2023-09-22T21:40:17.687 DEBUG:boto:Token: None 2023-09-22T21:40:17.687 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.687 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9qvp7RIExrbmRTrKDOMMOXIQwzA= 2023-09-22T21:40:17.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9qvp7RIExrbmRTrKDOMMOXIQwzA=', 'Content-Length': '0'} 2023-09-22T21:40:17.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db38b1b061541fdf-00650e09c1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.730 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:40:17.730 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:40:17.730 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.731 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.731 DEBUG:boto:Method: GET 2023-09-22T21:40:17.731 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.731 DEBUG:boto:Data: 2023-09-22T21:40:17.731 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.731 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.731 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.731 DEBUG:boto:Params: {} 2023-09-22T21:40:17.731 DEBUG:boto:Token: None 2023-09-22T21:40:17.732 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:40:17.732 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9qvp7RIExrbmRTrKDOMMOXIQwzA= 2023-09-22T21:40:17.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9qvp7RIExrbmRTrKDOMMOXIQwzA=', 'Content-Length': '0'} 2023-09-22T21:40:17.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004aedb865ce14914a-00650e09c1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.735 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:17.735 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:17.735 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:40:17.735 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:40:17.735 DEBUG:boto:Method: GET 2023-09-22T21:40:17.736 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:40:17.736 DEBUG:boto:Data: 2023-09-22T21:40:17.736 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.736 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.736 DEBUG:boto:Params: {} 2023-09-22T21:40:17.736 DEBUG:boto:Token: None 2023-09-22T21:40:17.736 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/ 2023-09-22T21:40:17.736 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ= 2023-09-22T21:40:17.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ=', 'Content-Length': '0'} 2023-09-22T21:40:17.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000306dafc8a0a91e6d-00650e09c1-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.782 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:17.783 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:17.783 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:17.783 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:40:17.783 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:40:17.783 DEBUG:boto:Method: GET 2023-09-22T21:40:17.783 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:40:17.783 DEBUG:boto:Data: 2023-09-22T21:40:17.783 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.783 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.784 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.784 DEBUG:boto:Params: {} 2023-09-22T21:40:17.784 DEBUG:boto:Token: None 2023-09-22T21:40:17.784 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/ 2023-09-22T21:40:17.784 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ= 2023-09-22T21:40:17.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ=', 'Content-Length': '0'} 2023-09-22T21:40:17.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005eb6544f2dda753a-00650e09c1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.790 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:17.790 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:40:17.791 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:40:17.791 DEBUG:boto:Method: HEAD 2023-09-22T21:40:17.791 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:40:17.791 DEBUG:boto:Data: 2023-09-22T21:40:17.791 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.791 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.791 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.791 DEBUG:boto:Params: {} 2023-09-22T21:40:17.791 DEBUG:boto:Token: None 2023-09-22T21:40:17.791 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj19 2023-09-22T21:40:17.792 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tU6X8A8e0ahRzcdSILGqpCYdNIA= 2023-09-22T21:40:17.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tU6X8A8e0ahRzcdSILGqpCYdNIA=', 'Content-Length': '0'} 2023-09-22T21:40:17.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ae938ef1f8e6794-00650e09c1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.795 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:40:17.795 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:40:17.795 DEBUG:boto:Method: HEAD 2023-09-22T21:40:17.795 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:40:17.795 DEBUG:boto:Data: 2023-09-22T21:40:17.795 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.795 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.795 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.795 DEBUG:boto:Params: {} 2023-09-22T21:40:17.795 DEBUG:boto:Token: None 2023-09-22T21:40:17.796 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj19 2023-09-22T21:40:17.796 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tU6X8A8e0ahRzcdSILGqpCYdNIA= 2023-09-22T21:40:17.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tU6X8A8e0ahRzcdSILGqpCYdNIA=', 'Content-Length': '0'} 2023-09-22T21:40:17.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec01934286d35cf0-00650e09c1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T21:40:17.799 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:40:17.799 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:40:17.799 DEBUG:boto:Method: GET 2023-09-22T21:40:17.799 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:40:17.799 DEBUG:boto:Data: 2023-09-22T21:40:17.799 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.799 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.799 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.800 DEBUG:boto:Params: {} 2023-09-22T21:40:17.800 DEBUG:boto:Token: None 2023-09-22T21:40:17.800 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj19 2023-09-22T21:40:17.800 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BwJoefh2zeKgUGMTqlDXx9gAcJM= 2023-09-22T21:40:17.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BwJoefh2zeKgUGMTqlDXx9gAcJM=', 'Content-Length': '0'} 2023-09-22T21:40:17.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8b1ccc5df314ddf-00650e09c1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.846 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:40:17.846 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:40:17.847 DEBUG:boto:Method: GET 2023-09-22T21:40:17.847 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:40:17.847 DEBUG:boto:Data: 2023-09-22T21:40:17.847 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.847 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.847 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.847 DEBUG:boto:Params: {} 2023-09-22T21:40:17.847 DEBUG:boto:Token: None 2023-09-22T21:40:17.847 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj19 2023-09-22T21:40:17.848 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BwJoefh2zeKgUGMTqlDXx9gAcJM= 2023-09-22T21:40:17.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BwJoefh2zeKgUGMTqlDXx9gAcJM=', 'Content-Length': '0'} 2023-09-22T21:40:17.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb60e765406ea13b-00650e09c1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.851 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:17.851 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:17.851 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:40:17.852 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:40:17.852 DEBUG:boto:Method: GET 2023-09-22T21:40:17.852 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:40:17.852 DEBUG:boto:Data: 2023-09-22T21:40:17.852 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.852 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.852 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.852 DEBUG:boto:Params: {} 2023-09-22T21:40:17.852 DEBUG:boto:Token: None 2023-09-22T21:40:17.852 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/ 2023-09-22T21:40:17.853 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ= 2023-09-22T21:40:17.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ=', 'Content-Length': '0'} 2023-09-22T21:40:17.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066c5c56869313b5e-00650e09c1-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.902 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:17.903 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:17.903 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:17.903 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:40:17.903 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:40:17.903 DEBUG:boto:Method: GET 2023-09-22T21:40:17.903 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:40:17.903 DEBUG:boto:Data: 2023-09-22T21:40:17.904 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.904 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.904 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.904 DEBUG:boto:Params: {} 2023-09-22T21:40:17.904 DEBUG:boto:Token: None 2023-09-22T21:40:17.904 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/ 2023-09-22T21:40:17.904 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ= 2023-09-22T21:40:17.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ=', 'Content-Length': '0'} 2023-09-22T21:40:17.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076ffe86459e65cdb-00650e09c1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.911 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:17.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:40:17.912 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:40:17.912 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:40:17.912 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.912 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.912 DEBUG:boto:Method: GET 2023-09-22T21:40:17.912 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.912 DEBUG:boto:Data: 2023-09-22T21:40:17.912 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.912 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.913 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.913 DEBUG:boto:Params: {} 2023-09-22T21:40:17.913 DEBUG:boto:Token: None 2023-09-22T21:40:17.913 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.913 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gYAV4TInKJueZIm14wdZD7KGYSQ= 2023-09-22T21:40:17.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gYAV4TInKJueZIm14wdZD7KGYSQ=', 'Content-Length': '0'} 2023-09-22T21:40:17.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063e16e1bba839524-00650e09c1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.916 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:40:17.916 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:40:17.916 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.916 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.917 DEBUG:boto:Method: GET 2023-09-22T21:40:17.917 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.917 DEBUG:boto:Data: 2023-09-22T21:40:17.917 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.917 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.917 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.917 DEBUG:boto:Params: {} 2023-09-22T21:40:17.917 DEBUG:boto:Token: None 2023-09-22T21:40:17.917 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.918 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gYAV4TInKJueZIm14wdZD7KGYSQ= 2023-09-22T21:40:17.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gYAV4TInKJueZIm14wdZD7KGYSQ=', 'Content-Length': '0'} 2023-09-22T21:40:17.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006966ec3e22a5b847-00650e09c1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.921 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:40:17.921 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:40:17.921 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.921 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.921 DEBUG:boto:Method: HEAD 2023-09-22T21:40:17.921 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.921 DEBUG:boto:Data: 2023-09-22T21:40:17.921 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.922 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.922 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.922 DEBUG:boto:Params: {} 2023-09-22T21:40:17.922 DEBUG:boto:Token: None 2023-09-22T21:40:17.922 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.922 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sgOMZrEr8dqL7GSIjdxxCvl6fX8= 2023-09-22T21:40:17.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sgOMZrEr8dqL7GSIjdxxCvl6fX8=', 'Content-Length': '0'} 2023-09-22T21:40:17.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef4782c9c29cb251-00650e09c1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.925 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:40:17.925 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:40:17.926 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.926 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.926 DEBUG:boto:Method: HEAD 2023-09-22T21:40:17.926 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.926 DEBUG:boto:Data: 2023-09-22T21:40:17.926 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.926 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.926 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.926 DEBUG:boto:Params: {} 2023-09-22T21:40:17.926 DEBUG:boto:Token: None 2023-09-22T21:40:17.927 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.927 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sgOMZrEr8dqL7GSIjdxxCvl6fX8= 2023-09-22T21:40:17.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sgOMZrEr8dqL7GSIjdxxCvl6fX8=', 'Content-Length': '0'} 2023-09-22T21:40:17.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006309d785aa234aeb-00650e09c1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:40:17.930 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:40:17.930 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:40:17.930 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.930 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.931 DEBUG:boto:Method: GET 2023-09-22T21:40:17.931 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.931 DEBUG:boto:Data: 2023-09-22T21:40:17.931 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.931 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.931 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.931 DEBUG:boto:Params: {} 2023-09-22T21:40:17.931 DEBUG:boto:Token: None 2023-09-22T21:40:17.931 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.931 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gYAV4TInKJueZIm14wdZD7KGYSQ= 2023-09-22T21:40:17.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gYAV4TInKJueZIm14wdZD7KGYSQ=', 'Content-Length': '0'} 2023-09-22T21:40:17.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa23974884b7a72b-00650e09c1-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.982 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:40:17.982 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:40:17.983 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.983 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.983 DEBUG:boto:Method: GET 2023-09-22T21:40:17.983 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.983 DEBUG:boto:Data: 2023-09-22T21:40:17.983 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.983 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:17.983 DEBUG:boto:Port: 8000 2023-09-22T21:40:17.983 DEBUG:boto:Params: {} 2023-09-22T21:40:17.983 DEBUG:boto:Token: None 2023-09-22T21:40:17.984 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:40:17.984 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gYAV4TInKJueZIm14wdZD7KGYSQ= 2023-09-22T21:40:17.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gYAV4TInKJueZIm14wdZD7KGYSQ=', 'Content-Length': '0'} 2023-09-22T21:40:17.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022578257b4ff1a50-00650e09c1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:17.987 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:17.987 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:17.987 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:40:17.987 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:40:17.987 DEBUG:boto:Method: GET 2023-09-22T21:40:17.987 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:40:17.987 DEBUG:boto:Data: 2023-09-22T21:40:17.988 DEBUG:boto:Headers: {} 2023-09-22T21:40:17.988 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:17.988 DEBUG:boto:Port: 8001 2023-09-22T21:40:17.988 DEBUG:boto:Params: {} 2023-09-22T21:40:17.988 DEBUG:boto:Token: None 2023-09-22T21:40:17.988 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:17 GMT /ffzdqu-59/ 2023-09-22T21:40:17.988 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ= 2023-09-22T21:40:17.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tqb0MBGQLuQJqodixN5zCAohERQ=', 'Content-Length': '0'} 2023-09-22T21:40:17.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf17d4746e9dd860-00650e09c1-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.034 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:18.035 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:18.035 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:18.035 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:40:18.035 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:40:18.035 DEBUG:boto:Method: GET 2023-09-22T21:40:18.035 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:40:18.035 DEBUG:boto:Data: 2023-09-22T21:40:18.035 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.035 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.036 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.036 DEBUG:boto:Params: {} 2023-09-22T21:40:18.036 DEBUG:boto:Token: None 2023-09-22T21:40:18.036 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/ 2023-09-22T21:40:18.036 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U= 2023-09-22T21:40:18.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U=', 'Content-Length': '0'} 2023-09-22T21:40:18.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d51e123bfe09646f-00650e09c2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.041 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:18.042 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:40:18.042 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:40:18.042 DEBUG:boto:Method: HEAD 2023-09-22T21:40:18.042 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:40:18.042 DEBUG:boto:Data: 2023-09-22T21:40:18.042 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.042 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.043 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.043 DEBUG:boto:Params: {} 2023-09-22T21:40:18.043 DEBUG:boto:Token: None 2023-09-22T21:40:18.043 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj22 2023-09-22T21:40:18.043 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2IWBF1c9UPDWF153MWa8N1OY3N4= 2023-09-22T21:40:18.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2IWBF1c9UPDWF153MWa8N1OY3N4=', 'Content-Length': '0'} 2023-09-22T21:40:18.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027ad0b1784b2e749-00650e09c2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.047 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:40:18.047 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:40:18.047 DEBUG:boto:Method: HEAD 2023-09-22T21:40:18.047 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:40:18.047 DEBUG:boto:Data: 2023-09-22T21:40:18.047 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.047 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.048 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.048 DEBUG:boto:Params: {} 2023-09-22T21:40:18.048 DEBUG:boto:Token: None 2023-09-22T21:40:18.048 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj22 2023-09-22T21:40:18.048 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2IWBF1c9UPDWF153MWa8N1OY3N4= 2023-09-22T21:40:18.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2IWBF1c9UPDWF153MWa8N1OY3N4=', 'Content-Length': '0'} 2023-09-22T21:40:18.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005657ab3ad1f80f37-00650e09c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:40:18.051 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:40:18.051 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:40:18.052 DEBUG:boto:Method: GET 2023-09-22T21:40:18.052 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:40:18.052 DEBUG:boto:Data: 2023-09-22T21:40:18.052 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.052 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.052 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.052 DEBUG:boto:Params: {} 2023-09-22T21:40:18.052 DEBUG:boto:Token: None 2023-09-22T21:40:18.052 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj22 2023-09-22T21:40:18.052 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wXK9Amvj9ugGkNAR0jQv8GPOTlY= 2023-09-22T21:40:18.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wXK9Amvj9ugGkNAR0jQv8GPOTlY=', 'Content-Length': '0'} 2023-09-22T21:40:18.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061994aa427832bc2-00650e09c2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.098 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:40:18.099 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:40:18.099 DEBUG:boto:Method: GET 2023-09-22T21:40:18.099 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:40:18.099 DEBUG:boto:Data: 2023-09-22T21:40:18.099 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.099 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.099 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.099 DEBUG:boto:Params: {} 2023-09-22T21:40:18.099 DEBUG:boto:Token: None 2023-09-22T21:40:18.100 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj22 2023-09-22T21:40:18.100 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wXK9Amvj9ugGkNAR0jQv8GPOTlY= 2023-09-22T21:40:18.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wXK9Amvj9ugGkNAR0jQv8GPOTlY=', 'Content-Length': '0'} 2023-09-22T21:40:18.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c658e476d2a786a-00650e09c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.103 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:18.104 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:18.104 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:40:18.104 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:40:18.104 DEBUG:boto:Method: GET 2023-09-22T21:40:18.104 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:40:18.104 DEBUG:boto:Data: 2023-09-22T21:40:18.104 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.104 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.104 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.104 DEBUG:boto:Params: {} 2023-09-22T21:40:18.105 DEBUG:boto:Token: None 2023-09-22T21:40:18.105 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/ 2023-09-22T21:40:18.105 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U= 2023-09-22T21:40:18.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U=', 'Content-Length': '0'} 2023-09-22T21:40:18.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5e66ee83f60749a-00650e09c2-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.150 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:18.151 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:18.151 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:18.151 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:40:18.151 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:40:18.151 DEBUG:boto:Method: GET 2023-09-22T21:40:18.151 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:40:18.151 DEBUG:boto:Data: 2023-09-22T21:40:18.152 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.152 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.152 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.152 DEBUG:boto:Params: {} 2023-09-22T21:40:18.152 DEBUG:boto:Token: None 2023-09-22T21:40:18.152 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/ 2023-09-22T21:40:18.152 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U= 2023-09-22T21:40:18.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U=', 'Content-Length': '0'} 2023-09-22T21:40:18.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095aeb3c12a7e5f83-00650e09c2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.159 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:18.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:40:18.159 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:40:18.159 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:40:18.159 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.159 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.159 DEBUG:boto:Method: GET 2023-09-22T21:40:18.160 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.160 DEBUG:boto:Data: 2023-09-22T21:40:18.160 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.160 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.160 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.160 DEBUG:boto:Params: {} 2023-09-22T21:40:18.160 DEBUG:boto:Token: None 2023-09-22T21:40:18.160 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.160 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H86XPJZkPgnxf5R6kjYM7vsVsp4= 2023-09-22T21:40:18.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H86XPJZkPgnxf5R6kjYM7vsVsp4=', 'Content-Length': '0'} 2023-09-22T21:40:18.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fc03a5feb70518e-00650e09c2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.206 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:40:18.207 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:40:18.207 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.207 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.207 DEBUG:boto:Method: GET 2023-09-22T21:40:18.207 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.207 DEBUG:boto:Data: 2023-09-22T21:40:18.207 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.207 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.207 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.207 DEBUG:boto:Params: {} 2023-09-22T21:40:18.208 DEBUG:boto:Token: None 2023-09-22T21:40:18.208 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.208 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H86XPJZkPgnxf5R6kjYM7vsVsp4= 2023-09-22T21:40:18.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H86XPJZkPgnxf5R6kjYM7vsVsp4=', 'Content-Length': '0'} 2023-09-22T21:40:18.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004332e600786a092b-00650e09c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.211 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:40:18.211 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:40:18.211 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.211 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.211 DEBUG:boto:Method: HEAD 2023-09-22T21:40:18.212 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.212 DEBUG:boto:Data: 2023-09-22T21:40:18.212 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.212 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.212 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.212 DEBUG:boto:Params: {} 2023-09-22T21:40:18.212 DEBUG:boto:Token: None 2023-09-22T21:40:18.212 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.212 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71LWdCxZz2/DvHvqWClluZ0lVbk= 2023-09-22T21:40:18.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71LWdCxZz2/DvHvqWClluZ0lVbk=', 'Content-Length': '0'} 2023-09-22T21:40:18.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d6ab6c37c613b9c-00650e09c2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.216 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:40:18.216 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:40:18.216 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.216 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.216 DEBUG:boto:Method: HEAD 2023-09-22T21:40:18.216 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.216 DEBUG:boto:Data: 2023-09-22T21:40:18.216 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.216 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.217 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.217 DEBUG:boto:Params: {} 2023-09-22T21:40:18.217 DEBUG:boto:Token: None 2023-09-22T21:40:18.217 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.217 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71LWdCxZz2/DvHvqWClluZ0lVbk= 2023-09-22T21:40:18.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71LWdCxZz2/DvHvqWClluZ0lVbk=', 'Content-Length': '0'} 2023-09-22T21:40:18.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000734f3896dedd5f2b-00650e09c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:40:18.220 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:40:18.220 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:40:18.220 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.220 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.220 DEBUG:boto:Method: GET 2023-09-22T21:40:18.220 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.220 DEBUG:boto:Data: 2023-09-22T21:40:18.221 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.221 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.221 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.221 DEBUG:boto:Params: {} 2023-09-22T21:40:18.221 DEBUG:boto:Token: None 2023-09-22T21:40:18.221 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.221 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H86XPJZkPgnxf5R6kjYM7vsVsp4= 2023-09-22T21:40:18.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H86XPJZkPgnxf5R6kjYM7vsVsp4=', 'Content-Length': '0'} 2023-09-22T21:40:18.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b7bac6e9afdb776-00650e09c2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.266 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:40:18.267 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:40:18.267 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.267 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.267 DEBUG:boto:Method: GET 2023-09-22T21:40:18.267 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.267 DEBUG:boto:Data: 2023-09-22T21:40:18.267 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.267 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.267 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.267 DEBUG:boto:Params: {} 2023-09-22T21:40:18.268 DEBUG:boto:Token: None 2023-09-22T21:40:18.268 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:40:18.268 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H86XPJZkPgnxf5R6kjYM7vsVsp4= 2023-09-22T21:40:18.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H86XPJZkPgnxf5R6kjYM7vsVsp4=', 'Content-Length': '0'} 2023-09-22T21:40:18.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000098f1bd4fe9c5606-00650e09c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.271 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:18.271 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:18.271 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:40:18.271 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:40:18.271 DEBUG:boto:Method: GET 2023-09-22T21:40:18.272 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:40:18.272 DEBUG:boto:Data: 2023-09-22T21:40:18.272 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.272 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.272 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.272 DEBUG:boto:Params: {} 2023-09-22T21:40:18.272 DEBUG:boto:Token: None 2023-09-22T21:40:18.272 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/ 2023-09-22T21:40:18.272 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U= 2023-09-22T21:40:18.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U=', 'Content-Length': '0'} 2023-09-22T21:40:18.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9f7d6987477d3b5-00650e09c2-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.318 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:18.319 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:18.319 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:18.319 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:40:18.320 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:40:18.320 DEBUG:boto:Method: GET 2023-09-22T21:40:18.320 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:40:18.320 DEBUG:boto:Data: 2023-09-22T21:40:18.320 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.321 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.321 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.321 DEBUG:boto:Params: {} 2023-09-22T21:40:18.321 DEBUG:boto:Token: None 2023-09-22T21:40:18.321 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/ 2023-09-22T21:40:18.321 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U= 2023-09-22T21:40:18.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U=', 'Content-Length': '0'} 2023-09-22T21:40:18.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a3e11b879033775-00650e09c2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.327 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:18.328 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:40:18.328 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:40:18.328 DEBUG:boto:Method: HEAD 2023-09-22T21:40:18.328 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:40:18.328 DEBUG:boto:Data: 2023-09-22T21:40:18.328 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.328 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.328 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.329 DEBUG:boto:Params: {} 2023-09-22T21:40:18.329 DEBUG:boto:Token: None 2023-09-22T21:40:18.329 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj33 2023-09-22T21:40:18.329 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:d3FjLqqPrphwHOrQbafmzMshMkw= 2023-09-22T21:40:18.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:d3FjLqqPrphwHOrQbafmzMshMkw=', 'Content-Length': '0'} 2023-09-22T21:40:18.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008114c97bf31ec024-00650e09c2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.333 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:40:18.333 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:40:18.333 DEBUG:boto:Method: HEAD 2023-09-22T21:40:18.333 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:40:18.333 DEBUG:boto:Data: 2023-09-22T21:40:18.333 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.333 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.333 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.333 DEBUG:boto:Params: {} 2023-09-22T21:40:18.334 DEBUG:boto:Token: None 2023-09-22T21:40:18.334 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj33 2023-09-22T21:40:18.334 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:d3FjLqqPrphwHOrQbafmzMshMkw= 2023-09-22T21:40:18.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:d3FjLqqPrphwHOrQbafmzMshMkw=', 'Content-Length': '0'} 2023-09-22T21:40:18.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e861ba412e92da49-00650e09c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:40:18.337 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:40:18.337 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:40:18.337 DEBUG:boto:Method: GET 2023-09-22T21:40:18.337 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:40:18.337 DEBUG:boto:Data: 2023-09-22T21:40:18.337 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.338 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.338 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.338 DEBUG:boto:Params: {} 2023-09-22T21:40:18.338 DEBUG:boto:Token: None 2023-09-22T21:40:18.338 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj33 2023-09-22T21:40:18.338 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OdTsZ+eovR7CBch/c7N9aPoq7Wc= 2023-09-22T21:40:18.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OdTsZ+eovR7CBch/c7N9aPoq7Wc=', 'Content-Length': '0'} 2023-09-22T21:40:18.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000634c9ed6696f19f4-00650e09c2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.382 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:40:18.383 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:40:18.383 DEBUG:boto:Method: GET 2023-09-22T21:40:18.383 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:40:18.383 DEBUG:boto:Data: 2023-09-22T21:40:18.383 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.383 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.383 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.383 DEBUG:boto:Params: {} 2023-09-22T21:40:18.383 DEBUG:boto:Token: None 2023-09-22T21:40:18.384 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj33 2023-09-22T21:40:18.384 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OdTsZ+eovR7CBch/c7N9aPoq7Wc= 2023-09-22T21:40:18.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OdTsZ+eovR7CBch/c7N9aPoq7Wc=', 'Content-Length': '0'} 2023-09-22T21:40:18.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee6c29b844a0fde1-00650e09c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.387 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:18.387 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:18.387 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:40:18.387 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:40:18.387 DEBUG:boto:Method: GET 2023-09-22T21:40:18.387 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:40:18.387 DEBUG:boto:Data: 2023-09-22T21:40:18.388 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.388 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.388 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.388 DEBUG:boto:Params: {} 2023-09-22T21:40:18.388 DEBUG:boto:Token: None 2023-09-22T21:40:18.388 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/ 2023-09-22T21:40:18.388 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U= 2023-09-22T21:40:18.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U=', 'Content-Length': '0'} 2023-09-22T21:40:18.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000735089910baeb570-00650e09c2-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.434 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:18.435 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:18.435 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:18.435 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:40:18.435 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:40:18.435 DEBUG:boto:Method: GET 2023-09-22T21:40:18.435 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:40:18.435 DEBUG:boto:Data: 2023-09-22T21:40:18.436 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.436 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.436 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.436 DEBUG:boto:Params: {} 2023-09-22T21:40:18.436 DEBUG:boto:Token: None 2023-09-22T21:40:18.436 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/ 2023-09-22T21:40:18.436 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U= 2023-09-22T21:40:18.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U=', 'Content-Length': '0'} 2023-09-22T21:40:18.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b4415155879cefc-00650e09c2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.443 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:18.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:40:18.443 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:40:18.443 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:40:18.443 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.443 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.444 DEBUG:boto:Method: GET 2023-09-22T21:40:18.444 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.444 DEBUG:boto:Data: 2023-09-22T21:40:18.444 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.444 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.444 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.444 DEBUG:boto:Params: {} 2023-09-22T21:40:18.444 DEBUG:boto:Token: None 2023-09-22T21:40:18.444 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.445 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vqT8PNEW3FcWgRDkYANO/iLZKpk= 2023-09-22T21:40:18.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vqT8PNEW3FcWgRDkYANO/iLZKpk=', 'Content-Length': '0'} 2023-09-22T21:40:18.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5e77fb4ab47a329-00650e09c2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.448 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:40:18.448 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:40:18.448 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.448 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.448 DEBUG:boto:Method: GET 2023-09-22T21:40:18.448 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.448 DEBUG:boto:Data: 2023-09-22T21:40:18.449 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.449 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.449 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.449 DEBUG:boto:Params: {} 2023-09-22T21:40:18.449 DEBUG:boto:Token: None 2023-09-22T21:40:18.449 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.449 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vqT8PNEW3FcWgRDkYANO/iLZKpk= 2023-09-22T21:40:18.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vqT8PNEW3FcWgRDkYANO/iLZKpk=', 'Content-Length': '0'} 2023-09-22T21:40:18.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037e9db3dd4cad692-00650e09c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.452 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:40:18.452 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:40:18.452 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.453 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.453 DEBUG:boto:Method: HEAD 2023-09-22T21:40:18.453 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.453 DEBUG:boto:Data: 2023-09-22T21:40:18.453 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.453 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.453 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.453 DEBUG:boto:Params: {} 2023-09-22T21:40:18.453 DEBUG:boto:Token: None 2023-09-22T21:40:18.454 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.454 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t13bLRIQP8ur1JCok8yl/lWD7ZA= 2023-09-22T21:40:18.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t13bLRIQP8ur1JCok8yl/lWD7ZA=', 'Content-Length': '0'} 2023-09-22T21:40:18.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dd6ad8cfadfbf33-00650e09c2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.456 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:40:18.457 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:40:18.457 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.457 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.457 DEBUG:boto:Method: HEAD 2023-09-22T21:40:18.457 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.457 DEBUG:boto:Data: 2023-09-22T21:40:18.457 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.457 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.457 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.457 DEBUG:boto:Params: {} 2023-09-22T21:40:18.458 DEBUG:boto:Token: None 2023-09-22T21:40:18.458 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.458 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t13bLRIQP8ur1JCok8yl/lWD7ZA= 2023-09-22T21:40:18.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t13bLRIQP8ur1JCok8yl/lWD7ZA=', 'Content-Length': '0'} 2023-09-22T21:40:18.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a0f6f9b8e9383e3-00650e09c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:40:18.461 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:40:18.461 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:40:18.461 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.461 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.461 DEBUG:boto:Method: GET 2023-09-22T21:40:18.461 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.462 DEBUG:boto:Data: 2023-09-22T21:40:18.462 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.462 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.462 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.462 DEBUG:boto:Params: {} 2023-09-22T21:40:18.462 DEBUG:boto:Token: None 2023-09-22T21:40:18.462 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.462 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vqT8PNEW3FcWgRDkYANO/iLZKpk= 2023-09-22T21:40:18.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vqT8PNEW3FcWgRDkYANO/iLZKpk=', 'Content-Length': '0'} 2023-09-22T21:40:18.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041965d77e74c74e9-00650e09c2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.506 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:40:18.506 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:40:18.507 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.507 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.507 DEBUG:boto:Method: GET 2023-09-22T21:40:18.507 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.507 DEBUG:boto:Data: 2023-09-22T21:40:18.507 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.507 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.507 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.507 DEBUG:boto:Params: {} 2023-09-22T21:40:18.508 DEBUG:boto:Token: None 2023-09-22T21:40:18.508 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:40:18.508 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vqT8PNEW3FcWgRDkYANO/iLZKpk= 2023-09-22T21:40:18.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vqT8PNEW3FcWgRDkYANO/iLZKpk=', 'Content-Length': '0'} 2023-09-22T21:40:18.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085c2b9bc4f573916-00650e09c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.511 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:18.511 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:18.511 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:40:18.512 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:40:18.512 DEBUG:boto:Method: GET 2023-09-22T21:40:18.512 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:40:18.512 DEBUG:boto:Data: 2023-09-22T21:40:18.512 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.512 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.512 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.512 DEBUG:boto:Params: {} 2023-09-22T21:40:18.512 DEBUG:boto:Token: None 2023-09-22T21:40:18.513 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/ 2023-09-22T21:40:18.513 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U= 2023-09-22T21:40:18.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U=', 'Content-Length': '0'} 2023-09-22T21:40:18.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a3d275b60d18f83-00650e09c2-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.562 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:18.563 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:18.563 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:18.563 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:40:18.563 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:40:18.563 DEBUG:boto:Method: GET 2023-09-22T21:40:18.563 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:40:18.563 DEBUG:boto:Data: 2023-09-22T21:40:18.564 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.564 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.564 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.564 DEBUG:boto:Params: {} 2023-09-22T21:40:18.564 DEBUG:boto:Token: None 2023-09-22T21:40:18.564 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/ 2023-09-22T21:40:18.564 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U= 2023-09-22T21:40:18.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U=', 'Content-Length': '0'} 2023-09-22T21:40:18.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d03601f9d2a88a9-00650e09c2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.571 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:18.571 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:40:18.571 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:40:18.571 DEBUG:boto:Method: HEAD 2023-09-22T21:40:18.571 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:40:18.572 DEBUG:boto:Data: 2023-09-22T21:40:18.572 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.572 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.572 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.572 DEBUG:boto:Params: {} 2023-09-22T21:40:18.572 DEBUG:boto:Token: None 2023-09-22T21:40:18.572 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj34 2023-09-22T21:40:18.572 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:u1pMTMviJfF33B16PsLiiOGKYSk= 2023-09-22T21:40:18.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:u1pMTMviJfF33B16PsLiiOGKYSk=', 'Content-Length': '0'} 2023-09-22T21:40:18.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089d02d70dd704e91-00650e09c2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.575 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:40:18.576 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:40:18.576 DEBUG:boto:Method: HEAD 2023-09-22T21:40:18.576 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:40:18.576 DEBUG:boto:Data: 2023-09-22T21:40:18.576 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.576 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.576 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.576 DEBUG:boto:Params: {} 2023-09-22T21:40:18.576 DEBUG:boto:Token: None 2023-09-22T21:40:18.577 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj34 2023-09-22T21:40:18.577 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:u1pMTMviJfF33B16PsLiiOGKYSk= 2023-09-22T21:40:18.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:u1pMTMviJfF33B16PsLiiOGKYSk=', 'Content-Length': '0'} 2023-09-22T21:40:18.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096c1737f92080788-00650e09c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:40:18.580 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:40:18.580 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:40:18.580 DEBUG:boto:Method: GET 2023-09-22T21:40:18.580 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:40:18.580 DEBUG:boto:Data: 2023-09-22T21:40:18.580 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.580 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.581 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.581 DEBUG:boto:Params: {} 2023-09-22T21:40:18.581 DEBUG:boto:Token: None 2023-09-22T21:40:18.581 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj34 2023-09-22T21:40:18.581 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GSYJIIon4oSb4RPYNie5uVfY888= 2023-09-22T21:40:18.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GSYJIIon4oSb4RPYNie5uVfY888=', 'Content-Length': '0'} 2023-09-22T21:40:18.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ac86429a92e7d34-00650e09c2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.630 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:40:18.630 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:40:18.631 DEBUG:boto:Method: GET 2023-09-22T21:40:18.631 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:40:18.631 DEBUG:boto:Data: 2023-09-22T21:40:18.631 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.631 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.631 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.631 DEBUG:boto:Params: {} 2023-09-22T21:40:18.631 DEBUG:boto:Token: None 2023-09-22T21:40:18.632 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj34 2023-09-22T21:40:18.632 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GSYJIIon4oSb4RPYNie5uVfY888= 2023-09-22T21:40:18.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GSYJIIon4oSb4RPYNie5uVfY888=', 'Content-Length': '0'} 2023-09-22T21:40:18.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9eaf9fa7bdb4465-00650e09c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.635 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:18.635 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:18.635 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:40:18.635 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:40:18.635 DEBUG:boto:Method: GET 2023-09-22T21:40:18.635 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:40:18.635 DEBUG:boto:Data: 2023-09-22T21:40:18.635 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.635 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.636 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.636 DEBUG:boto:Params: {} 2023-09-22T21:40:18.636 DEBUG:boto:Token: None 2023-09-22T21:40:18.636 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/ 2023-09-22T21:40:18.636 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U= 2023-09-22T21:40:18.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U=', 'Content-Length': '0'} 2023-09-22T21:40:18.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011ad5e2dd34f1714-00650e09c2-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.686 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:18.687 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:18.687 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:18.687 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:40:18.687 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:40:18.687 DEBUG:boto:Method: GET 2023-09-22T21:40:18.687 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:40:18.687 DEBUG:boto:Data: 2023-09-22T21:40:18.687 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.687 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.688 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.688 DEBUG:boto:Params: {} 2023-09-22T21:40:18.688 DEBUG:boto:Token: None 2023-09-22T21:40:18.688 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/ 2023-09-22T21:40:18.688 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U= 2023-09-22T21:40:18.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U=', 'Content-Length': '0'} 2023-09-22T21:40:18.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8e902a50fb7ed24-00650e09c2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.695 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:18.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T21:40:18.695 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:40:18.695 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:40:18.695 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.696 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.696 DEBUG:boto:Method: GET 2023-09-22T21:40:18.696 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.696 DEBUG:boto:Data: 2023-09-22T21:40:18.696 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.696 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.696 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.696 DEBUG:boto:Params: {} 2023-09-22T21:40:18.696 DEBUG:boto:Token: None 2023-09-22T21:40:18.697 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.697 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yajbn9Ss+SjibJ1R4FVkddY93UI= 2023-09-22T21:40:18.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yajbn9Ss+SjibJ1R4FVkddY93UI=', 'Content-Length': '0'} 2023-09-22T21:40:18.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099122fe80dc7f99d-00650e09c2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.700 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:40:18.700 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:40:18.700 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.700 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.700 DEBUG:boto:Method: GET 2023-09-22T21:40:18.701 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.701 DEBUG:boto:Data: 2023-09-22T21:40:18.701 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.701 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.701 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.701 DEBUG:boto:Params: {} 2023-09-22T21:40:18.701 DEBUG:boto:Token: None 2023-09-22T21:40:18.701 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.701 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yajbn9Ss+SjibJ1R4FVkddY93UI= 2023-09-22T21:40:18.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yajbn9Ss+SjibJ1R4FVkddY93UI=', 'Content-Length': '0'} 2023-09-22T21:40:18.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc4403d4d9a27c95-00650e09c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.704 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:40:18.705 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:40:18.705 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.705 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.705 DEBUG:boto:Method: HEAD 2023-09-22T21:40:18.705 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.705 DEBUG:boto:Data: 2023-09-22T21:40:18.705 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.705 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.705 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.705 DEBUG:boto:Params: {} 2023-09-22T21:40:18.706 DEBUG:boto:Token: None 2023-09-22T21:40:18.706 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.706 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bNV8FJ/gibTXbYI1IVK/OTZxaVo= 2023-09-22T21:40:18.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bNV8FJ/gibTXbYI1IVK/OTZxaVo=', 'Content-Length': '0'} 2023-09-22T21:40:18.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000186d1b32e4755072-00650e09c2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.709 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:40:18.709 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:40:18.709 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.709 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.709 DEBUG:boto:Method: HEAD 2023-09-22T21:40:18.709 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.709 DEBUG:boto:Data: 2023-09-22T21:40:18.709 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.710 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.710 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.710 DEBUG:boto:Params: {} 2023-09-22T21:40:18.710 DEBUG:boto:Token: None 2023-09-22T21:40:18.710 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.710 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bNV8FJ/gibTXbYI1IVK/OTZxaVo= 2023-09-22T21:40:18.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bNV8FJ/gibTXbYI1IVK/OTZxaVo=', 'Content-Length': '0'} 2023-09-22T21:40:18.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006523f590230f5b9c-00650e09c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T21:40:18.713 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:40:18.713 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:40:18.713 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.714 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.714 DEBUG:boto:Method: GET 2023-09-22T21:40:18.714 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.714 DEBUG:boto:Data: 2023-09-22T21:40:18.714 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.714 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.714 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.714 DEBUG:boto:Params: {} 2023-09-22T21:40:18.714 DEBUG:boto:Token: None 2023-09-22T21:40:18.715 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.715 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yajbn9Ss+SjibJ1R4FVkddY93UI= 2023-09-22T21:40:18.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yajbn9Ss+SjibJ1R4FVkddY93UI=', 'Content-Length': '0'} 2023-09-22T21:40:18.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a4d16048c8d076d-00650e09c2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.758 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:40:18.758 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:40:18.759 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.759 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.759 DEBUG:boto:Method: GET 2023-09-22T21:40:18.759 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.759 DEBUG:boto:Data: 2023-09-22T21:40:18.759 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.759 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.759 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.759 DEBUG:boto:Params: {} 2023-09-22T21:40:18.760 DEBUG:boto:Token: None 2023-09-22T21:40:18.760 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:40:18.760 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yajbn9Ss+SjibJ1R4FVkddY93UI= 2023-09-22T21:40:18.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yajbn9Ss+SjibJ1R4FVkddY93UI=', 'Content-Length': '0'} 2023-09-22T21:40:18.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005df9f14b10bac951-00650e09c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.763 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:18.763 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:18.764 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:40:18.764 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:40:18.764 DEBUG:boto:Method: GET 2023-09-22T21:40:18.764 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:40:18.764 DEBUG:boto:Data: 2023-09-22T21:40:18.764 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.764 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.764 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.764 DEBUG:boto:Params: {} 2023-09-22T21:40:18.765 DEBUG:boto:Token: None 2023-09-22T21:40:18.765 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/ 2023-09-22T21:40:18.765 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U= 2023-09-22T21:40:18.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U=', 'Content-Length': '0'} 2023-09-22T21:40:18.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d79ecacc15b070a0-00650e09c2-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.814 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:18.815 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:18.815 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:18.815 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:40:18.815 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:40:18.815 DEBUG:boto:Method: GET 2023-09-22T21:40:18.816 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:40:18.816 DEBUG:boto:Data: 2023-09-22T21:40:18.816 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.816 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.816 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.816 DEBUG:boto:Params: {} 2023-09-22T21:40:18.816 DEBUG:boto:Token: None 2023-09-22T21:40:18.817 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/ 2023-09-22T21:40:18.817 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U= 2023-09-22T21:40:18.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U=', 'Content-Length': '0'} 2023-09-22T21:40:18.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebe1b27e2d97d682-00650e09c2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.827 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:18.828 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:40:18.828 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:40:18.828 DEBUG:boto:Method: HEAD 2023-09-22T21:40:18.828 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:40:18.828 DEBUG:boto:Data: 2023-09-22T21:40:18.828 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.828 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.828 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.829 DEBUG:boto:Params: {} 2023-09-22T21:40:18.829 DEBUG:boto:Token: None 2023-09-22T21:40:18.829 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj38 2023-09-22T21:40:18.829 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EADI34TRNbna5bfJKw9MjphSfnc= 2023-09-22T21:40:18.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EADI34TRNbna5bfJKw9MjphSfnc=', 'Content-Length': '0'} 2023-09-22T21:40:18.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b728bd14ca8b3022-00650e09c2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.835 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:40:18.835 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:40:18.835 DEBUG:boto:Method: HEAD 2023-09-22T21:40:18.835 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:40:18.835 DEBUG:boto:Data: 2023-09-22T21:40:18.835 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.836 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.836 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.836 DEBUG:boto:Params: {} 2023-09-22T21:40:18.836 DEBUG:boto:Token: None 2023-09-22T21:40:18.836 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj38 2023-09-22T21:40:18.836 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EADI34TRNbna5bfJKw9MjphSfnc= 2023-09-22T21:40:18.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EADI34TRNbna5bfJKw9MjphSfnc=', 'Content-Length': '0'} 2023-09-22T21:40:18.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007121ea038e8f2bcc-00650e09c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T21:40:18.840 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:40:18.840 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:40:18.840 DEBUG:boto:Method: GET 2023-09-22T21:40:18.841 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:40:18.841 DEBUG:boto:Data: 2023-09-22T21:40:18.841 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.841 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.841 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.841 DEBUG:boto:Params: {} 2023-09-22T21:40:18.841 DEBUG:boto:Token: None 2023-09-22T21:40:18.842 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj38 2023-09-22T21:40:18.842 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CDjTHoK7PFpcmqQIKzb9MctnJnk= 2023-09-22T21:40:18.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CDjTHoK7PFpcmqQIKzb9MctnJnk=', 'Content-Length': '0'} 2023-09-22T21:40:18.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dc0a684aee6919d-00650e09c2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.886 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:40:18.887 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:40:18.887 DEBUG:boto:Method: GET 2023-09-22T21:40:18.887 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:40:18.887 DEBUG:boto:Data: 2023-09-22T21:40:18.887 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.887 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.887 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.887 DEBUG:boto:Params: {} 2023-09-22T21:40:18.887 DEBUG:boto:Token: None 2023-09-22T21:40:18.888 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj38 2023-09-22T21:40:18.888 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CDjTHoK7PFpcmqQIKzb9MctnJnk= 2023-09-22T21:40:18.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CDjTHoK7PFpcmqQIKzb9MctnJnk=', 'Content-Length': '0'} 2023-09-22T21:40:18.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a131caec133e1ec0-00650e09c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.891 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:18.891 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:18.891 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:40:18.891 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:40:18.891 DEBUG:boto:Method: GET 2023-09-22T21:40:18.891 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:40:18.892 DEBUG:boto:Data: 2023-09-22T21:40:18.892 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.892 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.892 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.892 DEBUG:boto:Params: {} 2023-09-22T21:40:18.892 DEBUG:boto:Token: None 2023-09-22T21:40:18.892 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/ 2023-09-22T21:40:18.892 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U= 2023-09-22T21:40:18.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U=', 'Content-Length': '0'} 2023-09-22T21:40:18.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008eacaea39b4a83a6-00650e09c2-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.938 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:18.939 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:18.939 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:18.939 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:40:18.939 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:40:18.939 DEBUG:boto:Method: GET 2023-09-22T21:40:18.939 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:40:18.939 DEBUG:boto:Data: 2023-09-22T21:40:18.939 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.939 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.940 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.940 DEBUG:boto:Params: {} 2023-09-22T21:40:18.940 DEBUG:boto:Token: None 2023-09-22T21:40:18.940 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/ 2023-09-22T21:40:18.940 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U= 2023-09-22T21:40:18.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9NKv9HzswVorYOH8dhPdNUFQD5U=', 'Content-Length': '0'} 2023-09-22T21:40:18.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a714ddcff98c46b2-00650e09c2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.947 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:18.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:40:18.947 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:40:18.947 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:40:18.948 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:18.948 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:18.948 DEBUG:boto:Method: GET 2023-09-22T21:40:18.948 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:18.948 DEBUG:boto:Data: 2023-09-22T21:40:18.948 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.948 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.948 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.949 DEBUG:boto:Params: {} 2023-09-22T21:40:18.949 DEBUG:boto:Token: None 2023-09-22T21:40:18.949 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:18.949 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7ynstCqEvnpHn5ru5K7FWp0kFr8= 2023-09-22T21:40:18.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7ynstCqEvnpHn5ru5K7FWp0kFr8=', 'Content-Length': '0'} 2023-09-22T21:40:18.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b304f9614b1628b-00650e09c2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.952 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:40:18.952 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:40:18.952 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:18.952 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:18.952 DEBUG:boto:Method: GET 2023-09-22T21:40:18.952 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:18.953 DEBUG:boto:Data: 2023-09-22T21:40:18.953 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.953 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.953 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.953 DEBUG:boto:Params: {} 2023-09-22T21:40:18.953 DEBUG:boto:Token: None 2023-09-22T21:40:18.953 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:18.953 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7ynstCqEvnpHn5ru5K7FWp0kFr8= 2023-09-22T21:40:18.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7ynstCqEvnpHn5ru5K7FWp0kFr8=', 'Content-Length': '0'} 2023-09-22T21:40:18.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b674f00694a37db8-00650e09c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.957 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:40:18.957 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:40:18.957 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:18.957 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:18.957 DEBUG:boto:Method: HEAD 2023-09-22T21:40:18.957 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:18.957 DEBUG:boto:Data: 2023-09-22T21:40:18.957 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.957 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.958 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.958 DEBUG:boto:Params: {} 2023-09-22T21:40:18.958 DEBUG:boto:Token: None 2023-09-22T21:40:18.958 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:18.958 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wX9T1XjeDvrpVvtBF74cc7USZaY= 2023-09-22T21:40:18.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wX9T1XjeDvrpVvtBF74cc7USZaY=', 'Content-Length': '0'} 2023-09-22T21:40:18.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000549cf329c5ce2587-00650e09c2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.961 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:40:18.961 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:40:18.961 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:18.962 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:18.962 DEBUG:boto:Method: HEAD 2023-09-22T21:40:18.962 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:18.962 DEBUG:boto:Data: 2023-09-22T21:40:18.962 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.962 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:18.962 DEBUG:boto:Port: 8000 2023-09-22T21:40:18.962 DEBUG:boto:Params: {} 2023-09-22T21:40:18.962 DEBUG:boto:Token: None 2023-09-22T21:40:18.962 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:18.963 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wX9T1XjeDvrpVvtBF74cc7USZaY= 2023-09-22T21:40:18.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wX9T1XjeDvrpVvtBF74cc7USZaY=', 'Content-Length': '0'} 2023-09-22T21:40:18.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4a3f02a7b4ce872-00650e09c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:18.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:40:18.966 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:40:18.966 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:40:18.966 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:18.966 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:18.966 DEBUG:boto:Method: GET 2023-09-22T21:40:18.966 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:18.966 DEBUG:boto:Data: 2023-09-22T21:40:18.966 DEBUG:boto:Headers: {} 2023-09-22T21:40:18.967 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:18.967 DEBUG:boto:Port: 8001 2023-09-22T21:40:18.967 DEBUG:boto:Params: {} 2023-09-22T21:40:18.967 DEBUG:boto:Token: None 2023-09-22T21:40:18.967 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:18 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:18.967 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7ynstCqEvnpHn5ru5K7FWp0kFr8= 2023-09-22T21:40:18.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7ynstCqEvnpHn5ru5K7FWp0kFr8=', 'Content-Length': '0'} 2023-09-22T21:40:18.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000788a03067772f743-00650e09c2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.014 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:40:19.014 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:40:19.015 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:19.015 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:19.015 DEBUG:boto:Method: GET 2023-09-22T21:40:19.015 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:19.015 DEBUG:boto:Data: 2023-09-22T21:40:19.015 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.015 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.015 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.015 DEBUG:boto:Params: {} 2023-09-22T21:40:19.015 DEBUG:boto:Token: None 2023-09-22T21:40:19.016 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:40:19.016 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mOtH6MYneDvmstgQAzN0I2W1RwI= 2023-09-22T21:40:19.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mOtH6MYneDvmstgQAzN0I2W1RwI=', 'Content-Length': '0'} 2023-09-22T21:40:19.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb3b78cc2e04afeb-00650e09c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.019 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:19.019 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:19.020 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:40:19.020 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:40:19.020 DEBUG:boto:Method: GET 2023-09-22T21:40:19.020 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:40:19.020 DEBUG:boto:Data: 2023-09-22T21:40:19.020 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.020 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.020 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.020 DEBUG:boto:Params: {} 2023-09-22T21:40:19.020 DEBUG:boto:Token: None 2023-09-22T21:40:19.021 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/ 2023-09-22T21:40:19.021 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ= 2023-09-22T21:40:19.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ=', 'Content-Length': '0'} 2023-09-22T21:40:19.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000623cc5b4febe7254-00650e09c3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.070 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:19.071 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:19.071 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:19.071 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:40:19.071 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:40:19.071 DEBUG:boto:Method: GET 2023-09-22T21:40:19.071 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:40:19.071 DEBUG:boto:Data: 2023-09-22T21:40:19.072 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.072 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.072 DEBUG:boto:Params: {} 2023-09-22T21:40:19.072 DEBUG:boto:Token: None 2023-09-22T21:40:19.072 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/ 2023-09-22T21:40:19.072 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ= 2023-09-22T21:40:19.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ=', 'Content-Length': '0'} 2023-09-22T21:40:19.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000874f6d6241f11aa9-00650e09c3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.079 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:19.079 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:40:19.079 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:40:19.079 DEBUG:boto:Method: HEAD 2023-09-22T21:40:19.079 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:40:19.080 DEBUG:boto:Data: 2023-09-22T21:40:19.080 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.080 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.080 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.080 DEBUG:boto:Params: {} 2023-09-22T21:40:19.080 DEBUG:boto:Token: None 2023-09-22T21:40:19.080 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj44 2023-09-22T21:40:19.080 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:X6dt+7T2KIBeLQsLZOKTkBf/fnc= 2023-09-22T21:40:19.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:X6dt+7T2KIBeLQsLZOKTkBf/fnc=', 'Content-Length': '0'} 2023-09-22T21:40:19.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fcdbf029a393925-00650e09c3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.083 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:40:19.084 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:40:19.084 DEBUG:boto:Method: HEAD 2023-09-22T21:40:19.084 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:40:19.084 DEBUG:boto:Data: 2023-09-22T21:40:19.084 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.084 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.084 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.084 DEBUG:boto:Params: {} 2023-09-22T21:40:19.085 DEBUG:boto:Token: None 2023-09-22T21:40:19.085 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj44 2023-09-22T21:40:19.085 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:X6dt+7T2KIBeLQsLZOKTkBf/fnc= 2023-09-22T21:40:19.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:X6dt+7T2KIBeLQsLZOKTkBf/fnc=', 'Content-Length': '0'} 2023-09-22T21:40:19.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000747fc801104a1307-00650e09c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:40:19.088 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:40:19.088 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:40:19.088 DEBUG:boto:Method: GET 2023-09-22T21:40:19.088 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:40:19.088 DEBUG:boto:Data: 2023-09-22T21:40:19.088 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.088 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.088 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.088 DEBUG:boto:Params: {} 2023-09-22T21:40:19.089 DEBUG:boto:Token: None 2023-09-22T21:40:19.089 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj44 2023-09-22T21:40:19.089 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NqonweMT25Tz4dDC63aBp7C8mT8= 2023-09-22T21:40:19.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NqonweMT25Tz4dDC63aBp7C8mT8=', 'Content-Length': '0'} 2023-09-22T21:40:19.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcb077b65847ccfd-00650e09c3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.134 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:40:19.135 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:40:19.135 DEBUG:boto:Method: GET 2023-09-22T21:40:19.135 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:40:19.135 DEBUG:boto:Data: 2023-09-22T21:40:19.135 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.135 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.135 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.135 DEBUG:boto:Params: {} 2023-09-22T21:40:19.136 DEBUG:boto:Token: None 2023-09-22T21:40:19.136 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj44 2023-09-22T21:40:19.136 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NqonweMT25Tz4dDC63aBp7C8mT8= 2023-09-22T21:40:19.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NqonweMT25Tz4dDC63aBp7C8mT8=', 'Content-Length': '0'} 2023-09-22T21:40:19.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a66792c89a0fbf55-00650e09c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.139 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:19.139 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:19.139 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:40:19.139 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:40:19.139 DEBUG:boto:Method: GET 2023-09-22T21:40:19.140 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:40:19.140 DEBUG:boto:Data: 2023-09-22T21:40:19.140 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.140 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.140 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.140 DEBUG:boto:Params: {} 2023-09-22T21:40:19.140 DEBUG:boto:Token: None 2023-09-22T21:40:19.140 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/ 2023-09-22T21:40:19.140 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ= 2023-09-22T21:40:19.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ=', 'Content-Length': '0'} 2023-09-22T21:40:19.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7c4ccf2b3c60edf-00650e09c3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.186 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:19.187 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:19.187 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:19.187 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:40:19.187 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:40:19.187 DEBUG:boto:Method: GET 2023-09-22T21:40:19.187 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:40:19.188 DEBUG:boto:Data: 2023-09-22T21:40:19.188 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.188 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.188 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.188 DEBUG:boto:Params: {} 2023-09-22T21:40:19.188 DEBUG:boto:Token: None 2023-09-22T21:40:19.188 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/ 2023-09-22T21:40:19.188 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ= 2023-09-22T21:40:19.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ=', 'Content-Length': '0'} 2023-09-22T21:40:19.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008efc7d0c08fba7ad-00650e09c3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.195 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:19.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:40:19.196 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:40:19.196 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:40:19.196 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.196 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.196 DEBUG:boto:Method: GET 2023-09-22T21:40:19.196 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.197 DEBUG:boto:Data: 2023-09-22T21:40:19.197 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.197 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.197 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.197 DEBUG:boto:Params: {} 2023-09-22T21:40:19.197 DEBUG:boto:Token: None 2023-09-22T21:40:19.197 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.197 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2xfRePXOJ48xyssLBzv71g59OsA= 2023-09-22T21:40:19.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2xfRePXOJ48xyssLBzv71g59OsA=', 'Content-Length': '0'} 2023-09-22T21:40:19.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a2f104306ee4f0e-00650e09c3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.201 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:40:19.201 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:40:19.201 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.201 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.201 DEBUG:boto:Method: GET 2023-09-22T21:40:19.201 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.201 DEBUG:boto:Data: 2023-09-22T21:40:19.201 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.201 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.202 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.202 DEBUG:boto:Params: {} 2023-09-22T21:40:19.202 DEBUG:boto:Token: None 2023-09-22T21:40:19.202 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.202 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2xfRePXOJ48xyssLBzv71g59OsA= 2023-09-22T21:40:19.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2xfRePXOJ48xyssLBzv71g59OsA=', 'Content-Length': '0'} 2023-09-22T21:40:19.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c716706d0807a335-00650e09c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.205 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:40:19.205 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:40:19.205 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.205 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.206 DEBUG:boto:Method: HEAD 2023-09-22T21:40:19.206 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.206 DEBUG:boto:Data: 2023-09-22T21:40:19.206 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.206 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.206 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.206 DEBUG:boto:Params: {} 2023-09-22T21:40:19.206 DEBUG:boto:Token: None 2023-09-22T21:40:19.206 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.207 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m7E6QQlfU2JFLIGHUZ+eICndOgQ= 2023-09-22T21:40:19.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m7E6QQlfU2JFLIGHUZ+eICndOgQ=', 'Content-Length': '0'} 2023-09-22T21:40:19.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd5c1b579ea162fc-00650e09c3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.209 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:40:19.210 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:40:19.210 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.210 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.210 DEBUG:boto:Method: HEAD 2023-09-22T21:40:19.210 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.210 DEBUG:boto:Data: 2023-09-22T21:40:19.210 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.210 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.210 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.211 DEBUG:boto:Params: {} 2023-09-22T21:40:19.211 DEBUG:boto:Token: None 2023-09-22T21:40:19.211 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.211 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m7E6QQlfU2JFLIGHUZ+eICndOgQ= 2023-09-22T21:40:19.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m7E6QQlfU2JFLIGHUZ+eICndOgQ=', 'Content-Length': '0'} 2023-09-22T21:40:19.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7973b8a7531df0a-00650e09c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:40:19.214 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:40:19.214 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:40:19.214 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.214 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.214 DEBUG:boto:Method: GET 2023-09-22T21:40:19.214 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.215 DEBUG:boto:Data: 2023-09-22T21:40:19.215 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.215 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.215 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.215 DEBUG:boto:Params: {} 2023-09-22T21:40:19.215 DEBUG:boto:Token: None 2023-09-22T21:40:19.215 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.215 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2xfRePXOJ48xyssLBzv71g59OsA= 2023-09-22T21:40:19.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2xfRePXOJ48xyssLBzv71g59OsA=', 'Content-Length': '0'} 2023-09-22T21:40:19.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000094c2591a73c9bbe-00650e09c3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.262 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:40:19.263 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:40:19.263 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.263 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.263 DEBUG:boto:Method: GET 2023-09-22T21:40:19.263 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.263 DEBUG:boto:Data: 2023-09-22T21:40:19.263 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.263 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.264 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.264 DEBUG:boto:Params: {} 2023-09-22T21:40:19.264 DEBUG:boto:Token: None 2023-09-22T21:40:19.264 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:40:19.264 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2xfRePXOJ48xyssLBzv71g59OsA= 2023-09-22T21:40:19.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2xfRePXOJ48xyssLBzv71g59OsA=', 'Content-Length': '0'} 2023-09-22T21:40:19.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003685e086c498b73f-00650e09c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.267 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:19.268 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:19.268 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:40:19.268 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:40:19.268 DEBUG:boto:Method: GET 2023-09-22T21:40:19.268 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:40:19.268 DEBUG:boto:Data: 2023-09-22T21:40:19.268 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.268 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.268 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.268 DEBUG:boto:Params: {} 2023-09-22T21:40:19.269 DEBUG:boto:Token: None 2023-09-22T21:40:19.269 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/ 2023-09-22T21:40:19.269 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ= 2023-09-22T21:40:19.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ=', 'Content-Length': '0'} 2023-09-22T21:40:19.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000085bc111de520dc7-00650e09c3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.318 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:19.319 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:19.319 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:19.319 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:40:19.319 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:40:19.319 DEBUG:boto:Method: GET 2023-09-22T21:40:19.319 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:40:19.320 DEBUG:boto:Data: 2023-09-22T21:40:19.320 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.320 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.320 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.320 DEBUG:boto:Params: {} 2023-09-22T21:40:19.320 DEBUG:boto:Token: None 2023-09-22T21:40:19.320 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/ 2023-09-22T21:40:19.320 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ= 2023-09-22T21:40:19.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ=', 'Content-Length': '0'} 2023-09-22T21:40:19.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000067ccb42bc161106-00650e09c3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.327 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:19.327 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:40:19.327 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:40:19.327 DEBUG:boto:Method: HEAD 2023-09-22T21:40:19.327 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:40:19.328 DEBUG:boto:Data: 2023-09-22T21:40:19.328 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.328 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.328 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.328 DEBUG:boto:Params: {} 2023-09-22T21:40:19.328 DEBUG:boto:Token: None 2023-09-22T21:40:19.328 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj51 2023-09-22T21:40:19.328 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:k5g5yh5Jk40vSuauEVvCEcAMJmA= 2023-09-22T21:40:19.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:k5g5yh5Jk40vSuauEVvCEcAMJmA=', 'Content-Length': '0'} 2023-09-22T21:40:19.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012c670e0791fd74d-00650e09c3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.331 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:40:19.331 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:40:19.331 DEBUG:boto:Method: HEAD 2023-09-22T21:40:19.332 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:40:19.332 DEBUG:boto:Data: 2023-09-22T21:40:19.332 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.332 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.332 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.332 DEBUG:boto:Params: {} 2023-09-22T21:40:19.332 DEBUG:boto:Token: None 2023-09-22T21:40:19.332 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj51 2023-09-22T21:40:19.333 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:k5g5yh5Jk40vSuauEVvCEcAMJmA= 2023-09-22T21:40:19.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:k5g5yh5Jk40vSuauEVvCEcAMJmA=', 'Content-Length': '0'} 2023-09-22T21:40:19.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b85c9d72af65f25-00650e09c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:40:19.335 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:40:19.335 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:40:19.336 DEBUG:boto:Method: GET 2023-09-22T21:40:19.336 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:40:19.336 DEBUG:boto:Data: 2023-09-22T21:40:19.336 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.336 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.336 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.336 DEBUG:boto:Params: {} 2023-09-22T21:40:19.336 DEBUG:boto:Token: None 2023-09-22T21:40:19.336 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj51 2023-09-22T21:40:19.337 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bgC5NcxNL3ODE3OBSwxF2exU4vo= 2023-09-22T21:40:19.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bgC5NcxNL3ODE3OBSwxF2exU4vo=', 'Content-Length': '0'} 2023-09-22T21:40:19.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cb7a5dac6f56169-00650e09c3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.382 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:40:19.383 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:40:19.383 DEBUG:boto:Method: GET 2023-09-22T21:40:19.383 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:40:19.383 DEBUG:boto:Data: 2023-09-22T21:40:19.383 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.383 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.383 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.383 DEBUG:boto:Params: {} 2023-09-22T21:40:19.384 DEBUG:boto:Token: None 2023-09-22T21:40:19.384 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj51 2023-09-22T21:40:19.384 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bgC5NcxNL3ODE3OBSwxF2exU4vo= 2023-09-22T21:40:19.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bgC5NcxNL3ODE3OBSwxF2exU4vo=', 'Content-Length': '0'} 2023-09-22T21:40:19.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000316b07630db8819a-00650e09c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.387 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:19.387 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:19.388 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:40:19.388 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:40:19.388 DEBUG:boto:Method: GET 2023-09-22T21:40:19.388 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:40:19.388 DEBUG:boto:Data: 2023-09-22T21:40:19.388 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.388 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.388 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.388 DEBUG:boto:Params: {} 2023-09-22T21:40:19.389 DEBUG:boto:Token: None 2023-09-22T21:40:19.389 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/ 2023-09-22T21:40:19.389 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ= 2023-09-22T21:40:19.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ=', 'Content-Length': '0'} 2023-09-22T21:40:19.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1d5202577113ef0-00650e09c3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.438 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:19.439 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:19.439 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:19.439 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:40:19.439 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:40:19.439 DEBUG:boto:Method: GET 2023-09-22T21:40:19.439 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:40:19.439 DEBUG:boto:Data: 2023-09-22T21:40:19.439 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.440 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.440 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.440 DEBUG:boto:Params: {} 2023-09-22T21:40:19.440 DEBUG:boto:Token: None 2023-09-22T21:40:19.440 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/ 2023-09-22T21:40:19.440 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ= 2023-09-22T21:40:19.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ=', 'Content-Length': '0'} 2023-09-22T21:40:19.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af62eee3d2da7aff-00650e09c3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.447 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:19.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:40:19.447 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:40:19.448 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:40:19.448 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.448 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.448 DEBUG:boto:Method: GET 2023-09-22T21:40:19.448 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.448 DEBUG:boto:Data: 2023-09-22T21:40:19.448 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.448 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.448 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.449 DEBUG:boto:Params: {} 2023-09-22T21:40:19.449 DEBUG:boto:Token: None 2023-09-22T21:40:19.449 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.449 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8zjSZz5zzHte+kbdRq1O+IsulLk= 2023-09-22T21:40:19.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8zjSZz5zzHte+kbdRq1O+IsulLk=', 'Content-Length': '0'} 2023-09-22T21:40:19.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0dc004f33c3466f-00650e09c3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.452 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:40:19.452 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:40:19.452 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.452 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.452 DEBUG:boto:Method: GET 2023-09-22T21:40:19.452 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.453 DEBUG:boto:Data: 2023-09-22T21:40:19.453 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.453 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.453 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.453 DEBUG:boto:Params: {} 2023-09-22T21:40:19.453 DEBUG:boto:Token: None 2023-09-22T21:40:19.453 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.453 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8zjSZz5zzHte+kbdRq1O+IsulLk= 2023-09-22T21:40:19.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8zjSZz5zzHte+kbdRq1O+IsulLk=', 'Content-Length': '0'} 2023-09-22T21:40:19.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bfc0c81914281ef-00650e09c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.456 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:40:19.456 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:40:19.456 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.457 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.457 DEBUG:boto:Method: HEAD 2023-09-22T21:40:19.457 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.457 DEBUG:boto:Data: 2023-09-22T21:40:19.457 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.457 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.457 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.457 DEBUG:boto:Params: {} 2023-09-22T21:40:19.457 DEBUG:boto:Token: None 2023-09-22T21:40:19.458 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.458 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hQl9AwSKTK2VSYR9HTkazN04rn4= 2023-09-22T21:40:19.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hQl9AwSKTK2VSYR9HTkazN04rn4=', 'Content-Length': '0'} 2023-09-22T21:40:19.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf004ddb7df138c4-00650e09c3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.460 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:40:19.461 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:40:19.461 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.461 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.461 DEBUG:boto:Method: HEAD 2023-09-22T21:40:19.461 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.461 DEBUG:boto:Data: 2023-09-22T21:40:19.461 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.461 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.461 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.461 DEBUG:boto:Params: {} 2023-09-22T21:40:19.462 DEBUG:boto:Token: None 2023-09-22T21:40:19.462 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.462 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hQl9AwSKTK2VSYR9HTkazN04rn4= 2023-09-22T21:40:19.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hQl9AwSKTK2VSYR9HTkazN04rn4=', 'Content-Length': '0'} 2023-09-22T21:40:19.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c81e19673fd00f5-00650e09c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:40:19.465 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:40:19.465 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:40:19.465 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.465 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.465 DEBUG:boto:Method: GET 2023-09-22T21:40:19.465 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.465 DEBUG:boto:Data: 2023-09-22T21:40:19.465 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.465 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.466 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.466 DEBUG:boto:Params: {} 2023-09-22T21:40:19.466 DEBUG:boto:Token: None 2023-09-22T21:40:19.466 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.466 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8zjSZz5zzHte+kbdRq1O+IsulLk= 2023-09-22T21:40:19.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8zjSZz5zzHte+kbdRq1O+IsulLk=', 'Content-Length': '0'} 2023-09-22T21:40:19.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b6d094999469fb5-00650e09c3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.510 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:40:19.511 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:40:19.511 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.511 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.511 DEBUG:boto:Method: GET 2023-09-22T21:40:19.511 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.511 DEBUG:boto:Data: 2023-09-22T21:40:19.511 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.511 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.512 DEBUG:boto:Params: {} 2023-09-22T21:40:19.512 DEBUG:boto:Token: None 2023-09-22T21:40:19.512 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:40:19.512 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8zjSZz5zzHte+kbdRq1O+IsulLk= 2023-09-22T21:40:19.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8zjSZz5zzHte+kbdRq1O+IsulLk=', 'Content-Length': '0'} 2023-09-22T21:40:19.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055cd5aece2e4ee1d-00650e09c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.515 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:19.515 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:19.516 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:40:19.516 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:40:19.516 DEBUG:boto:Method: GET 2023-09-22T21:40:19.516 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:40:19.516 DEBUG:boto:Data: 2023-09-22T21:40:19.516 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.516 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.516 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.516 DEBUG:boto:Params: {} 2023-09-22T21:40:19.517 DEBUG:boto:Token: None 2023-09-22T21:40:19.517 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/ 2023-09-22T21:40:19.517 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ= 2023-09-22T21:40:19.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ=', 'Content-Length': '0'} 2023-09-22T21:40:19.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002345008181351168-00650e09c3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.566 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:19.567 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:19.567 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:19.567 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:40:19.567 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:40:19.567 DEBUG:boto:Method: GET 2023-09-22T21:40:19.567 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:40:19.567 DEBUG:boto:Data: 2023-09-22T21:40:19.567 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.568 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.568 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.568 DEBUG:boto:Params: {} 2023-09-22T21:40:19.568 DEBUG:boto:Token: None 2023-09-22T21:40:19.568 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/ 2023-09-22T21:40:19.568 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ= 2023-09-22T21:40:19.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ=', 'Content-Length': '0'} 2023-09-22T21:40:19.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dd9b712499e2596-00650e09c3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.575 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:19.575 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:40:19.575 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:40:19.575 DEBUG:boto:Method: HEAD 2023-09-22T21:40:19.575 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:40:19.576 DEBUG:boto:Data: 2023-09-22T21:40:19.576 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.576 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.576 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.576 DEBUG:boto:Params: {} 2023-09-22T21:40:19.576 DEBUG:boto:Token: None 2023-09-22T21:40:19.576 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj55 2023-09-22T21:40:19.577 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+SI4Y0m9TGr6mZNA/7Y8nQOCSe0= 2023-09-22T21:40:19.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+SI4Y0m9TGr6mZNA/7Y8nQOCSe0=', 'Content-Length': '0'} 2023-09-22T21:40:19.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc36a6625ef34e38-00650e09c3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.580 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:40:19.580 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:40:19.580 DEBUG:boto:Method: HEAD 2023-09-22T21:40:19.580 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:40:19.580 DEBUG:boto:Data: 2023-09-22T21:40:19.580 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.581 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.581 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.581 DEBUG:boto:Params: {} 2023-09-22T21:40:19.581 DEBUG:boto:Token: None 2023-09-22T21:40:19.581 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj55 2023-09-22T21:40:19.581 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+SI4Y0m9TGr6mZNA/7Y8nQOCSe0= 2023-09-22T21:40:19.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+SI4Y0m9TGr6mZNA/7Y8nQOCSe0=', 'Content-Length': '0'} 2023-09-22T21:40:19.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000510506357da285aa-00650e09c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:40:19.585 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:40:19.585 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:40:19.585 DEBUG:boto:Method: GET 2023-09-22T21:40:19.585 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:40:19.585 DEBUG:boto:Data: 2023-09-22T21:40:19.586 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.586 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.586 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.586 DEBUG:boto:Params: {} 2023-09-22T21:40:19.586 DEBUG:boto:Token: None 2023-09-22T21:40:19.586 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj55 2023-09-22T21:40:19.586 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uuok8a0ls1RAxnizDMVTVnNNscE= 2023-09-22T21:40:19.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uuok8a0ls1RAxnizDMVTVnNNscE=', 'Content-Length': '0'} 2023-09-22T21:40:19.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000207ba59ca97d995a-00650e09c3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.630 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:40:19.631 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:40:19.631 DEBUG:boto:Method: GET 2023-09-22T21:40:19.631 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:40:19.631 DEBUG:boto:Data: 2023-09-22T21:40:19.631 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.631 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.632 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.632 DEBUG:boto:Params: {} 2023-09-22T21:40:19.632 DEBUG:boto:Token: None 2023-09-22T21:40:19.632 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj55 2023-09-22T21:40:19.632 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uuok8a0ls1RAxnizDMVTVnNNscE= 2023-09-22T21:40:19.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uuok8a0ls1RAxnizDMVTVnNNscE=', 'Content-Length': '0'} 2023-09-22T21:40:19.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e340d18ddc2020b-00650e09c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.636 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:19.636 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:19.636 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:40:19.637 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:40:19.637 DEBUG:boto:Method: GET 2023-09-22T21:40:19.637 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:40:19.637 DEBUG:boto:Data: 2023-09-22T21:40:19.637 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.637 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.638 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.638 DEBUG:boto:Params: {} 2023-09-22T21:40:19.638 DEBUG:boto:Token: None 2023-09-22T21:40:19.638 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/ 2023-09-22T21:40:19.638 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ= 2023-09-22T21:40:19.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ=', 'Content-Length': '0'} 2023-09-22T21:40:19.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013f456d4cd106eb0-00650e09c3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.690 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:19.691 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:19.691 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:19.691 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:40:19.691 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:40:19.692 DEBUG:boto:Method: GET 2023-09-22T21:40:19.692 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:40:19.692 DEBUG:boto:Data: 2023-09-22T21:40:19.692 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.692 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.692 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.692 DEBUG:boto:Params: {} 2023-09-22T21:40:19.693 DEBUG:boto:Token: None 2023-09-22T21:40:19.693 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/ 2023-09-22T21:40:19.693 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ= 2023-09-22T21:40:19.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ=', 'Content-Length': '0'} 2023-09-22T21:40:19.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a881f26d34c6053-00650e09c3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.700 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:19.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T21:40:19.701 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:40:19.701 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:40:19.701 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.701 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.702 DEBUG:boto:Method: GET 2023-09-22T21:40:19.702 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.702 DEBUG:boto:Data: 2023-09-22T21:40:19.702 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.702 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.702 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.702 DEBUG:boto:Params: {} 2023-09-22T21:40:19.702 DEBUG:boto:Token: None 2023-09-22T21:40:19.703 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.703 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QI2/8jjYQhh+zr59NP++xxR1r98= 2023-09-22T21:40:19.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QI2/8jjYQhh+zr59NP++xxR1r98=', 'Content-Length': '0'} 2023-09-22T21:40:19.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c12c64bb8e8a19a-00650e09c3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.706 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:40:19.706 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:40:19.707 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.707 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.707 DEBUG:boto:Method: GET 2023-09-22T21:40:19.707 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.707 DEBUG:boto:Data: 2023-09-22T21:40:19.707 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.707 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.707 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.708 DEBUG:boto:Params: {} 2023-09-22T21:40:19.708 DEBUG:boto:Token: None 2023-09-22T21:40:19.708 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.708 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QI2/8jjYQhh+zr59NP++xxR1r98= 2023-09-22T21:40:19.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QI2/8jjYQhh+zr59NP++xxR1r98=', 'Content-Length': '0'} 2023-09-22T21:40:19.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084e70e87273b9be3-00650e09c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.711 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:40:19.711 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:40:19.712 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.712 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.712 DEBUG:boto:Method: HEAD 2023-09-22T21:40:19.712 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.712 DEBUG:boto:Data: 2023-09-22T21:40:19.713 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.713 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.713 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.713 DEBUG:boto:Params: {} 2023-09-22T21:40:19.713 DEBUG:boto:Token: None 2023-09-22T21:40:19.713 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.713 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t59JFI7qQCkryjRdninYjPBQKe4= 2023-09-22T21:40:19.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t59JFI7qQCkryjRdninYjPBQKe4=', 'Content-Length': '0'} 2023-09-22T21:40:19.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003105ec5933a8924e-00650e09c3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.717 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:40:19.717 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:40:19.717 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.718 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.718 DEBUG:boto:Method: HEAD 2023-09-22T21:40:19.718 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.719 DEBUG:boto:Data: 2023-09-22T21:40:19.719 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.719 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.720 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.720 DEBUG:boto:Params: {} 2023-09-22T21:40:19.720 DEBUG:boto:Token: None 2023-09-22T21:40:19.721 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.721 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t59JFI7qQCkryjRdninYjPBQKe4= 2023-09-22T21:40:19.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t59JFI7qQCkryjRdninYjPBQKe4=', 'Content-Length': '0'} 2023-09-22T21:40:19.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048c06cad7f3d83df-00650e09c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T21:40:19.725 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:40:19.725 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:40:19.725 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.726 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.726 DEBUG:boto:Method: GET 2023-09-22T21:40:19.726 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.726 DEBUG:boto:Data: 2023-09-22T21:40:19.727 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.727 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.727 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.727 DEBUG:boto:Params: {} 2023-09-22T21:40:19.728 DEBUG:boto:Token: None 2023-09-22T21:40:19.728 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.729 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QI2/8jjYQhh+zr59NP++xxR1r98= 2023-09-22T21:40:19.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QI2/8jjYQhh+zr59NP++xxR1r98=', 'Content-Length': '0'} 2023-09-22T21:40:19.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055247c9d4851fbf5-00650e09c3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.778 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:40:19.779 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:40:19.779 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.779 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.779 DEBUG:boto:Method: GET 2023-09-22T21:40:19.779 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.779 DEBUG:boto:Data: 2023-09-22T21:40:19.779 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.779 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.779 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.780 DEBUG:boto:Params: {} 2023-09-22T21:40:19.780 DEBUG:boto:Token: None 2023-09-22T21:40:19.780 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:40:19.780 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QI2/8jjYQhh+zr59NP++xxR1r98= 2023-09-22T21:40:19.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QI2/8jjYQhh+zr59NP++xxR1r98=', 'Content-Length': '0'} 2023-09-22T21:40:19.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000967edd1aebd320b7-00650e09c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.783 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:19.784 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:19.784 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:40:19.784 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:40:19.784 DEBUG:boto:Method: GET 2023-09-22T21:40:19.784 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:40:19.784 DEBUG:boto:Data: 2023-09-22T21:40:19.784 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.784 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.784 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.784 DEBUG:boto:Params: {} 2023-09-22T21:40:19.785 DEBUG:boto:Token: None 2023-09-22T21:40:19.785 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/ 2023-09-22T21:40:19.785 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ= 2023-09-22T21:40:19.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ=', 'Content-Length': '0'} 2023-09-22T21:40:19.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000855e9948f5cafe72-00650e09c3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.790 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:19.790 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:19.790 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:19.791 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:40:19.791 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:40:19.791 DEBUG:boto:Method: GET 2023-09-22T21:40:19.791 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:40:19.791 DEBUG:boto:Data: 2023-09-22T21:40:19.791 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.791 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.792 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.792 DEBUG:boto:Params: {} 2023-09-22T21:40:19.792 DEBUG:boto:Token: None 2023-09-22T21:40:19.792 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/ 2023-09-22T21:40:19.792 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ= 2023-09-22T21:40:19.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ=', 'Content-Length': '0'} 2023-09-22T21:40:19.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e402ab594677cebd-00650e09c3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.846 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:19.847 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:40:19.847 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:40:19.847 DEBUG:boto:Method: HEAD 2023-09-22T21:40:19.847 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:40:19.847 DEBUG:boto:Data: 2023-09-22T21:40:19.847 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.847 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.847 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.847 DEBUG:boto:Params: {} 2023-09-22T21:40:19.848 DEBUG:boto:Token: None 2023-09-22T21:40:19.848 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj57 2023-09-22T21:40:19.848 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:o4JEcZL5vUBJTSrRA9olOKWnPZI= 2023-09-22T21:40:19.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:o4JEcZL5vUBJTSrRA9olOKWnPZI=', 'Content-Length': '0'} 2023-09-22T21:40:19.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007302b1291f70ef10-00650e09c3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.853 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:40:19.853 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:40:19.853 DEBUG:boto:Method: HEAD 2023-09-22T21:40:19.853 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:40:19.853 DEBUG:boto:Data: 2023-09-22T21:40:19.853 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.853 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.854 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.854 DEBUG:boto:Params: {} 2023-09-22T21:40:19.854 DEBUG:boto:Token: None 2023-09-22T21:40:19.854 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj57 2023-09-22T21:40:19.854 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:o4JEcZL5vUBJTSrRA9olOKWnPZI= 2023-09-22T21:40:19.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:o4JEcZL5vUBJTSrRA9olOKWnPZI=', 'Content-Length': '0'} 2023-09-22T21:40:19.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4cb8eb02cdefb48-00650e09c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T21:40:19.858 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:40:19.858 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:40:19.858 DEBUG:boto:Method: GET 2023-09-22T21:40:19.858 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:40:19.858 DEBUG:boto:Data: 2023-09-22T21:40:19.859 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.859 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.859 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.859 DEBUG:boto:Params: {} 2023-09-22T21:40:19.859 DEBUG:boto:Token: None 2023-09-22T21:40:19.859 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj57 2023-09-22T21:40:19.859 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P7uWpBgUmah3kYHfKjq0FJAeCU0= 2023-09-22T21:40:19.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P7uWpBgUmah3kYHfKjq0FJAeCU0=', 'Content-Length': '0'} 2023-09-22T21:40:19.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065ae58081b733ada-00650e09c3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.902 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:40:19.903 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:40:19.903 DEBUG:boto:Method: GET 2023-09-22T21:40:19.903 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:40:19.903 DEBUG:boto:Data: 2023-09-22T21:40:19.903 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.903 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.903 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.903 DEBUG:boto:Params: {} 2023-09-22T21:40:19.903 DEBUG:boto:Token: None 2023-09-22T21:40:19.904 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj57 2023-09-22T21:40:19.904 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P7uWpBgUmah3kYHfKjq0FJAeCU0= 2023-09-22T21:40:19.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P7uWpBgUmah3kYHfKjq0FJAeCU0=', 'Content-Length': '0'} 2023-09-22T21:40:19.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030de01883af5bebb-00650e09c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.907 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:19.907 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:19.907 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:40:19.908 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:40:19.908 DEBUG:boto:Method: GET 2023-09-22T21:40:19.908 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:40:19.908 DEBUG:boto:Data: 2023-09-22T21:40:19.908 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.908 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.908 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.908 DEBUG:boto:Params: {} 2023-09-22T21:40:19.908 DEBUG:boto:Token: None 2023-09-22T21:40:19.909 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/ 2023-09-22T21:40:19.909 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ= 2023-09-22T21:40:19.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ=', 'Content-Length': '0'} 2023-09-22T21:40:19.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072c6430f21a1cd4e-00650e09c3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.913 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:19.914 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:19.914 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:19.914 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:40:19.914 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:40:19.914 DEBUG:boto:Method: GET 2023-09-22T21:40:19.914 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:40:19.915 DEBUG:boto:Data: 2023-09-22T21:40:19.915 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.915 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.915 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.915 DEBUG:boto:Params: {} 2023-09-22T21:40:19.915 DEBUG:boto:Token: None 2023-09-22T21:40:19.916 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/ 2023-09-22T21:40:19.916 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ= 2023-09-22T21:40:19.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eliq/nKXjKpW1M4VhiiUdUeRGYQ=', 'Content-Length': '0'} 2023-09-22T21:40:19.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000858c62e5f1329d03-00650e09c3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.924 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:19.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:40:19.924 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:40:19.924 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:40:19.924 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:19.925 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:19.925 DEBUG:boto:Method: GET 2023-09-22T21:40:19.925 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:19.925 DEBUG:boto:Data: 2023-09-22T21:40:19.925 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.925 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:19.925 DEBUG:boto:Port: 8001 2023-09-22T21:40:19.925 DEBUG:boto:Params: {} 2023-09-22T21:40:19.925 DEBUG:boto:Token: None 2023-09-22T21:40:19.926 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:19.926 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4x//OlWWtxoIhE/xwNuxLNzjJRY= 2023-09-22T21:40:19.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4x//OlWWtxoIhE/xwNuxLNzjJRY=', 'Content-Length': '0'} 2023-09-22T21:40:19.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001c1cec32ed3d0d1-00650e09c3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:19.929 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:40:19.929 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:40:19.929 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:19.929 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:19.929 DEBUG:boto:Method: GET 2023-09-22T21:40:19.929 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:19.930 DEBUG:boto:Data: 2023-09-22T21:40:19.930 DEBUG:boto:Headers: {} 2023-09-22T21:40:19.930 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:19.930 DEBUG:boto:Port: 8000 2023-09-22T21:40:19.930 DEBUG:boto:Params: {} 2023-09-22T21:40:19.930 DEBUG:boto:Token: None 2023-09-22T21:40:19.930 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:19 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:19.930 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4x//OlWWtxoIhE/xwNuxLNzjJRY= 2023-09-22T21:40:19.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4x//OlWWtxoIhE/xwNuxLNzjJRY=', 'Content-Length': '0'} 2023-09-22T21:40:19.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc047d0ba54c1784-00650e09c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.367 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:40:20.367 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:40:20.367 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:20.367 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:20.367 DEBUG:boto:Method: HEAD 2023-09-22T21:40:20.367 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:20.368 DEBUG:boto:Data: 2023-09-22T21:40:20.368 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.368 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:20.368 DEBUG:boto:Port: 8001 2023-09-22T21:40:20.368 DEBUG:boto:Params: {} 2023-09-22T21:40:20.368 DEBUG:boto:Token: None 2023-09-22T21:40:20.368 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:20.369 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XbtyH8nFCsI7+GZruog/6pkb42w= 2023-09-22T21:40:20.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XbtyH8nFCsI7+GZruog/6pkb42w=', 'Content-Length': '0'} 2023-09-22T21:40:20.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7c4b1bf45d43023-00650e09c4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.372 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:40:20.373 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:40:20.373 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:20.373 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:20.373 DEBUG:boto:Method: HEAD 2023-09-22T21:40:20.373 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:20.373 DEBUG:boto:Data: 2023-09-22T21:40:20.373 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.374 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:20.374 DEBUG:boto:Port: 8000 2023-09-22T21:40:20.374 DEBUG:boto:Params: {} 2023-09-22T21:40:20.374 DEBUG:boto:Token: None 2023-09-22T21:40:20.374 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:20.375 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XbtyH8nFCsI7+GZruog/6pkb42w= 2023-09-22T21:40:20.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XbtyH8nFCsI7+GZruog/6pkb42w=', 'Content-Length': '0'} 2023-09-22T21:40:20.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7e4d5ca218fea63-00650e09c4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:40:20.378 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:40:20.378 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:40:20.378 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:20.378 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:20.379 DEBUG:boto:Method: GET 2023-09-22T21:40:20.379 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:20.379 DEBUG:boto:Data: 2023-09-22T21:40:20.379 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.379 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:20.379 DEBUG:boto:Port: 8001 2023-09-22T21:40:20.379 DEBUG:boto:Params: {} 2023-09-22T21:40:20.379 DEBUG:boto:Token: None 2023-09-22T21:40:20.379 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:20.380 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S7oXgKFbUcztieDtEtbCVrV6b/c= 2023-09-22T21:40:20.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S7oXgKFbUcztieDtEtbCVrV6b/c=', 'Content-Length': '0'} 2023-09-22T21:40:20.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a5da25ca926f2fd-00650e09c4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.426 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:40:20.427 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:40:20.427 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:20.427 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:20.427 DEBUG:boto:Method: GET 2023-09-22T21:40:20.427 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:20.427 DEBUG:boto:Data: 2023-09-22T21:40:20.427 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.427 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:20.428 DEBUG:boto:Port: 8000 2023-09-22T21:40:20.428 DEBUG:boto:Params: {} 2023-09-22T21:40:20.428 DEBUG:boto:Token: None 2023-09-22T21:40:20.428 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:40:20.428 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S7oXgKFbUcztieDtEtbCVrV6b/c= 2023-09-22T21:40:20.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S7oXgKFbUcztieDtEtbCVrV6b/c=', 'Content-Length': '0'} 2023-09-22T21:40:20.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006530ec6b4920c7f7-00650e09c4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.432 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:20.432 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:20.432 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:40:20.432 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:40:20.432 DEBUG:boto:Method: GET 2023-09-22T21:40:20.432 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:40:20.432 DEBUG:boto:Data: 2023-09-22T21:40:20.432 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.433 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:20.433 DEBUG:boto:Port: 8001 2023-09-22T21:40:20.433 DEBUG:boto:Params: {} 2023-09-22T21:40:20.433 DEBUG:boto:Token: None 2023-09-22T21:40:20.433 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/ 2023-09-22T21:40:20.433 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Whbokfexzg14G3sfq1hsAhnJzq8= 2023-09-22T21:40:20.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Whbokfexzg14G3sfq1hsAhnJzq8=', 'Content-Length': '0'} 2023-09-22T21:40:20.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002282cb0ee7b7c462-00650e09c4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.438 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:20.439 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:20.439 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:20.439 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:40:20.439 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:40:20.439 DEBUG:boto:Method: GET 2023-09-22T21:40:20.439 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:40:20.439 DEBUG:boto:Data: 2023-09-22T21:40:20.439 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.440 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:20.440 DEBUG:boto:Port: 8000 2023-09-22T21:40:20.440 DEBUG:boto:Params: {} 2023-09-22T21:40:20.440 DEBUG:boto:Token: None 2023-09-22T21:40:20.440 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/ 2023-09-22T21:40:20.440 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Whbokfexzg14G3sfq1hsAhnJzq8= 2023-09-22T21:40:20.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Whbokfexzg14G3sfq1hsAhnJzq8=', 'Content-Length': '0'} 2023-09-22T21:40:20.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009db588777b51065a-00650e09c4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.447 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:20.447 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:40:20.447 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:40:20.447 DEBUG:boto:Method: HEAD 2023-09-22T21:40:20.447 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:40:20.447 DEBUG:boto:Data: 2023-09-22T21:40:20.448 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.448 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:20.448 DEBUG:boto:Port: 8001 2023-09-22T21:40:20.448 DEBUG:boto:Params: {} 2023-09-22T21:40:20.448 DEBUG:boto:Token: None 2023-09-22T21:40:20.448 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj68 2023-09-22T21:40:20.448 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oChUC2pxrGrEm+JjMjJZCYJdJ8I= 2023-09-22T21:40:20.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oChUC2pxrGrEm+JjMjJZCYJdJ8I=', 'Content-Length': '0'} 2023-09-22T21:40:20.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080e863bf2e39e1fa-00650e09c4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.451 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:40:20.452 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:40:20.452 DEBUG:boto:Method: HEAD 2023-09-22T21:40:20.452 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:40:20.452 DEBUG:boto:Data: 2023-09-22T21:40:20.452 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.452 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:20.452 DEBUG:boto:Port: 8000 2023-09-22T21:40:20.452 DEBUG:boto:Params: {} 2023-09-22T21:40:20.453 DEBUG:boto:Token: None 2023-09-22T21:40:20.453 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj68 2023-09-22T21:40:20.453 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oChUC2pxrGrEm+JjMjJZCYJdJ8I= 2023-09-22T21:40:20.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oChUC2pxrGrEm+JjMjJZCYJdJ8I=', 'Content-Length': '0'} 2023-09-22T21:40:20.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e0b53507ae8a73c-00650e09c4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:40:20.457 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:40:20.457 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:40:20.457 DEBUG:boto:Method: GET 2023-09-22T21:40:20.457 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:40:20.457 DEBUG:boto:Data: 2023-09-22T21:40:20.457 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.457 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:20.457 DEBUG:boto:Port: 8001 2023-09-22T21:40:20.458 DEBUG:boto:Params: {} 2023-09-22T21:40:20.458 DEBUG:boto:Token: None 2023-09-22T21:40:20.458 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj68 2023-09-22T21:40:20.458 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y5YLxTiaCFwwv6BiK66N1cGqhy0= 2023-09-22T21:40:20.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y5YLxTiaCFwwv6BiK66N1cGqhy0=', 'Content-Length': '0'} 2023-09-22T21:40:20.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090c76ea3674605cf-00650e09c4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.502 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:40:20.520 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:40:20.520 DEBUG:boto:Method: GET 2023-09-22T21:40:20.520 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:40:20.521 DEBUG:boto:Data: 2023-09-22T21:40:20.521 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.521 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:20.521 DEBUG:boto:Port: 8000 2023-09-22T21:40:20.521 DEBUG:boto:Params: {} 2023-09-22T21:40:20.521 DEBUG:boto:Token: None 2023-09-22T21:40:20.521 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj68 2023-09-22T21:40:20.522 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y5YLxTiaCFwwv6BiK66N1cGqhy0= 2023-09-22T21:40:20.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y5YLxTiaCFwwv6BiK66N1cGqhy0=', 'Content-Length': '0'} 2023-09-22T21:40:20.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094baebfe24cd9585-00650e09c4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.525 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:20.525 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:20.525 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:40:20.525 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:40:20.526 DEBUG:boto:Method: GET 2023-09-22T21:40:20.526 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:40:20.526 DEBUG:boto:Data: 2023-09-22T21:40:20.526 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.526 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:20.526 DEBUG:boto:Port: 8001 2023-09-22T21:40:20.526 DEBUG:boto:Params: {} 2023-09-22T21:40:20.526 DEBUG:boto:Token: None 2023-09-22T21:40:20.527 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/ 2023-09-22T21:40:20.527 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Whbokfexzg14G3sfq1hsAhnJzq8= 2023-09-22T21:40:20.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Whbokfexzg14G3sfq1hsAhnJzq8=', 'Content-Length': '0'} 2023-09-22T21:40:20.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006560bbb64e0614d6-00650e09c4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.574 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:20.575 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:20.575 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:20.575 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:40:20.575 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:40:20.575 DEBUG:boto:Method: GET 2023-09-22T21:40:20.575 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:40:20.576 DEBUG:boto:Data: 2023-09-22T21:40:20.576 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.576 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:20.576 DEBUG:boto:Port: 8000 2023-09-22T21:40:20.576 DEBUG:boto:Params: {} 2023-09-22T21:40:20.576 DEBUG:boto:Token: None 2023-09-22T21:40:20.576 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/ 2023-09-22T21:40:20.576 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Whbokfexzg14G3sfq1hsAhnJzq8= 2023-09-22T21:40:20.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Whbokfexzg14G3sfq1hsAhnJzq8=', 'Content-Length': '0'} 2023-09-22T21:40:20.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f0de21a94b894e7-00650e09c4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.584 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:20.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T21:40:20.584 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:40:20.584 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:40:20.584 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.585 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.585 DEBUG:boto:Method: GET 2023-09-22T21:40:20.585 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.585 DEBUG:boto:Data: 2023-09-22T21:40:20.585 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.585 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:20.585 DEBUG:boto:Port: 8001 2023-09-22T21:40:20.585 DEBUG:boto:Params: {} 2023-09-22T21:40:20.585 DEBUG:boto:Token: None 2023-09-22T21:40:20.586 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.586 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uMlnO7ppSoo1B4JNQP+9ODxm08M= 2023-09-22T21:40:20.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uMlnO7ppSoo1B4JNQP+9ODxm08M=', 'Content-Length': '0'} 2023-09-22T21:40:20.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d925676f68c163f4-00650e09c4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.589 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:40:20.589 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:40:20.589 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.589 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.589 DEBUG:boto:Method: GET 2023-09-22T21:40:20.589 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.590 DEBUG:boto:Data: 2023-09-22T21:40:20.590 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.590 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:20.590 DEBUG:boto:Port: 8000 2023-09-22T21:40:20.590 DEBUG:boto:Params: {} 2023-09-22T21:40:20.590 DEBUG:boto:Token: None 2023-09-22T21:40:20.590 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.590 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uMlnO7ppSoo1B4JNQP+9ODxm08M= 2023-09-22T21:40:20.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uMlnO7ppSoo1B4JNQP+9ODxm08M=', 'Content-Length': '0'} 2023-09-22T21:40:20.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007023bcd4b3f70e31-00650e09c4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.634 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:40:20.635 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:40:20.635 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.635 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.635 DEBUG:boto:Method: HEAD 2023-09-22T21:40:20.635 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.635 DEBUG:boto:Data: 2023-09-22T21:40:20.635 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.635 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:20.635 DEBUG:boto:Port: 8001 2023-09-22T21:40:20.636 DEBUG:boto:Params: {} 2023-09-22T21:40:20.636 DEBUG:boto:Token: None 2023-09-22T21:40:20.636 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.636 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GrIWRK58xyBvKazKjuaLKYQVeE8= 2023-09-22T21:40:20.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GrIWRK58xyBvKazKjuaLKYQVeE8=', 'Content-Length': '0'} 2023-09-22T21:40:20.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc1b7e4631940624-00650e09c4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.639 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:40:20.639 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:40:20.640 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.640 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.640 DEBUG:boto:Method: HEAD 2023-09-22T21:40:20.640 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.640 DEBUG:boto:Data: 2023-09-22T21:40:20.640 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.640 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:20.640 DEBUG:boto:Port: 8000 2023-09-22T21:40:20.640 DEBUG:boto:Params: {} 2023-09-22T21:40:20.640 DEBUG:boto:Token: None 2023-09-22T21:40:20.641 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.641 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GrIWRK58xyBvKazKjuaLKYQVeE8= 2023-09-22T21:40:20.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GrIWRK58xyBvKazKjuaLKYQVeE8=', 'Content-Length': '0'} 2023-09-22T21:40:20.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1e8d9ad14b63d47-00650e09c4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T21:40:20.644 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:40:20.644 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:40:20.644 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.644 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.644 DEBUG:boto:Method: GET 2023-09-22T21:40:20.644 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.644 DEBUG:boto:Data: 2023-09-22T21:40:20.645 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.645 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:20.645 DEBUG:boto:Port: 8001 2023-09-22T21:40:20.645 DEBUG:boto:Params: {} 2023-09-22T21:40:20.645 DEBUG:boto:Token: None 2023-09-22T21:40:20.645 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.645 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uMlnO7ppSoo1B4JNQP+9ODxm08M= 2023-09-22T21:40:20.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uMlnO7ppSoo1B4JNQP+9ODxm08M=', 'Content-Length': '0'} 2023-09-22T21:40:20.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084ae11b9ed65a167-00650e09c4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.690 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:40:20.690 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:40:20.691 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.691 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.691 DEBUG:boto:Method: GET 2023-09-22T21:40:20.691 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.691 DEBUG:boto:Data: 2023-09-22T21:40:20.691 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.691 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:20.691 DEBUG:boto:Port: 8000 2023-09-22T21:40:20.691 DEBUG:boto:Params: {} 2023-09-22T21:40:20.692 DEBUG:boto:Token: None 2023-09-22T21:40:20.692 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:40:20.692 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uMlnO7ppSoo1B4JNQP+9ODxm08M= 2023-09-22T21:40:20.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uMlnO7ppSoo1B4JNQP+9ODxm08M=', 'Content-Length': '0'} 2023-09-22T21:40:20.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024378ac1e40fc8e0-00650e09c4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.695 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:20.695 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:20.696 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:40:20.696 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:40:20.696 DEBUG:boto:Method: GET 2023-09-22T21:40:20.696 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:40:20.696 DEBUG:boto:Data: 2023-09-22T21:40:20.696 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.696 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:20.696 DEBUG:boto:Port: 8001 2023-09-22T21:40:20.696 DEBUG:boto:Params: {} 2023-09-22T21:40:20.696 DEBUG:boto:Token: None 2023-09-22T21:40:20.697 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/ 2023-09-22T21:40:20.697 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Whbokfexzg14G3sfq1hsAhnJzq8= 2023-09-22T21:40:20.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Whbokfexzg14G3sfq1hsAhnJzq8=', 'Content-Length': '0'} 2023-09-22T21:40:20.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000009a1a3387f42fa7-00650e09c4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.702 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:20.702 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:20.703 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:20.703 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:40:20.703 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:40:20.703 DEBUG:boto:Method: GET 2023-09-22T21:40:20.703 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:40:20.703 DEBUG:boto:Data: 2023-09-22T21:40:20.703 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.703 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:20.703 DEBUG:boto:Port: 8000 2023-09-22T21:40:20.704 DEBUG:boto:Params: {} 2023-09-22T21:40:20.704 DEBUG:boto:Token: None 2023-09-22T21:40:20.704 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/ 2023-09-22T21:40:20.704 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Whbokfexzg14G3sfq1hsAhnJzq8= 2023-09-22T21:40:20.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Whbokfexzg14G3sfq1hsAhnJzq8=', 'Content-Length': '0'} 2023-09-22T21:40:20.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000097d10e3bdcb97a4-00650e09c4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.711 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:20.711 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:40:20.713 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:40:20.713 DEBUG:boto:Method: HEAD 2023-09-22T21:40:20.713 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:40:20.713 DEBUG:boto:Data: 2023-09-22T21:40:20.713 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.713 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:20.713 DEBUG:boto:Port: 8001 2023-09-22T21:40:20.714 DEBUG:boto:Params: {} 2023-09-22T21:40:20.714 DEBUG:boto:Token: None 2023-09-22T21:40:20.714 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj76 2023-09-22T21:40:20.714 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:I/cZO+qEJ66/Km0uQu/HNb5AqRc= 2023-09-22T21:40:20.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:I/cZO+qEJ66/Km0uQu/HNb5AqRc=', 'Content-Length': '0'} 2023-09-22T21:40:20.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f64ecf7806f62f78-00650e09c4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.717 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:40:20.717 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:40:20.717 DEBUG:boto:Method: HEAD 2023-09-22T21:40:20.717 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:40:20.717 DEBUG:boto:Data: 2023-09-22T21:40:20.718 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.718 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:20.718 DEBUG:boto:Port: 8000 2023-09-22T21:40:20.718 DEBUG:boto:Params: {} 2023-09-22T21:40:20.718 DEBUG:boto:Token: None 2023-09-22T21:40:20.718 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj76 2023-09-22T21:40:20.718 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:I/cZO+qEJ66/Km0uQu/HNb5AqRc= 2023-09-22T21:40:20.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:I/cZO+qEJ66/Km0uQu/HNb5AqRc=', 'Content-Length': '0'} 2023-09-22T21:40:20.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000212c99ac6988150a-00650e09c4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T21:40:20.721 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:40:20.722 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:40:20.722 DEBUG:boto:Method: GET 2023-09-22T21:40:20.722 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:40:20.722 DEBUG:boto:Data: 2023-09-22T21:40:20.722 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.722 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:20.722 DEBUG:boto:Port: 8001 2023-09-22T21:40:20.722 DEBUG:boto:Params: {} 2023-09-22T21:40:20.723 DEBUG:boto:Token: None 2023-09-22T21:40:20.723 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj76 2023-09-22T21:40:20.723 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2O1CqXj4UR3+E1126rkIQMVlRe8= 2023-09-22T21:40:20.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2O1CqXj4UR3+E1126rkIQMVlRe8=', 'Content-Length': '0'} 2023-09-22T21:40:20.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d13eec17af15e17-00650e09c4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.766 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:40:20.767 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:40:20.767 DEBUG:boto:Method: GET 2023-09-22T21:40:20.767 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:40:20.767 DEBUG:boto:Data: 2023-09-22T21:40:20.767 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.767 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:20.767 DEBUG:boto:Port: 8000 2023-09-22T21:40:20.768 DEBUG:boto:Params: {} 2023-09-22T21:40:20.768 DEBUG:boto:Token: None 2023-09-22T21:40:20.768 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj76 2023-09-22T21:40:20.768 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2O1CqXj4UR3+E1126rkIQMVlRe8= 2023-09-22T21:40:20.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2O1CqXj4UR3+E1126rkIQMVlRe8=', 'Content-Length': '0'} 2023-09-22T21:40:20.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db13940ad86ddd62-00650e09c4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.771 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:20.771 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:20.771 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:40:20.772 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:40:20.772 DEBUG:boto:Method: GET 2023-09-22T21:40:20.772 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:40:20.772 DEBUG:boto:Data: 2023-09-22T21:40:20.772 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.772 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:20.772 DEBUG:boto:Port: 8001 2023-09-22T21:40:20.772 DEBUG:boto:Params: {} 2023-09-22T21:40:20.772 DEBUG:boto:Token: None 2023-09-22T21:40:20.773 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/ 2023-09-22T21:40:20.773 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Whbokfexzg14G3sfq1hsAhnJzq8= 2023-09-22T21:40:20.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Whbokfexzg14G3sfq1hsAhnJzq8=', 'Content-Length': '0'} 2023-09-22T21:40:20.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c19057bed0b6ee8-00650e09c4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.818 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:20.819 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:20.819 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:20.819 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:40:20.819 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:40:20.819 DEBUG:boto:Method: GET 2023-09-22T21:40:20.819 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:40:20.819 DEBUG:boto:Data: 2023-09-22T21:40:20.820 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.820 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:20.820 DEBUG:boto:Port: 8000 2023-09-22T21:40:20.820 DEBUG:boto:Params: {} 2023-09-22T21:40:20.820 DEBUG:boto:Token: None 2023-09-22T21:40:20.820 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/ 2023-09-22T21:40:20.820 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Whbokfexzg14G3sfq1hsAhnJzq8= 2023-09-22T21:40:20.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Whbokfexzg14G3sfq1hsAhnJzq8=', 'Content-Length': '0'} 2023-09-22T21:40:20.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000592ea6e626e59434-00650e09c4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.827 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:20.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:40:20.828 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:40:20.828 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:40:20.828 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.828 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.828 DEBUG:boto:Method: GET 2023-09-22T21:40:20.828 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.828 DEBUG:boto:Data: 2023-09-22T21:40:20.828 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.829 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:20.829 DEBUG:boto:Port: 8001 2023-09-22T21:40:20.829 DEBUG:boto:Params: {} 2023-09-22T21:40:20.829 DEBUG:boto:Token: None 2023-09-22T21:40:20.829 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.829 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2klW7YUtlT+xM8UX0pG+U3nZ4BM= 2023-09-22T21:40:20.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2klW7YUtlT+xM8UX0pG+U3nZ4BM=', 'Content-Length': '0'} 2023-09-22T21:40:20.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2c21110fad76676-00650e09c4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.834 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:40:20.834 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:40:20.834 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.834 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.834 DEBUG:boto:Method: GET 2023-09-22T21:40:20.834 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.834 DEBUG:boto:Data: 2023-09-22T21:40:20.835 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.835 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:20.835 DEBUG:boto:Port: 8000 2023-09-22T21:40:20.835 DEBUG:boto:Params: {} 2023-09-22T21:40:20.835 DEBUG:boto:Token: None 2023-09-22T21:40:20.835 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.835 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2klW7YUtlT+xM8UX0pG+U3nZ4BM= 2023-09-22T21:40:20.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2klW7YUtlT+xM8UX0pG+U3nZ4BM=', 'Content-Length': '0'} 2023-09-22T21:40:20.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078745d67dd70fbdc-00650e09c4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.839 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:40:20.839 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:40:20.839 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.839 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.839 DEBUG:boto:Method: HEAD 2023-09-22T21:40:20.840 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.840 DEBUG:boto:Data: 2023-09-22T21:40:20.840 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.840 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:20.840 DEBUG:boto:Port: 8001 2023-09-22T21:40:20.840 DEBUG:boto:Params: {} 2023-09-22T21:40:20.840 DEBUG:boto:Token: None 2023-09-22T21:40:20.840 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.840 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F8QsGz4MLvJpUC0zY9hhJ1qrqf0= 2023-09-22T21:40:20.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F8QsGz4MLvJpUC0zY9hhJ1qrqf0=', 'Content-Length': '0'} 2023-09-22T21:40:20.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f595ff62bf1f718-00650e09c4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.844 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:40:20.844 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:40:20.844 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.844 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.844 DEBUG:boto:Method: HEAD 2023-09-22T21:40:20.844 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.844 DEBUG:boto:Data: 2023-09-22T21:40:20.844 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.844 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:20.845 DEBUG:boto:Port: 8000 2023-09-22T21:40:20.845 DEBUG:boto:Params: {} 2023-09-22T21:40:20.845 DEBUG:boto:Token: None 2023-09-22T21:40:20.845 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.845 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F8QsGz4MLvJpUC0zY9hhJ1qrqf0= 2023-09-22T21:40:20.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F8QsGz4MLvJpUC0zY9hhJ1qrqf0=', 'Content-Length': '0'} 2023-09-22T21:40:20.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecd9af26db915b5b-00650e09c4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:40:20.848 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:40:20.848 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:40:20.848 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.848 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.848 DEBUG:boto:Method: GET 2023-09-22T21:40:20.848 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.848 DEBUG:boto:Data: 2023-09-22T21:40:20.849 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.849 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:20.849 DEBUG:boto:Port: 8001 2023-09-22T21:40:20.849 DEBUG:boto:Params: {} 2023-09-22T21:40:20.849 DEBUG:boto:Token: None 2023-09-22T21:40:20.849 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.849 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2klW7YUtlT+xM8UX0pG+U3nZ4BM= 2023-09-22T21:40:20.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2klW7YUtlT+xM8UX0pG+U3nZ4BM=', 'Content-Length': '0'} 2023-09-22T21:40:20.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da6709255b7a45da-00650e09c4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.894 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:40:20.895 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:40:20.895 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.895 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.895 DEBUG:boto:Method: GET 2023-09-22T21:40:20.895 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.895 DEBUG:boto:Data: 2023-09-22T21:40:20.895 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.895 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:20.895 DEBUG:boto:Port: 8000 2023-09-22T21:40:20.896 DEBUG:boto:Params: {} 2023-09-22T21:40:20.896 DEBUG:boto:Token: None 2023-09-22T21:40:20.896 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:40:20.896 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2klW7YUtlT+xM8UX0pG+U3nZ4BM= 2023-09-22T21:40:20.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2klW7YUtlT+xM8UX0pG+U3nZ4BM=', 'Content-Length': '0'} 2023-09-22T21:40:20.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bda8583377cea91d-00650e09c4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.899 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:20.900 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:20.900 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:40:20.900 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:40:20.900 DEBUG:boto:Method: GET 2023-09-22T21:40:20.900 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:40:20.900 DEBUG:boto:Data: 2023-09-22T21:40:20.900 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.900 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:20.901 DEBUG:boto:Port: 8001 2023-09-22T21:40:20.901 DEBUG:boto:Params: {} 2023-09-22T21:40:20.901 DEBUG:boto:Token: None 2023-09-22T21:40:20.901 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/ 2023-09-22T21:40:20.901 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Whbokfexzg14G3sfq1hsAhnJzq8= 2023-09-22T21:40:20.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Whbokfexzg14G3sfq1hsAhnJzq8=', 'Content-Length': '0'} 2023-09-22T21:40:20.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd7f34179e70caa0-00650e09c4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.950 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:20.951 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:20.951 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:20.951 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:40:20.951 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:40:20.951 DEBUG:boto:Method: GET 2023-09-22T21:40:20.951 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:40:20.951 DEBUG:boto:Data: 2023-09-22T21:40:20.952 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.952 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:20.952 DEBUG:boto:Port: 8000 2023-09-22T21:40:20.952 DEBUG:boto:Params: {} 2023-09-22T21:40:20.952 DEBUG:boto:Token: None 2023-09-22T21:40:20.952 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/ 2023-09-22T21:40:20.952 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Whbokfexzg14G3sfq1hsAhnJzq8= 2023-09-22T21:40:20.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Whbokfexzg14G3sfq1hsAhnJzq8=', 'Content-Length': '0'} 2023-09-22T21:40:20.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085c1b8064b2c9879-00650e09c4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.959 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:20.959 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:40:20.959 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:40:20.959 DEBUG:boto:Method: HEAD 2023-09-22T21:40:20.960 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:40:20.960 DEBUG:boto:Data: 2023-09-22T21:40:20.960 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.960 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:20.960 DEBUG:boto:Port: 8001 2023-09-22T21:40:20.960 DEBUG:boto:Params: {} 2023-09-22T21:40:20.960 DEBUG:boto:Token: None 2023-09-22T21:40:20.960 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj85 2023-09-22T21:40:20.961 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x6ym+D8sdYM2vM4bSA9YeXyw+RY= 2023-09-22T21:40:20.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x6ym+D8sdYM2vM4bSA9YeXyw+RY=', 'Content-Length': '0'} 2023-09-22T21:40:20.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4b543f4de2e784f-00650e09c4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.963 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:40:20.964 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:40:20.964 DEBUG:boto:Method: HEAD 2023-09-22T21:40:20.964 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:40:20.964 DEBUG:boto:Data: 2023-09-22T21:40:20.964 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.964 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:20.964 DEBUG:boto:Port: 8000 2023-09-22T21:40:20.964 DEBUG:boto:Params: {} 2023-09-22T21:40:20.964 DEBUG:boto:Token: None 2023-09-22T21:40:20.965 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj85 2023-09-22T21:40:20.965 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x6ym+D8sdYM2vM4bSA9YeXyw+RY= 2023-09-22T21:40:20.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x6ym+D8sdYM2vM4bSA9YeXyw+RY=', 'Content-Length': '0'} 2023-09-22T21:40:20.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef0c53a8e8637130-00650e09c4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:20.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:40:20.968 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:40:20.968 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:40:20.968 DEBUG:boto:Method: GET 2023-09-22T21:40:20.968 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:40:20.968 DEBUG:boto:Data: 2023-09-22T21:40:20.968 DEBUG:boto:Headers: {} 2023-09-22T21:40:20.968 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:20.968 DEBUG:boto:Port: 8001 2023-09-22T21:40:20.969 DEBUG:boto:Params: {} 2023-09-22T21:40:20.969 DEBUG:boto:Token: None 2023-09-22T21:40:20.969 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:20 GMT /ffzdqu-59/obj85 2023-09-22T21:40:20.969 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zXoZve1D7Ic3sHpjcZFmK38SDl8= 2023-09-22T21:40:20.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zXoZve1D7Ic3sHpjcZFmK38SDl8=', 'Content-Length': '0'} 2023-09-22T21:40:20.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6f42e9dd674231e-00650e09c4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.014 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:40:21.025 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:40:21.025 DEBUG:boto:Method: GET 2023-09-22T21:40:21.025 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:40:21.025 DEBUG:boto:Data: 2023-09-22T21:40:21.025 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.026 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:21.026 DEBUG:boto:Port: 8000 2023-09-22T21:40:21.026 DEBUG:boto:Params: {} 2023-09-22T21:40:21.026 DEBUG:boto:Token: None 2023-09-22T21:40:21.026 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/obj85 2023-09-22T21:40:21.026 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:o8ju3sw7tF4AVoyV2WPcF6X5yBc= 2023-09-22T21:40:21.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:o8ju3sw7tF4AVoyV2WPcF6X5yBc=', 'Content-Length': '0'} 2023-09-22T21:40:21.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df95f8ac5f9d9d31-00650e09c5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.030 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:21.030 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:21.030 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:40:21.030 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:40:21.030 DEBUG:boto:Method: GET 2023-09-22T21:40:21.030 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:40:21.030 DEBUG:boto:Data: 2023-09-22T21:40:21.030 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.031 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:21.031 DEBUG:boto:Port: 8001 2023-09-22T21:40:21.031 DEBUG:boto:Params: {} 2023-09-22T21:40:21.031 DEBUG:boto:Token: None 2023-09-22T21:40:21.031 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/ 2023-09-22T21:40:21.031 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Sx+Ye+aIdwr2XiYA2GdrSvNCOdU= 2023-09-22T21:40:21.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Sx+Ye+aIdwr2XiYA2GdrSvNCOdU=', 'Content-Length': '0'} 2023-09-22T21:40:21.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb0b19c31f67fe98-00650e09c5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.036 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:21.037 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:21.037 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:21.037 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:40:21.037 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:40:21.037 DEBUG:boto:Method: GET 2023-09-22T21:40:21.037 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:40:21.038 DEBUG:boto:Data: 2023-09-22T21:40:21.038 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.038 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:21.038 DEBUG:boto:Port: 8000 2023-09-22T21:40:21.038 DEBUG:boto:Params: {} 2023-09-22T21:40:21.038 DEBUG:boto:Token: None 2023-09-22T21:40:21.038 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/ 2023-09-22T21:40:21.038 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Sx+Ye+aIdwr2XiYA2GdrSvNCOdU= 2023-09-22T21:40:21.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Sx+Ye+aIdwr2XiYA2GdrSvNCOdU=', 'Content-Length': '0'} 2023-09-22T21:40:21.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009300a3b451f71201-00650e09c5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.045 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:21.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T21:40:21.045 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:40:21.046 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:40:21.046 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.047 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.047 DEBUG:boto:Method: GET 2023-09-22T21:40:21.047 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.047 DEBUG:boto:Data: 2023-09-22T21:40:21.047 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.047 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:21.047 DEBUG:boto:Port: 8001 2023-09-22T21:40:21.048 DEBUG:boto:Params: {} 2023-09-22T21:40:21.048 DEBUG:boto:Token: None 2023-09-22T21:40:21.048 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.048 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:82cjLo/I+pe/h+PxK8G5ZGLYk8E= 2023-09-22T21:40:21.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:82cjLo/I+pe/h+PxK8G5ZGLYk8E=', 'Content-Length': '0'} 2023-09-22T21:40:21.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2813291163b84ca-00650e09c5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.052 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:40:21.052 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:40:21.052 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.053 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.053 DEBUG:boto:Method: GET 2023-09-22T21:40:21.053 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.053 DEBUG:boto:Data: 2023-09-22T21:40:21.053 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.053 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:21.053 DEBUG:boto:Port: 8000 2023-09-22T21:40:21.053 DEBUG:boto:Params: {} 2023-09-22T21:40:21.053 DEBUG:boto:Token: None 2023-09-22T21:40:21.053 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.054 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:82cjLo/I+pe/h+PxK8G5ZGLYk8E= 2023-09-22T21:40:21.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:82cjLo/I+pe/h+PxK8G5ZGLYk8E=', 'Content-Length': '0'} 2023-09-22T21:40:21.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d99f7cb084ebcc3b-00650e09c5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.057 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:40:21.057 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:40:21.057 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.057 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.058 DEBUG:boto:Method: HEAD 2023-09-22T21:40:21.058 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.058 DEBUG:boto:Data: 2023-09-22T21:40:21.058 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.058 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:21.058 DEBUG:boto:Port: 8001 2023-09-22T21:40:21.058 DEBUG:boto:Params: {} 2023-09-22T21:40:21.058 DEBUG:boto:Token: None 2023-09-22T21:40:21.059 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.059 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYnPJ+rOaA69aYhdv0yA0wXzGjM= 2023-09-22T21:40:21.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYnPJ+rOaA69aYhdv0yA0wXzGjM=', 'Content-Length': '0'} 2023-09-22T21:40:21.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dba1b5cd3cca27ad-00650e09c5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.062 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:40:21.062 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:40:21.062 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.062 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.062 DEBUG:boto:Method: HEAD 2023-09-22T21:40:21.062 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.062 DEBUG:boto:Data: 2023-09-22T21:40:21.063 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.063 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:21.063 DEBUG:boto:Port: 8000 2023-09-22T21:40:21.063 DEBUG:boto:Params: {} 2023-09-22T21:40:21.063 DEBUG:boto:Token: None 2023-09-22T21:40:21.063 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.063 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYnPJ+rOaA69aYhdv0yA0wXzGjM= 2023-09-22T21:40:21.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYnPJ+rOaA69aYhdv0yA0wXzGjM=', 'Content-Length': '0'} 2023-09-22T21:40:21.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bb7a4fb5c83b776-00650e09c5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T21:40:21.066 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:40:21.066 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:40:21.066 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.067 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.067 DEBUG:boto:Method: GET 2023-09-22T21:40:21.067 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.067 DEBUG:boto:Data: 2023-09-22T21:40:21.067 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.067 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:21.067 DEBUG:boto:Port: 8001 2023-09-22T21:40:21.067 DEBUG:boto:Params: {} 2023-09-22T21:40:21.067 DEBUG:boto:Token: None 2023-09-22T21:40:21.067 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.068 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:82cjLo/I+pe/h+PxK8G5ZGLYk8E= 2023-09-22T21:40:21.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:82cjLo/I+pe/h+PxK8G5ZGLYk8E=', 'Content-Length': '0'} 2023-09-22T21:40:21.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c1222b4c500b3b4-00650e09c5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.114 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:40:21.114 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:40:21.115 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.115 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.115 DEBUG:boto:Method: GET 2023-09-22T21:40:21.115 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.115 DEBUG:boto:Data: 2023-09-22T21:40:21.115 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.115 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:21.115 DEBUG:boto:Port: 8000 2023-09-22T21:40:21.115 DEBUG:boto:Params: {} 2023-09-22T21:40:21.116 DEBUG:boto:Token: None 2023-09-22T21:40:21.116 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:40:21.116 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:82cjLo/I+pe/h+PxK8G5ZGLYk8E= 2023-09-22T21:40:21.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:82cjLo/I+pe/h+PxK8G5ZGLYk8E=', 'Content-Length': '0'} 2023-09-22T21:40:21.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069961a8754f87491-00650e09c5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.119 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:21.119 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:21.120 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:40:21.120 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:40:21.121 DEBUG:boto:Method: GET 2023-09-22T21:40:21.121 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:40:21.121 DEBUG:boto:Data: 2023-09-22T21:40:21.121 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.121 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:21.121 DEBUG:boto:Port: 8001 2023-09-22T21:40:21.121 DEBUG:boto:Params: {} 2023-09-22T21:40:21.122 DEBUG:boto:Token: None 2023-09-22T21:40:21.122 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/ 2023-09-22T21:40:21.122 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Sx+Ye+aIdwr2XiYA2GdrSvNCOdU= 2023-09-22T21:40:21.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Sx+Ye+aIdwr2XiYA2GdrSvNCOdU=', 'Content-Length': '0'} 2023-09-22T21:40:21.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a616a6029981fbc6-00650e09c5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.127 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:21.127 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:21.127 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:21.127 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:40:21.128 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:40:21.128 DEBUG:boto:Method: GET 2023-09-22T21:40:21.128 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:40:21.128 DEBUG:boto:Data: 2023-09-22T21:40:21.128 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.129 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:21.129 DEBUG:boto:Port: 8000 2023-09-22T21:40:21.129 DEBUG:boto:Params: {} 2023-09-22T21:40:21.129 DEBUG:boto:Token: None 2023-09-22T21:40:21.129 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/ 2023-09-22T21:40:21.129 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Sx+Ye+aIdwr2XiYA2GdrSvNCOdU= 2023-09-22T21:40:21.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Sx+Ye+aIdwr2XiYA2GdrSvNCOdU=', 'Content-Length': '0'} 2023-09-22T21:40:21.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022eaec804eee841c-00650e09c5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.135 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:21.136 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:40:21.136 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:40:21.136 DEBUG:boto:Method: HEAD 2023-09-22T21:40:21.136 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:40:21.136 DEBUG:boto:Data: 2023-09-22T21:40:21.136 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.137 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:21.137 DEBUG:boto:Port: 8001 2023-09-22T21:40:21.137 DEBUG:boto:Params: {} 2023-09-22T21:40:21.137 DEBUG:boto:Token: None 2023-09-22T21:40:21.137 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/obj95 2023-09-22T21:40:21.137 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QiwbzUDF+EtF/7rHnYD+doYhedw= 2023-09-22T21:40:21.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QiwbzUDF+EtF/7rHnYD+doYhedw=', 'Content-Length': '0'} 2023-09-22T21:40:21.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8d0bed6be01f9e9-00650e09c5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.140 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:40:21.140 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:40:21.140 DEBUG:boto:Method: HEAD 2023-09-22T21:40:21.140 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:40:21.140 DEBUG:boto:Data: 2023-09-22T21:40:21.141 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.141 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:21.141 DEBUG:boto:Port: 8000 2023-09-22T21:40:21.141 DEBUG:boto:Params: {} 2023-09-22T21:40:21.141 DEBUG:boto:Token: None 2023-09-22T21:40:21.141 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/obj95 2023-09-22T21:40:21.141 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QiwbzUDF+EtF/7rHnYD+doYhedw= 2023-09-22T21:40:21.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QiwbzUDF+EtF/7rHnYD+doYhedw=', 'Content-Length': '0'} 2023-09-22T21:40:21.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009beacba238156ff2-00650e09c5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T21:40:21.144 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:40:21.145 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:40:21.145 DEBUG:boto:Method: GET 2023-09-22T21:40:21.145 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:40:21.145 DEBUG:boto:Data: 2023-09-22T21:40:21.145 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.145 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:21.145 DEBUG:boto:Port: 8001 2023-09-22T21:40:21.145 DEBUG:boto:Params: {} 2023-09-22T21:40:21.145 DEBUG:boto:Token: None 2023-09-22T21:40:21.146 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/obj95 2023-09-22T21:40:21.146 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RqtOOL93te07IMr6SPTdoAB6xkM= 2023-09-22T21:40:21.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RqtOOL93te07IMr6SPTdoAB6xkM=', 'Content-Length': '0'} 2023-09-22T21:40:21.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000837da3e34d30db08-00650e09c5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.194 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:40:21.195 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:40:21.195 DEBUG:boto:Method: GET 2023-09-22T21:40:21.195 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:40:21.195 DEBUG:boto:Data: 2023-09-22T21:40:21.195 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.195 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:21.195 DEBUG:boto:Port: 8000 2023-09-22T21:40:21.195 DEBUG:boto:Params: {} 2023-09-22T21:40:21.196 DEBUG:boto:Token: None 2023-09-22T21:40:21.196 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/obj95 2023-09-22T21:40:21.196 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RqtOOL93te07IMr6SPTdoAB6xkM= 2023-09-22T21:40:21.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RqtOOL93te07IMr6SPTdoAB6xkM=', 'Content-Length': '0'} 2023-09-22T21:40:21.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065545921f3ffda02-00650e09c5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.199 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:21.200 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:21.200 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:40:21.200 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:40:21.200 DEBUG:boto:Method: GET 2023-09-22T21:40:21.200 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:40:21.200 DEBUG:boto:Data: 2023-09-22T21:40:21.200 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.200 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:40:21.200 DEBUG:boto:Port: 8001 2023-09-22T21:40:21.201 DEBUG:boto:Params: {} 2023-09-22T21:40:21.201 DEBUG:boto:Token: None 2023-09-22T21:40:21.201 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/ 2023-09-22T21:40:21.201 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Sx+Ye+aIdwr2XiYA2GdrSvNCOdU= 2023-09-22T21:40:21.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Sx+Ye+aIdwr2XiYA2GdrSvNCOdU=', 'Content-Length': '0'} 2023-09-22T21:40:21.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d73bb9d2c75e1d41-00650e09c5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.206 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:21.206 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:21.206 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:21.207 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:40:21.207 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:40:21.207 DEBUG:boto:Method: GET 2023-09-22T21:40:21.207 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:40:21.207 DEBUG:boto:Data: 2023-09-22T21:40:21.207 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.207 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:21.207 DEBUG:boto:Port: 8000 2023-09-22T21:40:21.207 DEBUG:boto:Params: {} 2023-09-22T21:40:21.207 DEBUG:boto:Token: None 2023-09-22T21:40:21.208 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/ 2023-09-22T21:40:21.208 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Sx+Ye+aIdwr2XiYA2GdrSvNCOdU= 2023-09-22T21:40:21.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Sx+Ye+aIdwr2XiYA2GdrSvNCOdU=', 'Content-Length': '0'} 2023-09-22T21:40:21.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000611f00adf050a683-00650e09c5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.214 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:40:21.214 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-59 zones={a2, a1} 2023-09-22T21:40:21.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:40:21.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:40:21.215 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:40:21.261 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:40:21.261 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:40:21.319 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:40:21.319 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-59 2023-09-22T21:40:21.330 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 15 2023-09-22T21:40:21.391 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:21.391 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:21.391 DEBUG:boto:Method: HEAD 2023-09-22T21:40:21.392 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:40:21.392 DEBUG:boto:Data: 2023-09-22T21:40:21.392 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.392 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:21.392 DEBUG:boto:Port: 8000 2023-09-22T21:40:21.392 DEBUG:boto:Params: {} 2023-09-22T21:40:21.392 DEBUG:boto:Token: None 2023-09-22T21:40:21.392 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/ 2023-09-22T21:40:21.392 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0usU78TLmvMMIEFv/Fw2rAVy+dY= 2023-09-22T21:40:21.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0usU78TLmvMMIEFv/Fw2rAVy+dY=', 'Content-Length': '0'} 2023-09-22T21:40:21.396 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f77f62a9ba535325-00650e09c5-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.396 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:40:21.396 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:40:21.396 DEBUG:boto:Method: PUT 2023-09-22T21:40:21.396 DEBUG:boto:Path: /ffzdqu-59/obj23 2023-09-22T21:40:21.397 DEBUG:boto:Data: 2023-09-22T21:40:21.397 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:40:21.397 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:21.397 DEBUG:boto:Port: 8000 2023-09-22T21:40:21.397 DEBUG:boto:Params: {} 2023-09-22T21:40:21.397 DEBUG:boto:Token: None 2023-09-22T21:40:21.397 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/obj23 2023-09-22T21:40:21.397 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:py31CLxbLCNVJsEQOH7DShRqV7w= 2023-09-22T21:40:21.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:py31CLxbLCNVJsEQOH7DShRqV7w='} 2023-09-22T21:40:21.404 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f341573140855476-00650e09c5-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.404 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:21.405 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:21.405 DEBUG:boto:Method: HEAD 2023-09-22T21:40:21.405 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:40:21.405 DEBUG:boto:Data: 2023-09-22T21:40:21.405 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.405 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:21.405 DEBUG:boto:Port: 8000 2023-09-22T21:40:21.405 DEBUG:boto:Params: {} 2023-09-22T21:40:21.405 DEBUG:boto:Token: None 2023-09-22T21:40:21.406 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/ 2023-09-22T21:40:21.406 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0usU78TLmvMMIEFv/Fw2rAVy+dY= 2023-09-22T21:40:21.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0usU78TLmvMMIEFv/Fw2rAVy+dY=', 'Content-Length': '0'} 2023-09-22T21:40:21.409 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba2b77fb0ef8ebb4-00650e09c5-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.409 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:40:21.409 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:40:21.409 DEBUG:boto:Method: PUT 2023-09-22T21:40:21.409 DEBUG:boto:Path: /ffzdqu-59/obj46 2023-09-22T21:40:21.410 DEBUG:boto:Data: 2023-09-22T21:40:21.410 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:40:21.410 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:21.410 DEBUG:boto:Port: 8000 2023-09-22T21:40:21.410 DEBUG:boto:Params: {} 2023-09-22T21:40:21.410 DEBUG:boto:Token: None 2023-09-22T21:40:21.410 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/obj46 2023-09-22T21:40:21.410 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9bQCCwx5ILFNTqZ+wHXvVNDjrtg= 2023-09-22T21:40:21.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9bQCCwx5ILFNTqZ+wHXvVNDjrtg='} 2023-09-22T21:40:21.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000656fd1e38abbc68d-00650e09c5-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.455 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:21.455 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:21.455 DEBUG:boto:Method: HEAD 2023-09-22T21:40:21.455 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:40:21.455 DEBUG:boto:Data: 2023-09-22T21:40:21.455 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.455 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:21.455 DEBUG:boto:Port: 8000 2023-09-22T21:40:21.455 DEBUG:boto:Params: {} 2023-09-22T21:40:21.456 DEBUG:boto:Token: None 2023-09-22T21:40:21.456 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/ 2023-09-22T21:40:21.456 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0usU78TLmvMMIEFv/Fw2rAVy+dY= 2023-09-22T21:40:21.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0usU78TLmvMMIEFv/Fw2rAVy+dY=', 'Content-Length': '0'} 2023-09-22T21:40:21.460 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3a7c7cc49c02c15-00650e09c5-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.460 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:40:21.461 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:40:21.461 DEBUG:boto:Method: PUT 2023-09-22T21:40:21.461 DEBUG:boto:Path: /ffzdqu-59/obj69 2023-09-22T21:40:21.461 DEBUG:boto:Data: 2023-09-22T21:40:21.461 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:40:21.461 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:21.461 DEBUG:boto:Port: 8000 2023-09-22T21:40:21.461 DEBUG:boto:Params: {} 2023-09-22T21:40:21.461 DEBUG:boto:Token: None 2023-09-22T21:40:21.462 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/obj69 2023-09-22T21:40:21.462 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hy6hvTfpwC7ASYckpoiJ5ffUF0s= 2023-09-22T21:40:21.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hy6hvTfpwC7ASYckpoiJ5ffUF0s='} 2023-09-22T21:40:21.473 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000654ca48d805da98d-00650e09c5-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.473 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:21.473 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:21.474 DEBUG:boto:Method: HEAD 2023-09-22T21:40:21.474 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:40:21.474 DEBUG:boto:Data: 2023-09-22T21:40:21.474 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.474 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:21.474 DEBUG:boto:Port: 8000 2023-09-22T21:40:21.474 DEBUG:boto:Params: {} 2023-09-22T21:40:21.474 DEBUG:boto:Token: None 2023-09-22T21:40:21.474 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/ 2023-09-22T21:40:21.475 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0usU78TLmvMMIEFv/Fw2rAVy+dY= 2023-09-22T21:40:21.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0usU78TLmvMMIEFv/Fw2rAVy+dY=', 'Content-Length': '0'} 2023-09-22T21:40:21.479 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074a98c7c7be81165-00650e09c5-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.480 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:40:21.480 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:40:21.480 DEBUG:boto:Method: PUT 2023-09-22T21:40:21.480 DEBUG:boto:Path: /ffzdqu-59/obj92 2023-09-22T21:40:21.480 DEBUG:boto:Data: 2023-09-22T21:40:21.480 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:40:21.480 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:21.480 DEBUG:boto:Port: 8000 2023-09-22T21:40:21.480 DEBUG:boto:Params: {} 2023-09-22T21:40:21.481 DEBUG:boto:Token: None 2023-09-22T21:40:21.481 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/obj92 2023-09-22T21:40:21.481 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vWqXmLnDpSMNWf7n5uGi06fwU6g= 2023-09-22T21:40:21.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vWqXmLnDpSMNWf7n5uGi06fwU6g='} 2023-09-22T21:40:21.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6cc5b6249d8489b-00650e09c5-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.527 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:40:21.527 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:40:21.527 DEBUG:boto:Method: HEAD 2023-09-22T21:40:21.527 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:40:21.527 DEBUG:boto:Data: 2023-09-22T21:40:21.527 DEBUG:boto:Headers: {} 2023-09-22T21:40:21.528 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:21.528 DEBUG:boto:Port: 8000 2023-09-22T21:40:21.528 DEBUG:boto:Params: {} 2023-09-22T21:40:21.528 DEBUG:boto:Token: None 2023-09-22T21:40:21.528 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/ 2023-09-22T21:40:21.528 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0usU78TLmvMMIEFv/Fw2rAVy+dY= 2023-09-22T21:40:21.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0usU78TLmvMMIEFv/Fw2rAVy+dY=', 'Content-Length': '0'} 2023-09-22T21:40:21.534 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cbb5ca0ce8f47033-00650e09c5-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.534 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:40:21.534 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:40:21.535 DEBUG:boto:Method: PUT 2023-09-22T21:40:21.535 DEBUG:boto:Path: /ffzdqu-59/obj115 2023-09-22T21:40:21.535 DEBUG:boto:Data: 2023-09-22T21:40:21.535 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:40:21.535 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:40:21.535 DEBUG:boto:Port: 8000 2023-09-22T21:40:21.535 DEBUG:boto:Params: {} 2023-09-22T21:40:21.535 DEBUG:boto:Token: None 2023-09-22T21:40:21.536 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:40:21 GMT /ffzdqu-59/obj115 2023-09-22T21:40:21.536 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:l10CufMjaNXSIS1e3feGrzyBrmA= 2023-09-22T21:40:21.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:40:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:l10CufMjaNXSIS1e3feGrzyBrmA='} 2023-09-22T21:40:21.549 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029e81988e2292ae2-00650e09c5-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:40:21.549 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-59 2023-09-22T21:40:21.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:40:21.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:40:21.550 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:40:21.599 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:40:21.599 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:40:21.642 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:40:21.639+0000 7fb44894ea40 1 waiting to reach latest gen 3, on 2.. 2023-09-22T21:40:26.644 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:40:26.639+0000 7fb44894ea40 1 waiting to reach latest gen 3, on 2.. 2023-09-22T21:40:31.648 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:40:31.643+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:40:31.648 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , , , , , , , 00000000001.92.6, , , , , , ] 2023-09-22T21:40:31.648 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:40:36.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:40:36.647+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:40:36.653 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , , , , , , , 00000000001.92.6, , , , , , ] 2023-09-22T21:40:36.653 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:40:41.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:40:41.651+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:40:41.703 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , , , , , , , 00000000001.92.6, , , , , , ] 2023-09-22T21:40:41.704 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:40:46.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:40:46.655+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:40:46.659 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , , , , , , , 00000000001.92.6, , , , , , ] 2023-09-22T21:40:46.659 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:40:51.662 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:40:51.659+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:40:51.663 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , , , , , , , 00000000001.92.6, , , , , , ] 2023-09-22T21:40:51.663 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:40:56.666 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:40:56.663+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:40:56.666 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , , , , , , , 00000000001.92.6, , , , , , ] 2023-09-22T21:40:56.666 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:41:01.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:41:01.666+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:41:01.669 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , , , , , , , 00000000001.92.6, , , , , , ] 2023-09-22T21:41:01.669 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:41:06.672 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:41:06.666+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:41:06.673 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , , , , , , , 00000000001.92.6, , , , , , ] 2023-09-22T21:41:06.673 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:41:11.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:41:11.670+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:41:11.676 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , , , , , , , 00000000001.92.6, , , , , , ] 2023-09-22T21:41:11.676 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:41:16.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:41:16.674+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:41:16.680 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , , , , , , , 00000000001.92.6, , , , , , ] 2023-09-22T21:41:16.680 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:41:21.683 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:41:21.678+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:41:21.683 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , , , , , , , 00000000001.92.6, , , , , , ] 2023-09-22T21:41:21.683 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:41:26.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:41:26.682+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:41:26.687 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , , , , , , , 00000000001.92.6, , , , , , ] 2023-09-22T21:41:26.687 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:41:31.690 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:41:31.686+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:41:31.690 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , , , , , , , 00000000001.92.6, , , , , , ] 2023-09-22T21:41:31.690 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:41:36.693 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:41:36.690+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:41:36.693 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , , , , , , , 00000000001.92.6, , , , , , ] 2023-09-22T21:41:36.694 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:41:41.697 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:41:41.694+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:41:41.697 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , , , , , , , 00000000001.92.6, , , , , , ] 2023-09-22T21:41:41.697 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:41:46.700 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:41:46.694+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:41:46.700 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , , , , , , , 00000000001.92.6, , , , , , ] 2023-09-22T21:41:46.700 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:41:49.488 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:41:49.482+0000 7f092e9f8640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:41:51.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:41:51.698+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:41:51.704 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:41:51.704 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:41:56.707 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:41:56.702+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:41:56.708 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:41:56.708 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:01.711 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:42:01.706+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:42:01.711 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:01.711 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:06.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:42:06.710+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:42:06.714 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:06.715 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:11.718 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:42:11.714+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:42:11.718 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:11.718 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:16.721 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:42:16.718+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:42:16.722 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:16.722 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:21.725 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:42:21.721+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:42:21.725 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:21.725 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:26.728 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:42:26.725+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:42:27.754 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:27.754 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:31.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:42:31.725+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:42:31.732 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:31.732 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:36.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:42:36.729+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:42:36.736 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:36.736 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:41.739 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:42:41.737+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:42:41.739 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:41.739 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:46.742 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:42:46.741+0000 7fb44894ea40 1 waiting for incremental sync to catch up: 2023-09-22T21:42:46.743 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:46.743 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:51.746 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:42:51.745+0000 7fb44894ea40 1 bucket sync caught up with source: 2023-09-22T21:42:51.746 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:51.746 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.66.6, , , 00000000002.62.6, , 00000000002.64.6, , , , , , 00000000001.92.6, , 00000000001.70.6, , , , ] 2023-09-22T21:42:51.746 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:42:51.745+0000 7fb44894ea40 0 bucket checkpoint complete 2023-09-22T21:42:51.780 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-59 2023-09-22T21:42:51.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:42:51.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:42:51.781 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:42:51.877 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:42:51.877 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:42:51.923 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:42:51.921+0000 7f0d606d9a40 1 waiting for incremental sync to catch up: 2023-09-22T21:42:51.924 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.301.6, 00000000002.265.6, 00000000001.144.6, 00000000002.274.6, 00000000002.395.6, , , 00000000002.137.6, 00000000004.158.6, 00000000004.310.6] 2023-09-22T21:42:51.924 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.301.6, 00000000002.265.6, 00000000001.144.6, 00000000002.274.6, 00000000002.395.6, , , 00000000002.137.6, 00000000005.162.6, 00000000004.310.6] 2023-09-22T21:42:56.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:42:56.925+0000 7f0d606d9a40 1 waiting for incremental sync to catch up: 2023-09-22T21:42:56.925 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.301.6, 00000000002.265.6, 00000000001.144.6, 00000000002.274.6, 00000000002.395.6, , , 00000000002.137.6, 00000000004.158.6, 00000000004.310.6] 2023-09-22T21:42:56.925 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.301.6, 00000000002.265.6, 00000000001.144.6, 00000000002.274.6, 00000000002.395.6, , , 00000000002.137.6, 00000000005.162.6, 00000000004.310.6] 2023-09-22T21:43:01.927 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:43:01.925+0000 7f0d606d9a40 1 waiting for incremental sync to catch up: 2023-09-22T21:43:01.927 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.301.6, 00000000002.265.6, 00000000001.144.6, 00000000002.274.6, 00000000002.395.6, , , 00000000002.137.6, 00000000004.158.6, 00000000004.310.6] 2023-09-22T21:43:01.927 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.301.6, 00000000002.265.6, 00000000001.144.6, 00000000002.274.6, 00000000002.395.6, , , 00000000002.137.6, 00000000005.162.6, 00000000004.310.6] 2023-09-22T21:43:06.606 INFO:tasks.rgw.c1.client.0.smithi160.stdout:2023-09-22T21:43:06.605+0000 7fa9c1675640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:43:06.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:43:06.929+0000 7f0d606d9a40 1 bucket sync caught up with source: 2023-09-22T21:43:06.930 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.301.6, 00000000002.265.6, 00000000001.144.6, 00000000002.274.6, 00000000002.395.6, , , 00000000002.137.6, 00000000005.162.6, 00000000004.310.6] 2023-09-22T21:43:06.930 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.301.6, 00000000002.265.6, 00000000001.144.6, 00000000002.274.6, 00000000002.395.6, , , 00000000002.137.6, 00000000005.162.6, 00000000004.310.6] 2023-09-22T21:43:06.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:43:06.929+0000 7f0d606d9a40 0 bucket checkpoint complete 2023-09-22T21:43:06.957 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-59 zones={a2, a1} 2023-09-22T21:43:06.958 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:06.958 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:06.958 DEBUG:boto:Method: HEAD 2023-09-22T21:43:06.958 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:43:06.958 DEBUG:boto:Data: 2023-09-22T21:43:06.958 DEBUG:boto:Headers: {} 2023-09-22T21:43:06.959 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:06.959 DEBUG:boto:Port: 8001 2023-09-22T21:43:06.959 DEBUG:boto:Params: {} 2023-09-22T21:43:06.959 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:43:06.959 DEBUG:boto:Token: None 2023-09-22T21:43:06.960 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:06 GMT /ffzdqu-59/ 2023-09-22T21:43:06.960 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+27AWhUuIpjyJBld2Jkbw1q+HHY= 2023-09-22T21:43:06.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+27AWhUuIpjyJBld2Jkbw1q+HHY=', 'Content-Length': '0'} 2023-09-22T21:43:06.966 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ffd960423121dad-00650e0a6a-1b9c-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:06.966 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:06.966 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:06.967 DEBUG:boto:Method: HEAD 2023-09-22T21:43:06.967 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:43:06.967 DEBUG:boto:Data: 2023-09-22T21:43:06.967 DEBUG:boto:Headers: {} 2023-09-22T21:43:06.967 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:06.967 DEBUG:boto:Port: 8000 2023-09-22T21:43:06.967 DEBUG:boto:Params: {} 2023-09-22T21:43:06.967 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:43:06.968 DEBUG:boto:Token: None 2023-09-22T21:43:06.968 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:06 GMT /ffzdqu-59/ 2023-09-22T21:43:06.968 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+27AWhUuIpjyJBld2Jkbw1q+HHY= 2023-09-22T21:43:06.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+27AWhUuIpjyJBld2Jkbw1q+HHY=', 'Content-Length': '0'} 2023-09-22T21:43:06.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd34170218d8619c-00650e0a6a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:06.977 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:43:06.977 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:06.978 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:06.978 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:43:06.978 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:43:06.978 DEBUG:boto:Method: GET 2023-09-22T21:43:06.978 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:43:06.978 DEBUG:boto:Data: 2023-09-22T21:43:06.978 DEBUG:boto:Headers: {} 2023-09-22T21:43:06.978 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:06.978 DEBUG:boto:Port: 8001 2023-09-22T21:43:06.979 DEBUG:boto:Params: {} 2023-09-22T21:43:06.979 DEBUG:boto:Token: None 2023-09-22T21:43:06.979 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:06 GMT /ffzdqu-59/?versions 2023-09-22T21:43:06.979 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ndSVpUL1Ecg551BFFCTFYxMR7nU= 2023-09-22T21:43:06.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ndSVpUL1Ecg551BFFCTFYxMR7nU=', 'Content-Length': '0'} 2023-09-22T21:43:06.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000355bfde7ff85d717-00650e0a6a-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:06.983 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:06.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T21:43:06.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-22T21:43:06.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-22T21:43:06.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-22T21:43:06.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T21:43:06.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-22T21:43:06.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T21:43:06.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-22T21:43:06.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-22T21:43:06.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T21:43:06.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-22T21:43:06.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-22T21:43:06.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T21:43:06.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-22T21:43:06.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-22T21:43:06.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-22T21:43:06.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-22T21:43:06.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-22T21:43:06.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-22T21:43:06.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-22T21:43:06.987 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:43:06.987 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:06.987 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:06.987 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:43:06.987 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:43:06.987 DEBUG:boto:Method: GET 2023-09-22T21:43:06.987 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:43:06.987 DEBUG:boto:Data: 2023-09-22T21:43:06.987 DEBUG:boto:Headers: {} 2023-09-22T21:43:06.988 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:06.988 DEBUG:boto:Port: 8000 2023-09-22T21:43:06.988 DEBUG:boto:Params: {} 2023-09-22T21:43:06.988 DEBUG:boto:Token: None 2023-09-22T21:43:06.988 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:06 GMT /ffzdqu-59/?versions 2023-09-22T21:43:06.988 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ndSVpUL1Ecg551BFFCTFYxMR7nU= 2023-09-22T21:43:06.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ndSVpUL1Ecg551BFFCTFYxMR7nU=', 'Content-Length': '0'} 2023-09-22T21:43:06.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034bdffeaf8e1f453-00650e0a6a-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:06.994 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:06.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T21:43:06.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-22T21:43:06.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-22T21:43:06.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-22T21:43:06.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T21:43:06.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-22T21:43:06.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T21:43:06.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-22T21:43:06.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-22T21:43:06.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T21:43:06.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-22T21:43:06.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-22T21:43:06.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T21:43:06.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-22T21:43:06.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-22T21:43:06.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-22T21:43:06.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-22T21:43:06.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-22T21:43:06.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-22T21:43:06.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-22T21:43:06.997 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:06.997 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:06.997 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:43:06.998 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:43:06.998 DEBUG:boto:Method: GET 2023-09-22T21:43:06.998 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:43:06.998 DEBUG:boto:Data: 2023-09-22T21:43:06.998 DEBUG:boto:Headers: {} 2023-09-22T21:43:06.998 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:06.998 DEBUG:boto:Port: 8001 2023-09-22T21:43:06.998 DEBUG:boto:Params: {} 2023-09-22T21:43:06.998 DEBUG:boto:Token: None 2023-09-22T21:43:06.999 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:06 GMT /ffzdqu-59/?versions 2023-09-22T21:43:06.999 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ndSVpUL1Ecg551BFFCTFYxMR7nU= 2023-09-22T21:43:06.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ndSVpUL1Ecg551BFFCTFYxMR7nU=', 'Content-Length': '0'} 2023-09-22T21:43:07.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8086c6c01ed652d-00650e0a6a-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.002 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:07.004 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:07.004 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:07.004 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:43:07.004 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:43:07.004 DEBUG:boto:Method: GET 2023-09-22T21:43:07.004 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:43:07.004 DEBUG:boto:Data: 2023-09-22T21:43:07.004 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.004 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.005 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.005 DEBUG:boto:Params: {} 2023-09-22T21:43:07.005 DEBUG:boto:Token: None 2023-09-22T21:43:07.005 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/?versions 2023-09-22T21:43:07.005 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eBKySE/1pjPlA8mOp99KiG0JA18= 2023-09-22T21:43:07.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eBKySE/1pjPlA8mOp99KiG0JA18=', 'Content-Length': '0'} 2023-09-22T21:43:07.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d7d51b06707b0e3-00650e0a6b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.011 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:07.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:43:07.013 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:43:07.013 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:43:07.013 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.013 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.013 DEBUG:boto:Method: GET 2023-09-22T21:43:07.013 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.013 DEBUG:boto:Data: 2023-09-22T21:43:07.013 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.014 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.014 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.014 DEBUG:boto:Params: {} 2023-09-22T21:43:07.014 DEBUG:boto:Token: None 2023-09-22T21:43:07.014 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.014 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jJ80v7CtHITiZof9+fGBYFWrZqs= 2023-09-22T21:43:07.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jJ80v7CtHITiZof9+fGBYFWrZqs=', 'Content-Length': '0'} 2023-09-22T21:43:07.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c3366f3748bc20e-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.017 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:43:07.017 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:43:07.017 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.017 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.017 DEBUG:boto:Method: GET 2023-09-22T21:43:07.017 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.017 DEBUG:boto:Data: 2023-09-22T21:43:07.017 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.018 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.018 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.018 DEBUG:boto:Params: {} 2023-09-22T21:43:07.018 DEBUG:boto:Token: None 2023-09-22T21:43:07.018 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.018 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jJ80v7CtHITiZof9+fGBYFWrZqs= 2023-09-22T21:43:07.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jJ80v7CtHITiZof9+fGBYFWrZqs=', 'Content-Length': '0'} 2023-09-22T21:43:07.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f14ca62ff948154-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.066 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:43:07.067 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:43:07.067 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.067 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.067 DEBUG:boto:Method: HEAD 2023-09-22T21:43:07.067 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.067 DEBUG:boto:Data: 2023-09-22T21:43:07.067 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.067 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.067 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.068 DEBUG:boto:Params: {} 2023-09-22T21:43:07.068 DEBUG:boto:Token: None 2023-09-22T21:43:07.068 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.068 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bcOgePxXoJsJ6lJi2UQkzPXRp3c= 2023-09-22T21:43:07.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bcOgePxXoJsJ6lJi2UQkzPXRp3c=', 'Content-Length': '0'} 2023-09-22T21:43:07.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b02bb02147d681d-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.071 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:43:07.071 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:43:07.072 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.072 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.072 DEBUG:boto:Method: HEAD 2023-09-22T21:43:07.072 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.072 DEBUG:boto:Data: 2023-09-22T21:43:07.072 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.072 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.072 DEBUG:boto:Params: {} 2023-09-22T21:43:07.073 DEBUG:boto:Token: None 2023-09-22T21:43:07.073 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.073 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bcOgePxXoJsJ6lJi2UQkzPXRp3c= 2023-09-22T21:43:07.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bcOgePxXoJsJ6lJi2UQkzPXRp3c=', 'Content-Length': '0'} 2023-09-22T21:43:07.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002428dda9dd3e77e1-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:43:07.076 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:43:07.076 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:43:07.076 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.076 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.076 DEBUG:boto:Method: GET 2023-09-22T21:43:07.077 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.077 DEBUG:boto:Data: 2023-09-22T21:43:07.077 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.077 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.077 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.077 DEBUG:boto:Params: {} 2023-09-22T21:43:07.077 DEBUG:boto:Token: None 2023-09-22T21:43:07.077 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.078 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jJ80v7CtHITiZof9+fGBYFWrZqs= 2023-09-22T21:43:07.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jJ80v7CtHITiZof9+fGBYFWrZqs=', 'Content-Length': '0'} 2023-09-22T21:43:07.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d13a06ef2c69327-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.126 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:43:07.127 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:43:07.127 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.127 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.127 DEBUG:boto:Method: GET 2023-09-22T21:43:07.127 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.127 DEBUG:boto:Data: 2023-09-22T21:43:07.127 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.127 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.127 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.127 DEBUG:boto:Params: {} 2023-09-22T21:43:07.128 DEBUG:boto:Token: None 2023-09-22T21:43:07.128 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:43:07.128 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jJ80v7CtHITiZof9+fGBYFWrZqs= 2023-09-22T21:43:07.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jJ80v7CtHITiZof9+fGBYFWrZqs=', 'Content-Length': '0'} 2023-09-22T21:43:07.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000433e8639eec384b0-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.131 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:07.131 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:07.131 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:43:07.132 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:43:07.132 DEBUG:boto:Method: GET 2023-09-22T21:43:07.132 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:43:07.132 DEBUG:boto:Data: 2023-09-22T21:43:07.132 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.132 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.132 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.132 DEBUG:boto:Params: {} 2023-09-22T21:43:07.132 DEBUG:boto:Token: None 2023-09-22T21:43:07.133 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/ 2023-09-22T21:43:07.133 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E= 2023-09-22T21:43:07.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E=', 'Content-Length': '0'} 2023-09-22T21:43:07.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae6b9a55fb175ef5-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.137 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:07.138 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:07.138 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:07.138 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:43:07.138 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:43:07.138 DEBUG:boto:Method: GET 2023-09-22T21:43:07.138 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:43:07.139 DEBUG:boto:Data: 2023-09-22T21:43:07.139 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.139 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.139 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.139 DEBUG:boto:Params: {} 2023-09-22T21:43:07.139 DEBUG:boto:Token: None 2023-09-22T21:43:07.139 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/ 2023-09-22T21:43:07.139 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E= 2023-09-22T21:43:07.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E=', 'Content-Length': '0'} 2023-09-22T21:43:07.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d36291e08b2775b5-00650e0a6b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.147 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:07.148 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:43:07.148 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:43:07.148 DEBUG:boto:Method: HEAD 2023-09-22T21:43:07.148 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:43:07.148 DEBUG:boto:Data: 2023-09-22T21:43:07.148 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.148 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.148 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.149 DEBUG:boto:Params: {} 2023-09-22T21:43:07.149 DEBUG:boto:Token: None 2023-09-22T21:43:07.149 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj11 2023-09-22T21:43:07.149 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uJ8hV/YvqgUFxXJyAwTCJ8w16Cs= 2023-09-22T21:43:07.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uJ8hV/YvqgUFxXJyAwTCJ8w16Cs=', 'Content-Length': '0'} 2023-09-22T21:43:07.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dde979b11ec675ec-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.152 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:43:07.152 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:43:07.152 DEBUG:boto:Method: HEAD 2023-09-22T21:43:07.152 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:43:07.152 DEBUG:boto:Data: 2023-09-22T21:43:07.153 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.153 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.153 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.153 DEBUG:boto:Params: {} 2023-09-22T21:43:07.153 DEBUG:boto:Token: None 2023-09-22T21:43:07.153 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj11 2023-09-22T21:43:07.153 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uJ8hV/YvqgUFxXJyAwTCJ8w16Cs= 2023-09-22T21:43:07.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uJ8hV/YvqgUFxXJyAwTCJ8w16Cs=', 'Content-Length': '0'} 2023-09-22T21:43:07.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000555e0a30cc676353-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:43:07.156 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:43:07.156 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:43:07.157 DEBUG:boto:Method: GET 2023-09-22T21:43:07.157 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:43:07.157 DEBUG:boto:Data: 2023-09-22T21:43:07.157 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.157 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.157 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.157 DEBUG:boto:Params: {} 2023-09-22T21:43:07.158 DEBUG:boto:Token: None 2023-09-22T21:43:07.158 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj11 2023-09-22T21:43:07.158 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y/VgfDsc9P+31NytI+KPFa4F2fU= 2023-09-22T21:43:07.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y/VgfDsc9P+31NytI+KPFa4F2fU=', 'Content-Length': '0'} 2023-09-22T21:43:07.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d84f2c860aaedf19-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.202 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:43:07.203 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:43:07.203 DEBUG:boto:Method: GET 2023-09-22T21:43:07.203 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:43:07.203 DEBUG:boto:Data: 2023-09-22T21:43:07.203 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.204 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.204 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.204 DEBUG:boto:Params: {} 2023-09-22T21:43:07.204 DEBUG:boto:Token: None 2023-09-22T21:43:07.204 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj11 2023-09-22T21:43:07.204 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y/VgfDsc9P+31NytI+KPFa4F2fU= 2023-09-22T21:43:07.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y/VgfDsc9P+31NytI+KPFa4F2fU=', 'Content-Length': '0'} 2023-09-22T21:43:07.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ccd3fd85b02aa76-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.207 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:07.208 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:07.208 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:43:07.208 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:43:07.208 DEBUG:boto:Method: GET 2023-09-22T21:43:07.208 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:43:07.208 DEBUG:boto:Data: 2023-09-22T21:43:07.208 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.208 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.208 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.209 DEBUG:boto:Params: {} 2023-09-22T21:43:07.209 DEBUG:boto:Token: None 2023-09-22T21:43:07.209 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/ 2023-09-22T21:43:07.209 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E= 2023-09-22T21:43:07.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E=', 'Content-Length': '0'} 2023-09-22T21:43:07.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f320d97796cace90-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.214 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:07.215 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:07.215 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:07.215 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:43:07.215 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:43:07.215 DEBUG:boto:Method: GET 2023-09-22T21:43:07.216 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:43:07.216 DEBUG:boto:Data: 2023-09-22T21:43:07.216 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.216 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.216 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.216 DEBUG:boto:Params: {} 2023-09-22T21:43:07.216 DEBUG:boto:Token: None 2023-09-22T21:43:07.217 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/ 2023-09-22T21:43:07.217 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E= 2023-09-22T21:43:07.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E=', 'Content-Length': '0'} 2023-09-22T21:43:07.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c636706d2747a24f-00650e0a6b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.224 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:07.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-22T21:43:07.224 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:43:07.224 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:43:07.224 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.225 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.225 DEBUG:boto:Method: GET 2023-09-22T21:43:07.225 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.225 DEBUG:boto:Data: 2023-09-22T21:43:07.225 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.225 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.225 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.225 DEBUG:boto:Params: {} 2023-09-22T21:43:07.225 DEBUG:boto:Token: None 2023-09-22T21:43:07.226 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.226 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UnqwJGVzXqGoLUIJBIuzDsbrg28= 2023-09-22T21:43:07.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UnqwJGVzXqGoLUIJBIuzDsbrg28=', 'Content-Length': '0'} 2023-09-22T21:43:07.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c544c23835462705-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.229 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:43:07.229 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:43:07.229 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.229 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.229 DEBUG:boto:Method: GET 2023-09-22T21:43:07.230 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.230 DEBUG:boto:Data: 2023-09-22T21:43:07.230 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.230 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.230 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.230 DEBUG:boto:Params: {} 2023-09-22T21:43:07.230 DEBUG:boto:Token: None 2023-09-22T21:43:07.230 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.231 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UnqwJGVzXqGoLUIJBIuzDsbrg28= 2023-09-22T21:43:07.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UnqwJGVzXqGoLUIJBIuzDsbrg28=', 'Content-Length': '0'} 2023-09-22T21:43:07.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c6fd31fb15e2f22-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.234 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:43:07.234 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:43:07.234 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.234 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.234 DEBUG:boto:Method: HEAD 2023-09-22T21:43:07.234 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.235 DEBUG:boto:Data: 2023-09-22T21:43:07.235 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.235 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.235 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.235 DEBUG:boto:Params: {} 2023-09-22T21:43:07.235 DEBUG:boto:Token: None 2023-09-22T21:43:07.235 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.235 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DA2IEHEqv5tM7uG2Em+SgKqb3wI= 2023-09-22T21:43:07.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DA2IEHEqv5tM7uG2Em+SgKqb3wI=', 'Content-Length': '0'} 2023-09-22T21:43:07.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b646f2e8fed23a82-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.238 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:43:07.238 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:43:07.238 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.238 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.239 DEBUG:boto:Method: HEAD 2023-09-22T21:43:07.239 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.239 DEBUG:boto:Data: 2023-09-22T21:43:07.239 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.239 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.239 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.239 DEBUG:boto:Params: {} 2023-09-22T21:43:07.239 DEBUG:boto:Token: None 2023-09-22T21:43:07.240 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.240 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DA2IEHEqv5tM7uG2Em+SgKqb3wI= 2023-09-22T21:43:07.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DA2IEHEqv5tM7uG2Em+SgKqb3wI=', 'Content-Length': '0'} 2023-09-22T21:43:07.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052a45e7cffdea27f-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-22T21:43:07.243 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:43:07.243 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:43:07.243 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.243 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.243 DEBUG:boto:Method: GET 2023-09-22T21:43:07.243 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.243 DEBUG:boto:Data: 2023-09-22T21:43:07.243 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.243 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.244 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.244 DEBUG:boto:Params: {} 2023-09-22T21:43:07.244 DEBUG:boto:Token: None 2023-09-22T21:43:07.244 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.244 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UnqwJGVzXqGoLUIJBIuzDsbrg28= 2023-09-22T21:43:07.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UnqwJGVzXqGoLUIJBIuzDsbrg28=', 'Content-Length': '0'} 2023-09-22T21:43:07.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002529efb6b3ca57ab-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.247 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:43:07.247 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:43:07.248 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.248 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.248 DEBUG:boto:Method: GET 2023-09-22T21:43:07.248 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.248 DEBUG:boto:Data: 2023-09-22T21:43:07.248 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.248 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.248 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.248 DEBUG:boto:Params: {} 2023-09-22T21:43:07.248 DEBUG:boto:Token: None 2023-09-22T21:43:07.249 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:43:07.249 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UnqwJGVzXqGoLUIJBIuzDsbrg28= 2023-09-22T21:43:07.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UnqwJGVzXqGoLUIJBIuzDsbrg28=', 'Content-Length': '0'} 2023-09-22T21:43:07.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2e71c811a6798b2-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.252 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:07.252 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:07.252 DEBUG:boto:path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:43:07.252 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:43:07.252 DEBUG:boto:Method: GET 2023-09-22T21:43:07.252 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj115 2023-09-22T21:43:07.252 DEBUG:boto:Data: 2023-09-22T21:43:07.253 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.253 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.253 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.253 DEBUG:boto:Params: {} 2023-09-22T21:43:07.253 DEBUG:boto:Token: None 2023-09-22T21:43:07.253 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/ 2023-09-22T21:43:07.253 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E= 2023-09-22T21:43:07.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E=', 'Content-Length': '0'} 2023-09-22T21:43:07.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002373749e540d3754-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.298 DEBUG:boto:b'ffzdqu-59obj1151000falseobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:07.299 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:07.299 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:07.299 DEBUG:boto:path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:43:07.299 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:43:07.299 DEBUG:boto:Method: GET 2023-09-22T21:43:07.300 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj115 2023-09-22T21:43:07.300 DEBUG:boto:Data: 2023-09-22T21:43:07.300 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.300 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.300 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.300 DEBUG:boto:Params: {} 2023-09-22T21:43:07.300 DEBUG:boto:Token: None 2023-09-22T21:43:07.300 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/ 2023-09-22T21:43:07.301 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E= 2023-09-22T21:43:07.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E=', 'Content-Length': '0'} 2023-09-22T21:43:07.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfe9a11ec6fee620-00650e0a6b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.308 DEBUG:boto:b'ffzdqu-59obj1151000falseobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:07.308 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:43:07.309 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:43:07.309 DEBUG:boto:Method: HEAD 2023-09-22T21:43:07.309 DEBUG:boto:Path: /ffzdqu-59/obj115 2023-09-22T21:43:07.309 DEBUG:boto:Data: 2023-09-22T21:43:07.309 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.309 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.309 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.309 DEBUG:boto:Params: {} 2023-09-22T21:43:07.309 DEBUG:boto:Token: None 2023-09-22T21:43:07.310 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj115 2023-09-22T21:43:07.310 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:I8IadBVc4QqzsoC0Dvat0/1ON4k= 2023-09-22T21:43:07.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:I8IadBVc4QqzsoC0Dvat0/1ON4k=', 'Content-Length': '0'} 2023-09-22T21:43:07.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0943ba44560db4e-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.313 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:43:07.313 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:43:07.313 DEBUG:boto:Method: HEAD 2023-09-22T21:43:07.313 DEBUG:boto:Path: /ffzdqu-59/obj115 2023-09-22T21:43:07.313 DEBUG:boto:Data: 2023-09-22T21:43:07.313 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.313 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.313 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.314 DEBUG:boto:Params: {} 2023-09-22T21:43:07.314 DEBUG:boto:Token: None 2023-09-22T21:43:07.314 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj115 2023-09-22T21:43:07.314 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:I8IadBVc4QqzsoC0Dvat0/1ON4k= 2023-09-22T21:43:07.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:I8IadBVc4QqzsoC0Dvat0/1ON4k=', 'Content-Length': '0'} 2023-09-22T21:43:07.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bd29556cd4b0e56-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-22T21:43:07.317 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:43:07.318 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:43:07.318 DEBUG:boto:Method: GET 2023-09-22T21:43:07.318 DEBUG:boto:Path: /ffzdqu-59/obj115 2023-09-22T21:43:07.318 DEBUG:boto:Data: 2023-09-22T21:43:07.318 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.319 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.319 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.319 DEBUG:boto:Params: {} 2023-09-22T21:43:07.319 DEBUG:boto:Token: None 2023-09-22T21:43:07.320 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj115 2023-09-22T21:43:07.320 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ta/yDisoRmYSf4JS1avHkReQvgM= 2023-09-22T21:43:07.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ta/yDisoRmYSf4JS1avHkReQvgM=', 'Content-Length': '0'} 2023-09-22T21:43:07.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cab3a5ca85c7ec71-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.323 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:43:07.323 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:43:07.323 DEBUG:boto:Method: GET 2023-09-22T21:43:07.323 DEBUG:boto:Path: /ffzdqu-59/obj115 2023-09-22T21:43:07.323 DEBUG:boto:Data: 2023-09-22T21:43:07.323 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.323 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.324 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.324 DEBUG:boto:Params: {} 2023-09-22T21:43:07.324 DEBUG:boto:Token: None 2023-09-22T21:43:07.324 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj115 2023-09-22T21:43:07.324 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ta/yDisoRmYSf4JS1avHkReQvgM= 2023-09-22T21:43:07.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ta/yDisoRmYSf4JS1avHkReQvgM=', 'Content-Length': '0'} 2023-09-22T21:43:07.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feaf2afb41a4f693-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.370 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:07.371 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:07.371 DEBUG:boto:path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:43:07.371 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:43:07.371 DEBUG:boto:Method: GET 2023-09-22T21:43:07.371 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj115 2023-09-22T21:43:07.371 DEBUG:boto:Data: 2023-09-22T21:43:07.371 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.371 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.371 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.372 DEBUG:boto:Params: {} 2023-09-22T21:43:07.372 DEBUG:boto:Token: None 2023-09-22T21:43:07.372 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/ 2023-09-22T21:43:07.372 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E= 2023-09-22T21:43:07.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E=', 'Content-Length': '0'} 2023-09-22T21:43:07.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f62cdbed1d85a827-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.379 DEBUG:boto:b'ffzdqu-59obj1151000falseobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:07.379 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:07.380 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:07.380 DEBUG:boto:path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:43:07.380 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:43:07.380 DEBUG:boto:Method: GET 2023-09-22T21:43:07.380 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj115 2023-09-22T21:43:07.380 DEBUG:boto:Data: 2023-09-22T21:43:07.380 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.380 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.380 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.380 DEBUG:boto:Params: {} 2023-09-22T21:43:07.381 DEBUG:boto:Token: None 2023-09-22T21:43:07.381 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/ 2023-09-22T21:43:07.381 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E= 2023-09-22T21:43:07.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E=', 'Content-Length': '0'} 2023-09-22T21:43:07.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f779b483d0918a23-00650e0a6b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.390 DEBUG:boto:b'ffzdqu-59obj1151000falseobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:07.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:43:07.391 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:43:07.391 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:43:07.391 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.391 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.391 DEBUG:boto:Method: GET 2023-09-22T21:43:07.391 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.391 DEBUG:boto:Data: 2023-09-22T21:43:07.391 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.392 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.392 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.392 DEBUG:boto:Params: {} 2023-09-22T21:43:07.392 DEBUG:boto:Token: None 2023-09-22T21:43:07.392 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.392 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:W4LljlUdx2nSnGmxCc3DveXohwo= 2023-09-22T21:43:07.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:W4LljlUdx2nSnGmxCc3DveXohwo=', 'Content-Length': '0'} 2023-09-22T21:43:07.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003a0b43b48ae8f6c-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.396 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:43:07.396 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:43:07.396 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.396 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.396 DEBUG:boto:Method: GET 2023-09-22T21:43:07.396 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.396 DEBUG:boto:Data: 2023-09-22T21:43:07.396 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.396 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.397 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.397 DEBUG:boto:Params: {} 2023-09-22T21:43:07.397 DEBUG:boto:Token: None 2023-09-22T21:43:07.397 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.397 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:W4LljlUdx2nSnGmxCc3DveXohwo= 2023-09-22T21:43:07.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:W4LljlUdx2nSnGmxCc3DveXohwo=', 'Content-Length': '0'} 2023-09-22T21:43:07.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbeb72ec4de10bcf-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.400 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:43:07.400 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:43:07.400 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.400 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.401 DEBUG:boto:Method: HEAD 2023-09-22T21:43:07.401 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.401 DEBUG:boto:Data: 2023-09-22T21:43:07.401 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.401 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.401 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.401 DEBUG:boto:Params: {} 2023-09-22T21:43:07.401 DEBUG:boto:Token: None 2023-09-22T21:43:07.401 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.402 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OKSQKTQjBfP/5lnR3/rrtMe/xIc= 2023-09-22T21:43:07.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OKSQKTQjBfP/5lnR3/rrtMe/xIc=', 'Content-Length': '0'} 2023-09-22T21:43:07.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009602d7ca51416573-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.404 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:43:07.405 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:43:07.405 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.405 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.405 DEBUG:boto:Method: HEAD 2023-09-22T21:43:07.405 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.405 DEBUG:boto:Data: 2023-09-22T21:43:07.405 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.405 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.405 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.405 DEBUG:boto:Params: {} 2023-09-22T21:43:07.406 DEBUG:boto:Token: None 2023-09-22T21:43:07.406 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.406 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OKSQKTQjBfP/5lnR3/rrtMe/xIc= 2023-09-22T21:43:07.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OKSQKTQjBfP/5lnR3/rrtMe/xIc=', 'Content-Length': '0'} 2023-09-22T21:43:07.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db46c1aa1791ac32-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:43:07.409 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:43:07.409 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:43:07.409 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.409 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.409 DEBUG:boto:Method: GET 2023-09-22T21:43:07.409 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.409 DEBUG:boto:Data: 2023-09-22T21:43:07.409 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.410 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.410 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.410 DEBUG:boto:Params: {} 2023-09-22T21:43:07.410 DEBUG:boto:Token: None 2023-09-22T21:43:07.410 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.411 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:W4LljlUdx2nSnGmxCc3DveXohwo= 2023-09-22T21:43:07.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:W4LljlUdx2nSnGmxCc3DveXohwo=', 'Content-Length': '0'} 2023-09-22T21:43:07.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2d60ca13a590fd0-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.454 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:43:07.454 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:43:07.455 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.455 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.455 DEBUG:boto:Method: GET 2023-09-22T21:43:07.455 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.455 DEBUG:boto:Data: 2023-09-22T21:43:07.455 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.455 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.455 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.455 DEBUG:boto:Params: {} 2023-09-22T21:43:07.456 DEBUG:boto:Token: None 2023-09-22T21:43:07.456 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:43:07.456 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:W4LljlUdx2nSnGmxCc3DveXohwo= 2023-09-22T21:43:07.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:W4LljlUdx2nSnGmxCc3DveXohwo=', 'Content-Length': '0'} 2023-09-22T21:43:07.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057f0d98d2af3ac34-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.459 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:07.459 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:07.460 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:43:07.460 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:43:07.460 DEBUG:boto:Method: GET 2023-09-22T21:43:07.460 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:43:07.460 DEBUG:boto:Data: 2023-09-22T21:43:07.460 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.460 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.460 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.460 DEBUG:boto:Params: {} 2023-09-22T21:43:07.460 DEBUG:boto:Token: None 2023-09-22T21:43:07.461 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/ 2023-09-22T21:43:07.461 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E= 2023-09-22T21:43:07.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E=', 'Content-Length': '0'} 2023-09-22T21:43:07.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd39643150dfbe37-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.466 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:07.466 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:07.466 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:07.466 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:43:07.467 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:43:07.467 DEBUG:boto:Method: GET 2023-09-22T21:43:07.467 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:43:07.467 DEBUG:boto:Data: 2023-09-22T21:43:07.467 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.467 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.467 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.467 DEBUG:boto:Params: {} 2023-09-22T21:43:07.468 DEBUG:boto:Token: None 2023-09-22T21:43:07.468 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/ 2023-09-22T21:43:07.468 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E= 2023-09-22T21:43:07.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E=', 'Content-Length': '0'} 2023-09-22T21:43:07.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000569ed5cc5153970-00650e0a6b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.475 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:07.475 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:43:07.476 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:43:07.476 DEBUG:boto:Method: HEAD 2023-09-22T21:43:07.476 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:43:07.476 DEBUG:boto:Data: 2023-09-22T21:43:07.476 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.476 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.476 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.476 DEBUG:boto:Params: {} 2023-09-22T21:43:07.477 DEBUG:boto:Token: None 2023-09-22T21:43:07.477 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj17 2023-09-22T21:43:07.477 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DvNXZqJARhktpEjp4JMMMjouHnQ= 2023-09-22T21:43:07.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DvNXZqJARhktpEjp4JMMMjouHnQ=', 'Content-Length': '0'} 2023-09-22T21:43:07.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e22b54bd8ce73167-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.480 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:43:07.480 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:43:07.480 DEBUG:boto:Method: HEAD 2023-09-22T21:43:07.480 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:43:07.480 DEBUG:boto:Data: 2023-09-22T21:43:07.481 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.481 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.481 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.481 DEBUG:boto:Params: {} 2023-09-22T21:43:07.481 DEBUG:boto:Token: None 2023-09-22T21:43:07.481 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj17 2023-09-22T21:43:07.481 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DvNXZqJARhktpEjp4JMMMjouHnQ= 2023-09-22T21:43:07.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DvNXZqJARhktpEjp4JMMMjouHnQ=', 'Content-Length': '0'} 2023-09-22T21:43:07.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bc9f58fc10bc4e8-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:43:07.484 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:43:07.484 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:43:07.484 DEBUG:boto:Method: GET 2023-09-22T21:43:07.484 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:43:07.484 DEBUG:boto:Data: 2023-09-22T21:43:07.485 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.485 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.485 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.485 DEBUG:boto:Params: {} 2023-09-22T21:43:07.485 DEBUG:boto:Token: None 2023-09-22T21:43:07.485 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj17 2023-09-22T21:43:07.485 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7UO/VLA2tz8UviIA+dbsYELODuQ= 2023-09-22T21:43:07.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7UO/VLA2tz8UviIA+dbsYELODuQ=', 'Content-Length': '0'} 2023-09-22T21:43:07.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d75a808d8ff4fec9-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.534 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:43:07.535 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:43:07.535 DEBUG:boto:Method: GET 2023-09-22T21:43:07.535 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:43:07.535 DEBUG:boto:Data: 2023-09-22T21:43:07.535 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.535 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.535 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.535 DEBUG:boto:Params: {} 2023-09-22T21:43:07.535 DEBUG:boto:Token: None 2023-09-22T21:43:07.536 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj17 2023-09-22T21:43:07.536 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7UO/VLA2tz8UviIA+dbsYELODuQ= 2023-09-22T21:43:07.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7UO/VLA2tz8UviIA+dbsYELODuQ=', 'Content-Length': '0'} 2023-09-22T21:43:07.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a3b242db89e16fb-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.539 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:07.539 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:07.539 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:43:07.539 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:43:07.539 DEBUG:boto:Method: GET 2023-09-22T21:43:07.539 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:43:07.539 DEBUG:boto:Data: 2023-09-22T21:43:07.540 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.540 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.540 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.540 DEBUG:boto:Params: {} 2023-09-22T21:43:07.540 DEBUG:boto:Token: None 2023-09-22T21:43:07.540 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/ 2023-09-22T21:43:07.540 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E= 2023-09-22T21:43:07.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E=', 'Content-Length': '0'} 2023-09-22T21:43:07.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afcbeabbaa128e35-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.545 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:07.546 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:07.546 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:07.546 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:43:07.546 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:43:07.546 DEBUG:boto:Method: GET 2023-09-22T21:43:07.546 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:43:07.547 DEBUG:boto:Data: 2023-09-22T21:43:07.547 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.547 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.547 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.547 DEBUG:boto:Params: {} 2023-09-22T21:43:07.547 DEBUG:boto:Token: None 2023-09-22T21:43:07.547 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/ 2023-09-22T21:43:07.547 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E= 2023-09-22T21:43:07.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E=', 'Content-Length': '0'} 2023-09-22T21:43:07.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000362c0a42b1fbfb3b-00650e0a6b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.602 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:07.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T21:43:07.604 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:43:07.604 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:43:07.604 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.605 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.605 DEBUG:boto:Method: GET 2023-09-22T21:43:07.605 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.605 DEBUG:boto:Data: 2023-09-22T21:43:07.605 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.606 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.606 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.606 DEBUG:boto:Params: {} 2023-09-22T21:43:07.606 DEBUG:boto:Token: None 2023-09-22T21:43:07.606 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.607 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jFDCaCLnabO/Th7veVUgQLSsinI= 2023-09-22T21:43:07.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jFDCaCLnabO/Th7veVUgQLSsinI=', 'Content-Length': '0'} 2023-09-22T21:43:07.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ad8422c8c619c33-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.611 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:43:07.611 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:43:07.611 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.611 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.611 DEBUG:boto:Method: GET 2023-09-22T21:43:07.611 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.611 DEBUG:boto:Data: 2023-09-22T21:43:07.611 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.612 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.612 DEBUG:boto:Params: {} 2023-09-22T21:43:07.612 DEBUG:boto:Token: None 2023-09-22T21:43:07.612 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jFDCaCLnabO/Th7veVUgQLSsinI= 2023-09-22T21:43:07.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jFDCaCLnabO/Th7veVUgQLSsinI=', 'Content-Length': '0'} 2023-09-22T21:43:07.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c8a4131df55ca27-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.615 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:43:07.615 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:43:07.616 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.616 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.616 DEBUG:boto:Method: HEAD 2023-09-22T21:43:07.616 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.616 DEBUG:boto:Data: 2023-09-22T21:43:07.616 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.616 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.616 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.616 DEBUG:boto:Params: {} 2023-09-22T21:43:07.617 DEBUG:boto:Token: None 2023-09-22T21:43:07.617 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.617 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LKKH/JQpSkMLMf1YS4genk10WlE= 2023-09-22T21:43:07.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LKKH/JQpSkMLMf1YS4genk10WlE=', 'Content-Length': '0'} 2023-09-22T21:43:07.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c351b17d43988b5-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.620 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:43:07.620 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:43:07.620 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.620 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.620 DEBUG:boto:Method: HEAD 2023-09-22T21:43:07.620 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.621 DEBUG:boto:Data: 2023-09-22T21:43:07.621 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.621 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.621 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.621 DEBUG:boto:Params: {} 2023-09-22T21:43:07.621 DEBUG:boto:Token: None 2023-09-22T21:43:07.621 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.621 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LKKH/JQpSkMLMf1YS4genk10WlE= 2023-09-22T21:43:07.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LKKH/JQpSkMLMf1YS4genk10WlE=', 'Content-Length': '0'} 2023-09-22T21:43:07.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000167d7fc3b8a89360-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T21:43:07.624 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:43:07.625 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:43:07.625 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.625 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.625 DEBUG:boto:Method: GET 2023-09-22T21:43:07.625 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.625 DEBUG:boto:Data: 2023-09-22T21:43:07.625 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.625 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.625 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.625 DEBUG:boto:Params: {} 2023-09-22T21:43:07.626 DEBUG:boto:Token: None 2023-09-22T21:43:07.626 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.626 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jFDCaCLnabO/Th7veVUgQLSsinI= 2023-09-22T21:43:07.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jFDCaCLnabO/Th7veVUgQLSsinI=', 'Content-Length': '0'} 2023-09-22T21:43:07.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006949ac0ca257d55a-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.674 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:43:07.675 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:43:07.675 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.675 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.675 DEBUG:boto:Method: GET 2023-09-22T21:43:07.675 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.675 DEBUG:boto:Data: 2023-09-22T21:43:07.675 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.676 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.676 DEBUG:boto:Params: {} 2023-09-22T21:43:07.676 DEBUG:boto:Token: None 2023-09-22T21:43:07.676 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:43:07.676 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jFDCaCLnabO/Th7veVUgQLSsinI= 2023-09-22T21:43:07.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jFDCaCLnabO/Th7veVUgQLSsinI=', 'Content-Length': '0'} 2023-09-22T21:43:07.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075087fa8adca2224-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.679 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:07.679 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:07.679 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:43:07.680 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:43:07.680 DEBUG:boto:Method: GET 2023-09-22T21:43:07.680 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:43:07.680 DEBUG:boto:Data: 2023-09-22T21:43:07.680 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.680 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.680 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.680 DEBUG:boto:Params: {} 2023-09-22T21:43:07.680 DEBUG:boto:Token: None 2023-09-22T21:43:07.680 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/ 2023-09-22T21:43:07.681 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E= 2023-09-22T21:43:07.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E=', 'Content-Length': '0'} 2023-09-22T21:43:07.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ded22905f3c162c7-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.685 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:07.686 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:07.686 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:07.686 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:43:07.686 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:43:07.686 DEBUG:boto:Method: GET 2023-09-22T21:43:07.687 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:43:07.687 DEBUG:boto:Data: 2023-09-22T21:43:07.687 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.687 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.687 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.687 DEBUG:boto:Params: {} 2023-09-22T21:43:07.687 DEBUG:boto:Token: None 2023-09-22T21:43:07.687 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/ 2023-09-22T21:43:07.687 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E= 2023-09-22T21:43:07.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E=', 'Content-Length': '0'} 2023-09-22T21:43:07.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4a901dbb4d0e418-00650e0a6b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.742 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:07.743 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:43:07.743 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:43:07.743 DEBUG:boto:Method: HEAD 2023-09-22T21:43:07.743 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:43:07.743 DEBUG:boto:Data: 2023-09-22T21:43:07.743 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.743 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.743 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.744 DEBUG:boto:Params: {} 2023-09-22T21:43:07.744 DEBUG:boto:Token: None 2023-09-22T21:43:07.744 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj19 2023-09-22T21:43:07.744 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/VtxuN3GOJqVyrN0YMdlhHR9YvM= 2023-09-22T21:43:07.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/VtxuN3GOJqVyrN0YMdlhHR9YvM=', 'Content-Length': '0'} 2023-09-22T21:43:07.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f417640e870c9ae-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.747 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:43:07.747 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:43:07.747 DEBUG:boto:Method: HEAD 2023-09-22T21:43:07.747 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:43:07.748 DEBUG:boto:Data: 2023-09-22T21:43:07.748 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.748 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.748 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.748 DEBUG:boto:Params: {} 2023-09-22T21:43:07.748 DEBUG:boto:Token: None 2023-09-22T21:43:07.748 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj19 2023-09-22T21:43:07.748 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/VtxuN3GOJqVyrN0YMdlhHR9YvM= 2023-09-22T21:43:07.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/VtxuN3GOJqVyrN0YMdlhHR9YvM=', 'Content-Length': '0'} 2023-09-22T21:43:07.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c29c160da06aa082-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T21:43:07.752 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:43:07.752 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:43:07.752 DEBUG:boto:Method: GET 2023-09-22T21:43:07.752 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:43:07.752 DEBUG:boto:Data: 2023-09-22T21:43:07.752 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.752 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.752 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.752 DEBUG:boto:Params: {} 2023-09-22T21:43:07.753 DEBUG:boto:Token: None 2023-09-22T21:43:07.753 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj19 2023-09-22T21:43:07.753 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m9cn96xWFPbj/FCjXdxl68uYYDk= 2023-09-22T21:43:07.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m9cn96xWFPbj/FCjXdxl68uYYDk=', 'Content-Length': '0'} 2023-09-22T21:43:07.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039759f36cc79e922-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.798 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:43:07.799 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:43:07.799 DEBUG:boto:Method: GET 2023-09-22T21:43:07.799 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:43:07.799 DEBUG:boto:Data: 2023-09-22T21:43:07.799 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.799 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.799 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.799 DEBUG:boto:Params: {} 2023-09-22T21:43:07.800 DEBUG:boto:Token: None 2023-09-22T21:43:07.800 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj19 2023-09-22T21:43:07.800 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m9cn96xWFPbj/FCjXdxl68uYYDk= 2023-09-22T21:43:07.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m9cn96xWFPbj/FCjXdxl68uYYDk=', 'Content-Length': '0'} 2023-09-22T21:43:07.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000376afe9a4537644d-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.803 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:07.803 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:07.803 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:43:07.803 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:43:07.803 DEBUG:boto:Method: GET 2023-09-22T21:43:07.804 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:43:07.804 DEBUG:boto:Data: 2023-09-22T21:43:07.804 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.804 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.804 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.804 DEBUG:boto:Params: {} 2023-09-22T21:43:07.804 DEBUG:boto:Token: None 2023-09-22T21:43:07.804 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/ 2023-09-22T21:43:07.804 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E= 2023-09-22T21:43:07.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E=', 'Content-Length': '0'} 2023-09-22T21:43:07.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006366680822d7d883-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.854 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:07.855 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:07.855 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:07.856 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:43:07.856 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:43:07.856 DEBUG:boto:Method: GET 2023-09-22T21:43:07.856 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:43:07.856 DEBUG:boto:Data: 2023-09-22T21:43:07.857 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.857 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.857 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.857 DEBUG:boto:Params: {} 2023-09-22T21:43:07.857 DEBUG:boto:Token: None 2023-09-22T21:43:07.858 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/ 2023-09-22T21:43:07.858 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E= 2023-09-22T21:43:07.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E=', 'Content-Length': '0'} 2023-09-22T21:43:07.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021a1ccdc849ab308-00650e0a6b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.870 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:07.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:43:07.871 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:43:07.871 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:43:07.872 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.872 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.872 DEBUG:boto:Method: GET 2023-09-22T21:43:07.872 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.872 DEBUG:boto:Data: 2023-09-22T21:43:07.873 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.873 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.873 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.873 DEBUG:boto:Params: {} 2023-09-22T21:43:07.873 DEBUG:boto:Token: None 2023-09-22T21:43:07.874 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.874 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Mdu0nUWje7IDc3+l1r8FCQGnph8= 2023-09-22T21:43:07.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Mdu0nUWje7IDc3+l1r8FCQGnph8=', 'Content-Length': '0'} 2023-09-22T21:43:07.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bb86b2d0228b8be-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.877 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:43:07.877 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:43:07.877 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.878 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.878 DEBUG:boto:Method: GET 2023-09-22T21:43:07.878 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.878 DEBUG:boto:Data: 2023-09-22T21:43:07.878 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.879 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.879 DEBUG:boto:Params: {} 2023-09-22T21:43:07.879 DEBUG:boto:Token: None 2023-09-22T21:43:07.879 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.880 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Mdu0nUWje7IDc3+l1r8FCQGnph8= 2023-09-22T21:43:07.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Mdu0nUWje7IDc3+l1r8FCQGnph8=', 'Content-Length': '0'} 2023-09-22T21:43:07.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0c1b909d9b2af73-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.883 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:43:07.883 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:43:07.883 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.883 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.883 DEBUG:boto:Method: HEAD 2023-09-22T21:43:07.884 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.884 DEBUG:boto:Data: 2023-09-22T21:43:07.884 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.884 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.884 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.884 DEBUG:boto:Params: {} 2023-09-22T21:43:07.884 DEBUG:boto:Token: None 2023-09-22T21:43:07.884 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.885 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YUaMiZ455o4g/lmRSvZvHzsc65U= 2023-09-22T21:43:07.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YUaMiZ455o4g/lmRSvZvHzsc65U=', 'Content-Length': '0'} 2023-09-22T21:43:07.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007600f27c985e121-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.887 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:43:07.888 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:43:07.888 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.888 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.888 DEBUG:boto:Method: HEAD 2023-09-22T21:43:07.888 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.888 DEBUG:boto:Data: 2023-09-22T21:43:07.888 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.888 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.888 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.888 DEBUG:boto:Params: {} 2023-09-22T21:43:07.889 DEBUG:boto:Token: None 2023-09-22T21:43:07.889 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.889 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YUaMiZ455o4g/lmRSvZvHzsc65U= 2023-09-22T21:43:07.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YUaMiZ455o4g/lmRSvZvHzsc65U=', 'Content-Length': '0'} 2023-09-22T21:43:07.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8b32cc373b6bbd6-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:43:07.892 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:43:07.892 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:43:07.892 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.892 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.892 DEBUG:boto:Method: GET 2023-09-22T21:43:07.892 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.892 DEBUG:boto:Data: 2023-09-22T21:43:07.892 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.893 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.893 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.893 DEBUG:boto:Params: {} 2023-09-22T21:43:07.893 DEBUG:boto:Token: None 2023-09-22T21:43:07.893 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.893 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Mdu0nUWje7IDc3+l1r8FCQGnph8= 2023-09-22T21:43:07.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Mdu0nUWje7IDc3+l1r8FCQGnph8=', 'Content-Length': '0'} 2023-09-22T21:43:07.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040d51caf4b4920ae-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.938 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:43:07.938 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:43:07.938 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.939 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.939 DEBUG:boto:Method: GET 2023-09-22T21:43:07.939 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.939 DEBUG:boto:Data: 2023-09-22T21:43:07.939 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.939 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:07.939 DEBUG:boto:Port: 8000 2023-09-22T21:43:07.939 DEBUG:boto:Params: {} 2023-09-22T21:43:07.940 DEBUG:boto:Token: None 2023-09-22T21:43:07.940 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:43:07.940 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Mdu0nUWje7IDc3+l1r8FCQGnph8= 2023-09-22T21:43:07.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Mdu0nUWje7IDc3+l1r8FCQGnph8=', 'Content-Length': '0'} 2023-09-22T21:43:07.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4b4a553ccd4dee4-00650e0a6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.943 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:07.944 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:07.944 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:43:07.944 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:43:07.944 DEBUG:boto:Method: GET 2023-09-22T21:43:07.944 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:43:07.945 DEBUG:boto:Data: 2023-09-22T21:43:07.945 DEBUG:boto:Headers: {} 2023-09-22T21:43:07.945 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:07.945 DEBUG:boto:Port: 8001 2023-09-22T21:43:07.945 DEBUG:boto:Params: {} 2023-09-22T21:43:07.945 DEBUG:boto:Token: None 2023-09-22T21:43:07.945 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:07 GMT /ffzdqu-59/ 2023-09-22T21:43:07.945 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E= 2023-09-22T21:43:07.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3tXtTBsOFQW0Qkv7rHk7gyKri3E=', 'Content-Length': '0'} 2023-09-22T21:43:07.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087db96d823ac7955-00650e0a6b-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:07.998 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:07.999 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:07.999 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:07.999 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:43:07.999 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:43:08.000 DEBUG:boto:Method: GET 2023-09-22T21:43:08.000 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:43:08.000 DEBUG:boto:Data: 2023-09-22T21:43:08.000 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.000 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.000 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.001 DEBUG:boto:Params: {} 2023-09-22T21:43:08.001 DEBUG:boto:Token: None 2023-09-22T21:43:08.001 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/ 2023-09-22T21:43:08.002 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI= 2023-09-22T21:43:08.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI=', 'Content-Length': '0'} 2023-09-22T21:43:08.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0f81afefd8c039b-00650e0a6c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.009 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:08.009 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:43:08.009 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:43:08.010 DEBUG:boto:Method: HEAD 2023-09-22T21:43:08.010 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:43:08.010 DEBUG:boto:Data: 2023-09-22T21:43:08.010 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.010 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.010 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.010 DEBUG:boto:Params: {} 2023-09-22T21:43:08.010 DEBUG:boto:Token: None 2023-09-22T21:43:08.011 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj22 2023-09-22T21:43:08.011 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hahUTvK1o4j29j0HRVHhQVkD93k= 2023-09-22T21:43:08.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hahUTvK1o4j29j0HRVHhQVkD93k=', 'Content-Length': '0'} 2023-09-22T21:43:08.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009aef5667c2689c87-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.014 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:43:08.014 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:43:08.014 DEBUG:boto:Method: HEAD 2023-09-22T21:43:08.014 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:43:08.014 DEBUG:boto:Data: 2023-09-22T21:43:08.014 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.014 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.014 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.015 DEBUG:boto:Params: {} 2023-09-22T21:43:08.015 DEBUG:boto:Token: None 2023-09-22T21:43:08.015 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj22 2023-09-22T21:43:08.015 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hahUTvK1o4j29j0HRVHhQVkD93k= 2023-09-22T21:43:08.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hahUTvK1o4j29j0HRVHhQVkD93k=', 'Content-Length': '0'} 2023-09-22T21:43:08.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa9a47ab63494d43-00650e0a6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:43:08.018 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:43:08.018 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:43:08.018 DEBUG:boto:Method: GET 2023-09-22T21:43:08.018 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:43:08.018 DEBUG:boto:Data: 2023-09-22T21:43:08.018 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.018 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.018 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.019 DEBUG:boto:Params: {} 2023-09-22T21:43:08.019 DEBUG:boto:Token: None 2023-09-22T21:43:08.019 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj22 2023-09-22T21:43:08.019 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5KIuGe0O/Ddde7byIT7v9NqUjq8= 2023-09-22T21:43:08.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5KIuGe0O/Ddde7byIT7v9NqUjq8=', 'Content-Length': '0'} 2023-09-22T21:43:08.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fda1d21c1f4f2809-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.066 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:43:08.067 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:43:08.067 DEBUG:boto:Method: GET 2023-09-22T21:43:08.067 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:43:08.067 DEBUG:boto:Data: 2023-09-22T21:43:08.067 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.067 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.067 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.067 DEBUG:boto:Params: {} 2023-09-22T21:43:08.067 DEBUG:boto:Token: None 2023-09-22T21:43:08.068 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj22 2023-09-22T21:43:08.068 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5KIuGe0O/Ddde7byIT7v9NqUjq8= 2023-09-22T21:43:08.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5KIuGe0O/Ddde7byIT7v9NqUjq8=', 'Content-Length': '0'} 2023-09-22T21:43:08.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002be0c59db4196854-00650e0a6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.071 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:08.071 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:08.071 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:43:08.071 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:43:08.072 DEBUG:boto:Method: GET 2023-09-22T21:43:08.072 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:43:08.072 DEBUG:boto:Data: 2023-09-22T21:43:08.072 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.072 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.072 DEBUG:boto:Params: {} 2023-09-22T21:43:08.072 DEBUG:boto:Token: None 2023-09-22T21:43:08.072 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/ 2023-09-22T21:43:08.073 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI= 2023-09-22T21:43:08.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI=', 'Content-Length': '0'} 2023-09-22T21:43:08.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8c51989a50d788c-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.077 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:08.078 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:08.078 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:08.078 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:43:08.078 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:43:08.078 DEBUG:boto:Method: GET 2023-09-22T21:43:08.078 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:43:08.078 DEBUG:boto:Data: 2023-09-22T21:43:08.078 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.078 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.078 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.079 DEBUG:boto:Params: {} 2023-09-22T21:43:08.079 DEBUG:boto:Token: None 2023-09-22T21:43:08.079 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/ 2023-09-22T21:43:08.079 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI= 2023-09-22T21:43:08.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI=', 'Content-Length': '0'} 2023-09-22T21:43:08.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053ff70ddfe9996ed-00650e0a6c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.086 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:08.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-22T21:43:08.087 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:43:08.087 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:43:08.087 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.087 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.087 DEBUG:boto:Method: GET 2023-09-22T21:43:08.087 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.087 DEBUG:boto:Data: 2023-09-22T21:43:08.087 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.087 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.088 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.088 DEBUG:boto:Params: {} 2023-09-22T21:43:08.088 DEBUG:boto:Token: None 2023-09-22T21:43:08.088 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.088 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rdhWNFKW9qk8FYm0OJS3/Yp5bVI= 2023-09-22T21:43:08.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rdhWNFKW9qk8FYm0OJS3/Yp5bVI=', 'Content-Length': '0'} 2023-09-22T21:43:08.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e70fc0a9da282ca3-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.134 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:43:08.135 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:43:08.135 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.135 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.135 DEBUG:boto:Method: GET 2023-09-22T21:43:08.135 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.135 DEBUG:boto:Data: 2023-09-22T21:43:08.135 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.135 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.135 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.135 DEBUG:boto:Params: {} 2023-09-22T21:43:08.136 DEBUG:boto:Token: None 2023-09-22T21:43:08.136 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.136 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rdhWNFKW9qk8FYm0OJS3/Yp5bVI= 2023-09-22T21:43:08.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rdhWNFKW9qk8FYm0OJS3/Yp5bVI=', 'Content-Length': '0'} 2023-09-22T21:43:08.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023d383351748b45d-00650e0a6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.139 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:43:08.139 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:43:08.139 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.140 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.140 DEBUG:boto:Method: HEAD 2023-09-22T21:43:08.140 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.140 DEBUG:boto:Data: 2023-09-22T21:43:08.140 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.140 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.140 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.140 DEBUG:boto:Params: {} 2023-09-22T21:43:08.140 DEBUG:boto:Token: None 2023-09-22T21:43:08.141 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.141 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XsoZqpGgmnOQ05nj8j3ogYQEjBg= 2023-09-22T21:43:08.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XsoZqpGgmnOQ05nj8j3ogYQEjBg=', 'Content-Length': '0'} 2023-09-22T21:43:08.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009475117d51ccaa99-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.144 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:43:08.144 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:43:08.144 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.144 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.144 DEBUG:boto:Method: HEAD 2023-09-22T21:43:08.144 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.144 DEBUG:boto:Data: 2023-09-22T21:43:08.144 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.145 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.145 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.145 DEBUG:boto:Params: {} 2023-09-22T21:43:08.145 DEBUG:boto:Token: None 2023-09-22T21:43:08.145 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.145 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XsoZqpGgmnOQ05nj8j3ogYQEjBg= 2023-09-22T21:43:08.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XsoZqpGgmnOQ05nj8j3ogYQEjBg=', 'Content-Length': '0'} 2023-09-22T21:43:08.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3b8be85dce91ece-00650e0a6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-22T21:43:08.148 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:43:08.148 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:43:08.148 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.149 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.149 DEBUG:boto:Method: GET 2023-09-22T21:43:08.149 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.149 DEBUG:boto:Data: 2023-09-22T21:43:08.149 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.149 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.149 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.149 DEBUG:boto:Params: {} 2023-09-22T21:43:08.149 DEBUG:boto:Token: None 2023-09-22T21:43:08.150 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.150 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rdhWNFKW9qk8FYm0OJS3/Yp5bVI= 2023-09-22T21:43:08.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rdhWNFKW9qk8FYm0OJS3/Yp5bVI=', 'Content-Length': '0'} 2023-09-22T21:43:08.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b97583f43e810d99-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.194 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:43:08.195 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:43:08.195 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.195 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.195 DEBUG:boto:Method: GET 2023-09-22T21:43:08.195 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.195 DEBUG:boto:Data: 2023-09-22T21:43:08.195 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.195 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.195 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.196 DEBUG:boto:Params: {} 2023-09-22T21:43:08.196 DEBUG:boto:Token: None 2023-09-22T21:43:08.196 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:43:08.196 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rdhWNFKW9qk8FYm0OJS3/Yp5bVI= 2023-09-22T21:43:08.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rdhWNFKW9qk8FYm0OJS3/Yp5bVI=', 'Content-Length': '0'} 2023-09-22T21:43:08.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b5d6f689906019b-00650e0a6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.199 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:08.199 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:08.199 DEBUG:boto:path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:43:08.199 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:43:08.200 DEBUG:boto:Method: GET 2023-09-22T21:43:08.200 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj23 2023-09-22T21:43:08.200 DEBUG:boto:Data: 2023-09-22T21:43:08.200 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.200 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.200 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.200 DEBUG:boto:Params: {} 2023-09-22T21:43:08.200 DEBUG:boto:Token: None 2023-09-22T21:43:08.200 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/ 2023-09-22T21:43:08.201 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI= 2023-09-22T21:43:08.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI=', 'Content-Length': '0'} 2023-09-22T21:43:08.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002af3f4b8c0bfd62f-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.250 DEBUG:boto:b'ffzdqu-59obj231000falseobj232023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:08.251 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:08.251 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:08.251 DEBUG:boto:path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:43:08.251 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:43:08.251 DEBUG:boto:Method: GET 2023-09-22T21:43:08.251 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj23 2023-09-22T21:43:08.252 DEBUG:boto:Data: 2023-09-22T21:43:08.252 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.252 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.252 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.252 DEBUG:boto:Params: {} 2023-09-22T21:43:08.252 DEBUG:boto:Token: None 2023-09-22T21:43:08.252 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/ 2023-09-22T21:43:08.252 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI= 2023-09-22T21:43:08.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI=', 'Content-Length': '0'} 2023-09-22T21:43:08.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078a75010fe579bea-00650e0a6c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.260 DEBUG:boto:b'ffzdqu-59obj231000falseobj232023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:08.260 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:43:08.260 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:43:08.260 DEBUG:boto:Method: HEAD 2023-09-22T21:43:08.260 DEBUG:boto:Path: /ffzdqu-59/obj23 2023-09-22T21:43:08.261 DEBUG:boto:Data: 2023-09-22T21:43:08.261 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.261 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.261 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.261 DEBUG:boto:Params: {} 2023-09-22T21:43:08.261 DEBUG:boto:Token: None 2023-09-22T21:43:08.261 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj23 2023-09-22T21:43:08.261 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MHhlbji9ktOkaQQe9X1FqVRYHTI= 2023-09-22T21:43:08.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MHhlbji9ktOkaQQe9X1FqVRYHTI=', 'Content-Length': '0'} 2023-09-22T21:43:08.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a69602eaeaf4e55-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.264 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:43:08.264 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:43:08.264 DEBUG:boto:Method: HEAD 2023-09-22T21:43:08.265 DEBUG:boto:Path: /ffzdqu-59/obj23 2023-09-22T21:43:08.265 DEBUG:boto:Data: 2023-09-22T21:43:08.265 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.265 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.265 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.265 DEBUG:boto:Params: {} 2023-09-22T21:43:08.265 DEBUG:boto:Token: None 2023-09-22T21:43:08.265 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj23 2023-09-22T21:43:08.265 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MHhlbji9ktOkaQQe9X1FqVRYHTI= 2023-09-22T21:43:08.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MHhlbji9ktOkaQQe9X1FqVRYHTI=', 'Content-Length': '0'} 2023-09-22T21:43:08.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043656ed6630ad795-00650e0a6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-22T21:43:08.268 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:43:08.268 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:43:08.269 DEBUG:boto:Method: GET 2023-09-22T21:43:08.269 DEBUG:boto:Path: /ffzdqu-59/obj23 2023-09-22T21:43:08.269 DEBUG:boto:Data: 2023-09-22T21:43:08.269 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.269 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.269 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.269 DEBUG:boto:Params: {} 2023-09-22T21:43:08.269 DEBUG:boto:Token: None 2023-09-22T21:43:08.269 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj23 2023-09-22T21:43:08.270 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1IJnJJesSPnPjdb2otWYzQW3eQw= 2023-09-22T21:43:08.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1IJnJJesSPnPjdb2otWYzQW3eQw=', 'Content-Length': '0'} 2023-09-22T21:43:08.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000088687bd4e593e06-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.318 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:43:08.318 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:43:08.319 DEBUG:boto:Method: GET 2023-09-22T21:43:08.319 DEBUG:boto:Path: /ffzdqu-59/obj23 2023-09-22T21:43:08.319 DEBUG:boto:Data: 2023-09-22T21:43:08.319 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.319 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.319 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.319 DEBUG:boto:Params: {} 2023-09-22T21:43:08.320 DEBUG:boto:Token: None 2023-09-22T21:43:08.320 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj23 2023-09-22T21:43:08.320 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1IJnJJesSPnPjdb2otWYzQW3eQw= 2023-09-22T21:43:08.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1IJnJJesSPnPjdb2otWYzQW3eQw=', 'Content-Length': '0'} 2023-09-22T21:43:08.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087bffb02faf36ea7-00650e0a6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.323 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:08.323 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:08.323 DEBUG:boto:path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:43:08.324 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:43:08.324 DEBUG:boto:Method: GET 2023-09-22T21:43:08.324 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj23 2023-09-22T21:43:08.324 DEBUG:boto:Data: 2023-09-22T21:43:08.324 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.324 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.325 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.325 DEBUG:boto:Params: {} 2023-09-22T21:43:08.325 DEBUG:boto:Token: None 2023-09-22T21:43:08.325 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/ 2023-09-22T21:43:08.325 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI= 2023-09-22T21:43:08.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI=', 'Content-Length': '0'} 2023-09-22T21:43:08.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ce1125d65767cfb-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.374 DEBUG:boto:b'ffzdqu-59obj231000falseobj232023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:08.375 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:08.375 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:08.375 DEBUG:boto:path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:43:08.375 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:43:08.375 DEBUG:boto:Method: GET 2023-09-22T21:43:08.375 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj23 2023-09-22T21:43:08.376 DEBUG:boto:Data: 2023-09-22T21:43:08.376 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.376 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.376 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.376 DEBUG:boto:Params: {} 2023-09-22T21:43:08.376 DEBUG:boto:Token: None 2023-09-22T21:43:08.376 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/ 2023-09-22T21:43:08.376 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI= 2023-09-22T21:43:08.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI=', 'Content-Length': '0'} 2023-09-22T21:43:08.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000898da3413d7dcc6c-00650e0a6c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.383 DEBUG:boto:b'ffzdqu-59obj231000falseobj232023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:08.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:43:08.384 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:43:08.384 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:43:08.384 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.384 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.384 DEBUG:boto:Method: GET 2023-09-22T21:43:08.384 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.384 DEBUG:boto:Data: 2023-09-22T21:43:08.385 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.385 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.385 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.385 DEBUG:boto:Params: {} 2023-09-22T21:43:08.385 DEBUG:boto:Token: None 2023-09-22T21:43:08.385 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.385 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:laiAISGBUIdilEwJHdB0lFpA98o= 2023-09-22T21:43:08.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:laiAISGBUIdilEwJHdB0lFpA98o=', 'Content-Length': '0'} 2023-09-22T21:43:08.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7f70872f6be03b8-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.388 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:43:08.389 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:43:08.389 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.389 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.389 DEBUG:boto:Method: GET 2023-09-22T21:43:08.389 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.389 DEBUG:boto:Data: 2023-09-22T21:43:08.389 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.389 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.389 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.390 DEBUG:boto:Params: {} 2023-09-22T21:43:08.390 DEBUG:boto:Token: None 2023-09-22T21:43:08.390 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.390 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:laiAISGBUIdilEwJHdB0lFpA98o= 2023-09-22T21:43:08.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:laiAISGBUIdilEwJHdB0lFpA98o=', 'Content-Length': '0'} 2023-09-22T21:43:08.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fdd65e4a12415f1-00650e0a6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.393 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:43:08.393 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:43:08.393 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.393 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.393 DEBUG:boto:Method: HEAD 2023-09-22T21:43:08.394 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.394 DEBUG:boto:Data: 2023-09-22T21:43:08.394 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.394 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.394 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.394 DEBUG:boto:Params: {} 2023-09-22T21:43:08.394 DEBUG:boto:Token: None 2023-09-22T21:43:08.394 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.394 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vmQg4ieyAYR3AFwvlzGnzzvGBaQ= 2023-09-22T21:43:08.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vmQg4ieyAYR3AFwvlzGnzzvGBaQ=', 'Content-Length': '0'} 2023-09-22T21:43:08.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a31456298bfad8f6-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.397 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:43:08.398 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:43:08.398 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.398 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.398 DEBUG:boto:Method: HEAD 2023-09-22T21:43:08.398 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.398 DEBUG:boto:Data: 2023-09-22T21:43:08.398 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.398 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.398 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.399 DEBUG:boto:Params: {} 2023-09-22T21:43:08.399 DEBUG:boto:Token: None 2023-09-22T21:43:08.399 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.399 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vmQg4ieyAYR3AFwvlzGnzzvGBaQ= 2023-09-22T21:43:08.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vmQg4ieyAYR3AFwvlzGnzzvGBaQ=', 'Content-Length': '0'} 2023-09-22T21:43:08.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041e650cbe7ecc783-00650e0a6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:43:08.402 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:43:08.402 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:43:08.403 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.403 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.403 DEBUG:boto:Method: GET 2023-09-22T21:43:08.403 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.403 DEBUG:boto:Data: 2023-09-22T21:43:08.403 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.403 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.403 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.403 DEBUG:boto:Params: {} 2023-09-22T21:43:08.403 DEBUG:boto:Token: None 2023-09-22T21:43:08.404 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.404 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:laiAISGBUIdilEwJHdB0lFpA98o= 2023-09-22T21:43:08.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:laiAISGBUIdilEwJHdB0lFpA98o=', 'Content-Length': '0'} 2023-09-22T21:43:08.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e95768892dc61415-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.450 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:43:08.450 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:43:08.451 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.451 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.451 DEBUG:boto:Method: GET 2023-09-22T21:43:08.451 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.451 DEBUG:boto:Data: 2023-09-22T21:43:08.451 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.451 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.451 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.451 DEBUG:boto:Params: {} 2023-09-22T21:43:08.451 DEBUG:boto:Token: None 2023-09-22T21:43:08.452 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:43:08.452 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:laiAISGBUIdilEwJHdB0lFpA98o= 2023-09-22T21:43:08.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:laiAISGBUIdilEwJHdB0lFpA98o=', 'Content-Length': '0'} 2023-09-22T21:43:08.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000451e1df2640ef3-00650e0a6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.455 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:08.455 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:08.455 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:43:08.455 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:43:08.455 DEBUG:boto:Method: GET 2023-09-22T21:43:08.455 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:43:08.455 DEBUG:boto:Data: 2023-09-22T21:43:08.456 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.456 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.456 DEBUG:boto:Params: {} 2023-09-22T21:43:08.456 DEBUG:boto:Token: None 2023-09-22T21:43:08.456 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/ 2023-09-22T21:43:08.456 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI= 2023-09-22T21:43:08.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI=', 'Content-Length': '0'} 2023-09-22T21:43:08.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b71d90ce0fa80e90-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.502 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:08.503 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:08.503 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:08.503 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:43:08.503 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:43:08.503 DEBUG:boto:Method: GET 2023-09-22T21:43:08.503 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:43:08.504 DEBUG:boto:Data: 2023-09-22T21:43:08.504 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.504 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.504 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.504 DEBUG:boto:Params: {} 2023-09-22T21:43:08.504 DEBUG:boto:Token: None 2023-09-22T21:43:08.504 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/ 2023-09-22T21:43:08.504 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI= 2023-09-22T21:43:08.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI=', 'Content-Length': '0'} 2023-09-22T21:43:08.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c9a8a133955fe41-00650e0a6c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.512 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:08.512 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:43:08.512 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:43:08.512 DEBUG:boto:Method: HEAD 2023-09-22T21:43:08.512 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:43:08.512 DEBUG:boto:Data: 2023-09-22T21:43:08.513 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.513 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.513 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.513 DEBUG:boto:Params: {} 2023-09-22T21:43:08.513 DEBUG:boto:Token: None 2023-09-22T21:43:08.513 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj33 2023-09-22T21:43:08.513 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iaNneG48nABobWqStKdN8zv0Zp8= 2023-09-22T21:43:08.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iaNneG48nABobWqStKdN8zv0Zp8=', 'Content-Length': '0'} 2023-09-22T21:43:08.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bd37f274534561c-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.516 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:43:08.516 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:43:08.516 DEBUG:boto:Method: HEAD 2023-09-22T21:43:08.517 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:43:08.517 DEBUG:boto:Data: 2023-09-22T21:43:08.517 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.517 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.517 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.517 DEBUG:boto:Params: {} 2023-09-22T21:43:08.517 DEBUG:boto:Token: None 2023-09-22T21:43:08.517 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj33 2023-09-22T21:43:08.517 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iaNneG48nABobWqStKdN8zv0Zp8= 2023-09-22T21:43:08.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iaNneG48nABobWqStKdN8zv0Zp8=', 'Content-Length': '0'} 2023-09-22T21:43:08.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d10db32970a76d6b-00650e0a6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:43:08.520 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:43:08.521 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:43:08.521 DEBUG:boto:Method: GET 2023-09-22T21:43:08.521 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:43:08.521 DEBUG:boto:Data: 2023-09-22T21:43:08.521 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.521 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.521 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.521 DEBUG:boto:Params: {} 2023-09-22T21:43:08.521 DEBUG:boto:Token: None 2023-09-22T21:43:08.522 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj33 2023-09-22T21:43:08.522 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GvoIY0rlFTTBPsSAqVNJ/BfeI9M= 2023-09-22T21:43:08.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GvoIY0rlFTTBPsSAqVNJ/BfeI9M=', 'Content-Length': '0'} 2023-09-22T21:43:08.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e874d7f6fc70318-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.566 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:43:08.567 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:43:08.567 DEBUG:boto:Method: GET 2023-09-22T21:43:08.567 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:43:08.567 DEBUG:boto:Data: 2023-09-22T21:43:08.567 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.567 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.567 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.567 DEBUG:boto:Params: {} 2023-09-22T21:43:08.568 DEBUG:boto:Token: None 2023-09-22T21:43:08.568 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj33 2023-09-22T21:43:08.568 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GvoIY0rlFTTBPsSAqVNJ/BfeI9M= 2023-09-22T21:43:08.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GvoIY0rlFTTBPsSAqVNJ/BfeI9M=', 'Content-Length': '0'} 2023-09-22T21:43:08.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ad59db411751c64-00650e0a6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.571 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:08.571 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:08.571 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:43:08.572 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:43:08.572 DEBUG:boto:Method: GET 2023-09-22T21:43:08.572 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:43:08.572 DEBUG:boto:Data: 2023-09-22T21:43:08.572 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.572 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.572 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.572 DEBUG:boto:Params: {} 2023-09-22T21:43:08.572 DEBUG:boto:Token: None 2023-09-22T21:43:08.573 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/ 2023-09-22T21:43:08.573 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI= 2023-09-22T21:43:08.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI=', 'Content-Length': '0'} 2023-09-22T21:43:08.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007442b069bb8efc23-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.618 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:08.619 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:08.619 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:08.619 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:43:08.619 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:43:08.620 DEBUG:boto:Method: GET 2023-09-22T21:43:08.620 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:43:08.620 DEBUG:boto:Data: 2023-09-22T21:43:08.620 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.620 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.620 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.620 DEBUG:boto:Params: {} 2023-09-22T21:43:08.620 DEBUG:boto:Token: None 2023-09-22T21:43:08.621 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/ 2023-09-22T21:43:08.621 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI= 2023-09-22T21:43:08.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI=', 'Content-Length': '0'} 2023-09-22T21:43:08.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a1d3e420351281b-00650e0a6c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.628 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:08.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:43:08.628 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:43:08.628 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:43:08.628 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.628 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.629 DEBUG:boto:Method: GET 2023-09-22T21:43:08.629 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.629 DEBUG:boto:Data: 2023-09-22T21:43:08.629 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.629 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.629 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.629 DEBUG:boto:Params: {} 2023-09-22T21:43:08.629 DEBUG:boto:Token: None 2023-09-22T21:43:08.629 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.630 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Dl99HqHQvFb90F9tlQK6hWtT6n8= 2023-09-22T21:43:08.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Dl99HqHQvFb90F9tlQK6hWtT6n8=', 'Content-Length': '0'} 2023-09-22T21:43:08.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b154db3daec4185-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.633 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:43:08.633 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:43:08.633 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.633 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.633 DEBUG:boto:Method: GET 2023-09-22T21:43:08.633 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.634 DEBUG:boto:Data: 2023-09-22T21:43:08.634 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.634 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.634 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.634 DEBUG:boto:Params: {} 2023-09-22T21:43:08.634 DEBUG:boto:Token: None 2023-09-22T21:43:08.634 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.635 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Dl99HqHQvFb90F9tlQK6hWtT6n8= 2023-09-22T21:43:08.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Dl99HqHQvFb90F9tlQK6hWtT6n8=', 'Content-Length': '0'} 2023-09-22T21:43:08.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e3aaebdc463242c-00650e0a6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.638 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:43:08.638 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:43:08.638 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.638 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.639 DEBUG:boto:Method: HEAD 2023-09-22T21:43:08.639 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.639 DEBUG:boto:Data: 2023-09-22T21:43:08.639 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.639 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.639 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.639 DEBUG:boto:Params: {} 2023-09-22T21:43:08.639 DEBUG:boto:Token: None 2023-09-22T21:43:08.640 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.640 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eNnT/QtVO0zIdjJIowW02rV1mBw= 2023-09-22T21:43:08.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eNnT/QtVO0zIdjJIowW02rV1mBw=', 'Content-Length': '0'} 2023-09-22T21:43:08.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d89e53ce22aa53e0-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.642 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:43:08.643 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:43:08.643 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.643 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.643 DEBUG:boto:Method: HEAD 2023-09-22T21:43:08.643 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.643 DEBUG:boto:Data: 2023-09-22T21:43:08.643 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.643 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.643 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.644 DEBUG:boto:Params: {} 2023-09-22T21:43:08.644 DEBUG:boto:Token: None 2023-09-22T21:43:08.644 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.644 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eNnT/QtVO0zIdjJIowW02rV1mBw= 2023-09-22T21:43:08.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eNnT/QtVO0zIdjJIowW02rV1mBw=', 'Content-Length': '0'} 2023-09-22T21:43:08.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b24ad886d6727018-00650e0a6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:43:08.647 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:43:08.647 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:43:08.647 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.647 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.647 DEBUG:boto:Method: GET 2023-09-22T21:43:08.647 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.648 DEBUG:boto:Data: 2023-09-22T21:43:08.648 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.648 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.648 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.648 DEBUG:boto:Params: {} 2023-09-22T21:43:08.648 DEBUG:boto:Token: None 2023-09-22T21:43:08.648 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.648 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Dl99HqHQvFb90F9tlQK6hWtT6n8= 2023-09-22T21:43:08.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Dl99HqHQvFb90F9tlQK6hWtT6n8=', 'Content-Length': '0'} 2023-09-22T21:43:08.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2360e230cdfb949-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.694 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:43:08.695 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:43:08.695 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.695 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.695 DEBUG:boto:Method: GET 2023-09-22T21:43:08.695 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.695 DEBUG:boto:Data: 2023-09-22T21:43:08.695 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.695 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.696 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.696 DEBUG:boto:Params: {} 2023-09-22T21:43:08.696 DEBUG:boto:Token: None 2023-09-22T21:43:08.696 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:43:08.696 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Dl99HqHQvFb90F9tlQK6hWtT6n8= 2023-09-22T21:43:08.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Dl99HqHQvFb90F9tlQK6hWtT6n8=', 'Content-Length': '0'} 2023-09-22T21:43:08.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000545ea13c4fddee20-00650e0a6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.699 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:08.699 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:08.700 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:43:08.700 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:43:08.700 DEBUG:boto:Method: GET 2023-09-22T21:43:08.700 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:43:08.700 DEBUG:boto:Data: 2023-09-22T21:43:08.700 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.700 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.700 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.700 DEBUG:boto:Params: {} 2023-09-22T21:43:08.701 DEBUG:boto:Token: None 2023-09-22T21:43:08.701 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/ 2023-09-22T21:43:08.701 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI= 2023-09-22T21:43:08.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI=', 'Content-Length': '0'} 2023-09-22T21:43:08.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce4761dea7ba89e4-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.746 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:08.747 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:08.747 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:08.747 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:43:08.747 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:43:08.747 DEBUG:boto:Method: GET 2023-09-22T21:43:08.747 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:43:08.747 DEBUG:boto:Data: 2023-09-22T21:43:08.748 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.748 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.748 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.748 DEBUG:boto:Params: {} 2023-09-22T21:43:08.748 DEBUG:boto:Token: None 2023-09-22T21:43:08.748 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/ 2023-09-22T21:43:08.748 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI= 2023-09-22T21:43:08.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI=', 'Content-Length': '0'} 2023-09-22T21:43:08.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6763ae6be550ab7-00650e0a6c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.759 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:08.760 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:43:08.760 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:43:08.760 DEBUG:boto:Method: HEAD 2023-09-22T21:43:08.760 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:43:08.760 DEBUG:boto:Data: 2023-09-22T21:43:08.760 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.760 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.760 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.761 DEBUG:boto:Params: {} 2023-09-22T21:43:08.761 DEBUG:boto:Token: None 2023-09-22T21:43:08.761 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj34 2023-09-22T21:43:08.761 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6pyEFsXI5ln27UyTQ3wbztp5N8c= 2023-09-22T21:43:08.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6pyEFsXI5ln27UyTQ3wbztp5N8c=', 'Content-Length': '0'} 2023-09-22T21:43:08.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4cb1a29311b9322-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.765 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:43:08.765 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:43:08.765 DEBUG:boto:Method: HEAD 2023-09-22T21:43:08.765 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:43:08.765 DEBUG:boto:Data: 2023-09-22T21:43:08.765 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.765 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.765 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.765 DEBUG:boto:Params: {} 2023-09-22T21:43:08.766 DEBUG:boto:Token: None 2023-09-22T21:43:08.766 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj34 2023-09-22T21:43:08.766 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6pyEFsXI5ln27UyTQ3wbztp5N8c= 2023-09-22T21:43:08.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6pyEFsXI5ln27UyTQ3wbztp5N8c=', 'Content-Length': '0'} 2023-09-22T21:43:08.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d64de1e24d1d9f48-00650e0a6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:43:08.769 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:43:08.769 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:43:08.769 DEBUG:boto:Method: GET 2023-09-22T21:43:08.769 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:43:08.769 DEBUG:boto:Data: 2023-09-22T21:43:08.770 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.770 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.770 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.770 DEBUG:boto:Params: {} 2023-09-22T21:43:08.770 DEBUG:boto:Token: None 2023-09-22T21:43:08.770 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj34 2023-09-22T21:43:08.770 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IhFg/lsOZonJftx2mn3WHpLQXeY= 2023-09-22T21:43:08.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IhFg/lsOZonJftx2mn3WHpLQXeY=', 'Content-Length': '0'} 2023-09-22T21:43:08.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003642e993ecdefbbc-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.814 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:43:08.815 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:43:08.815 DEBUG:boto:Method: GET 2023-09-22T21:43:08.815 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:43:08.815 DEBUG:boto:Data: 2023-09-22T21:43:08.815 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.815 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.815 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.815 DEBUG:boto:Params: {} 2023-09-22T21:43:08.815 DEBUG:boto:Token: None 2023-09-22T21:43:08.816 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj34 2023-09-22T21:43:08.816 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IhFg/lsOZonJftx2mn3WHpLQXeY= 2023-09-22T21:43:08.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IhFg/lsOZonJftx2mn3WHpLQXeY=', 'Content-Length': '0'} 2023-09-22T21:43:08.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f059fa278520698-00650e0a6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.819 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:08.819 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:08.819 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:43:08.819 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:43:08.819 DEBUG:boto:Method: GET 2023-09-22T21:43:08.820 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:43:08.820 DEBUG:boto:Data: 2023-09-22T21:43:08.820 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.820 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.820 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.820 DEBUG:boto:Params: {} 2023-09-22T21:43:08.820 DEBUG:boto:Token: None 2023-09-22T21:43:08.820 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/ 2023-09-22T21:43:08.820 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI= 2023-09-22T21:43:08.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI=', 'Content-Length': '0'} 2023-09-22T21:43:08.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed533769af6cc690-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.866 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:08.867 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:08.867 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:08.867 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:43:08.867 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:43:08.868 DEBUG:boto:Method: GET 2023-09-22T21:43:08.868 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:43:08.868 DEBUG:boto:Data: 2023-09-22T21:43:08.868 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.868 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.868 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.868 DEBUG:boto:Params: {} 2023-09-22T21:43:08.868 DEBUG:boto:Token: None 2023-09-22T21:43:08.869 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/ 2023-09-22T21:43:08.869 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI= 2023-09-22T21:43:08.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI=', 'Content-Length': '0'} 2023-09-22T21:43:08.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004da9f4086b023a37-00650e0a6c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.877 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:08.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T21:43:08.877 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:43:08.877 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:43:08.878 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.878 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.878 DEBUG:boto:Method: GET 2023-09-22T21:43:08.878 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.878 DEBUG:boto:Data: 2023-09-22T21:43:08.878 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.878 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.879 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.879 DEBUG:boto:Params: {} 2023-09-22T21:43:08.879 DEBUG:boto:Token: None 2023-09-22T21:43:08.879 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.879 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IGMEwppo+Jg5jCOd15QKpS9liiU= 2023-09-22T21:43:08.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IGMEwppo+Jg5jCOd15QKpS9liiU=', 'Content-Length': '0'} 2023-09-22T21:43:08.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031ea0db832add5ae-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.882 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:43:08.883 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:43:08.883 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.883 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.883 DEBUG:boto:Method: GET 2023-09-22T21:43:08.883 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.883 DEBUG:boto:Data: 2023-09-22T21:43:08.883 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.883 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.883 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.883 DEBUG:boto:Params: {} 2023-09-22T21:43:08.884 DEBUG:boto:Token: None 2023-09-22T21:43:08.884 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.884 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IGMEwppo+Jg5jCOd15QKpS9liiU= 2023-09-22T21:43:08.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IGMEwppo+Jg5jCOd15QKpS9liiU=', 'Content-Length': '0'} 2023-09-22T21:43:08.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7f61e423ea3e2ac-00650e0a6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.930 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:43:08.931 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:43:08.931 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.931 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.931 DEBUG:boto:Method: HEAD 2023-09-22T21:43:08.931 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.931 DEBUG:boto:Data: 2023-09-22T21:43:08.931 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.931 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.931 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.932 DEBUG:boto:Params: {} 2023-09-22T21:43:08.932 DEBUG:boto:Token: None 2023-09-22T21:43:08.932 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.932 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6+0nkyIYpRaiXZQmQBiRwdoWsz0= 2023-09-22T21:43:08.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6+0nkyIYpRaiXZQmQBiRwdoWsz0=', 'Content-Length': '0'} 2023-09-22T21:43:08.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6cef75fe06cbc84-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.935 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:43:08.935 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:43:08.936 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.936 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.936 DEBUG:boto:Method: HEAD 2023-09-22T21:43:08.936 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.936 DEBUG:boto:Data: 2023-09-22T21:43:08.936 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.936 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.936 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.937 DEBUG:boto:Params: {} 2023-09-22T21:43:08.937 DEBUG:boto:Token: None 2023-09-22T21:43:08.937 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.937 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6+0nkyIYpRaiXZQmQBiRwdoWsz0= 2023-09-22T21:43:08.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6+0nkyIYpRaiXZQmQBiRwdoWsz0=', 'Content-Length': '0'} 2023-09-22T21:43:08.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cff28fba3bad1f8d-00650e0a6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T21:43:08.940 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:43:08.940 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:43:08.941 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.941 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.941 DEBUG:boto:Method: GET 2023-09-22T21:43:08.941 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.941 DEBUG:boto:Data: 2023-09-22T21:43:08.941 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.941 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.941 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.942 DEBUG:boto:Params: {} 2023-09-22T21:43:08.942 DEBUG:boto:Token: None 2023-09-22T21:43:08.942 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.942 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IGMEwppo+Jg5jCOd15QKpS9liiU= 2023-09-22T21:43:08.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IGMEwppo+Jg5jCOd15QKpS9liiU=', 'Content-Length': '0'} 2023-09-22T21:43:08.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f01329d333fabe4-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.986 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:43:08.987 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:43:08.987 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.987 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.987 DEBUG:boto:Method: GET 2023-09-22T21:43:08.987 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.987 DEBUG:boto:Data: 2023-09-22T21:43:08.987 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.987 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:08.987 DEBUG:boto:Port: 8000 2023-09-22T21:43:08.988 DEBUG:boto:Params: {} 2023-09-22T21:43:08.988 DEBUG:boto:Token: None 2023-09-22T21:43:08.988 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:43:08.988 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IGMEwppo+Jg5jCOd15QKpS9liiU= 2023-09-22T21:43:08.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IGMEwppo+Jg5jCOd15QKpS9liiU=', 'Content-Length': '0'} 2023-09-22T21:43:08.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bc4bfcf75076115-00650e0a6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:08.991 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:08.992 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:08.992 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:43:08.992 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:43:08.992 DEBUG:boto:Method: GET 2023-09-22T21:43:08.992 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:43:08.992 DEBUG:boto:Data: 2023-09-22T21:43:08.992 DEBUG:boto:Headers: {} 2023-09-22T21:43:08.992 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:08.992 DEBUG:boto:Port: 8001 2023-09-22T21:43:08.992 DEBUG:boto:Params: {} 2023-09-22T21:43:08.993 DEBUG:boto:Token: None 2023-09-22T21:43:08.993 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:08 GMT /ffzdqu-59/ 2023-09-22T21:43:08.993 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI= 2023-09-22T21:43:08.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TuRZwZ6KlpTx1qGuBxiLj1lJgTI=', 'Content-Length': '0'} 2023-09-22T21:43:08.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008887b10de6d5afa2-00650e0a6c-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.038 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:09.039 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:09.039 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:09.039 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:43:09.039 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:43:09.039 DEBUG:boto:Method: GET 2023-09-22T21:43:09.039 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:43:09.040 DEBUG:boto:Data: 2023-09-22T21:43:09.040 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.040 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.040 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.040 DEBUG:boto:Params: {} 2023-09-22T21:43:09.040 DEBUG:boto:Token: None 2023-09-22T21:43:09.040 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/ 2023-09-22T21:43:09.040 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ= 2023-09-22T21:43:09.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ=', 'Content-Length': '0'} 2023-09-22T21:43:09.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aa1ccf03ea6150d-00650e0a6d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.049 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:09.049 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:43:09.049 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:43:09.049 DEBUG:boto:Method: HEAD 2023-09-22T21:43:09.049 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:43:09.050 DEBUG:boto:Data: 2023-09-22T21:43:09.050 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.050 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.050 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.050 DEBUG:boto:Params: {} 2023-09-22T21:43:09.050 DEBUG:boto:Token: None 2023-09-22T21:43:09.050 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj38 2023-09-22T21:43:09.050 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yd0N9TWJ5NDIFUol5xetniGwgHY= 2023-09-22T21:43:09.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yd0N9TWJ5NDIFUol5xetniGwgHY=', 'Content-Length': '0'} 2023-09-22T21:43:09.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032e3efcd599a3166-00650e0a6d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.053 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:43:09.054 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:43:09.054 DEBUG:boto:Method: HEAD 2023-09-22T21:43:09.054 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:43:09.054 DEBUG:boto:Data: 2023-09-22T21:43:09.054 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.054 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.054 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.054 DEBUG:boto:Params: {} 2023-09-22T21:43:09.055 DEBUG:boto:Token: None 2023-09-22T21:43:09.055 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj38 2023-09-22T21:43:09.055 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yd0N9TWJ5NDIFUol5xetniGwgHY= 2023-09-22T21:43:09.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yd0N9TWJ5NDIFUol5xetniGwgHY=', 'Content-Length': '0'} 2023-09-22T21:43:09.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d91b3ecbb31c07e-00650e0a6d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T21:43:09.058 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:43:09.058 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:43:09.058 DEBUG:boto:Method: GET 2023-09-22T21:43:09.058 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:43:09.058 DEBUG:boto:Data: 2023-09-22T21:43:09.058 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.059 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.059 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.059 DEBUG:boto:Params: {} 2023-09-22T21:43:09.059 DEBUG:boto:Token: None 2023-09-22T21:43:09.059 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj38 2023-09-22T21:43:09.059 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Owg0AIruPuvE/d2Zn75y9CDINp0= 2023-09-22T21:43:09.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Owg0AIruPuvE/d2Zn75y9CDINp0=', 'Content-Length': '0'} 2023-09-22T21:43:09.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098ee07bf82c7a443-00650e0a6d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.106 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:43:09.107 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:43:09.107 DEBUG:boto:Method: GET 2023-09-22T21:43:09.107 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:43:09.107 DEBUG:boto:Data: 2023-09-22T21:43:09.107 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.107 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.108 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.108 DEBUG:boto:Params: {} 2023-09-22T21:43:09.108 DEBUG:boto:Token: None 2023-09-22T21:43:09.108 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj38 2023-09-22T21:43:09.109 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Owg0AIruPuvE/d2Zn75y9CDINp0= 2023-09-22T21:43:09.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Owg0AIruPuvE/d2Zn75y9CDINp0=', 'Content-Length': '0'} 2023-09-22T21:43:09.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9769bdfee100de3-00650e0a6d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.112 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:09.113 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:09.113 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:43:09.113 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:43:09.113 DEBUG:boto:Method: GET 2023-09-22T21:43:09.113 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:43:09.114 DEBUG:boto:Data: 2023-09-22T21:43:09.114 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.114 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.114 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.114 DEBUG:boto:Params: {} 2023-09-22T21:43:09.115 DEBUG:boto:Token: None 2023-09-22T21:43:09.115 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/ 2023-09-22T21:43:09.115 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ= 2023-09-22T21:43:09.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ=', 'Content-Length': '0'} 2023-09-22T21:43:09.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3836b3763ddcdfd-00650e0a6d-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.120 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:09.121 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:09.121 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:09.121 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:43:09.121 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:43:09.121 DEBUG:boto:Method: GET 2023-09-22T21:43:09.122 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:43:09.122 DEBUG:boto:Data: 2023-09-22T21:43:09.122 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.122 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.122 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.123 DEBUG:boto:Params: {} 2023-09-22T21:43:09.123 DEBUG:boto:Token: None 2023-09-22T21:43:09.123 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/ 2023-09-22T21:43:09.123 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ= 2023-09-22T21:43:09.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ=', 'Content-Length': '0'} 2023-09-22T21:43:09.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acadbca951c603cf-00650e0a6d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.174 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:09.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:43:09.175 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:43:09.175 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:43:09.176 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.176 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.176 DEBUG:boto:Method: GET 2023-09-22T21:43:09.176 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.176 DEBUG:boto:Data: 2023-09-22T21:43:09.176 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.177 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.177 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.177 DEBUG:boto:Params: {} 2023-09-22T21:43:09.177 DEBUG:boto:Token: None 2023-09-22T21:43:09.178 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.178 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cOYPCAEWVnWGQf0C6rpScNZbawY= 2023-09-22T21:43:09.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cOYPCAEWVnWGQf0C6rpScNZbawY=', 'Content-Length': '0'} 2023-09-22T21:43:09.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001950f5e7bf3d28ae-00650e0a6d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.222 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:43:09.223 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:43:09.224 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.224 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.224 DEBUG:boto:Method: GET 2023-09-22T21:43:09.224 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.224 DEBUG:boto:Data: 2023-09-22T21:43:09.224 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.224 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.224 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.225 DEBUG:boto:Params: {} 2023-09-22T21:43:09.225 DEBUG:boto:Token: None 2023-09-22T21:43:09.225 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.225 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cOYPCAEWVnWGQf0C6rpScNZbawY= 2023-09-22T21:43:09.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cOYPCAEWVnWGQf0C6rpScNZbawY=', 'Content-Length': '0'} 2023-09-22T21:43:09.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000632e7a7c741ecb77-00650e0a6d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.229 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:43:09.229 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:43:09.229 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.229 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.229 DEBUG:boto:Method: HEAD 2023-09-22T21:43:09.229 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.229 DEBUG:boto:Data: 2023-09-22T21:43:09.229 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.229 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.230 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.230 DEBUG:boto:Params: {} 2023-09-22T21:43:09.230 DEBUG:boto:Token: None 2023-09-22T21:43:09.230 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.230 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Kn2pBKF63/+2UoUPzYhByGB/JdE= 2023-09-22T21:43:09.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Kn2pBKF63/+2UoUPzYhByGB/JdE=', 'Content-Length': '0'} 2023-09-22T21:43:09.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b4618104094ca7d-00650e0a6d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.233 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:43:09.233 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:43:09.233 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.233 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.233 DEBUG:boto:Method: HEAD 2023-09-22T21:43:09.234 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.234 DEBUG:boto:Data: 2023-09-22T21:43:09.234 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.234 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.234 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.234 DEBUG:boto:Params: {} 2023-09-22T21:43:09.234 DEBUG:boto:Token: None 2023-09-22T21:43:09.234 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.234 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Kn2pBKF63/+2UoUPzYhByGB/JdE= 2023-09-22T21:43:09.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Kn2pBKF63/+2UoUPzYhByGB/JdE=', 'Content-Length': '0'} 2023-09-22T21:43:09.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007de43ab92e07af7f-00650e0a6d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:43:09.238 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:43:09.238 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:43:09.238 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.238 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.238 DEBUG:boto:Method: GET 2023-09-22T21:43:09.238 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.238 DEBUG:boto:Data: 2023-09-22T21:43:09.238 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.240 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.240 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.240 DEBUG:boto:Params: {} 2023-09-22T21:43:09.240 DEBUG:boto:Token: None 2023-09-22T21:43:09.240 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.240 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cOYPCAEWVnWGQf0C6rpScNZbawY= 2023-09-22T21:43:09.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cOYPCAEWVnWGQf0C6rpScNZbawY=', 'Content-Length': '0'} 2023-09-22T21:43:09.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbf4d5fc1e48de79-00650e0a6d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.290 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:43:09.291 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:43:09.291 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.291 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.291 DEBUG:boto:Method: GET 2023-09-22T21:43:09.291 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.291 DEBUG:boto:Data: 2023-09-22T21:43:09.291 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.291 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.291 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.292 DEBUG:boto:Params: {} 2023-09-22T21:43:09.292 DEBUG:boto:Token: None 2023-09-22T21:43:09.292 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:43:09.292 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cOYPCAEWVnWGQf0C6rpScNZbawY= 2023-09-22T21:43:09.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cOYPCAEWVnWGQf0C6rpScNZbawY=', 'Content-Length': '0'} 2023-09-22T21:43:09.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcab7e468b3406b7-00650e0a6d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.295 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:09.295 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:09.296 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:43:09.296 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:43:09.296 DEBUG:boto:Method: GET 2023-09-22T21:43:09.296 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:43:09.296 DEBUG:boto:Data: 2023-09-22T21:43:09.296 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.296 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.296 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.296 DEBUG:boto:Params: {} 2023-09-22T21:43:09.296 DEBUG:boto:Token: None 2023-09-22T21:43:09.297 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/ 2023-09-22T21:43:09.297 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ= 2023-09-22T21:43:09.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ=', 'Content-Length': '0'} 2023-09-22T21:43:09.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e1c30ec21319796-00650e0a6d-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.301 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:09.302 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:09.302 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:09.302 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:43:09.302 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:43:09.302 DEBUG:boto:Method: GET 2023-09-22T21:43:09.302 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:43:09.303 DEBUG:boto:Data: 2023-09-22T21:43:09.303 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.303 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.303 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.303 DEBUG:boto:Params: {} 2023-09-22T21:43:09.303 DEBUG:boto:Token: None 2023-09-22T21:43:09.303 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/ 2023-09-22T21:43:09.303 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ= 2023-09-22T21:43:09.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ=', 'Content-Length': '0'} 2023-09-22T21:43:09.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d0fb92d724e2f1c-00650e0a6d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.310 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:09.311 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:43:09.311 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:43:09.311 DEBUG:boto:Method: HEAD 2023-09-22T21:43:09.311 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:43:09.311 DEBUG:boto:Data: 2023-09-22T21:43:09.311 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.311 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.312 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.312 DEBUG:boto:Params: {} 2023-09-22T21:43:09.312 DEBUG:boto:Token: None 2023-09-22T21:43:09.312 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj44 2023-09-22T21:43:09.312 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b1vch7lX1Nd6VzDpuyGgPWyPr6s= 2023-09-22T21:43:09.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b1vch7lX1Nd6VzDpuyGgPWyPr6s=', 'Content-Length': '0'} 2023-09-22T21:43:09.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb1e762bad4979d8-00650e0a6d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.315 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:43:09.315 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:43:09.316 DEBUG:boto:Method: HEAD 2023-09-22T21:43:09.316 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:43:09.316 DEBUG:boto:Data: 2023-09-22T21:43:09.316 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.316 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.316 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.316 DEBUG:boto:Params: {} 2023-09-22T21:43:09.316 DEBUG:boto:Token: None 2023-09-22T21:43:09.317 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj44 2023-09-22T21:43:09.317 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b1vch7lX1Nd6VzDpuyGgPWyPr6s= 2023-09-22T21:43:09.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b1vch7lX1Nd6VzDpuyGgPWyPr6s=', 'Content-Length': '0'} 2023-09-22T21:43:09.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de6502307954e820-00650e0a6d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:43:09.320 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:43:09.320 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:43:09.320 DEBUG:boto:Method: GET 2023-09-22T21:43:09.320 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:43:09.320 DEBUG:boto:Data: 2023-09-22T21:43:09.321 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.321 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.321 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.321 DEBUG:boto:Params: {} 2023-09-22T21:43:09.321 DEBUG:boto:Token: None 2023-09-22T21:43:09.321 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj44 2023-09-22T21:43:09.321 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3EgV67w6rTuEQbkffZH9glAysDM= 2023-09-22T21:43:09.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3EgV67w6rTuEQbkffZH9glAysDM=', 'Content-Length': '0'} 2023-09-22T21:43:09.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008efba2f199210c83-00650e0a6d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.370 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:43:09.371 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:43:09.371 DEBUG:boto:Method: GET 2023-09-22T21:43:09.371 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:43:09.371 DEBUG:boto:Data: 2023-09-22T21:43:09.371 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.371 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.371 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.371 DEBUG:boto:Params: {} 2023-09-22T21:43:09.372 DEBUG:boto:Token: None 2023-09-22T21:43:09.372 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj44 2023-09-22T21:43:09.372 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3EgV67w6rTuEQbkffZH9glAysDM= 2023-09-22T21:43:09.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3EgV67w6rTuEQbkffZH9glAysDM=', 'Content-Length': '0'} 2023-09-22T21:43:09.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000208658b7263bee8e-00650e0a6d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.375 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:09.375 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:09.375 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:43:09.375 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:43:09.375 DEBUG:boto:Method: GET 2023-09-22T21:43:09.376 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:43:09.376 DEBUG:boto:Data: 2023-09-22T21:43:09.376 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.376 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.376 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.376 DEBUG:boto:Params: {} 2023-09-22T21:43:09.376 DEBUG:boto:Token: None 2023-09-22T21:43:09.376 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/ 2023-09-22T21:43:09.376 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ= 2023-09-22T21:43:09.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ=', 'Content-Length': '0'} 2023-09-22T21:43:09.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3da522ed52ce652-00650e0a6d-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.426 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:09.427 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:09.427 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:09.427 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:43:09.427 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:43:09.427 DEBUG:boto:Method: GET 2023-09-22T21:43:09.427 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:43:09.428 DEBUG:boto:Data: 2023-09-22T21:43:09.428 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.428 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.428 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.428 DEBUG:boto:Params: {} 2023-09-22T21:43:09.428 DEBUG:boto:Token: None 2023-09-22T21:43:09.428 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/ 2023-09-22T21:43:09.428 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ= 2023-09-22T21:43:09.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ=', 'Content-Length': '0'} 2023-09-22T21:43:09.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bd6e881451a3a9c-00650e0a6d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.436 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:09.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-22T21:43:09.436 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:43:09.436 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:43:09.437 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.437 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.437 DEBUG:boto:Method: GET 2023-09-22T21:43:09.437 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.437 DEBUG:boto:Data: 2023-09-22T21:43:09.437 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.437 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.437 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.437 DEBUG:boto:Params: {} 2023-09-22T21:43:09.438 DEBUG:boto:Token: None 2023-09-22T21:43:09.438 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.439 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:z1dj9M6Ru3fHjtaniSJ8R8cWYcI= 2023-09-22T21:43:09.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:z1dj9M6Ru3fHjtaniSJ8R8cWYcI=', 'Content-Length': '0'} 2023-09-22T21:43:09.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039fa5ea460ce47d1-00650e0a6d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.444 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:43:09.444 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:43:09.444 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.444 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.444 DEBUG:boto:Method: GET 2023-09-22T21:43:09.445 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.445 DEBUG:boto:Data: 2023-09-22T21:43:09.445 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.445 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.445 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.445 DEBUG:boto:Params: {} 2023-09-22T21:43:09.445 DEBUG:boto:Token: None 2023-09-22T21:43:09.445 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.446 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:z1dj9M6Ru3fHjtaniSJ8R8cWYcI= 2023-09-22T21:43:09.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:z1dj9M6Ru3fHjtaniSJ8R8cWYcI=', 'Content-Length': '0'} 2023-09-22T21:43:09.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006ce6c4299119b5a-00650e0a6d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.449 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:43:09.449 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:43:09.449 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.449 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.449 DEBUG:boto:Method: HEAD 2023-09-22T21:43:09.449 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.449 DEBUG:boto:Data: 2023-09-22T21:43:09.450 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.450 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.450 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.450 DEBUG:boto:Params: {} 2023-09-22T21:43:09.450 DEBUG:boto:Token: None 2023-09-22T21:43:09.450 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.450 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:s237XY+mxgHtD9iovynoogrxmik= 2023-09-22T21:43:09.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:s237XY+mxgHtD9iovynoogrxmik=', 'Content-Length': '0'} 2023-09-22T21:43:09.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff47f3ca1c90f006-00650e0a6d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.453 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:43:09.453 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:43:09.454 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.454 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.454 DEBUG:boto:Method: HEAD 2023-09-22T21:43:09.454 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.454 DEBUG:boto:Data: 2023-09-22T21:43:09.454 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.454 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.454 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.454 DEBUG:boto:Params: {} 2023-09-22T21:43:09.455 DEBUG:boto:Token: None 2023-09-22T21:43:09.455 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.455 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:s237XY+mxgHtD9iovynoogrxmik= 2023-09-22T21:43:09.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:s237XY+mxgHtD9iovynoogrxmik=', 'Content-Length': '0'} 2023-09-22T21:43:09.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b0530ff57943be7-00650e0a6d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-22T21:43:09.458 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:43:09.458 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:43:09.458 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.458 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.458 DEBUG:boto:Method: GET 2023-09-22T21:43:09.458 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.459 DEBUG:boto:Data: 2023-09-22T21:43:09.459 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.459 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.459 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.459 DEBUG:boto:Params: {} 2023-09-22T21:43:09.459 DEBUG:boto:Token: None 2023-09-22T21:43:09.459 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.459 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:z1dj9M6Ru3fHjtaniSJ8R8cWYcI= 2023-09-22T21:43:09.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:z1dj9M6Ru3fHjtaniSJ8R8cWYcI=', 'Content-Length': '0'} 2023-09-22T21:43:09.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cbd5108af326d53-00650e0a6d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.510 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:43:09.510 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:43:09.511 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.511 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.511 DEBUG:boto:Method: GET 2023-09-22T21:43:09.511 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.511 DEBUG:boto:Data: 2023-09-22T21:43:09.511 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.511 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.511 DEBUG:boto:Params: {} 2023-09-22T21:43:09.512 DEBUG:boto:Token: None 2023-09-22T21:43:09.512 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:43:09.512 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:z1dj9M6Ru3fHjtaniSJ8R8cWYcI= 2023-09-22T21:43:09.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:z1dj9M6Ru3fHjtaniSJ8R8cWYcI=', 'Content-Length': '0'} 2023-09-22T21:43:09.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d580019237d1d755-00650e0a6d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.515 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:09.515 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:09.515 DEBUG:boto:path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:43:09.515 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:43:09.516 DEBUG:boto:Method: GET 2023-09-22T21:43:09.516 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj46 2023-09-22T21:43:09.516 DEBUG:boto:Data: 2023-09-22T21:43:09.516 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.516 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.516 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.516 DEBUG:boto:Params: {} 2023-09-22T21:43:09.516 DEBUG:boto:Token: None 2023-09-22T21:43:09.516 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/ 2023-09-22T21:43:09.517 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ= 2023-09-22T21:43:09.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ=', 'Content-Length': '0'} 2023-09-22T21:43:09.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d19cc4bd50013ec6-00650e0a6d-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.521 DEBUG:boto:b'ffzdqu-59obj461000falseobj462023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:09.522 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:09.522 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:09.522 DEBUG:boto:path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:43:09.522 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:43:09.522 DEBUG:boto:Method: GET 2023-09-22T21:43:09.523 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj46 2023-09-22T21:43:09.523 DEBUG:boto:Data: 2023-09-22T21:43:09.523 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.523 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.523 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.523 DEBUG:boto:Params: {} 2023-09-22T21:43:09.523 DEBUG:boto:Token: None 2023-09-22T21:43:09.523 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/ 2023-09-22T21:43:09.523 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ= 2023-09-22T21:43:09.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ=', 'Content-Length': '0'} 2023-09-22T21:43:09.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7bd8b45bb122ff5-00650e0a6d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.530 DEBUG:boto:b'ffzdqu-59obj461000falseobj462023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:09.531 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:43:09.531 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:43:09.531 DEBUG:boto:Method: HEAD 2023-09-22T21:43:09.531 DEBUG:boto:Path: /ffzdqu-59/obj46 2023-09-22T21:43:09.531 DEBUG:boto:Data: 2023-09-22T21:43:09.531 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.531 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.531 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.531 DEBUG:boto:Params: {} 2023-09-22T21:43:09.532 DEBUG:boto:Token: None 2023-09-22T21:43:09.532 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj46 2023-09-22T21:43:09.532 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KgXbx4egwrHOOaX9EsGbjTxoheE= 2023-09-22T21:43:09.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KgXbx4egwrHOOaX9EsGbjTxoheE=', 'Content-Length': '0'} 2023-09-22T21:43:09.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a072be8898ad9eb9-00650e0a6d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.535 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:43:09.535 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:43:09.535 DEBUG:boto:Method: HEAD 2023-09-22T21:43:09.535 DEBUG:boto:Path: /ffzdqu-59/obj46 2023-09-22T21:43:09.535 DEBUG:boto:Data: 2023-09-22T21:43:09.535 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.536 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.536 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.536 DEBUG:boto:Params: {} 2023-09-22T21:43:09.536 DEBUG:boto:Token: None 2023-09-22T21:43:09.536 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj46 2023-09-22T21:43:09.536 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KgXbx4egwrHOOaX9EsGbjTxoheE= 2023-09-22T21:43:09.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KgXbx4egwrHOOaX9EsGbjTxoheE=', 'Content-Length': '0'} 2023-09-22T21:43:09.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071048fba80dfb794-00650e0a6d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-22T21:43:09.539 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:43:09.539 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:43:09.539 DEBUG:boto:Method: GET 2023-09-22T21:43:09.539 DEBUG:boto:Path: /ffzdqu-59/obj46 2023-09-22T21:43:09.539 DEBUG:boto:Data: 2023-09-22T21:43:09.540 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.540 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.540 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.540 DEBUG:boto:Params: {} 2023-09-22T21:43:09.540 DEBUG:boto:Token: None 2023-09-22T21:43:09.540 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj46 2023-09-22T21:43:09.540 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zJDmeLCycgorOJnim1grGohok+s= 2023-09-22T21:43:09.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zJDmeLCycgorOJnim1grGohok+s=', 'Content-Length': '0'} 2023-09-22T21:43:09.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080a9331fc614351e-00650e0a6d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.586 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:43:09.587 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:43:09.587 DEBUG:boto:Method: GET 2023-09-22T21:43:09.587 DEBUG:boto:Path: /ffzdqu-59/obj46 2023-09-22T21:43:09.587 DEBUG:boto:Data: 2023-09-22T21:43:09.587 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.587 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.587 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.587 DEBUG:boto:Params: {} 2023-09-22T21:43:09.587 DEBUG:boto:Token: None 2023-09-22T21:43:09.588 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj46 2023-09-22T21:43:09.588 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zJDmeLCycgorOJnim1grGohok+s= 2023-09-22T21:43:09.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zJDmeLCycgorOJnim1grGohok+s=', 'Content-Length': '0'} 2023-09-22T21:43:09.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be5d0635a1143e10-00650e0a6d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.591 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:09.591 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:09.591 DEBUG:boto:path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:43:09.591 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:43:09.591 DEBUG:boto:Method: GET 2023-09-22T21:43:09.591 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj46 2023-09-22T21:43:09.591 DEBUG:boto:Data: 2023-09-22T21:43:09.592 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.592 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.592 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.592 DEBUG:boto:Params: {} 2023-09-22T21:43:09.592 DEBUG:boto:Token: None 2023-09-22T21:43:09.592 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/ 2023-09-22T21:43:09.592 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ= 2023-09-22T21:43:09.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ=', 'Content-Length': '0'} 2023-09-22T21:43:09.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7c29f2795b6f7c7-00650e0a6d-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.597 DEBUG:boto:b'ffzdqu-59obj461000falseobj462023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:09.598 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:09.598 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:09.598 DEBUG:boto:path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:43:09.598 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:43:09.598 DEBUG:boto:Method: GET 2023-09-22T21:43:09.598 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj46 2023-09-22T21:43:09.598 DEBUG:boto:Data: 2023-09-22T21:43:09.598 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.598 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.599 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.599 DEBUG:boto:Params: {} 2023-09-22T21:43:09.599 DEBUG:boto:Token: None 2023-09-22T21:43:09.599 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/ 2023-09-22T21:43:09.599 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ= 2023-09-22T21:43:09.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ=', 'Content-Length': '0'} 2023-09-22T21:43:09.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098125491d6958fbb-00650e0a6d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.606 DEBUG:boto:b'ffzdqu-59obj461000falseobj462023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:09.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:43:09.606 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:43:09.606 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:43:09.607 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.607 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.607 DEBUG:boto:Method: GET 2023-09-22T21:43:09.607 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.607 DEBUG:boto:Data: 2023-09-22T21:43:09.607 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.607 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.607 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.607 DEBUG:boto:Params: {} 2023-09-22T21:43:09.608 DEBUG:boto:Token: None 2023-09-22T21:43:09.608 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.608 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tKVUUcLyyvOacXK9p2rN9BwDJRo= 2023-09-22T21:43:09.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tKVUUcLyyvOacXK9p2rN9BwDJRo=', 'Content-Length': '0'} 2023-09-22T21:43:09.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8c66811d09dc3dd-00650e0a6d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.611 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:43:09.611 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:43:09.611 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.611 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.611 DEBUG:boto:Method: GET 2023-09-22T21:43:09.612 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.612 DEBUG:boto:Data: 2023-09-22T21:43:09.612 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.612 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.612 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.612 DEBUG:boto:Params: {} 2023-09-22T21:43:09.612 DEBUG:boto:Token: None 2023-09-22T21:43:09.612 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tKVUUcLyyvOacXK9p2rN9BwDJRo= 2023-09-22T21:43:09.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tKVUUcLyyvOacXK9p2rN9BwDJRo=', 'Content-Length': '0'} 2023-09-22T21:43:09.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc260021d511449a-00650e0a6d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.615 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:43:09.616 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:43:09.616 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.616 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.616 DEBUG:boto:Method: HEAD 2023-09-22T21:43:09.616 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.616 DEBUG:boto:Data: 2023-09-22T21:43:09.616 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.616 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.616 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.617 DEBUG:boto:Params: {} 2023-09-22T21:43:09.617 DEBUG:boto:Token: None 2023-09-22T21:43:09.617 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.617 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+O85NTufgdDm4f1uH2jO+zeYXjk= 2023-09-22T21:43:09.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+O85NTufgdDm4f1uH2jO+zeYXjk=', 'Content-Length': '0'} 2023-09-22T21:43:09.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001464566dbd3b9c5d-00650e0a6d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.620 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:43:09.620 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:43:09.620 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.620 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.620 DEBUG:boto:Method: HEAD 2023-09-22T21:43:09.620 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.620 DEBUG:boto:Data: 2023-09-22T21:43:09.621 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.621 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.621 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.621 DEBUG:boto:Params: {} 2023-09-22T21:43:09.621 DEBUG:boto:Token: None 2023-09-22T21:43:09.621 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.621 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+O85NTufgdDm4f1uH2jO+zeYXjk= 2023-09-22T21:43:09.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+O85NTufgdDm4f1uH2jO+zeYXjk=', 'Content-Length': '0'} 2023-09-22T21:43:09.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a36756411d1d018-00650e0a6d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:43:09.624 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:43:09.624 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:43:09.625 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.625 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.625 DEBUG:boto:Method: GET 2023-09-22T21:43:09.625 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.625 DEBUG:boto:Data: 2023-09-22T21:43:09.625 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.625 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.625 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.625 DEBUG:boto:Params: {} 2023-09-22T21:43:09.626 DEBUG:boto:Token: None 2023-09-22T21:43:09.626 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.626 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tKVUUcLyyvOacXK9p2rN9BwDJRo= 2023-09-22T21:43:09.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tKVUUcLyyvOacXK9p2rN9BwDJRo=', 'Content-Length': '0'} 2023-09-22T21:43:09.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bcedd1684f022d4-00650e0a6d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.670 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:43:09.671 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:43:09.671 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.671 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.671 DEBUG:boto:Method: GET 2023-09-22T21:43:09.671 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.671 DEBUG:boto:Data: 2023-09-22T21:43:09.671 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.672 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.672 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.672 DEBUG:boto:Params: {} 2023-09-22T21:43:09.672 DEBUG:boto:Token: None 2023-09-22T21:43:09.672 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:43:09.672 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tKVUUcLyyvOacXK9p2rN9BwDJRo= 2023-09-22T21:43:09.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tKVUUcLyyvOacXK9p2rN9BwDJRo=', 'Content-Length': '0'} 2023-09-22T21:43:09.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000583479a6596cbeb3-00650e0a6d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.676 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:09.676 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:09.676 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:43:09.676 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:43:09.676 DEBUG:boto:Method: GET 2023-09-22T21:43:09.676 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:43:09.676 DEBUG:boto:Data: 2023-09-22T21:43:09.677 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.677 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.677 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.677 DEBUG:boto:Params: {} 2023-09-22T21:43:09.677 DEBUG:boto:Token: None 2023-09-22T21:43:09.677 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/ 2023-09-22T21:43:09.677 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ= 2023-09-22T21:43:09.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ=', 'Content-Length': '0'} 2023-09-22T21:43:09.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b02ae2eade9ca89-00650e0a6d-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.989 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:09.990 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:09.990 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:09.990 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:43:09.990 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:43:09.990 DEBUG:boto:Method: GET 2023-09-22T21:43:09.990 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:43:09.990 DEBUG:boto:Data: 2023-09-22T21:43:09.990 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.990 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:09.991 DEBUG:boto:Port: 8000 2023-09-22T21:43:09.991 DEBUG:boto:Params: {} 2023-09-22T21:43:09.991 DEBUG:boto:Token: None 2023-09-22T21:43:09.991 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/ 2023-09-22T21:43:09.991 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ= 2023-09-22T21:43:09.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HlHhLgJCwDwn9FcjjZEHxwMsLKQ=', 'Content-Length': '0'} 2023-09-22T21:43:09.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001280c7a9bf6609c5-00650e0a6d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:09.998 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:09.998 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:43:09.999 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:43:09.999 DEBUG:boto:Method: HEAD 2023-09-22T21:43:09.999 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:43:09.999 DEBUG:boto:Data: 2023-09-22T21:43:09.999 DEBUG:boto:Headers: {} 2023-09-22T21:43:09.999 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:09.999 DEBUG:boto:Port: 8001 2023-09-22T21:43:09.999 DEBUG:boto:Params: {} 2023-09-22T21:43:09.999 DEBUG:boto:Token: None 2023-09-22T21:43:10.000 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:09 GMT /ffzdqu-59/obj51 2023-09-22T21:43:10.000 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iGyDYell9wDBiH9ACov/XWcMizA= 2023-09-22T21:43:10.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iGyDYell9wDBiH9ACov/XWcMizA=', 'Content-Length': '0'} 2023-09-22T21:43:10.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5500d744008f649-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.003 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:43:10.003 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:43:10.003 DEBUG:boto:Method: HEAD 2023-09-22T21:43:10.003 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:43:10.003 DEBUG:boto:Data: 2023-09-22T21:43:10.003 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.003 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.003 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.003 DEBUG:boto:Params: {} 2023-09-22T21:43:10.004 DEBUG:boto:Token: None 2023-09-22T21:43:10.004 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj51 2023-09-22T21:43:10.004 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gHXvqekPakhpPttLnqzzVnSDwLU= 2023-09-22T21:43:10.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gHXvqekPakhpPttLnqzzVnSDwLU=', 'Content-Length': '0'} 2023-09-22T21:43:10.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae1bb6ef02a1eddd-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:43:10.007 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:43:10.007 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:43:10.007 DEBUG:boto:Method: GET 2023-09-22T21:43:10.007 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:43:10.007 DEBUG:boto:Data: 2023-09-22T21:43:10.007 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.007 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.007 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.008 DEBUG:boto:Params: {} 2023-09-22T21:43:10.008 DEBUG:boto:Token: None 2023-09-22T21:43:10.008 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj51 2023-09-22T21:43:10.008 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6C6/JQ9zU2IxflbCSWmUWPm0YAw= 2023-09-22T21:43:10.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6C6/JQ9zU2IxflbCSWmUWPm0YAw=', 'Content-Length': '0'} 2023-09-22T21:43:10.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059b473d34570183f-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.011 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:43:10.011 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:43:10.011 DEBUG:boto:Method: GET 2023-09-22T21:43:10.011 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:43:10.012 DEBUG:boto:Data: 2023-09-22T21:43:10.012 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.012 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.012 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.012 DEBUG:boto:Params: {} 2023-09-22T21:43:10.012 DEBUG:boto:Token: None 2023-09-22T21:43:10.012 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj51 2023-09-22T21:43:10.012 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6C6/JQ9zU2IxflbCSWmUWPm0YAw= 2023-09-22T21:43:10.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6C6/JQ9zU2IxflbCSWmUWPm0YAw=', 'Content-Length': '0'} 2023-09-22T21:43:10.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a925bc9d348ac5c-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.016 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:10.016 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:10.016 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:43:10.016 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:43:10.016 DEBUG:boto:Method: GET 2023-09-22T21:43:10.016 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:43:10.016 DEBUG:boto:Data: 2023-09-22T21:43:10.016 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.016 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.017 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.017 DEBUG:boto:Params: {} 2023-09-22T21:43:10.017 DEBUG:boto:Token: None 2023-09-22T21:43:10.017 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/ 2023-09-22T21:43:10.017 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs= 2023-09-22T21:43:10.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs=', 'Content-Length': '0'} 2023-09-22T21:43:10.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000156bab36c9bcfa85-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.062 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:10.063 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:10.063 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:10.063 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:43:10.063 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:43:10.063 DEBUG:boto:Method: GET 2023-09-22T21:43:10.064 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:43:10.064 DEBUG:boto:Data: 2023-09-22T21:43:10.064 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.064 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.064 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.064 DEBUG:boto:Params: {} 2023-09-22T21:43:10.064 DEBUG:boto:Token: None 2023-09-22T21:43:10.064 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/ 2023-09-22T21:43:10.065 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs= 2023-09-22T21:43:10.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs=', 'Content-Length': '0'} 2023-09-22T21:43:10.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d5f8332fdc0be2c-00650e0a6e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.072 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:10.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:43:10.072 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:43:10.073 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:43:10.073 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.073 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.073 DEBUG:boto:Method: GET 2023-09-22T21:43:10.073 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.073 DEBUG:boto:Data: 2023-09-22T21:43:10.073 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.073 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.073 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.074 DEBUG:boto:Params: {} 2023-09-22T21:43:10.074 DEBUG:boto:Token: None 2023-09-22T21:43:10.074 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.074 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yNwHyLEfkxZoyHGwUxDZK0CpQfQ= 2023-09-22T21:43:10.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yNwHyLEfkxZoyHGwUxDZK0CpQfQ=', 'Content-Length': '0'} 2023-09-22T21:43:10.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007362300498f5ed65-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.077 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:43:10.077 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:43:10.077 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.078 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.078 DEBUG:boto:Method: GET 2023-09-22T21:43:10.078 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.078 DEBUG:boto:Data: 2023-09-22T21:43:10.078 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.078 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.078 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.078 DEBUG:boto:Params: {} 2023-09-22T21:43:10.078 DEBUG:boto:Token: None 2023-09-22T21:43:10.079 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.079 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yNwHyLEfkxZoyHGwUxDZK0CpQfQ= 2023-09-22T21:43:10.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yNwHyLEfkxZoyHGwUxDZK0CpQfQ=', 'Content-Length': '0'} 2023-09-22T21:43:10.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005aa1acfc6e429a0c-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.122 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:43:10.123 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:43:10.123 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.123 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.123 DEBUG:boto:Method: HEAD 2023-09-22T21:43:10.123 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.123 DEBUG:boto:Data: 2023-09-22T21:43:10.123 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.123 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.123 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.124 DEBUG:boto:Params: {} 2023-09-22T21:43:10.124 DEBUG:boto:Token: None 2023-09-22T21:43:10.124 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.124 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1DyfNFiejMJprIALgeZ9YTJIj9s= 2023-09-22T21:43:10.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1DyfNFiejMJprIALgeZ9YTJIj9s=', 'Content-Length': '0'} 2023-09-22T21:43:10.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e002a5aa1b8d990d-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.127 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:43:10.127 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:43:10.128 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.128 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.128 DEBUG:boto:Method: HEAD 2023-09-22T21:43:10.128 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.128 DEBUG:boto:Data: 2023-09-22T21:43:10.128 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.128 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.128 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.128 DEBUG:boto:Params: {} 2023-09-22T21:43:10.129 DEBUG:boto:Token: None 2023-09-22T21:43:10.129 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.129 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1DyfNFiejMJprIALgeZ9YTJIj9s= 2023-09-22T21:43:10.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1DyfNFiejMJprIALgeZ9YTJIj9s=', 'Content-Length': '0'} 2023-09-22T21:43:10.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008783900ffec22b2b-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:43:10.132 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:43:10.132 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:43:10.132 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.132 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.132 DEBUG:boto:Method: GET 2023-09-22T21:43:10.132 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.133 DEBUG:boto:Data: 2023-09-22T21:43:10.133 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.133 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.133 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.133 DEBUG:boto:Params: {} 2023-09-22T21:43:10.133 DEBUG:boto:Token: None 2023-09-22T21:43:10.133 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.133 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yNwHyLEfkxZoyHGwUxDZK0CpQfQ= 2023-09-22T21:43:10.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yNwHyLEfkxZoyHGwUxDZK0CpQfQ=', 'Content-Length': '0'} 2023-09-22T21:43:10.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1ed0bed3ba2bed9-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.136 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:43:10.137 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:43:10.137 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.137 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.137 DEBUG:boto:Method: GET 2023-09-22T21:43:10.137 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.137 DEBUG:boto:Data: 2023-09-22T21:43:10.137 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.137 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.137 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.137 DEBUG:boto:Params: {} 2023-09-22T21:43:10.138 DEBUG:boto:Token: None 2023-09-22T21:43:10.138 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:43:10.138 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yNwHyLEfkxZoyHGwUxDZK0CpQfQ= 2023-09-22T21:43:10.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yNwHyLEfkxZoyHGwUxDZK0CpQfQ=', 'Content-Length': '0'} 2023-09-22T21:43:10.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3d5952d2715db3f-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.141 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:10.141 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:10.141 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:43:10.141 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:43:10.141 DEBUG:boto:Method: GET 2023-09-22T21:43:10.142 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:43:10.142 DEBUG:boto:Data: 2023-09-22T21:43:10.142 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.142 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.142 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.142 DEBUG:boto:Params: {} 2023-09-22T21:43:10.142 DEBUG:boto:Token: None 2023-09-22T21:43:10.142 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/ 2023-09-22T21:43:10.143 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs= 2023-09-22T21:43:10.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs=', 'Content-Length': '0'} 2023-09-22T21:43:10.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082928071b61586c8-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.194 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:10.195 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:10.195 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:10.195 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:43:10.195 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:43:10.195 DEBUG:boto:Method: GET 2023-09-22T21:43:10.195 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:43:10.195 DEBUG:boto:Data: 2023-09-22T21:43:10.196 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.196 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.196 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.196 DEBUG:boto:Params: {} 2023-09-22T21:43:10.196 DEBUG:boto:Token: None 2023-09-22T21:43:10.196 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/ 2023-09-22T21:43:10.196 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs= 2023-09-22T21:43:10.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs=', 'Content-Length': '0'} 2023-09-22T21:43:10.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e82456d9b30d3bf-00650e0a6e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.203 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:10.204 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:43:10.205 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:43:10.205 DEBUG:boto:Method: HEAD 2023-09-22T21:43:10.205 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:43:10.205 DEBUG:boto:Data: 2023-09-22T21:43:10.205 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.206 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.206 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.206 DEBUG:boto:Params: {} 2023-09-22T21:43:10.206 DEBUG:boto:Token: None 2023-09-22T21:43:10.206 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj55 2023-09-22T21:43:10.206 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Bayta7QXGpBTpY2SpyXdAa6lO/s= 2023-09-22T21:43:10.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Bayta7QXGpBTpY2SpyXdAa6lO/s=', 'Content-Length': '0'} 2023-09-22T21:43:10.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b319c26173ecd995-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.209 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:43:10.209 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:43:10.209 DEBUG:boto:Method: HEAD 2023-09-22T21:43:10.210 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:43:10.210 DEBUG:boto:Data: 2023-09-22T21:43:10.210 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.210 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.210 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.210 DEBUG:boto:Params: {} 2023-09-22T21:43:10.210 DEBUG:boto:Token: None 2023-09-22T21:43:10.210 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj55 2023-09-22T21:43:10.211 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Bayta7QXGpBTpY2SpyXdAa6lO/s= 2023-09-22T21:43:10.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Bayta7QXGpBTpY2SpyXdAa6lO/s=', 'Content-Length': '0'} 2023-09-22T21:43:10.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000037c09a27ac935b2-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:43:10.214 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:43:10.214 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:43:10.214 DEBUG:boto:Method: GET 2023-09-22T21:43:10.214 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:43:10.214 DEBUG:boto:Data: 2023-09-22T21:43:10.214 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.214 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.214 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.215 DEBUG:boto:Params: {} 2023-09-22T21:43:10.215 DEBUG:boto:Token: None 2023-09-22T21:43:10.215 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj55 2023-09-22T21:43:10.215 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lpODQ6F90ygegifnb2EmxnAXxKg= 2023-09-22T21:43:10.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lpODQ6F90ygegifnb2EmxnAXxKg=', 'Content-Length': '0'} 2023-09-22T21:43:10.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009380e5d518e0d169-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.258 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:43:10.259 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:43:10.259 DEBUG:boto:Method: GET 2023-09-22T21:43:10.259 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:43:10.259 DEBUG:boto:Data: 2023-09-22T21:43:10.260 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.260 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.260 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.260 DEBUG:boto:Params: {} 2023-09-22T21:43:10.260 DEBUG:boto:Token: None 2023-09-22T21:43:10.260 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj55 2023-09-22T21:43:10.261 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lpODQ6F90ygegifnb2EmxnAXxKg= 2023-09-22T21:43:10.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lpODQ6F90ygegifnb2EmxnAXxKg=', 'Content-Length': '0'} 2023-09-22T21:43:10.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000038102468775011f-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.264 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:10.264 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:10.265 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:43:10.265 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:43:10.265 DEBUG:boto:Method: GET 2023-09-22T21:43:10.265 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:43:10.265 DEBUG:boto:Data: 2023-09-22T21:43:10.266 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.266 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.266 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.266 DEBUG:boto:Params: {} 2023-09-22T21:43:10.266 DEBUG:boto:Token: None 2023-09-22T21:43:10.267 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/ 2023-09-22T21:43:10.267 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs= 2023-09-22T21:43:10.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs=', 'Content-Length': '0'} 2023-09-22T21:43:10.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b84d4158cee053e-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.273 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:10.274 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:10.274 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:10.274 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:43:10.275 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:43:10.275 DEBUG:boto:Method: GET 2023-09-22T21:43:10.275 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:43:10.275 DEBUG:boto:Data: 2023-09-22T21:43:10.275 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.276 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.276 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.276 DEBUG:boto:Params: {} 2023-09-22T21:43:10.276 DEBUG:boto:Token: None 2023-09-22T21:43:10.276 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/ 2023-09-22T21:43:10.277 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs= 2023-09-22T21:43:10.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs=', 'Content-Length': '0'} 2023-09-22T21:43:10.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020e80f7a43008c50-00650e0a6e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.284 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:10.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T21:43:10.284 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:43:10.285 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:43:10.285 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.285 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.285 DEBUG:boto:Method: GET 2023-09-22T21:43:10.285 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.285 DEBUG:boto:Data: 2023-09-22T21:43:10.285 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.285 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.285 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.285 DEBUG:boto:Params: {} 2023-09-22T21:43:10.286 DEBUG:boto:Token: None 2023-09-22T21:43:10.286 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.286 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vnDI42r2zFd6z8Q+o6nOw4PpAvE= 2023-09-22T21:43:10.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vnDI42r2zFd6z8Q+o6nOw4PpAvE=', 'Content-Length': '0'} 2023-09-22T21:43:10.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f0b4b6a00dcd532-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.334 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:43:10.334 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:43:10.334 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.335 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.335 DEBUG:boto:Method: GET 2023-09-22T21:43:10.335 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.335 DEBUG:boto:Data: 2023-09-22T21:43:10.335 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.335 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.335 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.335 DEBUG:boto:Params: {} 2023-09-22T21:43:10.335 DEBUG:boto:Token: None 2023-09-22T21:43:10.336 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.336 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vnDI42r2zFd6z8Q+o6nOw4PpAvE= 2023-09-22T21:43:10.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vnDI42r2zFd6z8Q+o6nOw4PpAvE=', 'Content-Length': '0'} 2023-09-22T21:43:10.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003afae86bf1773303-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.339 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:43:10.339 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:43:10.340 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.340 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.340 DEBUG:boto:Method: HEAD 2023-09-22T21:43:10.340 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.340 DEBUG:boto:Data: 2023-09-22T21:43:10.340 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.340 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.340 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.340 DEBUG:boto:Params: {} 2023-09-22T21:43:10.340 DEBUG:boto:Token: None 2023-09-22T21:43:10.341 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.341 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UUAiVs+IRf6MfCd2JeukTtttKIo= 2023-09-22T21:43:10.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UUAiVs+IRf6MfCd2JeukTtttKIo=', 'Content-Length': '0'} 2023-09-22T21:43:10.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad1bb8113936a6aa-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.344 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:43:10.344 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:43:10.344 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.344 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.344 DEBUG:boto:Method: HEAD 2023-09-22T21:43:10.344 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.344 DEBUG:boto:Data: 2023-09-22T21:43:10.344 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.345 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.345 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.345 DEBUG:boto:Params: {} 2023-09-22T21:43:10.345 DEBUG:boto:Token: None 2023-09-22T21:43:10.345 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.345 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UUAiVs+IRf6MfCd2JeukTtttKIo= 2023-09-22T21:43:10.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UUAiVs+IRf6MfCd2JeukTtttKIo=', 'Content-Length': '0'} 2023-09-22T21:43:10.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099bf081f3262e0da-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T21:43:10.348 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:43:10.349 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:43:10.349 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.349 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.349 DEBUG:boto:Method: GET 2023-09-22T21:43:10.349 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.349 DEBUG:boto:Data: 2023-09-22T21:43:10.349 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.349 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.349 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.350 DEBUG:boto:Params: {} 2023-09-22T21:43:10.350 DEBUG:boto:Token: None 2023-09-22T21:43:10.350 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.350 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vnDI42r2zFd6z8Q+o6nOw4PpAvE= 2023-09-22T21:43:10.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vnDI42r2zFd6z8Q+o6nOw4PpAvE=', 'Content-Length': '0'} 2023-09-22T21:43:10.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bca3de466221f4b-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.398 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:43:10.399 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:43:10.399 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.399 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.399 DEBUG:boto:Method: GET 2023-09-22T21:43:10.399 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.399 DEBUG:boto:Data: 2023-09-22T21:43:10.399 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.399 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.400 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.400 DEBUG:boto:Params: {} 2023-09-22T21:43:10.400 DEBUG:boto:Token: None 2023-09-22T21:43:10.400 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:43:10.400 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vnDI42r2zFd6z8Q+o6nOw4PpAvE= 2023-09-22T21:43:10.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vnDI42r2zFd6z8Q+o6nOw4PpAvE=', 'Content-Length': '0'} 2023-09-22T21:43:10.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d38b961e942e69cb-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.404 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:10.404 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:10.404 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:43:10.404 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:43:10.404 DEBUG:boto:Method: GET 2023-09-22T21:43:10.404 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:43:10.404 DEBUG:boto:Data: 2023-09-22T21:43:10.404 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.405 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.405 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.405 DEBUG:boto:Params: {} 2023-09-22T21:43:10.405 DEBUG:boto:Token: None 2023-09-22T21:43:10.405 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/ 2023-09-22T21:43:10.405 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs= 2023-09-22T21:43:10.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs=', 'Content-Length': '0'} 2023-09-22T21:43:10.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9d281454f036eeb-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.450 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:10.451 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:10.451 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:10.451 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:43:10.451 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:43:10.452 DEBUG:boto:Method: GET 2023-09-22T21:43:10.452 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:43:10.452 DEBUG:boto:Data: 2023-09-22T21:43:10.452 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.452 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.452 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.452 DEBUG:boto:Params: {} 2023-09-22T21:43:10.452 DEBUG:boto:Token: None 2023-09-22T21:43:10.452 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/ 2023-09-22T21:43:10.453 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs= 2023-09-22T21:43:10.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs=', 'Content-Length': '0'} 2023-09-22T21:43:10.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007549d048aced50a7-00650e0a6e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.460 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:10.460 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:43:10.461 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:43:10.461 DEBUG:boto:Method: HEAD 2023-09-22T21:43:10.461 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:43:10.461 DEBUG:boto:Data: 2023-09-22T21:43:10.461 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.461 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.461 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.461 DEBUG:boto:Params: {} 2023-09-22T21:43:10.462 DEBUG:boto:Token: None 2023-09-22T21:43:10.462 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj57 2023-09-22T21:43:10.462 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OwO0sqFD21KUhAGwAFKbC+UrOuM= 2023-09-22T21:43:10.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OwO0sqFD21KUhAGwAFKbC+UrOuM=', 'Content-Length': '0'} 2023-09-22T21:43:10.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019d35a929e71c3a2-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.465 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:43:10.465 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:43:10.465 DEBUG:boto:Method: HEAD 2023-09-22T21:43:10.465 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:43:10.466 DEBUG:boto:Data: 2023-09-22T21:43:10.466 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.466 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.466 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.466 DEBUG:boto:Params: {} 2023-09-22T21:43:10.466 DEBUG:boto:Token: None 2023-09-22T21:43:10.466 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj57 2023-09-22T21:43:10.466 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OwO0sqFD21KUhAGwAFKbC+UrOuM= 2023-09-22T21:43:10.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OwO0sqFD21KUhAGwAFKbC+UrOuM=', 'Content-Length': '0'} 2023-09-22T21:43:10.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000261263196a075c38-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T21:43:10.469 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:43:10.469 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:43:10.470 DEBUG:boto:Method: GET 2023-09-22T21:43:10.470 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:43:10.470 DEBUG:boto:Data: 2023-09-22T21:43:10.470 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.470 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.470 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.470 DEBUG:boto:Params: {} 2023-09-22T21:43:10.470 DEBUG:boto:Token: None 2023-09-22T21:43:10.470 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj57 2023-09-22T21:43:10.471 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4vwAPPRKldvBVL8VOa5zisBDRjs= 2023-09-22T21:43:10.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4vwAPPRKldvBVL8VOa5zisBDRjs=', 'Content-Length': '0'} 2023-09-22T21:43:10.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec8c2fed1d0fe647-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.514 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:43:10.515 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:43:10.515 DEBUG:boto:Method: GET 2023-09-22T21:43:10.515 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:43:10.515 DEBUG:boto:Data: 2023-09-22T21:43:10.515 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.515 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.515 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.515 DEBUG:boto:Params: {} 2023-09-22T21:43:10.515 DEBUG:boto:Token: None 2023-09-22T21:43:10.516 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj57 2023-09-22T21:43:10.516 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4vwAPPRKldvBVL8VOa5zisBDRjs= 2023-09-22T21:43:10.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4vwAPPRKldvBVL8VOa5zisBDRjs=', 'Content-Length': '0'} 2023-09-22T21:43:10.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000576fee1d0d350baa-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.520 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:10.520 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:10.520 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:43:10.520 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:43:10.521 DEBUG:boto:Method: GET 2023-09-22T21:43:10.521 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:43:10.521 DEBUG:boto:Data: 2023-09-22T21:43:10.521 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.521 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.521 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.521 DEBUG:boto:Params: {} 2023-09-22T21:43:10.521 DEBUG:boto:Token: None 2023-09-22T21:43:10.521 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/ 2023-09-22T21:43:10.522 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs= 2023-09-22T21:43:10.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs=', 'Content-Length': '0'} 2023-09-22T21:43:10.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1be02a63c4b0d33-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.527 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:10.527 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:10.527 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:10.528 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:43:10.528 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:43:10.528 DEBUG:boto:Method: GET 2023-09-22T21:43:10.528 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:43:10.528 DEBUG:boto:Data: 2023-09-22T21:43:10.528 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.528 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.528 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.529 DEBUG:boto:Params: {} 2023-09-22T21:43:10.529 DEBUG:boto:Token: None 2023-09-22T21:43:10.529 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/ 2023-09-22T21:43:10.529 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs= 2023-09-22T21:43:10.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs=', 'Content-Length': '0'} 2023-09-22T21:43:10.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f03846ba3c79684d-00650e0a6e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.536 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:10.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:43:10.537 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:43:10.537 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:43:10.537 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.537 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.537 DEBUG:boto:Method: GET 2023-09-22T21:43:10.537 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.537 DEBUG:boto:Data: 2023-09-22T21:43:10.538 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.538 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.538 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.538 DEBUG:boto:Params: {} 2023-09-22T21:43:10.538 DEBUG:boto:Token: None 2023-09-22T21:43:10.538 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.538 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eYlsnEq/twwdnY3HP/M4PD5yPJk= 2023-09-22T21:43:10.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eYlsnEq/twwdnY3HP/M4PD5yPJk=', 'Content-Length': '0'} 2023-09-22T21:43:10.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f39c7bf37ead3ae7-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.582 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:43:10.583 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:43:10.583 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.583 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.583 DEBUG:boto:Method: GET 2023-09-22T21:43:10.583 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.584 DEBUG:boto:Data: 2023-09-22T21:43:10.584 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.584 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.584 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.584 DEBUG:boto:Params: {} 2023-09-22T21:43:10.585 DEBUG:boto:Token: None 2023-09-22T21:43:10.585 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.585 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eYlsnEq/twwdnY3HP/M4PD5yPJk= 2023-09-22T21:43:10.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eYlsnEq/twwdnY3HP/M4PD5yPJk=', 'Content-Length': '0'} 2023-09-22T21:43:10.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fda44ec265db6a8-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.588 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:43:10.588 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:43:10.589 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.589 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.589 DEBUG:boto:Method: HEAD 2023-09-22T21:43:10.589 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.589 DEBUG:boto:Data: 2023-09-22T21:43:10.589 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.589 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.589 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.589 DEBUG:boto:Params: {} 2023-09-22T21:43:10.590 DEBUG:boto:Token: None 2023-09-22T21:43:10.590 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.590 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:A8i+tXEo7OGIa3f2gg+qQbsjrJY= 2023-09-22T21:43:10.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:A8i+tXEo7OGIa3f2gg+qQbsjrJY=', 'Content-Length': '0'} 2023-09-22T21:43:10.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000894440a79ad2bf42-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.593 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:43:10.593 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:43:10.593 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.593 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.594 DEBUG:boto:Method: HEAD 2023-09-22T21:43:10.594 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.594 DEBUG:boto:Data: 2023-09-22T21:43:10.594 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.594 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.594 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.594 DEBUG:boto:Params: {} 2023-09-22T21:43:10.594 DEBUG:boto:Token: None 2023-09-22T21:43:10.595 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.595 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:A8i+tXEo7OGIa3f2gg+qQbsjrJY= 2023-09-22T21:43:10.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:A8i+tXEo7OGIa3f2gg+qQbsjrJY=', 'Content-Length': '0'} 2023-09-22T21:43:10.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025cc9eeca2799fee-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:43:10.598 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:43:10.598 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:43:10.598 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.599 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.599 DEBUG:boto:Method: GET 2023-09-22T21:43:10.599 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.599 DEBUG:boto:Data: 2023-09-22T21:43:10.599 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.599 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.599 DEBUG:boto:Params: {} 2023-09-22T21:43:10.600 DEBUG:boto:Token: None 2023-09-22T21:43:10.600 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.600 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eYlsnEq/twwdnY3HP/M4PD5yPJk= 2023-09-22T21:43:10.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eYlsnEq/twwdnY3HP/M4PD5yPJk=', 'Content-Length': '0'} 2023-09-22T21:43:10.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062a081f4a4e75b95-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.646 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:43:10.646 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:43:10.647 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.647 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.647 DEBUG:boto:Method: GET 2023-09-22T21:43:10.647 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.647 DEBUG:boto:Data: 2023-09-22T21:43:10.647 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.647 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.647 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.647 DEBUG:boto:Params: {} 2023-09-22T21:43:10.648 DEBUG:boto:Token: None 2023-09-22T21:43:10.648 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:43:10.648 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eYlsnEq/twwdnY3HP/M4PD5yPJk= 2023-09-22T21:43:10.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eYlsnEq/twwdnY3HP/M4PD5yPJk=', 'Content-Length': '0'} 2023-09-22T21:43:10.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eede0eaeadc08c6-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.651 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:10.651 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:10.652 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:43:10.652 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:43:10.652 DEBUG:boto:Method: GET 2023-09-22T21:43:10.652 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:43:10.652 DEBUG:boto:Data: 2023-09-22T21:43:10.652 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.652 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.652 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.652 DEBUG:boto:Params: {} 2023-09-22T21:43:10.652 DEBUG:boto:Token: None 2023-09-22T21:43:10.653 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/ 2023-09-22T21:43:10.653 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs= 2023-09-22T21:43:10.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs=', 'Content-Length': '0'} 2023-09-22T21:43:10.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000217733552bb9fb31-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.658 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:10.658 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:10.658 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:10.659 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:43:10.659 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:43:10.659 DEBUG:boto:Method: GET 2023-09-22T21:43:10.659 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:43:10.659 DEBUG:boto:Data: 2023-09-22T21:43:10.659 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.659 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.659 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.659 DEBUG:boto:Params: {} 2023-09-22T21:43:10.660 DEBUG:boto:Token: None 2023-09-22T21:43:10.660 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/ 2023-09-22T21:43:10.660 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs= 2023-09-22T21:43:10.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs=', 'Content-Length': '0'} 2023-09-22T21:43:10.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb709cd5eed23906-00650e0a6e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.710 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:10.711 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:43:10.711 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:43:10.711 DEBUG:boto:Method: HEAD 2023-09-22T21:43:10.712 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:43:10.712 DEBUG:boto:Data: 2023-09-22T21:43:10.712 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.712 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.712 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.712 DEBUG:boto:Params: {} 2023-09-22T21:43:10.713 DEBUG:boto:Token: None 2023-09-22T21:43:10.713 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj68 2023-09-22T21:43:10.713 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QVK5TclkaVCtdDvdoDe12I+SqjU= 2023-09-22T21:43:10.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QVK5TclkaVCtdDvdoDe12I+SqjU=', 'Content-Length': '0'} 2023-09-22T21:43:10.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e04e8bc988403cf6-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.717 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:43:10.717 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:43:10.717 DEBUG:boto:Method: HEAD 2023-09-22T21:43:10.717 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:43:10.717 DEBUG:boto:Data: 2023-09-22T21:43:10.717 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.717 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.718 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.718 DEBUG:boto:Params: {} 2023-09-22T21:43:10.718 DEBUG:boto:Token: None 2023-09-22T21:43:10.718 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj68 2023-09-22T21:43:10.718 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QVK5TclkaVCtdDvdoDe12I+SqjU= 2023-09-22T21:43:10.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QVK5TclkaVCtdDvdoDe12I+SqjU=', 'Content-Length': '0'} 2023-09-22T21:43:10.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000675e314a1474cfa9-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:43:10.721 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:43:10.721 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:43:10.721 DEBUG:boto:Method: GET 2023-09-22T21:43:10.722 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:43:10.722 DEBUG:boto:Data: 2023-09-22T21:43:10.722 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.722 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.722 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.722 DEBUG:boto:Params: {} 2023-09-22T21:43:10.722 DEBUG:boto:Token: None 2023-09-22T21:43:10.723 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj68 2023-09-22T21:43:10.723 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3iOjdsFcaF7VSuTrZVkBmdE3wB4= 2023-09-22T21:43:10.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3iOjdsFcaF7VSuTrZVkBmdE3wB4=', 'Content-Length': '0'} 2023-09-22T21:43:10.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d64627dc56b0c784-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.766 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:43:10.767 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:43:10.767 DEBUG:boto:Method: GET 2023-09-22T21:43:10.767 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:43:10.767 DEBUG:boto:Data: 2023-09-22T21:43:10.767 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.767 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.767 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.767 DEBUG:boto:Params: {} 2023-09-22T21:43:10.768 DEBUG:boto:Token: None 2023-09-22T21:43:10.768 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj68 2023-09-22T21:43:10.768 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3iOjdsFcaF7VSuTrZVkBmdE3wB4= 2023-09-22T21:43:10.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3iOjdsFcaF7VSuTrZVkBmdE3wB4=', 'Content-Length': '0'} 2023-09-22T21:43:10.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb127914a74fde36-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.771 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:10.771 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:10.771 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:43:10.771 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:43:10.772 DEBUG:boto:Method: GET 2023-09-22T21:43:10.772 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:43:10.772 DEBUG:boto:Data: 2023-09-22T21:43:10.772 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.772 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.772 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.772 DEBUG:boto:Params: {} 2023-09-22T21:43:10.772 DEBUG:boto:Token: None 2023-09-22T21:43:10.773 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/ 2023-09-22T21:43:10.773 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs= 2023-09-22T21:43:10.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs=', 'Content-Length': '0'} 2023-09-22T21:43:10.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000190774b10cdea97c-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.778 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:10.779 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:10.779 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:10.779 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:43:10.780 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:43:10.780 DEBUG:boto:Method: GET 2023-09-22T21:43:10.780 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:43:10.780 DEBUG:boto:Data: 2023-09-22T21:43:10.780 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.781 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.781 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.781 DEBUG:boto:Params: {} 2023-09-22T21:43:10.781 DEBUG:boto:Token: None 2023-09-22T21:43:10.782 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/ 2023-09-22T21:43:10.782 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs= 2023-09-22T21:43:10.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs=', 'Content-Length': '0'} 2023-09-22T21:43:10.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013e7189c02654728-00650e0a6e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.789 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:10.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-22T21:43:10.790 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:43:10.790 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:43:10.790 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.790 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.790 DEBUG:boto:Method: GET 2023-09-22T21:43:10.790 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.790 DEBUG:boto:Data: 2023-09-22T21:43:10.791 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.791 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.791 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.791 DEBUG:boto:Params: {} 2023-09-22T21:43:10.791 DEBUG:boto:Token: None 2023-09-22T21:43:10.791 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.791 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KoQlZSjGW/9zKzgybPeUyEJKm7c= 2023-09-22T21:43:10.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KoQlZSjGW/9zKzgybPeUyEJKm7c=', 'Content-Length': '0'} 2023-09-22T21:43:10.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ee970a6e4216d39-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.795 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:43:10.795 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:43:10.795 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.795 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.795 DEBUG:boto:Method: GET 2023-09-22T21:43:10.795 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.795 DEBUG:boto:Data: 2023-09-22T21:43:10.795 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.796 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.796 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.796 DEBUG:boto:Params: {} 2023-09-22T21:43:10.796 DEBUG:boto:Token: None 2023-09-22T21:43:10.796 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.796 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KoQlZSjGW/9zKzgybPeUyEJKm7c= 2023-09-22T21:43:10.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KoQlZSjGW/9zKzgybPeUyEJKm7c=', 'Content-Length': '0'} 2023-09-22T21:43:10.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c029ceeb70fb6686-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.846 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:43:10.847 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:43:10.847 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.847 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.847 DEBUG:boto:Method: HEAD 2023-09-22T21:43:10.847 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.847 DEBUG:boto:Data: 2023-09-22T21:43:10.847 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.847 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.847 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.848 DEBUG:boto:Params: {} 2023-09-22T21:43:10.848 DEBUG:boto:Token: None 2023-09-22T21:43:10.848 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.848 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FDfcOGrWevmiWbipKKfm7BF/CNA= 2023-09-22T21:43:10.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FDfcOGrWevmiWbipKKfm7BF/CNA=', 'Content-Length': '0'} 2023-09-22T21:43:10.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ef0480e6655a7ab-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.851 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:43:10.851 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:43:10.852 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.852 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.852 DEBUG:boto:Method: HEAD 2023-09-22T21:43:10.852 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.853 DEBUG:boto:Data: 2023-09-22T21:43:10.853 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.853 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.853 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.853 DEBUG:boto:Params: {} 2023-09-22T21:43:10.853 DEBUG:boto:Token: None 2023-09-22T21:43:10.853 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.854 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FDfcOGrWevmiWbipKKfm7BF/CNA= 2023-09-22T21:43:10.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FDfcOGrWevmiWbipKKfm7BF/CNA=', 'Content-Length': '0'} 2023-09-22T21:43:10.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b50cef1874196d7b-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-22T21:43:10.857 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:43:10.857 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:43:10.857 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.857 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.857 DEBUG:boto:Method: GET 2023-09-22T21:43:10.857 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.857 DEBUG:boto:Data: 2023-09-22T21:43:10.857 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.858 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.858 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.858 DEBUG:boto:Params: {} 2023-09-22T21:43:10.858 DEBUG:boto:Token: None 2023-09-22T21:43:10.858 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.858 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KoQlZSjGW/9zKzgybPeUyEJKm7c= 2023-09-22T21:43:10.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KoQlZSjGW/9zKzgybPeUyEJKm7c=', 'Content-Length': '0'} 2023-09-22T21:43:10.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f3dc03a8138d849-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.902 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:43:10.903 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:43:10.903 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.903 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.903 DEBUG:boto:Method: GET 2023-09-22T21:43:10.903 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.903 DEBUG:boto:Data: 2023-09-22T21:43:10.903 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.903 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.903 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.903 DEBUG:boto:Params: {} 2023-09-22T21:43:10.904 DEBUG:boto:Token: None 2023-09-22T21:43:10.904 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:43:10.904 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KoQlZSjGW/9zKzgybPeUyEJKm7c= 2023-09-22T21:43:10.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KoQlZSjGW/9zKzgybPeUyEJKm7c=', 'Content-Length': '0'} 2023-09-22T21:43:10.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000055411140f0a72e5-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.907 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:10.907 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:10.908 DEBUG:boto:path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:43:10.908 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:43:10.908 DEBUG:boto:Method: GET 2023-09-22T21:43:10.908 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj69 2023-09-22T21:43:10.908 DEBUG:boto:Data: 2023-09-22T21:43:10.908 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.908 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.908 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.908 DEBUG:boto:Params: {} 2023-09-22T21:43:10.908 DEBUG:boto:Token: None 2023-09-22T21:43:10.909 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/ 2023-09-22T21:43:10.909 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs= 2023-09-22T21:43:10.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs=', 'Content-Length': '0'} 2023-09-22T21:43:10.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009347fe3a69e9cf06-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.914 DEBUG:boto:b'ffzdqu-59obj691000falseobj692023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:10.914 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:10.914 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:10.914 DEBUG:boto:path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:43:10.915 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:43:10.915 DEBUG:boto:Method: GET 2023-09-22T21:43:10.915 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj69 2023-09-22T21:43:10.915 DEBUG:boto:Data: 2023-09-22T21:43:10.915 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.915 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.915 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.915 DEBUG:boto:Params: {} 2023-09-22T21:43:10.915 DEBUG:boto:Token: None 2023-09-22T21:43:10.916 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/ 2023-09-22T21:43:10.916 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs= 2023-09-22T21:43:10.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs=', 'Content-Length': '0'} 2023-09-22T21:43:10.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f0d8afac808eea5-00650e0a6e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.923 DEBUG:boto:b'ffzdqu-59obj691000falseobj692023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:10.923 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:43:10.923 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:43:10.924 DEBUG:boto:Method: HEAD 2023-09-22T21:43:10.924 DEBUG:boto:Path: /ffzdqu-59/obj69 2023-09-22T21:43:10.924 DEBUG:boto:Data: 2023-09-22T21:43:10.924 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.924 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.924 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.924 DEBUG:boto:Params: {} 2023-09-22T21:43:10.924 DEBUG:boto:Token: None 2023-09-22T21:43:10.924 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj69 2023-09-22T21:43:10.925 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4wvTzK16oB1b1OEXxDxOQhtWaBw= 2023-09-22T21:43:10.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4wvTzK16oB1b1OEXxDxOQhtWaBw=', 'Content-Length': '0'} 2023-09-22T21:43:10.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e26694e7efdd3bc7-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.928 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:43:10.928 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:43:10.928 DEBUG:boto:Method: HEAD 2023-09-22T21:43:10.928 DEBUG:boto:Path: /ffzdqu-59/obj69 2023-09-22T21:43:10.928 DEBUG:boto:Data: 2023-09-22T21:43:10.928 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.928 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.928 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.928 DEBUG:boto:Params: {} 2023-09-22T21:43:10.929 DEBUG:boto:Token: None 2023-09-22T21:43:10.929 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj69 2023-09-22T21:43:10.929 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4wvTzK16oB1b1OEXxDxOQhtWaBw= 2023-09-22T21:43:10.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4wvTzK16oB1b1OEXxDxOQhtWaBw=', 'Content-Length': '0'} 2023-09-22T21:43:10.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bc67d06995b172a-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-22T21:43:10.932 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:43:10.932 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:43:10.932 DEBUG:boto:Method: GET 2023-09-22T21:43:10.932 DEBUG:boto:Path: /ffzdqu-59/obj69 2023-09-22T21:43:10.932 DEBUG:boto:Data: 2023-09-22T21:43:10.932 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.933 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.933 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.933 DEBUG:boto:Params: {} 2023-09-22T21:43:10.933 DEBUG:boto:Token: None 2023-09-22T21:43:10.933 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj69 2023-09-22T21:43:10.933 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MYMuDprl6EddEN9dQ0OYEtLS3GY= 2023-09-22T21:43:10.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MYMuDprl6EddEN9dQ0OYEtLS3GY=', 'Content-Length': '0'} 2023-09-22T21:43:10.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eb6cdf8cc4052e8-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.978 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:43:10.978 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:43:10.979 DEBUG:boto:Method: GET 2023-09-22T21:43:10.979 DEBUG:boto:Path: /ffzdqu-59/obj69 2023-09-22T21:43:10.979 DEBUG:boto:Data: 2023-09-22T21:43:10.979 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.979 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.979 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.979 DEBUG:boto:Params: {} 2023-09-22T21:43:10.979 DEBUG:boto:Token: None 2023-09-22T21:43:10.979 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/obj69 2023-09-22T21:43:10.980 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MYMuDprl6EddEN9dQ0OYEtLS3GY= 2023-09-22T21:43:10.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MYMuDprl6EddEN9dQ0OYEtLS3GY=', 'Content-Length': '0'} 2023-09-22T21:43:10.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f686f093d02115ca-00650e0a6e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.983 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:10.983 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:10.983 DEBUG:boto:path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:43:10.983 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:43:10.984 DEBUG:boto:Method: GET 2023-09-22T21:43:10.984 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj69 2023-09-22T21:43:10.984 DEBUG:boto:Data: 2023-09-22T21:43:10.984 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.984 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:10.984 DEBUG:boto:Port: 8001 2023-09-22T21:43:10.984 DEBUG:boto:Params: {} 2023-09-22T21:43:10.984 DEBUG:boto:Token: None 2023-09-22T21:43:10.984 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/ 2023-09-22T21:43:10.985 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs= 2023-09-22T21:43:10.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs=', 'Content-Length': '0'} 2023-09-22T21:43:10.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033aa60259e73b3a1-00650e0a6e-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.990 DEBUG:boto:b'ffzdqu-59obj691000falseobj692023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:10.990 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:10.990 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:10.990 DEBUG:boto:path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:43:10.990 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:43:10.990 DEBUG:boto:Method: GET 2023-09-22T21:43:10.991 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj69 2023-09-22T21:43:10.991 DEBUG:boto:Data: 2023-09-22T21:43:10.991 DEBUG:boto:Headers: {} 2023-09-22T21:43:10.991 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:10.991 DEBUG:boto:Port: 8000 2023-09-22T21:43:10.991 DEBUG:boto:Params: {} 2023-09-22T21:43:10.991 DEBUG:boto:Token: None 2023-09-22T21:43:10.991 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:10 GMT /ffzdqu-59/ 2023-09-22T21:43:10.991 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs= 2023-09-22T21:43:10.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aify2AfARxPYpS9PAPjvBZ/o2fs=', 'Content-Length': '0'} 2023-09-22T21:43:10.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3106342858b73c4-00650e0a6e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:10.999 DEBUG:boto:b'ffzdqu-59obj691000falseobj692023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:10.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T21:43:11.000 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:43:11.000 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:43:11.000 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.000 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.000 DEBUG:boto:Method: GET 2023-09-22T21:43:11.000 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.000 DEBUG:boto:Data: 2023-09-22T21:43:11.001 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.001 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.001 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.001 DEBUG:boto:Params: {} 2023-09-22T21:43:11.001 DEBUG:boto:Token: None 2023-09-22T21:43:11.001 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.001 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rc8p0VTXiK97yu/AGE0OlZcGAqk= 2023-09-22T21:43:11.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rc8p0VTXiK97yu/AGE0OlZcGAqk=', 'Content-Length': '0'} 2023-09-22T21:43:11.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002819f2a4deaafd7a-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.004 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:43:11.004 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:43:11.005 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.005 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.005 DEBUG:boto:Method: GET 2023-09-22T21:43:11.005 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.005 DEBUG:boto:Data: 2023-09-22T21:43:11.005 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.005 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.006 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.006 DEBUG:boto:Params: {} 2023-09-22T21:43:11.006 DEBUG:boto:Token: None 2023-09-22T21:43:11.006 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.006 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rc8p0VTXiK97yu/AGE0OlZcGAqk= 2023-09-22T21:43:11.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rc8p0VTXiK97yu/AGE0OlZcGAqk=', 'Content-Length': '0'} 2023-09-22T21:43:11.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d6433b965709892-00650e0a6f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.009 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:43:11.010 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:43:11.010 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.010 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.010 DEBUG:boto:Method: HEAD 2023-09-22T21:43:11.010 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.010 DEBUG:boto:Data: 2023-09-22T21:43:11.010 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.010 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.010 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.010 DEBUG:boto:Params: {} 2023-09-22T21:43:11.011 DEBUG:boto:Token: None 2023-09-22T21:43:11.011 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.011 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3hP+dPiuMyOZNGw3R9AIkYkufgM= 2023-09-22T21:43:11.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3hP+dPiuMyOZNGw3R9AIkYkufgM=', 'Content-Length': '0'} 2023-09-22T21:43:11.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029551281fb108d45-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.014 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:43:11.014 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:43:11.014 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.014 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.014 DEBUG:boto:Method: HEAD 2023-09-22T21:43:11.014 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.014 DEBUG:boto:Data: 2023-09-22T21:43:11.014 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.015 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.015 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.015 DEBUG:boto:Params: {} 2023-09-22T21:43:11.015 DEBUG:boto:Token: None 2023-09-22T21:43:11.015 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.015 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3hP+dPiuMyOZNGw3R9AIkYkufgM= 2023-09-22T21:43:11.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3hP+dPiuMyOZNGw3R9AIkYkufgM=', 'Content-Length': '0'} 2023-09-22T21:43:11.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a74512bcb2967ea-00650e0a6f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T21:43:11.020 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:43:11.020 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:43:11.021 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.021 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.021 DEBUG:boto:Method: GET 2023-09-22T21:43:11.021 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.021 DEBUG:boto:Data: 2023-09-22T21:43:11.021 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.021 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.021 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.021 DEBUG:boto:Params: {} 2023-09-22T21:43:11.021 DEBUG:boto:Token: None 2023-09-22T21:43:11.022 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.022 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rc8p0VTXiK97yu/AGE0OlZcGAqk= 2023-09-22T21:43:11.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rc8p0VTXiK97yu/AGE0OlZcGAqk=', 'Content-Length': '0'} 2023-09-22T21:43:11.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097ed4aa5d7cced5d-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.066 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:43:11.067 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:43:11.067 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.067 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.067 DEBUG:boto:Method: GET 2023-09-22T21:43:11.068 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.068 DEBUG:boto:Data: 2023-09-22T21:43:11.068 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.068 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.068 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.069 DEBUG:boto:Params: {} 2023-09-22T21:43:11.069 DEBUG:boto:Token: None 2023-09-22T21:43:11.069 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:43:11.070 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rc8p0VTXiK97yu/AGE0OlZcGAqk= 2023-09-22T21:43:11.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rc8p0VTXiK97yu/AGE0OlZcGAqk=', 'Content-Length': '0'} 2023-09-22T21:43:11.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000080d2bdd11410eca-00650e0a6f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.073 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:11.074 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:11.074 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:43:11.075 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:43:11.075 DEBUG:boto:Method: GET 2023-09-22T21:43:11.075 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:43:11.075 DEBUG:boto:Data: 2023-09-22T21:43:11.075 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.075 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.075 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.077 DEBUG:boto:Params: {} 2023-09-22T21:43:11.077 DEBUG:boto:Token: None 2023-09-22T21:43:11.078 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/ 2023-09-22T21:43:11.078 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4= 2023-09-22T21:43:11.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4=', 'Content-Length': '0'} 2023-09-22T21:43:11.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ab20a9632e32492-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.130 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:11.131 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:11.131 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:11.131 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:43:11.131 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:43:11.131 DEBUG:boto:Method: GET 2023-09-22T21:43:11.132 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:43:11.132 DEBUG:boto:Data: 2023-09-22T21:43:11.132 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.132 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.132 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.132 DEBUG:boto:Params: {} 2023-09-22T21:43:11.132 DEBUG:boto:Token: None 2023-09-22T21:43:11.132 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/ 2023-09-22T21:43:11.133 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4= 2023-09-22T21:43:11.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4=', 'Content-Length': '0'} 2023-09-22T21:43:11.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000398ece7bb6f1e22e-00650e0a6f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.141 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:11.141 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:43:11.141 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:43:11.141 DEBUG:boto:Method: HEAD 2023-09-22T21:43:11.141 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:43:11.142 DEBUG:boto:Data: 2023-09-22T21:43:11.142 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.142 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.142 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.142 DEBUG:boto:Params: {} 2023-09-22T21:43:11.142 DEBUG:boto:Token: None 2023-09-22T21:43:11.142 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj76 2023-09-22T21:43:11.142 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:I7eaceTH2B4Me/PrcjlMqW/ddzc= 2023-09-22T21:43:11.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:I7eaceTH2B4Me/PrcjlMqW/ddzc=', 'Content-Length': '0'} 2023-09-22T21:43:11.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdd52fbd7710952f-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.146 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:43:11.146 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:43:11.146 DEBUG:boto:Method: HEAD 2023-09-22T21:43:11.146 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:43:11.146 DEBUG:boto:Data: 2023-09-22T21:43:11.146 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.146 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.146 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.146 DEBUG:boto:Params: {} 2023-09-22T21:43:11.147 DEBUG:boto:Token: None 2023-09-22T21:43:11.147 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj76 2023-09-22T21:43:11.147 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:I7eaceTH2B4Me/PrcjlMqW/ddzc= 2023-09-22T21:43:11.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:I7eaceTH2B4Me/PrcjlMqW/ddzc=', 'Content-Length': '0'} 2023-09-22T21:43:11.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faa6dc7bd7130315-00650e0a6f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T21:43:11.150 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:43:11.150 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:43:11.151 DEBUG:boto:Method: GET 2023-09-22T21:43:11.151 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:43:11.151 DEBUG:boto:Data: 2023-09-22T21:43:11.151 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.152 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.152 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.152 DEBUG:boto:Params: {} 2023-09-22T21:43:11.152 DEBUG:boto:Token: None 2023-09-22T21:43:11.152 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj76 2023-09-22T21:43:11.153 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fOazXkgwR7V1pbN+jI9Lrdgu+AI= 2023-09-22T21:43:11.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fOazXkgwR7V1pbN+jI9Lrdgu+AI=', 'Content-Length': '0'} 2023-09-22T21:43:11.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5c6598beb2b3db3-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.202 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:43:11.203 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:43:11.203 DEBUG:boto:Method: GET 2023-09-22T21:43:11.203 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:43:11.203 DEBUG:boto:Data: 2023-09-22T21:43:11.203 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.203 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.203 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.203 DEBUG:boto:Params: {} 2023-09-22T21:43:11.203 DEBUG:boto:Token: None 2023-09-22T21:43:11.204 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj76 2023-09-22T21:43:11.204 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fOazXkgwR7V1pbN+jI9Lrdgu+AI= 2023-09-22T21:43:11.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fOazXkgwR7V1pbN+jI9Lrdgu+AI=', 'Content-Length': '0'} 2023-09-22T21:43:11.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059fb006373c26a27-00650e0a6f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.207 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:11.207 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:11.207 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:43:11.208 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:43:11.208 DEBUG:boto:Method: GET 2023-09-22T21:43:11.208 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:43:11.208 DEBUG:boto:Data: 2023-09-22T21:43:11.208 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.208 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.208 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.208 DEBUG:boto:Params: {} 2023-09-22T21:43:11.208 DEBUG:boto:Token: None 2023-09-22T21:43:11.208 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/ 2023-09-22T21:43:11.209 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4= 2023-09-22T21:43:11.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4=', 'Content-Length': '0'} 2023-09-22T21:43:11.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cedaec6d5247c4d2-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.258 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:11.259 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:11.259 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:11.259 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:43:11.259 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:43:11.259 DEBUG:boto:Method: GET 2023-09-22T21:43:11.260 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:43:11.260 DEBUG:boto:Data: 2023-09-22T21:43:11.260 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.260 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.260 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.260 DEBUG:boto:Params: {} 2023-09-22T21:43:11.260 DEBUG:boto:Token: None 2023-09-22T21:43:11.260 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/ 2023-09-22T21:43:11.260 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4= 2023-09-22T21:43:11.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4=', 'Content-Length': '0'} 2023-09-22T21:43:11.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c26f674fb614e368-00650e0a6f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.268 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:11.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:43:11.269 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:43:11.269 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:43:11.269 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.269 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.269 DEBUG:boto:Method: GET 2023-09-22T21:43:11.269 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.270 DEBUG:boto:Data: 2023-09-22T21:43:11.270 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.270 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.270 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.270 DEBUG:boto:Params: {} 2023-09-22T21:43:11.270 DEBUG:boto:Token: None 2023-09-22T21:43:11.270 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.270 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KMNzvZOEmmiHJPZgUCiKFiN3tRQ= 2023-09-22T21:43:11.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KMNzvZOEmmiHJPZgUCiKFiN3tRQ=', 'Content-Length': '0'} 2023-09-22T21:43:11.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b5d5141298e5f27-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.274 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:43:11.274 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:43:11.274 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.274 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.274 DEBUG:boto:Method: GET 2023-09-22T21:43:11.275 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.275 DEBUG:boto:Data: 2023-09-22T21:43:11.275 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.275 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.275 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.275 DEBUG:boto:Params: {} 2023-09-22T21:43:11.275 DEBUG:boto:Token: None 2023-09-22T21:43:11.275 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.275 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KMNzvZOEmmiHJPZgUCiKFiN3tRQ= 2023-09-22T21:43:11.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KMNzvZOEmmiHJPZgUCiKFiN3tRQ=', 'Content-Length': '0'} 2023-09-22T21:43:11.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000949123da374f9098-00650e0a6f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.279 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:43:11.279 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:43:11.279 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.279 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.279 DEBUG:boto:Method: HEAD 2023-09-22T21:43:11.279 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.279 DEBUG:boto:Data: 2023-09-22T21:43:11.279 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.279 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.281 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.281 DEBUG:boto:Params: {} 2023-09-22T21:43:11.282 DEBUG:boto:Token: None 2023-09-22T21:43:11.282 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.282 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pWW5ziCWN9/C1SwcQaonCnlO834= 2023-09-22T21:43:11.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pWW5ziCWN9/C1SwcQaonCnlO834=', 'Content-Length': '0'} 2023-09-22T21:43:11.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092aed8295f2a1a7f-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.285 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:43:11.285 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:43:11.285 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.285 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.286 DEBUG:boto:Method: HEAD 2023-09-22T21:43:11.286 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.286 DEBUG:boto:Data: 2023-09-22T21:43:11.286 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.286 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.286 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.286 DEBUG:boto:Params: {} 2023-09-22T21:43:11.286 DEBUG:boto:Token: None 2023-09-22T21:43:11.286 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.287 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pWW5ziCWN9/C1SwcQaonCnlO834= 2023-09-22T21:43:11.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pWW5ziCWN9/C1SwcQaonCnlO834=', 'Content-Length': '0'} 2023-09-22T21:43:11.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec6d2ee1b690b654-00650e0a6f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:43:11.290 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:43:11.290 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:43:11.290 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.290 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.291 DEBUG:boto:Method: GET 2023-09-22T21:43:11.291 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.291 DEBUG:boto:Data: 2023-09-22T21:43:11.291 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.291 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.291 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.291 DEBUG:boto:Params: {} 2023-09-22T21:43:11.291 DEBUG:boto:Token: None 2023-09-22T21:43:11.291 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.292 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KMNzvZOEmmiHJPZgUCiKFiN3tRQ= 2023-09-22T21:43:11.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KMNzvZOEmmiHJPZgUCiKFiN3tRQ=', 'Content-Length': '0'} 2023-09-22T21:43:11.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b536377f2027c8c-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.342 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:43:11.343 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:43:11.343 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.343 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.343 DEBUG:boto:Method: GET 2023-09-22T21:43:11.343 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.343 DEBUG:boto:Data: 2023-09-22T21:43:11.343 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.343 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.343 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.344 DEBUG:boto:Params: {} 2023-09-22T21:43:11.344 DEBUG:boto:Token: None 2023-09-22T21:43:11.344 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:43:11.344 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KMNzvZOEmmiHJPZgUCiKFiN3tRQ= 2023-09-22T21:43:11.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KMNzvZOEmmiHJPZgUCiKFiN3tRQ=', 'Content-Length': '0'} 2023-09-22T21:43:11.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b47f76af66b0ca0-00650e0a6f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.347 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:11.347 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:11.348 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:43:11.348 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:43:11.348 DEBUG:boto:Method: GET 2023-09-22T21:43:11.348 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:43:11.348 DEBUG:boto:Data: 2023-09-22T21:43:11.348 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.348 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.348 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.348 DEBUG:boto:Params: {} 2023-09-22T21:43:11.349 DEBUG:boto:Token: None 2023-09-22T21:43:11.349 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/ 2023-09-22T21:43:11.349 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4= 2023-09-22T21:43:11.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4=', 'Content-Length': '0'} 2023-09-22T21:43:11.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caf9df83099e058e-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.354 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:11.354 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:11.354 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:11.354 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:43:11.355 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:43:11.355 DEBUG:boto:Method: GET 2023-09-22T21:43:11.355 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:43:11.355 DEBUG:boto:Data: 2023-09-22T21:43:11.355 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.355 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.355 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.355 DEBUG:boto:Params: {} 2023-09-22T21:43:11.355 DEBUG:boto:Token: None 2023-09-22T21:43:11.355 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/ 2023-09-22T21:43:11.356 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4= 2023-09-22T21:43:11.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4=', 'Content-Length': '0'} 2023-09-22T21:43:11.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e821c4f8e1921b38-00650e0a6f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.406 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:11.407 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:43:11.407 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:43:11.407 DEBUG:boto:Method: HEAD 2023-09-22T21:43:11.407 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:43:11.407 DEBUG:boto:Data: 2023-09-22T21:43:11.407 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.408 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.408 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.408 DEBUG:boto:Params: {} 2023-09-22T21:43:11.408 DEBUG:boto:Token: None 2023-09-22T21:43:11.408 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj85 2023-09-22T21:43:11.408 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AxcxhewZ8yXP1Hm8S1EcJ4GtQ+c= 2023-09-22T21:43:11.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AxcxhewZ8yXP1Hm8S1EcJ4GtQ+c=', 'Content-Length': '0'} 2023-09-22T21:43:11.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e1ce70f5e9a164b-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.412 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:43:11.412 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:43:11.412 DEBUG:boto:Method: HEAD 2023-09-22T21:43:11.412 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:43:11.412 DEBUG:boto:Data: 2023-09-22T21:43:11.412 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.412 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.412 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.413 DEBUG:boto:Params: {} 2023-09-22T21:43:11.413 DEBUG:boto:Token: None 2023-09-22T21:43:11.413 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj85 2023-09-22T21:43:11.413 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AxcxhewZ8yXP1Hm8S1EcJ4GtQ+c= 2023-09-22T21:43:11.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AxcxhewZ8yXP1Hm8S1EcJ4GtQ+c=', 'Content-Length': '0'} 2023-09-22T21:43:11.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fa60f6f9fe935ad-00650e0a6f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:43:11.417 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:43:11.417 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:43:11.417 DEBUG:boto:Method: GET 2023-09-22T21:43:11.417 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:43:11.417 DEBUG:boto:Data: 2023-09-22T21:43:11.417 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.417 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.417 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.417 DEBUG:boto:Params: {} 2023-09-22T21:43:11.418 DEBUG:boto:Token: None 2023-09-22T21:43:11.418 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj85 2023-09-22T21:43:11.418 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2dX8d9tRuldpqq/ykB2zbQER180= 2023-09-22T21:43:11.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2dX8d9tRuldpqq/ykB2zbQER180=', 'Content-Length': '0'} 2023-09-22T21:43:11.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094d15694a3d3689d-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.462 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:43:11.463 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:43:11.463 DEBUG:boto:Method: GET 2023-09-22T21:43:11.463 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:43:11.463 DEBUG:boto:Data: 2023-09-22T21:43:11.463 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.463 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.463 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.463 DEBUG:boto:Params: {} 2023-09-22T21:43:11.463 DEBUG:boto:Token: None 2023-09-22T21:43:11.464 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj85 2023-09-22T21:43:11.464 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2dX8d9tRuldpqq/ykB2zbQER180= 2023-09-22T21:43:11.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2dX8d9tRuldpqq/ykB2zbQER180=', 'Content-Length': '0'} 2023-09-22T21:43:11.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a12becb810d747b2-00650e0a6f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.467 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:11.467 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:11.467 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:43:11.467 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:43:11.467 DEBUG:boto:Method: GET 2023-09-22T21:43:11.468 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:43:11.468 DEBUG:boto:Data: 2023-09-22T21:43:11.468 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.468 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.468 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.468 DEBUG:boto:Params: {} 2023-09-22T21:43:11.468 DEBUG:boto:Token: None 2023-09-22T21:43:11.468 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/ 2023-09-22T21:43:11.469 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4= 2023-09-22T21:43:11.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4=', 'Content-Length': '0'} 2023-09-22T21:43:11.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050b9b8fadf5942f4-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.473 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:11.473 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:11.474 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:11.474 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:43:11.474 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:43:11.474 DEBUG:boto:Method: GET 2023-09-22T21:43:11.474 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:43:11.474 DEBUG:boto:Data: 2023-09-22T21:43:11.474 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.474 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.475 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.475 DEBUG:boto:Params: {} 2023-09-22T21:43:11.475 DEBUG:boto:Token: None 2023-09-22T21:43:11.475 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/ 2023-09-22T21:43:11.475 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4= 2023-09-22T21:43:11.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4=', 'Content-Length': '0'} 2023-09-22T21:43:11.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a07210343f45fccb-00650e0a6f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.482 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:11.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-22T21:43:11.483 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:43:11.485 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:43:11.485 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.485 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.485 DEBUG:boto:Method: GET 2023-09-22T21:43:11.485 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.485 DEBUG:boto:Data: 2023-09-22T21:43:11.485 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.485 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.485 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.486 DEBUG:boto:Params: {} 2023-09-22T21:43:11.486 DEBUG:boto:Token: None 2023-09-22T21:43:11.486 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.486 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ncDWvp/2VaYlB9WCX2an4SDf0m0= 2023-09-22T21:43:11.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ncDWvp/2VaYlB9WCX2an4SDf0m0=', 'Content-Length': '0'} 2023-09-22T21:43:11.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f8850585e84e513-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.489 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:43:11.489 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:43:11.490 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.490 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.490 DEBUG:boto:Method: GET 2023-09-22T21:43:11.490 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.490 DEBUG:boto:Data: 2023-09-22T21:43:11.490 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.490 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.490 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.490 DEBUG:boto:Params: {} 2023-09-22T21:43:11.491 DEBUG:boto:Token: None 2023-09-22T21:43:11.491 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.491 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ncDWvp/2VaYlB9WCX2an4SDf0m0= 2023-09-22T21:43:11.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ncDWvp/2VaYlB9WCX2an4SDf0m0=', 'Content-Length': '0'} 2023-09-22T21:43:11.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cbb53cf4a0ade9b-00650e0a6f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.534 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:43:11.534 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:43:11.535 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.535 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.535 DEBUG:boto:Method: HEAD 2023-09-22T21:43:11.535 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.535 DEBUG:boto:Data: 2023-09-22T21:43:11.535 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.535 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.535 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.535 DEBUG:boto:Params: {} 2023-09-22T21:43:11.536 DEBUG:boto:Token: None 2023-09-22T21:43:11.536 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.536 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kP3xOfuJFdY9ImQ5bGtnmnkKXes= 2023-09-22T21:43:11.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kP3xOfuJFdY9ImQ5bGtnmnkKXes=', 'Content-Length': '0'} 2023-09-22T21:43:11.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050735296c49a7dc4-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.539 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:43:11.539 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:43:11.539 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.539 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.539 DEBUG:boto:Method: HEAD 2023-09-22T21:43:11.539 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.540 DEBUG:boto:Data: 2023-09-22T21:43:11.540 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.540 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.540 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.540 DEBUG:boto:Params: {} 2023-09-22T21:43:11.540 DEBUG:boto:Token: None 2023-09-22T21:43:11.540 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.540 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kP3xOfuJFdY9ImQ5bGtnmnkKXes= 2023-09-22T21:43:11.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kP3xOfuJFdY9ImQ5bGtnmnkKXes=', 'Content-Length': '0'} 2023-09-22T21:43:11.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee41a6e27763eb7f-00650e0a6f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-22T21:43:11.544 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:43:11.544 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:43:11.544 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.544 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.544 DEBUG:boto:Method: GET 2023-09-22T21:43:11.544 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.544 DEBUG:boto:Data: 2023-09-22T21:43:11.544 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.545 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.545 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.545 DEBUG:boto:Params: {} 2023-09-22T21:43:11.545 DEBUG:boto:Token: None 2023-09-22T21:43:11.545 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.545 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ncDWvp/2VaYlB9WCX2an4SDf0m0= 2023-09-22T21:43:11.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ncDWvp/2VaYlB9WCX2an4SDf0m0=', 'Content-Length': '0'} 2023-09-22T21:43:11.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3d565c48ec5ad8a-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.590 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:43:11.590 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:43:11.591 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.591 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.591 DEBUG:boto:Method: GET 2023-09-22T21:43:11.591 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.591 DEBUG:boto:Data: 2023-09-22T21:43:11.591 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.591 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.591 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.592 DEBUG:boto:Params: {} 2023-09-22T21:43:11.592 DEBUG:boto:Token: None 2023-09-22T21:43:11.592 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:43:11.592 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ncDWvp/2VaYlB9WCX2an4SDf0m0= 2023-09-22T21:43:11.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ncDWvp/2VaYlB9WCX2an4SDf0m0=', 'Content-Length': '0'} 2023-09-22T21:43:11.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009facf7e92ade9553-00650e0a6f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.595 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:11.595 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:11.595 DEBUG:boto:path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:43:11.596 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:43:11.596 DEBUG:boto:Method: GET 2023-09-22T21:43:11.596 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj92 2023-09-22T21:43:11.596 DEBUG:boto:Data: 2023-09-22T21:43:11.596 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.596 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.596 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.596 DEBUG:boto:Params: {} 2023-09-22T21:43:11.596 DEBUG:boto:Token: None 2023-09-22T21:43:11.597 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/ 2023-09-22T21:43:11.597 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4= 2023-09-22T21:43:11.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4=', 'Content-Length': '0'} 2023-09-22T21:43:11.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096946d427ad302b8-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.602 DEBUG:boto:b'ffzdqu-59obj921000falseobj922023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:11.602 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:11.603 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:11.603 DEBUG:boto:path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:43:11.603 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:43:11.603 DEBUG:boto:Method: GET 2023-09-22T21:43:11.603 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj92 2023-09-22T21:43:11.603 DEBUG:boto:Data: 2023-09-22T21:43:11.603 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.603 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.603 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.604 DEBUG:boto:Params: {} 2023-09-22T21:43:11.604 DEBUG:boto:Token: None 2023-09-22T21:43:11.604 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/ 2023-09-22T21:43:11.604 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4= 2023-09-22T21:43:11.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4=', 'Content-Length': '0'} 2023-09-22T21:43:11.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0b3099b31adda3d-00650e0a6f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.611 DEBUG:boto:b'ffzdqu-59obj921000falseobj922023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:11.611 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:43:11.612 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:43:11.612 DEBUG:boto:Method: HEAD 2023-09-22T21:43:11.612 DEBUG:boto:Path: /ffzdqu-59/obj92 2023-09-22T21:43:11.612 DEBUG:boto:Data: 2023-09-22T21:43:11.612 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.612 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.612 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.612 DEBUG:boto:Params: {} 2023-09-22T21:43:11.612 DEBUG:boto:Token: None 2023-09-22T21:43:11.613 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj92 2023-09-22T21:43:11.613 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NH8StkZ6M9yTeD9EM/O3mIUnN8Y= 2023-09-22T21:43:11.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NH8StkZ6M9yTeD9EM/O3mIUnN8Y=', 'Content-Length': '0'} 2023-09-22T21:43:11.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000507b5b66e36f79ec-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.616 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:43:11.616 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:43:11.616 DEBUG:boto:Method: HEAD 2023-09-22T21:43:11.616 DEBUG:boto:Path: /ffzdqu-59/obj92 2023-09-22T21:43:11.616 DEBUG:boto:Data: 2023-09-22T21:43:11.616 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.617 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.617 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.617 DEBUG:boto:Params: {} 2023-09-22T21:43:11.617 DEBUG:boto:Token: None 2023-09-22T21:43:11.617 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj92 2023-09-22T21:43:11.617 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NH8StkZ6M9yTeD9EM/O3mIUnN8Y= 2023-09-22T21:43:11.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NH8StkZ6M9yTeD9EM/O3mIUnN8Y=', 'Content-Length': '0'} 2023-09-22T21:43:11.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fa581a6773061cc-00650e0a6f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-22T21:43:11.620 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:43:11.620 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:43:11.620 DEBUG:boto:Method: GET 2023-09-22T21:43:11.620 DEBUG:boto:Path: /ffzdqu-59/obj92 2023-09-22T21:43:11.620 DEBUG:boto:Data: 2023-09-22T21:43:11.620 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.621 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.621 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.621 DEBUG:boto:Params: {} 2023-09-22T21:43:11.621 DEBUG:boto:Token: None 2023-09-22T21:43:11.621 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj92 2023-09-22T21:43:11.621 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CFGyE7JZC8jFMUgxLf+oiPN0bvI= 2023-09-22T21:43:11.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CFGyE7JZC8jFMUgxLf+oiPN0bvI=', 'Content-Length': '0'} 2023-09-22T21:43:11.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025bf072b29e3bd71-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.666 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:43:11.667 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:43:11.667 DEBUG:boto:Method: GET 2023-09-22T21:43:11.667 DEBUG:boto:Path: /ffzdqu-59/obj92 2023-09-22T21:43:11.667 DEBUG:boto:Data: 2023-09-22T21:43:11.667 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.667 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.667 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.668 DEBUG:boto:Params: {} 2023-09-22T21:43:11.668 DEBUG:boto:Token: None 2023-09-22T21:43:11.668 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj92 2023-09-22T21:43:11.668 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CFGyE7JZC8jFMUgxLf+oiPN0bvI= 2023-09-22T21:43:11.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CFGyE7JZC8jFMUgxLf+oiPN0bvI=', 'Content-Length': '0'} 2023-09-22T21:43:11.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f48a296fe3281758-00650e0a6f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.671 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:11.672 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:11.672 DEBUG:boto:path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:43:11.672 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:43:11.672 DEBUG:boto:Method: GET 2023-09-22T21:43:11.672 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj92 2023-09-22T21:43:11.672 DEBUG:boto:Data: 2023-09-22T21:43:11.672 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.672 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.673 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.673 DEBUG:boto:Params: {} 2023-09-22T21:43:11.673 DEBUG:boto:Token: None 2023-09-22T21:43:11.673 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/ 2023-09-22T21:43:11.673 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4= 2023-09-22T21:43:11.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4=', 'Content-Length': '0'} 2023-09-22T21:43:11.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5ad8f80749d97e6-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.678 DEBUG:boto:b'ffzdqu-59obj921000falseobj922023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:11.679 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:11.679 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:11.679 DEBUG:boto:path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:43:11.679 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:43:11.679 DEBUG:boto:Method: GET 2023-09-22T21:43:11.679 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj92 2023-09-22T21:43:11.680 DEBUG:boto:Data: 2023-09-22T21:43:11.680 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.680 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.680 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.680 DEBUG:boto:Params: {} 2023-09-22T21:43:11.680 DEBUG:boto:Token: None 2023-09-22T21:43:11.680 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/ 2023-09-22T21:43:11.680 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4= 2023-09-22T21:43:11.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4=', 'Content-Length': '0'} 2023-09-22T21:43:11.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b036da510197e0d-00650e0a6f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.687 DEBUG:boto:b'ffzdqu-59obj921000falseobj922023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:11.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T21:43:11.688 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:43:11.688 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:43:11.688 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.689 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.689 DEBUG:boto:Method: GET 2023-09-22T21:43:11.689 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.689 DEBUG:boto:Data: 2023-09-22T21:43:11.689 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.689 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.689 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.689 DEBUG:boto:Params: {} 2023-09-22T21:43:11.689 DEBUG:boto:Token: None 2023-09-22T21:43:11.690 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.690 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jRM3lFaDer4lRMCR74LbaXmu2Dk= 2023-09-22T21:43:11.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jRM3lFaDer4lRMCR74LbaXmu2Dk=', 'Content-Length': '0'} 2023-09-22T21:43:11.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000592310d4b5c13e99-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.693 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:43:11.693 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:43:11.693 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.693 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.693 DEBUG:boto:Method: GET 2023-09-22T21:43:11.693 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.693 DEBUG:boto:Data: 2023-09-22T21:43:11.693 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.694 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.694 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.694 DEBUG:boto:Params: {} 2023-09-22T21:43:11.694 DEBUG:boto:Token: None 2023-09-22T21:43:11.694 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.694 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jRM3lFaDer4lRMCR74LbaXmu2Dk= 2023-09-22T21:43:11.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jRM3lFaDer4lRMCR74LbaXmu2Dk=', 'Content-Length': '0'} 2023-09-22T21:43:11.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e448dbb3457f956-00650e0a6f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.697 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:43:11.697 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:43:11.697 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.698 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.698 DEBUG:boto:Method: HEAD 2023-09-22T21:43:11.698 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.698 DEBUG:boto:Data: 2023-09-22T21:43:11.698 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.698 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.698 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.698 DEBUG:boto:Params: {} 2023-09-22T21:43:11.698 DEBUG:boto:Token: None 2023-09-22T21:43:11.699 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.699 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:A/9Gu3nH9zYoNzusV2M2hXD6i7U= 2023-09-22T21:43:11.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:A/9Gu3nH9zYoNzusV2M2hXD6i7U=', 'Content-Length': '0'} 2023-09-22T21:43:11.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c8a33dc987d8438-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.702 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:43:11.702 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:43:11.702 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.702 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.702 DEBUG:boto:Method: HEAD 2023-09-22T21:43:11.702 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.702 DEBUG:boto:Data: 2023-09-22T21:43:11.702 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.703 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.703 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.703 DEBUG:boto:Params: {} 2023-09-22T21:43:11.703 DEBUG:boto:Token: None 2023-09-22T21:43:11.703 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.703 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:A/9Gu3nH9zYoNzusV2M2hXD6i7U= 2023-09-22T21:43:11.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:A/9Gu3nH9zYoNzusV2M2hXD6i7U=', 'Content-Length': '0'} 2023-09-22T21:43:11.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a210013f89468e47-00650e0a6f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T21:43:11.706 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:43:11.706 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:43:11.707 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.707 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.707 DEBUG:boto:Method: GET 2023-09-22T21:43:11.707 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.707 DEBUG:boto:Data: 2023-09-22T21:43:11.707 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.707 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.707 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.708 DEBUG:boto:Params: {} 2023-09-22T21:43:11.708 DEBUG:boto:Token: None 2023-09-22T21:43:11.708 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.708 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jRM3lFaDer4lRMCR74LbaXmu2Dk= 2023-09-22T21:43:11.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jRM3lFaDer4lRMCR74LbaXmu2Dk=', 'Content-Length': '0'} 2023-09-22T21:43:11.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009babd3f16d224387-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.754 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:43:11.755 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:43:11.755 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.755 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.755 DEBUG:boto:Method: GET 2023-09-22T21:43:11.755 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.755 DEBUG:boto:Data: 2023-09-22T21:43:11.755 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.756 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.756 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.756 DEBUG:boto:Params: {} 2023-09-22T21:43:11.756 DEBUG:boto:Token: None 2023-09-22T21:43:11.756 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:43:11.756 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jRM3lFaDer4lRMCR74LbaXmu2Dk= 2023-09-22T21:43:11.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jRM3lFaDer4lRMCR74LbaXmu2Dk=', 'Content-Length': '0'} 2023-09-22T21:43:11.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000969c6e8c95cc4024-00650e0a6f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.760 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:11.760 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:11.760 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:43:11.760 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:43:11.760 DEBUG:boto:Method: GET 2023-09-22T21:43:11.760 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:43:11.760 DEBUG:boto:Data: 2023-09-22T21:43:11.761 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.761 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.761 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.761 DEBUG:boto:Params: {} 2023-09-22T21:43:11.761 DEBUG:boto:Token: None 2023-09-22T21:43:11.761 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/ 2023-09-22T21:43:11.761 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4= 2023-09-22T21:43:11.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4=', 'Content-Length': '0'} 2023-09-22T21:43:11.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a27ef5e0f087a842-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.810 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:11.811 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:11.811 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:11.811 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:43:11.811 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:43:11.811 DEBUG:boto:Method: GET 2023-09-22T21:43:11.811 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:43:11.811 DEBUG:boto:Data: 2023-09-22T21:43:11.812 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.812 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.812 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.812 DEBUG:boto:Params: {} 2023-09-22T21:43:11.812 DEBUG:boto:Token: None 2023-09-22T21:43:11.812 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/ 2023-09-22T21:43:11.812 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4= 2023-09-22T21:43:11.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4=', 'Content-Length': '0'} 2023-09-22T21:43:11.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d1eda58d4a75a4e-00650e0a6f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.821 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:11.821 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:43:11.822 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:43:11.822 DEBUG:boto:Method: HEAD 2023-09-22T21:43:11.822 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:43:11.822 DEBUG:boto:Data: 2023-09-22T21:43:11.822 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.822 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.822 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.822 DEBUG:boto:Params: {} 2023-09-22T21:43:11.823 DEBUG:boto:Token: None 2023-09-22T21:43:11.823 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj95 2023-09-22T21:43:11.823 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VyNrSxZUB2/lF4HCI8AbGPqSUpo= 2023-09-22T21:43:11.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VyNrSxZUB2/lF4HCI8AbGPqSUpo=', 'Content-Length': '0'} 2023-09-22T21:43:11.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8f4af0a2c4fc371-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.827 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:43:11.827 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:43:11.827 DEBUG:boto:Method: HEAD 2023-09-22T21:43:11.827 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:43:11.827 DEBUG:boto:Data: 2023-09-22T21:43:11.827 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.827 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.827 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.828 DEBUG:boto:Params: {} 2023-09-22T21:43:11.828 DEBUG:boto:Token: None 2023-09-22T21:43:11.828 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj95 2023-09-22T21:43:11.828 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VyNrSxZUB2/lF4HCI8AbGPqSUpo= 2023-09-22T21:43:11.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VyNrSxZUB2/lF4HCI8AbGPqSUpo=', 'Content-Length': '0'} 2023-09-22T21:43:11.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fda782850a39e169-00650e0a6f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T21:43:11.831 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:43:11.831 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:43:11.831 DEBUG:boto:Method: GET 2023-09-22T21:43:11.831 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:43:11.831 DEBUG:boto:Data: 2023-09-22T21:43:11.832 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.832 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.832 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.832 DEBUG:boto:Params: {} 2023-09-22T21:43:11.832 DEBUG:boto:Token: None 2023-09-22T21:43:11.832 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj95 2023-09-22T21:43:11.832 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7lOE3NMc0PlnrevfNjFhzDuer2o= 2023-09-22T21:43:11.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7lOE3NMc0PlnrevfNjFhzDuer2o=', 'Content-Length': '0'} 2023-09-22T21:43:11.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004db203f48d7c9877-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.878 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:43:11.878 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:43:11.879 DEBUG:boto:Method: GET 2023-09-22T21:43:11.879 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:43:11.879 DEBUG:boto:Data: 2023-09-22T21:43:11.879 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.879 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.879 DEBUG:boto:Params: {} 2023-09-22T21:43:11.879 DEBUG:boto:Token: None 2023-09-22T21:43:11.880 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/obj95 2023-09-22T21:43:11.880 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7lOE3NMc0PlnrevfNjFhzDuer2o= 2023-09-22T21:43:11.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7lOE3NMc0PlnrevfNjFhzDuer2o=', 'Content-Length': '0'} 2023-09-22T21:43:11.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dc294442c1e497a-00650e0a6f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.883 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:11.883 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:11.883 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:43:11.883 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:43:11.883 DEBUG:boto:Method: GET 2023-09-22T21:43:11.883 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:43:11.884 DEBUG:boto:Data: 2023-09-22T21:43:11.884 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.884 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:43:11.884 DEBUG:boto:Port: 8001 2023-09-22T21:43:11.884 DEBUG:boto:Params: {} 2023-09-22T21:43:11.884 DEBUG:boto:Token: None 2023-09-22T21:43:11.884 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/ 2023-09-22T21:43:11.884 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4= 2023-09-22T21:43:11.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4=', 'Content-Length': '0'} 2023-09-22T21:43:11.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000570a403be7d6a538-00650e0a6f-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.930 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:11.931 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:11.931 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:11.931 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:43:11.931 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:43:11.931 DEBUG:boto:Method: GET 2023-09-22T21:43:11.931 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:43:11.931 DEBUG:boto:Data: 2023-09-22T21:43:11.932 DEBUG:boto:Headers: {} 2023-09-22T21:43:11.932 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:11.932 DEBUG:boto:Port: 8000 2023-09-22T21:43:11.932 DEBUG:boto:Params: {} 2023-09-22T21:43:11.932 DEBUG:boto:Token: None 2023-09-22T21:43:11.932 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:43:11 GMT /ffzdqu-59/ 2023-09-22T21:43:11.932 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4= 2023-09-22T21:43:11.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2nW+lBj9l4UxjYxEzTjdei3UG/4=', 'Content-Length': '0'} 2023-09-22T21:43:11.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ce946061a45abd0-00650e0a6f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:11.940 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:43:11.940 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-59 zones={a2, a1} 2023-09-22T21:43:11.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:43:11.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:43:11.941 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:43:11.987 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:43:11.987 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:43:12.049 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:43:12.049 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-59 2023-09-22T21:43:12.063 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 20 2023-09-22T21:43:12.128 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:12.128 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:12.128 DEBUG:boto:Method: HEAD 2023-09-22T21:43:12.128 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:43:12.129 DEBUG:boto:Data: 2023-09-22T21:43:12.129 DEBUG:boto:Headers: {} 2023-09-22T21:43:12.129 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:12.129 DEBUG:boto:Port: 8000 2023-09-22T21:43:12.129 DEBUG:boto:Params: {} 2023-09-22T21:43:12.129 DEBUG:boto:Token: None 2023-09-22T21:43:12.129 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:12 GMT /ffzdqu-59/ 2023-09-22T21:43:12.129 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zoUJiGT4qAJy300a2S4SbEZzDrw= 2023-09-22T21:43:12.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zoUJiGT4qAJy300a2S4SbEZzDrw=', 'Content-Length': '0'} 2023-09-22T21:43:12.133 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000246f376771847bd-00650e0a70-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:12.133 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:43:12.133 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:43:12.134 DEBUG:boto:Method: PUT 2023-09-22T21:43:12.134 DEBUG:boto:Path: /ffzdqu-59/obj29 2023-09-22T21:43:12.134 DEBUG:boto:Data: 2023-09-22T21:43:12.134 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:43:12.134 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:12.134 DEBUG:boto:Port: 8000 2023-09-22T21:43:12.134 DEBUG:boto:Params: {} 2023-09-22T21:43:12.134 DEBUG:boto:Token: None 2023-09-22T21:43:12.135 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:43:12 GMT /ffzdqu-59/obj29 2023-09-22T21:43:12.135 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZHduHKKN/KfdmtZ6YMVtT9Wh+m8= 2023-09-22T21:43:12.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:43:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZHduHKKN/KfdmtZ6YMVtT9Wh+m8='} 2023-09-22T21:43:12.141 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb6f922f46126446-00650e0a70-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:12.141 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:12.141 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:12.141 DEBUG:boto:Method: HEAD 2023-09-22T21:43:12.141 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:43:12.141 DEBUG:boto:Data: 2023-09-22T21:43:12.141 DEBUG:boto:Headers: {} 2023-09-22T21:43:12.141 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:12.142 DEBUG:boto:Port: 8000 2023-09-22T21:43:12.142 DEBUG:boto:Params: {} 2023-09-22T21:43:12.142 DEBUG:boto:Token: None 2023-09-22T21:43:12.142 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:12 GMT /ffzdqu-59/ 2023-09-22T21:43:12.142 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zoUJiGT4qAJy300a2S4SbEZzDrw= 2023-09-22T21:43:12.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zoUJiGT4qAJy300a2S4SbEZzDrw=', 'Content-Length': '0'} 2023-09-22T21:43:12.145 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008306e57b676e987e-00650e0a70-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:12.145 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:43:12.146 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:43:12.146 DEBUG:boto:Method: PUT 2023-09-22T21:43:12.146 DEBUG:boto:Path: /ffzdqu-59/obj58 2023-09-22T21:43:12.146 DEBUG:boto:Data: 2023-09-22T21:43:12.146 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:43:12.146 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:12.146 DEBUG:boto:Port: 8000 2023-09-22T21:43:12.146 DEBUG:boto:Params: {} 2023-09-22T21:43:12.146 DEBUG:boto:Token: None 2023-09-22T21:43:12.147 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:43:12 GMT /ffzdqu-59/obj58 2023-09-22T21:43:12.147 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5Axjc80ZQOcYJrf4S1P87qSY+zg= 2023-09-22T21:43:12.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:43:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5Axjc80ZQOcYJrf4S1P87qSY+zg='} 2023-09-22T21:43:12.153 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c11eea0724b4919a-00650e0a70-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:12.153 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:12.153 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:12.153 DEBUG:boto:Method: HEAD 2023-09-22T21:43:12.153 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:43:12.153 DEBUG:boto:Data: 2023-09-22T21:43:12.153 DEBUG:boto:Headers: {} 2023-09-22T21:43:12.153 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:12.154 DEBUG:boto:Port: 8000 2023-09-22T21:43:12.154 DEBUG:boto:Params: {} 2023-09-22T21:43:12.154 DEBUG:boto:Token: None 2023-09-22T21:43:12.154 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:12 GMT /ffzdqu-59/ 2023-09-22T21:43:12.154 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zoUJiGT4qAJy300a2S4SbEZzDrw= 2023-09-22T21:43:12.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zoUJiGT4qAJy300a2S4SbEZzDrw=', 'Content-Length': '0'} 2023-09-22T21:43:12.158 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000acd7cdb464a79228-00650e0a70-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:12.158 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:43:12.158 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:43:12.158 DEBUG:boto:Method: PUT 2023-09-22T21:43:12.158 DEBUG:boto:Path: /ffzdqu-59/obj87 2023-09-22T21:43:12.158 DEBUG:boto:Data: 2023-09-22T21:43:12.158 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:43:12.159 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:12.159 DEBUG:boto:Port: 8000 2023-09-22T21:43:12.159 DEBUG:boto:Params: {} 2023-09-22T21:43:12.159 DEBUG:boto:Token: None 2023-09-22T21:43:12.159 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:43:12 GMT /ffzdqu-59/obj87 2023-09-22T21:43:12.159 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ula3nNCNxxUwVFZqa9OVWGP1zIc= 2023-09-22T21:43:12.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:43:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ula3nNCNxxUwVFZqa9OVWGP1zIc='} 2023-09-22T21:43:12.165 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e156d7d893a3577-00650e0a70-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:12.165 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:12.165 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:12.165 DEBUG:boto:Method: HEAD 2023-09-22T21:43:12.166 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:43:12.166 DEBUG:boto:Data: 2023-09-22T21:43:12.166 DEBUG:boto:Headers: {} 2023-09-22T21:43:12.166 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:12.166 DEBUG:boto:Port: 8000 2023-09-22T21:43:12.166 DEBUG:boto:Params: {} 2023-09-22T21:43:12.166 DEBUG:boto:Token: None 2023-09-22T21:43:12.166 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:12 GMT /ffzdqu-59/ 2023-09-22T21:43:12.167 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zoUJiGT4qAJy300a2S4SbEZzDrw= 2023-09-22T21:43:12.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zoUJiGT4qAJy300a2S4SbEZzDrw=', 'Content-Length': '0'} 2023-09-22T21:43:12.169 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004c503ff31feeb40-00650e0a70-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:12.170 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:43:12.170 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:43:12.170 DEBUG:boto:Method: PUT 2023-09-22T21:43:12.170 DEBUG:boto:Path: /ffzdqu-59/obj116 2023-09-22T21:43:12.170 DEBUG:boto:Data: 2023-09-22T21:43:12.170 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:43:12.170 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:12.170 DEBUG:boto:Port: 8000 2023-09-22T21:43:12.170 DEBUG:boto:Params: {} 2023-09-22T21:43:12.171 DEBUG:boto:Token: None 2023-09-22T21:43:12.171 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:43:12 GMT /ffzdqu-59/obj116 2023-09-22T21:43:12.171 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Zl1aXcrml6WUuvwNuaNmgx9U+zA= 2023-09-22T21:43:12.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:43:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Zl1aXcrml6WUuvwNuaNmgx9U+zA='} 2023-09-22T21:43:12.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bcd511b39da908ea-00650e0a70-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:12.215 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:43:12.215 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:43:12.215 DEBUG:boto:Method: HEAD 2023-09-22T21:43:12.215 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:43:12.215 DEBUG:boto:Data: 2023-09-22T21:43:12.215 DEBUG:boto:Headers: {} 2023-09-22T21:43:12.215 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:12.215 DEBUG:boto:Port: 8000 2023-09-22T21:43:12.216 DEBUG:boto:Params: {} 2023-09-22T21:43:12.216 DEBUG:boto:Token: None 2023-09-22T21:43:12.216 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:43:12 GMT /ffzdqu-59/ 2023-09-22T21:43:12.216 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zoUJiGT4qAJy300a2S4SbEZzDrw= 2023-09-22T21:43:12.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:43:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zoUJiGT4qAJy300a2S4SbEZzDrw=', 'Content-Length': '0'} 2023-09-22T21:43:12.221 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3b4d3c21aa4d54f-00650e0a70-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:12.221 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:43:12.221 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:43:12.222 DEBUG:boto:Method: PUT 2023-09-22T21:43:12.222 DEBUG:boto:Path: /ffzdqu-59/obj145 2023-09-22T21:43:12.222 DEBUG:boto:Data: 2023-09-22T21:43:12.222 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:43:12.222 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:43:12.222 DEBUG:boto:Port: 8000 2023-09-22T21:43:12.222 DEBUG:boto:Params: {} 2023-09-22T21:43:12.222 DEBUG:boto:Token: None 2023-09-22T21:43:12.222 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:43:12 GMT /ffzdqu-59/obj145 2023-09-22T21:43:12.223 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ftxj2/lLxTV8e2L2o/L6+IhWBPQ= 2023-09-22T21:43:12.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:43:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ftxj2/lLxTV8e2L2o/L6+IhWBPQ='} 2023-09-22T21:43:12.232 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000039128edcfe7a275-00650e0a70-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:43:12.232 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-59 2023-09-22T21:43:12.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:43:12.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:43:12.233 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:43:12.280 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:43:12.281 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:43:12.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:43:12.321+0000 7fa915339a40 1 waiting to reach latest gen 4, on 3.. 2023-09-22T21:43:17.324 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:43:17.325+0000 7fa915339a40 1 waiting to reach latest gen 4, on 3.. 2023-09-22T21:43:22.325 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:43:22.325+0000 7fa915339a40 1 waiting to reach latest gen 4, on 3.. 2023-09-22T21:43:27.326 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:43:27.325+0000 7fa915339a40 1 waiting to reach latest gen 4, on 3.. 2023-09-22T21:43:32.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:43:32.329+0000 7fa915339a40 1 waiting for incremental sync to catch up: 2023-09-22T21:43:32.331 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.42.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-22T21:43:32.331 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.42.6, 00000000001.69.6, , , , , 00000000001.68.6, , , , , , , , , , , 00000000002.62.6, 00000000001.57.6, , , , , , , , , , ] 2023-09-22T21:43:37.335 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:43:37.332+0000 7fa915339a40 1 waiting for incremental sync to catch up: 2023-09-22T21:43:37.335 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.42.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-22T21:43:37.336 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.42.6, 00000000001.69.6, , , , , 00000000001.68.6, , , , , , , , , , , 00000000002.62.6, 00000000001.57.6, , , , , , , , , , ] 2023-09-22T21:43:42.339 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:43:42.336+0000 7fa915339a40 1 waiting for incremental sync to catch up: 2023-09-22T21:43:42.340 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.42.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-22T21:43:42.340 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.42.6, 00000000001.69.6, , , , , 00000000001.68.6, , , , , , , , , , , 00000000002.62.6, 00000000001.57.6, , , , , , , , , , ] 2023-09-22T21:43:47.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:43:47.340+0000 7fa915339a40 1 waiting for incremental sync to catch up: 2023-09-22T21:43:47.343 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.42.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-22T21:43:47.343 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.42.6, 00000000001.69.6, , , , , 00000000001.68.6, , , , , , , , , , , 00000000002.62.6, 00000000001.57.6, , , , , , , , , , ] 2023-09-22T21:43:52.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:43:52.344+0000 7fa915339a40 1 waiting for incremental sync to catch up: 2023-09-22T21:43:52.347 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.42.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-22T21:43:52.347 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.42.6, 00000000001.69.6, , , , , 00000000001.68.6, , , , , , , , , , , 00000000002.62.6, 00000000001.57.6, , , , , , , , , , ] 2023-09-22T21:43:57.352 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:43:57.352+0000 7fa915339a40 1 waiting for incremental sync to catch up: 2023-09-22T21:43:57.353 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.42.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-22T21:43:57.353 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.42.6, 00000000001.69.6, , , , , 00000000001.68.6, , , , , , , , , , , 00000000002.62.6, 00000000001.57.6, , , , , , , , , , ] 2023-09-22T21:44:02.356 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:44:02.356+0000 7fa915339a40 1 waiting for incremental sync to catch up: 2023-09-22T21:44:02.357 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.42.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-22T21:44:02.357 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.42.6, 00000000001.69.6, , , , , 00000000001.68.6, , , , , , , , , , , 00000000002.62.6, 00000000001.57.6, , , , , , , , , , ] 2023-09-22T21:44:07.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:44:07.360+0000 7fa915339a40 1 waiting for incremental sync to catch up: 2023-09-22T21:44:07.361 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.42.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-22T21:44:07.361 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.42.6, 00000000001.69.6, , , , , 00000000001.68.6, , , , , , , , , , , 00000000002.62.6, 00000000001.57.6, , , , , , , , , , ] 2023-09-22T21:44:12.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:44:12.364+0000 7fa915339a40 1 waiting for incremental sync to catch up: 2023-09-22T21:44:12.365 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.42.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-22T21:44:12.365 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.42.6, 00000000001.69.6, , , , , 00000000001.68.6, , , , , , , , , , , 00000000002.62.6, 00000000001.57.6, , , , , , , , , , ] 2023-09-22T21:44:17.369 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:44:17.368+0000 7fa915339a40 1 waiting for incremental sync to catch up: 2023-09-22T21:44:17.369 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.42.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-22T21:44:17.369 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.42.6, 00000000001.69.6, , , , , 00000000001.68.6, , , , , , , , , , , 00000000002.62.6, 00000000001.57.6, , , , , , , , , , ] 2023-09-22T21:44:22.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:44:22.372+0000 7fa915339a40 1 waiting for incremental sync to catch up: 2023-09-22T21:44:22.373 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.42.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-22T21:44:22.373 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.42.6, 00000000001.69.6, , , , , 00000000001.68.6, , , , , , , , , , , 00000000002.62.6, 00000000001.57.6, , , , , , , , , , ] 2023-09-22T21:44:27.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:44:27.376+0000 7fa915339a40 1 waiting for incremental sync to catch up: 2023-09-22T21:44:27.377 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.42.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-22T21:44:27.377 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.42.6, 00000000001.69.6, , , , , 00000000001.68.6, , , , , , , , , , , 00000000002.62.6, 00000000001.57.6, , , , , , , , , , ] 2023-09-22T21:44:32.381 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:44:32.380+0000 7fa915339a40 1 waiting for incremental sync to catch up: 2023-09-22T21:44:32.381 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.42.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-22T21:44:32.381 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.42.6, 00000000001.69.6, , , , , 00000000001.68.6, , , , , , , , , , , 00000000002.62.6, 00000000001.57.6, , , , , , , , , , ] 2023-09-22T21:44:37.385 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:44:37.384+0000 7fa915339a40 1 waiting for incremental sync to catch up: 2023-09-22T21:44:38.225 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.42.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-22T21:44:38.226 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.42.6, 00000000001.69.6, , , , , 00000000001.68.6, , , , , , , , , , , 00000000002.62.6, 00000000001.57.6, , , , , , , , , , ] 2023-09-22T21:44:42.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:44:42.388+0000 7fa915339a40 1 waiting for incremental sync to catch up: 2023-09-22T21:44:42.389 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.42.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-22T21:44:42.389 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.42.6, 00000000001.69.6, , , , , 00000000001.68.6, , , , , , , , , , , 00000000002.62.6, 00000000001.57.6, , , , , , , , , , ] 2023-09-22T21:44:47.392 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:44:47.392+0000 7fa915339a40 1 waiting for incremental sync to catch up: 2023-09-22T21:44:47.392 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.42.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-22T21:44:47.393 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.42.6, 00000000001.69.6, , , , , 00000000001.68.6, , , , , , , , , , , 00000000002.62.6, 00000000001.57.6, , , , , , , , , , ] 2023-09-22T21:44:52.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:44:52.396+0000 7fa915339a40 1 bucket sync caught up with source: 2023-09-22T21:44:52.397 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.42.6, 00000000001.69.6, , , , , 00000000001.68.6, , , , , , , , , , , 00000000002.62.6, 00000000001.57.6, , , , , , , , , , ] 2023-09-22T21:44:52.397 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.42.6, 00000000001.69.6, , , , , 00000000001.68.6, , , , , , , , , , , 00000000002.62.6, 00000000001.57.6, , , , , , , , , , ] 2023-09-22T21:44:52.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:44:52.396+0000 7fa915339a40 0 bucket checkpoint complete 2023-09-22T21:44:52.432 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-59 2023-09-22T21:44:52.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:44:52.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:44:52.433 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:44:52.482 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:44:52.482 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:44:52.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:44:52.523+0000 7f008513ba40 1 waiting for incremental sync to catch up: 2023-09-22T21:44:52.527 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.301.6, 00000000002.265.6, 00000000001.144.6, 00000000002.274.6, 00000000003.413.6, , , 00000000002.137.6, 00000000005.162.6, 00000000004.310.6] 2023-09-22T21:44:52.528 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.301.6, 00000000002.265.6, 00000000001.144.6, 00000000003.284.6, 00000000004.417.6, 00000000001.415.6, 00000000001.228.6, 00000000002.137.6, 00000000005.162.6, 00000000004.310.6] 2023-09-22T21:44:57.529 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:44:57.527+0000 7f008513ba40 1 waiting for incremental sync to catch up: 2023-09-22T21:44:57.530 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.301.6, 00000000002.265.6, 00000000001.144.6, 00000000003.284.6, 00000000003.413.6, 00000000001.415.6, 00000000001.228.6, 00000000002.137.6, 00000000005.162.6, 00000000004.310.6] 2023-09-22T21:44:57.530 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.301.6, 00000000002.265.6, 00000000001.144.6, 00000000003.284.6, 00000000004.417.6, 00000000001.415.6, 00000000001.228.6, 00000000002.137.6, 00000000005.162.6, 00000000004.310.6] 2023-09-22T21:45:02.532 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:45:02.527+0000 7f008513ba40 1 waiting for incremental sync to catch up: 2023-09-22T21:45:02.532 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.301.6, 00000000002.265.6, 00000000001.144.6, 00000000003.284.6, 00000000003.413.6, 00000000001.415.6, 00000000001.228.6, 00000000002.137.6, 00000000005.162.6, 00000000004.310.6] 2023-09-22T21:45:02.532 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.301.6, 00000000002.265.6, 00000000001.144.6, 00000000003.284.6, 00000000004.417.6, 00000000001.415.6, 00000000001.228.6, 00000000002.137.6, 00000000005.162.6, 00000000004.310.6] 2023-09-22T21:45:07.534 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:45:07.531+0000 7f008513ba40 1 bucket sync caught up with source: 2023-09-22T21:45:07.534 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.301.6, 00000000002.265.6, 00000000001.144.6, 00000000003.284.6, 00000000004.417.6, 00000000001.415.6, 00000000001.228.6, 00000000002.137.6, 00000000005.162.6, 00000000004.310.6] 2023-09-22T21:45:07.534 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.301.6, 00000000002.265.6, 00000000001.144.6, 00000000003.284.6, 00000000004.417.6, 00000000001.415.6, 00000000001.228.6, 00000000002.137.6, 00000000005.162.6, 00000000004.310.6] 2023-09-22T21:45:07.535 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:45:07.531+0000 7f008513ba40 0 bucket checkpoint complete 2023-09-22T21:45:07.567 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-59 zones={a2, a1} 2023-09-22T21:45:07.567 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:07.567 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:07.568 DEBUG:boto:Method: HEAD 2023-09-22T21:45:07.568 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:45:07.568 DEBUG:boto:Data: 2023-09-22T21:45:07.568 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.568 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:07.568 DEBUG:boto:Port: 8001 2023-09-22T21:45:07.568 DEBUG:boto:Params: {} 2023-09-22T21:45:07.569 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:45:07.569 DEBUG:boto:Token: None 2023-09-22T21:45:07.569 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/ 2023-09-22T21:45:07.569 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Bc42D6+XJHmW6p4vUx3u9halZiM= 2023-09-22T21:45:07.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Bc42D6+XJHmW6p4vUx3u9halZiM=', 'Content-Length': '0'} 2023-09-22T21:45:07.575 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4c5009450419329-00650e0ae3-1b9c-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.575 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:07.575 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:07.575 DEBUG:boto:Method: HEAD 2023-09-22T21:45:07.575 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:45:07.575 DEBUG:boto:Data: 2023-09-22T21:45:07.576 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.576 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:07.576 DEBUG:boto:Port: 8000 2023-09-22T21:45:07.576 DEBUG:boto:Params: {} 2023-09-22T21:45:07.576 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:45:07.576 DEBUG:boto:Token: None 2023-09-22T21:45:07.576 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/ 2023-09-22T21:45:07.577 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Bc42D6+XJHmW6p4vUx3u9halZiM= 2023-09-22T21:45:07.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Bc42D6+XJHmW6p4vUx3u9halZiM=', 'Content-Length': '0'} 2023-09-22T21:45:07.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000819f1223de45b942-00650e0ae3-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.583 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:45:07.583 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:07.583 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:07.583 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:45:07.583 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:45:07.583 DEBUG:boto:Method: GET 2023-09-22T21:45:07.583 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:45:07.583 DEBUG:boto:Data: 2023-09-22T21:45:07.583 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.584 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:07.584 DEBUG:boto:Port: 8001 2023-09-22T21:45:07.584 DEBUG:boto:Params: {} 2023-09-22T21:45:07.584 DEBUG:boto:Token: None 2023-09-22T21:45:07.584 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/?versions 2023-09-22T21:45:07.584 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ETbhy4Mf5wFXIiZkQ5RH5x37dB8= 2023-09-22T21:45:07.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ETbhy4Mf5wFXIiZkQ5RH5x37dB8=', 'Content-Length': '0'} 2023-09-22T21:45:07.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1a8715f8cb86e1e-00650e0ae3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.589 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:07.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T21:45:07.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-22T21:45:07.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-22T21:45:07.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-22T21:45:07.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-22T21:45:07.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-22T21:45:07.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T21:45:07.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-22T21:45:07.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-22T21:45:07.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T21:45:07.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-22T21:45:07.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-22T21:45:07.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T21:45:07.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-22T21:45:07.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-22T21:45:07.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T21:45:07.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-22T21:45:07.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-22T21:45:07.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-22T21:45:07.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-22T21:45:07.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-22T21:45:07.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-22T21:45:07.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-22T21:45:07.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-22T21:45:07.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-22T21:45:07.593 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:45:07.593 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:07.593 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:07.593 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:45:07.593 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:45:07.594 DEBUG:boto:Method: GET 2023-09-22T21:45:07.594 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:45:07.594 DEBUG:boto:Data: 2023-09-22T21:45:07.594 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.594 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:07.594 DEBUG:boto:Port: 8000 2023-09-22T21:45:07.594 DEBUG:boto:Params: {} 2023-09-22T21:45:07.594 DEBUG:boto:Token: None 2023-09-22T21:45:07.594 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/?versions 2023-09-22T21:45:07.595 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ETbhy4Mf5wFXIiZkQ5RH5x37dB8= 2023-09-22T21:45:07.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ETbhy4Mf5wFXIiZkQ5RH5x37dB8=', 'Content-Length': '0'} 2023-09-22T21:45:07.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004de11f95363db68c-00650e0ae3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.601 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:07.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T21:45:07.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-22T21:45:07.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-22T21:45:07.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-22T21:45:07.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-22T21:45:07.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-22T21:45:07.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T21:45:07.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-22T21:45:07.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-22T21:45:07.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T21:45:07.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-22T21:45:07.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-22T21:45:07.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T21:45:07.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-22T21:45:07.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-22T21:45:07.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T21:45:07.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-22T21:45:07.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-22T21:45:07.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-22T21:45:07.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-22T21:45:07.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-22T21:45:07.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-22T21:45:07.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-22T21:45:07.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-22T21:45:07.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-22T21:45:07.605 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:07.605 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:07.606 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:45:07.606 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:45:07.606 DEBUG:boto:Method: GET 2023-09-22T21:45:07.606 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:45:07.606 DEBUG:boto:Data: 2023-09-22T21:45:07.606 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.606 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:07.606 DEBUG:boto:Port: 8001 2023-09-22T21:45:07.606 DEBUG:boto:Params: {} 2023-09-22T21:45:07.607 DEBUG:boto:Token: None 2023-09-22T21:45:07.607 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/?versions 2023-09-22T21:45:07.607 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ETbhy4Mf5wFXIiZkQ5RH5x37dB8= 2023-09-22T21:45:07.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ETbhy4Mf5wFXIiZkQ5RH5x37dB8=', 'Content-Length': '0'} 2023-09-22T21:45:07.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d94b1f3a40b565c9-00650e0ae3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.611 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:07.612 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:07.612 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:07.612 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:45:07.613 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:45:07.613 DEBUG:boto:Method: GET 2023-09-22T21:45:07.613 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:45:07.613 DEBUG:boto:Data: 2023-09-22T21:45:07.613 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.613 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:07.613 DEBUG:boto:Port: 8000 2023-09-22T21:45:07.613 DEBUG:boto:Params: {} 2023-09-22T21:45:07.613 DEBUG:boto:Token: None 2023-09-22T21:45:07.614 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/?versions 2023-09-22T21:45:07.614 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ETbhy4Mf5wFXIiZkQ5RH5x37dB8= 2023-09-22T21:45:07.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ETbhy4Mf5wFXIiZkQ5RH5x37dB8=', 'Content-Length': '0'} 2023-09-22T21:45:07.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9e2b3f3fda15083-00650e0ae3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.620 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:07.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:45:07.622 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:45:07.622 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:45:07.622 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.622 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.622 DEBUG:boto:Method: GET 2023-09-22T21:45:07.622 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.622 DEBUG:boto:Data: 2023-09-22T21:45:07.623 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.623 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:07.623 DEBUG:boto:Port: 8001 2023-09-22T21:45:07.623 DEBUG:boto:Params: {} 2023-09-22T21:45:07.623 DEBUG:boto:Token: None 2023-09-22T21:45:07.623 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.623 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mDj48yFN16Kmakc43qER1lSrAxE= 2023-09-22T21:45:07.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mDj48yFN16Kmakc43qER1lSrAxE=', 'Content-Length': '0'} 2023-09-22T21:45:07.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9ab1c353ae95a32-00650e0ae3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.666 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:45:07.667 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:45:07.667 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.667 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.667 DEBUG:boto:Method: GET 2023-09-22T21:45:07.667 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.667 DEBUG:boto:Data: 2023-09-22T21:45:07.667 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.667 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:07.667 DEBUG:boto:Port: 8000 2023-09-22T21:45:07.668 DEBUG:boto:Params: {} 2023-09-22T21:45:07.668 DEBUG:boto:Token: None 2023-09-22T21:45:07.668 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.668 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mDj48yFN16Kmakc43qER1lSrAxE= 2023-09-22T21:45:07.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mDj48yFN16Kmakc43qER1lSrAxE=', 'Content-Length': '0'} 2023-09-22T21:45:07.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab66500ee68d8e56-00650e0ae3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.671 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:45:07.671 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:45:07.671 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.671 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.671 DEBUG:boto:Method: HEAD 2023-09-22T21:45:07.672 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.672 DEBUG:boto:Data: 2023-09-22T21:45:07.672 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.672 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:07.672 DEBUG:boto:Port: 8001 2023-09-22T21:45:07.672 DEBUG:boto:Params: {} 2023-09-22T21:45:07.672 DEBUG:boto:Token: None 2023-09-22T21:45:07.672 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.673 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:v5E0i0+Czu0qbIGSzFV11rCJ+u4= 2023-09-22T21:45:07.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:v5E0i0+Czu0qbIGSzFV11rCJ+u4=', 'Content-Length': '0'} 2023-09-22T21:45:07.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5492d293945bf94-00650e0ae3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.676 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:45:07.676 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:45:07.676 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.676 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.676 DEBUG:boto:Method: HEAD 2023-09-22T21:45:07.676 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.676 DEBUG:boto:Data: 2023-09-22T21:45:07.677 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.677 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:07.677 DEBUG:boto:Port: 8000 2023-09-22T21:45:07.677 DEBUG:boto:Params: {} 2023-09-22T21:45:07.677 DEBUG:boto:Token: None 2023-09-22T21:45:07.677 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.677 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:v5E0i0+Czu0qbIGSzFV11rCJ+u4= 2023-09-22T21:45:07.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:v5E0i0+Czu0qbIGSzFV11rCJ+u4=', 'Content-Length': '0'} 2023-09-22T21:45:07.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000742e1e152ebb722-00650e0ae3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:45:07.680 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:45:07.680 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:45:07.680 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.680 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.680 DEBUG:boto:Method: GET 2023-09-22T21:45:07.680 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.681 DEBUG:boto:Data: 2023-09-22T21:45:07.681 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.681 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:07.681 DEBUG:boto:Port: 8001 2023-09-22T21:45:07.681 DEBUG:boto:Params: {} 2023-09-22T21:45:07.681 DEBUG:boto:Token: None 2023-09-22T21:45:07.681 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.681 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mDj48yFN16Kmakc43qER1lSrAxE= 2023-09-22T21:45:07.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mDj48yFN16Kmakc43qER1lSrAxE=', 'Content-Length': '0'} 2023-09-22T21:45:07.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc34253a562324c5-00650e0ae3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.726 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:45:07.727 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:45:07.727 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.727 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.727 DEBUG:boto:Method: GET 2023-09-22T21:45:07.727 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.727 DEBUG:boto:Data: 2023-09-22T21:45:07.727 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.727 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:07.727 DEBUG:boto:Port: 8000 2023-09-22T21:45:07.728 DEBUG:boto:Params: {} 2023-09-22T21:45:07.728 DEBUG:boto:Token: None 2023-09-22T21:45:07.728 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:45:07.728 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mDj48yFN16Kmakc43qER1lSrAxE= 2023-09-22T21:45:07.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mDj48yFN16Kmakc43qER1lSrAxE=', 'Content-Length': '0'} 2023-09-22T21:45:07.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eabce1808201b38f-00650e0ae3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.731 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:07.732 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:07.732 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:45:07.732 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:45:07.732 DEBUG:boto:Method: GET 2023-09-22T21:45:07.732 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:45:07.732 DEBUG:boto:Data: 2023-09-22T21:45:07.732 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.732 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:07.733 DEBUG:boto:Port: 8001 2023-09-22T21:45:07.733 DEBUG:boto:Params: {} 2023-09-22T21:45:07.733 DEBUG:boto:Token: None 2023-09-22T21:45:07.733 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/ 2023-09-22T21:45:07.733 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IrN5ZaxsvQdc0RGdDMNXY/iOQY0= 2023-09-22T21:45:07.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IrN5ZaxsvQdc0RGdDMNXY/iOQY0=', 'Content-Length': '0'} 2023-09-22T21:45:07.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aca89e5db1ca5dd1-00650e0ae3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.778 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:07.779 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:07.779 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:07.779 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:45:07.779 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:45:07.779 DEBUG:boto:Method: GET 2023-09-22T21:45:07.780 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:45:07.780 DEBUG:boto:Data: 2023-09-22T21:45:07.780 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.780 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:07.780 DEBUG:boto:Port: 8000 2023-09-22T21:45:07.780 DEBUG:boto:Params: {} 2023-09-22T21:45:07.780 DEBUG:boto:Token: None 2023-09-22T21:45:07.780 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/ 2023-09-22T21:45:07.781 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IrN5ZaxsvQdc0RGdDMNXY/iOQY0= 2023-09-22T21:45:07.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IrN5ZaxsvQdc0RGdDMNXY/iOQY0=', 'Content-Length': '0'} 2023-09-22T21:45:07.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023f81b87dbcef3db-00650e0ae3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.790 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:07.790 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:45:07.790 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:45:07.790 DEBUG:boto:Method: HEAD 2023-09-22T21:45:07.791 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:45:07.791 DEBUG:boto:Data: 2023-09-22T21:45:07.791 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.791 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:07.791 DEBUG:boto:Port: 8001 2023-09-22T21:45:07.791 DEBUG:boto:Params: {} 2023-09-22T21:45:07.791 DEBUG:boto:Token: None 2023-09-22T21:45:07.791 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/obj11 2023-09-22T21:45:07.791 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pY0fsv3HQMrT7FrZvUHqC8u/7eA= 2023-09-22T21:45:07.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pY0fsv3HQMrT7FrZvUHqC8u/7eA=', 'Content-Length': '0'} 2023-09-22T21:45:07.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad54cc7f7f2b1a56-00650e0ae3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.795 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:45:07.795 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:45:07.795 DEBUG:boto:Method: HEAD 2023-09-22T21:45:07.795 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:45:07.795 DEBUG:boto:Data: 2023-09-22T21:45:07.795 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.795 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:07.795 DEBUG:boto:Port: 8000 2023-09-22T21:45:07.796 DEBUG:boto:Params: {} 2023-09-22T21:45:07.796 DEBUG:boto:Token: None 2023-09-22T21:45:07.796 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/obj11 2023-09-22T21:45:07.796 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pY0fsv3HQMrT7FrZvUHqC8u/7eA= 2023-09-22T21:45:07.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pY0fsv3HQMrT7FrZvUHqC8u/7eA=', 'Content-Length': '0'} 2023-09-22T21:45:07.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d61632723e3e98af-00650e0ae3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:45:07.799 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:45:07.799 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:45:07.799 DEBUG:boto:Method: GET 2023-09-22T21:45:07.800 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:45:07.800 DEBUG:boto:Data: 2023-09-22T21:45:07.800 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.800 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:07.800 DEBUG:boto:Port: 8001 2023-09-22T21:45:07.800 DEBUG:boto:Params: {} 2023-09-22T21:45:07.800 DEBUG:boto:Token: None 2023-09-22T21:45:07.800 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/obj11 2023-09-22T21:45:07.800 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hhgGJOALG8uloBJDHdVGPKPEDwg= 2023-09-22T21:45:07.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hhgGJOALG8uloBJDHdVGPKPEDwg=', 'Content-Length': '0'} 2023-09-22T21:45:07.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001760f55b5f951894-00650e0ae3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.804 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:45:07.804 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:45:07.804 DEBUG:boto:Method: GET 2023-09-22T21:45:07.804 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:45:07.804 DEBUG:boto:Data: 2023-09-22T21:45:07.804 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.804 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:07.804 DEBUG:boto:Port: 8000 2023-09-22T21:45:07.804 DEBUG:boto:Params: {} 2023-09-22T21:45:07.805 DEBUG:boto:Token: None 2023-09-22T21:45:07.805 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/obj11 2023-09-22T21:45:07.805 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hhgGJOALG8uloBJDHdVGPKPEDwg= 2023-09-22T21:45:07.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hhgGJOALG8uloBJDHdVGPKPEDwg=', 'Content-Length': '0'} 2023-09-22T21:45:07.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a285b2e6e5345e05-00650e0ae3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.808 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:07.808 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:07.808 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:45:07.808 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:45:07.808 DEBUG:boto:Method: GET 2023-09-22T21:45:07.808 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:45:07.808 DEBUG:boto:Data: 2023-09-22T21:45:07.808 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.809 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:07.809 DEBUG:boto:Port: 8001 2023-09-22T21:45:07.809 DEBUG:boto:Params: {} 2023-09-22T21:45:07.809 DEBUG:boto:Token: None 2023-09-22T21:45:07.809 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/ 2023-09-22T21:45:07.809 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IrN5ZaxsvQdc0RGdDMNXY/iOQY0= 2023-09-22T21:45:07.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IrN5ZaxsvQdc0RGdDMNXY/iOQY0=', 'Content-Length': '0'} 2023-09-22T21:45:07.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9234426b014c56e-00650e0ae3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.858 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:07.859 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:07.859 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:07.859 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:45:07.859 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:45:07.859 DEBUG:boto:Method: GET 2023-09-22T21:45:07.860 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:45:07.860 DEBUG:boto:Data: 2023-09-22T21:45:07.860 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.860 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:07.860 DEBUG:boto:Port: 8000 2023-09-22T21:45:07.860 DEBUG:boto:Params: {} 2023-09-22T21:45:07.860 DEBUG:boto:Token: None 2023-09-22T21:45:07.860 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/ 2023-09-22T21:45:07.860 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IrN5ZaxsvQdc0RGdDMNXY/iOQY0= 2023-09-22T21:45:07.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IrN5ZaxsvQdc0RGdDMNXY/iOQY0=', 'Content-Length': '0'} 2023-09-22T21:45:07.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a193b86c4c154624-00650e0ae3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.877 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:07.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-22T21:45:07.878 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:45:07.878 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:45:07.878 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.879 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.879 DEBUG:boto:Method: GET 2023-09-22T21:45:07.879 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.879 DEBUG:boto:Data: 2023-09-22T21:45:07.879 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:07.879 DEBUG:boto:Port: 8001 2023-09-22T21:45:07.879 DEBUG:boto:Params: {} 2023-09-22T21:45:07.879 DEBUG:boto:Token: None 2023-09-22T21:45:07.880 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.880 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pJLI3qMiwC7pMTcgn56uMlVlnrA= 2023-09-22T21:45:07.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pJLI3qMiwC7pMTcgn56uMlVlnrA=', 'Content-Length': '0'} 2023-09-22T21:45:07.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020216029dddbacbf-00650e0ae3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.883 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:45:07.883 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:45:07.884 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.884 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.884 DEBUG:boto:Method: GET 2023-09-22T21:45:07.884 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.884 DEBUG:boto:Data: 2023-09-22T21:45:07.884 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.884 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:07.884 DEBUG:boto:Port: 8000 2023-09-22T21:45:07.884 DEBUG:boto:Params: {} 2023-09-22T21:45:07.884 DEBUG:boto:Token: None 2023-09-22T21:45:07.885 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.885 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pJLI3qMiwC7pMTcgn56uMlVlnrA= 2023-09-22T21:45:07.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pJLI3qMiwC7pMTcgn56uMlVlnrA=', 'Content-Length': '0'} 2023-09-22T21:45:07.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c589e2f329014e5a-00650e0ae3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.934 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:45:07.935 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:45:07.935 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.935 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.935 DEBUG:boto:Method: HEAD 2023-09-22T21:45:07.935 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.935 DEBUG:boto:Data: 2023-09-22T21:45:07.935 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.935 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:07.935 DEBUG:boto:Port: 8001 2023-09-22T21:45:07.936 DEBUG:boto:Params: {} 2023-09-22T21:45:07.936 DEBUG:boto:Token: None 2023-09-22T21:45:07.936 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.936 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SXtm/FV18WuJYzhgltWn7YjCQPY= 2023-09-22T21:45:07.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SXtm/FV18WuJYzhgltWn7YjCQPY=', 'Content-Length': '0'} 2023-09-22T21:45:07.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005baff0b8890b6379-00650e0ae3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.939 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:45:07.939 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:45:07.940 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.940 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.940 DEBUG:boto:Method: HEAD 2023-09-22T21:45:07.940 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.940 DEBUG:boto:Data: 2023-09-22T21:45:07.940 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.940 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:07.940 DEBUG:boto:Port: 8000 2023-09-22T21:45:07.941 DEBUG:boto:Params: {} 2023-09-22T21:45:07.941 DEBUG:boto:Token: None 2023-09-22T21:45:07.941 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.941 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SXtm/FV18WuJYzhgltWn7YjCQPY= 2023-09-22T21:45:07.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SXtm/FV18WuJYzhgltWn7YjCQPY=', 'Content-Length': '0'} 2023-09-22T21:45:07.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003314f422fcfd32cf-00650e0ae3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-22T21:45:07.944 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:45:07.944 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:45:07.944 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.944 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.944 DEBUG:boto:Method: GET 2023-09-22T21:45:07.944 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.945 DEBUG:boto:Data: 2023-09-22T21:45:07.945 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.945 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:07.945 DEBUG:boto:Port: 8001 2023-09-22T21:45:07.945 DEBUG:boto:Params: {} 2023-09-22T21:45:07.945 DEBUG:boto:Token: None 2023-09-22T21:45:07.945 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.945 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pJLI3qMiwC7pMTcgn56uMlVlnrA= 2023-09-22T21:45:07.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pJLI3qMiwC7pMTcgn56uMlVlnrA=', 'Content-Length': '0'} 2023-09-22T21:45:07.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e156a6987a4fa935-00650e0ae3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.948 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:45:07.948 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:45:07.948 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.949 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.949 DEBUG:boto:Method: GET 2023-09-22T21:45:07.949 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.949 DEBUG:boto:Data: 2023-09-22T21:45:07.949 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.949 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:07.949 DEBUG:boto:Port: 8000 2023-09-22T21:45:07.949 DEBUG:boto:Params: {} 2023-09-22T21:45:07.949 DEBUG:boto:Token: None 2023-09-22T21:45:07.950 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:45:07.950 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pJLI3qMiwC7pMTcgn56uMlVlnrA= 2023-09-22T21:45:07.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pJLI3qMiwC7pMTcgn56uMlVlnrA=', 'Content-Length': '0'} 2023-09-22T21:45:07.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000739c7f120044b008-00650e0ae3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:07.953 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:07.954 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:07.954 DEBUG:boto:path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:45:07.954 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:45:07.954 DEBUG:boto:Method: GET 2023-09-22T21:45:07.954 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj115 2023-09-22T21:45:07.954 DEBUG:boto:Data: 2023-09-22T21:45:07.954 DEBUG:boto:Headers: {} 2023-09-22T21:45:07.954 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:07.955 DEBUG:boto:Port: 8001 2023-09-22T21:45:07.955 DEBUG:boto:Params: {} 2023-09-22T21:45:07.955 DEBUG:boto:Token: None 2023-09-22T21:45:07.955 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:07 GMT /ffzdqu-59/ 2023-09-22T21:45:07.955 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IrN5ZaxsvQdc0RGdDMNXY/iOQY0= 2023-09-22T21:45:07.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IrN5ZaxsvQdc0RGdDMNXY/iOQY0=', 'Content-Length': '0'} 2023-09-22T21:45:07.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c5c98159499a44a-00650e0ae3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.002 DEBUG:boto:b'ffzdqu-59obj1151000falseobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.003 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:08.003 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:08.004 DEBUG:boto:path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:45:08.004 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:45:08.004 DEBUG:boto:Method: GET 2023-09-22T21:45:08.004 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj115 2023-09-22T21:45:08.004 DEBUG:boto:Data: 2023-09-22T21:45:08.005 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.005 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.005 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.005 DEBUG:boto:Params: {} 2023-09-22T21:45:08.006 DEBUG:boto:Token: None 2023-09-22T21:45:08.006 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/ 2023-09-22T21:45:08.006 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU= 2023-09-22T21:45:08.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU=', 'Content-Length': '0'} 2023-09-22T21:45:08.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002641bf17b3caedd4-00650e0ae4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.015 DEBUG:boto:b'ffzdqu-59obj1151000falseobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.016 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:45:08.016 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:45:08.016 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.016 DEBUG:boto:Path: /ffzdqu-59/obj115 2023-09-22T21:45:08.016 DEBUG:boto:Data: 2023-09-22T21:45:08.017 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.017 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.017 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.017 DEBUG:boto:Params: {} 2023-09-22T21:45:08.017 DEBUG:boto:Token: None 2023-09-22T21:45:08.018 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj115 2023-09-22T21:45:08.018 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cn7bwdniNWAiRIglVskqUh+pm14= 2023-09-22T21:45:08.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cn7bwdniNWAiRIglVskqUh+pm14=', 'Content-Length': '0'} 2023-09-22T21:45:08.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000332beda3c54e5531-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.021 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:45:08.022 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:45:08.022 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.022 DEBUG:boto:Path: /ffzdqu-59/obj115 2023-09-22T21:45:08.022 DEBUG:boto:Data: 2023-09-22T21:45:08.023 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.023 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.023 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.023 DEBUG:boto:Params: {} 2023-09-22T21:45:08.024 DEBUG:boto:Token: None 2023-09-22T21:45:08.024 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj115 2023-09-22T21:45:08.024 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cn7bwdniNWAiRIglVskqUh+pm14= 2023-09-22T21:45:08.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cn7bwdniNWAiRIglVskqUh+pm14=', 'Content-Length': '0'} 2023-09-22T21:45:08.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4dc0a2360cee60f-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-22T21:45:08.028 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:45:08.028 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:45:08.028 DEBUG:boto:Method: GET 2023-09-22T21:45:08.029 DEBUG:boto:Path: /ffzdqu-59/obj115 2023-09-22T21:45:08.029 DEBUG:boto:Data: 2023-09-22T21:45:08.029 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.030 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.030 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.030 DEBUG:boto:Params: {} 2023-09-22T21:45:08.030 DEBUG:boto:Token: None 2023-09-22T21:45:08.031 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj115 2023-09-22T21:45:08.031 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7uJTIeEQDKw7DNpN0McRtg3bMnA= 2023-09-22T21:45:08.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7uJTIeEQDKw7DNpN0McRtg3bMnA=', 'Content-Length': '0'} 2023-09-22T21:45:08.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c5a8b6e4787a13d-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.034 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:45:08.034 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:45:08.035 DEBUG:boto:Method: GET 2023-09-22T21:45:08.035 DEBUG:boto:Path: /ffzdqu-59/obj115 2023-09-22T21:45:08.035 DEBUG:boto:Data: 2023-09-22T21:45:08.035 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.035 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.035 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.035 DEBUG:boto:Params: {} 2023-09-22T21:45:08.036 DEBUG:boto:Token: None 2023-09-22T21:45:08.036 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj115 2023-09-22T21:45:08.036 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7uJTIeEQDKw7DNpN0McRtg3bMnA= 2023-09-22T21:45:08.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7uJTIeEQDKw7DNpN0McRtg3bMnA=', 'Content-Length': '0'} 2023-09-22T21:45:08.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000524af154232a1fe-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.086 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:08.087 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:08.087 DEBUG:boto:path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:45:08.087 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:45:08.087 DEBUG:boto:Method: GET 2023-09-22T21:45:08.087 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj115 2023-09-22T21:45:08.087 DEBUG:boto:Data: 2023-09-22T21:45:08.087 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.088 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.088 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.088 DEBUG:boto:Params: {} 2023-09-22T21:45:08.088 DEBUG:boto:Token: None 2023-09-22T21:45:08.088 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/ 2023-09-22T21:45:08.088 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU= 2023-09-22T21:45:08.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU=', 'Content-Length': '0'} 2023-09-22T21:45:08.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038298bb1fe7f345b-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.093 DEBUG:boto:b'ffzdqu-59obj1151000falseobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.094 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:08.094 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:08.094 DEBUG:boto:path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:45:08.094 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:45:08.094 DEBUG:boto:Method: GET 2023-09-22T21:45:08.094 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj115 2023-09-22T21:45:08.094 DEBUG:boto:Data: 2023-09-22T21:45:08.094 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.095 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.095 DEBUG:boto:Params: {} 2023-09-22T21:45:08.095 DEBUG:boto:Token: None 2023-09-22T21:45:08.095 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/ 2023-09-22T21:45:08.095 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU= 2023-09-22T21:45:08.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU=', 'Content-Length': '0'} 2023-09-22T21:45:08.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b964c9fcaa81a57d-00650e0ae4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.103 DEBUG:boto:b'ffzdqu-59obj1151000falseobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-22T21:45:08.104 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:45:08.104 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:45:08.104 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.104 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.104 DEBUG:boto:Method: GET 2023-09-22T21:45:08.104 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.104 DEBUG:boto:Data: 2023-09-22T21:45:08.104 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.105 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.105 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.105 DEBUG:boto:Params: {} 2023-09-22T21:45:08.105 DEBUG:boto:Token: None 2023-09-22T21:45:08.105 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.105 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/k/+K4dktPobT/Yn80wAKo/TNfc= 2023-09-22T21:45:08.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/k/+K4dktPobT/Yn80wAKo/TNfc=', 'Content-Length': '0'} 2023-09-22T21:45:08.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1e0a4f63f5c4457-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.109 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:45:08.109 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:45:08.109 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.109 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.110 DEBUG:boto:Method: GET 2023-09-22T21:45:08.110 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.110 DEBUG:boto:Data: 2023-09-22T21:45:08.110 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.110 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.110 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.110 DEBUG:boto:Params: {} 2023-09-22T21:45:08.110 DEBUG:boto:Token: None 2023-09-22T21:45:08.111 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.111 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/k/+K4dktPobT/Yn80wAKo/TNfc= 2023-09-22T21:45:08.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/k/+K4dktPobT/Yn80wAKo/TNfc=', 'Content-Length': '0'} 2023-09-22T21:45:08.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b35276c4089f65c8-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.114 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:45:08.114 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:45:08.115 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.115 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.115 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.115 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.115 DEBUG:boto:Data: 2023-09-22T21:45:08.115 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.115 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.116 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.116 DEBUG:boto:Params: {} 2023-09-22T21:45:08.116 DEBUG:boto:Token: None 2023-09-22T21:45:08.116 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.116 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nxDs+V61XCgxC+U+64At72pGWP4= 2023-09-22T21:45:08.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nxDs+V61XCgxC+U+64At72pGWP4=', 'Content-Length': '0'} 2023-09-22T21:45:08.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072746b8c28b1781c-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.119 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:45:08.119 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:45:08.119 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.119 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.120 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.120 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.120 DEBUG:boto:Data: 2023-09-22T21:45:08.120 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.120 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.120 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.120 DEBUG:boto:Params: {} 2023-09-22T21:45:08.120 DEBUG:boto:Token: None 2023-09-22T21:45:08.121 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.121 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nxDs+V61XCgxC+U+64At72pGWP4= 2023-09-22T21:45:08.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nxDs+V61XCgxC+U+64At72pGWP4=', 'Content-Length': '0'} 2023-09-22T21:45:08.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025470e86a2f9afe9-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-22T21:45:08.124 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:45:08.124 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:45:08.124 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.124 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.124 DEBUG:boto:Method: GET 2023-09-22T21:45:08.125 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.125 DEBUG:boto:Data: 2023-09-22T21:45:08.125 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.125 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.125 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.125 DEBUG:boto:Params: {} 2023-09-22T21:45:08.125 DEBUG:boto:Token: None 2023-09-22T21:45:08.125 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.126 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/k/+K4dktPobT/Yn80wAKo/TNfc= 2023-09-22T21:45:08.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/k/+K4dktPobT/Yn80wAKo/TNfc=', 'Content-Length': '0'} 2023-09-22T21:45:08.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005195eca94d727fc7-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.170 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:45:08.171 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:45:08.171 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.171 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.171 DEBUG:boto:Method: GET 2023-09-22T21:45:08.171 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.171 DEBUG:boto:Data: 2023-09-22T21:45:08.171 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.172 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.172 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.172 DEBUG:boto:Params: {} 2023-09-22T21:45:08.172 DEBUG:boto:Token: None 2023-09-22T21:45:08.172 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:45:08.172 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/k/+K4dktPobT/Yn80wAKo/TNfc= 2023-09-22T21:45:08.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/k/+K4dktPobT/Yn80wAKo/TNfc=', 'Content-Length': '0'} 2023-09-22T21:45:08.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000676d6100de42fdf9-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.176 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:08.176 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:08.176 DEBUG:boto:path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:45:08.176 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:45:08.176 DEBUG:boto:Method: GET 2023-09-22T21:45:08.176 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj116 2023-09-22T21:45:08.177 DEBUG:boto:Data: 2023-09-22T21:45:08.177 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.177 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.177 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.177 DEBUG:boto:Params: {} 2023-09-22T21:45:08.177 DEBUG:boto:Token: None 2023-09-22T21:45:08.177 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/ 2023-09-22T21:45:08.177 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU= 2023-09-22T21:45:08.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU=', 'Content-Length': '0'} 2023-09-22T21:45:08.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e6e78da33c7d40f-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.183 DEBUG:boto:b'ffzdqu-59obj1161000falseobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.183 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:08.184 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:08.184 DEBUG:boto:path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:45:08.184 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:45:08.184 DEBUG:boto:Method: GET 2023-09-22T21:45:08.184 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj116 2023-09-22T21:45:08.184 DEBUG:boto:Data: 2023-09-22T21:45:08.184 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.184 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.184 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.185 DEBUG:boto:Params: {} 2023-09-22T21:45:08.185 DEBUG:boto:Token: None 2023-09-22T21:45:08.185 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/ 2023-09-22T21:45:08.185 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU= 2023-09-22T21:45:08.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU=', 'Content-Length': '0'} 2023-09-22T21:45:08.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf456b4e6ffd9549-00650e0ae4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.234 DEBUG:boto:b'ffzdqu-59obj1161000falseobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.235 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:45:08.235 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:45:08.235 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.235 DEBUG:boto:Path: /ffzdqu-59/obj116 2023-09-22T21:45:08.235 DEBUG:boto:Data: 2023-09-22T21:45:08.236 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.236 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.236 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.236 DEBUG:boto:Params: {} 2023-09-22T21:45:08.236 DEBUG:boto:Token: None 2023-09-22T21:45:08.236 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj116 2023-09-22T21:45:08.237 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Gzn0Q7ILNJxQDHw65TMHGQHtrGc= 2023-09-22T21:45:08.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Gzn0Q7ILNJxQDHw65TMHGQHtrGc=', 'Content-Length': '0'} 2023-09-22T21:45:08.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027b2f7dc0e91e59c-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.240 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:45:08.240 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:45:08.240 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.240 DEBUG:boto:Path: /ffzdqu-59/obj116 2023-09-22T21:45:08.240 DEBUG:boto:Data: 2023-09-22T21:45:08.241 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.241 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.241 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.241 DEBUG:boto:Params: {} 2023-09-22T21:45:08.241 DEBUG:boto:Token: None 2023-09-22T21:45:08.241 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj116 2023-09-22T21:45:08.241 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Gzn0Q7ILNJxQDHw65TMHGQHtrGc= 2023-09-22T21:45:08.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Gzn0Q7ILNJxQDHw65TMHGQHtrGc=', 'Content-Length': '0'} 2023-09-22T21:45:08.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eaf41c2ed5122ac-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-22T21:45:08.244 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:45:08.245 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:45:08.245 DEBUG:boto:Method: GET 2023-09-22T21:45:08.245 DEBUG:boto:Path: /ffzdqu-59/obj116 2023-09-22T21:45:08.245 DEBUG:boto:Data: 2023-09-22T21:45:08.245 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.245 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.245 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.246 DEBUG:boto:Params: {} 2023-09-22T21:45:08.246 DEBUG:boto:Token: None 2023-09-22T21:45:08.246 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj116 2023-09-22T21:45:08.246 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dCwqcQr2U+G7FWByu2J5My08wak= 2023-09-22T21:45:08.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dCwqcQr2U+G7FWByu2J5My08wak=', 'Content-Length': '0'} 2023-09-22T21:45:08.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f2aa7f676496b06-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.249 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:45:08.249 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:45:08.250 DEBUG:boto:Method: GET 2023-09-22T21:45:08.250 DEBUG:boto:Path: /ffzdqu-59/obj116 2023-09-22T21:45:08.250 DEBUG:boto:Data: 2023-09-22T21:45:08.250 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.250 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.250 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.250 DEBUG:boto:Params: {} 2023-09-22T21:45:08.250 DEBUG:boto:Token: None 2023-09-22T21:45:08.251 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj116 2023-09-22T21:45:08.251 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dCwqcQr2U+G7FWByu2J5My08wak= 2023-09-22T21:45:08.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dCwqcQr2U+G7FWByu2J5My08wak=', 'Content-Length': '0'} 2023-09-22T21:45:08.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000903f3dd38eed7923-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.254 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:08.254 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:08.254 DEBUG:boto:path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:45:08.254 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:45:08.254 DEBUG:boto:Method: GET 2023-09-22T21:45:08.255 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj116 2023-09-22T21:45:08.255 DEBUG:boto:Data: 2023-09-22T21:45:08.255 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.255 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.255 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.255 DEBUG:boto:Params: {} 2023-09-22T21:45:08.255 DEBUG:boto:Token: None 2023-09-22T21:45:08.255 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/ 2023-09-22T21:45:08.256 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU= 2023-09-22T21:45:08.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU=', 'Content-Length': '0'} 2023-09-22T21:45:08.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5f7e835f57f8973-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.261 DEBUG:boto:b'ffzdqu-59obj1161000falseobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.261 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:08.261 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:08.261 DEBUG:boto:path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:45:08.262 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:45:08.262 DEBUG:boto:Method: GET 2023-09-22T21:45:08.262 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj116 2023-09-22T21:45:08.262 DEBUG:boto:Data: 2023-09-22T21:45:08.262 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.262 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.262 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.263 DEBUG:boto:Params: {} 2023-09-22T21:45:08.263 DEBUG:boto:Token: None 2023-09-22T21:45:08.263 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/ 2023-09-22T21:45:08.263 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU= 2023-09-22T21:45:08.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU=', 'Content-Length': '0'} 2023-09-22T21:45:08.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aff2d22a1dbeffce-00650e0ae4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.273 DEBUG:boto:b'ffzdqu-59obj1161000falseobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-22T21:45:08.273 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:45:08.273 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:45:08.274 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.274 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.274 DEBUG:boto:Method: GET 2023-09-22T21:45:08.274 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.274 DEBUG:boto:Data: 2023-09-22T21:45:08.274 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.274 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.275 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.275 DEBUG:boto:Params: {} 2023-09-22T21:45:08.275 DEBUG:boto:Token: None 2023-09-22T21:45:08.275 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.275 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5WkRuHE44XnZOdmFvExI45Z29R0= 2023-09-22T21:45:08.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5WkRuHE44XnZOdmFvExI45Z29R0=', 'Content-Length': '0'} 2023-09-22T21:45:08.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4fecba893e1d53b-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.322 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:45:08.323 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:45:08.323 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.323 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.323 DEBUG:boto:Method: GET 2023-09-22T21:45:08.323 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.323 DEBUG:boto:Data: 2023-09-22T21:45:08.323 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.323 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.324 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.324 DEBUG:boto:Params: {} 2023-09-22T21:45:08.324 DEBUG:boto:Token: None 2023-09-22T21:45:08.324 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.324 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5WkRuHE44XnZOdmFvExI45Z29R0= 2023-09-22T21:45:08.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5WkRuHE44XnZOdmFvExI45Z29R0=', 'Content-Length': '0'} 2023-09-22T21:45:08.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007db4e426e70bb175-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.328 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:45:08.328 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:45:08.328 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.328 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.328 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.328 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.328 DEBUG:boto:Data: 2023-09-22T21:45:08.328 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.328 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.329 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.329 DEBUG:boto:Params: {} 2023-09-22T21:45:08.329 DEBUG:boto:Token: None 2023-09-22T21:45:08.329 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.329 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:muhTc7uHrw3nfHHSJ8WMq9de5GY= 2023-09-22T21:45:08.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:muhTc7uHrw3nfHHSJ8WMq9de5GY=', 'Content-Length': '0'} 2023-09-22T21:45:08.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052a89ceaad0ab6aa-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.332 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:45:08.332 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:45:08.332 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.332 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.332 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.332 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.333 DEBUG:boto:Data: 2023-09-22T21:45:08.333 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.333 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.333 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.333 DEBUG:boto:Params: {} 2023-09-22T21:45:08.333 DEBUG:boto:Token: None 2023-09-22T21:45:08.333 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.333 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:muhTc7uHrw3nfHHSJ8WMq9de5GY= 2023-09-22T21:45:08.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:muhTc7uHrw3nfHHSJ8WMq9de5GY=', 'Content-Length': '0'} 2023-09-22T21:45:08.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa336eac0482bd7d-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-22T21:45:08.337 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:45:08.337 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:45:08.337 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.337 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.337 DEBUG:boto:Method: GET 2023-09-22T21:45:08.337 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.337 DEBUG:boto:Data: 2023-09-22T21:45:08.337 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.337 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.337 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.338 DEBUG:boto:Params: {} 2023-09-22T21:45:08.338 DEBUG:boto:Token: None 2023-09-22T21:45:08.338 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.338 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5WkRuHE44XnZOdmFvExI45Z29R0= 2023-09-22T21:45:08.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5WkRuHE44XnZOdmFvExI45Z29R0=', 'Content-Length': '0'} 2023-09-22T21:45:08.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000272d3470f4e7a725-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.341 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:45:08.341 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:45:08.341 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.341 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.341 DEBUG:boto:Method: GET 2023-09-22T21:45:08.342 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.342 DEBUG:boto:Data: 2023-09-22T21:45:08.342 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.342 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.342 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.342 DEBUG:boto:Params: {} 2023-09-22T21:45:08.342 DEBUG:boto:Token: None 2023-09-22T21:45:08.342 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:45:08.343 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5WkRuHE44XnZOdmFvExI45Z29R0= 2023-09-22T21:45:08.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5WkRuHE44XnZOdmFvExI45Z29R0=', 'Content-Length': '0'} 2023-09-22T21:45:08.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ccfd640eeb7f376-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.386 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:08.387 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:08.387 DEBUG:boto:path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:45:08.387 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:45:08.387 DEBUG:boto:Method: GET 2023-09-22T21:45:08.387 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj145 2023-09-22T21:45:08.387 DEBUG:boto:Data: 2023-09-22T21:45:08.387 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.387 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.387 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.388 DEBUG:boto:Params: {} 2023-09-22T21:45:08.388 DEBUG:boto:Token: None 2023-09-22T21:45:08.388 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/ 2023-09-22T21:45:08.388 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU= 2023-09-22T21:45:08.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU=', 'Content-Length': '0'} 2023-09-22T21:45:08.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000016b71d1e44d5e29-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.394 DEBUG:boto:b'ffzdqu-59obj1451000falseobj1452023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.394 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:08.394 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:08.394 DEBUG:boto:path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:45:08.394 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:45:08.395 DEBUG:boto:Method: GET 2023-09-22T21:45:08.395 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj145 2023-09-22T21:45:08.395 DEBUG:boto:Data: 2023-09-22T21:45:08.395 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.395 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.395 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.395 DEBUG:boto:Params: {} 2023-09-22T21:45:08.395 DEBUG:boto:Token: None 2023-09-22T21:45:08.395 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/ 2023-09-22T21:45:08.396 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU= 2023-09-22T21:45:08.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU=', 'Content-Length': '0'} 2023-09-22T21:45:08.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9b246bac7bc9ca9-00650e0ae4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.404 DEBUG:boto:b'ffzdqu-59obj1451000falseobj1452023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.405 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:45:08.405 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:45:08.405 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.405 DEBUG:boto:Path: /ffzdqu-59/obj145 2023-09-22T21:45:08.405 DEBUG:boto:Data: 2023-09-22T21:45:08.405 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.405 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.405 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.405 DEBUG:boto:Params: {} 2023-09-22T21:45:08.406 DEBUG:boto:Token: None 2023-09-22T21:45:08.406 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj145 2023-09-22T21:45:08.406 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gqbcb/gYMHUpXBwYQvD2l0mRays= 2023-09-22T21:45:08.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gqbcb/gYMHUpXBwYQvD2l0mRays=', 'Content-Length': '0'} 2023-09-22T21:45:08.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066afa59590830f49-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.409 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:45:08.409 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:45:08.409 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.409 DEBUG:boto:Path: /ffzdqu-59/obj145 2023-09-22T21:45:08.409 DEBUG:boto:Data: 2023-09-22T21:45:08.409 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.410 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.410 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.410 DEBUG:boto:Params: {} 2023-09-22T21:45:08.410 DEBUG:boto:Token: None 2023-09-22T21:45:08.410 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj145 2023-09-22T21:45:08.410 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gqbcb/gYMHUpXBwYQvD2l0mRays= 2023-09-22T21:45:08.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gqbcb/gYMHUpXBwYQvD2l0mRays=', 'Content-Length': '0'} 2023-09-22T21:45:08.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1a7b6de8d12716a-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-22T21:45:08.414 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:45:08.414 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:45:08.414 DEBUG:boto:Method: GET 2023-09-22T21:45:08.414 DEBUG:boto:Path: /ffzdqu-59/obj145 2023-09-22T21:45:08.414 DEBUG:boto:Data: 2023-09-22T21:45:08.414 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.414 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.414 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.415 DEBUG:boto:Params: {} 2023-09-22T21:45:08.415 DEBUG:boto:Token: None 2023-09-22T21:45:08.415 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj145 2023-09-22T21:45:08.415 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wwlOoZG72jICZ32QhIuGh0FUca0= 2023-09-22T21:45:08.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wwlOoZG72jICZ32QhIuGh0FUca0=', 'Content-Length': '0'} 2023-09-22T21:45:08.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb4556053d12cd6f-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.462 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:45:08.463 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:45:08.463 DEBUG:boto:Method: GET 2023-09-22T21:45:08.463 DEBUG:boto:Path: /ffzdqu-59/obj145 2023-09-22T21:45:08.463 DEBUG:boto:Data: 2023-09-22T21:45:08.463 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.463 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.463 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.463 DEBUG:boto:Params: {} 2023-09-22T21:45:08.464 DEBUG:boto:Token: None 2023-09-22T21:45:08.464 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj145 2023-09-22T21:45:08.464 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wwlOoZG72jICZ32QhIuGh0FUca0= 2023-09-22T21:45:08.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wwlOoZG72jICZ32QhIuGh0FUca0=', 'Content-Length': '0'} 2023-09-22T21:45:08.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db361b438d9add5a-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.468 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:08.468 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:08.468 DEBUG:boto:path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:45:08.468 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:45:08.468 DEBUG:boto:Method: GET 2023-09-22T21:45:08.468 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj145 2023-09-22T21:45:08.468 DEBUG:boto:Data: 2023-09-22T21:45:08.468 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.469 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.469 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.469 DEBUG:boto:Params: {} 2023-09-22T21:45:08.469 DEBUG:boto:Token: None 2023-09-22T21:45:08.469 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/ 2023-09-22T21:45:08.469 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU= 2023-09-22T21:45:08.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU=', 'Content-Length': '0'} 2023-09-22T21:45:08.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3ec66f7db0a25f7-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.475 DEBUG:boto:b'ffzdqu-59obj1451000falseobj1452023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.475 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:08.476 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:08.476 DEBUG:boto:path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:45:08.476 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:45:08.476 DEBUG:boto:Method: GET 2023-09-22T21:45:08.476 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj145 2023-09-22T21:45:08.476 DEBUG:boto:Data: 2023-09-22T21:45:08.476 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.476 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.476 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.476 DEBUG:boto:Params: {} 2023-09-22T21:45:08.477 DEBUG:boto:Token: None 2023-09-22T21:45:08.477 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/ 2023-09-22T21:45:08.477 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU= 2023-09-22T21:45:08.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU=', 'Content-Length': '0'} 2023-09-22T21:45:08.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000642777e7a5c5ce64-00650e0ae4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.526 DEBUG:boto:b'ffzdqu-59obj1451000falseobj1452023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:45:08.527 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:45:08.527 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:45:08.527 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.527 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.528 DEBUG:boto:Method: GET 2023-09-22T21:45:08.528 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.528 DEBUG:boto:Data: 2023-09-22T21:45:08.528 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.528 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.528 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.528 DEBUG:boto:Params: {} 2023-09-22T21:45:08.528 DEBUG:boto:Token: None 2023-09-22T21:45:08.529 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.529 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8TTpnsyS6uHn8K4SfQr7GxrU1+E= 2023-09-22T21:45:08.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8TTpnsyS6uHn8K4SfQr7GxrU1+E=', 'Content-Length': '0'} 2023-09-22T21:45:08.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fba161a451a44f1-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.533 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:45:08.533 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:45:08.533 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.533 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.533 DEBUG:boto:Method: GET 2023-09-22T21:45:08.533 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.533 DEBUG:boto:Data: 2023-09-22T21:45:08.533 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.533 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.533 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.534 DEBUG:boto:Params: {} 2023-09-22T21:45:08.534 DEBUG:boto:Token: None 2023-09-22T21:45:08.534 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.534 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8TTpnsyS6uHn8K4SfQr7GxrU1+E= 2023-09-22T21:45:08.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8TTpnsyS6uHn8K4SfQr7GxrU1+E=', 'Content-Length': '0'} 2023-09-22T21:45:08.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efb2bb95bfab81b7-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.537 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:45:08.538 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:45:08.538 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.538 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.538 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.538 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.538 DEBUG:boto:Data: 2023-09-22T21:45:08.538 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.538 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.538 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.538 DEBUG:boto:Params: {} 2023-09-22T21:45:08.539 DEBUG:boto:Token: None 2023-09-22T21:45:08.539 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.539 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sjgA4LHTYINNVbTv3N833x/vj9U= 2023-09-22T21:45:08.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sjgA4LHTYINNVbTv3N833x/vj9U=', 'Content-Length': '0'} 2023-09-22T21:45:08.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6e097de0aa5ed11-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.542 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:45:08.542 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:45:08.542 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.542 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.542 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.542 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.542 DEBUG:boto:Data: 2023-09-22T21:45:08.542 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.542 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.543 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.543 DEBUG:boto:Params: {} 2023-09-22T21:45:08.543 DEBUG:boto:Token: None 2023-09-22T21:45:08.543 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.543 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sjgA4LHTYINNVbTv3N833x/vj9U= 2023-09-22T21:45:08.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sjgA4LHTYINNVbTv3N833x/vj9U=', 'Content-Length': '0'} 2023-09-22T21:45:08.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5bbb74df7129dbc-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:45:08.546 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:45:08.546 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:45:08.546 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.546 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.546 DEBUG:boto:Method: GET 2023-09-22T21:45:08.547 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.547 DEBUG:boto:Data: 2023-09-22T21:45:08.547 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.547 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.547 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.547 DEBUG:boto:Params: {} 2023-09-22T21:45:08.547 DEBUG:boto:Token: None 2023-09-22T21:45:08.547 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.547 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8TTpnsyS6uHn8K4SfQr7GxrU1+E= 2023-09-22T21:45:08.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8TTpnsyS6uHn8K4SfQr7GxrU1+E=', 'Content-Length': '0'} 2023-09-22T21:45:08.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a921e0bb39d92d8-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.551 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:45:08.551 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:45:08.551 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.551 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.551 DEBUG:boto:Method: GET 2023-09-22T21:45:08.552 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.552 DEBUG:boto:Data: 2023-09-22T21:45:08.552 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.552 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.552 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.552 DEBUG:boto:Params: {} 2023-09-22T21:45:08.552 DEBUG:boto:Token: None 2023-09-22T21:45:08.552 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:45:08.553 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8TTpnsyS6uHn8K4SfQr7GxrU1+E= 2023-09-22T21:45:08.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8TTpnsyS6uHn8K4SfQr7GxrU1+E=', 'Content-Length': '0'} 2023-09-22T21:45:08.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029a65c072b2edd0a-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.556 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:08.556 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:08.556 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:45:08.556 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:45:08.556 DEBUG:boto:Method: GET 2023-09-22T21:45:08.556 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:45:08.556 DEBUG:boto:Data: 2023-09-22T21:45:08.556 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.556 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.557 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.557 DEBUG:boto:Params: {} 2023-09-22T21:45:08.557 DEBUG:boto:Token: None 2023-09-22T21:45:08.557 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/ 2023-09-22T21:45:08.557 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU= 2023-09-22T21:45:08.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU=', 'Content-Length': '0'} 2023-09-22T21:45:08.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000649769db0ef159ba-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.602 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.603 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:08.603 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:08.603 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:45:08.603 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:45:08.603 DEBUG:boto:Method: GET 2023-09-22T21:45:08.603 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:45:08.603 DEBUG:boto:Data: 2023-09-22T21:45:08.604 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.604 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.604 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.604 DEBUG:boto:Params: {} 2023-09-22T21:45:08.604 DEBUG:boto:Token: None 2023-09-22T21:45:08.604 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/ 2023-09-22T21:45:08.604 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU= 2023-09-22T21:45:08.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU=', 'Content-Length': '0'} 2023-09-22T21:45:08.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005da194533f9d0146-00650e0ae4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.614 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.614 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:45:08.615 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:45:08.615 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.615 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:45:08.615 DEBUG:boto:Data: 2023-09-22T21:45:08.615 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.616 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.616 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.616 DEBUG:boto:Params: {} 2023-09-22T21:45:08.616 DEBUG:boto:Token: None 2023-09-22T21:45:08.616 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj17 2023-09-22T21:45:08.617 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:17ob3mlLzRvsYmY7rK3GQ5vUK6M= 2023-09-22T21:45:08.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:17ob3mlLzRvsYmY7rK3GQ5vUK6M=', 'Content-Length': '0'} 2023-09-22T21:45:08.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000597b0fb2d69c851f-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.620 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:45:08.621 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:45:08.621 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.621 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:45:08.621 DEBUG:boto:Data: 2023-09-22T21:45:08.621 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.621 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.622 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.622 DEBUG:boto:Params: {} 2023-09-22T21:45:08.622 DEBUG:boto:Token: None 2023-09-22T21:45:08.622 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj17 2023-09-22T21:45:08.623 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:17ob3mlLzRvsYmY7rK3GQ5vUK6M= 2023-09-22T21:45:08.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:17ob3mlLzRvsYmY7rK3GQ5vUK6M=', 'Content-Length': '0'} 2023-09-22T21:45:08.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4c116a22e68cb22-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:45:08.626 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:45:08.626 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:45:08.626 DEBUG:boto:Method: GET 2023-09-22T21:45:08.627 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:45:08.627 DEBUG:boto:Data: 2023-09-22T21:45:08.627 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.627 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.627 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.627 DEBUG:boto:Params: {} 2023-09-22T21:45:08.627 DEBUG:boto:Token: None 2023-09-22T21:45:08.628 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj17 2023-09-22T21:45:08.628 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D5VI6B6/R+er6ApDq/RtJd/PPwU= 2023-09-22T21:45:08.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D5VI6B6/R+er6ApDq/RtJd/PPwU=', 'Content-Length': '0'} 2023-09-22T21:45:08.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d22999c689ba6ef0-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.631 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:45:08.631 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:45:08.631 DEBUG:boto:Method: GET 2023-09-22T21:45:08.632 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:45:08.632 DEBUG:boto:Data: 2023-09-22T21:45:08.632 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.632 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.632 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.632 DEBUG:boto:Params: {} 2023-09-22T21:45:08.632 DEBUG:boto:Token: None 2023-09-22T21:45:08.632 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj17 2023-09-22T21:45:08.632 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D5VI6B6/R+er6ApDq/RtJd/PPwU= 2023-09-22T21:45:08.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D5VI6B6/R+er6ApDq/RtJd/PPwU=', 'Content-Length': '0'} 2023-09-22T21:45:08.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000571257740e0f0370-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.636 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:08.636 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:08.636 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:45:08.636 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:45:08.636 DEBUG:boto:Method: GET 2023-09-22T21:45:08.636 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:45:08.636 DEBUG:boto:Data: 2023-09-22T21:45:08.636 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.636 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.636 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.637 DEBUG:boto:Params: {} 2023-09-22T21:45:08.637 DEBUG:boto:Token: None 2023-09-22T21:45:08.637 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/ 2023-09-22T21:45:08.637 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU= 2023-09-22T21:45:08.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU=', 'Content-Length': '0'} 2023-09-22T21:45:08.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035a19496afbef2f5-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.682 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.683 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:08.683 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:08.683 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:45:08.683 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:45:08.683 DEBUG:boto:Method: GET 2023-09-22T21:45:08.684 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:45:08.684 DEBUG:boto:Data: 2023-09-22T21:45:08.684 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.684 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.684 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.684 DEBUG:boto:Params: {} 2023-09-22T21:45:08.684 DEBUG:boto:Token: None 2023-09-22T21:45:08.684 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/ 2023-09-22T21:45:08.685 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU= 2023-09-22T21:45:08.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU=', 'Content-Length': '0'} 2023-09-22T21:45:08.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090ca410059bc6bb4-00650e0ae4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.693 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T21:45:08.693 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:45:08.693 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:45:08.693 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.694 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.694 DEBUG:boto:Method: GET 2023-09-22T21:45:08.694 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.694 DEBUG:boto:Data: 2023-09-22T21:45:08.694 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.694 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.694 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.694 DEBUG:boto:Params: {} 2023-09-22T21:45:08.695 DEBUG:boto:Token: None 2023-09-22T21:45:08.695 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.695 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E8eZjyrjKx3n6boV0mXuQ/7FqI8= 2023-09-22T21:45:08.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E8eZjyrjKx3n6boV0mXuQ/7FqI8=', 'Content-Length': '0'} 2023-09-22T21:45:08.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e819fa6862cb758-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.698 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:45:08.699 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:45:08.699 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.699 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.699 DEBUG:boto:Method: GET 2023-09-22T21:45:08.699 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.699 DEBUG:boto:Data: 2023-09-22T21:45:08.699 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.700 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.700 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.700 DEBUG:boto:Params: {} 2023-09-22T21:45:08.700 DEBUG:boto:Token: None 2023-09-22T21:45:08.700 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.700 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E8eZjyrjKx3n6boV0mXuQ/7FqI8= 2023-09-22T21:45:08.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E8eZjyrjKx3n6boV0mXuQ/7FqI8=', 'Content-Length': '0'} 2023-09-22T21:45:08.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000249293c613fdb20b-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.746 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:45:08.747 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:45:08.747 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.747 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.747 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.747 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.747 DEBUG:boto:Data: 2023-09-22T21:45:08.747 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.747 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.747 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.747 DEBUG:boto:Params: {} 2023-09-22T21:45:08.748 DEBUG:boto:Token: None 2023-09-22T21:45:08.748 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.748 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+cb65StMy/YKHG+O3R3ihw25+5s= 2023-09-22T21:45:08.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+cb65StMy/YKHG+O3R3ihw25+5s=', 'Content-Length': '0'} 2023-09-22T21:45:08.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fca8b8f75f77a4c-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.751 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:45:08.751 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:45:08.751 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.751 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.751 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.752 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.752 DEBUG:boto:Data: 2023-09-22T21:45:08.752 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.752 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.752 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.752 DEBUG:boto:Params: {} 2023-09-22T21:45:08.752 DEBUG:boto:Token: None 2023-09-22T21:45:08.752 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.752 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+cb65StMy/YKHG+O3R3ihw25+5s= 2023-09-22T21:45:08.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+cb65StMy/YKHG+O3R3ihw25+5s=', 'Content-Length': '0'} 2023-09-22T21:45:08.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f432b3e588ba3d75-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T21:45:08.756 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:45:08.756 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:45:08.756 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.756 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.756 DEBUG:boto:Method: GET 2023-09-22T21:45:08.756 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.756 DEBUG:boto:Data: 2023-09-22T21:45:08.756 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.757 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.757 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.757 DEBUG:boto:Params: {} 2023-09-22T21:45:08.757 DEBUG:boto:Token: None 2023-09-22T21:45:08.757 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.757 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E8eZjyrjKx3n6boV0mXuQ/7FqI8= 2023-09-22T21:45:08.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E8eZjyrjKx3n6boV0mXuQ/7FqI8=', 'Content-Length': '0'} 2023-09-22T21:45:08.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ff3e21469313008-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.760 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:45:08.760 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:45:08.760 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.760 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.761 DEBUG:boto:Method: GET 2023-09-22T21:45:08.761 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.761 DEBUG:boto:Data: 2023-09-22T21:45:08.761 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.761 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.761 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.761 DEBUG:boto:Params: {} 2023-09-22T21:45:08.761 DEBUG:boto:Token: None 2023-09-22T21:45:08.761 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:45:08.761 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E8eZjyrjKx3n6boV0mXuQ/7FqI8= 2023-09-22T21:45:08.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E8eZjyrjKx3n6boV0mXuQ/7FqI8=', 'Content-Length': '0'} 2023-09-22T21:45:08.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000623754733436ef3a-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.765 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:08.765 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:08.766 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:45:08.766 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:45:08.766 DEBUG:boto:Method: GET 2023-09-22T21:45:08.766 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:45:08.766 DEBUG:boto:Data: 2023-09-22T21:45:08.766 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.766 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.766 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.766 DEBUG:boto:Params: {} 2023-09-22T21:45:08.767 DEBUG:boto:Token: None 2023-09-22T21:45:08.767 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/ 2023-09-22T21:45:08.767 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU= 2023-09-22T21:45:08.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU=', 'Content-Length': '0'} 2023-09-22T21:45:08.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee60f315846e0b5d-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.814 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.815 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:08.815 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:08.815 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:45:08.816 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:45:08.816 DEBUG:boto:Method: GET 2023-09-22T21:45:08.816 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:45:08.816 DEBUG:boto:Data: 2023-09-22T21:45:08.816 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.816 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.816 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.817 DEBUG:boto:Params: {} 2023-09-22T21:45:08.817 DEBUG:boto:Token: None 2023-09-22T21:45:08.817 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/ 2023-09-22T21:45:08.817 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU= 2023-09-22T21:45:08.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU=', 'Content-Length': '0'} 2023-09-22T21:45:08.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee3194f1b99068c4-00650e0ae4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.826 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.827 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:45:08.827 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:45:08.827 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.827 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:45:08.827 DEBUG:boto:Data: 2023-09-22T21:45:08.827 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.828 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.828 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.828 DEBUG:boto:Params: {} 2023-09-22T21:45:08.828 DEBUG:boto:Token: None 2023-09-22T21:45:08.828 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj19 2023-09-22T21:45:08.828 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:288KGIcp7UN77paw5ymW6yb0Yww= 2023-09-22T21:45:08.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:288KGIcp7UN77paw5ymW6yb0Yww=', 'Content-Length': '0'} 2023-09-22T21:45:08.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7b37d5ad30b4b75-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.831 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:45:08.831 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:45:08.831 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.832 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:45:08.832 DEBUG:boto:Data: 2023-09-22T21:45:08.832 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.832 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.832 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.832 DEBUG:boto:Params: {} 2023-09-22T21:45:08.832 DEBUG:boto:Token: None 2023-09-22T21:45:08.832 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj19 2023-09-22T21:45:08.832 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:288KGIcp7UN77paw5ymW6yb0Yww= 2023-09-22T21:45:08.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:288KGIcp7UN77paw5ymW6yb0Yww=', 'Content-Length': '0'} 2023-09-22T21:45:08.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000216dc502d58bdfce-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T21:45:08.835 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:45:08.836 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:45:08.836 DEBUG:boto:Method: GET 2023-09-22T21:45:08.836 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:45:08.836 DEBUG:boto:Data: 2023-09-22T21:45:08.836 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.836 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.836 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.836 DEBUG:boto:Params: {} 2023-09-22T21:45:08.836 DEBUG:boto:Token: None 2023-09-22T21:45:08.837 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj19 2023-09-22T21:45:08.837 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SxpNvvuM5rzxkV3K/EzGCzloto8= 2023-09-22T21:45:08.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SxpNvvuM5rzxkV3K/EzGCzloto8=', 'Content-Length': '0'} 2023-09-22T21:45:08.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032ef189a92b4e3b3-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.840 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:45:08.840 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:45:08.840 DEBUG:boto:Method: GET 2023-09-22T21:45:08.840 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:45:08.840 DEBUG:boto:Data: 2023-09-22T21:45:08.840 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.840 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.841 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.841 DEBUG:boto:Params: {} 2023-09-22T21:45:08.841 DEBUG:boto:Token: None 2023-09-22T21:45:08.841 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj19 2023-09-22T21:45:08.841 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SxpNvvuM5rzxkV3K/EzGCzloto8= 2023-09-22T21:45:08.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SxpNvvuM5rzxkV3K/EzGCzloto8=', 'Content-Length': '0'} 2023-09-22T21:45:08.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e491bc7ca5b192e-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.886 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:08.887 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:08.887 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:45:08.887 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:45:08.887 DEBUG:boto:Method: GET 2023-09-22T21:45:08.887 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:45:08.887 DEBUG:boto:Data: 2023-09-22T21:45:08.887 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.887 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.887 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.888 DEBUG:boto:Params: {} 2023-09-22T21:45:08.888 DEBUG:boto:Token: None 2023-09-22T21:45:08.888 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/ 2023-09-22T21:45:08.888 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU= 2023-09-22T21:45:08.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU=', 'Content-Length': '0'} 2023-09-22T21:45:08.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ae92a7818224d96-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.893 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.894 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:08.894 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:08.894 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:45:08.894 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:45:08.894 DEBUG:boto:Method: GET 2023-09-22T21:45:08.894 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:45:08.894 DEBUG:boto:Data: 2023-09-22T21:45:08.895 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.895 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.895 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.895 DEBUG:boto:Params: {} 2023-09-22T21:45:08.895 DEBUG:boto:Token: None 2023-09-22T21:45:08.895 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/ 2023-09-22T21:45:08.895 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU= 2023-09-22T21:45:08.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU=', 'Content-Length': '0'} 2023-09-22T21:45:08.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000117d391c226c29ae-00650e0ae4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.904 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:45:08.904 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:45:08.905 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:45:08.905 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.905 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.905 DEBUG:boto:Method: GET 2023-09-22T21:45:08.905 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.905 DEBUG:boto:Data: 2023-09-22T21:45:08.905 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.905 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.905 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.905 DEBUG:boto:Params: {} 2023-09-22T21:45:08.906 DEBUG:boto:Token: None 2023-09-22T21:45:08.906 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.906 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ThOl2+xWe2KkU7KfOENbkoN/Y5Y= 2023-09-22T21:45:08.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ThOl2+xWe2KkU7KfOENbkoN/Y5Y=', 'Content-Length': '0'} 2023-09-22T21:45:08.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9c414cae9549f01-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.950 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:45:08.951 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:45:08.951 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.951 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.951 DEBUG:boto:Method: GET 2023-09-22T21:45:08.951 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.951 DEBUG:boto:Data: 2023-09-22T21:45:08.951 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.951 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.951 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.951 DEBUG:boto:Params: {} 2023-09-22T21:45:08.952 DEBUG:boto:Token: None 2023-09-22T21:45:08.952 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.952 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ThOl2+xWe2KkU7KfOENbkoN/Y5Y= 2023-09-22T21:45:08.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ThOl2+xWe2KkU7KfOENbkoN/Y5Y=', 'Content-Length': '0'} 2023-09-22T21:45:08.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026e3b2ac424054e7-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.955 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:45:08.956 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:45:08.956 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.956 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.956 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.956 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.956 DEBUG:boto:Data: 2023-09-22T21:45:08.956 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.957 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.957 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.957 DEBUG:boto:Params: {} 2023-09-22T21:45:08.957 DEBUG:boto:Token: None 2023-09-22T21:45:08.957 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.957 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ig9RCEyhe56fZcT2gQqSamUL3TQ= 2023-09-22T21:45:08.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ig9RCEyhe56fZcT2gQqSamUL3TQ=', 'Content-Length': '0'} 2023-09-22T21:45:08.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f24efc9f761e6f49-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.960 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:45:08.960 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:45:08.960 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.960 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.961 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.961 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.961 DEBUG:boto:Data: 2023-09-22T21:45:08.961 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.961 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.961 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.961 DEBUG:boto:Params: {} 2023-09-22T21:45:08.961 DEBUG:boto:Token: None 2023-09-22T21:45:08.962 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.962 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ig9RCEyhe56fZcT2gQqSamUL3TQ= 2023-09-22T21:45:08.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ig9RCEyhe56fZcT2gQqSamUL3TQ=', 'Content-Length': '0'} 2023-09-22T21:45:08.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005eb7631538b80c77-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:45:08.965 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:45:08.965 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:45:08.965 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.965 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.965 DEBUG:boto:Method: GET 2023-09-22T21:45:08.965 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.966 DEBUG:boto:Data: 2023-09-22T21:45:08.966 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.966 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.966 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.966 DEBUG:boto:Params: {} 2023-09-22T21:45:08.966 DEBUG:boto:Token: None 2023-09-22T21:45:08.966 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.966 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ThOl2+xWe2KkU7KfOENbkoN/Y5Y= 2023-09-22T21:45:08.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ThOl2+xWe2KkU7KfOENbkoN/Y5Y=', 'Content-Length': '0'} 2023-09-22T21:45:08.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000287dade48450de61-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.969 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:45:08.969 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:45:08.970 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.970 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.970 DEBUG:boto:Method: GET 2023-09-22T21:45:08.970 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.970 DEBUG:boto:Data: 2023-09-22T21:45:08.970 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.970 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.970 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.970 DEBUG:boto:Params: {} 2023-09-22T21:45:08.971 DEBUG:boto:Token: None 2023-09-22T21:45:08.971 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:45:08.971 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ThOl2+xWe2KkU7KfOENbkoN/Y5Y= 2023-09-22T21:45:08.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ThOl2+xWe2KkU7KfOENbkoN/Y5Y=', 'Content-Length': '0'} 2023-09-22T21:45:08.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ea9ceb741de08a5-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.974 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:08.974 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:08.974 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:45:08.974 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:45:08.974 DEBUG:boto:Method: GET 2023-09-22T21:45:08.974 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:45:08.974 DEBUG:boto:Data: 2023-09-22T21:45:08.975 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.975 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.975 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.975 DEBUG:boto:Params: {} 2023-09-22T21:45:08.975 DEBUG:boto:Token: None 2023-09-22T21:45:08.975 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/ 2023-09-22T21:45:08.975 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU= 2023-09-22T21:45:08.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU=', 'Content-Length': '0'} 2023-09-22T21:45:08.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036235a033d488a57-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.980 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.981 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:08.981 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:08.981 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:45:08.981 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:45:08.981 DEBUG:boto:Method: GET 2023-09-22T21:45:08.981 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:45:08.982 DEBUG:boto:Data: 2023-09-22T21:45:08.982 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.982 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.982 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.982 DEBUG:boto:Params: {} 2023-09-22T21:45:08.982 DEBUG:boto:Token: None 2023-09-22T21:45:08.982 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/ 2023-09-22T21:45:08.982 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU= 2023-09-22T21:45:08.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:phT/HoQf5IoovDnLGGb7mkQxCoU=', 'Content-Length': '0'} 2023-09-22T21:45:08.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3a99afdf95a3105-00650e0ae4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.992 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:08.992 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:45:08.992 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:45:08.993 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.993 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:45:08.993 DEBUG:boto:Data: 2023-09-22T21:45:08.993 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.993 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:08.993 DEBUG:boto:Port: 8001 2023-09-22T21:45:08.993 DEBUG:boto:Params: {} 2023-09-22T21:45:08.993 DEBUG:boto:Token: None 2023-09-22T21:45:08.994 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj22 2023-09-22T21:45:08.994 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yISGfO6pmxJbSiR1maZGcLWhVBk= 2023-09-22T21:45:08.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yISGfO6pmxJbSiR1maZGcLWhVBk=', 'Content-Length': '0'} 2023-09-22T21:45:08.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f173ce820e45f50-00650e0ae4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:08.997 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:45:08.997 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:45:08.997 DEBUG:boto:Method: HEAD 2023-09-22T21:45:08.997 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:45:08.997 DEBUG:boto:Data: 2023-09-22T21:45:08.997 DEBUG:boto:Headers: {} 2023-09-22T21:45:08.997 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:08.997 DEBUG:boto:Port: 8000 2023-09-22T21:45:08.998 DEBUG:boto:Params: {} 2023-09-22T21:45:08.998 DEBUG:boto:Token: None 2023-09-22T21:45:08.998 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:08 GMT /ffzdqu-59/obj22 2023-09-22T21:45:08.998 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yISGfO6pmxJbSiR1maZGcLWhVBk= 2023-09-22T21:45:08.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yISGfO6pmxJbSiR1maZGcLWhVBk=', 'Content-Length': '0'} 2023-09-22T21:45:09.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c40127bbbfadc83c-00650e0ae4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:45:09.001 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:45:09.001 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:45:09.002 DEBUG:boto:Method: GET 2023-09-22T21:45:09.002 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:45:09.002 DEBUG:boto:Data: 2023-09-22T21:45:09.002 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.002 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.002 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.002 DEBUG:boto:Params: {} 2023-09-22T21:45:09.002 DEBUG:boto:Token: None 2023-09-22T21:45:09.002 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj22 2023-09-22T21:45:09.003 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6mPiYcCw2ZJYKIe6EJV1Ttj4Skc= 2023-09-22T21:45:09.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6mPiYcCw2ZJYKIe6EJV1Ttj4Skc=', 'Content-Length': '0'} 2023-09-22T21:45:09.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3ac634226a34328-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.046 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:45:09.047 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:45:09.047 DEBUG:boto:Method: GET 2023-09-22T21:45:09.047 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:45:09.047 DEBUG:boto:Data: 2023-09-22T21:45:09.047 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.047 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.047 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.047 DEBUG:boto:Params: {} 2023-09-22T21:45:09.048 DEBUG:boto:Token: None 2023-09-22T21:45:09.048 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj22 2023-09-22T21:45:09.048 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6mPiYcCw2ZJYKIe6EJV1Ttj4Skc= 2023-09-22T21:45:09.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6mPiYcCw2ZJYKIe6EJV1Ttj4Skc=', 'Content-Length': '0'} 2023-09-22T21:45:09.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009206761d84519e5a-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.051 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:09.051 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:09.051 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:45:09.052 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:45:09.052 DEBUG:boto:Method: GET 2023-09-22T21:45:09.052 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:45:09.052 DEBUG:boto:Data: 2023-09-22T21:45:09.052 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.052 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.052 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.052 DEBUG:boto:Params: {} 2023-09-22T21:45:09.052 DEBUG:boto:Token: None 2023-09-22T21:45:09.053 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/ 2023-09-22T21:45:09.053 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo= 2023-09-22T21:45:09.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo=', 'Content-Length': '0'} 2023-09-22T21:45:09.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000752352386f4a6e3a-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.057 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:09.058 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:09.058 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:09.058 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:45:09.058 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:45:09.058 DEBUG:boto:Method: GET 2023-09-22T21:45:09.058 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:45:09.058 DEBUG:boto:Data: 2023-09-22T21:45:09.059 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.059 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.059 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.059 DEBUG:boto:Params: {} 2023-09-22T21:45:09.059 DEBUG:boto:Token: None 2023-09-22T21:45:09.059 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/ 2023-09-22T21:45:09.059 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo= 2023-09-22T21:45:09.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo=', 'Content-Length': '0'} 2023-09-22T21:45:09.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d6e3306e5f8bb74-00650e0ae5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.068 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:09.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-22T21:45:09.069 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:45:09.069 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:45:09.069 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.069 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.069 DEBUG:boto:Method: GET 2023-09-22T21:45:09.069 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.070 DEBUG:boto:Data: 2023-09-22T21:45:09.070 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.070 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.070 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.070 DEBUG:boto:Params: {} 2023-09-22T21:45:09.070 DEBUG:boto:Token: None 2023-09-22T21:45:09.071 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.071 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AwWek0zgRMklUrU2kV7G9bI/tjg= 2023-09-22T21:45:09.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AwWek0zgRMklUrU2kV7G9bI/tjg=', 'Content-Length': '0'} 2023-09-22T21:45:09.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000351c59475be21ef1-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.074 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:45:09.074 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:45:09.074 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.074 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.074 DEBUG:boto:Method: GET 2023-09-22T21:45:09.074 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.075 DEBUG:boto:Data: 2023-09-22T21:45:09.075 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.075 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.075 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.075 DEBUG:boto:Params: {} 2023-09-22T21:45:09.075 DEBUG:boto:Token: None 2023-09-22T21:45:09.075 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.075 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AwWek0zgRMklUrU2kV7G9bI/tjg= 2023-09-22T21:45:09.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AwWek0zgRMklUrU2kV7G9bI/tjg=', 'Content-Length': '0'} 2023-09-22T21:45:09.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011206fe30ca1d3d5-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.078 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:45:09.078 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:45:09.079 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.079 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.079 DEBUG:boto:Method: HEAD 2023-09-22T21:45:09.079 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.079 DEBUG:boto:Data: 2023-09-22T21:45:09.079 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.079 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.079 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.079 DEBUG:boto:Params: {} 2023-09-22T21:45:09.080 DEBUG:boto:Token: None 2023-09-22T21:45:09.080 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.080 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RsA2Fl0m1xZmP/F0sKRUxkY+nEw= 2023-09-22T21:45:09.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RsA2Fl0m1xZmP/F0sKRUxkY+nEw=', 'Content-Length': '0'} 2023-09-22T21:45:09.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013b116d7cfe17849-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.083 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:45:09.083 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:45:09.083 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.083 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.083 DEBUG:boto:Method: HEAD 2023-09-22T21:45:09.083 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.084 DEBUG:boto:Data: 2023-09-22T21:45:09.084 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.084 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.084 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.084 DEBUG:boto:Params: {} 2023-09-22T21:45:09.084 DEBUG:boto:Token: None 2023-09-22T21:45:09.084 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.084 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RsA2Fl0m1xZmP/F0sKRUxkY+nEw= 2023-09-22T21:45:09.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RsA2Fl0m1xZmP/F0sKRUxkY+nEw=', 'Content-Length': '0'} 2023-09-22T21:45:09.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adb7895f225c11d6-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-22T21:45:09.088 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:45:09.088 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:45:09.088 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.088 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.088 DEBUG:boto:Method: GET 2023-09-22T21:45:09.088 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.088 DEBUG:boto:Data: 2023-09-22T21:45:09.088 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.088 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.088 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.089 DEBUG:boto:Params: {} 2023-09-22T21:45:09.089 DEBUG:boto:Token: None 2023-09-22T21:45:09.089 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.089 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AwWek0zgRMklUrU2kV7G9bI/tjg= 2023-09-22T21:45:09.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AwWek0zgRMklUrU2kV7G9bI/tjg=', 'Content-Length': '0'} 2023-09-22T21:45:09.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec781431cd0eef7f-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.138 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:45:09.139 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:45:09.139 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.139 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.139 DEBUG:boto:Method: GET 2023-09-22T21:45:09.139 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.139 DEBUG:boto:Data: 2023-09-22T21:45:09.139 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.139 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.140 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.140 DEBUG:boto:Params: {} 2023-09-22T21:45:09.140 DEBUG:boto:Token: None 2023-09-22T21:45:09.140 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:45:09.140 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AwWek0zgRMklUrU2kV7G9bI/tjg= 2023-09-22T21:45:09.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AwWek0zgRMklUrU2kV7G9bI/tjg=', 'Content-Length': '0'} 2023-09-22T21:45:09.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c90369a5da0f9903-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.143 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:09.144 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:09.144 DEBUG:boto:path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:45:09.144 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:45:09.144 DEBUG:boto:Method: GET 2023-09-22T21:45:09.144 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj23 2023-09-22T21:45:09.144 DEBUG:boto:Data: 2023-09-22T21:45:09.144 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.144 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.145 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.145 DEBUG:boto:Params: {} 2023-09-22T21:45:09.145 DEBUG:boto:Token: None 2023-09-22T21:45:09.145 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/ 2023-09-22T21:45:09.145 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo= 2023-09-22T21:45:09.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo=', 'Content-Length': '0'} 2023-09-22T21:45:09.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af8fb53bc94ad89d-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.151 DEBUG:boto:b'ffzdqu-59obj231000falseobj232023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:09.151 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:09.151 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:09.151 DEBUG:boto:path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:45:09.151 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:45:09.152 DEBUG:boto:Method: GET 2023-09-22T21:45:09.153 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj23 2023-09-22T21:45:09.153 DEBUG:boto:Data: 2023-09-22T21:45:09.154 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.154 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.154 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.154 DEBUG:boto:Params: {} 2023-09-22T21:45:09.154 DEBUG:boto:Token: None 2023-09-22T21:45:09.154 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/ 2023-09-22T21:45:09.154 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo= 2023-09-22T21:45:09.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo=', 'Content-Length': '0'} 2023-09-22T21:45:09.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047e886f846493c0a-00650e0ae5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.164 DEBUG:boto:b'ffzdqu-59obj231000falseobj232023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:09.164 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:45:09.164 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:45:09.164 DEBUG:boto:Method: HEAD 2023-09-22T21:45:09.165 DEBUG:boto:Path: /ffzdqu-59/obj23 2023-09-22T21:45:09.165 DEBUG:boto:Data: 2023-09-22T21:45:09.165 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.165 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.165 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.165 DEBUG:boto:Params: {} 2023-09-22T21:45:09.165 DEBUG:boto:Token: None 2023-09-22T21:45:09.166 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj23 2023-09-22T21:45:09.166 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:z5A08sNS8Gi3UODQ4i8aNSWXEBY= 2023-09-22T21:45:09.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:z5A08sNS8Gi3UODQ4i8aNSWXEBY=', 'Content-Length': '0'} 2023-09-22T21:45:09.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015ba6e9fae803230-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.169 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:45:09.169 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:45:09.169 DEBUG:boto:Method: HEAD 2023-09-22T21:45:09.169 DEBUG:boto:Path: /ffzdqu-59/obj23 2023-09-22T21:45:09.170 DEBUG:boto:Data: 2023-09-22T21:45:09.170 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.170 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.170 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.170 DEBUG:boto:Params: {} 2023-09-22T21:45:09.170 DEBUG:boto:Token: None 2023-09-22T21:45:09.170 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj23 2023-09-22T21:45:09.171 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:z5A08sNS8Gi3UODQ4i8aNSWXEBY= 2023-09-22T21:45:09.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:z5A08sNS8Gi3UODQ4i8aNSWXEBY=', 'Content-Length': '0'} 2023-09-22T21:45:09.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c988ece552d522c-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-22T21:45:09.174 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:45:09.174 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:45:09.174 DEBUG:boto:Method: GET 2023-09-22T21:45:09.174 DEBUG:boto:Path: /ffzdqu-59/obj23 2023-09-22T21:45:09.175 DEBUG:boto:Data: 2023-09-22T21:45:09.175 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.175 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.175 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.175 DEBUG:boto:Params: {} 2023-09-22T21:45:09.175 DEBUG:boto:Token: None 2023-09-22T21:45:09.175 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj23 2023-09-22T21:45:09.176 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tSfxlGbdZPhHFOA5p+FFQOj5poY= 2023-09-22T21:45:09.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tSfxlGbdZPhHFOA5p+FFQOj5poY=', 'Content-Length': '0'} 2023-09-22T21:45:09.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e47a48ca9e9c6687-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.179 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:45:09.179 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:45:09.179 DEBUG:boto:Method: GET 2023-09-22T21:45:09.179 DEBUG:boto:Path: /ffzdqu-59/obj23 2023-09-22T21:45:09.179 DEBUG:boto:Data: 2023-09-22T21:45:09.180 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.180 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.180 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.180 DEBUG:boto:Params: {} 2023-09-22T21:45:09.180 DEBUG:boto:Token: None 2023-09-22T21:45:09.180 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj23 2023-09-22T21:45:09.180 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tSfxlGbdZPhHFOA5p+FFQOj5poY= 2023-09-22T21:45:09.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tSfxlGbdZPhHFOA5p+FFQOj5poY=', 'Content-Length': '0'} 2023-09-22T21:45:09.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000431e3fa6e151e511-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.184 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:09.184 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:09.184 DEBUG:boto:path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:45:09.184 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:45:09.184 DEBUG:boto:Method: GET 2023-09-22T21:45:09.184 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj23 2023-09-22T21:45:09.184 DEBUG:boto:Data: 2023-09-22T21:45:09.184 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.184 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.185 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.185 DEBUG:boto:Params: {} 2023-09-22T21:45:09.185 DEBUG:boto:Token: None 2023-09-22T21:45:09.185 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/ 2023-09-22T21:45:09.185 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo= 2023-09-22T21:45:09.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo=', 'Content-Length': '0'} 2023-09-22T21:45:09.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a99aece44e03e110-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.234 DEBUG:boto:b'ffzdqu-59obj231000falseobj232023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:09.235 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:09.235 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:09.235 DEBUG:boto:path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:45:09.235 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:45:09.236 DEBUG:boto:Method: GET 2023-09-22T21:45:09.236 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj23 2023-09-22T21:45:09.236 DEBUG:boto:Data: 2023-09-22T21:45:09.236 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.236 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.236 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.236 DEBUG:boto:Params: {} 2023-09-22T21:45:09.236 DEBUG:boto:Token: None 2023-09-22T21:45:09.237 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/ 2023-09-22T21:45:09.237 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo= 2023-09-22T21:45:09.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo=', 'Content-Length': '0'} 2023-09-22T21:45:09.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000debb06989673a82e-00650e0ae5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.245 DEBUG:boto:b'ffzdqu-59obj231000falseobj232023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:09.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-22T21:45:09.246 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:45:09.246 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:45:09.246 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.246 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.247 DEBUG:boto:Method: GET 2023-09-22T21:45:09.247 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.247 DEBUG:boto:Data: 2023-09-22T21:45:09.247 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.247 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.247 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.247 DEBUG:boto:Params: {} 2023-09-22T21:45:09.247 DEBUG:boto:Token: None 2023-09-22T21:45:09.247 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.248 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WohhSuXVVp4zeSVmEV0+9MFaEHE= 2023-09-22T21:45:09.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WohhSuXVVp4zeSVmEV0+9MFaEHE=', 'Content-Length': '0'} 2023-09-22T21:45:09.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004e204b711bf488e-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.251 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:45:09.252 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:45:09.252 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.252 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.252 DEBUG:boto:Method: GET 2023-09-22T21:45:09.252 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.252 DEBUG:boto:Data: 2023-09-22T21:45:09.252 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.252 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.252 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.253 DEBUG:boto:Params: {} 2023-09-22T21:45:09.253 DEBUG:boto:Token: None 2023-09-22T21:45:09.253 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.253 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WohhSuXVVp4zeSVmEV0+9MFaEHE= 2023-09-22T21:45:09.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WohhSuXVVp4zeSVmEV0+9MFaEHE=', 'Content-Length': '0'} 2023-09-22T21:45:09.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2ad3bec0bc07ece-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.298 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:45:09.299 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:45:09.299 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.299 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.299 DEBUG:boto:Method: HEAD 2023-09-22T21:45:09.299 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.299 DEBUG:boto:Data: 2023-09-22T21:45:09.299 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.299 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.299 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.300 DEBUG:boto:Params: {} 2023-09-22T21:45:09.300 DEBUG:boto:Token: None 2023-09-22T21:45:09.300 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.300 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Xb8Y9o90lQn2OYP0WbnNJ0pxgCE= 2023-09-22T21:45:09.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Xb8Y9o90lQn2OYP0WbnNJ0pxgCE=', 'Content-Length': '0'} 2023-09-22T21:45:09.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002df32bfc37cd5d45-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.303 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:45:09.304 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:45:09.304 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.304 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.304 DEBUG:boto:Method: HEAD 2023-09-22T21:45:09.304 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.304 DEBUG:boto:Data: 2023-09-22T21:45:09.304 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.304 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.304 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.304 DEBUG:boto:Params: {} 2023-09-22T21:45:09.305 DEBUG:boto:Token: None 2023-09-22T21:45:09.305 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.305 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Xb8Y9o90lQn2OYP0WbnNJ0pxgCE= 2023-09-22T21:45:09.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Xb8Y9o90lQn2OYP0WbnNJ0pxgCE=', 'Content-Length': '0'} 2023-09-22T21:45:09.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9d7f3de9a3a1538-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-22T21:45:09.308 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:45:09.308 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:45:09.308 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.308 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.308 DEBUG:boto:Method: GET 2023-09-22T21:45:09.309 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.309 DEBUG:boto:Data: 2023-09-22T21:45:09.309 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.309 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.309 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.309 DEBUG:boto:Params: {} 2023-09-22T21:45:09.309 DEBUG:boto:Token: None 2023-09-22T21:45:09.309 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.309 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WohhSuXVVp4zeSVmEV0+9MFaEHE= 2023-09-22T21:45:09.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WohhSuXVVp4zeSVmEV0+9MFaEHE=', 'Content-Length': '0'} 2023-09-22T21:45:09.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e4c873b2b54537e-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.312 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:45:09.313 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:45:09.313 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.313 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.313 DEBUG:boto:Method: GET 2023-09-22T21:45:09.313 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.313 DEBUG:boto:Data: 2023-09-22T21:45:09.313 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.313 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.313 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.313 DEBUG:boto:Params: {} 2023-09-22T21:45:09.314 DEBUG:boto:Token: None 2023-09-22T21:45:09.314 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:45:09.314 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WohhSuXVVp4zeSVmEV0+9MFaEHE= 2023-09-22T21:45:09.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WohhSuXVVp4zeSVmEV0+9MFaEHE=', 'Content-Length': '0'} 2023-09-22T21:45:09.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048a935a0975bf305-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.317 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:09.317 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:09.317 DEBUG:boto:path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:45:09.317 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:45:09.317 DEBUG:boto:Method: GET 2023-09-22T21:45:09.317 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj29 2023-09-22T21:45:09.318 DEBUG:boto:Data: 2023-09-22T21:45:09.318 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.318 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.318 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.318 DEBUG:boto:Params: {} 2023-09-22T21:45:09.318 DEBUG:boto:Token: None 2023-09-22T21:45:09.318 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/ 2023-09-22T21:45:09.318 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo= 2023-09-22T21:45:09.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo=', 'Content-Length': '0'} 2023-09-22T21:45:09.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bffc70d8acba7bd8-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.366 DEBUG:boto:b'ffzdqu-59obj291000falseobj292023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:09.367 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:09.367 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:09.367 DEBUG:boto:path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:45:09.367 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:45:09.367 DEBUG:boto:Method: GET 2023-09-22T21:45:09.367 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj29 2023-09-22T21:45:09.367 DEBUG:boto:Data: 2023-09-22T21:45:09.368 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.368 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.368 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.368 DEBUG:boto:Params: {} 2023-09-22T21:45:09.368 DEBUG:boto:Token: None 2023-09-22T21:45:09.368 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/ 2023-09-22T21:45:09.368 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo= 2023-09-22T21:45:09.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo=', 'Content-Length': '0'} 2023-09-22T21:45:09.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042435d33202b544e-00650e0ae5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.376 DEBUG:boto:b'ffzdqu-59obj291000falseobj292023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:09.377 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:45:09.377 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:45:09.377 DEBUG:boto:Method: HEAD 2023-09-22T21:45:09.377 DEBUG:boto:Path: /ffzdqu-59/obj29 2023-09-22T21:45:09.377 DEBUG:boto:Data: 2023-09-22T21:45:09.377 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.378 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.378 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.378 DEBUG:boto:Params: {} 2023-09-22T21:45:09.378 DEBUG:boto:Token: None 2023-09-22T21:45:09.378 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj29 2023-09-22T21:45:09.378 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G7DxqByneIUZoXHeRKx50A2dK2g= 2023-09-22T21:45:09.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G7DxqByneIUZoXHeRKx50A2dK2g=', 'Content-Length': '0'} 2023-09-22T21:45:09.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2f95e803773855b-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.381 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:45:09.381 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:45:09.382 DEBUG:boto:Method: HEAD 2023-09-22T21:45:09.382 DEBUG:boto:Path: /ffzdqu-59/obj29 2023-09-22T21:45:09.382 DEBUG:boto:Data: 2023-09-22T21:45:09.382 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.382 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.382 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.382 DEBUG:boto:Params: {} 2023-09-22T21:45:09.382 DEBUG:boto:Token: None 2023-09-22T21:45:09.382 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj29 2023-09-22T21:45:09.383 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G7DxqByneIUZoXHeRKx50A2dK2g= 2023-09-22T21:45:09.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G7DxqByneIUZoXHeRKx50A2dK2g=', 'Content-Length': '0'} 2023-09-22T21:45:09.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eee6c339a49ea24e-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-22T21:45:09.386 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:45:09.386 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:45:09.386 DEBUG:boto:Method: GET 2023-09-22T21:45:09.386 DEBUG:boto:Path: /ffzdqu-59/obj29 2023-09-22T21:45:09.386 DEBUG:boto:Data: 2023-09-22T21:45:09.387 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.387 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.387 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.387 DEBUG:boto:Params: {} 2023-09-22T21:45:09.387 DEBUG:boto:Token: None 2023-09-22T21:45:09.387 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj29 2023-09-22T21:45:09.387 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sUusNJtwC3Z53mMiQP1CWCjBx5U= 2023-09-22T21:45:09.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sUusNJtwC3Z53mMiQP1CWCjBx5U=', 'Content-Length': '0'} 2023-09-22T21:45:09.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c3581f188f682e4-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.430 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:45:09.430 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:45:09.431 DEBUG:boto:Method: GET 2023-09-22T21:45:09.431 DEBUG:boto:Path: /ffzdqu-59/obj29 2023-09-22T21:45:09.431 DEBUG:boto:Data: 2023-09-22T21:45:09.431 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.431 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.431 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.431 DEBUG:boto:Params: {} 2023-09-22T21:45:09.431 DEBUG:boto:Token: None 2023-09-22T21:45:09.431 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj29 2023-09-22T21:45:09.432 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sUusNJtwC3Z53mMiQP1CWCjBx5U= 2023-09-22T21:45:09.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sUusNJtwC3Z53mMiQP1CWCjBx5U=', 'Content-Length': '0'} 2023-09-22T21:45:09.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fa7a1162de4df30-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.435 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:09.435 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:09.435 DEBUG:boto:path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:45:09.435 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:45:09.435 DEBUG:boto:Method: GET 2023-09-22T21:45:09.435 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj29 2023-09-22T21:45:09.435 DEBUG:boto:Data: 2023-09-22T21:45:09.435 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.436 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.436 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.436 DEBUG:boto:Params: {} 2023-09-22T21:45:09.436 DEBUG:boto:Token: None 2023-09-22T21:45:09.436 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/ 2023-09-22T21:45:09.436 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo= 2023-09-22T21:45:09.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo=', 'Content-Length': '0'} 2023-09-22T21:45:09.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c0f9f687b16dbba-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.442 DEBUG:boto:b'ffzdqu-59obj291000falseobj292023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:09.442 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:09.442 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:09.443 DEBUG:boto:path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:45:09.443 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:45:09.443 DEBUG:boto:Method: GET 2023-09-22T21:45:09.443 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj29 2023-09-22T21:45:09.443 DEBUG:boto:Data: 2023-09-22T21:45:09.443 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.443 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.443 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.444 DEBUG:boto:Params: {} 2023-09-22T21:45:09.444 DEBUG:boto:Token: None 2023-09-22T21:45:09.444 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/ 2023-09-22T21:45:09.444 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo= 2023-09-22T21:45:09.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo=', 'Content-Length': '0'} 2023-09-22T21:45:09.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032901264570f75b6-00650e0ae5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.452 DEBUG:boto:b'ffzdqu-59obj291000falseobj292023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:09.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:45:09.453 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:45:09.453 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:45:09.453 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.453 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.454 DEBUG:boto:Method: GET 2023-09-22T21:45:09.454 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.454 DEBUG:boto:Data: 2023-09-22T21:45:09.454 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.454 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.454 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.454 DEBUG:boto:Params: {} 2023-09-22T21:45:09.455 DEBUG:boto:Token: None 2023-09-22T21:45:09.455 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.455 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7y1GgL3Q80CHNpKmyZ+AlYEivjk= 2023-09-22T21:45:09.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7y1GgL3Q80CHNpKmyZ+AlYEivjk=', 'Content-Length': '0'} 2023-09-22T21:45:09.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007031f2c67dad7c34-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.498 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:45:09.499 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:45:09.499 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.499 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.499 DEBUG:boto:Method: GET 2023-09-22T21:45:09.499 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.499 DEBUG:boto:Data: 2023-09-22T21:45:09.499 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.499 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.499 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.499 DEBUG:boto:Params: {} 2023-09-22T21:45:09.500 DEBUG:boto:Token: None 2023-09-22T21:45:09.500 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.500 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7y1GgL3Q80CHNpKmyZ+AlYEivjk= 2023-09-22T21:45:09.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7y1GgL3Q80CHNpKmyZ+AlYEivjk=', 'Content-Length': '0'} 2023-09-22T21:45:09.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000775dc1887b248da3-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.504 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:45:09.504 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:45:09.504 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.504 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.505 DEBUG:boto:Method: HEAD 2023-09-22T21:45:09.505 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.505 DEBUG:boto:Data: 2023-09-22T21:45:09.505 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.505 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.506 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.506 DEBUG:boto:Params: {} 2023-09-22T21:45:09.506 DEBUG:boto:Token: None 2023-09-22T21:45:09.506 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.507 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KdpeS+q60rTfm4n+0KdUDIu2uhk= 2023-09-22T21:45:09.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KdpeS+q60rTfm4n+0KdUDIu2uhk=', 'Content-Length': '0'} 2023-09-22T21:45:09.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c44c86f5e756a90-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.510 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:45:09.510 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:45:09.511 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.511 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.511 DEBUG:boto:Method: HEAD 2023-09-22T21:45:09.511 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.511 DEBUG:boto:Data: 2023-09-22T21:45:09.511 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.511 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.511 DEBUG:boto:Params: {} 2023-09-22T21:45:09.512 DEBUG:boto:Token: None 2023-09-22T21:45:09.512 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.512 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KdpeS+q60rTfm4n+0KdUDIu2uhk= 2023-09-22T21:45:09.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KdpeS+q60rTfm4n+0KdUDIu2uhk=', 'Content-Length': '0'} 2023-09-22T21:45:09.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000096390fa29f8218b-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:45:09.515 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:45:09.516 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:45:09.516 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.516 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.516 DEBUG:boto:Method: GET 2023-09-22T21:45:09.516 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.517 DEBUG:boto:Data: 2023-09-22T21:45:09.517 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.517 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.517 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.517 DEBUG:boto:Params: {} 2023-09-22T21:45:09.517 DEBUG:boto:Token: None 2023-09-22T21:45:09.517 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.517 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7y1GgL3Q80CHNpKmyZ+AlYEivjk= 2023-09-22T21:45:09.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7y1GgL3Q80CHNpKmyZ+AlYEivjk=', 'Content-Length': '0'} 2023-09-22T21:45:09.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb75331fc90d46c9-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.562 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:45:09.563 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:45:09.563 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.563 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.563 DEBUG:boto:Method: GET 2023-09-22T21:45:09.564 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.564 DEBUG:boto:Data: 2023-09-22T21:45:09.564 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.564 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.564 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.564 DEBUG:boto:Params: {} 2023-09-22T21:45:09.564 DEBUG:boto:Token: None 2023-09-22T21:45:09.564 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:45:09.565 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7y1GgL3Q80CHNpKmyZ+AlYEivjk= 2023-09-22T21:45:09.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7y1GgL3Q80CHNpKmyZ+AlYEivjk=', 'Content-Length': '0'} 2023-09-22T21:45:09.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071151bc17d4fc6ab-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.568 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:09.568 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:09.568 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:45:09.568 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:45:09.568 DEBUG:boto:Method: GET 2023-09-22T21:45:09.568 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:45:09.568 DEBUG:boto:Data: 2023-09-22T21:45:09.569 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.569 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.569 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.569 DEBUG:boto:Params: {} 2023-09-22T21:45:09.569 DEBUG:boto:Token: None 2023-09-22T21:45:09.569 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/ 2023-09-22T21:45:09.569 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo= 2023-09-22T21:45:09.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo=', 'Content-Length': '0'} 2023-09-22T21:45:09.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7852e69d342c423-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.574 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:09.574 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:09.575 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:09.575 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:45:09.575 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:45:09.575 DEBUG:boto:Method: GET 2023-09-22T21:45:09.575 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:45:09.575 DEBUG:boto:Data: 2023-09-22T21:45:09.575 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.575 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.575 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.576 DEBUG:boto:Params: {} 2023-09-22T21:45:09.576 DEBUG:boto:Token: None 2023-09-22T21:45:09.576 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/ 2023-09-22T21:45:09.576 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo= 2023-09-22T21:45:09.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo=', 'Content-Length': '0'} 2023-09-22T21:45:09.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be7be910f817e27d-00650e0ae5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.630 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:09.631 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:45:09.631 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:45:09.631 DEBUG:boto:Method: HEAD 2023-09-22T21:45:09.631 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:45:09.631 DEBUG:boto:Data: 2023-09-22T21:45:09.632 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.632 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.632 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.632 DEBUG:boto:Params: {} 2023-09-22T21:45:09.632 DEBUG:boto:Token: None 2023-09-22T21:45:09.632 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj33 2023-09-22T21:45:09.632 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kxVw4qcHPRUQY4Q+rJv/U/7T7Fw= 2023-09-22T21:45:09.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kxVw4qcHPRUQY4Q+rJv/U/7T7Fw=', 'Content-Length': '0'} 2023-09-22T21:45:09.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074ac3689d9b221cf-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.636 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:45:09.636 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:45:09.636 DEBUG:boto:Method: HEAD 2023-09-22T21:45:09.636 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:45:09.636 DEBUG:boto:Data: 2023-09-22T21:45:09.636 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.636 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.636 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.636 DEBUG:boto:Params: {} 2023-09-22T21:45:09.637 DEBUG:boto:Token: None 2023-09-22T21:45:09.637 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj33 2023-09-22T21:45:09.637 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kxVw4qcHPRUQY4Q+rJv/U/7T7Fw= 2023-09-22T21:45:09.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kxVw4qcHPRUQY4Q+rJv/U/7T7Fw=', 'Content-Length': '0'} 2023-09-22T21:45:09.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b3a717ba6f95a8d-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:45:09.640 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:45:09.640 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:45:09.640 DEBUG:boto:Method: GET 2023-09-22T21:45:09.640 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:45:09.640 DEBUG:boto:Data: 2023-09-22T21:45:09.640 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.641 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.641 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.641 DEBUG:boto:Params: {} 2023-09-22T21:45:09.641 DEBUG:boto:Token: None 2023-09-22T21:45:09.641 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj33 2023-09-22T21:45:09.641 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7P6lsNFKxAiWOvCpXcMeCOmdjX4= 2023-09-22T21:45:09.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7P6lsNFKxAiWOvCpXcMeCOmdjX4=', 'Content-Length': '0'} 2023-09-22T21:45:09.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073f3c9574481b8f9-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.686 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:45:09.687 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:45:09.687 DEBUG:boto:Method: GET 2023-09-22T21:45:09.687 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:45:09.687 DEBUG:boto:Data: 2023-09-22T21:45:09.687 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.687 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.687 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.687 DEBUG:boto:Params: {} 2023-09-22T21:45:09.688 DEBUG:boto:Token: None 2023-09-22T21:45:09.688 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj33 2023-09-22T21:45:09.688 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7P6lsNFKxAiWOvCpXcMeCOmdjX4= 2023-09-22T21:45:09.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7P6lsNFKxAiWOvCpXcMeCOmdjX4=', 'Content-Length': '0'} 2023-09-22T21:45:09.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be4a91a22a1710da-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.691 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:09.691 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:09.691 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:45:09.692 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:45:09.692 DEBUG:boto:Method: GET 2023-09-22T21:45:09.692 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:45:09.692 DEBUG:boto:Data: 2023-09-22T21:45:09.692 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.692 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.692 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.692 DEBUG:boto:Params: {} 2023-09-22T21:45:09.693 DEBUG:boto:Token: None 2023-09-22T21:45:09.693 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/ 2023-09-22T21:45:09.693 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo= 2023-09-22T21:45:09.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo=', 'Content-Length': '0'} 2023-09-22T21:45:09.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000145de91cf16f8fec-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.738 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:09.739 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:09.739 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:09.739 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:45:09.739 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:45:09.740 DEBUG:boto:Method: GET 2023-09-22T21:45:09.740 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:45:09.740 DEBUG:boto:Data: 2023-09-22T21:45:09.740 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.740 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.740 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.740 DEBUG:boto:Params: {} 2023-09-22T21:45:09.741 DEBUG:boto:Token: None 2023-09-22T21:45:09.741 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/ 2023-09-22T21:45:09.741 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo= 2023-09-22T21:45:09.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo=', 'Content-Length': '0'} 2023-09-22T21:45:09.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acf55cbd0d64443b-00650e0ae5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.750 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:09.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:45:09.750 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:45:09.751 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:45:09.751 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.751 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.751 DEBUG:boto:Method: GET 2023-09-22T21:45:09.751 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.752 DEBUG:boto:Data: 2023-09-22T21:45:09.752 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.752 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.752 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.752 DEBUG:boto:Params: {} 2023-09-22T21:45:09.753 DEBUG:boto:Token: None 2023-09-22T21:45:09.753 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.753 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IzeQNE3TANZ6zA+K8KG2K3wrO5s= 2023-09-22T21:45:09.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IzeQNE3TANZ6zA+K8KG2K3wrO5s=', 'Content-Length': '0'} 2023-09-22T21:45:09.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad54433f38c76c30-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.756 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:45:09.756 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:45:09.756 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.756 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.757 DEBUG:boto:Method: GET 2023-09-22T21:45:09.757 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.757 DEBUG:boto:Data: 2023-09-22T21:45:09.757 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.757 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.757 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.757 DEBUG:boto:Params: {} 2023-09-22T21:45:09.758 DEBUG:boto:Token: None 2023-09-22T21:45:09.758 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.758 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IzeQNE3TANZ6zA+K8KG2K3wrO5s= 2023-09-22T21:45:09.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IzeQNE3TANZ6zA+K8KG2K3wrO5s=', 'Content-Length': '0'} 2023-09-22T21:45:09.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000992c3aa6e8ced1ee-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.761 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:45:09.761 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:45:09.762 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.762 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.762 DEBUG:boto:Method: HEAD 2023-09-22T21:45:09.762 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.762 DEBUG:boto:Data: 2023-09-22T21:45:09.762 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.762 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.763 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.763 DEBUG:boto:Params: {} 2023-09-22T21:45:09.763 DEBUG:boto:Token: None 2023-09-22T21:45:09.763 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.763 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ua/XoZBJUdt1iav6x3j9aYGtbQA= 2023-09-22T21:45:09.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ua/XoZBJUdt1iav6x3j9aYGtbQA=', 'Content-Length': '0'} 2023-09-22T21:45:09.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047d4f7c6be5b8327-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.766 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:45:09.767 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:45:09.767 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.767 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.767 DEBUG:boto:Method: HEAD 2023-09-22T21:45:09.767 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.767 DEBUG:boto:Data: 2023-09-22T21:45:09.767 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.767 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.768 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.768 DEBUG:boto:Params: {} 2023-09-22T21:45:09.768 DEBUG:boto:Token: None 2023-09-22T21:45:09.768 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.768 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ua/XoZBJUdt1iav6x3j9aYGtbQA= 2023-09-22T21:45:09.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ua/XoZBJUdt1iav6x3j9aYGtbQA=', 'Content-Length': '0'} 2023-09-22T21:45:09.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059c6820307ea808c-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:45:09.771 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:45:09.771 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:45:09.772 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.772 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.772 DEBUG:boto:Method: GET 2023-09-22T21:45:09.772 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.772 DEBUG:boto:Data: 2023-09-22T21:45:09.772 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.772 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.773 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.773 DEBUG:boto:Params: {} 2023-09-22T21:45:09.773 DEBUG:boto:Token: None 2023-09-22T21:45:09.773 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.773 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IzeQNE3TANZ6zA+K8KG2K3wrO5s= 2023-09-22T21:45:09.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IzeQNE3TANZ6zA+K8KG2K3wrO5s=', 'Content-Length': '0'} 2023-09-22T21:45:09.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088a72afc0b53b88e-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.818 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:45:09.819 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:45:09.819 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.819 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.819 DEBUG:boto:Method: GET 2023-09-22T21:45:09.819 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.819 DEBUG:boto:Data: 2023-09-22T21:45:09.820 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.820 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.820 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.820 DEBUG:boto:Params: {} 2023-09-22T21:45:09.820 DEBUG:boto:Token: None 2023-09-22T21:45:09.821 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:45:09.821 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IzeQNE3TANZ6zA+K8KG2K3wrO5s= 2023-09-22T21:45:09.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IzeQNE3TANZ6zA+K8KG2K3wrO5s=', 'Content-Length': '0'} 2023-09-22T21:45:09.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074dc564087d5b041-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.825 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:09.825 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:09.825 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:45:09.825 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:45:09.825 DEBUG:boto:Method: GET 2023-09-22T21:45:09.825 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:45:09.826 DEBUG:boto:Data: 2023-09-22T21:45:09.826 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.826 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.826 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.826 DEBUG:boto:Params: {} 2023-09-22T21:45:09.826 DEBUG:boto:Token: None 2023-09-22T21:45:09.827 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/ 2023-09-22T21:45:09.827 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo= 2023-09-22T21:45:09.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo=', 'Content-Length': '0'} 2023-09-22T21:45:09.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a490086a45f362a0-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.874 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:09.875 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:09.875 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:09.875 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:45:09.876 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:45:09.876 DEBUG:boto:Method: GET 2023-09-22T21:45:09.876 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:45:09.876 DEBUG:boto:Data: 2023-09-22T21:45:09.877 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.877 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.877 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.877 DEBUG:boto:Params: {} 2023-09-22T21:45:09.877 DEBUG:boto:Token: None 2023-09-22T21:45:09.878 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/ 2023-09-22T21:45:09.878 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo= 2023-09-22T21:45:09.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo=', 'Content-Length': '0'} 2023-09-22T21:45:09.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080e4d09558e2d9d2-00650e0ae5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.887 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:09.887 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:45:09.888 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:45:09.888 DEBUG:boto:Method: HEAD 2023-09-22T21:45:09.888 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:45:09.888 DEBUG:boto:Data: 2023-09-22T21:45:09.888 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.889 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.889 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.889 DEBUG:boto:Params: {} 2023-09-22T21:45:09.889 DEBUG:boto:Token: None 2023-09-22T21:45:09.889 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj34 2023-09-22T21:45:09.890 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Wig3IeEOjezkg1KlHhmV4/2Tr3E= 2023-09-22T21:45:09.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Wig3IeEOjezkg1KlHhmV4/2Tr3E=', 'Content-Length': '0'} 2023-09-22T21:45:09.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9685a307e54b57b-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.895 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:45:09.895 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:45:09.896 DEBUG:boto:Method: HEAD 2023-09-22T21:45:09.896 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:45:09.896 DEBUG:boto:Data: 2023-09-22T21:45:09.896 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.896 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.897 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.897 DEBUG:boto:Params: {} 2023-09-22T21:45:09.897 DEBUG:boto:Token: None 2023-09-22T21:45:09.897 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj34 2023-09-22T21:45:09.898 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Wig3IeEOjezkg1KlHhmV4/2Tr3E= 2023-09-22T21:45:09.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Wig3IeEOjezkg1KlHhmV4/2Tr3E=', 'Content-Length': '0'} 2023-09-22T21:45:09.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6de6efd0f1a447c-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:45:09.902 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:45:09.902 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:45:09.902 DEBUG:boto:Method: GET 2023-09-22T21:45:09.903 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:45:09.903 DEBUG:boto:Data: 2023-09-22T21:45:09.903 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.903 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.903 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.904 DEBUG:boto:Params: {} 2023-09-22T21:45:09.904 DEBUG:boto:Token: None 2023-09-22T21:45:09.904 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj34 2023-09-22T21:45:09.904 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jcnzHVHpU+mGztvBymmAAcFc/Nw= 2023-09-22T21:45:09.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jcnzHVHpU+mGztvBymmAAcFc/Nw=', 'Content-Length': '0'} 2023-09-22T21:45:09.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c37576071b5d0c00-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.950 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:45:09.951 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:45:09.951 DEBUG:boto:Method: GET 2023-09-22T21:45:09.951 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:45:09.951 DEBUG:boto:Data: 2023-09-22T21:45:09.952 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.952 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.952 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.952 DEBUG:boto:Params: {} 2023-09-22T21:45:09.953 DEBUG:boto:Token: None 2023-09-22T21:45:09.953 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj34 2023-09-22T21:45:09.953 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jcnzHVHpU+mGztvBymmAAcFc/Nw= 2023-09-22T21:45:09.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jcnzHVHpU+mGztvBymmAAcFc/Nw=', 'Content-Length': '0'} 2023-09-22T21:45:09.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000629dfc4359c42a0-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.957 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:09.957 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:09.957 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:45:09.957 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:45:09.957 DEBUG:boto:Method: GET 2023-09-22T21:45:09.958 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:45:09.958 DEBUG:boto:Data: 2023-09-22T21:45:09.958 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.958 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.958 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.958 DEBUG:boto:Params: {} 2023-09-22T21:45:09.958 DEBUG:boto:Token: None 2023-09-22T21:45:09.959 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/ 2023-09-22T21:45:09.959 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo= 2023-09-22T21:45:09.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo=', 'Content-Length': '0'} 2023-09-22T21:45:09.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6a31f00040762a6-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.964 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:09.964 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:09.964 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:09.964 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:45:09.964 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:45:09.965 DEBUG:boto:Method: GET 2023-09-22T21:45:09.965 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:45:09.965 DEBUG:boto:Data: 2023-09-22T21:45:09.965 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.965 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.965 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.965 DEBUG:boto:Params: {} 2023-09-22T21:45:09.965 DEBUG:boto:Token: None 2023-09-22T21:45:09.965 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/ 2023-09-22T21:45:09.966 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo= 2023-09-22T21:45:09.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LkZ8yD+WK+p+U9tg13Cp3S0oSpo=', 'Content-Length': '0'} 2023-09-22T21:45:09.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b4bd44b118e0bea-00650e0ae5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.974 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:09.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T21:45:09.975 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:45:09.975 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:45:09.975 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:09.975 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:09.975 DEBUG:boto:Method: GET 2023-09-22T21:45:09.976 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:09.976 DEBUG:boto:Data: 2023-09-22T21:45:09.976 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.976 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.976 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.976 DEBUG:boto:Params: {} 2023-09-22T21:45:09.976 DEBUG:boto:Token: None 2023-09-22T21:45:09.976 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:09.977 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0i18AnPNmkNb+z6wi37xFI1zbT0= 2023-09-22T21:45:09.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0i18AnPNmkNb+z6wi37xFI1zbT0=', 'Content-Length': '0'} 2023-09-22T21:45:09.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000095e0b85d515f6f-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.979 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:45:09.980 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:45:09.980 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:09.980 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:09.980 DEBUG:boto:Method: GET 2023-09-22T21:45:09.980 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:09.980 DEBUG:boto:Data: 2023-09-22T21:45:09.980 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.980 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.980 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.981 DEBUG:boto:Params: {} 2023-09-22T21:45:09.981 DEBUG:boto:Token: None 2023-09-22T21:45:09.981 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:09.981 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0i18AnPNmkNb+z6wi37xFI1zbT0= 2023-09-22T21:45:09.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0i18AnPNmkNb+z6wi37xFI1zbT0=', 'Content-Length': '0'} 2023-09-22T21:45:09.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045001df7815f9d6a-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.984 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:45:09.984 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:45:09.984 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:09.985 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:09.985 DEBUG:boto:Method: HEAD 2023-09-22T21:45:09.985 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:09.985 DEBUG:boto:Data: 2023-09-22T21:45:09.985 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.985 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.985 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.985 DEBUG:boto:Params: {} 2023-09-22T21:45:09.985 DEBUG:boto:Token: None 2023-09-22T21:45:09.986 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:09.986 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0KsF/PnC0wYi9Imd0pbNis+C5Ro= 2023-09-22T21:45:09.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0KsF/PnC0wYi9Imd0pbNis+C5Ro=', 'Content-Length': '0'} 2023-09-22T21:45:09.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5dd9ef5da6a4ac9-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.990 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:45:09.990 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:45:09.990 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:09.991 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:09.991 DEBUG:boto:Method: HEAD 2023-09-22T21:45:09.991 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:09.991 DEBUG:boto:Data: 2023-09-22T21:45:09.991 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.991 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:09.991 DEBUG:boto:Port: 8000 2023-09-22T21:45:09.991 DEBUG:boto:Params: {} 2023-09-22T21:45:09.991 DEBUG:boto:Token: None 2023-09-22T21:45:09.992 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:09.992 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0KsF/PnC0wYi9Imd0pbNis+C5Ro= 2023-09-22T21:45:09.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0KsF/PnC0wYi9Imd0pbNis+C5Ro=', 'Content-Length': '0'} 2023-09-22T21:45:09.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000471b4eed0749670a-00650e0ae5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:09.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T21:45:09.995 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:45:09.995 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:45:09.995 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:09.995 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:09.996 DEBUG:boto:Method: GET 2023-09-22T21:45:09.996 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:09.996 DEBUG:boto:Data: 2023-09-22T21:45:09.996 DEBUG:boto:Headers: {} 2023-09-22T21:45:09.996 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:09.996 DEBUG:boto:Port: 8001 2023-09-22T21:45:09.996 DEBUG:boto:Params: {} 2023-09-22T21:45:09.996 DEBUG:boto:Token: None 2023-09-22T21:45:09.996 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:09 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:09.997 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0i18AnPNmkNb+z6wi37xFI1zbT0= 2023-09-22T21:45:09.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0i18AnPNmkNb+z6wi37xFI1zbT0=', 'Content-Length': '0'} 2023-09-22T21:45:09.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a987995d64c9cf1-00650e0ae5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.042 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:45:10.043 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:45:10.043 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:10.043 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:10.043 DEBUG:boto:Method: GET 2023-09-22T21:45:10.043 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:10.043 DEBUG:boto:Data: 2023-09-22T21:45:10.043 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.043 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.044 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.044 DEBUG:boto:Params: {} 2023-09-22T21:45:10.044 DEBUG:boto:Token: None 2023-09-22T21:45:10.044 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:45:10.044 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YjxKzIDE+AIn1CCqmuA7E4QY69c= 2023-09-22T21:45:10.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YjxKzIDE+AIn1CCqmuA7E4QY69c=', 'Content-Length': '0'} 2023-09-22T21:45:10.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000702f4f4b234207fe-00650e0ae6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.047 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:10.048 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:10.048 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:45:10.048 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:45:10.048 DEBUG:boto:Method: GET 2023-09-22T21:45:10.048 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:45:10.048 DEBUG:boto:Data: 2023-09-22T21:45:10.048 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.048 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.048 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.049 DEBUG:boto:Params: {} 2023-09-22T21:45:10.049 DEBUG:boto:Token: None 2023-09-22T21:45:10.049 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/ 2023-09-22T21:45:10.049 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk= 2023-09-22T21:45:10.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk=', 'Content-Length': '0'} 2023-09-22T21:45:10.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d4d4843a453265e-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.054 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:10.054 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:10.055 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:10.055 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:45:10.055 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:45:10.055 DEBUG:boto:Method: GET 2023-09-22T21:45:10.055 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:45:10.055 DEBUG:boto:Data: 2023-09-22T21:45:10.055 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.056 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.056 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.056 DEBUG:boto:Params: {} 2023-09-22T21:45:10.056 DEBUG:boto:Token: None 2023-09-22T21:45:10.057 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/ 2023-09-22T21:45:10.057 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk= 2023-09-22T21:45:10.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk=', 'Content-Length': '0'} 2023-09-22T21:45:10.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d6333bbc2289553-00650e0ae6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.066 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:10.066 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:45:10.066 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:45:10.066 DEBUG:boto:Method: HEAD 2023-09-22T21:45:10.066 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:45:10.067 DEBUG:boto:Data: 2023-09-22T21:45:10.067 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.067 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.067 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.067 DEBUG:boto:Params: {} 2023-09-22T21:45:10.067 DEBUG:boto:Token: None 2023-09-22T21:45:10.067 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj38 2023-09-22T21:45:10.067 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZxnYDWIqtgYKuOX1Lv083/ZSD5U= 2023-09-22T21:45:10.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZxnYDWIqtgYKuOX1Lv083/ZSD5U=', 'Content-Length': '0'} 2023-09-22T21:45:10.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a1d06ada6d0bdcc-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.070 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:45:10.071 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:45:10.071 DEBUG:boto:Method: HEAD 2023-09-22T21:45:10.071 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:45:10.071 DEBUG:boto:Data: 2023-09-22T21:45:10.071 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.071 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.071 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.071 DEBUG:boto:Params: {} 2023-09-22T21:45:10.072 DEBUG:boto:Token: None 2023-09-22T21:45:10.072 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj38 2023-09-22T21:45:10.072 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZxnYDWIqtgYKuOX1Lv083/ZSD5U= 2023-09-22T21:45:10.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZxnYDWIqtgYKuOX1Lv083/ZSD5U=', 'Content-Length': '0'} 2023-09-22T21:45:10.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a651fa0d06a9a68c-00650e0ae6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T21:45:10.075 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:45:10.075 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:45:10.076 DEBUG:boto:Method: GET 2023-09-22T21:45:10.076 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:45:10.076 DEBUG:boto:Data: 2023-09-22T21:45:10.076 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.076 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.076 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.076 DEBUG:boto:Params: {} 2023-09-22T21:45:10.076 DEBUG:boto:Token: None 2023-09-22T21:45:10.077 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj38 2023-09-22T21:45:10.077 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C0/kTTYklh2t4Cg4WVjWuxdV7qQ= 2023-09-22T21:45:10.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C0/kTTYklh2t4Cg4WVjWuxdV7qQ=', 'Content-Length': '0'} 2023-09-22T21:45:10.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002531f9ca53453bd6-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.122 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:45:10.123 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:45:10.123 DEBUG:boto:Method: GET 2023-09-22T21:45:10.123 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:45:10.123 DEBUG:boto:Data: 2023-09-22T21:45:10.123 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.123 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.123 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.123 DEBUG:boto:Params: {} 2023-09-22T21:45:10.123 DEBUG:boto:Token: None 2023-09-22T21:45:10.124 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj38 2023-09-22T21:45:10.124 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C0/kTTYklh2t4Cg4WVjWuxdV7qQ= 2023-09-22T21:45:10.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C0/kTTYklh2t4Cg4WVjWuxdV7qQ=', 'Content-Length': '0'} 2023-09-22T21:45:10.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0159c77f5546b16-00650e0ae6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.127 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:10.127 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:10.127 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:45:10.127 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:45:10.127 DEBUG:boto:Method: GET 2023-09-22T21:45:10.127 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:45:10.128 DEBUG:boto:Data: 2023-09-22T21:45:10.128 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.128 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.128 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.128 DEBUG:boto:Params: {} 2023-09-22T21:45:10.128 DEBUG:boto:Token: None 2023-09-22T21:45:10.128 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/ 2023-09-22T21:45:10.128 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk= 2023-09-22T21:45:10.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk=', 'Content-Length': '0'} 2023-09-22T21:45:10.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f7c70ccc7b1e434-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.174 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:10.175 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:10.175 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:10.175 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:45:10.175 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:45:10.175 DEBUG:boto:Method: GET 2023-09-22T21:45:10.175 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:45:10.175 DEBUG:boto:Data: 2023-09-22T21:45:10.175 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.176 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.176 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.176 DEBUG:boto:Params: {} 2023-09-22T21:45:10.176 DEBUG:boto:Token: None 2023-09-22T21:45:10.176 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/ 2023-09-22T21:45:10.176 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk= 2023-09-22T21:45:10.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk=', 'Content-Length': '0'} 2023-09-22T21:45:10.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d48cb44ef0f3f658-00650e0ae6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.185 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:10.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:45:10.186 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:45:10.186 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:45:10.186 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.186 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.186 DEBUG:boto:Method: GET 2023-09-22T21:45:10.186 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.186 DEBUG:boto:Data: 2023-09-22T21:45:10.186 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.187 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.187 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.187 DEBUG:boto:Params: {} 2023-09-22T21:45:10.187 DEBUG:boto:Token: None 2023-09-22T21:45:10.187 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.187 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D+M1SBqvKud9CPC7gDkRSVFIHco= 2023-09-22T21:45:10.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D+M1SBqvKud9CPC7gDkRSVFIHco=', 'Content-Length': '0'} 2023-09-22T21:45:10.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085cb6424c10bf4d4-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.190 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:45:10.190 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:45:10.191 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.191 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.191 DEBUG:boto:Method: GET 2023-09-22T21:45:10.191 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.191 DEBUG:boto:Data: 2023-09-22T21:45:10.191 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.191 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.191 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.191 DEBUG:boto:Params: {} 2023-09-22T21:45:10.192 DEBUG:boto:Token: None 2023-09-22T21:45:10.192 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.192 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D+M1SBqvKud9CPC7gDkRSVFIHco= 2023-09-22T21:45:10.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D+M1SBqvKud9CPC7gDkRSVFIHco=', 'Content-Length': '0'} 2023-09-22T21:45:10.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b5269a491e214b9-00650e0ae6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.238 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:45:10.241 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:45:10.241 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.241 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.241 DEBUG:boto:Method: HEAD 2023-09-22T21:45:10.241 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.241 DEBUG:boto:Data: 2023-09-22T21:45:10.242 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.242 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.242 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.242 DEBUG:boto:Params: {} 2023-09-22T21:45:10.242 DEBUG:boto:Token: None 2023-09-22T21:45:10.242 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.243 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/YYjWyINVpfRWQfQVdQQJjCfgCs= 2023-09-22T21:45:10.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/YYjWyINVpfRWQfQVdQQJjCfgCs=', 'Content-Length': '0'} 2023-09-22T21:45:10.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7dd039b1ebe152d-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.246 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:45:10.246 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:45:10.246 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.246 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.246 DEBUG:boto:Method: HEAD 2023-09-22T21:45:10.247 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.247 DEBUG:boto:Data: 2023-09-22T21:45:10.247 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.247 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.247 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.247 DEBUG:boto:Params: {} 2023-09-22T21:45:10.248 DEBUG:boto:Token: None 2023-09-22T21:45:10.248 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.248 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/YYjWyINVpfRWQfQVdQQJjCfgCs= 2023-09-22T21:45:10.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/YYjWyINVpfRWQfQVdQQJjCfgCs=', 'Content-Length': '0'} 2023-09-22T21:45:10.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b1a797e61d33784-00650e0ae6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:45:10.251 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:45:10.251 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:45:10.251 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.252 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.252 DEBUG:boto:Method: GET 2023-09-22T21:45:10.252 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.252 DEBUG:boto:Data: 2023-09-22T21:45:10.252 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.252 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.252 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.252 DEBUG:boto:Params: {} 2023-09-22T21:45:10.253 DEBUG:boto:Token: None 2023-09-22T21:45:10.253 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.253 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D+M1SBqvKud9CPC7gDkRSVFIHco= 2023-09-22T21:45:10.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D+M1SBqvKud9CPC7gDkRSVFIHco=', 'Content-Length': '0'} 2023-09-22T21:45:10.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002eaf5559375be9e8-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.302 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:45:10.302 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:45:10.303 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.303 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.303 DEBUG:boto:Method: GET 2023-09-22T21:45:10.303 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.303 DEBUG:boto:Data: 2023-09-22T21:45:10.303 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.303 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.303 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.303 DEBUG:boto:Params: {} 2023-09-22T21:45:10.304 DEBUG:boto:Token: None 2023-09-22T21:45:10.304 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:45:10.304 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D+M1SBqvKud9CPC7gDkRSVFIHco= 2023-09-22T21:45:10.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D+M1SBqvKud9CPC7gDkRSVFIHco=', 'Content-Length': '0'} 2023-09-22T21:45:10.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d72bc23d970f0b79-00650e0ae6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.307 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:10.307 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:10.307 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:45:10.308 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:45:10.308 DEBUG:boto:Method: GET 2023-09-22T21:45:10.308 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:45:10.308 DEBUG:boto:Data: 2023-09-22T21:45:10.308 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.308 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.308 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.308 DEBUG:boto:Params: {} 2023-09-22T21:45:10.308 DEBUG:boto:Token: None 2023-09-22T21:45:10.309 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/ 2023-09-22T21:45:10.309 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk= 2023-09-22T21:45:10.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk=', 'Content-Length': '0'} 2023-09-22T21:45:10.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000814806b53a42df78-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.313 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:10.314 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:10.314 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:10.314 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:45:10.314 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:45:10.314 DEBUG:boto:Method: GET 2023-09-22T21:45:10.315 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:45:10.315 DEBUG:boto:Data: 2023-09-22T21:45:10.315 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.315 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.315 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.315 DEBUG:boto:Params: {} 2023-09-22T21:45:10.315 DEBUG:boto:Token: None 2023-09-22T21:45:10.315 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/ 2023-09-22T21:45:10.315 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk= 2023-09-22T21:45:10.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk=', 'Content-Length': '0'} 2023-09-22T21:45:10.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067e96bf7fbbad084-00650e0ae6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.324 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:10.324 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:45:10.324 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:45:10.325 DEBUG:boto:Method: HEAD 2023-09-22T21:45:10.325 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:45:10.325 DEBUG:boto:Data: 2023-09-22T21:45:10.325 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.325 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.325 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.325 DEBUG:boto:Params: {} 2023-09-22T21:45:10.325 DEBUG:boto:Token: None 2023-09-22T21:45:10.325 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj44 2023-09-22T21:45:10.326 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hXcTOxguZ38KIWbVOxGIgMWmEc4= 2023-09-22T21:45:10.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hXcTOxguZ38KIWbVOxGIgMWmEc4=', 'Content-Length': '0'} 2023-09-22T21:45:10.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9abe3b1813767a1-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.329 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:45:10.329 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:45:10.329 DEBUG:boto:Method: HEAD 2023-09-22T21:45:10.329 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:45:10.329 DEBUG:boto:Data: 2023-09-22T21:45:10.329 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.329 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.330 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.330 DEBUG:boto:Params: {} 2023-09-22T21:45:10.330 DEBUG:boto:Token: None 2023-09-22T21:45:10.330 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj44 2023-09-22T21:45:10.331 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hXcTOxguZ38KIWbVOxGIgMWmEc4= 2023-09-22T21:45:10.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hXcTOxguZ38KIWbVOxGIgMWmEc4=', 'Content-Length': '0'} 2023-09-22T21:45:10.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f24b49a890784edf-00650e0ae6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:45:10.334 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:45:10.334 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:45:10.334 DEBUG:boto:Method: GET 2023-09-22T21:45:10.334 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:45:10.334 DEBUG:boto:Data: 2023-09-22T21:45:10.334 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.335 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.335 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.335 DEBUG:boto:Params: {} 2023-09-22T21:45:10.335 DEBUG:boto:Token: None 2023-09-22T21:45:10.335 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj44 2023-09-22T21:45:10.335 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XM6DOTSvpakkOkj/CrT+r9k/I6c= 2023-09-22T21:45:10.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XM6DOTSvpakkOkj/CrT+r9k/I6c=', 'Content-Length': '0'} 2023-09-22T21:45:10.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec0d0dcee9daaf19-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.382 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:45:10.383 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:45:10.383 DEBUG:boto:Method: GET 2023-09-22T21:45:10.383 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:45:10.383 DEBUG:boto:Data: 2023-09-22T21:45:10.383 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.383 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.383 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.383 DEBUG:boto:Params: {} 2023-09-22T21:45:10.384 DEBUG:boto:Token: None 2023-09-22T21:45:10.384 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj44 2023-09-22T21:45:10.384 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XM6DOTSvpakkOkj/CrT+r9k/I6c= 2023-09-22T21:45:10.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XM6DOTSvpakkOkj/CrT+r9k/I6c=', 'Content-Length': '0'} 2023-09-22T21:45:10.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a6c61719933eae5-00650e0ae6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.388 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:10.388 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:10.388 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:45:10.388 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:45:10.388 DEBUG:boto:Method: GET 2023-09-22T21:45:10.388 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:45:10.388 DEBUG:boto:Data: 2023-09-22T21:45:10.388 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.389 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.389 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.389 DEBUG:boto:Params: {} 2023-09-22T21:45:10.389 DEBUG:boto:Token: None 2023-09-22T21:45:10.389 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/ 2023-09-22T21:45:10.389 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk= 2023-09-22T21:45:10.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk=', 'Content-Length': '0'} 2023-09-22T21:45:10.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054b813bf2f52115a-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.434 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:10.435 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:10.435 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:10.435 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:45:10.435 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:45:10.435 DEBUG:boto:Method: GET 2023-09-22T21:45:10.435 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:45:10.436 DEBUG:boto:Data: 2023-09-22T21:45:10.436 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.436 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.436 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.436 DEBUG:boto:Params: {} 2023-09-22T21:45:10.436 DEBUG:boto:Token: None 2023-09-22T21:45:10.436 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/ 2023-09-22T21:45:10.436 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk= 2023-09-22T21:45:10.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk=', 'Content-Length': '0'} 2023-09-22T21:45:10.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5b89f4320e0adaa-00650e0ae6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.445 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:10.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-22T21:45:10.446 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:45:10.446 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:45:10.446 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.446 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.446 DEBUG:boto:Method: GET 2023-09-22T21:45:10.446 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.446 DEBUG:boto:Data: 2023-09-22T21:45:10.447 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.447 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.447 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.447 DEBUG:boto:Params: {} 2023-09-22T21:45:10.447 DEBUG:boto:Token: None 2023-09-22T21:45:10.447 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.447 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5N57Hg5MpZ2z7kTQJ3TIICwexAc= 2023-09-22T21:45:10.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5N57Hg5MpZ2z7kTQJ3TIICwexAc=', 'Content-Length': '0'} 2023-09-22T21:45:10.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaacd34ac617f5e2-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.450 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:45:10.451 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:45:10.451 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.451 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.451 DEBUG:boto:Method: GET 2023-09-22T21:45:10.451 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.451 DEBUG:boto:Data: 2023-09-22T21:45:10.451 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.451 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.451 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.451 DEBUG:boto:Params: {} 2023-09-22T21:45:10.452 DEBUG:boto:Token: None 2023-09-22T21:45:10.452 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.452 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5N57Hg5MpZ2z7kTQJ3TIICwexAc= 2023-09-22T21:45:10.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5N57Hg5MpZ2z7kTQJ3TIICwexAc=', 'Content-Length': '0'} 2023-09-22T21:45:10.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000466341d3a361906e-00650e0ae6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.455 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:45:10.455 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:45:10.455 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.455 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.455 DEBUG:boto:Method: HEAD 2023-09-22T21:45:10.455 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.456 DEBUG:boto:Data: 2023-09-22T21:45:10.456 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.456 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.456 DEBUG:boto:Params: {} 2023-09-22T21:45:10.456 DEBUG:boto:Token: None 2023-09-22T21:45:10.456 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.457 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VxDBzPsZ9V9p+vASKjNROC92+sg= 2023-09-22T21:45:10.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VxDBzPsZ9V9p+vASKjNROC92+sg=', 'Content-Length': '0'} 2023-09-22T21:45:10.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002781dff061233731-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.459 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:45:10.460 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:45:10.460 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.460 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.460 DEBUG:boto:Method: HEAD 2023-09-22T21:45:10.460 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.460 DEBUG:boto:Data: 2023-09-22T21:45:10.460 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.460 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.461 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.461 DEBUG:boto:Params: {} 2023-09-22T21:45:10.461 DEBUG:boto:Token: None 2023-09-22T21:45:10.461 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.461 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VxDBzPsZ9V9p+vASKjNROC92+sg= 2023-09-22T21:45:10.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VxDBzPsZ9V9p+vASKjNROC92+sg=', 'Content-Length': '0'} 2023-09-22T21:45:10.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f8b52be499fc559-00650e0ae6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-22T21:45:10.464 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:45:10.464 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:45:10.464 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.464 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.465 DEBUG:boto:Method: GET 2023-09-22T21:45:10.465 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.465 DEBUG:boto:Data: 2023-09-22T21:45:10.465 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.465 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.465 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.465 DEBUG:boto:Params: {} 2023-09-22T21:45:10.465 DEBUG:boto:Token: None 2023-09-22T21:45:10.465 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.466 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5N57Hg5MpZ2z7kTQJ3TIICwexAc= 2023-09-22T21:45:10.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5N57Hg5MpZ2z7kTQJ3TIICwexAc=', 'Content-Length': '0'} 2023-09-22T21:45:10.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000824951742de238b6-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.510 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:45:10.511 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:45:10.511 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.511 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.511 DEBUG:boto:Method: GET 2023-09-22T21:45:10.511 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.511 DEBUG:boto:Data: 2023-09-22T21:45:10.511 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.511 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.512 DEBUG:boto:Params: {} 2023-09-22T21:45:10.512 DEBUG:boto:Token: None 2023-09-22T21:45:10.512 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:45:10.512 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5N57Hg5MpZ2z7kTQJ3TIICwexAc= 2023-09-22T21:45:10.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5N57Hg5MpZ2z7kTQJ3TIICwexAc=', 'Content-Length': '0'} 2023-09-22T21:45:10.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d4baf0ef2760981-00650e0ae6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.515 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:10.516 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:10.516 DEBUG:boto:path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:45:10.516 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:45:10.516 DEBUG:boto:Method: GET 2023-09-22T21:45:10.516 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj46 2023-09-22T21:45:10.516 DEBUG:boto:Data: 2023-09-22T21:45:10.516 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.516 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.516 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.516 DEBUG:boto:Params: {} 2023-09-22T21:45:10.517 DEBUG:boto:Token: None 2023-09-22T21:45:10.517 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/ 2023-09-22T21:45:10.517 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk= 2023-09-22T21:45:10.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk=', 'Content-Length': '0'} 2023-09-22T21:45:10.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000375ea17aedfd5aa5-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.562 DEBUG:boto:b'ffzdqu-59obj461000falseobj462023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:10.563 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:10.563 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:10.563 DEBUG:boto:path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:45:10.563 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:45:10.563 DEBUG:boto:Method: GET 2023-09-22T21:45:10.563 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj46 2023-09-22T21:45:10.564 DEBUG:boto:Data: 2023-09-22T21:45:10.564 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.564 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.564 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.564 DEBUG:boto:Params: {} 2023-09-22T21:45:10.564 DEBUG:boto:Token: None 2023-09-22T21:45:10.564 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/ 2023-09-22T21:45:10.564 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk= 2023-09-22T21:45:10.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk=', 'Content-Length': '0'} 2023-09-22T21:45:10.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e64772a520555109-00650e0ae6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.574 DEBUG:boto:b'ffzdqu-59obj461000falseobj462023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:10.574 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:45:10.574 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:45:10.574 DEBUG:boto:Method: HEAD 2023-09-22T21:45:10.574 DEBUG:boto:Path: /ffzdqu-59/obj46 2023-09-22T21:45:10.574 DEBUG:boto:Data: 2023-09-22T21:45:10.575 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.575 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.575 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.575 DEBUG:boto:Params: {} 2023-09-22T21:45:10.575 DEBUG:boto:Token: None 2023-09-22T21:45:10.575 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj46 2023-09-22T21:45:10.575 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Hl+5liJh8C+z6TVQtSOyklp/rpE= 2023-09-22T21:45:10.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Hl+5liJh8C+z6TVQtSOyklp/rpE=', 'Content-Length': '0'} 2023-09-22T21:45:10.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f3fe4a51a78d7fc-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.578 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:45:10.578 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:45:10.578 DEBUG:boto:Method: HEAD 2023-09-22T21:45:10.578 DEBUG:boto:Path: /ffzdqu-59/obj46 2023-09-22T21:45:10.579 DEBUG:boto:Data: 2023-09-22T21:45:10.579 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.579 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.579 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.579 DEBUG:boto:Params: {} 2023-09-22T21:45:10.579 DEBUG:boto:Token: None 2023-09-22T21:45:10.579 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj46 2023-09-22T21:45:10.579 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Hl+5liJh8C+z6TVQtSOyklp/rpE= 2023-09-22T21:45:10.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Hl+5liJh8C+z6TVQtSOyklp/rpE=', 'Content-Length': '0'} 2023-09-22T21:45:10.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003aec144dd6a28848-00650e0ae6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-22T21:45:10.583 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:45:10.583 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:45:10.583 DEBUG:boto:Method: GET 2023-09-22T21:45:10.583 DEBUG:boto:Path: /ffzdqu-59/obj46 2023-09-22T21:45:10.583 DEBUG:boto:Data: 2023-09-22T21:45:10.584 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.584 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.584 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.584 DEBUG:boto:Params: {} 2023-09-22T21:45:10.584 DEBUG:boto:Token: None 2023-09-22T21:45:10.584 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj46 2023-09-22T21:45:10.584 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rtH7qOSrDBHu8JVRghdOYnJvufU= 2023-09-22T21:45:10.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rtH7qOSrDBHu8JVRghdOYnJvufU=', 'Content-Length': '0'} 2023-09-22T21:45:10.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015fd5740fee1d3fe-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.630 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:45:10.631 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:45:10.631 DEBUG:boto:Method: GET 2023-09-22T21:45:10.631 DEBUG:boto:Path: /ffzdqu-59/obj46 2023-09-22T21:45:10.631 DEBUG:boto:Data: 2023-09-22T21:45:10.631 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.631 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.631 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.632 DEBUG:boto:Params: {} 2023-09-22T21:45:10.632 DEBUG:boto:Token: None 2023-09-22T21:45:10.632 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj46 2023-09-22T21:45:10.632 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rtH7qOSrDBHu8JVRghdOYnJvufU= 2023-09-22T21:45:10.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rtH7qOSrDBHu8JVRghdOYnJvufU=', 'Content-Length': '0'} 2023-09-22T21:45:10.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d28d3c6522e37a0-00650e0ae6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.635 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:10.635 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:10.636 DEBUG:boto:path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:45:10.636 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:45:10.636 DEBUG:boto:Method: GET 2023-09-22T21:45:10.636 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj46 2023-09-22T21:45:10.636 DEBUG:boto:Data: 2023-09-22T21:45:10.636 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.636 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.636 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.637 DEBUG:boto:Params: {} 2023-09-22T21:45:10.637 DEBUG:boto:Token: None 2023-09-22T21:45:10.637 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/ 2023-09-22T21:45:10.637 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk= 2023-09-22T21:45:10.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk=', 'Content-Length': '0'} 2023-09-22T21:45:10.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbf3da069d126f99-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.682 DEBUG:boto:b'ffzdqu-59obj461000falseobj462023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:10.683 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:10.683 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:10.683 DEBUG:boto:path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:45:10.683 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:45:10.683 DEBUG:boto:Method: GET 2023-09-22T21:45:10.683 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj46 2023-09-22T21:45:10.684 DEBUG:boto:Data: 2023-09-22T21:45:10.684 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.684 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.684 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.684 DEBUG:boto:Params: {} 2023-09-22T21:45:10.684 DEBUG:boto:Token: None 2023-09-22T21:45:10.684 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/ 2023-09-22T21:45:10.684 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk= 2023-09-22T21:45:10.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk=', 'Content-Length': '0'} 2023-09-22T21:45:10.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8262c10a397e7ef-00650e0ae6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.693 DEBUG:boto:b'ffzdqu-59obj461000falseobj462023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:10.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:45:10.693 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:45:10.693 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:45:10.694 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.694 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.694 DEBUG:boto:Method: GET 2023-09-22T21:45:10.694 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.694 DEBUG:boto:Data: 2023-09-22T21:45:10.694 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.694 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.694 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.694 DEBUG:boto:Params: {} 2023-09-22T21:45:10.695 DEBUG:boto:Token: None 2023-09-22T21:45:10.695 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.695 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:K0WyC+ALP+XmoW1Gg8uqG4dRvmU= 2023-09-22T21:45:10.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:K0WyC+ALP+XmoW1Gg8uqG4dRvmU=', 'Content-Length': '0'} 2023-09-22T21:45:10.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a2d78bcf5ee4c31-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.698 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:45:10.698 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:45:10.698 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.698 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.698 DEBUG:boto:Method: GET 2023-09-22T21:45:10.699 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.699 DEBUG:boto:Data: 2023-09-22T21:45:10.699 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.699 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.699 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.699 DEBUG:boto:Params: {} 2023-09-22T21:45:10.699 DEBUG:boto:Token: None 2023-09-22T21:45:10.699 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.699 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:K0WyC+ALP+XmoW1Gg8uqG4dRvmU= 2023-09-22T21:45:10.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:K0WyC+ALP+XmoW1Gg8uqG4dRvmU=', 'Content-Length': '0'} 2023-09-22T21:45:10.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000558c0f5fe65bf018-00650e0ae6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.703 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:45:10.703 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:45:10.703 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.703 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.703 DEBUG:boto:Method: HEAD 2023-09-22T21:45:10.703 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.703 DEBUG:boto:Data: 2023-09-22T21:45:10.703 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.703 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.704 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.704 DEBUG:boto:Params: {} 2023-09-22T21:45:10.704 DEBUG:boto:Token: None 2023-09-22T21:45:10.704 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.704 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qQEW9ApoiYVH0RXVYjGjmpKLEiY= 2023-09-22T21:45:10.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qQEW9ApoiYVH0RXVYjGjmpKLEiY=', 'Content-Length': '0'} 2023-09-22T21:45:10.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003777f1fbe980a309-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.707 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:45:10.707 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:45:10.707 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.707 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.707 DEBUG:boto:Method: HEAD 2023-09-22T21:45:10.708 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.708 DEBUG:boto:Data: 2023-09-22T21:45:10.708 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.708 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.708 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.708 DEBUG:boto:Params: {} 2023-09-22T21:45:10.708 DEBUG:boto:Token: None 2023-09-22T21:45:10.708 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.708 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qQEW9ApoiYVH0RXVYjGjmpKLEiY= 2023-09-22T21:45:10.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qQEW9ApoiYVH0RXVYjGjmpKLEiY=', 'Content-Length': '0'} 2023-09-22T21:45:10.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8135a39fcc67c5b-00650e0ae6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:45:10.712 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:45:10.712 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:45:10.712 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.712 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.712 DEBUG:boto:Method: GET 2023-09-22T21:45:10.712 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.712 DEBUG:boto:Data: 2023-09-22T21:45:10.712 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.712 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.713 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.713 DEBUG:boto:Params: {} 2023-09-22T21:45:10.713 DEBUG:boto:Token: None 2023-09-22T21:45:10.713 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.713 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:K0WyC+ALP+XmoW1Gg8uqG4dRvmU= 2023-09-22T21:45:10.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:K0WyC+ALP+XmoW1Gg8uqG4dRvmU=', 'Content-Length': '0'} 2023-09-22T21:45:10.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f9af3d5cd914f84-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.758 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:45:10.759 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:45:10.759 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.759 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.759 DEBUG:boto:Method: GET 2023-09-22T21:45:10.759 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.759 DEBUG:boto:Data: 2023-09-22T21:45:10.759 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.759 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.759 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.759 DEBUG:boto:Params: {} 2023-09-22T21:45:10.760 DEBUG:boto:Token: None 2023-09-22T21:45:10.760 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:45:10.760 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:K0WyC+ALP+XmoW1Gg8uqG4dRvmU= 2023-09-22T21:45:10.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:K0WyC+ALP+XmoW1Gg8uqG4dRvmU=', 'Content-Length': '0'} 2023-09-22T21:45:10.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d3c288cdc249174-00650e0ae6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.764 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:10.764 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:10.764 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:45:10.764 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:45:10.765 DEBUG:boto:Method: GET 2023-09-22T21:45:10.765 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:45:10.765 DEBUG:boto:Data: 2023-09-22T21:45:10.765 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.765 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.765 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.766 DEBUG:boto:Params: {} 2023-09-22T21:45:10.766 DEBUG:boto:Token: None 2023-09-22T21:45:10.766 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/ 2023-09-22T21:45:10.766 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk= 2023-09-22T21:45:10.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk=', 'Content-Length': '0'} 2023-09-22T21:45:10.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006594181fa528c80a-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.814 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:10.815 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:10.815 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:10.815 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:45:10.815 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:45:10.815 DEBUG:boto:Method: GET 2023-09-22T21:45:10.815 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:45:10.815 DEBUG:boto:Data: 2023-09-22T21:45:10.816 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.816 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.816 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.816 DEBUG:boto:Params: {} 2023-09-22T21:45:10.816 DEBUG:boto:Token: None 2023-09-22T21:45:10.816 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/ 2023-09-22T21:45:10.816 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk= 2023-09-22T21:45:10.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk=', 'Content-Length': '0'} 2023-09-22T21:45:10.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da800809ad22e181-00650e0ae6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.825 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:10.825 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:45:10.825 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:45:10.826 DEBUG:boto:Method: HEAD 2023-09-22T21:45:10.826 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:45:10.826 DEBUG:boto:Data: 2023-09-22T21:45:10.826 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.826 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.826 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.826 DEBUG:boto:Params: {} 2023-09-22T21:45:10.826 DEBUG:boto:Token: None 2023-09-22T21:45:10.826 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj51 2023-09-22T21:45:10.827 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ta6lK7zB2m+SqvZ8yVC8aXXm8Qo= 2023-09-22T21:45:10.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ta6lK7zB2m+SqvZ8yVC8aXXm8Qo=', 'Content-Length': '0'} 2023-09-22T21:45:10.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf4a95641604c005-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.829 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:45:10.830 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:45:10.830 DEBUG:boto:Method: HEAD 2023-09-22T21:45:10.830 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:45:10.830 DEBUG:boto:Data: 2023-09-22T21:45:10.830 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.830 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.830 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.830 DEBUG:boto:Params: {} 2023-09-22T21:45:10.831 DEBUG:boto:Token: None 2023-09-22T21:45:10.831 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj51 2023-09-22T21:45:10.831 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ta6lK7zB2m+SqvZ8yVC8aXXm8Qo= 2023-09-22T21:45:10.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ta6lK7zB2m+SqvZ8yVC8aXXm8Qo=', 'Content-Length': '0'} 2023-09-22T21:45:10.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fee9f549e8299d00-00650e0ae6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:45:10.834 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:45:10.834 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:45:10.834 DEBUG:boto:Method: GET 2023-09-22T21:45:10.835 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:45:10.835 DEBUG:boto:Data: 2023-09-22T21:45:10.835 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.835 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.835 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.835 DEBUG:boto:Params: {} 2023-09-22T21:45:10.835 DEBUG:boto:Token: None 2023-09-22T21:45:10.835 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj51 2023-09-22T21:45:10.836 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GZUX00q3FCYhAsxg4T/EIrg2OKw= 2023-09-22T21:45:10.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GZUX00q3FCYhAsxg4T/EIrg2OKw=', 'Content-Length': '0'} 2023-09-22T21:45:10.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007c4572eb75e488e-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.882 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:45:10.883 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:45:10.883 DEBUG:boto:Method: GET 2023-09-22T21:45:10.883 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:45:10.883 DEBUG:boto:Data: 2023-09-22T21:45:10.883 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.883 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.883 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.883 DEBUG:boto:Params: {} 2023-09-22T21:45:10.884 DEBUG:boto:Token: None 2023-09-22T21:45:10.884 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj51 2023-09-22T21:45:10.884 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GZUX00q3FCYhAsxg4T/EIrg2OKw= 2023-09-22T21:45:10.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GZUX00q3FCYhAsxg4T/EIrg2OKw=', 'Content-Length': '0'} 2023-09-22T21:45:10.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000609cbd84224aff7f-00650e0ae6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.887 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:10.888 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:10.888 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:45:10.888 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:45:10.888 DEBUG:boto:Method: GET 2023-09-22T21:45:10.888 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:45:10.888 DEBUG:boto:Data: 2023-09-22T21:45:10.888 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.888 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.888 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.888 DEBUG:boto:Params: {} 2023-09-22T21:45:10.889 DEBUG:boto:Token: None 2023-09-22T21:45:10.889 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/ 2023-09-22T21:45:10.889 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk= 2023-09-22T21:45:10.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk=', 'Content-Length': '0'} 2023-09-22T21:45:10.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ee94a3a506bc5c4-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.893 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:10.894 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:10.894 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:10.894 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:45:10.894 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:45:10.894 DEBUG:boto:Method: GET 2023-09-22T21:45:10.894 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:45:10.894 DEBUG:boto:Data: 2023-09-22T21:45:10.895 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.895 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.895 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.895 DEBUG:boto:Params: {} 2023-09-22T21:45:10.895 DEBUG:boto:Token: None 2023-09-22T21:45:10.895 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/ 2023-09-22T21:45:10.895 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk= 2023-09-22T21:45:10.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DmElH7OGof87CxLRjQJ9X3i/clk=', 'Content-Length': '0'} 2023-09-22T21:45:10.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ee26aeaca2a90da-00650e0ae6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.904 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:10.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:45:10.904 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:45:10.905 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:45:10.905 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:10.905 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:10.905 DEBUG:boto:Method: GET 2023-09-22T21:45:10.905 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:10.905 DEBUG:boto:Data: 2023-09-22T21:45:10.905 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.905 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.905 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.905 DEBUG:boto:Params: {} 2023-09-22T21:45:10.906 DEBUG:boto:Token: None 2023-09-22T21:45:10.906 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:10.906 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5OZkSkvAx2poRuUdnyYENlhL/hM= 2023-09-22T21:45:10.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5OZkSkvAx2poRuUdnyYENlhL/hM=', 'Content-Length': '0'} 2023-09-22T21:45:10.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed55b1f2f4d88ed2-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.950 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:45:10.951 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:45:10.951 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:10.951 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:10.951 DEBUG:boto:Method: GET 2023-09-22T21:45:10.951 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:10.951 DEBUG:boto:Data: 2023-09-22T21:45:10.951 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.951 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.951 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.951 DEBUG:boto:Params: {} 2023-09-22T21:45:10.952 DEBUG:boto:Token: None 2023-09-22T21:45:10.952 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:10.952 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5OZkSkvAx2poRuUdnyYENlhL/hM= 2023-09-22T21:45:10.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5OZkSkvAx2poRuUdnyYENlhL/hM=', 'Content-Length': '0'} 2023-09-22T21:45:10.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095ba10077dfeddca-00650e0ae6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.955 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:45:10.955 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:45:10.955 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:10.956 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:10.956 DEBUG:boto:Method: HEAD 2023-09-22T21:45:10.956 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:10.956 DEBUG:boto:Data: 2023-09-22T21:45:10.956 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.956 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.956 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.956 DEBUG:boto:Params: {} 2023-09-22T21:45:10.956 DEBUG:boto:Token: None 2023-09-22T21:45:10.956 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:10.957 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x8D30s3tQtIfH1vn62G5FZ580Sc= 2023-09-22T21:45:10.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x8D30s3tQtIfH1vn62G5FZ580Sc=', 'Content-Length': '0'} 2023-09-22T21:45:10.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb47a21eaaae3bc9-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.960 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:45:10.960 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:45:10.960 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:10.960 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:10.960 DEBUG:boto:Method: HEAD 2023-09-22T21:45:10.960 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:10.960 DEBUG:boto:Data: 2023-09-22T21:45:10.960 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.960 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:10.960 DEBUG:boto:Port: 8000 2023-09-22T21:45:10.961 DEBUG:boto:Params: {} 2023-09-22T21:45:10.961 DEBUG:boto:Token: None 2023-09-22T21:45:10.961 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:10.961 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x8D30s3tQtIfH1vn62G5FZ580Sc= 2023-09-22T21:45:10.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x8D30s3tQtIfH1vn62G5FZ580Sc=', 'Content-Length': '0'} 2023-09-22T21:45:10.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028f62dcc50994e47-00650e0ae6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:10.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:45:10.964 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:45:10.964 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:45:10.964 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:10.964 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:10.964 DEBUG:boto:Method: GET 2023-09-22T21:45:10.964 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:10.965 DEBUG:boto:Data: 2023-09-22T21:45:10.965 DEBUG:boto:Headers: {} 2023-09-22T21:45:10.965 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:10.965 DEBUG:boto:Port: 8001 2023-09-22T21:45:10.965 DEBUG:boto:Params: {} 2023-09-22T21:45:10.965 DEBUG:boto:Token: None 2023-09-22T21:45:10.965 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:10 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:10.965 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5OZkSkvAx2poRuUdnyYENlhL/hM= 2023-09-22T21:45:10.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5OZkSkvAx2poRuUdnyYENlhL/hM=', 'Content-Length': '0'} 2023-09-22T21:45:10.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000128458541374d8ec-00650e0ae6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.010 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:45:11.010 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:45:11.011 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:11.011 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:11.011 DEBUG:boto:Method: GET 2023-09-22T21:45:11.011 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:11.011 DEBUG:boto:Data: 2023-09-22T21:45:11.011 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.011 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.011 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.011 DEBUG:boto:Params: {} 2023-09-22T21:45:11.012 DEBUG:boto:Token: None 2023-09-22T21:45:11.012 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:45:11.012 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:39LrVeyO6JoNS+mWVuabsfp8C+o= 2023-09-22T21:45:11.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:39LrVeyO6JoNS+mWVuabsfp8C+o=', 'Content-Length': '0'} 2023-09-22T21:45:11.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebac2d233ae1b8fd-00650e0ae7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.015 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:11.015 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:11.015 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:45:11.016 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:45:11.016 DEBUG:boto:Method: GET 2023-09-22T21:45:11.016 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:45:11.016 DEBUG:boto:Data: 2023-09-22T21:45:11.016 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.016 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.016 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.016 DEBUG:boto:Params: {} 2023-09-22T21:45:11.016 DEBUG:boto:Token: None 2023-09-22T21:45:11.016 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/ 2023-09-22T21:45:11.017 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A= 2023-09-22T21:45:11.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A=', 'Content-Length': '0'} 2023-09-22T21:45:11.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9eef6554724653f-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.062 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:11.065 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:11.065 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:11.065 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:45:11.065 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:45:11.066 DEBUG:boto:Method: GET 2023-09-22T21:45:11.066 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:45:11.066 DEBUG:boto:Data: 2023-09-22T21:45:11.066 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.066 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.066 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.066 DEBUG:boto:Params: {} 2023-09-22T21:45:11.066 DEBUG:boto:Token: None 2023-09-22T21:45:11.067 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/ 2023-09-22T21:45:11.067 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A= 2023-09-22T21:45:11.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A=', 'Content-Length': '0'} 2023-09-22T21:45:11.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e70db1c2cbefa8db-00650e0ae7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.075 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:11.076 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:45:11.076 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:45:11.076 DEBUG:boto:Method: HEAD 2023-09-22T21:45:11.076 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:45:11.076 DEBUG:boto:Data: 2023-09-22T21:45:11.076 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.076 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.077 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.077 DEBUG:boto:Params: {} 2023-09-22T21:45:11.077 DEBUG:boto:Token: None 2023-09-22T21:45:11.077 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj55 2023-09-22T21:45:11.077 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zQL9870yanwEKa1ZQ0ifyg2xD0U= 2023-09-22T21:45:11.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zQL9870yanwEKa1ZQ0ifyg2xD0U=', 'Content-Length': '0'} 2023-09-22T21:45:11.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dcdd0e66f64b659-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.080 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:45:11.080 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:45:11.081 DEBUG:boto:Method: HEAD 2023-09-22T21:45:11.081 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:45:11.081 DEBUG:boto:Data: 2023-09-22T21:45:11.081 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.081 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.081 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.081 DEBUG:boto:Params: {} 2023-09-22T21:45:11.081 DEBUG:boto:Token: None 2023-09-22T21:45:11.081 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj55 2023-09-22T21:45:11.082 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zQL9870yanwEKa1ZQ0ifyg2xD0U= 2023-09-22T21:45:11.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zQL9870yanwEKa1ZQ0ifyg2xD0U=', 'Content-Length': '0'} 2023-09-22T21:45:11.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cb9b0efb338026e-00650e0ae7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:45:11.085 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:45:11.085 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:45:11.085 DEBUG:boto:Method: GET 2023-09-22T21:45:11.085 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:45:11.085 DEBUG:boto:Data: 2023-09-22T21:45:11.085 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.085 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.086 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.086 DEBUG:boto:Params: {} 2023-09-22T21:45:11.086 DEBUG:boto:Token: None 2023-09-22T21:45:11.086 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj55 2023-09-22T21:45:11.086 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:svrlbzfg0RrVJlfDxxrj2zKOuF8= 2023-09-22T21:45:11.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:svrlbzfg0RrVJlfDxxrj2zKOuF8=', 'Content-Length': '0'} 2023-09-22T21:45:11.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afa62b7a4e4b0245-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.130 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:45:11.131 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:45:11.131 DEBUG:boto:Method: GET 2023-09-22T21:45:11.131 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:45:11.131 DEBUG:boto:Data: 2023-09-22T21:45:11.131 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.131 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.131 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.131 DEBUG:boto:Params: {} 2023-09-22T21:45:11.132 DEBUG:boto:Token: None 2023-09-22T21:45:11.132 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj55 2023-09-22T21:45:11.132 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:svrlbzfg0RrVJlfDxxrj2zKOuF8= 2023-09-22T21:45:11.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:svrlbzfg0RrVJlfDxxrj2zKOuF8=', 'Content-Length': '0'} 2023-09-22T21:45:11.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d5f7d3a7ab0dad7-00650e0ae7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.135 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:11.135 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:11.135 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:45:11.135 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:45:11.136 DEBUG:boto:Method: GET 2023-09-22T21:45:11.136 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:45:11.136 DEBUG:boto:Data: 2023-09-22T21:45:11.136 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.136 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.136 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.136 DEBUG:boto:Params: {} 2023-09-22T21:45:11.136 DEBUG:boto:Token: None 2023-09-22T21:45:11.136 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/ 2023-09-22T21:45:11.137 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A= 2023-09-22T21:45:11.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A=', 'Content-Length': '0'} 2023-09-22T21:45:11.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035e07cd7f2c758ff-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.182 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:11.183 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:11.183 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:11.183 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:45:11.183 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:45:11.183 DEBUG:boto:Method: GET 2023-09-22T21:45:11.183 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:45:11.184 DEBUG:boto:Data: 2023-09-22T21:45:11.184 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.184 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.184 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.184 DEBUG:boto:Params: {} 2023-09-22T21:45:11.184 DEBUG:boto:Token: None 2023-09-22T21:45:11.184 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/ 2023-09-22T21:45:11.184 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A= 2023-09-22T21:45:11.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A=', 'Content-Length': '0'} 2023-09-22T21:45:11.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002117918ea04ca3ee-00650e0ae7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.193 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:11.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T21:45:11.194 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:45:11.194 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:45:11.194 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.194 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.194 DEBUG:boto:Method: GET 2023-09-22T21:45:11.194 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.194 DEBUG:boto:Data: 2023-09-22T21:45:11.195 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.195 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.195 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.195 DEBUG:boto:Params: {} 2023-09-22T21:45:11.195 DEBUG:boto:Token: None 2023-09-22T21:45:11.195 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.195 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PqaQ89+sSSzuW++I9w7ppRsEGQI= 2023-09-22T21:45:11.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PqaQ89+sSSzuW++I9w7ppRsEGQI=', 'Content-Length': '0'} 2023-09-22T21:45:11.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000897402fe19132ff4-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.198 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:45:11.198 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:45:11.198 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.199 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.199 DEBUG:boto:Method: GET 2023-09-22T21:45:11.199 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.199 DEBUG:boto:Data: 2023-09-22T21:45:11.199 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.199 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.199 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.199 DEBUG:boto:Params: {} 2023-09-22T21:45:11.199 DEBUG:boto:Token: None 2023-09-22T21:45:11.199 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.200 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PqaQ89+sSSzuW++I9w7ppRsEGQI= 2023-09-22T21:45:11.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PqaQ89+sSSzuW++I9w7ppRsEGQI=', 'Content-Length': '0'} 2023-09-22T21:45:11.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000285c043f40c623e7-00650e0ae7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.202 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:45:11.203 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:45:11.203 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.203 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.203 DEBUG:boto:Method: HEAD 2023-09-22T21:45:11.203 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.203 DEBUG:boto:Data: 2023-09-22T21:45:11.203 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.203 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.203 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.204 DEBUG:boto:Params: {} 2023-09-22T21:45:11.204 DEBUG:boto:Token: None 2023-09-22T21:45:11.204 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.204 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Qxgv7acPGLmBIXsnCMpM2GazFSg= 2023-09-22T21:45:11.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Qxgv7acPGLmBIXsnCMpM2GazFSg=', 'Content-Length': '0'} 2023-09-22T21:45:11.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000017af2c557c53f1c-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.207 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:45:11.207 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:45:11.207 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.207 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.208 DEBUG:boto:Method: HEAD 2023-09-22T21:45:11.208 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.208 DEBUG:boto:Data: 2023-09-22T21:45:11.208 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.208 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.208 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.208 DEBUG:boto:Params: {} 2023-09-22T21:45:11.208 DEBUG:boto:Token: None 2023-09-22T21:45:11.208 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.209 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Qxgv7acPGLmBIXsnCMpM2GazFSg= 2023-09-22T21:45:11.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Qxgv7acPGLmBIXsnCMpM2GazFSg=', 'Content-Length': '0'} 2023-09-22T21:45:11.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099f27f6e9750cae4-00650e0ae7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T21:45:11.212 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:45:11.212 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:45:11.212 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.212 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.212 DEBUG:boto:Method: GET 2023-09-22T21:45:11.212 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.212 DEBUG:boto:Data: 2023-09-22T21:45:11.213 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.213 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.213 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.213 DEBUG:boto:Params: {} 2023-09-22T21:45:11.213 DEBUG:boto:Token: None 2023-09-22T21:45:11.213 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.213 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PqaQ89+sSSzuW++I9w7ppRsEGQI= 2023-09-22T21:45:11.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PqaQ89+sSSzuW++I9w7ppRsEGQI=', 'Content-Length': '0'} 2023-09-22T21:45:11.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c7dbaabd2dfc1a8-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.258 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:45:11.259 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:45:11.259 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.259 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.259 DEBUG:boto:Method: GET 2023-09-22T21:45:11.259 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.259 DEBUG:boto:Data: 2023-09-22T21:45:11.259 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.259 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.259 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.260 DEBUG:boto:Params: {} 2023-09-22T21:45:11.260 DEBUG:boto:Token: None 2023-09-22T21:45:11.260 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:45:11.260 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PqaQ89+sSSzuW++I9w7ppRsEGQI= 2023-09-22T21:45:11.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PqaQ89+sSSzuW++I9w7ppRsEGQI=', 'Content-Length': '0'} 2023-09-22T21:45:11.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000775ee693916de3d2-00650e0ae7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.263 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:11.263 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:11.263 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:45:11.264 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:45:11.264 DEBUG:boto:Method: GET 2023-09-22T21:45:11.264 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:45:11.264 DEBUG:boto:Data: 2023-09-22T21:45:11.264 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.264 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.264 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.264 DEBUG:boto:Params: {} 2023-09-22T21:45:11.264 DEBUG:boto:Token: None 2023-09-22T21:45:11.265 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/ 2023-09-22T21:45:11.265 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A= 2023-09-22T21:45:11.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A=', 'Content-Length': '0'} 2023-09-22T21:45:11.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000059315fae0c3d146-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.314 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:11.315 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:11.315 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:11.315 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:45:11.315 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:45:11.316 DEBUG:boto:Method: GET 2023-09-22T21:45:11.316 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:45:11.316 DEBUG:boto:Data: 2023-09-22T21:45:11.316 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.316 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.316 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.316 DEBUG:boto:Params: {} 2023-09-22T21:45:11.317 DEBUG:boto:Token: None 2023-09-22T21:45:11.317 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/ 2023-09-22T21:45:11.317 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A= 2023-09-22T21:45:11.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A=', 'Content-Length': '0'} 2023-09-22T21:45:11.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1110d3088b0f80b-00650e0ae7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.327 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:11.328 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:45:11.328 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:45:11.328 DEBUG:boto:Method: HEAD 2023-09-22T21:45:11.328 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:45:11.328 DEBUG:boto:Data: 2023-09-22T21:45:11.328 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.328 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.329 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.329 DEBUG:boto:Params: {} 2023-09-22T21:45:11.329 DEBUG:boto:Token: None 2023-09-22T21:45:11.329 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj57 2023-09-22T21:45:11.329 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:snxD4H1Z9E+gHkJjmfrw4KBccJY= 2023-09-22T21:45:11.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:snxD4H1Z9E+gHkJjmfrw4KBccJY=', 'Content-Length': '0'} 2023-09-22T21:45:11.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a082fbe36cca4eb-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.332 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:45:11.332 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:45:11.332 DEBUG:boto:Method: HEAD 2023-09-22T21:45:11.332 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:45:11.332 DEBUG:boto:Data: 2023-09-22T21:45:11.333 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.333 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.333 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.333 DEBUG:boto:Params: {} 2023-09-22T21:45:11.333 DEBUG:boto:Token: None 2023-09-22T21:45:11.333 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj57 2023-09-22T21:45:11.333 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:snxD4H1Z9E+gHkJjmfrw4KBccJY= 2023-09-22T21:45:11.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:snxD4H1Z9E+gHkJjmfrw4KBccJY=', 'Content-Length': '0'} 2023-09-22T21:45:11.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006fa9a75417acda0-00650e0ae7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T21:45:11.337 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:45:11.337 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:45:11.337 DEBUG:boto:Method: GET 2023-09-22T21:45:11.337 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:45:11.337 DEBUG:boto:Data: 2023-09-22T21:45:11.337 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.337 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.337 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.337 DEBUG:boto:Params: {} 2023-09-22T21:45:11.338 DEBUG:boto:Token: None 2023-09-22T21:45:11.338 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj57 2023-09-22T21:45:11.338 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EvOHTdZWK8FWmERPfZDXe4wvfb0= 2023-09-22T21:45:11.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EvOHTdZWK8FWmERPfZDXe4wvfb0=', 'Content-Length': '0'} 2023-09-22T21:45:11.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd933ffda6534fa9-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.387 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:45:11.387 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:45:11.387 DEBUG:boto:Method: GET 2023-09-22T21:45:11.387 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:45:11.387 DEBUG:boto:Data: 2023-09-22T21:45:11.387 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.387 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.387 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.388 DEBUG:boto:Params: {} 2023-09-22T21:45:11.388 DEBUG:boto:Token: None 2023-09-22T21:45:11.388 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj57 2023-09-22T21:45:11.388 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EvOHTdZWK8FWmERPfZDXe4wvfb0= 2023-09-22T21:45:11.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EvOHTdZWK8FWmERPfZDXe4wvfb0=', 'Content-Length': '0'} 2023-09-22T21:45:11.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008593f3fa529fc8d2-00650e0ae7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.391 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:11.392 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:11.392 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:45:11.392 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:45:11.392 DEBUG:boto:Method: GET 2023-09-22T21:45:11.392 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:45:11.392 DEBUG:boto:Data: 2023-09-22T21:45:11.392 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.392 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.392 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.392 DEBUG:boto:Params: {} 2023-09-22T21:45:11.393 DEBUG:boto:Token: None 2023-09-22T21:45:11.393 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/ 2023-09-22T21:45:11.393 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A= 2023-09-22T21:45:11.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A=', 'Content-Length': '0'} 2023-09-22T21:45:11.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009405a215ab943e16-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.442 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:11.443 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:11.443 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:11.443 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:45:11.443 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:45:11.443 DEBUG:boto:Method: GET 2023-09-22T21:45:11.443 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:45:11.443 DEBUG:boto:Data: 2023-09-22T21:45:11.444 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.444 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.444 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.444 DEBUG:boto:Params: {} 2023-09-22T21:45:11.444 DEBUG:boto:Token: None 2023-09-22T21:45:11.444 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/ 2023-09-22T21:45:11.444 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A= 2023-09-22T21:45:11.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A=', 'Content-Length': '0'} 2023-09-22T21:45:11.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dc0af5e33706cfc-00650e0ae7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.452 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:11.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-22T21:45:11.453 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:45:11.453 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:45:11.453 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.453 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.453 DEBUG:boto:Method: GET 2023-09-22T21:45:11.453 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.453 DEBUG:boto:Data: 2023-09-22T21:45:11.454 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.454 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.454 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.454 DEBUG:boto:Params: {} 2023-09-22T21:45:11.454 DEBUG:boto:Token: None 2023-09-22T21:45:11.454 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.454 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Lgy5j6y+jGNCTs4yi42sYqUXPaU= 2023-09-22T21:45:11.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Lgy5j6y+jGNCTs4yi42sYqUXPaU=', 'Content-Length': '0'} 2023-09-22T21:45:11.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f431d77097aaa4a2-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.458 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:45:11.458 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:45:11.458 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.458 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.458 DEBUG:boto:Method: GET 2023-09-22T21:45:11.459 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.459 DEBUG:boto:Data: 2023-09-22T21:45:11.459 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.459 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.459 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.459 DEBUG:boto:Params: {} 2023-09-22T21:45:11.459 DEBUG:boto:Token: None 2023-09-22T21:45:11.459 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.459 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Lgy5j6y+jGNCTs4yi42sYqUXPaU= 2023-09-22T21:45:11.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Lgy5j6y+jGNCTs4yi42sYqUXPaU=', 'Content-Length': '0'} 2023-09-22T21:45:11.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e14bd9eef07df86-00650e0ae7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.463 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:45:11.463 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:45:11.463 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.463 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.463 DEBUG:boto:Method: HEAD 2023-09-22T21:45:11.463 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.463 DEBUG:boto:Data: 2023-09-22T21:45:11.463 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.463 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.464 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.464 DEBUG:boto:Params: {} 2023-09-22T21:45:11.464 DEBUG:boto:Token: None 2023-09-22T21:45:11.464 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.464 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Kpyfd/I9rxxGw4LvhTJBFZLP+fM= 2023-09-22T21:45:11.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Kpyfd/I9rxxGw4LvhTJBFZLP+fM=', 'Content-Length': '0'} 2023-09-22T21:45:11.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066fb39b5d32840d6-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.467 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:45:11.467 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:45:11.467 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.468 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.468 DEBUG:boto:Method: HEAD 2023-09-22T21:45:11.468 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.468 DEBUG:boto:Data: 2023-09-22T21:45:11.468 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.468 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.468 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.468 DEBUG:boto:Params: {} 2023-09-22T21:45:11.468 DEBUG:boto:Token: None 2023-09-22T21:45:11.468 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.469 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Kpyfd/I9rxxGw4LvhTJBFZLP+fM= 2023-09-22T21:45:11.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Kpyfd/I9rxxGw4LvhTJBFZLP+fM=', 'Content-Length': '0'} 2023-09-22T21:45:11.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b3d07eb7ab219c0-00650e0ae7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-22T21:45:11.472 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:45:11.472 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:45:11.472 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.472 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.472 DEBUG:boto:Method: GET 2023-09-22T21:45:11.472 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.472 DEBUG:boto:Data: 2023-09-22T21:45:11.473 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.473 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.473 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.473 DEBUG:boto:Params: {} 2023-09-22T21:45:11.473 DEBUG:boto:Token: None 2023-09-22T21:45:11.473 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.473 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Lgy5j6y+jGNCTs4yi42sYqUXPaU= 2023-09-22T21:45:11.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Lgy5j6y+jGNCTs4yi42sYqUXPaU=', 'Content-Length': '0'} 2023-09-22T21:45:11.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7dd5e17ed274b51-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.522 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:45:11.523 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:45:11.523 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.523 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.523 DEBUG:boto:Method: GET 2023-09-22T21:45:11.523 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.523 DEBUG:boto:Data: 2023-09-22T21:45:11.523 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.523 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.523 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.524 DEBUG:boto:Params: {} 2023-09-22T21:45:11.524 DEBUG:boto:Token: None 2023-09-22T21:45:11.524 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:45:11.524 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Lgy5j6y+jGNCTs4yi42sYqUXPaU= 2023-09-22T21:45:11.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Lgy5j6y+jGNCTs4yi42sYqUXPaU=', 'Content-Length': '0'} 2023-09-22T21:45:11.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc7984f0aa2b95c6-00650e0ae7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.527 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:11.528 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:11.528 DEBUG:boto:path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:45:11.528 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:45:11.528 DEBUG:boto:Method: GET 2023-09-22T21:45:11.528 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj58 2023-09-22T21:45:11.528 DEBUG:boto:Data: 2023-09-22T21:45:11.528 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.528 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.528 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.528 DEBUG:boto:Params: {} 2023-09-22T21:45:11.529 DEBUG:boto:Token: None 2023-09-22T21:45:11.529 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/ 2023-09-22T21:45:11.529 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A= 2023-09-22T21:45:11.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A=', 'Content-Length': '0'} 2023-09-22T21:45:11.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004572586096483b39-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.574 DEBUG:boto:b'ffzdqu-59obj581000falseobj582023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:11.575 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:11.575 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:11.575 DEBUG:boto:path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:45:11.575 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:45:11.575 DEBUG:boto:Method: GET 2023-09-22T21:45:11.575 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj58 2023-09-22T21:45:11.576 DEBUG:boto:Data: 2023-09-22T21:45:11.576 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.576 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.576 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.576 DEBUG:boto:Params: {} 2023-09-22T21:45:11.576 DEBUG:boto:Token: None 2023-09-22T21:45:11.576 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/ 2023-09-22T21:45:11.576 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A= 2023-09-22T21:45:11.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A=', 'Content-Length': '0'} 2023-09-22T21:45:11.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004883ef38f98ea274-00650e0ae7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.585 DEBUG:boto:b'ffzdqu-59obj581000falseobj582023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:11.585 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:45:11.585 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:45:11.586 DEBUG:boto:Method: HEAD 2023-09-22T21:45:11.586 DEBUG:boto:Path: /ffzdqu-59/obj58 2023-09-22T21:45:11.586 DEBUG:boto:Data: 2023-09-22T21:45:11.586 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.586 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.586 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.586 DEBUG:boto:Params: {} 2023-09-22T21:45:11.586 DEBUG:boto:Token: None 2023-09-22T21:45:11.587 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj58 2023-09-22T21:45:11.587 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kPXl2GZbX30R2eLNPDO/fSYe7QA= 2023-09-22T21:45:11.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kPXl2GZbX30R2eLNPDO/fSYe7QA=', 'Content-Length': '0'} 2023-09-22T21:45:11.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d3b0f562bed18bb-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.590 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:45:11.590 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:45:11.590 DEBUG:boto:Method: HEAD 2023-09-22T21:45:11.590 DEBUG:boto:Path: /ffzdqu-59/obj58 2023-09-22T21:45:11.590 DEBUG:boto:Data: 2023-09-22T21:45:11.590 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.591 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.591 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.591 DEBUG:boto:Params: {} 2023-09-22T21:45:11.591 DEBUG:boto:Token: None 2023-09-22T21:45:11.591 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj58 2023-09-22T21:45:11.591 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kPXl2GZbX30R2eLNPDO/fSYe7QA= 2023-09-22T21:45:11.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kPXl2GZbX30R2eLNPDO/fSYe7QA=', 'Content-Length': '0'} 2023-09-22T21:45:11.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002513b620cde5f038-00650e0ae7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-22T21:45:11.594 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:45:11.594 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:45:11.595 DEBUG:boto:Method: GET 2023-09-22T21:45:11.595 DEBUG:boto:Path: /ffzdqu-59/obj58 2023-09-22T21:45:11.595 DEBUG:boto:Data: 2023-09-22T21:45:11.595 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.595 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.595 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.595 DEBUG:boto:Params: {} 2023-09-22T21:45:11.595 DEBUG:boto:Token: None 2023-09-22T21:45:11.595 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj58 2023-09-22T21:45:11.596 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/6TK0CLk63Kf2h/H5sqws2YNPSE= 2023-09-22T21:45:11.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/6TK0CLk63Kf2h/H5sqws2YNPSE=', 'Content-Length': '0'} 2023-09-22T21:45:11.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d5ceada290b296f-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.642 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:45:11.643 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:45:11.643 DEBUG:boto:Method: GET 2023-09-22T21:45:11.643 DEBUG:boto:Path: /ffzdqu-59/obj58 2023-09-22T21:45:11.643 DEBUG:boto:Data: 2023-09-22T21:45:11.643 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.643 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.643 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.643 DEBUG:boto:Params: {} 2023-09-22T21:45:11.643 DEBUG:boto:Token: None 2023-09-22T21:45:11.644 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj58 2023-09-22T21:45:11.644 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/6TK0CLk63Kf2h/H5sqws2YNPSE= 2023-09-22T21:45:11.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/6TK0CLk63Kf2h/H5sqws2YNPSE=', 'Content-Length': '0'} 2023-09-22T21:45:11.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe8f9c771298384b-00650e0ae7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.647 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:11.647 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:11.647 DEBUG:boto:path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:45:11.648 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:45:11.648 DEBUG:boto:Method: GET 2023-09-22T21:45:11.648 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj58 2023-09-22T21:45:11.648 DEBUG:boto:Data: 2023-09-22T21:45:11.648 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.648 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.648 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.648 DEBUG:boto:Params: {} 2023-09-22T21:45:11.648 DEBUG:boto:Token: None 2023-09-22T21:45:11.649 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/ 2023-09-22T21:45:11.649 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A= 2023-09-22T21:45:11.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A=', 'Content-Length': '0'} 2023-09-22T21:45:11.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078c35428dc8912f7-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.653 DEBUG:boto:b'ffzdqu-59obj581000falseobj582023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:11.654 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:11.654 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:11.654 DEBUG:boto:path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:45:11.654 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:45:11.654 DEBUG:boto:Method: GET 2023-09-22T21:45:11.654 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj58 2023-09-22T21:45:11.654 DEBUG:boto:Data: 2023-09-22T21:45:11.654 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.655 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.655 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.655 DEBUG:boto:Params: {} 2023-09-22T21:45:11.655 DEBUG:boto:Token: None 2023-09-22T21:45:11.655 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/ 2023-09-22T21:45:11.655 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A= 2023-09-22T21:45:11.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A=', 'Content-Length': '0'} 2023-09-22T21:45:11.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c57fd793824d058-00650e0ae7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.663 DEBUG:boto:b'ffzdqu-59obj581000falseobj582023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:11.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:45:11.664 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:45:11.664 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:45:11.664 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.664 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.664 DEBUG:boto:Method: GET 2023-09-22T21:45:11.664 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.664 DEBUG:boto:Data: 2023-09-22T21:45:11.665 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.665 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.665 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.665 DEBUG:boto:Params: {} 2023-09-22T21:45:11.665 DEBUG:boto:Token: None 2023-09-22T21:45:11.665 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.665 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hxWU8ghGMaRKchn5bDdbVHMeZ8g= 2023-09-22T21:45:11.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hxWU8ghGMaRKchn5bDdbVHMeZ8g=', 'Content-Length': '0'} 2023-09-22T21:45:11.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c1a2ee88592e72e-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.714 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:45:11.715 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:45:11.715 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.715 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.715 DEBUG:boto:Method: GET 2023-09-22T21:45:11.715 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.715 DEBUG:boto:Data: 2023-09-22T21:45:11.715 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.715 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.715 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.715 DEBUG:boto:Params: {} 2023-09-22T21:45:11.716 DEBUG:boto:Token: None 2023-09-22T21:45:11.716 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.716 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hxWU8ghGMaRKchn5bDdbVHMeZ8g= 2023-09-22T21:45:11.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hxWU8ghGMaRKchn5bDdbVHMeZ8g=', 'Content-Length': '0'} 2023-09-22T21:45:11.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044d74ad29186a58e-00650e0ae7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.719 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:45:11.719 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:45:11.719 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.720 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.720 DEBUG:boto:Method: HEAD 2023-09-22T21:45:11.720 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.720 DEBUG:boto:Data: 2023-09-22T21:45:11.720 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.720 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.720 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.720 DEBUG:boto:Params: {} 2023-09-22T21:45:11.720 DEBUG:boto:Token: None 2023-09-22T21:45:11.721 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.721 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VS4v5UhzxLJ2uO1Si+Hb2db8Taw= 2023-09-22T21:45:11.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VS4v5UhzxLJ2uO1Si+Hb2db8Taw=', 'Content-Length': '0'} 2023-09-22T21:45:11.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064fa596fcc23988c-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.724 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:45:11.724 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:45:11.724 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.724 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.725 DEBUG:boto:Method: HEAD 2023-09-22T21:45:11.725 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.725 DEBUG:boto:Data: 2023-09-22T21:45:11.725 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.725 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.725 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.726 DEBUG:boto:Params: {} 2023-09-22T21:45:11.726 DEBUG:boto:Token: None 2023-09-22T21:45:11.726 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.726 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VS4v5UhzxLJ2uO1Si+Hb2db8Taw= 2023-09-22T21:45:11.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VS4v5UhzxLJ2uO1Si+Hb2db8Taw=', 'Content-Length': '0'} 2023-09-22T21:45:11.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdb85116a8c90571-00650e0ae7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:45:11.730 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:45:11.730 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:45:11.730 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.730 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.730 DEBUG:boto:Method: GET 2023-09-22T21:45:11.730 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.731 DEBUG:boto:Data: 2023-09-22T21:45:11.731 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.731 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.731 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.731 DEBUG:boto:Params: {} 2023-09-22T21:45:11.731 DEBUG:boto:Token: None 2023-09-22T21:45:11.731 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.731 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hxWU8ghGMaRKchn5bDdbVHMeZ8g= 2023-09-22T21:45:11.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hxWU8ghGMaRKchn5bDdbVHMeZ8g=', 'Content-Length': '0'} 2023-09-22T21:45:11.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b4362d5fd0fca20-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.778 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:45:11.779 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:45:11.779 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.779 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.779 DEBUG:boto:Method: GET 2023-09-22T21:45:11.779 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.779 DEBUG:boto:Data: 2023-09-22T21:45:11.779 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.779 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.779 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.780 DEBUG:boto:Params: {} 2023-09-22T21:45:11.780 DEBUG:boto:Token: None 2023-09-22T21:45:11.780 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:45:11.780 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hxWU8ghGMaRKchn5bDdbVHMeZ8g= 2023-09-22T21:45:11.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hxWU8ghGMaRKchn5bDdbVHMeZ8g=', 'Content-Length': '0'} 2023-09-22T21:45:11.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061c200d78884314f-00650e0ae7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.784 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:11.784 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:11.784 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:45:11.784 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:45:11.784 DEBUG:boto:Method: GET 2023-09-22T21:45:11.784 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:45:11.784 DEBUG:boto:Data: 2023-09-22T21:45:11.784 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.784 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.784 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.785 DEBUG:boto:Params: {} 2023-09-22T21:45:11.785 DEBUG:boto:Token: None 2023-09-22T21:45:11.785 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/ 2023-09-22T21:45:11.785 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A= 2023-09-22T21:45:11.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A=', 'Content-Length': '0'} 2023-09-22T21:45:11.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fbd9ba7f376bb3d-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.830 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:11.831 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:11.831 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:11.831 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:45:11.831 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:45:11.831 DEBUG:boto:Method: GET 2023-09-22T21:45:11.832 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:45:11.832 DEBUG:boto:Data: 2023-09-22T21:45:11.832 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.832 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.832 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.832 DEBUG:boto:Params: {} 2023-09-22T21:45:11.832 DEBUG:boto:Token: None 2023-09-22T21:45:11.833 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/ 2023-09-22T21:45:11.833 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A= 2023-09-22T21:45:11.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A=', 'Content-Length': '0'} 2023-09-22T21:45:11.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002be69833b08b09b6-00650e0ae7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.842 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:11.842 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:45:11.842 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:45:11.843 DEBUG:boto:Method: HEAD 2023-09-22T21:45:11.843 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:45:11.843 DEBUG:boto:Data: 2023-09-22T21:45:11.843 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.843 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.843 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.843 DEBUG:boto:Params: {} 2023-09-22T21:45:11.844 DEBUG:boto:Token: None 2023-09-22T21:45:11.844 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj68 2023-09-22T21:45:11.844 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f5//b7pObr3LX/V+kIiRh1lw3Ug= 2023-09-22T21:45:11.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f5//b7pObr3LX/V+kIiRh1lw3Ug=', 'Content-Length': '0'} 2023-09-22T21:45:11.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004be3627d4b21c391-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.847 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:45:11.847 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:45:11.848 DEBUG:boto:Method: HEAD 2023-09-22T21:45:11.848 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:45:11.848 DEBUG:boto:Data: 2023-09-22T21:45:11.848 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.848 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.848 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.848 DEBUG:boto:Params: {} 2023-09-22T21:45:11.848 DEBUG:boto:Token: None 2023-09-22T21:45:11.849 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj68 2023-09-22T21:45:11.849 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f5//b7pObr3LX/V+kIiRh1lw3Ug= 2023-09-22T21:45:11.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f5//b7pObr3LX/V+kIiRh1lw3Ug=', 'Content-Length': '0'} 2023-09-22T21:45:11.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c7b064e2fcb63a6-00650e0ae7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:45:11.852 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:45:11.853 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:45:11.853 DEBUG:boto:Method: GET 2023-09-22T21:45:11.853 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:45:11.853 DEBUG:boto:Data: 2023-09-22T21:45:11.853 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.853 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.854 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.854 DEBUG:boto:Params: {} 2023-09-22T21:45:11.854 DEBUG:boto:Token: None 2023-09-22T21:45:11.854 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj68 2023-09-22T21:45:11.854 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NB5TgYpNmtJDS9BvR7HkIm9Un8Q= 2023-09-22T21:45:11.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NB5TgYpNmtJDS9BvR7HkIm9Un8Q=', 'Content-Length': '0'} 2023-09-22T21:45:11.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d04c6cd9d1ce6b2a-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.898 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:45:11.899 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:45:11.899 DEBUG:boto:Method: GET 2023-09-22T21:45:11.899 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:45:11.899 DEBUG:boto:Data: 2023-09-22T21:45:11.899 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.899 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.900 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.900 DEBUG:boto:Params: {} 2023-09-22T21:45:11.900 DEBUG:boto:Token: None 2023-09-22T21:45:11.900 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj68 2023-09-22T21:45:11.900 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NB5TgYpNmtJDS9BvR7HkIm9Un8Q= 2023-09-22T21:45:11.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NB5TgYpNmtJDS9BvR7HkIm9Un8Q=', 'Content-Length': '0'} 2023-09-22T21:45:11.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003353b7532d032eef-00650e0ae7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.904 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:11.904 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:11.904 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:45:11.904 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:45:11.905 DEBUG:boto:Method: GET 2023-09-22T21:45:11.905 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:45:11.905 DEBUG:boto:Data: 2023-09-22T21:45:11.905 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.905 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.905 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.905 DEBUG:boto:Params: {} 2023-09-22T21:45:11.905 DEBUG:boto:Token: None 2023-09-22T21:45:11.906 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/ 2023-09-22T21:45:11.906 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A= 2023-09-22T21:45:11.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A=', 'Content-Length': '0'} 2023-09-22T21:45:11.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2c9bd45ea2e4cd0-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.954 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:11.955 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:11.955 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:11.955 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:45:11.955 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:45:11.955 DEBUG:boto:Method: GET 2023-09-22T21:45:11.956 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:45:11.956 DEBUG:boto:Data: 2023-09-22T21:45:11.956 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.956 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.956 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.956 DEBUG:boto:Params: {} 2023-09-22T21:45:11.956 DEBUG:boto:Token: None 2023-09-22T21:45:11.956 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/ 2023-09-22T21:45:11.957 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A= 2023-09-22T21:45:11.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LAGSqRo1uCYWADmToTOR8qkKS9A=', 'Content-Length': '0'} 2023-09-22T21:45:11.965 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e8f26cf763fc440-00650e0ae7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.965 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:11.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-22T21:45:11.966 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:45:11.966 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:45:11.966 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:11.966 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:11.966 DEBUG:boto:Method: GET 2023-09-22T21:45:11.966 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:11.967 DEBUG:boto:Data: 2023-09-22T21:45:11.967 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.967 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.967 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.967 DEBUG:boto:Params: {} 2023-09-22T21:45:11.967 DEBUG:boto:Token: None 2023-09-22T21:45:11.967 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:11.967 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Urg6hVxM7HdpEx9QvkvftLu7CH8= 2023-09-22T21:45:11.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Urg6hVxM7HdpEx9QvkvftLu7CH8=', 'Content-Length': '0'} 2023-09-22T21:45:11.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fadb426cb662cdf5-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.971 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:45:11.971 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:45:11.971 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:11.972 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:11.972 DEBUG:boto:Method: GET 2023-09-22T21:45:11.972 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:11.972 DEBUG:boto:Data: 2023-09-22T21:45:11.972 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.972 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.972 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.972 DEBUG:boto:Params: {} 2023-09-22T21:45:11.973 DEBUG:boto:Token: None 2023-09-22T21:45:11.973 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:11.973 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Urg6hVxM7HdpEx9QvkvftLu7CH8= 2023-09-22T21:45:11.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Urg6hVxM7HdpEx9QvkvftLu7CH8=', 'Content-Length': '0'} 2023-09-22T21:45:11.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a45fb96c7fb1660-00650e0ae7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.976 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:45:11.976 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:45:11.976 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:11.976 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:11.976 DEBUG:boto:Method: HEAD 2023-09-22T21:45:11.977 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:11.977 DEBUG:boto:Data: 2023-09-22T21:45:11.977 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.977 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.977 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.977 DEBUG:boto:Params: {} 2023-09-22T21:45:11.977 DEBUG:boto:Token: None 2023-09-22T21:45:11.977 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:11.978 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pd1oreLt0ADieFn7I7cW1ztE/Ic= 2023-09-22T21:45:11.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pd1oreLt0ADieFn7I7cW1ztE/Ic=', 'Content-Length': '0'} 2023-09-22T21:45:11.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f807ba5fa6b6a99-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.980 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:45:11.981 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:45:11.981 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:11.981 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:11.981 DEBUG:boto:Method: HEAD 2023-09-22T21:45:11.981 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:11.981 DEBUG:boto:Data: 2023-09-22T21:45:11.981 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.981 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:11.982 DEBUG:boto:Port: 8000 2023-09-22T21:45:11.982 DEBUG:boto:Params: {} 2023-09-22T21:45:11.982 DEBUG:boto:Token: None 2023-09-22T21:45:11.982 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:11.982 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pd1oreLt0ADieFn7I7cW1ztE/Ic= 2023-09-22T21:45:11.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pd1oreLt0ADieFn7I7cW1ztE/Ic=', 'Content-Length': '0'} 2023-09-22T21:45:11.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069607917d3dbbb9f-00650e0ae7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:11.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-22T21:45:11.986 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:45:11.986 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:45:11.986 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:11.986 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:11.986 DEBUG:boto:Method: GET 2023-09-22T21:45:11.986 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:11.986 DEBUG:boto:Data: 2023-09-22T21:45:11.987 DEBUG:boto:Headers: {} 2023-09-22T21:45:11.987 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:11.987 DEBUG:boto:Port: 8001 2023-09-22T21:45:11.987 DEBUG:boto:Params: {} 2023-09-22T21:45:11.987 DEBUG:boto:Token: None 2023-09-22T21:45:11.987 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:11 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:11.987 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Urg6hVxM7HdpEx9QvkvftLu7CH8= 2023-09-22T21:45:11.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Urg6hVxM7HdpEx9QvkvftLu7CH8=', 'Content-Length': '0'} 2023-09-22T21:45:11.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f33036c777fbb7c0-00650e0ae7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.034 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:45:12.035 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:45:12.035 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:12.035 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:12.035 DEBUG:boto:Method: GET 2023-09-22T21:45:12.035 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:12.035 DEBUG:boto:Data: 2023-09-22T21:45:12.035 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.035 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.036 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.036 DEBUG:boto:Params: {} 2023-09-22T21:45:12.036 DEBUG:boto:Token: None 2023-09-22T21:45:12.036 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:45:12.036 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uYQKM2lgxfy72xPq3wMyG9udqtk= 2023-09-22T21:45:12.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uYQKM2lgxfy72xPq3wMyG9udqtk=', 'Content-Length': '0'} 2023-09-22T21:45:12.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000821248a8f4798b2a-00650e0ae8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.040 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:12.040 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:12.040 DEBUG:boto:path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:45:12.040 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:45:12.040 DEBUG:boto:Method: GET 2023-09-22T21:45:12.040 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj69 2023-09-22T21:45:12.040 DEBUG:boto:Data: 2023-09-22T21:45:12.040 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.041 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.041 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.041 DEBUG:boto:Params: {} 2023-09-22T21:45:12.041 DEBUG:boto:Token: None 2023-09-22T21:45:12.041 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/ 2023-09-22T21:45:12.041 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0= 2023-09-22T21:45:12.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0=', 'Content-Length': '0'} 2023-09-22T21:45:12.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbf95da571039057-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.090 DEBUG:boto:b'ffzdqu-59obj691000falseobj692023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:12.091 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:12.091 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:12.091 DEBUG:boto:path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:45:12.091 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:45:12.092 DEBUG:boto:Method: GET 2023-09-22T21:45:12.092 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj69 2023-09-22T21:45:12.092 DEBUG:boto:Data: 2023-09-22T21:45:12.092 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.092 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.092 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.092 DEBUG:boto:Params: {} 2023-09-22T21:45:12.093 DEBUG:boto:Token: None 2023-09-22T21:45:12.093 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/ 2023-09-22T21:45:12.093 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0= 2023-09-22T21:45:12.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0=', 'Content-Length': '0'} 2023-09-22T21:45:12.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b441acc5b681ae2-00650e0ae8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.102 DEBUG:boto:b'ffzdqu-59obj691000falseobj692023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:12.102 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:45:12.103 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:45:12.103 DEBUG:boto:Method: HEAD 2023-09-22T21:45:12.103 DEBUG:boto:Path: /ffzdqu-59/obj69 2023-09-22T21:45:12.103 DEBUG:boto:Data: 2023-09-22T21:45:12.103 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.103 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.103 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.103 DEBUG:boto:Params: {} 2023-09-22T21:45:12.104 DEBUG:boto:Token: None 2023-09-22T21:45:12.104 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj69 2023-09-22T21:45:12.104 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bd5ISIIlQDzd7wTywdAxE3Qq3AQ= 2023-09-22T21:45:12.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bd5ISIIlQDzd7wTywdAxE3Qq3AQ=', 'Content-Length': '0'} 2023-09-22T21:45:12.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b368199c213a214-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.108 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:45:12.108 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:45:12.108 DEBUG:boto:Method: HEAD 2023-09-22T21:45:12.108 DEBUG:boto:Path: /ffzdqu-59/obj69 2023-09-22T21:45:12.108 DEBUG:boto:Data: 2023-09-22T21:45:12.108 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.109 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.109 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.109 DEBUG:boto:Params: {} 2023-09-22T21:45:12.109 DEBUG:boto:Token: None 2023-09-22T21:45:12.109 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj69 2023-09-22T21:45:12.109 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bd5ISIIlQDzd7wTywdAxE3Qq3AQ= 2023-09-22T21:45:12.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bd5ISIIlQDzd7wTywdAxE3Qq3AQ=', 'Content-Length': '0'} 2023-09-22T21:45:12.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053aff44ebae26846-00650e0ae8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-22T21:45:12.112 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:45:12.112 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:45:12.113 DEBUG:boto:Method: GET 2023-09-22T21:45:12.113 DEBUG:boto:Path: /ffzdqu-59/obj69 2023-09-22T21:45:12.113 DEBUG:boto:Data: 2023-09-22T21:45:12.113 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.113 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.113 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.113 DEBUG:boto:Params: {} 2023-09-22T21:45:12.113 DEBUG:boto:Token: None 2023-09-22T21:45:12.114 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj69 2023-09-22T21:45:12.114 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yiMOISsuUDOoenFPxrOliqswAh4= 2023-09-22T21:45:12.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yiMOISsuUDOoenFPxrOliqswAh4=', 'Content-Length': '0'} 2023-09-22T21:45:12.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2354750a6e888ef-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.158 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:45:12.159 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:45:12.159 DEBUG:boto:Method: GET 2023-09-22T21:45:12.159 DEBUG:boto:Path: /ffzdqu-59/obj69 2023-09-22T21:45:12.159 DEBUG:boto:Data: 2023-09-22T21:45:12.159 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.159 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.159 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.159 DEBUG:boto:Params: {} 2023-09-22T21:45:12.160 DEBUG:boto:Token: None 2023-09-22T21:45:12.160 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj69 2023-09-22T21:45:12.160 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yiMOISsuUDOoenFPxrOliqswAh4= 2023-09-22T21:45:12.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yiMOISsuUDOoenFPxrOliqswAh4=', 'Content-Length': '0'} 2023-09-22T21:45:12.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064877d0b64287589-00650e0ae8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.163 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:12.163 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:12.163 DEBUG:boto:path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:45:12.164 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:45:12.164 DEBUG:boto:Method: GET 2023-09-22T21:45:12.164 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj69 2023-09-22T21:45:12.164 DEBUG:boto:Data: 2023-09-22T21:45:12.164 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.164 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.165 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.165 DEBUG:boto:Params: {} 2023-09-22T21:45:12.165 DEBUG:boto:Token: None 2023-09-22T21:45:12.165 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/ 2023-09-22T21:45:12.165 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0= 2023-09-22T21:45:12.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0=', 'Content-Length': '0'} 2023-09-22T21:45:12.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4b58b28a2b2b9c1-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.210 DEBUG:boto:b'ffzdqu-59obj691000falseobj692023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:12.211 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:12.211 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:12.211 DEBUG:boto:path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:45:12.211 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:45:12.211 DEBUG:boto:Method: GET 2023-09-22T21:45:12.211 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj69 2023-09-22T21:45:12.212 DEBUG:boto:Data: 2023-09-22T21:45:12.212 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.213 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.213 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.213 DEBUG:boto:Params: {} 2023-09-22T21:45:12.213 DEBUG:boto:Token: None 2023-09-22T21:45:12.213 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/ 2023-09-22T21:45:12.213 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0= 2023-09-22T21:45:12.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0=', 'Content-Length': '0'} 2023-09-22T21:45:12.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c15040f77274832c-00650e0ae8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.222 DEBUG:boto:b'ffzdqu-59obj691000falseobj692023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:12.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T21:45:12.223 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:45:12.223 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:45:12.223 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.223 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.223 DEBUG:boto:Method: GET 2023-09-22T21:45:12.223 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.223 DEBUG:boto:Data: 2023-09-22T21:45:12.224 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.224 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.224 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.224 DEBUG:boto:Params: {} 2023-09-22T21:45:12.224 DEBUG:boto:Token: None 2023-09-22T21:45:12.224 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.224 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2mAhjeUCmzJpIt4A/9QvWjeCtOs= 2023-09-22T21:45:12.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2mAhjeUCmzJpIt4A/9QvWjeCtOs=', 'Content-Length': '0'} 2023-09-22T21:45:12.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d3baf6fbe08cf13-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.228 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:45:12.228 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:45:12.228 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.228 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.228 DEBUG:boto:Method: GET 2023-09-22T21:45:12.228 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.228 DEBUG:boto:Data: 2023-09-22T21:45:12.228 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.228 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.229 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.229 DEBUG:boto:Params: {} 2023-09-22T21:45:12.229 DEBUG:boto:Token: None 2023-09-22T21:45:12.229 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.229 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2mAhjeUCmzJpIt4A/9QvWjeCtOs= 2023-09-22T21:45:12.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2mAhjeUCmzJpIt4A/9QvWjeCtOs=', 'Content-Length': '0'} 2023-09-22T21:45:12.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a5e7c5318ef17bf-00650e0ae8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.232 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:45:12.232 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:45:12.233 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.233 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.233 DEBUG:boto:Method: HEAD 2023-09-22T21:45:12.233 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.233 DEBUG:boto:Data: 2023-09-22T21:45:12.233 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.233 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.233 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.233 DEBUG:boto:Params: {} 2023-09-22T21:45:12.233 DEBUG:boto:Token: None 2023-09-22T21:45:12.234 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.234 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:saJmPvWWsk5NyqMnhZxii7zkwtQ= 2023-09-22T21:45:12.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:saJmPvWWsk5NyqMnhZxii7zkwtQ=', 'Content-Length': '0'} 2023-09-22T21:45:12.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be80eca58a602a95-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.237 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:45:12.237 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:45:12.237 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.237 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.237 DEBUG:boto:Method: HEAD 2023-09-22T21:45:12.237 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.238 DEBUG:boto:Data: 2023-09-22T21:45:12.238 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.238 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.238 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.238 DEBUG:boto:Params: {} 2023-09-22T21:45:12.238 DEBUG:boto:Token: None 2023-09-22T21:45:12.238 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.238 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:saJmPvWWsk5NyqMnhZxii7zkwtQ= 2023-09-22T21:45:12.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:saJmPvWWsk5NyqMnhZxii7zkwtQ=', 'Content-Length': '0'} 2023-09-22T21:45:12.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029814ee5aac0ecb5-00650e0ae8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T21:45:12.242 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:45:12.242 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:45:12.242 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.243 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.243 DEBUG:boto:Method: GET 2023-09-22T21:45:12.243 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.243 DEBUG:boto:Data: 2023-09-22T21:45:12.243 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.243 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.243 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.243 DEBUG:boto:Params: {} 2023-09-22T21:45:12.243 DEBUG:boto:Token: None 2023-09-22T21:45:12.244 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.244 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2mAhjeUCmzJpIt4A/9QvWjeCtOs= 2023-09-22T21:45:12.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2mAhjeUCmzJpIt4A/9QvWjeCtOs=', 'Content-Length': '0'} 2023-09-22T21:45:12.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001999b5d08e789bad-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.294 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:45:12.295 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:45:12.295 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.295 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.295 DEBUG:boto:Method: GET 2023-09-22T21:45:12.295 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.295 DEBUG:boto:Data: 2023-09-22T21:45:12.295 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.295 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.297 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.297 DEBUG:boto:Params: {} 2023-09-22T21:45:12.297 DEBUG:boto:Token: None 2023-09-22T21:45:12.297 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:45:12.297 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2mAhjeUCmzJpIt4A/9QvWjeCtOs= 2023-09-22T21:45:12.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2mAhjeUCmzJpIt4A/9QvWjeCtOs=', 'Content-Length': '0'} 2023-09-22T21:45:12.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009df1f40ff7d72a2d-00650e0ae8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.301 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:12.301 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:12.301 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:45:12.301 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:45:12.301 DEBUG:boto:Method: GET 2023-09-22T21:45:12.301 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:45:12.302 DEBUG:boto:Data: 2023-09-22T21:45:12.302 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.302 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.302 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.302 DEBUG:boto:Params: {} 2023-09-22T21:45:12.302 DEBUG:boto:Token: None 2023-09-22T21:45:12.302 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/ 2023-09-22T21:45:12.302 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0= 2023-09-22T21:45:12.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0=', 'Content-Length': '0'} 2023-09-22T21:45:12.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a0251b8b47fb3fe-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.350 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:12.351 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:12.351 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:12.351 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:45:12.351 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:45:12.351 DEBUG:boto:Method: GET 2023-09-22T21:45:12.352 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:45:12.352 DEBUG:boto:Data: 2023-09-22T21:45:12.352 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.352 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.352 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.352 DEBUG:boto:Params: {} 2023-09-22T21:45:12.352 DEBUG:boto:Token: None 2023-09-22T21:45:12.352 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/ 2023-09-22T21:45:12.353 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0= 2023-09-22T21:45:12.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0=', 'Content-Length': '0'} 2023-09-22T21:45:12.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a57ff40ad0e0a17b-00650e0ae8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.361 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:12.362 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:45:12.362 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:45:12.362 DEBUG:boto:Method: HEAD 2023-09-22T21:45:12.362 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:45:12.362 DEBUG:boto:Data: 2023-09-22T21:45:12.362 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.362 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.362 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.362 DEBUG:boto:Params: {} 2023-09-22T21:45:12.363 DEBUG:boto:Token: None 2023-09-22T21:45:12.363 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj76 2023-09-22T21:45:12.363 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dhikOHRgbX7iEBgDF2eiZmxh1zw= 2023-09-22T21:45:12.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dhikOHRgbX7iEBgDF2eiZmxh1zw=', 'Content-Length': '0'} 2023-09-22T21:45:12.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1dfdc69d8836cd1-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.366 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:45:12.366 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:45:12.366 DEBUG:boto:Method: HEAD 2023-09-22T21:45:12.366 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:45:12.366 DEBUG:boto:Data: 2023-09-22T21:45:12.366 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.366 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.367 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.367 DEBUG:boto:Params: {} 2023-09-22T21:45:12.367 DEBUG:boto:Token: None 2023-09-22T21:45:12.367 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj76 2023-09-22T21:45:12.367 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dhikOHRgbX7iEBgDF2eiZmxh1zw= 2023-09-22T21:45:12.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dhikOHRgbX7iEBgDF2eiZmxh1zw=', 'Content-Length': '0'} 2023-09-22T21:45:12.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adf92ef0cd0fa44e-00650e0ae8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T21:45:12.370 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:45:12.370 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:45:12.370 DEBUG:boto:Method: GET 2023-09-22T21:45:12.370 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:45:12.371 DEBUG:boto:Data: 2023-09-22T21:45:12.371 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.371 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.371 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.371 DEBUG:boto:Params: {} 2023-09-22T21:45:12.371 DEBUG:boto:Token: None 2023-09-22T21:45:12.371 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj76 2023-09-22T21:45:12.371 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cj/JtkPfk35+2KFdHN38l5EYoYc= 2023-09-22T21:45:12.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cj/JtkPfk35+2KFdHN38l5EYoYc=', 'Content-Length': '0'} 2023-09-22T21:45:12.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057a4b81a9fd427a7-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.415 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:45:12.415 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:45:12.415 DEBUG:boto:Method: GET 2023-09-22T21:45:12.415 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:45:12.415 DEBUG:boto:Data: 2023-09-22T21:45:12.415 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.415 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.415 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.416 DEBUG:boto:Params: {} 2023-09-22T21:45:12.416 DEBUG:boto:Token: None 2023-09-22T21:45:12.416 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj76 2023-09-22T21:45:12.416 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cj/JtkPfk35+2KFdHN38l5EYoYc= 2023-09-22T21:45:12.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cj/JtkPfk35+2KFdHN38l5EYoYc=', 'Content-Length': '0'} 2023-09-22T21:45:12.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea9530d3a7201a64-00650e0ae8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.419 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:12.419 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:12.420 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:45:12.420 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:45:12.420 DEBUG:boto:Method: GET 2023-09-22T21:45:12.420 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:45:12.420 DEBUG:boto:Data: 2023-09-22T21:45:12.420 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.420 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.420 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.420 DEBUG:boto:Params: {} 2023-09-22T21:45:12.420 DEBUG:boto:Token: None 2023-09-22T21:45:12.421 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/ 2023-09-22T21:45:12.421 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0= 2023-09-22T21:45:12.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0=', 'Content-Length': '0'} 2023-09-22T21:45:12.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f24a86622c73dda-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.426 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:12.427 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:12.427 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:12.427 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:45:12.427 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:45:12.427 DEBUG:boto:Method: GET 2023-09-22T21:45:12.427 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:45:12.427 DEBUG:boto:Data: 2023-09-22T21:45:12.428 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.428 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.428 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.428 DEBUG:boto:Params: {} 2023-09-22T21:45:12.428 DEBUG:boto:Token: None 2023-09-22T21:45:12.428 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/ 2023-09-22T21:45:12.428 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0= 2023-09-22T21:45:12.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0=', 'Content-Length': '0'} 2023-09-22T21:45:12.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000766f0df510406b3-00650e0ae8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.436 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:12.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:45:12.437 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:45:12.437 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:45:12.437 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.437 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.437 DEBUG:boto:Method: GET 2023-09-22T21:45:12.437 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.437 DEBUG:boto:Data: 2023-09-22T21:45:12.437 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.437 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.438 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.438 DEBUG:boto:Params: {} 2023-09-22T21:45:12.438 DEBUG:boto:Token: None 2023-09-22T21:45:12.438 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.438 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kdcQ5c0s5VurWdecWbw7+etLRyc= 2023-09-22T21:45:12.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kdcQ5c0s5VurWdecWbw7+etLRyc=', 'Content-Length': '0'} 2023-09-22T21:45:12.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dc334c5de70f030-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.482 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:45:12.483 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:45:12.483 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.483 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.483 DEBUG:boto:Method: GET 2023-09-22T21:45:12.483 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.483 DEBUG:boto:Data: 2023-09-22T21:45:12.483 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.483 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.484 DEBUG:boto:Params: {} 2023-09-22T21:45:12.484 DEBUG:boto:Token: None 2023-09-22T21:45:12.484 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.484 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kdcQ5c0s5VurWdecWbw7+etLRyc= 2023-09-22T21:45:12.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kdcQ5c0s5VurWdecWbw7+etLRyc=', 'Content-Length': '0'} 2023-09-22T21:45:12.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8069bd6f3abd148-00650e0ae8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.487 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:45:12.488 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:45:12.488 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.488 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.488 DEBUG:boto:Method: HEAD 2023-09-22T21:45:12.488 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.488 DEBUG:boto:Data: 2023-09-22T21:45:12.488 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.488 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.488 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.488 DEBUG:boto:Params: {} 2023-09-22T21:45:12.489 DEBUG:boto:Token: None 2023-09-22T21:45:12.489 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.489 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4fJxwvzcLd4eUGgmeCfgLeSu4VY= 2023-09-22T21:45:12.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4fJxwvzcLd4eUGgmeCfgLeSu4VY=', 'Content-Length': '0'} 2023-09-22T21:45:12.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e84acffd6e12982f-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.492 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:45:12.492 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:45:12.492 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.492 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.492 DEBUG:boto:Method: HEAD 2023-09-22T21:45:12.492 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.492 DEBUG:boto:Data: 2023-09-22T21:45:12.493 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.493 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.493 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.493 DEBUG:boto:Params: {} 2023-09-22T21:45:12.493 DEBUG:boto:Token: None 2023-09-22T21:45:12.493 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.493 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4fJxwvzcLd4eUGgmeCfgLeSu4VY= 2023-09-22T21:45:12.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4fJxwvzcLd4eUGgmeCfgLeSu4VY=', 'Content-Length': '0'} 2023-09-22T21:45:12.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f7a71b00192b148-00650e0ae8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:45:12.497 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:45:12.497 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:45:12.497 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.497 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.497 DEBUG:boto:Method: GET 2023-09-22T21:45:12.497 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.497 DEBUG:boto:Data: 2023-09-22T21:45:12.497 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.497 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.498 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.498 DEBUG:boto:Params: {} 2023-09-22T21:45:12.498 DEBUG:boto:Token: None 2023-09-22T21:45:12.498 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.498 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kdcQ5c0s5VurWdecWbw7+etLRyc= 2023-09-22T21:45:12.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kdcQ5c0s5VurWdecWbw7+etLRyc=', 'Content-Length': '0'} 2023-09-22T21:45:12.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c1851a850d19525-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.542 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:45:12.543 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:45:12.543 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.543 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.543 DEBUG:boto:Method: GET 2023-09-22T21:45:12.543 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.543 DEBUG:boto:Data: 2023-09-22T21:45:12.543 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.543 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.543 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.544 DEBUG:boto:Params: {} 2023-09-22T21:45:12.544 DEBUG:boto:Token: None 2023-09-22T21:45:12.544 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:45:12.544 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kdcQ5c0s5VurWdecWbw7+etLRyc= 2023-09-22T21:45:12.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kdcQ5c0s5VurWdecWbw7+etLRyc=', 'Content-Length': '0'} 2023-09-22T21:45:12.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd3d110b936332d5-00650e0ae8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.548 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:12.548 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:12.548 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:45:12.548 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:45:12.548 DEBUG:boto:Method: GET 2023-09-22T21:45:12.548 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:45:12.548 DEBUG:boto:Data: 2023-09-22T21:45:12.548 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.549 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.549 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.549 DEBUG:boto:Params: {} 2023-09-22T21:45:12.549 DEBUG:boto:Token: None 2023-09-22T21:45:12.549 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/ 2023-09-22T21:45:12.549 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0= 2023-09-22T21:45:12.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0=', 'Content-Length': '0'} 2023-09-22T21:45:12.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0cd337edcae4660-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.594 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:12.595 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:12.595 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:12.595 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:45:12.595 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:45:12.595 DEBUG:boto:Method: GET 2023-09-22T21:45:12.595 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:45:12.596 DEBUG:boto:Data: 2023-09-22T21:45:12.596 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.596 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.596 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.596 DEBUG:boto:Params: {} 2023-09-22T21:45:12.596 DEBUG:boto:Token: None 2023-09-22T21:45:12.596 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/ 2023-09-22T21:45:12.596 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0= 2023-09-22T21:45:12.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0=', 'Content-Length': '0'} 2023-09-22T21:45:12.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042bc822641185c08-00650e0ae8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.605 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:12.605 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:45:12.605 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:45:12.606 DEBUG:boto:Method: HEAD 2023-09-22T21:45:12.606 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:45:12.606 DEBUG:boto:Data: 2023-09-22T21:45:12.606 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.606 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.606 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.606 DEBUG:boto:Params: {} 2023-09-22T21:45:12.606 DEBUG:boto:Token: None 2023-09-22T21:45:12.607 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj85 2023-09-22T21:45:12.607 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NaDX2vhf3l0/dcoLNT9AxqTNyY4= 2023-09-22T21:45:12.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NaDX2vhf3l0/dcoLNT9AxqTNyY4=', 'Content-Length': '0'} 2023-09-22T21:45:12.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000533d40c0e7d8d6c4-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.610 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:45:12.610 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:45:12.611 DEBUG:boto:Method: HEAD 2023-09-22T21:45:12.611 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:45:12.611 DEBUG:boto:Data: 2023-09-22T21:45:12.611 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.611 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.611 DEBUG:boto:Params: {} 2023-09-22T21:45:12.611 DEBUG:boto:Token: None 2023-09-22T21:45:12.611 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj85 2023-09-22T21:45:12.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NaDX2vhf3l0/dcoLNT9AxqTNyY4= 2023-09-22T21:45:12.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NaDX2vhf3l0/dcoLNT9AxqTNyY4=', 'Content-Length': '0'} 2023-09-22T21:45:12.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a91ee70cb32ce9c6-00650e0ae8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:45:12.615 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:45:12.615 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:45:12.615 DEBUG:boto:Method: GET 2023-09-22T21:45:12.615 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:45:12.615 DEBUG:boto:Data: 2023-09-22T21:45:12.615 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.615 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.615 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.616 DEBUG:boto:Params: {} 2023-09-22T21:45:12.616 DEBUG:boto:Token: None 2023-09-22T21:45:12.616 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj85 2023-09-22T21:45:12.616 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aTU/q1GIKp50P5AL434Y5rKM6f4= 2023-09-22T21:45:12.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aTU/q1GIKp50P5AL434Y5rKM6f4=', 'Content-Length': '0'} 2023-09-22T21:45:12.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c727f8c305dd0e23-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.662 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:45:12.663 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:45:12.663 DEBUG:boto:Method: GET 2023-09-22T21:45:12.663 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:45:12.663 DEBUG:boto:Data: 2023-09-22T21:45:12.663 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.663 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.663 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.663 DEBUG:boto:Params: {} 2023-09-22T21:45:12.663 DEBUG:boto:Token: None 2023-09-22T21:45:12.664 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj85 2023-09-22T21:45:12.664 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aTU/q1GIKp50P5AL434Y5rKM6f4= 2023-09-22T21:45:12.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aTU/q1GIKp50P5AL434Y5rKM6f4=', 'Content-Length': '0'} 2023-09-22T21:45:12.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b9ad1a7740b17f2-00650e0ae8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.667 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:12.667 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:12.667 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:45:12.668 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:45:12.668 DEBUG:boto:Method: GET 2023-09-22T21:45:12.668 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:45:12.668 DEBUG:boto:Data: 2023-09-22T21:45:12.668 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.668 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.668 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.668 DEBUG:boto:Params: {} 2023-09-22T21:45:12.668 DEBUG:boto:Token: None 2023-09-22T21:45:12.669 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/ 2023-09-22T21:45:12.669 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0= 2023-09-22T21:45:12.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0=', 'Content-Length': '0'} 2023-09-22T21:45:12.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d61136972fecba5c-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.673 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:12.674 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:12.674 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:12.674 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:45:12.674 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:45:12.674 DEBUG:boto:Method: GET 2023-09-22T21:45:12.674 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:45:12.674 DEBUG:boto:Data: 2023-09-22T21:45:12.674 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.675 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.675 DEBUG:boto:Params: {} 2023-09-22T21:45:12.675 DEBUG:boto:Token: None 2023-09-22T21:45:12.675 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/ 2023-09-22T21:45:12.675 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0= 2023-09-22T21:45:12.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0=', 'Content-Length': '0'} 2023-09-22T21:45:12.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003488d53034a62ab4-00650e0ae8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.684 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:12.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-22T21:45:12.684 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:45:12.685 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:45:12.685 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.685 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.685 DEBUG:boto:Method: GET 2023-09-22T21:45:12.685 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.685 DEBUG:boto:Data: 2023-09-22T21:45:12.685 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.685 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.685 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.686 DEBUG:boto:Params: {} 2023-09-22T21:45:12.686 DEBUG:boto:Token: None 2023-09-22T21:45:12.686 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.686 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YI9xJ6MZ/dWqe0x5sJ+3X3CuaYs= 2023-09-22T21:45:12.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YI9xJ6MZ/dWqe0x5sJ+3X3CuaYs=', 'Content-Length': '0'} 2023-09-22T21:45:12.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df7d7e26d1d7e201-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.730 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:45:12.731 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:45:12.731 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.731 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.731 DEBUG:boto:Method: GET 2023-09-22T21:45:12.731 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.731 DEBUG:boto:Data: 2023-09-22T21:45:12.732 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.732 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.732 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.732 DEBUG:boto:Params: {} 2023-09-22T21:45:12.732 DEBUG:boto:Token: None 2023-09-22T21:45:12.733 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.733 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YI9xJ6MZ/dWqe0x5sJ+3X3CuaYs= 2023-09-22T21:45:12.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YI9xJ6MZ/dWqe0x5sJ+3X3CuaYs=', 'Content-Length': '0'} 2023-09-22T21:45:12.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc3142453b29d8e1-00650e0ae8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.736 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:45:12.736 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:45:12.737 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.737 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.737 DEBUG:boto:Method: HEAD 2023-09-22T21:45:12.737 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.737 DEBUG:boto:Data: 2023-09-22T21:45:12.737 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.738 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.738 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.738 DEBUG:boto:Params: {} 2023-09-22T21:45:12.738 DEBUG:boto:Token: None 2023-09-22T21:45:12.738 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.739 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T21n94u4jxPuWUsjavA26G1YB9M= 2023-09-22T21:45:12.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T21n94u4jxPuWUsjavA26G1YB9M=', 'Content-Length': '0'} 2023-09-22T21:45:12.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a407d59f6cba35e-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.742 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:45:12.742 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:45:12.742 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.742 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.742 DEBUG:boto:Method: HEAD 2023-09-22T21:45:12.742 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.743 DEBUG:boto:Data: 2023-09-22T21:45:12.743 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.743 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.743 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.743 DEBUG:boto:Params: {} 2023-09-22T21:45:12.743 DEBUG:boto:Token: None 2023-09-22T21:45:12.743 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.743 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T21n94u4jxPuWUsjavA26G1YB9M= 2023-09-22T21:45:12.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T21n94u4jxPuWUsjavA26G1YB9M=', 'Content-Length': '0'} 2023-09-22T21:45:12.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064b62f6256ea87df-00650e0ae8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-22T21:45:12.746 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:45:12.747 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:45:12.747 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.747 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.747 DEBUG:boto:Method: GET 2023-09-22T21:45:12.747 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.747 DEBUG:boto:Data: 2023-09-22T21:45:12.747 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.747 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.747 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.747 DEBUG:boto:Params: {} 2023-09-22T21:45:12.748 DEBUG:boto:Token: None 2023-09-22T21:45:12.748 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.748 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YI9xJ6MZ/dWqe0x5sJ+3X3CuaYs= 2023-09-22T21:45:12.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YI9xJ6MZ/dWqe0x5sJ+3X3CuaYs=', 'Content-Length': '0'} 2023-09-22T21:45:12.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f137e5f454d85fbd-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.794 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:45:12.795 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:45:12.795 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.795 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.795 DEBUG:boto:Method: GET 2023-09-22T21:45:12.795 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.795 DEBUG:boto:Data: 2023-09-22T21:45:12.795 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.795 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.795 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.795 DEBUG:boto:Params: {} 2023-09-22T21:45:12.796 DEBUG:boto:Token: None 2023-09-22T21:45:12.796 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:45:12.796 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YI9xJ6MZ/dWqe0x5sJ+3X3CuaYs= 2023-09-22T21:45:12.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YI9xJ6MZ/dWqe0x5sJ+3X3CuaYs=', 'Content-Length': '0'} 2023-09-22T21:45:12.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006925d5bc4150b070-00650e0ae8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.800 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:12.800 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:12.800 DEBUG:boto:path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:45:12.800 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:45:12.800 DEBUG:boto:Method: GET 2023-09-22T21:45:12.800 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj87 2023-09-22T21:45:12.800 DEBUG:boto:Data: 2023-09-22T21:45:12.801 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.801 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.801 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.801 DEBUG:boto:Params: {} 2023-09-22T21:45:12.801 DEBUG:boto:Token: None 2023-09-22T21:45:12.801 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/ 2023-09-22T21:45:12.801 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0= 2023-09-22T21:45:12.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0=', 'Content-Length': '0'} 2023-09-22T21:45:12.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e89e49a43d839ab-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.846 DEBUG:boto:b'ffzdqu-59obj871000falseobj872023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:12.847 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:12.847 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:12.848 DEBUG:boto:path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:45:12.848 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:45:12.848 DEBUG:boto:Method: GET 2023-09-22T21:45:12.848 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj87 2023-09-22T21:45:12.848 DEBUG:boto:Data: 2023-09-22T21:45:12.849 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.849 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.849 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.849 DEBUG:boto:Params: {} 2023-09-22T21:45:12.850 DEBUG:boto:Token: None 2023-09-22T21:45:12.850 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/ 2023-09-22T21:45:12.850 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0= 2023-09-22T21:45:12.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0=', 'Content-Length': '0'} 2023-09-22T21:45:12.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f41d8999adf7a2b-00650e0ae8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.859 DEBUG:boto:b'ffzdqu-59obj871000falseobj872023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:12.860 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:45:12.860 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:45:12.860 DEBUG:boto:Method: HEAD 2023-09-22T21:45:12.860 DEBUG:boto:Path: /ffzdqu-59/obj87 2023-09-22T21:45:12.860 DEBUG:boto:Data: 2023-09-22T21:45:12.860 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.860 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.861 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.861 DEBUG:boto:Params: {} 2023-09-22T21:45:12.861 DEBUG:boto:Token: None 2023-09-22T21:45:12.861 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj87 2023-09-22T21:45:12.861 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m6ZOgELNzvrpz3IJkrtaUUznc8c= 2023-09-22T21:45:12.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m6ZOgELNzvrpz3IJkrtaUUznc8c=', 'Content-Length': '0'} 2023-09-22T21:45:12.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008685a71547241ef-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.864 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:45:12.864 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:45:12.864 DEBUG:boto:Method: HEAD 2023-09-22T21:45:12.864 DEBUG:boto:Path: /ffzdqu-59/obj87 2023-09-22T21:45:12.865 DEBUG:boto:Data: 2023-09-22T21:45:12.865 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.865 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.865 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.865 DEBUG:boto:Params: {} 2023-09-22T21:45:12.865 DEBUG:boto:Token: None 2023-09-22T21:45:12.865 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj87 2023-09-22T21:45:12.865 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m6ZOgELNzvrpz3IJkrtaUUznc8c= 2023-09-22T21:45:12.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m6ZOgELNzvrpz3IJkrtaUUznc8c=', 'Content-Length': '0'} 2023-09-22T21:45:12.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff4bf377a98ea0b1-00650e0ae8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-22T21:45:12.869 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:45:12.869 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:45:12.869 DEBUG:boto:Method: GET 2023-09-22T21:45:12.870 DEBUG:boto:Path: /ffzdqu-59/obj87 2023-09-22T21:45:12.870 DEBUG:boto:Data: 2023-09-22T21:45:12.870 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.870 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.870 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.870 DEBUG:boto:Params: {} 2023-09-22T21:45:12.871 DEBUG:boto:Token: None 2023-09-22T21:45:12.871 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj87 2023-09-22T21:45:12.871 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ogsBwP5E7trP/2eZKPO10d7aGls= 2023-09-22T21:45:12.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ogsBwP5E7trP/2eZKPO10d7aGls=', 'Content-Length': '0'} 2023-09-22T21:45:12.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d9ea341fd67c02c-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.914 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:45:12.915 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:45:12.915 DEBUG:boto:Method: GET 2023-09-22T21:45:12.915 DEBUG:boto:Path: /ffzdqu-59/obj87 2023-09-22T21:45:12.915 DEBUG:boto:Data: 2023-09-22T21:45:12.916 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.916 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.916 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.916 DEBUG:boto:Params: {} 2023-09-22T21:45:12.917 DEBUG:boto:Token: None 2023-09-22T21:45:12.917 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj87 2023-09-22T21:45:12.917 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ogsBwP5E7trP/2eZKPO10d7aGls= 2023-09-22T21:45:12.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ogsBwP5E7trP/2eZKPO10d7aGls=', 'Content-Length': '0'} 2023-09-22T21:45:12.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a634293c3c305703-00650e0ae8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.921 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:12.922 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:12.922 DEBUG:boto:path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:45:12.922 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:45:12.922 DEBUG:boto:Method: GET 2023-09-22T21:45:12.922 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj87 2023-09-22T21:45:12.922 DEBUG:boto:Data: 2023-09-22T21:45:12.922 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.922 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.923 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.923 DEBUG:boto:Params: {} 2023-09-22T21:45:12.923 DEBUG:boto:Token: None 2023-09-22T21:45:12.923 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/ 2023-09-22T21:45:12.923 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0= 2023-09-22T21:45:12.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0=', 'Content-Length': '0'} 2023-09-22T21:45:12.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aaa95e10801a5e83-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.929 DEBUG:boto:b'ffzdqu-59obj871000falseobj872023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:12.929 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:12.929 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:12.929 DEBUG:boto:path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:45:12.929 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:45:12.930 DEBUG:boto:Method: GET 2023-09-22T21:45:12.930 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj87 2023-09-22T21:45:12.930 DEBUG:boto:Data: 2023-09-22T21:45:12.930 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.930 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.930 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.931 DEBUG:boto:Params: {} 2023-09-22T21:45:12.931 DEBUG:boto:Token: None 2023-09-22T21:45:12.931 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/ 2023-09-22T21:45:12.931 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0= 2023-09-22T21:45:12.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gMMmOOKKRyrjNBPKDKnMKmJyeG0=', 'Content-Length': '0'} 2023-09-22T21:45:12.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9c608a9e9b03a79-00650e0ae8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.939 DEBUG:boto:b'ffzdqu-59obj871000falseobj872023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:12.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-22T21:45:12.940 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:45:12.940 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:45:12.940 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:12.941 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:12.941 DEBUG:boto:Method: GET 2023-09-22T21:45:12.941 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:12.941 DEBUG:boto:Data: 2023-09-22T21:45:12.941 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.941 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.941 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.942 DEBUG:boto:Params: {} 2023-09-22T21:45:12.942 DEBUG:boto:Token: None 2023-09-22T21:45:12.942 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:12.942 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XLQD1k482jebKBCpzrwSsgk2jtU= 2023-09-22T21:45:12.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XLQD1k482jebKBCpzrwSsgk2jtU=', 'Content-Length': '0'} 2023-09-22T21:45:12.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f51ae5619d2b003b-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.945 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:45:12.946 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:45:12.946 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:12.946 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:12.946 DEBUG:boto:Method: GET 2023-09-22T21:45:12.946 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:12.946 DEBUG:boto:Data: 2023-09-22T21:45:12.947 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.947 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.947 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.947 DEBUG:boto:Params: {} 2023-09-22T21:45:12.947 DEBUG:boto:Token: None 2023-09-22T21:45:12.947 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:12.948 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XLQD1k482jebKBCpzrwSsgk2jtU= 2023-09-22T21:45:12.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XLQD1k482jebKBCpzrwSsgk2jtU=', 'Content-Length': '0'} 2023-09-22T21:45:12.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5d07e5f7ef2d1e9-00650e0ae8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.951 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:45:12.951 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:45:12.951 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:12.951 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:12.951 DEBUG:boto:Method: HEAD 2023-09-22T21:45:12.951 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:12.951 DEBUG:boto:Data: 2023-09-22T21:45:12.951 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.952 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.952 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.952 DEBUG:boto:Params: {} 2023-09-22T21:45:12.952 DEBUG:boto:Token: None 2023-09-22T21:45:12.952 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:12.952 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KVLwUBtWJPomJrVHRog0plRpDhE= 2023-09-22T21:45:12.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KVLwUBtWJPomJrVHRog0plRpDhE=', 'Content-Length': '0'} 2023-09-22T21:45:12.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5fa6107ba09ec2a-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.955 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:45:12.955 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:45:12.955 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:12.955 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:12.955 DEBUG:boto:Method: HEAD 2023-09-22T21:45:12.955 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:12.956 DEBUG:boto:Data: 2023-09-22T21:45:12.956 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.956 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:12.956 DEBUG:boto:Port: 8000 2023-09-22T21:45:12.956 DEBUG:boto:Params: {} 2023-09-22T21:45:12.956 DEBUG:boto:Token: None 2023-09-22T21:45:12.956 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:12.956 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KVLwUBtWJPomJrVHRog0plRpDhE= 2023-09-22T21:45:12.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KVLwUBtWJPomJrVHRog0plRpDhE=', 'Content-Length': '0'} 2023-09-22T21:45:12.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c20d61a470d87cf0-00650e0ae8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:12.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-22T21:45:12.960 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:45:12.960 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:45:12.960 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:12.960 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:12.960 DEBUG:boto:Method: GET 2023-09-22T21:45:12.960 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:12.960 DEBUG:boto:Data: 2023-09-22T21:45:12.960 DEBUG:boto:Headers: {} 2023-09-22T21:45:12.960 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:12.961 DEBUG:boto:Port: 8001 2023-09-22T21:45:12.961 DEBUG:boto:Params: {} 2023-09-22T21:45:12.961 DEBUG:boto:Token: None 2023-09-22T21:45:12.961 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:12 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:12.961 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XLQD1k482jebKBCpzrwSsgk2jtU= 2023-09-22T21:45:12.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XLQD1k482jebKBCpzrwSsgk2jtU=', 'Content-Length': '0'} 2023-09-22T21:45:12.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052f677a59a79a208-00650e0ae8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.010 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:45:13.011 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:45:13.011 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:13.011 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:13.011 DEBUG:boto:Method: GET 2023-09-22T21:45:13.011 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:13.011 DEBUG:boto:Data: 2023-09-22T21:45:13.011 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.011 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.011 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.012 DEBUG:boto:Params: {} 2023-09-22T21:45:13.012 DEBUG:boto:Token: None 2023-09-22T21:45:13.012 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:45:13.012 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0ZQs+yMExCuMPfNLRg6vv4V0enQ= 2023-09-22T21:45:13.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0ZQs+yMExCuMPfNLRg6vv4V0enQ=', 'Content-Length': '0'} 2023-09-22T21:45:13.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a68d08225012517-00650e0ae9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.016 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:13.016 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:13.016 DEBUG:boto:path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:45:13.016 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:45:13.016 DEBUG:boto:Method: GET 2023-09-22T21:45:13.016 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj92 2023-09-22T21:45:13.017 DEBUG:boto:Data: 2023-09-22T21:45:13.017 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.017 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:13.017 DEBUG:boto:Port: 8001 2023-09-22T21:45:13.017 DEBUG:boto:Params: {} 2023-09-22T21:45:13.017 DEBUG:boto:Token: None 2023-09-22T21:45:13.018 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/ 2023-09-22T21:45:13.018 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GFy+wOcfqmqH1qEH21WOBozTnuY= 2023-09-22T21:45:13.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GFy+wOcfqmqH1qEH21WOBozTnuY=', 'Content-Length': '0'} 2023-09-22T21:45:13.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089c8bddb8d308cd2-00650e0ae9-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.070 DEBUG:boto:b'ffzdqu-59obj921000falseobj922023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:13.071 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:13.072 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:13.072 DEBUG:boto:path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:45:13.072 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:45:13.072 DEBUG:boto:Method: GET 2023-09-22T21:45:13.072 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj92 2023-09-22T21:45:13.073 DEBUG:boto:Data: 2023-09-22T21:45:13.073 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.073 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.073 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.073 DEBUG:boto:Params: {} 2023-09-22T21:45:13.074 DEBUG:boto:Token: None 2023-09-22T21:45:13.074 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/ 2023-09-22T21:45:13.074 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GFy+wOcfqmqH1qEH21WOBozTnuY= 2023-09-22T21:45:13.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GFy+wOcfqmqH1qEH21WOBozTnuY=', 'Content-Length': '0'} 2023-09-22T21:45:13.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000187f01331253cc84-00650e0ae9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.083 DEBUG:boto:b'ffzdqu-59obj921000falseobj922023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:13.084 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:45:13.084 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:45:13.085 DEBUG:boto:Method: HEAD 2023-09-22T21:45:13.085 DEBUG:boto:Path: /ffzdqu-59/obj92 2023-09-22T21:45:13.085 DEBUG:boto:Data: 2023-09-22T21:45:13.085 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.085 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:13.085 DEBUG:boto:Port: 8001 2023-09-22T21:45:13.085 DEBUG:boto:Params: {} 2023-09-22T21:45:13.086 DEBUG:boto:Token: None 2023-09-22T21:45:13.086 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/obj92 2023-09-22T21:45:13.086 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yNAryYwXvR3ihhdzMwZ/a+BlsAc= 2023-09-22T21:45:13.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yNAryYwXvR3ihhdzMwZ/a+BlsAc=', 'Content-Length': '0'} 2023-09-22T21:45:13.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000890c0ee2491c4b71-00650e0ae9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.089 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:45:13.090 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:45:13.090 DEBUG:boto:Method: HEAD 2023-09-22T21:45:13.090 DEBUG:boto:Path: /ffzdqu-59/obj92 2023-09-22T21:45:13.090 DEBUG:boto:Data: 2023-09-22T21:45:13.090 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.090 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.090 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.090 DEBUG:boto:Params: {} 2023-09-22T21:45:13.091 DEBUG:boto:Token: None 2023-09-22T21:45:13.091 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/obj92 2023-09-22T21:45:13.091 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yNAryYwXvR3ihhdzMwZ/a+BlsAc= 2023-09-22T21:45:13.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yNAryYwXvR3ihhdzMwZ/a+BlsAc=', 'Content-Length': '0'} 2023-09-22T21:45:13.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4d10815bb925f05-00650e0ae9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-22T21:45:13.095 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:45:13.095 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:45:13.095 DEBUG:boto:Method: GET 2023-09-22T21:45:13.095 DEBUG:boto:Path: /ffzdqu-59/obj92 2023-09-22T21:45:13.095 DEBUG:boto:Data: 2023-09-22T21:45:13.095 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:13.095 DEBUG:boto:Port: 8001 2023-09-22T21:45:13.095 DEBUG:boto:Params: {} 2023-09-22T21:45:13.095 DEBUG:boto:Token: None 2023-09-22T21:45:13.096 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/obj92 2023-09-22T21:45:13.096 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G/KksWxIoly6DDW+MbDXFz3kUe8= 2023-09-22T21:45:13.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G/KksWxIoly6DDW+MbDXFz3kUe8=', 'Content-Length': '0'} 2023-09-22T21:45:13.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c111838fc56dbf9-00650e0ae9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.142 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:45:13.143 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:45:13.143 DEBUG:boto:Method: GET 2023-09-22T21:45:13.143 DEBUG:boto:Path: /ffzdqu-59/obj92 2023-09-22T21:45:13.143 DEBUG:boto:Data: 2023-09-22T21:45:13.143 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.143 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.143 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.144 DEBUG:boto:Params: {} 2023-09-22T21:45:13.144 DEBUG:boto:Token: None 2023-09-22T21:45:13.144 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/obj92 2023-09-22T21:45:13.144 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G/KksWxIoly6DDW+MbDXFz3kUe8= 2023-09-22T21:45:13.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G/KksWxIoly6DDW+MbDXFz3kUe8=', 'Content-Length': '0'} 2023-09-22T21:45:13.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006977715c1b8a8d04-00650e0ae9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.147 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:13.147 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:13.148 DEBUG:boto:path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:45:13.148 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:45:13.148 DEBUG:boto:Method: GET 2023-09-22T21:45:13.148 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj92 2023-09-22T21:45:13.148 DEBUG:boto:Data: 2023-09-22T21:45:13.148 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.148 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:13.148 DEBUG:boto:Port: 8001 2023-09-22T21:45:13.148 DEBUG:boto:Params: {} 2023-09-22T21:45:13.149 DEBUG:boto:Token: None 2023-09-22T21:45:13.149 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/ 2023-09-22T21:45:13.149 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GFy+wOcfqmqH1qEH21WOBozTnuY= 2023-09-22T21:45:13.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GFy+wOcfqmqH1qEH21WOBozTnuY=', 'Content-Length': '0'} 2023-09-22T21:45:13.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b5e030104b8cc32-00650e0ae9-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.194 DEBUG:boto:b'ffzdqu-59obj921000falseobj922023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:13.195 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:13.195 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:13.195 DEBUG:boto:path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:45:13.195 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:45:13.196 DEBUG:boto:Method: GET 2023-09-22T21:45:13.196 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj92 2023-09-22T21:45:13.196 DEBUG:boto:Data: 2023-09-22T21:45:13.196 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.196 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.196 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.196 DEBUG:boto:Params: {} 2023-09-22T21:45:13.196 DEBUG:boto:Token: None 2023-09-22T21:45:13.197 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/ 2023-09-22T21:45:13.197 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GFy+wOcfqmqH1qEH21WOBozTnuY= 2023-09-22T21:45:13.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GFy+wOcfqmqH1qEH21WOBozTnuY=', 'Content-Length': '0'} 2023-09-22T21:45:13.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000861ca24b6b55e16e-00650e0ae9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.205 DEBUG:boto:b'ffzdqu-59obj921000falseobj922023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:13.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T21:45:13.206 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:45:13.206 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:45:13.206 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.206 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.206 DEBUG:boto:Method: GET 2023-09-22T21:45:13.206 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.206 DEBUG:boto:Data: 2023-09-22T21:45:13.206 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.207 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:13.207 DEBUG:boto:Port: 8001 2023-09-22T21:45:13.207 DEBUG:boto:Params: {} 2023-09-22T21:45:13.207 DEBUG:boto:Token: None 2023-09-22T21:45:13.207 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.207 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DOczAgM4Jwx/6XZZR4apoQVjsRM= 2023-09-22T21:45:13.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DOczAgM4Jwx/6XZZR4apoQVjsRM=', 'Content-Length': '0'} 2023-09-22T21:45:13.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009680cceeea1bf001-00650e0ae9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.211 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:45:13.211 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:45:13.211 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.212 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.212 DEBUG:boto:Method: GET 2023-09-22T21:45:13.212 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.212 DEBUG:boto:Data: 2023-09-22T21:45:13.213 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.213 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.213 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.213 DEBUG:boto:Params: {} 2023-09-22T21:45:13.213 DEBUG:boto:Token: None 2023-09-22T21:45:13.214 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.214 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DOczAgM4Jwx/6XZZR4apoQVjsRM= 2023-09-22T21:45:13.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DOczAgM4Jwx/6XZZR4apoQVjsRM=', 'Content-Length': '0'} 2023-09-22T21:45:13.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008733160d390f5cfa-00650e0ae9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.218 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:45:13.218 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:45:13.218 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.218 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.218 DEBUG:boto:Method: HEAD 2023-09-22T21:45:13.218 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.218 DEBUG:boto:Data: 2023-09-22T21:45:13.218 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.219 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:13.219 DEBUG:boto:Port: 8001 2023-09-22T21:45:13.219 DEBUG:boto:Params: {} 2023-09-22T21:45:13.219 DEBUG:boto:Token: None 2023-09-22T21:45:13.219 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.220 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yvqPDofay4OFU/+I15Rrvy2rSxc= 2023-09-22T21:45:13.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yvqPDofay4OFU/+I15Rrvy2rSxc=', 'Content-Length': '0'} 2023-09-22T21:45:13.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f995a4a738872110-00650e0ae9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.223 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:45:13.223 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:45:13.223 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.223 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.223 DEBUG:boto:Method: HEAD 2023-09-22T21:45:13.223 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.223 DEBUG:boto:Data: 2023-09-22T21:45:13.223 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.223 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.224 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.224 DEBUG:boto:Params: {} 2023-09-22T21:45:13.224 DEBUG:boto:Token: None 2023-09-22T21:45:13.224 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.224 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yvqPDofay4OFU/+I15Rrvy2rSxc= 2023-09-22T21:45:13.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yvqPDofay4OFU/+I15Rrvy2rSxc=', 'Content-Length': '0'} 2023-09-22T21:45:13.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046fab4a8ceebc19b-00650e0ae9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T21:45:13.227 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:45:13.227 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:45:13.228 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.228 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.228 DEBUG:boto:Method: GET 2023-09-22T21:45:13.228 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.228 DEBUG:boto:Data: 2023-09-22T21:45:13.228 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.228 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:13.228 DEBUG:boto:Port: 8001 2023-09-22T21:45:13.228 DEBUG:boto:Params: {} 2023-09-22T21:45:13.228 DEBUG:boto:Token: None 2023-09-22T21:45:13.229 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.229 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DOczAgM4Jwx/6XZZR4apoQVjsRM= 2023-09-22T21:45:13.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DOczAgM4Jwx/6XZZR4apoQVjsRM=', 'Content-Length': '0'} 2023-09-22T21:45:13.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f91ebb9abcf6855-00650e0ae9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.274 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:45:13.275 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:45:13.275 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.275 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.275 DEBUG:boto:Method: GET 2023-09-22T21:45:13.275 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.275 DEBUG:boto:Data: 2023-09-22T21:45:13.275 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.275 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.275 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.275 DEBUG:boto:Params: {} 2023-09-22T21:45:13.276 DEBUG:boto:Token: None 2023-09-22T21:45:13.276 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:45:13.276 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DOczAgM4Jwx/6XZZR4apoQVjsRM= 2023-09-22T21:45:13.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DOczAgM4Jwx/6XZZR4apoQVjsRM=', 'Content-Length': '0'} 2023-09-22T21:45:13.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061cab4f37f329570-00650e0ae9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.279 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:13.279 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:13.279 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:45:13.280 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:45:13.280 DEBUG:boto:Method: GET 2023-09-22T21:45:13.280 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:45:13.280 DEBUG:boto:Data: 2023-09-22T21:45:13.280 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.280 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:13.280 DEBUG:boto:Port: 8001 2023-09-22T21:45:13.280 DEBUG:boto:Params: {} 2023-09-22T21:45:13.281 DEBUG:boto:Token: None 2023-09-22T21:45:13.281 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/ 2023-09-22T21:45:13.281 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GFy+wOcfqmqH1qEH21WOBozTnuY= 2023-09-22T21:45:13.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GFy+wOcfqmqH1qEH21WOBozTnuY=', 'Content-Length': '0'} 2023-09-22T21:45:13.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066413ee5720d4ff8-00650e0ae9-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.286 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:13.286 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:13.287 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:13.287 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:45:13.287 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:45:13.287 DEBUG:boto:Method: GET 2023-09-22T21:45:13.287 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:45:13.287 DEBUG:boto:Data: 2023-09-22T21:45:13.287 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.287 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.287 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.288 DEBUG:boto:Params: {} 2023-09-22T21:45:13.288 DEBUG:boto:Token: None 2023-09-22T21:45:13.288 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/ 2023-09-22T21:45:13.288 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GFy+wOcfqmqH1qEH21WOBozTnuY= 2023-09-22T21:45:13.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GFy+wOcfqmqH1qEH21WOBozTnuY=', 'Content-Length': '0'} 2023-09-22T21:45:13.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043c1f214d4f1c385-00650e0ae9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.296 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:13.297 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:45:13.297 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:45:13.297 DEBUG:boto:Method: HEAD 2023-09-22T21:45:13.297 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:45:13.297 DEBUG:boto:Data: 2023-09-22T21:45:13.297 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.297 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:13.297 DEBUG:boto:Port: 8001 2023-09-22T21:45:13.297 DEBUG:boto:Params: {} 2023-09-22T21:45:13.298 DEBUG:boto:Token: None 2023-09-22T21:45:13.298 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/obj95 2023-09-22T21:45:13.298 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:L/qrSB05YgXOwPfWJWmGDoak7/E= 2023-09-22T21:45:13.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:L/qrSB05YgXOwPfWJWmGDoak7/E=', 'Content-Length': '0'} 2023-09-22T21:45:13.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4bf0c9962aea566-00650e0ae9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.301 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:45:13.301 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:45:13.301 DEBUG:boto:Method: HEAD 2023-09-22T21:45:13.301 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:45:13.302 DEBUG:boto:Data: 2023-09-22T21:45:13.302 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.302 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.302 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.302 DEBUG:boto:Params: {} 2023-09-22T21:45:13.302 DEBUG:boto:Token: None 2023-09-22T21:45:13.302 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/obj95 2023-09-22T21:45:13.303 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:L/qrSB05YgXOwPfWJWmGDoak7/E= 2023-09-22T21:45:13.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:L/qrSB05YgXOwPfWJWmGDoak7/E=', 'Content-Length': '0'} 2023-09-22T21:45:13.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d9bdaacaff4d190-00650e0ae9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T21:45:13.306 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:45:13.307 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:45:13.307 DEBUG:boto:Method: GET 2023-09-22T21:45:13.307 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:45:13.307 DEBUG:boto:Data: 2023-09-22T21:45:13.307 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.308 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:13.308 DEBUG:boto:Port: 8001 2023-09-22T21:45:13.308 DEBUG:boto:Params: {} 2023-09-22T21:45:13.308 DEBUG:boto:Token: None 2023-09-22T21:45:13.308 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/obj95 2023-09-22T21:45:13.309 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AIOcUND2CIYXf3+IdLK2ohurcNc= 2023-09-22T21:45:13.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AIOcUND2CIYXf3+IdLK2ohurcNc=', 'Content-Length': '0'} 2023-09-22T21:45:13.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032d35ade5898d73f-00650e0ae9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.354 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:45:13.355 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:45:13.355 DEBUG:boto:Method: GET 2023-09-22T21:45:13.355 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:45:13.355 DEBUG:boto:Data: 2023-09-22T21:45:13.355 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.355 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.355 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.355 DEBUG:boto:Params: {} 2023-09-22T21:45:13.356 DEBUG:boto:Token: None 2023-09-22T21:45:13.356 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/obj95 2023-09-22T21:45:13.356 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AIOcUND2CIYXf3+IdLK2ohurcNc= 2023-09-22T21:45:13.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AIOcUND2CIYXf3+IdLK2ohurcNc=', 'Content-Length': '0'} 2023-09-22T21:45:13.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000744cb5cf5f377c04-00650e0ae9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.359 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:13.359 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:13.359 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:45:13.359 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:45:13.359 DEBUG:boto:Method: GET 2023-09-22T21:45:13.359 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:45:13.360 DEBUG:boto:Data: 2023-09-22T21:45:13.360 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.360 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:45:13.360 DEBUG:boto:Port: 8001 2023-09-22T21:45:13.360 DEBUG:boto:Params: {} 2023-09-22T21:45:13.360 DEBUG:boto:Token: None 2023-09-22T21:45:13.360 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/ 2023-09-22T21:45:13.360 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GFy+wOcfqmqH1qEH21WOBozTnuY= 2023-09-22T21:45:13.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GFy+wOcfqmqH1qEH21WOBozTnuY=', 'Content-Length': '0'} 2023-09-22T21:45:13.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b608fbd13777c6f8-00650e0ae9-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.410 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:13.411 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:13.411 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:13.411 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:45:13.411 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:45:13.411 DEBUG:boto:Method: GET 2023-09-22T21:45:13.411 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:45:13.411 DEBUG:boto:Data: 2023-09-22T21:45:13.412 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.412 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.412 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.412 DEBUG:boto:Params: {} 2023-09-22T21:45:13.412 DEBUG:boto:Token: None 2023-09-22T21:45:13.412 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/ 2023-09-22T21:45:13.412 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GFy+wOcfqmqH1qEH21WOBozTnuY= 2023-09-22T21:45:13.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GFy+wOcfqmqH1qEH21WOBozTnuY=', 'Content-Length': '0'} 2023-09-22T21:45:13.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae115a0932196156-00650e0ae9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.421 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:45:13.422 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-59 zones={a2, a1} 2023-09-22T21:45:13.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:45:13.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:45:13.422 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:45:13.469 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:45:13.470 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:45:13.539 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:45:13.539 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-59 2023-09-22T21:45:13.557 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 25 2023-09-22T21:45:13.635 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:13.636 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:13.636 DEBUG:boto:Method: HEAD 2023-09-22T21:45:13.636 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:45:13.636 DEBUG:boto:Data: 2023-09-22T21:45:13.636 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.636 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.636 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.637 DEBUG:boto:Params: {} 2023-09-22T21:45:13.637 DEBUG:boto:Token: None 2023-09-22T21:45:13.637 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/ 2023-09-22T21:45:13.637 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5NQgxsxsDNFzqUpafbsPJP5USGU= 2023-09-22T21:45:13.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5NQgxsxsDNFzqUpafbsPJP5USGU=', 'Content-Length': '0'} 2023-09-22T21:45:13.641 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd6b5d87b1e870cc-00650e0ae9-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.642 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:45:13.642 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:45:13.642 DEBUG:boto:Method: PUT 2023-09-22T21:45:13.642 DEBUG:boto:Path: /ffzdqu-59/obj31 2023-09-22T21:45:13.642 DEBUG:boto:Data: 2023-09-22T21:45:13.642 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:45:13.642 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.642 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.643 DEBUG:boto:Params: {} 2023-09-22T21:45:13.643 DEBUG:boto:Token: None 2023-09-22T21:45:13.643 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/obj31 2023-09-22T21:45:13.643 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8dBUiWkpN0TCjON+dNNMtilJYAU= 2023-09-22T21:45:13.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8dBUiWkpN0TCjON+dNNMtilJYAU='} 2023-09-22T21:45:13.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005c0c97a50d3f2d0-00650e0ae9-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.650 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:13.650 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:13.650 DEBUG:boto:Method: HEAD 2023-09-22T21:45:13.650 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:45:13.650 DEBUG:boto:Data: 2023-09-22T21:45:13.650 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.651 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.651 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.651 DEBUG:boto:Params: {} 2023-09-22T21:45:13.651 DEBUG:boto:Token: None 2023-09-22T21:45:13.651 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/ 2023-09-22T21:45:13.651 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5NQgxsxsDNFzqUpafbsPJP5USGU= 2023-09-22T21:45:13.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5NQgxsxsDNFzqUpafbsPJP5USGU=', 'Content-Length': '0'} 2023-09-22T21:45:13.655 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dca44d0b5aee9aa2-00650e0ae9-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.655 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:45:13.655 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:45:13.656 DEBUG:boto:Method: PUT 2023-09-22T21:45:13.656 DEBUG:boto:Path: /ffzdqu-59/obj62 2023-09-22T21:45:13.656 DEBUG:boto:Data: 2023-09-22T21:45:13.656 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:45:13.656 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.656 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.656 DEBUG:boto:Params: {} 2023-09-22T21:45:13.656 DEBUG:boto:Token: None 2023-09-22T21:45:13.656 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/obj62 2023-09-22T21:45:13.657 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4jXwqd5tEO9eGmgZRAZLE1136vY= 2023-09-22T21:45:13.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4jXwqd5tEO9eGmgZRAZLE1136vY='} 2023-09-22T21:45:13.702 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a35241534d370fb4-00650e0ae9-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.703 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:13.703 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:13.703 DEBUG:boto:Method: HEAD 2023-09-22T21:45:13.703 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:45:13.703 DEBUG:boto:Data: 2023-09-22T21:45:13.703 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.703 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.704 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.704 DEBUG:boto:Params: {} 2023-09-22T21:45:13.704 DEBUG:boto:Token: None 2023-09-22T21:45:13.704 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/ 2023-09-22T21:45:13.704 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5NQgxsxsDNFzqUpafbsPJP5USGU= 2023-09-22T21:45:13.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5NQgxsxsDNFzqUpafbsPJP5USGU=', 'Content-Length': '0'} 2023-09-22T21:45:13.708 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000729d5e0e28a017da-00650e0ae9-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.708 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:45:13.708 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:45:13.709 DEBUG:boto:Method: PUT 2023-09-22T21:45:13.709 DEBUG:boto:Path: /ffzdqu-59/obj93 2023-09-22T21:45:13.709 DEBUG:boto:Data: 2023-09-22T21:45:13.709 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:45:13.709 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.709 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.709 DEBUG:boto:Params: {} 2023-09-22T21:45:13.709 DEBUG:boto:Token: None 2023-09-22T21:45:13.709 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/obj93 2023-09-22T21:45:13.710 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IPNlEK55OA8MxGy01a7VIdovU9w= 2023-09-22T21:45:13.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IPNlEK55OA8MxGy01a7VIdovU9w='} 2023-09-22T21:45:13.717 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095038914622d81d1-00650e0ae9-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.717 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:13.717 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:13.718 DEBUG:boto:Method: HEAD 2023-09-22T21:45:13.718 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:45:13.718 DEBUG:boto:Data: 2023-09-22T21:45:13.718 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.718 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.718 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.718 DEBUG:boto:Params: {} 2023-09-22T21:45:13.718 DEBUG:boto:Token: None 2023-09-22T21:45:13.718 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/ 2023-09-22T21:45:13.719 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5NQgxsxsDNFzqUpafbsPJP5USGU= 2023-09-22T21:45:13.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5NQgxsxsDNFzqUpafbsPJP5USGU=', 'Content-Length': '0'} 2023-09-22T21:45:13.723 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2dd2ab040c91b1e-00650e0ae9-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.723 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:45:13.723 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:45:13.723 DEBUG:boto:Method: PUT 2023-09-22T21:45:13.724 DEBUG:boto:Path: /ffzdqu-59/obj124 2023-09-22T21:45:13.724 DEBUG:boto:Data: 2023-09-22T21:45:13.724 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:45:13.724 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.724 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.724 DEBUG:boto:Params: {} 2023-09-22T21:45:13.724 DEBUG:boto:Token: None 2023-09-22T21:45:13.724 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/obj124 2023-09-22T21:45:13.724 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0Q4GcTwyA3eUmYQaFEZVxDO60oA= 2023-09-22T21:45:13.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0Q4GcTwyA3eUmYQaFEZVxDO60oA='} 2023-09-22T21:45:13.770 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a83b6bd1095e67a-00650e0ae9-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.771 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:45:13.771 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:45:13.771 DEBUG:boto:Method: HEAD 2023-09-22T21:45:13.771 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:45:13.771 DEBUG:boto:Data: 2023-09-22T21:45:13.771 DEBUG:boto:Headers: {} 2023-09-22T21:45:13.771 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.772 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.772 DEBUG:boto:Params: {} 2023-09-22T21:45:13.772 DEBUG:boto:Token: None 2023-09-22T21:45:13.772 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/ 2023-09-22T21:45:13.772 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5NQgxsxsDNFzqUpafbsPJP5USGU= 2023-09-22T21:45:13.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5NQgxsxsDNFzqUpafbsPJP5USGU=', 'Content-Length': '0'} 2023-09-22T21:45:13.778 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ed3c8e629061557-00650e0ae9-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.778 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:45:13.779 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:45:13.779 DEBUG:boto:Method: PUT 2023-09-22T21:45:13.779 DEBUG:boto:Path: /ffzdqu-59/obj155 2023-09-22T21:45:13.779 DEBUG:boto:Data: 2023-09-22T21:45:13.779 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:45:13.779 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:45:13.779 DEBUG:boto:Port: 8000 2023-09-22T21:45:13.779 DEBUG:boto:Params: {} 2023-09-22T21:45:13.780 DEBUG:boto:Token: None 2023-09-22T21:45:13.780 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:45:13 GMT /ffzdqu-59/obj155 2023-09-22T21:45:13.780 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PfS1qzd70xmNBmJDMf+ITXU2aTI= 2023-09-22T21:45:13.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:45:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PfS1qzd70xmNBmJDMf+ITXU2aTI='} 2023-09-22T21:45:13.789 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4ebbeb8cec147c7-00650e0ae9-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:45:13.790 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-59 2023-09-22T21:45:13.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:45:13.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:45:13.790 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:45:13.843 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:45:13.843 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:45:13.892 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:45:13.891+0000 7f444f94ba40 1 waiting to reach latest gen 5, on 4.. 2023-09-22T21:45:18.893 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:45:18.891+0000 7f444f94ba40 1 waiting to reach latest gen 5, on 4.. 2023-09-22T21:45:23.894 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:45:23.891+0000 7f444f94ba40 1 waiting to reach latest gen 5, on 4.. 2023-09-22T21:45:28.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:45:28.891+0000 7f444f94ba40 1 waiting to reach latest gen 5, on 4.. 2023-09-22T21:45:33.900 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:45:33.899+0000 7f444f94ba40 1 waiting for incremental sync to catch up: 2023-09-22T21:45:33.901 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , 00000000001.69.6, , , , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , ] 2023-09-22T21:45:33.901 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.167.6, , 00000000001.69.6, , 00000000001.101.6, , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-09-22T21:45:38.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:45:38.903+0000 7f444f94ba40 1 waiting for incremental sync to catch up: 2023-09-22T21:45:38.905 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , 00000000001.69.6, , , , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , ] 2023-09-22T21:45:38.905 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.167.6, , 00000000001.69.6, , 00000000001.101.6, , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-09-22T21:45:43.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:45:43.907+0000 7f444f94ba40 1 waiting for incremental sync to catch up: 2023-09-22T21:45:43.909 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , 00000000001.69.6, , , , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , ] 2023-09-22T21:45:43.909 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.167.6, , 00000000001.69.6, , 00000000001.101.6, , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-09-22T21:45:48.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:45:48.911+0000 7f444f94ba40 1 waiting for incremental sync to catch up: 2023-09-22T21:45:48.913 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , 00000000001.69.6, , , , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , ] 2023-09-22T21:45:48.913 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.167.6, , 00000000001.69.6, , 00000000001.101.6, , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-09-22T21:45:53.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:45:53.915+0000 7f444f94ba40 1 waiting for incremental sync to catch up: 2023-09-22T21:45:53.917 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , 00000000001.69.6, , , , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , ] 2023-09-22T21:45:53.917 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.167.6, , 00000000001.69.6, , 00000000001.101.6, , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-09-22T21:45:58.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:45:58.919+0000 7f444f94ba40 1 waiting for incremental sync to catch up: 2023-09-22T21:45:58.978 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , 00000000001.69.6, , , , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , ] 2023-09-22T21:45:58.978 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.167.6, , 00000000001.69.6, , 00000000001.101.6, , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-09-22T21:46:03.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:46:03.923+0000 7f444f94ba40 1 waiting for incremental sync to catch up: 2023-09-22T21:46:03.925 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , 00000000001.69.6, , , , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , ] 2023-09-22T21:46:03.925 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.167.6, , 00000000001.69.6, , 00000000001.101.6, , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-09-22T21:46:08.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:46:08.927+0000 7f444f94ba40 1 waiting for incremental sync to catch up: 2023-09-22T21:46:08.930 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , 00000000001.69.6, , , , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , ] 2023-09-22T21:46:08.930 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.167.6, , 00000000001.69.6, , 00000000001.101.6, , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-09-22T21:46:13.934 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:46:13.930+0000 7f444f94ba40 1 waiting for incremental sync to catch up: 2023-09-22T21:46:13.934 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , 00000000001.69.6, , , , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , ] 2023-09-22T21:46:13.934 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.167.6, , 00000000001.69.6, , 00000000001.101.6, , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-09-22T21:46:18.938 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:46:18.934+0000 7f444f94ba40 1 waiting for incremental sync to catch up: 2023-09-22T21:46:18.938 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , 00000000001.69.6, , , , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , ] 2023-09-22T21:46:18.939 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.167.6, , 00000000001.69.6, , 00000000001.101.6, , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-09-22T21:46:23.943 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:46:23.938+0000 7f444f94ba40 1 waiting for incremental sync to catch up: 2023-09-22T21:46:23.943 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , 00000000001.69.6, , , , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , ] 2023-09-22T21:46:23.943 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.167.6, , 00000000001.69.6, , 00000000001.101.6, , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-09-22T21:46:28.947 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:46:28.942+0000 7f444f94ba40 1 waiting for incremental sync to catch up: 2023-09-22T21:46:28.947 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , 00000000001.69.6, , , , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , ] 2023-09-22T21:46:28.948 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.167.6, , 00000000001.69.6, , 00000000001.101.6, , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-09-22T21:46:33.951 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:46:33.946+0000 7f444f94ba40 1 waiting for incremental sync to catch up: 2023-09-22T21:46:33.952 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , 00000000001.69.6, , , , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , ] 2023-09-22T21:46:33.952 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.167.6, , 00000000001.69.6, , 00000000001.101.6, , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-09-22T21:46:38.955 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:46:38.950+0000 7f444f94ba40 1 waiting for incremental sync to catch up: 2023-09-22T21:46:38.956 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , 00000000001.69.6, , , , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , ] 2023-09-22T21:46:38.956 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.167.6, , 00000000001.69.6, , 00000000001.101.6, , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-09-22T21:46:43.959 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:46:43.954+0000 7f444f94ba40 1 waiting for incremental sync to catch up: 2023-09-22T21:46:43.959 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , 00000000001.69.6, , , , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , ] 2023-09-22T21:46:43.960 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.167.6, , 00000000001.69.6, , 00000000001.101.6, , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-09-22T21:46:48.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:46:48.958+0000 7f444f94ba40 1 waiting for incremental sync to catch up: 2023-09-22T21:46:48.963 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , 00000000001.69.6, , , , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , ] 2023-09-22T21:46:48.963 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.167.6, , 00000000001.69.6, , 00000000001.101.6, , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-09-22T21:46:52.167 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:46:52.162+0000 7f092e9f8640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:46:53.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:46:53.962+0000 7f444f94ba40 1 bucket sync caught up with source: 2023-09-22T21:46:53.967 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.167.6, , 00000000001.69.6, , 00000000001.101.6, , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-09-22T21:46:53.968 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.167.6, , 00000000001.69.6, , 00000000001.101.6, , , , , , , , , , 00000000001.102.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-09-22T21:46:53.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:46:53.962+0000 7f444f94ba40 0 bucket checkpoint complete 2023-09-22T21:46:53.999 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-59 2023-09-22T21:46:53.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:46:53.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:46:53.999 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:46:54.047 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:46:54.048 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:46:54.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:46:54.086+0000 7fc6aa173a40 1 waiting for incremental sync to catch up: 2023-09-22T21:46:54.090 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.301.6, 00000000002.265.6, 00000000002.161.6, 00000000003.284.6, 00000000004.417.6, 00000000001.415.6, 00000000001.228.6, 00000000003.154.6, 00000000005.162.6, 00000000004.310.6] 2023-09-22T21:46:54.091 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.318.6, 00000000002.265.6, 00000000002.161.6, 00000000004.291.6, 00000000004.417.6, 00000000001.415.6, 00000000001.228.6, 00000000003.154.6, 00000000006.177.6, 00000000004.310.6] 2023-09-22T21:46:59.092 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:46:59.090+0000 7fc6aa173a40 1 waiting for incremental sync to catch up: 2023-09-22T21:46:59.093 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.301.6, 00000000002.265.6, 00000000002.161.6, 00000000004.291.6, 00000000004.417.6, 00000000001.415.6, 00000000001.228.6, 00000000003.154.6, 00000000005.162.6, 00000000004.310.6] 2023-09-22T21:46:59.093 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.318.6, 00000000002.265.6, 00000000002.161.6, 00000000004.291.6, 00000000004.417.6, 00000000001.415.6, 00000000001.228.6, 00000000003.154.6, 00000000006.177.6, 00000000004.310.6] 2023-09-22T21:47:04.094 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:47:04.090+0000 7fc6aa173a40 1 waiting for incremental sync to catch up: 2023-09-22T21:47:04.095 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.301.6, 00000000002.265.6, 00000000002.161.6, 00000000004.291.6, 00000000004.417.6, 00000000001.415.6, 00000000001.228.6, 00000000003.154.6, 00000000005.162.6, 00000000004.310.6] 2023-09-22T21:47:04.095 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.318.6, 00000000002.265.6, 00000000002.161.6, 00000000004.291.6, 00000000004.417.6, 00000000001.415.6, 00000000001.228.6, 00000000003.154.6, 00000000006.177.6, 00000000004.310.6] 2023-09-22T21:47:09.096 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:47:09.094+0000 7fc6aa173a40 1 bucket sync caught up with source: 2023-09-22T21:47:09.097 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.318.6, 00000000002.265.6, 00000000002.161.6, 00000000004.291.6, 00000000004.417.6, 00000000001.415.6, 00000000001.228.6, 00000000003.154.6, 00000000006.177.6, 00000000004.310.6] 2023-09-22T21:47:09.097 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.318.6, 00000000002.265.6, 00000000002.161.6, 00000000004.291.6, 00000000004.417.6, 00000000001.415.6, 00000000001.228.6, 00000000003.154.6, 00000000006.177.6, 00000000004.310.6] 2023-09-22T21:47:09.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:47:09.094+0000 7fc6aa173a40 0 bucket checkpoint complete 2023-09-22T21:47:09.124 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-59 zones={a2, a1} 2023-09-22T21:47:09.124 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:09.125 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:09.125 DEBUG:boto:Method: HEAD 2023-09-22T21:47:09.125 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:47:09.125 DEBUG:boto:Data: 2023-09-22T21:47:09.125 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.125 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.125 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.125 DEBUG:boto:Params: {} 2023-09-22T21:47:09.126 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:47:09.126 DEBUG:boto:Token: None 2023-09-22T21:47:09.126 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/ 2023-09-22T21:47:09.126 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QRL/069qASa37uSsHwiZhq/Wg8A= 2023-09-22T21:47:09.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QRL/069qASa37uSsHwiZhq/Wg8A=', 'Content-Length': '0'} 2023-09-22T21:47:09.131 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077b028c3bdc9d7ab-00650e0b5d-1b9c-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.132 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:09.132 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:09.132 DEBUG:boto:Method: HEAD 2023-09-22T21:47:09.132 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:47:09.132 DEBUG:boto:Data: 2023-09-22T21:47:09.229 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.229 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.229 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.230 DEBUG:boto:Params: {} 2023-09-22T21:47:09.230 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:47:09.230 DEBUG:boto:Token: None 2023-09-22T21:47:09.230 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/ 2023-09-22T21:47:09.230 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QRL/069qASa37uSsHwiZhq/Wg8A= 2023-09-22T21:47:09.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QRL/069qASa37uSsHwiZhq/Wg8A=', 'Content-Length': '0'} 2023-09-22T21:47:09.238 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000364e6ff9d5fa0744-00650e0b5d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.239 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:47:09.239 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:09.239 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:09.239 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:47:09.239 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:47:09.239 DEBUG:boto:Method: GET 2023-09-22T21:47:09.239 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:47:09.239 DEBUG:boto:Data: 2023-09-22T21:47:09.239 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.240 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.240 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.240 DEBUG:boto:Params: {} 2023-09-22T21:47:09.240 DEBUG:boto:Token: None 2023-09-22T21:47:09.240 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/?versions 2023-09-22T21:47:09.240 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VeQo0IDPDtiin4SUCTeRUhI0D6E= 2023-09-22T21:47:09.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VeQo0IDPDtiin4SUCTeRUhI0D6E=', 'Content-Length': '0'} 2023-09-22T21:47:09.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002661f08936664d44-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.246 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:09.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T21:47:09.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-22T21:47:09.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-22T21:47:09.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-09-22T21:47:09.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-22T21:47:09.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-09-22T21:47:09.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-22T21:47:09.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-22T21:47:09.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T21:47:09.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-22T21:47:09.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-22T21:47:09.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-09-22T21:47:09.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T21:47:09.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-22T21:47:09.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-22T21:47:09.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T21:47:09.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-22T21:47:09.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-22T21:47:09.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T21:47:09.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-22T21:47:09.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-22T21:47:09.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-09-22T21:47:09.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-22T21:47:09.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-22T21:47:09.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-22T21:47:09.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-22T21:47:09.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-22T21:47:09.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-22T21:47:09.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-09-22T21:47:09.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-22T21:47:09.251 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:47:09.251 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:09.251 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:09.251 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:47:09.251 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:47:09.251 DEBUG:boto:Method: GET 2023-09-22T21:47:09.251 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:47:09.251 DEBUG:boto:Data: 2023-09-22T21:47:09.251 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.251 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.252 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.252 DEBUG:boto:Params: {} 2023-09-22T21:47:09.252 DEBUG:boto:Token: None 2023-09-22T21:47:09.252 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/?versions 2023-09-22T21:47:09.252 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VeQo0IDPDtiin4SUCTeRUhI0D6E= 2023-09-22T21:47:09.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VeQo0IDPDtiin4SUCTeRUhI0D6E=', 'Content-Length': '0'} 2023-09-22T21:47:09.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4bf225b6dd77720-00650e0b5d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.261 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:09.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T21:47:09.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-22T21:47:09.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-22T21:47:09.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-09-22T21:47:09.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-22T21:47:09.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-09-22T21:47:09.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-22T21:47:09.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-22T21:47:09.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T21:47:09.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-22T21:47:09.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-22T21:47:09.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-09-22T21:47:09.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T21:47:09.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-22T21:47:09.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-22T21:47:09.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T21:47:09.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-22T21:47:09.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-22T21:47:09.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T21:47:09.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-22T21:47:09.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-22T21:47:09.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-09-22T21:47:09.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-22T21:47:09.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-22T21:47:09.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-22T21:47:09.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-22T21:47:09.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-22T21:47:09.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-22T21:47:09.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-09-22T21:47:09.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-22T21:47:09.265 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:09.265 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:09.265 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:47:09.265 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:47:09.266 DEBUG:boto:Method: GET 2023-09-22T21:47:09.266 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:47:09.266 DEBUG:boto:Data: 2023-09-22T21:47:09.266 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.266 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.266 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.266 DEBUG:boto:Params: {} 2023-09-22T21:47:09.266 DEBUG:boto:Token: None 2023-09-22T21:47:09.266 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/?versions 2023-09-22T21:47:09.267 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VeQo0IDPDtiin4SUCTeRUhI0D6E= 2023-09-22T21:47:09.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VeQo0IDPDtiin4SUCTeRUhI0D6E=', 'Content-Length': '0'} 2023-09-22T21:47:09.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cca5f06b9201e3fa-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.272 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:09.274 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:09.274 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:09.274 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:47:09.274 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:47:09.274 DEBUG:boto:Method: GET 2023-09-22T21:47:09.274 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:47:09.274 DEBUG:boto:Data: 2023-09-22T21:47:09.274 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.275 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.275 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.275 DEBUG:boto:Params: {} 2023-09-22T21:47:09.275 DEBUG:boto:Token: None 2023-09-22T21:47:09.275 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/?versions 2023-09-22T21:47:09.275 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VeQo0IDPDtiin4SUCTeRUhI0D6E= 2023-09-22T21:47:09.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VeQo0IDPDtiin4SUCTeRUhI0D6E=', 'Content-Length': '0'} 2023-09-22T21:47:09.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd3b026be54fcd1a-00650e0b5d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.284 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:09.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:47:09.286 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:47:09.286 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:47:09.286 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.286 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.286 DEBUG:boto:Method: GET 2023-09-22T21:47:09.286 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.287 DEBUG:boto:Data: 2023-09-22T21:47:09.287 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.287 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.287 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.287 DEBUG:boto:Params: {} 2023-09-22T21:47:09.287 DEBUG:boto:Token: None 2023-09-22T21:47:09.287 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.287 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:j1x4EXZQlCzM4MGPM08kTxBj7Ks= 2023-09-22T21:47:09.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:j1x4EXZQlCzM4MGPM08kTxBj7Ks=', 'Content-Length': '0'} 2023-09-22T21:47:09.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000122a6f3d7730f48f-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.291 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:47:09.291 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:47:09.291 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.291 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.291 DEBUG:boto:Method: GET 2023-09-22T21:47:09.291 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.291 DEBUG:boto:Data: 2023-09-22T21:47:09.291 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.292 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.292 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.292 DEBUG:boto:Params: {} 2023-09-22T21:47:09.292 DEBUG:boto:Token: None 2023-09-22T21:47:09.292 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.292 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:j1x4EXZQlCzM4MGPM08kTxBj7Ks= 2023-09-22T21:47:09.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:j1x4EXZQlCzM4MGPM08kTxBj7Ks=', 'Content-Length': '0'} 2023-09-22T21:47:09.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000844a4aa241accc10-00650e0b5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.338 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:47:09.338 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:47:09.339 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.339 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.339 DEBUG:boto:Method: HEAD 2023-09-22T21:47:09.339 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.339 DEBUG:boto:Data: 2023-09-22T21:47:09.339 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.339 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.339 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.339 DEBUG:boto:Params: {} 2023-09-22T21:47:09.339 DEBUG:boto:Token: None 2023-09-22T21:47:09.340 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.340 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iOyavk09vPBpIpLPI7fvB3n/uGk= 2023-09-22T21:47:09.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iOyavk09vPBpIpLPI7fvB3n/uGk=', 'Content-Length': '0'} 2023-09-22T21:47:09.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0dd6040684196c0-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.343 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:47:09.343 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:47:09.343 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.343 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.343 DEBUG:boto:Method: HEAD 2023-09-22T21:47:09.343 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.343 DEBUG:boto:Data: 2023-09-22T21:47:09.343 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.343 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.343 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.344 DEBUG:boto:Params: {} 2023-09-22T21:47:09.344 DEBUG:boto:Token: None 2023-09-22T21:47:09.344 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.344 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iOyavk09vPBpIpLPI7fvB3n/uGk= 2023-09-22T21:47:09.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iOyavk09vPBpIpLPI7fvB3n/uGk=', 'Content-Length': '0'} 2023-09-22T21:47:09.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006f2adf2b129a85f-00650e0b5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:47:09.347 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:47:09.347 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:47:09.347 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.347 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.348 DEBUG:boto:Method: GET 2023-09-22T21:47:09.348 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.348 DEBUG:boto:Data: 2023-09-22T21:47:09.348 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.348 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.348 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.348 DEBUG:boto:Params: {} 2023-09-22T21:47:09.348 DEBUG:boto:Token: None 2023-09-22T21:47:09.348 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.349 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:j1x4EXZQlCzM4MGPM08kTxBj7Ks= 2023-09-22T21:47:09.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:j1x4EXZQlCzM4MGPM08kTxBj7Ks=', 'Content-Length': '0'} 2023-09-22T21:47:09.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1f42865c1425126-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.394 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:47:09.394 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:47:09.395 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.395 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.395 DEBUG:boto:Method: GET 2023-09-22T21:47:09.395 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.395 DEBUG:boto:Data: 2023-09-22T21:47:09.395 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.395 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.395 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.395 DEBUG:boto:Params: {} 2023-09-22T21:47:09.396 DEBUG:boto:Token: None 2023-09-22T21:47:09.396 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:47:09.396 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:j1x4EXZQlCzM4MGPM08kTxBj7Ks= 2023-09-22T21:47:09.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:j1x4EXZQlCzM4MGPM08kTxBj7Ks=', 'Content-Length': '0'} 2023-09-22T21:47:09.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026ac3325cbf3a44d-00650e0b5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.399 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:09.399 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:09.399 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:47:09.400 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:47:09.400 DEBUG:boto:Method: GET 2023-09-22T21:47:09.400 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:47:09.400 DEBUG:boto:Data: 2023-09-22T21:47:09.400 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.400 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.400 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.400 DEBUG:boto:Params: {} 2023-09-22T21:47:09.400 DEBUG:boto:Token: None 2023-09-22T21:47:09.400 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/ 2023-09-22T21:47:09.401 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8= 2023-09-22T21:47:09.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8=', 'Content-Length': '0'} 2023-09-22T21:47:09.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000ced15310e63d78-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.405 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:09.406 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:09.406 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:09.406 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:47:09.406 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:47:09.406 DEBUG:boto:Method: GET 2023-09-22T21:47:09.406 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:47:09.406 DEBUG:boto:Data: 2023-09-22T21:47:09.407 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.407 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.407 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.407 DEBUG:boto:Params: {} 2023-09-22T21:47:09.407 DEBUG:boto:Token: None 2023-09-22T21:47:09.407 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/ 2023-09-22T21:47:09.407 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8= 2023-09-22T21:47:09.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8=', 'Content-Length': '0'} 2023-09-22T21:47:09.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cbcfe4564c065e6-00650e0b5d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.416 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:09.416 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:47:09.416 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:47:09.417 DEBUG:boto:Method: HEAD 2023-09-22T21:47:09.417 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:47:09.417 DEBUG:boto:Data: 2023-09-22T21:47:09.417 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.417 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.417 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.417 DEBUG:boto:Params: {} 2023-09-22T21:47:09.417 DEBUG:boto:Token: None 2023-09-22T21:47:09.417 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj11 2023-09-22T21:47:09.418 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ry3WcKO0rEuvX5mke1lg2iPlA0w= 2023-09-22T21:47:09.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ry3WcKO0rEuvX5mke1lg2iPlA0w=', 'Content-Length': '0'} 2023-09-22T21:47:09.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aff0058a2e39f98b-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.421 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:47:09.421 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:47:09.421 DEBUG:boto:Method: HEAD 2023-09-22T21:47:09.421 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:47:09.421 DEBUG:boto:Data: 2023-09-22T21:47:09.421 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.421 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.421 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.422 DEBUG:boto:Params: {} 2023-09-22T21:47:09.422 DEBUG:boto:Token: None 2023-09-22T21:47:09.422 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj11 2023-09-22T21:47:09.422 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ry3WcKO0rEuvX5mke1lg2iPlA0w= 2023-09-22T21:47:09.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ry3WcKO0rEuvX5mke1lg2iPlA0w=', 'Content-Length': '0'} 2023-09-22T21:47:09.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfa06c327bbfefd8-00650e0b5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:47:09.425 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:47:09.425 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:47:09.425 DEBUG:boto:Method: GET 2023-09-22T21:47:09.425 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:47:09.426 DEBUG:boto:Data: 2023-09-22T21:47:09.426 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.426 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.426 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.426 DEBUG:boto:Params: {} 2023-09-22T21:47:09.426 DEBUG:boto:Token: None 2023-09-22T21:47:09.426 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj11 2023-09-22T21:47:09.426 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4BtYRK3VXZU9D/jeNY/NyRHvz8E= 2023-09-22T21:47:09.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4BtYRK3VXZU9D/jeNY/NyRHvz8E=', 'Content-Length': '0'} 2023-09-22T21:47:09.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d835d3e40b15615-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.429 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:47:09.430 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:47:09.430 DEBUG:boto:Method: GET 2023-09-22T21:47:09.430 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:47:09.430 DEBUG:boto:Data: 2023-09-22T21:47:09.430 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.430 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.430 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.430 DEBUG:boto:Params: {} 2023-09-22T21:47:09.430 DEBUG:boto:Token: None 2023-09-22T21:47:09.431 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj11 2023-09-22T21:47:09.431 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4BtYRK3VXZU9D/jeNY/NyRHvz8E= 2023-09-22T21:47:09.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4BtYRK3VXZU9D/jeNY/NyRHvz8E=', 'Content-Length': '0'} 2023-09-22T21:47:09.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079a49180600895a9-00650e0b5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.475 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:09.475 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:09.475 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:47:09.475 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:47:09.475 DEBUG:boto:Method: GET 2023-09-22T21:47:09.475 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:47:09.475 DEBUG:boto:Data: 2023-09-22T21:47:09.475 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.475 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.476 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.476 DEBUG:boto:Params: {} 2023-09-22T21:47:09.476 DEBUG:boto:Token: None 2023-09-22T21:47:09.476 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/ 2023-09-22T21:47:09.476 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8= 2023-09-22T21:47:09.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8=', 'Content-Length': '0'} 2023-09-22T21:47:09.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb959a3fc730241a-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.482 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:09.483 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:09.483 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:09.483 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:47:09.483 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:47:09.483 DEBUG:boto:Method: GET 2023-09-22T21:47:09.483 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:47:09.483 DEBUG:boto:Data: 2023-09-22T21:47:09.483 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.484 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.484 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.484 DEBUG:boto:Params: {} 2023-09-22T21:47:09.484 DEBUG:boto:Token: None 2023-09-22T21:47:09.484 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/ 2023-09-22T21:47:09.484 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8= 2023-09-22T21:47:09.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8=', 'Content-Length': '0'} 2023-09-22T21:47:09.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adf8d94b2122f051-00650e0b5d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.493 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:09.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-22T21:47:09.494 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:47:09.494 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:47:09.494 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.494 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.494 DEBUG:boto:Method: GET 2023-09-22T21:47:09.494 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.494 DEBUG:boto:Data: 2023-09-22T21:47:09.494 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.494 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.494 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.495 DEBUG:boto:Params: {} 2023-09-22T21:47:09.495 DEBUG:boto:Token: None 2023-09-22T21:47:09.495 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.495 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bimHPOtkdOUsDD6B2knA7mdeC7Q= 2023-09-22T21:47:09.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bimHPOtkdOUsDD6B2knA7mdeC7Q=', 'Content-Length': '0'} 2023-09-22T21:47:09.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb45a549c5b972ac-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.539 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:47:09.539 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:47:09.539 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.539 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.539 DEBUG:boto:Method: GET 2023-09-22T21:47:09.539 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.539 DEBUG:boto:Data: 2023-09-22T21:47:09.539 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.539 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.540 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.540 DEBUG:boto:Params: {} 2023-09-22T21:47:09.540 DEBUG:boto:Token: None 2023-09-22T21:47:09.540 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.540 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bimHPOtkdOUsDD6B2knA7mdeC7Q= 2023-09-22T21:47:09.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bimHPOtkdOUsDD6B2knA7mdeC7Q=', 'Content-Length': '0'} 2023-09-22T21:47:09.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd3c39e7850d02b6-00650e0b5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.543 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:47:09.543 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:47:09.543 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.543 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.544 DEBUG:boto:Method: HEAD 2023-09-22T21:47:09.544 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.544 DEBUG:boto:Data: 2023-09-22T21:47:09.544 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.544 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.544 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.544 DEBUG:boto:Params: {} 2023-09-22T21:47:09.544 DEBUG:boto:Token: None 2023-09-22T21:47:09.544 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.545 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:31B/y8nlMvan0pe8J1sUyhvUd6w= 2023-09-22T21:47:09.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:31B/y8nlMvan0pe8J1sUyhvUd6w=', 'Content-Length': '0'} 2023-09-22T21:47:09.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090959416d606f37f-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.548 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:47:09.548 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:47:09.548 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.548 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.548 DEBUG:boto:Method: HEAD 2023-09-22T21:47:09.548 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.548 DEBUG:boto:Data: 2023-09-22T21:47:09.548 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.549 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.549 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.549 DEBUG:boto:Params: {} 2023-09-22T21:47:09.549 DEBUG:boto:Token: None 2023-09-22T21:47:09.549 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.549 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:31B/y8nlMvan0pe8J1sUyhvUd6w= 2023-09-22T21:47:09.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:31B/y8nlMvan0pe8J1sUyhvUd6w=', 'Content-Length': '0'} 2023-09-22T21:47:09.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002a13b5843f973c8-00650e0b5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-22T21:47:09.553 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:47:09.553 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:47:09.553 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.553 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.553 DEBUG:boto:Method: GET 2023-09-22T21:47:09.553 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.553 DEBUG:boto:Data: 2023-09-22T21:47:09.554 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.554 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.554 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.554 DEBUG:boto:Params: {} 2023-09-22T21:47:09.554 DEBUG:boto:Token: None 2023-09-22T21:47:09.554 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.554 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bimHPOtkdOUsDD6B2knA7mdeC7Q= 2023-09-22T21:47:09.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bimHPOtkdOUsDD6B2knA7mdeC7Q=', 'Content-Length': '0'} 2023-09-22T21:47:09.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d373567ca49c1ede-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.557 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:47:09.557 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:47:09.558 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.558 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.558 DEBUG:boto:Method: GET 2023-09-22T21:47:09.558 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.558 DEBUG:boto:Data: 2023-09-22T21:47:09.558 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.558 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.558 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.558 DEBUG:boto:Params: {} 2023-09-22T21:47:09.559 DEBUG:boto:Token: None 2023-09-22T21:47:09.559 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:47:09.559 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bimHPOtkdOUsDD6B2knA7mdeC7Q= 2023-09-22T21:47:09.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bimHPOtkdOUsDD6B2knA7mdeC7Q=', 'Content-Length': '0'} 2023-09-22T21:47:09.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df801d6d2a1affe5-00650e0b5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.602 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:09.603 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:09.603 DEBUG:boto:path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:47:09.603 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:47:09.603 DEBUG:boto:Method: GET 2023-09-22T21:47:09.603 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj115 2023-09-22T21:47:09.603 DEBUG:boto:Data: 2023-09-22T21:47:09.603 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.603 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.603 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.604 DEBUG:boto:Params: {} 2023-09-22T21:47:09.604 DEBUG:boto:Token: None 2023-09-22T21:47:09.604 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/ 2023-09-22T21:47:09.604 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8= 2023-09-22T21:47:09.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8=', 'Content-Length': '0'} 2023-09-22T21:47:09.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053484a4e10ecc091-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.609 DEBUG:boto:b'ffzdqu-59obj1151000falseobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:09.609 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:09.610 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:09.610 DEBUG:boto:path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:47:09.610 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:47:09.610 DEBUG:boto:Method: GET 2023-09-22T21:47:09.610 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj115 2023-09-22T21:47:09.610 DEBUG:boto:Data: 2023-09-22T21:47:09.610 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.611 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.611 DEBUG:boto:Params: {} 2023-09-22T21:47:09.611 DEBUG:boto:Token: None 2023-09-22T21:47:09.611 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/ 2023-09-22T21:47:09.611 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8= 2023-09-22T21:47:09.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8=', 'Content-Length': '0'} 2023-09-22T21:47:09.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012b3e0b673fcad16-00650e0b5d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.620 DEBUG:boto:b'ffzdqu-59obj1151000falseobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:09.620 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:47:09.621 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:47:09.621 DEBUG:boto:Method: HEAD 2023-09-22T21:47:09.621 DEBUG:boto:Path: /ffzdqu-59/obj115 2023-09-22T21:47:09.621 DEBUG:boto:Data: 2023-09-22T21:47:09.621 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.621 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.621 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.622 DEBUG:boto:Params: {} 2023-09-22T21:47:09.622 DEBUG:boto:Token: None 2023-09-22T21:47:09.622 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj115 2023-09-22T21:47:09.622 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zzJZ5vdVOYJx7GxI5TYJBQmlDDU= 2023-09-22T21:47:09.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zzJZ5vdVOYJx7GxI5TYJBQmlDDU=', 'Content-Length': '0'} 2023-09-22T21:47:09.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f21cb4d8acdada8f-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.625 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:47:09.625 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:47:09.625 DEBUG:boto:Method: HEAD 2023-09-22T21:47:09.626 DEBUG:boto:Path: /ffzdqu-59/obj115 2023-09-22T21:47:09.626 DEBUG:boto:Data: 2023-09-22T21:47:09.626 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.626 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.626 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.626 DEBUG:boto:Params: {} 2023-09-22T21:47:09.626 DEBUG:boto:Token: None 2023-09-22T21:47:09.627 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj115 2023-09-22T21:47:09.627 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zzJZ5vdVOYJx7GxI5TYJBQmlDDU= 2023-09-22T21:47:09.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zzJZ5vdVOYJx7GxI5TYJBQmlDDU=', 'Content-Length': '0'} 2023-09-22T21:47:09.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043438fa00407afd7-00650e0b5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-22T21:47:09.630 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:47:09.630 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:47:09.630 DEBUG:boto:Method: GET 2023-09-22T21:47:09.630 DEBUG:boto:Path: /ffzdqu-59/obj115 2023-09-22T21:47:09.630 DEBUG:boto:Data: 2023-09-22T21:47:09.630 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.630 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.631 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.631 DEBUG:boto:Params: {} 2023-09-22T21:47:09.631 DEBUG:boto:Token: None 2023-09-22T21:47:09.631 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj115 2023-09-22T21:47:09.631 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RfoOVot/ZP4nKREwrepcLqCI5EY= 2023-09-22T21:47:09.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RfoOVot/ZP4nKREwrepcLqCI5EY=', 'Content-Length': '0'} 2023-09-22T21:47:09.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aca4e16547029478-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.674 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:47:09.674 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:47:09.675 DEBUG:boto:Method: GET 2023-09-22T21:47:09.677 DEBUG:boto:Path: /ffzdqu-59/obj115 2023-09-22T21:47:09.677 DEBUG:boto:Data: 2023-09-22T21:47:09.677 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.677 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.677 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.677 DEBUG:boto:Params: {} 2023-09-22T21:47:09.677 DEBUG:boto:Token: None 2023-09-22T21:47:09.678 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj115 2023-09-22T21:47:09.678 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RfoOVot/ZP4nKREwrepcLqCI5EY= 2023-09-22T21:47:09.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RfoOVot/ZP4nKREwrepcLqCI5EY=', 'Content-Length': '0'} 2023-09-22T21:47:09.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c847507862d0991-00650e0b5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.681 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:09.681 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:09.681 DEBUG:boto:path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:47:09.681 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:47:09.681 DEBUG:boto:Method: GET 2023-09-22T21:47:09.682 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj115 2023-09-22T21:47:09.682 DEBUG:boto:Data: 2023-09-22T21:47:09.682 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.682 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.682 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.682 DEBUG:boto:Params: {} 2023-09-22T21:47:09.682 DEBUG:boto:Token: None 2023-09-22T21:47:09.682 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/ 2023-09-22T21:47:09.683 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8= 2023-09-22T21:47:09.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8=', 'Content-Length': '0'} 2023-09-22T21:47:09.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d3f7f0ab25f22e5-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.688 DEBUG:boto:b'ffzdqu-59obj1151000falseobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:09.688 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:09.689 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:09.689 DEBUG:boto:path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:47:09.689 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:47:09.689 DEBUG:boto:Method: GET 2023-09-22T21:47:09.689 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj115 2023-09-22T21:47:09.689 DEBUG:boto:Data: 2023-09-22T21:47:09.689 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.689 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.689 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.689 DEBUG:boto:Params: {} 2023-09-22T21:47:09.690 DEBUG:boto:Token: None 2023-09-22T21:47:09.690 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/ 2023-09-22T21:47:09.690 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8= 2023-09-22T21:47:09.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8=', 'Content-Length': '0'} 2023-09-22T21:47:09.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9b014ac477e119d-00650e0b5d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.699 DEBUG:boto:b'ffzdqu-59obj1151000falseobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:09.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-22T21:47:09.699 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:47:09.699 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:47:09.699 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.700 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.700 DEBUG:boto:Method: GET 2023-09-22T21:47:09.700 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.700 DEBUG:boto:Data: 2023-09-22T21:47:09.700 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.700 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.700 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.700 DEBUG:boto:Params: {} 2023-09-22T21:47:09.700 DEBUG:boto:Token: None 2023-09-22T21:47:09.701 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.701 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:woRsGYbIg08+UjS5nVkM8sNwq1Q= 2023-09-22T21:47:09.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:woRsGYbIg08+UjS5nVkM8sNwq1Q=', 'Content-Length': '0'} 2023-09-22T21:47:09.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051d1d470065e12e5-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.704 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:47:09.704 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:47:09.704 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.704 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.705 DEBUG:boto:Method: GET 2023-09-22T21:47:09.705 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.705 DEBUG:boto:Data: 2023-09-22T21:47:09.705 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.705 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.705 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.705 DEBUG:boto:Params: {} 2023-09-22T21:47:09.705 DEBUG:boto:Token: None 2023-09-22T21:47:09.705 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.706 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:woRsGYbIg08+UjS5nVkM8sNwq1Q= 2023-09-22T21:47:09.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:woRsGYbIg08+UjS5nVkM8sNwq1Q=', 'Content-Length': '0'} 2023-09-22T21:47:09.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e4139a23f594191-00650e0b5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.750 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:47:09.750 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:47:09.751 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.751 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.751 DEBUG:boto:Method: HEAD 2023-09-22T21:47:09.751 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.751 DEBUG:boto:Data: 2023-09-22T21:47:09.751 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.751 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.751 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.751 DEBUG:boto:Params: {} 2023-09-22T21:47:09.752 DEBUG:boto:Token: None 2023-09-22T21:47:09.752 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.752 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6lM2eLkXPJnJKSaTY/idodCkb0c= 2023-09-22T21:47:09.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6lM2eLkXPJnJKSaTY/idodCkb0c=', 'Content-Length': '0'} 2023-09-22T21:47:09.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041ce1b3c7d851f15-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.755 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:47:09.755 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:47:09.755 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.755 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.755 DEBUG:boto:Method: HEAD 2023-09-22T21:47:09.756 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.756 DEBUG:boto:Data: 2023-09-22T21:47:09.756 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.756 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.756 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.756 DEBUG:boto:Params: {} 2023-09-22T21:47:09.756 DEBUG:boto:Token: None 2023-09-22T21:47:09.756 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.757 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6lM2eLkXPJnJKSaTY/idodCkb0c= 2023-09-22T21:47:09.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6lM2eLkXPJnJKSaTY/idodCkb0c=', 'Content-Length': '0'} 2023-09-22T21:47:09.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf8ab787da0676f9-00650e0b5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-22T21:47:09.760 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:47:09.760 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:47:09.760 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.760 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.760 DEBUG:boto:Method: GET 2023-09-22T21:47:09.760 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.760 DEBUG:boto:Data: 2023-09-22T21:47:09.761 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.761 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.761 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.761 DEBUG:boto:Params: {} 2023-09-22T21:47:09.761 DEBUG:boto:Token: None 2023-09-22T21:47:09.761 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.761 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:woRsGYbIg08+UjS5nVkM8sNwq1Q= 2023-09-22T21:47:09.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:woRsGYbIg08+UjS5nVkM8sNwq1Q=', 'Content-Length': '0'} 2023-09-22T21:47:09.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000551f3f2a185c1e0c-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.810 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:47:09.810 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:47:09.811 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.811 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.811 DEBUG:boto:Method: GET 2023-09-22T21:47:09.811 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.811 DEBUG:boto:Data: 2023-09-22T21:47:09.811 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.811 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.811 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.811 DEBUG:boto:Params: {} 2023-09-22T21:47:09.812 DEBUG:boto:Token: None 2023-09-22T21:47:09.812 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:47:09.812 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:woRsGYbIg08+UjS5nVkM8sNwq1Q= 2023-09-22T21:47:09.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:woRsGYbIg08+UjS5nVkM8sNwq1Q=', 'Content-Length': '0'} 2023-09-22T21:47:09.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa25d8662cb0d3da-00650e0b5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.815 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:09.815 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:09.816 DEBUG:boto:path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:47:09.816 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:47:09.816 DEBUG:boto:Method: GET 2023-09-22T21:47:09.816 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj116 2023-09-22T21:47:09.816 DEBUG:boto:Data: 2023-09-22T21:47:09.816 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.816 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.816 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.816 DEBUG:boto:Params: {} 2023-09-22T21:47:09.817 DEBUG:boto:Token: None 2023-09-22T21:47:09.817 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/ 2023-09-22T21:47:09.817 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8= 2023-09-22T21:47:09.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8=', 'Content-Length': '0'} 2023-09-22T21:47:09.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e60c914a6dfedd24-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.822 DEBUG:boto:b'ffzdqu-59obj1161000falseobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:09.823 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:09.823 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:09.823 DEBUG:boto:path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:47:09.823 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:47:09.823 DEBUG:boto:Method: GET 2023-09-22T21:47:09.823 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj116 2023-09-22T21:47:09.824 DEBUG:boto:Data: 2023-09-22T21:47:09.824 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.824 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.824 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.824 DEBUG:boto:Params: {} 2023-09-22T21:47:09.824 DEBUG:boto:Token: None 2023-09-22T21:47:09.824 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/ 2023-09-22T21:47:09.825 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8= 2023-09-22T21:47:09.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8=', 'Content-Length': '0'} 2023-09-22T21:47:09.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e0a125d5651952d-00650e0b5d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.833 DEBUG:boto:b'ffzdqu-59obj1161000falseobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:09.833 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:47:09.834 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:47:09.834 DEBUG:boto:Method: HEAD 2023-09-22T21:47:09.834 DEBUG:boto:Path: /ffzdqu-59/obj116 2023-09-22T21:47:09.834 DEBUG:boto:Data: 2023-09-22T21:47:09.834 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.834 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.834 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.834 DEBUG:boto:Params: {} 2023-09-22T21:47:09.834 DEBUG:boto:Token: None 2023-09-22T21:47:09.835 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj116 2023-09-22T21:47:09.835 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EBEeBwIbD/4WKrUOQxYedHYO2/s= 2023-09-22T21:47:09.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EBEeBwIbD/4WKrUOQxYedHYO2/s=', 'Content-Length': '0'} 2023-09-22T21:47:09.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013cb00ed9a28231a-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.838 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:47:09.838 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:47:09.838 DEBUG:boto:Method: HEAD 2023-09-22T21:47:09.838 DEBUG:boto:Path: /ffzdqu-59/obj116 2023-09-22T21:47:09.838 DEBUG:boto:Data: 2023-09-22T21:47:09.838 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.838 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.838 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.838 DEBUG:boto:Params: {} 2023-09-22T21:47:09.839 DEBUG:boto:Token: None 2023-09-22T21:47:09.839 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj116 2023-09-22T21:47:09.839 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EBEeBwIbD/4WKrUOQxYedHYO2/s= 2023-09-22T21:47:09.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EBEeBwIbD/4WKrUOQxYedHYO2/s=', 'Content-Length': '0'} 2023-09-22T21:47:09.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5b3adea77f20cda-00650e0b5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-22T21:47:09.842 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:47:09.842 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:47:09.842 DEBUG:boto:Method: GET 2023-09-22T21:47:09.842 DEBUG:boto:Path: /ffzdqu-59/obj116 2023-09-22T21:47:09.842 DEBUG:boto:Data: 2023-09-22T21:47:09.842 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.842 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.842 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.843 DEBUG:boto:Params: {} 2023-09-22T21:47:09.843 DEBUG:boto:Token: None 2023-09-22T21:47:09.843 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj116 2023-09-22T21:47:09.843 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x9B9uT5vTwny+lA42ns9mYk6x3g= 2023-09-22T21:47:09.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x9B9uT5vTwny+lA42ns9mYk6x3g=', 'Content-Length': '0'} 2023-09-22T21:47:09.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbd0fc613b92ad1e-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.846 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:47:09.846 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:47:09.846 DEBUG:boto:Method: GET 2023-09-22T21:47:09.846 DEBUG:boto:Path: /ffzdqu-59/obj116 2023-09-22T21:47:09.846 DEBUG:boto:Data: 2023-09-22T21:47:09.847 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.847 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.847 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.847 DEBUG:boto:Params: {} 2023-09-22T21:47:09.847 DEBUG:boto:Token: None 2023-09-22T21:47:09.847 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj116 2023-09-22T21:47:09.847 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x9B9uT5vTwny+lA42ns9mYk6x3g= 2023-09-22T21:47:09.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x9B9uT5vTwny+lA42ns9mYk6x3g=', 'Content-Length': '0'} 2023-09-22T21:47:09.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076a54a28e0bc2ccb-00650e0b5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.894 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:09.894 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:09.895 DEBUG:boto:path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:47:09.895 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:47:09.895 DEBUG:boto:Method: GET 2023-09-22T21:47:09.895 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj116 2023-09-22T21:47:09.895 DEBUG:boto:Data: 2023-09-22T21:47:09.895 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.895 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.895 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.895 DEBUG:boto:Params: {} 2023-09-22T21:47:09.896 DEBUG:boto:Token: None 2023-09-22T21:47:09.896 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/ 2023-09-22T21:47:09.896 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8= 2023-09-22T21:47:09.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8=', 'Content-Length': '0'} 2023-09-22T21:47:09.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f2dddd2bbc5dc18-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.901 DEBUG:boto:b'ffzdqu-59obj1161000falseobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:09.902 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:09.902 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:09.902 DEBUG:boto:path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:47:09.902 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:47:09.902 DEBUG:boto:Method: GET 2023-09-22T21:47:09.902 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj116 2023-09-22T21:47:09.902 DEBUG:boto:Data: 2023-09-22T21:47:09.903 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.903 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.903 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.903 DEBUG:boto:Params: {} 2023-09-22T21:47:09.903 DEBUG:boto:Token: None 2023-09-22T21:47:09.903 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/ 2023-09-22T21:47:09.903 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8= 2023-09-22T21:47:09.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/L5U4fCrHASpoxP8VoStrR1fzc8=', 'Content-Length': '0'} 2023-09-22T21:47:09.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef0b6a7f70cfceb0-00650e0b5d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.912 DEBUG:boto:b'ffzdqu-59obj1161000falseobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:09.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-22T21:47:09.913 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:47:09.913 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:47:09.913 DEBUG:boto:path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.913 DEBUG:boto:auth_path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.913 DEBUG:boto:Method: GET 2023-09-22T21:47:09.913 DEBUG:boto:Path: /ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.913 DEBUG:boto:Data: 2023-09-22T21:47:09.913 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.914 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.914 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.914 DEBUG:boto:Params: {} 2023-09-22T21:47:09.914 DEBUG:boto:Token: None 2023-09-22T21:47:09.914 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.914 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2ixiwa1Mh0XAxMKu7PVipStvLBo= 2023-09-22T21:47:09.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2ixiwa1Mh0XAxMKu7PVipStvLBo=', 'Content-Length': '0'} 2023-09-22T21:47:09.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097c2e3289895ca6a-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.958 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:47:09.958 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:47:09.958 DEBUG:boto:path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.959 DEBUG:boto:auth_path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.959 DEBUG:boto:Method: GET 2023-09-22T21:47:09.959 DEBUG:boto:Path: /ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.959 DEBUG:boto:Data: 2023-09-22T21:47:09.959 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.959 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.959 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.959 DEBUG:boto:Params: {} 2023-09-22T21:47:09.959 DEBUG:boto:Token: None 2023-09-22T21:47:09.960 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.960 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2ixiwa1Mh0XAxMKu7PVipStvLBo= 2023-09-22T21:47:09.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2ixiwa1Mh0XAxMKu7PVipStvLBo=', 'Content-Length': '0'} 2023-09-22T21:47:09.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000200f27a105ac8f8-00650e0b5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.963 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:47:09.963 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:47:09.963 DEBUG:boto:path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.963 DEBUG:boto:auth_path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.964 DEBUG:boto:Method: HEAD 2023-09-22T21:47:09.964 DEBUG:boto:Path: /ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.964 DEBUG:boto:Data: 2023-09-22T21:47:09.964 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.964 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.964 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.964 DEBUG:boto:Params: {} 2023-09-22T21:47:09.964 DEBUG:boto:Token: None 2023-09-22T21:47:09.964 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.964 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rb6C4iC4pTN4Oz45btmIoDByoWY= 2023-09-22T21:47:09.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rb6C4iC4pTN4Oz45btmIoDByoWY=', 'Content-Length': '0'} 2023-09-22T21:47:09.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006339b3bb35a99b5-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.967 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:47:09.967 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:47:09.968 DEBUG:boto:path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.968 DEBUG:boto:auth_path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.968 DEBUG:boto:Method: HEAD 2023-09-22T21:47:09.968 DEBUG:boto:Path: /ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.968 DEBUG:boto:Data: 2023-09-22T21:47:09.968 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.968 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.968 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.968 DEBUG:boto:Params: {} 2023-09-22T21:47:09.969 DEBUG:boto:Token: None 2023-09-22T21:47:09.969 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.969 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rb6C4iC4pTN4Oz45btmIoDByoWY= 2023-09-22T21:47:09.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rb6C4iC4pTN4Oz45btmIoDByoWY=', 'Content-Length': '0'} 2023-09-22T21:47:09.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a63bbd131f081831-00650e0b5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-22T21:47:09.972 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:47:09.972 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:47:09.972 DEBUG:boto:path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.972 DEBUG:boto:auth_path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.972 DEBUG:boto:Method: GET 2023-09-22T21:47:09.972 DEBUG:boto:Path: /ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.973 DEBUG:boto:Data: 2023-09-22T21:47:09.973 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.973 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:09.973 DEBUG:boto:Port: 8001 2023-09-22T21:47:09.973 DEBUG:boto:Params: {} 2023-09-22T21:47:09.973 DEBUG:boto:Token: None 2023-09-22T21:47:09.973 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.973 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2ixiwa1Mh0XAxMKu7PVipStvLBo= 2023-09-22T21:47:09.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2ixiwa1Mh0XAxMKu7PVipStvLBo=', 'Content-Length': '0'} 2023-09-22T21:47:09.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047a78b6af2c75e06-00650e0b5d-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:09.976 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:47:09.976 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:47:09.977 DEBUG:boto:path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.977 DEBUG:boto:auth_path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.977 DEBUG:boto:Method: GET 2023-09-22T21:47:09.977 DEBUG:boto:Path: /ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.977 DEBUG:boto:Data: 2023-09-22T21:47:09.977 DEBUG:boto:Headers: {} 2023-09-22T21:47:09.977 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:09.977 DEBUG:boto:Port: 8000 2023-09-22T21:47:09.977 DEBUG:boto:Params: {} 2023-09-22T21:47:09.977 DEBUG:boto:Token: None 2023-09-22T21:47:09.978 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:09 GMT /ffzdqu-59/obj124?versionId=null 2023-09-22T21:47:09.978 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2ixiwa1Mh0XAxMKu7PVipStvLBo= 2023-09-22T21:47:09.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2ixiwa1Mh0XAxMKu7PVipStvLBo=', 'Content-Length': '0'} 2023-09-22T21:47:09.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c556b3a06905d89-00650e0b5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.026 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.027 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.027 DEBUG:boto:path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:47:10.027 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:47:10.027 DEBUG:boto:Method: GET 2023-09-22T21:47:10.027 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj124 2023-09-22T21:47:10.027 DEBUG:boto:Data: 2023-09-22T21:47:10.027 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.027 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.027 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.028 DEBUG:boto:Params: {} 2023-09-22T21:47:10.028 DEBUG:boto:Token: None 2023-09-22T21:47:10.028 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.028 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f55d0537069274b6-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.032 DEBUG:boto:b'ffzdqu-59obj1241000falseobj1242023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:10.033 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.033 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.033 DEBUG:boto:path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:47:10.033 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:47:10.033 DEBUG:boto:Method: GET 2023-09-22T21:47:10.034 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj124 2023-09-22T21:47:10.034 DEBUG:boto:Data: 2023-09-22T21:47:10.034 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.034 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.034 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.034 DEBUG:boto:Params: {} 2023-09-22T21:47:10.034 DEBUG:boto:Token: None 2023-09-22T21:47:10.034 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.035 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075a415cef898fbbf-00650e0b5e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.042 DEBUG:boto:b'ffzdqu-59obj1241000falseobj1242023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:10.043 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:47:10.043 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:47:10.043 DEBUG:boto:Method: HEAD 2023-09-22T21:47:10.043 DEBUG:boto:Path: /ffzdqu-59/obj124 2023-09-22T21:47:10.043 DEBUG:boto:Data: 2023-09-22T21:47:10.043 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.043 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.043 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.043 DEBUG:boto:Params: {} 2023-09-22T21:47:10.044 DEBUG:boto:Token: None 2023-09-22T21:47:10.044 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj124 2023-09-22T21:47:10.044 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wd1UuFPM+7k9POUg0ypZ2GPS4ek= 2023-09-22T21:47:10.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wd1UuFPM+7k9POUg0ypZ2GPS4ek=', 'Content-Length': '0'} 2023-09-22T21:47:10.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7dbfa1e3697b8b1-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.047 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:47:10.047 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:47:10.047 DEBUG:boto:Method: HEAD 2023-09-22T21:47:10.047 DEBUG:boto:Path: /ffzdqu-59/obj124 2023-09-22T21:47:10.047 DEBUG:boto:Data: 2023-09-22T21:47:10.047 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.047 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.047 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.048 DEBUG:boto:Params: {} 2023-09-22T21:47:10.048 DEBUG:boto:Token: None 2023-09-22T21:47:10.048 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj124 2023-09-22T21:47:10.048 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wd1UuFPM+7k9POUg0ypZ2GPS4ek= 2023-09-22T21:47:10.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wd1UuFPM+7k9POUg0ypZ2GPS4ek=', 'Content-Length': '0'} 2023-09-22T21:47:10.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da0f0fc2bd6ec787-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-22T21:47:10.051 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:47:10.051 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:47:10.051 DEBUG:boto:Method: GET 2023-09-22T21:47:10.051 DEBUG:boto:Path: /ffzdqu-59/obj124 2023-09-22T21:47:10.051 DEBUG:boto:Data: 2023-09-22T21:47:10.051 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.052 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.052 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.052 DEBUG:boto:Params: {} 2023-09-22T21:47:10.052 DEBUG:boto:Token: None 2023-09-22T21:47:10.052 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj124 2023-09-22T21:47:10.052 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e3blUGN2VfeGbz4m/QjGAU2ir2o= 2023-09-22T21:47:10.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e3blUGN2VfeGbz4m/QjGAU2ir2o=', 'Content-Length': '0'} 2023-09-22T21:47:10.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000559b9a505ef32b2d-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.098 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:47:10.099 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:47:10.099 DEBUG:boto:Method: GET 2023-09-22T21:47:10.099 DEBUG:boto:Path: /ffzdqu-59/obj124 2023-09-22T21:47:10.099 DEBUG:boto:Data: 2023-09-22T21:47:10.099 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.099 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.099 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.099 DEBUG:boto:Params: {} 2023-09-22T21:47:10.100 DEBUG:boto:Token: None 2023-09-22T21:47:10.100 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj124 2023-09-22T21:47:10.100 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e3blUGN2VfeGbz4m/QjGAU2ir2o= 2023-09-22T21:47:10.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e3blUGN2VfeGbz4m/QjGAU2ir2o=', 'Content-Length': '0'} 2023-09-22T21:47:10.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa3663ba08c3bd8c-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.103 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.103 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.104 DEBUG:boto:path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:47:10.104 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:47:10.104 DEBUG:boto:Method: GET 2023-09-22T21:47:10.104 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj124 2023-09-22T21:47:10.104 DEBUG:boto:Data: 2023-09-22T21:47:10.104 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.104 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.104 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.105 DEBUG:boto:Params: {} 2023-09-22T21:47:10.105 DEBUG:boto:Token: None 2023-09-22T21:47:10.105 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.105 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055ce2d1eceb14692-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.110 DEBUG:boto:b'ffzdqu-59obj1241000falseobj1242023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:10.110 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.110 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.110 DEBUG:boto:path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:47:10.111 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:47:10.111 DEBUG:boto:Method: GET 2023-09-22T21:47:10.111 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj124 2023-09-22T21:47:10.111 DEBUG:boto:Data: 2023-09-22T21:47:10.111 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.111 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.111 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.111 DEBUG:boto:Params: {} 2023-09-22T21:47:10.112 DEBUG:boto:Token: None 2023-09-22T21:47:10.112 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.112 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f068712ef01f8d51-00650e0b5e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.162 DEBUG:boto:b'ffzdqu-59obj1241000falseobj1242023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:10.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-22T21:47:10.163 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:47:10.163 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:47:10.163 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.163 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.163 DEBUG:boto:Method: GET 2023-09-22T21:47:10.163 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.163 DEBUG:boto:Data: 2023-09-22T21:47:10.164 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.164 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.164 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.164 DEBUG:boto:Params: {} 2023-09-22T21:47:10.164 DEBUG:boto:Token: None 2023-09-22T21:47:10.164 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.164 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jcVRfIN4pxQ56HjNHe3gyuQ5+YI= 2023-09-22T21:47:10.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jcVRfIN4pxQ56HjNHe3gyuQ5+YI=', 'Content-Length': '0'} 2023-09-22T21:47:10.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5a01dbf4f840d49-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.167 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:47:10.168 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:47:10.168 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.168 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.168 DEBUG:boto:Method: GET 2023-09-22T21:47:10.168 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.168 DEBUG:boto:Data: 2023-09-22T21:47:10.168 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.168 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.168 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.169 DEBUG:boto:Params: {} 2023-09-22T21:47:10.169 DEBUG:boto:Token: None 2023-09-22T21:47:10.169 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.169 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jcVRfIN4pxQ56HjNHe3gyuQ5+YI= 2023-09-22T21:47:10.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jcVRfIN4pxQ56HjNHe3gyuQ5+YI=', 'Content-Length': '0'} 2023-09-22T21:47:10.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a72e26b271273fc5-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.172 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:47:10.172 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:47:10.172 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.173 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.173 DEBUG:boto:Method: HEAD 2023-09-22T21:47:10.173 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.173 DEBUG:boto:Data: 2023-09-22T21:47:10.173 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.173 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.173 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.173 DEBUG:boto:Params: {} 2023-09-22T21:47:10.173 DEBUG:boto:Token: None 2023-09-22T21:47:10.174 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.174 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jZ5RrNQlDJ7K8QvA0pYs0DF6CDE= 2023-09-22T21:47:10.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jZ5RrNQlDJ7K8QvA0pYs0DF6CDE=', 'Content-Length': '0'} 2023-09-22T21:47:10.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000054e2a0f081e1c6d-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.177 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:47:10.177 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:47:10.177 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.177 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.177 DEBUG:boto:Method: HEAD 2023-09-22T21:47:10.177 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.177 DEBUG:boto:Data: 2023-09-22T21:47:10.177 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.177 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.178 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.178 DEBUG:boto:Params: {} 2023-09-22T21:47:10.178 DEBUG:boto:Token: None 2023-09-22T21:47:10.178 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.178 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jZ5RrNQlDJ7K8QvA0pYs0DF6CDE= 2023-09-22T21:47:10.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jZ5RrNQlDJ7K8QvA0pYs0DF6CDE=', 'Content-Length': '0'} 2023-09-22T21:47:10.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066404f427727a1d9-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-22T21:47:10.181 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:47:10.181 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:47:10.181 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.181 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.181 DEBUG:boto:Method: GET 2023-09-22T21:47:10.182 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.182 DEBUG:boto:Data: 2023-09-22T21:47:10.182 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.182 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.182 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.182 DEBUG:boto:Params: {} 2023-09-22T21:47:10.182 DEBUG:boto:Token: None 2023-09-22T21:47:10.182 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.182 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jcVRfIN4pxQ56HjNHe3gyuQ5+YI= 2023-09-22T21:47:10.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jcVRfIN4pxQ56HjNHe3gyuQ5+YI=', 'Content-Length': '0'} 2023-09-22T21:47:10.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f6efde422031c82-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.185 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:47:10.186 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:47:10.186 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.186 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.186 DEBUG:boto:Method: GET 2023-09-22T21:47:10.186 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.186 DEBUG:boto:Data: 2023-09-22T21:47:10.186 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.186 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.186 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.186 DEBUG:boto:Params: {} 2023-09-22T21:47:10.187 DEBUG:boto:Token: None 2023-09-22T21:47:10.187 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:47:10.187 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jcVRfIN4pxQ56HjNHe3gyuQ5+YI= 2023-09-22T21:47:10.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jcVRfIN4pxQ56HjNHe3gyuQ5+YI=', 'Content-Length': '0'} 2023-09-22T21:47:10.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abc94dfd2b1bbf87-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.190 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.190 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.190 DEBUG:boto:path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:47:10.190 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:47:10.190 DEBUG:boto:Method: GET 2023-09-22T21:47:10.190 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj145 2023-09-22T21:47:10.191 DEBUG:boto:Data: 2023-09-22T21:47:10.191 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.191 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.191 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.191 DEBUG:boto:Params: {} 2023-09-22T21:47:10.191 DEBUG:boto:Token: None 2023-09-22T21:47:10.191 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.191 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000997c042f53ffed35-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.196 DEBUG:boto:b'ffzdqu-59obj1451000falseobj1452023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:10.196 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.196 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.196 DEBUG:boto:path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:47:10.196 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:47:10.196 DEBUG:boto:Method: GET 2023-09-22T21:47:10.197 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj145 2023-09-22T21:47:10.197 DEBUG:boto:Data: 2023-09-22T21:47:10.197 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.197 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.197 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.197 DEBUG:boto:Params: {} 2023-09-22T21:47:10.197 DEBUG:boto:Token: None 2023-09-22T21:47:10.197 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.197 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d348b848c9560950-00650e0b5e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.207 DEBUG:boto:b'ffzdqu-59obj1451000falseobj1452023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:10.207 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:47:10.207 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:47:10.207 DEBUG:boto:Method: HEAD 2023-09-22T21:47:10.207 DEBUG:boto:Path: /ffzdqu-59/obj145 2023-09-22T21:47:10.207 DEBUG:boto:Data: 2023-09-22T21:47:10.208 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.208 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.208 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.208 DEBUG:boto:Params: {} 2023-09-22T21:47:10.208 DEBUG:boto:Token: None 2023-09-22T21:47:10.208 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj145 2023-09-22T21:47:10.208 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZxnOKbahJOjXp4Xcpwg2sap2k5w= 2023-09-22T21:47:10.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZxnOKbahJOjXp4Xcpwg2sap2k5w=', 'Content-Length': '0'} 2023-09-22T21:47:10.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058e08c4398b077c5-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.211 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:47:10.211 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:47:10.211 DEBUG:boto:Method: HEAD 2023-09-22T21:47:10.212 DEBUG:boto:Path: /ffzdqu-59/obj145 2023-09-22T21:47:10.212 DEBUG:boto:Data: 2023-09-22T21:47:10.212 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.212 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.212 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.212 DEBUG:boto:Params: {} 2023-09-22T21:47:10.212 DEBUG:boto:Token: None 2023-09-22T21:47:10.212 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj145 2023-09-22T21:47:10.212 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZxnOKbahJOjXp4Xcpwg2sap2k5w= 2023-09-22T21:47:10.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZxnOKbahJOjXp4Xcpwg2sap2k5w=', 'Content-Length': '0'} 2023-09-22T21:47:10.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d884cc5c8cc9bc-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-22T21:47:10.216 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:47:10.216 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:47:10.216 DEBUG:boto:Method: GET 2023-09-22T21:47:10.216 DEBUG:boto:Path: /ffzdqu-59/obj145 2023-09-22T21:47:10.216 DEBUG:boto:Data: 2023-09-22T21:47:10.216 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.216 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.216 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.216 DEBUG:boto:Params: {} 2023-09-22T21:47:10.216 DEBUG:boto:Token: None 2023-09-22T21:47:10.217 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj145 2023-09-22T21:47:10.217 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:k1f/blEJ30Os3bVxnqJ0hxN9wtg= 2023-09-22T21:47:10.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:k1f/blEJ30Os3bVxnqJ0hxN9wtg=', 'Content-Length': '0'} 2023-09-22T21:47:10.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1dd30b9f9e2fd94-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.262 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:47:10.263 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:47:10.263 DEBUG:boto:Method: GET 2023-09-22T21:47:10.263 DEBUG:boto:Path: /ffzdqu-59/obj145 2023-09-22T21:47:10.263 DEBUG:boto:Data: 2023-09-22T21:47:10.263 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.263 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.263 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.263 DEBUG:boto:Params: {} 2023-09-22T21:47:10.263 DEBUG:boto:Token: None 2023-09-22T21:47:10.264 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj145 2023-09-22T21:47:10.264 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:k1f/blEJ30Os3bVxnqJ0hxN9wtg= 2023-09-22T21:47:10.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:k1f/blEJ30Os3bVxnqJ0hxN9wtg=', 'Content-Length': '0'} 2023-09-22T21:47:10.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9f88677aea9b6c2-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.267 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.267 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.267 DEBUG:boto:path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:47:10.267 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:47:10.267 DEBUG:boto:Method: GET 2023-09-22T21:47:10.268 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj145 2023-09-22T21:47:10.268 DEBUG:boto:Data: 2023-09-22T21:47:10.268 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.268 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.268 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.268 DEBUG:boto:Params: {} 2023-09-22T21:47:10.268 DEBUG:boto:Token: None 2023-09-22T21:47:10.268 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.268 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ceaf578b7884fb3-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.273 DEBUG:boto:b'ffzdqu-59obj1451000falseobj1452023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:10.273 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.274 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.274 DEBUG:boto:path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:47:10.274 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:47:10.274 DEBUG:boto:Method: GET 2023-09-22T21:47:10.274 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj145 2023-09-22T21:47:10.274 DEBUG:boto:Data: 2023-09-22T21:47:10.274 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.274 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.274 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.275 DEBUG:boto:Params: {} 2023-09-22T21:47:10.275 DEBUG:boto:Token: None 2023-09-22T21:47:10.275 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.275 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000204c7353751d736a-00650e0b5e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.284 DEBUG:boto:b'ffzdqu-59obj1451000falseobj1452023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:10.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-22T21:47:10.284 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:47:10.285 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:47:10.285 DEBUG:boto:path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.285 DEBUG:boto:auth_path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.285 DEBUG:boto:Method: GET 2023-09-22T21:47:10.285 DEBUG:boto:Path: /ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.285 DEBUG:boto:Data: 2023-09-22T21:47:10.285 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.285 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.285 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.285 DEBUG:boto:Params: {} 2023-09-22T21:47:10.286 DEBUG:boto:Token: None 2023-09-22T21:47:10.286 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.286 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7kacbFLnaPlY0eLTLzxvDFfRc14= 2023-09-22T21:47:10.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7kacbFLnaPlY0eLTLzxvDFfRc14=', 'Content-Length': '0'} 2023-09-22T21:47:10.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4794ea4fe529520-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.289 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:47:10.289 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:47:10.289 DEBUG:boto:path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.290 DEBUG:boto:auth_path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.290 DEBUG:boto:Method: GET 2023-09-22T21:47:10.290 DEBUG:boto:Path: /ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.290 DEBUG:boto:Data: 2023-09-22T21:47:10.290 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.290 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.290 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.290 DEBUG:boto:Params: {} 2023-09-22T21:47:10.290 DEBUG:boto:Token: None 2023-09-22T21:47:10.291 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.291 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7kacbFLnaPlY0eLTLzxvDFfRc14= 2023-09-22T21:47:10.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7kacbFLnaPlY0eLTLzxvDFfRc14=', 'Content-Length': '0'} 2023-09-22T21:47:10.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004d667951cfb76ae-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.294 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:47:10.294 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:47:10.294 DEBUG:boto:path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.294 DEBUG:boto:auth_path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.294 DEBUG:boto:Method: HEAD 2023-09-22T21:47:10.294 DEBUG:boto:Path: /ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.294 DEBUG:boto:Data: 2023-09-22T21:47:10.294 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.295 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.295 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.295 DEBUG:boto:Params: {} 2023-09-22T21:47:10.295 DEBUG:boto:Token: None 2023-09-22T21:47:10.295 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.295 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7ySWGgWdgk55wcsOuHody0hkUBY= 2023-09-22T21:47:10.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7ySWGgWdgk55wcsOuHody0hkUBY=', 'Content-Length': '0'} 2023-09-22T21:47:10.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001694ca18aa6d8f0a-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.298 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:47:10.298 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:47:10.298 DEBUG:boto:path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.298 DEBUG:boto:auth_path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.298 DEBUG:boto:Method: HEAD 2023-09-22T21:47:10.299 DEBUG:boto:Path: /ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.299 DEBUG:boto:Data: 2023-09-22T21:47:10.299 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.299 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.299 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.299 DEBUG:boto:Params: {} 2023-09-22T21:47:10.299 DEBUG:boto:Token: None 2023-09-22T21:47:10.299 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.299 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7ySWGgWdgk55wcsOuHody0hkUBY= 2023-09-22T21:47:10.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7ySWGgWdgk55wcsOuHody0hkUBY=', 'Content-Length': '0'} 2023-09-22T21:47:10.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e7b000f40975a7d-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-22T21:47:10.303 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:47:10.303 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:47:10.303 DEBUG:boto:path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.303 DEBUG:boto:auth_path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.303 DEBUG:boto:Method: GET 2023-09-22T21:47:10.303 DEBUG:boto:Path: /ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.303 DEBUG:boto:Data: 2023-09-22T21:47:10.303 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.303 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.304 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.304 DEBUG:boto:Params: {} 2023-09-22T21:47:10.304 DEBUG:boto:Token: None 2023-09-22T21:47:10.304 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.304 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7kacbFLnaPlY0eLTLzxvDFfRc14= 2023-09-22T21:47:10.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7kacbFLnaPlY0eLTLzxvDFfRc14=', 'Content-Length': '0'} 2023-09-22T21:47:10.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000accbd7fe427627da-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.307 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:47:10.307 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:47:10.307 DEBUG:boto:path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.307 DEBUG:boto:auth_path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.307 DEBUG:boto:Method: GET 2023-09-22T21:47:10.307 DEBUG:boto:Path: /ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.308 DEBUG:boto:Data: 2023-09-22T21:47:10.308 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.308 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.308 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.308 DEBUG:boto:Params: {} 2023-09-22T21:47:10.308 DEBUG:boto:Token: None 2023-09-22T21:47:10.308 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj155?versionId=null 2023-09-22T21:47:10.308 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7kacbFLnaPlY0eLTLzxvDFfRc14= 2023-09-22T21:47:10.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7kacbFLnaPlY0eLTLzxvDFfRc14=', 'Content-Length': '0'} 2023-09-22T21:47:10.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000599175aa5343a541-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.354 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.355 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.355 DEBUG:boto:path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:47:10.355 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:47:10.355 DEBUG:boto:Method: GET 2023-09-22T21:47:10.355 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj155 2023-09-22T21:47:10.355 DEBUG:boto:Data: 2023-09-22T21:47:10.355 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.355 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.355 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.355 DEBUG:boto:Params: {} 2023-09-22T21:47:10.356 DEBUG:boto:Token: None 2023-09-22T21:47:10.356 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.356 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008df32e5a790612ce-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.361 DEBUG:boto:b'ffzdqu-59obj1551000falseobj1552023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:10.361 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.361 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.362 DEBUG:boto:path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:47:10.362 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:47:10.362 DEBUG:boto:Method: GET 2023-09-22T21:47:10.362 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj155 2023-09-22T21:47:10.362 DEBUG:boto:Data: 2023-09-22T21:47:10.362 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.362 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.362 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.363 DEBUG:boto:Params: {} 2023-09-22T21:47:10.363 DEBUG:boto:Token: None 2023-09-22T21:47:10.363 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.363 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093ee97215a503777-00650e0b5e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.372 DEBUG:boto:b'ffzdqu-59obj1551000falseobj1552023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:10.372 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:47:10.372 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:47:10.372 DEBUG:boto:Method: HEAD 2023-09-22T21:47:10.373 DEBUG:boto:Path: /ffzdqu-59/obj155 2023-09-22T21:47:10.373 DEBUG:boto:Data: 2023-09-22T21:47:10.373 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.373 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.373 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.373 DEBUG:boto:Params: {} 2023-09-22T21:47:10.373 DEBUG:boto:Token: None 2023-09-22T21:47:10.373 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj155 2023-09-22T21:47:10.374 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KLyRRYo+CpSnUvyNu+f/BeII3NE= 2023-09-22T21:47:10.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KLyRRYo+CpSnUvyNu+f/BeII3NE=', 'Content-Length': '0'} 2023-09-22T21:47:10.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088d1f672fdd06e20-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.377 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:47:10.377 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:47:10.377 DEBUG:boto:Method: HEAD 2023-09-22T21:47:10.377 DEBUG:boto:Path: /ffzdqu-59/obj155 2023-09-22T21:47:10.377 DEBUG:boto:Data: 2023-09-22T21:47:10.377 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.377 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.377 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.378 DEBUG:boto:Params: {} 2023-09-22T21:47:10.378 DEBUG:boto:Token: None 2023-09-22T21:47:10.378 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj155 2023-09-22T21:47:10.378 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KLyRRYo+CpSnUvyNu+f/BeII3NE= 2023-09-22T21:47:10.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KLyRRYo+CpSnUvyNu+f/BeII3NE=', 'Content-Length': '0'} 2023-09-22T21:47:10.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d5ebd3a38396b40-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-22T21:47:10.381 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:47:10.381 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:47:10.381 DEBUG:boto:Method: GET 2023-09-22T21:47:10.382 DEBUG:boto:Path: /ffzdqu-59/obj155 2023-09-22T21:47:10.382 DEBUG:boto:Data: 2023-09-22T21:47:10.382 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.382 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.382 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.382 DEBUG:boto:Params: {} 2023-09-22T21:47:10.382 DEBUG:boto:Token: None 2023-09-22T21:47:10.382 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj155 2023-09-22T21:47:10.383 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WKFsELLUkTARL4Xy5aN65QUU4P0= 2023-09-22T21:47:10.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WKFsELLUkTARL4Xy5aN65QUU4P0=', 'Content-Length': '0'} 2023-09-22T21:47:10.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f84593af5ecb6af-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.426 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:47:10.427 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:47:10.427 DEBUG:boto:Method: GET 2023-09-22T21:47:10.427 DEBUG:boto:Path: /ffzdqu-59/obj155 2023-09-22T21:47:10.427 DEBUG:boto:Data: 2023-09-22T21:47:10.427 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.427 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.427 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.428 DEBUG:boto:Params: {} 2023-09-22T21:47:10.428 DEBUG:boto:Token: None 2023-09-22T21:47:10.428 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj155 2023-09-22T21:47:10.428 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WKFsELLUkTARL4Xy5aN65QUU4P0= 2023-09-22T21:47:10.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WKFsELLUkTARL4Xy5aN65QUU4P0=', 'Content-Length': '0'} 2023-09-22T21:47:10.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000655b8f4926cb764f-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.432 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.432 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.432 DEBUG:boto:path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:47:10.432 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:47:10.432 DEBUG:boto:Method: GET 2023-09-22T21:47:10.432 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj155 2023-09-22T21:47:10.432 DEBUG:boto:Data: 2023-09-22T21:47:10.433 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.433 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.433 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.433 DEBUG:boto:Params: {} 2023-09-22T21:47:10.433 DEBUG:boto:Token: None 2023-09-22T21:47:10.433 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.433 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038e0b5a16de6e2d2-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.438 DEBUG:boto:b'ffzdqu-59obj1551000falseobj1552023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:10.438 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.439 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.439 DEBUG:boto:path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:47:10.439 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:47:10.439 DEBUG:boto:Method: GET 2023-09-22T21:47:10.439 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj155 2023-09-22T21:47:10.439 DEBUG:boto:Data: 2023-09-22T21:47:10.439 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.440 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.440 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.440 DEBUG:boto:Params: {} 2023-09-22T21:47:10.440 DEBUG:boto:Token: None 2023-09-22T21:47:10.440 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.440 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070eb1fb7f5340f41-00650e0b5e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.490 DEBUG:boto:b'ffzdqu-59obj1551000falseobj1552023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:10.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:47:10.491 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:47:10.491 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:47:10.491 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.491 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.491 DEBUG:boto:Method: GET 2023-09-22T21:47:10.491 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.491 DEBUG:boto:Data: 2023-09-22T21:47:10.492 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.492 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.492 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.492 DEBUG:boto:Params: {} 2023-09-22T21:47:10.492 DEBUG:boto:Token: None 2023-09-22T21:47:10.492 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.492 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6pbMOLQPm/pEdJSOBuVPPxkLAsc= 2023-09-22T21:47:10.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6pbMOLQPm/pEdJSOBuVPPxkLAsc=', 'Content-Length': '0'} 2023-09-22T21:47:10.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002674185306022bbb-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.495 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:47:10.496 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:47:10.496 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.496 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.496 DEBUG:boto:Method: GET 2023-09-22T21:47:10.496 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.496 DEBUG:boto:Data: 2023-09-22T21:47:10.496 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.496 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.496 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.496 DEBUG:boto:Params: {} 2023-09-22T21:47:10.497 DEBUG:boto:Token: None 2023-09-22T21:47:10.497 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.497 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6pbMOLQPm/pEdJSOBuVPPxkLAsc= 2023-09-22T21:47:10.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6pbMOLQPm/pEdJSOBuVPPxkLAsc=', 'Content-Length': '0'} 2023-09-22T21:47:10.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfd1d9bbd516f64a-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.500 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:47:10.500 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:47:10.500 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.500 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.501 DEBUG:boto:Method: HEAD 2023-09-22T21:47:10.501 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.501 DEBUG:boto:Data: 2023-09-22T21:47:10.501 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.501 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.501 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.501 DEBUG:boto:Params: {} 2023-09-22T21:47:10.501 DEBUG:boto:Token: None 2023-09-22T21:47:10.501 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.502 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b+JWNe6cz+OZJexPCt1Lmfe5CTk= 2023-09-22T21:47:10.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b+JWNe6cz+OZJexPCt1Lmfe5CTk=', 'Content-Length': '0'} 2023-09-22T21:47:10.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f31dcd8e86f4026-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.504 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:47:10.505 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:47:10.505 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.505 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.505 DEBUG:boto:Method: HEAD 2023-09-22T21:47:10.505 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.505 DEBUG:boto:Data: 2023-09-22T21:47:10.505 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.505 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.505 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.505 DEBUG:boto:Params: {} 2023-09-22T21:47:10.506 DEBUG:boto:Token: None 2023-09-22T21:47:10.506 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.506 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b+JWNe6cz+OZJexPCt1Lmfe5CTk= 2023-09-22T21:47:10.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b+JWNe6cz+OZJexPCt1Lmfe5CTk=', 'Content-Length': '0'} 2023-09-22T21:47:10.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077a1d50564cb270b-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:47:10.509 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:47:10.509 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:47:10.509 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.509 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.510 DEBUG:boto:Method: GET 2023-09-22T21:47:10.510 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.510 DEBUG:boto:Data: 2023-09-22T21:47:10.510 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.510 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.510 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.510 DEBUG:boto:Params: {} 2023-09-22T21:47:10.510 DEBUG:boto:Token: None 2023-09-22T21:47:10.510 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.511 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6pbMOLQPm/pEdJSOBuVPPxkLAsc= 2023-09-22T21:47:10.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6pbMOLQPm/pEdJSOBuVPPxkLAsc=', 'Content-Length': '0'} 2023-09-22T21:47:10.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dae77e11bc71183b-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.514 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:47:10.514 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:47:10.514 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.514 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.514 DEBUG:boto:Method: GET 2023-09-22T21:47:10.514 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.514 DEBUG:boto:Data: 2023-09-22T21:47:10.515 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.515 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.515 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.515 DEBUG:boto:Params: {} 2023-09-22T21:47:10.515 DEBUG:boto:Token: None 2023-09-22T21:47:10.515 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:47:10.515 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6pbMOLQPm/pEdJSOBuVPPxkLAsc= 2023-09-22T21:47:10.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6pbMOLQPm/pEdJSOBuVPPxkLAsc=', 'Content-Length': '0'} 2023-09-22T21:47:10.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e05c9896bcdcd746-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.519 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.519 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.519 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:47:10.519 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:47:10.519 DEBUG:boto:Method: GET 2023-09-22T21:47:10.519 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:47:10.519 DEBUG:boto:Data: 2023-09-22T21:47:10.520 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.520 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.520 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.520 DEBUG:boto:Params: {} 2023-09-22T21:47:10.520 DEBUG:boto:Token: None 2023-09-22T21:47:10.520 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.520 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d54bd96b7852d83-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.525 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:10.526 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.526 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.526 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:47:10.526 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:47:10.526 DEBUG:boto:Method: GET 2023-09-22T21:47:10.526 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:47:10.527 DEBUG:boto:Data: 2023-09-22T21:47:10.527 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.527 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.527 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.527 DEBUG:boto:Params: {} 2023-09-22T21:47:10.527 DEBUG:boto:Token: None 2023-09-22T21:47:10.527 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.527 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6ad6a38ab7efa1c-00650e0b5e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.582 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:10.583 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:47:10.583 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:47:10.583 DEBUG:boto:Method: HEAD 2023-09-22T21:47:10.583 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:47:10.583 DEBUG:boto:Data: 2023-09-22T21:47:10.583 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.583 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.583 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.584 DEBUG:boto:Params: {} 2023-09-22T21:47:10.584 DEBUG:boto:Token: None 2023-09-22T21:47:10.584 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj17 2023-09-22T21:47:10.584 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:48MTIMJrDiuu7+0/aXQ1W/zJ6Ok= 2023-09-22T21:47:10.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:48MTIMJrDiuu7+0/aXQ1W/zJ6Ok=', 'Content-Length': '0'} 2023-09-22T21:47:10.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b85857707e64057-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.587 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:47:10.587 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:47:10.588 DEBUG:boto:Method: HEAD 2023-09-22T21:47:10.588 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:47:10.588 DEBUG:boto:Data: 2023-09-22T21:47:10.588 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.588 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.588 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.588 DEBUG:boto:Params: {} 2023-09-22T21:47:10.588 DEBUG:boto:Token: None 2023-09-22T21:47:10.588 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj17 2023-09-22T21:47:10.589 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:48MTIMJrDiuu7+0/aXQ1W/zJ6Ok= 2023-09-22T21:47:10.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:48MTIMJrDiuu7+0/aXQ1W/zJ6Ok=', 'Content-Length': '0'} 2023-09-22T21:47:10.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5c36421c6bbc395-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:47:10.592 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:47:10.592 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:47:10.592 DEBUG:boto:Method: GET 2023-09-22T21:47:10.592 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:47:10.592 DEBUG:boto:Data: 2023-09-22T21:47:10.592 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.593 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.593 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.593 DEBUG:boto:Params: {} 2023-09-22T21:47:10.593 DEBUG:boto:Token: None 2023-09-22T21:47:10.593 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj17 2023-09-22T21:47:10.593 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:979/luvc0t6cC7B+/OeimA6/LGQ= 2023-09-22T21:47:10.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:979/luvc0t6cC7B+/OeimA6/LGQ=', 'Content-Length': '0'} 2023-09-22T21:47:10.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001773f9069ff4a915-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.638 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:47:10.639 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:47:10.639 DEBUG:boto:Method: GET 2023-09-22T21:47:10.639 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:47:10.639 DEBUG:boto:Data: 2023-09-22T21:47:10.639 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.639 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.639 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.639 DEBUG:boto:Params: {} 2023-09-22T21:47:10.640 DEBUG:boto:Token: None 2023-09-22T21:47:10.640 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj17 2023-09-22T21:47:10.640 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:979/luvc0t6cC7B+/OeimA6/LGQ= 2023-09-22T21:47:10.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:979/luvc0t6cC7B+/OeimA6/LGQ=', 'Content-Length': '0'} 2023-09-22T21:47:10.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c4cc107b4f8ac0e-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.643 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.643 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.643 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:47:10.643 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:47:10.644 DEBUG:boto:Method: GET 2023-09-22T21:47:10.644 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:47:10.644 DEBUG:boto:Data: 2023-09-22T21:47:10.644 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.644 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.644 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.644 DEBUG:boto:Params: {} 2023-09-22T21:47:10.644 DEBUG:boto:Token: None 2023-09-22T21:47:10.644 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.645 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087994c7da476ab68-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.650 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:10.650 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.650 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.650 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:47:10.651 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:47:10.651 DEBUG:boto:Method: GET 2023-09-22T21:47:10.651 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:47:10.651 DEBUG:boto:Data: 2023-09-22T21:47:10.651 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.651 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.651 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.651 DEBUG:boto:Params: {} 2023-09-22T21:47:10.651 DEBUG:boto:Token: None 2023-09-22T21:47:10.651 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.652 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000601a1c3ca400c9ce-00650e0b5e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.660 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:10.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T21:47:10.661 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:47:10.661 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:47:10.661 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.661 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.661 DEBUG:boto:Method: GET 2023-09-22T21:47:10.661 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.661 DEBUG:boto:Data: 2023-09-22T21:47:10.662 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.662 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.662 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.662 DEBUG:boto:Params: {} 2023-09-22T21:47:10.662 DEBUG:boto:Token: None 2023-09-22T21:47:10.662 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.662 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GTRIxWcFshWZwlME5F2z5hZH5P8= 2023-09-22T21:47:10.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GTRIxWcFshWZwlME5F2z5hZH5P8=', 'Content-Length': '0'} 2023-09-22T21:47:10.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbdc5aa910197dbc-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.665 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:47:10.666 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:47:10.666 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.666 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.666 DEBUG:boto:Method: GET 2023-09-22T21:47:10.666 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.666 DEBUG:boto:Data: 2023-09-22T21:47:10.666 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.666 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.666 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.666 DEBUG:boto:Params: {} 2023-09-22T21:47:10.667 DEBUG:boto:Token: None 2023-09-22T21:47:10.667 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.667 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GTRIxWcFshWZwlME5F2z5hZH5P8= 2023-09-22T21:47:10.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GTRIxWcFshWZwlME5F2z5hZH5P8=', 'Content-Length': '0'} 2023-09-22T21:47:10.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a099f67535a14b7-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.710 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:47:10.710 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:47:10.711 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.711 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.711 DEBUG:boto:Method: HEAD 2023-09-22T21:47:10.711 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.711 DEBUG:boto:Data: 2023-09-22T21:47:10.711 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.711 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.711 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.711 DEBUG:boto:Params: {} 2023-09-22T21:47:10.712 DEBUG:boto:Token: None 2023-09-22T21:47:10.712 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.712 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bC2pMqCbOTjMFcF6KmwFPXpaAH0= 2023-09-22T21:47:10.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bC2pMqCbOTjMFcF6KmwFPXpaAH0=', 'Content-Length': '0'} 2023-09-22T21:47:10.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019ced488cc8be6b9-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.715 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:47:10.715 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:47:10.716 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.716 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.716 DEBUG:boto:Method: HEAD 2023-09-22T21:47:10.716 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.716 DEBUG:boto:Data: 2023-09-22T21:47:10.716 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.716 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.716 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.716 DEBUG:boto:Params: {} 2023-09-22T21:47:10.717 DEBUG:boto:Token: None 2023-09-22T21:47:10.717 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.717 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bC2pMqCbOTjMFcF6KmwFPXpaAH0= 2023-09-22T21:47:10.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bC2pMqCbOTjMFcF6KmwFPXpaAH0=', 'Content-Length': '0'} 2023-09-22T21:47:10.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d79e99a87c94eb2-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T21:47:10.720 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:47:10.720 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:47:10.720 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.720 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.721 DEBUG:boto:Method: GET 2023-09-22T21:47:10.721 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.721 DEBUG:boto:Data: 2023-09-22T21:47:10.721 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.721 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.721 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.721 DEBUG:boto:Params: {} 2023-09-22T21:47:10.721 DEBUG:boto:Token: None 2023-09-22T21:47:10.721 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.721 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GTRIxWcFshWZwlME5F2z5hZH5P8= 2023-09-22T21:47:10.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GTRIxWcFshWZwlME5F2z5hZH5P8=', 'Content-Length': '0'} 2023-09-22T21:47:10.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012afa02445b74cde-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.766 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:47:10.767 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:47:10.767 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.767 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.767 DEBUG:boto:Method: GET 2023-09-22T21:47:10.767 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.767 DEBUG:boto:Data: 2023-09-22T21:47:10.767 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.767 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.767 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.768 DEBUG:boto:Params: {} 2023-09-22T21:47:10.768 DEBUG:boto:Token: None 2023-09-22T21:47:10.768 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:47:10.768 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GTRIxWcFshWZwlME5F2z5hZH5P8= 2023-09-22T21:47:10.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GTRIxWcFshWZwlME5F2z5hZH5P8=', 'Content-Length': '0'} 2023-09-22T21:47:10.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000538f0dd7db70e4be-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.771 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.771 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.771 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:47:10.771 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:47:10.771 DEBUG:boto:Method: GET 2023-09-22T21:47:10.771 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:47:10.772 DEBUG:boto:Data: 2023-09-22T21:47:10.772 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.772 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.772 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.772 DEBUG:boto:Params: {} 2023-09-22T21:47:10.772 DEBUG:boto:Token: None 2023-09-22T21:47:10.772 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.772 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1aa7f8dfa9242a8-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.777 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:10.778 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.778 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.778 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:47:10.778 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:47:10.778 DEBUG:boto:Method: GET 2023-09-22T21:47:10.778 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:47:10.779 DEBUG:boto:Data: 2023-09-22T21:47:10.779 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.779 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.779 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.779 DEBUG:boto:Params: {} 2023-09-22T21:47:10.779 DEBUG:boto:Token: None 2023-09-22T21:47:10.779 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.779 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090a90d65d2bb7169-00650e0b5e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.789 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:10.789 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:47:10.789 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:47:10.789 DEBUG:boto:Method: HEAD 2023-09-22T21:47:10.789 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:47:10.789 DEBUG:boto:Data: 2023-09-22T21:47:10.790 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.790 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.790 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.790 DEBUG:boto:Params: {} 2023-09-22T21:47:10.790 DEBUG:boto:Token: None 2023-09-22T21:47:10.790 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj19 2023-09-22T21:47:10.790 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:17dK6nsyPOPE1rHgZlWOT8kBCK0= 2023-09-22T21:47:10.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:17dK6nsyPOPE1rHgZlWOT8kBCK0=', 'Content-Length': '0'} 2023-09-22T21:47:10.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d2e6459ccc3c1a8-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.793 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:47:10.793 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:47:10.794 DEBUG:boto:Method: HEAD 2023-09-22T21:47:10.794 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:47:10.794 DEBUG:boto:Data: 2023-09-22T21:47:10.794 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.794 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.794 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.794 DEBUG:boto:Params: {} 2023-09-22T21:47:10.794 DEBUG:boto:Token: None 2023-09-22T21:47:10.795 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj19 2023-09-22T21:47:10.795 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:17dK6nsyPOPE1rHgZlWOT8kBCK0= 2023-09-22T21:47:10.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:17dK6nsyPOPE1rHgZlWOT8kBCK0=', 'Content-Length': '0'} 2023-09-22T21:47:10.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd94dd26fa35b1ec-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T21:47:10.798 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:47:10.798 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:47:10.798 DEBUG:boto:Method: GET 2023-09-22T21:47:10.798 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:47:10.798 DEBUG:boto:Data: 2023-09-22T21:47:10.799 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.799 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.799 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.799 DEBUG:boto:Params: {} 2023-09-22T21:47:10.799 DEBUG:boto:Token: None 2023-09-22T21:47:10.799 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj19 2023-09-22T21:47:10.799 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kjusvUbViwG+2eKX9gIHFx3aJgM= 2023-09-22T21:47:10.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kjusvUbViwG+2eKX9gIHFx3aJgM=', 'Content-Length': '0'} 2023-09-22T21:47:10.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063a553b8e7a34b48-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.802 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:47:10.803 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:47:10.803 DEBUG:boto:Method: GET 2023-09-22T21:47:10.803 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:47:10.803 DEBUG:boto:Data: 2023-09-22T21:47:10.803 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.803 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.803 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.803 DEBUG:boto:Params: {} 2023-09-22T21:47:10.803 DEBUG:boto:Token: None 2023-09-22T21:47:10.804 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj19 2023-09-22T21:47:10.804 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kjusvUbViwG+2eKX9gIHFx3aJgM= 2023-09-22T21:47:10.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kjusvUbViwG+2eKX9gIHFx3aJgM=', 'Content-Length': '0'} 2023-09-22T21:47:10.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053f818b83a5eeb8f-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.850 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.850 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.851 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:47:10.851 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:47:10.851 DEBUG:boto:Method: GET 2023-09-22T21:47:10.851 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:47:10.851 DEBUG:boto:Data: 2023-09-22T21:47:10.851 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.851 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.851 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.851 DEBUG:boto:Params: {} 2023-09-22T21:47:10.852 DEBUG:boto:Token: None 2023-09-22T21:47:10.852 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.852 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000835faec6f01a6b2c-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.857 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:10.857 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.858 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.858 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:47:10.858 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:47:10.858 DEBUG:boto:Method: GET 2023-09-22T21:47:10.858 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:47:10.858 DEBUG:boto:Data: 2023-09-22T21:47:10.858 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.859 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.859 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.859 DEBUG:boto:Params: {} 2023-09-22T21:47:10.859 DEBUG:boto:Token: None 2023-09-22T21:47:10.859 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.859 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f416ebe849621ec-00650e0b5e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.868 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:10.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:47:10.868 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:47:10.869 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:47:10.869 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.869 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.869 DEBUG:boto:Method: GET 2023-09-22T21:47:10.869 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.869 DEBUG:boto:Data: 2023-09-22T21:47:10.869 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.869 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.869 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.870 DEBUG:boto:Params: {} 2023-09-22T21:47:10.870 DEBUG:boto:Token: None 2023-09-22T21:47:10.870 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.870 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9l/4x5OcLhf6GRMM2gL7DsUDl5E= 2023-09-22T21:47:10.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9l/4x5OcLhf6GRMM2gL7DsUDl5E=', 'Content-Length': '0'} 2023-09-22T21:47:10.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000857d6bcd446d64b9-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.918 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:47:10.919 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:47:10.919 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.919 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.919 DEBUG:boto:Method: GET 2023-09-22T21:47:10.919 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.919 DEBUG:boto:Data: 2023-09-22T21:47:10.919 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.919 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.920 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.920 DEBUG:boto:Params: {} 2023-09-22T21:47:10.920 DEBUG:boto:Token: None 2023-09-22T21:47:10.920 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.920 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9l/4x5OcLhf6GRMM2gL7DsUDl5E= 2023-09-22T21:47:10.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9l/4x5OcLhf6GRMM2gL7DsUDl5E=', 'Content-Length': '0'} 2023-09-22T21:47:10.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084a27e83fdb871bf-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.924 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:47:10.924 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:47:10.924 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.924 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.924 DEBUG:boto:Method: HEAD 2023-09-22T21:47:10.924 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.924 DEBUG:boto:Data: 2023-09-22T21:47:10.924 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.925 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.925 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.925 DEBUG:boto:Params: {} 2023-09-22T21:47:10.925 DEBUG:boto:Token: None 2023-09-22T21:47:10.925 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.925 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6+vXXDeVfWMN+CGuzL8alugGxHM= 2023-09-22T21:47:10.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6+vXXDeVfWMN+CGuzL8alugGxHM=', 'Content-Length': '0'} 2023-09-22T21:47:10.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe7f3cb32cdb3d92-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.928 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:47:10.929 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:47:10.929 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.929 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.929 DEBUG:boto:Method: HEAD 2023-09-22T21:47:10.929 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.929 DEBUG:boto:Data: 2023-09-22T21:47:10.929 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.929 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.929 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.930 DEBUG:boto:Params: {} 2023-09-22T21:47:10.930 DEBUG:boto:Token: None 2023-09-22T21:47:10.930 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.930 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6+vXXDeVfWMN+CGuzL8alugGxHM= 2023-09-22T21:47:10.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6+vXXDeVfWMN+CGuzL8alugGxHM=', 'Content-Length': '0'} 2023-09-22T21:47:10.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000254c89b47297d53b-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:47:10.933 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:47:10.933 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:47:10.933 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.933 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.933 DEBUG:boto:Method: GET 2023-09-22T21:47:10.933 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.934 DEBUG:boto:Data: 2023-09-22T21:47:10.934 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.934 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.934 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.934 DEBUG:boto:Params: {} 2023-09-22T21:47:10.934 DEBUG:boto:Token: None 2023-09-22T21:47:10.934 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.934 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9l/4x5OcLhf6GRMM2gL7DsUDl5E= 2023-09-22T21:47:10.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9l/4x5OcLhf6GRMM2gL7DsUDl5E=', 'Content-Length': '0'} 2023-09-22T21:47:10.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070ea746f1d5c3126-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.938 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:47:10.938 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:47:10.938 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.938 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.938 DEBUG:boto:Method: GET 2023-09-22T21:47:10.938 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.939 DEBUG:boto:Data: 2023-09-22T21:47:10.939 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.939 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.939 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.939 DEBUG:boto:Params: {} 2023-09-22T21:47:10.939 DEBUG:boto:Token: None 2023-09-22T21:47:10.939 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:47:10.939 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9l/4x5OcLhf6GRMM2gL7DsUDl5E= 2023-09-22T21:47:10.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9l/4x5OcLhf6GRMM2gL7DsUDl5E=', 'Content-Length': '0'} 2023-09-22T21:47:10.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ed5373844e4a394-00650e0b5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.943 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.943 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.943 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:47:10.943 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:47:10.943 DEBUG:boto:Method: GET 2023-09-22T21:47:10.943 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:47:10.943 DEBUG:boto:Data: 2023-09-22T21:47:10.943 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.944 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:10.944 DEBUG:boto:Port: 8001 2023-09-22T21:47:10.944 DEBUG:boto:Params: {} 2023-09-22T21:47:10.944 DEBUG:boto:Token: None 2023-09-22T21:47:10.944 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.944 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007908db5804cf33d4-00650e0b5e-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:10.950 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:10.950 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:10.951 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:10.951 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:47:10.951 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:47:10.951 DEBUG:boto:Method: GET 2023-09-22T21:47:10.951 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:47:10.951 DEBUG:boto:Data: 2023-09-22T21:47:10.951 DEBUG:boto:Headers: {} 2023-09-22T21:47:10.951 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:10.952 DEBUG:boto:Port: 8000 2023-09-22T21:47:10.952 DEBUG:boto:Params: {} 2023-09-22T21:47:10.952 DEBUG:boto:Token: None 2023-09-22T21:47:10.952 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:10 GMT /ffzdqu-59/ 2023-09-22T21:47:10.952 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM= 2023-09-22T21:47:10.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/i7V4Sivr7ED/krKMTzETiLW5EM=', 'Content-Length': '0'} 2023-09-22T21:47:10.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c4018e37745017b-00650e0b5e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.002 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:11.003 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:47:11.003 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:47:11.003 DEBUG:boto:Method: HEAD 2023-09-22T21:47:11.003 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:47:11.003 DEBUG:boto:Data: 2023-09-22T21:47:11.003 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.003 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.003 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.004 DEBUG:boto:Params: {} 2023-09-22T21:47:11.004 DEBUG:boto:Token: None 2023-09-22T21:47:11.004 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj22 2023-09-22T21:47:11.004 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nhAe9w5yhQKl/K422LlRcmM4JPU= 2023-09-22T21:47:11.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nhAe9w5yhQKl/K422LlRcmM4JPU=', 'Content-Length': '0'} 2023-09-22T21:47:11.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9f67e0843c141d6-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.007 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:47:11.007 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:47:11.007 DEBUG:boto:Method: HEAD 2023-09-22T21:47:11.007 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:47:11.008 DEBUG:boto:Data: 2023-09-22T21:47:11.008 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.008 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.008 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.008 DEBUG:boto:Params: {} 2023-09-22T21:47:11.008 DEBUG:boto:Token: None 2023-09-22T21:47:11.008 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj22 2023-09-22T21:47:11.008 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nhAe9w5yhQKl/K422LlRcmM4JPU= 2023-09-22T21:47:11.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nhAe9w5yhQKl/K422LlRcmM4JPU=', 'Content-Length': '0'} 2023-09-22T21:47:11.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000155e1474d1aa2063-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:47:11.012 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:47:11.012 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:47:11.012 DEBUG:boto:Method: GET 2023-09-22T21:47:11.012 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:47:11.012 DEBUG:boto:Data: 2023-09-22T21:47:11.012 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.012 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.012 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.013 DEBUG:boto:Params: {} 2023-09-22T21:47:11.013 DEBUG:boto:Token: None 2023-09-22T21:47:11.013 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj22 2023-09-22T21:47:11.013 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J0FWSwV1BY4ZqBHyC0ti0KY41ls= 2023-09-22T21:47:11.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J0FWSwV1BY4ZqBHyC0ti0KY41ls=', 'Content-Length': '0'} 2023-09-22T21:47:11.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b25e1eb838f00efb-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.058 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:47:11.059 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:47:11.059 DEBUG:boto:Method: GET 2023-09-22T21:47:11.059 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:47:11.059 DEBUG:boto:Data: 2023-09-22T21:47:11.059 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.059 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.059 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.059 DEBUG:boto:Params: {} 2023-09-22T21:47:11.060 DEBUG:boto:Token: None 2023-09-22T21:47:11.060 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj22 2023-09-22T21:47:11.060 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J0FWSwV1BY4ZqBHyC0ti0KY41ls= 2023-09-22T21:47:11.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J0FWSwV1BY4ZqBHyC0ti0KY41ls=', 'Content-Length': '0'} 2023-09-22T21:47:11.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c53d7f897c0d950d-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.063 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:11.063 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:11.063 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:47:11.063 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:47:11.064 DEBUG:boto:Method: GET 2023-09-22T21:47:11.064 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:47:11.064 DEBUG:boto:Data: 2023-09-22T21:47:11.064 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.064 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.064 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.064 DEBUG:boto:Params: {} 2023-09-22T21:47:11.064 DEBUG:boto:Token: None 2023-09-22T21:47:11.065 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/ 2023-09-22T21:47:11.065 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0= 2023-09-22T21:47:11.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0=', 'Content-Length': '0'} 2023-09-22T21:47:11.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062478ac704b39f93-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.070 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:11.070 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:11.071 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:11.071 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:47:11.071 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:47:11.071 DEBUG:boto:Method: GET 2023-09-22T21:47:11.071 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:47:11.071 DEBUG:boto:Data: 2023-09-22T21:47:11.071 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.071 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.072 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.072 DEBUG:boto:Params: {} 2023-09-22T21:47:11.072 DEBUG:boto:Token: None 2023-09-22T21:47:11.072 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/ 2023-09-22T21:47:11.072 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0= 2023-09-22T21:47:11.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0=', 'Content-Length': '0'} 2023-09-22T21:47:11.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025c981cc09ab9306-00650e0b5f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.081 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:11.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-22T21:47:11.081 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:47:11.081 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:47:11.082 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.082 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.082 DEBUG:boto:Method: GET 2023-09-22T21:47:11.082 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.082 DEBUG:boto:Data: 2023-09-22T21:47:11.082 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.082 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.082 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.082 DEBUG:boto:Params: {} 2023-09-22T21:47:11.083 DEBUG:boto:Token: None 2023-09-22T21:47:11.083 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.083 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0mOEG3wj+8qXP9rzQCLwZSsYPk8= 2023-09-22T21:47:11.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0mOEG3wj+8qXP9rzQCLwZSsYPk8=', 'Content-Length': '0'} 2023-09-22T21:47:11.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043a348c99b7372b8-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.086 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:47:11.086 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:47:11.086 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.087 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.087 DEBUG:boto:Method: GET 2023-09-22T21:47:11.087 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.087 DEBUG:boto:Data: 2023-09-22T21:47:11.087 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.087 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.087 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.087 DEBUG:boto:Params: {} 2023-09-22T21:47:11.087 DEBUG:boto:Token: None 2023-09-22T21:47:11.088 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.088 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0mOEG3wj+8qXP9rzQCLwZSsYPk8= 2023-09-22T21:47:11.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0mOEG3wj+8qXP9rzQCLwZSsYPk8=', 'Content-Length': '0'} 2023-09-22T21:47:11.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000887e8ce3b6af7bcf-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.134 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:47:11.134 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:47:11.135 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.135 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.135 DEBUG:boto:Method: HEAD 2023-09-22T21:47:11.135 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.135 DEBUG:boto:Data: 2023-09-22T21:47:11.135 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.135 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.135 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.136 DEBUG:boto:Params: {} 2023-09-22T21:47:11.136 DEBUG:boto:Token: None 2023-09-22T21:47:11.136 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.136 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Oi5zXy7RS+XPPsAHaGEQpWO2/6U= 2023-09-22T21:47:11.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Oi5zXy7RS+XPPsAHaGEQpWO2/6U=', 'Content-Length': '0'} 2023-09-22T21:47:11.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070ff6c115e23edb5-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.139 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:47:11.139 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:47:11.140 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.140 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.140 DEBUG:boto:Method: HEAD 2023-09-22T21:47:11.140 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.140 DEBUG:boto:Data: 2023-09-22T21:47:11.140 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.140 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.140 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.140 DEBUG:boto:Params: {} 2023-09-22T21:47:11.141 DEBUG:boto:Token: None 2023-09-22T21:47:11.141 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.141 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Oi5zXy7RS+XPPsAHaGEQpWO2/6U= 2023-09-22T21:47:11.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Oi5zXy7RS+XPPsAHaGEQpWO2/6U=', 'Content-Length': '0'} 2023-09-22T21:47:11.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e26579745e7f31e3-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-22T21:47:11.145 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:47:11.145 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:47:11.145 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.145 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.145 DEBUG:boto:Method: GET 2023-09-22T21:47:11.145 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.145 DEBUG:boto:Data: 2023-09-22T21:47:11.146 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.146 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.146 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.146 DEBUG:boto:Params: {} 2023-09-22T21:47:11.146 DEBUG:boto:Token: None 2023-09-22T21:47:11.146 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.146 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0mOEG3wj+8qXP9rzQCLwZSsYPk8= 2023-09-22T21:47:11.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0mOEG3wj+8qXP9rzQCLwZSsYPk8=', 'Content-Length': '0'} 2023-09-22T21:47:11.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c786ebcfb7b6a8b-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.190 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:47:11.190 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:47:11.191 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.191 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.191 DEBUG:boto:Method: GET 2023-09-22T21:47:11.191 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.191 DEBUG:boto:Data: 2023-09-22T21:47:11.191 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.191 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.191 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.191 DEBUG:boto:Params: {} 2023-09-22T21:47:11.192 DEBUG:boto:Token: None 2023-09-22T21:47:11.192 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:47:11.192 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0mOEG3wj+8qXP9rzQCLwZSsYPk8= 2023-09-22T21:47:11.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0mOEG3wj+8qXP9rzQCLwZSsYPk8=', 'Content-Length': '0'} 2023-09-22T21:47:11.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084009472f424152f-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.195 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:11.195 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:11.195 DEBUG:boto:path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:47:11.195 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:47:11.196 DEBUG:boto:Method: GET 2023-09-22T21:47:11.196 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj23 2023-09-22T21:47:11.196 DEBUG:boto:Data: 2023-09-22T21:47:11.196 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.196 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.196 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.196 DEBUG:boto:Params: {} 2023-09-22T21:47:11.196 DEBUG:boto:Token: None 2023-09-22T21:47:11.196 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/ 2023-09-22T21:47:11.197 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0= 2023-09-22T21:47:11.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0=', 'Content-Length': '0'} 2023-09-22T21:47:11.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000673ebf5a741051ce-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.201 DEBUG:boto:b'ffzdqu-59obj231000falseobj232023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:11.201 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:11.202 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:11.202 DEBUG:boto:path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:47:11.202 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:47:11.202 DEBUG:boto:Method: GET 2023-09-22T21:47:11.202 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj23 2023-09-22T21:47:11.202 DEBUG:boto:Data: 2023-09-22T21:47:11.202 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.202 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.203 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.203 DEBUG:boto:Params: {} 2023-09-22T21:47:11.203 DEBUG:boto:Token: None 2023-09-22T21:47:11.203 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/ 2023-09-22T21:47:11.203 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0= 2023-09-22T21:47:11.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0=', 'Content-Length': '0'} 2023-09-22T21:47:11.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ceafd101b6c9936d-00650e0b5f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.212 DEBUG:boto:b'ffzdqu-59obj231000falseobj232023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:11.212 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:47:11.213 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:47:11.213 DEBUG:boto:Method: HEAD 2023-09-22T21:47:11.213 DEBUG:boto:Path: /ffzdqu-59/obj23 2023-09-22T21:47:11.213 DEBUG:boto:Data: 2023-09-22T21:47:11.213 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.213 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.213 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.213 DEBUG:boto:Params: {} 2023-09-22T21:47:11.214 DEBUG:boto:Token: None 2023-09-22T21:47:11.214 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj23 2023-09-22T21:47:11.214 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rmM9ufB1k6hAgi0LIdmgWgn6VEs= 2023-09-22T21:47:11.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rmM9ufB1k6hAgi0LIdmgWgn6VEs=', 'Content-Length': '0'} 2023-09-22T21:47:11.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a382746a854adb31-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.217 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:47:11.217 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:47:11.217 DEBUG:boto:Method: HEAD 2023-09-22T21:47:11.217 DEBUG:boto:Path: /ffzdqu-59/obj23 2023-09-22T21:47:11.217 DEBUG:boto:Data: 2023-09-22T21:47:11.217 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.217 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.217 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.217 DEBUG:boto:Params: {} 2023-09-22T21:47:11.218 DEBUG:boto:Token: None 2023-09-22T21:47:11.218 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj23 2023-09-22T21:47:11.218 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rmM9ufB1k6hAgi0LIdmgWgn6VEs= 2023-09-22T21:47:11.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rmM9ufB1k6hAgi0LIdmgWgn6VEs=', 'Content-Length': '0'} 2023-09-22T21:47:11.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e4696aebf8430d3-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-22T21:47:11.221 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:47:11.222 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:47:11.222 DEBUG:boto:Method: GET 2023-09-22T21:47:11.222 DEBUG:boto:Path: /ffzdqu-59/obj23 2023-09-22T21:47:11.222 DEBUG:boto:Data: 2023-09-22T21:47:11.222 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.222 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.222 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.222 DEBUG:boto:Params: {} 2023-09-22T21:47:11.222 DEBUG:boto:Token: None 2023-09-22T21:47:11.223 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj23 2023-09-22T21:47:11.223 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zmtLz4o5k9LhBMWq7D727fl38t0= 2023-09-22T21:47:11.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zmtLz4o5k9LhBMWq7D727fl38t0=', 'Content-Length': '0'} 2023-09-22T21:47:11.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f77146aa7df1a44-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.226 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:47:11.226 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:47:11.226 DEBUG:boto:Method: GET 2023-09-22T21:47:11.226 DEBUG:boto:Path: /ffzdqu-59/obj23 2023-09-22T21:47:11.226 DEBUG:boto:Data: 2023-09-22T21:47:11.227 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.227 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.227 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.227 DEBUG:boto:Params: {} 2023-09-22T21:47:11.227 DEBUG:boto:Token: None 2023-09-22T21:47:11.227 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj23 2023-09-22T21:47:11.227 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zmtLz4o5k9LhBMWq7D727fl38t0= 2023-09-22T21:47:11.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zmtLz4o5k9LhBMWq7D727fl38t0=', 'Content-Length': '0'} 2023-09-22T21:47:11.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8ac2076f662189b-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.270 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:11.271 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:11.271 DEBUG:boto:path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:47:11.271 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:47:11.271 DEBUG:boto:Method: GET 2023-09-22T21:47:11.271 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj23 2023-09-22T21:47:11.271 DEBUG:boto:Data: 2023-09-22T21:47:11.271 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.272 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.272 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.272 DEBUG:boto:Params: {} 2023-09-22T21:47:11.272 DEBUG:boto:Token: None 2023-09-22T21:47:11.272 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/ 2023-09-22T21:47:11.272 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0= 2023-09-22T21:47:11.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0=', 'Content-Length': '0'} 2023-09-22T21:47:11.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5f85a286ff0352c-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.278 DEBUG:boto:b'ffzdqu-59obj231000falseobj232023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:11.278 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:11.278 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:11.278 DEBUG:boto:path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:47:11.279 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:47:11.279 DEBUG:boto:Method: GET 2023-09-22T21:47:11.279 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj23 2023-09-22T21:47:11.279 DEBUG:boto:Data: 2023-09-22T21:47:11.279 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.279 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.279 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.279 DEBUG:boto:Params: {} 2023-09-22T21:47:11.279 DEBUG:boto:Token: None 2023-09-22T21:47:11.280 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/ 2023-09-22T21:47:11.280 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0= 2023-09-22T21:47:11.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0=', 'Content-Length': '0'} 2023-09-22T21:47:11.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027d804968ecb032b-00650e0b5f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.288 DEBUG:boto:b'ffzdqu-59obj231000falseobj232023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:11.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-22T21:47:11.289 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:47:11.289 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:47:11.289 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.289 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.289 DEBUG:boto:Method: GET 2023-09-22T21:47:11.289 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.289 DEBUG:boto:Data: 2023-09-22T21:47:11.289 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.290 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.290 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.290 DEBUG:boto:Params: {} 2023-09-22T21:47:11.290 DEBUG:boto:Token: None 2023-09-22T21:47:11.290 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.290 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:deOwihlUUdHzzzbYMvIZxuivs9U= 2023-09-22T21:47:11.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:deOwihlUUdHzzzbYMvIZxuivs9U=', 'Content-Length': '0'} 2023-09-22T21:47:11.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d4d67e154892ddc-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.334 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:47:11.334 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:47:11.334 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.335 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.335 DEBUG:boto:Method: GET 2023-09-22T21:47:11.335 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.335 DEBUG:boto:Data: 2023-09-22T21:47:11.335 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.335 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.335 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.335 DEBUG:boto:Params: {} 2023-09-22T21:47:11.336 DEBUG:boto:Token: None 2023-09-22T21:47:11.336 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.336 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:deOwihlUUdHzzzbYMvIZxuivs9U= 2023-09-22T21:47:11.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:deOwihlUUdHzzzbYMvIZxuivs9U=', 'Content-Length': '0'} 2023-09-22T21:47:11.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fda7597df52f2ebd-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.339 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:47:11.339 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:47:11.340 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.340 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.340 DEBUG:boto:Method: HEAD 2023-09-22T21:47:11.340 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.340 DEBUG:boto:Data: 2023-09-22T21:47:11.340 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.340 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.340 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.340 DEBUG:boto:Params: {} 2023-09-22T21:47:11.341 DEBUG:boto:Token: None 2023-09-22T21:47:11.341 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.341 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xvE4Gz/lVHUHbRQNX/a8ID5JeQU= 2023-09-22T21:47:11.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xvE4Gz/lVHUHbRQNX/a8ID5JeQU=', 'Content-Length': '0'} 2023-09-22T21:47:11.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e53768ab1c977fae-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.345 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:47:11.345 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:47:11.345 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.345 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.345 DEBUG:boto:Method: HEAD 2023-09-22T21:47:11.345 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.345 DEBUG:boto:Data: 2023-09-22T21:47:11.346 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.346 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.346 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.346 DEBUG:boto:Params: {} 2023-09-22T21:47:11.346 DEBUG:boto:Token: None 2023-09-22T21:47:11.346 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.346 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xvE4Gz/lVHUHbRQNX/a8ID5JeQU= 2023-09-22T21:47:11.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xvE4Gz/lVHUHbRQNX/a8ID5JeQU=', 'Content-Length': '0'} 2023-09-22T21:47:11.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9d419df6090389e-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-22T21:47:11.350 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:47:11.350 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:47:11.351 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.351 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.351 DEBUG:boto:Method: GET 2023-09-22T21:47:11.351 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.351 DEBUG:boto:Data: 2023-09-22T21:47:11.351 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.351 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.351 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.351 DEBUG:boto:Params: {} 2023-09-22T21:47:11.351 DEBUG:boto:Token: None 2023-09-22T21:47:11.352 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.352 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:deOwihlUUdHzzzbYMvIZxuivs9U= 2023-09-22T21:47:11.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:deOwihlUUdHzzzbYMvIZxuivs9U=', 'Content-Length': '0'} 2023-09-22T21:47:11.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca029aa6b9e74753-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.355 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:47:11.355 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:47:11.355 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.355 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.356 DEBUG:boto:Method: GET 2023-09-22T21:47:11.356 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.356 DEBUG:boto:Data: 2023-09-22T21:47:11.356 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.356 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.356 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.356 DEBUG:boto:Params: {} 2023-09-22T21:47:11.356 DEBUG:boto:Token: None 2023-09-22T21:47:11.357 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:47:11.357 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:deOwihlUUdHzzzbYMvIZxuivs9U= 2023-09-22T21:47:11.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:deOwihlUUdHzzzbYMvIZxuivs9U=', 'Content-Length': '0'} 2023-09-22T21:47:11.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7f3b1726042b311-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.360 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:11.360 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:11.360 DEBUG:boto:path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:47:11.360 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:47:11.361 DEBUG:boto:Method: GET 2023-09-22T21:47:11.361 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj29 2023-09-22T21:47:11.361 DEBUG:boto:Data: 2023-09-22T21:47:11.361 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.361 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.361 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.361 DEBUG:boto:Params: {} 2023-09-22T21:47:11.361 DEBUG:boto:Token: None 2023-09-22T21:47:11.361 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/ 2023-09-22T21:47:11.362 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0= 2023-09-22T21:47:11.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0=', 'Content-Length': '0'} 2023-09-22T21:47:11.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007130d12433239cc0-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.410 DEBUG:boto:b'ffzdqu-59obj291000falseobj292023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:11.411 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:11.411 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:11.411 DEBUG:boto:path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:47:11.411 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:47:11.411 DEBUG:boto:Method: GET 2023-09-22T21:47:11.411 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj29 2023-09-22T21:47:11.412 DEBUG:boto:Data: 2023-09-22T21:47:11.412 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.412 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.412 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.412 DEBUG:boto:Params: {} 2023-09-22T21:47:11.412 DEBUG:boto:Token: None 2023-09-22T21:47:11.412 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/ 2023-09-22T21:47:11.412 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0= 2023-09-22T21:47:11.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0=', 'Content-Length': '0'} 2023-09-22T21:47:11.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000869eb74ea1b07dd1-00650e0b5f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.422 DEBUG:boto:b'ffzdqu-59obj291000falseobj292023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:11.422 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:47:11.422 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:47:11.423 DEBUG:boto:Method: HEAD 2023-09-22T21:47:11.423 DEBUG:boto:Path: /ffzdqu-59/obj29 2023-09-22T21:47:11.423 DEBUG:boto:Data: 2023-09-22T21:47:11.423 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.423 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.423 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.423 DEBUG:boto:Params: {} 2023-09-22T21:47:11.423 DEBUG:boto:Token: None 2023-09-22T21:47:11.424 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj29 2023-09-22T21:47:11.424 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hHPOYzeEgMFsux4aytbxZ//zAv8= 2023-09-22T21:47:11.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hHPOYzeEgMFsux4aytbxZ//zAv8=', 'Content-Length': '0'} 2023-09-22T21:47:11.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003deaef801e78bf74-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.427 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:47:11.427 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:47:11.427 DEBUG:boto:Method: HEAD 2023-09-22T21:47:11.427 DEBUG:boto:Path: /ffzdqu-59/obj29 2023-09-22T21:47:11.427 DEBUG:boto:Data: 2023-09-22T21:47:11.427 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.428 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.428 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.428 DEBUG:boto:Params: {} 2023-09-22T21:47:11.428 DEBUG:boto:Token: None 2023-09-22T21:47:11.428 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj29 2023-09-22T21:47:11.428 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hHPOYzeEgMFsux4aytbxZ//zAv8= 2023-09-22T21:47:11.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hHPOYzeEgMFsux4aytbxZ//zAv8=', 'Content-Length': '0'} 2023-09-22T21:47:11.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ee0dcbb3df4847e-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-22T21:47:11.431 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:47:11.432 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:47:11.432 DEBUG:boto:Method: GET 2023-09-22T21:47:11.432 DEBUG:boto:Path: /ffzdqu-59/obj29 2023-09-22T21:47:11.432 DEBUG:boto:Data: 2023-09-22T21:47:11.432 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.432 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.432 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.432 DEBUG:boto:Params: {} 2023-09-22T21:47:11.432 DEBUG:boto:Token: None 2023-09-22T21:47:11.433 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj29 2023-09-22T21:47:11.433 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xL3HgqZ+modtaiXE+XgfyevYJ4Y= 2023-09-22T21:47:11.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xL3HgqZ+modtaiXE+XgfyevYJ4Y=', 'Content-Length': '0'} 2023-09-22T21:47:11.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009be15f6762f9f5da-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.436 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:47:11.436 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:47:11.436 DEBUG:boto:Method: GET 2023-09-22T21:47:11.436 DEBUG:boto:Path: /ffzdqu-59/obj29 2023-09-22T21:47:11.436 DEBUG:boto:Data: 2023-09-22T21:47:11.436 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.436 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.436 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.437 DEBUG:boto:Params: {} 2023-09-22T21:47:11.437 DEBUG:boto:Token: None 2023-09-22T21:47:11.437 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj29 2023-09-22T21:47:11.437 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xL3HgqZ+modtaiXE+XgfyevYJ4Y= 2023-09-22T21:47:11.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xL3HgqZ+modtaiXE+XgfyevYJ4Y=', 'Content-Length': '0'} 2023-09-22T21:47:11.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9876b888d2f4e29-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.482 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:11.482 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:11.483 DEBUG:boto:path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:47:11.483 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:47:11.483 DEBUG:boto:Method: GET 2023-09-22T21:47:11.483 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj29 2023-09-22T21:47:11.483 DEBUG:boto:Data: 2023-09-22T21:47:11.483 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.483 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.483 DEBUG:boto:Params: {} 2023-09-22T21:47:11.484 DEBUG:boto:Token: None 2023-09-22T21:47:11.484 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/ 2023-09-22T21:47:11.484 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0= 2023-09-22T21:47:11.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0=', 'Content-Length': '0'} 2023-09-22T21:47:11.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fe0e6484cdb1366-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.490 DEBUG:boto:b'ffzdqu-59obj291000falseobj292023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:11.490 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:11.491 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:11.491 DEBUG:boto:path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:47:11.491 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:47:11.491 DEBUG:boto:Method: GET 2023-09-22T21:47:11.491 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj29 2023-09-22T21:47:11.491 DEBUG:boto:Data: 2023-09-22T21:47:11.491 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.491 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.491 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.492 DEBUG:boto:Params: {} 2023-09-22T21:47:11.492 DEBUG:boto:Token: None 2023-09-22T21:47:11.492 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/ 2023-09-22T21:47:11.492 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0= 2023-09-22T21:47:11.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0=', 'Content-Length': '0'} 2023-09-22T21:47:11.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae3aea0602cb77a3-00650e0b5f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.501 DEBUG:boto:b'ffzdqu-59obj291000falseobj292023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:11.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-22T21:47:11.502 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:47:11.502 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:47:11.502 DEBUG:boto:path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.502 DEBUG:boto:auth_path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.502 DEBUG:boto:Method: GET 2023-09-22T21:47:11.502 DEBUG:boto:Path: /ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.502 DEBUG:boto:Data: 2023-09-22T21:47:11.502 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.503 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.503 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.503 DEBUG:boto:Params: {} 2023-09-22T21:47:11.503 DEBUG:boto:Token: None 2023-09-22T21:47:11.503 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.503 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QknxC+IXI2bPqW9OrbYia2D2ueI= 2023-09-22T21:47:11.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QknxC+IXI2bPqW9OrbYia2D2ueI=', 'Content-Length': '0'} 2023-09-22T21:47:11.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000341be7cad7ae9a6d-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.546 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:47:11.546 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:47:11.547 DEBUG:boto:path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.547 DEBUG:boto:auth_path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.547 DEBUG:boto:Method: GET 2023-09-22T21:47:11.547 DEBUG:boto:Path: /ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.547 DEBUG:boto:Data: 2023-09-22T21:47:11.547 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.547 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.547 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.548 DEBUG:boto:Params: {} 2023-09-22T21:47:11.548 DEBUG:boto:Token: None 2023-09-22T21:47:11.548 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.548 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QknxC+IXI2bPqW9OrbYia2D2ueI= 2023-09-22T21:47:11.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QknxC+IXI2bPqW9OrbYia2D2ueI=', 'Content-Length': '0'} 2023-09-22T21:47:11.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007764b9480def2985-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.551 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:47:11.551 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:47:11.552 DEBUG:boto:path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.552 DEBUG:boto:auth_path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.552 DEBUG:boto:Method: HEAD 2023-09-22T21:47:11.552 DEBUG:boto:Path: /ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.552 DEBUG:boto:Data: 2023-09-22T21:47:11.552 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.552 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.552 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.552 DEBUG:boto:Params: {} 2023-09-22T21:47:11.553 DEBUG:boto:Token: None 2023-09-22T21:47:11.553 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.553 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qJGzxEgE+BT/qGpnajf+e32d8Eg= 2023-09-22T21:47:11.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qJGzxEgE+BT/qGpnajf+e32d8Eg=', 'Content-Length': '0'} 2023-09-22T21:47:11.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021770fb31973c85d-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.556 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:47:11.556 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:47:11.556 DEBUG:boto:path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.556 DEBUG:boto:auth_path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.556 DEBUG:boto:Method: HEAD 2023-09-22T21:47:11.557 DEBUG:boto:Path: /ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.557 DEBUG:boto:Data: 2023-09-22T21:47:11.557 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.557 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.557 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.557 DEBUG:boto:Params: {} 2023-09-22T21:47:11.557 DEBUG:boto:Token: None 2023-09-22T21:47:11.557 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.558 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qJGzxEgE+BT/qGpnajf+e32d8Eg= 2023-09-22T21:47:11.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qJGzxEgE+BT/qGpnajf+e32d8Eg=', 'Content-Length': '0'} 2023-09-22T21:47:11.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb7cced1efed0263-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-22T21:47:11.561 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:47:11.561 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:47:11.561 DEBUG:boto:path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.561 DEBUG:boto:auth_path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.561 DEBUG:boto:Method: GET 2023-09-22T21:47:11.561 DEBUG:boto:Path: /ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.561 DEBUG:boto:Data: 2023-09-22T21:47:11.562 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.562 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.562 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.562 DEBUG:boto:Params: {} 2023-09-22T21:47:11.562 DEBUG:boto:Token: None 2023-09-22T21:47:11.562 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.562 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QknxC+IXI2bPqW9OrbYia2D2ueI= 2023-09-22T21:47:11.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QknxC+IXI2bPqW9OrbYia2D2ueI=', 'Content-Length': '0'} 2023-09-22T21:47:11.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094c602bad4a4df33-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.565 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:47:11.566 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:47:11.566 DEBUG:boto:path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.566 DEBUG:boto:auth_path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.566 DEBUG:boto:Method: GET 2023-09-22T21:47:11.566 DEBUG:boto:Path: /ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.566 DEBUG:boto:Data: 2023-09-22T21:47:11.566 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.566 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.567 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.567 DEBUG:boto:Params: {} 2023-09-22T21:47:11.567 DEBUG:boto:Token: None 2023-09-22T21:47:11.567 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj31?versionId=null 2023-09-22T21:47:11.567 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QknxC+IXI2bPqW9OrbYia2D2ueI= 2023-09-22T21:47:11.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QknxC+IXI2bPqW9OrbYia2D2ueI=', 'Content-Length': '0'} 2023-09-22T21:47:11.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e08d3e72afcc5d10-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.570 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:11.571 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:11.571 DEBUG:boto:path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:47:11.571 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:47:11.571 DEBUG:boto:Method: GET 2023-09-22T21:47:11.571 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj31 2023-09-22T21:47:11.571 DEBUG:boto:Data: 2023-09-22T21:47:11.571 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.571 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.572 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.572 DEBUG:boto:Params: {} 2023-09-22T21:47:11.572 DEBUG:boto:Token: None 2023-09-22T21:47:11.572 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/ 2023-09-22T21:47:11.572 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0= 2023-09-22T21:47:11.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0=', 'Content-Length': '0'} 2023-09-22T21:47:11.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a9c9b5cd4c7ea04-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.578 DEBUG:boto:b'ffzdqu-59obj311000falseobj312023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:11.579 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:11.579 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:11.579 DEBUG:boto:path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:47:11.579 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:47:11.579 DEBUG:boto:Method: GET 2023-09-22T21:47:11.579 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj31 2023-09-22T21:47:11.579 DEBUG:boto:Data: 2023-09-22T21:47:11.580 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.580 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.580 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.580 DEBUG:boto:Params: {} 2023-09-22T21:47:11.580 DEBUG:boto:Token: None 2023-09-22T21:47:11.580 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/ 2023-09-22T21:47:11.580 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0= 2023-09-22T21:47:11.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0=', 'Content-Length': '0'} 2023-09-22T21:47:11.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c68c1789948d8664-00650e0b5f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.630 DEBUG:boto:b'ffzdqu-59obj311000falseobj312023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:11.631 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:47:11.631 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:47:11.631 DEBUG:boto:Method: HEAD 2023-09-22T21:47:11.631 DEBUG:boto:Path: /ffzdqu-59/obj31 2023-09-22T21:47:11.631 DEBUG:boto:Data: 2023-09-22T21:47:11.631 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.631 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.632 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.632 DEBUG:boto:Params: {} 2023-09-22T21:47:11.632 DEBUG:boto:Token: None 2023-09-22T21:47:11.632 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj31 2023-09-22T21:47:11.632 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YeVxF464tZG6/3quIdy1id2G/hQ= 2023-09-22T21:47:11.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YeVxF464tZG6/3quIdy1id2G/hQ=', 'Content-Length': '0'} 2023-09-22T21:47:11.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002468dac5ae42f63b-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.635 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:47:11.636 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:47:11.636 DEBUG:boto:Method: HEAD 2023-09-22T21:47:11.636 DEBUG:boto:Path: /ffzdqu-59/obj31 2023-09-22T21:47:11.636 DEBUG:boto:Data: 2023-09-22T21:47:11.636 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.636 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.636 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.636 DEBUG:boto:Params: {} 2023-09-22T21:47:11.637 DEBUG:boto:Token: None 2023-09-22T21:47:11.637 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj31 2023-09-22T21:47:11.637 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YeVxF464tZG6/3quIdy1id2G/hQ= 2023-09-22T21:47:11.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YeVxF464tZG6/3quIdy1id2G/hQ=', 'Content-Length': '0'} 2023-09-22T21:47:11.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000769a3e6e368a5957-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-22T21:47:11.640 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:47:11.640 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:47:11.640 DEBUG:boto:Method: GET 2023-09-22T21:47:11.640 DEBUG:boto:Path: /ffzdqu-59/obj31 2023-09-22T21:47:11.641 DEBUG:boto:Data: 2023-09-22T21:47:11.641 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.641 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.641 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.641 DEBUG:boto:Params: {} 2023-09-22T21:47:11.641 DEBUG:boto:Token: None 2023-09-22T21:47:11.641 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj31 2023-09-22T21:47:11.641 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AVg2PbmVRZEigxqzrwe2w3PTh4o= 2023-09-22T21:47:11.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AVg2PbmVRZEigxqzrwe2w3PTh4o=', 'Content-Length': '0'} 2023-09-22T21:47:11.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6bd89ca704fd061-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.686 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:47:11.687 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:47:11.687 DEBUG:boto:Method: GET 2023-09-22T21:47:11.687 DEBUG:boto:Path: /ffzdqu-59/obj31 2023-09-22T21:47:11.687 DEBUG:boto:Data: 2023-09-22T21:47:11.687 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.687 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.687 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.687 DEBUG:boto:Params: {} 2023-09-22T21:47:11.688 DEBUG:boto:Token: None 2023-09-22T21:47:11.688 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj31 2023-09-22T21:47:11.688 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AVg2PbmVRZEigxqzrwe2w3PTh4o= 2023-09-22T21:47:11.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AVg2PbmVRZEigxqzrwe2w3PTh4o=', 'Content-Length': '0'} 2023-09-22T21:47:11.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffb9ee9b10dc63e5-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.691 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:11.692 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:11.692 DEBUG:boto:path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:47:11.692 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:47:11.692 DEBUG:boto:Method: GET 2023-09-22T21:47:11.692 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj31 2023-09-22T21:47:11.692 DEBUG:boto:Data: 2023-09-22T21:47:11.692 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.692 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.692 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.693 DEBUG:boto:Params: {} 2023-09-22T21:47:11.693 DEBUG:boto:Token: None 2023-09-22T21:47:11.693 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/ 2023-09-22T21:47:11.693 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0= 2023-09-22T21:47:11.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0=', 'Content-Length': '0'} 2023-09-22T21:47:11.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000970431eaccc4da5-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.698 DEBUG:boto:b'ffzdqu-59obj311000falseobj312023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:11.699 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:11.699 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:11.699 DEBUG:boto:path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:47:11.699 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:47:11.699 DEBUG:boto:Method: GET 2023-09-22T21:47:11.699 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj31 2023-09-22T21:47:11.700 DEBUG:boto:Data: 2023-09-22T21:47:11.700 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.700 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.700 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.700 DEBUG:boto:Params: {} 2023-09-22T21:47:11.700 DEBUG:boto:Token: None 2023-09-22T21:47:11.700 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/ 2023-09-22T21:47:11.700 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0= 2023-09-22T21:47:11.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0=', 'Content-Length': '0'} 2023-09-22T21:47:11.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aea7064ff4858376-00650e0b5f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.709 DEBUG:boto:b'ffzdqu-59obj311000falseobj312023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:11.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:47:11.710 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:47:11.710 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:47:11.710 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.710 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.710 DEBUG:boto:Method: GET 2023-09-22T21:47:11.710 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.710 DEBUG:boto:Data: 2023-09-22T21:47:11.710 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.711 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.711 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.711 DEBUG:boto:Params: {} 2023-09-22T21:47:11.711 DEBUG:boto:Token: None 2023-09-22T21:47:11.711 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.711 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ins0RGnmDdgVfpkFFc87zNM11NA= 2023-09-22T21:47:11.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ins0RGnmDdgVfpkFFc87zNM11NA=', 'Content-Length': '0'} 2023-09-22T21:47:11.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000378137b0aada6865-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.714 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:47:11.714 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:47:11.715 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.715 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.715 DEBUG:boto:Method: GET 2023-09-22T21:47:11.715 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.715 DEBUG:boto:Data: 2023-09-22T21:47:11.715 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.715 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.715 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.716 DEBUG:boto:Params: {} 2023-09-22T21:47:11.716 DEBUG:boto:Token: None 2023-09-22T21:47:11.716 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.716 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ins0RGnmDdgVfpkFFc87zNM11NA= 2023-09-22T21:47:11.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ins0RGnmDdgVfpkFFc87zNM11NA=', 'Content-Length': '0'} 2023-09-22T21:47:11.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaa60feb4e299dc0-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.762 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:47:11.763 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:47:11.763 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.763 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.763 DEBUG:boto:Method: HEAD 2023-09-22T21:47:11.763 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.763 DEBUG:boto:Data: 2023-09-22T21:47:11.763 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.763 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.763 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.764 DEBUG:boto:Params: {} 2023-09-22T21:47:11.764 DEBUG:boto:Token: None 2023-09-22T21:47:11.764 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.764 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+9IGrNFU+8piiudUGjNSMBpNObo= 2023-09-22T21:47:11.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+9IGrNFU+8piiudUGjNSMBpNObo=', 'Content-Length': '0'} 2023-09-22T21:47:11.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009025b291754871f5-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.767 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:47:11.767 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:47:11.767 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.767 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.767 DEBUG:boto:Method: HEAD 2023-09-22T21:47:11.767 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.768 DEBUG:boto:Data: 2023-09-22T21:47:11.768 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.768 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.768 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.768 DEBUG:boto:Params: {} 2023-09-22T21:47:11.768 DEBUG:boto:Token: None 2023-09-22T21:47:11.768 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.768 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+9IGrNFU+8piiudUGjNSMBpNObo= 2023-09-22T21:47:11.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+9IGrNFU+8piiudUGjNSMBpNObo=', 'Content-Length': '0'} 2023-09-22T21:47:11.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b14a50aa46d1614-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:47:11.771 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:47:11.772 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:47:11.772 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.772 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.772 DEBUG:boto:Method: GET 2023-09-22T21:47:11.772 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.772 DEBUG:boto:Data: 2023-09-22T21:47:11.772 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.772 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.772 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.772 DEBUG:boto:Params: {} 2023-09-22T21:47:11.773 DEBUG:boto:Token: None 2023-09-22T21:47:11.773 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.773 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ins0RGnmDdgVfpkFFc87zNM11NA= 2023-09-22T21:47:11.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ins0RGnmDdgVfpkFFc87zNM11NA=', 'Content-Length': '0'} 2023-09-22T21:47:11.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1420344e4c3b1e5-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.818 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:47:11.818 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:47:11.819 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.819 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.819 DEBUG:boto:Method: GET 2023-09-22T21:47:11.819 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.819 DEBUG:boto:Data: 2023-09-22T21:47:11.819 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.819 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.819 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.819 DEBUG:boto:Params: {} 2023-09-22T21:47:11.820 DEBUG:boto:Token: None 2023-09-22T21:47:11.820 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:47:11.820 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ins0RGnmDdgVfpkFFc87zNM11NA= 2023-09-22T21:47:11.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ins0RGnmDdgVfpkFFc87zNM11NA=', 'Content-Length': '0'} 2023-09-22T21:47:11.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e4b21b7b3fb5519-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.823 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:11.824 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:11.824 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:47:11.824 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:47:11.824 DEBUG:boto:Method: GET 2023-09-22T21:47:11.824 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:47:11.824 DEBUG:boto:Data: 2023-09-22T21:47:11.824 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.824 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.824 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.824 DEBUG:boto:Params: {} 2023-09-22T21:47:11.825 DEBUG:boto:Token: None 2023-09-22T21:47:11.825 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/ 2023-09-22T21:47:11.825 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0= 2023-09-22T21:47:11.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0=', 'Content-Length': '0'} 2023-09-22T21:47:11.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049d127aa5dc30919-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.830 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:11.831 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:11.831 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:11.831 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:47:11.831 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:47:11.831 DEBUG:boto:Method: GET 2023-09-22T21:47:11.831 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:47:11.831 DEBUG:boto:Data: 2023-09-22T21:47:11.831 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.831 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.832 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.832 DEBUG:boto:Params: {} 2023-09-22T21:47:11.832 DEBUG:boto:Token: None 2023-09-22T21:47:11.832 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/ 2023-09-22T21:47:11.832 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0= 2023-09-22T21:47:11.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0=', 'Content-Length': '0'} 2023-09-22T21:47:11.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000039f5e3f994c0238-00650e0b5f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.844 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:11.845 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:47:11.845 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:47:11.845 DEBUG:boto:Method: HEAD 2023-09-22T21:47:11.845 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:47:11.845 DEBUG:boto:Data: 2023-09-22T21:47:11.845 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.845 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.845 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.845 DEBUG:boto:Params: {} 2023-09-22T21:47:11.846 DEBUG:boto:Token: None 2023-09-22T21:47:11.846 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj33 2023-09-22T21:47:11.846 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jc/fcHXTT+TofRLhX3bSjJE+ggI= 2023-09-22T21:47:11.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jc/fcHXTT+TofRLhX3bSjJE+ggI=', 'Content-Length': '0'} 2023-09-22T21:47:11.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017b77a1dbe8700a6-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.849 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:47:11.849 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:47:11.849 DEBUG:boto:Method: HEAD 2023-09-22T21:47:11.849 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:47:11.850 DEBUG:boto:Data: 2023-09-22T21:47:11.850 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.850 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.850 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.850 DEBUG:boto:Params: {} 2023-09-22T21:47:11.850 DEBUG:boto:Token: None 2023-09-22T21:47:11.850 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj33 2023-09-22T21:47:11.850 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jc/fcHXTT+TofRLhX3bSjJE+ggI= 2023-09-22T21:47:11.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jc/fcHXTT+TofRLhX3bSjJE+ggI=', 'Content-Length': '0'} 2023-09-22T21:47:11.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cfeea6fb3a92ee5-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:47:11.854 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:47:11.854 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:47:11.854 DEBUG:boto:Method: GET 2023-09-22T21:47:11.854 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:47:11.854 DEBUG:boto:Data: 2023-09-22T21:47:11.854 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.854 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.854 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.855 DEBUG:boto:Params: {} 2023-09-22T21:47:11.855 DEBUG:boto:Token: None 2023-09-22T21:47:11.855 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj33 2023-09-22T21:47:11.855 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1WMDVg6dZ5p1cXDE2ezfU8/uaP0= 2023-09-22T21:47:11.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1WMDVg6dZ5p1cXDE2ezfU8/uaP0=', 'Content-Length': '0'} 2023-09-22T21:47:11.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc0d970b503c6080-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.858 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:47:11.858 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:47:11.858 DEBUG:boto:Method: GET 2023-09-22T21:47:11.858 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:47:11.858 DEBUG:boto:Data: 2023-09-22T21:47:11.858 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.859 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.859 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.859 DEBUG:boto:Params: {} 2023-09-22T21:47:11.859 DEBUG:boto:Token: None 2023-09-22T21:47:11.859 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj33 2023-09-22T21:47:11.859 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1WMDVg6dZ5p1cXDE2ezfU8/uaP0= 2023-09-22T21:47:11.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1WMDVg6dZ5p1cXDE2ezfU8/uaP0=', 'Content-Length': '0'} 2023-09-22T21:47:11.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007be9b3f968a879f7-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.902 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:11.902 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:11.903 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:47:11.903 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:47:11.903 DEBUG:boto:Method: GET 2023-09-22T21:47:11.903 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:47:11.903 DEBUG:boto:Data: 2023-09-22T21:47:11.903 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.903 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.903 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.904 DEBUG:boto:Params: {} 2023-09-22T21:47:11.904 DEBUG:boto:Token: None 2023-09-22T21:47:11.904 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/ 2023-09-22T21:47:11.904 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0= 2023-09-22T21:47:11.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0=', 'Content-Length': '0'} 2023-09-22T21:47:11.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000529abd740fae7adb-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.909 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:11.910 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:11.910 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:11.910 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:47:11.910 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:47:11.910 DEBUG:boto:Method: GET 2023-09-22T21:47:11.910 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:47:11.910 DEBUG:boto:Data: 2023-09-22T21:47:11.910 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.910 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.911 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.911 DEBUG:boto:Params: {} 2023-09-22T21:47:11.911 DEBUG:boto:Token: None 2023-09-22T21:47:11.911 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/ 2023-09-22T21:47:11.911 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0= 2023-09-22T21:47:11.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Oxcm2fL0mkpD3TsKP25TmHciDg0=', 'Content-Length': '0'} 2023-09-22T21:47:11.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009035506cf793245-00650e0b5f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.920 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:11.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:47:11.921 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:47:11.921 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:47:11.921 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.921 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.921 DEBUG:boto:Method: GET 2023-09-22T21:47:11.921 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.921 DEBUG:boto:Data: 2023-09-22T21:47:11.921 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.921 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.921 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.922 DEBUG:boto:Params: {} 2023-09-22T21:47:11.922 DEBUG:boto:Token: None 2023-09-22T21:47:11.922 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.922 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pRNlqYL0w1pmf7AzI9aILrPHTww= 2023-09-22T21:47:11.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pRNlqYL0w1pmf7AzI9aILrPHTww=', 'Content-Length': '0'} 2023-09-22T21:47:11.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ed5400e696e4dbe-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.966 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:47:11.966 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:47:11.967 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.967 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.967 DEBUG:boto:Method: GET 2023-09-22T21:47:11.967 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.967 DEBUG:boto:Data: 2023-09-22T21:47:11.967 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.967 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.967 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.967 DEBUG:boto:Params: {} 2023-09-22T21:47:11.967 DEBUG:boto:Token: None 2023-09-22T21:47:11.968 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.968 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pRNlqYL0w1pmf7AzI9aILrPHTww= 2023-09-22T21:47:11.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pRNlqYL0w1pmf7AzI9aILrPHTww=', 'Content-Length': '0'} 2023-09-22T21:47:11.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007dc5318043596a6-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.971 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:47:11.971 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:47:11.971 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.972 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.972 DEBUG:boto:Method: HEAD 2023-09-22T21:47:11.972 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.972 DEBUG:boto:Data: 2023-09-22T21:47:11.972 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.972 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.972 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.972 DEBUG:boto:Params: {} 2023-09-22T21:47:11.972 DEBUG:boto:Token: None 2023-09-22T21:47:11.973 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.973 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XvAkWJcEaYC1eVavzr+83FKAVqA= 2023-09-22T21:47:11.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XvAkWJcEaYC1eVavzr+83FKAVqA=', 'Content-Length': '0'} 2023-09-22T21:47:11.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9a697098974e493-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.976 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:47:11.976 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:47:11.976 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.976 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.976 DEBUG:boto:Method: HEAD 2023-09-22T21:47:11.976 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.977 DEBUG:boto:Data: 2023-09-22T21:47:11.977 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.977 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.977 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.977 DEBUG:boto:Params: {} 2023-09-22T21:47:11.977 DEBUG:boto:Token: None 2023-09-22T21:47:11.977 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.977 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XvAkWJcEaYC1eVavzr+83FKAVqA= 2023-09-22T21:47:11.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XvAkWJcEaYC1eVavzr+83FKAVqA=', 'Content-Length': '0'} 2023-09-22T21:47:11.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b507f53c66cd9fde-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:47:11.980 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:47:11.981 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:47:11.981 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.981 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.981 DEBUG:boto:Method: GET 2023-09-22T21:47:11.981 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.981 DEBUG:boto:Data: 2023-09-22T21:47:11.982 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.982 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:11.982 DEBUG:boto:Port: 8001 2023-09-22T21:47:11.982 DEBUG:boto:Params: {} 2023-09-22T21:47:11.982 DEBUG:boto:Token: None 2023-09-22T21:47:11.982 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.982 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pRNlqYL0w1pmf7AzI9aILrPHTww= 2023-09-22T21:47:11.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pRNlqYL0w1pmf7AzI9aILrPHTww=', 'Content-Length': '0'} 2023-09-22T21:47:11.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ed236bc3ac70bb9-00650e0b5f-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:11.985 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:47:11.985 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:47:11.985 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.986 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.986 DEBUG:boto:Method: GET 2023-09-22T21:47:11.986 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.986 DEBUG:boto:Data: 2023-09-22T21:47:11.986 DEBUG:boto:Headers: {} 2023-09-22T21:47:11.986 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:11.986 DEBUG:boto:Port: 8000 2023-09-22T21:47:11.986 DEBUG:boto:Params: {} 2023-09-22T21:47:11.986 DEBUG:boto:Token: None 2023-09-22T21:47:11.987 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:11 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:47:11.987 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pRNlqYL0w1pmf7AzI9aILrPHTww= 2023-09-22T21:47:11.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pRNlqYL0w1pmf7AzI9aILrPHTww=', 'Content-Length': '0'} 2023-09-22T21:47:11.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000914bcc8bc997250c-00650e0b5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.034 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:12.035 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:12.035 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:47:12.035 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:47:12.035 DEBUG:boto:Method: GET 2023-09-22T21:47:12.035 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:47:12.035 DEBUG:boto:Data: 2023-09-22T21:47:12.035 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.036 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.036 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.036 DEBUG:boto:Params: {} 2023-09-22T21:47:12.036 DEBUG:boto:Token: None 2023-09-22T21:47:12.036 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/ 2023-09-22T21:47:12.036 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA= 2023-09-22T21:47:12.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA=', 'Content-Length': '0'} 2023-09-22T21:47:12.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9aa871b6e71f2ad-00650e0b60-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.042 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:12.042 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:12.043 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:12.043 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:47:12.043 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:47:12.043 DEBUG:boto:Method: GET 2023-09-22T21:47:12.043 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:47:12.043 DEBUG:boto:Data: 2023-09-22T21:47:12.043 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.043 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.043 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.043 DEBUG:boto:Params: {} 2023-09-22T21:47:12.044 DEBUG:boto:Token: None 2023-09-22T21:47:12.044 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/ 2023-09-22T21:47:12.044 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA= 2023-09-22T21:47:12.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA=', 'Content-Length': '0'} 2023-09-22T21:47:12.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a039378398161cb-00650e0b60-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.052 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:12.053 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:47:12.053 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:47:12.053 DEBUG:boto:Method: HEAD 2023-09-22T21:47:12.053 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:47:12.053 DEBUG:boto:Data: 2023-09-22T21:47:12.053 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.053 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.053 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.054 DEBUG:boto:Params: {} 2023-09-22T21:47:12.054 DEBUG:boto:Token: None 2023-09-22T21:47:12.054 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj34 2023-09-22T21:47:12.054 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EI8yVvtheteRdZsSFXH8/CwXoRM= 2023-09-22T21:47:12.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EI8yVvtheteRdZsSFXH8/CwXoRM=', 'Content-Length': '0'} 2023-09-22T21:47:12.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c26224636900857d-00650e0b60-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.057 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:47:12.057 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:47:12.057 DEBUG:boto:Method: HEAD 2023-09-22T21:47:12.057 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:47:12.057 DEBUG:boto:Data: 2023-09-22T21:47:12.057 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.057 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.058 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.058 DEBUG:boto:Params: {} 2023-09-22T21:47:12.058 DEBUG:boto:Token: None 2023-09-22T21:47:12.058 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj34 2023-09-22T21:47:12.058 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EI8yVvtheteRdZsSFXH8/CwXoRM= 2023-09-22T21:47:12.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EI8yVvtheteRdZsSFXH8/CwXoRM=', 'Content-Length': '0'} 2023-09-22T21:47:12.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afe9053b170b75ae-00650e0b60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:47:12.062 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:47:12.062 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:47:12.062 DEBUG:boto:Method: GET 2023-09-22T21:47:12.062 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:47:12.062 DEBUG:boto:Data: 2023-09-22T21:47:12.062 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.062 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.062 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.063 DEBUG:boto:Params: {} 2023-09-22T21:47:12.063 DEBUG:boto:Token: None 2023-09-22T21:47:12.063 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj34 2023-09-22T21:47:12.063 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HVgtOcWmTWm+nInetp5NFFskz0Q= 2023-09-22T21:47:12.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HVgtOcWmTWm+nInetp5NFFskz0Q=', 'Content-Length': '0'} 2023-09-22T21:47:12.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f0923c7e5f57a5c-00650e0b60-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.110 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:47:12.110 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:47:12.111 DEBUG:boto:Method: GET 2023-09-22T21:47:12.111 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:47:12.111 DEBUG:boto:Data: 2023-09-22T21:47:12.111 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.111 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.111 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.111 DEBUG:boto:Params: {} 2023-09-22T21:47:12.111 DEBUG:boto:Token: None 2023-09-22T21:47:12.111 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj34 2023-09-22T21:47:12.112 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HVgtOcWmTWm+nInetp5NFFskz0Q= 2023-09-22T21:47:12.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HVgtOcWmTWm+nInetp5NFFskz0Q=', 'Content-Length': '0'} 2023-09-22T21:47:12.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9c90dad16214d5b-00650e0b60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.115 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:12.115 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:12.115 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:47:12.115 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:47:12.115 DEBUG:boto:Method: GET 2023-09-22T21:47:12.115 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:47:12.115 DEBUG:boto:Data: 2023-09-22T21:47:12.115 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.116 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.116 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.116 DEBUG:boto:Params: {} 2023-09-22T21:47:12.116 DEBUG:boto:Token: None 2023-09-22T21:47:12.116 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/ 2023-09-22T21:47:12.116 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA= 2023-09-22T21:47:12.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA=', 'Content-Length': '0'} 2023-09-22T21:47:12.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eed368a3b7abb0de-00650e0b60-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.162 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:12.163 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:12.163 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:12.163 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:47:12.164 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:47:12.164 DEBUG:boto:Method: GET 2023-09-22T21:47:12.164 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:47:12.164 DEBUG:boto:Data: 2023-09-22T21:47:12.164 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.164 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.164 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.164 DEBUG:boto:Params: {} 2023-09-22T21:47:12.165 DEBUG:boto:Token: None 2023-09-22T21:47:12.165 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/ 2023-09-22T21:47:12.165 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA= 2023-09-22T21:47:12.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA=', 'Content-Length': '0'} 2023-09-22T21:47:12.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b7f2f88c8189f85-00650e0b60-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.174 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:12.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T21:47:12.175 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:47:12.175 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:47:12.175 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.175 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.175 DEBUG:boto:Method: GET 2023-09-22T21:47:12.175 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.176 DEBUG:boto:Data: 2023-09-22T21:47:12.176 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.176 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.176 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.176 DEBUG:boto:Params: {} 2023-09-22T21:47:12.176 DEBUG:boto:Token: None 2023-09-22T21:47:12.176 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.176 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sXxTw7SGINv66I9kje1D2Alno5w= 2023-09-22T21:47:12.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sXxTw7SGINv66I9kje1D2Alno5w=', 'Content-Length': '0'} 2023-09-22T21:47:12.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6f0d5b89efdf9ab-00650e0b60-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.180 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:47:12.180 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:47:12.180 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.180 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.180 DEBUG:boto:Method: GET 2023-09-22T21:47:12.180 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.180 DEBUG:boto:Data: 2023-09-22T21:47:12.181 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.181 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.181 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.181 DEBUG:boto:Params: {} 2023-09-22T21:47:12.181 DEBUG:boto:Token: None 2023-09-22T21:47:12.181 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.181 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sXxTw7SGINv66I9kje1D2Alno5w= 2023-09-22T21:47:12.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sXxTw7SGINv66I9kje1D2Alno5w=', 'Content-Length': '0'} 2023-09-22T21:47:12.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000df0298cecaf79b4-00650e0b60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.226 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:47:12.227 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:47:12.227 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.227 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.227 DEBUG:boto:Method: HEAD 2023-09-22T21:47:12.227 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.227 DEBUG:boto:Data: 2023-09-22T21:47:12.227 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.227 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.227 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.228 DEBUG:boto:Params: {} 2023-09-22T21:47:12.228 DEBUG:boto:Token: None 2023-09-22T21:47:12.228 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.228 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0RfpkQu5pEYBtSMq/44B/FXX50U= 2023-09-22T21:47:12.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0RfpkQu5pEYBtSMq/44B/FXX50U=', 'Content-Length': '0'} 2023-09-22T21:47:12.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa3995ebac5f945e-00650e0b60-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.231 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:47:12.231 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:47:12.232 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.232 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.232 DEBUG:boto:Method: HEAD 2023-09-22T21:47:12.232 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.232 DEBUG:boto:Data: 2023-09-22T21:47:12.232 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.232 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.232 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.232 DEBUG:boto:Params: {} 2023-09-22T21:47:12.233 DEBUG:boto:Token: None 2023-09-22T21:47:12.233 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.233 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0RfpkQu5pEYBtSMq/44B/FXX50U= 2023-09-22T21:47:12.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0RfpkQu5pEYBtSMq/44B/FXX50U=', 'Content-Length': '0'} 2023-09-22T21:47:12.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083a771753d77fd67-00650e0b60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T21:47:12.236 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:47:12.236 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:47:12.236 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.237 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.237 DEBUG:boto:Method: GET 2023-09-22T21:47:12.237 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.237 DEBUG:boto:Data: 2023-09-22T21:47:12.237 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.237 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.237 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.237 DEBUG:boto:Params: {} 2023-09-22T21:47:12.237 DEBUG:boto:Token: None 2023-09-22T21:47:12.238 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.238 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sXxTw7SGINv66I9kje1D2Alno5w= 2023-09-22T21:47:12.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sXxTw7SGINv66I9kje1D2Alno5w=', 'Content-Length': '0'} 2023-09-22T21:47:12.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5799d9ef0d3586b-00650e0b60-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.282 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:47:12.282 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:47:12.283 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.283 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.283 DEBUG:boto:Method: GET 2023-09-22T21:47:12.283 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.283 DEBUG:boto:Data: 2023-09-22T21:47:12.283 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.283 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.283 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.283 DEBUG:boto:Params: {} 2023-09-22T21:47:12.284 DEBUG:boto:Token: None 2023-09-22T21:47:12.284 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:47:12.284 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sXxTw7SGINv66I9kje1D2Alno5w= 2023-09-22T21:47:12.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sXxTw7SGINv66I9kje1D2Alno5w=', 'Content-Length': '0'} 2023-09-22T21:47:12.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6d47c6e751b3bc1-00650e0b60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.287 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:12.287 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:12.287 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:47:12.287 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:47:12.287 DEBUG:boto:Method: GET 2023-09-22T21:47:12.287 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:47:12.288 DEBUG:boto:Data: 2023-09-22T21:47:12.288 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.288 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.288 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.288 DEBUG:boto:Params: {} 2023-09-22T21:47:12.288 DEBUG:boto:Token: None 2023-09-22T21:47:12.288 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/ 2023-09-22T21:47:12.288 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA= 2023-09-22T21:47:12.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA=', 'Content-Length': '0'} 2023-09-22T21:47:12.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d6997ec89465cde-00650e0b60-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.293 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:12.293 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:12.293 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:12.294 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:47:12.294 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:47:12.294 DEBUG:boto:Method: GET 2023-09-22T21:47:12.294 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:47:12.294 DEBUG:boto:Data: 2023-09-22T21:47:12.294 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.294 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.294 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.294 DEBUG:boto:Params: {} 2023-09-22T21:47:12.294 DEBUG:boto:Token: None 2023-09-22T21:47:12.295 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/ 2023-09-22T21:47:12.295 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA= 2023-09-22T21:47:12.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA=', 'Content-Length': '0'} 2023-09-22T21:47:12.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000820a5dae64f16f39-00650e0b60-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.303 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:12.304 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:47:12.304 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:47:12.304 DEBUG:boto:Method: HEAD 2023-09-22T21:47:12.304 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:47:12.304 DEBUG:boto:Data: 2023-09-22T21:47:12.304 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.304 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.304 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.304 DEBUG:boto:Params: {} 2023-09-22T21:47:12.305 DEBUG:boto:Token: None 2023-09-22T21:47:12.305 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj38 2023-09-22T21:47:12.305 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G4WhzTDo9bBrvU7ouW++TcgI8V8= 2023-09-22T21:47:12.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G4WhzTDo9bBrvU7ouW++TcgI8V8=', 'Content-Length': '0'} 2023-09-22T21:47:12.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000785d1d8271079cd-00650e0b60-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.308 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:47:12.308 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:47:12.308 DEBUG:boto:Method: HEAD 2023-09-22T21:47:12.309 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:47:12.309 DEBUG:boto:Data: 2023-09-22T21:47:12.309 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.309 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.309 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.309 DEBUG:boto:Params: {} 2023-09-22T21:47:12.309 DEBUG:boto:Token: None 2023-09-22T21:47:12.309 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj38 2023-09-22T21:47:12.309 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G4WhzTDo9bBrvU7ouW++TcgI8V8= 2023-09-22T21:47:12.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G4WhzTDo9bBrvU7ouW++TcgI8V8=', 'Content-Length': '0'} 2023-09-22T21:47:12.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080c50e258f3596f7-00650e0b60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T21:47:12.312 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:47:12.313 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:47:12.313 DEBUG:boto:Method: GET 2023-09-22T21:47:12.313 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:47:12.313 DEBUG:boto:Data: 2023-09-22T21:47:12.313 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.313 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.313 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.313 DEBUG:boto:Params: {} 2023-09-22T21:47:12.313 DEBUG:boto:Token: None 2023-09-22T21:47:12.314 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj38 2023-09-22T21:47:12.314 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hiVAcC7jTr29cacP8920iqFl99w= 2023-09-22T21:47:12.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hiVAcC7jTr29cacP8920iqFl99w=', 'Content-Length': '0'} 2023-09-22T21:47:12.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb35dbae4cfeeb39-00650e0b60-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.359 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:47:12.359 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:47:12.359 DEBUG:boto:Method: GET 2023-09-22T21:47:12.359 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:47:12.359 DEBUG:boto:Data: 2023-09-22T21:47:12.359 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.359 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.359 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.360 DEBUG:boto:Params: {} 2023-09-22T21:47:12.360 DEBUG:boto:Token: None 2023-09-22T21:47:12.360 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj38 2023-09-22T21:47:12.360 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hiVAcC7jTr29cacP8920iqFl99w= 2023-09-22T21:47:12.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hiVAcC7jTr29cacP8920iqFl99w=', 'Content-Length': '0'} 2023-09-22T21:47:12.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021ae387e63457443-00650e0b60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.363 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:12.364 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:12.364 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:47:12.364 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:47:12.364 DEBUG:boto:Method: GET 2023-09-22T21:47:12.364 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:47:12.364 DEBUG:boto:Data: 2023-09-22T21:47:12.364 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.364 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.364 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.365 DEBUG:boto:Params: {} 2023-09-22T21:47:12.365 DEBUG:boto:Token: None 2023-09-22T21:47:12.365 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/ 2023-09-22T21:47:12.365 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA= 2023-09-22T21:47:12.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA=', 'Content-Length': '0'} 2023-09-22T21:47:12.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049a1aa9f030c0115-00650e0b60-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.370 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:12.371 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:12.371 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:12.371 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:47:12.371 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:47:12.371 DEBUG:boto:Method: GET 2023-09-22T21:47:12.371 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:47:12.371 DEBUG:boto:Data: 2023-09-22T21:47:12.371 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.371 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.371 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.372 DEBUG:boto:Params: {} 2023-09-22T21:47:12.372 DEBUG:boto:Token: None 2023-09-22T21:47:12.372 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/ 2023-09-22T21:47:12.372 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA= 2023-09-22T21:47:12.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA=', 'Content-Length': '0'} 2023-09-22T21:47:12.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e3e2162290b7f0e-00650e0b60-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.380 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:12.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:47:12.381 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:47:12.381 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:47:12.381 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.381 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.381 DEBUG:boto:Method: GET 2023-09-22T21:47:12.381 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.381 DEBUG:boto:Data: 2023-09-22T21:47:12.382 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.382 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.382 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.382 DEBUG:boto:Params: {} 2023-09-22T21:47:12.382 DEBUG:boto:Token: None 2023-09-22T21:47:12.382 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.382 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MRygWrMXd0Ym3dlAVSmJ5QAkTDQ= 2023-09-22T21:47:12.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MRygWrMXd0Ym3dlAVSmJ5QAkTDQ=', 'Content-Length': '0'} 2023-09-22T21:47:12.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d40d0b507892e0c4-00650e0b60-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.385 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:47:12.385 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:47:12.385 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.386 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.386 DEBUG:boto:Method: GET 2023-09-22T21:47:12.386 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.386 DEBUG:boto:Data: 2023-09-22T21:47:12.386 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.386 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.386 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.386 DEBUG:boto:Params: {} 2023-09-22T21:47:12.386 DEBUG:boto:Token: None 2023-09-22T21:47:12.387 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.387 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MRygWrMXd0Ym3dlAVSmJ5QAkTDQ= 2023-09-22T21:47:12.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MRygWrMXd0Ym3dlAVSmJ5QAkTDQ=', 'Content-Length': '0'} 2023-09-22T21:47:12.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d539812d0fbcc935-00650e0b60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.390 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:47:12.390 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:47:12.390 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.390 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.390 DEBUG:boto:Method: HEAD 2023-09-22T21:47:12.390 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.390 DEBUG:boto:Data: 2023-09-22T21:47:12.390 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.390 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.391 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.391 DEBUG:boto:Params: {} 2023-09-22T21:47:12.391 DEBUG:boto:Token: None 2023-09-22T21:47:12.391 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.391 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0SddWuzTiaeDBw6zLpSELYJZSrc= 2023-09-22T21:47:12.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0SddWuzTiaeDBw6zLpSELYJZSrc=', 'Content-Length': '0'} 2023-09-22T21:47:12.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000204cea4db2220754-00650e0b60-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.394 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:47:12.394 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:47:12.394 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.394 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.394 DEBUG:boto:Method: HEAD 2023-09-22T21:47:12.395 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.395 DEBUG:boto:Data: 2023-09-22T21:47:12.395 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.395 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.395 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.395 DEBUG:boto:Params: {} 2023-09-22T21:47:12.395 DEBUG:boto:Token: None 2023-09-22T21:47:12.395 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.395 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0SddWuzTiaeDBw6zLpSELYJZSrc= 2023-09-22T21:47:12.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0SddWuzTiaeDBw6zLpSELYJZSrc=', 'Content-Length': '0'} 2023-09-22T21:47:12.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f9074e14b91f22e-00650e0b60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:47:12.398 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:47:12.399 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:47:12.399 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.399 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.399 DEBUG:boto:Method: GET 2023-09-22T21:47:12.399 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.399 DEBUG:boto:Data: 2023-09-22T21:47:12.399 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.399 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.399 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.399 DEBUG:boto:Params: {} 2023-09-22T21:47:12.400 DEBUG:boto:Token: None 2023-09-22T21:47:12.400 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.400 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MRygWrMXd0Ym3dlAVSmJ5QAkTDQ= 2023-09-22T21:47:12.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MRygWrMXd0Ym3dlAVSmJ5QAkTDQ=', 'Content-Length': '0'} 2023-09-22T21:47:12.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c2c2b261edb464e-00650e0b60-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.446 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:47:12.446 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:47:12.447 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.447 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.447 DEBUG:boto:Method: GET 2023-09-22T21:47:12.447 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.447 DEBUG:boto:Data: 2023-09-22T21:47:12.447 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.447 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.447 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.447 DEBUG:boto:Params: {} 2023-09-22T21:47:12.447 DEBUG:boto:Token: None 2023-09-22T21:47:12.448 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:47:12.448 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MRygWrMXd0Ym3dlAVSmJ5QAkTDQ= 2023-09-22T21:47:12.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MRygWrMXd0Ym3dlAVSmJ5QAkTDQ=', 'Content-Length': '0'} 2023-09-22T21:47:12.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2dc5e6598049234-00650e0b60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.451 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:12.451 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:12.451 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:47:12.451 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:47:12.451 DEBUG:boto:Method: GET 2023-09-22T21:47:12.451 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:47:12.451 DEBUG:boto:Data: 2023-09-22T21:47:12.452 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.452 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.452 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.452 DEBUG:boto:Params: {} 2023-09-22T21:47:12.452 DEBUG:boto:Token: None 2023-09-22T21:47:12.452 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/ 2023-09-22T21:47:12.452 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA= 2023-09-22T21:47:12.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA=', 'Content-Length': '0'} 2023-09-22T21:47:12.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002041dcce87082393-00650e0b60-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.498 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:12.499 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:12.499 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:12.499 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:47:12.499 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:47:12.499 DEBUG:boto:Method: GET 2023-09-22T21:47:12.499 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:47:12.499 DEBUG:boto:Data: 2023-09-22T21:47:12.500 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.500 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.500 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.500 DEBUG:boto:Params: {} 2023-09-22T21:47:12.500 DEBUG:boto:Token: None 2023-09-22T21:47:12.500 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/ 2023-09-22T21:47:12.500 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA= 2023-09-22T21:47:12.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA=', 'Content-Length': '0'} 2023-09-22T21:47:12.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6a325110723d690-00650e0b60-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.509 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:12.509 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:47:12.510 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:47:12.510 DEBUG:boto:Method: HEAD 2023-09-22T21:47:12.510 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:47:12.510 DEBUG:boto:Data: 2023-09-22T21:47:12.510 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.510 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.510 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.510 DEBUG:boto:Params: {} 2023-09-22T21:47:12.510 DEBUG:boto:Token: None 2023-09-22T21:47:12.511 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj44 2023-09-22T21:47:12.511 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wYC84WhaqB5gk7SdyuxDh7aToFY= 2023-09-22T21:47:12.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wYC84WhaqB5gk7SdyuxDh7aToFY=', 'Content-Length': '0'} 2023-09-22T21:47:12.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7e9aa531bb66e6a-00650e0b60-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.514 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:47:12.514 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:47:12.514 DEBUG:boto:Method: HEAD 2023-09-22T21:47:12.514 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:47:12.514 DEBUG:boto:Data: 2023-09-22T21:47:12.514 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.514 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.514 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.515 DEBUG:boto:Params: {} 2023-09-22T21:47:12.515 DEBUG:boto:Token: None 2023-09-22T21:47:12.515 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj44 2023-09-22T21:47:12.515 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wYC84WhaqB5gk7SdyuxDh7aToFY= 2023-09-22T21:47:12.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wYC84WhaqB5gk7SdyuxDh7aToFY=', 'Content-Length': '0'} 2023-09-22T21:47:12.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036bb85ba47a3d2d9-00650e0b60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:47:12.518 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:47:12.518 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:47:12.519 DEBUG:boto:Method: GET 2023-09-22T21:47:12.519 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:47:12.519 DEBUG:boto:Data: 2023-09-22T21:47:12.519 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.519 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.519 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.519 DEBUG:boto:Params: {} 2023-09-22T21:47:12.519 DEBUG:boto:Token: None 2023-09-22T21:47:12.520 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj44 2023-09-22T21:47:12.520 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m9Izi/iMl2tD48BUVmAt8+Uu0c0= 2023-09-22T21:47:12.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m9Izi/iMl2tD48BUVmAt8+Uu0c0=', 'Content-Length': '0'} 2023-09-22T21:47:12.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d47717e6dbab8832-00650e0b60-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.566 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:47:12.567 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:47:12.567 DEBUG:boto:Method: GET 2023-09-22T21:47:12.567 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:47:12.567 DEBUG:boto:Data: 2023-09-22T21:47:12.567 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.567 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.567 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.567 DEBUG:boto:Params: {} 2023-09-22T21:47:12.567 DEBUG:boto:Token: None 2023-09-22T21:47:12.568 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj44 2023-09-22T21:47:12.568 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m9Izi/iMl2tD48BUVmAt8+Uu0c0= 2023-09-22T21:47:12.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m9Izi/iMl2tD48BUVmAt8+Uu0c0=', 'Content-Length': '0'} 2023-09-22T21:47:12.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc21f4a4e2048892-00650e0b60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.571 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:12.571 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:12.572 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:47:12.572 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:47:12.572 DEBUG:boto:Method: GET 2023-09-22T21:47:12.572 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:47:12.572 DEBUG:boto:Data: 2023-09-22T21:47:12.572 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.572 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.572 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.572 DEBUG:boto:Params: {} 2023-09-22T21:47:12.573 DEBUG:boto:Token: None 2023-09-22T21:47:12.573 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/ 2023-09-22T21:47:12.573 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA= 2023-09-22T21:47:12.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA=', 'Content-Length': '0'} 2023-09-22T21:47:12.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000264bf774f1971166-00650e0b60-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.578 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:12.578 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:12.579 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:12.579 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:47:12.579 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:47:12.579 DEBUG:boto:Method: GET 2023-09-22T21:47:12.579 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:47:12.579 DEBUG:boto:Data: 2023-09-22T21:47:12.579 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.579 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.579 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.579 DEBUG:boto:Params: {} 2023-09-22T21:47:12.580 DEBUG:boto:Token: None 2023-09-22T21:47:12.580 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/ 2023-09-22T21:47:12.580 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA= 2023-09-22T21:47:12.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA=', 'Content-Length': '0'} 2023-09-22T21:47:12.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000804333caf6efe723-00650e0b60-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.592 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:12.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-22T21:47:12.593 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:47:12.593 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:47:12.593 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.593 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.594 DEBUG:boto:Method: GET 2023-09-22T21:47:12.594 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.594 DEBUG:boto:Data: 2023-09-22T21:47:12.594 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.594 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.594 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.594 DEBUG:boto:Params: {} 2023-09-22T21:47:12.594 DEBUG:boto:Token: None 2023-09-22T21:47:12.595 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.595 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ANkx4JenE3MHCJT8qq8ddgYwCzc= 2023-09-22T21:47:12.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ANkx4JenE3MHCJT8qq8ddgYwCzc=', 'Content-Length': '0'} 2023-09-22T21:47:12.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031306d1f7fd75f5b-00650e0b60-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.638 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:47:12.638 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:47:12.639 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.639 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.639 DEBUG:boto:Method: GET 2023-09-22T21:47:12.639 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.639 DEBUG:boto:Data: 2023-09-22T21:47:12.639 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.639 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.639 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.639 DEBUG:boto:Params: {} 2023-09-22T21:47:12.639 DEBUG:boto:Token: None 2023-09-22T21:47:12.640 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.640 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ANkx4JenE3MHCJT8qq8ddgYwCzc= 2023-09-22T21:47:12.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ANkx4JenE3MHCJT8qq8ddgYwCzc=', 'Content-Length': '0'} 2023-09-22T21:47:12.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d93dc8206947ea37-00650e0b60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.643 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:47:12.643 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:47:12.644 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.644 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.644 DEBUG:boto:Method: HEAD 2023-09-22T21:47:12.644 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.644 DEBUG:boto:Data: 2023-09-22T21:47:12.644 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.644 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.644 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.644 DEBUG:boto:Params: {} 2023-09-22T21:47:12.645 DEBUG:boto:Token: None 2023-09-22T21:47:12.645 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.645 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kh+Hi7L41Kry4xqEmtkINGugjiY= 2023-09-22T21:47:12.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kh+Hi7L41Kry4xqEmtkINGugjiY=', 'Content-Length': '0'} 2023-09-22T21:47:12.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eeee90e92520e591-00650e0b60-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.648 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:47:12.648 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:47:12.648 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.649 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.649 DEBUG:boto:Method: HEAD 2023-09-22T21:47:12.649 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.649 DEBUG:boto:Data: 2023-09-22T21:47:12.649 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.649 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.649 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.649 DEBUG:boto:Params: {} 2023-09-22T21:47:12.649 DEBUG:boto:Token: None 2023-09-22T21:47:12.650 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.650 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kh+Hi7L41Kry4xqEmtkINGugjiY= 2023-09-22T21:47:12.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kh+Hi7L41Kry4xqEmtkINGugjiY=', 'Content-Length': '0'} 2023-09-22T21:47:12.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab339d5ae673ca1a-00650e0b60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-22T21:47:12.653 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:47:12.653 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:47:12.653 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.653 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.653 DEBUG:boto:Method: GET 2023-09-22T21:47:12.654 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.654 DEBUG:boto:Data: 2023-09-22T21:47:12.654 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.654 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.654 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.654 DEBUG:boto:Params: {} 2023-09-22T21:47:12.654 DEBUG:boto:Token: None 2023-09-22T21:47:12.654 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.654 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ANkx4JenE3MHCJT8qq8ddgYwCzc= 2023-09-22T21:47:12.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ANkx4JenE3MHCJT8qq8ddgYwCzc=', 'Content-Length': '0'} 2023-09-22T21:47:12.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053b78c599fe1bad6-00650e0b60-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.702 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:47:12.702 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:47:12.703 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.703 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.703 DEBUG:boto:Method: GET 2023-09-22T21:47:12.703 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.703 DEBUG:boto:Data: 2023-09-22T21:47:12.703 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.703 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.703 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.703 DEBUG:boto:Params: {} 2023-09-22T21:47:12.704 DEBUG:boto:Token: None 2023-09-22T21:47:12.704 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:47:12.704 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ANkx4JenE3MHCJT8qq8ddgYwCzc= 2023-09-22T21:47:12.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ANkx4JenE3MHCJT8qq8ddgYwCzc=', 'Content-Length': '0'} 2023-09-22T21:47:12.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bd9b5f5950be9bd-00650e0b60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.707 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:12.707 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:12.707 DEBUG:boto:path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:47:12.707 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:47:12.708 DEBUG:boto:Method: GET 2023-09-22T21:47:12.708 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj46 2023-09-22T21:47:12.708 DEBUG:boto:Data: 2023-09-22T21:47:12.708 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.708 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.708 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.708 DEBUG:boto:Params: {} 2023-09-22T21:47:12.708 DEBUG:boto:Token: None 2023-09-22T21:47:12.708 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/ 2023-09-22T21:47:12.709 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA= 2023-09-22T21:47:12.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA=', 'Content-Length': '0'} 2023-09-22T21:47:12.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030a16390249a2ccd-00650e0b60-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.754 DEBUG:boto:b'ffzdqu-59obj461000falseobj462023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:12.755 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:12.755 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:12.755 DEBUG:boto:path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:47:12.755 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:47:12.755 DEBUG:boto:Method: GET 2023-09-22T21:47:12.756 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj46 2023-09-22T21:47:12.756 DEBUG:boto:Data: 2023-09-22T21:47:12.756 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.756 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.756 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.756 DEBUG:boto:Params: {} 2023-09-22T21:47:12.756 DEBUG:boto:Token: None 2023-09-22T21:47:12.756 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/ 2023-09-22T21:47:12.757 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA= 2023-09-22T21:47:12.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA=', 'Content-Length': '0'} 2023-09-22T21:47:12.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c66bfe731e6fc242-00650e0b60-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.766 DEBUG:boto:b'ffzdqu-59obj461000falseobj462023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:12.767 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:47:12.767 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:47:12.767 DEBUG:boto:Method: HEAD 2023-09-22T21:47:12.767 DEBUG:boto:Path: /ffzdqu-59/obj46 2023-09-22T21:47:12.767 DEBUG:boto:Data: 2023-09-22T21:47:12.767 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.767 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.768 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.768 DEBUG:boto:Params: {} 2023-09-22T21:47:12.768 DEBUG:boto:Token: None 2023-09-22T21:47:12.768 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj46 2023-09-22T21:47:12.768 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uLQMJSC3rGXJb78Dj9v/kF4fczk= 2023-09-22T21:47:12.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uLQMJSC3rGXJb78Dj9v/kF4fczk=', 'Content-Length': '0'} 2023-09-22T21:47:12.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d1fcf15722ba54d-00650e0b60-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.773 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:47:12.773 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:47:12.773 DEBUG:boto:Method: HEAD 2023-09-22T21:47:12.774 DEBUG:boto:Path: /ffzdqu-59/obj46 2023-09-22T21:47:12.774 DEBUG:boto:Data: 2023-09-22T21:47:12.774 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.774 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.774 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.774 DEBUG:boto:Params: {} 2023-09-22T21:47:12.774 DEBUG:boto:Token: None 2023-09-22T21:47:12.774 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj46 2023-09-22T21:47:12.775 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uLQMJSC3rGXJb78Dj9v/kF4fczk= 2023-09-22T21:47:12.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uLQMJSC3rGXJb78Dj9v/kF4fczk=', 'Content-Length': '0'} 2023-09-22T21:47:12.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ae1379f2491203d-00650e0b60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-22T21:47:12.779 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:47:12.779 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:47:12.779 DEBUG:boto:Method: GET 2023-09-22T21:47:12.779 DEBUG:boto:Path: /ffzdqu-59/obj46 2023-09-22T21:47:12.779 DEBUG:boto:Data: 2023-09-22T21:47:12.779 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.779 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.779 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.780 DEBUG:boto:Params: {} 2023-09-22T21:47:12.780 DEBUG:boto:Token: None 2023-09-22T21:47:12.780 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj46 2023-09-22T21:47:12.780 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RihHX+cqUCzjEVLdXwssyw3gE6k= 2023-09-22T21:47:12.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RihHX+cqUCzjEVLdXwssyw3gE6k=', 'Content-Length': '0'} 2023-09-22T21:47:12.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072ac41a04b61d7c3-00650e0b60-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.826 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:47:12.826 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:47:12.827 DEBUG:boto:Method: GET 2023-09-22T21:47:12.827 DEBUG:boto:Path: /ffzdqu-59/obj46 2023-09-22T21:47:12.827 DEBUG:boto:Data: 2023-09-22T21:47:12.827 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.827 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.827 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.827 DEBUG:boto:Params: {} 2023-09-22T21:47:12.827 DEBUG:boto:Token: None 2023-09-22T21:47:12.828 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj46 2023-09-22T21:47:12.828 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RihHX+cqUCzjEVLdXwssyw3gE6k= 2023-09-22T21:47:12.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RihHX+cqUCzjEVLdXwssyw3gE6k=', 'Content-Length': '0'} 2023-09-22T21:47:12.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b3c8ee43ca14dac-00650e0b60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.831 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:12.831 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:12.831 DEBUG:boto:path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:47:12.831 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:47:12.831 DEBUG:boto:Method: GET 2023-09-22T21:47:12.832 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj46 2023-09-22T21:47:12.832 DEBUG:boto:Data: 2023-09-22T21:47:12.832 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.832 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.832 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.832 DEBUG:boto:Params: {} 2023-09-22T21:47:12.832 DEBUG:boto:Token: None 2023-09-22T21:47:12.832 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/ 2023-09-22T21:47:12.832 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA= 2023-09-22T21:47:12.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA=', 'Content-Length': '0'} 2023-09-22T21:47:12.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000816014ee907e68b0-00650e0b60-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.878 DEBUG:boto:b'ffzdqu-59obj461000falseobj462023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:12.879 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:12.879 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:12.879 DEBUG:boto:path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:47:12.879 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:47:12.879 DEBUG:boto:Method: GET 2023-09-22T21:47:12.880 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj46 2023-09-22T21:47:12.880 DEBUG:boto:Data: 2023-09-22T21:47:12.880 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.880 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.880 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.880 DEBUG:boto:Params: {} 2023-09-22T21:47:12.880 DEBUG:boto:Token: None 2023-09-22T21:47:12.880 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/ 2023-09-22T21:47:12.881 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA= 2023-09-22T21:47:12.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA=', 'Content-Length': '0'} 2023-09-22T21:47:12.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e68baa751c9ce36f-00650e0b60-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.889 DEBUG:boto:b'ffzdqu-59obj461000falseobj462023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:12.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:47:12.890 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:47:12.890 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:47:12.890 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.890 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.890 DEBUG:boto:Method: GET 2023-09-22T21:47:12.890 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.890 DEBUG:boto:Data: 2023-09-22T21:47:12.890 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.891 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.891 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.891 DEBUG:boto:Params: {} 2023-09-22T21:47:12.891 DEBUG:boto:Token: None 2023-09-22T21:47:12.891 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.891 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kk/Ab7kkxpO/MsaIJFgZxyJDDck= 2023-09-22T21:47:12.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kk/Ab7kkxpO/MsaIJFgZxyJDDck=', 'Content-Length': '0'} 2023-09-22T21:47:12.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a125e255f7c7dfb2-00650e0b60-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.896 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:47:12.896 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:47:12.896 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.896 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.896 DEBUG:boto:Method: GET 2023-09-22T21:47:12.897 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.897 DEBUG:boto:Data: 2023-09-22T21:47:12.897 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.897 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.897 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.897 DEBUG:boto:Params: {} 2023-09-22T21:47:12.897 DEBUG:boto:Token: None 2023-09-22T21:47:12.897 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.898 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kk/Ab7kkxpO/MsaIJFgZxyJDDck= 2023-09-22T21:47:12.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kk/Ab7kkxpO/MsaIJFgZxyJDDck=', 'Content-Length': '0'} 2023-09-22T21:47:12.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000378dc6709b87aa4c-00650e0b60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.900 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:47:12.901 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:47:12.901 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.901 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.901 DEBUG:boto:Method: HEAD 2023-09-22T21:47:12.901 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.901 DEBUG:boto:Data: 2023-09-22T21:47:12.901 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.901 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.901 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.902 DEBUG:boto:Params: {} 2023-09-22T21:47:12.902 DEBUG:boto:Token: None 2023-09-22T21:47:12.902 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.902 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4agHEjJuo1DdDGgzv9Qks5nDHv0= 2023-09-22T21:47:12.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4agHEjJuo1DdDGgzv9Qks5nDHv0=', 'Content-Length': '0'} 2023-09-22T21:47:12.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd6c0b24aaf2493f-00650e0b60-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.905 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:47:12.906 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:47:12.906 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.906 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.906 DEBUG:boto:Method: HEAD 2023-09-22T21:47:12.906 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.906 DEBUG:boto:Data: 2023-09-22T21:47:12.906 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.906 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.906 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.906 DEBUG:boto:Params: {} 2023-09-22T21:47:12.907 DEBUG:boto:Token: None 2023-09-22T21:47:12.907 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.907 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4agHEjJuo1DdDGgzv9Qks5nDHv0= 2023-09-22T21:47:12.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4agHEjJuo1DdDGgzv9Qks5nDHv0=', 'Content-Length': '0'} 2023-09-22T21:47:12.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004062c866291b681f-00650e0b60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:47:12.910 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:47:12.910 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:47:12.910 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.910 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.911 DEBUG:boto:Method: GET 2023-09-22T21:47:12.911 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.911 DEBUG:boto:Data: 2023-09-22T21:47:12.911 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.911 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.911 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.911 DEBUG:boto:Params: {} 2023-09-22T21:47:12.911 DEBUG:boto:Token: None 2023-09-22T21:47:12.911 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.912 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kk/Ab7kkxpO/MsaIJFgZxyJDDck= 2023-09-22T21:47:12.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kk/Ab7kkxpO/MsaIJFgZxyJDDck=', 'Content-Length': '0'} 2023-09-22T21:47:12.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c966c96ed0cd2483-00650e0b60-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.958 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:47:12.959 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:47:12.959 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.959 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.959 DEBUG:boto:Method: GET 2023-09-22T21:47:12.959 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.959 DEBUG:boto:Data: 2023-09-22T21:47:12.959 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.959 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:12.959 DEBUG:boto:Port: 8000 2023-09-22T21:47:12.960 DEBUG:boto:Params: {} 2023-09-22T21:47:12.960 DEBUG:boto:Token: None 2023-09-22T21:47:12.960 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:47:12.960 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kk/Ab7kkxpO/MsaIJFgZxyJDDck= 2023-09-22T21:47:12.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kk/Ab7kkxpO/MsaIJFgZxyJDDck=', 'Content-Length': '0'} 2023-09-22T21:47:12.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b24dd80960101354-00650e0b60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:12.963 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:12.963 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:12.963 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:47:12.963 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:47:12.964 DEBUG:boto:Method: GET 2023-09-22T21:47:12.964 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:47:12.964 DEBUG:boto:Data: 2023-09-22T21:47:12.964 DEBUG:boto:Headers: {} 2023-09-22T21:47:12.964 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:12.964 DEBUG:boto:Port: 8001 2023-09-22T21:47:12.964 DEBUG:boto:Params: {} 2023-09-22T21:47:12.964 DEBUG:boto:Token: None 2023-09-22T21:47:12.964 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:12 GMT /ffzdqu-59/ 2023-09-22T21:47:12.965 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA= 2023-09-22T21:47:12.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wBkyeas04IwToIS+/Ip7rUPFUYA=', 'Content-Length': '0'} 2023-09-22T21:47:12.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009844a611a4d59eef-00650e0b60-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.010 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:13.011 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:13.011 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:13.011 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:47:13.011 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:47:13.011 DEBUG:boto:Method: GET 2023-09-22T21:47:13.011 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:47:13.011 DEBUG:boto:Data: 2023-09-22T21:47:13.012 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.012 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.012 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.012 DEBUG:boto:Params: {} 2023-09-22T21:47:13.012 DEBUG:boto:Token: None 2023-09-22T21:47:13.012 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/ 2023-09-22T21:47:13.012 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0= 2023-09-22T21:47:13.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0=', 'Content-Length': '0'} 2023-09-22T21:47:13.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ab20f68b55b3a53-00650e0b61-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.021 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:13.021 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:47:13.021 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:47:13.022 DEBUG:boto:Method: HEAD 2023-09-22T21:47:13.022 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:47:13.022 DEBUG:boto:Data: 2023-09-22T21:47:13.022 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.022 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.022 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.022 DEBUG:boto:Params: {} 2023-09-22T21:47:13.022 DEBUG:boto:Token: None 2023-09-22T21:47:13.023 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj51 2023-09-22T21:47:13.023 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZyGva4T1/GlJ91XKgMmopGfPzv4= 2023-09-22T21:47:13.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZyGva4T1/GlJ91XKgMmopGfPzv4=', 'Content-Length': '0'} 2023-09-22T21:47:13.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2e87c79eee9d217-00650e0b61-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.026 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:47:13.026 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:47:13.026 DEBUG:boto:Method: HEAD 2023-09-22T21:47:13.026 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:47:13.026 DEBUG:boto:Data: 2023-09-22T21:47:13.026 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.026 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.027 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.027 DEBUG:boto:Params: {} 2023-09-22T21:47:13.027 DEBUG:boto:Token: None 2023-09-22T21:47:13.027 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj51 2023-09-22T21:47:13.027 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZyGva4T1/GlJ91XKgMmopGfPzv4= 2023-09-22T21:47:13.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZyGva4T1/GlJ91XKgMmopGfPzv4=', 'Content-Length': '0'} 2023-09-22T21:47:13.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005c14da3149b21ab-00650e0b61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:47:13.030 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:47:13.030 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:47:13.030 DEBUG:boto:Method: GET 2023-09-22T21:47:13.030 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:47:13.030 DEBUG:boto:Data: 2023-09-22T21:47:13.031 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.031 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.031 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.031 DEBUG:boto:Params: {} 2023-09-22T21:47:13.031 DEBUG:boto:Token: None 2023-09-22T21:47:13.031 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj51 2023-09-22T21:47:13.031 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1xTP9YTC0GN/eFVHajaGwkaS/D0= 2023-09-22T21:47:13.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1xTP9YTC0GN/eFVHajaGwkaS/D0=', 'Content-Length': '0'} 2023-09-22T21:47:13.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a715960ad807771b-00650e0b61-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.082 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:47:13.082 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:47:13.083 DEBUG:boto:Method: GET 2023-09-22T21:47:13.083 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:47:13.083 DEBUG:boto:Data: 2023-09-22T21:47:13.083 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.083 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.083 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.083 DEBUG:boto:Params: {} 2023-09-22T21:47:13.084 DEBUG:boto:Token: None 2023-09-22T21:47:13.084 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj51 2023-09-22T21:47:13.084 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1xTP9YTC0GN/eFVHajaGwkaS/D0= 2023-09-22T21:47:13.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1xTP9YTC0GN/eFVHajaGwkaS/D0=', 'Content-Length': '0'} 2023-09-22T21:47:13.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000768e8ea6b8367320-00650e0b61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.087 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:13.087 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:13.088 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:47:13.088 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:47:13.088 DEBUG:boto:Method: GET 2023-09-22T21:47:13.088 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:47:13.088 DEBUG:boto:Data: 2023-09-22T21:47:13.088 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.088 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.088 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.088 DEBUG:boto:Params: {} 2023-09-22T21:47:13.089 DEBUG:boto:Token: None 2023-09-22T21:47:13.089 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/ 2023-09-22T21:47:13.089 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0= 2023-09-22T21:47:13.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0=', 'Content-Length': '0'} 2023-09-22T21:47:13.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2ee6be53a256c29-00650e0b61-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.094 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:13.094 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:13.094 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:13.094 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:47:13.094 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:47:13.094 DEBUG:boto:Method: GET 2023-09-22T21:47:13.095 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:47:13.095 DEBUG:boto:Data: 2023-09-22T21:47:13.095 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.095 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.095 DEBUG:boto:Params: {} 2023-09-22T21:47:13.095 DEBUG:boto:Token: None 2023-09-22T21:47:13.095 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/ 2023-09-22T21:47:13.095 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0= 2023-09-22T21:47:13.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0=', 'Content-Length': '0'} 2023-09-22T21:47:13.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f2d3c5fe8f957d9-00650e0b61-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.146 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:13.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:47:13.147 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:47:13.147 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:47:13.147 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.148 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.148 DEBUG:boto:Method: GET 2023-09-22T21:47:13.148 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.148 DEBUG:boto:Data: 2023-09-22T21:47:13.148 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.148 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.148 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.148 DEBUG:boto:Params: {} 2023-09-22T21:47:13.148 DEBUG:boto:Token: None 2023-09-22T21:47:13.149 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.149 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rpPv50UBbsx+NOd8MhNp+Hll724= 2023-09-22T21:47:13.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rpPv50UBbsx+NOd8MhNp+Hll724=', 'Content-Length': '0'} 2023-09-22T21:47:13.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006626eff9740fbba5-00650e0b61-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.194 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:47:13.194 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:47:13.195 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.195 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.195 DEBUG:boto:Method: GET 2023-09-22T21:47:13.195 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.195 DEBUG:boto:Data: 2023-09-22T21:47:13.195 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.195 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.195 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.195 DEBUG:boto:Params: {} 2023-09-22T21:47:13.196 DEBUG:boto:Token: None 2023-09-22T21:47:13.196 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.196 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rpPv50UBbsx+NOd8MhNp+Hll724= 2023-09-22T21:47:13.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rpPv50UBbsx+NOd8MhNp+Hll724=', 'Content-Length': '0'} 2023-09-22T21:47:13.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f67a839470c2d562-00650e0b61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.199 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:47:13.199 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:47:13.200 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.200 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.200 DEBUG:boto:Method: HEAD 2023-09-22T21:47:13.200 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.200 DEBUG:boto:Data: 2023-09-22T21:47:13.200 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.200 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.200 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.200 DEBUG:boto:Params: {} 2023-09-22T21:47:13.201 DEBUG:boto:Token: None 2023-09-22T21:47:13.201 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.201 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hMBOtY6BBSZOKstgoWdEsVA7li8= 2023-09-22T21:47:13.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hMBOtY6BBSZOKstgoWdEsVA7li8=', 'Content-Length': '0'} 2023-09-22T21:47:13.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fe62d95a1dd6224-00650e0b61-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.204 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:47:13.204 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:47:13.204 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.204 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.204 DEBUG:boto:Method: HEAD 2023-09-22T21:47:13.204 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.204 DEBUG:boto:Data: 2023-09-22T21:47:13.205 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.205 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.205 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.205 DEBUG:boto:Params: {} 2023-09-22T21:47:13.205 DEBUG:boto:Token: None 2023-09-22T21:47:13.205 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.205 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hMBOtY6BBSZOKstgoWdEsVA7li8= 2023-09-22T21:47:13.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hMBOtY6BBSZOKstgoWdEsVA7li8=', 'Content-Length': '0'} 2023-09-22T21:47:13.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba3482ac320ed78d-00650e0b61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:47:13.209 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:47:13.209 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:47:13.209 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.209 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.209 DEBUG:boto:Method: GET 2023-09-22T21:47:13.209 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.209 DEBUG:boto:Data: 2023-09-22T21:47:13.210 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.210 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.210 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.210 DEBUG:boto:Params: {} 2023-09-22T21:47:13.210 DEBUG:boto:Token: None 2023-09-22T21:47:13.210 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.210 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rpPv50UBbsx+NOd8MhNp+Hll724= 2023-09-22T21:47:13.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rpPv50UBbsx+NOd8MhNp+Hll724=', 'Content-Length': '0'} 2023-09-22T21:47:13.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b8c6b7fb19bac09-00650e0b61-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.254 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:47:13.255 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:47:13.255 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.255 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.255 DEBUG:boto:Method: GET 2023-09-22T21:47:13.255 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.255 DEBUG:boto:Data: 2023-09-22T21:47:13.255 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.255 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.256 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.256 DEBUG:boto:Params: {} 2023-09-22T21:47:13.256 DEBUG:boto:Token: None 2023-09-22T21:47:13.256 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:47:13.256 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rpPv50UBbsx+NOd8MhNp+Hll724= 2023-09-22T21:47:13.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rpPv50UBbsx+NOd8MhNp+Hll724=', 'Content-Length': '0'} 2023-09-22T21:47:13.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003497c8615f6a89ed-00650e0b61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.260 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:13.260 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:13.260 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:47:13.260 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:47:13.260 DEBUG:boto:Method: GET 2023-09-22T21:47:13.260 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:47:13.260 DEBUG:boto:Data: 2023-09-22T21:47:13.260 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.261 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.261 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.261 DEBUG:boto:Params: {} 2023-09-22T21:47:13.261 DEBUG:boto:Token: None 2023-09-22T21:47:13.261 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/ 2023-09-22T21:47:13.261 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0= 2023-09-22T21:47:13.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0=', 'Content-Length': '0'} 2023-09-22T21:47:13.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf57fe34d6876bc7-00650e0b61-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.306 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:13.307 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:13.307 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:13.307 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:47:13.307 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:47:13.307 DEBUG:boto:Method: GET 2023-09-22T21:47:13.307 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:47:13.308 DEBUG:boto:Data: 2023-09-22T21:47:13.308 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.308 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.308 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.308 DEBUG:boto:Params: {} 2023-09-22T21:47:13.308 DEBUG:boto:Token: None 2023-09-22T21:47:13.308 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/ 2023-09-22T21:47:13.308 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0= 2023-09-22T21:47:13.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0=', 'Content-Length': '0'} 2023-09-22T21:47:13.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000688c547b07cd1009-00650e0b61-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.318 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:13.318 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:47:13.319 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:47:13.319 DEBUG:boto:Method: HEAD 2023-09-22T21:47:13.319 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:47:13.319 DEBUG:boto:Data: 2023-09-22T21:47:13.319 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.319 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.319 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.319 DEBUG:boto:Params: {} 2023-09-22T21:47:13.319 DEBUG:boto:Token: None 2023-09-22T21:47:13.320 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj55 2023-09-22T21:47:13.320 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eXaJf/9XntISbFU8fYrz96Z/x6Y= 2023-09-22T21:47:13.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eXaJf/9XntISbFU8fYrz96Z/x6Y=', 'Content-Length': '0'} 2023-09-22T21:47:13.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2d705ff1a3becee-00650e0b61-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.323 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:47:13.323 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:47:13.323 DEBUG:boto:Method: HEAD 2023-09-22T21:47:13.324 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:47:13.324 DEBUG:boto:Data: 2023-09-22T21:47:13.324 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.324 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.324 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.324 DEBUG:boto:Params: {} 2023-09-22T21:47:13.324 DEBUG:boto:Token: None 2023-09-22T21:47:13.324 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj55 2023-09-22T21:47:13.324 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eXaJf/9XntISbFU8fYrz96Z/x6Y= 2023-09-22T21:47:13.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eXaJf/9XntISbFU8fYrz96Z/x6Y=', 'Content-Length': '0'} 2023-09-22T21:47:13.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0e9ff7461603e6d-00650e0b61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:47:13.328 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:47:13.328 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:47:13.328 DEBUG:boto:Method: GET 2023-09-22T21:47:13.329 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:47:13.329 DEBUG:boto:Data: 2023-09-22T21:47:13.329 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.329 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.329 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.329 DEBUG:boto:Params: {} 2023-09-22T21:47:13.329 DEBUG:boto:Token: None 2023-09-22T21:47:13.329 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj55 2023-09-22T21:47:13.329 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FsdX3ls4ViXafGWmFJtJ4lNfpKs= 2023-09-22T21:47:13.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FsdX3ls4ViXafGWmFJtJ4lNfpKs=', 'Content-Length': '0'} 2023-09-22T21:47:13.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005869f228458c1a7e-00650e0b61-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.374 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:47:13.374 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:47:13.375 DEBUG:boto:Method: GET 2023-09-22T21:47:13.375 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:47:13.375 DEBUG:boto:Data: 2023-09-22T21:47:13.375 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.375 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.375 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.375 DEBUG:boto:Params: {} 2023-09-22T21:47:13.375 DEBUG:boto:Token: None 2023-09-22T21:47:13.375 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj55 2023-09-22T21:47:13.376 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FsdX3ls4ViXafGWmFJtJ4lNfpKs= 2023-09-22T21:47:13.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FsdX3ls4ViXafGWmFJtJ4lNfpKs=', 'Content-Length': '0'} 2023-09-22T21:47:13.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003db7a63ad12700a8-00650e0b61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.379 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:13.379 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:13.379 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:47:13.379 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:47:13.380 DEBUG:boto:Method: GET 2023-09-22T21:47:13.380 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:47:13.380 DEBUG:boto:Data: 2023-09-22T21:47:13.380 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.380 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.380 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.380 DEBUG:boto:Params: {} 2023-09-22T21:47:13.380 DEBUG:boto:Token: None 2023-09-22T21:47:13.381 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/ 2023-09-22T21:47:13.381 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0= 2023-09-22T21:47:13.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0=', 'Content-Length': '0'} 2023-09-22T21:47:13.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000629848106ce9842-00650e0b61-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.426 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:13.427 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:13.427 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:13.427 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:47:13.427 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:47:13.427 DEBUG:boto:Method: GET 2023-09-22T21:47:13.427 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:47:13.428 DEBUG:boto:Data: 2023-09-22T21:47:13.428 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.428 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.428 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.428 DEBUG:boto:Params: {} 2023-09-22T21:47:13.428 DEBUG:boto:Token: None 2023-09-22T21:47:13.428 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/ 2023-09-22T21:47:13.428 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0= 2023-09-22T21:47:13.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0=', 'Content-Length': '0'} 2023-09-22T21:47:13.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000448ce657ea9aa8b3-00650e0b61-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.438 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:13.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T21:47:13.438 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:47:13.438 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:47:13.438 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.438 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.439 DEBUG:boto:Method: GET 2023-09-22T21:47:13.439 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.439 DEBUG:boto:Data: 2023-09-22T21:47:13.439 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.439 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.439 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.439 DEBUG:boto:Params: {} 2023-09-22T21:47:13.439 DEBUG:boto:Token: None 2023-09-22T21:47:13.439 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.440 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IDw+pPcsJPG/AZQLWjG1CfeHdyk= 2023-09-22T21:47:13.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IDw+pPcsJPG/AZQLWjG1CfeHdyk=', 'Content-Length': '0'} 2023-09-22T21:47:13.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f79f93103c8cbef-00650e0b61-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.443 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:47:13.443 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:47:13.443 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.443 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.443 DEBUG:boto:Method: GET 2023-09-22T21:47:13.443 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.444 DEBUG:boto:Data: 2023-09-22T21:47:13.444 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.444 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.444 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.444 DEBUG:boto:Params: {} 2023-09-22T21:47:13.444 DEBUG:boto:Token: None 2023-09-22T21:47:13.444 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.444 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IDw+pPcsJPG/AZQLWjG1CfeHdyk= 2023-09-22T21:47:13.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IDw+pPcsJPG/AZQLWjG1CfeHdyk=', 'Content-Length': '0'} 2023-09-22T21:47:13.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc6697e33564c4d3-00650e0b61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.494 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:47:13.495 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:47:13.495 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.495 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.495 DEBUG:boto:Method: HEAD 2023-09-22T21:47:13.495 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.495 DEBUG:boto:Data: 2023-09-22T21:47:13.495 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.495 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.495 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.495 DEBUG:boto:Params: {} 2023-09-22T21:47:13.496 DEBUG:boto:Token: None 2023-09-22T21:47:13.496 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.496 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eZUOcoa1FGvw7gaBzqQ8o/DdNR4= 2023-09-22T21:47:13.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eZUOcoa1FGvw7gaBzqQ8o/DdNR4=', 'Content-Length': '0'} 2023-09-22T21:47:13.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a768fc139d59910b-00650e0b61-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.499 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:47:13.499 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:47:13.499 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.499 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.499 DEBUG:boto:Method: HEAD 2023-09-22T21:47:13.499 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.499 DEBUG:boto:Data: 2023-09-22T21:47:13.500 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.500 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.500 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.500 DEBUG:boto:Params: {} 2023-09-22T21:47:13.500 DEBUG:boto:Token: None 2023-09-22T21:47:13.500 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.500 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eZUOcoa1FGvw7gaBzqQ8o/DdNR4= 2023-09-22T21:47:13.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eZUOcoa1FGvw7gaBzqQ8o/DdNR4=', 'Content-Length': '0'} 2023-09-22T21:47:13.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000025a1705c7d2bc26-00650e0b61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T21:47:13.504 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:47:13.504 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:47:13.504 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.504 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.504 DEBUG:boto:Method: GET 2023-09-22T21:47:13.504 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.504 DEBUG:boto:Data: 2023-09-22T21:47:13.504 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.504 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.505 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.505 DEBUG:boto:Params: {} 2023-09-22T21:47:13.505 DEBUG:boto:Token: None 2023-09-22T21:47:13.505 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.505 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IDw+pPcsJPG/AZQLWjG1CfeHdyk= 2023-09-22T21:47:13.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IDw+pPcsJPG/AZQLWjG1CfeHdyk=', 'Content-Length': '0'} 2023-09-22T21:47:13.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001deabeb44571de03-00650e0b61-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.550 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:47:13.551 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:47:13.551 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.551 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.551 DEBUG:boto:Method: GET 2023-09-22T21:47:13.551 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.551 DEBUG:boto:Data: 2023-09-22T21:47:13.551 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.551 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.551 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.552 DEBUG:boto:Params: {} 2023-09-22T21:47:13.552 DEBUG:boto:Token: None 2023-09-22T21:47:13.552 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:47:13.552 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IDw+pPcsJPG/AZQLWjG1CfeHdyk= 2023-09-22T21:47:13.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IDw+pPcsJPG/AZQLWjG1CfeHdyk=', 'Content-Length': '0'} 2023-09-22T21:47:13.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f4444c0d8eaf419-00650e0b61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.555 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:13.555 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:13.556 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:47:13.556 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:47:13.556 DEBUG:boto:Method: GET 2023-09-22T21:47:13.556 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:47:13.556 DEBUG:boto:Data: 2023-09-22T21:47:13.556 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.556 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.556 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.556 DEBUG:boto:Params: {} 2023-09-22T21:47:13.556 DEBUG:boto:Token: None 2023-09-22T21:47:13.557 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/ 2023-09-22T21:47:13.557 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0= 2023-09-22T21:47:13.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0=', 'Content-Length': '0'} 2023-09-22T21:47:13.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045263141083fcd50-00650e0b61-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.602 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:13.603 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:13.603 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:13.603 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:47:13.603 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:47:13.603 DEBUG:boto:Method: GET 2023-09-22T21:47:13.604 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:47:13.604 DEBUG:boto:Data: 2023-09-22T21:47:13.604 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.604 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.604 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.604 DEBUG:boto:Params: {} 2023-09-22T21:47:13.604 DEBUG:boto:Token: None 2023-09-22T21:47:13.605 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/ 2023-09-22T21:47:13.605 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0= 2023-09-22T21:47:13.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0=', 'Content-Length': '0'} 2023-09-22T21:47:13.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4a09dc4272c513e-00650e0b61-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.614 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:13.614 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:47:13.615 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:47:13.615 DEBUG:boto:Method: HEAD 2023-09-22T21:47:13.615 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:47:13.615 DEBUG:boto:Data: 2023-09-22T21:47:13.615 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.615 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.616 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.616 DEBUG:boto:Params: {} 2023-09-22T21:47:13.616 DEBUG:boto:Token: None 2023-09-22T21:47:13.616 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj57 2023-09-22T21:47:13.616 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ChQLGflnrk6hq+ritiMjMA+wBbM= 2023-09-22T21:47:13.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ChQLGflnrk6hq+ritiMjMA+wBbM=', 'Content-Length': '0'} 2023-09-22T21:47:13.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004431703302b5d449-00650e0b61-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.619 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:47:13.619 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:47:13.619 DEBUG:boto:Method: HEAD 2023-09-22T21:47:13.620 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:47:13.620 DEBUG:boto:Data: 2023-09-22T21:47:13.620 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.620 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.620 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.620 DEBUG:boto:Params: {} 2023-09-22T21:47:13.620 DEBUG:boto:Token: None 2023-09-22T21:47:13.620 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj57 2023-09-22T21:47:13.621 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ChQLGflnrk6hq+ritiMjMA+wBbM= 2023-09-22T21:47:13.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ChQLGflnrk6hq+ritiMjMA+wBbM=', 'Content-Length': '0'} 2023-09-22T21:47:13.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d83b570cdeb63f0b-00650e0b61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T21:47:13.623 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:47:13.624 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:47:13.624 DEBUG:boto:Method: GET 2023-09-22T21:47:13.624 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:47:13.624 DEBUG:boto:Data: 2023-09-22T21:47:13.624 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.624 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.624 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.624 DEBUG:boto:Params: {} 2023-09-22T21:47:13.624 DEBUG:boto:Token: None 2023-09-22T21:47:13.625 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj57 2023-09-22T21:47:13.625 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zxM/P+vJLGQ3wVA+fGCZN+Vjua8= 2023-09-22T21:47:13.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zxM/P+vJLGQ3wVA+fGCZN+Vjua8=', 'Content-Length': '0'} 2023-09-22T21:47:13.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ab3fac5b15ae0cf-00650e0b61-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.674 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:47:13.675 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:47:13.675 DEBUG:boto:Method: GET 2023-09-22T21:47:13.675 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:47:13.675 DEBUG:boto:Data: 2023-09-22T21:47:13.675 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.675 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.675 DEBUG:boto:Params: {} 2023-09-22T21:47:13.676 DEBUG:boto:Token: None 2023-09-22T21:47:13.676 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj57 2023-09-22T21:47:13.676 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zxM/P+vJLGQ3wVA+fGCZN+Vjua8= 2023-09-22T21:47:13.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zxM/P+vJLGQ3wVA+fGCZN+Vjua8=', 'Content-Length': '0'} 2023-09-22T21:47:13.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038e13a3a31e7820d-00650e0b61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.679 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:13.679 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:13.679 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:47:13.680 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:47:13.680 DEBUG:boto:Method: GET 2023-09-22T21:47:13.680 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:47:13.680 DEBUG:boto:Data: 2023-09-22T21:47:13.680 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.680 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.680 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.680 DEBUG:boto:Params: {} 2023-09-22T21:47:13.680 DEBUG:boto:Token: None 2023-09-22T21:47:13.681 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/ 2023-09-22T21:47:13.681 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0= 2023-09-22T21:47:13.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0=', 'Content-Length': '0'} 2023-09-22T21:47:13.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000edf46440d7f6c8f-00650e0b61-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.726 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:13.727 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:13.727 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:13.727 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:47:13.727 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:47:13.728 DEBUG:boto:Method: GET 2023-09-22T21:47:13.728 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:47:13.728 DEBUG:boto:Data: 2023-09-22T21:47:13.728 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.728 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.728 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.728 DEBUG:boto:Params: {} 2023-09-22T21:47:13.728 DEBUG:boto:Token: None 2023-09-22T21:47:13.728 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/ 2023-09-22T21:47:13.729 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0= 2023-09-22T21:47:13.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0=', 'Content-Length': '0'} 2023-09-22T21:47:13.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cb24888e6a76146-00650e0b61-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.737 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:13.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-22T21:47:13.738 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:47:13.738 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:47:13.738 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.738 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.738 DEBUG:boto:Method: GET 2023-09-22T21:47:13.738 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.738 DEBUG:boto:Data: 2023-09-22T21:47:13.738 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.738 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.739 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.739 DEBUG:boto:Params: {} 2023-09-22T21:47:13.739 DEBUG:boto:Token: None 2023-09-22T21:47:13.739 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.739 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jGbBdWzRay+Wt/NnfDMBrwm+1Gg= 2023-09-22T21:47:13.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jGbBdWzRay+Wt/NnfDMBrwm+1Gg=', 'Content-Length': '0'} 2023-09-22T21:47:13.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a440e60bb07d177e-00650e0b61-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.742 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:47:13.742 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:47:13.742 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.742 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.743 DEBUG:boto:Method: GET 2023-09-22T21:47:13.743 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.743 DEBUG:boto:Data: 2023-09-22T21:47:13.743 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.743 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.743 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.743 DEBUG:boto:Params: {} 2023-09-22T21:47:13.743 DEBUG:boto:Token: None 2023-09-22T21:47:13.743 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.744 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jGbBdWzRay+Wt/NnfDMBrwm+1Gg= 2023-09-22T21:47:13.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jGbBdWzRay+Wt/NnfDMBrwm+1Gg=', 'Content-Length': '0'} 2023-09-22T21:47:13.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1802ba4456542d8-00650e0b61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.790 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:47:13.790 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:47:13.791 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.791 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.791 DEBUG:boto:Method: HEAD 2023-09-22T21:47:13.791 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.791 DEBUG:boto:Data: 2023-09-22T21:47:13.791 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.791 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.791 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.791 DEBUG:boto:Params: {} 2023-09-22T21:47:13.791 DEBUG:boto:Token: None 2023-09-22T21:47:13.792 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.792 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:00FpVi7Utv7a42FY8jYWTBE4fyQ= 2023-09-22T21:47:13.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:00FpVi7Utv7a42FY8jYWTBE4fyQ=', 'Content-Length': '0'} 2023-09-22T21:47:13.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000338a03c005d10da9-00650e0b61-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.795 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:47:13.795 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:47:13.795 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.795 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.795 DEBUG:boto:Method: HEAD 2023-09-22T21:47:13.795 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.795 DEBUG:boto:Data: 2023-09-22T21:47:13.795 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.795 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.796 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.796 DEBUG:boto:Params: {} 2023-09-22T21:47:13.796 DEBUG:boto:Token: None 2023-09-22T21:47:13.796 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.796 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:00FpVi7Utv7a42FY8jYWTBE4fyQ= 2023-09-22T21:47:13.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:00FpVi7Utv7a42FY8jYWTBE4fyQ=', 'Content-Length': '0'} 2023-09-22T21:47:13.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eedf598957650567-00650e0b61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-22T21:47:13.800 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:47:13.800 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:47:13.800 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.800 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.800 DEBUG:boto:Method: GET 2023-09-22T21:47:13.800 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.800 DEBUG:boto:Data: 2023-09-22T21:47:13.800 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.800 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.801 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.801 DEBUG:boto:Params: {} 2023-09-22T21:47:13.801 DEBUG:boto:Token: None 2023-09-22T21:47:13.801 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.801 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jGbBdWzRay+Wt/NnfDMBrwm+1Gg= 2023-09-22T21:47:13.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jGbBdWzRay+Wt/NnfDMBrwm+1Gg=', 'Content-Length': '0'} 2023-09-22T21:47:13.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000502922f038c64231-00650e0b61-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.846 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:47:13.846 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:47:13.847 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.847 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.847 DEBUG:boto:Method: GET 2023-09-22T21:47:13.847 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.847 DEBUG:boto:Data: 2023-09-22T21:47:13.847 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.847 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.847 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.847 DEBUG:boto:Params: {} 2023-09-22T21:47:13.847 DEBUG:boto:Token: None 2023-09-22T21:47:13.848 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:47:13.848 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jGbBdWzRay+Wt/NnfDMBrwm+1Gg= 2023-09-22T21:47:13.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jGbBdWzRay+Wt/NnfDMBrwm+1Gg=', 'Content-Length': '0'} 2023-09-22T21:47:13.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7f7434ed4edfdca-00650e0b61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.852 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:13.852 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:13.852 DEBUG:boto:path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:47:13.852 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:47:13.852 DEBUG:boto:Method: GET 2023-09-22T21:47:13.852 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj58 2023-09-22T21:47:13.852 DEBUG:boto:Data: 2023-09-22T21:47:13.852 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.853 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.853 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.853 DEBUG:boto:Params: {} 2023-09-22T21:47:13.853 DEBUG:boto:Token: None 2023-09-22T21:47:13.853 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/ 2023-09-22T21:47:13.853 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0= 2023-09-22T21:47:13.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0=', 'Content-Length': '0'} 2023-09-22T21:47:13.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cd7baec3b036d19-00650e0b61-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.902 DEBUG:boto:b'ffzdqu-59obj581000falseobj582023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:13.903 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:13.903 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:13.903 DEBUG:boto:path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:47:13.903 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:47:13.903 DEBUG:boto:Method: GET 2023-09-22T21:47:13.903 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj58 2023-09-22T21:47:13.904 DEBUG:boto:Data: 2023-09-22T21:47:13.904 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.904 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.904 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.904 DEBUG:boto:Params: {} 2023-09-22T21:47:13.904 DEBUG:boto:Token: None 2023-09-22T21:47:13.904 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/ 2023-09-22T21:47:13.904 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0= 2023-09-22T21:47:13.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0=', 'Content-Length': '0'} 2023-09-22T21:47:13.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066547bbf4fc0336a-00650e0b61-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.913 DEBUG:boto:b'ffzdqu-59obj581000falseobj582023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:13.914 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:47:13.914 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:47:13.914 DEBUG:boto:Method: HEAD 2023-09-22T21:47:13.914 DEBUG:boto:Path: /ffzdqu-59/obj58 2023-09-22T21:47:13.914 DEBUG:boto:Data: 2023-09-22T21:47:13.914 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.914 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.914 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.915 DEBUG:boto:Params: {} 2023-09-22T21:47:13.915 DEBUG:boto:Token: None 2023-09-22T21:47:13.915 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj58 2023-09-22T21:47:13.915 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F6Sl/cn6xvwUq5oR/+gkz7CNEOE= 2023-09-22T21:47:13.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F6Sl/cn6xvwUq5oR/+gkz7CNEOE=', 'Content-Length': '0'} 2023-09-22T21:47:13.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb52cc348470b202-00650e0b61-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.918 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:47:13.918 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:47:13.918 DEBUG:boto:Method: HEAD 2023-09-22T21:47:13.918 DEBUG:boto:Path: /ffzdqu-59/obj58 2023-09-22T21:47:13.919 DEBUG:boto:Data: 2023-09-22T21:47:13.919 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.919 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.919 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.919 DEBUG:boto:Params: {} 2023-09-22T21:47:13.919 DEBUG:boto:Token: None 2023-09-22T21:47:13.919 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj58 2023-09-22T21:47:13.919 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F6Sl/cn6xvwUq5oR/+gkz7CNEOE= 2023-09-22T21:47:13.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F6Sl/cn6xvwUq5oR/+gkz7CNEOE=', 'Content-Length': '0'} 2023-09-22T21:47:13.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fccee8f17506bfc2-00650e0b61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-22T21:47:13.923 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:47:13.923 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:47:13.923 DEBUG:boto:Method: GET 2023-09-22T21:47:13.923 DEBUG:boto:Path: /ffzdqu-59/obj58 2023-09-22T21:47:13.923 DEBUG:boto:Data: 2023-09-22T21:47:13.923 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.923 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.923 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.923 DEBUG:boto:Params: {} 2023-09-22T21:47:13.924 DEBUG:boto:Token: None 2023-09-22T21:47:13.924 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj58 2023-09-22T21:47:13.924 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nOPq8Tl2XIH2WksvCZh3A21Bn2o= 2023-09-22T21:47:13.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nOPq8Tl2XIH2WksvCZh3A21Bn2o=', 'Content-Length': '0'} 2023-09-22T21:47:13.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000de70a48bf0c625d-00650e0b61-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.970 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:47:13.970 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:47:13.970 DEBUG:boto:Method: GET 2023-09-22T21:47:13.971 DEBUG:boto:Path: /ffzdqu-59/obj58 2023-09-22T21:47:13.971 DEBUG:boto:Data: 2023-09-22T21:47:13.971 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.971 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.971 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.971 DEBUG:boto:Params: {} 2023-09-22T21:47:13.971 DEBUG:boto:Token: None 2023-09-22T21:47:13.971 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/obj58 2023-09-22T21:47:13.971 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nOPq8Tl2XIH2WksvCZh3A21Bn2o= 2023-09-22T21:47:13.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nOPq8Tl2XIH2WksvCZh3A21Bn2o=', 'Content-Length': '0'} 2023-09-22T21:47:13.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f3ebb2721e8b836-00650e0b61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.975 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:13.975 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:13.975 DEBUG:boto:path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:47:13.975 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:47:13.975 DEBUG:boto:Method: GET 2023-09-22T21:47:13.975 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj58 2023-09-22T21:47:13.975 DEBUG:boto:Data: 2023-09-22T21:47:13.976 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.976 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:13.976 DEBUG:boto:Port: 8001 2023-09-22T21:47:13.976 DEBUG:boto:Params: {} 2023-09-22T21:47:13.976 DEBUG:boto:Token: None 2023-09-22T21:47:13.976 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/ 2023-09-22T21:47:13.976 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0= 2023-09-22T21:47:13.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0=', 'Content-Length': '0'} 2023-09-22T21:47:13.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024ca59652e3d2f3a-00650e0b61-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:13.982 DEBUG:boto:b'ffzdqu-59obj581000falseobj582023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:13.982 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:13.982 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:13.982 DEBUG:boto:path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:47:13.982 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:47:13.983 DEBUG:boto:Method: GET 2023-09-22T21:47:13.983 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj58 2023-09-22T21:47:13.983 DEBUG:boto:Data: 2023-09-22T21:47:13.983 DEBUG:boto:Headers: {} 2023-09-22T21:47:13.983 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:13.983 DEBUG:boto:Port: 8000 2023-09-22T21:47:13.983 DEBUG:boto:Params: {} 2023-09-22T21:47:13.983 DEBUG:boto:Token: None 2023-09-22T21:47:13.984 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:13 GMT /ffzdqu-59/ 2023-09-22T21:47:13.984 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0= 2023-09-22T21:47:13.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F9+PKKEoG/GJb6j3dT5RaOjb3G0=', 'Content-Length': '0'} 2023-09-22T21:47:13.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dd807b30c632297-00650e0b61-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.034 DEBUG:boto:b'ffzdqu-59obj581000falseobj582023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:14.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-22T21:47:14.035 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:47:14.035 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:47:14.035 DEBUG:boto:path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.035 DEBUG:boto:auth_path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.036 DEBUG:boto:Method: GET 2023-09-22T21:47:14.036 DEBUG:boto:Path: /ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.036 DEBUG:boto:Data: 2023-09-22T21:47:14.036 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.036 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.036 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.036 DEBUG:boto:Params: {} 2023-09-22T21:47:14.036 DEBUG:boto:Token: None 2023-09-22T21:47:14.037 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.037 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Geu3KDeOjDmQxX+eMM83k+FtloM= 2023-09-22T21:47:14.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Geu3KDeOjDmQxX+eMM83k+FtloM=', 'Content-Length': '0'} 2023-09-22T21:47:14.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000762b27693da62273-00650e0b62-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.082 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:47:14.082 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:47:14.083 DEBUG:boto:path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.083 DEBUG:boto:auth_path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.083 DEBUG:boto:Method: GET 2023-09-22T21:47:14.083 DEBUG:boto:Path: /ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.083 DEBUG:boto:Data: 2023-09-22T21:47:14.083 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.083 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.083 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.083 DEBUG:boto:Params: {} 2023-09-22T21:47:14.084 DEBUG:boto:Token: None 2023-09-22T21:47:14.084 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.084 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Geu3KDeOjDmQxX+eMM83k+FtloM= 2023-09-22T21:47:14.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Geu3KDeOjDmQxX+eMM83k+FtloM=', 'Content-Length': '0'} 2023-09-22T21:47:14.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aef7371794a9b3f3-00650e0b62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.088 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:47:14.089 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:47:14.089 DEBUG:boto:path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.089 DEBUG:boto:auth_path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.089 DEBUG:boto:Method: HEAD 2023-09-22T21:47:14.089 DEBUG:boto:Path: /ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.089 DEBUG:boto:Data: 2023-09-22T21:47:14.089 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.089 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.090 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.090 DEBUG:boto:Params: {} 2023-09-22T21:47:14.090 DEBUG:boto:Token: None 2023-09-22T21:47:14.090 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.090 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ehI/PK0odVycVH9vsb4D5xXiUWg= 2023-09-22T21:47:14.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ehI/PK0odVycVH9vsb4D5xXiUWg=', 'Content-Length': '0'} 2023-09-22T21:47:14.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005208550ff9f1215d-00650e0b62-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.094 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:47:14.094 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:47:14.094 DEBUG:boto:path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.094 DEBUG:boto:auth_path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.094 DEBUG:boto:Method: HEAD 2023-09-22T21:47:14.094 DEBUG:boto:Path: /ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.095 DEBUG:boto:Data: 2023-09-22T21:47:14.095 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.095 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.095 DEBUG:boto:Params: {} 2023-09-22T21:47:14.095 DEBUG:boto:Token: None 2023-09-22T21:47:14.095 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.096 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ehI/PK0odVycVH9vsb4D5xXiUWg= 2023-09-22T21:47:14.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ehI/PK0odVycVH9vsb4D5xXiUWg=', 'Content-Length': '0'} 2023-09-22T21:47:14.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066836c53626d7a62-00650e0b62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-22T21:47:14.099 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:47:14.099 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:47:14.099 DEBUG:boto:path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.099 DEBUG:boto:auth_path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.099 DEBUG:boto:Method: GET 2023-09-22T21:47:14.099 DEBUG:boto:Path: /ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.099 DEBUG:boto:Data: 2023-09-22T21:47:14.099 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.100 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.100 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.100 DEBUG:boto:Params: {} 2023-09-22T21:47:14.100 DEBUG:boto:Token: None 2023-09-22T21:47:14.100 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.100 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Geu3KDeOjDmQxX+eMM83k+FtloM= 2023-09-22T21:47:14.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Geu3KDeOjDmQxX+eMM83k+FtloM=', 'Content-Length': '0'} 2023-09-22T21:47:14.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017ded48aed2ceb50-00650e0b62-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.146 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:47:14.147 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:47:14.147 DEBUG:boto:path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.147 DEBUG:boto:auth_path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.147 DEBUG:boto:Method: GET 2023-09-22T21:47:14.147 DEBUG:boto:Path: /ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.147 DEBUG:boto:Data: 2023-09-22T21:47:14.147 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.147 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.148 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.148 DEBUG:boto:Params: {} 2023-09-22T21:47:14.148 DEBUG:boto:Token: None 2023-09-22T21:47:14.148 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj62?versionId=null 2023-09-22T21:47:14.148 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Geu3KDeOjDmQxX+eMM83k+FtloM= 2023-09-22T21:47:14.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Geu3KDeOjDmQxX+eMM83k+FtloM=', 'Content-Length': '0'} 2023-09-22T21:47:14.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e18b47a3780f68f-00650e0b62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.151 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:14.152 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:14.152 DEBUG:boto:path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:47:14.152 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:47:14.152 DEBUG:boto:Method: GET 2023-09-22T21:47:14.152 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj62 2023-09-22T21:47:14.152 DEBUG:boto:Data: 2023-09-22T21:47:14.152 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.152 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.153 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.153 DEBUG:boto:Params: {} 2023-09-22T21:47:14.153 DEBUG:boto:Token: None 2023-09-22T21:47:14.153 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/ 2023-09-22T21:47:14.153 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA= 2023-09-22T21:47:14.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA=', 'Content-Length': '0'} 2023-09-22T21:47:14.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ada4c75c83a8b58a-00650e0b62-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.158 DEBUG:boto:b'ffzdqu-59obj621000falseobj622023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:14.159 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:14.159 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:14.159 DEBUG:boto:path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:47:14.159 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:47:14.159 DEBUG:boto:Method: GET 2023-09-22T21:47:14.160 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj62 2023-09-22T21:47:14.160 DEBUG:boto:Data: 2023-09-22T21:47:14.160 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.160 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.160 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.160 DEBUG:boto:Params: {} 2023-09-22T21:47:14.160 DEBUG:boto:Token: None 2023-09-22T21:47:14.160 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/ 2023-09-22T21:47:14.161 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA= 2023-09-22T21:47:14.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA=', 'Content-Length': '0'} 2023-09-22T21:47:14.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000191c358eb19003be-00650e0b62-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.173 DEBUG:boto:b'ffzdqu-59obj621000falseobj622023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:14.173 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:47:14.173 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:47:14.173 DEBUG:boto:Method: HEAD 2023-09-22T21:47:14.174 DEBUG:boto:Path: /ffzdqu-59/obj62 2023-09-22T21:47:14.174 DEBUG:boto:Data: 2023-09-22T21:47:14.174 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.174 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.174 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.174 DEBUG:boto:Params: {} 2023-09-22T21:47:14.174 DEBUG:boto:Token: None 2023-09-22T21:47:14.174 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj62 2023-09-22T21:47:14.175 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UXZYR8/iZ6nkDg0yPdO4B3u59zU= 2023-09-22T21:47:14.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UXZYR8/iZ6nkDg0yPdO4B3u59zU=', 'Content-Length': '0'} 2023-09-22T21:47:14.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000355daf5e5de372-00650e0b62-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.178 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:47:14.178 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:47:14.178 DEBUG:boto:Method: HEAD 2023-09-22T21:47:14.178 DEBUG:boto:Path: /ffzdqu-59/obj62 2023-09-22T21:47:14.178 DEBUG:boto:Data: 2023-09-22T21:47:14.178 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.178 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.178 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.179 DEBUG:boto:Params: {} 2023-09-22T21:47:14.179 DEBUG:boto:Token: None 2023-09-22T21:47:14.179 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj62 2023-09-22T21:47:14.179 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UXZYR8/iZ6nkDg0yPdO4B3u59zU= 2023-09-22T21:47:14.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UXZYR8/iZ6nkDg0yPdO4B3u59zU=', 'Content-Length': '0'} 2023-09-22T21:47:14.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066e6c5fb56065d58-00650e0b62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-22T21:47:14.182 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:47:14.182 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:47:14.182 DEBUG:boto:Method: GET 2023-09-22T21:47:14.182 DEBUG:boto:Path: /ffzdqu-59/obj62 2023-09-22T21:47:14.182 DEBUG:boto:Data: 2023-09-22T21:47:14.182 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.183 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.183 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.183 DEBUG:boto:Params: {} 2023-09-22T21:47:14.183 DEBUG:boto:Token: None 2023-09-22T21:47:14.183 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj62 2023-09-22T21:47:14.183 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/j4XTQ5LnyEjx0MYD73L/HTkdr8= 2023-09-22T21:47:14.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/j4XTQ5LnyEjx0MYD73L/HTkdr8=', 'Content-Length': '0'} 2023-09-22T21:47:14.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c698e69e2638dfde-00650e0b62-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.226 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:47:14.226 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:47:14.227 DEBUG:boto:Method: GET 2023-09-22T21:47:14.227 DEBUG:boto:Path: /ffzdqu-59/obj62 2023-09-22T21:47:14.227 DEBUG:boto:Data: 2023-09-22T21:47:14.227 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.227 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.227 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.227 DEBUG:boto:Params: {} 2023-09-22T21:47:14.227 DEBUG:boto:Token: None 2023-09-22T21:47:14.227 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj62 2023-09-22T21:47:14.228 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/j4XTQ5LnyEjx0MYD73L/HTkdr8= 2023-09-22T21:47:14.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/j4XTQ5LnyEjx0MYD73L/HTkdr8=', 'Content-Length': '0'} 2023-09-22T21:47:14.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cd1969214a2dd0f-00650e0b62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.231 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:14.231 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:14.231 DEBUG:boto:path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:47:14.231 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:47:14.231 DEBUG:boto:Method: GET 2023-09-22T21:47:14.231 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj62 2023-09-22T21:47:14.231 DEBUG:boto:Data: 2023-09-22T21:47:14.232 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.232 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.232 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.232 DEBUG:boto:Params: {} 2023-09-22T21:47:14.232 DEBUG:boto:Token: None 2023-09-22T21:47:14.232 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/ 2023-09-22T21:47:14.233 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA= 2023-09-22T21:47:14.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA=', 'Content-Length': '0'} 2023-09-22T21:47:14.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fa97a6ac0a4412f-00650e0b62-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.238 DEBUG:boto:b'ffzdqu-59obj621000falseobj622023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:14.238 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:14.238 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:14.239 DEBUG:boto:path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:47:14.239 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:47:14.239 DEBUG:boto:Method: GET 2023-09-22T21:47:14.239 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj62 2023-09-22T21:47:14.239 DEBUG:boto:Data: 2023-09-22T21:47:14.239 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.239 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.239 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.239 DEBUG:boto:Params: {} 2023-09-22T21:47:14.239 DEBUG:boto:Token: None 2023-09-22T21:47:14.240 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/ 2023-09-22T21:47:14.240 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA= 2023-09-22T21:47:14.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA=', 'Content-Length': '0'} 2023-09-22T21:47:14.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d99fb451a18fbab-00650e0b62-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.249 DEBUG:boto:b'ffzdqu-59obj621000falseobj622023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:14.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:47:14.249 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:47:14.250 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:47:14.250 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.250 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.250 DEBUG:boto:Method: GET 2023-09-22T21:47:14.250 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.250 DEBUG:boto:Data: 2023-09-22T21:47:14.250 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.250 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.250 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.251 DEBUG:boto:Params: {} 2023-09-22T21:47:14.251 DEBUG:boto:Token: None 2023-09-22T21:47:14.251 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.251 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nEur6ljIOj1rG6c5N/rOvOC+6sY= 2023-09-22T21:47:14.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nEur6ljIOj1rG6c5N/rOvOC+6sY=', 'Content-Length': '0'} 2023-09-22T21:47:14.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f08bdf4c9bb67a7c-00650e0b62-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.254 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:47:14.254 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:47:14.255 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.255 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.255 DEBUG:boto:Method: GET 2023-09-22T21:47:14.255 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.255 DEBUG:boto:Data: 2023-09-22T21:47:14.255 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.255 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.255 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.255 DEBUG:boto:Params: {} 2023-09-22T21:47:14.256 DEBUG:boto:Token: None 2023-09-22T21:47:14.256 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.256 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nEur6ljIOj1rG6c5N/rOvOC+6sY= 2023-09-22T21:47:14.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nEur6ljIOj1rG6c5N/rOvOC+6sY=', 'Content-Length': '0'} 2023-09-22T21:47:14.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000321d928c9e8af8d0-00650e0b62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.259 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:47:14.259 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:47:14.259 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.259 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.259 DEBUG:boto:Method: HEAD 2023-09-22T21:47:14.259 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.260 DEBUG:boto:Data: 2023-09-22T21:47:14.260 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.260 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.260 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.260 DEBUG:boto:Params: {} 2023-09-22T21:47:14.260 DEBUG:boto:Token: None 2023-09-22T21:47:14.260 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.260 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Dc06T9l4Lpj87/rVFuANIdwvjcg= 2023-09-22T21:47:14.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Dc06T9l4Lpj87/rVFuANIdwvjcg=', 'Content-Length': '0'} 2023-09-22T21:47:14.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000915556e7ae25f608-00650e0b62-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.264 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:47:14.264 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:47:14.264 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.265 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.265 DEBUG:boto:Method: HEAD 2023-09-22T21:47:14.265 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.265 DEBUG:boto:Data: 2023-09-22T21:47:14.265 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.265 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.265 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.265 DEBUG:boto:Params: {} 2023-09-22T21:47:14.266 DEBUG:boto:Token: None 2023-09-22T21:47:14.266 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.266 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Dc06T9l4Lpj87/rVFuANIdwvjcg= 2023-09-22T21:47:14.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Dc06T9l4Lpj87/rVFuANIdwvjcg=', 'Content-Length': '0'} 2023-09-22T21:47:14.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ecc6b9522250b5e-00650e0b62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:47:14.269 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:47:14.269 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:47:14.269 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.269 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.270 DEBUG:boto:Method: GET 2023-09-22T21:47:14.270 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.270 DEBUG:boto:Data: 2023-09-22T21:47:14.270 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.270 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.270 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.270 DEBUG:boto:Params: {} 2023-09-22T21:47:14.270 DEBUG:boto:Token: None 2023-09-22T21:47:14.271 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.271 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nEur6ljIOj1rG6c5N/rOvOC+6sY= 2023-09-22T21:47:14.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nEur6ljIOj1rG6c5N/rOvOC+6sY=', 'Content-Length': '0'} 2023-09-22T21:47:14.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000596a7ab95595c86f-00650e0b62-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.318 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:47:14.318 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:47:14.319 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.319 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.319 DEBUG:boto:Method: GET 2023-09-22T21:47:14.319 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.319 DEBUG:boto:Data: 2023-09-22T21:47:14.319 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.319 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.319 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.320 DEBUG:boto:Params: {} 2023-09-22T21:47:14.320 DEBUG:boto:Token: None 2023-09-22T21:47:14.320 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:47:14.320 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nEur6ljIOj1rG6c5N/rOvOC+6sY= 2023-09-22T21:47:14.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nEur6ljIOj1rG6c5N/rOvOC+6sY=', 'Content-Length': '0'} 2023-09-22T21:47:14.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017b8a1d4562edee8-00650e0b62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.323 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:14.324 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:14.324 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:47:14.324 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:47:14.324 DEBUG:boto:Method: GET 2023-09-22T21:47:14.324 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:47:14.324 DEBUG:boto:Data: 2023-09-22T21:47:14.324 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.324 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.324 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.325 DEBUG:boto:Params: {} 2023-09-22T21:47:14.325 DEBUG:boto:Token: None 2023-09-22T21:47:14.325 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/ 2023-09-22T21:47:14.325 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA= 2023-09-22T21:47:14.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA=', 'Content-Length': '0'} 2023-09-22T21:47:14.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fecfa15be7a513a-00650e0b62-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.330 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:14.331 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:14.331 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:14.331 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:47:14.331 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:47:14.331 DEBUG:boto:Method: GET 2023-09-22T21:47:14.331 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:47:14.331 DEBUG:boto:Data: 2023-09-22T21:47:14.331 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.331 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.331 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.332 DEBUG:boto:Params: {} 2023-09-22T21:47:14.332 DEBUG:boto:Token: None 2023-09-22T21:47:14.332 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/ 2023-09-22T21:47:14.332 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA= 2023-09-22T21:47:14.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA=', 'Content-Length': '0'} 2023-09-22T21:47:14.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000948c37edb12958b6-00650e0b62-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.342 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:14.342 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:47:14.342 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:47:14.342 DEBUG:boto:Method: HEAD 2023-09-22T21:47:14.343 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:47:14.343 DEBUG:boto:Data: 2023-09-22T21:47:14.343 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.343 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.343 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.343 DEBUG:boto:Params: {} 2023-09-22T21:47:14.343 DEBUG:boto:Token: None 2023-09-22T21:47:14.343 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj68 2023-09-22T21:47:14.343 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1+nS0NKVO9P+jI0Sm9OFKCw5iyQ= 2023-09-22T21:47:14.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1+nS0NKVO9P+jI0Sm9OFKCw5iyQ=', 'Content-Length': '0'} 2023-09-22T21:47:14.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c890ae0a82e50cc0-00650e0b62-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.346 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:47:14.347 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:47:14.347 DEBUG:boto:Method: HEAD 2023-09-22T21:47:14.347 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:47:14.347 DEBUG:boto:Data: 2023-09-22T21:47:14.347 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.347 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.347 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.347 DEBUG:boto:Params: {} 2023-09-22T21:47:14.347 DEBUG:boto:Token: None 2023-09-22T21:47:14.348 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj68 2023-09-22T21:47:14.348 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1+nS0NKVO9P+jI0Sm9OFKCw5iyQ= 2023-09-22T21:47:14.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1+nS0NKVO9P+jI0Sm9OFKCw5iyQ=', 'Content-Length': '0'} 2023-09-22T21:47:14.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f75d2cd22ee27c37-00650e0b62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:47:14.351 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:47:14.351 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:47:14.351 DEBUG:boto:Method: GET 2023-09-22T21:47:14.351 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:47:14.351 DEBUG:boto:Data: 2023-09-22T21:47:14.352 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.352 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.352 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.352 DEBUG:boto:Params: {} 2023-09-22T21:47:14.352 DEBUG:boto:Token: None 2023-09-22T21:47:14.352 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj68 2023-09-22T21:47:14.352 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/EC9ITabBl48iH5/qotWl3/uu9M= 2023-09-22T21:47:14.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/EC9ITabBl48iH5/qotWl3/uu9M=', 'Content-Length': '0'} 2023-09-22T21:47:14.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000953a0998d5b275d6-00650e0b62-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.398 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:47:14.398 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:47:14.399 DEBUG:boto:Method: GET 2023-09-22T21:47:14.399 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:47:14.399 DEBUG:boto:Data: 2023-09-22T21:47:14.399 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.399 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.399 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.399 DEBUG:boto:Params: {} 2023-09-22T21:47:14.399 DEBUG:boto:Token: None 2023-09-22T21:47:14.399 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj68 2023-09-22T21:47:14.400 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/EC9ITabBl48iH5/qotWl3/uu9M= 2023-09-22T21:47:14.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/EC9ITabBl48iH5/qotWl3/uu9M=', 'Content-Length': '0'} 2023-09-22T21:47:14.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026bb9e013416da50-00650e0b62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.403 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:14.403 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:14.403 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:47:14.403 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:47:14.403 DEBUG:boto:Method: GET 2023-09-22T21:47:14.403 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:47:14.403 DEBUG:boto:Data: 2023-09-22T21:47:14.404 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.404 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.404 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.404 DEBUG:boto:Params: {} 2023-09-22T21:47:14.404 DEBUG:boto:Token: None 2023-09-22T21:47:14.404 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/ 2023-09-22T21:47:14.404 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA= 2023-09-22T21:47:14.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA=', 'Content-Length': '0'} 2023-09-22T21:47:14.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc730f22da051e33-00650e0b62-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.409 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:14.410 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:14.410 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:14.410 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:47:14.410 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:47:14.410 DEBUG:boto:Method: GET 2023-09-22T21:47:14.410 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:47:14.410 DEBUG:boto:Data: 2023-09-22T21:47:14.411 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.411 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.411 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.411 DEBUG:boto:Params: {} 2023-09-22T21:47:14.411 DEBUG:boto:Token: None 2023-09-22T21:47:14.411 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/ 2023-09-22T21:47:14.411 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA= 2023-09-22T21:47:14.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA=', 'Content-Length': '0'} 2023-09-22T21:47:14.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecb8997763e847fe-00650e0b62-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.462 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:14.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-22T21:47:14.463 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:47:14.463 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:47:14.463 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.463 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.463 DEBUG:boto:Method: GET 2023-09-22T21:47:14.463 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.464 DEBUG:boto:Data: 2023-09-22T21:47:14.464 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.464 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.464 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.464 DEBUG:boto:Params: {} 2023-09-22T21:47:14.464 DEBUG:boto:Token: None 2023-09-22T21:47:14.464 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.464 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q9Yy23F9gi3nVq/ytHo7sTF5sx0= 2023-09-22T21:47:14.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9Yy23F9gi3nVq/ytHo7sTF5sx0=', 'Content-Length': '0'} 2023-09-22T21:47:14.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002138b7c4a06f51ac-00650e0b62-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.468 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:47:14.468 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:47:14.468 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.468 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.468 DEBUG:boto:Method: GET 2023-09-22T21:47:14.468 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.468 DEBUG:boto:Data: 2023-09-22T21:47:14.468 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.469 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.469 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.469 DEBUG:boto:Params: {} 2023-09-22T21:47:14.469 DEBUG:boto:Token: None 2023-09-22T21:47:14.469 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.469 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q9Yy23F9gi3nVq/ytHo7sTF5sx0= 2023-09-22T21:47:14.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9Yy23F9gi3nVq/ytHo7sTF5sx0=', 'Content-Length': '0'} 2023-09-22T21:47:14.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006da7b01081930efd-00650e0b62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.472 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:47:14.472 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:47:14.472 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.473 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.473 DEBUG:boto:Method: HEAD 2023-09-22T21:47:14.473 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.473 DEBUG:boto:Data: 2023-09-22T21:47:14.473 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.473 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.473 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.473 DEBUG:boto:Params: {} 2023-09-22T21:47:14.473 DEBUG:boto:Token: None 2023-09-22T21:47:14.474 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.474 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vMzw5rNhjErpJ70MBhp5lkjttvs= 2023-09-22T21:47:14.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vMzw5rNhjErpJ70MBhp5lkjttvs=', 'Content-Length': '0'} 2023-09-22T21:47:14.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000934a4dfc1964e177-00650e0b62-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.477 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:47:14.477 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:47:14.477 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.477 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.477 DEBUG:boto:Method: HEAD 2023-09-22T21:47:14.477 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.477 DEBUG:boto:Data: 2023-09-22T21:47:14.477 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.478 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.478 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.478 DEBUG:boto:Params: {} 2023-09-22T21:47:14.478 DEBUG:boto:Token: None 2023-09-22T21:47:14.478 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.478 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vMzw5rNhjErpJ70MBhp5lkjttvs= 2023-09-22T21:47:14.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vMzw5rNhjErpJ70MBhp5lkjttvs=', 'Content-Length': '0'} 2023-09-22T21:47:14.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065243e11c83e958b-00650e0b62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-22T21:47:14.481 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:47:14.481 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:47:14.481 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.482 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.482 DEBUG:boto:Method: GET 2023-09-22T21:47:14.482 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.482 DEBUG:boto:Data: 2023-09-22T21:47:14.482 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.482 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.482 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.482 DEBUG:boto:Params: {} 2023-09-22T21:47:14.482 DEBUG:boto:Token: None 2023-09-22T21:47:14.483 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.483 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q9Yy23F9gi3nVq/ytHo7sTF5sx0= 2023-09-22T21:47:14.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9Yy23F9gi3nVq/ytHo7sTF5sx0=', 'Content-Length': '0'} 2023-09-22T21:47:14.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093a0e24a004faa5c-00650e0b62-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.526 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:47:14.527 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:47:14.527 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.527 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.527 DEBUG:boto:Method: GET 2023-09-22T21:47:14.527 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.527 DEBUG:boto:Data: 2023-09-22T21:47:14.527 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.527 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.527 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.528 DEBUG:boto:Params: {} 2023-09-22T21:47:14.528 DEBUG:boto:Token: None 2023-09-22T21:47:14.528 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:47:14.528 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q9Yy23F9gi3nVq/ytHo7sTF5sx0= 2023-09-22T21:47:14.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9Yy23F9gi3nVq/ytHo7sTF5sx0=', 'Content-Length': '0'} 2023-09-22T21:47:14.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000290c7c78414e1016-00650e0b62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.531 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:14.531 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:14.532 DEBUG:boto:path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:47:14.532 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:47:14.532 DEBUG:boto:Method: GET 2023-09-22T21:47:14.532 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj69 2023-09-22T21:47:14.532 DEBUG:boto:Data: 2023-09-22T21:47:14.532 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.532 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.532 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.532 DEBUG:boto:Params: {} 2023-09-22T21:47:14.532 DEBUG:boto:Token: None 2023-09-22T21:47:14.533 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/ 2023-09-22T21:47:14.533 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA= 2023-09-22T21:47:14.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA=', 'Content-Length': '0'} 2023-09-22T21:47:14.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d56c6e0e2daffc8-00650e0b62-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.538 DEBUG:boto:b'ffzdqu-59obj691000falseobj692023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:14.538 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:14.538 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:14.538 DEBUG:boto:path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:47:14.538 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:47:14.539 DEBUG:boto:Method: GET 2023-09-22T21:47:14.539 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj69 2023-09-22T21:47:14.539 DEBUG:boto:Data: 2023-09-22T21:47:14.539 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.539 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.539 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.539 DEBUG:boto:Params: {} 2023-09-22T21:47:14.539 DEBUG:boto:Token: None 2023-09-22T21:47:14.540 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/ 2023-09-22T21:47:14.540 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA= 2023-09-22T21:47:14.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA=', 'Content-Length': '0'} 2023-09-22T21:47:14.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d974a8328dcde5a3-00650e0b62-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.590 DEBUG:boto:b'ffzdqu-59obj691000falseobj692023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:14.708 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:47:14.708 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:47:14.708 DEBUG:boto:Method: HEAD 2023-09-22T21:47:14.708 DEBUG:boto:Path: /ffzdqu-59/obj69 2023-09-22T21:47:14.708 DEBUG:boto:Data: 2023-09-22T21:47:14.709 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.709 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.709 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.709 DEBUG:boto:Params: {} 2023-09-22T21:47:14.709 DEBUG:boto:Token: None 2023-09-22T21:47:14.709 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj69 2023-09-22T21:47:14.709 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cjW4v9BoYwq/b5ZKhW2nF2kZrfk= 2023-09-22T21:47:14.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cjW4v9BoYwq/b5ZKhW2nF2kZrfk=', 'Content-Length': '0'} 2023-09-22T21:47:14.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc47cf80abf4d333-00650e0b62-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.713 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:47:14.713 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:47:14.713 DEBUG:boto:Method: HEAD 2023-09-22T21:47:14.713 DEBUG:boto:Path: /ffzdqu-59/obj69 2023-09-22T21:47:14.713 DEBUG:boto:Data: 2023-09-22T21:47:14.713 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.713 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.714 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.714 DEBUG:boto:Params: {} 2023-09-22T21:47:14.714 DEBUG:boto:Token: None 2023-09-22T21:47:14.714 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj69 2023-09-22T21:47:14.714 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cjW4v9BoYwq/b5ZKhW2nF2kZrfk= 2023-09-22T21:47:14.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cjW4v9BoYwq/b5ZKhW2nF2kZrfk=', 'Content-Length': '0'} 2023-09-22T21:47:14.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000205225d3e11ace15-00650e0b62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-22T21:47:14.718 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:47:14.718 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:47:14.718 DEBUG:boto:Method: GET 2023-09-22T21:47:14.718 DEBUG:boto:Path: /ffzdqu-59/obj69 2023-09-22T21:47:14.718 DEBUG:boto:Data: 2023-09-22T21:47:14.718 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.718 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.718 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.719 DEBUG:boto:Params: {} 2023-09-22T21:47:14.719 DEBUG:boto:Token: None 2023-09-22T21:47:14.719 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj69 2023-09-22T21:47:14.719 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iGDF9xN19pjkSVrEBUVll5U63/U= 2023-09-22T21:47:14.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iGDF9xN19pjkSVrEBUVll5U63/U=', 'Content-Length': '0'} 2023-09-22T21:47:14.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000160d27f39b3f7f27-00650e0b62-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.762 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:47:14.763 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:47:14.763 DEBUG:boto:Method: GET 2023-09-22T21:47:14.763 DEBUG:boto:Path: /ffzdqu-59/obj69 2023-09-22T21:47:14.763 DEBUG:boto:Data: 2023-09-22T21:47:14.763 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.763 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.763 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.763 DEBUG:boto:Params: {} 2023-09-22T21:47:14.763 DEBUG:boto:Token: None 2023-09-22T21:47:14.764 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj69 2023-09-22T21:47:14.764 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iGDF9xN19pjkSVrEBUVll5U63/U= 2023-09-22T21:47:14.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iGDF9xN19pjkSVrEBUVll5U63/U=', 'Content-Length': '0'} 2023-09-22T21:47:14.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054fe445e2e7d7c98-00650e0b62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.767 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:14.767 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:14.767 DEBUG:boto:path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:47:14.767 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:47:14.767 DEBUG:boto:Method: GET 2023-09-22T21:47:14.768 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj69 2023-09-22T21:47:14.768 DEBUG:boto:Data: 2023-09-22T21:47:14.768 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.768 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.768 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.768 DEBUG:boto:Params: {} 2023-09-22T21:47:14.768 DEBUG:boto:Token: None 2023-09-22T21:47:14.768 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/ 2023-09-22T21:47:14.768 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA= 2023-09-22T21:47:14.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA=', 'Content-Length': '0'} 2023-09-22T21:47:14.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003df35c02f06ee4b5-00650e0b62-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.773 DEBUG:boto:b'ffzdqu-59obj691000falseobj692023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:14.774 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:14.774 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:14.774 DEBUG:boto:path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:47:14.774 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:47:14.774 DEBUG:boto:Method: GET 2023-09-22T21:47:14.774 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj69 2023-09-22T21:47:14.774 DEBUG:boto:Data: 2023-09-22T21:47:14.775 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.775 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.775 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.775 DEBUG:boto:Params: {} 2023-09-22T21:47:14.775 DEBUG:boto:Token: None 2023-09-22T21:47:14.775 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/ 2023-09-22T21:47:14.775 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA= 2023-09-22T21:47:14.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA=', 'Content-Length': '0'} 2023-09-22T21:47:14.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f960c6f01bca64f4-00650e0b62-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.785 DEBUG:boto:b'ffzdqu-59obj691000falseobj692023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:14.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T21:47:14.785 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:47:14.785 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:47:14.786 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.786 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.786 DEBUG:boto:Method: GET 2023-09-22T21:47:14.786 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.786 DEBUG:boto:Data: 2023-09-22T21:47:14.786 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.786 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.786 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.786 DEBUG:boto:Params: {} 2023-09-22T21:47:14.787 DEBUG:boto:Token: None 2023-09-22T21:47:14.787 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.787 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J5XxYSrwkOpau5TtUlVJcjPV7/Y= 2023-09-22T21:47:14.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J5XxYSrwkOpau5TtUlVJcjPV7/Y=', 'Content-Length': '0'} 2023-09-22T21:47:14.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa0e7be9119ac7f9-00650e0b62-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.830 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:47:14.830 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:47:14.831 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.831 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.831 DEBUG:boto:Method: GET 2023-09-22T21:47:14.831 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.831 DEBUG:boto:Data: 2023-09-22T21:47:14.831 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.831 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.831 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.831 DEBUG:boto:Params: {} 2023-09-22T21:47:14.831 DEBUG:boto:Token: None 2023-09-22T21:47:14.832 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.832 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J5XxYSrwkOpau5TtUlVJcjPV7/Y= 2023-09-22T21:47:14.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J5XxYSrwkOpau5TtUlVJcjPV7/Y=', 'Content-Length': '0'} 2023-09-22T21:47:14.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da537a77d3d03f1f-00650e0b62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.836 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:47:14.837 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:47:14.837 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.837 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.837 DEBUG:boto:Method: HEAD 2023-09-22T21:47:14.837 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.837 DEBUG:boto:Data: 2023-09-22T21:47:14.837 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.837 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.837 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.837 DEBUG:boto:Params: {} 2023-09-22T21:47:14.838 DEBUG:boto:Token: None 2023-09-22T21:47:14.838 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.838 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iKGoz1XHQ2HXwB3XbUDYng+6dqI= 2023-09-22T21:47:14.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iKGoz1XHQ2HXwB3XbUDYng+6dqI=', 'Content-Length': '0'} 2023-09-22T21:47:14.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003253ded577466122-00650e0b62-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.841 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:47:14.842 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:47:14.842 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.842 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.842 DEBUG:boto:Method: HEAD 2023-09-22T21:47:14.842 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.842 DEBUG:boto:Data: 2023-09-22T21:47:14.842 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.842 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.842 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.843 DEBUG:boto:Params: {} 2023-09-22T21:47:14.843 DEBUG:boto:Token: None 2023-09-22T21:47:14.843 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.843 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iKGoz1XHQ2HXwB3XbUDYng+6dqI= 2023-09-22T21:47:14.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iKGoz1XHQ2HXwB3XbUDYng+6dqI=', 'Content-Length': '0'} 2023-09-22T21:47:14.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abb6ea263cb3de4e-00650e0b62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T21:47:14.848 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:47:14.848 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:47:14.848 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.848 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.848 DEBUG:boto:Method: GET 2023-09-22T21:47:14.848 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.848 DEBUG:boto:Data: 2023-09-22T21:47:14.849 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.849 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.849 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.849 DEBUG:boto:Params: {} 2023-09-22T21:47:14.849 DEBUG:boto:Token: None 2023-09-22T21:47:14.849 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.849 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J5XxYSrwkOpau5TtUlVJcjPV7/Y= 2023-09-22T21:47:14.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J5XxYSrwkOpau5TtUlVJcjPV7/Y=', 'Content-Length': '0'} 2023-09-22T21:47:14.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000520169086d2910ab-00650e0b62-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.898 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:47:14.899 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:47:14.899 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.899 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.899 DEBUG:boto:Method: GET 2023-09-22T21:47:14.899 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.899 DEBUG:boto:Data: 2023-09-22T21:47:14.899 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.899 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.899 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.899 DEBUG:boto:Params: {} 2023-09-22T21:47:14.900 DEBUG:boto:Token: None 2023-09-22T21:47:14.900 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:47:14.900 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J5XxYSrwkOpau5TtUlVJcjPV7/Y= 2023-09-22T21:47:14.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J5XxYSrwkOpau5TtUlVJcjPV7/Y=', 'Content-Length': '0'} 2023-09-22T21:47:14.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d0a14900d724fe5-00650e0b62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.903 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:14.904 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:14.904 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:47:14.904 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:47:14.904 DEBUG:boto:Method: GET 2023-09-22T21:47:14.904 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:47:14.904 DEBUG:boto:Data: 2023-09-22T21:47:14.904 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.904 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.904 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.904 DEBUG:boto:Params: {} 2023-09-22T21:47:14.905 DEBUG:boto:Token: None 2023-09-22T21:47:14.905 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/ 2023-09-22T21:47:14.905 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA= 2023-09-22T21:47:14.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA=', 'Content-Length': '0'} 2023-09-22T21:47:14.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dd5e709f51143eb-00650e0b62-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.910 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:14.911 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:14.911 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:14.911 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:47:14.911 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:47:14.911 DEBUG:boto:Method: GET 2023-09-22T21:47:14.911 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:47:14.911 DEBUG:boto:Data: 2023-09-22T21:47:14.911 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.911 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.912 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.912 DEBUG:boto:Params: {} 2023-09-22T21:47:14.912 DEBUG:boto:Token: None 2023-09-22T21:47:14.912 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/ 2023-09-22T21:47:14.912 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA= 2023-09-22T21:47:14.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:22NWmJtoY8LMA6ay/o1NeanjGDA=', 'Content-Length': '0'} 2023-09-22T21:47:14.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ac35d4e7539ea5b-00650e0b62-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.966 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:14.967 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:47:14.967 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:47:14.967 DEBUG:boto:Method: HEAD 2023-09-22T21:47:14.967 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:47:14.967 DEBUG:boto:Data: 2023-09-22T21:47:14.967 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.967 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.968 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.968 DEBUG:boto:Params: {} 2023-09-22T21:47:14.968 DEBUG:boto:Token: None 2023-09-22T21:47:14.968 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj76 2023-09-22T21:47:14.968 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Vz0HPQW4K2wqpbk438VJqWceiHk= 2023-09-22T21:47:14.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Vz0HPQW4K2wqpbk438VJqWceiHk=', 'Content-Length': '0'} 2023-09-22T21:47:14.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000677e0c5a6ac3940f-00650e0b62-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.972 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:47:14.972 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:47:14.972 DEBUG:boto:Method: HEAD 2023-09-22T21:47:14.972 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:47:14.972 DEBUG:boto:Data: 2023-09-22T21:47:14.972 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.972 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:14.972 DEBUG:boto:Port: 8000 2023-09-22T21:47:14.972 DEBUG:boto:Params: {} 2023-09-22T21:47:14.973 DEBUG:boto:Token: None 2023-09-22T21:47:14.973 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj76 2023-09-22T21:47:14.973 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Vz0HPQW4K2wqpbk438VJqWceiHk= 2023-09-22T21:47:14.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Vz0HPQW4K2wqpbk438VJqWceiHk=', 'Content-Length': '0'} 2023-09-22T21:47:14.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bbb9ad104741182-00650e0b62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:14.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T21:47:14.976 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:47:14.976 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:47:14.976 DEBUG:boto:Method: GET 2023-09-22T21:47:14.976 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:47:14.976 DEBUG:boto:Data: 2023-09-22T21:47:14.976 DEBUG:boto:Headers: {} 2023-09-22T21:47:14.976 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:14.976 DEBUG:boto:Port: 8001 2023-09-22T21:47:14.977 DEBUG:boto:Params: {} 2023-09-22T21:47:14.977 DEBUG:boto:Token: None 2023-09-22T21:47:14.977 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:14 GMT /ffzdqu-59/obj76 2023-09-22T21:47:14.977 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BDRa7Pl4zh/0rz2vIqDLFpqs6eQ= 2023-09-22T21:47:14.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BDRa7Pl4zh/0rz2vIqDLFpqs6eQ=', 'Content-Length': '0'} 2023-09-22T21:47:14.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fb219a12c991d49-00650e0b62-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.022 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:47:15.023 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:47:15.023 DEBUG:boto:Method: GET 2023-09-22T21:47:15.023 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:47:15.023 DEBUG:boto:Data: 2023-09-22T21:47:15.023 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.023 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.023 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.023 DEBUG:boto:Params: {} 2023-09-22T21:47:15.024 DEBUG:boto:Token: None 2023-09-22T21:47:15.024 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj76 2023-09-22T21:47:15.024 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ym0uWn9Z1LrfLh1RGZzrTxaPu24= 2023-09-22T21:47:15.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ym0uWn9Z1LrfLh1RGZzrTxaPu24=', 'Content-Length': '0'} 2023-09-22T21:47:15.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccf3abd7ef629727-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.027 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:15.027 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:15.027 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:47:15.027 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:47:15.027 DEBUG:boto:Method: GET 2023-09-22T21:47:15.028 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:47:15.028 DEBUG:boto:Data: 2023-09-22T21:47:15.028 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.028 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.028 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.028 DEBUG:boto:Params: {} 2023-09-22T21:47:15.028 DEBUG:boto:Token: None 2023-09-22T21:47:15.028 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/ 2023-09-22T21:47:15.028 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI= 2023-09-22T21:47:15.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI=', 'Content-Length': '0'} 2023-09-22T21:47:15.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fff2b89eb0a693a2-00650e0b63-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.033 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:15.034 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:15.034 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:15.034 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:47:15.034 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:47:15.034 DEBUG:boto:Method: GET 2023-09-22T21:47:15.034 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:47:15.034 DEBUG:boto:Data: 2023-09-22T21:47:15.034 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.034 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.035 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.035 DEBUG:boto:Params: {} 2023-09-22T21:47:15.035 DEBUG:boto:Token: None 2023-09-22T21:47:15.035 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/ 2023-09-22T21:47:15.035 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI= 2023-09-22T21:47:15.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI=', 'Content-Length': '0'} 2023-09-22T21:47:15.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc571848229d3ee6-00650e0b63-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.044 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:15.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:47:15.044 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:47:15.045 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:47:15.045 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.045 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.045 DEBUG:boto:Method: GET 2023-09-22T21:47:15.045 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.045 DEBUG:boto:Data: 2023-09-22T21:47:15.045 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.045 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.045 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.045 DEBUG:boto:Params: {} 2023-09-22T21:47:15.046 DEBUG:boto:Token: None 2023-09-22T21:47:15.046 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.046 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:I0nGBuXbJ29TUuy0LqEPK1fQ+2I= 2023-09-22T21:47:15.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:I0nGBuXbJ29TUuy0LqEPK1fQ+2I=', 'Content-Length': '0'} 2023-09-22T21:47:15.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056f78eca51a245d6-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.049 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:47:15.049 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:47:15.049 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.049 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.050 DEBUG:boto:Method: GET 2023-09-22T21:47:15.050 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.050 DEBUG:boto:Data: 2023-09-22T21:47:15.050 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.050 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.050 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.050 DEBUG:boto:Params: {} 2023-09-22T21:47:15.050 DEBUG:boto:Token: None 2023-09-22T21:47:15.050 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.050 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:I0nGBuXbJ29TUuy0LqEPK1fQ+2I= 2023-09-22T21:47:15.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:I0nGBuXbJ29TUuy0LqEPK1fQ+2I=', 'Content-Length': '0'} 2023-09-22T21:47:15.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e65d413fcf5c1948-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.098 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:47:15.099 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:47:15.099 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.099 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.099 DEBUG:boto:Method: HEAD 2023-09-22T21:47:15.099 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.099 DEBUG:boto:Data: 2023-09-22T21:47:15.099 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.099 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.099 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.100 DEBUG:boto:Params: {} 2023-09-22T21:47:15.100 DEBUG:boto:Token: None 2023-09-22T21:47:15.100 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.100 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t/awjnJVWWevE9pRqh6KuBjzj8I= 2023-09-22T21:47:15.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t/awjnJVWWevE9pRqh6KuBjzj8I=', 'Content-Length': '0'} 2023-09-22T21:47:15.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a771fcc7bb4fe11c-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.103 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:47:15.103 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:47:15.103 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.103 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.104 DEBUG:boto:Method: HEAD 2023-09-22T21:47:15.104 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.104 DEBUG:boto:Data: 2023-09-22T21:47:15.104 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.104 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.104 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.104 DEBUG:boto:Params: {} 2023-09-22T21:47:15.104 DEBUG:boto:Token: None 2023-09-22T21:47:15.104 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.105 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t/awjnJVWWevE9pRqh6KuBjzj8I= 2023-09-22T21:47:15.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t/awjnJVWWevE9pRqh6KuBjzj8I=', 'Content-Length': '0'} 2023-09-22T21:47:15.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbe84f76b1d3ab55-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:47:15.108 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:47:15.108 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:47:15.108 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.108 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.108 DEBUG:boto:Method: GET 2023-09-22T21:47:15.108 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.108 DEBUG:boto:Data: 2023-09-22T21:47:15.108 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.108 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.109 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.109 DEBUG:boto:Params: {} 2023-09-22T21:47:15.109 DEBUG:boto:Token: None 2023-09-22T21:47:15.109 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.109 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:I0nGBuXbJ29TUuy0LqEPK1fQ+2I= 2023-09-22T21:47:15.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:I0nGBuXbJ29TUuy0LqEPK1fQ+2I=', 'Content-Length': '0'} 2023-09-22T21:47:15.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a23c708e5313e6a5-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.154 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:47:15.154 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:47:15.155 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.155 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.155 DEBUG:boto:Method: GET 2023-09-22T21:47:15.155 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.155 DEBUG:boto:Data: 2023-09-22T21:47:15.155 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.155 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.155 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.155 DEBUG:boto:Params: {} 2023-09-22T21:47:15.156 DEBUG:boto:Token: None 2023-09-22T21:47:15.156 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:47:15.156 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:I0nGBuXbJ29TUuy0LqEPK1fQ+2I= 2023-09-22T21:47:15.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:I0nGBuXbJ29TUuy0LqEPK1fQ+2I=', 'Content-Length': '0'} 2023-09-22T21:47:15.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ece8ac238b71bfa-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.159 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:15.159 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:15.159 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:47:15.160 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:47:15.160 DEBUG:boto:Method: GET 2023-09-22T21:47:15.160 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:47:15.160 DEBUG:boto:Data: 2023-09-22T21:47:15.160 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.160 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.160 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.160 DEBUG:boto:Params: {} 2023-09-22T21:47:15.160 DEBUG:boto:Token: None 2023-09-22T21:47:15.161 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/ 2023-09-22T21:47:15.161 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI= 2023-09-22T21:47:15.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI=', 'Content-Length': '0'} 2023-09-22T21:47:15.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2472a211324684e-00650e0b63-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.166 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:15.167 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:15.167 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:15.167 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:47:15.167 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:47:15.167 DEBUG:boto:Method: GET 2023-09-22T21:47:15.167 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:47:15.167 DEBUG:boto:Data: 2023-09-22T21:47:15.167 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.167 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.168 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.168 DEBUG:boto:Params: {} 2023-09-22T21:47:15.168 DEBUG:boto:Token: None 2023-09-22T21:47:15.168 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/ 2023-09-22T21:47:15.168 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI= 2023-09-22T21:47:15.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI=', 'Content-Length': '0'} 2023-09-22T21:47:15.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004595f34dd8da4adf-00650e0b63-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.177 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:15.177 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:47:15.177 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:47:15.177 DEBUG:boto:Method: HEAD 2023-09-22T21:47:15.178 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:47:15.178 DEBUG:boto:Data: 2023-09-22T21:47:15.178 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.178 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.178 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.178 DEBUG:boto:Params: {} 2023-09-22T21:47:15.178 DEBUG:boto:Token: None 2023-09-22T21:47:15.178 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj85 2023-09-22T21:47:15.178 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+/1nR3Z5WPhCFsiGzo9y4l/LMJE= 2023-09-22T21:47:15.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+/1nR3Z5WPhCFsiGzo9y4l/LMJE=', 'Content-Length': '0'} 2023-09-22T21:47:15.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007da61704cfcbff92-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.181 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:47:15.181 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:47:15.181 DEBUG:boto:Method: HEAD 2023-09-22T21:47:15.182 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:47:15.182 DEBUG:boto:Data: 2023-09-22T21:47:15.182 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.182 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.182 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.182 DEBUG:boto:Params: {} 2023-09-22T21:47:15.182 DEBUG:boto:Token: None 2023-09-22T21:47:15.182 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj85 2023-09-22T21:47:15.182 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+/1nR3Z5WPhCFsiGzo9y4l/LMJE= 2023-09-22T21:47:15.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+/1nR3Z5WPhCFsiGzo9y4l/LMJE=', 'Content-Length': '0'} 2023-09-22T21:47:15.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005efebac9cf38c46d-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:47:15.186 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:47:15.186 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:47:15.186 DEBUG:boto:Method: GET 2023-09-22T21:47:15.186 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:47:15.186 DEBUG:boto:Data: 2023-09-22T21:47:15.186 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.186 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.187 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.187 DEBUG:boto:Params: {} 2023-09-22T21:47:15.187 DEBUG:boto:Token: None 2023-09-22T21:47:15.187 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj85 2023-09-22T21:47:15.187 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RGlw1lX5yaexkfUSlPYNm1LtI6M= 2023-09-22T21:47:15.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RGlw1lX5yaexkfUSlPYNm1LtI6M=', 'Content-Length': '0'} 2023-09-22T21:47:15.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000762d2f74485a16a3-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.234 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:47:15.235 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:47:15.235 DEBUG:boto:Method: GET 2023-09-22T21:47:15.235 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:47:15.235 DEBUG:boto:Data: 2023-09-22T21:47:15.235 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.235 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.235 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.235 DEBUG:boto:Params: {} 2023-09-22T21:47:15.235 DEBUG:boto:Token: None 2023-09-22T21:47:15.236 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj85 2023-09-22T21:47:15.236 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RGlw1lX5yaexkfUSlPYNm1LtI6M= 2023-09-22T21:47:15.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RGlw1lX5yaexkfUSlPYNm1LtI6M=', 'Content-Length': '0'} 2023-09-22T21:47:15.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fed2447c41d75901-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.239 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:15.239 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:15.240 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:47:15.240 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:47:15.240 DEBUG:boto:Method: GET 2023-09-22T21:47:15.240 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:47:15.240 DEBUG:boto:Data: 2023-09-22T21:47:15.240 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.240 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.240 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.240 DEBUG:boto:Params: {} 2023-09-22T21:47:15.240 DEBUG:boto:Token: None 2023-09-22T21:47:15.241 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/ 2023-09-22T21:47:15.241 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI= 2023-09-22T21:47:15.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI=', 'Content-Length': '0'} 2023-09-22T21:47:15.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c85277569aeb2078-00650e0b63-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.286 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:15.287 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:15.287 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:15.287 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:47:15.287 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:47:15.287 DEBUG:boto:Method: GET 2023-09-22T21:47:15.287 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:47:15.288 DEBUG:boto:Data: 2023-09-22T21:47:15.288 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.288 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.288 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.288 DEBUG:boto:Params: {} 2023-09-22T21:47:15.288 DEBUG:boto:Token: None 2023-09-22T21:47:15.288 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/ 2023-09-22T21:47:15.288 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI= 2023-09-22T21:47:15.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI=', 'Content-Length': '0'} 2023-09-22T21:47:15.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f3df9e57ecb7101-00650e0b63-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.297 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:15.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-22T21:47:15.297 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:47:15.297 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:47:15.297 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.298 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.298 DEBUG:boto:Method: GET 2023-09-22T21:47:15.298 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.298 DEBUG:boto:Data: 2023-09-22T21:47:15.298 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.298 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.298 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.298 DEBUG:boto:Params: {} 2023-09-22T21:47:15.298 DEBUG:boto:Token: None 2023-09-22T21:47:15.298 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.299 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:90BobO0AVRZpu/Yipb7YeisDHHM= 2023-09-22T21:47:15.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:90BobO0AVRZpu/Yipb7YeisDHHM=', 'Content-Length': '0'} 2023-09-22T21:47:15.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fa67c7ed7d76903-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.302 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:47:15.302 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:47:15.302 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.302 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.302 DEBUG:boto:Method: GET 2023-09-22T21:47:15.302 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.302 DEBUG:boto:Data: 2023-09-22T21:47:15.302 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.303 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.303 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.303 DEBUG:boto:Params: {} 2023-09-22T21:47:15.303 DEBUG:boto:Token: None 2023-09-22T21:47:15.303 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.303 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:90BobO0AVRZpu/Yipb7YeisDHHM= 2023-09-22T21:47:15.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:90BobO0AVRZpu/Yipb7YeisDHHM=', 'Content-Length': '0'} 2023-09-22T21:47:15.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c850214ce8376537-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.306 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:47:15.306 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:47:15.306 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.306 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.307 DEBUG:boto:Method: HEAD 2023-09-22T21:47:15.307 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.307 DEBUG:boto:Data: 2023-09-22T21:47:15.307 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.307 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.307 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.307 DEBUG:boto:Params: {} 2023-09-22T21:47:15.307 DEBUG:boto:Token: None 2023-09-22T21:47:15.307 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.308 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1Djxcy4Uc2EX4RAgJU1HfMTukQA= 2023-09-22T21:47:15.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1Djxcy4Uc2EX4RAgJU1HfMTukQA=', 'Content-Length': '0'} 2023-09-22T21:47:15.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054c0884f0569315d-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.310 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:47:15.310 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:47:15.311 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.311 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.311 DEBUG:boto:Method: HEAD 2023-09-22T21:47:15.311 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.311 DEBUG:boto:Data: 2023-09-22T21:47:15.311 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.311 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.311 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.311 DEBUG:boto:Params: {} 2023-09-22T21:47:15.311 DEBUG:boto:Token: None 2023-09-22T21:47:15.312 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.312 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1Djxcy4Uc2EX4RAgJU1HfMTukQA= 2023-09-22T21:47:15.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1Djxcy4Uc2EX4RAgJU1HfMTukQA=', 'Content-Length': '0'} 2023-09-22T21:47:15.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006386dcb047eb8934-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-22T21:47:15.314 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:47:15.315 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:47:15.315 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.315 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.315 DEBUG:boto:Method: GET 2023-09-22T21:47:15.315 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.315 DEBUG:boto:Data: 2023-09-22T21:47:15.315 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.315 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.315 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.315 DEBUG:boto:Params: {} 2023-09-22T21:47:15.316 DEBUG:boto:Token: None 2023-09-22T21:47:15.316 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.316 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:90BobO0AVRZpu/Yipb7YeisDHHM= 2023-09-22T21:47:15.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:90BobO0AVRZpu/Yipb7YeisDHHM=', 'Content-Length': '0'} 2023-09-22T21:47:15.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c762f01fc37ce49c-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.366 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:47:15.367 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:47:15.367 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.367 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.367 DEBUG:boto:Method: GET 2023-09-22T21:47:15.367 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.367 DEBUG:boto:Data: 2023-09-22T21:47:15.367 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.367 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.367 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.368 DEBUG:boto:Params: {} 2023-09-22T21:47:15.368 DEBUG:boto:Token: None 2023-09-22T21:47:15.368 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:47:15.368 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:90BobO0AVRZpu/Yipb7YeisDHHM= 2023-09-22T21:47:15.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:90BobO0AVRZpu/Yipb7YeisDHHM=', 'Content-Length': '0'} 2023-09-22T21:47:15.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b0b72c8ad0d6d16-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.372 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:15.372 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:15.372 DEBUG:boto:path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:47:15.372 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:47:15.372 DEBUG:boto:Method: GET 2023-09-22T21:47:15.372 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj87 2023-09-22T21:47:15.372 DEBUG:boto:Data: 2023-09-22T21:47:15.372 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.373 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.373 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.373 DEBUG:boto:Params: {} 2023-09-22T21:47:15.373 DEBUG:boto:Token: None 2023-09-22T21:47:15.373 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/ 2023-09-22T21:47:15.373 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI= 2023-09-22T21:47:15.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI=', 'Content-Length': '0'} 2023-09-22T21:47:15.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7d8ec8205bd9cef-00650e0b63-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.378 DEBUG:boto:b'ffzdqu-59obj871000falseobj872023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:15.378 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:15.379 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:15.379 DEBUG:boto:path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:47:15.379 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:47:15.379 DEBUG:boto:Method: GET 2023-09-22T21:47:15.379 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj87 2023-09-22T21:47:15.379 DEBUG:boto:Data: 2023-09-22T21:47:15.379 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.379 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.379 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.379 DEBUG:boto:Params: {} 2023-09-22T21:47:15.380 DEBUG:boto:Token: None 2023-09-22T21:47:15.380 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/ 2023-09-22T21:47:15.380 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI= 2023-09-22T21:47:15.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI=', 'Content-Length': '0'} 2023-09-22T21:47:15.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b821b0ee775d362-00650e0b63-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.389 DEBUG:boto:b'ffzdqu-59obj871000falseobj872023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:15.389 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:47:15.389 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:47:15.390 DEBUG:boto:Method: HEAD 2023-09-22T21:47:15.390 DEBUG:boto:Path: /ffzdqu-59/obj87 2023-09-22T21:47:15.390 DEBUG:boto:Data: 2023-09-22T21:47:15.390 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.390 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.390 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.390 DEBUG:boto:Params: {} 2023-09-22T21:47:15.390 DEBUG:boto:Token: None 2023-09-22T21:47:15.390 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj87 2023-09-22T21:47:15.391 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PJ/ZXr/OmFCV2wBx38wCVWea7g0= 2023-09-22T21:47:15.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PJ/ZXr/OmFCV2wBx38wCVWea7g0=', 'Content-Length': '0'} 2023-09-22T21:47:15.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ca669a5b02b3d7e-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.393 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:47:15.393 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:47:15.394 DEBUG:boto:Method: HEAD 2023-09-22T21:47:15.394 DEBUG:boto:Path: /ffzdqu-59/obj87 2023-09-22T21:47:15.394 DEBUG:boto:Data: 2023-09-22T21:47:15.394 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.394 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.394 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.394 DEBUG:boto:Params: {} 2023-09-22T21:47:15.394 DEBUG:boto:Token: None 2023-09-22T21:47:15.394 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj87 2023-09-22T21:47:15.395 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PJ/ZXr/OmFCV2wBx38wCVWea7g0= 2023-09-22T21:47:15.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PJ/ZXr/OmFCV2wBx38wCVWea7g0=', 'Content-Length': '0'} 2023-09-22T21:47:15.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d3febf08a369442-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-22T21:47:15.398 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:47:15.398 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:47:15.398 DEBUG:boto:Method: GET 2023-09-22T21:47:15.398 DEBUG:boto:Path: /ffzdqu-59/obj87 2023-09-22T21:47:15.399 DEBUG:boto:Data: 2023-09-22T21:47:15.399 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.399 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.399 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.399 DEBUG:boto:Params: {} 2023-09-22T21:47:15.399 DEBUG:boto:Token: None 2023-09-22T21:47:15.399 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj87 2023-09-22T21:47:15.399 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UvDWLGPlwIeX/x6yUt33XSoPN+A= 2023-09-22T21:47:15.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UvDWLGPlwIeX/x6yUt33XSoPN+A=', 'Content-Length': '0'} 2023-09-22T21:47:15.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef3c8dcca4146a24-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.442 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:47:15.442 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:47:15.443 DEBUG:boto:Method: GET 2023-09-22T21:47:15.443 DEBUG:boto:Path: /ffzdqu-59/obj87 2023-09-22T21:47:15.443 DEBUG:boto:Data: 2023-09-22T21:47:15.443 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.443 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.443 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.443 DEBUG:boto:Params: {} 2023-09-22T21:47:15.443 DEBUG:boto:Token: None 2023-09-22T21:47:15.443 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj87 2023-09-22T21:47:15.444 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UvDWLGPlwIeX/x6yUt33XSoPN+A= 2023-09-22T21:47:15.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UvDWLGPlwIeX/x6yUt33XSoPN+A=', 'Content-Length': '0'} 2023-09-22T21:47:15.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f29b120b7b22cb0-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.447 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:15.447 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:15.447 DEBUG:boto:path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:47:15.447 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:47:15.447 DEBUG:boto:Method: GET 2023-09-22T21:47:15.447 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj87 2023-09-22T21:47:15.447 DEBUG:boto:Data: 2023-09-22T21:47:15.448 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.448 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.448 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.448 DEBUG:boto:Params: {} 2023-09-22T21:47:15.448 DEBUG:boto:Token: None 2023-09-22T21:47:15.448 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/ 2023-09-22T21:47:15.448 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI= 2023-09-22T21:47:15.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI=', 'Content-Length': '0'} 2023-09-22T21:47:15.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087f6f3fd2c3f5523-00650e0b63-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.453 DEBUG:boto:b'ffzdqu-59obj871000falseobj872023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:15.454 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:15.454 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:15.454 DEBUG:boto:path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:47:15.454 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:47:15.454 DEBUG:boto:Method: GET 2023-09-22T21:47:15.454 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj87 2023-09-22T21:47:15.454 DEBUG:boto:Data: 2023-09-22T21:47:15.454 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.455 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.455 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.455 DEBUG:boto:Params: {} 2023-09-22T21:47:15.455 DEBUG:boto:Token: None 2023-09-22T21:47:15.455 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/ 2023-09-22T21:47:15.455 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI= 2023-09-22T21:47:15.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI=', 'Content-Length': '0'} 2023-09-22T21:47:15.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec7051a1524101d8-00650e0b63-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.464 DEBUG:boto:b'ffzdqu-59obj871000falseobj872023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:15.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-22T21:47:15.465 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:47:15.465 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:47:15.465 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.465 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.465 DEBUG:boto:Method: GET 2023-09-22T21:47:15.465 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.465 DEBUG:boto:Data: 2023-09-22T21:47:15.465 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.465 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.466 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.466 DEBUG:boto:Params: {} 2023-09-22T21:47:15.466 DEBUG:boto:Token: None 2023-09-22T21:47:15.466 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.466 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VOr6jMIJE9f6F6XjqZ87va51E9U= 2023-09-22T21:47:15.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VOr6jMIJE9f6F6XjqZ87va51E9U=', 'Content-Length': '0'} 2023-09-22T21:47:15.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b66da26555cdaa27-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.469 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:47:15.469 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:47:15.469 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.469 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.470 DEBUG:boto:Method: GET 2023-09-22T21:47:15.470 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.470 DEBUG:boto:Data: 2023-09-22T21:47:15.470 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.470 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.470 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.470 DEBUG:boto:Params: {} 2023-09-22T21:47:15.470 DEBUG:boto:Token: None 2023-09-22T21:47:15.471 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.471 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VOr6jMIJE9f6F6XjqZ87va51E9U= 2023-09-22T21:47:15.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VOr6jMIJE9f6F6XjqZ87va51E9U=', 'Content-Length': '0'} 2023-09-22T21:47:15.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d277359dd6f280a-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.474 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:47:15.474 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:47:15.474 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.474 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.474 DEBUG:boto:Method: HEAD 2023-09-22T21:47:15.474 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.474 DEBUG:boto:Data: 2023-09-22T21:47:15.474 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.475 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.475 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.475 DEBUG:boto:Params: {} 2023-09-22T21:47:15.475 DEBUG:boto:Token: None 2023-09-22T21:47:15.475 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.475 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QAjeY/k75QqT+L2krtC67PIwVDs= 2023-09-22T21:47:15.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QAjeY/k75QqT+L2krtC67PIwVDs=', 'Content-Length': '0'} 2023-09-22T21:47:15.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006466accc101c0632-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.478 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:47:15.478 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:47:15.478 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.478 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.479 DEBUG:boto:Method: HEAD 2023-09-22T21:47:15.479 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.479 DEBUG:boto:Data: 2023-09-22T21:47:15.479 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.479 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.479 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.479 DEBUG:boto:Params: {} 2023-09-22T21:47:15.479 DEBUG:boto:Token: None 2023-09-22T21:47:15.479 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.480 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QAjeY/k75QqT+L2krtC67PIwVDs= 2023-09-22T21:47:15.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QAjeY/k75QqT+L2krtC67PIwVDs=', 'Content-Length': '0'} 2023-09-22T21:47:15.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a0db064e938f828-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-22T21:47:15.482 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:47:15.482 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:47:15.483 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.483 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.483 DEBUG:boto:Method: GET 2023-09-22T21:47:15.483 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.483 DEBUG:boto:Data: 2023-09-22T21:47:15.483 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.483 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.483 DEBUG:boto:Params: {} 2023-09-22T21:47:15.483 DEBUG:boto:Token: None 2023-09-22T21:47:15.484 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.484 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VOr6jMIJE9f6F6XjqZ87va51E9U= 2023-09-22T21:47:15.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VOr6jMIJE9f6F6XjqZ87va51E9U=', 'Content-Length': '0'} 2023-09-22T21:47:15.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008341e8ae2b209e79-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.530 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:47:15.530 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:47:15.531 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.531 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.531 DEBUG:boto:Method: GET 2023-09-22T21:47:15.531 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.531 DEBUG:boto:Data: 2023-09-22T21:47:15.531 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.531 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.531 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.531 DEBUG:boto:Params: {} 2023-09-22T21:47:15.531 DEBUG:boto:Token: None 2023-09-22T21:47:15.532 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:47:15.532 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VOr6jMIJE9f6F6XjqZ87va51E9U= 2023-09-22T21:47:15.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VOr6jMIJE9f6F6XjqZ87va51E9U=', 'Content-Length': '0'} 2023-09-22T21:47:15.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9ab29faa1f913ba-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.535 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:15.535 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:15.535 DEBUG:boto:path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:47:15.535 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:47:15.535 DEBUG:boto:Method: GET 2023-09-22T21:47:15.535 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj92 2023-09-22T21:47:15.535 DEBUG:boto:Data: 2023-09-22T21:47:15.535 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.536 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.536 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.536 DEBUG:boto:Params: {} 2023-09-22T21:47:15.536 DEBUG:boto:Token: None 2023-09-22T21:47:15.536 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/ 2023-09-22T21:47:15.536 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI= 2023-09-22T21:47:15.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI=', 'Content-Length': '0'} 2023-09-22T21:47:15.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082dbcd78cdb06019-00650e0b63-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.582 DEBUG:boto:b'ffzdqu-59obj921000falseobj922023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:15.583 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:15.583 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:15.583 DEBUG:boto:path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:47:15.583 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:47:15.583 DEBUG:boto:Method: GET 2023-09-22T21:47:15.583 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj92 2023-09-22T21:47:15.583 DEBUG:boto:Data: 2023-09-22T21:47:15.583 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.584 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.584 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.584 DEBUG:boto:Params: {} 2023-09-22T21:47:15.584 DEBUG:boto:Token: None 2023-09-22T21:47:15.584 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/ 2023-09-22T21:47:15.584 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI= 2023-09-22T21:47:15.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI=', 'Content-Length': '0'} 2023-09-22T21:47:15.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007942a198a5f5c5da-00650e0b63-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.593 DEBUG:boto:b'ffzdqu-59obj921000falseobj922023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:15.594 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:47:15.594 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:47:15.594 DEBUG:boto:Method: HEAD 2023-09-22T21:47:15.594 DEBUG:boto:Path: /ffzdqu-59/obj92 2023-09-22T21:47:15.594 DEBUG:boto:Data: 2023-09-22T21:47:15.594 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.594 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.594 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.595 DEBUG:boto:Params: {} 2023-09-22T21:47:15.595 DEBUG:boto:Token: None 2023-09-22T21:47:15.595 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj92 2023-09-22T21:47:15.595 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SA0lsqHaGn91KyPZxoOWhZ9w3FA= 2023-09-22T21:47:15.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SA0lsqHaGn91KyPZxoOWhZ9w3FA=', 'Content-Length': '0'} 2023-09-22T21:47:15.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044ec041532513a5c-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.598 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:47:15.598 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:47:15.598 DEBUG:boto:Method: HEAD 2023-09-22T21:47:15.599 DEBUG:boto:Path: /ffzdqu-59/obj92 2023-09-22T21:47:15.599 DEBUG:boto:Data: 2023-09-22T21:47:15.599 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.599 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.599 DEBUG:boto:Params: {} 2023-09-22T21:47:15.599 DEBUG:boto:Token: None 2023-09-22T21:47:15.599 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj92 2023-09-22T21:47:15.599 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SA0lsqHaGn91KyPZxoOWhZ9w3FA= 2023-09-22T21:47:15.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SA0lsqHaGn91KyPZxoOWhZ9w3FA=', 'Content-Length': '0'} 2023-09-22T21:47:15.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adfea19e18515155-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-22T21:47:15.602 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:47:15.602 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:47:15.602 DEBUG:boto:Method: GET 2023-09-22T21:47:15.603 DEBUG:boto:Path: /ffzdqu-59/obj92 2023-09-22T21:47:15.603 DEBUG:boto:Data: 2023-09-22T21:47:15.603 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.603 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.603 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.603 DEBUG:boto:Params: {} 2023-09-22T21:47:15.603 DEBUG:boto:Token: None 2023-09-22T21:47:15.603 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj92 2023-09-22T21:47:15.603 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eWNMtVVw0WYOnt2IRcbT7TgQj8o= 2023-09-22T21:47:15.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eWNMtVVw0WYOnt2IRcbT7TgQj8o=', 'Content-Length': '0'} 2023-09-22T21:47:15.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097440a260adec7af-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.650 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:47:15.650 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:47:15.651 DEBUG:boto:Method: GET 2023-09-22T21:47:15.651 DEBUG:boto:Path: /ffzdqu-59/obj92 2023-09-22T21:47:15.651 DEBUG:boto:Data: 2023-09-22T21:47:15.651 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.651 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.651 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.651 DEBUG:boto:Params: {} 2023-09-22T21:47:15.651 DEBUG:boto:Token: None 2023-09-22T21:47:15.651 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj92 2023-09-22T21:47:15.652 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eWNMtVVw0WYOnt2IRcbT7TgQj8o= 2023-09-22T21:47:15.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eWNMtVVw0WYOnt2IRcbT7TgQj8o=', 'Content-Length': '0'} 2023-09-22T21:47:15.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000580a75a6cd6877-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.655 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:15.655 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:15.655 DEBUG:boto:path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:47:15.655 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:47:15.655 DEBUG:boto:Method: GET 2023-09-22T21:47:15.655 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj92 2023-09-22T21:47:15.655 DEBUG:boto:Data: 2023-09-22T21:47:15.655 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.655 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.656 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.656 DEBUG:boto:Params: {} 2023-09-22T21:47:15.656 DEBUG:boto:Token: None 2023-09-22T21:47:15.656 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/ 2023-09-22T21:47:15.656 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI= 2023-09-22T21:47:15.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI=', 'Content-Length': '0'} 2023-09-22T21:47:15.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e3e49bc9dbd684e-00650e0b63-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.662 DEBUG:boto:b'ffzdqu-59obj921000falseobj922023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:15.662 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:15.662 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:15.662 DEBUG:boto:path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:47:15.663 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:47:15.663 DEBUG:boto:Method: GET 2023-09-22T21:47:15.663 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj92 2023-09-22T21:47:15.663 DEBUG:boto:Data: 2023-09-22T21:47:15.663 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.663 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.663 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.663 DEBUG:boto:Params: {} 2023-09-22T21:47:15.663 DEBUG:boto:Token: None 2023-09-22T21:47:15.663 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/ 2023-09-22T21:47:15.664 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI= 2023-09-22T21:47:15.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI=', 'Content-Length': '0'} 2023-09-22T21:47:15.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd632a007d2c3bb0-00650e0b63-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.672 DEBUG:boto:b'ffzdqu-59obj921000falseobj922023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:15.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-22T21:47:15.672 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:47:15.672 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:47:15.672 DEBUG:boto:path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.672 DEBUG:boto:auth_path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.672 DEBUG:boto:Method: GET 2023-09-22T21:47:15.673 DEBUG:boto:Path: /ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.673 DEBUG:boto:Data: 2023-09-22T21:47:15.673 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.673 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.673 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.673 DEBUG:boto:Params: {} 2023-09-22T21:47:15.673 DEBUG:boto:Token: None 2023-09-22T21:47:15.673 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.673 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EI6dtl9UUWrasqjGmBLGIctOiKk= 2023-09-22T21:47:15.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EI6dtl9UUWrasqjGmBLGIctOiKk=', 'Content-Length': '0'} 2023-09-22T21:47:15.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019a4c25a3914d99d-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.718 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:47:15.718 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:47:15.719 DEBUG:boto:path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.719 DEBUG:boto:auth_path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.719 DEBUG:boto:Method: GET 2023-09-22T21:47:15.719 DEBUG:boto:Path: /ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.719 DEBUG:boto:Data: 2023-09-22T21:47:15.719 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.719 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.719 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.719 DEBUG:boto:Params: {} 2023-09-22T21:47:15.720 DEBUG:boto:Token: None 2023-09-22T21:47:15.720 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.720 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EI6dtl9UUWrasqjGmBLGIctOiKk= 2023-09-22T21:47:15.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EI6dtl9UUWrasqjGmBLGIctOiKk=', 'Content-Length': '0'} 2023-09-22T21:47:15.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000248494eb6687f5ae-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.723 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:47:15.724 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:47:15.724 DEBUG:boto:path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.724 DEBUG:boto:auth_path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.724 DEBUG:boto:Method: HEAD 2023-09-22T21:47:15.724 DEBUG:boto:Path: /ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.724 DEBUG:boto:Data: 2023-09-22T21:47:15.724 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.724 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.725 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.725 DEBUG:boto:Params: {} 2023-09-22T21:47:15.725 DEBUG:boto:Token: None 2023-09-22T21:47:15.725 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.725 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hH/eh6PfdhqzoG4id3Tvh6XnSDw= 2023-09-22T21:47:15.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hH/eh6PfdhqzoG4id3Tvh6XnSDw=', 'Content-Length': '0'} 2023-09-22T21:47:15.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073dbfb818f4bb704-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.728 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:47:15.728 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:47:15.728 DEBUG:boto:path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.728 DEBUG:boto:auth_path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.728 DEBUG:boto:Method: HEAD 2023-09-22T21:47:15.729 DEBUG:boto:Path: /ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.729 DEBUG:boto:Data: 2023-09-22T21:47:15.729 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.729 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.729 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.729 DEBUG:boto:Params: {} 2023-09-22T21:47:15.729 DEBUG:boto:Token: None 2023-09-22T21:47:15.729 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.729 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hH/eh6PfdhqzoG4id3Tvh6XnSDw= 2023-09-22T21:47:15.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hH/eh6PfdhqzoG4id3Tvh6XnSDw=', 'Content-Length': '0'} 2023-09-22T21:47:15.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000911efb53cd9302ae-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-22T21:47:15.733 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:47:15.733 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:47:15.733 DEBUG:boto:path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.733 DEBUG:boto:auth_path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.733 DEBUG:boto:Method: GET 2023-09-22T21:47:15.733 DEBUG:boto:Path: /ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.733 DEBUG:boto:Data: 2023-09-22T21:47:15.733 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.734 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.734 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.734 DEBUG:boto:Params: {} 2023-09-22T21:47:15.734 DEBUG:boto:Token: None 2023-09-22T21:47:15.734 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.734 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EI6dtl9UUWrasqjGmBLGIctOiKk= 2023-09-22T21:47:15.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EI6dtl9UUWrasqjGmBLGIctOiKk=', 'Content-Length': '0'} 2023-09-22T21:47:15.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005972eb5da9aa8585-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.778 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:47:15.778 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:47:15.779 DEBUG:boto:path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.779 DEBUG:boto:auth_path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.779 DEBUG:boto:Method: GET 2023-09-22T21:47:15.779 DEBUG:boto:Path: /ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.779 DEBUG:boto:Data: 2023-09-22T21:47:15.779 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.779 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.779 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.779 DEBUG:boto:Params: {} 2023-09-22T21:47:15.779 DEBUG:boto:Token: None 2023-09-22T21:47:15.780 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj93?versionId=null 2023-09-22T21:47:15.780 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EI6dtl9UUWrasqjGmBLGIctOiKk= 2023-09-22T21:47:15.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EI6dtl9UUWrasqjGmBLGIctOiKk=', 'Content-Length': '0'} 2023-09-22T21:47:15.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079549a575d71287d-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.783 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:15.783 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:15.783 DEBUG:boto:path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:47:15.783 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:47:15.783 DEBUG:boto:Method: GET 2023-09-22T21:47:15.783 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj93 2023-09-22T21:47:15.783 DEBUG:boto:Data: 2023-09-22T21:47:15.783 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.783 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.784 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.784 DEBUG:boto:Params: {} 2023-09-22T21:47:15.784 DEBUG:boto:Token: None 2023-09-22T21:47:15.784 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/ 2023-09-22T21:47:15.784 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI= 2023-09-22T21:47:15.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI=', 'Content-Length': '0'} 2023-09-22T21:47:15.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b55ca9ded7b8feae-00650e0b63-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.830 DEBUG:boto:b'ffzdqu-59obj931000falseobj932023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:15.831 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:15.831 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:15.831 DEBUG:boto:path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:47:15.831 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:47:15.831 DEBUG:boto:Method: GET 2023-09-22T21:47:15.832 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj93 2023-09-22T21:47:15.832 DEBUG:boto:Data: 2023-09-22T21:47:15.832 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.832 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.832 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.832 DEBUG:boto:Params: {} 2023-09-22T21:47:15.832 DEBUG:boto:Token: None 2023-09-22T21:47:15.832 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/ 2023-09-22T21:47:15.833 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI= 2023-09-22T21:47:15.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI=', 'Content-Length': '0'} 2023-09-22T21:47:15.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071c9595e6cf0edc6-00650e0b63-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.841 DEBUG:boto:b'ffzdqu-59obj931000falseobj932023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:15.842 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:47:15.842 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:47:15.842 DEBUG:boto:Method: HEAD 2023-09-22T21:47:15.842 DEBUG:boto:Path: /ffzdqu-59/obj93 2023-09-22T21:47:15.842 DEBUG:boto:Data: 2023-09-22T21:47:15.842 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.842 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.842 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.843 DEBUG:boto:Params: {} 2023-09-22T21:47:15.843 DEBUG:boto:Token: None 2023-09-22T21:47:15.843 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj93 2023-09-22T21:47:15.843 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qha1iOp94wsSEihko6zUgBPKxKc= 2023-09-22T21:47:15.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qha1iOp94wsSEihko6zUgBPKxKc=', 'Content-Length': '0'} 2023-09-22T21:47:15.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d918a4f6dd8d8392-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.846 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:47:15.846 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:47:15.846 DEBUG:boto:Method: HEAD 2023-09-22T21:47:15.846 DEBUG:boto:Path: /ffzdqu-59/obj93 2023-09-22T21:47:15.847 DEBUG:boto:Data: 2023-09-22T21:47:15.847 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.847 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.847 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.847 DEBUG:boto:Params: {} 2023-09-22T21:47:15.847 DEBUG:boto:Token: None 2023-09-22T21:47:15.847 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj93 2023-09-22T21:47:15.847 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qha1iOp94wsSEihko6zUgBPKxKc= 2023-09-22T21:47:15.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qha1iOp94wsSEihko6zUgBPKxKc=', 'Content-Length': '0'} 2023-09-22T21:47:15.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fb680898e892eaa-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-22T21:47:15.850 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:47:15.851 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:47:15.851 DEBUG:boto:Method: GET 2023-09-22T21:47:15.851 DEBUG:boto:Path: /ffzdqu-59/obj93 2023-09-22T21:47:15.851 DEBUG:boto:Data: 2023-09-22T21:47:15.851 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.851 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.851 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.851 DEBUG:boto:Params: {} 2023-09-22T21:47:15.851 DEBUG:boto:Token: None 2023-09-22T21:47:15.851 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj93 2023-09-22T21:47:15.852 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uw3EEP+0mcBNyjRVjlDH1KAxYgc= 2023-09-22T21:47:15.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uw3EEP+0mcBNyjRVjlDH1KAxYgc=', 'Content-Length': '0'} 2023-09-22T21:47:15.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f079d65fdc06b9d-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.894 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:47:15.894 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:47:15.895 DEBUG:boto:Method: GET 2023-09-22T21:47:15.895 DEBUG:boto:Path: /ffzdqu-59/obj93 2023-09-22T21:47:15.895 DEBUG:boto:Data: 2023-09-22T21:47:15.895 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.895 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.895 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.895 DEBUG:boto:Params: {} 2023-09-22T21:47:15.895 DEBUG:boto:Token: None 2023-09-22T21:47:15.896 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj93 2023-09-22T21:47:15.896 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uw3EEP+0mcBNyjRVjlDH1KAxYgc= 2023-09-22T21:47:15.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uw3EEP+0mcBNyjRVjlDH1KAxYgc=', 'Content-Length': '0'} 2023-09-22T21:47:15.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000100c2aeadd7d49e-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.899 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:15.899 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:15.899 DEBUG:boto:path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:47:15.900 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:47:15.900 DEBUG:boto:Method: GET 2023-09-22T21:47:15.900 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj93 2023-09-22T21:47:15.900 DEBUG:boto:Data: 2023-09-22T21:47:15.900 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.900 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.900 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.900 DEBUG:boto:Params: {} 2023-09-22T21:47:15.900 DEBUG:boto:Token: None 2023-09-22T21:47:15.901 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/ 2023-09-22T21:47:15.901 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI= 2023-09-22T21:47:15.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI=', 'Content-Length': '0'} 2023-09-22T21:47:15.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002db4a236a539796e-00650e0b63-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.905 DEBUG:boto:b'ffzdqu-59obj931000falseobj932023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:15.906 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:15.906 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:15.906 DEBUG:boto:path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:47:15.906 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:47:15.906 DEBUG:boto:Method: GET 2023-09-22T21:47:15.906 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj93 2023-09-22T21:47:15.906 DEBUG:boto:Data: 2023-09-22T21:47:15.906 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.907 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.907 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.907 DEBUG:boto:Params: {} 2023-09-22T21:47:15.907 DEBUG:boto:Token: None 2023-09-22T21:47:15.907 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/ 2023-09-22T21:47:15.907 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI= 2023-09-22T21:47:15.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cEMGE/0r8KZu6uuNshTt/3B76WI=', 'Content-Length': '0'} 2023-09-22T21:47:15.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056a4e99a6f36f5a1-00650e0b63-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.917 DEBUG:boto:b'ffzdqu-59obj931000falseobj932023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:15.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T21:47:15.917 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:47:15.918 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:47:15.918 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:15.918 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:15.918 DEBUG:boto:Method: GET 2023-09-22T21:47:15.918 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:15.918 DEBUG:boto:Data: 2023-09-22T21:47:15.918 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.918 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.918 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.918 DEBUG:boto:Params: {} 2023-09-22T21:47:15.919 DEBUG:boto:Token: None 2023-09-22T21:47:15.919 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:15.919 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AJr7zdsqYeQnUe2abkfrim3VGw8= 2023-09-22T21:47:15.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AJr7zdsqYeQnUe2abkfrim3VGw8=', 'Content-Length': '0'} 2023-09-22T21:47:15.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021c9790add2e027c-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.962 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:47:15.962 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:47:15.963 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:15.963 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:15.963 DEBUG:boto:Method: GET 2023-09-22T21:47:15.963 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:15.963 DEBUG:boto:Data: 2023-09-22T21:47:15.963 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.963 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.963 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.963 DEBUG:boto:Params: {} 2023-09-22T21:47:15.964 DEBUG:boto:Token: None 2023-09-22T21:47:15.964 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:15.964 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AJr7zdsqYeQnUe2abkfrim3VGw8= 2023-09-22T21:47:15.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AJr7zdsqYeQnUe2abkfrim3VGw8=', 'Content-Length': '0'} 2023-09-22T21:47:15.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ed66d12c3ccb22f-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.967 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:47:15.967 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:47:15.968 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:15.968 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:15.968 DEBUG:boto:Method: HEAD 2023-09-22T21:47:15.968 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:15.968 DEBUG:boto:Data: 2023-09-22T21:47:15.968 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.968 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.968 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.968 DEBUG:boto:Params: {} 2023-09-22T21:47:15.968 DEBUG:boto:Token: None 2023-09-22T21:47:15.969 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:15.969 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r0BfmMXP2S+xaV2wLAOa59UgKDo= 2023-09-22T21:47:15.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r0BfmMXP2S+xaV2wLAOa59UgKDo=', 'Content-Length': '0'} 2023-09-22T21:47:15.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec78c078a5691106-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.972 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:47:15.972 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:47:15.972 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:15.972 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:15.972 DEBUG:boto:Method: HEAD 2023-09-22T21:47:15.972 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:15.972 DEBUG:boto:Data: 2023-09-22T21:47:15.973 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.973 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:15.973 DEBUG:boto:Port: 8000 2023-09-22T21:47:15.973 DEBUG:boto:Params: {} 2023-09-22T21:47:15.973 DEBUG:boto:Token: None 2023-09-22T21:47:15.973 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:15.973 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r0BfmMXP2S+xaV2wLAOa59UgKDo= 2023-09-22T21:47:15.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r0BfmMXP2S+xaV2wLAOa59UgKDo=', 'Content-Length': '0'} 2023-09-22T21:47:15.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000111e9377184cd89b-00650e0b63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:15.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T21:47:15.977 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:47:15.977 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:47:15.977 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:15.977 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:15.977 DEBUG:boto:Method: GET 2023-09-22T21:47:15.978 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:15.978 DEBUG:boto:Data: 2023-09-22T21:47:15.978 DEBUG:boto:Headers: {} 2023-09-22T21:47:15.978 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:15.978 DEBUG:boto:Port: 8001 2023-09-22T21:47:15.978 DEBUG:boto:Params: {} 2023-09-22T21:47:15.978 DEBUG:boto:Token: None 2023-09-22T21:47:15.978 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:15 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:15.978 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AJr7zdsqYeQnUe2abkfrim3VGw8= 2023-09-22T21:47:15.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AJr7zdsqYeQnUe2abkfrim3VGw8=', 'Content-Length': '0'} 2023-09-22T21:47:15.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a178fe519747271-00650e0b63-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:16.026 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:47:16.026 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:47:16.027 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:16.027 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:16.027 DEBUG:boto:Method: GET 2023-09-22T21:47:16.027 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:16.027 DEBUG:boto:Data: 2023-09-22T21:47:16.027 DEBUG:boto:Headers: {} 2023-09-22T21:47:16.027 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:16.027 DEBUG:boto:Port: 8000 2023-09-22T21:47:16.027 DEBUG:boto:Params: {} 2023-09-22T21:47:16.028 DEBUG:boto:Token: None 2023-09-22T21:47:16.028 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:16 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:47:16.028 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ezKIhhO0EogEUEu2F05hKnBQdbc= 2023-09-22T21:47:16.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ezKIhhO0EogEUEu2F05hKnBQdbc=', 'Content-Length': '0'} 2023-09-22T21:47:16.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000258d60ebb3773063-00650e0b64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:16.031 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:16.032 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:16.032 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:47:16.032 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:47:16.032 DEBUG:boto:Method: GET 2023-09-22T21:47:16.032 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:47:16.032 DEBUG:boto:Data: 2023-09-22T21:47:16.032 DEBUG:boto:Headers: {} 2023-09-22T21:47:16.032 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:16.032 DEBUG:boto:Port: 8001 2023-09-22T21:47:16.033 DEBUG:boto:Params: {} 2023-09-22T21:47:16.033 DEBUG:boto:Token: None 2023-09-22T21:47:16.033 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:16 GMT /ffzdqu-59/ 2023-09-22T21:47:16.033 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NddQe1+d0thT351UPG3RJJtOr3M= 2023-09-22T21:47:16.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NddQe1+d0thT351UPG3RJJtOr3M=', 'Content-Length': '0'} 2023-09-22T21:47:16.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093237a382fce5ea7-00650e0b64-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:16.078 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:16.079 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:16.079 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:16.079 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:47:16.079 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:47:16.079 DEBUG:boto:Method: GET 2023-09-22T21:47:16.080 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:47:16.080 DEBUG:boto:Data: 2023-09-22T21:47:16.080 DEBUG:boto:Headers: {} 2023-09-22T21:47:16.080 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:16.080 DEBUG:boto:Port: 8000 2023-09-22T21:47:16.080 DEBUG:boto:Params: {} 2023-09-22T21:47:16.080 DEBUG:boto:Token: None 2023-09-22T21:47:16.080 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:16 GMT /ffzdqu-59/ 2023-09-22T21:47:16.080 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NddQe1+d0thT351UPG3RJJtOr3M= 2023-09-22T21:47:16.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NddQe1+d0thT351UPG3RJJtOr3M=', 'Content-Length': '0'} 2023-09-22T21:47:16.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb1ae07056a13f21-00650e0b64-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:16.090 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:16.090 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:47:16.090 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:47:16.091 DEBUG:boto:Method: HEAD 2023-09-22T21:47:16.091 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:47:16.091 DEBUG:boto:Data: 2023-09-22T21:47:16.091 DEBUG:boto:Headers: {} 2023-09-22T21:47:16.091 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:16.091 DEBUG:boto:Port: 8001 2023-09-22T21:47:16.091 DEBUG:boto:Params: {} 2023-09-22T21:47:16.091 DEBUG:boto:Token: None 2023-09-22T21:47:16.091 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:16 GMT /ffzdqu-59/obj95 2023-09-22T21:47:16.092 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4r1Cdt2jBVvlWnQT9iWZArSYJPY= 2023-09-22T21:47:16.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4r1Cdt2jBVvlWnQT9iWZArSYJPY=', 'Content-Length': '0'} 2023-09-22T21:47:16.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a814505bfb010830-00650e0b64-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:16.094 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:47:16.095 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:47:16.095 DEBUG:boto:Method: HEAD 2023-09-22T21:47:16.095 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:47:16.095 DEBUG:boto:Data: 2023-09-22T21:47:16.095 DEBUG:boto:Headers: {} 2023-09-22T21:47:16.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:16.095 DEBUG:boto:Port: 8000 2023-09-22T21:47:16.095 DEBUG:boto:Params: {} 2023-09-22T21:47:16.096 DEBUG:boto:Token: None 2023-09-22T21:47:16.096 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:16 GMT /ffzdqu-59/obj95 2023-09-22T21:47:16.096 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4r1Cdt2jBVvlWnQT9iWZArSYJPY= 2023-09-22T21:47:16.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4r1Cdt2jBVvlWnQT9iWZArSYJPY=', 'Content-Length': '0'} 2023-09-22T21:47:16.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceb593bf2b1ad6ba-00650e0b64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:16.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T21:47:16.099 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:47:16.099 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:47:16.099 DEBUG:boto:Method: GET 2023-09-22T21:47:16.099 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:47:16.099 DEBUG:boto:Data: 2023-09-22T21:47:16.099 DEBUG:boto:Headers: {} 2023-09-22T21:47:16.099 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:16.099 DEBUG:boto:Port: 8001 2023-09-22T21:47:16.100 DEBUG:boto:Params: {} 2023-09-22T21:47:16.100 DEBUG:boto:Token: None 2023-09-22T21:47:16.100 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:16 GMT /ffzdqu-59/obj95 2023-09-22T21:47:16.100 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eP+8DN6FZdRfLVkalp8I9uAViS4= 2023-09-22T21:47:16.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eP+8DN6FZdRfLVkalp8I9uAViS4=', 'Content-Length': '0'} 2023-09-22T21:47:16.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e00f0c92f7cc51ce-00650e0b64-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:16.150 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:47:16.150 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:47:16.151 DEBUG:boto:Method: GET 2023-09-22T21:47:16.151 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:47:16.151 DEBUG:boto:Data: 2023-09-22T21:47:16.151 DEBUG:boto:Headers: {} 2023-09-22T21:47:16.151 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:16.151 DEBUG:boto:Port: 8000 2023-09-22T21:47:16.151 DEBUG:boto:Params: {} 2023-09-22T21:47:16.151 DEBUG:boto:Token: None 2023-09-22T21:47:16.152 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:16 GMT /ffzdqu-59/obj95 2023-09-22T21:47:16.152 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eP+8DN6FZdRfLVkalp8I9uAViS4= 2023-09-22T21:47:16.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eP+8DN6FZdRfLVkalp8I9uAViS4=', 'Content-Length': '0'} 2023-09-22T21:47:16.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dce5262c9cc98314-00650e0b64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:16.157 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:16.158 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:16.158 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:47:16.158 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:47:16.158 DEBUG:boto:Method: GET 2023-09-22T21:47:16.158 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:47:16.158 DEBUG:boto:Data: 2023-09-22T21:47:16.158 DEBUG:boto:Headers: {} 2023-09-22T21:47:16.158 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:47:16.158 DEBUG:boto:Port: 8001 2023-09-22T21:47:16.159 DEBUG:boto:Params: {} 2023-09-22T21:47:16.159 DEBUG:boto:Token: None 2023-09-22T21:47:16.159 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:16 GMT /ffzdqu-59/ 2023-09-22T21:47:16.159 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NddQe1+d0thT351UPG3RJJtOr3M= 2023-09-22T21:47:16.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NddQe1+d0thT351UPG3RJJtOr3M=', 'Content-Length': '0'} 2023-09-22T21:47:16.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b21d22fe657186b-00650e0b64-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:16.164 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:16.164 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:16.164 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:16.165 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:47:16.165 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:47:16.165 DEBUG:boto:Method: GET 2023-09-22T21:47:16.165 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:47:16.165 DEBUG:boto:Data: 2023-09-22T21:47:16.165 DEBUG:boto:Headers: {} 2023-09-22T21:47:16.165 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:16.165 DEBUG:boto:Port: 8000 2023-09-22T21:47:16.165 DEBUG:boto:Params: {} 2023-09-22T21:47:16.165 DEBUG:boto:Token: None 2023-09-22T21:47:16.166 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:47:16 GMT /ffzdqu-59/ 2023-09-22T21:47:16.166 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NddQe1+d0thT351UPG3RJJtOr3M= 2023-09-22T21:47:16.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NddQe1+d0thT351UPG3RJJtOr3M=', 'Content-Length': '0'} 2023-09-22T21:47:16.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e10ba91104e64ae3-00650e0b64-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:16.174 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:47:16.175 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-59 zones={a2, a1} 2023-09-22T21:47:16.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:47:16.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:47:16.176 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:47:16.220 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:47:16.220 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:47:16.285 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:47:16.285 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-59 2023-09-22T21:47:16.305 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 30 2023-09-22T21:47:16.388 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:16.389 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:16.389 DEBUG:boto:Method: HEAD 2023-09-22T21:47:16.389 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:47:16.389 DEBUG:boto:Data: 2023-09-22T21:47:16.389 DEBUG:boto:Headers: {} 2023-09-22T21:47:16.389 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:16.389 DEBUG:boto:Port: 8000 2023-09-22T21:47:16.389 DEBUG:boto:Params: {} 2023-09-22T21:47:16.389 DEBUG:boto:Token: None 2023-09-22T21:47:16.390 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:16 GMT /ffzdqu-59/ 2023-09-22T21:47:16.390 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YYQvi+sPLfTtLx3vVhiIgf25itM= 2023-09-22T21:47:16.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YYQvi+sPLfTtLx3vVhiIgf25itM=', 'Content-Length': '0'} 2023-09-22T21:47:16.394 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087e8889ac720e31b-00650e0b64-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:16.395 DEBUG:boto:path=/ffzdqu-59/obj37 2023-09-22T21:47:16.395 DEBUG:boto:auth_path=/ffzdqu-59/obj37 2023-09-22T21:47:16.395 DEBUG:boto:Method: PUT 2023-09-22T21:47:16.395 DEBUG:boto:Path: /ffzdqu-59/obj37 2023-09-22T21:47:16.395 DEBUG:boto:Data: 2023-09-22T21:47:16.395 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:47:16.395 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:16.395 DEBUG:boto:Port: 8000 2023-09-22T21:47:16.396 DEBUG:boto:Params: {} 2023-09-22T21:47:16.396 DEBUG:boto:Token: None 2023-09-22T21:47:16.396 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:47:16 GMT /ffzdqu-59/obj37 2023-09-22T21:47:16.396 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tf8Uf5SnPtJHZ04kdkMwOkwGuMA= 2023-09-22T21:47:16.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:47:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tf8Uf5SnPtJHZ04kdkMwOkwGuMA='} 2023-09-22T21:47:16.403 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f8cb947417e9e2f-00650e0b64-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:16.403 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:16.403 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:16.403 DEBUG:boto:Method: HEAD 2023-09-22T21:47:16.403 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:47:16.403 DEBUG:boto:Data: 2023-09-22T21:47:16.404 DEBUG:boto:Headers: {} 2023-09-22T21:47:16.404 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:16.404 DEBUG:boto:Port: 8000 2023-09-22T21:47:16.404 DEBUG:boto:Params: {} 2023-09-22T21:47:16.404 DEBUG:boto:Token: None 2023-09-22T21:47:16.404 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:16 GMT /ffzdqu-59/ 2023-09-22T21:47:16.404 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YYQvi+sPLfTtLx3vVhiIgf25itM= 2023-09-22T21:47:16.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YYQvi+sPLfTtLx3vVhiIgf25itM=', 'Content-Length': '0'} 2023-09-22T21:47:16.409 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4aa0a38fc658a1a-00650e0b64-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:16.409 DEBUG:boto:path=/ffzdqu-59/obj74 2023-09-22T21:47:16.409 DEBUG:boto:auth_path=/ffzdqu-59/obj74 2023-09-22T21:47:16.409 DEBUG:boto:Method: PUT 2023-09-22T21:47:16.409 DEBUG:boto:Path: /ffzdqu-59/obj74 2023-09-22T21:47:16.409 DEBUG:boto:Data: 2023-09-22T21:47:16.409 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:47:16.409 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:16.410 DEBUG:boto:Port: 8000 2023-09-22T21:47:16.410 DEBUG:boto:Params: {} 2023-09-22T21:47:16.410 DEBUG:boto:Token: None 2023-09-22T21:47:16.410 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:47:16 GMT /ffzdqu-59/obj74 2023-09-22T21:47:16.410 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:s+5TRhKea268kfkyNSOc4u85bmk= 2023-09-22T21:47:16.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:47:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:s+5TRhKea268kfkyNSOc4u85bmk='} 2023-09-22T21:47:16.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001043df326ed5ae6e-00650e0b64-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:16.455 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:16.455 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:16.455 DEBUG:boto:Method: HEAD 2023-09-22T21:47:16.455 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:47:16.455 DEBUG:boto:Data: 2023-09-22T21:47:16.455 DEBUG:boto:Headers: {} 2023-09-22T21:47:16.455 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:16.455 DEBUG:boto:Port: 8000 2023-09-22T21:47:16.455 DEBUG:boto:Params: {} 2023-09-22T21:47:16.456 DEBUG:boto:Token: None 2023-09-22T21:47:16.456 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:16 GMT /ffzdqu-59/ 2023-09-22T21:47:16.456 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YYQvi+sPLfTtLx3vVhiIgf25itM= 2023-09-22T21:47:16.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YYQvi+sPLfTtLx3vVhiIgf25itM=', 'Content-Length': '0'} 2023-09-22T21:47:16.462 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6f63ad8234cf76b-00650e0b64-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:16.463 DEBUG:boto:path=/ffzdqu-59/obj111 2023-09-22T21:47:16.463 DEBUG:boto:auth_path=/ffzdqu-59/obj111 2023-09-22T21:47:16.463 DEBUG:boto:Method: PUT 2023-09-22T21:47:16.463 DEBUG:boto:Path: /ffzdqu-59/obj111 2023-09-22T21:47:16.463 DEBUG:boto:Data: 2023-09-22T21:47:16.463 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:47:16.463 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:16.463 DEBUG:boto:Port: 8000 2023-09-22T21:47:16.463 DEBUG:boto:Params: {} 2023-09-22T21:47:16.464 DEBUG:boto:Token: None 2023-09-22T21:47:16.464 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:47:16 GMT /ffzdqu-59/obj111 2023-09-22T21:47:16.464 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pKHUYrSlT1VZIIOQOQNFE0fYQow= 2023-09-22T21:47:16.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:47:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pKHUYrSlT1VZIIOQOQNFE0fYQow='} 2023-09-22T21:47:16.474 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000958964cc084debe5-00650e0b64-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:16.474 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:16.474 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:16.475 DEBUG:boto:Method: HEAD 2023-09-22T21:47:16.475 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:47:16.475 DEBUG:boto:Data: 2023-09-22T21:47:16.475 DEBUG:boto:Headers: {} 2023-09-22T21:47:16.475 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:16.475 DEBUG:boto:Port: 8000 2023-09-22T21:47:16.475 DEBUG:boto:Params: {} 2023-09-22T21:47:16.475 DEBUG:boto:Token: None 2023-09-22T21:47:16.475 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:16 GMT /ffzdqu-59/ 2023-09-22T21:47:16.476 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YYQvi+sPLfTtLx3vVhiIgf25itM= 2023-09-22T21:47:16.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YYQvi+sPLfTtLx3vVhiIgf25itM=', 'Content-Length': '0'} 2023-09-22T21:47:16.482 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000758e6befdc68c433-00650e0b64-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:16.482 DEBUG:boto:path=/ffzdqu-59/obj148 2023-09-22T21:47:16.482 DEBUG:boto:auth_path=/ffzdqu-59/obj148 2023-09-22T21:47:16.482 DEBUG:boto:Method: PUT 2023-09-22T21:47:16.482 DEBUG:boto:Path: /ffzdqu-59/obj148 2023-09-22T21:47:16.483 DEBUG:boto:Data: 2023-09-22T21:47:16.483 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:47:16.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:16.483 DEBUG:boto:Port: 8000 2023-09-22T21:47:16.483 DEBUG:boto:Params: {} 2023-09-22T21:47:16.483 DEBUG:boto:Token: None 2023-09-22T21:47:16.483 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:47:16 GMT /ffzdqu-59/obj148 2023-09-22T21:47:16.483 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YVLD5Y5e63+48FLz0HK4j+FMyJk= 2023-09-22T21:47:16.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:47:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YVLD5Y5e63+48FLz0HK4j+FMyJk='} 2023-09-22T21:47:16.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000064de8fa40bd8211d-00650e0b64-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:16.527 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:47:16.527 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:47:16.527 DEBUG:boto:Method: HEAD 2023-09-22T21:47:16.527 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:47:16.527 DEBUG:boto:Data: 2023-09-22T21:47:16.527 DEBUG:boto:Headers: {} 2023-09-22T21:47:16.527 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:16.527 DEBUG:boto:Port: 8000 2023-09-22T21:47:16.528 DEBUG:boto:Params: {} 2023-09-22T21:47:16.528 DEBUG:boto:Token: None 2023-09-22T21:47:16.528 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:47:16 GMT /ffzdqu-59/ 2023-09-22T21:47:16.528 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YYQvi+sPLfTtLx3vVhiIgf25itM= 2023-09-22T21:47:16.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:47:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YYQvi+sPLfTtLx3vVhiIgf25itM=', 'Content-Length': '0'} 2023-09-22T21:47:16.538 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012acf07b650d0b01-00650e0b64-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:16.539 DEBUG:boto:path=/ffzdqu-59/obj185 2023-09-22T21:47:16.539 DEBUG:boto:auth_path=/ffzdqu-59/obj185 2023-09-22T21:47:16.539 DEBUG:boto:Method: PUT 2023-09-22T21:47:16.539 DEBUG:boto:Path: /ffzdqu-59/obj185 2023-09-22T21:47:16.539 DEBUG:boto:Data: 2023-09-22T21:47:16.539 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:47:16.540 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:47:16.540 DEBUG:boto:Port: 8000 2023-09-22T21:47:16.540 DEBUG:boto:Params: {} 2023-09-22T21:47:16.540 DEBUG:boto:Token: None 2023-09-22T21:47:16.540 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:47:16 GMT /ffzdqu-59/obj185 2023-09-22T21:47:16.540 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KNmWlZ4oxuFw+q6fWstyqt6sD4s= 2023-09-22T21:47:16.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:47:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KNmWlZ4oxuFw+q6fWstyqt6sD4s='} 2023-09-22T21:47:16.553 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007019ee182c56d62a-00650e0b64-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:47:16.553 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-59 2023-09-22T21:47:16.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:47:16.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:47:16.553 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:47:16.602 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:47:16.602 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:47:16.648 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:47:16.646+0000 7fbb8874aa40 1 waiting to reach latest gen 6, on 5.. 2023-09-22T21:47:21.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:47:21.646+0000 7fbb8874aa40 1 waiting to reach latest gen 6, on 5.. 2023-09-22T21:47:26.650 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:47:26.645+0000 7fbb8874aa40 1 waiting to reach latest gen 6, on 5.. 2023-09-22T21:47:31.651 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:47:31.645+0000 7fbb8874aa40 1 waiting to reach latest gen 6, on 5.. 2023-09-22T21:47:36.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:47:36.653+0000 7fbb8874aa40 1 waiting for incremental sync to catch up: 2023-09-22T21:47:36.657 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , , , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:47:36.657 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , 00000000002.76.6, , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:47:41.661 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:47:41.657+0000 7fbb8874aa40 1 waiting for incremental sync to catch up: 2023-09-22T21:47:41.662 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , , , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:47:41.662 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , 00000000002.76.6, , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:47:46.668 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:47:46.669+0000 7fbb8874aa40 1 waiting for incremental sync to catch up: 2023-09-22T21:47:46.669 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , , , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:47:46.669 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , 00000000002.76.6, , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:47:51.673 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:47:51.673+0000 7fbb8874aa40 1 waiting for incremental sync to catch up: 2023-09-22T21:47:51.673 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , , , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:47:51.673 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , 00000000002.76.6, , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:47:56.678 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:47:56.677+0000 7fbb8874aa40 1 waiting for incremental sync to catch up: 2023-09-22T21:47:56.953 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , , , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:47:56.953 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , 00000000002.76.6, , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:01.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:48:01.685+0000 7fbb8874aa40 1 waiting for incremental sync to catch up: 2023-09-22T21:48:01.684 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , , , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:01.685 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , 00000000002.76.6, , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:06.689 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:48:06.689+0000 7fbb8874aa40 1 waiting for incremental sync to catch up: 2023-09-22T21:48:06.689 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , , , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:06.689 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , 00000000002.76.6, , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:11.694 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:48:11.693+0000 7fbb8874aa40 1 waiting for incremental sync to catch up: 2023-09-22T21:48:11.695 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , , , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:11.695 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , 00000000002.76.6, , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:16.700 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:48:16.697+0000 7fbb8874aa40 1 waiting for incremental sync to catch up: 2023-09-22T21:48:16.700 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , , , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:16.700 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , 00000000002.76.6, , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:21.705 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:48:21.705+0000 7fbb8874aa40 1 waiting for incremental sync to catch up: 2023-09-22T21:48:21.705 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , , , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:21.705 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , 00000000002.76.6, , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:26.710 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:48:26.709+0000 7fbb8874aa40 1 waiting for incremental sync to catch up: 2023-09-22T21:48:26.711 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , , , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:26.711 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , 00000000002.76.6, , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:31.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:48:31.713+0000 7fbb8874aa40 1 waiting for incremental sync to catch up: 2023-09-22T21:48:31.716 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , , , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:31.716 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , 00000000002.76.6, , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:36.720 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:48:36.721+0000 7fbb8874aa40 1 waiting for incremental sync to catch up: 2023-09-22T21:48:36.721 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , , , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:36.721 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , 00000000002.76.6, , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:41.725 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:48:41.725+0000 7fbb8874aa40 1 waiting for incremental sync to catch up: 2023-09-22T21:48:41.725 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , , , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:41.726 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , 00000000002.76.6, , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:46.730 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:48:46.728+0000 7fbb8874aa40 1 waiting for incremental sync to catch up: 2023-09-22T21:48:46.730 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , , , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:46.731 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , 00000000002.76.6, , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:51.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:48:51.736+0000 7fbb8874aa40 1 waiting for incremental sync to catch up: 2023-09-22T21:48:51.737 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , , , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:51.737 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , 00000000002.76.6, , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:56.741 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:48:56.740+0000 7fbb8874aa40 1 bucket sync caught up with source: 2023-09-22T21:48:56.742 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , 00000000002.76.6, , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:56.742 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , 00000000002.76.6, , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:48:56.742 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:48:56.740+0000 7fbb8874aa40 0 bucket checkpoint complete 2023-09-22T21:48:56.776 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-59 2023-09-22T21:48:56.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:48:56.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:48:56.777 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:48:56.825 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:48:56.826 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:48:56.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:48:56.868+0000 7f031c127a40 1 waiting for incremental sync to catch up: 2023-09-22T21:48:56.869 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.318.6, 00000000002.265.6, 00000000002.161.6, 00000000005.295.6, 00000000004.417.6, 00000000001.415.6, 00000000001.228.6, 00000000004.158.6, 00000000006.177.6, 00000000006.329.6] 2023-09-22T21:48:56.869 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.318.6, 00000000002.265.6, 00000000002.161.6, 00000000005.295.6, 00000000004.417.6, 00000000001.415.6, 00000000001.228.6, 00000000004.158.6, 00000000007.181.6, 00000000006.329.6] 2023-09-22T21:49:01.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:01.868+0000 7f031c127a40 1 waiting for incremental sync to catch up: 2023-09-22T21:49:01.871 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.318.6, 00000000002.265.6, 00000000002.161.6, 00000000005.295.6, 00000000004.417.6, 00000000001.415.6, 00000000001.228.6, 00000000004.158.6, 00000000006.177.6, 00000000006.329.6] 2023-09-22T21:49:01.871 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.318.6, 00000000002.265.6, 00000000002.161.6, 00000000005.295.6, 00000000004.417.6, 00000000001.415.6, 00000000001.228.6, 00000000004.158.6, 00000000007.181.6, 00000000006.329.6] 2023-09-22T21:49:06.682 INFO:tasks.rgw.c1.client.0.smithi160.stdout:2023-09-22T21:49:06.680+0000 7fa9c1675640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:49:06.873 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:06.872+0000 7f031c127a40 1 bucket sync caught up with source: 2023-09-22T21:49:06.873 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.318.6, 00000000002.265.6, 00000000002.161.6, 00000000005.295.6, 00000000004.417.6, 00000000001.415.6, 00000000001.228.6, 00000000004.158.6, 00000000007.181.6, 00000000006.329.6] 2023-09-22T21:49:06.874 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.318.6, 00000000002.265.6, 00000000002.161.6, 00000000005.295.6, 00000000004.417.6, 00000000001.415.6, 00000000001.228.6, 00000000004.158.6, 00000000007.181.6, 00000000006.329.6] 2023-09-22T21:49:06.874 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:06.872+0000 7f031c127a40 0 bucket checkpoint complete 2023-09-22T21:49:06.903 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-59 zones={a2, a1} 2023-09-22T21:49:06.903 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:06.903 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:06.903 DEBUG:boto:Method: HEAD 2023-09-22T21:49:06.903 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:49:06.904 DEBUG:boto:Data: 2023-09-22T21:49:06.904 DEBUG:boto:Headers: {} 2023-09-22T21:49:06.904 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:06.904 DEBUG:boto:Port: 8001 2023-09-22T21:49:06.904 DEBUG:boto:Params: {} 2023-09-22T21:49:06.904 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:49:06.904 DEBUG:boto:Token: None 2023-09-22T21:49:06.905 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:06 GMT /ffzdqu-59/ 2023-09-22T21:49:06.905 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:j7k9CsSFcICsih/lWvX7accCn14= 2023-09-22T21:49:06.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:j7k9CsSFcICsih/lWvX7accCn14=', 'Content-Length': '0'} 2023-09-22T21:49:06.910 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a21fd4c52717036e-00650e0bd2-1b9c-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:49:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:06.910 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:06.911 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:06.911 DEBUG:boto:Method: HEAD 2023-09-22T21:49:06.911 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:49:06.911 DEBUG:boto:Data: 2023-09-22T21:49:06.911 DEBUG:boto:Headers: {} 2023-09-22T21:49:06.911 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:06.911 DEBUG:boto:Port: 8000 2023-09-22T21:49:06.911 DEBUG:boto:Params: {} 2023-09-22T21:49:06.912 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:49:06.912 DEBUG:boto:Token: None 2023-09-22T21:49:06.912 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:06 GMT /ffzdqu-59/ 2023-09-22T21:49:06.912 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:j7k9CsSFcICsih/lWvX7accCn14= 2023-09-22T21:49:06.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:j7k9CsSFcICsih/lWvX7accCn14=', 'Content-Length': '0'} 2023-09-22T21:49:06.918 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005725bab97f76461b-00650e0bd2-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:49:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:06.919 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:49:06.919 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:06.919 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:06.919 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:06.919 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:06.919 DEBUG:boto:Method: GET 2023-09-22T21:49:06.919 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:06.920 DEBUG:boto:Data: 2023-09-22T21:49:06.920 DEBUG:boto:Headers: {} 2023-09-22T21:49:06.920 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:06.920 DEBUG:boto:Port: 8001 2023-09-22T21:49:06.920 DEBUG:boto:Params: {} 2023-09-22T21:49:06.920 DEBUG:boto:Token: None 2023-09-22T21:49:06.920 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:06 GMT /ffzdqu-59/?versions 2023-09-22T21:49:06.921 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Hl5Aa/MQUrRFlqjp+3uyHeogFUY= 2023-09-22T21:49:06.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Hl5Aa/MQUrRFlqjp+3uyHeogFUY=', 'Content-Length': '0'} 2023-09-22T21:49:06.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098e3c5333fc2c3c1-00650e0bd2-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:06.925 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-22T21:47:16.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-22T21:47:16.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-22T21:47:16.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-22T21:47:16.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:06.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T21:49:06.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-09-22T21:49:06.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-22T21:49:06.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-22T21:49:06.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-09-22T21:49:06.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-22T21:49:06.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-09-22T21:49:06.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-09-22T21:49:06.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-22T21:49:06.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-09-22T21:49:06.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-22T21:49:06.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T21:49:06.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-22T21:49:06.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-22T21:49:06.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-09-22T21:49:06.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T21:49:06.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-22T21:49:06.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-09-22T21:49:06.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-22T21:49:06.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T21:49:06.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-22T21:49:06.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-22T21:49:06.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T21:49:06.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-22T21:49:06.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-22T21:49:06.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-09-22T21:49:06.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-22T21:49:06.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-22T21:49:06.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-09-22T21:49:06.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-22T21:49:06.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-22T21:49:06.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-22T21:49:06.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-22T21:49:06.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-09-22T21:49:06.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-22T21:49:06.931 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:49:06.931 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:06.931 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:06.931 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:06.932 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:06.932 DEBUG:boto:Method: GET 2023-09-22T21:49:06.932 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:06.932 DEBUG:boto:Data: 2023-09-22T21:49:06.932 DEBUG:boto:Headers: {} 2023-09-22T21:49:06.932 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:06.932 DEBUG:boto:Port: 8000 2023-09-22T21:49:06.932 DEBUG:boto:Params: {} 2023-09-22T21:49:06.933 DEBUG:boto:Token: None 2023-09-22T21:49:06.933 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:06 GMT /ffzdqu-59/?versions 2023-09-22T21:49:06.933 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Hl5Aa/MQUrRFlqjp+3uyHeogFUY= 2023-09-22T21:49:06.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Hl5Aa/MQUrRFlqjp+3uyHeogFUY=', 'Content-Length': '0'} 2023-09-22T21:49:06.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b49efb8590accb8c-00650e0bd2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:06.941 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-22T21:47:16.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-22T21:47:16.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-22T21:47:16.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-22T21:47:16.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:06.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T21:49:06.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-09-22T21:49:06.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-22T21:49:06.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-22T21:49:06.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-09-22T21:49:06.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-22T21:49:06.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-09-22T21:49:06.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-09-22T21:49:06.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-22T21:49:06.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-09-22T21:49:06.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-22T21:49:06.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T21:49:06.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-22T21:49:06.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-22T21:49:06.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-09-22T21:49:06.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T21:49:06.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-22T21:49:06.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-09-22T21:49:06.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-22T21:49:06.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T21:49:06.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-22T21:49:06.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-22T21:49:06.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T21:49:06.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-22T21:49:06.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-22T21:49:06.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-09-22T21:49:06.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-22T21:49:06.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-22T21:49:06.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-09-22T21:49:06.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-22T21:49:06.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-22T21:49:06.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-22T21:49:06.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-22T21:49:06.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-09-22T21:49:06.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-22T21:49:06.947 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:06.947 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:06.947 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:06.947 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:06.947 DEBUG:boto:Method: GET 2023-09-22T21:49:06.948 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:06.948 DEBUG:boto:Data: 2023-09-22T21:49:06.948 DEBUG:boto:Headers: {} 2023-09-22T21:49:06.948 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:06.948 DEBUG:boto:Port: 8001 2023-09-22T21:49:06.948 DEBUG:boto:Params: {} 2023-09-22T21:49:06.948 DEBUG:boto:Token: None 2023-09-22T21:49:06.948 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:06 GMT /ffzdqu-59/?versions 2023-09-22T21:49:06.949 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Hl5Aa/MQUrRFlqjp+3uyHeogFUY= 2023-09-22T21:49:06.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Hl5Aa/MQUrRFlqjp+3uyHeogFUY=', 'Content-Length': '0'} 2023-09-22T21:49:06.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056387f0bd5593c81-00650e0bd2-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:06.954 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-22T21:47:16.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-22T21:47:16.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-22T21:47:16.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-22T21:47:16.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:06.956 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:06.956 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:06.956 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:06.956 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:06.956 DEBUG:boto:Method: GET 2023-09-22T21:49:06.956 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:06.956 DEBUG:boto:Data: 2023-09-22T21:49:06.957 DEBUG:boto:Headers: {} 2023-09-22T21:49:06.957 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:06.957 DEBUG:boto:Port: 8000 2023-09-22T21:49:06.957 DEBUG:boto:Params: {} 2023-09-22T21:49:06.957 DEBUG:boto:Token: None 2023-09-22T21:49:06.957 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:06 GMT /ffzdqu-59/?versions 2023-09-22T21:49:06.957 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Hl5Aa/MQUrRFlqjp+3uyHeogFUY= 2023-09-22T21:49:06.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Hl5Aa/MQUrRFlqjp+3uyHeogFUY=', 'Content-Length': '0'} 2023-09-22T21:49:06.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000008d48e3f523be79-00650e0bd2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:06.966 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-22T21:47:16.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-22T21:47:16.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-22T21:47:16.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-22T21:47:16.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:06.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:49:06.968 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:49:06.968 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:49:06.968 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:06.968 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:06.968 DEBUG:boto:Method: GET 2023-09-22T21:49:06.968 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:06.969 DEBUG:boto:Data: 2023-09-22T21:49:06.969 DEBUG:boto:Headers: {} 2023-09-22T21:49:06.969 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:06.969 DEBUG:boto:Port: 8001 2023-09-22T21:49:06.969 DEBUG:boto:Params: {} 2023-09-22T21:49:06.969 DEBUG:boto:Token: None 2023-09-22T21:49:06.969 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:06 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:06.970 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cYDqP1i/wwh4Z3F7sqG4y+USUkU= 2023-09-22T21:49:06.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cYDqP1i/wwh4Z3F7sqG4y+USUkU=', 'Content-Length': '0'} 2023-09-22T21:49:06.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fb081c49b6dc483-00650e0bd2-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:06.973 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:49:06.973 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:49:06.973 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:06.973 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:06.973 DEBUG:boto:Method: GET 2023-09-22T21:49:06.973 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:06.973 DEBUG:boto:Data: 2023-09-22T21:49:06.974 DEBUG:boto:Headers: {} 2023-09-22T21:49:06.974 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:06.974 DEBUG:boto:Port: 8000 2023-09-22T21:49:06.974 DEBUG:boto:Params: {} 2023-09-22T21:49:06.974 DEBUG:boto:Token: None 2023-09-22T21:49:06.974 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:06 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:06.974 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cYDqP1i/wwh4Z3F7sqG4y+USUkU= 2023-09-22T21:49:06.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:06 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cYDqP1i/wwh4Z3F7sqG4y+USUkU=', 'Content-Length': '0'} 2023-09-22T21:49:06.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ae19061f216a205-00650e0bd2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.022 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:49:07.023 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:49:07.023 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:07.023 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:07.023 DEBUG:boto:Method: HEAD 2023-09-22T21:49:07.023 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:07.023 DEBUG:boto:Data: 2023-09-22T21:49:07.023 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.024 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.024 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.024 DEBUG:boto:Params: {} 2023-09-22T21:49:07.024 DEBUG:boto:Token: None 2023-09-22T21:49:07.024 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:07.024 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3QPXBXUr3TP2vmkGbtsQNnC7usk= 2023-09-22T21:49:07.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3QPXBXUr3TP2vmkGbtsQNnC7usk=', 'Content-Length': '0'} 2023-09-22T21:49:07.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002beb4d619a9df307-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.027 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:49:07.028 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:49:07.028 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:07.028 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:07.028 DEBUG:boto:Method: HEAD 2023-09-22T21:49:07.028 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:07.028 DEBUG:boto:Data: 2023-09-22T21:49:07.028 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.028 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.029 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.029 DEBUG:boto:Params: {} 2023-09-22T21:49:07.029 DEBUG:boto:Token: None 2023-09-22T21:49:07.029 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:07.029 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3QPXBXUr3TP2vmkGbtsQNnC7usk= 2023-09-22T21:49:07.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3QPXBXUr3TP2vmkGbtsQNnC7usk=', 'Content-Length': '0'} 2023-09-22T21:49:07.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe89e7c7a57c2f0a-00650e0bd3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:49:07.032 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:49:07.033 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:49:07.033 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:07.033 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:07.033 DEBUG:boto:Method: GET 2023-09-22T21:49:07.033 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:07.033 DEBUG:boto:Data: 2023-09-22T21:49:07.033 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.033 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.033 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.034 DEBUG:boto:Params: {} 2023-09-22T21:49:07.034 DEBUG:boto:Token: None 2023-09-22T21:49:07.034 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:07.034 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6KskhJ7lhjjoak87QvQnP01+Q0A= 2023-09-22T21:49:07.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6KskhJ7lhjjoak87QvQnP01+Q0A=', 'Content-Length': '0'} 2023-09-22T21:49:07.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005389a24b93d6e89b-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.082 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:49:07.083 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:49:07.083 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:07.083 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:07.083 DEBUG:boto:Method: GET 2023-09-22T21:49:07.083 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:07.083 DEBUG:boto:Data: 2023-09-22T21:49:07.083 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.083 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.084 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.084 DEBUG:boto:Params: {} 2023-09-22T21:49:07.084 DEBUG:boto:Token: None 2023-09-22T21:49:07.084 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:07.084 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6KskhJ7lhjjoak87QvQnP01+Q0A= 2023-09-22T21:49:07.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6KskhJ7lhjjoak87QvQnP01+Q0A=', 'Content-Length': '0'} 2023-09-22T21:49:07.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044a929cf694c4fdb-00650e0bd3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.087 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:07.088 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:07.088 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:49:07.088 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:49:07.088 DEBUG:boto:Method: GET 2023-09-22T21:49:07.088 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:49:07.088 DEBUG:boto:Data: 2023-09-22T21:49:07.088 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.088 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.089 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.089 DEBUG:boto:Params: {} 2023-09-22T21:49:07.089 DEBUG:boto:Token: None 2023-09-22T21:49:07.089 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/ 2023-09-22T21:49:07.089 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs= 2023-09-22T21:49:07.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs=', 'Content-Length': '0'} 2023-09-22T21:49:07.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004564e108ae51b3bb-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.095 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:07.096 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:07.096 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:07.096 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:49:07.097 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:49:07.097 DEBUG:boto:Method: GET 2023-09-22T21:49:07.097 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:49:07.097 DEBUG:boto:Data: 2023-09-22T21:49:07.097 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.098 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.098 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.098 DEBUG:boto:Params: {} 2023-09-22T21:49:07.098 DEBUG:boto:Token: None 2023-09-22T21:49:07.098 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/ 2023-09-22T21:49:07.099 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs= 2023-09-22T21:49:07.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs=', 'Content-Length': '0'} 2023-09-22T21:49:07.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f8ac10ebee6edc0-00650e0bd3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.109 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:07.110 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:49:07.110 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:49:07.110 DEBUG:boto:Method: HEAD 2023-09-22T21:49:07.110 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:49:07.110 DEBUG:boto:Data: 2023-09-22T21:49:07.110 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.110 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.111 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.111 DEBUG:boto:Params: {} 2023-09-22T21:49:07.111 DEBUG:boto:Token: None 2023-09-22T21:49:07.111 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj11 2023-09-22T21:49:07.111 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:arV2MeQb+/QW+sqim1sL5ovQBQk= 2023-09-22T21:49:07.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:arV2MeQb+/QW+sqim1sL5ovQBQk=', 'Content-Length': '0'} 2023-09-22T21:49:07.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000331f4232c5e9f37a-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.114 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:49:07.114 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:49:07.114 DEBUG:boto:Method: HEAD 2023-09-22T21:49:07.114 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:49:07.115 DEBUG:boto:Data: 2023-09-22T21:49:07.115 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.115 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.115 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.115 DEBUG:boto:Params: {} 2023-09-22T21:49:07.115 DEBUG:boto:Token: None 2023-09-22T21:49:07.115 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj11 2023-09-22T21:49:07.115 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:arV2MeQb+/QW+sqim1sL5ovQBQk= 2023-09-22T21:49:07.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:arV2MeQb+/QW+sqim1sL5ovQBQk=', 'Content-Length': '0'} 2023-09-22T21:49:07.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032e7cff5566d74fa-00650e0bd3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:49:07.119 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:49:07.119 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:49:07.119 DEBUG:boto:Method: GET 2023-09-22T21:49:07.119 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:49:07.119 DEBUG:boto:Data: 2023-09-22T21:49:07.119 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.119 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.120 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.120 DEBUG:boto:Params: {} 2023-09-22T21:49:07.120 DEBUG:boto:Token: None 2023-09-22T21:49:07.120 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj11 2023-09-22T21:49:07.120 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vuau8zktccNBEECKxPSbxGcHgKQ= 2023-09-22T21:49:07.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vuau8zktccNBEECKxPSbxGcHgKQ=', 'Content-Length': '0'} 2023-09-22T21:49:07.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000158048f9dc6d5df8-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.170 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:49:07.171 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:49:07.171 DEBUG:boto:Method: GET 2023-09-22T21:49:07.171 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:49:07.171 DEBUG:boto:Data: 2023-09-22T21:49:07.171 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.171 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.171 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.171 DEBUG:boto:Params: {} 2023-09-22T21:49:07.172 DEBUG:boto:Token: None 2023-09-22T21:49:07.172 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj11 2023-09-22T21:49:07.172 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vuau8zktccNBEECKxPSbxGcHgKQ= 2023-09-22T21:49:07.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vuau8zktccNBEECKxPSbxGcHgKQ=', 'Content-Length': '0'} 2023-09-22T21:49:07.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b807b6642492f082-00650e0bd3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.175 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:07.175 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:07.176 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:49:07.176 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:49:07.176 DEBUG:boto:Method: GET 2023-09-22T21:49:07.176 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:49:07.176 DEBUG:boto:Data: 2023-09-22T21:49:07.176 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.176 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.176 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.177 DEBUG:boto:Params: {} 2023-09-22T21:49:07.177 DEBUG:boto:Token: None 2023-09-22T21:49:07.177 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/ 2023-09-22T21:49:07.177 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs= 2023-09-22T21:49:07.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs=', 'Content-Length': '0'} 2023-09-22T21:49:07.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d459564518fba93f-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.182 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:07.182 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:07.183 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:07.183 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:49:07.183 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:49:07.183 DEBUG:boto:Method: GET 2023-09-22T21:49:07.183 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:49:07.183 DEBUG:boto:Data: 2023-09-22T21:49:07.183 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.183 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.184 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.184 DEBUG:boto:Params: {} 2023-09-22T21:49:07.184 DEBUG:boto:Token: None 2023-09-22T21:49:07.184 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/ 2023-09-22T21:49:07.184 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs= 2023-09-22T21:49:07.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs=', 'Content-Length': '0'} 2023-09-22T21:49:07.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cd5087018e6c3b7-00650e0bd3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.194 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:07.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-09-22T21:49:07.195 DEBUG:boto:path=/ffzdqu-59/obj111 2023-09-22T21:49:07.195 DEBUG:boto:auth_path=/ffzdqu-59/obj111 2023-09-22T21:49:07.195 DEBUG:boto:path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.195 DEBUG:boto:auth_path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.195 DEBUG:boto:Method: GET 2023-09-22T21:49:07.195 DEBUG:boto:Path: /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.195 DEBUG:boto:Data: 2023-09-22T21:49:07.196 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.196 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.196 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.196 DEBUG:boto:Params: {} 2023-09-22T21:49:07.196 DEBUG:boto:Token: None 2023-09-22T21:49:07.196 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.196 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GGiFXQ6oxU7c/hehCKAcb2yCu34= 2023-09-22T21:49:07.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GGiFXQ6oxU7c/hehCKAcb2yCu34=', 'Content-Length': '0'} 2023-09-22T21:49:07.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013440ee9e54e630d-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.242 DEBUG:boto:path=/ffzdqu-59/obj111 2023-09-22T21:49:07.243 DEBUG:boto:auth_path=/ffzdqu-59/obj111 2023-09-22T21:49:07.243 DEBUG:boto:path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.243 DEBUG:boto:auth_path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.243 DEBUG:boto:Method: GET 2023-09-22T21:49:07.243 DEBUG:boto:Path: /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.243 DEBUG:boto:Data: 2023-09-22T21:49:07.243 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.243 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.244 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.244 DEBUG:boto:Params: {} 2023-09-22T21:49:07.244 DEBUG:boto:Token: None 2023-09-22T21:49:07.244 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.244 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GGiFXQ6oxU7c/hehCKAcb2yCu34= 2023-09-22T21:49:07.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GGiFXQ6oxU7c/hehCKAcb2yCu34=', 'Content-Length': '0'} 2023-09-22T21:49:07.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085c4b6635433b65e-00650e0bd3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.247 DEBUG:boto:path=/ffzdqu-59/obj111 2023-09-22T21:49:07.248 DEBUG:boto:auth_path=/ffzdqu-59/obj111 2023-09-22T21:49:07.248 DEBUG:boto:path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.248 DEBUG:boto:auth_path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.248 DEBUG:boto:Method: HEAD 2023-09-22T21:49:07.248 DEBUG:boto:Path: /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.248 DEBUG:boto:Data: 2023-09-22T21:49:07.248 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.248 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.249 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.249 DEBUG:boto:Params: {} 2023-09-22T21:49:07.249 DEBUG:boto:Token: None 2023-09-22T21:49:07.249 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.249 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YAP1p4RoX9dphUGBP2ZOXK7ykT8= 2023-09-22T21:49:07.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YAP1p4RoX9dphUGBP2ZOXK7ykT8=', 'Content-Length': '0'} 2023-09-22T21:49:07.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3a8a571bb6bee6d-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.253 DEBUG:boto:path=/ffzdqu-59/obj111 2023-09-22T21:49:07.253 DEBUG:boto:auth_path=/ffzdqu-59/obj111 2023-09-22T21:49:07.253 DEBUG:boto:path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.253 DEBUG:boto:auth_path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.253 DEBUG:boto:Method: HEAD 2023-09-22T21:49:07.254 DEBUG:boto:Path: /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.254 DEBUG:boto:Data: 2023-09-22T21:49:07.254 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.254 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.254 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.255 DEBUG:boto:Params: {} 2023-09-22T21:49:07.255 DEBUG:boto:Token: None 2023-09-22T21:49:07.255 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.255 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YAP1p4RoX9dphUGBP2ZOXK7ykT8= 2023-09-22T21:49:07.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YAP1p4RoX9dphUGBP2ZOXK7ykT8=', 'Content-Length': '0'} 2023-09-22T21:49:07.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090e0aa4a5817b92f-00650e0bd3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-09-22T21:49:07.259 DEBUG:boto:path=/ffzdqu-59/obj111 2023-09-22T21:49:07.260 DEBUG:boto:auth_path=/ffzdqu-59/obj111 2023-09-22T21:49:07.260 DEBUG:boto:path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.260 DEBUG:boto:auth_path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.260 DEBUG:boto:Method: GET 2023-09-22T21:49:07.260 DEBUG:boto:Path: /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.261 DEBUG:boto:Data: 2023-09-22T21:49:07.261 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.261 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.261 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.261 DEBUG:boto:Params: {} 2023-09-22T21:49:07.262 DEBUG:boto:Token: None 2023-09-22T21:49:07.262 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.262 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GGiFXQ6oxU7c/hehCKAcb2yCu34= 2023-09-22T21:49:07.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GGiFXQ6oxU7c/hehCKAcb2yCu34=', 'Content-Length': '0'} 2023-09-22T21:49:07.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009df7a66663befab4-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.306 DEBUG:boto:path=/ffzdqu-59/obj111 2023-09-22T21:49:07.306 DEBUG:boto:auth_path=/ffzdqu-59/obj111 2023-09-22T21:49:07.307 DEBUG:boto:path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.307 DEBUG:boto:auth_path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.307 DEBUG:boto:Method: GET 2023-09-22T21:49:07.307 DEBUG:boto:Path: /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.307 DEBUG:boto:Data: 2023-09-22T21:49:07.307 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.307 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.307 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.307 DEBUG:boto:Params: {} 2023-09-22T21:49:07.308 DEBUG:boto:Token: None 2023-09-22T21:49:07.308 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:07.308 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GGiFXQ6oxU7c/hehCKAcb2yCu34= 2023-09-22T21:49:07.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GGiFXQ6oxU7c/hehCKAcb2yCu34=', 'Content-Length': '0'} 2023-09-22T21:49:07.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090b79e06caf3a302-00650e0bd3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.311 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:07.311 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:07.311 DEBUG:boto:path=/ffzdqu-59/?prefix=obj111 2023-09-22T21:49:07.311 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj111 2023-09-22T21:49:07.311 DEBUG:boto:Method: GET 2023-09-22T21:49:07.311 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj111 2023-09-22T21:49:07.312 DEBUG:boto:Data: 2023-09-22T21:49:07.312 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.312 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.312 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.312 DEBUG:boto:Params: {} 2023-09-22T21:49:07.312 DEBUG:boto:Token: None 2023-09-22T21:49:07.312 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/ 2023-09-22T21:49:07.312 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs= 2023-09-22T21:49:07.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs=', 'Content-Length': '0'} 2023-09-22T21:49:07.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003532f06671beaafe-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.362 DEBUG:boto:b'ffzdqu-59obj1111000falseobj1112023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:07.363 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:07.363 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:07.363 DEBUG:boto:path=/ffzdqu-59/?prefix=obj111 2023-09-22T21:49:07.363 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj111 2023-09-22T21:49:07.363 DEBUG:boto:Method: GET 2023-09-22T21:49:07.363 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj111 2023-09-22T21:49:07.364 DEBUG:boto:Data: 2023-09-22T21:49:07.364 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.364 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.364 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.364 DEBUG:boto:Params: {} 2023-09-22T21:49:07.364 DEBUG:boto:Token: None 2023-09-22T21:49:07.364 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/ 2023-09-22T21:49:07.364 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs= 2023-09-22T21:49:07.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs=', 'Content-Length': '0'} 2023-09-22T21:49:07.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063cb73c6750fb97c-00650e0bd3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.374 DEBUG:boto:b'ffzdqu-59obj1111000falseobj1112023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:07.375 DEBUG:boto:path=/ffzdqu-59/obj111 2023-09-22T21:49:07.375 DEBUG:boto:auth_path=/ffzdqu-59/obj111 2023-09-22T21:49:07.375 DEBUG:boto:Method: HEAD 2023-09-22T21:49:07.375 DEBUG:boto:Path: /ffzdqu-59/obj111 2023-09-22T21:49:07.375 DEBUG:boto:Data: 2023-09-22T21:49:07.376 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.376 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.376 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.376 DEBUG:boto:Params: {} 2023-09-22T21:49:07.376 DEBUG:boto:Token: None 2023-09-22T21:49:07.376 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj111 2023-09-22T21:49:07.377 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ziaGrY+Hf67ngDCsDDTco76Dj5w= 2023-09-22T21:49:07.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ziaGrY+Hf67ngDCsDDTco76Dj5w=', 'Content-Length': '0'} 2023-09-22T21:49:07.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078dcde33f58e9b3f-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.380 DEBUG:boto:path=/ffzdqu-59/obj111 2023-09-22T21:49:07.380 DEBUG:boto:auth_path=/ffzdqu-59/obj111 2023-09-22T21:49:07.381 DEBUG:boto:Method: HEAD 2023-09-22T21:49:07.381 DEBUG:boto:Path: /ffzdqu-59/obj111 2023-09-22T21:49:07.381 DEBUG:boto:Data: 2023-09-22T21:49:07.381 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.381 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.381 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.381 DEBUG:boto:Params: {} 2023-09-22T21:49:07.382 DEBUG:boto:Token: None 2023-09-22T21:49:07.382 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj111 2023-09-22T21:49:07.382 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ziaGrY+Hf67ngDCsDDTco76Dj5w= 2023-09-22T21:49:07.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ziaGrY+Hf67ngDCsDDTco76Dj5w=', 'Content-Length': '0'} 2023-09-22T21:49:07.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b40ae3f99dd1587c-00650e0bd3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-09-22T21:49:07.385 DEBUG:boto:path=/ffzdqu-59/obj111 2023-09-22T21:49:07.385 DEBUG:boto:auth_path=/ffzdqu-59/obj111 2023-09-22T21:49:07.386 DEBUG:boto:Method: GET 2023-09-22T21:49:07.386 DEBUG:boto:Path: /ffzdqu-59/obj111 2023-09-22T21:49:07.386 DEBUG:boto:Data: 2023-09-22T21:49:07.386 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.386 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.386 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.386 DEBUG:boto:Params: {} 2023-09-22T21:49:07.386 DEBUG:boto:Token: None 2023-09-22T21:49:07.387 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj111 2023-09-22T21:49:07.387 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Fk23wOE8lPnILPnSgaka7oPWe9k= 2023-09-22T21:49:07.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Fk23wOE8lPnILPnSgaka7oPWe9k=', 'Content-Length': '0'} 2023-09-22T21:49:07.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee33b686d15d9e7e-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.390 DEBUG:boto:path=/ffzdqu-59/obj111 2023-09-22T21:49:07.390 DEBUG:boto:auth_path=/ffzdqu-59/obj111 2023-09-22T21:49:07.390 DEBUG:boto:Method: GET 2023-09-22T21:49:07.390 DEBUG:boto:Path: /ffzdqu-59/obj111 2023-09-22T21:49:07.391 DEBUG:boto:Data: 2023-09-22T21:49:07.391 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.391 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.391 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.391 DEBUG:boto:Params: {} 2023-09-22T21:49:07.391 DEBUG:boto:Token: None 2023-09-22T21:49:07.391 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj111 2023-09-22T21:49:07.392 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Fk23wOE8lPnILPnSgaka7oPWe9k= 2023-09-22T21:49:07.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Fk23wOE8lPnILPnSgaka7oPWe9k=', 'Content-Length': '0'} 2023-09-22T21:49:07.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043131584a7885132-00650e0bd3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.434 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:07.435 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:07.435 DEBUG:boto:path=/ffzdqu-59/?prefix=obj111 2023-09-22T21:49:07.435 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj111 2023-09-22T21:49:07.435 DEBUG:boto:Method: GET 2023-09-22T21:49:07.435 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj111 2023-09-22T21:49:07.435 DEBUG:boto:Data: 2023-09-22T21:49:07.435 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.435 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.436 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.436 DEBUG:boto:Params: {} 2023-09-22T21:49:07.436 DEBUG:boto:Token: None 2023-09-22T21:49:07.436 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/ 2023-09-22T21:49:07.436 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs= 2023-09-22T21:49:07.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs=', 'Content-Length': '0'} 2023-09-22T21:49:07.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092d90da784bd0b16-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.441 DEBUG:boto:b'ffzdqu-59obj1111000falseobj1112023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:07.441 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:07.441 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:07.442 DEBUG:boto:path=/ffzdqu-59/?prefix=obj111 2023-09-22T21:49:07.442 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj111 2023-09-22T21:49:07.442 DEBUG:boto:Method: GET 2023-09-22T21:49:07.442 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj111 2023-09-22T21:49:07.442 DEBUG:boto:Data: 2023-09-22T21:49:07.442 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.442 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.442 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.443 DEBUG:boto:Params: {} 2023-09-22T21:49:07.443 DEBUG:boto:Token: None 2023-09-22T21:49:07.443 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/ 2023-09-22T21:49:07.443 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs= 2023-09-22T21:49:07.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs=', 'Content-Length': '0'} 2023-09-22T21:49:07.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4314bbcd005683f-00650e0bd3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.453 DEBUG:boto:b'ffzdqu-59obj1111000falseobj1112023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:07.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-22T21:49:07.454 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:49:07.454 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:49:07.455 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.455 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.455 DEBUG:boto:Method: GET 2023-09-22T21:49:07.455 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.455 DEBUG:boto:Data: 2023-09-22T21:49:07.456 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.456 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.456 DEBUG:boto:Params: {} 2023-09-22T21:49:07.456 DEBUG:boto:Token: None 2023-09-22T21:49:07.456 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.457 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y2wRgixhQDkI0sLm56ASbof5zsU= 2023-09-22T21:49:07.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y2wRgixhQDkI0sLm56ASbof5zsU=', 'Content-Length': '0'} 2023-09-22T21:49:07.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cae915f6eda11f05-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.502 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:49:07.503 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:49:07.503 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.503 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.503 DEBUG:boto:Method: GET 2023-09-22T21:49:07.503 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.503 DEBUG:boto:Data: 2023-09-22T21:49:07.503 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.503 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.504 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.504 DEBUG:boto:Params: {} 2023-09-22T21:49:07.504 DEBUG:boto:Token: None 2023-09-22T21:49:07.504 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.504 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y2wRgixhQDkI0sLm56ASbof5zsU= 2023-09-22T21:49:07.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y2wRgixhQDkI0sLm56ASbof5zsU=', 'Content-Length': '0'} 2023-09-22T21:49:07.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de48e75c0f7aaabc-00650e0bd3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.508 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:49:07.508 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:49:07.508 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.508 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.508 DEBUG:boto:Method: HEAD 2023-09-22T21:49:07.509 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.509 DEBUG:boto:Data: 2023-09-22T21:49:07.509 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.509 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.509 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.509 DEBUG:boto:Params: {} 2023-09-22T21:49:07.509 DEBUG:boto:Token: None 2023-09-22T21:49:07.509 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.510 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q18vpb1FqOac3v4gpWBfNj+r7wI= 2023-09-22T21:49:07.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q18vpb1FqOac3v4gpWBfNj+r7wI=', 'Content-Length': '0'} 2023-09-22T21:49:07.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099db4f5633009bdd-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.513 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:49:07.513 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:49:07.513 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.513 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.513 DEBUG:boto:Method: HEAD 2023-09-22T21:49:07.513 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.514 DEBUG:boto:Data: 2023-09-22T21:49:07.514 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.514 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.514 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.514 DEBUG:boto:Params: {} 2023-09-22T21:49:07.514 DEBUG:boto:Token: None 2023-09-22T21:49:07.514 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.515 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q18vpb1FqOac3v4gpWBfNj+r7wI= 2023-09-22T21:49:07.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q18vpb1FqOac3v4gpWBfNj+r7wI=', 'Content-Length': '0'} 2023-09-22T21:49:07.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000268c46d5eccca6b2-00650e0bd3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-22T21:49:07.518 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:49:07.518 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:49:07.518 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.518 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.518 DEBUG:boto:Method: GET 2023-09-22T21:49:07.518 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.519 DEBUG:boto:Data: 2023-09-22T21:49:07.519 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.519 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.519 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.519 DEBUG:boto:Params: {} 2023-09-22T21:49:07.519 DEBUG:boto:Token: None 2023-09-22T21:49:07.519 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.519 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y2wRgixhQDkI0sLm56ASbof5zsU= 2023-09-22T21:49:07.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y2wRgixhQDkI0sLm56ASbof5zsU=', 'Content-Length': '0'} 2023-09-22T21:49:07.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff506d7575469c7c-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.523 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:49:07.523 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:49:07.523 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.523 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.523 DEBUG:boto:Method: GET 2023-09-22T21:49:07.523 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.524 DEBUG:boto:Data: 2023-09-22T21:49:07.524 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.524 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.524 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.524 DEBUG:boto:Params: {} 2023-09-22T21:49:07.524 DEBUG:boto:Token: None 2023-09-22T21:49:07.524 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:07.524 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y2wRgixhQDkI0sLm56ASbof5zsU= 2023-09-22T21:49:07.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y2wRgixhQDkI0sLm56ASbof5zsU=', 'Content-Length': '0'} 2023-09-22T21:49:07.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a91609af277977b-00650e0bd3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.528 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:07.528 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:07.528 DEBUG:boto:path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:49:07.528 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:49:07.528 DEBUG:boto:Method: GET 2023-09-22T21:49:07.528 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj115 2023-09-22T21:49:07.528 DEBUG:boto:Data: 2023-09-22T21:49:07.528 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.529 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.529 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.529 DEBUG:boto:Params: {} 2023-09-22T21:49:07.529 DEBUG:boto:Token: None 2023-09-22T21:49:07.529 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/ 2023-09-22T21:49:07.529 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs= 2023-09-22T21:49:07.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs=', 'Content-Length': '0'} 2023-09-22T21:49:07.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008855340d82464bc9-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.534 DEBUG:boto:b'ffzdqu-59obj1151000falseobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:07.535 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:07.535 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:07.535 DEBUG:boto:path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:49:07.535 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:49:07.535 DEBUG:boto:Method: GET 2023-09-22T21:49:07.535 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj115 2023-09-22T21:49:07.535 DEBUG:boto:Data: 2023-09-22T21:49:07.536 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.536 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.536 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.536 DEBUG:boto:Params: {} 2023-09-22T21:49:07.536 DEBUG:boto:Token: None 2023-09-22T21:49:07.536 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/ 2023-09-22T21:49:07.536 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs= 2023-09-22T21:49:07.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs=', 'Content-Length': '0'} 2023-09-22T21:49:07.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010c03ce936bd292e-00650e0bd3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.586 DEBUG:boto:b'ffzdqu-59obj1151000falseobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:07.587 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:49:07.587 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:49:07.587 DEBUG:boto:Method: HEAD 2023-09-22T21:49:07.587 DEBUG:boto:Path: /ffzdqu-59/obj115 2023-09-22T21:49:07.587 DEBUG:boto:Data: 2023-09-22T21:49:07.587 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.587 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.588 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.588 DEBUG:boto:Params: {} 2023-09-22T21:49:07.588 DEBUG:boto:Token: None 2023-09-22T21:49:07.588 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj115 2023-09-22T21:49:07.588 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kGhATBFWM8kGG1s7NZNMtggAO4c= 2023-09-22T21:49:07.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kGhATBFWM8kGG1s7NZNMtggAO4c=', 'Content-Length': '0'} 2023-09-22T21:49:07.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9d2566cd90a94e5-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.591 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:49:07.591 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:49:07.592 DEBUG:boto:Method: HEAD 2023-09-22T21:49:07.592 DEBUG:boto:Path: /ffzdqu-59/obj115 2023-09-22T21:49:07.592 DEBUG:boto:Data: 2023-09-22T21:49:07.592 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.592 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.592 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.592 DEBUG:boto:Params: {} 2023-09-22T21:49:07.592 DEBUG:boto:Token: None 2023-09-22T21:49:07.593 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj115 2023-09-22T21:49:07.593 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kGhATBFWM8kGG1s7NZNMtggAO4c= 2023-09-22T21:49:07.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kGhATBFWM8kGG1s7NZNMtggAO4c=', 'Content-Length': '0'} 2023-09-22T21:49:07.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fa8217cafbd0741-00650e0bd3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-22T21:49:07.596 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:49:07.596 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:49:07.596 DEBUG:boto:Method: GET 2023-09-22T21:49:07.596 DEBUG:boto:Path: /ffzdqu-59/obj115 2023-09-22T21:49:07.596 DEBUG:boto:Data: 2023-09-22T21:49:07.596 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.596 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.597 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.597 DEBUG:boto:Params: {} 2023-09-22T21:49:07.597 DEBUG:boto:Token: None 2023-09-22T21:49:07.597 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj115 2023-09-22T21:49:07.597 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H6cyHtYSP1J0F2uV6kFteTVY2z8= 2023-09-22T21:49:07.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H6cyHtYSP1J0F2uV6kFteTVY2z8=', 'Content-Length': '0'} 2023-09-22T21:49:07.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad821294b71e1b96-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.642 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:49:07.642 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:49:07.643 DEBUG:boto:Method: GET 2023-09-22T21:49:07.643 DEBUG:boto:Path: /ffzdqu-59/obj115 2023-09-22T21:49:07.643 DEBUG:boto:Data: 2023-09-22T21:49:07.643 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.643 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.643 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.643 DEBUG:boto:Params: {} 2023-09-22T21:49:07.643 DEBUG:boto:Token: None 2023-09-22T21:49:07.643 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj115 2023-09-22T21:49:07.644 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H6cyHtYSP1J0F2uV6kFteTVY2z8= 2023-09-22T21:49:07.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H6cyHtYSP1J0F2uV6kFteTVY2z8=', 'Content-Length': '0'} 2023-09-22T21:49:07.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d190c71a29bf42f6-00650e0bd3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.647 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:07.647 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:07.647 DEBUG:boto:path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:49:07.647 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:49:07.647 DEBUG:boto:Method: GET 2023-09-22T21:49:07.647 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj115 2023-09-22T21:49:07.647 DEBUG:boto:Data: 2023-09-22T21:49:07.648 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.648 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.648 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.648 DEBUG:boto:Params: {} 2023-09-22T21:49:07.648 DEBUG:boto:Token: None 2023-09-22T21:49:07.648 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/ 2023-09-22T21:49:07.648 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs= 2023-09-22T21:49:07.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs=', 'Content-Length': '0'} 2023-09-22T21:49:07.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7d1598de35aea7f-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.653 DEBUG:boto:b'ffzdqu-59obj1151000falseobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:07.653 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:07.653 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:07.653 DEBUG:boto:path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:49:07.654 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:49:07.654 DEBUG:boto:Method: GET 2023-09-22T21:49:07.654 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj115 2023-09-22T21:49:07.654 DEBUG:boto:Data: 2023-09-22T21:49:07.654 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.654 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.654 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.654 DEBUG:boto:Params: {} 2023-09-22T21:49:07.655 DEBUG:boto:Token: None 2023-09-22T21:49:07.655 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/ 2023-09-22T21:49:07.655 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs= 2023-09-22T21:49:07.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs=', 'Content-Length': '0'} 2023-09-22T21:49:07.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c76295dd95330788-00650e0bd3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.665 DEBUG:boto:b'ffzdqu-59obj1151000falseobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:07.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-22T21:49:07.665 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:49:07.665 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:49:07.665 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.665 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.665 DEBUG:boto:Method: GET 2023-09-22T21:49:07.666 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.666 DEBUG:boto:Data: 2023-09-22T21:49:07.666 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.666 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.666 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.666 DEBUG:boto:Params: {} 2023-09-22T21:49:07.666 DEBUG:boto:Token: None 2023-09-22T21:49:07.666 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.667 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zJmu5a5DuqTz+B/E5eJtkw+2BX8= 2023-09-22T21:49:07.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zJmu5a5DuqTz+B/E5eJtkw+2BX8=', 'Content-Length': '0'} 2023-09-22T21:49:07.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c3aeace2f2f97bd-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.670 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:49:07.670 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:49:07.670 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.670 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.670 DEBUG:boto:Method: GET 2023-09-22T21:49:07.670 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.670 DEBUG:boto:Data: 2023-09-22T21:49:07.670 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.671 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.671 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.671 DEBUG:boto:Params: {} 2023-09-22T21:49:07.671 DEBUG:boto:Token: None 2023-09-22T21:49:07.671 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.671 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zJmu5a5DuqTz+B/E5eJtkw+2BX8= 2023-09-22T21:49:07.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zJmu5a5DuqTz+B/E5eJtkw+2BX8=', 'Content-Length': '0'} 2023-09-22T21:49:07.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfb02a008e6d791a-00650e0bd3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.714 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:49:07.715 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:49:07.715 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.715 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.715 DEBUG:boto:Method: HEAD 2023-09-22T21:49:07.715 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.715 DEBUG:boto:Data: 2023-09-22T21:49:07.715 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.715 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.716 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.716 DEBUG:boto:Params: {} 2023-09-22T21:49:07.716 DEBUG:boto:Token: None 2023-09-22T21:49:07.716 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.716 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sTriSxT+1fk5bbuktAbYR7Jv+mY= 2023-09-22T21:49:07.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sTriSxT+1fk5bbuktAbYR7Jv+mY=', 'Content-Length': '0'} 2023-09-22T21:49:07.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000241f0e10552ce8f8-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.719 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:49:07.720 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:49:07.720 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.720 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.720 DEBUG:boto:Method: HEAD 2023-09-22T21:49:07.721 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.721 DEBUG:boto:Data: 2023-09-22T21:49:07.721 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.721 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.721 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.722 DEBUG:boto:Params: {} 2023-09-22T21:49:07.722 DEBUG:boto:Token: None 2023-09-22T21:49:07.722 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.723 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sTriSxT+1fk5bbuktAbYR7Jv+mY= 2023-09-22T21:49:07.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sTriSxT+1fk5bbuktAbYR7Jv+mY=', 'Content-Length': '0'} 2023-09-22T21:49:07.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce2429a017c44c2f-00650e0bd3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-22T21:49:07.726 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:49:07.726 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:49:07.726 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.726 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.727 DEBUG:boto:Method: GET 2023-09-22T21:49:07.727 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.727 DEBUG:boto:Data: 2023-09-22T21:49:07.727 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.727 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.727 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.727 DEBUG:boto:Params: {} 2023-09-22T21:49:07.727 DEBUG:boto:Token: None 2023-09-22T21:49:07.727 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.728 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zJmu5a5DuqTz+B/E5eJtkw+2BX8= 2023-09-22T21:49:07.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zJmu5a5DuqTz+B/E5eJtkw+2BX8=', 'Content-Length': '0'} 2023-09-22T21:49:07.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1af67c9ded3c5f6-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.774 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:49:07.775 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:49:07.775 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.775 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.775 DEBUG:boto:Method: GET 2023-09-22T21:49:07.775 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.775 DEBUG:boto:Data: 2023-09-22T21:49:07.775 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.775 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.775 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.775 DEBUG:boto:Params: {} 2023-09-22T21:49:07.776 DEBUG:boto:Token: None 2023-09-22T21:49:07.776 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:07.776 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zJmu5a5DuqTz+B/E5eJtkw+2BX8= 2023-09-22T21:49:07.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zJmu5a5DuqTz+B/E5eJtkw+2BX8=', 'Content-Length': '0'} 2023-09-22T21:49:07.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d6539caebd6478a-00650e0bd3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.779 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:07.780 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:07.780 DEBUG:boto:path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:49:07.780 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:49:07.780 DEBUG:boto:Method: GET 2023-09-22T21:49:07.780 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj116 2023-09-22T21:49:07.780 DEBUG:boto:Data: 2023-09-22T21:49:07.780 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.780 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.780 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.780 DEBUG:boto:Params: {} 2023-09-22T21:49:07.781 DEBUG:boto:Token: None 2023-09-22T21:49:07.781 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/ 2023-09-22T21:49:07.781 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs= 2023-09-22T21:49:07.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs=', 'Content-Length': '0'} 2023-09-22T21:49:07.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8e9327b28414998-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.786 DEBUG:boto:b'ffzdqu-59obj1161000falseobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:07.787 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:07.787 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:07.787 DEBUG:boto:path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:49:07.787 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:49:07.787 DEBUG:boto:Method: GET 2023-09-22T21:49:07.787 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj116 2023-09-22T21:49:07.787 DEBUG:boto:Data: 2023-09-22T21:49:07.787 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.787 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.788 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.788 DEBUG:boto:Params: {} 2023-09-22T21:49:07.788 DEBUG:boto:Token: None 2023-09-22T21:49:07.788 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/ 2023-09-22T21:49:07.788 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs= 2023-09-22T21:49:07.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVl+LLAEEYznFIHq3qzqk8tiXDs=', 'Content-Length': '0'} 2023-09-22T21:49:07.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d3c235e797c56ec-00650e0bd3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.800 DEBUG:boto:b'ffzdqu-59obj1161000falseobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:07.800 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:49:07.800 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:49:07.800 DEBUG:boto:Method: HEAD 2023-09-22T21:49:07.800 DEBUG:boto:Path: /ffzdqu-59/obj116 2023-09-22T21:49:07.801 DEBUG:boto:Data: 2023-09-22T21:49:07.801 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.801 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.801 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.801 DEBUG:boto:Params: {} 2023-09-22T21:49:07.801 DEBUG:boto:Token: None 2023-09-22T21:49:07.801 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj116 2023-09-22T21:49:07.801 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x3DUCNu4okSF9ad3esovTKVfXB8= 2023-09-22T21:49:07.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x3DUCNu4okSF9ad3esovTKVfXB8=', 'Content-Length': '0'} 2023-09-22T21:49:07.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d7096d5bfbc92b6-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.804 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:49:07.804 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:49:07.804 DEBUG:boto:Method: HEAD 2023-09-22T21:49:07.805 DEBUG:boto:Path: /ffzdqu-59/obj116 2023-09-22T21:49:07.805 DEBUG:boto:Data: 2023-09-22T21:49:07.805 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.805 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:07.805 DEBUG:boto:Port: 8000 2023-09-22T21:49:07.805 DEBUG:boto:Params: {} 2023-09-22T21:49:07.805 DEBUG:boto:Token: None 2023-09-22T21:49:07.805 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj116 2023-09-22T21:49:07.805 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x3DUCNu4okSF9ad3esovTKVfXB8= 2023-09-22T21:49:07.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x3DUCNu4okSF9ad3esovTKVfXB8=', 'Content-Length': '0'} 2023-09-22T21:49:07.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b981e45b178ab51-00650e0bd3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-22T21:49:07.809 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:49:07.809 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:49:07.809 DEBUG:boto:Method: GET 2023-09-22T21:49:07.809 DEBUG:boto:Path: /ffzdqu-59/obj116 2023-09-22T21:49:07.809 DEBUG:boto:Data: 2023-09-22T21:49:07.809 DEBUG:boto:Headers: {} 2023-09-22T21:49:07.809 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:07.809 DEBUG:boto:Port: 8001 2023-09-22T21:49:07.809 DEBUG:boto:Params: {} 2023-09-22T21:49:07.810 DEBUG:boto:Token: None 2023-09-22T21:49:07.810 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:07 GMT /ffzdqu-59/obj116 2023-09-22T21:49:07.810 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GS3QhNxCKBEYWZ4wg8+fanqH9Bc= 2023-09-22T21:49:07.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GS3QhNxCKBEYWZ4wg8+fanqH9Bc=', 'Content-Length': '0'} 2023-09-22T21:49:07.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004105764f97202ac5-00650e0bd3-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:07.854 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:49:08.485 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:49:08.486 DEBUG:boto:Method: GET 2023-09-22T21:49:08.486 DEBUG:boto:Path: /ffzdqu-59/obj116 2023-09-22T21:49:08.486 DEBUG:boto:Data: 2023-09-22T21:49:08.486 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.487 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:08.487 DEBUG:boto:Port: 8000 2023-09-22T21:49:08.487 DEBUG:boto:Params: {} 2023-09-22T21:49:08.487 DEBUG:boto:Token: None 2023-09-22T21:49:08.487 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj116 2023-09-22T21:49:08.488 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MYZTMgQHDqtj4Wh069YVerSTHJU= 2023-09-22T21:49:08.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MYZTMgQHDqtj4Wh069YVerSTHJU=', 'Content-Length': '0'} 2023-09-22T21:49:08.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b184045dde682072-00650e0bd4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.492 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:08.492 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:08.493 DEBUG:boto:path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:49:08.493 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:49:08.493 DEBUG:boto:Method: GET 2023-09-22T21:49:08.493 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj116 2023-09-22T21:49:08.493 DEBUG:boto:Data: 2023-09-22T21:49:08.494 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.494 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:08.494 DEBUG:boto:Port: 8001 2023-09-22T21:49:08.494 DEBUG:boto:Params: {} 2023-09-22T21:49:08.494 DEBUG:boto:Token: None 2023-09-22T21:49:08.494 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/ 2023-09-22T21:49:08.494 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs= 2023-09-22T21:49:08.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs=', 'Content-Length': '0'} 2023-09-22T21:49:08.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000169206f93fc8096c-00650e0bd4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.499 DEBUG:boto:b'ffzdqu-59obj1161000falseobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:08.500 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:08.500 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:08.500 DEBUG:boto:path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:49:08.500 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:49:08.500 DEBUG:boto:Method: GET 2023-09-22T21:49:08.500 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj116 2023-09-22T21:49:08.501 DEBUG:boto:Data: 2023-09-22T21:49:08.501 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.501 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:08.501 DEBUG:boto:Port: 8000 2023-09-22T21:49:08.501 DEBUG:boto:Params: {} 2023-09-22T21:49:08.501 DEBUG:boto:Token: None 2023-09-22T21:49:08.501 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/ 2023-09-22T21:49:08.501 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs= 2023-09-22T21:49:08.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs=', 'Content-Length': '0'} 2023-09-22T21:49:08.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097d144e078bf31a9-00650e0bd4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.512 DEBUG:boto:b'ffzdqu-59obj1161000falseobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:08.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-22T21:49:08.512 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:49:08.512 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:49:08.512 DEBUG:boto:path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.512 DEBUG:boto:auth_path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.512 DEBUG:boto:Method: GET 2023-09-22T21:49:08.513 DEBUG:boto:Path: /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.513 DEBUG:boto:Data: 2023-09-22T21:49:08.513 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.513 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:08.513 DEBUG:boto:Port: 8001 2023-09-22T21:49:08.513 DEBUG:boto:Params: {} 2023-09-22T21:49:08.513 DEBUG:boto:Token: None 2023-09-22T21:49:08.513 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.513 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cOfBvjrOB80zL9Ee0PlBleVx6xU= 2023-09-22T21:49:08.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cOfBvjrOB80zL9Ee0PlBleVx6xU=', 'Content-Length': '0'} 2023-09-22T21:49:08.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d07dd07c26111e30-00650e0bd4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.517 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:49:08.517 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:49:08.517 DEBUG:boto:path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.517 DEBUG:boto:auth_path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.517 DEBUG:boto:Method: GET 2023-09-22T21:49:08.517 DEBUG:boto:Path: /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.517 DEBUG:boto:Data: 2023-09-22T21:49:08.518 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.518 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:08.518 DEBUG:boto:Port: 8000 2023-09-22T21:49:08.518 DEBUG:boto:Params: {} 2023-09-22T21:49:08.518 DEBUG:boto:Token: None 2023-09-22T21:49:08.518 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.518 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cOfBvjrOB80zL9Ee0PlBleVx6xU= 2023-09-22T21:49:08.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cOfBvjrOB80zL9Ee0PlBleVx6xU=', 'Content-Length': '0'} 2023-09-22T21:49:08.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000751292c9587d4b13-00650e0bd4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.521 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:49:08.521 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:49:08.522 DEBUG:boto:path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.522 DEBUG:boto:auth_path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.522 DEBUG:boto:Method: HEAD 2023-09-22T21:49:08.522 DEBUG:boto:Path: /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.522 DEBUG:boto:Data: 2023-09-22T21:49:08.522 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.522 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:08.522 DEBUG:boto:Port: 8001 2023-09-22T21:49:08.522 DEBUG:boto:Params: {} 2023-09-22T21:49:08.523 DEBUG:boto:Token: None 2023-09-22T21:49:08.523 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.523 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UPu+7ZyeEypKKSQsCrTRYc3wtjM= 2023-09-22T21:49:08.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UPu+7ZyeEypKKSQsCrTRYc3wtjM=', 'Content-Length': '0'} 2023-09-22T21:49:08.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1e62b1794993524-00650e0bd4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.526 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:49:08.526 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:49:08.526 DEBUG:boto:path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.526 DEBUG:boto:auth_path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.526 DEBUG:boto:Method: HEAD 2023-09-22T21:49:08.526 DEBUG:boto:Path: /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.526 DEBUG:boto:Data: 2023-09-22T21:49:08.527 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.527 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:08.527 DEBUG:boto:Port: 8000 2023-09-22T21:49:08.527 DEBUG:boto:Params: {} 2023-09-22T21:49:08.527 DEBUG:boto:Token: None 2023-09-22T21:49:08.527 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.527 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UPu+7ZyeEypKKSQsCrTRYc3wtjM= 2023-09-22T21:49:08.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UPu+7ZyeEypKKSQsCrTRYc3wtjM=', 'Content-Length': '0'} 2023-09-22T21:49:08.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000756d4134d9f4b903-00650e0bd4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-22T21:49:08.530 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:49:08.531 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:49:08.531 DEBUG:boto:path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.531 DEBUG:boto:auth_path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.531 DEBUG:boto:Method: GET 2023-09-22T21:49:08.531 DEBUG:boto:Path: /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.531 DEBUG:boto:Data: 2023-09-22T21:49:08.531 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.531 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:08.531 DEBUG:boto:Port: 8001 2023-09-22T21:49:08.531 DEBUG:boto:Params: {} 2023-09-22T21:49:08.532 DEBUG:boto:Token: None 2023-09-22T21:49:08.532 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.532 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cOfBvjrOB80zL9Ee0PlBleVx6xU= 2023-09-22T21:49:08.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cOfBvjrOB80zL9Ee0PlBleVx6xU=', 'Content-Length': '0'} 2023-09-22T21:49:08.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cfc502eddf996be-00650e0bd4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.535 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:49:08.535 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:49:08.535 DEBUG:boto:path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.535 DEBUG:boto:auth_path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.535 DEBUG:boto:Method: GET 2023-09-22T21:49:08.535 DEBUG:boto:Path: /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.535 DEBUG:boto:Data: 2023-09-22T21:49:08.536 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.536 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:08.536 DEBUG:boto:Port: 8000 2023-09-22T21:49:08.536 DEBUG:boto:Params: {} 2023-09-22T21:49:08.536 DEBUG:boto:Token: None 2023-09-22T21:49:08.536 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:08.536 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cOfBvjrOB80zL9Ee0PlBleVx6xU= 2023-09-22T21:49:08.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cOfBvjrOB80zL9Ee0PlBleVx6xU=', 'Content-Length': '0'} 2023-09-22T21:49:08.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064080ef609fde3f5-00650e0bd4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.540 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:08.541 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:08.541 DEBUG:boto:path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:49:08.541 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:49:08.541 DEBUG:boto:Method: GET 2023-09-22T21:49:08.541 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj124 2023-09-22T21:49:08.541 DEBUG:boto:Data: 2023-09-22T21:49:08.541 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.541 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:08.557 DEBUG:boto:Port: 8001 2023-09-22T21:49:08.557 DEBUG:boto:Params: {} 2023-09-22T21:49:08.557 DEBUG:boto:Token: None 2023-09-22T21:49:08.557 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/ 2023-09-22T21:49:08.557 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs= 2023-09-22T21:49:08.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs=', 'Content-Length': '0'} 2023-09-22T21:49:08.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c285de863b943cc0-00650e0bd4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.602 DEBUG:boto:b'ffzdqu-59obj1241000falseobj1242023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:08.603 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:08.603 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:08.603 DEBUG:boto:path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:49:08.603 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:49:08.605 DEBUG:boto:Method: GET 2023-09-22T21:49:08.605 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj124 2023-09-22T21:49:08.605 DEBUG:boto:Data: 2023-09-22T21:49:08.605 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.605 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:08.605 DEBUG:boto:Port: 8000 2023-09-22T21:49:08.605 DEBUG:boto:Params: {} 2023-09-22T21:49:08.605 DEBUG:boto:Token: None 2023-09-22T21:49:08.606 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/ 2023-09-22T21:49:08.606 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs= 2023-09-22T21:49:08.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs=', 'Content-Length': '0'} 2023-09-22T21:49:08.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca9ce0b65859b892-00650e0bd4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.616 DEBUG:boto:b'ffzdqu-59obj1241000falseobj1242023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:08.616 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:49:08.616 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:49:08.616 DEBUG:boto:Method: HEAD 2023-09-22T21:49:08.616 DEBUG:boto:Path: /ffzdqu-59/obj124 2023-09-22T21:49:08.617 DEBUG:boto:Data: 2023-09-22T21:49:08.617 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.617 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:08.617 DEBUG:boto:Port: 8001 2023-09-22T21:49:08.617 DEBUG:boto:Params: {} 2023-09-22T21:49:08.617 DEBUG:boto:Token: None 2023-09-22T21:49:08.617 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj124 2023-09-22T21:49:08.617 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x98/73ha2Q9OwaXWV7v0LVeaIWA= 2023-09-22T21:49:08.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x98/73ha2Q9OwaXWV7v0LVeaIWA=', 'Content-Length': '0'} 2023-09-22T21:49:08.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002508905496559f35-00650e0bd4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.620 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:49:08.620 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:49:08.621 DEBUG:boto:Method: HEAD 2023-09-22T21:49:08.621 DEBUG:boto:Path: /ffzdqu-59/obj124 2023-09-22T21:49:08.621 DEBUG:boto:Data: 2023-09-22T21:49:08.621 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.621 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:08.621 DEBUG:boto:Port: 8000 2023-09-22T21:49:08.621 DEBUG:boto:Params: {} 2023-09-22T21:49:08.621 DEBUG:boto:Token: None 2023-09-22T21:49:08.621 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj124 2023-09-22T21:49:08.622 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x98/73ha2Q9OwaXWV7v0LVeaIWA= 2023-09-22T21:49:08.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x98/73ha2Q9OwaXWV7v0LVeaIWA=', 'Content-Length': '0'} 2023-09-22T21:49:08.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084ed1c3dc1fff62a-00650e0bd4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-22T21:49:08.625 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:49:08.625 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:49:08.625 DEBUG:boto:Method: GET 2023-09-22T21:49:08.625 DEBUG:boto:Path: /ffzdqu-59/obj124 2023-09-22T21:49:08.625 DEBUG:boto:Data: 2023-09-22T21:49:08.625 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.625 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:08.625 DEBUG:boto:Port: 8001 2023-09-22T21:49:08.626 DEBUG:boto:Params: {} 2023-09-22T21:49:08.626 DEBUG:boto:Token: None 2023-09-22T21:49:08.626 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj124 2023-09-22T21:49:08.626 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Xwhb/ugffLREi4IRDJlnVj1kCT4= 2023-09-22T21:49:08.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Xwhb/ugffLREi4IRDJlnVj1kCT4=', 'Content-Length': '0'} 2023-09-22T21:49:08.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c248c26b64a91eb-00650e0bd4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.629 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:49:08.629 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:49:08.629 DEBUG:boto:Method: GET 2023-09-22T21:49:08.629 DEBUG:boto:Path: /ffzdqu-59/obj124 2023-09-22T21:49:08.629 DEBUG:boto:Data: 2023-09-22T21:49:08.630 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.630 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:08.630 DEBUG:boto:Port: 8000 2023-09-22T21:49:08.630 DEBUG:boto:Params: {} 2023-09-22T21:49:08.630 DEBUG:boto:Token: None 2023-09-22T21:49:08.630 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj124 2023-09-22T21:49:08.630 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Xwhb/ugffLREi4IRDJlnVj1kCT4= 2023-09-22T21:49:08.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Xwhb/ugffLREi4IRDJlnVj1kCT4=', 'Content-Length': '0'} 2023-09-22T21:49:08.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ded71fee7fa3a1f-00650e0bd4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.678 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:08.679 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:08.679 DEBUG:boto:path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:49:08.679 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:49:08.679 DEBUG:boto:Method: GET 2023-09-22T21:49:08.679 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj124 2023-09-22T21:49:08.679 DEBUG:boto:Data: 2023-09-22T21:49:08.679 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.679 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:08.679 DEBUG:boto:Port: 8001 2023-09-22T21:49:08.680 DEBUG:boto:Params: {} 2023-09-22T21:49:08.680 DEBUG:boto:Token: None 2023-09-22T21:49:08.680 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/ 2023-09-22T21:49:08.680 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs= 2023-09-22T21:49:08.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs=', 'Content-Length': '0'} 2023-09-22T21:49:08.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c40999b99484e42-00650e0bd4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.685 DEBUG:boto:b'ffzdqu-59obj1241000falseobj1242023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:08.685 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:08.686 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:08.686 DEBUG:boto:path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:49:08.686 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:49:08.686 DEBUG:boto:Method: GET 2023-09-22T21:49:08.686 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj124 2023-09-22T21:49:08.686 DEBUG:boto:Data: 2023-09-22T21:49:08.686 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.686 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:08.687 DEBUG:boto:Port: 8000 2023-09-22T21:49:08.687 DEBUG:boto:Params: {} 2023-09-22T21:49:08.687 DEBUG:boto:Token: None 2023-09-22T21:49:08.687 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/ 2023-09-22T21:49:08.687 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs= 2023-09-22T21:49:08.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs=', 'Content-Length': '0'} 2023-09-22T21:49:08.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de88e3da602e2169-00650e0bd4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.697 DEBUG:boto:b'ffzdqu-59obj1241000falseobj1242023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:08.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-22T21:49:08.698 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:49:08.698 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:49:08.698 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.698 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.698 DEBUG:boto:Method: GET 2023-09-22T21:49:08.698 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.698 DEBUG:boto:Data: 2023-09-22T21:49:08.699 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.699 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:08.699 DEBUG:boto:Port: 8001 2023-09-22T21:49:08.699 DEBUG:boto:Params: {} 2023-09-22T21:49:08.699 DEBUG:boto:Token: None 2023-09-22T21:49:08.699 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.699 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RlfWkMdMrdxHIcmh7cCnt+YDjLg= 2023-09-22T21:49:08.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RlfWkMdMrdxHIcmh7cCnt+YDjLg=', 'Content-Length': '0'} 2023-09-22T21:49:08.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e453785db6bc5fdd-00650e0bd4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.742 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:49:08.743 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:49:08.743 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.743 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.743 DEBUG:boto:Method: GET 2023-09-22T21:49:08.743 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.743 DEBUG:boto:Data: 2023-09-22T21:49:08.743 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.743 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:08.743 DEBUG:boto:Port: 8000 2023-09-22T21:49:08.743 DEBUG:boto:Params: {} 2023-09-22T21:49:08.744 DEBUG:boto:Token: None 2023-09-22T21:49:08.744 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.744 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RlfWkMdMrdxHIcmh7cCnt+YDjLg= 2023-09-22T21:49:08.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RlfWkMdMrdxHIcmh7cCnt+YDjLg=', 'Content-Length': '0'} 2023-09-22T21:49:08.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007478d6410092f180-00650e0bd4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.747 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:49:08.747 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:49:08.747 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.747 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.747 DEBUG:boto:Method: HEAD 2023-09-22T21:49:08.748 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.748 DEBUG:boto:Data: 2023-09-22T21:49:08.748 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.748 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:08.748 DEBUG:boto:Port: 8001 2023-09-22T21:49:08.748 DEBUG:boto:Params: {} 2023-09-22T21:49:08.748 DEBUG:boto:Token: None 2023-09-22T21:49:08.748 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.749 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G9umwfkA3uJrkBbq3gNuPgVUnZo= 2023-09-22T21:49:08.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G9umwfkA3uJrkBbq3gNuPgVUnZo=', 'Content-Length': '0'} 2023-09-22T21:49:08.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007efa8dcceafeaebf-00650e0bd4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.751 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:49:08.752 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:49:08.752 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.752 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.752 DEBUG:boto:Method: HEAD 2023-09-22T21:49:08.752 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.752 DEBUG:boto:Data: 2023-09-22T21:49:08.752 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.752 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:08.752 DEBUG:boto:Port: 8000 2023-09-22T21:49:08.753 DEBUG:boto:Params: {} 2023-09-22T21:49:08.753 DEBUG:boto:Token: None 2023-09-22T21:49:08.753 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.753 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G9umwfkA3uJrkBbq3gNuPgVUnZo= 2023-09-22T21:49:08.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G9umwfkA3uJrkBbq3gNuPgVUnZo=', 'Content-Length': '0'} 2023-09-22T21:49:08.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000627925c47559e3e7-00650e0bd4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-22T21:49:08.756 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:49:08.756 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:49:08.756 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.756 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.757 DEBUG:boto:Method: GET 2023-09-22T21:49:08.757 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.757 DEBUG:boto:Data: 2023-09-22T21:49:08.757 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.757 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:08.757 DEBUG:boto:Port: 8001 2023-09-22T21:49:08.757 DEBUG:boto:Params: {} 2023-09-22T21:49:08.757 DEBUG:boto:Token: None 2023-09-22T21:49:08.757 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.758 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RlfWkMdMrdxHIcmh7cCnt+YDjLg= 2023-09-22T21:49:08.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RlfWkMdMrdxHIcmh7cCnt+YDjLg=', 'Content-Length': '0'} 2023-09-22T21:49:08.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f500e52c479613b-00650e0bd4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.760 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:49:08.761 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:49:08.761 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.761 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.761 DEBUG:boto:Method: GET 2023-09-22T21:49:08.761 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.761 DEBUG:boto:Data: 2023-09-22T21:49:08.761 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.761 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:08.761 DEBUG:boto:Port: 8000 2023-09-22T21:49:08.762 DEBUG:boto:Params: {} 2023-09-22T21:49:08.762 DEBUG:boto:Token: None 2023-09-22T21:49:08.762 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:08.762 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RlfWkMdMrdxHIcmh7cCnt+YDjLg= 2023-09-22T21:49:08.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RlfWkMdMrdxHIcmh7cCnt+YDjLg=', 'Content-Length': '0'} 2023-09-22T21:49:08.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dddc21edc7020cdb-00650e0bd4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.810 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:08.811 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:08.811 DEBUG:boto:path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:49:08.811 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:49:08.813 DEBUG:boto:Method: GET 2023-09-22T21:49:08.813 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj145 2023-09-22T21:49:08.813 DEBUG:boto:Data: 2023-09-22T21:49:08.813 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.813 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:08.813 DEBUG:boto:Port: 8001 2023-09-22T21:49:08.813 DEBUG:boto:Params: {} 2023-09-22T21:49:08.814 DEBUG:boto:Token: None 2023-09-22T21:49:08.814 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/ 2023-09-22T21:49:08.814 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs= 2023-09-22T21:49:08.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs=', 'Content-Length': '0'} 2023-09-22T21:49:08.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfab553e9428b301-00650e0bd4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.819 DEBUG:boto:b'ffzdqu-59obj1451000falseobj1452023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:08.820 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:08.820 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:08.820 DEBUG:boto:path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:49:08.820 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:49:08.820 DEBUG:boto:Method: GET 2023-09-22T21:49:08.820 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj145 2023-09-22T21:49:08.820 DEBUG:boto:Data: 2023-09-22T21:49:08.820 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.820 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:08.821 DEBUG:boto:Port: 8000 2023-09-22T21:49:08.821 DEBUG:boto:Params: {} 2023-09-22T21:49:08.821 DEBUG:boto:Token: None 2023-09-22T21:49:08.821 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/ 2023-09-22T21:49:08.821 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs= 2023-09-22T21:49:08.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs=', 'Content-Length': '0'} 2023-09-22T21:49:08.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054b977a2e5d65b9e-00650e0bd4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.830 DEBUG:boto:b'ffzdqu-59obj1451000falseobj1452023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:08.831 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:49:08.831 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:49:08.831 DEBUG:boto:Method: HEAD 2023-09-22T21:49:08.831 DEBUG:boto:Path: /ffzdqu-59/obj145 2023-09-22T21:49:08.831 DEBUG:boto:Data: 2023-09-22T21:49:08.831 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.831 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:08.832 DEBUG:boto:Port: 8001 2023-09-22T21:49:08.832 DEBUG:boto:Params: {} 2023-09-22T21:49:08.832 DEBUG:boto:Token: None 2023-09-22T21:49:08.832 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj145 2023-09-22T21:49:08.832 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gG3LtgaTQsGJ3e5YvalCw3CY9g8= 2023-09-22T21:49:08.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gG3LtgaTQsGJ3e5YvalCw3CY9g8=', 'Content-Length': '0'} 2023-09-22T21:49:08.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c9d183028ca2e85-00650e0bd4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.835 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:49:08.835 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:49:08.835 DEBUG:boto:Method: HEAD 2023-09-22T21:49:08.835 DEBUG:boto:Path: /ffzdqu-59/obj145 2023-09-22T21:49:08.836 DEBUG:boto:Data: 2023-09-22T21:49:08.836 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.836 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:08.836 DEBUG:boto:Port: 8000 2023-09-22T21:49:08.836 DEBUG:boto:Params: {} 2023-09-22T21:49:08.836 DEBUG:boto:Token: None 2023-09-22T21:49:08.836 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj145 2023-09-22T21:49:08.836 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gG3LtgaTQsGJ3e5YvalCw3CY9g8= 2023-09-22T21:49:08.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gG3LtgaTQsGJ3e5YvalCw3CY9g8=', 'Content-Length': '0'} 2023-09-22T21:49:08.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e422371174159bb3-00650e0bd4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-22T21:49:08.840 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:49:08.840 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:49:08.840 DEBUG:boto:Method: GET 2023-09-22T21:49:08.840 DEBUG:boto:Path: /ffzdqu-59/obj145 2023-09-22T21:49:08.840 DEBUG:boto:Data: 2023-09-22T21:49:08.840 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.840 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:08.840 DEBUG:boto:Port: 8001 2023-09-22T21:49:08.840 DEBUG:boto:Params: {} 2023-09-22T21:49:08.840 DEBUG:boto:Token: None 2023-09-22T21:49:08.841 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj145 2023-09-22T21:49:08.841 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mdbvRK2Csy6MVkDUcE2SDLimmv4= 2023-09-22T21:49:08.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mdbvRK2Csy6MVkDUcE2SDLimmv4=', 'Content-Length': '0'} 2023-09-22T21:49:08.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b878e1662e4b658a-00650e0bd4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.890 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:49:08.891 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:49:08.891 DEBUG:boto:Method: GET 2023-09-22T21:49:08.891 DEBUG:boto:Path: /ffzdqu-59/obj145 2023-09-22T21:49:08.891 DEBUG:boto:Data: 2023-09-22T21:49:08.891 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.891 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:08.891 DEBUG:boto:Port: 8000 2023-09-22T21:49:08.891 DEBUG:boto:Params: {} 2023-09-22T21:49:08.891 DEBUG:boto:Token: None 2023-09-22T21:49:08.892 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj145 2023-09-22T21:49:08.892 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mdbvRK2Csy6MVkDUcE2SDLimmv4= 2023-09-22T21:49:08.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mdbvRK2Csy6MVkDUcE2SDLimmv4=', 'Content-Length': '0'} 2023-09-22T21:49:08.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7311f4bf73aaf3f-00650e0bd4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.895 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:08.895 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:08.895 DEBUG:boto:path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:49:08.895 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:49:08.896 DEBUG:boto:Method: GET 2023-09-22T21:49:08.896 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj145 2023-09-22T21:49:08.896 DEBUG:boto:Data: 2023-09-22T21:49:08.896 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.896 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:08.896 DEBUG:boto:Port: 8001 2023-09-22T21:49:08.896 DEBUG:boto:Params: {} 2023-09-22T21:49:08.896 DEBUG:boto:Token: None 2023-09-22T21:49:08.897 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/ 2023-09-22T21:49:08.897 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs= 2023-09-22T21:49:08.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs=', 'Content-Length': '0'} 2023-09-22T21:49:08.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063037086e195082e-00650e0bd4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.901 DEBUG:boto:b'ffzdqu-59obj1451000falseobj1452023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:08.902 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:08.902 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:08.902 DEBUG:boto:path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:49:08.902 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:49:08.902 DEBUG:boto:Method: GET 2023-09-22T21:49:08.903 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj145 2023-09-22T21:49:08.903 DEBUG:boto:Data: 2023-09-22T21:49:08.903 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.903 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:08.903 DEBUG:boto:Port: 8000 2023-09-22T21:49:08.903 DEBUG:boto:Params: {} 2023-09-22T21:49:08.903 DEBUG:boto:Token: None 2023-09-22T21:49:08.904 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/ 2023-09-22T21:49:08.904 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs= 2023-09-22T21:49:08.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs=', 'Content-Length': '0'} 2023-09-22T21:49:08.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea3ff54971054629-00650e0bd4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.954 DEBUG:boto:b'ffzdqu-59obj1451000falseobj1452023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:08.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-09-22T21:49:08.955 DEBUG:boto:path=/ffzdqu-59/obj148 2023-09-22T21:49:08.955 DEBUG:boto:auth_path=/ffzdqu-59/obj148 2023-09-22T21:49:08.955 DEBUG:boto:path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.955 DEBUG:boto:auth_path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.955 DEBUG:boto:Method: GET 2023-09-22T21:49:08.955 DEBUG:boto:Path: /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.955 DEBUG:boto:Data: 2023-09-22T21:49:08.956 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.956 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:08.956 DEBUG:boto:Port: 8001 2023-09-22T21:49:08.956 DEBUG:boto:Params: {} 2023-09-22T21:49:08.956 DEBUG:boto:Token: None 2023-09-22T21:49:08.956 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.956 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nW1WrxrG4hjHHsf1LpvbbJvol18= 2023-09-22T21:49:08.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nW1WrxrG4hjHHsf1LpvbbJvol18=', 'Content-Length': '0'} 2023-09-22T21:49:08.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6eeda061a9fe728-00650e0bd4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.960 DEBUG:boto:path=/ffzdqu-59/obj148 2023-09-22T21:49:08.960 DEBUG:boto:auth_path=/ffzdqu-59/obj148 2023-09-22T21:49:08.960 DEBUG:boto:path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.960 DEBUG:boto:auth_path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.961 DEBUG:boto:Method: GET 2023-09-22T21:49:08.961 DEBUG:boto:Path: /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.961 DEBUG:boto:Data: 2023-09-22T21:49:08.961 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.961 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:08.961 DEBUG:boto:Port: 8000 2023-09-22T21:49:08.961 DEBUG:boto:Params: {} 2023-09-22T21:49:08.961 DEBUG:boto:Token: None 2023-09-22T21:49:08.961 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.962 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nW1WrxrG4hjHHsf1LpvbbJvol18= 2023-09-22T21:49:08.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nW1WrxrG4hjHHsf1LpvbbJvol18=', 'Content-Length': '0'} 2023-09-22T21:49:08.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025de0fde144fda4f-00650e0bd4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.965 DEBUG:boto:path=/ffzdqu-59/obj148 2023-09-22T21:49:08.965 DEBUG:boto:auth_path=/ffzdqu-59/obj148 2023-09-22T21:49:08.965 DEBUG:boto:path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.965 DEBUG:boto:auth_path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.966 DEBUG:boto:Method: HEAD 2023-09-22T21:49:08.966 DEBUG:boto:Path: /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.966 DEBUG:boto:Data: 2023-09-22T21:49:08.966 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.966 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:08.966 DEBUG:boto:Port: 8001 2023-09-22T21:49:08.966 DEBUG:boto:Params: {} 2023-09-22T21:49:08.966 DEBUG:boto:Token: None 2023-09-22T21:49:08.966 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.967 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ku+bOpPPf/1yCTsjobDYM/fOb/w= 2023-09-22T21:49:08.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ku+bOpPPf/1yCTsjobDYM/fOb/w=', 'Content-Length': '0'} 2023-09-22T21:49:08.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5f42bd52b06061b-00650e0bd4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.969 DEBUG:boto:path=/ffzdqu-59/obj148 2023-09-22T21:49:08.969 DEBUG:boto:auth_path=/ffzdqu-59/obj148 2023-09-22T21:49:08.970 DEBUG:boto:path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.970 DEBUG:boto:auth_path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.970 DEBUG:boto:Method: HEAD 2023-09-22T21:49:08.970 DEBUG:boto:Path: /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.970 DEBUG:boto:Data: 2023-09-22T21:49:08.970 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.970 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:08.970 DEBUG:boto:Port: 8000 2023-09-22T21:49:08.970 DEBUG:boto:Params: {} 2023-09-22T21:49:08.971 DEBUG:boto:Token: None 2023-09-22T21:49:08.971 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.971 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ku+bOpPPf/1yCTsjobDYM/fOb/w= 2023-09-22T21:49:08.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ku+bOpPPf/1yCTsjobDYM/fOb/w=', 'Content-Length': '0'} 2023-09-22T21:49:08.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000898be02456238f08-00650e0bd4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-09-22T21:49:08.974 DEBUG:boto:path=/ffzdqu-59/obj148 2023-09-22T21:49:08.974 DEBUG:boto:auth_path=/ffzdqu-59/obj148 2023-09-22T21:49:08.974 DEBUG:boto:path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.974 DEBUG:boto:auth_path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.974 DEBUG:boto:Method: GET 2023-09-22T21:49:08.974 DEBUG:boto:Path: /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.975 DEBUG:boto:Data: 2023-09-22T21:49:08.975 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.975 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:08.975 DEBUG:boto:Port: 8001 2023-09-22T21:49:08.975 DEBUG:boto:Params: {} 2023-09-22T21:49:08.975 DEBUG:boto:Token: None 2023-09-22T21:49:08.975 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.975 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nW1WrxrG4hjHHsf1LpvbbJvol18= 2023-09-22T21:49:08.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nW1WrxrG4hjHHsf1LpvbbJvol18=', 'Content-Length': '0'} 2023-09-22T21:49:08.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce842032ceaecc10-00650e0bd4-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.978 DEBUG:boto:path=/ffzdqu-59/obj148 2023-09-22T21:49:08.978 DEBUG:boto:auth_path=/ffzdqu-59/obj148 2023-09-22T21:49:08.979 DEBUG:boto:path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.979 DEBUG:boto:auth_path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.979 DEBUG:boto:Method: GET 2023-09-22T21:49:08.979 DEBUG:boto:Path: /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.979 DEBUG:boto:Data: 2023-09-22T21:49:08.979 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.979 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:08.979 DEBUG:boto:Port: 8000 2023-09-22T21:49:08.979 DEBUG:boto:Params: {} 2023-09-22T21:49:08.979 DEBUG:boto:Token: None 2023-09-22T21:49:08.980 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:08.980 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nW1WrxrG4hjHHsf1LpvbbJvol18= 2023-09-22T21:49:08.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nW1WrxrG4hjHHsf1LpvbbJvol18=', 'Content-Length': '0'} 2023-09-22T21:49:08.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007871b4d10066fb33-00650e0bd4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.983 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:08.983 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:08.983 DEBUG:boto:path=/ffzdqu-59/?prefix=obj148 2023-09-22T21:49:08.983 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj148 2023-09-22T21:49:08.983 DEBUG:boto:Method: GET 2023-09-22T21:49:08.984 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj148 2023-09-22T21:49:08.984 DEBUG:boto:Data: 2023-09-22T21:49:08.984 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.984 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:08.984 DEBUG:boto:Port: 8001 2023-09-22T21:49:08.984 DEBUG:boto:Params: {} 2023-09-22T21:49:08.984 DEBUG:boto:Token: None 2023-09-22T21:49:08.984 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/ 2023-09-22T21:49:08.985 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs= 2023-09-22T21:49:08.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs=', 'Content-Length': '0'} 2023-09-22T21:49:08.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000778a1ea3bb9c8a44-00650e0bd4-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:08.989 DEBUG:boto:b'ffzdqu-59obj1481000falseobj1482023-09-22T21:47:16.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:08.989 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:08.990 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:08.990 DEBUG:boto:path=/ffzdqu-59/?prefix=obj148 2023-09-22T21:49:08.990 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj148 2023-09-22T21:49:08.990 DEBUG:boto:Method: GET 2023-09-22T21:49:08.990 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj148 2023-09-22T21:49:08.990 DEBUG:boto:Data: 2023-09-22T21:49:08.990 DEBUG:boto:Headers: {} 2023-09-22T21:49:08.990 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:08.990 DEBUG:boto:Port: 8000 2023-09-22T21:49:08.990 DEBUG:boto:Params: {} 2023-09-22T21:49:08.991 DEBUG:boto:Token: None 2023-09-22T21:49:08.991 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:08 GMT /ffzdqu-59/ 2023-09-22T21:49:08.991 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs= 2023-09-22T21:49:08.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wPsXGlbYVBQaVVMq9ax/sfrm2Xs=', 'Content-Length': '0'} 2023-09-22T21:49:09.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052d18111a59c9f1a-00650e0bd4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.002 DEBUG:boto:b'ffzdqu-59obj1481000falseobj1482023-09-22T21:47:16.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:09.002 DEBUG:boto:path=/ffzdqu-59/obj148 2023-09-22T21:49:09.003 DEBUG:boto:auth_path=/ffzdqu-59/obj148 2023-09-22T21:49:09.003 DEBUG:boto:Method: HEAD 2023-09-22T21:49:09.003 DEBUG:boto:Path: /ffzdqu-59/obj148 2023-09-22T21:49:09.003 DEBUG:boto:Data: 2023-09-22T21:49:09.003 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.003 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.003 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.003 DEBUG:boto:Params: {} 2023-09-22T21:49:09.003 DEBUG:boto:Token: None 2023-09-22T21:49:09.003 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj148 2023-09-22T21:49:09.004 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ezTXBJhLaKgDrFaeXi1fbgR3dn0= 2023-09-22T21:49:09.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ezTXBJhLaKgDrFaeXi1fbgR3dn0=', 'Content-Length': '0'} 2023-09-22T21:49:09.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000686ad24b587ede06-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.007 DEBUG:boto:path=/ffzdqu-59/obj148 2023-09-22T21:49:09.007 DEBUG:boto:auth_path=/ffzdqu-59/obj148 2023-09-22T21:49:09.007 DEBUG:boto:Method: HEAD 2023-09-22T21:49:09.007 DEBUG:boto:Path: /ffzdqu-59/obj148 2023-09-22T21:49:09.007 DEBUG:boto:Data: 2023-09-22T21:49:09.007 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.007 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.007 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.007 DEBUG:boto:Params: {} 2023-09-22T21:49:09.008 DEBUG:boto:Token: None 2023-09-22T21:49:09.008 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj148 2023-09-22T21:49:09.008 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ezTXBJhLaKgDrFaeXi1fbgR3dn0= 2023-09-22T21:49:09.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ezTXBJhLaKgDrFaeXi1fbgR3dn0=', 'Content-Length': '0'} 2023-09-22T21:49:09.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bdc6c1c590a33f7-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-09-22T21:49:09.011 DEBUG:boto:path=/ffzdqu-59/obj148 2023-09-22T21:49:09.011 DEBUG:boto:auth_path=/ffzdqu-59/obj148 2023-09-22T21:49:09.011 DEBUG:boto:Method: GET 2023-09-22T21:49:09.011 DEBUG:boto:Path: /ffzdqu-59/obj148 2023-09-22T21:49:09.011 DEBUG:boto:Data: 2023-09-22T21:49:09.011 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.012 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.012 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.012 DEBUG:boto:Params: {} 2023-09-22T21:49:09.012 DEBUG:boto:Token: None 2023-09-22T21:49:09.012 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj148 2023-09-22T21:49:09.012 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SmZS4Ebaq+Md4+CqfDbkIXbHnYo= 2023-09-22T21:49:09.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SmZS4Ebaq+Md4+CqfDbkIXbHnYo=', 'Content-Length': '0'} 2023-09-22T21:49:09.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bea5e82b81c62fd1-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.062 DEBUG:boto:path=/ffzdqu-59/obj148 2023-09-22T21:49:09.063 DEBUG:boto:auth_path=/ffzdqu-59/obj148 2023-09-22T21:49:09.063 DEBUG:boto:Method: GET 2023-09-22T21:49:09.063 DEBUG:boto:Path: /ffzdqu-59/obj148 2023-09-22T21:49:09.063 DEBUG:boto:Data: 2023-09-22T21:49:09.063 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.063 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.063 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.063 DEBUG:boto:Params: {} 2023-09-22T21:49:09.063 DEBUG:boto:Token: None 2023-09-22T21:49:09.064 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj148 2023-09-22T21:49:09.064 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SmZS4Ebaq+Md4+CqfDbkIXbHnYo= 2023-09-22T21:49:09.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SmZS4Ebaq+Md4+CqfDbkIXbHnYo=', 'Content-Length': '0'} 2023-09-22T21:49:09.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5fe3924d2bd8da0-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.067 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:09.067 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:09.067 DEBUG:boto:path=/ffzdqu-59/?prefix=obj148 2023-09-22T21:49:09.067 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj148 2023-09-22T21:49:09.067 DEBUG:boto:Method: GET 2023-09-22T21:49:09.068 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj148 2023-09-22T21:49:09.068 DEBUG:boto:Data: 2023-09-22T21:49:09.068 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.068 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.068 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.068 DEBUG:boto:Params: {} 2023-09-22T21:49:09.068 DEBUG:boto:Token: None 2023-09-22T21:49:09.068 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/ 2023-09-22T21:49:09.069 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4= 2023-09-22T21:49:09.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4=', 'Content-Length': '0'} 2023-09-22T21:49:09.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002378cd5691ff462f-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.074 DEBUG:boto:b'ffzdqu-59obj1481000falseobj1482023-09-22T21:47:16.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:09.075 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:09.075 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:09.075 DEBUG:boto:path=/ffzdqu-59/?prefix=obj148 2023-09-22T21:49:09.075 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj148 2023-09-22T21:49:09.075 DEBUG:boto:Method: GET 2023-09-22T21:49:09.075 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj148 2023-09-22T21:49:09.075 DEBUG:boto:Data: 2023-09-22T21:49:09.075 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.076 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.076 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.076 DEBUG:boto:Params: {} 2023-09-22T21:49:09.076 DEBUG:boto:Token: None 2023-09-22T21:49:09.076 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/ 2023-09-22T21:49:09.076 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4= 2023-09-22T21:49:09.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4=', 'Content-Length': '0'} 2023-09-22T21:49:09.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099e77aaa70b11e31-00650e0bd5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.086 DEBUG:boto:b'ffzdqu-59obj1481000falseobj1482023-09-22T21:47:16.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:09.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-22T21:49:09.087 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:49:09.087 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:49:09.087 DEBUG:boto:path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.087 DEBUG:boto:auth_path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.087 DEBUG:boto:Method: GET 2023-09-22T21:49:09.087 DEBUG:boto:Path: /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.087 DEBUG:boto:Data: 2023-09-22T21:49:09.087 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.088 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.088 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.088 DEBUG:boto:Params: {} 2023-09-22T21:49:09.088 DEBUG:boto:Token: None 2023-09-22T21:49:09.088 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.088 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Zt6j1jbK8RXmHniRJnUvJ7n+zj8= 2023-09-22T21:49:09.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Zt6j1jbK8RXmHniRJnUvJ7n+zj8=', 'Content-Length': '0'} 2023-09-22T21:49:09.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc76591789d09c63-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.091 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:49:09.091 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:49:09.091 DEBUG:boto:path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.091 DEBUG:boto:auth_path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.092 DEBUG:boto:Method: GET 2023-09-22T21:49:09.092 DEBUG:boto:Path: /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.092 DEBUG:boto:Data: 2023-09-22T21:49:09.092 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.092 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.092 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.092 DEBUG:boto:Params: {} 2023-09-22T21:49:09.092 DEBUG:boto:Token: None 2023-09-22T21:49:09.092 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.093 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Zt6j1jbK8RXmHniRJnUvJ7n+zj8= 2023-09-22T21:49:09.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Zt6j1jbK8RXmHniRJnUvJ7n+zj8=', 'Content-Length': '0'} 2023-09-22T21:49:09.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a6a8bfaa86e444d-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.096 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:49:09.096 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:49:09.096 DEBUG:boto:path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.096 DEBUG:boto:auth_path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.096 DEBUG:boto:Method: HEAD 2023-09-22T21:49:09.096 DEBUG:boto:Path: /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.096 DEBUG:boto:Data: 2023-09-22T21:49:09.096 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.096 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.096 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.097 DEBUG:boto:Params: {} 2023-09-22T21:49:09.097 DEBUG:boto:Token: None 2023-09-22T21:49:09.097 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.097 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Nno3IZBTifyWDlnAXcuq1JApwLw= 2023-09-22T21:49:09.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Nno3IZBTifyWDlnAXcuq1JApwLw=', 'Content-Length': '0'} 2023-09-22T21:49:09.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc78551d4e5a25a9-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.100 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:49:09.100 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:49:09.100 DEBUG:boto:path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.100 DEBUG:boto:auth_path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.100 DEBUG:boto:Method: HEAD 2023-09-22T21:49:09.100 DEBUG:boto:Path: /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.100 DEBUG:boto:Data: 2023-09-22T21:49:09.101 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.101 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.101 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.101 DEBUG:boto:Params: {} 2023-09-22T21:49:09.101 DEBUG:boto:Token: None 2023-09-22T21:49:09.101 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.101 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Nno3IZBTifyWDlnAXcuq1JApwLw= 2023-09-22T21:49:09.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Nno3IZBTifyWDlnAXcuq1JApwLw=', 'Content-Length': '0'} 2023-09-22T21:49:09.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0b6ea12aea9e34c-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-22T21:49:09.105 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:49:09.105 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:49:09.105 DEBUG:boto:path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.105 DEBUG:boto:auth_path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.105 DEBUG:boto:Method: GET 2023-09-22T21:49:09.105 DEBUG:boto:Path: /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.105 DEBUG:boto:Data: 2023-09-22T21:49:09.105 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.106 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.106 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.106 DEBUG:boto:Params: {} 2023-09-22T21:49:09.106 DEBUG:boto:Token: None 2023-09-22T21:49:09.106 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.106 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Zt6j1jbK8RXmHniRJnUvJ7n+zj8= 2023-09-22T21:49:09.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Zt6j1jbK8RXmHniRJnUvJ7n+zj8=', 'Content-Length': '0'} 2023-09-22T21:49:09.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6e6634043668189-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.150 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:49:09.150 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:49:09.150 DEBUG:boto:path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.151 DEBUG:boto:auth_path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.151 DEBUG:boto:Method: GET 2023-09-22T21:49:09.151 DEBUG:boto:Path: /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.151 DEBUG:boto:Data: 2023-09-22T21:49:09.151 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.151 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.151 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.151 DEBUG:boto:Params: {} 2023-09-22T21:49:09.151 DEBUG:boto:Token: None 2023-09-22T21:49:09.152 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:09.152 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Zt6j1jbK8RXmHniRJnUvJ7n+zj8= 2023-09-22T21:49:09.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Zt6j1jbK8RXmHniRJnUvJ7n+zj8=', 'Content-Length': '0'} 2023-09-22T21:49:09.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4192e18d1e09217-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.155 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:09.155 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:09.156 DEBUG:boto:path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:49:09.156 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:49:09.156 DEBUG:boto:Method: GET 2023-09-22T21:49:09.156 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj155 2023-09-22T21:49:09.156 DEBUG:boto:Data: 2023-09-22T21:49:09.156 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.156 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.156 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.156 DEBUG:boto:Params: {} 2023-09-22T21:49:09.157 DEBUG:boto:Token: None 2023-09-22T21:49:09.157 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/ 2023-09-22T21:49:09.157 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4= 2023-09-22T21:49:09.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4=', 'Content-Length': '0'} 2023-09-22T21:49:09.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4ef27600e4b3eb6-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.162 DEBUG:boto:b'ffzdqu-59obj1551000falseobj1552023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:09.163 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:09.163 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:09.163 DEBUG:boto:path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:49:09.163 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:49:09.163 DEBUG:boto:Method: GET 2023-09-22T21:49:09.163 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj155 2023-09-22T21:49:09.163 DEBUG:boto:Data: 2023-09-22T21:49:09.163 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.163 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.164 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.164 DEBUG:boto:Params: {} 2023-09-22T21:49:09.164 DEBUG:boto:Token: None 2023-09-22T21:49:09.164 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/ 2023-09-22T21:49:09.164 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4= 2023-09-22T21:49:09.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4=', 'Content-Length': '0'} 2023-09-22T21:49:09.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006af43d96661f86f4-00650e0bd5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.174 DEBUG:boto:b'ffzdqu-59obj1551000falseobj1552023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:09.175 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:49:09.175 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:49:09.175 DEBUG:boto:Method: HEAD 2023-09-22T21:49:09.175 DEBUG:boto:Path: /ffzdqu-59/obj155 2023-09-22T21:49:09.175 DEBUG:boto:Data: 2023-09-22T21:49:09.175 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.176 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.176 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.176 DEBUG:boto:Params: {} 2023-09-22T21:49:09.176 DEBUG:boto:Token: None 2023-09-22T21:49:09.176 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj155 2023-09-22T21:49:09.176 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LB5iqVJuMd2dBFW8hLl11tg2ItY= 2023-09-22T21:49:09.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LB5iqVJuMd2dBFW8hLl11tg2ItY=', 'Content-Length': '0'} 2023-09-22T21:49:09.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000566c2fc657b9c21f-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.179 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:49:09.179 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:49:09.179 DEBUG:boto:Method: HEAD 2023-09-22T21:49:09.179 DEBUG:boto:Path: /ffzdqu-59/obj155 2023-09-22T21:49:09.180 DEBUG:boto:Data: 2023-09-22T21:49:09.180 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.180 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.180 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.180 DEBUG:boto:Params: {} 2023-09-22T21:49:09.180 DEBUG:boto:Token: None 2023-09-22T21:49:09.180 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj155 2023-09-22T21:49:09.180 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LB5iqVJuMd2dBFW8hLl11tg2ItY= 2023-09-22T21:49:09.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LB5iqVJuMd2dBFW8hLl11tg2ItY=', 'Content-Length': '0'} 2023-09-22T21:49:09.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ad6c7974a39ac7a-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-22T21:49:09.183 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:49:09.184 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:49:09.184 DEBUG:boto:Method: GET 2023-09-22T21:49:09.184 DEBUG:boto:Path: /ffzdqu-59/obj155 2023-09-22T21:49:09.184 DEBUG:boto:Data: 2023-09-22T21:49:09.184 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.184 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.184 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.184 DEBUG:boto:Params: {} 2023-09-22T21:49:09.184 DEBUG:boto:Token: None 2023-09-22T21:49:09.185 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj155 2023-09-22T21:49:09.185 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YrzPPNrtAE0CrBulx4kw0VLIgJQ= 2023-09-22T21:49:09.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YrzPPNrtAE0CrBulx4kw0VLIgJQ=', 'Content-Length': '0'} 2023-09-22T21:49:09.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f895399e37a2946f-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.234 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:49:09.235 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:49:09.235 DEBUG:boto:Method: GET 2023-09-22T21:49:09.235 DEBUG:boto:Path: /ffzdqu-59/obj155 2023-09-22T21:49:09.235 DEBUG:boto:Data: 2023-09-22T21:49:09.235 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.235 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.235 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.235 DEBUG:boto:Params: {} 2023-09-22T21:49:09.236 DEBUG:boto:Token: None 2023-09-22T21:49:09.236 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj155 2023-09-22T21:49:09.236 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YrzPPNrtAE0CrBulx4kw0VLIgJQ= 2023-09-22T21:49:09.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YrzPPNrtAE0CrBulx4kw0VLIgJQ=', 'Content-Length': '0'} 2023-09-22T21:49:09.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050d21cb8e17051f9-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.239 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:09.240 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:09.240 DEBUG:boto:path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:49:09.240 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:49:09.240 DEBUG:boto:Method: GET 2023-09-22T21:49:09.240 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj155 2023-09-22T21:49:09.240 DEBUG:boto:Data: 2023-09-22T21:49:09.240 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.240 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.240 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.241 DEBUG:boto:Params: {} 2023-09-22T21:49:09.241 DEBUG:boto:Token: None 2023-09-22T21:49:09.241 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/ 2023-09-22T21:49:09.241 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4= 2023-09-22T21:49:09.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4=', 'Content-Length': '0'} 2023-09-22T21:49:09.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094bb634e61fe444b-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.246 DEBUG:boto:b'ffzdqu-59obj1551000falseobj1552023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:09.247 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:09.247 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:09.247 DEBUG:boto:path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:49:09.247 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:49:09.247 DEBUG:boto:Method: GET 2023-09-22T21:49:09.247 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj155 2023-09-22T21:49:09.247 DEBUG:boto:Data: 2023-09-22T21:49:09.247 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.247 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.248 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.248 DEBUG:boto:Params: {} 2023-09-22T21:49:09.248 DEBUG:boto:Token: None 2023-09-22T21:49:09.248 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/ 2023-09-22T21:49:09.248 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4= 2023-09-22T21:49:09.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4=', 'Content-Length': '0'} 2023-09-22T21:49:09.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000775328570baff0c9-00650e0bd5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.306 DEBUG:boto:b'ffzdqu-59obj1551000falseobj1552023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:09.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:49:09.307 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:49:09.307 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:49:09.307 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.307 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.307 DEBUG:boto:Method: GET 2023-09-22T21:49:09.307 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.307 DEBUG:boto:Data: 2023-09-22T21:49:09.307 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.308 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.308 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.308 DEBUG:boto:Params: {} 2023-09-22T21:49:09.308 DEBUG:boto:Token: None 2023-09-22T21:49:09.308 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.308 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eaZFsYCmOuDNHCz5MzO35Le87BE= 2023-09-22T21:49:09.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eaZFsYCmOuDNHCz5MzO35Le87BE=', 'Content-Length': '0'} 2023-09-22T21:49:09.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036e909501f215cbe-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.312 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:49:09.312 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:49:09.312 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.312 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.312 DEBUG:boto:Method: GET 2023-09-22T21:49:09.312 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.312 DEBUG:boto:Data: 2023-09-22T21:49:09.312 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.312 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.312 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.313 DEBUG:boto:Params: {} 2023-09-22T21:49:09.313 DEBUG:boto:Token: None 2023-09-22T21:49:09.313 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.313 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eaZFsYCmOuDNHCz5MzO35Le87BE= 2023-09-22T21:49:09.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eaZFsYCmOuDNHCz5MzO35Le87BE=', 'Content-Length': '0'} 2023-09-22T21:49:09.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6910e928065f64d-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.316 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:49:09.316 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:49:09.316 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.316 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.317 DEBUG:boto:Method: HEAD 2023-09-22T21:49:09.317 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.317 DEBUG:boto:Data: 2023-09-22T21:49:09.317 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.317 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.317 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.317 DEBUG:boto:Params: {} 2023-09-22T21:49:09.317 DEBUG:boto:Token: None 2023-09-22T21:49:09.317 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.318 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZrCYlBiGZX2n32gpHWzN+f1RiAY= 2023-09-22T21:49:09.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZrCYlBiGZX2n32gpHWzN+f1RiAY=', 'Content-Length': '0'} 2023-09-22T21:49:09.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a35dfccd518aaf04-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.321 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:49:09.321 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:49:09.321 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.321 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.321 DEBUG:boto:Method: HEAD 2023-09-22T21:49:09.321 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.321 DEBUG:boto:Data: 2023-09-22T21:49:09.321 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.322 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.322 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.322 DEBUG:boto:Params: {} 2023-09-22T21:49:09.322 DEBUG:boto:Token: None 2023-09-22T21:49:09.322 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.322 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZrCYlBiGZX2n32gpHWzN+f1RiAY= 2023-09-22T21:49:09.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZrCYlBiGZX2n32gpHWzN+f1RiAY=', 'Content-Length': '0'} 2023-09-22T21:49:09.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fca53bed1571b070-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:49:09.325 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:49:09.326 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:49:09.326 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.326 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.326 DEBUG:boto:Method: GET 2023-09-22T21:49:09.326 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.326 DEBUG:boto:Data: 2023-09-22T21:49:09.326 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.326 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.326 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.326 DEBUG:boto:Params: {} 2023-09-22T21:49:09.327 DEBUG:boto:Token: None 2023-09-22T21:49:09.327 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.327 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eaZFsYCmOuDNHCz5MzO35Le87BE= 2023-09-22T21:49:09.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eaZFsYCmOuDNHCz5MzO35Le87BE=', 'Content-Length': '0'} 2023-09-22T21:49:09.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc5a809cc1fd595e-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.330 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:49:09.330 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:49:09.330 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.331 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.331 DEBUG:boto:Method: GET 2023-09-22T21:49:09.331 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.331 DEBUG:boto:Data: 2023-09-22T21:49:09.331 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.331 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.331 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.331 DEBUG:boto:Params: {} 2023-09-22T21:49:09.331 DEBUG:boto:Token: None 2023-09-22T21:49:09.331 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:09.332 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eaZFsYCmOuDNHCz5MzO35Le87BE= 2023-09-22T21:49:09.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eaZFsYCmOuDNHCz5MzO35Le87BE=', 'Content-Length': '0'} 2023-09-22T21:49:09.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029afd2ff05cc28c4-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.335 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:09.335 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:09.335 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:49:09.335 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:49:09.335 DEBUG:boto:Method: GET 2023-09-22T21:49:09.335 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:49:09.335 DEBUG:boto:Data: 2023-09-22T21:49:09.335 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.335 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.335 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.336 DEBUG:boto:Params: {} 2023-09-22T21:49:09.336 DEBUG:boto:Token: None 2023-09-22T21:49:09.336 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/ 2023-09-22T21:49:09.336 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4= 2023-09-22T21:49:09.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4=', 'Content-Length': '0'} 2023-09-22T21:49:09.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d1ab1c8d2b307ab-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.341 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:09.341 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:09.342 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:09.342 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:49:09.342 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:49:09.342 DEBUG:boto:Method: GET 2023-09-22T21:49:09.342 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:49:09.342 DEBUG:boto:Data: 2023-09-22T21:49:09.342 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.342 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.342 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.343 DEBUG:boto:Params: {} 2023-09-22T21:49:09.343 DEBUG:boto:Token: None 2023-09-22T21:49:09.343 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/ 2023-09-22T21:49:09.343 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4= 2023-09-22T21:49:09.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4=', 'Content-Length': '0'} 2023-09-22T21:49:09.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e23750e0b263fc91-00650e0bd5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.394 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:09.395 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:49:09.395 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:49:09.395 DEBUG:boto:Method: HEAD 2023-09-22T21:49:09.395 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:49:09.395 DEBUG:boto:Data: 2023-09-22T21:49:09.395 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.396 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.396 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.396 DEBUG:boto:Params: {} 2023-09-22T21:49:09.396 DEBUG:boto:Token: None 2023-09-22T21:49:09.396 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj17 2023-09-22T21:49:09.396 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FUWnZT97s/SUl3BBkHIc8pxw0l0= 2023-09-22T21:49:09.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FUWnZT97s/SUl3BBkHIc8pxw0l0=', 'Content-Length': '0'} 2023-09-22T21:49:09.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fafaba4d59d3567a-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.400 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:49:09.400 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:49:09.400 DEBUG:boto:Method: HEAD 2023-09-22T21:49:09.400 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:49:09.400 DEBUG:boto:Data: 2023-09-22T21:49:09.400 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.400 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.400 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.400 DEBUG:boto:Params: {} 2023-09-22T21:49:09.401 DEBUG:boto:Token: None 2023-09-22T21:49:09.401 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj17 2023-09-22T21:49:09.401 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FUWnZT97s/SUl3BBkHIc8pxw0l0= 2023-09-22T21:49:09.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FUWnZT97s/SUl3BBkHIc8pxw0l0=', 'Content-Length': '0'} 2023-09-22T21:49:09.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7a5664f992cfaa1-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:49:09.404 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:49:09.404 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:49:09.404 DEBUG:boto:Method: GET 2023-09-22T21:49:09.404 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:49:09.405 DEBUG:boto:Data: 2023-09-22T21:49:09.405 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.405 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.405 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.405 DEBUG:boto:Params: {} 2023-09-22T21:49:09.405 DEBUG:boto:Token: None 2023-09-22T21:49:09.405 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj17 2023-09-22T21:49:09.405 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7w//t0BC4ktsjsqcloreNks3MgY= 2023-09-22T21:49:09.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7w//t0BC4ktsjsqcloreNks3MgY=', 'Content-Length': '0'} 2023-09-22T21:49:09.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020608d614442258b-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.450 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:49:09.451 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:49:09.451 DEBUG:boto:Method: GET 2023-09-22T21:49:09.451 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:49:09.451 DEBUG:boto:Data: 2023-09-22T21:49:09.451 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.451 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.451 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.451 DEBUG:boto:Params: {} 2023-09-22T21:49:09.452 DEBUG:boto:Token: None 2023-09-22T21:49:09.452 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj17 2023-09-22T21:49:09.452 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7w//t0BC4ktsjsqcloreNks3MgY= 2023-09-22T21:49:09.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7w//t0BC4ktsjsqcloreNks3MgY=', 'Content-Length': '0'} 2023-09-22T21:49:09.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f30cde3ad774397f-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.455 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:09.455 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:09.456 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:49:09.456 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:49:09.456 DEBUG:boto:Method: GET 2023-09-22T21:49:09.456 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:49:09.456 DEBUG:boto:Data: 2023-09-22T21:49:09.456 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.456 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.456 DEBUG:boto:Params: {} 2023-09-22T21:49:09.456 DEBUG:boto:Token: None 2023-09-22T21:49:09.457 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/ 2023-09-22T21:49:09.457 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4= 2023-09-22T21:49:09.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4=', 'Content-Length': '0'} 2023-09-22T21:49:09.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097bb50e0368aa993-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.461 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:09.462 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:09.462 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:09.462 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:49:09.462 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:49:09.462 DEBUG:boto:Method: GET 2023-09-22T21:49:09.462 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:49:09.462 DEBUG:boto:Data: 2023-09-22T21:49:09.462 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.463 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.463 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.463 DEBUG:boto:Params: {} 2023-09-22T21:49:09.463 DEBUG:boto:Token: None 2023-09-22T21:49:09.463 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/ 2023-09-22T21:49:09.463 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4= 2023-09-22T21:49:09.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4=', 'Content-Length': '0'} 2023-09-22T21:49:09.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcaf072d5b47ce81-00650e0bd5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.474 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:09.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-09-22T21:49:09.474 DEBUG:boto:path=/ffzdqu-59/obj185 2023-09-22T21:49:09.474 DEBUG:boto:auth_path=/ffzdqu-59/obj185 2023-09-22T21:49:09.474 DEBUG:boto:path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.475 DEBUG:boto:auth_path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.475 DEBUG:boto:Method: GET 2023-09-22T21:49:09.475 DEBUG:boto:Path: /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.475 DEBUG:boto:Data: 2023-09-22T21:49:09.475 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.475 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.475 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.475 DEBUG:boto:Params: {} 2023-09-22T21:49:09.475 DEBUG:boto:Token: None 2023-09-22T21:49:09.476 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.476 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wCOxswNO7BnLbyWm7zENKylbKm0= 2023-09-22T21:49:09.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wCOxswNO7BnLbyWm7zENKylbKm0=', 'Content-Length': '0'} 2023-09-22T21:49:09.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000010ba9cac541f6c1-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.479 DEBUG:boto:path=/ffzdqu-59/obj185 2023-09-22T21:49:09.479 DEBUG:boto:auth_path=/ffzdqu-59/obj185 2023-09-22T21:49:09.479 DEBUG:boto:path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.479 DEBUG:boto:auth_path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.479 DEBUG:boto:Method: GET 2023-09-22T21:49:09.479 DEBUG:boto:Path: /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.479 DEBUG:boto:Data: 2023-09-22T21:49:09.479 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.480 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.480 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.480 DEBUG:boto:Params: {} 2023-09-22T21:49:09.480 DEBUG:boto:Token: None 2023-09-22T21:49:09.480 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.480 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wCOxswNO7BnLbyWm7zENKylbKm0= 2023-09-22T21:49:09.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wCOxswNO7BnLbyWm7zENKylbKm0=', 'Content-Length': '0'} 2023-09-22T21:49:09.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f43378f76dcb6aaa-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.530 DEBUG:boto:path=/ffzdqu-59/obj185 2023-09-22T21:49:09.531 DEBUG:boto:auth_path=/ffzdqu-59/obj185 2023-09-22T21:49:09.531 DEBUG:boto:path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.531 DEBUG:boto:auth_path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.531 DEBUG:boto:Method: HEAD 2023-09-22T21:49:09.531 DEBUG:boto:Path: /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.531 DEBUG:boto:Data: 2023-09-22T21:49:09.531 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.531 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.531 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.532 DEBUG:boto:Params: {} 2023-09-22T21:49:09.532 DEBUG:boto:Token: None 2023-09-22T21:49:09.532 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.532 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yozAgFtNTuc0nkfpZpXdNT3BxCQ= 2023-09-22T21:49:09.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yozAgFtNTuc0nkfpZpXdNT3BxCQ=', 'Content-Length': '0'} 2023-09-22T21:49:09.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ded93eda15f96fbc-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.535 DEBUG:boto:path=/ffzdqu-59/obj185 2023-09-22T21:49:09.535 DEBUG:boto:auth_path=/ffzdqu-59/obj185 2023-09-22T21:49:09.535 DEBUG:boto:path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.536 DEBUG:boto:auth_path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.536 DEBUG:boto:Method: HEAD 2023-09-22T21:49:09.536 DEBUG:boto:Path: /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.536 DEBUG:boto:Data: 2023-09-22T21:49:09.536 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.536 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.536 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.536 DEBUG:boto:Params: {} 2023-09-22T21:49:09.536 DEBUG:boto:Token: None 2023-09-22T21:49:09.537 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.537 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yozAgFtNTuc0nkfpZpXdNT3BxCQ= 2023-09-22T21:49:09.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yozAgFtNTuc0nkfpZpXdNT3BxCQ=', 'Content-Length': '0'} 2023-09-22T21:49:09.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5ba9ad6452ff139-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-09-22T21:49:09.540 DEBUG:boto:path=/ffzdqu-59/obj185 2023-09-22T21:49:09.541 DEBUG:boto:auth_path=/ffzdqu-59/obj185 2023-09-22T21:49:09.541 DEBUG:boto:path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.541 DEBUG:boto:auth_path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.541 DEBUG:boto:Method: GET 2023-09-22T21:49:09.541 DEBUG:boto:Path: /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.541 DEBUG:boto:Data: 2023-09-22T21:49:09.541 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.541 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.541 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.541 DEBUG:boto:Params: {} 2023-09-22T21:49:09.542 DEBUG:boto:Token: None 2023-09-22T21:49:09.542 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.542 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wCOxswNO7BnLbyWm7zENKylbKm0= 2023-09-22T21:49:09.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wCOxswNO7BnLbyWm7zENKylbKm0=', 'Content-Length': '0'} 2023-09-22T21:49:09.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095aba7cd9a7f0ef7-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.586 DEBUG:boto:path=/ffzdqu-59/obj185 2023-09-22T21:49:09.586 DEBUG:boto:auth_path=/ffzdqu-59/obj185 2023-09-22T21:49:09.587 DEBUG:boto:path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.587 DEBUG:boto:auth_path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.587 DEBUG:boto:Method: GET 2023-09-22T21:49:09.587 DEBUG:boto:Path: /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.587 DEBUG:boto:Data: 2023-09-22T21:49:09.587 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.587 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.587 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.587 DEBUG:boto:Params: {} 2023-09-22T21:49:09.588 DEBUG:boto:Token: None 2023-09-22T21:49:09.588 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:09.588 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wCOxswNO7BnLbyWm7zENKylbKm0= 2023-09-22T21:49:09.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wCOxswNO7BnLbyWm7zENKylbKm0=', 'Content-Length': '0'} 2023-09-22T21:49:09.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ebc0bb81c5b2e02-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.591 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:09.592 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:09.592 DEBUG:boto:path=/ffzdqu-59/?prefix=obj185 2023-09-22T21:49:09.592 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj185 2023-09-22T21:49:09.592 DEBUG:boto:Method: GET 2023-09-22T21:49:09.592 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj185 2023-09-22T21:49:09.592 DEBUG:boto:Data: 2023-09-22T21:49:09.592 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.592 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.592 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.592 DEBUG:boto:Params: {} 2023-09-22T21:49:09.593 DEBUG:boto:Token: None 2023-09-22T21:49:09.593 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/ 2023-09-22T21:49:09.593 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4= 2023-09-22T21:49:09.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4=', 'Content-Length': '0'} 2023-09-22T21:49:09.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a470e0ce4bad1b68-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.597 DEBUG:boto:b'ffzdqu-59obj1851000falseobj1852023-09-22T21:47:16.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:09.598 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:09.598 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:09.598 DEBUG:boto:path=/ffzdqu-59/?prefix=obj185 2023-09-22T21:49:09.598 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj185 2023-09-22T21:49:09.598 DEBUG:boto:Method: GET 2023-09-22T21:49:09.599 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj185 2023-09-22T21:49:09.599 DEBUG:boto:Data: 2023-09-22T21:49:09.599 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.599 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.599 DEBUG:boto:Params: {} 2023-09-22T21:49:09.599 DEBUG:boto:Token: None 2023-09-22T21:49:09.599 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/ 2023-09-22T21:49:09.599 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4= 2023-09-22T21:49:09.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4=', 'Content-Length': '0'} 2023-09-22T21:49:09.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4268d918d99caa8-00650e0bd5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.609 DEBUG:boto:b'ffzdqu-59obj1851000falseobj1852023-09-22T21:47:16.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:09.609 DEBUG:boto:path=/ffzdqu-59/obj185 2023-09-22T21:49:09.610 DEBUG:boto:auth_path=/ffzdqu-59/obj185 2023-09-22T21:49:09.610 DEBUG:boto:Method: HEAD 2023-09-22T21:49:09.610 DEBUG:boto:Path: /ffzdqu-59/obj185 2023-09-22T21:49:09.610 DEBUG:boto:Data: 2023-09-22T21:49:09.610 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.610 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.610 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.610 DEBUG:boto:Params: {} 2023-09-22T21:49:09.611 DEBUG:boto:Token: None 2023-09-22T21:49:09.611 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj185 2023-09-22T21:49:09.611 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BLTF1VcX79rF0WhrNeowlNy58TQ= 2023-09-22T21:49:09.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BLTF1VcX79rF0WhrNeowlNy58TQ=', 'Content-Length': '0'} 2023-09-22T21:49:09.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c24034550174a2b-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.614 DEBUG:boto:path=/ffzdqu-59/obj185 2023-09-22T21:49:09.614 DEBUG:boto:auth_path=/ffzdqu-59/obj185 2023-09-22T21:49:09.614 DEBUG:boto:Method: HEAD 2023-09-22T21:49:09.614 DEBUG:boto:Path: /ffzdqu-59/obj185 2023-09-22T21:49:09.615 DEBUG:boto:Data: 2023-09-22T21:49:09.615 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.615 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.615 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.615 DEBUG:boto:Params: {} 2023-09-22T21:49:09.615 DEBUG:boto:Token: None 2023-09-22T21:49:09.615 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj185 2023-09-22T21:49:09.615 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BLTF1VcX79rF0WhrNeowlNy58TQ= 2023-09-22T21:49:09.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BLTF1VcX79rF0WhrNeowlNy58TQ=', 'Content-Length': '0'} 2023-09-22T21:49:09.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000872342a61b51e19-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-09-22T21:49:09.619 DEBUG:boto:path=/ffzdqu-59/obj185 2023-09-22T21:49:09.619 DEBUG:boto:auth_path=/ffzdqu-59/obj185 2023-09-22T21:49:09.619 DEBUG:boto:Method: GET 2023-09-22T21:49:09.619 DEBUG:boto:Path: /ffzdqu-59/obj185 2023-09-22T21:49:09.619 DEBUG:boto:Data: 2023-09-22T21:49:09.619 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.619 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.619 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.619 DEBUG:boto:Params: {} 2023-09-22T21:49:09.620 DEBUG:boto:Token: None 2023-09-22T21:49:09.620 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj185 2023-09-22T21:49:09.620 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bHRtyobYcPScVEPEr26own/a2Hg= 2023-09-22T21:49:09.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bHRtyobYcPScVEPEr26own/a2Hg=', 'Content-Length': '0'} 2023-09-22T21:49:09.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055d36f6d01245b23-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.623 DEBUG:boto:path=/ffzdqu-59/obj185 2023-09-22T21:49:09.623 DEBUG:boto:auth_path=/ffzdqu-59/obj185 2023-09-22T21:49:09.623 DEBUG:boto:Method: GET 2023-09-22T21:49:09.623 DEBUG:boto:Path: /ffzdqu-59/obj185 2023-09-22T21:49:09.623 DEBUG:boto:Data: 2023-09-22T21:49:09.623 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.623 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.624 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.624 DEBUG:boto:Params: {} 2023-09-22T21:49:09.624 DEBUG:boto:Token: None 2023-09-22T21:49:09.624 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj185 2023-09-22T21:49:09.624 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bHRtyobYcPScVEPEr26own/a2Hg= 2023-09-22T21:49:09.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bHRtyobYcPScVEPEr26own/a2Hg=', 'Content-Length': '0'} 2023-09-22T21:49:09.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc99d8a95e5abfda-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.674 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:09.675 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:09.675 DEBUG:boto:path=/ffzdqu-59/?prefix=obj185 2023-09-22T21:49:09.675 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj185 2023-09-22T21:49:09.675 DEBUG:boto:Method: GET 2023-09-22T21:49:09.675 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj185 2023-09-22T21:49:09.675 DEBUG:boto:Data: 2023-09-22T21:49:09.675 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.675 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.676 DEBUG:boto:Params: {} 2023-09-22T21:49:09.676 DEBUG:boto:Token: None 2023-09-22T21:49:09.676 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/ 2023-09-22T21:49:09.676 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4= 2023-09-22T21:49:09.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4=', 'Content-Length': '0'} 2023-09-22T21:49:09.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a7d80a2c11aa2e1-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.680 DEBUG:boto:b'ffzdqu-59obj1851000falseobj1852023-09-22T21:47:16.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:09.681 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:09.681 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:09.681 DEBUG:boto:path=/ffzdqu-59/?prefix=obj185 2023-09-22T21:49:09.681 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj185 2023-09-22T21:49:09.681 DEBUG:boto:Method: GET 2023-09-22T21:49:09.681 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj185 2023-09-22T21:49:09.681 DEBUG:boto:Data: 2023-09-22T21:49:09.682 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.682 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.682 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.682 DEBUG:boto:Params: {} 2023-09-22T21:49:09.682 DEBUG:boto:Token: None 2023-09-22T21:49:09.682 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/ 2023-09-22T21:49:09.682 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4= 2023-09-22T21:49:09.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4=', 'Content-Length': '0'} 2023-09-22T21:49:09.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050093b24dbf06063-00650e0bd5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.693 DEBUG:boto:b'ffzdqu-59obj1851000falseobj1852023-09-22T21:47:16.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:09.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T21:49:09.694 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:49:09.694 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:49:09.694 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.694 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.694 DEBUG:boto:Method: GET 2023-09-22T21:49:09.694 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.694 DEBUG:boto:Data: 2023-09-22T21:49:09.694 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.695 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.695 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.695 DEBUG:boto:Params: {} 2023-09-22T21:49:09.695 DEBUG:boto:Token: None 2023-09-22T21:49:09.695 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.695 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uCyMUR7pw0O9L9gxEfjjiXp295o= 2023-09-22T21:49:09.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uCyMUR7pw0O9L9gxEfjjiXp295o=', 'Content-Length': '0'} 2023-09-22T21:49:09.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004801d4aac05c093d-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.738 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:49:09.739 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:49:09.739 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.739 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.739 DEBUG:boto:Method: GET 2023-09-22T21:49:09.739 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.739 DEBUG:boto:Data: 2023-09-22T21:49:09.739 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.739 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.739 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.739 DEBUG:boto:Params: {} 2023-09-22T21:49:09.740 DEBUG:boto:Token: None 2023-09-22T21:49:09.740 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.740 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uCyMUR7pw0O9L9gxEfjjiXp295o= 2023-09-22T21:49:09.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uCyMUR7pw0O9L9gxEfjjiXp295o=', 'Content-Length': '0'} 2023-09-22T21:49:09.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006020c67e8f0dc6e9-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.744 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:49:09.744 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:49:09.744 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.744 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.744 DEBUG:boto:Method: HEAD 2023-09-22T21:49:09.744 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.744 DEBUG:boto:Data: 2023-09-22T21:49:09.744 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.744 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.744 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.745 DEBUG:boto:Params: {} 2023-09-22T21:49:09.745 DEBUG:boto:Token: None 2023-09-22T21:49:09.745 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.745 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MCWuC4M2xSfZHR5ATxf1lW1UUD8= 2023-09-22T21:49:09.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MCWuC4M2xSfZHR5ATxf1lW1UUD8=', 'Content-Length': '0'} 2023-09-22T21:49:09.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b22e983dd9627fe1-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.748 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:49:09.748 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:49:09.748 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.748 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.748 DEBUG:boto:Method: HEAD 2023-09-22T21:49:09.748 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.749 DEBUG:boto:Data: 2023-09-22T21:49:09.749 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.749 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.749 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.749 DEBUG:boto:Params: {} 2023-09-22T21:49:09.749 DEBUG:boto:Token: None 2023-09-22T21:49:09.749 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.749 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MCWuC4M2xSfZHR5ATxf1lW1UUD8= 2023-09-22T21:49:09.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MCWuC4M2xSfZHR5ATxf1lW1UUD8=', 'Content-Length': '0'} 2023-09-22T21:49:09.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e55e95dbb40a615b-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T21:49:09.753 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:49:09.753 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:49:09.753 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.753 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.753 DEBUG:boto:Method: GET 2023-09-22T21:49:09.753 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.753 DEBUG:boto:Data: 2023-09-22T21:49:09.753 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.753 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.754 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.754 DEBUG:boto:Params: {} 2023-09-22T21:49:09.754 DEBUG:boto:Token: None 2023-09-22T21:49:09.754 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.754 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uCyMUR7pw0O9L9gxEfjjiXp295o= 2023-09-22T21:49:09.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uCyMUR7pw0O9L9gxEfjjiXp295o=', 'Content-Length': '0'} 2023-09-22T21:49:09.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000408ab70923c3cc66-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.757 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:49:09.757 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:49:09.757 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.757 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.758 DEBUG:boto:Method: GET 2023-09-22T21:49:09.758 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.758 DEBUG:boto:Data: 2023-09-22T21:49:09.758 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.758 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.758 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.758 DEBUG:boto:Params: {} 2023-09-22T21:49:09.758 DEBUG:boto:Token: None 2023-09-22T21:49:09.758 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:09.759 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uCyMUR7pw0O9L9gxEfjjiXp295o= 2023-09-22T21:49:09.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uCyMUR7pw0O9L9gxEfjjiXp295o=', 'Content-Length': '0'} 2023-09-22T21:49:09.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071f243fee8d4e1d5-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.806 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:09.807 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:09.807 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:49:09.807 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:49:09.807 DEBUG:boto:Method: GET 2023-09-22T21:49:09.807 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:49:09.807 DEBUG:boto:Data: 2023-09-22T21:49:09.807 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.807 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.807 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.807 DEBUG:boto:Params: {} 2023-09-22T21:49:09.808 DEBUG:boto:Token: None 2023-09-22T21:49:09.808 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/ 2023-09-22T21:49:09.808 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4= 2023-09-22T21:49:09.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4=', 'Content-Length': '0'} 2023-09-22T21:49:09.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000b1e4d5bb85a893-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.813 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:09.814 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:09.814 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:09.814 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:49:09.814 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:49:09.814 DEBUG:boto:Method: GET 2023-09-22T21:49:09.814 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:49:09.814 DEBUG:boto:Data: 2023-09-22T21:49:09.815 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.815 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.815 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.815 DEBUG:boto:Params: {} 2023-09-22T21:49:09.815 DEBUG:boto:Token: None 2023-09-22T21:49:09.815 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/ 2023-09-22T21:49:09.815 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4= 2023-09-22T21:49:09.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4=', 'Content-Length': '0'} 2023-09-22T21:49:09.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd7fce7cad9e76b3-00650e0bd5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.828 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:09.828 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:49:09.828 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:49:09.828 DEBUG:boto:Method: HEAD 2023-09-22T21:49:09.828 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:49:09.828 DEBUG:boto:Data: 2023-09-22T21:49:09.829 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.829 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.829 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.829 DEBUG:boto:Params: {} 2023-09-22T21:49:09.829 DEBUG:boto:Token: None 2023-09-22T21:49:09.829 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj19 2023-09-22T21:49:09.829 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yAZUCa/f9pHGsTb88UcKew7qB3A= 2023-09-22T21:49:09.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yAZUCa/f9pHGsTb88UcKew7qB3A=', 'Content-Length': '0'} 2023-09-22T21:49:09.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000255c78c79e89a9c9-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.832 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:49:09.832 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:49:09.833 DEBUG:boto:Method: HEAD 2023-09-22T21:49:09.833 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:49:09.833 DEBUG:boto:Data: 2023-09-22T21:49:09.833 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.833 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.833 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.833 DEBUG:boto:Params: {} 2023-09-22T21:49:09.833 DEBUG:boto:Token: None 2023-09-22T21:49:09.833 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj19 2023-09-22T21:49:09.834 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yAZUCa/f9pHGsTb88UcKew7qB3A= 2023-09-22T21:49:09.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yAZUCa/f9pHGsTb88UcKew7qB3A=', 'Content-Length': '0'} 2023-09-22T21:49:09.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c266d2decfbd079-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T21:49:09.837 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:49:09.837 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:49:09.837 DEBUG:boto:Method: GET 2023-09-22T21:49:09.837 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:49:09.837 DEBUG:boto:Data: 2023-09-22T21:49:09.837 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.837 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.838 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.838 DEBUG:boto:Params: {} 2023-09-22T21:49:09.838 DEBUG:boto:Token: None 2023-09-22T21:49:09.838 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj19 2023-09-22T21:49:09.838 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PgFeLNhsGqcxfqT9+kPlWWNoQYg= 2023-09-22T21:49:09.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PgFeLNhsGqcxfqT9+kPlWWNoQYg=', 'Content-Length': '0'} 2023-09-22T21:49:09.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb14ef5a6a8be075-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.886 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:49:09.887 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:49:09.887 DEBUG:boto:Method: GET 2023-09-22T21:49:09.887 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:49:09.887 DEBUG:boto:Data: 2023-09-22T21:49:09.887 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.887 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.887 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.887 DEBUG:boto:Params: {} 2023-09-22T21:49:09.888 DEBUG:boto:Token: None 2023-09-22T21:49:09.888 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj19 2023-09-22T21:49:09.888 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PgFeLNhsGqcxfqT9+kPlWWNoQYg= 2023-09-22T21:49:09.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PgFeLNhsGqcxfqT9+kPlWWNoQYg=', 'Content-Length': '0'} 2023-09-22T21:49:09.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef0cb6151add2868-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.891 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:09.891 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:09.891 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:49:09.892 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:49:09.892 DEBUG:boto:Method: GET 2023-09-22T21:49:09.892 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:49:09.892 DEBUG:boto:Data: 2023-09-22T21:49:09.892 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.892 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.892 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.892 DEBUG:boto:Params: {} 2023-09-22T21:49:09.892 DEBUG:boto:Token: None 2023-09-22T21:49:09.893 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/ 2023-09-22T21:49:09.893 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4= 2023-09-22T21:49:09.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4=', 'Content-Length': '0'} 2023-09-22T21:49:09.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c577b320d546c3ee-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.897 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:09.898 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:09.898 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:09.898 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:49:09.898 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:49:09.898 DEBUG:boto:Method: GET 2023-09-22T21:49:09.898 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:49:09.899 DEBUG:boto:Data: 2023-09-22T21:49:09.899 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.899 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.899 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.899 DEBUG:boto:Params: {} 2023-09-22T21:49:09.899 DEBUG:boto:Token: None 2023-09-22T21:49:09.899 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/ 2023-09-22T21:49:09.899 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4= 2023-09-22T21:49:09.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4=', 'Content-Length': '0'} 2023-09-22T21:49:09.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d86bb776ae085be1-00650e0bd5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.950 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:09.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:49:09.951 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:49:09.951 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:49:09.951 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.951 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.952 DEBUG:boto:Method: GET 2023-09-22T21:49:09.952 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.952 DEBUG:boto:Data: 2023-09-22T21:49:09.952 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.952 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.952 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.952 DEBUG:boto:Params: {} 2023-09-22T21:49:09.952 DEBUG:boto:Token: None 2023-09-22T21:49:09.952 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.953 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KRHr8SPcJKEFuctyJCm/Wqt+qUc= 2023-09-22T21:49:09.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KRHr8SPcJKEFuctyJCm/Wqt+qUc=', 'Content-Length': '0'} 2023-09-22T21:49:09.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d47ebbf346d35957-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.956 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:49:09.956 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:49:09.956 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.956 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.956 DEBUG:boto:Method: GET 2023-09-22T21:49:09.957 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.957 DEBUG:boto:Data: 2023-09-22T21:49:09.957 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.957 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.957 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.957 DEBUG:boto:Params: {} 2023-09-22T21:49:09.957 DEBUG:boto:Token: None 2023-09-22T21:49:09.957 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.957 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KRHr8SPcJKEFuctyJCm/Wqt+qUc= 2023-09-22T21:49:09.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KRHr8SPcJKEFuctyJCm/Wqt+qUc=', 'Content-Length': '0'} 2023-09-22T21:49:09.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c26c3c3d9643df5-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.961 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:49:09.961 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:49:09.961 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.961 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.961 DEBUG:boto:Method: HEAD 2023-09-22T21:49:09.961 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.961 DEBUG:boto:Data: 2023-09-22T21:49:09.961 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.961 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.961 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.962 DEBUG:boto:Params: {} 2023-09-22T21:49:09.962 DEBUG:boto:Token: None 2023-09-22T21:49:09.962 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.962 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:u0i/oTxgJpg9yPAfNeD4OgUaKqY= 2023-09-22T21:49:09.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:u0i/oTxgJpg9yPAfNeD4OgUaKqY=', 'Content-Length': '0'} 2023-09-22T21:49:09.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b14be67616e1c328-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.965 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:49:09.965 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:49:09.965 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.966 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.966 DEBUG:boto:Method: HEAD 2023-09-22T21:49:09.966 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.966 DEBUG:boto:Data: 2023-09-22T21:49:09.966 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.966 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.966 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.966 DEBUG:boto:Params: {} 2023-09-22T21:49:09.966 DEBUG:boto:Token: None 2023-09-22T21:49:09.967 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.967 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:u0i/oTxgJpg9yPAfNeD4OgUaKqY= 2023-09-22T21:49:09.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:u0i/oTxgJpg9yPAfNeD4OgUaKqY=', 'Content-Length': '0'} 2023-09-22T21:49:09.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005530a08b823728cd-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:49:09.969 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:49:09.970 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:49:09.970 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.970 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.970 DEBUG:boto:Method: GET 2023-09-22T21:49:09.970 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.970 DEBUG:boto:Data: 2023-09-22T21:49:09.970 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.970 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.970 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.970 DEBUG:boto:Params: {} 2023-09-22T21:49:09.971 DEBUG:boto:Token: None 2023-09-22T21:49:09.971 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.971 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KRHr8SPcJKEFuctyJCm/Wqt+qUc= 2023-09-22T21:49:09.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KRHr8SPcJKEFuctyJCm/Wqt+qUc=', 'Content-Length': '0'} 2023-09-22T21:49:09.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b03135df1ccd7160-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.974 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:49:09.974 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:49:09.974 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.974 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.974 DEBUG:boto:Method: GET 2023-09-22T21:49:09.974 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.974 DEBUG:boto:Data: 2023-09-22T21:49:09.975 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.975 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.975 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.975 DEBUG:boto:Params: {} 2023-09-22T21:49:09.975 DEBUG:boto:Token: None 2023-09-22T21:49:09.975 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:09.975 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KRHr8SPcJKEFuctyJCm/Wqt+qUc= 2023-09-22T21:49:09.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KRHr8SPcJKEFuctyJCm/Wqt+qUc=', 'Content-Length': '0'} 2023-09-22T21:49:09.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f63e5e15182fbfc9-00650e0bd5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.978 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:09.979 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:09.979 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:49:09.979 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:49:09.979 DEBUG:boto:Method: GET 2023-09-22T21:49:09.979 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:49:09.979 DEBUG:boto:Data: 2023-09-22T21:49:09.979 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.979 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:09.979 DEBUG:boto:Port: 8001 2023-09-22T21:49:09.979 DEBUG:boto:Params: {} 2023-09-22T21:49:09.980 DEBUG:boto:Token: None 2023-09-22T21:49:09.980 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/ 2023-09-22T21:49:09.980 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4= 2023-09-22T21:49:09.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4=', 'Content-Length': '0'} 2023-09-22T21:49:09.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017974b433d3fa5dc-00650e0bd5-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:09.984 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:09.985 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:09.985 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:09.985 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:49:09.985 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:49:09.985 DEBUG:boto:Method: GET 2023-09-22T21:49:09.985 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:49:09.985 DEBUG:boto:Data: 2023-09-22T21:49:09.985 DEBUG:boto:Headers: {} 2023-09-22T21:49:09.986 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:09.986 DEBUG:boto:Port: 8000 2023-09-22T21:49:09.986 DEBUG:boto:Params: {} 2023-09-22T21:49:09.986 DEBUG:boto:Token: None 2023-09-22T21:49:09.986 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:09 GMT /ffzdqu-59/ 2023-09-22T21:49:09.986 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4= 2023-09-22T21:49:09.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5xUTrrvaCl+QqGzYQLK98wxF5Q4=', 'Content-Length': '0'} 2023-09-22T21:49:09.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf1a90df75d3a407-00650e0bd5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.038 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:10.039 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:49:10.039 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:49:10.039 DEBUG:boto:Method: HEAD 2023-09-22T21:49:10.039 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:49:10.039 DEBUG:boto:Data: 2023-09-22T21:49:10.040 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.040 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.040 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.040 DEBUG:boto:Params: {} 2023-09-22T21:49:10.040 DEBUG:boto:Token: None 2023-09-22T21:49:10.040 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj22 2023-09-22T21:49:10.040 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:V0TXulhSOhTVc00R0CoX8/vuMyk= 2023-09-22T21:49:10.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:V0TXulhSOhTVc00R0CoX8/vuMyk=', 'Content-Length': '0'} 2023-09-22T21:49:10.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027ce21d07ce423e5-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.043 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:49:10.043 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:49:10.044 DEBUG:boto:Method: HEAD 2023-09-22T21:49:10.044 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:49:10.044 DEBUG:boto:Data: 2023-09-22T21:49:10.044 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.044 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.044 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.044 DEBUG:boto:Params: {} 2023-09-22T21:49:10.044 DEBUG:boto:Token: None 2023-09-22T21:49:10.044 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj22 2023-09-22T21:49:10.045 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:V0TXulhSOhTVc00R0CoX8/vuMyk= 2023-09-22T21:49:10.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:V0TXulhSOhTVc00R0CoX8/vuMyk=', 'Content-Length': '0'} 2023-09-22T21:49:10.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e087bb985c41093-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:49:10.048 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:49:10.048 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:49:10.048 DEBUG:boto:Method: GET 2023-09-22T21:49:10.048 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:49:10.048 DEBUG:boto:Data: 2023-09-22T21:49:10.048 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.048 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.048 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.048 DEBUG:boto:Params: {} 2023-09-22T21:49:10.049 DEBUG:boto:Token: None 2023-09-22T21:49:10.049 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj22 2023-09-22T21:49:10.049 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qLSayoT36jql0Wgi+dJhKY1rfpA= 2023-09-22T21:49:10.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qLSayoT36jql0Wgi+dJhKY1rfpA=', 'Content-Length': '0'} 2023-09-22T21:49:10.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb138af113c391fc-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.098 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:49:10.099 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:49:10.099 DEBUG:boto:Method: GET 2023-09-22T21:49:10.099 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:49:10.099 DEBUG:boto:Data: 2023-09-22T21:49:10.099 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.099 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.099 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.099 DEBUG:boto:Params: {} 2023-09-22T21:49:10.099 DEBUG:boto:Token: None 2023-09-22T21:49:10.100 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj22 2023-09-22T21:49:10.100 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qLSayoT36jql0Wgi+dJhKY1rfpA= 2023-09-22T21:49:10.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qLSayoT36jql0Wgi+dJhKY1rfpA=', 'Content-Length': '0'} 2023-09-22T21:49:10.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dfe047187b5ba7f-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.103 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:10.103 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:10.103 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:49:10.103 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:49:10.103 DEBUG:boto:Method: GET 2023-09-22T21:49:10.103 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:49:10.103 DEBUG:boto:Data: 2023-09-22T21:49:10.104 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.104 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.104 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.104 DEBUG:boto:Params: {} 2023-09-22T21:49:10.104 DEBUG:boto:Token: None 2023-09-22T21:49:10.104 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/ 2023-09-22T21:49:10.104 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE= 2023-09-22T21:49:10.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE=', 'Content-Length': '0'} 2023-09-22T21:49:10.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1a1150ba657aacd-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.109 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:10.109 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:10.110 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:10.110 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:49:10.110 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:49:10.110 DEBUG:boto:Method: GET 2023-09-22T21:49:10.110 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:49:10.110 DEBUG:boto:Data: 2023-09-22T21:49:10.110 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.110 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.111 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.111 DEBUG:boto:Params: {} 2023-09-22T21:49:10.111 DEBUG:boto:Token: None 2023-09-22T21:49:10.111 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/ 2023-09-22T21:49:10.111 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE= 2023-09-22T21:49:10.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE=', 'Content-Length': '0'} 2023-09-22T21:49:10.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbee0bd822e30aa2-00650e0bd6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.121 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:10.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-22T21:49:10.122 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:49:10.122 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:49:10.122 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.122 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.122 DEBUG:boto:Method: GET 2023-09-22T21:49:10.122 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.122 DEBUG:boto:Data: 2023-09-22T21:49:10.122 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.122 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.123 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.123 DEBUG:boto:Params: {} 2023-09-22T21:49:10.123 DEBUG:boto:Token: None 2023-09-22T21:49:10.123 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.123 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HadH1NYvS78dKxw8AYOETgGSTP8= 2023-09-22T21:49:10.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HadH1NYvS78dKxw8AYOETgGSTP8=', 'Content-Length': '0'} 2023-09-22T21:49:10.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da461920f72296eb-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.126 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:49:10.126 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:49:10.126 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.126 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.126 DEBUG:boto:Method: GET 2023-09-22T21:49:10.127 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.127 DEBUG:boto:Data: 2023-09-22T21:49:10.127 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.127 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.127 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.127 DEBUG:boto:Params: {} 2023-09-22T21:49:10.127 DEBUG:boto:Token: None 2023-09-22T21:49:10.127 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.127 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HadH1NYvS78dKxw8AYOETgGSTP8= 2023-09-22T21:49:10.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HadH1NYvS78dKxw8AYOETgGSTP8=', 'Content-Length': '0'} 2023-09-22T21:49:10.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fab0c029e970dd6b-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.178 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:49:10.179 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:49:10.179 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.179 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.179 DEBUG:boto:Method: HEAD 2023-09-22T21:49:10.179 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.179 DEBUG:boto:Data: 2023-09-22T21:49:10.179 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.179 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.179 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.180 DEBUG:boto:Params: {} 2023-09-22T21:49:10.180 DEBUG:boto:Token: None 2023-09-22T21:49:10.180 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.180 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kFp71h2KdWUAsWuXQsAu7GLXwcg= 2023-09-22T21:49:10.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kFp71h2KdWUAsWuXQsAu7GLXwcg=', 'Content-Length': '0'} 2023-09-22T21:49:10.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065832f71f934bd52-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.183 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:49:10.183 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:49:10.183 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.183 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.184 DEBUG:boto:Method: HEAD 2023-09-22T21:49:10.184 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.184 DEBUG:boto:Data: 2023-09-22T21:49:10.184 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.184 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.184 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.184 DEBUG:boto:Params: {} 2023-09-22T21:49:10.184 DEBUG:boto:Token: None 2023-09-22T21:49:10.184 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.185 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kFp71h2KdWUAsWuXQsAu7GLXwcg= 2023-09-22T21:49:10.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kFp71h2KdWUAsWuXQsAu7GLXwcg=', 'Content-Length': '0'} 2023-09-22T21:49:10.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c441dc286ec86f5-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-22T21:49:10.188 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:49:10.188 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:49:10.188 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.188 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.188 DEBUG:boto:Method: GET 2023-09-22T21:49:10.188 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.188 DEBUG:boto:Data: 2023-09-22T21:49:10.188 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.189 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.189 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.189 DEBUG:boto:Params: {} 2023-09-22T21:49:10.189 DEBUG:boto:Token: None 2023-09-22T21:49:10.189 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.189 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HadH1NYvS78dKxw8AYOETgGSTP8= 2023-09-22T21:49:10.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HadH1NYvS78dKxw8AYOETgGSTP8=', 'Content-Length': '0'} 2023-09-22T21:49:10.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000282ff813a9d05ca7-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.234 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:49:10.235 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:49:10.235 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.235 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.235 DEBUG:boto:Method: GET 2023-09-22T21:49:10.235 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.235 DEBUG:boto:Data: 2023-09-22T21:49:10.235 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.235 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.235 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.235 DEBUG:boto:Params: {} 2023-09-22T21:49:10.236 DEBUG:boto:Token: None 2023-09-22T21:49:10.236 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:10.236 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HadH1NYvS78dKxw8AYOETgGSTP8= 2023-09-22T21:49:10.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HadH1NYvS78dKxw8AYOETgGSTP8=', 'Content-Length': '0'} 2023-09-22T21:49:10.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f63a2e8372a28b36-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.239 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:10.239 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:10.239 DEBUG:boto:path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:49:10.239 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:49:10.240 DEBUG:boto:Method: GET 2023-09-22T21:49:10.240 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj23 2023-09-22T21:49:10.240 DEBUG:boto:Data: 2023-09-22T21:49:10.240 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.240 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.240 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.240 DEBUG:boto:Params: {} 2023-09-22T21:49:10.240 DEBUG:boto:Token: None 2023-09-22T21:49:10.240 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/ 2023-09-22T21:49:10.241 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE= 2023-09-22T21:49:10.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE=', 'Content-Length': '0'} 2023-09-22T21:49:10.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047d8d312b8b0cb23-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.247 DEBUG:boto:b'ffzdqu-59obj231000falseobj232023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:10.247 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:10.247 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:10.247 DEBUG:boto:path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:49:10.247 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:49:10.248 DEBUG:boto:Method: GET 2023-09-22T21:49:10.248 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj23 2023-09-22T21:49:10.248 DEBUG:boto:Data: 2023-09-22T21:49:10.248 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.248 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.248 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.249 DEBUG:boto:Params: {} 2023-09-22T21:49:10.249 DEBUG:boto:Token: None 2023-09-22T21:49:10.249 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/ 2023-09-22T21:49:10.249 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE= 2023-09-22T21:49:10.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE=', 'Content-Length': '0'} 2023-09-22T21:49:10.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000742d8a6b6300c4fc-00650e0bd6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.259 DEBUG:boto:b'ffzdqu-59obj231000falseobj232023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:10.259 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:49:10.259 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:49:10.259 DEBUG:boto:Method: HEAD 2023-09-22T21:49:10.259 DEBUG:boto:Path: /ffzdqu-59/obj23 2023-09-22T21:49:10.259 DEBUG:boto:Data: 2023-09-22T21:49:10.260 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.260 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.260 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.260 DEBUG:boto:Params: {} 2023-09-22T21:49:10.260 DEBUG:boto:Token: None 2023-09-22T21:49:10.260 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj23 2023-09-22T21:49:10.260 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1h50oqU4t76yvs+DaRkKvjSo45U= 2023-09-22T21:49:10.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1h50oqU4t76yvs+DaRkKvjSo45U=', 'Content-Length': '0'} 2023-09-22T21:49:10.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d40478fd93381bbd-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.263 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:49:10.263 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:49:10.263 DEBUG:boto:Method: HEAD 2023-09-22T21:49:10.263 DEBUG:boto:Path: /ffzdqu-59/obj23 2023-09-22T21:49:10.263 DEBUG:boto:Data: 2023-09-22T21:49:10.264 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.264 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.264 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.264 DEBUG:boto:Params: {} 2023-09-22T21:49:10.264 DEBUG:boto:Token: None 2023-09-22T21:49:10.264 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj23 2023-09-22T21:49:10.264 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1h50oqU4t76yvs+DaRkKvjSo45U= 2023-09-22T21:49:10.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1h50oqU4t76yvs+DaRkKvjSo45U=', 'Content-Length': '0'} 2023-09-22T21:49:10.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9e83b75f3914d94-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-22T21:49:10.267 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:49:10.267 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:49:10.267 DEBUG:boto:Method: GET 2023-09-22T21:49:10.268 DEBUG:boto:Path: /ffzdqu-59/obj23 2023-09-22T21:49:10.268 DEBUG:boto:Data: 2023-09-22T21:49:10.268 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.268 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.268 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.268 DEBUG:boto:Params: {} 2023-09-22T21:49:10.268 DEBUG:boto:Token: None 2023-09-22T21:49:10.268 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj23 2023-09-22T21:49:10.268 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:L05MPPwkqQiWlnN9LGK/2qY3uWU= 2023-09-22T21:49:10.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:L05MPPwkqQiWlnN9LGK/2qY3uWU=', 'Content-Length': '0'} 2023-09-22T21:49:10.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7ce1314d558b0eb-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.318 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:49:10.319 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:49:10.319 DEBUG:boto:Method: GET 2023-09-22T21:49:10.319 DEBUG:boto:Path: /ffzdqu-59/obj23 2023-09-22T21:49:10.319 DEBUG:boto:Data: 2023-09-22T21:49:10.319 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.319 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.319 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.319 DEBUG:boto:Params: {} 2023-09-22T21:49:10.319 DEBUG:boto:Token: None 2023-09-22T21:49:10.320 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj23 2023-09-22T21:49:10.320 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:L05MPPwkqQiWlnN9LGK/2qY3uWU= 2023-09-22T21:49:10.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:L05MPPwkqQiWlnN9LGK/2qY3uWU=', 'Content-Length': '0'} 2023-09-22T21:49:10.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3cb813f7bf9900e-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.323 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:10.323 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:10.323 DEBUG:boto:path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:49:10.323 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:49:10.324 DEBUG:boto:Method: GET 2023-09-22T21:49:10.324 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj23 2023-09-22T21:49:10.324 DEBUG:boto:Data: 2023-09-22T21:49:10.324 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.324 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.324 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.324 DEBUG:boto:Params: {} 2023-09-22T21:49:10.324 DEBUG:boto:Token: None 2023-09-22T21:49:10.324 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/ 2023-09-22T21:49:10.325 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE= 2023-09-22T21:49:10.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE=', 'Content-Length': '0'} 2023-09-22T21:49:10.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc24c91db5905171-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.329 DEBUG:boto:b'ffzdqu-59obj231000falseobj232023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:10.330 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:10.330 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:10.330 DEBUG:boto:path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:49:10.330 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:49:10.330 DEBUG:boto:Method: GET 2023-09-22T21:49:10.330 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj23 2023-09-22T21:49:10.330 DEBUG:boto:Data: 2023-09-22T21:49:10.331 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.331 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.331 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.331 DEBUG:boto:Params: {} 2023-09-22T21:49:10.331 DEBUG:boto:Token: None 2023-09-22T21:49:10.331 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/ 2023-09-22T21:49:10.331 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE= 2023-09-22T21:49:10.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE=', 'Content-Length': '0'} 2023-09-22T21:49:10.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079293df820a3d7db-00650e0bd6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.341 DEBUG:boto:b'ffzdqu-59obj231000falseobj232023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:10.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-22T21:49:10.342 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:49:10.342 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:49:10.342 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.342 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.342 DEBUG:boto:Method: GET 2023-09-22T21:49:10.342 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.342 DEBUG:boto:Data: 2023-09-22T21:49:10.342 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.342 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.342 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.343 DEBUG:boto:Params: {} 2023-09-22T21:49:10.343 DEBUG:boto:Token: None 2023-09-22T21:49:10.343 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.343 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:edBwSAzQLBkb+5ouC/EmIqopqY8= 2023-09-22T21:49:10.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:edBwSAzQLBkb+5ouC/EmIqopqY8=', 'Content-Length': '0'} 2023-09-22T21:49:10.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fc6db71d1c81d61-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.346 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:49:10.346 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:49:10.346 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.346 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.346 DEBUG:boto:Method: GET 2023-09-22T21:49:10.347 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.347 DEBUG:boto:Data: 2023-09-22T21:49:10.347 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.347 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.347 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.347 DEBUG:boto:Params: {} 2023-09-22T21:49:10.347 DEBUG:boto:Token: None 2023-09-22T21:49:10.347 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.347 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:edBwSAzQLBkb+5ouC/EmIqopqY8= 2023-09-22T21:49:10.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:edBwSAzQLBkb+5ouC/EmIqopqY8=', 'Content-Length': '0'} 2023-09-22T21:49:10.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001200c5ee9982968e-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.350 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:49:10.351 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:49:10.351 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.351 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.351 DEBUG:boto:Method: HEAD 2023-09-22T21:49:10.351 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.351 DEBUG:boto:Data: 2023-09-22T21:49:10.351 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.351 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.351 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.352 DEBUG:boto:Params: {} 2023-09-22T21:49:10.352 DEBUG:boto:Token: None 2023-09-22T21:49:10.352 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.352 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sRmvwsCAoxx2ULp0jW4osvCpz8o= 2023-09-22T21:49:10.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sRmvwsCAoxx2ULp0jW4osvCpz8o=', 'Content-Length': '0'} 2023-09-22T21:49:10.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7eb29d000624ff7-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.355 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:49:10.355 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:49:10.355 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.355 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.355 DEBUG:boto:Method: HEAD 2023-09-22T21:49:10.355 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.355 DEBUG:boto:Data: 2023-09-22T21:49:10.356 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.356 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.356 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.356 DEBUG:boto:Params: {} 2023-09-22T21:49:10.356 DEBUG:boto:Token: None 2023-09-22T21:49:10.356 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.356 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sRmvwsCAoxx2ULp0jW4osvCpz8o= 2023-09-22T21:49:10.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sRmvwsCAoxx2ULp0jW4osvCpz8o=', 'Content-Length': '0'} 2023-09-22T21:49:10.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cb75a168019dcb2-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-22T21:49:10.360 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:49:10.360 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:49:10.360 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.360 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.360 DEBUG:boto:Method: GET 2023-09-22T21:49:10.360 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.360 DEBUG:boto:Data: 2023-09-22T21:49:10.360 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.360 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.360 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.361 DEBUG:boto:Params: {} 2023-09-22T21:49:10.361 DEBUG:boto:Token: None 2023-09-22T21:49:10.361 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.361 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:edBwSAzQLBkb+5ouC/EmIqopqY8= 2023-09-22T21:49:10.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:edBwSAzQLBkb+5ouC/EmIqopqY8=', 'Content-Length': '0'} 2023-09-22T21:49:10.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b5eaea9fcc02aec-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.406 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:49:10.407 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:49:10.407 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.407 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.407 DEBUG:boto:Method: GET 2023-09-22T21:49:10.407 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.407 DEBUG:boto:Data: 2023-09-22T21:49:10.407 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.407 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.408 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.408 DEBUG:boto:Params: {} 2023-09-22T21:49:10.408 DEBUG:boto:Token: None 2023-09-22T21:49:10.408 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:10.408 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:edBwSAzQLBkb+5ouC/EmIqopqY8= 2023-09-22T21:49:10.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:edBwSAzQLBkb+5ouC/EmIqopqY8=', 'Content-Length': '0'} 2023-09-22T21:49:10.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b662e1ad521d49e-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.411 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:10.411 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:10.412 DEBUG:boto:path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:49:10.412 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:49:10.412 DEBUG:boto:Method: GET 2023-09-22T21:49:10.412 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj29 2023-09-22T21:49:10.412 DEBUG:boto:Data: 2023-09-22T21:49:10.412 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.412 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.412 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.412 DEBUG:boto:Params: {} 2023-09-22T21:49:10.413 DEBUG:boto:Token: None 2023-09-22T21:49:10.413 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/ 2023-09-22T21:49:10.413 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE= 2023-09-22T21:49:10.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE=', 'Content-Length': '0'} 2023-09-22T21:49:10.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d7375d9b8af0f49-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.418 DEBUG:boto:b'ffzdqu-59obj291000falseobj292023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:10.419 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:10.419 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:10.419 DEBUG:boto:path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:49:10.419 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:49:10.419 DEBUG:boto:Method: GET 2023-09-22T21:49:10.419 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj29 2023-09-22T21:49:10.419 DEBUG:boto:Data: 2023-09-22T21:49:10.419 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.420 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.420 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.420 DEBUG:boto:Params: {} 2023-09-22T21:49:10.420 DEBUG:boto:Token: None 2023-09-22T21:49:10.420 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/ 2023-09-22T21:49:10.420 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE= 2023-09-22T21:49:10.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE=', 'Content-Length': '0'} 2023-09-22T21:49:10.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da9255408ff016d4-00650e0bd6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.431 DEBUG:boto:b'ffzdqu-59obj291000falseobj292023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:10.431 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:49:10.431 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:49:10.431 DEBUG:boto:Method: HEAD 2023-09-22T21:49:10.431 DEBUG:boto:Path: /ffzdqu-59/obj29 2023-09-22T21:49:10.431 DEBUG:boto:Data: 2023-09-22T21:49:10.431 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.432 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.432 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.432 DEBUG:boto:Params: {} 2023-09-22T21:49:10.432 DEBUG:boto:Token: None 2023-09-22T21:49:10.432 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj29 2023-09-22T21:49:10.432 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PLuy2+uRCBa9XveY+qRecLVcEk4= 2023-09-22T21:49:10.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PLuy2+uRCBa9XveY+qRecLVcEk4=', 'Content-Length': '0'} 2023-09-22T21:49:10.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd8bc5f7d0b7f14d-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.435 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:49:10.435 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:49:10.435 DEBUG:boto:Method: HEAD 2023-09-22T21:49:10.435 DEBUG:boto:Path: /ffzdqu-59/obj29 2023-09-22T21:49:10.435 DEBUG:boto:Data: 2023-09-22T21:49:10.436 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.436 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.436 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.436 DEBUG:boto:Params: {} 2023-09-22T21:49:10.436 DEBUG:boto:Token: None 2023-09-22T21:49:10.436 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj29 2023-09-22T21:49:10.436 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PLuy2+uRCBa9XveY+qRecLVcEk4= 2023-09-22T21:49:10.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PLuy2+uRCBa9XveY+qRecLVcEk4=', 'Content-Length': '0'} 2023-09-22T21:49:10.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002758129970fa9886-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-22T21:49:10.439 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:49:10.439 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:49:10.440 DEBUG:boto:Method: GET 2023-09-22T21:49:10.440 DEBUG:boto:Path: /ffzdqu-59/obj29 2023-09-22T21:49:10.440 DEBUG:boto:Data: 2023-09-22T21:49:10.440 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.440 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.440 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.440 DEBUG:boto:Params: {} 2023-09-22T21:49:10.440 DEBUG:boto:Token: None 2023-09-22T21:49:10.440 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj29 2023-09-22T21:49:10.441 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HNU/rz60yqtAGaGbdMuaAL5yDw0= 2023-09-22T21:49:10.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HNU/rz60yqtAGaGbdMuaAL5yDw0=', 'Content-Length': '0'} 2023-09-22T21:49:10.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af6e90beca0a5559-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.486 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:49:10.487 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:49:10.487 DEBUG:boto:Method: GET 2023-09-22T21:49:10.487 DEBUG:boto:Path: /ffzdqu-59/obj29 2023-09-22T21:49:10.487 DEBUG:boto:Data: 2023-09-22T21:49:10.487 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.487 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.487 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.487 DEBUG:boto:Params: {} 2023-09-22T21:49:10.488 DEBUG:boto:Token: None 2023-09-22T21:49:10.488 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj29 2023-09-22T21:49:10.488 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HNU/rz60yqtAGaGbdMuaAL5yDw0= 2023-09-22T21:49:10.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HNU/rz60yqtAGaGbdMuaAL5yDw0=', 'Content-Length': '0'} 2023-09-22T21:49:10.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4b7075ac5af2c65-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.491 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:10.491 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:10.492 DEBUG:boto:path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:49:10.492 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:49:10.492 DEBUG:boto:Method: GET 2023-09-22T21:49:10.492 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj29 2023-09-22T21:49:10.492 DEBUG:boto:Data: 2023-09-22T21:49:10.492 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.492 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.492 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.493 DEBUG:boto:Params: {} 2023-09-22T21:49:10.493 DEBUG:boto:Token: None 2023-09-22T21:49:10.493 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/ 2023-09-22T21:49:10.493 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE= 2023-09-22T21:49:10.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE=', 'Content-Length': '0'} 2023-09-22T21:49:10.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048f5480a1ac54252-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.497 DEBUG:boto:b'ffzdqu-59obj291000falseobj292023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:10.498 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:10.498 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:10.498 DEBUG:boto:path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:49:10.498 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:49:10.498 DEBUG:boto:Method: GET 2023-09-22T21:49:10.499 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj29 2023-09-22T21:49:10.499 DEBUG:boto:Data: 2023-09-22T21:49:10.499 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.499 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.499 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.499 DEBUG:boto:Params: {} 2023-09-22T21:49:10.499 DEBUG:boto:Token: None 2023-09-22T21:49:10.499 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/ 2023-09-22T21:49:10.499 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE= 2023-09-22T21:49:10.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE=', 'Content-Length': '0'} 2023-09-22T21:49:10.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f725c40fc464c78-00650e0bd6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.550 DEBUG:boto:b'ffzdqu-59obj291000falseobj292023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:10.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-22T21:49:10.551 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:49:10.551 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:49:10.551 DEBUG:boto:path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.551 DEBUG:boto:auth_path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.551 DEBUG:boto:Method: GET 2023-09-22T21:49:10.552 DEBUG:boto:Path: /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.552 DEBUG:boto:Data: 2023-09-22T21:49:10.552 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.552 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.552 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.552 DEBUG:boto:Params: {} 2023-09-22T21:49:10.552 DEBUG:boto:Token: None 2023-09-22T21:49:10.552 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.552 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:co3tmxpgxBzB2BEI9QKOV4/smAI= 2023-09-22T21:49:10.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:co3tmxpgxBzB2BEI9QKOV4/smAI=', 'Content-Length': '0'} 2023-09-22T21:49:10.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078c1119733d16066-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.555 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:49:10.556 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:49:10.556 DEBUG:boto:path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.556 DEBUG:boto:auth_path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.556 DEBUG:boto:Method: GET 2023-09-22T21:49:10.556 DEBUG:boto:Path: /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.556 DEBUG:boto:Data: 2023-09-22T21:49:10.556 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.556 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.556 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.557 DEBUG:boto:Params: {} 2023-09-22T21:49:10.557 DEBUG:boto:Token: None 2023-09-22T21:49:10.557 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.557 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:co3tmxpgxBzB2BEI9QKOV4/smAI= 2023-09-22T21:49:10.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:co3tmxpgxBzB2BEI9QKOV4/smAI=', 'Content-Length': '0'} 2023-09-22T21:49:10.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069dc6c54e5288466-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.560 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:49:10.561 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:49:10.561 DEBUG:boto:path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.561 DEBUG:boto:auth_path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.561 DEBUG:boto:Method: HEAD 2023-09-22T21:49:10.561 DEBUG:boto:Path: /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.561 DEBUG:boto:Data: 2023-09-22T21:49:10.561 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.561 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.561 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.561 DEBUG:boto:Params: {} 2023-09-22T21:49:10.562 DEBUG:boto:Token: None 2023-09-22T21:49:10.562 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.562 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jT0Ce2SK7vDyJub3FiNEUwLJ/jo= 2023-09-22T21:49:10.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jT0Ce2SK7vDyJub3FiNEUwLJ/jo=', 'Content-Length': '0'} 2023-09-22T21:49:10.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7d97792ed0f9343-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.565 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:49:10.565 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:49:10.565 DEBUG:boto:path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.565 DEBUG:boto:auth_path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.565 DEBUG:boto:Method: HEAD 2023-09-22T21:49:10.565 DEBUG:boto:Path: /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.566 DEBUG:boto:Data: 2023-09-22T21:49:10.566 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.566 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.566 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.566 DEBUG:boto:Params: {} 2023-09-22T21:49:10.566 DEBUG:boto:Token: None 2023-09-22T21:49:10.566 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.566 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jT0Ce2SK7vDyJub3FiNEUwLJ/jo= 2023-09-22T21:49:10.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jT0Ce2SK7vDyJub3FiNEUwLJ/jo=', 'Content-Length': '0'} 2023-09-22T21:49:10.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5e9745f8c0b49e3-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-22T21:49:10.570 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:49:10.570 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:49:10.570 DEBUG:boto:path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.570 DEBUG:boto:auth_path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.570 DEBUG:boto:Method: GET 2023-09-22T21:49:10.570 DEBUG:boto:Path: /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.570 DEBUG:boto:Data: 2023-09-22T21:49:10.570 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.570 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.571 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.571 DEBUG:boto:Params: {} 2023-09-22T21:49:10.571 DEBUG:boto:Token: None 2023-09-22T21:49:10.571 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.571 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:co3tmxpgxBzB2BEI9QKOV4/smAI= 2023-09-22T21:49:10.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:co3tmxpgxBzB2BEI9QKOV4/smAI=', 'Content-Length': '0'} 2023-09-22T21:49:10.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037bb1001d544630a-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.574 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:49:10.574 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:49:10.574 DEBUG:boto:path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.575 DEBUG:boto:auth_path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.575 DEBUG:boto:Method: GET 2023-09-22T21:49:10.575 DEBUG:boto:Path: /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.575 DEBUG:boto:Data: 2023-09-22T21:49:10.575 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.575 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.575 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.575 DEBUG:boto:Params: {} 2023-09-22T21:49:10.575 DEBUG:boto:Token: None 2023-09-22T21:49:10.575 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:10.576 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:co3tmxpgxBzB2BEI9QKOV4/smAI= 2023-09-22T21:49:10.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:co3tmxpgxBzB2BEI9QKOV4/smAI=', 'Content-Length': '0'} 2023-09-22T21:49:10.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b27740f39cb82522-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.579 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:10.579 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:10.579 DEBUG:boto:path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:49:10.579 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:49:10.580 DEBUG:boto:Method: GET 2023-09-22T21:49:10.580 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj31 2023-09-22T21:49:10.580 DEBUG:boto:Data: 2023-09-22T21:49:10.580 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.580 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.580 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.580 DEBUG:boto:Params: {} 2023-09-22T21:49:10.580 DEBUG:boto:Token: None 2023-09-22T21:49:10.580 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/ 2023-09-22T21:49:10.581 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE= 2023-09-22T21:49:10.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE=', 'Content-Length': '0'} 2023-09-22T21:49:10.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da92862f290888eb-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.589 DEBUG:boto:b'ffzdqu-59obj311000falseobj312023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:10.589 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:10.589 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:10.589 DEBUG:boto:path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:49:10.590 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:49:10.590 DEBUG:boto:Method: GET 2023-09-22T21:49:10.590 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj31 2023-09-22T21:49:10.590 DEBUG:boto:Data: 2023-09-22T21:49:10.590 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.590 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.590 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.590 DEBUG:boto:Params: {} 2023-09-22T21:49:10.590 DEBUG:boto:Token: None 2023-09-22T21:49:10.591 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/ 2023-09-22T21:49:10.591 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE= 2023-09-22T21:49:10.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE=', 'Content-Length': '0'} 2023-09-22T21:49:10.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012ccba85b2a09005-00650e0bd6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.642 DEBUG:boto:b'ffzdqu-59obj311000falseobj312023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:10.643 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:49:10.643 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:49:10.643 DEBUG:boto:Method: HEAD 2023-09-22T21:49:10.643 DEBUG:boto:Path: /ffzdqu-59/obj31 2023-09-22T21:49:10.643 DEBUG:boto:Data: 2023-09-22T21:49:10.643 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.644 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.644 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.644 DEBUG:boto:Params: {} 2023-09-22T21:49:10.644 DEBUG:boto:Token: None 2023-09-22T21:49:10.644 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj31 2023-09-22T21:49:10.644 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r3HDEhdq4mqH0zpz3tLlsks3jAQ= 2023-09-22T21:49:10.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r3HDEhdq4mqH0zpz3tLlsks3jAQ=', 'Content-Length': '0'} 2023-09-22T21:49:10.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071cdfe19ce947e7e-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.648 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:49:10.648 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:49:10.648 DEBUG:boto:Method: HEAD 2023-09-22T21:49:10.648 DEBUG:boto:Path: /ffzdqu-59/obj31 2023-09-22T21:49:10.648 DEBUG:boto:Data: 2023-09-22T21:49:10.648 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.648 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.649 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.649 DEBUG:boto:Params: {} 2023-09-22T21:49:10.649 DEBUG:boto:Token: None 2023-09-22T21:49:10.649 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj31 2023-09-22T21:49:10.649 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r3HDEhdq4mqH0zpz3tLlsks3jAQ= 2023-09-22T21:49:10.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r3HDEhdq4mqH0zpz3tLlsks3jAQ=', 'Content-Length': '0'} 2023-09-22T21:49:10.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6131d7ce9a04515-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-22T21:49:10.654 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:49:10.654 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:49:10.654 DEBUG:boto:Method: GET 2023-09-22T21:49:10.654 DEBUG:boto:Path: /ffzdqu-59/obj31 2023-09-22T21:49:10.654 DEBUG:boto:Data: 2023-09-22T21:49:10.654 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.654 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.654 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.655 DEBUG:boto:Params: {} 2023-09-22T21:49:10.655 DEBUG:boto:Token: None 2023-09-22T21:49:10.655 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj31 2023-09-22T21:49:10.655 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mTz8PBnUonjPU/tmLLWhD6TbSIU= 2023-09-22T21:49:10.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mTz8PBnUonjPU/tmLLWhD6TbSIU=', 'Content-Length': '0'} 2023-09-22T21:49:10.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2b0e5754c908b02-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.698 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:49:10.699 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:49:10.699 DEBUG:boto:Method: GET 2023-09-22T21:49:10.699 DEBUG:boto:Path: /ffzdqu-59/obj31 2023-09-22T21:49:10.699 DEBUG:boto:Data: 2023-09-22T21:49:10.699 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.699 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.699 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.699 DEBUG:boto:Params: {} 2023-09-22T21:49:10.699 DEBUG:boto:Token: None 2023-09-22T21:49:10.700 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj31 2023-09-22T21:49:10.700 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mTz8PBnUonjPU/tmLLWhD6TbSIU= 2023-09-22T21:49:10.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mTz8PBnUonjPU/tmLLWhD6TbSIU=', 'Content-Length': '0'} 2023-09-22T21:49:10.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbc4af4d2c92c6d1-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.703 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:10.703 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:10.703 DEBUG:boto:path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:49:10.704 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:49:10.704 DEBUG:boto:Method: GET 2023-09-22T21:49:10.704 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj31 2023-09-22T21:49:10.704 DEBUG:boto:Data: 2023-09-22T21:49:10.704 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.704 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.704 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.704 DEBUG:boto:Params: {} 2023-09-22T21:49:10.704 DEBUG:boto:Token: None 2023-09-22T21:49:10.705 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/ 2023-09-22T21:49:10.705 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE= 2023-09-22T21:49:10.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE=', 'Content-Length': '0'} 2023-09-22T21:49:10.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000642ca0cf4b3217d4-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.710 DEBUG:boto:b'ffzdqu-59obj311000falseobj312023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:10.710 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:10.710 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:10.710 DEBUG:boto:path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:49:10.711 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:49:10.711 DEBUG:boto:Method: GET 2023-09-22T21:49:10.711 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj31 2023-09-22T21:49:10.711 DEBUG:boto:Data: 2023-09-22T21:49:10.711 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.711 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.711 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.711 DEBUG:boto:Params: {} 2023-09-22T21:49:10.711 DEBUG:boto:Token: None 2023-09-22T21:49:10.712 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/ 2023-09-22T21:49:10.712 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE= 2023-09-22T21:49:10.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE=', 'Content-Length': '0'} 2023-09-22T21:49:10.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6c3557ca93823bb-00650e0bd6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.722 DEBUG:boto:b'ffzdqu-59obj311000falseobj312023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:10.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:49:10.723 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:49:10.723 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:49:10.723 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.723 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.723 DEBUG:boto:Method: GET 2023-09-22T21:49:10.723 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.723 DEBUG:boto:Data: 2023-09-22T21:49:10.724 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.724 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.724 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.724 DEBUG:boto:Params: {} 2023-09-22T21:49:10.724 DEBUG:boto:Token: None 2023-09-22T21:49:10.724 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.724 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jExv662jHsGZkVbZw/t/YBhc0mg= 2023-09-22T21:49:10.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jExv662jHsGZkVbZw/t/YBhc0mg=', 'Content-Length': '0'} 2023-09-22T21:49:10.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066c597b0d1bedc2d-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.727 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:49:10.727 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:49:10.728 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.728 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.728 DEBUG:boto:Method: GET 2023-09-22T21:49:10.728 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.728 DEBUG:boto:Data: 2023-09-22T21:49:10.728 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.728 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.728 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.728 DEBUG:boto:Params: {} 2023-09-22T21:49:10.728 DEBUG:boto:Token: None 2023-09-22T21:49:10.729 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.729 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jExv662jHsGZkVbZw/t/YBhc0mg= 2023-09-22T21:49:10.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jExv662jHsGZkVbZw/t/YBhc0mg=', 'Content-Length': '0'} 2023-09-22T21:49:10.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b14c56f750a84c0-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.774 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:49:10.774 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:49:10.775 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.775 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.775 DEBUG:boto:Method: HEAD 2023-09-22T21:49:10.775 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.775 DEBUG:boto:Data: 2023-09-22T21:49:10.775 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.775 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.775 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.775 DEBUG:boto:Params: {} 2023-09-22T21:49:10.776 DEBUG:boto:Token: None 2023-09-22T21:49:10.776 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.776 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J94C8u8GLrgRJgsHlVAWQFIMxT8= 2023-09-22T21:49:10.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J94C8u8GLrgRJgsHlVAWQFIMxT8=', 'Content-Length': '0'} 2023-09-22T21:49:10.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0188bb7769b11f4-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.779 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:49:10.780 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:49:10.780 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.780 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.780 DEBUG:boto:Method: HEAD 2023-09-22T21:49:10.780 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.780 DEBUG:boto:Data: 2023-09-22T21:49:10.780 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.780 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.780 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.781 DEBUG:boto:Params: {} 2023-09-22T21:49:10.781 DEBUG:boto:Token: None 2023-09-22T21:49:10.781 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.781 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J94C8u8GLrgRJgsHlVAWQFIMxT8= 2023-09-22T21:49:10.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J94C8u8GLrgRJgsHlVAWQFIMxT8=', 'Content-Length': '0'} 2023-09-22T21:49:10.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc883b16b5a1aed1-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:49:10.785 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:49:10.785 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:49:10.785 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.785 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.785 DEBUG:boto:Method: GET 2023-09-22T21:49:10.785 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.785 DEBUG:boto:Data: 2023-09-22T21:49:10.785 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.786 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.786 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.786 DEBUG:boto:Params: {} 2023-09-22T21:49:10.786 DEBUG:boto:Token: None 2023-09-22T21:49:10.786 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.786 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jExv662jHsGZkVbZw/t/YBhc0mg= 2023-09-22T21:49:10.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jExv662jHsGZkVbZw/t/YBhc0mg=', 'Content-Length': '0'} 2023-09-22T21:49:10.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089312d9e529161ef-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.830 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:49:10.830 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:49:10.831 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.831 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.831 DEBUG:boto:Method: GET 2023-09-22T21:49:10.831 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.831 DEBUG:boto:Data: 2023-09-22T21:49:10.831 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.831 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.831 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.831 DEBUG:boto:Params: {} 2023-09-22T21:49:10.832 DEBUG:boto:Token: None 2023-09-22T21:49:10.832 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:10.832 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jExv662jHsGZkVbZw/t/YBhc0mg= 2023-09-22T21:49:10.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jExv662jHsGZkVbZw/t/YBhc0mg=', 'Content-Length': '0'} 2023-09-22T21:49:10.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c55e1746b31f28d0-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.835 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:10.835 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:10.835 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:49:10.835 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:49:10.835 DEBUG:boto:Method: GET 2023-09-22T21:49:10.835 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:49:10.836 DEBUG:boto:Data: 2023-09-22T21:49:10.836 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.836 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.836 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.836 DEBUG:boto:Params: {} 2023-09-22T21:49:10.836 DEBUG:boto:Token: None 2023-09-22T21:49:10.836 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/ 2023-09-22T21:49:10.836 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE= 2023-09-22T21:49:10.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE=', 'Content-Length': '0'} 2023-09-22T21:49:10.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077fa6327cf1fe4df-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.842 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:10.842 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:10.842 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:10.842 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:49:10.842 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:49:10.842 DEBUG:boto:Method: GET 2023-09-22T21:49:10.843 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:49:10.843 DEBUG:boto:Data: 2023-09-22T21:49:10.843 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.843 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.843 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.843 DEBUG:boto:Params: {} 2023-09-22T21:49:10.843 DEBUG:boto:Token: None 2023-09-22T21:49:10.844 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/ 2023-09-22T21:49:10.844 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE= 2023-09-22T21:49:10.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE=', 'Content-Length': '0'} 2023-09-22T21:49:10.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c145e7eda3e6e9e3-00650e0bd6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.855 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:10.855 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:49:10.855 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:49:10.855 DEBUG:boto:Method: HEAD 2023-09-22T21:49:10.855 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:49:10.856 DEBUG:boto:Data: 2023-09-22T21:49:10.856 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.856 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.856 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.856 DEBUG:boto:Params: {} 2023-09-22T21:49:10.856 DEBUG:boto:Token: None 2023-09-22T21:49:10.856 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj33 2023-09-22T21:49:10.856 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2+QwOfsNUHxFt/a6yz03gBYafEQ= 2023-09-22T21:49:10.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2+QwOfsNUHxFt/a6yz03gBYafEQ=', 'Content-Length': '0'} 2023-09-22T21:49:10.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e634e7979de329f-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.860 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:49:10.861 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:49:10.861 DEBUG:boto:Method: HEAD 2023-09-22T21:49:10.861 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:49:10.861 DEBUG:boto:Data: 2023-09-22T21:49:10.861 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.861 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.861 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.861 DEBUG:boto:Params: {} 2023-09-22T21:49:10.862 DEBUG:boto:Token: None 2023-09-22T21:49:10.862 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj33 2023-09-22T21:49:10.862 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2+QwOfsNUHxFt/a6yz03gBYafEQ= 2023-09-22T21:49:10.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2+QwOfsNUHxFt/a6yz03gBYafEQ=', 'Content-Length': '0'} 2023-09-22T21:49:10.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064ff6c02661bee6a-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:49:10.865 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:49:10.865 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:49:10.865 DEBUG:boto:Method: GET 2023-09-22T21:49:10.866 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:49:10.866 DEBUG:boto:Data: 2023-09-22T21:49:10.866 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.866 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.866 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.866 DEBUG:boto:Params: {} 2023-09-22T21:49:10.866 DEBUG:boto:Token: None 2023-09-22T21:49:10.866 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj33 2023-09-22T21:49:10.867 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:z6umzZcLztH2/ZTTsXPveqrNtLw= 2023-09-22T21:49:10.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:z6umzZcLztH2/ZTTsXPveqrNtLw=', 'Content-Length': '0'} 2023-09-22T21:49:10.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006993df41affec1f8-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.914 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:49:10.914 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:49:10.915 DEBUG:boto:Method: GET 2023-09-22T21:49:10.915 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:49:10.915 DEBUG:boto:Data: 2023-09-22T21:49:10.915 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.915 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.915 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.915 DEBUG:boto:Params: {} 2023-09-22T21:49:10.915 DEBUG:boto:Token: None 2023-09-22T21:49:10.915 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj33 2023-09-22T21:49:10.916 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:z6umzZcLztH2/ZTTsXPveqrNtLw= 2023-09-22T21:49:10.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:z6umzZcLztH2/ZTTsXPveqrNtLw=', 'Content-Length': '0'} 2023-09-22T21:49:10.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f88ea440d74c8ee1-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.919 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:10.919 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:10.919 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:49:10.919 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:49:10.919 DEBUG:boto:Method: GET 2023-09-22T21:49:10.919 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:49:10.919 DEBUG:boto:Data: 2023-09-22T21:49:10.919 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.920 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.920 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.920 DEBUG:boto:Params: {} 2023-09-22T21:49:10.920 DEBUG:boto:Token: None 2023-09-22T21:49:10.920 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/ 2023-09-22T21:49:10.920 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE= 2023-09-22T21:49:10.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE=', 'Content-Length': '0'} 2023-09-22T21:49:10.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000052b0cd606037ecf-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.925 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:10.925 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:10.925 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:10.925 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:49:10.926 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:49:10.926 DEBUG:boto:Method: GET 2023-09-22T21:49:10.926 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:49:10.926 DEBUG:boto:Data: 2023-09-22T21:49:10.926 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.926 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.926 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.926 DEBUG:boto:Params: {} 2023-09-22T21:49:10.926 DEBUG:boto:Token: None 2023-09-22T21:49:10.927 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/ 2023-09-22T21:49:10.927 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE= 2023-09-22T21:49:10.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:25CACtdn7aau+I5dk5TtxEDbvIE=', 'Content-Length': '0'} 2023-09-22T21:49:10.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa181c41a171c3bf-00650e0bd6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.938 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:10.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:49:10.938 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:49:10.939 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:49:10.939 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:10.939 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:10.939 DEBUG:boto:Method: GET 2023-09-22T21:49:10.939 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:10.939 DEBUG:boto:Data: 2023-09-22T21:49:10.939 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.939 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.939 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.939 DEBUG:boto:Params: {} 2023-09-22T21:49:10.940 DEBUG:boto:Token: None 2023-09-22T21:49:10.940 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:10.940 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DXJuFQpSJ+zykPyybq2XpPkv0jI= 2023-09-22T21:49:10.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DXJuFQpSJ+zykPyybq2XpPkv0jI=', 'Content-Length': '0'} 2023-09-22T21:49:10.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e385433ff5f2ec03-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.943 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:49:10.943 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:49:10.943 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:10.943 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:10.943 DEBUG:boto:Method: GET 2023-09-22T21:49:10.944 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:10.944 DEBUG:boto:Data: 2023-09-22T21:49:10.944 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.944 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.944 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.944 DEBUG:boto:Params: {} 2023-09-22T21:49:10.944 DEBUG:boto:Token: None 2023-09-22T21:49:10.944 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:10.944 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DXJuFQpSJ+zykPyybq2XpPkv0jI= 2023-09-22T21:49:10.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DXJuFQpSJ+zykPyybq2XpPkv0jI=', 'Content-Length': '0'} 2023-09-22T21:49:10.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a9d120e51a669b4-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.948 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:49:10.948 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:49:10.948 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:10.948 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:10.948 DEBUG:boto:Method: HEAD 2023-09-22T21:49:10.948 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:10.948 DEBUG:boto:Data: 2023-09-22T21:49:10.949 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.949 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.949 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.949 DEBUG:boto:Params: {} 2023-09-22T21:49:10.949 DEBUG:boto:Token: None 2023-09-22T21:49:10.949 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:10.949 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iJykOZ48jrxfIlxxP5UdBbLh5fs= 2023-09-22T21:49:10.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iJykOZ48jrxfIlxxP5UdBbLh5fs=', 'Content-Length': '0'} 2023-09-22T21:49:10.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bb635b6a02837ea-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.952 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:49:10.952 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:49:10.952 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:10.952 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:10.953 DEBUG:boto:Method: HEAD 2023-09-22T21:49:10.953 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:10.953 DEBUG:boto:Data: 2023-09-22T21:49:10.953 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.953 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:10.953 DEBUG:boto:Port: 8000 2023-09-22T21:49:10.953 DEBUG:boto:Params: {} 2023-09-22T21:49:10.953 DEBUG:boto:Token: None 2023-09-22T21:49:10.953 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:10.953 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iJykOZ48jrxfIlxxP5UdBbLh5fs= 2023-09-22T21:49:10.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iJykOZ48jrxfIlxxP5UdBbLh5fs=', 'Content-Length': '0'} 2023-09-22T21:49:10.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000294a83cd90fcef8d-00650e0bd6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:10.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:49:10.957 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:49:10.957 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:49:10.957 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:10.957 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:10.957 DEBUG:boto:Method: GET 2023-09-22T21:49:10.957 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:10.957 DEBUG:boto:Data: 2023-09-22T21:49:10.957 DEBUG:boto:Headers: {} 2023-09-22T21:49:10.957 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:10.957 DEBUG:boto:Port: 8001 2023-09-22T21:49:10.958 DEBUG:boto:Params: {} 2023-09-22T21:49:10.958 DEBUG:boto:Token: None 2023-09-22T21:49:10.958 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:10 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:10.958 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DXJuFQpSJ+zykPyybq2XpPkv0jI= 2023-09-22T21:49:10.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DXJuFQpSJ+zykPyybq2XpPkv0jI=', 'Content-Length': '0'} 2023-09-22T21:49:10.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091053ece1aa9f1d7-00650e0bd6-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.002 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:49:11.003 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:49:11.003 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:11.003 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:11.003 DEBUG:boto:Method: GET 2023-09-22T21:49:11.003 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:11.003 DEBUG:boto:Data: 2023-09-22T21:49:11.003 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.003 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.003 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.004 DEBUG:boto:Params: {} 2023-09-22T21:49:11.004 DEBUG:boto:Token: None 2023-09-22T21:49:11.004 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:11.004 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MCpYhM8SLP2w93Z6gQLMJI3BaUw= 2023-09-22T21:49:11.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MCpYhM8SLP2w93Z6gQLMJI3BaUw=', 'Content-Length': '0'} 2023-09-22T21:49:11.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afc1a306646834d9-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.007 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.008 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.008 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:49:11.008 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:49:11.008 DEBUG:boto:Method: GET 2023-09-22T21:49:11.008 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:49:11.008 DEBUG:boto:Data: 2023-09-22T21:49:11.008 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.008 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.008 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.009 DEBUG:boto:Params: {} 2023-09-22T21:49:11.009 DEBUG:boto:Token: None 2023-09-22T21:49:11.009 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.009 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a27968151762269-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.014 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:11.014 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.014 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.014 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:49:11.014 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:49:11.015 DEBUG:boto:Method: GET 2023-09-22T21:49:11.015 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:49:11.015 DEBUG:boto:Data: 2023-09-22T21:49:11.015 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.015 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.015 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.015 DEBUG:boto:Params: {} 2023-09-22T21:49:11.015 DEBUG:boto:Token: None 2023-09-22T21:49:11.016 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.016 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000463418bbf3b8a8f3-00650e0bd7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.026 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:11.027 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:49:11.027 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:49:11.027 DEBUG:boto:Method: HEAD 2023-09-22T21:49:11.027 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:49:11.027 DEBUG:boto:Data: 2023-09-22T21:49:11.027 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.027 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.027 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.027 DEBUG:boto:Params: {} 2023-09-22T21:49:11.028 DEBUG:boto:Token: None 2023-09-22T21:49:11.028 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj34 2023-09-22T21:49:11.028 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qmTKS8ZKXpAlHuvVwicwbloYe9w= 2023-09-22T21:49:11.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qmTKS8ZKXpAlHuvVwicwbloYe9w=', 'Content-Length': '0'} 2023-09-22T21:49:11.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d7dd00827d33e54-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.031 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:49:11.031 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:49:11.031 DEBUG:boto:Method: HEAD 2023-09-22T21:49:11.031 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:49:11.031 DEBUG:boto:Data: 2023-09-22T21:49:11.031 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.032 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.032 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.032 DEBUG:boto:Params: {} 2023-09-22T21:49:11.032 DEBUG:boto:Token: None 2023-09-22T21:49:11.032 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj34 2023-09-22T21:49:11.032 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qmTKS8ZKXpAlHuvVwicwbloYe9w= 2023-09-22T21:49:11.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qmTKS8ZKXpAlHuvVwicwbloYe9w=', 'Content-Length': '0'} 2023-09-22T21:49:11.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb5464a655e918f8-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:49:11.035 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:49:11.035 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:49:11.035 DEBUG:boto:Method: GET 2023-09-22T21:49:11.036 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:49:11.036 DEBUG:boto:Data: 2023-09-22T21:49:11.036 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.036 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.036 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.036 DEBUG:boto:Params: {} 2023-09-22T21:49:11.036 DEBUG:boto:Token: None 2023-09-22T21:49:11.036 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj34 2023-09-22T21:49:11.036 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lxnPwwsO0c03Sx1Q7UaT2QqV7ds= 2023-09-22T21:49:11.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lxnPwwsO0c03Sx1Q7UaT2QqV7ds=', 'Content-Length': '0'} 2023-09-22T21:49:11.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b20b2c46aee5a3b1-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.086 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:49:11.087 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:49:11.087 DEBUG:boto:Method: GET 2023-09-22T21:49:11.087 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:49:11.087 DEBUG:boto:Data: 2023-09-22T21:49:11.087 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.087 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.087 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.087 DEBUG:boto:Params: {} 2023-09-22T21:49:11.087 DEBUG:boto:Token: None 2023-09-22T21:49:11.088 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj34 2023-09-22T21:49:11.088 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lxnPwwsO0c03Sx1Q7UaT2QqV7ds= 2023-09-22T21:49:11.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lxnPwwsO0c03Sx1Q7UaT2QqV7ds=', 'Content-Length': '0'} 2023-09-22T21:49:11.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6a4778994b9467d-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.091 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.091 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.091 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:49:11.091 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:49:11.092 DEBUG:boto:Method: GET 2023-09-22T21:49:11.092 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:49:11.092 DEBUG:boto:Data: 2023-09-22T21:49:11.092 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.092 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.092 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.092 DEBUG:boto:Params: {} 2023-09-22T21:49:11.092 DEBUG:boto:Token: None 2023-09-22T21:49:11.093 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.093 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040d2878378347df3-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.098 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:11.099 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.099 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.099 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:49:11.099 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:49:11.100 DEBUG:boto:Method: GET 2023-09-22T21:49:11.100 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:49:11.100 DEBUG:boto:Data: 2023-09-22T21:49:11.100 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.101 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.101 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.101 DEBUG:boto:Params: {} 2023-09-22T21:49:11.101 DEBUG:boto:Token: None 2023-09-22T21:49:11.102 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.102 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012d0d71e4e731f44-00650e0bd7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.154 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:11.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-09-22T21:49:11.155 DEBUG:boto:path=/ffzdqu-59/obj37 2023-09-22T21:49:11.155 DEBUG:boto:auth_path=/ffzdqu-59/obj37 2023-09-22T21:49:11.155 DEBUG:boto:path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.155 DEBUG:boto:auth_path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.156 DEBUG:boto:Method: GET 2023-09-22T21:49:11.156 DEBUG:boto:Path: /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.156 DEBUG:boto:Data: 2023-09-22T21:49:11.156 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.156 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.156 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.156 DEBUG:boto:Params: {} 2023-09-22T21:49:11.156 DEBUG:boto:Token: None 2023-09-22T21:49:11.156 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.157 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Wj9DBeegQ1VCy/Z39+Ro0TyK/OE= 2023-09-22T21:49:11.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Wj9DBeegQ1VCy/Z39+Ro0TyK/OE=', 'Content-Length': '0'} 2023-09-22T21:49:11.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009408f4ac5fcd62c0-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.160 DEBUG:boto:path=/ffzdqu-59/obj37 2023-09-22T21:49:11.161 DEBUG:boto:auth_path=/ffzdqu-59/obj37 2023-09-22T21:49:11.161 DEBUG:boto:path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.161 DEBUG:boto:auth_path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.161 DEBUG:boto:Method: GET 2023-09-22T21:49:11.161 DEBUG:boto:Path: /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.161 DEBUG:boto:Data: 2023-09-22T21:49:11.161 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.161 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.161 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.162 DEBUG:boto:Params: {} 2023-09-22T21:49:11.162 DEBUG:boto:Token: None 2023-09-22T21:49:11.162 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.162 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Wj9DBeegQ1VCy/Z39+Ro0TyK/OE= 2023-09-22T21:49:11.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Wj9DBeegQ1VCy/Z39+Ro0TyK/OE=', 'Content-Length': '0'} 2023-09-22T21:49:11.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4252ee819c413a0-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.166 DEBUG:boto:path=/ffzdqu-59/obj37 2023-09-22T21:49:11.166 DEBUG:boto:auth_path=/ffzdqu-59/obj37 2023-09-22T21:49:11.166 DEBUG:boto:path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.166 DEBUG:boto:auth_path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.166 DEBUG:boto:Method: HEAD 2023-09-22T21:49:11.166 DEBUG:boto:Path: /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.166 DEBUG:boto:Data: 2023-09-22T21:49:11.166 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.167 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.167 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.167 DEBUG:boto:Params: {} 2023-09-22T21:49:11.167 DEBUG:boto:Token: None 2023-09-22T21:49:11.167 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.167 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UFRDCqAIfWLLUk8NOQYUR9Bc36M= 2023-09-22T21:49:11.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UFRDCqAIfWLLUk8NOQYUR9Bc36M=', 'Content-Length': '0'} 2023-09-22T21:49:11.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb909c3817c193f3-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.170 DEBUG:boto:path=/ffzdqu-59/obj37 2023-09-22T21:49:11.170 DEBUG:boto:auth_path=/ffzdqu-59/obj37 2023-09-22T21:49:11.171 DEBUG:boto:path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.171 DEBUG:boto:auth_path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.171 DEBUG:boto:Method: HEAD 2023-09-22T21:49:11.171 DEBUG:boto:Path: /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.171 DEBUG:boto:Data: 2023-09-22T21:49:11.171 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.171 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.171 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.171 DEBUG:boto:Params: {} 2023-09-22T21:49:11.172 DEBUG:boto:Token: None 2023-09-22T21:49:11.172 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.172 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UFRDCqAIfWLLUk8NOQYUR9Bc36M= 2023-09-22T21:49:11.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UFRDCqAIfWLLUk8NOQYUR9Bc36M=', 'Content-Length': '0'} 2023-09-22T21:49:11.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df385c7c5da1e6f1-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-09-22T21:49:11.176 DEBUG:boto:path=/ffzdqu-59/obj37 2023-09-22T21:49:11.176 DEBUG:boto:auth_path=/ffzdqu-59/obj37 2023-09-22T21:49:11.176 DEBUG:boto:path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.176 DEBUG:boto:auth_path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.176 DEBUG:boto:Method: GET 2023-09-22T21:49:11.176 DEBUG:boto:Path: /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.176 DEBUG:boto:Data: 2023-09-22T21:49:11.176 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.177 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.177 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.177 DEBUG:boto:Params: {} 2023-09-22T21:49:11.177 DEBUG:boto:Token: None 2023-09-22T21:49:11.177 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.177 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Wj9DBeegQ1VCy/Z39+Ro0TyK/OE= 2023-09-22T21:49:11.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Wj9DBeegQ1VCy/Z39+Ro0TyK/OE=', 'Content-Length': '0'} 2023-09-22T21:49:11.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a50fcb72cbb6099-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.180 DEBUG:boto:path=/ffzdqu-59/obj37 2023-09-22T21:49:11.180 DEBUG:boto:auth_path=/ffzdqu-59/obj37 2023-09-22T21:49:11.181 DEBUG:boto:path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.181 DEBUG:boto:auth_path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.181 DEBUG:boto:Method: GET 2023-09-22T21:49:11.181 DEBUG:boto:Path: /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.181 DEBUG:boto:Data: 2023-09-22T21:49:11.181 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.181 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.181 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.182 DEBUG:boto:Params: {} 2023-09-22T21:49:11.182 DEBUG:boto:Token: None 2023-09-22T21:49:11.182 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:11.182 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Wj9DBeegQ1VCy/Z39+Ro0TyK/OE= 2023-09-22T21:49:11.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Wj9DBeegQ1VCy/Z39+Ro0TyK/OE=', 'Content-Length': '0'} 2023-09-22T21:49:11.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db46820f76f2f863-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.186 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.186 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.186 DEBUG:boto:path=/ffzdqu-59/?prefix=obj37 2023-09-22T21:49:11.186 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj37 2023-09-22T21:49:11.186 DEBUG:boto:Method: GET 2023-09-22T21:49:11.186 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj37 2023-09-22T21:49:11.186 DEBUG:boto:Data: 2023-09-22T21:49:11.186 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.187 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.187 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.187 DEBUG:boto:Params: {} 2023-09-22T21:49:11.187 DEBUG:boto:Token: None 2023-09-22T21:49:11.187 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.187 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0352a53640a5cb1-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.234 DEBUG:boto:b'ffzdqu-59obj371000falseobj372023-09-22T21:47:16.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:11.235 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.235 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.235 DEBUG:boto:path=/ffzdqu-59/?prefix=obj37 2023-09-22T21:49:11.235 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj37 2023-09-22T21:49:11.236 DEBUG:boto:Method: GET 2023-09-22T21:49:11.236 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj37 2023-09-22T21:49:11.236 DEBUG:boto:Data: 2023-09-22T21:49:11.236 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.237 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.237 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.237 DEBUG:boto:Params: {} 2023-09-22T21:49:11.237 DEBUG:boto:Token: None 2023-09-22T21:49:11.237 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.238 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025073020e1b5342b-00650e0bd7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.247 DEBUG:boto:b'ffzdqu-59obj371000falseobj372023-09-22T21:47:16.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:11.248 DEBUG:boto:path=/ffzdqu-59/obj37 2023-09-22T21:49:11.248 DEBUG:boto:auth_path=/ffzdqu-59/obj37 2023-09-22T21:49:11.248 DEBUG:boto:Method: HEAD 2023-09-22T21:49:11.248 DEBUG:boto:Path: /ffzdqu-59/obj37 2023-09-22T21:49:11.249 DEBUG:boto:Data: 2023-09-22T21:49:11.249 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.249 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.249 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.249 DEBUG:boto:Params: {} 2023-09-22T21:49:11.250 DEBUG:boto:Token: None 2023-09-22T21:49:11.250 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj37 2023-09-22T21:49:11.250 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aEtmCzS4FMf11B1tuHBzA4B2ClE= 2023-09-22T21:49:11.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aEtmCzS4FMf11B1tuHBzA4B2ClE=', 'Content-Length': '0'} 2023-09-22T21:49:11.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2c83a2123a2751c-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.253 DEBUG:boto:path=/ffzdqu-59/obj37 2023-09-22T21:49:11.254 DEBUG:boto:auth_path=/ffzdqu-59/obj37 2023-09-22T21:49:11.254 DEBUG:boto:Method: HEAD 2023-09-22T21:49:11.254 DEBUG:boto:Path: /ffzdqu-59/obj37 2023-09-22T21:49:11.254 DEBUG:boto:Data: 2023-09-22T21:49:11.255 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.255 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.255 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.255 DEBUG:boto:Params: {} 2023-09-22T21:49:11.255 DEBUG:boto:Token: None 2023-09-22T21:49:11.256 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj37 2023-09-22T21:49:11.256 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aEtmCzS4FMf11B1tuHBzA4B2ClE= 2023-09-22T21:49:11.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aEtmCzS4FMf11B1tuHBzA4B2ClE=', 'Content-Length': '0'} 2023-09-22T21:49:11.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdb239ff09361712-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-09-22T21:49:11.259 DEBUG:boto:path=/ffzdqu-59/obj37 2023-09-22T21:49:11.260 DEBUG:boto:auth_path=/ffzdqu-59/obj37 2023-09-22T21:49:11.260 DEBUG:boto:Method: GET 2023-09-22T21:49:11.260 DEBUG:boto:Path: /ffzdqu-59/obj37 2023-09-22T21:49:11.260 DEBUG:boto:Data: 2023-09-22T21:49:11.260 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.261 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.261 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.261 DEBUG:boto:Params: {} 2023-09-22T21:49:11.261 DEBUG:boto:Token: None 2023-09-22T21:49:11.261 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj37 2023-09-22T21:49:11.262 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:l7+NOzRT2TYmW9U5psl+yWhnlP4= 2023-09-22T21:49:11.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:l7+NOzRT2TYmW9U5psl+yWhnlP4=', 'Content-Length': '0'} 2023-09-22T21:49:11.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005749a464ea6a9453-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.265 DEBUG:boto:path=/ffzdqu-59/obj37 2023-09-22T21:49:11.265 DEBUG:boto:auth_path=/ffzdqu-59/obj37 2023-09-22T21:49:11.266 DEBUG:boto:Method: GET 2023-09-22T21:49:11.266 DEBUG:boto:Path: /ffzdqu-59/obj37 2023-09-22T21:49:11.266 DEBUG:boto:Data: 2023-09-22T21:49:11.266 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.266 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.267 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.267 DEBUG:boto:Params: {} 2023-09-22T21:49:11.267 DEBUG:boto:Token: None 2023-09-22T21:49:11.267 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj37 2023-09-22T21:49:11.267 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:l7+NOzRT2TYmW9U5psl+yWhnlP4= 2023-09-22T21:49:11.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:l7+NOzRT2TYmW9U5psl+yWhnlP4=', 'Content-Length': '0'} 2023-09-22T21:49:11.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a119b0adb6d88d16-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.318 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.318 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.319 DEBUG:boto:path=/ffzdqu-59/?prefix=obj37 2023-09-22T21:49:11.319 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj37 2023-09-22T21:49:11.319 DEBUG:boto:Method: GET 2023-09-22T21:49:11.319 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj37 2023-09-22T21:49:11.319 DEBUG:boto:Data: 2023-09-22T21:49:11.319 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.319 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.319 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.319 DEBUG:boto:Params: {} 2023-09-22T21:49:11.320 DEBUG:boto:Token: None 2023-09-22T21:49:11.320 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.320 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3d70c46a774f93f-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.325 DEBUG:boto:b'ffzdqu-59obj371000falseobj372023-09-22T21:47:16.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:11.325 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.325 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.325 DEBUG:boto:path=/ffzdqu-59/?prefix=obj37 2023-09-22T21:49:11.325 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj37 2023-09-22T21:49:11.326 DEBUG:boto:Method: GET 2023-09-22T21:49:11.326 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj37 2023-09-22T21:49:11.326 DEBUG:boto:Data: 2023-09-22T21:49:11.326 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.326 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.326 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.326 DEBUG:boto:Params: {} 2023-09-22T21:49:11.326 DEBUG:boto:Token: None 2023-09-22T21:49:11.326 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.327 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fc12ff50964ece0-00650e0bd7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.338 DEBUG:boto:b'ffzdqu-59obj371000falseobj372023-09-22T21:47:16.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:11.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T21:49:11.338 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:49:11.338 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:49:11.338 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.338 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.339 DEBUG:boto:Method: GET 2023-09-22T21:49:11.339 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.339 DEBUG:boto:Data: 2023-09-22T21:49:11.339 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.339 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.339 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.339 DEBUG:boto:Params: {} 2023-09-22T21:49:11.339 DEBUG:boto:Token: None 2023-09-22T21:49:11.339 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.340 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rarDiYfFKLeEdxhhFykvemM8YCQ= 2023-09-22T21:49:11.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rarDiYfFKLeEdxhhFykvemM8YCQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fe5abffff69836e-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.343 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:49:11.343 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:49:11.343 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.343 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.343 DEBUG:boto:Method: GET 2023-09-22T21:49:11.343 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.344 DEBUG:boto:Data: 2023-09-22T21:49:11.344 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.344 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.344 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.344 DEBUG:boto:Params: {} 2023-09-22T21:49:11.344 DEBUG:boto:Token: None 2023-09-22T21:49:11.344 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.344 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rarDiYfFKLeEdxhhFykvemM8YCQ= 2023-09-22T21:49:11.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rarDiYfFKLeEdxhhFykvemM8YCQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005359e09f15eda70d-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.348 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:49:11.348 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:49:11.348 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.348 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.348 DEBUG:boto:Method: HEAD 2023-09-22T21:49:11.348 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.348 DEBUG:boto:Data: 2023-09-22T21:49:11.348 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.348 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.349 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.349 DEBUG:boto:Params: {} 2023-09-22T21:49:11.349 DEBUG:boto:Token: None 2023-09-22T21:49:11.349 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.349 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Wh/rkYer1Sw9WaPzxpHthbDabbU= 2023-09-22T21:49:11.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Wh/rkYer1Sw9WaPzxpHthbDabbU=', 'Content-Length': '0'} 2023-09-22T21:49:11.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053e6827bb5688880-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.352 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:49:11.352 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:49:11.352 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.352 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.353 DEBUG:boto:Method: HEAD 2023-09-22T21:49:11.353 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.353 DEBUG:boto:Data: 2023-09-22T21:49:11.353 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.353 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.353 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.353 DEBUG:boto:Params: {} 2023-09-22T21:49:11.353 DEBUG:boto:Token: None 2023-09-22T21:49:11.353 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.354 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Wh/rkYer1Sw9WaPzxpHthbDabbU= 2023-09-22T21:49:11.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Wh/rkYer1Sw9WaPzxpHthbDabbU=', 'Content-Length': '0'} 2023-09-22T21:49:11.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000df6a952c1f27951-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T21:49:11.357 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:49:11.357 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:49:11.357 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.357 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.357 DEBUG:boto:Method: GET 2023-09-22T21:49:11.357 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.357 DEBUG:boto:Data: 2023-09-22T21:49:11.357 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.358 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.358 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.358 DEBUG:boto:Params: {} 2023-09-22T21:49:11.358 DEBUG:boto:Token: None 2023-09-22T21:49:11.358 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.358 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rarDiYfFKLeEdxhhFykvemM8YCQ= 2023-09-22T21:49:11.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rarDiYfFKLeEdxhhFykvemM8YCQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082d63e8473edefe6-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.406 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:49:11.406 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:49:11.407 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.407 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.407 DEBUG:boto:Method: GET 2023-09-22T21:49:11.407 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.407 DEBUG:boto:Data: 2023-09-22T21:49:11.407 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.407 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.407 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.407 DEBUG:boto:Params: {} 2023-09-22T21:49:11.408 DEBUG:boto:Token: None 2023-09-22T21:49:11.408 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:11.408 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rarDiYfFKLeEdxhhFykvemM8YCQ= 2023-09-22T21:49:11.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rarDiYfFKLeEdxhhFykvemM8YCQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097dc2bab0fff87bd-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.411 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.411 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.412 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:49:11.412 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:49:11.412 DEBUG:boto:Method: GET 2023-09-22T21:49:11.412 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:49:11.412 DEBUG:boto:Data: 2023-09-22T21:49:11.412 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.412 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.412 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.412 DEBUG:boto:Params: {} 2023-09-22T21:49:11.413 DEBUG:boto:Token: None 2023-09-22T21:49:11.413 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.413 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f693359e9b14e7a-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.418 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:11.418 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.418 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.418 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:49:11.418 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:49:11.418 DEBUG:boto:Method: GET 2023-09-22T21:49:11.419 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:49:11.419 DEBUG:boto:Data: 2023-09-22T21:49:11.419 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.419 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.419 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.419 DEBUG:boto:Params: {} 2023-09-22T21:49:11.419 DEBUG:boto:Token: None 2023-09-22T21:49:11.419 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.419 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006381b2e322098ced-00650e0bd7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.429 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:11.430 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:49:11.430 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:49:11.430 DEBUG:boto:Method: HEAD 2023-09-22T21:49:11.430 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:49:11.430 DEBUG:boto:Data: 2023-09-22T21:49:11.430 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.431 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.431 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.431 DEBUG:boto:Params: {} 2023-09-22T21:49:11.431 DEBUG:boto:Token: None 2023-09-22T21:49:11.431 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj38 2023-09-22T21:49:11.431 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ILz8BYzZjkD3QGU/2qlV4y+binc= 2023-09-22T21:49:11.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ILz8BYzZjkD3QGU/2qlV4y+binc=', 'Content-Length': '0'} 2023-09-22T21:49:11.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b05c34442968a9be-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.434 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:49:11.434 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:49:11.434 DEBUG:boto:Method: HEAD 2023-09-22T21:49:11.434 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:49:11.434 DEBUG:boto:Data: 2023-09-22T21:49:11.435 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.435 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.435 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.435 DEBUG:boto:Params: {} 2023-09-22T21:49:11.435 DEBUG:boto:Token: None 2023-09-22T21:49:11.435 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj38 2023-09-22T21:49:11.435 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ILz8BYzZjkD3QGU/2qlV4y+binc= 2023-09-22T21:49:11.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ILz8BYzZjkD3QGU/2qlV4y+binc=', 'Content-Length': '0'} 2023-09-22T21:49:11.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004523f996e51f7684-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T21:49:11.438 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:49:11.439 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:49:11.439 DEBUG:boto:Method: GET 2023-09-22T21:49:11.439 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:49:11.439 DEBUG:boto:Data: 2023-09-22T21:49:11.439 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.439 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.439 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.439 DEBUG:boto:Params: {} 2023-09-22T21:49:11.439 DEBUG:boto:Token: None 2023-09-22T21:49:11.440 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj38 2023-09-22T21:49:11.440 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mzMaKw4Ak+z2Apuy428tOa15ltw= 2023-09-22T21:49:11.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mzMaKw4Ak+z2Apuy428tOa15ltw=', 'Content-Length': '0'} 2023-09-22T21:49:11.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000177768fb633d64eb-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.443 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:49:11.443 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:49:11.443 DEBUG:boto:Method: GET 2023-09-22T21:49:11.443 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:49:11.443 DEBUG:boto:Data: 2023-09-22T21:49:11.443 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.443 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.443 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.443 DEBUG:boto:Params: {} 2023-09-22T21:49:11.444 DEBUG:boto:Token: None 2023-09-22T21:49:11.444 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj38 2023-09-22T21:49:11.444 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mzMaKw4Ak+z2Apuy428tOa15ltw= 2023-09-22T21:49:11.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mzMaKw4Ak+z2Apuy428tOa15ltw=', 'Content-Length': '0'} 2023-09-22T21:49:11.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004090f024d4ca25d-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.447 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.447 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.447 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:49:11.447 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:49:11.448 DEBUG:boto:Method: GET 2023-09-22T21:49:11.448 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:49:11.448 DEBUG:boto:Data: 2023-09-22T21:49:11.448 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.448 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.448 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.448 DEBUG:boto:Params: {} 2023-09-22T21:49:11.448 DEBUG:boto:Token: None 2023-09-22T21:49:11.448 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.449 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e8de9b2739c4616-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.498 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:11.499 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.499 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.499 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:49:11.499 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:49:11.499 DEBUG:boto:Method: GET 2023-09-22T21:49:11.499 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:49:11.500 DEBUG:boto:Data: 2023-09-22T21:49:11.500 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.500 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.500 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.500 DEBUG:boto:Params: {} 2023-09-22T21:49:11.500 DEBUG:boto:Token: None 2023-09-22T21:49:11.500 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.500 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f19896ec6b5a6fd0-00650e0bd7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.511 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:11.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:49:11.511 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:49:11.511 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:49:11.511 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.511 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.512 DEBUG:boto:Method: GET 2023-09-22T21:49:11.512 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.512 DEBUG:boto:Data: 2023-09-22T21:49:11.512 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.512 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.512 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.512 DEBUG:boto:Params: {} 2023-09-22T21:49:11.512 DEBUG:boto:Token: None 2023-09-22T21:49:11.512 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.513 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jXNR4JwN7huclNt0+Xr5xpQareY= 2023-09-22T21:49:11.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jXNR4JwN7huclNt0+Xr5xpQareY=', 'Content-Length': '0'} 2023-09-22T21:49:11.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df2e61e7bf857a7a-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.516 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:49:11.516 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:49:11.516 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.516 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.516 DEBUG:boto:Method: GET 2023-09-22T21:49:11.517 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.517 DEBUG:boto:Data: 2023-09-22T21:49:11.517 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.517 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.517 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.517 DEBUG:boto:Params: {} 2023-09-22T21:49:11.517 DEBUG:boto:Token: None 2023-09-22T21:49:11.517 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.517 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jXNR4JwN7huclNt0+Xr5xpQareY= 2023-09-22T21:49:11.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jXNR4JwN7huclNt0+Xr5xpQareY=', 'Content-Length': '0'} 2023-09-22T21:49:11.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffa96ba31911a024-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.566 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:49:11.566 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:49:11.566 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.567 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.567 DEBUG:boto:Method: HEAD 2023-09-22T21:49:11.567 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.567 DEBUG:boto:Data: 2023-09-22T21:49:11.567 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.567 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.567 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.567 DEBUG:boto:Params: {} 2023-09-22T21:49:11.567 DEBUG:boto:Token: None 2023-09-22T21:49:11.568 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.568 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C9JFkDKClXswzuo53PPRA0o0nG0= 2023-09-22T21:49:11.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C9JFkDKClXswzuo53PPRA0o0nG0=', 'Content-Length': '0'} 2023-09-22T21:49:11.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c93841017389edd2-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.571 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:49:11.571 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:49:11.571 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.571 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.571 DEBUG:boto:Method: HEAD 2023-09-22T21:49:11.571 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.571 DEBUG:boto:Data: 2023-09-22T21:49:11.572 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.572 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.572 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.572 DEBUG:boto:Params: {} 2023-09-22T21:49:11.572 DEBUG:boto:Token: None 2023-09-22T21:49:11.572 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.572 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:C9JFkDKClXswzuo53PPRA0o0nG0= 2023-09-22T21:49:11.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:C9JFkDKClXswzuo53PPRA0o0nG0=', 'Content-Length': '0'} 2023-09-22T21:49:11.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b062f515a01b59c-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:49:11.576 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:49:11.576 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:49:11.576 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.576 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.576 DEBUG:boto:Method: GET 2023-09-22T21:49:11.576 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.576 DEBUG:boto:Data: 2023-09-22T21:49:11.576 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.577 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.577 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.577 DEBUG:boto:Params: {} 2023-09-22T21:49:11.577 DEBUG:boto:Token: None 2023-09-22T21:49:11.577 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.577 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jXNR4JwN7huclNt0+Xr5xpQareY= 2023-09-22T21:49:11.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jXNR4JwN7huclNt0+Xr5xpQareY=', 'Content-Length': '0'} 2023-09-22T21:49:11.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055eca0bdeca90fa7-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.580 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:49:11.580 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:49:11.580 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.580 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.581 DEBUG:boto:Method: GET 2023-09-22T21:49:11.581 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.581 DEBUG:boto:Data: 2023-09-22T21:49:11.581 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.581 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.581 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.581 DEBUG:boto:Params: {} 2023-09-22T21:49:11.581 DEBUG:boto:Token: None 2023-09-22T21:49:11.582 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:11.582 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jXNR4JwN7huclNt0+Xr5xpQareY= 2023-09-22T21:49:11.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jXNR4JwN7huclNt0+Xr5xpQareY=', 'Content-Length': '0'} 2023-09-22T21:49:11.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000047e995c1512aeda-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.585 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.585 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.585 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:49:11.585 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:49:11.586 DEBUG:boto:Method: GET 2023-09-22T21:49:11.586 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:49:11.586 DEBUG:boto:Data: 2023-09-22T21:49:11.586 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.586 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.586 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.586 DEBUG:boto:Params: {} 2023-09-22T21:49:11.586 DEBUG:boto:Token: None 2023-09-22T21:49:11.586 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.587 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3e2278eb9a0620a-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.634 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:11.635 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.635 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.635 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:49:11.635 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:49:11.635 DEBUG:boto:Method: GET 2023-09-22T21:49:11.635 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:49:11.635 DEBUG:boto:Data: 2023-09-22T21:49:11.635 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.636 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.636 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.636 DEBUG:boto:Params: {} 2023-09-22T21:49:11.636 DEBUG:boto:Token: None 2023-09-22T21:49:11.636 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.636 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047bdc954dae3c01b-00650e0bd7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.646 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:11.646 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:49:11.647 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:49:11.647 DEBUG:boto:Method: HEAD 2023-09-22T21:49:11.647 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:49:11.647 DEBUG:boto:Data: 2023-09-22T21:49:11.647 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.647 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.647 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.647 DEBUG:boto:Params: {} 2023-09-22T21:49:11.648 DEBUG:boto:Token: None 2023-09-22T21:49:11.648 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj44 2023-09-22T21:49:11.648 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:k9vEBxHcCZBD8JP2LKO68o2miNo= 2023-09-22T21:49:11.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:k9vEBxHcCZBD8JP2LKO68o2miNo=', 'Content-Length': '0'} 2023-09-22T21:49:11.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008596439182055120-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.651 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:49:11.652 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:49:11.652 DEBUG:boto:Method: HEAD 2023-09-22T21:49:11.652 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:49:11.652 DEBUG:boto:Data: 2023-09-22T21:49:11.652 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.652 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.652 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.652 DEBUG:boto:Params: {} 2023-09-22T21:49:11.652 DEBUG:boto:Token: None 2023-09-22T21:49:11.653 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj44 2023-09-22T21:49:11.653 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:k9vEBxHcCZBD8JP2LKO68o2miNo= 2023-09-22T21:49:11.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:k9vEBxHcCZBD8JP2LKO68o2miNo=', 'Content-Length': '0'} 2023-09-22T21:49:11.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000601aea9d8dd814d9-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:49:11.656 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:49:11.656 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:49:11.656 DEBUG:boto:Method: GET 2023-09-22T21:49:11.656 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:49:11.656 DEBUG:boto:Data: 2023-09-22T21:49:11.656 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.657 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.657 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.657 DEBUG:boto:Params: {} 2023-09-22T21:49:11.657 DEBUG:boto:Token: None 2023-09-22T21:49:11.657 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj44 2023-09-22T21:49:11.657 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:z2Adc+Mj2srk70wky9jsOvFd+Os= 2023-09-22T21:49:11.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:z2Adc+Mj2srk70wky9jsOvFd+Os=', 'Content-Length': '0'} 2023-09-22T21:49:11.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063baf7a380c5ee4d-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.660 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:49:11.660 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:49:11.660 DEBUG:boto:Method: GET 2023-09-22T21:49:11.660 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:49:11.661 DEBUG:boto:Data: 2023-09-22T21:49:11.661 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.661 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.661 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.661 DEBUG:boto:Params: {} 2023-09-22T21:49:11.661 DEBUG:boto:Token: None 2023-09-22T21:49:11.661 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj44 2023-09-22T21:49:11.661 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:z2Adc+Mj2srk70wky9jsOvFd+Os= 2023-09-22T21:49:11.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:z2Adc+Mj2srk70wky9jsOvFd+Os=', 'Content-Length': '0'} 2023-09-22T21:49:11.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b6a80e5ecbf5241-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.710 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.711 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.711 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:49:11.711 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:49:11.711 DEBUG:boto:Method: GET 2023-09-22T21:49:11.711 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:49:11.711 DEBUG:boto:Data: 2023-09-22T21:49:11.711 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.711 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.711 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.712 DEBUG:boto:Params: {} 2023-09-22T21:49:11.712 DEBUG:boto:Token: None 2023-09-22T21:49:11.712 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.712 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a99defda59669e2f-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.717 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:11.717 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.717 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.718 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:49:11.718 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:49:11.718 DEBUG:boto:Method: GET 2023-09-22T21:49:11.718 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:49:11.718 DEBUG:boto:Data: 2023-09-22T21:49:11.718 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.718 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.718 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.718 DEBUG:boto:Params: {} 2023-09-22T21:49:11.719 DEBUG:boto:Token: None 2023-09-22T21:49:11.719 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.719 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d838c9f38fc5278-00650e0bd7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.729 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:11.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-22T21:49:11.730 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:49:11.730 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:49:11.730 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.730 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.730 DEBUG:boto:Method: GET 2023-09-22T21:49:11.731 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.731 DEBUG:boto:Data: 2023-09-22T21:49:11.731 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.731 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.731 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.731 DEBUG:boto:Params: {} 2023-09-22T21:49:11.731 DEBUG:boto:Token: None 2023-09-22T21:49:11.731 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.731 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iJ/egfAqqgSwukxtdnjG5DjYoTM= 2023-09-22T21:49:11.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iJ/egfAqqgSwukxtdnjG5DjYoTM=', 'Content-Length': '0'} 2023-09-22T21:49:11.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf1ee7660f5b285c-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.774 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:49:11.774 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:49:11.775 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.775 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.775 DEBUG:boto:Method: GET 2023-09-22T21:49:11.775 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.775 DEBUG:boto:Data: 2023-09-22T21:49:11.775 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.775 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.775 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.776 DEBUG:boto:Params: {} 2023-09-22T21:49:11.776 DEBUG:boto:Token: None 2023-09-22T21:49:11.776 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.776 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iJ/egfAqqgSwukxtdnjG5DjYoTM= 2023-09-22T21:49:11.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iJ/egfAqqgSwukxtdnjG5DjYoTM=', 'Content-Length': '0'} 2023-09-22T21:49:11.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f588c295f5328e4-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.779 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:49:11.779 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:49:11.779 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.780 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.780 DEBUG:boto:Method: HEAD 2023-09-22T21:49:11.780 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.780 DEBUG:boto:Data: 2023-09-22T21:49:11.780 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.780 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.780 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.780 DEBUG:boto:Params: {} 2023-09-22T21:49:11.780 DEBUG:boto:Token: None 2023-09-22T21:49:11.781 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.781 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6nBK4cGdqy3GSQm8EU+3TTpEl7A= 2023-09-22T21:49:11.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6nBK4cGdqy3GSQm8EU+3TTpEl7A=', 'Content-Length': '0'} 2023-09-22T21:49:11.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9923fb8dae5d832-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.784 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:49:11.784 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:49:11.784 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.784 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.785 DEBUG:boto:Method: HEAD 2023-09-22T21:49:11.785 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.785 DEBUG:boto:Data: 2023-09-22T21:49:11.785 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.785 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.785 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.785 DEBUG:boto:Params: {} 2023-09-22T21:49:11.785 DEBUG:boto:Token: None 2023-09-22T21:49:11.785 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.786 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6nBK4cGdqy3GSQm8EU+3TTpEl7A= 2023-09-22T21:49:11.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6nBK4cGdqy3GSQm8EU+3TTpEl7A=', 'Content-Length': '0'} 2023-09-22T21:49:11.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ac0335a55d6e738-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-22T21:49:11.789 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:49:11.789 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:49:11.789 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.789 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.789 DEBUG:boto:Method: GET 2023-09-22T21:49:11.790 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.790 DEBUG:boto:Data: 2023-09-22T21:49:11.790 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.790 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.790 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.790 DEBUG:boto:Params: {} 2023-09-22T21:49:11.790 DEBUG:boto:Token: None 2023-09-22T21:49:11.790 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.791 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iJ/egfAqqgSwukxtdnjG5DjYoTM= 2023-09-22T21:49:11.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iJ/egfAqqgSwukxtdnjG5DjYoTM=', 'Content-Length': '0'} 2023-09-22T21:49:11.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098742b2c03ae66a3-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.793 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:49:11.794 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:49:11.794 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.794 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.794 DEBUG:boto:Method: GET 2023-09-22T21:49:11.794 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.794 DEBUG:boto:Data: 2023-09-22T21:49:11.794 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.794 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.795 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.795 DEBUG:boto:Params: {} 2023-09-22T21:49:11.795 DEBUG:boto:Token: None 2023-09-22T21:49:11.795 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:11.795 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iJ/egfAqqgSwukxtdnjG5DjYoTM= 2023-09-22T21:49:11.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iJ/egfAqqgSwukxtdnjG5DjYoTM=', 'Content-Length': '0'} 2023-09-22T21:49:11.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e116ab0e2a195510-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.798 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.798 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.798 DEBUG:boto:path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:49:11.798 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:49:11.798 DEBUG:boto:Method: GET 2023-09-22T21:49:11.799 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj46 2023-09-22T21:49:11.799 DEBUG:boto:Data: 2023-09-22T21:49:11.799 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.799 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.799 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.799 DEBUG:boto:Params: {} 2023-09-22T21:49:11.799 DEBUG:boto:Token: None 2023-09-22T21:49:11.799 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.800 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f95e9ac27e8e15a5-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.804 DEBUG:boto:b'ffzdqu-59obj461000falseobj462023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:11.804 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.805 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.805 DEBUG:boto:path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:49:11.805 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:49:11.805 DEBUG:boto:Method: GET 2023-09-22T21:49:11.805 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj46 2023-09-22T21:49:11.805 DEBUG:boto:Data: 2023-09-22T21:49:11.805 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.805 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.805 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.805 DEBUG:boto:Params: {} 2023-09-22T21:49:11.806 DEBUG:boto:Token: None 2023-09-22T21:49:11.806 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.806 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001431c3ef4da68818-00650e0bd7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.816 DEBUG:boto:b'ffzdqu-59obj461000falseobj462023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:11.816 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:49:11.816 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:49:11.816 DEBUG:boto:Method: HEAD 2023-09-22T21:49:11.817 DEBUG:boto:Path: /ffzdqu-59/obj46 2023-09-22T21:49:11.817 DEBUG:boto:Data: 2023-09-22T21:49:11.817 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.817 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.817 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.817 DEBUG:boto:Params: {} 2023-09-22T21:49:11.817 DEBUG:boto:Token: None 2023-09-22T21:49:11.817 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj46 2023-09-22T21:49:11.818 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:l9vTsFvk3WolOpnfo2K7GFMW0wg= 2023-09-22T21:49:11.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:l9vTsFvk3WolOpnfo2K7GFMW0wg=', 'Content-Length': '0'} 2023-09-22T21:49:11.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d0e11cad92cc79a-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.821 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:49:11.821 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:49:11.821 DEBUG:boto:Method: HEAD 2023-09-22T21:49:11.821 DEBUG:boto:Path: /ffzdqu-59/obj46 2023-09-22T21:49:11.821 DEBUG:boto:Data: 2023-09-22T21:49:11.821 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.821 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.821 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.822 DEBUG:boto:Params: {} 2023-09-22T21:49:11.822 DEBUG:boto:Token: None 2023-09-22T21:49:11.822 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj46 2023-09-22T21:49:11.822 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:l9vTsFvk3WolOpnfo2K7GFMW0wg= 2023-09-22T21:49:11.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:l9vTsFvk3WolOpnfo2K7GFMW0wg=', 'Content-Length': '0'} 2023-09-22T21:49:11.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000500bd0b5af388daa-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-22T21:49:11.825 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:49:11.825 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:49:11.825 DEBUG:boto:Method: GET 2023-09-22T21:49:11.825 DEBUG:boto:Path: /ffzdqu-59/obj46 2023-09-22T21:49:11.825 DEBUG:boto:Data: 2023-09-22T21:49:11.826 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.826 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.826 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.826 DEBUG:boto:Params: {} 2023-09-22T21:49:11.826 DEBUG:boto:Token: None 2023-09-22T21:49:11.826 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj46 2023-09-22T21:49:11.826 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Vfws6s/bcdrxp6TT1U86yGiEbC0= 2023-09-22T21:49:11.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Vfws6s/bcdrxp6TT1U86yGiEbC0=', 'Content-Length': '0'} 2023-09-22T21:49:11.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bc1d82bf7a8c7e7-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.874 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:49:11.875 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:49:11.875 DEBUG:boto:Method: GET 2023-09-22T21:49:11.875 DEBUG:boto:Path: /ffzdqu-59/obj46 2023-09-22T21:49:11.875 DEBUG:boto:Data: 2023-09-22T21:49:11.876 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.876 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.876 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.876 DEBUG:boto:Params: {} 2023-09-22T21:49:11.876 DEBUG:boto:Token: None 2023-09-22T21:49:11.877 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj46 2023-09-22T21:49:11.877 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Vfws6s/bcdrxp6TT1U86yGiEbC0= 2023-09-22T21:49:11.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Vfws6s/bcdrxp6TT1U86yGiEbC0=', 'Content-Length': '0'} 2023-09-22T21:49:11.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000024d2490a402c747-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.880 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.880 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.880 DEBUG:boto:path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:49:11.880 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:49:11.881 DEBUG:boto:Method: GET 2023-09-22T21:49:11.881 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj46 2023-09-22T21:49:11.881 DEBUG:boto:Data: 2023-09-22T21:49:11.881 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.881 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.881 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.881 DEBUG:boto:Params: {} 2023-09-22T21:49:11.881 DEBUG:boto:Token: None 2023-09-22T21:49:11.881 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.882 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eed5020a89a1bda6-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.887 DEBUG:boto:b'ffzdqu-59obj461000falseobj462023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:11.887 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.887 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.887 DEBUG:boto:path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:49:11.888 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:49:11.888 DEBUG:boto:Method: GET 2023-09-22T21:49:11.888 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj46 2023-09-22T21:49:11.888 DEBUG:boto:Data: 2023-09-22T21:49:11.888 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.888 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.888 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.888 DEBUG:boto:Params: {} 2023-09-22T21:49:11.888 DEBUG:boto:Token: None 2023-09-22T21:49:11.889 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.889 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000846ddc38ff7526db-00650e0bd7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.899 DEBUG:boto:b'ffzdqu-59obj461000falseobj462023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:11.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:49:11.900 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:49:11.900 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:49:11.900 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.900 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.900 DEBUG:boto:Method: GET 2023-09-22T21:49:11.900 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.900 DEBUG:boto:Data: 2023-09-22T21:49:11.900 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.901 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.901 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.901 DEBUG:boto:Params: {} 2023-09-22T21:49:11.901 DEBUG:boto:Token: None 2023-09-22T21:49:11.901 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.901 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zrOWx7g8bFcFX3leU2Zii3mZggA= 2023-09-22T21:49:11.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zrOWx7g8bFcFX3leU2Zii3mZggA=', 'Content-Length': '0'} 2023-09-22T21:49:11.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052625750231c8839-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.905 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:49:11.905 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:49:11.905 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.905 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.905 DEBUG:boto:Method: GET 2023-09-22T21:49:11.905 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.906 DEBUG:boto:Data: 2023-09-22T21:49:11.906 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.906 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.906 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.906 DEBUG:boto:Params: {} 2023-09-22T21:49:11.906 DEBUG:boto:Token: None 2023-09-22T21:49:11.906 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.906 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zrOWx7g8bFcFX3leU2Zii3mZggA= 2023-09-22T21:49:11.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zrOWx7g8bFcFX3leU2Zii3mZggA=', 'Content-Length': '0'} 2023-09-22T21:49:11.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000193a3d1e981ca1d7-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.909 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:49:11.910 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:49:11.910 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.910 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.910 DEBUG:boto:Method: HEAD 2023-09-22T21:49:11.910 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.910 DEBUG:boto:Data: 2023-09-22T21:49:11.910 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.910 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.910 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.911 DEBUG:boto:Params: {} 2023-09-22T21:49:11.911 DEBUG:boto:Token: None 2023-09-22T21:49:11.911 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.911 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dVLxIc+sFN7XwLCXvEOo3gxRoro= 2023-09-22T21:49:11.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dVLxIc+sFN7XwLCXvEOo3gxRoro=', 'Content-Length': '0'} 2023-09-22T21:49:11.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f801f88eb2cfd08-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.914 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:49:11.914 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:49:11.914 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.914 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.914 DEBUG:boto:Method: HEAD 2023-09-22T21:49:11.915 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.915 DEBUG:boto:Data: 2023-09-22T21:49:11.915 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.915 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.915 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.915 DEBUG:boto:Params: {} 2023-09-22T21:49:11.915 DEBUG:boto:Token: None 2023-09-22T21:49:11.915 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.915 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dVLxIc+sFN7XwLCXvEOo3gxRoro= 2023-09-22T21:49:11.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dVLxIc+sFN7XwLCXvEOo3gxRoro=', 'Content-Length': '0'} 2023-09-22T21:49:11.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000affdc1d5b98d09ec-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:49:11.918 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:49:11.918 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:49:11.918 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.919 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.919 DEBUG:boto:Method: GET 2023-09-22T21:49:11.919 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.919 DEBUG:boto:Data: 2023-09-22T21:49:11.919 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.919 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.919 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.919 DEBUG:boto:Params: {} 2023-09-22T21:49:11.919 DEBUG:boto:Token: None 2023-09-22T21:49:11.919 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.920 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zrOWx7g8bFcFX3leU2Zii3mZggA= 2023-09-22T21:49:11.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zrOWx7g8bFcFX3leU2Zii3mZggA=', 'Content-Length': '0'} 2023-09-22T21:49:11.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4b6e8f336176bfa-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.970 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:49:11.970 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:49:11.971 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.971 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.971 DEBUG:boto:Method: GET 2023-09-22T21:49:11.971 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.971 DEBUG:boto:Data: 2023-09-22T21:49:11.971 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.971 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.971 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.971 DEBUG:boto:Params: {} 2023-09-22T21:49:11.971 DEBUG:boto:Token: None 2023-09-22T21:49:11.972 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:11.972 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zrOWx7g8bFcFX3leU2Zii3mZggA= 2023-09-22T21:49:11.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zrOWx7g8bFcFX3leU2Zii3mZggA=', 'Content-Length': '0'} 2023-09-22T21:49:11.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082177f9a2119017f-00650e0bd7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.975 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.975 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.975 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:49:11.975 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:49:11.975 DEBUG:boto:Method: GET 2023-09-22T21:49:11.975 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:49:11.975 DEBUG:boto:Data: 2023-09-22T21:49:11.975 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.976 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:11.976 DEBUG:boto:Port: 8001 2023-09-22T21:49:11.976 DEBUG:boto:Params: {} 2023-09-22T21:49:11.976 DEBUG:boto:Token: None 2023-09-22T21:49:11.976 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.976 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028a5c58b1401e82b-00650e0bd7-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:11.982 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:11.982 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:11.982 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:11.982 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:49:11.982 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:49:11.983 DEBUG:boto:Method: GET 2023-09-22T21:49:11.983 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:49:11.983 DEBUG:boto:Data: 2023-09-22T21:49:11.983 DEBUG:boto:Headers: {} 2023-09-22T21:49:11.983 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:11.983 DEBUG:boto:Port: 8000 2023-09-22T21:49:11.983 DEBUG:boto:Params: {} 2023-09-22T21:49:11.983 DEBUG:boto:Token: None 2023-09-22T21:49:11.983 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:11 GMT /ffzdqu-59/ 2023-09-22T21:49:11.984 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ= 2023-09-22T21:49:11.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Se0aYvIo0vyFMaFJlk/CCh5xJQ=', 'Content-Length': '0'} 2023-09-22T21:49:11.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d998bd700f1a820f-00650e0bd7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.034 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:12.035 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:49:12.035 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:49:12.035 DEBUG:boto:Method: HEAD 2023-09-22T21:49:12.035 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:49:12.035 DEBUG:boto:Data: 2023-09-22T21:49:12.035 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.035 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.035 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.036 DEBUG:boto:Params: {} 2023-09-22T21:49:12.036 DEBUG:boto:Token: None 2023-09-22T21:49:12.036 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj51 2023-09-22T21:49:12.036 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hamXOXZM+zbY5eOoPPW2l1Zfgx4= 2023-09-22T21:49:12.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hamXOXZM+zbY5eOoPPW2l1Zfgx4=', 'Content-Length': '0'} 2023-09-22T21:49:12.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4a8f1202c3ce057-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.039 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:49:12.039 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:49:12.039 DEBUG:boto:Method: HEAD 2023-09-22T21:49:12.039 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:49:12.039 DEBUG:boto:Data: 2023-09-22T21:49:12.040 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.040 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.040 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.040 DEBUG:boto:Params: {} 2023-09-22T21:49:12.040 DEBUG:boto:Token: None 2023-09-22T21:49:12.040 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj51 2023-09-22T21:49:12.040 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hamXOXZM+zbY5eOoPPW2l1Zfgx4= 2023-09-22T21:49:12.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hamXOXZM+zbY5eOoPPW2l1Zfgx4=', 'Content-Length': '0'} 2023-09-22T21:49:12.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4959699ff51c10f-00650e0bd8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:49:12.043 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:49:12.043 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:49:12.044 DEBUG:boto:Method: GET 2023-09-22T21:49:12.044 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:49:12.044 DEBUG:boto:Data: 2023-09-22T21:49:12.044 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.044 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.044 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.044 DEBUG:boto:Params: {} 2023-09-22T21:49:12.044 DEBUG:boto:Token: None 2023-09-22T21:49:12.045 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj51 2023-09-22T21:49:12.045 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:z243Y4HhpIxnwmijuvaiX7ySCho= 2023-09-22T21:49:12.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:z243Y4HhpIxnwmijuvaiX7ySCho=', 'Content-Length': '0'} 2023-09-22T21:49:12.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bcd70d4aa210155-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.048 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:49:12.048 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:49:12.048 DEBUG:boto:Method: GET 2023-09-22T21:49:12.048 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:49:12.048 DEBUG:boto:Data: 2023-09-22T21:49:12.048 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.048 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.048 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.049 DEBUG:boto:Params: {} 2023-09-22T21:49:12.049 DEBUG:boto:Token: None 2023-09-22T21:49:12.049 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj51 2023-09-22T21:49:12.049 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:z243Y4HhpIxnwmijuvaiX7ySCho= 2023-09-22T21:49:12.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:z243Y4HhpIxnwmijuvaiX7ySCho=', 'Content-Length': '0'} 2023-09-22T21:49:12.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b313cab8e478095a-00650e0bd8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.052 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:12.052 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:12.052 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:49:12.053 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:49:12.053 DEBUG:boto:Method: GET 2023-09-22T21:49:12.053 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:49:12.053 DEBUG:boto:Data: 2023-09-22T21:49:12.053 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.053 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.053 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.053 DEBUG:boto:Params: {} 2023-09-22T21:49:12.053 DEBUG:boto:Token: None 2023-09-22T21:49:12.054 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/ 2023-09-22T21:49:12.054 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U= 2023-09-22T21:49:12.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U=', 'Content-Length': '0'} 2023-09-22T21:49:12.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f34699d1f322b860-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.102 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:12.103 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:12.103 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:12.103 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:49:12.103 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:49:12.103 DEBUG:boto:Method: GET 2023-09-22T21:49:12.104 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:49:12.104 DEBUG:boto:Data: 2023-09-22T21:49:12.104 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.104 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.104 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.104 DEBUG:boto:Params: {} 2023-09-22T21:49:12.104 DEBUG:boto:Token: None 2023-09-22T21:49:12.105 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/ 2023-09-22T21:49:12.105 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U= 2023-09-22T21:49:12.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U=', 'Content-Length': '0'} 2023-09-22T21:49:12.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1eebd4819e2591b-00650e0bd8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.115 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:12.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:49:12.115 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:49:12.115 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:49:12.115 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.116 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.116 DEBUG:boto:Method: GET 2023-09-22T21:49:12.116 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.116 DEBUG:boto:Data: 2023-09-22T21:49:12.116 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.116 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.116 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.116 DEBUG:boto:Params: {} 2023-09-22T21:49:12.116 DEBUG:boto:Token: None 2023-09-22T21:49:12.117 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.117 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kgtbO2fZc9Y/73oSiKxwM4Mjcd4= 2023-09-22T21:49:12.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kgtbO2fZc9Y/73oSiKxwM4Mjcd4=', 'Content-Length': '0'} 2023-09-22T21:49:12.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000584ac4d6c8584c4f-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.120 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:49:12.120 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:49:12.120 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.120 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.121 DEBUG:boto:Method: GET 2023-09-22T21:49:12.121 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.121 DEBUG:boto:Data: 2023-09-22T21:49:12.121 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.121 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.121 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.121 DEBUG:boto:Params: {} 2023-09-22T21:49:12.121 DEBUG:boto:Token: None 2023-09-22T21:49:12.121 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.122 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kgtbO2fZc9Y/73oSiKxwM4Mjcd4= 2023-09-22T21:49:12.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kgtbO2fZc9Y/73oSiKxwM4Mjcd4=', 'Content-Length': '0'} 2023-09-22T21:49:12.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b31b31f1b06dc936-00650e0bd8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.166 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:49:12.167 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:49:12.167 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.167 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.167 DEBUG:boto:Method: HEAD 2023-09-22T21:49:12.167 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.167 DEBUG:boto:Data: 2023-09-22T21:49:12.167 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.167 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.167 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.168 DEBUG:boto:Params: {} 2023-09-22T21:49:12.168 DEBUG:boto:Token: None 2023-09-22T21:49:12.168 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.168 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b5eSqnmomLf9bfarJX8RTCIxiM8= 2023-09-22T21:49:12.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b5eSqnmomLf9bfarJX8RTCIxiM8=', 'Content-Length': '0'} 2023-09-22T21:49:12.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000368aee64dc275370-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.171 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:49:12.172 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:49:12.172 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.172 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.172 DEBUG:boto:Method: HEAD 2023-09-22T21:49:12.172 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.172 DEBUG:boto:Data: 2023-09-22T21:49:12.173 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.173 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.173 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.173 DEBUG:boto:Params: {} 2023-09-22T21:49:12.173 DEBUG:boto:Token: None 2023-09-22T21:49:12.173 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.174 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b5eSqnmomLf9bfarJX8RTCIxiM8= 2023-09-22T21:49:12.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b5eSqnmomLf9bfarJX8RTCIxiM8=', 'Content-Length': '0'} 2023-09-22T21:49:12.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ae592e0c085f547-00650e0bd8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:49:12.178 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:49:12.178 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:49:12.178 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.178 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.179 DEBUG:boto:Method: GET 2023-09-22T21:49:12.179 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.179 DEBUG:boto:Data: 2023-09-22T21:49:12.179 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.179 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.180 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.180 DEBUG:boto:Params: {} 2023-09-22T21:49:12.180 DEBUG:boto:Token: None 2023-09-22T21:49:12.180 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.180 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kgtbO2fZc9Y/73oSiKxwM4Mjcd4= 2023-09-22T21:49:12.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kgtbO2fZc9Y/73oSiKxwM4Mjcd4=', 'Content-Length': '0'} 2023-09-22T21:49:12.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031adf178ded96e3b-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.183 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:49:12.184 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:49:12.184 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.184 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.184 DEBUG:boto:Method: GET 2023-09-22T21:49:12.184 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.184 DEBUG:boto:Data: 2023-09-22T21:49:12.184 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.184 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.184 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.185 DEBUG:boto:Params: {} 2023-09-22T21:49:12.185 DEBUG:boto:Token: None 2023-09-22T21:49:12.185 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:12.185 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kgtbO2fZc9Y/73oSiKxwM4Mjcd4= 2023-09-22T21:49:12.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kgtbO2fZc9Y/73oSiKxwM4Mjcd4=', 'Content-Length': '0'} 2023-09-22T21:49:12.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019f2f8cdd58db1d0-00650e0bd8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.188 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:12.188 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:12.188 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:49:12.188 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:49:12.188 DEBUG:boto:Method: GET 2023-09-22T21:49:12.189 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:49:12.189 DEBUG:boto:Data: 2023-09-22T21:49:12.189 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.189 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.189 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.189 DEBUG:boto:Params: {} 2023-09-22T21:49:12.189 DEBUG:boto:Token: None 2023-09-22T21:49:12.189 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/ 2023-09-22T21:49:12.189 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U= 2023-09-22T21:49:12.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U=', 'Content-Length': '0'} 2023-09-22T21:49:12.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb35287dd11a023b-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.234 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:12.235 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:12.235 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:12.235 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:49:12.235 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:49:12.235 DEBUG:boto:Method: GET 2023-09-22T21:49:12.235 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:49:12.235 DEBUG:boto:Data: 2023-09-22T21:49:12.236 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.236 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.236 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.236 DEBUG:boto:Params: {} 2023-09-22T21:49:12.236 DEBUG:boto:Token: None 2023-09-22T21:49:12.236 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/ 2023-09-22T21:49:12.236 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U= 2023-09-22T21:49:12.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U=', 'Content-Length': '0'} 2023-09-22T21:49:12.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0dc0544364f1ad9-00650e0bd8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.246 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:12.247 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:49:12.247 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:49:12.247 DEBUG:boto:Method: HEAD 2023-09-22T21:49:12.247 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:49:12.247 DEBUG:boto:Data: 2023-09-22T21:49:12.247 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.247 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.247 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.247 DEBUG:boto:Params: {} 2023-09-22T21:49:12.247 DEBUG:boto:Token: None 2023-09-22T21:49:12.248 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj55 2023-09-22T21:49:12.248 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nUpL5KMHXsFyvYsuVOnV4ke3ySY= 2023-09-22T21:49:12.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nUpL5KMHXsFyvYsuVOnV4ke3ySY=', 'Content-Length': '0'} 2023-09-22T21:49:12.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f662f40a5b7fb966-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.251 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:49:12.251 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:49:12.251 DEBUG:boto:Method: HEAD 2023-09-22T21:49:12.251 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:49:12.252 DEBUG:boto:Data: 2023-09-22T21:49:12.252 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.252 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.252 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.252 DEBUG:boto:Params: {} 2023-09-22T21:49:12.252 DEBUG:boto:Token: None 2023-09-22T21:49:12.252 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj55 2023-09-22T21:49:12.252 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nUpL5KMHXsFyvYsuVOnV4ke3ySY= 2023-09-22T21:49:12.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nUpL5KMHXsFyvYsuVOnV4ke3ySY=', 'Content-Length': '0'} 2023-09-22T21:49:12.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d52a6f825578830-00650e0bd8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:49:12.256 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:49:12.256 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:49:12.256 DEBUG:boto:Method: GET 2023-09-22T21:49:12.256 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:49:12.256 DEBUG:boto:Data: 2023-09-22T21:49:12.256 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.256 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.256 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.256 DEBUG:boto:Params: {} 2023-09-22T21:49:12.256 DEBUG:boto:Token: None 2023-09-22T21:49:12.257 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj55 2023-09-22T21:49:12.257 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QJsv3vJLMW7XwtYxhlktzQg0HAM= 2023-09-22T21:49:12.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QJsv3vJLMW7XwtYxhlktzQg0HAM=', 'Content-Length': '0'} 2023-09-22T21:49:12.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf3263ea8db4f43d-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.260 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:49:12.260 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:49:12.260 DEBUG:boto:Method: GET 2023-09-22T21:49:12.260 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:49:12.260 DEBUG:boto:Data: 2023-09-22T21:49:12.260 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.260 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.261 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.261 DEBUG:boto:Params: {} 2023-09-22T21:49:12.261 DEBUG:boto:Token: None 2023-09-22T21:49:12.261 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj55 2023-09-22T21:49:12.261 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QJsv3vJLMW7XwtYxhlktzQg0HAM= 2023-09-22T21:49:12.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QJsv3vJLMW7XwtYxhlktzQg0HAM=', 'Content-Length': '0'} 2023-09-22T21:49:12.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f2453311714cf97-00650e0bd8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.306 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:12.307 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:12.307 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:49:12.307 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:49:12.307 DEBUG:boto:Method: GET 2023-09-22T21:49:12.307 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:49:12.307 DEBUG:boto:Data: 2023-09-22T21:49:12.307 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.307 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.307 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.307 DEBUG:boto:Params: {} 2023-09-22T21:49:12.308 DEBUG:boto:Token: None 2023-09-22T21:49:12.308 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/ 2023-09-22T21:49:12.308 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U= 2023-09-22T21:49:12.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U=', 'Content-Length': '0'} 2023-09-22T21:49:12.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2add83edbec27cd-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.313 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:12.314 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:12.314 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:12.314 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:49:12.314 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:49:12.314 DEBUG:boto:Method: GET 2023-09-22T21:49:12.314 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:49:12.315 DEBUG:boto:Data: 2023-09-22T21:49:12.315 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.315 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.315 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.315 DEBUG:boto:Params: {} 2023-09-22T21:49:12.315 DEBUG:boto:Token: None 2023-09-22T21:49:12.315 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/ 2023-09-22T21:49:12.315 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U= 2023-09-22T21:49:12.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U=', 'Content-Length': '0'} 2023-09-22T21:49:12.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de4ac0690214cdda-00650e0bd8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.325 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:12.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T21:49:12.325 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:49:12.325 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:49:12.326 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.326 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.326 DEBUG:boto:Method: GET 2023-09-22T21:49:12.326 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.326 DEBUG:boto:Data: 2023-09-22T21:49:12.326 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.326 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.326 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.326 DEBUG:boto:Params: {} 2023-09-22T21:49:12.326 DEBUG:boto:Token: None 2023-09-22T21:49:12.327 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.327 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rOqlbvQ4i+08hz61XlnKN1PW370= 2023-09-22T21:49:12.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rOqlbvQ4i+08hz61XlnKN1PW370=', 'Content-Length': '0'} 2023-09-22T21:49:12.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075118a35d0cd55ef-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.370 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:49:12.370 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:49:12.371 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.371 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.371 DEBUG:boto:Method: GET 2023-09-22T21:49:12.371 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.371 DEBUG:boto:Data: 2023-09-22T21:49:12.371 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.371 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.371 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.371 DEBUG:boto:Params: {} 2023-09-22T21:49:12.372 DEBUG:boto:Token: None 2023-09-22T21:49:12.372 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.372 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rOqlbvQ4i+08hz61XlnKN1PW370= 2023-09-22T21:49:12.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rOqlbvQ4i+08hz61XlnKN1PW370=', 'Content-Length': '0'} 2023-09-22T21:49:12.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a8b386d4a1f6990-00650e0bd8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.375 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:49:12.375 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:49:12.375 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.376 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.376 DEBUG:boto:Method: HEAD 2023-09-22T21:49:12.376 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.376 DEBUG:boto:Data: 2023-09-22T21:49:12.376 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.376 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.376 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.376 DEBUG:boto:Params: {} 2023-09-22T21:49:12.376 DEBUG:boto:Token: None 2023-09-22T21:49:12.377 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.377 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EcSaSKotzYz24Mnl3f4KQZiDw30= 2023-09-22T21:49:12.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EcSaSKotzYz24Mnl3f4KQZiDw30=', 'Content-Length': '0'} 2023-09-22T21:49:12.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008870feaa42fcf2a2-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.380 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:49:12.380 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:49:12.380 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.380 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.380 DEBUG:boto:Method: HEAD 2023-09-22T21:49:12.380 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.380 DEBUG:boto:Data: 2023-09-22T21:49:12.380 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.380 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.381 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.381 DEBUG:boto:Params: {} 2023-09-22T21:49:12.381 DEBUG:boto:Token: None 2023-09-22T21:49:12.381 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.381 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EcSaSKotzYz24Mnl3f4KQZiDw30= 2023-09-22T21:49:12.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EcSaSKotzYz24Mnl3f4KQZiDw30=', 'Content-Length': '0'} 2023-09-22T21:49:12.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fde61ace2451b87-00650e0bd8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T21:49:12.384 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:49:12.384 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:49:12.384 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.384 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.385 DEBUG:boto:Method: GET 2023-09-22T21:49:12.385 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.385 DEBUG:boto:Data: 2023-09-22T21:49:12.385 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.385 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.385 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.385 DEBUG:boto:Params: {} 2023-09-22T21:49:12.385 DEBUG:boto:Token: None 2023-09-22T21:49:12.385 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.385 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rOqlbvQ4i+08hz61XlnKN1PW370= 2023-09-22T21:49:12.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rOqlbvQ4i+08hz61XlnKN1PW370=', 'Content-Length': '0'} 2023-09-22T21:49:12.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f4ce40ea4ef6428-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.388 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:49:12.389 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:49:12.389 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.389 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.389 DEBUG:boto:Method: GET 2023-09-22T21:49:12.389 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.389 DEBUG:boto:Data: 2023-09-22T21:49:12.389 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.389 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.389 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.389 DEBUG:boto:Params: {} 2023-09-22T21:49:12.390 DEBUG:boto:Token: None 2023-09-22T21:49:12.390 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:12.390 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rOqlbvQ4i+08hz61XlnKN1PW370= 2023-09-22T21:49:12.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rOqlbvQ4i+08hz61XlnKN1PW370=', 'Content-Length': '0'} 2023-09-22T21:49:12.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b33ee4eeff6b3fb-00650e0bd8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.434 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:12.434 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:12.435 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:49:12.435 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:49:12.435 DEBUG:boto:Method: GET 2023-09-22T21:49:12.435 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:49:12.435 DEBUG:boto:Data: 2023-09-22T21:49:12.436 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.436 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.436 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.436 DEBUG:boto:Params: {} 2023-09-22T21:49:12.436 DEBUG:boto:Token: None 2023-09-22T21:49:12.437 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/ 2023-09-22T21:49:12.437 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U= 2023-09-22T21:49:12.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U=', 'Content-Length': '0'} 2023-09-22T21:49:12.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019a8cd68cbe87752-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.442 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:12.442 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:12.442 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:12.443 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:49:12.443 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:49:12.443 DEBUG:boto:Method: GET 2023-09-22T21:49:12.443 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:49:12.443 DEBUG:boto:Data: 2023-09-22T21:49:12.443 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.443 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.443 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.443 DEBUG:boto:Params: {} 2023-09-22T21:49:12.443 DEBUG:boto:Token: None 2023-09-22T21:49:12.444 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/ 2023-09-22T21:49:12.444 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U= 2023-09-22T21:49:12.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U=', 'Content-Length': '0'} 2023-09-22T21:49:12.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000962eb5d3d08e70df-00650e0bd8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.453 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:12.454 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:49:12.454 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:49:12.454 DEBUG:boto:Method: HEAD 2023-09-22T21:49:12.454 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:49:12.454 DEBUG:boto:Data: 2023-09-22T21:49:12.454 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.455 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.455 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.455 DEBUG:boto:Params: {} 2023-09-22T21:49:12.455 DEBUG:boto:Token: None 2023-09-22T21:49:12.455 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj57 2023-09-22T21:49:12.455 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vGEVUUek+oF0iqYxP6ejc0Ch40k= 2023-09-22T21:49:12.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vGEVUUek+oF0iqYxP6ejc0Ch40k=', 'Content-Length': '0'} 2023-09-22T21:49:12.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f5cfc1745e1992c-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.458 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:49:12.458 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:49:12.458 DEBUG:boto:Method: HEAD 2023-09-22T21:49:12.458 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:49:12.458 DEBUG:boto:Data: 2023-09-22T21:49:12.459 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.459 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.459 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.459 DEBUG:boto:Params: {} 2023-09-22T21:49:12.459 DEBUG:boto:Token: None 2023-09-22T21:49:12.459 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj57 2023-09-22T21:49:12.459 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vGEVUUek+oF0iqYxP6ejc0Ch40k= 2023-09-22T21:49:12.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vGEVUUek+oF0iqYxP6ejc0Ch40k=', 'Content-Length': '0'} 2023-09-22T21:49:12.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6f5fff2c51924b2-00650e0bd8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T21:49:12.462 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:49:12.462 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:49:12.462 DEBUG:boto:Method: GET 2023-09-22T21:49:12.462 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:49:12.463 DEBUG:boto:Data: 2023-09-22T21:49:12.463 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.463 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.463 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.463 DEBUG:boto:Params: {} 2023-09-22T21:49:12.463 DEBUG:boto:Token: None 2023-09-22T21:49:12.463 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj57 2023-09-22T21:49:12.463 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PWedZsQKJkyXwtJuOUIf1pH7uG0= 2023-09-22T21:49:12.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PWedZsQKJkyXwtJuOUIf1pH7uG0=', 'Content-Length': '0'} 2023-09-22T21:49:12.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dc37265de0b0cba-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.510 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:49:12.511 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:49:12.511 DEBUG:boto:Method: GET 2023-09-22T21:49:12.511 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:49:12.511 DEBUG:boto:Data: 2023-09-22T21:49:12.511 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.511 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.511 DEBUG:boto:Params: {} 2023-09-22T21:49:12.511 DEBUG:boto:Token: None 2023-09-22T21:49:12.512 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj57 2023-09-22T21:49:12.512 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PWedZsQKJkyXwtJuOUIf1pH7uG0= 2023-09-22T21:49:12.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PWedZsQKJkyXwtJuOUIf1pH7uG0=', 'Content-Length': '0'} 2023-09-22T21:49:12.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004842902c7305c314-00650e0bd8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.515 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:12.515 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:12.515 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:49:12.515 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:49:12.515 DEBUG:boto:Method: GET 2023-09-22T21:49:12.515 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:49:12.515 DEBUG:boto:Data: 2023-09-22T21:49:12.516 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.516 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.516 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.516 DEBUG:boto:Params: {} 2023-09-22T21:49:12.516 DEBUG:boto:Token: None 2023-09-22T21:49:12.516 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/ 2023-09-22T21:49:12.516 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U= 2023-09-22T21:49:12.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U=', 'Content-Length': '0'} 2023-09-22T21:49:12.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca53dca266284e7f-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.521 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:12.522 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:12.522 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:12.522 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:49:12.522 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:49:12.522 DEBUG:boto:Method: GET 2023-09-22T21:49:12.523 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:49:12.523 DEBUG:boto:Data: 2023-09-22T21:49:12.523 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.523 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.523 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.523 DEBUG:boto:Params: {} 2023-09-22T21:49:12.524 DEBUG:boto:Token: None 2023-09-22T21:49:12.524 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/ 2023-09-22T21:49:12.524 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U= 2023-09-22T21:49:12.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U=', 'Content-Length': '0'} 2023-09-22T21:49:12.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5e1db152668e16a-00650e0bd8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.578 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:12.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-22T21:49:12.579 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:49:12.579 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:49:12.579 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.579 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.579 DEBUG:boto:Method: GET 2023-09-22T21:49:12.579 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.580 DEBUG:boto:Data: 2023-09-22T21:49:12.580 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.580 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.580 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.580 DEBUG:boto:Params: {} 2023-09-22T21:49:12.580 DEBUG:boto:Token: None 2023-09-22T21:49:12.580 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.580 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jR5rgg4apX8muISX31Cv98Y8Drk= 2023-09-22T21:49:12.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jR5rgg4apX8muISX31Cv98Y8Drk=', 'Content-Length': '0'} 2023-09-22T21:49:12.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002217d92f8f08bc7f-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.584 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:49:12.584 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:49:12.584 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.584 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.584 DEBUG:boto:Method: GET 2023-09-22T21:49:12.584 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.584 DEBUG:boto:Data: 2023-09-22T21:49:12.584 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.585 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.585 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.585 DEBUG:boto:Params: {} 2023-09-22T21:49:12.585 DEBUG:boto:Token: None 2023-09-22T21:49:12.585 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.585 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jR5rgg4apX8muISX31Cv98Y8Drk= 2023-09-22T21:49:12.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jR5rgg4apX8muISX31Cv98Y8Drk=', 'Content-Length': '0'} 2023-09-22T21:49:12.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001285461c8d672e62-00650e0bd8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.588 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:49:12.589 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:49:12.589 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.589 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.589 DEBUG:boto:Method: HEAD 2023-09-22T21:49:12.589 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.589 DEBUG:boto:Data: 2023-09-22T21:49:12.589 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.589 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.589 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.589 DEBUG:boto:Params: {} 2023-09-22T21:49:12.590 DEBUG:boto:Token: None 2023-09-22T21:49:12.590 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.590 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QJIp1d0udOSASdtB+v3A8nRjumo= 2023-09-22T21:49:12.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QJIp1d0udOSASdtB+v3A8nRjumo=', 'Content-Length': '0'} 2023-09-22T21:49:12.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012db2319964e86dc-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.593 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:49:12.593 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:49:12.594 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.594 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.594 DEBUG:boto:Method: HEAD 2023-09-22T21:49:12.594 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.594 DEBUG:boto:Data: 2023-09-22T21:49:12.594 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.594 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.594 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.594 DEBUG:boto:Params: {} 2023-09-22T21:49:12.594 DEBUG:boto:Token: None 2023-09-22T21:49:12.595 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.595 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QJIp1d0udOSASdtB+v3A8nRjumo= 2023-09-22T21:49:12.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QJIp1d0udOSASdtB+v3A8nRjumo=', 'Content-Length': '0'} 2023-09-22T21:49:12.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b58577fdc4fe70ce-00650e0bd8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-22T21:49:12.598 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:49:12.598 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:49:12.598 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.598 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.598 DEBUG:boto:Method: GET 2023-09-22T21:49:12.598 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.599 DEBUG:boto:Data: 2023-09-22T21:49:12.599 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.599 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.599 DEBUG:boto:Params: {} 2023-09-22T21:49:12.599 DEBUG:boto:Token: None 2023-09-22T21:49:12.599 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.599 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jR5rgg4apX8muISX31Cv98Y8Drk= 2023-09-22T21:49:12.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jR5rgg4apX8muISX31Cv98Y8Drk=', 'Content-Length': '0'} 2023-09-22T21:49:12.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088e209fc3c752eca-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.602 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:49:12.603 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:49:12.603 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.603 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.603 DEBUG:boto:Method: GET 2023-09-22T21:49:12.603 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.603 DEBUG:boto:Data: 2023-09-22T21:49:12.603 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.604 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.604 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.604 DEBUG:boto:Params: {} 2023-09-22T21:49:12.604 DEBUG:boto:Token: None 2023-09-22T21:49:12.604 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:12.604 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jR5rgg4apX8muISX31Cv98Y8Drk= 2023-09-22T21:49:12.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jR5rgg4apX8muISX31Cv98Y8Drk=', 'Content-Length': '0'} 2023-09-22T21:49:12.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084225305c26a232a-00650e0bd8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.607 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:12.607 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:12.608 DEBUG:boto:path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:49:12.608 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:49:12.608 DEBUG:boto:Method: GET 2023-09-22T21:49:12.608 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj58 2023-09-22T21:49:12.608 DEBUG:boto:Data: 2023-09-22T21:49:12.608 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.608 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.608 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.608 DEBUG:boto:Params: {} 2023-09-22T21:49:12.609 DEBUG:boto:Token: None 2023-09-22T21:49:12.609 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/ 2023-09-22T21:49:12.609 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U= 2023-09-22T21:49:12.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U=', 'Content-Length': '0'} 2023-09-22T21:49:12.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008621ee6422ce6a08-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.613 DEBUG:boto:b'ffzdqu-59obj581000falseobj582023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:12.614 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:12.614 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:12.614 DEBUG:boto:path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:49:12.614 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:49:12.614 DEBUG:boto:Method: GET 2023-09-22T21:49:12.615 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj58 2023-09-22T21:49:12.615 DEBUG:boto:Data: 2023-09-22T21:49:12.615 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.615 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.615 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.615 DEBUG:boto:Params: {} 2023-09-22T21:49:12.615 DEBUG:boto:Token: None 2023-09-22T21:49:12.615 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/ 2023-09-22T21:49:12.615 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U= 2023-09-22T21:49:12.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U=', 'Content-Length': '0'} 2023-09-22T21:49:12.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036177284226160ef-00650e0bd8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.666 DEBUG:boto:b'ffzdqu-59obj581000falseobj582023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:12.800 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:49:12.800 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:49:12.800 DEBUG:boto:Method: HEAD 2023-09-22T21:49:12.800 DEBUG:boto:Path: /ffzdqu-59/obj58 2023-09-22T21:49:12.800 DEBUG:boto:Data: 2023-09-22T21:49:12.801 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.801 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.801 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.801 DEBUG:boto:Params: {} 2023-09-22T21:49:12.801 DEBUG:boto:Token: None 2023-09-22T21:49:12.801 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj58 2023-09-22T21:49:12.801 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zHkZKYEAfcqffwifNh9TmfV/j7Y= 2023-09-22T21:49:12.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zHkZKYEAfcqffwifNh9TmfV/j7Y=', 'Content-Length': '0'} 2023-09-22T21:49:12.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d55fa8f5211fa12c-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.805 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:49:12.805 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:49:12.805 DEBUG:boto:Method: HEAD 2023-09-22T21:49:12.805 DEBUG:boto:Path: /ffzdqu-59/obj58 2023-09-22T21:49:12.805 DEBUG:boto:Data: 2023-09-22T21:49:12.805 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.805 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.805 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.806 DEBUG:boto:Params: {} 2023-09-22T21:49:12.806 DEBUG:boto:Token: None 2023-09-22T21:49:12.806 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj58 2023-09-22T21:49:12.806 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zHkZKYEAfcqffwifNh9TmfV/j7Y= 2023-09-22T21:49:12.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zHkZKYEAfcqffwifNh9TmfV/j7Y=', 'Content-Length': '0'} 2023-09-22T21:49:12.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000eeeef8b15a5b025-00650e0bd8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-22T21:49:12.809 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:49:12.810 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:49:12.810 DEBUG:boto:Method: GET 2023-09-22T21:49:12.810 DEBUG:boto:Path: /ffzdqu-59/obj58 2023-09-22T21:49:12.810 DEBUG:boto:Data: 2023-09-22T21:49:12.810 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.810 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.810 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.810 DEBUG:boto:Params: {} 2023-09-22T21:49:12.810 DEBUG:boto:Token: None 2023-09-22T21:49:12.811 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj58 2023-09-22T21:49:12.811 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YanqYmBdKVouAcNbkjYLMldevAg= 2023-09-22T21:49:12.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YanqYmBdKVouAcNbkjYLMldevAg=', 'Content-Length': '0'} 2023-09-22T21:49:12.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a1fdd7b5b8d9b3d-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.854 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:49:12.855 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:49:12.855 DEBUG:boto:Method: GET 2023-09-22T21:49:12.855 DEBUG:boto:Path: /ffzdqu-59/obj58 2023-09-22T21:49:12.855 DEBUG:boto:Data: 2023-09-22T21:49:12.855 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.855 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.855 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.855 DEBUG:boto:Params: {} 2023-09-22T21:49:12.855 DEBUG:boto:Token: None 2023-09-22T21:49:12.856 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj58 2023-09-22T21:49:12.856 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YanqYmBdKVouAcNbkjYLMldevAg= 2023-09-22T21:49:12.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YanqYmBdKVouAcNbkjYLMldevAg=', 'Content-Length': '0'} 2023-09-22T21:49:12.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0ec1dadf77360f6-00650e0bd8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.859 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:12.859 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:12.859 DEBUG:boto:path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:49:12.859 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:49:12.860 DEBUG:boto:Method: GET 2023-09-22T21:49:12.860 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj58 2023-09-22T21:49:12.860 DEBUG:boto:Data: 2023-09-22T21:49:12.860 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.860 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.860 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.860 DEBUG:boto:Params: {} 2023-09-22T21:49:12.860 DEBUG:boto:Token: None 2023-09-22T21:49:12.860 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/ 2023-09-22T21:49:12.861 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U= 2023-09-22T21:49:12.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U=', 'Content-Length': '0'} 2023-09-22T21:49:12.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000895f93471e003ff9-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.865 DEBUG:boto:b'ffzdqu-59obj581000falseobj582023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:12.866 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:12.866 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:12.866 DEBUG:boto:path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:49:12.866 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:49:12.866 DEBUG:boto:Method: GET 2023-09-22T21:49:12.866 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj58 2023-09-22T21:49:12.866 DEBUG:boto:Data: 2023-09-22T21:49:12.866 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.867 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.867 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.867 DEBUG:boto:Params: {} 2023-09-22T21:49:12.867 DEBUG:boto:Token: None 2023-09-22T21:49:12.867 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/ 2023-09-22T21:49:12.867 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U= 2023-09-22T21:49:12.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U=', 'Content-Length': '0'} 2023-09-22T21:49:12.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056e3c982adf7a897-00650e0bd8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.879 DEBUG:boto:b'ffzdqu-59obj581000falseobj582023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:12.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-22T21:49:12.879 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:49:12.879 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:49:12.879 DEBUG:boto:path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.880 DEBUG:boto:auth_path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.880 DEBUG:boto:Method: GET 2023-09-22T21:49:12.880 DEBUG:boto:Path: /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.880 DEBUG:boto:Data: 2023-09-22T21:49:12.880 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.880 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.880 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.880 DEBUG:boto:Params: {} 2023-09-22T21:49:12.880 DEBUG:boto:Token: None 2023-09-22T21:49:12.880 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.881 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qxunZ1unkdfqQFGMjtFxNbJ+4RM= 2023-09-22T21:49:12.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qxunZ1unkdfqQFGMjtFxNbJ+4RM=', 'Content-Length': '0'} 2023-09-22T21:49:12.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0b94bd2c75f5465-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.884 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:49:12.884 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:49:12.885 DEBUG:boto:path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.885 DEBUG:boto:auth_path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.885 DEBUG:boto:Method: GET 2023-09-22T21:49:12.885 DEBUG:boto:Path: /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.885 DEBUG:boto:Data: 2023-09-22T21:49:12.885 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.885 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.885 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.885 DEBUG:boto:Params: {} 2023-09-22T21:49:12.886 DEBUG:boto:Token: None 2023-09-22T21:49:12.886 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.886 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qxunZ1unkdfqQFGMjtFxNbJ+4RM= 2023-09-22T21:49:12.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qxunZ1unkdfqQFGMjtFxNbJ+4RM=', 'Content-Length': '0'} 2023-09-22T21:49:12.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078efce96ac0ac0f1-00650e0bd8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.889 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:49:12.889 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:49:12.889 DEBUG:boto:path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.889 DEBUG:boto:auth_path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.889 DEBUG:boto:Method: HEAD 2023-09-22T21:49:12.890 DEBUG:boto:Path: /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.890 DEBUG:boto:Data: 2023-09-22T21:49:12.890 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.890 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.890 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.890 DEBUG:boto:Params: {} 2023-09-22T21:49:12.890 DEBUG:boto:Token: None 2023-09-22T21:49:12.890 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.891 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CHvBc+K4TwaG72BvJeo2LV5smZE= 2023-09-22T21:49:12.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CHvBc+K4TwaG72BvJeo2LV5smZE=', 'Content-Length': '0'} 2023-09-22T21:49:12.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064cd7c1d49ea6705-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.893 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:49:12.893 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:49:12.893 DEBUG:boto:path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.894 DEBUG:boto:auth_path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.894 DEBUG:boto:Method: HEAD 2023-09-22T21:49:12.894 DEBUG:boto:Path: /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.894 DEBUG:boto:Data: 2023-09-22T21:49:12.894 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.894 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.894 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.894 DEBUG:boto:Params: {} 2023-09-22T21:49:12.894 DEBUG:boto:Token: None 2023-09-22T21:49:12.895 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.895 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CHvBc+K4TwaG72BvJeo2LV5smZE= 2023-09-22T21:49:12.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CHvBc+K4TwaG72BvJeo2LV5smZE=', 'Content-Length': '0'} 2023-09-22T21:49:12.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007472c44fe3a36ed0-00650e0bd8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-22T21:49:12.898 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:49:12.899 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:49:12.899 DEBUG:boto:path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.899 DEBUG:boto:auth_path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.899 DEBUG:boto:Method: GET 2023-09-22T21:49:12.899 DEBUG:boto:Path: /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.899 DEBUG:boto:Data: 2023-09-22T21:49:12.899 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.899 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.899 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.899 DEBUG:boto:Params: {} 2023-09-22T21:49:12.900 DEBUG:boto:Token: None 2023-09-22T21:49:12.900 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.900 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qxunZ1unkdfqQFGMjtFxNbJ+4RM= 2023-09-22T21:49:12.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qxunZ1unkdfqQFGMjtFxNbJ+4RM=', 'Content-Length': '0'} 2023-09-22T21:49:12.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a12a4903b9989803-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.946 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:49:12.947 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:49:12.947 DEBUG:boto:path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.947 DEBUG:boto:auth_path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.947 DEBUG:boto:Method: GET 2023-09-22T21:49:12.947 DEBUG:boto:Path: /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.947 DEBUG:boto:Data: 2023-09-22T21:49:12.947 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.948 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.948 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.948 DEBUG:boto:Params: {} 2023-09-22T21:49:12.948 DEBUG:boto:Token: None 2023-09-22T21:49:12.948 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:12.949 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qxunZ1unkdfqQFGMjtFxNbJ+4RM= 2023-09-22T21:49:12.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qxunZ1unkdfqQFGMjtFxNbJ+4RM=', 'Content-Length': '0'} 2023-09-22T21:49:12.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f956fbd3160595e2-00650e0bd8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.952 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:12.952 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:12.952 DEBUG:boto:path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:49:12.952 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:49:12.952 DEBUG:boto:Method: GET 2023-09-22T21:49:12.953 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj62 2023-09-22T21:49:12.953 DEBUG:boto:Data: 2023-09-22T21:49:12.953 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.953 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:12.953 DEBUG:boto:Port: 8001 2023-09-22T21:49:12.953 DEBUG:boto:Params: {} 2023-09-22T21:49:12.953 DEBUG:boto:Token: None 2023-09-22T21:49:12.953 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/ 2023-09-22T21:49:12.954 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U= 2023-09-22T21:49:12.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U=', 'Content-Length': '0'} 2023-09-22T21:49:12.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7eac9c20f20060a-00650e0bd8-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:12.959 DEBUG:boto:b'ffzdqu-59obj621000falseobj622023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:12.959 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:12.959 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:12.959 DEBUG:boto:path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:49:12.959 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:49:12.960 DEBUG:boto:Method: GET 2023-09-22T21:49:12.960 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj62 2023-09-22T21:49:12.960 DEBUG:boto:Data: 2023-09-22T21:49:12.960 DEBUG:boto:Headers: {} 2023-09-22T21:49:12.960 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:12.960 DEBUG:boto:Port: 8000 2023-09-22T21:49:12.960 DEBUG:boto:Params: {} 2023-09-22T21:49:12.960 DEBUG:boto:Token: None 2023-09-22T21:49:12.961 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:12 GMT /ffzdqu-59/ 2023-09-22T21:49:12.961 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U= 2023-09-22T21:49:12.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T7IStkHdRF0+QyePPAZyYcCt3/U=', 'Content-Length': '0'} 2023-09-22T21:49:12.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005de04058e3c18b9d-00650e0bd8-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.018 DEBUG:boto:b'ffzdqu-59obj621000falseobj622023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:13.019 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:49:13.019 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:49:13.019 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.019 DEBUG:boto:Path: /ffzdqu-59/obj62 2023-09-22T21:49:13.019 DEBUG:boto:Data: 2023-09-22T21:49:13.020 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.020 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.020 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.020 DEBUG:boto:Params: {} 2023-09-22T21:49:13.020 DEBUG:boto:Token: None 2023-09-22T21:49:13.020 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj62 2023-09-22T21:49:13.020 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:97tVDeVgVBMhIXNemg8nGV4yd0s= 2023-09-22T21:49:13.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:97tVDeVgVBMhIXNemg8nGV4yd0s=', 'Content-Length': '0'} 2023-09-22T21:49:13.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005f21a409b6adfe4-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.023 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:49:13.023 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:49:13.024 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.024 DEBUG:boto:Path: /ffzdqu-59/obj62 2023-09-22T21:49:13.024 DEBUG:boto:Data: 2023-09-22T21:49:13.024 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.024 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.024 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.024 DEBUG:boto:Params: {} 2023-09-22T21:49:13.024 DEBUG:boto:Token: None 2023-09-22T21:49:13.024 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj62 2023-09-22T21:49:13.025 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:97tVDeVgVBMhIXNemg8nGV4yd0s= 2023-09-22T21:49:13.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:97tVDeVgVBMhIXNemg8nGV4yd0s=', 'Content-Length': '0'} 2023-09-22T21:49:13.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f822733b5b16030-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-22T21:49:13.028 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:49:13.028 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:49:13.028 DEBUG:boto:Method: GET 2023-09-22T21:49:13.028 DEBUG:boto:Path: /ffzdqu-59/obj62 2023-09-22T21:49:13.028 DEBUG:boto:Data: 2023-09-22T21:49:13.028 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.028 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.029 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.029 DEBUG:boto:Params: {} 2023-09-22T21:49:13.029 DEBUG:boto:Token: None 2023-09-22T21:49:13.029 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj62 2023-09-22T21:49:13.029 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eCIZceuVs1ELnfq3WPyciheAgvI= 2023-09-22T21:49:13.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eCIZceuVs1ELnfq3WPyciheAgvI=', 'Content-Length': '0'} 2023-09-22T21:49:13.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f81104d96a6a83f5-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.032 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:49:13.033 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:49:13.033 DEBUG:boto:Method: GET 2023-09-22T21:49:13.033 DEBUG:boto:Path: /ffzdqu-59/obj62 2023-09-22T21:49:13.033 DEBUG:boto:Data: 2023-09-22T21:49:13.033 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.033 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.033 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.033 DEBUG:boto:Params: {} 2023-09-22T21:49:13.033 DEBUG:boto:Token: None 2023-09-22T21:49:13.034 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj62 2023-09-22T21:49:13.034 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eCIZceuVs1ELnfq3WPyciheAgvI= 2023-09-22T21:49:13.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eCIZceuVs1ELnfq3WPyciheAgvI=', 'Content-Length': '0'} 2023-09-22T21:49:13.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c73169875d4bf2d4-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.037 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:13.037 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:13.037 DEBUG:boto:path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:49:13.037 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:49:13.037 DEBUG:boto:Method: GET 2023-09-22T21:49:13.037 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj62 2023-09-22T21:49:13.037 DEBUG:boto:Data: 2023-09-22T21:49:13.037 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.038 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.038 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.038 DEBUG:boto:Params: {} 2023-09-22T21:49:13.038 DEBUG:boto:Token: None 2023-09-22T21:49:13.038 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/ 2023-09-22T21:49:13.038 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg= 2023-09-22T21:49:13.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg=', 'Content-Length': '0'} 2023-09-22T21:49:13.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075feac876de8f6f1-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.086 DEBUG:boto:b'ffzdqu-59obj621000falseobj622023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:13.087 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:13.087 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:13.087 DEBUG:boto:path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:49:13.087 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:49:13.087 DEBUG:boto:Method: GET 2023-09-22T21:49:13.088 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj62 2023-09-22T21:49:13.088 DEBUG:boto:Data: 2023-09-22T21:49:13.088 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.088 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.088 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.088 DEBUG:boto:Params: {} 2023-09-22T21:49:13.088 DEBUG:boto:Token: None 2023-09-22T21:49:13.089 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/ 2023-09-22T21:49:13.089 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg= 2023-09-22T21:49:13.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg=', 'Content-Length': '0'} 2023-09-22T21:49:13.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef2797c76420bd9d-00650e0bd9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.099 DEBUG:boto:b'ffzdqu-59obj621000falseobj622023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:13.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:49:13.099 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:49:13.099 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:49:13.099 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.100 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.100 DEBUG:boto:Method: GET 2023-09-22T21:49:13.100 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.100 DEBUG:boto:Data: 2023-09-22T21:49:13.100 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.100 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.100 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.100 DEBUG:boto:Params: {} 2023-09-22T21:49:13.100 DEBUG:boto:Token: None 2023-09-22T21:49:13.101 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.101 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yZjsUMgzSG+utQZw3CwXzr+VHIY= 2023-09-22T21:49:13.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yZjsUMgzSG+utQZw3CwXzr+VHIY=', 'Content-Length': '0'} 2023-09-22T21:49:13.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063e25cb0b4decb37-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.104 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:49:13.104 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:49:13.104 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.104 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.104 DEBUG:boto:Method: GET 2023-09-22T21:49:13.104 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.104 DEBUG:boto:Data: 2023-09-22T21:49:13.105 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.105 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.105 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.105 DEBUG:boto:Params: {} 2023-09-22T21:49:13.105 DEBUG:boto:Token: None 2023-09-22T21:49:13.105 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.105 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yZjsUMgzSG+utQZw3CwXzr+VHIY= 2023-09-22T21:49:13.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yZjsUMgzSG+utQZw3CwXzr+VHIY=', 'Content-Length': '0'} 2023-09-22T21:49:13.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed3cc6e52d5c72ac-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.150 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:49:13.151 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:49:13.151 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.151 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.151 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.151 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.151 DEBUG:boto:Data: 2023-09-22T21:49:13.151 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.151 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.152 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.152 DEBUG:boto:Params: {} 2023-09-22T21:49:13.152 DEBUG:boto:Token: None 2023-09-22T21:49:13.152 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.152 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6BWsS94d3pfs4bNMQT2luY4Kx0o= 2023-09-22T21:49:13.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6BWsS94d3pfs4bNMQT2luY4Kx0o=', 'Content-Length': '0'} 2023-09-22T21:49:13.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046bb37b3bfa04388-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.155 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:49:13.155 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:49:13.156 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.156 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.156 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.156 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.156 DEBUG:boto:Data: 2023-09-22T21:49:13.156 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.156 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.156 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.156 DEBUG:boto:Params: {} 2023-09-22T21:49:13.157 DEBUG:boto:Token: None 2023-09-22T21:49:13.157 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.157 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6BWsS94d3pfs4bNMQT2luY4Kx0o= 2023-09-22T21:49:13.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6BWsS94d3pfs4bNMQT2luY4Kx0o=', 'Content-Length': '0'} 2023-09-22T21:49:13.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5a73517d2840c94-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:49:13.160 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:49:13.161 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:49:13.161 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.161 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.161 DEBUG:boto:Method: GET 2023-09-22T21:49:13.161 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.161 DEBUG:boto:Data: 2023-09-22T21:49:13.161 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.161 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.162 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.162 DEBUG:boto:Params: {} 2023-09-22T21:49:13.162 DEBUG:boto:Token: None 2023-09-22T21:49:13.162 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.162 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yZjsUMgzSG+utQZw3CwXzr+VHIY= 2023-09-22T21:49:13.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yZjsUMgzSG+utQZw3CwXzr+VHIY=', 'Content-Length': '0'} 2023-09-22T21:49:13.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091a8d638006ef523-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.165 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:49:13.165 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:49:13.165 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.166 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.166 DEBUG:boto:Method: GET 2023-09-22T21:49:13.166 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.166 DEBUG:boto:Data: 2023-09-22T21:49:13.166 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.166 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.166 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.166 DEBUG:boto:Params: {} 2023-09-22T21:49:13.167 DEBUG:boto:Token: None 2023-09-22T21:49:13.167 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:13.167 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yZjsUMgzSG+utQZw3CwXzr+VHIY= 2023-09-22T21:49:13.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yZjsUMgzSG+utQZw3CwXzr+VHIY=', 'Content-Length': '0'} 2023-09-22T21:49:13.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ed149aa2133030d-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.170 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:13.170 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:13.170 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:49:13.170 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:49:13.170 DEBUG:boto:Method: GET 2023-09-22T21:49:13.171 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:49:13.171 DEBUG:boto:Data: 2023-09-22T21:49:13.171 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.171 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.171 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.171 DEBUG:boto:Params: {} 2023-09-22T21:49:13.171 DEBUG:boto:Token: None 2023-09-22T21:49:13.171 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/ 2023-09-22T21:49:13.172 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg= 2023-09-22T21:49:13.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg=', 'Content-Length': '0'} 2023-09-22T21:49:13.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000249b0389e79f0b16-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.176 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:13.177 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:13.177 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:13.177 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:49:13.177 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:49:13.177 DEBUG:boto:Method: GET 2023-09-22T21:49:13.177 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:49:13.178 DEBUG:boto:Data: 2023-09-22T21:49:13.178 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.178 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.178 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.178 DEBUG:boto:Params: {} 2023-09-22T21:49:13.178 DEBUG:boto:Token: None 2023-09-22T21:49:13.178 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/ 2023-09-22T21:49:13.178 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg= 2023-09-22T21:49:13.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg=', 'Content-Length': '0'} 2023-09-22T21:49:13.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2d80e3801be2cf2-00650e0bd9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.188 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:13.189 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:49:13.189 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:49:13.189 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.189 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:49:13.189 DEBUG:boto:Data: 2023-09-22T21:49:13.189 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.190 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.190 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.190 DEBUG:boto:Params: {} 2023-09-22T21:49:13.190 DEBUG:boto:Token: None 2023-09-22T21:49:13.190 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj68 2023-09-22T21:49:13.190 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WLdPMcvbt91rjUF8YF+4KS6+c7M= 2023-09-22T21:49:13.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WLdPMcvbt91rjUF8YF+4KS6+c7M=', 'Content-Length': '0'} 2023-09-22T21:49:13.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a29e7fa89d26d433-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.194 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:49:13.194 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:49:13.194 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.194 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:49:13.194 DEBUG:boto:Data: 2023-09-22T21:49:13.194 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.194 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.195 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.195 DEBUG:boto:Params: {} 2023-09-22T21:49:13.195 DEBUG:boto:Token: None 2023-09-22T21:49:13.195 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj68 2023-09-22T21:49:13.195 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WLdPMcvbt91rjUF8YF+4KS6+c7M= 2023-09-22T21:49:13.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WLdPMcvbt91rjUF8YF+4KS6+c7M=', 'Content-Length': '0'} 2023-09-22T21:49:13.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000817535bd66638ff6-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:49:13.198 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:49:13.198 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:49:13.199 DEBUG:boto:Method: GET 2023-09-22T21:49:13.199 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:49:13.199 DEBUG:boto:Data: 2023-09-22T21:49:13.199 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.199 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.199 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.199 DEBUG:boto:Params: {} 2023-09-22T21:49:13.199 DEBUG:boto:Token: None 2023-09-22T21:49:13.199 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj68 2023-09-22T21:49:13.200 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JBdhDHsKGuBo9hMdyH5NKg5bvoQ= 2023-09-22T21:49:13.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JBdhDHsKGuBo9hMdyH5NKg5bvoQ=', 'Content-Length': '0'} 2023-09-22T21:49:13.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f15a1d86612c2614-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.246 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:49:13.247 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:49:13.247 DEBUG:boto:Method: GET 2023-09-22T21:49:13.247 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:49:13.247 DEBUG:boto:Data: 2023-09-22T21:49:13.247 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.247 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.247 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.248 DEBUG:boto:Params: {} 2023-09-22T21:49:13.248 DEBUG:boto:Token: None 2023-09-22T21:49:13.248 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj68 2023-09-22T21:49:13.248 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JBdhDHsKGuBo9hMdyH5NKg5bvoQ= 2023-09-22T21:49:13.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JBdhDHsKGuBo9hMdyH5NKg5bvoQ=', 'Content-Length': '0'} 2023-09-22T21:49:13.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ad96862a34449d0-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.251 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:13.252 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:13.252 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:49:13.252 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:49:13.252 DEBUG:boto:Method: GET 2023-09-22T21:49:13.252 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:49:13.252 DEBUG:boto:Data: 2023-09-22T21:49:13.252 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.253 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.253 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.253 DEBUG:boto:Params: {} 2023-09-22T21:49:13.253 DEBUG:boto:Token: None 2023-09-22T21:49:13.253 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/ 2023-09-22T21:49:13.253 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg= 2023-09-22T21:49:13.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg=', 'Content-Length': '0'} 2023-09-22T21:49:13.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002eeff3985b042706-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.258 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:13.259 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:13.259 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:13.259 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:49:13.259 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:49:13.259 DEBUG:boto:Method: GET 2023-09-22T21:49:13.259 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:49:13.259 DEBUG:boto:Data: 2023-09-22T21:49:13.260 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.260 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.260 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.260 DEBUG:boto:Params: {} 2023-09-22T21:49:13.260 DEBUG:boto:Token: None 2023-09-22T21:49:13.260 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/ 2023-09-22T21:49:13.261 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg= 2023-09-22T21:49:13.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg=', 'Content-Length': '0'} 2023-09-22T21:49:13.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000513bf8052b419a5d-00650e0bd9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.271 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:13.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-22T21:49:13.272 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:49:13.272 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:49:13.272 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.273 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.273 DEBUG:boto:Method: GET 2023-09-22T21:49:13.273 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.273 DEBUG:boto:Data: 2023-09-22T21:49:13.273 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.274 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.274 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.274 DEBUG:boto:Params: {} 2023-09-22T21:49:13.274 DEBUG:boto:Token: None 2023-09-22T21:49:13.275 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.275 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AseW3ux6BSHUb5Lyk66/S/vKoDU= 2023-09-22T21:49:13.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AseW3ux6BSHUb5Lyk66/S/vKoDU=', 'Content-Length': '0'} 2023-09-22T21:49:13.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064150bd9b3276845-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.279 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:49:13.279 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:49:13.279 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.279 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.279 DEBUG:boto:Method: GET 2023-09-22T21:49:13.279 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.279 DEBUG:boto:Data: 2023-09-22T21:49:13.280 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.280 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.280 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.280 DEBUG:boto:Params: {} 2023-09-22T21:49:13.280 DEBUG:boto:Token: None 2023-09-22T21:49:13.280 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.281 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AseW3ux6BSHUb5Lyk66/S/vKoDU= 2023-09-22T21:49:13.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AseW3ux6BSHUb5Lyk66/S/vKoDU=', 'Content-Length': '0'} 2023-09-22T21:49:13.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0417863738a793e-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.284 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:49:13.284 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:49:13.284 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.284 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.285 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.285 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.285 DEBUG:boto:Data: 2023-09-22T21:49:13.285 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.285 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.285 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.285 DEBUG:boto:Params: {} 2023-09-22T21:49:13.285 DEBUG:boto:Token: None 2023-09-22T21:49:13.285 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.286 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HoVR6nRP/IgORPk9pPFwoxpEW6Y= 2023-09-22T21:49:13.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HoVR6nRP/IgORPk9pPFwoxpEW6Y=', 'Content-Length': '0'} 2023-09-22T21:49:13.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1560f7f4cd2830a-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.288 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:49:13.289 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:49:13.289 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.289 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.289 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.289 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.289 DEBUG:boto:Data: 2023-09-22T21:49:13.289 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.289 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.289 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.290 DEBUG:boto:Params: {} 2023-09-22T21:49:13.290 DEBUG:boto:Token: None 2023-09-22T21:49:13.290 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.290 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HoVR6nRP/IgORPk9pPFwoxpEW6Y= 2023-09-22T21:49:13.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HoVR6nRP/IgORPk9pPFwoxpEW6Y=', 'Content-Length': '0'} 2023-09-22T21:49:13.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054cf0fabf8e0e94e-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-22T21:49:13.294 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:49:13.294 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:49:13.294 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.294 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.294 DEBUG:boto:Method: GET 2023-09-22T21:49:13.294 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.294 DEBUG:boto:Data: 2023-09-22T21:49:13.295 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.295 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.295 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.295 DEBUG:boto:Params: {} 2023-09-22T21:49:13.295 DEBUG:boto:Token: None 2023-09-22T21:49:13.295 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.295 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AseW3ux6BSHUb5Lyk66/S/vKoDU= 2023-09-22T21:49:13.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AseW3ux6BSHUb5Lyk66/S/vKoDU=', 'Content-Length': '0'} 2023-09-22T21:49:13.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae4c25c35958b4fc-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.338 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:49:13.338 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:49:13.339 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.339 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.339 DEBUG:boto:Method: GET 2023-09-22T21:49:13.339 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.339 DEBUG:boto:Data: 2023-09-22T21:49:13.339 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.339 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.339 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.339 DEBUG:boto:Params: {} 2023-09-22T21:49:13.339 DEBUG:boto:Token: None 2023-09-22T21:49:13.340 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:13.340 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AseW3ux6BSHUb5Lyk66/S/vKoDU= 2023-09-22T21:49:13.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AseW3ux6BSHUb5Lyk66/S/vKoDU=', 'Content-Length': '0'} 2023-09-22T21:49:13.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087d9db72c0475d7c-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.343 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:13.343 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:13.343 DEBUG:boto:path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:49:13.343 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:49:13.344 DEBUG:boto:Method: GET 2023-09-22T21:49:13.344 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj69 2023-09-22T21:49:13.344 DEBUG:boto:Data: 2023-09-22T21:49:13.344 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.344 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.344 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.344 DEBUG:boto:Params: {} 2023-09-22T21:49:13.344 DEBUG:boto:Token: None 2023-09-22T21:49:13.344 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/ 2023-09-22T21:49:13.345 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg= 2023-09-22T21:49:13.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg=', 'Content-Length': '0'} 2023-09-22T21:49:13.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007077dc52d865e499-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.349 DEBUG:boto:b'ffzdqu-59obj691000falseobj692023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:13.349 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:13.350 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:13.350 DEBUG:boto:path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:49:13.350 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:49:13.350 DEBUG:boto:Method: GET 2023-09-22T21:49:13.350 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj69 2023-09-22T21:49:13.350 DEBUG:boto:Data: 2023-09-22T21:49:13.350 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.350 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.350 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.351 DEBUG:boto:Params: {} 2023-09-22T21:49:13.351 DEBUG:boto:Token: None 2023-09-22T21:49:13.351 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/ 2023-09-22T21:49:13.351 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg= 2023-09-22T21:49:13.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg=', 'Content-Length': '0'} 2023-09-22T21:49:13.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b7a00ec7fbdf9c1-00650e0bd9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.361 DEBUG:boto:b'ffzdqu-59obj691000falseobj692023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:13.361 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:49:13.361 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:49:13.361 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.361 DEBUG:boto:Path: /ffzdqu-59/obj69 2023-09-22T21:49:13.361 DEBUG:boto:Data: 2023-09-22T21:49:13.362 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.362 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.362 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.362 DEBUG:boto:Params: {} 2023-09-22T21:49:13.362 DEBUG:boto:Token: None 2023-09-22T21:49:13.362 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj69 2023-09-22T21:49:13.362 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:My4iOc/rbQzlW2KsNXTjW2cqaFI= 2023-09-22T21:49:13.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:My4iOc/rbQzlW2KsNXTjW2cqaFI=', 'Content-Length': '0'} 2023-09-22T21:49:13.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000136d1da1f3b4cc67-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.365 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:49:13.365 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:49:13.365 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.366 DEBUG:boto:Path: /ffzdqu-59/obj69 2023-09-22T21:49:13.366 DEBUG:boto:Data: 2023-09-22T21:49:13.366 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.366 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.366 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.366 DEBUG:boto:Params: {} 2023-09-22T21:49:13.366 DEBUG:boto:Token: None 2023-09-22T21:49:13.366 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj69 2023-09-22T21:49:13.367 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:My4iOc/rbQzlW2KsNXTjW2cqaFI= 2023-09-22T21:49:13.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:My4iOc/rbQzlW2KsNXTjW2cqaFI=', 'Content-Length': '0'} 2023-09-22T21:49:13.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a083d6f5dd1e754c-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-22T21:49:13.370 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:49:13.370 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:49:13.370 DEBUG:boto:Method: GET 2023-09-22T21:49:13.370 DEBUG:boto:Path: /ffzdqu-59/obj69 2023-09-22T21:49:13.370 DEBUG:boto:Data: 2023-09-22T21:49:13.370 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.370 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.370 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.371 DEBUG:boto:Params: {} 2023-09-22T21:49:13.371 DEBUG:boto:Token: None 2023-09-22T21:49:13.371 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj69 2023-09-22T21:49:13.371 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7buMM1xhBJ1n+6BBI/OtTYSESrw= 2023-09-22T21:49:13.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7buMM1xhBJ1n+6BBI/OtTYSESrw=', 'Content-Length': '0'} 2023-09-22T21:49:13.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8dff28de9dd8175-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.374 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:49:13.374 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:49:13.374 DEBUG:boto:Method: GET 2023-09-22T21:49:13.374 DEBUG:boto:Path: /ffzdqu-59/obj69 2023-09-22T21:49:13.374 DEBUG:boto:Data: 2023-09-22T21:49:13.374 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.375 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.375 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.375 DEBUG:boto:Params: {} 2023-09-22T21:49:13.375 DEBUG:boto:Token: None 2023-09-22T21:49:13.375 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj69 2023-09-22T21:49:13.375 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7buMM1xhBJ1n+6BBI/OtTYSESrw= 2023-09-22T21:49:13.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7buMM1xhBJ1n+6BBI/OtTYSESrw=', 'Content-Length': '0'} 2023-09-22T21:49:13.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074720204fffee421-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.378 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:13.378 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:13.378 DEBUG:boto:path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:49:13.378 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:49:13.378 DEBUG:boto:Method: GET 2023-09-22T21:49:13.379 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj69 2023-09-22T21:49:13.379 DEBUG:boto:Data: 2023-09-22T21:49:13.379 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.379 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.379 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.379 DEBUG:boto:Params: {} 2023-09-22T21:49:13.379 DEBUG:boto:Token: None 2023-09-22T21:49:13.379 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/ 2023-09-22T21:49:13.379 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg= 2023-09-22T21:49:13.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg=', 'Content-Length': '0'} 2023-09-22T21:49:13.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057388bfd25fc8fd7-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.384 DEBUG:boto:b'ffzdqu-59obj691000falseobj692023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:13.385 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:13.385 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:13.385 DEBUG:boto:path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:49:13.385 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:49:13.385 DEBUG:boto:Method: GET 2023-09-22T21:49:13.385 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj69 2023-09-22T21:49:13.385 DEBUG:boto:Data: 2023-09-22T21:49:13.386 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.386 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.386 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.386 DEBUG:boto:Params: {} 2023-09-22T21:49:13.386 DEBUG:boto:Token: None 2023-09-22T21:49:13.386 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/ 2023-09-22T21:49:13.386 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg= 2023-09-22T21:49:13.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg=', 'Content-Length': '0'} 2023-09-22T21:49:13.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c102455a7428707a-00650e0bd9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.396 DEBUG:boto:b'ffzdqu-59obj691000falseobj692023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:13.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-09-22T21:49:13.396 DEBUG:boto:path=/ffzdqu-59/obj74 2023-09-22T21:49:13.396 DEBUG:boto:auth_path=/ffzdqu-59/obj74 2023-09-22T21:49:13.396 DEBUG:boto:path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.397 DEBUG:boto:auth_path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.397 DEBUG:boto:Method: GET 2023-09-22T21:49:13.397 DEBUG:boto:Path: /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.397 DEBUG:boto:Data: 2023-09-22T21:49:13.397 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.397 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.397 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.397 DEBUG:boto:Params: {} 2023-09-22T21:49:13.397 DEBUG:boto:Token: None 2023-09-22T21:49:13.397 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.398 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9ObeuXc+BB4S1w+oAwfVBh745M8= 2023-09-22T21:49:13.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9ObeuXc+BB4S1w+oAwfVBh745M8=', 'Content-Length': '0'} 2023-09-22T21:49:13.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d06bf75a7ecc35ef-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.442 DEBUG:boto:path=/ffzdqu-59/obj74 2023-09-22T21:49:13.442 DEBUG:boto:auth_path=/ffzdqu-59/obj74 2023-09-22T21:49:13.443 DEBUG:boto:path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.443 DEBUG:boto:auth_path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.443 DEBUG:boto:Method: GET 2023-09-22T21:49:13.443 DEBUG:boto:Path: /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.443 DEBUG:boto:Data: 2023-09-22T21:49:13.443 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.443 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.443 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.443 DEBUG:boto:Params: {} 2023-09-22T21:49:13.443 DEBUG:boto:Token: None 2023-09-22T21:49:13.444 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.444 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9ObeuXc+BB4S1w+oAwfVBh745M8= 2023-09-22T21:49:13.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9ObeuXc+BB4S1w+oAwfVBh745M8=', 'Content-Length': '0'} 2023-09-22T21:49:13.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002239f7328e40f280-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.447 DEBUG:boto:path=/ffzdqu-59/obj74 2023-09-22T21:49:13.447 DEBUG:boto:auth_path=/ffzdqu-59/obj74 2023-09-22T21:49:13.447 DEBUG:boto:path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.447 DEBUG:boto:auth_path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.447 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.447 DEBUG:boto:Path: /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.447 DEBUG:boto:Data: 2023-09-22T21:49:13.447 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.447 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.448 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.448 DEBUG:boto:Params: {} 2023-09-22T21:49:13.448 DEBUG:boto:Token: None 2023-09-22T21:49:13.448 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.448 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Lx8Y90KO/GmWsAWejLmXYdO1yVU= 2023-09-22T21:49:13.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Lx8Y90KO/GmWsAWejLmXYdO1yVU=', 'Content-Length': '0'} 2023-09-22T21:49:13.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063004d9b45ce6d98-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.451 DEBUG:boto:path=/ffzdqu-59/obj74 2023-09-22T21:49:13.451 DEBUG:boto:auth_path=/ffzdqu-59/obj74 2023-09-22T21:49:13.451 DEBUG:boto:path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.451 DEBUG:boto:auth_path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.451 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.451 DEBUG:boto:Path: /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.452 DEBUG:boto:Data: 2023-09-22T21:49:13.452 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.452 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.452 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.452 DEBUG:boto:Params: {} 2023-09-22T21:49:13.452 DEBUG:boto:Token: None 2023-09-22T21:49:13.452 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.452 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Lx8Y90KO/GmWsAWejLmXYdO1yVU= 2023-09-22T21:49:13.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Lx8Y90KO/GmWsAWejLmXYdO1yVU=', 'Content-Length': '0'} 2023-09-22T21:49:13.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024740b776b69de15-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-09-22T21:49:13.456 DEBUG:boto:path=/ffzdqu-59/obj74 2023-09-22T21:49:13.456 DEBUG:boto:auth_path=/ffzdqu-59/obj74 2023-09-22T21:49:13.456 DEBUG:boto:path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.456 DEBUG:boto:auth_path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.456 DEBUG:boto:Method: GET 2023-09-22T21:49:13.457 DEBUG:boto:Path: /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.457 DEBUG:boto:Data: 2023-09-22T21:49:13.457 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.457 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.457 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.457 DEBUG:boto:Params: {} 2023-09-22T21:49:13.457 DEBUG:boto:Token: None 2023-09-22T21:49:13.457 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.457 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9ObeuXc+BB4S1w+oAwfVBh745M8= 2023-09-22T21:49:13.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9ObeuXc+BB4S1w+oAwfVBh745M8=', 'Content-Length': '0'} 2023-09-22T21:49:13.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065c5fda93c0a7e7c-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.460 DEBUG:boto:path=/ffzdqu-59/obj74 2023-09-22T21:49:13.461 DEBUG:boto:auth_path=/ffzdqu-59/obj74 2023-09-22T21:49:13.461 DEBUG:boto:path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.461 DEBUG:boto:auth_path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.461 DEBUG:boto:Method: GET 2023-09-22T21:49:13.461 DEBUG:boto:Path: /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.461 DEBUG:boto:Data: 2023-09-22T21:49:13.461 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.461 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.461 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.461 DEBUG:boto:Params: {} 2023-09-22T21:49:13.462 DEBUG:boto:Token: None 2023-09-22T21:49:13.462 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:13.462 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9ObeuXc+BB4S1w+oAwfVBh745M8= 2023-09-22T21:49:13.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9ObeuXc+BB4S1w+oAwfVBh745M8=', 'Content-Length': '0'} 2023-09-22T21:49:13.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001159b9626dcd516b-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.506 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:13.506 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:13.507 DEBUG:boto:path=/ffzdqu-59/?prefix=obj74 2023-09-22T21:49:13.507 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj74 2023-09-22T21:49:13.507 DEBUG:boto:Method: GET 2023-09-22T21:49:13.507 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj74 2023-09-22T21:49:13.507 DEBUG:boto:Data: 2023-09-22T21:49:13.507 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.507 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.507 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.507 DEBUG:boto:Params: {} 2023-09-22T21:49:13.507 DEBUG:boto:Token: None 2023-09-22T21:49:13.508 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/ 2023-09-22T21:49:13.508 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg= 2023-09-22T21:49:13.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg=', 'Content-Length': '0'} 2023-09-22T21:49:13.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072295f91abded1e6-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.512 DEBUG:boto:b'ffzdqu-59obj741000falseobj742023-09-22T21:47:16.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:13.513 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:13.513 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:13.513 DEBUG:boto:path=/ffzdqu-59/?prefix=obj74 2023-09-22T21:49:13.513 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj74 2023-09-22T21:49:13.513 DEBUG:boto:Method: GET 2023-09-22T21:49:13.513 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj74 2023-09-22T21:49:13.513 DEBUG:boto:Data: 2023-09-22T21:49:13.513 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.513 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.514 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.514 DEBUG:boto:Params: {} 2023-09-22T21:49:13.514 DEBUG:boto:Token: None 2023-09-22T21:49:13.514 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/ 2023-09-22T21:49:13.514 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg= 2023-09-22T21:49:13.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg=', 'Content-Length': '0'} 2023-09-22T21:49:13.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d01b130b8295c9c6-00650e0bd9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.524 DEBUG:boto:b'ffzdqu-59obj741000falseobj742023-09-22T21:47:16.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:13.524 DEBUG:boto:path=/ffzdqu-59/obj74 2023-09-22T21:49:13.524 DEBUG:boto:auth_path=/ffzdqu-59/obj74 2023-09-22T21:49:13.524 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.525 DEBUG:boto:Path: /ffzdqu-59/obj74 2023-09-22T21:49:13.525 DEBUG:boto:Data: 2023-09-22T21:49:13.525 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.525 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.525 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.525 DEBUG:boto:Params: {} 2023-09-22T21:49:13.525 DEBUG:boto:Token: None 2023-09-22T21:49:13.525 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj74 2023-09-22T21:49:13.525 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:d+5Ycgaz9LmAp5oVVQTFOnGoiY8= 2023-09-22T21:49:13.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:d+5Ycgaz9LmAp5oVVQTFOnGoiY8=', 'Content-Length': '0'} 2023-09-22T21:49:13.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c0c4116e0d9f411-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.528 DEBUG:boto:path=/ffzdqu-59/obj74 2023-09-22T21:49:13.529 DEBUG:boto:auth_path=/ffzdqu-59/obj74 2023-09-22T21:49:13.529 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.529 DEBUG:boto:Path: /ffzdqu-59/obj74 2023-09-22T21:49:13.529 DEBUG:boto:Data: 2023-09-22T21:49:13.529 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.529 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.529 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.529 DEBUG:boto:Params: {} 2023-09-22T21:49:13.529 DEBUG:boto:Token: None 2023-09-22T21:49:13.530 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj74 2023-09-22T21:49:13.530 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:d+5Ycgaz9LmAp5oVVQTFOnGoiY8= 2023-09-22T21:49:13.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:d+5Ycgaz9LmAp5oVVQTFOnGoiY8=', 'Content-Length': '0'} 2023-09-22T21:49:13.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000108111d8d171864e-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-09-22T21:49:13.533 DEBUG:boto:path=/ffzdqu-59/obj74 2023-09-22T21:49:13.533 DEBUG:boto:auth_path=/ffzdqu-59/obj74 2023-09-22T21:49:13.533 DEBUG:boto:Method: GET 2023-09-22T21:49:13.534 DEBUG:boto:Path: /ffzdqu-59/obj74 2023-09-22T21:49:13.534 DEBUG:boto:Data: 2023-09-22T21:49:13.534 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.534 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.534 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.534 DEBUG:boto:Params: {} 2023-09-22T21:49:13.534 DEBUG:boto:Token: None 2023-09-22T21:49:13.534 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj74 2023-09-22T21:49:13.534 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rfzkDolTKsXDn3UnbHtjO77t91U= 2023-09-22T21:49:13.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rfzkDolTKsXDn3UnbHtjO77t91U=', 'Content-Length': '0'} 2023-09-22T21:49:13.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006dfec5d12ae3295-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.578 DEBUG:boto:path=/ffzdqu-59/obj74 2023-09-22T21:49:13.578 DEBUG:boto:auth_path=/ffzdqu-59/obj74 2023-09-22T21:49:13.579 DEBUG:boto:Method: GET 2023-09-22T21:49:13.579 DEBUG:boto:Path: /ffzdqu-59/obj74 2023-09-22T21:49:13.579 DEBUG:boto:Data: 2023-09-22T21:49:13.579 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.579 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.579 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.579 DEBUG:boto:Params: {} 2023-09-22T21:49:13.579 DEBUG:boto:Token: None 2023-09-22T21:49:13.580 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj74 2023-09-22T21:49:13.580 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rfzkDolTKsXDn3UnbHtjO77t91U= 2023-09-22T21:49:13.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rfzkDolTKsXDn3UnbHtjO77t91U=', 'Content-Length': '0'} 2023-09-22T21:49:13.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000623cef3994094021-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.583 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:13.583 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:13.583 DEBUG:boto:path=/ffzdqu-59/?prefix=obj74 2023-09-22T21:49:13.583 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj74 2023-09-22T21:49:13.583 DEBUG:boto:Method: GET 2023-09-22T21:49:13.583 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj74 2023-09-22T21:49:13.583 DEBUG:boto:Data: 2023-09-22T21:49:13.583 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.583 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.584 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.584 DEBUG:boto:Params: {} 2023-09-22T21:49:13.584 DEBUG:boto:Token: None 2023-09-22T21:49:13.584 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/ 2023-09-22T21:49:13.584 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg= 2023-09-22T21:49:13.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg=', 'Content-Length': '0'} 2023-09-22T21:49:13.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038560a31d0174589-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.589 DEBUG:boto:b'ffzdqu-59obj741000falseobj742023-09-22T21:47:16.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:13.589 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:13.589 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:13.589 DEBUG:boto:path=/ffzdqu-59/?prefix=obj74 2023-09-22T21:49:13.589 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj74 2023-09-22T21:49:13.590 DEBUG:boto:Method: GET 2023-09-22T21:49:13.590 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj74 2023-09-22T21:49:13.590 DEBUG:boto:Data: 2023-09-22T21:49:13.590 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.590 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.590 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.590 DEBUG:boto:Params: {} 2023-09-22T21:49:13.590 DEBUG:boto:Token: None 2023-09-22T21:49:13.590 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/ 2023-09-22T21:49:13.591 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg= 2023-09-22T21:49:13.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg=', 'Content-Length': '0'} 2023-09-22T21:49:13.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066f6e7b61bf2ce77-00650e0bd9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.642 DEBUG:boto:b'ffzdqu-59obj741000falseobj742023-09-22T21:47:16.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:13.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T21:49:13.643 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:49:13.643 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:49:13.643 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.643 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.643 DEBUG:boto:Method: GET 2023-09-22T21:49:13.643 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.644 DEBUG:boto:Data: 2023-09-22T21:49:13.644 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.644 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.644 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.644 DEBUG:boto:Params: {} 2023-09-22T21:49:13.644 DEBUG:boto:Token: None 2023-09-22T21:49:13.644 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.644 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uDLOqPopqhciGDXBjPLmnTZlAjE= 2023-09-22T21:49:13.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uDLOqPopqhciGDXBjPLmnTZlAjE=', 'Content-Length': '0'} 2023-09-22T21:49:13.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000903741d4f6ab660a-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.648 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:49:13.648 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:49:13.648 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.648 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.648 DEBUG:boto:Method: GET 2023-09-22T21:49:13.649 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.649 DEBUG:boto:Data: 2023-09-22T21:49:13.649 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.649 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.649 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.649 DEBUG:boto:Params: {} 2023-09-22T21:49:13.649 DEBUG:boto:Token: None 2023-09-22T21:49:13.649 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.649 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uDLOqPopqhciGDXBjPLmnTZlAjE= 2023-09-22T21:49:13.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uDLOqPopqhciGDXBjPLmnTZlAjE=', 'Content-Length': '0'} 2023-09-22T21:49:13.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de67381d1b90c5a5-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.653 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:49:13.653 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:49:13.653 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.653 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.653 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.654 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.654 DEBUG:boto:Data: 2023-09-22T21:49:13.654 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.654 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.654 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.654 DEBUG:boto:Params: {} 2023-09-22T21:49:13.654 DEBUG:boto:Token: None 2023-09-22T21:49:13.654 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.655 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MmSUinuMuC08NZ5NDlYUWFF7rRg= 2023-09-22T21:49:13.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MmSUinuMuC08NZ5NDlYUWFF7rRg=', 'Content-Length': '0'} 2023-09-22T21:49:13.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa60f6a2fe408713-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.658 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:49:13.658 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:49:13.658 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.658 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.658 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.658 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.658 DEBUG:boto:Data: 2023-09-22T21:49:13.658 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.658 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.659 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.659 DEBUG:boto:Params: {} 2023-09-22T21:49:13.659 DEBUG:boto:Token: None 2023-09-22T21:49:13.659 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.659 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MmSUinuMuC08NZ5NDlYUWFF7rRg= 2023-09-22T21:49:13.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MmSUinuMuC08NZ5NDlYUWFF7rRg=', 'Content-Length': '0'} 2023-09-22T21:49:13.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040c3a32fd50fc341-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T21:49:13.662 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:49:13.662 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:49:13.662 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.663 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.663 DEBUG:boto:Method: GET 2023-09-22T21:49:13.663 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.663 DEBUG:boto:Data: 2023-09-22T21:49:13.663 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.663 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.663 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.663 DEBUG:boto:Params: {} 2023-09-22T21:49:13.663 DEBUG:boto:Token: None 2023-09-22T21:49:13.663 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.664 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uDLOqPopqhciGDXBjPLmnTZlAjE= 2023-09-22T21:49:13.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uDLOqPopqhciGDXBjPLmnTZlAjE=', 'Content-Length': '0'} 2023-09-22T21:49:13.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e55daf6f3df7a3a-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.667 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:49:13.667 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:49:13.667 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.667 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.667 DEBUG:boto:Method: GET 2023-09-22T21:49:13.667 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.667 DEBUG:boto:Data: 2023-09-22T21:49:13.667 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.668 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.668 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.668 DEBUG:boto:Params: {} 2023-09-22T21:49:13.668 DEBUG:boto:Token: None 2023-09-22T21:49:13.668 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:13.668 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uDLOqPopqhciGDXBjPLmnTZlAjE= 2023-09-22T21:49:13.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uDLOqPopqhciGDXBjPLmnTZlAjE=', 'Content-Length': '0'} 2023-09-22T21:49:13.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bde95432cb8b3fbb-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.671 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:13.671 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:13.671 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:49:13.672 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:49:13.672 DEBUG:boto:Method: GET 2023-09-22T21:49:13.672 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:49:13.672 DEBUG:boto:Data: 2023-09-22T21:49:13.672 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.672 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.672 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.672 DEBUG:boto:Params: {} 2023-09-22T21:49:13.672 DEBUG:boto:Token: None 2023-09-22T21:49:13.672 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/ 2023-09-22T21:49:13.673 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg= 2023-09-22T21:49:13.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg=', 'Content-Length': '0'} 2023-09-22T21:49:13.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009acf07d630dd71d7-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.718 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:13.719 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:13.719 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:13.719 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:49:13.719 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:49:13.719 DEBUG:boto:Method: GET 2023-09-22T21:49:13.719 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:49:13.719 DEBUG:boto:Data: 2023-09-22T21:49:13.719 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.719 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.720 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.720 DEBUG:boto:Params: {} 2023-09-22T21:49:13.720 DEBUG:boto:Token: None 2023-09-22T21:49:13.720 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/ 2023-09-22T21:49:13.720 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg= 2023-09-22T21:49:13.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg=', 'Content-Length': '0'} 2023-09-22T21:49:13.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc965ca714e39f9d-00650e0bd9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.731 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:13.731 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:49:13.731 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:49:13.731 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.731 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:49:13.731 DEBUG:boto:Data: 2023-09-22T21:49:13.732 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.732 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.732 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.732 DEBUG:boto:Params: {} 2023-09-22T21:49:13.732 DEBUG:boto:Token: None 2023-09-22T21:49:13.732 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj76 2023-09-22T21:49:13.732 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9IntFAkv0Kcsbamhh0BFZ8lXu+o= 2023-09-22T21:49:13.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9IntFAkv0Kcsbamhh0BFZ8lXu+o=', 'Content-Length': '0'} 2023-09-22T21:49:13.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004882be27dd1a7ee3-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.735 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:49:13.736 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:49:13.736 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.736 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:49:13.736 DEBUG:boto:Data: 2023-09-22T21:49:13.736 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.736 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.736 DEBUG:boto:Params: {} 2023-09-22T21:49:13.736 DEBUG:boto:Token: None 2023-09-22T21:49:13.737 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj76 2023-09-22T21:49:13.737 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9IntFAkv0Kcsbamhh0BFZ8lXu+o= 2023-09-22T21:49:13.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9IntFAkv0Kcsbamhh0BFZ8lXu+o=', 'Content-Length': '0'} 2023-09-22T21:49:13.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4aed68332521309-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T21:49:13.740 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:49:13.740 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:49:13.740 DEBUG:boto:Method: GET 2023-09-22T21:49:13.740 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:49:13.740 DEBUG:boto:Data: 2023-09-22T21:49:13.740 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.740 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.741 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.741 DEBUG:boto:Params: {} 2023-09-22T21:49:13.741 DEBUG:boto:Token: None 2023-09-22T21:49:13.741 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj76 2023-09-22T21:49:13.741 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cFrM+bxm9idVainhCxgjz98+MDM= 2023-09-22T21:49:13.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cFrM+bxm9idVainhCxgjz98+MDM=', 'Content-Length': '0'} 2023-09-22T21:49:13.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1599e34bfdfef7d-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.744 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:49:13.744 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:49:13.744 DEBUG:boto:Method: GET 2023-09-22T21:49:13.744 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:49:13.745 DEBUG:boto:Data: 2023-09-22T21:49:13.745 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.745 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.745 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.745 DEBUG:boto:Params: {} 2023-09-22T21:49:13.745 DEBUG:boto:Token: None 2023-09-22T21:49:13.745 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj76 2023-09-22T21:49:13.745 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cFrM+bxm9idVainhCxgjz98+MDM= 2023-09-22T21:49:13.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cFrM+bxm9idVainhCxgjz98+MDM=', 'Content-Length': '0'} 2023-09-22T21:49:13.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ebd9aeb4916a1f0-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.790 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:13.791 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:13.791 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:49:13.791 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:49:13.791 DEBUG:boto:Method: GET 2023-09-22T21:49:13.791 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:49:13.791 DEBUG:boto:Data: 2023-09-22T21:49:13.791 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.791 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.791 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.791 DEBUG:boto:Params: {} 2023-09-22T21:49:13.792 DEBUG:boto:Token: None 2023-09-22T21:49:13.792 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/ 2023-09-22T21:49:13.792 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg= 2023-09-22T21:49:13.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg=', 'Content-Length': '0'} 2023-09-22T21:49:13.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000613487f47ac2d47a-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.797 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:13.797 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:13.797 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:13.797 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:49:13.797 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:49:13.797 DEBUG:boto:Method: GET 2023-09-22T21:49:13.798 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:49:13.798 DEBUG:boto:Data: 2023-09-22T21:49:13.798 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.798 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.798 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.798 DEBUG:boto:Params: {} 2023-09-22T21:49:13.798 DEBUG:boto:Token: None 2023-09-22T21:49:13.798 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/ 2023-09-22T21:49:13.798 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg= 2023-09-22T21:49:13.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg=', 'Content-Length': '0'} 2023-09-22T21:49:13.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000a35f6d490e9ecf-00650e0bd9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.808 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:13.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:49:13.809 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:49:13.809 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:49:13.809 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.809 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.809 DEBUG:boto:Method: GET 2023-09-22T21:49:13.809 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.809 DEBUG:boto:Data: 2023-09-22T21:49:13.809 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.809 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.810 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.810 DEBUG:boto:Params: {} 2023-09-22T21:49:13.810 DEBUG:boto:Token: None 2023-09-22T21:49:13.810 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.810 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EZia/zc8CUgJTMxygcFMKF1EKig= 2023-09-22T21:49:13.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EZia/zc8CUgJTMxygcFMKF1EKig=', 'Content-Length': '0'} 2023-09-22T21:49:13.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000793cd381abd26e2e-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.854 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:49:13.855 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:49:13.855 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.855 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.855 DEBUG:boto:Method: GET 2023-09-22T21:49:13.855 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.855 DEBUG:boto:Data: 2023-09-22T21:49:13.855 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.855 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.855 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.855 DEBUG:boto:Params: {} 2023-09-22T21:49:13.856 DEBUG:boto:Token: None 2023-09-22T21:49:13.856 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.856 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EZia/zc8CUgJTMxygcFMKF1EKig= 2023-09-22T21:49:13.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EZia/zc8CUgJTMxygcFMKF1EKig=', 'Content-Length': '0'} 2023-09-22T21:49:13.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064542e2a13f6854b-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.860 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:49:13.860 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:49:13.861 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.861 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.861 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.861 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.861 DEBUG:boto:Data: 2023-09-22T21:49:13.861 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.861 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.861 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.861 DEBUG:boto:Params: {} 2023-09-22T21:49:13.861 DEBUG:boto:Token: None 2023-09-22T21:49:13.861 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.862 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0czNQ5u8xW848WS49o97Wkjyo2g= 2023-09-22T21:49:13.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0czNQ5u8xW848WS49o97Wkjyo2g=', 'Content-Length': '0'} 2023-09-22T21:49:13.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089f7b4952304cb21-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.865 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:49:13.865 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:49:13.865 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.866 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.866 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.866 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.866 DEBUG:boto:Data: 2023-09-22T21:49:13.866 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.866 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.866 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.866 DEBUG:boto:Params: {} 2023-09-22T21:49:13.866 DEBUG:boto:Token: None 2023-09-22T21:49:13.867 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.867 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0czNQ5u8xW848WS49o97Wkjyo2g= 2023-09-22T21:49:13.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0czNQ5u8xW848WS49o97Wkjyo2g=', 'Content-Length': '0'} 2023-09-22T21:49:13.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c12ade35c61cf297-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:49:13.870 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:49:13.870 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:49:13.870 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.870 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.870 DEBUG:boto:Method: GET 2023-09-22T21:49:13.870 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.870 DEBUG:boto:Data: 2023-09-22T21:49:13.871 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.871 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.871 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.871 DEBUG:boto:Params: {} 2023-09-22T21:49:13.871 DEBUG:boto:Token: None 2023-09-22T21:49:13.871 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.871 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EZia/zc8CUgJTMxygcFMKF1EKig= 2023-09-22T21:49:13.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EZia/zc8CUgJTMxygcFMKF1EKig=', 'Content-Length': '0'} 2023-09-22T21:49:13.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060e51f2bc12ff0b4-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.874 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:49:13.874 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:49:13.875 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.875 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.875 DEBUG:boto:Method: GET 2023-09-22T21:49:13.875 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.875 DEBUG:boto:Data: 2023-09-22T21:49:13.875 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.875 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.875 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.875 DEBUG:boto:Params: {} 2023-09-22T21:49:13.875 DEBUG:boto:Token: None 2023-09-22T21:49:13.876 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:13.876 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EZia/zc8CUgJTMxygcFMKF1EKig= 2023-09-22T21:49:13.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EZia/zc8CUgJTMxygcFMKF1EKig=', 'Content-Length': '0'} 2023-09-22T21:49:13.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007982d31511d687f3-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.879 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:13.879 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:13.879 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:49:13.879 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:49:13.879 DEBUG:boto:Method: GET 2023-09-22T21:49:13.879 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:49:13.879 DEBUG:boto:Data: 2023-09-22T21:49:13.879 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.880 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.880 DEBUG:boto:Params: {} 2023-09-22T21:49:13.880 DEBUG:boto:Token: None 2023-09-22T21:49:13.880 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/ 2023-09-22T21:49:13.880 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg= 2023-09-22T21:49:13.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg=', 'Content-Length': '0'} 2023-09-22T21:49:13.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6950a4a0b5ca674-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.885 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:13.885 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:13.885 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:13.885 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:49:13.886 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:49:13.886 DEBUG:boto:Method: GET 2023-09-22T21:49:13.886 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:49:13.886 DEBUG:boto:Data: 2023-09-22T21:49:13.886 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.886 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.886 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.886 DEBUG:boto:Params: {} 2023-09-22T21:49:13.886 DEBUG:boto:Token: None 2023-09-22T21:49:13.887 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/ 2023-09-22T21:49:13.887 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg= 2023-09-22T21:49:13.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg=', 'Content-Length': '0'} 2023-09-22T21:49:13.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd3419dc1a4ec2a0-00650e0bd9-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.938 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:13.939 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:49:13.939 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:49:13.939 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.939 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:49:13.939 DEBUG:boto:Data: 2023-09-22T21:49:13.939 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.939 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.939 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.940 DEBUG:boto:Params: {} 2023-09-22T21:49:13.940 DEBUG:boto:Token: None 2023-09-22T21:49:13.940 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj85 2023-09-22T21:49:13.940 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UewztbUMCKowdPgCucZOrCTEfV8= 2023-09-22T21:49:13.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UewztbUMCKowdPgCucZOrCTEfV8=', 'Content-Length': '0'} 2023-09-22T21:49:13.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f591955d7b87d38d-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.943 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:49:13.943 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:49:13.943 DEBUG:boto:Method: HEAD 2023-09-22T21:49:13.943 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:49:13.944 DEBUG:boto:Data: 2023-09-22T21:49:13.944 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.944 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.944 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.944 DEBUG:boto:Params: {} 2023-09-22T21:49:13.944 DEBUG:boto:Token: None 2023-09-22T21:49:13.944 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj85 2023-09-22T21:49:13.944 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UewztbUMCKowdPgCucZOrCTEfV8= 2023-09-22T21:49:13.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UewztbUMCKowdPgCucZOrCTEfV8=', 'Content-Length': '0'} 2023-09-22T21:49:13.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c46525ca8796e5f9-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:49:13.947 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:49:13.947 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:49:13.948 DEBUG:boto:Method: GET 2023-09-22T21:49:13.948 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:49:13.948 DEBUG:boto:Data: 2023-09-22T21:49:13.948 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.948 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.948 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.948 DEBUG:boto:Params: {} 2023-09-22T21:49:13.948 DEBUG:boto:Token: None 2023-09-22T21:49:13.948 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj85 2023-09-22T21:49:13.949 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kL7ZCcF4c9n2BnugJY/bdO4pNUg= 2023-09-22T21:49:13.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kL7ZCcF4c9n2BnugJY/bdO4pNUg=', 'Content-Length': '0'} 2023-09-22T21:49:13.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005606d17d5b2d259d-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.952 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:49:13.952 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:49:13.953 DEBUG:boto:Method: GET 2023-09-22T21:49:13.953 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:49:13.953 DEBUG:boto:Data: 2023-09-22T21:49:13.953 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.953 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:13.953 DEBUG:boto:Port: 8000 2023-09-22T21:49:13.953 DEBUG:boto:Params: {} 2023-09-22T21:49:13.953 DEBUG:boto:Token: None 2023-09-22T21:49:13.953 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/obj85 2023-09-22T21:49:13.954 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kL7ZCcF4c9n2BnugJY/bdO4pNUg= 2023-09-22T21:49:13.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kL7ZCcF4c9n2BnugJY/bdO4pNUg=', 'Content-Length': '0'} 2023-09-22T21:49:13.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db3216877e054afe-00650e0bd9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:13.956 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:13.956 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:13.956 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:49:13.957 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:49:13.957 DEBUG:boto:Method: GET 2023-09-22T21:49:13.957 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:49:13.957 DEBUG:boto:Data: 2023-09-22T21:49:13.957 DEBUG:boto:Headers: {} 2023-09-22T21:49:13.957 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:13.957 DEBUG:boto:Port: 8001 2023-09-22T21:49:13.957 DEBUG:boto:Params: {} 2023-09-22T21:49:13.957 DEBUG:boto:Token: None 2023-09-22T21:49:13.957 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:13 GMT /ffzdqu-59/ 2023-09-22T21:49:13.958 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg= 2023-09-22T21:49:13.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/z4MxNEW4BXAMOeIy8aoufFZWxg=', 'Content-Length': '0'} 2023-09-22T21:49:13.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000244ed0355353f6db-00650e0bd9-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.002 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:14.003 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:14.003 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:14.003 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:49:14.003 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:49:14.003 DEBUG:boto:Method: GET 2023-09-22T21:49:14.003 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:49:14.003 DEBUG:boto:Data: 2023-09-22T21:49:14.003 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.004 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.004 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.004 DEBUG:boto:Params: {} 2023-09-22T21:49:14.004 DEBUG:boto:Token: None 2023-09-22T21:49:14.004 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/ 2023-09-22T21:49:14.004 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o= 2023-09-22T21:49:14.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o=', 'Content-Length': '0'} 2023-09-22T21:49:14.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007815cf3a5f5a99d4-00650e0bda-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.014 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:14.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-22T21:49:14.015 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:49:14.015 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:49:14.015 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.015 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.015 DEBUG:boto:Method: GET 2023-09-22T21:49:14.015 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.015 DEBUG:boto:Data: 2023-09-22T21:49:14.015 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.016 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.016 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.016 DEBUG:boto:Params: {} 2023-09-22T21:49:14.016 DEBUG:boto:Token: None 2023-09-22T21:49:14.016 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.016 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cTHtgj4CHrJMu1j9hW6kDVnpeww= 2023-09-22T21:49:14.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cTHtgj4CHrJMu1j9hW6kDVnpeww=', 'Content-Length': '0'} 2023-09-22T21:49:14.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6a2cabd18d01a28-00650e0bda-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.020 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:49:14.020 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:49:14.020 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.020 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.020 DEBUG:boto:Method: GET 2023-09-22T21:49:14.020 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.020 DEBUG:boto:Data: 2023-09-22T21:49:14.020 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.020 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.021 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.021 DEBUG:boto:Params: {} 2023-09-22T21:49:14.021 DEBUG:boto:Token: None 2023-09-22T21:49:14.021 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.021 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cTHtgj4CHrJMu1j9hW6kDVnpeww= 2023-09-22T21:49:14.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cTHtgj4CHrJMu1j9hW6kDVnpeww=', 'Content-Length': '0'} 2023-09-22T21:49:14.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed6e3e618bcdb25c-00650e0bda-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.066 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:49:14.067 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:49:14.067 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.067 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.067 DEBUG:boto:Method: HEAD 2023-09-22T21:49:14.067 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.067 DEBUG:boto:Data: 2023-09-22T21:49:14.067 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.067 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.067 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.067 DEBUG:boto:Params: {} 2023-09-22T21:49:14.068 DEBUG:boto:Token: None 2023-09-22T21:49:14.068 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.068 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tcEiHPi58zRBjb/1yGuyphLbkME= 2023-09-22T21:49:14.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tcEiHPi58zRBjb/1yGuyphLbkME=', 'Content-Length': '0'} 2023-09-22T21:49:14.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091ab41113dcf2485-00650e0bda-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.071 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:49:14.071 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:49:14.071 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.071 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.071 DEBUG:boto:Method: HEAD 2023-09-22T21:49:14.071 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.072 DEBUG:boto:Data: 2023-09-22T21:49:14.072 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.072 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.072 DEBUG:boto:Params: {} 2023-09-22T21:49:14.072 DEBUG:boto:Token: None 2023-09-22T21:49:14.072 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.072 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tcEiHPi58zRBjb/1yGuyphLbkME= 2023-09-22T21:49:14.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tcEiHPi58zRBjb/1yGuyphLbkME=', 'Content-Length': '0'} 2023-09-22T21:49:14.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c8469b6805f5127-00650e0bda-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-22T21:49:14.076 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:49:14.076 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:49:14.076 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.076 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.076 DEBUG:boto:Method: GET 2023-09-22T21:49:14.076 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.076 DEBUG:boto:Data: 2023-09-22T21:49:14.076 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.076 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.076 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.077 DEBUG:boto:Params: {} 2023-09-22T21:49:14.077 DEBUG:boto:Token: None 2023-09-22T21:49:14.077 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.077 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cTHtgj4CHrJMu1j9hW6kDVnpeww= 2023-09-22T21:49:14.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cTHtgj4CHrJMu1j9hW6kDVnpeww=', 'Content-Length': '0'} 2023-09-22T21:49:14.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077ef807639f98905-00650e0bda-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.080 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:49:14.080 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:49:14.080 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.080 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.080 DEBUG:boto:Method: GET 2023-09-22T21:49:14.080 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.081 DEBUG:boto:Data: 2023-09-22T21:49:14.081 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.081 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.081 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.081 DEBUG:boto:Params: {} 2023-09-22T21:49:14.081 DEBUG:boto:Token: None 2023-09-22T21:49:14.081 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:14.081 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cTHtgj4CHrJMu1j9hW6kDVnpeww= 2023-09-22T21:49:14.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cTHtgj4CHrJMu1j9hW6kDVnpeww=', 'Content-Length': '0'} 2023-09-22T21:49:14.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c73994de0dfa691-00650e0bda-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.084 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:14.085 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:14.085 DEBUG:boto:path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:49:14.085 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:49:14.085 DEBUG:boto:Method: GET 2023-09-22T21:49:14.085 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj87 2023-09-22T21:49:14.085 DEBUG:boto:Data: 2023-09-22T21:49:14.085 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.085 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.085 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.085 DEBUG:boto:Params: {} 2023-09-22T21:49:14.086 DEBUG:boto:Token: None 2023-09-22T21:49:14.086 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/ 2023-09-22T21:49:14.086 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o= 2023-09-22T21:49:14.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o=', 'Content-Length': '0'} 2023-09-22T21:49:14.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc36c5590ed4f8af-00650e0bda-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.134 DEBUG:boto:b'ffzdqu-59obj871000falseobj872023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:14.137 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:14.137 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:14.137 DEBUG:boto:path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:49:14.137 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:49:14.137 DEBUG:boto:Method: GET 2023-09-22T21:49:14.138 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj87 2023-09-22T21:49:14.138 DEBUG:boto:Data: 2023-09-22T21:49:14.138 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.138 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.138 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.138 DEBUG:boto:Params: {} 2023-09-22T21:49:14.138 DEBUG:boto:Token: None 2023-09-22T21:49:14.139 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/ 2023-09-22T21:49:14.139 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o= 2023-09-22T21:49:14.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o=', 'Content-Length': '0'} 2023-09-22T21:49:14.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f95e292bc44d3ca4-00650e0bda-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.149 DEBUG:boto:b'ffzdqu-59obj871000falseobj872023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:14.149 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:49:14.149 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:49:14.150 DEBUG:boto:Method: HEAD 2023-09-22T21:49:14.150 DEBUG:boto:Path: /ffzdqu-59/obj87 2023-09-22T21:49:14.150 DEBUG:boto:Data: 2023-09-22T21:49:14.150 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.150 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.150 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.150 DEBUG:boto:Params: {} 2023-09-22T21:49:14.150 DEBUG:boto:Token: None 2023-09-22T21:49:14.151 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj87 2023-09-22T21:49:14.151 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QNCyEHfEnVMSiSm0W07TbQEVQW0= 2023-09-22T21:49:14.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QNCyEHfEnVMSiSm0W07TbQEVQW0=', 'Content-Length': '0'} 2023-09-22T21:49:14.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005de38ccb94d47ddf-00650e0bda-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.154 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:49:14.154 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:49:14.154 DEBUG:boto:Method: HEAD 2023-09-22T21:49:14.154 DEBUG:boto:Path: /ffzdqu-59/obj87 2023-09-22T21:49:14.154 DEBUG:boto:Data: 2023-09-22T21:49:14.154 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.154 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.154 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.155 DEBUG:boto:Params: {} 2023-09-22T21:49:14.155 DEBUG:boto:Token: None 2023-09-22T21:49:14.155 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj87 2023-09-22T21:49:14.155 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QNCyEHfEnVMSiSm0W07TbQEVQW0= 2023-09-22T21:49:14.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QNCyEHfEnVMSiSm0W07TbQEVQW0=', 'Content-Length': '0'} 2023-09-22T21:49:14.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d11a8448835ebd6e-00650e0bda-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-22T21:49:14.158 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:49:14.158 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:49:14.158 DEBUG:boto:Method: GET 2023-09-22T21:49:14.158 DEBUG:boto:Path: /ffzdqu-59/obj87 2023-09-22T21:49:14.158 DEBUG:boto:Data: 2023-09-22T21:49:14.159 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.159 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.159 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.159 DEBUG:boto:Params: {} 2023-09-22T21:49:14.159 DEBUG:boto:Token: None 2023-09-22T21:49:14.159 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj87 2023-09-22T21:49:14.159 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YYISVUnB3ypB6a0KtBjOUKwOVh0= 2023-09-22T21:49:14.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YYISVUnB3ypB6a0KtBjOUKwOVh0=', 'Content-Length': '0'} 2023-09-22T21:49:14.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c40c6c3206f83609-00650e0bda-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.162 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:49:14.162 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:49:14.163 DEBUG:boto:Method: GET 2023-09-22T21:49:14.163 DEBUG:boto:Path: /ffzdqu-59/obj87 2023-09-22T21:49:14.163 DEBUG:boto:Data: 2023-09-22T21:49:14.163 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.163 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.163 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.163 DEBUG:boto:Params: {} 2023-09-22T21:49:14.163 DEBUG:boto:Token: None 2023-09-22T21:49:14.163 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj87 2023-09-22T21:49:14.163 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YYISVUnB3ypB6a0KtBjOUKwOVh0= 2023-09-22T21:49:14.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YYISVUnB3ypB6a0KtBjOUKwOVh0=', 'Content-Length': '0'} 2023-09-22T21:49:14.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011ae588f53b75bc4-00650e0bda-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.210 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:14.211 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:14.211 DEBUG:boto:path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:49:14.211 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:49:14.211 DEBUG:boto:Method: GET 2023-09-22T21:49:14.211 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj87 2023-09-22T21:49:14.211 DEBUG:boto:Data: 2023-09-22T21:49:14.211 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.211 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.211 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.212 DEBUG:boto:Params: {} 2023-09-22T21:49:14.212 DEBUG:boto:Token: None 2023-09-22T21:49:14.212 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/ 2023-09-22T21:49:14.212 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o= 2023-09-22T21:49:14.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o=', 'Content-Length': '0'} 2023-09-22T21:49:14.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c931f8f4bada7309-00650e0bda-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.217 DEBUG:boto:b'ffzdqu-59obj871000falseobj872023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:14.218 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:14.218 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:14.218 DEBUG:boto:path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:49:14.218 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:49:14.218 DEBUG:boto:Method: GET 2023-09-22T21:49:14.218 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj87 2023-09-22T21:49:14.219 DEBUG:boto:Data: 2023-09-22T21:49:14.219 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.219 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.219 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.219 DEBUG:boto:Params: {} 2023-09-22T21:49:14.219 DEBUG:boto:Token: None 2023-09-22T21:49:14.219 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/ 2023-09-22T21:49:14.219 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o= 2023-09-22T21:49:14.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o=', 'Content-Length': '0'} 2023-09-22T21:49:14.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b7d507a326967c3-00650e0bda-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.229 DEBUG:boto:b'ffzdqu-59obj871000falseobj872023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:14.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-22T21:49:14.230 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:49:14.230 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:49:14.230 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.230 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.230 DEBUG:boto:Method: GET 2023-09-22T21:49:14.230 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.231 DEBUG:boto:Data: 2023-09-22T21:49:14.231 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.231 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.231 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.231 DEBUG:boto:Params: {} 2023-09-22T21:49:14.231 DEBUG:boto:Token: None 2023-09-22T21:49:14.231 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.231 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zIbe+uSJWIMY9dXLIDPb+82mWzY= 2023-09-22T21:49:14.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zIbe+uSJWIMY9dXLIDPb+82mWzY=', 'Content-Length': '0'} 2023-09-22T21:49:14.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbc2079e900023b0-00650e0bda-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.235 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:49:14.235 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:49:14.235 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.235 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.235 DEBUG:boto:Method: GET 2023-09-22T21:49:14.235 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.235 DEBUG:boto:Data: 2023-09-22T21:49:14.235 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.236 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.236 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.236 DEBUG:boto:Params: {} 2023-09-22T21:49:14.236 DEBUG:boto:Token: None 2023-09-22T21:49:14.236 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.236 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zIbe+uSJWIMY9dXLIDPb+82mWzY= 2023-09-22T21:49:14.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zIbe+uSJWIMY9dXLIDPb+82mWzY=', 'Content-Length': '0'} 2023-09-22T21:49:14.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000759c5f9e04233e8d-00650e0bda-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.239 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:49:14.239 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:49:14.239 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.239 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.240 DEBUG:boto:Method: HEAD 2023-09-22T21:49:14.240 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.240 DEBUG:boto:Data: 2023-09-22T21:49:14.240 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.240 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.240 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.240 DEBUG:boto:Params: {} 2023-09-22T21:49:14.240 DEBUG:boto:Token: None 2023-09-22T21:49:14.240 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.241 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1L02ngi1/Pqoz15nSBOPVpRQCtg= 2023-09-22T21:49:14.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1L02ngi1/Pqoz15nSBOPVpRQCtg=', 'Content-Length': '0'} 2023-09-22T21:49:14.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab970507e13d4982-00650e0bda-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.243 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:49:14.244 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:49:14.244 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.244 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.244 DEBUG:boto:Method: HEAD 2023-09-22T21:49:14.244 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.244 DEBUG:boto:Data: 2023-09-22T21:49:14.244 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.244 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.244 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.244 DEBUG:boto:Params: {} 2023-09-22T21:49:14.245 DEBUG:boto:Token: None 2023-09-22T21:49:14.245 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.245 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1L02ngi1/Pqoz15nSBOPVpRQCtg= 2023-09-22T21:49:14.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1L02ngi1/Pqoz15nSBOPVpRQCtg=', 'Content-Length': '0'} 2023-09-22T21:49:14.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098f82ac5b5a217af-00650e0bda-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-22T21:49:14.248 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:49:14.248 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:49:14.248 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.248 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.248 DEBUG:boto:Method: GET 2023-09-22T21:49:14.248 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.249 DEBUG:boto:Data: 2023-09-22T21:49:14.249 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.249 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.249 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.249 DEBUG:boto:Params: {} 2023-09-22T21:49:14.249 DEBUG:boto:Token: None 2023-09-22T21:49:14.249 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.249 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zIbe+uSJWIMY9dXLIDPb+82mWzY= 2023-09-22T21:49:14.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zIbe+uSJWIMY9dXLIDPb+82mWzY=', 'Content-Length': '0'} 2023-09-22T21:49:14.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001187bfc0885d42c6-00650e0bda-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.294 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:49:14.295 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:49:14.295 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.295 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.295 DEBUG:boto:Method: GET 2023-09-22T21:49:14.295 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.295 DEBUG:boto:Data: 2023-09-22T21:49:14.295 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.295 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.295 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.295 DEBUG:boto:Params: {} 2023-09-22T21:49:14.296 DEBUG:boto:Token: None 2023-09-22T21:49:14.296 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:14.296 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zIbe+uSJWIMY9dXLIDPb+82mWzY= 2023-09-22T21:49:14.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zIbe+uSJWIMY9dXLIDPb+82mWzY=', 'Content-Length': '0'} 2023-09-22T21:49:14.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085f906e4a8854083-00650e0bda-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.299 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:14.299 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:14.299 DEBUG:boto:path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:49:14.299 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:49:14.299 DEBUG:boto:Method: GET 2023-09-22T21:49:14.300 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj92 2023-09-22T21:49:14.300 DEBUG:boto:Data: 2023-09-22T21:49:14.300 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.300 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.300 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.300 DEBUG:boto:Params: {} 2023-09-22T21:49:14.300 DEBUG:boto:Token: None 2023-09-22T21:49:14.300 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/ 2023-09-22T21:49:14.300 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o= 2023-09-22T21:49:14.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o=', 'Content-Length': '0'} 2023-09-22T21:49:14.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f47336739e49d37-00650e0bda-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.305 DEBUG:boto:b'ffzdqu-59obj921000falseobj922023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:14.305 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:14.305 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:14.306 DEBUG:boto:path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:49:14.306 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:49:14.306 DEBUG:boto:Method: GET 2023-09-22T21:49:14.306 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj92 2023-09-22T21:49:14.306 DEBUG:boto:Data: 2023-09-22T21:49:14.306 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.306 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.306 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.306 DEBUG:boto:Params: {} 2023-09-22T21:49:14.306 DEBUG:boto:Token: None 2023-09-22T21:49:14.307 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/ 2023-09-22T21:49:14.307 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o= 2023-09-22T21:49:14.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o=', 'Content-Length': '0'} 2023-09-22T21:49:14.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abf4c7be76faf623-00650e0bda-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.316 DEBUG:boto:b'ffzdqu-59obj921000falseobj922023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:14.317 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:49:14.317 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:49:14.317 DEBUG:boto:Method: HEAD 2023-09-22T21:49:14.317 DEBUG:boto:Path: /ffzdqu-59/obj92 2023-09-22T21:49:14.317 DEBUG:boto:Data: 2023-09-22T21:49:14.317 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.317 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.317 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.317 DEBUG:boto:Params: {} 2023-09-22T21:49:14.318 DEBUG:boto:Token: None 2023-09-22T21:49:14.318 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj92 2023-09-22T21:49:14.318 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8cL87TdTFpiyZKzOpw2gii4iRpY= 2023-09-22T21:49:14.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8cL87TdTFpiyZKzOpw2gii4iRpY=', 'Content-Length': '0'} 2023-09-22T21:49:14.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbe553e76a3d2976-00650e0bda-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.321 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:49:14.321 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:49:14.321 DEBUG:boto:Method: HEAD 2023-09-22T21:49:14.321 DEBUG:boto:Path: /ffzdqu-59/obj92 2023-09-22T21:49:14.321 DEBUG:boto:Data: 2023-09-22T21:49:14.321 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.321 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.321 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.322 DEBUG:boto:Params: {} 2023-09-22T21:49:14.322 DEBUG:boto:Token: None 2023-09-22T21:49:14.322 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj92 2023-09-22T21:49:14.322 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8cL87TdTFpiyZKzOpw2gii4iRpY= 2023-09-22T21:49:14.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8cL87TdTFpiyZKzOpw2gii4iRpY=', 'Content-Length': '0'} 2023-09-22T21:49:14.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a54bc2538c599d60-00650e0bda-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-22T21:49:14.325 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:49:14.325 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:49:14.325 DEBUG:boto:Method: GET 2023-09-22T21:49:14.325 DEBUG:boto:Path: /ffzdqu-59/obj92 2023-09-22T21:49:14.325 DEBUG:boto:Data: 2023-09-22T21:49:14.326 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.326 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.326 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.326 DEBUG:boto:Params: {} 2023-09-22T21:49:14.326 DEBUG:boto:Token: None 2023-09-22T21:49:14.326 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj92 2023-09-22T21:49:14.326 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fx0Dc4ZfT1QXptN0/ZnpYrMXwHU= 2023-09-22T21:49:14.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fx0Dc4ZfT1QXptN0/ZnpYrMXwHU=', 'Content-Length': '0'} 2023-09-22T21:49:14.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0eeaa41a61f6db2-00650e0bda-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.329 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:49:14.330 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:49:14.330 DEBUG:boto:Method: GET 2023-09-22T21:49:14.330 DEBUG:boto:Path: /ffzdqu-59/obj92 2023-09-22T21:49:14.330 DEBUG:boto:Data: 2023-09-22T21:49:14.330 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.330 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.330 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.330 DEBUG:boto:Params: {} 2023-09-22T21:49:14.330 DEBUG:boto:Token: None 2023-09-22T21:49:14.331 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj92 2023-09-22T21:49:14.331 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fx0Dc4ZfT1QXptN0/ZnpYrMXwHU= 2023-09-22T21:49:14.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fx0Dc4ZfT1QXptN0/ZnpYrMXwHU=', 'Content-Length': '0'} 2023-09-22T21:49:14.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b4e6e202706e4f7-00650e0bda-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.334 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:14.334 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:14.334 DEBUG:boto:path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:49:14.334 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:49:14.334 DEBUG:boto:Method: GET 2023-09-22T21:49:14.335 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj92 2023-09-22T21:49:14.335 DEBUG:boto:Data: 2023-09-22T21:49:14.335 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.335 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.335 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.335 DEBUG:boto:Params: {} 2023-09-22T21:49:14.335 DEBUG:boto:Token: None 2023-09-22T21:49:14.335 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/ 2023-09-22T21:49:14.335 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o= 2023-09-22T21:49:14.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o=', 'Content-Length': '0'} 2023-09-22T21:49:14.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fac719f55fd9f4d5-00650e0bda-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.341 DEBUG:boto:b'ffzdqu-59obj921000falseobj922023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:14.341 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:14.341 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:14.341 DEBUG:boto:path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:49:14.341 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:49:14.341 DEBUG:boto:Method: GET 2023-09-22T21:49:14.341 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj92 2023-09-22T21:49:14.341 DEBUG:boto:Data: 2023-09-22T21:49:14.342 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.342 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.342 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.342 DEBUG:boto:Params: {} 2023-09-22T21:49:14.342 DEBUG:boto:Token: None 2023-09-22T21:49:14.342 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/ 2023-09-22T21:49:14.342 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o= 2023-09-22T21:49:14.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o=', 'Content-Length': '0'} 2023-09-22T21:49:14.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cacedd8cfb2e9246-00650e0bda-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.352 DEBUG:boto:b'ffzdqu-59obj921000falseobj922023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:14.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-22T21:49:14.352 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:49:14.353 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:49:14.353 DEBUG:boto:path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.353 DEBUG:boto:auth_path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.353 DEBUG:boto:Method: GET 2023-09-22T21:49:14.353 DEBUG:boto:Path: /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.353 DEBUG:boto:Data: 2023-09-22T21:49:14.353 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.353 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.353 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.353 DEBUG:boto:Params: {} 2023-09-22T21:49:14.354 DEBUG:boto:Token: None 2023-09-22T21:49:14.354 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.354 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y+C3MXBrHX/s38N9VAxuXbD7h04= 2023-09-22T21:49:14.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y+C3MXBrHX/s38N9VAxuXbD7h04=', 'Content-Length': '0'} 2023-09-22T21:49:14.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000755fa3bdde714f8f-00650e0bda-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.398 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:49:14.398 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:49:14.399 DEBUG:boto:path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.399 DEBUG:boto:auth_path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.399 DEBUG:boto:Method: GET 2023-09-22T21:49:14.399 DEBUG:boto:Path: /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.399 DEBUG:boto:Data: 2023-09-22T21:49:14.399 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.399 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.399 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.399 DEBUG:boto:Params: {} 2023-09-22T21:49:14.400 DEBUG:boto:Token: None 2023-09-22T21:49:14.400 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.400 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y+C3MXBrHX/s38N9VAxuXbD7h04= 2023-09-22T21:49:14.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y+C3MXBrHX/s38N9VAxuXbD7h04=', 'Content-Length': '0'} 2023-09-22T21:49:14.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0f22f4df439072d-00650e0bda-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.403 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:49:14.403 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:49:14.403 DEBUG:boto:path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.403 DEBUG:boto:auth_path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.403 DEBUG:boto:Method: HEAD 2023-09-22T21:49:14.404 DEBUG:boto:Path: /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.404 DEBUG:boto:Data: 2023-09-22T21:49:14.404 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.404 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.404 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.404 DEBUG:boto:Params: {} 2023-09-22T21:49:14.404 DEBUG:boto:Token: None 2023-09-22T21:49:14.404 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.404 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dvhpzuysOvfY9i40tQPi3rkIfvc= 2023-09-22T21:49:14.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dvhpzuysOvfY9i40tQPi3rkIfvc=', 'Content-Length': '0'} 2023-09-22T21:49:14.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bd84fd23d612f27-00650e0bda-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.408 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:49:14.408 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:49:14.408 DEBUG:boto:path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.408 DEBUG:boto:auth_path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.408 DEBUG:boto:Method: HEAD 2023-09-22T21:49:14.408 DEBUG:boto:Path: /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.408 DEBUG:boto:Data: 2023-09-22T21:49:14.409 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.409 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.409 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.409 DEBUG:boto:Params: {} 2023-09-22T21:49:14.409 DEBUG:boto:Token: None 2023-09-22T21:49:14.409 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.409 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dvhpzuysOvfY9i40tQPi3rkIfvc= 2023-09-22T21:49:14.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dvhpzuysOvfY9i40tQPi3rkIfvc=', 'Content-Length': '0'} 2023-09-22T21:49:14.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7cfa16961a17552-00650e0bda-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-22T21:49:14.412 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:49:14.412 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:49:14.412 DEBUG:boto:path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.413 DEBUG:boto:auth_path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.413 DEBUG:boto:Method: GET 2023-09-22T21:49:14.413 DEBUG:boto:Path: /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.413 DEBUG:boto:Data: 2023-09-22T21:49:14.413 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.413 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.413 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.413 DEBUG:boto:Params: {} 2023-09-22T21:49:14.413 DEBUG:boto:Token: None 2023-09-22T21:49:14.414 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.414 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y+C3MXBrHX/s38N9VAxuXbD7h04= 2023-09-22T21:49:14.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y+C3MXBrHX/s38N9VAxuXbD7h04=', 'Content-Length': '0'} 2023-09-22T21:49:14.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020a7ffdb687dab18-00650e0bda-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.462 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:49:14.463 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:49:14.463 DEBUG:boto:path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.463 DEBUG:boto:auth_path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.463 DEBUG:boto:Method: GET 2023-09-22T21:49:14.463 DEBUG:boto:Path: /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.463 DEBUG:boto:Data: 2023-09-22T21:49:14.463 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.463 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.463 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.464 DEBUG:boto:Params: {} 2023-09-22T21:49:14.464 DEBUG:boto:Token: None 2023-09-22T21:49:14.464 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:14.464 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y+C3MXBrHX/s38N9VAxuXbD7h04= 2023-09-22T21:49:14.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y+C3MXBrHX/s38N9VAxuXbD7h04=', 'Content-Length': '0'} 2023-09-22T21:49:14.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063ca81eef9088ab0-00650e0bda-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.467 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:14.468 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:14.468 DEBUG:boto:path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:49:14.468 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:49:14.468 DEBUG:boto:Method: GET 2023-09-22T21:49:14.468 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj93 2023-09-22T21:49:14.468 DEBUG:boto:Data: 2023-09-22T21:49:14.468 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.468 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.468 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.468 DEBUG:boto:Params: {} 2023-09-22T21:49:14.469 DEBUG:boto:Token: None 2023-09-22T21:49:14.469 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/ 2023-09-22T21:49:14.469 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o= 2023-09-22T21:49:14.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o=', 'Content-Length': '0'} 2023-09-22T21:49:14.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f27c66a5c083be4-00650e0bda-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.518 DEBUG:boto:b'ffzdqu-59obj931000falseobj932023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:14.519 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:14.519 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:14.519 DEBUG:boto:path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:49:14.519 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:49:14.519 DEBUG:boto:Method: GET 2023-09-22T21:49:14.519 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj93 2023-09-22T21:49:14.520 DEBUG:boto:Data: 2023-09-22T21:49:14.520 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.520 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.520 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.520 DEBUG:boto:Params: {} 2023-09-22T21:49:14.520 DEBUG:boto:Token: None 2023-09-22T21:49:14.520 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/ 2023-09-22T21:49:14.520 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o= 2023-09-22T21:49:14.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o=', 'Content-Length': '0'} 2023-09-22T21:49:14.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3e89ea20a6d868e-00650e0bda-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.530 DEBUG:boto:b'ffzdqu-59obj931000falseobj932023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:14.530 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:49:14.531 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:49:14.531 DEBUG:boto:Method: HEAD 2023-09-22T21:49:14.531 DEBUG:boto:Path: /ffzdqu-59/obj93 2023-09-22T21:49:14.531 DEBUG:boto:Data: 2023-09-22T21:49:14.531 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.531 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.531 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.531 DEBUG:boto:Params: {} 2023-09-22T21:49:14.531 DEBUG:boto:Token: None 2023-09-22T21:49:14.532 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj93 2023-09-22T21:49:14.532 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dvvpjJPB7Uk2Wg7DYrYZkKcobYs= 2023-09-22T21:49:14.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dvvpjJPB7Uk2Wg7DYrYZkKcobYs=', 'Content-Length': '0'} 2023-09-22T21:49:14.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000898df875e9308cbf-00650e0bda-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.534 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:49:14.535 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:49:14.535 DEBUG:boto:Method: HEAD 2023-09-22T21:49:14.535 DEBUG:boto:Path: /ffzdqu-59/obj93 2023-09-22T21:49:14.535 DEBUG:boto:Data: 2023-09-22T21:49:14.535 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.535 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.535 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.535 DEBUG:boto:Params: {} 2023-09-22T21:49:14.535 DEBUG:boto:Token: None 2023-09-22T21:49:14.536 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj93 2023-09-22T21:49:14.536 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dvvpjJPB7Uk2Wg7DYrYZkKcobYs= 2023-09-22T21:49:14.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dvvpjJPB7Uk2Wg7DYrYZkKcobYs=', 'Content-Length': '0'} 2023-09-22T21:49:14.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf59d53f4819f452-00650e0bda-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-22T21:49:14.539 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:49:14.539 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:49:14.539 DEBUG:boto:Method: GET 2023-09-22T21:49:14.539 DEBUG:boto:Path: /ffzdqu-59/obj93 2023-09-22T21:49:14.539 DEBUG:boto:Data: 2023-09-22T21:49:14.539 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.539 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.540 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.540 DEBUG:boto:Params: {} 2023-09-22T21:49:14.540 DEBUG:boto:Token: None 2023-09-22T21:49:14.540 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj93 2023-09-22T21:49:14.540 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JklQlB9VRy9VAMJvDQY7RPRi+GA= 2023-09-22T21:49:14.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JklQlB9VRy9VAMJvDQY7RPRi+GA=', 'Content-Length': '0'} 2023-09-22T21:49:14.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007171e31e9225feec-00650e0bda-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.586 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:49:14.587 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:49:14.587 DEBUG:boto:Method: GET 2023-09-22T21:49:14.587 DEBUG:boto:Path: /ffzdqu-59/obj93 2023-09-22T21:49:14.587 DEBUG:boto:Data: 2023-09-22T21:49:14.587 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.587 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.587 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.587 DEBUG:boto:Params: {} 2023-09-22T21:49:14.588 DEBUG:boto:Token: None 2023-09-22T21:49:14.588 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj93 2023-09-22T21:49:14.588 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JklQlB9VRy9VAMJvDQY7RPRi+GA= 2023-09-22T21:49:14.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JklQlB9VRy9VAMJvDQY7RPRi+GA=', 'Content-Length': '0'} 2023-09-22T21:49:14.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024cf6ae5c7a8ffe4-00650e0bda-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.591 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:14.591 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:14.591 DEBUG:boto:path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:49:14.591 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:49:14.591 DEBUG:boto:Method: GET 2023-09-22T21:49:14.592 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj93 2023-09-22T21:49:14.592 DEBUG:boto:Data: 2023-09-22T21:49:14.592 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.592 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.592 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.592 DEBUG:boto:Params: {} 2023-09-22T21:49:14.592 DEBUG:boto:Token: None 2023-09-22T21:49:14.592 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/ 2023-09-22T21:49:14.592 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o= 2023-09-22T21:49:14.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o=', 'Content-Length': '0'} 2023-09-22T21:49:14.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e0685c4c4584375-00650e0bda-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.638 DEBUG:boto:b'ffzdqu-59obj931000falseobj932023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:14.639 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:14.639 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:14.639 DEBUG:boto:path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:49:14.639 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:49:14.639 DEBUG:boto:Method: GET 2023-09-22T21:49:14.639 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj93 2023-09-22T21:49:14.639 DEBUG:boto:Data: 2023-09-22T21:49:14.639 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.640 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.640 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.640 DEBUG:boto:Params: {} 2023-09-22T21:49:14.640 DEBUG:boto:Token: None 2023-09-22T21:49:14.640 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/ 2023-09-22T21:49:14.640 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o= 2023-09-22T21:49:14.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o=', 'Content-Length': '0'} 2023-09-22T21:49:14.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb26c3fecf83a667-00650e0bda-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.651 DEBUG:boto:b'ffzdqu-59obj931000falseobj932023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:14.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T21:49:14.651 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:49:14.651 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:49:14.651 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.651 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.652 DEBUG:boto:Method: GET 2023-09-22T21:49:14.652 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.652 DEBUG:boto:Data: 2023-09-22T21:49:14.652 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.652 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.652 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.652 DEBUG:boto:Params: {} 2023-09-22T21:49:14.652 DEBUG:boto:Token: None 2023-09-22T21:49:14.652 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.652 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6ta7jT5hu0dkDMusZg9nYOKicCc= 2023-09-22T21:49:14.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6ta7jT5hu0dkDMusZg9nYOKicCc=', 'Content-Length': '0'} 2023-09-22T21:49:14.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9d93fbee46efca0-00650e0bda-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.656 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:49:14.656 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:49:14.656 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.656 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.656 DEBUG:boto:Method: GET 2023-09-22T21:49:14.656 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.656 DEBUG:boto:Data: 2023-09-22T21:49:14.656 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.657 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.657 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.657 DEBUG:boto:Params: {} 2023-09-22T21:49:14.657 DEBUG:boto:Token: None 2023-09-22T21:49:14.657 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.657 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6ta7jT5hu0dkDMusZg9nYOKicCc= 2023-09-22T21:49:14.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6ta7jT5hu0dkDMusZg9nYOKicCc=', 'Content-Length': '0'} 2023-09-22T21:49:14.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf7b7d56aad8e3d0-00650e0bda-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.660 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:49:14.660 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:49:14.661 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.661 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.661 DEBUG:boto:Method: HEAD 2023-09-22T21:49:14.661 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.661 DEBUG:boto:Data: 2023-09-22T21:49:14.661 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.661 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.661 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.661 DEBUG:boto:Params: {} 2023-09-22T21:49:14.661 DEBUG:boto:Token: None 2023-09-22T21:49:14.662 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.662 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S/GG9yfCOOkrCwgHD0oEjoK9Ifw= 2023-09-22T21:49:14.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S/GG9yfCOOkrCwgHD0oEjoK9Ifw=', 'Content-Length': '0'} 2023-09-22T21:49:14.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032616365e604a475-00650e0bda-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.665 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:49:14.665 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:49:14.665 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.665 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.665 DEBUG:boto:Method: HEAD 2023-09-22T21:49:14.665 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.665 DEBUG:boto:Data: 2023-09-22T21:49:14.665 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.665 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.666 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.666 DEBUG:boto:Params: {} 2023-09-22T21:49:14.666 DEBUG:boto:Token: None 2023-09-22T21:49:14.666 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.666 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S/GG9yfCOOkrCwgHD0oEjoK9Ifw= 2023-09-22T21:49:14.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S/GG9yfCOOkrCwgHD0oEjoK9Ifw=', 'Content-Length': '0'} 2023-09-22T21:49:14.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e54dbbee8fa71bc8-00650e0bda-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T21:49:14.669 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:49:14.669 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:49:14.669 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.669 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.669 DEBUG:boto:Method: GET 2023-09-22T21:49:14.670 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.670 DEBUG:boto:Data: 2023-09-22T21:49:14.670 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.670 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.670 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.670 DEBUG:boto:Params: {} 2023-09-22T21:49:14.670 DEBUG:boto:Token: None 2023-09-22T21:49:14.670 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.670 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6ta7jT5hu0dkDMusZg9nYOKicCc= 2023-09-22T21:49:14.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6ta7jT5hu0dkDMusZg9nYOKicCc=', 'Content-Length': '0'} 2023-09-22T21:49:14.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d579d7770e85cd9-00650e0bda-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.714 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:49:14.714 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:49:14.715 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.715 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.715 DEBUG:boto:Method: GET 2023-09-22T21:49:14.715 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.715 DEBUG:boto:Data: 2023-09-22T21:49:14.715 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.715 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.715 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.715 DEBUG:boto:Params: {} 2023-09-22T21:49:14.716 DEBUG:boto:Token: None 2023-09-22T21:49:14.716 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:14.716 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6ta7jT5hu0dkDMusZg9nYOKicCc= 2023-09-22T21:49:14.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6ta7jT5hu0dkDMusZg9nYOKicCc=', 'Content-Length': '0'} 2023-09-22T21:49:14.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ce547522dc841d3-00650e0bda-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.719 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:14.719 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:14.720 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:49:14.720 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:49:14.720 DEBUG:boto:Method: GET 2023-09-22T21:49:14.720 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:49:14.720 DEBUG:boto:Data: 2023-09-22T21:49:14.720 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.720 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.720 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.720 DEBUG:boto:Params: {} 2023-09-22T21:49:14.720 DEBUG:boto:Token: None 2023-09-22T21:49:14.721 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/ 2023-09-22T21:49:14.721 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o= 2023-09-22T21:49:14.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o=', 'Content-Length': '0'} 2023-09-22T21:49:14.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006321de247c7891fa-00650e0bda-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.766 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:14.767 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:14.767 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:14.767 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:49:14.767 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:49:14.767 DEBUG:boto:Method: GET 2023-09-22T21:49:14.767 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:49:14.767 DEBUG:boto:Data: 2023-09-22T21:49:14.768 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.768 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.768 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.768 DEBUG:boto:Params: {} 2023-09-22T21:49:14.768 DEBUG:boto:Token: None 2023-09-22T21:49:14.768 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/ 2023-09-22T21:49:14.768 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o= 2023-09-22T21:49:14.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o=', 'Content-Length': '0'} 2023-09-22T21:49:14.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4839f42e88dc6a8-00650e0bda-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.778 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:14.778 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:49:14.778 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:49:14.778 DEBUG:boto:Method: HEAD 2023-09-22T21:49:14.779 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:49:14.779 DEBUG:boto:Data: 2023-09-22T21:49:14.779 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.779 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.779 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.779 DEBUG:boto:Params: {} 2023-09-22T21:49:14.779 DEBUG:boto:Token: None 2023-09-22T21:49:14.779 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj95 2023-09-22T21:49:14.779 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rtQePmppUewx/mW7tW73XX++b0I= 2023-09-22T21:49:14.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rtQePmppUewx/mW7tW73XX++b0I=', 'Content-Length': '0'} 2023-09-22T21:49:14.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e54e699968b78e91-00650e0bda-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.782 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:49:14.783 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:49:14.783 DEBUG:boto:Method: HEAD 2023-09-22T21:49:14.783 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:49:14.783 DEBUG:boto:Data: 2023-09-22T21:49:14.783 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.783 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.783 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.783 DEBUG:boto:Params: {} 2023-09-22T21:49:14.783 DEBUG:boto:Token: None 2023-09-22T21:49:14.784 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj95 2023-09-22T21:49:14.784 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rtQePmppUewx/mW7tW73XX++b0I= 2023-09-22T21:49:14.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rtQePmppUewx/mW7tW73XX++b0I=', 'Content-Length': '0'} 2023-09-22T21:49:14.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3e5ab49a70d792c-00650e0bda-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T21:49:14.787 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:49:14.787 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:49:14.787 DEBUG:boto:Method: GET 2023-09-22T21:49:14.787 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:49:14.787 DEBUG:boto:Data: 2023-09-22T21:49:14.788 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.788 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.788 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.788 DEBUG:boto:Params: {} 2023-09-22T21:49:14.788 DEBUG:boto:Token: None 2023-09-22T21:49:14.788 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj95 2023-09-22T21:49:14.788 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h51DLCOanWehGPEc+eljEPcNCTc= 2023-09-22T21:49:14.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h51DLCOanWehGPEc+eljEPcNCTc=', 'Content-Length': '0'} 2023-09-22T21:49:14.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc5942ccdeffa8c4-00650e0bda-1b9c-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.834 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:49:14.834 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:49:14.835 DEBUG:boto:Method: GET 2023-09-22T21:49:14.835 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:49:14.835 DEBUG:boto:Data: 2023-09-22T21:49:14.835 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.835 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.835 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.835 DEBUG:boto:Params: {} 2023-09-22T21:49:14.835 DEBUG:boto:Token: None 2023-09-22T21:49:14.835 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/obj95 2023-09-22T21:49:14.835 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h51DLCOanWehGPEc+eljEPcNCTc= 2023-09-22T21:49:14.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h51DLCOanWehGPEc+eljEPcNCTc=', 'Content-Length': '0'} 2023-09-22T21:49:14.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed4daab7d6d346bc-00650e0bda-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.841 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:14.841 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:14.841 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:49:14.841 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:49:14.841 DEBUG:boto:Method: GET 2023-09-22T21:49:14.841 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:49:14.842 DEBUG:boto:Data: 2023-09-22T21:49:14.842 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.842 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:14.842 DEBUG:boto:Port: 8001 2023-09-22T21:49:14.842 DEBUG:boto:Params: {} 2023-09-22T21:49:14.842 DEBUG:boto:Token: None 2023-09-22T21:49:14.842 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/ 2023-09-22T21:49:14.842 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o= 2023-09-22T21:49:14.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o=', 'Content-Length': '0'} 2023-09-22T21:49:14.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afde74792bea578f-00650e0bda-1b9c-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.894 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:14.895 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:14.895 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:14.895 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:49:14.895 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:49:14.895 DEBUG:boto:Method: GET 2023-09-22T21:49:14.895 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:49:14.895 DEBUG:boto:Data: 2023-09-22T21:49:14.895 DEBUG:boto:Headers: {} 2023-09-22T21:49:14.896 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:14.896 DEBUG:boto:Port: 8000 2023-09-22T21:49:14.896 DEBUG:boto:Params: {} 2023-09-22T21:49:14.896 DEBUG:boto:Token: None 2023-09-22T21:49:14.896 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:14 GMT /ffzdqu-59/ 2023-09-22T21:49:14.896 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o= 2023-09-22T21:49:14.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5t228Y4hheXWnT2rukjN4wHwS7o=', 'Content-Length': '0'} 2023-09-22T21:49:14.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091115375ccb9e274-00650e0bda-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:14.907 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:14.907 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-59 zones={a2, a1} 2023-09-22T21:49:14.907 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-09-22T21:49:14.908 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T21:49:14.909 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:49:14.908+0000 7f095023b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 71437) UID: 0 2023-09-22T21:49:14.909 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:49:14.908+0000 7f0953a82a40 -1 shutting down 2023-09-22T21:49:17.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-09-22T21:49:21.007 INFO:tasks.rgw.c1.client.1:Stopped 2023-09-22T21:49:21.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:49:21.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:49:21.008 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:49:21.055 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:21.055 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:21.115 INFO:teuthology.orchestra.run.smithi160.stdout:[{"name":"obj11","instance":"","ver":{"pool":13,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:36:31.700091Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_Pkz8j96zSE214e7cZA5WLMpUxa_4yWJ","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj111","instance":"","ver":{"pool":13,"epoch":13},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:47:16.468300Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_MKlnWaktc97omVVBwPkvYdLh8k7ZSnM","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj115","instance":"","ver":{"pool":13,"epoch":2},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:40:21.539635Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_9bQmOcTt9ax2e0yH5t_AlUfJxiIA7py","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj116","instance":"","ver":{"pool":13,"epoch":2},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:43:12.174400Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_vtPhEIXBMvCC2ryrgfLX3zra8L7tc_1","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj124","instance":"","ver":{"pool":13,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:45:13.728341Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_KtIQ0jQcZCxhpumoBdkLKryN5Eaw_Vq","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj145","instance":"","ver":{"pool":13,"epoch":2},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:43:12.226320Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_TYMFD6C0uKehALnNKCNJXp-YUu8-2JN","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj148","instance":"","ver":{"pool":13,"epoch":8},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:47:16.487798Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_px1IAhIk7JnaklGHKXlOLqK6D5ktf2s","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj155","instance":"","ver":{"pool":13,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:45:13.783971Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_HbJksjNqHPygBW913J35gWyKonFvJfL","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj17","instance":"","ver":{"pool":13,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:36:48.678004Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_XxFwtJlPJ3h9Yft4fBNsWMsg2LDG-6o","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj185","instance":"","ver":{"pool":13,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:47:16.544730Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_3AX6e2tOyypdvEtjlfqq0jK7rlZq7xN","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj19","instance":"","ver":{"pool":13,"epoch":12},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:38:31.615370Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"__YHqDjPQB2CSwLehDIjOposy-TZVwF8","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj22","instance":"","ver":{"pool":13,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:36:31.712430Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_JJTlIaQAcOvPJjwA5y9tRKZNSkFi5bD","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj23","instance":"","ver":{"pool":13,"epoch":8},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:40:21.400499Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_HIH1GnIMzugCPIxVs_z-KA4MYrgMfnO","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj29","instance":"","ver":{"pool":13,"epoch":2},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:43:12.137994Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_QxZG2cmiacsuMjvmHAHIKul5ds0EFby","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj31","instance":"","ver":{"pool":13,"epoch":2},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:45:13.646846Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_6UPEbwGXME7Tyc0sZCsX7G2PsvqJvs4","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj33","instance":"","ver":{"pool":13,"epoch":1},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:36:31.769832Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_bpsgMnioNgdbQ11vVJ8mhzGNgxa8gIZ","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj34","instance":"","ver":{"pool":13,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:36:48.689308Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_B5VbiY9oVsFon_f-beefDOQpiYw0H24","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj37","instance":"","ver":{"pool":13,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:47:16.399903Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_aPn5sqPezN_Vlu_NJPqE6gPnQ-hDo7j","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj38","instance":"","ver":{"pool":13,"epoch":16},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:38:31.671148Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_5Y5ycxS0g9LZHu-QbZxhATQFw5wsWPs","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj44","instance":"","ver":{"pool":13,"epoch":9},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:36:31.787636Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_xmcusXaUSlNoiME4lfm911SRTLL-P8s","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj46","instance":"","ver":{"pool":13,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:40:21.413515Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_AJCnbroo0_azsOjmKKeD-t33kDExChs","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj51","instance":"","ver":{"pool":13,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:36:48.744226Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_mLByrMbyteHqjjDYkuA1B2ZjahEdKB-","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj55","instance":"","ver":{"pool":13,"epoch":2},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:36:31.838216Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_dHc8Y0YapiK57MugdT5YK1ftB6IbDST","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj57","instance":"","ver":{"pool":13,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:38:31.686153Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_8ax8KU9Pq762WdOtucUujXXooMo63lQ","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj58","instance":"","ver":{"pool":13,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:43:12.150080Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_QfdQ-QmGtMXXpMqYjxQiTJc4ycsHpti","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj62","instance":"","ver":{"pool":13,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:45:13.660030Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_QlVyV0ebPkbKPzuSNSJ9_IBWgzzCLgj","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj68","instance":"","ver":{"pool":13,"epoch":1},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:36:48.760922Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_7WBWP-MWY2fCbMQltMtyCJRHCJCImyp","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj69","instance":"","ver":{"pool":13,"epoch":2},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:40:21.465384Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_5rFwJBJedarG4iUDlCIrDJSn0EqAWdK","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj74","instance":"","ver":{"pool":13,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:47:16.413955Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_qw1-ckq3064kiN4jcyLw4t2WUeBvX85","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj76","instance":"","ver":{"pool":13,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:38:31.739068Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_c7tuVn5GUWUaByTC4ycl50boBOoXI89","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj85","instance":"","ver":{"pool":13,"epoch":2},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:36:48.813905Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_hyi6gz0k_Y-f6el6Q-g6iOfYQh7B7AP","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj87","instance":"","ver":{"pool":13,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:43:12.162489Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_KW5MfNu6_SHrDuG6TYkj67RrIW3Acgz","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj92","instance":"","ver":{"pool":13,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:40:21.484079Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_fGh7VbrHWhvSm4N5_FgEnfpsYIUONq9","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj93","instance":"","ver":{"pool":13,"epoch":10},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:45:13.713251Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_vZ4CXGHsFTbPrKz0fHGvWunmJlh94Qi","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj95","instance":"","ver":{"pool":13,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:38:31.759207Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_uDnLS8H-m5iPyufnQzcUJgMXWQLiznf","flags":0,"pending_map":[],"versioned_epoch":0}] 2023-09-22T21:49:21.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:21.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:21.116 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj11 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:21.204 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:21.204 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:21.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:21.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:21.311 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj111 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:21.393 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:21.393 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:21.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:21.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:21.503 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj115 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:21.580 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:21.580 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:21.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:21.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:21.692 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj116 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:21.771 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:21.771 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:21.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:21.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:21.881 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj124 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:21.959 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:21.959 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:22.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:22.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:22.057 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj145 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:22.137 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:22.137 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:22.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:22.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:22.234 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj148 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:22.318 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:22.318 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:22.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:22.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:22.412 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj155 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:22.492 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:22.492 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:22.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:22.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:22.585 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj17 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:22.661 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:22.662 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:22.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:22.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:22.756 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj185 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:22.835 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:22.835 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:22.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:22.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:22.932 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj19 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:23.012 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:23.013 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:23.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:23.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:23.107 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj22 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:23.186 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:23.187 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:23.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:23.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:23.279 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj23 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:23.359 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:23.359 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:23.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:23.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:23.452 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj29 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:23.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-09-22T21:49:23.527 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:23.527 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:23.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:23.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:23.687 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj31 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:23.729 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:23.741 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:23.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:23.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:23.824 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj33 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:23.902 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:23.903 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:23.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:23.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:23.998 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj34 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:24.079 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:24.079 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:24.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:24.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:24.174 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj37 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:24.253 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:24.254 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:24.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:24.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:24.346 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj38 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:24.427 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:24.427 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:24.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:24.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:24.523 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj44 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:24.601 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:24.602 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:24.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:24.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:24.695 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj46 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:24.773 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:24.773 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:24.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:24.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:24.869 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj51 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:24.948 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:24.948 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:25.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:25.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:25.051 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:25.124 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:25.125 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:25.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:25.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:25.223 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj57 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:25.301 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:25.302 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:25.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:25.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:25.402 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj58 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:25.476 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:25.476 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:25.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:25.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:25.571 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj62 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:25.651 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:25.652 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:25.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:25.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:25.744 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:25.828 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:25.828 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:25.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:25.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:25.926 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj69 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:25.999 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:25.999 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:26.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:26.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:26.091 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:26.171 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:26.171 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:26.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:26.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:26.270 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj76 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:26.350 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:26.350 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:26.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:26.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:26.445 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj85 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:26.519 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:26.520 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:26.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:26.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:26.619 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj87 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:26.695 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:26.695 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:26.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:26.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:26.790 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj92 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:26.873 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:26.873 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:26.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:26.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:26.969 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj93 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:27.045 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:27.045 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:27.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:27.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:27.138 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --object obj95 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:27.212 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:27.213 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:27.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:27.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:27.306 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog trim --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:27.377 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:27.377 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:27.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:27.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:27.433 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:27.514 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:27.514 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:27.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:27.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:49:27.564 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:49:27.639 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:27.640 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:27.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.712+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: sync status for source bucket: init. lease is: not taken. stop indications is: 0 2023-09-22T21:49:27.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.712+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: waiting on bucket lease 2023-09-22T21:49:27.719 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.716+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: status after acquiring the lock is: init 2023-09-22T21:49:27.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.732+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: listing bucket for full sync 2023-09-22T21:49:27.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj11 2023-09-22T21:49:27.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj11]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj11 op=0 op_state=1 2023-09-22T21:49:27.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj11]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj11[0] 2023-09-22T21:49:27.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj111 2023-09-22T21:49:27.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj111]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj111 op=0 op_state=1 2023-09-22T21:49:27.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj111]: : injecting data sync error on key=obj111 2023-09-22T21:49:27.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:27.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj111]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj111 2023-09-22T21:49:27.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj115 2023-09-22T21:49:27.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj115]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj115 op=0 op_state=1 2023-09-22T21:49:27.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj115]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj115[0] 2023-09-22T21:49:27.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj116 2023-09-22T21:49:27.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj116]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj116 op=0 op_state=1 2023-09-22T21:49:27.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj116]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj116[0] 2023-09-22T21:49:27.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj124 2023-09-22T21:49:27.785 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj124]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj124 op=0 op_state=1 2023-09-22T21:49:27.785 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj124]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj124[0] 2023-09-22T21:49:27.785 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj145 2023-09-22T21:49:27.785 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj145]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj145 op=0 op_state=1 2023-09-22T21:49:27.785 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj145]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj145[0] 2023-09-22T21:49:27.785 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj148 2023-09-22T21:49:27.786 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj148]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj148 op=0 op_state=1 2023-09-22T21:49:27.786 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj148]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj148[0] 2023-09-22T21:49:27.786 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj155 2023-09-22T21:49:27.786 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj155]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj155 op=0 op_state=1 2023-09-22T21:49:27.786 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj155]: : injecting data sync error on key=obj155 2023-09-22T21:49:27.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:27.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj155]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj155 2023-09-22T21:49:27.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj17 2023-09-22T21:49:27.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj17]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj17 op=0 op_state=1 2023-09-22T21:49:27.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj17]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj17[0] 2023-09-22T21:49:27.788 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj185 2023-09-22T21:49:27.788 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj185]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj185 op=0 op_state=1 2023-09-22T21:49:27.788 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj185]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj185[0] 2023-09-22T21:49:27.788 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj19 2023-09-22T21:49:27.788 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj19]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj19 op=0 op_state=1 2023-09-22T21:49:27.789 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj19]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj19[0] 2023-09-22T21:49:27.789 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj22 2023-09-22T21:49:27.789 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj22]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj22 op=0 op_state=1 2023-09-22T21:49:27.789 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj22]: : injecting data sync error on key=obj22 2023-09-22T21:49:27.789 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj22]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:27.790 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj22]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj22 2023-09-22T21:49:27.790 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj23 2023-09-22T21:49:27.790 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj23]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj23 op=0 op_state=1 2023-09-22T21:49:27.790 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj23]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj23[0] 2023-09-22T21:49:27.790 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj29 2023-09-22T21:49:27.790 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj29]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj29 op=0 op_state=1 2023-09-22T21:49:27.791 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj29]: : injecting data sync error on key=obj29 2023-09-22T21:49:27.791 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:27.791 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj29]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj29 2023-09-22T21:49:27.791 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj31 2023-09-22T21:49:27.791 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj31]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj31 op=0 op_state=1 2023-09-22T21:49:27.791 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj31]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj31[0] 2023-09-22T21:49:27.791 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj33 2023-09-22T21:49:27.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj33]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj33 op=0 op_state=1 2023-09-22T21:49:27.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj33]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj33[0] 2023-09-22T21:49:27.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj34 2023-09-22T21:49:27.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj34]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj34 op=0 op_state=1 2023-09-22T21:49:27.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj34]: : injecting data sync error on key=obj34 2023-09-22T21:49:27.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj34]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:27.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj34]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj34 2023-09-22T21:49:27.793 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj37 2023-09-22T21:49:27.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj37]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj37 op=0 op_state=1 2023-09-22T21:49:27.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj37]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj37[0] 2023-09-22T21:49:27.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj38 2023-09-22T21:49:27.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj38]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj38 op=0 op_state=1 2023-09-22T21:49:27.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj38]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj38[0] 2023-09-22T21:49:27.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj44 2023-09-22T21:49:27.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj44]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj44 op=0 op_state=1 2023-09-22T21:49:27.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj44]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj44[0] 2023-09-22T21:49:27.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj46 2023-09-22T21:49:27.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj46]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj46 op=0 op_state=1 2023-09-22T21:49:27.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.780+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj46]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj46[0] 2023-09-22T21:49:27.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.784+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:27.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.784+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj51 2023-09-22T21:49:27.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.784+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj51]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj51 op=0 op_state=1 2023-09-22T21:49:27.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.784+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj51]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj51[0] 2023-09-22T21:49:27.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.788+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:27.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.788+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj55 2023-09-22T21:49:27.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.788+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj55]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj55 op=0 op_state=1 2023-09-22T21:49:27.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.788+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj55]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj55[0] 2023-09-22T21:49:27.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.788+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:27.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.788+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj57 2023-09-22T21:49:27.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.788+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj57]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj57 op=0 op_state=1 2023-09-22T21:49:27.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.788+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj57]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj57[0] 2023-09-22T21:49:27.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.788+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:27.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.788+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj58 2023-09-22T21:49:27.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.788+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj58]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj58 op=0 op_state=1 2023-09-22T21:49:27.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.788+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj58]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj58[0] 2023-09-22T21:49:27.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.788+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:27.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.788+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj62 2023-09-22T21:49:27.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.788+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj62]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj62 op=0 op_state=1 2023-09-22T21:49:27.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.788+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj62]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj62[0] 2023-09-22T21:49:27.817 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj124]: success 2023-09-22T21:49:27.817 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj68 2023-09-22T21:49:27.817 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj68]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj68 op=0 op_state=1 2023-09-22T21:49:27.817 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj68]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj68[0] 2023-09-22T21:49:27.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj33]: success 2023-09-22T21:49:27.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj38]: success 2023-09-22T21:49:27.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj19]: success 2023-09-22T21:49:27.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj37]: success 2023-09-22T21:49:27.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj116]: success 2023-09-22T21:49:27.820 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj17]: success 2023-09-22T21:49:27.820 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj69 2023-09-22T21:49:27.820 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj69]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj69 op=0 op_state=1 2023-09-22T21:49:27.820 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj69]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj69[0] 2023-09-22T21:49:27.820 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj74 2023-09-22T21:49:27.820 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj74]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj74 op=0 op_state=1 2023-09-22T21:49:27.820 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj115]: success 2023-09-22T21:49:27.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj74]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj74[0] 2023-09-22T21:49:27.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj76 2023-09-22T21:49:27.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj76]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj76 op=0 op_state=1 2023-09-22T21:49:27.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj76]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj76[0] 2023-09-22T21:49:27.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj51]: success 2023-09-22T21:49:27.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj85 2023-09-22T21:49:27.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj85]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj85 op=0 op_state=1 2023-09-22T21:49:27.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj85]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj85[0] 2023-09-22T21:49:27.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj87 2023-09-22T21:49:27.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj87]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj87 op=0 op_state=1 2023-09-22T21:49:27.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj87]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj87[0] 2023-09-22T21:49:27.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj92 2023-09-22T21:49:27.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj92]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj92 op=0 op_state=1 2023-09-22T21:49:27.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj92]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj92[0] 2023-09-22T21:49:27.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj93 2023-09-22T21:49:27.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj93]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj93 op=0 op_state=1 2023-09-22T21:49:27.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj93]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj93[0] 2023-09-22T21:49:27.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj95 2023-09-22T21:49:27.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj95]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj95 op=0 op_state=1 2023-09-22T21:49:27.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.816+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj95]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj95[0] 2023-09-22T21:49:27.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.820+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj148]: success 2023-09-22T21:49:27.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.820+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj145]: success 2023-09-22T21:49:27.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.820+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj44]: success 2023-09-22T21:49:27.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.820+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj46]: success 2023-09-22T21:49:27.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.820+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj31]: success 2023-09-22T21:49:27.825 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.820+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj185]: success 2023-09-22T21:49:27.825 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.824+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj11]: success 2023-09-22T21:49:27.825 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.824+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: updating marker oid=bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24 marker=obj11 2023-09-22T21:49:27.825 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.824+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj23]: success 2023-09-22T21:49:27.838 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.836+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj62]: success 2023-09-22T21:49:27.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.844+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj55]: success 2023-09-22T21:49:27.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.844+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj58]: success 2023-09-22T21:49:27.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.844+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj57]: success 2023-09-22T21:49:27.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.868+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj68]: success 2023-09-22T21:49:27.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.868+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj92]: success 2023-09-22T21:49:27.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.868+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj87]: success 2023-09-22T21:49:27.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.868+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj74]: success 2023-09-22T21:49:27.873 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.872+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj95]: success 2023-09-22T21:49:27.874 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.872+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj76]: success 2023-09-22T21:49:27.874 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.872+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj93]: success 2023-09-22T21:49:27.875 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.872+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj85]: success 2023-09-22T21:49:27.875 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.872+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj69]: success 2023-09-22T21:49:27.875 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.872+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: backing out with sync_status=-5 2023-09-22T21:49:27.875 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.872+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: ERROR: full sync failed. error: -5 2023-09-22T21:49:27.877 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.876+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-09-22T21:49:27.877 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.876+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: waiting on bucket lease 2023-09-22T21:49:27.883 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.880+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: status after acquiring the lock is: full 2023-09-22T21:49:27.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.896+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: listing bucket for full sync 2023-09-22T21:49:27.947 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj11 2023-09-22T21:49:27.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj11]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj11 op=0 op_state=1 2023-09-22T21:49:27.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj11]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj11[0] 2023-09-22T21:49:27.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj111 2023-09-22T21:49:27.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj111]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj111 op=0 op_state=1 2023-09-22T21:49:27.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj111]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj111[0] 2023-09-22T21:49:27.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj115 2023-09-22T21:49:27.949 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj115]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj115 op=0 op_state=1 2023-09-22T21:49:27.949 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj115]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj115[0] 2023-09-22T21:49:27.949 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj116 2023-09-22T21:49:27.949 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj116]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj116 op=0 op_state=1 2023-09-22T21:49:27.949 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj116]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj116[0] 2023-09-22T21:49:27.949 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj124 2023-09-22T21:49:27.949 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj124]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj124 op=0 op_state=1 2023-09-22T21:49:27.949 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj124]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj124[0] 2023-09-22T21:49:27.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj145 2023-09-22T21:49:27.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj145]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj145 op=0 op_state=1 2023-09-22T21:49:27.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj145]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj145[0] 2023-09-22T21:49:27.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj148 2023-09-22T21:49:27.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj148]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj148 op=0 op_state=1 2023-09-22T21:49:27.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj148]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj148[0] 2023-09-22T21:49:27.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj155 2023-09-22T21:49:27.951 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj155]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj155 op=0 op_state=1 2023-09-22T21:49:27.951 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj155]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj155[0] 2023-09-22T21:49:27.951 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj17 2023-09-22T21:49:27.951 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj17]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj17 op=0 op_state=1 2023-09-22T21:49:27.951 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj17]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj17[0] 2023-09-22T21:49:27.951 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj185 2023-09-22T21:49:27.951 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj185]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj185 op=0 op_state=1 2023-09-22T21:49:27.951 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.944+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj185]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj185[0] 2023-09-22T21:49:27.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj19 2023-09-22T21:49:27.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj19]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj19 op=0 op_state=1 2023-09-22T21:49:27.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj19]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj19[0] 2023-09-22T21:49:27.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj22 2023-09-22T21:49:27.953 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj22]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj22 op=0 op_state=1 2023-09-22T21:49:27.953 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj22]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj22[0] 2023-09-22T21:49:27.953 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj23 2023-09-22T21:49:27.953 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj23]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj23 op=0 op_state=1 2023-09-22T21:49:27.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj23]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj23[0] 2023-09-22T21:49:27.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj29 2023-09-22T21:49:27.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj29]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj29 op=0 op_state=1 2023-09-22T21:49:27.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj29]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj29[0] 2023-09-22T21:49:27.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj31 2023-09-22T21:49:27.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj31]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj31 op=0 op_state=1 2023-09-22T21:49:27.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj31]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj31[0] 2023-09-22T21:49:27.955 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj33 2023-09-22T21:49:27.955 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj33]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj33 op=0 op_state=1 2023-09-22T21:49:27.955 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj33]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj33[0] 2023-09-22T21:49:27.955 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj34 2023-09-22T21:49:27.955 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj34]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj34 op=0 op_state=1 2023-09-22T21:49:27.956 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj34]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj34[0] 2023-09-22T21:49:27.956 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.948+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj37 2023-09-22T21:49:27.956 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.952+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj37]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj37 op=0 op_state=1 2023-09-22T21:49:27.956 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.952+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj37]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj37[0] 2023-09-22T21:49:27.956 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.952+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj38 2023-09-22T21:49:27.956 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.952+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj38]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj38 op=0 op_state=1 2023-09-22T21:49:27.956 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.952+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj38]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj38[0] 2023-09-22T21:49:27.957 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.952+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj44 2023-09-22T21:49:27.957 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.952+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj44]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj44 op=0 op_state=1 2023-09-22T21:49:27.957 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.952+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj44]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj44[0] 2023-09-22T21:49:27.957 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.952+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj46 2023-09-22T21:49:27.957 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.952+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj46]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj46 op=0 op_state=1 2023-09-22T21:49:27.957 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.952+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj46]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj46[0] 2023-09-22T21:49:27.957 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.952+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj124]: success 2023-09-22T21:49:27.957 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.952+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj51 2023-09-22T21:49:27.958 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.952+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj51]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj51 op=0 op_state=1 2023-09-22T21:49:27.958 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.952+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj51]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj51[0] 2023-09-22T21:49:27.960 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.960+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj11]: success 2023-09-22T21:49:27.961 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.960+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj148]: success 2023-09-22T21:49:27.961 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.960+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj115]: success 2023-09-22T21:49:27.961 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.960+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj116]: success 2023-09-22T21:49:27.962 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.960+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj17]: success 2023-09-22T21:49:27.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.960+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj23]: success 2023-09-22T21:49:27.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.960+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj55 2023-09-22T21:49:27.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.960+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj55]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj55 op=0 op_state=1 2023-09-22T21:49:27.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.960+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj55]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj55[0] 2023-09-22T21:49:27.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.960+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj185]: success 2023-09-22T21:49:27.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.960+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj57 2023-09-22T21:49:27.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.960+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj57]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj57 op=0 op_state=1 2023-09-22T21:49:27.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.960+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj145]: success 2023-09-22T21:49:27.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.960+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj57]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj57[0] 2023-09-22T21:49:27.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj19]: success 2023-09-22T21:49:27.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj58 2023-09-22T21:49:27.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj58]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj58 op=0 op_state=1 2023-09-22T21:49:27.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj58]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj58[0] 2023-09-22T21:49:27.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj62 2023-09-22T21:49:27.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj62]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj62 op=0 op_state=1 2023-09-22T21:49:27.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj62]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj62[0] 2023-09-22T21:49:27.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj68 2023-09-22T21:49:27.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj68]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj68 op=0 op_state=1 2023-09-22T21:49:27.969 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj33]: success 2023-09-22T21:49:27.969 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj68]: : injecting data sync error on key=obj68 2023-09-22T21:49:27.969 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj38]: success 2023-09-22T21:49:27.969 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj31]: success 2023-09-22T21:49:27.969 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj51]: success 2023-09-22T21:49:27.969 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj44]: success 2023-09-22T21:49:27.969 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj46]: success 2023-09-22T21:49:27.970 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj37]: success 2023-09-22T21:49:27.970 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj68]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:27.970 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj68]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj68 2023-09-22T21:49:27.970 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj69 2023-09-22T21:49:27.970 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj69]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj69 op=0 op_state=1 2023-09-22T21:49:27.970 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj69]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj69[0] 2023-09-22T21:49:27.970 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj74 2023-09-22T21:49:27.971 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj74]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj74 op=0 op_state=1 2023-09-22T21:49:27.971 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj74]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj74[0] 2023-09-22T21:49:27.971 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj76 2023-09-22T21:49:27.971 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj76]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj76 op=0 op_state=1 2023-09-22T21:49:27.971 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj76]: : injecting data sync error on key=obj76 2023-09-22T21:49:27.971 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.964+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj76]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:27.971 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.968+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj76]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj76 2023-09-22T21:49:27.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.968+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj85 2023-09-22T21:49:27.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.968+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj85]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj85 op=0 op_state=1 2023-09-22T21:49:27.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.968+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj85]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj85[0] 2023-09-22T21:49:27.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.968+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj87 2023-09-22T21:49:27.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.968+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj87]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj87 op=0 op_state=1 2023-09-22T21:49:27.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.968+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj87]: : injecting data sync error on key=obj87 2023-09-22T21:49:27.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.968+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:27.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.968+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj87]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj87 2023-09-22T21:49:27.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.968+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj92 2023-09-22T21:49:27.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.968+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj92]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj92 op=0 op_state=1 2023-09-22T21:49:27.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.968+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj92]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj92[0] 2023-09-22T21:49:27.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.968+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj93 2023-09-22T21:49:27.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.968+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj93]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj93 op=0 op_state=1 2023-09-22T21:49:27.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.968+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj93]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj93[0] 2023-09-22T21:49:27.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.968+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj95 2023-09-22T21:49:27.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.968+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj95]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj95 op=0 op_state=1 2023-09-22T21:49:27.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.968+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj95]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj95[0] 2023-09-22T21:49:27.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.968+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj55]: success 2023-09-22T21:49:27.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.968+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj155]: success 2023-09-22T21:49:27.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.972+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj111]: success 2023-09-22T21:49:27.976 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.972+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: updating marker oid=bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24 marker=obj19 2023-09-22T21:49:27.976 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.972+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj62]: success 2023-09-22T21:49:27.976 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.972+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj69]: success 2023-09-22T21:49:27.976 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.972+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj58]: success 2023-09-22T21:49:27.977 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.976+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj85]: success 2023-09-22T21:49:27.977 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.976+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj92]: success 2023-09-22T21:49:27.978 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.976+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj74]: success 2023-09-22T21:49:27.978 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.976+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj95]: success 2023-09-22T21:49:27.978 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.976+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj57]: success 2023-09-22T21:49:27.978 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.976+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:27.978 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.976+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:27.978 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.976+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:27.978 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.976+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj22]: success 2023-09-22T21:49:27.979 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:27.976+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj93]: success 2023-09-22T21:49:28.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.004+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj29]: success 2023-09-22T21:49:28.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.004+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: updating marker oid=bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24 marker=obj33 2023-09-22T21:49:28.013 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.012+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj34]: success 2023-09-22T21:49:28.013 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.012+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: updating marker oid=bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24 marker=obj62 2023-09-22T21:49:28.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.016+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: backing out with sync_status=-5 2023-09-22T21:49:28.016 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.016+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: ERROR: full sync failed. error: -5 2023-09-22T21:49:28.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.016+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.016+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: waiting on bucket lease 2023-09-22T21:49:28.028 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.028+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: status after acquiring the lock is: full 2023-09-22T21:49:28.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.052+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: listing bucket for full sync 2023-09-22T21:49:28.116 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj11 2023-09-22T21:49:28.116 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj11]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj11 op=0 op_state=1 2023-09-22T21:49:28.117 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj11]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj11[0] 2023-09-22T21:49:28.117 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj111 2023-09-22T21:49:28.117 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj111]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj111 op=0 op_state=1 2023-09-22T21:49:28.117 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj111]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj111[0] 2023-09-22T21:49:28.117 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj115 2023-09-22T21:49:28.117 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj115]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj115 op=0 op_state=1 2023-09-22T21:49:28.117 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj115]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj115[0] 2023-09-22T21:49:28.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj116 2023-09-22T21:49:28.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj116]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj116 op=0 op_state=1 2023-09-22T21:49:28.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj116]: : injecting data sync error on key=obj116 2023-09-22T21:49:28.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj116]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:28.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj116]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj116 2023-09-22T21:49:28.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj124 2023-09-22T21:49:28.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj124]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj124 op=0 op_state=1 2023-09-22T21:49:28.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj124]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj124[0] 2023-09-22T21:49:28.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj145 2023-09-22T21:49:28.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj145]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj145 op=0 op_state=1 2023-09-22T21:49:28.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj145]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj145[0] 2023-09-22T21:49:28.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj148 2023-09-22T21:49:28.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj148]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj148 op=0 op_state=1 2023-09-22T21:49:28.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.112+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj148]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj148[0] 2023-09-22T21:49:28.120 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj155 2023-09-22T21:49:28.120 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj155]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj155 op=0 op_state=1 2023-09-22T21:49:28.120 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj155]: : injecting data sync error on key=obj155 2023-09-22T21:49:28.120 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:28.120 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj155]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj155 2023-09-22T21:49:28.120 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj17 2023-09-22T21:49:28.120 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj17]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj17 op=0 op_state=1 2023-09-22T21:49:28.120 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj17]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj17[0] 2023-09-22T21:49:28.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj185 2023-09-22T21:49:28.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj185]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj185 op=0 op_state=1 2023-09-22T21:49:28.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj185]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj185[0] 2023-09-22T21:49:28.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj19 2023-09-22T21:49:28.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj19]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj19 op=0 op_state=1 2023-09-22T21:49:28.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj19]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj19[0] 2023-09-22T21:49:28.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj22 2023-09-22T21:49:28.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj22]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj22 op=0 op_state=1 2023-09-22T21:49:28.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj22]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj22[0] 2023-09-22T21:49:28.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj23 2023-09-22T21:49:28.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj23]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj23 op=0 op_state=1 2023-09-22T21:49:28.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj23]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj23[0] 2023-09-22T21:49:28.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj29 2023-09-22T21:49:28.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj29]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj29 op=0 op_state=1 2023-09-22T21:49:28.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj29]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj29[0] 2023-09-22T21:49:28.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj31 2023-09-22T21:49:28.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj31]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj31 op=0 op_state=1 2023-09-22T21:49:28.126 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj31]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj31[0] 2023-09-22T21:49:28.126 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj33 2023-09-22T21:49:28.126 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj33]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj33 op=0 op_state=1 2023-09-22T21:49:28.126 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj33]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj33[0] 2023-09-22T21:49:28.126 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj34 2023-09-22T21:49:28.126 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj34]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj34 op=0 op_state=1 2023-09-22T21:49:28.126 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj34]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj34[0] 2023-09-22T21:49:28.126 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj37 2023-09-22T21:49:28.127 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj37]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj37 op=0 op_state=1 2023-09-22T21:49:28.127 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj37]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj37[0] 2023-09-22T21:49:28.127 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj38 2023-09-22T21:49:28.127 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj38]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj38 op=0 op_state=1 2023-09-22T21:49:28.127 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj38]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj38[0] 2023-09-22T21:49:28.127 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj44 2023-09-22T21:49:28.127 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj44]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj44 op=0 op_state=1 2023-09-22T21:49:28.127 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj44]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj44[0] 2023-09-22T21:49:28.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj46 2023-09-22T21:49:28.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj46]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj46 op=0 op_state=1 2023-09-22T21:49:28.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.116+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj46]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj46[0] 2023-09-22T21:49:28.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.124+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:28.129 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.124+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj51 2023-09-22T21:49:28.129 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.124+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj51]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj51 op=0 op_state=1 2023-09-22T21:49:28.129 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.124+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj51]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj51[0] 2023-09-22T21:49:28.129 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.124+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:28.129 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.124+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj55 2023-09-22T21:49:28.129 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.124+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj55]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj55 op=0 op_state=1 2023-09-22T21:49:28.129 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.124+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj55]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj55[0] 2023-09-22T21:49:28.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj124]: success 2023-09-22T21:49:28.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj115]: success 2023-09-22T21:49:28.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj11]: success 2023-09-22T21:49:28.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj111]: success 2023-09-22T21:49:28.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj57 2023-09-22T21:49:28.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj57]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj57 op=0 op_state=1 2023-09-22T21:49:28.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj57]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj57[0] 2023-09-22T21:49:28.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj23]: success 2023-09-22T21:49:28.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj58 2023-09-22T21:49:28.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj58]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj58 op=0 op_state=1 2023-09-22T21:49:28.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj58]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj58[0] 2023-09-22T21:49:28.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj62 2023-09-22T21:49:28.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj62]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj62 op=0 op_state=1 2023-09-22T21:49:28.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj62]: : injecting data sync error on key=obj62 2023-09-22T21:49:28.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj62]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:28.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj62]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj62 2023-09-22T21:49:28.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj68 2023-09-22T21:49:28.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj68]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj68 op=0 op_state=1 2023-09-22T21:49:28.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj68]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj68[0] 2023-09-22T21:49:28.137 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj69 2023-09-22T21:49:28.137 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj69]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj69 op=0 op_state=1 2023-09-22T21:49:28.137 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj69]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj69[0] 2023-09-22T21:49:28.137 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.132+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj185]: success 2023-09-22T21:49:28.137 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj29]: success 2023-09-22T21:49:28.138 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj145]: success 2023-09-22T21:49:28.138 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj34]: success 2023-09-22T21:49:28.138 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj46]: success 2023-09-22T21:49:28.138 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj37]: success 2023-09-22T21:49:28.138 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj74 2023-09-22T21:49:28.138 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj74]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj74 op=0 op_state=1 2023-09-22T21:49:28.138 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj33]: success 2023-09-22T21:49:28.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj22]: success 2023-09-22T21:49:28.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj17]: success 2023-09-22T21:49:28.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj19]: success 2023-09-22T21:49:28.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj38]: success 2023-09-22T21:49:28.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj148]: success 2023-09-22T21:49:28.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj31]: success 2023-09-22T21:49:28.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj44]: success 2023-09-22T21:49:28.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj74]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj74[0] 2023-09-22T21:49:28.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj76 2023-09-22T21:49:28.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj76]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj76 op=0 op_state=1 2023-09-22T21:49:28.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj76]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj76[0] 2023-09-22T21:49:28.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj85 2023-09-22T21:49:28.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj85]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj85 op=0 op_state=1 2023-09-22T21:49:28.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj85]: : injecting data sync error on key=obj85 2023-09-22T21:49:28.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:28.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj85]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj85 2023-09-22T21:49:28.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj87 2023-09-22T21:49:28.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj87]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj87 op=0 op_state=1 2023-09-22T21:49:28.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj87]: : injecting data sync error on key=obj87 2023-09-22T21:49:28.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:28.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj87]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj87 2023-09-22T21:49:28.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj92 2023-09-22T21:49:28.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj92]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj92 op=0 op_state=1 2023-09-22T21:49:28.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj92]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj92[0] 2023-09-22T21:49:28.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj93 2023-09-22T21:49:28.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj93]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj93 op=0 op_state=1 2023-09-22T21:49:28.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj93]: : injecting data sync error on key=obj93 2023-09-22T21:49:28.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj93]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:28.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj93]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj93 2023-09-22T21:49:28.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj95 2023-09-22T21:49:28.143 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj95]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj95 op=0 op_state=1 2023-09-22T21:49:28.143 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj95]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj95[0] 2023-09-22T21:49:28.143 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj55]: success 2023-09-22T21:49:28.143 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.136+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj51]: success 2023-09-22T21:49:28.144 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.144+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj69]: success 2023-09-22T21:49:28.144 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.144+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj57]: success 2023-09-22T21:49:28.145 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.144+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj76]: success 2023-09-22T21:49:28.145 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.144+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj58]: success 2023-09-22T21:49:28.145 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.144+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj74]: success 2023-09-22T21:49:28.145 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.144+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj68]: success 2023-09-22T21:49:28.146 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.144+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:28.146 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.144+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj92]: success 2023-09-22T21:49:28.147 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.144+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:28.147 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.144+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:28.147 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.144+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:28.147 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.144+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj95]: success 2023-09-22T21:49:28.147 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.148+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: updating marker oid=bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24 marker=obj115 2023-09-22T21:49:28.150 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.148+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: backing out with sync_status=-5 2023-09-22T21:49:28.150 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.148+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: ERROR: full sync failed. error: -5 2023-09-22T21:49:28.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.152+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.154 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.152+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: waiting on bucket lease 2023-09-22T21:49:28.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.208+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: status after acquiring the lock is: full 2023-09-22T21:49:28.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.244+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: listing bucket for full sync 2023-09-22T21:49:28.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj11 2023-09-22T21:49:28.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj11]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj11 op=0 op_state=1 2023-09-22T21:49:28.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj11]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj11[0] 2023-09-22T21:49:28.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj111 2023-09-22T21:49:28.303 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj111]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj111 op=0 op_state=1 2023-09-22T21:49:28.303 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj111]: : injecting data sync error on key=obj111 2023-09-22T21:49:28.303 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:28.303 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj111]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj111 2023-09-22T21:49:28.303 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj115 2023-09-22T21:49:28.303 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj115]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj115 op=0 op_state=1 2023-09-22T21:49:28.303 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj115]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj115[0] 2023-09-22T21:49:28.304 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj116 2023-09-22T21:49:28.304 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj116]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj116 op=0 op_state=1 2023-09-22T21:49:28.304 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj116]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj116[0] 2023-09-22T21:49:28.304 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj124 2023-09-22T21:49:28.304 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj124]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj124 op=0 op_state=1 2023-09-22T21:49:28.304 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj124]: : injecting data sync error on key=obj124 2023-09-22T21:49:28.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj124]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:28.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj124]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj124 2023-09-22T21:49:28.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj145 2023-09-22T21:49:28.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj145]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj145 op=0 op_state=1 2023-09-22T21:49:28.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj145]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj145[0] 2023-09-22T21:49:28.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj148 2023-09-22T21:49:28.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj148]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj148 op=0 op_state=1 2023-09-22T21:49:28.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj148]: : injecting data sync error on key=obj148 2023-09-22T21:49:28.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj148]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:28.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj148]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj148 2023-09-22T21:49:28.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj155 2023-09-22T21:49:28.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj155]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj155 op=0 op_state=1 2023-09-22T21:49:28.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj155]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj155[0] 2023-09-22T21:49:28.307 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj17 2023-09-22T21:49:28.308 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj17]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj17 op=0 op_state=1 2023-09-22T21:49:28.308 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.300+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj17]: : injecting data sync error on key=obj17 2023-09-22T21:49:28.308 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj17]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:28.308 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj17]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj17 2023-09-22T21:49:28.308 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj185 2023-09-22T21:49:28.309 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj185]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj185 op=0 op_state=1 2023-09-22T21:49:28.309 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj185]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj185[0] 2023-09-22T21:49:28.309 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj19 2023-09-22T21:49:28.309 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj19]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj19 op=0 op_state=1 2023-09-22T21:49:28.309 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj19]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj19[0] 2023-09-22T21:49:28.309 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj22 2023-09-22T21:49:28.309 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj22]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj22 op=0 op_state=1 2023-09-22T21:49:28.310 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj22]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj22[0] 2023-09-22T21:49:28.310 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj23 2023-09-22T21:49:28.310 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj23]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj23 op=0 op_state=1 2023-09-22T21:49:28.310 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj23]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj23[0] 2023-09-22T21:49:28.310 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj29 2023-09-22T21:49:28.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj29]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj29 op=0 op_state=1 2023-09-22T21:49:28.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj29]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj29[0] 2023-09-22T21:49:28.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj31 2023-09-22T21:49:28.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj31]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj31 op=0 op_state=1 2023-09-22T21:49:28.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj31]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj31[0] 2023-09-22T21:49:28.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj33 2023-09-22T21:49:28.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj33]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj33 op=0 op_state=1 2023-09-22T21:49:28.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.304+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj33]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj33[0] 2023-09-22T21:49:28.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj34 2023-09-22T21:49:28.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj34]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj34 op=0 op_state=1 2023-09-22T21:49:28.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj34]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj34[0] 2023-09-22T21:49:28.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj37 2023-09-22T21:49:28.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj37]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj37 op=0 op_state=1 2023-09-22T21:49:28.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj37]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj37[0] 2023-09-22T21:49:28.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj38 2023-09-22T21:49:28.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj38]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj38 op=0 op_state=1 2023-09-22T21:49:28.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj38]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj38[0] 2023-09-22T21:49:28.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj44 2023-09-22T21:49:28.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj44]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj44 op=0 op_state=1 2023-09-22T21:49:28.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj44]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj44[0] 2023-09-22T21:49:28.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj11]: success 2023-09-22T21:49:28.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj116]: success 2023-09-22T21:49:28.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj46 2023-09-22T21:49:28.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj46]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj46 op=0 op_state=1 2023-09-22T21:49:28.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj46]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj46[0] 2023-09-22T21:49:28.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:28.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:28.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:28.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj51 2023-09-22T21:49:28.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj51]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj51 op=0 op_state=1 2023-09-22T21:49:28.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj51]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj51[0] 2023-09-22T21:49:28.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj55 2023-09-22T21:49:28.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj55]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj55 op=0 op_state=1 2023-09-22T21:49:28.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.308+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj55]: : injecting data sync error on key=obj55 2023-09-22T21:49:28.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.312+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj55]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:28.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.312+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj55]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj55 2023-09-22T21:49:28.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.312+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj57 2023-09-22T21:49:28.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.312+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj57]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj57 op=0 op_state=1 2023-09-22T21:49:28.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.312+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj145]: success 2023-09-22T21:49:28.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.312+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj115]: success 2023-09-22T21:49:28.319 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.312+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj57]: : injecting data sync error on key=obj57 2023-09-22T21:49:28.319 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.312+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj57]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:28.319 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.312+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj57]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj57 2023-09-22T21:49:28.319 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj58 2023-09-22T21:49:28.319 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj58]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj58 op=0 op_state=1 2023-09-22T21:49:28.319 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj58]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj58[0] 2023-09-22T21:49:28.319 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj62 2023-09-22T21:49:28.319 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj62]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj62 op=0 op_state=1 2023-09-22T21:49:28.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj62]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj62[0] 2023-09-22T21:49:28.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:28.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj19]: success 2023-09-22T21:49:28.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj68 2023-09-22T21:49:28.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj68]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj68 op=0 op_state=1 2023-09-22T21:49:28.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj68]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj68[0] 2023-09-22T21:49:28.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj29]: success 2023-09-22T21:49:28.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj69 2023-09-22T21:49:28.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj69]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj69 op=0 op_state=1 2023-09-22T21:49:28.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj185]: success 2023-09-22T21:49:28.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj69]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj69[0] 2023-09-22T21:49:28.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj22]: success 2023-09-22T21:49:28.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj74 2023-09-22T21:49:28.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj74]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj74 op=0 op_state=1 2023-09-22T21:49:28.322 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj74]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj74[0] 2023-09-22T21:49:28.322 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj76 2023-09-22T21:49:28.322 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj76]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj76 op=0 op_state=1 2023-09-22T21:49:28.322 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj76]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj76[0] 2023-09-22T21:49:28.322 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj85 2023-09-22T21:49:28.322 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj85]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj85 op=0 op_state=1 2023-09-22T21:49:28.322 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj85]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj85[0] 2023-09-22T21:49:28.322 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj87 2023-09-22T21:49:28.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj87]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj87 op=0 op_state=1 2023-09-22T21:49:28.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.316+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj87]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj87[0] 2023-09-22T21:49:28.324 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.320+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj92 2023-09-22T21:49:28.324 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.320+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj92]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj92 op=0 op_state=1 2023-09-22T21:49:28.325 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.320+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj92]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj92[0] 2023-09-22T21:49:28.325 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.320+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj155]: success 2023-09-22T21:49:28.325 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.320+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj93 2023-09-22T21:49:28.325 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.320+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj93]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj93 op=0 op_state=1 2023-09-22T21:49:28.325 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.320+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj93]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj93[0] 2023-09-22T21:49:28.325 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.320+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj33]: success 2023-09-22T21:49:28.325 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.320+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj23]: success 2023-09-22T21:49:28.326 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.320+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj31]: success 2023-09-22T21:49:28.326 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.324+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj95 2023-09-22T21:49:28.326 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.324+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj95]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj95 op=0 op_state=1 2023-09-22T21:49:28.326 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.324+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj34]: success 2023-09-22T21:49:28.326 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.324+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj95]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj95[0] 2023-09-22T21:49:28.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.324+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:28.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.324+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:28.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.324+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj37]: success 2023-09-22T21:49:28.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.324+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj44]: success 2023-09-22T21:49:28.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.328+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj38]: success 2023-09-22T21:49:28.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.328+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj76]: success 2023-09-22T21:49:28.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.328+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj69]: success 2023-09-22T21:49:28.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.328+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj85]: success 2023-09-22T21:49:28.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.328+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj51]: success 2023-09-22T21:49:28.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.328+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj74]: success 2023-09-22T21:49:28.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.328+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj62]: success 2023-09-22T21:49:28.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.328+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj58]: success 2023-09-22T21:49:28.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.328+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj68]: success 2023-09-22T21:49:28.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.328+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj46]: success 2023-09-22T21:49:28.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.328+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj93]: success 2023-09-22T21:49:28.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.328+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj92]: success 2023-09-22T21:49:28.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.328+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj87]: success 2023-09-22T21:49:28.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.332+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj95]: success 2023-09-22T21:49:28.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.332+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: updating marker oid=bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24 marker=obj11 2023-09-22T21:49:28.335 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.332+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: backing out with sync_status=-5 2023-09-22T21:49:28.335 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.332+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: ERROR: full sync failed. error: -5 2023-09-22T21:49:28.338 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.336+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.338 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.336+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: waiting on bucket lease 2023-09-22T21:49:28.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.392+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: status after acquiring the lock is: full 2023-09-22T21:49:28.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.420+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: listing bucket for full sync 2023-09-22T21:49:28.483 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj11 2023-09-22T21:49:28.485 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj11]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj11 op=0 op_state=1 2023-09-22T21:49:28.485 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj11]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj11[0] 2023-09-22T21:49:28.485 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj111 2023-09-22T21:49:28.485 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj111]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj111 op=0 op_state=1 2023-09-22T21:49:28.485 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj111]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj111[0] 2023-09-22T21:49:28.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj115 2023-09-22T21:49:28.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj115]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj115 op=0 op_state=1 2023-09-22T21:49:28.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj115]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj115[0] 2023-09-22T21:49:28.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj116 2023-09-22T21:49:28.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj116]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj116 op=0 op_state=1 2023-09-22T21:49:28.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj116]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj116[0] 2023-09-22T21:49:28.487 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj124 2023-09-22T21:49:28.487 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj124]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj124 op=0 op_state=1 2023-09-22T21:49:28.487 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj124]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj124[0] 2023-09-22T21:49:28.487 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj145 2023-09-22T21:49:28.487 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj145]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj145 op=0 op_state=1 2023-09-22T21:49:28.488 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj145]: : injecting data sync error on key=obj145 2023-09-22T21:49:28.488 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj145]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:28.488 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj145]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj145 2023-09-22T21:49:28.488 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj148 2023-09-22T21:49:28.488 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj148]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj148 op=0 op_state=1 2023-09-22T21:49:28.489 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj148]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj148[0] 2023-09-22T21:49:28.489 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj155 2023-09-22T21:49:28.489 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj155]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj155 op=0 op_state=1 2023-09-22T21:49:28.489 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj155]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj155[0] 2023-09-22T21:49:28.489 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj17 2023-09-22T21:49:28.490 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj17]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj17 op=0 op_state=1 2023-09-22T21:49:28.490 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj17]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj17[0] 2023-09-22T21:49:28.490 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.480+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj185 2023-09-22T21:49:28.490 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj185]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj185 op=0 op_state=1 2023-09-22T21:49:28.491 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj185]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj185[0] 2023-09-22T21:49:28.496 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj19 2023-09-22T21:49:28.496 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj19]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj19 op=0 op_state=1 2023-09-22T21:49:28.496 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj19]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj19[0] 2023-09-22T21:49:28.496 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj22 2023-09-22T21:49:28.497 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj22]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj22 op=0 op_state=1 2023-09-22T21:49:28.497 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj22]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj22[0] 2023-09-22T21:49:28.497 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj23 2023-09-22T21:49:28.497 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj23]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj23 op=0 op_state=1 2023-09-22T21:49:28.497 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj23]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj23[0] 2023-09-22T21:49:28.498 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj29 2023-09-22T21:49:28.498 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj29]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj29 op=0 op_state=1 2023-09-22T21:49:28.498 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj29]: : injecting data sync error on key=obj29 2023-09-22T21:49:28.498 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:28.498 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj29]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj29 2023-09-22T21:49:28.498 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj31 2023-09-22T21:49:28.499 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj31]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj31 op=0 op_state=1 2023-09-22T21:49:28.499 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj31]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj31[0] 2023-09-22T21:49:28.499 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj33 2023-09-22T21:49:28.499 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj33]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj33 op=0 op_state=1 2023-09-22T21:49:28.499 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.484+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj33]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj33[0] 2023-09-22T21:49:28.499 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.488+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj34 2023-09-22T21:49:28.499 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.488+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj34]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj34 op=0 op_state=1 2023-09-22T21:49:28.499 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.488+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj34]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj34[0] 2023-09-22T21:49:28.500 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.488+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj37 2023-09-22T21:49:28.500 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.488+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj37]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj37 op=0 op_state=1 2023-09-22T21:49:28.500 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.488+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj37]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj37[0] 2023-09-22T21:49:28.500 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.488+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj38 2023-09-22T21:49:28.500 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.488+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj38]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj38 op=0 op_state=1 2023-09-22T21:49:28.500 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.488+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj38]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj38[0] 2023-09-22T21:49:28.500 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.488+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj44 2023-09-22T21:49:28.500 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.492+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj44]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj44 op=0 op_state=1 2023-09-22T21:49:28.501 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.492+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj44]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj44[0] 2023-09-22T21:49:28.501 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.492+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj46 2023-09-22T21:49:28.501 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.492+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj46]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj46 op=0 op_state=1 2023-09-22T21:49:28.501 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.492+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj46]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj46[0] 2023-09-22T21:49:28.501 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.492+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj116]: success 2023-09-22T21:49:28.501 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.492+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj111]: success 2023-09-22T21:49:28.501 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.492+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj115]: success 2023-09-22T21:49:28.501 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.492+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj51 2023-09-22T21:49:28.502 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.492+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj51]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj51 op=0 op_state=1 2023-09-22T21:49:28.502 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.492+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj51]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj51[0] 2023-09-22T21:49:28.502 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.492+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj55 2023-09-22T21:49:28.502 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.492+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj55]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj55 op=0 op_state=1 2023-09-22T21:49:28.502 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.492+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj55]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj55[0] 2023-09-22T21:49:28.502 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.492+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj57 2023-09-22T21:49:28.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.496+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj57]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj57 op=0 op_state=1 2023-09-22T21:49:28.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.496+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj57]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj57[0] 2023-09-22T21:49:28.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.496+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj185]: success 2023-09-22T21:49:28.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.496+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:28.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.496+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj58 2023-09-22T21:49:28.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.496+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj58]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj58 op=0 op_state=1 2023-09-22T21:49:28.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.496+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj58]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj58[0] 2023-09-22T21:49:28.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.496+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj62 2023-09-22T21:49:28.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.496+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj62]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj62 op=0 op_state=1 2023-09-22T21:49:28.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.496+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj62]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj62[0] 2023-09-22T21:49:28.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.496+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj148]: success 2023-09-22T21:49:28.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj11]: success 2023-09-22T21:49:28.508 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj19]: success 2023-09-22T21:49:28.508 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj124]: success 2023-09-22T21:49:28.508 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj23]: success 2023-09-22T21:49:28.508 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj17]: success 2023-09-22T21:49:28.508 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj155]: success 2023-09-22T21:49:28.508 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj33]: success 2023-09-22T21:49:28.508 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj68 2023-09-22T21:49:28.509 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj68]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj68 op=0 op_state=1 2023-09-22T21:49:28.509 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj22]: success 2023-09-22T21:49:28.509 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj68]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj68[0] 2023-09-22T21:49:28.509 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj37]: success 2023-09-22T21:49:28.509 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj69 2023-09-22T21:49:28.509 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj69]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj69 op=0 op_state=1 2023-09-22T21:49:28.509 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj34]: success 2023-09-22T21:49:28.509 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj69]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj69[0] 2023-09-22T21:49:28.510 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj74 2023-09-22T21:49:28.510 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj74]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj74 op=0 op_state=1 2023-09-22T21:49:28.510 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj74]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj74[0] 2023-09-22T21:49:28.510 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj31]: success 2023-09-22T21:49:28.510 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj76 2023-09-22T21:49:28.510 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj76]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj76 op=0 op_state=1 2023-09-22T21:49:28.510 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj76]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj76[0] 2023-09-22T21:49:28.511 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj85 2023-09-22T21:49:28.511 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj85]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj85 op=0 op_state=1 2023-09-22T21:49:28.511 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.500+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj85]: : injecting data sync error on key=obj85 2023-09-22T21:49:28.511 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.504+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2023-09-22T21:49:28.511 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.504+0000 7f7d302aea40 0 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj85]: ERROR: failed to sync object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj85 2023-09-22T21:49:28.511 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.504+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj87 2023-09-22T21:49:28.511 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.504+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj87]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj87 op=0 op_state=1 2023-09-22T21:49:28.511 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.504+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj87]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj87[0] 2023-09-22T21:49:28.512 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.504+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj92 2023-09-22T21:49:28.512 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.504+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj92]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj92 op=0 op_state=1 2023-09-22T21:49:28.512 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.504+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj92]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj92[0] 2023-09-22T21:49:28.512 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.504+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj93 2023-09-22T21:49:28.512 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.504+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj93]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj93 op=0 op_state=1 2023-09-22T21:49:28.512 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.504+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj93]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj93[0] 2023-09-22T21:49:28.513 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.504+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj95 2023-09-22T21:49:28.514 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.504+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj95]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj95 op=0 op_state=1 2023-09-22T21:49:28.514 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.508+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj95]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj95[0] 2023-09-22T21:49:28.514 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.508+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj55]: success 2023-09-22T21:49:28.514 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.508+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj44]: success 2023-09-22T21:49:28.514 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.508+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:28.514 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.508+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj57]: success 2023-09-22T21:49:28.514 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.508+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj58]: success 2023-09-22T21:49:28.515 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.508+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj51]: success 2023-09-22T21:49:28.515 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.508+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj46]: success 2023-09-22T21:49:28.515 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.508+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj38]: success 2023-09-22T21:49:28.515 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.508+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj69]: success 2023-09-22T21:49:28.515 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.508+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj62]: success 2023-09-22T21:49:28.515 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.512+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj74]: success 2023-09-22T21:49:28.515 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.512+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj76]: success 2023-09-22T21:49:28.516 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.512+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj87]: success 2023-09-22T21:49:28.516 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.512+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj92]: success 2023-09-22T21:49:28.516 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.512+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: a sync operation returned error 2023-09-22T21:49:28.516 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.512+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj68]: success 2023-09-22T21:49:28.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.512+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj95]: success 2023-09-22T21:49:28.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.512+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj93]: success 2023-09-22T21:49:28.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.512+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: updating marker oid=bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24 marker=obj124 2023-09-22T21:49:28.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.516+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: backing out with sync_status=-5 2023-09-22T21:49:28.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.516+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: ERROR: full sync failed. error: -5 2023-09-22T21:49:28.521 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.520+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.521 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.520+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: waiting on bucket lease 2023-09-22T21:49:28.573 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.572+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: status after acquiring the lock is: full 2023-09-22T21:49:28.610 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.608+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: listing bucket for full sync 2023-09-22T21:49:28.668 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.664+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj11 2023-09-22T21:49:28.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.664+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj11]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj11 op=0 op_state=1 2023-09-22T21:49:28.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.664+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj11]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj11[0] 2023-09-22T21:49:28.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.664+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj111 2023-09-22T21:49:28.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.664+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj111]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj111 op=0 op_state=1 2023-09-22T21:49:28.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.664+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj111]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj111[0] 2023-09-22T21:49:28.670 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.664+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj115 2023-09-22T21:49:28.670 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.664+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj115]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj115 op=0 op_state=1 2023-09-22T21:49:28.670 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.664+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj115]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj115[0] 2023-09-22T21:49:28.670 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj116 2023-09-22T21:49:28.670 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj116]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj116 op=0 op_state=1 2023-09-22T21:49:28.670 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj116]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj116[0] 2023-09-22T21:49:28.670 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj124 2023-09-22T21:49:28.671 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj124]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj124 op=0 op_state=1 2023-09-22T21:49:28.671 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj124]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj124[0] 2023-09-22T21:49:28.671 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj145 2023-09-22T21:49:28.671 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj145]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj145 op=0 op_state=1 2023-09-22T21:49:28.671 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj145]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj145[0] 2023-09-22T21:49:28.671 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj148 2023-09-22T21:49:28.671 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj148]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj148 op=0 op_state=1 2023-09-22T21:49:28.671 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj148]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj148[0] 2023-09-22T21:49:28.672 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj155 2023-09-22T21:49:28.672 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj155]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj155 op=0 op_state=1 2023-09-22T21:49:28.672 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj155]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj155[0] 2023-09-22T21:49:28.672 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj17 2023-09-22T21:49:28.672 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj17]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj17 op=0 op_state=1 2023-09-22T21:49:28.672 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj17]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj17[0] 2023-09-22T21:49:28.672 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj185 2023-09-22T21:49:28.672 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj185]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj185 op=0 op_state=1 2023-09-22T21:49:28.673 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj185]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj185[0] 2023-09-22T21:49:28.673 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj19 2023-09-22T21:49:28.673 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj19]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj19 op=0 op_state=1 2023-09-22T21:49:28.673 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj19]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj19[0] 2023-09-22T21:49:28.673 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj22 2023-09-22T21:49:28.673 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj22]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj22 op=0 op_state=1 2023-09-22T21:49:28.673 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj22]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj22[0] 2023-09-22T21:49:28.674 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj23 2023-09-22T21:49:28.674 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj23]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj23 op=0 op_state=1 2023-09-22T21:49:28.674 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj23]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj23[0] 2023-09-22T21:49:28.674 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj29 2023-09-22T21:49:28.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj29]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj29 op=0 op_state=1 2023-09-22T21:49:28.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj29]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj29[0] 2023-09-22T21:49:28.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj31 2023-09-22T21:49:28.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj31]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj31 op=0 op_state=1 2023-09-22T21:49:28.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj31]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj31[0] 2023-09-22T21:49:28.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj33 2023-09-22T21:49:28.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj33]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj33 op=0 op_state=1 2023-09-22T21:49:28.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj33]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj33[0] 2023-09-22T21:49:28.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj34 2023-09-22T21:49:28.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj34]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj34 op=0 op_state=1 2023-09-22T21:49:28.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj34]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj34[0] 2023-09-22T21:49:28.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj37 2023-09-22T21:49:28.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj37]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj37 op=0 op_state=1 2023-09-22T21:49:28.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj37]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj37[0] 2023-09-22T21:49:28.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj38 2023-09-22T21:49:28.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj38]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj38 op=0 op_state=1 2023-09-22T21:49:28.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj38]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj38[0] 2023-09-22T21:49:28.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj44 2023-09-22T21:49:28.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj44]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj44 op=0 op_state=1 2023-09-22T21:49:28.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj44]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj44[0] 2023-09-22T21:49:28.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj46 2023-09-22T21:49:28.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj46]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj46 op=0 op_state=1 2023-09-22T21:49:28.678 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.668+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj46]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj46[0] 2023-09-22T21:49:28.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.676+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj111]: success 2023-09-22T21:49:28.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.676+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj11]: success 2023-09-22T21:49:28.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.676+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj116]: success 2023-09-22T21:49:28.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.676+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj51 2023-09-22T21:49:28.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.676+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj51]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj51 op=0 op_state=1 2023-09-22T21:49:28.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.676+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj51]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj51[0] 2023-09-22T21:49:28.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.676+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj55 2023-09-22T21:49:28.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.676+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj55]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj55 op=0 op_state=1 2023-09-22T21:49:28.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.676+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj55]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj55[0] 2023-09-22T21:49:28.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.676+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj57 2023-09-22T21:49:28.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.676+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj57]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj57 op=0 op_state=1 2023-09-22T21:49:28.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.676+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj57]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj57[0] 2023-09-22T21:49:28.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.676+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj124]: success 2023-09-22T21:49:28.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.676+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj58 2023-09-22T21:49:28.681 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.676+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj58]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj58 op=0 op_state=1 2023-09-22T21:49:28.681 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.676+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj58]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj58[0] 2023-09-22T21:49:28.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.684+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj23]: success 2023-09-22T21:49:28.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.684+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj19]: success 2023-09-22T21:49:28.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.684+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj22]: success 2023-09-22T21:49:28.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj29]: success 2023-09-22T21:49:28.689 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj115]: success 2023-09-22T21:49:28.689 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj33]: success 2023-09-22T21:49:28.689 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj185]: success 2023-09-22T21:49:28.689 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj17]: success 2023-09-22T21:49:28.689 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj62 2023-09-22T21:49:28.689 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj62]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj62 op=0 op_state=1 2023-09-22T21:49:28.689 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj62]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj62[0] 2023-09-22T21:49:28.690 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj44]: success 2023-09-22T21:49:28.690 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj68 2023-09-22T21:49:28.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj68]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj68 op=0 op_state=1 2023-09-22T21:49:28.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj68]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj68[0] 2023-09-22T21:49:28.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj34]: success 2023-09-22T21:49:28.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj31]: success 2023-09-22T21:49:28.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj46]: success 2023-09-22T21:49:28.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj145]: success 2023-09-22T21:49:28.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: updating marker oid=bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24 marker=obj145 2023-09-22T21:49:28.692 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj148]: success 2023-09-22T21:49:28.692 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj69 2023-09-22T21:49:28.692 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj69]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj69 op=0 op_state=1 2023-09-22T21:49:28.692 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj37]: success 2023-09-22T21:49:28.692 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj69]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj69[0] 2023-09-22T21:49:28.692 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj155]: success 2023-09-22T21:49:28.692 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj74 2023-09-22T21:49:28.692 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj74]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj74 op=0 op_state=1 2023-09-22T21:49:28.693 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj74]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj74[0] 2023-09-22T21:49:28.693 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj76 2023-09-22T21:49:28.693 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj76]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj76 op=0 op_state=1 2023-09-22T21:49:28.693 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj76]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj76[0] 2023-09-22T21:49:28.693 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj85 2023-09-22T21:49:28.694 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj85]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj85 op=0 op_state=1 2023-09-22T21:49:28.694 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj85]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj85[0] 2023-09-22T21:49:28.694 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj87 2023-09-22T21:49:28.694 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj87]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj87 op=0 op_state=1 2023-09-22T21:49:28.694 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj87]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj87[0] 2023-09-22T21:49:28.694 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj92 2023-09-22T21:49:28.694 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj92]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj92 op=0 op_state=1 2023-09-22T21:49:28.694 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj92]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj92[0] 2023-09-22T21:49:28.695 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj38]: success 2023-09-22T21:49:28.695 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj55]: success 2023-09-22T21:49:28.695 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj93 2023-09-22T21:49:28.695 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj93]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj93 op=0 op_state=1 2023-09-22T21:49:28.695 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj93]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj93[0] 2023-09-22T21:49:28.695 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: [full sync] syncing object: ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0/obj95 2023-09-22T21:49:28.695 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj95]: bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b=bucket sync single entry (source_zone=f132bdb9-20a7-4073-a932-8f99da20691e) b= log_entry=obj95 op=0 op_state=1 2023-09-22T21:49:28.696 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.688+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj95]: bucket sync: sync obj: f132bdb9-20a7-4073-a932-8f99da20691e/:ffzdqu-59[f132bdb9-20a7-4073-a932-8f99da20691e.5729.24])/obj95[0] 2023-09-22T21:49:28.696 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.692+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj51]: success 2023-09-22T21:49:28.698 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.696+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj58]: success 2023-09-22T21:49:28.698 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.696+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj57]: success 2023-09-22T21:49:28.698 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.696+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj62]: success 2023-09-22T21:49:28.698 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.696+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj68]: success 2023-09-22T21:49:28.699 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.696+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: updating marker oid=bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24 marker=obj68 2023-09-22T21:49:28.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.700+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj95]: success 2023-09-22T21:49:28.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.700+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj69]: success 2023-09-22T21:49:28.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.700+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj92]: success 2023-09-22T21:49:28.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.700+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj93]: success 2023-09-22T21:49:28.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.700+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj85]: success 2023-09-22T21:49:28.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.704+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj74]: success 2023-09-22T21:49:28.705 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.704+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj87]: success 2023-09-22T21:49:28.705 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.704+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:entry[obj76]: success 2023-09-22T21:49:28.705 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.704+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: updating marker oid=bucket.full-sync-status.f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24 marker=obj95 2023-09-22T21:49:28.708 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.708+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: set bucket state=incremental 2023-09-22T21:49:28.712 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.712+0000 7f7d302aea40 5 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:full_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: bucket status objv={r=_2yDf8SiyyhM6jV-dA8WVUn_:151,w=:0} 2023-09-22T21:49:28.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.712+0000 7f7d302aea40 10 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: WARNING: requested sync of past generation 0 < 6, returning success 2023-09-22T21:49:28.720 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.720+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.720 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.720+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.720 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.720+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.721 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.720+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.721 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.720+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: sync status for source bucket shard: 2 2023-09-22T21:49:28.721 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.720+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:0]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.722 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.720+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.722 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.720+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:1]: sync status for source bucket shard: 2 2023-09-22T21:49:28.722 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.720+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:4]: sync status for source bucket shard: 2 2023-09-22T21:49:28.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.720+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:1]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:1]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.720+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:4]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:4]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.720+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.724 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.720+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.724 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.720+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:2]: sync status for source bucket shard: 2 2023-09-22T21:49:28.724 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.720+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:2]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:2]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.724+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:6]: sync status for source bucket shard: 2 2023-09-22T21:49:28.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.724+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.724+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.724+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.724+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:6]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:6]: listing bilog for incremental sync; position=00000000002.116.6 2023-09-22T21:49:28.727 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.724+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:7]: sync status for source bucket shard: 2 2023-09-22T21:49:28.727 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.724+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:3]: sync status for source bucket shard: 2 2023-09-22T21:49:28.727 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.724+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:7]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:7]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.727 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.724+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:3]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:3]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.728 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.724+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.729 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.728+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:10]: sync status for source bucket shard: 2 2023-09-22T21:49:28.729 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.728+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.729 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.728+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.729 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.728+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.729 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.728+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:10]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:10]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.730 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.728+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.730 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.728+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:5]: sync status for source bucket shard: 2 2023-09-22T21:49:28.730 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.728+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:5]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:5]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.731 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.728+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:8]: sync status for source bucket shard: 2 2023-09-22T21:49:28.731 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.728+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:9]: sync status for source bucket shard: 2 2023-09-22T21:49:28.731 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.728+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:8]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:8]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.731 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.728+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:9]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:9]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.728+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.728+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:13]: sync status for source bucket shard: 2 2023-09-22T21:49:28.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.728+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:13]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:13]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.728+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:14]: sync status for source bucket shard: 2 2023-09-22T21:49:28.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.732+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:14]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:14]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.732+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.732+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.732+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:11]: sync status for source bucket shard: 2 2023-09-22T21:49:28.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.732+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:11]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:11]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.732+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:12]: sync status for source bucket shard: 2 2023-09-22T21:49:28.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.732+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:12]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:12]: listing bilog for incremental sync; position=00000000002.92.6 2023-09-22T21:49:28.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.736+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:17]: sync status for source bucket shard: 2 2023-09-22T21:49:28.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.736+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:17]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:17]: listing bilog for incremental sync; position=00000000002.76.6 2023-09-22T21:49:28.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.736+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.736+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.736+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.736+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.736+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:16]: sync status for source bucket shard: 2 2023-09-22T21:49:28.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.736+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:16]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:16]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.736+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.739 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.736+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:20]: sync status for source bucket shard: 2 2023-09-22T21:49:28.739 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.736+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.739 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.736+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:20]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:20]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.739 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.736+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.739 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.736+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:22]: sync status for source bucket shard: 2 2023-09-22T21:49:28.739 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.736+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:18]: sync status for source bucket shard: 2 2023-09-22T21:49:28.740 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.736+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:22]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:22]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.740 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.736+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:18]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:18]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.741 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.736+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:21]: sync status for source bucket shard: 2 2023-09-22T21:49:28.741 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.740+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:21]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:21]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.741 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.740+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.741 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.740+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:23]: sync status for source bucket shard: 2 2023-09-22T21:49:28.741 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.740+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:23]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:23]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.742 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.740+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.742 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.740+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.742 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.740+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.742 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.740+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.743 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.740+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:25]: sync status for source bucket shard: 2 2023-09-22T21:49:28.743 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.740+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:24]: sync status for source bucket shard: 2 2023-09-22T21:49:28.743 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.740+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:25]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:25]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.743 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.740+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:24]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:24]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.744 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.744 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.745 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.745 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:29]: sync status for source bucket shard: 2 2023-09-22T21:49:28.745 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:26]: sync status for source bucket shard: 2 2023-09-22T21:49:28.745 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:27]: sync status for source bucket shard: 2 2023-09-22T21:49:28.745 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:29]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:29]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.745 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:26]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:26]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.745 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:27]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:27]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.745 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.746 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.746 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.746 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:28]: sync status for source bucket shard: 2 2023-09-22T21:49:28.746 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:30]: sync status for source bucket shard: 2 2023-09-22T21:49:28.746 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:32]: sync status for source bucket shard: 2 2023-09-22T21:49:28.747 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:28]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:28]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.747 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:30]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:30]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.747 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:32]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:32]: listing bilog for incremental sync; position=00000000001.79.6 2023-09-22T21:49:28.748 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:31]: sync status for source bucket shard: 2 2023-09-22T21:49:28.748 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:31]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:31]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.748 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-22T21:49:28.749 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:34]: sync status for source bucket shard: 2 2023-09-22T21:49:28.749 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:34]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:34]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.749 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:15]: sync status for source bucket shard: 2 2023-09-22T21:49:28.749 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:15]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:15]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.749 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:33]: sync status for source bucket shard: 2 2023-09-22T21:49:28.749 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.744+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:19]: sync status for source bucket shard: 2 2023-09-22T21:49:28.749 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.748+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:33]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:33]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.749 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.748+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:19]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:19]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.750 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.748+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:35]: sync status for source bucket shard: 2 2023-09-22T21:49:28.750 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.748+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:35]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:35]: listing bilog for incremental sync; position= 2023-09-22T21:49:28.750 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.748+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:36]: sync status for source bucket shard: 2 2023-09-22T21:49:28.750 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:28.748+0000 7f7d302aea40 20 RGW-SYNC:bucket[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24<-ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:36]:inc_sync[ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24:36]: listing bilog for incremental sync; position=00000000001.114.6 2023-09-22T21:49:28.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:49:28.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:49:28.873 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:49:28.961 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:28.961 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:29.021 INFO:teuthology.orchestra.run.smithi160.stdout:[{"name":"obj11","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:36:31.700091Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.12363723853317213104","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj111","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:47:16.468300Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.10775254410936052709","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj115","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:40:21.539635Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.3019691625852512994","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj116","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:43:12.174400Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.13606801312311740650","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj124","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:45:13.728341Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.1910571590078555770","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj145","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:43:12.226320Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.257031655597253237","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj148","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:47:16.487798Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.7268404783358681373","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj155","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:45:13.783971Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.16495487761126606791","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj17","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:36:48.678004Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.2114338151981654649","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj185","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:47:16.544730Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.8077749194256209450","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj19","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:38:31.615370Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.2476747126528959952","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj22","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:36:31.712430Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.976569876042823191","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj23","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:40:21.400499Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.17528387093750174838","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj29","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:43:12.137994Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.18117860557671523398","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj31","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:45:13.646846Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.414552692897346256","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj33","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:36:31.769832Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.9912488319201322520","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj34","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:36:48.689308Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.14203132595590134555","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj37","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:47:16.399903Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.8038003150642126383","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj38","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:38:31.671148Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.11570793449813111911","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj44","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:36:31.787636Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.818104836198532302","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj46","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:40:21.413515Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.7309291495460947597","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj51","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:36:48.744226Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.4368120013164008883","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj55","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:36:31.838216Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.3819500922135588014","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj57","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:38:31.686153Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.11534772636580058268","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj58","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:43:12.150080Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.13915817215022895514","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj62","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:45:13.660030Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.11768540602306072500","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj68","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:36:48.760922Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.15518292049764251269","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj69","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:40:21.465384Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.7299390023711697293","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj74","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:47:16.413955Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.1172025735747317358","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj76","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:38:31.739068Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.1006900334917667181","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj85","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:36:48.813905Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.17300703406980465861","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj87","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:43:12.162489Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.5626523695409935735","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj92","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:40:21.484079Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.12019081983248844955","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj93","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:45:13.713251Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.10737576657220895185","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj95","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:38:31.759207Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"f132bdb9-20a7-4073-a932-8f99da20691e.5727.16901934387143548317","flags":0,"pending_map":[],"versioned_epoch":0}] 2023-09-22T21:49:29.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:49:29.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:49:29.023 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:49:29.059 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:29.060 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:29.115 INFO:teuthology.orchestra.run.smithi160.stdout:[{"name":"obj11","instance":"","ver":{"pool":13,"epoch":9},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:36:31.700091Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_RJNojFL0AEcQ56XtrnFF1IpxqM4Kv7i","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj111","instance":"","ver":{"pool":13,"epoch":17},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:47:16.468300Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_GAOk2PzmRdgVrbfZsGAaz31EuSlj7A3","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj115","instance":"","ver":{"pool":13,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:40:21.539635Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_A07EOVgz2R710s-en74sH7I4xVq9TJE","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj116","instance":"","ver":{"pool":13,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:43:12.174400Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_cANNlny-XjEXLYwXmxP35AUlf0ArzPW","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj124","instance":"","ver":{"pool":13,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:45:13.728341Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_kbm8c8a87_C5jpMbwjNVcKnjsAi_U9U","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj145","instance":"","ver":{"pool":13,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:43:12.226320Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_4kft1U3iX-tvFlpgP6NCqPIm9PQsm2p","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj148","instance":"","ver":{"pool":13,"epoch":11},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:47:16.487798Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_zhvsT83jm3M2OLVPASfnExzwzYh4IWC","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj155","instance":"","ver":{"pool":13,"epoch":12},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:45:13.783971Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_LlsJqVbdULxpVqBlwaMruX4m0C7AgDQ","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj17","instance":"","ver":{"pool":13,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:36:48.678004Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_xszvkHEg8r1mg6mgjASFut18jgnrMDj","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj185","instance":"","ver":{"pool":13,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:47:16.544730Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_BKPh3N97e8ZGzUNR5TCFRB9M9eOLky3","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj19","instance":"","ver":{"pool":13,"epoch":16},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:38:31.615370Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_gFRAFvedkncY1dy5KO02F4GKrciekUd","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj22","instance":"","ver":{"pool":13,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:36:31.712430Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_Te-mQqiTDY3-4jXZhiRGkzNSz0gcxvk","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj23","instance":"","ver":{"pool":13,"epoch":10},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:40:21.400499Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_TpMSJx_BID4eIT25SaLvF0E72DGMLml","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj29","instance":"","ver":{"pool":13,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:43:12.137994Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_F_M7qPPZaP0hQNvOfiQQz3cfTe14uFL","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj31","instance":"","ver":{"pool":13,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:45:13.646846Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_8qI_nOgJZ1_0tkM-RGm85XBxEAuCwBU","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj33","instance":"","ver":{"pool":13,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:36:31.769832Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_N56QvZ6SxoOVWsDA1G4a7rwTx0VHqjb","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj34","instance":"","ver":{"pool":13,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:36:48.689308Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_8rd2_dC5TutLJBUiHhozal7Ma6VnECj","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj37","instance":"","ver":{"pool":13,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:47:16.399903Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_9Zk4a_JrvNKJlPf16rue5n1je0vb87D","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj38","instance":"","ver":{"pool":13,"epoch":18},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:38:31.671148Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_nDYO1Vtg8codeCz0anPKcqJnZSXq8Oi","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj44","instance":"","ver":{"pool":13,"epoch":13},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:36:31.787636Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_CFdSqCoNMc_j-dwSjvV3Mi6EWWWSb3M","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj46","instance":"","ver":{"pool":13,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:40:21.413515Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_chnkPQi7sVSVD-sN-GD4lJnhsm91ewT","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj51","instance":"","ver":{"pool":13,"epoch":9},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:36:48.744226Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_Kwex2OMl9tzYOUO4U7qRlnb3J_Tzp8L","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj55","instance":"","ver":{"pool":13,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:36:31.838216Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_-ifvZwwjW35SVikY9ekZsSX4RAnGeYn","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj57","instance":"","ver":{"pool":13,"epoch":9},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:38:31.686153Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_HQcTm7rxQiA1uX72Tg5uwWiMpE-ouTb","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj58","instance":"","ver":{"pool":13,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2023-09-22T21:43:12.150080Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_VSwAXTbs8AAcewNesvGrMNrxQ1mYzV5","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj62","instance":"","ver":{"pool":13,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:45:13.660030Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_5__EEOBuYNjFOZBDCVcjaSMySvzBzgV","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj68","instance":"","ver":{"pool":13,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:36:48.760922Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_CSWD2NT7_oIInMdX4-6ReoZ9QpJMkzH","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj69","instance":"","ver":{"pool":13,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:40:21.465384Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_zxuTf4SmQ0OoBZT6MM35EO9PHxe9vCH","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj74","instance":"","ver":{"pool":13,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:47:16.413955Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_9I3Bf3gUXXRsNq9Voke-JSciwWylGqf","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj76","instance":"","ver":{"pool":13,"epoch":10},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:38:31.739068Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_bVER7BvMh3pwrbvdWQdqWVO_9Ug4uTJ","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj85","instance":"","ver":{"pool":13,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:36:48.813905Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_RK1aofnPdOKk_d1X2fd9l9BmI1-11eH","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj87","instance":"","ver":{"pool":13,"epoch":9},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2023-09-22T21:43:12.162489Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_IAt8iJD6951eT2nZmEgKmh0QEvCH_VY","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj92","instance":"","ver":{"pool":13,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:40:21.484079Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"__G84-rkSY7WyTAsQJlRsmBIy0Ej54pw","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj93","instance":"","ver":{"pool":13,"epoch":14},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2023-09-22T21:45:13.713251Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_IAiNDFsK42GaG9kQVnno3WdgKkCii0x","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj95","instance":"","ver":{"pool":13,"epoch":10},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2023-09-22T21:38:31.759207Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"rgw-multisite-test-user","owner_display_name":"Multisite Test User","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_N__2_SCc5oyZne1FWrTbfN4gm0drzmd","flags":0,"pending_map":[],"versioned_epoch":0}] 2023-09-22T21:49:29.116 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-22T21:49:29.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-22T21:49:29.161 INFO:tasks.rgw.c1.client.1:Started 2023-09-22T21:49:29.162 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:29.173 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T21:49:29.173 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:49:29.173 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:49:29.173 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T21:49:29.174 INFO:teuthology.orchestra.run.smithi160.stderr:curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T21:49:30.175 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:30.187 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:49:30.187 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:49:30.188 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 138k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-22T21:49:30.189 INFO:teuthology.orchestra.run.smithi160.stdout:anonymous 2023-09-22T21:49:30.189 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-59 2023-09-22T21:49:30.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:49:30.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:49:30.190 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:49:30.285 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:30.286 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:30.335 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:30.332+0000 7ff4920f5a40 1 bucket sync caught up with source: 2023-09-22T21:49:30.335 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , 00000000002.76.6, , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:49:30.335 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.116.6, , , , , , 00000000002.92.6, , , , , 00000000002.76.6, , , , , , , , , , , , , , , 00000000001.79.6, , , , 00000000001.114.6] 2023-09-22T21:49:30.335 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:30.332+0000 7ff4920f5a40 0 bucket checkpoint complete 2023-09-22T21:49:30.352 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-59 2023-09-22T21:49:30.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:49:30.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:49:30.352 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:49:30.395 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:30.395 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:30.439 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:30.436+0000 7f844dfd3a40 1 waiting for incremental sync to catch up: 2023-09-22T21:49:30.439 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000009.334.6, 00000000002.265.6, 00000000006.173.6, 00000000015.317.6, 00000000012.455.6, 00000000001.415.6, 00000000003.241.6, 00000000012.176.6, 00000000021.211.6, 00000000006.329.6] 2023-09-22T21:49:30.439 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000009.334.6, 00000000006.290.6, 00000000006.173.6, 00000000015.317.6, 00000000012.455.6, 00000000003.451.6, 00000000003.241.6, 00000000012.176.6, 00000000021.211.6, 00000000018.355.6] 2023-09-22T21:49:35.441 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:35.440+0000 7f844dfd3a40 1 bucket sync caught up with source: 2023-09-22T21:49:35.441 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000009.334.6, 00000000006.290.6, 00000000006.173.6, 00000000015.317.6, 00000000012.455.6, 00000000003.451.6, 00000000003.241.6, 00000000012.176.6, 00000000021.211.6, 00000000018.355.6] 2023-09-22T21:49:35.441 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000009.334.6, 00000000006.290.6, 00000000006.173.6, 00000000015.317.6, 00000000012.455.6, 00000000003.451.6, 00000000003.241.6, 00000000012.176.6, 00000000021.211.6, 00000000018.355.6] 2023-09-22T21:49:35.441 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:35.440+0000 7f844dfd3a40 0 bucket checkpoint complete 2023-09-22T21:49:35.478 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-59 zones={a2, a1} 2023-09-22T21:49:35.478 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:35.479 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:35.479 DEBUG:boto:Method: HEAD 2023-09-22T21:49:35.479 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:49:35.479 DEBUG:boto:Data: 2023-09-22T21:49:35.479 DEBUG:boto:Headers: {} 2023-09-22T21:49:35.479 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:35.479 DEBUG:boto:Port: 8001 2023-09-22T21:49:35.479 DEBUG:boto:Params: {} 2023-09-22T21:49:35.480 DEBUG:boto:Token: None 2023-09-22T21:49:35.480 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:35 GMT /ffzdqu-59/ 2023-09-22T21:49:35.480 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Sn9sWhUgEbB25STeuvQZhZhWqgQ= 2023-09-22T21:49:35.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Sn9sWhUgEbB25STeuvQZhZhWqgQ=', 'Content-Length': '0'} 2023-09-22T21:49:35.481 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2023-09-22T21:49:35.481 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:49:36.271 DEBUG:boto:Token: None 2023-09-22T21:49:36.271 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:35 GMT /ffzdqu-59/ 2023-09-22T21:49:36.272 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Sn9sWhUgEbB25STeuvQZhZhWqgQ= 2023-09-22T21:49:36.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:35 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Sn9sWhUgEbB25STeuvQZhZhWqgQ=', 'Content-Length': '0'} 2023-09-22T21:49:36.282 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1e0770a27cf3a1f-00650e0bf0-1d9d-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.283 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:36.283 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:36.283 DEBUG:boto:Method: HEAD 2023-09-22T21:49:36.283 DEBUG:boto:Path: /ffzdqu-59/ 2023-09-22T21:49:36.283 DEBUG:boto:Data: 2023-09-22T21:49:36.283 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.283 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.283 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.283 DEBUG:boto:Params: {} 2023-09-22T21:49:36.284 DEBUG:boto:Token: None 2023-09-22T21:49:36.284 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/ 2023-09-22T21:49:36.284 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ygRH5P/DRHupzfY4ssC7hIfDoLc= 2023-09-22T21:49:36.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ygRH5P/DRHupzfY4ssC7hIfDoLc=', 'Content-Length': '0'} 2023-09-22T21:49:36.294 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b35e2b57896912f7-00650e0bf0-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.294 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:49:36.294 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:36.295 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:36.295 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:36.295 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:36.295 DEBUG:boto:Method: GET 2023-09-22T21:49:36.295 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:36.295 DEBUG:boto:Data: 2023-09-22T21:49:36.295 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.295 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.296 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.296 DEBUG:boto:Params: {} 2023-09-22T21:49:36.296 DEBUG:boto:Token: None 2023-09-22T21:49:36.296 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/?versions 2023-09-22T21:49:36.296 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gwtJ7pfJV8kLPOutt8La5oh4vX0= 2023-09-22T21:49:36.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gwtJ7pfJV8kLPOutt8La5oh4vX0=', 'Content-Length': '0'} 2023-09-22T21:49:36.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2be69ea1b4126c4-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.303 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-22T21:47:16.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-22T21:47:16.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-22T21:47:16.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-22T21:47:16.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:36.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T21:49:36.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-09-22T21:49:36.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-22T21:49:36.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-22T21:49:36.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-09-22T21:49:36.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-22T21:49:36.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-09-22T21:49:36.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-09-22T21:49:36.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-22T21:49:36.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-09-22T21:49:36.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-22T21:49:36.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T21:49:36.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-22T21:49:36.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-22T21:49:36.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-09-22T21:49:36.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T21:49:36.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-22T21:49:36.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-09-22T21:49:36.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-22T21:49:36.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T21:49:36.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-22T21:49:36.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-22T21:49:36.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T21:49:36.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-22T21:49:36.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-22T21:49:36.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-09-22T21:49:36.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-22T21:49:36.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-22T21:49:36.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-09-22T21:49:36.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-22T21:49:36.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-22T21:49:36.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-22T21:49:36.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-22T21:49:36.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-09-22T21:49:36.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-22T21:49:36.309 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:49:36.309 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:36.309 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:36.309 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:36.309 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:36.309 DEBUG:boto:Method: GET 2023-09-22T21:49:36.309 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:36.309 DEBUG:boto:Data: 2023-09-22T21:49:36.309 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.309 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.310 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.310 DEBUG:boto:Params: {} 2023-09-22T21:49:36.310 DEBUG:boto:Token: None 2023-09-22T21:49:36.310 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/?versions 2023-09-22T21:49:36.310 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gwtJ7pfJV8kLPOutt8La5oh4vX0= 2023-09-22T21:49:36.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gwtJ7pfJV8kLPOutt8La5oh4vX0=', 'Content-Length': '0'} 2023-09-22T21:49:36.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002343131084259bbc-00650e0bf0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.320 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-22T21:47:16.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-22T21:47:16.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-22T21:47:16.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-22T21:47:16.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:36.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T21:49:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-09-22T21:49:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-22T21:49:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-22T21:49:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-09-22T21:49:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-22T21:49:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-09-22T21:49:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-09-22T21:49:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-22T21:49:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-09-22T21:49:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-22T21:49:36.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T21:49:36.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-22T21:49:36.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-22T21:49:36.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-09-22T21:49:36.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T21:49:36.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-22T21:49:36.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-09-22T21:49:36.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-22T21:49:36.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T21:49:36.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-22T21:49:36.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-22T21:49:36.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T21:49:36.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-22T21:49:36.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-22T21:49:36.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-09-22T21:49:36.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-22T21:49:36.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-22T21:49:36.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-09-22T21:49:36.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-22T21:49:36.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-22T21:49:36.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-22T21:49:36.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-22T21:49:36.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-09-22T21:49:36.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-22T21:49:36.325 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:36.325 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:36.325 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:36.325 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:36.325 DEBUG:boto:Method: GET 2023-09-22T21:49:36.325 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:36.325 DEBUG:boto:Data: 2023-09-22T21:49:36.326 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.326 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.326 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.326 DEBUG:boto:Params: {} 2023-09-22T21:49:36.326 DEBUG:boto:Token: None 2023-09-22T21:49:36.326 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/?versions 2023-09-22T21:49:36.326 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gwtJ7pfJV8kLPOutt8La5oh4vX0= 2023-09-22T21:49:36.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gwtJ7pfJV8kLPOutt8La5oh4vX0=', 'Content-Length': '0'} 2023-09-22T21:49:36.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059b61c879f3f39ee-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.332 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-22T21:47:16.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-22T21:47:16.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-22T21:47:16.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-22T21:47:16.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:36.334 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:36.334 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:36.334 DEBUG:boto:path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:36.335 DEBUG:boto:auth_path=/ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:36.335 DEBUG:boto:Method: GET 2023-09-22T21:49:36.335 DEBUG:boto:Path: /ffzdqu-59/?versions&max-keys=999 2023-09-22T21:49:36.335 DEBUG:boto:Data: 2023-09-22T21:49:36.335 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.335 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.335 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.335 DEBUG:boto:Params: {} 2023-09-22T21:49:36.335 DEBUG:boto:Token: None 2023-09-22T21:49:36.335 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/?versions 2023-09-22T21:49:36.336 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gwtJ7pfJV8kLPOutt8La5oh4vX0= 2023-09-22T21:49:36.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gwtJ7pfJV8kLPOutt8La5oh4vX0=', 'Content-Length': '0'} 2023-09-22T21:49:36.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066e1f25e5c57daf1-00650e0bf0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.348 DEBUG:boto:b'ffzdqu-59999falseobj11nulltrue2023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-22T21:47:16.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-22T21:47:16.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-22T21:47:16.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-22T21:47:16.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:36.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:49:36.350 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:49:36.350 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:49:36.350 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.351 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.351 DEBUG:boto:Method: GET 2023-09-22T21:49:36.351 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.351 DEBUG:boto:Data: 2023-09-22T21:49:36.351 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.351 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.351 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.351 DEBUG:boto:Params: {} 2023-09-22T21:49:36.351 DEBUG:boto:Token: None 2023-09-22T21:49:36.351 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.352 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ym/zZuTlcIjZFnBQ0iIGrHS51GI= 2023-09-22T21:49:36.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ym/zZuTlcIjZFnBQ0iIGrHS51GI=', 'Content-Length': '0'} 2023-09-22T21:49:36.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8bdc51262b72fb7-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.398 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:49:36.398 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:49:36.399 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.399 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.399 DEBUG:boto:Method: GET 2023-09-22T21:49:36.399 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.399 DEBUG:boto:Data: 2023-09-22T21:49:36.399 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.399 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.399 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.399 DEBUG:boto:Params: {} 2023-09-22T21:49:36.400 DEBUG:boto:Token: None 2023-09-22T21:49:36.400 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.400 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ym/zZuTlcIjZFnBQ0iIGrHS51GI= 2023-09-22T21:49:36.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ym/zZuTlcIjZFnBQ0iIGrHS51GI=', 'Content-Length': '0'} 2023-09-22T21:49:36.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a38ecec35b184252-00650e0bf0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.403 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:49:36.404 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:49:36.404 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.404 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.404 DEBUG:boto:Method: HEAD 2023-09-22T21:49:36.404 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.404 DEBUG:boto:Data: 2023-09-22T21:49:36.404 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.404 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.404 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.405 DEBUG:boto:Params: {} 2023-09-22T21:49:36.405 DEBUG:boto:Token: None 2023-09-22T21:49:36.405 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.405 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VwMFEqiBTROJWMRp5qB9UvEn/EI= 2023-09-22T21:49:36.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VwMFEqiBTROJWMRp5qB9UvEn/EI=', 'Content-Length': '0'} 2023-09-22T21:49:36.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e60d46914af65705-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.408 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:49:36.408 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:49:36.408 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.409 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.409 DEBUG:boto:Method: HEAD 2023-09-22T21:49:36.409 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.409 DEBUG:boto:Data: 2023-09-22T21:49:36.409 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.409 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.409 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.409 DEBUG:boto:Params: {} 2023-09-22T21:49:36.410 DEBUG:boto:Token: None 2023-09-22T21:49:36.410 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.410 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VwMFEqiBTROJWMRp5qB9UvEn/EI= 2023-09-22T21:49:36.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VwMFEqiBTROJWMRp5qB9UvEn/EI=', 'Content-Length': '0'} 2023-09-22T21:49:36.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da47fb97ee90f253-00650e0bf0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:49:36.413 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:49:36.413 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:49:36.414 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.414 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.414 DEBUG:boto:Method: GET 2023-09-22T21:49:36.414 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.414 DEBUG:boto:Data: 2023-09-22T21:49:36.414 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.414 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.414 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.414 DEBUG:boto:Params: {} 2023-09-22T21:49:36.415 DEBUG:boto:Token: None 2023-09-22T21:49:36.415 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.415 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ym/zZuTlcIjZFnBQ0iIGrHS51GI= 2023-09-22T21:49:36.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ym/zZuTlcIjZFnBQ0iIGrHS51GI=', 'Content-Length': '0'} 2023-09-22T21:49:36.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000464d9f3c0aed0a0d-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.458 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:49:36.459 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:49:36.459 DEBUG:boto:path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.459 DEBUG:boto:auth_path=/ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.459 DEBUG:boto:Method: GET 2023-09-22T21:49:36.459 DEBUG:boto:Path: /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.459 DEBUG:boto:Data: 2023-09-22T21:49:36.459 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.459 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.459 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.459 DEBUG:boto:Params: {} 2023-09-22T21:49:36.460 DEBUG:boto:Token: None 2023-09-22T21:49:36.460 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj11?versionId=null 2023-09-22T21:49:36.460 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ym/zZuTlcIjZFnBQ0iIGrHS51GI= 2023-09-22T21:49:36.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ym/zZuTlcIjZFnBQ0iIGrHS51GI=', 'Content-Length': '0'} 2023-09-22T21:49:36.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017c4468aced5df35-00650e0bf0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.464 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:36.464 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:36.464 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:49:36.464 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:49:36.464 DEBUG:boto:Method: GET 2023-09-22T21:49:36.464 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:49:36.465 DEBUG:boto:Data: 2023-09-22T21:49:36.465 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.465 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.465 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.465 DEBUG:boto:Params: {} 2023-09-22T21:49:36.465 DEBUG:boto:Token: None 2023-09-22T21:49:36.465 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/ 2023-09-22T21:49:36.465 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U= 2023-09-22T21:49:36.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U=', 'Content-Length': '0'} 2023-09-22T21:49:36.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000635ed23e3f002e17-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.470 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:36.471 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:36.471 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:36.471 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:49:36.471 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:49:36.472 DEBUG:boto:Method: GET 2023-09-22T21:49:36.472 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:49:36.472 DEBUG:boto:Data: 2023-09-22T21:49:36.472 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.472 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.472 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.472 DEBUG:boto:Params: {} 2023-09-22T21:49:36.472 DEBUG:boto:Token: None 2023-09-22T21:49:36.472 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/ 2023-09-22T21:49:36.473 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U= 2023-09-22T21:49:36.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U=', 'Content-Length': '0'} 2023-09-22T21:49:36.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000388a38c51301f53c-00650e0bf0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.483 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:36.483 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:49:36.483 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:49:36.483 DEBUG:boto:Method: HEAD 2023-09-22T21:49:36.484 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:49:36.484 DEBUG:boto:Data: 2023-09-22T21:49:36.484 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.484 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.484 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.484 DEBUG:boto:Params: {} 2023-09-22T21:49:36.484 DEBUG:boto:Token: None 2023-09-22T21:49:36.484 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj11 2023-09-22T21:49:36.484 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UAtU7r4CF7/DaquH4dNtmSHn3lU= 2023-09-22T21:49:36.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UAtU7r4CF7/DaquH4dNtmSHn3lU=', 'Content-Length': '0'} 2023-09-22T21:49:36.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce0d9b35065e602a-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.488 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:49:36.488 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:49:36.488 DEBUG:boto:Method: HEAD 2023-09-22T21:49:36.488 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:49:36.488 DEBUG:boto:Data: 2023-09-22T21:49:36.488 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.488 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.488 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.488 DEBUG:boto:Params: {} 2023-09-22T21:49:36.489 DEBUG:boto:Token: None 2023-09-22T21:49:36.489 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj11 2023-09-22T21:49:36.489 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UAtU7r4CF7/DaquH4dNtmSHn3lU= 2023-09-22T21:49:36.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UAtU7r4CF7/DaquH4dNtmSHn3lU=', 'Content-Length': '0'} 2023-09-22T21:49:36.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec11ecc8695ccfbd-00650e0bf0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:49:36.492 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:49:36.492 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:49:36.492 DEBUG:boto:Method: GET 2023-09-22T21:49:36.492 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:49:36.492 DEBUG:boto:Data: 2023-09-22T21:49:36.492 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.493 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.493 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.493 DEBUG:boto:Params: {} 2023-09-22T21:49:36.493 DEBUG:boto:Token: None 2023-09-22T21:49:36.493 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj11 2023-09-22T21:49:36.493 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7rVy79PxB6qOPWWVV/TiIAZ+lYE= 2023-09-22T21:49:36.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7rVy79PxB6qOPWWVV/TiIAZ+lYE=', 'Content-Length': '0'} 2023-09-22T21:49:36.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000833675834be252ff-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.542 DEBUG:boto:path=/ffzdqu-59/obj11 2023-09-22T21:49:36.543 DEBUG:boto:auth_path=/ffzdqu-59/obj11 2023-09-22T21:49:36.543 DEBUG:boto:Method: GET 2023-09-22T21:49:36.543 DEBUG:boto:Path: /ffzdqu-59/obj11 2023-09-22T21:49:36.543 DEBUG:boto:Data: 2023-09-22T21:49:36.543 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.543 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.543 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.544 DEBUG:boto:Params: {} 2023-09-22T21:49:36.544 DEBUG:boto:Token: None 2023-09-22T21:49:36.544 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj11 2023-09-22T21:49:36.544 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7rVy79PxB6qOPWWVV/TiIAZ+lYE= 2023-09-22T21:49:36.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7rVy79PxB6qOPWWVV/TiIAZ+lYE=', 'Content-Length': '0'} 2023-09-22T21:49:36.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000923b9fe6c811cf50-00650e0bf0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.548 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:36.548 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:36.548 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:49:36.548 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:49:36.548 DEBUG:boto:Method: GET 2023-09-22T21:49:36.548 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:49:36.549 DEBUG:boto:Data: 2023-09-22T21:49:36.549 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.549 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.549 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.549 DEBUG:boto:Params: {} 2023-09-22T21:49:36.549 DEBUG:boto:Token: None 2023-09-22T21:49:36.549 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/ 2023-09-22T21:49:36.549 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U= 2023-09-22T21:49:36.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U=', 'Content-Length': '0'} 2023-09-22T21:49:36.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f32816104ce49dd2-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.555 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:36.555 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:36.555 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:36.555 DEBUG:boto:path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:49:36.556 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj11 2023-09-22T21:49:36.556 DEBUG:boto:Method: GET 2023-09-22T21:49:36.556 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj11 2023-09-22T21:49:36.556 DEBUG:boto:Data: 2023-09-22T21:49:36.556 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.556 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.556 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.556 DEBUG:boto:Params: {} 2023-09-22T21:49:36.556 DEBUG:boto:Token: None 2023-09-22T21:49:36.556 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/ 2023-09-22T21:49:36.557 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U= 2023-09-22T21:49:36.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U=', 'Content-Length': '0'} 2023-09-22T21:49:36.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000764b85ac573daaf2-00650e0bf0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.567 DEBUG:boto:b'ffzdqu-59obj111000falseobj112023-09-22T21:36:31.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:36.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-09-22T21:49:36.567 DEBUG:boto:path=/ffzdqu-59/obj111 2023-09-22T21:49:36.568 DEBUG:boto:auth_path=/ffzdqu-59/obj111 2023-09-22T21:49:36.568 DEBUG:boto:path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.568 DEBUG:boto:auth_path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.568 DEBUG:boto:Method: GET 2023-09-22T21:49:36.568 DEBUG:boto:Path: /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.568 DEBUG:boto:Data: 2023-09-22T21:49:36.568 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.568 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.568 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.568 DEBUG:boto:Params: {} 2023-09-22T21:49:36.569 DEBUG:boto:Token: None 2023-09-22T21:49:36.569 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.569 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dzlGQ6n9g/ouwYZ8IYwBk15EgGs= 2023-09-22T21:49:36.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dzlGQ6n9g/ouwYZ8IYwBk15EgGs=', 'Content-Length': '0'} 2023-09-22T21:49:36.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e490657df5b08d01-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.573 DEBUG:boto:path=/ffzdqu-59/obj111 2023-09-22T21:49:36.573 DEBUG:boto:auth_path=/ffzdqu-59/obj111 2023-09-22T21:49:36.573 DEBUG:boto:path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.573 DEBUG:boto:auth_path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.573 DEBUG:boto:Method: GET 2023-09-22T21:49:36.573 DEBUG:boto:Path: /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.573 DEBUG:boto:Data: 2023-09-22T21:49:36.573 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.574 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.574 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.574 DEBUG:boto:Params: {} 2023-09-22T21:49:36.574 DEBUG:boto:Token: None 2023-09-22T21:49:36.574 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.574 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dzlGQ6n9g/ouwYZ8IYwBk15EgGs= 2023-09-22T21:49:36.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dzlGQ6n9g/ouwYZ8IYwBk15EgGs=', 'Content-Length': '0'} 2023-09-22T21:49:36.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062065b401689cd4a-00650e0bf0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.578 DEBUG:boto:path=/ffzdqu-59/obj111 2023-09-22T21:49:36.578 DEBUG:boto:auth_path=/ffzdqu-59/obj111 2023-09-22T21:49:36.578 DEBUG:boto:path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.578 DEBUG:boto:auth_path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.578 DEBUG:boto:Method: HEAD 2023-09-22T21:49:36.578 DEBUG:boto:Path: /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.579 DEBUG:boto:Data: 2023-09-22T21:49:36.579 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.579 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.579 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.579 DEBUG:boto:Params: {} 2023-09-22T21:49:36.579 DEBUG:boto:Token: None 2023-09-22T21:49:36.579 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.579 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jGKLWS4fOYEFH//OTqaj5V/QAXs= 2023-09-22T21:49:36.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jGKLWS4fOYEFH//OTqaj5V/QAXs=', 'Content-Length': '0'} 2023-09-22T21:49:36.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bda26a2aed5f8ee-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.582 DEBUG:boto:path=/ffzdqu-59/obj111 2023-09-22T21:49:36.583 DEBUG:boto:auth_path=/ffzdqu-59/obj111 2023-09-22T21:49:36.583 DEBUG:boto:path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.583 DEBUG:boto:auth_path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.583 DEBUG:boto:Method: HEAD 2023-09-22T21:49:36.583 DEBUG:boto:Path: /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.583 DEBUG:boto:Data: 2023-09-22T21:49:36.583 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.583 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.583 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.583 DEBUG:boto:Params: {} 2023-09-22T21:49:36.584 DEBUG:boto:Token: None 2023-09-22T21:49:36.584 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.584 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jGKLWS4fOYEFH//OTqaj5V/QAXs= 2023-09-22T21:49:36.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jGKLWS4fOYEFH//OTqaj5V/QAXs=', 'Content-Length': '0'} 2023-09-22T21:49:36.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e94ebd6cbbcfb3b-00650e0bf0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-09-22T21:49:36.587 DEBUG:boto:path=/ffzdqu-59/obj111 2023-09-22T21:49:36.587 DEBUG:boto:auth_path=/ffzdqu-59/obj111 2023-09-22T21:49:36.587 DEBUG:boto:path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.587 DEBUG:boto:auth_path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.587 DEBUG:boto:Method: GET 2023-09-22T21:49:36.588 DEBUG:boto:Path: /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.588 DEBUG:boto:Data: 2023-09-22T21:49:36.588 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.588 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.588 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.588 DEBUG:boto:Params: {} 2023-09-22T21:49:36.588 DEBUG:boto:Token: None 2023-09-22T21:49:36.588 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.588 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dzlGQ6n9g/ouwYZ8IYwBk15EgGs= 2023-09-22T21:49:36.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dzlGQ6n9g/ouwYZ8IYwBk15EgGs=', 'Content-Length': '0'} 2023-09-22T21:49:36.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000004a1dc082a65d17-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.634 DEBUG:boto:path=/ffzdqu-59/obj111 2023-09-22T21:49:36.634 DEBUG:boto:auth_path=/ffzdqu-59/obj111 2023-09-22T21:49:36.635 DEBUG:boto:path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.635 DEBUG:boto:auth_path=/ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.635 DEBUG:boto:Method: GET 2023-09-22T21:49:36.635 DEBUG:boto:Path: /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.635 DEBUG:boto:Data: 2023-09-22T21:49:36.635 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.635 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.635 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.635 DEBUG:boto:Params: {} 2023-09-22T21:49:36.636 DEBUG:boto:Token: None 2023-09-22T21:49:36.636 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj111?versionId=null 2023-09-22T21:49:36.636 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dzlGQ6n9g/ouwYZ8IYwBk15EgGs= 2023-09-22T21:49:36.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dzlGQ6n9g/ouwYZ8IYwBk15EgGs=', 'Content-Length': '0'} 2023-09-22T21:49:36.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000185c8ea1e7be7b77-00650e0bf0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.639 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:36.639 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:36.639 DEBUG:boto:path=/ffzdqu-59/?prefix=obj111 2023-09-22T21:49:36.639 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj111 2023-09-22T21:49:36.639 DEBUG:boto:Method: GET 2023-09-22T21:49:36.640 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj111 2023-09-22T21:49:36.640 DEBUG:boto:Data: 2023-09-22T21:49:36.640 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.640 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.640 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.640 DEBUG:boto:Params: {} 2023-09-22T21:49:36.640 DEBUG:boto:Token: None 2023-09-22T21:49:36.640 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/ 2023-09-22T21:49:36.641 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U= 2023-09-22T21:49:36.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U=', 'Content-Length': '0'} 2023-09-22T21:49:36.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020c56ff2015bd6dd-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.645 DEBUG:boto:b'ffzdqu-59obj1111000falseobj1112023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:36.646 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:36.646 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:36.646 DEBUG:boto:path=/ffzdqu-59/?prefix=obj111 2023-09-22T21:49:36.646 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj111 2023-09-22T21:49:36.646 DEBUG:boto:Method: GET 2023-09-22T21:49:36.646 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj111 2023-09-22T21:49:36.646 DEBUG:boto:Data: 2023-09-22T21:49:36.647 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.647 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.647 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.647 DEBUG:boto:Params: {} 2023-09-22T21:49:36.647 DEBUG:boto:Token: None 2023-09-22T21:49:36.647 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/ 2023-09-22T21:49:36.647 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U= 2023-09-22T21:49:36.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U=', 'Content-Length': '0'} 2023-09-22T21:49:36.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083b887bc00cec2ff-00650e0bf0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.698 DEBUG:boto:b'ffzdqu-59obj1111000falseobj1112023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:36.699 DEBUG:boto:path=/ffzdqu-59/obj111 2023-09-22T21:49:36.699 DEBUG:boto:auth_path=/ffzdqu-59/obj111 2023-09-22T21:49:36.699 DEBUG:boto:Method: HEAD 2023-09-22T21:49:36.699 DEBUG:boto:Path: /ffzdqu-59/obj111 2023-09-22T21:49:36.699 DEBUG:boto:Data: 2023-09-22T21:49:36.699 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.699 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.700 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.700 DEBUG:boto:Params: {} 2023-09-22T21:49:36.700 DEBUG:boto:Token: None 2023-09-22T21:49:36.700 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj111 2023-09-22T21:49:36.700 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Xhxrvq5Tv0SmFfjgehrPgSM4Kog= 2023-09-22T21:49:36.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Xhxrvq5Tv0SmFfjgehrPgSM4Kog=', 'Content-Length': '0'} 2023-09-22T21:49:36.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e16b12fcf3b2863-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.703 DEBUG:boto:path=/ffzdqu-59/obj111 2023-09-22T21:49:36.704 DEBUG:boto:auth_path=/ffzdqu-59/obj111 2023-09-22T21:49:36.704 DEBUG:boto:Method: HEAD 2023-09-22T21:49:36.704 DEBUG:boto:Path: /ffzdqu-59/obj111 2023-09-22T21:49:36.704 DEBUG:boto:Data: 2023-09-22T21:49:36.704 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.704 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.705 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.705 DEBUG:boto:Params: {} 2023-09-22T21:49:36.705 DEBUG:boto:Token: None 2023-09-22T21:49:36.705 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj111 2023-09-22T21:49:36.705 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Xhxrvq5Tv0SmFfjgehrPgSM4Kog= 2023-09-22T21:49:36.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Xhxrvq5Tv0SmFfjgehrPgSM4Kog=', 'Content-Length': '0'} 2023-09-22T21:49:36.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078e37314d5a4c750-00650e0bf0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-09-22T21:49:36.708 DEBUG:boto:path=/ffzdqu-59/obj111 2023-09-22T21:49:36.708 DEBUG:boto:auth_path=/ffzdqu-59/obj111 2023-09-22T21:49:36.708 DEBUG:boto:Method: GET 2023-09-22T21:49:36.708 DEBUG:boto:Path: /ffzdqu-59/obj111 2023-09-22T21:49:36.708 DEBUG:boto:Data: 2023-09-22T21:49:36.709 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.709 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.709 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.709 DEBUG:boto:Params: {} 2023-09-22T21:49:36.709 DEBUG:boto:Token: None 2023-09-22T21:49:36.709 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj111 2023-09-22T21:49:36.709 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4de+MEw2FVVUXy45MDgUh711A70= 2023-09-22T21:49:36.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4de+MEw2FVVUXy45MDgUh711A70=', 'Content-Length': '0'} 2023-09-22T21:49:36.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbfb28035c17301d-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.758 DEBUG:boto:path=/ffzdqu-59/obj111 2023-09-22T21:49:36.758 DEBUG:boto:auth_path=/ffzdqu-59/obj111 2023-09-22T21:49:36.759 DEBUG:boto:Method: GET 2023-09-22T21:49:36.759 DEBUG:boto:Path: /ffzdqu-59/obj111 2023-09-22T21:49:36.759 DEBUG:boto:Data: 2023-09-22T21:49:36.759 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.759 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.759 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.759 DEBUG:boto:Params: {} 2023-09-22T21:49:36.759 DEBUG:boto:Token: None 2023-09-22T21:49:36.760 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj111 2023-09-22T21:49:36.760 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4de+MEw2FVVUXy45MDgUh711A70= 2023-09-22T21:49:36.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4de+MEw2FVVUXy45MDgUh711A70=', 'Content-Length': '0'} 2023-09-22T21:49:36.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d43d2e312855d75a-00650e0bf0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.763 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:36.763 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:36.763 DEBUG:boto:path=/ffzdqu-59/?prefix=obj111 2023-09-22T21:49:36.764 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj111 2023-09-22T21:49:36.764 DEBUG:boto:Method: GET 2023-09-22T21:49:36.764 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj111 2023-09-22T21:49:36.764 DEBUG:boto:Data: 2023-09-22T21:49:36.764 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.764 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.764 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.764 DEBUG:boto:Params: {} 2023-09-22T21:49:36.764 DEBUG:boto:Token: None 2023-09-22T21:49:36.765 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/ 2023-09-22T21:49:36.765 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U= 2023-09-22T21:49:36.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U=', 'Content-Length': '0'} 2023-09-22T21:49:36.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039b63e32f082d23c-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.769 DEBUG:boto:b'ffzdqu-59obj1111000falseobj1112023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:36.770 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:36.770 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:36.770 DEBUG:boto:path=/ffzdqu-59/?prefix=obj111 2023-09-22T21:49:36.770 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj111 2023-09-22T21:49:36.770 DEBUG:boto:Method: GET 2023-09-22T21:49:36.771 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj111 2023-09-22T21:49:36.771 DEBUG:boto:Data: 2023-09-22T21:49:36.771 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.771 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.771 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.771 DEBUG:boto:Params: {} 2023-09-22T21:49:36.771 DEBUG:boto:Token: None 2023-09-22T21:49:36.771 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/ 2023-09-22T21:49:36.771 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U= 2023-09-22T21:49:36.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U=', 'Content-Length': '0'} 2023-09-22T21:49:36.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000809f920255d98a0d-00650e0bf0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.781 DEBUG:boto:b'ffzdqu-59obj1111000falseobj1112023-09-22T21:47:16.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:36.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-22T21:49:36.782 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:49:36.782 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:49:36.782 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.782 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.782 DEBUG:boto:Method: GET 2023-09-22T21:49:36.782 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.783 DEBUG:boto:Data: 2023-09-22T21:49:36.783 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.783 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.783 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.783 DEBUG:boto:Params: {} 2023-09-22T21:49:36.783 DEBUG:boto:Token: None 2023-09-22T21:49:36.783 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.783 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uemizDsC5UVzcKcXZh6Ld/WWDhA= 2023-09-22T21:49:36.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uemizDsC5UVzcKcXZh6Ld/WWDhA=', 'Content-Length': '0'} 2023-09-22T21:49:36.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034ac81fc806281d0-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.788 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:49:36.788 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:49:36.788 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.788 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.788 DEBUG:boto:Method: GET 2023-09-22T21:49:36.788 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.788 DEBUG:boto:Data: 2023-09-22T21:49:36.788 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.788 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.788 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.789 DEBUG:boto:Params: {} 2023-09-22T21:49:36.789 DEBUG:boto:Token: None 2023-09-22T21:49:36.789 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.789 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uemizDsC5UVzcKcXZh6Ld/WWDhA= 2023-09-22T21:49:36.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uemizDsC5UVzcKcXZh6Ld/WWDhA=', 'Content-Length': '0'} 2023-09-22T21:49:36.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ad9991fbc4210da-00650e0bf0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.838 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:49:36.838 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:49:36.839 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.839 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.839 DEBUG:boto:Method: HEAD 2023-09-22T21:49:36.839 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.839 DEBUG:boto:Data: 2023-09-22T21:49:36.839 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.839 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.839 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.839 DEBUG:boto:Params: {} 2023-09-22T21:49:36.839 DEBUG:boto:Token: None 2023-09-22T21:49:36.840 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.840 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GnXScwWJqW9tTmtDXqZYB84YwMA= 2023-09-22T21:49:36.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GnXScwWJqW9tTmtDXqZYB84YwMA=', 'Content-Length': '0'} 2023-09-22T21:49:36.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000382c4c5767df94b5-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.843 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:49:36.843 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:49:36.843 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.843 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.843 DEBUG:boto:Method: HEAD 2023-09-22T21:49:36.843 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.843 DEBUG:boto:Data: 2023-09-22T21:49:36.843 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.844 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.844 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.844 DEBUG:boto:Params: {} 2023-09-22T21:49:36.844 DEBUG:boto:Token: None 2023-09-22T21:49:36.844 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.844 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GnXScwWJqW9tTmtDXqZYB84YwMA= 2023-09-22T21:49:36.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GnXScwWJqW9tTmtDXqZYB84YwMA=', 'Content-Length': '0'} 2023-09-22T21:49:36.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b765916bfcd45f0-00650e0bf0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-22T21:49:36.848 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:49:36.848 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:49:36.848 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.848 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.848 DEBUG:boto:Method: GET 2023-09-22T21:49:36.848 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.848 DEBUG:boto:Data: 2023-09-22T21:49:36.848 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.848 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.848 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.848 DEBUG:boto:Params: {} 2023-09-22T21:49:36.849 DEBUG:boto:Token: None 2023-09-22T21:49:36.849 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.849 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uemizDsC5UVzcKcXZh6Ld/WWDhA= 2023-09-22T21:49:36.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uemizDsC5UVzcKcXZh6Ld/WWDhA=', 'Content-Length': '0'} 2023-09-22T21:49:36.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f97c92dcdd419f2-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.894 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:49:36.895 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:49:36.895 DEBUG:boto:path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.895 DEBUG:boto:auth_path=/ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.895 DEBUG:boto:Method: GET 2023-09-22T21:49:36.895 DEBUG:boto:Path: /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.895 DEBUG:boto:Data: 2023-09-22T21:49:36.895 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.895 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.895 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.896 DEBUG:boto:Params: {} 2023-09-22T21:49:36.896 DEBUG:boto:Token: None 2023-09-22T21:49:36.896 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj115?versionId=null 2023-09-22T21:49:36.897 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uemizDsC5UVzcKcXZh6Ld/WWDhA= 2023-09-22T21:49:36.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uemizDsC5UVzcKcXZh6Ld/WWDhA=', 'Content-Length': '0'} 2023-09-22T21:49:36.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7d5dfcd755c99d7-00650e0bf0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.901 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:36.901 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:36.901 DEBUG:boto:path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:49:36.901 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:49:36.902 DEBUG:boto:Method: GET 2023-09-22T21:49:36.902 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj115 2023-09-22T21:49:36.902 DEBUG:boto:Data: 2023-09-22T21:49:36.902 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.902 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.902 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.902 DEBUG:boto:Params: {} 2023-09-22T21:49:36.902 DEBUG:boto:Token: None 2023-09-22T21:49:36.902 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/ 2023-09-22T21:49:36.903 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U= 2023-09-22T21:49:36.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U=', 'Content-Length': '0'} 2023-09-22T21:49:36.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076928ddd44f71c65-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.908 DEBUG:boto:b'ffzdqu-59obj1151000falseobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:36.908 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:36.908 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:36.908 DEBUG:boto:path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:49:36.908 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:49:36.909 DEBUG:boto:Method: GET 2023-09-22T21:49:36.909 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj115 2023-09-22T21:49:36.909 DEBUG:boto:Data: 2023-09-22T21:49:36.909 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.909 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.909 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.909 DEBUG:boto:Params: {} 2023-09-22T21:49:36.909 DEBUG:boto:Token: None 2023-09-22T21:49:36.909 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/ 2023-09-22T21:49:36.910 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U= 2023-09-22T21:49:36.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U=', 'Content-Length': '0'} 2023-09-22T21:49:36.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8acffa188d4e783-00650e0bf0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.919 DEBUG:boto:b'ffzdqu-59obj1151000falseobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:36.920 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:49:36.920 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:49:36.920 DEBUG:boto:Method: HEAD 2023-09-22T21:49:36.920 DEBUG:boto:Path: /ffzdqu-59/obj115 2023-09-22T21:49:36.920 DEBUG:boto:Data: 2023-09-22T21:49:36.920 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.920 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.920 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.921 DEBUG:boto:Params: {} 2023-09-22T21:49:36.921 DEBUG:boto:Token: None 2023-09-22T21:49:36.921 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj115 2023-09-22T21:49:36.921 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+EsWej/TGJ0+PRQCI9pOswclwCM= 2023-09-22T21:49:36.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+EsWej/TGJ0+PRQCI9pOswclwCM=', 'Content-Length': '0'} 2023-09-22T21:49:36.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000470955fecfb56686-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.924 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:49:36.924 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:49:36.924 DEBUG:boto:Method: HEAD 2023-09-22T21:49:36.924 DEBUG:boto:Path: /ffzdqu-59/obj115 2023-09-22T21:49:36.924 DEBUG:boto:Data: 2023-09-22T21:49:36.925 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.925 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.925 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.925 DEBUG:boto:Params: {} 2023-09-22T21:49:36.925 DEBUG:boto:Token: None 2023-09-22T21:49:36.925 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj115 2023-09-22T21:49:36.925 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+EsWej/TGJ0+PRQCI9pOswclwCM= 2023-09-22T21:49:36.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+EsWej/TGJ0+PRQCI9pOswclwCM=', 'Content-Length': '0'} 2023-09-22T21:49:36.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e589ea876ce4df40-00650e0bf0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-22T21:49:36.928 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:49:36.928 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:49:36.929 DEBUG:boto:Method: GET 2023-09-22T21:49:36.929 DEBUG:boto:Path: /ffzdqu-59/obj115 2023-09-22T21:49:36.929 DEBUG:boto:Data: 2023-09-22T21:49:36.929 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.929 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.929 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.929 DEBUG:boto:Params: {} 2023-09-22T21:49:36.929 DEBUG:boto:Token: None 2023-09-22T21:49:36.929 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj115 2023-09-22T21:49:36.929 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Xb0winXUxLEb8w5X8/RtzWD53G8= 2023-09-22T21:49:36.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Xb0winXUxLEb8w5X8/RtzWD53G8=', 'Content-Length': '0'} 2023-09-22T21:49:36.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd091b870dbbbee2-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.978 DEBUG:boto:path=/ffzdqu-59/obj115 2023-09-22T21:49:36.978 DEBUG:boto:auth_path=/ffzdqu-59/obj115 2023-09-22T21:49:36.979 DEBUG:boto:Method: GET 2023-09-22T21:49:36.979 DEBUG:boto:Path: /ffzdqu-59/obj115 2023-09-22T21:49:36.979 DEBUG:boto:Data: 2023-09-22T21:49:36.979 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.979 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.979 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.979 DEBUG:boto:Params: {} 2023-09-22T21:49:36.979 DEBUG:boto:Token: None 2023-09-22T21:49:36.979 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/obj115 2023-09-22T21:49:36.980 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Xb0winXUxLEb8w5X8/RtzWD53G8= 2023-09-22T21:49:36.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Xb0winXUxLEb8w5X8/RtzWD53G8=', 'Content-Length': '0'} 2023-09-22T21:49:36.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ddd1d38e7e05669-00650e0bf0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.983 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:36.983 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:36.983 DEBUG:boto:path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:49:36.983 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:49:36.983 DEBUG:boto:Method: GET 2023-09-22T21:49:36.983 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj115 2023-09-22T21:49:36.984 DEBUG:boto:Data: 2023-09-22T21:49:36.984 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.984 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:36.984 DEBUG:boto:Port: 8001 2023-09-22T21:49:36.984 DEBUG:boto:Params: {} 2023-09-22T21:49:36.984 DEBUG:boto:Token: None 2023-09-22T21:49:36.984 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/ 2023-09-22T21:49:36.984 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U= 2023-09-22T21:49:36.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U=', 'Content-Length': '0'} 2023-09-22T21:49:36.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060cc384e7bd9dc13-00650e0bf0-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:36.990 DEBUG:boto:b'ffzdqu-59obj1151000falseobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:36.990 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:36.991 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:36.991 DEBUG:boto:path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:49:36.991 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj115 2023-09-22T21:49:36.991 DEBUG:boto:Method: GET 2023-09-22T21:49:36.991 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj115 2023-09-22T21:49:36.991 DEBUG:boto:Data: 2023-09-22T21:49:36.991 DEBUG:boto:Headers: {} 2023-09-22T21:49:36.991 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:36.991 DEBUG:boto:Port: 8000 2023-09-22T21:49:36.992 DEBUG:boto:Params: {} 2023-09-22T21:49:36.992 DEBUG:boto:Token: None 2023-09-22T21:49:36.992 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:36 GMT /ffzdqu-59/ 2023-09-22T21:49:36.992 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U= 2023-09-22T21:49:36.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:puTBl9zeFHabS41Kzwgl4+aX0/U=', 'Content-Length': '0'} 2023-09-22T21:49:37.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efc39b9ee7312108-00650e0bf0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.002 DEBUG:boto:b'ffzdqu-59obj1151000falseobj1152023-09-22T21:40:21.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:37.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-22T21:49:37.002 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:49:37.002 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:49:37.003 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.003 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.003 DEBUG:boto:Method: GET 2023-09-22T21:49:37.003 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.003 DEBUG:boto:Data: 2023-09-22T21:49:37.003 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.003 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.003 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.003 DEBUG:boto:Params: {} 2023-09-22T21:49:37.004 DEBUG:boto:Token: None 2023-09-22T21:49:37.004 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.004 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MQ75m4HHBjD6QKFcY1FBHsZ9OAc= 2023-09-22T21:49:37.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MQ75m4HHBjD6QKFcY1FBHsZ9OAc=', 'Content-Length': '0'} 2023-09-22T21:49:37.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e80bd06f9f170ee2-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.008 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:49:37.008 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:49:37.008 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.008 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.008 DEBUG:boto:Method: GET 2023-09-22T21:49:37.008 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.008 DEBUG:boto:Data: 2023-09-22T21:49:37.008 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.008 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.009 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.009 DEBUG:boto:Params: {} 2023-09-22T21:49:37.009 DEBUG:boto:Token: None 2023-09-22T21:49:37.009 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.009 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MQ75m4HHBjD6QKFcY1FBHsZ9OAc= 2023-09-22T21:49:37.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MQ75m4HHBjD6QKFcY1FBHsZ9OAc=', 'Content-Length': '0'} 2023-09-22T21:49:37.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dda59018ca9be611-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.012 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:49:37.012 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:49:37.012 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.012 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.012 DEBUG:boto:Method: HEAD 2023-09-22T21:49:37.013 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.013 DEBUG:boto:Data: 2023-09-22T21:49:37.013 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.013 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.013 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.013 DEBUG:boto:Params: {} 2023-09-22T21:49:37.013 DEBUG:boto:Token: None 2023-09-22T21:49:37.013 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.013 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x9gKFh9GvrM77ujH5p2DpJvkF2Y= 2023-09-22T21:49:37.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x9gKFh9GvrM77ujH5p2DpJvkF2Y=', 'Content-Length': '0'} 2023-09-22T21:49:37.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000705374e3acf1efb8-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.016 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:49:37.016 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:49:37.017 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.017 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.017 DEBUG:boto:Method: HEAD 2023-09-22T21:49:37.017 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.017 DEBUG:boto:Data: 2023-09-22T21:49:37.017 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.017 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.017 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.017 DEBUG:boto:Params: {} 2023-09-22T21:49:37.017 DEBUG:boto:Token: None 2023-09-22T21:49:37.018 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.018 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x9gKFh9GvrM77ujH5p2DpJvkF2Y= 2023-09-22T21:49:37.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x9gKFh9GvrM77ujH5p2DpJvkF2Y=', 'Content-Length': '0'} 2023-09-22T21:49:37.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e689d7b82d3f41cb-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-22T21:49:37.021 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:49:37.021 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:49:37.021 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.021 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.021 DEBUG:boto:Method: GET 2023-09-22T21:49:37.021 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.022 DEBUG:boto:Data: 2023-09-22T21:49:37.022 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.022 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.022 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.022 DEBUG:boto:Params: {} 2023-09-22T21:49:37.022 DEBUG:boto:Token: None 2023-09-22T21:49:37.022 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.022 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MQ75m4HHBjD6QKFcY1FBHsZ9OAc= 2023-09-22T21:49:37.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MQ75m4HHBjD6QKFcY1FBHsZ9OAc=', 'Content-Length': '0'} 2023-09-22T21:49:37.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006e08a97e3d3defe-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.070 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:49:37.070 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:49:37.071 DEBUG:boto:path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.071 DEBUG:boto:auth_path=/ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.071 DEBUG:boto:Method: GET 2023-09-22T21:49:37.071 DEBUG:boto:Path: /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.071 DEBUG:boto:Data: 2023-09-22T21:49:37.071 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.071 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.071 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.071 DEBUG:boto:Params: {} 2023-09-22T21:49:37.072 DEBUG:boto:Token: None 2023-09-22T21:49:37.072 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj116?versionId=null 2023-09-22T21:49:37.072 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MQ75m4HHBjD6QKFcY1FBHsZ9OAc= 2023-09-22T21:49:37.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MQ75m4HHBjD6QKFcY1FBHsZ9OAc=', 'Content-Length': '0'} 2023-09-22T21:49:37.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9f28af5bd3b8d91-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.075 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:37.075 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:37.075 DEBUG:boto:path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:49:37.075 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:49:37.075 DEBUG:boto:Method: GET 2023-09-22T21:49:37.075 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj116 2023-09-22T21:49:37.076 DEBUG:boto:Data: 2023-09-22T21:49:37.076 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.076 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.076 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.076 DEBUG:boto:Params: {} 2023-09-22T21:49:37.076 DEBUG:boto:Token: None 2023-09-22T21:49:37.076 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/ 2023-09-22T21:49:37.076 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM= 2023-09-22T21:49:37.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM=', 'Content-Length': '0'} 2023-09-22T21:49:37.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000166b24084e691c9a-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.082 DEBUG:boto:b'ffzdqu-59obj1161000falseobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:37.082 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:37.085 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:37.085 DEBUG:boto:path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:49:37.085 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:49:37.085 DEBUG:boto:Method: GET 2023-09-22T21:49:37.085 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj116 2023-09-22T21:49:37.085 DEBUG:boto:Data: 2023-09-22T21:49:37.086 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.086 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.086 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.086 DEBUG:boto:Params: {} 2023-09-22T21:49:37.086 DEBUG:boto:Token: None 2023-09-22T21:49:37.086 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/ 2023-09-22T21:49:37.087 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM= 2023-09-22T21:49:37.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM=', 'Content-Length': '0'} 2023-09-22T21:49:37.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015a712b1c0dcd550-00650e0bf1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.097 DEBUG:boto:b'ffzdqu-59obj1161000falseobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:37.098 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:49:37.098 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:49:37.098 DEBUG:boto:Method: HEAD 2023-09-22T21:49:37.098 DEBUG:boto:Path: /ffzdqu-59/obj116 2023-09-22T21:49:37.098 DEBUG:boto:Data: 2023-09-22T21:49:37.098 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.098 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.098 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.099 DEBUG:boto:Params: {} 2023-09-22T21:49:37.099 DEBUG:boto:Token: None 2023-09-22T21:49:37.099 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj116 2023-09-22T21:49:37.099 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IvAxKPMPF3FtTB448cN2b0SD1os= 2023-09-22T21:49:37.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IvAxKPMPF3FtTB448cN2b0SD1os=', 'Content-Length': '0'} 2023-09-22T21:49:37.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9e60fe376a4ce22-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.102 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:49:37.102 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:49:37.102 DEBUG:boto:Method: HEAD 2023-09-22T21:49:37.102 DEBUG:boto:Path: /ffzdqu-59/obj116 2023-09-22T21:49:37.102 DEBUG:boto:Data: 2023-09-22T21:49:37.102 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.103 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.103 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.103 DEBUG:boto:Params: {} 2023-09-22T21:49:37.103 DEBUG:boto:Token: None 2023-09-22T21:49:37.103 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj116 2023-09-22T21:49:37.103 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IvAxKPMPF3FtTB448cN2b0SD1os= 2023-09-22T21:49:37.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IvAxKPMPF3FtTB448cN2b0SD1os=', 'Content-Length': '0'} 2023-09-22T21:49:37.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001b809d9ac3d68c8-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-22T21:49:37.106 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:49:37.106 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:49:37.106 DEBUG:boto:Method: GET 2023-09-22T21:49:37.107 DEBUG:boto:Path: /ffzdqu-59/obj116 2023-09-22T21:49:37.107 DEBUG:boto:Data: 2023-09-22T21:49:37.107 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.107 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.107 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.107 DEBUG:boto:Params: {} 2023-09-22T21:49:37.107 DEBUG:boto:Token: None 2023-09-22T21:49:37.107 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj116 2023-09-22T21:49:37.107 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8dlTjRk1DdPSZHopydAKzlTUGKM= 2023-09-22T21:49:37.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8dlTjRk1DdPSZHopydAKzlTUGKM=', 'Content-Length': '0'} 2023-09-22T21:49:37.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d40844d0859dffbf-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.111 DEBUG:boto:path=/ffzdqu-59/obj116 2023-09-22T21:49:37.111 DEBUG:boto:auth_path=/ffzdqu-59/obj116 2023-09-22T21:49:37.111 DEBUG:boto:Method: GET 2023-09-22T21:49:37.111 DEBUG:boto:Path: /ffzdqu-59/obj116 2023-09-22T21:49:37.111 DEBUG:boto:Data: 2023-09-22T21:49:37.111 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.111 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.111 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.111 DEBUG:boto:Params: {} 2023-09-22T21:49:37.112 DEBUG:boto:Token: None 2023-09-22T21:49:37.112 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj116 2023-09-22T21:49:37.112 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8dlTjRk1DdPSZHopydAKzlTUGKM= 2023-09-22T21:49:37.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8dlTjRk1DdPSZHopydAKzlTUGKM=', 'Content-Length': '0'} 2023-09-22T21:49:37.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021a4262eb8fd7abe-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.115 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:37.115 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:37.115 DEBUG:boto:path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:49:37.115 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:49:37.115 DEBUG:boto:Method: GET 2023-09-22T21:49:37.115 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj116 2023-09-22T21:49:37.115 DEBUG:boto:Data: 2023-09-22T21:49:37.115 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.115 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.116 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.116 DEBUG:boto:Params: {} 2023-09-22T21:49:37.116 DEBUG:boto:Token: None 2023-09-22T21:49:37.116 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/ 2023-09-22T21:49:37.116 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM= 2023-09-22T21:49:37.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM=', 'Content-Length': '0'} 2023-09-22T21:49:37.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c43830e97948f9c1-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.162 DEBUG:boto:b'ffzdqu-59obj1161000falseobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:37.163 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:37.163 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:37.163 DEBUG:boto:path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:49:37.163 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj116 2023-09-22T21:49:37.163 DEBUG:boto:Method: GET 2023-09-22T21:49:37.163 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj116 2023-09-22T21:49:37.164 DEBUG:boto:Data: 2023-09-22T21:49:37.164 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.164 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.164 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.164 DEBUG:boto:Params: {} 2023-09-22T21:49:37.164 DEBUG:boto:Token: None 2023-09-22T21:49:37.164 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/ 2023-09-22T21:49:37.164 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM= 2023-09-22T21:49:37.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM=', 'Content-Length': '0'} 2023-09-22T21:49:37.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047cbe66197d5ccec-00650e0bf1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.175 DEBUG:boto:b'ffzdqu-59obj1161000falseobj1162023-09-22T21:43:12.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:37.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-22T21:49:37.176 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:49:37.176 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:49:37.176 DEBUG:boto:path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.176 DEBUG:boto:auth_path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.176 DEBUG:boto:Method: GET 2023-09-22T21:49:37.176 DEBUG:boto:Path: /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.177 DEBUG:boto:Data: 2023-09-22T21:49:37.177 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.177 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.177 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.177 DEBUG:boto:Params: {} 2023-09-22T21:49:37.177 DEBUG:boto:Token: None 2023-09-22T21:49:37.177 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.177 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nfSErUn05aCgLvGLduFB1z45coQ= 2023-09-22T21:49:37.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nfSErUn05aCgLvGLduFB1z45coQ=', 'Content-Length': '0'} 2023-09-22T21:49:37.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c58a3a9255f0ff50-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.181 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:49:37.181 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:49:37.181 DEBUG:boto:path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.181 DEBUG:boto:auth_path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.182 DEBUG:boto:Method: GET 2023-09-22T21:49:37.182 DEBUG:boto:Path: /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.182 DEBUG:boto:Data: 2023-09-22T21:49:37.182 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.182 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.182 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.182 DEBUG:boto:Params: {} 2023-09-22T21:49:37.182 DEBUG:boto:Token: None 2023-09-22T21:49:37.182 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.183 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nfSErUn05aCgLvGLduFB1z45coQ= 2023-09-22T21:49:37.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nfSErUn05aCgLvGLduFB1z45coQ=', 'Content-Length': '0'} 2023-09-22T21:49:37.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052e7ff972edb8248-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.226 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:49:37.226 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:49:37.227 DEBUG:boto:path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.227 DEBUG:boto:auth_path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.227 DEBUG:boto:Method: HEAD 2023-09-22T21:49:37.227 DEBUG:boto:Path: /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.227 DEBUG:boto:Data: 2023-09-22T21:49:37.227 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.227 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.227 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.227 DEBUG:boto:Params: {} 2023-09-22T21:49:37.228 DEBUG:boto:Token: None 2023-09-22T21:49:37.228 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.228 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KLmhBoOq8QRLk/Zh982FEwUe6Sc= 2023-09-22T21:49:37.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KLmhBoOq8QRLk/Zh982FEwUe6Sc=', 'Content-Length': '0'} 2023-09-22T21:49:37.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb7d18ef35ee913c-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.231 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:49:37.231 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:49:37.232 DEBUG:boto:path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.232 DEBUG:boto:auth_path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.232 DEBUG:boto:Method: HEAD 2023-09-22T21:49:37.232 DEBUG:boto:Path: /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.232 DEBUG:boto:Data: 2023-09-22T21:49:37.232 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.232 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.232 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.232 DEBUG:boto:Params: {} 2023-09-22T21:49:37.233 DEBUG:boto:Token: None 2023-09-22T21:49:37.233 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.233 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KLmhBoOq8QRLk/Zh982FEwUe6Sc= 2023-09-22T21:49:37.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KLmhBoOq8QRLk/Zh982FEwUe6Sc=', 'Content-Length': '0'} 2023-09-22T21:49:37.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000646fffcfd848cfbe-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-22T21:49:37.236 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:49:37.236 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:49:37.236 DEBUG:boto:path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.236 DEBUG:boto:auth_path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.236 DEBUG:boto:Method: GET 2023-09-22T21:49:37.236 DEBUG:boto:Path: /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.236 DEBUG:boto:Data: 2023-09-22T21:49:37.236 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.237 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.237 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.237 DEBUG:boto:Params: {} 2023-09-22T21:49:37.237 DEBUG:boto:Token: None 2023-09-22T21:49:37.237 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.237 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nfSErUn05aCgLvGLduFB1z45coQ= 2023-09-22T21:49:37.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nfSErUn05aCgLvGLduFB1z45coQ=', 'Content-Length': '0'} 2023-09-22T21:49:37.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000f05df3458ec3da-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.241 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:49:37.241 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:49:37.241 DEBUG:boto:path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.241 DEBUG:boto:auth_path=/ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.241 DEBUG:boto:Method: GET 2023-09-22T21:49:37.241 DEBUG:boto:Path: /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.241 DEBUG:boto:Data: 2023-09-22T21:49:37.241 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.241 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.242 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.242 DEBUG:boto:Params: {} 2023-09-22T21:49:37.242 DEBUG:boto:Token: None 2023-09-22T21:49:37.242 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj124?versionId=null 2023-09-22T21:49:37.242 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nfSErUn05aCgLvGLduFB1z45coQ= 2023-09-22T21:49:37.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nfSErUn05aCgLvGLduFB1z45coQ=', 'Content-Length': '0'} 2023-09-22T21:49:37.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c01201af3c145f16-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.245 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:37.245 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:37.246 DEBUG:boto:path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:49:37.246 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:49:37.246 DEBUG:boto:Method: GET 2023-09-22T21:49:37.246 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj124 2023-09-22T21:49:37.246 DEBUG:boto:Data: 2023-09-22T21:49:37.246 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.246 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.246 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.246 DEBUG:boto:Params: {} 2023-09-22T21:49:37.247 DEBUG:boto:Token: None 2023-09-22T21:49:37.247 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/ 2023-09-22T21:49:37.247 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM= 2023-09-22T21:49:37.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM=', 'Content-Length': '0'} 2023-09-22T21:49:37.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aaa768957eeb77f3-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.294 DEBUG:boto:b'ffzdqu-59obj1241000falseobj1242023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:37.295 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:37.295 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:37.295 DEBUG:boto:path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:49:37.295 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:49:37.295 DEBUG:boto:Method: GET 2023-09-22T21:49:37.295 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj124 2023-09-22T21:49:37.295 DEBUG:boto:Data: 2023-09-22T21:49:37.295 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.296 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.296 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.296 DEBUG:boto:Params: {} 2023-09-22T21:49:37.296 DEBUG:boto:Token: None 2023-09-22T21:49:37.296 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/ 2023-09-22T21:49:37.296 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM= 2023-09-22T21:49:37.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM=', 'Content-Length': '0'} 2023-09-22T21:49:37.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4726975b50ca464-00650e0bf1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.306 DEBUG:boto:b'ffzdqu-59obj1241000falseobj1242023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:37.306 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:49:37.307 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:49:37.307 DEBUG:boto:Method: HEAD 2023-09-22T21:49:37.307 DEBUG:boto:Path: /ffzdqu-59/obj124 2023-09-22T21:49:37.307 DEBUG:boto:Data: 2023-09-22T21:49:37.307 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.307 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.307 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.307 DEBUG:boto:Params: {} 2023-09-22T21:49:37.307 DEBUG:boto:Token: None 2023-09-22T21:49:37.308 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj124 2023-09-22T21:49:37.308 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Vg78796l1ycwZhOToy7hdzuabvw= 2023-09-22T21:49:37.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Vg78796l1ycwZhOToy7hdzuabvw=', 'Content-Length': '0'} 2023-09-22T21:49:37.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079883aa3d764a44f-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.311 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:49:37.311 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:49:37.311 DEBUG:boto:Method: HEAD 2023-09-22T21:49:37.311 DEBUG:boto:Path: /ffzdqu-59/obj124 2023-09-22T21:49:37.311 DEBUG:boto:Data: 2023-09-22T21:49:37.311 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.312 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.312 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.312 DEBUG:boto:Params: {} 2023-09-22T21:49:37.312 DEBUG:boto:Token: None 2023-09-22T21:49:37.312 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj124 2023-09-22T21:49:37.312 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Vg78796l1ycwZhOToy7hdzuabvw= 2023-09-22T21:49:37.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Vg78796l1ycwZhOToy7hdzuabvw=', 'Content-Length': '0'} 2023-09-22T21:49:37.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092ddc90bf1ffa3a8-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-22T21:49:37.315 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:49:37.315 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:49:37.315 DEBUG:boto:Method: GET 2023-09-22T21:49:37.316 DEBUG:boto:Path: /ffzdqu-59/obj124 2023-09-22T21:49:37.316 DEBUG:boto:Data: 2023-09-22T21:49:37.316 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.316 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.316 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.316 DEBUG:boto:Params: {} 2023-09-22T21:49:37.316 DEBUG:boto:Token: None 2023-09-22T21:49:37.316 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj124 2023-09-22T21:49:37.316 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hf/NrKHE760wTQOuEoJeiwbV/gc= 2023-09-22T21:49:37.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hf/NrKHE760wTQOuEoJeiwbV/gc=', 'Content-Length': '0'} 2023-09-22T21:49:37.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000306d0fc04e96c41d-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.320 DEBUG:boto:path=/ffzdqu-59/obj124 2023-09-22T21:49:37.320 DEBUG:boto:auth_path=/ffzdqu-59/obj124 2023-09-22T21:49:37.320 DEBUG:boto:Method: GET 2023-09-22T21:49:37.320 DEBUG:boto:Path: /ffzdqu-59/obj124 2023-09-22T21:49:37.320 DEBUG:boto:Data: 2023-09-22T21:49:37.320 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.320 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.320 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.320 DEBUG:boto:Params: {} 2023-09-22T21:49:37.321 DEBUG:boto:Token: None 2023-09-22T21:49:37.321 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj124 2023-09-22T21:49:37.321 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hf/NrKHE760wTQOuEoJeiwbV/gc= 2023-09-22T21:49:37.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hf/NrKHE760wTQOuEoJeiwbV/gc=', 'Content-Length': '0'} 2023-09-22T21:49:37.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000add27565f0ebe0f8-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.366 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:37.367 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:37.367 DEBUG:boto:path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:49:37.367 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:49:37.367 DEBUG:boto:Method: GET 2023-09-22T21:49:37.367 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj124 2023-09-22T21:49:37.367 DEBUG:boto:Data: 2023-09-22T21:49:37.367 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.367 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.367 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.367 DEBUG:boto:Params: {} 2023-09-22T21:49:37.368 DEBUG:boto:Token: None 2023-09-22T21:49:37.368 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/ 2023-09-22T21:49:37.368 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM= 2023-09-22T21:49:37.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM=', 'Content-Length': '0'} 2023-09-22T21:49:37.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf25455d73484a7e-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.373 DEBUG:boto:b'ffzdqu-59obj1241000falseobj1242023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:37.374 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:37.374 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:37.374 DEBUG:boto:path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:49:37.374 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj124 2023-09-22T21:49:37.374 DEBUG:boto:Method: GET 2023-09-22T21:49:37.374 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj124 2023-09-22T21:49:37.374 DEBUG:boto:Data: 2023-09-22T21:49:37.375 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.375 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.375 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.375 DEBUG:boto:Params: {} 2023-09-22T21:49:37.375 DEBUG:boto:Token: None 2023-09-22T21:49:37.375 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/ 2023-09-22T21:49:37.375 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM= 2023-09-22T21:49:37.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM=', 'Content-Length': '0'} 2023-09-22T21:49:37.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b69ad0c601abf10-00650e0bf1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.385 DEBUG:boto:b'ffzdqu-59obj1241000falseobj1242023-09-22T21:45:13.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:37.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-22T21:49:37.385 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:49:37.385 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:49:37.385 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.386 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.386 DEBUG:boto:Method: GET 2023-09-22T21:49:37.386 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.386 DEBUG:boto:Data: 2023-09-22T21:49:37.386 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.386 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.386 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.386 DEBUG:boto:Params: {} 2023-09-22T21:49:37.386 DEBUG:boto:Token: None 2023-09-22T21:49:37.387 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.387 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:klomIGlLefUXL/mXt4Oeg/edDks= 2023-09-22T21:49:37.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:klomIGlLefUXL/mXt4Oeg/edDks=', 'Content-Length': '0'} 2023-09-22T21:49:37.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084aa36e6734e25c0-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.430 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:49:37.430 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:49:37.431 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.431 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.431 DEBUG:boto:Method: GET 2023-09-22T21:49:37.431 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.431 DEBUG:boto:Data: 2023-09-22T21:49:37.431 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.431 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.431 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.431 DEBUG:boto:Params: {} 2023-09-22T21:49:37.432 DEBUG:boto:Token: None 2023-09-22T21:49:37.432 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.432 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:klomIGlLefUXL/mXt4Oeg/edDks= 2023-09-22T21:49:37.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:klomIGlLefUXL/mXt4Oeg/edDks=', 'Content-Length': '0'} 2023-09-22T21:49:37.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0b2bc0589e4f31e-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.435 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:49:37.435 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:49:37.435 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.435 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.436 DEBUG:boto:Method: HEAD 2023-09-22T21:49:37.436 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.436 DEBUG:boto:Data: 2023-09-22T21:49:37.436 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.436 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.436 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.436 DEBUG:boto:Params: {} 2023-09-22T21:49:37.436 DEBUG:boto:Token: None 2023-09-22T21:49:37.436 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.437 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7a5nBOPPMVcOgVxZlu5hUSbpH0c= 2023-09-22T21:49:37.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7a5nBOPPMVcOgVxZlu5hUSbpH0c=', 'Content-Length': '0'} 2023-09-22T21:49:37.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba4975dcf08ef527-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.440 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:49:37.440 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:49:37.440 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.440 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.440 DEBUG:boto:Method: HEAD 2023-09-22T21:49:37.440 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.440 DEBUG:boto:Data: 2023-09-22T21:49:37.440 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.441 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.441 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.441 DEBUG:boto:Params: {} 2023-09-22T21:49:37.441 DEBUG:boto:Token: None 2023-09-22T21:49:37.441 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.441 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7a5nBOPPMVcOgVxZlu5hUSbpH0c= 2023-09-22T21:49:37.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7a5nBOPPMVcOgVxZlu5hUSbpH0c=', 'Content-Length': '0'} 2023-09-22T21:49:37.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000399f85c949938726-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-22T21:49:37.445 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:49:37.445 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:49:37.445 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.445 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.445 DEBUG:boto:Method: GET 2023-09-22T21:49:37.445 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.445 DEBUG:boto:Data: 2023-09-22T21:49:37.445 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.445 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.445 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.446 DEBUG:boto:Params: {} 2023-09-22T21:49:37.446 DEBUG:boto:Token: None 2023-09-22T21:49:37.446 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.446 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:klomIGlLefUXL/mXt4Oeg/edDks= 2023-09-22T21:49:37.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:klomIGlLefUXL/mXt4Oeg/edDks=', 'Content-Length': '0'} 2023-09-22T21:49:37.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b304ea5bc35b9769-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.449 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:49:37.449 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:49:37.450 DEBUG:boto:path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.450 DEBUG:boto:auth_path=/ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.450 DEBUG:boto:Method: GET 2023-09-22T21:49:37.450 DEBUG:boto:Path: /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.450 DEBUG:boto:Data: 2023-09-22T21:49:37.450 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.450 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.450 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.450 DEBUG:boto:Params: {} 2023-09-22T21:49:37.451 DEBUG:boto:Token: None 2023-09-22T21:49:37.451 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj145?versionId=null 2023-09-22T21:49:37.451 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:klomIGlLefUXL/mXt4Oeg/edDks= 2023-09-22T21:49:37.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:klomIGlLefUXL/mXt4Oeg/edDks=', 'Content-Length': '0'} 2023-09-22T21:49:37.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2c4b79b36d51e8b-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.494 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:37.494 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:37.495 DEBUG:boto:path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:49:37.495 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:49:37.495 DEBUG:boto:Method: GET 2023-09-22T21:49:37.495 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj145 2023-09-22T21:49:37.495 DEBUG:boto:Data: 2023-09-22T21:49:37.495 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.497 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.497 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.497 DEBUG:boto:Params: {} 2023-09-22T21:49:37.497 DEBUG:boto:Token: None 2023-09-22T21:49:37.497 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/ 2023-09-22T21:49:37.498 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM= 2023-09-22T21:49:37.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM=', 'Content-Length': '0'} 2023-09-22T21:49:37.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff92cae6fe46b40f-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.502 DEBUG:boto:b'ffzdqu-59obj1451000falseobj1452023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:37.503 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:37.503 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:37.503 DEBUG:boto:path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:49:37.503 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:49:37.503 DEBUG:boto:Method: GET 2023-09-22T21:49:37.503 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj145 2023-09-22T21:49:37.504 DEBUG:boto:Data: 2023-09-22T21:49:37.504 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.504 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.504 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.504 DEBUG:boto:Params: {} 2023-09-22T21:49:37.504 DEBUG:boto:Token: None 2023-09-22T21:49:37.504 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/ 2023-09-22T21:49:37.504 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM= 2023-09-22T21:49:37.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM=', 'Content-Length': '0'} 2023-09-22T21:49:37.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc637d85fe013cdf-00650e0bf1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.514 DEBUG:boto:b'ffzdqu-59obj1451000falseobj1452023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:37.515 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:49:37.515 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:49:37.515 DEBUG:boto:Method: HEAD 2023-09-22T21:49:37.515 DEBUG:boto:Path: /ffzdqu-59/obj145 2023-09-22T21:49:37.515 DEBUG:boto:Data: 2023-09-22T21:49:37.515 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.515 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.515 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.516 DEBUG:boto:Params: {} 2023-09-22T21:49:37.516 DEBUG:boto:Token: None 2023-09-22T21:49:37.516 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj145 2023-09-22T21:49:37.516 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7DX1QXdgP0SNRTChjy9baW9LnpM= 2023-09-22T21:49:37.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7DX1QXdgP0SNRTChjy9baW9LnpM=', 'Content-Length': '0'} 2023-09-22T21:49:37.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2c0d61a3950ec3d-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.519 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:49:37.519 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:49:37.519 DEBUG:boto:Method: HEAD 2023-09-22T21:49:37.519 DEBUG:boto:Path: /ffzdqu-59/obj145 2023-09-22T21:49:37.519 DEBUG:boto:Data: 2023-09-22T21:49:37.520 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.520 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.520 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.520 DEBUG:boto:Params: {} 2023-09-22T21:49:37.520 DEBUG:boto:Token: None 2023-09-22T21:49:37.520 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj145 2023-09-22T21:49:37.520 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7DX1QXdgP0SNRTChjy9baW9LnpM= 2023-09-22T21:49:37.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7DX1QXdgP0SNRTChjy9baW9LnpM=', 'Content-Length': '0'} 2023-09-22T21:49:37.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f34f715ffc10633-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-22T21:49:37.524 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:49:37.524 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:49:37.524 DEBUG:boto:Method: GET 2023-09-22T21:49:37.524 DEBUG:boto:Path: /ffzdqu-59/obj145 2023-09-22T21:49:37.524 DEBUG:boto:Data: 2023-09-22T21:49:37.524 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.524 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.525 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.525 DEBUG:boto:Params: {} 2023-09-22T21:49:37.525 DEBUG:boto:Token: None 2023-09-22T21:49:37.525 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj145 2023-09-22T21:49:37.525 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MeE19zQc5zduucT/vhyr0vPhd6A= 2023-09-22T21:49:37.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MeE19zQc5zduucT/vhyr0vPhd6A=', 'Content-Length': '0'} 2023-09-22T21:49:37.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f07fc504e6ce4b8-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.570 DEBUG:boto:path=/ffzdqu-59/obj145 2023-09-22T21:49:37.571 DEBUG:boto:auth_path=/ffzdqu-59/obj145 2023-09-22T21:49:37.571 DEBUG:boto:Method: GET 2023-09-22T21:49:37.571 DEBUG:boto:Path: /ffzdqu-59/obj145 2023-09-22T21:49:37.571 DEBUG:boto:Data: 2023-09-22T21:49:37.571 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.571 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.571 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.571 DEBUG:boto:Params: {} 2023-09-22T21:49:37.571 DEBUG:boto:Token: None 2023-09-22T21:49:37.572 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj145 2023-09-22T21:49:37.572 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MeE19zQc5zduucT/vhyr0vPhd6A= 2023-09-22T21:49:37.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MeE19zQc5zduucT/vhyr0vPhd6A=', 'Content-Length': '0'} 2023-09-22T21:49:37.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d74e009fa6e17b96-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.575 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:37.575 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:37.575 DEBUG:boto:path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:49:37.575 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:49:37.575 DEBUG:boto:Method: GET 2023-09-22T21:49:37.576 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj145 2023-09-22T21:49:37.576 DEBUG:boto:Data: 2023-09-22T21:49:37.576 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.576 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.576 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.576 DEBUG:boto:Params: {} 2023-09-22T21:49:37.576 DEBUG:boto:Token: None 2023-09-22T21:49:37.576 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/ 2023-09-22T21:49:37.576 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM= 2023-09-22T21:49:37.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM=', 'Content-Length': '0'} 2023-09-22T21:49:37.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041d8b97cef919d9f-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.581 DEBUG:boto:b'ffzdqu-59obj1451000falseobj1452023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:37.582 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:37.582 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:37.582 DEBUG:boto:path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:49:37.582 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj145 2023-09-22T21:49:37.582 DEBUG:boto:Method: GET 2023-09-22T21:49:37.582 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj145 2023-09-22T21:49:37.582 DEBUG:boto:Data: 2023-09-22T21:49:37.583 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.583 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.583 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.583 DEBUG:boto:Params: {} 2023-09-22T21:49:37.583 DEBUG:boto:Token: None 2023-09-22T21:49:37.583 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/ 2023-09-22T21:49:37.583 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM= 2023-09-22T21:49:37.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM=', 'Content-Length': '0'} 2023-09-22T21:49:37.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef7e3a865273862c-00650e0bf1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.638 DEBUG:boto:b'ffzdqu-59obj1451000falseobj1452023-09-22T21:43:12.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:37.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-09-22T21:49:37.639 DEBUG:boto:path=/ffzdqu-59/obj148 2023-09-22T21:49:37.639 DEBUG:boto:auth_path=/ffzdqu-59/obj148 2023-09-22T21:49:37.639 DEBUG:boto:path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.639 DEBUG:boto:auth_path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.639 DEBUG:boto:Method: GET 2023-09-22T21:49:37.639 DEBUG:boto:Path: /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.640 DEBUG:boto:Data: 2023-09-22T21:49:37.640 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.640 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.640 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.640 DEBUG:boto:Params: {} 2023-09-22T21:49:37.640 DEBUG:boto:Token: None 2023-09-22T21:49:37.640 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.640 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7O4f06q+249nzB4xVp2OTvfeetA= 2023-09-22T21:49:37.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7O4f06q+249nzB4xVp2OTvfeetA=', 'Content-Length': '0'} 2023-09-22T21:49:37.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004651c4cd2b5918fb-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.644 DEBUG:boto:path=/ffzdqu-59/obj148 2023-09-22T21:49:37.644 DEBUG:boto:auth_path=/ffzdqu-59/obj148 2023-09-22T21:49:37.644 DEBUG:boto:path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.644 DEBUG:boto:auth_path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.644 DEBUG:boto:Method: GET 2023-09-22T21:49:37.644 DEBUG:boto:Path: /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.645 DEBUG:boto:Data: 2023-09-22T21:49:37.645 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.645 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.645 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.645 DEBUG:boto:Params: {} 2023-09-22T21:49:37.645 DEBUG:boto:Token: None 2023-09-22T21:49:37.645 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.645 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7O4f06q+249nzB4xVp2OTvfeetA= 2023-09-22T21:49:37.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7O4f06q+249nzB4xVp2OTvfeetA=', 'Content-Length': '0'} 2023-09-22T21:49:37.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020f3ed16d8d5b119-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.649 DEBUG:boto:path=/ffzdqu-59/obj148 2023-09-22T21:49:37.649 DEBUG:boto:auth_path=/ffzdqu-59/obj148 2023-09-22T21:49:37.649 DEBUG:boto:path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.649 DEBUG:boto:auth_path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.649 DEBUG:boto:Method: HEAD 2023-09-22T21:49:37.649 DEBUG:boto:Path: /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.650 DEBUG:boto:Data: 2023-09-22T21:49:37.650 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.650 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.650 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.650 DEBUG:boto:Params: {} 2023-09-22T21:49:37.650 DEBUG:boto:Token: None 2023-09-22T21:49:37.650 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.650 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oYrkGMFGWnFTdxMSJ+CRYfPh8vM= 2023-09-22T21:49:37.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oYrkGMFGWnFTdxMSJ+CRYfPh8vM=', 'Content-Length': '0'} 2023-09-22T21:49:37.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3f2625b2c6858c8-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.653 DEBUG:boto:path=/ffzdqu-59/obj148 2023-09-22T21:49:37.653 DEBUG:boto:auth_path=/ffzdqu-59/obj148 2023-09-22T21:49:37.653 DEBUG:boto:path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.654 DEBUG:boto:auth_path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.654 DEBUG:boto:Method: HEAD 2023-09-22T21:49:37.654 DEBUG:boto:Path: /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.654 DEBUG:boto:Data: 2023-09-22T21:49:37.654 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.654 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.654 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.654 DEBUG:boto:Params: {} 2023-09-22T21:49:37.654 DEBUG:boto:Token: None 2023-09-22T21:49:37.655 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.655 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oYrkGMFGWnFTdxMSJ+CRYfPh8vM= 2023-09-22T21:49:37.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oYrkGMFGWnFTdxMSJ+CRYfPh8vM=', 'Content-Length': '0'} 2023-09-22T21:49:37.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa8ac14d4d090360-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-09-22T21:49:37.658 DEBUG:boto:path=/ffzdqu-59/obj148 2023-09-22T21:49:37.658 DEBUG:boto:auth_path=/ffzdqu-59/obj148 2023-09-22T21:49:37.658 DEBUG:boto:path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.658 DEBUG:boto:auth_path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.658 DEBUG:boto:Method: GET 2023-09-22T21:49:37.658 DEBUG:boto:Path: /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.658 DEBUG:boto:Data: 2023-09-22T21:49:37.659 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.659 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.659 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.659 DEBUG:boto:Params: {} 2023-09-22T21:49:37.659 DEBUG:boto:Token: None 2023-09-22T21:49:37.659 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.659 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7O4f06q+249nzB4xVp2OTvfeetA= 2023-09-22T21:49:37.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7O4f06q+249nzB4xVp2OTvfeetA=', 'Content-Length': '0'} 2023-09-22T21:49:37.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3677336abdee60d-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.663 DEBUG:boto:path=/ffzdqu-59/obj148 2023-09-22T21:49:37.663 DEBUG:boto:auth_path=/ffzdqu-59/obj148 2023-09-22T21:49:37.663 DEBUG:boto:path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.663 DEBUG:boto:auth_path=/ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.663 DEBUG:boto:Method: GET 2023-09-22T21:49:37.663 DEBUG:boto:Path: /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.663 DEBUG:boto:Data: 2023-09-22T21:49:37.663 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.663 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.663 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.664 DEBUG:boto:Params: {} 2023-09-22T21:49:37.664 DEBUG:boto:Token: None 2023-09-22T21:49:37.664 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj148?versionId=null 2023-09-22T21:49:37.664 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7O4f06q+249nzB4xVp2OTvfeetA= 2023-09-22T21:49:37.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7O4f06q+249nzB4xVp2OTvfeetA=', 'Content-Length': '0'} 2023-09-22T21:49:37.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dc8423635c577dd-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.667 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:37.667 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:37.667 DEBUG:boto:path=/ffzdqu-59/?prefix=obj148 2023-09-22T21:49:37.667 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj148 2023-09-22T21:49:37.668 DEBUG:boto:Method: GET 2023-09-22T21:49:37.668 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj148 2023-09-22T21:49:37.668 DEBUG:boto:Data: 2023-09-22T21:49:37.668 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.668 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.668 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.668 DEBUG:boto:Params: {} 2023-09-22T21:49:37.668 DEBUG:boto:Token: None 2023-09-22T21:49:37.668 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/ 2023-09-22T21:49:37.669 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM= 2023-09-22T21:49:37.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM=', 'Content-Length': '0'} 2023-09-22T21:49:37.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aef57fb2b35201bf-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.718 DEBUG:boto:b'ffzdqu-59obj1481000falseobj1482023-09-22T21:47:16.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:37.719 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:37.719 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:37.719 DEBUG:boto:path=/ffzdqu-59/?prefix=obj148 2023-09-22T21:49:37.719 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj148 2023-09-22T21:49:37.719 DEBUG:boto:Method: GET 2023-09-22T21:49:37.719 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj148 2023-09-22T21:49:37.719 DEBUG:boto:Data: 2023-09-22T21:49:37.719 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.719 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.720 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.720 DEBUG:boto:Params: {} 2023-09-22T21:49:37.720 DEBUG:boto:Token: None 2023-09-22T21:49:37.720 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/ 2023-09-22T21:49:37.720 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM= 2023-09-22T21:49:37.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM=', 'Content-Length': '0'} 2023-09-22T21:49:37.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c367c6c716baa372-00650e0bf1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.731 DEBUG:boto:b'ffzdqu-59obj1481000falseobj1482023-09-22T21:47:16.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:37.731 DEBUG:boto:path=/ffzdqu-59/obj148 2023-09-22T21:49:37.731 DEBUG:boto:auth_path=/ffzdqu-59/obj148 2023-09-22T21:49:37.731 DEBUG:boto:Method: HEAD 2023-09-22T21:49:37.732 DEBUG:boto:Path: /ffzdqu-59/obj148 2023-09-22T21:49:37.732 DEBUG:boto:Data: 2023-09-22T21:49:37.732 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.732 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.732 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.732 DEBUG:boto:Params: {} 2023-09-22T21:49:37.732 DEBUG:boto:Token: None 2023-09-22T21:49:37.732 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj148 2023-09-22T21:49:37.732 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:L3piJII3xa4obEVha+sEJQ1TMsw= 2023-09-22T21:49:37.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:L3piJII3xa4obEVha+sEJQ1TMsw=', 'Content-Length': '0'} 2023-09-22T21:49:37.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7f8c81c7c5337a2-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.735 DEBUG:boto:path=/ffzdqu-59/obj148 2023-09-22T21:49:37.736 DEBUG:boto:auth_path=/ffzdqu-59/obj148 2023-09-22T21:49:37.736 DEBUG:boto:Method: HEAD 2023-09-22T21:49:37.736 DEBUG:boto:Path: /ffzdqu-59/obj148 2023-09-22T21:49:37.736 DEBUG:boto:Data: 2023-09-22T21:49:37.736 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.736 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.736 DEBUG:boto:Params: {} 2023-09-22T21:49:37.736 DEBUG:boto:Token: None 2023-09-22T21:49:37.737 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj148 2023-09-22T21:49:37.737 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:L3piJII3xa4obEVha+sEJQ1TMsw= 2023-09-22T21:49:37.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:L3piJII3xa4obEVha+sEJQ1TMsw=', 'Content-Length': '0'} 2023-09-22T21:49:37.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca5d1be6a4346430-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-09-22T21:49:37.740 DEBUG:boto:path=/ffzdqu-59/obj148 2023-09-22T21:49:37.740 DEBUG:boto:auth_path=/ffzdqu-59/obj148 2023-09-22T21:49:37.740 DEBUG:boto:Method: GET 2023-09-22T21:49:37.740 DEBUG:boto:Path: /ffzdqu-59/obj148 2023-09-22T21:49:37.740 DEBUG:boto:Data: 2023-09-22T21:49:37.740 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.740 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.741 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.741 DEBUG:boto:Params: {} 2023-09-22T21:49:37.741 DEBUG:boto:Token: None 2023-09-22T21:49:37.741 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj148 2023-09-22T21:49:37.741 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Z1phO8N0V2o955dlirYCDvqWHmk= 2023-09-22T21:49:37.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Z1phO8N0V2o955dlirYCDvqWHmk=', 'Content-Length': '0'} 2023-09-22T21:49:37.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb8d7d277df527ab-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.744 DEBUG:boto:path=/ffzdqu-59/obj148 2023-09-22T21:49:37.744 DEBUG:boto:auth_path=/ffzdqu-59/obj148 2023-09-22T21:49:37.744 DEBUG:boto:Method: GET 2023-09-22T21:49:37.744 DEBUG:boto:Path: /ffzdqu-59/obj148 2023-09-22T21:49:37.744 DEBUG:boto:Data: 2023-09-22T21:49:37.745 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.745 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.745 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.745 DEBUG:boto:Params: {} 2023-09-22T21:49:37.745 DEBUG:boto:Token: None 2023-09-22T21:49:37.745 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj148 2023-09-22T21:49:37.745 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Z1phO8N0V2o955dlirYCDvqWHmk= 2023-09-22T21:49:37.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Z1phO8N0V2o955dlirYCDvqWHmk=', 'Content-Length': '0'} 2023-09-22T21:49:37.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000259697416ceee749-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.790 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:37.861 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:37.862 DEBUG:boto:path=/ffzdqu-59/?prefix=obj148 2023-09-22T21:49:37.862 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj148 2023-09-22T21:49:37.862 DEBUG:boto:Method: GET 2023-09-22T21:49:37.862 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj148 2023-09-22T21:49:37.862 DEBUG:boto:Data: 2023-09-22T21:49:37.862 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.862 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.862 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.863 DEBUG:boto:Params: {} 2023-09-22T21:49:37.863 DEBUG:boto:Token: None 2023-09-22T21:49:37.863 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/ 2023-09-22T21:49:37.863 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM= 2023-09-22T21:49:37.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM=', 'Content-Length': '0'} 2023-09-22T21:49:37.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020740dff44b62d4b-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.870 DEBUG:boto:b'ffzdqu-59obj1481000falseobj1482023-09-22T21:47:16.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:37.870 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:37.870 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:37.870 DEBUG:boto:path=/ffzdqu-59/?prefix=obj148 2023-09-22T21:49:37.870 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj148 2023-09-22T21:49:37.871 DEBUG:boto:Method: GET 2023-09-22T21:49:37.871 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj148 2023-09-22T21:49:37.871 DEBUG:boto:Data: 2023-09-22T21:49:37.871 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.871 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.871 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.871 DEBUG:boto:Params: {} 2023-09-22T21:49:37.871 DEBUG:boto:Token: None 2023-09-22T21:49:37.871 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/ 2023-09-22T21:49:37.872 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM= 2023-09-22T21:49:37.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM=', 'Content-Length': '0'} 2023-09-22T21:49:37.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5687c7347e58918-00650e0bf1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.883 DEBUG:boto:b'ffzdqu-59obj1481000falseobj1482023-09-22T21:47:16.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:37.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-22T21:49:37.883 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:49:37.883 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:49:37.883 DEBUG:boto:path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.883 DEBUG:boto:auth_path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.884 DEBUG:boto:Method: GET 2023-09-22T21:49:37.884 DEBUG:boto:Path: /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.884 DEBUG:boto:Data: 2023-09-22T21:49:37.884 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.884 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.884 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.884 DEBUG:boto:Params: {} 2023-09-22T21:49:37.884 DEBUG:boto:Token: None 2023-09-22T21:49:37.885 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.885 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SNBdkI2q+2XEYfkQN8BJkNnXlZA= 2023-09-22T21:49:37.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNBdkI2q+2XEYfkQN8BJkNnXlZA=', 'Content-Length': '0'} 2023-09-22T21:49:37.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000666a9e30a9c63cd0-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.934 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:49:37.935 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:49:37.935 DEBUG:boto:path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.935 DEBUG:boto:auth_path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.935 DEBUG:boto:Method: GET 2023-09-22T21:49:37.935 DEBUG:boto:Path: /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.935 DEBUG:boto:Data: 2023-09-22T21:49:37.935 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.935 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.935 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.935 DEBUG:boto:Params: {} 2023-09-22T21:49:37.936 DEBUG:boto:Token: None 2023-09-22T21:49:37.936 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.936 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SNBdkI2q+2XEYfkQN8BJkNnXlZA= 2023-09-22T21:49:37.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNBdkI2q+2XEYfkQN8BJkNnXlZA=', 'Content-Length': '0'} 2023-09-22T21:49:37.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2bdd360ce0d882a-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.939 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:49:37.939 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:49:37.939 DEBUG:boto:path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.939 DEBUG:boto:auth_path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.940 DEBUG:boto:Method: HEAD 2023-09-22T21:49:37.940 DEBUG:boto:Path: /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.940 DEBUG:boto:Data: 2023-09-22T21:49:37.940 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.940 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.940 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.940 DEBUG:boto:Params: {} 2023-09-22T21:49:37.940 DEBUG:boto:Token: None 2023-09-22T21:49:37.940 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.941 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:X4Z73JrZJ8VZtxyOZYwR3naXbbc= 2023-09-22T21:49:37.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:X4Z73JrZJ8VZtxyOZYwR3naXbbc=', 'Content-Length': '0'} 2023-09-22T21:49:37.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000671a580b48acc62d-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.944 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:49:37.944 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:49:37.944 DEBUG:boto:path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.944 DEBUG:boto:auth_path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.944 DEBUG:boto:Method: HEAD 2023-09-22T21:49:37.944 DEBUG:boto:Path: /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.944 DEBUG:boto:Data: 2023-09-22T21:49:37.944 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.945 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.945 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.945 DEBUG:boto:Params: {} 2023-09-22T21:49:37.945 DEBUG:boto:Token: None 2023-09-22T21:49:37.945 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.945 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:X4Z73JrZJ8VZtxyOZYwR3naXbbc= 2023-09-22T21:49:37.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:X4Z73JrZJ8VZtxyOZYwR3naXbbc=', 'Content-Length': '0'} 2023-09-22T21:49:37.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a58efcc606cbc5a-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-22T21:49:37.949 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:49:37.949 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:49:37.949 DEBUG:boto:path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.949 DEBUG:boto:auth_path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.949 DEBUG:boto:Method: GET 2023-09-22T21:49:37.949 DEBUG:boto:Path: /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.949 DEBUG:boto:Data: 2023-09-22T21:49:37.950 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.950 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.950 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.950 DEBUG:boto:Params: {} 2023-09-22T21:49:37.950 DEBUG:boto:Token: None 2023-09-22T21:49:37.950 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.950 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SNBdkI2q+2XEYfkQN8BJkNnXlZA= 2023-09-22T21:49:37.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNBdkI2q+2XEYfkQN8BJkNnXlZA=', 'Content-Length': '0'} 2023-09-22T21:49:37.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcddecfba86f4a5b-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.954 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:49:37.954 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:49:37.954 DEBUG:boto:path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.955 DEBUG:boto:auth_path=/ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.955 DEBUG:boto:Method: GET 2023-09-22T21:49:37.955 DEBUG:boto:Path: /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.955 DEBUG:boto:Data: 2023-09-22T21:49:37.955 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.955 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.955 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.955 DEBUG:boto:Params: {} 2023-09-22T21:49:37.955 DEBUG:boto:Token: None 2023-09-22T21:49:37.955 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/obj155?versionId=null 2023-09-22T21:49:37.956 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SNBdkI2q+2XEYfkQN8BJkNnXlZA= 2023-09-22T21:49:37.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNBdkI2q+2XEYfkQN8BJkNnXlZA=', 'Content-Length': '0'} 2023-09-22T21:49:37.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088e681dd87183a7e-00650e0bf1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.959 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:37.959 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:37.959 DEBUG:boto:path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:49:37.959 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:49:37.959 DEBUG:boto:Method: GET 2023-09-22T21:49:37.959 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj155 2023-09-22T21:49:37.959 DEBUG:boto:Data: 2023-09-22T21:49:37.959 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.959 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:37.960 DEBUG:boto:Port: 8001 2023-09-22T21:49:37.960 DEBUG:boto:Params: {} 2023-09-22T21:49:37.960 DEBUG:boto:Token: None 2023-09-22T21:49:37.960 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/ 2023-09-22T21:49:37.960 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM= 2023-09-22T21:49:37.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM=', 'Content-Length': '0'} 2023-09-22T21:49:37.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e79d7a1a89049adf-00650e0bf1-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:37.965 DEBUG:boto:b'ffzdqu-59obj1551000falseobj1552023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:37.965 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:37.965 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:37.965 DEBUG:boto:path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:49:37.966 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:49:37.966 DEBUG:boto:Method: GET 2023-09-22T21:49:37.966 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj155 2023-09-22T21:49:37.966 DEBUG:boto:Data: 2023-09-22T21:49:37.966 DEBUG:boto:Headers: {} 2023-09-22T21:49:37.966 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:37.967 DEBUG:boto:Port: 8000 2023-09-22T21:49:37.967 DEBUG:boto:Params: {} 2023-09-22T21:49:37.967 DEBUG:boto:Token: None 2023-09-22T21:49:37.967 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:37 GMT /ffzdqu-59/ 2023-09-22T21:49:37.967 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM= 2023-09-22T21:49:37.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F0bRmAOaXQ6qznJsgFEkvsiRAGM=', 'Content-Length': '0'} 2023-09-22T21:49:37.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbbb5c0da44693cd-00650e0bf1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.026 DEBUG:boto:b'ffzdqu-59obj1551000falseobj1552023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:38.027 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:49:38.027 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:49:38.027 DEBUG:boto:Method: HEAD 2023-09-22T21:49:38.027 DEBUG:boto:Path: /ffzdqu-59/obj155 2023-09-22T21:49:38.027 DEBUG:boto:Data: 2023-09-22T21:49:38.027 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.027 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.027 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.027 DEBUG:boto:Params: {} 2023-09-22T21:49:38.028 DEBUG:boto:Token: None 2023-09-22T21:49:38.028 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj155 2023-09-22T21:49:38.028 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kk16UNmG4JEYiRfIslBPZMbBk9I= 2023-09-22T21:49:38.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kk16UNmG4JEYiRfIslBPZMbBk9I=', 'Content-Length': '0'} 2023-09-22T21:49:38.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fe26aedf42c0b55-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.031 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:49:38.031 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:49:38.031 DEBUG:boto:Method: HEAD 2023-09-22T21:49:38.031 DEBUG:boto:Path: /ffzdqu-59/obj155 2023-09-22T21:49:38.031 DEBUG:boto:Data: 2023-09-22T21:49:38.032 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.032 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.032 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.032 DEBUG:boto:Params: {} 2023-09-22T21:49:38.032 DEBUG:boto:Token: None 2023-09-22T21:49:38.032 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj155 2023-09-22T21:49:38.032 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kk16UNmG4JEYiRfIslBPZMbBk9I= 2023-09-22T21:49:38.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kk16UNmG4JEYiRfIslBPZMbBk9I=', 'Content-Length': '0'} 2023-09-22T21:49:38.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093c5a159d8b41522-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-22T21:49:38.036 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:49:38.036 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:49:38.036 DEBUG:boto:Method: GET 2023-09-22T21:49:38.036 DEBUG:boto:Path: /ffzdqu-59/obj155 2023-09-22T21:49:38.036 DEBUG:boto:Data: 2023-09-22T21:49:38.036 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.036 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.036 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.036 DEBUG:boto:Params: {} 2023-09-22T21:49:38.036 DEBUG:boto:Token: None 2023-09-22T21:49:38.037 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj155 2023-09-22T21:49:38.037 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WIe5aaoxNFQjVK1H00b5kloO7sw= 2023-09-22T21:49:38.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WIe5aaoxNFQjVK1H00b5kloO7sw=', 'Content-Length': '0'} 2023-09-22T21:49:38.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000058e9c289522248e-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.082 DEBUG:boto:path=/ffzdqu-59/obj155 2023-09-22T21:49:38.082 DEBUG:boto:auth_path=/ffzdqu-59/obj155 2023-09-22T21:49:38.083 DEBUG:boto:Method: GET 2023-09-22T21:49:38.083 DEBUG:boto:Path: /ffzdqu-59/obj155 2023-09-22T21:49:38.083 DEBUG:boto:Data: 2023-09-22T21:49:38.083 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.083 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.083 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.083 DEBUG:boto:Params: {} 2023-09-22T21:49:38.083 DEBUG:boto:Token: None 2023-09-22T21:49:38.083 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj155 2023-09-22T21:49:38.084 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WIe5aaoxNFQjVK1H00b5kloO7sw= 2023-09-22T21:49:38.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WIe5aaoxNFQjVK1H00b5kloO7sw=', 'Content-Length': '0'} 2023-09-22T21:49:38.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043636b6edab7e528-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.087 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:38.087 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:38.087 DEBUG:boto:path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:49:38.087 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:49:38.087 DEBUG:boto:Method: GET 2023-09-22T21:49:38.087 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj155 2023-09-22T21:49:38.087 DEBUG:boto:Data: 2023-09-22T21:49:38.087 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.088 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.088 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.088 DEBUG:boto:Params: {} 2023-09-22T21:49:38.088 DEBUG:boto:Token: None 2023-09-22T21:49:38.088 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/ 2023-09-22T21:49:38.088 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU= 2023-09-22T21:49:38.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU=', 'Content-Length': '0'} 2023-09-22T21:49:38.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078fed2345f3b6507-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.093 DEBUG:boto:b'ffzdqu-59obj1551000falseobj1552023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:38.093 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:38.093 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:38.094 DEBUG:boto:path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:49:38.094 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj155 2023-09-22T21:49:38.094 DEBUG:boto:Method: GET 2023-09-22T21:49:38.094 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj155 2023-09-22T21:49:38.094 DEBUG:boto:Data: 2023-09-22T21:49:38.094 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.094 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.094 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.094 DEBUG:boto:Params: {} 2023-09-22T21:49:38.094 DEBUG:boto:Token: None 2023-09-22T21:49:38.095 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/ 2023-09-22T21:49:38.095 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU= 2023-09-22T21:49:38.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU=', 'Content-Length': '0'} 2023-09-22T21:49:38.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028adf13044b3f4e5-00650e0bf2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.105 DEBUG:boto:b'ffzdqu-59obj1551000falseobj1552023-09-22T21:45:13.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:38.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:49:38.105 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:49:38.105 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:49:38.106 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.106 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.106 DEBUG:boto:Method: GET 2023-09-22T21:49:38.106 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.106 DEBUG:boto:Data: 2023-09-22T21:49:38.106 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.106 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.106 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.106 DEBUG:boto:Params: {} 2023-09-22T21:49:38.107 DEBUG:boto:Token: None 2023-09-22T21:49:38.107 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.107 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:koLRpWM6rCyT9HBWzv3I7b+djh8= 2023-09-22T21:49:38.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:koLRpWM6rCyT9HBWzv3I7b+djh8=', 'Content-Length': '0'} 2023-09-22T21:49:38.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d3c47162bcd1d95-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.112 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:49:38.112 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:49:38.112 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.112 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.112 DEBUG:boto:Method: GET 2023-09-22T21:49:38.112 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.112 DEBUG:boto:Data: 2023-09-22T21:49:38.113 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.113 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.113 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.113 DEBUG:boto:Params: {} 2023-09-22T21:49:38.113 DEBUG:boto:Token: None 2023-09-22T21:49:38.113 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.113 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:koLRpWM6rCyT9HBWzv3I7b+djh8= 2023-09-22T21:49:38.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:koLRpWM6rCyT9HBWzv3I7b+djh8=', 'Content-Length': '0'} 2023-09-22T21:49:38.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c652650d7d8a7a51-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.158 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:49:38.159 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:49:38.159 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.159 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.159 DEBUG:boto:Method: HEAD 2023-09-22T21:49:38.159 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.159 DEBUG:boto:Data: 2023-09-22T21:49:38.159 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.159 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.159 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.159 DEBUG:boto:Params: {} 2023-09-22T21:49:38.160 DEBUG:boto:Token: None 2023-09-22T21:49:38.160 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.160 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fVKZbH6iuRJx88tF1YBOINQ8OK0= 2023-09-22T21:49:38.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fVKZbH6iuRJx88tF1YBOINQ8OK0=', 'Content-Length': '0'} 2023-09-22T21:49:38.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca46d1473593337e-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.163 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:49:38.163 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:49:38.163 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.163 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.163 DEBUG:boto:Method: HEAD 2023-09-22T21:49:38.163 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.164 DEBUG:boto:Data: 2023-09-22T21:49:38.164 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.164 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.164 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.164 DEBUG:boto:Params: {} 2023-09-22T21:49:38.164 DEBUG:boto:Token: None 2023-09-22T21:49:38.164 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.164 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fVKZbH6iuRJx88tF1YBOINQ8OK0= 2023-09-22T21:49:38.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fVKZbH6iuRJx88tF1YBOINQ8OK0=', 'Content-Length': '0'} 2023-09-22T21:49:38.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092ebeb7631b1d45a-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:49:38.168 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:49:38.168 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:49:38.168 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.168 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.168 DEBUG:boto:Method: GET 2023-09-22T21:49:38.168 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.168 DEBUG:boto:Data: 2023-09-22T21:49:38.168 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.168 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.168 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.169 DEBUG:boto:Params: {} 2023-09-22T21:49:38.169 DEBUG:boto:Token: None 2023-09-22T21:49:38.169 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.169 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:koLRpWM6rCyT9HBWzv3I7b+djh8= 2023-09-22T21:49:38.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:koLRpWM6rCyT9HBWzv3I7b+djh8=', 'Content-Length': '0'} 2023-09-22T21:49:38.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071bc3f2734bcf32d-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.214 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:49:38.214 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:49:38.215 DEBUG:boto:path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.215 DEBUG:boto:auth_path=/ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.215 DEBUG:boto:Method: GET 2023-09-22T21:49:38.215 DEBUG:boto:Path: /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.215 DEBUG:boto:Data: 2023-09-22T21:49:38.215 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.215 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.215 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.215 DEBUG:boto:Params: {} 2023-09-22T21:49:38.216 DEBUG:boto:Token: None 2023-09-22T21:49:38.216 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj17?versionId=null 2023-09-22T21:49:38.216 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:koLRpWM6rCyT9HBWzv3I7b+djh8= 2023-09-22T21:49:38.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:koLRpWM6rCyT9HBWzv3I7b+djh8=', 'Content-Length': '0'} 2023-09-22T21:49:38.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccb4dae8b84fa32b-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.219 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:38.219 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:38.219 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:49:38.220 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:49:38.220 DEBUG:boto:Method: GET 2023-09-22T21:49:38.220 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:49:38.220 DEBUG:boto:Data: 2023-09-22T21:49:38.220 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.220 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.220 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.220 DEBUG:boto:Params: {} 2023-09-22T21:49:38.220 DEBUG:boto:Token: None 2023-09-22T21:49:38.220 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/ 2023-09-22T21:49:38.221 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU= 2023-09-22T21:49:38.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU=', 'Content-Length': '0'} 2023-09-22T21:49:38.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000800d2932bfe50efc-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.225 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:38.226 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:38.226 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:38.226 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:49:38.226 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:49:38.226 DEBUG:boto:Method: GET 2023-09-22T21:49:38.227 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:49:38.227 DEBUG:boto:Data: 2023-09-22T21:49:38.227 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.227 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.227 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.227 DEBUG:boto:Params: {} 2023-09-22T21:49:38.227 DEBUG:boto:Token: None 2023-09-22T21:49:38.227 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/ 2023-09-22T21:49:38.227 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU= 2023-09-22T21:49:38.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU=', 'Content-Length': '0'} 2023-09-22T21:49:38.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b002fd4168787b2a-00650e0bf2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.237 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:38.238 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:49:38.238 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:49:38.238 DEBUG:boto:Method: HEAD 2023-09-22T21:49:38.238 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:49:38.238 DEBUG:boto:Data: 2023-09-22T21:49:38.238 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.238 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.238 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.239 DEBUG:boto:Params: {} 2023-09-22T21:49:38.239 DEBUG:boto:Token: None 2023-09-22T21:49:38.239 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj17 2023-09-22T21:49:38.239 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9CKLQRljxfc53FdexY052WMITq4= 2023-09-22T21:49:38.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9CKLQRljxfc53FdexY052WMITq4=', 'Content-Length': '0'} 2023-09-22T21:49:38.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000254e2216c4080790-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.242 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:49:38.242 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:49:38.242 DEBUG:boto:Method: HEAD 2023-09-22T21:49:38.242 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:49:38.243 DEBUG:boto:Data: 2023-09-22T21:49:38.243 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.243 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.243 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.243 DEBUG:boto:Params: {} 2023-09-22T21:49:38.243 DEBUG:boto:Token: None 2023-09-22T21:49:38.243 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj17 2023-09-22T21:49:38.243 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9CKLQRljxfc53FdexY052WMITq4= 2023-09-22T21:49:38.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9CKLQRljxfc53FdexY052WMITq4=', 'Content-Length': '0'} 2023-09-22T21:49:38.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016d59aa967e3ca91-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:49:38.246 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:49:38.247 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:49:38.247 DEBUG:boto:Method: GET 2023-09-22T21:49:38.247 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:49:38.247 DEBUG:boto:Data: 2023-09-22T21:49:38.247 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.247 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.247 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.247 DEBUG:boto:Params: {} 2023-09-22T21:49:38.247 DEBUG:boto:Token: None 2023-09-22T21:49:38.248 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj17 2023-09-22T21:49:38.248 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+chiQVbbbg3x8YjYMl/epelp5h8= 2023-09-22T21:49:38.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+chiQVbbbg3x8YjYMl/epelp5h8=', 'Content-Length': '0'} 2023-09-22T21:49:38.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d593cc705e7c30d-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.251 DEBUG:boto:path=/ffzdqu-59/obj17 2023-09-22T21:49:38.251 DEBUG:boto:auth_path=/ffzdqu-59/obj17 2023-09-22T21:49:38.251 DEBUG:boto:Method: GET 2023-09-22T21:49:38.252 DEBUG:boto:Path: /ffzdqu-59/obj17 2023-09-22T21:49:38.252 DEBUG:boto:Data: 2023-09-22T21:49:38.252 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.252 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.252 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.252 DEBUG:boto:Params: {} 2023-09-22T21:49:38.252 DEBUG:boto:Token: None 2023-09-22T21:49:38.252 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj17 2023-09-22T21:49:38.252 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+chiQVbbbg3x8YjYMl/epelp5h8= 2023-09-22T21:49:38.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+chiQVbbbg3x8YjYMl/epelp5h8=', 'Content-Length': '0'} 2023-09-22T21:49:38.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc8f05dc515ae98d-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.302 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:38.303 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:38.303 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:49:38.303 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:49:38.303 DEBUG:boto:Method: GET 2023-09-22T21:49:38.303 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:49:38.303 DEBUG:boto:Data: 2023-09-22T21:49:38.303 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.303 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.303 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.303 DEBUG:boto:Params: {} 2023-09-22T21:49:38.304 DEBUG:boto:Token: None 2023-09-22T21:49:38.304 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/ 2023-09-22T21:49:38.304 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU= 2023-09-22T21:49:38.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU=', 'Content-Length': '0'} 2023-09-22T21:49:38.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce89974a9c343f60-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.309 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:38.309 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:38.310 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:38.310 DEBUG:boto:path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:49:38.310 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj17 2023-09-22T21:49:38.310 DEBUG:boto:Method: GET 2023-09-22T21:49:38.310 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj17 2023-09-22T21:49:38.310 DEBUG:boto:Data: 2023-09-22T21:49:38.310 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.310 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.311 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.311 DEBUG:boto:Params: {} 2023-09-22T21:49:38.311 DEBUG:boto:Token: None 2023-09-22T21:49:38.311 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/ 2023-09-22T21:49:38.311 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU= 2023-09-22T21:49:38.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU=', 'Content-Length': '0'} 2023-09-22T21:49:38.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a975274def17acf-00650e0bf2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.321 DEBUG:boto:b'ffzdqu-59obj171000falseobj172023-09-22T21:36:48.678Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:38.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-09-22T21:49:38.322 DEBUG:boto:path=/ffzdqu-59/obj185 2023-09-22T21:49:38.322 DEBUG:boto:auth_path=/ffzdqu-59/obj185 2023-09-22T21:49:38.322 DEBUG:boto:path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.322 DEBUG:boto:auth_path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.322 DEBUG:boto:Method: GET 2023-09-22T21:49:38.322 DEBUG:boto:Path: /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.322 DEBUG:boto:Data: 2023-09-22T21:49:38.322 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.322 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.322 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.323 DEBUG:boto:Params: {} 2023-09-22T21:49:38.323 DEBUG:boto:Token: None 2023-09-22T21:49:38.323 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.323 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0ZawdHcDtIacrDSQiCT2gwFbOCY= 2023-09-22T21:49:38.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0ZawdHcDtIacrDSQiCT2gwFbOCY=', 'Content-Length': '0'} 2023-09-22T21:49:38.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001827d18e5bbed91e-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.366 DEBUG:boto:path=/ffzdqu-59/obj185 2023-09-22T21:49:38.367 DEBUG:boto:auth_path=/ffzdqu-59/obj185 2023-09-22T21:49:38.367 DEBUG:boto:path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.367 DEBUG:boto:auth_path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.367 DEBUG:boto:Method: GET 2023-09-22T21:49:38.367 DEBUG:boto:Path: /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.367 DEBUG:boto:Data: 2023-09-22T21:49:38.367 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.367 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.368 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.368 DEBUG:boto:Params: {} 2023-09-22T21:49:38.368 DEBUG:boto:Token: None 2023-09-22T21:49:38.368 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.368 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0ZawdHcDtIacrDSQiCT2gwFbOCY= 2023-09-22T21:49:38.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0ZawdHcDtIacrDSQiCT2gwFbOCY=', 'Content-Length': '0'} 2023-09-22T21:49:38.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021d42c03d9db2981-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.371 DEBUG:boto:path=/ffzdqu-59/obj185 2023-09-22T21:49:38.372 DEBUG:boto:auth_path=/ffzdqu-59/obj185 2023-09-22T21:49:38.372 DEBUG:boto:path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.372 DEBUG:boto:auth_path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.372 DEBUG:boto:Method: HEAD 2023-09-22T21:49:38.372 DEBUG:boto:Path: /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.372 DEBUG:boto:Data: 2023-09-22T21:49:38.372 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.372 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.372 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.372 DEBUG:boto:Params: {} 2023-09-22T21:49:38.373 DEBUG:boto:Token: None 2023-09-22T21:49:38.373 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.373 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HcGo5BvqczEjiQQeyWWqvNwkYGI= 2023-09-22T21:49:38.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HcGo5BvqczEjiQQeyWWqvNwkYGI=', 'Content-Length': '0'} 2023-09-22T21:49:38.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c24971f1be98e98-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.376 DEBUG:boto:path=/ffzdqu-59/obj185 2023-09-22T21:49:38.376 DEBUG:boto:auth_path=/ffzdqu-59/obj185 2023-09-22T21:49:38.376 DEBUG:boto:path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.376 DEBUG:boto:auth_path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.376 DEBUG:boto:Method: HEAD 2023-09-22T21:49:38.376 DEBUG:boto:Path: /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.376 DEBUG:boto:Data: 2023-09-22T21:49:38.377 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.377 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.377 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.377 DEBUG:boto:Params: {} 2023-09-22T21:49:38.377 DEBUG:boto:Token: None 2023-09-22T21:49:38.377 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.377 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HcGo5BvqczEjiQQeyWWqvNwkYGI= 2023-09-22T21:49:38.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HcGo5BvqczEjiQQeyWWqvNwkYGI=', 'Content-Length': '0'} 2023-09-22T21:49:38.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040f0fbbb967affe8-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-09-22T21:49:38.380 DEBUG:boto:path=/ffzdqu-59/obj185 2023-09-22T21:49:38.380 DEBUG:boto:auth_path=/ffzdqu-59/obj185 2023-09-22T21:49:38.381 DEBUG:boto:path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.381 DEBUG:boto:auth_path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.381 DEBUG:boto:Method: GET 2023-09-22T21:49:38.381 DEBUG:boto:Path: /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.381 DEBUG:boto:Data: 2023-09-22T21:49:38.381 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.381 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.381 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.381 DEBUG:boto:Params: {} 2023-09-22T21:49:38.381 DEBUG:boto:Token: None 2023-09-22T21:49:38.382 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.382 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0ZawdHcDtIacrDSQiCT2gwFbOCY= 2023-09-22T21:49:38.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0ZawdHcDtIacrDSQiCT2gwFbOCY=', 'Content-Length': '0'} 2023-09-22T21:49:38.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5b139d555d611d6-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.385 DEBUG:boto:path=/ffzdqu-59/obj185 2023-09-22T21:49:38.385 DEBUG:boto:auth_path=/ffzdqu-59/obj185 2023-09-22T21:49:38.385 DEBUG:boto:path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.385 DEBUG:boto:auth_path=/ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.385 DEBUG:boto:Method: GET 2023-09-22T21:49:38.385 DEBUG:boto:Path: /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.385 DEBUG:boto:Data: 2023-09-22T21:49:38.385 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.386 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.386 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.386 DEBUG:boto:Params: {} 2023-09-22T21:49:38.386 DEBUG:boto:Token: None 2023-09-22T21:49:38.386 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj185?versionId=null 2023-09-22T21:49:38.386 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0ZawdHcDtIacrDSQiCT2gwFbOCY= 2023-09-22T21:49:38.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0ZawdHcDtIacrDSQiCT2gwFbOCY=', 'Content-Length': '0'} 2023-09-22T21:49:38.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008234077442dcd4a0-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.430 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:38.430 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:38.431 DEBUG:boto:path=/ffzdqu-59/?prefix=obj185 2023-09-22T21:49:38.431 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj185 2023-09-22T21:49:38.431 DEBUG:boto:Method: GET 2023-09-22T21:49:38.431 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj185 2023-09-22T21:49:38.431 DEBUG:boto:Data: 2023-09-22T21:49:38.431 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.431 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.431 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.431 DEBUG:boto:Params: {} 2023-09-22T21:49:38.431 DEBUG:boto:Token: None 2023-09-22T21:49:38.432 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/ 2023-09-22T21:49:38.432 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU= 2023-09-22T21:49:38.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU=', 'Content-Length': '0'} 2023-09-22T21:49:38.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007504708898e0213a-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.437 DEBUG:boto:b'ffzdqu-59obj1851000falseobj1852023-09-22T21:47:16.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:38.438 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:38.438 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:38.438 DEBUG:boto:path=/ffzdqu-59/?prefix=obj185 2023-09-22T21:49:38.438 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj185 2023-09-22T21:49:38.438 DEBUG:boto:Method: GET 2023-09-22T21:49:38.439 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj185 2023-09-22T21:49:38.439 DEBUG:boto:Data: 2023-09-22T21:49:38.439 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.439 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.439 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.439 DEBUG:boto:Params: {} 2023-09-22T21:49:38.439 DEBUG:boto:Token: None 2023-09-22T21:49:38.439 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/ 2023-09-22T21:49:38.440 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU= 2023-09-22T21:49:38.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU=', 'Content-Length': '0'} 2023-09-22T21:49:38.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc2edf3121c64b26-00650e0bf2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.455 DEBUG:boto:b'ffzdqu-59obj1851000falseobj1852023-09-22T21:47:16.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:38.456 DEBUG:boto:path=/ffzdqu-59/obj185 2023-09-22T21:49:38.456 DEBUG:boto:auth_path=/ffzdqu-59/obj185 2023-09-22T21:49:38.456 DEBUG:boto:Method: HEAD 2023-09-22T21:49:38.456 DEBUG:boto:Path: /ffzdqu-59/obj185 2023-09-22T21:49:38.456 DEBUG:boto:Data: 2023-09-22T21:49:38.456 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.457 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.457 DEBUG:boto:Params: {} 2023-09-22T21:49:38.457 DEBUG:boto:Token: None 2023-09-22T21:49:38.457 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj185 2023-09-22T21:49:38.457 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fb2ph2RAgmgbqOTVu58qBY7QZ9M= 2023-09-22T21:49:38.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fb2ph2RAgmgbqOTVu58qBY7QZ9M=', 'Content-Length': '0'} 2023-09-22T21:49:38.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000998f2860a50db818-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.460 DEBUG:boto:path=/ffzdqu-59/obj185 2023-09-22T21:49:38.460 DEBUG:boto:auth_path=/ffzdqu-59/obj185 2023-09-22T21:49:38.461 DEBUG:boto:Method: HEAD 2023-09-22T21:49:38.461 DEBUG:boto:Path: /ffzdqu-59/obj185 2023-09-22T21:49:38.461 DEBUG:boto:Data: 2023-09-22T21:49:38.461 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.461 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.461 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.461 DEBUG:boto:Params: {} 2023-09-22T21:49:38.461 DEBUG:boto:Token: None 2023-09-22T21:49:38.461 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj185 2023-09-22T21:49:38.462 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fb2ph2RAgmgbqOTVu58qBY7QZ9M= 2023-09-22T21:49:38.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fb2ph2RAgmgbqOTVu58qBY7QZ9M=', 'Content-Length': '0'} 2023-09-22T21:49:38.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097505fac5e33541d-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-09-22T21:49:38.465 DEBUG:boto:path=/ffzdqu-59/obj185 2023-09-22T21:49:38.465 DEBUG:boto:auth_path=/ffzdqu-59/obj185 2023-09-22T21:49:38.465 DEBUG:boto:Method: GET 2023-09-22T21:49:38.465 DEBUG:boto:Path: /ffzdqu-59/obj185 2023-09-22T21:49:38.465 DEBUG:boto:Data: 2023-09-22T21:49:38.465 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.465 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.465 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.465 DEBUG:boto:Params: {} 2023-09-22T21:49:38.466 DEBUG:boto:Token: None 2023-09-22T21:49:38.466 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj185 2023-09-22T21:49:38.466 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:odNBCjPyLEX5yvafS5UGJGq7ec4= 2023-09-22T21:49:38.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:odNBCjPyLEX5yvafS5UGJGq7ec4=', 'Content-Length': '0'} 2023-09-22T21:49:38.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098e3739295e5c836-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.514 DEBUG:boto:path=/ffzdqu-59/obj185 2023-09-22T21:49:38.515 DEBUG:boto:auth_path=/ffzdqu-59/obj185 2023-09-22T21:49:38.515 DEBUG:boto:Method: GET 2023-09-22T21:49:38.515 DEBUG:boto:Path: /ffzdqu-59/obj185 2023-09-22T21:49:38.515 DEBUG:boto:Data: 2023-09-22T21:49:38.515 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.515 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.515 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.515 DEBUG:boto:Params: {} 2023-09-22T21:49:38.515 DEBUG:boto:Token: None 2023-09-22T21:49:38.516 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj185 2023-09-22T21:49:38.516 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:odNBCjPyLEX5yvafS5UGJGq7ec4= 2023-09-22T21:49:38.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:odNBCjPyLEX5yvafS5UGJGq7ec4=', 'Content-Length': '0'} 2023-09-22T21:49:38.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020164175a9eeb0ac-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.522 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:38.522 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:38.522 DEBUG:boto:path=/ffzdqu-59/?prefix=obj185 2023-09-22T21:49:38.522 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj185 2023-09-22T21:49:38.522 DEBUG:boto:Method: GET 2023-09-22T21:49:38.522 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj185 2023-09-22T21:49:38.522 DEBUG:boto:Data: 2023-09-22T21:49:38.522 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.522 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.523 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.523 DEBUG:boto:Params: {} 2023-09-22T21:49:38.523 DEBUG:boto:Token: None 2023-09-22T21:49:38.523 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/ 2023-09-22T21:49:38.523 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU= 2023-09-22T21:49:38.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU=', 'Content-Length': '0'} 2023-09-22T21:49:38.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d523acae5db0a10e-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.528 DEBUG:boto:b'ffzdqu-59obj1851000falseobj1852023-09-22T21:47:16.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:38.529 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:38.529 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:38.529 DEBUG:boto:path=/ffzdqu-59/?prefix=obj185 2023-09-22T21:49:38.529 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj185 2023-09-22T21:49:38.529 DEBUG:boto:Method: GET 2023-09-22T21:49:38.529 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj185 2023-09-22T21:49:38.530 DEBUG:boto:Data: 2023-09-22T21:49:38.530 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.530 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.530 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.530 DEBUG:boto:Params: {} 2023-09-22T21:49:38.530 DEBUG:boto:Token: None 2023-09-22T21:49:38.530 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/ 2023-09-22T21:49:38.530 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU= 2023-09-22T21:49:38.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU=', 'Content-Length': '0'} 2023-09-22T21:49:38.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000add297de8299814d-00650e0bf2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.582 DEBUG:boto:b'ffzdqu-59obj1851000falseobj1852023-09-22T21:47:16.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:38.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T21:49:38.583 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:49:38.583 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:49:38.583 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.583 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.583 DEBUG:boto:Method: GET 2023-09-22T21:49:38.584 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.584 DEBUG:boto:Data: 2023-09-22T21:49:38.584 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.584 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.584 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.584 DEBUG:boto:Params: {} 2023-09-22T21:49:38.584 DEBUG:boto:Token: None 2023-09-22T21:49:38.584 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.584 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r+fmTSG7pVWuHgvxIYQpOJNKWD0= 2023-09-22T21:49:38.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r+fmTSG7pVWuHgvxIYQpOJNKWD0=', 'Content-Length': '0'} 2023-09-22T21:49:38.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088f0edacdf469304-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.588 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:49:38.588 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:49:38.588 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.588 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.589 DEBUG:boto:Method: GET 2023-09-22T21:49:38.589 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.589 DEBUG:boto:Data: 2023-09-22T21:49:38.589 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.589 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.589 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.589 DEBUG:boto:Params: {} 2023-09-22T21:49:38.589 DEBUG:boto:Token: None 2023-09-22T21:49:38.589 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.589 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r+fmTSG7pVWuHgvxIYQpOJNKWD0= 2023-09-22T21:49:38.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r+fmTSG7pVWuHgvxIYQpOJNKWD0=', 'Content-Length': '0'} 2023-09-22T21:49:38.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002292ab2d0bfe28c4-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.594 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:49:38.594 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:49:38.595 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.595 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.595 DEBUG:boto:Method: HEAD 2023-09-22T21:49:38.595 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.595 DEBUG:boto:Data: 2023-09-22T21:49:38.595 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.595 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.595 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.595 DEBUG:boto:Params: {} 2023-09-22T21:49:38.595 DEBUG:boto:Token: None 2023-09-22T21:49:38.596 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.596 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MgQ4ApNy14MXaLc72lWoysZ9tg0= 2023-09-22T21:49:38.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MgQ4ApNy14MXaLc72lWoysZ9tg0=', 'Content-Length': '0'} 2023-09-22T21:49:38.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038515900680fb8b5-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.600 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:49:38.600 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:49:38.600 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.600 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.600 DEBUG:boto:Method: HEAD 2023-09-22T21:49:38.601 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.601 DEBUG:boto:Data: 2023-09-22T21:49:38.601 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.601 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.601 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.601 DEBUG:boto:Params: {} 2023-09-22T21:49:38.601 DEBUG:boto:Token: None 2023-09-22T21:49:38.601 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.602 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MgQ4ApNy14MXaLc72lWoysZ9tg0= 2023-09-22T21:49:38.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MgQ4ApNy14MXaLc72lWoysZ9tg0=', 'Content-Length': '0'} 2023-09-22T21:49:38.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072e8de98e13a6156-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T21:49:38.605 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:49:38.605 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:49:38.605 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.605 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.605 DEBUG:boto:Method: GET 2023-09-22T21:49:38.605 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.605 DEBUG:boto:Data: 2023-09-22T21:49:38.605 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.606 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.606 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.606 DEBUG:boto:Params: {} 2023-09-22T21:49:38.606 DEBUG:boto:Token: None 2023-09-22T21:49:38.606 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.606 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r+fmTSG7pVWuHgvxIYQpOJNKWD0= 2023-09-22T21:49:38.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r+fmTSG7pVWuHgvxIYQpOJNKWD0=', 'Content-Length': '0'} 2023-09-22T21:49:38.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066c811ee9703ec1c-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.650 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:49:38.650 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:49:38.650 DEBUG:boto:path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.651 DEBUG:boto:auth_path=/ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.651 DEBUG:boto:Method: GET 2023-09-22T21:49:38.651 DEBUG:boto:Path: /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.651 DEBUG:boto:Data: 2023-09-22T21:49:38.651 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.651 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.651 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.651 DEBUG:boto:Params: {} 2023-09-22T21:49:38.651 DEBUG:boto:Token: None 2023-09-22T21:49:38.651 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj19?versionId=null 2023-09-22T21:49:38.652 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r+fmTSG7pVWuHgvxIYQpOJNKWD0= 2023-09-22T21:49:38.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r+fmTSG7pVWuHgvxIYQpOJNKWD0=', 'Content-Length': '0'} 2023-09-22T21:49:38.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000aef5563a1e2f35-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.655 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:38.655 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:38.655 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:49:38.655 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:49:38.655 DEBUG:boto:Method: GET 2023-09-22T21:49:38.655 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:49:38.655 DEBUG:boto:Data: 2023-09-22T21:49:38.656 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.656 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.656 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.656 DEBUG:boto:Params: {} 2023-09-22T21:49:38.656 DEBUG:boto:Token: None 2023-09-22T21:49:38.656 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/ 2023-09-22T21:49:38.656 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU= 2023-09-22T21:49:38.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU=', 'Content-Length': '0'} 2023-09-22T21:49:38.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028bd40509e9e75da-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.661 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:38.661 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:38.661 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:38.662 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:49:38.662 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:49:38.662 DEBUG:boto:Method: GET 2023-09-22T21:49:38.662 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:49:38.662 DEBUG:boto:Data: 2023-09-22T21:49:38.662 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.662 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.662 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.662 DEBUG:boto:Params: {} 2023-09-22T21:49:38.663 DEBUG:boto:Token: None 2023-09-22T21:49:38.663 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/ 2023-09-22T21:49:38.663 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU= 2023-09-22T21:49:38.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU=', 'Content-Length': '0'} 2023-09-22T21:49:38.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041cbcd28cee6514a-00650e0bf2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.714 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:38.715 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:49:38.715 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:49:38.715 DEBUG:boto:Method: HEAD 2023-09-22T21:49:38.715 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:49:38.715 DEBUG:boto:Data: 2023-09-22T21:49:38.715 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.715 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.715 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.716 DEBUG:boto:Params: {} 2023-09-22T21:49:38.716 DEBUG:boto:Token: None 2023-09-22T21:49:38.716 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj19 2023-09-22T21:49:38.716 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AyUYhO3l0JejupOja8NY4w6PKeA= 2023-09-22T21:49:38.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AyUYhO3l0JejupOja8NY4w6PKeA=', 'Content-Length': '0'} 2023-09-22T21:49:38.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d891d25538d4f58-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.719 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:49:38.719 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:49:38.719 DEBUG:boto:Method: HEAD 2023-09-22T21:49:38.719 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:49:38.720 DEBUG:boto:Data: 2023-09-22T21:49:38.720 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.720 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.720 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.720 DEBUG:boto:Params: {} 2023-09-22T21:49:38.720 DEBUG:boto:Token: None 2023-09-22T21:49:38.720 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj19 2023-09-22T21:49:38.720 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AyUYhO3l0JejupOja8NY4w6PKeA= 2023-09-22T21:49:38.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AyUYhO3l0JejupOja8NY4w6PKeA=', 'Content-Length': '0'} 2023-09-22T21:49:38.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000537e7763d9a80b25-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T21:49:38.726 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:49:38.726 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:49:38.726 DEBUG:boto:Method: GET 2023-09-22T21:49:38.726 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:49:38.726 DEBUG:boto:Data: 2023-09-22T21:49:38.726 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.726 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.726 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.726 DEBUG:boto:Params: {} 2023-09-22T21:49:38.727 DEBUG:boto:Token: None 2023-09-22T21:49:38.727 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj19 2023-09-22T21:49:38.727 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gEWoGRIH+Ze+2l+J55zxmZuWvM8= 2023-09-22T21:49:38.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gEWoGRIH+Ze+2l+J55zxmZuWvM8=', 'Content-Length': '0'} 2023-09-22T21:49:38.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004d901acf29649a1-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.730 DEBUG:boto:path=/ffzdqu-59/obj19 2023-09-22T21:49:38.730 DEBUG:boto:auth_path=/ffzdqu-59/obj19 2023-09-22T21:49:38.730 DEBUG:boto:Method: GET 2023-09-22T21:49:38.730 DEBUG:boto:Path: /ffzdqu-59/obj19 2023-09-22T21:49:38.730 DEBUG:boto:Data: 2023-09-22T21:49:38.731 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.731 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.731 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.731 DEBUG:boto:Params: {} 2023-09-22T21:49:38.731 DEBUG:boto:Token: None 2023-09-22T21:49:38.731 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj19 2023-09-22T21:49:38.731 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gEWoGRIH+Ze+2l+J55zxmZuWvM8= 2023-09-22T21:49:38.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gEWoGRIH+Ze+2l+J55zxmZuWvM8=', 'Content-Length': '0'} 2023-09-22T21:49:38.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fb429c9fb77269e-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.734 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:38.734 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:38.734 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:49:38.734 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:49:38.735 DEBUG:boto:Method: GET 2023-09-22T21:49:38.735 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:49:38.735 DEBUG:boto:Data: 2023-09-22T21:49:38.735 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.735 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.735 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.735 DEBUG:boto:Params: {} 2023-09-22T21:49:38.735 DEBUG:boto:Token: None 2023-09-22T21:49:38.735 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/ 2023-09-22T21:49:38.735 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU= 2023-09-22T21:49:38.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU=', 'Content-Length': '0'} 2023-09-22T21:49:38.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000979a035ba23f73a4-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.741 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:38.741 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:38.741 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:38.741 DEBUG:boto:path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:49:38.741 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj19 2023-09-22T21:49:38.742 DEBUG:boto:Method: GET 2023-09-22T21:49:38.742 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj19 2023-09-22T21:49:38.742 DEBUG:boto:Data: 2023-09-22T21:49:38.742 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.742 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.742 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.742 DEBUG:boto:Params: {} 2023-09-22T21:49:38.742 DEBUG:boto:Token: None 2023-09-22T21:49:38.742 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/ 2023-09-22T21:49:38.743 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU= 2023-09-22T21:49:38.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU=', 'Content-Length': '0'} 2023-09-22T21:49:38.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf0a86febc22a5ec-00650e0bf2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.753 DEBUG:boto:b'ffzdqu-59obj191000falseobj192023-09-22T21:38:31.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:38.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:49:38.754 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:49:38.754 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:49:38.754 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.754 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.754 DEBUG:boto:Method: GET 2023-09-22T21:49:38.754 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.754 DEBUG:boto:Data: 2023-09-22T21:49:38.755 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.755 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.755 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.755 DEBUG:boto:Params: {} 2023-09-22T21:49:38.755 DEBUG:boto:Token: None 2023-09-22T21:49:38.755 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.755 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:54Q+9CQkoPS4V/kz8dxxSVQfrKk= 2023-09-22T21:49:38.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:54Q+9CQkoPS4V/kz8dxxSVQfrKk=', 'Content-Length': '0'} 2023-09-22T21:49:38.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000621a0083a9c91d61-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.802 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:49:38.802 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:49:38.803 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.803 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.803 DEBUG:boto:Method: GET 2023-09-22T21:49:38.803 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.803 DEBUG:boto:Data: 2023-09-22T21:49:38.803 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.803 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.803 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.803 DEBUG:boto:Params: {} 2023-09-22T21:49:38.804 DEBUG:boto:Token: None 2023-09-22T21:49:38.804 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.804 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:54Q+9CQkoPS4V/kz8dxxSVQfrKk= 2023-09-22T21:49:38.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:54Q+9CQkoPS4V/kz8dxxSVQfrKk=', 'Content-Length': '0'} 2023-09-22T21:49:38.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040759d1fd167d251-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.808 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:49:38.808 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:49:38.808 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.808 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.808 DEBUG:boto:Method: HEAD 2023-09-22T21:49:38.808 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.808 DEBUG:boto:Data: 2023-09-22T21:49:38.808 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.809 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.809 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.809 DEBUG:boto:Params: {} 2023-09-22T21:49:38.809 DEBUG:boto:Token: None 2023-09-22T21:49:38.809 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.809 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rWjJ4KHYqT+dtjP6XXn5qdrzfs0= 2023-09-22T21:49:38.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rWjJ4KHYqT+dtjP6XXn5qdrzfs0=', 'Content-Length': '0'} 2023-09-22T21:49:38.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f990892719fd3e8-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.812 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:49:38.812 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:49:38.812 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.812 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.813 DEBUG:boto:Method: HEAD 2023-09-22T21:49:38.813 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.813 DEBUG:boto:Data: 2023-09-22T21:49:38.813 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.813 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.813 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.813 DEBUG:boto:Params: {} 2023-09-22T21:49:38.813 DEBUG:boto:Token: None 2023-09-22T21:49:38.813 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.814 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rWjJ4KHYqT+dtjP6XXn5qdrzfs0= 2023-09-22T21:49:38.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rWjJ4KHYqT+dtjP6XXn5qdrzfs0=', 'Content-Length': '0'} 2023-09-22T21:49:38.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006156f9ea2508a7ba-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:49:38.817 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:49:38.817 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:49:38.817 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.818 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.818 DEBUG:boto:Method: GET 2023-09-22T21:49:38.818 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.818 DEBUG:boto:Data: 2023-09-22T21:49:38.818 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.818 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.818 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.818 DEBUG:boto:Params: {} 2023-09-22T21:49:38.818 DEBUG:boto:Token: None 2023-09-22T21:49:38.819 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.819 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:54Q+9CQkoPS4V/kz8dxxSVQfrKk= 2023-09-22T21:49:38.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:54Q+9CQkoPS4V/kz8dxxSVQfrKk=', 'Content-Length': '0'} 2023-09-22T21:49:38.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b10a2d72e235bf07-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.822 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:49:38.822 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:49:38.822 DEBUG:boto:path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.822 DEBUG:boto:auth_path=/ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.822 DEBUG:boto:Method: GET 2023-09-22T21:49:38.822 DEBUG:boto:Path: /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.822 DEBUG:boto:Data: 2023-09-22T21:49:38.823 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.823 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.823 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.823 DEBUG:boto:Params: {} 2023-09-22T21:49:38.823 DEBUG:boto:Token: None 2023-09-22T21:49:38.823 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj22?versionId=null 2023-09-22T21:49:38.823 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:54Q+9CQkoPS4V/kz8dxxSVQfrKk= 2023-09-22T21:49:38.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:54Q+9CQkoPS4V/kz8dxxSVQfrKk=', 'Content-Length': '0'} 2023-09-22T21:49:38.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8f740d25da72eda-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.870 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:38.871 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:38.871 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:49:38.871 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:49:38.871 DEBUG:boto:Method: GET 2023-09-22T21:49:38.871 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:49:38.871 DEBUG:boto:Data: 2023-09-22T21:49:38.871 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.871 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.871 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.871 DEBUG:boto:Params: {} 2023-09-22T21:49:38.872 DEBUG:boto:Token: None 2023-09-22T21:49:38.872 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/ 2023-09-22T21:49:38.872 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU= 2023-09-22T21:49:38.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU=', 'Content-Length': '0'} 2023-09-22T21:49:38.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057a95609fdfb5d67-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.877 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:38.877 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:38.877 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:38.877 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:49:38.877 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:49:38.878 DEBUG:boto:Method: GET 2023-09-22T21:49:38.878 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:49:38.878 DEBUG:boto:Data: 2023-09-22T21:49:38.878 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.878 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.878 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.878 DEBUG:boto:Params: {} 2023-09-22T21:49:38.878 DEBUG:boto:Token: None 2023-09-22T21:49:38.879 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/ 2023-09-22T21:49:38.879 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU= 2023-09-22T21:49:38.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU=', 'Content-Length': '0'} 2023-09-22T21:49:38.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a2e9f5fcd0a4962-00650e0bf2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.889 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:38.889 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:49:38.889 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:49:38.889 DEBUG:boto:Method: HEAD 2023-09-22T21:49:38.890 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:49:38.890 DEBUG:boto:Data: 2023-09-22T21:49:38.890 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.890 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.890 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.890 DEBUG:boto:Params: {} 2023-09-22T21:49:38.890 DEBUG:boto:Token: None 2023-09-22T21:49:38.890 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj22 2023-09-22T21:49:38.891 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Irk9YOmvRj3s1CBDml0blABLOW8= 2023-09-22T21:49:38.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Irk9YOmvRj3s1CBDml0blABLOW8=', 'Content-Length': '0'} 2023-09-22T21:49:38.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d936e606619740ac-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.894 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:49:38.894 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:49:38.894 DEBUG:boto:Method: HEAD 2023-09-22T21:49:38.894 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:49:38.894 DEBUG:boto:Data: 2023-09-22T21:49:38.894 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.894 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.894 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.895 DEBUG:boto:Params: {} 2023-09-22T21:49:38.895 DEBUG:boto:Token: None 2023-09-22T21:49:38.895 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj22 2023-09-22T21:49:38.895 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Irk9YOmvRj3s1CBDml0blABLOW8= 2023-09-22T21:49:38.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Irk9YOmvRj3s1CBDml0blABLOW8=', 'Content-Length': '0'} 2023-09-22T21:49:38.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b600e40710f446e6-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:49:38.898 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:49:38.898 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:49:38.898 DEBUG:boto:Method: GET 2023-09-22T21:49:38.898 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:49:38.898 DEBUG:boto:Data: 2023-09-22T21:49:38.898 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.899 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.899 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.899 DEBUG:boto:Params: {} 2023-09-22T21:49:38.899 DEBUG:boto:Token: None 2023-09-22T21:49:38.899 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj22 2023-09-22T21:49:38.899 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y1HhCAr2Sidc43oYY/A2OzDv5vs= 2023-09-22T21:49:38.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y1HhCAr2Sidc43oYY/A2OzDv5vs=', 'Content-Length': '0'} 2023-09-22T21:49:38.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067e52066c7d183f0-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.946 DEBUG:boto:path=/ffzdqu-59/obj22 2023-09-22T21:49:38.947 DEBUG:boto:auth_path=/ffzdqu-59/obj22 2023-09-22T21:49:38.947 DEBUG:boto:Method: GET 2023-09-22T21:49:38.947 DEBUG:boto:Path: /ffzdqu-59/obj22 2023-09-22T21:49:38.947 DEBUG:boto:Data: 2023-09-22T21:49:38.947 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.947 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.947 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.947 DEBUG:boto:Params: {} 2023-09-22T21:49:38.947 DEBUG:boto:Token: None 2023-09-22T21:49:38.948 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/obj22 2023-09-22T21:49:38.948 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y1HhCAr2Sidc43oYY/A2OzDv5vs= 2023-09-22T21:49:38.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y1HhCAr2Sidc43oYY/A2OzDv5vs=', 'Content-Length': '0'} 2023-09-22T21:49:38.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6ea98630bce1224-00650e0bf2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.951 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:38.951 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:38.951 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:49:38.951 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:49:38.951 DEBUG:boto:Method: GET 2023-09-22T21:49:38.952 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:49:38.952 DEBUG:boto:Data: 2023-09-22T21:49:38.952 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.952 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:38.952 DEBUG:boto:Port: 8001 2023-09-22T21:49:38.952 DEBUG:boto:Params: {} 2023-09-22T21:49:38.952 DEBUG:boto:Token: None 2023-09-22T21:49:38.952 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/ 2023-09-22T21:49:38.953 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU= 2023-09-22T21:49:38.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU=', 'Content-Length': '0'} 2023-09-22T21:49:38.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e85ff67a46fd616-00650e0bf2-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:38.957 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:38.958 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:38.958 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:38.958 DEBUG:boto:path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:49:38.958 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj22 2023-09-22T21:49:38.958 DEBUG:boto:Method: GET 2023-09-22T21:49:38.958 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj22 2023-09-22T21:49:38.959 DEBUG:boto:Data: 2023-09-22T21:49:38.959 DEBUG:boto:Headers: {} 2023-09-22T21:49:38.959 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:38.959 DEBUG:boto:Port: 8000 2023-09-22T21:49:38.959 DEBUG:boto:Params: {} 2023-09-22T21:49:38.959 DEBUG:boto:Token: None 2023-09-22T21:49:38.959 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:38 GMT /ffzdqu-59/ 2023-09-22T21:49:38.959 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU= 2023-09-22T21:49:38.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vj8wD81U90NLHHO0rIMT+V+ciZU=', 'Content-Length': '0'} 2023-09-22T21:49:38.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000823c69f60fb1d43d-00650e0bf2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.010 DEBUG:boto:b'ffzdqu-59obj221000falseobj222023-09-22T21:36:31.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:39.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-22T21:49:39.011 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:49:39.011 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:49:39.011 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.011 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.011 DEBUG:boto:Method: GET 2023-09-22T21:49:39.011 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.012 DEBUG:boto:Data: 2023-09-22T21:49:39.012 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.012 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.012 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.012 DEBUG:boto:Params: {} 2023-09-22T21:49:39.012 DEBUG:boto:Token: None 2023-09-22T21:49:39.012 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.012 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Hgig0AzzqJnUpTek+NBPyXYkyms= 2023-09-22T21:49:39.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Hgig0AzzqJnUpTek+NBPyXYkyms=', 'Content-Length': '0'} 2023-09-22T21:49:39.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1d34521c375c98b-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.016 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:49:39.016 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:49:39.016 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.016 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.017 DEBUG:boto:Method: GET 2023-09-22T21:49:39.017 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.017 DEBUG:boto:Data: 2023-09-22T21:49:39.017 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.017 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.017 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.017 DEBUG:boto:Params: {} 2023-09-22T21:49:39.017 DEBUG:boto:Token: None 2023-09-22T21:49:39.017 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.018 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Hgig0AzzqJnUpTek+NBPyXYkyms= 2023-09-22T21:49:39.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Hgig0AzzqJnUpTek+NBPyXYkyms=', 'Content-Length': '0'} 2023-09-22T21:49:39.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005774f35ba0578b40-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.021 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:49:39.021 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:49:39.021 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.021 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.021 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.022 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.022 DEBUG:boto:Data: 2023-09-22T21:49:39.022 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.022 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.022 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.022 DEBUG:boto:Params: {} 2023-09-22T21:49:39.022 DEBUG:boto:Token: None 2023-09-22T21:49:39.022 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.023 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GWDUar45SXjFbRfQtysF51Rv7F0= 2023-09-22T21:49:39.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GWDUar45SXjFbRfQtysF51Rv7F0=', 'Content-Length': '0'} 2023-09-22T21:49:39.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024e7a4ff3464d7dc-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.026 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:49:39.026 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:49:39.026 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.026 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.026 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.026 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.027 DEBUG:boto:Data: 2023-09-22T21:49:39.027 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.027 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.027 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.027 DEBUG:boto:Params: {} 2023-09-22T21:49:39.027 DEBUG:boto:Token: None 2023-09-22T21:49:39.027 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.027 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GWDUar45SXjFbRfQtysF51Rv7F0= 2023-09-22T21:49:39.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GWDUar45SXjFbRfQtysF51Rv7F0=', 'Content-Length': '0'} 2023-09-22T21:49:39.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009201979505097160-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-22T21:49:39.031 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:49:39.031 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:49:39.031 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.031 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.031 DEBUG:boto:Method: GET 2023-09-22T21:49:39.031 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.031 DEBUG:boto:Data: 2023-09-22T21:49:39.031 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.032 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.032 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.032 DEBUG:boto:Params: {} 2023-09-22T21:49:39.032 DEBUG:boto:Token: None 2023-09-22T21:49:39.032 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.032 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Hgig0AzzqJnUpTek+NBPyXYkyms= 2023-09-22T21:49:39.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Hgig0AzzqJnUpTek+NBPyXYkyms=', 'Content-Length': '0'} 2023-09-22T21:49:39.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067fec9dcf0f423cc-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.035 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:49:39.036 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:49:39.036 DEBUG:boto:path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.036 DEBUG:boto:auth_path=/ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.036 DEBUG:boto:Method: GET 2023-09-22T21:49:39.036 DEBUG:boto:Path: /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.036 DEBUG:boto:Data: 2023-09-22T21:49:39.036 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.036 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.036 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.036 DEBUG:boto:Params: {} 2023-09-22T21:49:39.037 DEBUG:boto:Token: None 2023-09-22T21:49:39.037 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj23?versionId=null 2023-09-22T21:49:39.037 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Hgig0AzzqJnUpTek+NBPyXYkyms= 2023-09-22T21:49:39.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Hgig0AzzqJnUpTek+NBPyXYkyms=', 'Content-Length': '0'} 2023-09-22T21:49:39.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a4cbb2dbdf7b675-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.040 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:39.040 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:39.041 DEBUG:boto:path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:49:39.041 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:49:39.041 DEBUG:boto:Method: GET 2023-09-22T21:49:39.041 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj23 2023-09-22T21:49:39.041 DEBUG:boto:Data: 2023-09-22T21:49:39.041 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.041 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.041 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.041 DEBUG:boto:Params: {} 2023-09-22T21:49:39.041 DEBUG:boto:Token: None 2023-09-22T21:49:39.042 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/ 2023-09-22T21:49:39.042 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU= 2023-09-22T21:49:39.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU=', 'Content-Length': '0'} 2023-09-22T21:49:39.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000530c7d2ce5ab91ca-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.090 DEBUG:boto:b'ffzdqu-59obj231000falseobj232023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:39.091 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:39.091 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:39.091 DEBUG:boto:path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:49:39.091 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:49:39.091 DEBUG:boto:Method: GET 2023-09-22T21:49:39.091 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj23 2023-09-22T21:49:39.091 DEBUG:boto:Data: 2023-09-22T21:49:39.091 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.092 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.092 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.092 DEBUG:boto:Params: {} 2023-09-22T21:49:39.092 DEBUG:boto:Token: None 2023-09-22T21:49:39.092 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/ 2023-09-22T21:49:39.092 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU= 2023-09-22T21:49:39.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU=', 'Content-Length': '0'} 2023-09-22T21:49:39.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b87daf4f250d25e9-00650e0bf3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.101 DEBUG:boto:b'ffzdqu-59obj231000falseobj232023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:39.102 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:49:39.102 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:49:39.102 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.102 DEBUG:boto:Path: /ffzdqu-59/obj23 2023-09-22T21:49:39.102 DEBUG:boto:Data: 2023-09-22T21:49:39.102 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.103 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.103 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.103 DEBUG:boto:Params: {} 2023-09-22T21:49:39.103 DEBUG:boto:Token: None 2023-09-22T21:49:39.103 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj23 2023-09-22T21:49:39.103 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8rrXd4f/BilGapMYejQ1soXyGd4= 2023-09-22T21:49:39.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8rrXd4f/BilGapMYejQ1soXyGd4=', 'Content-Length': '0'} 2023-09-22T21:49:39.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e973a075a6a5659-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.106 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:49:39.106 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:49:39.106 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.107 DEBUG:boto:Path: /ffzdqu-59/obj23 2023-09-22T21:49:39.107 DEBUG:boto:Data: 2023-09-22T21:49:39.107 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.107 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.107 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.107 DEBUG:boto:Params: {} 2023-09-22T21:49:39.107 DEBUG:boto:Token: None 2023-09-22T21:49:39.107 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj23 2023-09-22T21:49:39.107 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8rrXd4f/BilGapMYejQ1soXyGd4= 2023-09-22T21:49:39.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8rrXd4f/BilGapMYejQ1soXyGd4=', 'Content-Length': '0'} 2023-09-22T21:49:39.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1bc9c9984ab2815-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-22T21:49:39.111 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:49:39.111 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:49:39.111 DEBUG:boto:Method: GET 2023-09-22T21:49:39.111 DEBUG:boto:Path: /ffzdqu-59/obj23 2023-09-22T21:49:39.111 DEBUG:boto:Data: 2023-09-22T21:49:39.111 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.111 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.111 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.111 DEBUG:boto:Params: {} 2023-09-22T21:49:39.112 DEBUG:boto:Token: None 2023-09-22T21:49:39.112 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj23 2023-09-22T21:49:39.112 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D0TFC3PrW4P26OUxsxgwz2kWris= 2023-09-22T21:49:39.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D0TFC3PrW4P26OUxsxgwz2kWris=', 'Content-Length': '0'} 2023-09-22T21:49:39.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000779aeeb95af3b10c-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.115 DEBUG:boto:path=/ffzdqu-59/obj23 2023-09-22T21:49:39.115 DEBUG:boto:auth_path=/ffzdqu-59/obj23 2023-09-22T21:49:39.115 DEBUG:boto:Method: GET 2023-09-22T21:49:39.115 DEBUG:boto:Path: /ffzdqu-59/obj23 2023-09-22T21:49:39.115 DEBUG:boto:Data: 2023-09-22T21:49:39.115 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.116 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.116 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.116 DEBUG:boto:Params: {} 2023-09-22T21:49:39.116 DEBUG:boto:Token: None 2023-09-22T21:49:39.116 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj23 2023-09-22T21:49:39.116 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D0TFC3PrW4P26OUxsxgwz2kWris= 2023-09-22T21:49:39.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D0TFC3PrW4P26OUxsxgwz2kWris=', 'Content-Length': '0'} 2023-09-22T21:49:39.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024f136dd9ccdddb7-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.162 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:39.162 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:39.163 DEBUG:boto:path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:49:39.163 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:49:39.163 DEBUG:boto:Method: GET 2023-09-22T21:49:39.163 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj23 2023-09-22T21:49:39.163 DEBUG:boto:Data: 2023-09-22T21:49:39.163 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.163 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.163 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.163 DEBUG:boto:Params: {} 2023-09-22T21:49:39.164 DEBUG:boto:Token: None 2023-09-22T21:49:39.164 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/ 2023-09-22T21:49:39.164 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU= 2023-09-22T21:49:39.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU=', 'Content-Length': '0'} 2023-09-22T21:49:39.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e58551cd916c1bf-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.169 DEBUG:boto:b'ffzdqu-59obj231000falseobj232023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:39.169 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:39.169 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:39.169 DEBUG:boto:path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:49:39.169 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj23 2023-09-22T21:49:39.169 DEBUG:boto:Method: GET 2023-09-22T21:49:39.170 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj23 2023-09-22T21:49:39.170 DEBUG:boto:Data: 2023-09-22T21:49:39.170 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.170 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.170 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.170 DEBUG:boto:Params: {} 2023-09-22T21:49:39.170 DEBUG:boto:Token: None 2023-09-22T21:49:39.170 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/ 2023-09-22T21:49:39.170 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU= 2023-09-22T21:49:39.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU=', 'Content-Length': '0'} 2023-09-22T21:49:39.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036ef06af318478e0-00650e0bf3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.180 DEBUG:boto:b'ffzdqu-59obj231000falseobj232023-09-22T21:40:21.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:39.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-22T21:49:39.181 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:49:39.181 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:49:39.181 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.181 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.181 DEBUG:boto:Method: GET 2023-09-22T21:49:39.182 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.182 DEBUG:boto:Data: 2023-09-22T21:49:39.182 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.182 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.182 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.182 DEBUG:boto:Params: {} 2023-09-22T21:49:39.182 DEBUG:boto:Token: None 2023-09-22T21:49:39.182 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.182 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:95njpviPcla4jAmTtqMftWJ0IFM= 2023-09-22T21:49:39.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:95njpviPcla4jAmTtqMftWJ0IFM=', 'Content-Length': '0'} 2023-09-22T21:49:39.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054084ee58ac79ba1-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.186 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:49:39.186 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:49:39.186 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.186 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.187 DEBUG:boto:Method: GET 2023-09-22T21:49:39.187 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.187 DEBUG:boto:Data: 2023-09-22T21:49:39.187 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.187 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.187 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.187 DEBUG:boto:Params: {} 2023-09-22T21:49:39.187 DEBUG:boto:Token: None 2023-09-22T21:49:39.187 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.188 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:95njpviPcla4jAmTtqMftWJ0IFM= 2023-09-22T21:49:39.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:95njpviPcla4jAmTtqMftWJ0IFM=', 'Content-Length': '0'} 2023-09-22T21:49:39.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000026497b92ab7ca94-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.191 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:49:39.191 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:49:39.191 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.191 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.191 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.192 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.192 DEBUG:boto:Data: 2023-09-22T21:49:39.192 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.192 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.192 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.192 DEBUG:boto:Params: {} 2023-09-22T21:49:39.192 DEBUG:boto:Token: None 2023-09-22T21:49:39.192 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.192 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fyw9/Js96RAeg6h+Cg4j0BGt4Aw= 2023-09-22T21:49:39.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fyw9/Js96RAeg6h+Cg4j0BGt4Aw=', 'Content-Length': '0'} 2023-09-22T21:49:39.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ffb6e47681de037-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.195 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:49:39.196 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:49:39.196 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.196 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.196 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.196 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.196 DEBUG:boto:Data: 2023-09-22T21:49:39.196 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.196 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.196 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.196 DEBUG:boto:Params: {} 2023-09-22T21:49:39.197 DEBUG:boto:Token: None 2023-09-22T21:49:39.197 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.197 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fyw9/Js96RAeg6h+Cg4j0BGt4Aw= 2023-09-22T21:49:39.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fyw9/Js96RAeg6h+Cg4j0BGt4Aw=', 'Content-Length': '0'} 2023-09-22T21:49:39.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b4cd93e37632bd0-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-22T21:49:39.200 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:49:39.200 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:49:39.200 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.200 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.200 DEBUG:boto:Method: GET 2023-09-22T21:49:39.200 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.201 DEBUG:boto:Data: 2023-09-22T21:49:39.201 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.201 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.201 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.201 DEBUG:boto:Params: {} 2023-09-22T21:49:39.201 DEBUG:boto:Token: None 2023-09-22T21:49:39.201 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.201 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:95njpviPcla4jAmTtqMftWJ0IFM= 2023-09-22T21:49:39.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:95njpviPcla4jAmTtqMftWJ0IFM=', 'Content-Length': '0'} 2023-09-22T21:49:39.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc0453308fd4ab90-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.246 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:49:39.246 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:49:39.247 DEBUG:boto:path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.247 DEBUG:boto:auth_path=/ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.247 DEBUG:boto:Method: GET 2023-09-22T21:49:39.247 DEBUG:boto:Path: /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.247 DEBUG:boto:Data: 2023-09-22T21:49:39.247 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.247 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.247 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.247 DEBUG:boto:Params: {} 2023-09-22T21:49:39.247 DEBUG:boto:Token: None 2023-09-22T21:49:39.248 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj29?versionId=null 2023-09-22T21:49:39.248 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:95njpviPcla4jAmTtqMftWJ0IFM= 2023-09-22T21:49:39.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:95njpviPcla4jAmTtqMftWJ0IFM=', 'Content-Length': '0'} 2023-09-22T21:49:39.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e7bf5179e402fcc-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.251 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:39.251 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:39.252 DEBUG:boto:path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:49:39.252 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:49:39.252 DEBUG:boto:Method: GET 2023-09-22T21:49:39.252 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj29 2023-09-22T21:49:39.252 DEBUG:boto:Data: 2023-09-22T21:49:39.252 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.252 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.252 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.252 DEBUG:boto:Params: {} 2023-09-22T21:49:39.253 DEBUG:boto:Token: None 2023-09-22T21:49:39.253 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/ 2023-09-22T21:49:39.253 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU= 2023-09-22T21:49:39.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU=', 'Content-Length': '0'} 2023-09-22T21:49:39.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000704bb1ea31ede711-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.258 DEBUG:boto:b'ffzdqu-59obj291000falseobj292023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:39.258 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:39.258 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:39.259 DEBUG:boto:path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:49:39.259 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:49:39.259 DEBUG:boto:Method: GET 2023-09-22T21:49:39.259 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj29 2023-09-22T21:49:39.259 DEBUG:boto:Data: 2023-09-22T21:49:39.259 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.259 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.259 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.259 DEBUG:boto:Params: {} 2023-09-22T21:49:39.259 DEBUG:boto:Token: None 2023-09-22T21:49:39.260 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/ 2023-09-22T21:49:39.260 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU= 2023-09-22T21:49:39.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU=', 'Content-Length': '0'} 2023-09-22T21:49:39.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bacd57ac133b2884-00650e0bf3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.270 DEBUG:boto:b'ffzdqu-59obj291000falseobj292023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:39.270 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:49:39.270 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:49:39.270 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.270 DEBUG:boto:Path: /ffzdqu-59/obj29 2023-09-22T21:49:39.271 DEBUG:boto:Data: 2023-09-22T21:49:39.271 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.271 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.271 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.271 DEBUG:boto:Params: {} 2023-09-22T21:49:39.271 DEBUG:boto:Token: None 2023-09-22T21:49:39.271 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj29 2023-09-22T21:49:39.271 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m3Y1mdqpkW0P1NCId8mk1wg9Grk= 2023-09-22T21:49:39.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m3Y1mdqpkW0P1NCId8mk1wg9Grk=', 'Content-Length': '0'} 2023-09-22T21:49:39.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000504f2dafbb455c33-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.274 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:49:39.274 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:49:39.275 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.275 DEBUG:boto:Path: /ffzdqu-59/obj29 2023-09-22T21:49:39.275 DEBUG:boto:Data: 2023-09-22T21:49:39.275 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.275 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.275 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.275 DEBUG:boto:Params: {} 2023-09-22T21:49:39.275 DEBUG:boto:Token: None 2023-09-22T21:49:39.275 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj29 2023-09-22T21:49:39.276 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m3Y1mdqpkW0P1NCId8mk1wg9Grk= 2023-09-22T21:49:39.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m3Y1mdqpkW0P1NCId8mk1wg9Grk=', 'Content-Length': '0'} 2023-09-22T21:49:39.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049ef978b79cf8342-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-22T21:49:39.279 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:49:39.279 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:49:39.279 DEBUG:boto:Method: GET 2023-09-22T21:49:39.279 DEBUG:boto:Path: /ffzdqu-59/obj29 2023-09-22T21:49:39.279 DEBUG:boto:Data: 2023-09-22T21:49:39.279 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.279 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.279 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.279 DEBUG:boto:Params: {} 2023-09-22T21:49:39.280 DEBUG:boto:Token: None 2023-09-22T21:49:39.280 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj29 2023-09-22T21:49:39.280 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qWTAk7dN38VsmeO3mjk8jDpXlmE= 2023-09-22T21:49:39.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qWTAk7dN38VsmeO3mjk8jDpXlmE=', 'Content-Length': '0'} 2023-09-22T21:49:39.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004895f200b62f86b1-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.283 DEBUG:boto:path=/ffzdqu-59/obj29 2023-09-22T21:49:39.283 DEBUG:boto:auth_path=/ffzdqu-59/obj29 2023-09-22T21:49:39.283 DEBUG:boto:Method: GET 2023-09-22T21:49:39.283 DEBUG:boto:Path: /ffzdqu-59/obj29 2023-09-22T21:49:39.283 DEBUG:boto:Data: 2023-09-22T21:49:39.284 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.284 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.284 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.284 DEBUG:boto:Params: {} 2023-09-22T21:49:39.284 DEBUG:boto:Token: None 2023-09-22T21:49:39.284 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj29 2023-09-22T21:49:39.284 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qWTAk7dN38VsmeO3mjk8jDpXlmE= 2023-09-22T21:49:39.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qWTAk7dN38VsmeO3mjk8jDpXlmE=', 'Content-Length': '0'} 2023-09-22T21:49:39.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce3c15355c3be5ff-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.334 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:39.334 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:39.335 DEBUG:boto:path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:49:39.335 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:49:39.335 DEBUG:boto:Method: GET 2023-09-22T21:49:39.335 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj29 2023-09-22T21:49:39.335 DEBUG:boto:Data: 2023-09-22T21:49:39.335 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.335 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.335 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.335 DEBUG:boto:Params: {} 2023-09-22T21:49:39.336 DEBUG:boto:Token: None 2023-09-22T21:49:39.336 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/ 2023-09-22T21:49:39.336 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU= 2023-09-22T21:49:39.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU=', 'Content-Length': '0'} 2023-09-22T21:49:39.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd9237929eb81fba-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.341 DEBUG:boto:b'ffzdqu-59obj291000falseobj292023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:39.342 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:39.342 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:39.342 DEBUG:boto:path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:49:39.342 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj29 2023-09-22T21:49:39.342 DEBUG:boto:Method: GET 2023-09-22T21:49:39.342 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj29 2023-09-22T21:49:39.342 DEBUG:boto:Data: 2023-09-22T21:49:39.342 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.343 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.343 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.343 DEBUG:boto:Params: {} 2023-09-22T21:49:39.343 DEBUG:boto:Token: None 2023-09-22T21:49:39.343 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/ 2023-09-22T21:49:39.343 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU= 2023-09-22T21:49:39.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU=', 'Content-Length': '0'} 2023-09-22T21:49:39.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c00c5124d1187787-00650e0bf3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.353 DEBUG:boto:b'ffzdqu-59obj291000falseobj292023-09-22T21:43:12.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:39.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-22T21:49:39.353 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:49:39.354 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:49:39.354 DEBUG:boto:path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.354 DEBUG:boto:auth_path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.354 DEBUG:boto:Method: GET 2023-09-22T21:49:39.354 DEBUG:boto:Path: /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.354 DEBUG:boto:Data: 2023-09-22T21:49:39.354 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.354 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.354 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.354 DEBUG:boto:Params: {} 2023-09-22T21:49:39.355 DEBUG:boto:Token: None 2023-09-22T21:49:39.355 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.355 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EenYo0MT1LYc3kwzBIFHchi8ZOg= 2023-09-22T21:49:39.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EenYo0MT1LYc3kwzBIFHchi8ZOg=', 'Content-Length': '0'} 2023-09-22T21:49:39.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058d11e922f192f8c-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.402 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:49:39.402 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:49:39.403 DEBUG:boto:path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.403 DEBUG:boto:auth_path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.403 DEBUG:boto:Method: GET 2023-09-22T21:49:39.403 DEBUG:boto:Path: /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.403 DEBUG:boto:Data: 2023-09-22T21:49:39.403 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.403 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.403 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.403 DEBUG:boto:Params: {} 2023-09-22T21:49:39.404 DEBUG:boto:Token: None 2023-09-22T21:49:39.404 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.404 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EenYo0MT1LYc3kwzBIFHchi8ZOg= 2023-09-22T21:49:39.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EenYo0MT1LYc3kwzBIFHchi8ZOg=', 'Content-Length': '0'} 2023-09-22T21:49:39.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce01c59c74a59889-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.407 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:49:39.407 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:49:39.407 DEBUG:boto:path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.407 DEBUG:boto:auth_path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.408 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.408 DEBUG:boto:Path: /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.408 DEBUG:boto:Data: 2023-09-22T21:49:39.408 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.408 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.408 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.408 DEBUG:boto:Params: {} 2023-09-22T21:49:39.408 DEBUG:boto:Token: None 2023-09-22T21:49:39.408 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.409 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:O1KGSO1TCXRSCnS8XIZIiH2JsyY= 2023-09-22T21:49:39.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:O1KGSO1TCXRSCnS8XIZIiH2JsyY=', 'Content-Length': '0'} 2023-09-22T21:49:39.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e903ff4266f085d-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.411 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:49:39.412 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:49:39.412 DEBUG:boto:path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.412 DEBUG:boto:auth_path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.412 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.412 DEBUG:boto:Path: /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.412 DEBUG:boto:Data: 2023-09-22T21:49:39.412 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.412 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.412 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.413 DEBUG:boto:Params: {} 2023-09-22T21:49:39.413 DEBUG:boto:Token: None 2023-09-22T21:49:39.413 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.413 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:O1KGSO1TCXRSCnS8XIZIiH2JsyY= 2023-09-22T21:49:39.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:O1KGSO1TCXRSCnS8XIZIiH2JsyY=', 'Content-Length': '0'} 2023-09-22T21:49:39.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001409431f8806dbce-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-22T21:49:39.416 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:49:39.416 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:49:39.416 DEBUG:boto:path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.416 DEBUG:boto:auth_path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.417 DEBUG:boto:Method: GET 2023-09-22T21:49:39.417 DEBUG:boto:Path: /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.417 DEBUG:boto:Data: 2023-09-22T21:49:39.417 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.417 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.417 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.417 DEBUG:boto:Params: {} 2023-09-22T21:49:39.417 DEBUG:boto:Token: None 2023-09-22T21:49:39.417 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.418 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EenYo0MT1LYc3kwzBIFHchi8ZOg= 2023-09-22T21:49:39.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EenYo0MT1LYc3kwzBIFHchi8ZOg=', 'Content-Length': '0'} 2023-09-22T21:49:39.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000139fef008388ad6c-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.420 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:49:39.421 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:49:39.421 DEBUG:boto:path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.421 DEBUG:boto:auth_path=/ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.421 DEBUG:boto:Method: GET 2023-09-22T21:49:39.421 DEBUG:boto:Path: /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.421 DEBUG:boto:Data: 2023-09-22T21:49:39.421 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.421 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.421 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.421 DEBUG:boto:Params: {} 2023-09-22T21:49:39.422 DEBUG:boto:Token: None 2023-09-22T21:49:39.422 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj31?versionId=null 2023-09-22T21:49:39.422 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EenYo0MT1LYc3kwzBIFHchi8ZOg= 2023-09-22T21:49:39.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EenYo0MT1LYc3kwzBIFHchi8ZOg=', 'Content-Length': '0'} 2023-09-22T21:49:39.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc44ef01b23ef17a-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.466 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:39.466 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:39.467 DEBUG:boto:path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:49:39.467 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:49:39.467 DEBUG:boto:Method: GET 2023-09-22T21:49:39.467 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj31 2023-09-22T21:49:39.467 DEBUG:boto:Data: 2023-09-22T21:49:39.467 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.467 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.467 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.467 DEBUG:boto:Params: {} 2023-09-22T21:49:39.467 DEBUG:boto:Token: None 2023-09-22T21:49:39.468 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/ 2023-09-22T21:49:39.468 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU= 2023-09-22T21:49:39.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU=', 'Content-Length': '0'} 2023-09-22T21:49:39.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abc8d8b86ef7cc3a-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.473 DEBUG:boto:b'ffzdqu-59obj311000falseobj312023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:39.473 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:39.473 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:39.473 DEBUG:boto:path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:49:39.473 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:49:39.474 DEBUG:boto:Method: GET 2023-09-22T21:49:39.474 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj31 2023-09-22T21:49:39.474 DEBUG:boto:Data: 2023-09-22T21:49:39.474 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.474 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.474 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.474 DEBUG:boto:Params: {} 2023-09-22T21:49:39.474 DEBUG:boto:Token: None 2023-09-22T21:49:39.474 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/ 2023-09-22T21:49:39.475 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU= 2023-09-22T21:49:39.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU=', 'Content-Length': '0'} 2023-09-22T21:49:39.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011f451083616d860-00650e0bf3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.485 DEBUG:boto:b'ffzdqu-59obj311000falseobj312023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:39.485 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:49:39.485 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:49:39.485 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.486 DEBUG:boto:Path: /ffzdqu-59/obj31 2023-09-22T21:49:39.486 DEBUG:boto:Data: 2023-09-22T21:49:39.486 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.486 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.486 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.486 DEBUG:boto:Params: {} 2023-09-22T21:49:39.486 DEBUG:boto:Token: None 2023-09-22T21:49:39.486 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj31 2023-09-22T21:49:39.486 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bAh5aKScKFAbpJbGyoLKoQeDGdU= 2023-09-22T21:49:39.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bAh5aKScKFAbpJbGyoLKoQeDGdU=', 'Content-Length': '0'} 2023-09-22T21:49:39.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000d1d8cb1e98ce2b-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.489 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:49:39.489 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:49:39.490 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.490 DEBUG:boto:Path: /ffzdqu-59/obj31 2023-09-22T21:49:39.490 DEBUG:boto:Data: 2023-09-22T21:49:39.490 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.490 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.490 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.490 DEBUG:boto:Params: {} 2023-09-22T21:49:39.490 DEBUG:boto:Token: None 2023-09-22T21:49:39.490 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj31 2023-09-22T21:49:39.491 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bAh5aKScKFAbpJbGyoLKoQeDGdU= 2023-09-22T21:49:39.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bAh5aKScKFAbpJbGyoLKoQeDGdU=', 'Content-Length': '0'} 2023-09-22T21:49:39.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a4a06b99ad78ccb-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-22T21:49:39.494 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:49:39.494 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:49:39.494 DEBUG:boto:Method: GET 2023-09-22T21:49:39.495 DEBUG:boto:Path: /ffzdqu-59/obj31 2023-09-22T21:49:39.495 DEBUG:boto:Data: 2023-09-22T21:49:39.495 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.495 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.495 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.495 DEBUG:boto:Params: {} 2023-09-22T21:49:39.495 DEBUG:boto:Token: None 2023-09-22T21:49:39.495 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj31 2023-09-22T21:49:39.495 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8cUpavv93TnjLTSxLcurxJnu5ng= 2023-09-22T21:49:39.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8cUpavv93TnjLTSxLcurxJnu5ng=', 'Content-Length': '0'} 2023-09-22T21:49:39.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e8d81769e276054-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.546 DEBUG:boto:path=/ffzdqu-59/obj31 2023-09-22T21:49:39.547 DEBUG:boto:auth_path=/ffzdqu-59/obj31 2023-09-22T21:49:39.547 DEBUG:boto:Method: GET 2023-09-22T21:49:39.547 DEBUG:boto:Path: /ffzdqu-59/obj31 2023-09-22T21:49:39.547 DEBUG:boto:Data: 2023-09-22T21:49:39.547 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.547 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.547 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.547 DEBUG:boto:Params: {} 2023-09-22T21:49:39.548 DEBUG:boto:Token: None 2023-09-22T21:49:39.548 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj31 2023-09-22T21:49:39.548 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8cUpavv93TnjLTSxLcurxJnu5ng= 2023-09-22T21:49:39.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8cUpavv93TnjLTSxLcurxJnu5ng=', 'Content-Length': '0'} 2023-09-22T21:49:39.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2a0c294460caec8-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.552 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:39.552 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:39.552 DEBUG:boto:path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:49:39.552 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:49:39.552 DEBUG:boto:Method: GET 2023-09-22T21:49:39.552 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj31 2023-09-22T21:49:39.552 DEBUG:boto:Data: 2023-09-22T21:49:39.552 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.552 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.553 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.553 DEBUG:boto:Params: {} 2023-09-22T21:49:39.553 DEBUG:boto:Token: None 2023-09-22T21:49:39.553 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/ 2023-09-22T21:49:39.553 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU= 2023-09-22T21:49:39.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU=', 'Content-Length': '0'} 2023-09-22T21:49:39.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f933d5638f316b03-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.557 DEBUG:boto:b'ffzdqu-59obj311000falseobj312023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:39.558 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:39.558 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:39.558 DEBUG:boto:path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:49:39.558 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj31 2023-09-22T21:49:39.558 DEBUG:boto:Method: GET 2023-09-22T21:49:39.559 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj31 2023-09-22T21:49:39.559 DEBUG:boto:Data: 2023-09-22T21:49:39.559 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.559 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.559 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.559 DEBUG:boto:Params: {} 2023-09-22T21:49:39.559 DEBUG:boto:Token: None 2023-09-22T21:49:39.559 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/ 2023-09-22T21:49:39.559 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU= 2023-09-22T21:49:39.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU=', 'Content-Length': '0'} 2023-09-22T21:49:39.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c942954b3fbe78a-00650e0bf3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.618 DEBUG:boto:b'ffzdqu-59obj311000falseobj312023-09-22T21:45:13.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:39.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:49:39.619 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:49:39.619 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:49:39.619 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.619 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.619 DEBUG:boto:Method: GET 2023-09-22T21:49:39.619 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.620 DEBUG:boto:Data: 2023-09-22T21:49:39.620 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.620 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.620 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.620 DEBUG:boto:Params: {} 2023-09-22T21:49:39.620 DEBUG:boto:Token: None 2023-09-22T21:49:39.620 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.620 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0P7wO7aUBJVga3F1c9ypMx6h+9E= 2023-09-22T21:49:39.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0P7wO7aUBJVga3F1c9ypMx6h+9E=', 'Content-Length': '0'} 2023-09-22T21:49:39.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095de91e3b7003d2d-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.624 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:49:39.624 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:49:39.624 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.624 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.624 DEBUG:boto:Method: GET 2023-09-22T21:49:39.625 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.625 DEBUG:boto:Data: 2023-09-22T21:49:39.625 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.625 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.625 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.625 DEBUG:boto:Params: {} 2023-09-22T21:49:39.625 DEBUG:boto:Token: None 2023-09-22T21:49:39.625 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.625 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0P7wO7aUBJVga3F1c9ypMx6h+9E= 2023-09-22T21:49:39.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0P7wO7aUBJVga3F1c9ypMx6h+9E=', 'Content-Length': '0'} 2023-09-22T21:49:39.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000062455065e03b2a-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.628 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:49:39.628 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:49:39.629 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.629 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.629 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.629 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.629 DEBUG:boto:Data: 2023-09-22T21:49:39.629 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.629 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.629 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.629 DEBUG:boto:Params: {} 2023-09-22T21:49:39.629 DEBUG:boto:Token: None 2023-09-22T21:49:39.630 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.630 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LX9fuQPTqolL3PPxuE6Up+/zJok= 2023-09-22T21:49:39.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LX9fuQPTqolL3PPxuE6Up+/zJok=', 'Content-Length': '0'} 2023-09-22T21:49:39.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0a4e28baf9a4553-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.632 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:49:39.633 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:49:39.633 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.633 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.633 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.633 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.633 DEBUG:boto:Data: 2023-09-22T21:49:39.633 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.633 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.634 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.634 DEBUG:boto:Params: {} 2023-09-22T21:49:39.634 DEBUG:boto:Token: None 2023-09-22T21:49:39.634 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.634 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LX9fuQPTqolL3PPxuE6Up+/zJok= 2023-09-22T21:49:39.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LX9fuQPTqolL3PPxuE6Up+/zJok=', 'Content-Length': '0'} 2023-09-22T21:49:39.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032ddda49bad5aedd-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:49:39.637 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:49:39.637 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:49:39.637 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.637 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.637 DEBUG:boto:Method: GET 2023-09-22T21:49:39.637 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.638 DEBUG:boto:Data: 2023-09-22T21:49:39.638 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.638 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.638 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.638 DEBUG:boto:Params: {} 2023-09-22T21:49:39.638 DEBUG:boto:Token: None 2023-09-22T21:49:39.638 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.638 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0P7wO7aUBJVga3F1c9ypMx6h+9E= 2023-09-22T21:49:39.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0P7wO7aUBJVga3F1c9ypMx6h+9E=', 'Content-Length': '0'} 2023-09-22T21:49:39.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdec4898ef76aaf0-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.642 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:49:39.642 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:49:39.642 DEBUG:boto:path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.642 DEBUG:boto:auth_path=/ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.642 DEBUG:boto:Method: GET 2023-09-22T21:49:39.642 DEBUG:boto:Path: /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.642 DEBUG:boto:Data: 2023-09-22T21:49:39.642 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.643 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.643 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.643 DEBUG:boto:Params: {} 2023-09-22T21:49:39.643 DEBUG:boto:Token: None 2023-09-22T21:49:39.643 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj33?versionId=null 2023-09-22T21:49:39.643 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0P7wO7aUBJVga3F1c9ypMx6h+9E= 2023-09-22T21:49:39.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0P7wO7aUBJVga3F1c9ypMx6h+9E=', 'Content-Length': '0'} 2023-09-22T21:49:39.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efbfef718506135d-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.646 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:39.646 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:39.646 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:49:39.646 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:49:39.647 DEBUG:boto:Method: GET 2023-09-22T21:49:39.647 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:49:39.647 DEBUG:boto:Data: 2023-09-22T21:49:39.647 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.647 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.647 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.647 DEBUG:boto:Params: {} 2023-09-22T21:49:39.647 DEBUG:boto:Token: None 2023-09-22T21:49:39.647 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/ 2023-09-22T21:49:39.648 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU= 2023-09-22T21:49:39.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU=', 'Content-Length': '0'} 2023-09-22T21:49:39.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f25012e404d3ec97-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.652 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:39.653 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:39.653 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:39.653 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:49:39.653 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:49:39.653 DEBUG:boto:Method: GET 2023-09-22T21:49:39.653 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:49:39.653 DEBUG:boto:Data: 2023-09-22T21:49:39.654 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.654 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.654 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.654 DEBUG:boto:Params: {} 2023-09-22T21:49:39.654 DEBUG:boto:Token: None 2023-09-22T21:49:39.654 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/ 2023-09-22T21:49:39.654 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU= 2023-09-22T21:49:39.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU=', 'Content-Length': '0'} 2023-09-22T21:49:39.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000994e3168d3e56c25-00650e0bf3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.664 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:39.665 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:49:39.665 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:49:39.665 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.665 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:49:39.665 DEBUG:boto:Data: 2023-09-22T21:49:39.665 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.665 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.666 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.666 DEBUG:boto:Params: {} 2023-09-22T21:49:39.666 DEBUG:boto:Token: None 2023-09-22T21:49:39.666 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj33 2023-09-22T21:49:39.666 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yLCtCzqcIHl1mHmP+tYlOxJpxM0= 2023-09-22T21:49:39.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yLCtCzqcIHl1mHmP+tYlOxJpxM0=', 'Content-Length': '0'} 2023-09-22T21:49:39.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005b31ba0cce96563-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.669 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:49:39.669 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:49:39.670 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.670 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:49:39.670 DEBUG:boto:Data: 2023-09-22T21:49:39.670 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.670 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.670 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.670 DEBUG:boto:Params: {} 2023-09-22T21:49:39.670 DEBUG:boto:Token: None 2023-09-22T21:49:39.670 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj33 2023-09-22T21:49:39.671 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yLCtCzqcIHl1mHmP+tYlOxJpxM0= 2023-09-22T21:49:39.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yLCtCzqcIHl1mHmP+tYlOxJpxM0=', 'Content-Length': '0'} 2023-09-22T21:49:39.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e521d2140a4ab2f2-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:49:39.673 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:49:39.674 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:49:39.674 DEBUG:boto:Method: GET 2023-09-22T21:49:39.674 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:49:39.674 DEBUG:boto:Data: 2023-09-22T21:49:39.674 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.674 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.674 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.674 DEBUG:boto:Params: {} 2023-09-22T21:49:39.674 DEBUG:boto:Token: None 2023-09-22T21:49:39.675 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj33 2023-09-22T21:49:39.675 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Rr+DP1dC6agLou4BkACUz4TKq7M= 2023-09-22T21:49:39.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rr+DP1dC6agLou4BkACUz4TKq7M=', 'Content-Length': '0'} 2023-09-22T21:49:39.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037dbd2c82268440d-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.722 DEBUG:boto:path=/ffzdqu-59/obj33 2023-09-22T21:49:39.722 DEBUG:boto:auth_path=/ffzdqu-59/obj33 2023-09-22T21:49:39.723 DEBUG:boto:Method: GET 2023-09-22T21:49:39.723 DEBUG:boto:Path: /ffzdqu-59/obj33 2023-09-22T21:49:39.723 DEBUG:boto:Data: 2023-09-22T21:49:39.723 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.723 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.723 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.723 DEBUG:boto:Params: {} 2023-09-22T21:49:39.723 DEBUG:boto:Token: None 2023-09-22T21:49:39.723 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj33 2023-09-22T21:49:39.724 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Rr+DP1dC6agLou4BkACUz4TKq7M= 2023-09-22T21:49:39.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rr+DP1dC6agLou4BkACUz4TKq7M=', 'Content-Length': '0'} 2023-09-22T21:49:39.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006af122a6ce70ed90-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.727 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:39.727 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:39.727 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:49:39.727 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:49:39.728 DEBUG:boto:Method: GET 2023-09-22T21:49:39.728 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:49:39.728 DEBUG:boto:Data: 2023-09-22T21:49:39.728 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.728 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.728 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.728 DEBUG:boto:Params: {} 2023-09-22T21:49:39.728 DEBUG:boto:Token: None 2023-09-22T21:49:39.728 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/ 2023-09-22T21:49:39.729 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU= 2023-09-22T21:49:39.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU=', 'Content-Length': '0'} 2023-09-22T21:49:39.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b57f18e28087edc-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.734 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:39.734 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:39.734 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:39.734 DEBUG:boto:path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:49:39.735 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj33 2023-09-22T21:49:39.735 DEBUG:boto:Method: GET 2023-09-22T21:49:39.735 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj33 2023-09-22T21:49:39.735 DEBUG:boto:Data: 2023-09-22T21:49:39.735 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.735 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.735 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.735 DEBUG:boto:Params: {} 2023-09-22T21:49:39.735 DEBUG:boto:Token: None 2023-09-22T21:49:39.736 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/ 2023-09-22T21:49:39.736 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU= 2023-09-22T21:49:39.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU=', 'Content-Length': '0'} 2023-09-22T21:49:39.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fae030453d8e101-00650e0bf3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.745 DEBUG:boto:b'ffzdqu-59obj331000falseobj332023-09-22T21:36:31.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:39.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:49:39.746 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:49:39.746 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:49:39.746 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.746 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.746 DEBUG:boto:Method: GET 2023-09-22T21:49:39.746 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.746 DEBUG:boto:Data: 2023-09-22T21:49:39.747 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.747 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.747 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.747 DEBUG:boto:Params: {} 2023-09-22T21:49:39.747 DEBUG:boto:Token: None 2023-09-22T21:49:39.750 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.750 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Ni14eYB5qExaeZmaCpx1da+DoA= 2023-09-22T21:49:39.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Ni14eYB5qExaeZmaCpx1da+DoA=', 'Content-Length': '0'} 2023-09-22T21:49:39.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062f1a8aa432b6cdf-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.753 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:49:39.753 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:49:39.753 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.754 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.754 DEBUG:boto:Method: GET 2023-09-22T21:49:39.754 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.754 DEBUG:boto:Data: 2023-09-22T21:49:39.754 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.754 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.754 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.754 DEBUG:boto:Params: {} 2023-09-22T21:49:39.754 DEBUG:boto:Token: None 2023-09-22T21:49:39.755 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.755 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Ni14eYB5qExaeZmaCpx1da+DoA= 2023-09-22T21:49:39.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Ni14eYB5qExaeZmaCpx1da+DoA=', 'Content-Length': '0'} 2023-09-22T21:49:39.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7c150966a220edc-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.758 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:49:39.758 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:49:39.758 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.758 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.758 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.758 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.759 DEBUG:boto:Data: 2023-09-22T21:49:39.759 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.759 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.759 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.759 DEBUG:boto:Params: {} 2023-09-22T21:49:39.759 DEBUG:boto:Token: None 2023-09-22T21:49:39.759 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.759 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOh37GMjuWXGxkWoqdNnotpm/U4= 2023-09-22T21:49:39.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOh37GMjuWXGxkWoqdNnotpm/U4=', 'Content-Length': '0'} 2023-09-22T21:49:39.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1ff3ce56ff37c8c-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.762 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:49:39.762 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:49:39.762 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.763 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.763 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.763 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.763 DEBUG:boto:Data: 2023-09-22T21:49:39.763 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.763 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.763 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.763 DEBUG:boto:Params: {} 2023-09-22T21:49:39.763 DEBUG:boto:Token: None 2023-09-22T21:49:39.764 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.764 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOh37GMjuWXGxkWoqdNnotpm/U4= 2023-09-22T21:49:39.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOh37GMjuWXGxkWoqdNnotpm/U4=', 'Content-Length': '0'} 2023-09-22T21:49:39.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e8d55b907bad03d-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:49:39.767 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:49:39.767 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:49:39.767 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.767 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.767 DEBUG:boto:Method: GET 2023-09-22T21:49:39.767 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.767 DEBUG:boto:Data: 2023-09-22T21:49:39.767 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.767 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.768 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.768 DEBUG:boto:Params: {} 2023-09-22T21:49:39.768 DEBUG:boto:Token: None 2023-09-22T21:49:39.768 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.768 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Ni14eYB5qExaeZmaCpx1da+DoA= 2023-09-22T21:49:39.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Ni14eYB5qExaeZmaCpx1da+DoA=', 'Content-Length': '0'} 2023-09-22T21:49:39.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009af978aef4e0da17-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.771 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:49:39.771 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:49:39.771 DEBUG:boto:path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.771 DEBUG:boto:auth_path=/ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.772 DEBUG:boto:Method: GET 2023-09-22T21:49:39.772 DEBUG:boto:Path: /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.772 DEBUG:boto:Data: 2023-09-22T21:49:39.772 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.772 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.772 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.772 DEBUG:boto:Params: {} 2023-09-22T21:49:39.772 DEBUG:boto:Token: None 2023-09-22T21:49:39.772 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj34?versionId=null 2023-09-22T21:49:39.772 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/Ni14eYB5qExaeZmaCpx1da+DoA= 2023-09-22T21:49:39.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/Ni14eYB5qExaeZmaCpx1da+DoA=', 'Content-Length': '0'} 2023-09-22T21:49:39.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000361ddd736cf4d8b1-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.818 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:39.818 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:39.819 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:49:39.819 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:49:39.819 DEBUG:boto:Method: GET 2023-09-22T21:49:39.819 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:49:39.819 DEBUG:boto:Data: 2023-09-22T21:49:39.819 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.819 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.819 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.819 DEBUG:boto:Params: {} 2023-09-22T21:49:39.819 DEBUG:boto:Token: None 2023-09-22T21:49:39.820 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/ 2023-09-22T21:49:39.820 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU= 2023-09-22T21:49:39.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU=', 'Content-Length': '0'} 2023-09-22T21:49:39.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a497a9212f657c8e-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.825 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:39.826 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:39.826 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:39.826 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:49:39.826 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:49:39.826 DEBUG:boto:Method: GET 2023-09-22T21:49:39.826 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:49:39.826 DEBUG:boto:Data: 2023-09-22T21:49:39.826 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.827 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.827 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.827 DEBUG:boto:Params: {} 2023-09-22T21:49:39.827 DEBUG:boto:Token: None 2023-09-22T21:49:39.827 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/ 2023-09-22T21:49:39.827 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU= 2023-09-22T21:49:39.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU=', 'Content-Length': '0'} 2023-09-22T21:49:39.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b32680087ca9ec7-00650e0bf3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.837 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:39.837 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:49:39.837 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:49:39.837 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.838 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:49:39.838 DEBUG:boto:Data: 2023-09-22T21:49:39.838 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.838 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.838 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.838 DEBUG:boto:Params: {} 2023-09-22T21:49:39.838 DEBUG:boto:Token: None 2023-09-22T21:49:39.838 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj34 2023-09-22T21:49:39.838 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Nz5sJ6sXEvFxXR3tXYiBNUpvPRk= 2023-09-22T21:49:39.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Nz5sJ6sXEvFxXR3tXYiBNUpvPRk=', 'Content-Length': '0'} 2023-09-22T21:49:39.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f7eaa92cbe6d909-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.841 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:49:39.842 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:49:39.842 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.842 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:49:39.842 DEBUG:boto:Data: 2023-09-22T21:49:39.842 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.842 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.842 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.842 DEBUG:boto:Params: {} 2023-09-22T21:49:39.842 DEBUG:boto:Token: None 2023-09-22T21:49:39.843 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj34 2023-09-22T21:49:39.843 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Nz5sJ6sXEvFxXR3tXYiBNUpvPRk= 2023-09-22T21:49:39.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Nz5sJ6sXEvFxXR3tXYiBNUpvPRk=', 'Content-Length': '0'} 2023-09-22T21:49:39.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a961bbddf979206-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:49:39.846 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:49:39.846 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:49:39.847 DEBUG:boto:Method: GET 2023-09-22T21:49:39.847 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:49:39.847 DEBUG:boto:Data: 2023-09-22T21:49:39.847 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.847 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.847 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.847 DEBUG:boto:Params: {} 2023-09-22T21:49:39.847 DEBUG:boto:Token: None 2023-09-22T21:49:39.847 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj34 2023-09-22T21:49:39.848 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cgrgI/KeVFdVnw7Xxc/Ov1ekn6E= 2023-09-22T21:49:39.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cgrgI/KeVFdVnw7Xxc/Ov1ekn6E=', 'Content-Length': '0'} 2023-09-22T21:49:39.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037c6c2e6553106a3-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.894 DEBUG:boto:path=/ffzdqu-59/obj34 2023-09-22T21:49:39.895 DEBUG:boto:auth_path=/ffzdqu-59/obj34 2023-09-22T21:49:39.895 DEBUG:boto:Method: GET 2023-09-22T21:49:39.895 DEBUG:boto:Path: /ffzdqu-59/obj34 2023-09-22T21:49:39.895 DEBUG:boto:Data: 2023-09-22T21:49:39.895 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.895 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.895 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.895 DEBUG:boto:Params: {} 2023-09-22T21:49:39.896 DEBUG:boto:Token: None 2023-09-22T21:49:39.896 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj34 2023-09-22T21:49:39.896 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cgrgI/KeVFdVnw7Xxc/Ov1ekn6E= 2023-09-22T21:49:39.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cgrgI/KeVFdVnw7Xxc/Ov1ekn6E=', 'Content-Length': '0'} 2023-09-22T21:49:39.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000477336809c32d287-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.899 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:39.899 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:39.900 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:49:39.900 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:49:39.900 DEBUG:boto:Method: GET 2023-09-22T21:49:39.900 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:49:39.900 DEBUG:boto:Data: 2023-09-22T21:49:39.900 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.900 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.900 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.900 DEBUG:boto:Params: {} 2023-09-22T21:49:39.901 DEBUG:boto:Token: None 2023-09-22T21:49:39.901 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/ 2023-09-22T21:49:39.901 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU= 2023-09-22T21:49:39.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU=', 'Content-Length': '0'} 2023-09-22T21:49:39.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f804fefc40053ee-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.906 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:39.907 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:39.907 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:39.907 DEBUG:boto:path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:49:39.907 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj34 2023-09-22T21:49:39.907 DEBUG:boto:Method: GET 2023-09-22T21:49:39.907 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj34 2023-09-22T21:49:39.908 DEBUG:boto:Data: 2023-09-22T21:49:39.908 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.908 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.908 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.908 DEBUG:boto:Params: {} 2023-09-22T21:49:39.908 DEBUG:boto:Token: None 2023-09-22T21:49:39.908 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/ 2023-09-22T21:49:39.908 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU= 2023-09-22T21:49:39.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU=', 'Content-Length': '0'} 2023-09-22T21:49:39.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049bd2fb2201931af-00650e0bf3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.958 DEBUG:boto:b'ffzdqu-59obj341000falseobj342023-09-22T21:36:48.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:39.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-09-22T21:49:39.960 DEBUG:boto:path=/ffzdqu-59/obj37 2023-09-22T21:49:39.960 DEBUG:boto:auth_path=/ffzdqu-59/obj37 2023-09-22T21:49:39.960 DEBUG:boto:path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.960 DEBUG:boto:auth_path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.960 DEBUG:boto:Method: GET 2023-09-22T21:49:39.960 DEBUG:boto:Path: /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.961 DEBUG:boto:Data: 2023-09-22T21:49:39.961 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.961 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.961 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.961 DEBUG:boto:Params: {} 2023-09-22T21:49:39.961 DEBUG:boto:Token: None 2023-09-22T21:49:39.961 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.961 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8QhWRx6RIOZvA0iOSbGaIj/3ca4= 2023-09-22T21:49:39.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8QhWRx6RIOZvA0iOSbGaIj/3ca4=', 'Content-Length': '0'} 2023-09-22T21:49:39.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044a55cc67647b1e3-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.965 DEBUG:boto:path=/ffzdqu-59/obj37 2023-09-22T21:49:39.965 DEBUG:boto:auth_path=/ffzdqu-59/obj37 2023-09-22T21:49:39.966 DEBUG:boto:path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.966 DEBUG:boto:auth_path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.966 DEBUG:boto:Method: GET 2023-09-22T21:49:39.966 DEBUG:boto:Path: /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.966 DEBUG:boto:Data: 2023-09-22T21:49:39.966 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.966 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.966 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.966 DEBUG:boto:Params: {} 2023-09-22T21:49:39.966 DEBUG:boto:Token: None 2023-09-22T21:49:39.967 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.967 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8QhWRx6RIOZvA0iOSbGaIj/3ca4= 2023-09-22T21:49:39.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8QhWRx6RIOZvA0iOSbGaIj/3ca4=', 'Content-Length': '0'} 2023-09-22T21:49:39.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093f704ac2ccdc108-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.970 DEBUG:boto:path=/ffzdqu-59/obj37 2023-09-22T21:49:39.970 DEBUG:boto:auth_path=/ffzdqu-59/obj37 2023-09-22T21:49:39.970 DEBUG:boto:path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.970 DEBUG:boto:auth_path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.970 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.970 DEBUG:boto:Path: /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.971 DEBUG:boto:Data: 2023-09-22T21:49:39.971 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.971 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.971 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.971 DEBUG:boto:Params: {} 2023-09-22T21:49:39.971 DEBUG:boto:Token: None 2023-09-22T21:49:39.971 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.971 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QVCCBgN3I2qiHOgzsaXmajbrQxI= 2023-09-22T21:49:39.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QVCCBgN3I2qiHOgzsaXmajbrQxI=', 'Content-Length': '0'} 2023-09-22T21:49:39.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000882fe3fe5f857a4d-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.975 DEBUG:boto:path=/ffzdqu-59/obj37 2023-09-22T21:49:39.975 DEBUG:boto:auth_path=/ffzdqu-59/obj37 2023-09-22T21:49:39.975 DEBUG:boto:path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.975 DEBUG:boto:auth_path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.975 DEBUG:boto:Method: HEAD 2023-09-22T21:49:39.975 DEBUG:boto:Path: /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.975 DEBUG:boto:Data: 2023-09-22T21:49:39.975 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.976 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.976 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.976 DEBUG:boto:Params: {} 2023-09-22T21:49:39.976 DEBUG:boto:Token: None 2023-09-22T21:49:39.976 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.976 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QVCCBgN3I2qiHOgzsaXmajbrQxI= 2023-09-22T21:49:39.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QVCCBgN3I2qiHOgzsaXmajbrQxI=', 'Content-Length': '0'} 2023-09-22T21:49:39.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083d6bed3ffbbf80e-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-09-22T21:49:39.980 DEBUG:boto:path=/ffzdqu-59/obj37 2023-09-22T21:49:39.980 DEBUG:boto:auth_path=/ffzdqu-59/obj37 2023-09-22T21:49:39.980 DEBUG:boto:path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.980 DEBUG:boto:auth_path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.980 DEBUG:boto:Method: GET 2023-09-22T21:49:39.980 DEBUG:boto:Path: /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.980 DEBUG:boto:Data: 2023-09-22T21:49:39.980 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.980 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.980 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.980 DEBUG:boto:Params: {} 2023-09-22T21:49:39.981 DEBUG:boto:Token: None 2023-09-22T21:49:39.981 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.981 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8QhWRx6RIOZvA0iOSbGaIj/3ca4= 2023-09-22T21:49:39.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8QhWRx6RIOZvA0iOSbGaIj/3ca4=', 'Content-Length': '0'} 2023-09-22T21:49:39.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000667461f0abc4125a-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.984 DEBUG:boto:path=/ffzdqu-59/obj37 2023-09-22T21:49:39.984 DEBUG:boto:auth_path=/ffzdqu-59/obj37 2023-09-22T21:49:39.984 DEBUG:boto:path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.984 DEBUG:boto:auth_path=/ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.984 DEBUG:boto:Method: GET 2023-09-22T21:49:39.984 DEBUG:boto:Path: /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.985 DEBUG:boto:Data: 2023-09-22T21:49:39.985 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.985 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:39.985 DEBUG:boto:Port: 8000 2023-09-22T21:49:39.985 DEBUG:boto:Params: {} 2023-09-22T21:49:39.985 DEBUG:boto:Token: None 2023-09-22T21:49:39.985 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/obj37?versionId=null 2023-09-22T21:49:39.985 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8QhWRx6RIOZvA0iOSbGaIj/3ca4= 2023-09-22T21:49:39.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8QhWRx6RIOZvA0iOSbGaIj/3ca4=', 'Content-Length': '0'} 2023-09-22T21:49:39.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006601754cbb73be7c-00650e0bf3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:39.989 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:39.989 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:39.990 DEBUG:boto:path=/ffzdqu-59/?prefix=obj37 2023-09-22T21:49:39.990 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj37 2023-09-22T21:49:39.990 DEBUG:boto:Method: GET 2023-09-22T21:49:39.990 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj37 2023-09-22T21:49:39.990 DEBUG:boto:Data: 2023-09-22T21:49:39.990 DEBUG:boto:Headers: {} 2023-09-22T21:49:39.990 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:39.990 DEBUG:boto:Port: 8001 2023-09-22T21:49:39.990 DEBUG:boto:Params: {} 2023-09-22T21:49:39.990 DEBUG:boto:Token: None 2023-09-22T21:49:39.991 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:39 GMT /ffzdqu-59/ 2023-09-22T21:49:39.991 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU= 2023-09-22T21:49:39.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x1rjavapDJlYx4t3/FyiE9N2frU=', 'Content-Length': '0'} 2023-09-22T21:49:39.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027cf3d4b0e45c4cf-00650e0bf3-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.038 DEBUG:boto:b'ffzdqu-59obj371000falseobj372023-09-22T21:47:16.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:40.039 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:40.039 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:40.039 DEBUG:boto:path=/ffzdqu-59/?prefix=obj37 2023-09-22T21:49:40.039 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj37 2023-09-22T21:49:40.039 DEBUG:boto:Method: GET 2023-09-22T21:49:40.039 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj37 2023-09-22T21:49:40.040 DEBUG:boto:Data: 2023-09-22T21:49:40.040 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.040 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.040 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.040 DEBUG:boto:Params: {} 2023-09-22T21:49:40.040 DEBUG:boto:Token: None 2023-09-22T21:49:40.040 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/ 2023-09-22T21:49:40.040 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk= 2023-09-22T21:49:40.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk=', 'Content-Length': '0'} 2023-09-22T21:49:40.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4ba108ad4436448-00650e0bf4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.050 DEBUG:boto:b'ffzdqu-59obj371000falseobj372023-09-22T21:47:16.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:40.050 DEBUG:boto:path=/ffzdqu-59/obj37 2023-09-22T21:49:40.051 DEBUG:boto:auth_path=/ffzdqu-59/obj37 2023-09-22T21:49:40.051 DEBUG:boto:Method: HEAD 2023-09-22T21:49:40.051 DEBUG:boto:Path: /ffzdqu-59/obj37 2023-09-22T21:49:40.051 DEBUG:boto:Data: 2023-09-22T21:49:40.051 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.051 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.051 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.051 DEBUG:boto:Params: {} 2023-09-22T21:49:40.051 DEBUG:boto:Token: None 2023-09-22T21:49:40.052 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj37 2023-09-22T21:49:40.052 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:X75MVmMUpoeAlS1xLrHC3IOkF7I= 2023-09-22T21:49:40.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:X75MVmMUpoeAlS1xLrHC3IOkF7I=', 'Content-Length': '0'} 2023-09-22T21:49:40.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095e7f444870e0e9b-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.055 DEBUG:boto:path=/ffzdqu-59/obj37 2023-09-22T21:49:40.055 DEBUG:boto:auth_path=/ffzdqu-59/obj37 2023-09-22T21:49:40.055 DEBUG:boto:Method: HEAD 2023-09-22T21:49:40.055 DEBUG:boto:Path: /ffzdqu-59/obj37 2023-09-22T21:49:40.055 DEBUG:boto:Data: 2023-09-22T21:49:40.055 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.055 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.055 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.055 DEBUG:boto:Params: {} 2023-09-22T21:49:40.056 DEBUG:boto:Token: None 2023-09-22T21:49:40.056 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj37 2023-09-22T21:49:40.056 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:X75MVmMUpoeAlS1xLrHC3IOkF7I= 2023-09-22T21:49:40.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:X75MVmMUpoeAlS1xLrHC3IOkF7I=', 'Content-Length': '0'} 2023-09-22T21:49:40.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f05588283d699c37-00650e0bf4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-09-22T21:49:40.059 DEBUG:boto:path=/ffzdqu-59/obj37 2023-09-22T21:49:40.059 DEBUG:boto:auth_path=/ffzdqu-59/obj37 2023-09-22T21:49:40.059 DEBUG:boto:Method: GET 2023-09-22T21:49:40.060 DEBUG:boto:Path: /ffzdqu-59/obj37 2023-09-22T21:49:40.060 DEBUG:boto:Data: 2023-09-22T21:49:40.060 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.060 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.060 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.060 DEBUG:boto:Params: {} 2023-09-22T21:49:40.060 DEBUG:boto:Token: None 2023-09-22T21:49:40.060 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj37 2023-09-22T21:49:40.060 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gTFTlRDBU1+uU3dezyS64KH/4ag= 2023-09-22T21:49:40.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gTFTlRDBU1+uU3dezyS64KH/4ag=', 'Content-Length': '0'} 2023-09-22T21:49:40.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4788a20f98a2a8e-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.063 DEBUG:boto:path=/ffzdqu-59/obj37 2023-09-22T21:49:40.063 DEBUG:boto:auth_path=/ffzdqu-59/obj37 2023-09-22T21:49:40.064 DEBUG:boto:Method: GET 2023-09-22T21:49:40.064 DEBUG:boto:Path: /ffzdqu-59/obj37 2023-09-22T21:49:40.064 DEBUG:boto:Data: 2023-09-22T21:49:40.064 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.064 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.064 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.064 DEBUG:boto:Params: {} 2023-09-22T21:49:40.064 DEBUG:boto:Token: None 2023-09-22T21:49:40.064 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj37 2023-09-22T21:49:40.064 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gTFTlRDBU1+uU3dezyS64KH/4ag= 2023-09-22T21:49:40.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gTFTlRDBU1+uU3dezyS64KH/4ag=', 'Content-Length': '0'} 2023-09-22T21:49:40.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a195204784a29e19-00650e0bf4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.110 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:40.111 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:40.111 DEBUG:boto:path=/ffzdqu-59/?prefix=obj37 2023-09-22T21:49:40.111 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj37 2023-09-22T21:49:40.111 DEBUG:boto:Method: GET 2023-09-22T21:49:40.111 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj37 2023-09-22T21:49:40.111 DEBUG:boto:Data: 2023-09-22T21:49:40.111 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.111 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.112 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.112 DEBUG:boto:Params: {} 2023-09-22T21:49:40.112 DEBUG:boto:Token: None 2023-09-22T21:49:40.112 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/ 2023-09-22T21:49:40.112 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk= 2023-09-22T21:49:40.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk=', 'Content-Length': '0'} 2023-09-22T21:49:40.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e032b97ac397050-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.117 DEBUG:boto:b'ffzdqu-59obj371000falseobj372023-09-22T21:47:16.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:40.118 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:40.118 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:40.118 DEBUG:boto:path=/ffzdqu-59/?prefix=obj37 2023-09-22T21:49:40.118 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj37 2023-09-22T21:49:40.118 DEBUG:boto:Method: GET 2023-09-22T21:49:40.118 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj37 2023-09-22T21:49:40.119 DEBUG:boto:Data: 2023-09-22T21:49:40.119 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.119 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.119 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.119 DEBUG:boto:Params: {} 2023-09-22T21:49:40.119 DEBUG:boto:Token: None 2023-09-22T21:49:40.119 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/ 2023-09-22T21:49:40.119 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk= 2023-09-22T21:49:40.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk=', 'Content-Length': '0'} 2023-09-22T21:49:40.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007da77bbb06bc3e2f-00650e0bf4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.130 DEBUG:boto:b'ffzdqu-59obj371000falseobj372023-09-22T21:47:16.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:40.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T21:49:40.130 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:49:40.130 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:49:40.131 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.131 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.131 DEBUG:boto:Method: GET 2023-09-22T21:49:40.131 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.131 DEBUG:boto:Data: 2023-09-22T21:49:40.131 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.131 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.131 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.131 DEBUG:boto:Params: {} 2023-09-22T21:49:40.131 DEBUG:boto:Token: None 2023-09-22T21:49:40.132 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.132 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pM1L2hnlmgXKQhpqqU04JqGcFOs= 2023-09-22T21:49:40.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pM1L2hnlmgXKQhpqqU04JqGcFOs=', 'Content-Length': '0'} 2023-09-22T21:49:40.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000531b677e8be301e8-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.178 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:49:40.178 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:49:40.179 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.179 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.179 DEBUG:boto:Method: GET 2023-09-22T21:49:40.179 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.179 DEBUG:boto:Data: 2023-09-22T21:49:40.179 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.179 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.179 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.179 DEBUG:boto:Params: {} 2023-09-22T21:49:40.179 DEBUG:boto:Token: None 2023-09-22T21:49:40.180 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.180 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pM1L2hnlmgXKQhpqqU04JqGcFOs= 2023-09-22T21:49:40.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pM1L2hnlmgXKQhpqqU04JqGcFOs=', 'Content-Length': '0'} 2023-09-22T21:49:40.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3aa9d8c1df3ea21-00650e0bf4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.183 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:49:40.183 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:49:40.183 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.183 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.183 DEBUG:boto:Method: HEAD 2023-09-22T21:49:40.184 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.184 DEBUG:boto:Data: 2023-09-22T21:49:40.184 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.184 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.184 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.184 DEBUG:boto:Params: {} 2023-09-22T21:49:40.184 DEBUG:boto:Token: None 2023-09-22T21:49:40.184 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.185 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rwrV8pIjoipssQ+yxmdMfJqIxmE= 2023-09-22T21:49:40.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rwrV8pIjoipssQ+yxmdMfJqIxmE=', 'Content-Length': '0'} 2023-09-22T21:49:40.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059e08da4636581d6-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.188 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:49:40.188 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:49:40.188 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.188 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.188 DEBUG:boto:Method: HEAD 2023-09-22T21:49:40.188 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.189 DEBUG:boto:Data: 2023-09-22T21:49:40.189 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.189 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.189 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.189 DEBUG:boto:Params: {} 2023-09-22T21:49:40.189 DEBUG:boto:Token: None 2023-09-22T21:49:40.189 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.189 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rwrV8pIjoipssQ+yxmdMfJqIxmE= 2023-09-22T21:49:40.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rwrV8pIjoipssQ+yxmdMfJqIxmE=', 'Content-Length': '0'} 2023-09-22T21:49:40.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c61ea4fff34e0db5-00650e0bf4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T21:49:40.192 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:49:40.192 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:49:40.192 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.192 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.192 DEBUG:boto:Method: GET 2023-09-22T21:49:40.193 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.193 DEBUG:boto:Data: 2023-09-22T21:49:40.193 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.193 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.193 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.193 DEBUG:boto:Params: {} 2023-09-22T21:49:40.193 DEBUG:boto:Token: None 2023-09-22T21:49:40.193 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.193 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pM1L2hnlmgXKQhpqqU04JqGcFOs= 2023-09-22T21:49:40.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pM1L2hnlmgXKQhpqqU04JqGcFOs=', 'Content-Length': '0'} 2023-09-22T21:49:40.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000117b56e62b1f2410-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.196 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:49:40.197 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:49:40.197 DEBUG:boto:path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.197 DEBUG:boto:auth_path=/ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.197 DEBUG:boto:Method: GET 2023-09-22T21:49:40.197 DEBUG:boto:Path: /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.197 DEBUG:boto:Data: 2023-09-22T21:49:40.197 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.197 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.197 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.197 DEBUG:boto:Params: {} 2023-09-22T21:49:40.198 DEBUG:boto:Token: None 2023-09-22T21:49:40.198 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj38?versionId=null 2023-09-22T21:49:40.198 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pM1L2hnlmgXKQhpqqU04JqGcFOs= 2023-09-22T21:49:40.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pM1L2hnlmgXKQhpqqU04JqGcFOs=', 'Content-Length': '0'} 2023-09-22T21:49:40.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e04087c0582235a8-00650e0bf4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.242 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:40.242 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:40.243 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:49:40.243 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:49:40.243 DEBUG:boto:Method: GET 2023-09-22T21:49:40.243 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:49:40.243 DEBUG:boto:Data: 2023-09-22T21:49:40.243 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.243 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.243 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.243 DEBUG:boto:Params: {} 2023-09-22T21:49:40.243 DEBUG:boto:Token: None 2023-09-22T21:49:40.244 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/ 2023-09-22T21:49:40.244 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk= 2023-09-22T21:49:40.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk=', 'Content-Length': '0'} 2023-09-22T21:49:40.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000defaae04bb8651a7-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.249 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:40.249 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:40.250 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:40.250 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:49:40.250 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:49:40.250 DEBUG:boto:Method: GET 2023-09-22T21:49:40.250 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:49:40.250 DEBUG:boto:Data: 2023-09-22T21:49:40.250 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.250 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.250 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.250 DEBUG:boto:Params: {} 2023-09-22T21:49:40.251 DEBUG:boto:Token: None 2023-09-22T21:49:40.251 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/ 2023-09-22T21:49:40.251 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk= 2023-09-22T21:49:40.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk=', 'Content-Length': '0'} 2023-09-22T21:49:40.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe76f2996c8a8405-00650e0bf4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.260 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:40.261 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:49:40.261 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:49:40.261 DEBUG:boto:Method: HEAD 2023-09-22T21:49:40.261 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:49:40.261 DEBUG:boto:Data: 2023-09-22T21:49:40.261 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.261 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.262 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.262 DEBUG:boto:Params: {} 2023-09-22T21:49:40.262 DEBUG:boto:Token: None 2023-09-22T21:49:40.262 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj38 2023-09-22T21:49:40.262 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AkV5vUR9PXXq8J4DExrgQiCfR34= 2023-09-22T21:49:40.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AkV5vUR9PXXq8J4DExrgQiCfR34=', 'Content-Length': '0'} 2023-09-22T21:49:40.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6c4db3c511ac1c6-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.265 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:49:40.265 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:49:40.265 DEBUG:boto:Method: HEAD 2023-09-22T21:49:40.265 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:49:40.265 DEBUG:boto:Data: 2023-09-22T21:49:40.265 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.265 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.265 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.266 DEBUG:boto:Params: {} 2023-09-22T21:49:40.266 DEBUG:boto:Token: None 2023-09-22T21:49:40.266 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj38 2023-09-22T21:49:40.266 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AkV5vUR9PXXq8J4DExrgQiCfR34= 2023-09-22T21:49:40.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AkV5vUR9PXXq8J4DExrgQiCfR34=', 'Content-Length': '0'} 2023-09-22T21:49:40.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3e672730d3147ac-00650e0bf4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T21:49:40.269 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:49:40.269 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:49:40.269 DEBUG:boto:Method: GET 2023-09-22T21:49:40.270 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:49:40.270 DEBUG:boto:Data: 2023-09-22T21:49:40.270 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.270 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.270 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.270 DEBUG:boto:Params: {} 2023-09-22T21:49:40.270 DEBUG:boto:Token: None 2023-09-22T21:49:40.270 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj38 2023-09-22T21:49:40.270 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NP/wOw8INnUND1O9/5YWOOwMX94= 2023-09-22T21:49:40.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NP/wOw8INnUND1O9/5YWOOwMX94=', 'Content-Length': '0'} 2023-09-22T21:49:40.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac0052ad62696c45-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.318 DEBUG:boto:path=/ffzdqu-59/obj38 2023-09-22T21:49:40.318 DEBUG:boto:auth_path=/ffzdqu-59/obj38 2023-09-22T21:49:40.319 DEBUG:boto:Method: GET 2023-09-22T21:49:40.319 DEBUG:boto:Path: /ffzdqu-59/obj38 2023-09-22T21:49:40.319 DEBUG:boto:Data: 2023-09-22T21:49:40.319 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.319 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.319 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.319 DEBUG:boto:Params: {} 2023-09-22T21:49:40.319 DEBUG:boto:Token: None 2023-09-22T21:49:40.319 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj38 2023-09-22T21:49:40.320 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NP/wOw8INnUND1O9/5YWOOwMX94= 2023-09-22T21:49:40.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NP/wOw8INnUND1O9/5YWOOwMX94=', 'Content-Length': '0'} 2023-09-22T21:49:40.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000471e7b90280bb72b-00650e0bf4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.323 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:40.323 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:40.323 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:49:40.323 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:49:40.323 DEBUG:boto:Method: GET 2023-09-22T21:49:40.323 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:49:40.323 DEBUG:boto:Data: 2023-09-22T21:49:40.323 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.324 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.324 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.324 DEBUG:boto:Params: {} 2023-09-22T21:49:40.324 DEBUG:boto:Token: None 2023-09-22T21:49:40.324 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/ 2023-09-22T21:49:40.324 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk= 2023-09-22T21:49:40.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk=', 'Content-Length': '0'} 2023-09-22T21:49:40.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000343a161052158b23-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.329 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:40.329 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:40.330 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:40.330 DEBUG:boto:path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:49:40.330 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj38 2023-09-22T21:49:40.330 DEBUG:boto:Method: GET 2023-09-22T21:49:40.330 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj38 2023-09-22T21:49:40.330 DEBUG:boto:Data: 2023-09-22T21:49:40.330 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.330 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.330 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.331 DEBUG:boto:Params: {} 2023-09-22T21:49:40.331 DEBUG:boto:Token: None 2023-09-22T21:49:40.331 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/ 2023-09-22T21:49:40.331 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk= 2023-09-22T21:49:40.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk=', 'Content-Length': '0'} 2023-09-22T21:49:40.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3994a16db3bf50f-00650e0bf4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.386 DEBUG:boto:b'ffzdqu-59obj381000falseobj382023-09-22T21:38:31.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:40.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:49:40.387 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:49:40.387 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:49:40.387 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.387 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.387 DEBUG:boto:Method: GET 2023-09-22T21:49:40.387 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.387 DEBUG:boto:Data: 2023-09-22T21:49:40.387 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.388 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.388 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.388 DEBUG:boto:Params: {} 2023-09-22T21:49:40.388 DEBUG:boto:Token: None 2023-09-22T21:49:40.388 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.388 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+wp2JlBKmspNhKryffCf62pcKWI= 2023-09-22T21:49:40.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+wp2JlBKmspNhKryffCf62pcKWI=', 'Content-Length': '0'} 2023-09-22T21:49:40.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a50b3ab5f44cd117-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.392 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:49:40.392 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:49:40.392 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.392 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.392 DEBUG:boto:Method: GET 2023-09-22T21:49:40.392 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.392 DEBUG:boto:Data: 2023-09-22T21:49:40.392 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.392 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.392 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.393 DEBUG:boto:Params: {} 2023-09-22T21:49:40.393 DEBUG:boto:Token: None 2023-09-22T21:49:40.393 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.393 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+wp2JlBKmspNhKryffCf62pcKWI= 2023-09-22T21:49:40.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+wp2JlBKmspNhKryffCf62pcKWI=', 'Content-Length': '0'} 2023-09-22T21:49:40.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b74d046be589546d-00650e0bf4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.396 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:49:40.396 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:49:40.396 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.397 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.397 DEBUG:boto:Method: HEAD 2023-09-22T21:49:40.397 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.397 DEBUG:boto:Data: 2023-09-22T21:49:40.397 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.397 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.397 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.397 DEBUG:boto:Params: {} 2023-09-22T21:49:40.397 DEBUG:boto:Token: None 2023-09-22T21:49:40.398 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.398 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DbjoF2Lj17jkbFWuiIdN2PbluVU= 2023-09-22T21:49:40.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DbjoF2Lj17jkbFWuiIdN2PbluVU=', 'Content-Length': '0'} 2023-09-22T21:49:40.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b16bb8d568df99d-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.400 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:49:40.401 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:49:40.401 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.401 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.401 DEBUG:boto:Method: HEAD 2023-09-22T21:49:40.401 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.401 DEBUG:boto:Data: 2023-09-22T21:49:40.401 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.401 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.402 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.402 DEBUG:boto:Params: {} 2023-09-22T21:49:40.402 DEBUG:boto:Token: None 2023-09-22T21:49:40.402 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.402 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DbjoF2Lj17jkbFWuiIdN2PbluVU= 2023-09-22T21:49:40.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DbjoF2Lj17jkbFWuiIdN2PbluVU=', 'Content-Length': '0'} 2023-09-22T21:49:40.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b5f6de7bed5e7fa-00650e0bf4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:49:40.405 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:49:40.406 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:49:40.406 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.406 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.406 DEBUG:boto:Method: GET 2023-09-22T21:49:40.406 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.406 DEBUG:boto:Data: 2023-09-22T21:49:40.406 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.406 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.406 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.407 DEBUG:boto:Params: {} 2023-09-22T21:49:40.407 DEBUG:boto:Token: None 2023-09-22T21:49:40.407 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.407 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+wp2JlBKmspNhKryffCf62pcKWI= 2023-09-22T21:49:40.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+wp2JlBKmspNhKryffCf62pcKWI=', 'Content-Length': '0'} 2023-09-22T21:49:40.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000883077622b1bff90-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.410 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:49:40.410 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:49:40.410 DEBUG:boto:path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.410 DEBUG:boto:auth_path=/ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.411 DEBUG:boto:Method: GET 2023-09-22T21:49:40.411 DEBUG:boto:Path: /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.411 DEBUG:boto:Data: 2023-09-22T21:49:40.411 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.411 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.411 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.411 DEBUG:boto:Params: {} 2023-09-22T21:49:40.411 DEBUG:boto:Token: None 2023-09-22T21:49:40.411 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj44?versionId=null 2023-09-22T21:49:40.412 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+wp2JlBKmspNhKryffCf62pcKWI= 2023-09-22T21:49:40.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+wp2JlBKmspNhKryffCf62pcKWI=', 'Content-Length': '0'} 2023-09-22T21:49:40.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000de977ba7dc3ebd-00650e0bf4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.415 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:40.415 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:40.415 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:49:40.415 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:49:40.415 DEBUG:boto:Method: GET 2023-09-22T21:49:40.415 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:49:40.415 DEBUG:boto:Data: 2023-09-22T21:49:40.415 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.415 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.415 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.416 DEBUG:boto:Params: {} 2023-09-22T21:49:40.416 DEBUG:boto:Token: None 2023-09-22T21:49:40.416 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/ 2023-09-22T21:49:40.416 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk= 2023-09-22T21:49:40.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk=', 'Content-Length': '0'} 2023-09-22T21:49:40.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015a0fb4fa122ab50-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.421 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:40.421 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:40.421 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:40.421 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:49:40.421 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:49:40.421 DEBUG:boto:Method: GET 2023-09-22T21:49:40.421 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:49:40.422 DEBUG:boto:Data: 2023-09-22T21:49:40.422 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.422 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.422 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.422 DEBUG:boto:Params: {} 2023-09-22T21:49:40.422 DEBUG:boto:Token: None 2023-09-22T21:49:40.422 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/ 2023-09-22T21:49:40.422 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk= 2023-09-22T21:49:40.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk=', 'Content-Length': '0'} 2023-09-22T21:49:40.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adcfdb969b763c7a-00650e0bf4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.474 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:40.475 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:49:40.475 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:49:40.475 DEBUG:boto:Method: HEAD 2023-09-22T21:49:40.475 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:49:40.475 DEBUG:boto:Data: 2023-09-22T21:49:40.475 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.475 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.475 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.475 DEBUG:boto:Params: {} 2023-09-22T21:49:40.476 DEBUG:boto:Token: None 2023-09-22T21:49:40.476 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj44 2023-09-22T21:49:40.476 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EbiP5C5zyRlTkFIx0Voj1s+tiNQ= 2023-09-22T21:49:40.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EbiP5C5zyRlTkFIx0Voj1s+tiNQ=', 'Content-Length': '0'} 2023-09-22T21:49:40.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057941f6d9ce77e94-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.479 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:49:40.479 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:49:40.479 DEBUG:boto:Method: HEAD 2023-09-22T21:49:40.479 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:49:40.479 DEBUG:boto:Data: 2023-09-22T21:49:40.479 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.480 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.480 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.480 DEBUG:boto:Params: {} 2023-09-22T21:49:40.480 DEBUG:boto:Token: None 2023-09-22T21:49:40.480 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj44 2023-09-22T21:49:40.480 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EbiP5C5zyRlTkFIx0Voj1s+tiNQ= 2023-09-22T21:49:40.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EbiP5C5zyRlTkFIx0Voj1s+tiNQ=', 'Content-Length': '0'} 2023-09-22T21:49:40.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5273a15d01387e6-00650e0bf4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:49:40.483 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:49:40.483 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:49:40.484 DEBUG:boto:Method: GET 2023-09-22T21:49:40.484 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:49:40.484 DEBUG:boto:Data: 2023-09-22T21:49:40.484 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.484 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.484 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.484 DEBUG:boto:Params: {} 2023-09-22T21:49:40.484 DEBUG:boto:Token: None 2023-09-22T21:49:40.484 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj44 2023-09-22T21:49:40.485 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tKyZtPpzuDmOeOXFYOv9NvrbZic= 2023-09-22T21:49:40.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tKyZtPpzuDmOeOXFYOv9NvrbZic=', 'Content-Length': '0'} 2023-09-22T21:49:40.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e63190cf6771c8bc-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.534 DEBUG:boto:path=/ffzdqu-59/obj44 2023-09-22T21:49:40.534 DEBUG:boto:auth_path=/ffzdqu-59/obj44 2023-09-22T21:49:40.535 DEBUG:boto:Method: GET 2023-09-22T21:49:40.535 DEBUG:boto:Path: /ffzdqu-59/obj44 2023-09-22T21:49:40.535 DEBUG:boto:Data: 2023-09-22T21:49:40.535 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.535 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.535 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.535 DEBUG:boto:Params: {} 2023-09-22T21:49:40.535 DEBUG:boto:Token: None 2023-09-22T21:49:40.535 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj44 2023-09-22T21:49:40.536 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tKyZtPpzuDmOeOXFYOv9NvrbZic= 2023-09-22T21:49:40.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tKyZtPpzuDmOeOXFYOv9NvrbZic=', 'Content-Length': '0'} 2023-09-22T21:49:40.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce1e8581b3273fd8-00650e0bf4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.539 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:40.539 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:40.539 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:49:40.539 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:49:40.539 DEBUG:boto:Method: GET 2023-09-22T21:49:40.539 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:49:40.539 DEBUG:boto:Data: 2023-09-22T21:49:40.539 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.539 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.539 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.540 DEBUG:boto:Params: {} 2023-09-22T21:49:40.540 DEBUG:boto:Token: None 2023-09-22T21:49:40.540 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/ 2023-09-22T21:49:40.540 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk= 2023-09-22T21:49:40.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk=', 'Content-Length': '0'} 2023-09-22T21:49:40.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae684ffd4b1897dd-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.546 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:40.546 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:40.546 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:40.546 DEBUG:boto:path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:49:40.546 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj44 2023-09-22T21:49:40.546 DEBUG:boto:Method: GET 2023-09-22T21:49:40.546 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj44 2023-09-22T21:49:40.547 DEBUG:boto:Data: 2023-09-22T21:49:40.547 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.547 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.547 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.547 DEBUG:boto:Params: {} 2023-09-22T21:49:40.547 DEBUG:boto:Token: None 2023-09-22T21:49:40.547 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/ 2023-09-22T21:49:40.547 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk= 2023-09-22T21:49:40.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk=', 'Content-Length': '0'} 2023-09-22T21:49:40.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9a6eb58f4b25487-00650e0bf4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.557 DEBUG:boto:b'ffzdqu-59obj441000falseobj442023-09-22T21:36:31.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:40.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-22T21:49:40.558 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:49:40.558 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:49:40.558 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.558 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.558 DEBUG:boto:Method: GET 2023-09-22T21:49:40.558 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.558 DEBUG:boto:Data: 2023-09-22T21:49:40.558 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.558 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.559 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.559 DEBUG:boto:Params: {} 2023-09-22T21:49:40.559 DEBUG:boto:Token: None 2023-09-22T21:49:40.559 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.559 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/K9qP3VSrGrkgkZ/uf5+nEoIbXA= 2023-09-22T21:49:40.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/K9qP3VSrGrkgkZ/uf5+nEoIbXA=', 'Content-Length': '0'} 2023-09-22T21:49:40.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a6d4d317218f127-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.562 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:49:40.562 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:49:40.563 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.563 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.563 DEBUG:boto:Method: GET 2023-09-22T21:49:40.563 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.563 DEBUG:boto:Data: 2023-09-22T21:49:40.563 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.563 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.563 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.563 DEBUG:boto:Params: {} 2023-09-22T21:49:40.564 DEBUG:boto:Token: None 2023-09-22T21:49:40.564 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.564 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/K9qP3VSrGrkgkZ/uf5+nEoIbXA= 2023-09-22T21:49:40.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/K9qP3VSrGrkgkZ/uf5+nEoIbXA=', 'Content-Length': '0'} 2023-09-22T21:49:40.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5d1311b7a9d0251-00650e0bf4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.610 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:49:40.610 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:49:40.611 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.611 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.611 DEBUG:boto:Method: HEAD 2023-09-22T21:49:40.611 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.611 DEBUG:boto:Data: 2023-09-22T21:49:40.611 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.611 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.611 DEBUG:boto:Params: {} 2023-09-22T21:49:40.611 DEBUG:boto:Token: None 2023-09-22T21:49:40.612 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Mj/mfF8uFQOni5rpdTTdLJNcO6E= 2023-09-22T21:49:40.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Mj/mfF8uFQOni5rpdTTdLJNcO6E=', 'Content-Length': '0'} 2023-09-22T21:49:40.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b0f78529d0b4c57-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.615 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:49:40.615 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:49:40.615 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.615 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.615 DEBUG:boto:Method: HEAD 2023-09-22T21:49:40.615 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.615 DEBUG:boto:Data: 2023-09-22T21:49:40.616 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.616 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.616 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.616 DEBUG:boto:Params: {} 2023-09-22T21:49:40.616 DEBUG:boto:Token: None 2023-09-22T21:49:40.616 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.616 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Mj/mfF8uFQOni5rpdTTdLJNcO6E= 2023-09-22T21:49:40.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Mj/mfF8uFQOni5rpdTTdLJNcO6E=', 'Content-Length': '0'} 2023-09-22T21:49:40.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaefb8f814f51c36-00650e0bf4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-22T21:49:40.619 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:49:40.619 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:49:40.619 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.619 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.620 DEBUG:boto:Method: GET 2023-09-22T21:49:40.620 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.620 DEBUG:boto:Data: 2023-09-22T21:49:40.620 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.620 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.620 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.620 DEBUG:boto:Params: {} 2023-09-22T21:49:40.620 DEBUG:boto:Token: None 2023-09-22T21:49:40.620 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.620 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/K9qP3VSrGrkgkZ/uf5+nEoIbXA= 2023-09-22T21:49:40.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/K9qP3VSrGrkgkZ/uf5+nEoIbXA=', 'Content-Length': '0'} 2023-09-22T21:49:40.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c79cf3e304352d6-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.666 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:49:40.666 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:49:40.667 DEBUG:boto:path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.667 DEBUG:boto:auth_path=/ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.667 DEBUG:boto:Method: GET 2023-09-22T21:49:40.667 DEBUG:boto:Path: /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.667 DEBUG:boto:Data: 2023-09-22T21:49:40.667 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.667 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.667 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.667 DEBUG:boto:Params: {} 2023-09-22T21:49:40.668 DEBUG:boto:Token: None 2023-09-22T21:49:40.668 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj46?versionId=null 2023-09-22T21:49:40.668 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/K9qP3VSrGrkgkZ/uf5+nEoIbXA= 2023-09-22T21:49:40.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/K9qP3VSrGrkgkZ/uf5+nEoIbXA=', 'Content-Length': '0'} 2023-09-22T21:49:40.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056196256e0aa510f-00650e0bf4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.671 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:40.671 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:40.671 DEBUG:boto:path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:49:40.671 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:49:40.671 DEBUG:boto:Method: GET 2023-09-22T21:49:40.671 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj46 2023-09-22T21:49:40.672 DEBUG:boto:Data: 2023-09-22T21:49:40.672 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.672 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.672 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.672 DEBUG:boto:Params: {} 2023-09-22T21:49:40.672 DEBUG:boto:Token: None 2023-09-22T21:49:40.672 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/ 2023-09-22T21:49:40.672 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk= 2023-09-22T21:49:40.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk=', 'Content-Length': '0'} 2023-09-22T21:49:40.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093f3bd68fe63dea7-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.677 DEBUG:boto:b'ffzdqu-59obj461000falseobj462023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:40.678 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:40.678 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:40.678 DEBUG:boto:path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:49:40.678 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:49:40.678 DEBUG:boto:Method: GET 2023-09-22T21:49:40.678 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj46 2023-09-22T21:49:40.678 DEBUG:boto:Data: 2023-09-22T21:49:40.679 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.679 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.679 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.679 DEBUG:boto:Params: {} 2023-09-22T21:49:40.679 DEBUG:boto:Token: None 2023-09-22T21:49:40.679 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/ 2023-09-22T21:49:40.679 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk= 2023-09-22T21:49:40.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk=', 'Content-Length': '0'} 2023-09-22T21:49:40.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061982d3a9523aec2-00650e0bf4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.847 DEBUG:boto:b'ffzdqu-59obj461000falseobj462023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:40.848 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:49:40.848 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:49:40.848 DEBUG:boto:Method: HEAD 2023-09-22T21:49:40.848 DEBUG:boto:Path: /ffzdqu-59/obj46 2023-09-22T21:49:40.848 DEBUG:boto:Data: 2023-09-22T21:49:40.848 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.849 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.849 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.849 DEBUG:boto:Params: {} 2023-09-22T21:49:40.849 DEBUG:boto:Token: None 2023-09-22T21:49:40.849 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj46 2023-09-22T21:49:40.849 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+SM3wjWbQUIvfC65Y41P60o/w70= 2023-09-22T21:49:40.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+SM3wjWbQUIvfC65Y41P60o/w70=', 'Content-Length': '0'} 2023-09-22T21:49:40.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017008f996fa5681d-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.852 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:49:40.853 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:49:40.853 DEBUG:boto:Method: HEAD 2023-09-22T21:49:40.853 DEBUG:boto:Path: /ffzdqu-59/obj46 2023-09-22T21:49:40.853 DEBUG:boto:Data: 2023-09-22T21:49:40.853 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.853 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.853 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.853 DEBUG:boto:Params: {} 2023-09-22T21:49:40.853 DEBUG:boto:Token: None 2023-09-22T21:49:40.854 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj46 2023-09-22T21:49:40.854 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+SM3wjWbQUIvfC65Y41P60o/w70= 2023-09-22T21:49:40.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+SM3wjWbQUIvfC65Y41P60o/w70=', 'Content-Length': '0'} 2023-09-22T21:49:40.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007eb23b1bd3c6344b-00650e0bf4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-22T21:49:40.858 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:49:40.858 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:49:40.859 DEBUG:boto:Method: GET 2023-09-22T21:49:40.859 DEBUG:boto:Path: /ffzdqu-59/obj46 2023-09-22T21:49:40.859 DEBUG:boto:Data: 2023-09-22T21:49:40.859 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.859 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.859 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.859 DEBUG:boto:Params: {} 2023-09-22T21:49:40.859 DEBUG:boto:Token: None 2023-09-22T21:49:40.860 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj46 2023-09-22T21:49:40.860 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GibLDeti32krzJUL4RhAB9D5ktA= 2023-09-22T21:49:40.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GibLDeti32krzJUL4RhAB9D5ktA=', 'Content-Length': '0'} 2023-09-22T21:49:40.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085fae1f89c29d04d-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.864 DEBUG:boto:path=/ffzdqu-59/obj46 2023-09-22T21:49:40.864 DEBUG:boto:auth_path=/ffzdqu-59/obj46 2023-09-22T21:49:40.864 DEBUG:boto:Method: GET 2023-09-22T21:49:40.864 DEBUG:boto:Path: /ffzdqu-59/obj46 2023-09-22T21:49:40.864 DEBUG:boto:Data: 2023-09-22T21:49:40.864 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.864 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.864 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.865 DEBUG:boto:Params: {} 2023-09-22T21:49:40.865 DEBUG:boto:Token: None 2023-09-22T21:49:40.865 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj46 2023-09-22T21:49:40.865 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GibLDeti32krzJUL4RhAB9D5ktA= 2023-09-22T21:49:40.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GibLDeti32krzJUL4RhAB9D5ktA=', 'Content-Length': '0'} 2023-09-22T21:49:40.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061384a2fe83d6c31-00650e0bf4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.868 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:40.869 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:40.869 DEBUG:boto:path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:49:40.869 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:49:40.869 DEBUG:boto:Method: GET 2023-09-22T21:49:40.869 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj46 2023-09-22T21:49:40.869 DEBUG:boto:Data: 2023-09-22T21:49:40.869 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.869 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.869 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.870 DEBUG:boto:Params: {} 2023-09-22T21:49:40.870 DEBUG:boto:Token: None 2023-09-22T21:49:40.870 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/ 2023-09-22T21:49:40.870 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk= 2023-09-22T21:49:40.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk=', 'Content-Length': '0'} 2023-09-22T21:49:40.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b89fc0c11dea59b-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.918 DEBUG:boto:b'ffzdqu-59obj461000falseobj462023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:40.919 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:40.919 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:40.919 DEBUG:boto:path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:49:40.919 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj46 2023-09-22T21:49:40.919 DEBUG:boto:Method: GET 2023-09-22T21:49:40.919 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj46 2023-09-22T21:49:40.919 DEBUG:boto:Data: 2023-09-22T21:49:40.919 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.920 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.920 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.920 DEBUG:boto:Params: {} 2023-09-22T21:49:40.920 DEBUG:boto:Token: None 2023-09-22T21:49:40.920 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/ 2023-09-22T21:49:40.920 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk= 2023-09-22T21:49:40.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sl0+Qp/c/95QKa9tEhJm898A0Dk=', 'Content-Length': '0'} 2023-09-22T21:49:40.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9babd78bef27bad-00650e0bf4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.930 DEBUG:boto:b'ffzdqu-59obj461000falseobj462023-09-22T21:40:21.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:40.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:49:40.930 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:49:40.930 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:49:40.930 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.931 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.931 DEBUG:boto:Method: GET 2023-09-22T21:49:40.931 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.931 DEBUG:boto:Data: 2023-09-22T21:49:40.931 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.931 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.931 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.931 DEBUG:boto:Params: {} 2023-09-22T21:49:40.931 DEBUG:boto:Token: None 2023-09-22T21:49:40.932 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.932 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T5ySGZtrGchL3YVKnaLJVCaeBKs= 2023-09-22T21:49:40.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T5ySGZtrGchL3YVKnaLJVCaeBKs=', 'Content-Length': '0'} 2023-09-22T21:49:40.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4cd2d31e172bf29-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.935 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:49:40.935 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:49:40.935 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.935 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.936 DEBUG:boto:Method: GET 2023-09-22T21:49:40.936 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.936 DEBUG:boto:Data: 2023-09-22T21:49:40.936 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.936 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.936 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.936 DEBUG:boto:Params: {} 2023-09-22T21:49:40.936 DEBUG:boto:Token: None 2023-09-22T21:49:40.936 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.937 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T5ySGZtrGchL3YVKnaLJVCaeBKs= 2023-09-22T21:49:40.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T5ySGZtrGchL3YVKnaLJVCaeBKs=', 'Content-Length': '0'} 2023-09-22T21:49:40.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000147967910b177990-00650e0bf4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.982 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:49:40.982 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:49:40.983 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.983 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.983 DEBUG:boto:Method: HEAD 2023-09-22T21:49:40.983 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.983 DEBUG:boto:Data: 2023-09-22T21:49:40.983 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.983 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.983 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.983 DEBUG:boto:Params: {} 2023-09-22T21:49:40.984 DEBUG:boto:Token: None 2023-09-22T21:49:40.984 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.984 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wnOTekUjgafBhL14ZSel4oiY4bg= 2023-09-22T21:49:40.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wnOTekUjgafBhL14ZSel4oiY4bg=', 'Content-Length': '0'} 2023-09-22T21:49:40.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebec4a1c1c3d8439-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.987 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:49:40.987 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:49:40.987 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.987 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.987 DEBUG:boto:Method: HEAD 2023-09-22T21:49:40.987 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.988 DEBUG:boto:Data: 2023-09-22T21:49:40.988 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.988 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.988 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.988 DEBUG:boto:Params: {} 2023-09-22T21:49:40.988 DEBUG:boto:Token: None 2023-09-22T21:49:40.988 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.988 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wnOTekUjgafBhL14ZSel4oiY4bg= 2023-09-22T21:49:40.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wnOTekUjgafBhL14ZSel4oiY4bg=', 'Content-Length': '0'} 2023-09-22T21:49:40.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6b717c5021a7d25-00650e0bf4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:49:40.992 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:49:40.992 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:49:40.992 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.992 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.992 DEBUG:boto:Method: GET 2023-09-22T21:49:40.992 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.992 DEBUG:boto:Data: 2023-09-22T21:49:40.992 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.993 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:40.993 DEBUG:boto:Port: 8001 2023-09-22T21:49:40.993 DEBUG:boto:Params: {} 2023-09-22T21:49:40.993 DEBUG:boto:Token: None 2023-09-22T21:49:40.993 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.993 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T5ySGZtrGchL3YVKnaLJVCaeBKs= 2023-09-22T21:49:40.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T5ySGZtrGchL3YVKnaLJVCaeBKs=', 'Content-Length': '0'} 2023-09-22T21:49:40.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c60cca73cb74add-00650e0bf4-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:40.996 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:49:40.997 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:49:40.997 DEBUG:boto:path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.997 DEBUG:boto:auth_path=/ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.997 DEBUG:boto:Method: GET 2023-09-22T21:49:40.997 DEBUG:boto:Path: /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.997 DEBUG:boto:Data: 2023-09-22T21:49:40.997 DEBUG:boto:Headers: {} 2023-09-22T21:49:40.997 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:40.997 DEBUG:boto:Port: 8000 2023-09-22T21:49:40.997 DEBUG:boto:Params: {} 2023-09-22T21:49:40.998 DEBUG:boto:Token: None 2023-09-22T21:49:40.998 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:40 GMT /ffzdqu-59/obj51?versionId=null 2023-09-22T21:49:40.998 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T5ySGZtrGchL3YVKnaLJVCaeBKs= 2023-09-22T21:49:40.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T5ySGZtrGchL3YVKnaLJVCaeBKs=', 'Content-Length': '0'} 2023-09-22T21:49:41.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fef4f6330ce4643-00650e0bf4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.001 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:41.001 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:41.001 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:49:41.001 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:49:41.002 DEBUG:boto:Method: GET 2023-09-22T21:49:41.002 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:49:41.002 DEBUG:boto:Data: 2023-09-22T21:49:41.002 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.002 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.002 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.002 DEBUG:boto:Params: {} 2023-09-22T21:49:41.002 DEBUG:boto:Token: None 2023-09-22T21:49:41.002 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/ 2023-09-22T21:49:41.003 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk= 2023-09-22T21:49:41.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk=', 'Content-Length': '0'} 2023-09-22T21:49:41.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4c549ee35e037a6-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.050 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:41.051 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:41.051 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:41.051 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:49:41.051 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:49:41.052 DEBUG:boto:Method: GET 2023-09-22T21:49:41.052 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:49:41.052 DEBUG:boto:Data: 2023-09-22T21:49:41.052 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.052 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.052 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.052 DEBUG:boto:Params: {} 2023-09-22T21:49:41.052 DEBUG:boto:Token: None 2023-09-22T21:49:41.052 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/ 2023-09-22T21:49:41.053 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk= 2023-09-22T21:49:41.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk=', 'Content-Length': '0'} 2023-09-22T21:49:41.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f0e804965d71340-00650e0bf5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.062 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:41.062 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:49:41.062 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:49:41.063 DEBUG:boto:Method: HEAD 2023-09-22T21:49:41.063 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:49:41.063 DEBUG:boto:Data: 2023-09-22T21:49:41.063 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.063 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.063 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.063 DEBUG:boto:Params: {} 2023-09-22T21:49:41.063 DEBUG:boto:Token: None 2023-09-22T21:49:41.063 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj51 2023-09-22T21:49:41.064 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mrx/I2ZRBQtUqmQ6eGdZmaNZv1o= 2023-09-22T21:49:41.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mrx/I2ZRBQtUqmQ6eGdZmaNZv1o=', 'Content-Length': '0'} 2023-09-22T21:49:41.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043ae42413c47b5ce-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.067 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:49:41.067 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:49:41.067 DEBUG:boto:Method: HEAD 2023-09-22T21:49:41.067 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:49:41.067 DEBUG:boto:Data: 2023-09-22T21:49:41.067 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.067 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.067 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.067 DEBUG:boto:Params: {} 2023-09-22T21:49:41.068 DEBUG:boto:Token: None 2023-09-22T21:49:41.068 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj51 2023-09-22T21:49:41.068 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mrx/I2ZRBQtUqmQ6eGdZmaNZv1o= 2023-09-22T21:49:41.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mrx/I2ZRBQtUqmQ6eGdZmaNZv1o=', 'Content-Length': '0'} 2023-09-22T21:49:41.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cf183e1344d457e-00650e0bf5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:49:41.071 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:49:41.071 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:49:41.071 DEBUG:boto:Method: GET 2023-09-22T21:49:41.071 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:49:41.071 DEBUG:boto:Data: 2023-09-22T21:49:41.071 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.071 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.072 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.072 DEBUG:boto:Params: {} 2023-09-22T21:49:41.072 DEBUG:boto:Token: None 2023-09-22T21:49:41.072 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj51 2023-09-22T21:49:41.072 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JmsDYpqzBOdVXFqUEM37dPzKfGQ= 2023-09-22T21:49:41.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JmsDYpqzBOdVXFqUEM37dPzKfGQ=', 'Content-Length': '0'} 2023-09-22T21:49:41.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a81fa41aca279eb-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.075 DEBUG:boto:path=/ffzdqu-59/obj51 2023-09-22T21:49:41.075 DEBUG:boto:auth_path=/ffzdqu-59/obj51 2023-09-22T21:49:41.075 DEBUG:boto:Method: GET 2023-09-22T21:49:41.075 DEBUG:boto:Path: /ffzdqu-59/obj51 2023-09-22T21:49:41.075 DEBUG:boto:Data: 2023-09-22T21:49:41.076 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.076 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.076 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.076 DEBUG:boto:Params: {} 2023-09-22T21:49:41.076 DEBUG:boto:Token: None 2023-09-22T21:49:41.076 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj51 2023-09-22T21:49:41.076 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JmsDYpqzBOdVXFqUEM37dPzKfGQ= 2023-09-22T21:49:41.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JmsDYpqzBOdVXFqUEM37dPzKfGQ=', 'Content-Length': '0'} 2023-09-22T21:49:41.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f418a3ae2ae526cf-00650e0bf5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.122 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:41.123 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:41.123 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:49:41.123 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:49:41.123 DEBUG:boto:Method: GET 2023-09-22T21:49:41.123 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:49:41.123 DEBUG:boto:Data: 2023-09-22T21:49:41.123 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.123 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.123 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.123 DEBUG:boto:Params: {} 2023-09-22T21:49:41.124 DEBUG:boto:Token: None 2023-09-22T21:49:41.124 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/ 2023-09-22T21:49:41.124 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk= 2023-09-22T21:49:41.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk=', 'Content-Length': '0'} 2023-09-22T21:49:41.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1c9b31d5af946b4-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.129 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:41.129 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:41.129 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:41.130 DEBUG:boto:path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:49:41.130 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj51 2023-09-22T21:49:41.130 DEBUG:boto:Method: GET 2023-09-22T21:49:41.130 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj51 2023-09-22T21:49:41.130 DEBUG:boto:Data: 2023-09-22T21:49:41.130 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.130 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.130 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.130 DEBUG:boto:Params: {} 2023-09-22T21:49:41.131 DEBUG:boto:Token: None 2023-09-22T21:49:41.131 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/ 2023-09-22T21:49:41.131 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk= 2023-09-22T21:49:41.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk=', 'Content-Length': '0'} 2023-09-22T21:49:41.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8caa10c9afebdd9-00650e0bf5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.141 DEBUG:boto:b'ffzdqu-59obj511000falseobj512023-09-22T21:36:48.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:41.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:49:41.141 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:49:41.142 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:49:41.142 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.142 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.142 DEBUG:boto:Method: GET 2023-09-22T21:49:41.142 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.142 DEBUG:boto:Data: 2023-09-22T21:49:41.142 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.142 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.142 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.143 DEBUG:boto:Params: {} 2023-09-22T21:49:41.143 DEBUG:boto:Token: None 2023-09-22T21:49:41.143 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.143 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+AJenDt1m5nlaJdIAmo7AeCPJZM= 2023-09-22T21:49:41.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+AJenDt1m5nlaJdIAmo7AeCPJZM=', 'Content-Length': '0'} 2023-09-22T21:49:41.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e03defa5838cae7d-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.190 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:49:41.190 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:49:41.191 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.191 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.191 DEBUG:boto:Method: GET 2023-09-22T21:49:41.191 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.191 DEBUG:boto:Data: 2023-09-22T21:49:41.191 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.191 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.191 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.191 DEBUG:boto:Params: {} 2023-09-22T21:49:41.192 DEBUG:boto:Token: None 2023-09-22T21:49:41.192 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.192 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+AJenDt1m5nlaJdIAmo7AeCPJZM= 2023-09-22T21:49:41.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+AJenDt1m5nlaJdIAmo7AeCPJZM=', 'Content-Length': '0'} 2023-09-22T21:49:41.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b44e7a76477739a-00650e0bf5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.195 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:49:41.195 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:49:41.195 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.195 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.196 DEBUG:boto:Method: HEAD 2023-09-22T21:49:41.196 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.196 DEBUG:boto:Data: 2023-09-22T21:49:41.196 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.196 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.196 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.196 DEBUG:boto:Params: {} 2023-09-22T21:49:41.196 DEBUG:boto:Token: None 2023-09-22T21:49:41.196 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.197 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:B5KNxeGT18ukP16sAGIiL7kP0eA= 2023-09-22T21:49:41.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:B5KNxeGT18ukP16sAGIiL7kP0eA=', 'Content-Length': '0'} 2023-09-22T21:49:41.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000564f1b81edf2d8b4-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.200 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:49:41.200 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:49:41.200 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.200 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.200 DEBUG:boto:Method: HEAD 2023-09-22T21:49:41.200 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.200 DEBUG:boto:Data: 2023-09-22T21:49:41.200 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.200 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.200 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.201 DEBUG:boto:Params: {} 2023-09-22T21:49:41.201 DEBUG:boto:Token: None 2023-09-22T21:49:41.201 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.201 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:B5KNxeGT18ukP16sAGIiL7kP0eA= 2023-09-22T21:49:41.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:B5KNxeGT18ukP16sAGIiL7kP0eA=', 'Content-Length': '0'} 2023-09-22T21:49:41.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000987b3e8ea4a5aa49-00650e0bf5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:49:41.204 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:49:41.204 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:49:41.204 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.204 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.204 DEBUG:boto:Method: GET 2023-09-22T21:49:41.205 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.205 DEBUG:boto:Data: 2023-09-22T21:49:41.205 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.205 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.205 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.205 DEBUG:boto:Params: {} 2023-09-22T21:49:41.205 DEBUG:boto:Token: None 2023-09-22T21:49:41.205 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.206 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+AJenDt1m5nlaJdIAmo7AeCPJZM= 2023-09-22T21:49:41.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+AJenDt1m5nlaJdIAmo7AeCPJZM=', 'Content-Length': '0'} 2023-09-22T21:49:41.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aa65da3f603b30b-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.209 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:49:41.209 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:49:41.209 DEBUG:boto:path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.209 DEBUG:boto:auth_path=/ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.209 DEBUG:boto:Method: GET 2023-09-22T21:49:41.209 DEBUG:boto:Path: /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.209 DEBUG:boto:Data: 2023-09-22T21:49:41.209 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.209 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.210 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.210 DEBUG:boto:Params: {} 2023-09-22T21:49:41.210 DEBUG:boto:Token: None 2023-09-22T21:49:41.210 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj55?versionId=null 2023-09-22T21:49:41.210 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+AJenDt1m5nlaJdIAmo7AeCPJZM= 2023-09-22T21:49:41.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+AJenDt1m5nlaJdIAmo7AeCPJZM=', 'Content-Length': '0'} 2023-09-22T21:49:41.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5bbe7d0eff46123-00650e0bf5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.254 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:41.254 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:41.255 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:49:41.255 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:49:41.255 DEBUG:boto:Method: GET 2023-09-22T21:49:41.255 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:49:41.255 DEBUG:boto:Data: 2023-09-22T21:49:41.255 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.255 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.255 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.255 DEBUG:boto:Params: {} 2023-09-22T21:49:41.255 DEBUG:boto:Token: None 2023-09-22T21:49:41.256 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/ 2023-09-22T21:49:41.256 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk= 2023-09-22T21:49:41.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk=', 'Content-Length': '0'} 2023-09-22T21:49:41.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000606735582e231e90-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.261 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:41.261 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:41.261 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:41.261 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:49:41.261 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:49:41.261 DEBUG:boto:Method: GET 2023-09-22T21:49:41.262 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:49:41.262 DEBUG:boto:Data: 2023-09-22T21:49:41.262 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.262 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.262 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.262 DEBUG:boto:Params: {} 2023-09-22T21:49:41.262 DEBUG:boto:Token: None 2023-09-22T21:49:41.262 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/ 2023-09-22T21:49:41.262 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk= 2023-09-22T21:49:41.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk=', 'Content-Length': '0'} 2023-09-22T21:49:41.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027591d5479c56395-00650e0bf5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.272 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:41.272 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:49:41.272 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:49:41.272 DEBUG:boto:Method: HEAD 2023-09-22T21:49:41.273 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:49:41.273 DEBUG:boto:Data: 2023-09-22T21:49:41.273 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.273 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.273 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.273 DEBUG:boto:Params: {} 2023-09-22T21:49:41.273 DEBUG:boto:Token: None 2023-09-22T21:49:41.273 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj55 2023-09-22T21:49:41.273 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nwVCS7ly2bWIxFr9Ju7fkknQt9Y= 2023-09-22T21:49:41.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nwVCS7ly2bWIxFr9Ju7fkknQt9Y=', 'Content-Length': '0'} 2023-09-22T21:49:41.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076172391f9e5bf93-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.276 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:49:41.276 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:49:41.276 DEBUG:boto:Method: HEAD 2023-09-22T21:49:41.277 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:49:41.277 DEBUG:boto:Data: 2023-09-22T21:49:41.277 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.277 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.277 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.277 DEBUG:boto:Params: {} 2023-09-22T21:49:41.277 DEBUG:boto:Token: None 2023-09-22T21:49:41.277 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj55 2023-09-22T21:49:41.277 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nwVCS7ly2bWIxFr9Ju7fkknQt9Y= 2023-09-22T21:49:41.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nwVCS7ly2bWIxFr9Ju7fkknQt9Y=', 'Content-Length': '0'} 2023-09-22T21:49:41.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b09b8bcaa592f8d-00650e0bf5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:49:41.281 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:49:41.281 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:49:41.281 DEBUG:boto:Method: GET 2023-09-22T21:49:41.281 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:49:41.281 DEBUG:boto:Data: 2023-09-22T21:49:41.281 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.281 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.281 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.282 DEBUG:boto:Params: {} 2023-09-22T21:49:41.282 DEBUG:boto:Token: None 2023-09-22T21:49:41.282 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj55 2023-09-22T21:49:41.282 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/A3nyo5ISxJxeYnGRI5IPNDNFAQ= 2023-09-22T21:49:41.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/A3nyo5ISxJxeYnGRI5IPNDNFAQ=', 'Content-Length': '0'} 2023-09-22T21:49:41.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000274239c4bcb4ebdd-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.326 DEBUG:boto:path=/ffzdqu-59/obj55 2023-09-22T21:49:41.326 DEBUG:boto:auth_path=/ffzdqu-59/obj55 2023-09-22T21:49:41.326 DEBUG:boto:Method: GET 2023-09-22T21:49:41.327 DEBUG:boto:Path: /ffzdqu-59/obj55 2023-09-22T21:49:41.327 DEBUG:boto:Data: 2023-09-22T21:49:41.327 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.327 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.327 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.327 DEBUG:boto:Params: {} 2023-09-22T21:49:41.327 DEBUG:boto:Token: None 2023-09-22T21:49:41.327 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj55 2023-09-22T21:49:41.327 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/A3nyo5ISxJxeYnGRI5IPNDNFAQ= 2023-09-22T21:49:41.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/A3nyo5ISxJxeYnGRI5IPNDNFAQ=', 'Content-Length': '0'} 2023-09-22T21:49:41.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6a9ae9241648877-00650e0bf5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.331 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:41.331 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:41.331 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:49:41.331 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:49:41.332 DEBUG:boto:Method: GET 2023-09-22T21:49:41.332 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:49:41.332 DEBUG:boto:Data: 2023-09-22T21:49:41.332 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.332 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.332 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.332 DEBUG:boto:Params: {} 2023-09-22T21:49:41.332 DEBUG:boto:Token: None 2023-09-22T21:49:41.332 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/ 2023-09-22T21:49:41.333 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk= 2023-09-22T21:49:41.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk=', 'Content-Length': '0'} 2023-09-22T21:49:41.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014260cfa4ba4463a-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.338 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:41.338 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:41.338 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:41.338 DEBUG:boto:path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:49:41.338 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj55 2023-09-22T21:49:41.339 DEBUG:boto:Method: GET 2023-09-22T21:49:41.339 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj55 2023-09-22T21:49:41.339 DEBUG:boto:Data: 2023-09-22T21:49:41.339 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.339 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.339 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.339 DEBUG:boto:Params: {} 2023-09-22T21:49:41.339 DEBUG:boto:Token: None 2023-09-22T21:49:41.339 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/ 2023-09-22T21:49:41.340 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk= 2023-09-22T21:49:41.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk=', 'Content-Length': '0'} 2023-09-22T21:49:41.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000918a2b634626d617-00650e0bf5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.390 DEBUG:boto:b'ffzdqu-59obj551000falseobj552023-09-22T21:36:31.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:41.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T21:49:41.406 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:49:41.406 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:49:41.406 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.406 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.406 DEBUG:boto:Method: GET 2023-09-22T21:49:41.406 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.406 DEBUG:boto:Data: 2023-09-22T21:49:41.406 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.406 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.407 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.407 DEBUG:boto:Params: {} 2023-09-22T21:49:41.407 DEBUG:boto:Token: None 2023-09-22T21:49:41.407 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.407 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0EhIOYjo3o8gKDG/OTKAUdlD3m4= 2023-09-22T21:49:41.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0EhIOYjo3o8gKDG/OTKAUdlD3m4=', 'Content-Length': '0'} 2023-09-22T21:49:41.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a8305e3b5e3bcc0-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.410 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:49:41.411 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:49:41.411 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.411 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.411 DEBUG:boto:Method: GET 2023-09-22T21:49:41.411 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.411 DEBUG:boto:Data: 2023-09-22T21:49:41.411 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.411 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.411 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.411 DEBUG:boto:Params: {} 2023-09-22T21:49:41.412 DEBUG:boto:Token: None 2023-09-22T21:49:41.412 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.412 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0EhIOYjo3o8gKDG/OTKAUdlD3m4= 2023-09-22T21:49:41.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0EhIOYjo3o8gKDG/OTKAUdlD3m4=', 'Content-Length': '0'} 2023-09-22T21:49:41.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5605bd79cdc899d-00650e0bf5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.415 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:49:41.415 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:49:41.415 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.415 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.416 DEBUG:boto:Method: HEAD 2023-09-22T21:49:41.416 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.416 DEBUG:boto:Data: 2023-09-22T21:49:41.416 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.416 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.416 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.416 DEBUG:boto:Params: {} 2023-09-22T21:49:41.416 DEBUG:boto:Token: None 2023-09-22T21:49:41.416 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.417 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZSbqz7dZZEV3f6KIc3EuGItjB+Y= 2023-09-22T21:49:41.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZSbqz7dZZEV3f6KIc3EuGItjB+Y=', 'Content-Length': '0'} 2023-09-22T21:49:41.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2baa9a48ec7f1c0-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.420 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:49:41.420 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:49:41.420 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.420 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.420 DEBUG:boto:Method: HEAD 2023-09-22T21:49:41.420 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.421 DEBUG:boto:Data: 2023-09-22T21:49:41.421 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.421 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.421 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.421 DEBUG:boto:Params: {} 2023-09-22T21:49:41.421 DEBUG:boto:Token: None 2023-09-22T21:49:41.421 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.421 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZSbqz7dZZEV3f6KIc3EuGItjB+Y= 2023-09-22T21:49:41.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZSbqz7dZZEV3f6KIc3EuGItjB+Y=', 'Content-Length': '0'} 2023-09-22T21:49:41.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6628c1a06318bf7-00650e0bf5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T21:49:41.424 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:49:41.424 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:49:41.424 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.425 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.425 DEBUG:boto:Method: GET 2023-09-22T21:49:41.425 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.425 DEBUG:boto:Data: 2023-09-22T21:49:41.425 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.425 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.425 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.425 DEBUG:boto:Params: {} 2023-09-22T21:49:41.425 DEBUG:boto:Token: None 2023-09-22T21:49:41.425 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.426 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0EhIOYjo3o8gKDG/OTKAUdlD3m4= 2023-09-22T21:49:41.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0EhIOYjo3o8gKDG/OTKAUdlD3m4=', 'Content-Length': '0'} 2023-09-22T21:49:41.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0776d899f96cf09-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.474 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:49:41.474 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:49:41.475 DEBUG:boto:path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.475 DEBUG:boto:auth_path=/ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.475 DEBUG:boto:Method: GET 2023-09-22T21:49:41.475 DEBUG:boto:Path: /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.475 DEBUG:boto:Data: 2023-09-22T21:49:41.475 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.475 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.475 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.475 DEBUG:boto:Params: {} 2023-09-22T21:49:41.476 DEBUG:boto:Token: None 2023-09-22T21:49:41.476 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj57?versionId=null 2023-09-22T21:49:41.476 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0EhIOYjo3o8gKDG/OTKAUdlD3m4= 2023-09-22T21:49:41.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0EhIOYjo3o8gKDG/OTKAUdlD3m4=', 'Content-Length': '0'} 2023-09-22T21:49:41.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a64084511267222-00650e0bf5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.479 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:41.479 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:41.480 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:49:41.480 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:49:41.480 DEBUG:boto:Method: GET 2023-09-22T21:49:41.480 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:49:41.480 DEBUG:boto:Data: 2023-09-22T21:49:41.480 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.480 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.480 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.480 DEBUG:boto:Params: {} 2023-09-22T21:49:41.480 DEBUG:boto:Token: None 2023-09-22T21:49:41.481 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/ 2023-09-22T21:49:41.481 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk= 2023-09-22T21:49:41.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk=', 'Content-Length': '0'} 2023-09-22T21:49:41.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b33bdf9372ba21d-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.486 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:41.487 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:41.487 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:41.487 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:49:41.487 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:49:41.487 DEBUG:boto:Method: GET 2023-09-22T21:49:41.487 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:49:41.487 DEBUG:boto:Data: 2023-09-22T21:49:41.487 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.487 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.487 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.488 DEBUG:boto:Params: {} 2023-09-22T21:49:41.488 DEBUG:boto:Token: None 2023-09-22T21:49:41.488 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/ 2023-09-22T21:49:41.488 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk= 2023-09-22T21:49:41.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk=', 'Content-Length': '0'} 2023-09-22T21:49:41.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4dbf70b3703c710-00650e0bf5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.542 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:41.543 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:49:41.543 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:49:41.543 DEBUG:boto:Method: HEAD 2023-09-22T21:49:41.543 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:49:41.543 DEBUG:boto:Data: 2023-09-22T21:49:41.543 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.543 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.543 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.543 DEBUG:boto:Params: {} 2023-09-22T21:49:41.544 DEBUG:boto:Token: None 2023-09-22T21:49:41.544 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj57 2023-09-22T21:49:41.544 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FrZQIdxvhmT/BzhsZT+pvczJ8Ys= 2023-09-22T21:49:41.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FrZQIdxvhmT/BzhsZT+pvczJ8Ys=', 'Content-Length': '0'} 2023-09-22T21:49:41.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019f10c0c58d83d5f-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.547 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:49:41.547 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:49:41.547 DEBUG:boto:Method: HEAD 2023-09-22T21:49:41.547 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:49:41.547 DEBUG:boto:Data: 2023-09-22T21:49:41.548 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.548 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.548 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.548 DEBUG:boto:Params: {} 2023-09-22T21:49:41.548 DEBUG:boto:Token: None 2023-09-22T21:49:41.548 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj57 2023-09-22T21:49:41.548 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FrZQIdxvhmT/BzhsZT+pvczJ8Ys= 2023-09-22T21:49:41.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FrZQIdxvhmT/BzhsZT+pvczJ8Ys=', 'Content-Length': '0'} 2023-09-22T21:49:41.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6576ffd53989489-00650e0bf5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T21:49:41.552 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:49:41.552 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:49:41.552 DEBUG:boto:Method: GET 2023-09-22T21:49:41.552 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:49:41.552 DEBUG:boto:Data: 2023-09-22T21:49:41.552 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.552 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.552 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.552 DEBUG:boto:Params: {} 2023-09-22T21:49:41.552 DEBUG:boto:Token: None 2023-09-22T21:49:41.553 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj57 2023-09-22T21:49:41.553 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:diEQ3v2j3pEY2utXXXuXYNrk3I0= 2023-09-22T21:49:41.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:diEQ3v2j3pEY2utXXXuXYNrk3I0=', 'Content-Length': '0'} 2023-09-22T21:49:41.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000821ef50f745bd458-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.556 DEBUG:boto:path=/ffzdqu-59/obj57 2023-09-22T21:49:41.556 DEBUG:boto:auth_path=/ffzdqu-59/obj57 2023-09-22T21:49:41.556 DEBUG:boto:Method: GET 2023-09-22T21:49:41.556 DEBUG:boto:Path: /ffzdqu-59/obj57 2023-09-22T21:49:41.556 DEBUG:boto:Data: 2023-09-22T21:49:41.556 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.556 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.556 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.556 DEBUG:boto:Params: {} 2023-09-22T21:49:41.557 DEBUG:boto:Token: None 2023-09-22T21:49:41.557 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj57 2023-09-22T21:49:41.557 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:diEQ3v2j3pEY2utXXXuXYNrk3I0= 2023-09-22T21:49:41.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:diEQ3v2j3pEY2utXXXuXYNrk3I0=', 'Content-Length': '0'} 2023-09-22T21:49:41.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056b49839fa1b0e4c-00650e0bf5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.560 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:41.560 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:41.560 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:49:41.560 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:49:41.561 DEBUG:boto:Method: GET 2023-09-22T21:49:41.561 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:49:41.561 DEBUG:boto:Data: 2023-09-22T21:49:41.561 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.561 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.561 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.561 DEBUG:boto:Params: {} 2023-09-22T21:49:41.561 DEBUG:boto:Token: None 2023-09-22T21:49:41.561 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/ 2023-09-22T21:49:41.561 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk= 2023-09-22T21:49:41.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk=', 'Content-Length': '0'} 2023-09-22T21:49:41.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc95992810a1d46e-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.606 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:41.607 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:41.607 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:41.607 DEBUG:boto:path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:49:41.607 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj57 2023-09-22T21:49:41.607 DEBUG:boto:Method: GET 2023-09-22T21:49:41.607 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj57 2023-09-22T21:49:41.607 DEBUG:boto:Data: 2023-09-22T21:49:41.607 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.607 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.608 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.608 DEBUG:boto:Params: {} 2023-09-22T21:49:41.608 DEBUG:boto:Token: None 2023-09-22T21:49:41.608 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/ 2023-09-22T21:49:41.608 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk= 2023-09-22T21:49:41.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk=', 'Content-Length': '0'} 2023-09-22T21:49:41.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a79519243cf9697d-00650e0bf5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.618 DEBUG:boto:b'ffzdqu-59obj571000falseobj572023-09-22T21:38:31.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:41.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-22T21:49:41.619 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:49:41.619 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:49:41.619 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.619 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.619 DEBUG:boto:Method: GET 2023-09-22T21:49:41.619 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.620 DEBUG:boto:Data: 2023-09-22T21:49:41.620 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.620 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.620 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.620 DEBUG:boto:Params: {} 2023-09-22T21:49:41.620 DEBUG:boto:Token: None 2023-09-22T21:49:41.620 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.620 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M5VBd5hI5YC/wgPKncb67pICYH8= 2023-09-22T21:49:41.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M5VBd5hI5YC/wgPKncb67pICYH8=', 'Content-Length': '0'} 2023-09-22T21:49:41.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a4ebf9cc6d2d225-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.624 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:49:41.624 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:49:41.624 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.624 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.624 DEBUG:boto:Method: GET 2023-09-22T21:49:41.624 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.625 DEBUG:boto:Data: 2023-09-22T21:49:41.625 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.625 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.625 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.625 DEBUG:boto:Params: {} 2023-09-22T21:49:41.625 DEBUG:boto:Token: None 2023-09-22T21:49:41.625 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.625 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M5VBd5hI5YC/wgPKncb67pICYH8= 2023-09-22T21:49:41.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M5VBd5hI5YC/wgPKncb67pICYH8=', 'Content-Length': '0'} 2023-09-22T21:49:41.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fe91d684370c527-00650e0bf5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.670 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:49:41.671 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:49:41.671 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.671 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.671 DEBUG:boto:Method: HEAD 2023-09-22T21:49:41.671 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.671 DEBUG:boto:Data: 2023-09-22T21:49:41.671 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.671 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.671 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.672 DEBUG:boto:Params: {} 2023-09-22T21:49:41.672 DEBUG:boto:Token: None 2023-09-22T21:49:41.672 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.672 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:A0jWxX0+zFiOQpcL+1AwFLw3M8I= 2023-09-22T21:49:41.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:A0jWxX0+zFiOQpcL+1AwFLw3M8I=', 'Content-Length': '0'} 2023-09-22T21:49:41.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000210ef80b3741f962-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.675 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:49:41.675 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:49:41.676 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.676 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.676 DEBUG:boto:Method: HEAD 2023-09-22T21:49:41.676 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.676 DEBUG:boto:Data: 2023-09-22T21:49:41.676 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.676 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.676 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.676 DEBUG:boto:Params: {} 2023-09-22T21:49:41.676 DEBUG:boto:Token: None 2023-09-22T21:49:41.677 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.677 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:A0jWxX0+zFiOQpcL+1AwFLw3M8I= 2023-09-22T21:49:41.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:A0jWxX0+zFiOQpcL+1AwFLw3M8I=', 'Content-Length': '0'} 2023-09-22T21:49:41.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b6980ba6961105d-00650e0bf5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-22T21:49:41.680 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:49:41.680 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:49:41.680 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.680 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.680 DEBUG:boto:Method: GET 2023-09-22T21:49:41.680 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.680 DEBUG:boto:Data: 2023-09-22T21:49:41.681 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.681 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.681 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.681 DEBUG:boto:Params: {} 2023-09-22T21:49:41.681 DEBUG:boto:Token: None 2023-09-22T21:49:41.681 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.681 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M5VBd5hI5YC/wgPKncb67pICYH8= 2023-09-22T21:49:41.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M5VBd5hI5YC/wgPKncb67pICYH8=', 'Content-Length': '0'} 2023-09-22T21:49:41.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e10618812a56b72-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.684 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:49:41.684 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:49:41.685 DEBUG:boto:path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.685 DEBUG:boto:auth_path=/ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.685 DEBUG:boto:Method: GET 2023-09-22T21:49:41.685 DEBUG:boto:Path: /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.685 DEBUG:boto:Data: 2023-09-22T21:49:41.685 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.685 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.685 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.685 DEBUG:boto:Params: {} 2023-09-22T21:49:41.685 DEBUG:boto:Token: None 2023-09-22T21:49:41.686 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj58?versionId=null 2023-09-22T21:49:41.686 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M5VBd5hI5YC/wgPKncb67pICYH8= 2023-09-22T21:49:41.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M5VBd5hI5YC/wgPKncb67pICYH8=', 'Content-Length': '0'} 2023-09-22T21:49:41.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d13fa7bd8f3fdc66-00650e0bf5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.689 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:41.689 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:41.689 DEBUG:boto:path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:49:41.689 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:49:41.690 DEBUG:boto:Method: GET 2023-09-22T21:49:41.690 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj58 2023-09-22T21:49:41.690 DEBUG:boto:Data: 2023-09-22T21:49:41.690 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.690 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.690 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.690 DEBUG:boto:Params: {} 2023-09-22T21:49:41.690 DEBUG:boto:Token: None 2023-09-22T21:49:41.690 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/ 2023-09-22T21:49:41.691 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk= 2023-09-22T21:49:41.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk=', 'Content-Length': '0'} 2023-09-22T21:49:41.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de87bde7496dd02a-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.738 DEBUG:boto:b'ffzdqu-59obj581000falseobj582023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:41.739 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:41.739 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:41.739 DEBUG:boto:path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:49:41.739 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:49:41.739 DEBUG:boto:Method: GET 2023-09-22T21:49:41.739 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj58 2023-09-22T21:49:41.739 DEBUG:boto:Data: 2023-09-22T21:49:41.739 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.740 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.740 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.740 DEBUG:boto:Params: {} 2023-09-22T21:49:41.740 DEBUG:boto:Token: None 2023-09-22T21:49:41.740 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/ 2023-09-22T21:49:41.740 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk= 2023-09-22T21:49:41.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk=', 'Content-Length': '0'} 2023-09-22T21:49:41.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a12308958187ae30-00650e0bf5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.751 DEBUG:boto:b'ffzdqu-59obj581000falseobj582023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:41.751 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:49:41.751 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:49:41.751 DEBUG:boto:Method: HEAD 2023-09-22T21:49:41.751 DEBUG:boto:Path: /ffzdqu-59/obj58 2023-09-22T21:49:41.751 DEBUG:boto:Data: 2023-09-22T21:49:41.752 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.752 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.752 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.752 DEBUG:boto:Params: {} 2023-09-22T21:49:41.752 DEBUG:boto:Token: None 2023-09-22T21:49:41.752 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj58 2023-09-22T21:49:41.752 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tHWMs4OD+u+uZfulc+KGNjuFpJE= 2023-09-22T21:49:41.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tHWMs4OD+u+uZfulc+KGNjuFpJE=', 'Content-Length': '0'} 2023-09-22T21:49:41.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3f13da19648cea8-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.755 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:49:41.755 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:49:41.756 DEBUG:boto:Method: HEAD 2023-09-22T21:49:41.756 DEBUG:boto:Path: /ffzdqu-59/obj58 2023-09-22T21:49:41.756 DEBUG:boto:Data: 2023-09-22T21:49:41.756 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.756 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.756 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.756 DEBUG:boto:Params: {} 2023-09-22T21:49:41.756 DEBUG:boto:Token: None 2023-09-22T21:49:41.756 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj58 2023-09-22T21:49:41.757 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tHWMs4OD+u+uZfulc+KGNjuFpJE= 2023-09-22T21:49:41.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tHWMs4OD+u+uZfulc+KGNjuFpJE=', 'Content-Length': '0'} 2023-09-22T21:49:41.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdc7c90917a2c91a-00650e0bf5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-22T21:49:41.760 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:49:41.760 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:49:41.760 DEBUG:boto:Method: GET 2023-09-22T21:49:41.760 DEBUG:boto:Path: /ffzdqu-59/obj58 2023-09-22T21:49:41.760 DEBUG:boto:Data: 2023-09-22T21:49:41.760 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.761 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.761 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.761 DEBUG:boto:Params: {} 2023-09-22T21:49:41.761 DEBUG:boto:Token: None 2023-09-22T21:49:41.761 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj58 2023-09-22T21:49:41.761 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JlKeJsTHZp2gOhhjH8itrJ2ARtU= 2023-09-22T21:49:41.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JlKeJsTHZp2gOhhjH8itrJ2ARtU=', 'Content-Length': '0'} 2023-09-22T21:49:41.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1a4f012a08a966f-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.764 DEBUG:boto:path=/ffzdqu-59/obj58 2023-09-22T21:49:41.764 DEBUG:boto:auth_path=/ffzdqu-59/obj58 2023-09-22T21:49:41.764 DEBUG:boto:Method: GET 2023-09-22T21:49:41.764 DEBUG:boto:Path: /ffzdqu-59/obj58 2023-09-22T21:49:41.765 DEBUG:boto:Data: 2023-09-22T21:49:41.765 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.765 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.765 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.765 DEBUG:boto:Params: {} 2023-09-22T21:49:41.765 DEBUG:boto:Token: None 2023-09-22T21:49:41.765 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj58 2023-09-22T21:49:41.765 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JlKeJsTHZp2gOhhjH8itrJ2ARtU= 2023-09-22T21:49:41.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JlKeJsTHZp2gOhhjH8itrJ2ARtU=', 'Content-Length': '0'} 2023-09-22T21:49:41.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddca1ca78919b48d-00650e0bf5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.810 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:41.811 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:41.811 DEBUG:boto:path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:49:41.811 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:49:41.811 DEBUG:boto:Method: GET 2023-09-22T21:49:41.811 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj58 2023-09-22T21:49:41.811 DEBUG:boto:Data: 2023-09-22T21:49:41.811 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.811 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.811 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.811 DEBUG:boto:Params: {} 2023-09-22T21:49:41.812 DEBUG:boto:Token: None 2023-09-22T21:49:41.812 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/ 2023-09-22T21:49:41.812 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk= 2023-09-22T21:49:41.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk=', 'Content-Length': '0'} 2023-09-22T21:49:41.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000039ad27dc477561f-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.819 DEBUG:boto:b'ffzdqu-59obj581000falseobj582023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:41.819 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:41.819 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:41.819 DEBUG:boto:path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:49:41.819 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj58 2023-09-22T21:49:41.820 DEBUG:boto:Method: GET 2023-09-22T21:49:41.820 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj58 2023-09-22T21:49:41.820 DEBUG:boto:Data: 2023-09-22T21:49:41.820 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.820 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.820 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.820 DEBUG:boto:Params: {} 2023-09-22T21:49:41.820 DEBUG:boto:Token: None 2023-09-22T21:49:41.820 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/ 2023-09-22T21:49:41.821 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk= 2023-09-22T21:49:41.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk=', 'Content-Length': '0'} 2023-09-22T21:49:41.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032588e64de778f22-00650e0bf5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.830 DEBUG:boto:b'ffzdqu-59obj581000falseobj582023-09-22T21:43:12.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:41.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-22T21:49:41.831 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:49:41.831 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:49:41.831 DEBUG:boto:path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.831 DEBUG:boto:auth_path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.831 DEBUG:boto:Method: GET 2023-09-22T21:49:41.831 DEBUG:boto:Path: /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.831 DEBUG:boto:Data: 2023-09-22T21:49:41.831 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.832 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.832 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.832 DEBUG:boto:Params: {} 2023-09-22T21:49:41.832 DEBUG:boto:Token: None 2023-09-22T21:49:41.832 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.832 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3JWH6+zvGYZHgtD6QjQPHfEz3+M= 2023-09-22T21:49:41.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3JWH6+zvGYZHgtD6QjQPHfEz3+M=', 'Content-Length': '0'} 2023-09-22T21:49:41.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067b69999eb63e724-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.878 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:49:41.878 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:49:41.879 DEBUG:boto:path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.879 DEBUG:boto:auth_path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.879 DEBUG:boto:Method: GET 2023-09-22T21:49:41.879 DEBUG:boto:Path: /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.879 DEBUG:boto:Data: 2023-09-22T21:49:41.879 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.879 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.879 DEBUG:boto:Params: {} 2023-09-22T21:49:41.880 DEBUG:boto:Token: None 2023-09-22T21:49:41.880 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.880 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3JWH6+zvGYZHgtD6QjQPHfEz3+M= 2023-09-22T21:49:41.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3JWH6+zvGYZHgtD6QjQPHfEz3+M=', 'Content-Length': '0'} 2023-09-22T21:49:41.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000674c67c875334ba8-00650e0bf5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.883 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:49:41.883 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:49:41.883 DEBUG:boto:path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.883 DEBUG:boto:auth_path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.884 DEBUG:boto:Method: HEAD 2023-09-22T21:49:41.884 DEBUG:boto:Path: /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.884 DEBUG:boto:Data: 2023-09-22T21:49:41.884 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.884 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.884 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.884 DEBUG:boto:Params: {} 2023-09-22T21:49:41.884 DEBUG:boto:Token: None 2023-09-22T21:49:41.884 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.885 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:K6xLFM4FWu5JnSSWlP2oOFoAfSQ= 2023-09-22T21:49:41.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:K6xLFM4FWu5JnSSWlP2oOFoAfSQ=', 'Content-Length': '0'} 2023-09-22T21:49:41.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bacd675e9b484b43-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.888 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:49:41.888 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:49:41.888 DEBUG:boto:path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.888 DEBUG:boto:auth_path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.888 DEBUG:boto:Method: HEAD 2023-09-22T21:49:41.888 DEBUG:boto:Path: /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.888 DEBUG:boto:Data: 2023-09-22T21:49:41.889 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.889 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.889 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.889 DEBUG:boto:Params: {} 2023-09-22T21:49:41.889 DEBUG:boto:Token: None 2023-09-22T21:49:41.889 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.889 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:K6xLFM4FWu5JnSSWlP2oOFoAfSQ= 2023-09-22T21:49:41.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:K6xLFM4FWu5JnSSWlP2oOFoAfSQ=', 'Content-Length': '0'} 2023-09-22T21:49:41.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6ab5f325f43d931-00650e0bf5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-22T21:49:41.892 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:49:41.892 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:49:41.893 DEBUG:boto:path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.893 DEBUG:boto:auth_path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.893 DEBUG:boto:Method: GET 2023-09-22T21:49:41.893 DEBUG:boto:Path: /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.893 DEBUG:boto:Data: 2023-09-22T21:49:41.893 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.893 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.893 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.893 DEBUG:boto:Params: {} 2023-09-22T21:49:41.893 DEBUG:boto:Token: None 2023-09-22T21:49:41.894 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.894 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3JWH6+zvGYZHgtD6QjQPHfEz3+M= 2023-09-22T21:49:41.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3JWH6+zvGYZHgtD6QjQPHfEz3+M=', 'Content-Length': '0'} 2023-09-22T21:49:41.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeb92543bf99106b-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.897 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:49:41.897 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:49:41.897 DEBUG:boto:path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.897 DEBUG:boto:auth_path=/ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.897 DEBUG:boto:Method: GET 2023-09-22T21:49:41.898 DEBUG:boto:Path: /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.898 DEBUG:boto:Data: 2023-09-22T21:49:41.898 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.898 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.898 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.898 DEBUG:boto:Params: {} 2023-09-22T21:49:41.898 DEBUG:boto:Token: None 2023-09-22T21:49:41.898 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj62?versionId=null 2023-09-22T21:49:41.899 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3JWH6+zvGYZHgtD6QjQPHfEz3+M= 2023-09-22T21:49:41.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3JWH6+zvGYZHgtD6QjQPHfEz3+M=', 'Content-Length': '0'} 2023-09-22T21:49:41.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078a651237b9a62c2-00650e0bf5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.946 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:41.947 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:41.947 DEBUG:boto:path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:49:41.947 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:49:41.947 DEBUG:boto:Method: GET 2023-09-22T21:49:41.947 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj62 2023-09-22T21:49:41.947 DEBUG:boto:Data: 2023-09-22T21:49:41.947 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.947 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.947 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.947 DEBUG:boto:Params: {} 2023-09-22T21:49:41.948 DEBUG:boto:Token: None 2023-09-22T21:49:41.948 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/ 2023-09-22T21:49:41.948 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk= 2023-09-22T21:49:41.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk=', 'Content-Length': '0'} 2023-09-22T21:49:41.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ed85c2a41b4d57c-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.953 DEBUG:boto:b'ffzdqu-59obj621000falseobj622023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:41.953 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:41.953 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:41.953 DEBUG:boto:path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:49:41.953 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:49:41.954 DEBUG:boto:Method: GET 2023-09-22T21:49:41.954 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj62 2023-09-22T21:49:41.954 DEBUG:boto:Data: 2023-09-22T21:49:41.954 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.954 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.954 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.954 DEBUG:boto:Params: {} 2023-09-22T21:49:41.954 DEBUG:boto:Token: None 2023-09-22T21:49:41.955 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/ 2023-09-22T21:49:41.955 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk= 2023-09-22T21:49:41.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71xwiGkDTmn07JFjwC5zS51vbpk=', 'Content-Length': '0'} 2023-09-22T21:49:41.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000302c28358557b4ac-00650e0bf5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.965 DEBUG:boto:b'ffzdqu-59obj621000falseobj622023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:41.965 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:49:41.965 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:49:41.965 DEBUG:boto:Method: HEAD 2023-09-22T21:49:41.965 DEBUG:boto:Path: /ffzdqu-59/obj62 2023-09-22T21:49:41.965 DEBUG:boto:Data: 2023-09-22T21:49:41.965 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.966 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.966 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.966 DEBUG:boto:Params: {} 2023-09-22T21:49:41.966 DEBUG:boto:Token: None 2023-09-22T21:49:41.966 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj62 2023-09-22T21:49:41.966 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mx1U+1zirycevDPyLcI7rz4Z1Tw= 2023-09-22T21:49:41.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mx1U+1zirycevDPyLcI7rz4Z1Tw=', 'Content-Length': '0'} 2023-09-22T21:49:41.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d406306c7494eed5-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.969 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:49:41.969 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:49:41.970 DEBUG:boto:Method: HEAD 2023-09-22T21:49:41.970 DEBUG:boto:Path: /ffzdqu-59/obj62 2023-09-22T21:49:41.970 DEBUG:boto:Data: 2023-09-22T21:49:41.970 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.970 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:41.970 DEBUG:boto:Port: 8000 2023-09-22T21:49:41.970 DEBUG:boto:Params: {} 2023-09-22T21:49:41.970 DEBUG:boto:Token: None 2023-09-22T21:49:41.971 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj62 2023-09-22T21:49:41.971 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mx1U+1zirycevDPyLcI7rz4Z1Tw= 2023-09-22T21:49:41.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mx1U+1zirycevDPyLcI7rz4Z1Tw=', 'Content-Length': '0'} 2023-09-22T21:49:41.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d38c98a898c5c3b0-00650e0bf5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:41.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-22T21:49:41.974 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:49:41.974 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:49:41.974 DEBUG:boto:Method: GET 2023-09-22T21:49:41.974 DEBUG:boto:Path: /ffzdqu-59/obj62 2023-09-22T21:49:41.974 DEBUG:boto:Data: 2023-09-22T21:49:41.974 DEBUG:boto:Headers: {} 2023-09-22T21:49:41.974 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:41.975 DEBUG:boto:Port: 8001 2023-09-22T21:49:41.975 DEBUG:boto:Params: {} 2023-09-22T21:49:41.975 DEBUG:boto:Token: None 2023-09-22T21:49:41.975 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:41 GMT /ffzdqu-59/obj62 2023-09-22T21:49:41.975 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Pzc0mz2nfREpfnaVENIJXiSvKHs= 2023-09-22T21:49:41.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Pzc0mz2nfREpfnaVENIJXiSvKHs=', 'Content-Length': '0'} 2023-09-22T21:49:41.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086eb9e0c11fe2967-00650e0bf5-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.018 DEBUG:boto:path=/ffzdqu-59/obj62 2023-09-22T21:49:42.018 DEBUG:boto:auth_path=/ffzdqu-59/obj62 2023-09-22T21:49:42.019 DEBUG:boto:Method: GET 2023-09-22T21:49:42.019 DEBUG:boto:Path: /ffzdqu-59/obj62 2023-09-22T21:49:42.019 DEBUG:boto:Data: 2023-09-22T21:49:42.019 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.019 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.019 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.019 DEBUG:boto:Params: {} 2023-09-22T21:49:42.019 DEBUG:boto:Token: None 2023-09-22T21:49:42.019 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj62 2023-09-22T21:49:42.020 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y0Cw8MC61pg8bx1eEtNxLx12p3E= 2023-09-22T21:49:42.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y0Cw8MC61pg8bx1eEtNxLx12p3E=', 'Content-Length': '0'} 2023-09-22T21:49:42.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8c02f6e1fd6c0ac-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.023 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:42.023 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:42.023 DEBUG:boto:path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:49:42.023 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:49:42.023 DEBUG:boto:Method: GET 2023-09-22T21:49:42.023 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj62 2023-09-22T21:49:42.023 DEBUG:boto:Data: 2023-09-22T21:49:42.024 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.024 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.024 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.024 DEBUG:boto:Params: {} 2023-09-22T21:49:42.024 DEBUG:boto:Token: None 2023-09-22T21:49:42.024 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/ 2023-09-22T21:49:42.024 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE= 2023-09-22T21:49:42.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE=', 'Content-Length': '0'} 2023-09-22T21:49:42.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd6fa2b16288ab4f-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.030 DEBUG:boto:b'ffzdqu-59obj621000falseobj622023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:42.030 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:42.031 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:42.031 DEBUG:boto:path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:49:42.031 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj62 2023-09-22T21:49:42.031 DEBUG:boto:Method: GET 2023-09-22T21:49:42.031 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj62 2023-09-22T21:49:42.031 DEBUG:boto:Data: 2023-09-22T21:49:42.031 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.031 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.031 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.031 DEBUG:boto:Params: {} 2023-09-22T21:49:42.031 DEBUG:boto:Token: None 2023-09-22T21:49:42.032 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/ 2023-09-22T21:49:42.032 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE= 2023-09-22T21:49:42.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE=', 'Content-Length': '0'} 2023-09-22T21:49:42.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff95762ddcb18b98-00650e0bf6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.082 DEBUG:boto:b'ffzdqu-59obj621000falseobj622023-09-22T21:45:13.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:42.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:49:42.083 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:49:42.083 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:49:42.083 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.083 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.083 DEBUG:boto:Method: GET 2023-09-22T21:49:42.083 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.083 DEBUG:boto:Data: 2023-09-22T21:49:42.083 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.084 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.084 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.084 DEBUG:boto:Params: {} 2023-09-22T21:49:42.084 DEBUG:boto:Token: None 2023-09-22T21:49:42.084 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.084 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6UrgghXpyDCwf49l5UmEJhi9KwE= 2023-09-22T21:49:42.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6UrgghXpyDCwf49l5UmEJhi9KwE=', 'Content-Length': '0'} 2023-09-22T21:49:42.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7d4c41883dc0255-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.088 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:49:42.088 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:49:42.088 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.088 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.088 DEBUG:boto:Method: GET 2023-09-22T21:49:42.088 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.088 DEBUG:boto:Data: 2023-09-22T21:49:42.088 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.089 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.089 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.089 DEBUG:boto:Params: {} 2023-09-22T21:49:42.089 DEBUG:boto:Token: None 2023-09-22T21:49:42.089 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.089 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6UrgghXpyDCwf49l5UmEJhi9KwE= 2023-09-22T21:49:42.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6UrgghXpyDCwf49l5UmEJhi9KwE=', 'Content-Length': '0'} 2023-09-22T21:49:42.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a7ba8408ed9599c-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.092 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:49:42.092 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:49:42.092 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.093 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.093 DEBUG:boto:Method: HEAD 2023-09-22T21:49:42.093 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.093 DEBUG:boto:Data: 2023-09-22T21:49:42.093 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.093 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.093 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.093 DEBUG:boto:Params: {} 2023-09-22T21:49:42.093 DEBUG:boto:Token: None 2023-09-22T21:49:42.093 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.094 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:A7iVbUrGVfU9Ke5PtewA3Q1jZVc= 2023-09-22T21:49:42.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:A7iVbUrGVfU9Ke5PtewA3Q1jZVc=', 'Content-Length': '0'} 2023-09-22T21:49:42.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bc0b11ec0567e7d-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.097 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:49:42.097 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:49:42.097 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.097 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.097 DEBUG:boto:Method: HEAD 2023-09-22T21:49:42.097 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.098 DEBUG:boto:Data: 2023-09-22T21:49:42.098 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.098 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.098 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.098 DEBUG:boto:Params: {} 2023-09-22T21:49:42.098 DEBUG:boto:Token: None 2023-09-22T21:49:42.098 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.098 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:A7iVbUrGVfU9Ke5PtewA3Q1jZVc= 2023-09-22T21:49:42.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:A7iVbUrGVfU9Ke5PtewA3Q1jZVc=', 'Content-Length': '0'} 2023-09-22T21:49:42.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000046d38192b47486a-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:49:42.101 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:49:42.102 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:49:42.102 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.102 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.102 DEBUG:boto:Method: GET 2023-09-22T21:49:42.102 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.102 DEBUG:boto:Data: 2023-09-22T21:49:42.102 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.102 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.102 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.103 DEBUG:boto:Params: {} 2023-09-22T21:49:42.103 DEBUG:boto:Token: None 2023-09-22T21:49:42.103 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.103 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6UrgghXpyDCwf49l5UmEJhi9KwE= 2023-09-22T21:49:42.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6UrgghXpyDCwf49l5UmEJhi9KwE=', 'Content-Length': '0'} 2023-09-22T21:49:42.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee7eb4c0472287ea-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.106 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:49:42.106 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:49:42.106 DEBUG:boto:path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.107 DEBUG:boto:auth_path=/ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.107 DEBUG:boto:Method: GET 2023-09-22T21:49:42.107 DEBUG:boto:Path: /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.107 DEBUG:boto:Data: 2023-09-22T21:49:42.107 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.107 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.107 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.107 DEBUG:boto:Params: {} 2023-09-22T21:49:42.107 DEBUG:boto:Token: None 2023-09-22T21:49:42.108 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj68?versionId=null 2023-09-22T21:49:42.108 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6UrgghXpyDCwf49l5UmEJhi9KwE= 2023-09-22T21:49:42.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6UrgghXpyDCwf49l5UmEJhi9KwE=', 'Content-Length': '0'} 2023-09-22T21:49:42.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005afc75b37d36444f-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.111 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:42.111 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:42.111 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:49:42.111 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:49:42.111 DEBUG:boto:Method: GET 2023-09-22T21:49:42.111 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:49:42.111 DEBUG:boto:Data: 2023-09-22T21:49:42.111 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.111 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.111 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.112 DEBUG:boto:Params: {} 2023-09-22T21:49:42.112 DEBUG:boto:Token: None 2023-09-22T21:49:42.112 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/ 2023-09-22T21:49:42.112 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE= 2023-09-22T21:49:42.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE=', 'Content-Length': '0'} 2023-09-22T21:49:42.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000280bea5f2c6108d5-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.117 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:42.117 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:42.118 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:42.118 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:49:42.118 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:49:42.118 DEBUG:boto:Method: GET 2023-09-22T21:49:42.118 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:49:42.118 DEBUG:boto:Data: 2023-09-22T21:49:42.118 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.118 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.119 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.119 DEBUG:boto:Params: {} 2023-09-22T21:49:42.119 DEBUG:boto:Token: None 2023-09-22T21:49:42.119 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/ 2023-09-22T21:49:42.119 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE= 2023-09-22T21:49:42.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE=', 'Content-Length': '0'} 2023-09-22T21:49:42.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef72917c57948b3c-00650e0bf6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.174 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:42.175 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:49:42.175 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:49:42.175 DEBUG:boto:Method: HEAD 2023-09-22T21:49:42.175 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:49:42.175 DEBUG:boto:Data: 2023-09-22T21:49:42.175 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.175 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.175 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.175 DEBUG:boto:Params: {} 2023-09-22T21:49:42.176 DEBUG:boto:Token: None 2023-09-22T21:49:42.176 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj68 2023-09-22T21:49:42.176 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3FmoiIkGrd63MkUfw6msN6oM9+U= 2023-09-22T21:49:42.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3FmoiIkGrd63MkUfw6msN6oM9+U=', 'Content-Length': '0'} 2023-09-22T21:49:42.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000166b2b52d22657a3-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.179 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:49:42.179 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:49:42.179 DEBUG:boto:Method: HEAD 2023-09-22T21:49:42.179 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:49:42.179 DEBUG:boto:Data: 2023-09-22T21:49:42.180 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.180 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.180 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.180 DEBUG:boto:Params: {} 2023-09-22T21:49:42.180 DEBUG:boto:Token: None 2023-09-22T21:49:42.180 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj68 2023-09-22T21:49:42.180 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3FmoiIkGrd63MkUfw6msN6oM9+U= 2023-09-22T21:49:42.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3FmoiIkGrd63MkUfw6msN6oM9+U=', 'Content-Length': '0'} 2023-09-22T21:49:42.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055baf9c8f5df6da2-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:49:42.183 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:49:42.184 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:49:42.184 DEBUG:boto:Method: GET 2023-09-22T21:49:42.184 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:49:42.184 DEBUG:boto:Data: 2023-09-22T21:49:42.184 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.184 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.184 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.184 DEBUG:boto:Params: {} 2023-09-22T21:49:42.184 DEBUG:boto:Token: None 2023-09-22T21:49:42.185 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj68 2023-09-22T21:49:42.185 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wc0Ns4u9nB4o6aCO5e8YFUe+oaY= 2023-09-22T21:49:42.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wc0Ns4u9nB4o6aCO5e8YFUe+oaY=', 'Content-Length': '0'} 2023-09-22T21:49:42.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9af2eb1118b81f0-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.234 DEBUG:boto:path=/ffzdqu-59/obj68 2023-09-22T21:49:42.234 DEBUG:boto:auth_path=/ffzdqu-59/obj68 2023-09-22T21:49:42.235 DEBUG:boto:Method: GET 2023-09-22T21:49:42.235 DEBUG:boto:Path: /ffzdqu-59/obj68 2023-09-22T21:49:42.235 DEBUG:boto:Data: 2023-09-22T21:49:42.235 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.235 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.235 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.235 DEBUG:boto:Params: {} 2023-09-22T21:49:42.235 DEBUG:boto:Token: None 2023-09-22T21:49:42.235 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj68 2023-09-22T21:49:42.236 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wc0Ns4u9nB4o6aCO5e8YFUe+oaY= 2023-09-22T21:49:42.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wc0Ns4u9nB4o6aCO5e8YFUe+oaY=', 'Content-Length': '0'} 2023-09-22T21:49:42.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009efbe8ebdd0486d6-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.239 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:42.239 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:42.239 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:49:42.239 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:49:42.239 DEBUG:boto:Method: GET 2023-09-22T21:49:42.239 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:49:42.239 DEBUG:boto:Data: 2023-09-22T21:49:42.239 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.240 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.240 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.240 DEBUG:boto:Params: {} 2023-09-22T21:49:42.240 DEBUG:boto:Token: None 2023-09-22T21:49:42.240 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/ 2023-09-22T21:49:42.240 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE= 2023-09-22T21:49:42.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE=', 'Content-Length': '0'} 2023-09-22T21:49:42.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea39479984360142-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.245 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:42.246 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:42.246 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:42.246 DEBUG:boto:path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:49:42.246 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj68 2023-09-22T21:49:42.246 DEBUG:boto:Method: GET 2023-09-22T21:49:42.246 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj68 2023-09-22T21:49:42.246 DEBUG:boto:Data: 2023-09-22T21:49:42.246 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.246 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.247 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.247 DEBUG:boto:Params: {} 2023-09-22T21:49:42.247 DEBUG:boto:Token: None 2023-09-22T21:49:42.247 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/ 2023-09-22T21:49:42.247 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE= 2023-09-22T21:49:42.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE=', 'Content-Length': '0'} 2023-09-22T21:49:42.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b70f5bb746623281-00650e0bf6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.258 DEBUG:boto:b'ffzdqu-59obj681000falseobj682023-09-22T21:36:48.760Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:42.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-22T21:49:42.259 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:49:42.259 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:49:42.259 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.259 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.259 DEBUG:boto:Method: GET 2023-09-22T21:49:42.259 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.259 DEBUG:boto:Data: 2023-09-22T21:49:42.259 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.259 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.259 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.260 DEBUG:boto:Params: {} 2023-09-22T21:49:42.260 DEBUG:boto:Token: None 2023-09-22T21:49:42.260 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.260 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lJ5uDywGzehgEcJC1Rj9FtYlIUA= 2023-09-22T21:49:42.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lJ5uDywGzehgEcJC1Rj9FtYlIUA=', 'Content-Length': '0'} 2023-09-22T21:49:42.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099127bb4b4a7c9e0-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.263 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:49:42.264 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:49:42.264 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.264 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.264 DEBUG:boto:Method: GET 2023-09-22T21:49:42.264 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.264 DEBUG:boto:Data: 2023-09-22T21:49:42.264 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.264 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.264 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.265 DEBUG:boto:Params: {} 2023-09-22T21:49:42.265 DEBUG:boto:Token: None 2023-09-22T21:49:42.265 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.265 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lJ5uDywGzehgEcJC1Rj9FtYlIUA= 2023-09-22T21:49:42.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lJ5uDywGzehgEcJC1Rj9FtYlIUA=', 'Content-Length': '0'} 2023-09-22T21:49:42.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038bc81fc5d0285fb-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.314 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:49:42.315 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:49:42.315 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.315 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.315 DEBUG:boto:Method: HEAD 2023-09-22T21:49:42.315 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.315 DEBUG:boto:Data: 2023-09-22T21:49:42.315 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.315 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.315 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.315 DEBUG:boto:Params: {} 2023-09-22T21:49:42.316 DEBUG:boto:Token: None 2023-09-22T21:49:42.316 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.316 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DebHjFRE7v20KgRsq6W+xhp4k+4= 2023-09-22T21:49:42.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DebHjFRE7v20KgRsq6W+xhp4k+4=', 'Content-Length': '0'} 2023-09-22T21:49:42.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef9c503e4042b3d5-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.319 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:49:42.319 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:49:42.319 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.319 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.319 DEBUG:boto:Method: HEAD 2023-09-22T21:49:42.319 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.320 DEBUG:boto:Data: 2023-09-22T21:49:42.320 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.320 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.320 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.320 DEBUG:boto:Params: {} 2023-09-22T21:49:42.320 DEBUG:boto:Token: None 2023-09-22T21:49:42.320 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.320 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DebHjFRE7v20KgRsq6W+xhp4k+4= 2023-09-22T21:49:42.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DebHjFRE7v20KgRsq6W+xhp4k+4=', 'Content-Length': '0'} 2023-09-22T21:49:42.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bcc6c11a465d7f6-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-22T21:49:42.324 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:49:42.324 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:49:42.324 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.324 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.324 DEBUG:boto:Method: GET 2023-09-22T21:49:42.324 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.324 DEBUG:boto:Data: 2023-09-22T21:49:42.324 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.324 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.324 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.325 DEBUG:boto:Params: {} 2023-09-22T21:49:42.325 DEBUG:boto:Token: None 2023-09-22T21:49:42.325 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.325 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lJ5uDywGzehgEcJC1Rj9FtYlIUA= 2023-09-22T21:49:42.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lJ5uDywGzehgEcJC1Rj9FtYlIUA=', 'Content-Length': '0'} 2023-09-22T21:49:42.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e41bd3d0ad72a7e2-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.370 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:49:42.370 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:49:42.371 DEBUG:boto:path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.371 DEBUG:boto:auth_path=/ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.371 DEBUG:boto:Method: GET 2023-09-22T21:49:42.371 DEBUG:boto:Path: /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.371 DEBUG:boto:Data: 2023-09-22T21:49:42.371 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.371 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.371 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.371 DEBUG:boto:Params: {} 2023-09-22T21:49:42.371 DEBUG:boto:Token: None 2023-09-22T21:49:42.372 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj69?versionId=null 2023-09-22T21:49:42.372 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lJ5uDywGzehgEcJC1Rj9FtYlIUA= 2023-09-22T21:49:42.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lJ5uDywGzehgEcJC1Rj9FtYlIUA=', 'Content-Length': '0'} 2023-09-22T21:49:42.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000054880a6a0caaab2-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.375 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:42.375 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:42.375 DEBUG:boto:path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:49:42.375 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:49:42.375 DEBUG:boto:Method: GET 2023-09-22T21:49:42.376 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj69 2023-09-22T21:49:42.376 DEBUG:boto:Data: 2023-09-22T21:49:42.376 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.376 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.376 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.376 DEBUG:boto:Params: {} 2023-09-22T21:49:42.376 DEBUG:boto:Token: None 2023-09-22T21:49:42.376 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/ 2023-09-22T21:49:42.376 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE= 2023-09-22T21:49:42.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE=', 'Content-Length': '0'} 2023-09-22T21:49:42.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb7d25903c69a15e-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.382 DEBUG:boto:b'ffzdqu-59obj691000falseobj692023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:42.383 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:42.383 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:42.383 DEBUG:boto:path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:49:42.383 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:49:42.383 DEBUG:boto:Method: GET 2023-09-22T21:49:42.383 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj69 2023-09-22T21:49:42.383 DEBUG:boto:Data: 2023-09-22T21:49:42.383 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.383 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.383 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.383 DEBUG:boto:Params: {} 2023-09-22T21:49:42.384 DEBUG:boto:Token: None 2023-09-22T21:49:42.384 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/ 2023-09-22T21:49:42.384 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE= 2023-09-22T21:49:42.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE=', 'Content-Length': '0'} 2023-09-22T21:49:42.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000911f3e75f000d082-00650e0bf6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.394 DEBUG:boto:b'ffzdqu-59obj691000falseobj692023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:42.394 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:49:42.394 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:49:42.395 DEBUG:boto:Method: HEAD 2023-09-22T21:49:42.395 DEBUG:boto:Path: /ffzdqu-59/obj69 2023-09-22T21:49:42.395 DEBUG:boto:Data: 2023-09-22T21:49:42.395 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.395 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.395 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.395 DEBUG:boto:Params: {} 2023-09-22T21:49:42.395 DEBUG:boto:Token: None 2023-09-22T21:49:42.395 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj69 2023-09-22T21:49:42.395 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZJbLpkk6TMybyMVWNDiCrzV03V0= 2023-09-22T21:49:42.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZJbLpkk6TMybyMVWNDiCrzV03V0=', 'Content-Length': '0'} 2023-09-22T21:49:42.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032704dd30c4a2f06-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.398 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:49:42.399 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:49:42.399 DEBUG:boto:Method: HEAD 2023-09-22T21:49:42.399 DEBUG:boto:Path: /ffzdqu-59/obj69 2023-09-22T21:49:42.399 DEBUG:boto:Data: 2023-09-22T21:49:42.399 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.399 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.399 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.399 DEBUG:boto:Params: {} 2023-09-22T21:49:42.399 DEBUG:boto:Token: None 2023-09-22T21:49:42.400 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj69 2023-09-22T21:49:42.400 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZJbLpkk6TMybyMVWNDiCrzV03V0= 2023-09-22T21:49:42.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZJbLpkk6TMybyMVWNDiCrzV03V0=', 'Content-Length': '0'} 2023-09-22T21:49:42.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007763b31307763f32-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-22T21:49:42.403 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:49:42.403 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:49:42.403 DEBUG:boto:Method: GET 2023-09-22T21:49:42.403 DEBUG:boto:Path: /ffzdqu-59/obj69 2023-09-22T21:49:42.403 DEBUG:boto:Data: 2023-09-22T21:49:42.403 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.403 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.403 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.404 DEBUG:boto:Params: {} 2023-09-22T21:49:42.404 DEBUG:boto:Token: None 2023-09-22T21:49:42.404 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj69 2023-09-22T21:49:42.404 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Xk+hnCdG3UqL3FzCE85JyVUlt/0= 2023-09-22T21:49:42.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Xk+hnCdG3UqL3FzCE85JyVUlt/0=', 'Content-Length': '0'} 2023-09-22T21:49:42.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a399c54a7a6adc09-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.407 DEBUG:boto:path=/ffzdqu-59/obj69 2023-09-22T21:49:42.407 DEBUG:boto:auth_path=/ffzdqu-59/obj69 2023-09-22T21:49:42.407 DEBUG:boto:Method: GET 2023-09-22T21:49:42.407 DEBUG:boto:Path: /ffzdqu-59/obj69 2023-09-22T21:49:42.408 DEBUG:boto:Data: 2023-09-22T21:49:42.408 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.408 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.408 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.408 DEBUG:boto:Params: {} 2023-09-22T21:49:42.408 DEBUG:boto:Token: None 2023-09-22T21:49:42.408 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj69 2023-09-22T21:49:42.408 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Xk+hnCdG3UqL3FzCE85JyVUlt/0= 2023-09-22T21:49:42.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Xk+hnCdG3UqL3FzCE85JyVUlt/0=', 'Content-Length': '0'} 2023-09-22T21:49:42.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad1700be4e0a714d-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.454 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:42.455 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:42.455 DEBUG:boto:path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:49:42.455 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:49:42.455 DEBUG:boto:Method: GET 2023-09-22T21:49:42.455 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj69 2023-09-22T21:49:42.455 DEBUG:boto:Data: 2023-09-22T21:49:42.455 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.455 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.455 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.455 DEBUG:boto:Params: {} 2023-09-22T21:49:42.456 DEBUG:boto:Token: None 2023-09-22T21:49:42.456 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/ 2023-09-22T21:49:42.456 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE= 2023-09-22T21:49:42.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE=', 'Content-Length': '0'} 2023-09-22T21:49:42.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f4ad3e1560f0c06-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.461 DEBUG:boto:b'ffzdqu-59obj691000falseobj692023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:42.461 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:42.461 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:42.462 DEBUG:boto:path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:49:42.462 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj69 2023-09-22T21:49:42.462 DEBUG:boto:Method: GET 2023-09-22T21:49:42.462 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj69 2023-09-22T21:49:42.462 DEBUG:boto:Data: 2023-09-22T21:49:42.462 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.462 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.462 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.462 DEBUG:boto:Params: {} 2023-09-22T21:49:42.463 DEBUG:boto:Token: None 2023-09-22T21:49:42.463 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/ 2023-09-22T21:49:42.463 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE= 2023-09-22T21:49:42.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE=', 'Content-Length': '0'} 2023-09-22T21:49:42.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098f4d2997e8bd54e-00650e0bf6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.473 DEBUG:boto:b'ffzdqu-59obj691000falseobj692023-09-22T21:40:21.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:42.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-09-22T21:49:42.473 DEBUG:boto:path=/ffzdqu-59/obj74 2023-09-22T21:49:42.474 DEBUG:boto:auth_path=/ffzdqu-59/obj74 2023-09-22T21:49:42.474 DEBUG:boto:path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.474 DEBUG:boto:auth_path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.474 DEBUG:boto:Method: GET 2023-09-22T21:49:42.474 DEBUG:boto:Path: /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.474 DEBUG:boto:Data: 2023-09-22T21:49:42.474 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.474 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.474 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.475 DEBUG:boto:Params: {} 2023-09-22T21:49:42.475 DEBUG:boto:Token: None 2023-09-22T21:49:42.475 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.475 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Tm+71kahqgR8MlgcQPpih7PH8ZU= 2023-09-22T21:49:42.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Tm+71kahqgR8MlgcQPpih7PH8ZU=', 'Content-Length': '0'} 2023-09-22T21:49:42.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007301fe7c43cf03b3-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.522 DEBUG:boto:path=/ffzdqu-59/obj74 2023-09-22T21:49:42.522 DEBUG:boto:auth_path=/ffzdqu-59/obj74 2023-09-22T21:49:42.523 DEBUG:boto:path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.523 DEBUG:boto:auth_path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.523 DEBUG:boto:Method: GET 2023-09-22T21:49:42.523 DEBUG:boto:Path: /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.523 DEBUG:boto:Data: 2023-09-22T21:49:42.523 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.523 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.523 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.523 DEBUG:boto:Params: {} 2023-09-22T21:49:42.523 DEBUG:boto:Token: None 2023-09-22T21:49:42.524 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.524 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Tm+71kahqgR8MlgcQPpih7PH8ZU= 2023-09-22T21:49:42.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Tm+71kahqgR8MlgcQPpih7PH8ZU=', 'Content-Length': '0'} 2023-09-22T21:49:42.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af00ad0feec3f752-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.527 DEBUG:boto:path=/ffzdqu-59/obj74 2023-09-22T21:49:42.527 DEBUG:boto:auth_path=/ffzdqu-59/obj74 2023-09-22T21:49:42.527 DEBUG:boto:path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.527 DEBUG:boto:auth_path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.527 DEBUG:boto:Method: HEAD 2023-09-22T21:49:42.527 DEBUG:boto:Path: /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.527 DEBUG:boto:Data: 2023-09-22T21:49:42.528 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.528 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.528 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.528 DEBUG:boto:Params: {} 2023-09-22T21:49:42.528 DEBUG:boto:Token: None 2023-09-22T21:49:42.528 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.528 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h31Ed4pbksHjUVQ5OiJoy+myEXo= 2023-09-22T21:49:42.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h31Ed4pbksHjUVQ5OiJoy+myEXo=', 'Content-Length': '0'} 2023-09-22T21:49:42.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cdaf5e6032077ca-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.531 DEBUG:boto:path=/ffzdqu-59/obj74 2023-09-22T21:49:42.531 DEBUG:boto:auth_path=/ffzdqu-59/obj74 2023-09-22T21:49:42.531 DEBUG:boto:path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.531 DEBUG:boto:auth_path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.531 DEBUG:boto:Method: HEAD 2023-09-22T21:49:42.532 DEBUG:boto:Path: /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.532 DEBUG:boto:Data: 2023-09-22T21:49:42.532 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.532 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.532 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.532 DEBUG:boto:Params: {} 2023-09-22T21:49:42.532 DEBUG:boto:Token: None 2023-09-22T21:49:42.532 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.532 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h31Ed4pbksHjUVQ5OiJoy+myEXo= 2023-09-22T21:49:42.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h31Ed4pbksHjUVQ5OiJoy+myEXo=', 'Content-Length': '0'} 2023-09-22T21:49:42.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ff464654c076574-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-09-22T21:49:42.536 DEBUG:boto:path=/ffzdqu-59/obj74 2023-09-22T21:49:42.536 DEBUG:boto:auth_path=/ffzdqu-59/obj74 2023-09-22T21:49:42.536 DEBUG:boto:path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.537 DEBUG:boto:auth_path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.537 DEBUG:boto:Method: GET 2023-09-22T21:49:42.537 DEBUG:boto:Path: /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.537 DEBUG:boto:Data: 2023-09-22T21:49:42.537 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.537 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.537 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.537 DEBUG:boto:Params: {} 2023-09-22T21:49:42.537 DEBUG:boto:Token: None 2023-09-22T21:49:42.537 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.538 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Tm+71kahqgR8MlgcQPpih7PH8ZU= 2023-09-22T21:49:42.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Tm+71kahqgR8MlgcQPpih7PH8ZU=', 'Content-Length': '0'} 2023-09-22T21:49:42.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c17587cd10952a9-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.541 DEBUG:boto:path=/ffzdqu-59/obj74 2023-09-22T21:49:42.541 DEBUG:boto:auth_path=/ffzdqu-59/obj74 2023-09-22T21:49:42.541 DEBUG:boto:path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.541 DEBUG:boto:auth_path=/ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.541 DEBUG:boto:Method: GET 2023-09-22T21:49:42.541 DEBUG:boto:Path: /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.541 DEBUG:boto:Data: 2023-09-22T21:49:42.541 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.541 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.541 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.542 DEBUG:boto:Params: {} 2023-09-22T21:49:42.542 DEBUG:boto:Token: None 2023-09-22T21:49:42.542 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj74?versionId=null 2023-09-22T21:49:42.542 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Tm+71kahqgR8MlgcQPpih7PH8ZU= 2023-09-22T21:49:42.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Tm+71kahqgR8MlgcQPpih7PH8ZU=', 'Content-Length': '0'} 2023-09-22T21:49:42.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df827528ddad4e9a-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.586 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:42.587 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:42.587 DEBUG:boto:path=/ffzdqu-59/?prefix=obj74 2023-09-22T21:49:42.587 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj74 2023-09-22T21:49:42.587 DEBUG:boto:Method: GET 2023-09-22T21:49:42.587 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj74 2023-09-22T21:49:42.587 DEBUG:boto:Data: 2023-09-22T21:49:42.587 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.587 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.587 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.587 DEBUG:boto:Params: {} 2023-09-22T21:49:42.587 DEBUG:boto:Token: None 2023-09-22T21:49:42.588 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/ 2023-09-22T21:49:42.588 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE= 2023-09-22T21:49:42.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE=', 'Content-Length': '0'} 2023-09-22T21:49:42.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005be8db55398d8c16-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.593 DEBUG:boto:b'ffzdqu-59obj741000falseobj742023-09-22T21:47:16.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:42.594 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:42.594 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:42.594 DEBUG:boto:path=/ffzdqu-59/?prefix=obj74 2023-09-22T21:49:42.594 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj74 2023-09-22T21:49:42.594 DEBUG:boto:Method: GET 2023-09-22T21:49:42.595 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj74 2023-09-22T21:49:42.595 DEBUG:boto:Data: 2023-09-22T21:49:42.595 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.595 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.595 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.595 DEBUG:boto:Params: {} 2023-09-22T21:49:42.595 DEBUG:boto:Token: None 2023-09-22T21:49:42.596 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/ 2023-09-22T21:49:42.596 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE= 2023-09-22T21:49:42.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE=', 'Content-Length': '0'} 2023-09-22T21:49:42.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029528f79314911a8-00650e0bf6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.607 DEBUG:boto:b'ffzdqu-59obj741000falseobj742023-09-22T21:47:16.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:42.608 DEBUG:boto:path=/ffzdqu-59/obj74 2023-09-22T21:49:42.608 DEBUG:boto:auth_path=/ffzdqu-59/obj74 2023-09-22T21:49:42.608 DEBUG:boto:Method: HEAD 2023-09-22T21:49:42.608 DEBUG:boto:Path: /ffzdqu-59/obj74 2023-09-22T21:49:42.608 DEBUG:boto:Data: 2023-09-22T21:49:42.608 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.608 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.609 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.609 DEBUG:boto:Params: {} 2023-09-22T21:49:42.609 DEBUG:boto:Token: None 2023-09-22T21:49:42.609 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj74 2023-09-22T21:49:42.609 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NuIyiQ2qQDfMOgas/yA4hk0UnL0= 2023-09-22T21:49:42.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NuIyiQ2qQDfMOgas/yA4hk0UnL0=', 'Content-Length': '0'} 2023-09-22T21:49:42.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbcd0c70d6a6cd33-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.612 DEBUG:boto:path=/ffzdqu-59/obj74 2023-09-22T21:49:42.612 DEBUG:boto:auth_path=/ffzdqu-59/obj74 2023-09-22T21:49:42.612 DEBUG:boto:Method: HEAD 2023-09-22T21:49:42.612 DEBUG:boto:Path: /ffzdqu-59/obj74 2023-09-22T21:49:42.613 DEBUG:boto:Data: 2023-09-22T21:49:42.613 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.613 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.613 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.613 DEBUG:boto:Params: {} 2023-09-22T21:49:42.613 DEBUG:boto:Token: None 2023-09-22T21:49:42.613 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj74 2023-09-22T21:49:42.613 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NuIyiQ2qQDfMOgas/yA4hk0UnL0= 2023-09-22T21:49:42.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NuIyiQ2qQDfMOgas/yA4hk0UnL0=', 'Content-Length': '0'} 2023-09-22T21:49:42.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a14d3c64263d538c-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-09-22T21:49:42.617 DEBUG:boto:path=/ffzdqu-59/obj74 2023-09-22T21:49:42.617 DEBUG:boto:auth_path=/ffzdqu-59/obj74 2023-09-22T21:49:42.617 DEBUG:boto:Method: GET 2023-09-22T21:49:42.617 DEBUG:boto:Path: /ffzdqu-59/obj74 2023-09-22T21:49:42.617 DEBUG:boto:Data: 2023-09-22T21:49:42.618 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.618 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.618 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.618 DEBUG:boto:Params: {} 2023-09-22T21:49:42.618 DEBUG:boto:Token: None 2023-09-22T21:49:42.618 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj74 2023-09-22T21:49:42.618 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WI9UGrLio8SvqvRgh57KfSdhogc= 2023-09-22T21:49:42.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WI9UGrLio8SvqvRgh57KfSdhogc=', 'Content-Length': '0'} 2023-09-22T21:49:42.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe308f5dd36a620a-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.662 DEBUG:boto:path=/ffzdqu-59/obj74 2023-09-22T21:49:42.663 DEBUG:boto:auth_path=/ffzdqu-59/obj74 2023-09-22T21:49:42.663 DEBUG:boto:Method: GET 2023-09-22T21:49:42.663 DEBUG:boto:Path: /ffzdqu-59/obj74 2023-09-22T21:49:42.663 DEBUG:boto:Data: 2023-09-22T21:49:42.663 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.663 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.663 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.663 DEBUG:boto:Params: {} 2023-09-22T21:49:42.663 DEBUG:boto:Token: None 2023-09-22T21:49:42.664 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj74 2023-09-22T21:49:42.664 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WI9UGrLio8SvqvRgh57KfSdhogc= 2023-09-22T21:49:42.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WI9UGrLio8SvqvRgh57KfSdhogc=', 'Content-Length': '0'} 2023-09-22T21:49:42.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:47:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ce3ab261a65f4fa-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.667 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:42.667 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:42.668 DEBUG:boto:path=/ffzdqu-59/?prefix=obj74 2023-09-22T21:49:42.668 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj74 2023-09-22T21:49:42.668 DEBUG:boto:Method: GET 2023-09-22T21:49:42.668 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj74 2023-09-22T21:49:42.668 DEBUG:boto:Data: 2023-09-22T21:49:42.668 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.668 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.668 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.668 DEBUG:boto:Params: {} 2023-09-22T21:49:42.668 DEBUG:boto:Token: None 2023-09-22T21:49:42.669 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/ 2023-09-22T21:49:42.669 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE= 2023-09-22T21:49:42.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE=', 'Content-Length': '0'} 2023-09-22T21:49:42.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000693bd29f881434ca-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.673 DEBUG:boto:b'ffzdqu-59obj741000falseobj742023-09-22T21:47:16.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:42.674 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:42.674 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:42.674 DEBUG:boto:path=/ffzdqu-59/?prefix=obj74 2023-09-22T21:49:42.674 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj74 2023-09-22T21:49:42.674 DEBUG:boto:Method: GET 2023-09-22T21:49:42.674 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj74 2023-09-22T21:49:42.675 DEBUG:boto:Data: 2023-09-22T21:49:42.675 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.675 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.675 DEBUG:boto:Params: {} 2023-09-22T21:49:42.675 DEBUG:boto:Token: None 2023-09-22T21:49:42.675 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/ 2023-09-22T21:49:42.675 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE= 2023-09-22T21:49:42.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE=', 'Content-Length': '0'} 2023-09-22T21:49:42.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9bcfb1f25252570-00650e0bf6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.726 DEBUG:boto:b'ffzdqu-59obj741000falseobj742023-09-22T21:47:16.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:42.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T21:49:42.727 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:49:42.727 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:49:42.727 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.727 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.727 DEBUG:boto:Method: GET 2023-09-22T21:49:42.727 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.727 DEBUG:boto:Data: 2023-09-22T21:49:42.728 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.728 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.728 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.728 DEBUG:boto:Params: {} 2023-09-22T21:49:42.728 DEBUG:boto:Token: None 2023-09-22T21:49:42.728 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.728 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xAB+Sy+nAKlLaVHX0NfhOW8YfWc= 2023-09-22T21:49:42.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xAB+Sy+nAKlLaVHX0NfhOW8YfWc=', 'Content-Length': '0'} 2023-09-22T21:49:42.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070671e17c236b67a-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.732 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:49:42.732 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:49:42.732 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.732 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.732 DEBUG:boto:Method: GET 2023-09-22T21:49:42.732 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.732 DEBUG:boto:Data: 2023-09-22T21:49:42.733 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.733 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.733 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.733 DEBUG:boto:Params: {} 2023-09-22T21:49:42.733 DEBUG:boto:Token: None 2023-09-22T21:49:42.733 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.733 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xAB+Sy+nAKlLaVHX0NfhOW8YfWc= 2023-09-22T21:49:42.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xAB+Sy+nAKlLaVHX0NfhOW8YfWc=', 'Content-Length': '0'} 2023-09-22T21:49:42.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee696cd35d2d7652-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.736 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:49:42.736 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:49:42.737 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.737 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.737 DEBUG:boto:Method: HEAD 2023-09-22T21:49:42.737 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.737 DEBUG:boto:Data: 2023-09-22T21:49:42.737 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.737 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.737 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.737 DEBUG:boto:Params: {} 2023-09-22T21:49:42.737 DEBUG:boto:Token: None 2023-09-22T21:49:42.738 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.738 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XaY14VBc8rOlMMX7Boravaw8tUk= 2023-09-22T21:49:42.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XaY14VBc8rOlMMX7Boravaw8tUk=', 'Content-Length': '0'} 2023-09-22T21:49:42.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026d7d3492b82193d-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.741 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:49:42.741 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:49:42.741 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.741 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.741 DEBUG:boto:Method: HEAD 2023-09-22T21:49:42.741 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.741 DEBUG:boto:Data: 2023-09-22T21:49:42.742 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.742 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.742 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.742 DEBUG:boto:Params: {} 2023-09-22T21:49:42.742 DEBUG:boto:Token: None 2023-09-22T21:49:42.742 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.742 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XaY14VBc8rOlMMX7Boravaw8tUk= 2023-09-22T21:49:42.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XaY14VBc8rOlMMX7Boravaw8tUk=', 'Content-Length': '0'} 2023-09-22T21:49:42.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ffb74adeb662d58-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T21:49:42.746 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:49:42.746 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:49:42.746 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.746 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.746 DEBUG:boto:Method: GET 2023-09-22T21:49:42.746 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.746 DEBUG:boto:Data: 2023-09-22T21:49:42.746 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.747 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.747 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.747 DEBUG:boto:Params: {} 2023-09-22T21:49:42.747 DEBUG:boto:Token: None 2023-09-22T21:49:42.747 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.747 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xAB+Sy+nAKlLaVHX0NfhOW8YfWc= 2023-09-22T21:49:42.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xAB+Sy+nAKlLaVHX0NfhOW8YfWc=', 'Content-Length': '0'} 2023-09-22T21:49:42.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b35fcec5e086f03-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.750 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:49:42.750 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:49:42.750 DEBUG:boto:path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.750 DEBUG:boto:auth_path=/ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.751 DEBUG:boto:Method: GET 2023-09-22T21:49:42.751 DEBUG:boto:Path: /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.751 DEBUG:boto:Data: 2023-09-22T21:49:42.751 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.751 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.751 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.751 DEBUG:boto:Params: {} 2023-09-22T21:49:42.751 DEBUG:boto:Token: None 2023-09-22T21:49:42.751 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj76?versionId=null 2023-09-22T21:49:42.751 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xAB+Sy+nAKlLaVHX0NfhOW8YfWc= 2023-09-22T21:49:42.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xAB+Sy+nAKlLaVHX0NfhOW8YfWc=', 'Content-Length': '0'} 2023-09-22T21:49:42.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f7472f00e7b6178-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.754 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:42.755 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:42.755 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:49:42.755 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:49:42.755 DEBUG:boto:Method: GET 2023-09-22T21:49:42.755 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:49:42.755 DEBUG:boto:Data: 2023-09-22T21:49:42.755 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.755 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.755 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.755 DEBUG:boto:Params: {} 2023-09-22T21:49:42.756 DEBUG:boto:Token: None 2023-09-22T21:49:42.756 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/ 2023-09-22T21:49:42.756 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE= 2023-09-22T21:49:42.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE=', 'Content-Length': '0'} 2023-09-22T21:49:42.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb6554360564c704-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.761 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:42.761 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:42.762 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:42.762 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:49:42.762 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:49:42.762 DEBUG:boto:Method: GET 2023-09-22T21:49:42.762 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:49:42.762 DEBUG:boto:Data: 2023-09-22T21:49:42.762 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.762 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.762 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.763 DEBUG:boto:Params: {} 2023-09-22T21:49:42.763 DEBUG:boto:Token: None 2023-09-22T21:49:42.763 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/ 2023-09-22T21:49:42.763 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE= 2023-09-22T21:49:42.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE=', 'Content-Length': '0'} 2023-09-22T21:49:42.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005eb6e83179b1da8e-00650e0bf6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.814 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:42.815 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:49:42.815 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:49:42.815 DEBUG:boto:Method: HEAD 2023-09-22T21:49:42.815 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:49:42.815 DEBUG:boto:Data: 2023-09-22T21:49:42.815 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.815 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.815 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.816 DEBUG:boto:Params: {} 2023-09-22T21:49:42.816 DEBUG:boto:Token: None 2023-09-22T21:49:42.816 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj76 2023-09-22T21:49:42.816 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tlKDj2RzmKkK1RWUtk3I7p0VJbw= 2023-09-22T21:49:42.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tlKDj2RzmKkK1RWUtk3I7p0VJbw=', 'Content-Length': '0'} 2023-09-22T21:49:42.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000868af1317d98625b-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.819 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:49:42.819 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:49:42.819 DEBUG:boto:Method: HEAD 2023-09-22T21:49:42.820 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:49:42.820 DEBUG:boto:Data: 2023-09-22T21:49:42.820 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.820 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.820 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.820 DEBUG:boto:Params: {} 2023-09-22T21:49:42.820 DEBUG:boto:Token: None 2023-09-22T21:49:42.820 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj76 2023-09-22T21:49:42.821 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tlKDj2RzmKkK1RWUtk3I7p0VJbw= 2023-09-22T21:49:42.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tlKDj2RzmKkK1RWUtk3I7p0VJbw=', 'Content-Length': '0'} 2023-09-22T21:49:42.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc493883e518b07a-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T21:49:42.825 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:49:42.825 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:49:42.825 DEBUG:boto:Method: GET 2023-09-22T21:49:42.825 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:49:42.826 DEBUG:boto:Data: 2023-09-22T21:49:42.826 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.826 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.826 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.826 DEBUG:boto:Params: {} 2023-09-22T21:49:42.826 DEBUG:boto:Token: None 2023-09-22T21:49:42.826 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj76 2023-09-22T21:49:42.826 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mMyDoTwUvOqW1nnwe2EIGz5yhuA= 2023-09-22T21:49:42.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mMyDoTwUvOqW1nnwe2EIGz5yhuA=', 'Content-Length': '0'} 2023-09-22T21:49:42.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8510c0ba09b1e41-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.874 DEBUG:boto:path=/ffzdqu-59/obj76 2023-09-22T21:49:42.875 DEBUG:boto:auth_path=/ffzdqu-59/obj76 2023-09-22T21:49:42.875 DEBUG:boto:Method: GET 2023-09-22T21:49:42.875 DEBUG:boto:Path: /ffzdqu-59/obj76 2023-09-22T21:49:42.875 DEBUG:boto:Data: 2023-09-22T21:49:42.875 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.875 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.875 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.875 DEBUG:boto:Params: {} 2023-09-22T21:49:42.875 DEBUG:boto:Token: None 2023-09-22T21:49:42.876 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj76 2023-09-22T21:49:42.876 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mMyDoTwUvOqW1nnwe2EIGz5yhuA= 2023-09-22T21:49:42.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mMyDoTwUvOqW1nnwe2EIGz5yhuA=', 'Content-Length': '0'} 2023-09-22T21:49:42.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c2dec4fa5339dae-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.880 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:42.881 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:42.881 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:49:42.881 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:49:42.881 DEBUG:boto:Method: GET 2023-09-22T21:49:42.881 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:49:42.881 DEBUG:boto:Data: 2023-09-22T21:49:42.881 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.881 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.881 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.881 DEBUG:boto:Params: {} 2023-09-22T21:49:42.882 DEBUG:boto:Token: None 2023-09-22T21:49:42.882 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/ 2023-09-22T21:49:42.882 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE= 2023-09-22T21:49:42.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE=', 'Content-Length': '0'} 2023-09-22T21:49:42.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fa58956bb235563-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.890 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:42.891 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:42.891 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:42.891 DEBUG:boto:path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:49:42.891 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj76 2023-09-22T21:49:42.891 DEBUG:boto:Method: GET 2023-09-22T21:49:42.891 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj76 2023-09-22T21:49:42.891 DEBUG:boto:Data: 2023-09-22T21:49:42.892 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.892 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.892 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.892 DEBUG:boto:Params: {} 2023-09-22T21:49:42.892 DEBUG:boto:Token: None 2023-09-22T21:49:42.892 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/ 2023-09-22T21:49:42.892 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE= 2023-09-22T21:49:42.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zSMJO2++v7cOw9OJ5n+9FDEAbRE=', 'Content-Length': '0'} 2023-09-22T21:49:42.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000544128b2b2aab9c3-00650e0bf6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.903 DEBUG:boto:b'ffzdqu-59obj761000falseobj762023-09-22T21:38:31.739Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:42.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:49:42.903 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:49:42.903 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:49:42.903 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:42.903 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:42.904 DEBUG:boto:Method: GET 2023-09-22T21:49:42.904 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:42.904 DEBUG:boto:Data: 2023-09-22T21:49:42.904 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.904 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.904 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.904 DEBUG:boto:Params: {} 2023-09-22T21:49:42.904 DEBUG:boto:Token: None 2023-09-22T21:49:42.904 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:42.905 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rNcygszBKWa9XYU6Vpfj4mqA4vc= 2023-09-22T21:49:42.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rNcygszBKWa9XYU6Vpfj4mqA4vc=', 'Content-Length': '0'} 2023-09-22T21:49:42.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c250f5af99232b09-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.908 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:49:42.908 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:49:42.908 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:42.908 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:42.908 DEBUG:boto:Method: GET 2023-09-22T21:49:42.908 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:42.908 DEBUG:boto:Data: 2023-09-22T21:49:42.909 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.909 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.909 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.909 DEBUG:boto:Params: {} 2023-09-22T21:49:42.909 DEBUG:boto:Token: None 2023-09-22T21:49:42.909 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:42.909 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rNcygszBKWa9XYU6Vpfj4mqA4vc= 2023-09-22T21:49:42.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rNcygszBKWa9XYU6Vpfj4mqA4vc=', 'Content-Length': '0'} 2023-09-22T21:49:42.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bcded55a9fa5d80-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.954 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:49:42.954 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:49:42.955 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:42.955 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:42.955 DEBUG:boto:Method: HEAD 2023-09-22T21:49:42.955 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:42.955 DEBUG:boto:Data: 2023-09-22T21:49:42.955 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.955 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.955 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.955 DEBUG:boto:Params: {} 2023-09-22T21:49:42.956 DEBUG:boto:Token: None 2023-09-22T21:49:42.956 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:42.956 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f9M7JVJtOKUChFWtFgnbSCyBUMY= 2023-09-22T21:49:42.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f9M7JVJtOKUChFWtFgnbSCyBUMY=', 'Content-Length': '0'} 2023-09-22T21:49:42.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059552664e5442a6c-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.959 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:49:42.959 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:49:42.959 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:42.960 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:42.960 DEBUG:boto:Method: HEAD 2023-09-22T21:49:42.960 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:42.960 DEBUG:boto:Data: 2023-09-22T21:49:42.960 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.960 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:42.960 DEBUG:boto:Port: 8000 2023-09-22T21:49:42.960 DEBUG:boto:Params: {} 2023-09-22T21:49:42.960 DEBUG:boto:Token: None 2023-09-22T21:49:42.961 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:42.961 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f9M7JVJtOKUChFWtFgnbSCyBUMY= 2023-09-22T21:49:42.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f9M7JVJtOKUChFWtFgnbSCyBUMY=', 'Content-Length': '0'} 2023-09-22T21:49:42.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3436b0531433319-00650e0bf6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:42.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:49:42.964 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:49:42.964 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:49:42.964 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:42.964 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:42.964 DEBUG:boto:Method: GET 2023-09-22T21:49:42.964 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:42.964 DEBUG:boto:Data: 2023-09-22T21:49:42.965 DEBUG:boto:Headers: {} 2023-09-22T21:49:42.965 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:42.965 DEBUG:boto:Port: 8001 2023-09-22T21:49:42.965 DEBUG:boto:Params: {} 2023-09-22T21:49:42.965 DEBUG:boto:Token: None 2023-09-22T21:49:42.965 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:42 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:42.965 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rNcygszBKWa9XYU6Vpfj4mqA4vc= 2023-09-22T21:49:42.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rNcygszBKWa9XYU6Vpfj4mqA4vc=', 'Content-Length': '0'} 2023-09-22T21:49:42.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc6080dbd5d8a202-00650e0bf6-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.010 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:49:43.010 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:49:43.010 DEBUG:boto:path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:43.011 DEBUG:boto:auth_path=/ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:43.011 DEBUG:boto:Method: GET 2023-09-22T21:49:43.011 DEBUG:boto:Path: /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:43.011 DEBUG:boto:Data: 2023-09-22T21:49:43.011 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.011 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.011 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.011 DEBUG:boto:Params: {} 2023-09-22T21:49:43.011 DEBUG:boto:Token: None 2023-09-22T21:49:43.011 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj85?versionId=null 2023-09-22T21:49:43.012 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hvt9YzbCwcqMXHp20eU7J9EeEBU= 2023-09-22T21:49:43.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hvt9YzbCwcqMXHp20eU7J9EeEBU=', 'Content-Length': '0'} 2023-09-22T21:49:43.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000728a3a6657d912dc-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.015 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:43.015 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:43.015 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:49:43.015 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:49:43.015 DEBUG:boto:Method: GET 2023-09-22T21:49:43.015 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:49:43.016 DEBUG:boto:Data: 2023-09-22T21:49:43.016 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.016 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.016 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.016 DEBUG:boto:Params: {} 2023-09-22T21:49:43.016 DEBUG:boto:Token: None 2023-09-22T21:49:43.016 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/ 2023-09-22T21:49:43.016 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw= 2023-09-22T21:49:43.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw=', 'Content-Length': '0'} 2023-09-22T21:49:43.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d39d5a16b0bc196-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.021 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:43.022 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:43.022 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:43.022 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:49:43.022 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:49:43.022 DEBUG:boto:Method: GET 2023-09-22T21:49:43.022 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:49:43.022 DEBUG:boto:Data: 2023-09-22T21:49:43.022 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.022 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.023 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.023 DEBUG:boto:Params: {} 2023-09-22T21:49:43.023 DEBUG:boto:Token: None 2023-09-22T21:49:43.023 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/ 2023-09-22T21:49:43.023 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw= 2023-09-22T21:49:43.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw=', 'Content-Length': '0'} 2023-09-22T21:49:43.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010076e99efbf189b-00650e0bf7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.032 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:43.033 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:49:43.033 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:49:43.033 DEBUG:boto:Method: HEAD 2023-09-22T21:49:43.033 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:49:43.033 DEBUG:boto:Data: 2023-09-22T21:49:43.033 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.033 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.034 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.034 DEBUG:boto:Params: {} 2023-09-22T21:49:43.034 DEBUG:boto:Token: None 2023-09-22T21:49:43.034 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj85 2023-09-22T21:49:43.034 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IApJNO9QQaF+uZWPEdlUxo6xXs8= 2023-09-22T21:49:43.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IApJNO9QQaF+uZWPEdlUxo6xXs8=', 'Content-Length': '0'} 2023-09-22T21:49:43.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0929aed4fabb795-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.037 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:49:43.037 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:49:43.037 DEBUG:boto:Method: HEAD 2023-09-22T21:49:43.038 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:49:43.038 DEBUG:boto:Data: 2023-09-22T21:49:43.038 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.038 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.038 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.038 DEBUG:boto:Params: {} 2023-09-22T21:49:43.038 DEBUG:boto:Token: None 2023-09-22T21:49:43.038 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj85 2023-09-22T21:49:43.038 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IApJNO9QQaF+uZWPEdlUxo6xXs8= 2023-09-22T21:49:43.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IApJNO9QQaF+uZWPEdlUxo6xXs8=', 'Content-Length': '0'} 2023-09-22T21:49:43.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ffdc16b3955650b-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:49:43.041 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:49:43.041 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:49:43.041 DEBUG:boto:Method: GET 2023-09-22T21:49:43.042 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:49:43.042 DEBUG:boto:Data: 2023-09-22T21:49:43.042 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.042 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.042 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.042 DEBUG:boto:Params: {} 2023-09-22T21:49:43.042 DEBUG:boto:Token: None 2023-09-22T21:49:43.042 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj85 2023-09-22T21:49:43.042 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CmkdYovtn7YmIBObqHT1r27GvZE= 2023-09-22T21:49:43.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CmkdYovtn7YmIBObqHT1r27GvZE=', 'Content-Length': '0'} 2023-09-22T21:49:43.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000893c7a1a1df8f1d3-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.046 DEBUG:boto:path=/ffzdqu-59/obj85 2023-09-22T21:49:43.046 DEBUG:boto:auth_path=/ffzdqu-59/obj85 2023-09-22T21:49:43.046 DEBUG:boto:Method: GET 2023-09-22T21:49:43.046 DEBUG:boto:Path: /ffzdqu-59/obj85 2023-09-22T21:49:43.046 DEBUG:boto:Data: 2023-09-22T21:49:43.046 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.046 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.046 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.046 DEBUG:boto:Params: {} 2023-09-22T21:49:43.047 DEBUG:boto:Token: None 2023-09-22T21:49:43.047 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj85 2023-09-22T21:49:43.047 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CmkdYovtn7YmIBObqHT1r27GvZE= 2023-09-22T21:49:43.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CmkdYovtn7YmIBObqHT1r27GvZE=', 'Content-Length': '0'} 2023-09-22T21:49:43.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:36:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac062c7f6290d8ff-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.090 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:43.091 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:43.091 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:49:43.091 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:49:43.091 DEBUG:boto:Method: GET 2023-09-22T21:49:43.091 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:49:43.091 DEBUG:boto:Data: 2023-09-22T21:49:43.091 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.091 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.091 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.092 DEBUG:boto:Params: {} 2023-09-22T21:49:43.092 DEBUG:boto:Token: None 2023-09-22T21:49:43.092 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/ 2023-09-22T21:49:43.092 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw= 2023-09-22T21:49:43.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw=', 'Content-Length': '0'} 2023-09-22T21:49:43.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000712fb6fba26f538a-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.097 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:43.097 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:43.097 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:43.097 DEBUG:boto:path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:49:43.097 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj85 2023-09-22T21:49:43.097 DEBUG:boto:Method: GET 2023-09-22T21:49:43.097 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj85 2023-09-22T21:49:43.098 DEBUG:boto:Data: 2023-09-22T21:49:43.098 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.098 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.098 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.098 DEBUG:boto:Params: {} 2023-09-22T21:49:43.098 DEBUG:boto:Token: None 2023-09-22T21:49:43.098 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/ 2023-09-22T21:49:43.098 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw= 2023-09-22T21:49:43.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw=', 'Content-Length': '0'} 2023-09-22T21:49:43.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083a4a7facf90107b-00650e0bf7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.108 DEBUG:boto:b'ffzdqu-59obj851000falseobj852023-09-22T21:36:48.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:43.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-22T21:49:43.109 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:49:43.109 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:49:43.109 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.109 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.109 DEBUG:boto:Method: GET 2023-09-22T21:49:43.109 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.109 DEBUG:boto:Data: 2023-09-22T21:49:43.109 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.109 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.109 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.110 DEBUG:boto:Params: {} 2023-09-22T21:49:43.110 DEBUG:boto:Token: None 2023-09-22T21:49:43.110 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.110 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2zCuW/oO6+nYTNRiwwSp4hTjimE= 2023-09-22T21:49:43.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2zCuW/oO6+nYTNRiwwSp4hTjimE=', 'Content-Length': '0'} 2023-09-22T21:49:43.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc8596f47ae02273-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.158 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:49:43.158 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:49:43.159 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.159 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.159 DEBUG:boto:Method: GET 2023-09-22T21:49:43.159 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.159 DEBUG:boto:Data: 2023-09-22T21:49:43.159 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.159 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.159 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.159 DEBUG:boto:Params: {} 2023-09-22T21:49:43.159 DEBUG:boto:Token: None 2023-09-22T21:49:43.160 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.160 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2zCuW/oO6+nYTNRiwwSp4hTjimE= 2023-09-22T21:49:43.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2zCuW/oO6+nYTNRiwwSp4hTjimE=', 'Content-Length': '0'} 2023-09-22T21:49:43.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b266379fe008cc3-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.163 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:49:43.163 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:49:43.163 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.163 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.163 DEBUG:boto:Method: HEAD 2023-09-22T21:49:43.164 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.164 DEBUG:boto:Data: 2023-09-22T21:49:43.164 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.164 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.164 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.164 DEBUG:boto:Params: {} 2023-09-22T21:49:43.164 DEBUG:boto:Token: None 2023-09-22T21:49:43.164 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.165 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:o8KAX25jqn4/89seexiRA1kMEfg= 2023-09-22T21:49:43.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:o8KAX25jqn4/89seexiRA1kMEfg=', 'Content-Length': '0'} 2023-09-22T21:49:43.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbf8bb365db67360-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.168 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:49:43.168 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:49:43.168 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.168 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.168 DEBUG:boto:Method: HEAD 2023-09-22T21:49:43.168 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.168 DEBUG:boto:Data: 2023-09-22T21:49:43.168 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.168 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.168 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.169 DEBUG:boto:Params: {} 2023-09-22T21:49:43.169 DEBUG:boto:Token: None 2023-09-22T21:49:43.169 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.169 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:o8KAX25jqn4/89seexiRA1kMEfg= 2023-09-22T21:49:43.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:o8KAX25jqn4/89seexiRA1kMEfg=', 'Content-Length': '0'} 2023-09-22T21:49:43.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054edde42457fa46f-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-22T21:49:43.172 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:49:43.172 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:49:43.172 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.172 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.173 DEBUG:boto:Method: GET 2023-09-22T21:49:43.173 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.173 DEBUG:boto:Data: 2023-09-22T21:49:43.173 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.173 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.173 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.173 DEBUG:boto:Params: {} 2023-09-22T21:49:43.173 DEBUG:boto:Token: None 2023-09-22T21:49:43.173 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.174 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2zCuW/oO6+nYTNRiwwSp4hTjimE= 2023-09-22T21:49:43.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2zCuW/oO6+nYTNRiwwSp4hTjimE=', 'Content-Length': '0'} 2023-09-22T21:49:43.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001733aee2276ece6c-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.176 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:49:43.176 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:49:43.177 DEBUG:boto:path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.177 DEBUG:boto:auth_path=/ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.177 DEBUG:boto:Method: GET 2023-09-22T21:49:43.177 DEBUG:boto:Path: /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.177 DEBUG:boto:Data: 2023-09-22T21:49:43.177 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.177 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.177 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.177 DEBUG:boto:Params: {} 2023-09-22T21:49:43.177 DEBUG:boto:Token: None 2023-09-22T21:49:43.178 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj87?versionId=null 2023-09-22T21:49:43.178 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2zCuW/oO6+nYTNRiwwSp4hTjimE= 2023-09-22T21:49:43.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2zCuW/oO6+nYTNRiwwSp4hTjimE=', 'Content-Length': '0'} 2023-09-22T21:49:43.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075f989ec4bb7a51e-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.222 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:43.223 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:43.223 DEBUG:boto:path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:49:43.223 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:49:43.223 DEBUG:boto:Method: GET 2023-09-22T21:49:43.223 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj87 2023-09-22T21:49:43.223 DEBUG:boto:Data: 2023-09-22T21:49:43.223 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.223 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.223 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.224 DEBUG:boto:Params: {} 2023-09-22T21:49:43.224 DEBUG:boto:Token: None 2023-09-22T21:49:43.224 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/ 2023-09-22T21:49:43.224 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw= 2023-09-22T21:49:43.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw=', 'Content-Length': '0'} 2023-09-22T21:49:43.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e636dff1a0790e95-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.229 DEBUG:boto:b'ffzdqu-59obj871000falseobj872023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:43.230 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:43.230 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:43.230 DEBUG:boto:path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:49:43.230 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:49:43.230 DEBUG:boto:Method: GET 2023-09-22T21:49:43.230 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj87 2023-09-22T21:49:43.230 DEBUG:boto:Data: 2023-09-22T21:49:43.230 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.230 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.231 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.231 DEBUG:boto:Params: {} 2023-09-22T21:49:43.231 DEBUG:boto:Token: None 2023-09-22T21:49:43.231 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/ 2023-09-22T21:49:43.231 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw= 2023-09-22T21:49:43.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw=', 'Content-Length': '0'} 2023-09-22T21:49:43.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7a1a2224ab34f05-00650e0bf7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.241 DEBUG:boto:b'ffzdqu-59obj871000falseobj872023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:43.241 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:49:43.241 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:49:43.241 DEBUG:boto:Method: HEAD 2023-09-22T21:49:43.241 DEBUG:boto:Path: /ffzdqu-59/obj87 2023-09-22T21:49:43.242 DEBUG:boto:Data: 2023-09-22T21:49:43.242 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.242 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.242 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.242 DEBUG:boto:Params: {} 2023-09-22T21:49:43.242 DEBUG:boto:Token: None 2023-09-22T21:49:43.242 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj87 2023-09-22T21:49:43.242 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iATBKKhtjV48j9mpEmB6YfADU58= 2023-09-22T21:49:43.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iATBKKhtjV48j9mpEmB6YfADU58=', 'Content-Length': '0'} 2023-09-22T21:49:43.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd7bb78b14b346f8-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.245 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:49:43.245 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:49:43.246 DEBUG:boto:Method: HEAD 2023-09-22T21:49:43.246 DEBUG:boto:Path: /ffzdqu-59/obj87 2023-09-22T21:49:43.246 DEBUG:boto:Data: 2023-09-22T21:49:43.246 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.246 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.246 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.246 DEBUG:boto:Params: {} 2023-09-22T21:49:43.246 DEBUG:boto:Token: None 2023-09-22T21:49:43.247 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj87 2023-09-22T21:49:43.247 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iATBKKhtjV48j9mpEmB6YfADU58= 2023-09-22T21:49:43.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iATBKKhtjV48j9mpEmB6YfADU58=', 'Content-Length': '0'} 2023-09-22T21:49:43.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cf5335e782ae63a-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-22T21:49:43.250 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:49:43.250 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:49:43.250 DEBUG:boto:Method: GET 2023-09-22T21:49:43.250 DEBUG:boto:Path: /ffzdqu-59/obj87 2023-09-22T21:49:43.251 DEBUG:boto:Data: 2023-09-22T21:49:43.251 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.251 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.251 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.251 DEBUG:boto:Params: {} 2023-09-22T21:49:43.251 DEBUG:boto:Token: None 2023-09-22T21:49:43.251 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj87 2023-09-22T21:49:43.251 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ahwLUC3+6r4nh+TSwLmvbEXl3Co= 2023-09-22T21:49:43.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ahwLUC3+6r4nh+TSwLmvbEXl3Co=', 'Content-Length': '0'} 2023-09-22T21:49:43.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c22d74b2e3de6ac2-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.298 DEBUG:boto:path=/ffzdqu-59/obj87 2023-09-22T21:49:43.299 DEBUG:boto:auth_path=/ffzdqu-59/obj87 2023-09-22T21:49:43.299 DEBUG:boto:Method: GET 2023-09-22T21:49:43.299 DEBUG:boto:Path: /ffzdqu-59/obj87 2023-09-22T21:49:43.299 DEBUG:boto:Data: 2023-09-22T21:49:43.299 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.299 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.299 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.299 DEBUG:boto:Params: {} 2023-09-22T21:49:43.299 DEBUG:boto:Token: None 2023-09-22T21:49:43.300 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj87 2023-09-22T21:49:43.300 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ahwLUC3+6r4nh+TSwLmvbEXl3Co= 2023-09-22T21:49:43.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ahwLUC3+6r4nh+TSwLmvbEXl3Co=', 'Content-Length': '0'} 2023-09-22T21:49:43.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:43:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050a3f398b50ccce4-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.303 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:43.303 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:43.303 DEBUG:boto:path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:49:43.303 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:49:43.303 DEBUG:boto:Method: GET 2023-09-22T21:49:43.303 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj87 2023-09-22T21:49:43.303 DEBUG:boto:Data: 2023-09-22T21:49:43.303 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.303 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.304 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.304 DEBUG:boto:Params: {} 2023-09-22T21:49:43.304 DEBUG:boto:Token: None 2023-09-22T21:49:43.304 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/ 2023-09-22T21:49:43.304 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw= 2023-09-22T21:49:43.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw=', 'Content-Length': '0'} 2023-09-22T21:49:43.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000347210b507862c69-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.309 DEBUG:boto:b'ffzdqu-59obj871000falseobj872023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:43.309 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:43.309 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:43.310 DEBUG:boto:path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:49:43.310 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj87 2023-09-22T21:49:43.310 DEBUG:boto:Method: GET 2023-09-22T21:49:43.310 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj87 2023-09-22T21:49:43.310 DEBUG:boto:Data: 2023-09-22T21:49:43.310 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.310 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.310 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.310 DEBUG:boto:Params: {} 2023-09-22T21:49:43.310 DEBUG:boto:Token: None 2023-09-22T21:49:43.311 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/ 2023-09-22T21:49:43.311 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw= 2023-09-22T21:49:43.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw=', 'Content-Length': '0'} 2023-09-22T21:49:43.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033047e358be29397-00650e0bf7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.362 DEBUG:boto:b'ffzdqu-59obj871000falseobj872023-09-22T21:43:12.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:43.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-22T21:49:43.363 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:49:43.363 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:49:43.363 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.363 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.363 DEBUG:boto:Method: GET 2023-09-22T21:49:43.363 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.364 DEBUG:boto:Data: 2023-09-22T21:49:43.364 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.364 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.364 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.364 DEBUG:boto:Params: {} 2023-09-22T21:49:43.364 DEBUG:boto:Token: None 2023-09-22T21:49:43.364 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.364 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e0jSUfHc9yctv1oYhMTMll8BxeQ= 2023-09-22T21:49:43.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e0jSUfHc9yctv1oYhMTMll8BxeQ=', 'Content-Length': '0'} 2023-09-22T21:49:43.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd7e874885d10a9c-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.368 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:49:43.368 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:49:43.368 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.368 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.368 DEBUG:boto:Method: GET 2023-09-22T21:49:43.368 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.369 DEBUG:boto:Data: 2023-09-22T21:49:43.369 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.369 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.369 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.369 DEBUG:boto:Params: {} 2023-09-22T21:49:43.369 DEBUG:boto:Token: None 2023-09-22T21:49:43.369 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.369 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e0jSUfHc9yctv1oYhMTMll8BxeQ= 2023-09-22T21:49:43.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e0jSUfHc9yctv1oYhMTMll8BxeQ=', 'Content-Length': '0'} 2023-09-22T21:49:43.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad5e0d68be0f18b9-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.373 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:49:43.373 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:49:43.373 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.373 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.373 DEBUG:boto:Method: HEAD 2023-09-22T21:49:43.373 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.373 DEBUG:boto:Data: 2023-09-22T21:49:43.373 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.374 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.374 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.374 DEBUG:boto:Params: {} 2023-09-22T21:49:43.374 DEBUG:boto:Token: None 2023-09-22T21:49:43.374 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.374 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:p+Fhfo5WFcZkKh6jyOOVuYBjiGE= 2023-09-22T21:49:43.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:p+Fhfo5WFcZkKh6jyOOVuYBjiGE=', 'Content-Length': '0'} 2023-09-22T21:49:43.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043d4ed4a10fac190-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.377 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:49:43.377 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:49:43.377 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.378 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.378 DEBUG:boto:Method: HEAD 2023-09-22T21:49:43.378 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.378 DEBUG:boto:Data: 2023-09-22T21:49:43.378 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.378 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.378 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.378 DEBUG:boto:Params: {} 2023-09-22T21:49:43.378 DEBUG:boto:Token: None 2023-09-22T21:49:43.379 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.379 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:p+Fhfo5WFcZkKh6jyOOVuYBjiGE= 2023-09-22T21:49:43.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:p+Fhfo5WFcZkKh6jyOOVuYBjiGE=', 'Content-Length': '0'} 2023-09-22T21:49:43.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2daa83e57500656-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-22T21:49:43.382 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:49:43.382 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:49:43.382 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.382 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.382 DEBUG:boto:Method: GET 2023-09-22T21:49:43.382 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.382 DEBUG:boto:Data: 2023-09-22T21:49:43.383 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.383 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.383 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.383 DEBUG:boto:Params: {} 2023-09-22T21:49:43.383 DEBUG:boto:Token: None 2023-09-22T21:49:43.383 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.383 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e0jSUfHc9yctv1oYhMTMll8BxeQ= 2023-09-22T21:49:43.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e0jSUfHc9yctv1oYhMTMll8BxeQ=', 'Content-Length': '0'} 2023-09-22T21:49:43.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3855eddc41c1d91-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.386 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:49:43.386 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:49:43.387 DEBUG:boto:path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.387 DEBUG:boto:auth_path=/ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.387 DEBUG:boto:Method: GET 2023-09-22T21:49:43.387 DEBUG:boto:Path: /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.387 DEBUG:boto:Data: 2023-09-22T21:49:43.387 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.387 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.387 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.387 DEBUG:boto:Params: {} 2023-09-22T21:49:43.387 DEBUG:boto:Token: None 2023-09-22T21:49:43.388 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj92?versionId=null 2023-09-22T21:49:43.388 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:e0jSUfHc9yctv1oYhMTMll8BxeQ= 2023-09-22T21:49:43.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:e0jSUfHc9yctv1oYhMTMll8BxeQ=', 'Content-Length': '0'} 2023-09-22T21:49:43.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073de3d8bb7be2d90-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.391 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:43.391 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:43.391 DEBUG:boto:path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:49:43.391 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:49:43.391 DEBUG:boto:Method: GET 2023-09-22T21:49:43.391 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj92 2023-09-22T21:49:43.391 DEBUG:boto:Data: 2023-09-22T21:49:43.391 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.391 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.391 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.392 DEBUG:boto:Params: {} 2023-09-22T21:49:43.392 DEBUG:boto:Token: None 2023-09-22T21:49:43.392 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/ 2023-09-22T21:49:43.392 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw= 2023-09-22T21:49:43.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw=', 'Content-Length': '0'} 2023-09-22T21:49:43.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c02a57f63743a4b5-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.397 DEBUG:boto:b'ffzdqu-59obj921000falseobj922023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:43.397 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:43.397 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:43.398 DEBUG:boto:path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:49:43.398 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:49:43.398 DEBUG:boto:Method: GET 2023-09-22T21:49:43.398 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj92 2023-09-22T21:49:43.398 DEBUG:boto:Data: 2023-09-22T21:49:43.398 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.398 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.398 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.398 DEBUG:boto:Params: {} 2023-09-22T21:49:43.398 DEBUG:boto:Token: None 2023-09-22T21:49:43.399 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/ 2023-09-22T21:49:43.399 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw= 2023-09-22T21:49:43.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw=', 'Content-Length': '0'} 2023-09-22T21:49:43.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034e113b3762545ec-00650e0bf7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.450 DEBUG:boto:b'ffzdqu-59obj921000falseobj922023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:43.451 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:49:43.451 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:49:43.451 DEBUG:boto:Method: HEAD 2023-09-22T21:49:43.451 DEBUG:boto:Path: /ffzdqu-59/obj92 2023-09-22T21:49:43.451 DEBUG:boto:Data: 2023-09-22T21:49:43.451 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.451 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.451 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.452 DEBUG:boto:Params: {} 2023-09-22T21:49:43.452 DEBUG:boto:Token: None 2023-09-22T21:49:43.452 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj92 2023-09-22T21:49:43.452 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f/8amQ1gR5UMGlp3N+1qFIsBJuM= 2023-09-22T21:49:43.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f/8amQ1gR5UMGlp3N+1qFIsBJuM=', 'Content-Length': '0'} 2023-09-22T21:49:43.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085b0e05aa7f17e55-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.455 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:49:43.455 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:49:43.455 DEBUG:boto:Method: HEAD 2023-09-22T21:49:43.456 DEBUG:boto:Path: /ffzdqu-59/obj92 2023-09-22T21:49:43.456 DEBUG:boto:Data: 2023-09-22T21:49:43.456 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.456 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.456 DEBUG:boto:Params: {} 2023-09-22T21:49:43.456 DEBUG:boto:Token: None 2023-09-22T21:49:43.456 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj92 2023-09-22T21:49:43.457 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f/8amQ1gR5UMGlp3N+1qFIsBJuM= 2023-09-22T21:49:43.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f/8amQ1gR5UMGlp3N+1qFIsBJuM=', 'Content-Length': '0'} 2023-09-22T21:49:43.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045eb69b2ad243369-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-22T21:49:43.460 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:49:43.460 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:49:43.460 DEBUG:boto:Method: GET 2023-09-22T21:49:43.460 DEBUG:boto:Path: /ffzdqu-59/obj92 2023-09-22T21:49:43.460 DEBUG:boto:Data: 2023-09-22T21:49:43.460 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.460 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.461 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.461 DEBUG:boto:Params: {} 2023-09-22T21:49:43.461 DEBUG:boto:Token: None 2023-09-22T21:49:43.461 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj92 2023-09-22T21:49:43.461 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eMLOoYDXLGaP1lb74zME7JbEUKk= 2023-09-22T21:49:43.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eMLOoYDXLGaP1lb74zME7JbEUKk=', 'Content-Length': '0'} 2023-09-22T21:49:43.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dcf46260ed008cc-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.506 DEBUG:boto:path=/ffzdqu-59/obj92 2023-09-22T21:49:43.506 DEBUG:boto:auth_path=/ffzdqu-59/obj92 2023-09-22T21:49:43.507 DEBUG:boto:Method: GET 2023-09-22T21:49:43.507 DEBUG:boto:Path: /ffzdqu-59/obj92 2023-09-22T21:49:43.507 DEBUG:boto:Data: 2023-09-22T21:49:43.507 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.507 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.507 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.507 DEBUG:boto:Params: {} 2023-09-22T21:49:43.507 DEBUG:boto:Token: None 2023-09-22T21:49:43.508 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj92 2023-09-22T21:49:43.508 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eMLOoYDXLGaP1lb74zME7JbEUKk= 2023-09-22T21:49:43.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eMLOoYDXLGaP1lb74zME7JbEUKk=', 'Content-Length': '0'} 2023-09-22T21:49:43.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:40:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9d4c75ab9884f2c-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.511 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:43.511 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:43.511 DEBUG:boto:path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:49:43.512 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:49:43.512 DEBUG:boto:Method: GET 2023-09-22T21:49:43.512 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj92 2023-09-22T21:49:43.512 DEBUG:boto:Data: 2023-09-22T21:49:43.512 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.512 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.512 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.512 DEBUG:boto:Params: {} 2023-09-22T21:49:43.512 DEBUG:boto:Token: None 2023-09-22T21:49:43.513 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/ 2023-09-22T21:49:43.513 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw= 2023-09-22T21:49:43.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw=', 'Content-Length': '0'} 2023-09-22T21:49:43.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d84dad42c620ccba-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.518 DEBUG:boto:b'ffzdqu-59obj921000falseobj922023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:43.518 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:43.518 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:43.518 DEBUG:boto:path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:49:43.518 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj92 2023-09-22T21:49:43.519 DEBUG:boto:Method: GET 2023-09-22T21:49:43.519 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj92 2023-09-22T21:49:43.519 DEBUG:boto:Data: 2023-09-22T21:49:43.519 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.519 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.519 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.519 DEBUG:boto:Params: {} 2023-09-22T21:49:43.519 DEBUG:boto:Token: None 2023-09-22T21:49:43.519 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/ 2023-09-22T21:49:43.519 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw= 2023-09-22T21:49:43.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw=', 'Content-Length': '0'} 2023-09-22T21:49:43.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000181b5a81f73909f4-00650e0bf7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.530 DEBUG:boto:b'ffzdqu-59obj921000falseobj922023-09-22T21:40:21.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:43.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-22T21:49:43.530 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:49:43.531 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:49:43.531 DEBUG:boto:path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.531 DEBUG:boto:auth_path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.531 DEBUG:boto:Method: GET 2023-09-22T21:49:43.531 DEBUG:boto:Path: /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.531 DEBUG:boto:Data: 2023-09-22T21:49:43.531 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.531 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.531 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.531 DEBUG:boto:Params: {} 2023-09-22T21:49:43.532 DEBUG:boto:Token: None 2023-09-22T21:49:43.532 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.532 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dj1A61arUIVHGgRmJLpHeljEwrg= 2023-09-22T21:49:43.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dj1A61arUIVHGgRmJLpHeljEwrg=', 'Content-Length': '0'} 2023-09-22T21:49:43.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a529c5121e0a9c6-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.535 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:49:43.536 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:49:43.536 DEBUG:boto:path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.536 DEBUG:boto:auth_path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.536 DEBUG:boto:Method: GET 2023-09-22T21:49:43.536 DEBUG:boto:Path: /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.536 DEBUG:boto:Data: 2023-09-22T21:49:43.536 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.536 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.536 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.536 DEBUG:boto:Params: {} 2023-09-22T21:49:43.537 DEBUG:boto:Token: None 2023-09-22T21:49:43.537 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.537 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dj1A61arUIVHGgRmJLpHeljEwrg= 2023-09-22T21:49:43.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dj1A61arUIVHGgRmJLpHeljEwrg=', 'Content-Length': '0'} 2023-09-22T21:49:43.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045aae7a397f0edd1-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.582 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:49:43.583 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:49:43.583 DEBUG:boto:path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.583 DEBUG:boto:auth_path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.583 DEBUG:boto:Method: HEAD 2023-09-22T21:49:43.583 DEBUG:boto:Path: /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.583 DEBUG:boto:Data: 2023-09-22T21:49:43.583 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.583 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.583 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.583 DEBUG:boto:Params: {} 2023-09-22T21:49:43.584 DEBUG:boto:Token: None 2023-09-22T21:49:43.584 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.584 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7TwBlBvBlbTYLRNyRz+N9Ew+x0U= 2023-09-22T21:49:43.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7TwBlBvBlbTYLRNyRz+N9Ew+x0U=', 'Content-Length': '0'} 2023-09-22T21:49:43.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcd20ca09131df01-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.587 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:49:43.587 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:49:43.587 DEBUG:boto:path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.587 DEBUG:boto:auth_path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.588 DEBUG:boto:Method: HEAD 2023-09-22T21:49:43.588 DEBUG:boto:Path: /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.588 DEBUG:boto:Data: 2023-09-22T21:49:43.588 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.588 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.588 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.588 DEBUG:boto:Params: {} 2023-09-22T21:49:43.588 DEBUG:boto:Token: None 2023-09-22T21:49:43.588 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.589 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7TwBlBvBlbTYLRNyRz+N9Ew+x0U= 2023-09-22T21:49:43.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7TwBlBvBlbTYLRNyRz+N9Ew+x0U=', 'Content-Length': '0'} 2023-09-22T21:49:43.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c623f2ebabb6164d-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-22T21:49:43.592 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:49:43.592 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:49:43.592 DEBUG:boto:path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.592 DEBUG:boto:auth_path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.592 DEBUG:boto:Method: GET 2023-09-22T21:49:43.593 DEBUG:boto:Path: /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.593 DEBUG:boto:Data: 2023-09-22T21:49:43.593 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.593 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.593 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.593 DEBUG:boto:Params: {} 2023-09-22T21:49:43.593 DEBUG:boto:Token: None 2023-09-22T21:49:43.593 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.593 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dj1A61arUIVHGgRmJLpHeljEwrg= 2023-09-22T21:49:43.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dj1A61arUIVHGgRmJLpHeljEwrg=', 'Content-Length': '0'} 2023-09-22T21:49:43.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002866eff45a8be544-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.638 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:49:43.638 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:49:43.639 DEBUG:boto:path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.639 DEBUG:boto:auth_path=/ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.639 DEBUG:boto:Method: GET 2023-09-22T21:49:43.639 DEBUG:boto:Path: /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.639 DEBUG:boto:Data: 2023-09-22T21:49:43.639 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.639 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.639 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.639 DEBUG:boto:Params: {} 2023-09-22T21:49:43.639 DEBUG:boto:Token: None 2023-09-22T21:49:43.640 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj93?versionId=null 2023-09-22T21:49:43.640 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dj1A61arUIVHGgRmJLpHeljEwrg= 2023-09-22T21:49:43.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dj1A61arUIVHGgRmJLpHeljEwrg=', 'Content-Length': '0'} 2023-09-22T21:49:43.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c42f8c480a8abe89-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.643 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:43.643 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:43.643 DEBUG:boto:path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:49:43.643 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:49:43.644 DEBUG:boto:Method: GET 2023-09-22T21:49:43.644 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj93 2023-09-22T21:49:43.644 DEBUG:boto:Data: 2023-09-22T21:49:43.644 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.644 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.644 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.644 DEBUG:boto:Params: {} 2023-09-22T21:49:43.644 DEBUG:boto:Token: None 2023-09-22T21:49:43.644 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/ 2023-09-22T21:49:43.645 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw= 2023-09-22T21:49:43.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw=', 'Content-Length': '0'} 2023-09-22T21:49:43.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb4b65ed38f7d30d-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.649 DEBUG:boto:b'ffzdqu-59obj931000falseobj932023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:43.650 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:43.650 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:43.650 DEBUG:boto:path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:49:43.650 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:49:43.650 DEBUG:boto:Method: GET 2023-09-22T21:49:43.650 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj93 2023-09-22T21:49:43.650 DEBUG:boto:Data: 2023-09-22T21:49:43.650 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.651 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.651 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.651 DEBUG:boto:Params: {} 2023-09-22T21:49:43.651 DEBUG:boto:Token: None 2023-09-22T21:49:43.651 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/ 2023-09-22T21:49:43.651 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw= 2023-09-22T21:49:43.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw=', 'Content-Length': '0'} 2023-09-22T21:49:43.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c786906b08cfdd14-00650e0bf7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.661 DEBUG:boto:b'ffzdqu-59obj931000falseobj932023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:43.661 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:49:43.662 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:49:43.662 DEBUG:boto:Method: HEAD 2023-09-22T21:49:43.662 DEBUG:boto:Path: /ffzdqu-59/obj93 2023-09-22T21:49:43.662 DEBUG:boto:Data: 2023-09-22T21:49:43.662 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.662 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.662 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.662 DEBUG:boto:Params: {} 2023-09-22T21:49:43.662 DEBUG:boto:Token: None 2023-09-22T21:49:43.663 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj93 2023-09-22T21:49:43.663 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hbS929ojh1Jjv1FZ0EdimvElx5Y= 2023-09-22T21:49:43.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hbS929ojh1Jjv1FZ0EdimvElx5Y=', 'Content-Length': '0'} 2023-09-22T21:49:43.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b84d5716d9eebbe2-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.666 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:49:43.666 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:49:43.666 DEBUG:boto:Method: HEAD 2023-09-22T21:49:43.666 DEBUG:boto:Path: /ffzdqu-59/obj93 2023-09-22T21:49:43.666 DEBUG:boto:Data: 2023-09-22T21:49:43.666 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.666 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.667 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.667 DEBUG:boto:Params: {} 2023-09-22T21:49:43.667 DEBUG:boto:Token: None 2023-09-22T21:49:43.667 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj93 2023-09-22T21:49:43.667 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hbS929ojh1Jjv1FZ0EdimvElx5Y= 2023-09-22T21:49:43.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hbS929ojh1Jjv1FZ0EdimvElx5Y=', 'Content-Length': '0'} 2023-09-22T21:49:43.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f48892cc92b6866-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-22T21:49:43.670 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:49:43.670 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:49:43.671 DEBUG:boto:Method: GET 2023-09-22T21:49:43.671 DEBUG:boto:Path: /ffzdqu-59/obj93 2023-09-22T21:49:43.671 DEBUG:boto:Data: 2023-09-22T21:49:43.671 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.671 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.671 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.671 DEBUG:boto:Params: {} 2023-09-22T21:49:43.671 DEBUG:boto:Token: None 2023-09-22T21:49:43.671 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj93 2023-09-22T21:49:43.672 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WbTzkStx64TnN++Vncf9Db83dUc= 2023-09-22T21:49:43.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WbTzkStx64TnN++Vncf9Db83dUc=', 'Content-Length': '0'} 2023-09-22T21:49:43.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000342d8868e1f7e22b-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.718 DEBUG:boto:path=/ffzdqu-59/obj93 2023-09-22T21:49:43.718 DEBUG:boto:auth_path=/ffzdqu-59/obj93 2023-09-22T21:49:43.718 DEBUG:boto:Method: GET 2023-09-22T21:49:43.719 DEBUG:boto:Path: /ffzdqu-59/obj93 2023-09-22T21:49:43.719 DEBUG:boto:Data: 2023-09-22T21:49:43.719 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.719 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.719 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.719 DEBUG:boto:Params: {} 2023-09-22T21:49:43.719 DEBUG:boto:Token: None 2023-09-22T21:49:43.719 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj93 2023-09-22T21:49:43.720 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WbTzkStx64TnN++Vncf9Db83dUc= 2023-09-22T21:49:43.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WbTzkStx64TnN++Vncf9Db83dUc=', 'Content-Length': '0'} 2023-09-22T21:49:43.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:45:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bbba426093d4314-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.723 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:43.723 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:43.723 DEBUG:boto:path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:49:43.724 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:49:43.724 DEBUG:boto:Method: GET 2023-09-22T21:49:43.724 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj93 2023-09-22T21:49:43.724 DEBUG:boto:Data: 2023-09-22T21:49:43.724 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.724 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.724 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.724 DEBUG:boto:Params: {} 2023-09-22T21:49:43.724 DEBUG:boto:Token: None 2023-09-22T21:49:43.725 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/ 2023-09-22T21:49:43.725 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw= 2023-09-22T21:49:43.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw=', 'Content-Length': '0'} 2023-09-22T21:49:43.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021acf630bdfaa15b-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.770 DEBUG:boto:b'ffzdqu-59obj931000falseobj932023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:43.771 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:43.771 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:43.771 DEBUG:boto:path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:49:43.771 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj93 2023-09-22T21:49:43.771 DEBUG:boto:Method: GET 2023-09-22T21:49:43.771 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj93 2023-09-22T21:49:43.771 DEBUG:boto:Data: 2023-09-22T21:49:43.772 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.772 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.772 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.772 DEBUG:boto:Params: {} 2023-09-22T21:49:43.772 DEBUG:boto:Token: None 2023-09-22T21:49:43.772 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/ 2023-09-22T21:49:43.772 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw= 2023-09-22T21:49:43.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw=', 'Content-Length': '0'} 2023-09-22T21:49:43.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc0357418ea08b79-00650e0bf7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.784 DEBUG:boto:b'ffzdqu-59obj931000falseobj932023-09-22T21:45:13.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:43.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T21:49:43.784 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:49:43.784 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:49:43.784 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.784 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.785 DEBUG:boto:Method: GET 2023-09-22T21:49:43.785 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.785 DEBUG:boto:Data: 2023-09-22T21:49:43.785 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.785 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.785 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.785 DEBUG:boto:Params: {} 2023-09-22T21:49:43.785 DEBUG:boto:Token: None 2023-09-22T21:49:43.785 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.786 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/RogvlQnUWHWgibmbIDkbsfqUvo= 2023-09-22T21:49:43.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/RogvlQnUWHWgibmbIDkbsfqUvo=', 'Content-Length': '0'} 2023-09-22T21:49:43.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000550bd7b124cd49fe-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.789 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:49:43.789 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:49:43.789 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.789 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.789 DEBUG:boto:Method: GET 2023-09-22T21:49:43.789 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.789 DEBUG:boto:Data: 2023-09-22T21:49:43.789 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.789 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.790 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.790 DEBUG:boto:Params: {} 2023-09-22T21:49:43.790 DEBUG:boto:Token: None 2023-09-22T21:49:43.790 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.790 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/RogvlQnUWHWgibmbIDkbsfqUvo= 2023-09-22T21:49:43.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/RogvlQnUWHWgibmbIDkbsfqUvo=', 'Content-Length': '0'} 2023-09-22T21:49:43.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000962eb73ed1489acd-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.793 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:49:43.794 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:49:43.794 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.794 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.794 DEBUG:boto:Method: HEAD 2023-09-22T21:49:43.794 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.794 DEBUG:boto:Data: 2023-09-22T21:49:43.794 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.794 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.794 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.794 DEBUG:boto:Params: {} 2023-09-22T21:49:43.795 DEBUG:boto:Token: None 2023-09-22T21:49:43.795 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.795 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:d1X8R0Pa3GFIKgPk9hMlVRZP1Dg= 2023-09-22T21:49:43.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:d1X8R0Pa3GFIKgPk9hMlVRZP1Dg=', 'Content-Length': '0'} 2023-09-22T21:49:43.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d489a5d1dfbddcf-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.798 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:49:43.798 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:49:43.798 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.798 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.798 DEBUG:boto:Method: HEAD 2023-09-22T21:49:43.798 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.798 DEBUG:boto:Data: 2023-09-22T21:49:43.799 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.799 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.799 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.799 DEBUG:boto:Params: {} 2023-09-22T21:49:43.799 DEBUG:boto:Token: None 2023-09-22T21:49:43.799 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.799 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:d1X8R0Pa3GFIKgPk9hMlVRZP1Dg= 2023-09-22T21:49:43.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:d1X8R0Pa3GFIKgPk9hMlVRZP1Dg=', 'Content-Length': '0'} 2023-09-22T21:49:43.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001444e2859bd42187-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T21:49:43.802 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:49:43.802 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:49:43.803 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.803 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.803 DEBUG:boto:Method: GET 2023-09-22T21:49:43.803 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.803 DEBUG:boto:Data: 2023-09-22T21:49:43.803 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.803 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.803 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.803 DEBUG:boto:Params: {} 2023-09-22T21:49:43.803 DEBUG:boto:Token: None 2023-09-22T21:49:43.804 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.804 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/RogvlQnUWHWgibmbIDkbsfqUvo= 2023-09-22T21:49:43.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/RogvlQnUWHWgibmbIDkbsfqUvo=', 'Content-Length': '0'} 2023-09-22T21:49:43.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f24d6433a4b9024-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.850 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:49:43.850 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:49:43.850 DEBUG:boto:path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.851 DEBUG:boto:auth_path=/ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.851 DEBUG:boto:Method: GET 2023-09-22T21:49:43.851 DEBUG:boto:Path: /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.851 DEBUG:boto:Data: 2023-09-22T21:49:43.851 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.851 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.851 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.851 DEBUG:boto:Params: {} 2023-09-22T21:49:43.851 DEBUG:boto:Token: None 2023-09-22T21:49:43.852 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj95?versionId=null 2023-09-22T21:49:43.852 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/RogvlQnUWHWgibmbIDkbsfqUvo= 2023-09-22T21:49:43.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/RogvlQnUWHWgibmbIDkbsfqUvo=', 'Content-Length': '0'} 2023-09-22T21:49:43.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcdca2357ecb12e0-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.855 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:43.855 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:43.855 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:49:43.855 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:49:43.855 DEBUG:boto:Method: GET 2023-09-22T21:49:43.856 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:49:43.856 DEBUG:boto:Data: 2023-09-22T21:49:43.856 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.856 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.856 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.856 DEBUG:boto:Params: {} 2023-09-22T21:49:43.856 DEBUG:boto:Token: None 2023-09-22T21:49:43.856 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/ 2023-09-22T21:49:43.856 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw= 2023-09-22T21:49:43.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw=', 'Content-Length': '0'} 2023-09-22T21:49:43.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000529e6a45722da3ee-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.862 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:43.862 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:43.862 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:43.862 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:49:43.862 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:49:43.863 DEBUG:boto:Method: GET 2023-09-22T21:49:43.863 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:49:43.863 DEBUG:boto:Data: 2023-09-22T21:49:43.863 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.863 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.863 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.863 DEBUG:boto:Params: {} 2023-09-22T21:49:43.863 DEBUG:boto:Token: None 2023-09-22T21:49:43.863 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/ 2023-09-22T21:49:43.864 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw= 2023-09-22T21:49:43.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw=', 'Content-Length': '0'} 2023-09-22T21:49:43.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008086ef3a336f6c6-00650e0bf7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.873 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:43.873 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:49:43.873 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:49:43.873 DEBUG:boto:Method: HEAD 2023-09-22T21:49:43.874 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:49:43.874 DEBUG:boto:Data: 2023-09-22T21:49:43.874 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.874 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.874 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.874 DEBUG:boto:Params: {} 2023-09-22T21:49:43.874 DEBUG:boto:Token: None 2023-09-22T21:49:43.874 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj95 2023-09-22T21:49:43.875 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hGO5ErLjinxidy8mqNmBH5QGUPM= 2023-09-22T21:49:43.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hGO5ErLjinxidy8mqNmBH5QGUPM=', 'Content-Length': '0'} 2023-09-22T21:49:43.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bac8d51bc31dbae0-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.877 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:49:43.878 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:49:43.878 DEBUG:boto:Method: HEAD 2023-09-22T21:49:43.878 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:49:43.878 DEBUG:boto:Data: 2023-09-22T21:49:43.878 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.878 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.878 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.878 DEBUG:boto:Params: {} 2023-09-22T21:49:43.878 DEBUG:boto:Token: None 2023-09-22T21:49:43.879 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj95 2023-09-22T21:49:43.879 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hGO5ErLjinxidy8mqNmBH5QGUPM= 2023-09-22T21:49:43.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hGO5ErLjinxidy8mqNmBH5QGUPM=', 'Content-Length': '0'} 2023-09-22T21:49:43.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c7547b0bd96ab01-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T21:49:43.882 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:49:43.882 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:49:43.882 DEBUG:boto:Method: GET 2023-09-22T21:49:43.882 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:49:43.882 DEBUG:boto:Data: 2023-09-22T21:49:43.883 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.883 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.883 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.883 DEBUG:boto:Params: {} 2023-09-22T21:49:43.883 DEBUG:boto:Token: None 2023-09-22T21:49:43.883 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj95 2023-09-22T21:49:43.883 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:25QwR8qYeWzhxMcQETdWal24MRY= 2023-09-22T21:49:43.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:25QwR8qYeWzhxMcQETdWal24MRY=', 'Content-Length': '0'} 2023-09-22T21:49:43.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-59:f132bdb9-20a7-4073-a932-8f99da20691e.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003da30f0841d4747a-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.926 DEBUG:boto:path=/ffzdqu-59/obj95 2023-09-22T21:49:43.926 DEBUG:boto:auth_path=/ffzdqu-59/obj95 2023-09-22T21:49:43.927 DEBUG:boto:Method: GET 2023-09-22T21:49:43.927 DEBUG:boto:Path: /ffzdqu-59/obj95 2023-09-22T21:49:43.927 DEBUG:boto:Data: 2023-09-22T21:49:43.927 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.927 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.927 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.927 DEBUG:boto:Params: {} 2023-09-22T21:49:43.927 DEBUG:boto:Token: None 2023-09-22T21:49:43.928 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/obj95 2023-09-22T21:49:43.928 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:25QwR8qYeWzhxMcQETdWal24MRY= 2023-09-22T21:49:43.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:25QwR8qYeWzhxMcQETdWal24MRY=', 'Content-Length': '0'} 2023-09-22T21:49:43.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:38:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e7dd2acb32fdb1f-00650e0bf7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.931 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:43.931 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:43.931 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:49:43.931 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:49:43.932 DEBUG:boto:Method: GET 2023-09-22T21:49:43.932 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:49:43.932 DEBUG:boto:Data: 2023-09-22T21:49:43.932 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.932 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:49:43.932 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.932 DEBUG:boto:Params: {} 2023-09-22T21:49:43.932 DEBUG:boto:Token: None 2023-09-22T21:49:43.932 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/ 2023-09-22T21:49:43.933 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw= 2023-09-22T21:49:43.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw=', 'Content-Length': '0'} 2023-09-22T21:49:43.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6c26e8ed2c9eafd-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.938 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:43.938 DEBUG:boto:path=/ffzdqu-59/ 2023-09-22T21:49:43.938 DEBUG:boto:auth_path=/ffzdqu-59/ 2023-09-22T21:49:43.938 DEBUG:boto:path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:49:43.939 DEBUG:boto:auth_path=/ffzdqu-59/?prefix=obj95 2023-09-22T21:49:43.939 DEBUG:boto:Method: GET 2023-09-22T21:49:43.939 DEBUG:boto:Path: /ffzdqu-59/?prefix=obj95 2023-09-22T21:49:43.939 DEBUG:boto:Data: 2023-09-22T21:49:43.939 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.939 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:49:43.939 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.939 DEBUG:boto:Params: {} 2023-09-22T21:49:43.939 DEBUG:boto:Token: None 2023-09-22T21:49:43.940 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:49:43 GMT /ffzdqu-59/ 2023-09-22T21:49:43.940 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw= 2023-09-22T21:49:43.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:49:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pTXV63al/23dRoSt88NuUL1zYEw=', 'Content-Length': '0'} 2023-09-22T21:49:43.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054593f8f0d53e973-00650e0bf7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.949 DEBUG:boto:b'ffzdqu-59obj951000falseobj952023-09-22T21:38:31.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:49:43.950 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-59 zones={a2, a1} 2023-09-22T21:49:43.950 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... ok 2023-09-22T21:49:43.954 INFO:rgw_multi.tests:create role zone=a1 name=roles-ffzdqu-1 2023-09-22T21:49:43.954 DEBUG:boto:Method: POST 2023-09-22T21:49:43.954 DEBUG:boto:Path: / 2023-09-22T21:49:43.954 DEBUG:boto:Data: 2023-09-22T21:49:43.954 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.954 DEBUG:boto:Host: smithi160.front.sepia.ceph.com 2023-09-22T21:49:43.954 DEBUG:boto:Port: 8000 2023-09-22T21:49:43.955 DEBUG:boto:Params: {'RoleName': 'roles-ffzdqu-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-09-22T21:49:43.955 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:49:43.955 DEBUG:boto:Token: None 2023-09-22T21:49:43.955 DEBUG:boto:CanonicalRequest: POST / host:smithi160.front.sepia.ceph.com:8000 x-amz-date:20230922T214943Z host;x-amz-date cafff31873b6d15713bff69ed68eedf7ddedd90e88c70b56a8d3a6b035037b8d 2023-09-22T21:49:43.955 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230922T214943Z 20230922/front/smithi160/aws4_request 49912425cfc4d56bc3ac564c8b90ec90244952a037ed1fb8e980b2c5d7c84ef2 2023-09-22T21:49:43.955 DEBUG:boto:Signature: 78f20d2d429516dfad926b9ab5e4a4b86792a907535561bbd6df0fd0eafa2543 2023-09-22T21:49:43.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230922T214943Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RX9T9XC1DLDGURC46TDQ/20230922/front/smithi160/aws4_request,SignedHeaders=host;x-amz-date,Signature=78f20d2d429516dfad926b9ab5e4a4b86792a907535561bbd6df0fd0eafa2543', 'Host': 'smithi160.front.sepia.ceph.com:8000'} 2023-09-22T21:49:43.980 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a4bc71fe8cd0e53b-00650e0bf7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '723'), ('Date', 'Fri, 22 Sep 2023 21:49:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:43.980 DEBUG:boto:b'197fda3e-8291-42ad-9d50-2573d7498b0broles-ffzdqu-1/arn:aws:iam:::role/roles-ffzdqu-12023-09-22T21:49:43.964Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000a4bc71fe8cd0e53b-00650e0bf7-165f-a1' 2023-09-22T21:49:43.981 INFO:rgw_multi.tests:create role zone=a2 name=roles-ffzdqu-2 2023-09-22T21:49:43.981 DEBUG:boto:Method: POST 2023-09-22T21:49:43.981 DEBUG:boto:Path: / 2023-09-22T21:49:43.981 DEBUG:boto:Data: 2023-09-22T21:49:43.982 DEBUG:boto:Headers: {} 2023-09-22T21:49:43.982 DEBUG:boto:Host: smithi160.front.sepia.ceph.com 2023-09-22T21:49:43.982 DEBUG:boto:Port: 8001 2023-09-22T21:49:43.982 DEBUG:boto:Params: {'RoleName': 'roles-ffzdqu-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-09-22T21:49:43.982 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:49:43.982 DEBUG:boto:Token: None 2023-09-22T21:49:43.982 DEBUG:boto:CanonicalRequest: POST / host:smithi160.front.sepia.ceph.com:8001 x-amz-date:20230922T214943Z host;x-amz-date 73eb5ba9f28e722c04b606a48cd9db0bc74b5fdd16beea7a7bf199aacd0df0a9 2023-09-22T21:49:43.983 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230922T214943Z 20230922/front/smithi160/aws4_request 315fb55976e23254180f7e92cfaf0265ff89341eaf8f51a6e21d3750aabe4520 2023-09-22T21:49:43.983 DEBUG:boto:Signature: 208a6e3eef7295c77bb441dfe5ca70e63918b3acfb693bf21a5c4a742d2373a6 2023-09-22T21:49:43.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230922T214943Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RX9T9XC1DLDGURC46TDQ/20230922/front/smithi160/aws4_request,SignedHeaders=host;x-amz-date,Signature=208a6e3eef7295c77bb441dfe5ca70e63918b3acfb693bf21a5c4a742d2373a6', 'Host': 'smithi160.front.sepia.ceph.com:8001'} 2023-09-22T21:49:44.025 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007771ef05d440ec7f-00650e0bf7-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '722'), ('Date', 'Fri, 22 Sep 2023 21:49:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:49:44.025 DEBUG:boto:b'09b48eb6-5f0b-4445-88a9-c807813157a4roles-ffzdqu-2/arn:aws:iam:::role/roles-ffzdqu-22023-09-22T21:49:44.13Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx000007771ef05d440ec7f-00650e0bf7-1d9d-a2' 2023-09-22T21:49:44.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:49:44.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:49:44.026 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:49:44.066 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:44.066 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:44.106 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:49:44.127 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"1_1695419384.000133_332.1","last_update":"2023-09-22T21:49:44.000133Z"},{"marker":"1_1695419236.317321_547.1","last_update":"2023-09-22T21:47:16.317321Z"},{"marker":"1_1695419383.971498_240.1","last_update":"2023-09-22T21:49:43.971498Z"}] 2023-09-22T21:49:44.127 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419236.317321_547.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:49:44.127 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:49:44.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:49:44.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:49:44.128 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:49:44.206 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:44.303 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:44.304 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:44.240+0000 7f827ff71a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:49:44.304 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:44.244+0000 7f827ff71a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:49:44.305 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:49:44Z"} 2023-09-22T21:49:44.306 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:49:44Z"} 2023-09-22T21:49:44.306 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695419236.317321_547.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:47:16.317321Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:49:44.306 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419236.317321_547.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:49:44.306 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695419236.317321_547.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:49:44.306 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1695419384.000133_332.1 target=, shard=3 master=1_1695419383.971498_240.1 target=1_1695416704.185257_96.1 2023-09-22T21:49:49.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:49:49.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:49:49.308 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:49:49.352 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:49.352 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:49.390 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:49.388+0000 7fb323864a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:49:49.391 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:49.392+0000 7fb323864a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:49:49.410 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:49:49Z"} 2023-09-22T21:49:49.410 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:49:49Z"} 2023-09-22T21:49:49.411 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695419236.317321_547.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:47:16.317321Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:49:49.411 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419236.317321_547.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:49:49.411 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695419236.317321_547.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:49:49.411 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1695419384.000133_332.1 target=, shard=3 master=1_1695419383.971498_240.1 target=1_1695416704.185257_96.1 2023-09-22T21:49:54.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:49:54.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:49:54.413 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:49:54.457 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:54.457 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:54.496 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:54.496+0000 7fdeacc7aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:49:54.497 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:54.496+0000 7fdeacc7aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:49:54.517 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:49:54Z"} 2023-09-22T21:49:54.517 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:49:54Z"} 2023-09-22T21:49:54.517 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695419236.317321_547.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:47:16.317321Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:49:54.517 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419236.317321_547.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:49:54.518 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695419236.317321_547.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:49:54.518 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1695419384.000133_332.1 target=, shard=3 master=1_1695419383.971498_240.1 target=1_1695416704.185257_96.1 2023-09-22T21:49:59.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:49:59.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:49:59.519 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:49:59.562 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:49:59.562 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:49:59.601 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:59.600+0000 7f7f6a1a3a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:49:59.602 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:49:59.600+0000 7f7f6a1a3a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:49:59.621 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:49:59Z"} 2023-09-22T21:49:59.621 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:49:59Z"} 2023-09-22T21:49:59.621 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695419236.317321_547.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:47:16.317321Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:49:59.621 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419236.317321_547.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:49:59.622 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695419236.317321_547.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:49:59.622 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1695419384.000133_332.1 target=, shard=3 master=1_1695419383.971498_240.1 target=1_1695416704.185257_96.1 2023-09-22T21:50:04.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:04.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:04.623 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:50:04.665 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:50:04.666 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:50:04.705 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:04.703+0000 7feb9cfdfa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:04.705 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:04.703+0000 7feb9cfdfa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:04.723 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:04Z"} 2023-09-22T21:50:04.863 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:04Z"} 2023-09-22T21:50:04.864 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695419236.317321_547.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:47:16.317321Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:50:04.864 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419236.317321_547.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:50:04.864 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695419236.317321_547.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:50:04.864 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1695419384.000133_332.1 target=, shard=3 master=1_1695419383.971498_240.1 target=1_1695416704.185257_96.1 2023-09-22T21:50:09.865 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:09.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:09.865 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:50:09.909 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:50:09.909 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:50:09.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:09.947+0000 7f21fcd2aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:09.949 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:09.947+0000 7f21fcd2aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:09.968 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:09Z"} 2023-09-22T21:50:09.969 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:09Z"} 2023-09-22T21:50:09.969 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695419236.317321_547.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:47:16.317321Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:50:09.969 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419236.317321_547.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:50:09.969 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695419236.317321_547.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:50:09.969 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1695419384.000133_332.1 target=, shard=3 master=1_1695419383.971498_240.1 target=1_1695416704.185257_96.1 2023-09-22T21:50:14.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:14.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:14.971 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:50:15.014 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:50:15.015 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:50:15.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:15.051+0000 7fc20db27a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:15.053 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:15.051+0000 7fc20db27a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:15.072 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:15Z"} 2023-09-22T21:50:15.072 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:15Z"} 2023-09-22T21:50:15.072 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695419236.317321_547.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:47:16.317321Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:50:15.072 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419236.317321_547.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:50:15.072 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695419236.317321_547.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:50:15.072 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1695419384.000133_332.1 target=, shard=3 master=1_1695419383.971498_240.1 target=1_1695416704.185257_96.1 2023-09-22T21:50:20.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:20.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:20.074 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:50:20.118 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:50:20.118 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:50:20.158 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:20.155+0000 7f9cec8f2a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:20.159 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:20.159+0000 7f9cec8f2a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:20.178 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:20Z"} 2023-09-22T21:50:20.178 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:20Z"} 2023-09-22T21:50:20.178 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695419236.317321_547.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:47:16.317321Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:50:20.179 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419236.317321_547.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:50:20.179 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695419236.317321_547.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:50:20.179 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1695419384.000133_332.1 target=, shard=3 master=1_1695419383.971498_240.1 target=1_1695416704.185257_96.1 2023-09-22T21:50:25.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:25.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:25.180 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:50:25.223 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:50:25.223 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:50:25.265 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:25.263+0000 7facb05aba40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:25.266 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:25.263+0000 7facb05aba40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:25.284 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:25Z"} 2023-09-22T21:50:25.284 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:25Z"} 2023-09-22T21:50:26.476 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695419236.317321_547.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:47:16.317321Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:50:26.476 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419236.317321_547.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:50:26.476 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695419236.317321_547.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:50:26.477 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1695419384.000133_332.1 target=, shard=3 master=1_1695419383.971498_240.1 target=1_1695416704.185257_96.1 2023-09-22T21:50:31.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:31.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:31.478 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:50:31.521 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:50:31.521 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:50:31.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:31.559+0000 7f96c8e61a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:31.561 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:31.559+0000 7f96c8e61a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:31.580 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:31Z"} 2023-09-22T21:50:31.580 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:31Z"} 2023-09-22T21:50:31.580 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695419236.317321_547.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:47:16.317321Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:50:31.580 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419236.317321_547.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:50:31.580 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695419236.317321_547.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:50:31.581 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1695419384.000133_332.1 target=, shard=3 master=1_1695419383.971498_240.1 target=1_1695416704.185257_96.1 2023-09-22T21:50:36.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:36.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:36.583 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:50:36.625 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:50:36.626 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:50:36.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:36.663+0000 7f76912b0a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:36.665 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:36.663+0000 7f76912b0a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:36.684 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:36Z"} 2023-09-22T21:50:36.684 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:36Z"} 2023-09-22T21:50:36.684 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695419236.317321_547.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:47:16.317321Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:50:36.685 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419236.317321_547.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:50:36.685 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695419236.317321_547.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:50:36.685 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1695419384.000133_332.1 target=, shard=3 master=1_1695419383.971498_240.1 target=1_1695416704.185257_96.1 2023-09-22T21:50:41.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:41.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:41.686 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:50:41.730 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:50:41.730 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:50:41.769 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:41.767+0000 7f7fc3ed7a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:41.770 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:41.767+0000 7f7fc3ed7a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:41.788 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:41Z"} 2023-09-22T21:50:41.788 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:41Z"} 2023-09-22T21:50:41.788 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695419236.317321_547.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:47:16.317321Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:50:41.788 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419236.317321_547.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:50:41.789 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695419236.317321_547.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:50:41.789 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1695419384.000133_332.1 target=, shard=3 master=1_1695419383.971498_240.1 target=1_1695416704.185257_96.1 2023-09-22T21:50:46.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:46.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:46.790 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:50:46.832 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:50:46.833 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:50:46.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:46.871+0000 7f29b62d4a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:47.026 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:46.871+0000 7f29b62d4a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:47.027 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:46Z"} 2023-09-22T21:50:47.027 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:46Z"} 2023-09-22T21:50:47.027 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695419236.317321_547.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:47:16.317321Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:50:47.027 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419236.317321_547.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:50:47.028 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695419236.317321_547.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:50:47.028 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1695419384.000133_332.1 target=, shard=3 master=1_1695419383.971498_240.1 target=1_1695416704.185257_96.1 2023-09-22T21:50:52.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:52.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:52.029 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:50:52.073 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:50:52.073 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:50:52.110 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:52.107+0000 7f6dcba96a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:52.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:52.111+0000 7f6dcba96a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:52.130 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:52Z"} 2023-09-22T21:50:52.130 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:52Z"} 2023-09-22T21:50:52.130 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695419236.317321_547.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:47:16.317321Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:50:52.131 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419236.317321_547.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:50:52.131 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695419236.317321_547.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:50:52.131 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1695419384.000133_332.1 target=, shard=3 master=1_1695419383.971498_240.1 target=1_1695416704.185257_96.1 2023-09-22T21:50:57.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:57.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:50:57.133 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:50:57.178 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:50:57.178 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:50:57.215 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:57.215+0000 7f054ae54a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:57.216 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:50:57.215+0000 7f054ae54a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:50:57.235 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:57Z"} 2023-09-22T21:50:57.235 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695416704.185257_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:05:04.185257Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:50:57Z"} 2023-09-22T21:50:57.235 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695419236.317321_547.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:47:16.317321Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695416704.185257_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:05:04.185257Z', 'realm_epoch': 2}}] 2023-09-22T21:50:57.235 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419236.317321_547.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:50:57.235 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '', 2: '1_1695419236.317321_547.1', 3: '1_1695416704.185257_96.1'} 2023-09-22T21:50:57.236 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1695419384.000133_332.1 target=, shard=3 master=1_1695419383.971498_240.1 target=1_1695416704.185257_96.1 2023-09-22T21:51:02.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:51:02.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:51:02.236 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:51:02.281 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:51:02.281 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:51:02.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:51:02.319+0000 7f44be06fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:51:02.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:51:02.319+0000 7f44be06fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:51:02.342 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:51:02Z"} 2023-09-22T21:51:02.342 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419236.317321_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:47:16.317321Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:51:02Z"} 2023-09-22T21:51:02.342 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695419236.317321_547.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:47:16.317321Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T21:51:02.342 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419236.317321_547.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:51:02.342 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419236.317321_547.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:51:02.342 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:51:02.343 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-ffzdqu-1 zones={a2, a1} 2023-09-22T21:51:02.343 DEBUG:boto:Method: POST 2023-09-22T21:51:02.343 DEBUG:boto:Path: / 2023-09-22T21:51:02.343 DEBUG:boto:Data: 2023-09-22T21:51:02.343 DEBUG:boto:Headers: {} 2023-09-22T21:51:02.343 DEBUG:boto:Host: smithi160.front.sepia.ceph.com 2023-09-22T21:51:02.343 DEBUG:boto:Port: 8001 2023-09-22T21:51:02.343 DEBUG:boto:Params: {'RoleName': 'roles-ffzdqu-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-09-22T21:51:02.344 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:51:02.344 DEBUG:boto:Token: None 2023-09-22T21:51:02.344 DEBUG:boto:CanonicalRequest: POST / host:smithi160.front.sepia.ceph.com:8001 x-amz-date:20230922T215102Z host;x-amz-date 29b837d17716a5fd3db49f9d0e9b4896eaad45d5684dd6e386672c3fa9a44a24 2023-09-22T21:51:02.344 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230922T215102Z 20230922/front/smithi160/aws4_request b78375f122bcf081995e3f0d3908297f3f57fbda1a8546e0339a37542a81b551 2023-09-22T21:51:02.344 DEBUG:boto:Signature: 7e7f9fd47b74df5349af2bb21bddd4abf1d69feaa527d8740d851cbe915643db 2023-09-22T21:51:02.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230922T215102Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RX9T9XC1DLDGURC46TDQ/20230922/front/smithi160/aws4_request,SignedHeaders=host;x-amz-date,Signature=7e7f9fd47b74df5349af2bb21bddd4abf1d69feaa527d8740d851cbe915643db', 'Host': 'smithi160.front.sepia.ceph.com:8001'} 2023-09-22T21:51:02.348 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004e658b8f9fc61cfd-00650e0c46-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '711'), ('Date', 'Fri, 22 Sep 2023 21:51:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:51:02.348 DEBUG:boto:b'tx000004e658b8f9fc61cfd-00650e0c46-1d9d-a2197fda3e-8291-42ad-9d50-2573d7498b0broles-ffzdqu-1/arn:aws:iam:::role/roles-ffzdqu-12023-09-22T21:51:00.176Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-09-22T21:51:02.349 DEBUG:boto:Method: POST 2023-09-22T21:51:02.349 DEBUG:boto:Path: / 2023-09-22T21:51:02.349 DEBUG:boto:Data: 2023-09-22T21:51:02.349 DEBUG:boto:Headers: {} 2023-09-22T21:51:02.349 DEBUG:boto:Host: smithi160.front.sepia.ceph.com 2023-09-22T21:51:02.349 DEBUG:boto:Port: 8000 2023-09-22T21:51:02.349 DEBUG:boto:Params: {'RoleName': 'roles-ffzdqu-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-09-22T21:51:02.350 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:51:02.350 DEBUG:boto:Token: None 2023-09-22T21:51:02.350 DEBUG:boto:CanonicalRequest: POST / host:smithi160.front.sepia.ceph.com:8000 x-amz-date:20230922T215102Z host;x-amz-date 29b837d17716a5fd3db49f9d0e9b4896eaad45d5684dd6e386672c3fa9a44a24 2023-09-22T21:51:02.350 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230922T215102Z 20230922/front/smithi160/aws4_request 903da6d84e65ae25a0d10a14928d8e24f8f03bb33a5791cda9a1eec04939f257 2023-09-22T21:51:02.350 DEBUG:boto:Signature: 5b52b8a451540da51b689bf6c1ccff6a599b9d94272906df524d0dcf656dd028 2023-09-22T21:51:02.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230922T215102Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RX9T9XC1DLDGURC46TDQ/20230922/front/smithi160/aws4_request,SignedHeaders=host;x-amz-date,Signature=5b52b8a451540da51b689bf6c1ccff6a599b9d94272906df524d0dcf656dd028', 'Host': 'smithi160.front.sepia.ceph.com:8000'} 2023-09-22T21:51:02.353 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000260906119cb75fde-00650e0c46-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '711'), ('Date', 'Fri, 22 Sep 2023 21:51:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:51:02.353 DEBUG:boto:b'tx00000260906119cb75fde-00650e0c46-165f-a1197fda3e-8291-42ad-9d50-2573d7498b0broles-ffzdqu-1/arn:aws:iam:::role/roles-ffzdqu-12023-09-22T21:49:43.964Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-09-22T21:51:02.354 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-ffzdqu-1 2023-09-22T21:51:02.354 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-ffzdqu-1 zones={a2, a1} 2023-09-22T21:51:02.354 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-ffzdqu-2 zones={a1, a2} 2023-09-22T21:51:02.354 DEBUG:boto:Method: POST 2023-09-22T21:51:02.355 DEBUG:boto:Path: / 2023-09-22T21:51:02.355 DEBUG:boto:Data: 2023-09-22T21:51:02.355 DEBUG:boto:Headers: {} 2023-09-22T21:51:02.355 DEBUG:boto:Host: smithi160.front.sepia.ceph.com 2023-09-22T21:51:02.355 DEBUG:boto:Port: 8000 2023-09-22T21:51:02.355 DEBUG:boto:Params: {'RoleName': 'roles-ffzdqu-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-09-22T21:51:02.355 DEBUG:boto:Token: None 2023-09-22T21:51:02.355 DEBUG:boto:CanonicalRequest: POST / host:smithi160.front.sepia.ceph.com:8000 x-amz-date:20230922T215102Z host;x-amz-date aba46c4d745923f8a5d140fe358572712df8023359a5fc1d2dfcb44bb3735e9f 2023-09-22T21:51:02.355 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230922T215102Z 20230922/front/smithi160/aws4_request 66dc1afe55e2672533331c740a286386d34071a76e37c9e7760e7e46ff3b0dc6 2023-09-22T21:51:02.356 DEBUG:boto:Signature: 2f26128f64567d4c014793f15775b11ff645691b5267cde83349acf02e274760 2023-09-22T21:51:02.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230922T215102Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RX9T9XC1DLDGURC46TDQ/20230922/front/smithi160/aws4_request,SignedHeaders=host;x-amz-date,Signature=2f26128f64567d4c014793f15775b11ff645691b5267cde83349acf02e274760', 'Host': 'smithi160.front.sepia.ceph.com:8000'} 2023-09-22T21:51:02.357 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003dbb4124aa778650-00650e0c46-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '711'), ('Date', 'Fri, 22 Sep 2023 21:51:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:51:02.357 DEBUG:boto:b'tx000003dbb4124aa778650-00650e0c46-165f-a109b48eb6-5f0b-4445-88a9-c807813157a4roles-ffzdqu-2/arn:aws:iam:::role/roles-ffzdqu-22023-09-22T21:49:43.992Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-09-22T21:51:02.358 DEBUG:boto:Method: POST 2023-09-22T21:51:02.358 DEBUG:boto:Path: / 2023-09-22T21:51:02.358 DEBUG:boto:Data: 2023-09-22T21:51:02.358 DEBUG:boto:Headers: {} 2023-09-22T21:51:02.358 DEBUG:boto:Host: smithi160.front.sepia.ceph.com 2023-09-22T21:51:02.358 DEBUG:boto:Port: 8001 2023-09-22T21:51:02.358 DEBUG:boto:Params: {'RoleName': 'roles-ffzdqu-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-09-22T21:51:02.359 DEBUG:boto:Token: None 2023-09-22T21:51:02.359 DEBUG:boto:CanonicalRequest: POST / host:smithi160.front.sepia.ceph.com:8001 x-amz-date:20230922T215102Z host;x-amz-date aba46c4d745923f8a5d140fe358572712df8023359a5fc1d2dfcb44bb3735e9f 2023-09-22T21:51:02.359 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230922T215102Z 20230922/front/smithi160/aws4_request bea1a45d0c79c6a2eb50230ff832588d226bedd05378d1fa4c27d8c0b2b98399 2023-09-22T21:51:02.359 DEBUG:boto:Signature: 71cac2dd9b2b42c38e42a3e2034e96917d2d5d2dca29b65ae106b76e7a8f96bf 2023-09-22T21:51:02.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230922T215102Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RX9T9XC1DLDGURC46TDQ/20230922/front/smithi160/aws4_request,SignedHeaders=host;x-amz-date,Signature=71cac2dd9b2b42c38e42a3e2034e96917d2d5d2dca29b65ae106b76e7a8f96bf', 'Host': 'smithi160.front.sepia.ceph.com:8001'} 2023-09-22T21:51:02.361 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006a9124039363e910-00650e0c46-1d9d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '711'), ('Date', 'Fri, 22 Sep 2023 21:51:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:51:02.361 DEBUG:boto:b'tx000006a9124039363e910-00650e0c46-1d9d-a209b48eb6-5f0b-4445-88a9-c807813157a4roles-ffzdqu-2/arn:aws:iam:::role/roles-ffzdqu-22023-09-22T21:49:43.992Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-09-22T21:51:02.362 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-ffzdqu-2 2023-09-22T21:51:02.362 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-ffzdqu-2 zones={a1, a2} 2023-09-22T21:51:02.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ok 2023-09-22T21:51:02.363 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:51:02.363 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:51:02.363 DEBUG:boto:Method: PUT 2023-09-22T21:51:02.363 DEBUG:boto:Path: /ffzdqu-60/ 2023-09-22T21:51:02.363 DEBUG:boto:Data: 2023-09-22T21:51:02.363 DEBUG:boto:Headers: {} 2023-09-22T21:51:02.363 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:51:02.363 DEBUG:boto:Port: 8000 2023-09-22T21:51:02.364 DEBUG:boto:Params: {} 2023-09-22T21:51:02.364 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:51:02.364 DEBUG:boto:Token: None 2023-09-22T21:51:02.364 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:51:02 GMT /ffzdqu-60/ 2023-09-22T21:51:02.364 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QqjVbcK5EcTQygcF+8VB8uytIXM= 2023-09-22T21:51:02.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:51:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QqjVbcK5EcTQygcF+8VB8uytIXM=', 'Content-Length': '0'} 2023-09-22T21:51:02.385 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002c9fcc342bb05419-00650e0c46-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:51:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:51:02.386 DEBUG:rgw_multi.tests:created bucket=ffzdqu-60 2023-09-22T21:51:02.386 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:51:02.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:51:02.386 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:51:02.420 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:51:02.420 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:51:02.457 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:51:02.476 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"1_1695419384.000133_332.1","last_update":"2023-09-22T21:49:44.000133Z"},{"marker":"1_1695419462.385536_558.1","last_update":"2023-09-22T21:51:02.385536Z"},{"marker":"1_1695419383.971498_240.1","last_update":"2023-09-22T21:49:43.971498Z"}] 2023-09-22T21:51:02.476 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419462.385536_558.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:51:02.476 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:51:02.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:51:02.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:51:02.476 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:51:02.560 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:51:02.560 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:51:02.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:51:02.599+0000 7f2f2b166a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:51:02.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:51:02.603+0000 7f2f2b166a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:51:02.622 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419462.377302_555.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:51:02.377302Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:51:02Z"} 2023-09-22T21:51:02.623 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419462.377302_555.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:51:02.377302Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:51:02Z"} 2023-09-22T21:51:02.623 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695419462.377302_555.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:51:02.377302Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T21:51:02.623 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419462.385536_558.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:51:02.623 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419462.377302_555.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:51:02.623 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695419462.385536_558.1 target=1_1695419462.377302_555.1 2023-09-22T21:51:07.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:51:07.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:51:07.625 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:51:07.669 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:51:07.669 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:51:07.707 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:51:07.707+0000 7fd747520a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:51:07.708 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:51:07.707+0000 7fd747520a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:51:07.727 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419462.385536_558.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:51:02.385536Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:51:07Z"} 2023-09-22T21:51:07.727 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419462.385536_558.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:51:02.385536Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:51:07Z"} 2023-09-22T21:51:07.727 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695419462.385536_558.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:51:02.385536Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T21:51:07.727 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419462.385536_558.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:51:07.728 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419462.385536_558.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:51:07.728 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:51:07.728 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-09-22T21:51:07.728 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T21:51:07.772 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:51:07.771+0000 7f07d97fe640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 77633) UID: 0 2023-09-22T21:51:07.772 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:51:07.771+0000 7f07dd045a40 -1 shutting down 2023-09-22T21:51:12.721 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-09-22T21:51:13.829 INFO:tasks.rgw.c1.client.1:Stopped 2023-09-22T21:51:13.830 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:51:13.830 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:51:13.830 DEBUG:boto:Method: HEAD 2023-09-22T21:51:13.830 DEBUG:boto:Path: /ffzdqu-60/ 2023-09-22T21:51:13.831 DEBUG:boto:Data: 2023-09-22T21:51:13.831 DEBUG:boto:Headers: {} 2023-09-22T21:51:13.831 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:51:13.831 DEBUG:boto:Port: 8000 2023-09-22T21:51:13.831 DEBUG:boto:Params: {} 2023-09-22T21:51:13.831 DEBUG:boto:Token: None 2023-09-22T21:51:13.832 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:51:13 GMT /ffzdqu-60/ 2023-09-22T21:51:13.832 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GS2n+xSBvmcDQWgWdvuwS7ACBaQ= 2023-09-22T21:51:13.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:51:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GS2n+xSBvmcDQWgWdvuwS7ACBaQ=', 'Content-Length': '0'} 2023-09-22T21:51:13.837 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c1caa73eb835213-00650e0c51-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:51:13.837 DEBUG:boto:path=/ffzdqu-60/obj11 2023-09-22T21:51:13.837 DEBUG:boto:auth_path=/ffzdqu-60/obj11 2023-09-22T21:51:13.837 DEBUG:boto:Method: PUT 2023-09-22T21:51:13.838 DEBUG:boto:Path: /ffzdqu-60/obj11 2023-09-22T21:51:13.838 DEBUG:boto:Data: 2023-09-22T21:51:13.838 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:51:13.838 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:51:13.838 DEBUG:boto:Port: 8000 2023-09-22T21:51:13.838 DEBUG:boto:Params: {} 2023-09-22T21:51:13.839 DEBUG:boto:Token: None 2023-09-22T21:51:13.839 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:51:13 GMT /ffzdqu-60/obj11 2023-09-22T21:51:13.839 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8xvtJvdQLcHMmO0xurJ1tJI6guI= 2023-09-22T21:51:13.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:51:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8xvtJvdQLcHMmO0xurJ1tJI6guI='} 2023-09-22T21:51:13.860 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a72365cb6d5bb12-00650e0c51-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:51:13.861 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:51:13.861 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:51:13.861 DEBUG:boto:Method: HEAD 2023-09-22T21:51:13.861 DEBUG:boto:Path: /ffzdqu-60/ 2023-09-22T21:51:13.861 DEBUG:boto:Data: 2023-09-22T21:51:13.862 DEBUG:boto:Headers: {} 2023-09-22T21:51:13.862 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:51:13.862 DEBUG:boto:Port: 8000 2023-09-22T21:51:13.862 DEBUG:boto:Params: {} 2023-09-22T21:51:13.862 DEBUG:boto:Token: None 2023-09-22T21:51:13.862 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:51:13 GMT /ffzdqu-60/ 2023-09-22T21:51:13.863 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GS2n+xSBvmcDQWgWdvuwS7ACBaQ= 2023-09-22T21:51:13.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:51:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GS2n+xSBvmcDQWgWdvuwS7ACBaQ=', 'Content-Length': '0'} 2023-09-22T21:51:13.868 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0d89b5b561f2b7a-00650e0c51-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:51:13.868 DEBUG:boto:path=/ffzdqu-60/obj22 2023-09-22T21:51:13.869 DEBUG:boto:auth_path=/ffzdqu-60/obj22 2023-09-22T21:51:13.869 DEBUG:boto:Method: PUT 2023-09-22T21:51:13.869 DEBUG:boto:Path: /ffzdqu-60/obj22 2023-09-22T21:51:13.869 DEBUG:boto:Data: 2023-09-22T21:51:13.869 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:51:13.869 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:51:13.869 DEBUG:boto:Port: 8000 2023-09-22T21:51:13.870 DEBUG:boto:Params: {} 2023-09-22T21:51:13.870 DEBUG:boto:Token: None 2023-09-22T21:51:13.870 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:51:13 GMT /ffzdqu-60/obj22 2023-09-22T21:51:13.870 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wRmrH6meFL4tQ7nQcX9QQWzBxNA= 2023-09-22T21:51:13.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:51:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wRmrH6meFL4tQ7nQcX9QQWzBxNA='} 2023-09-22T21:51:13.915 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d762343d0321be0c-00650e0c51-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:51:13.915 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:51:13.915 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:51:13.915 DEBUG:boto:Method: HEAD 2023-09-22T21:51:13.915 DEBUG:boto:Path: /ffzdqu-60/ 2023-09-22T21:51:13.916 DEBUG:boto:Data: 2023-09-22T21:51:13.916 DEBUG:boto:Headers: {} 2023-09-22T21:51:13.916 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:51:13.916 DEBUG:boto:Port: 8000 2023-09-22T21:51:13.916 DEBUG:boto:Params: {} 2023-09-22T21:51:13.916 DEBUG:boto:Token: None 2023-09-22T21:51:13.917 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:51:13 GMT /ffzdqu-60/ 2023-09-22T21:51:13.917 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GS2n+xSBvmcDQWgWdvuwS7ACBaQ= 2023-09-22T21:51:13.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:51:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GS2n+xSBvmcDQWgWdvuwS7ACBaQ=', 'Content-Length': '0'} 2023-09-22T21:51:13.921 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ab161b37c47829a-00650e0c51-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:51:13.921 DEBUG:boto:path=/ffzdqu-60/obj33 2023-09-22T21:51:13.922 DEBUG:boto:auth_path=/ffzdqu-60/obj33 2023-09-22T21:51:13.922 DEBUG:boto:Method: PUT 2023-09-22T21:51:13.922 DEBUG:boto:Path: /ffzdqu-60/obj33 2023-09-22T21:51:13.922 DEBUG:boto:Data: 2023-09-22T21:51:13.922 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:51:13.922 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:51:13.923 DEBUG:boto:Port: 8000 2023-09-22T21:51:13.923 DEBUG:boto:Params: {} 2023-09-22T21:51:13.923 DEBUG:boto:Token: None 2023-09-22T21:51:13.923 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:51:13 GMT /ffzdqu-60/obj33 2023-09-22T21:51:13.923 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BpSkQFHLL6lRDmowc9ZtSaA5Nq0= 2023-09-22T21:51:13.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:51:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BpSkQFHLL6lRDmowc9ZtSaA5Nq0='} 2023-09-22T21:51:13.936 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000071915457322ca2e1-00650e0c51-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:51:13.936 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:51:13.937 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:51:13.937 DEBUG:boto:Method: HEAD 2023-09-22T21:51:13.937 DEBUG:boto:Path: /ffzdqu-60/ 2023-09-22T21:51:13.937 DEBUG:boto:Data: 2023-09-22T21:51:13.937 DEBUG:boto:Headers: {} 2023-09-22T21:51:13.937 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:51:13.937 DEBUG:boto:Port: 8000 2023-09-22T21:51:13.938 DEBUG:boto:Params: {} 2023-09-22T21:51:13.938 DEBUG:boto:Token: None 2023-09-22T21:51:13.938 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:51:13 GMT /ffzdqu-60/ 2023-09-22T21:51:13.938 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GS2n+xSBvmcDQWgWdvuwS7ACBaQ= 2023-09-22T21:51:13.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:51:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GS2n+xSBvmcDQWgWdvuwS7ACBaQ=', 'Content-Length': '0'} 2023-09-22T21:51:13.943 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b49c7c2448c55680-00650e0c51-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:51:13.943 DEBUG:boto:path=/ffzdqu-60/obj44 2023-09-22T21:51:13.943 DEBUG:boto:auth_path=/ffzdqu-60/obj44 2023-09-22T21:51:13.943 DEBUG:boto:Method: PUT 2023-09-22T21:51:13.944 DEBUG:boto:Path: /ffzdqu-60/obj44 2023-09-22T21:51:13.944 DEBUG:boto:Data: 2023-09-22T21:51:13.944 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:51:13.944 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:51:13.944 DEBUG:boto:Port: 8000 2023-09-22T21:51:13.944 DEBUG:boto:Params: {} 2023-09-22T21:51:13.944 DEBUG:boto:Token: None 2023-09-22T21:51:13.945 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:51:13 GMT /ffzdqu-60/obj44 2023-09-22T21:51:13.945 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8DIe0E+9H2sbuhy0OmAEOJ4z02w= 2023-09-22T21:51:13.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:51:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8DIe0E+9H2sbuhy0OmAEOJ4z02w='} 2023-09-22T21:51:13.991 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f852265f5165c6f4-00650e0c51-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:51:13.991 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:51:13.991 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:51:13.991 DEBUG:boto:Method: HEAD 2023-09-22T21:51:13.992 DEBUG:boto:Path: /ffzdqu-60/ 2023-09-22T21:51:13.992 DEBUG:boto:Data: 2023-09-22T21:51:13.992 DEBUG:boto:Headers: {} 2023-09-22T21:51:13.992 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:51:13.992 DEBUG:boto:Port: 8000 2023-09-22T21:51:13.992 DEBUG:boto:Params: {} 2023-09-22T21:51:13.992 DEBUG:boto:Token: None 2023-09-22T21:51:13.993 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:51:13 GMT /ffzdqu-60/ 2023-09-22T21:51:13.993 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GS2n+xSBvmcDQWgWdvuwS7ACBaQ= 2023-09-22T21:51:13.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:51:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GS2n+xSBvmcDQWgWdvuwS7ACBaQ=', 'Content-Length': '0'} 2023-09-22T21:51:13.997 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca83431db54cdf3d-00650e0c51-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:51:13.998 DEBUG:boto:path=/ffzdqu-60/obj55 2023-09-22T21:51:13.998 DEBUG:boto:auth_path=/ffzdqu-60/obj55 2023-09-22T21:51:13.998 DEBUG:boto:Method: PUT 2023-09-22T21:51:13.998 DEBUG:boto:Path: /ffzdqu-60/obj55 2023-09-22T21:51:13.999 DEBUG:boto:Data: 2023-09-22T21:51:13.999 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:51:13.999 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:51:13.999 DEBUG:boto:Port: 8000 2023-09-22T21:51:13.999 DEBUG:boto:Params: {} 2023-09-22T21:51:13.999 DEBUG:boto:Token: None 2023-09-22T21:51:14.000 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:51:14 GMT /ffzdqu-60/obj55 2023-09-22T21:51:14.000 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:39QtUt45Ndid93CRmdy2ehZB0gQ= 2023-09-22T21:51:14.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:51:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:39QtUt45Ndid93CRmdy2ehZB0gQ='} 2023-09-22T21:51:14.009 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6577ac1791e4f40-00650e0c51-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:51:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:51:14.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:51:14.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:51:14.009 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:51:14.054 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:51:14.054 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:51:14.093 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:51:14.091+0000 7f863c195a40 5 RGW-SYNC:data: waiting on data sync status lease 2023-09-22T21:51:14.094 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:51:14.091+0000 7f863c195a40 5 RGW-SYNC:data: acquired data sync status lease 2023-09-22T21:51:14.120 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-22T21:51:14.121 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-22T21:51:14.165 INFO:tasks.rgw.c1.client.1:Started 2023-09-22T21:51:14.165 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T21:51:14.175 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T21:51:14.176 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:51:14.176 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:51:14.176 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T21:51:14.176 INFO:teuthology.orchestra.run.smithi160.stderr:curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T21:51:15.178 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T21:51:15.189 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:51:15.189 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:51:15.190 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 135k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-22T21:51:15.191 INFO:teuthology.orchestra.run.smithi160.stdout:anonymous 2023-09-22T21:51:15.191 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-60 2023-09-22T21:51:15.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:51:15.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:51:15.192 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-60 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:51:15.289 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:51:15.289 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:51:15.334 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:51:15.331+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:51:17.090 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:51:17.087+0000 7fda862b5640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:51:18.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-09-22T21:51:20.335 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:51:20.330+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:51:25.336 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:51:25.334+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:51:30.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:51:30.334+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:51:35.338 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:51:35.334+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:51:40.339 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:51:40.334+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:51:45.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:51:45.338+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:51:50.341 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:51:50.338+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:51:55.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:51:55.338+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:52:00.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:52:00.338+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:52:05.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:52:05.342+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:52:10.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:52:10.342+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:52:15.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:52:15.342+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:52:20.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:52:20.342+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:52:25.348 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:52:25.346+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:52:30.349 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:52:30.346+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:52:35.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:52:35.345+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:52:40.351 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:52:40.345+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:52:45.352 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:52:45.349+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:52:50.353 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:52:50.349+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:52:55.354 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:52:55.353+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:53:00.355 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:53:00.353+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:53:05.356 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:53:05.357+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:53:10.357 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:53:10.357+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:53:15.358 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:53:15.357+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:53:20.359 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:53:20.357+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:53:25.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:53:25.361+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:53:30.361 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:53:30.361+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:53:35.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:53:35.361+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:53:40.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:53:40.365+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:53:45.365 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:53:45.365+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:53:50.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:53:50.365+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:53:55.367 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:53:55.364+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:54:00.368 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:54:00.368+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:54:05.369 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:54:05.368+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:54:10.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:54:10.368+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:54:15.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:54:15.372+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:54:20.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:54:20.372+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:54:25.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:54:25.372+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:54:30.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:54:30.376+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:54:34.938 INFO:tasks.rgw.c1.client.0.smithi160.stdout:2023-09-22T21:54:34.936+0000 7fa9c1675640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:54:35.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:54:35.376+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:54:40.378 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:54:40.376+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:54:45.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:54:45.380+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:54:50.381 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:54:50.380+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:54:55.382 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:54:55.380+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:55:00.383 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:55:00.380+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:55:05.384 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:55:05.384+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:55:10.385 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:55:10.383+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:55:15.386 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:55:15.383+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:55:20.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:55:20.383+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:55:25.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:55:25.387+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:55:30.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:55:30.387+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:55:35.390 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:55:35.387+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:55:40.391 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:55:40.391+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:55:45.392 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:55:45.391+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:55:50.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:55:50.391+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:55:55.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:55:55.391+0000 7fda1763ea40 1 waiting to reach incremental sync.. 2023-09-22T21:56:00.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:56:00.395+0000 7fda1763ea40 1 bucket sync caught up with source: 2023-09-22T21:56:00.399 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.67.6, 00000000002.69.6, 00000000002.84.6] 2023-09-22T21:56:00.399 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.67.6, 00000000002.69.6, 00000000002.84.6] 2023-09-22T21:56:00.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:56:00.395+0000 7fda1763ea40 0 bucket checkpoint complete 2023-09-22T21:56:00.431 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-60 2023-09-22T21:56:00.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:56:00.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:56:00.432 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-60 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:56:00.481 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:56:00.481 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:56:00.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:56:00.523+0000 7f0bbda4fa40 1 bucket sync caught up with source: 2023-09-22T21:56:00.525 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.278.6, 00000000002.340.6, 00000000002.361.6] 2023-09-22T21:56:00.526 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.278.6, 00000000002.340.6, 00000000002.361.6] 2023-09-22T21:56:00.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:56:00.523+0000 7f0bbda4fa40 0 bucket checkpoint complete 2023-09-22T21:56:00.543 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-60 zones={a2, a1} 2023-09-22T21:56:00.543 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:00.544 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:00.544 DEBUG:boto:Method: HEAD 2023-09-22T21:56:00.544 DEBUG:boto:Path: /ffzdqu-60/ 2023-09-22T21:56:00.544 DEBUG:boto:Data: 2023-09-22T21:56:00.544 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.544 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:00.544 DEBUG:boto:Port: 8001 2023-09-22T21:56:00.544 DEBUG:boto:Params: {} 2023-09-22T21:56:00.545 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:56:00.545 DEBUG:boto:Token: None 2023-09-22T21:56:00.545 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/ 2023-09-22T21:56:00.545 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LGbrq50QoWERjkSU6oV1s7YmZ14= 2023-09-22T21:56:00.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LGbrq50QoWERjkSU6oV1s7YmZ14=', 'Content-Length': '0'} 2023-09-22T21:56:00.551 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb5b7ca3193cd132-00650e0d70-1e5d-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.551 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:00.552 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:00.552 DEBUG:boto:Method: HEAD 2023-09-22T21:56:00.552 DEBUG:boto:Path: /ffzdqu-60/ 2023-09-22T21:56:00.552 DEBUG:boto:Data: 2023-09-22T21:56:00.552 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.552 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:00.552 DEBUG:boto:Port: 8000 2023-09-22T21:56:00.552 DEBUG:boto:Params: {} 2023-09-22T21:56:00.553 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:56:00.553 DEBUG:boto:Token: None 2023-09-22T21:56:00.553 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/ 2023-09-22T21:56:00.553 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LGbrq50QoWERjkSU6oV1s7YmZ14= 2023-09-22T21:56:00.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LGbrq50QoWERjkSU6oV1s7YmZ14=', 'Content-Length': '0'} 2023-09-22T21:56:00.556 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091cb12a18f3e6845-00650e0d70-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.557 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:56:00.557 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:00.557 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:00.557 DEBUG:boto:path=/ffzdqu-60/?versions&max-keys=999 2023-09-22T21:56:00.557 DEBUG:boto:auth_path=/ffzdqu-60/?versions&max-keys=999 2023-09-22T21:56:00.557 DEBUG:boto:Method: GET 2023-09-22T21:56:00.557 DEBUG:boto:Path: /ffzdqu-60/?versions&max-keys=999 2023-09-22T21:56:00.557 DEBUG:boto:Data: 2023-09-22T21:56:00.558 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.558 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:00.558 DEBUG:boto:Port: 8001 2023-09-22T21:56:00.558 DEBUG:boto:Params: {} 2023-09-22T21:56:00.558 DEBUG:boto:Token: None 2023-09-22T21:56:00.558 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/?versions 2023-09-22T21:56:00.558 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rEMDq9K3EPV1r1AsRtuVFpi2cGE= 2023-09-22T21:56:00.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rEMDq9K3EPV1r1AsRtuVFpi2cGE=', 'Content-Length': '0'} 2023-09-22T21:56:00.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a04f80d185e922af-00650e0d70-1e5d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.561 DEBUG:boto:b'ffzdqu-60999falseobj11nulltrue2023-09-22T21:51:13.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:51:13.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:51:13.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:51:13.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:51:14.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:00.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T21:56:00.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T21:56:00.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T21:56:00.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T21:56:00.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T21:56:00.562 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:56:00.562 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:00.562 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:00.563 DEBUG:boto:path=/ffzdqu-60/?versions&max-keys=999 2023-09-22T21:56:00.563 DEBUG:boto:auth_path=/ffzdqu-60/?versions&max-keys=999 2023-09-22T21:56:00.563 DEBUG:boto:Method: GET 2023-09-22T21:56:00.563 DEBUG:boto:Path: /ffzdqu-60/?versions&max-keys=999 2023-09-22T21:56:00.563 DEBUG:boto:Data: 2023-09-22T21:56:00.563 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.563 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:00.563 DEBUG:boto:Port: 8000 2023-09-22T21:56:00.563 DEBUG:boto:Params: {} 2023-09-22T21:56:00.564 DEBUG:boto:Token: None 2023-09-22T21:56:00.564 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/?versions 2023-09-22T21:56:00.564 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rEMDq9K3EPV1r1AsRtuVFpi2cGE= 2023-09-22T21:56:00.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rEMDq9K3EPV1r1AsRtuVFpi2cGE=', 'Content-Length': '0'} 2023-09-22T21:56:00.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000826e1a418b456cc3-00650e0d70-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.566 DEBUG:boto:b'ffzdqu-60999falseobj11nulltrue2023-09-22T21:51:13.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:51:13.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:51:13.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:51:13.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:51:14.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:00.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T21:56:00.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T21:56:00.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T21:56:00.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T21:56:00.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T21:56:00.568 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:00.568 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:00.568 DEBUG:boto:path=/ffzdqu-60/?versions&max-keys=999 2023-09-22T21:56:00.568 DEBUG:boto:auth_path=/ffzdqu-60/?versions&max-keys=999 2023-09-22T21:56:00.568 DEBUG:boto:Method: GET 2023-09-22T21:56:00.568 DEBUG:boto:Path: /ffzdqu-60/?versions&max-keys=999 2023-09-22T21:56:00.568 DEBUG:boto:Data: 2023-09-22T21:56:00.568 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.568 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:00.569 DEBUG:boto:Port: 8001 2023-09-22T21:56:00.569 DEBUG:boto:Params: {} 2023-09-22T21:56:00.569 DEBUG:boto:Token: None 2023-09-22T21:56:00.569 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/?versions 2023-09-22T21:56:00.569 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rEMDq9K3EPV1r1AsRtuVFpi2cGE= 2023-09-22T21:56:00.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rEMDq9K3EPV1r1AsRtuVFpi2cGE=', 'Content-Length': '0'} 2023-09-22T21:56:00.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036872bc6ee7a37c0-00650e0d70-1e5d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.572 DEBUG:boto:b'ffzdqu-60999falseobj11nulltrue2023-09-22T21:51:13.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:51:13.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:51:13.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:51:13.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:51:14.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:00.572 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:00.572 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:00.573 DEBUG:boto:path=/ffzdqu-60/?versions&max-keys=999 2023-09-22T21:56:00.573 DEBUG:boto:auth_path=/ffzdqu-60/?versions&max-keys=999 2023-09-22T21:56:00.573 DEBUG:boto:Method: GET 2023-09-22T21:56:00.573 DEBUG:boto:Path: /ffzdqu-60/?versions&max-keys=999 2023-09-22T21:56:00.573 DEBUG:boto:Data: 2023-09-22T21:56:00.573 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.573 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:00.573 DEBUG:boto:Port: 8000 2023-09-22T21:56:00.573 DEBUG:boto:Params: {} 2023-09-22T21:56:00.574 DEBUG:boto:Token: None 2023-09-22T21:56:00.574 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/?versions 2023-09-22T21:56:00.574 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rEMDq9K3EPV1r1AsRtuVFpi2cGE= 2023-09-22T21:56:00.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rEMDq9K3EPV1r1AsRtuVFpi2cGE=', 'Content-Length': '0'} 2023-09-22T21:56:00.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e423a4422f645fd-00650e0d70-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.577 DEBUG:boto:b'ffzdqu-60999falseobj11nulltrue2023-09-22T21:51:13.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:51:13.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:51:13.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:51:13.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:51:14.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:00.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:56:00.577 DEBUG:boto:path=/ffzdqu-60/obj11 2023-09-22T21:56:00.578 DEBUG:boto:auth_path=/ffzdqu-60/obj11 2023-09-22T21:56:00.578 DEBUG:boto:path=/ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.578 DEBUG:boto:auth_path=/ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.578 DEBUG:boto:Method: GET 2023-09-22T21:56:00.578 DEBUG:boto:Path: /ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.578 DEBUG:boto:Data: 2023-09-22T21:56:00.578 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.578 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:00.578 DEBUG:boto:Port: 8001 2023-09-22T21:56:00.578 DEBUG:boto:Params: {} 2023-09-22T21:56:00.579 DEBUG:boto:Token: None 2023-09-22T21:56:00.579 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.579 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FNYCsHb0wN8xErCkIEpttJaFOaE= 2023-09-22T21:56:00.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FNYCsHb0wN8xErCkIEpttJaFOaE=', 'Content-Length': '0'} 2023-09-22T21:56:00.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007de47c04cf8431d2-00650e0d70-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.582 DEBUG:boto:path=/ffzdqu-60/obj11 2023-09-22T21:56:00.582 DEBUG:boto:auth_path=/ffzdqu-60/obj11 2023-09-22T21:56:00.582 DEBUG:boto:path=/ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.582 DEBUG:boto:auth_path=/ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.582 DEBUG:boto:Method: GET 2023-09-22T21:56:00.582 DEBUG:boto:Path: /ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.582 DEBUG:boto:Data: 2023-09-22T21:56:00.583 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.583 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:00.583 DEBUG:boto:Port: 8000 2023-09-22T21:56:00.583 DEBUG:boto:Params: {} 2023-09-22T21:56:00.583 DEBUG:boto:Token: None 2023-09-22T21:56:00.583 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.583 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FNYCsHb0wN8xErCkIEpttJaFOaE= 2023-09-22T21:56:00.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FNYCsHb0wN8xErCkIEpttJaFOaE=', 'Content-Length': '0'} 2023-09-22T21:56:00.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009af77e97f154ca6f-00650e0d70-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.585 DEBUG:boto:path=/ffzdqu-60/obj11 2023-09-22T21:56:00.585 DEBUG:boto:auth_path=/ffzdqu-60/obj11 2023-09-22T21:56:00.585 DEBUG:boto:path=/ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.586 DEBUG:boto:auth_path=/ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.586 DEBUG:boto:Method: HEAD 2023-09-22T21:56:00.586 DEBUG:boto:Path: /ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.586 DEBUG:boto:Data: 2023-09-22T21:56:00.586 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.586 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:00.586 DEBUG:boto:Port: 8001 2023-09-22T21:56:00.586 DEBUG:boto:Params: {} 2023-09-22T21:56:00.586 DEBUG:boto:Token: None 2023-09-22T21:56:00.587 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.587 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9CF4xjX8n5XDNEqnMldl+zw5iFA= 2023-09-22T21:56:00.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9CF4xjX8n5XDNEqnMldl+zw5iFA=', 'Content-Length': '0'} 2023-09-22T21:56:00.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e31d2e7a953bda01-00650e0d70-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.589 DEBUG:boto:path=/ffzdqu-60/obj11 2023-09-22T21:56:00.589 DEBUG:boto:auth_path=/ffzdqu-60/obj11 2023-09-22T21:56:00.589 DEBUG:boto:path=/ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.589 DEBUG:boto:auth_path=/ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.589 DEBUG:boto:Method: HEAD 2023-09-22T21:56:00.589 DEBUG:boto:Path: /ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.589 DEBUG:boto:Data: 2023-09-22T21:56:00.589 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.590 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:00.590 DEBUG:boto:Port: 8000 2023-09-22T21:56:00.590 DEBUG:boto:Params: {} 2023-09-22T21:56:00.590 DEBUG:boto:Token: None 2023-09-22T21:56:00.590 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.590 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9CF4xjX8n5XDNEqnMldl+zw5iFA= 2023-09-22T21:56:00.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9CF4xjX8n5XDNEqnMldl+zw5iFA=', 'Content-Length': '0'} 2023-09-22T21:56:00.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001741347c96b53afd-00650e0d70-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:56:00.592 DEBUG:boto:path=/ffzdqu-60/obj11 2023-09-22T21:56:00.592 DEBUG:boto:auth_path=/ffzdqu-60/obj11 2023-09-22T21:56:00.593 DEBUG:boto:path=/ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.593 DEBUG:boto:auth_path=/ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.593 DEBUG:boto:Method: GET 2023-09-22T21:56:00.593 DEBUG:boto:Path: /ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.593 DEBUG:boto:Data: 2023-09-22T21:56:00.593 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.593 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:00.593 DEBUG:boto:Port: 8001 2023-09-22T21:56:00.593 DEBUG:boto:Params: {} 2023-09-22T21:56:00.593 DEBUG:boto:Token: None 2023-09-22T21:56:00.594 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.594 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FNYCsHb0wN8xErCkIEpttJaFOaE= 2023-09-22T21:56:00.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FNYCsHb0wN8xErCkIEpttJaFOaE=', 'Content-Length': '0'} 2023-09-22T21:56:00.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ab863a6944b5eaf-00650e0d70-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.596 DEBUG:boto:path=/ffzdqu-60/obj11 2023-09-22T21:56:00.596 DEBUG:boto:auth_path=/ffzdqu-60/obj11 2023-09-22T21:56:00.596 DEBUG:boto:path=/ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.596 DEBUG:boto:auth_path=/ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.596 DEBUG:boto:Method: GET 2023-09-22T21:56:00.596 DEBUG:boto:Path: /ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.596 DEBUG:boto:Data: 2023-09-22T21:56:00.596 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.597 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:00.597 DEBUG:boto:Port: 8000 2023-09-22T21:56:00.597 DEBUG:boto:Params: {} 2023-09-22T21:56:00.597 DEBUG:boto:Token: None 2023-09-22T21:56:00.597 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj11?versionId=null 2023-09-22T21:56:00.597 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FNYCsHb0wN8xErCkIEpttJaFOaE= 2023-09-22T21:56:00.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FNYCsHb0wN8xErCkIEpttJaFOaE=', 'Content-Length': '0'} 2023-09-22T21:56:00.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040a2d63602b0d681-00650e0d70-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.599 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:00.599 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:00.599 DEBUG:boto:path=/ffzdqu-60/?prefix=obj11 2023-09-22T21:56:00.599 DEBUG:boto:auth_path=/ffzdqu-60/?prefix=obj11 2023-09-22T21:56:00.600 DEBUG:boto:Method: GET 2023-09-22T21:56:00.600 DEBUG:boto:Path: /ffzdqu-60/?prefix=obj11 2023-09-22T21:56:00.600 DEBUG:boto:Data: 2023-09-22T21:56:00.600 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.600 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:00.600 DEBUG:boto:Port: 8001 2023-09-22T21:56:00.600 DEBUG:boto:Params: {} 2023-09-22T21:56:00.600 DEBUG:boto:Token: None 2023-09-22T21:56:00.600 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/ 2023-09-22T21:56:00.601 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rsFaj0CDtNrDG0XtSB+wFeEmLWo= 2023-09-22T21:56:00.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rsFaj0CDtNrDG0XtSB+wFeEmLWo=', 'Content-Length': '0'} 2023-09-22T21:56:00.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065acb071c236c5d7-00650e0d70-1e5d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.603 DEBUG:boto:b'ffzdqu-60obj111000falseobj112023-09-22T21:51:13.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:00.604 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:00.604 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:00.604 DEBUG:boto:path=/ffzdqu-60/?prefix=obj11 2023-09-22T21:56:00.604 DEBUG:boto:auth_path=/ffzdqu-60/?prefix=obj11 2023-09-22T21:56:00.604 DEBUG:boto:Method: GET 2023-09-22T21:56:00.604 DEBUG:boto:Path: /ffzdqu-60/?prefix=obj11 2023-09-22T21:56:00.604 DEBUG:boto:Data: 2023-09-22T21:56:00.604 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.604 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:00.605 DEBUG:boto:Port: 8000 2023-09-22T21:56:00.605 DEBUG:boto:Params: {} 2023-09-22T21:56:00.605 DEBUG:boto:Token: None 2023-09-22T21:56:00.605 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/ 2023-09-22T21:56:00.605 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rsFaj0CDtNrDG0XtSB+wFeEmLWo= 2023-09-22T21:56:00.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rsFaj0CDtNrDG0XtSB+wFeEmLWo=', 'Content-Length': '0'} 2023-09-22T21:56:00.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000976aaaaeda2f4a71-00650e0d70-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.608 DEBUG:boto:b'ffzdqu-60obj111000falseobj112023-09-22T21:51:13.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:00.608 DEBUG:boto:path=/ffzdqu-60/obj11 2023-09-22T21:56:00.609 DEBUG:boto:auth_path=/ffzdqu-60/obj11 2023-09-22T21:56:00.609 DEBUG:boto:Method: HEAD 2023-09-22T21:56:00.609 DEBUG:boto:Path: /ffzdqu-60/obj11 2023-09-22T21:56:00.609 DEBUG:boto:Data: 2023-09-22T21:56:00.609 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.609 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:00.609 DEBUG:boto:Port: 8001 2023-09-22T21:56:00.609 DEBUG:boto:Params: {} 2023-09-22T21:56:00.609 DEBUG:boto:Token: None 2023-09-22T21:56:00.610 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj11 2023-09-22T21:56:00.610 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OGgdhVLQgHGTLg128rq2+vVN90Y= 2023-09-22T21:56:00.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OGgdhVLQgHGTLg128rq2+vVN90Y=', 'Content-Length': '0'} 2023-09-22T21:56:00.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000883658d8af2ecf57-00650e0d70-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.611 DEBUG:boto:path=/ffzdqu-60/obj11 2023-09-22T21:56:00.612 DEBUG:boto:auth_path=/ffzdqu-60/obj11 2023-09-22T21:56:00.612 DEBUG:boto:Method: HEAD 2023-09-22T21:56:00.612 DEBUG:boto:Path: /ffzdqu-60/obj11 2023-09-22T21:56:00.612 DEBUG:boto:Data: 2023-09-22T21:56:00.612 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.612 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:00.612 DEBUG:boto:Port: 8000 2023-09-22T21:56:00.612 DEBUG:boto:Params: {} 2023-09-22T21:56:00.613 DEBUG:boto:Token: None 2023-09-22T21:56:00.613 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj11 2023-09-22T21:56:00.613 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OGgdhVLQgHGTLg128rq2+vVN90Y= 2023-09-22T21:56:00.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OGgdhVLQgHGTLg128rq2+vVN90Y=', 'Content-Length': '0'} 2023-09-22T21:56:00.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be2780b3ae4a2045-00650e0d70-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:56:00.615 DEBUG:boto:path=/ffzdqu-60/obj11 2023-09-22T21:56:00.615 DEBUG:boto:auth_path=/ffzdqu-60/obj11 2023-09-22T21:56:00.615 DEBUG:boto:Method: GET 2023-09-22T21:56:00.615 DEBUG:boto:Path: /ffzdqu-60/obj11 2023-09-22T21:56:00.615 DEBUG:boto:Data: 2023-09-22T21:56:00.615 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.615 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:00.615 DEBUG:boto:Port: 8001 2023-09-22T21:56:00.616 DEBUG:boto:Params: {} 2023-09-22T21:56:00.616 DEBUG:boto:Token: None 2023-09-22T21:56:00.616 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj11 2023-09-22T21:56:00.616 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3DIuOHFjm+GJ9KsFgyrYS0+R8XE= 2023-09-22T21:56:00.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3DIuOHFjm+GJ9KsFgyrYS0+R8XE=', 'Content-Length': '0'} 2023-09-22T21:56:00.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffada62a443da02e-00650e0d70-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.658 DEBUG:boto:path=/ffzdqu-60/obj11 2023-09-22T21:56:00.658 DEBUG:boto:auth_path=/ffzdqu-60/obj11 2023-09-22T21:56:00.659 DEBUG:boto:Method: GET 2023-09-22T21:56:00.659 DEBUG:boto:Path: /ffzdqu-60/obj11 2023-09-22T21:56:00.659 DEBUG:boto:Data: 2023-09-22T21:56:00.659 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.659 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:00.659 DEBUG:boto:Port: 8000 2023-09-22T21:56:00.659 DEBUG:boto:Params: {} 2023-09-22T21:56:00.659 DEBUG:boto:Token: None 2023-09-22T21:56:00.659 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj11 2023-09-22T21:56:00.660 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3DIuOHFjm+GJ9KsFgyrYS0+R8XE= 2023-09-22T21:56:00.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3DIuOHFjm+GJ9KsFgyrYS0+R8XE=', 'Content-Length': '0'} 2023-09-22T21:56:00.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c9345517f0b0c9b-00650e0d70-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.663 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:00.663 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:00.663 DEBUG:boto:path=/ffzdqu-60/?prefix=obj11 2023-09-22T21:56:00.663 DEBUG:boto:auth_path=/ffzdqu-60/?prefix=obj11 2023-09-22T21:56:00.663 DEBUG:boto:Method: GET 2023-09-22T21:56:00.663 DEBUG:boto:Path: /ffzdqu-60/?prefix=obj11 2023-09-22T21:56:00.664 DEBUG:boto:Data: 2023-09-22T21:56:00.664 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.664 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:00.664 DEBUG:boto:Port: 8001 2023-09-22T21:56:00.664 DEBUG:boto:Params: {} 2023-09-22T21:56:00.664 DEBUG:boto:Token: None 2023-09-22T21:56:00.664 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/ 2023-09-22T21:56:00.664 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rsFaj0CDtNrDG0XtSB+wFeEmLWo= 2023-09-22T21:56:00.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rsFaj0CDtNrDG0XtSB+wFeEmLWo=', 'Content-Length': '0'} 2023-09-22T21:56:00.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf893aceb6c6108e-00650e0d70-1e5d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.710 DEBUG:boto:b'ffzdqu-60obj111000falseobj112023-09-22T21:51:13.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:00.711 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:00.711 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:00.711 DEBUG:boto:path=/ffzdqu-60/?prefix=obj11 2023-09-22T21:56:00.711 DEBUG:boto:auth_path=/ffzdqu-60/?prefix=obj11 2023-09-22T21:56:00.711 DEBUG:boto:Method: GET 2023-09-22T21:56:00.711 DEBUG:boto:Path: /ffzdqu-60/?prefix=obj11 2023-09-22T21:56:00.711 DEBUG:boto:Data: 2023-09-22T21:56:00.711 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.712 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:00.712 DEBUG:boto:Port: 8000 2023-09-22T21:56:00.712 DEBUG:boto:Params: {} 2023-09-22T21:56:00.712 DEBUG:boto:Token: None 2023-09-22T21:56:00.712 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/ 2023-09-22T21:56:00.712 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rsFaj0CDtNrDG0XtSB+wFeEmLWo= 2023-09-22T21:56:00.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rsFaj0CDtNrDG0XtSB+wFeEmLWo=', 'Content-Length': '0'} 2023-09-22T21:56:00.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000066706b71237de1d-00650e0d70-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.717 DEBUG:boto:b'ffzdqu-60obj111000falseobj112023-09-22T21:51:13.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:00.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:56:00.718 DEBUG:boto:path=/ffzdqu-60/obj22 2023-09-22T21:56:00.718 DEBUG:boto:auth_path=/ffzdqu-60/obj22 2023-09-22T21:56:00.718 DEBUG:boto:path=/ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.718 DEBUG:boto:auth_path=/ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.718 DEBUG:boto:Method: GET 2023-09-22T21:56:00.718 DEBUG:boto:Path: /ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.718 DEBUG:boto:Data: 2023-09-22T21:56:00.718 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.718 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:00.719 DEBUG:boto:Port: 8001 2023-09-22T21:56:00.719 DEBUG:boto:Params: {} 2023-09-22T21:56:00.719 DEBUG:boto:Token: None 2023-09-22T21:56:00.719 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.719 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ePZL7yP6lKrPx+UbDmGLQDKW5Hg= 2023-09-22T21:56:00.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ePZL7yP6lKrPx+UbDmGLQDKW5Hg=', 'Content-Length': '0'} 2023-09-22T21:56:00.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000249ef554ed7dd0c8-00650e0d70-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.723 DEBUG:boto:path=/ffzdqu-60/obj22 2023-09-22T21:56:00.723 DEBUG:boto:auth_path=/ffzdqu-60/obj22 2023-09-22T21:56:00.723 DEBUG:boto:path=/ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.723 DEBUG:boto:auth_path=/ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.723 DEBUG:boto:Method: GET 2023-09-22T21:56:00.723 DEBUG:boto:Path: /ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.723 DEBUG:boto:Data: 2023-09-22T21:56:00.723 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.724 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:00.724 DEBUG:boto:Port: 8000 2023-09-22T21:56:00.724 DEBUG:boto:Params: {} 2023-09-22T21:56:00.724 DEBUG:boto:Token: None 2023-09-22T21:56:00.724 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.724 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ePZL7yP6lKrPx+UbDmGLQDKW5Hg= 2023-09-22T21:56:00.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ePZL7yP6lKrPx+UbDmGLQDKW5Hg=', 'Content-Length': '0'} 2023-09-22T21:56:00.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e855314440d9ca1a-00650e0d70-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.727 DEBUG:boto:path=/ffzdqu-60/obj22 2023-09-22T21:56:00.727 DEBUG:boto:auth_path=/ffzdqu-60/obj22 2023-09-22T21:56:00.727 DEBUG:boto:path=/ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.727 DEBUG:boto:auth_path=/ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.728 DEBUG:boto:Method: HEAD 2023-09-22T21:56:00.728 DEBUG:boto:Path: /ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.728 DEBUG:boto:Data: 2023-09-22T21:56:00.728 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.728 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:00.728 DEBUG:boto:Port: 8001 2023-09-22T21:56:00.728 DEBUG:boto:Params: {} 2023-09-22T21:56:00.728 DEBUG:boto:Token: None 2023-09-22T21:56:00.728 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.728 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kuBauLE6NWMo5PA0WpJoeOgFKPA= 2023-09-22T21:56:00.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kuBauLE6NWMo5PA0WpJoeOgFKPA=', 'Content-Length': '0'} 2023-09-22T21:56:00.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010ffe08859dd140b-00650e0d70-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.731 DEBUG:boto:path=/ffzdqu-60/obj22 2023-09-22T21:56:00.731 DEBUG:boto:auth_path=/ffzdqu-60/obj22 2023-09-22T21:56:00.731 DEBUG:boto:path=/ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.732 DEBUG:boto:auth_path=/ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.732 DEBUG:boto:Method: HEAD 2023-09-22T21:56:00.732 DEBUG:boto:Path: /ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.732 DEBUG:boto:Data: 2023-09-22T21:56:00.732 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.732 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:00.732 DEBUG:boto:Port: 8000 2023-09-22T21:56:00.732 DEBUG:boto:Params: {} 2023-09-22T21:56:00.732 DEBUG:boto:Token: None 2023-09-22T21:56:00.733 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.733 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kuBauLE6NWMo5PA0WpJoeOgFKPA= 2023-09-22T21:56:00.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kuBauLE6NWMo5PA0WpJoeOgFKPA=', 'Content-Length': '0'} 2023-09-22T21:56:00.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d363045945dc77b0-00650e0d70-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:56:00.736 DEBUG:boto:path=/ffzdqu-60/obj22 2023-09-22T21:56:00.736 DEBUG:boto:auth_path=/ffzdqu-60/obj22 2023-09-22T21:56:00.736 DEBUG:boto:path=/ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.736 DEBUG:boto:auth_path=/ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.736 DEBUG:boto:Method: GET 2023-09-22T21:56:00.736 DEBUG:boto:Path: /ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.736 DEBUG:boto:Data: 2023-09-22T21:56:00.736 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:00.736 DEBUG:boto:Port: 8001 2023-09-22T21:56:00.737 DEBUG:boto:Params: {} 2023-09-22T21:56:00.737 DEBUG:boto:Token: None 2023-09-22T21:56:00.737 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.737 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ePZL7yP6lKrPx+UbDmGLQDKW5Hg= 2023-09-22T21:56:00.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ePZL7yP6lKrPx+UbDmGLQDKW5Hg=', 'Content-Length': '0'} 2023-09-22T21:56:00.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084237397d668435f-00650e0d70-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.786 DEBUG:boto:path=/ffzdqu-60/obj22 2023-09-22T21:56:00.786 DEBUG:boto:auth_path=/ffzdqu-60/obj22 2023-09-22T21:56:00.786 DEBUG:boto:path=/ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.787 DEBUG:boto:auth_path=/ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.787 DEBUG:boto:Method: GET 2023-09-22T21:56:00.787 DEBUG:boto:Path: /ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.787 DEBUG:boto:Data: 2023-09-22T21:56:00.787 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.787 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:00.787 DEBUG:boto:Port: 8000 2023-09-22T21:56:00.787 DEBUG:boto:Params: {} 2023-09-22T21:56:00.787 DEBUG:boto:Token: None 2023-09-22T21:56:00.787 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj22?versionId=null 2023-09-22T21:56:00.788 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ePZL7yP6lKrPx+UbDmGLQDKW5Hg= 2023-09-22T21:56:00.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ePZL7yP6lKrPx+UbDmGLQDKW5Hg=', 'Content-Length': '0'} 2023-09-22T21:56:00.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d63b646f519382f7-00650e0d70-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.791 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:00.791 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:00.791 DEBUG:boto:path=/ffzdqu-60/?prefix=obj22 2023-09-22T21:56:00.791 DEBUG:boto:auth_path=/ffzdqu-60/?prefix=obj22 2023-09-22T21:56:00.791 DEBUG:boto:Method: GET 2023-09-22T21:56:00.791 DEBUG:boto:Path: /ffzdqu-60/?prefix=obj22 2023-09-22T21:56:00.791 DEBUG:boto:Data: 2023-09-22T21:56:00.791 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.792 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:00.792 DEBUG:boto:Port: 8001 2023-09-22T21:56:00.792 DEBUG:boto:Params: {} 2023-09-22T21:56:00.792 DEBUG:boto:Token: None 2023-09-22T21:56:00.792 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/ 2023-09-22T21:56:00.792 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rsFaj0CDtNrDG0XtSB+wFeEmLWo= 2023-09-22T21:56:00.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rsFaj0CDtNrDG0XtSB+wFeEmLWo=', 'Content-Length': '0'} 2023-09-22T21:56:00.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012961fefb1bd1e71-00650e0d70-1e5d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.838 DEBUG:boto:b'ffzdqu-60obj221000falseobj222023-09-22T21:51:13.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:00.839 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:00.839 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:00.839 DEBUG:boto:path=/ffzdqu-60/?prefix=obj22 2023-09-22T21:56:00.839 DEBUG:boto:auth_path=/ffzdqu-60/?prefix=obj22 2023-09-22T21:56:00.839 DEBUG:boto:Method: GET 2023-09-22T21:56:00.839 DEBUG:boto:Path: /ffzdqu-60/?prefix=obj22 2023-09-22T21:56:00.839 DEBUG:boto:Data: 2023-09-22T21:56:00.839 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.839 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:00.840 DEBUG:boto:Port: 8000 2023-09-22T21:56:00.840 DEBUG:boto:Params: {} 2023-09-22T21:56:00.840 DEBUG:boto:Token: None 2023-09-22T21:56:00.840 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/ 2023-09-22T21:56:00.840 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rsFaj0CDtNrDG0XtSB+wFeEmLWo= 2023-09-22T21:56:00.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rsFaj0CDtNrDG0XtSB+wFeEmLWo=', 'Content-Length': '0'} 2023-09-22T21:56:00.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008ff9bda66a4278a-00650e0d70-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.845 DEBUG:boto:b'ffzdqu-60obj221000falseobj222023-09-22T21:51:13.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:00.845 DEBUG:boto:path=/ffzdqu-60/obj22 2023-09-22T21:56:00.845 DEBUG:boto:auth_path=/ffzdqu-60/obj22 2023-09-22T21:56:00.845 DEBUG:boto:Method: HEAD 2023-09-22T21:56:00.845 DEBUG:boto:Path: /ffzdqu-60/obj22 2023-09-22T21:56:00.845 DEBUG:boto:Data: 2023-09-22T21:56:00.845 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.846 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:00.846 DEBUG:boto:Port: 8001 2023-09-22T21:56:00.846 DEBUG:boto:Params: {} 2023-09-22T21:56:00.846 DEBUG:boto:Token: None 2023-09-22T21:56:00.846 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj22 2023-09-22T21:56:00.846 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bQ3bKvppiFVXf6ZhvYY6RVk8HzQ= 2023-09-22T21:56:00.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bQ3bKvppiFVXf6ZhvYY6RVk8HzQ=', 'Content-Length': '0'} 2023-09-22T21:56:00.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfb2fd0fa13f7a83-00650e0d70-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.849 DEBUG:boto:path=/ffzdqu-60/obj22 2023-09-22T21:56:00.849 DEBUG:boto:auth_path=/ffzdqu-60/obj22 2023-09-22T21:56:00.849 DEBUG:boto:Method: HEAD 2023-09-22T21:56:00.849 DEBUG:boto:Path: /ffzdqu-60/obj22 2023-09-22T21:56:00.850 DEBUG:boto:Data: 2023-09-22T21:56:00.850 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.850 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:00.850 DEBUG:boto:Port: 8000 2023-09-22T21:56:00.850 DEBUG:boto:Params: {} 2023-09-22T21:56:00.850 DEBUG:boto:Token: None 2023-09-22T21:56:00.850 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj22 2023-09-22T21:56:00.850 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bQ3bKvppiFVXf6ZhvYY6RVk8HzQ= 2023-09-22T21:56:00.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bQ3bKvppiFVXf6ZhvYY6RVk8HzQ=', 'Content-Length': '0'} 2023-09-22T21:56:00.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd2e39bdf6ec4ddf-00650e0d70-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:56:00.854 DEBUG:boto:path=/ffzdqu-60/obj22 2023-09-22T21:56:00.854 DEBUG:boto:auth_path=/ffzdqu-60/obj22 2023-09-22T21:56:00.854 DEBUG:boto:Method: GET 2023-09-22T21:56:00.854 DEBUG:boto:Path: /ffzdqu-60/obj22 2023-09-22T21:56:00.855 DEBUG:boto:Data: 2023-09-22T21:56:00.855 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.855 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:00.855 DEBUG:boto:Port: 8001 2023-09-22T21:56:00.855 DEBUG:boto:Params: {} 2023-09-22T21:56:00.855 DEBUG:boto:Token: None 2023-09-22T21:56:00.856 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj22 2023-09-22T21:56:00.856 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NxuRTPmjdYLY0Mzv8tbjF325m0c= 2023-09-22T21:56:00.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NxuRTPmjdYLY0Mzv8tbjF325m0c=', 'Content-Length': '0'} 2023-09-22T21:56:00.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d0da146df7852f9-00650e0d70-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.906 DEBUG:boto:path=/ffzdqu-60/obj22 2023-09-22T21:56:00.907 DEBUG:boto:auth_path=/ffzdqu-60/obj22 2023-09-22T21:56:00.907 DEBUG:boto:Method: GET 2023-09-22T21:56:00.907 DEBUG:boto:Path: /ffzdqu-60/obj22 2023-09-22T21:56:00.907 DEBUG:boto:Data: 2023-09-22T21:56:00.907 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.907 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:00.907 DEBUG:boto:Port: 8000 2023-09-22T21:56:00.907 DEBUG:boto:Params: {} 2023-09-22T21:56:00.908 DEBUG:boto:Token: None 2023-09-22T21:56:00.908 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj22 2023-09-22T21:56:00.908 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NxuRTPmjdYLY0Mzv8tbjF325m0c= 2023-09-22T21:56:00.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NxuRTPmjdYLY0Mzv8tbjF325m0c=', 'Content-Length': '0'} 2023-09-22T21:56:00.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004136e70a721a3060-00650e0d70-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.912 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:00.912 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:00.912 DEBUG:boto:path=/ffzdqu-60/?prefix=obj22 2023-09-22T21:56:00.912 DEBUG:boto:auth_path=/ffzdqu-60/?prefix=obj22 2023-09-22T21:56:00.912 DEBUG:boto:Method: GET 2023-09-22T21:56:00.912 DEBUG:boto:Path: /ffzdqu-60/?prefix=obj22 2023-09-22T21:56:00.912 DEBUG:boto:Data: 2023-09-22T21:56:00.912 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.912 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:00.913 DEBUG:boto:Port: 8001 2023-09-22T21:56:00.913 DEBUG:boto:Params: {} 2023-09-22T21:56:00.913 DEBUG:boto:Token: None 2023-09-22T21:56:00.913 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/ 2023-09-22T21:56:00.913 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rsFaj0CDtNrDG0XtSB+wFeEmLWo= 2023-09-22T21:56:00.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rsFaj0CDtNrDG0XtSB+wFeEmLWo=', 'Content-Length': '0'} 2023-09-22T21:56:00.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009270dd3e95d95151-00650e0d70-1e5d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.962 DEBUG:boto:b'ffzdqu-60obj221000falseobj222023-09-22T21:51:13.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:00.963 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:00.963 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:00.963 DEBUG:boto:path=/ffzdqu-60/?prefix=obj22 2023-09-22T21:56:00.963 DEBUG:boto:auth_path=/ffzdqu-60/?prefix=obj22 2023-09-22T21:56:00.963 DEBUG:boto:Method: GET 2023-09-22T21:56:00.963 DEBUG:boto:Path: /ffzdqu-60/?prefix=obj22 2023-09-22T21:56:00.963 DEBUG:boto:Data: 2023-09-22T21:56:00.963 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.964 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:00.964 DEBUG:boto:Port: 8000 2023-09-22T21:56:00.964 DEBUG:boto:Params: {} 2023-09-22T21:56:00.964 DEBUG:boto:Token: None 2023-09-22T21:56:00.964 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/ 2023-09-22T21:56:00.964 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rsFaj0CDtNrDG0XtSB+wFeEmLWo= 2023-09-22T21:56:00.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rsFaj0CDtNrDG0XtSB+wFeEmLWo=', 'Content-Length': '0'} 2023-09-22T21:56:00.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fffcb8e151acae65-00650e0d70-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.970 DEBUG:boto:b'ffzdqu-60obj221000falseobj222023-09-22T21:51:13.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:00.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:56:00.970 DEBUG:boto:path=/ffzdqu-60/obj33 2023-09-22T21:56:00.970 DEBUG:boto:auth_path=/ffzdqu-60/obj33 2023-09-22T21:56:00.970 DEBUG:boto:path=/ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:00.971 DEBUG:boto:auth_path=/ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:00.971 DEBUG:boto:Method: GET 2023-09-22T21:56:00.971 DEBUG:boto:Path: /ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:00.971 DEBUG:boto:Data: 2023-09-22T21:56:00.971 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.971 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:00.971 DEBUG:boto:Port: 8001 2023-09-22T21:56:00.971 DEBUG:boto:Params: {} 2023-09-22T21:56:00.971 DEBUG:boto:Token: None 2023-09-22T21:56:00.971 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:00.972 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bBK+Mr7vCpHWfe4QbBzJYaC6GN4= 2023-09-22T21:56:00.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bBK+Mr7vCpHWfe4QbBzJYaC6GN4=', 'Content-Length': '0'} 2023-09-22T21:56:00.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002f908b4cb71265f-00650e0d70-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.975 DEBUG:boto:path=/ffzdqu-60/obj33 2023-09-22T21:56:00.975 DEBUG:boto:auth_path=/ffzdqu-60/obj33 2023-09-22T21:56:00.975 DEBUG:boto:path=/ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:00.975 DEBUG:boto:auth_path=/ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:00.975 DEBUG:boto:Method: GET 2023-09-22T21:56:00.975 DEBUG:boto:Path: /ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:00.976 DEBUG:boto:Data: 2023-09-22T21:56:00.976 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.976 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:00.976 DEBUG:boto:Port: 8000 2023-09-22T21:56:00.976 DEBUG:boto:Params: {} 2023-09-22T21:56:00.976 DEBUG:boto:Token: None 2023-09-22T21:56:00.976 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:00.976 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bBK+Mr7vCpHWfe4QbBzJYaC6GN4= 2023-09-22T21:56:00.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bBK+Mr7vCpHWfe4QbBzJYaC6GN4=', 'Content-Length': '0'} 2023-09-22T21:56:00.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd2cf268cf0d304c-00650e0d70-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.979 DEBUG:boto:path=/ffzdqu-60/obj33 2023-09-22T21:56:00.979 DEBUG:boto:auth_path=/ffzdqu-60/obj33 2023-09-22T21:56:00.979 DEBUG:boto:path=/ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:00.980 DEBUG:boto:auth_path=/ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:00.980 DEBUG:boto:Method: HEAD 2023-09-22T21:56:00.980 DEBUG:boto:Path: /ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:00.980 DEBUG:boto:Data: 2023-09-22T21:56:00.980 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.980 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:00.980 DEBUG:boto:Port: 8001 2023-09-22T21:56:00.980 DEBUG:boto:Params: {} 2023-09-22T21:56:00.980 DEBUG:boto:Token: None 2023-09-22T21:56:00.980 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:00.981 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:w6nar7xbzH8RB7GeIZn2T6RTk3M= 2023-09-22T21:56:00.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:w6nar7xbzH8RB7GeIZn2T6RTk3M=', 'Content-Length': '0'} 2023-09-22T21:56:00.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019b0f9e1fd073d54-00650e0d70-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.984 DEBUG:boto:path=/ffzdqu-60/obj33 2023-09-22T21:56:00.984 DEBUG:boto:auth_path=/ffzdqu-60/obj33 2023-09-22T21:56:00.984 DEBUG:boto:path=/ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:00.984 DEBUG:boto:auth_path=/ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:00.984 DEBUG:boto:Method: HEAD 2023-09-22T21:56:00.984 DEBUG:boto:Path: /ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:00.984 DEBUG:boto:Data: 2023-09-22T21:56:00.984 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.985 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:00.985 DEBUG:boto:Port: 8000 2023-09-22T21:56:00.985 DEBUG:boto:Params: {} 2023-09-22T21:56:00.985 DEBUG:boto:Token: None 2023-09-22T21:56:00.985 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:00.985 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:w6nar7xbzH8RB7GeIZn2T6RTk3M= 2023-09-22T21:56:00.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:w6nar7xbzH8RB7GeIZn2T6RTk3M=', 'Content-Length': '0'} 2023-09-22T21:56:00.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3b2570d941bc780-00650e0d70-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:00.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:56:00.988 DEBUG:boto:path=/ffzdqu-60/obj33 2023-09-22T21:56:00.988 DEBUG:boto:auth_path=/ffzdqu-60/obj33 2023-09-22T21:56:00.988 DEBUG:boto:path=/ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:00.988 DEBUG:boto:auth_path=/ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:00.989 DEBUG:boto:Method: GET 2023-09-22T21:56:00.989 DEBUG:boto:Path: /ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:00.989 DEBUG:boto:Data: 2023-09-22T21:56:00.989 DEBUG:boto:Headers: {} 2023-09-22T21:56:00.989 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:00.989 DEBUG:boto:Port: 8001 2023-09-22T21:56:00.989 DEBUG:boto:Params: {} 2023-09-22T21:56:00.989 DEBUG:boto:Token: None 2023-09-22T21:56:00.989 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:00 GMT /ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:00.989 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bBK+Mr7vCpHWfe4QbBzJYaC6GN4= 2023-09-22T21:56:00.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:00 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bBK+Mr7vCpHWfe4QbBzJYaC6GN4=', 'Content-Length': '0'} 2023-09-22T21:56:00.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097aa75419148d08b-00650e0d70-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.034 DEBUG:boto:path=/ffzdqu-60/obj33 2023-09-22T21:56:01.034 DEBUG:boto:auth_path=/ffzdqu-60/obj33 2023-09-22T21:56:01.035 DEBUG:boto:path=/ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:01.035 DEBUG:boto:auth_path=/ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:01.035 DEBUG:boto:Method: GET 2023-09-22T21:56:01.035 DEBUG:boto:Path: /ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:01.035 DEBUG:boto:Data: 2023-09-22T21:56:01.035 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.035 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:01.035 DEBUG:boto:Port: 8000 2023-09-22T21:56:01.035 DEBUG:boto:Params: {} 2023-09-22T21:56:01.035 DEBUG:boto:Token: None 2023-09-22T21:56:01.036 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj33?versionId=null 2023-09-22T21:56:01.036 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PmxFNmrNFRQ8Cftv1kYzR7lpmR8= 2023-09-22T21:56:01.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PmxFNmrNFRQ8Cftv1kYzR7lpmR8=', 'Content-Length': '0'} 2023-09-22T21:56:01.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5c1afd2c6f8e5ae-00650e0d71-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.039 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:01.039 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:01.039 DEBUG:boto:path=/ffzdqu-60/?prefix=obj33 2023-09-22T21:56:01.039 DEBUG:boto:auth_path=/ffzdqu-60/?prefix=obj33 2023-09-22T21:56:01.039 DEBUG:boto:Method: GET 2023-09-22T21:56:01.039 DEBUG:boto:Path: /ffzdqu-60/?prefix=obj33 2023-09-22T21:56:01.039 DEBUG:boto:Data: 2023-09-22T21:56:01.040 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.040 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:01.040 DEBUG:boto:Port: 8001 2023-09-22T21:56:01.040 DEBUG:boto:Params: {} 2023-09-22T21:56:01.040 DEBUG:boto:Token: None 2023-09-22T21:56:01.040 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/ 2023-09-22T21:56:01.040 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o= 2023-09-22T21:56:01.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o=', 'Content-Length': '0'} 2023-09-22T21:56:01.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbb296b324c13eb3-00650e0d71-1e5d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.086 DEBUG:boto:b'ffzdqu-60obj331000falseobj332023-09-22T21:51:13.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:01.087 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:01.087 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:01.087 DEBUG:boto:path=/ffzdqu-60/?prefix=obj33 2023-09-22T21:56:01.087 DEBUG:boto:auth_path=/ffzdqu-60/?prefix=obj33 2023-09-22T21:56:01.087 DEBUG:boto:Method: GET 2023-09-22T21:56:01.087 DEBUG:boto:Path: /ffzdqu-60/?prefix=obj33 2023-09-22T21:56:01.087 DEBUG:boto:Data: 2023-09-22T21:56:01.088 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.088 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:01.088 DEBUG:boto:Port: 8000 2023-09-22T21:56:01.088 DEBUG:boto:Params: {} 2023-09-22T21:56:01.088 DEBUG:boto:Token: None 2023-09-22T21:56:01.088 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/ 2023-09-22T21:56:01.088 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o= 2023-09-22T21:56:01.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o=', 'Content-Length': '0'} 2023-09-22T21:56:01.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023ea3ff94073911c-00650e0d71-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.093 DEBUG:boto:b'ffzdqu-60obj331000falseobj332023-09-22T21:51:13.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:01.094 DEBUG:boto:path=/ffzdqu-60/obj33 2023-09-22T21:56:01.094 DEBUG:boto:auth_path=/ffzdqu-60/obj33 2023-09-22T21:56:01.094 DEBUG:boto:Method: HEAD 2023-09-22T21:56:01.094 DEBUG:boto:Path: /ffzdqu-60/obj33 2023-09-22T21:56:01.094 DEBUG:boto:Data: 2023-09-22T21:56:01.094 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.094 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:01.094 DEBUG:boto:Port: 8001 2023-09-22T21:56:01.094 DEBUG:boto:Params: {} 2023-09-22T21:56:01.095 DEBUG:boto:Token: None 2023-09-22T21:56:01.095 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj33 2023-09-22T21:56:01.095 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tD6Knhp4WSX6ZC1161VXfQXaoYU= 2023-09-22T21:56:01.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tD6Knhp4WSX6ZC1161VXfQXaoYU=', 'Content-Length': '0'} 2023-09-22T21:56:01.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000552bc98206c72389-00650e0d71-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.098 DEBUG:boto:path=/ffzdqu-60/obj33 2023-09-22T21:56:01.098 DEBUG:boto:auth_path=/ffzdqu-60/obj33 2023-09-22T21:56:01.098 DEBUG:boto:Method: HEAD 2023-09-22T21:56:01.098 DEBUG:boto:Path: /ffzdqu-60/obj33 2023-09-22T21:56:01.098 DEBUG:boto:Data: 2023-09-22T21:56:01.098 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.098 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:01.099 DEBUG:boto:Port: 8000 2023-09-22T21:56:01.099 DEBUG:boto:Params: {} 2023-09-22T21:56:01.099 DEBUG:boto:Token: None 2023-09-22T21:56:01.099 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj33 2023-09-22T21:56:01.099 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tD6Knhp4WSX6ZC1161VXfQXaoYU= 2023-09-22T21:56:01.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tD6Knhp4WSX6ZC1161VXfQXaoYU=', 'Content-Length': '0'} 2023-09-22T21:56:01.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092a02ff8dadcaaeb-00650e0d71-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:56:01.102 DEBUG:boto:path=/ffzdqu-60/obj33 2023-09-22T21:56:01.102 DEBUG:boto:auth_path=/ffzdqu-60/obj33 2023-09-22T21:56:01.102 DEBUG:boto:Method: GET 2023-09-22T21:56:01.103 DEBUG:boto:Path: /ffzdqu-60/obj33 2023-09-22T21:56:01.103 DEBUG:boto:Data: 2023-09-22T21:56:01.103 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.103 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:01.103 DEBUG:boto:Port: 8001 2023-09-22T21:56:01.103 DEBUG:boto:Params: {} 2023-09-22T21:56:01.103 DEBUG:boto:Token: None 2023-09-22T21:56:01.103 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj33 2023-09-22T21:56:01.103 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CFLwo989XmAjZ0mu8ZZTX6ZwoiM= 2023-09-22T21:56:01.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CFLwo989XmAjZ0mu8ZZTX6ZwoiM=', 'Content-Length': '0'} 2023-09-22T21:56:01.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9294c65e80bb6b9-00650e0d71-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.150 DEBUG:boto:path=/ffzdqu-60/obj33 2023-09-22T21:56:01.151 DEBUG:boto:auth_path=/ffzdqu-60/obj33 2023-09-22T21:56:01.151 DEBUG:boto:Method: GET 2023-09-22T21:56:01.151 DEBUG:boto:Path: /ffzdqu-60/obj33 2023-09-22T21:56:01.151 DEBUG:boto:Data: 2023-09-22T21:56:01.151 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.151 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:01.151 DEBUG:boto:Port: 8000 2023-09-22T21:56:01.151 DEBUG:boto:Params: {} 2023-09-22T21:56:01.151 DEBUG:boto:Token: None 2023-09-22T21:56:01.152 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj33 2023-09-22T21:56:01.152 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CFLwo989XmAjZ0mu8ZZTX6ZwoiM= 2023-09-22T21:56:01.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CFLwo989XmAjZ0mu8ZZTX6ZwoiM=', 'Content-Length': '0'} 2023-09-22T21:56:01.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cea581e4ed531fa6-00650e0d71-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.155 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:01.155 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:01.155 DEBUG:boto:path=/ffzdqu-60/?prefix=obj33 2023-09-22T21:56:01.155 DEBUG:boto:auth_path=/ffzdqu-60/?prefix=obj33 2023-09-22T21:56:01.155 DEBUG:boto:Method: GET 2023-09-22T21:56:01.155 DEBUG:boto:Path: /ffzdqu-60/?prefix=obj33 2023-09-22T21:56:01.155 DEBUG:boto:Data: 2023-09-22T21:56:01.156 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.156 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:01.156 DEBUG:boto:Port: 8001 2023-09-22T21:56:01.156 DEBUG:boto:Params: {} 2023-09-22T21:56:01.156 DEBUG:boto:Token: None 2023-09-22T21:56:01.156 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/ 2023-09-22T21:56:01.156 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o= 2023-09-22T21:56:01.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o=', 'Content-Length': '0'} 2023-09-22T21:56:01.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f15e754b1b7cc4b-00650e0d71-1e5d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.206 DEBUG:boto:b'ffzdqu-60obj331000falseobj332023-09-22T21:51:13.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:01.207 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:01.207 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:01.207 DEBUG:boto:path=/ffzdqu-60/?prefix=obj33 2023-09-22T21:56:01.207 DEBUG:boto:auth_path=/ffzdqu-60/?prefix=obj33 2023-09-22T21:56:01.207 DEBUG:boto:Method: GET 2023-09-22T21:56:01.207 DEBUG:boto:Path: /ffzdqu-60/?prefix=obj33 2023-09-22T21:56:01.207 DEBUG:boto:Data: 2023-09-22T21:56:01.208 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.208 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:01.208 DEBUG:boto:Port: 8000 2023-09-22T21:56:01.208 DEBUG:boto:Params: {} 2023-09-22T21:56:01.208 DEBUG:boto:Token: None 2023-09-22T21:56:01.208 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/ 2023-09-22T21:56:01.208 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o= 2023-09-22T21:56:01.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o=', 'Content-Length': '0'} 2023-09-22T21:56:01.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad1cba5fbdaf7cdc-00650e0d71-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.213 DEBUG:boto:b'ffzdqu-60obj331000falseobj332023-09-22T21:51:13.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:01.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:56:01.213 DEBUG:boto:path=/ffzdqu-60/obj44 2023-09-22T21:56:01.213 DEBUG:boto:auth_path=/ffzdqu-60/obj44 2023-09-22T21:56:01.214 DEBUG:boto:path=/ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.214 DEBUG:boto:auth_path=/ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.214 DEBUG:boto:Method: GET 2023-09-22T21:56:01.214 DEBUG:boto:Path: /ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.214 DEBUG:boto:Data: 2023-09-22T21:56:01.214 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.214 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:01.214 DEBUG:boto:Port: 8001 2023-09-22T21:56:01.214 DEBUG:boto:Params: {} 2023-09-22T21:56:01.215 DEBUG:boto:Token: None 2023-09-22T21:56:01.215 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.215 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Dog+0oQlK7QYJY08oEmsB22WTks= 2023-09-22T21:56:01.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Dog+0oQlK7QYJY08oEmsB22WTks=', 'Content-Length': '0'} 2023-09-22T21:56:01.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3627c56201aa7a3-00650e0d71-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.218 DEBUG:boto:path=/ffzdqu-60/obj44 2023-09-22T21:56:01.218 DEBUG:boto:auth_path=/ffzdqu-60/obj44 2023-09-22T21:56:01.218 DEBUG:boto:path=/ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.218 DEBUG:boto:auth_path=/ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.218 DEBUG:boto:Method: GET 2023-09-22T21:56:01.219 DEBUG:boto:Path: /ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.219 DEBUG:boto:Data: 2023-09-22T21:56:01.219 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.219 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:01.219 DEBUG:boto:Port: 8000 2023-09-22T21:56:01.219 DEBUG:boto:Params: {} 2023-09-22T21:56:01.219 DEBUG:boto:Token: None 2023-09-22T21:56:01.219 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.219 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Dog+0oQlK7QYJY08oEmsB22WTks= 2023-09-22T21:56:01.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Dog+0oQlK7QYJY08oEmsB22WTks=', 'Content-Length': '0'} 2023-09-22T21:56:01.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee894c4635bba50b-00650e0d71-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.222 DEBUG:boto:path=/ffzdqu-60/obj44 2023-09-22T21:56:01.223 DEBUG:boto:auth_path=/ffzdqu-60/obj44 2023-09-22T21:56:01.223 DEBUG:boto:path=/ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.223 DEBUG:boto:auth_path=/ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.223 DEBUG:boto:Method: HEAD 2023-09-22T21:56:01.223 DEBUG:boto:Path: /ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.223 DEBUG:boto:Data: 2023-09-22T21:56:01.223 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.223 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:01.223 DEBUG:boto:Port: 8001 2023-09-22T21:56:01.224 DEBUG:boto:Params: {} 2023-09-22T21:56:01.224 DEBUG:boto:Token: None 2023-09-22T21:56:01.224 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.224 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OojmqjSKF+JjLdyzKxACZZxDkFg= 2023-09-22T21:56:01.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OojmqjSKF+JjLdyzKxACZZxDkFg=', 'Content-Length': '0'} 2023-09-22T21:56:01.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d0dbd3fadd1366c-00650e0d71-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.227 DEBUG:boto:path=/ffzdqu-60/obj44 2023-09-22T21:56:01.227 DEBUG:boto:auth_path=/ffzdqu-60/obj44 2023-09-22T21:56:01.227 DEBUG:boto:path=/ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.227 DEBUG:boto:auth_path=/ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.227 DEBUG:boto:Method: HEAD 2023-09-22T21:56:01.227 DEBUG:boto:Path: /ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.227 DEBUG:boto:Data: 2023-09-22T21:56:01.227 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.228 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:01.228 DEBUG:boto:Port: 8000 2023-09-22T21:56:01.228 DEBUG:boto:Params: {} 2023-09-22T21:56:01.228 DEBUG:boto:Token: None 2023-09-22T21:56:01.228 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.228 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OojmqjSKF+JjLdyzKxACZZxDkFg= 2023-09-22T21:56:01.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OojmqjSKF+JjLdyzKxACZZxDkFg=', 'Content-Length': '0'} 2023-09-22T21:56:01.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b36d16ba302c84ff-00650e0d71-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:56:01.231 DEBUG:boto:path=/ffzdqu-60/obj44 2023-09-22T21:56:01.231 DEBUG:boto:auth_path=/ffzdqu-60/obj44 2023-09-22T21:56:01.231 DEBUG:boto:path=/ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.231 DEBUG:boto:auth_path=/ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.231 DEBUG:boto:Method: GET 2023-09-22T21:56:01.232 DEBUG:boto:Path: /ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.232 DEBUG:boto:Data: 2023-09-22T21:56:01.232 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.232 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:01.232 DEBUG:boto:Port: 8001 2023-09-22T21:56:01.232 DEBUG:boto:Params: {} 2023-09-22T21:56:01.232 DEBUG:boto:Token: None 2023-09-22T21:56:01.232 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.232 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Dog+0oQlK7QYJY08oEmsB22WTks= 2023-09-22T21:56:01.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Dog+0oQlK7QYJY08oEmsB22WTks=', 'Content-Length': '0'} 2023-09-22T21:56:01.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a139ccb8ae7a4965-00650e0d71-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.278 DEBUG:boto:path=/ffzdqu-60/obj44 2023-09-22T21:56:01.279 DEBUG:boto:auth_path=/ffzdqu-60/obj44 2023-09-22T21:56:01.279 DEBUG:boto:path=/ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.279 DEBUG:boto:auth_path=/ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.279 DEBUG:boto:Method: GET 2023-09-22T21:56:01.279 DEBUG:boto:Path: /ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.279 DEBUG:boto:Data: 2023-09-22T21:56:01.279 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.279 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:01.279 DEBUG:boto:Port: 8000 2023-09-22T21:56:01.279 DEBUG:boto:Params: {} 2023-09-22T21:56:01.280 DEBUG:boto:Token: None 2023-09-22T21:56:01.280 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj44?versionId=null 2023-09-22T21:56:01.280 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Dog+0oQlK7QYJY08oEmsB22WTks= 2023-09-22T21:56:01.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Dog+0oQlK7QYJY08oEmsB22WTks=', 'Content-Length': '0'} 2023-09-22T21:56:01.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4ce37e16a37a465-00650e0d71-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.283 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:01.283 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:01.283 DEBUG:boto:path=/ffzdqu-60/?prefix=obj44 2023-09-22T21:56:01.284 DEBUG:boto:auth_path=/ffzdqu-60/?prefix=obj44 2023-09-22T21:56:01.284 DEBUG:boto:Method: GET 2023-09-22T21:56:01.284 DEBUG:boto:Path: /ffzdqu-60/?prefix=obj44 2023-09-22T21:56:01.284 DEBUG:boto:Data: 2023-09-22T21:56:01.284 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.284 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:01.284 DEBUG:boto:Port: 8001 2023-09-22T21:56:01.284 DEBUG:boto:Params: {} 2023-09-22T21:56:01.285 DEBUG:boto:Token: None 2023-09-22T21:56:01.285 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/ 2023-09-22T21:56:01.285 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o= 2023-09-22T21:56:01.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o=', 'Content-Length': '0'} 2023-09-22T21:56:01.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017352f92ecb852f2-00650e0d71-1e5d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.330 DEBUG:boto:b'ffzdqu-60obj441000falseobj442023-09-22T21:51:13.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:01.331 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:01.331 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:01.331 DEBUG:boto:path=/ffzdqu-60/?prefix=obj44 2023-09-22T21:56:01.331 DEBUG:boto:auth_path=/ffzdqu-60/?prefix=obj44 2023-09-22T21:56:01.331 DEBUG:boto:Method: GET 2023-09-22T21:56:01.331 DEBUG:boto:Path: /ffzdqu-60/?prefix=obj44 2023-09-22T21:56:01.331 DEBUG:boto:Data: 2023-09-22T21:56:01.331 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.332 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:01.332 DEBUG:boto:Port: 8000 2023-09-22T21:56:01.332 DEBUG:boto:Params: {} 2023-09-22T21:56:01.332 DEBUG:boto:Token: None 2023-09-22T21:56:01.332 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/ 2023-09-22T21:56:01.332 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o= 2023-09-22T21:56:01.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o=', 'Content-Length': '0'} 2023-09-22T21:56:01.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be2d68553586b97a-00650e0d71-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.337 DEBUG:boto:b'ffzdqu-60obj441000falseobj442023-09-22T21:51:13.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:01.338 DEBUG:boto:path=/ffzdqu-60/obj44 2023-09-22T21:56:01.338 DEBUG:boto:auth_path=/ffzdqu-60/obj44 2023-09-22T21:56:01.338 DEBUG:boto:Method: HEAD 2023-09-22T21:56:01.338 DEBUG:boto:Path: /ffzdqu-60/obj44 2023-09-22T21:56:01.338 DEBUG:boto:Data: 2023-09-22T21:56:01.338 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.338 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:01.338 DEBUG:boto:Port: 8001 2023-09-22T21:56:01.338 DEBUG:boto:Params: {} 2023-09-22T21:56:01.339 DEBUG:boto:Token: None 2023-09-22T21:56:01.339 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj44 2023-09-22T21:56:01.339 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iz0Cm7MlTus9yWNCSJJ7xy/ctRU= 2023-09-22T21:56:01.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iz0Cm7MlTus9yWNCSJJ7xy/ctRU=', 'Content-Length': '0'} 2023-09-22T21:56:01.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a11af4665cdd89ec-00650e0d71-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.342 DEBUG:boto:path=/ffzdqu-60/obj44 2023-09-22T21:56:01.342 DEBUG:boto:auth_path=/ffzdqu-60/obj44 2023-09-22T21:56:01.342 DEBUG:boto:Method: HEAD 2023-09-22T21:56:01.342 DEBUG:boto:Path: /ffzdqu-60/obj44 2023-09-22T21:56:01.343 DEBUG:boto:Data: 2023-09-22T21:56:01.343 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.343 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:01.343 DEBUG:boto:Port: 8000 2023-09-22T21:56:01.343 DEBUG:boto:Params: {} 2023-09-22T21:56:01.343 DEBUG:boto:Token: None 2023-09-22T21:56:01.343 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj44 2023-09-22T21:56:01.343 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iz0Cm7MlTus9yWNCSJJ7xy/ctRU= 2023-09-22T21:56:01.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iz0Cm7MlTus9yWNCSJJ7xy/ctRU=', 'Content-Length': '0'} 2023-09-22T21:56:01.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bfb31e8a12706b4-00650e0d71-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:56:01.347 DEBUG:boto:path=/ffzdqu-60/obj44 2023-09-22T21:56:01.347 DEBUG:boto:auth_path=/ffzdqu-60/obj44 2023-09-22T21:56:01.347 DEBUG:boto:Method: GET 2023-09-22T21:56:01.347 DEBUG:boto:Path: /ffzdqu-60/obj44 2023-09-22T21:56:01.347 DEBUG:boto:Data: 2023-09-22T21:56:01.347 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.347 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:01.347 DEBUG:boto:Port: 8001 2023-09-22T21:56:01.347 DEBUG:boto:Params: {} 2023-09-22T21:56:01.347 DEBUG:boto:Token: None 2023-09-22T21:56:01.348 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj44 2023-09-22T21:56:01.348 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xU3ZiSkA3pPWr8MaPD7VQIQS0Og= 2023-09-22T21:56:01.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xU3ZiSkA3pPWr8MaPD7VQIQS0Og=', 'Content-Length': '0'} 2023-09-22T21:56:01.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012ca58cc79eea589-00650e0d71-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.394 DEBUG:boto:path=/ffzdqu-60/obj44 2023-09-22T21:56:01.394 DEBUG:boto:auth_path=/ffzdqu-60/obj44 2023-09-22T21:56:01.395 DEBUG:boto:Method: GET 2023-09-22T21:56:01.395 DEBUG:boto:Path: /ffzdqu-60/obj44 2023-09-22T21:56:01.395 DEBUG:boto:Data: 2023-09-22T21:56:01.395 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.395 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:01.395 DEBUG:boto:Port: 8000 2023-09-22T21:56:01.395 DEBUG:boto:Params: {} 2023-09-22T21:56:01.395 DEBUG:boto:Token: None 2023-09-22T21:56:01.396 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj44 2023-09-22T21:56:01.396 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xU3ZiSkA3pPWr8MaPD7VQIQS0Og= 2023-09-22T21:56:01.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xU3ZiSkA3pPWr8MaPD7VQIQS0Og=', 'Content-Length': '0'} 2023-09-22T21:56:01.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000323250a041ce26ae-00650e0d71-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.399 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:01.399 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:01.399 DEBUG:boto:path=/ffzdqu-60/?prefix=obj44 2023-09-22T21:56:01.399 DEBUG:boto:auth_path=/ffzdqu-60/?prefix=obj44 2023-09-22T21:56:01.399 DEBUG:boto:Method: GET 2023-09-22T21:56:01.400 DEBUG:boto:Path: /ffzdqu-60/?prefix=obj44 2023-09-22T21:56:01.400 DEBUG:boto:Data: 2023-09-22T21:56:01.400 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.400 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:01.400 DEBUG:boto:Port: 8001 2023-09-22T21:56:01.400 DEBUG:boto:Params: {} 2023-09-22T21:56:01.400 DEBUG:boto:Token: None 2023-09-22T21:56:01.400 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/ 2023-09-22T21:56:01.400 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o= 2023-09-22T21:56:01.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o=', 'Content-Length': '0'} 2023-09-22T21:56:01.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0218a8cf39baa25-00650e0d71-1e5d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.405 DEBUG:boto:b'ffzdqu-60obj441000falseobj442023-09-22T21:51:13.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:01.405 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:01.405 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:01.406 DEBUG:boto:path=/ffzdqu-60/?prefix=obj44 2023-09-22T21:56:01.406 DEBUG:boto:auth_path=/ffzdqu-60/?prefix=obj44 2023-09-22T21:56:01.406 DEBUG:boto:Method: GET 2023-09-22T21:56:01.406 DEBUG:boto:Path: /ffzdqu-60/?prefix=obj44 2023-09-22T21:56:01.406 DEBUG:boto:Data: 2023-09-22T21:56:01.406 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.406 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:01.406 DEBUG:boto:Port: 8000 2023-09-22T21:56:01.406 DEBUG:boto:Params: {} 2023-09-22T21:56:01.407 DEBUG:boto:Token: None 2023-09-22T21:56:01.407 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/ 2023-09-22T21:56:01.407 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o= 2023-09-22T21:56:01.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o=', 'Content-Length': '0'} 2023-09-22T21:56:01.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000710eb58b35299fa-00650e0d71-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.411 DEBUG:boto:b'ffzdqu-60obj441000falseobj442023-09-22T21:51:13.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:01.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:56:01.412 DEBUG:boto:path=/ffzdqu-60/obj55 2023-09-22T21:56:01.412 DEBUG:boto:auth_path=/ffzdqu-60/obj55 2023-09-22T21:56:01.412 DEBUG:boto:path=/ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.412 DEBUG:boto:auth_path=/ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.412 DEBUG:boto:Method: GET 2023-09-22T21:56:01.413 DEBUG:boto:Path: /ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.413 DEBUG:boto:Data: 2023-09-22T21:56:01.413 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.413 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:01.413 DEBUG:boto:Port: 8001 2023-09-22T21:56:01.413 DEBUG:boto:Params: {} 2023-09-22T21:56:01.413 DEBUG:boto:Token: None 2023-09-22T21:56:01.413 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.413 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T7y/UzVuXJU6IYxMjKHejBLEG/o= 2023-09-22T21:56:01.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T7y/UzVuXJU6IYxMjKHejBLEG/o=', 'Content-Length': '0'} 2023-09-22T21:56:01.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004155d395ce15f353-00650e0d71-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.458 DEBUG:boto:path=/ffzdqu-60/obj55 2023-09-22T21:56:01.459 DEBUG:boto:auth_path=/ffzdqu-60/obj55 2023-09-22T21:56:01.459 DEBUG:boto:path=/ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.459 DEBUG:boto:auth_path=/ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.459 DEBUG:boto:Method: GET 2023-09-22T21:56:01.459 DEBUG:boto:Path: /ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.459 DEBUG:boto:Data: 2023-09-22T21:56:01.459 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.459 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:01.459 DEBUG:boto:Port: 8000 2023-09-22T21:56:01.460 DEBUG:boto:Params: {} 2023-09-22T21:56:01.460 DEBUG:boto:Token: None 2023-09-22T21:56:01.460 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.460 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T7y/UzVuXJU6IYxMjKHejBLEG/o= 2023-09-22T21:56:01.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T7y/UzVuXJU6IYxMjKHejBLEG/o=', 'Content-Length': '0'} 2023-09-22T21:56:01.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091dace28a0c0b25b-00650e0d71-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.463 DEBUG:boto:path=/ffzdqu-60/obj55 2023-09-22T21:56:01.463 DEBUG:boto:auth_path=/ffzdqu-60/obj55 2023-09-22T21:56:01.463 DEBUG:boto:path=/ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.463 DEBUG:boto:auth_path=/ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.463 DEBUG:boto:Method: HEAD 2023-09-22T21:56:01.463 DEBUG:boto:Path: /ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.464 DEBUG:boto:Data: 2023-09-22T21:56:01.464 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.464 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:01.464 DEBUG:boto:Port: 8001 2023-09-22T21:56:01.464 DEBUG:boto:Params: {} 2023-09-22T21:56:01.464 DEBUG:boto:Token: None 2023-09-22T21:56:01.464 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.464 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3lZAkY4AyI2x8X6YAbYZnNlEQsE= 2023-09-22T21:56:01.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3lZAkY4AyI2x8X6YAbYZnNlEQsE=', 'Content-Length': '0'} 2023-09-22T21:56:01.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9e5942837305914-00650e0d71-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.470 DEBUG:boto:path=/ffzdqu-60/obj55 2023-09-22T21:56:01.470 DEBUG:boto:auth_path=/ffzdqu-60/obj55 2023-09-22T21:56:01.470 DEBUG:boto:path=/ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.470 DEBUG:boto:auth_path=/ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.470 DEBUG:boto:Method: HEAD 2023-09-22T21:56:01.470 DEBUG:boto:Path: /ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.471 DEBUG:boto:Data: 2023-09-22T21:56:01.471 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.471 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:01.471 DEBUG:boto:Port: 8000 2023-09-22T21:56:01.471 DEBUG:boto:Params: {} 2023-09-22T21:56:01.471 DEBUG:boto:Token: None 2023-09-22T21:56:01.471 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.471 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3lZAkY4AyI2x8X6YAbYZnNlEQsE= 2023-09-22T21:56:01.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3lZAkY4AyI2x8X6YAbYZnNlEQsE=', 'Content-Length': '0'} 2023-09-22T21:56:01.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037efd1b9e3a58a70-00650e0d71-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:56:01.475 DEBUG:boto:path=/ffzdqu-60/obj55 2023-09-22T21:56:01.475 DEBUG:boto:auth_path=/ffzdqu-60/obj55 2023-09-22T21:56:01.475 DEBUG:boto:path=/ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.475 DEBUG:boto:auth_path=/ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.475 DEBUG:boto:Method: GET 2023-09-22T21:56:01.475 DEBUG:boto:Path: /ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.475 DEBUG:boto:Data: 2023-09-22T21:56:01.475 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.476 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:01.476 DEBUG:boto:Port: 8001 2023-09-22T21:56:01.476 DEBUG:boto:Params: {} 2023-09-22T21:56:01.476 DEBUG:boto:Token: None 2023-09-22T21:56:01.476 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.476 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T7y/UzVuXJU6IYxMjKHejBLEG/o= 2023-09-22T21:56:01.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T7y/UzVuXJU6IYxMjKHejBLEG/o=', 'Content-Length': '0'} 2023-09-22T21:56:01.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3a89753142ef2a5-00650e0d71-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.522 DEBUG:boto:path=/ffzdqu-60/obj55 2023-09-22T21:56:01.523 DEBUG:boto:auth_path=/ffzdqu-60/obj55 2023-09-22T21:56:01.523 DEBUG:boto:path=/ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.523 DEBUG:boto:auth_path=/ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.523 DEBUG:boto:Method: GET 2023-09-22T21:56:01.523 DEBUG:boto:Path: /ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.523 DEBUG:boto:Data: 2023-09-22T21:56:01.523 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.523 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:01.523 DEBUG:boto:Port: 8000 2023-09-22T21:56:01.523 DEBUG:boto:Params: {} 2023-09-22T21:56:01.524 DEBUG:boto:Token: None 2023-09-22T21:56:01.524 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj55?versionId=null 2023-09-22T21:56:01.524 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T7y/UzVuXJU6IYxMjKHejBLEG/o= 2023-09-22T21:56:01.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T7y/UzVuXJU6IYxMjKHejBLEG/o=', 'Content-Length': '0'} 2023-09-22T21:56:01.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000115f6891d81743ac-00650e0d71-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.527 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:01.527 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:01.527 DEBUG:boto:path=/ffzdqu-60/?prefix=obj55 2023-09-22T21:56:01.528 DEBUG:boto:auth_path=/ffzdqu-60/?prefix=obj55 2023-09-22T21:56:01.528 DEBUG:boto:Method: GET 2023-09-22T21:56:01.528 DEBUG:boto:Path: /ffzdqu-60/?prefix=obj55 2023-09-22T21:56:01.528 DEBUG:boto:Data: 2023-09-22T21:56:01.528 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.528 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:01.528 DEBUG:boto:Port: 8001 2023-09-22T21:56:01.528 DEBUG:boto:Params: {} 2023-09-22T21:56:01.529 DEBUG:boto:Token: None 2023-09-22T21:56:01.529 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/ 2023-09-22T21:56:01.529 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o= 2023-09-22T21:56:01.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o=', 'Content-Length': '0'} 2023-09-22T21:56:01.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006267e5c19c9c1b9d-00650e0d71-1e5d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.574 DEBUG:boto:b'ffzdqu-60obj551000falseobj552023-09-22T21:51:14.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:01.575 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:01.575 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:01.575 DEBUG:boto:path=/ffzdqu-60/?prefix=obj55 2023-09-22T21:56:01.575 DEBUG:boto:auth_path=/ffzdqu-60/?prefix=obj55 2023-09-22T21:56:01.575 DEBUG:boto:Method: GET 2023-09-22T21:56:01.575 DEBUG:boto:Path: /ffzdqu-60/?prefix=obj55 2023-09-22T21:56:01.575 DEBUG:boto:Data: 2023-09-22T21:56:01.576 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.576 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:01.576 DEBUG:boto:Port: 8000 2023-09-22T21:56:01.576 DEBUG:boto:Params: {} 2023-09-22T21:56:01.576 DEBUG:boto:Token: None 2023-09-22T21:56:01.576 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/ 2023-09-22T21:56:01.576 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o= 2023-09-22T21:56:01.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o=', 'Content-Length': '0'} 2023-09-22T21:56:01.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034db98fe26c4ecef-00650e0d71-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.581 DEBUG:boto:b'ffzdqu-60obj551000falseobj552023-09-22T21:51:14.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:01.581 DEBUG:boto:path=/ffzdqu-60/obj55 2023-09-22T21:56:01.581 DEBUG:boto:auth_path=/ffzdqu-60/obj55 2023-09-22T21:56:01.581 DEBUG:boto:Method: HEAD 2023-09-22T21:56:01.581 DEBUG:boto:Path: /ffzdqu-60/obj55 2023-09-22T21:56:01.581 DEBUG:boto:Data: 2023-09-22T21:56:01.581 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.582 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:01.582 DEBUG:boto:Port: 8001 2023-09-22T21:56:01.582 DEBUG:boto:Params: {} 2023-09-22T21:56:01.582 DEBUG:boto:Token: None 2023-09-22T21:56:01.582 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj55 2023-09-22T21:56:01.582 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EVYumi1QnT/qrJ0Bii2d3Vb/Cco= 2023-09-22T21:56:01.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EVYumi1QnT/qrJ0Bii2d3Vb/Cco=', 'Content-Length': '0'} 2023-09-22T21:56:01.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001bce73e2aefd55a-00650e0d71-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.585 DEBUG:boto:path=/ffzdqu-60/obj55 2023-09-22T21:56:01.585 DEBUG:boto:auth_path=/ffzdqu-60/obj55 2023-09-22T21:56:01.585 DEBUG:boto:Method: HEAD 2023-09-22T21:56:01.586 DEBUG:boto:Path: /ffzdqu-60/obj55 2023-09-22T21:56:01.586 DEBUG:boto:Data: 2023-09-22T21:56:01.586 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.586 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:01.586 DEBUG:boto:Port: 8000 2023-09-22T21:56:01.586 DEBUG:boto:Params: {} 2023-09-22T21:56:01.586 DEBUG:boto:Token: None 2023-09-22T21:56:01.586 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj55 2023-09-22T21:56:01.587 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EVYumi1QnT/qrJ0Bii2d3Vb/Cco= 2023-09-22T21:56:01.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EVYumi1QnT/qrJ0Bii2d3Vb/Cco=', 'Content-Length': '0'} 2023-09-22T21:56:01.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000929028ea229f3bc-00650e0d71-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:56:01.590 DEBUG:boto:path=/ffzdqu-60/obj55 2023-09-22T21:56:01.590 DEBUG:boto:auth_path=/ffzdqu-60/obj55 2023-09-22T21:56:01.590 DEBUG:boto:Method: GET 2023-09-22T21:56:01.590 DEBUG:boto:Path: /ffzdqu-60/obj55 2023-09-22T21:56:01.590 DEBUG:boto:Data: 2023-09-22T21:56:01.590 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.590 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:01.590 DEBUG:boto:Port: 8001 2023-09-22T21:56:01.591 DEBUG:boto:Params: {} 2023-09-22T21:56:01.591 DEBUG:boto:Token: None 2023-09-22T21:56:01.591 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj55 2023-09-22T21:56:01.591 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rEELYjKX2dkKZUh2yqyVeYo8oQQ= 2023-09-22T21:56:01.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rEELYjKX2dkKZUh2yqyVeYo8oQQ=', 'Content-Length': '0'} 2023-09-22T21:56:01.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-60:f132bdb9-20a7-4073-a932-8f99da20691e.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015670622e7ec06b8-00650e0d71-1e5d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.634 DEBUG:boto:path=/ffzdqu-60/obj55 2023-09-22T21:56:01.634 DEBUG:boto:auth_path=/ffzdqu-60/obj55 2023-09-22T21:56:01.635 DEBUG:boto:Method: GET 2023-09-22T21:56:01.635 DEBUG:boto:Path: /ffzdqu-60/obj55 2023-09-22T21:56:01.635 DEBUG:boto:Data: 2023-09-22T21:56:01.635 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.635 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:01.635 DEBUG:boto:Port: 8000 2023-09-22T21:56:01.635 DEBUG:boto:Params: {} 2023-09-22T21:56:01.635 DEBUG:boto:Token: None 2023-09-22T21:56:01.636 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/obj55 2023-09-22T21:56:01.636 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rEELYjKX2dkKZUh2yqyVeYo8oQQ= 2023-09-22T21:56:01.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rEELYjKX2dkKZUh2yqyVeYo8oQQ=', 'Content-Length': '0'} 2023-09-22T21:56:01.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:51:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a589cb045cba7224-00650e0d71-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.639 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:01.639 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:01.639 DEBUG:boto:path=/ffzdqu-60/?prefix=obj55 2023-09-22T21:56:01.639 DEBUG:boto:auth_path=/ffzdqu-60/?prefix=obj55 2023-09-22T21:56:01.639 DEBUG:boto:Method: GET 2023-09-22T21:56:01.639 DEBUG:boto:Path: /ffzdqu-60/?prefix=obj55 2023-09-22T21:56:01.640 DEBUG:boto:Data: 2023-09-22T21:56:01.640 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.640 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:01.640 DEBUG:boto:Port: 8001 2023-09-22T21:56:01.640 DEBUG:boto:Params: {} 2023-09-22T21:56:01.640 DEBUG:boto:Token: None 2023-09-22T21:56:01.640 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/ 2023-09-22T21:56:01.640 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o= 2023-09-22T21:56:01.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o=', 'Content-Length': '0'} 2023-09-22T21:56:01.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5c59dfb2afb2b6f-00650e0d71-1e5d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.686 DEBUG:boto:b'ffzdqu-60obj551000falseobj552023-09-22T21:51:14.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:01.687 DEBUG:boto:path=/ffzdqu-60/ 2023-09-22T21:56:01.687 DEBUG:boto:auth_path=/ffzdqu-60/ 2023-09-22T21:56:01.687 DEBUG:boto:path=/ffzdqu-60/?prefix=obj55 2023-09-22T21:56:01.687 DEBUG:boto:auth_path=/ffzdqu-60/?prefix=obj55 2023-09-22T21:56:01.687 DEBUG:boto:Method: GET 2023-09-22T21:56:01.687 DEBUG:boto:Path: /ffzdqu-60/?prefix=obj55 2023-09-22T21:56:01.687 DEBUG:boto:Data: 2023-09-22T21:56:01.688 DEBUG:boto:Headers: {} 2023-09-22T21:56:01.688 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:01.688 DEBUG:boto:Port: 8000 2023-09-22T21:56:01.688 DEBUG:boto:Params: {} 2023-09-22T21:56:01.688 DEBUG:boto:Token: None 2023-09-22T21:56:01.688 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:56:01 GMT /ffzdqu-60/ 2023-09-22T21:56:01.688 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o= 2023-09-22T21:56:01.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:01 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ESSB6/kW1JIlY0VfmcKb3G/4b5o=', 'Content-Length': '0'} 2023-09-22T21:56:01.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d1c7a697cdfbe81-00650e0d71-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:01.693 DEBUG:boto:b'ffzdqu-60obj551000falseobj552023-09-22T21:51:14.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:56:01.693 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-60 zones={a2, a1} 2023-09-22T21:56:01.694 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-09-22T21:56:01.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:56:01.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:56:01.694 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:56:01.737 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:56:01.737 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:56:01.801 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"00000000000000000000:00000000000000018751","last_update":"2023-09-22T21:47:16.364707Z"},{"marker":"00000000000000000000:00000000000000030598","last_update":"2023-09-22T21:51:13.956930Z"},{"marker":"00000000000000000000:00000000000000027349","last_update":"2023-09-22T21:51:17.512324Z"},{"marker":"00000000000000000000:00000000000000026257","last_update":"2023-09-22T21:51:17.515944Z"}] 2023-09-22T21:56:01.801 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000018751', 1: '00000000000000000000:00000000000000030598', 2: '00000000000000000000:00000000000000027349', 3: '00000000000000000000:00000000000000026257'} 2023-09-22T21:56:01.802 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-22T21:56:01.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:56:01.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:56:01.802 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:56:01.884 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:56:01.885 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:56:01.944 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17766500030570482017},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000018751","next_step_marker":"","total_entries":9,"pos":0,"timestamp":"2023-09-22T21:47:16.364707Z"}},{"key":1,"val":{"status":"full-sync","marker":"ffzdqu-43:f132bdb9-20a7-4073-a932-8f99da20691e.5729.8:0","next_step_marker":"00000000000000000000:00000000000000030598","total_entries":51,"pos":34,"timestamp":"2023-09-22T21:51:13.956930Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000027349","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:51:17.512324Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000026257","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:51:17.515944Z"}}]},"full_sync":{"total":60,"complete":43},"current_time":"2023-09-22T21:56:01Z"} 2023-09-22T21:56:01.944 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17766500030570482017},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000018751","next_step_marker":"","total_entries":9,"pos":0,"timestamp":"2023-09-22T21:47:16.364707Z"}},{"key":1,"val":{"status":"full-sync","marker":"ffzdqu-43:f132bdb9-20a7-4073-a932-8f99da20691e.5729.8:0","next_step_marker":"00000000000000000000:00000000000000030598","total_entries":51,"pos":34,"timestamp":"2023-09-22T21:51:13.956930Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000027349","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:51:17.512324Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000026257","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:51:17.515944Z"}}]},"full_sync":{"total":60,"complete":43},"current_time":"2023-09-22T21:56:01Z"} 2023-09-22T21:56:01.945 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018751', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-22T21:47:16.364707Z'}}, {'key': 1, 'val': {'status': 'full-sync', 'marker': 'ffzdqu-43:f132bdb9-20a7-4073-a932-8f99da20691e.5729.8:0', 'next_step_marker': '00000000000000000000:00000000000000030598', 'total_entries': 51, 'pos': 34, 'timestamp': '2023-09-22T21:51:13.956930Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027349', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:51:17.512324Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026257', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:51:17.515944Z'}}] 2023-09-22T21:56:01.945 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000018751', 1: '00000000000000000000:00000000000000030598', 2: '00000000000000000000:00000000000000027349', 3: '00000000000000000000:00000000000000026257'} 2023-09-22T21:56:01.945 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000018751', 1: 'ffzdqu-43:f132bdb9-20a7-4073-a932-8f99da20691e.5729.8:0', 2: '00000000000000000000:00000000000000027349', 3: '00000000000000000000:00000000000000026257'} 2023-09-22T21:56:01.945 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-22T21:56:01.945 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-09-22T21:56:01.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:56:01.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:56:01.946 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:56:02.025 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:56:02.025 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:56:02.089 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"00000000000000000000:00000000000000005159","last_update":"2023-09-22T21:49:28.827000Z"},{"marker":"00000000000000000000:00000000000000014601","last_update":"2023-09-22T21:53:18.309214Z"},{"marker":"00000000000000000000:00000000000000012775","last_update":"2023-09-22T21:53:26.270738Z"},{"marker":"00000000000000000000:00000000000000010828","last_update":"2023-09-22T21:54:18.280719Z"}] 2023-09-22T21:56:02.090 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000005159', 1: '00000000000000000000:00000000000000014601', 2: '00000000000000000000:00000000000000012775', 3: '00000000000000000000:00000000000000010828'} 2023-09-22T21:56:02.090 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-09-22T21:56:02.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:56:02.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:56:02.091 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:56:02.167 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:56:02.167 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:56:02.228 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10476880264182873081},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005159","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:28.827000Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000014601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:53:18.309214Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012775","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:53:26.270738Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000010828","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:54:18.280719Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:56:02Z"} 2023-09-22T21:56:02.228 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10476880264182873081},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005159","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:28.827000Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000014601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:53:18.309214Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012775","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:53:26.270738Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000010828","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:54:18.280719Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:56:02Z"} 2023-09-22T21:56:02.228 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005159', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:28.827000Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014601', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:53:18.309214Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012775', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:53:26.270738Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:54:18.280719Z'}}] 2023-09-22T21:56:02.228 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005159', 1: '00000000000000000000:00000000000000014601', 2: '00000000000000000000:00000000000000012775', 3: '00000000000000000000:00000000000000010828'} 2023-09-22T21:56:02.228 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005159', 1: '00000000000000000000:00000000000000014601', 2: '00000000000000000000:00000000000000012775', 3: '00000000000000000000:00000000000000010828'} 2023-09-22T21:56:02.229 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-09-22T21:56:02.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... ok 2023-09-22T21:56:02.230 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:02.230 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:02.230 DEBUG:boto:Method: PUT 2023-09-22T21:56:02.230 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:02.230 DEBUG:boto:Data: 2023-09-22T21:56:02.230 DEBUG:boto:Headers: {} 2023-09-22T21:56:02.230 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:02.231 DEBUG:boto:Port: 8000 2023-09-22T21:56:02.231 DEBUG:boto:Params: {} 2023-09-22T21:56:02.231 DEBUG:boto:Token: None 2023-09-22T21:56:02.231 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:56:02 GMT /ffzdqu-61/ 2023-09-22T21:56:02.231 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h9zKgkw4NLEgRR993CQSiBK6nxI= 2023-09-22T21:56:02.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:02 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h9zKgkw4NLEgRR993CQSiBK6nxI=', 'Content-Length': '0'} 2023-09-22T21:56:02.250 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006cd285652f1ef60f-00650e0d72-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:02.250 DEBUG:rgw_multi.tests:created bucket=ffzdqu-61 2023-09-22T21:56:02.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:56:02.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:56:02.251 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:56:02.302 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:56:02.303 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:56:02.349 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:56:02.372 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"1_1695419384.000133_332.1","last_update":"2023-09-22T21:49:44.000133Z"},{"marker":"1_1695419762.251118_563.1","last_update":"2023-09-22T21:56:02.251118Z"},{"marker":"1_1695419383.971498_240.1","last_update":"2023-09-22T21:49:43.971498Z"}] 2023-09-22T21:56:02.372 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419762.251118_563.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:56:02.373 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:56:02.373 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:56:02.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:56:02.373 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:56:02.449 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:56:02.449 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:56:02.487 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:56:02.487+0000 7fab5e71ba40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:56:02.488 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:56:02.487+0000 7fab5e71ba40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:56:02.506 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419762.251118_563.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:56:02.251118Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:56:02Z"} 2023-09-22T21:56:02.506 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419762.251118_563.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:56:02.251118Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:56:02Z"} 2023-09-22T21:56:02.506 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695419762.251118_563.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:56:02.251118Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T21:56:02.506 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419762.251118_563.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:56:02.506 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419762.251118_563.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:56:02.507 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:56:02.507 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-09-22T21:56:02.507 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T21:56:02.547 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:56:02.547+0000 7fdaa7af8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 80262) UID: 0 2023-09-22T21:56:02.547 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:56:02.547+0000 7fdaab33fa40 -1 shutting down 2023-09-22T21:56:03.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-09-22T21:56:08.608 INFO:tasks.rgw.c1.client.1:Stopped 2023-09-22T21:56:08.609 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:08.609 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:08.609 DEBUG:boto:Method: HEAD 2023-09-22T21:56:08.609 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:08.609 DEBUG:boto:Data: 2023-09-22T21:56:08.609 DEBUG:boto:Headers: {} 2023-09-22T21:56:08.609 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:08.609 DEBUG:boto:Port: 8000 2023-09-22T21:56:08.610 DEBUG:boto:Params: {} 2023-09-22T21:56:08.610 DEBUG:boto:Token: None 2023-09-22T21:56:08.610 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:08 GMT /ffzdqu-61/ 2023-09-22T21:56:08.610 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QJuMcjeS5lNJK3VyjHXvS1Pr6WM= 2023-09-22T21:56:08.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QJuMcjeS5lNJK3VyjHXvS1Pr6WM=', 'Content-Length': '0'} 2023-09-22T21:56:08.614 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab813913b392373e-00650e0d78-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:08.614 DEBUG:boto:path=/ffzdqu-61/obj11 2023-09-22T21:56:08.615 DEBUG:boto:auth_path=/ffzdqu-61/obj11 2023-09-22T21:56:08.615 DEBUG:boto:Method: PUT 2023-09-22T21:56:08.615 DEBUG:boto:Path: /ffzdqu-61/obj11 2023-09-22T21:56:08.615 DEBUG:boto:Data: 2023-09-22T21:56:08.615 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:08.615 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:08.615 DEBUG:boto:Port: 8000 2023-09-22T21:56:08.615 DEBUG:boto:Params: {} 2023-09-22T21:56:08.615 DEBUG:boto:Token: None 2023-09-22T21:56:08.616 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:08 GMT /ffzdqu-61/obj11 2023-09-22T21:56:08.616 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0WOWQM9nqBVUNa+H9LT1/CzkBlI= 2023-09-22T21:56:08.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0WOWQM9nqBVUNa+H9LT1/CzkBlI='} 2023-09-22T21:56:08.633 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001627b0bd36ea67a7-00650e0d78-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:08.633 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:08.633 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:08.633 DEBUG:boto:Method: HEAD 2023-09-22T21:56:08.633 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:08.633 DEBUG:boto:Data: 2023-09-22T21:56:08.634 DEBUG:boto:Headers: {} 2023-09-22T21:56:08.634 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:08.634 DEBUG:boto:Port: 8000 2023-09-22T21:56:08.634 DEBUG:boto:Params: {} 2023-09-22T21:56:08.634 DEBUG:boto:Token: None 2023-09-22T21:56:08.634 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:08 GMT /ffzdqu-61/ 2023-09-22T21:56:08.634 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QJuMcjeS5lNJK3VyjHXvS1Pr6WM= 2023-09-22T21:56:08.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QJuMcjeS5lNJK3VyjHXvS1Pr6WM=', 'Content-Length': '0'} 2023-09-22T21:56:08.638 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065c928fbdc203bd8-00650e0d78-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:08.638 DEBUG:boto:path=/ffzdqu-61/obj22 2023-09-22T21:56:08.639 DEBUG:boto:auth_path=/ffzdqu-61/obj22 2023-09-22T21:56:08.639 DEBUG:boto:Method: PUT 2023-09-22T21:56:08.639 DEBUG:boto:Path: /ffzdqu-61/obj22 2023-09-22T21:56:08.639 DEBUG:boto:Data: 2023-09-22T21:56:08.639 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:08.639 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:08.639 DEBUG:boto:Port: 8000 2023-09-22T21:56:08.639 DEBUG:boto:Params: {} 2023-09-22T21:56:08.639 DEBUG:boto:Token: None 2023-09-22T21:56:08.640 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:08 GMT /ffzdqu-61/obj22 2023-09-22T21:56:08.640 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F6uDYFYLxwI78ji7RbPb/BlmqUo= 2023-09-22T21:56:08.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F6uDYFYLxwI78ji7RbPb/BlmqUo='} 2023-09-22T21:56:08.649 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6819a49f36e35f7-00650e0d78-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:08.649 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:08.649 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:08.649 DEBUG:boto:Method: HEAD 2023-09-22T21:56:08.649 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:08.650 DEBUG:boto:Data: 2023-09-22T21:56:08.650 DEBUG:boto:Headers: {} 2023-09-22T21:56:08.650 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:08.650 DEBUG:boto:Port: 8000 2023-09-22T21:56:08.650 DEBUG:boto:Params: {} 2023-09-22T21:56:08.650 DEBUG:boto:Token: None 2023-09-22T21:56:08.650 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:08 GMT /ffzdqu-61/ 2023-09-22T21:56:08.650 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QJuMcjeS5lNJK3VyjHXvS1Pr6WM= 2023-09-22T21:56:08.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QJuMcjeS5lNJK3VyjHXvS1Pr6WM=', 'Content-Length': '0'} 2023-09-22T21:56:08.654 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000112fcef66eba4a08-00650e0d78-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:08.654 DEBUG:boto:path=/ffzdqu-61/obj33 2023-09-22T21:56:08.655 DEBUG:boto:auth_path=/ffzdqu-61/obj33 2023-09-22T21:56:08.655 DEBUG:boto:Method: PUT 2023-09-22T21:56:08.655 DEBUG:boto:Path: /ffzdqu-61/obj33 2023-09-22T21:56:08.655 DEBUG:boto:Data: 2023-09-22T21:56:08.655 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:08.655 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:08.655 DEBUG:boto:Port: 8000 2023-09-22T21:56:08.655 DEBUG:boto:Params: {} 2023-09-22T21:56:08.655 DEBUG:boto:Token: None 2023-09-22T21:56:08.656 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:08 GMT /ffzdqu-61/obj33 2023-09-22T21:56:08.656 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8bTcDlBsjiysaAWATNHAA6cGU9k= 2023-09-22T21:56:08.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8bTcDlBsjiysaAWATNHAA6cGU9k='} 2023-09-22T21:56:08.669 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6994f9e3c11e7ce-00650e0d78-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:08.669 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:08.669 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:08.670 DEBUG:boto:Method: HEAD 2023-09-22T21:56:08.670 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:08.670 DEBUG:boto:Data: 2023-09-22T21:56:08.670 DEBUG:boto:Headers: {} 2023-09-22T21:56:08.670 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:08.670 DEBUG:boto:Port: 8000 2023-09-22T21:56:08.670 DEBUG:boto:Params: {} 2023-09-22T21:56:08.670 DEBUG:boto:Token: None 2023-09-22T21:56:08.670 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:08 GMT /ffzdqu-61/ 2023-09-22T21:56:08.671 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QJuMcjeS5lNJK3VyjHXvS1Pr6WM= 2023-09-22T21:56:08.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QJuMcjeS5lNJK3VyjHXvS1Pr6WM=', 'Content-Length': '0'} 2023-09-22T21:56:08.674 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f64b2158a3cbbbd9-00650e0d78-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:08.675 DEBUG:boto:path=/ffzdqu-61/obj44 2023-09-22T21:56:08.675 DEBUG:boto:auth_path=/ffzdqu-61/obj44 2023-09-22T21:56:08.675 DEBUG:boto:Method: PUT 2023-09-22T21:56:08.675 DEBUG:boto:Path: /ffzdqu-61/obj44 2023-09-22T21:56:08.675 DEBUG:boto:Data: 2023-09-22T21:56:08.675 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:08.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:08.676 DEBUG:boto:Port: 8000 2023-09-22T21:56:08.676 DEBUG:boto:Params: {} 2023-09-22T21:56:08.676 DEBUG:boto:Token: None 2023-09-22T21:56:08.676 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:08 GMT /ffzdqu-61/obj44 2023-09-22T21:56:08.676 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dsXjyVmi4MJDYSgIcCeFY0Mxe5E= 2023-09-22T21:56:08.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dsXjyVmi4MJDYSgIcCeFY0Mxe5E='} 2023-09-22T21:56:08.718 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3737bd357c27f6e-00650e0d78-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:08.719 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:08.719 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:08.719 DEBUG:boto:Method: HEAD 2023-09-22T21:56:08.719 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:08.719 DEBUG:boto:Data: 2023-09-22T21:56:08.719 DEBUG:boto:Headers: {} 2023-09-22T21:56:08.719 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:08.719 DEBUG:boto:Port: 8000 2023-09-22T21:56:08.719 DEBUG:boto:Params: {} 2023-09-22T21:56:08.720 DEBUG:boto:Token: None 2023-09-22T21:56:08.720 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:08 GMT /ffzdqu-61/ 2023-09-22T21:56:08.720 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QJuMcjeS5lNJK3VyjHXvS1Pr6WM= 2023-09-22T21:56:08.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QJuMcjeS5lNJK3VyjHXvS1Pr6WM=', 'Content-Length': '0'} 2023-09-22T21:56:08.724 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000201502fb869a3367-00650e0d78-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:08.724 DEBUG:boto:path=/ffzdqu-61/obj55 2023-09-22T21:56:08.724 DEBUG:boto:auth_path=/ffzdqu-61/obj55 2023-09-22T21:56:08.724 DEBUG:boto:Method: PUT 2023-09-22T21:56:08.724 DEBUG:boto:Path: /ffzdqu-61/obj55 2023-09-22T21:56:08.725 DEBUG:boto:Data: 2023-09-22T21:56:08.725 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:08.725 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:08.725 DEBUG:boto:Port: 8000 2023-09-22T21:56:08.725 DEBUG:boto:Params: {} 2023-09-22T21:56:08.725 DEBUG:boto:Token: None 2023-09-22T21:56:08.725 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:08 GMT /ffzdqu-61/obj55 2023-09-22T21:56:08.725 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3TC7OZkLBE+/WCw4SCjhu9eYrDA= 2023-09-22T21:56:08.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3TC7OZkLBE+/WCw4SCjhu9eYrDA='} 2023-09-22T21:56:08.734 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba88476327ef59c9-00650e0d78-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:08.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:56:08.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:56:08.735 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-61 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:56:08.781 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:56:08.781 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:56:08.837 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:56:08.837 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-61 2023-09-22T21:56:08.844 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 5 2023-09-22T21:56:08.891 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:08.891 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:08.891 DEBUG:boto:Method: HEAD 2023-09-22T21:56:08.892 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:08.892 DEBUG:boto:Data: 2023-09-22T21:56:08.892 DEBUG:boto:Headers: {} 2023-09-22T21:56:08.892 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:08.892 DEBUG:boto:Port: 8000 2023-09-22T21:56:08.892 DEBUG:boto:Params: {} 2023-09-22T21:56:08.892 DEBUG:boto:Token: None 2023-09-22T21:56:08.892 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:08 GMT /ffzdqu-61/ 2023-09-22T21:56:08.893 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QJuMcjeS5lNJK3VyjHXvS1Pr6WM= 2023-09-22T21:56:08.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QJuMcjeS5lNJK3VyjHXvS1Pr6WM=', 'Content-Length': '0'} 2023-09-22T21:56:08.895 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000900de60940bdba8d-00650e0d78-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:08.896 DEBUG:boto:path=/ffzdqu-61/obj17 2023-09-22T21:56:08.896 DEBUG:boto:auth_path=/ffzdqu-61/obj17 2023-09-22T21:56:08.896 DEBUG:boto:Method: PUT 2023-09-22T21:56:08.896 DEBUG:boto:Path: /ffzdqu-61/obj17 2023-09-22T21:56:08.896 DEBUG:boto:Data: 2023-09-22T21:56:08.896 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:08.896 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:08.896 DEBUG:boto:Port: 8000 2023-09-22T21:56:08.897 DEBUG:boto:Params: {} 2023-09-22T21:56:08.897 DEBUG:boto:Token: None 2023-09-22T21:56:08.897 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:08 GMT /ffzdqu-61/obj17 2023-09-22T21:56:08.897 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:767xDUp0XAJPMmrSEE+XS7fmkcY= 2023-09-22T21:56:08.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:767xDUp0XAJPMmrSEE+XS7fmkcY='} 2023-09-22T21:56:08.938 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007407442441fa1f4f-00650e0d78-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:08.939 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:08.939 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:08.939 DEBUG:boto:Method: HEAD 2023-09-22T21:56:08.939 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:08.939 DEBUG:boto:Data: 2023-09-22T21:56:08.939 DEBUG:boto:Headers: {} 2023-09-22T21:56:08.939 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:08.939 DEBUG:boto:Port: 8000 2023-09-22T21:56:08.940 DEBUG:boto:Params: {} 2023-09-22T21:56:08.940 DEBUG:boto:Token: None 2023-09-22T21:56:08.940 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:08 GMT /ffzdqu-61/ 2023-09-22T21:56:08.940 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QJuMcjeS5lNJK3VyjHXvS1Pr6WM= 2023-09-22T21:56:08.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QJuMcjeS5lNJK3VyjHXvS1Pr6WM=', 'Content-Length': '0'} 2023-09-22T21:56:08.943 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c22ade80105b439-00650e0d78-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:08.943 DEBUG:boto:path=/ffzdqu-61/obj34 2023-09-22T21:56:08.943 DEBUG:boto:auth_path=/ffzdqu-61/obj34 2023-09-22T21:56:08.944 DEBUG:boto:Method: PUT 2023-09-22T21:56:08.944 DEBUG:boto:Path: /ffzdqu-61/obj34 2023-09-22T21:56:08.944 DEBUG:boto:Data: 2023-09-22T21:56:08.944 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:08.944 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:08.944 DEBUG:boto:Port: 8000 2023-09-22T21:56:08.944 DEBUG:boto:Params: {} 2023-09-22T21:56:08.944 DEBUG:boto:Token: None 2023-09-22T21:56:08.945 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:08 GMT /ffzdqu-61/obj34 2023-09-22T21:56:08.945 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IxW8YzhMTdK8CmOzkW0haCT7Lhs= 2023-09-22T21:56:08.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IxW8YzhMTdK8CmOzkW0haCT7Lhs='} 2023-09-22T21:56:08.952 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055824f77e656befe-00650e0d78-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:08.953 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:08.953 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:08.953 DEBUG:boto:Method: HEAD 2023-09-22T21:56:08.953 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:08.953 DEBUG:boto:Data: 2023-09-22T21:56:08.953 DEBUG:boto:Headers: {} 2023-09-22T21:56:08.953 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:08.953 DEBUG:boto:Port: 8000 2023-09-22T21:56:08.954 DEBUG:boto:Params: {} 2023-09-22T21:56:08.954 DEBUG:boto:Token: None 2023-09-22T21:56:08.954 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:08 GMT /ffzdqu-61/ 2023-09-22T21:56:08.954 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QJuMcjeS5lNJK3VyjHXvS1Pr6WM= 2023-09-22T21:56:08.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QJuMcjeS5lNJK3VyjHXvS1Pr6WM=', 'Content-Length': '0'} 2023-09-22T21:56:08.957 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a7551efeed9453a-00650e0d78-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:08.958 DEBUG:boto:path=/ffzdqu-61/obj51 2023-09-22T21:56:08.958 DEBUG:boto:auth_path=/ffzdqu-61/obj51 2023-09-22T21:56:08.958 DEBUG:boto:Method: PUT 2023-09-22T21:56:08.958 DEBUG:boto:Path: /ffzdqu-61/obj51 2023-09-22T21:56:08.958 DEBUG:boto:Data: 2023-09-22T21:56:08.958 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:08.958 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:08.959 DEBUG:boto:Port: 8000 2023-09-22T21:56:08.959 DEBUG:boto:Params: {} 2023-09-22T21:56:08.959 DEBUG:boto:Token: None 2023-09-22T21:56:08.959 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:08 GMT /ffzdqu-61/obj51 2023-09-22T21:56:08.959 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E+IqACyF6Fjs6h/GeCZQpSnYD5M= 2023-09-22T21:56:08.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E+IqACyF6Fjs6h/GeCZQpSnYD5M='} 2023-09-22T21:56:09.002 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c627ac3994b2e04-00650e0d78-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.003 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:09.003 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:09.003 DEBUG:boto:Method: HEAD 2023-09-22T21:56:09.003 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:09.003 DEBUG:boto:Data: 2023-09-22T21:56:09.003 DEBUG:boto:Headers: {} 2023-09-22T21:56:09.003 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.003 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.004 DEBUG:boto:Params: {} 2023-09-22T21:56:09.004 DEBUG:boto:Token: None 2023-09-22T21:56:09.004 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/ 2023-09-22T21:56:09.004 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4= 2023-09-22T21:56:09.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4=', 'Content-Length': '0'} 2023-09-22T21:56:09.009 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000367417db25c4b174-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.009 DEBUG:boto:path=/ffzdqu-61/obj68 2023-09-22T21:56:09.009 DEBUG:boto:auth_path=/ffzdqu-61/obj68 2023-09-22T21:56:09.009 DEBUG:boto:Method: PUT 2023-09-22T21:56:09.009 DEBUG:boto:Path: /ffzdqu-61/obj68 2023-09-22T21:56:09.009 DEBUG:boto:Data: 2023-09-22T21:56:09.009 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:09.010 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.010 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.010 DEBUG:boto:Params: {} 2023-09-22T21:56:09.010 DEBUG:boto:Token: None 2023-09-22T21:56:09.010 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/obj68 2023-09-22T21:56:09.010 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TclhTHD1l+j2RiyTSSXSK07UqDU= 2023-09-22T21:56:09.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TclhTHD1l+j2RiyTSSXSK07UqDU='} 2023-09-22T21:56:09.023 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa98bc549889c2c1-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.023 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:09.023 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:09.023 DEBUG:boto:Method: HEAD 2023-09-22T21:56:09.023 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:09.024 DEBUG:boto:Data: 2023-09-22T21:56:09.024 DEBUG:boto:Headers: {} 2023-09-22T21:56:09.024 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.024 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.024 DEBUG:boto:Params: {} 2023-09-22T21:56:09.024 DEBUG:boto:Token: None 2023-09-22T21:56:09.024 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/ 2023-09-22T21:56:09.024 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4= 2023-09-22T21:56:09.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4=', 'Content-Length': '0'} 2023-09-22T21:56:09.029 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a8b28dac223f84f-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.030 DEBUG:boto:path=/ffzdqu-61/obj85 2023-09-22T21:56:09.030 DEBUG:boto:auth_path=/ffzdqu-61/obj85 2023-09-22T21:56:09.030 DEBUG:boto:Method: PUT 2023-09-22T21:56:09.030 DEBUG:boto:Path: /ffzdqu-61/obj85 2023-09-22T21:56:09.030 DEBUG:boto:Data: 2023-09-22T21:56:09.030 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:09.030 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.031 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.031 DEBUG:boto:Params: {} 2023-09-22T21:56:09.031 DEBUG:boto:Token: None 2023-09-22T21:56:09.031 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/obj85 2023-09-22T21:56:09.031 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yNIyLjpssWr0bZeXkS2TgRMRj94= 2023-09-22T21:56:09.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yNIyLjpssWr0bZeXkS2TgRMRj94='} 2023-09-22T21:56:09.074 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bcee438c0972f973-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:56:09.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:56:09.075 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-61 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:56:09.120 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:56:09.121 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:56:09.179 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:56:09.179 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-61 2023-09-22T21:56:09.189 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 10 2023-09-22T21:56:09.239 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-09-22T21:56:09.249 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:09.249 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:09.249 DEBUG:boto:Method: HEAD 2023-09-22T21:56:09.249 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:09.250 DEBUG:boto:Data: 2023-09-22T21:56:09.250 DEBUG:boto:Headers: {} 2023-09-22T21:56:09.250 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.250 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.250 DEBUG:boto:Params: {} 2023-09-22T21:56:09.250 DEBUG:boto:Token: None 2023-09-22T21:56:09.250 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/ 2023-09-22T21:56:09.250 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4= 2023-09-22T21:56:09.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4=', 'Content-Length': '0'} 2023-09-22T21:56:09.254 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002703be8a3198edb-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.254 DEBUG:boto:path=/ffzdqu-61/obj19 2023-09-22T21:56:09.254 DEBUG:boto:auth_path=/ffzdqu-61/obj19 2023-09-22T21:56:09.255 DEBUG:boto:Method: PUT 2023-09-22T21:56:09.255 DEBUG:boto:Path: /ffzdqu-61/obj19 2023-09-22T21:56:09.255 DEBUG:boto:Data: 2023-09-22T21:56:09.255 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:09.255 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.255 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.255 DEBUG:boto:Params: {} 2023-09-22T21:56:09.255 DEBUG:boto:Token: None 2023-09-22T21:56:09.256 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/obj19 2023-09-22T21:56:09.256 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fFwmiDMl8Kn8TpEEkoSE/IvEIzc= 2023-09-22T21:56:09.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fFwmiDMl8Kn8TpEEkoSE/IvEIzc='} 2023-09-22T21:56:09.263 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc902ab38789a51d-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.263 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:09.263 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:09.263 DEBUG:boto:Method: HEAD 2023-09-22T21:56:09.263 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:09.263 DEBUG:boto:Data: 2023-09-22T21:56:09.264 DEBUG:boto:Headers: {} 2023-09-22T21:56:09.264 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.264 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.264 DEBUG:boto:Params: {} 2023-09-22T21:56:09.264 DEBUG:boto:Token: None 2023-09-22T21:56:09.264 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/ 2023-09-22T21:56:09.264 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4= 2023-09-22T21:56:09.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4=', 'Content-Length': '0'} 2023-09-22T21:56:09.267 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f547cb5f23d63430-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.267 DEBUG:boto:path=/ffzdqu-61/obj38 2023-09-22T21:56:09.267 DEBUG:boto:auth_path=/ffzdqu-61/obj38 2023-09-22T21:56:09.268 DEBUG:boto:Method: PUT 2023-09-22T21:56:09.268 DEBUG:boto:Path: /ffzdqu-61/obj38 2023-09-22T21:56:09.268 DEBUG:boto:Data: 2023-09-22T21:56:09.268 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:09.268 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.268 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.268 DEBUG:boto:Params: {} 2023-09-22T21:56:09.268 DEBUG:boto:Token: None 2023-09-22T21:56:09.269 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/obj38 2023-09-22T21:56:09.269 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/tZQvfgeAarfDuVo00s+rbFG0II= 2023-09-22T21:56:09.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/tZQvfgeAarfDuVo00s+rbFG0II='} 2023-09-22T21:56:09.273 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005a0ab03d7f480ee-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.273 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:09.273 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:09.274 DEBUG:boto:Method: HEAD 2023-09-22T21:56:09.274 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:09.274 DEBUG:boto:Data: 2023-09-22T21:56:09.274 DEBUG:boto:Headers: {} 2023-09-22T21:56:09.274 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.274 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.274 DEBUG:boto:Params: {} 2023-09-22T21:56:09.274 DEBUG:boto:Token: None 2023-09-22T21:56:09.274 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/ 2023-09-22T21:56:09.275 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4= 2023-09-22T21:56:09.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4=', 'Content-Length': '0'} 2023-09-22T21:56:09.278 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc9ed8a7eecbfaf0-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.278 DEBUG:boto:path=/ffzdqu-61/obj57 2023-09-22T21:56:09.278 DEBUG:boto:auth_path=/ffzdqu-61/obj57 2023-09-22T21:56:09.278 DEBUG:boto:Method: PUT 2023-09-22T21:56:09.279 DEBUG:boto:Path: /ffzdqu-61/obj57 2023-09-22T21:56:09.279 DEBUG:boto:Data: 2023-09-22T21:56:09.279 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:09.279 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.279 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.279 DEBUG:boto:Params: {} 2023-09-22T21:56:09.279 DEBUG:boto:Token: None 2023-09-22T21:56:09.279 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/obj57 2023-09-22T21:56:09.279 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zFAYwlP5lQILjBRljvTHQJQ5/lU= 2023-09-22T21:56:09.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zFAYwlP5lQILjBRljvTHQJQ5/lU='} 2023-09-22T21:56:09.326 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cac3477b8a0a3cab-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.327 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:09.327 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:09.327 DEBUG:boto:Method: HEAD 2023-09-22T21:56:09.327 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:09.327 DEBUG:boto:Data: 2023-09-22T21:56:09.327 DEBUG:boto:Headers: {} 2023-09-22T21:56:09.327 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.327 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.328 DEBUG:boto:Params: {} 2023-09-22T21:56:09.328 DEBUG:boto:Token: None 2023-09-22T21:56:09.328 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/ 2023-09-22T21:56:09.328 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4= 2023-09-22T21:56:09.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4=', 'Content-Length': '0'} 2023-09-22T21:56:09.332 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097467e62cd0f9854-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.332 DEBUG:boto:path=/ffzdqu-61/obj76 2023-09-22T21:56:09.333 DEBUG:boto:auth_path=/ffzdqu-61/obj76 2023-09-22T21:56:09.333 DEBUG:boto:Method: PUT 2023-09-22T21:56:09.333 DEBUG:boto:Path: /ffzdqu-61/obj76 2023-09-22T21:56:09.333 DEBUG:boto:Data: 2023-09-22T21:56:09.333 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:09.333 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.333 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.333 DEBUG:boto:Params: {} 2023-09-22T21:56:09.334 DEBUG:boto:Token: None 2023-09-22T21:56:09.334 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/obj76 2023-09-22T21:56:09.334 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KSRWhNBL27y5LpBnVE+7uwfr+sQ= 2023-09-22T21:56:09.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KSRWhNBL27y5LpBnVE+7uwfr+sQ='} 2023-09-22T21:56:09.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db0add04e040e39d-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.346 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:09.346 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:09.346 DEBUG:boto:Method: HEAD 2023-09-22T21:56:09.346 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:09.347 DEBUG:boto:Data: 2023-09-22T21:56:09.347 DEBUG:boto:Headers: {} 2023-09-22T21:56:09.347 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.347 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.347 DEBUG:boto:Params: {} 2023-09-22T21:56:09.347 DEBUG:boto:Token: None 2023-09-22T21:56:09.347 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/ 2023-09-22T21:56:09.347 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4= 2023-09-22T21:56:09.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4=', 'Content-Length': '0'} 2023-09-22T21:56:09.352 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053f28d766281e475-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.352 DEBUG:boto:path=/ffzdqu-61/obj95 2023-09-22T21:56:09.353 DEBUG:boto:auth_path=/ffzdqu-61/obj95 2023-09-22T21:56:09.353 DEBUG:boto:Method: PUT 2023-09-22T21:56:09.353 DEBUG:boto:Path: /ffzdqu-61/obj95 2023-09-22T21:56:09.353 DEBUG:boto:Data: 2023-09-22T21:56:09.353 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:09.353 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.353 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.353 DEBUG:boto:Params: {} 2023-09-22T21:56:09.353 DEBUG:boto:Token: None 2023-09-22T21:56:09.354 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/obj95 2023-09-22T21:56:09.354 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GwED+kp/kLm0cYJa1zwQpmVjoDs= 2023-09-22T21:56:09.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GwED+kp/kLm0cYJa1zwQpmVjoDs='} 2023-09-22T21:56:09.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000050853d4474df1c7d-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:56:09.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:56:09.399 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-61 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:56:09.444 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:56:09.445 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:56:09.511 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:56:09.511 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-61 2023-09-22T21:56:09.522 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 15 2023-09-22T21:56:09.581 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:09.581 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:09.581 DEBUG:boto:Method: HEAD 2023-09-22T21:56:09.581 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:09.582 DEBUG:boto:Data: 2023-09-22T21:56:09.582 DEBUG:boto:Headers: {} 2023-09-22T21:56:09.582 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.582 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.582 DEBUG:boto:Params: {} 2023-09-22T21:56:09.582 DEBUG:boto:Token: None 2023-09-22T21:56:09.582 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/ 2023-09-22T21:56:09.583 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4= 2023-09-22T21:56:09.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4=', 'Content-Length': '0'} 2023-09-22T21:56:09.586 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000fdd908276fedf25-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.586 DEBUG:boto:path=/ffzdqu-61/obj23 2023-09-22T21:56:09.586 DEBUG:boto:auth_path=/ffzdqu-61/obj23 2023-09-22T21:56:09.587 DEBUG:boto:Method: PUT 2023-09-22T21:56:09.587 DEBUG:boto:Path: /ffzdqu-61/obj23 2023-09-22T21:56:09.587 DEBUG:boto:Data: 2023-09-22T21:56:09.587 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:09.587 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.587 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.587 DEBUG:boto:Params: {} 2023-09-22T21:56:09.587 DEBUG:boto:Token: None 2023-09-22T21:56:09.588 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/obj23 2023-09-22T21:56:09.588 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yosqja0OlIm17uXZrw/KyW+44Ds= 2023-09-22T21:56:09.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yosqja0OlIm17uXZrw/KyW+44Ds='} 2023-09-22T21:56:09.594 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006e1f72036c7078b6-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.594 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:09.595 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:09.595 DEBUG:boto:Method: HEAD 2023-09-22T21:56:09.595 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:09.595 DEBUG:boto:Data: 2023-09-22T21:56:09.595 DEBUG:boto:Headers: {} 2023-09-22T21:56:09.595 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.595 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.595 DEBUG:boto:Params: {} 2023-09-22T21:56:09.595 DEBUG:boto:Token: None 2023-09-22T21:56:09.596 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/ 2023-09-22T21:56:09.596 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4= 2023-09-22T21:56:09.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4=', 'Content-Length': '0'} 2023-09-22T21:56:09.598 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c9e1e4e0f36fb23-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.599 DEBUG:boto:path=/ffzdqu-61/obj46 2023-09-22T21:56:09.599 DEBUG:boto:auth_path=/ffzdqu-61/obj46 2023-09-22T21:56:09.599 DEBUG:boto:Method: PUT 2023-09-22T21:56:09.599 DEBUG:boto:Path: /ffzdqu-61/obj46 2023-09-22T21:56:09.599 DEBUG:boto:Data: 2023-09-22T21:56:09.599 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:09.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.599 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.600 DEBUG:boto:Params: {} 2023-09-22T21:56:09.600 DEBUG:boto:Token: None 2023-09-22T21:56:09.600 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/obj46 2023-09-22T21:56:09.600 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mugczRy4dKq1fGrnXzFBAS4W8mQ= 2023-09-22T21:56:09.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mugczRy4dKq1fGrnXzFBAS4W8mQ='} 2023-09-22T21:56:09.606 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000018a697545f9856dc-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.606 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:09.606 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:09.606 DEBUG:boto:Method: HEAD 2023-09-22T21:56:09.607 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:09.607 DEBUG:boto:Data: 2023-09-22T21:56:09.607 DEBUG:boto:Headers: {} 2023-09-22T21:56:09.607 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.607 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.607 DEBUG:boto:Params: {} 2023-09-22T21:56:09.607 DEBUG:boto:Token: None 2023-09-22T21:56:09.607 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/ 2023-09-22T21:56:09.607 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4= 2023-09-22T21:56:09.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4=', 'Content-Length': '0'} 2023-09-22T21:56:09.610 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cafde39298db208d-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.610 DEBUG:boto:path=/ffzdqu-61/obj69 2023-09-22T21:56:09.611 DEBUG:boto:auth_path=/ffzdqu-61/obj69 2023-09-22T21:56:09.611 DEBUG:boto:Method: PUT 2023-09-22T21:56:09.611 DEBUG:boto:Path: /ffzdqu-61/obj69 2023-09-22T21:56:09.611 DEBUG:boto:Data: 2023-09-22T21:56:09.611 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:09.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.611 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.611 DEBUG:boto:Params: {} 2023-09-22T21:56:09.611 DEBUG:boto:Token: None 2023-09-22T21:56:09.612 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/obj69 2023-09-22T21:56:09.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oQRELds2gGSV4dubpkBpPA0u8V8= 2023-09-22T21:56:09.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oQRELds2gGSV4dubpkBpPA0u8V8='} 2023-09-22T21:56:09.654 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002ae5d18380a589cf-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.655 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:09.655 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:09.655 DEBUG:boto:Method: HEAD 2023-09-22T21:56:09.655 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:09.655 DEBUG:boto:Data: 2023-09-22T21:56:09.655 DEBUG:boto:Headers: {} 2023-09-22T21:56:09.655 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.655 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.655 DEBUG:boto:Params: {} 2023-09-22T21:56:09.656 DEBUG:boto:Token: None 2023-09-22T21:56:09.656 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/ 2023-09-22T21:56:09.656 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4= 2023-09-22T21:56:09.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4=', 'Content-Length': '0'} 2023-09-22T21:56:09.661 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5e7ddc93b870086-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.661 DEBUG:boto:path=/ffzdqu-61/obj92 2023-09-22T21:56:09.661 DEBUG:boto:auth_path=/ffzdqu-61/obj92 2023-09-22T21:56:09.661 DEBUG:boto:Method: PUT 2023-09-22T21:56:09.662 DEBUG:boto:Path: /ffzdqu-61/obj92 2023-09-22T21:56:09.662 DEBUG:boto:Data: 2023-09-22T21:56:09.662 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:09.662 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.662 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.662 DEBUG:boto:Params: {} 2023-09-22T21:56:09.662 DEBUG:boto:Token: None 2023-09-22T21:56:09.662 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/obj92 2023-09-22T21:56:09.662 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M65sR/AuBZU8qSf8L2rK8iVQqhI= 2023-09-22T21:56:09.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M65sR/AuBZU8qSf8L2rK8iVQqhI='} 2023-09-22T21:56:09.673 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000247abf89db7f62f5-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.673 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:09.673 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:09.673 DEBUG:boto:Method: HEAD 2023-09-22T21:56:09.673 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:09.674 DEBUG:boto:Data: 2023-09-22T21:56:09.674 DEBUG:boto:Headers: {} 2023-09-22T21:56:09.674 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.674 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.674 DEBUG:boto:Params: {} 2023-09-22T21:56:09.674 DEBUG:boto:Token: None 2023-09-22T21:56:09.674 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/ 2023-09-22T21:56:09.674 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4= 2023-09-22T21:56:09.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4=', 'Content-Length': '0'} 2023-09-22T21:56:09.680 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6bd1f4719af2827-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.680 DEBUG:boto:path=/ffzdqu-61/obj115 2023-09-22T21:56:09.680 DEBUG:boto:auth_path=/ffzdqu-61/obj115 2023-09-22T21:56:09.680 DEBUG:boto:Method: PUT 2023-09-22T21:56:09.680 DEBUG:boto:Path: /ffzdqu-61/obj115 2023-09-22T21:56:09.680 DEBUG:boto:Data: 2023-09-22T21:56:09.680 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:09.681 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.681 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.681 DEBUG:boto:Params: {} 2023-09-22T21:56:09.681 DEBUG:boto:Token: None 2023-09-22T21:56:09.681 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/obj115 2023-09-22T21:56:09.681 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gEZG6sluE9US25ti0zg0IxneP40= 2023-09-22T21:56:09.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gEZG6sluE9US25ti0zg0IxneP40='} 2023-09-22T21:56:09.726 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b9eb70d438ce26d2-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:56:09.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:56:09.727 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-61 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:56:09.773 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:56:09.773 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:56:09.834 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:56:09.834 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-61 2023-09-22T21:56:09.848 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 20 2023-09-22T21:56:09.920 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:09.920 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:09.920 DEBUG:boto:Method: HEAD 2023-09-22T21:56:09.920 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:09.920 DEBUG:boto:Data: 2023-09-22T21:56:09.920 DEBUG:boto:Headers: {} 2023-09-22T21:56:09.920 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.921 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.921 DEBUG:boto:Params: {} 2023-09-22T21:56:09.921 DEBUG:boto:Token: None 2023-09-22T21:56:09.921 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/ 2023-09-22T21:56:09.921 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4= 2023-09-22T21:56:09.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4=', 'Content-Length': '0'} 2023-09-22T21:56:09.925 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074a7b9b28c83c4ff-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.925 DEBUG:boto:path=/ffzdqu-61/obj29 2023-09-22T21:56:09.925 DEBUG:boto:auth_path=/ffzdqu-61/obj29 2023-09-22T21:56:09.925 DEBUG:boto:Method: PUT 2023-09-22T21:56:09.925 DEBUG:boto:Path: /ffzdqu-61/obj29 2023-09-22T21:56:09.925 DEBUG:boto:Data: 2023-09-22T21:56:09.926 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:09.926 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.926 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.926 DEBUG:boto:Params: {} 2023-09-22T21:56:09.926 DEBUG:boto:Token: None 2023-09-22T21:56:09.926 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/obj29 2023-09-22T21:56:09.926 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NVwp220gWKtZ69FzMnfZcibo+6g= 2023-09-22T21:56:09.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NVwp220gWKtZ69FzMnfZcibo+6g='} 2023-09-22T21:56:09.932 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a9f324a48c21cff-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.932 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:09.933 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:09.933 DEBUG:boto:Method: HEAD 2023-09-22T21:56:09.933 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:09.933 DEBUG:boto:Data: 2023-09-22T21:56:09.933 DEBUG:boto:Headers: {} 2023-09-22T21:56:09.933 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.933 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.933 DEBUG:boto:Params: {} 2023-09-22T21:56:09.933 DEBUG:boto:Token: None 2023-09-22T21:56:09.933 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/ 2023-09-22T21:56:09.934 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4= 2023-09-22T21:56:09.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4=', 'Content-Length': '0'} 2023-09-22T21:56:09.937 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ecc9da02bcec113d-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.937 DEBUG:boto:path=/ffzdqu-61/obj58 2023-09-22T21:56:09.937 DEBUG:boto:auth_path=/ffzdqu-61/obj58 2023-09-22T21:56:09.937 DEBUG:boto:Method: PUT 2023-09-22T21:56:09.937 DEBUG:boto:Path: /ffzdqu-61/obj58 2023-09-22T21:56:09.937 DEBUG:boto:Data: 2023-09-22T21:56:09.938 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:09.938 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.938 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.938 DEBUG:boto:Params: {} 2023-09-22T21:56:09.938 DEBUG:boto:Token: None 2023-09-22T21:56:09.938 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/obj58 2023-09-22T21:56:09.938 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jUHL9vSIPOLQYURt3ECNXaqwsBw= 2023-09-22T21:56:09.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jUHL9vSIPOLQYURt3ECNXaqwsBw='} 2023-09-22T21:56:09.944 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f096f273c9b73a73-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.944 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:09.945 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:09.945 DEBUG:boto:Method: HEAD 2023-09-22T21:56:09.945 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:09.945 DEBUG:boto:Data: 2023-09-22T21:56:09.945 DEBUG:boto:Headers: {} 2023-09-22T21:56:09.945 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.945 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.945 DEBUG:boto:Params: {} 2023-09-22T21:56:09.945 DEBUG:boto:Token: None 2023-09-22T21:56:09.945 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/ 2023-09-22T21:56:09.946 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4= 2023-09-22T21:56:09.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4=', 'Content-Length': '0'} 2023-09-22T21:56:09.949 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2e9103c7ab06bb1-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.949 DEBUG:boto:path=/ffzdqu-61/obj87 2023-09-22T21:56:09.949 DEBUG:boto:auth_path=/ffzdqu-61/obj87 2023-09-22T21:56:09.949 DEBUG:boto:Method: PUT 2023-09-22T21:56:09.949 DEBUG:boto:Path: /ffzdqu-61/obj87 2023-09-22T21:56:09.950 DEBUG:boto:Data: 2023-09-22T21:56:09.950 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:09.950 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.950 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.950 DEBUG:boto:Params: {} 2023-09-22T21:56:09.950 DEBUG:boto:Token: None 2023-09-22T21:56:09.950 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/obj87 2023-09-22T21:56:09.950 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H930txKEYBRmLwQsRw3GBsiukgc= 2023-09-22T21:56:09.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H930txKEYBRmLwQsRw3GBsiukgc='} 2023-09-22T21:56:09.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000063d92726fbcff4ef-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:09.995 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:09.995 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:09.995 DEBUG:boto:Method: HEAD 2023-09-22T21:56:09.995 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:09.995 DEBUG:boto:Data: 2023-09-22T21:56:09.995 DEBUG:boto:Headers: {} 2023-09-22T21:56:09.995 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:09.995 DEBUG:boto:Port: 8000 2023-09-22T21:56:09.995 DEBUG:boto:Params: {} 2023-09-22T21:56:09.996 DEBUG:boto:Token: None 2023-09-22T21:56:09.996 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:09 GMT /ffzdqu-61/ 2023-09-22T21:56:09.996 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4= 2023-09-22T21:56:09.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:E9fZAJM1ZrKg+iWGK/ehJP9veI4=', 'Content-Length': '0'} 2023-09-22T21:56:10.000 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca0934661c7cb735-00650e0d79-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.000 DEBUG:boto:path=/ffzdqu-61/obj116 2023-09-22T21:56:10.000 DEBUG:boto:auth_path=/ffzdqu-61/obj116 2023-09-22T21:56:10.001 DEBUG:boto:Method: PUT 2023-09-22T21:56:10.001 DEBUG:boto:Path: /ffzdqu-61/obj116 2023-09-22T21:56:10.001 DEBUG:boto:Data: 2023-09-22T21:56:10.001 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:10.001 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.001 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.001 DEBUG:boto:Params: {} 2023-09-22T21:56:10.001 DEBUG:boto:Token: None 2023-09-22T21:56:10.002 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/obj116 2023-09-22T21:56:10.002 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XUjEydoeJKcMC8b5l4V12Jy0xi0= 2023-09-22T21:56:10.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XUjEydoeJKcMC8b5l4V12Jy0xi0='} 2023-09-22T21:56:10.009 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000edb079990340372b-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.009 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:10.009 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:10.009 DEBUG:boto:Method: HEAD 2023-09-22T21:56:10.009 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:10.009 DEBUG:boto:Data: 2023-09-22T21:56:10.009 DEBUG:boto:Headers: {} 2023-09-22T21:56:10.010 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.010 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.010 DEBUG:boto:Params: {} 2023-09-22T21:56:10.010 DEBUG:boto:Token: None 2023-09-22T21:56:10.010 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/ 2023-09-22T21:56:10.010 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U= 2023-09-22T21:56:10.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U=', 'Content-Length': '0'} 2023-09-22T21:56:10.015 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000adc0f88c78fa7d87-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.015 DEBUG:boto:path=/ffzdqu-61/obj145 2023-09-22T21:56:10.015 DEBUG:boto:auth_path=/ffzdqu-61/obj145 2023-09-22T21:56:10.015 DEBUG:boto:Method: PUT 2023-09-22T21:56:10.015 DEBUG:boto:Path: /ffzdqu-61/obj145 2023-09-22T21:56:10.016 DEBUG:boto:Data: 2023-09-22T21:56:10.016 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:10.016 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.016 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.016 DEBUG:boto:Params: {} 2023-09-22T21:56:10.016 DEBUG:boto:Token: None 2023-09-22T21:56:10.016 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/obj145 2023-09-22T21:56:10.016 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:szSq+evcd6+/VCW9gvlPGCJ87sg= 2023-09-22T21:56:10.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:szSq+evcd6+/VCW9gvlPGCJ87sg='} 2023-09-22T21:56:10.062 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a26b02f7b58f15b-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:56:10.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:56:10.063 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-61 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:56:10.109 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:56:10.109 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:56:10.172 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:56:10.172 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-61 2023-09-22T21:56:10.190 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 25 2023-09-22T21:56:10.268 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:10.268 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:10.268 DEBUG:boto:Method: HEAD 2023-09-22T21:56:10.268 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:10.268 DEBUG:boto:Data: 2023-09-22T21:56:10.268 DEBUG:boto:Headers: {} 2023-09-22T21:56:10.269 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.269 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.269 DEBUG:boto:Params: {} 2023-09-22T21:56:10.269 DEBUG:boto:Token: None 2023-09-22T21:56:10.269 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/ 2023-09-22T21:56:10.269 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U= 2023-09-22T21:56:10.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U=', 'Content-Length': '0'} 2023-09-22T21:56:10.273 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000969f54c71272bc90-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.273 DEBUG:boto:path=/ffzdqu-61/obj31 2023-09-22T21:56:10.273 DEBUG:boto:auth_path=/ffzdqu-61/obj31 2023-09-22T21:56:10.273 DEBUG:boto:Method: PUT 2023-09-22T21:56:10.273 DEBUG:boto:Path: /ffzdqu-61/obj31 2023-09-22T21:56:10.274 DEBUG:boto:Data: 2023-09-22T21:56:10.274 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:10.274 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.274 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.274 DEBUG:boto:Params: {} 2023-09-22T21:56:10.274 DEBUG:boto:Token: None 2023-09-22T21:56:10.274 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/obj31 2023-09-22T21:56:10.274 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UDybd5useI0vX7639ZtbDrAnsJA= 2023-09-22T21:56:10.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UDybd5useI0vX7639ZtbDrAnsJA='} 2023-09-22T21:56:10.281 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049dffa5077d6df24-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.281 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:10.281 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:10.281 DEBUG:boto:Method: HEAD 2023-09-22T21:56:10.281 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:10.281 DEBUG:boto:Data: 2023-09-22T21:56:10.281 DEBUG:boto:Headers: {} 2023-09-22T21:56:10.282 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.282 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.282 DEBUG:boto:Params: {} 2023-09-22T21:56:10.282 DEBUG:boto:Token: None 2023-09-22T21:56:10.282 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/ 2023-09-22T21:56:10.282 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U= 2023-09-22T21:56:10.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U=', 'Content-Length': '0'} 2023-09-22T21:56:10.286 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009801cf76ddb244dc-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.286 DEBUG:boto:path=/ffzdqu-61/obj62 2023-09-22T21:56:10.286 DEBUG:boto:auth_path=/ffzdqu-61/obj62 2023-09-22T21:56:10.286 DEBUG:boto:Method: PUT 2023-09-22T21:56:10.286 DEBUG:boto:Path: /ffzdqu-61/obj62 2023-09-22T21:56:10.286 DEBUG:boto:Data: 2023-09-22T21:56:10.286 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:10.287 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.287 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.287 DEBUG:boto:Params: {} 2023-09-22T21:56:10.287 DEBUG:boto:Token: None 2023-09-22T21:56:10.287 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/obj62 2023-09-22T21:56:10.287 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ttl4sq3WMy0kq7zzwRckjx1H/Nc= 2023-09-22T21:56:10.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ttl4sq3WMy0kq7zzwRckjx1H/Nc='} 2023-09-22T21:56:10.293 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ef16544f89afced-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.294 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:10.294 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:10.294 DEBUG:boto:Method: HEAD 2023-09-22T21:56:10.294 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:10.294 DEBUG:boto:Data: 2023-09-22T21:56:10.294 DEBUG:boto:Headers: {} 2023-09-22T21:56:10.294 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.294 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.295 DEBUG:boto:Params: {} 2023-09-22T21:56:10.295 DEBUG:boto:Token: None 2023-09-22T21:56:10.295 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/ 2023-09-22T21:56:10.295 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U= 2023-09-22T21:56:10.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U=', 'Content-Length': '0'} 2023-09-22T21:56:10.299 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000875a7a68eccd73bb-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.299 DEBUG:boto:path=/ffzdqu-61/obj93 2023-09-22T21:56:10.299 DEBUG:boto:auth_path=/ffzdqu-61/obj93 2023-09-22T21:56:10.299 DEBUG:boto:Method: PUT 2023-09-22T21:56:10.299 DEBUG:boto:Path: /ffzdqu-61/obj93 2023-09-22T21:56:10.299 DEBUG:boto:Data: 2023-09-22T21:56:10.300 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:10.300 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.300 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.300 DEBUG:boto:Params: {} 2023-09-22T21:56:10.300 DEBUG:boto:Token: None 2023-09-22T21:56:10.300 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/obj93 2023-09-22T21:56:10.300 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LeMVHf5/OikrQCEHxR+q/lP6MjY= 2023-09-22T21:56:10.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LeMVHf5/OikrQCEHxR+q/lP6MjY='} 2023-09-22T21:56:10.342 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d7da4bb499daad92-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.343 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:10.343 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:10.343 DEBUG:boto:Method: HEAD 2023-09-22T21:56:10.343 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:10.343 DEBUG:boto:Data: 2023-09-22T21:56:10.343 DEBUG:boto:Headers: {} 2023-09-22T21:56:10.343 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.343 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.343 DEBUG:boto:Params: {} 2023-09-22T21:56:10.344 DEBUG:boto:Token: None 2023-09-22T21:56:10.344 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/ 2023-09-22T21:56:10.344 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U= 2023-09-22T21:56:10.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U=', 'Content-Length': '0'} 2023-09-22T21:56:10.348 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c82bbae63d653e6-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.349 DEBUG:boto:path=/ffzdqu-61/obj124 2023-09-22T21:56:10.349 DEBUG:boto:auth_path=/ffzdqu-61/obj124 2023-09-22T21:56:10.349 DEBUG:boto:Method: PUT 2023-09-22T21:56:10.349 DEBUG:boto:Path: /ffzdqu-61/obj124 2023-09-22T21:56:10.349 DEBUG:boto:Data: 2023-09-22T21:56:10.349 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:10.349 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.349 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.350 DEBUG:boto:Params: {} 2023-09-22T21:56:10.350 DEBUG:boto:Token: None 2023-09-22T21:56:10.350 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/obj124 2023-09-22T21:56:10.350 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ls7CzZ3TjFUnVS6CZETnMxG3UvE= 2023-09-22T21:56:10.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ls7CzZ3TjFUnVS6CZETnMxG3UvE='} 2023-09-22T21:56:10.360 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000015037e01dc7dc9c8-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.361 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:10.361 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:10.361 DEBUG:boto:Method: HEAD 2023-09-22T21:56:10.361 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:10.361 DEBUG:boto:Data: 2023-09-22T21:56:10.361 DEBUG:boto:Headers: {} 2023-09-22T21:56:10.361 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.361 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.361 DEBUG:boto:Params: {} 2023-09-22T21:56:10.362 DEBUG:boto:Token: None 2023-09-22T21:56:10.362 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/ 2023-09-22T21:56:10.362 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U= 2023-09-22T21:56:10.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U=', 'Content-Length': '0'} 2023-09-22T21:56:10.368 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000399a6378278828b7-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.368 DEBUG:boto:path=/ffzdqu-61/obj155 2023-09-22T21:56:10.368 DEBUG:boto:auth_path=/ffzdqu-61/obj155 2023-09-22T21:56:10.368 DEBUG:boto:Method: PUT 2023-09-22T21:56:10.368 DEBUG:boto:Path: /ffzdqu-61/obj155 2023-09-22T21:56:10.369 DEBUG:boto:Data: 2023-09-22T21:56:10.369 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:10.369 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.369 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.369 DEBUG:boto:Params: {} 2023-09-22T21:56:10.369 DEBUG:boto:Token: None 2023-09-22T21:56:10.369 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/obj155 2023-09-22T21:56:10.369 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:W6dC/e01PUj5LSnoEmehzGpNklg= 2023-09-22T21:56:10.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:W6dC/e01PUj5LSnoEmehzGpNklg='} 2023-09-22T21:56:10.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aa4b7e1edb5f0dd4-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:56:10.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:56:10.419 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-61 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:56:10.465 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:56:10.465 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:56:10.530 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T21:56:10.530 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-61 2023-09-22T21:56:10.549 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 30 2023-09-22T21:56:10.639 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:10.639 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:10.639 DEBUG:boto:Method: HEAD 2023-09-22T21:56:10.639 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:10.640 DEBUG:boto:Data: 2023-09-22T21:56:10.640 DEBUG:boto:Headers: {} 2023-09-22T21:56:10.640 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.640 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.640 DEBUG:boto:Params: {} 2023-09-22T21:56:10.640 DEBUG:boto:Token: None 2023-09-22T21:56:10.640 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/ 2023-09-22T21:56:10.640 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U= 2023-09-22T21:56:10.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U=', 'Content-Length': '0'} 2023-09-22T21:56:10.646 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009374fa547bebfcb8-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.646 DEBUG:boto:path=/ffzdqu-61/obj37 2023-09-22T21:56:10.646 DEBUG:boto:auth_path=/ffzdqu-61/obj37 2023-09-22T21:56:10.646 DEBUG:boto:Method: PUT 2023-09-22T21:56:10.646 DEBUG:boto:Path: /ffzdqu-61/obj37 2023-09-22T21:56:10.647 DEBUG:boto:Data: 2023-09-22T21:56:10.647 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:10.647 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.647 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.647 DEBUG:boto:Params: {} 2023-09-22T21:56:10.647 DEBUG:boto:Token: None 2023-09-22T21:56:10.647 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/obj37 2023-09-22T21:56:10.647 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KjlaXevLC8OpO64Ql4E27TOxSZo= 2023-09-22T21:56:10.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KjlaXevLC8OpO64Ql4E27TOxSZo='} 2023-09-22T21:56:10.655 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000090eb9f8cf6c3a5f3-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.655 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:10.655 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:10.656 DEBUG:boto:Method: HEAD 2023-09-22T21:56:10.656 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:10.656 DEBUG:boto:Data: 2023-09-22T21:56:10.656 DEBUG:boto:Headers: {} 2023-09-22T21:56:10.656 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.656 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.656 DEBUG:boto:Params: {} 2023-09-22T21:56:10.656 DEBUG:boto:Token: None 2023-09-22T21:56:10.656 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/ 2023-09-22T21:56:10.656 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U= 2023-09-22T21:56:10.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U=', 'Content-Length': '0'} 2023-09-22T21:56:10.661 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092a85c0c8bc76316-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.662 DEBUG:boto:path=/ffzdqu-61/obj74 2023-09-22T21:56:10.662 DEBUG:boto:auth_path=/ffzdqu-61/obj74 2023-09-22T21:56:10.662 DEBUG:boto:Method: PUT 2023-09-22T21:56:10.662 DEBUG:boto:Path: /ffzdqu-61/obj74 2023-09-22T21:56:10.662 DEBUG:boto:Data: 2023-09-22T21:56:10.662 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:10.662 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.662 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.663 DEBUG:boto:Params: {} 2023-09-22T21:56:10.663 DEBUG:boto:Token: None 2023-09-22T21:56:10.663 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/obj74 2023-09-22T21:56:10.663 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SY27pKv+P1oKXZ8y/aPjVyX54Zo= 2023-09-22T21:56:10.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SY27pKv+P1oKXZ8y/aPjVyX54Zo='} 2023-09-22T21:56:10.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000839a35d774232953-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.707 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:10.707 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:10.707 DEBUG:boto:Method: HEAD 2023-09-22T21:56:10.707 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:10.707 DEBUG:boto:Data: 2023-09-22T21:56:10.707 DEBUG:boto:Headers: {} 2023-09-22T21:56:10.707 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.707 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.707 DEBUG:boto:Params: {} 2023-09-22T21:56:10.708 DEBUG:boto:Token: None 2023-09-22T21:56:10.708 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/ 2023-09-22T21:56:10.708 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U= 2023-09-22T21:56:10.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U=', 'Content-Length': '0'} 2023-09-22T21:56:10.714 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9a58d0841099e1b-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.714 DEBUG:boto:path=/ffzdqu-61/obj111 2023-09-22T21:56:10.714 DEBUG:boto:auth_path=/ffzdqu-61/obj111 2023-09-22T21:56:10.714 DEBUG:boto:Method: PUT 2023-09-22T21:56:10.714 DEBUG:boto:Path: /ffzdqu-61/obj111 2023-09-22T21:56:10.715 DEBUG:boto:Data: 2023-09-22T21:56:10.715 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:10.715 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.715 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.715 DEBUG:boto:Params: {} 2023-09-22T21:56:10.715 DEBUG:boto:Token: None 2023-09-22T21:56:10.715 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/obj111 2023-09-22T21:56:10.715 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fX0+Ef7cmc8z9/i4W8VHtXcL5ic= 2023-09-22T21:56:10.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fX0+Ef7cmc8z9/i4W8VHtXcL5ic='} 2023-09-22T21:56:10.728 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c7d992ae1b1a314-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.728 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:10.728 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:10.729 DEBUG:boto:Method: HEAD 2023-09-22T21:56:10.729 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:10.729 DEBUG:boto:Data: 2023-09-22T21:56:10.729 DEBUG:boto:Headers: {} 2023-09-22T21:56:10.729 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.729 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.729 DEBUG:boto:Params: {} 2023-09-22T21:56:10.729 DEBUG:boto:Token: None 2023-09-22T21:56:10.729 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/ 2023-09-22T21:56:10.730 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U= 2023-09-22T21:56:10.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U=', 'Content-Length': '0'} 2023-09-22T21:56:10.737 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007257f91491e53af4-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.738 DEBUG:boto:path=/ffzdqu-61/obj148 2023-09-22T21:56:10.738 DEBUG:boto:auth_path=/ffzdqu-61/obj148 2023-09-22T21:56:10.738 DEBUG:boto:Method: PUT 2023-09-22T21:56:10.738 DEBUG:boto:Path: /ffzdqu-61/obj148 2023-09-22T21:56:10.738 DEBUG:boto:Data: 2023-09-22T21:56:10.738 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:10.738 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.738 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.739 DEBUG:boto:Params: {} 2023-09-22T21:56:10.739 DEBUG:boto:Token: None 2023-09-22T21:56:10.739 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/obj148 2023-09-22T21:56:10.739 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wvEvZa9no2AMqkmwtCdaHtrrywU= 2023-09-22T21:56:10.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wvEvZa9no2AMqkmwtCdaHtrrywU='} 2023-09-22T21:56:10.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025e1d4b2031f3b9b-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.783 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:56:10.783 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:56:10.783 DEBUG:boto:Method: HEAD 2023-09-22T21:56:10.783 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:56:10.783 DEBUG:boto:Data: 2023-09-22T21:56:10.783 DEBUG:boto:Headers: {} 2023-09-22T21:56:10.783 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.783 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.783 DEBUG:boto:Params: {} 2023-09-22T21:56:10.783 DEBUG:boto:Token: None 2023-09-22T21:56:10.784 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/ 2023-09-22T21:56:10.784 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U= 2023-09-22T21:56:10.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:r6T7nIQzqWb/TGwS2zND4Xcq88U=', 'Content-Length': '0'} 2023-09-22T21:56:10.792 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081bcaf218e2b14cd-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.792 DEBUG:boto:path=/ffzdqu-61/obj185 2023-09-22T21:56:10.792 DEBUG:boto:auth_path=/ffzdqu-61/obj185 2023-09-22T21:56:10.792 DEBUG:boto:Method: PUT 2023-09-22T21:56:10.792 DEBUG:boto:Path: /ffzdqu-61/obj185 2023-09-22T21:56:10.792 DEBUG:boto:Data: 2023-09-22T21:56:10.793 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:56:10.793 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:56:10.793 DEBUG:boto:Port: 8000 2023-09-22T21:56:10.793 DEBUG:boto:Params: {} 2023-09-22T21:56:10.793 DEBUG:boto:Token: None 2023-09-22T21:56:10.793 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:56:10 GMT /ffzdqu-61/obj185 2023-09-22T21:56:10.793 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n353Oc5X0UJYiOnS9iHxADYLErQ= 2023-09-22T21:56:10.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:56:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n353Oc5X0UJYiOnS9iHxADYLErQ='} 2023-09-22T21:56:10.806 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9b471aa24563886-00650e0d7a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:56:10.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:56:10.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:56:10.807 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:56:10.850 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:56:10.851 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:56:10.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:56:10.895+0000 7f589da92a40 5 RGW-SYNC:data: waiting on data sync status lease 2023-09-22T21:56:10.896 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:56:10.895+0000 7f589da92a40 5 RGW-SYNC:data: acquired data sync status lease 2023-09-22T21:56:10.923 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-22T21:56:10.923 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-22T21:56:10.968 INFO:tasks.rgw.c1.client.1:Started 2023-09-22T21:56:10.968 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:10.977 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:56:10.977 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:56:10.977 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T21:56:10.978 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T21:56:10.978 INFO:teuthology.orchestra.run.smithi160.stderr:curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T21:56:11.980 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T21:56:11.990 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:56:11.991 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:56:11.993 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 73564 0 --:--:-- --:--:-- --:--:-- 104k 2023-09-22T21:56:11.994 INFO:teuthology.orchestra.run.smithi160.stdout:anonymous 2023-09-22T21:56:11.994 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-61 2023-09-22T21:56:11.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:56:11.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:56:11.995 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:56:12.086 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:56:12.087 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:56:12.130 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:56:12.127+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:56:13.951 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:56:13.951+0000 7ff64a379640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:56:17.131 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:56:17.131+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:56:22.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:56:22.130+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:56:27.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:56:27.130+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:56:32.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:56:32.130+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:56:37.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:56:37.130+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:56:42.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:56:42.130+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:56:47.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:56:47.130+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:56:52.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:56:52.130+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:56:57.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:56:57.130+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:57:02.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:57:02.134+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:57:07.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:57:07.134+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:57:12.137 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:57:12.138+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:57:17.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:57:17.134+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:57:22.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:57:22.138+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:57:27.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:57:27.138+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:57:32.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:57:32.141+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:57:37.143 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:57:37.137+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:57:42.144 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:57:42.141+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:57:47.145 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:57:47.141+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:57:52.146 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:57:52.145+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:57:57.147 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:57:57.149+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:58:02.148 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:58:02.149+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:58:07.150 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:58:07.149+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:58:12.151 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:58:12.149+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:58:17.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:58:17.153+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:58:22.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:58:22.153+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:58:27.154 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:58:27.153+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:58:32.155 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:58:32.153+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:58:37.157 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:58:37.157+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:58:42.157 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:58:42.156+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:58:47.158 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:58:47.156+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:58:52.159 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:58:52.160+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:58:57.160 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:58:57.160+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:59:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:59:02.160+0000 7f3984979a40 1 waiting to reach incremental sync.. 2023-09-22T21:59:07.168 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:59:07.168+0000 7f3984979a40 1 bucket sync caught up with source: 2023-09-22T21:59:07.168 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.99.6, , , , , , 00000000002.91.6, , , , , 00000000002.126.6, , , , , , , , , , , , , , , 00000000001.94.6, , , , 00000000001.93.6] 2023-09-22T21:59:07.168 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.99.6, , , , , , 00000000002.91.6, , , , , 00000000002.126.6, , , , , , , , , , , , , , , 00000000001.94.6, , , , 00000000001.93.6] 2023-09-22T21:59:07.169 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:59:07.168+0000 7f3984979a40 0 bucket checkpoint complete 2023-09-22T21:59:07.200 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-61 2023-09-22T21:59:07.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:59:07.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:59:07.201 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:59:07.251 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:59:07.251 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:59:07.296 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:59:07.296+0000 7f5d62400a40 1 bucket sync caught up with source: 2023-09-22T21:59:07.296 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.243.6, 00000000002.212.6, 00000000002.214.6, 00000000005.250.6, 00000000004.188.6, 00000000001.220.6, 00000000001.158.6, 00000000004.296.6, 00000000007.340.6, 00000000006.202.6] 2023-09-22T21:59:07.296 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.243.6, 00000000002.212.6, 00000000002.214.6, 00000000005.250.6, 00000000004.188.6, 00000000001.220.6, 00000000001.158.6, 00000000004.296.6, 00000000007.340.6, 00000000006.202.6] 2023-09-22T21:59:07.296 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:59:07.296+0000 7f5d62400a40 0 bucket checkpoint complete 2023-09-22T21:59:07.314 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-61 zones={a2, a1} 2023-09-22T21:59:07.314 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:07.314 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:07.314 DEBUG:boto:Method: HEAD 2023-09-22T21:59:07.315 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:59:07.315 DEBUG:boto:Data: 2023-09-22T21:59:07.315 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.315 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.315 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.315 DEBUG:boto:Params: {} 2023-09-22T21:59:07.315 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T21:59:07.316 DEBUG:boto:Token: None 2023-09-22T21:59:07.316 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/ 2023-09-22T21:59:07.316 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:INwiNxp39E7otw0FbWNA5wVHEPo= 2023-09-22T21:59:07.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:INwiNxp39E7otw0FbWNA5wVHEPo=', 'Content-Length': '0'} 2023-09-22T21:59:07.322 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a45e8c8d65437d8b-00650e0e2b-1edb-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.322 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:07.322 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:07.322 DEBUG:boto:Method: HEAD 2023-09-22T21:59:07.323 DEBUG:boto:Path: /ffzdqu-61/ 2023-09-22T21:59:07.323 DEBUG:boto:Data: 2023-09-22T21:59:07.323 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.323 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.323 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.323 DEBUG:boto:Params: {} 2023-09-22T21:59:07.323 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T21:59:07.323 DEBUG:boto:Token: None 2023-09-22T21:59:07.324 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/ 2023-09-22T21:59:07.324 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:INwiNxp39E7otw0FbWNA5wVHEPo= 2023-09-22T21:59:07.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:INwiNxp39E7otw0FbWNA5wVHEPo=', 'Content-Length': '0'} 2023-09-22T21:59:07.328 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be6a0d5c4f8102be-00650e0e2b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.329 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:59:07.329 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:07.329 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:07.329 DEBUG:boto:path=/ffzdqu-61/?versions&max-keys=999 2023-09-22T21:59:07.329 DEBUG:boto:auth_path=/ffzdqu-61/?versions&max-keys=999 2023-09-22T21:59:07.329 DEBUG:boto:Method: GET 2023-09-22T21:59:07.329 DEBUG:boto:Path: /ffzdqu-61/?versions&max-keys=999 2023-09-22T21:59:07.329 DEBUG:boto:Data: 2023-09-22T21:59:07.330 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.330 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.330 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.330 DEBUG:boto:Params: {} 2023-09-22T21:59:07.330 DEBUG:boto:Token: None 2023-09-22T21:59:07.330 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/?versions 2023-09-22T21:59:07.330 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:a7EAz26AKQDc6p3mLLC18QPhkKI= 2023-09-22T21:59:07.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:a7EAz26AKQDc6p3mLLC18QPhkKI=', 'Content-Length': '0'} 2023-09-22T21:59:07.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2c3cbb13dd7284e-00650e0e2b-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.333 DEBUG:boto:b'ffzdqu-61999falseobj11nulltrue2023-09-22T21:56:08.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-22T21:56:10.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:56:09.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T21:56:10.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T21:56:10.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T21:56:10.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-22T21:56:10.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T21:56:10.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:56:08.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-22T21:56:10.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:56:09.260Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:56:08.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:56:09.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T21:56:09.931Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T21:56:10.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:56:08.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:56:08.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-22T21:56:10.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:56:09.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:56:08.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:56:09.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:56:08.964Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:56:08.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:56:09.284Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T21:56:09.943Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T21:56:10.292Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:56:09.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:56:09.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-22T21:56:10.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:56:09.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:56:09.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T21:56:09.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:56:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T21:56:10.304Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:56:09.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:07.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T21:59:07.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-09-22T21:59:07.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-22T21:59:07.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-22T21:59:07.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-09-22T21:59:07.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-22T21:59:07.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-09-22T21:59:07.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-09-22T21:59:07.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-22T21:59:07.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-09-22T21:59:07.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-22T21:59:07.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T21:59:07.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-22T21:59:07.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-22T21:59:07.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-09-22T21:59:07.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T21:59:07.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-22T21:59:07.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-09-22T21:59:07.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-22T21:59:07.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T21:59:07.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-22T21:59:07.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-22T21:59:07.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T21:59:07.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-22T21:59:07.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-22T21:59:07.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-09-22T21:59:07.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-22T21:59:07.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-22T21:59:07.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-09-22T21:59:07.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-22T21:59:07.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-22T21:59:07.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-22T21:59:07.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-22T21:59:07.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-09-22T21:59:07.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-22T21:59:07.339 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:59:07.339 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:07.340 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:07.340 DEBUG:boto:path=/ffzdqu-61/?versions&max-keys=999 2023-09-22T21:59:07.340 DEBUG:boto:auth_path=/ffzdqu-61/?versions&max-keys=999 2023-09-22T21:59:07.340 DEBUG:boto:Method: GET 2023-09-22T21:59:07.340 DEBUG:boto:Path: /ffzdqu-61/?versions&max-keys=999 2023-09-22T21:59:07.340 DEBUG:boto:Data: 2023-09-22T21:59:07.340 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.341 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.341 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.341 DEBUG:boto:Params: {} 2023-09-22T21:59:07.341 DEBUG:boto:Token: None 2023-09-22T21:59:07.341 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/?versions 2023-09-22T21:59:07.341 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:a7EAz26AKQDc6p3mLLC18QPhkKI= 2023-09-22T21:59:07.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:a7EAz26AKQDc6p3mLLC18QPhkKI=', 'Content-Length': '0'} 2023-09-22T21:59:07.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bca9d214ef9a47c7-00650e0e2b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.346 DEBUG:boto:b'ffzdqu-61999falseobj11nulltrue2023-09-22T21:56:08.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-22T21:56:10.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:56:09.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T21:56:10.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T21:56:10.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T21:56:10.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-22T21:56:10.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T21:56:10.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:56:08.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-22T21:56:10.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:56:09.260Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:56:08.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:56:09.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T21:56:09.931Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T21:56:10.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:56:08.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:56:08.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-22T21:56:10.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:56:09.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:56:08.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:56:09.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:56:08.964Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:56:08.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:56:09.284Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T21:56:09.943Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T21:56:10.292Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:56:09.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:56:09.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-22T21:56:10.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:56:09.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:56:09.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T21:56:09.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:56:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T21:56:10.304Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:56:09.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:07.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T21:59:07.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-09-22T21:59:07.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-22T21:59:07.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-22T21:59:07.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-09-22T21:59:07.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-22T21:59:07.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-09-22T21:59:07.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-09-22T21:59:07.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-22T21:59:07.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-09-22T21:59:07.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-22T21:59:07.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T21:59:07.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-22T21:59:07.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-22T21:59:07.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-09-22T21:59:07.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T21:59:07.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-22T21:59:07.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-09-22T21:59:07.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-22T21:59:07.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T21:59:07.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-22T21:59:07.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-22T21:59:07.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T21:59:07.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-22T21:59:07.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-22T21:59:07.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-09-22T21:59:07.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-22T21:59:07.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-22T21:59:07.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-09-22T21:59:07.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-22T21:59:07.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-22T21:59:07.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-22T21:59:07.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-22T21:59:07.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-09-22T21:59:07.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-22T21:59:07.352 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:07.352 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:07.352 DEBUG:boto:path=/ffzdqu-61/?versions&max-keys=999 2023-09-22T21:59:07.352 DEBUG:boto:auth_path=/ffzdqu-61/?versions&max-keys=999 2023-09-22T21:59:07.353 DEBUG:boto:Method: GET 2023-09-22T21:59:07.353 DEBUG:boto:Path: /ffzdqu-61/?versions&max-keys=999 2023-09-22T21:59:07.353 DEBUG:boto:Data: 2023-09-22T21:59:07.353 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.353 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.353 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.353 DEBUG:boto:Params: {} 2023-09-22T21:59:07.354 DEBUG:boto:Token: None 2023-09-22T21:59:07.354 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/?versions 2023-09-22T21:59:07.354 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:a7EAz26AKQDc6p3mLLC18QPhkKI= 2023-09-22T21:59:07.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:a7EAz26AKQDc6p3mLLC18QPhkKI=', 'Content-Length': '0'} 2023-09-22T21:59:07.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068a1bda282ccac0a-00650e0e2b-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.357 DEBUG:boto:b'ffzdqu-61999falseobj11nulltrue2023-09-22T21:56:08.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-22T21:56:10.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:56:09.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T21:56:10.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T21:56:10.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T21:56:10.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-22T21:56:10.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T21:56:10.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:56:08.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-22T21:56:10.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:56:09.260Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:56:08.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:56:09.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T21:56:09.931Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T21:56:10.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:56:08.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:56:08.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-22T21:56:10.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:56:09.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:56:08.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:56:09.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:56:08.964Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:56:08.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:56:09.284Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T21:56:09.943Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T21:56:10.292Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:56:09.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:56:09.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-22T21:56:10.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:56:09.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:56:09.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T21:56:09.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:56:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T21:56:10.304Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:56:09.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:07.359 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:07.359 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:07.359 DEBUG:boto:path=/ffzdqu-61/?versions&max-keys=999 2023-09-22T21:59:07.359 DEBUG:boto:auth_path=/ffzdqu-61/?versions&max-keys=999 2023-09-22T21:59:07.359 DEBUG:boto:Method: GET 2023-09-22T21:59:07.359 DEBUG:boto:Path: /ffzdqu-61/?versions&max-keys=999 2023-09-22T21:59:07.359 DEBUG:boto:Data: 2023-09-22T21:59:07.359 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.360 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.360 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.360 DEBUG:boto:Params: {} 2023-09-22T21:59:07.360 DEBUG:boto:Token: None 2023-09-22T21:59:07.360 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/?versions 2023-09-22T21:59:07.360 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:a7EAz26AKQDc6p3mLLC18QPhkKI= 2023-09-22T21:59:07.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:a7EAz26AKQDc6p3mLLC18QPhkKI=', 'Content-Length': '0'} 2023-09-22T21:59:07.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002762e7e2ff42e98-00650e0e2b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.365 DEBUG:boto:b'ffzdqu-61999falseobj11nulltrue2023-09-22T21:56:08.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-22T21:56:10.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T21:56:09.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T21:56:10.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T21:56:10.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T21:56:10.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-22T21:56:10.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T21:56:10.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T21:56:08.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-22T21:56:10.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T21:56:09.260Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T21:56:08.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T21:56:09.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T21:56:09.931Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T21:56:10.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T21:56:08.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T21:56:08.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-22T21:56:10.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T21:56:09.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T21:56:08.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T21:56:09.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T21:56:08.964Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T21:56:08.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T21:56:09.284Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T21:56:09.943Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T21:56:10.292Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T21:56:09.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T21:56:09.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-22T21:56:10.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T21:56:09.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T21:56:09.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T21:56:09.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T21:56:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T21:56:10.304Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T21:56:09.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:07.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:59:07.368 DEBUG:boto:path=/ffzdqu-61/obj11 2023-09-22T21:59:07.368 DEBUG:boto:auth_path=/ffzdqu-61/obj11 2023-09-22T21:59:07.368 DEBUG:boto:path=/ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.368 DEBUG:boto:auth_path=/ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.368 DEBUG:boto:Method: GET 2023-09-22T21:59:07.368 DEBUG:boto:Path: /ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.369 DEBUG:boto:Data: 2023-09-22T21:59:07.369 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.369 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.369 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.369 DEBUG:boto:Params: {} 2023-09-22T21:59:07.369 DEBUG:boto:Token: None 2023-09-22T21:59:07.369 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.370 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f3MaA3VT+leV/gsfmkm8v7iFznI= 2023-09-22T21:59:07.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f3MaA3VT+leV/gsfmkm8v7iFznI=', 'Content-Length': '0'} 2023-09-22T21:59:07.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f05c8d2ad71d087-00650e0e2b-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.372 DEBUG:boto:path=/ffzdqu-61/obj11 2023-09-22T21:59:07.372 DEBUG:boto:auth_path=/ffzdqu-61/obj11 2023-09-22T21:59:07.372 DEBUG:boto:path=/ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.372 DEBUG:boto:auth_path=/ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.372 DEBUG:boto:Method: GET 2023-09-22T21:59:07.373 DEBUG:boto:Path: /ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.373 DEBUG:boto:Data: 2023-09-22T21:59:07.373 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.373 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.373 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.373 DEBUG:boto:Params: {} 2023-09-22T21:59:07.373 DEBUG:boto:Token: None 2023-09-22T21:59:07.374 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.374 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f3MaA3VT+leV/gsfmkm8v7iFznI= 2023-09-22T21:59:07.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f3MaA3VT+leV/gsfmkm8v7iFznI=', 'Content-Length': '0'} 2023-09-22T21:59:07.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0c36393f471f539-00650e0e2b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.418 DEBUG:boto:path=/ffzdqu-61/obj11 2023-09-22T21:59:07.419 DEBUG:boto:auth_path=/ffzdqu-61/obj11 2023-09-22T21:59:07.419 DEBUG:boto:path=/ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.419 DEBUG:boto:auth_path=/ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.419 DEBUG:boto:Method: HEAD 2023-09-22T21:59:07.419 DEBUG:boto:Path: /ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.419 DEBUG:boto:Data: 2023-09-22T21:59:07.419 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.419 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.419 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.419 DEBUG:boto:Params: {} 2023-09-22T21:59:07.420 DEBUG:boto:Token: None 2023-09-22T21:59:07.420 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.420 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FWWOaSkvi5KlMIL8FXiMstHrMNI= 2023-09-22T21:59:07.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FWWOaSkvi5KlMIL8FXiMstHrMNI=', 'Content-Length': '0'} 2023-09-22T21:59:07.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2d764adeacbef86-00650e0e2b-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.423 DEBUG:boto:path=/ffzdqu-61/obj11 2023-09-22T21:59:07.423 DEBUG:boto:auth_path=/ffzdqu-61/obj11 2023-09-22T21:59:07.423 DEBUG:boto:path=/ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.423 DEBUG:boto:auth_path=/ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.423 DEBUG:boto:Method: HEAD 2023-09-22T21:59:07.423 DEBUG:boto:Path: /ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.424 DEBUG:boto:Data: 2023-09-22T21:59:07.424 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.424 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.424 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.424 DEBUG:boto:Params: {} 2023-09-22T21:59:07.424 DEBUG:boto:Token: None 2023-09-22T21:59:07.424 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.424 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FWWOaSkvi5KlMIL8FXiMstHrMNI= 2023-09-22T21:59:07.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FWWOaSkvi5KlMIL8FXiMstHrMNI=', 'Content-Length': '0'} 2023-09-22T21:59:07.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005110499dc4627b6b-00650e0e2b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:59:07.428 DEBUG:boto:path=/ffzdqu-61/obj11 2023-09-22T21:59:07.428 DEBUG:boto:auth_path=/ffzdqu-61/obj11 2023-09-22T21:59:07.428 DEBUG:boto:path=/ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.428 DEBUG:boto:auth_path=/ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.428 DEBUG:boto:Method: GET 2023-09-22T21:59:07.428 DEBUG:boto:Path: /ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.428 DEBUG:boto:Data: 2023-09-22T21:59:07.428 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.428 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.428 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.429 DEBUG:boto:Params: {} 2023-09-22T21:59:07.429 DEBUG:boto:Token: None 2023-09-22T21:59:07.429 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.429 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f3MaA3VT+leV/gsfmkm8v7iFznI= 2023-09-22T21:59:07.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f3MaA3VT+leV/gsfmkm8v7iFznI=', 'Content-Length': '0'} 2023-09-22T21:59:07.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aa3d2642805b288-00650e0e2b-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.474 DEBUG:boto:path=/ffzdqu-61/obj11 2023-09-22T21:59:07.474 DEBUG:boto:auth_path=/ffzdqu-61/obj11 2023-09-22T21:59:07.475 DEBUG:boto:path=/ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.475 DEBUG:boto:auth_path=/ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.475 DEBUG:boto:Method: GET 2023-09-22T21:59:07.475 DEBUG:boto:Path: /ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.475 DEBUG:boto:Data: 2023-09-22T21:59:07.475 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.475 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.475 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.475 DEBUG:boto:Params: {} 2023-09-22T21:59:07.476 DEBUG:boto:Token: None 2023-09-22T21:59:07.476 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj11?versionId=null 2023-09-22T21:59:07.476 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f3MaA3VT+leV/gsfmkm8v7iFznI= 2023-09-22T21:59:07.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f3MaA3VT+leV/gsfmkm8v7iFznI=', 'Content-Length': '0'} 2023-09-22T21:59:07.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091e3565180e5093e-00650e0e2b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.479 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:07.479 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:07.479 DEBUG:boto:path=/ffzdqu-61/?prefix=obj11 2023-09-22T21:59:07.480 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj11 2023-09-22T21:59:07.480 DEBUG:boto:Method: GET 2023-09-22T21:59:07.480 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj11 2023-09-22T21:59:07.480 DEBUG:boto:Data: 2023-09-22T21:59:07.480 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.480 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.480 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.480 DEBUG:boto:Params: {} 2023-09-22T21:59:07.480 DEBUG:boto:Token: None 2023-09-22T21:59:07.480 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/ 2023-09-22T21:59:07.481 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro= 2023-09-22T21:59:07.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro=', 'Content-Length': '0'} 2023-09-22T21:59:07.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b8051919cd7b540-00650e0e2b-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.486 DEBUG:boto:b'ffzdqu-61obj111000falseobj112023-09-22T21:56:08.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-22T21:56:10.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:56:09.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T21:56:10.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:07.486 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:07.487 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:07.487 DEBUG:boto:path=/ffzdqu-61/?prefix=obj11 2023-09-22T21:59:07.487 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj11 2023-09-22T21:59:07.487 DEBUG:boto:Method: GET 2023-09-22T21:59:07.487 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj11 2023-09-22T21:59:07.487 DEBUG:boto:Data: 2023-09-22T21:59:07.487 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.487 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.487 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.487 DEBUG:boto:Params: {} 2023-09-22T21:59:07.488 DEBUG:boto:Token: None 2023-09-22T21:59:07.488 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/ 2023-09-22T21:59:07.488 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro= 2023-09-22T21:59:07.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro=', 'Content-Length': '0'} 2023-09-22T21:59:07.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053bd3b420a074391-00650e0e2b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.499 DEBUG:boto:b'ffzdqu-61obj111000falseobj112023-09-22T21:56:08.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-22T21:56:10.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:56:09.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T21:56:10.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:07.500 DEBUG:boto:path=/ffzdqu-61/obj11 2023-09-22T21:59:07.500 DEBUG:boto:auth_path=/ffzdqu-61/obj11 2023-09-22T21:59:07.500 DEBUG:boto:Method: HEAD 2023-09-22T21:59:07.500 DEBUG:boto:Path: /ffzdqu-61/obj11 2023-09-22T21:59:07.500 DEBUG:boto:Data: 2023-09-22T21:59:07.501 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.501 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.501 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.501 DEBUG:boto:Params: {} 2023-09-22T21:59:07.501 DEBUG:boto:Token: None 2023-09-22T21:59:07.501 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj11 2023-09-22T21:59:07.501 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bQhq1DH+fxPYbPgNb0VawdbTWFo= 2023-09-22T21:59:07.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bQhq1DH+fxPYbPgNb0VawdbTWFo=', 'Content-Length': '0'} 2023-09-22T21:59:07.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e2bd46e46606473-00650e0e2b-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.504 DEBUG:boto:path=/ffzdqu-61/obj11 2023-09-22T21:59:07.505 DEBUG:boto:auth_path=/ffzdqu-61/obj11 2023-09-22T21:59:07.505 DEBUG:boto:Method: HEAD 2023-09-22T21:59:07.505 DEBUG:boto:Path: /ffzdqu-61/obj11 2023-09-22T21:59:07.505 DEBUG:boto:Data: 2023-09-22T21:59:07.505 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.505 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.505 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.505 DEBUG:boto:Params: {} 2023-09-22T21:59:07.506 DEBUG:boto:Token: None 2023-09-22T21:59:07.506 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj11 2023-09-22T21:59:07.506 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bQhq1DH+fxPYbPgNb0VawdbTWFo= 2023-09-22T21:59:07.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bQhq1DH+fxPYbPgNb0VawdbTWFo=', 'Content-Length': '0'} 2023-09-22T21:59:07.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfe60087eac53576-00650e0e2b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T21:59:07.509 DEBUG:boto:path=/ffzdqu-61/obj11 2023-09-22T21:59:07.509 DEBUG:boto:auth_path=/ffzdqu-61/obj11 2023-09-22T21:59:07.509 DEBUG:boto:Method: GET 2023-09-22T21:59:07.509 DEBUG:boto:Path: /ffzdqu-61/obj11 2023-09-22T21:59:07.509 DEBUG:boto:Data: 2023-09-22T21:59:07.510 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.510 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.510 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.510 DEBUG:boto:Params: {} 2023-09-22T21:59:07.510 DEBUG:boto:Token: None 2023-09-22T21:59:07.510 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj11 2023-09-22T21:59:07.510 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+fz7BBFV6EAuIw92oKfYw7GlTpw= 2023-09-22T21:59:07.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+fz7BBFV6EAuIw92oKfYw7GlTpw=', 'Content-Length': '0'} 2023-09-22T21:59:07.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004458ad87a26fcc5e-00650e0e2b-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.554 DEBUG:boto:path=/ffzdqu-61/obj11 2023-09-22T21:59:07.554 DEBUG:boto:auth_path=/ffzdqu-61/obj11 2023-09-22T21:59:07.554 DEBUG:boto:Method: GET 2023-09-22T21:59:07.555 DEBUG:boto:Path: /ffzdqu-61/obj11 2023-09-22T21:59:07.555 DEBUG:boto:Data: 2023-09-22T21:59:07.555 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.555 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.555 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.555 DEBUG:boto:Params: {} 2023-09-22T21:59:07.555 DEBUG:boto:Token: None 2023-09-22T21:59:07.555 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj11 2023-09-22T21:59:07.555 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+fz7BBFV6EAuIw92oKfYw7GlTpw= 2023-09-22T21:59:07.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+fz7BBFV6EAuIw92oKfYw7GlTpw=', 'Content-Length': '0'} 2023-09-22T21:59:07.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072ba80339985274f-00650e0e2b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.558 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:07.559 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:07.559 DEBUG:boto:path=/ffzdqu-61/?prefix=obj11 2023-09-22T21:59:07.559 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj11 2023-09-22T21:59:07.559 DEBUG:boto:Method: GET 2023-09-22T21:59:07.559 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj11 2023-09-22T21:59:07.559 DEBUG:boto:Data: 2023-09-22T21:59:07.559 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.559 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.559 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.559 DEBUG:boto:Params: {} 2023-09-22T21:59:07.560 DEBUG:boto:Token: None 2023-09-22T21:59:07.560 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/ 2023-09-22T21:59:07.560 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro= 2023-09-22T21:59:07.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro=', 'Content-Length': '0'} 2023-09-22T21:59:07.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d753578c63e7a178-00650e0e2b-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.565 DEBUG:boto:b'ffzdqu-61obj111000falseobj112023-09-22T21:56:08.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-22T21:56:10.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:56:09.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T21:56:10.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:07.565 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:07.566 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:07.566 DEBUG:boto:path=/ffzdqu-61/?prefix=obj11 2023-09-22T21:59:07.566 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj11 2023-09-22T21:59:07.566 DEBUG:boto:Method: GET 2023-09-22T21:59:07.566 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj11 2023-09-22T21:59:07.566 DEBUG:boto:Data: 2023-09-22T21:59:07.566 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.566 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.566 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.567 DEBUG:boto:Params: {} 2023-09-22T21:59:07.567 DEBUG:boto:Token: None 2023-09-22T21:59:07.567 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/ 2023-09-22T21:59:07.567 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro= 2023-09-22T21:59:07.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro=', 'Content-Length': '0'} 2023-09-22T21:59:07.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce37d07d2bd59f25-00650e0e2b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.577 DEBUG:boto:b'ffzdqu-61obj111000falseobj112023-09-22T21:56:08.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-22T21:56:10.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T21:56:09.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T21:56:10.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:07.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-09-22T21:59:07.578 DEBUG:boto:path=/ffzdqu-61/obj111 2023-09-22T21:59:07.578 DEBUG:boto:auth_path=/ffzdqu-61/obj111 2023-09-22T21:59:07.578 DEBUG:boto:path=/ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.578 DEBUG:boto:auth_path=/ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.578 DEBUG:boto:Method: GET 2023-09-22T21:59:07.578 DEBUG:boto:Path: /ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.578 DEBUG:boto:Data: 2023-09-22T21:59:07.578 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.578 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.578 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.579 DEBUG:boto:Params: {} 2023-09-22T21:59:07.579 DEBUG:boto:Token: None 2023-09-22T21:59:07.579 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.579 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xNe3VvXorcKAtyHDvKIGy5m4kpM= 2023-09-22T21:59:07.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xNe3VvXorcKAtyHDvKIGy5m4kpM=', 'Content-Length': '0'} 2023-09-22T21:59:07.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000882f42400c923dbb-00650e0e2b-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.582 DEBUG:boto:path=/ffzdqu-61/obj111 2023-09-22T21:59:07.582 DEBUG:boto:auth_path=/ffzdqu-61/obj111 2023-09-22T21:59:07.583 DEBUG:boto:path=/ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.583 DEBUG:boto:auth_path=/ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.583 DEBUG:boto:Method: GET 2023-09-22T21:59:07.583 DEBUG:boto:Path: /ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.583 DEBUG:boto:Data: 2023-09-22T21:59:07.583 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.583 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.583 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.583 DEBUG:boto:Params: {} 2023-09-22T21:59:07.583 DEBUG:boto:Token: None 2023-09-22T21:59:07.584 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.584 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xNe3VvXorcKAtyHDvKIGy5m4kpM= 2023-09-22T21:59:07.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xNe3VvXorcKAtyHDvKIGy5m4kpM=', 'Content-Length': '0'} 2023-09-22T21:59:07.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000622504b8501f3219-00650e0e2b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.587 DEBUG:boto:path=/ffzdqu-61/obj111 2023-09-22T21:59:07.587 DEBUG:boto:auth_path=/ffzdqu-61/obj111 2023-09-22T21:59:07.587 DEBUG:boto:path=/ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.587 DEBUG:boto:auth_path=/ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.587 DEBUG:boto:Method: HEAD 2023-09-22T21:59:07.587 DEBUG:boto:Path: /ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.587 DEBUG:boto:Data: 2023-09-22T21:59:07.588 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.588 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.588 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.588 DEBUG:boto:Params: {} 2023-09-22T21:59:07.588 DEBUG:boto:Token: None 2023-09-22T21:59:07.588 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.588 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TNAIKlAqKSuY6WM81PzUMWpodP8= 2023-09-22T21:59:07.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TNAIKlAqKSuY6WM81PzUMWpodP8=', 'Content-Length': '0'} 2023-09-22T21:59:07.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019fdedf7f42d0dff-00650e0e2b-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.591 DEBUG:boto:path=/ffzdqu-61/obj111 2023-09-22T21:59:07.591 DEBUG:boto:auth_path=/ffzdqu-61/obj111 2023-09-22T21:59:07.591 DEBUG:boto:path=/ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.591 DEBUG:boto:auth_path=/ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.592 DEBUG:boto:Method: HEAD 2023-09-22T21:59:07.592 DEBUG:boto:Path: /ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.592 DEBUG:boto:Data: 2023-09-22T21:59:07.592 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.592 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.592 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.592 DEBUG:boto:Params: {} 2023-09-22T21:59:07.592 DEBUG:boto:Token: None 2023-09-22T21:59:07.592 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.592 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TNAIKlAqKSuY6WM81PzUMWpodP8= 2023-09-22T21:59:07.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TNAIKlAqKSuY6WM81PzUMWpodP8=', 'Content-Length': '0'} 2023-09-22T21:59:07.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011516880aa024a0e-00650e0e2b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-09-22T21:59:07.596 DEBUG:boto:path=/ffzdqu-61/obj111 2023-09-22T21:59:07.596 DEBUG:boto:auth_path=/ffzdqu-61/obj111 2023-09-22T21:59:07.596 DEBUG:boto:path=/ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.596 DEBUG:boto:auth_path=/ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.596 DEBUG:boto:Method: GET 2023-09-22T21:59:07.596 DEBUG:boto:Path: /ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.596 DEBUG:boto:Data: 2023-09-22T21:59:07.596 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.596 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.596 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.597 DEBUG:boto:Params: {} 2023-09-22T21:59:07.597 DEBUG:boto:Token: None 2023-09-22T21:59:07.597 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.597 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xNe3VvXorcKAtyHDvKIGy5m4kpM= 2023-09-22T21:59:07.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xNe3VvXorcKAtyHDvKIGy5m4kpM=', 'Content-Length': '0'} 2023-09-22T21:59:07.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000653a911b3b3072e1-00650e0e2b-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.642 DEBUG:boto:path=/ffzdqu-61/obj111 2023-09-22T21:59:07.642 DEBUG:boto:auth_path=/ffzdqu-61/obj111 2023-09-22T21:59:07.643 DEBUG:boto:path=/ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.643 DEBUG:boto:auth_path=/ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.643 DEBUG:boto:Method: GET 2023-09-22T21:59:07.643 DEBUG:boto:Path: /ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.643 DEBUG:boto:Data: 2023-09-22T21:59:07.643 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.643 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.643 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.643 DEBUG:boto:Params: {} 2023-09-22T21:59:07.644 DEBUG:boto:Token: None 2023-09-22T21:59:07.644 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj111?versionId=null 2023-09-22T21:59:07.644 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xNe3VvXorcKAtyHDvKIGy5m4kpM= 2023-09-22T21:59:07.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xNe3VvXorcKAtyHDvKIGy5m4kpM=', 'Content-Length': '0'} 2023-09-22T21:59:07.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b15a14b4dca55e6d-00650e0e2b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.647 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:07.648 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:07.648 DEBUG:boto:path=/ffzdqu-61/?prefix=obj111 2023-09-22T21:59:07.648 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj111 2023-09-22T21:59:07.648 DEBUG:boto:Method: GET 2023-09-22T21:59:07.648 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj111 2023-09-22T21:59:07.648 DEBUG:boto:Data: 2023-09-22T21:59:07.648 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.648 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.649 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.649 DEBUG:boto:Params: {} 2023-09-22T21:59:07.649 DEBUG:boto:Token: None 2023-09-22T21:59:07.649 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/ 2023-09-22T21:59:07.649 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro= 2023-09-22T21:59:07.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro=', 'Content-Length': '0'} 2023-09-22T21:59:07.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afe4da1573c2cc53-00650e0e2b-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.694 DEBUG:boto:b'ffzdqu-61obj1111000falseobj1112023-09-22T21:56:10.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:07.695 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:07.695 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:07.695 DEBUG:boto:path=/ffzdqu-61/?prefix=obj111 2023-09-22T21:59:07.695 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj111 2023-09-22T21:59:07.695 DEBUG:boto:Method: GET 2023-09-22T21:59:07.695 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj111 2023-09-22T21:59:07.696 DEBUG:boto:Data: 2023-09-22T21:59:07.696 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.696 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.696 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.696 DEBUG:boto:Params: {} 2023-09-22T21:59:07.696 DEBUG:boto:Token: None 2023-09-22T21:59:07.696 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/ 2023-09-22T21:59:07.697 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro= 2023-09-22T21:59:07.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro=', 'Content-Length': '0'} 2023-09-22T21:59:07.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d734890a975369c4-00650e0e2b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.707 DEBUG:boto:b'ffzdqu-61obj1111000falseobj1112023-09-22T21:56:10.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:07.707 DEBUG:boto:path=/ffzdqu-61/obj111 2023-09-22T21:59:07.707 DEBUG:boto:auth_path=/ffzdqu-61/obj111 2023-09-22T21:59:07.707 DEBUG:boto:Method: HEAD 2023-09-22T21:59:07.707 DEBUG:boto:Path: /ffzdqu-61/obj111 2023-09-22T21:59:07.708 DEBUG:boto:Data: 2023-09-22T21:59:07.708 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.708 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.708 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.708 DEBUG:boto:Params: {} 2023-09-22T21:59:07.708 DEBUG:boto:Token: None 2023-09-22T21:59:07.708 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj111 2023-09-22T21:59:07.709 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fSRr+0b9VCQxdUD6jiQn5ZNAavY= 2023-09-22T21:59:07.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fSRr+0b9VCQxdUD6jiQn5ZNAavY=', 'Content-Length': '0'} 2023-09-22T21:59:07.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d8c0ac113fd34ed-00650e0e2b-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.712 DEBUG:boto:path=/ffzdqu-61/obj111 2023-09-22T21:59:07.712 DEBUG:boto:auth_path=/ffzdqu-61/obj111 2023-09-22T21:59:07.712 DEBUG:boto:Method: HEAD 2023-09-22T21:59:07.712 DEBUG:boto:Path: /ffzdqu-61/obj111 2023-09-22T21:59:07.712 DEBUG:boto:Data: 2023-09-22T21:59:07.712 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.712 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.712 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.712 DEBUG:boto:Params: {} 2023-09-22T21:59:07.712 DEBUG:boto:Token: None 2023-09-22T21:59:07.713 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj111 2023-09-22T21:59:07.713 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fSRr+0b9VCQxdUD6jiQn5ZNAavY= 2023-09-22T21:59:07.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fSRr+0b9VCQxdUD6jiQn5ZNAavY=', 'Content-Length': '0'} 2023-09-22T21:59:07.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e78d82ee590548e-00650e0e2b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-09-22T21:59:07.716 DEBUG:boto:path=/ffzdqu-61/obj111 2023-09-22T21:59:07.716 DEBUG:boto:auth_path=/ffzdqu-61/obj111 2023-09-22T21:59:07.716 DEBUG:boto:Method: GET 2023-09-22T21:59:07.716 DEBUG:boto:Path: /ffzdqu-61/obj111 2023-09-22T21:59:07.716 DEBUG:boto:Data: 2023-09-22T21:59:07.716 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.716 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.716 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.716 DEBUG:boto:Params: {} 2023-09-22T21:59:07.717 DEBUG:boto:Token: None 2023-09-22T21:59:07.717 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj111 2023-09-22T21:59:07.717 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:01sQ+vMBmdDjvVoIKgdNGb5ZUq4= 2023-09-22T21:59:07.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:01sQ+vMBmdDjvVoIKgdNGb5ZUq4=', 'Content-Length': '0'} 2023-09-22T21:59:07.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095b866a823d809ad-00650e0e2b-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.762 DEBUG:boto:path=/ffzdqu-61/obj111 2023-09-22T21:59:07.762 DEBUG:boto:auth_path=/ffzdqu-61/obj111 2023-09-22T21:59:07.762 DEBUG:boto:Method: GET 2023-09-22T21:59:07.763 DEBUG:boto:Path: /ffzdqu-61/obj111 2023-09-22T21:59:07.763 DEBUG:boto:Data: 2023-09-22T21:59:07.763 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.763 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.763 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.763 DEBUG:boto:Params: {} 2023-09-22T21:59:07.763 DEBUG:boto:Token: None 2023-09-22T21:59:07.763 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj111 2023-09-22T21:59:07.763 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:01sQ+vMBmdDjvVoIKgdNGb5ZUq4= 2023-09-22T21:59:07.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:01sQ+vMBmdDjvVoIKgdNGb5ZUq4=', 'Content-Length': '0'} 2023-09-22T21:59:07.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d187e5749bb65896-00650e0e2b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.766 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:07.767 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:07.767 DEBUG:boto:path=/ffzdqu-61/?prefix=obj111 2023-09-22T21:59:07.767 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj111 2023-09-22T21:59:07.767 DEBUG:boto:Method: GET 2023-09-22T21:59:07.767 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj111 2023-09-22T21:59:07.767 DEBUG:boto:Data: 2023-09-22T21:59:07.767 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.767 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.767 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.767 DEBUG:boto:Params: {} 2023-09-22T21:59:07.768 DEBUG:boto:Token: None 2023-09-22T21:59:07.768 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/ 2023-09-22T21:59:07.768 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro= 2023-09-22T21:59:07.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro=', 'Content-Length': '0'} 2023-09-22T21:59:07.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ca58ba3fe50af08-00650e0e2b-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.773 DEBUG:boto:b'ffzdqu-61obj1111000falseobj1112023-09-22T21:56:10.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:07.773 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:07.774 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:07.774 DEBUG:boto:path=/ffzdqu-61/?prefix=obj111 2023-09-22T21:59:07.774 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj111 2023-09-22T21:59:07.774 DEBUG:boto:Method: GET 2023-09-22T21:59:07.774 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj111 2023-09-22T21:59:07.774 DEBUG:boto:Data: 2023-09-22T21:59:07.774 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.774 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.774 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.775 DEBUG:boto:Params: {} 2023-09-22T21:59:07.775 DEBUG:boto:Token: None 2023-09-22T21:59:07.775 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/ 2023-09-22T21:59:07.775 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro= 2023-09-22T21:59:07.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro=', 'Content-Length': '0'} 2023-09-22T21:59:07.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c933776bff002816-00650e0e2b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.785 DEBUG:boto:b'ffzdqu-61obj1111000falseobj1112023-09-22T21:56:10.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:07.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-22T21:59:07.785 DEBUG:boto:path=/ffzdqu-61/obj115 2023-09-22T21:59:07.785 DEBUG:boto:auth_path=/ffzdqu-61/obj115 2023-09-22T21:59:07.786 DEBUG:boto:path=/ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.786 DEBUG:boto:auth_path=/ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.786 DEBUG:boto:Method: GET 2023-09-22T21:59:07.786 DEBUG:boto:Path: /ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.786 DEBUG:boto:Data: 2023-09-22T21:59:07.786 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.786 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.786 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.786 DEBUG:boto:Params: {} 2023-09-22T21:59:07.786 DEBUG:boto:Token: None 2023-09-22T21:59:07.787 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.787 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4eE6xcRss+RrY3+zhLWG0SBZWag= 2023-09-22T21:59:07.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4eE6xcRss+RrY3+zhLWG0SBZWag=', 'Content-Length': '0'} 2023-09-22T21:59:07.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfe8dd06626dad8a-00650e0e2b-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.790 DEBUG:boto:path=/ffzdqu-61/obj115 2023-09-22T21:59:07.790 DEBUG:boto:auth_path=/ffzdqu-61/obj115 2023-09-22T21:59:07.790 DEBUG:boto:path=/ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.790 DEBUG:boto:auth_path=/ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.790 DEBUG:boto:Method: GET 2023-09-22T21:59:07.790 DEBUG:boto:Path: /ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.790 DEBUG:boto:Data: 2023-09-22T21:59:07.790 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.791 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.791 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.791 DEBUG:boto:Params: {} 2023-09-22T21:59:07.791 DEBUG:boto:Token: None 2023-09-22T21:59:07.791 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.791 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4eE6xcRss+RrY3+zhLWG0SBZWag= 2023-09-22T21:59:07.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4eE6xcRss+RrY3+zhLWG0SBZWag=', 'Content-Length': '0'} 2023-09-22T21:59:07.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000359191b379a3b2a1-00650e0e2b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.834 DEBUG:boto:path=/ffzdqu-61/obj115 2023-09-22T21:59:07.834 DEBUG:boto:auth_path=/ffzdqu-61/obj115 2023-09-22T21:59:07.834 DEBUG:boto:path=/ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.835 DEBUG:boto:auth_path=/ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.835 DEBUG:boto:Method: HEAD 2023-09-22T21:59:07.835 DEBUG:boto:Path: /ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.835 DEBUG:boto:Data: 2023-09-22T21:59:07.835 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.835 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.835 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.835 DEBUG:boto:Params: {} 2023-09-22T21:59:07.835 DEBUG:boto:Token: None 2023-09-22T21:59:07.835 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.836 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wALfkBu2uAX3qbNCntVHZjihOLs= 2023-09-22T21:59:07.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wALfkBu2uAX3qbNCntVHZjihOLs=', 'Content-Length': '0'} 2023-09-22T21:59:07.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ffe98a067b49f23-00650e0e2b-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.839 DEBUG:boto:path=/ffzdqu-61/obj115 2023-09-22T21:59:07.839 DEBUG:boto:auth_path=/ffzdqu-61/obj115 2023-09-22T21:59:07.839 DEBUG:boto:path=/ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.839 DEBUG:boto:auth_path=/ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.839 DEBUG:boto:Method: HEAD 2023-09-22T21:59:07.839 DEBUG:boto:Path: /ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.839 DEBUG:boto:Data: 2023-09-22T21:59:07.839 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.839 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.840 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.840 DEBUG:boto:Params: {} 2023-09-22T21:59:07.840 DEBUG:boto:Token: None 2023-09-22T21:59:07.840 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.840 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wALfkBu2uAX3qbNCntVHZjihOLs= 2023-09-22T21:59:07.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wALfkBu2uAX3qbNCntVHZjihOLs=', 'Content-Length': '0'} 2023-09-22T21:59:07.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c082044a2dcaf4a-00650e0e2b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-22T21:59:07.843 DEBUG:boto:path=/ffzdqu-61/obj115 2023-09-22T21:59:07.843 DEBUG:boto:auth_path=/ffzdqu-61/obj115 2023-09-22T21:59:07.843 DEBUG:boto:path=/ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.843 DEBUG:boto:auth_path=/ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.844 DEBUG:boto:Method: GET 2023-09-22T21:59:07.844 DEBUG:boto:Path: /ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.844 DEBUG:boto:Data: 2023-09-22T21:59:07.844 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.844 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.844 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.844 DEBUG:boto:Params: {} 2023-09-22T21:59:07.844 DEBUG:boto:Token: None 2023-09-22T21:59:07.844 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.845 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4eE6xcRss+RrY3+zhLWG0SBZWag= 2023-09-22T21:59:07.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4eE6xcRss+RrY3+zhLWG0SBZWag=', 'Content-Length': '0'} 2023-09-22T21:59:07.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5f93db7a4a38563-00650e0e2b-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.890 DEBUG:boto:path=/ffzdqu-61/obj115 2023-09-22T21:59:07.890 DEBUG:boto:auth_path=/ffzdqu-61/obj115 2023-09-22T21:59:07.890 DEBUG:boto:path=/ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.890 DEBUG:boto:auth_path=/ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.891 DEBUG:boto:Method: GET 2023-09-22T21:59:07.891 DEBUG:boto:Path: /ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.891 DEBUG:boto:Data: 2023-09-22T21:59:07.891 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.891 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.891 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.891 DEBUG:boto:Params: {} 2023-09-22T21:59:07.891 DEBUG:boto:Token: None 2023-09-22T21:59:07.891 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj115?versionId=null 2023-09-22T21:59:07.892 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4eE6xcRss+RrY3+zhLWG0SBZWag= 2023-09-22T21:59:07.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4eE6xcRss+RrY3+zhLWG0SBZWag=', 'Content-Length': '0'} 2023-09-22T21:59:07.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000980f38453e343c1b-00650e0e2b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.896 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:07.896 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:07.896 DEBUG:boto:path=/ffzdqu-61/?prefix=obj115 2023-09-22T21:59:07.896 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj115 2023-09-22T21:59:07.896 DEBUG:boto:Method: GET 2023-09-22T21:59:07.896 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj115 2023-09-22T21:59:07.896 DEBUG:boto:Data: 2023-09-22T21:59:07.897 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.897 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.897 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.897 DEBUG:boto:Params: {} 2023-09-22T21:59:07.897 DEBUG:boto:Token: None 2023-09-22T21:59:07.897 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/ 2023-09-22T21:59:07.897 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro= 2023-09-22T21:59:07.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro=', 'Content-Length': '0'} 2023-09-22T21:59:07.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006910c5ae740323f4-00650e0e2b-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.904 DEBUG:boto:b'ffzdqu-61obj1151000falseobj1152023-09-22T21:56:09.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:07.904 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:07.904 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:07.904 DEBUG:boto:path=/ffzdqu-61/?prefix=obj115 2023-09-22T21:59:07.904 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj115 2023-09-22T21:59:07.904 DEBUG:boto:Method: GET 2023-09-22T21:59:07.905 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj115 2023-09-22T21:59:07.905 DEBUG:boto:Data: 2023-09-22T21:59:07.905 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.905 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.905 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.905 DEBUG:boto:Params: {} 2023-09-22T21:59:07.905 DEBUG:boto:Token: None 2023-09-22T21:59:07.905 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/ 2023-09-22T21:59:07.905 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro= 2023-09-22T21:59:07.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro=', 'Content-Length': '0'} 2023-09-22T21:59:07.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000148a9e78df2d662-00650e0e2b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.918 DEBUG:boto:b'ffzdqu-61obj1151000falseobj1152023-09-22T21:56:09.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:07.919 DEBUG:boto:path=/ffzdqu-61/obj115 2023-09-22T21:59:07.919 DEBUG:boto:auth_path=/ffzdqu-61/obj115 2023-09-22T21:59:07.919 DEBUG:boto:Method: HEAD 2023-09-22T21:59:07.919 DEBUG:boto:Path: /ffzdqu-61/obj115 2023-09-22T21:59:07.919 DEBUG:boto:Data: 2023-09-22T21:59:07.919 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.919 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.919 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.919 DEBUG:boto:Params: {} 2023-09-22T21:59:07.920 DEBUG:boto:Token: None 2023-09-22T21:59:07.920 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj115 2023-09-22T21:59:07.920 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tguG/4MyFPVcpY35sXPSOjh7uUw= 2023-09-22T21:59:07.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tguG/4MyFPVcpY35sXPSOjh7uUw=', 'Content-Length': '0'} 2023-09-22T21:59:07.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000490974327332bf9c-00650e0e2b-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.923 DEBUG:boto:path=/ffzdqu-61/obj115 2023-09-22T21:59:07.923 DEBUG:boto:auth_path=/ffzdqu-61/obj115 2023-09-22T21:59:07.923 DEBUG:boto:Method: HEAD 2023-09-22T21:59:07.923 DEBUG:boto:Path: /ffzdqu-61/obj115 2023-09-22T21:59:07.923 DEBUG:boto:Data: 2023-09-22T21:59:07.923 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.923 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.924 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.924 DEBUG:boto:Params: {} 2023-09-22T21:59:07.924 DEBUG:boto:Token: None 2023-09-22T21:59:07.924 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj115 2023-09-22T21:59:07.924 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tguG/4MyFPVcpY35sXPSOjh7uUw= 2023-09-22T21:59:07.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tguG/4MyFPVcpY35sXPSOjh7uUw=', 'Content-Length': '0'} 2023-09-22T21:59:07.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036897e329f530d55-00650e0e2b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-22T21:59:07.927 DEBUG:boto:path=/ffzdqu-61/obj115 2023-09-22T21:59:07.927 DEBUG:boto:auth_path=/ffzdqu-61/obj115 2023-09-22T21:59:07.927 DEBUG:boto:Method: GET 2023-09-22T21:59:07.927 DEBUG:boto:Path: /ffzdqu-61/obj115 2023-09-22T21:59:07.927 DEBUG:boto:Data: 2023-09-22T21:59:07.927 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.928 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.928 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.928 DEBUG:boto:Params: {} 2023-09-22T21:59:07.928 DEBUG:boto:Token: None 2023-09-22T21:59:07.928 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj115 2023-09-22T21:59:07.928 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rbXJZWArf0uS3soF9eIwQz/uudw= 2023-09-22T21:59:07.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rbXJZWArf0uS3soF9eIwQz/uudw=', 'Content-Length': '0'} 2023-09-22T21:59:07.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005425978e0f6cd5f4-00650e0e2b-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.974 DEBUG:boto:path=/ffzdqu-61/obj115 2023-09-22T21:59:07.974 DEBUG:boto:auth_path=/ffzdqu-61/obj115 2023-09-22T21:59:07.974 DEBUG:boto:Method: GET 2023-09-22T21:59:07.975 DEBUG:boto:Path: /ffzdqu-61/obj115 2023-09-22T21:59:07.975 DEBUG:boto:Data: 2023-09-22T21:59:07.975 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.975 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.975 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.975 DEBUG:boto:Params: {} 2023-09-22T21:59:07.975 DEBUG:boto:Token: None 2023-09-22T21:59:07.975 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/obj115 2023-09-22T21:59:07.975 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rbXJZWArf0uS3soF9eIwQz/uudw= 2023-09-22T21:59:07.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rbXJZWArf0uS3soF9eIwQz/uudw=', 'Content-Length': '0'} 2023-09-22T21:59:07.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000595faa2249b837be-00650e0e2b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.979 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:07.979 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:07.979 DEBUG:boto:path=/ffzdqu-61/?prefix=obj115 2023-09-22T21:59:07.979 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj115 2023-09-22T21:59:07.979 DEBUG:boto:Method: GET 2023-09-22T21:59:07.979 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj115 2023-09-22T21:59:07.979 DEBUG:boto:Data: 2023-09-22T21:59:07.979 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.979 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.979 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.980 DEBUG:boto:Params: {} 2023-09-22T21:59:07.980 DEBUG:boto:Token: None 2023-09-22T21:59:07.980 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/ 2023-09-22T21:59:07.980 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro= 2023-09-22T21:59:07.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro=', 'Content-Length': '0'} 2023-09-22T21:59:07.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092f125685fdbabf9-00650e0e2b-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.986 DEBUG:boto:b'ffzdqu-61obj1151000falseobj1152023-09-22T21:56:09.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:07.986 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:07.986 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:07.986 DEBUG:boto:path=/ffzdqu-61/?prefix=obj115 2023-09-22T21:59:07.987 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj115 2023-09-22T21:59:07.987 DEBUG:boto:Method: GET 2023-09-22T21:59:07.987 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj115 2023-09-22T21:59:07.987 DEBUG:boto:Data: 2023-09-22T21:59:07.987 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.987 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:07.987 DEBUG:boto:Port: 8000 2023-09-22T21:59:07.987 DEBUG:boto:Params: {} 2023-09-22T21:59:07.987 DEBUG:boto:Token: None 2023-09-22T21:59:07.988 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:07 GMT /ffzdqu-61/ 2023-09-22T21:59:07.988 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro= 2023-09-22T21:59:07.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:07 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8yY/VVNo124n6/A0r/HbKhS+fro=', 'Content-Length': '0'} 2023-09-22T21:59:07.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4e63eb7fca8ff85-00650e0e2b-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:07.998 DEBUG:boto:b'ffzdqu-61obj1151000falseobj1152023-09-22T21:56:09.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:07.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-22T21:59:07.998 DEBUG:boto:path=/ffzdqu-61/obj116 2023-09-22T21:59:07.999 DEBUG:boto:auth_path=/ffzdqu-61/obj116 2023-09-22T21:59:07.999 DEBUG:boto:path=/ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:07.999 DEBUG:boto:auth_path=/ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:07.999 DEBUG:boto:Method: GET 2023-09-22T21:59:07.999 DEBUG:boto:Path: /ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:07.999 DEBUG:boto:Data: 2023-09-22T21:59:07.999 DEBUG:boto:Headers: {} 2023-09-22T21:59:07.999 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:07.999 DEBUG:boto:Port: 8001 2023-09-22T21:59:07.999 DEBUG:boto:Params: {} 2023-09-22T21:59:08.000 DEBUG:boto:Token: None 2023-09-22T21:59:08.000 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:08.000 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kRVIHGHyM7RKMqeH7xhsmzq5RuY= 2023-09-22T21:59:08.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kRVIHGHyM7RKMqeH7xhsmzq5RuY=', 'Content-Length': '0'} 2023-09-22T21:59:08.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b39b8f44856ace1-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.003 DEBUG:boto:path=/ffzdqu-61/obj116 2023-09-22T21:59:08.003 DEBUG:boto:auth_path=/ffzdqu-61/obj116 2023-09-22T21:59:08.003 DEBUG:boto:path=/ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:08.003 DEBUG:boto:auth_path=/ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:08.003 DEBUG:boto:Method: GET 2023-09-22T21:59:08.004 DEBUG:boto:Path: /ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:08.004 DEBUG:boto:Data: 2023-09-22T21:59:08.004 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.004 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.004 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.004 DEBUG:boto:Params: {} 2023-09-22T21:59:08.004 DEBUG:boto:Token: None 2023-09-22T21:59:08.004 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:08.004 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kRVIHGHyM7RKMqeH7xhsmzq5RuY= 2023-09-22T21:59:08.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kRVIHGHyM7RKMqeH7xhsmzq5RuY=', 'Content-Length': '0'} 2023-09-22T21:59:08.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008834b3a5572bb3f2-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.007 DEBUG:boto:path=/ffzdqu-61/obj116 2023-09-22T21:59:08.008 DEBUG:boto:auth_path=/ffzdqu-61/obj116 2023-09-22T21:59:08.008 DEBUG:boto:path=/ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:08.008 DEBUG:boto:auth_path=/ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:08.008 DEBUG:boto:Method: HEAD 2023-09-22T21:59:08.008 DEBUG:boto:Path: /ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:08.008 DEBUG:boto:Data: 2023-09-22T21:59:08.008 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.008 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.008 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.008 DEBUG:boto:Params: {} 2023-09-22T21:59:08.009 DEBUG:boto:Token: None 2023-09-22T21:59:08.009 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:08.009 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jq3ok51HIhRSkQx4ElTqLdjXeMc= 2023-09-22T21:59:08.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jq3ok51HIhRSkQx4ElTqLdjXeMc=', 'Content-Length': '0'} 2023-09-22T21:59:08.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001d45ad0b1db354b-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.012 DEBUG:boto:path=/ffzdqu-61/obj116 2023-09-22T21:59:08.012 DEBUG:boto:auth_path=/ffzdqu-61/obj116 2023-09-22T21:59:08.012 DEBUG:boto:path=/ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:08.012 DEBUG:boto:auth_path=/ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:08.012 DEBUG:boto:Method: HEAD 2023-09-22T21:59:08.012 DEBUG:boto:Path: /ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:08.012 DEBUG:boto:Data: 2023-09-22T21:59:08.012 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.013 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.013 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.013 DEBUG:boto:Params: {} 2023-09-22T21:59:08.013 DEBUG:boto:Token: None 2023-09-22T21:59:08.013 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:08.013 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jq3ok51HIhRSkQx4ElTqLdjXeMc= 2023-09-22T21:59:08.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jq3ok51HIhRSkQx4ElTqLdjXeMc=', 'Content-Length': '0'} 2023-09-22T21:59:08.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010cc5a6d6d587f08-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-22T21:59:08.016 DEBUG:boto:path=/ffzdqu-61/obj116 2023-09-22T21:59:08.016 DEBUG:boto:auth_path=/ffzdqu-61/obj116 2023-09-22T21:59:08.016 DEBUG:boto:path=/ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:08.016 DEBUG:boto:auth_path=/ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:08.017 DEBUG:boto:Method: GET 2023-09-22T21:59:08.017 DEBUG:boto:Path: /ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:08.017 DEBUG:boto:Data: 2023-09-22T21:59:08.017 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.017 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.017 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.017 DEBUG:boto:Params: {} 2023-09-22T21:59:08.017 DEBUG:boto:Token: None 2023-09-22T21:59:08.017 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:08.018 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kRVIHGHyM7RKMqeH7xhsmzq5RuY= 2023-09-22T21:59:08.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kRVIHGHyM7RKMqeH7xhsmzq5RuY=', 'Content-Length': '0'} 2023-09-22T21:59:08.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000002ad1b4f5b03d7e-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.062 DEBUG:boto:path=/ffzdqu-61/obj116 2023-09-22T21:59:08.157 DEBUG:boto:auth_path=/ffzdqu-61/obj116 2023-09-22T21:59:08.157 DEBUG:boto:path=/ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:08.157 DEBUG:boto:auth_path=/ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:08.157 DEBUG:boto:Method: GET 2023-09-22T21:59:08.158 DEBUG:boto:Path: /ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:08.158 DEBUG:boto:Data: 2023-09-22T21:59:08.158 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.158 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.158 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.158 DEBUG:boto:Params: {} 2023-09-22T21:59:08.158 DEBUG:boto:Token: None 2023-09-22T21:59:08.159 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj116?versionId=null 2023-09-22T21:59:08.159 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kRVIHGHyM7RKMqeH7xhsmzq5RuY= 2023-09-22T21:59:08.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kRVIHGHyM7RKMqeH7xhsmzq5RuY=', 'Content-Length': '0'} 2023-09-22T21:59:08.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ad50bf9a6b87c91-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.162 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:08.162 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:08.162 DEBUG:boto:path=/ffzdqu-61/?prefix=obj116 2023-09-22T21:59:08.163 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj116 2023-09-22T21:59:08.163 DEBUG:boto:Method: GET 2023-09-22T21:59:08.163 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj116 2023-09-22T21:59:08.163 DEBUG:boto:Data: 2023-09-22T21:59:08.163 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.163 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.164 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.164 DEBUG:boto:Params: {} 2023-09-22T21:59:08.164 DEBUG:boto:Token: None 2023-09-22T21:59:08.164 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/ 2023-09-22T21:59:08.164 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8= 2023-09-22T21:59:08.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8=', 'Content-Length': '0'} 2023-09-22T21:59:08.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4e8608234927a92-00650e0e2c-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.169 DEBUG:boto:b'ffzdqu-61obj1161000falseobj1162023-09-22T21:56:10.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:08.169 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:08.170 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:08.170 DEBUG:boto:path=/ffzdqu-61/?prefix=obj116 2023-09-22T21:59:08.170 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj116 2023-09-22T21:59:08.170 DEBUG:boto:Method: GET 2023-09-22T21:59:08.170 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj116 2023-09-22T21:59:08.170 DEBUG:boto:Data: 2023-09-22T21:59:08.170 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.170 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.171 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.171 DEBUG:boto:Params: {} 2023-09-22T21:59:08.171 DEBUG:boto:Token: None 2023-09-22T21:59:08.171 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/ 2023-09-22T21:59:08.171 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8= 2023-09-22T21:59:08.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8=', 'Content-Length': '0'} 2023-09-22T21:59:08.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000994b8d73793b66ab-00650e0e2c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.181 DEBUG:boto:b'ffzdqu-61obj1161000falseobj1162023-09-22T21:56:10.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:08.182 DEBUG:boto:path=/ffzdqu-61/obj116 2023-09-22T21:59:08.182 DEBUG:boto:auth_path=/ffzdqu-61/obj116 2023-09-22T21:59:08.183 DEBUG:boto:Method: HEAD 2023-09-22T21:59:08.183 DEBUG:boto:Path: /ffzdqu-61/obj116 2023-09-22T21:59:08.183 DEBUG:boto:Data: 2023-09-22T21:59:08.183 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.183 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.184 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.184 DEBUG:boto:Params: {} 2023-09-22T21:59:08.184 DEBUG:boto:Token: None 2023-09-22T21:59:08.184 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj116 2023-09-22T21:59:08.185 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XbtLzpTrPadAtcHojC3OXyZRJIs= 2023-09-22T21:59:08.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XbtLzpTrPadAtcHojC3OXyZRJIs=', 'Content-Length': '0'} 2023-09-22T21:59:08.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017b37a7031894ffb-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.188 DEBUG:boto:path=/ffzdqu-61/obj116 2023-09-22T21:59:08.188 DEBUG:boto:auth_path=/ffzdqu-61/obj116 2023-09-22T21:59:08.188 DEBUG:boto:Method: HEAD 2023-09-22T21:59:08.189 DEBUG:boto:Path: /ffzdqu-61/obj116 2023-09-22T21:59:08.189 DEBUG:boto:Data: 2023-09-22T21:59:08.189 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.189 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.190 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.190 DEBUG:boto:Params: {} 2023-09-22T21:59:08.190 DEBUG:boto:Token: None 2023-09-22T21:59:08.190 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj116 2023-09-22T21:59:08.190 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XbtLzpTrPadAtcHojC3OXyZRJIs= 2023-09-22T21:59:08.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XbtLzpTrPadAtcHojC3OXyZRJIs=', 'Content-Length': '0'} 2023-09-22T21:59:08.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f011d66485e12eef-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-22T21:59:08.194 DEBUG:boto:path=/ffzdqu-61/obj116 2023-09-22T21:59:08.194 DEBUG:boto:auth_path=/ffzdqu-61/obj116 2023-09-22T21:59:08.195 DEBUG:boto:Method: GET 2023-09-22T21:59:08.195 DEBUG:boto:Path: /ffzdqu-61/obj116 2023-09-22T21:59:08.195 DEBUG:boto:Data: 2023-09-22T21:59:08.195 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.195 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.196 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.196 DEBUG:boto:Params: {} 2023-09-22T21:59:08.196 DEBUG:boto:Token: None 2023-09-22T21:59:08.197 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj116 2023-09-22T21:59:08.197 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bvWIY5nKf4LspxUcZ1lbxQ0BPF4= 2023-09-22T21:59:08.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bvWIY5nKf4LspxUcZ1lbxQ0BPF4=', 'Content-Length': '0'} 2023-09-22T21:59:08.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008866788f5d525dc9-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.200 DEBUG:boto:path=/ffzdqu-61/obj116 2023-09-22T21:59:08.201 DEBUG:boto:auth_path=/ffzdqu-61/obj116 2023-09-22T21:59:08.201 DEBUG:boto:Method: GET 2023-09-22T21:59:08.201 DEBUG:boto:Path: /ffzdqu-61/obj116 2023-09-22T21:59:08.201 DEBUG:boto:Data: 2023-09-22T21:59:08.201 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.202 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.202 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.202 DEBUG:boto:Params: {} 2023-09-22T21:59:08.202 DEBUG:boto:Token: None 2023-09-22T21:59:08.203 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj116 2023-09-22T21:59:08.203 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bvWIY5nKf4LspxUcZ1lbxQ0BPF4= 2023-09-22T21:59:08.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bvWIY5nKf4LspxUcZ1lbxQ0BPF4=', 'Content-Length': '0'} 2023-09-22T21:59:08.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048da3430b7c938d7-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.206 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:08.206 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:08.207 DEBUG:boto:path=/ffzdqu-61/?prefix=obj116 2023-09-22T21:59:08.207 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj116 2023-09-22T21:59:08.207 DEBUG:boto:Method: GET 2023-09-22T21:59:08.207 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj116 2023-09-22T21:59:08.207 DEBUG:boto:Data: 2023-09-22T21:59:08.207 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.208 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.208 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.208 DEBUG:boto:Params: {} 2023-09-22T21:59:08.208 DEBUG:boto:Token: None 2023-09-22T21:59:08.209 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/ 2023-09-22T21:59:08.209 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8= 2023-09-22T21:59:08.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8=', 'Content-Length': '0'} 2023-09-22T21:59:08.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097738701657d31d3-00650e0e2c-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.254 DEBUG:boto:b'ffzdqu-61obj1161000falseobj1162023-09-22T21:56:10.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:08.255 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:08.255 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:08.255 DEBUG:boto:path=/ffzdqu-61/?prefix=obj116 2023-09-22T21:59:08.255 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj116 2023-09-22T21:59:08.256 DEBUG:boto:Method: GET 2023-09-22T21:59:08.256 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj116 2023-09-22T21:59:08.256 DEBUG:boto:Data: 2023-09-22T21:59:08.256 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.256 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.257 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.257 DEBUG:boto:Params: {} 2023-09-22T21:59:08.257 DEBUG:boto:Token: None 2023-09-22T21:59:08.257 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/ 2023-09-22T21:59:08.258 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8= 2023-09-22T21:59:08.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8=', 'Content-Length': '0'} 2023-09-22T21:59:08.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfc7ad473d66d123-00650e0e2c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.268 DEBUG:boto:b'ffzdqu-61obj1161000falseobj1162023-09-22T21:56:10.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:08.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-22T21:59:08.268 DEBUG:boto:path=/ffzdqu-61/obj124 2023-09-22T21:59:08.269 DEBUG:boto:auth_path=/ffzdqu-61/obj124 2023-09-22T21:59:08.269 DEBUG:boto:path=/ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.269 DEBUG:boto:auth_path=/ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.269 DEBUG:boto:Method: GET 2023-09-22T21:59:08.269 DEBUG:boto:Path: /ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.269 DEBUG:boto:Data: 2023-09-22T21:59:08.270 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.270 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.270 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.270 DEBUG:boto:Params: {} 2023-09-22T21:59:08.271 DEBUG:boto:Token: None 2023-09-22T21:59:08.271 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.271 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fdTZ2H8oGbNjwQBvV8cjdD4UUls= 2023-09-22T21:59:08.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fdTZ2H8oGbNjwQBvV8cjdD4UUls=', 'Content-Length': '0'} 2023-09-22T21:59:08.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000350e20d290e1ee10-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.275 DEBUG:boto:path=/ffzdqu-61/obj124 2023-09-22T21:59:08.275 DEBUG:boto:auth_path=/ffzdqu-61/obj124 2023-09-22T21:59:08.275 DEBUG:boto:path=/ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.275 DEBUG:boto:auth_path=/ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.275 DEBUG:boto:Method: GET 2023-09-22T21:59:08.276 DEBUG:boto:Path: /ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.276 DEBUG:boto:Data: 2023-09-22T21:59:08.276 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.276 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.276 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.277 DEBUG:boto:Params: {} 2023-09-22T21:59:08.277 DEBUG:boto:Token: None 2023-09-22T21:59:08.277 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.277 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fdTZ2H8oGbNjwQBvV8cjdD4UUls= 2023-09-22T21:59:08.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fdTZ2H8oGbNjwQBvV8cjdD4UUls=', 'Content-Length': '0'} 2023-09-22T21:59:08.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003bc1fd258a9ee2e-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.281 DEBUG:boto:path=/ffzdqu-61/obj124 2023-09-22T21:59:08.281 DEBUG:boto:auth_path=/ffzdqu-61/obj124 2023-09-22T21:59:08.281 DEBUG:boto:path=/ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.282 DEBUG:boto:auth_path=/ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.282 DEBUG:boto:Method: HEAD 2023-09-22T21:59:08.282 DEBUG:boto:Path: /ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.282 DEBUG:boto:Data: 2023-09-22T21:59:08.282 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.282 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.283 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.283 DEBUG:boto:Params: {} 2023-09-22T21:59:08.283 DEBUG:boto:Token: None 2023-09-22T21:59:08.283 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.284 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TuWS3cyfUdbRuuzhMnl0SoU4LGU= 2023-09-22T21:59:08.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TuWS3cyfUdbRuuzhMnl0SoU4LGU=', 'Content-Length': '0'} 2023-09-22T21:59:08.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9388e10fb0f95af-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.287 DEBUG:boto:path=/ffzdqu-61/obj124 2023-09-22T21:59:08.287 DEBUG:boto:auth_path=/ffzdqu-61/obj124 2023-09-22T21:59:08.287 DEBUG:boto:path=/ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.287 DEBUG:boto:auth_path=/ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.288 DEBUG:boto:Method: HEAD 2023-09-22T21:59:08.288 DEBUG:boto:Path: /ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.288 DEBUG:boto:Data: 2023-09-22T21:59:08.288 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.288 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.288 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.289 DEBUG:boto:Params: {} 2023-09-22T21:59:08.289 DEBUG:boto:Token: None 2023-09-22T21:59:08.289 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.289 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TuWS3cyfUdbRuuzhMnl0SoU4LGU= 2023-09-22T21:59:08.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TuWS3cyfUdbRuuzhMnl0SoU4LGU=', 'Content-Length': '0'} 2023-09-22T21:59:08.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c18cba84f6483dd-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-22T21:59:08.293 DEBUG:boto:path=/ffzdqu-61/obj124 2023-09-22T21:59:08.293 DEBUG:boto:auth_path=/ffzdqu-61/obj124 2023-09-22T21:59:08.293 DEBUG:boto:path=/ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.294 DEBUG:boto:auth_path=/ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.294 DEBUG:boto:Method: GET 2023-09-22T21:59:08.294 DEBUG:boto:Path: /ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.294 DEBUG:boto:Data: 2023-09-22T21:59:08.294 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.295 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.295 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.295 DEBUG:boto:Params: {} 2023-09-22T21:59:08.295 DEBUG:boto:Token: None 2023-09-22T21:59:08.295 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.296 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fdTZ2H8oGbNjwQBvV8cjdD4UUls= 2023-09-22T21:59:08.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fdTZ2H8oGbNjwQBvV8cjdD4UUls=', 'Content-Length': '0'} 2023-09-22T21:59:08.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1dd500a09f1fab4-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.342 DEBUG:boto:path=/ffzdqu-61/obj124 2023-09-22T21:59:08.342 DEBUG:boto:auth_path=/ffzdqu-61/obj124 2023-09-22T21:59:08.343 DEBUG:boto:path=/ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.343 DEBUG:boto:auth_path=/ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.343 DEBUG:boto:Method: GET 2023-09-22T21:59:08.343 DEBUG:boto:Path: /ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.343 DEBUG:boto:Data: 2023-09-22T21:59:08.344 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.344 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.344 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.344 DEBUG:boto:Params: {} 2023-09-22T21:59:08.344 DEBUG:boto:Token: None 2023-09-22T21:59:08.345 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj124?versionId=null 2023-09-22T21:59:08.345 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fdTZ2H8oGbNjwQBvV8cjdD4UUls= 2023-09-22T21:59:08.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fdTZ2H8oGbNjwQBvV8cjdD4UUls=', 'Content-Length': '0'} 2023-09-22T21:59:08.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8641cc987af3faa-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.348 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:08.348 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:08.349 DEBUG:boto:path=/ffzdqu-61/?prefix=obj124 2023-09-22T21:59:08.349 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj124 2023-09-22T21:59:08.349 DEBUG:boto:Method: GET 2023-09-22T21:59:08.349 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj124 2023-09-22T21:59:08.349 DEBUG:boto:Data: 2023-09-22T21:59:08.350 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.350 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.350 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.350 DEBUG:boto:Params: {} 2023-09-22T21:59:08.350 DEBUG:boto:Token: None 2023-09-22T21:59:08.351 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/ 2023-09-22T21:59:08.351 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8= 2023-09-22T21:59:08.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8=', 'Content-Length': '0'} 2023-09-22T21:59:08.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010701b3f1e596fb8-00650e0e2c-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.356 DEBUG:boto:b'ffzdqu-61obj1241000falseobj1242023-09-22T21:56:10.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:08.357 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:08.357 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:08.357 DEBUG:boto:path=/ffzdqu-61/?prefix=obj124 2023-09-22T21:59:08.357 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj124 2023-09-22T21:59:08.358 DEBUG:boto:Method: GET 2023-09-22T21:59:08.358 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj124 2023-09-22T21:59:08.358 DEBUG:boto:Data: 2023-09-22T21:59:08.358 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.358 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.359 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.359 DEBUG:boto:Params: {} 2023-09-22T21:59:08.359 DEBUG:boto:Token: None 2023-09-22T21:59:08.359 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/ 2023-09-22T21:59:08.360 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8= 2023-09-22T21:59:08.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8=', 'Content-Length': '0'} 2023-09-22T21:59:08.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6ff2bdd3938738c-00650e0e2c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.410 DEBUG:boto:b'ffzdqu-61obj1241000falseobj1242023-09-22T21:56:10.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:08.411 DEBUG:boto:path=/ffzdqu-61/obj124 2023-09-22T21:59:08.411 DEBUG:boto:auth_path=/ffzdqu-61/obj124 2023-09-22T21:59:08.411 DEBUG:boto:Method: HEAD 2023-09-22T21:59:08.411 DEBUG:boto:Path: /ffzdqu-61/obj124 2023-09-22T21:59:08.412 DEBUG:boto:Data: 2023-09-22T21:59:08.412 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.412 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.412 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.412 DEBUG:boto:Params: {} 2023-09-22T21:59:08.413 DEBUG:boto:Token: None 2023-09-22T21:59:08.413 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj124 2023-09-22T21:59:08.413 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yfCsIx3IBRFMXis0wdq+BBS3XqM= 2023-09-22T21:59:08.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yfCsIx3IBRFMXis0wdq+BBS3XqM=', 'Content-Length': '0'} 2023-09-22T21:59:08.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b9219149b110af1-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.417 DEBUG:boto:path=/ffzdqu-61/obj124 2023-09-22T21:59:08.417 DEBUG:boto:auth_path=/ffzdqu-61/obj124 2023-09-22T21:59:08.417 DEBUG:boto:Method: HEAD 2023-09-22T21:59:08.418 DEBUG:boto:Path: /ffzdqu-61/obj124 2023-09-22T21:59:08.418 DEBUG:boto:Data: 2023-09-22T21:59:08.418 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.418 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.418 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.419 DEBUG:boto:Params: {} 2023-09-22T21:59:08.419 DEBUG:boto:Token: None 2023-09-22T21:59:08.419 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj124 2023-09-22T21:59:08.419 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yfCsIx3IBRFMXis0wdq+BBS3XqM= 2023-09-22T21:59:08.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yfCsIx3IBRFMXis0wdq+BBS3XqM=', 'Content-Length': '0'} 2023-09-22T21:59:08.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c0acb6081327ffd-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-22T21:59:08.423 DEBUG:boto:path=/ffzdqu-61/obj124 2023-09-22T21:59:08.423 DEBUG:boto:auth_path=/ffzdqu-61/obj124 2023-09-22T21:59:08.424 DEBUG:boto:Method: GET 2023-09-22T21:59:08.424 DEBUG:boto:Path: /ffzdqu-61/obj124 2023-09-22T21:59:08.424 DEBUG:boto:Data: 2023-09-22T21:59:08.424 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.424 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.424 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.425 DEBUG:boto:Params: {} 2023-09-22T21:59:08.425 DEBUG:boto:Token: None 2023-09-22T21:59:08.425 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj124 2023-09-22T21:59:08.425 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gRP3RWFsurIx4kE8GRZJwlbCPeE= 2023-09-22T21:59:08.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gRP3RWFsurIx4kE8GRZJwlbCPeE=', 'Content-Length': '0'} 2023-09-22T21:59:08.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6239c621f54c677-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.429 DEBUG:boto:path=/ffzdqu-61/obj124 2023-09-22T21:59:08.429 DEBUG:boto:auth_path=/ffzdqu-61/obj124 2023-09-22T21:59:08.429 DEBUG:boto:Method: GET 2023-09-22T21:59:08.430 DEBUG:boto:Path: /ffzdqu-61/obj124 2023-09-22T21:59:08.430 DEBUG:boto:Data: 2023-09-22T21:59:08.430 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.430 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.430 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.431 DEBUG:boto:Params: {} 2023-09-22T21:59:08.431 DEBUG:boto:Token: None 2023-09-22T21:59:08.431 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj124 2023-09-22T21:59:08.431 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gRP3RWFsurIx4kE8GRZJwlbCPeE= 2023-09-22T21:59:08.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gRP3RWFsurIx4kE8GRZJwlbCPeE=', 'Content-Length': '0'} 2023-09-22T21:59:08.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092d33d66097d53a1-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.435 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:08.435 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:08.435 DEBUG:boto:path=/ffzdqu-61/?prefix=obj124 2023-09-22T21:59:08.435 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj124 2023-09-22T21:59:08.436 DEBUG:boto:Method: GET 2023-09-22T21:59:08.436 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj124 2023-09-22T21:59:08.436 DEBUG:boto:Data: 2023-09-22T21:59:08.436 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.436 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.436 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.437 DEBUG:boto:Params: {} 2023-09-22T21:59:08.437 DEBUG:boto:Token: None 2023-09-22T21:59:08.437 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/ 2023-09-22T21:59:08.438 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8= 2023-09-22T21:59:08.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8=', 'Content-Length': '0'} 2023-09-22T21:59:08.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000deba6d22c7bfea11-00650e0e2c-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.443 DEBUG:boto:b'ffzdqu-61obj1241000falseobj1242023-09-22T21:56:10.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:08.443 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:08.443 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:08.444 DEBUG:boto:path=/ffzdqu-61/?prefix=obj124 2023-09-22T21:59:08.444 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj124 2023-09-22T21:59:08.444 DEBUG:boto:Method: GET 2023-09-22T21:59:08.444 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj124 2023-09-22T21:59:08.444 DEBUG:boto:Data: 2023-09-22T21:59:08.445 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.445 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.445 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.445 DEBUG:boto:Params: {} 2023-09-22T21:59:08.446 DEBUG:boto:Token: None 2023-09-22T21:59:08.446 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/ 2023-09-22T21:59:08.446 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8= 2023-09-22T21:59:08.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8=', 'Content-Length': '0'} 2023-09-22T21:59:08.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ce9bee65842779e-00650e0e2c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.498 DEBUG:boto:b'ffzdqu-61obj1241000falseobj1242023-09-22T21:56:10.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:08.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-22T21:59:08.499 DEBUG:boto:path=/ffzdqu-61/obj145 2023-09-22T21:59:08.499 DEBUG:boto:auth_path=/ffzdqu-61/obj145 2023-09-22T21:59:08.499 DEBUG:boto:path=/ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.500 DEBUG:boto:auth_path=/ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.500 DEBUG:boto:Method: GET 2023-09-22T21:59:08.500 DEBUG:boto:Path: /ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.500 DEBUG:boto:Data: 2023-09-22T21:59:08.500 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.501 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.501 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.501 DEBUG:boto:Params: {} 2023-09-22T21:59:08.501 DEBUG:boto:Token: None 2023-09-22T21:59:08.501 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.502 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ASsO+4zC2889yaoQAlzlK9KAD9o= 2023-09-22T21:59:08.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ASsO+4zC2889yaoQAlzlK9KAD9o=', 'Content-Length': '0'} 2023-09-22T21:59:08.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000115bcbe8fde17fce-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.505 DEBUG:boto:path=/ffzdqu-61/obj145 2023-09-22T21:59:08.505 DEBUG:boto:auth_path=/ffzdqu-61/obj145 2023-09-22T21:59:08.505 DEBUG:boto:path=/ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.506 DEBUG:boto:auth_path=/ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.506 DEBUG:boto:Method: GET 2023-09-22T21:59:08.506 DEBUG:boto:Path: /ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.506 DEBUG:boto:Data: 2023-09-22T21:59:08.506 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.507 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.507 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.507 DEBUG:boto:Params: {} 2023-09-22T21:59:08.507 DEBUG:boto:Token: None 2023-09-22T21:59:08.508 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.508 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ASsO+4zC2889yaoQAlzlK9KAD9o= 2023-09-22T21:59:08.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ASsO+4zC2889yaoQAlzlK9KAD9o=', 'Content-Length': '0'} 2023-09-22T21:59:08.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000208b32bc23338305-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.511 DEBUG:boto:path=/ffzdqu-61/obj145 2023-09-22T21:59:08.511 DEBUG:boto:auth_path=/ffzdqu-61/obj145 2023-09-22T21:59:08.512 DEBUG:boto:path=/ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.512 DEBUG:boto:auth_path=/ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.512 DEBUG:boto:Method: HEAD 2023-09-22T21:59:08.512 DEBUG:boto:Path: /ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.513 DEBUG:boto:Data: 2023-09-22T21:59:08.513 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.513 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.513 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.513 DEBUG:boto:Params: {} 2023-09-22T21:59:08.514 DEBUG:boto:Token: None 2023-09-22T21:59:08.514 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.514 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CzhL2VAqw0HBdf+cEy57S58LC3Y= 2023-09-22T21:59:08.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CzhL2VAqw0HBdf+cEy57S58LC3Y=', 'Content-Length': '0'} 2023-09-22T21:59:08.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000763656f47a3601ac-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.518 DEBUG:boto:path=/ffzdqu-61/obj145 2023-09-22T21:59:08.518 DEBUG:boto:auth_path=/ffzdqu-61/obj145 2023-09-22T21:59:08.518 DEBUG:boto:path=/ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.519 DEBUG:boto:auth_path=/ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.519 DEBUG:boto:Method: HEAD 2023-09-22T21:59:08.519 DEBUG:boto:Path: /ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.519 DEBUG:boto:Data: 2023-09-22T21:59:08.519 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.520 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.520 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.520 DEBUG:boto:Params: {} 2023-09-22T21:59:08.520 DEBUG:boto:Token: None 2023-09-22T21:59:08.521 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.521 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CzhL2VAqw0HBdf+cEy57S58LC3Y= 2023-09-22T21:59:08.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CzhL2VAqw0HBdf+cEy57S58LC3Y=', 'Content-Length': '0'} 2023-09-22T21:59:08.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078be7392dc336596-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-22T21:59:08.524 DEBUG:boto:path=/ffzdqu-61/obj145 2023-09-22T21:59:08.525 DEBUG:boto:auth_path=/ffzdqu-61/obj145 2023-09-22T21:59:08.525 DEBUG:boto:path=/ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.525 DEBUG:boto:auth_path=/ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.525 DEBUG:boto:Method: GET 2023-09-22T21:59:08.525 DEBUG:boto:Path: /ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.526 DEBUG:boto:Data: 2023-09-22T21:59:08.526 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.526 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.526 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.526 DEBUG:boto:Params: {} 2023-09-22T21:59:08.527 DEBUG:boto:Token: None 2023-09-22T21:59:08.527 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.527 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ASsO+4zC2889yaoQAlzlK9KAD9o= 2023-09-22T21:59:08.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ASsO+4zC2889yaoQAlzlK9KAD9o=', 'Content-Length': '0'} 2023-09-22T21:59:08.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c45da86eb049eac-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.531 DEBUG:boto:path=/ffzdqu-61/obj145 2023-09-22T21:59:08.531 DEBUG:boto:auth_path=/ffzdqu-61/obj145 2023-09-22T21:59:08.531 DEBUG:boto:path=/ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.532 DEBUG:boto:auth_path=/ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.532 DEBUG:boto:Method: GET 2023-09-22T21:59:08.532 DEBUG:boto:Path: /ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.532 DEBUG:boto:Data: 2023-09-22T21:59:08.532 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.533 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.533 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.533 DEBUG:boto:Params: {} 2023-09-22T21:59:08.533 DEBUG:boto:Token: None 2023-09-22T21:59:08.533 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj145?versionId=null 2023-09-22T21:59:08.534 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ASsO+4zC2889yaoQAlzlK9KAD9o= 2023-09-22T21:59:08.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ASsO+4zC2889yaoQAlzlK9KAD9o=', 'Content-Length': '0'} 2023-09-22T21:59:08.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4b5078b3f340be8-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.578 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:08.578 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:08.579 DEBUG:boto:path=/ffzdqu-61/?prefix=obj145 2023-09-22T21:59:08.579 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj145 2023-09-22T21:59:08.579 DEBUG:boto:Method: GET 2023-09-22T21:59:08.579 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj145 2023-09-22T21:59:08.579 DEBUG:boto:Data: 2023-09-22T21:59:08.580 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.580 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.580 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.580 DEBUG:boto:Params: {} 2023-09-22T21:59:08.580 DEBUG:boto:Token: None 2023-09-22T21:59:08.581 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/ 2023-09-22T21:59:08.581 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8= 2023-09-22T21:59:08.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8=', 'Content-Length': '0'} 2023-09-22T21:59:08.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015d6db98c27b1d47-00650e0e2c-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.589 DEBUG:boto:b'ffzdqu-61obj1451000falseobj1452023-09-22T21:56:10.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:08.589 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:08.589 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:08.589 DEBUG:boto:path=/ffzdqu-61/?prefix=obj145 2023-09-22T21:59:08.590 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj145 2023-09-22T21:59:08.590 DEBUG:boto:Method: GET 2023-09-22T21:59:08.590 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj145 2023-09-22T21:59:08.590 DEBUG:boto:Data: 2023-09-22T21:59:08.590 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.591 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.591 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.591 DEBUG:boto:Params: {} 2023-09-22T21:59:08.591 DEBUG:boto:Token: None 2023-09-22T21:59:08.592 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/ 2023-09-22T21:59:08.592 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8= 2023-09-22T21:59:08.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8=', 'Content-Length': '0'} 2023-09-22T21:59:08.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc0be207b01ce083-00650e0e2c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.604 DEBUG:boto:b'ffzdqu-61obj1451000falseobj1452023-09-22T21:56:10.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:08.604 DEBUG:boto:path=/ffzdqu-61/obj145 2023-09-22T21:59:08.604 DEBUG:boto:auth_path=/ffzdqu-61/obj145 2023-09-22T21:59:08.605 DEBUG:boto:Method: HEAD 2023-09-22T21:59:08.605 DEBUG:boto:Path: /ffzdqu-61/obj145 2023-09-22T21:59:08.605 DEBUG:boto:Data: 2023-09-22T21:59:08.605 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.605 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.606 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.606 DEBUG:boto:Params: {} 2023-09-22T21:59:08.606 DEBUG:boto:Token: None 2023-09-22T21:59:08.606 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj145 2023-09-22T21:59:08.606 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OmFuRBoxCwduTZwn54q3ZEZGA/U= 2023-09-22T21:59:08.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OmFuRBoxCwduTZwn54q3ZEZGA/U=', 'Content-Length': '0'} 2023-09-22T21:59:08.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd9c2e135abcaf87-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.610 DEBUG:boto:path=/ffzdqu-61/obj145 2023-09-22T21:59:08.610 DEBUG:boto:auth_path=/ffzdqu-61/obj145 2023-09-22T21:59:08.610 DEBUG:boto:Method: HEAD 2023-09-22T21:59:08.610 DEBUG:boto:Path: /ffzdqu-61/obj145 2023-09-22T21:59:08.611 DEBUG:boto:Data: 2023-09-22T21:59:08.611 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.611 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.611 DEBUG:boto:Params: {} 2023-09-22T21:59:08.612 DEBUG:boto:Token: None 2023-09-22T21:59:08.612 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj145 2023-09-22T21:59:08.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OmFuRBoxCwduTZwn54q3ZEZGA/U= 2023-09-22T21:59:08.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OmFuRBoxCwduTZwn54q3ZEZGA/U=', 'Content-Length': '0'} 2023-09-22T21:59:08.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d2f985d4abb49b5-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-22T21:59:08.616 DEBUG:boto:path=/ffzdqu-61/obj145 2023-09-22T21:59:08.616 DEBUG:boto:auth_path=/ffzdqu-61/obj145 2023-09-22T21:59:08.616 DEBUG:boto:Method: GET 2023-09-22T21:59:08.616 DEBUG:boto:Path: /ffzdqu-61/obj145 2023-09-22T21:59:08.616 DEBUG:boto:Data: 2023-09-22T21:59:08.617 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.617 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.617 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.617 DEBUG:boto:Params: {} 2023-09-22T21:59:08.617 DEBUG:boto:Token: None 2023-09-22T21:59:08.618 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj145 2023-09-22T21:59:08.618 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cUTXXso4TJkZQYmNstuEUQTS+JE= 2023-09-22T21:59:08.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cUTXXso4TJkZQYmNstuEUQTS+JE=', 'Content-Length': '0'} 2023-09-22T21:59:08.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062280e805f015dec-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.622 DEBUG:boto:path=/ffzdqu-61/obj145 2023-09-22T21:59:08.622 DEBUG:boto:auth_path=/ffzdqu-61/obj145 2023-09-22T21:59:08.622 DEBUG:boto:Method: GET 2023-09-22T21:59:08.622 DEBUG:boto:Path: /ffzdqu-61/obj145 2023-09-22T21:59:08.623 DEBUG:boto:Data: 2023-09-22T21:59:08.623 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.623 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.623 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.623 DEBUG:boto:Params: {} 2023-09-22T21:59:08.624 DEBUG:boto:Token: None 2023-09-22T21:59:08.624 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj145 2023-09-22T21:59:08.624 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cUTXXso4TJkZQYmNstuEUQTS+JE= 2023-09-22T21:59:08.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cUTXXso4TJkZQYmNstuEUQTS+JE=', 'Content-Length': '0'} 2023-09-22T21:59:08.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081db8b1ae1a715e6-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.670 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:08.670 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:08.671 DEBUG:boto:path=/ffzdqu-61/?prefix=obj145 2023-09-22T21:59:08.671 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj145 2023-09-22T21:59:08.671 DEBUG:boto:Method: GET 2023-09-22T21:59:08.671 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj145 2023-09-22T21:59:08.671 DEBUG:boto:Data: 2023-09-22T21:59:08.672 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.672 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.672 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.672 DEBUG:boto:Params: {} 2023-09-22T21:59:08.672 DEBUG:boto:Token: None 2023-09-22T21:59:08.673 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/ 2023-09-22T21:59:08.673 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8= 2023-09-22T21:59:08.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8=', 'Content-Length': '0'} 2023-09-22T21:59:08.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0a13d5010ff7215-00650e0e2c-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.682 DEBUG:boto:b'ffzdqu-61obj1451000falseobj1452023-09-22T21:56:10.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:08.682 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:08.682 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:08.683 DEBUG:boto:path=/ffzdqu-61/?prefix=obj145 2023-09-22T21:59:08.683 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj145 2023-09-22T21:59:08.683 DEBUG:boto:Method: GET 2023-09-22T21:59:08.683 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj145 2023-09-22T21:59:08.683 DEBUG:boto:Data: 2023-09-22T21:59:08.684 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.684 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.684 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.684 DEBUG:boto:Params: {} 2023-09-22T21:59:08.685 DEBUG:boto:Token: None 2023-09-22T21:59:08.685 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/ 2023-09-22T21:59:08.685 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8= 2023-09-22T21:59:08.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8=', 'Content-Length': '0'} 2023-09-22T21:59:08.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c84e8669957e34f2-00650e0e2c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.696 DEBUG:boto:b'ffzdqu-61obj1451000falseobj1452023-09-22T21:56:10.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:08.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-09-22T21:59:08.697 DEBUG:boto:path=/ffzdqu-61/obj148 2023-09-22T21:59:08.697 DEBUG:boto:auth_path=/ffzdqu-61/obj148 2023-09-22T21:59:08.697 DEBUG:boto:path=/ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.698 DEBUG:boto:auth_path=/ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.698 DEBUG:boto:Method: GET 2023-09-22T21:59:08.698 DEBUG:boto:Path: /ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.698 DEBUG:boto:Data: 2023-09-22T21:59:08.698 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.699 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.699 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.699 DEBUG:boto:Params: {} 2023-09-22T21:59:08.699 DEBUG:boto:Token: None 2023-09-22T21:59:08.700 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.700 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n53KrmL7GNmmHLVDf8ZL97sStZA= 2023-09-22T21:59:08.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n53KrmL7GNmmHLVDf8ZL97sStZA=', 'Content-Length': '0'} 2023-09-22T21:59:08.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000727059761242fbbc-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.746 DEBUG:boto:path=/ffzdqu-61/obj148 2023-09-22T21:59:08.746 DEBUG:boto:auth_path=/ffzdqu-61/obj148 2023-09-22T21:59:08.746 DEBUG:boto:path=/ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.747 DEBUG:boto:auth_path=/ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.747 DEBUG:boto:Method: GET 2023-09-22T21:59:08.747 DEBUG:boto:Path: /ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.747 DEBUG:boto:Data: 2023-09-22T21:59:08.747 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.747 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.747 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.747 DEBUG:boto:Params: {} 2023-09-22T21:59:08.748 DEBUG:boto:Token: None 2023-09-22T21:59:08.748 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.748 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n53KrmL7GNmmHLVDf8ZL97sStZA= 2023-09-22T21:59:08.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n53KrmL7GNmmHLVDf8ZL97sStZA=', 'Content-Length': '0'} 2023-09-22T21:59:08.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074af1265d107436a-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.751 DEBUG:boto:path=/ffzdqu-61/obj148 2023-09-22T21:59:08.752 DEBUG:boto:auth_path=/ffzdqu-61/obj148 2023-09-22T21:59:08.752 DEBUG:boto:path=/ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.752 DEBUG:boto:auth_path=/ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.752 DEBUG:boto:Method: HEAD 2023-09-22T21:59:08.752 DEBUG:boto:Path: /ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.752 DEBUG:boto:Data: 2023-09-22T21:59:08.753 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.753 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.753 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.753 DEBUG:boto:Params: {} 2023-09-22T21:59:08.753 DEBUG:boto:Token: None 2023-09-22T21:59:08.753 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.753 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DiRz6nfnBRV3H46OwMySY5oVnjg= 2023-09-22T21:59:08.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DiRz6nfnBRV3H46OwMySY5oVnjg=', 'Content-Length': '0'} 2023-09-22T21:59:08.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc40db139ade64aa-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.756 DEBUG:boto:path=/ffzdqu-61/obj148 2023-09-22T21:59:08.757 DEBUG:boto:auth_path=/ffzdqu-61/obj148 2023-09-22T21:59:08.757 DEBUG:boto:path=/ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.757 DEBUG:boto:auth_path=/ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.757 DEBUG:boto:Method: HEAD 2023-09-22T21:59:08.757 DEBUG:boto:Path: /ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.757 DEBUG:boto:Data: 2023-09-22T21:59:08.758 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.758 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.758 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.758 DEBUG:boto:Params: {} 2023-09-22T21:59:08.758 DEBUG:boto:Token: None 2023-09-22T21:59:08.758 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.758 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DiRz6nfnBRV3H46OwMySY5oVnjg= 2023-09-22T21:59:08.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DiRz6nfnBRV3H46OwMySY5oVnjg=', 'Content-Length': '0'} 2023-09-22T21:59:08.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6495e4fbc7c01e3-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-09-22T21:59:08.762 DEBUG:boto:path=/ffzdqu-61/obj148 2023-09-22T21:59:08.762 DEBUG:boto:auth_path=/ffzdqu-61/obj148 2023-09-22T21:59:08.762 DEBUG:boto:path=/ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.762 DEBUG:boto:auth_path=/ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.762 DEBUG:boto:Method: GET 2023-09-22T21:59:08.762 DEBUG:boto:Path: /ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.762 DEBUG:boto:Data: 2023-09-22T21:59:08.763 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.763 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.763 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.763 DEBUG:boto:Params: {} 2023-09-22T21:59:08.763 DEBUG:boto:Token: None 2023-09-22T21:59:08.763 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.763 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n53KrmL7GNmmHLVDf8ZL97sStZA= 2023-09-22T21:59:08.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n53KrmL7GNmmHLVDf8ZL97sStZA=', 'Content-Length': '0'} 2023-09-22T21:59:08.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055dbddb3d40290b2-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.767 DEBUG:boto:path=/ffzdqu-61/obj148 2023-09-22T21:59:08.767 DEBUG:boto:auth_path=/ffzdqu-61/obj148 2023-09-22T21:59:08.767 DEBUG:boto:path=/ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.767 DEBUG:boto:auth_path=/ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.767 DEBUG:boto:Method: GET 2023-09-22T21:59:08.767 DEBUG:boto:Path: /ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.767 DEBUG:boto:Data: 2023-09-22T21:59:08.768 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.768 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.768 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.768 DEBUG:boto:Params: {} 2023-09-22T21:59:08.768 DEBUG:boto:Token: None 2023-09-22T21:59:08.768 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj148?versionId=null 2023-09-22T21:59:08.768 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n53KrmL7GNmmHLVDf8ZL97sStZA= 2023-09-22T21:59:08.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n53KrmL7GNmmHLVDf8ZL97sStZA=', 'Content-Length': '0'} 2023-09-22T21:59:08.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054ca6fa819a89c21-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.772 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:08.772 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:08.772 DEBUG:boto:path=/ffzdqu-61/?prefix=obj148 2023-09-22T21:59:08.772 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj148 2023-09-22T21:59:08.772 DEBUG:boto:Method: GET 2023-09-22T21:59:08.772 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj148 2023-09-22T21:59:08.772 DEBUG:boto:Data: 2023-09-22T21:59:08.773 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.773 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.773 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.773 DEBUG:boto:Params: {} 2023-09-22T21:59:08.773 DEBUG:boto:Token: None 2023-09-22T21:59:08.773 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/ 2023-09-22T21:59:08.773 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8= 2023-09-22T21:59:08.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8=', 'Content-Length': '0'} 2023-09-22T21:59:08.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000444ba54cf3757be8-00650e0e2c-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.784 DEBUG:boto:b'ffzdqu-61obj1481000falseobj1482023-09-22T21:56:10.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:08.785 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:08.785 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:08.785 DEBUG:boto:path=/ffzdqu-61/?prefix=obj148 2023-09-22T21:59:08.785 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj148 2023-09-22T21:59:08.785 DEBUG:boto:Method: GET 2023-09-22T21:59:08.786 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj148 2023-09-22T21:59:08.786 DEBUG:boto:Data: 2023-09-22T21:59:08.786 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.786 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.786 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.786 DEBUG:boto:Params: {} 2023-09-22T21:59:08.786 DEBUG:boto:Token: None 2023-09-22T21:59:08.787 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/ 2023-09-22T21:59:08.787 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8= 2023-09-22T21:59:08.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8=', 'Content-Length': '0'} 2023-09-22T21:59:08.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000886c6cedeb179e46-00650e0e2c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.838 DEBUG:boto:b'ffzdqu-61obj1481000falseobj1482023-09-22T21:56:10.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:08.839 DEBUG:boto:path=/ffzdqu-61/obj148 2023-09-22T21:59:08.839 DEBUG:boto:auth_path=/ffzdqu-61/obj148 2023-09-22T21:59:08.839 DEBUG:boto:Method: HEAD 2023-09-22T21:59:08.839 DEBUG:boto:Path: /ffzdqu-61/obj148 2023-09-22T21:59:08.839 DEBUG:boto:Data: 2023-09-22T21:59:08.839 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.839 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.840 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.840 DEBUG:boto:Params: {} 2023-09-22T21:59:08.840 DEBUG:boto:Token: None 2023-09-22T21:59:08.840 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj148 2023-09-22T21:59:08.840 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZCecUl6uaZ48n42AnfMhUdbGbIQ= 2023-09-22T21:59:08.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZCecUl6uaZ48n42AnfMhUdbGbIQ=', 'Content-Length': '0'} 2023-09-22T21:59:08.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e8d3fcafc7baf71-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.843 DEBUG:boto:path=/ffzdqu-61/obj148 2023-09-22T21:59:08.843 DEBUG:boto:auth_path=/ffzdqu-61/obj148 2023-09-22T21:59:08.844 DEBUG:boto:Method: HEAD 2023-09-22T21:59:08.844 DEBUG:boto:Path: /ffzdqu-61/obj148 2023-09-22T21:59:08.844 DEBUG:boto:Data: 2023-09-22T21:59:08.844 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.844 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.844 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.844 DEBUG:boto:Params: {} 2023-09-22T21:59:08.844 DEBUG:boto:Token: None 2023-09-22T21:59:08.844 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj148 2023-09-22T21:59:08.845 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZCecUl6uaZ48n42AnfMhUdbGbIQ= 2023-09-22T21:59:08.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZCecUl6uaZ48n42AnfMhUdbGbIQ=', 'Content-Length': '0'} 2023-09-22T21:59:08.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005637dd40825f552d-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-09-22T21:59:08.848 DEBUG:boto:path=/ffzdqu-61/obj148 2023-09-22T21:59:08.848 DEBUG:boto:auth_path=/ffzdqu-61/obj148 2023-09-22T21:59:08.848 DEBUG:boto:Method: GET 2023-09-22T21:59:08.848 DEBUG:boto:Path: /ffzdqu-61/obj148 2023-09-22T21:59:08.848 DEBUG:boto:Data: 2023-09-22T21:59:08.848 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.848 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.848 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.848 DEBUG:boto:Params: {} 2023-09-22T21:59:08.849 DEBUG:boto:Token: None 2023-09-22T21:59:08.849 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj148 2023-09-22T21:59:08.849 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PG8PWfSkpeEfXrOV8vtGUQm5s9o= 2023-09-22T21:59:08.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PG8PWfSkpeEfXrOV8vtGUQm5s9o=', 'Content-Length': '0'} 2023-09-22T21:59:08.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd13f7809f70018c-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.894 DEBUG:boto:path=/ffzdqu-61/obj148 2023-09-22T21:59:08.894 DEBUG:boto:auth_path=/ffzdqu-61/obj148 2023-09-22T21:59:08.894 DEBUG:boto:Method: GET 2023-09-22T21:59:08.895 DEBUG:boto:Path: /ffzdqu-61/obj148 2023-09-22T21:59:08.895 DEBUG:boto:Data: 2023-09-22T21:59:08.895 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.895 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.895 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.895 DEBUG:boto:Params: {} 2023-09-22T21:59:08.895 DEBUG:boto:Token: None 2023-09-22T21:59:08.895 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj148 2023-09-22T21:59:08.896 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PG8PWfSkpeEfXrOV8vtGUQm5s9o= 2023-09-22T21:59:08.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PG8PWfSkpeEfXrOV8vtGUQm5s9o=', 'Content-Length': '0'} 2023-09-22T21:59:08.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b04cede2541c04f-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.899 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:08.899 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:08.899 DEBUG:boto:path=/ffzdqu-61/?prefix=obj148 2023-09-22T21:59:08.899 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj148 2023-09-22T21:59:08.899 DEBUG:boto:Method: GET 2023-09-22T21:59:08.899 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj148 2023-09-22T21:59:08.899 DEBUG:boto:Data: 2023-09-22T21:59:08.899 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.899 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.900 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.900 DEBUG:boto:Params: {} 2023-09-22T21:59:08.900 DEBUG:boto:Token: None 2023-09-22T21:59:08.900 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/ 2023-09-22T21:59:08.900 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8= 2023-09-22T21:59:08.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8=', 'Content-Length': '0'} 2023-09-22T21:59:08.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026b749b9fbb628c6-00650e0e2c-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.905 DEBUG:boto:b'ffzdqu-61obj1481000falseobj1482023-09-22T21:56:10.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:08.906 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:08.906 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:08.906 DEBUG:boto:path=/ffzdqu-61/?prefix=obj148 2023-09-22T21:59:08.906 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj148 2023-09-22T21:59:08.906 DEBUG:boto:Method: GET 2023-09-22T21:59:08.906 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj148 2023-09-22T21:59:08.906 DEBUG:boto:Data: 2023-09-22T21:59:08.906 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.906 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.907 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.907 DEBUG:boto:Params: {} 2023-09-22T21:59:08.907 DEBUG:boto:Token: None 2023-09-22T21:59:08.907 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/ 2023-09-22T21:59:08.907 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8= 2023-09-22T21:59:08.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DnnpyMMACeXPhA7iYUI7+Y7nhV8=', 'Content-Length': '0'} 2023-09-22T21:59:08.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f555165775242b1-00650e0e2c-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.917 DEBUG:boto:b'ffzdqu-61obj1481000falseobj1482023-09-22T21:56:10.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:08.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-22T21:59:08.918 DEBUG:boto:path=/ffzdqu-61/obj155 2023-09-22T21:59:08.918 DEBUG:boto:auth_path=/ffzdqu-61/obj155 2023-09-22T21:59:08.918 DEBUG:boto:path=/ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:08.918 DEBUG:boto:auth_path=/ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:08.918 DEBUG:boto:Method: GET 2023-09-22T21:59:08.918 DEBUG:boto:Path: /ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:08.918 DEBUG:boto:Data: 2023-09-22T21:59:08.918 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.918 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.919 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.919 DEBUG:boto:Params: {} 2023-09-22T21:59:08.919 DEBUG:boto:Token: None 2023-09-22T21:59:08.919 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:08.919 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tK+zP0y4Zt1nX1y1LECkSWEKd+s= 2023-09-22T21:59:08.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tK+zP0y4Zt1nX1y1LECkSWEKd+s=', 'Content-Length': '0'} 2023-09-22T21:59:08.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092fcebd069ca3e15-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.922 DEBUG:boto:path=/ffzdqu-61/obj155 2023-09-22T21:59:08.923 DEBUG:boto:auth_path=/ffzdqu-61/obj155 2023-09-22T21:59:08.923 DEBUG:boto:path=/ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:08.923 DEBUG:boto:auth_path=/ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:08.923 DEBUG:boto:Method: GET 2023-09-22T21:59:08.923 DEBUG:boto:Path: /ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:08.923 DEBUG:boto:Data: 2023-09-22T21:59:08.923 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.923 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.924 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.924 DEBUG:boto:Params: {} 2023-09-22T21:59:08.924 DEBUG:boto:Token: None 2023-09-22T21:59:08.924 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:08.924 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tK+zP0y4Zt1nX1y1LECkSWEKd+s= 2023-09-22T21:59:08.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tK+zP0y4Zt1nX1y1LECkSWEKd+s=', 'Content-Length': '0'} 2023-09-22T21:59:08.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b03700f21564dde4-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.970 DEBUG:boto:path=/ffzdqu-61/obj155 2023-09-22T21:59:08.970 DEBUG:boto:auth_path=/ffzdqu-61/obj155 2023-09-22T21:59:08.970 DEBUG:boto:path=/ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:08.971 DEBUG:boto:auth_path=/ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:08.971 DEBUG:boto:Method: HEAD 2023-09-22T21:59:08.971 DEBUG:boto:Path: /ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:08.971 DEBUG:boto:Data: 2023-09-22T21:59:08.971 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.971 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.971 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.971 DEBUG:boto:Params: {} 2023-09-22T21:59:08.972 DEBUG:boto:Token: None 2023-09-22T21:59:08.972 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:08.972 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UebOUNqv02mkWOQ5E7OdIvoEhl8= 2023-09-22T21:59:08.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UebOUNqv02mkWOQ5E7OdIvoEhl8=', 'Content-Length': '0'} 2023-09-22T21:59:08.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cffb8333d2daeedb-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.975 DEBUG:boto:path=/ffzdqu-61/obj155 2023-09-22T21:59:08.975 DEBUG:boto:auth_path=/ffzdqu-61/obj155 2023-09-22T21:59:08.975 DEBUG:boto:path=/ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:08.975 DEBUG:boto:auth_path=/ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:08.976 DEBUG:boto:Method: HEAD 2023-09-22T21:59:08.976 DEBUG:boto:Path: /ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:08.976 DEBUG:boto:Data: 2023-09-22T21:59:08.976 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.976 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:08.976 DEBUG:boto:Port: 8000 2023-09-22T21:59:08.976 DEBUG:boto:Params: {} 2023-09-22T21:59:08.976 DEBUG:boto:Token: None 2023-09-22T21:59:08.976 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:08.977 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UebOUNqv02mkWOQ5E7OdIvoEhl8= 2023-09-22T21:59:08.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UebOUNqv02mkWOQ5E7OdIvoEhl8=', 'Content-Length': '0'} 2023-09-22T21:59:08.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006486a612f5e969d9-00650e0e2c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:08.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-22T21:59:08.980 DEBUG:boto:path=/ffzdqu-61/obj155 2023-09-22T21:59:08.980 DEBUG:boto:auth_path=/ffzdqu-61/obj155 2023-09-22T21:59:08.980 DEBUG:boto:path=/ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:08.980 DEBUG:boto:auth_path=/ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:08.980 DEBUG:boto:Method: GET 2023-09-22T21:59:08.980 DEBUG:boto:Path: /ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:08.980 DEBUG:boto:Data: 2023-09-22T21:59:08.980 DEBUG:boto:Headers: {} 2023-09-22T21:59:08.980 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:08.981 DEBUG:boto:Port: 8001 2023-09-22T21:59:08.981 DEBUG:boto:Params: {} 2023-09-22T21:59:08.981 DEBUG:boto:Token: None 2023-09-22T21:59:08.981 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:08 GMT /ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:08.981 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tK+zP0y4Zt1nX1y1LECkSWEKd+s= 2023-09-22T21:59:08.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tK+zP0y4Zt1nX1y1LECkSWEKd+s=', 'Content-Length': '0'} 2023-09-22T21:59:08.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005df37a583ffe0e0c-00650e0e2c-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.026 DEBUG:boto:path=/ffzdqu-61/obj155 2023-09-22T21:59:09.026 DEBUG:boto:auth_path=/ffzdqu-61/obj155 2023-09-22T21:59:09.026 DEBUG:boto:path=/ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:09.026 DEBUG:boto:auth_path=/ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:09.027 DEBUG:boto:Method: GET 2023-09-22T21:59:09.027 DEBUG:boto:Path: /ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:09.027 DEBUG:boto:Data: 2023-09-22T21:59:09.027 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.027 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.027 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.027 DEBUG:boto:Params: {} 2023-09-22T21:59:09.027 DEBUG:boto:Token: None 2023-09-22T21:59:09.028 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj155?versionId=null 2023-09-22T21:59:09.028 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m8LxWyfaTuOhJ+4+fCqCv0aVKPg= 2023-09-22T21:59:09.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m8LxWyfaTuOhJ+4+fCqCv0aVKPg=', 'Content-Length': '0'} 2023-09-22T21:59:09.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad29afb9e2fbb893-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.031 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:09.031 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:09.031 DEBUG:boto:path=/ffzdqu-61/?prefix=obj155 2023-09-22T21:59:09.031 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj155 2023-09-22T21:59:09.031 DEBUG:boto:Method: GET 2023-09-22T21:59:09.031 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj155 2023-09-22T21:59:09.031 DEBUG:boto:Data: 2023-09-22T21:59:09.032 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.032 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.032 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.032 DEBUG:boto:Params: {} 2023-09-22T21:59:09.032 DEBUG:boto:Token: None 2023-09-22T21:59:09.032 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/ 2023-09-22T21:59:09.032 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ= 2023-09-22T21:59:09.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ=', 'Content-Length': '0'} 2023-09-22T21:59:09.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f01a5688e1e8e705-00650e0e2d-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.037 DEBUG:boto:b'ffzdqu-61obj1551000falseobj1552023-09-22T21:56:10.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:09.038 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:09.038 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:09.038 DEBUG:boto:path=/ffzdqu-61/?prefix=obj155 2023-09-22T21:59:09.038 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj155 2023-09-22T21:59:09.038 DEBUG:boto:Method: GET 2023-09-22T21:59:09.038 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj155 2023-09-22T21:59:09.038 DEBUG:boto:Data: 2023-09-22T21:59:09.039 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.039 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.039 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.039 DEBUG:boto:Params: {} 2023-09-22T21:59:09.039 DEBUG:boto:Token: None 2023-09-22T21:59:09.039 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/ 2023-09-22T21:59:09.039 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ= 2023-09-22T21:59:09.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ=', 'Content-Length': '0'} 2023-09-22T21:59:09.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080ac5321605c3a10-00650e0e2d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.049 DEBUG:boto:b'ffzdqu-61obj1551000falseobj1552023-09-22T21:56:10.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:09.049 DEBUG:boto:path=/ffzdqu-61/obj155 2023-09-22T21:59:09.050 DEBUG:boto:auth_path=/ffzdqu-61/obj155 2023-09-22T21:59:09.050 DEBUG:boto:Method: HEAD 2023-09-22T21:59:09.050 DEBUG:boto:Path: /ffzdqu-61/obj155 2023-09-22T21:59:09.050 DEBUG:boto:Data: 2023-09-22T21:59:09.050 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.050 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.050 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.050 DEBUG:boto:Params: {} 2023-09-22T21:59:09.051 DEBUG:boto:Token: None 2023-09-22T21:59:09.051 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj155 2023-09-22T21:59:09.051 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4CLriK3L/Vuef44GTT6m6Bf+ZkU= 2023-09-22T21:59:09.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4CLriK3L/Vuef44GTT6m6Bf+ZkU=', 'Content-Length': '0'} 2023-09-22T21:59:09.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038d9e54d6f53ae83-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.054 DEBUG:boto:path=/ffzdqu-61/obj155 2023-09-22T21:59:09.054 DEBUG:boto:auth_path=/ffzdqu-61/obj155 2023-09-22T21:59:09.054 DEBUG:boto:Method: HEAD 2023-09-22T21:59:09.054 DEBUG:boto:Path: /ffzdqu-61/obj155 2023-09-22T21:59:09.055 DEBUG:boto:Data: 2023-09-22T21:59:09.055 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.055 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.055 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.055 DEBUG:boto:Params: {} 2023-09-22T21:59:09.055 DEBUG:boto:Token: None 2023-09-22T21:59:09.055 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj155 2023-09-22T21:59:09.055 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4CLriK3L/Vuef44GTT6m6Bf+ZkU= 2023-09-22T21:59:09.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4CLriK3L/Vuef44GTT6m6Bf+ZkU=', 'Content-Length': '0'} 2023-09-22T21:59:09.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c169c95d8aa26468-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-22T21:59:09.059 DEBUG:boto:path=/ffzdqu-61/obj155 2023-09-22T21:59:09.059 DEBUG:boto:auth_path=/ffzdqu-61/obj155 2023-09-22T21:59:09.059 DEBUG:boto:Method: GET 2023-09-22T21:59:09.059 DEBUG:boto:Path: /ffzdqu-61/obj155 2023-09-22T21:59:09.059 DEBUG:boto:Data: 2023-09-22T21:59:09.059 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.059 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.060 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.060 DEBUG:boto:Params: {} 2023-09-22T21:59:09.060 DEBUG:boto:Token: None 2023-09-22T21:59:09.060 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj155 2023-09-22T21:59:09.060 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2AHfo0zp4gTLAEvYutk0HKc/73g= 2023-09-22T21:59:09.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2AHfo0zp4gTLAEvYutk0HKc/73g=', 'Content-Length': '0'} 2023-09-22T21:59:09.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e4c4cfded74ee35-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.063 DEBUG:boto:path=/ffzdqu-61/obj155 2023-09-22T21:59:09.063 DEBUG:boto:auth_path=/ffzdqu-61/obj155 2023-09-22T21:59:09.063 DEBUG:boto:Method: GET 2023-09-22T21:59:09.063 DEBUG:boto:Path: /ffzdqu-61/obj155 2023-09-22T21:59:09.063 DEBUG:boto:Data: 2023-09-22T21:59:09.064 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.064 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.064 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.064 DEBUG:boto:Params: {} 2023-09-22T21:59:09.064 DEBUG:boto:Token: None 2023-09-22T21:59:09.064 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj155 2023-09-22T21:59:09.065 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2AHfo0zp4gTLAEvYutk0HKc/73g= 2023-09-22T21:59:09.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2AHfo0zp4gTLAEvYutk0HKc/73g=', 'Content-Length': '0'} 2023-09-22T21:59:09.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c185297994ba809c-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.110 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:09.110 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:09.110 DEBUG:boto:path=/ffzdqu-61/?prefix=obj155 2023-09-22T21:59:09.111 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj155 2023-09-22T21:59:09.111 DEBUG:boto:Method: GET 2023-09-22T21:59:09.111 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj155 2023-09-22T21:59:09.111 DEBUG:boto:Data: 2023-09-22T21:59:09.111 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.111 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.111 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.111 DEBUG:boto:Params: {} 2023-09-22T21:59:09.112 DEBUG:boto:Token: None 2023-09-22T21:59:09.112 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/ 2023-09-22T21:59:09.112 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ= 2023-09-22T21:59:09.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ=', 'Content-Length': '0'} 2023-09-22T21:59:09.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daa6aafbfdafd19c-00650e0e2d-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.117 DEBUG:boto:b'ffzdqu-61obj1551000falseobj1552023-09-22T21:56:10.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:09.118 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:09.118 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:09.118 DEBUG:boto:path=/ffzdqu-61/?prefix=obj155 2023-09-22T21:59:09.118 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj155 2023-09-22T21:59:09.118 DEBUG:boto:Method: GET 2023-09-22T21:59:09.119 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj155 2023-09-22T21:59:09.119 DEBUG:boto:Data: 2023-09-22T21:59:09.119 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.119 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.119 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.119 DEBUG:boto:Params: {} 2023-09-22T21:59:09.119 DEBUG:boto:Token: None 2023-09-22T21:59:09.119 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/ 2023-09-22T21:59:09.119 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ= 2023-09-22T21:59:09.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ=', 'Content-Length': '0'} 2023-09-22T21:59:09.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000174437f12258d82d-00650e0e2d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.129 DEBUG:boto:b'ffzdqu-61obj1551000falseobj1552023-09-22T21:56:10.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:09.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:59:09.129 DEBUG:boto:path=/ffzdqu-61/obj17 2023-09-22T21:59:09.129 DEBUG:boto:auth_path=/ffzdqu-61/obj17 2023-09-22T21:59:09.129 DEBUG:boto:path=/ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.129 DEBUG:boto:auth_path=/ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.130 DEBUG:boto:Method: GET 2023-09-22T21:59:09.130 DEBUG:boto:Path: /ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.130 DEBUG:boto:Data: 2023-09-22T21:59:09.130 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.130 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.130 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.130 DEBUG:boto:Params: {} 2023-09-22T21:59:09.130 DEBUG:boto:Token: None 2023-09-22T21:59:09.131 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.131 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CnUHqOvLyQorOtv8tVDf73SdU88= 2023-09-22T21:59:09.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CnUHqOvLyQorOtv8tVDf73SdU88=', 'Content-Length': '0'} 2023-09-22T21:59:09.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0945a57f520ee58-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.174 DEBUG:boto:path=/ffzdqu-61/obj17 2023-09-22T21:59:09.174 DEBUG:boto:auth_path=/ffzdqu-61/obj17 2023-09-22T21:59:09.174 DEBUG:boto:path=/ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.175 DEBUG:boto:auth_path=/ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.175 DEBUG:boto:Method: GET 2023-09-22T21:59:09.175 DEBUG:boto:Path: /ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.175 DEBUG:boto:Data: 2023-09-22T21:59:09.175 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.175 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.175 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.175 DEBUG:boto:Params: {} 2023-09-22T21:59:09.176 DEBUG:boto:Token: None 2023-09-22T21:59:09.176 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.176 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CnUHqOvLyQorOtv8tVDf73SdU88= 2023-09-22T21:59:09.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CnUHqOvLyQorOtv8tVDf73SdU88=', 'Content-Length': '0'} 2023-09-22T21:59:09.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff1a45182ef75a63-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.179 DEBUG:boto:path=/ffzdqu-61/obj17 2023-09-22T21:59:09.180 DEBUG:boto:auth_path=/ffzdqu-61/obj17 2023-09-22T21:59:09.180 DEBUG:boto:path=/ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.180 DEBUG:boto:auth_path=/ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.180 DEBUG:boto:Method: HEAD 2023-09-22T21:59:09.180 DEBUG:boto:Path: /ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.180 DEBUG:boto:Data: 2023-09-22T21:59:09.180 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.180 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.180 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.181 DEBUG:boto:Params: {} 2023-09-22T21:59:09.181 DEBUG:boto:Token: None 2023-09-22T21:59:09.181 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.181 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D5Ps7CAOnNuvFmGSGFzatMhnq30= 2023-09-22T21:59:09.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D5Ps7CAOnNuvFmGSGFzatMhnq30=', 'Content-Length': '0'} 2023-09-22T21:59:09.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002a23e37c8bca6e4-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.184 DEBUG:boto:path=/ffzdqu-61/obj17 2023-09-22T21:59:09.184 DEBUG:boto:auth_path=/ffzdqu-61/obj17 2023-09-22T21:59:09.184 DEBUG:boto:path=/ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.185 DEBUG:boto:auth_path=/ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.185 DEBUG:boto:Method: HEAD 2023-09-22T21:59:09.185 DEBUG:boto:Path: /ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.185 DEBUG:boto:Data: 2023-09-22T21:59:09.185 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.185 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.185 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.185 DEBUG:boto:Params: {} 2023-09-22T21:59:09.185 DEBUG:boto:Token: None 2023-09-22T21:59:09.186 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.186 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D5Ps7CAOnNuvFmGSGFzatMhnq30= 2023-09-22T21:59:09.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D5Ps7CAOnNuvFmGSGFzatMhnq30=', 'Content-Length': '0'} 2023-09-22T21:59:09.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e75c8fbf3e55d53-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:59:09.189 DEBUG:boto:path=/ffzdqu-61/obj17 2023-09-22T21:59:09.189 DEBUG:boto:auth_path=/ffzdqu-61/obj17 2023-09-22T21:59:09.189 DEBUG:boto:path=/ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.189 DEBUG:boto:auth_path=/ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.189 DEBUG:boto:Method: GET 2023-09-22T21:59:09.189 DEBUG:boto:Path: /ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.190 DEBUG:boto:Data: 2023-09-22T21:59:09.190 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.190 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.190 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.190 DEBUG:boto:Params: {} 2023-09-22T21:59:09.190 DEBUG:boto:Token: None 2023-09-22T21:59:09.190 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.191 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CnUHqOvLyQorOtv8tVDf73SdU88= 2023-09-22T21:59:09.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CnUHqOvLyQorOtv8tVDf73SdU88=', 'Content-Length': '0'} 2023-09-22T21:59:09.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000acfe64c02f1c2b5-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.194 DEBUG:boto:path=/ffzdqu-61/obj17 2023-09-22T21:59:09.194 DEBUG:boto:auth_path=/ffzdqu-61/obj17 2023-09-22T21:59:09.194 DEBUG:boto:path=/ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.194 DEBUG:boto:auth_path=/ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.194 DEBUG:boto:Method: GET 2023-09-22T21:59:09.194 DEBUG:boto:Path: /ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.194 DEBUG:boto:Data: 2023-09-22T21:59:09.194 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.195 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.195 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.195 DEBUG:boto:Params: {} 2023-09-22T21:59:09.195 DEBUG:boto:Token: None 2023-09-22T21:59:09.195 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj17?versionId=null 2023-09-22T21:59:09.195 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CnUHqOvLyQorOtv8tVDf73SdU88= 2023-09-22T21:59:09.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CnUHqOvLyQorOtv8tVDf73SdU88=', 'Content-Length': '0'} 2023-09-22T21:59:09.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5596a1e6d29b3a5-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.199 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:09.199 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:09.199 DEBUG:boto:path=/ffzdqu-61/?prefix=obj17 2023-09-22T21:59:09.199 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj17 2023-09-22T21:59:09.199 DEBUG:boto:Method: GET 2023-09-22T21:59:09.199 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj17 2023-09-22T21:59:09.199 DEBUG:boto:Data: 2023-09-22T21:59:09.199 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.199 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.200 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.200 DEBUG:boto:Params: {} 2023-09-22T21:59:09.200 DEBUG:boto:Token: None 2023-09-22T21:59:09.200 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/ 2023-09-22T21:59:09.200 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ= 2023-09-22T21:59:09.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ=', 'Content-Length': '0'} 2023-09-22T21:59:09.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081cd0edef3e4787f-00650e0e2d-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.205 DEBUG:boto:b'ffzdqu-61obj171000falseobj172023-09-22T21:56:08.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:09.206 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:09.206 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:09.206 DEBUG:boto:path=/ffzdqu-61/?prefix=obj17 2023-09-22T21:59:09.206 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj17 2023-09-22T21:59:09.206 DEBUG:boto:Method: GET 2023-09-22T21:59:09.206 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj17 2023-09-22T21:59:09.207 DEBUG:boto:Data: 2023-09-22T21:59:09.207 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.207 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.207 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.207 DEBUG:boto:Params: {} 2023-09-22T21:59:09.207 DEBUG:boto:Token: None 2023-09-22T21:59:09.207 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/ 2023-09-22T21:59:09.207 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ= 2023-09-22T21:59:09.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ=', 'Content-Length': '0'} 2023-09-22T21:59:09.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac84b77f4845c5db-00650e0e2d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.258 DEBUG:boto:b'ffzdqu-61obj171000falseobj172023-09-22T21:56:08.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:09.259 DEBUG:boto:path=/ffzdqu-61/obj17 2023-09-22T21:59:09.259 DEBUG:boto:auth_path=/ffzdqu-61/obj17 2023-09-22T21:59:09.259 DEBUG:boto:Method: HEAD 2023-09-22T21:59:09.259 DEBUG:boto:Path: /ffzdqu-61/obj17 2023-09-22T21:59:09.259 DEBUG:boto:Data: 2023-09-22T21:59:09.259 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.259 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.259 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.260 DEBUG:boto:Params: {} 2023-09-22T21:59:09.260 DEBUG:boto:Token: None 2023-09-22T21:59:09.260 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj17 2023-09-22T21:59:09.260 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pp9O/nk3AzROVf34MNm5VPTpMuk= 2023-09-22T21:59:09.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pp9O/nk3AzROVf34MNm5VPTpMuk=', 'Content-Length': '0'} 2023-09-22T21:59:09.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd4f8bca6d8ade00-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.263 DEBUG:boto:path=/ffzdqu-61/obj17 2023-09-22T21:59:09.263 DEBUG:boto:auth_path=/ffzdqu-61/obj17 2023-09-22T21:59:09.263 DEBUG:boto:Method: HEAD 2023-09-22T21:59:09.264 DEBUG:boto:Path: /ffzdqu-61/obj17 2023-09-22T21:59:09.264 DEBUG:boto:Data: 2023-09-22T21:59:09.264 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.264 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.264 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.264 DEBUG:boto:Params: {} 2023-09-22T21:59:09.264 DEBUG:boto:Token: None 2023-09-22T21:59:09.264 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj17 2023-09-22T21:59:09.264 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pp9O/nk3AzROVf34MNm5VPTpMuk= 2023-09-22T21:59:09.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pp9O/nk3AzROVf34MNm5VPTpMuk=', 'Content-Length': '0'} 2023-09-22T21:59:09.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006394063a87fb6acb-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T21:59:09.268 DEBUG:boto:path=/ffzdqu-61/obj17 2023-09-22T21:59:09.268 DEBUG:boto:auth_path=/ffzdqu-61/obj17 2023-09-22T21:59:09.268 DEBUG:boto:Method: GET 2023-09-22T21:59:09.268 DEBUG:boto:Path: /ffzdqu-61/obj17 2023-09-22T21:59:09.268 DEBUG:boto:Data: 2023-09-22T21:59:09.268 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.268 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.268 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.269 DEBUG:boto:Params: {} 2023-09-22T21:59:09.269 DEBUG:boto:Token: None 2023-09-22T21:59:09.269 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj17 2023-09-22T21:59:09.269 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gXJh09TqQh9IQzRghdCOPrrXsy0= 2023-09-22T21:59:09.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gXJh09TqQh9IQzRghdCOPrrXsy0=', 'Content-Length': '0'} 2023-09-22T21:59:09.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d21ba782e2befdd-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.314 DEBUG:boto:path=/ffzdqu-61/obj17 2023-09-22T21:59:09.314 DEBUG:boto:auth_path=/ffzdqu-61/obj17 2023-09-22T21:59:09.315 DEBUG:boto:Method: GET 2023-09-22T21:59:09.315 DEBUG:boto:Path: /ffzdqu-61/obj17 2023-09-22T21:59:09.315 DEBUG:boto:Data: 2023-09-22T21:59:09.315 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.315 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.315 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.315 DEBUG:boto:Params: {} 2023-09-22T21:59:09.315 DEBUG:boto:Token: None 2023-09-22T21:59:09.315 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj17 2023-09-22T21:59:09.316 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gXJh09TqQh9IQzRghdCOPrrXsy0= 2023-09-22T21:59:09.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gXJh09TqQh9IQzRghdCOPrrXsy0=', 'Content-Length': '0'} 2023-09-22T21:59:09.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9328fc7c8346780-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.319 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:09.319 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:09.319 DEBUG:boto:path=/ffzdqu-61/?prefix=obj17 2023-09-22T21:59:09.319 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj17 2023-09-22T21:59:09.319 DEBUG:boto:Method: GET 2023-09-22T21:59:09.319 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj17 2023-09-22T21:59:09.319 DEBUG:boto:Data: 2023-09-22T21:59:09.319 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.320 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.320 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.320 DEBUG:boto:Params: {} 2023-09-22T21:59:09.320 DEBUG:boto:Token: None 2023-09-22T21:59:09.320 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/ 2023-09-22T21:59:09.320 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ= 2023-09-22T21:59:09.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ=', 'Content-Length': '0'} 2023-09-22T21:59:09.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a36045b37d8717c-00650e0e2d-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.325 DEBUG:boto:b'ffzdqu-61obj171000falseobj172023-09-22T21:56:08.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:09.325 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:09.325 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:09.326 DEBUG:boto:path=/ffzdqu-61/?prefix=obj17 2023-09-22T21:59:09.326 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj17 2023-09-22T21:59:09.326 DEBUG:boto:Method: GET 2023-09-22T21:59:09.326 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj17 2023-09-22T21:59:09.326 DEBUG:boto:Data: 2023-09-22T21:59:09.326 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.326 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.327 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.327 DEBUG:boto:Params: {} 2023-09-22T21:59:09.327 DEBUG:boto:Token: None 2023-09-22T21:59:09.327 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/ 2023-09-22T21:59:09.327 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ= 2023-09-22T21:59:09.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ=', 'Content-Length': '0'} 2023-09-22T21:59:09.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063b40469704cafdc-00650e0e2d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.336 DEBUG:boto:b'ffzdqu-61obj171000falseobj172023-09-22T21:56:08.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:09.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-09-22T21:59:09.337 DEBUG:boto:path=/ffzdqu-61/obj185 2023-09-22T21:59:09.337 DEBUG:boto:auth_path=/ffzdqu-61/obj185 2023-09-22T21:59:09.337 DEBUG:boto:path=/ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.337 DEBUG:boto:auth_path=/ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.338 DEBUG:boto:Method: GET 2023-09-22T21:59:09.338 DEBUG:boto:Path: /ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.338 DEBUG:boto:Data: 2023-09-22T21:59:09.338 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.338 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.338 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.338 DEBUG:boto:Params: {} 2023-09-22T21:59:09.338 DEBUG:boto:Token: None 2023-09-22T21:59:09.338 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.339 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZYr4JSEE+csgD6e0xxjQeNjoRO4= 2023-09-22T21:59:09.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZYr4JSEE+csgD6e0xxjQeNjoRO4=', 'Content-Length': '0'} 2023-09-22T21:59:09.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9598805847f811c-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.342 DEBUG:boto:path=/ffzdqu-61/obj185 2023-09-22T21:59:09.342 DEBUG:boto:auth_path=/ffzdqu-61/obj185 2023-09-22T21:59:09.342 DEBUG:boto:path=/ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.342 DEBUG:boto:auth_path=/ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.342 DEBUG:boto:Method: GET 2023-09-22T21:59:09.343 DEBUG:boto:Path: /ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.343 DEBUG:boto:Data: 2023-09-22T21:59:09.343 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.343 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.343 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.343 DEBUG:boto:Params: {} 2023-09-22T21:59:09.343 DEBUG:boto:Token: None 2023-09-22T21:59:09.343 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.343 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZYr4JSEE+csgD6e0xxjQeNjoRO4= 2023-09-22T21:59:09.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZYr4JSEE+csgD6e0xxjQeNjoRO4=', 'Content-Length': '0'} 2023-09-22T21:59:09.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000717644add156a0ec-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.386 DEBUG:boto:path=/ffzdqu-61/obj185 2023-09-22T21:59:09.386 DEBUG:boto:auth_path=/ffzdqu-61/obj185 2023-09-22T21:59:09.387 DEBUG:boto:path=/ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.387 DEBUG:boto:auth_path=/ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.387 DEBUG:boto:Method: HEAD 2023-09-22T21:59:09.387 DEBUG:boto:Path: /ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.387 DEBUG:boto:Data: 2023-09-22T21:59:09.387 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.387 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.387 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.387 DEBUG:boto:Params: {} 2023-09-22T21:59:09.388 DEBUG:boto:Token: None 2023-09-22T21:59:09.388 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.388 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rTdDXjdqspX3grqjRa7q1iOUcmU= 2023-09-22T21:59:09.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rTdDXjdqspX3grqjRa7q1iOUcmU=', 'Content-Length': '0'} 2023-09-22T21:59:09.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075c5f0dbd579f4a4-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.391 DEBUG:boto:path=/ffzdqu-61/obj185 2023-09-22T21:59:09.391 DEBUG:boto:auth_path=/ffzdqu-61/obj185 2023-09-22T21:59:09.391 DEBUG:boto:path=/ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.391 DEBUG:boto:auth_path=/ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.391 DEBUG:boto:Method: HEAD 2023-09-22T21:59:09.392 DEBUG:boto:Path: /ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.392 DEBUG:boto:Data: 2023-09-22T21:59:09.392 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.392 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.392 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.392 DEBUG:boto:Params: {} 2023-09-22T21:59:09.392 DEBUG:boto:Token: None 2023-09-22T21:59:09.392 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.392 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rTdDXjdqspX3grqjRa7q1iOUcmU= 2023-09-22T21:59:09.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rTdDXjdqspX3grqjRa7q1iOUcmU=', 'Content-Length': '0'} 2023-09-22T21:59:09.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c82cda08d08e060-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-09-22T21:59:09.396 DEBUG:boto:path=/ffzdqu-61/obj185 2023-09-22T21:59:09.396 DEBUG:boto:auth_path=/ffzdqu-61/obj185 2023-09-22T21:59:09.396 DEBUG:boto:path=/ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.396 DEBUG:boto:auth_path=/ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.396 DEBUG:boto:Method: GET 2023-09-22T21:59:09.396 DEBUG:boto:Path: /ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.396 DEBUG:boto:Data: 2023-09-22T21:59:09.396 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.397 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.397 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.397 DEBUG:boto:Params: {} 2023-09-22T21:59:09.397 DEBUG:boto:Token: None 2023-09-22T21:59:09.397 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.397 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZYr4JSEE+csgD6e0xxjQeNjoRO4= 2023-09-22T21:59:09.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZYr4JSEE+csgD6e0xxjQeNjoRO4=', 'Content-Length': '0'} 2023-09-22T21:59:09.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a57a21bb549ef3c-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.442 DEBUG:boto:path=/ffzdqu-61/obj185 2023-09-22T21:59:09.443 DEBUG:boto:auth_path=/ffzdqu-61/obj185 2023-09-22T21:59:09.443 DEBUG:boto:path=/ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.443 DEBUG:boto:auth_path=/ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.443 DEBUG:boto:Method: GET 2023-09-22T21:59:09.443 DEBUG:boto:Path: /ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.443 DEBUG:boto:Data: 2023-09-22T21:59:09.443 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.443 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.444 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.444 DEBUG:boto:Params: {} 2023-09-22T21:59:09.444 DEBUG:boto:Token: None 2023-09-22T21:59:09.444 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj185?versionId=null 2023-09-22T21:59:09.444 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZYr4JSEE+csgD6e0xxjQeNjoRO4= 2023-09-22T21:59:09.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZYr4JSEE+csgD6e0xxjQeNjoRO4=', 'Content-Length': '0'} 2023-09-22T21:59:09.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092fd858d38a265b8-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.448 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:09.448 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:09.448 DEBUG:boto:path=/ffzdqu-61/?prefix=obj185 2023-09-22T21:59:09.448 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj185 2023-09-22T21:59:09.448 DEBUG:boto:Method: GET 2023-09-22T21:59:09.448 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj185 2023-09-22T21:59:09.449 DEBUG:boto:Data: 2023-09-22T21:59:09.449 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.449 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.449 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.449 DEBUG:boto:Params: {} 2023-09-22T21:59:09.449 DEBUG:boto:Token: None 2023-09-22T21:59:09.449 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/ 2023-09-22T21:59:09.449 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ= 2023-09-22T21:59:09.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ=', 'Content-Length': '0'} 2023-09-22T21:59:09.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bb11788003cb98c-00650e0e2d-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.454 DEBUG:boto:b'ffzdqu-61obj1851000falseobj1852023-09-22T21:56:10.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:09.455 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:09.455 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:09.455 DEBUG:boto:path=/ffzdqu-61/?prefix=obj185 2023-09-22T21:59:09.455 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj185 2023-09-22T21:59:09.455 DEBUG:boto:Method: GET 2023-09-22T21:59:09.455 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj185 2023-09-22T21:59:09.456 DEBUG:boto:Data: 2023-09-22T21:59:09.456 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.456 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.456 DEBUG:boto:Params: {} 2023-09-22T21:59:09.456 DEBUG:boto:Token: None 2023-09-22T21:59:09.456 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/ 2023-09-22T21:59:09.456 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ= 2023-09-22T21:59:09.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ=', 'Content-Length': '0'} 2023-09-22T21:59:09.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bc877cd5d418a85-00650e0e2d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.466 DEBUG:boto:b'ffzdqu-61obj1851000falseobj1852023-09-22T21:56:10.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:09.466 DEBUG:boto:path=/ffzdqu-61/obj185 2023-09-22T21:59:09.466 DEBUG:boto:auth_path=/ffzdqu-61/obj185 2023-09-22T21:59:09.466 DEBUG:boto:Method: HEAD 2023-09-22T21:59:09.467 DEBUG:boto:Path: /ffzdqu-61/obj185 2023-09-22T21:59:09.467 DEBUG:boto:Data: 2023-09-22T21:59:09.467 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.467 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.467 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.467 DEBUG:boto:Params: {} 2023-09-22T21:59:09.467 DEBUG:boto:Token: None 2023-09-22T21:59:09.467 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj185 2023-09-22T21:59:09.467 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rtWem700ul0ywTBvz4NLIjNIQt4= 2023-09-22T21:59:09.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rtWem700ul0ywTBvz4NLIjNIQt4=', 'Content-Length': '0'} 2023-09-22T21:59:09.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff1d78ded362e547-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.470 DEBUG:boto:path=/ffzdqu-61/obj185 2023-09-22T21:59:09.470 DEBUG:boto:auth_path=/ffzdqu-61/obj185 2023-09-22T21:59:09.471 DEBUG:boto:Method: HEAD 2023-09-22T21:59:09.471 DEBUG:boto:Path: /ffzdqu-61/obj185 2023-09-22T21:59:09.471 DEBUG:boto:Data: 2023-09-22T21:59:09.471 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.471 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.471 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.471 DEBUG:boto:Params: {} 2023-09-22T21:59:09.471 DEBUG:boto:Token: None 2023-09-22T21:59:09.471 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj185 2023-09-22T21:59:09.472 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rtWem700ul0ywTBvz4NLIjNIQt4= 2023-09-22T21:59:09.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rtWem700ul0ywTBvz4NLIjNIQt4=', 'Content-Length': '0'} 2023-09-22T21:59:09.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008add0edcc2837b37-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-09-22T21:59:09.475 DEBUG:boto:path=/ffzdqu-61/obj185 2023-09-22T21:59:09.475 DEBUG:boto:auth_path=/ffzdqu-61/obj185 2023-09-22T21:59:09.475 DEBUG:boto:Method: GET 2023-09-22T21:59:09.475 DEBUG:boto:Path: /ffzdqu-61/obj185 2023-09-22T21:59:09.475 DEBUG:boto:Data: 2023-09-22T21:59:09.475 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.475 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.475 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.475 DEBUG:boto:Params: {} 2023-09-22T21:59:09.475 DEBUG:boto:Token: None 2023-09-22T21:59:09.476 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj185 2023-09-22T21:59:09.476 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2OoiR3+Rj6tObx3Q17sOBE5aqQE= 2023-09-22T21:59:09.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2OoiR3+Rj6tObx3Q17sOBE5aqQE=', 'Content-Length': '0'} 2023-09-22T21:59:09.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053b60297d5f1e466-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.479 DEBUG:boto:path=/ffzdqu-61/obj185 2023-09-22T21:59:09.479 DEBUG:boto:auth_path=/ffzdqu-61/obj185 2023-09-22T21:59:09.479 DEBUG:boto:Method: GET 2023-09-22T21:59:09.479 DEBUG:boto:Path: /ffzdqu-61/obj185 2023-09-22T21:59:09.479 DEBUG:boto:Data: 2023-09-22T21:59:09.479 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.480 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.480 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.480 DEBUG:boto:Params: {} 2023-09-22T21:59:09.480 DEBUG:boto:Token: None 2023-09-22T21:59:09.480 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj185 2023-09-22T21:59:09.480 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2OoiR3+Rj6tObx3Q17sOBE5aqQE= 2023-09-22T21:59:09.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2OoiR3+Rj6tObx3Q17sOBE5aqQE=', 'Content-Length': '0'} 2023-09-22T21:59:09.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e1ea3f0a2c18046-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.526 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:09.526 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:09.526 DEBUG:boto:path=/ffzdqu-61/?prefix=obj185 2023-09-22T21:59:09.527 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj185 2023-09-22T21:59:09.527 DEBUG:boto:Method: GET 2023-09-22T21:59:09.527 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj185 2023-09-22T21:59:09.527 DEBUG:boto:Data: 2023-09-22T21:59:09.527 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.527 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.527 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.527 DEBUG:boto:Params: {} 2023-09-22T21:59:09.527 DEBUG:boto:Token: None 2023-09-22T21:59:09.528 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/ 2023-09-22T21:59:09.528 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ= 2023-09-22T21:59:09.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ=', 'Content-Length': '0'} 2023-09-22T21:59:09.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068acc2e6a9adfdbd-00650e0e2d-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.532 DEBUG:boto:b'ffzdqu-61obj1851000falseobj1852023-09-22T21:56:10.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:09.533 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:09.533 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:09.533 DEBUG:boto:path=/ffzdqu-61/?prefix=obj185 2023-09-22T21:59:09.533 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj185 2023-09-22T21:59:09.533 DEBUG:boto:Method: GET 2023-09-22T21:59:09.534 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj185 2023-09-22T21:59:09.534 DEBUG:boto:Data: 2023-09-22T21:59:09.534 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.534 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.534 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.534 DEBUG:boto:Params: {} 2023-09-22T21:59:09.534 DEBUG:boto:Token: None 2023-09-22T21:59:09.534 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/ 2023-09-22T21:59:09.534 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ= 2023-09-22T21:59:09.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ=', 'Content-Length': '0'} 2023-09-22T21:59:09.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bad8c5caeff3c75-00650e0e2d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.544 DEBUG:boto:b'ffzdqu-61obj1851000falseobj1852023-09-22T21:56:10.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:09.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T21:59:09.545 DEBUG:boto:path=/ffzdqu-61/obj19 2023-09-22T21:59:09.545 DEBUG:boto:auth_path=/ffzdqu-61/obj19 2023-09-22T21:59:09.545 DEBUG:boto:path=/ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.545 DEBUG:boto:auth_path=/ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.545 DEBUG:boto:Method: GET 2023-09-22T21:59:09.545 DEBUG:boto:Path: /ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.545 DEBUG:boto:Data: 2023-09-22T21:59:09.545 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.545 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.545 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.546 DEBUG:boto:Params: {} 2023-09-22T21:59:09.546 DEBUG:boto:Token: None 2023-09-22T21:59:09.546 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.546 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y7AOFKnY2HyE8arROt0LqBvY0tE= 2023-09-22T21:59:09.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y7AOFKnY2HyE8arROt0LqBvY0tE=', 'Content-Length': '0'} 2023-09-22T21:59:09.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3b35278950c356f-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.590 DEBUG:boto:path=/ffzdqu-61/obj19 2023-09-22T21:59:09.590 DEBUG:boto:auth_path=/ffzdqu-61/obj19 2023-09-22T21:59:09.590 DEBUG:boto:path=/ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.591 DEBUG:boto:auth_path=/ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.591 DEBUG:boto:Method: GET 2023-09-22T21:59:09.591 DEBUG:boto:Path: /ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.591 DEBUG:boto:Data: 2023-09-22T21:59:09.591 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.591 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.591 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.591 DEBUG:boto:Params: {} 2023-09-22T21:59:09.591 DEBUG:boto:Token: None 2023-09-22T21:59:09.592 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.592 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y7AOFKnY2HyE8arROt0LqBvY0tE= 2023-09-22T21:59:09.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y7AOFKnY2HyE8arROt0LqBvY0tE=', 'Content-Length': '0'} 2023-09-22T21:59:09.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fad863b6bd974b56-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.595 DEBUG:boto:path=/ffzdqu-61/obj19 2023-09-22T21:59:09.595 DEBUG:boto:auth_path=/ffzdqu-61/obj19 2023-09-22T21:59:09.595 DEBUG:boto:path=/ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.595 DEBUG:boto:auth_path=/ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.595 DEBUG:boto:Method: HEAD 2023-09-22T21:59:09.595 DEBUG:boto:Path: /ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.596 DEBUG:boto:Data: 2023-09-22T21:59:09.596 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.596 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.596 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.596 DEBUG:boto:Params: {} 2023-09-22T21:59:09.596 DEBUG:boto:Token: None 2023-09-22T21:59:09.596 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.596 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EdNT/YBgFRNSXA94gKMz2slrMGw= 2023-09-22T21:59:09.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EdNT/YBgFRNSXA94gKMz2slrMGw=', 'Content-Length': '0'} 2023-09-22T21:59:09.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d639bad57751809-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.599 DEBUG:boto:path=/ffzdqu-61/obj19 2023-09-22T21:59:09.599 DEBUG:boto:auth_path=/ffzdqu-61/obj19 2023-09-22T21:59:09.599 DEBUG:boto:path=/ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.600 DEBUG:boto:auth_path=/ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.600 DEBUG:boto:Method: HEAD 2023-09-22T21:59:09.600 DEBUG:boto:Path: /ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.600 DEBUG:boto:Data: 2023-09-22T21:59:09.600 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.600 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.600 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.600 DEBUG:boto:Params: {} 2023-09-22T21:59:09.600 DEBUG:boto:Token: None 2023-09-22T21:59:09.601 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.601 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EdNT/YBgFRNSXA94gKMz2slrMGw= 2023-09-22T21:59:09.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EdNT/YBgFRNSXA94gKMz2slrMGw=', 'Content-Length': '0'} 2023-09-22T21:59:09.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6b9883e625c783f-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T21:59:09.604 DEBUG:boto:path=/ffzdqu-61/obj19 2023-09-22T21:59:09.604 DEBUG:boto:auth_path=/ffzdqu-61/obj19 2023-09-22T21:59:09.604 DEBUG:boto:path=/ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.604 DEBUG:boto:auth_path=/ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.604 DEBUG:boto:Method: GET 2023-09-22T21:59:09.604 DEBUG:boto:Path: /ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.604 DEBUG:boto:Data: 2023-09-22T21:59:09.604 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.605 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.605 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.605 DEBUG:boto:Params: {} 2023-09-22T21:59:09.605 DEBUG:boto:Token: None 2023-09-22T21:59:09.605 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.605 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y7AOFKnY2HyE8arROt0LqBvY0tE= 2023-09-22T21:59:09.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y7AOFKnY2HyE8arROt0LqBvY0tE=', 'Content-Length': '0'} 2023-09-22T21:59:09.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e556847790f291e-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.608 DEBUG:boto:path=/ffzdqu-61/obj19 2023-09-22T21:59:09.608 DEBUG:boto:auth_path=/ffzdqu-61/obj19 2023-09-22T21:59:09.609 DEBUG:boto:path=/ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.609 DEBUG:boto:auth_path=/ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.609 DEBUG:boto:Method: GET 2023-09-22T21:59:09.609 DEBUG:boto:Path: /ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.609 DEBUG:boto:Data: 2023-09-22T21:59:09.609 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.609 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.609 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.609 DEBUG:boto:Params: {} 2023-09-22T21:59:09.610 DEBUG:boto:Token: None 2023-09-22T21:59:09.610 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj19?versionId=null 2023-09-22T21:59:09.610 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y7AOFKnY2HyE8arROt0LqBvY0tE= 2023-09-22T21:59:09.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y7AOFKnY2HyE8arROt0LqBvY0tE=', 'Content-Length': '0'} 2023-09-22T21:59:09.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fe753de78c96bc7-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.654 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:09.654 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:09.654 DEBUG:boto:path=/ffzdqu-61/?prefix=obj19 2023-09-22T21:59:09.655 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj19 2023-09-22T21:59:09.655 DEBUG:boto:Method: GET 2023-09-22T21:59:09.655 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj19 2023-09-22T21:59:09.655 DEBUG:boto:Data: 2023-09-22T21:59:09.655 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.655 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.655 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.655 DEBUG:boto:Params: {} 2023-09-22T21:59:09.655 DEBUG:boto:Token: None 2023-09-22T21:59:09.656 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/ 2023-09-22T21:59:09.656 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ= 2023-09-22T21:59:09.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ=', 'Content-Length': '0'} 2023-09-22T21:59:09.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4c958a96a957630-00650e0e2d-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.661 DEBUG:boto:b'ffzdqu-61obj191000falseobj192023-09-22T21:56:09.260Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:09.661 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:09.662 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:09.662 DEBUG:boto:path=/ffzdqu-61/?prefix=obj19 2023-09-22T21:59:09.662 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj19 2023-09-22T21:59:09.662 DEBUG:boto:Method: GET 2023-09-22T21:59:09.662 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj19 2023-09-22T21:59:09.662 DEBUG:boto:Data: 2023-09-22T21:59:09.662 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.662 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.662 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.663 DEBUG:boto:Params: {} 2023-09-22T21:59:09.663 DEBUG:boto:Token: None 2023-09-22T21:59:09.663 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/ 2023-09-22T21:59:09.663 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ= 2023-09-22T21:59:09.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ=', 'Content-Length': '0'} 2023-09-22T21:59:09.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005460a69759310b70-00650e0e2d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.676 DEBUG:boto:b'ffzdqu-61obj191000falseobj192023-09-22T21:56:09.260Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:09.677 DEBUG:boto:path=/ffzdqu-61/obj19 2023-09-22T21:59:09.677 DEBUG:boto:auth_path=/ffzdqu-61/obj19 2023-09-22T21:59:09.677 DEBUG:boto:Method: HEAD 2023-09-22T21:59:09.677 DEBUG:boto:Path: /ffzdqu-61/obj19 2023-09-22T21:59:09.677 DEBUG:boto:Data: 2023-09-22T21:59:09.677 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.677 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.677 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.678 DEBUG:boto:Params: {} 2023-09-22T21:59:09.678 DEBUG:boto:Token: None 2023-09-22T21:59:09.678 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj19 2023-09-22T21:59:09.678 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nWDKt8JQvtvifH5U7kHWFN68kIw= 2023-09-22T21:59:09.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nWDKt8JQvtvifH5U7kHWFN68kIw=', 'Content-Length': '0'} 2023-09-22T21:59:09.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a7258c5087ac44f-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.681 DEBUG:boto:path=/ffzdqu-61/obj19 2023-09-22T21:59:09.681 DEBUG:boto:auth_path=/ffzdqu-61/obj19 2023-09-22T21:59:09.681 DEBUG:boto:Method: HEAD 2023-09-22T21:59:09.681 DEBUG:boto:Path: /ffzdqu-61/obj19 2023-09-22T21:59:09.681 DEBUG:boto:Data: 2023-09-22T21:59:09.681 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.682 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.682 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.682 DEBUG:boto:Params: {} 2023-09-22T21:59:09.682 DEBUG:boto:Token: None 2023-09-22T21:59:09.682 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj19 2023-09-22T21:59:09.682 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nWDKt8JQvtvifH5U7kHWFN68kIw= 2023-09-22T21:59:09.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nWDKt8JQvtvifH5U7kHWFN68kIw=', 'Content-Length': '0'} 2023-09-22T21:59:09.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006079687d0391d655-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T21:59:09.686 DEBUG:boto:path=/ffzdqu-61/obj19 2023-09-22T21:59:09.686 DEBUG:boto:auth_path=/ffzdqu-61/obj19 2023-09-22T21:59:09.686 DEBUG:boto:Method: GET 2023-09-22T21:59:09.686 DEBUG:boto:Path: /ffzdqu-61/obj19 2023-09-22T21:59:09.687 DEBUG:boto:Data: 2023-09-22T21:59:09.687 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.687 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.687 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.687 DEBUG:boto:Params: {} 2023-09-22T21:59:09.687 DEBUG:boto:Token: None 2023-09-22T21:59:09.687 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj19 2023-09-22T21:59:09.687 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7QACVWofT1eezZBAeNodxrpWIWA= 2023-09-22T21:59:09.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7QACVWofT1eezZBAeNodxrpWIWA=', 'Content-Length': '0'} 2023-09-22T21:59:09.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000857af774823294-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.730 DEBUG:boto:path=/ffzdqu-61/obj19 2023-09-22T21:59:09.730 DEBUG:boto:auth_path=/ffzdqu-61/obj19 2023-09-22T21:59:09.730 DEBUG:boto:Method: GET 2023-09-22T21:59:09.731 DEBUG:boto:Path: /ffzdqu-61/obj19 2023-09-22T21:59:09.731 DEBUG:boto:Data: 2023-09-22T21:59:09.731 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.731 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.731 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.731 DEBUG:boto:Params: {} 2023-09-22T21:59:09.731 DEBUG:boto:Token: None 2023-09-22T21:59:09.731 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj19 2023-09-22T21:59:09.731 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7QACVWofT1eezZBAeNodxrpWIWA= 2023-09-22T21:59:09.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7QACVWofT1eezZBAeNodxrpWIWA=', 'Content-Length': '0'} 2023-09-22T21:59:09.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cdecb205a59bec0-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.735 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:09.735 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:09.735 DEBUG:boto:path=/ffzdqu-61/?prefix=obj19 2023-09-22T21:59:09.735 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj19 2023-09-22T21:59:09.735 DEBUG:boto:Method: GET 2023-09-22T21:59:09.736 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj19 2023-09-22T21:59:09.736 DEBUG:boto:Data: 2023-09-22T21:59:09.736 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.736 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.736 DEBUG:boto:Params: {} 2023-09-22T21:59:09.736 DEBUG:boto:Token: None 2023-09-22T21:59:09.736 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/ 2023-09-22T21:59:09.736 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ= 2023-09-22T21:59:09.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ=', 'Content-Length': '0'} 2023-09-22T21:59:09.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f32b1ec1db37f5a4-00650e0e2d-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.743 DEBUG:boto:b'ffzdqu-61obj191000falseobj192023-09-22T21:56:09.260Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:09.743 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:09.743 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:09.743 DEBUG:boto:path=/ffzdqu-61/?prefix=obj19 2023-09-22T21:59:09.744 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj19 2023-09-22T21:59:09.744 DEBUG:boto:Method: GET 2023-09-22T21:59:09.744 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj19 2023-09-22T21:59:09.744 DEBUG:boto:Data: 2023-09-22T21:59:09.744 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.744 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.744 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.744 DEBUG:boto:Params: {} 2023-09-22T21:59:09.744 DEBUG:boto:Token: None 2023-09-22T21:59:09.745 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/ 2023-09-22T21:59:09.745 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ= 2023-09-22T21:59:09.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ=', 'Content-Length': '0'} 2023-09-22T21:59:09.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a3305bd2eb10465-00650e0e2d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.798 DEBUG:boto:b'ffzdqu-61obj191000falseobj192023-09-22T21:56:09.260Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:09.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:59:09.799 DEBUG:boto:path=/ffzdqu-61/obj22 2023-09-22T21:59:09.799 DEBUG:boto:auth_path=/ffzdqu-61/obj22 2023-09-22T21:59:09.800 DEBUG:boto:path=/ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.800 DEBUG:boto:auth_path=/ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.800 DEBUG:boto:Method: GET 2023-09-22T21:59:09.800 DEBUG:boto:Path: /ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.800 DEBUG:boto:Data: 2023-09-22T21:59:09.800 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.801 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.801 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.801 DEBUG:boto:Params: {} 2023-09-22T21:59:09.801 DEBUG:boto:Token: None 2023-09-22T21:59:09.802 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.802 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QLH8oaHqyayk2vaBhI3POwEj1g0= 2023-09-22T21:59:09.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QLH8oaHqyayk2vaBhI3POwEj1g0=', 'Content-Length': '0'} 2023-09-22T21:59:09.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6a3861038e29eec-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.805 DEBUG:boto:path=/ffzdqu-61/obj22 2023-09-22T21:59:09.806 DEBUG:boto:auth_path=/ffzdqu-61/obj22 2023-09-22T21:59:09.806 DEBUG:boto:path=/ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.806 DEBUG:boto:auth_path=/ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.806 DEBUG:boto:Method: GET 2023-09-22T21:59:09.806 DEBUG:boto:Path: /ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.807 DEBUG:boto:Data: 2023-09-22T21:59:09.807 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.807 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.807 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.807 DEBUG:boto:Params: {} 2023-09-22T21:59:09.807 DEBUG:boto:Token: None 2023-09-22T21:59:09.808 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.808 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QLH8oaHqyayk2vaBhI3POwEj1g0= 2023-09-22T21:59:09.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QLH8oaHqyayk2vaBhI3POwEj1g0=', 'Content-Length': '0'} 2023-09-22T21:59:09.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcd9e40d96b2eb8b-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.811 DEBUG:boto:path=/ffzdqu-61/obj22 2023-09-22T21:59:09.811 DEBUG:boto:auth_path=/ffzdqu-61/obj22 2023-09-22T21:59:09.811 DEBUG:boto:path=/ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.811 DEBUG:boto:auth_path=/ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.812 DEBUG:boto:Method: HEAD 2023-09-22T21:59:09.812 DEBUG:boto:Path: /ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.812 DEBUG:boto:Data: 2023-09-22T21:59:09.812 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.812 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.812 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.813 DEBUG:boto:Params: {} 2023-09-22T21:59:09.813 DEBUG:boto:Token: None 2023-09-22T21:59:09.813 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.813 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6d/LWlhJRUvaT88KChiD+sm1Jzo= 2023-09-22T21:59:09.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6d/LWlhJRUvaT88KChiD+sm1Jzo=', 'Content-Length': '0'} 2023-09-22T21:59:09.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acb6c1a271ecded2-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.816 DEBUG:boto:path=/ffzdqu-61/obj22 2023-09-22T21:59:09.816 DEBUG:boto:auth_path=/ffzdqu-61/obj22 2023-09-22T21:59:09.816 DEBUG:boto:path=/ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.817 DEBUG:boto:auth_path=/ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.817 DEBUG:boto:Method: HEAD 2023-09-22T21:59:09.817 DEBUG:boto:Path: /ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.817 DEBUG:boto:Data: 2023-09-22T21:59:09.817 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.817 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.818 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.818 DEBUG:boto:Params: {} 2023-09-22T21:59:09.818 DEBUG:boto:Token: None 2023-09-22T21:59:09.818 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.819 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6d/LWlhJRUvaT88KChiD+sm1Jzo= 2023-09-22T21:59:09.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6d/LWlhJRUvaT88KChiD+sm1Jzo=', 'Content-Length': '0'} 2023-09-22T21:59:09.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003612ad6430214750-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:59:09.822 DEBUG:boto:path=/ffzdqu-61/obj22 2023-09-22T21:59:09.822 DEBUG:boto:auth_path=/ffzdqu-61/obj22 2023-09-22T21:59:09.822 DEBUG:boto:path=/ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.822 DEBUG:boto:auth_path=/ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.822 DEBUG:boto:Method: GET 2023-09-22T21:59:09.823 DEBUG:boto:Path: /ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.823 DEBUG:boto:Data: 2023-09-22T21:59:09.823 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.823 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.823 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.823 DEBUG:boto:Params: {} 2023-09-22T21:59:09.824 DEBUG:boto:Token: None 2023-09-22T21:59:09.824 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.824 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QLH8oaHqyayk2vaBhI3POwEj1g0= 2023-09-22T21:59:09.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QLH8oaHqyayk2vaBhI3POwEj1g0=', 'Content-Length': '0'} 2023-09-22T21:59:09.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095b233d5f60be2f3-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.827 DEBUG:boto:path=/ffzdqu-61/obj22 2023-09-22T21:59:09.827 DEBUG:boto:auth_path=/ffzdqu-61/obj22 2023-09-22T21:59:09.827 DEBUG:boto:path=/ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.827 DEBUG:boto:auth_path=/ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.828 DEBUG:boto:Method: GET 2023-09-22T21:59:09.828 DEBUG:boto:Path: /ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.828 DEBUG:boto:Data: 2023-09-22T21:59:09.828 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.828 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.828 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.828 DEBUG:boto:Params: {} 2023-09-22T21:59:09.829 DEBUG:boto:Token: None 2023-09-22T21:59:09.829 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj22?versionId=null 2023-09-22T21:59:09.829 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QLH8oaHqyayk2vaBhI3POwEj1g0= 2023-09-22T21:59:09.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QLH8oaHqyayk2vaBhI3POwEj1g0=', 'Content-Length': '0'} 2023-09-22T21:59:09.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007242b157a808c73c-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.832 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:09.832 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:09.833 DEBUG:boto:path=/ffzdqu-61/?prefix=obj22 2023-09-22T21:59:09.833 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj22 2023-09-22T21:59:09.833 DEBUG:boto:Method: GET 2023-09-22T21:59:09.833 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj22 2023-09-22T21:59:09.833 DEBUG:boto:Data: 2023-09-22T21:59:09.833 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.834 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.834 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.834 DEBUG:boto:Params: {} 2023-09-22T21:59:09.834 DEBUG:boto:Token: None 2023-09-22T21:59:09.834 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/ 2023-09-22T21:59:09.835 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ= 2023-09-22T21:59:09.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ=', 'Content-Length': '0'} 2023-09-22T21:59:09.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daa7db1799a60db6-00650e0e2d-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.882 DEBUG:boto:b'ffzdqu-61obj221000falseobj222023-09-22T21:56:08.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:09.883 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:09.883 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:09.883 DEBUG:boto:path=/ffzdqu-61/?prefix=obj22 2023-09-22T21:59:09.883 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj22 2023-09-22T21:59:09.883 DEBUG:boto:Method: GET 2023-09-22T21:59:09.883 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj22 2023-09-22T21:59:09.884 DEBUG:boto:Data: 2023-09-22T21:59:09.884 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.884 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.884 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.884 DEBUG:boto:Params: {} 2023-09-22T21:59:09.884 DEBUG:boto:Token: None 2023-09-22T21:59:09.885 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/ 2023-09-22T21:59:09.885 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ= 2023-09-22T21:59:09.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ=', 'Content-Length': '0'} 2023-09-22T21:59:09.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1538a6a81fe4bf5-00650e0e2d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.894 DEBUG:boto:b'ffzdqu-61obj221000falseobj222023-09-22T21:56:08.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:09.895 DEBUG:boto:path=/ffzdqu-61/obj22 2023-09-22T21:59:09.895 DEBUG:boto:auth_path=/ffzdqu-61/obj22 2023-09-22T21:59:09.895 DEBUG:boto:Method: HEAD 2023-09-22T21:59:09.895 DEBUG:boto:Path: /ffzdqu-61/obj22 2023-09-22T21:59:09.895 DEBUG:boto:Data: 2023-09-22T21:59:09.895 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.896 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.896 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.896 DEBUG:boto:Params: {} 2023-09-22T21:59:09.896 DEBUG:boto:Token: None 2023-09-22T21:59:09.896 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj22 2023-09-22T21:59:09.896 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LWQbIFPALrqx8BO2ojI/LDajaus= 2023-09-22T21:59:09.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LWQbIFPALrqx8BO2ojI/LDajaus=', 'Content-Length': '0'} 2023-09-22T21:59:09.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000931818dfd9b6820-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.899 DEBUG:boto:path=/ffzdqu-61/obj22 2023-09-22T21:59:09.900 DEBUG:boto:auth_path=/ffzdqu-61/obj22 2023-09-22T21:59:09.900 DEBUG:boto:Method: HEAD 2023-09-22T21:59:09.900 DEBUG:boto:Path: /ffzdqu-61/obj22 2023-09-22T21:59:09.900 DEBUG:boto:Data: 2023-09-22T21:59:09.900 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.900 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.901 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.901 DEBUG:boto:Params: {} 2023-09-22T21:59:09.901 DEBUG:boto:Token: None 2023-09-22T21:59:09.901 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj22 2023-09-22T21:59:09.901 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LWQbIFPALrqx8BO2ojI/LDajaus= 2023-09-22T21:59:09.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LWQbIFPALrqx8BO2ojI/LDajaus=', 'Content-Length': '0'} 2023-09-22T21:59:09.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4cf221b2256630b-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T21:59:09.905 DEBUG:boto:path=/ffzdqu-61/obj22 2023-09-22T21:59:09.905 DEBUG:boto:auth_path=/ffzdqu-61/obj22 2023-09-22T21:59:09.905 DEBUG:boto:Method: GET 2023-09-22T21:59:09.905 DEBUG:boto:Path: /ffzdqu-61/obj22 2023-09-22T21:59:09.906 DEBUG:boto:Data: 2023-09-22T21:59:09.906 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.906 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.906 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.906 DEBUG:boto:Params: {} 2023-09-22T21:59:09.906 DEBUG:boto:Token: None 2023-09-22T21:59:09.907 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj22 2023-09-22T21:59:09.907 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xe072evR9o2YkSuyj/Nzw9z/wE0= 2023-09-22T21:59:09.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xe072evR9o2YkSuyj/Nzw9z/wE0=', 'Content-Length': '0'} 2023-09-22T21:59:09.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000916878207dadbb96-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.910 DEBUG:boto:path=/ffzdqu-61/obj22 2023-09-22T21:59:09.910 DEBUG:boto:auth_path=/ffzdqu-61/obj22 2023-09-22T21:59:09.910 DEBUG:boto:Method: GET 2023-09-22T21:59:09.910 DEBUG:boto:Path: /ffzdqu-61/obj22 2023-09-22T21:59:09.910 DEBUG:boto:Data: 2023-09-22T21:59:09.910 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.910 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.911 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.911 DEBUG:boto:Params: {} 2023-09-22T21:59:09.911 DEBUG:boto:Token: None 2023-09-22T21:59:09.911 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj22 2023-09-22T21:59:09.911 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xe072evR9o2YkSuyj/Nzw9z/wE0= 2023-09-22T21:59:09.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xe072evR9o2YkSuyj/Nzw9z/wE0=', 'Content-Length': '0'} 2023-09-22T21:59:09.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050da4d7f94c3fee2-00650e0e2d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.954 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:09.954 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:09.955 DEBUG:boto:path=/ffzdqu-61/?prefix=obj22 2023-09-22T21:59:09.955 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj22 2023-09-22T21:59:09.955 DEBUG:boto:Method: GET 2023-09-22T21:59:09.955 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj22 2023-09-22T21:59:09.955 DEBUG:boto:Data: 2023-09-22T21:59:09.955 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.955 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.955 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.955 DEBUG:boto:Params: {} 2023-09-22T21:59:09.956 DEBUG:boto:Token: None 2023-09-22T21:59:09.956 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/ 2023-09-22T21:59:09.956 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ= 2023-09-22T21:59:09.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ=', 'Content-Length': '0'} 2023-09-22T21:59:09.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b373bb7218624a36-00650e0e2d-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.961 DEBUG:boto:b'ffzdqu-61obj221000falseobj222023-09-22T21:56:08.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:09.961 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:09.962 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:09.962 DEBUG:boto:path=/ffzdqu-61/?prefix=obj22 2023-09-22T21:59:09.962 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj22 2023-09-22T21:59:09.962 DEBUG:boto:Method: GET 2023-09-22T21:59:09.962 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj22 2023-09-22T21:59:09.962 DEBUG:boto:Data: 2023-09-22T21:59:09.962 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.963 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:09.963 DEBUG:boto:Port: 8000 2023-09-22T21:59:09.963 DEBUG:boto:Params: {} 2023-09-22T21:59:09.963 DEBUG:boto:Token: None 2023-09-22T21:59:09.963 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/ 2023-09-22T21:59:09.963 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ= 2023-09-22T21:59:09.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8grLGoobzBP0NODn4tANntM8tQ=', 'Content-Length': '0'} 2023-09-22T21:59:09.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f61d2a33910a5bf8-00650e0e2d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:09.973 DEBUG:boto:b'ffzdqu-61obj221000falseobj222023-09-22T21:56:08.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:09.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-22T21:59:09.973 DEBUG:boto:path=/ffzdqu-61/obj23 2023-09-22T21:59:09.973 DEBUG:boto:auth_path=/ffzdqu-61/obj23 2023-09-22T21:59:09.974 DEBUG:boto:path=/ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:09.974 DEBUG:boto:auth_path=/ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:09.974 DEBUG:boto:Method: GET 2023-09-22T21:59:09.974 DEBUG:boto:Path: /ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:09.974 DEBUG:boto:Data: 2023-09-22T21:59:09.974 DEBUG:boto:Headers: {} 2023-09-22T21:59:09.974 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:09.974 DEBUG:boto:Port: 8001 2023-09-22T21:59:09.975 DEBUG:boto:Params: {} 2023-09-22T21:59:09.975 DEBUG:boto:Token: None 2023-09-22T21:59:09.975 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:09 GMT /ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:09.975 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:umlrPW1PHTW2RNt3WSJdvgm/ci4= 2023-09-22T21:59:09.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:09 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:umlrPW1PHTW2RNt3WSJdvgm/ci4=', 'Content-Length': '0'} 2023-09-22T21:59:09.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f739047b23ca4dec-00650e0e2d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.018 DEBUG:boto:path=/ffzdqu-61/obj23 2023-09-22T21:59:10.018 DEBUG:boto:auth_path=/ffzdqu-61/obj23 2023-09-22T21:59:10.018 DEBUG:boto:path=/ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:10.019 DEBUG:boto:auth_path=/ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:10.019 DEBUG:boto:Method: GET 2023-09-22T21:59:10.019 DEBUG:boto:Path: /ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:10.019 DEBUG:boto:Data: 2023-09-22T21:59:10.019 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.019 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.019 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.019 DEBUG:boto:Params: {} 2023-09-22T21:59:10.020 DEBUG:boto:Token: None 2023-09-22T21:59:10.020 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:10.020 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rT/EAE/QCnDdPm/ELPHPaVMVq80= 2023-09-22T21:59:10.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rT/EAE/QCnDdPm/ELPHPaVMVq80=', 'Content-Length': '0'} 2023-09-22T21:59:10.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2ba5fae08d97d68-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.023 DEBUG:boto:path=/ffzdqu-61/obj23 2023-09-22T21:59:10.023 DEBUG:boto:auth_path=/ffzdqu-61/obj23 2023-09-22T21:59:10.023 DEBUG:boto:path=/ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:10.023 DEBUG:boto:auth_path=/ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:10.023 DEBUG:boto:Method: HEAD 2023-09-22T21:59:10.024 DEBUG:boto:Path: /ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:10.024 DEBUG:boto:Data: 2023-09-22T21:59:10.024 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.024 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.024 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.024 DEBUG:boto:Params: {} 2023-09-22T21:59:10.024 DEBUG:boto:Token: None 2023-09-22T21:59:10.024 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:10.025 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3AzpI7iv03tp3oQuAeoj5SksBGI= 2023-09-22T21:59:10.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3AzpI7iv03tp3oQuAeoj5SksBGI=', 'Content-Length': '0'} 2023-09-22T21:59:10.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e44971159220041-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.027 DEBUG:boto:path=/ffzdqu-61/obj23 2023-09-22T21:59:10.027 DEBUG:boto:auth_path=/ffzdqu-61/obj23 2023-09-22T21:59:10.028 DEBUG:boto:path=/ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:10.028 DEBUG:boto:auth_path=/ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:10.028 DEBUG:boto:Method: HEAD 2023-09-22T21:59:10.028 DEBUG:boto:Path: /ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:10.028 DEBUG:boto:Data: 2023-09-22T21:59:10.028 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.028 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.028 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.029 DEBUG:boto:Params: {} 2023-09-22T21:59:10.029 DEBUG:boto:Token: None 2023-09-22T21:59:10.029 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:10.029 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3AzpI7iv03tp3oQuAeoj5SksBGI= 2023-09-22T21:59:10.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3AzpI7iv03tp3oQuAeoj5SksBGI=', 'Content-Length': '0'} 2023-09-22T21:59:10.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f9016a03b3abbdc-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-22T21:59:10.032 DEBUG:boto:path=/ffzdqu-61/obj23 2023-09-22T21:59:10.033 DEBUG:boto:auth_path=/ffzdqu-61/obj23 2023-09-22T21:59:10.033 DEBUG:boto:path=/ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:10.033 DEBUG:boto:auth_path=/ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:10.033 DEBUG:boto:Method: GET 2023-09-22T21:59:10.033 DEBUG:boto:Path: /ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:10.033 DEBUG:boto:Data: 2023-09-22T21:59:10.033 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.033 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.034 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.034 DEBUG:boto:Params: {} 2023-09-22T21:59:10.034 DEBUG:boto:Token: None 2023-09-22T21:59:10.034 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:10.034 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rT/EAE/QCnDdPm/ELPHPaVMVq80= 2023-09-22T21:59:10.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rT/EAE/QCnDdPm/ELPHPaVMVq80=', 'Content-Length': '0'} 2023-09-22T21:59:10.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000585f0d452f45c7d5-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.038 DEBUG:boto:path=/ffzdqu-61/obj23 2023-09-22T21:59:10.038 DEBUG:boto:auth_path=/ffzdqu-61/obj23 2023-09-22T21:59:10.038 DEBUG:boto:path=/ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:10.038 DEBUG:boto:auth_path=/ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:10.038 DEBUG:boto:Method: GET 2023-09-22T21:59:10.038 DEBUG:boto:Path: /ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:10.039 DEBUG:boto:Data: 2023-09-22T21:59:10.039 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.039 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.039 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.039 DEBUG:boto:Params: {} 2023-09-22T21:59:10.039 DEBUG:boto:Token: None 2023-09-22T21:59:10.039 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj23?versionId=null 2023-09-22T21:59:10.039 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rT/EAE/QCnDdPm/ELPHPaVMVq80= 2023-09-22T21:59:10.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rT/EAE/QCnDdPm/ELPHPaVMVq80=', 'Content-Length': '0'} 2023-09-22T21:59:10.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe74845a2d6c68cd-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.086 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:10.086 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:10.087 DEBUG:boto:path=/ffzdqu-61/?prefix=obj23 2023-09-22T21:59:10.087 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj23 2023-09-22T21:59:10.087 DEBUG:boto:Method: GET 2023-09-22T21:59:10.087 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj23 2023-09-22T21:59:10.087 DEBUG:boto:Data: 2023-09-22T21:59:10.087 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.087 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.087 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.087 DEBUG:boto:Params: {} 2023-09-22T21:59:10.088 DEBUG:boto:Token: None 2023-09-22T21:59:10.088 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/ 2023-09-22T21:59:10.088 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ= 2023-09-22T21:59:10.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ=', 'Content-Length': '0'} 2023-09-22T21:59:10.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0eb0ab38a6e48c2-00650e0e2e-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.093 DEBUG:boto:b'ffzdqu-61obj231000falseobj232023-09-22T21:56:09.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:10.094 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:10.094 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:10.094 DEBUG:boto:path=/ffzdqu-61/?prefix=obj23 2023-09-22T21:59:10.094 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj23 2023-09-22T21:59:10.094 DEBUG:boto:Method: GET 2023-09-22T21:59:10.094 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj23 2023-09-22T21:59:10.094 DEBUG:boto:Data: 2023-09-22T21:59:10.095 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.095 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.095 DEBUG:boto:Params: {} 2023-09-22T21:59:10.095 DEBUG:boto:Token: None 2023-09-22T21:59:10.095 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/ 2023-09-22T21:59:10.095 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ= 2023-09-22T21:59:10.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ=', 'Content-Length': '0'} 2023-09-22T21:59:10.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ee6e16894174b4b-00650e0e2e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.106 DEBUG:boto:b'ffzdqu-61obj231000falseobj232023-09-22T21:56:09.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:10.106 DEBUG:boto:path=/ffzdqu-61/obj23 2023-09-22T21:59:10.106 DEBUG:boto:auth_path=/ffzdqu-61/obj23 2023-09-22T21:59:10.106 DEBUG:boto:Method: HEAD 2023-09-22T21:59:10.106 DEBUG:boto:Path: /ffzdqu-61/obj23 2023-09-22T21:59:10.107 DEBUG:boto:Data: 2023-09-22T21:59:10.107 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.107 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.107 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.107 DEBUG:boto:Params: {} 2023-09-22T21:59:10.107 DEBUG:boto:Token: None 2023-09-22T21:59:10.107 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj23 2023-09-22T21:59:10.108 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+LEg/jFq5PgOlN6+aDTKe1hkWJE= 2023-09-22T21:59:10.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+LEg/jFq5PgOlN6+aDTKe1hkWJE=', 'Content-Length': '0'} 2023-09-22T21:59:10.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6364a0ef399dd6d-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.110 DEBUG:boto:path=/ffzdqu-61/obj23 2023-09-22T21:59:10.111 DEBUG:boto:auth_path=/ffzdqu-61/obj23 2023-09-22T21:59:10.111 DEBUG:boto:Method: HEAD 2023-09-22T21:59:10.111 DEBUG:boto:Path: /ffzdqu-61/obj23 2023-09-22T21:59:10.111 DEBUG:boto:Data: 2023-09-22T21:59:10.111 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.111 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.111 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.111 DEBUG:boto:Params: {} 2023-09-22T21:59:10.112 DEBUG:boto:Token: None 2023-09-22T21:59:10.112 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj23 2023-09-22T21:59:10.112 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+LEg/jFq5PgOlN6+aDTKe1hkWJE= 2023-09-22T21:59:10.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+LEg/jFq5PgOlN6+aDTKe1hkWJE=', 'Content-Length': '0'} 2023-09-22T21:59:10.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000207d087b5f7c092c-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-22T21:59:10.115 DEBUG:boto:path=/ffzdqu-61/obj23 2023-09-22T21:59:10.115 DEBUG:boto:auth_path=/ffzdqu-61/obj23 2023-09-22T21:59:10.115 DEBUG:boto:Method: GET 2023-09-22T21:59:10.116 DEBUG:boto:Path: /ffzdqu-61/obj23 2023-09-22T21:59:10.116 DEBUG:boto:Data: 2023-09-22T21:59:10.116 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.116 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.116 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.116 DEBUG:boto:Params: {} 2023-09-22T21:59:10.116 DEBUG:boto:Token: None 2023-09-22T21:59:10.117 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj23 2023-09-22T21:59:10.117 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n2oDRfYkTm25l6YZIgx2zJQ647I= 2023-09-22T21:59:10.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n2oDRfYkTm25l6YZIgx2zJQ647I=', 'Content-Length': '0'} 2023-09-22T21:59:10.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000123434bf7a15922d-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.162 DEBUG:boto:path=/ffzdqu-61/obj23 2023-09-22T21:59:10.162 DEBUG:boto:auth_path=/ffzdqu-61/obj23 2023-09-22T21:59:10.163 DEBUG:boto:Method: GET 2023-09-22T21:59:10.163 DEBUG:boto:Path: /ffzdqu-61/obj23 2023-09-22T21:59:10.163 DEBUG:boto:Data: 2023-09-22T21:59:10.163 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.163 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.163 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.163 DEBUG:boto:Params: {} 2023-09-22T21:59:10.163 DEBUG:boto:Token: None 2023-09-22T21:59:10.164 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj23 2023-09-22T21:59:10.164 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n2oDRfYkTm25l6YZIgx2zJQ647I= 2023-09-22T21:59:10.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n2oDRfYkTm25l6YZIgx2zJQ647I=', 'Content-Length': '0'} 2023-09-22T21:59:10.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000539296212b33a4db-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.167 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:10.167 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:10.167 DEBUG:boto:path=/ffzdqu-61/?prefix=obj23 2023-09-22T21:59:10.167 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj23 2023-09-22T21:59:10.167 DEBUG:boto:Method: GET 2023-09-22T21:59:10.167 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj23 2023-09-22T21:59:10.168 DEBUG:boto:Data: 2023-09-22T21:59:10.168 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.168 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.168 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.168 DEBUG:boto:Params: {} 2023-09-22T21:59:10.168 DEBUG:boto:Token: None 2023-09-22T21:59:10.168 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/ 2023-09-22T21:59:10.168 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ= 2023-09-22T21:59:10.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ=', 'Content-Length': '0'} 2023-09-22T21:59:10.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e830638be11138d-00650e0e2e-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.176 DEBUG:boto:b'ffzdqu-61obj231000falseobj232023-09-22T21:56:09.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:10.176 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:10.176 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:10.177 DEBUG:boto:path=/ffzdqu-61/?prefix=obj23 2023-09-22T21:59:10.177 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj23 2023-09-22T21:59:10.177 DEBUG:boto:Method: GET 2023-09-22T21:59:10.177 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj23 2023-09-22T21:59:10.177 DEBUG:boto:Data: 2023-09-22T21:59:10.177 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.177 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.178 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.178 DEBUG:boto:Params: {} 2023-09-22T21:59:10.178 DEBUG:boto:Token: None 2023-09-22T21:59:10.178 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/ 2023-09-22T21:59:10.178 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ= 2023-09-22T21:59:10.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ=', 'Content-Length': '0'} 2023-09-22T21:59:10.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbde8dc630e5635e-00650e0e2e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.230 DEBUG:boto:b'ffzdqu-61obj231000falseobj232023-09-22T21:56:09.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:10.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-22T21:59:10.231 DEBUG:boto:path=/ffzdqu-61/obj29 2023-09-22T21:59:10.231 DEBUG:boto:auth_path=/ffzdqu-61/obj29 2023-09-22T21:59:10.231 DEBUG:boto:path=/ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.231 DEBUG:boto:auth_path=/ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.232 DEBUG:boto:Method: GET 2023-09-22T21:59:10.232 DEBUG:boto:Path: /ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.232 DEBUG:boto:Data: 2023-09-22T21:59:10.232 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.232 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.232 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.232 DEBUG:boto:Params: {} 2023-09-22T21:59:10.233 DEBUG:boto:Token: None 2023-09-22T21:59:10.233 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.233 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G2jSsr1bE3AU10lVMveYm8peHgM= 2023-09-22T21:59:10.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G2jSsr1bE3AU10lVMveYm8peHgM=', 'Content-Length': '0'} 2023-09-22T21:59:10.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058088a605148c24a-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.236 DEBUG:boto:path=/ffzdqu-61/obj29 2023-09-22T21:59:10.237 DEBUG:boto:auth_path=/ffzdqu-61/obj29 2023-09-22T21:59:10.237 DEBUG:boto:path=/ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.237 DEBUG:boto:auth_path=/ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.237 DEBUG:boto:Method: GET 2023-09-22T21:59:10.237 DEBUG:boto:Path: /ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.237 DEBUG:boto:Data: 2023-09-22T21:59:10.237 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.237 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.238 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.238 DEBUG:boto:Params: {} 2023-09-22T21:59:10.238 DEBUG:boto:Token: None 2023-09-22T21:59:10.238 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.238 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G2jSsr1bE3AU10lVMveYm8peHgM= 2023-09-22T21:59:10.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G2jSsr1bE3AU10lVMveYm8peHgM=', 'Content-Length': '0'} 2023-09-22T21:59:10.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5d97610584959fd-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.241 DEBUG:boto:path=/ffzdqu-61/obj29 2023-09-22T21:59:10.241 DEBUG:boto:auth_path=/ffzdqu-61/obj29 2023-09-22T21:59:10.241 DEBUG:boto:path=/ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.242 DEBUG:boto:auth_path=/ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.242 DEBUG:boto:Method: HEAD 2023-09-22T21:59:10.242 DEBUG:boto:Path: /ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.242 DEBUG:boto:Data: 2023-09-22T21:59:10.242 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.242 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.242 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.242 DEBUG:boto:Params: {} 2023-09-22T21:59:10.243 DEBUG:boto:Token: None 2023-09-22T21:59:10.243 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.243 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JK1BPYLSn5oeXLX4TSHTvcNE9Ss= 2023-09-22T21:59:10.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JK1BPYLSn5oeXLX4TSHTvcNE9Ss=', 'Content-Length': '0'} 2023-09-22T21:59:10.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e67fd6730bb494cf-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.246 DEBUG:boto:path=/ffzdqu-61/obj29 2023-09-22T21:59:10.246 DEBUG:boto:auth_path=/ffzdqu-61/obj29 2023-09-22T21:59:10.246 DEBUG:boto:path=/ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.246 DEBUG:boto:auth_path=/ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.247 DEBUG:boto:Method: HEAD 2023-09-22T21:59:10.247 DEBUG:boto:Path: /ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.247 DEBUG:boto:Data: 2023-09-22T21:59:10.247 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.247 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.247 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.247 DEBUG:boto:Params: {} 2023-09-22T21:59:10.247 DEBUG:boto:Token: None 2023-09-22T21:59:10.248 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.248 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JK1BPYLSn5oeXLX4TSHTvcNE9Ss= 2023-09-22T21:59:10.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JK1BPYLSn5oeXLX4TSHTvcNE9Ss=', 'Content-Length': '0'} 2023-09-22T21:59:10.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c89ba567a2a524ad-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-22T21:59:10.251 DEBUG:boto:path=/ffzdqu-61/obj29 2023-09-22T21:59:10.251 DEBUG:boto:auth_path=/ffzdqu-61/obj29 2023-09-22T21:59:10.251 DEBUG:boto:path=/ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.251 DEBUG:boto:auth_path=/ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.251 DEBUG:boto:Method: GET 2023-09-22T21:59:10.251 DEBUG:boto:Path: /ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.252 DEBUG:boto:Data: 2023-09-22T21:59:10.252 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.252 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.252 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.252 DEBUG:boto:Params: {} 2023-09-22T21:59:10.252 DEBUG:boto:Token: None 2023-09-22T21:59:10.252 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.252 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G2jSsr1bE3AU10lVMveYm8peHgM= 2023-09-22T21:59:10.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G2jSsr1bE3AU10lVMveYm8peHgM=', 'Content-Length': '0'} 2023-09-22T21:59:10.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033609371455e6b4e-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.256 DEBUG:boto:path=/ffzdqu-61/obj29 2023-09-22T21:59:10.256 DEBUG:boto:auth_path=/ffzdqu-61/obj29 2023-09-22T21:59:10.256 DEBUG:boto:path=/ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.256 DEBUG:boto:auth_path=/ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.256 DEBUG:boto:Method: GET 2023-09-22T21:59:10.256 DEBUG:boto:Path: /ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.256 DEBUG:boto:Data: 2023-09-22T21:59:10.257 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.257 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.257 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.257 DEBUG:boto:Params: {} 2023-09-22T21:59:10.257 DEBUG:boto:Token: None 2023-09-22T21:59:10.257 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj29?versionId=null 2023-09-22T21:59:10.257 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G2jSsr1bE3AU10lVMveYm8peHgM= 2023-09-22T21:59:10.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G2jSsr1bE3AU10lVMveYm8peHgM=', 'Content-Length': '0'} 2023-09-22T21:59:10.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018826713507120ed-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.260 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:10.260 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:10.260 DEBUG:boto:path=/ffzdqu-61/?prefix=obj29 2023-09-22T21:59:10.261 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj29 2023-09-22T21:59:10.261 DEBUG:boto:Method: GET 2023-09-22T21:59:10.261 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj29 2023-09-22T21:59:10.261 DEBUG:boto:Data: 2023-09-22T21:59:10.261 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.261 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.261 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.261 DEBUG:boto:Params: {} 2023-09-22T21:59:10.261 DEBUG:boto:Token: None 2023-09-22T21:59:10.262 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/ 2023-09-22T21:59:10.262 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ= 2023-09-22T21:59:10.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ=', 'Content-Length': '0'} 2023-09-22T21:59:10.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2f11e301c291534-00650e0e2e-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.310 DEBUG:boto:b'ffzdqu-61obj291000falseobj292023-09-22T21:56:09.931Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:10.311 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:10.311 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:10.311 DEBUG:boto:path=/ffzdqu-61/?prefix=obj29 2023-09-22T21:59:10.311 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj29 2023-09-22T21:59:10.311 DEBUG:boto:Method: GET 2023-09-22T21:59:10.311 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj29 2023-09-22T21:59:10.311 DEBUG:boto:Data: 2023-09-22T21:59:10.312 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.312 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.312 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.312 DEBUG:boto:Params: {} 2023-09-22T21:59:10.312 DEBUG:boto:Token: None 2023-09-22T21:59:10.312 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/ 2023-09-22T21:59:10.312 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ= 2023-09-22T21:59:10.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ=', 'Content-Length': '0'} 2023-09-22T21:59:10.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000884535d34e8edf4f-00650e0e2e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.322 DEBUG:boto:b'ffzdqu-61obj291000falseobj292023-09-22T21:56:09.931Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:10.322 DEBUG:boto:path=/ffzdqu-61/obj29 2023-09-22T21:59:10.322 DEBUG:boto:auth_path=/ffzdqu-61/obj29 2023-09-22T21:59:10.323 DEBUG:boto:Method: HEAD 2023-09-22T21:59:10.323 DEBUG:boto:Path: /ffzdqu-61/obj29 2023-09-22T21:59:10.323 DEBUG:boto:Data: 2023-09-22T21:59:10.323 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.323 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.323 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.323 DEBUG:boto:Params: {} 2023-09-22T21:59:10.324 DEBUG:boto:Token: None 2023-09-22T21:59:10.324 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj29 2023-09-22T21:59:10.324 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H9JkRB/xSvVCCD+T3jucFf72gEA= 2023-09-22T21:59:10.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H9JkRB/xSvVCCD+T3jucFf72gEA=', 'Content-Length': '0'} 2023-09-22T21:59:10.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af0f84ba8b6a8e2a-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.327 DEBUG:boto:path=/ffzdqu-61/obj29 2023-09-22T21:59:10.327 DEBUG:boto:auth_path=/ffzdqu-61/obj29 2023-09-22T21:59:10.327 DEBUG:boto:Method: HEAD 2023-09-22T21:59:10.327 DEBUG:boto:Path: /ffzdqu-61/obj29 2023-09-22T21:59:10.327 DEBUG:boto:Data: 2023-09-22T21:59:10.328 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.328 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.328 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.328 DEBUG:boto:Params: {} 2023-09-22T21:59:10.328 DEBUG:boto:Token: None 2023-09-22T21:59:10.328 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj29 2023-09-22T21:59:10.328 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H9JkRB/xSvVCCD+T3jucFf72gEA= 2023-09-22T21:59:10.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H9JkRB/xSvVCCD+T3jucFf72gEA=', 'Content-Length': '0'} 2023-09-22T21:59:10.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd61f2d3dd968352-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-22T21:59:10.332 DEBUG:boto:path=/ffzdqu-61/obj29 2023-09-22T21:59:10.332 DEBUG:boto:auth_path=/ffzdqu-61/obj29 2023-09-22T21:59:10.332 DEBUG:boto:Method: GET 2023-09-22T21:59:10.332 DEBUG:boto:Path: /ffzdqu-61/obj29 2023-09-22T21:59:10.332 DEBUG:boto:Data: 2023-09-22T21:59:10.332 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.332 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.332 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.332 DEBUG:boto:Params: {} 2023-09-22T21:59:10.333 DEBUG:boto:Token: None 2023-09-22T21:59:10.333 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj29 2023-09-22T21:59:10.333 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y0t1VFFhI1VsszHGPfSAF5mKYZg= 2023-09-22T21:59:10.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y0t1VFFhI1VsszHGPfSAF5mKYZg=', 'Content-Length': '0'} 2023-09-22T21:59:10.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d85d1a222f530d1-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.336 DEBUG:boto:path=/ffzdqu-61/obj29 2023-09-22T21:59:10.336 DEBUG:boto:auth_path=/ffzdqu-61/obj29 2023-09-22T21:59:10.336 DEBUG:boto:Method: GET 2023-09-22T21:59:10.336 DEBUG:boto:Path: /ffzdqu-61/obj29 2023-09-22T21:59:10.336 DEBUG:boto:Data: 2023-09-22T21:59:10.336 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.337 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.337 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.337 DEBUG:boto:Params: {} 2023-09-22T21:59:10.337 DEBUG:boto:Token: None 2023-09-22T21:59:10.337 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj29 2023-09-22T21:59:10.337 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y0t1VFFhI1VsszHGPfSAF5mKYZg= 2023-09-22T21:59:10.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y0t1VFFhI1VsszHGPfSAF5mKYZg=', 'Content-Length': '0'} 2023-09-22T21:59:10.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2dfbdf9b2e761e1-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.382 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:10.382 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:10.383 DEBUG:boto:path=/ffzdqu-61/?prefix=obj29 2023-09-22T21:59:10.383 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj29 2023-09-22T21:59:10.383 DEBUG:boto:Method: GET 2023-09-22T21:59:10.383 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj29 2023-09-22T21:59:10.383 DEBUG:boto:Data: 2023-09-22T21:59:10.383 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.383 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.383 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.384 DEBUG:boto:Params: {} 2023-09-22T21:59:10.384 DEBUG:boto:Token: None 2023-09-22T21:59:10.384 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/ 2023-09-22T21:59:10.384 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ= 2023-09-22T21:59:10.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ=', 'Content-Length': '0'} 2023-09-22T21:59:10.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7d6b17720e28ede-00650e0e2e-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.389 DEBUG:boto:b'ffzdqu-61obj291000falseobj292023-09-22T21:56:09.931Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:10.390 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:10.390 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:10.390 DEBUG:boto:path=/ffzdqu-61/?prefix=obj29 2023-09-22T21:59:10.390 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj29 2023-09-22T21:59:10.390 DEBUG:boto:Method: GET 2023-09-22T21:59:10.390 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj29 2023-09-22T21:59:10.391 DEBUG:boto:Data: 2023-09-22T21:59:10.391 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.391 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.391 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.391 DEBUG:boto:Params: {} 2023-09-22T21:59:10.391 DEBUG:boto:Token: None 2023-09-22T21:59:10.391 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/ 2023-09-22T21:59:10.391 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ= 2023-09-22T21:59:10.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ=', 'Content-Length': '0'} 2023-09-22T21:59:10.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc27a5802d2966d5-00650e0e2e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.401 DEBUG:boto:b'ffzdqu-61obj291000falseobj292023-09-22T21:56:09.931Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:10.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-22T21:59:10.402 DEBUG:boto:path=/ffzdqu-61/obj31 2023-09-22T21:59:10.402 DEBUG:boto:auth_path=/ffzdqu-61/obj31 2023-09-22T21:59:10.402 DEBUG:boto:path=/ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.402 DEBUG:boto:auth_path=/ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.403 DEBUG:boto:Method: GET 2023-09-22T21:59:10.403 DEBUG:boto:Path: /ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.403 DEBUG:boto:Data: 2023-09-22T21:59:10.403 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.403 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.403 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.403 DEBUG:boto:Params: {} 2023-09-22T21:59:10.403 DEBUG:boto:Token: None 2023-09-22T21:59:10.404 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.404 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F8sRVyYciyiyiVzUvXNdH8TYKQc= 2023-09-22T21:59:10.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F8sRVyYciyiyiVzUvXNdH8TYKQc=', 'Content-Length': '0'} 2023-09-22T21:59:10.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e6400def55bc4e0-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.450 DEBUG:boto:path=/ffzdqu-61/obj31 2023-09-22T21:59:10.450 DEBUG:boto:auth_path=/ffzdqu-61/obj31 2023-09-22T21:59:10.451 DEBUG:boto:path=/ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.451 DEBUG:boto:auth_path=/ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.451 DEBUG:boto:Method: GET 2023-09-22T21:59:10.451 DEBUG:boto:Path: /ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.451 DEBUG:boto:Data: 2023-09-22T21:59:10.451 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.451 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.451 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.451 DEBUG:boto:Params: {} 2023-09-22T21:59:10.452 DEBUG:boto:Token: None 2023-09-22T21:59:10.452 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.452 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F8sRVyYciyiyiVzUvXNdH8TYKQc= 2023-09-22T21:59:10.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F8sRVyYciyiyiVzUvXNdH8TYKQc=', 'Content-Length': '0'} 2023-09-22T21:59:10.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006005f14f4c34d892-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.455 DEBUG:boto:path=/ffzdqu-61/obj31 2023-09-22T21:59:10.455 DEBUG:boto:auth_path=/ffzdqu-61/obj31 2023-09-22T21:59:10.456 DEBUG:boto:path=/ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.456 DEBUG:boto:auth_path=/ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.456 DEBUG:boto:Method: HEAD 2023-09-22T21:59:10.456 DEBUG:boto:Path: /ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.456 DEBUG:boto:Data: 2023-09-22T21:59:10.456 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.456 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.456 DEBUG:boto:Params: {} 2023-09-22T21:59:10.457 DEBUG:boto:Token: None 2023-09-22T21:59:10.457 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.457 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6d+KdbjaIWqRe21iNLYMWSNJQ20= 2023-09-22T21:59:10.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6d+KdbjaIWqRe21iNLYMWSNJQ20=', 'Content-Length': '0'} 2023-09-22T21:59:10.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045bbe25fcd279c3a-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.460 DEBUG:boto:path=/ffzdqu-61/obj31 2023-09-22T21:59:10.460 DEBUG:boto:auth_path=/ffzdqu-61/obj31 2023-09-22T21:59:10.460 DEBUG:boto:path=/ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.460 DEBUG:boto:auth_path=/ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.460 DEBUG:boto:Method: HEAD 2023-09-22T21:59:10.460 DEBUG:boto:Path: /ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.461 DEBUG:boto:Data: 2023-09-22T21:59:10.461 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.461 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.461 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.461 DEBUG:boto:Params: {} 2023-09-22T21:59:10.461 DEBUG:boto:Token: None 2023-09-22T21:59:10.461 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.462 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6d+KdbjaIWqRe21iNLYMWSNJQ20= 2023-09-22T21:59:10.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6d+KdbjaIWqRe21iNLYMWSNJQ20=', 'Content-Length': '0'} 2023-09-22T21:59:10.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ace80e0a3a45eb6f-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-22T21:59:10.464 DEBUG:boto:path=/ffzdqu-61/obj31 2023-09-22T21:59:10.465 DEBUG:boto:auth_path=/ffzdqu-61/obj31 2023-09-22T21:59:10.465 DEBUG:boto:path=/ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.465 DEBUG:boto:auth_path=/ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.465 DEBUG:boto:Method: GET 2023-09-22T21:59:10.465 DEBUG:boto:Path: /ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.465 DEBUG:boto:Data: 2023-09-22T21:59:10.465 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.465 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.466 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.466 DEBUG:boto:Params: {} 2023-09-22T21:59:10.466 DEBUG:boto:Token: None 2023-09-22T21:59:10.466 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.466 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F8sRVyYciyiyiVzUvXNdH8TYKQc= 2023-09-22T21:59:10.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F8sRVyYciyiyiVzUvXNdH8TYKQc=', 'Content-Length': '0'} 2023-09-22T21:59:10.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a232ac06d0c2fc9-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.469 DEBUG:boto:path=/ffzdqu-61/obj31 2023-09-22T21:59:10.469 DEBUG:boto:auth_path=/ffzdqu-61/obj31 2023-09-22T21:59:10.469 DEBUG:boto:path=/ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.469 DEBUG:boto:auth_path=/ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.470 DEBUG:boto:Method: GET 2023-09-22T21:59:10.470 DEBUG:boto:Path: /ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.470 DEBUG:boto:Data: 2023-09-22T21:59:10.470 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.470 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.470 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.470 DEBUG:boto:Params: {} 2023-09-22T21:59:10.470 DEBUG:boto:Token: None 2023-09-22T21:59:10.471 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj31?versionId=null 2023-09-22T21:59:10.471 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:F8sRVyYciyiyiVzUvXNdH8TYKQc= 2023-09-22T21:59:10.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:F8sRVyYciyiyiVzUvXNdH8TYKQc=', 'Content-Length': '0'} 2023-09-22T21:59:10.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7aefc5ea3bc5a10-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.474 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:10.474 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:10.474 DEBUG:boto:path=/ffzdqu-61/?prefix=obj31 2023-09-22T21:59:10.474 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj31 2023-09-22T21:59:10.475 DEBUG:boto:Method: GET 2023-09-22T21:59:10.475 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj31 2023-09-22T21:59:10.475 DEBUG:boto:Data: 2023-09-22T21:59:10.475 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.475 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.475 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.475 DEBUG:boto:Params: {} 2023-09-22T21:59:10.475 DEBUG:boto:Token: None 2023-09-22T21:59:10.476 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/ 2023-09-22T21:59:10.476 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ= 2023-09-22T21:59:10.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ=', 'Content-Length': '0'} 2023-09-22T21:59:10.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039d7ef0a37be372a-00650e0e2e-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.481 DEBUG:boto:b'ffzdqu-61obj311000falseobj312023-09-22T21:56:10.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:10.481 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:10.482 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:10.482 DEBUG:boto:path=/ffzdqu-61/?prefix=obj31 2023-09-22T21:59:10.482 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj31 2023-09-22T21:59:10.482 DEBUG:boto:Method: GET 2023-09-22T21:59:10.482 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj31 2023-09-22T21:59:10.482 DEBUG:boto:Data: 2023-09-22T21:59:10.482 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.482 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.482 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.483 DEBUG:boto:Params: {} 2023-09-22T21:59:10.483 DEBUG:boto:Token: None 2023-09-22T21:59:10.483 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/ 2023-09-22T21:59:10.483 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ= 2023-09-22T21:59:10.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ=', 'Content-Length': '0'} 2023-09-22T21:59:10.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000740931e0bf304d4e-00650e0e2e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.493 DEBUG:boto:b'ffzdqu-61obj311000falseobj312023-09-22T21:56:10.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:10.493 DEBUG:boto:path=/ffzdqu-61/obj31 2023-09-22T21:59:10.493 DEBUG:boto:auth_path=/ffzdqu-61/obj31 2023-09-22T21:59:10.493 DEBUG:boto:Method: HEAD 2023-09-22T21:59:10.493 DEBUG:boto:Path: /ffzdqu-61/obj31 2023-09-22T21:59:10.494 DEBUG:boto:Data: 2023-09-22T21:59:10.494 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.494 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.494 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.494 DEBUG:boto:Params: {} 2023-09-22T21:59:10.494 DEBUG:boto:Token: None 2023-09-22T21:59:10.494 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj31 2023-09-22T21:59:10.494 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Na0INyAgLLSFtxd/blDe7zLfg+8= 2023-09-22T21:59:10.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Na0INyAgLLSFtxd/blDe7zLfg+8=', 'Content-Length': '0'} 2023-09-22T21:59:10.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e9f476d9b99b9ec-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.498 DEBUG:boto:path=/ffzdqu-61/obj31 2023-09-22T21:59:10.498 DEBUG:boto:auth_path=/ffzdqu-61/obj31 2023-09-22T21:59:10.498 DEBUG:boto:Method: HEAD 2023-09-22T21:59:10.498 DEBUG:boto:Path: /ffzdqu-61/obj31 2023-09-22T21:59:10.498 DEBUG:boto:Data: 2023-09-22T21:59:10.498 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.499 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.499 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.499 DEBUG:boto:Params: {} 2023-09-22T21:59:10.499 DEBUG:boto:Token: None 2023-09-22T21:59:10.499 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj31 2023-09-22T21:59:10.499 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Na0INyAgLLSFtxd/blDe7zLfg+8= 2023-09-22T21:59:10.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Na0INyAgLLSFtxd/blDe7zLfg+8=', 'Content-Length': '0'} 2023-09-22T21:59:10.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a29918c557f0abea-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-22T21:59:10.502 DEBUG:boto:path=/ffzdqu-61/obj31 2023-09-22T21:59:10.502 DEBUG:boto:auth_path=/ffzdqu-61/obj31 2023-09-22T21:59:10.502 DEBUG:boto:Method: GET 2023-09-22T21:59:10.503 DEBUG:boto:Path: /ffzdqu-61/obj31 2023-09-22T21:59:10.503 DEBUG:boto:Data: 2023-09-22T21:59:10.503 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.503 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.503 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.503 DEBUG:boto:Params: {} 2023-09-22T21:59:10.503 DEBUG:boto:Token: None 2023-09-22T21:59:10.503 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj31 2023-09-22T21:59:10.504 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zsek0daJddnsBl+/NP5pyRv+n8M= 2023-09-22T21:59:10.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zsek0daJddnsBl+/NP5pyRv+n8M=', 'Content-Length': '0'} 2023-09-22T21:59:10.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077bb33abe78fb6c6-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.550 DEBUG:boto:path=/ffzdqu-61/obj31 2023-09-22T21:59:10.551 DEBUG:boto:auth_path=/ffzdqu-61/obj31 2023-09-22T21:59:10.551 DEBUG:boto:Method: GET 2023-09-22T21:59:10.551 DEBUG:boto:Path: /ffzdqu-61/obj31 2023-09-22T21:59:10.551 DEBUG:boto:Data: 2023-09-22T21:59:10.551 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.551 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.551 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.551 DEBUG:boto:Params: {} 2023-09-22T21:59:10.552 DEBUG:boto:Token: None 2023-09-22T21:59:10.552 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj31 2023-09-22T21:59:10.552 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zsek0daJddnsBl+/NP5pyRv+n8M= 2023-09-22T21:59:10.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zsek0daJddnsBl+/NP5pyRv+n8M=', 'Content-Length': '0'} 2023-09-22T21:59:10.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f7782f42091153b-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.555 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:10.555 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:10.555 DEBUG:boto:path=/ffzdqu-61/?prefix=obj31 2023-09-22T21:59:10.556 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj31 2023-09-22T21:59:10.556 DEBUG:boto:Method: GET 2023-09-22T21:59:10.556 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj31 2023-09-22T21:59:10.556 DEBUG:boto:Data: 2023-09-22T21:59:10.556 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.556 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.556 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.556 DEBUG:boto:Params: {} 2023-09-22T21:59:10.557 DEBUG:boto:Token: None 2023-09-22T21:59:10.557 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/ 2023-09-22T21:59:10.557 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ= 2023-09-22T21:59:10.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ=', 'Content-Length': '0'} 2023-09-22T21:59:10.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007571af891fb1994f-00650e0e2e-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.563 DEBUG:boto:b'ffzdqu-61obj311000falseobj312023-09-22T21:56:10.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:10.564 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:10.564 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:10.564 DEBUG:boto:path=/ffzdqu-61/?prefix=obj31 2023-09-22T21:59:10.564 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj31 2023-09-22T21:59:10.564 DEBUG:boto:Method: GET 2023-09-22T21:59:10.564 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj31 2023-09-22T21:59:10.565 DEBUG:boto:Data: 2023-09-22T21:59:10.565 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.565 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.565 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.565 DEBUG:boto:Params: {} 2023-09-22T21:59:10.565 DEBUG:boto:Token: None 2023-09-22T21:59:10.565 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/ 2023-09-22T21:59:10.566 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ= 2023-09-22T21:59:10.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ=', 'Content-Length': '0'} 2023-09-22T21:59:10.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000355cc21be4252a20-00650e0e2e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.575 DEBUG:boto:b'ffzdqu-61obj311000falseobj312023-09-22T21:56:10.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:10.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:59:10.576 DEBUG:boto:path=/ffzdqu-61/obj33 2023-09-22T21:59:10.576 DEBUG:boto:auth_path=/ffzdqu-61/obj33 2023-09-22T21:59:10.576 DEBUG:boto:path=/ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.576 DEBUG:boto:auth_path=/ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.577 DEBUG:boto:Method: GET 2023-09-22T21:59:10.577 DEBUG:boto:Path: /ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.577 DEBUG:boto:Data: 2023-09-22T21:59:10.577 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.577 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.577 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.577 DEBUG:boto:Params: {} 2023-09-22T21:59:10.577 DEBUG:boto:Token: None 2023-09-22T21:59:10.577 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.578 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Hah3UW/AMHSFuWMp+lutwhlw2j4= 2023-09-22T21:59:10.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Hah3UW/AMHSFuWMp+lutwhlw2j4=', 'Content-Length': '0'} 2023-09-22T21:59:10.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac9dfeab3effdcad-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.581 DEBUG:boto:path=/ffzdqu-61/obj33 2023-09-22T21:59:10.581 DEBUG:boto:auth_path=/ffzdqu-61/obj33 2023-09-22T21:59:10.581 DEBUG:boto:path=/ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.581 DEBUG:boto:auth_path=/ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.581 DEBUG:boto:Method: GET 2023-09-22T21:59:10.581 DEBUG:boto:Path: /ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.582 DEBUG:boto:Data: 2023-09-22T21:59:10.582 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.582 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.582 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.582 DEBUG:boto:Params: {} 2023-09-22T21:59:10.582 DEBUG:boto:Token: None 2023-09-22T21:59:10.582 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.582 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Hah3UW/AMHSFuWMp+lutwhlw2j4= 2023-09-22T21:59:10.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Hah3UW/AMHSFuWMp+lutwhlw2j4=', 'Content-Length': '0'} 2023-09-22T21:59:10.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072cfb38778e9f9ad-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.586 DEBUG:boto:path=/ffzdqu-61/obj33 2023-09-22T21:59:10.586 DEBUG:boto:auth_path=/ffzdqu-61/obj33 2023-09-22T21:59:10.586 DEBUG:boto:path=/ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.586 DEBUG:boto:auth_path=/ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.586 DEBUG:boto:Method: HEAD 2023-09-22T21:59:10.586 DEBUG:boto:Path: /ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.587 DEBUG:boto:Data: 2023-09-22T21:59:10.587 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.587 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.587 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.587 DEBUG:boto:Params: {} 2023-09-22T21:59:10.587 DEBUG:boto:Token: None 2023-09-22T21:59:10.587 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.587 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/9Cc5ME/EAshZtViB37KpsgVm4I= 2023-09-22T21:59:10.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/9Cc5ME/EAshZtViB37KpsgVm4I=', 'Content-Length': '0'} 2023-09-22T21:59:10.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7fc78c1c0c8a5ee-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.590 DEBUG:boto:path=/ffzdqu-61/obj33 2023-09-22T21:59:10.591 DEBUG:boto:auth_path=/ffzdqu-61/obj33 2023-09-22T21:59:10.591 DEBUG:boto:path=/ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.591 DEBUG:boto:auth_path=/ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.591 DEBUG:boto:Method: HEAD 2023-09-22T21:59:10.591 DEBUG:boto:Path: /ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.591 DEBUG:boto:Data: 2023-09-22T21:59:10.591 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.591 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.592 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.592 DEBUG:boto:Params: {} 2023-09-22T21:59:10.592 DEBUG:boto:Token: None 2023-09-22T21:59:10.592 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.592 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/9Cc5ME/EAshZtViB37KpsgVm4I= 2023-09-22T21:59:10.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/9Cc5ME/EAshZtViB37KpsgVm4I=', 'Content-Length': '0'} 2023-09-22T21:59:10.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000505a25f1db7b26f8-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:59:10.595 DEBUG:boto:path=/ffzdqu-61/obj33 2023-09-22T21:59:10.595 DEBUG:boto:auth_path=/ffzdqu-61/obj33 2023-09-22T21:59:10.595 DEBUG:boto:path=/ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.595 DEBUG:boto:auth_path=/ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.595 DEBUG:boto:Method: GET 2023-09-22T21:59:10.596 DEBUG:boto:Path: /ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.596 DEBUG:boto:Data: 2023-09-22T21:59:10.596 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.596 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.596 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.596 DEBUG:boto:Params: {} 2023-09-22T21:59:10.596 DEBUG:boto:Token: None 2023-09-22T21:59:10.596 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.597 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Hah3UW/AMHSFuWMp+lutwhlw2j4= 2023-09-22T21:59:10.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Hah3UW/AMHSFuWMp+lutwhlw2j4=', 'Content-Length': '0'} 2023-09-22T21:59:10.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000557233ea23972b1-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.599 DEBUG:boto:path=/ffzdqu-61/obj33 2023-09-22T21:59:10.600 DEBUG:boto:auth_path=/ffzdqu-61/obj33 2023-09-22T21:59:10.600 DEBUG:boto:path=/ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.600 DEBUG:boto:auth_path=/ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.600 DEBUG:boto:Method: GET 2023-09-22T21:59:10.600 DEBUG:boto:Path: /ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.600 DEBUG:boto:Data: 2023-09-22T21:59:10.600 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.601 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.601 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.601 DEBUG:boto:Params: {} 2023-09-22T21:59:10.601 DEBUG:boto:Token: None 2023-09-22T21:59:10.601 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj33?versionId=null 2023-09-22T21:59:10.601 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Hah3UW/AMHSFuWMp+lutwhlw2j4= 2023-09-22T21:59:10.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Hah3UW/AMHSFuWMp+lutwhlw2j4=', 'Content-Length': '0'} 2023-09-22T21:59:10.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fc07470a939eb12-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.646 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:10.646 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:10.647 DEBUG:boto:path=/ffzdqu-61/?prefix=obj33 2023-09-22T21:59:10.647 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj33 2023-09-22T21:59:10.647 DEBUG:boto:Method: GET 2023-09-22T21:59:10.647 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj33 2023-09-22T21:59:10.647 DEBUG:boto:Data: 2023-09-22T21:59:10.647 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.648 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.648 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.648 DEBUG:boto:Params: {} 2023-09-22T21:59:10.648 DEBUG:boto:Token: None 2023-09-22T21:59:10.648 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/ 2023-09-22T21:59:10.648 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ= 2023-09-22T21:59:10.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ=', 'Content-Length': '0'} 2023-09-22T21:59:10.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bf0f2d2dfe98ee5-00650e0e2e-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.653 DEBUG:boto:b'ffzdqu-61obj331000falseobj332023-09-22T21:56:08.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:10.653 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:10.654 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:10.654 DEBUG:boto:path=/ffzdqu-61/?prefix=obj33 2023-09-22T21:59:10.654 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj33 2023-09-22T21:59:10.654 DEBUG:boto:Method: GET 2023-09-22T21:59:10.654 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj33 2023-09-22T21:59:10.654 DEBUG:boto:Data: 2023-09-22T21:59:10.654 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.654 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.655 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.655 DEBUG:boto:Params: {} 2023-09-22T21:59:10.655 DEBUG:boto:Token: None 2023-09-22T21:59:10.655 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/ 2023-09-22T21:59:10.655 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ= 2023-09-22T21:59:10.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ=', 'Content-Length': '0'} 2023-09-22T21:59:10.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be247b5ea1b016b0-00650e0e2e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.665 DEBUG:boto:b'ffzdqu-61obj331000falseobj332023-09-22T21:56:08.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:10.665 DEBUG:boto:path=/ffzdqu-61/obj33 2023-09-22T21:59:10.666 DEBUG:boto:auth_path=/ffzdqu-61/obj33 2023-09-22T21:59:10.666 DEBUG:boto:Method: HEAD 2023-09-22T21:59:10.666 DEBUG:boto:Path: /ffzdqu-61/obj33 2023-09-22T21:59:10.666 DEBUG:boto:Data: 2023-09-22T21:59:10.666 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.666 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.666 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.666 DEBUG:boto:Params: {} 2023-09-22T21:59:10.667 DEBUG:boto:Token: None 2023-09-22T21:59:10.667 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj33 2023-09-22T21:59:10.667 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m7FDZJIFq3tmRdITacljAfAMaA8= 2023-09-22T21:59:10.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m7FDZJIFq3tmRdITacljAfAMaA8=', 'Content-Length': '0'} 2023-09-22T21:59:10.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d32e671fcbdfdf-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.670 DEBUG:boto:path=/ffzdqu-61/obj33 2023-09-22T21:59:10.670 DEBUG:boto:auth_path=/ffzdqu-61/obj33 2023-09-22T21:59:10.670 DEBUG:boto:Method: HEAD 2023-09-22T21:59:10.671 DEBUG:boto:Path: /ffzdqu-61/obj33 2023-09-22T21:59:10.671 DEBUG:boto:Data: 2023-09-22T21:59:10.671 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.671 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.671 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.671 DEBUG:boto:Params: {} 2023-09-22T21:59:10.671 DEBUG:boto:Token: None 2023-09-22T21:59:10.671 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj33 2023-09-22T21:59:10.672 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:m7FDZJIFq3tmRdITacljAfAMaA8= 2023-09-22T21:59:10.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:m7FDZJIFq3tmRdITacljAfAMaA8=', 'Content-Length': '0'} 2023-09-22T21:59:10.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077280e1ff0d037b7-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T21:59:10.675 DEBUG:boto:path=/ffzdqu-61/obj33 2023-09-22T21:59:10.675 DEBUG:boto:auth_path=/ffzdqu-61/obj33 2023-09-22T21:59:10.675 DEBUG:boto:Method: GET 2023-09-22T21:59:10.675 DEBUG:boto:Path: /ffzdqu-61/obj33 2023-09-22T21:59:10.675 DEBUG:boto:Data: 2023-09-22T21:59:10.675 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.675 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.676 DEBUG:boto:Params: {} 2023-09-22T21:59:10.676 DEBUG:boto:Token: None 2023-09-22T21:59:10.676 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj33 2023-09-22T21:59:10.676 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+dKk3iryI7ttEwZnTfofPdgxb+o= 2023-09-22T21:59:10.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+dKk3iryI7ttEwZnTfofPdgxb+o=', 'Content-Length': '0'} 2023-09-22T21:59:10.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d971a6e2034f1ea-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.722 DEBUG:boto:path=/ffzdqu-61/obj33 2023-09-22T21:59:10.722 DEBUG:boto:auth_path=/ffzdqu-61/obj33 2023-09-22T21:59:10.723 DEBUG:boto:Method: GET 2023-09-22T21:59:10.723 DEBUG:boto:Path: /ffzdqu-61/obj33 2023-09-22T21:59:10.723 DEBUG:boto:Data: 2023-09-22T21:59:10.723 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.723 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.723 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.723 DEBUG:boto:Params: {} 2023-09-22T21:59:10.723 DEBUG:boto:Token: None 2023-09-22T21:59:10.724 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj33 2023-09-22T21:59:10.724 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+dKk3iryI7ttEwZnTfofPdgxb+o= 2023-09-22T21:59:10.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+dKk3iryI7ttEwZnTfofPdgxb+o=', 'Content-Length': '0'} 2023-09-22T21:59:10.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007141623fb22ea144-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.727 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:10.728 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:10.728 DEBUG:boto:path=/ffzdqu-61/?prefix=obj33 2023-09-22T21:59:10.728 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj33 2023-09-22T21:59:10.728 DEBUG:boto:Method: GET 2023-09-22T21:59:10.728 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj33 2023-09-22T21:59:10.728 DEBUG:boto:Data: 2023-09-22T21:59:10.728 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.728 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.728 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.729 DEBUG:boto:Params: {} 2023-09-22T21:59:10.729 DEBUG:boto:Token: None 2023-09-22T21:59:10.729 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/ 2023-09-22T21:59:10.729 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ= 2023-09-22T21:59:10.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ=', 'Content-Length': '0'} 2023-09-22T21:59:10.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028b71b5877ecde24-00650e0e2e-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.735 DEBUG:boto:b'ffzdqu-61obj331000falseobj332023-09-22T21:56:08.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:10.735 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:10.735 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:10.735 DEBUG:boto:path=/ffzdqu-61/?prefix=obj33 2023-09-22T21:59:10.735 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj33 2023-09-22T21:59:10.735 DEBUG:boto:Method: GET 2023-09-22T21:59:10.736 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj33 2023-09-22T21:59:10.736 DEBUG:boto:Data: 2023-09-22T21:59:10.736 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.736 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.736 DEBUG:boto:Params: {} 2023-09-22T21:59:10.736 DEBUG:boto:Token: None 2023-09-22T21:59:10.736 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/ 2023-09-22T21:59:10.737 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ= 2023-09-22T21:59:10.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ=', 'Content-Length': '0'} 2023-09-22T21:59:10.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002491272ed902c7be-00650e0e2e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.790 DEBUG:boto:b'ffzdqu-61obj331000falseobj332023-09-22T21:56:08.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:10.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:59:10.791 DEBUG:boto:path=/ffzdqu-61/obj34 2023-09-22T21:59:10.791 DEBUG:boto:auth_path=/ffzdqu-61/obj34 2023-09-22T21:59:10.791 DEBUG:boto:path=/ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.791 DEBUG:boto:auth_path=/ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.791 DEBUG:boto:Method: GET 2023-09-22T21:59:10.792 DEBUG:boto:Path: /ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.792 DEBUG:boto:Data: 2023-09-22T21:59:10.792 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.792 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.792 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.792 DEBUG:boto:Params: {} 2023-09-22T21:59:10.792 DEBUG:boto:Token: None 2023-09-22T21:59:10.793 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.793 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y0tw0WIUgGHzDvjD7fUF1B6JA5U= 2023-09-22T21:59:10.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y0tw0WIUgGHzDvjD7fUF1B6JA5U=', 'Content-Length': '0'} 2023-09-22T21:59:10.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005126e51a5c7c22ff-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.796 DEBUG:boto:path=/ffzdqu-61/obj34 2023-09-22T21:59:10.797 DEBUG:boto:auth_path=/ffzdqu-61/obj34 2023-09-22T21:59:10.797 DEBUG:boto:path=/ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.797 DEBUG:boto:auth_path=/ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.797 DEBUG:boto:Method: GET 2023-09-22T21:59:10.797 DEBUG:boto:Path: /ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.797 DEBUG:boto:Data: 2023-09-22T21:59:10.797 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.797 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.797 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.798 DEBUG:boto:Params: {} 2023-09-22T21:59:10.798 DEBUG:boto:Token: None 2023-09-22T21:59:10.798 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.798 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y0tw0WIUgGHzDvjD7fUF1B6JA5U= 2023-09-22T21:59:10.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y0tw0WIUgGHzDvjD7fUF1B6JA5U=', 'Content-Length': '0'} 2023-09-22T21:59:10.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086a60a6e6357a373-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.801 DEBUG:boto:path=/ffzdqu-61/obj34 2023-09-22T21:59:10.801 DEBUG:boto:auth_path=/ffzdqu-61/obj34 2023-09-22T21:59:10.802 DEBUG:boto:path=/ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.802 DEBUG:boto:auth_path=/ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.802 DEBUG:boto:Method: HEAD 2023-09-22T21:59:10.802 DEBUG:boto:Path: /ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.802 DEBUG:boto:Data: 2023-09-22T21:59:10.802 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.802 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.802 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.803 DEBUG:boto:Params: {} 2023-09-22T21:59:10.803 DEBUG:boto:Token: None 2023-09-22T21:59:10.803 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.803 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8kCn5qO6R2M84z4WkBeAyyArfEU= 2023-09-22T21:59:10.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8kCn5qO6R2M84z4WkBeAyyArfEU=', 'Content-Length': '0'} 2023-09-22T21:59:10.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000341aea9f55d82959-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.806 DEBUG:boto:path=/ffzdqu-61/obj34 2023-09-22T21:59:10.807 DEBUG:boto:auth_path=/ffzdqu-61/obj34 2023-09-22T21:59:10.807 DEBUG:boto:path=/ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.807 DEBUG:boto:auth_path=/ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.807 DEBUG:boto:Method: HEAD 2023-09-22T21:59:10.807 DEBUG:boto:Path: /ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.807 DEBUG:boto:Data: 2023-09-22T21:59:10.807 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.807 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.807 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.808 DEBUG:boto:Params: {} 2023-09-22T21:59:10.808 DEBUG:boto:Token: None 2023-09-22T21:59:10.808 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.808 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8kCn5qO6R2M84z4WkBeAyyArfEU= 2023-09-22T21:59:10.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8kCn5qO6R2M84z4WkBeAyyArfEU=', 'Content-Length': '0'} 2023-09-22T21:59:10.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe05701835f2c747-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:59:10.814 DEBUG:boto:path=/ffzdqu-61/obj34 2023-09-22T21:59:10.814 DEBUG:boto:auth_path=/ffzdqu-61/obj34 2023-09-22T21:59:10.814 DEBUG:boto:path=/ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.814 DEBUG:boto:auth_path=/ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.814 DEBUG:boto:Method: GET 2023-09-22T21:59:10.814 DEBUG:boto:Path: /ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.815 DEBUG:boto:Data: 2023-09-22T21:59:10.815 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.815 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.815 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.815 DEBUG:boto:Params: {} 2023-09-22T21:59:10.815 DEBUG:boto:Token: None 2023-09-22T21:59:10.815 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.815 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y0tw0WIUgGHzDvjD7fUF1B6JA5U= 2023-09-22T21:59:10.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y0tw0WIUgGHzDvjD7fUF1B6JA5U=', 'Content-Length': '0'} 2023-09-22T21:59:10.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001267809f48bd2122-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.819 DEBUG:boto:path=/ffzdqu-61/obj34 2023-09-22T21:59:10.819 DEBUG:boto:auth_path=/ffzdqu-61/obj34 2023-09-22T21:59:10.819 DEBUG:boto:path=/ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.819 DEBUG:boto:auth_path=/ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.819 DEBUG:boto:Method: GET 2023-09-22T21:59:10.819 DEBUG:boto:Path: /ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.820 DEBUG:boto:Data: 2023-09-22T21:59:10.820 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.820 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.820 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.820 DEBUG:boto:Params: {} 2023-09-22T21:59:10.820 DEBUG:boto:Token: None 2023-09-22T21:59:10.820 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj34?versionId=null 2023-09-22T21:59:10.821 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y0tw0WIUgGHzDvjD7fUF1B6JA5U= 2023-09-22T21:59:10.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y0tw0WIUgGHzDvjD7fUF1B6JA5U=', 'Content-Length': '0'} 2023-09-22T21:59:10.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002994bed422a6533a-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.827 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:10.827 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:10.827 DEBUG:boto:path=/ffzdqu-61/?prefix=obj34 2023-09-22T21:59:10.828 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj34 2023-09-22T21:59:10.828 DEBUG:boto:Method: GET 2023-09-22T21:59:10.828 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj34 2023-09-22T21:59:10.828 DEBUG:boto:Data: 2023-09-22T21:59:10.828 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.828 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.828 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.828 DEBUG:boto:Params: {} 2023-09-22T21:59:10.829 DEBUG:boto:Token: None 2023-09-22T21:59:10.829 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/ 2023-09-22T21:59:10.829 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ= 2023-09-22T21:59:10.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ=', 'Content-Length': '0'} 2023-09-22T21:59:10.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002a40701b3000f4a-00650e0e2e-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.874 DEBUG:boto:b'ffzdqu-61obj341000falseobj342023-09-22T21:56:08.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:10.875 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:10.875 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:10.875 DEBUG:boto:path=/ffzdqu-61/?prefix=obj34 2023-09-22T21:59:10.875 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj34 2023-09-22T21:59:10.875 DEBUG:boto:Method: GET 2023-09-22T21:59:10.875 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj34 2023-09-22T21:59:10.875 DEBUG:boto:Data: 2023-09-22T21:59:10.876 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.876 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.876 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.876 DEBUG:boto:Params: {} 2023-09-22T21:59:10.876 DEBUG:boto:Token: None 2023-09-22T21:59:10.876 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/ 2023-09-22T21:59:10.876 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ= 2023-09-22T21:59:10.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ=', 'Content-Length': '0'} 2023-09-22T21:59:10.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099f2820418a44c41-00650e0e2e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.894 DEBUG:boto:b'ffzdqu-61obj341000falseobj342023-09-22T21:56:08.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:10.894 DEBUG:boto:path=/ffzdqu-61/obj34 2023-09-22T21:59:10.894 DEBUG:boto:auth_path=/ffzdqu-61/obj34 2023-09-22T21:59:10.894 DEBUG:boto:Method: HEAD 2023-09-22T21:59:10.895 DEBUG:boto:Path: /ffzdqu-61/obj34 2023-09-22T21:59:10.895 DEBUG:boto:Data: 2023-09-22T21:59:10.895 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.895 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.895 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.896 DEBUG:boto:Params: {} 2023-09-22T21:59:10.896 DEBUG:boto:Token: None 2023-09-22T21:59:10.896 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj34 2023-09-22T21:59:10.896 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1iFpW/mPbqPryrHgvqHxmAX0GAc= 2023-09-22T21:59:10.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1iFpW/mPbqPryrHgvqHxmAX0GAc=', 'Content-Length': '0'} 2023-09-22T21:59:10.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdb80db1063857f4-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.900 DEBUG:boto:path=/ffzdqu-61/obj34 2023-09-22T21:59:10.900 DEBUG:boto:auth_path=/ffzdqu-61/obj34 2023-09-22T21:59:10.900 DEBUG:boto:Method: HEAD 2023-09-22T21:59:10.900 DEBUG:boto:Path: /ffzdqu-61/obj34 2023-09-22T21:59:10.900 DEBUG:boto:Data: 2023-09-22T21:59:10.900 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.901 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.901 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.901 DEBUG:boto:Params: {} 2023-09-22T21:59:10.901 DEBUG:boto:Token: None 2023-09-22T21:59:10.901 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj34 2023-09-22T21:59:10.901 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1iFpW/mPbqPryrHgvqHxmAX0GAc= 2023-09-22T21:59:10.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1iFpW/mPbqPryrHgvqHxmAX0GAc=', 'Content-Length': '0'} 2023-09-22T21:59:10.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099c12386c704c347-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T21:59:10.905 DEBUG:boto:path=/ffzdqu-61/obj34 2023-09-22T21:59:10.905 DEBUG:boto:auth_path=/ffzdqu-61/obj34 2023-09-22T21:59:10.905 DEBUG:boto:Method: GET 2023-09-22T21:59:10.906 DEBUG:boto:Path: /ffzdqu-61/obj34 2023-09-22T21:59:10.906 DEBUG:boto:Data: 2023-09-22T21:59:10.906 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.906 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.906 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.906 DEBUG:boto:Params: {} 2023-09-22T21:59:10.906 DEBUG:boto:Token: None 2023-09-22T21:59:10.906 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj34 2023-09-22T21:59:10.907 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IFilpLG03yFKVGgv4NIdzMShCS8= 2023-09-22T21:59:10.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IFilpLG03yFKVGgv4NIdzMShCS8=', 'Content-Length': '0'} 2023-09-22T21:59:10.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f14bc11e83bc37f5-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.911 DEBUG:boto:path=/ffzdqu-61/obj34 2023-09-22T21:59:10.911 DEBUG:boto:auth_path=/ffzdqu-61/obj34 2023-09-22T21:59:10.911 DEBUG:boto:Method: GET 2023-09-22T21:59:10.911 DEBUG:boto:Path: /ffzdqu-61/obj34 2023-09-22T21:59:10.912 DEBUG:boto:Data: 2023-09-22T21:59:10.912 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.912 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.912 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.912 DEBUG:boto:Params: {} 2023-09-22T21:59:10.912 DEBUG:boto:Token: None 2023-09-22T21:59:10.912 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj34 2023-09-22T21:59:10.912 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IFilpLG03yFKVGgv4NIdzMShCS8= 2023-09-22T21:59:10.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IFilpLG03yFKVGgv4NIdzMShCS8=', 'Content-Length': '0'} 2023-09-22T21:59:10.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000334d1720c71f4a75-00650e0e2e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.958 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:10.959 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:10.959 DEBUG:boto:path=/ffzdqu-61/?prefix=obj34 2023-09-22T21:59:10.959 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj34 2023-09-22T21:59:10.959 DEBUG:boto:Method: GET 2023-09-22T21:59:10.959 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj34 2023-09-22T21:59:10.959 DEBUG:boto:Data: 2023-09-22T21:59:10.959 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.960 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.960 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.960 DEBUG:boto:Params: {} 2023-09-22T21:59:10.960 DEBUG:boto:Token: None 2023-09-22T21:59:10.960 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/ 2023-09-22T21:59:10.960 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ= 2023-09-22T21:59:10.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ=', 'Content-Length': '0'} 2023-09-22T21:59:10.965 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038a587cd9458274d-00650e0e2e-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.965 DEBUG:boto:b'ffzdqu-61obj341000falseobj342023-09-22T21:56:08.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:10.966 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:10.966 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:10.966 DEBUG:boto:path=/ffzdqu-61/?prefix=obj34 2023-09-22T21:59:10.966 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj34 2023-09-22T21:59:10.966 DEBUG:boto:Method: GET 2023-09-22T21:59:10.967 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj34 2023-09-22T21:59:10.967 DEBUG:boto:Data: 2023-09-22T21:59:10.967 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.967 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:10.967 DEBUG:boto:Port: 8000 2023-09-22T21:59:10.967 DEBUG:boto:Params: {} 2023-09-22T21:59:10.967 DEBUG:boto:Token: None 2023-09-22T21:59:10.967 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/ 2023-09-22T21:59:10.968 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ= 2023-09-22T21:59:10.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8YSnxv+B743iqnmMbE3GleCWhgQ=', 'Content-Length': '0'} 2023-09-22T21:59:10.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c05d7a5b70bbd62-00650e0e2e-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:10.978 DEBUG:boto:b'ffzdqu-61obj341000falseobj342023-09-22T21:56:08.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:10.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-09-22T21:59:10.978 DEBUG:boto:path=/ffzdqu-61/obj37 2023-09-22T21:59:10.979 DEBUG:boto:auth_path=/ffzdqu-61/obj37 2023-09-22T21:59:10.979 DEBUG:boto:path=/ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:10.979 DEBUG:boto:auth_path=/ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:10.979 DEBUG:boto:Method: GET 2023-09-22T21:59:10.979 DEBUG:boto:Path: /ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:10.979 DEBUG:boto:Data: 2023-09-22T21:59:10.979 DEBUG:boto:Headers: {} 2023-09-22T21:59:10.979 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:10.980 DEBUG:boto:Port: 8001 2023-09-22T21:59:10.980 DEBUG:boto:Params: {} 2023-09-22T21:59:10.980 DEBUG:boto:Token: None 2023-09-22T21:59:10.980 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:10 GMT /ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:10.980 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EJvuhZ9csWCmM1t5uJVRKb4EZYY= 2023-09-22T21:59:10.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:10 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EJvuhZ9csWCmM1t5uJVRKb4EZYY=', 'Content-Length': '0'} 2023-09-22T21:59:10.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099487291e1943cf9-00650e0e2e-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.026 DEBUG:boto:path=/ffzdqu-61/obj37 2023-09-22T21:59:11.026 DEBUG:boto:auth_path=/ffzdqu-61/obj37 2023-09-22T21:59:11.027 DEBUG:boto:path=/ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:11.027 DEBUG:boto:auth_path=/ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:11.027 DEBUG:boto:Method: GET 2023-09-22T21:59:11.027 DEBUG:boto:Path: /ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:11.027 DEBUG:boto:Data: 2023-09-22T21:59:11.027 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.027 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.027 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.027 DEBUG:boto:Params: {} 2023-09-22T21:59:11.028 DEBUG:boto:Token: None 2023-09-22T21:59:11.028 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:11.028 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KtQK2mTziQ8OYgci+/t0dbbk1fo= 2023-09-22T21:59:11.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KtQK2mTziQ8OYgci+/t0dbbk1fo=', 'Content-Length': '0'} 2023-09-22T21:59:11.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c69ce19bb0efb70-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.031 DEBUG:boto:path=/ffzdqu-61/obj37 2023-09-22T21:59:11.031 DEBUG:boto:auth_path=/ffzdqu-61/obj37 2023-09-22T21:59:11.031 DEBUG:boto:path=/ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:11.031 DEBUG:boto:auth_path=/ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:11.032 DEBUG:boto:Method: HEAD 2023-09-22T21:59:11.032 DEBUG:boto:Path: /ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:11.032 DEBUG:boto:Data: 2023-09-22T21:59:11.032 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.032 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.032 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.032 DEBUG:boto:Params: {} 2023-09-22T21:59:11.032 DEBUG:boto:Token: None 2023-09-22T21:59:11.033 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:11.033 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0SpmoUQZo10v4NXTwFK46dAdBCY= 2023-09-22T21:59:11.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0SpmoUQZo10v4NXTwFK46dAdBCY=', 'Content-Length': '0'} 2023-09-22T21:59:11.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084df12b30f0f5493-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.036 DEBUG:boto:path=/ffzdqu-61/obj37 2023-09-22T21:59:11.036 DEBUG:boto:auth_path=/ffzdqu-61/obj37 2023-09-22T21:59:11.036 DEBUG:boto:path=/ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:11.036 DEBUG:boto:auth_path=/ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:11.036 DEBUG:boto:Method: HEAD 2023-09-22T21:59:11.037 DEBUG:boto:Path: /ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:11.037 DEBUG:boto:Data: 2023-09-22T21:59:11.037 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.037 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.037 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.037 DEBUG:boto:Params: {} 2023-09-22T21:59:11.037 DEBUG:boto:Token: None 2023-09-22T21:59:11.037 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:11.038 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0SpmoUQZo10v4NXTwFK46dAdBCY= 2023-09-22T21:59:11.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0SpmoUQZo10v4NXTwFK46dAdBCY=', 'Content-Length': '0'} 2023-09-22T21:59:11.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010798249f0cbd9f0-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-09-22T21:59:11.041 DEBUG:boto:path=/ffzdqu-61/obj37 2023-09-22T21:59:11.041 DEBUG:boto:auth_path=/ffzdqu-61/obj37 2023-09-22T21:59:11.041 DEBUG:boto:path=/ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:11.041 DEBUG:boto:auth_path=/ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:11.041 DEBUG:boto:Method: GET 2023-09-22T21:59:11.041 DEBUG:boto:Path: /ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:11.042 DEBUG:boto:Data: 2023-09-22T21:59:11.042 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.042 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.042 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.042 DEBUG:boto:Params: {} 2023-09-22T21:59:11.042 DEBUG:boto:Token: None 2023-09-22T21:59:11.042 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:11.042 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KtQK2mTziQ8OYgci+/t0dbbk1fo= 2023-09-22T21:59:11.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KtQK2mTziQ8OYgci+/t0dbbk1fo=', 'Content-Length': '0'} 2023-09-22T21:59:11.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080e051790a8ac7b6-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.046 DEBUG:boto:path=/ffzdqu-61/obj37 2023-09-22T21:59:11.046 DEBUG:boto:auth_path=/ffzdqu-61/obj37 2023-09-22T21:59:11.046 DEBUG:boto:path=/ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:11.046 DEBUG:boto:auth_path=/ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:11.046 DEBUG:boto:Method: GET 2023-09-22T21:59:11.047 DEBUG:boto:Path: /ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:11.047 DEBUG:boto:Data: 2023-09-22T21:59:11.047 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.047 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.047 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.047 DEBUG:boto:Params: {} 2023-09-22T21:59:11.047 DEBUG:boto:Token: None 2023-09-22T21:59:11.047 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj37?versionId=null 2023-09-22T21:59:11.048 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KtQK2mTziQ8OYgci+/t0dbbk1fo= 2023-09-22T21:59:11.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KtQK2mTziQ8OYgci+/t0dbbk1fo=', 'Content-Length': '0'} 2023-09-22T21:59:11.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021a1ac4073e1304a-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.051 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:11.051 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:11.051 DEBUG:boto:path=/ffzdqu-61/?prefix=obj37 2023-09-22T21:59:11.051 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj37 2023-09-22T21:59:11.051 DEBUG:boto:Method: GET 2023-09-22T21:59:11.051 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj37 2023-09-22T21:59:11.051 DEBUG:boto:Data: 2023-09-22T21:59:11.051 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.052 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.052 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.052 DEBUG:boto:Params: {} 2023-09-22T21:59:11.052 DEBUG:boto:Token: None 2023-09-22T21:59:11.052 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/ 2023-09-22T21:59:11.052 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10= 2023-09-22T21:59:11.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10=', 'Content-Length': '0'} 2023-09-22T21:59:11.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fe7080011e1046e-00650e0e2f-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.057 DEBUG:boto:b'ffzdqu-61obj371000falseobj372023-09-22T21:56:10.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:11.058 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:11.058 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:11.058 DEBUG:boto:path=/ffzdqu-61/?prefix=obj37 2023-09-22T21:59:11.058 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj37 2023-09-22T21:59:11.058 DEBUG:boto:Method: GET 2023-09-22T21:59:11.058 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj37 2023-09-22T21:59:11.058 DEBUG:boto:Data: 2023-09-22T21:59:11.059 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.059 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.059 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.059 DEBUG:boto:Params: {} 2023-09-22T21:59:11.059 DEBUG:boto:Token: None 2023-09-22T21:59:11.059 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/ 2023-09-22T21:59:11.059 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10= 2023-09-22T21:59:11.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10=', 'Content-Length': '0'} 2023-09-22T21:59:11.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e765484237446890-00650e0e2f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.110 DEBUG:boto:b'ffzdqu-61obj371000falseobj372023-09-22T21:56:10.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:11.111 DEBUG:boto:path=/ffzdqu-61/obj37 2023-09-22T21:59:11.111 DEBUG:boto:auth_path=/ffzdqu-61/obj37 2023-09-22T21:59:11.111 DEBUG:boto:Method: HEAD 2023-09-22T21:59:11.111 DEBUG:boto:Path: /ffzdqu-61/obj37 2023-09-22T21:59:11.111 DEBUG:boto:Data: 2023-09-22T21:59:11.111 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.111 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.112 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.112 DEBUG:boto:Params: {} 2023-09-22T21:59:11.112 DEBUG:boto:Token: None 2023-09-22T21:59:11.112 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj37 2023-09-22T21:59:11.112 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WX/JBiEGDL5BEvDqaP40EXqwMjg= 2023-09-22T21:59:11.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WX/JBiEGDL5BEvDqaP40EXqwMjg=', 'Content-Length': '0'} 2023-09-22T21:59:11.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a425cef8cbecee8-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.115 DEBUG:boto:path=/ffzdqu-61/obj37 2023-09-22T21:59:11.115 DEBUG:boto:auth_path=/ffzdqu-61/obj37 2023-09-22T21:59:11.116 DEBUG:boto:Method: HEAD 2023-09-22T21:59:11.116 DEBUG:boto:Path: /ffzdqu-61/obj37 2023-09-22T21:59:11.116 DEBUG:boto:Data: 2023-09-22T21:59:11.116 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.116 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.116 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.116 DEBUG:boto:Params: {} 2023-09-22T21:59:11.116 DEBUG:boto:Token: None 2023-09-22T21:59:11.117 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj37 2023-09-22T21:59:11.117 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WX/JBiEGDL5BEvDqaP40EXqwMjg= 2023-09-22T21:59:11.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WX/JBiEGDL5BEvDqaP40EXqwMjg=', 'Content-Length': '0'} 2023-09-22T21:59:11.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b10e128f61938ec8-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-09-22T21:59:11.120 DEBUG:boto:path=/ffzdqu-61/obj37 2023-09-22T21:59:11.120 DEBUG:boto:auth_path=/ffzdqu-61/obj37 2023-09-22T21:59:11.120 DEBUG:boto:Method: GET 2023-09-22T21:59:11.120 DEBUG:boto:Path: /ffzdqu-61/obj37 2023-09-22T21:59:11.120 DEBUG:boto:Data: 2023-09-22T21:59:11.121 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.121 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.121 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.121 DEBUG:boto:Params: {} 2023-09-22T21:59:11.121 DEBUG:boto:Token: None 2023-09-22T21:59:11.121 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj37 2023-09-22T21:59:11.121 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sLQkxUcwhDFiKLGXjaur0l/KOsQ= 2023-09-22T21:59:11.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sLQkxUcwhDFiKLGXjaur0l/KOsQ=', 'Content-Length': '0'} 2023-09-22T21:59:11.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038afa63c44d7c06e-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.166 DEBUG:boto:path=/ffzdqu-61/obj37 2023-09-22T21:59:11.166 DEBUG:boto:auth_path=/ffzdqu-61/obj37 2023-09-22T21:59:11.167 DEBUG:boto:Method: GET 2023-09-22T21:59:11.167 DEBUG:boto:Path: /ffzdqu-61/obj37 2023-09-22T21:59:11.167 DEBUG:boto:Data: 2023-09-22T21:59:11.167 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.167 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.167 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.168 DEBUG:boto:Params: {} 2023-09-22T21:59:11.168 DEBUG:boto:Token: None 2023-09-22T21:59:11.168 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj37 2023-09-22T21:59:11.168 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sLQkxUcwhDFiKLGXjaur0l/KOsQ= 2023-09-22T21:59:11.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sLQkxUcwhDFiKLGXjaur0l/KOsQ=', 'Content-Length': '0'} 2023-09-22T21:59:11.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004266859327fe2032-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.172 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:11.172 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:11.172 DEBUG:boto:path=/ffzdqu-61/?prefix=obj37 2023-09-22T21:59:11.172 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj37 2023-09-22T21:59:11.172 DEBUG:boto:Method: GET 2023-09-22T21:59:11.172 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj37 2023-09-22T21:59:11.173 DEBUG:boto:Data: 2023-09-22T21:59:11.173 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.173 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.173 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.173 DEBUG:boto:Params: {} 2023-09-22T21:59:11.173 DEBUG:boto:Token: None 2023-09-22T21:59:11.174 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/ 2023-09-22T21:59:11.174 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10= 2023-09-22T21:59:11.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10=', 'Content-Length': '0'} 2023-09-22T21:59:11.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5ad3e79cb0f96df-00650e0e2f-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.179 DEBUG:boto:b'ffzdqu-61obj371000falseobj372023-09-22T21:56:10.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:11.179 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:11.179 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:11.180 DEBUG:boto:path=/ffzdqu-61/?prefix=obj37 2023-09-22T21:59:11.180 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj37 2023-09-22T21:59:11.180 DEBUG:boto:Method: GET 2023-09-22T21:59:11.180 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj37 2023-09-22T21:59:11.180 DEBUG:boto:Data: 2023-09-22T21:59:11.180 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.181 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.181 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.181 DEBUG:boto:Params: {} 2023-09-22T21:59:11.181 DEBUG:boto:Token: None 2023-09-22T21:59:11.181 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/ 2023-09-22T21:59:11.181 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10= 2023-09-22T21:59:11.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10=', 'Content-Length': '0'} 2023-09-22T21:59:11.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000253a1f6580dead52-00650e0e2f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.193 DEBUG:boto:b'ffzdqu-61obj371000falseobj372023-09-22T21:56:10.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:11.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T21:59:11.194 DEBUG:boto:path=/ffzdqu-61/obj38 2023-09-22T21:59:11.194 DEBUG:boto:auth_path=/ffzdqu-61/obj38 2023-09-22T21:59:11.194 DEBUG:boto:path=/ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.194 DEBUG:boto:auth_path=/ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.195 DEBUG:boto:Method: GET 2023-09-22T21:59:11.195 DEBUG:boto:Path: /ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.195 DEBUG:boto:Data: 2023-09-22T21:59:11.195 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.195 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.195 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.196 DEBUG:boto:Params: {} 2023-09-22T21:59:11.196 DEBUG:boto:Token: None 2023-09-22T21:59:11.196 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.196 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xiNdHyHA3pNGtDF5I2IxAPb+wJQ= 2023-09-22T21:59:11.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xiNdHyHA3pNGtDF5I2IxAPb+wJQ=', 'Content-Length': '0'} 2023-09-22T21:59:11.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddd58c2d667941af-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.199 DEBUG:boto:path=/ffzdqu-61/obj38 2023-09-22T21:59:11.200 DEBUG:boto:auth_path=/ffzdqu-61/obj38 2023-09-22T21:59:11.200 DEBUG:boto:path=/ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.200 DEBUG:boto:auth_path=/ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.200 DEBUG:boto:Method: GET 2023-09-22T21:59:11.200 DEBUG:boto:Path: /ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.200 DEBUG:boto:Data: 2023-09-22T21:59:11.200 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.201 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.201 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.201 DEBUG:boto:Params: {} 2023-09-22T21:59:11.201 DEBUG:boto:Token: None 2023-09-22T21:59:11.201 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.201 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xiNdHyHA3pNGtDF5I2IxAPb+wJQ= 2023-09-22T21:59:11.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xiNdHyHA3pNGtDF5I2IxAPb+wJQ=', 'Content-Length': '0'} 2023-09-22T21:59:11.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000875cc7ee9b232cd4-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.246 DEBUG:boto:path=/ffzdqu-61/obj38 2023-09-22T21:59:11.247 DEBUG:boto:auth_path=/ffzdqu-61/obj38 2023-09-22T21:59:11.247 DEBUG:boto:path=/ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.247 DEBUG:boto:auth_path=/ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.247 DEBUG:boto:Method: HEAD 2023-09-22T21:59:11.247 DEBUG:boto:Path: /ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.247 DEBUG:boto:Data: 2023-09-22T21:59:11.247 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.248 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.248 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.248 DEBUG:boto:Params: {} 2023-09-22T21:59:11.248 DEBUG:boto:Token: None 2023-09-22T21:59:11.248 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.248 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UcsOFQrQ39wDujfEludaim4laPw= 2023-09-22T21:59:11.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UcsOFQrQ39wDujfEludaim4laPw=', 'Content-Length': '0'} 2023-09-22T21:59:11.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041e108543250ae73-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.252 DEBUG:boto:path=/ffzdqu-61/obj38 2023-09-22T21:59:11.252 DEBUG:boto:auth_path=/ffzdqu-61/obj38 2023-09-22T21:59:11.253 DEBUG:boto:path=/ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.253 DEBUG:boto:auth_path=/ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.253 DEBUG:boto:Method: HEAD 2023-09-22T21:59:11.253 DEBUG:boto:Path: /ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.253 DEBUG:boto:Data: 2023-09-22T21:59:11.253 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.253 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.253 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.254 DEBUG:boto:Params: {} 2023-09-22T21:59:11.254 DEBUG:boto:Token: None 2023-09-22T21:59:11.254 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.254 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UcsOFQrQ39wDujfEludaim4laPw= 2023-09-22T21:59:11.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UcsOFQrQ39wDujfEludaim4laPw=', 'Content-Length': '0'} 2023-09-22T21:59:11.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a63947435c0f44e-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T21:59:11.258 DEBUG:boto:path=/ffzdqu-61/obj38 2023-09-22T21:59:11.258 DEBUG:boto:auth_path=/ffzdqu-61/obj38 2023-09-22T21:59:11.258 DEBUG:boto:path=/ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.258 DEBUG:boto:auth_path=/ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.258 DEBUG:boto:Method: GET 2023-09-22T21:59:11.258 DEBUG:boto:Path: /ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.258 DEBUG:boto:Data: 2023-09-22T21:59:11.259 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.259 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.259 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.259 DEBUG:boto:Params: {} 2023-09-22T21:59:11.259 DEBUG:boto:Token: None 2023-09-22T21:59:11.259 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.259 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xiNdHyHA3pNGtDF5I2IxAPb+wJQ= 2023-09-22T21:59:11.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xiNdHyHA3pNGtDF5I2IxAPb+wJQ=', 'Content-Length': '0'} 2023-09-22T21:59:11.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000628fce08140153a7-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.302 DEBUG:boto:path=/ffzdqu-61/obj38 2023-09-22T21:59:11.303 DEBUG:boto:auth_path=/ffzdqu-61/obj38 2023-09-22T21:59:11.303 DEBUG:boto:path=/ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.303 DEBUG:boto:auth_path=/ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.303 DEBUG:boto:Method: GET 2023-09-22T21:59:11.303 DEBUG:boto:Path: /ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.303 DEBUG:boto:Data: 2023-09-22T21:59:11.303 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.303 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.304 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.304 DEBUG:boto:Params: {} 2023-09-22T21:59:11.304 DEBUG:boto:Token: None 2023-09-22T21:59:11.304 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj38?versionId=null 2023-09-22T21:59:11.304 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xiNdHyHA3pNGtDF5I2IxAPb+wJQ= 2023-09-22T21:59:11.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xiNdHyHA3pNGtDF5I2IxAPb+wJQ=', 'Content-Length': '0'} 2023-09-22T21:59:11.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1a3f742a0ef33b6-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.308 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:11.308 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:11.308 DEBUG:boto:path=/ffzdqu-61/?prefix=obj38 2023-09-22T21:59:11.308 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj38 2023-09-22T21:59:11.308 DEBUG:boto:Method: GET 2023-09-22T21:59:11.308 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj38 2023-09-22T21:59:11.308 DEBUG:boto:Data: 2023-09-22T21:59:11.308 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.309 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.309 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.309 DEBUG:boto:Params: {} 2023-09-22T21:59:11.309 DEBUG:boto:Token: None 2023-09-22T21:59:11.309 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/ 2023-09-22T21:59:11.309 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10= 2023-09-22T21:59:11.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10=', 'Content-Length': '0'} 2023-09-22T21:59:11.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec51422264588a02-00650e0e2f-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.314 DEBUG:boto:b'ffzdqu-61obj381000falseobj382023-09-22T21:56:09.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:11.315 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:11.315 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:11.315 DEBUG:boto:path=/ffzdqu-61/?prefix=obj38 2023-09-22T21:59:11.315 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj38 2023-09-22T21:59:11.315 DEBUG:boto:Method: GET 2023-09-22T21:59:11.316 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj38 2023-09-22T21:59:11.316 DEBUG:boto:Data: 2023-09-22T21:59:11.316 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.316 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.316 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.316 DEBUG:boto:Params: {} 2023-09-22T21:59:11.316 DEBUG:boto:Token: None 2023-09-22T21:59:11.317 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/ 2023-09-22T21:59:11.317 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10= 2023-09-22T21:59:11.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10=', 'Content-Length': '0'} 2023-09-22T21:59:11.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4a52826bb4d5d5a-00650e0e2f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.327 DEBUG:boto:b'ffzdqu-61obj381000falseobj382023-09-22T21:56:09.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:11.327 DEBUG:boto:path=/ffzdqu-61/obj38 2023-09-22T21:59:11.327 DEBUG:boto:auth_path=/ffzdqu-61/obj38 2023-09-22T21:59:11.327 DEBUG:boto:Method: HEAD 2023-09-22T21:59:11.327 DEBUG:boto:Path: /ffzdqu-61/obj38 2023-09-22T21:59:11.328 DEBUG:boto:Data: 2023-09-22T21:59:11.328 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.328 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.328 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.328 DEBUG:boto:Params: {} 2023-09-22T21:59:11.328 DEBUG:boto:Token: None 2023-09-22T21:59:11.329 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj38 2023-09-22T21:59:11.329 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ctkREUVnugo0KQ7DVD+nMR4YZ+c= 2023-09-22T21:59:11.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ctkREUVnugo0KQ7DVD+nMR4YZ+c=', 'Content-Length': '0'} 2023-09-22T21:59:11.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a982e5ec7b8136ed-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.332 DEBUG:boto:path=/ffzdqu-61/obj38 2023-09-22T21:59:11.332 DEBUG:boto:auth_path=/ffzdqu-61/obj38 2023-09-22T21:59:11.332 DEBUG:boto:Method: HEAD 2023-09-22T21:59:11.332 DEBUG:boto:Path: /ffzdqu-61/obj38 2023-09-22T21:59:11.332 DEBUG:boto:Data: 2023-09-22T21:59:11.332 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.333 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.333 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.333 DEBUG:boto:Params: {} 2023-09-22T21:59:11.333 DEBUG:boto:Token: None 2023-09-22T21:59:11.333 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj38 2023-09-22T21:59:11.333 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ctkREUVnugo0KQ7DVD+nMR4YZ+c= 2023-09-22T21:59:11.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ctkREUVnugo0KQ7DVD+nMR4YZ+c=', 'Content-Length': '0'} 2023-09-22T21:59:11.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c55f4aa0d0e12a1-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T21:59:11.336 DEBUG:boto:path=/ffzdqu-61/obj38 2023-09-22T21:59:11.336 DEBUG:boto:auth_path=/ffzdqu-61/obj38 2023-09-22T21:59:11.337 DEBUG:boto:Method: GET 2023-09-22T21:59:11.337 DEBUG:boto:Path: /ffzdqu-61/obj38 2023-09-22T21:59:11.337 DEBUG:boto:Data: 2023-09-22T21:59:11.337 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.337 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.337 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.337 DEBUG:boto:Params: {} 2023-09-22T21:59:11.337 DEBUG:boto:Token: None 2023-09-22T21:59:11.337 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj38 2023-09-22T21:59:11.338 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uXvH7HLtUR+8lL8nSjQVTu7LpOE= 2023-09-22T21:59:11.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uXvH7HLtUR+8lL8nSjQVTu7LpOE=', 'Content-Length': '0'} 2023-09-22T21:59:11.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e30a6aedf5c7eac-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.382 DEBUG:boto:path=/ffzdqu-61/obj38 2023-09-22T21:59:11.382 DEBUG:boto:auth_path=/ffzdqu-61/obj38 2023-09-22T21:59:11.382 DEBUG:boto:Method: GET 2023-09-22T21:59:11.383 DEBUG:boto:Path: /ffzdqu-61/obj38 2023-09-22T21:59:11.383 DEBUG:boto:Data: 2023-09-22T21:59:11.383 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.383 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.383 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.383 DEBUG:boto:Params: {} 2023-09-22T21:59:11.383 DEBUG:boto:Token: None 2023-09-22T21:59:11.383 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj38 2023-09-22T21:59:11.383 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uXvH7HLtUR+8lL8nSjQVTu7LpOE= 2023-09-22T21:59:11.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uXvH7HLtUR+8lL8nSjQVTu7LpOE=', 'Content-Length': '0'} 2023-09-22T21:59:11.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b05b3292a54057ee-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.387 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:11.387 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:11.387 DEBUG:boto:path=/ffzdqu-61/?prefix=obj38 2023-09-22T21:59:11.387 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj38 2023-09-22T21:59:11.387 DEBUG:boto:Method: GET 2023-09-22T21:59:11.387 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj38 2023-09-22T21:59:11.387 DEBUG:boto:Data: 2023-09-22T21:59:11.387 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.387 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.388 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.388 DEBUG:boto:Params: {} 2023-09-22T21:59:11.388 DEBUG:boto:Token: None 2023-09-22T21:59:11.388 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/ 2023-09-22T21:59:11.388 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10= 2023-09-22T21:59:11.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10=', 'Content-Length': '0'} 2023-09-22T21:59:11.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b21da16ca62df2c0-00650e0e2f-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.393 DEBUG:boto:b'ffzdqu-61obj381000falseobj382023-09-22T21:56:09.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:11.393 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:11.393 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:11.394 DEBUG:boto:path=/ffzdqu-61/?prefix=obj38 2023-09-22T21:59:11.394 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj38 2023-09-22T21:59:11.394 DEBUG:boto:Method: GET 2023-09-22T21:59:11.394 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj38 2023-09-22T21:59:11.394 DEBUG:boto:Data: 2023-09-22T21:59:11.394 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.394 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.394 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.394 DEBUG:boto:Params: {} 2023-09-22T21:59:11.394 DEBUG:boto:Token: None 2023-09-22T21:59:11.395 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/ 2023-09-22T21:59:11.395 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10= 2023-09-22T21:59:11.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10=', 'Content-Length': '0'} 2023-09-22T21:59:11.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acda5f0735c2cb83-00650e0e2f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.405 DEBUG:boto:b'ffzdqu-61obj381000falseobj382023-09-22T21:56:09.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:11.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:59:11.405 DEBUG:boto:path=/ffzdqu-61/obj44 2023-09-22T21:59:11.405 DEBUG:boto:auth_path=/ffzdqu-61/obj44 2023-09-22T21:59:11.405 DEBUG:boto:path=/ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.405 DEBUG:boto:auth_path=/ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.406 DEBUG:boto:Method: GET 2023-09-22T21:59:11.406 DEBUG:boto:Path: /ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.406 DEBUG:boto:Data: 2023-09-22T21:59:11.406 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.406 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.406 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.406 DEBUG:boto:Params: {} 2023-09-22T21:59:11.406 DEBUG:boto:Token: None 2023-09-22T21:59:11.406 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.407 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FtBprd4f74r82DeS5anVRrwez/k= 2023-09-22T21:59:11.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FtBprd4f74r82DeS5anVRrwez/k=', 'Content-Length': '0'} 2023-09-22T21:59:11.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c787aff39c45d40a-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.410 DEBUG:boto:path=/ffzdqu-61/obj44 2023-09-22T21:59:11.410 DEBUG:boto:auth_path=/ffzdqu-61/obj44 2023-09-22T21:59:11.410 DEBUG:boto:path=/ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.410 DEBUG:boto:auth_path=/ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.411 DEBUG:boto:Method: GET 2023-09-22T21:59:11.411 DEBUG:boto:Path: /ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.411 DEBUG:boto:Data: 2023-09-22T21:59:11.411 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.411 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.411 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.411 DEBUG:boto:Params: {} 2023-09-22T21:59:11.411 DEBUG:boto:Token: None 2023-09-22T21:59:11.411 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.412 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FtBprd4f74r82DeS5anVRrwez/k= 2023-09-22T21:59:11.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FtBprd4f74r82DeS5anVRrwez/k=', 'Content-Length': '0'} 2023-09-22T21:59:11.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ca6b496d69b40c0-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.415 DEBUG:boto:path=/ffzdqu-61/obj44 2023-09-22T21:59:11.415 DEBUG:boto:auth_path=/ffzdqu-61/obj44 2023-09-22T21:59:11.415 DEBUG:boto:path=/ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.415 DEBUG:boto:auth_path=/ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.415 DEBUG:boto:Method: HEAD 2023-09-22T21:59:11.415 DEBUG:boto:Path: /ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.416 DEBUG:boto:Data: 2023-09-22T21:59:11.416 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.416 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.416 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.416 DEBUG:boto:Params: {} 2023-09-22T21:59:11.416 DEBUG:boto:Token: None 2023-09-22T21:59:11.416 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.416 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dJzGQvS1XVvq8ZnxCsjG8yDOz8Q= 2023-09-22T21:59:11.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dJzGQvS1XVvq8ZnxCsjG8yDOz8Q=', 'Content-Length': '0'} 2023-09-22T21:59:11.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b68b12ea6ee43fb7-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.419 DEBUG:boto:path=/ffzdqu-61/obj44 2023-09-22T21:59:11.419 DEBUG:boto:auth_path=/ffzdqu-61/obj44 2023-09-22T21:59:11.420 DEBUG:boto:path=/ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.420 DEBUG:boto:auth_path=/ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.420 DEBUG:boto:Method: HEAD 2023-09-22T21:59:11.420 DEBUG:boto:Path: /ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.420 DEBUG:boto:Data: 2023-09-22T21:59:11.420 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.420 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.420 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.420 DEBUG:boto:Params: {} 2023-09-22T21:59:11.421 DEBUG:boto:Token: None 2023-09-22T21:59:11.421 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.421 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dJzGQvS1XVvq8ZnxCsjG8yDOz8Q= 2023-09-22T21:59:11.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dJzGQvS1XVvq8ZnxCsjG8yDOz8Q=', 'Content-Length': '0'} 2023-09-22T21:59:11.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000723e6426f34dfb2-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:59:11.424 DEBUG:boto:path=/ffzdqu-61/obj44 2023-09-22T21:59:11.424 DEBUG:boto:auth_path=/ffzdqu-61/obj44 2023-09-22T21:59:11.424 DEBUG:boto:path=/ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.424 DEBUG:boto:auth_path=/ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.424 DEBUG:boto:Method: GET 2023-09-22T21:59:11.424 DEBUG:boto:Path: /ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.424 DEBUG:boto:Data: 2023-09-22T21:59:11.425 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.425 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.425 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.425 DEBUG:boto:Params: {} 2023-09-22T21:59:11.425 DEBUG:boto:Token: None 2023-09-22T21:59:11.425 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.425 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FtBprd4f74r82DeS5anVRrwez/k= 2023-09-22T21:59:11.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FtBprd4f74r82DeS5anVRrwez/k=', 'Content-Length': '0'} 2023-09-22T21:59:11.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9f20af41b961ab1-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.470 DEBUG:boto:path=/ffzdqu-61/obj44 2023-09-22T21:59:11.470 DEBUG:boto:auth_path=/ffzdqu-61/obj44 2023-09-22T21:59:11.470 DEBUG:boto:path=/ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.471 DEBUG:boto:auth_path=/ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.471 DEBUG:boto:Method: GET 2023-09-22T21:59:11.471 DEBUG:boto:Path: /ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.471 DEBUG:boto:Data: 2023-09-22T21:59:11.471 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.471 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.471 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.471 DEBUG:boto:Params: {} 2023-09-22T21:59:11.471 DEBUG:boto:Token: None 2023-09-22T21:59:11.472 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj44?versionId=null 2023-09-22T21:59:11.472 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FtBprd4f74r82DeS5anVRrwez/k= 2023-09-22T21:59:11.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FtBprd4f74r82DeS5anVRrwez/k=', 'Content-Length': '0'} 2023-09-22T21:59:11.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005188eaa1a1a5d169-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.475 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:11.475 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:11.475 DEBUG:boto:path=/ffzdqu-61/?prefix=obj44 2023-09-22T21:59:11.475 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj44 2023-09-22T21:59:11.475 DEBUG:boto:Method: GET 2023-09-22T21:59:11.475 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj44 2023-09-22T21:59:11.475 DEBUG:boto:Data: 2023-09-22T21:59:11.475 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.476 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.476 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.476 DEBUG:boto:Params: {} 2023-09-22T21:59:11.476 DEBUG:boto:Token: None 2023-09-22T21:59:11.476 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/ 2023-09-22T21:59:11.476 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10= 2023-09-22T21:59:11.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10=', 'Content-Length': '0'} 2023-09-22T21:59:11.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4ba31aac540471f-00650e0e2f-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.481 DEBUG:boto:b'ffzdqu-61obj441000falseobj442023-09-22T21:56:08.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:11.482 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:11.482 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:11.482 DEBUG:boto:path=/ffzdqu-61/?prefix=obj44 2023-09-22T21:59:11.482 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj44 2023-09-22T21:59:11.482 DEBUG:boto:Method: GET 2023-09-22T21:59:11.482 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj44 2023-09-22T21:59:11.482 DEBUG:boto:Data: 2023-09-22T21:59:11.482 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.483 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.483 DEBUG:boto:Params: {} 2023-09-22T21:59:11.483 DEBUG:boto:Token: None 2023-09-22T21:59:11.483 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/ 2023-09-22T21:59:11.483 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10= 2023-09-22T21:59:11.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10=', 'Content-Length': '0'} 2023-09-22T21:59:11.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d2aaa786391a1f5-00650e0e2f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.493 DEBUG:boto:b'ffzdqu-61obj441000falseobj442023-09-22T21:56:08.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:11.493 DEBUG:boto:path=/ffzdqu-61/obj44 2023-09-22T21:59:11.493 DEBUG:boto:auth_path=/ffzdqu-61/obj44 2023-09-22T21:59:11.494 DEBUG:boto:Method: HEAD 2023-09-22T21:59:11.494 DEBUG:boto:Path: /ffzdqu-61/obj44 2023-09-22T21:59:11.494 DEBUG:boto:Data: 2023-09-22T21:59:11.494 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.494 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.494 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.494 DEBUG:boto:Params: {} 2023-09-22T21:59:11.494 DEBUG:boto:Token: None 2023-09-22T21:59:11.494 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj44 2023-09-22T21:59:11.495 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FYfY/b7c8/FleuQ1aXPhiDVzmmA= 2023-09-22T21:59:11.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FYfY/b7c8/FleuQ1aXPhiDVzmmA=', 'Content-Length': '0'} 2023-09-22T21:59:11.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e952e06adaf513b-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.497 DEBUG:boto:path=/ffzdqu-61/obj44 2023-09-22T21:59:11.498 DEBUG:boto:auth_path=/ffzdqu-61/obj44 2023-09-22T21:59:11.498 DEBUG:boto:Method: HEAD 2023-09-22T21:59:11.498 DEBUG:boto:Path: /ffzdqu-61/obj44 2023-09-22T21:59:11.498 DEBUG:boto:Data: 2023-09-22T21:59:11.498 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.498 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.498 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.498 DEBUG:boto:Params: {} 2023-09-22T21:59:11.498 DEBUG:boto:Token: None 2023-09-22T21:59:11.499 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj44 2023-09-22T21:59:11.499 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FYfY/b7c8/FleuQ1aXPhiDVzmmA= 2023-09-22T21:59:11.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FYfY/b7c8/FleuQ1aXPhiDVzmmA=', 'Content-Length': '0'} 2023-09-22T21:59:11.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041a6b639c971630e-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T21:59:11.502 DEBUG:boto:path=/ffzdqu-61/obj44 2023-09-22T21:59:11.502 DEBUG:boto:auth_path=/ffzdqu-61/obj44 2023-09-22T21:59:11.502 DEBUG:boto:Method: GET 2023-09-22T21:59:11.502 DEBUG:boto:Path: /ffzdqu-61/obj44 2023-09-22T21:59:11.502 DEBUG:boto:Data: 2023-09-22T21:59:11.502 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.502 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.502 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.503 DEBUG:boto:Params: {} 2023-09-22T21:59:11.503 DEBUG:boto:Token: None 2023-09-22T21:59:11.503 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj44 2023-09-22T21:59:11.503 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lYrBi81K2NnDdLGaCOJpBrAR34U= 2023-09-22T21:59:11.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lYrBi81K2NnDdLGaCOJpBrAR34U=', 'Content-Length': '0'} 2023-09-22T21:59:11.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068d5291f61aab561-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.506 DEBUG:boto:path=/ffzdqu-61/obj44 2023-09-22T21:59:11.506 DEBUG:boto:auth_path=/ffzdqu-61/obj44 2023-09-22T21:59:11.506 DEBUG:boto:Method: GET 2023-09-22T21:59:11.506 DEBUG:boto:Path: /ffzdqu-61/obj44 2023-09-22T21:59:11.506 DEBUG:boto:Data: 2023-09-22T21:59:11.507 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.507 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.507 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.507 DEBUG:boto:Params: {} 2023-09-22T21:59:11.507 DEBUG:boto:Token: None 2023-09-22T21:59:11.507 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj44 2023-09-22T21:59:11.507 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lYrBi81K2NnDdLGaCOJpBrAR34U= 2023-09-22T21:59:11.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lYrBi81K2NnDdLGaCOJpBrAR34U=', 'Content-Length': '0'} 2023-09-22T21:59:11.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a53faf575b277b5-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.510 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:11.511 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:11.511 DEBUG:boto:path=/ffzdqu-61/?prefix=obj44 2023-09-22T21:59:11.511 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj44 2023-09-22T21:59:11.511 DEBUG:boto:Method: GET 2023-09-22T21:59:11.511 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj44 2023-09-22T21:59:11.511 DEBUG:boto:Data: 2023-09-22T21:59:11.511 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.511 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.511 DEBUG:boto:Params: {} 2023-09-22T21:59:11.512 DEBUG:boto:Token: None 2023-09-22T21:59:11.512 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/ 2023-09-22T21:59:11.512 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10= 2023-09-22T21:59:11.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10=', 'Content-Length': '0'} 2023-09-22T21:59:11.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000326f499b86d6907a-00650e0e2f-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.518 DEBUG:boto:b'ffzdqu-61obj441000falseobj442023-09-22T21:56:08.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:11.518 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:11.518 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:11.518 DEBUG:boto:path=/ffzdqu-61/?prefix=obj44 2023-09-22T21:59:11.518 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj44 2023-09-22T21:59:11.519 DEBUG:boto:Method: GET 2023-09-22T21:59:11.519 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj44 2023-09-22T21:59:11.519 DEBUG:boto:Data: 2023-09-22T21:59:11.519 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.519 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.519 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.519 DEBUG:boto:Params: {} 2023-09-22T21:59:11.519 DEBUG:boto:Token: None 2023-09-22T21:59:11.519 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/ 2023-09-22T21:59:11.520 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10= 2023-09-22T21:59:11.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10=', 'Content-Length': '0'} 2023-09-22T21:59:11.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008aa2c30a5cdd1a48-00650e0e2f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.574 DEBUG:boto:b'ffzdqu-61obj441000falseobj442023-09-22T21:56:08.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:11.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-22T21:59:11.575 DEBUG:boto:path=/ffzdqu-61/obj46 2023-09-22T21:59:11.575 DEBUG:boto:auth_path=/ffzdqu-61/obj46 2023-09-22T21:59:11.575 DEBUG:boto:path=/ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.575 DEBUG:boto:auth_path=/ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.575 DEBUG:boto:Method: GET 2023-09-22T21:59:11.575 DEBUG:boto:Path: /ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.575 DEBUG:boto:Data: 2023-09-22T21:59:11.575 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.576 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.576 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.576 DEBUG:boto:Params: {} 2023-09-22T21:59:11.576 DEBUG:boto:Token: None 2023-09-22T21:59:11.576 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.576 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mgovsJTweH5CQuIA6aNMcIJf3IQ= 2023-09-22T21:59:11.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mgovsJTweH5CQuIA6aNMcIJf3IQ=', 'Content-Length': '0'} 2023-09-22T21:59:11.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093429c28a72c3fcc-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.579 DEBUG:boto:path=/ffzdqu-61/obj46 2023-09-22T21:59:11.580 DEBUG:boto:auth_path=/ffzdqu-61/obj46 2023-09-22T21:59:11.580 DEBUG:boto:path=/ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.580 DEBUG:boto:auth_path=/ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.580 DEBUG:boto:Method: GET 2023-09-22T21:59:11.580 DEBUG:boto:Path: /ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.580 DEBUG:boto:Data: 2023-09-22T21:59:11.580 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.580 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.580 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.580 DEBUG:boto:Params: {} 2023-09-22T21:59:11.581 DEBUG:boto:Token: None 2023-09-22T21:59:11.581 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.581 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mgovsJTweH5CQuIA6aNMcIJf3IQ= 2023-09-22T21:59:11.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mgovsJTweH5CQuIA6aNMcIJf3IQ=', 'Content-Length': '0'} 2023-09-22T21:59:11.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c540436ee7186441-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.584 DEBUG:boto:path=/ffzdqu-61/obj46 2023-09-22T21:59:11.584 DEBUG:boto:auth_path=/ffzdqu-61/obj46 2023-09-22T21:59:11.584 DEBUG:boto:path=/ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.584 DEBUG:boto:auth_path=/ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.584 DEBUG:boto:Method: HEAD 2023-09-22T21:59:11.584 DEBUG:boto:Path: /ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.584 DEBUG:boto:Data: 2023-09-22T21:59:11.585 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.585 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.585 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.585 DEBUG:boto:Params: {} 2023-09-22T21:59:11.585 DEBUG:boto:Token: None 2023-09-22T21:59:11.585 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.585 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:02XJqD1iq8ZwHWImO504dh3dP48= 2023-09-22T21:59:11.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:02XJqD1iq8ZwHWImO504dh3dP48=', 'Content-Length': '0'} 2023-09-22T21:59:11.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ef60f141cd89d4e-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.588 DEBUG:boto:path=/ffzdqu-61/obj46 2023-09-22T21:59:11.588 DEBUG:boto:auth_path=/ffzdqu-61/obj46 2023-09-22T21:59:11.589 DEBUG:boto:path=/ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.589 DEBUG:boto:auth_path=/ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.589 DEBUG:boto:Method: HEAD 2023-09-22T21:59:11.589 DEBUG:boto:Path: /ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.589 DEBUG:boto:Data: 2023-09-22T21:59:11.589 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.589 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.589 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.589 DEBUG:boto:Params: {} 2023-09-22T21:59:11.589 DEBUG:boto:Token: None 2023-09-22T21:59:11.590 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.590 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:02XJqD1iq8ZwHWImO504dh3dP48= 2023-09-22T21:59:11.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:02XJqD1iq8ZwHWImO504dh3dP48=', 'Content-Length': '0'} 2023-09-22T21:59:11.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000821dce3ddd296748-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-22T21:59:11.593 DEBUG:boto:path=/ffzdqu-61/obj46 2023-09-22T21:59:11.593 DEBUG:boto:auth_path=/ffzdqu-61/obj46 2023-09-22T21:59:11.593 DEBUG:boto:path=/ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.593 DEBUG:boto:auth_path=/ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.593 DEBUG:boto:Method: GET 2023-09-22T21:59:11.593 DEBUG:boto:Path: /ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.593 DEBUG:boto:Data: 2023-09-22T21:59:11.593 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.593 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.593 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.594 DEBUG:boto:Params: {} 2023-09-22T21:59:11.594 DEBUG:boto:Token: None 2023-09-22T21:59:11.594 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.594 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mgovsJTweH5CQuIA6aNMcIJf3IQ= 2023-09-22T21:59:11.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mgovsJTweH5CQuIA6aNMcIJf3IQ=', 'Content-Length': '0'} 2023-09-22T21:59:11.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001478fa94d33304b8-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.638 DEBUG:boto:path=/ffzdqu-61/obj46 2023-09-22T21:59:11.638 DEBUG:boto:auth_path=/ffzdqu-61/obj46 2023-09-22T21:59:11.638 DEBUG:boto:path=/ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.639 DEBUG:boto:auth_path=/ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.639 DEBUG:boto:Method: GET 2023-09-22T21:59:11.639 DEBUG:boto:Path: /ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.639 DEBUG:boto:Data: 2023-09-22T21:59:11.639 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.639 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.639 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.639 DEBUG:boto:Params: {} 2023-09-22T21:59:11.639 DEBUG:boto:Token: None 2023-09-22T21:59:11.640 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj46?versionId=null 2023-09-22T21:59:11.640 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mgovsJTweH5CQuIA6aNMcIJf3IQ= 2023-09-22T21:59:11.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mgovsJTweH5CQuIA6aNMcIJf3IQ=', 'Content-Length': '0'} 2023-09-22T21:59:11.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b55cbc2d611c74df-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.643 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:11.643 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:11.643 DEBUG:boto:path=/ffzdqu-61/?prefix=obj46 2023-09-22T21:59:11.643 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj46 2023-09-22T21:59:11.643 DEBUG:boto:Method: GET 2023-09-22T21:59:11.643 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj46 2023-09-22T21:59:11.643 DEBUG:boto:Data: 2023-09-22T21:59:11.643 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.644 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.644 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.644 DEBUG:boto:Params: {} 2023-09-22T21:59:11.644 DEBUG:boto:Token: None 2023-09-22T21:59:11.644 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/ 2023-09-22T21:59:11.644 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10= 2023-09-22T21:59:11.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10=', 'Content-Length': '0'} 2023-09-22T21:59:11.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b68b043b675c7421-00650e0e2f-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.650 DEBUG:boto:b'ffzdqu-61obj461000falseobj462023-09-22T21:56:09.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:11.650 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:11.650 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:11.650 DEBUG:boto:path=/ffzdqu-61/?prefix=obj46 2023-09-22T21:59:11.650 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj46 2023-09-22T21:59:11.651 DEBUG:boto:Method: GET 2023-09-22T21:59:11.651 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj46 2023-09-22T21:59:11.651 DEBUG:boto:Data: 2023-09-22T21:59:11.651 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.651 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.651 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.651 DEBUG:boto:Params: {} 2023-09-22T21:59:11.651 DEBUG:boto:Token: None 2023-09-22T21:59:11.651 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/ 2023-09-22T21:59:11.652 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10= 2023-09-22T21:59:11.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10=', 'Content-Length': '0'} 2023-09-22T21:59:11.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aaae189e6ceef594-00650e0e2f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.710 DEBUG:boto:b'ffzdqu-61obj461000falseobj462023-09-22T21:56:09.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:11.711 DEBUG:boto:path=/ffzdqu-61/obj46 2023-09-22T21:59:11.711 DEBUG:boto:auth_path=/ffzdqu-61/obj46 2023-09-22T21:59:11.711 DEBUG:boto:Method: HEAD 2023-09-22T21:59:11.711 DEBUG:boto:Path: /ffzdqu-61/obj46 2023-09-22T21:59:11.711 DEBUG:boto:Data: 2023-09-22T21:59:11.711 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.711 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.711 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.712 DEBUG:boto:Params: {} 2023-09-22T21:59:11.712 DEBUG:boto:Token: None 2023-09-22T21:59:11.712 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj46 2023-09-22T21:59:11.712 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t9b5IKnLooL4oJtnUmJi9WIwqJk= 2023-09-22T21:59:11.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t9b5IKnLooL4oJtnUmJi9WIwqJk=', 'Content-Length': '0'} 2023-09-22T21:59:11.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a0eea3a6a2c6342-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.715 DEBUG:boto:path=/ffzdqu-61/obj46 2023-09-22T21:59:11.715 DEBUG:boto:auth_path=/ffzdqu-61/obj46 2023-09-22T21:59:11.715 DEBUG:boto:Method: HEAD 2023-09-22T21:59:11.715 DEBUG:boto:Path: /ffzdqu-61/obj46 2023-09-22T21:59:11.716 DEBUG:boto:Data: 2023-09-22T21:59:11.716 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.716 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.716 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.716 DEBUG:boto:Params: {} 2023-09-22T21:59:11.716 DEBUG:boto:Token: None 2023-09-22T21:59:11.716 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj46 2023-09-22T21:59:11.716 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t9b5IKnLooL4oJtnUmJi9WIwqJk= 2023-09-22T21:59:11.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t9b5IKnLooL4oJtnUmJi9WIwqJk=', 'Content-Length': '0'} 2023-09-22T21:59:11.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001299633a5e1df323-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-22T21:59:11.720 DEBUG:boto:path=/ffzdqu-61/obj46 2023-09-22T21:59:11.720 DEBUG:boto:auth_path=/ffzdqu-61/obj46 2023-09-22T21:59:11.720 DEBUG:boto:Method: GET 2023-09-22T21:59:11.720 DEBUG:boto:Path: /ffzdqu-61/obj46 2023-09-22T21:59:11.720 DEBUG:boto:Data: 2023-09-22T21:59:11.720 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.720 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.720 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.720 DEBUG:boto:Params: {} 2023-09-22T21:59:11.721 DEBUG:boto:Token: None 2023-09-22T21:59:11.721 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj46 2023-09-22T21:59:11.721 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:47xDotCisxNaq+cJ7JxifXWEzVk= 2023-09-22T21:59:11.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:47xDotCisxNaq+cJ7JxifXWEzVk=', 'Content-Length': '0'} 2023-09-22T21:59:11.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004839f29407e32b76-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.724 DEBUG:boto:path=/ffzdqu-61/obj46 2023-09-22T21:59:11.724 DEBUG:boto:auth_path=/ffzdqu-61/obj46 2023-09-22T21:59:11.724 DEBUG:boto:Method: GET 2023-09-22T21:59:11.725 DEBUG:boto:Path: /ffzdqu-61/obj46 2023-09-22T21:59:11.725 DEBUG:boto:Data: 2023-09-22T21:59:11.725 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.725 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.725 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.725 DEBUG:boto:Params: {} 2023-09-22T21:59:11.725 DEBUG:boto:Token: None 2023-09-22T21:59:11.725 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj46 2023-09-22T21:59:11.725 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:47xDotCisxNaq+cJ7JxifXWEzVk= 2023-09-22T21:59:11.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:47xDotCisxNaq+cJ7JxifXWEzVk=', 'Content-Length': '0'} 2023-09-22T21:59:11.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000096bb739d7e5ae6e-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.728 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:11.728 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:11.728 DEBUG:boto:path=/ffzdqu-61/?prefix=obj46 2023-09-22T21:59:11.728 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj46 2023-09-22T21:59:11.729 DEBUG:boto:Method: GET 2023-09-22T21:59:11.729 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj46 2023-09-22T21:59:11.729 DEBUG:boto:Data: 2023-09-22T21:59:11.729 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.729 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.729 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.729 DEBUG:boto:Params: {} 2023-09-22T21:59:11.729 DEBUG:boto:Token: None 2023-09-22T21:59:11.729 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/ 2023-09-22T21:59:11.730 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10= 2023-09-22T21:59:11.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10=', 'Content-Length': '0'} 2023-09-22T21:59:11.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9e89dc265fb8151-00650e0e2f-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.778 DEBUG:boto:b'ffzdqu-61obj461000falseobj462023-09-22T21:56:09.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:11.779 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:11.779 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:11.779 DEBUG:boto:path=/ffzdqu-61/?prefix=obj46 2023-09-22T21:59:11.779 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj46 2023-09-22T21:59:11.779 DEBUG:boto:Method: GET 2023-09-22T21:59:11.779 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj46 2023-09-22T21:59:11.779 DEBUG:boto:Data: 2023-09-22T21:59:11.779 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.779 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.779 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.780 DEBUG:boto:Params: {} 2023-09-22T21:59:11.780 DEBUG:boto:Token: None 2023-09-22T21:59:11.780 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/ 2023-09-22T21:59:11.780 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10= 2023-09-22T21:59:11.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10=', 'Content-Length': '0'} 2023-09-22T21:59:11.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2b6d6713899dc68-00650e0e2f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.790 DEBUG:boto:b'ffzdqu-61obj461000falseobj462023-09-22T21:56:09.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:11.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:59:11.790 DEBUG:boto:path=/ffzdqu-61/obj51 2023-09-22T21:59:11.791 DEBUG:boto:auth_path=/ffzdqu-61/obj51 2023-09-22T21:59:11.791 DEBUG:boto:path=/ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.791 DEBUG:boto:auth_path=/ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.791 DEBUG:boto:Method: GET 2023-09-22T21:59:11.791 DEBUG:boto:Path: /ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.791 DEBUG:boto:Data: 2023-09-22T21:59:11.791 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.791 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.791 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.791 DEBUG:boto:Params: {} 2023-09-22T21:59:11.792 DEBUG:boto:Token: None 2023-09-22T21:59:11.792 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.792 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D77UkzEIInMjl5r19GJAn7EQ3bc= 2023-09-22T21:59:11.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D77UkzEIInMjl5r19GJAn7EQ3bc=', 'Content-Length': '0'} 2023-09-22T21:59:11.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd2517dd27f20dbb-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.795 DEBUG:boto:path=/ffzdqu-61/obj51 2023-09-22T21:59:11.795 DEBUG:boto:auth_path=/ffzdqu-61/obj51 2023-09-22T21:59:11.795 DEBUG:boto:path=/ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.795 DEBUG:boto:auth_path=/ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.796 DEBUG:boto:Method: GET 2023-09-22T21:59:11.796 DEBUG:boto:Path: /ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.796 DEBUG:boto:Data: 2023-09-22T21:59:11.796 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.796 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.796 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.796 DEBUG:boto:Params: {} 2023-09-22T21:59:11.796 DEBUG:boto:Token: None 2023-09-22T21:59:11.796 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.797 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D77UkzEIInMjl5r19GJAn7EQ3bc= 2023-09-22T21:59:11.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D77UkzEIInMjl5r19GJAn7EQ3bc=', 'Content-Length': '0'} 2023-09-22T21:59:11.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f87ab05c1bb9df9a-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.842 DEBUG:boto:path=/ffzdqu-61/obj51 2023-09-22T21:59:11.842 DEBUG:boto:auth_path=/ffzdqu-61/obj51 2023-09-22T21:59:11.843 DEBUG:boto:path=/ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.843 DEBUG:boto:auth_path=/ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.843 DEBUG:boto:Method: HEAD 2023-09-22T21:59:11.843 DEBUG:boto:Path: /ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.843 DEBUG:boto:Data: 2023-09-22T21:59:11.843 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.843 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.843 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.843 DEBUG:boto:Params: {} 2023-09-22T21:59:11.844 DEBUG:boto:Token: None 2023-09-22T21:59:11.844 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.844 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8fCaEIrBMr8cY1xEATIsYL/FUYQ= 2023-09-22T21:59:11.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8fCaEIrBMr8cY1xEATIsYL/FUYQ=', 'Content-Length': '0'} 2023-09-22T21:59:11.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001554bc2a86c87b50-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.847 DEBUG:boto:path=/ffzdqu-61/obj51 2023-09-22T21:59:11.847 DEBUG:boto:auth_path=/ffzdqu-61/obj51 2023-09-22T21:59:11.847 DEBUG:boto:path=/ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.847 DEBUG:boto:auth_path=/ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.847 DEBUG:boto:Method: HEAD 2023-09-22T21:59:11.848 DEBUG:boto:Path: /ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.848 DEBUG:boto:Data: 2023-09-22T21:59:11.848 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.848 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.848 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.848 DEBUG:boto:Params: {} 2023-09-22T21:59:11.848 DEBUG:boto:Token: None 2023-09-22T21:59:11.848 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.848 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8fCaEIrBMr8cY1xEATIsYL/FUYQ= 2023-09-22T21:59:11.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8fCaEIrBMr8cY1xEATIsYL/FUYQ=', 'Content-Length': '0'} 2023-09-22T21:59:11.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008178da6d44b73ada-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:59:11.851 DEBUG:boto:path=/ffzdqu-61/obj51 2023-09-22T21:59:11.851 DEBUG:boto:auth_path=/ffzdqu-61/obj51 2023-09-22T21:59:11.851 DEBUG:boto:path=/ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.852 DEBUG:boto:auth_path=/ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.852 DEBUG:boto:Method: GET 2023-09-22T21:59:11.852 DEBUG:boto:Path: /ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.852 DEBUG:boto:Data: 2023-09-22T21:59:11.852 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.852 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.852 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.852 DEBUG:boto:Params: {} 2023-09-22T21:59:11.852 DEBUG:boto:Token: None 2023-09-22T21:59:11.853 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.853 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D77UkzEIInMjl5r19GJAn7EQ3bc= 2023-09-22T21:59:11.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D77UkzEIInMjl5r19GJAn7EQ3bc=', 'Content-Length': '0'} 2023-09-22T21:59:11.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e34bc30a78b87bd1-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.856 DEBUG:boto:path=/ffzdqu-61/obj51 2023-09-22T21:59:11.856 DEBUG:boto:auth_path=/ffzdqu-61/obj51 2023-09-22T21:59:11.856 DEBUG:boto:path=/ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.856 DEBUG:boto:auth_path=/ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.856 DEBUG:boto:Method: GET 2023-09-22T21:59:11.856 DEBUG:boto:Path: /ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.856 DEBUG:boto:Data: 2023-09-22T21:59:11.856 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.856 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.856 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.857 DEBUG:boto:Params: {} 2023-09-22T21:59:11.857 DEBUG:boto:Token: None 2023-09-22T21:59:11.857 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj51?versionId=null 2023-09-22T21:59:11.857 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D77UkzEIInMjl5r19GJAn7EQ3bc= 2023-09-22T21:59:11.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D77UkzEIInMjl5r19GJAn7EQ3bc=', 'Content-Length': '0'} 2023-09-22T21:59:11.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f80103eeb552409c-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.860 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:11.860 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:11.860 DEBUG:boto:path=/ffzdqu-61/?prefix=obj51 2023-09-22T21:59:11.860 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj51 2023-09-22T21:59:11.860 DEBUG:boto:Method: GET 2023-09-22T21:59:11.860 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj51 2023-09-22T21:59:11.860 DEBUG:boto:Data: 2023-09-22T21:59:11.861 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.861 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.861 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.861 DEBUG:boto:Params: {} 2023-09-22T21:59:11.861 DEBUG:boto:Token: None 2023-09-22T21:59:11.861 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/ 2023-09-22T21:59:11.861 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10= 2023-09-22T21:59:11.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10=', 'Content-Length': '0'} 2023-09-22T21:59:11.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c4f9754e9729c42-00650e0e2f-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.914 DEBUG:boto:b'ffzdqu-61obj511000falseobj512023-09-22T21:56:08.964Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:11.915 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:11.915 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:11.915 DEBUG:boto:path=/ffzdqu-61/?prefix=obj51 2023-09-22T21:59:11.915 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj51 2023-09-22T21:59:11.915 DEBUG:boto:Method: GET 2023-09-22T21:59:11.915 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj51 2023-09-22T21:59:11.915 DEBUG:boto:Data: 2023-09-22T21:59:11.916 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.916 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.916 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.916 DEBUG:boto:Params: {} 2023-09-22T21:59:11.916 DEBUG:boto:Token: None 2023-09-22T21:59:11.916 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/ 2023-09-22T21:59:11.916 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10= 2023-09-22T21:59:11.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10=', 'Content-Length': '0'} 2023-09-22T21:59:11.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d29f935c8aac64d-00650e0e2f-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.926 DEBUG:boto:b'ffzdqu-61obj511000falseobj512023-09-22T21:56:08.964Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:11.926 DEBUG:boto:path=/ffzdqu-61/obj51 2023-09-22T21:59:11.926 DEBUG:boto:auth_path=/ffzdqu-61/obj51 2023-09-22T21:59:11.926 DEBUG:boto:Method: HEAD 2023-09-22T21:59:11.926 DEBUG:boto:Path: /ffzdqu-61/obj51 2023-09-22T21:59:11.927 DEBUG:boto:Data: 2023-09-22T21:59:11.927 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.927 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.927 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.927 DEBUG:boto:Params: {} 2023-09-22T21:59:11.927 DEBUG:boto:Token: None 2023-09-22T21:59:11.927 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj51 2023-09-22T21:59:11.927 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y+P4IQV1wjGIbKchT/hoXu5c3Pc= 2023-09-22T21:59:11.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y+P4IQV1wjGIbKchT/hoXu5c3Pc=', 'Content-Length': '0'} 2023-09-22T21:59:11.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9c1a2d5e84ff34c-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.931 DEBUG:boto:path=/ffzdqu-61/obj51 2023-09-22T21:59:11.931 DEBUG:boto:auth_path=/ffzdqu-61/obj51 2023-09-22T21:59:11.931 DEBUG:boto:Method: HEAD 2023-09-22T21:59:11.931 DEBUG:boto:Path: /ffzdqu-61/obj51 2023-09-22T21:59:11.931 DEBUG:boto:Data: 2023-09-22T21:59:11.931 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.931 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.931 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.932 DEBUG:boto:Params: {} 2023-09-22T21:59:11.932 DEBUG:boto:Token: None 2023-09-22T21:59:11.932 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj51 2023-09-22T21:59:11.932 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y+P4IQV1wjGIbKchT/hoXu5c3Pc= 2023-09-22T21:59:11.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y+P4IQV1wjGIbKchT/hoXu5c3Pc=', 'Content-Length': '0'} 2023-09-22T21:59:11.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009263fba5faaf11af-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T21:59:11.935 DEBUG:boto:path=/ffzdqu-61/obj51 2023-09-22T21:59:11.935 DEBUG:boto:auth_path=/ffzdqu-61/obj51 2023-09-22T21:59:11.935 DEBUG:boto:Method: GET 2023-09-22T21:59:11.935 DEBUG:boto:Path: /ffzdqu-61/obj51 2023-09-22T21:59:11.935 DEBUG:boto:Data: 2023-09-22T21:59:11.935 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.936 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.936 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.936 DEBUG:boto:Params: {} 2023-09-22T21:59:11.936 DEBUG:boto:Token: None 2023-09-22T21:59:11.936 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj51 2023-09-22T21:59:11.936 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jk+JKA3mPePJ5CqIOUQAWcjvD4c= 2023-09-22T21:59:11.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jk+JKA3mPePJ5CqIOUQAWcjvD4c=', 'Content-Length': '0'} 2023-09-22T21:59:11.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012eebab5228cec63-00650e0e2f-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.939 DEBUG:boto:path=/ffzdqu-61/obj51 2023-09-22T21:59:11.939 DEBUG:boto:auth_path=/ffzdqu-61/obj51 2023-09-22T21:59:11.940 DEBUG:boto:Method: GET 2023-09-22T21:59:11.940 DEBUG:boto:Path: /ffzdqu-61/obj51 2023-09-22T21:59:11.940 DEBUG:boto:Data: 2023-09-22T21:59:11.940 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.940 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:11.940 DEBUG:boto:Port: 8000 2023-09-22T21:59:11.940 DEBUG:boto:Params: {} 2023-09-22T21:59:11.940 DEBUG:boto:Token: None 2023-09-22T21:59:11.940 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/obj51 2023-09-22T21:59:11.941 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jk+JKA3mPePJ5CqIOUQAWcjvD4c= 2023-09-22T21:59:11.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jk+JKA3mPePJ5CqIOUQAWcjvD4c=', 'Content-Length': '0'} 2023-09-22T21:59:11.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc38cb20f8606f63-00650e0e2f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.986 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:11.986 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:11.987 DEBUG:boto:path=/ffzdqu-61/?prefix=obj51 2023-09-22T21:59:11.987 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj51 2023-09-22T21:59:11.987 DEBUG:boto:Method: GET 2023-09-22T21:59:11.987 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj51 2023-09-22T21:59:11.987 DEBUG:boto:Data: 2023-09-22T21:59:11.987 DEBUG:boto:Headers: {} 2023-09-22T21:59:11.987 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:11.987 DEBUG:boto:Port: 8001 2023-09-22T21:59:11.987 DEBUG:boto:Params: {} 2023-09-22T21:59:11.987 DEBUG:boto:Token: None 2023-09-22T21:59:11.988 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:11 GMT /ffzdqu-61/ 2023-09-22T21:59:11.988 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10= 2023-09-22T21:59:11.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:11 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2t5LUCNc4VucgL5yYM2QZo/Tl10=', 'Content-Length': '0'} 2023-09-22T21:59:11.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d69427bb572bdb4a-00650e0e2f-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:11.993 DEBUG:boto:b'ffzdqu-61obj511000falseobj512023-09-22T21:56:08.964Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:11.993 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:11.993 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:11.994 DEBUG:boto:path=/ffzdqu-61/?prefix=obj51 2023-09-22T21:59:11.994 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj51 2023-09-22T21:59:11.994 DEBUG:boto:Method: GET 2023-09-22T21:59:11.994 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj51 2023-09-22T21:59:11.994 DEBUG:boto:Data: 2023-09-22T21:59:12.055 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.055 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.055 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.055 DEBUG:boto:Params: {} 2023-09-22T21:59:12.055 DEBUG:boto:Token: None 2023-09-22T21:59:12.056 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/ 2023-09-22T21:59:12.056 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI= 2023-09-22T21:59:12.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI=', 'Content-Length': '0'} 2023-09-22T21:59:12.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007187a4a73a790b9d-00650e0e30-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.066 DEBUG:boto:b'ffzdqu-61obj511000falseobj512023-09-22T21:56:08.964Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:12.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:59:12.066 DEBUG:boto:path=/ffzdqu-61/obj55 2023-09-22T21:59:12.066 DEBUG:boto:auth_path=/ffzdqu-61/obj55 2023-09-22T21:59:12.066 DEBUG:boto:path=/ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.066 DEBUG:boto:auth_path=/ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.066 DEBUG:boto:Method: GET 2023-09-22T21:59:12.067 DEBUG:boto:Path: /ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.067 DEBUG:boto:Data: 2023-09-22T21:59:12.067 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.067 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.067 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.067 DEBUG:boto:Params: {} 2023-09-22T21:59:12.067 DEBUG:boto:Token: None 2023-09-22T21:59:12.067 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.068 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uQxUeDdtkaeIjMvo2TREcQiUoNQ= 2023-09-22T21:59:12.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uQxUeDdtkaeIjMvo2TREcQiUoNQ=', 'Content-Length': '0'} 2023-09-22T21:59:12.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd0d0a10d1ee6b05-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.071 DEBUG:boto:path=/ffzdqu-61/obj55 2023-09-22T21:59:12.071 DEBUG:boto:auth_path=/ffzdqu-61/obj55 2023-09-22T21:59:12.071 DEBUG:boto:path=/ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.071 DEBUG:boto:auth_path=/ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.071 DEBUG:boto:Method: GET 2023-09-22T21:59:12.071 DEBUG:boto:Path: /ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.072 DEBUG:boto:Data: 2023-09-22T21:59:12.072 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.072 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.072 DEBUG:boto:Params: {} 2023-09-22T21:59:12.072 DEBUG:boto:Token: None 2023-09-22T21:59:12.072 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.072 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uQxUeDdtkaeIjMvo2TREcQiUoNQ= 2023-09-22T21:59:12.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uQxUeDdtkaeIjMvo2TREcQiUoNQ=', 'Content-Length': '0'} 2023-09-22T21:59:12.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e2c7c36dc6d63ae-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.076 DEBUG:boto:path=/ffzdqu-61/obj55 2023-09-22T21:59:12.076 DEBUG:boto:auth_path=/ffzdqu-61/obj55 2023-09-22T21:59:12.076 DEBUG:boto:path=/ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.076 DEBUG:boto:auth_path=/ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.076 DEBUG:boto:Method: HEAD 2023-09-22T21:59:12.076 DEBUG:boto:Path: /ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.076 DEBUG:boto:Data: 2023-09-22T21:59:12.076 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.076 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.077 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.077 DEBUG:boto:Params: {} 2023-09-22T21:59:12.077 DEBUG:boto:Token: None 2023-09-22T21:59:12.077 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.077 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ://8sp9tUVsRJDNl/QV6JLWsF3lY= 2023-09-22T21:59:12.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ://8sp9tUVsRJDNl/QV6JLWsF3lY=', 'Content-Length': '0'} 2023-09-22T21:59:12.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005549224d13b332a-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.080 DEBUG:boto:path=/ffzdqu-61/obj55 2023-09-22T21:59:12.080 DEBUG:boto:auth_path=/ffzdqu-61/obj55 2023-09-22T21:59:12.081 DEBUG:boto:path=/ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.081 DEBUG:boto:auth_path=/ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.081 DEBUG:boto:Method: HEAD 2023-09-22T21:59:12.081 DEBUG:boto:Path: /ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.081 DEBUG:boto:Data: 2023-09-22T21:59:12.081 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.081 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.081 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.081 DEBUG:boto:Params: {} 2023-09-22T21:59:12.081 DEBUG:boto:Token: None 2023-09-22T21:59:12.082 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.082 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ://8sp9tUVsRJDNl/QV6JLWsF3lY= 2023-09-22T21:59:12.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ://8sp9tUVsRJDNl/QV6JLWsF3lY=', 'Content-Length': '0'} 2023-09-22T21:59:12.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066c2fe867c9e1807-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:59:12.085 DEBUG:boto:path=/ffzdqu-61/obj55 2023-09-22T21:59:12.085 DEBUG:boto:auth_path=/ffzdqu-61/obj55 2023-09-22T21:59:12.085 DEBUG:boto:path=/ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.085 DEBUG:boto:auth_path=/ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.085 DEBUG:boto:Method: GET 2023-09-22T21:59:12.086 DEBUG:boto:Path: /ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.086 DEBUG:boto:Data: 2023-09-22T21:59:12.086 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.086 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.086 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.086 DEBUG:boto:Params: {} 2023-09-22T21:59:12.086 DEBUG:boto:Token: None 2023-09-22T21:59:12.086 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.086 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uQxUeDdtkaeIjMvo2TREcQiUoNQ= 2023-09-22T21:59:12.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uQxUeDdtkaeIjMvo2TREcQiUoNQ=', 'Content-Length': '0'} 2023-09-22T21:59:12.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046966abfe01c28da-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.134 DEBUG:boto:path=/ffzdqu-61/obj55 2023-09-22T21:59:12.134 DEBUG:boto:auth_path=/ffzdqu-61/obj55 2023-09-22T21:59:12.135 DEBUG:boto:path=/ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.135 DEBUG:boto:auth_path=/ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.135 DEBUG:boto:Method: GET 2023-09-22T21:59:12.135 DEBUG:boto:Path: /ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.135 DEBUG:boto:Data: 2023-09-22T21:59:12.135 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.135 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.135 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.135 DEBUG:boto:Params: {} 2023-09-22T21:59:12.136 DEBUG:boto:Token: None 2023-09-22T21:59:12.136 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj55?versionId=null 2023-09-22T21:59:12.136 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uQxUeDdtkaeIjMvo2TREcQiUoNQ= 2023-09-22T21:59:12.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uQxUeDdtkaeIjMvo2TREcQiUoNQ=', 'Content-Length': '0'} 2023-09-22T21:59:12.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000647a3d8118d07998-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.139 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:12.139 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:12.139 DEBUG:boto:path=/ffzdqu-61/?prefix=obj55 2023-09-22T21:59:12.139 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj55 2023-09-22T21:59:12.140 DEBUG:boto:Method: GET 2023-09-22T21:59:12.140 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj55 2023-09-22T21:59:12.140 DEBUG:boto:Data: 2023-09-22T21:59:12.140 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.140 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.140 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.140 DEBUG:boto:Params: {} 2023-09-22T21:59:12.140 DEBUG:boto:Token: None 2023-09-22T21:59:12.140 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/ 2023-09-22T21:59:12.141 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI= 2023-09-22T21:59:12.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI=', 'Content-Length': '0'} 2023-09-22T21:59:12.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aef5fb6372c155ea-00650e0e30-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.146 DEBUG:boto:b'ffzdqu-61obj551000falseobj552023-09-22T21:56:08.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:12.146 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:12.147 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:12.147 DEBUG:boto:path=/ffzdqu-61/?prefix=obj55 2023-09-22T21:59:12.147 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj55 2023-09-22T21:59:12.147 DEBUG:boto:Method: GET 2023-09-22T21:59:12.147 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj55 2023-09-22T21:59:12.147 DEBUG:boto:Data: 2023-09-22T21:59:12.147 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.147 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.147 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.148 DEBUG:boto:Params: {} 2023-09-22T21:59:12.148 DEBUG:boto:Token: None 2023-09-22T21:59:12.148 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/ 2023-09-22T21:59:12.148 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI= 2023-09-22T21:59:12.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI=', 'Content-Length': '0'} 2023-09-22T21:59:12.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edce6efb607a8d43-00650e0e30-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.198 DEBUG:boto:b'ffzdqu-61obj551000falseobj552023-09-22T21:56:08.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:12.198 DEBUG:boto:path=/ffzdqu-61/obj55 2023-09-22T21:59:12.199 DEBUG:boto:auth_path=/ffzdqu-61/obj55 2023-09-22T21:59:12.199 DEBUG:boto:Method: HEAD 2023-09-22T21:59:12.199 DEBUG:boto:Path: /ffzdqu-61/obj55 2023-09-22T21:59:12.199 DEBUG:boto:Data: 2023-09-22T21:59:12.199 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.199 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.199 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.199 DEBUG:boto:Params: {} 2023-09-22T21:59:12.200 DEBUG:boto:Token: None 2023-09-22T21:59:12.200 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj55 2023-09-22T21:59:12.200 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zZGczmt4d3HRMqP07Y4FFZZYzYU= 2023-09-22T21:59:12.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zZGczmt4d3HRMqP07Y4FFZZYzYU=', 'Content-Length': '0'} 2023-09-22T21:59:12.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3083057b11e28be-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.203 DEBUG:boto:path=/ffzdqu-61/obj55 2023-09-22T21:59:12.203 DEBUG:boto:auth_path=/ffzdqu-61/obj55 2023-09-22T21:59:12.203 DEBUG:boto:Method: HEAD 2023-09-22T21:59:12.203 DEBUG:boto:Path: /ffzdqu-61/obj55 2023-09-22T21:59:12.203 DEBUG:boto:Data: 2023-09-22T21:59:12.204 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.204 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.204 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.204 DEBUG:boto:Params: {} 2023-09-22T21:59:12.204 DEBUG:boto:Token: None 2023-09-22T21:59:12.204 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj55 2023-09-22T21:59:12.204 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zZGczmt4d3HRMqP07Y4FFZZYzYU= 2023-09-22T21:59:12.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zZGczmt4d3HRMqP07Y4FFZZYzYU=', 'Content-Length': '0'} 2023-09-22T21:59:12.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055c7e6ee16f5e961-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T21:59:12.207 DEBUG:boto:path=/ffzdqu-61/obj55 2023-09-22T21:59:12.208 DEBUG:boto:auth_path=/ffzdqu-61/obj55 2023-09-22T21:59:12.208 DEBUG:boto:Method: GET 2023-09-22T21:59:12.208 DEBUG:boto:Path: /ffzdqu-61/obj55 2023-09-22T21:59:12.208 DEBUG:boto:Data: 2023-09-22T21:59:12.208 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.208 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.208 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.208 DEBUG:boto:Params: {} 2023-09-22T21:59:12.208 DEBUG:boto:Token: None 2023-09-22T21:59:12.209 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj55 2023-09-22T21:59:12.209 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tELDJj2iyjqdODTv7Dp6F5QdC8s= 2023-09-22T21:59:12.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tELDJj2iyjqdODTv7Dp6F5QdC8s=', 'Content-Length': '0'} 2023-09-22T21:59:12.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000516cf90f4cf44719-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.212 DEBUG:boto:path=/ffzdqu-61/obj55 2023-09-22T21:59:12.212 DEBUG:boto:auth_path=/ffzdqu-61/obj55 2023-09-22T21:59:12.212 DEBUG:boto:Method: GET 2023-09-22T21:59:12.212 DEBUG:boto:Path: /ffzdqu-61/obj55 2023-09-22T21:59:12.212 DEBUG:boto:Data: 2023-09-22T21:59:12.212 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.212 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.212 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.213 DEBUG:boto:Params: {} 2023-09-22T21:59:12.213 DEBUG:boto:Token: None 2023-09-22T21:59:12.213 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj55 2023-09-22T21:59:12.213 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tELDJj2iyjqdODTv7Dp6F5QdC8s= 2023-09-22T21:59:12.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tELDJj2iyjqdODTv7Dp6F5QdC8s=', 'Content-Length': '0'} 2023-09-22T21:59:12.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035c5dc6acafd3262-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.216 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:12.217 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:12.217 DEBUG:boto:path=/ffzdqu-61/?prefix=obj55 2023-09-22T21:59:12.217 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj55 2023-09-22T21:59:12.217 DEBUG:boto:Method: GET 2023-09-22T21:59:12.217 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj55 2023-09-22T21:59:12.217 DEBUG:boto:Data: 2023-09-22T21:59:12.217 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.217 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.217 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.217 DEBUG:boto:Params: {} 2023-09-22T21:59:12.218 DEBUG:boto:Token: None 2023-09-22T21:59:12.218 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/ 2023-09-22T21:59:12.218 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI= 2023-09-22T21:59:12.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI=', 'Content-Length': '0'} 2023-09-22T21:59:12.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f89382d1d69917d0-00650e0e30-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.266 DEBUG:boto:b'ffzdqu-61obj551000falseobj552023-09-22T21:56:08.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:12.267 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:12.267 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:12.267 DEBUG:boto:path=/ffzdqu-61/?prefix=obj55 2023-09-22T21:59:12.267 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj55 2023-09-22T21:59:12.267 DEBUG:boto:Method: GET 2023-09-22T21:59:12.267 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj55 2023-09-22T21:59:12.267 DEBUG:boto:Data: 2023-09-22T21:59:12.267 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.267 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.268 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.268 DEBUG:boto:Params: {} 2023-09-22T21:59:12.268 DEBUG:boto:Token: None 2023-09-22T21:59:12.268 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/ 2023-09-22T21:59:12.268 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI= 2023-09-22T21:59:12.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI=', 'Content-Length': '0'} 2023-09-22T21:59:12.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e5c391e77e37f1b-00650e0e30-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.278 DEBUG:boto:b'ffzdqu-61obj551000falseobj552023-09-22T21:56:08.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:12.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T21:59:12.278 DEBUG:boto:path=/ffzdqu-61/obj57 2023-09-22T21:59:12.278 DEBUG:boto:auth_path=/ffzdqu-61/obj57 2023-09-22T21:59:12.279 DEBUG:boto:path=/ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.279 DEBUG:boto:auth_path=/ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.279 DEBUG:boto:Method: GET 2023-09-22T21:59:12.279 DEBUG:boto:Path: /ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.279 DEBUG:boto:Data: 2023-09-22T21:59:12.279 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.279 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.279 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.279 DEBUG:boto:Params: {} 2023-09-22T21:59:12.279 DEBUG:boto:Token: None 2023-09-22T21:59:12.280 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.280 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hmq0KhNH2UTDi3fmQwkDF9UGRjM= 2023-09-22T21:59:12.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hmq0KhNH2UTDi3fmQwkDF9UGRjM=', 'Content-Length': '0'} 2023-09-22T21:59:12.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4892bd221168b32-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.283 DEBUG:boto:path=/ffzdqu-61/obj57 2023-09-22T21:59:12.283 DEBUG:boto:auth_path=/ffzdqu-61/obj57 2023-09-22T21:59:12.283 DEBUG:boto:path=/ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.283 DEBUG:boto:auth_path=/ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.283 DEBUG:boto:Method: GET 2023-09-22T21:59:12.284 DEBUG:boto:Path: /ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.284 DEBUG:boto:Data: 2023-09-22T21:59:12.284 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.284 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.284 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.284 DEBUG:boto:Params: {} 2023-09-22T21:59:12.284 DEBUG:boto:Token: None 2023-09-22T21:59:12.284 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.284 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hmq0KhNH2UTDi3fmQwkDF9UGRjM= 2023-09-22T21:59:12.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hmq0KhNH2UTDi3fmQwkDF9UGRjM=', 'Content-Length': '0'} 2023-09-22T21:59:12.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047945671991e6153-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.334 DEBUG:boto:path=/ffzdqu-61/obj57 2023-09-22T21:59:12.334 DEBUG:boto:auth_path=/ffzdqu-61/obj57 2023-09-22T21:59:12.335 DEBUG:boto:path=/ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.335 DEBUG:boto:auth_path=/ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.335 DEBUG:boto:Method: HEAD 2023-09-22T21:59:12.335 DEBUG:boto:Path: /ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.335 DEBUG:boto:Data: 2023-09-22T21:59:12.335 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.335 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.335 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.335 DEBUG:boto:Params: {} 2023-09-22T21:59:12.335 DEBUG:boto:Token: None 2023-09-22T21:59:12.336 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.336 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:c6Mg/sQ8u0zTjGdLm//Bqbs26Aw= 2023-09-22T21:59:12.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:c6Mg/sQ8u0zTjGdLm//Bqbs26Aw=', 'Content-Length': '0'} 2023-09-22T21:59:12.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc535f5d59e9dc03-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.339 DEBUG:boto:path=/ffzdqu-61/obj57 2023-09-22T21:59:12.339 DEBUG:boto:auth_path=/ffzdqu-61/obj57 2023-09-22T21:59:12.339 DEBUG:boto:path=/ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.339 DEBUG:boto:auth_path=/ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.339 DEBUG:boto:Method: HEAD 2023-09-22T21:59:12.339 DEBUG:boto:Path: /ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.339 DEBUG:boto:Data: 2023-09-22T21:59:12.340 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.340 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.340 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.340 DEBUG:boto:Params: {} 2023-09-22T21:59:12.340 DEBUG:boto:Token: None 2023-09-22T21:59:12.340 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.340 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:c6Mg/sQ8u0zTjGdLm//Bqbs26Aw= 2023-09-22T21:59:12.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:c6Mg/sQ8u0zTjGdLm//Bqbs26Aw=', 'Content-Length': '0'} 2023-09-22T21:59:12.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4a1fb141a519cd8-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T21:59:12.343 DEBUG:boto:path=/ffzdqu-61/obj57 2023-09-22T21:59:12.344 DEBUG:boto:auth_path=/ffzdqu-61/obj57 2023-09-22T21:59:12.344 DEBUG:boto:path=/ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.344 DEBUG:boto:auth_path=/ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.344 DEBUG:boto:Method: GET 2023-09-22T21:59:12.344 DEBUG:boto:Path: /ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.344 DEBUG:boto:Data: 2023-09-22T21:59:12.344 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.344 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.344 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.344 DEBUG:boto:Params: {} 2023-09-22T21:59:12.345 DEBUG:boto:Token: None 2023-09-22T21:59:12.345 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.345 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hmq0KhNH2UTDi3fmQwkDF9UGRjM= 2023-09-22T21:59:12.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hmq0KhNH2UTDi3fmQwkDF9UGRjM=', 'Content-Length': '0'} 2023-09-22T21:59:12.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000407bb1f3efd2539f-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.348 DEBUG:boto:path=/ffzdqu-61/obj57 2023-09-22T21:59:12.348 DEBUG:boto:auth_path=/ffzdqu-61/obj57 2023-09-22T21:59:12.348 DEBUG:boto:path=/ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.348 DEBUG:boto:auth_path=/ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.348 DEBUG:boto:Method: GET 2023-09-22T21:59:12.348 DEBUG:boto:Path: /ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.348 DEBUG:boto:Data: 2023-09-22T21:59:12.349 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.349 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.349 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.349 DEBUG:boto:Params: {} 2023-09-22T21:59:12.349 DEBUG:boto:Token: None 2023-09-22T21:59:12.349 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj57?versionId=null 2023-09-22T21:59:12.349 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hmq0KhNH2UTDi3fmQwkDF9UGRjM= 2023-09-22T21:59:12.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hmq0KhNH2UTDi3fmQwkDF9UGRjM=', 'Content-Length': '0'} 2023-09-22T21:59:12.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000049f921b7de494d7-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.352 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:12.352 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:12.353 DEBUG:boto:path=/ffzdqu-61/?prefix=obj57 2023-09-22T21:59:12.353 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj57 2023-09-22T21:59:12.353 DEBUG:boto:Method: GET 2023-09-22T21:59:12.353 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj57 2023-09-22T21:59:12.353 DEBUG:boto:Data: 2023-09-22T21:59:12.353 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.353 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.353 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.353 DEBUG:boto:Params: {} 2023-09-22T21:59:12.354 DEBUG:boto:Token: None 2023-09-22T21:59:12.354 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/ 2023-09-22T21:59:12.354 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI= 2023-09-22T21:59:12.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI=', 'Content-Length': '0'} 2023-09-22T21:59:12.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b06913b7c3799d54-00650e0e30-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.406 DEBUG:boto:b'ffzdqu-61obj571000falseobj572023-09-22T21:56:09.284Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:12.407 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:12.407 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:12.407 DEBUG:boto:path=/ffzdqu-61/?prefix=obj57 2023-09-22T21:59:12.407 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj57 2023-09-22T21:59:12.407 DEBUG:boto:Method: GET 2023-09-22T21:59:12.407 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj57 2023-09-22T21:59:12.407 DEBUG:boto:Data: 2023-09-22T21:59:12.407 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.407 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.408 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.408 DEBUG:boto:Params: {} 2023-09-22T21:59:12.408 DEBUG:boto:Token: None 2023-09-22T21:59:12.408 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/ 2023-09-22T21:59:12.408 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI= 2023-09-22T21:59:12.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI=', 'Content-Length': '0'} 2023-09-22T21:59:12.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de2efee7460999cf-00650e0e30-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.418 DEBUG:boto:b'ffzdqu-61obj571000falseobj572023-09-22T21:56:09.284Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:12.418 DEBUG:boto:path=/ffzdqu-61/obj57 2023-09-22T21:59:12.418 DEBUG:boto:auth_path=/ffzdqu-61/obj57 2023-09-22T21:59:12.418 DEBUG:boto:Method: HEAD 2023-09-22T21:59:12.418 DEBUG:boto:Path: /ffzdqu-61/obj57 2023-09-22T21:59:12.419 DEBUG:boto:Data: 2023-09-22T21:59:12.419 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.419 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.419 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.419 DEBUG:boto:Params: {} 2023-09-22T21:59:12.419 DEBUG:boto:Token: None 2023-09-22T21:59:12.419 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj57 2023-09-22T21:59:12.419 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+RpDro53vTF/hrE5m7gnBxvtc4c= 2023-09-22T21:59:12.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+RpDro53vTF/hrE5m7gnBxvtc4c=', 'Content-Length': '0'} 2023-09-22T21:59:12.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002094e8359ddf24ae-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.422 DEBUG:boto:path=/ffzdqu-61/obj57 2023-09-22T21:59:12.422 DEBUG:boto:auth_path=/ffzdqu-61/obj57 2023-09-22T21:59:12.423 DEBUG:boto:Method: HEAD 2023-09-22T21:59:12.423 DEBUG:boto:Path: /ffzdqu-61/obj57 2023-09-22T21:59:12.423 DEBUG:boto:Data: 2023-09-22T21:59:12.423 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.423 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.423 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.423 DEBUG:boto:Params: {} 2023-09-22T21:59:12.423 DEBUG:boto:Token: None 2023-09-22T21:59:12.423 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj57 2023-09-22T21:59:12.424 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+RpDro53vTF/hrE5m7gnBxvtc4c= 2023-09-22T21:59:12.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+RpDro53vTF/hrE5m7gnBxvtc4c=', 'Content-Length': '0'} 2023-09-22T21:59:12.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f386f3ec73ebd11-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T21:59:12.427 DEBUG:boto:path=/ffzdqu-61/obj57 2023-09-22T21:59:12.427 DEBUG:boto:auth_path=/ffzdqu-61/obj57 2023-09-22T21:59:12.427 DEBUG:boto:Method: GET 2023-09-22T21:59:12.427 DEBUG:boto:Path: /ffzdqu-61/obj57 2023-09-22T21:59:12.428 DEBUG:boto:Data: 2023-09-22T21:59:12.428 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.428 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.428 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.428 DEBUG:boto:Params: {} 2023-09-22T21:59:12.428 DEBUG:boto:Token: None 2023-09-22T21:59:12.428 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj57 2023-09-22T21:59:12.428 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/5BS0EMRl6GJ0ZdhgH+tL9mlKLw= 2023-09-22T21:59:12.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/5BS0EMRl6GJ0ZdhgH+tL9mlKLw=', 'Content-Length': '0'} 2023-09-22T21:59:12.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007348affb05120e0c-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.431 DEBUG:boto:path=/ffzdqu-61/obj57 2023-09-22T21:59:12.432 DEBUG:boto:auth_path=/ffzdqu-61/obj57 2023-09-22T21:59:12.432 DEBUG:boto:Method: GET 2023-09-22T21:59:12.432 DEBUG:boto:Path: /ffzdqu-61/obj57 2023-09-22T21:59:12.432 DEBUG:boto:Data: 2023-09-22T21:59:12.432 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.432 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.432 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.432 DEBUG:boto:Params: {} 2023-09-22T21:59:12.432 DEBUG:boto:Token: None 2023-09-22T21:59:12.433 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj57 2023-09-22T21:59:12.433 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/5BS0EMRl6GJ0ZdhgH+tL9mlKLw= 2023-09-22T21:59:12.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/5BS0EMRl6GJ0ZdhgH+tL9mlKLw=', 'Content-Length': '0'} 2023-09-22T21:59:12.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a52db8b224d4c7b-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.482 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:12.482 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:12.483 DEBUG:boto:path=/ffzdqu-61/?prefix=obj57 2023-09-22T21:59:12.483 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj57 2023-09-22T21:59:12.483 DEBUG:boto:Method: GET 2023-09-22T21:59:12.483 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj57 2023-09-22T21:59:12.483 DEBUG:boto:Data: 2023-09-22T21:59:12.483 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.483 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.483 DEBUG:boto:Params: {} 2023-09-22T21:59:12.484 DEBUG:boto:Token: None 2023-09-22T21:59:12.484 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/ 2023-09-22T21:59:12.484 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI= 2023-09-22T21:59:12.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI=', 'Content-Length': '0'} 2023-09-22T21:59:12.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7d9bef311ab5b9b-00650e0e30-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.489 DEBUG:boto:b'ffzdqu-61obj571000falseobj572023-09-22T21:56:09.284Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:12.490 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:12.490 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:12.490 DEBUG:boto:path=/ffzdqu-61/?prefix=obj57 2023-09-22T21:59:12.490 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj57 2023-09-22T21:59:12.490 DEBUG:boto:Method: GET 2023-09-22T21:59:12.490 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj57 2023-09-22T21:59:12.490 DEBUG:boto:Data: 2023-09-22T21:59:12.491 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.491 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.491 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.491 DEBUG:boto:Params: {} 2023-09-22T21:59:12.491 DEBUG:boto:Token: None 2023-09-22T21:59:12.491 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/ 2023-09-22T21:59:12.491 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI= 2023-09-22T21:59:12.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI=', 'Content-Length': '0'} 2023-09-22T21:59:12.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a791497bc8a2927-00650e0e30-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.500 DEBUG:boto:b'ffzdqu-61obj571000falseobj572023-09-22T21:56:09.284Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:12.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-22T21:59:12.501 DEBUG:boto:path=/ffzdqu-61/obj58 2023-09-22T21:59:12.501 DEBUG:boto:auth_path=/ffzdqu-61/obj58 2023-09-22T21:59:12.501 DEBUG:boto:path=/ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.501 DEBUG:boto:auth_path=/ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.501 DEBUG:boto:Method: GET 2023-09-22T21:59:12.501 DEBUG:boto:Path: /ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.501 DEBUG:boto:Data: 2023-09-22T21:59:12.501 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.502 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.502 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.502 DEBUG:boto:Params: {} 2023-09-22T21:59:12.502 DEBUG:boto:Token: None 2023-09-22T21:59:12.502 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.502 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZSv3QVZkb5moDTLf/XPldR3JlTs= 2023-09-22T21:59:12.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZSv3QVZkb5moDTLf/XPldR3JlTs=', 'Content-Length': '0'} 2023-09-22T21:59:12.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000840596fb85e2aa29-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.505 DEBUG:boto:path=/ffzdqu-61/obj58 2023-09-22T21:59:12.505 DEBUG:boto:auth_path=/ffzdqu-61/obj58 2023-09-22T21:59:12.505 DEBUG:boto:path=/ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.506 DEBUG:boto:auth_path=/ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.506 DEBUG:boto:Method: GET 2023-09-22T21:59:12.506 DEBUG:boto:Path: /ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.506 DEBUG:boto:Data: 2023-09-22T21:59:12.506 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.506 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.506 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.506 DEBUG:boto:Params: {} 2023-09-22T21:59:12.506 DEBUG:boto:Token: None 2023-09-22T21:59:12.507 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.507 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZSv3QVZkb5moDTLf/XPldR3JlTs= 2023-09-22T21:59:12.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZSv3QVZkb5moDTLf/XPldR3JlTs=', 'Content-Length': '0'} 2023-09-22T21:59:12.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026f3973be8ce1698-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.510 DEBUG:boto:path=/ffzdqu-61/obj58 2023-09-22T21:59:12.510 DEBUG:boto:auth_path=/ffzdqu-61/obj58 2023-09-22T21:59:12.511 DEBUG:boto:path=/ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.511 DEBUG:boto:auth_path=/ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.511 DEBUG:boto:Method: HEAD 2023-09-22T21:59:12.511 DEBUG:boto:Path: /ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.511 DEBUG:boto:Data: 2023-09-22T21:59:12.511 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.511 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.511 DEBUG:boto:Params: {} 2023-09-22T21:59:12.511 DEBUG:boto:Token: None 2023-09-22T21:59:12.512 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.512 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h3GNZd51JL7QjmKYg2bFBdc7KDY= 2023-09-22T21:59:12.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h3GNZd51JL7QjmKYg2bFBdc7KDY=', 'Content-Length': '0'} 2023-09-22T21:59:12.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091b32b90cd182be8-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.515 DEBUG:boto:path=/ffzdqu-61/obj58 2023-09-22T21:59:12.515 DEBUG:boto:auth_path=/ffzdqu-61/obj58 2023-09-22T21:59:12.515 DEBUG:boto:path=/ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.515 DEBUG:boto:auth_path=/ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.515 DEBUG:boto:Method: HEAD 2023-09-22T21:59:12.515 DEBUG:boto:Path: /ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.515 DEBUG:boto:Data: 2023-09-22T21:59:12.515 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.516 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.516 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.516 DEBUG:boto:Params: {} 2023-09-22T21:59:12.516 DEBUG:boto:Token: None 2023-09-22T21:59:12.516 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.516 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h3GNZd51JL7QjmKYg2bFBdc7KDY= 2023-09-22T21:59:12.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h3GNZd51JL7QjmKYg2bFBdc7KDY=', 'Content-Length': '0'} 2023-09-22T21:59:12.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012bd2fe8ba346939-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-22T21:59:12.519 DEBUG:boto:path=/ffzdqu-61/obj58 2023-09-22T21:59:12.519 DEBUG:boto:auth_path=/ffzdqu-61/obj58 2023-09-22T21:59:12.520 DEBUG:boto:path=/ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.520 DEBUG:boto:auth_path=/ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.520 DEBUG:boto:Method: GET 2023-09-22T21:59:12.520 DEBUG:boto:Path: /ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.520 DEBUG:boto:Data: 2023-09-22T21:59:12.520 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.520 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.520 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.520 DEBUG:boto:Params: {} 2023-09-22T21:59:12.521 DEBUG:boto:Token: None 2023-09-22T21:59:12.521 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.521 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZSv3QVZkb5moDTLf/XPldR3JlTs= 2023-09-22T21:59:12.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZSv3QVZkb5moDTLf/XPldR3JlTs=', 'Content-Length': '0'} 2023-09-22T21:59:12.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e04ff6e7c2798da3-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.566 DEBUG:boto:path=/ffzdqu-61/obj58 2023-09-22T21:59:12.567 DEBUG:boto:auth_path=/ffzdqu-61/obj58 2023-09-22T21:59:12.567 DEBUG:boto:path=/ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.567 DEBUG:boto:auth_path=/ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.567 DEBUG:boto:Method: GET 2023-09-22T21:59:12.567 DEBUG:boto:Path: /ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.567 DEBUG:boto:Data: 2023-09-22T21:59:12.567 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.567 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.567 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.568 DEBUG:boto:Params: {} 2023-09-22T21:59:12.568 DEBUG:boto:Token: None 2023-09-22T21:59:12.568 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj58?versionId=null 2023-09-22T21:59:12.568 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZSv3QVZkb5moDTLf/XPldR3JlTs= 2023-09-22T21:59:12.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZSv3QVZkb5moDTLf/XPldR3JlTs=', 'Content-Length': '0'} 2023-09-22T21:59:12.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a388fa4e5db0f97a-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.572 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:12.572 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:12.572 DEBUG:boto:path=/ffzdqu-61/?prefix=obj58 2023-09-22T21:59:12.572 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj58 2023-09-22T21:59:12.572 DEBUG:boto:Method: GET 2023-09-22T21:59:12.572 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj58 2023-09-22T21:59:12.572 DEBUG:boto:Data: 2023-09-22T21:59:12.572 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.572 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.573 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.573 DEBUG:boto:Params: {} 2023-09-22T21:59:12.573 DEBUG:boto:Token: None 2023-09-22T21:59:12.573 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/ 2023-09-22T21:59:12.573 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI= 2023-09-22T21:59:12.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI=', 'Content-Length': '0'} 2023-09-22T21:59:12.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b38c9749d8115ec0-00650e0e30-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.578 DEBUG:boto:b'ffzdqu-61obj581000falseobj582023-09-22T21:56:09.943Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:12.578 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:12.578 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:12.578 DEBUG:boto:path=/ffzdqu-61/?prefix=obj58 2023-09-22T21:59:12.578 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj58 2023-09-22T21:59:12.579 DEBUG:boto:Method: GET 2023-09-22T21:59:12.579 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj58 2023-09-22T21:59:12.579 DEBUG:boto:Data: 2023-09-22T21:59:12.579 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.579 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.579 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.579 DEBUG:boto:Params: {} 2023-09-22T21:59:12.579 DEBUG:boto:Token: None 2023-09-22T21:59:12.579 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/ 2023-09-22T21:59:12.580 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI= 2023-09-22T21:59:12.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI=', 'Content-Length': '0'} 2023-09-22T21:59:12.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1a622aa9dc6ac51-00650e0e30-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.634 DEBUG:boto:b'ffzdqu-61obj581000falseobj582023-09-22T21:56:09.943Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:12.635 DEBUG:boto:path=/ffzdqu-61/obj58 2023-09-22T21:59:12.635 DEBUG:boto:auth_path=/ffzdqu-61/obj58 2023-09-22T21:59:12.635 DEBUG:boto:Method: HEAD 2023-09-22T21:59:12.635 DEBUG:boto:Path: /ffzdqu-61/obj58 2023-09-22T21:59:12.635 DEBUG:boto:Data: 2023-09-22T21:59:12.635 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.635 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.635 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.635 DEBUG:boto:Params: {} 2023-09-22T21:59:12.636 DEBUG:boto:Token: None 2023-09-22T21:59:12.636 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj58 2023-09-22T21:59:12.636 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71tFUY0obqL8+n73Gbs6jSZ4Pb4= 2023-09-22T21:59:12.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71tFUY0obqL8+n73Gbs6jSZ4Pb4=', 'Content-Length': '0'} 2023-09-22T21:59:12.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9e642751094e749-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.639 DEBUG:boto:path=/ffzdqu-61/obj58 2023-09-22T21:59:12.639 DEBUG:boto:auth_path=/ffzdqu-61/obj58 2023-09-22T21:59:12.639 DEBUG:boto:Method: HEAD 2023-09-22T21:59:12.639 DEBUG:boto:Path: /ffzdqu-61/obj58 2023-09-22T21:59:12.639 DEBUG:boto:Data: 2023-09-22T21:59:12.639 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.639 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.640 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.640 DEBUG:boto:Params: {} 2023-09-22T21:59:12.640 DEBUG:boto:Token: None 2023-09-22T21:59:12.640 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj58 2023-09-22T21:59:12.640 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:71tFUY0obqL8+n73Gbs6jSZ4Pb4= 2023-09-22T21:59:12.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:71tFUY0obqL8+n73Gbs6jSZ4Pb4=', 'Content-Length': '0'} 2023-09-22T21:59:12.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009aac7c8bf99a0eaa-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-22T21:59:12.643 DEBUG:boto:path=/ffzdqu-61/obj58 2023-09-22T21:59:12.643 DEBUG:boto:auth_path=/ffzdqu-61/obj58 2023-09-22T21:59:12.643 DEBUG:boto:Method: GET 2023-09-22T21:59:12.644 DEBUG:boto:Path: /ffzdqu-61/obj58 2023-09-22T21:59:12.644 DEBUG:boto:Data: 2023-09-22T21:59:12.644 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.644 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.644 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.644 DEBUG:boto:Params: {} 2023-09-22T21:59:12.644 DEBUG:boto:Token: None 2023-09-22T21:59:12.644 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj58 2023-09-22T21:59:12.645 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xMJ0i6qM+5kwm9/au9JlMW3eU90= 2023-09-22T21:59:12.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xMJ0i6qM+5kwm9/au9JlMW3eU90=', 'Content-Length': '0'} 2023-09-22T21:59:12.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a39275c53ac1ff9-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.648 DEBUG:boto:path=/ffzdqu-61/obj58 2023-09-22T21:59:12.648 DEBUG:boto:auth_path=/ffzdqu-61/obj58 2023-09-22T21:59:12.648 DEBUG:boto:Method: GET 2023-09-22T21:59:12.648 DEBUG:boto:Path: /ffzdqu-61/obj58 2023-09-22T21:59:12.648 DEBUG:boto:Data: 2023-09-22T21:59:12.648 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.648 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.648 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.648 DEBUG:boto:Params: {} 2023-09-22T21:59:12.648 DEBUG:boto:Token: None 2023-09-22T21:59:12.649 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj58 2023-09-22T21:59:12.649 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xMJ0i6qM+5kwm9/au9JlMW3eU90= 2023-09-22T21:59:12.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xMJ0i6qM+5kwm9/au9JlMW3eU90=', 'Content-Length': '0'} 2023-09-22T21:59:12.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cb42f808066e8ec-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.652 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:12.652 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:12.652 DEBUG:boto:path=/ffzdqu-61/?prefix=obj58 2023-09-22T21:59:12.652 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj58 2023-09-22T21:59:12.652 DEBUG:boto:Method: GET 2023-09-22T21:59:12.652 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj58 2023-09-22T21:59:12.652 DEBUG:boto:Data: 2023-09-22T21:59:12.652 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.653 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.653 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.653 DEBUG:boto:Params: {} 2023-09-22T21:59:12.653 DEBUG:boto:Token: None 2023-09-22T21:59:12.653 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/ 2023-09-22T21:59:12.653 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI= 2023-09-22T21:59:12.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI=', 'Content-Length': '0'} 2023-09-22T21:59:12.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa322757fca53a35-00650e0e30-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.702 DEBUG:boto:b'ffzdqu-61obj581000falseobj582023-09-22T21:56:09.943Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:12.703 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:12.703 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:12.703 DEBUG:boto:path=/ffzdqu-61/?prefix=obj58 2023-09-22T21:59:12.703 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj58 2023-09-22T21:59:12.703 DEBUG:boto:Method: GET 2023-09-22T21:59:12.703 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj58 2023-09-22T21:59:12.703 DEBUG:boto:Data: 2023-09-22T21:59:12.703 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.703 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.704 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.704 DEBUG:boto:Params: {} 2023-09-22T21:59:12.704 DEBUG:boto:Token: None 2023-09-22T21:59:12.704 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/ 2023-09-22T21:59:12.704 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI= 2023-09-22T21:59:12.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI=', 'Content-Length': '0'} 2023-09-22T21:59:12.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a73433580b5f9822-00650e0e30-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.713 DEBUG:boto:b'ffzdqu-61obj581000falseobj582023-09-22T21:56:09.943Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:12.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-22T21:59:12.714 DEBUG:boto:path=/ffzdqu-61/obj62 2023-09-22T21:59:12.714 DEBUG:boto:auth_path=/ffzdqu-61/obj62 2023-09-22T21:59:12.714 DEBUG:boto:path=/ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.714 DEBUG:boto:auth_path=/ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.714 DEBUG:boto:Method: GET 2023-09-22T21:59:12.714 DEBUG:boto:Path: /ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.715 DEBUG:boto:Data: 2023-09-22T21:59:12.715 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.715 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.715 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.715 DEBUG:boto:Params: {} 2023-09-22T21:59:12.715 DEBUG:boto:Token: None 2023-09-22T21:59:12.715 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.715 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jdvkiIHbQ1VPNwOsPbNESASoTlI= 2023-09-22T21:59:12.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jdvkiIHbQ1VPNwOsPbNESASoTlI=', 'Content-Length': '0'} 2023-09-22T21:59:12.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025c17d0cecec76c8-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.719 DEBUG:boto:path=/ffzdqu-61/obj62 2023-09-22T21:59:12.719 DEBUG:boto:auth_path=/ffzdqu-61/obj62 2023-09-22T21:59:12.719 DEBUG:boto:path=/ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.719 DEBUG:boto:auth_path=/ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.719 DEBUG:boto:Method: GET 2023-09-22T21:59:12.719 DEBUG:boto:Path: /ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.719 DEBUG:boto:Data: 2023-09-22T21:59:12.720 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.720 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.720 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.720 DEBUG:boto:Params: {} 2023-09-22T21:59:12.720 DEBUG:boto:Token: None 2023-09-22T21:59:12.720 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.720 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jdvkiIHbQ1VPNwOsPbNESASoTlI= 2023-09-22T21:59:12.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jdvkiIHbQ1VPNwOsPbNESASoTlI=', 'Content-Length': '0'} 2023-09-22T21:59:12.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002498cd5e45432001-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.770 DEBUG:boto:path=/ffzdqu-61/obj62 2023-09-22T21:59:12.770 DEBUG:boto:auth_path=/ffzdqu-61/obj62 2023-09-22T21:59:12.770 DEBUG:boto:path=/ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.771 DEBUG:boto:auth_path=/ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.771 DEBUG:boto:Method: HEAD 2023-09-22T21:59:12.771 DEBUG:boto:Path: /ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.771 DEBUG:boto:Data: 2023-09-22T21:59:12.771 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.771 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.771 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.771 DEBUG:boto:Params: {} 2023-09-22T21:59:12.771 DEBUG:boto:Token: None 2023-09-22T21:59:12.772 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.772 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:s/hjJqq4JPbDchWYU1eOwaEEdjQ= 2023-09-22T21:59:12.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:s/hjJqq4JPbDchWYU1eOwaEEdjQ=', 'Content-Length': '0'} 2023-09-22T21:59:12.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b990e9f44b9e89b5-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.775 DEBUG:boto:path=/ffzdqu-61/obj62 2023-09-22T21:59:12.775 DEBUG:boto:auth_path=/ffzdqu-61/obj62 2023-09-22T21:59:12.775 DEBUG:boto:path=/ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.775 DEBUG:boto:auth_path=/ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.775 DEBUG:boto:Method: HEAD 2023-09-22T21:59:12.775 DEBUG:boto:Path: /ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.775 DEBUG:boto:Data: 2023-09-22T21:59:12.775 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.776 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.776 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.776 DEBUG:boto:Params: {} 2023-09-22T21:59:12.776 DEBUG:boto:Token: None 2023-09-22T21:59:12.776 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.776 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:s/hjJqq4JPbDchWYU1eOwaEEdjQ= 2023-09-22T21:59:12.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:s/hjJqq4JPbDchWYU1eOwaEEdjQ=', 'Content-Length': '0'} 2023-09-22T21:59:12.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a9e0b16915d1b8c-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-22T21:59:12.779 DEBUG:boto:path=/ffzdqu-61/obj62 2023-09-22T21:59:12.780 DEBUG:boto:auth_path=/ffzdqu-61/obj62 2023-09-22T21:59:12.780 DEBUG:boto:path=/ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.780 DEBUG:boto:auth_path=/ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.780 DEBUG:boto:Method: GET 2023-09-22T21:59:12.780 DEBUG:boto:Path: /ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.780 DEBUG:boto:Data: 2023-09-22T21:59:12.780 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.780 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.780 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.780 DEBUG:boto:Params: {} 2023-09-22T21:59:12.781 DEBUG:boto:Token: None 2023-09-22T21:59:12.781 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.781 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jdvkiIHbQ1VPNwOsPbNESASoTlI= 2023-09-22T21:59:12.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jdvkiIHbQ1VPNwOsPbNESASoTlI=', 'Content-Length': '0'} 2023-09-22T21:59:12.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005da6b50b60d69e7b-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.784 DEBUG:boto:path=/ffzdqu-61/obj62 2023-09-22T21:59:12.784 DEBUG:boto:auth_path=/ffzdqu-61/obj62 2023-09-22T21:59:12.784 DEBUG:boto:path=/ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.784 DEBUG:boto:auth_path=/ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.785 DEBUG:boto:Method: GET 2023-09-22T21:59:12.785 DEBUG:boto:Path: /ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.785 DEBUG:boto:Data: 2023-09-22T21:59:12.785 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.785 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.785 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.785 DEBUG:boto:Params: {} 2023-09-22T21:59:12.785 DEBUG:boto:Token: None 2023-09-22T21:59:12.785 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj62?versionId=null 2023-09-22T21:59:12.786 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jdvkiIHbQ1VPNwOsPbNESASoTlI= 2023-09-22T21:59:12.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jdvkiIHbQ1VPNwOsPbNESASoTlI=', 'Content-Length': '0'} 2023-09-22T21:59:12.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a365f4cc67bfdbc2-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.788 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:12.789 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:12.789 DEBUG:boto:path=/ffzdqu-61/?prefix=obj62 2023-09-22T21:59:12.789 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj62 2023-09-22T21:59:12.789 DEBUG:boto:Method: GET 2023-09-22T21:59:12.789 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj62 2023-09-22T21:59:12.789 DEBUG:boto:Data: 2023-09-22T21:59:12.789 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.789 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.789 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.789 DEBUG:boto:Params: {} 2023-09-22T21:59:12.790 DEBUG:boto:Token: None 2023-09-22T21:59:12.790 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/ 2023-09-22T21:59:12.790 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI= 2023-09-22T21:59:12.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI=', 'Content-Length': '0'} 2023-09-22T21:59:12.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044b2fd85f37a27cc-00650e0e30-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.838 DEBUG:boto:b'ffzdqu-61obj621000falseobj622023-09-22T21:56:10.292Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:12.839 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:12.839 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:12.839 DEBUG:boto:path=/ffzdqu-61/?prefix=obj62 2023-09-22T21:59:12.839 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj62 2023-09-22T21:59:12.839 DEBUG:boto:Method: GET 2023-09-22T21:59:12.839 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj62 2023-09-22T21:59:12.839 DEBUG:boto:Data: 2023-09-22T21:59:12.839 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.840 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.840 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.840 DEBUG:boto:Params: {} 2023-09-22T21:59:12.840 DEBUG:boto:Token: None 2023-09-22T21:59:12.840 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/ 2023-09-22T21:59:12.840 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI= 2023-09-22T21:59:12.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI=', 'Content-Length': '0'} 2023-09-22T21:59:12.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adf2a07f704662f8-00650e0e30-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.849 DEBUG:boto:b'ffzdqu-61obj621000falseobj622023-09-22T21:56:10.292Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:12.850 DEBUG:boto:path=/ffzdqu-61/obj62 2023-09-22T21:59:12.850 DEBUG:boto:auth_path=/ffzdqu-61/obj62 2023-09-22T21:59:12.850 DEBUG:boto:Method: HEAD 2023-09-22T21:59:12.850 DEBUG:boto:Path: /ffzdqu-61/obj62 2023-09-22T21:59:12.850 DEBUG:boto:Data: 2023-09-22T21:59:12.850 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.850 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.850 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.851 DEBUG:boto:Params: {} 2023-09-22T21:59:12.851 DEBUG:boto:Token: None 2023-09-22T21:59:12.851 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj62 2023-09-22T21:59:12.851 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b2Y22dZGXPJun1TW9rS4AgvU+XU= 2023-09-22T21:59:12.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b2Y22dZGXPJun1TW9rS4AgvU+XU=', 'Content-Length': '0'} 2023-09-22T21:59:12.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003364c45beb9c8e9b-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.854 DEBUG:boto:path=/ffzdqu-61/obj62 2023-09-22T21:59:12.854 DEBUG:boto:auth_path=/ffzdqu-61/obj62 2023-09-22T21:59:12.854 DEBUG:boto:Method: HEAD 2023-09-22T21:59:12.854 DEBUG:boto:Path: /ffzdqu-61/obj62 2023-09-22T21:59:12.854 DEBUG:boto:Data: 2023-09-22T21:59:12.854 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.855 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.855 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.855 DEBUG:boto:Params: {} 2023-09-22T21:59:12.855 DEBUG:boto:Token: None 2023-09-22T21:59:12.855 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj62 2023-09-22T21:59:12.855 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:b2Y22dZGXPJun1TW9rS4AgvU+XU= 2023-09-22T21:59:12.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:b2Y22dZGXPJun1TW9rS4AgvU+XU=', 'Content-Length': '0'} 2023-09-22T21:59:12.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ce8824c35e4ea0b-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-22T21:59:12.858 DEBUG:boto:path=/ffzdqu-61/obj62 2023-09-22T21:59:12.858 DEBUG:boto:auth_path=/ffzdqu-61/obj62 2023-09-22T21:59:12.858 DEBUG:boto:Method: GET 2023-09-22T21:59:12.859 DEBUG:boto:Path: /ffzdqu-61/obj62 2023-09-22T21:59:12.859 DEBUG:boto:Data: 2023-09-22T21:59:12.859 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.859 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.859 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.859 DEBUG:boto:Params: {} 2023-09-22T21:59:12.859 DEBUG:boto:Token: None 2023-09-22T21:59:12.859 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj62 2023-09-22T21:59:12.859 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EuwmbrAEklx+Xsf3/epmJMr+OZ4= 2023-09-22T21:59:12.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EuwmbrAEklx+Xsf3/epmJMr+OZ4=', 'Content-Length': '0'} 2023-09-22T21:59:12.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4c3e501fed2ff2f-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.862 DEBUG:boto:path=/ffzdqu-61/obj62 2023-09-22T21:59:12.863 DEBUG:boto:auth_path=/ffzdqu-61/obj62 2023-09-22T21:59:12.863 DEBUG:boto:Method: GET 2023-09-22T21:59:12.863 DEBUG:boto:Path: /ffzdqu-61/obj62 2023-09-22T21:59:12.863 DEBUG:boto:Data: 2023-09-22T21:59:12.863 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.863 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.863 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.863 DEBUG:boto:Params: {} 2023-09-22T21:59:12.863 DEBUG:boto:Token: None 2023-09-22T21:59:12.864 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj62 2023-09-22T21:59:12.864 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EuwmbrAEklx+Xsf3/epmJMr+OZ4= 2023-09-22T21:59:12.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EuwmbrAEklx+Xsf3/epmJMr+OZ4=', 'Content-Length': '0'} 2023-09-22T21:59:12.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e586e110de2c3dd6-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.910 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:12.910 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:12.910 DEBUG:boto:path=/ffzdqu-61/?prefix=obj62 2023-09-22T21:59:12.911 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj62 2023-09-22T21:59:12.911 DEBUG:boto:Method: GET 2023-09-22T21:59:12.911 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj62 2023-09-22T21:59:12.911 DEBUG:boto:Data: 2023-09-22T21:59:12.911 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.911 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.911 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.911 DEBUG:boto:Params: {} 2023-09-22T21:59:12.911 DEBUG:boto:Token: None 2023-09-22T21:59:12.912 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/ 2023-09-22T21:59:12.912 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI= 2023-09-22T21:59:12.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI=', 'Content-Length': '0'} 2023-09-22T21:59:12.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050c344e844b63e25-00650e0e30-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.916 DEBUG:boto:b'ffzdqu-61obj621000falseobj622023-09-22T21:56:10.292Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:12.917 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:12.917 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:12.917 DEBUG:boto:path=/ffzdqu-61/?prefix=obj62 2023-09-22T21:59:12.917 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj62 2023-09-22T21:59:12.917 DEBUG:boto:Method: GET 2023-09-22T21:59:12.917 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj62 2023-09-22T21:59:12.917 DEBUG:boto:Data: 2023-09-22T21:59:12.918 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.918 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.918 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.918 DEBUG:boto:Params: {} 2023-09-22T21:59:12.918 DEBUG:boto:Token: None 2023-09-22T21:59:12.918 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/ 2023-09-22T21:59:12.918 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI= 2023-09-22T21:59:12.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S+H9IynkKdNZoSSvZYNEhXnx9XI=', 'Content-Length': '0'} 2023-09-22T21:59:12.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e44f4ea0942c9837-00650e0e30-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.932 DEBUG:boto:b'ffzdqu-61obj621000falseobj622023-09-22T21:56:10.292Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:12.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:59:12.933 DEBUG:boto:path=/ffzdqu-61/obj68 2023-09-22T21:59:12.933 DEBUG:boto:auth_path=/ffzdqu-61/obj68 2023-09-22T21:59:12.933 DEBUG:boto:path=/ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.933 DEBUG:boto:auth_path=/ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.933 DEBUG:boto:Method: GET 2023-09-22T21:59:12.933 DEBUG:boto:Path: /ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.934 DEBUG:boto:Data: 2023-09-22T21:59:12.934 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.934 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.934 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.934 DEBUG:boto:Params: {} 2023-09-22T21:59:12.934 DEBUG:boto:Token: None 2023-09-22T21:59:12.934 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.934 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iPnMNI0g8C7sw/EOKstTQaDyh7w= 2023-09-22T21:59:12.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iPnMNI0g8C7sw/EOKstTQaDyh7w=', 'Content-Length': '0'} 2023-09-22T21:59:12.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000886329f7ab261f7b-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.938 DEBUG:boto:path=/ffzdqu-61/obj68 2023-09-22T21:59:12.938 DEBUG:boto:auth_path=/ffzdqu-61/obj68 2023-09-22T21:59:12.938 DEBUG:boto:path=/ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.938 DEBUG:boto:auth_path=/ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.938 DEBUG:boto:Method: GET 2023-09-22T21:59:12.938 DEBUG:boto:Path: /ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.938 DEBUG:boto:Data: 2023-09-22T21:59:12.938 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.939 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.939 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.939 DEBUG:boto:Params: {} 2023-09-22T21:59:12.939 DEBUG:boto:Token: None 2023-09-22T21:59:12.939 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.939 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iPnMNI0g8C7sw/EOKstTQaDyh7w= 2023-09-22T21:59:12.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iPnMNI0g8C7sw/EOKstTQaDyh7w=', 'Content-Length': '0'} 2023-09-22T21:59:12.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022ba68676d4f89b0-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.942 DEBUG:boto:path=/ffzdqu-61/obj68 2023-09-22T21:59:12.942 DEBUG:boto:auth_path=/ffzdqu-61/obj68 2023-09-22T21:59:12.943 DEBUG:boto:path=/ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.943 DEBUG:boto:auth_path=/ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.943 DEBUG:boto:Method: HEAD 2023-09-22T21:59:12.943 DEBUG:boto:Path: /ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.943 DEBUG:boto:Data: 2023-09-22T21:59:12.943 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.943 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.943 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.944 DEBUG:boto:Params: {} 2023-09-22T21:59:12.944 DEBUG:boto:Token: None 2023-09-22T21:59:12.944 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.944 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:s/KehSiwaOy1qGkPVukrh80dBy0= 2023-09-22T21:59:12.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:s/KehSiwaOy1qGkPVukrh80dBy0=', 'Content-Length': '0'} 2023-09-22T21:59:12.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b784b55df52e509a-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.947 DEBUG:boto:path=/ffzdqu-61/obj68 2023-09-22T21:59:12.947 DEBUG:boto:auth_path=/ffzdqu-61/obj68 2023-09-22T21:59:12.947 DEBUG:boto:path=/ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.947 DEBUG:boto:auth_path=/ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.947 DEBUG:boto:Method: HEAD 2023-09-22T21:59:12.948 DEBUG:boto:Path: /ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.948 DEBUG:boto:Data: 2023-09-22T21:59:12.948 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.948 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.948 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.948 DEBUG:boto:Params: {} 2023-09-22T21:59:12.948 DEBUG:boto:Token: None 2023-09-22T21:59:12.948 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.948 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:s/KehSiwaOy1qGkPVukrh80dBy0= 2023-09-22T21:59:12.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:s/KehSiwaOy1qGkPVukrh80dBy0=', 'Content-Length': '0'} 2023-09-22T21:59:12.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8cbaa5ba31023ee-00650e0e30-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:59:12.952 DEBUG:boto:path=/ffzdqu-61/obj68 2023-09-22T21:59:12.952 DEBUG:boto:auth_path=/ffzdqu-61/obj68 2023-09-22T21:59:12.952 DEBUG:boto:path=/ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.952 DEBUG:boto:auth_path=/ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.952 DEBUG:boto:Method: GET 2023-09-22T21:59:12.952 DEBUG:boto:Path: /ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.952 DEBUG:boto:Data: 2023-09-22T21:59:12.953 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.953 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:12.953 DEBUG:boto:Port: 8001 2023-09-22T21:59:12.953 DEBUG:boto:Params: {} 2023-09-22T21:59:12.953 DEBUG:boto:Token: None 2023-09-22T21:59:12.953 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:12 GMT /ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.953 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iPnMNI0g8C7sw/EOKstTQaDyh7w= 2023-09-22T21:59:12.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:12 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iPnMNI0g8C7sw/EOKstTQaDyh7w=', 'Content-Length': '0'} 2023-09-22T21:59:12.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051dd30f5d9ff63ad-00650e0e30-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:12.998 DEBUG:boto:path=/ffzdqu-61/obj68 2023-09-22T21:59:12.998 DEBUG:boto:auth_path=/ffzdqu-61/obj68 2023-09-22T21:59:12.999 DEBUG:boto:path=/ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.999 DEBUG:boto:auth_path=/ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.999 DEBUG:boto:Method: GET 2023-09-22T21:59:12.999 DEBUG:boto:Path: /ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:12.999 DEBUG:boto:Data: 2023-09-22T21:59:12.999 DEBUG:boto:Headers: {} 2023-09-22T21:59:12.999 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:12.999 DEBUG:boto:Port: 8000 2023-09-22T21:59:12.999 DEBUG:boto:Params: {} 2023-09-22T21:59:13.000 DEBUG:boto:Token: None 2023-09-22T21:59:13.000 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj68?versionId=null 2023-09-22T21:59:13.000 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:78NB/8JXj1fB3lhMlh2gHxS05eM= 2023-09-22T21:59:13.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:78NB/8JXj1fB3lhMlh2gHxS05eM=', 'Content-Length': '0'} 2023-09-22T21:59:13.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4e4824e98be8c84-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.003 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:13.003 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:13.003 DEBUG:boto:path=/ffzdqu-61/?prefix=obj68 2023-09-22T21:59:13.003 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj68 2023-09-22T21:59:13.004 DEBUG:boto:Method: GET 2023-09-22T21:59:13.004 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj68 2023-09-22T21:59:13.004 DEBUG:boto:Data: 2023-09-22T21:59:13.004 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.004 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.004 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.004 DEBUG:boto:Params: {} 2023-09-22T21:59:13.004 DEBUG:boto:Token: None 2023-09-22T21:59:13.004 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/ 2023-09-22T21:59:13.005 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw= 2023-09-22T21:59:13.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw=', 'Content-Length': '0'} 2023-09-22T21:59:13.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d32e72b7d0481dc-00650e0e31-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.009 DEBUG:boto:b'ffzdqu-61obj681000falseobj682023-09-22T21:56:09.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:13.010 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:13.010 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:13.010 DEBUG:boto:path=/ffzdqu-61/?prefix=obj68 2023-09-22T21:59:13.010 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj68 2023-09-22T21:59:13.010 DEBUG:boto:Method: GET 2023-09-22T21:59:13.010 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj68 2023-09-22T21:59:13.011 DEBUG:boto:Data: 2023-09-22T21:59:13.011 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.011 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.011 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.011 DEBUG:boto:Params: {} 2023-09-22T21:59:13.011 DEBUG:boto:Token: None 2023-09-22T21:59:13.011 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/ 2023-09-22T21:59:13.011 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw= 2023-09-22T21:59:13.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw=', 'Content-Length': '0'} 2023-09-22T21:59:13.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6db76f2895131f0-00650e0e31-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.062 DEBUG:boto:b'ffzdqu-61obj681000falseobj682023-09-22T21:56:09.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:13.063 DEBUG:boto:path=/ffzdqu-61/obj68 2023-09-22T21:59:13.063 DEBUG:boto:auth_path=/ffzdqu-61/obj68 2023-09-22T21:59:13.063 DEBUG:boto:Method: HEAD 2023-09-22T21:59:13.063 DEBUG:boto:Path: /ffzdqu-61/obj68 2023-09-22T21:59:13.063 DEBUG:boto:Data: 2023-09-22T21:59:13.063 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.063 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.063 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.063 DEBUG:boto:Params: {} 2023-09-22T21:59:13.064 DEBUG:boto:Token: None 2023-09-22T21:59:13.064 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj68 2023-09-22T21:59:13.064 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9ySrTMFuRXz1182BxpXHdQCBnOE= 2023-09-22T21:59:13.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9ySrTMFuRXz1182BxpXHdQCBnOE=', 'Content-Length': '0'} 2023-09-22T21:59:13.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5757f1b3404e9d5-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.067 DEBUG:boto:path=/ffzdqu-61/obj68 2023-09-22T21:59:13.067 DEBUG:boto:auth_path=/ffzdqu-61/obj68 2023-09-22T21:59:13.067 DEBUG:boto:Method: HEAD 2023-09-22T21:59:13.067 DEBUG:boto:Path: /ffzdqu-61/obj68 2023-09-22T21:59:13.067 DEBUG:boto:Data: 2023-09-22T21:59:13.067 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.068 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.068 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.068 DEBUG:boto:Params: {} 2023-09-22T21:59:13.068 DEBUG:boto:Token: None 2023-09-22T21:59:13.068 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj68 2023-09-22T21:59:13.068 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:9ySrTMFuRXz1182BxpXHdQCBnOE= 2023-09-22T21:59:13.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:9ySrTMFuRXz1182BxpXHdQCBnOE=', 'Content-Length': '0'} 2023-09-22T21:59:13.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055f477aaf1e87731-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T21:59:13.071 DEBUG:boto:path=/ffzdqu-61/obj68 2023-09-22T21:59:13.071 DEBUG:boto:auth_path=/ffzdqu-61/obj68 2023-09-22T21:59:13.071 DEBUG:boto:Method: GET 2023-09-22T21:59:13.071 DEBUG:boto:Path: /ffzdqu-61/obj68 2023-09-22T21:59:13.072 DEBUG:boto:Data: 2023-09-22T21:59:13.072 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.072 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.072 DEBUG:boto:Params: {} 2023-09-22T21:59:13.072 DEBUG:boto:Token: None 2023-09-22T21:59:13.072 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj68 2023-09-22T21:59:13.072 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gr2NpX5+VuYypMlDXvXnY0A8kis= 2023-09-22T21:59:13.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gr2NpX5+VuYypMlDXvXnY0A8kis=', 'Content-Length': '0'} 2023-09-22T21:59:13.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e209cde65b35f079-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.075 DEBUG:boto:path=/ffzdqu-61/obj68 2023-09-22T21:59:13.075 DEBUG:boto:auth_path=/ffzdqu-61/obj68 2023-09-22T21:59:13.075 DEBUG:boto:Method: GET 2023-09-22T21:59:13.076 DEBUG:boto:Path: /ffzdqu-61/obj68 2023-09-22T21:59:13.076 DEBUG:boto:Data: 2023-09-22T21:59:13.076 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.076 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.076 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.076 DEBUG:boto:Params: {} 2023-09-22T21:59:13.076 DEBUG:boto:Token: None 2023-09-22T21:59:13.076 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj68 2023-09-22T21:59:13.076 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gr2NpX5+VuYypMlDXvXnY0A8kis= 2023-09-22T21:59:13.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gr2NpX5+VuYypMlDXvXnY0A8kis=', 'Content-Length': '0'} 2023-09-22T21:59:13.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4293a622cced2e8-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.079 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:13.080 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:13.080 DEBUG:boto:path=/ffzdqu-61/?prefix=obj68 2023-09-22T21:59:13.080 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj68 2023-09-22T21:59:13.080 DEBUG:boto:Method: GET 2023-09-22T21:59:13.080 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj68 2023-09-22T21:59:13.080 DEBUG:boto:Data: 2023-09-22T21:59:13.080 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.080 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.080 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.080 DEBUG:boto:Params: {} 2023-09-22T21:59:13.081 DEBUG:boto:Token: None 2023-09-22T21:59:13.081 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/ 2023-09-22T21:59:13.081 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw= 2023-09-22T21:59:13.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw=', 'Content-Length': '0'} 2023-09-22T21:59:13.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1982f4db6361ed9-00650e0e31-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.130 DEBUG:boto:b'ffzdqu-61obj681000falseobj682023-09-22T21:56:09.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:13.131 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:13.131 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:13.131 DEBUG:boto:path=/ffzdqu-61/?prefix=obj68 2023-09-22T21:59:13.131 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj68 2023-09-22T21:59:13.131 DEBUG:boto:Method: GET 2023-09-22T21:59:13.131 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj68 2023-09-22T21:59:13.131 DEBUG:boto:Data: 2023-09-22T21:59:13.131 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.131 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.131 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.132 DEBUG:boto:Params: {} 2023-09-22T21:59:13.132 DEBUG:boto:Token: None 2023-09-22T21:59:13.132 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/ 2023-09-22T21:59:13.132 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw= 2023-09-22T21:59:13.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw=', 'Content-Length': '0'} 2023-09-22T21:59:13.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a56aa3bb629ebbea-00650e0e31-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.142 DEBUG:boto:b'ffzdqu-61obj681000falseobj682023-09-22T21:56:09.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:13.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-22T21:59:13.143 DEBUG:boto:path=/ffzdqu-61/obj69 2023-09-22T21:59:13.143 DEBUG:boto:auth_path=/ffzdqu-61/obj69 2023-09-22T21:59:13.143 DEBUG:boto:path=/ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.143 DEBUG:boto:auth_path=/ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.143 DEBUG:boto:Method: GET 2023-09-22T21:59:13.143 DEBUG:boto:Path: /ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.143 DEBUG:boto:Data: 2023-09-22T21:59:13.144 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.144 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.144 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.144 DEBUG:boto:Params: {} 2023-09-22T21:59:13.144 DEBUG:boto:Token: None 2023-09-22T21:59:13.144 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.144 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jhd7bkwAo1C/tUweJLyIpduGoIM= 2023-09-22T21:59:13.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jhd7bkwAo1C/tUweJLyIpduGoIM=', 'Content-Length': '0'} 2023-09-22T21:59:13.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000202b7eb10e67d675-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.147 DEBUG:boto:path=/ffzdqu-61/obj69 2023-09-22T21:59:13.147 DEBUG:boto:auth_path=/ffzdqu-61/obj69 2023-09-22T21:59:13.148 DEBUG:boto:path=/ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.148 DEBUG:boto:auth_path=/ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.148 DEBUG:boto:Method: GET 2023-09-22T21:59:13.148 DEBUG:boto:Path: /ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.148 DEBUG:boto:Data: 2023-09-22T21:59:13.148 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.148 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.148 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.148 DEBUG:boto:Params: {} 2023-09-22T21:59:13.148 DEBUG:boto:Token: None 2023-09-22T21:59:13.149 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.149 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jhd7bkwAo1C/tUweJLyIpduGoIM= 2023-09-22T21:59:13.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jhd7bkwAo1C/tUweJLyIpduGoIM=', 'Content-Length': '0'} 2023-09-22T21:59:13.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000132300e071950ec-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.194 DEBUG:boto:path=/ffzdqu-61/obj69 2023-09-22T21:59:13.194 DEBUG:boto:auth_path=/ffzdqu-61/obj69 2023-09-22T21:59:13.195 DEBUG:boto:path=/ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.195 DEBUG:boto:auth_path=/ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.195 DEBUG:boto:Method: HEAD 2023-09-22T21:59:13.195 DEBUG:boto:Path: /ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.195 DEBUG:boto:Data: 2023-09-22T21:59:13.195 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.195 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.195 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.196 DEBUG:boto:Params: {} 2023-09-22T21:59:13.196 DEBUG:boto:Token: None 2023-09-22T21:59:13.196 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.196 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yLGCFtdN6zY9TuH3ALTpc3Edv9s= 2023-09-22T21:59:13.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yLGCFtdN6zY9TuH3ALTpc3Edv9s=', 'Content-Length': '0'} 2023-09-22T21:59:13.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000931bb137b1b302fa-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.199 DEBUG:boto:path=/ffzdqu-61/obj69 2023-09-22T21:59:13.199 DEBUG:boto:auth_path=/ffzdqu-61/obj69 2023-09-22T21:59:13.199 DEBUG:boto:path=/ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.200 DEBUG:boto:auth_path=/ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.200 DEBUG:boto:Method: HEAD 2023-09-22T21:59:13.200 DEBUG:boto:Path: /ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.200 DEBUG:boto:Data: 2023-09-22T21:59:13.200 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.200 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.200 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.200 DEBUG:boto:Params: {} 2023-09-22T21:59:13.200 DEBUG:boto:Token: None 2023-09-22T21:59:13.201 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.201 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yLGCFtdN6zY9TuH3ALTpc3Edv9s= 2023-09-22T21:59:13.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yLGCFtdN6zY9TuH3ALTpc3Edv9s=', 'Content-Length': '0'} 2023-09-22T21:59:13.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073259a9086f85d9f-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-22T21:59:13.204 DEBUG:boto:path=/ffzdqu-61/obj69 2023-09-22T21:59:13.204 DEBUG:boto:auth_path=/ffzdqu-61/obj69 2023-09-22T21:59:13.204 DEBUG:boto:path=/ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.204 DEBUG:boto:auth_path=/ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.204 DEBUG:boto:Method: GET 2023-09-22T21:59:13.205 DEBUG:boto:Path: /ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.205 DEBUG:boto:Data: 2023-09-22T21:59:13.205 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.205 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.205 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.205 DEBUG:boto:Params: {} 2023-09-22T21:59:13.205 DEBUG:boto:Token: None 2023-09-22T21:59:13.205 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.206 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jhd7bkwAo1C/tUweJLyIpduGoIM= 2023-09-22T21:59:13.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jhd7bkwAo1C/tUweJLyIpduGoIM=', 'Content-Length': '0'} 2023-09-22T21:59:13.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034c68a5dda32d29a-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.209 DEBUG:boto:path=/ffzdqu-61/obj69 2023-09-22T21:59:13.209 DEBUG:boto:auth_path=/ffzdqu-61/obj69 2023-09-22T21:59:13.209 DEBUG:boto:path=/ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.209 DEBUG:boto:auth_path=/ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.209 DEBUG:boto:Method: GET 2023-09-22T21:59:13.209 DEBUG:boto:Path: /ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.209 DEBUG:boto:Data: 2023-09-22T21:59:13.210 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.210 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.210 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.210 DEBUG:boto:Params: {} 2023-09-22T21:59:13.210 DEBUG:boto:Token: None 2023-09-22T21:59:13.210 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj69?versionId=null 2023-09-22T21:59:13.210 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jhd7bkwAo1C/tUweJLyIpduGoIM= 2023-09-22T21:59:13.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jhd7bkwAo1C/tUweJLyIpduGoIM=', 'Content-Length': '0'} 2023-09-22T21:59:13.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000837dafcb534c0418-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.213 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:13.213 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:13.214 DEBUG:boto:path=/ffzdqu-61/?prefix=obj69 2023-09-22T21:59:13.214 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj69 2023-09-22T21:59:13.214 DEBUG:boto:Method: GET 2023-09-22T21:59:13.214 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj69 2023-09-22T21:59:13.214 DEBUG:boto:Data: 2023-09-22T21:59:13.214 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.214 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.214 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.215 DEBUG:boto:Params: {} 2023-09-22T21:59:13.215 DEBUG:boto:Token: None 2023-09-22T21:59:13.215 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/ 2023-09-22T21:59:13.215 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw= 2023-09-22T21:59:13.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw=', 'Content-Length': '0'} 2023-09-22T21:59:13.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2d0af667a17f254-00650e0e31-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.223 DEBUG:boto:b'ffzdqu-61obj691000falseobj692023-09-22T21:56:09.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:13.224 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:13.224 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:13.224 DEBUG:boto:path=/ffzdqu-61/?prefix=obj69 2023-09-22T21:59:13.224 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj69 2023-09-22T21:59:13.224 DEBUG:boto:Method: GET 2023-09-22T21:59:13.224 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj69 2023-09-22T21:59:13.224 DEBUG:boto:Data: 2023-09-22T21:59:13.224 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.224 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.225 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.225 DEBUG:boto:Params: {} 2023-09-22T21:59:13.225 DEBUG:boto:Token: None 2023-09-22T21:59:13.225 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/ 2023-09-22T21:59:13.225 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw= 2023-09-22T21:59:13.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw=', 'Content-Length': '0'} 2023-09-22T21:59:13.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004122853f7e2ea1b-00650e0e31-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.236 DEBUG:boto:b'ffzdqu-61obj691000falseobj692023-09-22T21:56:09.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:13.236 DEBUG:boto:path=/ffzdqu-61/obj69 2023-09-22T21:59:13.236 DEBUG:boto:auth_path=/ffzdqu-61/obj69 2023-09-22T21:59:13.236 DEBUG:boto:Method: HEAD 2023-09-22T21:59:13.236 DEBUG:boto:Path: /ffzdqu-61/obj69 2023-09-22T21:59:13.237 DEBUG:boto:Data: 2023-09-22T21:59:13.237 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.237 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.237 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.237 DEBUG:boto:Params: {} 2023-09-22T21:59:13.237 DEBUG:boto:Token: None 2023-09-22T21:59:13.237 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj69 2023-09-22T21:59:13.237 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ao0Y6sK7xn9aXAh97q6wimc+QFc= 2023-09-22T21:59:13.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ao0Y6sK7xn9aXAh97q6wimc+QFc=', 'Content-Length': '0'} 2023-09-22T21:59:13.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081a611d4f5233573-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.240 DEBUG:boto:path=/ffzdqu-61/obj69 2023-09-22T21:59:13.240 DEBUG:boto:auth_path=/ffzdqu-61/obj69 2023-09-22T21:59:13.240 DEBUG:boto:Method: HEAD 2023-09-22T21:59:13.241 DEBUG:boto:Path: /ffzdqu-61/obj69 2023-09-22T21:59:13.241 DEBUG:boto:Data: 2023-09-22T21:59:13.241 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.241 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.241 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.241 DEBUG:boto:Params: {} 2023-09-22T21:59:13.241 DEBUG:boto:Token: None 2023-09-22T21:59:13.241 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj69 2023-09-22T21:59:13.241 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ao0Y6sK7xn9aXAh97q6wimc+QFc= 2023-09-22T21:59:13.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ao0Y6sK7xn9aXAh97q6wimc+QFc=', 'Content-Length': '0'} 2023-09-22T21:59:13.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cfae3d7e31f1c77-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-22T21:59:13.245 DEBUG:boto:path=/ffzdqu-61/obj69 2023-09-22T21:59:13.245 DEBUG:boto:auth_path=/ffzdqu-61/obj69 2023-09-22T21:59:13.245 DEBUG:boto:Method: GET 2023-09-22T21:59:13.245 DEBUG:boto:Path: /ffzdqu-61/obj69 2023-09-22T21:59:13.245 DEBUG:boto:Data: 2023-09-22T21:59:13.245 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.245 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.245 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.245 DEBUG:boto:Params: {} 2023-09-22T21:59:13.246 DEBUG:boto:Token: None 2023-09-22T21:59:13.246 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj69 2023-09-22T21:59:13.246 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pyqlHFJR48ebO/7MLoysVqfZ9vI= 2023-09-22T21:59:13.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pyqlHFJR48ebO/7MLoysVqfZ9vI=', 'Content-Length': '0'} 2023-09-22T21:59:13.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095ba039ff7a2515f-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.290 DEBUG:boto:path=/ffzdqu-61/obj69 2023-09-22T21:59:13.290 DEBUG:boto:auth_path=/ffzdqu-61/obj69 2023-09-22T21:59:13.290 DEBUG:boto:Method: GET 2023-09-22T21:59:13.291 DEBUG:boto:Path: /ffzdqu-61/obj69 2023-09-22T21:59:13.291 DEBUG:boto:Data: 2023-09-22T21:59:13.291 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.291 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.291 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.291 DEBUG:boto:Params: {} 2023-09-22T21:59:13.291 DEBUG:boto:Token: None 2023-09-22T21:59:13.291 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj69 2023-09-22T21:59:13.291 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pyqlHFJR48ebO/7MLoysVqfZ9vI= 2023-09-22T21:59:13.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pyqlHFJR48ebO/7MLoysVqfZ9vI=', 'Content-Length': '0'} 2023-09-22T21:59:13.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049f0f3cf2c6ab946-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.295 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:13.295 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:13.295 DEBUG:boto:path=/ffzdqu-61/?prefix=obj69 2023-09-22T21:59:13.295 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj69 2023-09-22T21:59:13.295 DEBUG:boto:Method: GET 2023-09-22T21:59:13.295 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj69 2023-09-22T21:59:13.295 DEBUG:boto:Data: 2023-09-22T21:59:13.295 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.295 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.296 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.296 DEBUG:boto:Params: {} 2023-09-22T21:59:13.296 DEBUG:boto:Token: None 2023-09-22T21:59:13.296 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/ 2023-09-22T21:59:13.296 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw= 2023-09-22T21:59:13.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw=', 'Content-Length': '0'} 2023-09-22T21:59:13.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e89d3899b16ca99b-00650e0e31-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.301 DEBUG:boto:b'ffzdqu-61obj691000falseobj692023-09-22T21:56:09.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:13.301 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:13.301 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:13.302 DEBUG:boto:path=/ffzdqu-61/?prefix=obj69 2023-09-22T21:59:13.302 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj69 2023-09-22T21:59:13.302 DEBUG:boto:Method: GET 2023-09-22T21:59:13.302 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj69 2023-09-22T21:59:13.302 DEBUG:boto:Data: 2023-09-22T21:59:13.302 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.302 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.302 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.302 DEBUG:boto:Params: {} 2023-09-22T21:59:13.303 DEBUG:boto:Token: None 2023-09-22T21:59:13.303 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/ 2023-09-22T21:59:13.303 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw= 2023-09-22T21:59:13.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw=', 'Content-Length': '0'} 2023-09-22T21:59:13.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000307bac6bccfbd03f-00650e0e31-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.315 DEBUG:boto:b'ffzdqu-61obj691000falseobj692023-09-22T21:56:09.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:13.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-09-22T21:59:13.315 DEBUG:boto:path=/ffzdqu-61/obj74 2023-09-22T21:59:13.315 DEBUG:boto:auth_path=/ffzdqu-61/obj74 2023-09-22T21:59:13.315 DEBUG:boto:path=/ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.316 DEBUG:boto:auth_path=/ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.316 DEBUG:boto:Method: GET 2023-09-22T21:59:13.316 DEBUG:boto:Path: /ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.316 DEBUG:boto:Data: 2023-09-22T21:59:13.316 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.316 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.316 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.316 DEBUG:boto:Params: {} 2023-09-22T21:59:13.316 DEBUG:boto:Token: None 2023-09-22T21:59:13.316 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.317 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CWwGcHQFxDAB0bAHAFnQWwAHTPg= 2023-09-22T21:59:13.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CWwGcHQFxDAB0bAHAFnQWwAHTPg=', 'Content-Length': '0'} 2023-09-22T21:59:13.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023277c1a5debd4de-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.320 DEBUG:boto:path=/ffzdqu-61/obj74 2023-09-22T21:59:13.320 DEBUG:boto:auth_path=/ffzdqu-61/obj74 2023-09-22T21:59:13.320 DEBUG:boto:path=/ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.320 DEBUG:boto:auth_path=/ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.320 DEBUG:boto:Method: GET 2023-09-22T21:59:13.320 DEBUG:boto:Path: /ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.320 DEBUG:boto:Data: 2023-09-22T21:59:13.320 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.320 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.321 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.321 DEBUG:boto:Params: {} 2023-09-22T21:59:13.321 DEBUG:boto:Token: None 2023-09-22T21:59:13.321 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.321 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CWwGcHQFxDAB0bAHAFnQWwAHTPg= 2023-09-22T21:59:13.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CWwGcHQFxDAB0bAHAFnQWwAHTPg=', 'Content-Length': '0'} 2023-09-22T21:59:13.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012cd31946fafa75c-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.324 DEBUG:boto:path=/ffzdqu-61/obj74 2023-09-22T21:59:13.324 DEBUG:boto:auth_path=/ffzdqu-61/obj74 2023-09-22T21:59:13.324 DEBUG:boto:path=/ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.324 DEBUG:boto:auth_path=/ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.325 DEBUG:boto:Method: HEAD 2023-09-22T21:59:13.325 DEBUG:boto:Path: /ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.325 DEBUG:boto:Data: 2023-09-22T21:59:13.325 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.325 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.325 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.325 DEBUG:boto:Params: {} 2023-09-22T21:59:13.325 DEBUG:boto:Token: None 2023-09-22T21:59:13.325 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.326 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YRlWeVfguW2WzalqK0VISd3n4RA= 2023-09-22T21:59:13.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YRlWeVfguW2WzalqK0VISd3n4RA=', 'Content-Length': '0'} 2023-09-22T21:59:13.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2cb71af49fe6fd1-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.328 DEBUG:boto:path=/ffzdqu-61/obj74 2023-09-22T21:59:13.329 DEBUG:boto:auth_path=/ffzdqu-61/obj74 2023-09-22T21:59:13.329 DEBUG:boto:path=/ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.329 DEBUG:boto:auth_path=/ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.329 DEBUG:boto:Method: HEAD 2023-09-22T21:59:13.329 DEBUG:boto:Path: /ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.329 DEBUG:boto:Data: 2023-09-22T21:59:13.329 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.329 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.329 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.329 DEBUG:boto:Params: {} 2023-09-22T21:59:13.330 DEBUG:boto:Token: None 2023-09-22T21:59:13.330 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.330 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YRlWeVfguW2WzalqK0VISd3n4RA= 2023-09-22T21:59:13.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YRlWeVfguW2WzalqK0VISd3n4RA=', 'Content-Length': '0'} 2023-09-22T21:59:13.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006faba5149d18f036-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-09-22T21:59:13.333 DEBUG:boto:path=/ffzdqu-61/obj74 2023-09-22T21:59:13.333 DEBUG:boto:auth_path=/ffzdqu-61/obj74 2023-09-22T21:59:13.333 DEBUG:boto:path=/ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.333 DEBUG:boto:auth_path=/ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.333 DEBUG:boto:Method: GET 2023-09-22T21:59:13.333 DEBUG:boto:Path: /ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.334 DEBUG:boto:Data: 2023-09-22T21:59:13.334 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.334 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.334 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.334 DEBUG:boto:Params: {} 2023-09-22T21:59:13.334 DEBUG:boto:Token: None 2023-09-22T21:59:13.334 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.334 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CWwGcHQFxDAB0bAHAFnQWwAHTPg= 2023-09-22T21:59:13.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CWwGcHQFxDAB0bAHAFnQWwAHTPg=', 'Content-Length': '0'} 2023-09-22T21:59:13.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000197f14e3e66d11b5-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.378 DEBUG:boto:path=/ffzdqu-61/obj74 2023-09-22T21:59:13.378 DEBUG:boto:auth_path=/ffzdqu-61/obj74 2023-09-22T21:59:13.379 DEBUG:boto:path=/ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.379 DEBUG:boto:auth_path=/ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.379 DEBUG:boto:Method: GET 2023-09-22T21:59:13.379 DEBUG:boto:Path: /ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.379 DEBUG:boto:Data: 2023-09-22T21:59:13.379 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.379 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.379 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.379 DEBUG:boto:Params: {} 2023-09-22T21:59:13.379 DEBUG:boto:Token: None 2023-09-22T21:59:13.380 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj74?versionId=null 2023-09-22T21:59:13.380 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CWwGcHQFxDAB0bAHAFnQWwAHTPg= 2023-09-22T21:59:13.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CWwGcHQFxDAB0bAHAFnQWwAHTPg=', 'Content-Length': '0'} 2023-09-22T21:59:13.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000259bc8ca16c81243-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.383 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:13.383 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:13.383 DEBUG:boto:path=/ffzdqu-61/?prefix=obj74 2023-09-22T21:59:13.383 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj74 2023-09-22T21:59:13.383 DEBUG:boto:Method: GET 2023-09-22T21:59:13.384 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj74 2023-09-22T21:59:13.384 DEBUG:boto:Data: 2023-09-22T21:59:13.384 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.384 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.384 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.384 DEBUG:boto:Params: {} 2023-09-22T21:59:13.384 DEBUG:boto:Token: None 2023-09-22T21:59:13.384 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/ 2023-09-22T21:59:13.385 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw= 2023-09-22T21:59:13.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw=', 'Content-Length': '0'} 2023-09-22T21:59:13.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006962b0f07713eb5c-00650e0e31-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.390 DEBUG:boto:b'ffzdqu-61obj741000falseobj742023-09-22T21:56:10.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:13.390 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:13.390 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:13.390 DEBUG:boto:path=/ffzdqu-61/?prefix=obj74 2023-09-22T21:59:13.390 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj74 2023-09-22T21:59:13.391 DEBUG:boto:Method: GET 2023-09-22T21:59:13.391 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj74 2023-09-22T21:59:13.391 DEBUG:boto:Data: 2023-09-22T21:59:13.391 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.391 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.391 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.391 DEBUG:boto:Params: {} 2023-09-22T21:59:13.391 DEBUG:boto:Token: None 2023-09-22T21:59:13.391 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/ 2023-09-22T21:59:13.391 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw= 2023-09-22T21:59:13.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw=', 'Content-Length': '0'} 2023-09-22T21:59:13.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000638ca5603f499c1a-00650e0e31-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.401 DEBUG:boto:b'ffzdqu-61obj741000falseobj742023-09-22T21:56:10.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:13.401 DEBUG:boto:path=/ffzdqu-61/obj74 2023-09-22T21:59:13.401 DEBUG:boto:auth_path=/ffzdqu-61/obj74 2023-09-22T21:59:13.401 DEBUG:boto:Method: HEAD 2023-09-22T21:59:13.402 DEBUG:boto:Path: /ffzdqu-61/obj74 2023-09-22T21:59:13.402 DEBUG:boto:Data: 2023-09-22T21:59:13.402 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.402 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.402 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.402 DEBUG:boto:Params: {} 2023-09-22T21:59:13.402 DEBUG:boto:Token: None 2023-09-22T21:59:13.402 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj74 2023-09-22T21:59:13.402 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wx3oCit/2Ri0yvamnalWPMJTmfY= 2023-09-22T21:59:13.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wx3oCit/2Ri0yvamnalWPMJTmfY=', 'Content-Length': '0'} 2023-09-22T21:59:13.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cf4f08515e3f192-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.406 DEBUG:boto:path=/ffzdqu-61/obj74 2023-09-22T21:59:13.406 DEBUG:boto:auth_path=/ffzdqu-61/obj74 2023-09-22T21:59:13.406 DEBUG:boto:Method: HEAD 2023-09-22T21:59:13.406 DEBUG:boto:Path: /ffzdqu-61/obj74 2023-09-22T21:59:13.406 DEBUG:boto:Data: 2023-09-22T21:59:13.406 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.406 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.407 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.407 DEBUG:boto:Params: {} 2023-09-22T21:59:13.407 DEBUG:boto:Token: None 2023-09-22T21:59:13.407 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj74 2023-09-22T21:59:13.407 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wx3oCit/2Ri0yvamnalWPMJTmfY= 2023-09-22T21:59:13.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wx3oCit/2Ri0yvamnalWPMJTmfY=', 'Content-Length': '0'} 2023-09-22T21:59:13.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2217ea7a18116d2-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-09-22T21:59:13.410 DEBUG:boto:path=/ffzdqu-61/obj74 2023-09-22T21:59:13.410 DEBUG:boto:auth_path=/ffzdqu-61/obj74 2023-09-22T21:59:13.411 DEBUG:boto:Method: GET 2023-09-22T21:59:13.411 DEBUG:boto:Path: /ffzdqu-61/obj74 2023-09-22T21:59:13.411 DEBUG:boto:Data: 2023-09-22T21:59:13.411 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.411 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.411 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.411 DEBUG:boto:Params: {} 2023-09-22T21:59:13.411 DEBUG:boto:Token: None 2023-09-22T21:59:13.411 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj74 2023-09-22T21:59:13.412 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SH6Uhn/4vv3RyaatM4jlI2JS+nU= 2023-09-22T21:59:13.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SH6Uhn/4vv3RyaatM4jlI2JS+nU=', 'Content-Length': '0'} 2023-09-22T21:59:13.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026afb74b199707a8-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.415 DEBUG:boto:path=/ffzdqu-61/obj74 2023-09-22T21:59:13.415 DEBUG:boto:auth_path=/ffzdqu-61/obj74 2023-09-22T21:59:13.415 DEBUG:boto:Method: GET 2023-09-22T21:59:13.415 DEBUG:boto:Path: /ffzdqu-61/obj74 2023-09-22T21:59:13.415 DEBUG:boto:Data: 2023-09-22T21:59:13.415 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.416 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.416 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.416 DEBUG:boto:Params: {} 2023-09-22T21:59:13.416 DEBUG:boto:Token: None 2023-09-22T21:59:13.416 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj74 2023-09-22T21:59:13.416 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SH6Uhn/4vv3RyaatM4jlI2JS+nU= 2023-09-22T21:59:13.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SH6Uhn/4vv3RyaatM4jlI2JS+nU=', 'Content-Length': '0'} 2023-09-22T21:59:13.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3b181c746bb5fad-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.419 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:13.419 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:13.419 DEBUG:boto:path=/ffzdqu-61/?prefix=obj74 2023-09-22T21:59:13.420 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj74 2023-09-22T21:59:13.420 DEBUG:boto:Method: GET 2023-09-22T21:59:13.420 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj74 2023-09-22T21:59:13.420 DEBUG:boto:Data: 2023-09-22T21:59:13.420 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.420 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.420 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.420 DEBUG:boto:Params: {} 2023-09-22T21:59:13.420 DEBUG:boto:Token: None 2023-09-22T21:59:13.420 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/ 2023-09-22T21:59:13.421 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw= 2023-09-22T21:59:13.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw=', 'Content-Length': '0'} 2023-09-22T21:59:13.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad18649384874be0-00650e0e31-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.466 DEBUG:boto:b'ffzdqu-61obj741000falseobj742023-09-22T21:56:10.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:13.466 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:13.467 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:13.467 DEBUG:boto:path=/ffzdqu-61/?prefix=obj74 2023-09-22T21:59:13.467 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj74 2023-09-22T21:59:13.467 DEBUG:boto:Method: GET 2023-09-22T21:59:13.467 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj74 2023-09-22T21:59:13.467 DEBUG:boto:Data: 2023-09-22T21:59:13.467 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.467 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.467 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.467 DEBUG:boto:Params: {} 2023-09-22T21:59:13.468 DEBUG:boto:Token: None 2023-09-22T21:59:13.468 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/ 2023-09-22T21:59:13.468 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw= 2023-09-22T21:59:13.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw=', 'Content-Length': '0'} 2023-09-22T21:59:13.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000283a909598eaf846-00650e0e31-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.477 DEBUG:boto:b'ffzdqu-61obj741000falseobj742023-09-22T21:56:10.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:13.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T21:59:13.477 DEBUG:boto:path=/ffzdqu-61/obj76 2023-09-22T21:59:13.478 DEBUG:boto:auth_path=/ffzdqu-61/obj76 2023-09-22T21:59:13.478 DEBUG:boto:path=/ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.478 DEBUG:boto:auth_path=/ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.478 DEBUG:boto:Method: GET 2023-09-22T21:59:13.478 DEBUG:boto:Path: /ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.478 DEBUG:boto:Data: 2023-09-22T21:59:13.478 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.478 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.478 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.479 DEBUG:boto:Params: {} 2023-09-22T21:59:13.479 DEBUG:boto:Token: None 2023-09-22T21:59:13.479 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.479 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:poTpndXEuoS9qBoB3r77gn1KTFY= 2023-09-22T21:59:13.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:poTpndXEuoS9qBoB3r77gn1KTFY=', 'Content-Length': '0'} 2023-09-22T21:59:13.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a5e4dbb610bd495-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.482 DEBUG:boto:path=/ffzdqu-61/obj76 2023-09-22T21:59:13.482 DEBUG:boto:auth_path=/ffzdqu-61/obj76 2023-09-22T21:59:13.482 DEBUG:boto:path=/ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.482 DEBUG:boto:auth_path=/ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.482 DEBUG:boto:Method: GET 2023-09-22T21:59:13.483 DEBUG:boto:Path: /ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.483 DEBUG:boto:Data: 2023-09-22T21:59:13.483 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.483 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.483 DEBUG:boto:Params: {} 2023-09-22T21:59:13.483 DEBUG:boto:Token: None 2023-09-22T21:59:13.483 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.483 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:poTpndXEuoS9qBoB3r77gn1KTFY= 2023-09-22T21:59:13.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:poTpndXEuoS9qBoB3r77gn1KTFY=', 'Content-Length': '0'} 2023-09-22T21:59:13.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bc7c5c661f46f15-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.530 DEBUG:boto:path=/ffzdqu-61/obj76 2023-09-22T21:59:13.530 DEBUG:boto:auth_path=/ffzdqu-61/obj76 2023-09-22T21:59:13.530 DEBUG:boto:path=/ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.531 DEBUG:boto:auth_path=/ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.531 DEBUG:boto:Method: HEAD 2023-09-22T21:59:13.531 DEBUG:boto:Path: /ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.531 DEBUG:boto:Data: 2023-09-22T21:59:13.531 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.531 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.531 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.531 DEBUG:boto:Params: {} 2023-09-22T21:59:13.531 DEBUG:boto:Token: None 2023-09-22T21:59:13.532 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.532 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MLq19ckzlnTrIvRBdGTz7Kghgqo= 2023-09-22T21:59:13.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MLq19ckzlnTrIvRBdGTz7Kghgqo=', 'Content-Length': '0'} 2023-09-22T21:59:13.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6b19e77737261c0-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.535 DEBUG:boto:path=/ffzdqu-61/obj76 2023-09-22T21:59:13.535 DEBUG:boto:auth_path=/ffzdqu-61/obj76 2023-09-22T21:59:13.535 DEBUG:boto:path=/ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.535 DEBUG:boto:auth_path=/ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.535 DEBUG:boto:Method: HEAD 2023-09-22T21:59:13.535 DEBUG:boto:Path: /ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.535 DEBUG:boto:Data: 2023-09-22T21:59:13.536 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.536 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.536 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.536 DEBUG:boto:Params: {} 2023-09-22T21:59:13.536 DEBUG:boto:Token: None 2023-09-22T21:59:13.536 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.536 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MLq19ckzlnTrIvRBdGTz7Kghgqo= 2023-09-22T21:59:13.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MLq19ckzlnTrIvRBdGTz7Kghgqo=', 'Content-Length': '0'} 2023-09-22T21:59:13.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e2890ebd9e2f864-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T21:59:13.539 DEBUG:boto:path=/ffzdqu-61/obj76 2023-09-22T21:59:13.539 DEBUG:boto:auth_path=/ffzdqu-61/obj76 2023-09-22T21:59:13.539 DEBUG:boto:path=/ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.539 DEBUG:boto:auth_path=/ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.540 DEBUG:boto:Method: GET 2023-09-22T21:59:13.540 DEBUG:boto:Path: /ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.540 DEBUG:boto:Data: 2023-09-22T21:59:13.540 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.540 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.540 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.540 DEBUG:boto:Params: {} 2023-09-22T21:59:13.540 DEBUG:boto:Token: None 2023-09-22T21:59:13.540 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.540 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:poTpndXEuoS9qBoB3r77gn1KTFY= 2023-09-22T21:59:13.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:poTpndXEuoS9qBoB3r77gn1KTFY=', 'Content-Length': '0'} 2023-09-22T21:59:13.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000951458d5815952ac-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.544 DEBUG:boto:path=/ffzdqu-61/obj76 2023-09-22T21:59:13.544 DEBUG:boto:auth_path=/ffzdqu-61/obj76 2023-09-22T21:59:13.544 DEBUG:boto:path=/ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.544 DEBUG:boto:auth_path=/ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.544 DEBUG:boto:Method: GET 2023-09-22T21:59:13.544 DEBUG:boto:Path: /ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.544 DEBUG:boto:Data: 2023-09-22T21:59:13.544 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.544 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.544 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.545 DEBUG:boto:Params: {} 2023-09-22T21:59:13.545 DEBUG:boto:Token: None 2023-09-22T21:59:13.545 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj76?versionId=null 2023-09-22T21:59:13.545 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:poTpndXEuoS9qBoB3r77gn1KTFY= 2023-09-22T21:59:13.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:poTpndXEuoS9qBoB3r77gn1KTFY=', 'Content-Length': '0'} 2023-09-22T21:59:13.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019a0594676093dd0-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.548 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:13.548 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:13.548 DEBUG:boto:path=/ffzdqu-61/?prefix=obj76 2023-09-22T21:59:13.548 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj76 2023-09-22T21:59:13.548 DEBUG:boto:Method: GET 2023-09-22T21:59:13.548 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj76 2023-09-22T21:59:13.549 DEBUG:boto:Data: 2023-09-22T21:59:13.549 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.549 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.549 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.549 DEBUG:boto:Params: {} 2023-09-22T21:59:13.549 DEBUG:boto:Token: None 2023-09-22T21:59:13.549 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/ 2023-09-22T21:59:13.549 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw= 2023-09-22T21:59:13.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw=', 'Content-Length': '0'} 2023-09-22T21:59:13.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a2517dcc9efbbd2-00650e0e31-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.594 DEBUG:boto:b'ffzdqu-61obj761000falseobj762023-09-22T21:56:09.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:13.595 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:13.595 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:13.595 DEBUG:boto:path=/ffzdqu-61/?prefix=obj76 2023-09-22T21:59:13.595 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj76 2023-09-22T21:59:13.595 DEBUG:boto:Method: GET 2023-09-22T21:59:13.595 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj76 2023-09-22T21:59:13.595 DEBUG:boto:Data: 2023-09-22T21:59:13.595 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.595 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.596 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.596 DEBUG:boto:Params: {} 2023-09-22T21:59:13.596 DEBUG:boto:Token: None 2023-09-22T21:59:13.596 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/ 2023-09-22T21:59:13.596 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw= 2023-09-22T21:59:13.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw=', 'Content-Length': '0'} 2023-09-22T21:59:13.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050c16f9fd40a7dae-00650e0e31-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.605 DEBUG:boto:b'ffzdqu-61obj761000falseobj762023-09-22T21:56:09.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:13.605 DEBUG:boto:path=/ffzdqu-61/obj76 2023-09-22T21:59:13.606 DEBUG:boto:auth_path=/ffzdqu-61/obj76 2023-09-22T21:59:13.606 DEBUG:boto:Method: HEAD 2023-09-22T21:59:13.606 DEBUG:boto:Path: /ffzdqu-61/obj76 2023-09-22T21:59:13.606 DEBUG:boto:Data: 2023-09-22T21:59:13.606 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.606 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.606 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.606 DEBUG:boto:Params: {} 2023-09-22T21:59:13.607 DEBUG:boto:Token: None 2023-09-22T21:59:13.607 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj76 2023-09-22T21:59:13.607 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:k0jvDFxvjMI463cGjwEYL0OK5Zw= 2023-09-22T21:59:13.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:k0jvDFxvjMI463cGjwEYL0OK5Zw=', 'Content-Length': '0'} 2023-09-22T21:59:13.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000456093fef9a79278-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.610 DEBUG:boto:path=/ffzdqu-61/obj76 2023-09-22T21:59:13.610 DEBUG:boto:auth_path=/ffzdqu-61/obj76 2023-09-22T21:59:13.610 DEBUG:boto:Method: HEAD 2023-09-22T21:59:13.610 DEBUG:boto:Path: /ffzdqu-61/obj76 2023-09-22T21:59:13.611 DEBUG:boto:Data: 2023-09-22T21:59:13.611 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.611 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.611 DEBUG:boto:Params: {} 2023-09-22T21:59:13.611 DEBUG:boto:Token: None 2023-09-22T21:59:13.611 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj76 2023-09-22T21:59:13.611 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:k0jvDFxvjMI463cGjwEYL0OK5Zw= 2023-09-22T21:59:13.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:k0jvDFxvjMI463cGjwEYL0OK5Zw=', 'Content-Length': '0'} 2023-09-22T21:59:13.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003769807b813f1255-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T21:59:13.615 DEBUG:boto:path=/ffzdqu-61/obj76 2023-09-22T21:59:13.615 DEBUG:boto:auth_path=/ffzdqu-61/obj76 2023-09-22T21:59:13.615 DEBUG:boto:Method: GET 2023-09-22T21:59:13.615 DEBUG:boto:Path: /ffzdqu-61/obj76 2023-09-22T21:59:13.615 DEBUG:boto:Data: 2023-09-22T21:59:13.615 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.615 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.615 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.616 DEBUG:boto:Params: {} 2023-09-22T21:59:13.616 DEBUG:boto:Token: None 2023-09-22T21:59:13.616 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj76 2023-09-22T21:59:13.616 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0sJv/b95clqiq1W4ggj4dIZ2PqQ= 2023-09-22T21:59:13.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0sJv/b95clqiq1W4ggj4dIZ2PqQ=', 'Content-Length': '0'} 2023-09-22T21:59:13.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000402ac108ed246e5e-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.619 DEBUG:boto:path=/ffzdqu-61/obj76 2023-09-22T21:59:13.619 DEBUG:boto:auth_path=/ffzdqu-61/obj76 2023-09-22T21:59:13.619 DEBUG:boto:Method: GET 2023-09-22T21:59:13.619 DEBUG:boto:Path: /ffzdqu-61/obj76 2023-09-22T21:59:13.619 DEBUG:boto:Data: 2023-09-22T21:59:13.620 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.620 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.620 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.620 DEBUG:boto:Params: {} 2023-09-22T21:59:13.620 DEBUG:boto:Token: None 2023-09-22T21:59:13.620 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj76 2023-09-22T21:59:13.620 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0sJv/b95clqiq1W4ggj4dIZ2PqQ= 2023-09-22T21:59:13.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0sJv/b95clqiq1W4ggj4dIZ2PqQ=', 'Content-Length': '0'} 2023-09-22T21:59:13.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b32a2baa5e8ba3c0-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.666 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:13.666 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:13.667 DEBUG:boto:path=/ffzdqu-61/?prefix=obj76 2023-09-22T21:59:13.667 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj76 2023-09-22T21:59:13.667 DEBUG:boto:Method: GET 2023-09-22T21:59:13.667 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj76 2023-09-22T21:59:13.667 DEBUG:boto:Data: 2023-09-22T21:59:13.667 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.667 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.667 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.667 DEBUG:boto:Params: {} 2023-09-22T21:59:13.668 DEBUG:boto:Token: None 2023-09-22T21:59:13.668 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/ 2023-09-22T21:59:13.668 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw= 2023-09-22T21:59:13.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw=', 'Content-Length': '0'} 2023-09-22T21:59:13.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d781d879bd5919a-00650e0e31-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.673 DEBUG:boto:b'ffzdqu-61obj761000falseobj762023-09-22T21:56:09.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:13.674 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:13.674 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:13.674 DEBUG:boto:path=/ffzdqu-61/?prefix=obj76 2023-09-22T21:59:13.674 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj76 2023-09-22T21:59:13.674 DEBUG:boto:Method: GET 2023-09-22T21:59:13.674 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj76 2023-09-22T21:59:13.675 DEBUG:boto:Data: 2023-09-22T21:59:13.675 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.675 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.675 DEBUG:boto:Params: {} 2023-09-22T21:59:13.675 DEBUG:boto:Token: None 2023-09-22T21:59:13.675 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/ 2023-09-22T21:59:13.675 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw= 2023-09-22T21:59:13.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw=', 'Content-Length': '0'} 2023-09-22T21:59:13.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006069db7089e5ab28-00650e0e31-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.685 DEBUG:boto:b'ffzdqu-61obj761000falseobj762023-09-22T21:56:09.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:13.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:59:13.685 DEBUG:boto:path=/ffzdqu-61/obj85 2023-09-22T21:59:13.686 DEBUG:boto:auth_path=/ffzdqu-61/obj85 2023-09-22T21:59:13.686 DEBUG:boto:path=/ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.686 DEBUG:boto:auth_path=/ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.686 DEBUG:boto:Method: GET 2023-09-22T21:59:13.686 DEBUG:boto:Path: /ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.686 DEBUG:boto:Data: 2023-09-22T21:59:13.686 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.686 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.687 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.687 DEBUG:boto:Params: {} 2023-09-22T21:59:13.687 DEBUG:boto:Token: None 2023-09-22T21:59:13.687 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.687 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KKS9+2Wg1JTCxY+SwuKq+RaKBN4= 2023-09-22T21:59:13.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KKS9+2Wg1JTCxY+SwuKq+RaKBN4=', 'Content-Length': '0'} 2023-09-22T21:59:13.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfc11eda009fef04-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.690 DEBUG:boto:path=/ffzdqu-61/obj85 2023-09-22T21:59:13.690 DEBUG:boto:auth_path=/ffzdqu-61/obj85 2023-09-22T21:59:13.691 DEBUG:boto:path=/ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.691 DEBUG:boto:auth_path=/ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.691 DEBUG:boto:Method: GET 2023-09-22T21:59:13.691 DEBUG:boto:Path: /ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.691 DEBUG:boto:Data: 2023-09-22T21:59:13.691 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.691 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.691 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.691 DEBUG:boto:Params: {} 2023-09-22T21:59:13.692 DEBUG:boto:Token: None 2023-09-22T21:59:13.692 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.692 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KKS9+2Wg1JTCxY+SwuKq+RaKBN4= 2023-09-22T21:59:13.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KKS9+2Wg1JTCxY+SwuKq+RaKBN4=', 'Content-Length': '0'} 2023-09-22T21:59:13.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6967570e599e034-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.695 DEBUG:boto:path=/ffzdqu-61/obj85 2023-09-22T21:59:13.695 DEBUG:boto:auth_path=/ffzdqu-61/obj85 2023-09-22T21:59:13.695 DEBUG:boto:path=/ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.695 DEBUG:boto:auth_path=/ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.695 DEBUG:boto:Method: HEAD 2023-09-22T21:59:13.696 DEBUG:boto:Path: /ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.696 DEBUG:boto:Data: 2023-09-22T21:59:13.696 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.696 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.696 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.696 DEBUG:boto:Params: {} 2023-09-22T21:59:13.696 DEBUG:boto:Token: None 2023-09-22T21:59:13.696 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.696 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3Jsy5yOrxaVID4Eiq/tjBTu89yk= 2023-09-22T21:59:13.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3Jsy5yOrxaVID4Eiq/tjBTu89yk=', 'Content-Length': '0'} 2023-09-22T21:59:13.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050ebfd6ca6e72076-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.699 DEBUG:boto:path=/ffzdqu-61/obj85 2023-09-22T21:59:13.700 DEBUG:boto:auth_path=/ffzdqu-61/obj85 2023-09-22T21:59:13.700 DEBUG:boto:path=/ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.700 DEBUG:boto:auth_path=/ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.700 DEBUG:boto:Method: HEAD 2023-09-22T21:59:13.700 DEBUG:boto:Path: /ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.700 DEBUG:boto:Data: 2023-09-22T21:59:13.700 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.700 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.700 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.700 DEBUG:boto:Params: {} 2023-09-22T21:59:13.701 DEBUG:boto:Token: None 2023-09-22T21:59:13.701 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.701 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3Jsy5yOrxaVID4Eiq/tjBTu89yk= 2023-09-22T21:59:13.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3Jsy5yOrxaVID4Eiq/tjBTu89yk=', 'Content-Length': '0'} 2023-09-22T21:59:13.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057c977aff87d0984-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:59:13.704 DEBUG:boto:path=/ffzdqu-61/obj85 2023-09-22T21:59:13.704 DEBUG:boto:auth_path=/ffzdqu-61/obj85 2023-09-22T21:59:13.704 DEBUG:boto:path=/ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.704 DEBUG:boto:auth_path=/ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.704 DEBUG:boto:Method: GET 2023-09-22T21:59:13.704 DEBUG:boto:Path: /ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.705 DEBUG:boto:Data: 2023-09-22T21:59:13.705 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.705 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.705 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.705 DEBUG:boto:Params: {} 2023-09-22T21:59:13.705 DEBUG:boto:Token: None 2023-09-22T21:59:13.705 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.705 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KKS9+2Wg1JTCxY+SwuKq+RaKBN4= 2023-09-22T21:59:13.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KKS9+2Wg1JTCxY+SwuKq+RaKBN4=', 'Content-Length': '0'} 2023-09-22T21:59:13.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067f38471a92c102d-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.750 DEBUG:boto:path=/ffzdqu-61/obj85 2023-09-22T21:59:13.877 DEBUG:boto:auth_path=/ffzdqu-61/obj85 2023-09-22T21:59:13.877 DEBUG:boto:path=/ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.877 DEBUG:boto:auth_path=/ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.877 DEBUG:boto:Method: GET 2023-09-22T21:59:13.877 DEBUG:boto:Path: /ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.877 DEBUG:boto:Data: 2023-09-22T21:59:13.877 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.878 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.878 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.878 DEBUG:boto:Params: {} 2023-09-22T21:59:13.878 DEBUG:boto:Token: None 2023-09-22T21:59:13.878 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj85?versionId=null 2023-09-22T21:59:13.878 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KKS9+2Wg1JTCxY+SwuKq+RaKBN4= 2023-09-22T21:59:13.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KKS9+2Wg1JTCxY+SwuKq+RaKBN4=', 'Content-Length': '0'} 2023-09-22T21:59:13.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef7049c52c38e9ae-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.884 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:13.884 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:13.884 DEBUG:boto:path=/ffzdqu-61/?prefix=obj85 2023-09-22T21:59:13.884 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj85 2023-09-22T21:59:13.884 DEBUG:boto:Method: GET 2023-09-22T21:59:13.884 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj85 2023-09-22T21:59:13.884 DEBUG:boto:Data: 2023-09-22T21:59:13.884 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.885 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.885 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.885 DEBUG:boto:Params: {} 2023-09-22T21:59:13.885 DEBUG:boto:Token: None 2023-09-22T21:59:13.885 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/ 2023-09-22T21:59:13.885 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw= 2023-09-22T21:59:13.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw=', 'Content-Length': '0'} 2023-09-22T21:59:13.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078baefc38ebc1d17-00650e0e31-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.892 DEBUG:boto:b'ffzdqu-61obj851000falseobj852023-09-22T21:56:09.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:13.892 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:13.892 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:13.893 DEBUG:boto:path=/ffzdqu-61/?prefix=obj85 2023-09-22T21:59:13.893 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj85 2023-09-22T21:59:13.893 DEBUG:boto:Method: GET 2023-09-22T21:59:13.893 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj85 2023-09-22T21:59:13.893 DEBUG:boto:Data: 2023-09-22T21:59:13.893 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.893 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.893 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.893 DEBUG:boto:Params: {} 2023-09-22T21:59:13.894 DEBUG:boto:Token: None 2023-09-22T21:59:13.894 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/ 2023-09-22T21:59:13.894 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw= 2023-09-22T21:59:13.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw=', 'Content-Length': '0'} 2023-09-22T21:59:13.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b7962cd8ea822eb-00650e0e31-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.903 DEBUG:boto:b'ffzdqu-61obj851000falseobj852023-09-22T21:56:09.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:13.904 DEBUG:boto:path=/ffzdqu-61/obj85 2023-09-22T21:59:13.904 DEBUG:boto:auth_path=/ffzdqu-61/obj85 2023-09-22T21:59:13.904 DEBUG:boto:Method: HEAD 2023-09-22T21:59:13.904 DEBUG:boto:Path: /ffzdqu-61/obj85 2023-09-22T21:59:13.904 DEBUG:boto:Data: 2023-09-22T21:59:13.904 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.905 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.905 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.905 DEBUG:boto:Params: {} 2023-09-22T21:59:13.905 DEBUG:boto:Token: None 2023-09-22T21:59:13.905 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj85 2023-09-22T21:59:13.905 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aQ23vyxpTbs/gEHYKlbDoRq4x4E= 2023-09-22T21:59:13.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aQ23vyxpTbs/gEHYKlbDoRq4x4E=', 'Content-Length': '0'} 2023-09-22T21:59:13.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022f50679fd24f46b-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.908 DEBUG:boto:path=/ffzdqu-61/obj85 2023-09-22T21:59:13.908 DEBUG:boto:auth_path=/ffzdqu-61/obj85 2023-09-22T21:59:13.908 DEBUG:boto:Method: HEAD 2023-09-22T21:59:13.909 DEBUG:boto:Path: /ffzdqu-61/obj85 2023-09-22T21:59:13.909 DEBUG:boto:Data: 2023-09-22T21:59:13.909 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.909 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.909 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.909 DEBUG:boto:Params: {} 2023-09-22T21:59:13.909 DEBUG:boto:Token: None 2023-09-22T21:59:13.909 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj85 2023-09-22T21:59:13.909 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aQ23vyxpTbs/gEHYKlbDoRq4x4E= 2023-09-22T21:59:13.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aQ23vyxpTbs/gEHYKlbDoRq4x4E=', 'Content-Length': '0'} 2023-09-22T21:59:13.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1f36a7cdfdd30cb-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T21:59:13.912 DEBUG:boto:path=/ffzdqu-61/obj85 2023-09-22T21:59:13.912 DEBUG:boto:auth_path=/ffzdqu-61/obj85 2023-09-22T21:59:13.913 DEBUG:boto:Method: GET 2023-09-22T21:59:13.913 DEBUG:boto:Path: /ffzdqu-61/obj85 2023-09-22T21:59:13.913 DEBUG:boto:Data: 2023-09-22T21:59:13.913 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.913 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.913 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.913 DEBUG:boto:Params: {} 2023-09-22T21:59:13.913 DEBUG:boto:Token: None 2023-09-22T21:59:13.913 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj85 2023-09-22T21:59:13.914 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:USo2R2pzmLG6mQskDDZqdnq08wk= 2023-09-22T21:59:13.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:USo2R2pzmLG6mQskDDZqdnq08wk=', 'Content-Length': '0'} 2023-09-22T21:59:13.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002f9f23b43347ac1-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.917 DEBUG:boto:path=/ffzdqu-61/obj85 2023-09-22T21:59:13.917 DEBUG:boto:auth_path=/ffzdqu-61/obj85 2023-09-22T21:59:13.917 DEBUG:boto:Method: GET 2023-09-22T21:59:13.917 DEBUG:boto:Path: /ffzdqu-61/obj85 2023-09-22T21:59:13.917 DEBUG:boto:Data: 2023-09-22T21:59:13.917 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.917 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.917 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.918 DEBUG:boto:Params: {} 2023-09-22T21:59:13.918 DEBUG:boto:Token: None 2023-09-22T21:59:13.918 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj85 2023-09-22T21:59:13.918 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:USo2R2pzmLG6mQskDDZqdnq08wk= 2023-09-22T21:59:13.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:USo2R2pzmLG6mQskDDZqdnq08wk=', 'Content-Length': '0'} 2023-09-22T21:59:13.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002979597995ddb775-00650e0e31-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.962 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:13.962 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:13.963 DEBUG:boto:path=/ffzdqu-61/?prefix=obj85 2023-09-22T21:59:13.963 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj85 2023-09-22T21:59:13.963 DEBUG:boto:Method: GET 2023-09-22T21:59:13.963 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj85 2023-09-22T21:59:13.963 DEBUG:boto:Data: 2023-09-22T21:59:13.963 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.963 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.963 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.963 DEBUG:boto:Params: {} 2023-09-22T21:59:13.964 DEBUG:boto:Token: None 2023-09-22T21:59:13.964 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/ 2023-09-22T21:59:13.964 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw= 2023-09-22T21:59:13.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw=', 'Content-Length': '0'} 2023-09-22T21:59:13.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e4f7e1c9084c321-00650e0e31-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.968 DEBUG:boto:b'ffzdqu-61obj851000falseobj852023-09-22T21:56:09.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:13.969 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:13.969 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:13.969 DEBUG:boto:path=/ffzdqu-61/?prefix=obj85 2023-09-22T21:59:13.969 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj85 2023-09-22T21:59:13.969 DEBUG:boto:Method: GET 2023-09-22T21:59:13.969 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj85 2023-09-22T21:59:13.970 DEBUG:boto:Data: 2023-09-22T21:59:13.970 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.970 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:13.970 DEBUG:boto:Port: 8000 2023-09-22T21:59:13.970 DEBUG:boto:Params: {} 2023-09-22T21:59:13.970 DEBUG:boto:Token: None 2023-09-22T21:59:13.970 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/ 2023-09-22T21:59:13.970 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw= 2023-09-22T21:59:13.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:NOEcBv+Px01iFWgaQsgDmnXdlPw=', 'Content-Length': '0'} 2023-09-22T21:59:13.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001775fcdf2881d0d2-00650e0e31-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:13.982 DEBUG:boto:b'ffzdqu-61obj851000falseobj852023-09-22T21:56:09.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:13.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-22T21:59:13.983 DEBUG:boto:path=/ffzdqu-61/obj87 2023-09-22T21:59:13.983 DEBUG:boto:auth_path=/ffzdqu-61/obj87 2023-09-22T21:59:13.983 DEBUG:boto:path=/ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:13.983 DEBUG:boto:auth_path=/ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:13.983 DEBUG:boto:Method: GET 2023-09-22T21:59:13.983 DEBUG:boto:Path: /ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:13.983 DEBUG:boto:Data: 2023-09-22T21:59:13.983 DEBUG:boto:Headers: {} 2023-09-22T21:59:13.984 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:13.984 DEBUG:boto:Port: 8001 2023-09-22T21:59:13.984 DEBUG:boto:Params: {} 2023-09-22T21:59:13.984 DEBUG:boto:Token: None 2023-09-22T21:59:13.984 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:13 GMT /ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:13.984 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SK8Ap/oyXRkyUuADiRICLn5dxh4= 2023-09-22T21:59:13.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SK8Ap/oyXRkyUuADiRICLn5dxh4=', 'Content-Length': '0'} 2023-09-22T21:59:13.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe0945b5ad5e1f52-00650e0e31-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.030 DEBUG:boto:path=/ffzdqu-61/obj87 2023-09-22T21:59:14.030 DEBUG:boto:auth_path=/ffzdqu-61/obj87 2023-09-22T21:59:14.031 DEBUG:boto:path=/ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:14.031 DEBUG:boto:auth_path=/ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:14.031 DEBUG:boto:Method: GET 2023-09-22T21:59:14.031 DEBUG:boto:Path: /ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:14.031 DEBUG:boto:Data: 2023-09-22T21:59:14.031 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.031 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.031 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.031 DEBUG:boto:Params: {} 2023-09-22T21:59:14.031 DEBUG:boto:Token: None 2023-09-22T21:59:14.032 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:14.032 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wMO6ubd4CL3k5Cmb7OLKeNiq4s4= 2023-09-22T21:59:14.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wMO6ubd4CL3k5Cmb7OLKeNiq4s4=', 'Content-Length': '0'} 2023-09-22T21:59:14.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052fc9e7c4220e135-00650e0e32-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.035 DEBUG:boto:path=/ffzdqu-61/obj87 2023-09-22T21:59:14.035 DEBUG:boto:auth_path=/ffzdqu-61/obj87 2023-09-22T21:59:14.035 DEBUG:boto:path=/ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:14.035 DEBUG:boto:auth_path=/ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:14.035 DEBUG:boto:Method: HEAD 2023-09-22T21:59:14.035 DEBUG:boto:Path: /ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:14.036 DEBUG:boto:Data: 2023-09-22T21:59:14.036 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.036 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.036 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.036 DEBUG:boto:Params: {} 2023-09-22T21:59:14.036 DEBUG:boto:Token: None 2023-09-22T21:59:14.036 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:14.036 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XugrWJ6XeLS8rhZIKUer9507IFs= 2023-09-22T21:59:14.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XugrWJ6XeLS8rhZIKUer9507IFs=', 'Content-Length': '0'} 2023-09-22T21:59:14.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000819dca42361e8d92-00650e0e32-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.039 DEBUG:boto:path=/ffzdqu-61/obj87 2023-09-22T21:59:14.039 DEBUG:boto:auth_path=/ffzdqu-61/obj87 2023-09-22T21:59:14.039 DEBUG:boto:path=/ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:14.039 DEBUG:boto:auth_path=/ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:14.040 DEBUG:boto:Method: HEAD 2023-09-22T21:59:14.040 DEBUG:boto:Path: /ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:14.040 DEBUG:boto:Data: 2023-09-22T21:59:14.040 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.040 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.040 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.040 DEBUG:boto:Params: {} 2023-09-22T21:59:14.040 DEBUG:boto:Token: None 2023-09-22T21:59:14.040 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:14.041 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XugrWJ6XeLS8rhZIKUer9507IFs= 2023-09-22T21:59:14.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XugrWJ6XeLS8rhZIKUer9507IFs=', 'Content-Length': '0'} 2023-09-22T21:59:14.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000147bedf15698c6c-00650e0e32-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-22T21:59:14.044 DEBUG:boto:path=/ffzdqu-61/obj87 2023-09-22T21:59:14.044 DEBUG:boto:auth_path=/ffzdqu-61/obj87 2023-09-22T21:59:14.044 DEBUG:boto:path=/ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:14.044 DEBUG:boto:auth_path=/ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:14.044 DEBUG:boto:Method: GET 2023-09-22T21:59:14.044 DEBUG:boto:Path: /ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:14.044 DEBUG:boto:Data: 2023-09-22T21:59:14.044 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.044 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.045 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.045 DEBUG:boto:Params: {} 2023-09-22T21:59:14.045 DEBUG:boto:Token: None 2023-09-22T21:59:14.045 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:14.045 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wMO6ubd4CL3k5Cmb7OLKeNiq4s4= 2023-09-22T21:59:14.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wMO6ubd4CL3k5Cmb7OLKeNiq4s4=', 'Content-Length': '0'} 2023-09-22T21:59:14.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce3b44457114b09e-00650e0e32-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.048 DEBUG:boto:path=/ffzdqu-61/obj87 2023-09-22T21:59:14.048 DEBUG:boto:auth_path=/ffzdqu-61/obj87 2023-09-22T21:59:14.048 DEBUG:boto:path=/ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:14.048 DEBUG:boto:auth_path=/ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:14.048 DEBUG:boto:Method: GET 2023-09-22T21:59:14.049 DEBUG:boto:Path: /ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:14.049 DEBUG:boto:Data: 2023-09-22T21:59:14.049 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.049 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.049 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.049 DEBUG:boto:Params: {} 2023-09-22T21:59:14.049 DEBUG:boto:Token: None 2023-09-22T21:59:14.049 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj87?versionId=null 2023-09-22T21:59:14.049 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wMO6ubd4CL3k5Cmb7OLKeNiq4s4= 2023-09-22T21:59:14.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wMO6ubd4CL3k5Cmb7OLKeNiq4s4=', 'Content-Length': '0'} 2023-09-22T21:59:14.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2ac08ce6e3f9fc6-00650e0e32-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.098 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:14.098 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:14.099 DEBUG:boto:path=/ffzdqu-61/?prefix=obj87 2023-09-22T21:59:14.099 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj87 2023-09-22T21:59:14.099 DEBUG:boto:Method: GET 2023-09-22T21:59:14.099 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj87 2023-09-22T21:59:14.099 DEBUG:boto:Data: 2023-09-22T21:59:14.099 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.099 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.099 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.099 DEBUG:boto:Params: {} 2023-09-22T21:59:14.099 DEBUG:boto:Token: None 2023-09-22T21:59:14.100 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/ 2023-09-22T21:59:14.100 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo= 2023-09-22T21:59:14.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo=', 'Content-Length': '0'} 2023-09-22T21:59:14.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b0cbd55b691c832-00650e0e32-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.105 DEBUG:boto:b'ffzdqu-61obj871000falseobj872023-09-22T21:56:09.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:14.105 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:14.105 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:14.105 DEBUG:boto:path=/ffzdqu-61/?prefix=obj87 2023-09-22T21:59:14.105 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj87 2023-09-22T21:59:14.105 DEBUG:boto:Method: GET 2023-09-22T21:59:14.106 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj87 2023-09-22T21:59:14.106 DEBUG:boto:Data: 2023-09-22T21:59:14.106 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.106 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.106 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.106 DEBUG:boto:Params: {} 2023-09-22T21:59:14.106 DEBUG:boto:Token: None 2023-09-22T21:59:14.106 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/ 2023-09-22T21:59:14.106 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo= 2023-09-22T21:59:14.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo=', 'Content-Length': '0'} 2023-09-22T21:59:14.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2beca842ccbd0df-00650e0e32-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.116 DEBUG:boto:b'ffzdqu-61obj871000falseobj872023-09-22T21:56:09.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:14.117 DEBUG:boto:path=/ffzdqu-61/obj87 2023-09-22T21:59:14.117 DEBUG:boto:auth_path=/ffzdqu-61/obj87 2023-09-22T21:59:14.117 DEBUG:boto:Method: HEAD 2023-09-22T21:59:14.117 DEBUG:boto:Path: /ffzdqu-61/obj87 2023-09-22T21:59:14.117 DEBUG:boto:Data: 2023-09-22T21:59:14.117 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.117 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.117 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.117 DEBUG:boto:Params: {} 2023-09-22T21:59:14.118 DEBUG:boto:Token: None 2023-09-22T21:59:14.118 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj87 2023-09-22T21:59:14.118 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3MPxN7Q9K2QrvbQEKOm/kHKnKro= 2023-09-22T21:59:14.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3MPxN7Q9K2QrvbQEKOm/kHKnKro=', 'Content-Length': '0'} 2023-09-22T21:59:14.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007643f04ef39da563-00650e0e32-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.121 DEBUG:boto:path=/ffzdqu-61/obj87 2023-09-22T21:59:14.121 DEBUG:boto:auth_path=/ffzdqu-61/obj87 2023-09-22T21:59:14.121 DEBUG:boto:Method: HEAD 2023-09-22T21:59:14.121 DEBUG:boto:Path: /ffzdqu-61/obj87 2023-09-22T21:59:14.121 DEBUG:boto:Data: 2023-09-22T21:59:14.121 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.122 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.122 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.122 DEBUG:boto:Params: {} 2023-09-22T21:59:14.122 DEBUG:boto:Token: None 2023-09-22T21:59:14.122 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj87 2023-09-22T21:59:14.122 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3MPxN7Q9K2QrvbQEKOm/kHKnKro= 2023-09-22T21:59:14.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3MPxN7Q9K2QrvbQEKOm/kHKnKro=', 'Content-Length': '0'} 2023-09-22T21:59:14.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002841e7875c87ae9c-00650e0e32-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-22T21:59:14.125 DEBUG:boto:path=/ffzdqu-61/obj87 2023-09-22T21:59:14.125 DEBUG:boto:auth_path=/ffzdqu-61/obj87 2023-09-22T21:59:14.125 DEBUG:boto:Method: GET 2023-09-22T21:59:14.125 DEBUG:boto:Path: /ffzdqu-61/obj87 2023-09-22T21:59:14.126 DEBUG:boto:Data: 2023-09-22T21:59:14.126 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.126 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.126 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.126 DEBUG:boto:Params: {} 2023-09-22T21:59:14.126 DEBUG:boto:Token: None 2023-09-22T21:59:14.126 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj87 2023-09-22T21:59:14.126 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P+MNg2q0FI2zQSFbH12mqcHEO94= 2023-09-22T21:59:14.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P+MNg2q0FI2zQSFbH12mqcHEO94=', 'Content-Length': '0'} 2023-09-22T21:59:14.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dba30d33ecf6c50-00650e0e32-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.170 DEBUG:boto:path=/ffzdqu-61/obj87 2023-09-22T21:59:14.170 DEBUG:boto:auth_path=/ffzdqu-61/obj87 2023-09-22T21:59:14.170 DEBUG:boto:Method: GET 2023-09-22T21:59:14.171 DEBUG:boto:Path: /ffzdqu-61/obj87 2023-09-22T21:59:14.171 DEBUG:boto:Data: 2023-09-22T21:59:14.171 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.171 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.171 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.171 DEBUG:boto:Params: {} 2023-09-22T21:59:14.171 DEBUG:boto:Token: None 2023-09-22T21:59:14.171 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj87 2023-09-22T21:59:14.171 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:P+MNg2q0FI2zQSFbH12mqcHEO94= 2023-09-22T21:59:14.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:P+MNg2q0FI2zQSFbH12mqcHEO94=', 'Content-Length': '0'} 2023-09-22T21:59:14.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063559c2f05ace0b8-00650e0e32-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.175 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:14.175 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:14.175 DEBUG:boto:path=/ffzdqu-61/?prefix=obj87 2023-09-22T21:59:14.175 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj87 2023-09-22T21:59:14.175 DEBUG:boto:Method: GET 2023-09-22T21:59:14.175 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj87 2023-09-22T21:59:14.175 DEBUG:boto:Data: 2023-09-22T21:59:14.175 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.176 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.176 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.176 DEBUG:boto:Params: {} 2023-09-22T21:59:14.176 DEBUG:boto:Token: None 2023-09-22T21:59:14.176 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/ 2023-09-22T21:59:14.176 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo= 2023-09-22T21:59:14.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo=', 'Content-Length': '0'} 2023-09-22T21:59:14.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e11230295e01f5d-00650e0e32-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.181 DEBUG:boto:b'ffzdqu-61obj871000falseobj872023-09-22T21:56:09.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:14.181 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:14.181 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:14.182 DEBUG:boto:path=/ffzdqu-61/?prefix=obj87 2023-09-22T21:59:14.182 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj87 2023-09-22T21:59:14.182 DEBUG:boto:Method: GET 2023-09-22T21:59:14.182 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj87 2023-09-22T21:59:14.182 DEBUG:boto:Data: 2023-09-22T21:59:14.182 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.182 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.182 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.182 DEBUG:boto:Params: {} 2023-09-22T21:59:14.183 DEBUG:boto:Token: None 2023-09-22T21:59:14.183 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/ 2023-09-22T21:59:14.183 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo= 2023-09-22T21:59:14.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo=', 'Content-Length': '0'} 2023-09-22T21:59:14.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ea3614a7605ad4c-00650e0e32-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.238 DEBUG:boto:b'ffzdqu-61obj871000falseobj872023-09-22T21:56:09.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:14.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-22T21:59:14.239 DEBUG:boto:path=/ffzdqu-61/obj92 2023-09-22T21:59:14.239 DEBUG:boto:auth_path=/ffzdqu-61/obj92 2023-09-22T21:59:14.239 DEBUG:boto:path=/ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.239 DEBUG:boto:auth_path=/ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.239 DEBUG:boto:Method: GET 2023-09-22T21:59:14.239 DEBUG:boto:Path: /ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.239 DEBUG:boto:Data: 2023-09-22T21:59:14.239 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.240 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.240 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.240 DEBUG:boto:Params: {} 2023-09-22T21:59:14.240 DEBUG:boto:Token: None 2023-09-22T21:59:14.240 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.240 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H0kvUIiEpYo9M8RSyRjX8bMjp4k= 2023-09-22T21:59:14.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H0kvUIiEpYo9M8RSyRjX8bMjp4k=', 'Content-Length': '0'} 2023-09-22T21:59:14.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea1735358af38437-00650e0e32-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.244 DEBUG:boto:path=/ffzdqu-61/obj92 2023-09-22T21:59:14.244 DEBUG:boto:auth_path=/ffzdqu-61/obj92 2023-09-22T21:59:14.244 DEBUG:boto:path=/ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.244 DEBUG:boto:auth_path=/ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.244 DEBUG:boto:Method: GET 2023-09-22T21:59:14.244 DEBUG:boto:Path: /ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.245 DEBUG:boto:Data: 2023-09-22T21:59:14.245 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.245 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.245 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.245 DEBUG:boto:Params: {} 2023-09-22T21:59:14.245 DEBUG:boto:Token: None 2023-09-22T21:59:14.245 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.245 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H0kvUIiEpYo9M8RSyRjX8bMjp4k= 2023-09-22T21:59:14.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H0kvUIiEpYo9M8RSyRjX8bMjp4k=', 'Content-Length': '0'} 2023-09-22T21:59:14.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068999d8a5806db6c-00650e0e32-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.250 DEBUG:boto:path=/ffzdqu-61/obj92 2023-09-22T21:59:14.250 DEBUG:boto:auth_path=/ffzdqu-61/obj92 2023-09-22T21:59:14.250 DEBUG:boto:path=/ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.250 DEBUG:boto:auth_path=/ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.250 DEBUG:boto:Method: HEAD 2023-09-22T21:59:14.250 DEBUG:boto:Path: /ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.250 DEBUG:boto:Data: 2023-09-22T21:59:14.251 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.251 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.251 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.251 DEBUG:boto:Params: {} 2023-09-22T21:59:14.251 DEBUG:boto:Token: None 2023-09-22T21:59:14.251 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.251 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1zInu2GbXqQz4f54JmAQjGT/qoE= 2023-09-22T21:59:14.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1zInu2GbXqQz4f54JmAQjGT/qoE=', 'Content-Length': '0'} 2023-09-22T21:59:14.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073e618fc3f4e2d9d-00650e0e32-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.254 DEBUG:boto:path=/ffzdqu-61/obj92 2023-09-22T21:59:14.254 DEBUG:boto:auth_path=/ffzdqu-61/obj92 2023-09-22T21:59:14.254 DEBUG:boto:path=/ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.254 DEBUG:boto:auth_path=/ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.254 DEBUG:boto:Method: HEAD 2023-09-22T21:59:14.254 DEBUG:boto:Path: /ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.255 DEBUG:boto:Data: 2023-09-22T21:59:14.255 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.255 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.255 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.255 DEBUG:boto:Params: {} 2023-09-22T21:59:14.255 DEBUG:boto:Token: None 2023-09-22T21:59:14.255 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.255 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1zInu2GbXqQz4f54JmAQjGT/qoE= 2023-09-22T21:59:14.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1zInu2GbXqQz4f54JmAQjGT/qoE=', 'Content-Length': '0'} 2023-09-22T21:59:14.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8ec6032e990b29d-00650e0e32-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-22T21:59:14.258 DEBUG:boto:path=/ffzdqu-61/obj92 2023-09-22T21:59:14.258 DEBUG:boto:auth_path=/ffzdqu-61/obj92 2023-09-22T21:59:14.259 DEBUG:boto:path=/ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.259 DEBUG:boto:auth_path=/ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.259 DEBUG:boto:Method: GET 2023-09-22T21:59:14.259 DEBUG:boto:Path: /ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.259 DEBUG:boto:Data: 2023-09-22T21:59:14.259 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.259 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.259 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.259 DEBUG:boto:Params: {} 2023-09-22T21:59:14.259 DEBUG:boto:Token: None 2023-09-22T21:59:14.260 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.260 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H0kvUIiEpYo9M8RSyRjX8bMjp4k= 2023-09-22T21:59:14.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H0kvUIiEpYo9M8RSyRjX8bMjp4k=', 'Content-Length': '0'} 2023-09-22T21:59:14.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036a7e6f2b7b7d69a-00650e0e32-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.263 DEBUG:boto:path=/ffzdqu-61/obj92 2023-09-22T21:59:14.263 DEBUG:boto:auth_path=/ffzdqu-61/obj92 2023-09-22T21:59:14.263 DEBUG:boto:path=/ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.263 DEBUG:boto:auth_path=/ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.263 DEBUG:boto:Method: GET 2023-09-22T21:59:14.263 DEBUG:boto:Path: /ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.263 DEBUG:boto:Data: 2023-09-22T21:59:14.263 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.263 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.264 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.264 DEBUG:boto:Params: {} 2023-09-22T21:59:14.264 DEBUG:boto:Token: None 2023-09-22T21:59:14.264 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj92?versionId=null 2023-09-22T21:59:14.264 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:H0kvUIiEpYo9M8RSyRjX8bMjp4k= 2023-09-22T21:59:14.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:H0kvUIiEpYo9M8RSyRjX8bMjp4k=', 'Content-Length': '0'} 2023-09-22T21:59:14.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aff417dcc4c7be8e-00650e0e32-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.267 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:14.267 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:14.267 DEBUG:boto:path=/ffzdqu-61/?prefix=obj92 2023-09-22T21:59:14.267 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj92 2023-09-22T21:59:14.267 DEBUG:boto:Method: GET 2023-09-22T21:59:14.268 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj92 2023-09-22T21:59:14.268 DEBUG:boto:Data: 2023-09-22T21:59:14.268 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.268 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.268 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.268 DEBUG:boto:Params: {} 2023-09-22T21:59:14.268 DEBUG:boto:Token: None 2023-09-22T21:59:14.268 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/ 2023-09-22T21:59:14.268 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo= 2023-09-22T21:59:14.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo=', 'Content-Length': '0'} 2023-09-22T21:59:14.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e92b998eb6320fff-00650e0e32-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.318 DEBUG:boto:b'ffzdqu-61obj921000falseobj922023-09-22T21:56:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:14.319 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:14.319 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:14.319 DEBUG:boto:path=/ffzdqu-61/?prefix=obj92 2023-09-22T21:59:14.319 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj92 2023-09-22T21:59:14.319 DEBUG:boto:Method: GET 2023-09-22T21:59:14.319 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj92 2023-09-22T21:59:14.319 DEBUG:boto:Data: 2023-09-22T21:59:14.319 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.319 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.320 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.320 DEBUG:boto:Params: {} 2023-09-22T21:59:14.320 DEBUG:boto:Token: None 2023-09-22T21:59:14.320 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/ 2023-09-22T21:59:14.320 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo= 2023-09-22T21:59:14.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo=', 'Content-Length': '0'} 2023-09-22T21:59:14.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9206227f59a98d3-00650e0e32-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.329 DEBUG:boto:b'ffzdqu-61obj921000falseobj922023-09-22T21:56:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:14.329 DEBUG:boto:path=/ffzdqu-61/obj92 2023-09-22T21:59:14.330 DEBUG:boto:auth_path=/ffzdqu-61/obj92 2023-09-22T21:59:14.330 DEBUG:boto:Method: HEAD 2023-09-22T21:59:14.330 DEBUG:boto:Path: /ffzdqu-61/obj92 2023-09-22T21:59:14.330 DEBUG:boto:Data: 2023-09-22T21:59:14.330 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.330 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.330 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.330 DEBUG:boto:Params: {} 2023-09-22T21:59:14.330 DEBUG:boto:Token: None 2023-09-22T21:59:14.331 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj92 2023-09-22T21:59:14.331 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5ENIpAdEYm4wFkAEWxGu9KoCh4A= 2023-09-22T21:59:14.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5ENIpAdEYm4wFkAEWxGu9KoCh4A=', 'Content-Length': '0'} 2023-09-22T21:59:14.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099d2fb14ed7abc65-00650e0e32-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.334 DEBUG:boto:path=/ffzdqu-61/obj92 2023-09-22T21:59:14.334 DEBUG:boto:auth_path=/ffzdqu-61/obj92 2023-09-22T21:59:14.334 DEBUG:boto:Method: HEAD 2023-09-22T21:59:14.334 DEBUG:boto:Path: /ffzdqu-61/obj92 2023-09-22T21:59:14.334 DEBUG:boto:Data: 2023-09-22T21:59:14.334 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.334 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.335 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.335 DEBUG:boto:Params: {} 2023-09-22T21:59:14.335 DEBUG:boto:Token: None 2023-09-22T21:59:14.335 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj92 2023-09-22T21:59:14.335 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5ENIpAdEYm4wFkAEWxGu9KoCh4A= 2023-09-22T21:59:14.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5ENIpAdEYm4wFkAEWxGu9KoCh4A=', 'Content-Length': '0'} 2023-09-22T21:59:14.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044ce50b376f07b8f-00650e0e32-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-22T21:59:14.338 DEBUG:boto:path=/ffzdqu-61/obj92 2023-09-22T21:59:14.338 DEBUG:boto:auth_path=/ffzdqu-61/obj92 2023-09-22T21:59:14.338 DEBUG:boto:Method: GET 2023-09-22T21:59:14.338 DEBUG:boto:Path: /ffzdqu-61/obj92 2023-09-22T21:59:14.338 DEBUG:boto:Data: 2023-09-22T21:59:14.338 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.339 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.339 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.339 DEBUG:boto:Params: {} 2023-09-22T21:59:14.339 DEBUG:boto:Token: None 2023-09-22T21:59:14.339 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj92 2023-09-22T21:59:14.339 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f84zQStfLYZTrEevMKYnH1dCwL0= 2023-09-22T21:59:14.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f84zQStfLYZTrEevMKYnH1dCwL0=', 'Content-Length': '0'} 2023-09-22T21:59:14.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000b88345a082c4fa-00650e0e32-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.342 DEBUG:boto:path=/ffzdqu-61/obj92 2023-09-22T21:59:14.342 DEBUG:boto:auth_path=/ffzdqu-61/obj92 2023-09-22T21:59:14.342 DEBUG:boto:Method: GET 2023-09-22T21:59:14.342 DEBUG:boto:Path: /ffzdqu-61/obj92 2023-09-22T21:59:14.343 DEBUG:boto:Data: 2023-09-22T21:59:14.343 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.343 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.343 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.343 DEBUG:boto:Params: {} 2023-09-22T21:59:14.343 DEBUG:boto:Token: None 2023-09-22T21:59:14.343 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj92 2023-09-22T21:59:14.343 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f84zQStfLYZTrEevMKYnH1dCwL0= 2023-09-22T21:59:14.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f84zQStfLYZTrEevMKYnH1dCwL0=', 'Content-Length': '0'} 2023-09-22T21:59:14.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008614346598a96ae9-00650e0e32-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.386 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:14.386 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:14.386 DEBUG:boto:path=/ffzdqu-61/?prefix=obj92 2023-09-22T21:59:14.387 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj92 2023-09-22T21:59:14.387 DEBUG:boto:Method: GET 2023-09-22T21:59:14.387 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj92 2023-09-22T21:59:14.387 DEBUG:boto:Data: 2023-09-22T21:59:14.387 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.387 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.387 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.387 DEBUG:boto:Params: {} 2023-09-22T21:59:14.387 DEBUG:boto:Token: None 2023-09-22T21:59:14.388 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/ 2023-09-22T21:59:14.388 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo= 2023-09-22T21:59:14.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo=', 'Content-Length': '0'} 2023-09-22T21:59:14.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023c224fc2f0c4cd1-00650e0e32-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.434 DEBUG:boto:b'ffzdqu-61obj921000falseobj922023-09-22T21:56:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:14.435 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:14.435 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:14.435 DEBUG:boto:path=/ffzdqu-61/?prefix=obj92 2023-09-22T21:59:14.435 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj92 2023-09-22T21:59:14.435 DEBUG:boto:Method: GET 2023-09-22T21:59:14.435 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj92 2023-09-22T21:59:14.435 DEBUG:boto:Data: 2023-09-22T21:59:14.435 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.436 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.436 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.436 DEBUG:boto:Params: {} 2023-09-22T21:59:14.436 DEBUG:boto:Token: None 2023-09-22T21:59:14.436 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/ 2023-09-22T21:59:14.436 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo= 2023-09-22T21:59:14.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo=', 'Content-Length': '0'} 2023-09-22T21:59:14.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc1b1c8b7f751515-00650e0e32-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.445 DEBUG:boto:b'ffzdqu-61obj921000falseobj922023-09-22T21:56:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:14.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-22T21:59:14.446 DEBUG:boto:path=/ffzdqu-61/obj93 2023-09-22T21:59:14.446 DEBUG:boto:auth_path=/ffzdqu-61/obj93 2023-09-22T21:59:14.446 DEBUG:boto:path=/ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.446 DEBUG:boto:auth_path=/ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.446 DEBUG:boto:Method: GET 2023-09-22T21:59:14.446 DEBUG:boto:Path: /ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.446 DEBUG:boto:Data: 2023-09-22T21:59:14.446 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.446 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.447 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.447 DEBUG:boto:Params: {} 2023-09-22T21:59:14.447 DEBUG:boto:Token: None 2023-09-22T21:59:14.447 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.447 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G435XRFcQ+fSv4h1FKs5SwuZxqE= 2023-09-22T21:59:14.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G435XRFcQ+fSv4h1FKs5SwuZxqE=', 'Content-Length': '0'} 2023-09-22T21:59:14.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000686b85d2b49119b2-00650e0e32-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.450 DEBUG:boto:path=/ffzdqu-61/obj93 2023-09-22T21:59:14.450 DEBUG:boto:auth_path=/ffzdqu-61/obj93 2023-09-22T21:59:14.450 DEBUG:boto:path=/ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.451 DEBUG:boto:auth_path=/ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.451 DEBUG:boto:Method: GET 2023-09-22T21:59:14.451 DEBUG:boto:Path: /ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.451 DEBUG:boto:Data: 2023-09-22T21:59:14.451 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.451 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.451 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.451 DEBUG:boto:Params: {} 2023-09-22T21:59:14.451 DEBUG:boto:Token: None 2023-09-22T21:59:14.452 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.452 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G435XRFcQ+fSv4h1FKs5SwuZxqE= 2023-09-22T21:59:14.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G435XRFcQ+fSv4h1FKs5SwuZxqE=', 'Content-Length': '0'} 2023-09-22T21:59:14.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdfb734396aef450-00650e0e32-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.455 DEBUG:boto:path=/ffzdqu-61/obj93 2023-09-22T21:59:14.455 DEBUG:boto:auth_path=/ffzdqu-61/obj93 2023-09-22T21:59:14.455 DEBUG:boto:path=/ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.455 DEBUG:boto:auth_path=/ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.455 DEBUG:boto:Method: HEAD 2023-09-22T21:59:14.455 DEBUG:boto:Path: /ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.455 DEBUG:boto:Data: 2023-09-22T21:59:14.455 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.455 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.456 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.456 DEBUG:boto:Params: {} 2023-09-22T21:59:14.456 DEBUG:boto:Token: None 2023-09-22T21:59:14.456 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.456 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D8vzgHTz88MD3Sp11RAx1cVQtdY= 2023-09-22T21:59:14.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D8vzgHTz88MD3Sp11RAx1cVQtdY=', 'Content-Length': '0'} 2023-09-22T21:59:14.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c16bfc87ab73ac8e-00650e0e32-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.459 DEBUG:boto:path=/ffzdqu-61/obj93 2023-09-22T21:59:14.459 DEBUG:boto:auth_path=/ffzdqu-61/obj93 2023-09-22T21:59:14.459 DEBUG:boto:path=/ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.459 DEBUG:boto:auth_path=/ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.459 DEBUG:boto:Method: HEAD 2023-09-22T21:59:14.460 DEBUG:boto:Path: /ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.460 DEBUG:boto:Data: 2023-09-22T21:59:14.460 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.460 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.460 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.460 DEBUG:boto:Params: {} 2023-09-22T21:59:14.460 DEBUG:boto:Token: None 2023-09-22T21:59:14.460 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.460 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D8vzgHTz88MD3Sp11RAx1cVQtdY= 2023-09-22T21:59:14.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D8vzgHTz88MD3Sp11RAx1cVQtdY=', 'Content-Length': '0'} 2023-09-22T21:59:14.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2a9d42c0cac9d2b-00650e0e32-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-22T21:59:14.464 DEBUG:boto:path=/ffzdqu-61/obj93 2023-09-22T21:59:14.464 DEBUG:boto:auth_path=/ffzdqu-61/obj93 2023-09-22T21:59:14.464 DEBUG:boto:path=/ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.464 DEBUG:boto:auth_path=/ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.464 DEBUG:boto:Method: GET 2023-09-22T21:59:14.464 DEBUG:boto:Path: /ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.464 DEBUG:boto:Data: 2023-09-22T21:59:14.464 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.464 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.464 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.465 DEBUG:boto:Params: {} 2023-09-22T21:59:14.465 DEBUG:boto:Token: None 2023-09-22T21:59:14.465 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.465 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G435XRFcQ+fSv4h1FKs5SwuZxqE= 2023-09-22T21:59:14.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G435XRFcQ+fSv4h1FKs5SwuZxqE=', 'Content-Length': '0'} 2023-09-22T21:59:14.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000654504b0175b1afe-00650e0e32-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.510 DEBUG:boto:path=/ffzdqu-61/obj93 2023-09-22T21:59:14.510 DEBUG:boto:auth_path=/ffzdqu-61/obj93 2023-09-22T21:59:14.510 DEBUG:boto:path=/ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.511 DEBUG:boto:auth_path=/ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.511 DEBUG:boto:Method: GET 2023-09-22T21:59:14.511 DEBUG:boto:Path: /ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.511 DEBUG:boto:Data: 2023-09-22T21:59:14.511 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.511 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.511 DEBUG:boto:Params: {} 2023-09-22T21:59:14.511 DEBUG:boto:Token: None 2023-09-22T21:59:14.512 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj93?versionId=null 2023-09-22T21:59:14.512 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G435XRFcQ+fSv4h1FKs5SwuZxqE= 2023-09-22T21:59:14.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G435XRFcQ+fSv4h1FKs5SwuZxqE=', 'Content-Length': '0'} 2023-09-22T21:59:14.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcc4d40a896a1d79-00650e0e32-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.515 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:14.515 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:14.515 DEBUG:boto:path=/ffzdqu-61/?prefix=obj93 2023-09-22T21:59:14.515 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj93 2023-09-22T21:59:14.515 DEBUG:boto:Method: GET 2023-09-22T21:59:14.516 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj93 2023-09-22T21:59:14.516 DEBUG:boto:Data: 2023-09-22T21:59:14.516 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.516 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.516 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.516 DEBUG:boto:Params: {} 2023-09-22T21:59:14.516 DEBUG:boto:Token: None 2023-09-22T21:59:14.516 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/ 2023-09-22T21:59:14.516 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo= 2023-09-22T21:59:14.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo=', 'Content-Length': '0'} 2023-09-22T21:59:14.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cde1bc27868bfd96-00650e0e32-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.525 DEBUG:boto:b'ffzdqu-61obj931000falseobj932023-09-22T21:56:10.304Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:14.525 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:14.525 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:14.525 DEBUG:boto:path=/ffzdqu-61/?prefix=obj93 2023-09-22T21:59:14.525 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj93 2023-09-22T21:59:14.526 DEBUG:boto:Method: GET 2023-09-22T21:59:14.526 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj93 2023-09-22T21:59:14.526 DEBUG:boto:Data: 2023-09-22T21:59:14.526 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.526 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.526 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.526 DEBUG:boto:Params: {} 2023-09-22T21:59:14.526 DEBUG:boto:Token: None 2023-09-22T21:59:14.526 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/ 2023-09-22T21:59:14.527 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo= 2023-09-22T21:59:14.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo=', 'Content-Length': '0'} 2023-09-22T21:59:14.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec9dd939abccfbf6-00650e0e32-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.578 DEBUG:boto:b'ffzdqu-61obj931000falseobj932023-09-22T21:56:10.304Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:14.579 DEBUG:boto:path=/ffzdqu-61/obj93 2023-09-22T21:59:14.579 DEBUG:boto:auth_path=/ffzdqu-61/obj93 2023-09-22T21:59:14.579 DEBUG:boto:Method: HEAD 2023-09-22T21:59:14.579 DEBUG:boto:Path: /ffzdqu-61/obj93 2023-09-22T21:59:14.579 DEBUG:boto:Data: 2023-09-22T21:59:14.579 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.579 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.579 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.579 DEBUG:boto:Params: {} 2023-09-22T21:59:14.580 DEBUG:boto:Token: None 2023-09-22T21:59:14.580 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj93 2023-09-22T21:59:14.580 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eKBJR+yhXGYFZYRwW23xQDt0dGU= 2023-09-22T21:59:14.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eKBJR+yhXGYFZYRwW23xQDt0dGU=', 'Content-Length': '0'} 2023-09-22T21:59:14.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebe55a4e0d85c8a6-00650e0e32-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.583 DEBUG:boto:path=/ffzdqu-61/obj93 2023-09-22T21:59:14.583 DEBUG:boto:auth_path=/ffzdqu-61/obj93 2023-09-22T21:59:14.583 DEBUG:boto:Method: HEAD 2023-09-22T21:59:14.583 DEBUG:boto:Path: /ffzdqu-61/obj93 2023-09-22T21:59:14.584 DEBUG:boto:Data: 2023-09-22T21:59:14.584 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.584 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.584 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.584 DEBUG:boto:Params: {} 2023-09-22T21:59:14.584 DEBUG:boto:Token: None 2023-09-22T21:59:14.584 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj93 2023-09-22T21:59:14.584 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eKBJR+yhXGYFZYRwW23xQDt0dGU= 2023-09-22T21:59:14.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eKBJR+yhXGYFZYRwW23xQDt0dGU=', 'Content-Length': '0'} 2023-09-22T21:59:14.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d837e6131ae7a8c3-00650e0e32-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-22T21:59:14.587 DEBUG:boto:path=/ffzdqu-61/obj93 2023-09-22T21:59:14.588 DEBUG:boto:auth_path=/ffzdqu-61/obj93 2023-09-22T21:59:14.588 DEBUG:boto:Method: GET 2023-09-22T21:59:14.588 DEBUG:boto:Path: /ffzdqu-61/obj93 2023-09-22T21:59:14.588 DEBUG:boto:Data: 2023-09-22T21:59:14.588 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.588 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.588 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.588 DEBUG:boto:Params: {} 2023-09-22T21:59:14.588 DEBUG:boto:Token: None 2023-09-22T21:59:14.589 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj93 2023-09-22T21:59:14.589 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Seu+3KNFAOBsLHjD8OpkUUe1fz0= 2023-09-22T21:59:14.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Seu+3KNFAOBsLHjD8OpkUUe1fz0=', 'Content-Length': '0'} 2023-09-22T21:59:14.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e76725e92c3aefd8-00650e0e32-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.638 DEBUG:boto:path=/ffzdqu-61/obj93 2023-09-22T21:59:14.638 DEBUG:boto:auth_path=/ffzdqu-61/obj93 2023-09-22T21:59:14.638 DEBUG:boto:Method: GET 2023-09-22T21:59:14.639 DEBUG:boto:Path: /ffzdqu-61/obj93 2023-09-22T21:59:14.639 DEBUG:boto:Data: 2023-09-22T21:59:14.639 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.639 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.639 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.639 DEBUG:boto:Params: {} 2023-09-22T21:59:14.639 DEBUG:boto:Token: None 2023-09-22T21:59:14.639 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj93 2023-09-22T21:59:14.639 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Seu+3KNFAOBsLHjD8OpkUUe1fz0= 2023-09-22T21:59:14.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Seu+3KNFAOBsLHjD8OpkUUe1fz0=', 'Content-Length': '0'} 2023-09-22T21:59:14.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5ea05f4b6b59424-00650e0e32-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.642 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:14.643 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:14.643 DEBUG:boto:path=/ffzdqu-61/?prefix=obj93 2023-09-22T21:59:14.643 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj93 2023-09-22T21:59:14.643 DEBUG:boto:Method: GET 2023-09-22T21:59:14.643 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj93 2023-09-22T21:59:14.643 DEBUG:boto:Data: 2023-09-22T21:59:14.643 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.643 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.643 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.644 DEBUG:boto:Params: {} 2023-09-22T21:59:14.644 DEBUG:boto:Token: None 2023-09-22T21:59:14.644 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/ 2023-09-22T21:59:14.644 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo= 2023-09-22T21:59:14.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo=', 'Content-Length': '0'} 2023-09-22T21:59:14.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb79f639fa449e70-00650e0e32-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.649 DEBUG:boto:b'ffzdqu-61obj931000falseobj932023-09-22T21:56:10.304Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:14.649 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:14.649 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:14.649 DEBUG:boto:path=/ffzdqu-61/?prefix=obj93 2023-09-22T21:59:14.649 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj93 2023-09-22T21:59:14.649 DEBUG:boto:Method: GET 2023-09-22T21:59:14.650 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj93 2023-09-22T21:59:14.650 DEBUG:boto:Data: 2023-09-22T21:59:14.650 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.650 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.650 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.650 DEBUG:boto:Params: {} 2023-09-22T21:59:14.650 DEBUG:boto:Token: None 2023-09-22T21:59:14.650 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/ 2023-09-22T21:59:14.650 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo= 2023-09-22T21:59:14.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo=', 'Content-Length': '0'} 2023-09-22T21:59:14.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b73bb7fa866b270c-00650e0e32-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.664 DEBUG:boto:b'ffzdqu-61obj931000falseobj932023-09-22T21:56:10.304Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:14.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T21:59:14.665 DEBUG:boto:path=/ffzdqu-61/obj95 2023-09-22T21:59:14.665 DEBUG:boto:auth_path=/ffzdqu-61/obj95 2023-09-22T21:59:14.665 DEBUG:boto:path=/ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.665 DEBUG:boto:auth_path=/ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.665 DEBUG:boto:Method: GET 2023-09-22T21:59:14.665 DEBUG:boto:Path: /ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.665 DEBUG:boto:Data: 2023-09-22T21:59:14.666 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.666 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.666 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.666 DEBUG:boto:Params: {} 2023-09-22T21:59:14.666 DEBUG:boto:Token: None 2023-09-22T21:59:14.666 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.666 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cOm2APe55Z37A7Bkz6I8vIaH4Lo= 2023-09-22T21:59:14.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cOm2APe55Z37A7Bkz6I8vIaH4Lo=', 'Content-Length': '0'} 2023-09-22T21:59:14.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b09ead5e409c13e-00650e0e32-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.670 DEBUG:boto:path=/ffzdqu-61/obj95 2023-09-22T21:59:14.670 DEBUG:boto:auth_path=/ffzdqu-61/obj95 2023-09-22T21:59:14.670 DEBUG:boto:path=/ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.670 DEBUG:boto:auth_path=/ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.670 DEBUG:boto:Method: GET 2023-09-22T21:59:14.670 DEBUG:boto:Path: /ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.670 DEBUG:boto:Data: 2023-09-22T21:59:14.670 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.670 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.671 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.671 DEBUG:boto:Params: {} 2023-09-22T21:59:14.671 DEBUG:boto:Token: None 2023-09-22T21:59:14.671 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.671 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cOm2APe55Z37A7Bkz6I8vIaH4Lo= 2023-09-22T21:59:14.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cOm2APe55Z37A7Bkz6I8vIaH4Lo=', 'Content-Length': '0'} 2023-09-22T21:59:14.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e85871de6bf32b1-00650e0e32-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.722 DEBUG:boto:path=/ffzdqu-61/obj95 2023-09-22T21:59:14.722 DEBUG:boto:auth_path=/ffzdqu-61/obj95 2023-09-22T21:59:14.722 DEBUG:boto:path=/ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.723 DEBUG:boto:auth_path=/ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.723 DEBUG:boto:Method: HEAD 2023-09-22T21:59:14.723 DEBUG:boto:Path: /ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.723 DEBUG:boto:Data: 2023-09-22T21:59:14.723 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.723 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.723 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.723 DEBUG:boto:Params: {} 2023-09-22T21:59:14.723 DEBUG:boto:Token: None 2023-09-22T21:59:14.724 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.724 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Camy3OODzG07WrHTN2u+9E+817s= 2023-09-22T21:59:14.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Camy3OODzG07WrHTN2u+9E+817s=', 'Content-Length': '0'} 2023-09-22T21:59:14.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd20d7f43a03fd7c-00650e0e32-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.727 DEBUG:boto:path=/ffzdqu-61/obj95 2023-09-22T21:59:14.727 DEBUG:boto:auth_path=/ffzdqu-61/obj95 2023-09-22T21:59:14.727 DEBUG:boto:path=/ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.727 DEBUG:boto:auth_path=/ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.727 DEBUG:boto:Method: HEAD 2023-09-22T21:59:14.727 DEBUG:boto:Path: /ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.728 DEBUG:boto:Data: 2023-09-22T21:59:14.728 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.728 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.728 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.728 DEBUG:boto:Params: {} 2023-09-22T21:59:14.728 DEBUG:boto:Token: None 2023-09-22T21:59:14.728 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.728 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Camy3OODzG07WrHTN2u+9E+817s= 2023-09-22T21:59:14.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Camy3OODzG07WrHTN2u+9E+817s=', 'Content-Length': '0'} 2023-09-22T21:59:14.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000080fbe3362e19f4e-00650e0e32-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T21:59:14.731 DEBUG:boto:path=/ffzdqu-61/obj95 2023-09-22T21:59:14.731 DEBUG:boto:auth_path=/ffzdqu-61/obj95 2023-09-22T21:59:14.731 DEBUG:boto:path=/ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.732 DEBUG:boto:auth_path=/ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.732 DEBUG:boto:Method: GET 2023-09-22T21:59:14.732 DEBUG:boto:Path: /ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.732 DEBUG:boto:Data: 2023-09-22T21:59:14.732 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.732 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.732 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.732 DEBUG:boto:Params: {} 2023-09-22T21:59:14.732 DEBUG:boto:Token: None 2023-09-22T21:59:14.733 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.733 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cOm2APe55Z37A7Bkz6I8vIaH4Lo= 2023-09-22T21:59:14.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cOm2APe55Z37A7Bkz6I8vIaH4Lo=', 'Content-Length': '0'} 2023-09-22T21:59:14.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f888fa9f6fdbe1ac-00650e0e32-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.782 DEBUG:boto:path=/ffzdqu-61/obj95 2023-09-22T21:59:14.782 DEBUG:boto:auth_path=/ffzdqu-61/obj95 2023-09-22T21:59:14.783 DEBUG:boto:path=/ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.783 DEBUG:boto:auth_path=/ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.783 DEBUG:boto:Method: GET 2023-09-22T21:59:14.783 DEBUG:boto:Path: /ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.783 DEBUG:boto:Data: 2023-09-22T21:59:14.783 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.783 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.783 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.783 DEBUG:boto:Params: {} 2023-09-22T21:59:14.784 DEBUG:boto:Token: None 2023-09-22T21:59:14.784 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj95?versionId=null 2023-09-22T21:59:14.784 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cOm2APe55Z37A7Bkz6I8vIaH4Lo= 2023-09-22T21:59:14.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cOm2APe55Z37A7Bkz6I8vIaH4Lo=', 'Content-Length': '0'} 2023-09-22T21:59:14.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d06ec533eef20351-00650e0e32-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.788 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:14.788 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:14.788 DEBUG:boto:path=/ffzdqu-61/?prefix=obj95 2023-09-22T21:59:14.788 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj95 2023-09-22T21:59:14.788 DEBUG:boto:Method: GET 2023-09-22T21:59:14.788 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj95 2023-09-22T21:59:14.788 DEBUG:boto:Data: 2023-09-22T21:59:14.788 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.789 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.789 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.789 DEBUG:boto:Params: {} 2023-09-22T21:59:14.789 DEBUG:boto:Token: None 2023-09-22T21:59:14.789 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/ 2023-09-22T21:59:14.789 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo= 2023-09-22T21:59:14.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo=', 'Content-Length': '0'} 2023-09-22T21:59:14.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb098f8a553b20c2-00650e0e32-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.834 DEBUG:boto:b'ffzdqu-61obj951000falseobj952023-09-22T21:56:09.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:14.834 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:14.835 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:14.835 DEBUG:boto:path=/ffzdqu-61/?prefix=obj95 2023-09-22T21:59:14.835 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj95 2023-09-22T21:59:14.835 DEBUG:boto:Method: GET 2023-09-22T21:59:14.835 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj95 2023-09-22T21:59:14.835 DEBUG:boto:Data: 2023-09-22T21:59:14.835 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.835 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.835 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.836 DEBUG:boto:Params: {} 2023-09-22T21:59:14.836 DEBUG:boto:Token: None 2023-09-22T21:59:14.836 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/ 2023-09-22T21:59:14.836 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo= 2023-09-22T21:59:14.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo=', 'Content-Length': '0'} 2023-09-22T21:59:14.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adcc3e80c108921a-00650e0e32-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.847 DEBUG:boto:b'ffzdqu-61obj951000falseobj952023-09-22T21:56:09.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:14.847 DEBUG:boto:path=/ffzdqu-61/obj95 2023-09-22T21:59:14.847 DEBUG:boto:auth_path=/ffzdqu-61/obj95 2023-09-22T21:59:14.847 DEBUG:boto:Method: HEAD 2023-09-22T21:59:14.847 DEBUG:boto:Path: /ffzdqu-61/obj95 2023-09-22T21:59:14.847 DEBUG:boto:Data: 2023-09-22T21:59:14.848 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.848 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.848 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.848 DEBUG:boto:Params: {} 2023-09-22T21:59:14.848 DEBUG:boto:Token: None 2023-09-22T21:59:14.848 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj95 2023-09-22T21:59:14.848 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AtERQHhmhHbsI/4mK0ATbhJ6M9g= 2023-09-22T21:59:14.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AtERQHhmhHbsI/4mK0ATbhJ6M9g=', 'Content-Length': '0'} 2023-09-22T21:59:14.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ad7ef8861fec633-00650e0e32-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.852 DEBUG:boto:path=/ffzdqu-61/obj95 2023-09-22T21:59:14.852 DEBUG:boto:auth_path=/ffzdqu-61/obj95 2023-09-22T21:59:14.852 DEBUG:boto:Method: HEAD 2023-09-22T21:59:14.852 DEBUG:boto:Path: /ffzdqu-61/obj95 2023-09-22T21:59:14.852 DEBUG:boto:Data: 2023-09-22T21:59:14.852 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.852 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.852 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.852 DEBUG:boto:Params: {} 2023-09-22T21:59:14.853 DEBUG:boto:Token: None 2023-09-22T21:59:14.853 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj95 2023-09-22T21:59:14.853 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AtERQHhmhHbsI/4mK0ATbhJ6M9g= 2023-09-22T21:59:14.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AtERQHhmhHbsI/4mK0ATbhJ6M9g=', 'Content-Length': '0'} 2023-09-22T21:59:14.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d171b4d9ba7e292-00650e0e32-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T21:59:14.857 DEBUG:boto:path=/ffzdqu-61/obj95 2023-09-22T21:59:14.857 DEBUG:boto:auth_path=/ffzdqu-61/obj95 2023-09-22T21:59:14.857 DEBUG:boto:Method: GET 2023-09-22T21:59:14.857 DEBUG:boto:Path: /ffzdqu-61/obj95 2023-09-22T21:59:14.857 DEBUG:boto:Data: 2023-09-22T21:59:14.857 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.857 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.857 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.858 DEBUG:boto:Params: {} 2023-09-22T21:59:14.858 DEBUG:boto:Token: None 2023-09-22T21:59:14.858 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj95 2023-09-22T21:59:14.858 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PGV7dUDGTkmAFc6pqNit6serZgY= 2023-09-22T21:59:14.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PGV7dUDGTkmAFc6pqNit6serZgY=', 'Content-Length': '0'} 2023-09-22T21:59:14.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-61:f132bdb9-20a7-4073-a932-8f99da20691e.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060987a2cf299c1af-00650e0e32-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.902 DEBUG:boto:path=/ffzdqu-61/obj95 2023-09-22T21:59:14.902 DEBUG:boto:auth_path=/ffzdqu-61/obj95 2023-09-22T21:59:14.903 DEBUG:boto:Method: GET 2023-09-22T21:59:14.903 DEBUG:boto:Path: /ffzdqu-61/obj95 2023-09-22T21:59:14.903 DEBUG:boto:Data: 2023-09-22T21:59:14.903 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.903 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.903 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.903 DEBUG:boto:Params: {} 2023-09-22T21:59:14.903 DEBUG:boto:Token: None 2023-09-22T21:59:14.903 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/obj95 2023-09-22T21:59:14.904 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PGV7dUDGTkmAFc6pqNit6serZgY= 2023-09-22T21:59:14.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PGV7dUDGTkmAFc6pqNit6serZgY=', 'Content-Length': '0'} 2023-09-22T21:59:14.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1b05df89d1416c8-00650e0e32-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.907 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:14.907 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:14.907 DEBUG:boto:path=/ffzdqu-61/?prefix=obj95 2023-09-22T21:59:14.907 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj95 2023-09-22T21:59:14.907 DEBUG:boto:Method: GET 2023-09-22T21:59:14.907 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj95 2023-09-22T21:59:14.908 DEBUG:boto:Data: 2023-09-22T21:59:14.908 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.908 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:14.908 DEBUG:boto:Port: 8001 2023-09-22T21:59:14.908 DEBUG:boto:Params: {} 2023-09-22T21:59:14.908 DEBUG:boto:Token: None 2023-09-22T21:59:14.908 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/ 2023-09-22T21:59:14.908 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo= 2023-09-22T21:59:14.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo=', 'Content-Length': '0'} 2023-09-22T21:59:14.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000029369842d9b5072-00650e0e32-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.914 DEBUG:boto:b'ffzdqu-61obj951000falseobj952023-09-22T21:56:09.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:14.914 DEBUG:boto:path=/ffzdqu-61/ 2023-09-22T21:59:14.914 DEBUG:boto:auth_path=/ffzdqu-61/ 2023-09-22T21:59:14.914 DEBUG:boto:path=/ffzdqu-61/?prefix=obj95 2023-09-22T21:59:14.915 DEBUG:boto:auth_path=/ffzdqu-61/?prefix=obj95 2023-09-22T21:59:14.915 DEBUG:boto:Method: GET 2023-09-22T21:59:14.915 DEBUG:boto:Path: /ffzdqu-61/?prefix=obj95 2023-09-22T21:59:14.915 DEBUG:boto:Data: 2023-09-22T21:59:14.915 DEBUG:boto:Headers: {} 2023-09-22T21:59:14.915 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:14.915 DEBUG:boto:Port: 8000 2023-09-22T21:59:14.915 DEBUG:boto:Params: {} 2023-09-22T21:59:14.915 DEBUG:boto:Token: None 2023-09-22T21:59:14.915 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:14 GMT /ffzdqu-61/ 2023-09-22T21:59:14.916 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo= 2023-09-22T21:59:14.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:32MGJPNB4bmo11FtzwINtPukueo=', 'Content-Length': '0'} 2023-09-22T21:59:14.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b674ed80935830fc-00650e0e32-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:14.970 DEBUG:boto:b'ffzdqu-61obj951000falseobj952023-09-22T21:56:09.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:14.971 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-61 zones={a2, a1} 2023-09-22T21:59:14.971 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-09-22T21:59:14.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:59:14.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:59:14.971 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:59:15.013 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:59:15.013 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:59:15.077 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"00000000000000000000:00000000000000023140","last_update":"2023-09-22T21:56:25.521611Z"},{"marker":"00000000000000000000:00000000000000037247","last_update":"2023-09-22T21:56:10.806425Z"},{"marker":"00000000000000000000:00000000000000033007","last_update":"2023-09-22T21:56:25.525405Z"},{"marker":"00000000000000000000:00000000000000032629","last_update":"2023-09-22T21:56:25.528804Z"}] 2023-09-22T21:59:15.078 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000023140', 1: '00000000000000000000:00000000000000037247', 2: '00000000000000000000:00000000000000033007', 3: '00000000000000000000:00000000000000032629'} 2023-09-22T21:59:15.078 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-22T21:59:15.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:59:15.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:59:15.078 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:59:15.158 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:59:15.159 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:59:15.213 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":5568447401830590353},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000023140","next_step_marker":"","total_entries":9,"pos":0,"timestamp":"2023-09-22T21:56:25.521611Z"}},{"key":1,"val":{"status":"full-sync","marker":"ffzdqu-43:f132bdb9-20a7-4073-a932-8f99da20691e.5729.8:0","next_step_marker":"00000000000000000000:00000000000000037247","total_entries":52,"pos":34,"timestamp":"2023-09-22T21:56:10.806425Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000033007","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:56:25.525405Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000032629","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:56:25.528804Z"}}]},"full_sync":{"total":61,"complete":43},"current_time":"2023-09-22T21:59:15Z"} 2023-09-22T21:59:15.213 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":5568447401830590353},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000023140","next_step_marker":"","total_entries":9,"pos":0,"timestamp":"2023-09-22T21:56:25.521611Z"}},{"key":1,"val":{"status":"full-sync","marker":"ffzdqu-43:f132bdb9-20a7-4073-a932-8f99da20691e.5729.8:0","next_step_marker":"00000000000000000000:00000000000000037247","total_entries":52,"pos":34,"timestamp":"2023-09-22T21:56:10.806425Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000033007","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:56:25.525405Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000032629","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:56:25.528804Z"}}]},"full_sync":{"total":61,"complete":43},"current_time":"2023-09-22T21:59:15Z"} 2023-09-22T21:59:15.214 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023140', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-22T21:56:25.521611Z'}}, {'key': 1, 'val': {'status': 'full-sync', 'marker': 'ffzdqu-43:f132bdb9-20a7-4073-a932-8f99da20691e.5729.8:0', 'next_step_marker': '00000000000000000000:00000000000000037247', 'total_entries': 52, 'pos': 34, 'timestamp': '2023-09-22T21:56:10.806425Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033007', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:56:25.525405Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032629', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:56:25.528804Z'}}] 2023-09-22T21:59:15.214 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000023140', 1: '00000000000000000000:00000000000000037247', 2: '00000000000000000000:00000000000000033007', 3: '00000000000000000000:00000000000000032629'} 2023-09-22T21:59:15.214 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000023140', 1: 'ffzdqu-43:f132bdb9-20a7-4073-a932-8f99da20691e.5729.8:0', 2: '00000000000000000000:00000000000000033007', 3: '00000000000000000000:00000000000000032629'} 2023-09-22T21:59:15.214 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-22T21:59:15.214 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-09-22T21:59:15.214 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:59:15.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:59:15.217 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:59:15.287 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:59:15.288 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:59:15.352 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"00000000000000000000:00000000000000005582","last_update":"2023-09-22T21:58:01.075511Z"},{"marker":"00000000000000000000:00000000000000015165","last_update":"2023-09-22T21:58:01.078980Z"},{"marker":"00000000000000000000:00000000000000013621","last_update":"2023-09-22T21:58:01.082292Z"},{"marker":"00000000000000000000:00000000000000011535","last_update":"2023-09-22T21:58:01.085895Z"}] 2023-09-22T21:59:15.353 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000015165', 2: '00000000000000000000:00000000000000013621', 3: '00000000000000000000:00000000000000011535'} 2023-09-22T21:59:15.353 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-09-22T21:59:15.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:59:15.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:59:15.353 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:59:15.439 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:59:15.439 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:59:15.497 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10476880264182873081},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005582","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:58:01.075511Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000015165","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:58:01.078980Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000013621","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:58:01.082292Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000011535","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:58:01.085895Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:59:15Z"} 2023-09-22T21:59:15.497 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10476880264182873081},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005582","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:58:01.075511Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000015165","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:58:01.078980Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000013621","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:58:01.082292Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000011535","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:58:01.085895Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2023-09-22T21:59:15Z"} 2023-09-22T21:59:15.497 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:58:01.075511Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015165', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:58:01.078980Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013621', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:58:01.082292Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011535', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:58:01.085895Z'}}] 2023-09-22T21:59:15.498 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000015165', 2: '00000000000000000000:00000000000000013621', 3: '00000000000000000000:00000000000000011535'} 2023-09-22T21:59:15.498 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000015165', 2: '00000000000000000000:00000000000000013621', 3: '00000000000000000000:00000000000000011535'} 2023-09-22T21:59:15.498 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-09-22T21:59:15.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... ok 2023-09-22T21:59:15.499 DEBUG:boto:path=/ffzdqu-62/ 2023-09-22T21:59:15.500 DEBUG:boto:auth_path=/ffzdqu-62/ 2023-09-22T21:59:15.500 DEBUG:boto:Method: PUT 2023-09-22T21:59:15.500 DEBUG:boto:Path: /ffzdqu-62/ 2023-09-22T21:59:15.500 DEBUG:boto:Data: 2023-09-22T21:59:15.500 DEBUG:boto:Headers: {} 2023-09-22T21:59:15.500 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:15.500 DEBUG:boto:Port: 8000 2023-09-22T21:59:15.500 DEBUG:boto:Params: {} 2023-09-22T21:59:15.500 DEBUG:boto:Token: None 2023-09-22T21:59:15.501 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 21:59:15 GMT /ffzdqu-62/ 2023-09-22T21:59:15.501 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KIjkGkLx+Aoy4Z2hwb2ldHKgSNw= 2023-09-22T21:59:15.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KIjkGkLx+Aoy4Z2hwb2ldHKgSNw=', 'Content-Length': '0'} 2023-09-22T21:59:15.520 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000072964fee0a5ba293-00650e0e33-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:15.520 DEBUG:rgw_multi.tests:created bucket=ffzdqu-62 2023-09-22T21:59:15.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:59:15.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:59:15.520 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:59:15.553 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:59:15.553 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:59:15.590 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T21:59:15.609 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"1_1695419384.000133_332.1","last_update":"2023-09-22T21:49:44.000133Z"},{"marker":"1_1695419955.520713_604.1","last_update":"2023-09-22T21:59:15.520713Z"},{"marker":"1_1695419383.971498_240.1","last_update":"2023-09-22T21:49:43.971498Z"}] 2023-09-22T21:59:15.609 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419955.520713_604.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:59:15.609 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T21:59:15.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:59:15.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T21:59:15.610 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T21:59:15.692 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:59:15.692 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:59:15.728 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:59:15.728+0000 7fcb262e4a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:59:15.729 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:59:15.728+0000 7fcb262e4a40 20 RGW-SYNC:meta: read sync status 2023-09-22T21:59:15.747 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419955.520713_604.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:59:15.520713Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:59:15Z"} 2023-09-22T21:59:15.747 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419955.520713_604.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:59:15.520713Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:59:15Z"} 2023-09-22T21:59:15.748 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695419955.520713_604.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:59:15.520713Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T21:59:15.748 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419955.520713_604.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:59:15.748 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419955.520713_604.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T21:59:15.748 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T21:59:15.748 DEBUG:boto:path=/ffzdqu-62/ 2023-09-22T21:59:15.748 DEBUG:boto:auth_path=/ffzdqu-62/ 2023-09-22T21:59:15.748 DEBUG:boto:Method: HEAD 2023-09-22T21:59:15.749 DEBUG:boto:Path: /ffzdqu-62/ 2023-09-22T21:59:15.749 DEBUG:boto:Data: 2023-09-22T21:59:15.749 DEBUG:boto:Headers: {} 2023-09-22T21:59:15.749 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:15.749 DEBUG:boto:Port: 8000 2023-09-22T21:59:15.749 DEBUG:boto:Params: {} 2023-09-22T21:59:15.749 DEBUG:boto:Token: None 2023-09-22T21:59:15.750 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:15 GMT /ffzdqu-62/ 2023-09-22T21:59:15.750 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wQNcODYrvpGbytqbF0wr7kur69g= 2023-09-22T21:59:15.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wQNcODYrvpGbytqbF0wr7kur69g=', 'Content-Length': '0'} 2023-09-22T21:59:15.753 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034dfe618db7807c2-00650e0e33-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:15.753 DEBUG:boto:path=/ffzdqu-62/dummy 2023-09-22T21:59:15.753 DEBUG:boto:auth_path=/ffzdqu-62/dummy 2023-09-22T21:59:15.753 DEBUG:boto:Method: PUT 2023-09-22T21:59:15.753 DEBUG:boto:Path: /ffzdqu-62/dummy 2023-09-22T21:59:15.754 DEBUG:boto:Data: 2023-09-22T21:59:15.754 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:59:15.754 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:15.754 DEBUG:boto:Port: 8000 2023-09-22T21:59:15.754 DEBUG:boto:Params: {} 2023-09-22T21:59:15.754 DEBUG:boto:Token: None 2023-09-22T21:59:15.754 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:59:15 GMT /ffzdqu-62/dummy 2023-09-22T21:59:15.754 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZhGAl+27u2pexLtw8H4f0bHB9C0= 2023-09-22T21:59:15.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:59:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZhGAl+27u2pexLtw8H4f0bHB9C0='} 2023-09-22T21:59:15.763 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3ff3a170283450e-00650e0e33-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:15.763 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-62 2023-09-22T21:59:15.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:59:15.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:59:15.763 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:59:15.834 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:59:15.835 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:59:15.874 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:59:15.872+0000 7f28598d8a40 1 waiting to reach incremental sync.. 2023-09-22T21:59:20.875 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:59:20.876+0000 7f28598d8a40 1 waiting to reach incremental sync.. 2023-09-22T21:59:25.878 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:59:25.876+0000 7f28598d8a40 1 bucket sync caught up with source: 2023-09-22T21:59:25.878 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T21:59:25.879 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T21:59:25.879 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:59:25.876+0000 7f28598d8a40 0 bucket checkpoint complete 2023-09-22T21:59:25.907 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-62 2023-09-22T21:59:25.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:59:25.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T21:59:25.908 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T21:59:25.955 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:59:25.955 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:59:26.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:59:26.000+0000 7f4b643d9a40 1 waiting to reach incremental sync.. 2023-09-22T21:59:31.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:59:31.000+0000 7f4b643d9a40 1 waiting to reach incremental sync.. 2023-09-22T21:59:36.001 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:59:36.000+0000 7f4b643d9a40 1 waiting to reach incremental sync.. 2023-09-22T21:59:36.005 INFO:tasks.rgw.c1.client.0.smithi160.stdout:2023-09-22T21:59:36.004+0000 7fa9c1675640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T21:59:41.004 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:59:41.004+0000 7f4b643d9a40 1 bucket sync caught up with source: 2023-09-22T21:59:41.005 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.362.6, , , , , ] 2023-09-22T21:59:41.005 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.362.6, , , , , ] 2023-09-22T21:59:41.006 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:59:41.004+0000 7f4b643d9a40 0 bucket checkpoint complete 2023-09-22T21:59:41.035 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-62 zones={a2, a1} 2023-09-22T21:59:41.035 DEBUG:boto:path=/ffzdqu-62/ 2023-09-22T21:59:41.035 DEBUG:boto:auth_path=/ffzdqu-62/ 2023-09-22T21:59:41.035 DEBUG:boto:Method: HEAD 2023-09-22T21:59:41.035 DEBUG:boto:Path: /ffzdqu-62/ 2023-09-22T21:59:41.036 DEBUG:boto:Data: 2023-09-22T21:59:41.036 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.036 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:41.036 DEBUG:boto:Port: 8001 2023-09-22T21:59:41.036 DEBUG:boto:Params: {} 2023-09-22T21:59:41.036 DEBUG:boto:Token: None 2023-09-22T21:59:41.036 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/ 2023-09-22T21:59:41.037 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI= 2023-09-22T21:59:41.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI=', 'Content-Length': '0'} 2023-09-22T21:59:41.040 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005841c00c28494a3d-00650e0e4d-1edb-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.041 DEBUG:boto:path=/ffzdqu-62/ 2023-09-22T21:59:41.041 DEBUG:boto:auth_path=/ffzdqu-62/ 2023-09-22T21:59:41.041 DEBUG:boto:Method: HEAD 2023-09-22T21:59:41.041 DEBUG:boto:Path: /ffzdqu-62/ 2023-09-22T21:59:41.041 DEBUG:boto:Data: 2023-09-22T21:59:41.041 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.041 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:41.041 DEBUG:boto:Port: 8000 2023-09-22T21:59:41.042 DEBUG:boto:Params: {} 2023-09-22T21:59:41.042 DEBUG:boto:Token: None 2023-09-22T21:59:41.042 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/ 2023-09-22T21:59:41.042 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI= 2023-09-22T21:59:41.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI=', 'Content-Length': '0'} 2023-09-22T21:59:41.047 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c18b68f049027c63-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.048 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T21:59:41.048 DEBUG:boto:path=/ffzdqu-62/ 2023-09-22T21:59:41.048 DEBUG:boto:auth_path=/ffzdqu-62/ 2023-09-22T21:59:41.048 DEBUG:boto:path=/ffzdqu-62/?versions&max-keys=999 2023-09-22T21:59:41.048 DEBUG:boto:auth_path=/ffzdqu-62/?versions&max-keys=999 2023-09-22T21:59:41.048 DEBUG:boto:Method: GET 2023-09-22T21:59:41.048 DEBUG:boto:Path: /ffzdqu-62/?versions&max-keys=999 2023-09-22T21:59:41.048 DEBUG:boto:Data: 2023-09-22T21:59:41.049 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.049 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:41.049 DEBUG:boto:Port: 8001 2023-09-22T21:59:41.049 DEBUG:boto:Params: {} 2023-09-22T21:59:41.049 DEBUG:boto:Token: None 2023-09-22T21:59:41.049 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/?versions 2023-09-22T21:59:41.049 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:whFfJb2kG8JY8Y2809KJdFShzdY= 2023-09-22T21:59:41.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:whFfJb2kG8JY8Y2809KJdFShzdY=', 'Content-Length': '0'} 2023-09-22T21:59:41.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af6689742bfcd8a2-00650e0e4d-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.055 DEBUG:boto:b'ffzdqu-62999falsedummynulltrue2023-09-22T21:59:15.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:41.056 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-22T21:59:41.056 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T21:59:41.056 DEBUG:boto:path=/ffzdqu-62/ 2023-09-22T21:59:41.056 DEBUG:boto:auth_path=/ffzdqu-62/ 2023-09-22T21:59:41.056 DEBUG:boto:path=/ffzdqu-62/?versions&max-keys=999 2023-09-22T21:59:41.056 DEBUG:boto:auth_path=/ffzdqu-62/?versions&max-keys=999 2023-09-22T21:59:41.056 DEBUG:boto:Method: GET 2023-09-22T21:59:41.057 DEBUG:boto:Path: /ffzdqu-62/?versions&max-keys=999 2023-09-22T21:59:41.057 DEBUG:boto:Data: 2023-09-22T21:59:41.057 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.057 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:41.057 DEBUG:boto:Port: 8000 2023-09-22T21:59:41.057 DEBUG:boto:Params: {} 2023-09-22T21:59:41.057 DEBUG:boto:Token: None 2023-09-22T21:59:41.057 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/?versions 2023-09-22T21:59:41.058 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:whFfJb2kG8JY8Y2809KJdFShzdY= 2023-09-22T21:59:41.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:whFfJb2kG8JY8Y2809KJdFShzdY=', 'Content-Length': '0'} 2023-09-22T21:59:41.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc3a124040236878-00650e0e4d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.062 DEBUG:boto:b'ffzdqu-62999falsedummynulltrue2023-09-22T21:59:15.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:41.063 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-22T21:59:41.063 DEBUG:boto:path=/ffzdqu-62/ 2023-09-22T21:59:41.063 DEBUG:boto:auth_path=/ffzdqu-62/ 2023-09-22T21:59:41.063 DEBUG:boto:path=/ffzdqu-62/?versions&max-keys=999 2023-09-22T21:59:41.063 DEBUG:boto:auth_path=/ffzdqu-62/?versions&max-keys=999 2023-09-22T21:59:41.063 DEBUG:boto:Method: GET 2023-09-22T21:59:41.063 DEBUG:boto:Path: /ffzdqu-62/?versions&max-keys=999 2023-09-22T21:59:41.064 DEBUG:boto:Data: 2023-09-22T21:59:41.064 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.064 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:41.064 DEBUG:boto:Port: 8001 2023-09-22T21:59:41.064 DEBUG:boto:Params: {} 2023-09-22T21:59:41.064 DEBUG:boto:Token: None 2023-09-22T21:59:41.064 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/?versions 2023-09-22T21:59:41.064 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:whFfJb2kG8JY8Y2809KJdFShzdY= 2023-09-22T21:59:41.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:whFfJb2kG8JY8Y2809KJdFShzdY=', 'Content-Length': '0'} 2023-09-22T21:59:41.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdeff3a8510f0bef-00650e0e4d-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.114 DEBUG:boto:b'ffzdqu-62999falsedummynulltrue2023-09-22T21:59:15.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:41.115 DEBUG:boto:path=/ffzdqu-62/ 2023-09-22T21:59:41.115 DEBUG:boto:auth_path=/ffzdqu-62/ 2023-09-22T21:59:41.115 DEBUG:boto:path=/ffzdqu-62/?versions&max-keys=999 2023-09-22T21:59:41.115 DEBUG:boto:auth_path=/ffzdqu-62/?versions&max-keys=999 2023-09-22T21:59:41.115 DEBUG:boto:Method: GET 2023-09-22T21:59:41.116 DEBUG:boto:Path: /ffzdqu-62/?versions&max-keys=999 2023-09-22T21:59:41.116 DEBUG:boto:Data: 2023-09-22T21:59:41.116 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.116 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:41.116 DEBUG:boto:Port: 8000 2023-09-22T21:59:41.116 DEBUG:boto:Params: {} 2023-09-22T21:59:41.116 DEBUG:boto:Token: None 2023-09-22T21:59:41.116 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/?versions 2023-09-22T21:59:41.117 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:whFfJb2kG8JY8Y2809KJdFShzdY= 2023-09-22T21:59:41.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:whFfJb2kG8JY8Y2809KJdFShzdY=', 'Content-Length': '0'} 2023-09-22T21:59:41.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001361a8be13945673-00650e0e4d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.122 DEBUG:boto:b'ffzdqu-62999falsedummynulltrue2023-09-22T21:59:15.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:41.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-22T21:59:41.122 DEBUG:boto:path=/ffzdqu-62/dummy 2023-09-22T21:59:41.122 DEBUG:boto:auth_path=/ffzdqu-62/dummy 2023-09-22T21:59:41.123 DEBUG:boto:path=/ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.123 DEBUG:boto:auth_path=/ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.123 DEBUG:boto:Method: GET 2023-09-22T21:59:41.123 DEBUG:boto:Path: /ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.123 DEBUG:boto:Data: 2023-09-22T21:59:41.123 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.123 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:41.123 DEBUG:boto:Port: 8001 2023-09-22T21:59:41.123 DEBUG:boto:Params: {} 2023-09-22T21:59:41.124 DEBUG:boto:Token: None 2023-09-22T21:59:41.124 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.124 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XK8T/qDhiV47hG/1jddhZE/IwdE= 2023-09-22T21:59:41.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XK8T/qDhiV47hG/1jddhZE/IwdE=', 'Content-Length': '0'} 2023-09-22T21:59:41.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-62:f132bdb9-20a7-4073-a932-8f99da20691e.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bd328ee38fd55a3-00650e0e4d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.127 DEBUG:boto:path=/ffzdqu-62/dummy 2023-09-22T21:59:41.127 DEBUG:boto:auth_path=/ffzdqu-62/dummy 2023-09-22T21:59:41.127 DEBUG:boto:path=/ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.128 DEBUG:boto:auth_path=/ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.128 DEBUG:boto:Method: GET 2023-09-22T21:59:41.128 DEBUG:boto:Path: /ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.128 DEBUG:boto:Data: 2023-09-22T21:59:41.128 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.128 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:41.128 DEBUG:boto:Port: 8000 2023-09-22T21:59:41.128 DEBUG:boto:Params: {} 2023-09-22T21:59:41.128 DEBUG:boto:Token: None 2023-09-22T21:59:41.129 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.129 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XK8T/qDhiV47hG/1jddhZE/IwdE= 2023-09-22T21:59:41.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XK8T/qDhiV47hG/1jddhZE/IwdE=', 'Content-Length': '0'} 2023-09-22T21:59:41.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078520e28c4f8d683-00650e0e4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.132 DEBUG:boto:path=/ffzdqu-62/dummy 2023-09-22T21:59:41.132 DEBUG:boto:auth_path=/ffzdqu-62/dummy 2023-09-22T21:59:41.132 DEBUG:boto:path=/ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.132 DEBUG:boto:auth_path=/ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.133 DEBUG:boto:Method: HEAD 2023-09-22T21:59:41.133 DEBUG:boto:Path: /ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.133 DEBUG:boto:Data: 2023-09-22T21:59:41.133 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.133 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:41.133 DEBUG:boto:Port: 8001 2023-09-22T21:59:41.133 DEBUG:boto:Params: {} 2023-09-22T21:59:41.133 DEBUG:boto:Token: None 2023-09-22T21:59:41.134 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.134 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BShKvrVw3lQWzqreBuyS1KfnsGY= 2023-09-22T21:59:41.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BShKvrVw3lQWzqreBuyS1KfnsGY=', 'Content-Length': '0'} 2023-09-22T21:59:41.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-62:f132bdb9-20a7-4073-a932-8f99da20691e.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff34d390cd8c32d3-00650e0e4d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.137 DEBUG:boto:path=/ffzdqu-62/dummy 2023-09-22T21:59:41.137 DEBUG:boto:auth_path=/ffzdqu-62/dummy 2023-09-22T21:59:41.137 DEBUG:boto:path=/ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.137 DEBUG:boto:auth_path=/ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.137 DEBUG:boto:Method: HEAD 2023-09-22T21:59:41.137 DEBUG:boto:Path: /ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.138 DEBUG:boto:Data: 2023-09-22T21:59:41.138 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.138 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:41.138 DEBUG:boto:Port: 8000 2023-09-22T21:59:41.138 DEBUG:boto:Params: {} 2023-09-22T21:59:41.138 DEBUG:boto:Token: None 2023-09-22T21:59:41.138 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.138 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BShKvrVw3lQWzqreBuyS1KfnsGY= 2023-09-22T21:59:41.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BShKvrVw3lQWzqreBuyS1KfnsGY=', 'Content-Length': '0'} 2023-09-22T21:59:41.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081994114629c5c41-00650e0e4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-22T21:59:41.142 DEBUG:boto:path=/ffzdqu-62/dummy 2023-09-22T21:59:41.142 DEBUG:boto:auth_path=/ffzdqu-62/dummy 2023-09-22T21:59:41.142 DEBUG:boto:path=/ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.142 DEBUG:boto:auth_path=/ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.142 DEBUG:boto:Method: GET 2023-09-22T21:59:41.142 DEBUG:boto:Path: /ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.142 DEBUG:boto:Data: 2023-09-22T21:59:41.142 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.143 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:41.143 DEBUG:boto:Port: 8001 2023-09-22T21:59:41.143 DEBUG:boto:Params: {} 2023-09-22T21:59:41.143 DEBUG:boto:Token: None 2023-09-22T21:59:41.143 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.143 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XK8T/qDhiV47hG/1jddhZE/IwdE= 2023-09-22T21:59:41.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XK8T/qDhiV47hG/1jddhZE/IwdE=', 'Content-Length': '0'} 2023-09-22T21:59:41.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-62:f132bdb9-20a7-4073-a932-8f99da20691e.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e59c2afd494104f-00650e0e4d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.146 DEBUG:boto:path=/ffzdqu-62/dummy 2023-09-22T21:59:41.146 DEBUG:boto:auth_path=/ffzdqu-62/dummy 2023-09-22T21:59:41.146 DEBUG:boto:path=/ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.147 DEBUG:boto:auth_path=/ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.147 DEBUG:boto:Method: GET 2023-09-22T21:59:41.147 DEBUG:boto:Path: /ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.147 DEBUG:boto:Data: 2023-09-22T21:59:41.147 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.147 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:41.147 DEBUG:boto:Port: 8000 2023-09-22T21:59:41.147 DEBUG:boto:Params: {} 2023-09-22T21:59:41.147 DEBUG:boto:Token: None 2023-09-22T21:59:41.148 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/dummy?versionId=null 2023-09-22T21:59:41.148 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XK8T/qDhiV47hG/1jddhZE/IwdE= 2023-09-22T21:59:41.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XK8T/qDhiV47hG/1jddhZE/IwdE=', 'Content-Length': '0'} 2023-09-22T21:59:41.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000031a2b72c9cc6b6d-00650e0e4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.198 DEBUG:boto:path=/ffzdqu-62/ 2023-09-22T21:59:41.199 DEBUG:boto:auth_path=/ffzdqu-62/ 2023-09-22T21:59:41.199 DEBUG:boto:path=/ffzdqu-62/?prefix=dummy 2023-09-22T21:59:41.199 DEBUG:boto:auth_path=/ffzdqu-62/?prefix=dummy 2023-09-22T21:59:41.199 DEBUG:boto:Method: GET 2023-09-22T21:59:41.199 DEBUG:boto:Path: /ffzdqu-62/?prefix=dummy 2023-09-22T21:59:41.199 DEBUG:boto:Data: 2023-09-22T21:59:41.199 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.200 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:41.200 DEBUG:boto:Port: 8001 2023-09-22T21:59:41.200 DEBUG:boto:Params: {} 2023-09-22T21:59:41.200 DEBUG:boto:Token: None 2023-09-22T21:59:41.200 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/ 2023-09-22T21:59:41.200 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HyhWzt8jfbBeBMV01XtFDFD3y28= 2023-09-22T21:59:41.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HyhWzt8jfbBeBMV01XtFDFD3y28=', 'Content-Length': '0'} 2023-09-22T21:59:41.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef0709acb1ffd8ad-00650e0e4d-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.250 DEBUG:boto:b'ffzdqu-62dummy1000falsedummy2023-09-22T21:59:15.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:41.251 DEBUG:boto:path=/ffzdqu-62/ 2023-09-22T21:59:41.251 DEBUG:boto:auth_path=/ffzdqu-62/ 2023-09-22T21:59:41.251 DEBUG:boto:path=/ffzdqu-62/?prefix=dummy 2023-09-22T21:59:41.251 DEBUG:boto:auth_path=/ffzdqu-62/?prefix=dummy 2023-09-22T21:59:41.251 DEBUG:boto:Method: GET 2023-09-22T21:59:41.251 DEBUG:boto:Path: /ffzdqu-62/?prefix=dummy 2023-09-22T21:59:41.251 DEBUG:boto:Data: 2023-09-22T21:59:41.252 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.252 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:41.252 DEBUG:boto:Port: 8000 2023-09-22T21:59:41.252 DEBUG:boto:Params: {} 2023-09-22T21:59:41.252 DEBUG:boto:Token: None 2023-09-22T21:59:41.252 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/ 2023-09-22T21:59:41.252 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HyhWzt8jfbBeBMV01XtFDFD3y28= 2023-09-22T21:59:41.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HyhWzt8jfbBeBMV01XtFDFD3y28=', 'Content-Length': '0'} 2023-09-22T21:59:41.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004adc563163cd9231-00650e0e4d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.258 DEBUG:boto:b'ffzdqu-62dummy1000falsedummy2023-09-22T21:59:15.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:41.259 DEBUG:boto:path=/ffzdqu-62/dummy 2023-09-22T21:59:41.259 DEBUG:boto:auth_path=/ffzdqu-62/dummy 2023-09-22T21:59:41.259 DEBUG:boto:Method: HEAD 2023-09-22T21:59:41.259 DEBUG:boto:Path: /ffzdqu-62/dummy 2023-09-22T21:59:41.259 DEBUG:boto:Data: 2023-09-22T21:59:41.259 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.259 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:41.259 DEBUG:boto:Port: 8001 2023-09-22T21:59:41.260 DEBUG:boto:Params: {} 2023-09-22T21:59:41.260 DEBUG:boto:Token: None 2023-09-22T21:59:41.260 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/dummy 2023-09-22T21:59:41.260 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+2ONW0AV9/gDBySOy0YU9MxEp6s= 2023-09-22T21:59:41.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+2ONW0AV9/gDBySOy0YU9MxEp6s=', 'Content-Length': '0'} 2023-09-22T21:59:41.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-62:f132bdb9-20a7-4073-a932-8f99da20691e.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000422c5817d1ea1d3-00650e0e4d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.263 DEBUG:boto:path=/ffzdqu-62/dummy 2023-09-22T21:59:41.263 DEBUG:boto:auth_path=/ffzdqu-62/dummy 2023-09-22T21:59:41.263 DEBUG:boto:Method: HEAD 2023-09-22T21:59:41.263 DEBUG:boto:Path: /ffzdqu-62/dummy 2023-09-22T21:59:41.263 DEBUG:boto:Data: 2023-09-22T21:59:41.264 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.264 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:41.264 DEBUG:boto:Port: 8000 2023-09-22T21:59:41.264 DEBUG:boto:Params: {} 2023-09-22T21:59:41.264 DEBUG:boto:Token: None 2023-09-22T21:59:41.264 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/dummy 2023-09-22T21:59:41.265 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+2ONW0AV9/gDBySOy0YU9MxEp6s= 2023-09-22T21:59:41.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+2ONW0AV9/gDBySOy0YU9MxEp6s=', 'Content-Length': '0'} 2023-09-22T21:59:41.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086406a2144efe5c9-00650e0e4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-22T21:59:41.268 DEBUG:boto:path=/ffzdqu-62/dummy 2023-09-22T21:59:41.268 DEBUG:boto:auth_path=/ffzdqu-62/dummy 2023-09-22T21:59:41.268 DEBUG:boto:Method: GET 2023-09-22T21:59:41.268 DEBUG:boto:Path: /ffzdqu-62/dummy 2023-09-22T21:59:41.268 DEBUG:boto:Data: 2023-09-22T21:59:41.268 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.268 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:41.268 DEBUG:boto:Port: 8001 2023-09-22T21:59:41.268 DEBUG:boto:Params: {} 2023-09-22T21:59:41.269 DEBUG:boto:Token: None 2023-09-22T21:59:41.269 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/dummy 2023-09-22T21:59:41.269 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FdjZcrUTxfvgqZgBlkXN4zVMr7A= 2023-09-22T21:59:41.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FdjZcrUTxfvgqZgBlkXN4zVMr7A=', 'Content-Length': '0'} 2023-09-22T21:59:41.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-62:f132bdb9-20a7-4073-a932-8f99da20691e.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2c000c86bea9e97-00650e0e4d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.314 DEBUG:boto:path=/ffzdqu-62/dummy 2023-09-22T21:59:41.314 DEBUG:boto:auth_path=/ffzdqu-62/dummy 2023-09-22T21:59:41.315 DEBUG:boto:Method: GET 2023-09-22T21:59:41.315 DEBUG:boto:Path: /ffzdqu-62/dummy 2023-09-22T21:59:41.315 DEBUG:boto:Data: 2023-09-22T21:59:41.315 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.315 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:41.315 DEBUG:boto:Port: 8000 2023-09-22T21:59:41.315 DEBUG:boto:Params: {} 2023-09-22T21:59:41.315 DEBUG:boto:Token: None 2023-09-22T21:59:41.316 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/dummy 2023-09-22T21:59:41.316 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FdjZcrUTxfvgqZgBlkXN4zVMr7A= 2023-09-22T21:59:41.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FdjZcrUTxfvgqZgBlkXN4zVMr7A=', 'Content-Length': '0'} 2023-09-22T21:59:41.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4f8a7da0dc8e06d-00650e0e4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.319 DEBUG:boto:path=/ffzdqu-62/ 2023-09-22T21:59:41.319 DEBUG:boto:auth_path=/ffzdqu-62/ 2023-09-22T21:59:41.319 DEBUG:boto:path=/ffzdqu-62/?prefix=dummy 2023-09-22T21:59:41.319 DEBUG:boto:auth_path=/ffzdqu-62/?prefix=dummy 2023-09-22T21:59:41.320 DEBUG:boto:Method: GET 2023-09-22T21:59:41.320 DEBUG:boto:Path: /ffzdqu-62/?prefix=dummy 2023-09-22T21:59:41.320 DEBUG:boto:Data: 2023-09-22T21:59:41.320 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.320 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:41.320 DEBUG:boto:Port: 8001 2023-09-22T21:59:41.320 DEBUG:boto:Params: {} 2023-09-22T21:59:41.320 DEBUG:boto:Token: None 2023-09-22T21:59:41.320 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/ 2023-09-22T21:59:41.321 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HyhWzt8jfbBeBMV01XtFDFD3y28= 2023-09-22T21:59:41.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HyhWzt8jfbBeBMV01XtFDFD3y28=', 'Content-Length': '0'} 2023-09-22T21:59:41.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047374d626296d3f8-00650e0e4d-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.326 DEBUG:boto:b'ffzdqu-62dummy1000falsedummy2023-09-22T21:59:15.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:41.327 DEBUG:boto:path=/ffzdqu-62/ 2023-09-22T21:59:41.327 DEBUG:boto:auth_path=/ffzdqu-62/ 2023-09-22T21:59:41.327 DEBUG:boto:path=/ffzdqu-62/?prefix=dummy 2023-09-22T21:59:41.327 DEBUG:boto:auth_path=/ffzdqu-62/?prefix=dummy 2023-09-22T21:59:41.327 DEBUG:boto:Method: GET 2023-09-22T21:59:41.327 DEBUG:boto:Path: /ffzdqu-62/?prefix=dummy 2023-09-22T21:59:41.327 DEBUG:boto:Data: 2023-09-22T21:59:41.327 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.327 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:41.328 DEBUG:boto:Port: 8000 2023-09-22T21:59:41.328 DEBUG:boto:Params: {} 2023-09-22T21:59:41.328 DEBUG:boto:Token: None 2023-09-22T21:59:41.328 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/ 2023-09-22T21:59:41.328 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HyhWzt8jfbBeBMV01XtFDFD3y28= 2023-09-22T21:59:41.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HyhWzt8jfbBeBMV01XtFDFD3y28=', 'Content-Length': '0'} 2023-09-22T21:59:41.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1a64e6a54359ba3-00650e0e4d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.333 DEBUG:boto:b'ffzdqu-62dummy1000falsedummy2023-09-22T21:59:15.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T21:59:41.334 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-62 zones={a2, a1} 2023-09-22T21:59:41.334 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-09-22T21:59:41.334 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T21:59:41.335 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:59:41.336+0000 7ff66bbbc640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 82185) UID: 0 2023-09-22T21:59:41.336 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:59:41.336+0000 7ff66f403a40 -1 shutting down 2023-09-22T21:59:41.359 INFO:tasks.rgw.c1.client.1.smithi160.stdout:./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7ff64a379640 time 2023-09-22T21:59:41.362087+0000 2023-09-22T21:59:41.359 INFO:tasks.rgw.c1.client.1.smithi160.stdout:./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T21:59:41.362 INFO:tasks.rgw.c1.client.1.smithi160.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T21:59:41.362 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7ff672f27883] 2023-09-22T21:59:41.362 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7ff672f27a37] 2023-09-22T21:59:41.362 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x559da71b446c] 2023-09-22T21:59:41.362 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x559da7596910] 2023-09-22T21:59:41.363 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x559da71c056e] 2023-09-22T21:59:41.363 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x559da71c1e7b] 2023-09-22T21:59:41.363 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x559da71c2ced] 2023-09-22T21:59:41.363 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x559da75c9712] 2023-09-22T21:59:41.363 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 9: radosgw(+0x836db8) [0x559da7374db8] 2023-09-22T21:59:41.363 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x559da7375753] 2023-09-22T21:59:41.363 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff671bf6b43] 2023-09-22T21:59:41.363 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff671c88a00] 2023-09-22T21:59:41.364 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:59:41.360+0000 7ff64a379640 -1 ./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7ff64a379640 time 2023-09-22T21:59:41.362087+0000 2023-09-22T21:59:41.364 INFO:tasks.rgw.c1.client.1.smithi160.stdout:./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T21:59:41.364 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T21:59:41.364 INFO:tasks.rgw.c1.client.1.smithi160.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T21:59:41.364 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7ff672f27883] 2023-09-22T21:59:41.365 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7ff672f27a37] 2023-09-22T21:59:41.365 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x559da71b446c] 2023-09-22T21:59:41.365 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x559da7596910] 2023-09-22T21:59:41.365 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x559da71c056e] 2023-09-22T21:59:41.365 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x559da71c1e7b] 2023-09-22T21:59:41.365 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x559da71c2ced] 2023-09-22T21:59:41.365 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x559da75c9712] 2023-09-22T21:59:41.365 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 9: radosgw(+0x836db8) [0x559da7374db8] 2023-09-22T21:59:41.366 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x559da7375753] 2023-09-22T21:59:41.366 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff671bf6b43] 2023-09-22T21:59:41.366 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff671c88a00] 2023-09-22T21:59:41.366 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T21:59:41.366 INFO:tasks.rgw.c1.client.1.smithi160.stdout:*** Caught signal (Aborted) ** 2023-09-22T21:59:41.366 INFO:tasks.rgw.c1.client.1.smithi160.stdout: in thread 7ff64a379640 thread_name:data-sync 2023-09-22T21:59:41.367 INFO:tasks.rgw.c1.client.1.smithi160.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T21:59:41.367 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff671ba4520] 2023-09-22T21:59:41.367 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2: pthread_kill() 2023-09-22T21:59:41.367 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 3: raise() 2023-09-22T21:59:41.367 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 4: abort() 2023-09-22T21:59:41.368 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7ff672f278dd] 2023-09-22T21:59:41.368 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7ff672f27a37] 2023-09-22T21:59:41.368 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x559da71b446c] 2023-09-22T21:59:41.368 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x559da7596910] 2023-09-22T21:59:41.368 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x559da71c056e] 2023-09-22T21:59:41.368 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x559da71c1e7b] 2023-09-22T21:59:41.368 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x559da71c2ced] 2023-09-22T21:59:41.368 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x559da75c9712] 2023-09-22T21:59:41.369 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 13: radosgw(+0x836db8) [0x559da7374db8] 2023-09-22T21:59:41.369 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x559da7375753] 2023-09-22T21:59:41.369 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff671bf6b43] 2023-09-22T21:59:41.369 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff671c88a00] 2023-09-22T21:59:41.369 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T21:59:41.364+0000 7ff64a379640 -1 *** Caught signal (Aborted) ** 2023-09-22T21:59:41.369 INFO:tasks.rgw.c1.client.1.smithi160.stdout: in thread 7ff64a379640 thread_name:data-sync 2023-09-22T21:59:41.369 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T21:59:41.370 INFO:tasks.rgw.c1.client.1.smithi160.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T21:59:41.370 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff671ba4520] 2023-09-22T21:59:41.370 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2: pthread_kill() 2023-09-22T21:59:41.370 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 3: raise() 2023-09-22T21:59:41.370 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 4: abort() 2023-09-22T21:59:41.370 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7ff672f278dd] 2023-09-22T21:59:41.370 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7ff672f27a37] 2023-09-22T21:59:41.370 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x559da71b446c] 2023-09-22T21:59:41.371 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x559da7596910] 2023-09-22T21:59:41.371 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x559da71c056e] 2023-09-22T21:59:41.371 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x559da71c1e7b] 2023-09-22T21:59:41.371 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x559da71c2ced] 2023-09-22T21:59:41.371 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x559da75c9712] 2023-09-22T21:59:41.371 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 13: radosgw(+0x836db8) [0x559da7374db8] 2023-09-22T21:59:41.371 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x559da7375753] 2023-09-22T21:59:41.372 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff671bf6b43] 2023-09-22T21:59:41.372 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff671c88a00] 2023-09-22T21:59:41.372 INFO:tasks.rgw.c1.client.1.smithi160.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-09-22T21:59:41.372 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T21:59:41.392 INFO:tasks.rgw.c1.client.1.smithi160.stdout: -87> 2023-09-22T21:59:41.336+0000 7ff66bbbc640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 82185) UID: 0 2023-09-22T21:59:41.393 INFO:tasks.rgw.c1.client.1.smithi160.stdout: -84> 2023-09-22T21:59:41.336+0000 7ff66f403a40 -1 shutting down 2023-09-22T21:59:41.393 INFO:tasks.rgw.c1.client.1.smithi160.stdout: -1> 2023-09-22T21:59:41.360+0000 7ff64a379640 -1 ./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7ff64a379640 time 2023-09-22T21:59:41.362087+0000 2023-09-22T21:59:41.393 INFO:tasks.rgw.c1.client.1.smithi160.stdout:./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T21:59:41.393 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T21:59:41.393 INFO:tasks.rgw.c1.client.1.smithi160.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T21:59:41.393 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7ff672f27883] 2023-09-22T21:59:41.393 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7ff672f27a37] 2023-09-22T21:59:41.394 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x559da71b446c] 2023-09-22T21:59:41.394 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x559da7596910] 2023-09-22T21:59:41.394 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x559da71c056e] 2023-09-22T21:59:41.394 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x559da71c1e7b] 2023-09-22T21:59:41.394 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x559da71c2ced] 2023-09-22T21:59:41.394 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x559da75c9712] 2023-09-22T21:59:41.394 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 9: radosgw(+0x836db8) [0x559da7374db8] 2023-09-22T21:59:41.394 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x559da7375753] 2023-09-22T21:59:41.395 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff671bf6b43] 2023-09-22T21:59:41.395 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff671c88a00] 2023-09-22T21:59:41.395 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T21:59:41.395 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 0> 2023-09-22T21:59:41.364+0000 7ff64a379640 -1 *** Caught signal (Aborted) ** 2023-09-22T21:59:41.395 INFO:tasks.rgw.c1.client.1.smithi160.stdout: in thread 7ff64a379640 thread_name:data-sync 2023-09-22T21:59:41.395 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T21:59:41.395 INFO:tasks.rgw.c1.client.1.smithi160.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T21:59:41.395 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff671ba4520] 2023-09-22T21:59:41.395 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2: pthread_kill() 2023-09-22T21:59:41.396 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 3: raise() 2023-09-22T21:59:41.396 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 4: abort() 2023-09-22T21:59:41.396 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7ff672f278dd] 2023-09-22T21:59:41.396 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7ff672f27a37] 2023-09-22T21:59:41.396 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x559da71b446c] 2023-09-22T21:59:41.396 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x559da7596910] 2023-09-22T21:59:41.396 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x559da71c056e] 2023-09-22T21:59:41.396 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x559da71c1e7b] 2023-09-22T21:59:41.397 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x559da71c2ced] 2023-09-22T21:59:41.397 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x559da75c9712] 2023-09-22T21:59:41.397 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 13: radosgw(+0x836db8) [0x559da7374db8] 2023-09-22T21:59:41.397 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x559da7375753] 2023-09-22T21:59:41.397 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff671bf6b43] 2023-09-22T21:59:41.397 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff671c88a00] 2023-09-22T21:59:41.397 INFO:tasks.rgw.c1.client.1.smithi160.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-09-22T21:59:41.397 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T21:59:41.408 INFO:tasks.rgw.c1.client.1.smithi160.stderr:daemon-helper: command crashed with signal 11 2023-09-22T21:59:41.422 INFO:tasks.rgw.c1.client.1:Stopped 2023-09-22T21:59:41.422 DEBUG:boto:path=/ffzdqu-62/ 2023-09-22T21:59:41.423 DEBUG:boto:auth_path=/ffzdqu-62/ 2023-09-22T21:59:41.423 DEBUG:boto:Method: HEAD 2023-09-22T21:59:41.423 DEBUG:boto:Path: /ffzdqu-62/ 2023-09-22T21:59:41.423 DEBUG:boto:Data: 2023-09-22T21:59:41.423 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.423 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:41.423 DEBUG:boto:Port: 8000 2023-09-22T21:59:41.423 DEBUG:boto:Params: {} 2023-09-22T21:59:41.424 DEBUG:boto:Token: None 2023-09-22T21:59:41.424 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/ 2023-09-22T21:59:41.424 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI= 2023-09-22T21:59:41.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI=', 'Content-Length': '0'} 2023-09-22T21:59:41.428 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000180c83bddb08a93d-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.428 DEBUG:boto:path=/ffzdqu-62/obj11 2023-09-22T21:59:41.428 DEBUG:boto:auth_path=/ffzdqu-62/obj11 2023-09-22T21:59:41.428 DEBUG:boto:Method: PUT 2023-09-22T21:59:41.429 DEBUG:boto:Path: /ffzdqu-62/obj11 2023-09-22T21:59:41.429 DEBUG:boto:Data: 2023-09-22T21:59:41.429 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:59:41.429 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:41.429 DEBUG:boto:Port: 8000 2023-09-22T21:59:41.429 DEBUG:boto:Params: {} 2023-09-22T21:59:41.429 DEBUG:boto:Token: None 2023-09-22T21:59:41.429 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/obj11 2023-09-22T21:59:41.430 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4gLRkTN0hXrI7R6APS7/ZMe+U+A= 2023-09-22T21:59:41.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4gLRkTN0hXrI7R6APS7/ZMe+U+A='} 2023-09-22T21:59:41.478 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd6caa301c837722-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.479 DEBUG:boto:path=/ffzdqu-62/ 2023-09-22T21:59:41.479 DEBUG:boto:auth_path=/ffzdqu-62/ 2023-09-22T21:59:41.479 DEBUG:boto:Method: HEAD 2023-09-22T21:59:41.479 DEBUG:boto:Path: /ffzdqu-62/ 2023-09-22T21:59:41.479 DEBUG:boto:Data: 2023-09-22T21:59:41.480 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.480 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:41.480 DEBUG:boto:Port: 8000 2023-09-22T21:59:41.480 DEBUG:boto:Params: {} 2023-09-22T21:59:41.480 DEBUG:boto:Token: None 2023-09-22T21:59:41.480 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/ 2023-09-22T21:59:41.480 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI= 2023-09-22T21:59:41.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI=', 'Content-Length': '0'} 2023-09-22T21:59:41.484 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4c856d9265de288-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.484 DEBUG:boto:path=/ffzdqu-62/obj22 2023-09-22T21:59:41.484 DEBUG:boto:auth_path=/ffzdqu-62/obj22 2023-09-22T21:59:41.484 DEBUG:boto:Method: PUT 2023-09-22T21:59:41.485 DEBUG:boto:Path: /ffzdqu-62/obj22 2023-09-22T21:59:41.485 DEBUG:boto:Data: 2023-09-22T21:59:41.485 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:59:41.485 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:41.485 DEBUG:boto:Port: 8000 2023-09-22T21:59:41.485 DEBUG:boto:Params: {} 2023-09-22T21:59:41.485 DEBUG:boto:Token: None 2023-09-22T21:59:41.485 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/obj22 2023-09-22T21:59:41.486 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G1sCdLXauStwzYcFXcA69scYsU8= 2023-09-22T21:59:41.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G1sCdLXauStwzYcFXcA69scYsU8='} 2023-09-22T21:59:41.493 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa346b8ca634108a-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.493 DEBUG:boto:path=/ffzdqu-62/ 2023-09-22T21:59:41.494 DEBUG:boto:auth_path=/ffzdqu-62/ 2023-09-22T21:59:41.494 DEBUG:boto:Method: HEAD 2023-09-22T21:59:41.494 DEBUG:boto:Path: /ffzdqu-62/ 2023-09-22T21:59:41.494 DEBUG:boto:Data: 2023-09-22T21:59:41.494 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.494 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:41.494 DEBUG:boto:Port: 8000 2023-09-22T21:59:41.495 DEBUG:boto:Params: {} 2023-09-22T21:59:41.495 DEBUG:boto:Token: None 2023-09-22T21:59:41.495 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/ 2023-09-22T21:59:41.495 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI= 2023-09-22T21:59:41.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI=', 'Content-Length': '0'} 2023-09-22T21:59:41.499 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023576e4b74b72bdf-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.499 DEBUG:boto:path=/ffzdqu-62/obj33 2023-09-22T21:59:41.499 DEBUG:boto:auth_path=/ffzdqu-62/obj33 2023-09-22T21:59:41.499 DEBUG:boto:Method: PUT 2023-09-22T21:59:41.499 DEBUG:boto:Path: /ffzdqu-62/obj33 2023-09-22T21:59:41.499 DEBUG:boto:Data: 2023-09-22T21:59:41.500 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:59:41.500 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:41.500 DEBUG:boto:Port: 8000 2023-09-22T21:59:41.500 DEBUG:boto:Params: {} 2023-09-22T21:59:41.500 DEBUG:boto:Token: None 2023-09-22T21:59:41.500 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/obj33 2023-09-22T21:59:41.500 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5r7YCnwtALBZ8g6teOUDmzpKZSM= 2023-09-22T21:59:41.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5r7YCnwtALBZ8g6teOUDmzpKZSM='} 2023-09-22T21:59:41.542 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4878125c6453179-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.543 DEBUG:boto:path=/ffzdqu-62/ 2023-09-22T21:59:41.543 DEBUG:boto:auth_path=/ffzdqu-62/ 2023-09-22T21:59:41.543 DEBUG:boto:Method: HEAD 2023-09-22T21:59:41.543 DEBUG:boto:Path: /ffzdqu-62/ 2023-09-22T21:59:41.543 DEBUG:boto:Data: 2023-09-22T21:59:41.543 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.543 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:41.544 DEBUG:boto:Port: 8000 2023-09-22T21:59:41.544 DEBUG:boto:Params: {} 2023-09-22T21:59:41.544 DEBUG:boto:Token: None 2023-09-22T21:59:41.544 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/ 2023-09-22T21:59:41.544 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI= 2023-09-22T21:59:41.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI=', 'Content-Length': '0'} 2023-09-22T21:59:41.549 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fda0651de41f6af6-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.549 DEBUG:boto:path=/ffzdqu-62/obj44 2023-09-22T21:59:41.549 DEBUG:boto:auth_path=/ffzdqu-62/obj44 2023-09-22T21:59:41.550 DEBUG:boto:Method: PUT 2023-09-22T21:59:41.550 DEBUG:boto:Path: /ffzdqu-62/obj44 2023-09-22T21:59:41.550 DEBUG:boto:Data: 2023-09-22T21:59:41.550 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:59:41.550 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:41.550 DEBUG:boto:Port: 8000 2023-09-22T21:59:41.550 DEBUG:boto:Params: {} 2023-09-22T21:59:41.550 DEBUG:boto:Token: None 2023-09-22T21:59:41.551 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/obj44 2023-09-22T21:59:41.551 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8xoiE6mm+GiSZ9zpX4igRXA7LGE= 2023-09-22T21:59:41.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8xoiE6mm+GiSZ9zpX4igRXA7LGE='} 2023-09-22T21:59:41.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008538c06a8ce0727a-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.563 DEBUG:boto:path=/ffzdqu-62/ 2023-09-22T21:59:41.563 DEBUG:boto:auth_path=/ffzdqu-62/ 2023-09-22T21:59:41.563 DEBUG:boto:Method: HEAD 2023-09-22T21:59:41.563 DEBUG:boto:Path: /ffzdqu-62/ 2023-09-22T21:59:41.563 DEBUG:boto:Data: 2023-09-22T21:59:41.563 DEBUG:boto:Headers: {} 2023-09-22T21:59:41.564 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:41.564 DEBUG:boto:Port: 8000 2023-09-22T21:59:41.564 DEBUG:boto:Params: {} 2023-09-22T21:59:41.564 DEBUG:boto:Token: None 2023-09-22T21:59:41.564 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/ 2023-09-22T21:59:41.564 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI= 2023-09-22T21:59:41.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI=', 'Content-Length': '0'} 2023-09-22T21:59:41.568 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1df83734ae2dd87-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.568 DEBUG:boto:path=/ffzdqu-62/obj55 2023-09-22T21:59:41.568 DEBUG:boto:auth_path=/ffzdqu-62/obj55 2023-09-22T21:59:41.568 DEBUG:boto:Method: PUT 2023-09-22T21:59:41.569 DEBUG:boto:Path: /ffzdqu-62/obj55 2023-09-22T21:59:41.569 DEBUG:boto:Data: 2023-09-22T21:59:41.569 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T21:59:41.569 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T21:59:41.569 DEBUG:boto:Port: 8000 2023-09-22T21:59:41.569 DEBUG:boto:Params: {} 2023-09-22T21:59:41.569 DEBUG:boto:Token: None 2023-09-22T21:59:41.570 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 21:59:41 GMT /ffzdqu-62/obj55 2023-09-22T21:59:41.570 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZXktj0CnX8k04WHPiK/Rm8H+I2c= 2023-09-22T21:59:41.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZXktj0CnX8k04WHPiK/Rm8H+I2c='} 2023-09-22T21:59:41.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ffa4b8fb304a038f-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T21:59:41.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:59:41.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T21:59:41.615 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T21:59:41.659 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T21:59:41.659 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T21:59:41.705 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:59:41.704+0000 7f4898cafa40 5 RGW-SYNC:data: waiting on data sync status lease 2023-09-22T21:59:41.706 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T21:59:41.704+0000 7f4898cafa40 5 RGW-SYNC:data: acquired data sync status lease 2023-09-22T21:59:41.732 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-22T21:59:41.733 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-22T21:59:41.734 INFO:tasks.rgw.c1.client.1:Started 2023-09-22T21:59:41.735 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:41.743 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:59:41.743 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:59:41.743 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T21:59:41.744 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T21:59:41.744 INFO:teuthology.orchestra.run.smithi160.stderr:curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T21:59:42.745 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:42.922 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:59:42.922 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:59:42.922 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T21:59:42.923 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T21:59:42.923 INFO:teuthology.orchestra.run.smithi160.stderr:curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T21:59:44.924 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:44.935 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:59:44.936 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:59:44.937 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T21:59:44.937 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T21:59:44.937 INFO:teuthology.orchestra.run.smithi160.stderr:curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T21:59:48.938 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:48.951 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:59:48.951 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:59:48.951 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T21:59:48.952 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T21:59:48.952 INFO:teuthology.orchestra.run.smithi160.stderr:curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T21:59:56.953 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T21:59:56.965 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T21:59:56.965 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T21:59:56.966 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T21:59:56.966 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T21:59:56.967 INFO:teuthology.orchestra.run.smithi160.stderr:curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T22:00:12.968 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:00:12.979 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:00:12.979 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T22:00:12.980 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 136k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-22T22:00:12.981 INFO:teuthology.orchestra.run.smithi160.stdout:anonymous 2023-09-22T22:00:12.981 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-62 2023-09-22T22:00:12.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:00:12.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:00:12.982 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T22:00:13.070 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:00:13.070 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:00:13.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:00:13.111+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:00:13.114 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:00:13.114 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:00:14.151 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T22:00:14.147+0000 7f7bd38e3640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T22:00:18.116 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:00:18.115+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:00:18.116 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:00:18.116 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:00:23.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:00:23.115+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:00:23.118 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:00:23.119 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:00:28.121 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:00:28.119+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:00:28.121 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:00:28.121 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:00:33.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:00:33.123+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:00:33.123 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:00:33.123 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:00:38.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:00:38.123+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:00:38.125 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:00:38.126 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:00:43.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:00:43.127+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:00:43.128 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:00:43.129 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:00:48.130 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:00:48.127+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:00:48.131 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:00:48.131 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:00:53.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:00:53.135+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:00:53.136 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:00:53.136 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:00:58.138 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:00:58.135+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:00:58.500 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:00:58.501 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:01:03.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:01:03.138+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:01:03.141 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:01:03.141 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:01:08.143 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:01:08.142+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:01:08.143 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:01:08.143 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:01:13.145 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:01:13.142+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:01:13.146 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:01:13.146 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:01:18.148 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:01:18.146+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:01:18.148 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:01:18.148 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:01:23.150 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:01:23.146+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:01:23.150 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:01:23.150 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:01:28.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:01:28.150+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:01:28.153 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:01:28.153 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:01:33.156 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:01:33.154+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:01:33.156 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:01:33.156 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:01:38.158 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:01:38.154+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:01:38.158 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:01:38.158 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:01:43.160 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:01:43.158+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:01:43.160 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:01:43.161 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:01:48.163 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:01:48.162+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:01:48.163 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:01:48.163 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:01:53.165 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:01:53.162+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:01:53.165 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:01:53.165 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:01:58.168 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:01:58.166+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:01:58.168 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:01:58.169 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:02:03.170 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:02:03.166+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:02:03.170 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:02:03.170 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:02:08.173 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:02:08.170+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:02:08.173 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:02:08.173 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:02:13.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:02:13.173+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:02:13.176 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:02:13.176 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:02:18.178 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:02:18.173+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:02:18.179 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:02:18.179 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:02:23.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:02:23.181+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:02:23.184 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:02:23.184 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:02:28.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:02:28.185+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:02:28.187 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:02:28.187 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:02:33.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:02:33.189+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:02:33.190 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:02:33.190 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:02:38.193 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:02:38.193+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:02:38.194 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:02:38.194 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:02:43.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:02:43.197+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:02:43.196 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:02:43.197 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:02:48.199 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:02:48.197+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:02:48.199 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:02:48.199 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:02:53.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:02:53.201+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:02:53.201 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:02:53.202 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:02:58.203 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:02:58.205+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:02:58.203 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:02:58.204 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:03.206 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:03:03.205+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:03:03.206 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:03.206 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:08.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:03:08.209+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:03:08.209 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:08.209 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:13.210 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:03:13.209+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:03:13.211 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:13.211 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:18.212 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:03:18.213+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:03:18.213 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:18.213 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:23.215 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:03:23.217+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:03:23.215 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:23.215 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:28.218 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:03:28.216+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:03:28.218 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:28.218 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:33.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:03:33.220+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:03:33.220 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:33.220 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:38.222 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:03:38.220+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:03:38.222 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:38.222 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:43.224 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:03:43.224+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:03:43.224 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:43.225 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:48.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:03:48.228+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:03:48.227 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:48.227 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:53.229 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:03:53.228+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:03:53.230 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:53.230 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:58.232 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:03:58.232+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:03:58.232 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:03:58.232 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:03.234 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:04:03.232+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:04:03.234 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:03.234 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:08.237 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:04:08.236+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:04:08.237 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:08.237 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:13.239 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:04:13.240+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:04:13.239 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:13.239 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:18.242 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:04:18.240+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:04:18.242 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:18.243 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:23.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:04:23.244+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:04:23.244 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:23.244 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:28.246 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:04:28.244+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:04:28.247 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:28.247 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:33.250 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:04:33.248+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:04:33.250 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:33.250 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:38.251 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:04:38.251+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:04:38.251 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:38.251 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:43.253 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:04:43.251+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:04:43.253 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:43.253 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:48.256 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:04:48.255+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:04:48.256 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:48.256 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:53.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:04:53.255+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:04:53.258 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:53.258 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:58.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:04:58.259+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:04:58.260 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:04:58.260 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:05:03.262 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:05:03.259+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:05:03.263 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:05:03.263 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:05:08.265 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:05:08.263+0000 7fc33889ba40 0 bucket checkpoint timed out waiting for incremental sync to catch up 2023-09-22T22:05:08.266 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:05:08.263+0000 7fc33889ba40 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-22T22:05:08.266 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:05:08.263+0000 7fc33889ba40 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-22T22:05:08.303 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-09-22T22:05:08.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... FAIL 2023-09-22T22:05:08.312 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:08.312 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:08.312 DEBUG:boto:Method: PUT 2023-09-22T22:05:08.312 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:08.312 DEBUG:boto:Data: 2023-09-22T22:05:08.312 DEBUG:boto:Headers: {} 2023-09-22T22:05:08.312 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:08.313 DEBUG:boto:Port: 8000 2023-09-22T22:05:08.313 DEBUG:boto:Params: {} 2023-09-22T22:05:08.313 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T22:05:08.313 DEBUG:boto:Token: None 2023-09-22T22:05:08.313 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 22:05:08 GMT /ffzdqu-63/ 2023-09-22T22:05:08.314 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0wh2zKc2uZo0OGrGQmNMw9Iw/GY= 2023-09-22T22:05:08.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0wh2zKc2uZo0OGrGQmNMw9Iw/GY=', 'Content-Length': '0'} 2023-09-22T22:05:08.361 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000000f82e8cc78f9cc1-00650e0f94-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:08.361 DEBUG:rgw_multi.tests:created bucket=ffzdqu-63 2023-09-22T22:05:08.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:05:08.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:05:08.362 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:05:08.405 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:05:08.406 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:05:08.446 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T22:05:08.469 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"1_1695419384.000133_332.1","last_update":"2023-09-22T21:49:44.000133Z"},{"marker":"1_1695420308.359160_609.1","last_update":"2023-09-22T22:05:08.359160Z"},{"marker":"1_1695419383.971498_240.1","last_update":"2023-09-22T21:49:43.971498Z"}] 2023-09-22T22:05:08.469 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420308.359160_609.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:05:08.469 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T22:05:08.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:05:08.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:05:08.470 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:05:08.549 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:05:08.549 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:05:08.585 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:05:08.583+0000 7fdebc370a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:05:08.586 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:05:08.583+0000 7fdebc370a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:05:08.603 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420308.344032_606.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:05:08.344032Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:05:08Z"} 2023-09-22T22:05:08.604 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420308.344032_606.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:05:08.344032Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:05:08Z"} 2023-09-22T22:05:08.604 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420308.344032_606.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:05:08.344032Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:05:08.604 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420308.359160_609.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:05:08.604 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420308.344032_606.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:05:08.604 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695420308.359160_609.1 target=1_1695420308.344032_606.1 2023-09-22T22:05:13.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:05:13.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:05:13.606 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:05:13.650 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:05:13.650 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:05:13.690 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:05:13.687+0000 7f4c4f92ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:05:13.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:05:13.691+0000 7f4c4f92ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:05:13.709 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420308.359160_609.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:05:08.359160Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:05:13Z"} 2023-09-22T22:05:13.709 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420308.359160_609.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:05:08.359160Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:05:13Z"} 2023-09-22T22:05:13.710 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420308.359160_609.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:05:08.359160Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:05:13.710 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420308.359160_609.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:05:13.710 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420308.359160_609.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:05:13.710 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T22:05:13.710 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:13.710 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:13.711 DEBUG:boto:Method: HEAD 2023-09-22T22:05:13.711 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:13.711 DEBUG:boto:Data: 2023-09-22T22:05:13.711 DEBUG:boto:Headers: {} 2023-09-22T22:05:13.711 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:13.711 DEBUG:boto:Port: 8000 2023-09-22T22:05:13.711 DEBUG:boto:Params: {} 2023-09-22T22:05:13.711 DEBUG:boto:Token: None 2023-09-22T22:05:13.712 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:13 GMT /ffzdqu-63/ 2023-09-22T22:05:13.712 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y8MimPEpPTm4vni54PayPyzDfeA= 2023-09-22T22:05:13.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y8MimPEpPTm4vni54PayPyzDfeA=', 'Content-Length': '0'} 2023-09-22T22:05:13.714 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006432f24b88b85b58-00650e0f99-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:13.715 DEBUG:boto:path=/ffzdqu-63/obj11 2023-09-22T22:05:13.715 DEBUG:boto:auth_path=/ffzdqu-63/obj11 2023-09-22T22:05:13.715 DEBUG:boto:Method: PUT 2023-09-22T22:05:13.715 DEBUG:boto:Path: /ffzdqu-63/obj11 2023-09-22T22:05:13.715 DEBUG:boto:Data: 2023-09-22T22:05:13.715 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:13.715 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:13.715 DEBUG:boto:Port: 8000 2023-09-22T22:05:13.715 DEBUG:boto:Params: {} 2023-09-22T22:05:13.715 DEBUG:boto:Token: None 2023-09-22T22:05:13.716 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:13 GMT /ffzdqu-63/obj11 2023-09-22T22:05:13.716 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Mqp0cA7RwXh1w24OznCpY4mbNiw= 2023-09-22T22:05:13.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Mqp0cA7RwXh1w24OznCpY4mbNiw='} 2023-09-22T22:05:13.723 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002d9325dd78e9fc92-00650e0f99-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:13.723 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:13.723 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:13.723 DEBUG:boto:Method: HEAD 2023-09-22T22:05:13.723 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:13.723 DEBUG:boto:Data: 2023-09-22T22:05:13.724 DEBUG:boto:Headers: {} 2023-09-22T22:05:13.724 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:13.724 DEBUG:boto:Port: 8000 2023-09-22T22:05:13.724 DEBUG:boto:Params: {} 2023-09-22T22:05:13.724 DEBUG:boto:Token: None 2023-09-22T22:05:13.724 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:13 GMT /ffzdqu-63/ 2023-09-22T22:05:13.724 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y8MimPEpPTm4vni54PayPyzDfeA= 2023-09-22T22:05:13.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y8MimPEpPTm4vni54PayPyzDfeA=', 'Content-Length': '0'} 2023-09-22T22:05:13.727 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2740b101737ba92-00650e0f99-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:13.727 DEBUG:boto:path=/ffzdqu-63/obj22 2023-09-22T22:05:13.727 DEBUG:boto:auth_path=/ffzdqu-63/obj22 2023-09-22T22:05:13.727 DEBUG:boto:Method: PUT 2023-09-22T22:05:13.727 DEBUG:boto:Path: /ffzdqu-63/obj22 2023-09-22T22:05:13.727 DEBUG:boto:Data: 2023-09-22T22:05:13.727 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:13.728 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:13.728 DEBUG:boto:Port: 8000 2023-09-22T22:05:13.728 DEBUG:boto:Params: {} 2023-09-22T22:05:13.728 DEBUG:boto:Token: None 2023-09-22T22:05:13.728 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:13 GMT /ffzdqu-63/obj22 2023-09-22T22:05:13.728 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4PO+0BWdlhmkonqM/RkKBEcJeYo= 2023-09-22T22:05:13.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4PO+0BWdlhmkonqM/RkKBEcJeYo='} 2023-09-22T22:05:13.734 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000065fa785ca600546-00650e0f99-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:13.734 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:13.735 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:13.735 DEBUG:boto:Method: HEAD 2023-09-22T22:05:13.735 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:13.735 DEBUG:boto:Data: 2023-09-22T22:05:13.735 DEBUG:boto:Headers: {} 2023-09-22T22:05:13.735 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:13.735 DEBUG:boto:Port: 8000 2023-09-22T22:05:13.735 DEBUG:boto:Params: {} 2023-09-22T22:05:13.735 DEBUG:boto:Token: None 2023-09-22T22:05:13.735 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:13 GMT /ffzdqu-63/ 2023-09-22T22:05:13.736 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y8MimPEpPTm4vni54PayPyzDfeA= 2023-09-22T22:05:13.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y8MimPEpPTm4vni54PayPyzDfeA=', 'Content-Length': '0'} 2023-09-22T22:05:13.738 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000917309ca6945d4c5-00650e0f99-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:13.738 DEBUG:boto:path=/ffzdqu-63/obj33 2023-09-22T22:05:13.738 DEBUG:boto:auth_path=/ffzdqu-63/obj33 2023-09-22T22:05:13.738 DEBUG:boto:Method: PUT 2023-09-22T22:05:13.738 DEBUG:boto:Path: /ffzdqu-63/obj33 2023-09-22T22:05:13.738 DEBUG:boto:Data: 2023-09-22T22:05:13.739 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:13.739 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:13.739 DEBUG:boto:Port: 8000 2023-09-22T22:05:13.739 DEBUG:boto:Params: {} 2023-09-22T22:05:13.739 DEBUG:boto:Token: None 2023-09-22T22:05:13.739 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:13 GMT /ffzdqu-63/obj33 2023-09-22T22:05:13.739 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:g8aZfu6MYIHYbI6jD1NpzQvPNM4= 2023-09-22T22:05:13.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8aZfu6MYIHYbI6jD1NpzQvPNM4='} 2023-09-22T22:05:13.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025e8c2ef910991e9-00650e0f99-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:13.783 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:13.783 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:13.783 DEBUG:boto:Method: HEAD 2023-09-22T22:05:13.783 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:13.783 DEBUG:boto:Data: 2023-09-22T22:05:13.783 DEBUG:boto:Headers: {} 2023-09-22T22:05:13.783 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:13.783 DEBUG:boto:Port: 8000 2023-09-22T22:05:13.783 DEBUG:boto:Params: {} 2023-09-22T22:05:13.784 DEBUG:boto:Token: None 2023-09-22T22:05:13.784 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:13 GMT /ffzdqu-63/ 2023-09-22T22:05:13.784 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y8MimPEpPTm4vni54PayPyzDfeA= 2023-09-22T22:05:13.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y8MimPEpPTm4vni54PayPyzDfeA=', 'Content-Length': '0'} 2023-09-22T22:05:13.787 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001354c5e777c700e5-00650e0f99-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:13.787 DEBUG:boto:path=/ffzdqu-63/obj44 2023-09-22T22:05:13.787 DEBUG:boto:auth_path=/ffzdqu-63/obj44 2023-09-22T22:05:13.787 DEBUG:boto:Method: PUT 2023-09-22T22:05:13.788 DEBUG:boto:Path: /ffzdqu-63/obj44 2023-09-22T22:05:13.788 DEBUG:boto:Data: 2023-09-22T22:05:13.788 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:13.788 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:13.788 DEBUG:boto:Port: 8000 2023-09-22T22:05:13.788 DEBUG:boto:Params: {} 2023-09-22T22:05:13.788 DEBUG:boto:Token: None 2023-09-22T22:05:13.788 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:13 GMT /ffzdqu-63/obj44 2023-09-22T22:05:13.788 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jQLzJCUjG0g15NpdomH6h+WjAp4= 2023-09-22T22:05:13.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jQLzJCUjG0g15NpdomH6h+WjAp4='} 2023-09-22T22:05:13.801 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f778c8ffd7c8e9fa-00650e0f99-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:13.801 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:13.801 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:13.801 DEBUG:boto:Method: HEAD 2023-09-22T22:05:13.801 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:13.801 DEBUG:boto:Data: 2023-09-22T22:05:13.802 DEBUG:boto:Headers: {} 2023-09-22T22:05:13.802 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:13.802 DEBUG:boto:Port: 8000 2023-09-22T22:05:13.802 DEBUG:boto:Params: {} 2023-09-22T22:05:13.802 DEBUG:boto:Token: None 2023-09-22T22:05:13.802 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:13 GMT /ffzdqu-63/ 2023-09-22T22:05:13.802 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Y8MimPEpPTm4vni54PayPyzDfeA= 2023-09-22T22:05:13.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y8MimPEpPTm4vni54PayPyzDfeA=', 'Content-Length': '0'} 2023-09-22T22:05:13.806 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ab6e84cd1beea4f-00650e0f99-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:13.807 DEBUG:boto:path=/ffzdqu-63/obj55 2023-09-22T22:05:13.807 DEBUG:boto:auth_path=/ffzdqu-63/obj55 2023-09-22T22:05:13.807 DEBUG:boto:Method: PUT 2023-09-22T22:05:13.807 DEBUG:boto:Path: /ffzdqu-63/obj55 2023-09-22T22:05:13.807 DEBUG:boto:Data: 2023-09-22T22:05:13.807 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:13.807 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:13.807 DEBUG:boto:Port: 8000 2023-09-22T22:05:13.807 DEBUG:boto:Params: {} 2023-09-22T22:05:13.808 DEBUG:boto:Token: None 2023-09-22T22:05:13.808 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:13 GMT /ffzdqu-63/obj55 2023-09-22T22:05:13.808 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yFPyanA5+4dDAjR2a3uYRWqFUUc= 2023-09-22T22:05:13.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yFPyanA5+4dDAjR2a3uYRWqFUUc='} 2023-09-22T22:05:13.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000672ef00ed86a81c5-00650e0f99-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:13.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:05:13.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:05:13.851 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:05:13.903 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:05:14.028 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:05:14.030 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T22:05:14.030 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-63 2023-09-22T22:05:14.030 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 5 2023-09-22T22:05:14.031 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:14.031 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:14.031 DEBUG:boto:Method: HEAD 2023-09-22T22:05:14.031 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:14.031 DEBUG:boto:Data: 2023-09-22T22:05:14.032 DEBUG:boto:Headers: {} 2023-09-22T22:05:14.032 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.032 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.032 DEBUG:boto:Params: {} 2023-09-22T22:05:14.032 DEBUG:boto:Token: None 2023-09-22T22:05:14.032 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/ 2023-09-22T22:05:14.032 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:05:14.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:05:14.035 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000865f3d0993c5d69e-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.036 DEBUG:boto:path=/ffzdqu-63/obj17 2023-09-22T22:05:14.036 DEBUG:boto:auth_path=/ffzdqu-63/obj17 2023-09-22T22:05:14.036 DEBUG:boto:Method: PUT 2023-09-22T22:05:14.036 DEBUG:boto:Path: /ffzdqu-63/obj17 2023-09-22T22:05:14.036 DEBUG:boto:Data: 2023-09-22T22:05:14.036 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:14.036 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.036 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.037 DEBUG:boto:Params: {} 2023-09-22T22:05:14.037 DEBUG:boto:Token: None 2023-09-22T22:05:14.037 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/obj17 2023-09-22T22:05:14.037 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lXNddYeGDSb63G5cVg4qzyLAF70= 2023-09-22T22:05:14.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lXNddYeGDSb63G5cVg4qzyLAF70='} 2023-09-22T22:05:14.043 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c43503f5585cdd90-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.043 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:14.043 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:14.044 DEBUG:boto:Method: HEAD 2023-09-22T22:05:14.044 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:14.044 DEBUG:boto:Data: 2023-09-22T22:05:14.044 DEBUG:boto:Headers: {} 2023-09-22T22:05:14.044 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.044 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.044 DEBUG:boto:Params: {} 2023-09-22T22:05:14.044 DEBUG:boto:Token: None 2023-09-22T22:05:14.045 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/ 2023-09-22T22:05:14.045 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:05:14.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:05:14.048 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4558d2007f58520-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.048 DEBUG:boto:path=/ffzdqu-63/obj34 2023-09-22T22:05:14.048 DEBUG:boto:auth_path=/ffzdqu-63/obj34 2023-09-22T22:05:14.048 DEBUG:boto:Method: PUT 2023-09-22T22:05:14.048 DEBUG:boto:Path: /ffzdqu-63/obj34 2023-09-22T22:05:14.048 DEBUG:boto:Data: 2023-09-22T22:05:14.048 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:14.048 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.049 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.049 DEBUG:boto:Params: {} 2023-09-22T22:05:14.049 DEBUG:boto:Token: None 2023-09-22T22:05:14.049 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/obj34 2023-09-22T22:05:14.049 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AG/R2Wjh/SaEsZLn6PywH6ODNt0= 2023-09-22T22:05:14.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AG/R2Wjh/SaEsZLn6PywH6ODNt0='} 2023-09-22T22:05:14.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ea01f3a2244014fe-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.095 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:14.095 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:14.095 DEBUG:boto:Method: HEAD 2023-09-22T22:05:14.095 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:14.095 DEBUG:boto:Data: 2023-09-22T22:05:14.095 DEBUG:boto:Headers: {} 2023-09-22T22:05:14.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.096 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.096 DEBUG:boto:Params: {} 2023-09-22T22:05:14.096 DEBUG:boto:Token: None 2023-09-22T22:05:14.096 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/ 2023-09-22T22:05:14.096 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:05:14.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:05:14.099 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f49f091017a70b7-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.099 DEBUG:boto:path=/ffzdqu-63/obj51 2023-09-22T22:05:14.100 DEBUG:boto:auth_path=/ffzdqu-63/obj51 2023-09-22T22:05:14.100 DEBUG:boto:Method: PUT 2023-09-22T22:05:14.100 DEBUG:boto:Path: /ffzdqu-63/obj51 2023-09-22T22:05:14.100 DEBUG:boto:Data: 2023-09-22T22:05:14.100 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:14.100 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.100 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.100 DEBUG:boto:Params: {} 2023-09-22T22:05:14.100 DEBUG:boto:Token: None 2023-09-22T22:05:14.101 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/obj51 2023-09-22T22:05:14.101 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IH50w5lC5jH2AgFH0DMPn3+e+yM= 2023-09-22T22:05:14.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IH50w5lC5jH2AgFH0DMPn3+e+yM='} 2023-09-22T22:05:14.108 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e5e51708af1c50f-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.108 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:14.108 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:14.108 DEBUG:boto:Method: HEAD 2023-09-22T22:05:14.108 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:14.108 DEBUG:boto:Data: 2023-09-22T22:05:14.108 DEBUG:boto:Headers: {} 2023-09-22T22:05:14.108 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.108 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.109 DEBUG:boto:Params: {} 2023-09-22T22:05:14.109 DEBUG:boto:Token: None 2023-09-22T22:05:14.109 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/ 2023-09-22T22:05:14.109 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:05:14.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:05:14.112 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d7af7dab2627f42-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.112 DEBUG:boto:path=/ffzdqu-63/obj68 2023-09-22T22:05:14.112 DEBUG:boto:auth_path=/ffzdqu-63/obj68 2023-09-22T22:05:14.112 DEBUG:boto:Method: PUT 2023-09-22T22:05:14.112 DEBUG:boto:Path: /ffzdqu-63/obj68 2023-09-22T22:05:14.112 DEBUG:boto:Data: 2023-09-22T22:05:14.112 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:14.113 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.113 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.113 DEBUG:boto:Params: {} 2023-09-22T22:05:14.113 DEBUG:boto:Token: None 2023-09-22T22:05:14.113 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/obj68 2023-09-22T22:05:14.113 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IXu8VRNR+NJWTbIl0pMZf/XpQGk= 2023-09-22T22:05:14.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IXu8VRNR+NJWTbIl0pMZf/XpQGk='} 2023-09-22T22:05:14.162 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c1667f795b06156-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.163 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:14.163 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:14.163 DEBUG:boto:Method: HEAD 2023-09-22T22:05:14.163 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:14.163 DEBUG:boto:Data: 2023-09-22T22:05:14.163 DEBUG:boto:Headers: {} 2023-09-22T22:05:14.163 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.163 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.163 DEBUG:boto:Params: {} 2023-09-22T22:05:14.164 DEBUG:boto:Token: None 2023-09-22T22:05:14.164 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/ 2023-09-22T22:05:14.164 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:05:14.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:05:14.169 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b597f27b3f0411a8-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.169 DEBUG:boto:path=/ffzdqu-63/obj85 2023-09-22T22:05:14.170 DEBUG:boto:auth_path=/ffzdqu-63/obj85 2023-09-22T22:05:14.170 DEBUG:boto:Method: PUT 2023-09-22T22:05:14.170 DEBUG:boto:Path: /ffzdqu-63/obj85 2023-09-22T22:05:14.170 DEBUG:boto:Data: 2023-09-22T22:05:14.170 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:14.170 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.170 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.170 DEBUG:boto:Params: {} 2023-09-22T22:05:14.171 DEBUG:boto:Token: None 2023-09-22T22:05:14.171 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/obj85 2023-09-22T22:05:14.171 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:s9touLmomUYRfyAKA8YWGv1Jrx4= 2023-09-22T22:05:14.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:s9touLmomUYRfyAKA8YWGv1Jrx4='} 2023-09-22T22:05:14.184 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033af09fa02b48893-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.184 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:05:14.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:05:14.185 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:05:14.231 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:05:14.231 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:05:14.292 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T22:05:14.292 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-63 2023-09-22T22:05:14.302 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 10 2023-09-22T22:05:14.359 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:14.360 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:14.360 DEBUG:boto:Method: HEAD 2023-09-22T22:05:14.360 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:14.360 DEBUG:boto:Data: 2023-09-22T22:05:14.360 DEBUG:boto:Headers: {} 2023-09-22T22:05:14.360 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.360 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.360 DEBUG:boto:Params: {} 2023-09-22T22:05:14.361 DEBUG:boto:Token: None 2023-09-22T22:05:14.361 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/ 2023-09-22T22:05:14.361 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:05:14.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:05:14.364 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013eff33f8cd624f4-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.364 DEBUG:boto:path=/ffzdqu-63/obj19 2023-09-22T22:05:14.365 DEBUG:boto:auth_path=/ffzdqu-63/obj19 2023-09-22T22:05:14.365 DEBUG:boto:Method: PUT 2023-09-22T22:05:14.365 DEBUG:boto:Path: /ffzdqu-63/obj19 2023-09-22T22:05:14.365 DEBUG:boto:Data: 2023-09-22T22:05:14.365 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:14.365 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.365 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.365 DEBUG:boto:Params: {} 2023-09-22T22:05:14.365 DEBUG:boto:Token: None 2023-09-22T22:05:14.366 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/obj19 2023-09-22T22:05:14.366 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:c0T76fAEzGZH1VwiIdFiU7YnOhg= 2023-09-22T22:05:14.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:c0T76fAEzGZH1VwiIdFiU7YnOhg='} 2023-09-22T22:05:14.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9994d9cf90a8189-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.415 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:14.415 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:14.415 DEBUG:boto:Method: HEAD 2023-09-22T22:05:14.415 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:14.415 DEBUG:boto:Data: 2023-09-22T22:05:14.415 DEBUG:boto:Headers: {} 2023-09-22T22:05:14.416 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.416 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.416 DEBUG:boto:Params: {} 2023-09-22T22:05:14.416 DEBUG:boto:Token: None 2023-09-22T22:05:14.416 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/ 2023-09-22T22:05:14.416 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:05:14.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:05:14.421 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006281ecd2f5bfd9a1-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.422 DEBUG:boto:path=/ffzdqu-63/obj38 2023-09-22T22:05:14.422 DEBUG:boto:auth_path=/ffzdqu-63/obj38 2023-09-22T22:05:14.422 DEBUG:boto:Method: PUT 2023-09-22T22:05:14.422 DEBUG:boto:Path: /ffzdqu-63/obj38 2023-09-22T22:05:14.422 DEBUG:boto:Data: 2023-09-22T22:05:14.422 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:14.422 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.422 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.423 DEBUG:boto:Params: {} 2023-09-22T22:05:14.423 DEBUG:boto:Token: None 2023-09-22T22:05:14.423 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/obj38 2023-09-22T22:05:14.423 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HwUSNQqnTDqfjat88BxMKEZY5zc= 2023-09-22T22:05:14.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HwUSNQqnTDqfjat88BxMKEZY5zc='} 2023-09-22T22:05:14.431 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca9537e3eabf3431-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.431 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:14.432 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:14.432 DEBUG:boto:Method: HEAD 2023-09-22T22:05:14.432 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:14.432 DEBUG:boto:Data: 2023-09-22T22:05:14.432 DEBUG:boto:Headers: {} 2023-09-22T22:05:14.432 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.432 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.432 DEBUG:boto:Params: {} 2023-09-22T22:05:14.432 DEBUG:boto:Token: None 2023-09-22T22:05:14.432 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/ 2023-09-22T22:05:14.433 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:05:14.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:05:14.438 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ed655c3d7d8da68-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.438 DEBUG:boto:path=/ffzdqu-63/obj57 2023-09-22T22:05:14.438 DEBUG:boto:auth_path=/ffzdqu-63/obj57 2023-09-22T22:05:14.438 DEBUG:boto:Method: PUT 2023-09-22T22:05:14.438 DEBUG:boto:Path: /ffzdqu-63/obj57 2023-09-22T22:05:14.438 DEBUG:boto:Data: 2023-09-22T22:05:14.439 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:14.439 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.439 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.439 DEBUG:boto:Params: {} 2023-09-22T22:05:14.439 DEBUG:boto:Token: None 2023-09-22T22:05:14.439 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/obj57 2023-09-22T22:05:14.439 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:stdWr3aP9yy8Ke87dKHHrTL7Q8U= 2023-09-22T22:05:14.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:stdWr3aP9yy8Ke87dKHHrTL7Q8U='} 2023-09-22T22:05:14.486 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2830b6b48bc65ef-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.487 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:14.487 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:14.487 DEBUG:boto:Method: HEAD 2023-09-22T22:05:14.487 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:14.487 DEBUG:boto:Data: 2023-09-22T22:05:14.487 DEBUG:boto:Headers: {} 2023-09-22T22:05:14.487 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.487 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.487 DEBUG:boto:Params: {} 2023-09-22T22:05:14.488 DEBUG:boto:Token: None 2023-09-22T22:05:14.488 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/ 2023-09-22T22:05:14.488 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:05:14.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:05:14.493 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e049e49825599422-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.493 DEBUG:boto:path=/ffzdqu-63/obj76 2023-09-22T22:05:14.494 DEBUG:boto:auth_path=/ffzdqu-63/obj76 2023-09-22T22:05:14.494 DEBUG:boto:Method: PUT 2023-09-22T22:05:14.494 DEBUG:boto:Path: /ffzdqu-63/obj76 2023-09-22T22:05:14.494 DEBUG:boto:Data: 2023-09-22T22:05:14.494 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:14.494 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.494 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.494 DEBUG:boto:Params: {} 2023-09-22T22:05:14.494 DEBUG:boto:Token: None 2023-09-22T22:05:14.495 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/obj76 2023-09-22T22:05:14.495 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:llzLD2d0/sGxl1olFiWU4THw58M= 2023-09-22T22:05:14.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:llzLD2d0/sGxl1olFiWU4THw58M='} 2023-09-22T22:05:14.508 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aa16129cf5ea7409-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.509 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:14.509 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:14.509 DEBUG:boto:Method: HEAD 2023-09-22T22:05:14.509 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:14.509 DEBUG:boto:Data: 2023-09-22T22:05:14.509 DEBUG:boto:Headers: {} 2023-09-22T22:05:14.509 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.509 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.509 DEBUG:boto:Params: {} 2023-09-22T22:05:14.510 DEBUG:boto:Token: None 2023-09-22T22:05:14.510 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/ 2023-09-22T22:05:14.510 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:05:14.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:05:14.515 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3571597473b3761-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.515 DEBUG:boto:path=/ffzdqu-63/obj95 2023-09-22T22:05:14.515 DEBUG:boto:auth_path=/ffzdqu-63/obj95 2023-09-22T22:05:14.516 DEBUG:boto:Method: PUT 2023-09-22T22:05:14.516 DEBUG:boto:Path: /ffzdqu-63/obj95 2023-09-22T22:05:14.516 DEBUG:boto:Data: 2023-09-22T22:05:14.516 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:14.516 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.516 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.516 DEBUG:boto:Params: {} 2023-09-22T22:05:14.516 DEBUG:boto:Token: None 2023-09-22T22:05:14.517 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/obj95 2023-09-22T22:05:14.517 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VYXVWdh4OESFZcQStNAho8zZ0kQ= 2023-09-22T22:05:14.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VYXVWdh4OESFZcQStNAho8zZ0kQ='} 2023-09-22T22:05:14.558 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000080ffc7252cd9cf8e-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:05:14.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:05:14.559 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:05:14.605 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:05:14.605 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:05:14.666 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T22:05:14.667 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-63 2023-09-22T22:05:14.678 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 15 2023-09-22T22:05:14.742 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:14.742 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:14.742 DEBUG:boto:Method: HEAD 2023-09-22T22:05:14.742 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:14.742 DEBUG:boto:Data: 2023-09-22T22:05:14.742 DEBUG:boto:Headers: {} 2023-09-22T22:05:14.743 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.743 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.743 DEBUG:boto:Params: {} 2023-09-22T22:05:14.743 DEBUG:boto:Token: None 2023-09-22T22:05:14.743 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/ 2023-09-22T22:05:14.743 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:05:14.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:05:14.747 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006aa1ce981f7f358e-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.747 DEBUG:boto:path=/ffzdqu-63/obj23 2023-09-22T22:05:14.747 DEBUG:boto:auth_path=/ffzdqu-63/obj23 2023-09-22T22:05:14.747 DEBUG:boto:Method: PUT 2023-09-22T22:05:14.747 DEBUG:boto:Path: /ffzdqu-63/obj23 2023-09-22T22:05:14.748 DEBUG:boto:Data: 2023-09-22T22:05:14.748 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:14.748 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.748 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.748 DEBUG:boto:Params: {} 2023-09-22T22:05:14.748 DEBUG:boto:Token: None 2023-09-22T22:05:14.748 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/obj23 2023-09-22T22:05:14.748 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3tGrlYdc+EP0GZQWRw6nfS5t27w= 2023-09-22T22:05:14.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3tGrlYdc+EP0GZQWRw6nfS5t27w='} 2023-09-22T22:05:14.755 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000004f9c3ae3f54e10c-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.755 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:14.755 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:14.755 DEBUG:boto:Method: HEAD 2023-09-22T22:05:14.755 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:14.755 DEBUG:boto:Data: 2023-09-22T22:05:14.756 DEBUG:boto:Headers: {} 2023-09-22T22:05:14.756 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.756 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.756 DEBUG:boto:Params: {} 2023-09-22T22:05:14.756 DEBUG:boto:Token: None 2023-09-22T22:05:14.756 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/ 2023-09-22T22:05:14.756 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:05:14.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:05:14.759 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dab71441e84d11e9-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.759 DEBUG:boto:path=/ffzdqu-63/obj46 2023-09-22T22:05:14.760 DEBUG:boto:auth_path=/ffzdqu-63/obj46 2023-09-22T22:05:14.760 DEBUG:boto:Method: PUT 2023-09-22T22:05:14.760 DEBUG:boto:Path: /ffzdqu-63/obj46 2023-09-22T22:05:14.760 DEBUG:boto:Data: 2023-09-22T22:05:14.760 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:14.760 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.760 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.760 DEBUG:boto:Params: {} 2023-09-22T22:05:14.760 DEBUG:boto:Token: None 2023-09-22T22:05:14.761 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/obj46 2023-09-22T22:05:14.761 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eMPrMyqdKcrlbhrhS9WvhBusD6I= 2023-09-22T22:05:14.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eMPrMyqdKcrlbhrhS9WvhBusD6I='} 2023-09-22T22:05:14.802 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001317e00bbc8d3cb4-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.803 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:14.803 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:14.803 DEBUG:boto:Method: HEAD 2023-09-22T22:05:14.803 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:14.803 DEBUG:boto:Data: 2023-09-22T22:05:14.803 DEBUG:boto:Headers: {} 2023-09-22T22:05:14.803 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.803 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.803 DEBUG:boto:Params: {} 2023-09-22T22:05:14.804 DEBUG:boto:Token: None 2023-09-22T22:05:14.804 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/ 2023-09-22T22:05:14.804 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:05:14.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:05:14.808 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f20f29a2bab43c8b-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.809 DEBUG:boto:path=/ffzdqu-63/obj69 2023-09-22T22:05:14.809 DEBUG:boto:auth_path=/ffzdqu-63/obj69 2023-09-22T22:05:14.809 DEBUG:boto:Method: PUT 2023-09-22T22:05:14.809 DEBUG:boto:Path: /ffzdqu-63/obj69 2023-09-22T22:05:14.809 DEBUG:boto:Data: 2023-09-22T22:05:14.809 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:14.809 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.809 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.809 DEBUG:boto:Params: {} 2023-09-22T22:05:14.810 DEBUG:boto:Token: None 2023-09-22T22:05:14.810 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/obj69 2023-09-22T22:05:14.810 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+hWMoOsiPMnOTOmpzj+etVLzPeQ= 2023-09-22T22:05:14.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+hWMoOsiPMnOTOmpzj+etVLzPeQ='} 2023-09-22T22:05:14.820 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009471ab76139b55ea-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.820 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:14.821 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:14.821 DEBUG:boto:Method: HEAD 2023-09-22T22:05:14.821 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:14.821 DEBUG:boto:Data: 2023-09-22T22:05:14.821 DEBUG:boto:Headers: {} 2023-09-22T22:05:14.821 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.821 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.821 DEBUG:boto:Params: {} 2023-09-22T22:05:14.821 DEBUG:boto:Token: None 2023-09-22T22:05:14.822 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/ 2023-09-22T22:05:14.822 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:05:14.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:05:14.826 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d1c15fbe9600701-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.826 DEBUG:boto:path=/ffzdqu-63/obj92 2023-09-22T22:05:14.826 DEBUG:boto:auth_path=/ffzdqu-63/obj92 2023-09-22T22:05:14.826 DEBUG:boto:Method: PUT 2023-09-22T22:05:14.826 DEBUG:boto:Path: /ffzdqu-63/obj92 2023-09-22T22:05:14.827 DEBUG:boto:Data: 2023-09-22T22:05:14.827 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:14.827 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.827 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.827 DEBUG:boto:Params: {} 2023-09-22T22:05:14.827 DEBUG:boto:Token: None 2023-09-22T22:05:14.827 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/obj92 2023-09-22T22:05:14.827 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:i6cbNbg0Z7NPXhvCWrP3vg1+dU0= 2023-09-22T22:05:14.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i6cbNbg0Z7NPXhvCWrP3vg1+dU0='} 2023-09-22T22:05:14.874 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000933c5a4a71a9a23-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.875 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:14.875 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:14.875 DEBUG:boto:Method: HEAD 2023-09-22T22:05:14.875 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:14.875 DEBUG:boto:Data: 2023-09-22T22:05:14.875 DEBUG:boto:Headers: {} 2023-09-22T22:05:14.875 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.875 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.875 DEBUG:boto:Params: {} 2023-09-22T22:05:14.876 DEBUG:boto:Token: None 2023-09-22T22:05:14.876 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/ 2023-09-22T22:05:14.876 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:05:14.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:05:14.887 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bad6d89b72f4fc49-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.887 DEBUG:boto:path=/ffzdqu-63/obj115 2023-09-22T22:05:14.887 DEBUG:boto:auth_path=/ffzdqu-63/obj115 2023-09-22T22:05:14.887 DEBUG:boto:Method: PUT 2023-09-22T22:05:14.887 DEBUG:boto:Path: /ffzdqu-63/obj115 2023-09-22T22:05:14.887 DEBUG:boto:Data: 2023-09-22T22:05:14.887 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:14.888 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:14.888 DEBUG:boto:Port: 8000 2023-09-22T22:05:14.888 DEBUG:boto:Params: {} 2023-09-22T22:05:14.888 DEBUG:boto:Token: None 2023-09-22T22:05:14.888 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:14 GMT /ffzdqu-63/obj115 2023-09-22T22:05:14.888 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GLTkCkeu56p5fjGmIuRWLw3hC20= 2023-09-22T22:05:14.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GLTkCkeu56p5fjGmIuRWLw3hC20='} 2023-09-22T22:05:14.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ccf4c3747ec87fe5-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:14.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:05:14.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:05:14.903 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:05:14.950 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:05:14.950 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:05:15.016 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T22:05:15.017 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-63 2023-09-22T22:05:15.030 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 20 2023-09-22T22:05:15.099 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:15.099 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:15.099 DEBUG:boto:Method: HEAD 2023-09-22T22:05:15.099 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:15.099 DEBUG:boto:Data: 2023-09-22T22:05:15.099 DEBUG:boto:Headers: {} 2023-09-22T22:05:15.099 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.099 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.099 DEBUG:boto:Params: {} 2023-09-22T22:05:15.100 DEBUG:boto:Token: None 2023-09-22T22:05:15.100 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/ 2023-09-22T22:05:15.100 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:05:15.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:05:15.104 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002832cb6d1cc84109-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.104 DEBUG:boto:path=/ffzdqu-63/obj29 2023-09-22T22:05:15.104 DEBUG:boto:auth_path=/ffzdqu-63/obj29 2023-09-22T22:05:15.104 DEBUG:boto:Method: PUT 2023-09-22T22:05:15.105 DEBUG:boto:Path: /ffzdqu-63/obj29 2023-09-22T22:05:15.105 DEBUG:boto:Data: 2023-09-22T22:05:15.105 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:15.105 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.105 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.105 DEBUG:boto:Params: {} 2023-09-22T22:05:15.105 DEBUG:boto:Token: None 2023-09-22T22:05:15.106 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/obj29 2023-09-22T22:05:15.106 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kIp1rDDxoKSTTYsulvt8L/NwYD8= 2023-09-22T22:05:15.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kIp1rDDxoKSTTYsulvt8L/NwYD8='} 2023-09-22T22:05:15.126 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T22:05:15.123+0000 7f7bd38e3640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T22:05:15.150 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000051ecbc9bd2f4ba5-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.151 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:15.151 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:15.151 DEBUG:boto:Method: HEAD 2023-09-22T22:05:15.151 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:15.151 DEBUG:boto:Data: 2023-09-22T22:05:15.151 DEBUG:boto:Headers: {} 2023-09-22T22:05:15.151 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.152 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.152 DEBUG:boto:Params: {} 2023-09-22T22:05:15.152 DEBUG:boto:Token: None 2023-09-22T22:05:15.152 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/ 2023-09-22T22:05:15.152 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:05:15.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:05:15.155 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051b67d5900e62123-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.156 DEBUG:boto:path=/ffzdqu-63/obj58 2023-09-22T22:05:15.156 DEBUG:boto:auth_path=/ffzdqu-63/obj58 2023-09-22T22:05:15.156 DEBUG:boto:Method: PUT 2023-09-22T22:05:15.156 DEBUG:boto:Path: /ffzdqu-63/obj58 2023-09-22T22:05:15.156 DEBUG:boto:Data: 2023-09-22T22:05:15.157 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:15.157 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.157 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.157 DEBUG:boto:Params: {} 2023-09-22T22:05:15.157 DEBUG:boto:Token: None 2023-09-22T22:05:15.157 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/obj58 2023-09-22T22:05:15.157 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aEMCz98dlJp+nOIUAQofTt1yj68= 2023-09-22T22:05:15.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aEMCz98dlJp+nOIUAQofTt1yj68='} 2023-09-22T22:05:15.164 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fed1b496f80b5029-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.165 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:15.165 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:15.165 DEBUG:boto:Method: HEAD 2023-09-22T22:05:15.165 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:15.165 DEBUG:boto:Data: 2023-09-22T22:05:15.165 DEBUG:boto:Headers: {} 2023-09-22T22:05:15.165 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.166 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.166 DEBUG:boto:Params: {} 2023-09-22T22:05:15.166 DEBUG:boto:Token: None 2023-09-22T22:05:15.166 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/ 2023-09-22T22:05:15.166 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:05:15.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:05:15.170 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004599ea8d618541be-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.170 DEBUG:boto:path=/ffzdqu-63/obj87 2023-09-22T22:05:15.171 DEBUG:boto:auth_path=/ffzdqu-63/obj87 2023-09-22T22:05:15.171 DEBUG:boto:Method: PUT 2023-09-22T22:05:15.171 DEBUG:boto:Path: /ffzdqu-63/obj87 2023-09-22T22:05:15.171 DEBUG:boto:Data: 2023-09-22T22:05:15.171 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:15.171 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.171 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.171 DEBUG:boto:Params: {} 2023-09-22T22:05:15.172 DEBUG:boto:Token: None 2023-09-22T22:05:15.172 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/obj87 2023-09-22T22:05:15.172 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0suG7HiA5ukwIJkzgsXt/p1xYu0= 2023-09-22T22:05:15.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0suG7HiA5ukwIJkzgsXt/p1xYu0='} 2023-09-22T22:05:15.218 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077cd2d471f7c2ea6-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.219 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:15.219 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:15.219 DEBUG:boto:Method: HEAD 2023-09-22T22:05:15.219 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:15.219 DEBUG:boto:Data: 2023-09-22T22:05:15.219 DEBUG:boto:Headers: {} 2023-09-22T22:05:15.219 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.219 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.219 DEBUG:boto:Params: {} 2023-09-22T22:05:15.220 DEBUG:boto:Token: None 2023-09-22T22:05:15.220 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/ 2023-09-22T22:05:15.220 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:05:15.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:05:15.224 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089176a389c21f5fe-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.224 DEBUG:boto:path=/ffzdqu-63/obj116 2023-09-22T22:05:15.224 DEBUG:boto:auth_path=/ffzdqu-63/obj116 2023-09-22T22:05:15.225 DEBUG:boto:Method: PUT 2023-09-22T22:05:15.225 DEBUG:boto:Path: /ffzdqu-63/obj116 2023-09-22T22:05:15.225 DEBUG:boto:Data: 2023-09-22T22:05:15.225 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:15.225 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.225 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.225 DEBUG:boto:Params: {} 2023-09-22T22:05:15.225 DEBUG:boto:Token: None 2023-09-22T22:05:15.226 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/obj116 2023-09-22T22:05:15.226 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sP36Z/dhzdVuZapG8/K0J+xlWwo= 2023-09-22T22:05:15.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sP36Z/dhzdVuZapG8/K0J+xlWwo='} 2023-09-22T22:05:15.235 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c7c2944c503a268-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.235 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:15.235 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:15.235 DEBUG:boto:Method: HEAD 2023-09-22T22:05:15.236 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:15.236 DEBUG:boto:Data: 2023-09-22T22:05:15.236 DEBUG:boto:Headers: {} 2023-09-22T22:05:15.236 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.236 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.236 DEBUG:boto:Params: {} 2023-09-22T22:05:15.236 DEBUG:boto:Token: None 2023-09-22T22:05:15.236 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/ 2023-09-22T22:05:15.237 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:05:15.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:05:15.241 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000896a6a1c2d115896-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.242 DEBUG:boto:path=/ffzdqu-63/obj145 2023-09-22T22:05:15.242 DEBUG:boto:auth_path=/ffzdqu-63/obj145 2023-09-22T22:05:15.242 DEBUG:boto:Method: PUT 2023-09-22T22:05:15.242 DEBUG:boto:Path: /ffzdqu-63/obj145 2023-09-22T22:05:15.242 DEBUG:boto:Data: 2023-09-22T22:05:15.242 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:15.242 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.242 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.243 DEBUG:boto:Params: {} 2023-09-22T22:05:15.243 DEBUG:boto:Token: None 2023-09-22T22:05:15.243 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/obj145 2023-09-22T22:05:15.243 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vMRgIH1FUTDDKCRAbfaIZxncqJs= 2023-09-22T22:05:15.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vMRgIH1FUTDDKCRAbfaIZxncqJs='} 2023-09-22T22:05:15.286 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a9d69cc3ea6a4c0-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:05:15.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:05:15.287 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:05:15.332 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:05:15.332 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:05:15.396 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T22:05:15.396 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-63 2023-09-22T22:05:15.414 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 25 2023-09-22T22:05:15.491 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:15.492 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:15.492 DEBUG:boto:Method: HEAD 2023-09-22T22:05:15.492 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:15.492 DEBUG:boto:Data: 2023-09-22T22:05:15.492 DEBUG:boto:Headers: {} 2023-09-22T22:05:15.492 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.492 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.492 DEBUG:boto:Params: {} 2023-09-22T22:05:15.493 DEBUG:boto:Token: None 2023-09-22T22:05:15.493 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/ 2023-09-22T22:05:15.493 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:05:15.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:05:15.497 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000129595926ac7888f-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.497 DEBUG:boto:path=/ffzdqu-63/obj31 2023-09-22T22:05:15.497 DEBUG:boto:auth_path=/ffzdqu-63/obj31 2023-09-22T22:05:15.498 DEBUG:boto:Method: PUT 2023-09-22T22:05:15.498 DEBUG:boto:Path: /ffzdqu-63/obj31 2023-09-22T22:05:15.498 DEBUG:boto:Data: 2023-09-22T22:05:15.498 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:15.498 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.498 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.498 DEBUG:boto:Params: {} 2023-09-22T22:05:15.498 DEBUG:boto:Token: None 2023-09-22T22:05:15.498 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/obj31 2023-09-22T22:05:15.499 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iNbxX/mL1V2KR/2syAASToA/DBk= 2023-09-22T22:05:15.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iNbxX/mL1V2KR/2syAASToA/DBk='} 2023-09-22T22:05:15.505 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a6bafeb9e27af81-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.505 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:15.505 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:15.505 DEBUG:boto:Method: HEAD 2023-09-22T22:05:15.505 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:15.505 DEBUG:boto:Data: 2023-09-22T22:05:15.506 DEBUG:boto:Headers: {} 2023-09-22T22:05:15.506 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.506 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.506 DEBUG:boto:Params: {} 2023-09-22T22:05:15.506 DEBUG:boto:Token: None 2023-09-22T22:05:15.506 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/ 2023-09-22T22:05:15.506 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:05:15.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:05:15.510 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ef2590a159f94ef-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.510 DEBUG:boto:path=/ffzdqu-63/obj62 2023-09-22T22:05:15.510 DEBUG:boto:auth_path=/ffzdqu-63/obj62 2023-09-22T22:05:15.510 DEBUG:boto:Method: PUT 2023-09-22T22:05:15.510 DEBUG:boto:Path: /ffzdqu-63/obj62 2023-09-22T22:05:15.510 DEBUG:boto:Data: 2023-09-22T22:05:15.510 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:15.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.511 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.511 DEBUG:boto:Params: {} 2023-09-22T22:05:15.511 DEBUG:boto:Token: None 2023-09-22T22:05:15.511 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/obj62 2023-09-22T22:05:15.511 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:D2jo28niSCXgi7HYsXd9/haHhsU= 2023-09-22T22:05:15.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D2jo28niSCXgi7HYsXd9/haHhsU='} 2023-09-22T22:05:15.517 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029923cb4d956979c-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.517 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:15.517 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:15.518 DEBUG:boto:Method: HEAD 2023-09-22T22:05:15.518 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:15.518 DEBUG:boto:Data: 2023-09-22T22:05:15.518 DEBUG:boto:Headers: {} 2023-09-22T22:05:15.518 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.518 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.518 DEBUG:boto:Params: {} 2023-09-22T22:05:15.518 DEBUG:boto:Token: None 2023-09-22T22:05:15.518 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/ 2023-09-22T22:05:15.519 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:05:15.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:05:15.522 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094d9e25885d11b4e-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.522 DEBUG:boto:path=/ffzdqu-63/obj93 2023-09-22T22:05:15.522 DEBUG:boto:auth_path=/ffzdqu-63/obj93 2023-09-22T22:05:15.522 DEBUG:boto:Method: PUT 2023-09-22T22:05:15.522 DEBUG:boto:Path: /ffzdqu-63/obj93 2023-09-22T22:05:15.522 DEBUG:boto:Data: 2023-09-22T22:05:15.522 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:15.522 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.523 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.523 DEBUG:boto:Params: {} 2023-09-22T22:05:15.523 DEBUG:boto:Token: None 2023-09-22T22:05:15.523 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/obj93 2023-09-22T22:05:15.523 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dYgkq02BfzT4wkkcCxlDQ+GtYGc= 2023-09-22T22:05:15.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dYgkq02BfzT4wkkcCxlDQ+GtYGc='} 2023-09-22T22:05:15.566 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000651b6193bcda5f16-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.567 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:15.567 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:15.567 DEBUG:boto:Method: HEAD 2023-09-22T22:05:15.567 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:15.567 DEBUG:boto:Data: 2023-09-22T22:05:15.567 DEBUG:boto:Headers: {} 2023-09-22T22:05:15.567 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.567 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.567 DEBUG:boto:Params: {} 2023-09-22T22:05:15.568 DEBUG:boto:Token: None 2023-09-22T22:05:15.568 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/ 2023-09-22T22:05:15.568 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:05:15.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:05:15.574 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f46f35ed90e6561-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.574 DEBUG:boto:path=/ffzdqu-63/obj124 2023-09-22T22:05:15.574 DEBUG:boto:auth_path=/ffzdqu-63/obj124 2023-09-22T22:05:15.574 DEBUG:boto:Method: PUT 2023-09-22T22:05:15.575 DEBUG:boto:Path: /ffzdqu-63/obj124 2023-09-22T22:05:15.575 DEBUG:boto:Data: 2023-09-22T22:05:15.575 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:15.575 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.575 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.575 DEBUG:boto:Params: {} 2023-09-22T22:05:15.575 DEBUG:boto:Token: None 2023-09-22T22:05:15.575 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/obj124 2023-09-22T22:05:15.575 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JH0S79ijGJEsCfgXk4bwynZwoe4= 2023-09-22T22:05:15.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JH0S79ijGJEsCfgXk4bwynZwoe4='} 2023-09-22T22:05:15.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000afc5e19c91a90e39-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.587 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:15.587 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:15.587 DEBUG:boto:Method: HEAD 2023-09-22T22:05:15.587 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:15.587 DEBUG:boto:Data: 2023-09-22T22:05:15.587 DEBUG:boto:Headers: {} 2023-09-22T22:05:15.587 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.587 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.587 DEBUG:boto:Params: {} 2023-09-22T22:05:15.588 DEBUG:boto:Token: None 2023-09-22T22:05:15.588 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/ 2023-09-22T22:05:15.588 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:05:15.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:05:15.594 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b6f6ac4d1d9c4dd-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.594 DEBUG:boto:path=/ffzdqu-63/obj155 2023-09-22T22:05:15.594 DEBUG:boto:auth_path=/ffzdqu-63/obj155 2023-09-22T22:05:15.594 DEBUG:boto:Method: PUT 2023-09-22T22:05:15.594 DEBUG:boto:Path: /ffzdqu-63/obj155 2023-09-22T22:05:15.594 DEBUG:boto:Data: 2023-09-22T22:05:15.595 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:15.595 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.595 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.595 DEBUG:boto:Params: {} 2023-09-22T22:05:15.595 DEBUG:boto:Token: None 2023-09-22T22:05:15.595 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/obj155 2023-09-22T22:05:15.595 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7v9Y2stHJ4mejDL9AhsPSn6FmfA= 2023-09-22T22:05:15.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7v9Y2stHJ4mejDL9AhsPSn6FmfA='} 2023-09-22T22:05:15.639 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043bc08fe4cf5df7e-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:05:15.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:05:15.639 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:05:15.686 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:05:15.686 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:05:15.749 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T22:05:15.750 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-63 2023-09-22T22:05:15.769 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 30 2023-09-22T22:05:15.849 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:15.850 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:15.850 DEBUG:boto:Method: HEAD 2023-09-22T22:05:15.850 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:15.850 DEBUG:boto:Data: 2023-09-22T22:05:15.851 DEBUG:boto:Headers: {} 2023-09-22T22:05:15.851 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.851 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.851 DEBUG:boto:Params: {} 2023-09-22T22:05:15.851 DEBUG:boto:Token: None 2023-09-22T22:05:15.852 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/ 2023-09-22T22:05:15.852 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:05:15.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:05:15.857 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056db82d9ef6ac0b4-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.857 DEBUG:boto:path=/ffzdqu-63/obj37 2023-09-22T22:05:15.857 DEBUG:boto:auth_path=/ffzdqu-63/obj37 2023-09-22T22:05:15.857 DEBUG:boto:Method: PUT 2023-09-22T22:05:15.857 DEBUG:boto:Path: /ffzdqu-63/obj37 2023-09-22T22:05:15.857 DEBUG:boto:Data: 2023-09-22T22:05:15.858 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:15.858 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.858 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.858 DEBUG:boto:Params: {} 2023-09-22T22:05:15.858 DEBUG:boto:Token: None 2023-09-22T22:05:15.858 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/obj37 2023-09-22T22:05:15.858 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hdnzWO0AxYLo9l10XUBQmoXyCSM= 2023-09-22T22:05:15.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hdnzWO0AxYLo9l10XUBQmoXyCSM='} 2023-09-22T22:05:15.865 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002319d1a35bbda96a-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.866 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:15.866 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:15.866 DEBUG:boto:Method: HEAD 2023-09-22T22:05:15.866 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:15.866 DEBUG:boto:Data: 2023-09-22T22:05:15.866 DEBUG:boto:Headers: {} 2023-09-22T22:05:15.866 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.866 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.866 DEBUG:boto:Params: {} 2023-09-22T22:05:15.867 DEBUG:boto:Token: None 2023-09-22T22:05:15.867 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/ 2023-09-22T22:05:15.867 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:05:15.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:05:15.871 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e11f48cd954fecf4-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.871 DEBUG:boto:path=/ffzdqu-63/obj74 2023-09-22T22:05:15.872 DEBUG:boto:auth_path=/ffzdqu-63/obj74 2023-09-22T22:05:15.872 DEBUG:boto:Method: PUT 2023-09-22T22:05:15.872 DEBUG:boto:Path: /ffzdqu-63/obj74 2023-09-22T22:05:15.872 DEBUG:boto:Data: 2023-09-22T22:05:15.872 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:15.872 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.872 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.872 DEBUG:boto:Params: {} 2023-09-22T22:05:15.872 DEBUG:boto:Token: None 2023-09-22T22:05:15.873 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/obj74 2023-09-22T22:05:15.873 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WWNgAXfi7Cbj3xBBHPG/uynL4wM= 2023-09-22T22:05:15.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WWNgAXfi7Cbj3xBBHPG/uynL4wM='} 2023-09-22T22:05:15.918 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036ebdd69127b76fb-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.919 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:15.919 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:15.919 DEBUG:boto:Method: HEAD 2023-09-22T22:05:15.919 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:15.919 DEBUG:boto:Data: 2023-09-22T22:05:15.919 DEBUG:boto:Headers: {} 2023-09-22T22:05:15.919 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.919 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.919 DEBUG:boto:Params: {} 2023-09-22T22:05:15.920 DEBUG:boto:Token: None 2023-09-22T22:05:15.920 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/ 2023-09-22T22:05:15.920 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:05:15.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:05:15.926 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7f915c4c42735e8-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.926 DEBUG:boto:path=/ffzdqu-63/obj111 2023-09-22T22:05:15.926 DEBUG:boto:auth_path=/ffzdqu-63/obj111 2023-09-22T22:05:15.926 DEBUG:boto:Method: PUT 2023-09-22T22:05:15.926 DEBUG:boto:Path: /ffzdqu-63/obj111 2023-09-22T22:05:15.927 DEBUG:boto:Data: 2023-09-22T22:05:15.927 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:15.927 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.927 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.927 DEBUG:boto:Params: {} 2023-09-22T22:05:15.927 DEBUG:boto:Token: None 2023-09-22T22:05:15.927 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/obj111 2023-09-22T22:05:15.927 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7XvmPlMh+LB4HYDZ7Rk/Zq2mGRo= 2023-09-22T22:05:15.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7XvmPlMh+LB4HYDZ7Rk/Zq2mGRo='} 2023-09-22T22:05:15.938 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000023401e58001074e4-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.938 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:15.938 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:15.938 DEBUG:boto:Method: HEAD 2023-09-22T22:05:15.938 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:15.938 DEBUG:boto:Data: 2023-09-22T22:05:15.938 DEBUG:boto:Headers: {} 2023-09-22T22:05:15.938 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.939 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.939 DEBUG:boto:Params: {} 2023-09-22T22:05:15.939 DEBUG:boto:Token: None 2023-09-22T22:05:15.939 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/ 2023-09-22T22:05:15.939 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:05:15.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:05:15.944 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008dc4b5f3a3fa8fd8-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.945 DEBUG:boto:path=/ffzdqu-63/obj148 2023-09-22T22:05:15.945 DEBUG:boto:auth_path=/ffzdqu-63/obj148 2023-09-22T22:05:15.945 DEBUG:boto:Method: PUT 2023-09-22T22:05:15.945 DEBUG:boto:Path: /ffzdqu-63/obj148 2023-09-22T22:05:15.945 DEBUG:boto:Data: 2023-09-22T22:05:15.945 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:15.945 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.945 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.945 DEBUG:boto:Params: {} 2023-09-22T22:05:15.946 DEBUG:boto:Token: None 2023-09-22T22:05:15.946 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/obj148 2023-09-22T22:05:15.946 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Iddyv3VPHpH5c8+rKlIbp6xr6v4= 2023-09-22T22:05:15.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Iddyv3VPHpH5c8+rKlIbp6xr6v4='} 2023-09-22T22:05:15.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c864a3356cb227a-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.990 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:15.991 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:15.991 DEBUG:boto:Method: HEAD 2023-09-22T22:05:15.991 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:15.991 DEBUG:boto:Data: 2023-09-22T22:05:15.991 DEBUG:boto:Headers: {} 2023-09-22T22:05:15.991 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:15.991 DEBUG:boto:Port: 8000 2023-09-22T22:05:15.991 DEBUG:boto:Params: {} 2023-09-22T22:05:15.991 DEBUG:boto:Token: None 2023-09-22T22:05:15.992 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:15 GMT /ffzdqu-63/ 2023-09-22T22:05:15.992 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:05:15.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:05:15.999 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005247631bd05b5516-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:15.999 DEBUG:boto:path=/ffzdqu-63/obj185 2023-09-22T22:05:16.000 DEBUG:boto:auth_path=/ffzdqu-63/obj185 2023-09-22T22:05:16.000 DEBUG:boto:Method: PUT 2023-09-22T22:05:16.000 DEBUG:boto:Path: /ffzdqu-63/obj185 2023-09-22T22:05:16.000 DEBUG:boto:Data: 2023-09-22T22:05:16.000 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:16.000 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:16.000 DEBUG:boto:Port: 8000 2023-09-22T22:05:16.000 DEBUG:boto:Params: {} 2023-09-22T22:05:16.000 DEBUG:boto:Token: None 2023-09-22T22:05:16.001 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:16 GMT /ffzdqu-63/obj185 2023-09-22T22:05:16.001 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GelrMoTGZdYolxxpOaE99j2cTwk= 2023-09-22T22:05:16.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GelrMoTGZdYolxxpOaE99j2cTwk='} 2023-09-22T22:05:16.011 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004fe5c955ae744bc3-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:16.011 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-63 2023-09-22T22:05:16.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:05:16.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:05:16.011 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T22:05:16.059 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:05:16.060 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:05:16.104 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:05:16.103+0000 7fa098d9ca40 1 waiting to reach incremental sync.. 2023-09-22T22:05:21.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:05:21.103+0000 7fa098d9ca40 1 waiting to reach incremental sync.. 2023-09-22T22:05:26.112 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:05:26.111+0000 7fa098d9ca40 1 bucket sync caught up with source: 2023-09-22T22:05:26.112 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:05:26.112 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:05:26.112 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:05:26.111+0000 7fa098d9ca40 0 bucket checkpoint complete 2023-09-22T22:05:26.144 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-63 2023-09-22T22:05:26.144 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:05:26.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:05:26.144 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-63 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T22:05:26.235 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:05:26.235 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:05:26.280 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:05:26.279+0000 7fd3ea06da40 1 waiting to reach incremental sync.. 2023-09-22T22:05:31.281 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:05:31.279+0000 7fd3ea06da40 1 waiting to reach incremental sync.. 2023-09-22T22:05:31.418 INFO:tasks.rgw.c1.client.0.smithi160.stdout:2023-09-22T22:05:31.415+0000 7fa9c1675640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T22:05:36.284 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:05:36.283+0000 7fd3ea06da40 1 bucket sync caught up with source: 2023-09-22T22:05:36.284 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.192.6, 00000000002.259.6, 00000000002.426.6, 00000000005.288.6, 00000000004.263.6, 00000000001.298.6, 00000000001.186.6, 00000000004.161.6, 00000000007.358.6, 00000000006.281.6] 2023-09-22T22:05:36.284 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.192.6, 00000000002.259.6, 00000000002.426.6, 00000000005.288.6, 00000000004.263.6, 00000000001.298.6, 00000000001.186.6, 00000000004.161.6, 00000000007.358.6, 00000000006.281.6] 2023-09-22T22:05:36.284 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:05:36.283+0000 7fd3ea06da40 0 bucket checkpoint complete 2023-09-22T22:05:36.317 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-63 zones={a2, a1} 2023-09-22T22:05:36.317 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:36.317 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:36.317 DEBUG:boto:Method: HEAD 2023-09-22T22:05:36.317 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:36.318 DEBUG:boto:Data: 2023-09-22T22:05:36.318 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.318 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.318 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.318 DEBUG:boto:Params: {} 2023-09-22T22:05:36.318 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T22:05:36.318 DEBUG:boto:Token: None 2023-09-22T22:05:36.319 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/ 2023-09-22T22:05:36.319 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VXStreklr320+CMWJc/abq+53FA= 2023-09-22T22:05:36.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VXStreklr320+CMWJc/abq+53FA=', 'Content-Length': '0'} 2023-09-22T22:05:36.326 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6d33b4290658527-00650e0fb0-1f3d-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.326 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:36.326 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:36.326 DEBUG:boto:Method: HEAD 2023-09-22T22:05:36.326 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:36.326 DEBUG:boto:Data: 2023-09-22T22:05:36.326 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.327 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.327 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.327 DEBUG:boto:Params: {} 2023-09-22T22:05:36.327 DEBUG:boto:Token: None 2023-09-22T22:05:36.327 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/ 2023-09-22T22:05:36.327 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VXStreklr320+CMWJc/abq+53FA= 2023-09-22T22:05:36.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VXStreklr320+CMWJc/abq+53FA=', 'Content-Length': '0'} 2023-09-22T22:05:36.332 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d74f39d3c7811d5-00650e0fb0-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.333 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T22:05:36.333 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:36.333 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:36.333 DEBUG:boto:path=/ffzdqu-63/?versions&max-keys=999 2023-09-22T22:05:36.333 DEBUG:boto:auth_path=/ffzdqu-63/?versions&max-keys=999 2023-09-22T22:05:36.333 DEBUG:boto:Method: GET 2023-09-22T22:05:36.333 DEBUG:boto:Path: /ffzdqu-63/?versions&max-keys=999 2023-09-22T22:05:36.333 DEBUG:boto:Data: 2023-09-22T22:05:36.334 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.334 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.334 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.334 DEBUG:boto:Params: {} 2023-09-22T22:05:36.334 DEBUG:boto:Token: None 2023-09-22T22:05:36.334 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/?versions 2023-09-22T22:05:36.334 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:O7ggq92aKlvvTpA9x89lddg1c2o= 2023-09-22T22:05:36.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:O7ggq92aKlvvTpA9x89lddg1c2o=', 'Content-Length': '0'} 2023-09-22T22:05:36.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009eb2c1fcd8f2f875-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.338 DEBUG:boto:b'ffzdqu-63999falseobj11nulltrue2023-09-22T22:05:13.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T22:05:15.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T22:05:15.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-22T22:05:15.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T22:05:15.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T22:05:14.042Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-22T22:05:16.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T22:05:14.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T22:05:13.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T22:05:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T22:05:15.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T22:05:15.503Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T22:05:13.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T22:05:14.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-22T22:05:15.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T22:05:14.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T22:05:13.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T22:05:14.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T22:05:14.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T22:05:13.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T22:05:14.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T22:05:15.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T22:05:15.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T22:05:14.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T22:05:14.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-22T22:05:15.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T22:05:14.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T22:05:14.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T22:05:15.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T22:05:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T22:05:15.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T22:05:14.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:36.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T22:05:36.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-09-22T22:05:36.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-22T22:05:36.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-22T22:05:36.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-09-22T22:05:36.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-22T22:05:36.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-09-22T22:05:36.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-09-22T22:05:36.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-22T22:05:36.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-09-22T22:05:36.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-22T22:05:36.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T22:05:36.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-22T22:05:36.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-22T22:05:36.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-09-22T22:05:36.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T22:05:36.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-22T22:05:36.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-09-22T22:05:36.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-22T22:05:36.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T22:05:36.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-22T22:05:36.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-22T22:05:36.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T22:05:36.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-22T22:05:36.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-22T22:05:36.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-09-22T22:05:36.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-22T22:05:36.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-22T22:05:36.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-09-22T22:05:36.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-22T22:05:36.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-22T22:05:36.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-22T22:05:36.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-22T22:05:36.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-09-22T22:05:36.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-22T22:05:36.344 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T22:05:36.344 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:36.344 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:36.344 DEBUG:boto:path=/ffzdqu-63/?versions&max-keys=999 2023-09-22T22:05:36.344 DEBUG:boto:auth_path=/ffzdqu-63/?versions&max-keys=999 2023-09-22T22:05:36.344 DEBUG:boto:Method: GET 2023-09-22T22:05:36.345 DEBUG:boto:Path: /ffzdqu-63/?versions&max-keys=999 2023-09-22T22:05:36.345 DEBUG:boto:Data: 2023-09-22T22:05:36.345 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.345 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.345 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.345 DEBUG:boto:Params: {} 2023-09-22T22:05:36.345 DEBUG:boto:Token: None 2023-09-22T22:05:36.345 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/?versions 2023-09-22T22:05:36.345 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:O7ggq92aKlvvTpA9x89lddg1c2o= 2023-09-22T22:05:36.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:O7ggq92aKlvvTpA9x89lddg1c2o=', 'Content-Length': '0'} 2023-09-22T22:05:36.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013ed9518d5f93a51-00650e0fb0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.353 DEBUG:boto:b'ffzdqu-63999falseobj11nulltrue2023-09-22T22:05:13.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T22:05:15.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T22:05:15.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-22T22:05:15.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T22:05:15.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T22:05:14.042Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-22T22:05:16.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T22:05:14.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T22:05:13.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T22:05:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T22:05:15.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T22:05:15.503Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T22:05:13.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T22:05:14.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-22T22:05:15.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T22:05:14.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T22:05:13.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T22:05:14.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T22:05:14.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T22:05:13.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T22:05:14.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T22:05:15.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T22:05:15.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T22:05:14.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T22:05:14.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-22T22:05:15.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T22:05:14.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T22:05:14.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T22:05:15.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T22:05:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T22:05:15.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T22:05:14.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:36.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T22:05:36.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-09-22T22:05:36.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-22T22:05:36.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-22T22:05:36.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-09-22T22:05:36.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-22T22:05:36.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-09-22T22:05:36.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-09-22T22:05:36.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-22T22:05:36.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-09-22T22:05:36.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-22T22:05:36.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T22:05:36.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-22T22:05:36.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-22T22:05:36.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-09-22T22:05:36.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T22:05:36.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-22T22:05:36.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-09-22T22:05:36.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-22T22:05:36.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T22:05:36.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-22T22:05:36.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-22T22:05:36.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T22:05:36.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-22T22:05:36.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-22T22:05:36.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-09-22T22:05:36.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-22T22:05:36.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-22T22:05:36.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-09-22T22:05:36.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-22T22:05:36.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-22T22:05:36.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-22T22:05:36.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-22T22:05:36.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-09-22T22:05:36.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-22T22:05:36.358 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:36.358 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:36.359 DEBUG:boto:path=/ffzdqu-63/?versions&max-keys=999 2023-09-22T22:05:36.359 DEBUG:boto:auth_path=/ffzdqu-63/?versions&max-keys=999 2023-09-22T22:05:36.359 DEBUG:boto:Method: GET 2023-09-22T22:05:36.359 DEBUG:boto:Path: /ffzdqu-63/?versions&max-keys=999 2023-09-22T22:05:36.359 DEBUG:boto:Data: 2023-09-22T22:05:36.359 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.359 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.359 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.359 DEBUG:boto:Params: {} 2023-09-22T22:05:36.359 DEBUG:boto:Token: None 2023-09-22T22:05:36.360 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/?versions 2023-09-22T22:05:36.360 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:O7ggq92aKlvvTpA9x89lddg1c2o= 2023-09-22T22:05:36.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:O7ggq92aKlvvTpA9x89lddg1c2o=', 'Content-Length': '0'} 2023-09-22T22:05:36.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e572f4a226894df-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.365 DEBUG:boto:b'ffzdqu-63999falseobj11nulltrue2023-09-22T22:05:13.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T22:05:15.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T22:05:15.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-22T22:05:15.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T22:05:15.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T22:05:14.042Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-22T22:05:16.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T22:05:14.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T22:05:13.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T22:05:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T22:05:15.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T22:05:15.503Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T22:05:13.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T22:05:14.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-22T22:05:15.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T22:05:14.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T22:05:13.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T22:05:14.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T22:05:14.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T22:05:13.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T22:05:14.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T22:05:15.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T22:05:15.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T22:05:14.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T22:05:14.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-22T22:05:15.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T22:05:14.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T22:05:14.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T22:05:15.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T22:05:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T22:05:15.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T22:05:14.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:36.367 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:36.367 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:36.367 DEBUG:boto:path=/ffzdqu-63/?versions&max-keys=999 2023-09-22T22:05:36.367 DEBUG:boto:auth_path=/ffzdqu-63/?versions&max-keys=999 2023-09-22T22:05:36.367 DEBUG:boto:Method: GET 2023-09-22T22:05:36.367 DEBUG:boto:Path: /ffzdqu-63/?versions&max-keys=999 2023-09-22T22:05:36.368 DEBUG:boto:Data: 2023-09-22T22:05:36.368 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.368 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.368 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.368 DEBUG:boto:Params: {} 2023-09-22T22:05:36.368 DEBUG:boto:Token: None 2023-09-22T22:05:36.368 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/?versions 2023-09-22T22:05:36.368 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:O7ggq92aKlvvTpA9x89lddg1c2o= 2023-09-22T22:05:36.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:O7ggq92aKlvvTpA9x89lddg1c2o=', 'Content-Length': '0'} 2023-09-22T22:05:36.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a85d29cc575a5a6-00650e0fb0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.376 DEBUG:boto:b'ffzdqu-63999falseobj11nulltrue2023-09-22T22:05:13.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T22:05:15.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T22:05:15.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-22T22:05:15.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T22:05:15.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T22:05:14.042Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-22T22:05:16.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T22:05:14.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T22:05:13.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T22:05:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T22:05:15.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T22:05:15.503Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T22:05:13.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T22:05:14.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-22T22:05:15.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T22:05:14.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T22:05:13.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T22:05:14.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T22:05:14.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T22:05:13.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T22:05:14.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T22:05:15.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T22:05:15.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T22:05:14.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T22:05:14.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-22T22:05:15.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T22:05:14.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T22:05:14.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T22:05:15.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T22:05:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T22:05:15.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T22:05:14.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:36.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T22:05:36.380 DEBUG:boto:path=/ffzdqu-63/obj11 2023-09-22T22:05:36.380 DEBUG:boto:auth_path=/ffzdqu-63/obj11 2023-09-22T22:05:36.380 DEBUG:boto:path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.380 DEBUG:boto:auth_path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.380 DEBUG:boto:Method: GET 2023-09-22T22:05:36.381 DEBUG:boto:Path: /ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.381 DEBUG:boto:Data: 2023-09-22T22:05:36.381 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.381 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.381 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.381 DEBUG:boto:Params: {} 2023-09-22T22:05:36.381 DEBUG:boto:Token: None 2023-09-22T22:05:36.381 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.382 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gK6tup572M7PH0hYpjVUVvYT3p4= 2023-09-22T22:05:36.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gK6tup572M7PH0hYpjVUVvYT3p4=', 'Content-Length': '0'} 2023-09-22T22:05:36.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a4a6a060dc61120-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.426 DEBUG:boto:path=/ffzdqu-63/obj11 2023-09-22T22:05:36.427 DEBUG:boto:auth_path=/ffzdqu-63/obj11 2023-09-22T22:05:36.427 DEBUG:boto:path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.427 DEBUG:boto:auth_path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.427 DEBUG:boto:Method: GET 2023-09-22T22:05:36.427 DEBUG:boto:Path: /ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.427 DEBUG:boto:Data: 2023-09-22T22:05:36.427 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.427 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.427 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.427 DEBUG:boto:Params: {} 2023-09-22T22:05:36.428 DEBUG:boto:Token: None 2023-09-22T22:05:36.428 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.428 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gK6tup572M7PH0hYpjVUVvYT3p4= 2023-09-22T22:05:36.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gK6tup572M7PH0hYpjVUVvYT3p4=', 'Content-Length': '0'} 2023-09-22T22:05:36.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec201df7b0a21461-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.431 DEBUG:boto:path=/ffzdqu-63/obj11 2023-09-22T22:05:36.431 DEBUG:boto:auth_path=/ffzdqu-63/obj11 2023-09-22T22:05:36.431 DEBUG:boto:path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.431 DEBUG:boto:auth_path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.432 DEBUG:boto:Method: HEAD 2023-09-22T22:05:36.432 DEBUG:boto:Path: /ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.432 DEBUG:boto:Data: 2023-09-22T22:05:36.432 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.432 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.432 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.432 DEBUG:boto:Params: {} 2023-09-22T22:05:36.432 DEBUG:boto:Token: None 2023-09-22T22:05:36.432 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.433 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2Lz8XfQ7iVz2YjcPc7hdSq3HDfo= 2023-09-22T22:05:36.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2Lz8XfQ7iVz2YjcPc7hdSq3HDfo=', 'Content-Length': '0'} 2023-09-22T22:05:36.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070ede2a4dfbfaec6-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.435 DEBUG:boto:path=/ffzdqu-63/obj11 2023-09-22T22:05:36.436 DEBUG:boto:auth_path=/ffzdqu-63/obj11 2023-09-22T22:05:36.436 DEBUG:boto:path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.436 DEBUG:boto:auth_path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.436 DEBUG:boto:Method: HEAD 2023-09-22T22:05:36.436 DEBUG:boto:Path: /ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.436 DEBUG:boto:Data: 2023-09-22T22:05:36.436 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.436 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.436 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.436 DEBUG:boto:Params: {} 2023-09-22T22:05:36.437 DEBUG:boto:Token: None 2023-09-22T22:05:36.437 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.437 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2Lz8XfQ7iVz2YjcPc7hdSq3HDfo= 2023-09-22T22:05:36.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2Lz8XfQ7iVz2YjcPc7hdSq3HDfo=', 'Content-Length': '0'} 2023-09-22T22:05:36.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063814989ec024c8d-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T22:05:36.441 DEBUG:boto:path=/ffzdqu-63/obj11 2023-09-22T22:05:36.441 DEBUG:boto:auth_path=/ffzdqu-63/obj11 2023-09-22T22:05:36.441 DEBUG:boto:path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.441 DEBUG:boto:auth_path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.441 DEBUG:boto:Method: GET 2023-09-22T22:05:36.442 DEBUG:boto:Path: /ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.442 DEBUG:boto:Data: 2023-09-22T22:05:36.442 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.442 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.442 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.442 DEBUG:boto:Params: {} 2023-09-22T22:05:36.442 DEBUG:boto:Token: None 2023-09-22T22:05:36.442 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.443 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gK6tup572M7PH0hYpjVUVvYT3p4= 2023-09-22T22:05:36.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gK6tup572M7PH0hYpjVUVvYT3p4=', 'Content-Length': '0'} 2023-09-22T22:05:36.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a2e033f9cf70d01-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.486 DEBUG:boto:path=/ffzdqu-63/obj11 2023-09-22T22:05:36.486 DEBUG:boto:auth_path=/ffzdqu-63/obj11 2023-09-22T22:05:36.487 DEBUG:boto:path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.487 DEBUG:boto:auth_path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.487 DEBUG:boto:Method: GET 2023-09-22T22:05:36.487 DEBUG:boto:Path: /ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.487 DEBUG:boto:Data: 2023-09-22T22:05:36.487 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.487 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.487 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.488 DEBUG:boto:Params: {} 2023-09-22T22:05:36.488 DEBUG:boto:Token: None 2023-09-22T22:05:36.488 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj11?versionId=null 2023-09-22T22:05:36.488 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gK6tup572M7PH0hYpjVUVvYT3p4= 2023-09-22T22:05:36.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gK6tup572M7PH0hYpjVUVvYT3p4=', 'Content-Length': '0'} 2023-09-22T22:05:36.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e61fc05ec4a4c8c-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.491 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:36.492 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:36.492 DEBUG:boto:path=/ffzdqu-63/?prefix=obj11 2023-09-22T22:05:36.492 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj11 2023-09-22T22:05:36.492 DEBUG:boto:Method: GET 2023-09-22T22:05:36.492 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj11 2023-09-22T22:05:36.492 DEBUG:boto:Data: 2023-09-22T22:05:36.492 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.492 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.493 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.493 DEBUG:boto:Params: {} 2023-09-22T22:05:36.493 DEBUG:boto:Token: None 2023-09-22T22:05:36.493 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/ 2023-09-22T22:05:36.493 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:05:36.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:05:36.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060480d56b3a8c981-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.498 DEBUG:boto:b'ffzdqu-63obj111000falseobj112023-09-22T22:05:13.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:36.498 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:36.499 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:36.499 DEBUG:boto:path=/ffzdqu-63/?prefix=obj11 2023-09-22T22:05:36.499 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj11 2023-09-22T22:05:36.499 DEBUG:boto:Method: GET 2023-09-22T22:05:36.499 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj11 2023-09-22T22:05:36.499 DEBUG:boto:Data: 2023-09-22T22:05:36.499 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.499 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.500 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.500 DEBUG:boto:Params: {} 2023-09-22T22:05:36.500 DEBUG:boto:Token: None 2023-09-22T22:05:36.500 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/ 2023-09-22T22:05:36.500 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:05:36.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:05:36.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006137252ed1f936e6-00650e0fb0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.510 DEBUG:boto:b'ffzdqu-63obj111000falseobj112023-09-22T22:05:13.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:36.510 DEBUG:boto:path=/ffzdqu-63/obj11 2023-09-22T22:05:36.510 DEBUG:boto:auth_path=/ffzdqu-63/obj11 2023-09-22T22:05:36.511 DEBUG:boto:Method: HEAD 2023-09-22T22:05:36.511 DEBUG:boto:Path: /ffzdqu-63/obj11 2023-09-22T22:05:36.511 DEBUG:boto:Data: 2023-09-22T22:05:36.511 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.511 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.511 DEBUG:boto:Params: {} 2023-09-22T22:05:36.511 DEBUG:boto:Token: None 2023-09-22T22:05:36.512 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj11 2023-09-22T22:05:36.512 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KHeCs1xe8XbuG/JRm7kTYcqUOP4= 2023-09-22T22:05:36.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KHeCs1xe8XbuG/JRm7kTYcqUOP4=', 'Content-Length': '0'} 2023-09-22T22:05:36.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d36fd6f5d9576e7a-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.515 DEBUG:boto:path=/ffzdqu-63/obj11 2023-09-22T22:05:36.515 DEBUG:boto:auth_path=/ffzdqu-63/obj11 2023-09-22T22:05:36.515 DEBUG:boto:Method: HEAD 2023-09-22T22:05:36.515 DEBUG:boto:Path: /ffzdqu-63/obj11 2023-09-22T22:05:36.516 DEBUG:boto:Data: 2023-09-22T22:05:36.516 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.516 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.516 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.516 DEBUG:boto:Params: {} 2023-09-22T22:05:36.516 DEBUG:boto:Token: None 2023-09-22T22:05:36.517 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj11 2023-09-22T22:05:36.517 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:KHeCs1xe8XbuG/JRm7kTYcqUOP4= 2023-09-22T22:05:36.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KHeCs1xe8XbuG/JRm7kTYcqUOP4=', 'Content-Length': '0'} 2023-09-22T22:05:36.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000887eb3f3cab7f66f-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T22:05:36.520 DEBUG:boto:path=/ffzdqu-63/obj11 2023-09-22T22:05:36.520 DEBUG:boto:auth_path=/ffzdqu-63/obj11 2023-09-22T22:05:36.521 DEBUG:boto:Method: GET 2023-09-22T22:05:36.521 DEBUG:boto:Path: /ffzdqu-63/obj11 2023-09-22T22:05:36.521 DEBUG:boto:Data: 2023-09-22T22:05:36.521 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.521 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.521 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.521 DEBUG:boto:Params: {} 2023-09-22T22:05:36.521 DEBUG:boto:Token: None 2023-09-22T22:05:36.522 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj11 2023-09-22T22:05:36.522 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:09IrybW+1pJkZWTqnf6rYKBAfbQ= 2023-09-22T22:05:36.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:09IrybW+1pJkZWTqnf6rYKBAfbQ=', 'Content-Length': '0'} 2023-09-22T22:05:36.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc62df9cad051adf-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.570 DEBUG:boto:path=/ffzdqu-63/obj11 2023-09-22T22:05:36.571 DEBUG:boto:auth_path=/ffzdqu-63/obj11 2023-09-22T22:05:36.571 DEBUG:boto:Method: GET 2023-09-22T22:05:36.571 DEBUG:boto:Path: /ffzdqu-63/obj11 2023-09-22T22:05:36.571 DEBUG:boto:Data: 2023-09-22T22:05:36.571 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.572 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.572 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.572 DEBUG:boto:Params: {} 2023-09-22T22:05:36.572 DEBUG:boto:Token: None 2023-09-22T22:05:36.573 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj11 2023-09-22T22:05:36.573 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:09IrybW+1pJkZWTqnf6rYKBAfbQ= 2023-09-22T22:05:36.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:09IrybW+1pJkZWTqnf6rYKBAfbQ=', 'Content-Length': '0'} 2023-09-22T22:05:36.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006838e2b63c765dc1-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.577 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:36.577 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:36.577 DEBUG:boto:path=/ffzdqu-63/?prefix=obj11 2023-09-22T22:05:36.577 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj11 2023-09-22T22:05:36.577 DEBUG:boto:Method: GET 2023-09-22T22:05:36.577 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj11 2023-09-22T22:05:36.578 DEBUG:boto:Data: 2023-09-22T22:05:36.578 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.578 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.578 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.578 DEBUG:boto:Params: {} 2023-09-22T22:05:36.579 DEBUG:boto:Token: None 2023-09-22T22:05:36.579 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/ 2023-09-22T22:05:36.579 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:05:36.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:05:36.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc9789d92f5b4612-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.584 DEBUG:boto:b'ffzdqu-63obj111000falseobj112023-09-22T22:05:13.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:36.585 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:36.585 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:36.585 DEBUG:boto:path=/ffzdqu-63/?prefix=obj11 2023-09-22T22:05:36.585 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj11 2023-09-22T22:05:36.585 DEBUG:boto:Method: GET 2023-09-22T22:05:36.585 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj11 2023-09-22T22:05:36.586 DEBUG:boto:Data: 2023-09-22T22:05:36.586 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.586 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.586 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.586 DEBUG:boto:Params: {} 2023-09-22T22:05:36.587 DEBUG:boto:Token: None 2023-09-22T22:05:36.587 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/ 2023-09-22T22:05:36.587 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:05:36.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:05:36.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000832e403a963ed282-00650e0fb0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.596 DEBUG:boto:b'ffzdqu-63obj111000falseobj112023-09-22T22:05:13.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:36.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-09-22T22:05:36.597 DEBUG:boto:path=/ffzdqu-63/obj111 2023-09-22T22:05:36.597 DEBUG:boto:auth_path=/ffzdqu-63/obj111 2023-09-22T22:05:36.597 DEBUG:boto:path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.598 DEBUG:boto:auth_path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.598 DEBUG:boto:Method: GET 2023-09-22T22:05:36.598 DEBUG:boto:Path: /ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.598 DEBUG:boto:Data: 2023-09-22T22:05:36.598 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.599 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.599 DEBUG:boto:Params: {} 2023-09-22T22:05:36.599 DEBUG:boto:Token: None 2023-09-22T22:05:36.599 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.600 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rf6f7mlJjp4Txk+CGqyquLg2aFE= 2023-09-22T22:05:36.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rf6f7mlJjp4Txk+CGqyquLg2aFE=', 'Content-Length': '0'} 2023-09-22T22:05:36.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f39031c3b12938fa-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.604 DEBUG:boto:path=/ffzdqu-63/obj111 2023-09-22T22:05:36.604 DEBUG:boto:auth_path=/ffzdqu-63/obj111 2023-09-22T22:05:36.604 DEBUG:boto:path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.604 DEBUG:boto:auth_path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.604 DEBUG:boto:Method: GET 2023-09-22T22:05:36.604 DEBUG:boto:Path: /ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.604 DEBUG:boto:Data: 2023-09-22T22:05:36.605 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.605 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.605 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.605 DEBUG:boto:Params: {} 2023-09-22T22:05:36.605 DEBUG:boto:Token: None 2023-09-22T22:05:36.605 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.605 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rf6f7mlJjp4Txk+CGqyquLg2aFE= 2023-09-22T22:05:36.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rf6f7mlJjp4Txk+CGqyquLg2aFE=', 'Content-Length': '0'} 2023-09-22T22:05:36.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007bc36bd8c6b14dc-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.608 DEBUG:boto:path=/ffzdqu-63/obj111 2023-09-22T22:05:36.609 DEBUG:boto:auth_path=/ffzdqu-63/obj111 2023-09-22T22:05:36.609 DEBUG:boto:path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.609 DEBUG:boto:auth_path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.609 DEBUG:boto:Method: HEAD 2023-09-22T22:05:36.609 DEBUG:boto:Path: /ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.609 DEBUG:boto:Data: 2023-09-22T22:05:36.609 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.609 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.609 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.609 DEBUG:boto:Params: {} 2023-09-22T22:05:36.610 DEBUG:boto:Token: None 2023-09-22T22:05:36.610 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.610 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0nkDBBO1MT4EbIS/IaRKV8sonRY= 2023-09-22T22:05:36.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0nkDBBO1MT4EbIS/IaRKV8sonRY=', 'Content-Length': '0'} 2023-09-22T22:05:36.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006fe431921d5790c-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.613 DEBUG:boto:path=/ffzdqu-63/obj111 2023-09-22T22:05:36.613 DEBUG:boto:auth_path=/ffzdqu-63/obj111 2023-09-22T22:05:36.613 DEBUG:boto:path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.613 DEBUG:boto:auth_path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.613 DEBUG:boto:Method: HEAD 2023-09-22T22:05:36.613 DEBUG:boto:Path: /ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.614 DEBUG:boto:Data: 2023-09-22T22:05:36.614 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.614 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.614 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.614 DEBUG:boto:Params: {} 2023-09-22T22:05:36.614 DEBUG:boto:Token: None 2023-09-22T22:05:36.614 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.614 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0nkDBBO1MT4EbIS/IaRKV8sonRY= 2023-09-22T22:05:36.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0nkDBBO1MT4EbIS/IaRKV8sonRY=', 'Content-Length': '0'} 2023-09-22T22:05:36.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bb8cac9933267aa-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-09-22T22:05:36.618 DEBUG:boto:path=/ffzdqu-63/obj111 2023-09-22T22:05:36.618 DEBUG:boto:auth_path=/ffzdqu-63/obj111 2023-09-22T22:05:36.618 DEBUG:boto:path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.618 DEBUG:boto:auth_path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.618 DEBUG:boto:Method: GET 2023-09-22T22:05:36.618 DEBUG:boto:Path: /ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.619 DEBUG:boto:Data: 2023-09-22T22:05:36.619 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.619 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.619 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.619 DEBUG:boto:Params: {} 2023-09-22T22:05:36.619 DEBUG:boto:Token: None 2023-09-22T22:05:36.619 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.619 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rf6f7mlJjp4Txk+CGqyquLg2aFE= 2023-09-22T22:05:36.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rf6f7mlJjp4Txk+CGqyquLg2aFE=', 'Content-Length': '0'} 2023-09-22T22:05:36.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd245a3cc6b21bd5-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.670 DEBUG:boto:path=/ffzdqu-63/obj111 2023-09-22T22:05:36.671 DEBUG:boto:auth_path=/ffzdqu-63/obj111 2023-09-22T22:05:36.671 DEBUG:boto:path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.671 DEBUG:boto:auth_path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.671 DEBUG:boto:Method: GET 2023-09-22T22:05:36.671 DEBUG:boto:Path: /ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.672 DEBUG:boto:Data: 2023-09-22T22:05:36.672 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.672 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.672 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.672 DEBUG:boto:Params: {} 2023-09-22T22:05:36.672 DEBUG:boto:Token: None 2023-09-22T22:05:36.673 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj111?versionId=null 2023-09-22T22:05:36.673 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rf6f7mlJjp4Txk+CGqyquLg2aFE= 2023-09-22T22:05:36.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rf6f7mlJjp4Txk+CGqyquLg2aFE=', 'Content-Length': '0'} 2023-09-22T22:05:36.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b581b5f7d7c32fd-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.677 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:36.677 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:36.677 DEBUG:boto:path=/ffzdqu-63/?prefix=obj111 2023-09-22T22:05:36.677 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj111 2023-09-22T22:05:36.677 DEBUG:boto:Method: GET 2023-09-22T22:05:36.678 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj111 2023-09-22T22:05:36.678 DEBUG:boto:Data: 2023-09-22T22:05:36.678 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.678 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.678 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.678 DEBUG:boto:Params: {} 2023-09-22T22:05:36.679 DEBUG:boto:Token: None 2023-09-22T22:05:36.679 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/ 2023-09-22T22:05:36.679 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:05:36.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:05:36.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000312fb36ea09c1f30-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.685 DEBUG:boto:b'ffzdqu-63obj1111000falseobj1112023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:36.685 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:36.686 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:36.686 DEBUG:boto:path=/ffzdqu-63/?prefix=obj111 2023-09-22T22:05:36.686 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj111 2023-09-22T22:05:36.686 DEBUG:boto:Method: GET 2023-09-22T22:05:36.686 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj111 2023-09-22T22:05:36.687 DEBUG:boto:Data: 2023-09-22T22:05:36.687 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.687 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.687 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.687 DEBUG:boto:Params: {} 2023-09-22T22:05:36.687 DEBUG:boto:Token: None 2023-09-22T22:05:36.688 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/ 2023-09-22T22:05:36.688 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:05:36.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:05:36.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d201db9727eb268c-00650e0fb0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.698 DEBUG:boto:b'ffzdqu-63obj1111000falseobj1112023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:36.699 DEBUG:boto:path=/ffzdqu-63/obj111 2023-09-22T22:05:36.699 DEBUG:boto:auth_path=/ffzdqu-63/obj111 2023-09-22T22:05:36.699 DEBUG:boto:Method: HEAD 2023-09-22T22:05:36.699 DEBUG:boto:Path: /ffzdqu-63/obj111 2023-09-22T22:05:36.699 DEBUG:boto:Data: 2023-09-22T22:05:36.700 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.700 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.700 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.700 DEBUG:boto:Params: {} 2023-09-22T22:05:36.700 DEBUG:boto:Token: None 2023-09-22T22:05:36.701 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj111 2023-09-22T22:05:36.701 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:z0a/mZkGiedAfDJkcNmi99Zhlac= 2023-09-22T22:05:36.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:z0a/mZkGiedAfDJkcNmi99Zhlac=', 'Content-Length': '0'} 2023-09-22T22:05:36.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0e428a662c4bcdb-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.704 DEBUG:boto:path=/ffzdqu-63/obj111 2023-09-22T22:05:36.704 DEBUG:boto:auth_path=/ffzdqu-63/obj111 2023-09-22T22:05:36.705 DEBUG:boto:Method: HEAD 2023-09-22T22:05:36.705 DEBUG:boto:Path: /ffzdqu-63/obj111 2023-09-22T22:05:36.705 DEBUG:boto:Data: 2023-09-22T22:05:36.705 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.705 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.706 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.706 DEBUG:boto:Params: {} 2023-09-22T22:05:36.706 DEBUG:boto:Token: None 2023-09-22T22:05:36.706 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj111 2023-09-22T22:05:36.707 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:z0a/mZkGiedAfDJkcNmi99Zhlac= 2023-09-22T22:05:36.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:z0a/mZkGiedAfDJkcNmi99Zhlac=', 'Content-Length': '0'} 2023-09-22T22:05:36.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bb1a73c69efb1a3-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-09-22T22:05:36.710 DEBUG:boto:path=/ffzdqu-63/obj111 2023-09-22T22:05:36.710 DEBUG:boto:auth_path=/ffzdqu-63/obj111 2023-09-22T22:05:36.711 DEBUG:boto:Method: GET 2023-09-22T22:05:36.711 DEBUG:boto:Path: /ffzdqu-63/obj111 2023-09-22T22:05:36.711 DEBUG:boto:Data: 2023-09-22T22:05:36.711 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.711 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.712 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.712 DEBUG:boto:Params: {} 2023-09-22T22:05:36.712 DEBUG:boto:Token: None 2023-09-22T22:05:36.712 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj111 2023-09-22T22:05:36.713 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LSISURKMbShRzKpzgeowsJPP900= 2023-09-22T22:05:36.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LSISURKMbShRzKpzgeowsJPP900=', 'Content-Length': '0'} 2023-09-22T22:05:36.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000534929e4039723ee-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.758 DEBUG:boto:path=/ffzdqu-63/obj111 2023-09-22T22:05:36.759 DEBUG:boto:auth_path=/ffzdqu-63/obj111 2023-09-22T22:05:36.759 DEBUG:boto:Method: GET 2023-09-22T22:05:36.759 DEBUG:boto:Path: /ffzdqu-63/obj111 2023-09-22T22:05:36.759 DEBUG:boto:Data: 2023-09-22T22:05:36.759 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.760 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.760 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.760 DEBUG:boto:Params: {} 2023-09-22T22:05:36.760 DEBUG:boto:Token: None 2023-09-22T22:05:36.760 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj111 2023-09-22T22:05:36.761 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LSISURKMbShRzKpzgeowsJPP900= 2023-09-22T22:05:36.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LSISURKMbShRzKpzgeowsJPP900=', 'Content-Length': '0'} 2023-09-22T22:05:36.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a38c933f35fe8228-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.764 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:36.765 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:36.765 DEBUG:boto:path=/ffzdqu-63/?prefix=obj111 2023-09-22T22:05:36.765 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj111 2023-09-22T22:05:36.765 DEBUG:boto:Method: GET 2023-09-22T22:05:36.765 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj111 2023-09-22T22:05:36.765 DEBUG:boto:Data: 2023-09-22T22:05:36.765 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.765 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.765 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.766 DEBUG:boto:Params: {} 2023-09-22T22:05:36.766 DEBUG:boto:Token: None 2023-09-22T22:05:36.766 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/ 2023-09-22T22:05:36.766 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:05:36.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:05:36.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab5b478bbe01cdc7-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.771 DEBUG:boto:b'ffzdqu-63obj1111000falseobj1112023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:36.772 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:36.772 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:36.772 DEBUG:boto:path=/ffzdqu-63/?prefix=obj111 2023-09-22T22:05:36.772 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj111 2023-09-22T22:05:36.772 DEBUG:boto:Method: GET 2023-09-22T22:05:36.772 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj111 2023-09-22T22:05:36.772 DEBUG:boto:Data: 2023-09-22T22:05:36.773 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.773 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.773 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.773 DEBUG:boto:Params: {} 2023-09-22T22:05:36.773 DEBUG:boto:Token: None 2023-09-22T22:05:36.773 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/ 2023-09-22T22:05:36.773 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:05:36.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:05:36.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000646fb577b2284a4e-00650e0fb0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.822 DEBUG:boto:b'ffzdqu-63obj1111000falseobj1112023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:36.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-22T22:05:36.823 DEBUG:boto:path=/ffzdqu-63/obj115 2023-09-22T22:05:36.823 DEBUG:boto:auth_path=/ffzdqu-63/obj115 2023-09-22T22:05:36.823 DEBUG:boto:path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.823 DEBUG:boto:auth_path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.823 DEBUG:boto:Method: GET 2023-09-22T22:05:36.823 DEBUG:boto:Path: /ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.823 DEBUG:boto:Data: 2023-09-22T22:05:36.824 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.824 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.824 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.824 DEBUG:boto:Params: {} 2023-09-22T22:05:36.824 DEBUG:boto:Token: None 2023-09-22T22:05:36.824 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.824 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bAtWTfki/8437idRxzK0O7aHA20= 2023-09-22T22:05:36.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bAtWTfki/8437idRxzK0O7aHA20=', 'Content-Length': '0'} 2023-09-22T22:05:36.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dbb1a219675b672-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.827 DEBUG:boto:path=/ffzdqu-63/obj115 2023-09-22T22:05:36.828 DEBUG:boto:auth_path=/ffzdqu-63/obj115 2023-09-22T22:05:36.828 DEBUG:boto:path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.828 DEBUG:boto:auth_path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.828 DEBUG:boto:Method: GET 2023-09-22T22:05:36.828 DEBUG:boto:Path: /ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.828 DEBUG:boto:Data: 2023-09-22T22:05:36.828 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.828 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.828 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.828 DEBUG:boto:Params: {} 2023-09-22T22:05:36.829 DEBUG:boto:Token: None 2023-09-22T22:05:36.829 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.829 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bAtWTfki/8437idRxzK0O7aHA20= 2023-09-22T22:05:36.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bAtWTfki/8437idRxzK0O7aHA20=', 'Content-Length': '0'} 2023-09-22T22:05:36.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d5e936ba2c8aa0f-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.832 DEBUG:boto:path=/ffzdqu-63/obj115 2023-09-22T22:05:36.832 DEBUG:boto:auth_path=/ffzdqu-63/obj115 2023-09-22T22:05:36.832 DEBUG:boto:path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.832 DEBUG:boto:auth_path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.832 DEBUG:boto:Method: HEAD 2023-09-22T22:05:36.832 DEBUG:boto:Path: /ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.832 DEBUG:boto:Data: 2023-09-22T22:05:36.832 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.833 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.833 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.833 DEBUG:boto:Params: {} 2023-09-22T22:05:36.833 DEBUG:boto:Token: None 2023-09-22T22:05:36.833 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.833 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EbiCdk54o041V0Sn0jb2Rhz/yKI= 2023-09-22T22:05:36.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EbiCdk54o041V0Sn0jb2Rhz/yKI=', 'Content-Length': '0'} 2023-09-22T22:05:36.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004067b6ca30857d0c-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.836 DEBUG:boto:path=/ffzdqu-63/obj115 2023-09-22T22:05:36.836 DEBUG:boto:auth_path=/ffzdqu-63/obj115 2023-09-22T22:05:36.836 DEBUG:boto:path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.836 DEBUG:boto:auth_path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.837 DEBUG:boto:Method: HEAD 2023-09-22T22:05:36.837 DEBUG:boto:Path: /ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.837 DEBUG:boto:Data: 2023-09-22T22:05:36.837 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.837 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.837 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.837 DEBUG:boto:Params: {} 2023-09-22T22:05:36.837 DEBUG:boto:Token: None 2023-09-22T22:05:36.837 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.838 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EbiCdk54o041V0Sn0jb2Rhz/yKI= 2023-09-22T22:05:36.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EbiCdk54o041V0Sn0jb2Rhz/yKI=', 'Content-Length': '0'} 2023-09-22T22:05:36.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1612045599955aa-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-22T22:05:36.840 DEBUG:boto:path=/ffzdqu-63/obj115 2023-09-22T22:05:36.841 DEBUG:boto:auth_path=/ffzdqu-63/obj115 2023-09-22T22:05:36.841 DEBUG:boto:path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.841 DEBUG:boto:auth_path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.841 DEBUG:boto:Method: GET 2023-09-22T22:05:36.841 DEBUG:boto:Path: /ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.841 DEBUG:boto:Data: 2023-09-22T22:05:36.841 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.841 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.841 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.841 DEBUG:boto:Params: {} 2023-09-22T22:05:36.842 DEBUG:boto:Token: None 2023-09-22T22:05:36.842 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.842 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bAtWTfki/8437idRxzK0O7aHA20= 2023-09-22T22:05:36.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bAtWTfki/8437idRxzK0O7aHA20=', 'Content-Length': '0'} 2023-09-22T22:05:36.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023b7aded51193570-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.886 DEBUG:boto:path=/ffzdqu-63/obj115 2023-09-22T22:05:36.886 DEBUG:boto:auth_path=/ffzdqu-63/obj115 2023-09-22T22:05:36.886 DEBUG:boto:path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.887 DEBUG:boto:auth_path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.887 DEBUG:boto:Method: GET 2023-09-22T22:05:36.887 DEBUG:boto:Path: /ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.887 DEBUG:boto:Data: 2023-09-22T22:05:36.887 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.887 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.887 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.887 DEBUG:boto:Params: {} 2023-09-22T22:05:36.887 DEBUG:boto:Token: None 2023-09-22T22:05:36.888 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj115?versionId=null 2023-09-22T22:05:36.888 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bAtWTfki/8437idRxzK0O7aHA20= 2023-09-22T22:05:36.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bAtWTfki/8437idRxzK0O7aHA20=', 'Content-Length': '0'} 2023-09-22T22:05:36.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9a2e34fe36b66e9-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.891 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:36.891 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:36.891 DEBUG:boto:path=/ffzdqu-63/?prefix=obj115 2023-09-22T22:05:36.891 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj115 2023-09-22T22:05:36.891 DEBUG:boto:Method: GET 2023-09-22T22:05:36.891 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj115 2023-09-22T22:05:36.892 DEBUG:boto:Data: 2023-09-22T22:05:36.892 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.892 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.892 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.892 DEBUG:boto:Params: {} 2023-09-22T22:05:36.892 DEBUG:boto:Token: None 2023-09-22T22:05:36.892 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/ 2023-09-22T22:05:36.892 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:05:36.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:05:36.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058493da9ffbb324b-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.898 DEBUG:boto:b'ffzdqu-63obj1151000falseobj1152023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:36.898 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:36.898 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:36.899 DEBUG:boto:path=/ffzdqu-63/?prefix=obj115 2023-09-22T22:05:36.899 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj115 2023-09-22T22:05:36.899 DEBUG:boto:Method: GET 2023-09-22T22:05:36.899 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj115 2023-09-22T22:05:36.899 DEBUG:boto:Data: 2023-09-22T22:05:36.899 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.899 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.899 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.899 DEBUG:boto:Params: {} 2023-09-22T22:05:36.899 DEBUG:boto:Token: None 2023-09-22T22:05:36.900 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/ 2023-09-22T22:05:36.900 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:05:36.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:05:36.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2e9dfe7853187d6-00650e0fb0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.950 DEBUG:boto:b'ffzdqu-63obj1151000falseobj1152023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:36.951 DEBUG:boto:path=/ffzdqu-63/obj115 2023-09-22T22:05:36.951 DEBUG:boto:auth_path=/ffzdqu-63/obj115 2023-09-22T22:05:36.951 DEBUG:boto:Method: HEAD 2023-09-22T22:05:36.951 DEBUG:boto:Path: /ffzdqu-63/obj115 2023-09-22T22:05:36.951 DEBUG:boto:Data: 2023-09-22T22:05:36.951 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.952 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.952 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.952 DEBUG:boto:Params: {} 2023-09-22T22:05:36.952 DEBUG:boto:Token: None 2023-09-22T22:05:36.952 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj115 2023-09-22T22:05:36.952 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sz80IZbeOjCbhlSRMQcTgciJEo0= 2023-09-22T22:05:36.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sz80IZbeOjCbhlSRMQcTgciJEo0=', 'Content-Length': '0'} 2023-09-22T22:05:36.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef1e227c0a5f3340-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.955 DEBUG:boto:path=/ffzdqu-63/obj115 2023-09-22T22:05:36.956 DEBUG:boto:auth_path=/ffzdqu-63/obj115 2023-09-22T22:05:36.956 DEBUG:boto:Method: HEAD 2023-09-22T22:05:36.956 DEBUG:boto:Path: /ffzdqu-63/obj115 2023-09-22T22:05:36.956 DEBUG:boto:Data: 2023-09-22T22:05:36.956 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.956 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.956 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.956 DEBUG:boto:Params: {} 2023-09-22T22:05:36.956 DEBUG:boto:Token: None 2023-09-22T22:05:36.957 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj115 2023-09-22T22:05:36.957 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sz80IZbeOjCbhlSRMQcTgciJEo0= 2023-09-22T22:05:36.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sz80IZbeOjCbhlSRMQcTgciJEo0=', 'Content-Length': '0'} 2023-09-22T22:05:36.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ba44dba5145a7e2-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-22T22:05:36.960 DEBUG:boto:path=/ffzdqu-63/obj115 2023-09-22T22:05:36.960 DEBUG:boto:auth_path=/ffzdqu-63/obj115 2023-09-22T22:05:36.961 DEBUG:boto:Method: GET 2023-09-22T22:05:36.961 DEBUG:boto:Path: /ffzdqu-63/obj115 2023-09-22T22:05:36.961 DEBUG:boto:Data: 2023-09-22T22:05:36.961 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.961 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.961 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.961 DEBUG:boto:Params: {} 2023-09-22T22:05:36.961 DEBUG:boto:Token: None 2023-09-22T22:05:36.962 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj115 2023-09-22T22:05:36.962 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qXk8SOzcDL/NA19AdctX3W0Le98= 2023-09-22T22:05:36.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qXk8SOzcDL/NA19AdctX3W0Le98=', 'Content-Length': '0'} 2023-09-22T22:05:36.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039f10b229860dfd9-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.965 DEBUG:boto:path=/ffzdqu-63/obj115 2023-09-22T22:05:36.965 DEBUG:boto:auth_path=/ffzdqu-63/obj115 2023-09-22T22:05:36.965 DEBUG:boto:Method: GET 2023-09-22T22:05:36.965 DEBUG:boto:Path: /ffzdqu-63/obj115 2023-09-22T22:05:36.965 DEBUG:boto:Data: 2023-09-22T22:05:36.965 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.966 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.966 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.966 DEBUG:boto:Params: {} 2023-09-22T22:05:36.966 DEBUG:boto:Token: None 2023-09-22T22:05:36.966 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj115 2023-09-22T22:05:36.966 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qXk8SOzcDL/NA19AdctX3W0Le98= 2023-09-22T22:05:36.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qXk8SOzcDL/NA19AdctX3W0Le98=', 'Content-Length': '0'} 2023-09-22T22:05:36.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c35f51444026a28b-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.969 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:36.969 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:36.970 DEBUG:boto:path=/ffzdqu-63/?prefix=obj115 2023-09-22T22:05:36.970 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj115 2023-09-22T22:05:36.970 DEBUG:boto:Method: GET 2023-09-22T22:05:36.970 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj115 2023-09-22T22:05:36.970 DEBUG:boto:Data: 2023-09-22T22:05:36.970 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.970 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.970 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.971 DEBUG:boto:Params: {} 2023-09-22T22:05:36.971 DEBUG:boto:Token: None 2023-09-22T22:05:36.971 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/ 2023-09-22T22:05:36.971 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:05:36.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:05:36.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4cd6b7da165b58b-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.978 DEBUG:boto:b'ffzdqu-63obj1151000falseobj1152023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:36.978 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:36.978 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:36.978 DEBUG:boto:path=/ffzdqu-63/?prefix=obj115 2023-09-22T22:05:36.979 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj115 2023-09-22T22:05:36.979 DEBUG:boto:Method: GET 2023-09-22T22:05:36.979 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj115 2023-09-22T22:05:36.979 DEBUG:boto:Data: 2023-09-22T22:05:36.979 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.979 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:36.979 DEBUG:boto:Port: 8000 2023-09-22T22:05:36.979 DEBUG:boto:Params: {} 2023-09-22T22:05:36.979 DEBUG:boto:Token: None 2023-09-22T22:05:36.979 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/ 2023-09-22T22:05:36.980 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:05:36.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:05:36.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004d89ba1be40e7cd-00650e0fb0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:36.990 DEBUG:boto:b'ffzdqu-63obj1151000falseobj1152023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:36.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-22T22:05:36.990 DEBUG:boto:path=/ffzdqu-63/obj116 2023-09-22T22:05:36.990 DEBUG:boto:auth_path=/ffzdqu-63/obj116 2023-09-22T22:05:36.991 DEBUG:boto:path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:36.991 DEBUG:boto:auth_path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:36.991 DEBUG:boto:Method: GET 2023-09-22T22:05:36.991 DEBUG:boto:Path: /ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:36.991 DEBUG:boto:Data: 2023-09-22T22:05:36.991 DEBUG:boto:Headers: {} 2023-09-22T22:05:36.991 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:36.991 DEBUG:boto:Port: 8001 2023-09-22T22:05:36.991 DEBUG:boto:Params: {} 2023-09-22T22:05:36.991 DEBUG:boto:Token: None 2023-09-22T22:05:36.992 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:36 GMT /ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:36.992 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DgPoQzSTMvcsJnb95MaKCSnRZOk= 2023-09-22T22:05:36.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DgPoQzSTMvcsJnb95MaKCSnRZOk=', 'Content-Length': '0'} 2023-09-22T22:05:36.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5fc3a0d8ca16e71-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.038 DEBUG:boto:path=/ffzdqu-63/obj116 2023-09-22T22:05:37.038 DEBUG:boto:auth_path=/ffzdqu-63/obj116 2023-09-22T22:05:37.039 DEBUG:boto:path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:37.039 DEBUG:boto:auth_path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:37.039 DEBUG:boto:Method: GET 2023-09-22T22:05:37.039 DEBUG:boto:Path: /ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:37.039 DEBUG:boto:Data: 2023-09-22T22:05:37.039 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.039 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.039 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.039 DEBUG:boto:Params: {} 2023-09-22T22:05:37.040 DEBUG:boto:Token: None 2023-09-22T22:05:37.040 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:37.040 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ywoI8NscNvthzFZ95UP+gtA+u/4= 2023-09-22T22:05:37.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ywoI8NscNvthzFZ95UP+gtA+u/4=', 'Content-Length': '0'} 2023-09-22T22:05:37.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ebcc4955b29a47b-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.043 DEBUG:boto:path=/ffzdqu-63/obj116 2023-09-22T22:05:37.043 DEBUG:boto:auth_path=/ffzdqu-63/obj116 2023-09-22T22:05:37.043 DEBUG:boto:path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:37.044 DEBUG:boto:auth_path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:37.044 DEBUG:boto:Method: HEAD 2023-09-22T22:05:37.044 DEBUG:boto:Path: /ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:37.044 DEBUG:boto:Data: 2023-09-22T22:05:37.044 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.044 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.044 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.044 DEBUG:boto:Params: {} 2023-09-22T22:05:37.044 DEBUG:boto:Token: None 2023-09-22T22:05:37.045 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:37.045 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ItjSYSfu6Ro5dVzLCkfK7ZAjJI8= 2023-09-22T22:05:37.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ItjSYSfu6Ro5dVzLCkfK7ZAjJI8=', 'Content-Length': '0'} 2023-09-22T22:05:37.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078fca83f4322e533-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.048 DEBUG:boto:path=/ffzdqu-63/obj116 2023-09-22T22:05:37.048 DEBUG:boto:auth_path=/ffzdqu-63/obj116 2023-09-22T22:05:37.048 DEBUG:boto:path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:37.048 DEBUG:boto:auth_path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:37.048 DEBUG:boto:Method: HEAD 2023-09-22T22:05:37.049 DEBUG:boto:Path: /ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:37.049 DEBUG:boto:Data: 2023-09-22T22:05:37.049 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.049 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.049 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.049 DEBUG:boto:Params: {} 2023-09-22T22:05:37.049 DEBUG:boto:Token: None 2023-09-22T22:05:37.049 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:37.049 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ItjSYSfu6Ro5dVzLCkfK7ZAjJI8= 2023-09-22T22:05:37.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ItjSYSfu6Ro5dVzLCkfK7ZAjJI8=', 'Content-Length': '0'} 2023-09-22T22:05:37.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000249cfb996a20ae13-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-22T22:05:37.053 DEBUG:boto:path=/ffzdqu-63/obj116 2023-09-22T22:05:37.053 DEBUG:boto:auth_path=/ffzdqu-63/obj116 2023-09-22T22:05:37.053 DEBUG:boto:path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:37.053 DEBUG:boto:auth_path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:37.053 DEBUG:boto:Method: GET 2023-09-22T22:05:37.053 DEBUG:boto:Path: /ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:37.054 DEBUG:boto:Data: 2023-09-22T22:05:37.054 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.054 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.054 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.054 DEBUG:boto:Params: {} 2023-09-22T22:05:37.054 DEBUG:boto:Token: None 2023-09-22T22:05:37.054 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:37.054 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ywoI8NscNvthzFZ95UP+gtA+u/4= 2023-09-22T22:05:37.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ywoI8NscNvthzFZ95UP+gtA+u/4=', 'Content-Length': '0'} 2023-09-22T22:05:37.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd99a9d1f0545c5f-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.057 DEBUG:boto:path=/ffzdqu-63/obj116 2023-09-22T22:05:37.058 DEBUG:boto:auth_path=/ffzdqu-63/obj116 2023-09-22T22:05:37.058 DEBUG:boto:path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:37.058 DEBUG:boto:auth_path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:37.058 DEBUG:boto:Method: GET 2023-09-22T22:05:37.058 DEBUG:boto:Path: /ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:37.058 DEBUG:boto:Data: 2023-09-22T22:05:37.058 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.058 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.058 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.059 DEBUG:boto:Params: {} 2023-09-22T22:05:37.059 DEBUG:boto:Token: None 2023-09-22T22:05:37.059 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj116?versionId=null 2023-09-22T22:05:37.059 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ywoI8NscNvthzFZ95UP+gtA+u/4= 2023-09-22T22:05:37.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ywoI8NscNvthzFZ95UP+gtA+u/4=', 'Content-Length': '0'} 2023-09-22T22:05:37.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042d8c148bd91439e-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.106 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:37.107 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:37.107 DEBUG:boto:path=/ffzdqu-63/?prefix=obj116 2023-09-22T22:05:37.107 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj116 2023-09-22T22:05:37.107 DEBUG:boto:Method: GET 2023-09-22T22:05:37.107 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj116 2023-09-22T22:05:37.107 DEBUG:boto:Data: 2023-09-22T22:05:37.107 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.107 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.107 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.107 DEBUG:boto:Params: {} 2023-09-22T22:05:37.108 DEBUG:boto:Token: None 2023-09-22T22:05:37.108 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/ 2023-09-22T22:05:37.108 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:05:37.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:05:37.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005998ffb3b1146888-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.113 DEBUG:boto:b'ffzdqu-63obj1161000falseobj1162023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:37.113 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:37.113 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:37.113 DEBUG:boto:path=/ffzdqu-63/?prefix=obj116 2023-09-22T22:05:37.113 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj116 2023-09-22T22:05:37.114 DEBUG:boto:Method: GET 2023-09-22T22:05:37.114 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj116 2023-09-22T22:05:37.114 DEBUG:boto:Data: 2023-09-22T22:05:37.114 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.114 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.114 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.114 DEBUG:boto:Params: {} 2023-09-22T22:05:37.114 DEBUG:boto:Token: None 2023-09-22T22:05:37.114 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/ 2023-09-22T22:05:37.115 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:05:37.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:05:37.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092f57873179a0c86-00650e0fb1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.125 DEBUG:boto:b'ffzdqu-63obj1161000falseobj1162023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:37.125 DEBUG:boto:path=/ffzdqu-63/obj116 2023-09-22T22:05:37.126 DEBUG:boto:auth_path=/ffzdqu-63/obj116 2023-09-22T22:05:37.126 DEBUG:boto:Method: HEAD 2023-09-22T22:05:37.126 DEBUG:boto:Path: /ffzdqu-63/obj116 2023-09-22T22:05:37.126 DEBUG:boto:Data: 2023-09-22T22:05:37.126 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.126 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.126 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.126 DEBUG:boto:Params: {} 2023-09-22T22:05:37.126 DEBUG:boto:Token: None 2023-09-22T22:05:37.127 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj116 2023-09-22T22:05:37.127 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SNy564q59PqpevZU7YqZw2Ow2Ok= 2023-09-22T22:05:37.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNy564q59PqpevZU7YqZw2Ow2Ok=', 'Content-Length': '0'} 2023-09-22T22:05:37.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eabf64d9cbf2364b-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.130 DEBUG:boto:path=/ffzdqu-63/obj116 2023-09-22T22:05:37.130 DEBUG:boto:auth_path=/ffzdqu-63/obj116 2023-09-22T22:05:37.130 DEBUG:boto:Method: HEAD 2023-09-22T22:05:37.130 DEBUG:boto:Path: /ffzdqu-63/obj116 2023-09-22T22:05:37.130 DEBUG:boto:Data: 2023-09-22T22:05:37.130 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.130 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.130 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.131 DEBUG:boto:Params: {} 2023-09-22T22:05:37.131 DEBUG:boto:Token: None 2023-09-22T22:05:37.131 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj116 2023-09-22T22:05:37.131 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:SNy564q59PqpevZU7YqZw2Ow2Ok= 2023-09-22T22:05:37.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNy564q59PqpevZU7YqZw2Ow2Ok=', 'Content-Length': '0'} 2023-09-22T22:05:37.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000897c0b22b7903809-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-22T22:05:37.134 DEBUG:boto:path=/ffzdqu-63/obj116 2023-09-22T22:05:37.134 DEBUG:boto:auth_path=/ffzdqu-63/obj116 2023-09-22T22:05:37.134 DEBUG:boto:Method: GET 2023-09-22T22:05:37.134 DEBUG:boto:Path: /ffzdqu-63/obj116 2023-09-22T22:05:37.134 DEBUG:boto:Data: 2023-09-22T22:05:37.135 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.135 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.135 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.135 DEBUG:boto:Params: {} 2023-09-22T22:05:37.135 DEBUG:boto:Token: None 2023-09-22T22:05:37.135 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj116 2023-09-22T22:05:37.135 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Iwm/YA0lAjBkQN7d1/db24gxRtI= 2023-09-22T22:05:37.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Iwm/YA0lAjBkQN7d1/db24gxRtI=', 'Content-Length': '0'} 2023-09-22T22:05:37.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008873557ff6198e81-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.178 DEBUG:boto:path=/ffzdqu-63/obj116 2023-09-22T22:05:37.178 DEBUG:boto:auth_path=/ffzdqu-63/obj116 2023-09-22T22:05:37.179 DEBUG:boto:Method: GET 2023-09-22T22:05:37.179 DEBUG:boto:Path: /ffzdqu-63/obj116 2023-09-22T22:05:37.179 DEBUG:boto:Data: 2023-09-22T22:05:37.179 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.179 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.179 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.179 DEBUG:boto:Params: {} 2023-09-22T22:05:37.179 DEBUG:boto:Token: None 2023-09-22T22:05:37.179 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj116 2023-09-22T22:05:37.180 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Iwm/YA0lAjBkQN7d1/db24gxRtI= 2023-09-22T22:05:37.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Iwm/YA0lAjBkQN7d1/db24gxRtI=', 'Content-Length': '0'} 2023-09-22T22:05:37.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f5868f14d4352b4-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.183 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:37.183 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:37.183 DEBUG:boto:path=/ffzdqu-63/?prefix=obj116 2023-09-22T22:05:37.183 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj116 2023-09-22T22:05:37.183 DEBUG:boto:Method: GET 2023-09-22T22:05:37.183 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj116 2023-09-22T22:05:37.183 DEBUG:boto:Data: 2023-09-22T22:05:37.184 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.184 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.184 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.184 DEBUG:boto:Params: {} 2023-09-22T22:05:37.184 DEBUG:boto:Token: None 2023-09-22T22:05:37.184 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/ 2023-09-22T22:05:37.184 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:05:37.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:05:37.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052929a4ce3575c8d-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.189 DEBUG:boto:b'ffzdqu-63obj1161000falseobj1162023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:37.189 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:37.190 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:37.190 DEBUG:boto:path=/ffzdqu-63/?prefix=obj116 2023-09-22T22:05:37.190 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj116 2023-09-22T22:05:37.190 DEBUG:boto:Method: GET 2023-09-22T22:05:37.190 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj116 2023-09-22T22:05:37.190 DEBUG:boto:Data: 2023-09-22T22:05:37.190 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.190 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.190 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.190 DEBUG:boto:Params: {} 2023-09-22T22:05:37.191 DEBUG:boto:Token: None 2023-09-22T22:05:37.191 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/ 2023-09-22T22:05:37.191 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:05:37.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:05:37.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fe1c90601233434-00650e0fb1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.242 DEBUG:boto:b'ffzdqu-63obj1161000falseobj1162023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:37.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-22T22:05:37.243 DEBUG:boto:path=/ffzdqu-63/obj124 2023-09-22T22:05:37.243 DEBUG:boto:auth_path=/ffzdqu-63/obj124 2023-09-22T22:05:37.243 DEBUG:boto:path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.243 DEBUG:boto:auth_path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.243 DEBUG:boto:Method: GET 2023-09-22T22:05:37.243 DEBUG:boto:Path: /ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.243 DEBUG:boto:Data: 2023-09-22T22:05:37.244 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.244 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.244 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.244 DEBUG:boto:Params: {} 2023-09-22T22:05:37.244 DEBUG:boto:Token: None 2023-09-22T22:05:37.244 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.244 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fimquc5/yzLqcG+GY6rBy4s5Jtg= 2023-09-22T22:05:37.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fimquc5/yzLqcG+GY6rBy4s5Jtg=', 'Content-Length': '0'} 2023-09-22T22:05:37.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f34ac2c295499f69-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.247 DEBUG:boto:path=/ffzdqu-63/obj124 2023-09-22T22:05:37.248 DEBUG:boto:auth_path=/ffzdqu-63/obj124 2023-09-22T22:05:37.248 DEBUG:boto:path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.248 DEBUG:boto:auth_path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.248 DEBUG:boto:Method: GET 2023-09-22T22:05:37.248 DEBUG:boto:Path: /ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.248 DEBUG:boto:Data: 2023-09-22T22:05:37.248 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.248 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.248 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.248 DEBUG:boto:Params: {} 2023-09-22T22:05:37.249 DEBUG:boto:Token: None 2023-09-22T22:05:37.249 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.249 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fimquc5/yzLqcG+GY6rBy4s5Jtg= 2023-09-22T22:05:37.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fimquc5/yzLqcG+GY6rBy4s5Jtg=', 'Content-Length': '0'} 2023-09-22T22:05:37.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039b2d95eb2ef30f1-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.252 DEBUG:boto:path=/ffzdqu-63/obj124 2023-09-22T22:05:37.252 DEBUG:boto:auth_path=/ffzdqu-63/obj124 2023-09-22T22:05:37.252 DEBUG:boto:path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.252 DEBUG:boto:auth_path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.252 DEBUG:boto:Method: HEAD 2023-09-22T22:05:37.253 DEBUG:boto:Path: /ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.253 DEBUG:boto:Data: 2023-09-22T22:05:37.253 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.253 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.253 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.253 DEBUG:boto:Params: {} 2023-09-22T22:05:37.253 DEBUG:boto:Token: None 2023-09-22T22:05:37.253 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.253 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vGDXcV6Xpa8jP6j3wzrovUDwdl4= 2023-09-22T22:05:37.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vGDXcV6Xpa8jP6j3wzrovUDwdl4=', 'Content-Length': '0'} 2023-09-22T22:05:37.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b3431009229c932-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.257 DEBUG:boto:path=/ffzdqu-63/obj124 2023-09-22T22:05:37.257 DEBUG:boto:auth_path=/ffzdqu-63/obj124 2023-09-22T22:05:37.257 DEBUG:boto:path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.257 DEBUG:boto:auth_path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.257 DEBUG:boto:Method: HEAD 2023-09-22T22:05:37.257 DEBUG:boto:Path: /ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.257 DEBUG:boto:Data: 2023-09-22T22:05:37.257 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.257 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.258 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.258 DEBUG:boto:Params: {} 2023-09-22T22:05:37.258 DEBUG:boto:Token: None 2023-09-22T22:05:37.258 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.258 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:vGDXcV6Xpa8jP6j3wzrovUDwdl4= 2023-09-22T22:05:37.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vGDXcV6Xpa8jP6j3wzrovUDwdl4=', 'Content-Length': '0'} 2023-09-22T22:05:37.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000639c8b1b6670b538-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-22T22:05:37.261 DEBUG:boto:path=/ffzdqu-63/obj124 2023-09-22T22:05:37.262 DEBUG:boto:auth_path=/ffzdqu-63/obj124 2023-09-22T22:05:37.262 DEBUG:boto:path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.262 DEBUG:boto:auth_path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.262 DEBUG:boto:Method: GET 2023-09-22T22:05:37.262 DEBUG:boto:Path: /ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.262 DEBUG:boto:Data: 2023-09-22T22:05:37.262 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.262 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.262 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.262 DEBUG:boto:Params: {} 2023-09-22T22:05:37.263 DEBUG:boto:Token: None 2023-09-22T22:05:37.263 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.263 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fimquc5/yzLqcG+GY6rBy4s5Jtg= 2023-09-22T22:05:37.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fimquc5/yzLqcG+GY6rBy4s5Jtg=', 'Content-Length': '0'} 2023-09-22T22:05:37.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fd54ef6bb1355d4-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.306 DEBUG:boto:path=/ffzdqu-63/obj124 2023-09-22T22:05:37.307 DEBUG:boto:auth_path=/ffzdqu-63/obj124 2023-09-22T22:05:37.307 DEBUG:boto:path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.307 DEBUG:boto:auth_path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.307 DEBUG:boto:Method: GET 2023-09-22T22:05:37.307 DEBUG:boto:Path: /ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.307 DEBUG:boto:Data: 2023-09-22T22:05:37.307 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.307 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.307 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.308 DEBUG:boto:Params: {} 2023-09-22T22:05:37.308 DEBUG:boto:Token: None 2023-09-22T22:05:37.308 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj124?versionId=null 2023-09-22T22:05:37.308 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fimquc5/yzLqcG+GY6rBy4s5Jtg= 2023-09-22T22:05:37.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fimquc5/yzLqcG+GY6rBy4s5Jtg=', 'Content-Length': '0'} 2023-09-22T22:05:37.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a13e859d1a74a8f7-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.311 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:37.311 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:37.311 DEBUG:boto:path=/ffzdqu-63/?prefix=obj124 2023-09-22T22:05:37.311 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj124 2023-09-22T22:05:37.311 DEBUG:boto:Method: GET 2023-09-22T22:05:37.311 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj124 2023-09-22T22:05:37.312 DEBUG:boto:Data: 2023-09-22T22:05:37.312 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.312 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.312 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.312 DEBUG:boto:Params: {} 2023-09-22T22:05:37.312 DEBUG:boto:Token: None 2023-09-22T22:05:37.312 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/ 2023-09-22T22:05:37.312 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:05:37.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:05:37.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dec6b3e31380df43-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.317 DEBUG:boto:b'ffzdqu-63obj1241000falseobj1242023-09-22T22:05:15.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:37.317 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:37.317 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:37.318 DEBUG:boto:path=/ffzdqu-63/?prefix=obj124 2023-09-22T22:05:37.318 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj124 2023-09-22T22:05:37.318 DEBUG:boto:Method: GET 2023-09-22T22:05:37.318 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj124 2023-09-22T22:05:37.318 DEBUG:boto:Data: 2023-09-22T22:05:37.318 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.318 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.318 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.318 DEBUG:boto:Params: {} 2023-09-22T22:05:37.319 DEBUG:boto:Token: None 2023-09-22T22:05:37.319 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/ 2023-09-22T22:05:37.319 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:05:37.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:05:37.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e410f379998149b1-00650e0fb1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.374 DEBUG:boto:b'ffzdqu-63obj1241000falseobj1242023-09-22T22:05:15.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:37.375 DEBUG:boto:path=/ffzdqu-63/obj124 2023-09-22T22:05:37.375 DEBUG:boto:auth_path=/ffzdqu-63/obj124 2023-09-22T22:05:37.375 DEBUG:boto:Method: HEAD 2023-09-22T22:05:37.375 DEBUG:boto:Path: /ffzdqu-63/obj124 2023-09-22T22:05:37.375 DEBUG:boto:Data: 2023-09-22T22:05:37.375 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.376 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.376 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.376 DEBUG:boto:Params: {} 2023-09-22T22:05:37.376 DEBUG:boto:Token: None 2023-09-22T22:05:37.376 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj124 2023-09-22T22:05:37.376 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:14s102Q24wkER37WtUb5udNTZf0= 2023-09-22T22:05:37.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:14s102Q24wkER37WtUb5udNTZf0=', 'Content-Length': '0'} 2023-09-22T22:05:37.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee28d331bbe3273b-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.383 DEBUG:boto:path=/ffzdqu-63/obj124 2023-09-22T22:05:37.383 DEBUG:boto:auth_path=/ffzdqu-63/obj124 2023-09-22T22:05:37.383 DEBUG:boto:Method: HEAD 2023-09-22T22:05:37.384 DEBUG:boto:Path: /ffzdqu-63/obj124 2023-09-22T22:05:37.384 DEBUG:boto:Data: 2023-09-22T22:05:37.384 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.384 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.384 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.384 DEBUG:boto:Params: {} 2023-09-22T22:05:37.384 DEBUG:boto:Token: None 2023-09-22T22:05:37.384 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj124 2023-09-22T22:05:37.385 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:14s102Q24wkER37WtUb5udNTZf0= 2023-09-22T22:05:37.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:14s102Q24wkER37WtUb5udNTZf0=', 'Content-Length': '0'} 2023-09-22T22:05:37.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000101acc8dde6c1d37-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-22T22:05:37.389 DEBUG:boto:path=/ffzdqu-63/obj124 2023-09-22T22:05:37.389 DEBUG:boto:auth_path=/ffzdqu-63/obj124 2023-09-22T22:05:37.389 DEBUG:boto:Method: GET 2023-09-22T22:05:37.389 DEBUG:boto:Path: /ffzdqu-63/obj124 2023-09-22T22:05:37.390 DEBUG:boto:Data: 2023-09-22T22:05:37.390 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.390 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.390 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.390 DEBUG:boto:Params: {} 2023-09-22T22:05:37.390 DEBUG:boto:Token: None 2023-09-22T22:05:37.390 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj124 2023-09-22T22:05:37.390 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IlWMbrbMFvShKjznecvygLxWe4Q= 2023-09-22T22:05:37.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IlWMbrbMFvShKjznecvygLxWe4Q=', 'Content-Length': '0'} 2023-09-22T22:05:37.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a8d69d7aa0a8dbc-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.394 DEBUG:boto:path=/ffzdqu-63/obj124 2023-09-22T22:05:37.394 DEBUG:boto:auth_path=/ffzdqu-63/obj124 2023-09-22T22:05:37.394 DEBUG:boto:Method: GET 2023-09-22T22:05:37.394 DEBUG:boto:Path: /ffzdqu-63/obj124 2023-09-22T22:05:37.395 DEBUG:boto:Data: 2023-09-22T22:05:37.395 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.395 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.395 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.395 DEBUG:boto:Params: {} 2023-09-22T22:05:37.395 DEBUG:boto:Token: None 2023-09-22T22:05:37.395 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj124 2023-09-22T22:05:37.395 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:IlWMbrbMFvShKjznecvygLxWe4Q= 2023-09-22T22:05:37.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IlWMbrbMFvShKjznecvygLxWe4Q=', 'Content-Length': '0'} 2023-09-22T22:05:37.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceef9f7a6fe32da9-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.398 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:37.399 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:37.399 DEBUG:boto:path=/ffzdqu-63/?prefix=obj124 2023-09-22T22:05:37.399 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj124 2023-09-22T22:05:37.399 DEBUG:boto:Method: GET 2023-09-22T22:05:37.399 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj124 2023-09-22T22:05:37.399 DEBUG:boto:Data: 2023-09-22T22:05:37.399 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.399 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.399 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.399 DEBUG:boto:Params: {} 2023-09-22T22:05:37.400 DEBUG:boto:Token: None 2023-09-22T22:05:37.400 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/ 2023-09-22T22:05:37.400 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:05:37.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:05:37.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c6e3ff48298e893-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.412 DEBUG:boto:b'ffzdqu-63obj1241000falseobj1242023-09-22T22:05:15.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:37.413 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:37.413 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:37.413 DEBUG:boto:path=/ffzdqu-63/?prefix=obj124 2023-09-22T22:05:37.413 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj124 2023-09-22T22:05:37.413 DEBUG:boto:Method: GET 2023-09-22T22:05:37.413 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj124 2023-09-22T22:05:37.413 DEBUG:boto:Data: 2023-09-22T22:05:37.413 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.413 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.414 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.414 DEBUG:boto:Params: {} 2023-09-22T22:05:37.414 DEBUG:boto:Token: None 2023-09-22T22:05:37.414 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/ 2023-09-22T22:05:37.414 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:05:37.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:05:37.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3d884c2c889f5a0-00650e0fb1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.474 DEBUG:boto:b'ffzdqu-63obj1241000falseobj1242023-09-22T22:05:15.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:37.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-22T22:05:37.475 DEBUG:boto:path=/ffzdqu-63/obj145 2023-09-22T22:05:37.475 DEBUG:boto:auth_path=/ffzdqu-63/obj145 2023-09-22T22:05:37.475 DEBUG:boto:path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.475 DEBUG:boto:auth_path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.475 DEBUG:boto:Method: GET 2023-09-22T22:05:37.475 DEBUG:boto:Path: /ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.476 DEBUG:boto:Data: 2023-09-22T22:05:37.476 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.476 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.476 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.476 DEBUG:boto:Params: {} 2023-09-22T22:05:37.476 DEBUG:boto:Token: None 2023-09-22T22:05:37.476 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.476 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rpXSdE1Jf+8b/bv3qzZXP+ATqTE= 2023-09-22T22:05:37.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rpXSdE1Jf+8b/bv3qzZXP+ATqTE=', 'Content-Length': '0'} 2023-09-22T22:05:37.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5b6aa8ce015e947-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.479 DEBUG:boto:path=/ffzdqu-63/obj145 2023-09-22T22:05:37.479 DEBUG:boto:auth_path=/ffzdqu-63/obj145 2023-09-22T22:05:37.480 DEBUG:boto:path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.480 DEBUG:boto:auth_path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.480 DEBUG:boto:Method: GET 2023-09-22T22:05:37.480 DEBUG:boto:Path: /ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.480 DEBUG:boto:Data: 2023-09-22T22:05:37.480 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.480 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.480 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.480 DEBUG:boto:Params: {} 2023-09-22T22:05:37.480 DEBUG:boto:Token: None 2023-09-22T22:05:37.481 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.481 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rpXSdE1Jf+8b/bv3qzZXP+ATqTE= 2023-09-22T22:05:37.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rpXSdE1Jf+8b/bv3qzZXP+ATqTE=', 'Content-Length': '0'} 2023-09-22T22:05:37.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005241ad99e2d3577c-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.483 DEBUG:boto:path=/ffzdqu-63/obj145 2023-09-22T22:05:37.484 DEBUG:boto:auth_path=/ffzdqu-63/obj145 2023-09-22T22:05:37.484 DEBUG:boto:path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.484 DEBUG:boto:auth_path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.484 DEBUG:boto:Method: HEAD 2023-09-22T22:05:37.484 DEBUG:boto:Path: /ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.484 DEBUG:boto:Data: 2023-09-22T22:05:37.484 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.484 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.484 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.484 DEBUG:boto:Params: {} 2023-09-22T22:05:37.485 DEBUG:boto:Token: None 2023-09-22T22:05:37.485 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.485 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Snte4OVVpw1kbAUX7eGjtLpjL1M= 2023-09-22T22:05:37.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Snte4OVVpw1kbAUX7eGjtLpjL1M=', 'Content-Length': '0'} 2023-09-22T22:05:37.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094c4ebd2459b467d-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.487 DEBUG:boto:path=/ffzdqu-63/obj145 2023-09-22T22:05:37.487 DEBUG:boto:auth_path=/ffzdqu-63/obj145 2023-09-22T22:05:37.487 DEBUG:boto:path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.487 DEBUG:boto:auth_path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.487 DEBUG:boto:Method: HEAD 2023-09-22T22:05:37.488 DEBUG:boto:Path: /ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.488 DEBUG:boto:Data: 2023-09-22T22:05:37.488 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.488 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.488 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.488 DEBUG:boto:Params: {} 2023-09-22T22:05:37.488 DEBUG:boto:Token: None 2023-09-22T22:05:37.488 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.488 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Snte4OVVpw1kbAUX7eGjtLpjL1M= 2023-09-22T22:05:37.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Snte4OVVpw1kbAUX7eGjtLpjL1M=', 'Content-Length': '0'} 2023-09-22T22:05:37.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e02a0201cb2f1100-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-22T22:05:37.491 DEBUG:boto:path=/ffzdqu-63/obj145 2023-09-22T22:05:37.491 DEBUG:boto:auth_path=/ffzdqu-63/obj145 2023-09-22T22:05:37.491 DEBUG:boto:path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.491 DEBUG:boto:auth_path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.491 DEBUG:boto:Method: GET 2023-09-22T22:05:37.491 DEBUG:boto:Path: /ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.492 DEBUG:boto:Data: 2023-09-22T22:05:37.492 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.492 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.492 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.492 DEBUG:boto:Params: {} 2023-09-22T22:05:37.492 DEBUG:boto:Token: None 2023-09-22T22:05:37.492 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.492 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rpXSdE1Jf+8b/bv3qzZXP+ATqTE= 2023-09-22T22:05:37.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rpXSdE1Jf+8b/bv3qzZXP+ATqTE=', 'Content-Length': '0'} 2023-09-22T22:05:37.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003212822771f48c93-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.542 DEBUG:boto:path=/ffzdqu-63/obj145 2023-09-22T22:05:37.542 DEBUG:boto:auth_path=/ffzdqu-63/obj145 2023-09-22T22:05:37.543 DEBUG:boto:path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.543 DEBUG:boto:auth_path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.543 DEBUG:boto:Method: GET 2023-09-22T22:05:37.543 DEBUG:boto:Path: /ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.543 DEBUG:boto:Data: 2023-09-22T22:05:37.543 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.543 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.543 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.543 DEBUG:boto:Params: {} 2023-09-22T22:05:37.543 DEBUG:boto:Token: None 2023-09-22T22:05:37.544 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj145?versionId=null 2023-09-22T22:05:37.544 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rpXSdE1Jf+8b/bv3qzZXP+ATqTE= 2023-09-22T22:05:37.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rpXSdE1Jf+8b/bv3qzZXP+ATqTE=', 'Content-Length': '0'} 2023-09-22T22:05:37.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a4b1a056808af7c-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.547 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:37.548 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:37.548 DEBUG:boto:path=/ffzdqu-63/?prefix=obj145 2023-09-22T22:05:37.548 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj145 2023-09-22T22:05:37.548 DEBUG:boto:Method: GET 2023-09-22T22:05:37.548 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj145 2023-09-22T22:05:37.548 DEBUG:boto:Data: 2023-09-22T22:05:37.548 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.548 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.548 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.549 DEBUG:boto:Params: {} 2023-09-22T22:05:37.549 DEBUG:boto:Token: None 2023-09-22T22:05:37.549 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/ 2023-09-22T22:05:37.549 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:05:37.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:05:37.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd6bf72a20b35356-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.555 DEBUG:boto:b'ffzdqu-63obj1451000falseobj1452023-09-22T22:05:15.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:37.555 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:37.555 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:37.556 DEBUG:boto:path=/ffzdqu-63/?prefix=obj145 2023-09-22T22:05:37.556 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj145 2023-09-22T22:05:37.556 DEBUG:boto:Method: GET 2023-09-22T22:05:37.556 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj145 2023-09-22T22:05:37.556 DEBUG:boto:Data: 2023-09-22T22:05:37.556 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.556 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.556 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.556 DEBUG:boto:Params: {} 2023-09-22T22:05:37.556 DEBUG:boto:Token: None 2023-09-22T22:05:37.557 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/ 2023-09-22T22:05:37.557 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:05:37.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:05:37.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adc7438b332c669e-00650e0fb1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.610 DEBUG:boto:b'ffzdqu-63obj1451000falseobj1452023-09-22T22:05:15.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:37.611 DEBUG:boto:path=/ffzdqu-63/obj145 2023-09-22T22:05:37.611 DEBUG:boto:auth_path=/ffzdqu-63/obj145 2023-09-22T22:05:37.611 DEBUG:boto:Method: HEAD 2023-09-22T22:05:37.611 DEBUG:boto:Path: /ffzdqu-63/obj145 2023-09-22T22:05:37.611 DEBUG:boto:Data: 2023-09-22T22:05:37.611 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.612 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.612 DEBUG:boto:Params: {} 2023-09-22T22:05:37.612 DEBUG:boto:Token: None 2023-09-22T22:05:37.612 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj145 2023-09-22T22:05:37.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rAIDCTDNYgh8n/gnx/w/dhAy1wQ= 2023-09-22T22:05:37.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rAIDCTDNYgh8n/gnx/w/dhAy1wQ=', 'Content-Length': '0'} 2023-09-22T22:05:37.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df5fd033994040a5-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.616 DEBUG:boto:path=/ffzdqu-63/obj145 2023-09-22T22:05:37.616 DEBUG:boto:auth_path=/ffzdqu-63/obj145 2023-09-22T22:05:37.616 DEBUG:boto:Method: HEAD 2023-09-22T22:05:37.616 DEBUG:boto:Path: /ffzdqu-63/obj145 2023-09-22T22:05:37.616 DEBUG:boto:Data: 2023-09-22T22:05:37.616 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.616 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.617 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.617 DEBUG:boto:Params: {} 2023-09-22T22:05:37.617 DEBUG:boto:Token: None 2023-09-22T22:05:37.617 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj145 2023-09-22T22:05:37.617 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rAIDCTDNYgh8n/gnx/w/dhAy1wQ= 2023-09-22T22:05:37.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rAIDCTDNYgh8n/gnx/w/dhAy1wQ=', 'Content-Length': '0'} 2023-09-22T22:05:37.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e845f370efe5417-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-22T22:05:37.621 DEBUG:boto:path=/ffzdqu-63/obj145 2023-09-22T22:05:37.621 DEBUG:boto:auth_path=/ffzdqu-63/obj145 2023-09-22T22:05:37.621 DEBUG:boto:Method: GET 2023-09-22T22:05:37.621 DEBUG:boto:Path: /ffzdqu-63/obj145 2023-09-22T22:05:37.621 DEBUG:boto:Data: 2023-09-22T22:05:37.621 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.621 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.621 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.621 DEBUG:boto:Params: {} 2023-09-22T22:05:37.622 DEBUG:boto:Token: None 2023-09-22T22:05:37.622 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj145 2023-09-22T22:05:37.622 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WK1Uq3ChfGjO0Z5q8sKXj6rPtss= 2023-09-22T22:05:37.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WK1Uq3ChfGjO0Z5q8sKXj6rPtss=', 'Content-Length': '0'} 2023-09-22T22:05:37.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5fee5be6b42a29e-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.625 DEBUG:boto:path=/ffzdqu-63/obj145 2023-09-22T22:05:37.625 DEBUG:boto:auth_path=/ffzdqu-63/obj145 2023-09-22T22:05:37.625 DEBUG:boto:Method: GET 2023-09-22T22:05:37.625 DEBUG:boto:Path: /ffzdqu-63/obj145 2023-09-22T22:05:37.625 DEBUG:boto:Data: 2023-09-22T22:05:37.625 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.626 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.626 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.626 DEBUG:boto:Params: {} 2023-09-22T22:05:37.626 DEBUG:boto:Token: None 2023-09-22T22:05:37.626 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj145 2023-09-22T22:05:37.626 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WK1Uq3ChfGjO0Z5q8sKXj6rPtss= 2023-09-22T22:05:37.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WK1Uq3ChfGjO0Z5q8sKXj6rPtss=', 'Content-Length': '0'} 2023-09-22T22:05:37.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000cbabe1f3d45625-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.629 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:37.629 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:37.629 DEBUG:boto:path=/ffzdqu-63/?prefix=obj145 2023-09-22T22:05:37.629 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj145 2023-09-22T22:05:37.630 DEBUG:boto:Method: GET 2023-09-22T22:05:37.630 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj145 2023-09-22T22:05:37.630 DEBUG:boto:Data: 2023-09-22T22:05:37.630 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.630 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.630 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.630 DEBUG:boto:Params: {} 2023-09-22T22:05:37.630 DEBUG:boto:Token: None 2023-09-22T22:05:37.630 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/ 2023-09-22T22:05:37.631 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:05:37.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:05:37.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052e1d00682bad0fd-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.635 DEBUG:boto:b'ffzdqu-63obj1451000falseobj1452023-09-22T22:05:15.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:37.636 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:37.636 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:37.636 DEBUG:boto:path=/ffzdqu-63/?prefix=obj145 2023-09-22T22:05:37.636 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj145 2023-09-22T22:05:37.636 DEBUG:boto:Method: GET 2023-09-22T22:05:37.637 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj145 2023-09-22T22:05:37.637 DEBUG:boto:Data: 2023-09-22T22:05:37.637 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.637 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.637 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.637 DEBUG:boto:Params: {} 2023-09-22T22:05:37.637 DEBUG:boto:Token: None 2023-09-22T22:05:37.637 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/ 2023-09-22T22:05:37.637 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:05:37.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:05:37.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8237ca2630cff22-00650e0fb1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.647 DEBUG:boto:b'ffzdqu-63obj1451000falseobj1452023-09-22T22:05:15.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:37.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-09-22T22:05:37.647 DEBUG:boto:path=/ffzdqu-63/obj148 2023-09-22T22:05:37.647 DEBUG:boto:auth_path=/ffzdqu-63/obj148 2023-09-22T22:05:37.647 DEBUG:boto:path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.648 DEBUG:boto:auth_path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.648 DEBUG:boto:Method: GET 2023-09-22T22:05:37.648 DEBUG:boto:Path: /ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.648 DEBUG:boto:Data: 2023-09-22T22:05:37.648 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.648 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.648 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.648 DEBUG:boto:Params: {} 2023-09-22T22:05:37.648 DEBUG:boto:Token: None 2023-09-22T22:05:37.649 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.649 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1FsSUUWdrIzchAa0x6aIrJfSDPM= 2023-09-22T22:05:37.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1FsSUUWdrIzchAa0x6aIrJfSDPM=', 'Content-Length': '0'} 2023-09-22T22:05:37.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac41978b5b84eeb4-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.694 DEBUG:boto:path=/ffzdqu-63/obj148 2023-09-22T22:05:37.695 DEBUG:boto:auth_path=/ffzdqu-63/obj148 2023-09-22T22:05:37.695 DEBUG:boto:path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.695 DEBUG:boto:auth_path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.695 DEBUG:boto:Method: GET 2023-09-22T22:05:37.695 DEBUG:boto:Path: /ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.695 DEBUG:boto:Data: 2023-09-22T22:05:37.695 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.695 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.695 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.696 DEBUG:boto:Params: {} 2023-09-22T22:05:37.696 DEBUG:boto:Token: None 2023-09-22T22:05:37.696 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.696 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1FsSUUWdrIzchAa0x6aIrJfSDPM= 2023-09-22T22:05:37.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1FsSUUWdrIzchAa0x6aIrJfSDPM=', 'Content-Length': '0'} 2023-09-22T22:05:37.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f283182eaf4893d4-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.699 DEBUG:boto:path=/ffzdqu-63/obj148 2023-09-22T22:05:37.700 DEBUG:boto:auth_path=/ffzdqu-63/obj148 2023-09-22T22:05:37.700 DEBUG:boto:path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.700 DEBUG:boto:auth_path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.700 DEBUG:boto:Method: HEAD 2023-09-22T22:05:37.700 DEBUG:boto:Path: /ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.700 DEBUG:boto:Data: 2023-09-22T22:05:37.700 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.700 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.700 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.700 DEBUG:boto:Params: {} 2023-09-22T22:05:37.701 DEBUG:boto:Token: None 2023-09-22T22:05:37.701 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.701 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MQcJNQcHn8mqm2qbMpf8G386OUA= 2023-09-22T22:05:37.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MQcJNQcHn8mqm2qbMpf8G386OUA=', 'Content-Length': '0'} 2023-09-22T22:05:37.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f7c3d7ced628824-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.704 DEBUG:boto:path=/ffzdqu-63/obj148 2023-09-22T22:05:37.704 DEBUG:boto:auth_path=/ffzdqu-63/obj148 2023-09-22T22:05:37.704 DEBUG:boto:path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.704 DEBUG:boto:auth_path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.704 DEBUG:boto:Method: HEAD 2023-09-22T22:05:37.704 DEBUG:boto:Path: /ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.705 DEBUG:boto:Data: 2023-09-22T22:05:37.705 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.705 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.705 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.705 DEBUG:boto:Params: {} 2023-09-22T22:05:37.705 DEBUG:boto:Token: None 2023-09-22T22:05:37.705 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.705 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MQcJNQcHn8mqm2qbMpf8G386OUA= 2023-09-22T22:05:37.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MQcJNQcHn8mqm2qbMpf8G386OUA=', 'Content-Length': '0'} 2023-09-22T22:05:37.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f3f0f034ceb9066-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-09-22T22:05:37.708 DEBUG:boto:path=/ffzdqu-63/obj148 2023-09-22T22:05:37.708 DEBUG:boto:auth_path=/ffzdqu-63/obj148 2023-09-22T22:05:37.708 DEBUG:boto:path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.708 DEBUG:boto:auth_path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.709 DEBUG:boto:Method: GET 2023-09-22T22:05:37.709 DEBUG:boto:Path: /ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.709 DEBUG:boto:Data: 2023-09-22T22:05:37.709 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.709 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.709 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.709 DEBUG:boto:Params: {} 2023-09-22T22:05:37.709 DEBUG:boto:Token: None 2023-09-22T22:05:37.710 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.710 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1FsSUUWdrIzchAa0x6aIrJfSDPM= 2023-09-22T22:05:37.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1FsSUUWdrIzchAa0x6aIrJfSDPM=', 'Content-Length': '0'} 2023-09-22T22:05:37.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000761cc8075eaf6a10-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.713 DEBUG:boto:path=/ffzdqu-63/obj148 2023-09-22T22:05:37.713 DEBUG:boto:auth_path=/ffzdqu-63/obj148 2023-09-22T22:05:37.713 DEBUG:boto:path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.713 DEBUG:boto:auth_path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.713 DEBUG:boto:Method: GET 2023-09-22T22:05:37.713 DEBUG:boto:Path: /ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.713 DEBUG:boto:Data: 2023-09-22T22:05:37.713 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.713 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.714 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.714 DEBUG:boto:Params: {} 2023-09-22T22:05:37.714 DEBUG:boto:Token: None 2023-09-22T22:05:37.714 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj148?versionId=null 2023-09-22T22:05:37.714 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1FsSUUWdrIzchAa0x6aIrJfSDPM= 2023-09-22T22:05:37.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1FsSUUWdrIzchAa0x6aIrJfSDPM=', 'Content-Length': '0'} 2023-09-22T22:05:37.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fe0a5448b84326c-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.758 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:37.758 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:37.759 DEBUG:boto:path=/ffzdqu-63/?prefix=obj148 2023-09-22T22:05:37.759 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj148 2023-09-22T22:05:37.759 DEBUG:boto:Method: GET 2023-09-22T22:05:37.759 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj148 2023-09-22T22:05:37.759 DEBUG:boto:Data: 2023-09-22T22:05:37.759 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.759 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.759 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.759 DEBUG:boto:Params: {} 2023-09-22T22:05:37.759 DEBUG:boto:Token: None 2023-09-22T22:05:37.760 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/ 2023-09-22T22:05:37.760 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:05:37.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:05:37.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb3082faef0f1ac0-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.765 DEBUG:boto:b'ffzdqu-63obj1481000falseobj1482023-09-22T22:05:15.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:37.765 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:37.765 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:37.765 DEBUG:boto:path=/ffzdqu-63/?prefix=obj148 2023-09-22T22:05:37.765 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj148 2023-09-22T22:05:37.766 DEBUG:boto:Method: GET 2023-09-22T22:05:37.766 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj148 2023-09-22T22:05:37.766 DEBUG:boto:Data: 2023-09-22T22:05:37.766 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.766 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.766 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.766 DEBUG:boto:Params: {} 2023-09-22T22:05:37.766 DEBUG:boto:Token: None 2023-09-22T22:05:37.766 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/ 2023-09-22T22:05:37.767 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:05:37.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:05:37.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b5a980b03fe545b-00650e0fb1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.777 DEBUG:boto:b'ffzdqu-63obj1481000falseobj1482023-09-22T22:05:15.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:37.778 DEBUG:boto:path=/ffzdqu-63/obj148 2023-09-22T22:05:37.778 DEBUG:boto:auth_path=/ffzdqu-63/obj148 2023-09-22T22:05:37.778 DEBUG:boto:Method: HEAD 2023-09-22T22:05:37.778 DEBUG:boto:Path: /ffzdqu-63/obj148 2023-09-22T22:05:37.778 DEBUG:boto:Data: 2023-09-22T22:05:37.778 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.779 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.779 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.779 DEBUG:boto:Params: {} 2023-09-22T22:05:37.779 DEBUG:boto:Token: None 2023-09-22T22:05:37.779 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj148 2023-09-22T22:05:37.779 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gysKSRyvW8oABzgqf+Pc4iOLASk= 2023-09-22T22:05:37.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gysKSRyvW8oABzgqf+Pc4iOLASk=', 'Content-Length': '0'} 2023-09-22T22:05:37.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ca4c6679929f7b0-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.783 DEBUG:boto:path=/ffzdqu-63/obj148 2023-09-22T22:05:37.783 DEBUG:boto:auth_path=/ffzdqu-63/obj148 2023-09-22T22:05:37.783 DEBUG:boto:Method: HEAD 2023-09-22T22:05:37.783 DEBUG:boto:Path: /ffzdqu-63/obj148 2023-09-22T22:05:37.783 DEBUG:boto:Data: 2023-09-22T22:05:37.783 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.783 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.783 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.784 DEBUG:boto:Params: {} 2023-09-22T22:05:37.784 DEBUG:boto:Token: None 2023-09-22T22:05:37.784 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj148 2023-09-22T22:05:37.784 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gysKSRyvW8oABzgqf+Pc4iOLASk= 2023-09-22T22:05:37.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gysKSRyvW8oABzgqf+Pc4iOLASk=', 'Content-Length': '0'} 2023-09-22T22:05:37.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004534251f3012e58a-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-09-22T22:05:37.788 DEBUG:boto:path=/ffzdqu-63/obj148 2023-09-22T22:05:37.788 DEBUG:boto:auth_path=/ffzdqu-63/obj148 2023-09-22T22:05:37.788 DEBUG:boto:Method: GET 2023-09-22T22:05:37.788 DEBUG:boto:Path: /ffzdqu-63/obj148 2023-09-22T22:05:37.788 DEBUG:boto:Data: 2023-09-22T22:05:37.788 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.788 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.788 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.788 DEBUG:boto:Params: {} 2023-09-22T22:05:37.789 DEBUG:boto:Token: None 2023-09-22T22:05:37.789 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj148 2023-09-22T22:05:37.789 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Hd1vwzwyq55uyeFNUjYJyCoj5oE= 2023-09-22T22:05:37.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Hd1vwzwyq55uyeFNUjYJyCoj5oE=', 'Content-Length': '0'} 2023-09-22T22:05:37.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a419ddbcabf44380-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.838 DEBUG:boto:path=/ffzdqu-63/obj148 2023-09-22T22:05:37.838 DEBUG:boto:auth_path=/ffzdqu-63/obj148 2023-09-22T22:05:37.839 DEBUG:boto:Method: GET 2023-09-22T22:05:37.839 DEBUG:boto:Path: /ffzdqu-63/obj148 2023-09-22T22:05:37.839 DEBUG:boto:Data: 2023-09-22T22:05:37.839 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.839 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.839 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.839 DEBUG:boto:Params: {} 2023-09-22T22:05:37.839 DEBUG:boto:Token: None 2023-09-22T22:05:37.839 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj148 2023-09-22T22:05:37.840 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Hd1vwzwyq55uyeFNUjYJyCoj5oE= 2023-09-22T22:05:37.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Hd1vwzwyq55uyeFNUjYJyCoj5oE=', 'Content-Length': '0'} 2023-09-22T22:05:37.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006548f7309d96f5a7-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.843 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:37.843 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:37.843 DEBUG:boto:path=/ffzdqu-63/?prefix=obj148 2023-09-22T22:05:37.844 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj148 2023-09-22T22:05:37.844 DEBUG:boto:Method: GET 2023-09-22T22:05:37.844 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj148 2023-09-22T22:05:37.844 DEBUG:boto:Data: 2023-09-22T22:05:37.844 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.844 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.844 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.844 DEBUG:boto:Params: {} 2023-09-22T22:05:37.844 DEBUG:boto:Token: None 2023-09-22T22:05:37.845 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/ 2023-09-22T22:05:37.845 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:05:37.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:05:37.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000826cf2c3f3ba8855-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.850 DEBUG:boto:b'ffzdqu-63obj1481000falseobj1482023-09-22T22:05:15.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:37.850 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:37.851 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:37.851 DEBUG:boto:path=/ffzdqu-63/?prefix=obj148 2023-09-22T22:05:37.852 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj148 2023-09-22T22:05:37.852 DEBUG:boto:Method: GET 2023-09-22T22:05:37.852 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj148 2023-09-22T22:05:37.853 DEBUG:boto:Data: 2023-09-22T22:05:37.853 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.853 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.853 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.853 DEBUG:boto:Params: {} 2023-09-22T22:05:37.854 DEBUG:boto:Token: None 2023-09-22T22:05:37.854 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/ 2023-09-22T22:05:37.854 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:05:37.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:05:37.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3c5f1c4c1f37e21-00650e0fb1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.906 DEBUG:boto:b'ffzdqu-63obj1481000falseobj1482023-09-22T22:05:15.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:37.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-22T22:05:37.907 DEBUG:boto:path=/ffzdqu-63/obj155 2023-09-22T22:05:37.907 DEBUG:boto:auth_path=/ffzdqu-63/obj155 2023-09-22T22:05:37.907 DEBUG:boto:path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.907 DEBUG:boto:auth_path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.907 DEBUG:boto:Method: GET 2023-09-22T22:05:37.908 DEBUG:boto:Path: /ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.908 DEBUG:boto:Data: 2023-09-22T22:05:37.908 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.908 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.908 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.908 DEBUG:boto:Params: {} 2023-09-22T22:05:37.908 DEBUG:boto:Token: None 2023-09-22T22:05:37.908 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.908 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/r3b2nZoGCN4rZRqE+a3kMF52tI= 2023-09-22T22:05:37.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/r3b2nZoGCN4rZRqE+a3kMF52tI=', 'Content-Length': '0'} 2023-09-22T22:05:37.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ebdfff93cea22cf-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.912 DEBUG:boto:path=/ffzdqu-63/obj155 2023-09-22T22:05:37.912 DEBUG:boto:auth_path=/ffzdqu-63/obj155 2023-09-22T22:05:37.912 DEBUG:boto:path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.912 DEBUG:boto:auth_path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.913 DEBUG:boto:Method: GET 2023-09-22T22:05:37.913 DEBUG:boto:Path: /ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.913 DEBUG:boto:Data: 2023-09-22T22:05:37.913 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.913 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.913 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.913 DEBUG:boto:Params: {} 2023-09-22T22:05:37.913 DEBUG:boto:Token: None 2023-09-22T22:05:37.913 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.914 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/r3b2nZoGCN4rZRqE+a3kMF52tI= 2023-09-22T22:05:37.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/r3b2nZoGCN4rZRqE+a3kMF52tI=', 'Content-Length': '0'} 2023-09-22T22:05:37.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000166675c4e7e37ad-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.917 DEBUG:boto:path=/ffzdqu-63/obj155 2023-09-22T22:05:37.917 DEBUG:boto:auth_path=/ffzdqu-63/obj155 2023-09-22T22:05:37.917 DEBUG:boto:path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.917 DEBUG:boto:auth_path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.917 DEBUG:boto:Method: HEAD 2023-09-22T22:05:37.917 DEBUG:boto:Path: /ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.918 DEBUG:boto:Data: 2023-09-22T22:05:37.918 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.918 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.918 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.918 DEBUG:boto:Params: {} 2023-09-22T22:05:37.918 DEBUG:boto:Token: None 2023-09-22T22:05:37.918 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.918 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HGvCOXD6BzC7MWgjACb9EiKznek= 2023-09-22T22:05:37.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HGvCOXD6BzC7MWgjACb9EiKznek=', 'Content-Length': '0'} 2023-09-22T22:05:37.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004028e195f741fe1e-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.921 DEBUG:boto:path=/ffzdqu-63/obj155 2023-09-22T22:05:37.921 DEBUG:boto:auth_path=/ffzdqu-63/obj155 2023-09-22T22:05:37.921 DEBUG:boto:path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.922 DEBUG:boto:auth_path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.922 DEBUG:boto:Method: HEAD 2023-09-22T22:05:37.922 DEBUG:boto:Path: /ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.922 DEBUG:boto:Data: 2023-09-22T22:05:37.922 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.922 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.922 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.922 DEBUG:boto:Params: {} 2023-09-22T22:05:37.922 DEBUG:boto:Token: None 2023-09-22T22:05:37.923 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.923 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HGvCOXD6BzC7MWgjACb9EiKznek= 2023-09-22T22:05:37.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HGvCOXD6BzC7MWgjACb9EiKznek=', 'Content-Length': '0'} 2023-09-22T22:05:37.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1441b98f7d2bcc7-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-22T22:05:37.926 DEBUG:boto:path=/ffzdqu-63/obj155 2023-09-22T22:05:37.926 DEBUG:boto:auth_path=/ffzdqu-63/obj155 2023-09-22T22:05:37.926 DEBUG:boto:path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.926 DEBUG:boto:auth_path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.926 DEBUG:boto:Method: GET 2023-09-22T22:05:37.926 DEBUG:boto:Path: /ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.926 DEBUG:boto:Data: 2023-09-22T22:05:37.927 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.927 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.927 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.927 DEBUG:boto:Params: {} 2023-09-22T22:05:37.927 DEBUG:boto:Token: None 2023-09-22T22:05:37.927 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.927 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/r3b2nZoGCN4rZRqE+a3kMF52tI= 2023-09-22T22:05:37.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/r3b2nZoGCN4rZRqE+a3kMF52tI=', 'Content-Length': '0'} 2023-09-22T22:05:37.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063cddd83b1de0120-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.930 DEBUG:boto:path=/ffzdqu-63/obj155 2023-09-22T22:05:37.930 DEBUG:boto:auth_path=/ffzdqu-63/obj155 2023-09-22T22:05:37.930 DEBUG:boto:path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.931 DEBUG:boto:auth_path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.931 DEBUG:boto:Method: GET 2023-09-22T22:05:37.931 DEBUG:boto:Path: /ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.931 DEBUG:boto:Data: 2023-09-22T22:05:37.931 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.931 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.931 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.931 DEBUG:boto:Params: {} 2023-09-22T22:05:37.931 DEBUG:boto:Token: None 2023-09-22T22:05:37.932 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj155?versionId=null 2023-09-22T22:05:37.932 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/r3b2nZoGCN4rZRqE+a3kMF52tI= 2023-09-22T22:05:37.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/r3b2nZoGCN4rZRqE+a3kMF52tI=', 'Content-Length': '0'} 2023-09-22T22:05:37.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e0d08267af0c129-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.935 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:37.935 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:37.936 DEBUG:boto:path=/ffzdqu-63/?prefix=obj155 2023-09-22T22:05:37.936 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj155 2023-09-22T22:05:37.936 DEBUG:boto:Method: GET 2023-09-22T22:05:37.936 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj155 2023-09-22T22:05:37.936 DEBUG:boto:Data: 2023-09-22T22:05:37.936 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.936 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.936 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.936 DEBUG:boto:Params: {} 2023-09-22T22:05:37.937 DEBUG:boto:Token: None 2023-09-22T22:05:37.937 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/ 2023-09-22T22:05:37.937 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:05:37.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:05:37.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007622abdf63face86-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.945 DEBUG:boto:b'ffzdqu-63obj1551000falseobj1552023-09-22T22:05:15.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:37.946 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:37.946 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:37.946 DEBUG:boto:path=/ffzdqu-63/?prefix=obj155 2023-09-22T22:05:37.946 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj155 2023-09-22T22:05:37.946 DEBUG:boto:Method: GET 2023-09-22T22:05:37.946 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj155 2023-09-22T22:05:37.946 DEBUG:boto:Data: 2023-09-22T22:05:37.946 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.946 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.947 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.947 DEBUG:boto:Params: {} 2023-09-22T22:05:37.947 DEBUG:boto:Token: None 2023-09-22T22:05:37.947 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/ 2023-09-22T22:05:37.947 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:05:37.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:05:37.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae12186613b9ff39-00650e0fb1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.958 DEBUG:boto:b'ffzdqu-63obj1551000falseobj1552023-09-22T22:05:15.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:37.958 DEBUG:boto:path=/ffzdqu-63/obj155 2023-09-22T22:05:37.958 DEBUG:boto:auth_path=/ffzdqu-63/obj155 2023-09-22T22:05:37.958 DEBUG:boto:Method: HEAD 2023-09-22T22:05:37.958 DEBUG:boto:Path: /ffzdqu-63/obj155 2023-09-22T22:05:37.959 DEBUG:boto:Data: 2023-09-22T22:05:37.959 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.959 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.959 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.959 DEBUG:boto:Params: {} 2023-09-22T22:05:37.959 DEBUG:boto:Token: None 2023-09-22T22:05:37.959 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj155 2023-09-22T22:05:37.959 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GjtWMwyl58/OW+4yn5UHOt+TGaY= 2023-09-22T22:05:37.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GjtWMwyl58/OW+4yn5UHOt+TGaY=', 'Content-Length': '0'} 2023-09-22T22:05:37.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000626186fb584522b5-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.963 DEBUG:boto:path=/ffzdqu-63/obj155 2023-09-22T22:05:37.963 DEBUG:boto:auth_path=/ffzdqu-63/obj155 2023-09-22T22:05:37.963 DEBUG:boto:Method: HEAD 2023-09-22T22:05:37.963 DEBUG:boto:Path: /ffzdqu-63/obj155 2023-09-22T22:05:37.963 DEBUG:boto:Data: 2023-09-22T22:05:37.963 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.963 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:37.963 DEBUG:boto:Port: 8000 2023-09-22T22:05:37.964 DEBUG:boto:Params: {} 2023-09-22T22:05:37.964 DEBUG:boto:Token: None 2023-09-22T22:05:37.964 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj155 2023-09-22T22:05:37.964 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GjtWMwyl58/OW+4yn5UHOt+TGaY= 2023-09-22T22:05:37.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GjtWMwyl58/OW+4yn5UHOt+TGaY=', 'Content-Length': '0'} 2023-09-22T22:05:37.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bdf64349edbdda0-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:37.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-22T22:05:37.967 DEBUG:boto:path=/ffzdqu-63/obj155 2023-09-22T22:05:37.967 DEBUG:boto:auth_path=/ffzdqu-63/obj155 2023-09-22T22:05:37.967 DEBUG:boto:Method: GET 2023-09-22T22:05:37.967 DEBUG:boto:Path: /ffzdqu-63/obj155 2023-09-22T22:05:37.968 DEBUG:boto:Data: 2023-09-22T22:05:37.968 DEBUG:boto:Headers: {} 2023-09-22T22:05:37.968 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:37.968 DEBUG:boto:Port: 8001 2023-09-22T22:05:37.968 DEBUG:boto:Params: {} 2023-09-22T22:05:37.968 DEBUG:boto:Token: None 2023-09-22T22:05:37.968 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:37 GMT /ffzdqu-63/obj155 2023-09-22T22:05:37.968 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:az/I6PpZgmMxAwxehRm7Q9ZcfvU= 2023-09-22T22:05:37.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:az/I6PpZgmMxAwxehRm7Q9ZcfvU=', 'Content-Length': '0'} 2023-09-22T22:05:37.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071c0e85a18aaf4f0-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.014 DEBUG:boto:path=/ffzdqu-63/obj155 2023-09-22T22:05:38.014 DEBUG:boto:auth_path=/ffzdqu-63/obj155 2023-09-22T22:05:38.015 DEBUG:boto:Method: GET 2023-09-22T22:05:38.015 DEBUG:boto:Path: /ffzdqu-63/obj155 2023-09-22T22:05:38.015 DEBUG:boto:Data: 2023-09-22T22:05:38.015 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.015 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.015 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.015 DEBUG:boto:Params: {} 2023-09-22T22:05:38.015 DEBUG:boto:Token: None 2023-09-22T22:05:38.015 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj155 2023-09-22T22:05:38.016 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dZcYrQQQgYXtouD5NHQgD66YT14= 2023-09-22T22:05:38.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dZcYrQQQgYXtouD5NHQgD66YT14=', 'Content-Length': '0'} 2023-09-22T22:05:38.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f3df1447a5e8f77-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.020 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:38.020 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:38.020 DEBUG:boto:path=/ffzdqu-63/?prefix=obj155 2023-09-22T22:05:38.020 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj155 2023-09-22T22:05:38.020 DEBUG:boto:Method: GET 2023-09-22T22:05:38.021 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj155 2023-09-22T22:05:38.021 DEBUG:boto:Data: 2023-09-22T22:05:38.021 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.021 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.021 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.021 DEBUG:boto:Params: {} 2023-09-22T22:05:38.021 DEBUG:boto:Token: None 2023-09-22T22:05:38.021 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/ 2023-09-22T22:05:38.021 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:05:38.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:05:38.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000787f3eab71dba3ea-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.027 DEBUG:boto:b'ffzdqu-63obj1551000falseobj1552023-09-22T22:05:15.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:38.027 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:38.027 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:38.027 DEBUG:boto:path=/ffzdqu-63/?prefix=obj155 2023-09-22T22:05:38.028 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj155 2023-09-22T22:05:38.028 DEBUG:boto:Method: GET 2023-09-22T22:05:38.028 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj155 2023-09-22T22:05:38.028 DEBUG:boto:Data: 2023-09-22T22:05:38.028 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.028 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.028 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.028 DEBUG:boto:Params: {} 2023-09-22T22:05:38.028 DEBUG:boto:Token: None 2023-09-22T22:05:38.029 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/ 2023-09-22T22:05:38.029 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:05:38.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:05:38.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cad3fff41d453911-00650e0fb2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.037 DEBUG:boto:b'ffzdqu-63obj1551000falseobj1552023-09-22T22:05:15.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:38.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T22:05:38.038 DEBUG:boto:path=/ffzdqu-63/obj17 2023-09-22T22:05:38.038 DEBUG:boto:auth_path=/ffzdqu-63/obj17 2023-09-22T22:05:38.038 DEBUG:boto:path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.038 DEBUG:boto:auth_path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.038 DEBUG:boto:Method: GET 2023-09-22T22:05:38.039 DEBUG:boto:Path: /ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.039 DEBUG:boto:Data: 2023-09-22T22:05:38.039 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.039 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.039 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.039 DEBUG:boto:Params: {} 2023-09-22T22:05:38.039 DEBUG:boto:Token: None 2023-09-22T22:05:38.039 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.039 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FRf/wY7RmkbHhpXzpcfDUiUdQQQ= 2023-09-22T22:05:38.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FRf/wY7RmkbHhpXzpcfDUiUdQQQ=', 'Content-Length': '0'} 2023-09-22T22:05:38.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa710e449efde951-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.043 DEBUG:boto:path=/ffzdqu-63/obj17 2023-09-22T22:05:38.043 DEBUG:boto:auth_path=/ffzdqu-63/obj17 2023-09-22T22:05:38.043 DEBUG:boto:path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.043 DEBUG:boto:auth_path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.043 DEBUG:boto:Method: GET 2023-09-22T22:05:38.043 DEBUG:boto:Path: /ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.043 DEBUG:boto:Data: 2023-09-22T22:05:38.044 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.044 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.044 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.044 DEBUG:boto:Params: {} 2023-09-22T22:05:38.044 DEBUG:boto:Token: None 2023-09-22T22:05:38.044 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.044 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FRf/wY7RmkbHhpXzpcfDUiUdQQQ= 2023-09-22T22:05:38.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FRf/wY7RmkbHhpXzpcfDUiUdQQQ=', 'Content-Length': '0'} 2023-09-22T22:05:38.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007961c31d9d1c856c-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.048 DEBUG:boto:path=/ffzdqu-63/obj17 2023-09-22T22:05:38.048 DEBUG:boto:auth_path=/ffzdqu-63/obj17 2023-09-22T22:05:38.048 DEBUG:boto:path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.048 DEBUG:boto:auth_path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.048 DEBUG:boto:Method: HEAD 2023-09-22T22:05:38.048 DEBUG:boto:Path: /ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.048 DEBUG:boto:Data: 2023-09-22T22:05:38.048 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.048 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.049 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.049 DEBUG:boto:Params: {} 2023-09-22T22:05:38.049 DEBUG:boto:Token: None 2023-09-22T22:05:38.049 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.049 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pUrmJMf7HZPsWZTeFGNt4v63WcY= 2023-09-22T22:05:38.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pUrmJMf7HZPsWZTeFGNt4v63WcY=', 'Content-Length': '0'} 2023-09-22T22:05:38.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090464193a7dfceaa-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.052 DEBUG:boto:path=/ffzdqu-63/obj17 2023-09-22T22:05:38.052 DEBUG:boto:auth_path=/ffzdqu-63/obj17 2023-09-22T22:05:38.052 DEBUG:boto:path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.053 DEBUG:boto:auth_path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.053 DEBUG:boto:Method: HEAD 2023-09-22T22:05:38.053 DEBUG:boto:Path: /ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.053 DEBUG:boto:Data: 2023-09-22T22:05:38.053 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.054 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.054 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.054 DEBUG:boto:Params: {} 2023-09-22T22:05:38.054 DEBUG:boto:Token: None 2023-09-22T22:05:38.055 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.055 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pUrmJMf7HZPsWZTeFGNt4v63WcY= 2023-09-22T22:05:38.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pUrmJMf7HZPsWZTeFGNt4v63WcY=', 'Content-Length': '0'} 2023-09-22T22:05:38.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008217709c9968f3b8-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T22:05:38.058 DEBUG:boto:path=/ffzdqu-63/obj17 2023-09-22T22:05:38.058 DEBUG:boto:auth_path=/ffzdqu-63/obj17 2023-09-22T22:05:38.059 DEBUG:boto:path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.059 DEBUG:boto:auth_path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.059 DEBUG:boto:Method: GET 2023-09-22T22:05:38.059 DEBUG:boto:Path: /ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.059 DEBUG:boto:Data: 2023-09-22T22:05:38.059 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.060 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.060 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.060 DEBUG:boto:Params: {} 2023-09-22T22:05:38.060 DEBUG:boto:Token: None 2023-09-22T22:05:38.060 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.060 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FRf/wY7RmkbHhpXzpcfDUiUdQQQ= 2023-09-22T22:05:38.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FRf/wY7RmkbHhpXzpcfDUiUdQQQ=', 'Content-Length': '0'} 2023-09-22T22:05:38.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dbcb1a587756744-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.106 DEBUG:boto:path=/ffzdqu-63/obj17 2023-09-22T22:05:38.107 DEBUG:boto:auth_path=/ffzdqu-63/obj17 2023-09-22T22:05:38.107 DEBUG:boto:path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.107 DEBUG:boto:auth_path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.107 DEBUG:boto:Method: GET 2023-09-22T22:05:38.107 DEBUG:boto:Path: /ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.107 DEBUG:boto:Data: 2023-09-22T22:05:38.107 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.107 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.107 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.108 DEBUG:boto:Params: {} 2023-09-22T22:05:38.108 DEBUG:boto:Token: None 2023-09-22T22:05:38.108 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj17?versionId=null 2023-09-22T22:05:38.108 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FRf/wY7RmkbHhpXzpcfDUiUdQQQ= 2023-09-22T22:05:38.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FRf/wY7RmkbHhpXzpcfDUiUdQQQ=', 'Content-Length': '0'} 2023-09-22T22:05:38.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4afffda0acfedaa-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.111 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:38.112 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:38.112 DEBUG:boto:path=/ffzdqu-63/?prefix=obj17 2023-09-22T22:05:38.112 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj17 2023-09-22T22:05:38.112 DEBUG:boto:Method: GET 2023-09-22T22:05:38.112 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj17 2023-09-22T22:05:38.112 DEBUG:boto:Data: 2023-09-22T22:05:38.112 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.112 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.113 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.113 DEBUG:boto:Params: {} 2023-09-22T22:05:38.113 DEBUG:boto:Token: None 2023-09-22T22:05:38.113 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/ 2023-09-22T22:05:38.113 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:05:38.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:05:38.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000309628ae6239c313-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.118 DEBUG:boto:b'ffzdqu-63obj171000falseobj172023-09-22T22:05:14.042Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:38.119 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:38.119 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:38.119 DEBUG:boto:path=/ffzdqu-63/?prefix=obj17 2023-09-22T22:05:38.119 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj17 2023-09-22T22:05:38.120 DEBUG:boto:Method: GET 2023-09-22T22:05:38.120 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj17 2023-09-22T22:05:38.120 DEBUG:boto:Data: 2023-09-22T22:05:38.120 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.120 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.120 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.120 DEBUG:boto:Params: {} 2023-09-22T22:05:38.120 DEBUG:boto:Token: None 2023-09-22T22:05:38.121 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/ 2023-09-22T22:05:38.121 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:05:38.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:05:38.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024391218acc26d08-00650e0fb2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.130 DEBUG:boto:b'ffzdqu-63obj171000falseobj172023-09-22T22:05:14.042Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:38.131 DEBUG:boto:path=/ffzdqu-63/obj17 2023-09-22T22:05:38.131 DEBUG:boto:auth_path=/ffzdqu-63/obj17 2023-09-22T22:05:38.131 DEBUG:boto:Method: HEAD 2023-09-22T22:05:38.131 DEBUG:boto:Path: /ffzdqu-63/obj17 2023-09-22T22:05:38.131 DEBUG:boto:Data: 2023-09-22T22:05:38.131 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.131 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.131 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.132 DEBUG:boto:Params: {} 2023-09-22T22:05:38.132 DEBUG:boto:Token: None 2023-09-22T22:05:38.132 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj17 2023-09-22T22:05:38.132 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JURMpqWay8zeev7BdMS8rF2HLj4= 2023-09-22T22:05:38.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JURMpqWay8zeev7BdMS8rF2HLj4=', 'Content-Length': '0'} 2023-09-22T22:05:38.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b3c8f8cbcf7b7e1-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.135 DEBUG:boto:path=/ffzdqu-63/obj17 2023-09-22T22:05:38.135 DEBUG:boto:auth_path=/ffzdqu-63/obj17 2023-09-22T22:05:38.135 DEBUG:boto:Method: HEAD 2023-09-22T22:05:38.135 DEBUG:boto:Path: /ffzdqu-63/obj17 2023-09-22T22:05:38.135 DEBUG:boto:Data: 2023-09-22T22:05:38.135 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.136 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.136 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.136 DEBUG:boto:Params: {} 2023-09-22T22:05:38.136 DEBUG:boto:Token: None 2023-09-22T22:05:38.136 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj17 2023-09-22T22:05:38.136 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JURMpqWay8zeev7BdMS8rF2HLj4= 2023-09-22T22:05:38.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JURMpqWay8zeev7BdMS8rF2HLj4=', 'Content-Length': '0'} 2023-09-22T22:05:38.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7a153910268140c-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-22T22:05:38.139 DEBUG:boto:path=/ffzdqu-63/obj17 2023-09-22T22:05:38.139 DEBUG:boto:auth_path=/ffzdqu-63/obj17 2023-09-22T22:05:38.139 DEBUG:boto:Method: GET 2023-09-22T22:05:38.139 DEBUG:boto:Path: /ffzdqu-63/obj17 2023-09-22T22:05:38.140 DEBUG:boto:Data: 2023-09-22T22:05:38.140 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.140 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.140 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.140 DEBUG:boto:Params: {} 2023-09-22T22:05:38.140 DEBUG:boto:Token: None 2023-09-22T22:05:38.140 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj17 2023-09-22T22:05:38.140 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:N4a+iMde9IqqVErDRW47vwIAn90= 2023-09-22T22:05:38.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:N4a+iMde9IqqVErDRW47vwIAn90=', 'Content-Length': '0'} 2023-09-22T22:05:38.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cf6c31b452be838-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.186 DEBUG:boto:path=/ffzdqu-63/obj17 2023-09-22T22:05:38.186 DEBUG:boto:auth_path=/ffzdqu-63/obj17 2023-09-22T22:05:38.187 DEBUG:boto:Method: GET 2023-09-22T22:05:38.187 DEBUG:boto:Path: /ffzdqu-63/obj17 2023-09-22T22:05:38.187 DEBUG:boto:Data: 2023-09-22T22:05:38.187 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.187 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.187 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.187 DEBUG:boto:Params: {} 2023-09-22T22:05:38.187 DEBUG:boto:Token: None 2023-09-22T22:05:38.187 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj17 2023-09-22T22:05:38.187 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:N4a+iMde9IqqVErDRW47vwIAn90= 2023-09-22T22:05:38.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:N4a+iMde9IqqVErDRW47vwIAn90=', 'Content-Length': '0'} 2023-09-22T22:05:38.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000374c37bb7c8dcab7-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.190 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:38.191 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:38.191 DEBUG:boto:path=/ffzdqu-63/?prefix=obj17 2023-09-22T22:05:38.191 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj17 2023-09-22T22:05:38.191 DEBUG:boto:Method: GET 2023-09-22T22:05:38.191 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj17 2023-09-22T22:05:38.191 DEBUG:boto:Data: 2023-09-22T22:05:38.191 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.191 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.191 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.192 DEBUG:boto:Params: {} 2023-09-22T22:05:38.192 DEBUG:boto:Token: None 2023-09-22T22:05:38.192 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/ 2023-09-22T22:05:38.192 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:05:38.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:05:38.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008482153ede7b7306-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.197 DEBUG:boto:b'ffzdqu-63obj171000falseobj172023-09-22T22:05:14.042Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:38.197 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:38.197 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:38.197 DEBUG:boto:path=/ffzdqu-63/?prefix=obj17 2023-09-22T22:05:38.198 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj17 2023-09-22T22:05:38.198 DEBUG:boto:Method: GET 2023-09-22T22:05:38.198 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj17 2023-09-22T22:05:38.198 DEBUG:boto:Data: 2023-09-22T22:05:38.198 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.198 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.198 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.198 DEBUG:boto:Params: {} 2023-09-22T22:05:38.198 DEBUG:boto:Token: None 2023-09-22T22:05:38.199 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/ 2023-09-22T22:05:38.199 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:05:38.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:05:38.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051a9847759cacb67-00650e0fb2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.250 DEBUG:boto:b'ffzdqu-63obj171000falseobj172023-09-22T22:05:14.042Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:38.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-09-22T22:05:38.251 DEBUG:boto:path=/ffzdqu-63/obj185 2023-09-22T22:05:38.251 DEBUG:boto:auth_path=/ffzdqu-63/obj185 2023-09-22T22:05:38.251 DEBUG:boto:path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.251 DEBUG:boto:auth_path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.251 DEBUG:boto:Method: GET 2023-09-22T22:05:38.251 DEBUG:boto:Path: /ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.252 DEBUG:boto:Data: 2023-09-22T22:05:38.252 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.252 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.252 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.252 DEBUG:boto:Params: {} 2023-09-22T22:05:38.252 DEBUG:boto:Token: None 2023-09-22T22:05:38.252 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.252 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cw136yPN3EsoInc2teF951FLeag= 2023-09-22T22:05:38.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cw136yPN3EsoInc2teF951FLeag=', 'Content-Length': '0'} 2023-09-22T22:05:38.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074e96ed73f3410f5-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.256 DEBUG:boto:path=/ffzdqu-63/obj185 2023-09-22T22:05:38.256 DEBUG:boto:auth_path=/ffzdqu-63/obj185 2023-09-22T22:05:38.256 DEBUG:boto:path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.256 DEBUG:boto:auth_path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.256 DEBUG:boto:Method: GET 2023-09-22T22:05:38.256 DEBUG:boto:Path: /ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.256 DEBUG:boto:Data: 2023-09-22T22:05:38.256 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.257 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.257 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.257 DEBUG:boto:Params: {} 2023-09-22T22:05:38.257 DEBUG:boto:Token: None 2023-09-22T22:05:38.257 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.257 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cw136yPN3EsoInc2teF951FLeag= 2023-09-22T22:05:38.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cw136yPN3EsoInc2teF951FLeag=', 'Content-Length': '0'} 2023-09-22T22:05:38.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007619a835c48002fe-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.260 DEBUG:boto:path=/ffzdqu-63/obj185 2023-09-22T22:05:38.260 DEBUG:boto:auth_path=/ffzdqu-63/obj185 2023-09-22T22:05:38.260 DEBUG:boto:path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.261 DEBUG:boto:auth_path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.261 DEBUG:boto:Method: HEAD 2023-09-22T22:05:38.261 DEBUG:boto:Path: /ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.261 DEBUG:boto:Data: 2023-09-22T22:05:38.261 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.261 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.261 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.261 DEBUG:boto:Params: {} 2023-09-22T22:05:38.261 DEBUG:boto:Token: None 2023-09-22T22:05:38.261 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.262 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TgKshhcY7np0AHaA0O2nEx92YTk= 2023-09-22T22:05:38.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TgKshhcY7np0AHaA0O2nEx92YTk=', 'Content-Length': '0'} 2023-09-22T22:05:38.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015d4abed2374518c-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.265 DEBUG:boto:path=/ffzdqu-63/obj185 2023-09-22T22:05:38.265 DEBUG:boto:auth_path=/ffzdqu-63/obj185 2023-09-22T22:05:38.265 DEBUG:boto:path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.265 DEBUG:boto:auth_path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.265 DEBUG:boto:Method: HEAD 2023-09-22T22:05:38.265 DEBUG:boto:Path: /ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.265 DEBUG:boto:Data: 2023-09-22T22:05:38.265 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.265 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.266 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.266 DEBUG:boto:Params: {} 2023-09-22T22:05:38.266 DEBUG:boto:Token: None 2023-09-22T22:05:38.266 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.266 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TgKshhcY7np0AHaA0O2nEx92YTk= 2023-09-22T22:05:38.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TgKshhcY7np0AHaA0O2nEx92YTk=', 'Content-Length': '0'} 2023-09-22T22:05:38.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f45c400ed775fe1f-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-09-22T22:05:38.269 DEBUG:boto:path=/ffzdqu-63/obj185 2023-09-22T22:05:38.269 DEBUG:boto:auth_path=/ffzdqu-63/obj185 2023-09-22T22:05:38.269 DEBUG:boto:path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.269 DEBUG:boto:auth_path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.269 DEBUG:boto:Method: GET 2023-09-22T22:05:38.269 DEBUG:boto:Path: /ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.270 DEBUG:boto:Data: 2023-09-22T22:05:38.270 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.270 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.270 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.270 DEBUG:boto:Params: {} 2023-09-22T22:05:38.270 DEBUG:boto:Token: None 2023-09-22T22:05:38.270 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.270 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cw136yPN3EsoInc2teF951FLeag= 2023-09-22T22:05:38.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cw136yPN3EsoInc2teF951FLeag=', 'Content-Length': '0'} 2023-09-22T22:05:38.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af733d122f929e5b-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.274 DEBUG:boto:path=/ffzdqu-63/obj185 2023-09-22T22:05:38.274 DEBUG:boto:auth_path=/ffzdqu-63/obj185 2023-09-22T22:05:38.274 DEBUG:boto:path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.274 DEBUG:boto:auth_path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.274 DEBUG:boto:Method: GET 2023-09-22T22:05:38.274 DEBUG:boto:Path: /ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.275 DEBUG:boto:Data: 2023-09-22T22:05:38.275 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.275 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.275 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.275 DEBUG:boto:Params: {} 2023-09-22T22:05:38.275 DEBUG:boto:Token: None 2023-09-22T22:05:38.275 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj185?versionId=null 2023-09-22T22:05:38.276 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cw136yPN3EsoInc2teF951FLeag= 2023-09-22T22:05:38.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cw136yPN3EsoInc2teF951FLeag=', 'Content-Length': '0'} 2023-09-22T22:05:38.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1b70c448a34635f-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.279 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:38.279 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:38.279 DEBUG:boto:path=/ffzdqu-63/?prefix=obj185 2023-09-22T22:05:38.279 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj185 2023-09-22T22:05:38.279 DEBUG:boto:Method: GET 2023-09-22T22:05:38.279 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj185 2023-09-22T22:05:38.280 DEBUG:boto:Data: 2023-09-22T22:05:38.280 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.280 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.280 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.280 DEBUG:boto:Params: {} 2023-09-22T22:05:38.280 DEBUG:boto:Token: None 2023-09-22T22:05:38.280 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/ 2023-09-22T22:05:38.280 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:05:38.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:05:38.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2ea3bdbb459f85e-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.286 DEBUG:boto:b'ffzdqu-63obj1851000falseobj1852023-09-22T22:05:16.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:38.286 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:38.286 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:38.286 DEBUG:boto:path=/ffzdqu-63/?prefix=obj185 2023-09-22T22:05:38.286 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj185 2023-09-22T22:05:38.287 DEBUG:boto:Method: GET 2023-09-22T22:05:38.287 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj185 2023-09-22T22:05:38.287 DEBUG:boto:Data: 2023-09-22T22:05:38.287 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.287 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.287 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.287 DEBUG:boto:Params: {} 2023-09-22T22:05:38.287 DEBUG:boto:Token: None 2023-09-22T22:05:38.288 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/ 2023-09-22T22:05:38.288 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:05:38.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:05:38.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fb4b6d083cae21f-00650e0fb2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.343 DEBUG:boto:b'ffzdqu-63obj1851000falseobj1852023-09-22T22:05:16.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:38.385 DEBUG:boto:path=/ffzdqu-63/obj185 2023-09-22T22:05:38.385 DEBUG:boto:auth_path=/ffzdqu-63/obj185 2023-09-22T22:05:38.385 DEBUG:boto:Method: HEAD 2023-09-22T22:05:38.385 DEBUG:boto:Path: /ffzdqu-63/obj185 2023-09-22T22:05:38.385 DEBUG:boto:Data: 2023-09-22T22:05:38.386 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.386 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.386 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.386 DEBUG:boto:Params: {} 2023-09-22T22:05:38.386 DEBUG:boto:Token: None 2023-09-22T22:05:38.386 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj185 2023-09-22T22:05:38.386 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pMg3V+2j+fAJ1vliynRgm8oFlUs= 2023-09-22T22:05:38.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pMg3V+2j+fAJ1vliynRgm8oFlUs=', 'Content-Length': '0'} 2023-09-22T22:05:38.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006720f8912ebbbef0-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.389 DEBUG:boto:path=/ffzdqu-63/obj185 2023-09-22T22:05:38.390 DEBUG:boto:auth_path=/ffzdqu-63/obj185 2023-09-22T22:05:38.390 DEBUG:boto:Method: HEAD 2023-09-22T22:05:38.390 DEBUG:boto:Path: /ffzdqu-63/obj185 2023-09-22T22:05:38.390 DEBUG:boto:Data: 2023-09-22T22:05:38.390 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.390 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.390 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.390 DEBUG:boto:Params: {} 2023-09-22T22:05:38.391 DEBUG:boto:Token: None 2023-09-22T22:05:38.391 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj185 2023-09-22T22:05:38.391 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pMg3V+2j+fAJ1vliynRgm8oFlUs= 2023-09-22T22:05:38.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pMg3V+2j+fAJ1vliynRgm8oFlUs=', 'Content-Length': '0'} 2023-09-22T22:05:38.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f615d8c8d04353a8-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-09-22T22:05:38.394 DEBUG:boto:path=/ffzdqu-63/obj185 2023-09-22T22:05:38.394 DEBUG:boto:auth_path=/ffzdqu-63/obj185 2023-09-22T22:05:38.394 DEBUG:boto:Method: GET 2023-09-22T22:05:38.394 DEBUG:boto:Path: /ffzdqu-63/obj185 2023-09-22T22:05:38.394 DEBUG:boto:Data: 2023-09-22T22:05:38.395 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.395 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.395 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.395 DEBUG:boto:Params: {} 2023-09-22T22:05:38.395 DEBUG:boto:Token: None 2023-09-22T22:05:38.395 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj185 2023-09-22T22:05:38.395 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uLkr6f+CxMIS95Ctz0AKfhpOxOk= 2023-09-22T22:05:38.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uLkr6f+CxMIS95Ctz0AKfhpOxOk=', 'Content-Length': '0'} 2023-09-22T22:05:38.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5569765fa8460cd-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.398 DEBUG:boto:path=/ffzdqu-63/obj185 2023-09-22T22:05:38.399 DEBUG:boto:auth_path=/ffzdqu-63/obj185 2023-09-22T22:05:38.399 DEBUG:boto:Method: GET 2023-09-22T22:05:38.399 DEBUG:boto:Path: /ffzdqu-63/obj185 2023-09-22T22:05:38.399 DEBUG:boto:Data: 2023-09-22T22:05:38.399 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.399 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.399 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.399 DEBUG:boto:Params: {} 2023-09-22T22:05:38.399 DEBUG:boto:Token: None 2023-09-22T22:05:38.399 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj185 2023-09-22T22:05:38.400 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uLkr6f+CxMIS95Ctz0AKfhpOxOk= 2023-09-22T22:05:38.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uLkr6f+CxMIS95Ctz0AKfhpOxOk=', 'Content-Length': '0'} 2023-09-22T22:05:38.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047093f47440eeefb-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.403 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:38.403 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:38.403 DEBUG:boto:path=/ffzdqu-63/?prefix=obj185 2023-09-22T22:05:38.403 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj185 2023-09-22T22:05:38.403 DEBUG:boto:Method: GET 2023-09-22T22:05:38.403 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj185 2023-09-22T22:05:38.403 DEBUG:boto:Data: 2023-09-22T22:05:38.403 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.403 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.403 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.404 DEBUG:boto:Params: {} 2023-09-22T22:05:38.404 DEBUG:boto:Token: None 2023-09-22T22:05:38.404 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/ 2023-09-22T22:05:38.404 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:05:38.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:05:38.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac8a4e422609233f-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.410 DEBUG:boto:b'ffzdqu-63obj1851000falseobj1852023-09-22T22:05:16.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:38.410 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:38.410 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:38.410 DEBUG:boto:path=/ffzdqu-63/?prefix=obj185 2023-09-22T22:05:38.410 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj185 2023-09-22T22:05:38.410 DEBUG:boto:Method: GET 2023-09-22T22:05:38.411 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj185 2023-09-22T22:05:38.411 DEBUG:boto:Data: 2023-09-22T22:05:38.411 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.411 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.411 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.411 DEBUG:boto:Params: {} 2023-09-22T22:05:38.411 DEBUG:boto:Token: None 2023-09-22T22:05:38.411 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/ 2023-09-22T22:05:38.411 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:05:38.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:05:38.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087cd61095a07fe70-00650e0fb2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.422 DEBUG:boto:b'ffzdqu-63obj1851000falseobj1852023-09-22T22:05:16.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:38.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T22:05:38.422 DEBUG:boto:path=/ffzdqu-63/obj19 2023-09-22T22:05:38.423 DEBUG:boto:auth_path=/ffzdqu-63/obj19 2023-09-22T22:05:38.423 DEBUG:boto:path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.423 DEBUG:boto:auth_path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.423 DEBUG:boto:Method: GET 2023-09-22T22:05:38.423 DEBUG:boto:Path: /ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.423 DEBUG:boto:Data: 2023-09-22T22:05:38.423 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.423 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.423 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.423 DEBUG:boto:Params: {} 2023-09-22T22:05:38.424 DEBUG:boto:Token: None 2023-09-22T22:05:38.424 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.424 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YP7fbermt89LPn4/xK1q24DFRMs= 2023-09-22T22:05:38.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YP7fbermt89LPn4/xK1q24DFRMs=', 'Content-Length': '0'} 2023-09-22T22:05:38.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006c0d8d322b2ee91-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.470 DEBUG:boto:path=/ffzdqu-63/obj19 2023-09-22T22:05:38.471 DEBUG:boto:auth_path=/ffzdqu-63/obj19 2023-09-22T22:05:38.471 DEBUG:boto:path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.471 DEBUG:boto:auth_path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.471 DEBUG:boto:Method: GET 2023-09-22T22:05:38.471 DEBUG:boto:Path: /ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.471 DEBUG:boto:Data: 2023-09-22T22:05:38.471 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.471 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.471 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.471 DEBUG:boto:Params: {} 2023-09-22T22:05:38.471 DEBUG:boto:Token: None 2023-09-22T22:05:38.472 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.472 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YP7fbermt89LPn4/xK1q24DFRMs= 2023-09-22T22:05:38.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YP7fbermt89LPn4/xK1q24DFRMs=', 'Content-Length': '0'} 2023-09-22T22:05:38.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002aad02e3de7e30fa-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.475 DEBUG:boto:path=/ffzdqu-63/obj19 2023-09-22T22:05:38.475 DEBUG:boto:auth_path=/ffzdqu-63/obj19 2023-09-22T22:05:38.475 DEBUG:boto:path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.476 DEBUG:boto:auth_path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.476 DEBUG:boto:Method: HEAD 2023-09-22T22:05:38.476 DEBUG:boto:Path: /ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.476 DEBUG:boto:Data: 2023-09-22T22:05:38.476 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.476 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.476 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.476 DEBUG:boto:Params: {} 2023-09-22T22:05:38.476 DEBUG:boto:Token: None 2023-09-22T22:05:38.476 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.477 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4igrPJ24sXQIlAPRW8xDu1FZp6s= 2023-09-22T22:05:38.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4igrPJ24sXQIlAPRW8xDu1FZp6s=', 'Content-Length': '0'} 2023-09-22T22:05:38.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000477c565dca241f72-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.480 DEBUG:boto:path=/ffzdqu-63/obj19 2023-09-22T22:05:38.480 DEBUG:boto:auth_path=/ffzdqu-63/obj19 2023-09-22T22:05:38.480 DEBUG:boto:path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.480 DEBUG:boto:auth_path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.480 DEBUG:boto:Method: HEAD 2023-09-22T22:05:38.480 DEBUG:boto:Path: /ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.480 DEBUG:boto:Data: 2023-09-22T22:05:38.480 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.481 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.481 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.481 DEBUG:boto:Params: {} 2023-09-22T22:05:38.481 DEBUG:boto:Token: None 2023-09-22T22:05:38.481 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.481 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4igrPJ24sXQIlAPRW8xDu1FZp6s= 2023-09-22T22:05:38.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4igrPJ24sXQIlAPRW8xDu1FZp6s=', 'Content-Length': '0'} 2023-09-22T22:05:38.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002163748f8b65c586-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T22:05:38.484 DEBUG:boto:path=/ffzdqu-63/obj19 2023-09-22T22:05:38.484 DEBUG:boto:auth_path=/ffzdqu-63/obj19 2023-09-22T22:05:38.484 DEBUG:boto:path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.484 DEBUG:boto:auth_path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.484 DEBUG:boto:Method: GET 2023-09-22T22:05:38.484 DEBUG:boto:Path: /ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.485 DEBUG:boto:Data: 2023-09-22T22:05:38.485 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.485 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.485 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.485 DEBUG:boto:Params: {} 2023-09-22T22:05:38.485 DEBUG:boto:Token: None 2023-09-22T22:05:38.485 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.485 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YP7fbermt89LPn4/xK1q24DFRMs= 2023-09-22T22:05:38.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YP7fbermt89LPn4/xK1q24DFRMs=', 'Content-Length': '0'} 2023-09-22T22:05:38.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068fb07adcd7fb493-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.488 DEBUG:boto:path=/ffzdqu-63/obj19 2023-09-22T22:05:38.488 DEBUG:boto:auth_path=/ffzdqu-63/obj19 2023-09-22T22:05:38.488 DEBUG:boto:path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.489 DEBUG:boto:auth_path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.489 DEBUG:boto:Method: GET 2023-09-22T22:05:38.489 DEBUG:boto:Path: /ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.489 DEBUG:boto:Data: 2023-09-22T22:05:38.489 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.489 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.489 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.489 DEBUG:boto:Params: {} 2023-09-22T22:05:38.489 DEBUG:boto:Token: None 2023-09-22T22:05:38.490 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj19?versionId=null 2023-09-22T22:05:38.490 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YP7fbermt89LPn4/xK1q24DFRMs= 2023-09-22T22:05:38.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YP7fbermt89LPn4/xK1q24DFRMs=', 'Content-Length': '0'} 2023-09-22T22:05:38.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d49be1a739a4bea-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.534 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:38.535 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:38.535 DEBUG:boto:path=/ffzdqu-63/?prefix=obj19 2023-09-22T22:05:38.535 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj19 2023-09-22T22:05:38.535 DEBUG:boto:Method: GET 2023-09-22T22:05:38.535 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj19 2023-09-22T22:05:38.535 DEBUG:boto:Data: 2023-09-22T22:05:38.535 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.536 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.536 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.536 DEBUG:boto:Params: {} 2023-09-22T22:05:38.536 DEBUG:boto:Token: None 2023-09-22T22:05:38.536 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/ 2023-09-22T22:05:38.536 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:05:38.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:05:38.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8d0a53470dd0d8f-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.541 DEBUG:boto:b'ffzdqu-63obj191000falseobj192023-09-22T22:05:14.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:38.542 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:38.542 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:38.542 DEBUG:boto:path=/ffzdqu-63/?prefix=obj19 2023-09-22T22:05:38.542 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj19 2023-09-22T22:05:38.542 DEBUG:boto:Method: GET 2023-09-22T22:05:38.542 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj19 2023-09-22T22:05:38.543 DEBUG:boto:Data: 2023-09-22T22:05:38.543 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.543 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.543 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.543 DEBUG:boto:Params: {} 2023-09-22T22:05:38.543 DEBUG:boto:Token: None 2023-09-22T22:05:38.543 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/ 2023-09-22T22:05:38.543 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:05:38.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:05:38.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000877baa40f1e0d04d-00650e0fb2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.555 DEBUG:boto:b'ffzdqu-63obj191000falseobj192023-09-22T22:05:14.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:38.555 DEBUG:boto:path=/ffzdqu-63/obj19 2023-09-22T22:05:38.556 DEBUG:boto:auth_path=/ffzdqu-63/obj19 2023-09-22T22:05:38.556 DEBUG:boto:Method: HEAD 2023-09-22T22:05:38.556 DEBUG:boto:Path: /ffzdqu-63/obj19 2023-09-22T22:05:38.556 DEBUG:boto:Data: 2023-09-22T22:05:38.556 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.556 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.557 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.557 DEBUG:boto:Params: {} 2023-09-22T22:05:38.557 DEBUG:boto:Token: None 2023-09-22T22:05:38.557 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj19 2023-09-22T22:05:38.557 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jsJlU8zOjTlocXj8qntbg0oX9hk= 2023-09-22T22:05:38.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jsJlU8zOjTlocXj8qntbg0oX9hk=', 'Content-Length': '0'} 2023-09-22T22:05:38.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a1630da456aff9a-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.561 DEBUG:boto:path=/ffzdqu-63/obj19 2023-09-22T22:05:38.561 DEBUG:boto:auth_path=/ffzdqu-63/obj19 2023-09-22T22:05:38.561 DEBUG:boto:Method: HEAD 2023-09-22T22:05:38.561 DEBUG:boto:Path: /ffzdqu-63/obj19 2023-09-22T22:05:38.562 DEBUG:boto:Data: 2023-09-22T22:05:38.562 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.562 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.562 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.562 DEBUG:boto:Params: {} 2023-09-22T22:05:38.563 DEBUG:boto:Token: None 2023-09-22T22:05:38.563 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj19 2023-09-22T22:05:38.563 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jsJlU8zOjTlocXj8qntbg0oX9hk= 2023-09-22T22:05:38.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jsJlU8zOjTlocXj8qntbg0oX9hk=', 'Content-Length': '0'} 2023-09-22T22:05:38.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007060795459fb0b70-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-22T22:05:38.566 DEBUG:boto:path=/ffzdqu-63/obj19 2023-09-22T22:05:38.566 DEBUG:boto:auth_path=/ffzdqu-63/obj19 2023-09-22T22:05:38.566 DEBUG:boto:Method: GET 2023-09-22T22:05:38.567 DEBUG:boto:Path: /ffzdqu-63/obj19 2023-09-22T22:05:38.567 DEBUG:boto:Data: 2023-09-22T22:05:38.567 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.567 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.567 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.567 DEBUG:boto:Params: {} 2023-09-22T22:05:38.567 DEBUG:boto:Token: None 2023-09-22T22:05:38.567 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj19 2023-09-22T22:05:38.568 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:U3a7Gj49WDb/5DJDVH+bLe1jSqs= 2023-09-22T22:05:38.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:U3a7Gj49WDb/5DJDVH+bLe1jSqs=', 'Content-Length': '0'} 2023-09-22T22:05:38.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005337e483281872d2-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.614 DEBUG:boto:path=/ffzdqu-63/obj19 2023-09-22T22:05:38.614 DEBUG:boto:auth_path=/ffzdqu-63/obj19 2023-09-22T22:05:38.615 DEBUG:boto:Method: GET 2023-09-22T22:05:38.615 DEBUG:boto:Path: /ffzdqu-63/obj19 2023-09-22T22:05:38.615 DEBUG:boto:Data: 2023-09-22T22:05:38.615 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.615 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.615 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.615 DEBUG:boto:Params: {} 2023-09-22T22:05:38.615 DEBUG:boto:Token: None 2023-09-22T22:05:38.616 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj19 2023-09-22T22:05:38.616 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:U3a7Gj49WDb/5DJDVH+bLe1jSqs= 2023-09-22T22:05:38.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:U3a7Gj49WDb/5DJDVH+bLe1jSqs=', 'Content-Length': '0'} 2023-09-22T22:05:38.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000192d55f07a15447e-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.619 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:38.619 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:38.619 DEBUG:boto:path=/ffzdqu-63/?prefix=obj19 2023-09-22T22:05:38.620 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj19 2023-09-22T22:05:38.620 DEBUG:boto:Method: GET 2023-09-22T22:05:38.620 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj19 2023-09-22T22:05:38.620 DEBUG:boto:Data: 2023-09-22T22:05:38.620 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.620 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.620 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.620 DEBUG:boto:Params: {} 2023-09-22T22:05:38.620 DEBUG:boto:Token: None 2023-09-22T22:05:38.621 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/ 2023-09-22T22:05:38.621 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:05:38.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:05:38.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f0879996eae6d01-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.626 DEBUG:boto:b'ffzdqu-63obj191000falseobj192023-09-22T22:05:14.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:38.626 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:38.627 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:38.627 DEBUG:boto:path=/ffzdqu-63/?prefix=obj19 2023-09-22T22:05:38.627 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj19 2023-09-22T22:05:38.627 DEBUG:boto:Method: GET 2023-09-22T22:05:38.627 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj19 2023-09-22T22:05:38.627 DEBUG:boto:Data: 2023-09-22T22:05:38.627 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.627 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.627 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.628 DEBUG:boto:Params: {} 2023-09-22T22:05:38.628 DEBUG:boto:Token: None 2023-09-22T22:05:38.628 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/ 2023-09-22T22:05:38.628 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:05:38.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:05:38.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080973d1f9001190c-00650e0fb2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.682 DEBUG:boto:b'ffzdqu-63obj191000falseobj192023-09-22T22:05:14.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:38.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T22:05:38.683 DEBUG:boto:path=/ffzdqu-63/obj22 2023-09-22T22:05:38.683 DEBUG:boto:auth_path=/ffzdqu-63/obj22 2023-09-22T22:05:38.683 DEBUG:boto:path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.683 DEBUG:boto:auth_path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.684 DEBUG:boto:Method: GET 2023-09-22T22:05:38.684 DEBUG:boto:Path: /ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.684 DEBUG:boto:Data: 2023-09-22T22:05:38.684 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.684 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.684 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.684 DEBUG:boto:Params: {} 2023-09-22T22:05:38.684 DEBUG:boto:Token: None 2023-09-22T22:05:38.684 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.685 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uKUdqjUmyu3eKy806idSG+PkgVg= 2023-09-22T22:05:38.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKUdqjUmyu3eKy806idSG+PkgVg=', 'Content-Length': '0'} 2023-09-22T22:05:38.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005494c9e69654ad7b-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.688 DEBUG:boto:path=/ffzdqu-63/obj22 2023-09-22T22:05:38.688 DEBUG:boto:auth_path=/ffzdqu-63/obj22 2023-09-22T22:05:38.688 DEBUG:boto:path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.688 DEBUG:boto:auth_path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.688 DEBUG:boto:Method: GET 2023-09-22T22:05:38.689 DEBUG:boto:Path: /ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.689 DEBUG:boto:Data: 2023-09-22T22:05:38.689 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.689 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.689 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.689 DEBUG:boto:Params: {} 2023-09-22T22:05:38.689 DEBUG:boto:Token: None 2023-09-22T22:05:38.689 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.690 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uKUdqjUmyu3eKy806idSG+PkgVg= 2023-09-22T22:05:38.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKUdqjUmyu3eKy806idSG+PkgVg=', 'Content-Length': '0'} 2023-09-22T22:05:38.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd697dbffbdb0d19-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.693 DEBUG:boto:path=/ffzdqu-63/obj22 2023-09-22T22:05:38.693 DEBUG:boto:auth_path=/ffzdqu-63/obj22 2023-09-22T22:05:38.693 DEBUG:boto:path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.694 DEBUG:boto:auth_path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.694 DEBUG:boto:Method: HEAD 2023-09-22T22:05:38.694 DEBUG:boto:Path: /ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.694 DEBUG:boto:Data: 2023-09-22T22:05:38.694 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.694 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.694 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.694 DEBUG:boto:Params: {} 2023-09-22T22:05:38.694 DEBUG:boto:Token: None 2023-09-22T22:05:38.695 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.695 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5pjWoqz+Io6DTZ0ox/6cWXkKUl8= 2023-09-22T22:05:38.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5pjWoqz+Io6DTZ0ox/6cWXkKUl8=', 'Content-Length': '0'} 2023-09-22T22:05:38.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ec95bff7049be2d-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.698 DEBUG:boto:path=/ffzdqu-63/obj22 2023-09-22T22:05:38.698 DEBUG:boto:auth_path=/ffzdqu-63/obj22 2023-09-22T22:05:38.698 DEBUG:boto:path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.698 DEBUG:boto:auth_path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.698 DEBUG:boto:Method: HEAD 2023-09-22T22:05:38.698 DEBUG:boto:Path: /ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.698 DEBUG:boto:Data: 2023-09-22T22:05:38.698 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.699 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.699 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.699 DEBUG:boto:Params: {} 2023-09-22T22:05:38.699 DEBUG:boto:Token: None 2023-09-22T22:05:38.699 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.699 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:5pjWoqz+Io6DTZ0ox/6cWXkKUl8= 2023-09-22T22:05:38.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5pjWoqz+Io6DTZ0ox/6cWXkKUl8=', 'Content-Length': '0'} 2023-09-22T22:05:38.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b45d0d6140c188e1-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T22:05:38.703 DEBUG:boto:path=/ffzdqu-63/obj22 2023-09-22T22:05:38.703 DEBUG:boto:auth_path=/ffzdqu-63/obj22 2023-09-22T22:05:38.704 DEBUG:boto:path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.704 DEBUG:boto:auth_path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.704 DEBUG:boto:Method: GET 2023-09-22T22:05:38.704 DEBUG:boto:Path: /ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.705 DEBUG:boto:Data: 2023-09-22T22:05:38.705 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.705 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.705 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.705 DEBUG:boto:Params: {} 2023-09-22T22:05:38.706 DEBUG:boto:Token: None 2023-09-22T22:05:38.706 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.706 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uKUdqjUmyu3eKy806idSG+PkgVg= 2023-09-22T22:05:38.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKUdqjUmyu3eKy806idSG+PkgVg=', 'Content-Length': '0'} 2023-09-22T22:05:38.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000453b478858be92eb-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.710 DEBUG:boto:path=/ffzdqu-63/obj22 2023-09-22T22:05:38.710 DEBUG:boto:auth_path=/ffzdqu-63/obj22 2023-09-22T22:05:38.710 DEBUG:boto:path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.710 DEBUG:boto:auth_path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.710 DEBUG:boto:Method: GET 2023-09-22T22:05:38.710 DEBUG:boto:Path: /ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.710 DEBUG:boto:Data: 2023-09-22T22:05:38.711 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.711 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.711 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.711 DEBUG:boto:Params: {} 2023-09-22T22:05:38.711 DEBUG:boto:Token: None 2023-09-22T22:05:38.711 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj22?versionId=null 2023-09-22T22:05:38.711 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uKUdqjUmyu3eKy806idSG+PkgVg= 2023-09-22T22:05:38.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKUdqjUmyu3eKy806idSG+PkgVg=', 'Content-Length': '0'} 2023-09-22T22:05:38.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c77a6ee1502bbb4-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.714 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:38.714 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:38.714 DEBUG:boto:path=/ffzdqu-63/?prefix=obj22 2023-09-22T22:05:38.715 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj22 2023-09-22T22:05:38.715 DEBUG:boto:Method: GET 2023-09-22T22:05:38.715 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj22 2023-09-22T22:05:38.715 DEBUG:boto:Data: 2023-09-22T22:05:38.715 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.715 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.715 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.715 DEBUG:boto:Params: {} 2023-09-22T22:05:38.715 DEBUG:boto:Token: None 2023-09-22T22:05:38.716 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/ 2023-09-22T22:05:38.716 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:05:38.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:05:38.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005da0e44ce556cbe4-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.721 DEBUG:boto:b'ffzdqu-63obj221000falseobj222023-09-22T22:05:13.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:38.721 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:38.721 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:38.721 DEBUG:boto:path=/ffzdqu-63/?prefix=obj22 2023-09-22T22:05:38.721 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj22 2023-09-22T22:05:38.722 DEBUG:boto:Method: GET 2023-09-22T22:05:38.722 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj22 2023-09-22T22:05:38.722 DEBUG:boto:Data: 2023-09-22T22:05:38.722 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.722 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.722 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.722 DEBUG:boto:Params: {} 2023-09-22T22:05:38.722 DEBUG:boto:Token: None 2023-09-22T22:05:38.723 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/ 2023-09-22T22:05:38.723 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:05:38.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:05:38.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029b9e6cfb0d99493-00650e0fb2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.734 DEBUG:boto:b'ffzdqu-63obj221000falseobj222023-09-22T22:05:13.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:38.734 DEBUG:boto:path=/ffzdqu-63/obj22 2023-09-22T22:05:38.735 DEBUG:boto:auth_path=/ffzdqu-63/obj22 2023-09-22T22:05:38.735 DEBUG:boto:Method: HEAD 2023-09-22T22:05:38.735 DEBUG:boto:Path: /ffzdqu-63/obj22 2023-09-22T22:05:38.735 DEBUG:boto:Data: 2023-09-22T22:05:38.735 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.735 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.735 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.735 DEBUG:boto:Params: {} 2023-09-22T22:05:38.736 DEBUG:boto:Token: None 2023-09-22T22:05:38.736 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj22 2023-09-22T22:05:38.736 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6/8iwinmk9X8X5xu2R82CFUr+ao= 2023-09-22T22:05:38.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6/8iwinmk9X8X5xu2R82CFUr+ao=', 'Content-Length': '0'} 2023-09-22T22:05:38.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be68dcc1b349e694-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.739 DEBUG:boto:path=/ffzdqu-63/obj22 2023-09-22T22:05:38.739 DEBUG:boto:auth_path=/ffzdqu-63/obj22 2023-09-22T22:05:38.740 DEBUG:boto:Method: HEAD 2023-09-22T22:05:38.740 DEBUG:boto:Path: /ffzdqu-63/obj22 2023-09-22T22:05:38.740 DEBUG:boto:Data: 2023-09-22T22:05:38.741 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.741 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.741 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.741 DEBUG:boto:Params: {} 2023-09-22T22:05:38.741 DEBUG:boto:Token: None 2023-09-22T22:05:38.742 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj22 2023-09-22T22:05:38.742 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6/8iwinmk9X8X5xu2R82CFUr+ao= 2023-09-22T22:05:38.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6/8iwinmk9X8X5xu2R82CFUr+ao=', 'Content-Length': '0'} 2023-09-22T22:05:38.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6140c99c441ef01-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T22:05:38.745 DEBUG:boto:path=/ffzdqu-63/obj22 2023-09-22T22:05:38.745 DEBUG:boto:auth_path=/ffzdqu-63/obj22 2023-09-22T22:05:38.746 DEBUG:boto:Method: GET 2023-09-22T22:05:38.746 DEBUG:boto:Path: /ffzdqu-63/obj22 2023-09-22T22:05:38.746 DEBUG:boto:Data: 2023-09-22T22:05:38.746 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.746 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.746 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.746 DEBUG:boto:Params: {} 2023-09-22T22:05:38.746 DEBUG:boto:Token: None 2023-09-22T22:05:38.747 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj22 2023-09-22T22:05:38.747 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UFltGioGQCl7b08erZxcTqoxwY0= 2023-09-22T22:05:38.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UFltGioGQCl7b08erZxcTqoxwY0=', 'Content-Length': '0'} 2023-09-22T22:05:38.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae9bc5452efc153c-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.794 DEBUG:boto:path=/ffzdqu-63/obj22 2023-09-22T22:05:38.795 DEBUG:boto:auth_path=/ffzdqu-63/obj22 2023-09-22T22:05:38.795 DEBUG:boto:Method: GET 2023-09-22T22:05:38.795 DEBUG:boto:Path: /ffzdqu-63/obj22 2023-09-22T22:05:38.795 DEBUG:boto:Data: 2023-09-22T22:05:38.795 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.795 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.795 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.795 DEBUG:boto:Params: {} 2023-09-22T22:05:38.795 DEBUG:boto:Token: None 2023-09-22T22:05:38.796 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj22 2023-09-22T22:05:38.796 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UFltGioGQCl7b08erZxcTqoxwY0= 2023-09-22T22:05:38.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UFltGioGQCl7b08erZxcTqoxwY0=', 'Content-Length': '0'} 2023-09-22T22:05:38.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8e1f06e0f5b9b38-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.799 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:38.799 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:38.799 DEBUG:boto:path=/ffzdqu-63/?prefix=obj22 2023-09-22T22:05:38.799 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj22 2023-09-22T22:05:38.799 DEBUG:boto:Method: GET 2023-09-22T22:05:38.799 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj22 2023-09-22T22:05:38.799 DEBUG:boto:Data: 2023-09-22T22:05:38.800 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.800 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.800 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.800 DEBUG:boto:Params: {} 2023-09-22T22:05:38.800 DEBUG:boto:Token: None 2023-09-22T22:05:38.800 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/ 2023-09-22T22:05:38.801 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:05:38.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:05:38.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087068558de212b1f-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.806 DEBUG:boto:b'ffzdqu-63obj221000falseobj222023-09-22T22:05:13.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:38.807 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:38.807 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:38.807 DEBUG:boto:path=/ffzdqu-63/?prefix=obj22 2023-09-22T22:05:38.807 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj22 2023-09-22T22:05:38.807 DEBUG:boto:Method: GET 2023-09-22T22:05:38.808 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj22 2023-09-22T22:05:38.808 DEBUG:boto:Data: 2023-09-22T22:05:38.808 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.808 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.808 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.808 DEBUG:boto:Params: {} 2023-09-22T22:05:38.809 DEBUG:boto:Token: None 2023-09-22T22:05:38.809 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/ 2023-09-22T22:05:38.809 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:05:38.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:05:38.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c23f3d6d02b9e03e-00650e0fb2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.819 DEBUG:boto:b'ffzdqu-63obj221000falseobj222023-09-22T22:05:13.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:38.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-22T22:05:38.820 DEBUG:boto:path=/ffzdqu-63/obj23 2023-09-22T22:05:38.820 DEBUG:boto:auth_path=/ffzdqu-63/obj23 2023-09-22T22:05:38.820 DEBUG:boto:path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.820 DEBUG:boto:auth_path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.820 DEBUG:boto:Method: GET 2023-09-22T22:05:38.820 DEBUG:boto:Path: /ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.821 DEBUG:boto:Data: 2023-09-22T22:05:38.821 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.821 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.821 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.821 DEBUG:boto:Params: {} 2023-09-22T22:05:38.821 DEBUG:boto:Token: None 2023-09-22T22:05:38.821 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.821 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oxli9BcBB93LqGHCI14abQMgJDw= 2023-09-22T22:05:38.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oxli9BcBB93LqGHCI14abQMgJDw=', 'Content-Length': '0'} 2023-09-22T22:05:38.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000359e1ff76a119e56-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.825 DEBUG:boto:path=/ffzdqu-63/obj23 2023-09-22T22:05:38.825 DEBUG:boto:auth_path=/ffzdqu-63/obj23 2023-09-22T22:05:38.825 DEBUG:boto:path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.825 DEBUG:boto:auth_path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.825 DEBUG:boto:Method: GET 2023-09-22T22:05:38.825 DEBUG:boto:Path: /ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.825 DEBUG:boto:Data: 2023-09-22T22:05:38.825 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.825 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.825 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.826 DEBUG:boto:Params: {} 2023-09-22T22:05:38.826 DEBUG:boto:Token: None 2023-09-22T22:05:38.826 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.826 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oxli9BcBB93LqGHCI14abQMgJDw= 2023-09-22T22:05:38.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oxli9BcBB93LqGHCI14abQMgJDw=', 'Content-Length': '0'} 2023-09-22T22:05:38.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb21d030c552c2d6-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.829 DEBUG:boto:path=/ffzdqu-63/obj23 2023-09-22T22:05:38.829 DEBUG:boto:auth_path=/ffzdqu-63/obj23 2023-09-22T22:05:38.829 DEBUG:boto:path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.830 DEBUG:boto:auth_path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.830 DEBUG:boto:Method: HEAD 2023-09-22T22:05:38.830 DEBUG:boto:Path: /ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.830 DEBUG:boto:Data: 2023-09-22T22:05:38.830 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.830 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.830 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.830 DEBUG:boto:Params: {} 2023-09-22T22:05:38.830 DEBUG:boto:Token: None 2023-09-22T22:05:38.831 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.831 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Bpitv75Q5PWZCZdaBSdJqnF3f2g= 2023-09-22T22:05:38.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Bpitv75Q5PWZCZdaBSdJqnF3f2g=', 'Content-Length': '0'} 2023-09-22T22:05:38.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5249dd1995a9a0d-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.834 DEBUG:boto:path=/ffzdqu-63/obj23 2023-09-22T22:05:38.834 DEBUG:boto:auth_path=/ffzdqu-63/obj23 2023-09-22T22:05:38.834 DEBUG:boto:path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.834 DEBUG:boto:auth_path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.835 DEBUG:boto:Method: HEAD 2023-09-22T22:05:38.835 DEBUG:boto:Path: /ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.835 DEBUG:boto:Data: 2023-09-22T22:05:38.835 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.835 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.835 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.835 DEBUG:boto:Params: {} 2023-09-22T22:05:38.835 DEBUG:boto:Token: None 2023-09-22T22:05:38.835 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.836 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Bpitv75Q5PWZCZdaBSdJqnF3f2g= 2023-09-22T22:05:38.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Bpitv75Q5PWZCZdaBSdJqnF3f2g=', 'Content-Length': '0'} 2023-09-22T22:05:38.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077403d8f5813fc8b-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-22T22:05:38.839 DEBUG:boto:path=/ffzdqu-63/obj23 2023-09-22T22:05:38.839 DEBUG:boto:auth_path=/ffzdqu-63/obj23 2023-09-22T22:05:38.839 DEBUG:boto:path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.839 DEBUG:boto:auth_path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.839 DEBUG:boto:Method: GET 2023-09-22T22:05:38.839 DEBUG:boto:Path: /ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.839 DEBUG:boto:Data: 2023-09-22T22:05:38.839 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.839 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.840 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.840 DEBUG:boto:Params: {} 2023-09-22T22:05:38.840 DEBUG:boto:Token: None 2023-09-22T22:05:38.840 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.840 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oxli9BcBB93LqGHCI14abQMgJDw= 2023-09-22T22:05:38.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oxli9BcBB93LqGHCI14abQMgJDw=', 'Content-Length': '0'} 2023-09-22T22:05:38.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2864726713c5b00-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.886 DEBUG:boto:path=/ffzdqu-63/obj23 2023-09-22T22:05:38.887 DEBUG:boto:auth_path=/ffzdqu-63/obj23 2023-09-22T22:05:38.887 DEBUG:boto:path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.887 DEBUG:boto:auth_path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.887 DEBUG:boto:Method: GET 2023-09-22T22:05:38.887 DEBUG:boto:Path: /ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.887 DEBUG:boto:Data: 2023-09-22T22:05:38.887 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.887 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.888 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.888 DEBUG:boto:Params: {} 2023-09-22T22:05:38.888 DEBUG:boto:Token: None 2023-09-22T22:05:38.888 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj23?versionId=null 2023-09-22T22:05:38.888 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oxli9BcBB93LqGHCI14abQMgJDw= 2023-09-22T22:05:38.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oxli9BcBB93LqGHCI14abQMgJDw=', 'Content-Length': '0'} 2023-09-22T22:05:38.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee26cfcbb390f9bd-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.892 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:38.892 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:38.892 DEBUG:boto:path=/ffzdqu-63/?prefix=obj23 2023-09-22T22:05:38.892 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj23 2023-09-22T22:05:38.892 DEBUG:boto:Method: GET 2023-09-22T22:05:38.892 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj23 2023-09-22T22:05:38.892 DEBUG:boto:Data: 2023-09-22T22:05:38.892 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.892 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.893 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.893 DEBUG:boto:Params: {} 2023-09-22T22:05:38.893 DEBUG:boto:Token: None 2023-09-22T22:05:38.893 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/ 2023-09-22T22:05:38.893 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:05:38.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:05:38.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079a090f9529b599f-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.900 DEBUG:boto:b'ffzdqu-63obj231000falseobj232023-09-22T22:05:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:38.900 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:38.900 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:38.901 DEBUG:boto:path=/ffzdqu-63/?prefix=obj23 2023-09-22T22:05:38.901 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj23 2023-09-22T22:05:38.901 DEBUG:boto:Method: GET 2023-09-22T22:05:38.901 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj23 2023-09-22T22:05:38.901 DEBUG:boto:Data: 2023-09-22T22:05:38.901 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.902 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.902 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.902 DEBUG:boto:Params: {} 2023-09-22T22:05:38.902 DEBUG:boto:Token: None 2023-09-22T22:05:38.902 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/ 2023-09-22T22:05:38.903 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:05:38.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:05:38.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000575672fafbd50dbc-00650e0fb2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.958 DEBUG:boto:b'ffzdqu-63obj231000falseobj232023-09-22T22:05:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:38.959 DEBUG:boto:path=/ffzdqu-63/obj23 2023-09-22T22:05:38.959 DEBUG:boto:auth_path=/ffzdqu-63/obj23 2023-09-22T22:05:38.959 DEBUG:boto:Method: HEAD 2023-09-22T22:05:38.959 DEBUG:boto:Path: /ffzdqu-63/obj23 2023-09-22T22:05:38.959 DEBUG:boto:Data: 2023-09-22T22:05:38.959 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.959 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.960 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.960 DEBUG:boto:Params: {} 2023-09-22T22:05:38.960 DEBUG:boto:Token: None 2023-09-22T22:05:38.960 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj23 2023-09-22T22:05:38.960 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Iq2aVAou8qHmjEvrP9XuDx3pDuw= 2023-09-22T22:05:38.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Iq2aVAou8qHmjEvrP9XuDx3pDuw=', 'Content-Length': '0'} 2023-09-22T22:05:38.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076b3176df4c3d1cb-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.963 DEBUG:boto:path=/ffzdqu-63/obj23 2023-09-22T22:05:38.963 DEBUG:boto:auth_path=/ffzdqu-63/obj23 2023-09-22T22:05:38.963 DEBUG:boto:Method: HEAD 2023-09-22T22:05:38.964 DEBUG:boto:Path: /ffzdqu-63/obj23 2023-09-22T22:05:38.964 DEBUG:boto:Data: 2023-09-22T22:05:38.964 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.964 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.964 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.964 DEBUG:boto:Params: {} 2023-09-22T22:05:38.964 DEBUG:boto:Token: None 2023-09-22T22:05:38.964 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj23 2023-09-22T22:05:38.964 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Iq2aVAou8qHmjEvrP9XuDx3pDuw= 2023-09-22T22:05:38.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Iq2aVAou8qHmjEvrP9XuDx3pDuw=', 'Content-Length': '0'} 2023-09-22T22:05:38.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b67265d6c7313d7b-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-22T22:05:38.967 DEBUG:boto:path=/ffzdqu-63/obj23 2023-09-22T22:05:38.967 DEBUG:boto:auth_path=/ffzdqu-63/obj23 2023-09-22T22:05:38.968 DEBUG:boto:Method: GET 2023-09-22T22:05:38.968 DEBUG:boto:Path: /ffzdqu-63/obj23 2023-09-22T22:05:38.968 DEBUG:boto:Data: 2023-09-22T22:05:38.968 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.968 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.968 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.968 DEBUG:boto:Params: {} 2023-09-22T22:05:38.968 DEBUG:boto:Token: None 2023-09-22T22:05:38.968 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj23 2023-09-22T22:05:38.969 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aVkXjSK7EvlH4eOfLgIiMk8Qo6k= 2023-09-22T22:05:38.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aVkXjSK7EvlH4eOfLgIiMk8Qo6k=', 'Content-Length': '0'} 2023-09-22T22:05:38.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b320d99b16dbfcbb-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.971 DEBUG:boto:path=/ffzdqu-63/obj23 2023-09-22T22:05:38.971 DEBUG:boto:auth_path=/ffzdqu-63/obj23 2023-09-22T22:05:38.972 DEBUG:boto:Method: GET 2023-09-22T22:05:38.972 DEBUG:boto:Path: /ffzdqu-63/obj23 2023-09-22T22:05:38.972 DEBUG:boto:Data: 2023-09-22T22:05:38.972 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.972 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:38.972 DEBUG:boto:Port: 8000 2023-09-22T22:05:38.972 DEBUG:boto:Params: {} 2023-09-22T22:05:38.972 DEBUG:boto:Token: None 2023-09-22T22:05:38.972 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/obj23 2023-09-22T22:05:38.972 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aVkXjSK7EvlH4eOfLgIiMk8Qo6k= 2023-09-22T22:05:38.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aVkXjSK7EvlH4eOfLgIiMk8Qo6k=', 'Content-Length': '0'} 2023-09-22T22:05:38.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4bb4c4af63124d7-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:38.976 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:38.976 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:38.976 DEBUG:boto:path=/ffzdqu-63/?prefix=obj23 2023-09-22T22:05:38.976 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj23 2023-09-22T22:05:38.976 DEBUG:boto:Method: GET 2023-09-22T22:05:38.976 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj23 2023-09-22T22:05:38.976 DEBUG:boto:Data: 2023-09-22T22:05:38.977 DEBUG:boto:Headers: {} 2023-09-22T22:05:38.977 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:38.977 DEBUG:boto:Port: 8001 2023-09-22T22:05:38.977 DEBUG:boto:Params: {} 2023-09-22T22:05:38.977 DEBUG:boto:Token: None 2023-09-22T22:05:38.977 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:38 GMT /ffzdqu-63/ 2023-09-22T22:05:38.977 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:05:38.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:05:38.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000592bbdb4a8b9d762-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.022 DEBUG:boto:b'ffzdqu-63obj231000falseobj232023-09-22T22:05:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:39.023 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:39.023 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:39.023 DEBUG:boto:path=/ffzdqu-63/?prefix=obj23 2023-09-22T22:05:39.023 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj23 2023-09-22T22:05:39.023 DEBUG:boto:Method: GET 2023-09-22T22:05:39.023 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj23 2023-09-22T22:05:39.023 DEBUG:boto:Data: 2023-09-22T22:05:39.023 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.023 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.024 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.024 DEBUG:boto:Params: {} 2023-09-22T22:05:39.024 DEBUG:boto:Token: None 2023-09-22T22:05:39.024 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/ 2023-09-22T22:05:39.024 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:05:39.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:05:39.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042b120036a9614de-00650e0fb3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.035 DEBUG:boto:b'ffzdqu-63obj231000falseobj232023-09-22T22:05:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:39.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-22T22:05:39.035 DEBUG:boto:path=/ffzdqu-63/obj29 2023-09-22T22:05:39.035 DEBUG:boto:auth_path=/ffzdqu-63/obj29 2023-09-22T22:05:39.035 DEBUG:boto:path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.035 DEBUG:boto:auth_path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.036 DEBUG:boto:Method: GET 2023-09-22T22:05:39.036 DEBUG:boto:Path: /ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.036 DEBUG:boto:Data: 2023-09-22T22:05:39.036 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.036 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.036 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.036 DEBUG:boto:Params: {} 2023-09-22T22:05:39.036 DEBUG:boto:Token: None 2023-09-22T22:05:39.036 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.036 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AwMeS/DjBkFg9H83verViX3M0XA= 2023-09-22T22:05:39.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AwMeS/DjBkFg9H83verViX3M0XA=', 'Content-Length': '0'} 2023-09-22T22:05:39.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff3f40fc268dd671-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.039 DEBUG:boto:path=/ffzdqu-63/obj29 2023-09-22T22:05:39.039 DEBUG:boto:auth_path=/ffzdqu-63/obj29 2023-09-22T22:05:39.040 DEBUG:boto:path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.040 DEBUG:boto:auth_path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.040 DEBUG:boto:Method: GET 2023-09-22T22:05:39.040 DEBUG:boto:Path: /ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.040 DEBUG:boto:Data: 2023-09-22T22:05:39.040 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.040 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.040 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.040 DEBUG:boto:Params: {} 2023-09-22T22:05:39.040 DEBUG:boto:Token: None 2023-09-22T22:05:39.041 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.041 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AwMeS/DjBkFg9H83verViX3M0XA= 2023-09-22T22:05:39.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AwMeS/DjBkFg9H83verViX3M0XA=', 'Content-Length': '0'} 2023-09-22T22:05:39.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b24d6ac3281b773-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.090 DEBUG:boto:path=/ffzdqu-63/obj29 2023-09-22T22:05:39.090 DEBUG:boto:auth_path=/ffzdqu-63/obj29 2023-09-22T22:05:39.091 DEBUG:boto:path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.091 DEBUG:boto:auth_path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.091 DEBUG:boto:Method: HEAD 2023-09-22T22:05:39.091 DEBUG:boto:Path: /ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.091 DEBUG:boto:Data: 2023-09-22T22:05:39.091 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.091 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.091 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.091 DEBUG:boto:Params: {} 2023-09-22T22:05:39.092 DEBUG:boto:Token: None 2023-09-22T22:05:39.092 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.092 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3o8gKiOp6CwJw3znoVfqAQsZ+fg= 2023-09-22T22:05:39.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3o8gKiOp6CwJw3znoVfqAQsZ+fg=', 'Content-Length': '0'} 2023-09-22T22:05:39.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068b9912fbaf0f9e4-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.095 DEBUG:boto:path=/ffzdqu-63/obj29 2023-09-22T22:05:39.095 DEBUG:boto:auth_path=/ffzdqu-63/obj29 2023-09-22T22:05:39.095 DEBUG:boto:path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.096 DEBUG:boto:auth_path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.096 DEBUG:boto:Method: HEAD 2023-09-22T22:05:39.096 DEBUG:boto:Path: /ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.096 DEBUG:boto:Data: 2023-09-22T22:05:39.096 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.096 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.096 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.096 DEBUG:boto:Params: {} 2023-09-22T22:05:39.096 DEBUG:boto:Token: None 2023-09-22T22:05:39.097 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.097 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3o8gKiOp6CwJw3znoVfqAQsZ+fg= 2023-09-22T22:05:39.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3o8gKiOp6CwJw3znoVfqAQsZ+fg=', 'Content-Length': '0'} 2023-09-22T22:05:39.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a33ee75060918ef-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-22T22:05:39.100 DEBUG:boto:path=/ffzdqu-63/obj29 2023-09-22T22:05:39.100 DEBUG:boto:auth_path=/ffzdqu-63/obj29 2023-09-22T22:05:39.100 DEBUG:boto:path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.100 DEBUG:boto:auth_path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.100 DEBUG:boto:Method: GET 2023-09-22T22:05:39.100 DEBUG:boto:Path: /ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.100 DEBUG:boto:Data: 2023-09-22T22:05:39.101 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.101 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.101 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.101 DEBUG:boto:Params: {} 2023-09-22T22:05:39.101 DEBUG:boto:Token: None 2023-09-22T22:05:39.101 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.101 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AwMeS/DjBkFg9H83verViX3M0XA= 2023-09-22T22:05:39.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AwMeS/DjBkFg9H83verViX3M0XA=', 'Content-Length': '0'} 2023-09-22T22:05:39.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003398f0422d27f14c-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.104 DEBUG:boto:path=/ffzdqu-63/obj29 2023-09-22T22:05:39.104 DEBUG:boto:auth_path=/ffzdqu-63/obj29 2023-09-22T22:05:39.104 DEBUG:boto:path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.105 DEBUG:boto:auth_path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.105 DEBUG:boto:Method: GET 2023-09-22T22:05:39.105 DEBUG:boto:Path: /ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.105 DEBUG:boto:Data: 2023-09-22T22:05:39.105 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.105 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.105 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.105 DEBUG:boto:Params: {} 2023-09-22T22:05:39.105 DEBUG:boto:Token: None 2023-09-22T22:05:39.106 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj29?versionId=null 2023-09-22T22:05:39.106 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AwMeS/DjBkFg9H83verViX3M0XA= 2023-09-22T22:05:39.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AwMeS/DjBkFg9H83verViX3M0XA=', 'Content-Length': '0'} 2023-09-22T22:05:39.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af4463cca8d93f94-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.109 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:39.109 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:39.109 DEBUG:boto:path=/ffzdqu-63/?prefix=obj29 2023-09-22T22:05:39.109 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj29 2023-09-22T22:05:39.110 DEBUG:boto:Method: GET 2023-09-22T22:05:39.110 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj29 2023-09-22T22:05:39.110 DEBUG:boto:Data: 2023-09-22T22:05:39.110 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.110 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.110 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.110 DEBUG:boto:Params: {} 2023-09-22T22:05:39.110 DEBUG:boto:Token: None 2023-09-22T22:05:39.110 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/ 2023-09-22T22:05:39.111 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:05:39.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:05:39.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088b74ae85052b6d6-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.158 DEBUG:boto:b'ffzdqu-63obj291000falseobj292023-09-22T22:05:15.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:39.159 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:39.159 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:39.159 DEBUG:boto:path=/ffzdqu-63/?prefix=obj29 2023-09-22T22:05:39.159 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj29 2023-09-22T22:05:39.159 DEBUG:boto:Method: GET 2023-09-22T22:05:39.159 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj29 2023-09-22T22:05:39.159 DEBUG:boto:Data: 2023-09-22T22:05:39.160 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.160 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.160 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.160 DEBUG:boto:Params: {} 2023-09-22T22:05:39.160 DEBUG:boto:Token: None 2023-09-22T22:05:39.160 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/ 2023-09-22T22:05:39.160 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:05:39.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:05:39.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e7d73912268547d-00650e0fb3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.170 DEBUG:boto:b'ffzdqu-63obj291000falseobj292023-09-22T22:05:15.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:39.170 DEBUG:boto:path=/ffzdqu-63/obj29 2023-09-22T22:05:39.170 DEBUG:boto:auth_path=/ffzdqu-63/obj29 2023-09-22T22:05:39.170 DEBUG:boto:Method: HEAD 2023-09-22T22:05:39.171 DEBUG:boto:Path: /ffzdqu-63/obj29 2023-09-22T22:05:39.171 DEBUG:boto:Data: 2023-09-22T22:05:39.171 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.171 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.171 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.171 DEBUG:boto:Params: {} 2023-09-22T22:05:39.171 DEBUG:boto:Token: None 2023-09-22T22:05:39.171 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj29 2023-09-22T22:05:39.171 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JWub2c7eUqWLLfjqOnwpDNVJuSc= 2023-09-22T22:05:39.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JWub2c7eUqWLLfjqOnwpDNVJuSc=', 'Content-Length': '0'} 2023-09-22T22:05:39.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f40a6caf60a505b0-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.174 DEBUG:boto:path=/ffzdqu-63/obj29 2023-09-22T22:05:39.175 DEBUG:boto:auth_path=/ffzdqu-63/obj29 2023-09-22T22:05:39.175 DEBUG:boto:Method: HEAD 2023-09-22T22:05:39.175 DEBUG:boto:Path: /ffzdqu-63/obj29 2023-09-22T22:05:39.175 DEBUG:boto:Data: 2023-09-22T22:05:39.175 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.175 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.175 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.175 DEBUG:boto:Params: {} 2023-09-22T22:05:39.175 DEBUG:boto:Token: None 2023-09-22T22:05:39.176 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj29 2023-09-22T22:05:39.176 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:JWub2c7eUqWLLfjqOnwpDNVJuSc= 2023-09-22T22:05:39.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JWub2c7eUqWLLfjqOnwpDNVJuSc=', 'Content-Length': '0'} 2023-09-22T22:05:39.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000516a5ee4774fb5a8-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-22T22:05:39.179 DEBUG:boto:path=/ffzdqu-63/obj29 2023-09-22T22:05:39.179 DEBUG:boto:auth_path=/ffzdqu-63/obj29 2023-09-22T22:05:39.179 DEBUG:boto:Method: GET 2023-09-22T22:05:39.179 DEBUG:boto:Path: /ffzdqu-63/obj29 2023-09-22T22:05:39.179 DEBUG:boto:Data: 2023-09-22T22:05:39.179 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.179 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.179 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.180 DEBUG:boto:Params: {} 2023-09-22T22:05:39.180 DEBUG:boto:Token: None 2023-09-22T22:05:39.180 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj29 2023-09-22T22:05:39.180 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DWJz/hx/zSxNkzLJ0MKYyBWTgkA= 2023-09-22T22:05:39.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DWJz/hx/zSxNkzLJ0MKYyBWTgkA=', 'Content-Length': '0'} 2023-09-22T22:05:39.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008668e39986cbac48-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.183 DEBUG:boto:path=/ffzdqu-63/obj29 2023-09-22T22:05:39.183 DEBUG:boto:auth_path=/ffzdqu-63/obj29 2023-09-22T22:05:39.183 DEBUG:boto:Method: GET 2023-09-22T22:05:39.183 DEBUG:boto:Path: /ffzdqu-63/obj29 2023-09-22T22:05:39.183 DEBUG:boto:Data: 2023-09-22T22:05:39.183 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.183 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.184 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.184 DEBUG:boto:Params: {} 2023-09-22T22:05:39.184 DEBUG:boto:Token: None 2023-09-22T22:05:39.184 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj29 2023-09-22T22:05:39.184 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DWJz/hx/zSxNkzLJ0MKYyBWTgkA= 2023-09-22T22:05:39.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DWJz/hx/zSxNkzLJ0MKYyBWTgkA=', 'Content-Length': '0'} 2023-09-22T22:05:39.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2754a5ddaf90189-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.234 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:39.234 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:39.234 DEBUG:boto:path=/ffzdqu-63/?prefix=obj29 2023-09-22T22:05:39.235 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj29 2023-09-22T22:05:39.235 DEBUG:boto:Method: GET 2023-09-22T22:05:39.235 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj29 2023-09-22T22:05:39.235 DEBUG:boto:Data: 2023-09-22T22:05:39.235 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.235 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.235 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.235 DEBUG:boto:Params: {} 2023-09-22T22:05:39.235 DEBUG:boto:Token: None 2023-09-22T22:05:39.235 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/ 2023-09-22T22:05:39.236 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:05:39.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:05:39.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd520a49b4fda6c6-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.241 DEBUG:boto:b'ffzdqu-63obj291000falseobj292023-09-22T22:05:15.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:39.241 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:39.241 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:39.241 DEBUG:boto:path=/ffzdqu-63/?prefix=obj29 2023-09-22T22:05:39.241 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj29 2023-09-22T22:05:39.242 DEBUG:boto:Method: GET 2023-09-22T22:05:39.242 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj29 2023-09-22T22:05:39.242 DEBUG:boto:Data: 2023-09-22T22:05:39.242 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.242 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.242 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.242 DEBUG:boto:Params: {} 2023-09-22T22:05:39.242 DEBUG:boto:Token: None 2023-09-22T22:05:39.242 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/ 2023-09-22T22:05:39.243 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:05:39.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:05:39.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008064abf1a118978c-00650e0fb3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.253 DEBUG:boto:b'ffzdqu-63obj291000falseobj292023-09-22T22:05:15.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:39.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-22T22:05:39.253 DEBUG:boto:path=/ffzdqu-63/obj31 2023-09-22T22:05:39.253 DEBUG:boto:auth_path=/ffzdqu-63/obj31 2023-09-22T22:05:39.253 DEBUG:boto:path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.253 DEBUG:boto:auth_path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.253 DEBUG:boto:Method: GET 2023-09-22T22:05:39.253 DEBUG:boto:Path: /ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.254 DEBUG:boto:Data: 2023-09-22T22:05:39.254 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.254 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.254 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.254 DEBUG:boto:Params: {} 2023-09-22T22:05:39.254 DEBUG:boto:Token: None 2023-09-22T22:05:39.254 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.254 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HDwtIZ5FCc+AzD4PeMoL/WzVD1c= 2023-09-22T22:05:39.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HDwtIZ5FCc+AzD4PeMoL/WzVD1c=', 'Content-Length': '0'} 2023-09-22T22:05:39.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f96eecc85d3edd72-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.298 DEBUG:boto:path=/ffzdqu-63/obj31 2023-09-22T22:05:39.299 DEBUG:boto:auth_path=/ffzdqu-63/obj31 2023-09-22T22:05:39.299 DEBUG:boto:path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.299 DEBUG:boto:auth_path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.299 DEBUG:boto:Method: GET 2023-09-22T22:05:39.299 DEBUG:boto:Path: /ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.299 DEBUG:boto:Data: 2023-09-22T22:05:39.299 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.299 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.299 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.299 DEBUG:boto:Params: {} 2023-09-22T22:05:39.300 DEBUG:boto:Token: None 2023-09-22T22:05:39.300 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.300 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HDwtIZ5FCc+AzD4PeMoL/WzVD1c= 2023-09-22T22:05:39.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HDwtIZ5FCc+AzD4PeMoL/WzVD1c=', 'Content-Length': '0'} 2023-09-22T22:05:39.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8a42d40429f8eab-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.303 DEBUG:boto:path=/ffzdqu-63/obj31 2023-09-22T22:05:39.304 DEBUG:boto:auth_path=/ffzdqu-63/obj31 2023-09-22T22:05:39.304 DEBUG:boto:path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.304 DEBUG:boto:auth_path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.304 DEBUG:boto:Method: HEAD 2023-09-22T22:05:39.304 DEBUG:boto:Path: /ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.304 DEBUG:boto:Data: 2023-09-22T22:05:39.304 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.304 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.304 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.304 DEBUG:boto:Params: {} 2023-09-22T22:05:39.304 DEBUG:boto:Token: None 2023-09-22T22:05:39.305 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.305 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DkC61oRYFxQorw6xprFQppMYyMU= 2023-09-22T22:05:39.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DkC61oRYFxQorw6xprFQppMYyMU=', 'Content-Length': '0'} 2023-09-22T22:05:39.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002817a9ecb2cbd048-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.308 DEBUG:boto:path=/ffzdqu-63/obj31 2023-09-22T22:05:39.308 DEBUG:boto:auth_path=/ffzdqu-63/obj31 2023-09-22T22:05:39.308 DEBUG:boto:path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.308 DEBUG:boto:auth_path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.309 DEBUG:boto:Method: HEAD 2023-09-22T22:05:39.309 DEBUG:boto:Path: /ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.309 DEBUG:boto:Data: 2023-09-22T22:05:39.309 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.309 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.309 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.309 DEBUG:boto:Params: {} 2023-09-22T22:05:39.309 DEBUG:boto:Token: None 2023-09-22T22:05:39.309 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.309 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DkC61oRYFxQorw6xprFQppMYyMU= 2023-09-22T22:05:39.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DkC61oRYFxQorw6xprFQppMYyMU=', 'Content-Length': '0'} 2023-09-22T22:05:39.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bebf9f7d320423a0-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-22T22:05:39.313 DEBUG:boto:path=/ffzdqu-63/obj31 2023-09-22T22:05:39.313 DEBUG:boto:auth_path=/ffzdqu-63/obj31 2023-09-22T22:05:39.313 DEBUG:boto:path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.313 DEBUG:boto:auth_path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.313 DEBUG:boto:Method: GET 2023-09-22T22:05:39.313 DEBUG:boto:Path: /ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.313 DEBUG:boto:Data: 2023-09-22T22:05:39.313 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.313 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.314 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.314 DEBUG:boto:Params: {} 2023-09-22T22:05:39.314 DEBUG:boto:Token: None 2023-09-22T22:05:39.314 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.314 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HDwtIZ5FCc+AzD4PeMoL/WzVD1c= 2023-09-22T22:05:39.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HDwtIZ5FCc+AzD4PeMoL/WzVD1c=', 'Content-Length': '0'} 2023-09-22T22:05:39.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097138f34adf1ead4-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.317 DEBUG:boto:path=/ffzdqu-63/obj31 2023-09-22T22:05:39.317 DEBUG:boto:auth_path=/ffzdqu-63/obj31 2023-09-22T22:05:39.317 DEBUG:boto:path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.317 DEBUG:boto:auth_path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.318 DEBUG:boto:Method: GET 2023-09-22T22:05:39.318 DEBUG:boto:Path: /ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.318 DEBUG:boto:Data: 2023-09-22T22:05:39.318 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.318 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.318 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.318 DEBUG:boto:Params: {} 2023-09-22T22:05:39.318 DEBUG:boto:Token: None 2023-09-22T22:05:39.318 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj31?versionId=null 2023-09-22T22:05:39.318 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HDwtIZ5FCc+AzD4PeMoL/WzVD1c= 2023-09-22T22:05:39.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HDwtIZ5FCc+AzD4PeMoL/WzVD1c=', 'Content-Length': '0'} 2023-09-22T22:05:39.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d827460594327c4b-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.362 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:39.362 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:39.363 DEBUG:boto:path=/ffzdqu-63/?prefix=obj31 2023-09-22T22:05:39.363 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj31 2023-09-22T22:05:39.363 DEBUG:boto:Method: GET 2023-09-22T22:05:39.363 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj31 2023-09-22T22:05:39.363 DEBUG:boto:Data: 2023-09-22T22:05:39.363 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.363 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.363 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.363 DEBUG:boto:Params: {} 2023-09-22T22:05:39.363 DEBUG:boto:Token: None 2023-09-22T22:05:39.364 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/ 2023-09-22T22:05:39.364 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:05:39.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:05:39.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf0494438f3478c8-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.369 DEBUG:boto:b'ffzdqu-63obj311000falseobj312023-09-22T22:05:15.503Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:39.369 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:39.369 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:39.369 DEBUG:boto:path=/ffzdqu-63/?prefix=obj31 2023-09-22T22:05:39.370 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj31 2023-09-22T22:05:39.370 DEBUG:boto:Method: GET 2023-09-22T22:05:39.370 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj31 2023-09-22T22:05:39.370 DEBUG:boto:Data: 2023-09-22T22:05:39.370 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.370 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.370 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.370 DEBUG:boto:Params: {} 2023-09-22T22:05:39.370 DEBUG:boto:Token: None 2023-09-22T22:05:39.371 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/ 2023-09-22T22:05:39.371 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:05:39.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:05:39.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d912145cffbd6ce-00650e0fb3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.380 DEBUG:boto:b'ffzdqu-63obj311000falseobj312023-09-22T22:05:15.503Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:39.380 DEBUG:boto:path=/ffzdqu-63/obj31 2023-09-22T22:05:39.380 DEBUG:boto:auth_path=/ffzdqu-63/obj31 2023-09-22T22:05:39.381 DEBUG:boto:Method: HEAD 2023-09-22T22:05:39.381 DEBUG:boto:Path: /ffzdqu-63/obj31 2023-09-22T22:05:39.381 DEBUG:boto:Data: 2023-09-22T22:05:39.381 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.381 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.381 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.381 DEBUG:boto:Params: {} 2023-09-22T22:05:39.381 DEBUG:boto:Token: None 2023-09-22T22:05:39.381 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj31 2023-09-22T22:05:39.381 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LIpvjZrJylbKg/l1We+79JQ7i2g= 2023-09-22T22:05:39.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LIpvjZrJylbKg/l1We+79JQ7i2g=', 'Content-Length': '0'} 2023-09-22T22:05:39.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e17a3e2abb29713-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.384 DEBUG:boto:path=/ffzdqu-63/obj31 2023-09-22T22:05:39.384 DEBUG:boto:auth_path=/ffzdqu-63/obj31 2023-09-22T22:05:39.384 DEBUG:boto:Method: HEAD 2023-09-22T22:05:39.385 DEBUG:boto:Path: /ffzdqu-63/obj31 2023-09-22T22:05:39.385 DEBUG:boto:Data: 2023-09-22T22:05:39.385 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.385 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.385 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.385 DEBUG:boto:Params: {} 2023-09-22T22:05:39.385 DEBUG:boto:Token: None 2023-09-22T22:05:39.385 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj31 2023-09-22T22:05:39.385 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LIpvjZrJylbKg/l1We+79JQ7i2g= 2023-09-22T22:05:39.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LIpvjZrJylbKg/l1We+79JQ7i2g=', 'Content-Length': '0'} 2023-09-22T22:05:39.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032f025391c77a505-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-22T22:05:39.388 DEBUG:boto:path=/ffzdqu-63/obj31 2023-09-22T22:05:39.389 DEBUG:boto:auth_path=/ffzdqu-63/obj31 2023-09-22T22:05:39.389 DEBUG:boto:Method: GET 2023-09-22T22:05:39.389 DEBUG:boto:Path: /ffzdqu-63/obj31 2023-09-22T22:05:39.389 DEBUG:boto:Data: 2023-09-22T22:05:39.389 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.389 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.389 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.389 DEBUG:boto:Params: {} 2023-09-22T22:05:39.389 DEBUG:boto:Token: None 2023-09-22T22:05:39.389 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj31 2023-09-22T22:05:39.390 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UtVR3iphNz55cxcYasbdnV4SvQg= 2023-09-22T22:05:39.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UtVR3iphNz55cxcYasbdnV4SvQg=', 'Content-Length': '0'} 2023-09-22T22:05:39.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d80fc0aa04281d40-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.434 DEBUG:boto:path=/ffzdqu-63/obj31 2023-09-22T22:05:39.434 DEBUG:boto:auth_path=/ffzdqu-63/obj31 2023-09-22T22:05:39.435 DEBUG:boto:Method: GET 2023-09-22T22:05:39.435 DEBUG:boto:Path: /ffzdqu-63/obj31 2023-09-22T22:05:39.435 DEBUG:boto:Data: 2023-09-22T22:05:39.435 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.435 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.435 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.435 DEBUG:boto:Params: {} 2023-09-22T22:05:39.435 DEBUG:boto:Token: None 2023-09-22T22:05:39.435 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj31 2023-09-22T22:05:39.436 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UtVR3iphNz55cxcYasbdnV4SvQg= 2023-09-22T22:05:39.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UtVR3iphNz55cxcYasbdnV4SvQg=', 'Content-Length': '0'} 2023-09-22T22:05:39.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055f982dd88188b4f-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.439 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:39.439 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:39.439 DEBUG:boto:path=/ffzdqu-63/?prefix=obj31 2023-09-22T22:05:39.440 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj31 2023-09-22T22:05:39.440 DEBUG:boto:Method: GET 2023-09-22T22:05:39.440 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj31 2023-09-22T22:05:39.440 DEBUG:boto:Data: 2023-09-22T22:05:39.440 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.440 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.440 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.441 DEBUG:boto:Params: {} 2023-09-22T22:05:39.441 DEBUG:boto:Token: None 2023-09-22T22:05:39.441 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/ 2023-09-22T22:05:39.441 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:05:39.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:05:39.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a7c062ebf0443d0-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.446 DEBUG:boto:b'ffzdqu-63obj311000falseobj312023-09-22T22:05:15.503Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:39.446 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:39.446 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:39.447 DEBUG:boto:path=/ffzdqu-63/?prefix=obj31 2023-09-22T22:05:39.447 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj31 2023-09-22T22:05:39.447 DEBUG:boto:Method: GET 2023-09-22T22:05:39.447 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj31 2023-09-22T22:05:39.447 DEBUG:boto:Data: 2023-09-22T22:05:39.447 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.447 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.447 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.447 DEBUG:boto:Params: {} 2023-09-22T22:05:39.447 DEBUG:boto:Token: None 2023-09-22T22:05:39.448 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/ 2023-09-22T22:05:39.448 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:05:39.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:05:39.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d33d447e69c5f13b-00650e0fb3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.498 DEBUG:boto:b'ffzdqu-63obj311000falseobj312023-09-22T22:05:15.503Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:39.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T22:05:39.499 DEBUG:boto:path=/ffzdqu-63/obj33 2023-09-22T22:05:39.499 DEBUG:boto:auth_path=/ffzdqu-63/obj33 2023-09-22T22:05:39.500 DEBUG:boto:path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.500 DEBUG:boto:auth_path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.500 DEBUG:boto:Method: GET 2023-09-22T22:05:39.500 DEBUG:boto:Path: /ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.501 DEBUG:boto:Data: 2023-09-22T22:05:39.501 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.501 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.501 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.501 DEBUG:boto:Params: {} 2023-09-22T22:05:39.502 DEBUG:boto:Token: None 2023-09-22T22:05:39.502 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.502 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x29jAnhEnsxmC8n6NBw2mHN3zbk= 2023-09-22T22:05:39.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x29jAnhEnsxmC8n6NBw2mHN3zbk=', 'Content-Length': '0'} 2023-09-22T22:05:39.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7e326dd848ff20c-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.506 DEBUG:boto:path=/ffzdqu-63/obj33 2023-09-22T22:05:39.506 DEBUG:boto:auth_path=/ffzdqu-63/obj33 2023-09-22T22:05:39.506 DEBUG:boto:path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.506 DEBUG:boto:auth_path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.506 DEBUG:boto:Method: GET 2023-09-22T22:05:39.507 DEBUG:boto:Path: /ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.507 DEBUG:boto:Data: 2023-09-22T22:05:39.507 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.507 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.507 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.507 DEBUG:boto:Params: {} 2023-09-22T22:05:39.507 DEBUG:boto:Token: None 2023-09-22T22:05:39.507 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.507 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x29jAnhEnsxmC8n6NBw2mHN3zbk= 2023-09-22T22:05:39.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x29jAnhEnsxmC8n6NBw2mHN3zbk=', 'Content-Length': '0'} 2023-09-22T22:05:39.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e82fe82121a5bcd6-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.511 DEBUG:boto:path=/ffzdqu-63/obj33 2023-09-22T22:05:39.511 DEBUG:boto:auth_path=/ffzdqu-63/obj33 2023-09-22T22:05:39.511 DEBUG:boto:path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.511 DEBUG:boto:auth_path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.511 DEBUG:boto:Method: HEAD 2023-09-22T22:05:39.511 DEBUG:boto:Path: /ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.511 DEBUG:boto:Data: 2023-09-22T22:05:39.512 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.512 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.512 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.512 DEBUG:boto:Params: {} 2023-09-22T22:05:39.512 DEBUG:boto:Token: None 2023-09-22T22:05:39.512 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.512 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6I8SE0rvDP48Lm246yzuZ8INmgA= 2023-09-22T22:05:39.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6I8SE0rvDP48Lm246yzuZ8INmgA=', 'Content-Length': '0'} 2023-09-22T22:05:39.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000833fca8f951a4e26-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.515 DEBUG:boto:path=/ffzdqu-63/obj33 2023-09-22T22:05:39.515 DEBUG:boto:auth_path=/ffzdqu-63/obj33 2023-09-22T22:05:39.515 DEBUG:boto:path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.516 DEBUG:boto:auth_path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.516 DEBUG:boto:Method: HEAD 2023-09-22T22:05:39.516 DEBUG:boto:Path: /ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.516 DEBUG:boto:Data: 2023-09-22T22:05:39.516 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.516 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.516 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.516 DEBUG:boto:Params: {} 2023-09-22T22:05:39.516 DEBUG:boto:Token: None 2023-09-22T22:05:39.517 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.517 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6I8SE0rvDP48Lm246yzuZ8INmgA= 2023-09-22T22:05:39.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6I8SE0rvDP48Lm246yzuZ8INmgA=', 'Content-Length': '0'} 2023-09-22T22:05:39.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df4a68159ec8cde3-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T22:05:39.521 DEBUG:boto:path=/ffzdqu-63/obj33 2023-09-22T22:05:39.521 DEBUG:boto:auth_path=/ffzdqu-63/obj33 2023-09-22T22:05:39.521 DEBUG:boto:path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.521 DEBUG:boto:auth_path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.521 DEBUG:boto:Method: GET 2023-09-22T22:05:39.521 DEBUG:boto:Path: /ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.522 DEBUG:boto:Data: 2023-09-22T22:05:39.522 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.522 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.522 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.522 DEBUG:boto:Params: {} 2023-09-22T22:05:39.522 DEBUG:boto:Token: None 2023-09-22T22:05:39.522 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.522 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x29jAnhEnsxmC8n6NBw2mHN3zbk= 2023-09-22T22:05:39.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x29jAnhEnsxmC8n6NBw2mHN3zbk=', 'Content-Length': '0'} 2023-09-22T22:05:39.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fad5a05dd2d79e0-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.526 DEBUG:boto:path=/ffzdqu-63/obj33 2023-09-22T22:05:39.526 DEBUG:boto:auth_path=/ffzdqu-63/obj33 2023-09-22T22:05:39.526 DEBUG:boto:path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.526 DEBUG:boto:auth_path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.526 DEBUG:boto:Method: GET 2023-09-22T22:05:39.526 DEBUG:boto:Path: /ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.526 DEBUG:boto:Data: 2023-09-22T22:05:39.527 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.527 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.527 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.527 DEBUG:boto:Params: {} 2023-09-22T22:05:39.527 DEBUG:boto:Token: None 2023-09-22T22:05:39.527 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj33?versionId=null 2023-09-22T22:05:39.527 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:x29jAnhEnsxmC8n6NBw2mHN3zbk= 2023-09-22T22:05:39.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x29jAnhEnsxmC8n6NBw2mHN3zbk=', 'Content-Length': '0'} 2023-09-22T22:05:39.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaa3f517b9d3a21b-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.570 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:39.570 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:39.571 DEBUG:boto:path=/ffzdqu-63/?prefix=obj33 2023-09-22T22:05:39.571 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj33 2023-09-22T22:05:39.571 DEBUG:boto:Method: GET 2023-09-22T22:05:39.571 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj33 2023-09-22T22:05:39.571 DEBUG:boto:Data: 2023-09-22T22:05:39.571 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.571 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.571 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.571 DEBUG:boto:Params: {} 2023-09-22T22:05:39.571 DEBUG:boto:Token: None 2023-09-22T22:05:39.572 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/ 2023-09-22T22:05:39.572 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:05:39.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:05:39.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002baac6a38303c5d9-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.577 DEBUG:boto:b'ffzdqu-63obj331000falseobj332023-09-22T22:05:13.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:39.578 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:39.578 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:39.578 DEBUG:boto:path=/ffzdqu-63/?prefix=obj33 2023-09-22T22:05:39.578 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj33 2023-09-22T22:05:39.578 DEBUG:boto:Method: GET 2023-09-22T22:05:39.578 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj33 2023-09-22T22:05:39.578 DEBUG:boto:Data: 2023-09-22T22:05:39.578 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.579 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.579 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.579 DEBUG:boto:Params: {} 2023-09-22T22:05:39.579 DEBUG:boto:Token: None 2023-09-22T22:05:39.579 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/ 2023-09-22T22:05:39.579 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:05:39.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:05:39.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017b7cf8b6a56bacd-00650e0fb3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.589 DEBUG:boto:b'ffzdqu-63obj331000falseobj332023-09-22T22:05:13.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:39.589 DEBUG:boto:path=/ffzdqu-63/obj33 2023-09-22T22:05:39.589 DEBUG:boto:auth_path=/ffzdqu-63/obj33 2023-09-22T22:05:39.590 DEBUG:boto:Method: HEAD 2023-09-22T22:05:39.590 DEBUG:boto:Path: /ffzdqu-63/obj33 2023-09-22T22:05:39.590 DEBUG:boto:Data: 2023-09-22T22:05:39.590 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.590 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.590 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.590 DEBUG:boto:Params: {} 2023-09-22T22:05:39.590 DEBUG:boto:Token: None 2023-09-22T22:05:39.591 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj33 2023-09-22T22:05:39.591 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:N/IVs3J91C0KhLuGK4kYR6utcfU= 2023-09-22T22:05:39.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:N/IVs3J91C0KhLuGK4kYR6utcfU=', 'Content-Length': '0'} 2023-09-22T22:05:39.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019884ccea0b47cad-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.594 DEBUG:boto:path=/ffzdqu-63/obj33 2023-09-22T22:05:39.594 DEBUG:boto:auth_path=/ffzdqu-63/obj33 2023-09-22T22:05:39.594 DEBUG:boto:Method: HEAD 2023-09-22T22:05:39.594 DEBUG:boto:Path: /ffzdqu-63/obj33 2023-09-22T22:05:39.594 DEBUG:boto:Data: 2023-09-22T22:05:39.594 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.594 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.594 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.595 DEBUG:boto:Params: {} 2023-09-22T22:05:39.595 DEBUG:boto:Token: None 2023-09-22T22:05:39.595 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj33 2023-09-22T22:05:39.595 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:N/IVs3J91C0KhLuGK4kYR6utcfU= 2023-09-22T22:05:39.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:N/IVs3J91C0KhLuGK4kYR6utcfU=', 'Content-Length': '0'} 2023-09-22T22:05:39.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc1ada81bd96cfc7-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T22:05:39.598 DEBUG:boto:path=/ffzdqu-63/obj33 2023-09-22T22:05:39.598 DEBUG:boto:auth_path=/ffzdqu-63/obj33 2023-09-22T22:05:39.598 DEBUG:boto:Method: GET 2023-09-22T22:05:39.598 DEBUG:boto:Path: /ffzdqu-63/obj33 2023-09-22T22:05:39.599 DEBUG:boto:Data: 2023-09-22T22:05:39.599 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.599 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.599 DEBUG:boto:Params: {} 2023-09-22T22:05:39.599 DEBUG:boto:Token: None 2023-09-22T22:05:39.599 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj33 2023-09-22T22:05:39.599 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+uGjrCnLehXGrMFsWcxXMAknNC8= 2023-09-22T22:05:39.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+uGjrCnLehXGrMFsWcxXMAknNC8=', 'Content-Length': '0'} 2023-09-22T22:05:39.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f1e002538c05171-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.642 DEBUG:boto:path=/ffzdqu-63/obj33 2023-09-22T22:05:39.642 DEBUG:boto:auth_path=/ffzdqu-63/obj33 2023-09-22T22:05:39.643 DEBUG:boto:Method: GET 2023-09-22T22:05:39.643 DEBUG:boto:Path: /ffzdqu-63/obj33 2023-09-22T22:05:39.643 DEBUG:boto:Data: 2023-09-22T22:05:39.643 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.643 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.643 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.643 DEBUG:boto:Params: {} 2023-09-22T22:05:39.643 DEBUG:boto:Token: None 2023-09-22T22:05:39.643 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj33 2023-09-22T22:05:39.644 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:+uGjrCnLehXGrMFsWcxXMAknNC8= 2023-09-22T22:05:39.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+uGjrCnLehXGrMFsWcxXMAknNC8=', 'Content-Length': '0'} 2023-09-22T22:05:39.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d225002f2dcd82a0-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.647 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:39.647 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:39.647 DEBUG:boto:path=/ffzdqu-63/?prefix=obj33 2023-09-22T22:05:39.647 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj33 2023-09-22T22:05:39.647 DEBUG:boto:Method: GET 2023-09-22T22:05:39.648 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj33 2023-09-22T22:05:39.648 DEBUG:boto:Data: 2023-09-22T22:05:39.648 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.648 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.648 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.648 DEBUG:boto:Params: {} 2023-09-22T22:05:39.648 DEBUG:boto:Token: None 2023-09-22T22:05:39.648 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/ 2023-09-22T22:05:39.648 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:05:39.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:05:39.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096bfcbebb8cd1c64-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.653 DEBUG:boto:b'ffzdqu-63obj331000falseobj332023-09-22T22:05:13.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:39.653 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:39.654 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:39.654 DEBUG:boto:path=/ffzdqu-63/?prefix=obj33 2023-09-22T22:05:39.654 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj33 2023-09-22T22:05:39.654 DEBUG:boto:Method: GET 2023-09-22T22:05:39.654 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj33 2023-09-22T22:05:39.654 DEBUG:boto:Data: 2023-09-22T22:05:39.654 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.654 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.654 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.655 DEBUG:boto:Params: {} 2023-09-22T22:05:39.655 DEBUG:boto:Token: None 2023-09-22T22:05:39.655 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/ 2023-09-22T22:05:39.655 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:05:39.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:05:39.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d7117af21d17025-00650e0fb3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.706 DEBUG:boto:b'ffzdqu-63obj331000falseobj332023-09-22T22:05:13.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:39.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T22:05:39.707 DEBUG:boto:path=/ffzdqu-63/obj34 2023-09-22T22:05:39.707 DEBUG:boto:auth_path=/ffzdqu-63/obj34 2023-09-22T22:05:39.707 DEBUG:boto:path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.707 DEBUG:boto:auth_path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.707 DEBUG:boto:Method: GET 2023-09-22T22:05:39.707 DEBUG:boto:Path: /ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.707 DEBUG:boto:Data: 2023-09-22T22:05:39.708 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.708 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.708 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.708 DEBUG:boto:Params: {} 2023-09-22T22:05:39.708 DEBUG:boto:Token: None 2023-09-22T22:05:39.708 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.708 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:akqqJFTLpznzrfYm6gSxDkWWKmM= 2023-09-22T22:05:39.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:akqqJFTLpznzrfYm6gSxDkWWKmM=', 'Content-Length': '0'} 2023-09-22T22:05:39.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001170bd4d3b75b471-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.712 DEBUG:boto:path=/ffzdqu-63/obj34 2023-09-22T22:05:39.712 DEBUG:boto:auth_path=/ffzdqu-63/obj34 2023-09-22T22:05:39.712 DEBUG:boto:path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.712 DEBUG:boto:auth_path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.712 DEBUG:boto:Method: GET 2023-09-22T22:05:39.712 DEBUG:boto:Path: /ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.712 DEBUG:boto:Data: 2023-09-22T22:05:39.712 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.712 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.713 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.713 DEBUG:boto:Params: {} 2023-09-22T22:05:39.713 DEBUG:boto:Token: None 2023-09-22T22:05:39.713 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.713 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:akqqJFTLpznzrfYm6gSxDkWWKmM= 2023-09-22T22:05:39.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:akqqJFTLpznzrfYm6gSxDkWWKmM=', 'Content-Length': '0'} 2023-09-22T22:05:39.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059ff2cac97a09245-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.716 DEBUG:boto:path=/ffzdqu-63/obj34 2023-09-22T22:05:39.716 DEBUG:boto:auth_path=/ffzdqu-63/obj34 2023-09-22T22:05:39.716 DEBUG:boto:path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.716 DEBUG:boto:auth_path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.716 DEBUG:boto:Method: HEAD 2023-09-22T22:05:39.717 DEBUG:boto:Path: /ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.717 DEBUG:boto:Data: 2023-09-22T22:05:39.717 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.717 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.717 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.717 DEBUG:boto:Params: {} 2023-09-22T22:05:39.717 DEBUG:boto:Token: None 2023-09-22T22:05:39.717 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.717 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:j2cm0uxtNtIwGktDoULu3X6YM34= 2023-09-22T22:05:39.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:j2cm0uxtNtIwGktDoULu3X6YM34=', 'Content-Length': '0'} 2023-09-22T22:05:39.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f6fa505f7b990b1-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.720 DEBUG:boto:path=/ffzdqu-63/obj34 2023-09-22T22:05:39.720 DEBUG:boto:auth_path=/ffzdqu-63/obj34 2023-09-22T22:05:39.721 DEBUG:boto:path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.721 DEBUG:boto:auth_path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.721 DEBUG:boto:Method: HEAD 2023-09-22T22:05:39.721 DEBUG:boto:Path: /ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.721 DEBUG:boto:Data: 2023-09-22T22:05:39.721 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.721 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.721 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.721 DEBUG:boto:Params: {} 2023-09-22T22:05:39.721 DEBUG:boto:Token: None 2023-09-22T22:05:39.722 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.722 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:j2cm0uxtNtIwGktDoULu3X6YM34= 2023-09-22T22:05:39.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:j2cm0uxtNtIwGktDoULu3X6YM34=', 'Content-Length': '0'} 2023-09-22T22:05:39.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb488d1e2332e825-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T22:05:39.725 DEBUG:boto:path=/ffzdqu-63/obj34 2023-09-22T22:05:39.725 DEBUG:boto:auth_path=/ffzdqu-63/obj34 2023-09-22T22:05:39.726 DEBUG:boto:path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.726 DEBUG:boto:auth_path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.726 DEBUG:boto:Method: GET 2023-09-22T22:05:39.726 DEBUG:boto:Path: /ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.726 DEBUG:boto:Data: 2023-09-22T22:05:39.726 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.726 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.726 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.726 DEBUG:boto:Params: {} 2023-09-22T22:05:39.726 DEBUG:boto:Token: None 2023-09-22T22:05:39.727 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.727 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:akqqJFTLpznzrfYm6gSxDkWWKmM= 2023-09-22T22:05:39.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:akqqJFTLpznzrfYm6gSxDkWWKmM=', 'Content-Length': '0'} 2023-09-22T22:05:39.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000991cc56ffb5bcaf8-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.774 DEBUG:boto:path=/ffzdqu-63/obj34 2023-09-22T22:05:39.774 DEBUG:boto:auth_path=/ffzdqu-63/obj34 2023-09-22T22:05:39.774 DEBUG:boto:path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.775 DEBUG:boto:auth_path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.775 DEBUG:boto:Method: GET 2023-09-22T22:05:39.775 DEBUG:boto:Path: /ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.775 DEBUG:boto:Data: 2023-09-22T22:05:39.775 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.775 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.775 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.775 DEBUG:boto:Params: {} 2023-09-22T22:05:39.775 DEBUG:boto:Token: None 2023-09-22T22:05:39.775 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj34?versionId=null 2023-09-22T22:05:39.776 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:akqqJFTLpznzrfYm6gSxDkWWKmM= 2023-09-22T22:05:39.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:akqqJFTLpznzrfYm6gSxDkWWKmM=', 'Content-Length': '0'} 2023-09-22T22:05:39.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df55a959e46f7796-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.779 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:39.779 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:39.779 DEBUG:boto:path=/ffzdqu-63/?prefix=obj34 2023-09-22T22:05:39.779 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj34 2023-09-22T22:05:39.779 DEBUG:boto:Method: GET 2023-09-22T22:05:39.780 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj34 2023-09-22T22:05:39.780 DEBUG:boto:Data: 2023-09-22T22:05:39.780 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.780 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.780 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.780 DEBUG:boto:Params: {} 2023-09-22T22:05:39.780 DEBUG:boto:Token: None 2023-09-22T22:05:39.780 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/ 2023-09-22T22:05:39.780 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:05:39.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:05:39.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078d6f8516f374040-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.785 DEBUG:boto:b'ffzdqu-63obj341000falseobj342023-09-22T22:05:14.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:39.786 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:39.786 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:39.786 DEBUG:boto:path=/ffzdqu-63/?prefix=obj34 2023-09-22T22:05:39.786 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj34 2023-09-22T22:05:39.786 DEBUG:boto:Method: GET 2023-09-22T22:05:39.786 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj34 2023-09-22T22:05:39.786 DEBUG:boto:Data: 2023-09-22T22:05:39.786 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.786 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.787 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.787 DEBUG:boto:Params: {} 2023-09-22T22:05:39.787 DEBUG:boto:Token: None 2023-09-22T22:05:39.787 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/ 2023-09-22T22:05:39.787 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:05:39.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:05:39.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088f1caebd0bd8295-00650e0fb3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.842 DEBUG:boto:b'ffzdqu-63obj341000falseobj342023-09-22T22:05:14.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:39.843 DEBUG:boto:path=/ffzdqu-63/obj34 2023-09-22T22:05:39.843 DEBUG:boto:auth_path=/ffzdqu-63/obj34 2023-09-22T22:05:39.843 DEBUG:boto:Method: HEAD 2023-09-22T22:05:39.843 DEBUG:boto:Path: /ffzdqu-63/obj34 2023-09-22T22:05:39.843 DEBUG:boto:Data: 2023-09-22T22:05:39.843 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.843 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.843 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.843 DEBUG:boto:Params: {} 2023-09-22T22:05:39.844 DEBUG:boto:Token: None 2023-09-22T22:05:39.844 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj34 2023-09-22T22:05:39.844 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ynEgdWlacHXX3fqI8nl8YBSAtrA= 2023-09-22T22:05:39.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ynEgdWlacHXX3fqI8nl8YBSAtrA=', 'Content-Length': '0'} 2023-09-22T22:05:39.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000225eced0f33d2051-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.847 DEBUG:boto:path=/ffzdqu-63/obj34 2023-09-22T22:05:39.847 DEBUG:boto:auth_path=/ffzdqu-63/obj34 2023-09-22T22:05:39.847 DEBUG:boto:Method: HEAD 2023-09-22T22:05:39.847 DEBUG:boto:Path: /ffzdqu-63/obj34 2023-09-22T22:05:39.847 DEBUG:boto:Data: 2023-09-22T22:05:39.847 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.848 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.848 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.848 DEBUG:boto:Params: {} 2023-09-22T22:05:39.848 DEBUG:boto:Token: None 2023-09-22T22:05:39.848 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj34 2023-09-22T22:05:39.848 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ynEgdWlacHXX3fqI8nl8YBSAtrA= 2023-09-22T22:05:39.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ynEgdWlacHXX3fqI8nl8YBSAtrA=', 'Content-Length': '0'} 2023-09-22T22:05:39.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae30d85fd8da6b9d-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-22T22:05:39.851 DEBUG:boto:path=/ffzdqu-63/obj34 2023-09-22T22:05:39.851 DEBUG:boto:auth_path=/ffzdqu-63/obj34 2023-09-22T22:05:39.852 DEBUG:boto:Method: GET 2023-09-22T22:05:39.852 DEBUG:boto:Path: /ffzdqu-63/obj34 2023-09-22T22:05:39.852 DEBUG:boto:Data: 2023-09-22T22:05:39.852 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.852 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.852 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.852 DEBUG:boto:Params: {} 2023-09-22T22:05:39.852 DEBUG:boto:Token: None 2023-09-22T22:05:39.852 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj34 2023-09-22T22:05:39.853 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oj/WlJNtJ5n3m1hE/jJL/5tQeds= 2023-09-22T22:05:39.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oj/WlJNtJ5n3m1hE/jJL/5tQeds=', 'Content-Length': '0'} 2023-09-22T22:05:39.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6b5b6002cac5ac8-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.856 DEBUG:boto:path=/ffzdqu-63/obj34 2023-09-22T22:05:39.856 DEBUG:boto:auth_path=/ffzdqu-63/obj34 2023-09-22T22:05:39.856 DEBUG:boto:Method: GET 2023-09-22T22:05:39.856 DEBUG:boto:Path: /ffzdqu-63/obj34 2023-09-22T22:05:39.856 DEBUG:boto:Data: 2023-09-22T22:05:39.857 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.857 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.857 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.857 DEBUG:boto:Params: {} 2023-09-22T22:05:39.857 DEBUG:boto:Token: None 2023-09-22T22:05:39.857 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj34 2023-09-22T22:05:39.857 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oj/WlJNtJ5n3m1hE/jJL/5tQeds= 2023-09-22T22:05:39.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oj/WlJNtJ5n3m1hE/jJL/5tQeds=', 'Content-Length': '0'} 2023-09-22T22:05:39.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dc8f3a6d0c74040-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.861 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:39.861 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:39.861 DEBUG:boto:path=/ffzdqu-63/?prefix=obj34 2023-09-22T22:05:39.861 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj34 2023-09-22T22:05:39.861 DEBUG:boto:Method: GET 2023-09-22T22:05:39.861 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj34 2023-09-22T22:05:39.861 DEBUG:boto:Data: 2023-09-22T22:05:39.861 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.861 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.861 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.862 DEBUG:boto:Params: {} 2023-09-22T22:05:39.862 DEBUG:boto:Token: None 2023-09-22T22:05:39.862 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/ 2023-09-22T22:05:39.862 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:05:39.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:05:39.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eacf51c4f6c934ef-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.910 DEBUG:boto:b'ffzdqu-63obj341000falseobj342023-09-22T22:05:14.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:39.911 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:39.911 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:39.911 DEBUG:boto:path=/ffzdqu-63/?prefix=obj34 2023-09-22T22:05:39.911 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj34 2023-09-22T22:05:39.911 DEBUG:boto:Method: GET 2023-09-22T22:05:39.911 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj34 2023-09-22T22:05:39.911 DEBUG:boto:Data: 2023-09-22T22:05:39.911 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.911 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.912 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.912 DEBUG:boto:Params: {} 2023-09-22T22:05:39.912 DEBUG:boto:Token: None 2023-09-22T22:05:39.912 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/ 2023-09-22T22:05:39.912 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:05:39.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:05:39.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8aba1039505164e-00650e0fb3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.929 DEBUG:boto:b'ffzdqu-63obj341000falseobj342023-09-22T22:05:14.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:39.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-09-22T22:05:39.929 DEBUG:boto:path=/ffzdqu-63/obj37 2023-09-22T22:05:39.930 DEBUG:boto:auth_path=/ffzdqu-63/obj37 2023-09-22T22:05:39.930 DEBUG:boto:path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:39.930 DEBUG:boto:auth_path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:39.930 DEBUG:boto:Method: GET 2023-09-22T22:05:39.930 DEBUG:boto:Path: /ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:39.930 DEBUG:boto:Data: 2023-09-22T22:05:39.930 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.930 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.930 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.931 DEBUG:boto:Params: {} 2023-09-22T22:05:39.931 DEBUG:boto:Token: None 2023-09-22T22:05:39.931 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:39.931 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tMMV6l4vg0+L8hoMus+8oWv1N1U= 2023-09-22T22:05:39.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tMMV6l4vg0+L8hoMus+8oWv1N1U=', 'Content-Length': '0'} 2023-09-22T22:05:39.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da2d9eef6987f003-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.934 DEBUG:boto:path=/ffzdqu-63/obj37 2023-09-22T22:05:39.934 DEBUG:boto:auth_path=/ffzdqu-63/obj37 2023-09-22T22:05:39.934 DEBUG:boto:path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:39.934 DEBUG:boto:auth_path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:39.934 DEBUG:boto:Method: GET 2023-09-22T22:05:39.935 DEBUG:boto:Path: /ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:39.935 DEBUG:boto:Data: 2023-09-22T22:05:39.935 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.935 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.935 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.935 DEBUG:boto:Params: {} 2023-09-22T22:05:39.935 DEBUG:boto:Token: None 2023-09-22T22:05:39.935 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:39.935 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tMMV6l4vg0+L8hoMus+8oWv1N1U= 2023-09-22T22:05:39.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tMMV6l4vg0+L8hoMus+8oWv1N1U=', 'Content-Length': '0'} 2023-09-22T22:05:39.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af78e68916ed7fc7-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.986 DEBUG:boto:path=/ffzdqu-63/obj37 2023-09-22T22:05:39.986 DEBUG:boto:auth_path=/ffzdqu-63/obj37 2023-09-22T22:05:39.987 DEBUG:boto:path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:39.987 DEBUG:boto:auth_path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:39.987 DEBUG:boto:Method: HEAD 2023-09-22T22:05:39.987 DEBUG:boto:Path: /ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:39.987 DEBUG:boto:Data: 2023-09-22T22:05:39.987 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.987 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.987 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.987 DEBUG:boto:Params: {} 2023-09-22T22:05:39.988 DEBUG:boto:Token: None 2023-09-22T22:05:39.988 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:39.988 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8WRoQzi1CkUkYQIzpuA6lbdMmRc= 2023-09-22T22:05:39.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8WRoQzi1CkUkYQIzpuA6lbdMmRc=', 'Content-Length': '0'} 2023-09-22T22:05:39.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000396c13d5e764fc0e-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.991 DEBUG:boto:path=/ffzdqu-63/obj37 2023-09-22T22:05:39.991 DEBUG:boto:auth_path=/ffzdqu-63/obj37 2023-09-22T22:05:39.991 DEBUG:boto:path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:39.991 DEBUG:boto:auth_path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:39.991 DEBUG:boto:Method: HEAD 2023-09-22T22:05:39.991 DEBUG:boto:Path: /ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:39.991 DEBUG:boto:Data: 2023-09-22T22:05:39.992 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.992 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:39.992 DEBUG:boto:Port: 8000 2023-09-22T22:05:39.992 DEBUG:boto:Params: {} 2023-09-22T22:05:39.992 DEBUG:boto:Token: None 2023-09-22T22:05:39.992 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:39.992 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8WRoQzi1CkUkYQIzpuA6lbdMmRc= 2023-09-22T22:05:39.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8WRoQzi1CkUkYQIzpuA6lbdMmRc=', 'Content-Length': '0'} 2023-09-22T22:05:39.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000776fe2ea47775a0a-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:39.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-09-22T22:05:39.996 DEBUG:boto:path=/ffzdqu-63/obj37 2023-09-22T22:05:39.996 DEBUG:boto:auth_path=/ffzdqu-63/obj37 2023-09-22T22:05:39.996 DEBUG:boto:path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:39.996 DEBUG:boto:auth_path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:39.996 DEBUG:boto:Method: GET 2023-09-22T22:05:39.996 DEBUG:boto:Path: /ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:39.996 DEBUG:boto:Data: 2023-09-22T22:05:39.997 DEBUG:boto:Headers: {} 2023-09-22T22:05:39.997 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:39.997 DEBUG:boto:Port: 8001 2023-09-22T22:05:39.997 DEBUG:boto:Params: {} 2023-09-22T22:05:39.997 DEBUG:boto:Token: None 2023-09-22T22:05:39.997 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:39 GMT /ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:39.997 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tMMV6l4vg0+L8hoMus+8oWv1N1U= 2023-09-22T22:05:39.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tMMV6l4vg0+L8hoMus+8oWv1N1U=', 'Content-Length': '0'} 2023-09-22T22:05:40.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dae22ee003619dc1-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.001 DEBUG:boto:path=/ffzdqu-63/obj37 2023-09-22T22:05:40.001 DEBUG:boto:auth_path=/ffzdqu-63/obj37 2023-09-22T22:05:40.001 DEBUG:boto:path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:40.001 DEBUG:boto:auth_path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:40.001 DEBUG:boto:Method: GET 2023-09-22T22:05:40.001 DEBUG:boto:Path: /ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:40.002 DEBUG:boto:Data: 2023-09-22T22:05:40.002 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.002 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.002 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.002 DEBUG:boto:Params: {} 2023-09-22T22:05:40.002 DEBUG:boto:Token: None 2023-09-22T22:05:40.002 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj37?versionId=null 2023-09-22T22:05:40.002 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RRgZJRYs+Zpq3Z33XrYHN1Uix4I= 2023-09-22T22:05:40.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RRgZJRYs+Zpq3Z33XrYHN1Uix4I=', 'Content-Length': '0'} 2023-09-22T22:05:40.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000415545d9856815d-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.006 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.006 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.006 DEBUG:boto:path=/ffzdqu-63/?prefix=obj37 2023-09-22T22:05:40.006 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj37 2023-09-22T22:05:40.006 DEBUG:boto:Method: GET 2023-09-22T22:05:40.006 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj37 2023-09-22T22:05:40.006 DEBUG:boto:Data: 2023-09-22T22:05:40.006 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.006 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.007 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.007 DEBUG:boto:Params: {} 2023-09-22T22:05:40.007 DEBUG:boto:Token: None 2023-09-22T22:05:40.007 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.007 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d066638a228bca24-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.058 DEBUG:boto:b'ffzdqu-63obj371000falseobj372023-09-22T22:05:15.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:40.059 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.059 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.059 DEBUG:boto:path=/ffzdqu-63/?prefix=obj37 2023-09-22T22:05:40.059 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj37 2023-09-22T22:05:40.059 DEBUG:boto:Method: GET 2023-09-22T22:05:40.059 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj37 2023-09-22T22:05:40.060 DEBUG:boto:Data: 2023-09-22T22:05:40.060 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.060 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.060 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.060 DEBUG:boto:Params: {} 2023-09-22T22:05:40.060 DEBUG:boto:Token: None 2023-09-22T22:05:40.060 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.060 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000367a57655a16ef04-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.070 DEBUG:boto:b'ffzdqu-63obj371000falseobj372023-09-22T22:05:15.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:40.070 DEBUG:boto:path=/ffzdqu-63/obj37 2023-09-22T22:05:40.070 DEBUG:boto:auth_path=/ffzdqu-63/obj37 2023-09-22T22:05:40.071 DEBUG:boto:Method: HEAD 2023-09-22T22:05:40.071 DEBUG:boto:Path: /ffzdqu-63/obj37 2023-09-22T22:05:40.071 DEBUG:boto:Data: 2023-09-22T22:05:40.071 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.071 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.071 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.071 DEBUG:boto:Params: {} 2023-09-22T22:05:40.071 DEBUG:boto:Token: None 2023-09-22T22:05:40.071 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj37 2023-09-22T22:05:40.072 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1CIbB0EUuvhCN0rZPCh+C8Rtm1g= 2023-09-22T22:05:40.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1CIbB0EUuvhCN0rZPCh+C8Rtm1g=', 'Content-Length': '0'} 2023-09-22T22:05:40.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012930c9b1df54a81-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.075 DEBUG:boto:path=/ffzdqu-63/obj37 2023-09-22T22:05:40.075 DEBUG:boto:auth_path=/ffzdqu-63/obj37 2023-09-22T22:05:40.075 DEBUG:boto:Method: HEAD 2023-09-22T22:05:40.075 DEBUG:boto:Path: /ffzdqu-63/obj37 2023-09-22T22:05:40.075 DEBUG:boto:Data: 2023-09-22T22:05:40.075 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.075 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.075 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.076 DEBUG:boto:Params: {} 2023-09-22T22:05:40.076 DEBUG:boto:Token: None 2023-09-22T22:05:40.076 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj37 2023-09-22T22:05:40.076 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1CIbB0EUuvhCN0rZPCh+C8Rtm1g= 2023-09-22T22:05:40.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1CIbB0EUuvhCN0rZPCh+C8Rtm1g=', 'Content-Length': '0'} 2023-09-22T22:05:40.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9ccc51ede129352-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-09-22T22:05:40.079 DEBUG:boto:path=/ffzdqu-63/obj37 2023-09-22T22:05:40.079 DEBUG:boto:auth_path=/ffzdqu-63/obj37 2023-09-22T22:05:40.079 DEBUG:boto:Method: GET 2023-09-22T22:05:40.079 DEBUG:boto:Path: /ffzdqu-63/obj37 2023-09-22T22:05:40.079 DEBUG:boto:Data: 2023-09-22T22:05:40.080 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.080 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.080 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.080 DEBUG:boto:Params: {} 2023-09-22T22:05:40.080 DEBUG:boto:Token: None 2023-09-22T22:05:40.080 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj37 2023-09-22T22:05:40.080 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1gkWHUmBkAXSOau12UA+Pa/tTVQ= 2023-09-22T22:05:40.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1gkWHUmBkAXSOau12UA+Pa/tTVQ=', 'Content-Length': '0'} 2023-09-22T22:05:40.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e4bfbe063dc7a2f-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.083 DEBUG:boto:path=/ffzdqu-63/obj37 2023-09-22T22:05:40.084 DEBUG:boto:auth_path=/ffzdqu-63/obj37 2023-09-22T22:05:40.084 DEBUG:boto:Method: GET 2023-09-22T22:05:40.084 DEBUG:boto:Path: /ffzdqu-63/obj37 2023-09-22T22:05:40.084 DEBUG:boto:Data: 2023-09-22T22:05:40.084 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.084 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.084 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.084 DEBUG:boto:Params: {} 2023-09-22T22:05:40.085 DEBUG:boto:Token: None 2023-09-22T22:05:40.085 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj37 2023-09-22T22:05:40.085 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1gkWHUmBkAXSOau12UA+Pa/tTVQ= 2023-09-22T22:05:40.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1gkWHUmBkAXSOau12UA+Pa/tTVQ=', 'Content-Length': '0'} 2023-09-22T22:05:40.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d1bacb6d3635a78-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.130 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.131 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.131 DEBUG:boto:path=/ffzdqu-63/?prefix=obj37 2023-09-22T22:05:40.131 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj37 2023-09-22T22:05:40.133 DEBUG:boto:Method: GET 2023-09-22T22:05:40.133 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj37 2023-09-22T22:05:40.133 DEBUG:boto:Data: 2023-09-22T22:05:40.133 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.134 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.134 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.134 DEBUG:boto:Params: {} 2023-09-22T22:05:40.134 DEBUG:boto:Token: None 2023-09-22T22:05:40.134 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.134 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d732a3651e466b3-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.141 DEBUG:boto:b'ffzdqu-63obj371000falseobj372023-09-22T22:05:15.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:40.141 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.141 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.141 DEBUG:boto:path=/ffzdqu-63/?prefix=obj37 2023-09-22T22:05:40.141 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj37 2023-09-22T22:05:40.142 DEBUG:boto:Method: GET 2023-09-22T22:05:40.142 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj37 2023-09-22T22:05:40.142 DEBUG:boto:Data: 2023-09-22T22:05:40.142 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.142 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.142 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.142 DEBUG:boto:Params: {} 2023-09-22T22:05:40.142 DEBUG:boto:Token: None 2023-09-22T22:05:40.142 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.143 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b892a9ba676902d-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.155 DEBUG:boto:b'ffzdqu-63obj371000falseobj372023-09-22T22:05:15.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:40.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T22:05:40.156 DEBUG:boto:path=/ffzdqu-63/obj38 2023-09-22T22:05:40.156 DEBUG:boto:auth_path=/ffzdqu-63/obj38 2023-09-22T22:05:40.156 DEBUG:boto:path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.156 DEBUG:boto:auth_path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.156 DEBUG:boto:Method: GET 2023-09-22T22:05:40.156 DEBUG:boto:Path: /ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.156 DEBUG:boto:Data: 2023-09-22T22:05:40.156 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.156 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.157 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.157 DEBUG:boto:Params: {} 2023-09-22T22:05:40.157 DEBUG:boto:Token: None 2023-09-22T22:05:40.157 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.157 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fPNN++EmkVj7/Pyc745/p+ve9xs= 2023-09-22T22:05:40.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fPNN++EmkVj7/Pyc745/p+ve9xs=', 'Content-Length': '0'} 2023-09-22T22:05:40.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e1088a566b4f34c-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.202 DEBUG:boto:path=/ffzdqu-63/obj38 2023-09-22T22:05:40.202 DEBUG:boto:auth_path=/ffzdqu-63/obj38 2023-09-22T22:05:40.203 DEBUG:boto:path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.203 DEBUG:boto:auth_path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.203 DEBUG:boto:Method: GET 2023-09-22T22:05:40.203 DEBUG:boto:Path: /ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.203 DEBUG:boto:Data: 2023-09-22T22:05:40.203 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.203 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.203 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.203 DEBUG:boto:Params: {} 2023-09-22T22:05:40.203 DEBUG:boto:Token: None 2023-09-22T22:05:40.204 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.204 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fPNN++EmkVj7/Pyc745/p+ve9xs= 2023-09-22T22:05:40.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fPNN++EmkVj7/Pyc745/p+ve9xs=', 'Content-Length': '0'} 2023-09-22T22:05:40.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c822640d7327322d-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.207 DEBUG:boto:path=/ffzdqu-63/obj38 2023-09-22T22:05:40.207 DEBUG:boto:auth_path=/ffzdqu-63/obj38 2023-09-22T22:05:40.207 DEBUG:boto:path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.208 DEBUG:boto:auth_path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.208 DEBUG:boto:Method: HEAD 2023-09-22T22:05:40.208 DEBUG:boto:Path: /ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.208 DEBUG:boto:Data: 2023-09-22T22:05:40.208 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.208 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.208 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.208 DEBUG:boto:Params: {} 2023-09-22T22:05:40.208 DEBUG:boto:Token: None 2023-09-22T22:05:40.208 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.209 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/rKp9M7/U2mw7ynEHzUc6eZkAJE= 2023-09-22T22:05:40.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/rKp9M7/U2mw7ynEHzUc6eZkAJE=', 'Content-Length': '0'} 2023-09-22T22:05:40.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d91caefe6fda4041-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.212 DEBUG:boto:path=/ffzdqu-63/obj38 2023-09-22T22:05:40.212 DEBUG:boto:auth_path=/ffzdqu-63/obj38 2023-09-22T22:05:40.212 DEBUG:boto:path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.212 DEBUG:boto:auth_path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.212 DEBUG:boto:Method: HEAD 2023-09-22T22:05:40.212 DEBUG:boto:Path: /ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.212 DEBUG:boto:Data: 2023-09-22T22:05:40.212 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.212 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.212 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.213 DEBUG:boto:Params: {} 2023-09-22T22:05:40.213 DEBUG:boto:Token: None 2023-09-22T22:05:40.213 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.213 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/rKp9M7/U2mw7ynEHzUc6eZkAJE= 2023-09-22T22:05:40.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/rKp9M7/U2mw7ynEHzUc6eZkAJE=', 'Content-Length': '0'} 2023-09-22T22:05:40.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007da552b6eb1f5705-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T22:05:40.216 DEBUG:boto:path=/ffzdqu-63/obj38 2023-09-22T22:05:40.216 DEBUG:boto:auth_path=/ffzdqu-63/obj38 2023-09-22T22:05:40.216 DEBUG:boto:path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.216 DEBUG:boto:auth_path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.216 DEBUG:boto:Method: GET 2023-09-22T22:05:40.217 DEBUG:boto:Path: /ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.217 DEBUG:boto:Data: 2023-09-22T22:05:40.217 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.217 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.217 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.217 DEBUG:boto:Params: {} 2023-09-22T22:05:40.217 DEBUG:boto:Token: None 2023-09-22T22:05:40.218 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.218 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fPNN++EmkVj7/Pyc745/p+ve9xs= 2023-09-22T22:05:40.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fPNN++EmkVj7/Pyc745/p+ve9xs=', 'Content-Length': '0'} 2023-09-22T22:05:40.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042f2a0e10276ef4e-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.223 DEBUG:boto:path=/ffzdqu-63/obj38 2023-09-22T22:05:40.223 DEBUG:boto:auth_path=/ffzdqu-63/obj38 2023-09-22T22:05:40.223 DEBUG:boto:path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.223 DEBUG:boto:auth_path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.223 DEBUG:boto:Method: GET 2023-09-22T22:05:40.223 DEBUG:boto:Path: /ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.223 DEBUG:boto:Data: 2023-09-22T22:05:40.223 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.223 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.224 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.224 DEBUG:boto:Params: {} 2023-09-22T22:05:40.224 DEBUG:boto:Token: None 2023-09-22T22:05:40.224 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj38?versionId=null 2023-09-22T22:05:40.224 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fPNN++EmkVj7/Pyc745/p+ve9xs= 2023-09-22T22:05:40.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fPNN++EmkVj7/Pyc745/p+ve9xs=', 'Content-Length': '0'} 2023-09-22T22:05:40.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e15f32fcdcb3291-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.270 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.271 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.271 DEBUG:boto:path=/ffzdqu-63/?prefix=obj38 2023-09-22T22:05:40.271 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj38 2023-09-22T22:05:40.271 DEBUG:boto:Method: GET 2023-09-22T22:05:40.271 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj38 2023-09-22T22:05:40.271 DEBUG:boto:Data: 2023-09-22T22:05:40.271 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.271 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.271 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.271 DEBUG:boto:Params: {} 2023-09-22T22:05:40.272 DEBUG:boto:Token: None 2023-09-22T22:05:40.272 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.272 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090eb80a566ea4088-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.277 DEBUG:boto:b'ffzdqu-63obj381000falseobj382023-09-22T22:05:14.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:40.278 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.278 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.278 DEBUG:boto:path=/ffzdqu-63/?prefix=obj38 2023-09-22T22:05:40.278 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj38 2023-09-22T22:05:40.278 DEBUG:boto:Method: GET 2023-09-22T22:05:40.278 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj38 2023-09-22T22:05:40.278 DEBUG:boto:Data: 2023-09-22T22:05:40.278 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.278 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.279 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.279 DEBUG:boto:Params: {} 2023-09-22T22:05:40.279 DEBUG:boto:Token: None 2023-09-22T22:05:40.279 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.279 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052fb9ef93f16fc56-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.290 DEBUG:boto:b'ffzdqu-63obj381000falseobj382023-09-22T22:05:14.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:40.290 DEBUG:boto:path=/ffzdqu-63/obj38 2023-09-22T22:05:40.291 DEBUG:boto:auth_path=/ffzdqu-63/obj38 2023-09-22T22:05:40.291 DEBUG:boto:Method: HEAD 2023-09-22T22:05:40.291 DEBUG:boto:Path: /ffzdqu-63/obj38 2023-09-22T22:05:40.291 DEBUG:boto:Data: 2023-09-22T22:05:40.291 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.291 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.291 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.291 DEBUG:boto:Params: {} 2023-09-22T22:05:40.291 DEBUG:boto:Token: None 2023-09-22T22:05:40.292 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj38 2023-09-22T22:05:40.292 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RhXSwvM/7Vc6nh8WPji+Uqw47/w= 2023-09-22T22:05:40.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RhXSwvM/7Vc6nh8WPji+Uqw47/w=', 'Content-Length': '0'} 2023-09-22T22:05:40.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ce304f7052a33ad-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.295 DEBUG:boto:path=/ffzdqu-63/obj38 2023-09-22T22:05:40.295 DEBUG:boto:auth_path=/ffzdqu-63/obj38 2023-09-22T22:05:40.295 DEBUG:boto:Method: HEAD 2023-09-22T22:05:40.295 DEBUG:boto:Path: /ffzdqu-63/obj38 2023-09-22T22:05:40.295 DEBUG:boto:Data: 2023-09-22T22:05:40.296 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.296 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.296 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.296 DEBUG:boto:Params: {} 2023-09-22T22:05:40.296 DEBUG:boto:Token: None 2023-09-22T22:05:40.296 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj38 2023-09-22T22:05:40.296 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RhXSwvM/7Vc6nh8WPji+Uqw47/w= 2023-09-22T22:05:40.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RhXSwvM/7Vc6nh8WPji+Uqw47/w=', 'Content-Length': '0'} 2023-09-22T22:05:40.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f75c05b19793157-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-22T22:05:40.301 DEBUG:boto:path=/ffzdqu-63/obj38 2023-09-22T22:05:40.301 DEBUG:boto:auth_path=/ffzdqu-63/obj38 2023-09-22T22:05:40.301 DEBUG:boto:Method: GET 2023-09-22T22:05:40.301 DEBUG:boto:Path: /ffzdqu-63/obj38 2023-09-22T22:05:40.301 DEBUG:boto:Data: 2023-09-22T22:05:40.301 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.302 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.302 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.302 DEBUG:boto:Params: {} 2023-09-22T22:05:40.302 DEBUG:boto:Token: None 2023-09-22T22:05:40.302 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj38 2023-09-22T22:05:40.302 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J6qmxcd842GC8frB3hrMRAusR00= 2023-09-22T22:05:40.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J6qmxcd842GC8frB3hrMRAusR00=', 'Content-Length': '0'} 2023-09-22T22:05:40.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c86fc0e5200a69a-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.346 DEBUG:boto:path=/ffzdqu-63/obj38 2023-09-22T22:05:40.347 DEBUG:boto:auth_path=/ffzdqu-63/obj38 2023-09-22T22:05:40.347 DEBUG:boto:Method: GET 2023-09-22T22:05:40.347 DEBUG:boto:Path: /ffzdqu-63/obj38 2023-09-22T22:05:40.347 DEBUG:boto:Data: 2023-09-22T22:05:40.347 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.347 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.347 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.347 DEBUG:boto:Params: {} 2023-09-22T22:05:40.347 DEBUG:boto:Token: None 2023-09-22T22:05:40.348 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj38 2023-09-22T22:05:40.348 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:J6qmxcd842GC8frB3hrMRAusR00= 2023-09-22T22:05:40.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J6qmxcd842GC8frB3hrMRAusR00=', 'Content-Length': '0'} 2023-09-22T22:05:40.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005abd6d04298c3479-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.352 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.352 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.352 DEBUG:boto:path=/ffzdqu-63/?prefix=obj38 2023-09-22T22:05:40.352 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj38 2023-09-22T22:05:40.352 DEBUG:boto:Method: GET 2023-09-22T22:05:40.352 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj38 2023-09-22T22:05:40.352 DEBUG:boto:Data: 2023-09-22T22:05:40.352 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.352 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.353 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.353 DEBUG:boto:Params: {} 2023-09-22T22:05:40.353 DEBUG:boto:Token: None 2023-09-22T22:05:40.353 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.353 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0ccaa28b63fa1ed-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.358 DEBUG:boto:b'ffzdqu-63obj381000falseobj382023-09-22T22:05:14.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:40.358 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.359 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.359 DEBUG:boto:path=/ffzdqu-63/?prefix=obj38 2023-09-22T22:05:40.359 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj38 2023-09-22T22:05:40.359 DEBUG:boto:Method: GET 2023-09-22T22:05:40.359 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj38 2023-09-22T22:05:40.359 DEBUG:boto:Data: 2023-09-22T22:05:40.359 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.359 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.359 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.359 DEBUG:boto:Params: {} 2023-09-22T22:05:40.360 DEBUG:boto:Token: None 2023-09-22T22:05:40.360 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.360 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb00e72b24913425-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.410 DEBUG:boto:b'ffzdqu-63obj381000falseobj382023-09-22T22:05:14.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:40.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T22:05:40.411 DEBUG:boto:path=/ffzdqu-63/obj44 2023-09-22T22:05:40.411 DEBUG:boto:auth_path=/ffzdqu-63/obj44 2023-09-22T22:05:40.411 DEBUG:boto:path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.411 DEBUG:boto:auth_path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.411 DEBUG:boto:Method: GET 2023-09-22T22:05:40.411 DEBUG:boto:Path: /ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.411 DEBUG:boto:Data: 2023-09-22T22:05:40.411 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.412 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.412 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.412 DEBUG:boto:Params: {} 2023-09-22T22:05:40.412 DEBUG:boto:Token: None 2023-09-22T22:05:40.412 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.412 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AlajPbA0VfaL+rN7XEawl/36qdg= 2023-09-22T22:05:40.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AlajPbA0VfaL+rN7XEawl/36qdg=', 'Content-Length': '0'} 2023-09-22T22:05:40.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cd6aa3a0f2b3ec5-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.415 DEBUG:boto:path=/ffzdqu-63/obj44 2023-09-22T22:05:40.416 DEBUG:boto:auth_path=/ffzdqu-63/obj44 2023-09-22T22:05:40.416 DEBUG:boto:path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.416 DEBUG:boto:auth_path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.416 DEBUG:boto:Method: GET 2023-09-22T22:05:40.416 DEBUG:boto:Path: /ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.416 DEBUG:boto:Data: 2023-09-22T22:05:40.416 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.416 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.416 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.417 DEBUG:boto:Params: {} 2023-09-22T22:05:40.417 DEBUG:boto:Token: None 2023-09-22T22:05:40.417 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.417 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AlajPbA0VfaL+rN7XEawl/36qdg= 2023-09-22T22:05:40.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AlajPbA0VfaL+rN7XEawl/36qdg=', 'Content-Length': '0'} 2023-09-22T22:05:40.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee7c53020e6dcb56-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.420 DEBUG:boto:path=/ffzdqu-63/obj44 2023-09-22T22:05:40.420 DEBUG:boto:auth_path=/ffzdqu-63/obj44 2023-09-22T22:05:40.420 DEBUG:boto:path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.420 DEBUG:boto:auth_path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.421 DEBUG:boto:Method: HEAD 2023-09-22T22:05:40.421 DEBUG:boto:Path: /ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.421 DEBUG:boto:Data: 2023-09-22T22:05:40.421 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.421 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.421 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.421 DEBUG:boto:Params: {} 2023-09-22T22:05:40.421 DEBUG:boto:Token: None 2023-09-22T22:05:40.421 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.422 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0J2r0OAitL6gcisFFJPLjCpGF3k= 2023-09-22T22:05:40.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0J2r0OAitL6gcisFFJPLjCpGF3k=', 'Content-Length': '0'} 2023-09-22T22:05:40.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000654bc0358b42a01c-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.425 DEBUG:boto:path=/ffzdqu-63/obj44 2023-09-22T22:05:40.425 DEBUG:boto:auth_path=/ffzdqu-63/obj44 2023-09-22T22:05:40.425 DEBUG:boto:path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.425 DEBUG:boto:auth_path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.425 DEBUG:boto:Method: HEAD 2023-09-22T22:05:40.425 DEBUG:boto:Path: /ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.425 DEBUG:boto:Data: 2023-09-22T22:05:40.425 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.425 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.426 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.426 DEBUG:boto:Params: {} 2023-09-22T22:05:40.426 DEBUG:boto:Token: None 2023-09-22T22:05:40.426 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.426 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0J2r0OAitL6gcisFFJPLjCpGF3k= 2023-09-22T22:05:40.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0J2r0OAitL6gcisFFJPLjCpGF3k=', 'Content-Length': '0'} 2023-09-22T22:05:40.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051070ed008a011d5-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T22:05:40.429 DEBUG:boto:path=/ffzdqu-63/obj44 2023-09-22T22:05:40.429 DEBUG:boto:auth_path=/ffzdqu-63/obj44 2023-09-22T22:05:40.429 DEBUG:boto:path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.429 DEBUG:boto:auth_path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.429 DEBUG:boto:Method: GET 2023-09-22T22:05:40.430 DEBUG:boto:Path: /ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.430 DEBUG:boto:Data: 2023-09-22T22:05:40.430 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.430 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.430 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.430 DEBUG:boto:Params: {} 2023-09-22T22:05:40.430 DEBUG:boto:Token: None 2023-09-22T22:05:40.430 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.431 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AlajPbA0VfaL+rN7XEawl/36qdg= 2023-09-22T22:05:40.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AlajPbA0VfaL+rN7XEawl/36qdg=', 'Content-Length': '0'} 2023-09-22T22:05:40.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b4535e10c947621-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.434 DEBUG:boto:path=/ffzdqu-63/obj44 2023-09-22T22:05:40.434 DEBUG:boto:auth_path=/ffzdqu-63/obj44 2023-09-22T22:05:40.434 DEBUG:boto:path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.434 DEBUG:boto:auth_path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.434 DEBUG:boto:Method: GET 2023-09-22T22:05:40.434 DEBUG:boto:Path: /ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.434 DEBUG:boto:Data: 2023-09-22T22:05:40.434 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.434 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.434 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.435 DEBUG:boto:Params: {} 2023-09-22T22:05:40.435 DEBUG:boto:Token: None 2023-09-22T22:05:40.435 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj44?versionId=null 2023-09-22T22:05:40.435 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AlajPbA0VfaL+rN7XEawl/36qdg= 2023-09-22T22:05:40.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AlajPbA0VfaL+rN7XEawl/36qdg=', 'Content-Length': '0'} 2023-09-22T22:05:40.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f58da9067887741-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.438 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.438 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.438 DEBUG:boto:path=/ffzdqu-63/?prefix=obj44 2023-09-22T22:05:40.438 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj44 2023-09-22T22:05:40.439 DEBUG:boto:Method: GET 2023-09-22T22:05:40.439 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj44 2023-09-22T22:05:40.439 DEBUG:boto:Data: 2023-09-22T22:05:40.439 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.439 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.439 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.439 DEBUG:boto:Params: {} 2023-09-22T22:05:40.439 DEBUG:boto:Token: None 2023-09-22T22:05:40.439 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.440 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000539fbad568c89e96-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.445 DEBUG:boto:b'ffzdqu-63obj441000falseobj442023-09-22T22:05:13.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:40.445 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.446 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.446 DEBUG:boto:path=/ffzdqu-63/?prefix=obj44 2023-09-22T22:05:40.446 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj44 2023-09-22T22:05:40.446 DEBUG:boto:Method: GET 2023-09-22T22:05:40.446 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj44 2023-09-22T22:05:40.446 DEBUG:boto:Data: 2023-09-22T22:05:40.446 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.446 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.446 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.447 DEBUG:boto:Params: {} 2023-09-22T22:05:40.447 DEBUG:boto:Token: None 2023-09-22T22:05:40.447 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.447 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4086c4779db3cbc-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.458 DEBUG:boto:b'ffzdqu-63obj441000falseobj442023-09-22T22:05:13.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:40.459 DEBUG:boto:path=/ffzdqu-63/obj44 2023-09-22T22:05:40.459 DEBUG:boto:auth_path=/ffzdqu-63/obj44 2023-09-22T22:05:40.459 DEBUG:boto:Method: HEAD 2023-09-22T22:05:40.459 DEBUG:boto:Path: /ffzdqu-63/obj44 2023-09-22T22:05:40.459 DEBUG:boto:Data: 2023-09-22T22:05:40.459 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.459 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.459 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.459 DEBUG:boto:Params: {} 2023-09-22T22:05:40.460 DEBUG:boto:Token: None 2023-09-22T22:05:40.460 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj44 2023-09-22T22:05:40.460 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kcwDutLOF3sQt2VKad4KG1HjJUM= 2023-09-22T22:05:40.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kcwDutLOF3sQt2VKad4KG1HjJUM=', 'Content-Length': '0'} 2023-09-22T22:05:40.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9bde5035d17c939-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.463 DEBUG:boto:path=/ffzdqu-63/obj44 2023-09-22T22:05:40.463 DEBUG:boto:auth_path=/ffzdqu-63/obj44 2023-09-22T22:05:40.463 DEBUG:boto:Method: HEAD 2023-09-22T22:05:40.463 DEBUG:boto:Path: /ffzdqu-63/obj44 2023-09-22T22:05:40.463 DEBUG:boto:Data: 2023-09-22T22:05:40.463 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.463 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.463 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.463 DEBUG:boto:Params: {} 2023-09-22T22:05:40.464 DEBUG:boto:Token: None 2023-09-22T22:05:40.464 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj44 2023-09-22T22:05:40.464 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kcwDutLOF3sQt2VKad4KG1HjJUM= 2023-09-22T22:05:40.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kcwDutLOF3sQt2VKad4KG1HjJUM=', 'Content-Length': '0'} 2023-09-22T22:05:40.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a5d9731e36d9d54-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T22:05:40.467 DEBUG:boto:path=/ffzdqu-63/obj44 2023-09-22T22:05:40.467 DEBUG:boto:auth_path=/ffzdqu-63/obj44 2023-09-22T22:05:40.467 DEBUG:boto:Method: GET 2023-09-22T22:05:40.467 DEBUG:boto:Path: /ffzdqu-63/obj44 2023-09-22T22:05:40.467 DEBUG:boto:Data: 2023-09-22T22:05:40.467 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.467 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.467 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.468 DEBUG:boto:Params: {} 2023-09-22T22:05:40.468 DEBUG:boto:Token: None 2023-09-22T22:05:40.468 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj44 2023-09-22T22:05:40.468 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7C8xyDVQwG31/49Siciixifpay4= 2023-09-22T22:05:40.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7C8xyDVQwG31/49Siciixifpay4=', 'Content-Length': '0'} 2023-09-22T22:05:40.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000569d1b39ee59ec34-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.514 DEBUG:boto:path=/ffzdqu-63/obj44 2023-09-22T22:05:40.515 DEBUG:boto:auth_path=/ffzdqu-63/obj44 2023-09-22T22:05:40.515 DEBUG:boto:Method: GET 2023-09-22T22:05:40.515 DEBUG:boto:Path: /ffzdqu-63/obj44 2023-09-22T22:05:40.515 DEBUG:boto:Data: 2023-09-22T22:05:40.515 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.515 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.515 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.515 DEBUG:boto:Params: {} 2023-09-22T22:05:40.515 DEBUG:boto:Token: None 2023-09-22T22:05:40.516 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj44 2023-09-22T22:05:40.516 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7C8xyDVQwG31/49Siciixifpay4= 2023-09-22T22:05:40.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7C8xyDVQwG31/49Siciixifpay4=', 'Content-Length': '0'} 2023-09-22T22:05:40.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f9b984cc1ebdc84-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.519 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.520 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.520 DEBUG:boto:path=/ffzdqu-63/?prefix=obj44 2023-09-22T22:05:40.520 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj44 2023-09-22T22:05:40.520 DEBUG:boto:Method: GET 2023-09-22T22:05:40.520 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj44 2023-09-22T22:05:40.520 DEBUG:boto:Data: 2023-09-22T22:05:40.520 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.520 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.520 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.520 DEBUG:boto:Params: {} 2023-09-22T22:05:40.521 DEBUG:boto:Token: None 2023-09-22T22:05:40.521 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.521 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006780df58dd8c8b7d-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.526 DEBUG:boto:b'ffzdqu-63obj441000falseobj442023-09-22T22:05:13.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:40.526 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.527 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.527 DEBUG:boto:path=/ffzdqu-63/?prefix=obj44 2023-09-22T22:05:40.527 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj44 2023-09-22T22:05:40.527 DEBUG:boto:Method: GET 2023-09-22T22:05:40.527 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj44 2023-09-22T22:05:40.527 DEBUG:boto:Data: 2023-09-22T22:05:40.527 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.527 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.527 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.527 DEBUG:boto:Params: {} 2023-09-22T22:05:40.528 DEBUG:boto:Token: None 2023-09-22T22:05:40.528 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.528 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000798a5a6fee473949-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.537 DEBUG:boto:b'ffzdqu-63obj441000falseobj442023-09-22T22:05:13.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:40.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-22T22:05:40.538 DEBUG:boto:path=/ffzdqu-63/obj46 2023-09-22T22:05:40.538 DEBUG:boto:auth_path=/ffzdqu-63/obj46 2023-09-22T22:05:40.538 DEBUG:boto:path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.538 DEBUG:boto:auth_path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.538 DEBUG:boto:Method: GET 2023-09-22T22:05:40.538 DEBUG:boto:Path: /ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.539 DEBUG:boto:Data: 2023-09-22T22:05:40.539 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.539 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.539 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.539 DEBUG:boto:Params: {} 2023-09-22T22:05:40.539 DEBUG:boto:Token: None 2023-09-22T22:05:40.539 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.539 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sU1gAwvZdmrCJZ9hGDFtZYU1UEM= 2023-09-22T22:05:40.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sU1gAwvZdmrCJZ9hGDFtZYU1UEM=', 'Content-Length': '0'} 2023-09-22T22:05:40.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7126b2c582d8956-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.544 DEBUG:boto:path=/ffzdqu-63/obj46 2023-09-22T22:05:40.544 DEBUG:boto:auth_path=/ffzdqu-63/obj46 2023-09-22T22:05:40.544 DEBUG:boto:path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.544 DEBUG:boto:auth_path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.544 DEBUG:boto:Method: GET 2023-09-22T22:05:40.544 DEBUG:boto:Path: /ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.544 DEBUG:boto:Data: 2023-09-22T22:05:40.544 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.544 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.545 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.545 DEBUG:boto:Params: {} 2023-09-22T22:05:40.545 DEBUG:boto:Token: None 2023-09-22T22:05:40.545 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.545 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sU1gAwvZdmrCJZ9hGDFtZYU1UEM= 2023-09-22T22:05:40.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sU1gAwvZdmrCJZ9hGDFtZYU1UEM=', 'Content-Length': '0'} 2023-09-22T22:05:40.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e2fd840357ff1f2-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.549 DEBUG:boto:path=/ffzdqu-63/obj46 2023-09-22T22:05:40.549 DEBUG:boto:auth_path=/ffzdqu-63/obj46 2023-09-22T22:05:40.549 DEBUG:boto:path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.549 DEBUG:boto:auth_path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.549 DEBUG:boto:Method: HEAD 2023-09-22T22:05:40.549 DEBUG:boto:Path: /ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.549 DEBUG:boto:Data: 2023-09-22T22:05:40.550 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.550 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.550 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.550 DEBUG:boto:Params: {} 2023-09-22T22:05:40.550 DEBUG:boto:Token: None 2023-09-22T22:05:40.550 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.550 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rCWjQak7U9auKqBxhpzCkYnuc4o= 2023-09-22T22:05:40.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rCWjQak7U9auKqBxhpzCkYnuc4o=', 'Content-Length': '0'} 2023-09-22T22:05:40.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014cbf5d661c22a7c-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.553 DEBUG:boto:path=/ffzdqu-63/obj46 2023-09-22T22:05:40.553 DEBUG:boto:auth_path=/ffzdqu-63/obj46 2023-09-22T22:05:40.553 DEBUG:boto:path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.554 DEBUG:boto:auth_path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.554 DEBUG:boto:Method: HEAD 2023-09-22T22:05:40.554 DEBUG:boto:Path: /ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.554 DEBUG:boto:Data: 2023-09-22T22:05:40.554 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.554 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.554 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.554 DEBUG:boto:Params: {} 2023-09-22T22:05:40.554 DEBUG:boto:Token: None 2023-09-22T22:05:40.555 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.555 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rCWjQak7U9auKqBxhpzCkYnuc4o= 2023-09-22T22:05:40.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rCWjQak7U9auKqBxhpzCkYnuc4o=', 'Content-Length': '0'} 2023-09-22T22:05:40.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000087f9dcbc2a309b0-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-22T22:05:40.558 DEBUG:boto:path=/ffzdqu-63/obj46 2023-09-22T22:05:40.558 DEBUG:boto:auth_path=/ffzdqu-63/obj46 2023-09-22T22:05:40.558 DEBUG:boto:path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.558 DEBUG:boto:auth_path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.558 DEBUG:boto:Method: GET 2023-09-22T22:05:40.558 DEBUG:boto:Path: /ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.559 DEBUG:boto:Data: 2023-09-22T22:05:40.559 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.559 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.559 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.559 DEBUG:boto:Params: {} 2023-09-22T22:05:40.559 DEBUG:boto:Token: None 2023-09-22T22:05:40.559 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.559 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sU1gAwvZdmrCJZ9hGDFtZYU1UEM= 2023-09-22T22:05:40.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sU1gAwvZdmrCJZ9hGDFtZYU1UEM=', 'Content-Length': '0'} 2023-09-22T22:05:40.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd5b6f0e06461a0f-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.610 DEBUG:boto:path=/ffzdqu-63/obj46 2023-09-22T22:05:40.611 DEBUG:boto:auth_path=/ffzdqu-63/obj46 2023-09-22T22:05:40.611 DEBUG:boto:path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.611 DEBUG:boto:auth_path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.611 DEBUG:boto:Method: GET 2023-09-22T22:05:40.611 DEBUG:boto:Path: /ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.611 DEBUG:boto:Data: 2023-09-22T22:05:40.611 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.611 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.612 DEBUG:boto:Params: {} 2023-09-22T22:05:40.612 DEBUG:boto:Token: None 2023-09-22T22:05:40.612 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj46?versionId=null 2023-09-22T22:05:40.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:sU1gAwvZdmrCJZ9hGDFtZYU1UEM= 2023-09-22T22:05:40.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sU1gAwvZdmrCJZ9hGDFtZYU1UEM=', 'Content-Length': '0'} 2023-09-22T22:05:40.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029c9bc8dcfcd978f-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.615 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.616 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.616 DEBUG:boto:path=/ffzdqu-63/?prefix=obj46 2023-09-22T22:05:40.616 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj46 2023-09-22T22:05:40.616 DEBUG:boto:Method: GET 2023-09-22T22:05:40.616 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj46 2023-09-22T22:05:40.616 DEBUG:boto:Data: 2023-09-22T22:05:40.616 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.616 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.616 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.616 DEBUG:boto:Params: {} 2023-09-22T22:05:40.617 DEBUG:boto:Token: None 2023-09-22T22:05:40.617 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.617 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9dc419cef4220cf-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.622 DEBUG:boto:b'ffzdqu-63obj461000falseobj462023-09-22T22:05:14.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:40.622 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.623 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.623 DEBUG:boto:path=/ffzdqu-63/?prefix=obj46 2023-09-22T22:05:40.623 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj46 2023-09-22T22:05:40.623 DEBUG:boto:Method: GET 2023-09-22T22:05:40.623 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj46 2023-09-22T22:05:40.623 DEBUG:boto:Data: 2023-09-22T22:05:40.623 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.623 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.623 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.623 DEBUG:boto:Params: {} 2023-09-22T22:05:40.624 DEBUG:boto:Token: None 2023-09-22T22:05:40.624 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.624 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003424316997bffdc6-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.634 DEBUG:boto:b'ffzdqu-63obj461000falseobj462023-09-22T22:05:14.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:40.634 DEBUG:boto:path=/ffzdqu-63/obj46 2023-09-22T22:05:40.634 DEBUG:boto:auth_path=/ffzdqu-63/obj46 2023-09-22T22:05:40.634 DEBUG:boto:Method: HEAD 2023-09-22T22:05:40.635 DEBUG:boto:Path: /ffzdqu-63/obj46 2023-09-22T22:05:40.635 DEBUG:boto:Data: 2023-09-22T22:05:40.635 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.635 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.635 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.635 DEBUG:boto:Params: {} 2023-09-22T22:05:40.635 DEBUG:boto:Token: None 2023-09-22T22:05:40.635 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj46 2023-09-22T22:05:40.635 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zyJvmpDY8irsZn3g9bhDv+qYLJQ= 2023-09-22T22:05:40.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zyJvmpDY8irsZn3g9bhDv+qYLJQ=', 'Content-Length': '0'} 2023-09-22T22:05:40.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c18221e987028d7-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.638 DEBUG:boto:path=/ffzdqu-63/obj46 2023-09-22T22:05:40.638 DEBUG:boto:auth_path=/ffzdqu-63/obj46 2023-09-22T22:05:40.639 DEBUG:boto:Method: HEAD 2023-09-22T22:05:40.639 DEBUG:boto:Path: /ffzdqu-63/obj46 2023-09-22T22:05:40.639 DEBUG:boto:Data: 2023-09-22T22:05:40.639 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.639 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.639 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.639 DEBUG:boto:Params: {} 2023-09-22T22:05:40.639 DEBUG:boto:Token: None 2023-09-22T22:05:40.639 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj46 2023-09-22T22:05:40.640 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zyJvmpDY8irsZn3g9bhDv+qYLJQ= 2023-09-22T22:05:40.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zyJvmpDY8irsZn3g9bhDv+qYLJQ=', 'Content-Length': '0'} 2023-09-22T22:05:40.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000963026ba8dbf0701-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-22T22:05:40.642 DEBUG:boto:path=/ffzdqu-63/obj46 2023-09-22T22:05:40.643 DEBUG:boto:auth_path=/ffzdqu-63/obj46 2023-09-22T22:05:40.643 DEBUG:boto:Method: GET 2023-09-22T22:05:40.643 DEBUG:boto:Path: /ffzdqu-63/obj46 2023-09-22T22:05:40.643 DEBUG:boto:Data: 2023-09-22T22:05:40.643 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.643 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.643 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.643 DEBUG:boto:Params: {} 2023-09-22T22:05:40.643 DEBUG:boto:Token: None 2023-09-22T22:05:40.643 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj46 2023-09-22T22:05:40.644 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VOD2KbsFlyTKPzaXrDb8UvqNDI8= 2023-09-22T22:05:40.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VOD2KbsFlyTKPzaXrDb8UvqNDI8=', 'Content-Length': '0'} 2023-09-22T22:05:40.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c452f974b5a6130a-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.647 DEBUG:boto:path=/ffzdqu-63/obj46 2023-09-22T22:05:40.647 DEBUG:boto:auth_path=/ffzdqu-63/obj46 2023-09-22T22:05:40.647 DEBUG:boto:Method: GET 2023-09-22T22:05:40.647 DEBUG:boto:Path: /ffzdqu-63/obj46 2023-09-22T22:05:40.647 DEBUG:boto:Data: 2023-09-22T22:05:40.647 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.647 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.647 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.647 DEBUG:boto:Params: {} 2023-09-22T22:05:40.648 DEBUG:boto:Token: None 2023-09-22T22:05:40.648 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj46 2023-09-22T22:05:40.648 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VOD2KbsFlyTKPzaXrDb8UvqNDI8= 2023-09-22T22:05:40.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VOD2KbsFlyTKPzaXrDb8UvqNDI8=', 'Content-Length': '0'} 2023-09-22T22:05:40.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fd289c8cf5f0793-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.651 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.651 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.651 DEBUG:boto:path=/ffzdqu-63/?prefix=obj46 2023-09-22T22:05:40.651 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj46 2023-09-22T22:05:40.652 DEBUG:boto:Method: GET 2023-09-22T22:05:40.652 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj46 2023-09-22T22:05:40.652 DEBUG:boto:Data: 2023-09-22T22:05:40.652 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.652 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.652 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.652 DEBUG:boto:Params: {} 2023-09-22T22:05:40.652 DEBUG:boto:Token: None 2023-09-22T22:05:40.652 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.653 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acb4f2a08479023c-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.698 DEBUG:boto:b'ffzdqu-63obj461000falseobj462023-09-22T22:05:14.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:40.699 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.699 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.699 DEBUG:boto:path=/ffzdqu-63/?prefix=obj46 2023-09-22T22:05:40.699 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj46 2023-09-22T22:05:40.699 DEBUG:boto:Method: GET 2023-09-22T22:05:40.699 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj46 2023-09-22T22:05:40.699 DEBUG:boto:Data: 2023-09-22T22:05:40.700 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.700 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.700 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.700 DEBUG:boto:Params: {} 2023-09-22T22:05:40.700 DEBUG:boto:Token: None 2023-09-22T22:05:40.700 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.700 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c01980ba945dee9-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.710 DEBUG:boto:b'ffzdqu-63obj461000falseobj462023-09-22T22:05:14.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:40.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T22:05:40.711 DEBUG:boto:path=/ffzdqu-63/obj51 2023-09-22T22:05:40.711 DEBUG:boto:auth_path=/ffzdqu-63/obj51 2023-09-22T22:05:40.711 DEBUG:boto:path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.711 DEBUG:boto:auth_path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.711 DEBUG:boto:Method: GET 2023-09-22T22:05:40.711 DEBUG:boto:Path: /ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.711 DEBUG:boto:Data: 2023-09-22T22:05:40.711 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.711 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.711 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.712 DEBUG:boto:Params: {} 2023-09-22T22:05:40.712 DEBUG:boto:Token: None 2023-09-22T22:05:40.712 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.712 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DcFBQbb0OaJX99+jaf+cusiuYKU= 2023-09-22T22:05:40.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DcFBQbb0OaJX99+jaf+cusiuYKU=', 'Content-Length': '0'} 2023-09-22T22:05:40.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eefa6e42bad3adfc-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.715 DEBUG:boto:path=/ffzdqu-63/obj51 2023-09-22T22:05:40.715 DEBUG:boto:auth_path=/ffzdqu-63/obj51 2023-09-22T22:05:40.715 DEBUG:boto:path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.715 DEBUG:boto:auth_path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.716 DEBUG:boto:Method: GET 2023-09-22T22:05:40.716 DEBUG:boto:Path: /ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.716 DEBUG:boto:Data: 2023-09-22T22:05:40.716 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.716 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.716 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.716 DEBUG:boto:Params: {} 2023-09-22T22:05:40.716 DEBUG:boto:Token: None 2023-09-22T22:05:40.716 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.717 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DcFBQbb0OaJX99+jaf+cusiuYKU= 2023-09-22T22:05:40.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DcFBQbb0OaJX99+jaf+cusiuYKU=', 'Content-Length': '0'} 2023-09-22T22:05:40.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc8541b3cf277c4e-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.766 DEBUG:boto:path=/ffzdqu-63/obj51 2023-09-22T22:05:40.767 DEBUG:boto:auth_path=/ffzdqu-63/obj51 2023-09-22T22:05:40.767 DEBUG:boto:path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.767 DEBUG:boto:auth_path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.767 DEBUG:boto:Method: HEAD 2023-09-22T22:05:40.767 DEBUG:boto:Path: /ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.767 DEBUG:boto:Data: 2023-09-22T22:05:40.767 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.767 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.767 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.767 DEBUG:boto:Params: {} 2023-09-22T22:05:40.768 DEBUG:boto:Token: None 2023-09-22T22:05:40.768 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.768 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CFG5ela/Bcg3ZsO3/J7LOU8DOE4= 2023-09-22T22:05:40.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CFG5ela/Bcg3ZsO3/J7LOU8DOE4=', 'Content-Length': '0'} 2023-09-22T22:05:40.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070f700ea0cdfbdc1-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.771 DEBUG:boto:path=/ffzdqu-63/obj51 2023-09-22T22:05:40.771 DEBUG:boto:auth_path=/ffzdqu-63/obj51 2023-09-22T22:05:40.772 DEBUG:boto:path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.772 DEBUG:boto:auth_path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.772 DEBUG:boto:Method: HEAD 2023-09-22T22:05:40.772 DEBUG:boto:Path: /ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.772 DEBUG:boto:Data: 2023-09-22T22:05:40.772 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.772 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.772 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.772 DEBUG:boto:Params: {} 2023-09-22T22:05:40.772 DEBUG:boto:Token: None 2023-09-22T22:05:40.773 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.773 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CFG5ela/Bcg3ZsO3/J7LOU8DOE4= 2023-09-22T22:05:40.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CFG5ela/Bcg3ZsO3/J7LOU8DOE4=', 'Content-Length': '0'} 2023-09-22T22:05:40.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ceee6ec1d3c3261-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T22:05:40.776 DEBUG:boto:path=/ffzdqu-63/obj51 2023-09-22T22:05:40.776 DEBUG:boto:auth_path=/ffzdqu-63/obj51 2023-09-22T22:05:40.776 DEBUG:boto:path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.777 DEBUG:boto:auth_path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.777 DEBUG:boto:Method: GET 2023-09-22T22:05:40.777 DEBUG:boto:Path: /ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.777 DEBUG:boto:Data: 2023-09-22T22:05:40.777 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.777 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.777 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.777 DEBUG:boto:Params: {} 2023-09-22T22:05:40.777 DEBUG:boto:Token: None 2023-09-22T22:05:40.777 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.778 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DcFBQbb0OaJX99+jaf+cusiuYKU= 2023-09-22T22:05:40.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DcFBQbb0OaJX99+jaf+cusiuYKU=', 'Content-Length': '0'} 2023-09-22T22:05:40.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007df33980debd6bb9-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.781 DEBUG:boto:path=/ffzdqu-63/obj51 2023-09-22T22:05:40.781 DEBUG:boto:auth_path=/ffzdqu-63/obj51 2023-09-22T22:05:40.781 DEBUG:boto:path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.781 DEBUG:boto:auth_path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.781 DEBUG:boto:Method: GET 2023-09-22T22:05:40.782 DEBUG:boto:Path: /ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.782 DEBUG:boto:Data: 2023-09-22T22:05:40.782 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.782 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.782 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.782 DEBUG:boto:Params: {} 2023-09-22T22:05:40.782 DEBUG:boto:Token: None 2023-09-22T22:05:40.782 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj51?versionId=null 2023-09-22T22:05:40.782 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:DcFBQbb0OaJX99+jaf+cusiuYKU= 2023-09-22T22:05:40.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DcFBQbb0OaJX99+jaf+cusiuYKU=', 'Content-Length': '0'} 2023-09-22T22:05:40.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045e51e66f2cacd10-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.786 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.786 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.786 DEBUG:boto:path=/ffzdqu-63/?prefix=obj51 2023-09-22T22:05:40.786 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj51 2023-09-22T22:05:40.787 DEBUG:boto:Method: GET 2023-09-22T22:05:40.787 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj51 2023-09-22T22:05:40.787 DEBUG:boto:Data: 2023-09-22T22:05:40.787 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.787 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.787 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.787 DEBUG:boto:Params: {} 2023-09-22T22:05:40.787 DEBUG:boto:Token: None 2023-09-22T22:05:40.788 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.788 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000834516f94cbc612a-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.793 DEBUG:boto:b'ffzdqu-63obj511000falseobj512023-09-22T22:05:14.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:40.793 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.793 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.793 DEBUG:boto:path=/ffzdqu-63/?prefix=obj51 2023-09-22T22:05:40.793 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj51 2023-09-22T22:05:40.793 DEBUG:boto:Method: GET 2023-09-22T22:05:40.793 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj51 2023-09-22T22:05:40.794 DEBUG:boto:Data: 2023-09-22T22:05:40.794 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.794 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.794 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.794 DEBUG:boto:Params: {} 2023-09-22T22:05:40.794 DEBUG:boto:Token: None 2023-09-22T22:05:40.794 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.794 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002487477748ad3629-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.803 DEBUG:boto:b'ffzdqu-63obj511000falseobj512023-09-22T22:05:14.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:40.804 DEBUG:boto:path=/ffzdqu-63/obj51 2023-09-22T22:05:40.804 DEBUG:boto:auth_path=/ffzdqu-63/obj51 2023-09-22T22:05:40.804 DEBUG:boto:Method: HEAD 2023-09-22T22:05:40.804 DEBUG:boto:Path: /ffzdqu-63/obj51 2023-09-22T22:05:40.804 DEBUG:boto:Data: 2023-09-22T22:05:40.804 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.804 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.805 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.805 DEBUG:boto:Params: {} 2023-09-22T22:05:40.805 DEBUG:boto:Token: None 2023-09-22T22:05:40.805 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj51 2023-09-22T22:05:40.805 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WKOEbnuB8JJ42i6P2knu+nTfxQg= 2023-09-22T22:05:40.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WKOEbnuB8JJ42i6P2knu+nTfxQg=', 'Content-Length': '0'} 2023-09-22T22:05:40.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043cbb3165587527e-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.808 DEBUG:boto:path=/ffzdqu-63/obj51 2023-09-22T22:05:40.808 DEBUG:boto:auth_path=/ffzdqu-63/obj51 2023-09-22T22:05:40.808 DEBUG:boto:Method: HEAD 2023-09-22T22:05:40.808 DEBUG:boto:Path: /ffzdqu-63/obj51 2023-09-22T22:05:40.809 DEBUG:boto:Data: 2023-09-22T22:05:40.809 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.809 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.809 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.809 DEBUG:boto:Params: {} 2023-09-22T22:05:40.809 DEBUG:boto:Token: None 2023-09-22T22:05:40.809 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj51 2023-09-22T22:05:40.809 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WKOEbnuB8JJ42i6P2knu+nTfxQg= 2023-09-22T22:05:40.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WKOEbnuB8JJ42i6P2knu+nTfxQg=', 'Content-Length': '0'} 2023-09-22T22:05:40.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006295efec391ecb95-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-22T22:05:40.813 DEBUG:boto:path=/ffzdqu-63/obj51 2023-09-22T22:05:40.813 DEBUG:boto:auth_path=/ffzdqu-63/obj51 2023-09-22T22:05:40.813 DEBUG:boto:Method: GET 2023-09-22T22:05:40.813 DEBUG:boto:Path: /ffzdqu-63/obj51 2023-09-22T22:05:40.813 DEBUG:boto:Data: 2023-09-22T22:05:40.814 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.814 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.814 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.814 DEBUG:boto:Params: {} 2023-09-22T22:05:40.814 DEBUG:boto:Token: None 2023-09-22T22:05:40.814 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj51 2023-09-22T22:05:40.814 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ms/4XGScL/CGCtr5A8umD+CySoA= 2023-09-22T22:05:40.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ms/4XGScL/CGCtr5A8umD+CySoA=', 'Content-Length': '0'} 2023-09-22T22:05:40.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000642e74f8bb4aa855-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.862 DEBUG:boto:path=/ffzdqu-63/obj51 2023-09-22T22:05:40.863 DEBUG:boto:auth_path=/ffzdqu-63/obj51 2023-09-22T22:05:40.863 DEBUG:boto:Method: GET 2023-09-22T22:05:40.863 DEBUG:boto:Path: /ffzdqu-63/obj51 2023-09-22T22:05:40.863 DEBUG:boto:Data: 2023-09-22T22:05:40.863 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.863 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.863 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.864 DEBUG:boto:Params: {} 2023-09-22T22:05:40.864 DEBUG:boto:Token: None 2023-09-22T22:05:40.864 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj51 2023-09-22T22:05:40.864 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ms/4XGScL/CGCtr5A8umD+CySoA= 2023-09-22T22:05:40.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ms/4XGScL/CGCtr5A8umD+CySoA=', 'Content-Length': '0'} 2023-09-22T22:05:40.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008985ba2faef078d5-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.867 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.867 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.868 DEBUG:boto:path=/ffzdqu-63/?prefix=obj51 2023-09-22T22:05:40.868 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj51 2023-09-22T22:05:40.868 DEBUG:boto:Method: GET 2023-09-22T22:05:40.868 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj51 2023-09-22T22:05:40.868 DEBUG:boto:Data: 2023-09-22T22:05:40.868 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.869 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.869 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.869 DEBUG:boto:Params: {} 2023-09-22T22:05:40.869 DEBUG:boto:Token: None 2023-09-22T22:05:40.869 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.869 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ab3b155ebb20af6-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.875 DEBUG:boto:b'ffzdqu-63obj511000falseobj512023-09-22T22:05:14.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:40.875 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.875 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.875 DEBUG:boto:path=/ffzdqu-63/?prefix=obj51 2023-09-22T22:05:40.876 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj51 2023-09-22T22:05:40.876 DEBUG:boto:Method: GET 2023-09-22T22:05:40.876 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj51 2023-09-22T22:05:40.876 DEBUG:boto:Data: 2023-09-22T22:05:40.876 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.876 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.876 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.876 DEBUG:boto:Params: {} 2023-09-22T22:05:40.877 DEBUG:boto:Token: None 2023-09-22T22:05:40.877 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.877 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd6872bfb2ec3af9-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.886 DEBUG:boto:b'ffzdqu-63obj511000falseobj512023-09-22T22:05:14.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:40.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T22:05:40.887 DEBUG:boto:path=/ffzdqu-63/obj55 2023-09-22T22:05:40.887 DEBUG:boto:auth_path=/ffzdqu-63/obj55 2023-09-22T22:05:40.887 DEBUG:boto:path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.887 DEBUG:boto:auth_path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.887 DEBUG:boto:Method: GET 2023-09-22T22:05:40.887 DEBUG:boto:Path: /ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.887 DEBUG:boto:Data: 2023-09-22T22:05:40.888 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.888 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.888 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.888 DEBUG:boto:Params: {} 2023-09-22T22:05:40.888 DEBUG:boto:Token: None 2023-09-22T22:05:40.888 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.889 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3ptq3BCD3fgbA9BKIyI3LINdX3c= 2023-09-22T22:05:40.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3ptq3BCD3fgbA9BKIyI3LINdX3c=', 'Content-Length': '0'} 2023-09-22T22:05:40.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1cf258526174f24-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.892 DEBUG:boto:path=/ffzdqu-63/obj55 2023-09-22T22:05:40.892 DEBUG:boto:auth_path=/ffzdqu-63/obj55 2023-09-22T22:05:40.892 DEBUG:boto:path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.892 DEBUG:boto:auth_path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.893 DEBUG:boto:Method: GET 2023-09-22T22:05:40.893 DEBUG:boto:Path: /ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.893 DEBUG:boto:Data: 2023-09-22T22:05:40.893 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.893 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.893 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.893 DEBUG:boto:Params: {} 2023-09-22T22:05:40.893 DEBUG:boto:Token: None 2023-09-22T22:05:40.894 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.894 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3ptq3BCD3fgbA9BKIyI3LINdX3c= 2023-09-22T22:05:40.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3ptq3BCD3fgbA9BKIyI3LINdX3c=', 'Content-Length': '0'} 2023-09-22T22:05:40.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067b6ea8a14e61c84-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.897 DEBUG:boto:path=/ffzdqu-63/obj55 2023-09-22T22:05:40.898 DEBUG:boto:auth_path=/ffzdqu-63/obj55 2023-09-22T22:05:40.898 DEBUG:boto:path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.898 DEBUG:boto:auth_path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.898 DEBUG:boto:Method: HEAD 2023-09-22T22:05:40.898 DEBUG:boto:Path: /ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.898 DEBUG:boto:Data: 2023-09-22T22:05:40.898 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.898 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.899 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.899 DEBUG:boto:Params: {} 2023-09-22T22:05:40.899 DEBUG:boto:Token: None 2023-09-22T22:05:40.899 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.899 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eRBy4rjXgmJYBbmSRVHNtZHjmDQ= 2023-09-22T22:05:40.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eRBy4rjXgmJYBbmSRVHNtZHjmDQ=', 'Content-Length': '0'} 2023-09-22T22:05:40.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f5ce8751bd1c179-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.903 DEBUG:boto:path=/ffzdqu-63/obj55 2023-09-22T22:05:40.903 DEBUG:boto:auth_path=/ffzdqu-63/obj55 2023-09-22T22:05:40.903 DEBUG:boto:path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.903 DEBUG:boto:auth_path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.903 DEBUG:boto:Method: HEAD 2023-09-22T22:05:40.904 DEBUG:boto:Path: /ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.904 DEBUG:boto:Data: 2023-09-22T22:05:40.904 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.904 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.904 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.904 DEBUG:boto:Params: {} 2023-09-22T22:05:40.904 DEBUG:boto:Token: None 2023-09-22T22:05:40.904 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.905 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eRBy4rjXgmJYBbmSRVHNtZHjmDQ= 2023-09-22T22:05:40.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eRBy4rjXgmJYBbmSRVHNtZHjmDQ=', 'Content-Length': '0'} 2023-09-22T22:05:40.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f69bb510f6bf077-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T22:05:40.909 DEBUG:boto:path=/ffzdqu-63/obj55 2023-09-22T22:05:40.909 DEBUG:boto:auth_path=/ffzdqu-63/obj55 2023-09-22T22:05:40.909 DEBUG:boto:path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.909 DEBUG:boto:auth_path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.909 DEBUG:boto:Method: GET 2023-09-22T22:05:40.910 DEBUG:boto:Path: /ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.910 DEBUG:boto:Data: 2023-09-22T22:05:40.910 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.910 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.910 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.910 DEBUG:boto:Params: {} 2023-09-22T22:05:40.910 DEBUG:boto:Token: None 2023-09-22T22:05:40.911 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.911 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3ptq3BCD3fgbA9BKIyI3LINdX3c= 2023-09-22T22:05:40.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3ptq3BCD3fgbA9BKIyI3LINdX3c=', 'Content-Length': '0'} 2023-09-22T22:05:40.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afb572b9a42ad78a-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.958 DEBUG:boto:path=/ffzdqu-63/obj55 2023-09-22T22:05:40.959 DEBUG:boto:auth_path=/ffzdqu-63/obj55 2023-09-22T22:05:40.959 DEBUG:boto:path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.959 DEBUG:boto:auth_path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.959 DEBUG:boto:Method: GET 2023-09-22T22:05:40.959 DEBUG:boto:Path: /ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.959 DEBUG:boto:Data: 2023-09-22T22:05:40.959 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.959 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.959 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.960 DEBUG:boto:Params: {} 2023-09-22T22:05:40.960 DEBUG:boto:Token: None 2023-09-22T22:05:40.960 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/obj55?versionId=null 2023-09-22T22:05:40.960 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3ptq3BCD3fgbA9BKIyI3LINdX3c= 2023-09-22T22:05:40.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3ptq3BCD3fgbA9BKIyI3LINdX3c=', 'Content-Length': '0'} 2023-09-22T22:05:40.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8762b2a4a430ac7-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.963 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.963 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.964 DEBUG:boto:path=/ffzdqu-63/?prefix=obj55 2023-09-22T22:05:40.964 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj55 2023-09-22T22:05:40.964 DEBUG:boto:Method: GET 2023-09-22T22:05:40.964 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj55 2023-09-22T22:05:40.964 DEBUG:boto:Data: 2023-09-22T22:05:40.964 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.964 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:40.964 DEBUG:boto:Port: 8001 2023-09-22T22:05:40.965 DEBUG:boto:Params: {} 2023-09-22T22:05:40.965 DEBUG:boto:Token: None 2023-09-22T22:05:40.965 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.965 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e8913117861d887-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:40.971 DEBUG:boto:b'ffzdqu-63obj551000falseobj552023-09-22T22:05:13.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:40.971 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:40.972 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:40.972 DEBUG:boto:path=/ffzdqu-63/?prefix=obj55 2023-09-22T22:05:40.972 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj55 2023-09-22T22:05:40.972 DEBUG:boto:Method: GET 2023-09-22T22:05:40.973 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj55 2023-09-22T22:05:40.973 DEBUG:boto:Data: 2023-09-22T22:05:40.973 DEBUG:boto:Headers: {} 2023-09-22T22:05:40.973 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:40.974 DEBUG:boto:Port: 8000 2023-09-22T22:05:40.974 DEBUG:boto:Params: {} 2023-09-22T22:05:40.974 DEBUG:boto:Token: None 2023-09-22T22:05:40.974 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:40 GMT /ffzdqu-63/ 2023-09-22T22:05:40.975 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:05:40.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:05:40.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c189833f4ff5bf8-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.026 DEBUG:boto:b'ffzdqu-63obj551000falseobj552023-09-22T22:05:13.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:41.027 DEBUG:boto:path=/ffzdqu-63/obj55 2023-09-22T22:05:41.027 DEBUG:boto:auth_path=/ffzdqu-63/obj55 2023-09-22T22:05:41.027 DEBUG:boto:Method: HEAD 2023-09-22T22:05:41.027 DEBUG:boto:Path: /ffzdqu-63/obj55 2023-09-22T22:05:41.027 DEBUG:boto:Data: 2023-09-22T22:05:41.027 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.027 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.027 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.028 DEBUG:boto:Params: {} 2023-09-22T22:05:41.028 DEBUG:boto:Token: None 2023-09-22T22:05:41.028 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj55 2023-09-22T22:05:41.028 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:p1yxpdoeBeMXiDXoFgdM1QisM9E= 2023-09-22T22:05:41.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:p1yxpdoeBeMXiDXoFgdM1QisM9E=', 'Content-Length': '0'} 2023-09-22T22:05:41.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042d25124f0526d2f-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.031 DEBUG:boto:path=/ffzdqu-63/obj55 2023-09-22T22:05:41.031 DEBUG:boto:auth_path=/ffzdqu-63/obj55 2023-09-22T22:05:41.032 DEBUG:boto:Method: HEAD 2023-09-22T22:05:41.032 DEBUG:boto:Path: /ffzdqu-63/obj55 2023-09-22T22:05:41.032 DEBUG:boto:Data: 2023-09-22T22:05:41.032 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.032 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.032 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.032 DEBUG:boto:Params: {} 2023-09-22T22:05:41.032 DEBUG:boto:Token: None 2023-09-22T22:05:41.033 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj55 2023-09-22T22:05:41.033 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:p1yxpdoeBeMXiDXoFgdM1QisM9E= 2023-09-22T22:05:41.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:p1yxpdoeBeMXiDXoFgdM1QisM9E=', 'Content-Length': '0'} 2023-09-22T22:05:41.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092f7dde3d2343233-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T22:05:41.037 DEBUG:boto:path=/ffzdqu-63/obj55 2023-09-22T22:05:41.037 DEBUG:boto:auth_path=/ffzdqu-63/obj55 2023-09-22T22:05:41.037 DEBUG:boto:Method: GET 2023-09-22T22:05:41.037 DEBUG:boto:Path: /ffzdqu-63/obj55 2023-09-22T22:05:41.037 DEBUG:boto:Data: 2023-09-22T22:05:41.038 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.038 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.038 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.038 DEBUG:boto:Params: {} 2023-09-22T22:05:41.038 DEBUG:boto:Token: None 2023-09-22T22:05:41.038 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj55 2023-09-22T22:05:41.038 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:u7VlXbGC0TAhSa9bnJSCMsX68m8= 2023-09-22T22:05:41.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:u7VlXbGC0TAhSa9bnJSCMsX68m8=', 'Content-Length': '0'} 2023-09-22T22:05:41.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000611f96be3d4678c9-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.042 DEBUG:boto:path=/ffzdqu-63/obj55 2023-09-22T22:05:41.042 DEBUG:boto:auth_path=/ffzdqu-63/obj55 2023-09-22T22:05:41.042 DEBUG:boto:Method: GET 2023-09-22T22:05:41.042 DEBUG:boto:Path: /ffzdqu-63/obj55 2023-09-22T22:05:41.042 DEBUG:boto:Data: 2023-09-22T22:05:41.042 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.042 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.043 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.043 DEBUG:boto:Params: {} 2023-09-22T22:05:41.043 DEBUG:boto:Token: None 2023-09-22T22:05:41.043 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj55 2023-09-22T22:05:41.043 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:u7VlXbGC0TAhSa9bnJSCMsX68m8= 2023-09-22T22:05:41.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:u7VlXbGC0TAhSa9bnJSCMsX68m8=', 'Content-Length': '0'} 2023-09-22T22:05:41.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be109c05f0a6fbe1-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.046 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:41.047 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:41.047 DEBUG:boto:path=/ffzdqu-63/?prefix=obj55 2023-09-22T22:05:41.047 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj55 2023-09-22T22:05:41.047 DEBUG:boto:Method: GET 2023-09-22T22:05:41.047 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj55 2023-09-22T22:05:41.047 DEBUG:boto:Data: 2023-09-22T22:05:41.047 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.047 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.047 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.048 DEBUG:boto:Params: {} 2023-09-22T22:05:41.048 DEBUG:boto:Token: None 2023-09-22T22:05:41.048 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/ 2023-09-22T22:05:41.048 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:05:41.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:05:41.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8a05e608eaa07ac-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.054 DEBUG:boto:b'ffzdqu-63obj551000falseobj552023-09-22T22:05:13.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:41.054 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:41.054 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:41.054 DEBUG:boto:path=/ffzdqu-63/?prefix=obj55 2023-09-22T22:05:41.055 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj55 2023-09-22T22:05:41.055 DEBUG:boto:Method: GET 2023-09-22T22:05:41.055 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj55 2023-09-22T22:05:41.055 DEBUG:boto:Data: 2023-09-22T22:05:41.055 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.055 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.055 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.055 DEBUG:boto:Params: {} 2023-09-22T22:05:41.056 DEBUG:boto:Token: None 2023-09-22T22:05:41.056 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/ 2023-09-22T22:05:41.056 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:05:41.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:05:41.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009eddf353ffaec0c7-00650e0fb5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.066 DEBUG:boto:b'ffzdqu-63obj551000falseobj552023-09-22T22:05:13.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:41.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T22:05:41.067 DEBUG:boto:path=/ffzdqu-63/obj57 2023-09-22T22:05:41.067 DEBUG:boto:auth_path=/ffzdqu-63/obj57 2023-09-22T22:05:41.067 DEBUG:boto:path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.067 DEBUG:boto:auth_path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.067 DEBUG:boto:Method: GET 2023-09-22T22:05:41.067 DEBUG:boto:Path: /ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.068 DEBUG:boto:Data: 2023-09-22T22:05:41.068 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.068 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.068 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.068 DEBUG:boto:Params: {} 2023-09-22T22:05:41.068 DEBUG:boto:Token: None 2023-09-22T22:05:41.068 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.069 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8CsBBhFi5YgAfSzFr/cJdELMHdo= 2023-09-22T22:05:41.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8CsBBhFi5YgAfSzFr/cJdELMHdo=', 'Content-Length': '0'} 2023-09-22T22:05:41.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008255446dceac2161-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.118 DEBUG:boto:path=/ffzdqu-63/obj57 2023-09-22T22:05:41.118 DEBUG:boto:auth_path=/ffzdqu-63/obj57 2023-09-22T22:05:41.119 DEBUG:boto:path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.119 DEBUG:boto:auth_path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.119 DEBUG:boto:Method: GET 2023-09-22T22:05:41.119 DEBUG:boto:Path: /ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.119 DEBUG:boto:Data: 2023-09-22T22:05:41.119 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.119 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.119 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.120 DEBUG:boto:Params: {} 2023-09-22T22:05:41.120 DEBUG:boto:Token: None 2023-09-22T22:05:41.120 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.120 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8CsBBhFi5YgAfSzFr/cJdELMHdo= 2023-09-22T22:05:41.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8CsBBhFi5YgAfSzFr/cJdELMHdo=', 'Content-Length': '0'} 2023-09-22T22:05:41.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000613b7fe38da60e3d-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.123 DEBUG:boto:path=/ffzdqu-63/obj57 2023-09-22T22:05:41.124 DEBUG:boto:auth_path=/ffzdqu-63/obj57 2023-09-22T22:05:41.124 DEBUG:boto:path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.124 DEBUG:boto:auth_path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.124 DEBUG:boto:Method: HEAD 2023-09-22T22:05:41.124 DEBUG:boto:Path: /ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.124 DEBUG:boto:Data: 2023-09-22T22:05:41.124 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.124 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.125 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.125 DEBUG:boto:Params: {} 2023-09-22T22:05:41.125 DEBUG:boto:Token: None 2023-09-22T22:05:41.125 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.125 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fPqGlTXpzWHIs305gl/MLFDk4pE= 2023-09-22T22:05:41.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fPqGlTXpzWHIs305gl/MLFDk4pE=', 'Content-Length': '0'} 2023-09-22T22:05:41.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c4cc9e1f9836d31-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.128 DEBUG:boto:path=/ffzdqu-63/obj57 2023-09-22T22:05:41.128 DEBUG:boto:auth_path=/ffzdqu-63/obj57 2023-09-22T22:05:41.129 DEBUG:boto:path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.129 DEBUG:boto:auth_path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.129 DEBUG:boto:Method: HEAD 2023-09-22T22:05:41.129 DEBUG:boto:Path: /ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.129 DEBUG:boto:Data: 2023-09-22T22:05:41.129 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.129 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.129 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.130 DEBUG:boto:Params: {} 2023-09-22T22:05:41.130 DEBUG:boto:Token: None 2023-09-22T22:05:41.130 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.130 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:fPqGlTXpzWHIs305gl/MLFDk4pE= 2023-09-22T22:05:41.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fPqGlTXpzWHIs305gl/MLFDk4pE=', 'Content-Length': '0'} 2023-09-22T22:05:41.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a53fdde1f3038005-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T22:05:41.133 DEBUG:boto:path=/ffzdqu-63/obj57 2023-09-22T22:05:41.134 DEBUG:boto:auth_path=/ffzdqu-63/obj57 2023-09-22T22:05:41.134 DEBUG:boto:path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.134 DEBUG:boto:auth_path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.134 DEBUG:boto:Method: GET 2023-09-22T22:05:41.134 DEBUG:boto:Path: /ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.134 DEBUG:boto:Data: 2023-09-22T22:05:41.134 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.134 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.134 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.135 DEBUG:boto:Params: {} 2023-09-22T22:05:41.135 DEBUG:boto:Token: None 2023-09-22T22:05:41.135 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.135 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8CsBBhFi5YgAfSzFr/cJdELMHdo= 2023-09-22T22:05:41.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8CsBBhFi5YgAfSzFr/cJdELMHdo=', 'Content-Length': '0'} 2023-09-22T22:05:41.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065c94b3f6db2ef81-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.138 DEBUG:boto:path=/ffzdqu-63/obj57 2023-09-22T22:05:41.139 DEBUG:boto:auth_path=/ffzdqu-63/obj57 2023-09-22T22:05:41.139 DEBUG:boto:path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.139 DEBUG:boto:auth_path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.139 DEBUG:boto:Method: GET 2023-09-22T22:05:41.139 DEBUG:boto:Path: /ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.139 DEBUG:boto:Data: 2023-09-22T22:05:41.139 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.139 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.139 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.140 DEBUG:boto:Params: {} 2023-09-22T22:05:41.140 DEBUG:boto:Token: None 2023-09-22T22:05:41.140 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj57?versionId=null 2023-09-22T22:05:41.140 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8CsBBhFi5YgAfSzFr/cJdELMHdo= 2023-09-22T22:05:41.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8CsBBhFi5YgAfSzFr/cJdELMHdo=', 'Content-Length': '0'} 2023-09-22T22:05:41.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035064f8b0e87d8dd-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.186 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:41.187 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:41.187 DEBUG:boto:path=/ffzdqu-63/?prefix=obj57 2023-09-22T22:05:41.187 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj57 2023-09-22T22:05:41.187 DEBUG:boto:Method: GET 2023-09-22T22:05:41.187 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj57 2023-09-22T22:05:41.187 DEBUG:boto:Data: 2023-09-22T22:05:41.187 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.187 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.187 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.188 DEBUG:boto:Params: {} 2023-09-22T22:05:41.188 DEBUG:boto:Token: None 2023-09-22T22:05:41.188 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/ 2023-09-22T22:05:41.188 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:05:41.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:05:41.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bff49cfe290e6f27-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.193 DEBUG:boto:b'ffzdqu-63obj571000falseobj572023-09-22T22:05:14.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:41.193 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:41.193 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:41.194 DEBUG:boto:path=/ffzdqu-63/?prefix=obj57 2023-09-22T22:05:41.194 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj57 2023-09-22T22:05:41.194 DEBUG:boto:Method: GET 2023-09-22T22:05:41.194 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj57 2023-09-22T22:05:41.194 DEBUG:boto:Data: 2023-09-22T22:05:41.194 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.194 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.194 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.195 DEBUG:boto:Params: {} 2023-09-22T22:05:41.195 DEBUG:boto:Token: None 2023-09-22T22:05:41.195 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/ 2023-09-22T22:05:41.195 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:05:41.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:05:41.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000262169911eae6152-00650e0fb5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.205 DEBUG:boto:b'ffzdqu-63obj571000falseobj572023-09-22T22:05:14.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:41.205 DEBUG:boto:path=/ffzdqu-63/obj57 2023-09-22T22:05:41.205 DEBUG:boto:auth_path=/ffzdqu-63/obj57 2023-09-22T22:05:41.206 DEBUG:boto:Method: HEAD 2023-09-22T22:05:41.206 DEBUG:boto:Path: /ffzdqu-63/obj57 2023-09-22T22:05:41.206 DEBUG:boto:Data: 2023-09-22T22:05:41.206 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.206 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.206 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.206 DEBUG:boto:Params: {} 2023-09-22T22:05:41.206 DEBUG:boto:Token: None 2023-09-22T22:05:41.207 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj57 2023-09-22T22:05:41.207 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uK5xPtcDCpkqD3lfImiha3CLhVg= 2023-09-22T22:05:41.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uK5xPtcDCpkqD3lfImiha3CLhVg=', 'Content-Length': '0'} 2023-09-22T22:05:41.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec7691b086d73334-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.210 DEBUG:boto:path=/ffzdqu-63/obj57 2023-09-22T22:05:41.210 DEBUG:boto:auth_path=/ffzdqu-63/obj57 2023-09-22T22:05:41.210 DEBUG:boto:Method: HEAD 2023-09-22T22:05:41.210 DEBUG:boto:Path: /ffzdqu-63/obj57 2023-09-22T22:05:41.211 DEBUG:boto:Data: 2023-09-22T22:05:41.211 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.211 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.211 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.211 DEBUG:boto:Params: {} 2023-09-22T22:05:41.211 DEBUG:boto:Token: None 2023-09-22T22:05:41.211 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj57 2023-09-22T22:05:41.211 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uK5xPtcDCpkqD3lfImiha3CLhVg= 2023-09-22T22:05:41.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uK5xPtcDCpkqD3lfImiha3CLhVg=', 'Content-Length': '0'} 2023-09-22T22:05:41.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002da948e1533b27d1-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-22T22:05:41.215 DEBUG:boto:path=/ffzdqu-63/obj57 2023-09-22T22:05:41.215 DEBUG:boto:auth_path=/ffzdqu-63/obj57 2023-09-22T22:05:41.215 DEBUG:boto:Method: GET 2023-09-22T22:05:41.215 DEBUG:boto:Path: /ffzdqu-63/obj57 2023-09-22T22:05:41.215 DEBUG:boto:Data: 2023-09-22T22:05:41.215 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.215 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.215 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.216 DEBUG:boto:Params: {} 2023-09-22T22:05:41.216 DEBUG:boto:Token: None 2023-09-22T22:05:41.216 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj57 2023-09-22T22:05:41.216 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xq2IXsMfuiLnRPNx5OYeGCZbTLY= 2023-09-22T22:05:41.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xq2IXsMfuiLnRPNx5OYeGCZbTLY=', 'Content-Length': '0'} 2023-09-22T22:05:41.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ecba7049d573e47-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.266 DEBUG:boto:path=/ffzdqu-63/obj57 2023-09-22T22:05:41.266 DEBUG:boto:auth_path=/ffzdqu-63/obj57 2023-09-22T22:05:41.266 DEBUG:boto:Method: GET 2023-09-22T22:05:41.267 DEBUG:boto:Path: /ffzdqu-63/obj57 2023-09-22T22:05:41.267 DEBUG:boto:Data: 2023-09-22T22:05:41.267 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.267 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.267 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.267 DEBUG:boto:Params: {} 2023-09-22T22:05:41.267 DEBUG:boto:Token: None 2023-09-22T22:05:41.267 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj57 2023-09-22T22:05:41.268 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:xq2IXsMfuiLnRPNx5OYeGCZbTLY= 2023-09-22T22:05:41.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xq2IXsMfuiLnRPNx5OYeGCZbTLY=', 'Content-Length': '0'} 2023-09-22T22:05:41.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d9448956e03e8e4-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.271 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:41.271 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:41.271 DEBUG:boto:path=/ffzdqu-63/?prefix=obj57 2023-09-22T22:05:41.272 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj57 2023-09-22T22:05:41.272 DEBUG:boto:Method: GET 2023-09-22T22:05:41.272 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj57 2023-09-22T22:05:41.272 DEBUG:boto:Data: 2023-09-22T22:05:41.272 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.272 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.272 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.272 DEBUG:boto:Params: {} 2023-09-22T22:05:41.273 DEBUG:boto:Token: None 2023-09-22T22:05:41.273 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/ 2023-09-22T22:05:41.273 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:05:41.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:05:41.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dd883514bca0081-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.278 DEBUG:boto:b'ffzdqu-63obj571000falseobj572023-09-22T22:05:14.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:41.279 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:41.279 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:41.279 DEBUG:boto:path=/ffzdqu-63/?prefix=obj57 2023-09-22T22:05:41.279 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj57 2023-09-22T22:05:41.279 DEBUG:boto:Method: GET 2023-09-22T22:05:41.280 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj57 2023-09-22T22:05:41.280 DEBUG:boto:Data: 2023-09-22T22:05:41.280 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.280 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.280 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.280 DEBUG:boto:Params: {} 2023-09-22T22:05:41.280 DEBUG:boto:Token: None 2023-09-22T22:05:41.280 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/ 2023-09-22T22:05:41.281 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:05:41.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:05:41.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fec6d0a1e23f0c4c-00650e0fb5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.330 DEBUG:boto:b'ffzdqu-63obj571000falseobj572023-09-22T22:05:14.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:41.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-22T22:05:41.331 DEBUG:boto:path=/ffzdqu-63/obj58 2023-09-22T22:05:41.331 DEBUG:boto:auth_path=/ffzdqu-63/obj58 2023-09-22T22:05:41.331 DEBUG:boto:path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.331 DEBUG:boto:auth_path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.331 DEBUG:boto:Method: GET 2023-09-22T22:05:41.332 DEBUG:boto:Path: /ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.332 DEBUG:boto:Data: 2023-09-22T22:05:41.332 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.332 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.332 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.332 DEBUG:boto:Params: {} 2023-09-22T22:05:41.332 DEBUG:boto:Token: None 2023-09-22T22:05:41.332 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.333 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HbElvGukCjTOKLFNPpD6yC9cRwc= 2023-09-22T22:05:41.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HbElvGukCjTOKLFNPpD6yC9cRwc=', 'Content-Length': '0'} 2023-09-22T22:05:41.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e6c2321e13e57bf-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.336 DEBUG:boto:path=/ffzdqu-63/obj58 2023-09-22T22:05:41.336 DEBUG:boto:auth_path=/ffzdqu-63/obj58 2023-09-22T22:05:41.336 DEBUG:boto:path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.336 DEBUG:boto:auth_path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.337 DEBUG:boto:Method: GET 2023-09-22T22:05:41.337 DEBUG:boto:Path: /ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.337 DEBUG:boto:Data: 2023-09-22T22:05:41.337 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.337 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.337 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.337 DEBUG:boto:Params: {} 2023-09-22T22:05:41.337 DEBUG:boto:Token: None 2023-09-22T22:05:41.338 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.338 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HbElvGukCjTOKLFNPpD6yC9cRwc= 2023-09-22T22:05:41.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HbElvGukCjTOKLFNPpD6yC9cRwc=', 'Content-Length': '0'} 2023-09-22T22:05:41.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b716b4e98b3e6a4-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.341 DEBUG:boto:path=/ffzdqu-63/obj58 2023-09-22T22:05:41.341 DEBUG:boto:auth_path=/ffzdqu-63/obj58 2023-09-22T22:05:41.341 DEBUG:boto:path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.342 DEBUG:boto:auth_path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.342 DEBUG:boto:Method: HEAD 2023-09-22T22:05:41.342 DEBUG:boto:Path: /ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.342 DEBUG:boto:Data: 2023-09-22T22:05:41.342 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.342 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.342 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.342 DEBUG:boto:Params: {} 2023-09-22T22:05:41.343 DEBUG:boto:Token: None 2023-09-22T22:05:41.343 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.343 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYqbM+TCxQfw3D9Y04x8aeqNjzE= 2023-09-22T22:05:41.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYqbM+TCxQfw3D9Y04x8aeqNjzE=', 'Content-Length': '0'} 2023-09-22T22:05:41.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009907954b1d9932d6-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.346 DEBUG:boto:path=/ffzdqu-63/obj58 2023-09-22T22:05:41.346 DEBUG:boto:auth_path=/ffzdqu-63/obj58 2023-09-22T22:05:41.346 DEBUG:boto:path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.346 DEBUG:boto:auth_path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.346 DEBUG:boto:Method: HEAD 2023-09-22T22:05:41.347 DEBUG:boto:Path: /ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.347 DEBUG:boto:Data: 2023-09-22T22:05:41.347 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.347 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.347 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.347 DEBUG:boto:Params: {} 2023-09-22T22:05:41.347 DEBUG:boto:Token: None 2023-09-22T22:05:41.347 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.348 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iYqbM+TCxQfw3D9Y04x8aeqNjzE= 2023-09-22T22:05:41.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYqbM+TCxQfw3D9Y04x8aeqNjzE=', 'Content-Length': '0'} 2023-09-22T22:05:41.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000998df9d409e41c54-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-22T22:05:41.351 DEBUG:boto:path=/ffzdqu-63/obj58 2023-09-22T22:05:41.351 DEBUG:boto:auth_path=/ffzdqu-63/obj58 2023-09-22T22:05:41.351 DEBUG:boto:path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.351 DEBUG:boto:auth_path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.351 DEBUG:boto:Method: GET 2023-09-22T22:05:41.351 DEBUG:boto:Path: /ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.352 DEBUG:boto:Data: 2023-09-22T22:05:41.352 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.352 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.352 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.352 DEBUG:boto:Params: {} 2023-09-22T22:05:41.352 DEBUG:boto:Token: None 2023-09-22T22:05:41.352 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.353 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HbElvGukCjTOKLFNPpD6yC9cRwc= 2023-09-22T22:05:41.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HbElvGukCjTOKLFNPpD6yC9cRwc=', 'Content-Length': '0'} 2023-09-22T22:05:41.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072e8047b715e3c4e-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.356 DEBUG:boto:path=/ffzdqu-63/obj58 2023-09-22T22:05:41.356 DEBUG:boto:auth_path=/ffzdqu-63/obj58 2023-09-22T22:05:41.356 DEBUG:boto:path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.356 DEBUG:boto:auth_path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.356 DEBUG:boto:Method: GET 2023-09-22T22:05:41.356 DEBUG:boto:Path: /ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.356 DEBUG:boto:Data: 2023-09-22T22:05:41.357 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.357 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.357 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.357 DEBUG:boto:Params: {} 2023-09-22T22:05:41.357 DEBUG:boto:Token: None 2023-09-22T22:05:41.357 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj58?versionId=null 2023-09-22T22:05:41.357 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:HbElvGukCjTOKLFNPpD6yC9cRwc= 2023-09-22T22:05:41.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HbElvGukCjTOKLFNPpD6yC9cRwc=', 'Content-Length': '0'} 2023-09-22T22:05:41.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4e6446e5f97be4f-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.361 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:41.361 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:41.361 DEBUG:boto:path=/ffzdqu-63/?prefix=obj58 2023-09-22T22:05:41.361 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj58 2023-09-22T22:05:41.361 DEBUG:boto:Method: GET 2023-09-22T22:05:41.361 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj58 2023-09-22T22:05:41.361 DEBUG:boto:Data: 2023-09-22T22:05:41.361 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.362 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.362 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.362 DEBUG:boto:Params: {} 2023-09-22T22:05:41.362 DEBUG:boto:Token: None 2023-09-22T22:05:41.362 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/ 2023-09-22T22:05:41.362 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:05:41.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:05:41.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e74436b4c1be076-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.414 DEBUG:boto:b'ffzdqu-63obj581000falseobj582023-09-22T22:05:15.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:41.415 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:41.415 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:41.415 DEBUG:boto:path=/ffzdqu-63/?prefix=obj58 2023-09-22T22:05:41.415 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj58 2023-09-22T22:05:41.416 DEBUG:boto:Method: GET 2023-09-22T22:05:41.416 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj58 2023-09-22T22:05:41.416 DEBUG:boto:Data: 2023-09-22T22:05:41.416 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.416 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.416 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.416 DEBUG:boto:Params: {} 2023-09-22T22:05:41.416 DEBUG:boto:Token: None 2023-09-22T22:05:41.417 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/ 2023-09-22T22:05:41.417 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:05:41.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:05:41.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcd014ef0edf6a98-00650e0fb5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.428 DEBUG:boto:b'ffzdqu-63obj581000falseobj582023-09-22T22:05:15.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:41.428 DEBUG:boto:path=/ffzdqu-63/obj58 2023-09-22T22:05:41.429 DEBUG:boto:auth_path=/ffzdqu-63/obj58 2023-09-22T22:05:41.429 DEBUG:boto:Method: HEAD 2023-09-22T22:05:41.429 DEBUG:boto:Path: /ffzdqu-63/obj58 2023-09-22T22:05:41.429 DEBUG:boto:Data: 2023-09-22T22:05:41.429 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.429 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.429 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.429 DEBUG:boto:Params: {} 2023-09-22T22:05:41.430 DEBUG:boto:Token: None 2023-09-22T22:05:41.430 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj58 2023-09-22T22:05:41.430 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h0+BDr4RhYNH8kxT8vIN6L410WI= 2023-09-22T22:05:41.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h0+BDr4RhYNH8kxT8vIN6L410WI=', 'Content-Length': '0'} 2023-09-22T22:05:41.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b1baaa9fe56bdc8-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.433 DEBUG:boto:path=/ffzdqu-63/obj58 2023-09-22T22:05:41.433 DEBUG:boto:auth_path=/ffzdqu-63/obj58 2023-09-22T22:05:41.434 DEBUG:boto:Method: HEAD 2023-09-22T22:05:41.434 DEBUG:boto:Path: /ffzdqu-63/obj58 2023-09-22T22:05:41.434 DEBUG:boto:Data: 2023-09-22T22:05:41.434 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.434 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.434 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.434 DEBUG:boto:Params: {} 2023-09-22T22:05:41.434 DEBUG:boto:Token: None 2023-09-22T22:05:41.435 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj58 2023-09-22T22:05:41.435 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h0+BDr4RhYNH8kxT8vIN6L410WI= 2023-09-22T22:05:41.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h0+BDr4RhYNH8kxT8vIN6L410WI=', 'Content-Length': '0'} 2023-09-22T22:05:41.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007536c4fae8a32207-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-22T22:05:41.438 DEBUG:boto:path=/ffzdqu-63/obj58 2023-09-22T22:05:41.438 DEBUG:boto:auth_path=/ffzdqu-63/obj58 2023-09-22T22:05:41.439 DEBUG:boto:Method: GET 2023-09-22T22:05:41.439 DEBUG:boto:Path: /ffzdqu-63/obj58 2023-09-22T22:05:41.439 DEBUG:boto:Data: 2023-09-22T22:05:41.439 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.439 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.439 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.439 DEBUG:boto:Params: {} 2023-09-22T22:05:41.439 DEBUG:boto:Token: None 2023-09-22T22:05:41.440 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj58 2023-09-22T22:05:41.440 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f6ieXfrzUXpJFKJBmm4QnzeNHD8= 2023-09-22T22:05:41.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f6ieXfrzUXpJFKJBmm4QnzeNHD8=', 'Content-Length': '0'} 2023-09-22T22:05:41.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006edf53d85477c00d-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.444 DEBUG:boto:path=/ffzdqu-63/obj58 2023-09-22T22:05:41.444 DEBUG:boto:auth_path=/ffzdqu-63/obj58 2023-09-22T22:05:41.444 DEBUG:boto:Method: GET 2023-09-22T22:05:41.444 DEBUG:boto:Path: /ffzdqu-63/obj58 2023-09-22T22:05:41.444 DEBUG:boto:Data: 2023-09-22T22:05:41.445 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.445 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.445 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.445 DEBUG:boto:Params: {} 2023-09-22T22:05:41.445 DEBUG:boto:Token: None 2023-09-22T22:05:41.445 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj58 2023-09-22T22:05:41.445 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f6ieXfrzUXpJFKJBmm4QnzeNHD8= 2023-09-22T22:05:41.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f6ieXfrzUXpJFKJBmm4QnzeNHD8=', 'Content-Length': '0'} 2023-09-22T22:05:41.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045570e9517a1c3ca-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.490 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:41.491 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:41.491 DEBUG:boto:path=/ffzdqu-63/?prefix=obj58 2023-09-22T22:05:41.491 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj58 2023-09-22T22:05:41.491 DEBUG:boto:Method: GET 2023-09-22T22:05:41.491 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj58 2023-09-22T22:05:41.491 DEBUG:boto:Data: 2023-09-22T22:05:41.491 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.491 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.492 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.492 DEBUG:boto:Params: {} 2023-09-22T22:05:41.492 DEBUG:boto:Token: None 2023-09-22T22:05:41.492 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/ 2023-09-22T22:05:41.492 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:05:41.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:05:41.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000122a84c4823b4295-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.498 DEBUG:boto:b'ffzdqu-63obj581000falseobj582023-09-22T22:05:15.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:41.498 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:41.498 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:41.499 DEBUG:boto:path=/ffzdqu-63/?prefix=obj58 2023-09-22T22:05:41.499 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj58 2023-09-22T22:05:41.499 DEBUG:boto:Method: GET 2023-09-22T22:05:41.499 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj58 2023-09-22T22:05:41.499 DEBUG:boto:Data: 2023-09-22T22:05:41.499 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.499 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.499 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.499 DEBUG:boto:Params: {} 2023-09-22T22:05:41.500 DEBUG:boto:Token: None 2023-09-22T22:05:41.500 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/ 2023-09-22T22:05:41.500 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:05:41.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:05:41.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030d81954336e03ad-00650e0fb5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.510 DEBUG:boto:b'ffzdqu-63obj581000falseobj582023-09-22T22:05:15.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:41.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-22T22:05:41.510 DEBUG:boto:path=/ffzdqu-63/obj62 2023-09-22T22:05:41.510 DEBUG:boto:auth_path=/ffzdqu-63/obj62 2023-09-22T22:05:41.510 DEBUG:boto:path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.511 DEBUG:boto:auth_path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.511 DEBUG:boto:Method: GET 2023-09-22T22:05:41.511 DEBUG:boto:Path: /ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.511 DEBUG:boto:Data: 2023-09-22T22:05:41.511 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.511 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.511 DEBUG:boto:Params: {} 2023-09-22T22:05:41.512 DEBUG:boto:Token: None 2023-09-22T22:05:41.512 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.512 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wbDUsltvuvF9UTqNaGPyZLId0VU= 2023-09-22T22:05:41.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wbDUsltvuvF9UTqNaGPyZLId0VU=', 'Content-Length': '0'} 2023-09-22T22:05:41.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a24e759710dcf652-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.515 DEBUG:boto:path=/ffzdqu-63/obj62 2023-09-22T22:05:41.515 DEBUG:boto:auth_path=/ffzdqu-63/obj62 2023-09-22T22:05:41.516 DEBUG:boto:path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.516 DEBUG:boto:auth_path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.516 DEBUG:boto:Method: GET 2023-09-22T22:05:41.516 DEBUG:boto:Path: /ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.516 DEBUG:boto:Data: 2023-09-22T22:05:41.516 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.516 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.516 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.516 DEBUG:boto:Params: {} 2023-09-22T22:05:41.517 DEBUG:boto:Token: None 2023-09-22T22:05:41.517 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.517 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wbDUsltvuvF9UTqNaGPyZLId0VU= 2023-09-22T22:05:41.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wbDUsltvuvF9UTqNaGPyZLId0VU=', 'Content-Length': '0'} 2023-09-22T22:05:41.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034283780a8078570-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.520 DEBUG:boto:path=/ffzdqu-63/obj62 2023-09-22T22:05:41.520 DEBUG:boto:auth_path=/ffzdqu-63/obj62 2023-09-22T22:05:41.521 DEBUG:boto:path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.521 DEBUG:boto:auth_path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.521 DEBUG:boto:Method: HEAD 2023-09-22T22:05:41.521 DEBUG:boto:Path: /ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.521 DEBUG:boto:Data: 2023-09-22T22:05:41.521 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.521 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.521 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.521 DEBUG:boto:Params: {} 2023-09-22T22:05:41.522 DEBUG:boto:Token: None 2023-09-22T22:05:41.522 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.522 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hSqI/+W2+leER6HP7uWrcJCMklo= 2023-09-22T22:05:41.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hSqI/+W2+leER6HP7uWrcJCMklo=', 'Content-Length': '0'} 2023-09-22T22:05:41.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089f0b8fc0e40c5e0-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.525 DEBUG:boto:path=/ffzdqu-63/obj62 2023-09-22T22:05:41.525 DEBUG:boto:auth_path=/ffzdqu-63/obj62 2023-09-22T22:05:41.525 DEBUG:boto:path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.525 DEBUG:boto:auth_path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.525 DEBUG:boto:Method: HEAD 2023-09-22T22:05:41.525 DEBUG:boto:Path: /ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.525 DEBUG:boto:Data: 2023-09-22T22:05:41.525 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.526 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.526 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.526 DEBUG:boto:Params: {} 2023-09-22T22:05:41.526 DEBUG:boto:Token: None 2023-09-22T22:05:41.526 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.526 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hSqI/+W2+leER6HP7uWrcJCMklo= 2023-09-22T22:05:41.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hSqI/+W2+leER6HP7uWrcJCMklo=', 'Content-Length': '0'} 2023-09-22T22:05:41.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005282fd0ae15b2169-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-22T22:05:41.529 DEBUG:boto:path=/ffzdqu-63/obj62 2023-09-22T22:05:41.529 DEBUG:boto:auth_path=/ffzdqu-63/obj62 2023-09-22T22:05:41.530 DEBUG:boto:path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.530 DEBUG:boto:auth_path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.530 DEBUG:boto:Method: GET 2023-09-22T22:05:41.530 DEBUG:boto:Path: /ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.530 DEBUG:boto:Data: 2023-09-22T22:05:41.530 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.530 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.530 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.530 DEBUG:boto:Params: {} 2023-09-22T22:05:41.531 DEBUG:boto:Token: None 2023-09-22T22:05:41.531 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.531 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wbDUsltvuvF9UTqNaGPyZLId0VU= 2023-09-22T22:05:41.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wbDUsltvuvF9UTqNaGPyZLId0VU=', 'Content-Length': '0'} 2023-09-22T22:05:41.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b01a68f56b5ab94a-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.574 DEBUG:boto:path=/ffzdqu-63/obj62 2023-09-22T22:05:41.574 DEBUG:boto:auth_path=/ffzdqu-63/obj62 2023-09-22T22:05:41.575 DEBUG:boto:path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.575 DEBUG:boto:auth_path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.575 DEBUG:boto:Method: GET 2023-09-22T22:05:41.575 DEBUG:boto:Path: /ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.575 DEBUG:boto:Data: 2023-09-22T22:05:41.575 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.575 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.575 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.575 DEBUG:boto:Params: {} 2023-09-22T22:05:41.576 DEBUG:boto:Token: None 2023-09-22T22:05:41.576 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj62?versionId=null 2023-09-22T22:05:41.576 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wbDUsltvuvF9UTqNaGPyZLId0VU= 2023-09-22T22:05:41.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wbDUsltvuvF9UTqNaGPyZLId0VU=', 'Content-Length': '0'} 2023-09-22T22:05:41.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012bbc9898269f31c-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.580 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:41.580 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:41.580 DEBUG:boto:path=/ffzdqu-63/?prefix=obj62 2023-09-22T22:05:41.580 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj62 2023-09-22T22:05:41.580 DEBUG:boto:Method: GET 2023-09-22T22:05:41.580 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj62 2023-09-22T22:05:41.580 DEBUG:boto:Data: 2023-09-22T22:05:41.580 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.580 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.581 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.581 DEBUG:boto:Params: {} 2023-09-22T22:05:41.581 DEBUG:boto:Token: None 2023-09-22T22:05:41.581 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/ 2023-09-22T22:05:41.581 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:05:41.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:05:41.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dd73cf56469f1c8-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.586 DEBUG:boto:b'ffzdqu-63obj621000falseobj622023-09-22T22:05:15.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:41.586 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:41.587 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:41.587 DEBUG:boto:path=/ffzdqu-63/?prefix=obj62 2023-09-22T22:05:41.587 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj62 2023-09-22T22:05:41.587 DEBUG:boto:Method: GET 2023-09-22T22:05:41.587 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj62 2023-09-22T22:05:41.587 DEBUG:boto:Data: 2023-09-22T22:05:41.587 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.587 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.587 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.588 DEBUG:boto:Params: {} 2023-09-22T22:05:41.588 DEBUG:boto:Token: None 2023-09-22T22:05:41.588 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/ 2023-09-22T22:05:41.588 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:05:41.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:05:41.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060e0bbcb811116f0-00650e0fb5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.598 DEBUG:boto:b'ffzdqu-63obj621000falseobj622023-09-22T22:05:15.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:41.598 DEBUG:boto:path=/ffzdqu-63/obj62 2023-09-22T22:05:41.599 DEBUG:boto:auth_path=/ffzdqu-63/obj62 2023-09-22T22:05:41.599 DEBUG:boto:Method: HEAD 2023-09-22T22:05:41.599 DEBUG:boto:Path: /ffzdqu-63/obj62 2023-09-22T22:05:41.599 DEBUG:boto:Data: 2023-09-22T22:05:41.599 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.599 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.599 DEBUG:boto:Params: {} 2023-09-22T22:05:41.599 DEBUG:boto:Token: None 2023-09-22T22:05:41.600 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj62 2023-09-22T22:05:41.600 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OAIpxDfV7JfYRtQO4F4/8FNuutU= 2023-09-22T22:05:41.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OAIpxDfV7JfYRtQO4F4/8FNuutU=', 'Content-Length': '0'} 2023-09-22T22:05:41.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022a7133c83242bd4-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.603 DEBUG:boto:path=/ffzdqu-63/obj62 2023-09-22T22:05:41.603 DEBUG:boto:auth_path=/ffzdqu-63/obj62 2023-09-22T22:05:41.603 DEBUG:boto:Method: HEAD 2023-09-22T22:05:41.603 DEBUG:boto:Path: /ffzdqu-63/obj62 2023-09-22T22:05:41.603 DEBUG:boto:Data: 2023-09-22T22:05:41.603 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.603 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.603 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.603 DEBUG:boto:Params: {} 2023-09-22T22:05:41.604 DEBUG:boto:Token: None 2023-09-22T22:05:41.604 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj62 2023-09-22T22:05:41.604 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OAIpxDfV7JfYRtQO4F4/8FNuutU= 2023-09-22T22:05:41.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OAIpxDfV7JfYRtQO4F4/8FNuutU=', 'Content-Length': '0'} 2023-09-22T22:05:41.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa41ac04520480fb-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-22T22:05:41.607 DEBUG:boto:path=/ffzdqu-63/obj62 2023-09-22T22:05:41.607 DEBUG:boto:auth_path=/ffzdqu-63/obj62 2023-09-22T22:05:41.607 DEBUG:boto:Method: GET 2023-09-22T22:05:41.607 DEBUG:boto:Path: /ffzdqu-63/obj62 2023-09-22T22:05:41.608 DEBUG:boto:Data: 2023-09-22T22:05:41.608 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.608 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.608 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.608 DEBUG:boto:Params: {} 2023-09-22T22:05:41.608 DEBUG:boto:Token: None 2023-09-22T22:05:41.608 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj62 2023-09-22T22:05:41.608 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AiDD/6aEAw13Up4dF71VPs9J60A= 2023-09-22T22:05:41.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AiDD/6aEAw13Up4dF71VPs9J60A=', 'Content-Length': '0'} 2023-09-22T22:05:41.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006d89a03516c5689-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.658 DEBUG:boto:path=/ffzdqu-63/obj62 2023-09-22T22:05:41.658 DEBUG:boto:auth_path=/ffzdqu-63/obj62 2023-09-22T22:05:41.658 DEBUG:boto:Method: GET 2023-09-22T22:05:41.659 DEBUG:boto:Path: /ffzdqu-63/obj62 2023-09-22T22:05:41.659 DEBUG:boto:Data: 2023-09-22T22:05:41.659 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.659 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.659 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.659 DEBUG:boto:Params: {} 2023-09-22T22:05:41.659 DEBUG:boto:Token: None 2023-09-22T22:05:41.659 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj62 2023-09-22T22:05:41.659 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AiDD/6aEAw13Up4dF71VPs9J60A= 2023-09-22T22:05:41.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AiDD/6aEAw13Up4dF71VPs9J60A=', 'Content-Length': '0'} 2023-09-22T22:05:41.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a40a82cfcbfcb3ee-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.663 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:41.663 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:41.663 DEBUG:boto:path=/ffzdqu-63/?prefix=obj62 2023-09-22T22:05:41.663 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj62 2023-09-22T22:05:41.663 DEBUG:boto:Method: GET 2023-09-22T22:05:41.663 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj62 2023-09-22T22:05:41.663 DEBUG:boto:Data: 2023-09-22T22:05:41.663 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.663 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.664 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.664 DEBUG:boto:Params: {} 2023-09-22T22:05:41.664 DEBUG:boto:Token: None 2023-09-22T22:05:41.664 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/ 2023-09-22T22:05:41.664 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:05:41.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:05:41.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cef660f9e6db5b0b-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.669 DEBUG:boto:b'ffzdqu-63obj621000falseobj622023-09-22T22:05:15.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:41.669 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:41.670 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:41.670 DEBUG:boto:path=/ffzdqu-63/?prefix=obj62 2023-09-22T22:05:41.670 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj62 2023-09-22T22:05:41.670 DEBUG:boto:Method: GET 2023-09-22T22:05:41.670 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj62 2023-09-22T22:05:41.670 DEBUG:boto:Data: 2023-09-22T22:05:41.670 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.670 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.670 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.671 DEBUG:boto:Params: {} 2023-09-22T22:05:41.671 DEBUG:boto:Token: None 2023-09-22T22:05:41.671 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/ 2023-09-22T22:05:41.671 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:05:41.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:05:41.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e685ec332aad672-00650e0fb5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.722 DEBUG:boto:b'ffzdqu-63obj621000falseobj622023-09-22T22:05:15.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:41.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T22:05:41.723 DEBUG:boto:path=/ffzdqu-63/obj68 2023-09-22T22:05:41.723 DEBUG:boto:auth_path=/ffzdqu-63/obj68 2023-09-22T22:05:41.723 DEBUG:boto:path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.723 DEBUG:boto:auth_path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.723 DEBUG:boto:Method: GET 2023-09-22T22:05:41.723 DEBUG:boto:Path: /ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.724 DEBUG:boto:Data: 2023-09-22T22:05:41.724 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.724 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.724 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.724 DEBUG:boto:Params: {} 2023-09-22T22:05:41.724 DEBUG:boto:Token: None 2023-09-22T22:05:41.724 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.724 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FvITgYTqoW0R9xO0TwNclNZrNFM= 2023-09-22T22:05:41.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FvITgYTqoW0R9xO0TwNclNZrNFM=', 'Content-Length': '0'} 2023-09-22T22:05:41.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b6e27662c02f00b-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.728 DEBUG:boto:path=/ffzdqu-63/obj68 2023-09-22T22:05:41.728 DEBUG:boto:auth_path=/ffzdqu-63/obj68 2023-09-22T22:05:41.728 DEBUG:boto:path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.728 DEBUG:boto:auth_path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.728 DEBUG:boto:Method: GET 2023-09-22T22:05:41.728 DEBUG:boto:Path: /ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.728 DEBUG:boto:Data: 2023-09-22T22:05:41.728 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.728 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.729 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.729 DEBUG:boto:Params: {} 2023-09-22T22:05:41.729 DEBUG:boto:Token: None 2023-09-22T22:05:41.729 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.729 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FvITgYTqoW0R9xO0TwNclNZrNFM= 2023-09-22T22:05:41.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FvITgYTqoW0R9xO0TwNclNZrNFM=', 'Content-Length': '0'} 2023-09-22T22:05:41.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000989f29a810556aa7-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.732 DEBUG:boto:path=/ffzdqu-63/obj68 2023-09-22T22:05:41.732 DEBUG:boto:auth_path=/ffzdqu-63/obj68 2023-09-22T22:05:41.732 DEBUG:boto:path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.733 DEBUG:boto:auth_path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.733 DEBUG:boto:Method: HEAD 2023-09-22T22:05:41.733 DEBUG:boto:Path: /ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.733 DEBUG:boto:Data: 2023-09-22T22:05:41.733 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.733 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.733 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.733 DEBUG:boto:Params: {} 2023-09-22T22:05:41.733 DEBUG:boto:Token: None 2023-09-22T22:05:41.734 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.734 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jCI47/21CweQP6sNI/kiuQzkkWQ= 2023-09-22T22:05:41.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jCI47/21CweQP6sNI/kiuQzkkWQ=', 'Content-Length': '0'} 2023-09-22T22:05:41.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4e0688f1e6333a6-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.737 DEBUG:boto:path=/ffzdqu-63/obj68 2023-09-22T22:05:41.737 DEBUG:boto:auth_path=/ffzdqu-63/obj68 2023-09-22T22:05:41.737 DEBUG:boto:path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.737 DEBUG:boto:auth_path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.738 DEBUG:boto:Method: HEAD 2023-09-22T22:05:41.738 DEBUG:boto:Path: /ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.738 DEBUG:boto:Data: 2023-09-22T22:05:41.738 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.738 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.738 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.738 DEBUG:boto:Params: {} 2023-09-22T22:05:41.738 DEBUG:boto:Token: None 2023-09-22T22:05:41.738 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.739 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jCI47/21CweQP6sNI/kiuQzkkWQ= 2023-09-22T22:05:41.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jCI47/21CweQP6sNI/kiuQzkkWQ=', 'Content-Length': '0'} 2023-09-22T22:05:41.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001814f87731355365-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T22:05:41.742 DEBUG:boto:path=/ffzdqu-63/obj68 2023-09-22T22:05:41.742 DEBUG:boto:auth_path=/ffzdqu-63/obj68 2023-09-22T22:05:41.742 DEBUG:boto:path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.742 DEBUG:boto:auth_path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.742 DEBUG:boto:Method: GET 2023-09-22T22:05:41.742 DEBUG:boto:Path: /ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.742 DEBUG:boto:Data: 2023-09-22T22:05:41.743 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.743 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.743 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.743 DEBUG:boto:Params: {} 2023-09-22T22:05:41.743 DEBUG:boto:Token: None 2023-09-22T22:05:41.743 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.743 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FvITgYTqoW0R9xO0TwNclNZrNFM= 2023-09-22T22:05:41.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FvITgYTqoW0R9xO0TwNclNZrNFM=', 'Content-Length': '0'} 2023-09-22T22:05:41.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaa4341dda2526e5-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.746 DEBUG:boto:path=/ffzdqu-63/obj68 2023-09-22T22:05:41.746 DEBUG:boto:auth_path=/ffzdqu-63/obj68 2023-09-22T22:05:41.747 DEBUG:boto:path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.747 DEBUG:boto:auth_path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.747 DEBUG:boto:Method: GET 2023-09-22T22:05:41.747 DEBUG:boto:Path: /ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.747 DEBUG:boto:Data: 2023-09-22T22:05:41.747 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.747 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.747 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.747 DEBUG:boto:Params: {} 2023-09-22T22:05:41.747 DEBUG:boto:Token: None 2023-09-22T22:05:41.748 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj68?versionId=null 2023-09-22T22:05:41.748 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:FvITgYTqoW0R9xO0TwNclNZrNFM= 2023-09-22T22:05:41.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FvITgYTqoW0R9xO0TwNclNZrNFM=', 'Content-Length': '0'} 2023-09-22T22:05:41.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b335ae60c5c2d00d-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.751 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:41.751 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:41.751 DEBUG:boto:path=/ffzdqu-63/?prefix=obj68 2023-09-22T22:05:41.751 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj68 2023-09-22T22:05:41.751 DEBUG:boto:Method: GET 2023-09-22T22:05:41.751 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj68 2023-09-22T22:05:41.751 DEBUG:boto:Data: 2023-09-22T22:05:41.751 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.751 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.752 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.752 DEBUG:boto:Params: {} 2023-09-22T22:05:41.752 DEBUG:boto:Token: None 2023-09-22T22:05:41.752 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/ 2023-09-22T22:05:41.752 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:05:41.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:05:41.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029cb7e95b699c5e1-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.760 DEBUG:boto:b'ffzdqu-63obj681000falseobj682023-09-22T22:05:14.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:41.761 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:41.761 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:41.761 DEBUG:boto:path=/ffzdqu-63/?prefix=obj68 2023-09-22T22:05:41.761 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj68 2023-09-22T22:05:41.761 DEBUG:boto:Method: GET 2023-09-22T22:05:41.761 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj68 2023-09-22T22:05:41.761 DEBUG:boto:Data: 2023-09-22T22:05:41.762 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.762 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.762 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.762 DEBUG:boto:Params: {} 2023-09-22T22:05:41.762 DEBUG:boto:Token: None 2023-09-22T22:05:41.762 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/ 2023-09-22T22:05:41.762 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:05:41.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:05:41.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000243fb6129496c153-00650e0fb5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.822 DEBUG:boto:b'ffzdqu-63obj681000falseobj682023-09-22T22:05:14.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:41.823 DEBUG:boto:path=/ffzdqu-63/obj68 2023-09-22T22:05:41.823 DEBUG:boto:auth_path=/ffzdqu-63/obj68 2023-09-22T22:05:41.823 DEBUG:boto:Method: HEAD 2023-09-22T22:05:41.823 DEBUG:boto:Path: /ffzdqu-63/obj68 2023-09-22T22:05:41.823 DEBUG:boto:Data: 2023-09-22T22:05:41.824 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.824 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.824 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.824 DEBUG:boto:Params: {} 2023-09-22T22:05:41.824 DEBUG:boto:Token: None 2023-09-22T22:05:41.824 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj68 2023-09-22T22:05:41.824 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WAx3PX9RhdxC2kGJX4HxiNImGRs= 2023-09-22T22:05:41.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WAx3PX9RhdxC2kGJX4HxiNImGRs=', 'Content-Length': '0'} 2023-09-22T22:05:41.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000118af6acae65e701-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.827 DEBUG:boto:path=/ffzdqu-63/obj68 2023-09-22T22:05:41.828 DEBUG:boto:auth_path=/ffzdqu-63/obj68 2023-09-22T22:05:41.828 DEBUG:boto:Method: HEAD 2023-09-22T22:05:41.828 DEBUG:boto:Path: /ffzdqu-63/obj68 2023-09-22T22:05:41.828 DEBUG:boto:Data: 2023-09-22T22:05:41.828 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.828 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.828 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.828 DEBUG:boto:Params: {} 2023-09-22T22:05:41.828 DEBUG:boto:Token: None 2023-09-22T22:05:41.829 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj68 2023-09-22T22:05:41.829 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WAx3PX9RhdxC2kGJX4HxiNImGRs= 2023-09-22T22:05:41.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WAx3PX9RhdxC2kGJX4HxiNImGRs=', 'Content-Length': '0'} 2023-09-22T22:05:41.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003042b73f170eea02-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-22T22:05:41.832 DEBUG:boto:path=/ffzdqu-63/obj68 2023-09-22T22:05:41.832 DEBUG:boto:auth_path=/ffzdqu-63/obj68 2023-09-22T22:05:41.832 DEBUG:boto:Method: GET 2023-09-22T22:05:41.832 DEBUG:boto:Path: /ffzdqu-63/obj68 2023-09-22T22:05:41.832 DEBUG:boto:Data: 2023-09-22T22:05:41.832 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.832 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.833 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.833 DEBUG:boto:Params: {} 2023-09-22T22:05:41.833 DEBUG:boto:Token: None 2023-09-22T22:05:41.833 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj68 2023-09-22T22:05:41.833 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:owXkfr9fm1zG6i3JCGUj3vvou1Q= 2023-09-22T22:05:41.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:owXkfr9fm1zG6i3JCGUj3vvou1Q=', 'Content-Length': '0'} 2023-09-22T22:05:41.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000320ea172f4d93ae2-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.878 DEBUG:boto:path=/ffzdqu-63/obj68 2023-09-22T22:05:41.878 DEBUG:boto:auth_path=/ffzdqu-63/obj68 2023-09-22T22:05:41.879 DEBUG:boto:Method: GET 2023-09-22T22:05:41.879 DEBUG:boto:Path: /ffzdqu-63/obj68 2023-09-22T22:05:41.879 DEBUG:boto:Data: 2023-09-22T22:05:41.879 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.879 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.879 DEBUG:boto:Params: {} 2023-09-22T22:05:41.879 DEBUG:boto:Token: None 2023-09-22T22:05:41.880 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj68 2023-09-22T22:05:41.880 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:owXkfr9fm1zG6i3JCGUj3vvou1Q= 2023-09-22T22:05:41.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:owXkfr9fm1zG6i3JCGUj3vvou1Q=', 'Content-Length': '0'} 2023-09-22T22:05:41.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2f69cbd8d819fbf-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.883 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:41.883 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:41.883 DEBUG:boto:path=/ffzdqu-63/?prefix=obj68 2023-09-22T22:05:41.883 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj68 2023-09-22T22:05:41.883 DEBUG:boto:Method: GET 2023-09-22T22:05:41.884 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj68 2023-09-22T22:05:41.884 DEBUG:boto:Data: 2023-09-22T22:05:41.884 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.884 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.884 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.884 DEBUG:boto:Params: {} 2023-09-22T22:05:41.884 DEBUG:boto:Token: None 2023-09-22T22:05:41.884 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/ 2023-09-22T22:05:41.884 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:05:41.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:05:41.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7dcdeafabcf38a9-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.890 DEBUG:boto:b'ffzdqu-63obj681000falseobj682023-09-22T22:05:14.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:41.891 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:41.891 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:41.891 DEBUG:boto:path=/ffzdqu-63/?prefix=obj68 2023-09-22T22:05:41.891 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj68 2023-09-22T22:05:41.891 DEBUG:boto:Method: GET 2023-09-22T22:05:41.891 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj68 2023-09-22T22:05:41.891 DEBUG:boto:Data: 2023-09-22T22:05:41.891 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.891 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.891 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.892 DEBUG:boto:Params: {} 2023-09-22T22:05:41.892 DEBUG:boto:Token: None 2023-09-22T22:05:41.892 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/ 2023-09-22T22:05:41.892 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:05:41.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:05:41.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af3dd1b84a32c9c6-00650e0fb5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.903 DEBUG:boto:b'ffzdqu-63obj681000falseobj682023-09-22T22:05:14.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:41.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-22T22:05:41.903 DEBUG:boto:path=/ffzdqu-63/obj69 2023-09-22T22:05:41.903 DEBUG:boto:auth_path=/ffzdqu-63/obj69 2023-09-22T22:05:41.903 DEBUG:boto:path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:41.904 DEBUG:boto:auth_path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:41.904 DEBUG:boto:Method: GET 2023-09-22T22:05:41.904 DEBUG:boto:Path: /ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:41.904 DEBUG:boto:Data: 2023-09-22T22:05:41.904 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.904 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.904 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.904 DEBUG:boto:Params: {} 2023-09-22T22:05:41.904 DEBUG:boto:Token: None 2023-09-22T22:05:41.905 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:41.905 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7TUY0Fp5mj5/UaiaSX8JIZVGOrg= 2023-09-22T22:05:41.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7TUY0Fp5mj5/UaiaSX8JIZVGOrg=', 'Content-Length': '0'} 2023-09-22T22:05:41.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3836e6d25837e6c-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.908 DEBUG:boto:path=/ffzdqu-63/obj69 2023-09-22T22:05:41.908 DEBUG:boto:auth_path=/ffzdqu-63/obj69 2023-09-22T22:05:41.908 DEBUG:boto:path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:41.908 DEBUG:boto:auth_path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:41.909 DEBUG:boto:Method: GET 2023-09-22T22:05:41.909 DEBUG:boto:Path: /ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:41.909 DEBUG:boto:Data: 2023-09-22T22:05:41.909 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.909 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.909 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.909 DEBUG:boto:Params: {} 2023-09-22T22:05:41.909 DEBUG:boto:Token: None 2023-09-22T22:05:41.909 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:41.910 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7TUY0Fp5mj5/UaiaSX8JIZVGOrg= 2023-09-22T22:05:41.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7TUY0Fp5mj5/UaiaSX8JIZVGOrg=', 'Content-Length': '0'} 2023-09-22T22:05:41.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000191b24d825b7629-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.958 DEBUG:boto:path=/ffzdqu-63/obj69 2023-09-22T22:05:41.958 DEBUG:boto:auth_path=/ffzdqu-63/obj69 2023-09-22T22:05:41.959 DEBUG:boto:path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:41.959 DEBUG:boto:auth_path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:41.959 DEBUG:boto:Method: HEAD 2023-09-22T22:05:41.959 DEBUG:boto:Path: /ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:41.959 DEBUG:boto:Data: 2023-09-22T22:05:41.959 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.959 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.959 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.959 DEBUG:boto:Params: {} 2023-09-22T22:05:41.960 DEBUG:boto:Token: None 2023-09-22T22:05:41.960 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:41.960 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GRhT5FWesUOzCSxYZVCzfPcOSzQ= 2023-09-22T22:05:41.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GRhT5FWesUOzCSxYZVCzfPcOSzQ=', 'Content-Length': '0'} 2023-09-22T22:05:41.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b08ffedab928b3e-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.963 DEBUG:boto:path=/ffzdqu-63/obj69 2023-09-22T22:05:41.963 DEBUG:boto:auth_path=/ffzdqu-63/obj69 2023-09-22T22:05:41.963 DEBUG:boto:path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:41.963 DEBUG:boto:auth_path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:41.963 DEBUG:boto:Method: HEAD 2023-09-22T22:05:41.963 DEBUG:boto:Path: /ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:41.964 DEBUG:boto:Data: 2023-09-22T22:05:41.964 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.964 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:41.964 DEBUG:boto:Port: 8000 2023-09-22T22:05:41.964 DEBUG:boto:Params: {} 2023-09-22T22:05:41.964 DEBUG:boto:Token: None 2023-09-22T22:05:41.964 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:41.964 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:GRhT5FWesUOzCSxYZVCzfPcOSzQ= 2023-09-22T22:05:41.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GRhT5FWesUOzCSxYZVCzfPcOSzQ=', 'Content-Length': '0'} 2023-09-22T22:05:41.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b82628b2c1f4349-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:41.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-22T22:05:41.968 DEBUG:boto:path=/ffzdqu-63/obj69 2023-09-22T22:05:41.968 DEBUG:boto:auth_path=/ffzdqu-63/obj69 2023-09-22T22:05:41.968 DEBUG:boto:path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:41.968 DEBUG:boto:auth_path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:41.968 DEBUG:boto:Method: GET 2023-09-22T22:05:41.968 DEBUG:boto:Path: /ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:41.969 DEBUG:boto:Data: 2023-09-22T22:05:41.969 DEBUG:boto:Headers: {} 2023-09-22T22:05:41.969 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:41.969 DEBUG:boto:Port: 8001 2023-09-22T22:05:41.969 DEBUG:boto:Params: {} 2023-09-22T22:05:41.969 DEBUG:boto:Token: None 2023-09-22T22:05:41.969 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:41 GMT /ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:41.969 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7TUY0Fp5mj5/UaiaSX8JIZVGOrg= 2023-09-22T22:05:41.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7TUY0Fp5mj5/UaiaSX8JIZVGOrg=', 'Content-Length': '0'} 2023-09-22T22:05:41.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd354ff5395730db-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.014 DEBUG:boto:path=/ffzdqu-63/obj69 2023-09-22T22:05:42.014 DEBUG:boto:auth_path=/ffzdqu-63/obj69 2023-09-22T22:05:42.015 DEBUG:boto:path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:42.015 DEBUG:boto:auth_path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:42.015 DEBUG:boto:Method: GET 2023-09-22T22:05:42.015 DEBUG:boto:Path: /ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:42.015 DEBUG:boto:Data: 2023-09-22T22:05:42.015 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.015 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.015 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.015 DEBUG:boto:Params: {} 2023-09-22T22:05:42.015 DEBUG:boto:Token: None 2023-09-22T22:05:42.016 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj69?versionId=null 2023-09-22T22:05:42.016 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:PguPhR+3E82NW/SdNY4vTsfm8wA= 2023-09-22T22:05:42.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PguPhR+3E82NW/SdNY4vTsfm8wA=', 'Content-Length': '0'} 2023-09-22T22:05:42.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058e90bedf4799e97-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.019 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:42.019 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:42.019 DEBUG:boto:path=/ffzdqu-63/?prefix=obj69 2023-09-22T22:05:42.020 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj69 2023-09-22T22:05:42.020 DEBUG:boto:Method: GET 2023-09-22T22:05:42.020 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj69 2023-09-22T22:05:42.020 DEBUG:boto:Data: 2023-09-22T22:05:42.020 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.020 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.020 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.020 DEBUG:boto:Params: {} 2023-09-22T22:05:42.020 DEBUG:boto:Token: None 2023-09-22T22:05:42.020 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/ 2023-09-22T22:05:42.021 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:05:42.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:05:42.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046d2941ebaabbb8e-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.025 DEBUG:boto:b'ffzdqu-63obj691000falseobj692023-09-22T22:05:14.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:42.025 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:42.026 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:42.026 DEBUG:boto:path=/ffzdqu-63/?prefix=obj69 2023-09-22T22:05:42.026 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj69 2023-09-22T22:05:42.026 DEBUG:boto:Method: GET 2023-09-22T22:05:42.026 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj69 2023-09-22T22:05:42.026 DEBUG:boto:Data: 2023-09-22T22:05:42.026 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.026 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.026 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.027 DEBUG:boto:Params: {} 2023-09-22T22:05:42.027 DEBUG:boto:Token: None 2023-09-22T22:05:42.027 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/ 2023-09-22T22:05:42.027 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:05:42.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:05:42.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000215fc50e2785c593-00650e0fb6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.037 DEBUG:boto:b'ffzdqu-63obj691000falseobj692023-09-22T22:05:14.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:42.038 DEBUG:boto:path=/ffzdqu-63/obj69 2023-09-22T22:05:42.038 DEBUG:boto:auth_path=/ffzdqu-63/obj69 2023-09-22T22:05:42.038 DEBUG:boto:Method: HEAD 2023-09-22T22:05:42.038 DEBUG:boto:Path: /ffzdqu-63/obj69 2023-09-22T22:05:42.038 DEBUG:boto:Data: 2023-09-22T22:05:42.038 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.038 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.038 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.038 DEBUG:boto:Params: {} 2023-09-22T22:05:42.039 DEBUG:boto:Token: None 2023-09-22T22:05:42.039 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj69 2023-09-22T22:05:42.039 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XuZpWUYn0ag7tSzsAEQNYwdaRcQ= 2023-09-22T22:05:42.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XuZpWUYn0ag7tSzsAEQNYwdaRcQ=', 'Content-Length': '0'} 2023-09-22T22:05:42.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c970e6d100ea051-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.042 DEBUG:boto:path=/ffzdqu-63/obj69 2023-09-22T22:05:42.042 DEBUG:boto:auth_path=/ffzdqu-63/obj69 2023-09-22T22:05:42.042 DEBUG:boto:Method: HEAD 2023-09-22T22:05:42.042 DEBUG:boto:Path: /ffzdqu-63/obj69 2023-09-22T22:05:42.042 DEBUG:boto:Data: 2023-09-22T22:05:42.042 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.042 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.042 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.043 DEBUG:boto:Params: {} 2023-09-22T22:05:42.043 DEBUG:boto:Token: None 2023-09-22T22:05:42.043 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj69 2023-09-22T22:05:42.043 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XuZpWUYn0ag7tSzsAEQNYwdaRcQ= 2023-09-22T22:05:42.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XuZpWUYn0ag7tSzsAEQNYwdaRcQ=', 'Content-Length': '0'} 2023-09-22T22:05:42.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e7f5053d6e98b65-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-22T22:05:42.046 DEBUG:boto:path=/ffzdqu-63/obj69 2023-09-22T22:05:42.046 DEBUG:boto:auth_path=/ffzdqu-63/obj69 2023-09-22T22:05:42.046 DEBUG:boto:Method: GET 2023-09-22T22:05:42.046 DEBUG:boto:Path: /ffzdqu-63/obj69 2023-09-22T22:05:42.047 DEBUG:boto:Data: 2023-09-22T22:05:42.047 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.047 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.047 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.047 DEBUG:boto:Params: {} 2023-09-22T22:05:42.047 DEBUG:boto:Token: None 2023-09-22T22:05:42.047 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj69 2023-09-22T22:05:42.047 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZIdMz94KnwG7/RbWGfTl2jQr12E= 2023-09-22T22:05:42.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZIdMz94KnwG7/RbWGfTl2jQr12E=', 'Content-Length': '0'} 2023-09-22T22:05:42.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087b76c4e17738f4c-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.050 DEBUG:boto:path=/ffzdqu-63/obj69 2023-09-22T22:05:42.050 DEBUG:boto:auth_path=/ffzdqu-63/obj69 2023-09-22T22:05:42.051 DEBUG:boto:Method: GET 2023-09-22T22:05:42.051 DEBUG:boto:Path: /ffzdqu-63/obj69 2023-09-22T22:05:42.051 DEBUG:boto:Data: 2023-09-22T22:05:42.051 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.051 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.051 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.051 DEBUG:boto:Params: {} 2023-09-22T22:05:42.051 DEBUG:boto:Token: None 2023-09-22T22:05:42.051 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj69 2023-09-22T22:05:42.052 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ZIdMz94KnwG7/RbWGfTl2jQr12E= 2023-09-22T22:05:42.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZIdMz94KnwG7/RbWGfTl2jQr12E=', 'Content-Length': '0'} 2023-09-22T22:05:42.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a9fc905960e9b1b-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.094 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:42.095 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:42.095 DEBUG:boto:path=/ffzdqu-63/?prefix=obj69 2023-09-22T22:05:42.095 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj69 2023-09-22T22:05:42.095 DEBUG:boto:Method: GET 2023-09-22T22:05:42.095 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj69 2023-09-22T22:05:42.095 DEBUG:boto:Data: 2023-09-22T22:05:42.095 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.095 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.096 DEBUG:boto:Params: {} 2023-09-22T22:05:42.096 DEBUG:boto:Token: None 2023-09-22T22:05:42.096 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/ 2023-09-22T22:05:42.096 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:05:42.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:05:42.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e7efd0b0c33d1d2-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.101 DEBUG:boto:b'ffzdqu-63obj691000falseobj692023-09-22T22:05:14.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:42.101 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:42.101 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:42.101 DEBUG:boto:path=/ffzdqu-63/?prefix=obj69 2023-09-22T22:05:42.101 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj69 2023-09-22T22:05:42.101 DEBUG:boto:Method: GET 2023-09-22T22:05:42.102 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj69 2023-09-22T22:05:42.102 DEBUG:boto:Data: 2023-09-22T22:05:42.102 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.102 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.102 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.102 DEBUG:boto:Params: {} 2023-09-22T22:05:42.102 DEBUG:boto:Token: None 2023-09-22T22:05:42.102 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/ 2023-09-22T22:05:42.102 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:05:42.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:05:42.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d98b473427b5e24f-00650e0fb6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.113 DEBUG:boto:b'ffzdqu-63obj691000falseobj692023-09-22T22:05:14.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:42.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-09-22T22:05:42.113 DEBUG:boto:path=/ffzdqu-63/obj74 2023-09-22T22:05:42.113 DEBUG:boto:auth_path=/ffzdqu-63/obj74 2023-09-22T22:05:42.113 DEBUG:boto:path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.113 DEBUG:boto:auth_path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.113 DEBUG:boto:Method: GET 2023-09-22T22:05:42.114 DEBUG:boto:Path: /ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.114 DEBUG:boto:Data: 2023-09-22T22:05:42.114 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.114 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.114 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.114 DEBUG:boto:Params: {} 2023-09-22T22:05:42.114 DEBUG:boto:Token: None 2023-09-22T22:05:42.114 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.115 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tPNM71+HlmzSKA7c15maruaCOY8= 2023-09-22T22:05:42.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tPNM71+HlmzSKA7c15maruaCOY8=', 'Content-Length': '0'} 2023-09-22T22:05:42.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000423db6dd3a6b3d6c-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.158 DEBUG:boto:path=/ffzdqu-63/obj74 2023-09-22T22:05:42.159 DEBUG:boto:auth_path=/ffzdqu-63/obj74 2023-09-22T22:05:42.159 DEBUG:boto:path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.159 DEBUG:boto:auth_path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.159 DEBUG:boto:Method: GET 2023-09-22T22:05:42.159 DEBUG:boto:Path: /ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.159 DEBUG:boto:Data: 2023-09-22T22:05:42.159 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.159 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.159 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.160 DEBUG:boto:Params: {} 2023-09-22T22:05:42.160 DEBUG:boto:Token: None 2023-09-22T22:05:42.160 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.160 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tPNM71+HlmzSKA7c15maruaCOY8= 2023-09-22T22:05:42.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tPNM71+HlmzSKA7c15maruaCOY8=', 'Content-Length': '0'} 2023-09-22T22:05:42.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2e1ddc7c9ca387d-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.164 DEBUG:boto:path=/ffzdqu-63/obj74 2023-09-22T22:05:42.164 DEBUG:boto:auth_path=/ffzdqu-63/obj74 2023-09-22T22:05:42.164 DEBUG:boto:path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.164 DEBUG:boto:auth_path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.164 DEBUG:boto:Method: HEAD 2023-09-22T22:05:42.164 DEBUG:boto:Path: /ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.164 DEBUG:boto:Data: 2023-09-22T22:05:42.164 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.164 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.164 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.165 DEBUG:boto:Params: {} 2023-09-22T22:05:42.165 DEBUG:boto:Token: None 2023-09-22T22:05:42.165 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.165 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/v7h9/hPrOGDciltnsOJT2B7UJo= 2023-09-22T22:05:42.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/v7h9/hPrOGDciltnsOJT2B7UJo=', 'Content-Length': '0'} 2023-09-22T22:05:42.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000954fa1fa68ea669b-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.168 DEBUG:boto:path=/ffzdqu-63/obj74 2023-09-22T22:05:42.168 DEBUG:boto:auth_path=/ffzdqu-63/obj74 2023-09-22T22:05:42.168 DEBUG:boto:path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.168 DEBUG:boto:auth_path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.168 DEBUG:boto:Method: HEAD 2023-09-22T22:05:42.168 DEBUG:boto:Path: /ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.168 DEBUG:boto:Data: 2023-09-22T22:05:42.169 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.169 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.169 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.169 DEBUG:boto:Params: {} 2023-09-22T22:05:42.169 DEBUG:boto:Token: None 2023-09-22T22:05:42.169 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.169 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/v7h9/hPrOGDciltnsOJT2B7UJo= 2023-09-22T22:05:42.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/v7h9/hPrOGDciltnsOJT2B7UJo=', 'Content-Length': '0'} 2023-09-22T22:05:42.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feb9b46e29068adc-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-09-22T22:05:42.172 DEBUG:boto:path=/ffzdqu-63/obj74 2023-09-22T22:05:42.172 DEBUG:boto:auth_path=/ffzdqu-63/obj74 2023-09-22T22:05:42.172 DEBUG:boto:path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.172 DEBUG:boto:auth_path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.172 DEBUG:boto:Method: GET 2023-09-22T22:05:42.173 DEBUG:boto:Path: /ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.173 DEBUG:boto:Data: 2023-09-22T22:05:42.173 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.173 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.173 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.173 DEBUG:boto:Params: {} 2023-09-22T22:05:42.173 DEBUG:boto:Token: None 2023-09-22T22:05:42.173 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.173 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tPNM71+HlmzSKA7c15maruaCOY8= 2023-09-22T22:05:42.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tPNM71+HlmzSKA7c15maruaCOY8=', 'Content-Length': '0'} 2023-09-22T22:05:42.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033528eba14e727b4-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.176 DEBUG:boto:path=/ffzdqu-63/obj74 2023-09-22T22:05:42.176 DEBUG:boto:auth_path=/ffzdqu-63/obj74 2023-09-22T22:05:42.177 DEBUG:boto:path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.177 DEBUG:boto:auth_path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.177 DEBUG:boto:Method: GET 2023-09-22T22:05:42.177 DEBUG:boto:Path: /ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.177 DEBUG:boto:Data: 2023-09-22T22:05:42.177 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.177 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.177 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.177 DEBUG:boto:Params: {} 2023-09-22T22:05:42.177 DEBUG:boto:Token: None 2023-09-22T22:05:42.178 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj74?versionId=null 2023-09-22T22:05:42.178 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:tPNM71+HlmzSKA7c15maruaCOY8= 2023-09-22T22:05:42.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tPNM71+HlmzSKA7c15maruaCOY8=', 'Content-Length': '0'} 2023-09-22T22:05:42.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b4931cbd7a633cb-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.226 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:42.227 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:42.227 DEBUG:boto:path=/ffzdqu-63/?prefix=obj74 2023-09-22T22:05:42.227 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj74 2023-09-22T22:05:42.227 DEBUG:boto:Method: GET 2023-09-22T22:05:42.227 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj74 2023-09-22T22:05:42.227 DEBUG:boto:Data: 2023-09-22T22:05:42.227 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.227 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.227 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.227 DEBUG:boto:Params: {} 2023-09-22T22:05:42.228 DEBUG:boto:Token: None 2023-09-22T22:05:42.228 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/ 2023-09-22T22:05:42.228 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:05:42.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:05:42.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f548e7e535e5b0a-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.233 DEBUG:boto:b'ffzdqu-63obj741000falseobj742023-09-22T22:05:15.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:42.233 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:42.233 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:42.233 DEBUG:boto:path=/ffzdqu-63/?prefix=obj74 2023-09-22T22:05:42.233 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj74 2023-09-22T22:05:42.234 DEBUG:boto:Method: GET 2023-09-22T22:05:42.234 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj74 2023-09-22T22:05:42.234 DEBUG:boto:Data: 2023-09-22T22:05:42.234 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.234 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.234 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.234 DEBUG:boto:Params: {} 2023-09-22T22:05:42.234 DEBUG:boto:Token: None 2023-09-22T22:05:42.234 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/ 2023-09-22T22:05:42.235 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:05:42.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:05:42.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006846d15f619e174f-00650e0fb6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.244 DEBUG:boto:b'ffzdqu-63obj741000falseobj742023-09-22T22:05:15.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:42.245 DEBUG:boto:path=/ffzdqu-63/obj74 2023-09-22T22:05:42.245 DEBUG:boto:auth_path=/ffzdqu-63/obj74 2023-09-22T22:05:42.245 DEBUG:boto:Method: HEAD 2023-09-22T22:05:42.245 DEBUG:boto:Path: /ffzdqu-63/obj74 2023-09-22T22:05:42.245 DEBUG:boto:Data: 2023-09-22T22:05:42.245 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.245 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.245 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.246 DEBUG:boto:Params: {} 2023-09-22T22:05:42.246 DEBUG:boto:Token: None 2023-09-22T22:05:42.246 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj74 2023-09-22T22:05:42.246 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:flTxT3yLAOliRRvPQBmCso8P5IY= 2023-09-22T22:05:42.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:flTxT3yLAOliRRvPQBmCso8P5IY=', 'Content-Length': '0'} 2023-09-22T22:05:42.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067b073d144472003-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.249 DEBUG:boto:path=/ffzdqu-63/obj74 2023-09-22T22:05:42.249 DEBUG:boto:auth_path=/ffzdqu-63/obj74 2023-09-22T22:05:42.249 DEBUG:boto:Method: HEAD 2023-09-22T22:05:42.249 DEBUG:boto:Path: /ffzdqu-63/obj74 2023-09-22T22:05:42.249 DEBUG:boto:Data: 2023-09-22T22:05:42.250 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.250 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.250 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.250 DEBUG:boto:Params: {} 2023-09-22T22:05:42.250 DEBUG:boto:Token: None 2023-09-22T22:05:42.250 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj74 2023-09-22T22:05:42.250 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:flTxT3yLAOliRRvPQBmCso8P5IY= 2023-09-22T22:05:42.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:flTxT3yLAOliRRvPQBmCso8P5IY=', 'Content-Length': '0'} 2023-09-22T22:05:42.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086ac682da727bb55-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-09-22T22:05:42.253 DEBUG:boto:path=/ffzdqu-63/obj74 2023-09-22T22:05:42.253 DEBUG:boto:auth_path=/ffzdqu-63/obj74 2023-09-22T22:05:42.253 DEBUG:boto:Method: GET 2023-09-22T22:05:42.254 DEBUG:boto:Path: /ffzdqu-63/obj74 2023-09-22T22:05:42.254 DEBUG:boto:Data: 2023-09-22T22:05:42.254 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.254 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.254 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.254 DEBUG:boto:Params: {} 2023-09-22T22:05:42.254 DEBUG:boto:Token: None 2023-09-22T22:05:42.254 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj74 2023-09-22T22:05:42.254 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:In1MH66XZQ53RNE9LoyKTYvUUYw= 2023-09-22T22:05:42.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:In1MH66XZQ53RNE9LoyKTYvUUYw=', 'Content-Length': '0'} 2023-09-22T22:05:42.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4204d5b18eeb341-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.298 DEBUG:boto:path=/ffzdqu-63/obj74 2023-09-22T22:05:42.299 DEBUG:boto:auth_path=/ffzdqu-63/obj74 2023-09-22T22:05:42.299 DEBUG:boto:Method: GET 2023-09-22T22:05:42.299 DEBUG:boto:Path: /ffzdqu-63/obj74 2023-09-22T22:05:42.299 DEBUG:boto:Data: 2023-09-22T22:05:42.299 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.299 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.299 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.299 DEBUG:boto:Params: {} 2023-09-22T22:05:42.299 DEBUG:boto:Token: None 2023-09-22T22:05:42.300 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj74 2023-09-22T22:05:42.300 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:In1MH66XZQ53RNE9LoyKTYvUUYw= 2023-09-22T22:05:42.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:In1MH66XZQ53RNE9LoyKTYvUUYw=', 'Content-Length': '0'} 2023-09-22T22:05:42.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003d75cf950d49090-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.303 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:42.303 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:42.303 DEBUG:boto:path=/ffzdqu-63/?prefix=obj74 2023-09-22T22:05:42.303 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj74 2023-09-22T22:05:42.303 DEBUG:boto:Method: GET 2023-09-22T22:05:42.303 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj74 2023-09-22T22:05:42.304 DEBUG:boto:Data: 2023-09-22T22:05:42.304 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.304 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.304 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.304 DEBUG:boto:Params: {} 2023-09-22T22:05:42.304 DEBUG:boto:Token: None 2023-09-22T22:05:42.304 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/ 2023-09-22T22:05:42.304 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:05:42.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:05:42.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd4c5ac36a87aa15-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.309 DEBUG:boto:b'ffzdqu-63obj741000falseobj742023-09-22T22:05:15.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:42.309 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:42.309 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:42.309 DEBUG:boto:path=/ffzdqu-63/?prefix=obj74 2023-09-22T22:05:42.309 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj74 2023-09-22T22:05:42.309 DEBUG:boto:Method: GET 2023-09-22T22:05:42.310 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj74 2023-09-22T22:05:42.310 DEBUG:boto:Data: 2023-09-22T22:05:42.310 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.310 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.310 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.310 DEBUG:boto:Params: {} 2023-09-22T22:05:42.310 DEBUG:boto:Token: None 2023-09-22T22:05:42.310 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/ 2023-09-22T22:05:42.310 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:05:42.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:05:42.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0adfe3f1b698438-00650e0fb6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.362 DEBUG:boto:b'ffzdqu-63obj741000falseobj742023-09-22T22:05:15.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:42.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T22:05:42.363 DEBUG:boto:path=/ffzdqu-63/obj76 2023-09-22T22:05:42.363 DEBUG:boto:auth_path=/ffzdqu-63/obj76 2023-09-22T22:05:42.363 DEBUG:boto:path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.363 DEBUG:boto:auth_path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.363 DEBUG:boto:Method: GET 2023-09-22T22:05:42.363 DEBUG:boto:Path: /ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.363 DEBUG:boto:Data: 2023-09-22T22:05:42.364 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.364 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.364 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.364 DEBUG:boto:Params: {} 2023-09-22T22:05:42.364 DEBUG:boto:Token: None 2023-09-22T22:05:42.364 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.364 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XhtPes5kmI6PMA9ufpMHuL+MZ3g= 2023-09-22T22:05:42.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XhtPes5kmI6PMA9ufpMHuL+MZ3g=', 'Content-Length': '0'} 2023-09-22T22:05:42.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001aa8584759d67829-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.368 DEBUG:boto:path=/ffzdqu-63/obj76 2023-09-22T22:05:42.368 DEBUG:boto:auth_path=/ffzdqu-63/obj76 2023-09-22T22:05:42.368 DEBUG:boto:path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.368 DEBUG:boto:auth_path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.368 DEBUG:boto:Method: GET 2023-09-22T22:05:42.368 DEBUG:boto:Path: /ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.368 DEBUG:boto:Data: 2023-09-22T22:05:42.368 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.368 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.369 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.369 DEBUG:boto:Params: {} 2023-09-22T22:05:42.369 DEBUG:boto:Token: None 2023-09-22T22:05:42.369 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.369 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XhtPes5kmI6PMA9ufpMHuL+MZ3g= 2023-09-22T22:05:42.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XhtPes5kmI6PMA9ufpMHuL+MZ3g=', 'Content-Length': '0'} 2023-09-22T22:05:42.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a59b793eedff0045-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.372 DEBUG:boto:path=/ffzdqu-63/obj76 2023-09-22T22:05:42.372 DEBUG:boto:auth_path=/ffzdqu-63/obj76 2023-09-22T22:05:42.373 DEBUG:boto:path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.373 DEBUG:boto:auth_path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.373 DEBUG:boto:Method: HEAD 2023-09-22T22:05:42.373 DEBUG:boto:Path: /ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.373 DEBUG:boto:Data: 2023-09-22T22:05:42.373 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.373 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.373 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.373 DEBUG:boto:Params: {} 2023-09-22T22:05:42.373 DEBUG:boto:Token: None 2023-09-22T22:05:42.374 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.374 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iOSpS1HmLIA1O66s6b3jyn4GP/I= 2023-09-22T22:05:42.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iOSpS1HmLIA1O66s6b3jyn4GP/I=', 'Content-Length': '0'} 2023-09-22T22:05:42.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000370ffb98985e7f5b-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.377 DEBUG:boto:path=/ffzdqu-63/obj76 2023-09-22T22:05:42.377 DEBUG:boto:auth_path=/ffzdqu-63/obj76 2023-09-22T22:05:42.377 DEBUG:boto:path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.377 DEBUG:boto:auth_path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.377 DEBUG:boto:Method: HEAD 2023-09-22T22:05:42.377 DEBUG:boto:Path: /ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.377 DEBUG:boto:Data: 2023-09-22T22:05:42.377 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.377 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.377 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.378 DEBUG:boto:Params: {} 2023-09-22T22:05:42.378 DEBUG:boto:Token: None 2023-09-22T22:05:42.378 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.378 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:iOSpS1HmLIA1O66s6b3jyn4GP/I= 2023-09-22T22:05:42.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iOSpS1HmLIA1O66s6b3jyn4GP/I=', 'Content-Length': '0'} 2023-09-22T22:05:42.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077b4ac220e50eadb-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T22:05:42.381 DEBUG:boto:path=/ffzdqu-63/obj76 2023-09-22T22:05:42.381 DEBUG:boto:auth_path=/ffzdqu-63/obj76 2023-09-22T22:05:42.381 DEBUG:boto:path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.381 DEBUG:boto:auth_path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.382 DEBUG:boto:Method: GET 2023-09-22T22:05:42.382 DEBUG:boto:Path: /ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.382 DEBUG:boto:Data: 2023-09-22T22:05:42.382 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.382 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.382 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.382 DEBUG:boto:Params: {} 2023-09-22T22:05:42.382 DEBUG:boto:Token: None 2023-09-22T22:05:42.382 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.383 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XhtPes5kmI6PMA9ufpMHuL+MZ3g= 2023-09-22T22:05:42.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XhtPes5kmI6PMA9ufpMHuL+MZ3g=', 'Content-Length': '0'} 2023-09-22T22:05:42.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d13defd02da2293d-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.386 DEBUG:boto:path=/ffzdqu-63/obj76 2023-09-22T22:05:42.386 DEBUG:boto:auth_path=/ffzdqu-63/obj76 2023-09-22T22:05:42.386 DEBUG:boto:path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.386 DEBUG:boto:auth_path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.386 DEBUG:boto:Method: GET 2023-09-22T22:05:42.386 DEBUG:boto:Path: /ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.386 DEBUG:boto:Data: 2023-09-22T22:05:42.386 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.386 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.386 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.387 DEBUG:boto:Params: {} 2023-09-22T22:05:42.387 DEBUG:boto:Token: None 2023-09-22T22:05:42.387 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj76?versionId=null 2023-09-22T22:05:42.387 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XhtPes5kmI6PMA9ufpMHuL+MZ3g= 2023-09-22T22:05:42.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XhtPes5kmI6PMA9ufpMHuL+MZ3g=', 'Content-Length': '0'} 2023-09-22T22:05:42.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fc2f81b735439dd-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.390 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:42.390 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:42.390 DEBUG:boto:path=/ffzdqu-63/?prefix=obj76 2023-09-22T22:05:42.390 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj76 2023-09-22T22:05:42.390 DEBUG:boto:Method: GET 2023-09-22T22:05:42.391 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj76 2023-09-22T22:05:42.391 DEBUG:boto:Data: 2023-09-22T22:05:42.391 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.391 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.391 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.391 DEBUG:boto:Params: {} 2023-09-22T22:05:42.391 DEBUG:boto:Token: None 2023-09-22T22:05:42.391 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/ 2023-09-22T22:05:42.391 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:05:42.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:05:42.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e3eaabaef7bbdc9-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.396 DEBUG:boto:b'ffzdqu-63obj761000falseobj762023-09-22T22:05:14.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:42.396 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:42.397 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:42.397 DEBUG:boto:path=/ffzdqu-63/?prefix=obj76 2023-09-22T22:05:42.397 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj76 2023-09-22T22:05:42.397 DEBUG:boto:Method: GET 2023-09-22T22:05:42.397 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj76 2023-09-22T22:05:42.397 DEBUG:boto:Data: 2023-09-22T22:05:42.397 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.397 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.397 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.397 DEBUG:boto:Params: {} 2023-09-22T22:05:42.398 DEBUG:boto:Token: None 2023-09-22T22:05:42.398 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/ 2023-09-22T22:05:42.398 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:05:42.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:05:42.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5fd959e7ee0b6e4-00650e0fb6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.408 DEBUG:boto:b'ffzdqu-63obj761000falseobj762023-09-22T22:05:14.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:42.409 DEBUG:boto:path=/ffzdqu-63/obj76 2023-09-22T22:05:42.409 DEBUG:boto:auth_path=/ffzdqu-63/obj76 2023-09-22T22:05:42.409 DEBUG:boto:Method: HEAD 2023-09-22T22:05:42.409 DEBUG:boto:Path: /ffzdqu-63/obj76 2023-09-22T22:05:42.409 DEBUG:boto:Data: 2023-09-22T22:05:42.409 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.409 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.409 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.409 DEBUG:boto:Params: {} 2023-09-22T22:05:42.410 DEBUG:boto:Token: None 2023-09-22T22:05:42.410 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj76 2023-09-22T22:05:42.410 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RlF3qM95snPe8w86eFgpkp8eTts= 2023-09-22T22:05:42.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RlF3qM95snPe8w86eFgpkp8eTts=', 'Content-Length': '0'} 2023-09-22T22:05:42.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000829df133941764e8-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.413 DEBUG:boto:path=/ffzdqu-63/obj76 2023-09-22T22:05:42.413 DEBUG:boto:auth_path=/ffzdqu-63/obj76 2023-09-22T22:05:42.413 DEBUG:boto:Method: HEAD 2023-09-22T22:05:42.413 DEBUG:boto:Path: /ffzdqu-63/obj76 2023-09-22T22:05:42.413 DEBUG:boto:Data: 2023-09-22T22:05:42.414 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.414 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.414 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.414 DEBUG:boto:Params: {} 2023-09-22T22:05:42.414 DEBUG:boto:Token: None 2023-09-22T22:05:42.414 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj76 2023-09-22T22:05:42.414 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RlF3qM95snPe8w86eFgpkp8eTts= 2023-09-22T22:05:42.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RlF3qM95snPe8w86eFgpkp8eTts=', 'Content-Length': '0'} 2023-09-22T22:05:42.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055765f2fc6ee7222-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-22T22:05:42.418 DEBUG:boto:path=/ffzdqu-63/obj76 2023-09-22T22:05:42.418 DEBUG:boto:auth_path=/ffzdqu-63/obj76 2023-09-22T22:05:42.418 DEBUG:boto:Method: GET 2023-09-22T22:05:42.418 DEBUG:boto:Path: /ffzdqu-63/obj76 2023-09-22T22:05:42.418 DEBUG:boto:Data: 2023-09-22T22:05:42.418 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.418 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.418 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.418 DEBUG:boto:Params: {} 2023-09-22T22:05:42.418 DEBUG:boto:Token: None 2023-09-22T22:05:42.419 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj76 2023-09-22T22:05:42.419 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UmkDJ7t6hLn0Uubp2FtPINP5hCg= 2023-09-22T22:05:42.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UmkDJ7t6hLn0Uubp2FtPINP5hCg=', 'Content-Length': '0'} 2023-09-22T22:05:42.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d72383635c8e594-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.466 DEBUG:boto:path=/ffzdqu-63/obj76 2023-09-22T22:05:42.467 DEBUG:boto:auth_path=/ffzdqu-63/obj76 2023-09-22T22:05:42.467 DEBUG:boto:Method: GET 2023-09-22T22:05:42.467 DEBUG:boto:Path: /ffzdqu-63/obj76 2023-09-22T22:05:42.467 DEBUG:boto:Data: 2023-09-22T22:05:42.467 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.467 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.467 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.467 DEBUG:boto:Params: {} 2023-09-22T22:05:42.467 DEBUG:boto:Token: None 2023-09-22T22:05:42.468 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj76 2023-09-22T22:05:42.468 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:UmkDJ7t6hLn0Uubp2FtPINP5hCg= 2023-09-22T22:05:42.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UmkDJ7t6hLn0Uubp2FtPINP5hCg=', 'Content-Length': '0'} 2023-09-22T22:05:42.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b2f54e1541fb448-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.471 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:42.471 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:42.471 DEBUG:boto:path=/ffzdqu-63/?prefix=obj76 2023-09-22T22:05:42.471 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj76 2023-09-22T22:05:42.471 DEBUG:boto:Method: GET 2023-09-22T22:05:42.471 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj76 2023-09-22T22:05:42.471 DEBUG:boto:Data: 2023-09-22T22:05:42.471 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.472 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.472 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.472 DEBUG:boto:Params: {} 2023-09-22T22:05:42.472 DEBUG:boto:Token: None 2023-09-22T22:05:42.472 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/ 2023-09-22T22:05:42.472 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:05:42.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:05:42.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f22fc0de71a74d2-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.477 DEBUG:boto:b'ffzdqu-63obj761000falseobj762023-09-22T22:05:14.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:42.478 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:42.478 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:42.478 DEBUG:boto:path=/ffzdqu-63/?prefix=obj76 2023-09-22T22:05:42.478 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj76 2023-09-22T22:05:42.478 DEBUG:boto:Method: GET 2023-09-22T22:05:42.478 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj76 2023-09-22T22:05:42.478 DEBUG:boto:Data: 2023-09-22T22:05:42.479 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.479 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.479 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.479 DEBUG:boto:Params: {} 2023-09-22T22:05:42.479 DEBUG:boto:Token: None 2023-09-22T22:05:42.479 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/ 2023-09-22T22:05:42.479 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:05:42.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:05:42.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d71e533cb37437a-00650e0fb6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.489 DEBUG:boto:b'ffzdqu-63obj761000falseobj762023-09-22T22:05:14.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:42.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T22:05:42.489 DEBUG:boto:path=/ffzdqu-63/obj85 2023-09-22T22:05:42.489 DEBUG:boto:auth_path=/ffzdqu-63/obj85 2023-09-22T22:05:42.490 DEBUG:boto:path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.490 DEBUG:boto:auth_path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.490 DEBUG:boto:Method: GET 2023-09-22T22:05:42.490 DEBUG:boto:Path: /ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.490 DEBUG:boto:Data: 2023-09-22T22:05:42.490 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.490 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.490 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.490 DEBUG:boto:Params: {} 2023-09-22T22:05:42.490 DEBUG:boto:Token: None 2023-09-22T22:05:42.491 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.491 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XFiKSnT2fI/EJYAJdbjDlH7zFxc= 2023-09-22T22:05:42.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XFiKSnT2fI/EJYAJdbjDlH7zFxc=', 'Content-Length': '0'} 2023-09-22T22:05:42.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a6859daa8257774-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.494 DEBUG:boto:path=/ffzdqu-63/obj85 2023-09-22T22:05:42.494 DEBUG:boto:auth_path=/ffzdqu-63/obj85 2023-09-22T22:05:42.494 DEBUG:boto:path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.494 DEBUG:boto:auth_path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.494 DEBUG:boto:Method: GET 2023-09-22T22:05:42.494 DEBUG:boto:Path: /ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.495 DEBUG:boto:Data: 2023-09-22T22:05:42.495 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.495 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.495 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.495 DEBUG:boto:Params: {} 2023-09-22T22:05:42.495 DEBUG:boto:Token: None 2023-09-22T22:05:42.495 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.495 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XFiKSnT2fI/EJYAJdbjDlH7zFxc= 2023-09-22T22:05:42.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XFiKSnT2fI/EJYAJdbjDlH7zFxc=', 'Content-Length': '0'} 2023-09-22T22:05:42.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d953d6a8b92f8ee-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.498 DEBUG:boto:path=/ffzdqu-63/obj85 2023-09-22T22:05:42.499 DEBUG:boto:auth_path=/ffzdqu-63/obj85 2023-09-22T22:05:42.499 DEBUG:boto:path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.499 DEBUG:boto:auth_path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.499 DEBUG:boto:Method: HEAD 2023-09-22T22:05:42.499 DEBUG:boto:Path: /ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.499 DEBUG:boto:Data: 2023-09-22T22:05:42.499 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.499 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.499 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.499 DEBUG:boto:Params: {} 2023-09-22T22:05:42.500 DEBUG:boto:Token: None 2023-09-22T22:05:42.500 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.500 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lEVWTlsW//gSpxH99f5L91P5uF4= 2023-09-22T22:05:42.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lEVWTlsW//gSpxH99f5L91P5uF4=', 'Content-Length': '0'} 2023-09-22T22:05:42.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c4f9a920b1d1b8b-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.503 DEBUG:boto:path=/ffzdqu-63/obj85 2023-09-22T22:05:42.503 DEBUG:boto:auth_path=/ffzdqu-63/obj85 2023-09-22T22:05:42.503 DEBUG:boto:path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.503 DEBUG:boto:auth_path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.503 DEBUG:boto:Method: HEAD 2023-09-22T22:05:42.503 DEBUG:boto:Path: /ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.503 DEBUG:boto:Data: 2023-09-22T22:05:42.504 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.504 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.504 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.504 DEBUG:boto:Params: {} 2023-09-22T22:05:42.504 DEBUG:boto:Token: None 2023-09-22T22:05:42.504 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.504 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:lEVWTlsW//gSpxH99f5L91P5uF4= 2023-09-22T22:05:42.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lEVWTlsW//gSpxH99f5L91P5uF4=', 'Content-Length': '0'} 2023-09-22T22:05:42.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000961e4470e1c80130-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T22:05:42.509 DEBUG:boto:path=/ffzdqu-63/obj85 2023-09-22T22:05:42.509 DEBUG:boto:auth_path=/ffzdqu-63/obj85 2023-09-22T22:05:42.509 DEBUG:boto:path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.509 DEBUG:boto:auth_path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.510 DEBUG:boto:Method: GET 2023-09-22T22:05:42.510 DEBUG:boto:Path: /ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.510 DEBUG:boto:Data: 2023-09-22T22:05:42.510 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.510 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.510 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.510 DEBUG:boto:Params: {} 2023-09-22T22:05:42.510 DEBUG:boto:Token: None 2023-09-22T22:05:42.510 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.511 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XFiKSnT2fI/EJYAJdbjDlH7zFxc= 2023-09-22T22:05:42.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XFiKSnT2fI/EJYAJdbjDlH7zFxc=', 'Content-Length': '0'} 2023-09-22T22:05:42.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000318e6f3436147728-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.558 DEBUG:boto:path=/ffzdqu-63/obj85 2023-09-22T22:05:42.559 DEBUG:boto:auth_path=/ffzdqu-63/obj85 2023-09-22T22:05:42.559 DEBUG:boto:path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.559 DEBUG:boto:auth_path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.559 DEBUG:boto:Method: GET 2023-09-22T22:05:42.560 DEBUG:boto:Path: /ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.560 DEBUG:boto:Data: 2023-09-22T22:05:42.560 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.560 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.560 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.560 DEBUG:boto:Params: {} 2023-09-22T22:05:42.561 DEBUG:boto:Token: None 2023-09-22T22:05:42.561 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj85?versionId=null 2023-09-22T22:05:42.561 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XFiKSnT2fI/EJYAJdbjDlH7zFxc= 2023-09-22T22:05:42.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XFiKSnT2fI/EJYAJdbjDlH7zFxc=', 'Content-Length': '0'} 2023-09-22T22:05:42.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7ac7096d579a0d5-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.565 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:42.565 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:42.565 DEBUG:boto:path=/ffzdqu-63/?prefix=obj85 2023-09-22T22:05:42.565 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj85 2023-09-22T22:05:42.566 DEBUG:boto:Method: GET 2023-09-22T22:05:42.566 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj85 2023-09-22T22:05:42.566 DEBUG:boto:Data: 2023-09-22T22:05:42.566 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.566 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.566 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.566 DEBUG:boto:Params: {} 2023-09-22T22:05:42.566 DEBUG:boto:Token: None 2023-09-22T22:05:42.567 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/ 2023-09-22T22:05:42.567 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:05:42.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:05:42.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053f1f08f920a54ae-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.573 DEBUG:boto:b'ffzdqu-63obj851000falseobj852023-09-22T22:05:14.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:42.573 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:42.573 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:42.573 DEBUG:boto:path=/ffzdqu-63/?prefix=obj85 2023-09-22T22:05:42.574 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj85 2023-09-22T22:05:42.574 DEBUG:boto:Method: GET 2023-09-22T22:05:42.574 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj85 2023-09-22T22:05:42.574 DEBUG:boto:Data: 2023-09-22T22:05:42.574 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.574 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.574 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.575 DEBUG:boto:Params: {} 2023-09-22T22:05:42.575 DEBUG:boto:Token: None 2023-09-22T22:05:42.575 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/ 2023-09-22T22:05:42.575 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:05:42.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:05:42.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b731c1dd6abe5aa-00650e0fb6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.590 DEBUG:boto:b'ffzdqu-63obj851000falseobj852023-09-22T22:05:14.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:42.591 DEBUG:boto:path=/ffzdqu-63/obj85 2023-09-22T22:05:42.591 DEBUG:boto:auth_path=/ffzdqu-63/obj85 2023-09-22T22:05:42.591 DEBUG:boto:Method: HEAD 2023-09-22T22:05:42.592 DEBUG:boto:Path: /ffzdqu-63/obj85 2023-09-22T22:05:42.592 DEBUG:boto:Data: 2023-09-22T22:05:42.592 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.592 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.593 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.593 DEBUG:boto:Params: {} 2023-09-22T22:05:42.593 DEBUG:boto:Token: None 2023-09-22T22:05:42.593 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj85 2023-09-22T22:05:42.593 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QwbVYsdT6+OATixnENPz646WkV0= 2023-09-22T22:05:42.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QwbVYsdT6+OATixnENPz646WkV0=', 'Content-Length': '0'} 2023-09-22T22:05:42.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df54bcd96b2558f0-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.596 DEBUG:boto:path=/ffzdqu-63/obj85 2023-09-22T22:05:42.596 DEBUG:boto:auth_path=/ffzdqu-63/obj85 2023-09-22T22:05:42.596 DEBUG:boto:Method: HEAD 2023-09-22T22:05:42.596 DEBUG:boto:Path: /ffzdqu-63/obj85 2023-09-22T22:05:42.597 DEBUG:boto:Data: 2023-09-22T22:05:42.597 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.597 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.597 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.597 DEBUG:boto:Params: {} 2023-09-22T22:05:42.597 DEBUG:boto:Token: None 2023-09-22T22:05:42.597 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj85 2023-09-22T22:05:42.597 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QwbVYsdT6+OATixnENPz646WkV0= 2023-09-22T22:05:42.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QwbVYsdT6+OATixnENPz646WkV0=', 'Content-Length': '0'} 2023-09-22T22:05:42.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b183eab5d148f93-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-22T22:05:42.601 DEBUG:boto:path=/ffzdqu-63/obj85 2023-09-22T22:05:42.601 DEBUG:boto:auth_path=/ffzdqu-63/obj85 2023-09-22T22:05:42.601 DEBUG:boto:Method: GET 2023-09-22T22:05:42.601 DEBUG:boto:Path: /ffzdqu-63/obj85 2023-09-22T22:05:42.601 DEBUG:boto:Data: 2023-09-22T22:05:42.601 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.601 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.601 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.602 DEBUG:boto:Params: {} 2023-09-22T22:05:42.602 DEBUG:boto:Token: None 2023-09-22T22:05:42.602 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj85 2023-09-22T22:05:42.602 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hLkorFe6avqqyK+UdVJFyaKzO5Q= 2023-09-22T22:05:42.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hLkorFe6avqqyK+UdVJFyaKzO5Q=', 'Content-Length': '0'} 2023-09-22T22:05:42.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f44d381b220bd587-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.646 DEBUG:boto:path=/ffzdqu-63/obj85 2023-09-22T22:05:42.646 DEBUG:boto:auth_path=/ffzdqu-63/obj85 2023-09-22T22:05:42.647 DEBUG:boto:Method: GET 2023-09-22T22:05:42.647 DEBUG:boto:Path: /ffzdqu-63/obj85 2023-09-22T22:05:42.647 DEBUG:boto:Data: 2023-09-22T22:05:42.647 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.647 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.647 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.647 DEBUG:boto:Params: {} 2023-09-22T22:05:42.647 DEBUG:boto:Token: None 2023-09-22T22:05:42.648 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj85 2023-09-22T22:05:42.648 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hLkorFe6avqqyK+UdVJFyaKzO5Q= 2023-09-22T22:05:42.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hLkorFe6avqqyK+UdVJFyaKzO5Q=', 'Content-Length': '0'} 2023-09-22T22:05:42.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008515c7831d6608eb-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.651 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:42.651 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:42.652 DEBUG:boto:path=/ffzdqu-63/?prefix=obj85 2023-09-22T22:05:42.652 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj85 2023-09-22T22:05:42.652 DEBUG:boto:Method: GET 2023-09-22T22:05:42.652 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj85 2023-09-22T22:05:42.652 DEBUG:boto:Data: 2023-09-22T22:05:42.652 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.652 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.652 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.653 DEBUG:boto:Params: {} 2023-09-22T22:05:42.653 DEBUG:boto:Token: None 2023-09-22T22:05:42.653 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/ 2023-09-22T22:05:42.653 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:05:42.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:05:42.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb55de93f7bbd64f-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.658 DEBUG:boto:b'ffzdqu-63obj851000falseobj852023-09-22T22:05:14.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:42.658 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:42.658 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:42.658 DEBUG:boto:path=/ffzdqu-63/?prefix=obj85 2023-09-22T22:05:42.658 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj85 2023-09-22T22:05:42.658 DEBUG:boto:Method: GET 2023-09-22T22:05:42.659 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj85 2023-09-22T22:05:42.659 DEBUG:boto:Data: 2023-09-22T22:05:42.659 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.659 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.659 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.659 DEBUG:boto:Params: {} 2023-09-22T22:05:42.659 DEBUG:boto:Token: None 2023-09-22T22:05:42.659 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/ 2023-09-22T22:05:42.660 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:05:42.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:05:42.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b9dc39856b34a36-00650e0fb6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.710 DEBUG:boto:b'ffzdqu-63obj851000falseobj852023-09-22T22:05:14.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:42.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-22T22:05:42.711 DEBUG:boto:path=/ffzdqu-63/obj87 2023-09-22T22:05:42.711 DEBUG:boto:auth_path=/ffzdqu-63/obj87 2023-09-22T22:05:42.711 DEBUG:boto:path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.711 DEBUG:boto:auth_path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.711 DEBUG:boto:Method: GET 2023-09-22T22:05:42.711 DEBUG:boto:Path: /ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.712 DEBUG:boto:Data: 2023-09-22T22:05:42.712 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.712 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.712 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.712 DEBUG:boto:Params: {} 2023-09-22T22:05:42.712 DEBUG:boto:Token: None 2023-09-22T22:05:42.712 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.712 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CKfJelByYNISJSHaqmuJh6qp0Nk= 2023-09-22T22:05:42.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CKfJelByYNISJSHaqmuJh6qp0Nk=', 'Content-Length': '0'} 2023-09-22T22:05:42.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7109c6feadd5157-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.716 DEBUG:boto:path=/ffzdqu-63/obj87 2023-09-22T22:05:42.716 DEBUG:boto:auth_path=/ffzdqu-63/obj87 2023-09-22T22:05:42.716 DEBUG:boto:path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.716 DEBUG:boto:auth_path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.716 DEBUG:boto:Method: GET 2023-09-22T22:05:42.716 DEBUG:boto:Path: /ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.716 DEBUG:boto:Data: 2023-09-22T22:05:42.717 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.717 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.717 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.717 DEBUG:boto:Params: {} 2023-09-22T22:05:42.717 DEBUG:boto:Token: None 2023-09-22T22:05:42.717 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.717 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CKfJelByYNISJSHaqmuJh6qp0Nk= 2023-09-22T22:05:42.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CKfJelByYNISJSHaqmuJh6qp0Nk=', 'Content-Length': '0'} 2023-09-22T22:05:42.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056d7a94cf5fab6ee-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.721 DEBUG:boto:path=/ffzdqu-63/obj87 2023-09-22T22:05:42.721 DEBUG:boto:auth_path=/ffzdqu-63/obj87 2023-09-22T22:05:42.721 DEBUG:boto:path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.721 DEBUG:boto:auth_path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.721 DEBUG:boto:Method: HEAD 2023-09-22T22:05:42.721 DEBUG:boto:Path: /ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.721 DEBUG:boto:Data: 2023-09-22T22:05:42.721 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.722 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.722 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.722 DEBUG:boto:Params: {} 2023-09-22T22:05:42.722 DEBUG:boto:Token: None 2023-09-22T22:05:42.722 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.722 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wtj2Nxk9QvK/yIRVIwi57snoMlY= 2023-09-22T22:05:42.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wtj2Nxk9QvK/yIRVIwi57snoMlY=', 'Content-Length': '0'} 2023-09-22T22:05:42.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000602e4d46feef14c6-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.725 DEBUG:boto:path=/ffzdqu-63/obj87 2023-09-22T22:05:42.725 DEBUG:boto:auth_path=/ffzdqu-63/obj87 2023-09-22T22:05:42.726 DEBUG:boto:path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.726 DEBUG:boto:auth_path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.726 DEBUG:boto:Method: HEAD 2023-09-22T22:05:42.726 DEBUG:boto:Path: /ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.726 DEBUG:boto:Data: 2023-09-22T22:05:42.726 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.726 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.726 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.726 DEBUG:boto:Params: {} 2023-09-22T22:05:42.727 DEBUG:boto:Token: None 2023-09-22T22:05:42.727 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.727 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wtj2Nxk9QvK/yIRVIwi57snoMlY= 2023-09-22T22:05:42.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wtj2Nxk9QvK/yIRVIwi57snoMlY=', 'Content-Length': '0'} 2023-09-22T22:05:42.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dcc488c062df2dd-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-22T22:05:42.730 DEBUG:boto:path=/ffzdqu-63/obj87 2023-09-22T22:05:42.730 DEBUG:boto:auth_path=/ffzdqu-63/obj87 2023-09-22T22:05:42.731 DEBUG:boto:path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.731 DEBUG:boto:auth_path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.731 DEBUG:boto:Method: GET 2023-09-22T22:05:42.731 DEBUG:boto:Path: /ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.731 DEBUG:boto:Data: 2023-09-22T22:05:42.731 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.731 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.731 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.731 DEBUG:boto:Params: {} 2023-09-22T22:05:42.732 DEBUG:boto:Token: None 2023-09-22T22:05:42.732 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.732 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CKfJelByYNISJSHaqmuJh6qp0Nk= 2023-09-22T22:05:42.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CKfJelByYNISJSHaqmuJh6qp0Nk=', 'Content-Length': '0'} 2023-09-22T22:05:42.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003685b716af586cdf-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.735 DEBUG:boto:path=/ffzdqu-63/obj87 2023-09-22T22:05:42.735 DEBUG:boto:auth_path=/ffzdqu-63/obj87 2023-09-22T22:05:42.735 DEBUG:boto:path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.735 DEBUG:boto:auth_path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.735 DEBUG:boto:Method: GET 2023-09-22T22:05:42.735 DEBUG:boto:Path: /ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.736 DEBUG:boto:Data: 2023-09-22T22:05:42.736 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.736 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.736 DEBUG:boto:Params: {} 2023-09-22T22:05:42.736 DEBUG:boto:Token: None 2023-09-22T22:05:42.736 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj87?versionId=null 2023-09-22T22:05:42.736 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CKfJelByYNISJSHaqmuJh6qp0Nk= 2023-09-22T22:05:42.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CKfJelByYNISJSHaqmuJh6qp0Nk=', 'Content-Length': '0'} 2023-09-22T22:05:42.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000117554cd964fff75-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.740 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:42.740 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:42.740 DEBUG:boto:path=/ffzdqu-63/?prefix=obj87 2023-09-22T22:05:42.740 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj87 2023-09-22T22:05:42.740 DEBUG:boto:Method: GET 2023-09-22T22:05:42.740 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj87 2023-09-22T22:05:42.740 DEBUG:boto:Data: 2023-09-22T22:05:42.740 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.741 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.741 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.741 DEBUG:boto:Params: {} 2023-09-22T22:05:42.741 DEBUG:boto:Token: None 2023-09-22T22:05:42.741 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/ 2023-09-22T22:05:42.741 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:05:42.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:05:42.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000440ce94247197f79-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.786 DEBUG:boto:b'ffzdqu-63obj871000falseobj872023-09-22T22:05:15.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:42.787 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:42.787 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:42.787 DEBUG:boto:path=/ffzdqu-63/?prefix=obj87 2023-09-22T22:05:42.787 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj87 2023-09-22T22:05:42.787 DEBUG:boto:Method: GET 2023-09-22T22:05:42.787 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj87 2023-09-22T22:05:42.788 DEBUG:boto:Data: 2023-09-22T22:05:42.788 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.788 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.788 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.788 DEBUG:boto:Params: {} 2023-09-22T22:05:42.788 DEBUG:boto:Token: None 2023-09-22T22:05:42.788 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/ 2023-09-22T22:05:42.788 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:05:42.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:05:42.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa858d4c41ecd0b8-00650e0fb6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.797 DEBUG:boto:b'ffzdqu-63obj871000falseobj872023-09-22T22:05:15.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:42.798 DEBUG:boto:path=/ffzdqu-63/obj87 2023-09-22T22:05:42.798 DEBUG:boto:auth_path=/ffzdqu-63/obj87 2023-09-22T22:05:42.798 DEBUG:boto:Method: HEAD 2023-09-22T22:05:42.798 DEBUG:boto:Path: /ffzdqu-63/obj87 2023-09-22T22:05:42.798 DEBUG:boto:Data: 2023-09-22T22:05:42.798 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.799 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.799 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.799 DEBUG:boto:Params: {} 2023-09-22T22:05:42.799 DEBUG:boto:Token: None 2023-09-22T22:05:42.799 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj87 2023-09-22T22:05:42.800 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Z5bmL+vtCiDJZ2LH5VR1VOZhMlk= 2023-09-22T22:05:42.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Z5bmL+vtCiDJZ2LH5VR1VOZhMlk=', 'Content-Length': '0'} 2023-09-22T22:05:42.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d5e1af4a5009030-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.803 DEBUG:boto:path=/ffzdqu-63/obj87 2023-09-22T22:05:42.804 DEBUG:boto:auth_path=/ffzdqu-63/obj87 2023-09-22T22:05:42.804 DEBUG:boto:Method: HEAD 2023-09-22T22:05:42.804 DEBUG:boto:Path: /ffzdqu-63/obj87 2023-09-22T22:05:42.804 DEBUG:boto:Data: 2023-09-22T22:05:42.804 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.804 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.804 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.804 DEBUG:boto:Params: {} 2023-09-22T22:05:42.805 DEBUG:boto:Token: None 2023-09-22T22:05:42.805 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj87 2023-09-22T22:05:42.805 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Z5bmL+vtCiDJZ2LH5VR1VOZhMlk= 2023-09-22T22:05:42.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Z5bmL+vtCiDJZ2LH5VR1VOZhMlk=', 'Content-Length': '0'} 2023-09-22T22:05:42.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c9d2f0634825791-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-22T22:05:42.808 DEBUG:boto:path=/ffzdqu-63/obj87 2023-09-22T22:05:42.808 DEBUG:boto:auth_path=/ffzdqu-63/obj87 2023-09-22T22:05:42.808 DEBUG:boto:Method: GET 2023-09-22T22:05:42.808 DEBUG:boto:Path: /ffzdqu-63/obj87 2023-09-22T22:05:42.808 DEBUG:boto:Data: 2023-09-22T22:05:42.809 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.809 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.809 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.809 DEBUG:boto:Params: {} 2023-09-22T22:05:42.809 DEBUG:boto:Token: None 2023-09-22T22:05:42.809 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj87 2023-09-22T22:05:42.809 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gWJjvC4faFx7WeQ/iK9x/p+vD8M= 2023-09-22T22:05:42.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gWJjvC4faFx7WeQ/iK9x/p+vD8M=', 'Content-Length': '0'} 2023-09-22T22:05:42.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cfefe4d1dd7ecc4-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.812 DEBUG:boto:path=/ffzdqu-63/obj87 2023-09-22T22:05:42.813 DEBUG:boto:auth_path=/ffzdqu-63/obj87 2023-09-22T22:05:42.813 DEBUG:boto:Method: GET 2023-09-22T22:05:42.813 DEBUG:boto:Path: /ffzdqu-63/obj87 2023-09-22T22:05:42.813 DEBUG:boto:Data: 2023-09-22T22:05:42.813 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.813 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.813 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.813 DEBUG:boto:Params: {} 2023-09-22T22:05:42.814 DEBUG:boto:Token: None 2023-09-22T22:05:42.814 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj87 2023-09-22T22:05:42.814 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gWJjvC4faFx7WeQ/iK9x/p+vD8M= 2023-09-22T22:05:42.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gWJjvC4faFx7WeQ/iK9x/p+vD8M=', 'Content-Length': '0'} 2023-09-22T22:05:42.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff812c1974eba416-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.858 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:42.859 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:42.859 DEBUG:boto:path=/ffzdqu-63/?prefix=obj87 2023-09-22T22:05:42.859 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj87 2023-09-22T22:05:42.859 DEBUG:boto:Method: GET 2023-09-22T22:05:42.859 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj87 2023-09-22T22:05:42.859 DEBUG:boto:Data: 2023-09-22T22:05:42.859 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.859 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.860 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.860 DEBUG:boto:Params: {} 2023-09-22T22:05:42.860 DEBUG:boto:Token: None 2023-09-22T22:05:42.860 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/ 2023-09-22T22:05:42.860 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:05:42.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:05:42.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea88e8fadacf8c43-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.866 DEBUG:boto:b'ffzdqu-63obj871000falseobj872023-09-22T22:05:15.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:42.866 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:42.866 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:42.866 DEBUG:boto:path=/ffzdqu-63/?prefix=obj87 2023-09-22T22:05:42.867 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj87 2023-09-22T22:05:42.867 DEBUG:boto:Method: GET 2023-09-22T22:05:42.867 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj87 2023-09-22T22:05:42.867 DEBUG:boto:Data: 2023-09-22T22:05:42.867 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.867 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.867 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.867 DEBUG:boto:Params: {} 2023-09-22T22:05:42.868 DEBUG:boto:Token: None 2023-09-22T22:05:42.868 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/ 2023-09-22T22:05:42.868 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:05:42.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:05:42.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fd33b2d4a58cc8a-00650e0fb6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.877 DEBUG:boto:b'ffzdqu-63obj871000falseobj872023-09-22T22:05:15.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:42.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-22T22:05:42.878 DEBUG:boto:path=/ffzdqu-63/obj92 2023-09-22T22:05:42.878 DEBUG:boto:auth_path=/ffzdqu-63/obj92 2023-09-22T22:05:42.878 DEBUG:boto:path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.878 DEBUG:boto:auth_path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.878 DEBUG:boto:Method: GET 2023-09-22T22:05:42.878 DEBUG:boto:Path: /ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.879 DEBUG:boto:Data: 2023-09-22T22:05:42.879 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.879 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.879 DEBUG:boto:Params: {} 2023-09-22T22:05:42.879 DEBUG:boto:Token: None 2023-09-22T22:05:42.879 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.879 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ek75pf0KD5sk02d9jIDkruZC414= 2023-09-22T22:05:42.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ek75pf0KD5sk02d9jIDkruZC414=', 'Content-Length': '0'} 2023-09-22T22:05:42.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d13a2bf83f54466f-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.930 DEBUG:boto:path=/ffzdqu-63/obj92 2023-09-22T22:05:42.930 DEBUG:boto:auth_path=/ffzdqu-63/obj92 2023-09-22T22:05:42.931 DEBUG:boto:path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.931 DEBUG:boto:auth_path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.931 DEBUG:boto:Method: GET 2023-09-22T22:05:42.931 DEBUG:boto:Path: /ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.931 DEBUG:boto:Data: 2023-09-22T22:05:42.931 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.931 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.931 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.931 DEBUG:boto:Params: {} 2023-09-22T22:05:42.932 DEBUG:boto:Token: None 2023-09-22T22:05:42.932 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.932 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ek75pf0KD5sk02d9jIDkruZC414= 2023-09-22T22:05:42.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ek75pf0KD5sk02d9jIDkruZC414=', 'Content-Length': '0'} 2023-09-22T22:05:42.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050f6fe454b846a3c-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.935 DEBUG:boto:path=/ffzdqu-63/obj92 2023-09-22T22:05:42.936 DEBUG:boto:auth_path=/ffzdqu-63/obj92 2023-09-22T22:05:42.936 DEBUG:boto:path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.936 DEBUG:boto:auth_path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.936 DEBUG:boto:Method: HEAD 2023-09-22T22:05:42.936 DEBUG:boto:Path: /ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.936 DEBUG:boto:Data: 2023-09-22T22:05:42.936 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.936 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.936 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.937 DEBUG:boto:Params: {} 2023-09-22T22:05:42.937 DEBUG:boto:Token: None 2023-09-22T22:05:42.937 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.937 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kUlk563M3++sOPTb0DANqmhZhZA= 2023-09-22T22:05:42.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kUlk563M3++sOPTb0DANqmhZhZA=', 'Content-Length': '0'} 2023-09-22T22:05:42.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a15dc3bda061267-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.942 DEBUG:boto:path=/ffzdqu-63/obj92 2023-09-22T22:05:42.942 DEBUG:boto:auth_path=/ffzdqu-63/obj92 2023-09-22T22:05:42.943 DEBUG:boto:path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.943 DEBUG:boto:auth_path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.943 DEBUG:boto:Method: HEAD 2023-09-22T22:05:42.943 DEBUG:boto:Path: /ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.944 DEBUG:boto:Data: 2023-09-22T22:05:42.944 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.944 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.944 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.944 DEBUG:boto:Params: {} 2023-09-22T22:05:42.944 DEBUG:boto:Token: None 2023-09-22T22:05:42.944 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.944 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kUlk563M3++sOPTb0DANqmhZhZA= 2023-09-22T22:05:42.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kUlk563M3++sOPTb0DANqmhZhZA=', 'Content-Length': '0'} 2023-09-22T22:05:42.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b42eb422fbb0c120-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-22T22:05:42.947 DEBUG:boto:path=/ffzdqu-63/obj92 2023-09-22T22:05:42.948 DEBUG:boto:auth_path=/ffzdqu-63/obj92 2023-09-22T22:05:42.948 DEBUG:boto:path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.948 DEBUG:boto:auth_path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.948 DEBUG:boto:Method: GET 2023-09-22T22:05:42.948 DEBUG:boto:Path: /ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.948 DEBUG:boto:Data: 2023-09-22T22:05:42.948 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.948 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.948 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.948 DEBUG:boto:Params: {} 2023-09-22T22:05:42.949 DEBUG:boto:Token: None 2023-09-22T22:05:42.949 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.949 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ek75pf0KD5sk02d9jIDkruZC414= 2023-09-22T22:05:42.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ek75pf0KD5sk02d9jIDkruZC414=', 'Content-Length': '0'} 2023-09-22T22:05:42.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aae509fc9a53935-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.952 DEBUG:boto:path=/ffzdqu-63/obj92 2023-09-22T22:05:42.952 DEBUG:boto:auth_path=/ffzdqu-63/obj92 2023-09-22T22:05:42.952 DEBUG:boto:path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.952 DEBUG:boto:auth_path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.952 DEBUG:boto:Method: GET 2023-09-22T22:05:42.952 DEBUG:boto:Path: /ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.952 DEBUG:boto:Data: 2023-09-22T22:05:42.953 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.953 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:42.953 DEBUG:boto:Port: 8000 2023-09-22T22:05:42.953 DEBUG:boto:Params: {} 2023-09-22T22:05:42.953 DEBUG:boto:Token: None 2023-09-22T22:05:42.953 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:42 GMT /ffzdqu-63/obj92?versionId=null 2023-09-22T22:05:42.953 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ek75pf0KD5sk02d9jIDkruZC414= 2023-09-22T22:05:42.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ek75pf0KD5sk02d9jIDkruZC414=', 'Content-Length': '0'} 2023-09-22T22:05:42.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c04738da06b152bb-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:42.998 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:42.998 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:42.999 DEBUG:boto:path=/ffzdqu-63/?prefix=obj92 2023-09-22T22:05:42.999 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj92 2023-09-22T22:05:42.999 DEBUG:boto:Method: GET 2023-09-22T22:05:42.999 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj92 2023-09-22T22:05:42.999 DEBUG:boto:Data: 2023-09-22T22:05:42.999 DEBUG:boto:Headers: {} 2023-09-22T22:05:42.999 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:42.999 DEBUG:boto:Port: 8001 2023-09-22T22:05:42.999 DEBUG:boto:Params: {} 2023-09-22T22:05:43.000 DEBUG:boto:Token: None 2023-09-22T22:05:43.000 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/ 2023-09-22T22:05:43.000 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:05:43.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:05:43.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5a8769997658bfb-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.005 DEBUG:boto:b'ffzdqu-63obj921000falseobj922023-09-22T22:05:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:43.005 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:43.005 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:43.005 DEBUG:boto:path=/ffzdqu-63/?prefix=obj92 2023-09-22T22:05:43.005 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj92 2023-09-22T22:05:43.005 DEBUG:boto:Method: GET 2023-09-22T22:05:43.006 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj92 2023-09-22T22:05:43.006 DEBUG:boto:Data: 2023-09-22T22:05:43.006 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.006 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:43.006 DEBUG:boto:Port: 8000 2023-09-22T22:05:43.006 DEBUG:boto:Params: {} 2023-09-22T22:05:43.006 DEBUG:boto:Token: None 2023-09-22T22:05:43.006 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/ 2023-09-22T22:05:43.006 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:05:43.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:05:43.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e61ebc2ebf31ef1e-00650e0fb7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.016 DEBUG:boto:b'ffzdqu-63obj921000falseobj922023-09-22T22:05:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:43.016 DEBUG:boto:path=/ffzdqu-63/obj92 2023-09-22T22:05:43.016 DEBUG:boto:auth_path=/ffzdqu-63/obj92 2023-09-22T22:05:43.016 DEBUG:boto:Method: HEAD 2023-09-22T22:05:43.016 DEBUG:boto:Path: /ffzdqu-63/obj92 2023-09-22T22:05:43.017 DEBUG:boto:Data: 2023-09-22T22:05:43.017 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.017 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:43.017 DEBUG:boto:Port: 8001 2023-09-22T22:05:43.017 DEBUG:boto:Params: {} 2023-09-22T22:05:43.017 DEBUG:boto:Token: None 2023-09-22T22:05:43.017 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj92 2023-09-22T22:05:43.017 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Xoawnu6ZJYh3bmipyFQPfw86qto= 2023-09-22T22:05:43.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Xoawnu6ZJYh3bmipyFQPfw86qto=', 'Content-Length': '0'} 2023-09-22T22:05:43.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd750d17298b9ed1-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.021 DEBUG:boto:path=/ffzdqu-63/obj92 2023-09-22T22:05:43.021 DEBUG:boto:auth_path=/ffzdqu-63/obj92 2023-09-22T22:05:43.021 DEBUG:boto:Method: HEAD 2023-09-22T22:05:43.021 DEBUG:boto:Path: /ffzdqu-63/obj92 2023-09-22T22:05:43.021 DEBUG:boto:Data: 2023-09-22T22:05:43.021 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.021 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:43.021 DEBUG:boto:Port: 8000 2023-09-22T22:05:43.021 DEBUG:boto:Params: {} 2023-09-22T22:05:43.022 DEBUG:boto:Token: None 2023-09-22T22:05:43.022 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj92 2023-09-22T22:05:43.022 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Xoawnu6ZJYh3bmipyFQPfw86qto= 2023-09-22T22:05:43.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Xoawnu6ZJYh3bmipyFQPfw86qto=', 'Content-Length': '0'} 2023-09-22T22:05:43.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dae871acb95fd40a-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-22T22:05:43.025 DEBUG:boto:path=/ffzdqu-63/obj92 2023-09-22T22:05:43.025 DEBUG:boto:auth_path=/ffzdqu-63/obj92 2023-09-22T22:05:43.025 DEBUG:boto:Method: GET 2023-09-22T22:05:43.025 DEBUG:boto:Path: /ffzdqu-63/obj92 2023-09-22T22:05:43.026 DEBUG:boto:Data: 2023-09-22T22:05:43.026 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.026 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:43.026 DEBUG:boto:Port: 8001 2023-09-22T22:05:43.026 DEBUG:boto:Params: {} 2023-09-22T22:05:43.026 DEBUG:boto:Token: None 2023-09-22T22:05:43.026 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj92 2023-09-22T22:05:43.026 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zT9I0avIVi8tvstlZYsM3LrbNTw= 2023-09-22T22:05:43.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zT9I0avIVi8tvstlZYsM3LrbNTw=', 'Content-Length': '0'} 2023-09-22T22:05:43.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eba8c361fac76430-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.070 DEBUG:boto:path=/ffzdqu-63/obj92 2023-09-22T22:05:43.071 DEBUG:boto:auth_path=/ffzdqu-63/obj92 2023-09-22T22:05:43.071 DEBUG:boto:Method: GET 2023-09-22T22:05:43.071 DEBUG:boto:Path: /ffzdqu-63/obj92 2023-09-22T22:05:43.071 DEBUG:boto:Data: 2023-09-22T22:05:43.071 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.071 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:43.071 DEBUG:boto:Port: 8000 2023-09-22T22:05:43.071 DEBUG:boto:Params: {} 2023-09-22T22:05:43.072 DEBUG:boto:Token: None 2023-09-22T22:05:43.072 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj92 2023-09-22T22:05:43.072 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:zT9I0avIVi8tvstlZYsM3LrbNTw= 2023-09-22T22:05:43.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zT9I0avIVi8tvstlZYsM3LrbNTw=', 'Content-Length': '0'} 2023-09-22T22:05:43.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feaf4b64d2445192-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.075 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:43.075 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:43.075 DEBUG:boto:path=/ffzdqu-63/?prefix=obj92 2023-09-22T22:05:43.076 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj92 2023-09-22T22:05:43.076 DEBUG:boto:Method: GET 2023-09-22T22:05:43.076 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj92 2023-09-22T22:05:43.076 DEBUG:boto:Data: 2023-09-22T22:05:43.076 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.076 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:43.076 DEBUG:boto:Port: 8001 2023-09-22T22:05:43.076 DEBUG:boto:Params: {} 2023-09-22T22:05:43.076 DEBUG:boto:Token: None 2023-09-22T22:05:43.077 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/ 2023-09-22T22:05:43.077 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:05:43.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:05:43.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081ea4d0a7328b3b0-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.082 DEBUG:boto:b'ffzdqu-63obj921000falseobj922023-09-22T22:05:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:43.082 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:43.082 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:43.082 DEBUG:boto:path=/ffzdqu-63/?prefix=obj92 2023-09-22T22:05:43.082 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj92 2023-09-22T22:05:43.082 DEBUG:boto:Method: GET 2023-09-22T22:05:43.082 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj92 2023-09-22T22:05:43.083 DEBUG:boto:Data: 2023-09-22T22:05:43.083 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.083 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:43.083 DEBUG:boto:Port: 8000 2023-09-22T22:05:43.083 DEBUG:boto:Params: {} 2023-09-22T22:05:43.083 DEBUG:boto:Token: None 2023-09-22T22:05:43.083 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/ 2023-09-22T22:05:43.083 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:05:43.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:05:43.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af92349e91331298-00650e0fb7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.134 DEBUG:boto:b'ffzdqu-63obj921000falseobj922023-09-22T22:05:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:43.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-22T22:05:43.135 DEBUG:boto:path=/ffzdqu-63/obj93 2023-09-22T22:05:43.135 DEBUG:boto:auth_path=/ffzdqu-63/obj93 2023-09-22T22:05:43.135 DEBUG:boto:path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.135 DEBUG:boto:auth_path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.135 DEBUG:boto:Method: GET 2023-09-22T22:05:43.135 DEBUG:boto:Path: /ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.135 DEBUG:boto:Data: 2023-09-22T22:05:43.136 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.136 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:43.136 DEBUG:boto:Port: 8001 2023-09-22T22:05:43.136 DEBUG:boto:Params: {} 2023-09-22T22:05:43.136 DEBUG:boto:Token: None 2023-09-22T22:05:43.136 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.136 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MJHMR+UXtiWxykgr1S7QRov7Opw= 2023-09-22T22:05:43.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MJHMR+UXtiWxykgr1S7QRov7Opw=', 'Content-Length': '0'} 2023-09-22T22:05:43.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000df549528d6cb950-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.140 DEBUG:boto:path=/ffzdqu-63/obj93 2023-09-22T22:05:43.140 DEBUG:boto:auth_path=/ffzdqu-63/obj93 2023-09-22T22:05:43.140 DEBUG:boto:path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.140 DEBUG:boto:auth_path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.140 DEBUG:boto:Method: GET 2023-09-22T22:05:43.140 DEBUG:boto:Path: /ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.140 DEBUG:boto:Data: 2023-09-22T22:05:43.140 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.140 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:43.141 DEBUG:boto:Port: 8000 2023-09-22T22:05:43.141 DEBUG:boto:Params: {} 2023-09-22T22:05:43.141 DEBUG:boto:Token: None 2023-09-22T22:05:43.141 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.141 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MJHMR+UXtiWxykgr1S7QRov7Opw= 2023-09-22T22:05:43.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MJHMR+UXtiWxykgr1S7QRov7Opw=', 'Content-Length': '0'} 2023-09-22T22:05:43.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b931e11a0e9e2aaa-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.144 DEBUG:boto:path=/ffzdqu-63/obj93 2023-09-22T22:05:43.144 DEBUG:boto:auth_path=/ffzdqu-63/obj93 2023-09-22T22:05:43.144 DEBUG:boto:path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.144 DEBUG:boto:auth_path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.144 DEBUG:boto:Method: HEAD 2023-09-22T22:05:43.145 DEBUG:boto:Path: /ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.145 DEBUG:boto:Data: 2023-09-22T22:05:43.145 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.145 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:43.145 DEBUG:boto:Port: 8001 2023-09-22T22:05:43.145 DEBUG:boto:Params: {} 2023-09-22T22:05:43.145 DEBUG:boto:Token: None 2023-09-22T22:05:43.145 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.145 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y99FCKWsNpvW/KadOooWXZJ1Xlc= 2023-09-22T22:05:43.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y99FCKWsNpvW/KadOooWXZJ1Xlc=', 'Content-Length': '0'} 2023-09-22T22:05:43.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032801624a45bcb25-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.148 DEBUG:boto:path=/ffzdqu-63/obj93 2023-09-22T22:05:43.149 DEBUG:boto:auth_path=/ffzdqu-63/obj93 2023-09-22T22:05:43.149 DEBUG:boto:path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.149 DEBUG:boto:auth_path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.149 DEBUG:boto:Method: HEAD 2023-09-22T22:05:43.149 DEBUG:boto:Path: /ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.149 DEBUG:boto:Data: 2023-09-22T22:05:43.149 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.149 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:43.149 DEBUG:boto:Port: 8000 2023-09-22T22:05:43.149 DEBUG:boto:Params: {} 2023-09-22T22:05:43.150 DEBUG:boto:Token: None 2023-09-22T22:05:43.150 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.150 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:y99FCKWsNpvW/KadOooWXZJ1Xlc= 2023-09-22T22:05:43.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y99FCKWsNpvW/KadOooWXZJ1Xlc=', 'Content-Length': '0'} 2023-09-22T22:05:43.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085b7c989b343f2f2-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-22T22:05:43.153 DEBUG:boto:path=/ffzdqu-63/obj93 2023-09-22T22:05:43.153 DEBUG:boto:auth_path=/ffzdqu-63/obj93 2023-09-22T22:05:43.154 DEBUG:boto:path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.154 DEBUG:boto:auth_path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.154 DEBUG:boto:Method: GET 2023-09-22T22:05:43.154 DEBUG:boto:Path: /ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.154 DEBUG:boto:Data: 2023-09-22T22:05:43.154 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.154 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:43.154 DEBUG:boto:Port: 8001 2023-09-22T22:05:43.154 DEBUG:boto:Params: {} 2023-09-22T22:05:43.155 DEBUG:boto:Token: None 2023-09-22T22:05:43.155 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.155 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MJHMR+UXtiWxykgr1S7QRov7Opw= 2023-09-22T22:05:43.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MJHMR+UXtiWxykgr1S7QRov7Opw=', 'Content-Length': '0'} 2023-09-22T22:05:43.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045b2d3c928126130-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.198 DEBUG:boto:path=/ffzdqu-63/obj93 2023-09-22T22:05:43.198 DEBUG:boto:auth_path=/ffzdqu-63/obj93 2023-09-22T22:05:43.199 DEBUG:boto:path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.199 DEBUG:boto:auth_path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.199 DEBUG:boto:Method: GET 2023-09-22T22:05:43.199 DEBUG:boto:Path: /ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.199 DEBUG:boto:Data: 2023-09-22T22:05:43.199 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.199 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:43.199 DEBUG:boto:Port: 8000 2023-09-22T22:05:43.199 DEBUG:boto:Params: {} 2023-09-22T22:05:43.200 DEBUG:boto:Token: None 2023-09-22T22:05:43.200 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj93?versionId=null 2023-09-22T22:05:43.200 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MJHMR+UXtiWxykgr1S7QRov7Opw= 2023-09-22T22:05:43.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MJHMR+UXtiWxykgr1S7QRov7Opw=', 'Content-Length': '0'} 2023-09-22T22:05:43.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d2789ade8e39eea-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.203 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:43.204 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:43.204 DEBUG:boto:path=/ffzdqu-63/?prefix=obj93 2023-09-22T22:05:43.204 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj93 2023-09-22T22:05:43.204 DEBUG:boto:Method: GET 2023-09-22T22:05:43.204 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj93 2023-09-22T22:05:43.204 DEBUG:boto:Data: 2023-09-22T22:05:43.204 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.204 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:43.204 DEBUG:boto:Port: 8001 2023-09-22T22:05:43.204 DEBUG:boto:Params: {} 2023-09-22T22:05:43.205 DEBUG:boto:Token: None 2023-09-22T22:05:43.205 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/ 2023-09-22T22:05:43.205 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:05:43.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:05:43.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f9db05a74d3fdcc-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.210 DEBUG:boto:b'ffzdqu-63obj931000falseobj932023-09-22T22:05:15.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:43.211 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:43.211 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:43.211 DEBUG:boto:path=/ffzdqu-63/?prefix=obj93 2023-09-22T22:05:43.211 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj93 2023-09-22T22:05:43.211 DEBUG:boto:Method: GET 2023-09-22T22:05:43.211 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj93 2023-09-22T22:05:43.212 DEBUG:boto:Data: 2023-09-22T22:05:43.212 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.212 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:43.212 DEBUG:boto:Port: 8000 2023-09-22T22:05:43.212 DEBUG:boto:Params: {} 2023-09-22T22:05:43.212 DEBUG:boto:Token: None 2023-09-22T22:05:43.212 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/ 2023-09-22T22:05:43.212 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:05:43.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:05:43.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067de43cc42f2afc0-00650e0fb7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.266 DEBUG:boto:b'ffzdqu-63obj931000falseobj932023-09-22T22:05:15.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:43.267 DEBUG:boto:path=/ffzdqu-63/obj93 2023-09-22T22:05:43.267 DEBUG:boto:auth_path=/ffzdqu-63/obj93 2023-09-22T22:05:43.267 DEBUG:boto:Method: HEAD 2023-09-22T22:05:43.267 DEBUG:boto:Path: /ffzdqu-63/obj93 2023-09-22T22:05:43.267 DEBUG:boto:Data: 2023-09-22T22:05:43.267 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.267 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:43.268 DEBUG:boto:Port: 8001 2023-09-22T22:05:43.268 DEBUG:boto:Params: {} 2023-09-22T22:05:43.268 DEBUG:boto:Token: None 2023-09-22T22:05:43.268 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj93 2023-09-22T22:05:43.268 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dtMbyFsRW47rjHEhOFQwuXz9hpM= 2023-09-22T22:05:43.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dtMbyFsRW47rjHEhOFQwuXz9hpM=', 'Content-Length': '0'} 2023-09-22T22:05:43.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054bfdb49acad5d45-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.271 DEBUG:boto:path=/ffzdqu-63/obj93 2023-09-22T22:05:43.272 DEBUG:boto:auth_path=/ffzdqu-63/obj93 2023-09-22T22:05:43.272 DEBUG:boto:Method: HEAD 2023-09-22T22:05:43.272 DEBUG:boto:Path: /ffzdqu-63/obj93 2023-09-22T22:05:43.272 DEBUG:boto:Data: 2023-09-22T22:05:43.272 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.272 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:43.272 DEBUG:boto:Port: 8000 2023-09-22T22:05:43.272 DEBUG:boto:Params: {} 2023-09-22T22:05:43.273 DEBUG:boto:Token: None 2023-09-22T22:05:43.273 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj93 2023-09-22T22:05:43.273 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:dtMbyFsRW47rjHEhOFQwuXz9hpM= 2023-09-22T22:05:43.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dtMbyFsRW47rjHEhOFQwuXz9hpM=', 'Content-Length': '0'} 2023-09-22T22:05:43.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6a2058ab3ae83b4-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-22T22:05:43.276 DEBUG:boto:path=/ffzdqu-63/obj93 2023-09-22T22:05:43.276 DEBUG:boto:auth_path=/ffzdqu-63/obj93 2023-09-22T22:05:43.276 DEBUG:boto:Method: GET 2023-09-22T22:05:43.277 DEBUG:boto:Path: /ffzdqu-63/obj93 2023-09-22T22:05:43.277 DEBUG:boto:Data: 2023-09-22T22:05:43.277 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.277 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:43.277 DEBUG:boto:Port: 8001 2023-09-22T22:05:43.277 DEBUG:boto:Params: {} 2023-09-22T22:05:43.277 DEBUG:boto:Token: None 2023-09-22T22:05:43.277 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj93 2023-09-22T22:05:43.277 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:27ZdIEbfvvbN/C67Eg/86x5tT2w= 2023-09-22T22:05:43.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:27ZdIEbfvvbN/C67Eg/86x5tT2w=', 'Content-Length': '0'} 2023-09-22T22:05:43.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027c319405451b894-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.326 DEBUG:boto:path=/ffzdqu-63/obj93 2023-09-22T22:05:43.326 DEBUG:boto:auth_path=/ffzdqu-63/obj93 2023-09-22T22:05:43.327 DEBUG:boto:Method: GET 2023-09-22T22:05:43.327 DEBUG:boto:Path: /ffzdqu-63/obj93 2023-09-22T22:05:43.327 DEBUG:boto:Data: 2023-09-22T22:05:43.327 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.327 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:43.327 DEBUG:boto:Port: 8000 2023-09-22T22:05:43.327 DEBUG:boto:Params: {} 2023-09-22T22:05:43.327 DEBUG:boto:Token: None 2023-09-22T22:05:43.328 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj93 2023-09-22T22:05:43.328 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:27ZdIEbfvvbN/C67Eg/86x5tT2w= 2023-09-22T22:05:43.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:27ZdIEbfvvbN/C67Eg/86x5tT2w=', 'Content-Length': '0'} 2023-09-22T22:05:43.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a4251d6470d2a8f-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.331 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:43.331 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:43.332 DEBUG:boto:path=/ffzdqu-63/?prefix=obj93 2023-09-22T22:05:43.332 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj93 2023-09-22T22:05:43.332 DEBUG:boto:Method: GET 2023-09-22T22:05:43.332 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj93 2023-09-22T22:05:43.332 DEBUG:boto:Data: 2023-09-22T22:05:43.332 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.332 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:43.332 DEBUG:boto:Port: 8001 2023-09-22T22:05:43.332 DEBUG:boto:Params: {} 2023-09-22T22:05:43.332 DEBUG:boto:Token: None 2023-09-22T22:05:43.333 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/ 2023-09-22T22:05:43.333 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:05:43.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:05:43.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e46988d20788bf21-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.339 DEBUG:boto:b'ffzdqu-63obj931000falseobj932023-09-22T22:05:15.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:43.339 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:43.340 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:43.340 DEBUG:boto:path=/ffzdqu-63/?prefix=obj93 2023-09-22T22:05:43.340 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj93 2023-09-22T22:05:43.340 DEBUG:boto:Method: GET 2023-09-22T22:05:43.340 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj93 2023-09-22T22:05:43.340 DEBUG:boto:Data: 2023-09-22T22:05:43.340 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.340 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:43.340 DEBUG:boto:Port: 8000 2023-09-22T22:05:43.340 DEBUG:boto:Params: {} 2023-09-22T22:05:43.341 DEBUG:boto:Token: None 2023-09-22T22:05:43.341 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/ 2023-09-22T22:05:43.341 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:05:43.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:05:43.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036e922a8ac213b79-00650e0fb7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.351 DEBUG:boto:b'ffzdqu-63obj931000falseobj932023-09-22T22:05:15.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:43.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T22:05:43.351 DEBUG:boto:path=/ffzdqu-63/obj95 2023-09-22T22:05:43.351 DEBUG:boto:auth_path=/ffzdqu-63/obj95 2023-09-22T22:05:43.351 DEBUG:boto:path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.352 DEBUG:boto:auth_path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.352 DEBUG:boto:Method: GET 2023-09-22T22:05:43.352 DEBUG:boto:Path: /ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.352 DEBUG:boto:Data: 2023-09-22T22:05:43.352 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.352 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:43.352 DEBUG:boto:Port: 8001 2023-09-22T22:05:43.352 DEBUG:boto:Params: {} 2023-09-22T22:05:43.352 DEBUG:boto:Token: None 2023-09-22T22:05:43.353 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.353 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ia92LecqngRIpRSDl1q0G510zaE= 2023-09-22T22:05:43.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ia92LecqngRIpRSDl1q0G510zaE=', 'Content-Length': '0'} 2023-09-22T22:05:43.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceda11c762259ed5-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.356 DEBUG:boto:path=/ffzdqu-63/obj95 2023-09-22T22:05:43.356 DEBUG:boto:auth_path=/ffzdqu-63/obj95 2023-09-22T22:05:43.356 DEBUG:boto:path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.356 DEBUG:boto:auth_path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.356 DEBUG:boto:Method: GET 2023-09-22T22:05:43.357 DEBUG:boto:Path: /ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.357 DEBUG:boto:Data: 2023-09-22T22:05:43.357 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.357 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:43.357 DEBUG:boto:Port: 8000 2023-09-22T22:05:43.357 DEBUG:boto:Params: {} 2023-09-22T22:05:43.357 DEBUG:boto:Token: None 2023-09-22T22:05:43.357 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.357 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ia92LecqngRIpRSDl1q0G510zaE= 2023-09-22T22:05:43.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ia92LecqngRIpRSDl1q0G510zaE=', 'Content-Length': '0'} 2023-09-22T22:05:43.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000631c134403fbd08-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.402 DEBUG:boto:path=/ffzdqu-63/obj95 2023-09-22T22:05:43.403 DEBUG:boto:auth_path=/ffzdqu-63/obj95 2023-09-22T22:05:43.403 DEBUG:boto:path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.403 DEBUG:boto:auth_path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.403 DEBUG:boto:Method: HEAD 2023-09-22T22:05:43.403 DEBUG:boto:Path: /ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.403 DEBUG:boto:Data: 2023-09-22T22:05:43.403 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.403 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:43.403 DEBUG:boto:Port: 8001 2023-09-22T22:05:43.403 DEBUG:boto:Params: {} 2023-09-22T22:05:43.404 DEBUG:boto:Token: None 2023-09-22T22:05:43.404 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.404 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bDTtTPCVZIaTFzn65i84LVySOwc= 2023-09-22T22:05:43.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bDTtTPCVZIaTFzn65i84LVySOwc=', 'Content-Length': '0'} 2023-09-22T22:05:43.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003502a5a32d87456f-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.407 DEBUG:boto:path=/ffzdqu-63/obj95 2023-09-22T22:05:43.407 DEBUG:boto:auth_path=/ffzdqu-63/obj95 2023-09-22T22:05:43.407 DEBUG:boto:path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.407 DEBUG:boto:auth_path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.407 DEBUG:boto:Method: HEAD 2023-09-22T22:05:43.407 DEBUG:boto:Path: /ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.408 DEBUG:boto:Data: 2023-09-22T22:05:43.408 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.408 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:43.408 DEBUG:boto:Port: 8000 2023-09-22T22:05:43.408 DEBUG:boto:Params: {} 2023-09-22T22:05:43.408 DEBUG:boto:Token: None 2023-09-22T22:05:43.408 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.408 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bDTtTPCVZIaTFzn65i84LVySOwc= 2023-09-22T22:05:43.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bDTtTPCVZIaTFzn65i84LVySOwc=', 'Content-Length': '0'} 2023-09-22T22:05:43.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff7e5b1f832b8f75-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T22:05:43.411 DEBUG:boto:path=/ffzdqu-63/obj95 2023-09-22T22:05:43.412 DEBUG:boto:auth_path=/ffzdqu-63/obj95 2023-09-22T22:05:43.412 DEBUG:boto:path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.412 DEBUG:boto:auth_path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.412 DEBUG:boto:Method: GET 2023-09-22T22:05:43.412 DEBUG:boto:Path: /ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.412 DEBUG:boto:Data: 2023-09-22T22:05:43.412 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.412 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:43.412 DEBUG:boto:Port: 8001 2023-09-22T22:05:43.412 DEBUG:boto:Params: {} 2023-09-22T22:05:43.412 DEBUG:boto:Token: None 2023-09-22T22:05:43.413 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.413 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ia92LecqngRIpRSDl1q0G510zaE= 2023-09-22T22:05:43.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ia92LecqngRIpRSDl1q0G510zaE=', 'Content-Length': '0'} 2023-09-22T22:05:43.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b01abe5e973224e-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.458 DEBUG:boto:path=/ffzdqu-63/obj95 2023-09-22T22:05:43.458 DEBUG:boto:auth_path=/ffzdqu-63/obj95 2023-09-22T22:05:43.458 DEBUG:boto:path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.459 DEBUG:boto:auth_path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.459 DEBUG:boto:Method: GET 2023-09-22T22:05:43.459 DEBUG:boto:Path: /ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.459 DEBUG:boto:Data: 2023-09-22T22:05:43.459 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.459 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:43.459 DEBUG:boto:Port: 8000 2023-09-22T22:05:43.459 DEBUG:boto:Params: {} 2023-09-22T22:05:43.459 DEBUG:boto:Token: None 2023-09-22T22:05:43.460 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj95?versionId=null 2023-09-22T22:05:43.460 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Ia92LecqngRIpRSDl1q0G510zaE= 2023-09-22T22:05:43.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ia92LecqngRIpRSDl1q0G510zaE=', 'Content-Length': '0'} 2023-09-22T22:05:43.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c83833b1a2b623ab-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.463 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:43.463 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:43.463 DEBUG:boto:path=/ffzdqu-63/?prefix=obj95 2023-09-22T22:05:43.464 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj95 2023-09-22T22:05:43.464 DEBUG:boto:Method: GET 2023-09-22T22:05:43.464 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj95 2023-09-22T22:05:43.464 DEBUG:boto:Data: 2023-09-22T22:05:43.464 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.464 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:43.464 DEBUG:boto:Port: 8001 2023-09-22T22:05:43.464 DEBUG:boto:Params: {} 2023-09-22T22:05:43.465 DEBUG:boto:Token: None 2023-09-22T22:05:43.465 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/ 2023-09-22T22:05:43.465 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:05:43.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:05:43.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdbb106311797da9-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.470 DEBUG:boto:b'ffzdqu-63obj951000falseobj952023-09-22T22:05:14.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:43.470 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:43.470 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:43.470 DEBUG:boto:path=/ffzdqu-63/?prefix=obj95 2023-09-22T22:05:43.471 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj95 2023-09-22T22:05:43.471 DEBUG:boto:Method: GET 2023-09-22T22:05:43.471 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj95 2023-09-22T22:05:43.471 DEBUG:boto:Data: 2023-09-22T22:05:43.471 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.471 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:43.471 DEBUG:boto:Port: 8000 2023-09-22T22:05:43.471 DEBUG:boto:Params: {} 2023-09-22T22:05:43.471 DEBUG:boto:Token: None 2023-09-22T22:05:43.471 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/ 2023-09-22T22:05:43.472 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:05:43.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:05:43.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b52cb48c3f52dd24-00650e0fb7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.482 DEBUG:boto:b'ffzdqu-63obj951000falseobj952023-09-22T22:05:14.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:43.482 DEBUG:boto:path=/ffzdqu-63/obj95 2023-09-22T22:05:43.482 DEBUG:boto:auth_path=/ffzdqu-63/obj95 2023-09-22T22:05:43.482 DEBUG:boto:Method: HEAD 2023-09-22T22:05:43.482 DEBUG:boto:Path: /ffzdqu-63/obj95 2023-09-22T22:05:43.483 DEBUG:boto:Data: 2023-09-22T22:05:43.483 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:43.483 DEBUG:boto:Port: 8001 2023-09-22T22:05:43.483 DEBUG:boto:Params: {} 2023-09-22T22:05:43.483 DEBUG:boto:Token: None 2023-09-22T22:05:43.483 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj95 2023-09-22T22:05:43.483 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:a0jh7PUag7FVXT6w50icyHMXdqE= 2023-09-22T22:05:43.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:a0jh7PUag7FVXT6w50icyHMXdqE=', 'Content-Length': '0'} 2023-09-22T22:05:43.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079fdf798b91ea29a-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.486 DEBUG:boto:path=/ffzdqu-63/obj95 2023-09-22T22:05:43.487 DEBUG:boto:auth_path=/ffzdqu-63/obj95 2023-09-22T22:05:43.487 DEBUG:boto:Method: HEAD 2023-09-22T22:05:43.487 DEBUG:boto:Path: /ffzdqu-63/obj95 2023-09-22T22:05:43.487 DEBUG:boto:Data: 2023-09-22T22:05:43.487 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.487 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:43.487 DEBUG:boto:Port: 8000 2023-09-22T22:05:43.487 DEBUG:boto:Params: {} 2023-09-22T22:05:43.487 DEBUG:boto:Token: None 2023-09-22T22:05:43.488 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj95 2023-09-22T22:05:43.488 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:a0jh7PUag7FVXT6w50icyHMXdqE= 2023-09-22T22:05:43.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:a0jh7PUag7FVXT6w50icyHMXdqE=', 'Content-Length': '0'} 2023-09-22T22:05:43.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f531060cf65234a-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-22T22:05:43.491 DEBUG:boto:path=/ffzdqu-63/obj95 2023-09-22T22:05:43.491 DEBUG:boto:auth_path=/ffzdqu-63/obj95 2023-09-22T22:05:43.491 DEBUG:boto:Method: GET 2023-09-22T22:05:43.491 DEBUG:boto:Path: /ffzdqu-63/obj95 2023-09-22T22:05:43.491 DEBUG:boto:Data: 2023-09-22T22:05:43.491 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.491 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:43.492 DEBUG:boto:Port: 8001 2023-09-22T22:05:43.492 DEBUG:boto:Params: {} 2023-09-22T22:05:43.492 DEBUG:boto:Token: None 2023-09-22T22:05:43.492 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj95 2023-09-22T22:05:43.492 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oGIKI8L+3KtzhKk95AkGl66lFxc= 2023-09-22T22:05:43.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oGIKI8L+3KtzhKk95AkGl66lFxc=', 'Content-Length': '0'} 2023-09-22T22:05:43.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c0ea2c93989a02e-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.538 DEBUG:boto:path=/ffzdqu-63/obj95 2023-09-22T22:05:43.539 DEBUG:boto:auth_path=/ffzdqu-63/obj95 2023-09-22T22:05:43.539 DEBUG:boto:Method: GET 2023-09-22T22:05:43.539 DEBUG:boto:Path: /ffzdqu-63/obj95 2023-09-22T22:05:43.539 DEBUG:boto:Data: 2023-09-22T22:05:43.539 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.539 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:43.539 DEBUG:boto:Port: 8000 2023-09-22T22:05:43.539 DEBUG:boto:Params: {} 2023-09-22T22:05:43.539 DEBUG:boto:Token: None 2023-09-22T22:05:43.540 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/obj95 2023-09-22T22:05:43.540 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:oGIKI8L+3KtzhKk95AkGl66lFxc= 2023-09-22T22:05:43.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oGIKI8L+3KtzhKk95AkGl66lFxc=', 'Content-Length': '0'} 2023-09-22T22:05:43.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af3fa6b1bab5c4c1-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.543 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:43.543 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:43.543 DEBUG:boto:path=/ffzdqu-63/?prefix=obj95 2023-09-22T22:05:43.543 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj95 2023-09-22T22:05:43.543 DEBUG:boto:Method: GET 2023-09-22T22:05:43.543 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj95 2023-09-22T22:05:43.543 DEBUG:boto:Data: 2023-09-22T22:05:43.543 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.544 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:43.544 DEBUG:boto:Port: 8001 2023-09-22T22:05:43.544 DEBUG:boto:Params: {} 2023-09-22T22:05:43.544 DEBUG:boto:Token: None 2023-09-22T22:05:43.544 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/ 2023-09-22T22:05:43.544 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:05:43.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:05:43.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005470b2c4940f9782-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.594 DEBUG:boto:b'ffzdqu-63obj951000falseobj952023-09-22T22:05:14.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:43.595 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:43.595 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:43.595 DEBUG:boto:path=/ffzdqu-63/?prefix=obj95 2023-09-22T22:05:43.595 DEBUG:boto:auth_path=/ffzdqu-63/?prefix=obj95 2023-09-22T22:05:43.595 DEBUG:boto:Method: GET 2023-09-22T22:05:43.596 DEBUG:boto:Path: /ffzdqu-63/?prefix=obj95 2023-09-22T22:05:43.596 DEBUG:boto:Data: 2023-09-22T22:05:43.596 DEBUG:boto:Headers: {} 2023-09-22T22:05:43.596 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:43.596 DEBUG:boto:Port: 8000 2023-09-22T22:05:43.596 DEBUG:boto:Params: {} 2023-09-22T22:05:43.596 DEBUG:boto:Token: None 2023-09-22T22:05:43.596 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:05:43 GMT /ffzdqu-63/ 2023-09-22T22:05:43.597 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:05:43.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:05:43.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b816606b29d1ab97-00650e0fb7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:43.607 DEBUG:boto:b'ffzdqu-63obj951000falseobj952023-09-22T22:05:14.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:05:43.607 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-63 zones={a2, a1} 2023-09-22T22:05:43.607 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-09-22T22:05:43.608 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T22:05:43.609 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T22:05:43.607+0000 7f7bf5126640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 83780) UID: 0 2023-09-22T22:05:43.609 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T22:05:43.607+0000 7f7bf896da40 -1 shutting down 2023-09-22T22:05:45.425 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-09-22T22:05:49.707 INFO:tasks.rgw.c1.client.1:Stopped 2023-09-22T22:05:49.707 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:49.707 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:49.708 DEBUG:boto:Method: HEAD 2023-09-22T22:05:49.708 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:49.708 DEBUG:boto:Data: 2023-09-22T22:05:49.708 DEBUG:boto:Headers: {} 2023-09-22T22:05:49.708 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:49.708 DEBUG:boto:Port: 8000 2023-09-22T22:05:49.708 DEBUG:boto:Params: {} 2023-09-22T22:05:49.708 DEBUG:boto:Token: None 2023-09-22T22:05:49.709 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:49 GMT /ffzdqu-63/ 2023-09-22T22:05:49.709 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rT0I7C+K3yRSjyCLsgXGysOO0b8= 2023-09-22T22:05:49.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rT0I7C+K3yRSjyCLsgXGysOO0b8=', 'Content-Length': '0'} 2023-09-22T22:05:49.716 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000968603b916a7246a-00650e0fbd-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:49.717 DEBUG:boto:path=/ffzdqu-63/obj37 2023-09-22T22:05:49.717 DEBUG:boto:auth_path=/ffzdqu-63/obj37 2023-09-22T22:05:49.717 DEBUG:boto:Method: PUT 2023-09-22T22:05:49.717 DEBUG:boto:Path: /ffzdqu-63/obj37 2023-09-22T22:05:49.717 DEBUG:boto:Data: 2023-09-22T22:05:49.717 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:49.718 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:49.718 DEBUG:boto:Port: 8000 2023-09-22T22:05:49.718 DEBUG:boto:Params: {} 2023-09-22T22:05:49.718 DEBUG:boto:Token: None 2023-09-22T22:05:49.718 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:49 GMT /ffzdqu-63/obj37 2023-09-22T22:05:49.718 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MBOmitfRUaiQ7ahB1puT6IgiQgg= 2023-09-22T22:05:49.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MBOmitfRUaiQ7ahB1puT6IgiQgg='} 2023-09-22T22:05:49.736 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003aea74f3019ee6f8-00650e0fbd-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:49.736 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:49.737 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:49.737 DEBUG:boto:Method: HEAD 2023-09-22T22:05:49.737 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:49.737 DEBUG:boto:Data: 2023-09-22T22:05:49.737 DEBUG:boto:Headers: {} 2023-09-22T22:05:49.737 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:49.737 DEBUG:boto:Port: 8000 2023-09-22T22:05:49.737 DEBUG:boto:Params: {} 2023-09-22T22:05:49.737 DEBUG:boto:Token: None 2023-09-22T22:05:49.738 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:49 GMT /ffzdqu-63/ 2023-09-22T22:05:49.738 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rT0I7C+K3yRSjyCLsgXGysOO0b8= 2023-09-22T22:05:49.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rT0I7C+K3yRSjyCLsgXGysOO0b8=', 'Content-Length': '0'} 2023-09-22T22:05:49.746 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037d8d61a9a3e2fbd-00650e0fbd-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:49.746 DEBUG:boto:path=/ffzdqu-63/obj74 2023-09-22T22:05:49.746 DEBUG:boto:auth_path=/ffzdqu-63/obj74 2023-09-22T22:05:49.746 DEBUG:boto:Method: PUT 2023-09-22T22:05:49.746 DEBUG:boto:Path: /ffzdqu-63/obj74 2023-09-22T22:05:49.746 DEBUG:boto:Data: 2023-09-22T22:05:49.746 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:49.747 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:49.747 DEBUG:boto:Port: 8000 2023-09-22T22:05:49.747 DEBUG:boto:Params: {} 2023-09-22T22:05:49.747 DEBUG:boto:Token: None 2023-09-22T22:05:49.747 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:49 GMT /ffzdqu-63/obj74 2023-09-22T22:05:49.747 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:d43Ajibp+jx8/SOOBEYtaS65C4w= 2023-09-22T22:05:49.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:d43Ajibp+jx8/SOOBEYtaS65C4w='} 2023-09-22T22:05:49.790 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000097b4c4595abcf586-00650e0fbd-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:49.791 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:49.791 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:49.791 DEBUG:boto:Method: HEAD 2023-09-22T22:05:49.791 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:49.791 DEBUG:boto:Data: 2023-09-22T22:05:49.791 DEBUG:boto:Headers: {} 2023-09-22T22:05:49.791 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:49.792 DEBUG:boto:Port: 8000 2023-09-22T22:05:49.792 DEBUG:boto:Params: {} 2023-09-22T22:05:49.792 DEBUG:boto:Token: None 2023-09-22T22:05:49.792 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:49 GMT /ffzdqu-63/ 2023-09-22T22:05:49.792 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rT0I7C+K3yRSjyCLsgXGysOO0b8= 2023-09-22T22:05:49.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rT0I7C+K3yRSjyCLsgXGysOO0b8=', 'Content-Length': '0'} 2023-09-22T22:05:49.800 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b24205224fb8e782-00650e0fbd-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:49.800 DEBUG:boto:path=/ffzdqu-63/obj111 2023-09-22T22:05:49.800 DEBUG:boto:auth_path=/ffzdqu-63/obj111 2023-09-22T22:05:49.800 DEBUG:boto:Method: PUT 2023-09-22T22:05:49.800 DEBUG:boto:Path: /ffzdqu-63/obj111 2023-09-22T22:05:49.800 DEBUG:boto:Data: 2023-09-22T22:05:49.801 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:49.801 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:49.801 DEBUG:boto:Port: 8000 2023-09-22T22:05:49.801 DEBUG:boto:Params: {} 2023-09-22T22:05:49.801 DEBUG:boto:Token: None 2023-09-22T22:05:49.801 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:49 GMT /ffzdqu-63/obj111 2023-09-22T22:05:49.801 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:79RWPyslkGteYGbPoiGUZX1JTh4= 2023-09-22T22:05:49.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:79RWPyslkGteYGbPoiGUZX1JTh4='} 2023-09-22T22:05:49.819 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006e3f0f09a2edee92-00650e0fbd-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:49.820 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:49.820 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:49.820 DEBUG:boto:Method: HEAD 2023-09-22T22:05:49.820 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:49.820 DEBUG:boto:Data: 2023-09-22T22:05:49.820 DEBUG:boto:Headers: {} 2023-09-22T22:05:49.820 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:49.820 DEBUG:boto:Port: 8000 2023-09-22T22:05:49.820 DEBUG:boto:Params: {} 2023-09-22T22:05:49.821 DEBUG:boto:Token: None 2023-09-22T22:05:49.821 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:49 GMT /ffzdqu-63/ 2023-09-22T22:05:49.821 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rT0I7C+K3yRSjyCLsgXGysOO0b8= 2023-09-22T22:05:49.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rT0I7C+K3yRSjyCLsgXGysOO0b8=', 'Content-Length': '0'} 2023-09-22T22:05:49.828 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092cafe344ec4baca-00650e0fbd-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:49.828 DEBUG:boto:path=/ffzdqu-63/obj148 2023-09-22T22:05:49.828 DEBUG:boto:auth_path=/ffzdqu-63/obj148 2023-09-22T22:05:49.829 DEBUG:boto:Method: PUT 2023-09-22T22:05:49.829 DEBUG:boto:Path: /ffzdqu-63/obj148 2023-09-22T22:05:49.829 DEBUG:boto:Data: 2023-09-22T22:05:49.829 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:49.829 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:49.829 DEBUG:boto:Port: 8000 2023-09-22T22:05:49.829 DEBUG:boto:Params: {} 2023-09-22T22:05:49.829 DEBUG:boto:Token: None 2023-09-22T22:05:49.830 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:49 GMT /ffzdqu-63/obj148 2023-09-22T22:05:49.830 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:XSMuiUUrf93SZDCqnYPHpvX/Vlo= 2023-09-22T22:05:49.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XSMuiUUrf93SZDCqnYPHpvX/Vlo='} 2023-09-22T22:05:49.874 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005907c3d0b573ef55-00650e0fbd-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:49.875 DEBUG:boto:path=/ffzdqu-63/ 2023-09-22T22:05:49.875 DEBUG:boto:auth_path=/ffzdqu-63/ 2023-09-22T22:05:49.875 DEBUG:boto:Method: HEAD 2023-09-22T22:05:49.875 DEBUG:boto:Path: /ffzdqu-63/ 2023-09-22T22:05:49.875 DEBUG:boto:Data: 2023-09-22T22:05:49.875 DEBUG:boto:Headers: {} 2023-09-22T22:05:49.875 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:49.875 DEBUG:boto:Port: 8000 2023-09-22T22:05:49.875 DEBUG:boto:Params: {} 2023-09-22T22:05:49.876 DEBUG:boto:Token: None 2023-09-22T22:05:49.876 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:05:49 GMT /ffzdqu-63/ 2023-09-22T22:05:49.876 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:rT0I7C+K3yRSjyCLsgXGysOO0b8= 2023-09-22T22:05:49.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rT0I7C+K3yRSjyCLsgXGysOO0b8=', 'Content-Length': '0'} 2023-09-22T22:05:49.884 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc95a2fa55aabea9-00650e0fbd-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:49.884 DEBUG:boto:path=/ffzdqu-63/obj185 2023-09-22T22:05:49.884 DEBUG:boto:auth_path=/ffzdqu-63/obj185 2023-09-22T22:05:49.884 DEBUG:boto:Method: PUT 2023-09-22T22:05:49.884 DEBUG:boto:Path: /ffzdqu-63/obj185 2023-09-22T22:05:49.885 DEBUG:boto:Data: 2023-09-22T22:05:49.885 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:05:49.885 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:05:49.885 DEBUG:boto:Port: 8000 2023-09-22T22:05:49.885 DEBUG:boto:Params: {} 2023-09-22T22:05:49.885 DEBUG:boto:Token: None 2023-09-22T22:05:49.885 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:05:49 GMT /ffzdqu-63/obj185 2023-09-22T22:05:49.885 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G02xL4IoMYhyvGUMTA0QEA0D3mk= 2023-09-22T22:05:49.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G02xL4IoMYhyvGUMTA0QEA0D3mk='} 2023-09-22T22:05:49.903 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055c526d52dddd8ca-00650e0fbd-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:05:49.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:05:49.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:05:49.904 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:05:49.947 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:05:49.947 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:05:49.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:05:49.982+0000 7f6149764a40 5 RGW-SYNC:data: waiting on data sync status lease 2023-09-22T22:05:49.987 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:05:49.986+0000 7f6149764a40 5 RGW-SYNC:data: acquired data sync status lease 2023-09-22T22:05:50.015 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-22T22:05:50.015 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-22T22:05:50.060 INFO:tasks.rgw.c1.client.1:Started 2023-09-22T22:05:50.060 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:50.069 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:05:50.069 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T22:05:50.069 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T22:05:50.070 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T22:05:50.071 INFO:teuthology.orchestra.run.smithi160.stderr:curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T22:05:51.071 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:05:51.083 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:05:51.083 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T22:05:51.086 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 60554 0 --:--:-- --:--:-- --:--:-- 71333 2023-09-22T22:05:51.087 INFO:teuthology.orchestra.run.smithi160.stdout:anonymous 2023-09-22T22:05:51.087 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-63 2023-09-22T22:05:51.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:05:51.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:05:51.088 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T22:05:51.130 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-09-22T22:05:51.178 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:05:51.178 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:05:51.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:05:51.222+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:05:51.226 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:05:51.226 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:05:53.122 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T22:05:53.122+0000 7f2d4a614640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T22:05:56.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:05:56.230+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:05:56.232 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:05:56.232 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:06:01.237 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:06:01.234+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:06:01.238 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:06:01.238 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:06:06.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:06:06.242+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:06:06.243 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:06:06.243 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:06:11.248 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:06:11.246+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:06:11.249 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:06:11.249 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:06:16.254 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:06:16.250+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:06:16.254 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:06:16.254 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:06:21.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:06:21.258+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:06:21.260 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:06:21.260 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:06:26.265 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:06:26.262+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:06:26.265 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:06:26.266 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:06:31.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:06:31.270+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:06:31.271 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:06:31.271 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:06:36.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:06:36.274+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:06:36.277 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:06:36.277 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:06:41.282 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:06:41.278+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:06:41.282 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:06:41.282 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:06:46.287 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:06:46.286+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:06:46.288 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:06:46.288 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:06:51.293 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:06:51.290+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:06:51.336 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:06:51.336 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:06:56.298 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:06:56.298+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:06:56.299 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:06:56.299 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:07:01.304 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:07:01.302+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:07:01.304 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:07:01.305 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:07:06.310 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:07:06.305+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:07:06.310 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:07:06.310 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:07:11.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:07:11.313+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:07:11.315 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:07:11.316 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:07:16.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:07:16.317+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:07:16.321 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:07:16.321 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:07:21.326 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:07:21.329+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:07:21.327 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:07:21.327 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:07:26.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:07:26.333+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:07:26.332 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:07:26.332 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:07:31.338 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:07:31.341+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:07:31.338 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:07:31.339 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:07:36.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:07:36.345+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:07:36.344 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:07:36.344 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:07:41.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:07:41.349+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:07:41.351 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:07:41.351 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:07:46.355 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:07:46.357+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:07:46.355 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:07:46.356 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:07:51.361 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:07:51.361+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:07:51.361 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:07:51.361 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:07:56.369 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:07:56.369+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:07:56.369 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:07:56.369 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:08:01.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:08:01.377+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:08:01.376 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:08:01.376 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:08:06.382 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:08:06.381+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:08:06.382 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:08:06.383 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:08:11.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:08:11.389+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:08:11.388 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:08:11.388 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:08:16.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:08:16.392+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:08:16.393 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:08:16.393 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:08:21.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:08:21.400+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:08:21.401 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:08:21.401 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:08:26.407 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:08:26.408+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:08:26.407 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:08:26.407 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:08:31.412 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:08:31.412+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:08:31.413 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:08:31.413 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:08:36.418 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:08:36.416+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:08:36.418 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:08:36.418 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:08:41.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:08:41.424+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:08:41.423 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:08:41.423 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:08:46.430 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:08:46.428+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:08:46.430 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:08:46.430 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:08:51.435 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:08:51.436+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:08:51.435 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:08:51.435 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:08:56.441 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:08:56.440+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:08:56.441 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:08:56.441 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:09:01.446 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:09:01.448+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:09:01.447 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:09:01.447 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:09:06.452 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:09:06.452+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:09:06.452 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:09:06.452 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:09:11.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:09:11.460+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:09:11.459 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:09:11.459 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:09:16.465 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:09:16.464+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:09:16.465 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:09:16.465 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:09:21.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:09:21.468+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:09:21.470 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:09:21.471 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:09:26.476 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:09:26.476+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:09:26.476 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:09:26.476 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:09:31.481 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:09:31.479+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:09:31.482 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:09:31.482 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:09:36.489 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:09:36.487+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:09:36.490 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:09:36.490 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:09:41.495 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:09:41.495+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:09:41.495 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:09:41.495 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:09:46.500 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:09:46.499+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:09:46.501 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:09:46.501 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:09:51.508 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:09:51.507+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:09:51.508 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:09:51.508 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:09:56.513 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:09:56.511+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:09:56.513 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:09:56.514 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:10:01.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:10:01.519+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:10:01.519 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:10:01.519 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:10:06.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:10:06.523+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:10:06.524 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:10:06.524 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:10:11.529 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:10:11.527+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:10:11.530 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:10:11.530 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:10:16.535 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:10:16.535+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:10:16.535 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:10:16.535 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:10:21.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:10:21.539+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:10:21.541 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:10:21.541 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:10:26.547 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:10:26.547+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:10:26.547 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:10:26.547 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:10:31.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:10:31.551+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:10:31.553 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:10:31.553 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:10:36.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:10:36.559+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:10:36.559 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:10:36.560 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:10:41.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:10:41.563+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:10:41.565 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:10:41.565 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:10:46.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:10:46.566+0000 7f34ca72ca40 0 bucket checkpoint timed out waiting for incremental sync to catch up 2023-09-22T22:10:46.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:10:46.566+0000 7f34ca72ca40 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-22T22:10:46.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:10:46.566+0000 7f34ca72ca40 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-22T22:10:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-09-22T22:10:46.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2023-09-22T22:10:46.610 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:10:46.611 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:10:46.611 DEBUG:boto:Method: PUT 2023-09-22T22:10:46.611 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:10:46.611 DEBUG:boto:Data: 2023-09-22T22:10:46.611 DEBUG:boto:Headers: {} 2023-09-22T22:10:46.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:10:46.611 DEBUG:boto:Port: 8000 2023-09-22T22:10:46.611 DEBUG:boto:Params: {} 2023-09-22T22:10:46.612 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T22:10:46.612 DEBUG:boto:Token: None 2023-09-22T22:10:46.612 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 22:10:46 GMT /ffzdqu-64/ 2023-09-22T22:10:46.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CVrBAGWAaMdXd+MXHcUiAhbZSNw= 2023-09-22T22:10:46.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:10:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CVrBAGWAaMdXd+MXHcUiAhbZSNw=', 'Content-Length': '0'} 2023-09-22T22:10:46.645 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f9dd883816c1da9e-00650e10e6-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:10:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:10:46.645 DEBUG:rgw_multi.tests:created bucket=ffzdqu-64 2023-09-22T22:10:46.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:10:46.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:10:46.646 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:10:46.686 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:10:46.686 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:10:46.725 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T22:10:46.745 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"1_1695419384.000133_332.1","last_update":"2023-09-22T21:49:44.000133Z"},{"marker":"1_1695420646.644925_651.1","last_update":"2023-09-22T22:10:46.644925Z"},{"marker":"1_1695419383.971498_240.1","last_update":"2023-09-22T21:49:43.971498Z"}] 2023-09-22T22:10:46.745 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420646.644925_651.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:10:46.745 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T22:10:46.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:10:46.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:10:46.746 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:10:46.826 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:10:46.826 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:10:46.864 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:10:46.862+0000 7f97e2059a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:10:46.865 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:10:46.862+0000 7f97e2059a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:10:46.883 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420646.633184_648.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:10:46.633184Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:10:46Z"} 2023-09-22T22:10:46.884 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420646.633184_648.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:10:46.633184Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:10:46Z"} 2023-09-22T22:10:46.884 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420646.633184_648.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:10:46.633184Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:10:46.884 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420646.644925_651.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:10:46.884 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420646.633184_648.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:10:46.884 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695420646.644925_651.1 target=1_1695420646.633184_648.1 2023-09-22T22:10:51.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:10:51.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:10:51.886 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:10:51.934 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:10:51.934 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:10:51.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:10:51.974+0000 7faef7ccba40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:10:51.976 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:10:51.974+0000 7faef7ccba40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:10:51.995 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420646.644925_651.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:10:46.644925Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:10:51Z"} 2023-09-22T22:10:51.995 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420646.644925_651.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:10:46.644925Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:10:51Z"} 2023-09-22T22:10:51.995 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420646.644925_651.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:10:46.644925Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:10:51.996 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420646.644925_651.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:10:51.996 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420646.644925_651.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:10:51.996 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T22:10:51.996 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:10:51.996 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:10:51.996 DEBUG:boto:Method: HEAD 2023-09-22T22:10:51.996 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:10:51.997 DEBUG:boto:Data: 2023-09-22T22:10:51.997 DEBUG:boto:Headers: {} 2023-09-22T22:10:51.997 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:10:51.997 DEBUG:boto:Port: 8000 2023-09-22T22:10:51.997 DEBUG:boto:Params: {} 2023-09-22T22:10:51.997 DEBUG:boto:Token: None 2023-09-22T22:10:51.997 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:10:51 GMT /ffzdqu-64/ 2023-09-22T22:10:51.997 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Z6slShZpeKfl90AtKNO/Ej/+b/I= 2023-09-22T22:10:51.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:10:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Z6slShZpeKfl90AtKNO/Ej/+b/I=', 'Content-Length': '0'} 2023-09-22T22:10:52.000 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfdee417852908f0-00650e10eb-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:10:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:10:52.000 DEBUG:boto:path=/ffzdqu-64/obj11 2023-09-22T22:10:52.001 DEBUG:boto:auth_path=/ffzdqu-64/obj11 2023-09-22T22:10:52.001 DEBUG:boto:Method: PUT 2023-09-22T22:10:52.001 DEBUG:boto:Path: /ffzdqu-64/obj11 2023-09-22T22:10:52.001 DEBUG:boto:Data: 2023-09-22T22:10:52.001 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:10:52.001 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:10:52.001 DEBUG:boto:Port: 8000 2023-09-22T22:10:52.001 DEBUG:boto:Params: {} 2023-09-22T22:10:52.001 DEBUG:boto:Token: None 2023-09-22T22:10:52.002 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:10:52 GMT /ffzdqu-64/obj11 2023-09-22T22:10:52.002 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:AfUO7EktB8KLz4WJe2mYlBkZ22g= 2023-09-22T22:10:52.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:10:52 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AfUO7EktB8KLz4WJe2mYlBkZ22g='} 2023-09-22T22:10:52.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000166a03fd3ab0386b-00650e10ec-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:10:52.010 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:10:52.010 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:10:52.011 DEBUG:boto:Method: HEAD 2023-09-22T22:10:52.011 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:10:52.011 DEBUG:boto:Data: 2023-09-22T22:10:52.011 DEBUG:boto:Headers: {} 2023-09-22T22:10:52.011 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:10:52.011 DEBUG:boto:Port: 8000 2023-09-22T22:10:52.011 DEBUG:boto:Params: {} 2023-09-22T22:10:52.011 DEBUG:boto:Token: None 2023-09-22T22:10:52.011 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:10:52 GMT /ffzdqu-64/ 2023-09-22T22:10:52.012 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3/A/+jKsD2yNrKEJV2T0CSQSx6A= 2023-09-22T22:10:52.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:10:52 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3/A/+jKsD2yNrKEJV2T0CSQSx6A=', 'Content-Length': '0'} 2023-09-22T22:10:52.013 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f4a5859f6b3e959-00650e10ec-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:10:52.014 DEBUG:boto:path=/ffzdqu-64/obj22 2023-09-22T22:10:52.014 DEBUG:boto:auth_path=/ffzdqu-64/obj22 2023-09-22T22:10:52.014 DEBUG:boto:Method: PUT 2023-09-22T22:10:52.014 DEBUG:boto:Path: /ffzdqu-64/obj22 2023-09-22T22:10:52.014 DEBUG:boto:Data: 2023-09-22T22:10:52.014 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:10:52.015 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:10:52.015 DEBUG:boto:Port: 8000 2023-09-22T22:10:52.015 DEBUG:boto:Params: {} 2023-09-22T22:10:52.015 DEBUG:boto:Token: None 2023-09-22T22:10:52.015 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:10:52 GMT /ffzdqu-64/obj22 2023-09-22T22:10:52.015 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:/jSv1AmuZr4EQkURslnSHbyqFBQ= 2023-09-22T22:10:52.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:10:52 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/jSv1AmuZr4EQkURslnSHbyqFBQ='} 2023-09-22T22:10:52.019 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c47d6e4efbb7a9df-00650e10ec-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:10:52.020 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:10:52.020 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:10:52.020 DEBUG:boto:Method: HEAD 2023-09-22T22:10:52.020 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:10:52.020 DEBUG:boto:Data: 2023-09-22T22:10:52.020 DEBUG:boto:Headers: {} 2023-09-22T22:10:52.020 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:10:52.020 DEBUG:boto:Port: 8000 2023-09-22T22:10:52.020 DEBUG:boto:Params: {} 2023-09-22T22:10:52.021 DEBUG:boto:Token: None 2023-09-22T22:10:52.021 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:10:52 GMT /ffzdqu-64/ 2023-09-22T22:10:52.021 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3/A/+jKsD2yNrKEJV2T0CSQSx6A= 2023-09-22T22:10:52.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:10:52 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3/A/+jKsD2yNrKEJV2T0CSQSx6A=', 'Content-Length': '0'} 2023-09-22T22:10:52.023 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000220629b97d22a251-00650e10ec-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:10:52.023 DEBUG:boto:path=/ffzdqu-64/obj33 2023-09-22T22:10:52.023 DEBUG:boto:auth_path=/ffzdqu-64/obj33 2023-09-22T22:10:52.023 DEBUG:boto:Method: PUT 2023-09-22T22:10:52.024 DEBUG:boto:Path: /ffzdqu-64/obj33 2023-09-22T22:10:52.024 DEBUG:boto:Data: 2023-09-22T22:10:52.024 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:10:52.024 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:10:52.024 DEBUG:boto:Port: 8000 2023-09-22T22:10:52.024 DEBUG:boto:Params: {} 2023-09-22T22:10:52.024 DEBUG:boto:Token: None 2023-09-22T22:10:52.024 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:10:52 GMT /ffzdqu-64/obj33 2023-09-22T22:10:52.024 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VpMRv+3K5JePX/aPQNYw8OQfGFQ= 2023-09-22T22:10:52.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:10:52 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VpMRv+3K5JePX/aPQNYw8OQfGFQ='} 2023-09-22T22:10:52.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091899caa7cf4e99c-00650e10ec-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:10:52.071 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:10:52.071 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:10:52.071 DEBUG:boto:Method: HEAD 2023-09-22T22:10:52.071 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:10:52.071 DEBUG:boto:Data: 2023-09-22T22:10:52.071 DEBUG:boto:Headers: {} 2023-09-22T22:10:52.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:10:52.072 DEBUG:boto:Port: 8000 2023-09-22T22:10:52.072 DEBUG:boto:Params: {} 2023-09-22T22:10:52.072 DEBUG:boto:Token: None 2023-09-22T22:10:52.072 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:10:52 GMT /ffzdqu-64/ 2023-09-22T22:10:52.072 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3/A/+jKsD2yNrKEJV2T0CSQSx6A= 2023-09-22T22:10:52.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:10:52 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3/A/+jKsD2yNrKEJV2T0CSQSx6A=', 'Content-Length': '0'} 2023-09-22T22:10:52.075 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a89f8c1ae2d30af5-00650e10ec-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:10:52.076 DEBUG:boto:path=/ffzdqu-64/obj44 2023-09-22T22:10:52.076 DEBUG:boto:auth_path=/ffzdqu-64/obj44 2023-09-22T22:10:52.076 DEBUG:boto:Method: PUT 2023-09-22T22:10:52.076 DEBUG:boto:Path: /ffzdqu-64/obj44 2023-09-22T22:10:52.076 DEBUG:boto:Data: 2023-09-22T22:10:52.076 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:10:52.077 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:10:52.077 DEBUG:boto:Port: 8000 2023-09-22T22:10:52.077 DEBUG:boto:Params: {} 2023-09-22T22:10:52.077 DEBUG:boto:Token: None 2023-09-22T22:10:52.077 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:10:52 GMT /ffzdqu-64/obj44 2023-09-22T22:10:52.077 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0ASaUJR6s6qUhGad9kceN3r3QK8= 2023-09-22T22:10:52.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:10:52 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0ASaUJR6s6qUhGad9kceN3r3QK8='} 2023-09-22T22:10:52.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008cc88a2a50ae0bbc-00650e10ec-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:10:52.094 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:10:52.094 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:10:52.094 DEBUG:boto:Method: HEAD 2023-09-22T22:10:52.094 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:10:52.095 DEBUG:boto:Data: 2023-09-22T22:10:52.095 DEBUG:boto:Headers: {} 2023-09-22T22:10:52.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:10:52.095 DEBUG:boto:Port: 8000 2023-09-22T22:10:52.095 DEBUG:boto:Params: {} 2023-09-22T22:10:52.095 DEBUG:boto:Token: None 2023-09-22T22:10:52.095 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:10:52 GMT /ffzdqu-64/ 2023-09-22T22:10:52.095 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3/A/+jKsD2yNrKEJV2T0CSQSx6A= 2023-09-22T22:10:52.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:10:52 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3/A/+jKsD2yNrKEJV2T0CSQSx6A=', 'Content-Length': '0'} 2023-09-22T22:10:52.099 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe6e7f39170f0a3d-00650e10ec-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:10:52.099 DEBUG:boto:path=/ffzdqu-64/obj55 2023-09-22T22:10:52.100 DEBUG:boto:auth_path=/ffzdqu-64/obj55 2023-09-22T22:10:52.100 DEBUG:boto:Method: PUT 2023-09-22T22:10:52.100 DEBUG:boto:Path: /ffzdqu-64/obj55 2023-09-22T22:10:52.100 DEBUG:boto:Data: 2023-09-22T22:10:52.100 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:10:52.100 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:10:52.100 DEBUG:boto:Port: 8000 2023-09-22T22:10:52.100 DEBUG:boto:Params: {} 2023-09-22T22:10:52.100 DEBUG:boto:Token: None 2023-09-22T22:10:52.101 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:10:52 GMT /ffzdqu-64/obj55 2023-09-22T22:10:52.101 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bIMgps7X0pCjRAepVC8u6tKLNXM= 2023-09-22T22:10:52.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:10:52 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bIMgps7X0pCjRAepVC8u6tKLNXM='} 2023-09-22T22:10:52.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b170bbfffadf91e-00650e10ec-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:10:52.147 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-64 2023-09-22T22:10:52.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:10:52.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:10:52.147 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T22:10:52.197 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:10:52.197 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:10:52.240 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:10:52.238+0000 7f43099e3a40 1 waiting to reach incremental sync.. 2023-09-22T22:10:53.127 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T22:10:53.126+0000 7f2d4a614640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T22:10:57.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:10:57.242+0000 7f43099e3a40 1 bucket sync caught up with source: 2023-09-22T22:10:57.243 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.107.6, 00000000002.84.6, 00000000002.115.6] 2023-09-22T22:10:57.243 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.107.6, 00000000002.84.6, 00000000002.115.6] 2023-09-22T22:10:57.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:10:57.242+0000 7f43099e3a40 0 bucket checkpoint complete 2023-09-22T22:10:57.278 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-64 2023-09-22T22:10:57.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:10:57.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:10:57.279 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T22:10:57.327 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:10:57.327 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:10:57.370 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:10:57.370+0000 7f63e8855a40 1 waiting to reach incremental sync.. 2023-09-22T22:11:02.371 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:11:02.370+0000 7f63e8855a40 1 waiting to reach incremental sync.. 2023-09-22T22:11:07.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:11:07.370+0000 7f63e8855a40 1 waiting to reach incremental sync.. 2023-09-22T22:11:12.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:11:12.370+0000 7f63e8855a40 1 waiting to reach incremental sync.. 2023-09-22T22:11:12.547 INFO:tasks.rgw.c1.client.0.smithi160.stdout:2023-09-22T22:11:12.546+0000 7fa9c1675640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T22:11:17.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:11:17.374+0000 7f63e8855a40 1 bucket sync caught up with source: 2023-09-22T22:11:17.376 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.367.6, 00000000002.283.6, 00000000002.239.6] 2023-09-22T22:11:17.376 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.367.6, 00000000002.283.6, 00000000002.239.6] 2023-09-22T22:11:17.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:11:17.374+0000 7f63e8855a40 0 bucket checkpoint complete 2023-09-22T22:11:17.410 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ffzdqu-64 zones={a2, a1} 2023-09-22T22:11:17.410 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:17.410 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:17.411 DEBUG:boto:Method: HEAD 2023-09-22T22:11:17.411 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:17.411 DEBUG:boto:Data: 2023-09-22T22:11:17.411 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.411 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:17.411 DEBUG:boto:Port: 8001 2023-09-22T22:11:17.411 DEBUG:boto:Params: {} 2023-09-22T22:11:17.412 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T22:11:17.412 DEBUG:boto:Token: None 2023-09-22T22:11:17.412 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/ 2023-09-22T22:11:17.412 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MzzFtzOyR23+7IZDebU2fVumaUk= 2023-09-22T22:11:17.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MzzFtzOyR23+7IZDebU2fVumaUk=', 'Content-Length': '0'} 2023-09-22T22:11:17.421 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000546f4f365db72675-00650e1105-1fae-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.421 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:17.421 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:17.422 DEBUG:boto:Method: HEAD 2023-09-22T22:11:17.422 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:17.422 DEBUG:boto:Data: 2023-09-22T22:11:17.422 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.422 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:17.422 DEBUG:boto:Port: 8000 2023-09-22T22:11:17.422 DEBUG:boto:Params: {} 2023-09-22T22:11:17.422 DEBUG:boto:Token: None 2023-09-22T22:11:17.423 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/ 2023-09-22T22:11:17.423 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MzzFtzOyR23+7IZDebU2fVumaUk= 2023-09-22T22:11:17.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MzzFtzOyR23+7IZDebU2fVumaUk=', 'Content-Length': '0'} 2023-09-22T22:11:17.426 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6fdaf47d6099dc4-00650e1105-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.426 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-22T22:11:17.427 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:17.427 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:17.427 DEBUG:boto:path=/ffzdqu-64/?versions&max-keys=999 2023-09-22T22:11:17.427 DEBUG:boto:auth_path=/ffzdqu-64/?versions&max-keys=999 2023-09-22T22:11:17.427 DEBUG:boto:Method: GET 2023-09-22T22:11:17.427 DEBUG:boto:Path: /ffzdqu-64/?versions&max-keys=999 2023-09-22T22:11:17.427 DEBUG:boto:Data: 2023-09-22T22:11:17.427 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.428 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:17.428 DEBUG:boto:Port: 8001 2023-09-22T22:11:17.428 DEBUG:boto:Params: {} 2023-09-22T22:11:17.428 DEBUG:boto:Token: None 2023-09-22T22:11:17.428 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/?versions 2023-09-22T22:11:17.428 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6tEvQMrNLoSeNyxhnQYIMaZBat8= 2023-09-22T22:11:17.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6tEvQMrNLoSeNyxhnQYIMaZBat8=', 'Content-Length': '0'} 2023-09-22T22:11:17.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018be0ef4e8980cd9-00650e1105-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.432 DEBUG:boto:b'ffzdqu-64999falseobj11nulltrue2023-09-22T22:10:52.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T22:10:52.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T22:10:52.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T22:10:52.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T22:10:52.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:17.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T22:11:17.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T22:11:17.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T22:11:17.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T22:11:17.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T22:11:17.433 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-22T22:11:17.434 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:17.434 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:17.434 DEBUG:boto:path=/ffzdqu-64/?versions&max-keys=999 2023-09-22T22:11:17.434 DEBUG:boto:auth_path=/ffzdqu-64/?versions&max-keys=999 2023-09-22T22:11:17.434 DEBUG:boto:Method: GET 2023-09-22T22:11:17.434 DEBUG:boto:Path: /ffzdqu-64/?versions&max-keys=999 2023-09-22T22:11:17.434 DEBUG:boto:Data: 2023-09-22T22:11:17.434 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.434 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:17.435 DEBUG:boto:Port: 8000 2023-09-22T22:11:17.435 DEBUG:boto:Params: {} 2023-09-22T22:11:17.435 DEBUG:boto:Token: None 2023-09-22T22:11:17.435 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/?versions 2023-09-22T22:11:17.435 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6tEvQMrNLoSeNyxhnQYIMaZBat8= 2023-09-22T22:11:17.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6tEvQMrNLoSeNyxhnQYIMaZBat8=', 'Content-Length': '0'} 2023-09-22T22:11:17.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6f98e2897e5a0ac-00650e1105-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.439 DEBUG:boto:b'ffzdqu-64999falseobj11nulltrue2023-09-22T22:10:52.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T22:10:52.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T22:10:52.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T22:10:52.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T22:10:52.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:17.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-22T22:11:17.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-22T22:11:17.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-22T22:11:17.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-22T22:11:17.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-22T22:11:17.440 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:17.440 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:17.441 DEBUG:boto:path=/ffzdqu-64/?versions&max-keys=999 2023-09-22T22:11:17.441 DEBUG:boto:auth_path=/ffzdqu-64/?versions&max-keys=999 2023-09-22T22:11:17.441 DEBUG:boto:Method: GET 2023-09-22T22:11:17.441 DEBUG:boto:Path: /ffzdqu-64/?versions&max-keys=999 2023-09-22T22:11:17.441 DEBUG:boto:Data: 2023-09-22T22:11:17.441 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.441 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:17.441 DEBUG:boto:Port: 8001 2023-09-22T22:11:17.441 DEBUG:boto:Params: {} 2023-09-22T22:11:17.441 DEBUG:boto:Token: None 2023-09-22T22:11:17.442 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/?versions 2023-09-22T22:11:17.442 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6tEvQMrNLoSeNyxhnQYIMaZBat8= 2023-09-22T22:11:17.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6tEvQMrNLoSeNyxhnQYIMaZBat8=', 'Content-Length': '0'} 2023-09-22T22:11:17.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b33931ac92f66663-00650e1105-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.446 DEBUG:boto:b'ffzdqu-64999falseobj11nulltrue2023-09-22T22:10:52.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T22:10:52.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T22:10:52.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T22:10:52.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T22:10:52.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:17.446 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:17.447 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:17.447 DEBUG:boto:path=/ffzdqu-64/?versions&max-keys=999 2023-09-22T22:11:17.447 DEBUG:boto:auth_path=/ffzdqu-64/?versions&max-keys=999 2023-09-22T22:11:17.447 DEBUG:boto:Method: GET 2023-09-22T22:11:17.447 DEBUG:boto:Path: /ffzdqu-64/?versions&max-keys=999 2023-09-22T22:11:17.447 DEBUG:boto:Data: 2023-09-22T22:11:17.447 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.447 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:17.447 DEBUG:boto:Port: 8000 2023-09-22T22:11:17.448 DEBUG:boto:Params: {} 2023-09-22T22:11:17.448 DEBUG:boto:Token: None 2023-09-22T22:11:17.448 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/?versions 2023-09-22T22:11:17.448 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:6tEvQMrNLoSeNyxhnQYIMaZBat8= 2023-09-22T22:11:17.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6tEvQMrNLoSeNyxhnQYIMaZBat8=', 'Content-Length': '0'} 2023-09-22T22:11:17.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f397ed7ea8762e10-00650e1105-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.452 DEBUG:boto:b'ffzdqu-64999falseobj11nulltrue2023-09-22T22:10:52.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T22:10:52.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T22:10:52.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T22:10:52.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T22:10:52.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:17.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T22:11:17.453 DEBUG:boto:path=/ffzdqu-64/obj11 2023-09-22T22:11:17.453 DEBUG:boto:auth_path=/ffzdqu-64/obj11 2023-09-22T22:11:17.454 DEBUG:boto:path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.454 DEBUG:boto:auth_path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.454 DEBUG:boto:Method: GET 2023-09-22T22:11:17.454 DEBUG:boto:Path: /ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.454 DEBUG:boto:Data: 2023-09-22T22:11:17.454 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.454 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:17.454 DEBUG:boto:Port: 8001 2023-09-22T22:11:17.454 DEBUG:boto:Params: {} 2023-09-22T22:11:17.455 DEBUG:boto:Token: None 2023-09-22T22:11:17.455 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.455 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MAGCo+HTDcx7bvFqY8x75Ofvj6s= 2023-09-22T22:11:17.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MAGCo+HTDcx7bvFqY8x75Ofvj6s=', 'Content-Length': '0'} 2023-09-22T22:11:17.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000611778ff93b56430-00650e1105-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.459 DEBUG:boto:path=/ffzdqu-64/obj11 2023-09-22T22:11:17.459 DEBUG:boto:auth_path=/ffzdqu-64/obj11 2023-09-22T22:11:17.459 DEBUG:boto:path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.459 DEBUG:boto:auth_path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.459 DEBUG:boto:Method: GET 2023-09-22T22:11:17.459 DEBUG:boto:Path: /ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.459 DEBUG:boto:Data: 2023-09-22T22:11:17.459 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.459 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:17.460 DEBUG:boto:Port: 8000 2023-09-22T22:11:17.460 DEBUG:boto:Params: {} 2023-09-22T22:11:17.460 DEBUG:boto:Token: None 2023-09-22T22:11:17.460 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.460 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MAGCo+HTDcx7bvFqY8x75Ofvj6s= 2023-09-22T22:11:17.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MAGCo+HTDcx7bvFqY8x75Ofvj6s=', 'Content-Length': '0'} 2023-09-22T22:11:17.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd7cc4c09ec378f0-00650e1105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.506 DEBUG:boto:path=/ffzdqu-64/obj11 2023-09-22T22:11:17.507 DEBUG:boto:auth_path=/ffzdqu-64/obj11 2023-09-22T22:11:17.507 DEBUG:boto:path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.507 DEBUG:boto:auth_path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.507 DEBUG:boto:Method: HEAD 2023-09-22T22:11:17.507 DEBUG:boto:Path: /ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.507 DEBUG:boto:Data: 2023-09-22T22:11:17.507 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.507 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:17.508 DEBUG:boto:Port: 8001 2023-09-22T22:11:17.508 DEBUG:boto:Params: {} 2023-09-22T22:11:17.508 DEBUG:boto:Token: None 2023-09-22T22:11:17.508 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.508 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:B8qQKr2P7CwCAj23Dz0W4CjdEM0= 2023-09-22T22:11:17.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:B8qQKr2P7CwCAj23Dz0W4CjdEM0=', 'Content-Length': '0'} 2023-09-22T22:11:17.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006663de5db3a9d0cb-00650e1105-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.511 DEBUG:boto:path=/ffzdqu-64/obj11 2023-09-22T22:11:17.511 DEBUG:boto:auth_path=/ffzdqu-64/obj11 2023-09-22T22:11:17.512 DEBUG:boto:path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.512 DEBUG:boto:auth_path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.512 DEBUG:boto:Method: HEAD 2023-09-22T22:11:17.512 DEBUG:boto:Path: /ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.512 DEBUG:boto:Data: 2023-09-22T22:11:17.512 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.512 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:17.512 DEBUG:boto:Port: 8000 2023-09-22T22:11:17.512 DEBUG:boto:Params: {} 2023-09-22T22:11:17.513 DEBUG:boto:Token: None 2023-09-22T22:11:17.513 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.513 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:B8qQKr2P7CwCAj23Dz0W4CjdEM0= 2023-09-22T22:11:17.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:B8qQKr2P7CwCAj23Dz0W4CjdEM0=', 'Content-Length': '0'} 2023-09-22T22:11:17.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000791e414cfe2b0bbc-00650e1105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T22:11:17.516 DEBUG:boto:path=/ffzdqu-64/obj11 2023-09-22T22:11:17.516 DEBUG:boto:auth_path=/ffzdqu-64/obj11 2023-09-22T22:11:17.516 DEBUG:boto:path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.516 DEBUG:boto:auth_path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.516 DEBUG:boto:Method: GET 2023-09-22T22:11:17.517 DEBUG:boto:Path: /ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.517 DEBUG:boto:Data: 2023-09-22T22:11:17.517 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.517 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:17.517 DEBUG:boto:Port: 8001 2023-09-22T22:11:17.517 DEBUG:boto:Params: {} 2023-09-22T22:11:17.517 DEBUG:boto:Token: None 2023-09-22T22:11:17.517 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.518 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MAGCo+HTDcx7bvFqY8x75Ofvj6s= 2023-09-22T22:11:17.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MAGCo+HTDcx7bvFqY8x75Ofvj6s=', 'Content-Length': '0'} 2023-09-22T22:11:17.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b02322b564edaad3-00650e1105-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.562 DEBUG:boto:path=/ffzdqu-64/obj11 2023-09-22T22:11:17.563 DEBUG:boto:auth_path=/ffzdqu-64/obj11 2023-09-22T22:11:17.563 DEBUG:boto:path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.563 DEBUG:boto:auth_path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.563 DEBUG:boto:Method: GET 2023-09-22T22:11:17.563 DEBUG:boto:Path: /ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.563 DEBUG:boto:Data: 2023-09-22T22:11:17.563 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.563 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:17.563 DEBUG:boto:Port: 8000 2023-09-22T22:11:17.564 DEBUG:boto:Params: {} 2023-09-22T22:11:17.564 DEBUG:boto:Token: None 2023-09-22T22:11:17.564 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/obj11?versionId=null 2023-09-22T22:11:17.564 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:MAGCo+HTDcx7bvFqY8x75Ofvj6s= 2023-09-22T22:11:17.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MAGCo+HTDcx7bvFqY8x75Ofvj6s=', 'Content-Length': '0'} 2023-09-22T22:11:17.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c31a133abd63d705-00650e1105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.568 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:17.568 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:17.568 DEBUG:boto:path=/ffzdqu-64/?prefix=obj11 2023-09-22T22:11:17.568 DEBUG:boto:auth_path=/ffzdqu-64/?prefix=obj11 2023-09-22T22:11:17.568 DEBUG:boto:Method: GET 2023-09-22T22:11:17.568 DEBUG:boto:Path: /ffzdqu-64/?prefix=obj11 2023-09-22T22:11:17.568 DEBUG:boto:Data: 2023-09-22T22:11:17.568 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.568 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:17.569 DEBUG:boto:Port: 8001 2023-09-22T22:11:17.569 DEBUG:boto:Params: {} 2023-09-22T22:11:17.569 DEBUG:boto:Token: None 2023-09-22T22:11:17.569 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/ 2023-09-22T22:11:17.569 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs= 2023-09-22T22:11:17.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs=', 'Content-Length': '0'} 2023-09-22T22:11:17.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000796307c7439d1aec-00650e1105-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.614 DEBUG:boto:b'ffzdqu-64obj111000falseobj112023-09-22T22:10:52.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:17.615 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:17.615 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:17.615 DEBUG:boto:path=/ffzdqu-64/?prefix=obj11 2023-09-22T22:11:17.615 DEBUG:boto:auth_path=/ffzdqu-64/?prefix=obj11 2023-09-22T22:11:17.615 DEBUG:boto:Method: GET 2023-09-22T22:11:17.615 DEBUG:boto:Path: /ffzdqu-64/?prefix=obj11 2023-09-22T22:11:17.616 DEBUG:boto:Data: 2023-09-22T22:11:17.616 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.616 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:17.616 DEBUG:boto:Port: 8000 2023-09-22T22:11:17.616 DEBUG:boto:Params: {} 2023-09-22T22:11:17.616 DEBUG:boto:Token: None 2023-09-22T22:11:17.616 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/ 2023-09-22T22:11:17.616 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs= 2023-09-22T22:11:17.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs=', 'Content-Length': '0'} 2023-09-22T22:11:17.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fed5ff75ab9911a4-00650e1105-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.621 DEBUG:boto:b'ffzdqu-64obj111000falseobj112023-09-22T22:10:52.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:17.622 DEBUG:boto:path=/ffzdqu-64/obj11 2023-09-22T22:11:17.622 DEBUG:boto:auth_path=/ffzdqu-64/obj11 2023-09-22T22:11:17.622 DEBUG:boto:Method: HEAD 2023-09-22T22:11:17.622 DEBUG:boto:Path: /ffzdqu-64/obj11 2023-09-22T22:11:17.622 DEBUG:boto:Data: 2023-09-22T22:11:17.622 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.622 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:17.622 DEBUG:boto:Port: 8001 2023-09-22T22:11:17.622 DEBUG:boto:Params: {} 2023-09-22T22:11:17.623 DEBUG:boto:Token: None 2023-09-22T22:11:17.623 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/obj11 2023-09-22T22:11:17.623 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7k87VKrrWndsR5w/94krjYTTNC0= 2023-09-22T22:11:17.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7k87VKrrWndsR5w/94krjYTTNC0=', 'Content-Length': '0'} 2023-09-22T22:11:17.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeca03ade649dac6-00650e1105-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.626 DEBUG:boto:path=/ffzdqu-64/obj11 2023-09-22T22:11:17.626 DEBUG:boto:auth_path=/ffzdqu-64/obj11 2023-09-22T22:11:17.626 DEBUG:boto:Method: HEAD 2023-09-22T22:11:17.626 DEBUG:boto:Path: /ffzdqu-64/obj11 2023-09-22T22:11:17.626 DEBUG:boto:Data: 2023-09-22T22:11:17.627 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.627 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:17.627 DEBUG:boto:Port: 8000 2023-09-22T22:11:17.627 DEBUG:boto:Params: {} 2023-09-22T22:11:17.627 DEBUG:boto:Token: None 2023-09-22T22:11:17.627 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/obj11 2023-09-22T22:11:17.627 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:7k87VKrrWndsR5w/94krjYTTNC0= 2023-09-22T22:11:17.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7k87VKrrWndsR5w/94krjYTTNC0=', 'Content-Length': '0'} 2023-09-22T22:11:17.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd0f457f96fdb82e-00650e1105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-22T22:11:17.630 DEBUG:boto:path=/ffzdqu-64/obj11 2023-09-22T22:11:17.631 DEBUG:boto:auth_path=/ffzdqu-64/obj11 2023-09-22T22:11:17.631 DEBUG:boto:Method: GET 2023-09-22T22:11:17.631 DEBUG:boto:Path: /ffzdqu-64/obj11 2023-09-22T22:11:17.631 DEBUG:boto:Data: 2023-09-22T22:11:17.631 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.631 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:17.631 DEBUG:boto:Port: 8001 2023-09-22T22:11:17.631 DEBUG:boto:Params: {} 2023-09-22T22:11:17.631 DEBUG:boto:Token: None 2023-09-22T22:11:17.632 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/obj11 2023-09-22T22:11:17.632 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QylKQUMlzPV7IrCtK3iyx3XSMRs= 2023-09-22T22:11:17.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QylKQUMlzPV7IrCtK3iyx3XSMRs=', 'Content-Length': '0'} 2023-09-22T22:11:17.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007657fcf73be387ad-00650e1105-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.678 DEBUG:boto:path=/ffzdqu-64/obj11 2023-09-22T22:11:17.679 DEBUG:boto:auth_path=/ffzdqu-64/obj11 2023-09-22T22:11:17.679 DEBUG:boto:Method: GET 2023-09-22T22:11:17.679 DEBUG:boto:Path: /ffzdqu-64/obj11 2023-09-22T22:11:17.679 DEBUG:boto:Data: 2023-09-22T22:11:17.679 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.679 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:17.679 DEBUG:boto:Port: 8000 2023-09-22T22:11:17.679 DEBUG:boto:Params: {} 2023-09-22T22:11:17.680 DEBUG:boto:Token: None 2023-09-22T22:11:17.680 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/obj11 2023-09-22T22:11:17.680 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:QylKQUMlzPV7IrCtK3iyx3XSMRs= 2023-09-22T22:11:17.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QylKQUMlzPV7IrCtK3iyx3XSMRs=', 'Content-Length': '0'} 2023-09-22T22:11:17.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f72abdc5d17afce6-00650e1105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.683 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:17.683 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:17.684 DEBUG:boto:path=/ffzdqu-64/?prefix=obj11 2023-09-22T22:11:17.684 DEBUG:boto:auth_path=/ffzdqu-64/?prefix=obj11 2023-09-22T22:11:17.684 DEBUG:boto:Method: GET 2023-09-22T22:11:17.684 DEBUG:boto:Path: /ffzdqu-64/?prefix=obj11 2023-09-22T22:11:17.684 DEBUG:boto:Data: 2023-09-22T22:11:17.684 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.684 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:17.684 DEBUG:boto:Port: 8001 2023-09-22T22:11:17.684 DEBUG:boto:Params: {} 2023-09-22T22:11:17.685 DEBUG:boto:Token: None 2023-09-22T22:11:17.685 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/ 2023-09-22T22:11:17.685 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs= 2023-09-22T22:11:17.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs=', 'Content-Length': '0'} 2023-09-22T22:11:17.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006693fe62e1f5a16e-00650e1105-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.734 DEBUG:boto:b'ffzdqu-64obj111000falseobj112023-09-22T22:10:52.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:17.735 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:17.735 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:17.735 DEBUG:boto:path=/ffzdqu-64/?prefix=obj11 2023-09-22T22:11:17.735 DEBUG:boto:auth_path=/ffzdqu-64/?prefix=obj11 2023-09-22T22:11:17.735 DEBUG:boto:Method: GET 2023-09-22T22:11:17.735 DEBUG:boto:Path: /ffzdqu-64/?prefix=obj11 2023-09-22T22:11:17.736 DEBUG:boto:Data: 2023-09-22T22:11:17.736 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:17.736 DEBUG:boto:Port: 8000 2023-09-22T22:11:17.736 DEBUG:boto:Params: {} 2023-09-22T22:11:17.736 DEBUG:boto:Token: None 2023-09-22T22:11:17.736 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/ 2023-09-22T22:11:17.736 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs= 2023-09-22T22:11:17.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs=', 'Content-Length': '0'} 2023-09-22T22:11:17.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0ebe320f96c9f44-00650e1105-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.742 DEBUG:boto:b'ffzdqu-64obj111000falseobj112023-09-22T22:10:52.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:17.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T22:11:17.742 DEBUG:boto:path=/ffzdqu-64/obj22 2023-09-22T22:11:17.743 DEBUG:boto:auth_path=/ffzdqu-64/obj22 2023-09-22T22:11:17.743 DEBUG:boto:path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.743 DEBUG:boto:auth_path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.743 DEBUG:boto:Method: GET 2023-09-22T22:11:17.743 DEBUG:boto:Path: /ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.743 DEBUG:boto:Data: 2023-09-22T22:11:17.743 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.743 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:17.743 DEBUG:boto:Port: 8001 2023-09-22T22:11:17.744 DEBUG:boto:Params: {} 2023-09-22T22:11:17.744 DEBUG:boto:Token: None 2023-09-22T22:11:17.744 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.744 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ROBOZpWv40872mMlG8N3bfx+Z6c= 2023-09-22T22:11:17.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ROBOZpWv40872mMlG8N3bfx+Z6c=', 'Content-Length': '0'} 2023-09-22T22:11:17.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004558cb9f7bdec855-00650e1105-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.747 DEBUG:boto:path=/ffzdqu-64/obj22 2023-09-22T22:11:17.748 DEBUG:boto:auth_path=/ffzdqu-64/obj22 2023-09-22T22:11:17.748 DEBUG:boto:path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.748 DEBUG:boto:auth_path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.748 DEBUG:boto:Method: GET 2023-09-22T22:11:17.748 DEBUG:boto:Path: /ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.748 DEBUG:boto:Data: 2023-09-22T22:11:17.748 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.748 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:17.748 DEBUG:boto:Port: 8000 2023-09-22T22:11:17.748 DEBUG:boto:Params: {} 2023-09-22T22:11:17.749 DEBUG:boto:Token: None 2023-09-22T22:11:17.749 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.749 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ROBOZpWv40872mMlG8N3bfx+Z6c= 2023-09-22T22:11:17.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ROBOZpWv40872mMlG8N3bfx+Z6c=', 'Content-Length': '0'} 2023-09-22T22:11:17.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067c1d0c82bde647b-00650e1105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.798 DEBUG:boto:path=/ffzdqu-64/obj22 2023-09-22T22:11:17.799 DEBUG:boto:auth_path=/ffzdqu-64/obj22 2023-09-22T22:11:17.799 DEBUG:boto:path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.799 DEBUG:boto:auth_path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.799 DEBUG:boto:Method: HEAD 2023-09-22T22:11:17.799 DEBUG:boto:Path: /ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.799 DEBUG:boto:Data: 2023-09-22T22:11:17.799 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.799 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:17.799 DEBUG:boto:Port: 8001 2023-09-22T22:11:17.800 DEBUG:boto:Params: {} 2023-09-22T22:11:17.800 DEBUG:boto:Token: None 2023-09-22T22:11:17.800 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.800 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jPzYE2Z7psQmD1uzQMxqQR/NBHs= 2023-09-22T22:11:17.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jPzYE2Z7psQmD1uzQMxqQR/NBHs=', 'Content-Length': '0'} 2023-09-22T22:11:17.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5200ff1037059c3-00650e1105-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.803 DEBUG:boto:path=/ffzdqu-64/obj22 2023-09-22T22:11:17.803 DEBUG:boto:auth_path=/ffzdqu-64/obj22 2023-09-22T22:11:17.804 DEBUG:boto:path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.804 DEBUG:boto:auth_path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.804 DEBUG:boto:Method: HEAD 2023-09-22T22:11:17.804 DEBUG:boto:Path: /ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.804 DEBUG:boto:Data: 2023-09-22T22:11:17.804 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.804 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:17.804 DEBUG:boto:Port: 8000 2023-09-22T22:11:17.805 DEBUG:boto:Params: {} 2023-09-22T22:11:17.805 DEBUG:boto:Token: None 2023-09-22T22:11:17.805 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.805 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jPzYE2Z7psQmD1uzQMxqQR/NBHs= 2023-09-22T22:11:17.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jPzYE2Z7psQmD1uzQMxqQR/NBHs=', 'Content-Length': '0'} 2023-09-22T22:11:17.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d9c7d09a33ab01a-00650e1105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T22:11:17.808 DEBUG:boto:path=/ffzdqu-64/obj22 2023-09-22T22:11:17.808 DEBUG:boto:auth_path=/ffzdqu-64/obj22 2023-09-22T22:11:17.808 DEBUG:boto:path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.809 DEBUG:boto:auth_path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.809 DEBUG:boto:Method: GET 2023-09-22T22:11:17.809 DEBUG:boto:Path: /ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.809 DEBUG:boto:Data: 2023-09-22T22:11:17.809 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.809 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:17.809 DEBUG:boto:Port: 8001 2023-09-22T22:11:17.809 DEBUG:boto:Params: {} 2023-09-22T22:11:17.809 DEBUG:boto:Token: None 2023-09-22T22:11:17.810 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.810 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ROBOZpWv40872mMlG8N3bfx+Z6c= 2023-09-22T22:11:17.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ROBOZpWv40872mMlG8N3bfx+Z6c=', 'Content-Length': '0'} 2023-09-22T22:11:17.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bcf2a0a8c66eb4d-00650e1105-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.854 DEBUG:boto:path=/ffzdqu-64/obj22 2023-09-22T22:11:17.855 DEBUG:boto:auth_path=/ffzdqu-64/obj22 2023-09-22T22:11:17.855 DEBUG:boto:path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.855 DEBUG:boto:auth_path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.855 DEBUG:boto:Method: GET 2023-09-22T22:11:17.855 DEBUG:boto:Path: /ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.855 DEBUG:boto:Data: 2023-09-22T22:11:17.855 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.855 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:17.855 DEBUG:boto:Port: 8000 2023-09-22T22:11:17.856 DEBUG:boto:Params: {} 2023-09-22T22:11:17.856 DEBUG:boto:Token: None 2023-09-22T22:11:17.856 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/obj22?versionId=null 2023-09-22T22:11:17.856 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ROBOZpWv40872mMlG8N3bfx+Z6c= 2023-09-22T22:11:17.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ROBOZpWv40872mMlG8N3bfx+Z6c=', 'Content-Length': '0'} 2023-09-22T22:11:17.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091876ed8dee1768b-00650e1105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.859 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:17.860 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:17.860 DEBUG:boto:path=/ffzdqu-64/?prefix=obj22 2023-09-22T22:11:17.860 DEBUG:boto:auth_path=/ffzdqu-64/?prefix=obj22 2023-09-22T22:11:17.860 DEBUG:boto:Method: GET 2023-09-22T22:11:17.860 DEBUG:boto:Path: /ffzdqu-64/?prefix=obj22 2023-09-22T22:11:17.860 DEBUG:boto:Data: 2023-09-22T22:11:17.860 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.860 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:17.860 DEBUG:boto:Port: 8001 2023-09-22T22:11:17.860 DEBUG:boto:Params: {} 2023-09-22T22:11:17.861 DEBUG:boto:Token: None 2023-09-22T22:11:17.861 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/ 2023-09-22T22:11:17.861 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs= 2023-09-22T22:11:17.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs=', 'Content-Length': '0'} 2023-09-22T22:11:17.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000863f58de38cb9f6a-00650e1105-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.910 DEBUG:boto:b'ffzdqu-64obj221000falseobj222023-09-22T22:10:52.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:17.911 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:17.911 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:17.911 DEBUG:boto:path=/ffzdqu-64/?prefix=obj22 2023-09-22T22:11:17.911 DEBUG:boto:auth_path=/ffzdqu-64/?prefix=obj22 2023-09-22T22:11:17.911 DEBUG:boto:Method: GET 2023-09-22T22:11:17.912 DEBUG:boto:Path: /ffzdqu-64/?prefix=obj22 2023-09-22T22:11:17.912 DEBUG:boto:Data: 2023-09-22T22:11:17.912 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.912 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:17.912 DEBUG:boto:Port: 8000 2023-09-22T22:11:17.912 DEBUG:boto:Params: {} 2023-09-22T22:11:17.912 DEBUG:boto:Token: None 2023-09-22T22:11:17.912 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/ 2023-09-22T22:11:17.913 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs= 2023-09-22T22:11:17.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs=', 'Content-Length': '0'} 2023-09-22T22:11:17.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d818191acbd380f-00650e1105-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.918 DEBUG:boto:b'ffzdqu-64obj221000falseobj222023-09-22T22:10:52.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:17.918 DEBUG:boto:path=/ffzdqu-64/obj22 2023-09-22T22:11:17.918 DEBUG:boto:auth_path=/ffzdqu-64/obj22 2023-09-22T22:11:17.918 DEBUG:boto:Method: HEAD 2023-09-22T22:11:17.918 DEBUG:boto:Path: /ffzdqu-64/obj22 2023-09-22T22:11:17.918 DEBUG:boto:Data: 2023-09-22T22:11:17.919 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.919 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:17.919 DEBUG:boto:Port: 8001 2023-09-22T22:11:17.919 DEBUG:boto:Params: {} 2023-09-22T22:11:17.919 DEBUG:boto:Token: None 2023-09-22T22:11:17.919 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/obj22 2023-09-22T22:11:17.919 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:L0y0XpyXOV5C7XhRCyMgsrwiqd8= 2023-09-22T22:11:17.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:L0y0XpyXOV5C7XhRCyMgsrwiqd8=', 'Content-Length': '0'} 2023-09-22T22:11:17.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6d7ce0bc8d47c35-00650e1105-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.922 DEBUG:boto:path=/ffzdqu-64/obj22 2023-09-22T22:11:17.923 DEBUG:boto:auth_path=/ffzdqu-64/obj22 2023-09-22T22:11:17.923 DEBUG:boto:Method: HEAD 2023-09-22T22:11:17.923 DEBUG:boto:Path: /ffzdqu-64/obj22 2023-09-22T22:11:17.923 DEBUG:boto:Data: 2023-09-22T22:11:17.923 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.923 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:17.923 DEBUG:boto:Port: 8000 2023-09-22T22:11:17.923 DEBUG:boto:Params: {} 2023-09-22T22:11:17.924 DEBUG:boto:Token: None 2023-09-22T22:11:17.924 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/obj22 2023-09-22T22:11:17.924 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:L0y0XpyXOV5C7XhRCyMgsrwiqd8= 2023-09-22T22:11:17.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:L0y0XpyXOV5C7XhRCyMgsrwiqd8=', 'Content-Length': '0'} 2023-09-22T22:11:17.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f56aea8ea7166d5-00650e1105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-22T22:11:17.927 DEBUG:boto:path=/ffzdqu-64/obj22 2023-09-22T22:11:17.927 DEBUG:boto:auth_path=/ffzdqu-64/obj22 2023-09-22T22:11:17.927 DEBUG:boto:Method: GET 2023-09-22T22:11:17.927 DEBUG:boto:Path: /ffzdqu-64/obj22 2023-09-22T22:11:17.927 DEBUG:boto:Data: 2023-09-22T22:11:17.928 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.928 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:17.928 DEBUG:boto:Port: 8001 2023-09-22T22:11:17.928 DEBUG:boto:Params: {} 2023-09-22T22:11:17.928 DEBUG:boto:Token: None 2023-09-22T22:11:17.928 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/obj22 2023-09-22T22:11:17.928 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VRM6bIR0b/eaR4udd9WdLZ14Vv0= 2023-09-22T22:11:17.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VRM6bIR0b/eaR4udd9WdLZ14Vv0=', 'Content-Length': '0'} 2023-09-22T22:11:17.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000823eea96d83d2424-00650e1105-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.974 DEBUG:boto:path=/ffzdqu-64/obj22 2023-09-22T22:11:17.975 DEBUG:boto:auth_path=/ffzdqu-64/obj22 2023-09-22T22:11:17.975 DEBUG:boto:Method: GET 2023-09-22T22:11:17.975 DEBUG:boto:Path: /ffzdqu-64/obj22 2023-09-22T22:11:17.975 DEBUG:boto:Data: 2023-09-22T22:11:17.975 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.975 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:17.975 DEBUG:boto:Port: 8000 2023-09-22T22:11:17.975 DEBUG:boto:Params: {} 2023-09-22T22:11:17.975 DEBUG:boto:Token: None 2023-09-22T22:11:17.976 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/obj22 2023-09-22T22:11:17.976 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:VRM6bIR0b/eaR4udd9WdLZ14Vv0= 2023-09-22T22:11:17.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VRM6bIR0b/eaR4udd9WdLZ14Vv0=', 'Content-Length': '0'} 2023-09-22T22:11:17.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fde57c1cd2dc8df3-00650e1105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:17.979 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:17.979 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:17.979 DEBUG:boto:path=/ffzdqu-64/?prefix=obj22 2023-09-22T22:11:17.980 DEBUG:boto:auth_path=/ffzdqu-64/?prefix=obj22 2023-09-22T22:11:17.980 DEBUG:boto:Method: GET 2023-09-22T22:11:17.980 DEBUG:boto:Path: /ffzdqu-64/?prefix=obj22 2023-09-22T22:11:17.980 DEBUG:boto:Data: 2023-09-22T22:11:17.980 DEBUG:boto:Headers: {} 2023-09-22T22:11:17.980 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:17.980 DEBUG:boto:Port: 8001 2023-09-22T22:11:17.980 DEBUG:boto:Params: {} 2023-09-22T22:11:17.980 DEBUG:boto:Token: None 2023-09-22T22:11:17.981 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:17 GMT /ffzdqu-64/ 2023-09-22T22:11:17.981 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs= 2023-09-22T22:11:17.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs=', 'Content-Length': '0'} 2023-09-22T22:11:17.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4a36318d2f336b4-00650e1105-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.026 DEBUG:boto:b'ffzdqu-64obj221000falseobj222023-09-22T22:10:52.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:18.027 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:18.027 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:18.027 DEBUG:boto:path=/ffzdqu-64/?prefix=obj22 2023-09-22T22:11:18.027 DEBUG:boto:auth_path=/ffzdqu-64/?prefix=obj22 2023-09-22T22:11:18.027 DEBUG:boto:Method: GET 2023-09-22T22:11:18.028 DEBUG:boto:Path: /ffzdqu-64/?prefix=obj22 2023-09-22T22:11:18.028 DEBUG:boto:Data: 2023-09-22T22:11:18.028 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.028 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.028 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.028 DEBUG:boto:Params: {} 2023-09-22T22:11:18.028 DEBUG:boto:Token: None 2023-09-22T22:11:18.029 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/ 2023-09-22T22:11:18.029 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:11:18.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:11:18.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a17d1d72343e8bc9-00650e1106-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.034 DEBUG:boto:b'ffzdqu-64obj221000falseobj222023-09-22T22:10:52.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:18.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T22:11:18.035 DEBUG:boto:path=/ffzdqu-64/obj33 2023-09-22T22:11:18.035 DEBUG:boto:auth_path=/ffzdqu-64/obj33 2023-09-22T22:11:18.037 DEBUG:boto:path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.037 DEBUG:boto:auth_path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.037 DEBUG:boto:Method: GET 2023-09-22T22:11:18.037 DEBUG:boto:Path: /ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.037 DEBUG:boto:Data: 2023-09-22T22:11:18.037 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.037 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:18.038 DEBUG:boto:Port: 8001 2023-09-22T22:11:18.038 DEBUG:boto:Params: {} 2023-09-22T22:11:18.038 DEBUG:boto:Token: None 2023-09-22T22:11:18.038 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.038 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h/sE5s8+uqC0h7kQymg9eoylTJQ= 2023-09-22T22:11:18.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h/sE5s8+uqC0h7kQymg9eoylTJQ=', 'Content-Length': '0'} 2023-09-22T22:11:18.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e132d24b1a50e3cf-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.042 DEBUG:boto:path=/ffzdqu-64/obj33 2023-09-22T22:11:18.042 DEBUG:boto:auth_path=/ffzdqu-64/obj33 2023-09-22T22:11:18.042 DEBUG:boto:path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.042 DEBUG:boto:auth_path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.042 DEBUG:boto:Method: GET 2023-09-22T22:11:18.042 DEBUG:boto:Path: /ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.042 DEBUG:boto:Data: 2023-09-22T22:11:18.043 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.043 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.043 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.043 DEBUG:boto:Params: {} 2023-09-22T22:11:18.043 DEBUG:boto:Token: None 2023-09-22T22:11:18.043 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.043 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h/sE5s8+uqC0h7kQymg9eoylTJQ= 2023-09-22T22:11:18.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h/sE5s8+uqC0h7kQymg9eoylTJQ=', 'Content-Length': '0'} 2023-09-22T22:11:18.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f240b5af23d4815-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.094 DEBUG:boto:path=/ffzdqu-64/obj33 2023-09-22T22:11:18.095 DEBUG:boto:auth_path=/ffzdqu-64/obj33 2023-09-22T22:11:18.095 DEBUG:boto:path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.095 DEBUG:boto:auth_path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.095 DEBUG:boto:Method: HEAD 2023-09-22T22:11:18.095 DEBUG:boto:Path: /ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.095 DEBUG:boto:Data: 2023-09-22T22:11:18.095 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:18.095 DEBUG:boto:Port: 8001 2023-09-22T22:11:18.096 DEBUG:boto:Params: {} 2023-09-22T22:11:18.096 DEBUG:boto:Token: None 2023-09-22T22:11:18.096 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.096 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gS8ZgG27cZOVxGRWfVbj0tuZ7Nc= 2023-09-22T22:11:18.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gS8ZgG27cZOVxGRWfVbj0tuZ7Nc=', 'Content-Length': '0'} 2023-09-22T22:11:18.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d874b3b249b3f94-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.099 DEBUG:boto:path=/ffzdqu-64/obj33 2023-09-22T22:11:18.099 DEBUG:boto:auth_path=/ffzdqu-64/obj33 2023-09-22T22:11:18.099 DEBUG:boto:path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.099 DEBUG:boto:auth_path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.100 DEBUG:boto:Method: HEAD 2023-09-22T22:11:18.100 DEBUG:boto:Path: /ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.100 DEBUG:boto:Data: 2023-09-22T22:11:18.100 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.100 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.100 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.100 DEBUG:boto:Params: {} 2023-09-22T22:11:18.100 DEBUG:boto:Token: None 2023-09-22T22:11:18.100 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.101 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gS8ZgG27cZOVxGRWfVbj0tuZ7Nc= 2023-09-22T22:11:18.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gS8ZgG27cZOVxGRWfVbj0tuZ7Nc=', 'Content-Length': '0'} 2023-09-22T22:11:18.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acb5eb1d03e88bba-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T22:11:18.104 DEBUG:boto:path=/ffzdqu-64/obj33 2023-09-22T22:11:18.104 DEBUG:boto:auth_path=/ffzdqu-64/obj33 2023-09-22T22:11:18.104 DEBUG:boto:path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.104 DEBUG:boto:auth_path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.104 DEBUG:boto:Method: GET 2023-09-22T22:11:18.105 DEBUG:boto:Path: /ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.105 DEBUG:boto:Data: 2023-09-22T22:11:18.105 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.105 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:18.105 DEBUG:boto:Port: 8001 2023-09-22T22:11:18.105 DEBUG:boto:Params: {} 2023-09-22T22:11:18.105 DEBUG:boto:Token: None 2023-09-22T22:11:18.105 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.105 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h/sE5s8+uqC0h7kQymg9eoylTJQ= 2023-09-22T22:11:18.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h/sE5s8+uqC0h7kQymg9eoylTJQ=', 'Content-Length': '0'} 2023-09-22T22:11:18.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008b4ae1ef9547c8a-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.150 DEBUG:boto:path=/ffzdqu-64/obj33 2023-09-22T22:11:18.151 DEBUG:boto:auth_path=/ffzdqu-64/obj33 2023-09-22T22:11:18.151 DEBUG:boto:path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.151 DEBUG:boto:auth_path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.151 DEBUG:boto:Method: GET 2023-09-22T22:11:18.151 DEBUG:boto:Path: /ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.151 DEBUG:boto:Data: 2023-09-22T22:11:18.151 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.151 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.151 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.152 DEBUG:boto:Params: {} 2023-09-22T22:11:18.152 DEBUG:boto:Token: None 2023-09-22T22:11:18.152 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj33?versionId=null 2023-09-22T22:11:18.152 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:h/sE5s8+uqC0h7kQymg9eoylTJQ= 2023-09-22T22:11:18.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h/sE5s8+uqC0h7kQymg9eoylTJQ=', 'Content-Length': '0'} 2023-09-22T22:11:18.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbae06ba4dc22f1c-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.155 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:18.156 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:18.156 DEBUG:boto:path=/ffzdqu-64/?prefix=obj33 2023-09-22T22:11:18.156 DEBUG:boto:auth_path=/ffzdqu-64/?prefix=obj33 2023-09-22T22:11:18.156 DEBUG:boto:Method: GET 2023-09-22T22:11:18.156 DEBUG:boto:Path: /ffzdqu-64/?prefix=obj33 2023-09-22T22:11:18.156 DEBUG:boto:Data: 2023-09-22T22:11:18.156 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.156 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:18.156 DEBUG:boto:Port: 8001 2023-09-22T22:11:18.156 DEBUG:boto:Params: {} 2023-09-22T22:11:18.157 DEBUG:boto:Token: None 2023-09-22T22:11:18.157 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/ 2023-09-22T22:11:18.157 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:11:18.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:11:18.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e601812d41a057b-00650e1106-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.202 DEBUG:boto:b'ffzdqu-64obj331000falseobj332023-09-22T22:10:52.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:18.203 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:18.203 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:18.203 DEBUG:boto:path=/ffzdqu-64/?prefix=obj33 2023-09-22T22:11:18.203 DEBUG:boto:auth_path=/ffzdqu-64/?prefix=obj33 2023-09-22T22:11:18.203 DEBUG:boto:Method: GET 2023-09-22T22:11:18.203 DEBUG:boto:Path: /ffzdqu-64/?prefix=obj33 2023-09-22T22:11:18.204 DEBUG:boto:Data: 2023-09-22T22:11:18.204 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.204 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.204 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.204 DEBUG:boto:Params: {} 2023-09-22T22:11:18.204 DEBUG:boto:Token: None 2023-09-22T22:11:18.204 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/ 2023-09-22T22:11:18.204 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:11:18.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:11:18.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c41d94e5d3c5ad3-00650e1106-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.210 DEBUG:boto:b'ffzdqu-64obj331000falseobj332023-09-22T22:10:52.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:18.210 DEBUG:boto:path=/ffzdqu-64/obj33 2023-09-22T22:11:18.210 DEBUG:boto:auth_path=/ffzdqu-64/obj33 2023-09-22T22:11:18.210 DEBUG:boto:Method: HEAD 2023-09-22T22:11:18.210 DEBUG:boto:Path: /ffzdqu-64/obj33 2023-09-22T22:11:18.211 DEBUG:boto:Data: 2023-09-22T22:11:18.211 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.211 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:18.211 DEBUG:boto:Port: 8001 2023-09-22T22:11:18.211 DEBUG:boto:Params: {} 2023-09-22T22:11:18.211 DEBUG:boto:Token: None 2023-09-22T22:11:18.211 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj33 2023-09-22T22:11:18.211 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0p86mvaEnhR35P9nBEliaWuLsTQ= 2023-09-22T22:11:18.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0p86mvaEnhR35P9nBEliaWuLsTQ=', 'Content-Length': '0'} 2023-09-22T22:11:18.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000936931d4f0eac951-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.214 DEBUG:boto:path=/ffzdqu-64/obj33 2023-09-22T22:11:18.215 DEBUG:boto:auth_path=/ffzdqu-64/obj33 2023-09-22T22:11:18.215 DEBUG:boto:Method: HEAD 2023-09-22T22:11:18.215 DEBUG:boto:Path: /ffzdqu-64/obj33 2023-09-22T22:11:18.215 DEBUG:boto:Data: 2023-09-22T22:11:18.215 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.215 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.215 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.215 DEBUG:boto:Params: {} 2023-09-22T22:11:18.216 DEBUG:boto:Token: None 2023-09-22T22:11:18.216 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj33 2023-09-22T22:11:18.216 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:0p86mvaEnhR35P9nBEliaWuLsTQ= 2023-09-22T22:11:18.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0p86mvaEnhR35P9nBEliaWuLsTQ=', 'Content-Length': '0'} 2023-09-22T22:11:18.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3ecdfa7b8ee644f-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-22T22:11:18.219 DEBUG:boto:path=/ffzdqu-64/obj33 2023-09-22T22:11:18.219 DEBUG:boto:auth_path=/ffzdqu-64/obj33 2023-09-22T22:11:18.220 DEBUG:boto:Method: GET 2023-09-22T22:11:18.220 DEBUG:boto:Path: /ffzdqu-64/obj33 2023-09-22T22:11:18.220 DEBUG:boto:Data: 2023-09-22T22:11:18.220 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.220 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:18.220 DEBUG:boto:Port: 8001 2023-09-22T22:11:18.220 DEBUG:boto:Params: {} 2023-09-22T22:11:18.220 DEBUG:boto:Token: None 2023-09-22T22:11:18.220 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj33 2023-09-22T22:11:18.221 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TXAsBiDixehk/DPj8QFyPlMmXLo= 2023-09-22T22:11:18.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TXAsBiDixehk/DPj8QFyPlMmXLo=', 'Content-Length': '0'} 2023-09-22T22:11:18.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000227c7baa2b4463cf-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.266 DEBUG:boto:path=/ffzdqu-64/obj33 2023-09-22T22:11:18.267 DEBUG:boto:auth_path=/ffzdqu-64/obj33 2023-09-22T22:11:18.267 DEBUG:boto:Method: GET 2023-09-22T22:11:18.267 DEBUG:boto:Path: /ffzdqu-64/obj33 2023-09-22T22:11:18.267 DEBUG:boto:Data: 2023-09-22T22:11:18.267 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.267 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.267 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.267 DEBUG:boto:Params: {} 2023-09-22T22:11:18.268 DEBUG:boto:Token: None 2023-09-22T22:11:18.268 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj33 2023-09-22T22:11:18.268 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:TXAsBiDixehk/DPj8QFyPlMmXLo= 2023-09-22T22:11:18.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TXAsBiDixehk/DPj8QFyPlMmXLo=', 'Content-Length': '0'} 2023-09-22T22:11:18.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001db047f2c9c68558-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.271 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:18.271 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:18.272 DEBUG:boto:path=/ffzdqu-64/?prefix=obj33 2023-09-22T22:11:18.272 DEBUG:boto:auth_path=/ffzdqu-64/?prefix=obj33 2023-09-22T22:11:18.272 DEBUG:boto:Method: GET 2023-09-22T22:11:18.272 DEBUG:boto:Path: /ffzdqu-64/?prefix=obj33 2023-09-22T22:11:18.272 DEBUG:boto:Data: 2023-09-22T22:11:18.272 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.272 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:18.272 DEBUG:boto:Port: 8001 2023-09-22T22:11:18.272 DEBUG:boto:Params: {} 2023-09-22T22:11:18.273 DEBUG:boto:Token: None 2023-09-22T22:11:18.273 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/ 2023-09-22T22:11:18.273 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:11:18.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:11:18.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000981ab63a22fb03f9-00650e1106-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.278 DEBUG:boto:b'ffzdqu-64obj331000falseobj332023-09-22T22:10:52.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:18.278 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:18.278 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:18.278 DEBUG:boto:path=/ffzdqu-64/?prefix=obj33 2023-09-22T22:11:18.279 DEBUG:boto:auth_path=/ffzdqu-64/?prefix=obj33 2023-09-22T22:11:18.279 DEBUG:boto:Method: GET 2023-09-22T22:11:18.279 DEBUG:boto:Path: /ffzdqu-64/?prefix=obj33 2023-09-22T22:11:18.279 DEBUG:boto:Data: 2023-09-22T22:11:18.279 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.279 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.279 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.279 DEBUG:boto:Params: {} 2023-09-22T22:11:18.279 DEBUG:boto:Token: None 2023-09-22T22:11:18.280 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/ 2023-09-22T22:11:18.280 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:11:18.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:11:18.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b40a942474e045ea-00650e1106-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.326 DEBUG:boto:b'ffzdqu-64obj331000falseobj332023-09-22T22:10:52.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:18.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T22:11:18.327 DEBUG:boto:path=/ffzdqu-64/obj44 2023-09-22T22:11:18.327 DEBUG:boto:auth_path=/ffzdqu-64/obj44 2023-09-22T22:11:18.327 DEBUG:boto:path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.327 DEBUG:boto:auth_path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.327 DEBUG:boto:Method: GET 2023-09-22T22:11:18.327 DEBUG:boto:Path: /ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.328 DEBUG:boto:Data: 2023-09-22T22:11:18.328 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.328 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:18.328 DEBUG:boto:Port: 8001 2023-09-22T22:11:18.328 DEBUG:boto:Params: {} 2023-09-22T22:11:18.328 DEBUG:boto:Token: None 2023-09-22T22:11:18.328 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.328 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yXb8zIL4PX0AVgFTslA41mui64s= 2023-09-22T22:11:18.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yXb8zIL4PX0AVgFTslA41mui64s=', 'Content-Length': '0'} 2023-09-22T22:11:18.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec77702b78f073c3-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.374 DEBUG:boto:path=/ffzdqu-64/obj44 2023-09-22T22:11:18.375 DEBUG:boto:auth_path=/ffzdqu-64/obj44 2023-09-22T22:11:18.375 DEBUG:boto:path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.375 DEBUG:boto:auth_path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.375 DEBUG:boto:Method: GET 2023-09-22T22:11:18.375 DEBUG:boto:Path: /ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.375 DEBUG:boto:Data: 2023-09-22T22:11:18.375 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.375 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.375 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.375 DEBUG:boto:Params: {} 2023-09-22T22:11:18.376 DEBUG:boto:Token: None 2023-09-22T22:11:18.376 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.376 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yXb8zIL4PX0AVgFTslA41mui64s= 2023-09-22T22:11:18.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yXb8zIL4PX0AVgFTslA41mui64s=', 'Content-Length': '0'} 2023-09-22T22:11:18.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b563afac71d444f-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.379 DEBUG:boto:path=/ffzdqu-64/obj44 2023-09-22T22:11:18.380 DEBUG:boto:auth_path=/ffzdqu-64/obj44 2023-09-22T22:11:18.380 DEBUG:boto:path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.380 DEBUG:boto:auth_path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.380 DEBUG:boto:Method: HEAD 2023-09-22T22:11:18.380 DEBUG:boto:Path: /ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.380 DEBUG:boto:Data: 2023-09-22T22:11:18.380 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.380 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:18.380 DEBUG:boto:Port: 8001 2023-09-22T22:11:18.380 DEBUG:boto:Params: {} 2023-09-22T22:11:18.381 DEBUG:boto:Token: None 2023-09-22T22:11:18.381 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.381 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EFrz8+M7D/L5kogGoKGkduN5PAE= 2023-09-22T22:11:18.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EFrz8+M7D/L5kogGoKGkduN5PAE=', 'Content-Length': '0'} 2023-09-22T22:11:18.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000477d7a94e000bc59-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.384 DEBUG:boto:path=/ffzdqu-64/obj44 2023-09-22T22:11:18.384 DEBUG:boto:auth_path=/ffzdqu-64/obj44 2023-09-22T22:11:18.384 DEBUG:boto:path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.385 DEBUG:boto:auth_path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.385 DEBUG:boto:Method: HEAD 2023-09-22T22:11:18.385 DEBUG:boto:Path: /ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.385 DEBUG:boto:Data: 2023-09-22T22:11:18.385 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.385 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.385 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.385 DEBUG:boto:Params: {} 2023-09-22T22:11:18.385 DEBUG:boto:Token: None 2023-09-22T22:11:18.386 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.386 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:EFrz8+M7D/L5kogGoKGkduN5PAE= 2023-09-22T22:11:18.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EFrz8+M7D/L5kogGoKGkduN5PAE=', 'Content-Length': '0'} 2023-09-22T22:11:18.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e768442f3f1ebdb-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T22:11:18.389 DEBUG:boto:path=/ffzdqu-64/obj44 2023-09-22T22:11:18.389 DEBUG:boto:auth_path=/ffzdqu-64/obj44 2023-09-22T22:11:18.389 DEBUG:boto:path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.390 DEBUG:boto:auth_path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.390 DEBUG:boto:Method: GET 2023-09-22T22:11:18.390 DEBUG:boto:Path: /ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.390 DEBUG:boto:Data: 2023-09-22T22:11:18.390 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.390 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:18.390 DEBUG:boto:Port: 8001 2023-09-22T22:11:18.390 DEBUG:boto:Params: {} 2023-09-22T22:11:18.390 DEBUG:boto:Token: None 2023-09-22T22:11:18.391 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.391 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yXb8zIL4PX0AVgFTslA41mui64s= 2023-09-22T22:11:18.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yXb8zIL4PX0AVgFTslA41mui64s=', 'Content-Length': '0'} 2023-09-22T22:11:18.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005abf7f3f3979aa89-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.438 DEBUG:boto:path=/ffzdqu-64/obj44 2023-09-22T22:11:18.439 DEBUG:boto:auth_path=/ffzdqu-64/obj44 2023-09-22T22:11:18.439 DEBUG:boto:path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.439 DEBUG:boto:auth_path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.439 DEBUG:boto:Method: GET 2023-09-22T22:11:18.439 DEBUG:boto:Path: /ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.439 DEBUG:boto:Data: 2023-09-22T22:11:18.439 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.439 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.440 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.440 DEBUG:boto:Params: {} 2023-09-22T22:11:18.440 DEBUG:boto:Token: None 2023-09-22T22:11:18.440 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj44?versionId=null 2023-09-22T22:11:18.440 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:yXb8zIL4PX0AVgFTslA41mui64s= 2023-09-22T22:11:18.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yXb8zIL4PX0AVgFTslA41mui64s=', 'Content-Length': '0'} 2023-09-22T22:11:18.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000566ab893974e345f-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.446 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:18.446 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:18.446 DEBUG:boto:path=/ffzdqu-64/?prefix=obj44 2023-09-22T22:11:18.446 DEBUG:boto:auth_path=/ffzdqu-64/?prefix=obj44 2023-09-22T22:11:18.446 DEBUG:boto:Method: GET 2023-09-22T22:11:18.446 DEBUG:boto:Path: /ffzdqu-64/?prefix=obj44 2023-09-22T22:11:18.446 DEBUG:boto:Data: 2023-09-22T22:11:18.446 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.446 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:18.447 DEBUG:boto:Port: 8001 2023-09-22T22:11:18.447 DEBUG:boto:Params: {} 2023-09-22T22:11:18.447 DEBUG:boto:Token: None 2023-09-22T22:11:18.447 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/ 2023-09-22T22:11:18.447 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:11:18.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:11:18.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba935293d1da6759-00650e1106-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.498 DEBUG:boto:b'ffzdqu-64obj441000falseobj442023-09-22T22:10:52.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:18.499 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:18.499 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:18.499 DEBUG:boto:path=/ffzdqu-64/?prefix=obj44 2023-09-22T22:11:18.499 DEBUG:boto:auth_path=/ffzdqu-64/?prefix=obj44 2023-09-22T22:11:18.499 DEBUG:boto:Method: GET 2023-09-22T22:11:18.500 DEBUG:boto:Path: /ffzdqu-64/?prefix=obj44 2023-09-22T22:11:18.500 DEBUG:boto:Data: 2023-09-22T22:11:18.500 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.500 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.500 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.500 DEBUG:boto:Params: {} 2023-09-22T22:11:18.500 DEBUG:boto:Token: None 2023-09-22T22:11:18.500 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/ 2023-09-22T22:11:18.500 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:11:18.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:11:18.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ed3458085a37b8f-00650e1106-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.505 DEBUG:boto:b'ffzdqu-64obj441000falseobj442023-09-22T22:10:52.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:18.505 DEBUG:boto:path=/ffzdqu-64/obj44 2023-09-22T22:11:18.506 DEBUG:boto:auth_path=/ffzdqu-64/obj44 2023-09-22T22:11:18.506 DEBUG:boto:Method: HEAD 2023-09-22T22:11:18.506 DEBUG:boto:Path: /ffzdqu-64/obj44 2023-09-22T22:11:18.506 DEBUG:boto:Data: 2023-09-22T22:11:18.506 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.506 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:18.506 DEBUG:boto:Port: 8001 2023-09-22T22:11:18.506 DEBUG:boto:Params: {} 2023-09-22T22:11:18.507 DEBUG:boto:Token: None 2023-09-22T22:11:18.507 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj44 2023-09-22T22:11:18.507 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jeDa/cqB1y93s2UXJDHALNQSh8U= 2023-09-22T22:11:18.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jeDa/cqB1y93s2UXJDHALNQSh8U=', 'Content-Length': '0'} 2023-09-22T22:11:18.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001995f0bd9f892cd1-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.510 DEBUG:boto:path=/ffzdqu-64/obj44 2023-09-22T22:11:18.510 DEBUG:boto:auth_path=/ffzdqu-64/obj44 2023-09-22T22:11:18.510 DEBUG:boto:Method: HEAD 2023-09-22T22:11:18.510 DEBUG:boto:Path: /ffzdqu-64/obj44 2023-09-22T22:11:18.510 DEBUG:boto:Data: 2023-09-22T22:11:18.510 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.511 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.511 DEBUG:boto:Params: {} 2023-09-22T22:11:18.511 DEBUG:boto:Token: None 2023-09-22T22:11:18.511 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj44 2023-09-22T22:11:18.511 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:jeDa/cqB1y93s2UXJDHALNQSh8U= 2023-09-22T22:11:18.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jeDa/cqB1y93s2UXJDHALNQSh8U=', 'Content-Length': '0'} 2023-09-22T22:11:18.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d12dfdfd0047492f-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-22T22:11:18.514 DEBUG:boto:path=/ffzdqu-64/obj44 2023-09-22T22:11:18.515 DEBUG:boto:auth_path=/ffzdqu-64/obj44 2023-09-22T22:11:18.515 DEBUG:boto:Method: GET 2023-09-22T22:11:18.515 DEBUG:boto:Path: /ffzdqu-64/obj44 2023-09-22T22:11:18.515 DEBUG:boto:Data: 2023-09-22T22:11:18.515 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.515 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:18.515 DEBUG:boto:Port: 8001 2023-09-22T22:11:18.515 DEBUG:boto:Params: {} 2023-09-22T22:11:18.515 DEBUG:boto:Token: None 2023-09-22T22:11:18.515 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj44 2023-09-22T22:11:18.516 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:osifHDik/HLbMKBiy7FXtBP1jxI= 2023-09-22T22:11:18.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:osifHDik/HLbMKBiy7FXtBP1jxI=', 'Content-Length': '0'} 2023-09-22T22:11:18.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048c66e3d93c6a785-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.562 DEBUG:boto:path=/ffzdqu-64/obj44 2023-09-22T22:11:18.563 DEBUG:boto:auth_path=/ffzdqu-64/obj44 2023-09-22T22:11:18.563 DEBUG:boto:Method: GET 2023-09-22T22:11:18.563 DEBUG:boto:Path: /ffzdqu-64/obj44 2023-09-22T22:11:18.563 DEBUG:boto:Data: 2023-09-22T22:11:18.563 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.563 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.563 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.563 DEBUG:boto:Params: {} 2023-09-22T22:11:18.563 DEBUG:boto:Token: None 2023-09-22T22:11:18.564 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj44 2023-09-22T22:11:18.564 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:osifHDik/HLbMKBiy7FXtBP1jxI= 2023-09-22T22:11:18.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:osifHDik/HLbMKBiy7FXtBP1jxI=', 'Content-Length': '0'} 2023-09-22T22:11:18.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000824e71dc4ef2cecb-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.567 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:18.567 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:18.567 DEBUG:boto:path=/ffzdqu-64/?prefix=obj44 2023-09-22T22:11:18.567 DEBUG:boto:auth_path=/ffzdqu-64/?prefix=obj44 2023-09-22T22:11:18.567 DEBUG:boto:Method: GET 2023-09-22T22:11:18.567 DEBUG:boto:Path: /ffzdqu-64/?prefix=obj44 2023-09-22T22:11:18.567 DEBUG:boto:Data: 2023-09-22T22:11:18.567 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.568 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:18.568 DEBUG:boto:Port: 8001 2023-09-22T22:11:18.568 DEBUG:boto:Params: {} 2023-09-22T22:11:18.568 DEBUG:boto:Token: None 2023-09-22T22:11:18.568 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/ 2023-09-22T22:11:18.568 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:11:18.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:11:18.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000505d4c4cf904ce49-00650e1106-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.574 DEBUG:boto:b'ffzdqu-64obj441000falseobj442023-09-22T22:10:52.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:18.574 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:18.575 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:18.575 DEBUG:boto:path=/ffzdqu-64/?prefix=obj44 2023-09-22T22:11:18.575 DEBUG:boto:auth_path=/ffzdqu-64/?prefix=obj44 2023-09-22T22:11:18.575 DEBUG:boto:Method: GET 2023-09-22T22:11:18.575 DEBUG:boto:Path: /ffzdqu-64/?prefix=obj44 2023-09-22T22:11:18.575 DEBUG:boto:Data: 2023-09-22T22:11:18.575 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.575 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.575 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.576 DEBUG:boto:Params: {} 2023-09-22T22:11:18.576 DEBUG:boto:Token: None 2023-09-22T22:11:18.576 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/ 2023-09-22T22:11:18.576 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:11:18.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:11:18.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a47f2c7b20888279-00650e1106-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.622 DEBUG:boto:b'ffzdqu-64obj441000falseobj442023-09-22T22:10:52.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:18.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T22:11:18.623 DEBUG:boto:path=/ffzdqu-64/obj55 2023-09-22T22:11:18.623 DEBUG:boto:auth_path=/ffzdqu-64/obj55 2023-09-22T22:11:18.623 DEBUG:boto:path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.623 DEBUG:boto:auth_path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.623 DEBUG:boto:Method: GET 2023-09-22T22:11:18.623 DEBUG:boto:Path: /ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.624 DEBUG:boto:Data: 2023-09-22T22:11:18.624 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.624 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:18.624 DEBUG:boto:Port: 8001 2023-09-22T22:11:18.624 DEBUG:boto:Params: {} 2023-09-22T22:11:18.624 DEBUG:boto:Token: None 2023-09-22T22:11:18.624 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.625 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qfw5ymVdzxsE7OnZlBe3N/gB3PM= 2023-09-22T22:11:18.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qfw5ymVdzxsE7OnZlBe3N/gB3PM=', 'Content-Length': '0'} 2023-09-22T22:11:18.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f592ec8c2b881425-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.674 DEBUG:boto:path=/ffzdqu-64/obj55 2023-09-22T22:11:18.675 DEBUG:boto:auth_path=/ffzdqu-64/obj55 2023-09-22T22:11:18.675 DEBUG:boto:path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.675 DEBUG:boto:auth_path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.675 DEBUG:boto:Method: GET 2023-09-22T22:11:18.675 DEBUG:boto:Path: /ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.675 DEBUG:boto:Data: 2023-09-22T22:11:18.675 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.676 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.676 DEBUG:boto:Params: {} 2023-09-22T22:11:18.676 DEBUG:boto:Token: None 2023-09-22T22:11:18.676 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.676 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qfw5ymVdzxsE7OnZlBe3N/gB3PM= 2023-09-22T22:11:18.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qfw5ymVdzxsE7OnZlBe3N/gB3PM=', 'Content-Length': '0'} 2023-09-22T22:11:18.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ede9fd95026a523e-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.680 DEBUG:boto:path=/ffzdqu-64/obj55 2023-09-22T22:11:18.680 DEBUG:boto:auth_path=/ffzdqu-64/obj55 2023-09-22T22:11:18.680 DEBUG:boto:path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.680 DEBUG:boto:auth_path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.680 DEBUG:boto:Method: HEAD 2023-09-22T22:11:18.680 DEBUG:boto:Path: /ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.680 DEBUG:boto:Data: 2023-09-22T22:11:18.680 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.680 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:18.681 DEBUG:boto:Port: 8001 2023-09-22T22:11:18.681 DEBUG:boto:Params: {} 2023-09-22T22:11:18.681 DEBUG:boto:Token: None 2023-09-22T22:11:18.681 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.682 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kygRCK9S92gFrL9ECLO/EiV5Lg0= 2023-09-22T22:11:18.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kygRCK9S92gFrL9ECLO/EiV5Lg0=', 'Content-Length': '0'} 2023-09-22T22:11:18.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b62a92c5f16c8747-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.685 DEBUG:boto:path=/ffzdqu-64/obj55 2023-09-22T22:11:18.685 DEBUG:boto:auth_path=/ffzdqu-64/obj55 2023-09-22T22:11:18.685 DEBUG:boto:path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.685 DEBUG:boto:auth_path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.685 DEBUG:boto:Method: HEAD 2023-09-22T22:11:18.685 DEBUG:boto:Path: /ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.685 DEBUG:boto:Data: 2023-09-22T22:11:18.686 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.686 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.686 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.686 DEBUG:boto:Params: {} 2023-09-22T22:11:18.686 DEBUG:boto:Token: None 2023-09-22T22:11:18.686 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.686 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:kygRCK9S92gFrL9ECLO/EiV5Lg0= 2023-09-22T22:11:18.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kygRCK9S92gFrL9ECLO/EiV5Lg0=', 'Content-Length': '0'} 2023-09-22T22:11:18.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057f94b80ffb4371f-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T22:11:18.690 DEBUG:boto:path=/ffzdqu-64/obj55 2023-09-22T22:11:18.690 DEBUG:boto:auth_path=/ffzdqu-64/obj55 2023-09-22T22:11:18.690 DEBUG:boto:path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.690 DEBUG:boto:auth_path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.690 DEBUG:boto:Method: GET 2023-09-22T22:11:18.690 DEBUG:boto:Path: /ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.690 DEBUG:boto:Data: 2023-09-22T22:11:18.691 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.691 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:18.691 DEBUG:boto:Port: 8001 2023-09-22T22:11:18.691 DEBUG:boto:Params: {} 2023-09-22T22:11:18.691 DEBUG:boto:Token: None 2023-09-22T22:11:18.691 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.691 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qfw5ymVdzxsE7OnZlBe3N/gB3PM= 2023-09-22T22:11:18.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qfw5ymVdzxsE7OnZlBe3N/gB3PM=', 'Content-Length': '0'} 2023-09-22T22:11:18.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083be85d1138ef417-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.738 DEBUG:boto:path=/ffzdqu-64/obj55 2023-09-22T22:11:18.739 DEBUG:boto:auth_path=/ffzdqu-64/obj55 2023-09-22T22:11:18.739 DEBUG:boto:path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.739 DEBUG:boto:auth_path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.739 DEBUG:boto:Method: GET 2023-09-22T22:11:18.739 DEBUG:boto:Path: /ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.739 DEBUG:boto:Data: 2023-09-22T22:11:18.739 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.739 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.740 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.740 DEBUG:boto:Params: {} 2023-09-22T22:11:18.740 DEBUG:boto:Token: None 2023-09-22T22:11:18.740 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj55?versionId=null 2023-09-22T22:11:18.740 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:qfw5ymVdzxsE7OnZlBe3N/gB3PM= 2023-09-22T22:11:18.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qfw5ymVdzxsE7OnZlBe3N/gB3PM=', 'Content-Length': '0'} 2023-09-22T22:11:18.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed46ca68500dc13a-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.744 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:18.744 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:18.744 DEBUG:boto:path=/ffzdqu-64/?prefix=obj55 2023-09-22T22:11:18.744 DEBUG:boto:auth_path=/ffzdqu-64/?prefix=obj55 2023-09-22T22:11:18.744 DEBUG:boto:Method: GET 2023-09-22T22:11:18.744 DEBUG:boto:Path: /ffzdqu-64/?prefix=obj55 2023-09-22T22:11:18.744 DEBUG:boto:Data: 2023-09-22T22:11:18.744 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.745 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:18.745 DEBUG:boto:Port: 8001 2023-09-22T22:11:18.745 DEBUG:boto:Params: {} 2023-09-22T22:11:18.745 DEBUG:boto:Token: None 2023-09-22T22:11:18.745 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/ 2023-09-22T22:11:18.745 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:11:18.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:11:18.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007720670c841af18c-00650e1106-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.794 DEBUG:boto:b'ffzdqu-64obj551000falseobj552023-09-22T22:10:52.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:18.795 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:18.795 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:18.795 DEBUG:boto:path=/ffzdqu-64/?prefix=obj55 2023-09-22T22:11:18.795 DEBUG:boto:auth_path=/ffzdqu-64/?prefix=obj55 2023-09-22T22:11:18.795 DEBUG:boto:Method: GET 2023-09-22T22:11:18.796 DEBUG:boto:Path: /ffzdqu-64/?prefix=obj55 2023-09-22T22:11:18.796 DEBUG:boto:Data: 2023-09-22T22:11:18.796 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.796 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.796 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.796 DEBUG:boto:Params: {} 2023-09-22T22:11:18.796 DEBUG:boto:Token: None 2023-09-22T22:11:18.796 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/ 2023-09-22T22:11:18.797 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:11:18.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:11:18.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b41754d97f2c9949-00650e1106-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.801 DEBUG:boto:b'ffzdqu-64obj551000falseobj552023-09-22T22:10:52.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:18.802 DEBUG:boto:path=/ffzdqu-64/obj55 2023-09-22T22:11:18.802 DEBUG:boto:auth_path=/ffzdqu-64/obj55 2023-09-22T22:11:18.802 DEBUG:boto:Method: HEAD 2023-09-22T22:11:18.802 DEBUG:boto:Path: /ffzdqu-64/obj55 2023-09-22T22:11:18.802 DEBUG:boto:Data: 2023-09-22T22:11:18.803 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.803 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:18.803 DEBUG:boto:Port: 8001 2023-09-22T22:11:18.803 DEBUG:boto:Params: {} 2023-09-22T22:11:18.803 DEBUG:boto:Token: None 2023-09-22T22:11:18.803 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj55 2023-09-22T22:11:18.803 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pecKVGx4oXRPPnAi9iK1/tbmTdk= 2023-09-22T22:11:18.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pecKVGx4oXRPPnAi9iK1/tbmTdk=', 'Content-Length': '0'} 2023-09-22T22:11:18.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2404bb52d56e978-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.806 DEBUG:boto:path=/ffzdqu-64/obj55 2023-09-22T22:11:18.806 DEBUG:boto:auth_path=/ffzdqu-64/obj55 2023-09-22T22:11:18.807 DEBUG:boto:Method: HEAD 2023-09-22T22:11:18.807 DEBUG:boto:Path: /ffzdqu-64/obj55 2023-09-22T22:11:18.807 DEBUG:boto:Data: 2023-09-22T22:11:18.807 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.807 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.807 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.807 DEBUG:boto:Params: {} 2023-09-22T22:11:18.808 DEBUG:boto:Token: None 2023-09-22T22:11:18.808 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj55 2023-09-22T22:11:18.808 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:pecKVGx4oXRPPnAi9iK1/tbmTdk= 2023-09-22T22:11:18.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pecKVGx4oXRPPnAi9iK1/tbmTdk=', 'Content-Length': '0'} 2023-09-22T22:11:18.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004aa0076d7610814b-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-22T22:11:18.811 DEBUG:boto:path=/ffzdqu-64/obj55 2023-09-22T22:11:18.811 DEBUG:boto:auth_path=/ffzdqu-64/obj55 2023-09-22T22:11:18.811 DEBUG:boto:Method: GET 2023-09-22T22:11:18.812 DEBUG:boto:Path: /ffzdqu-64/obj55 2023-09-22T22:11:18.812 DEBUG:boto:Data: 2023-09-22T22:11:18.812 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.812 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:18.812 DEBUG:boto:Port: 8001 2023-09-22T22:11:18.812 DEBUG:boto:Params: {} 2023-09-22T22:11:18.812 DEBUG:boto:Token: None 2023-09-22T22:11:18.812 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj55 2023-09-22T22:11:18.813 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wFqrtU41sGomtABbuMFt4q8MokQ= 2023-09-22T22:11:18.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wFqrtU41sGomtABbuMFt4q8MokQ=', 'Content-Length': '0'} 2023-09-22T22:11:18.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d23acbc3db479ac9-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.858 DEBUG:boto:path=/ffzdqu-64/obj55 2023-09-22T22:11:18.859 DEBUG:boto:auth_path=/ffzdqu-64/obj55 2023-09-22T22:11:18.859 DEBUG:boto:Method: GET 2023-09-22T22:11:18.859 DEBUG:boto:Path: /ffzdqu-64/obj55 2023-09-22T22:11:18.859 DEBUG:boto:Data: 2023-09-22T22:11:18.859 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.860 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.860 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.860 DEBUG:boto:Params: {} 2023-09-22T22:11:18.860 DEBUG:boto:Token: None 2023-09-22T22:11:18.860 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/obj55 2023-09-22T22:11:18.861 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wFqrtU41sGomtABbuMFt4q8MokQ= 2023-09-22T22:11:18.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wFqrtU41sGomtABbuMFt4q8MokQ=', 'Content-Length': '0'} 2023-09-22T22:11:18.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d441ebca96ba323e-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.864 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:18.864 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:18.865 DEBUG:boto:path=/ffzdqu-64/?prefix=obj55 2023-09-22T22:11:18.865 DEBUG:boto:auth_path=/ffzdqu-64/?prefix=obj55 2023-09-22T22:11:18.865 DEBUG:boto:Method: GET 2023-09-22T22:11:18.865 DEBUG:boto:Path: /ffzdqu-64/?prefix=obj55 2023-09-22T22:11:18.865 DEBUG:boto:Data: 2023-09-22T22:11:18.865 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.865 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:18.865 DEBUG:boto:Port: 8001 2023-09-22T22:11:18.866 DEBUG:boto:Params: {} 2023-09-22T22:11:18.866 DEBUG:boto:Token: None 2023-09-22T22:11:18.866 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/ 2023-09-22T22:11:18.866 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:11:18.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:11:18.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b709c4d1688ce1ff-00650e1106-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.914 DEBUG:boto:b'ffzdqu-64obj551000falseobj552023-09-22T22:10:52.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:18.915 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:18.915 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:18.915 DEBUG:boto:path=/ffzdqu-64/?prefix=obj55 2023-09-22T22:11:18.915 DEBUG:boto:auth_path=/ffzdqu-64/?prefix=obj55 2023-09-22T22:11:18.915 DEBUG:boto:Method: GET 2023-09-22T22:11:18.915 DEBUG:boto:Path: /ffzdqu-64/?prefix=obj55 2023-09-22T22:11:18.916 DEBUG:boto:Data: 2023-09-22T22:11:18.916 DEBUG:boto:Headers: {} 2023-09-22T22:11:18.916 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:18.916 DEBUG:boto:Port: 8000 2023-09-22T22:11:18.916 DEBUG:boto:Params: {} 2023-09-22T22:11:18.916 DEBUG:boto:Token: None 2023-09-22T22:11:18.916 DEBUG:boto:StringToSign: GET Fri, 22 Sep 2023 22:11:18 GMT /ffzdqu-64/ 2023-09-22T22:11:18.916 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:11:18.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:11:18.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005398374f8e4f07d2-00650e1106-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:18.921 DEBUG:boto:b'ffzdqu-64obj551000falseobj552023-09-22T22:10:52.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:11:18.922 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ffzdqu-64 zones={a2, a1} 2023-09-22T22:11:18.922 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-09-22T22:11:18.922 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T22:11:18.923 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T22:11:18.922+0000 7f2d6be57640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 85618) UID: 0 2023-09-22T22:11:18.923 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T22:11:18.922+0000 7f2d6f69ea40 -1 shutting down 2023-09-22T22:11:18.948 INFO:tasks.rgw.c1.client.1.smithi160.stdout:./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7f2d4a614640 time 2023-09-22T22:11:18.952159+0000 2023-09-22T22:11:18.948 INFO:tasks.rgw.c1.client.1.smithi160.stdout:./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T22:11:18.951 INFO:tasks.rgw.c1.client.1.smithi160.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:11:18.951 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f2d731c2883] 2023-09-22T22:11:18.951 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f2d731c2a37] 2023-09-22T22:11:18.952 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x55698763b46c] 2023-09-22T22:11:18.952 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x556987a1d910] 2023-09-22T22:11:18.952 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x55698764756e] 2023-09-22T22:11:18.952 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x556987648e7b] 2023-09-22T22:11:18.952 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x556987649ced] 2023-09-22T22:11:18.952 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x556987a50712] 2023-09-22T22:11:18.952 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 9: radosgw(+0x836db8) [0x5569877fbdb8] 2023-09-22T22:11:18.952 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x5569877fc753] 2023-09-22T22:11:18.952 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2d71e91b43] 2023-09-22T22:11:18.953 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2d71f23a00] 2023-09-22T22:11:18.953 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T22:11:18.950+0000 7f2d4a614640 -1 ./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7f2d4a614640 time 2023-09-22T22:11:18.952159+0000 2023-09-22T22:11:18.953 INFO:tasks.rgw.c1.client.1.smithi160.stdout:./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T22:11:18.953 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T22:11:18.953 INFO:tasks.rgw.c1.client.1.smithi160.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:11:18.953 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f2d731c2883] 2023-09-22T22:11:18.953 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f2d731c2a37] 2023-09-22T22:11:18.953 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x55698763b46c] 2023-09-22T22:11:18.953 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x556987a1d910] 2023-09-22T22:11:18.954 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x55698764756e] 2023-09-22T22:11:18.954 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x556987648e7b] 2023-09-22T22:11:18.954 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x556987649ced] 2023-09-22T22:11:18.954 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x556987a50712] 2023-09-22T22:11:18.954 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 9: radosgw(+0x836db8) [0x5569877fbdb8] 2023-09-22T22:11:18.954 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x5569877fc753] 2023-09-22T22:11:18.954 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2d71e91b43] 2023-09-22T22:11:18.954 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2d71f23a00] 2023-09-22T22:11:18.954 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T22:11:18.955 INFO:tasks.rgw.c1.client.1.smithi160.stdout:*** Caught signal (Aborted) ** 2023-09-22T22:11:18.955 INFO:tasks.rgw.c1.client.1.smithi160.stdout: in thread 7f2d4a614640 thread_name:data-sync 2023-09-22T22:11:18.955 INFO:tasks.rgw.c1.client.1.smithi160.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:11:18.955 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2d71e3f520] 2023-09-22T22:11:18.955 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2: pthread_kill() 2023-09-22T22:11:18.956 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 3: raise() 2023-09-22T22:11:18.956 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 4: abort() 2023-09-22T22:11:18.956 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f2d731c28dd] 2023-09-22T22:11:18.956 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f2d731c2a37] 2023-09-22T22:11:18.956 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x55698763b46c] 2023-09-22T22:11:18.956 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x556987a1d910] 2023-09-22T22:11:18.956 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x55698764756e] 2023-09-22T22:11:18.956 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x556987648e7b] 2023-09-22T22:11:18.957 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x556987649ced] 2023-09-22T22:11:18.957 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x556987a50712] 2023-09-22T22:11:18.957 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 13: radosgw(+0x836db8) [0x5569877fbdb8] 2023-09-22T22:11:18.957 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x5569877fc753] 2023-09-22T22:11:18.957 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2d71e91b43] 2023-09-22T22:11:18.957 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2d71f23a00] 2023-09-22T22:11:18.957 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T22:11:18.954+0000 7f2d4a614640 -1 *** Caught signal (Aborted) ** 2023-09-22T22:11:18.957 INFO:tasks.rgw.c1.client.1.smithi160.stdout: in thread 7f2d4a614640 thread_name:data-sync 2023-09-22T22:11:18.958 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T22:11:18.958 INFO:tasks.rgw.c1.client.1.smithi160.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:11:18.958 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2d71e3f520] 2023-09-22T22:11:18.958 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2: pthread_kill() 2023-09-22T22:11:18.958 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 3: raise() 2023-09-22T22:11:18.958 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 4: abort() 2023-09-22T22:11:18.958 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f2d731c28dd] 2023-09-22T22:11:18.958 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f2d731c2a37] 2023-09-22T22:11:18.958 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x55698763b46c] 2023-09-22T22:11:18.959 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x556987a1d910] 2023-09-22T22:11:18.959 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x55698764756e] 2023-09-22T22:11:18.959 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x556987648e7b] 2023-09-22T22:11:18.959 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x556987649ced] 2023-09-22T22:11:18.959 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x556987a50712] 2023-09-22T22:11:18.959 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 13: radosgw(+0x836db8) [0x5569877fbdb8] 2023-09-22T22:11:18.959 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x5569877fc753] 2023-09-22T22:11:18.959 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2d71e91b43] 2023-09-22T22:11:18.960 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2d71f23a00] 2023-09-22T22:11:18.960 INFO:tasks.rgw.c1.client.1.smithi160.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-09-22T22:11:18.960 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T22:11:18.982 INFO:tasks.rgw.c1.client.1.smithi160.stdout: -89> 2023-09-22T22:11:18.922+0000 7f2d6be57640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 85618) UID: 0 2023-09-22T22:11:18.982 INFO:tasks.rgw.c1.client.1.smithi160.stdout: -86> 2023-09-22T22:11:18.922+0000 7f2d6f69ea40 -1 shutting down 2023-09-22T22:11:18.982 INFO:tasks.rgw.c1.client.1.smithi160.stdout: -1> 2023-09-22T22:11:18.950+0000 7f2d4a614640 -1 ./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7f2d4a614640 time 2023-09-22T22:11:18.952159+0000 2023-09-22T22:11:18.982 INFO:tasks.rgw.c1.client.1.smithi160.stdout:./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T22:11:18.982 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T22:11:18.982 INFO:tasks.rgw.c1.client.1.smithi160.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:11:18.982 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f2d731c2883] 2023-09-22T22:11:18.982 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f2d731c2a37] 2023-09-22T22:11:18.983 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x55698763b46c] 2023-09-22T22:11:18.983 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x556987a1d910] 2023-09-22T22:11:18.983 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x55698764756e] 2023-09-22T22:11:18.983 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x556987648e7b] 2023-09-22T22:11:18.983 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x556987649ced] 2023-09-22T22:11:18.983 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x556987a50712] 2023-09-22T22:11:18.983 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 9: radosgw(+0x836db8) [0x5569877fbdb8] 2023-09-22T22:11:18.983 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x5569877fc753] 2023-09-22T22:11:18.983 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2d71e91b43] 2023-09-22T22:11:18.983 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2d71f23a00] 2023-09-22T22:11:18.984 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T22:11:18.984 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 0> 2023-09-22T22:11:18.954+0000 7f2d4a614640 -1 *** Caught signal (Aborted) ** 2023-09-22T22:11:18.984 INFO:tasks.rgw.c1.client.1.smithi160.stdout: in thread 7f2d4a614640 thread_name:data-sync 2023-09-22T22:11:18.984 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T22:11:18.984 INFO:tasks.rgw.c1.client.1.smithi160.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:11:18.984 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2d71e3f520] 2023-09-22T22:11:18.984 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2: pthread_kill() 2023-09-22T22:11:18.984 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 3: raise() 2023-09-22T22:11:18.984 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 4: abort() 2023-09-22T22:11:18.985 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f2d731c28dd] 2023-09-22T22:11:18.985 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f2d731c2a37] 2023-09-22T22:11:18.985 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x55698763b46c] 2023-09-22T22:11:18.985 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x556987a1d910] 2023-09-22T22:11:18.985 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x55698764756e] 2023-09-22T22:11:18.985 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x556987648e7b] 2023-09-22T22:11:18.985 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x556987649ced] 2023-09-22T22:11:18.985 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x556987a50712] 2023-09-22T22:11:18.985 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 13: radosgw(+0x836db8) [0x5569877fbdb8] 2023-09-22T22:11:18.985 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x5569877fc753] 2023-09-22T22:11:18.986 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2d71e91b43] 2023-09-22T22:11:18.986 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2d71f23a00] 2023-09-22T22:11:18.986 INFO:tasks.rgw.c1.client.1.smithi160.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-09-22T22:11:18.986 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T22:11:18.995 INFO:tasks.rgw.c1.client.1.smithi160.stderr:daemon-helper: command crashed with signal 11 2023-09-22T22:11:19.008 INFO:tasks.rgw.c1.client.1:Stopped 2023-09-22T22:11:19.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:11:19.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:11:19.009 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-64 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:11:19.094 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:11:19.094 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:11:19.149 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T22:11:19.150 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-64 2023-09-22T22:11:19.156 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 5 2023-09-22T22:11:19.206 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:19.207 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:19.207 DEBUG:boto:Method: HEAD 2023-09-22T22:11:19.207 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:19.207 DEBUG:boto:Data: 2023-09-22T22:11:19.207 DEBUG:boto:Headers: {} 2023-09-22T22:11:19.207 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.207 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.207 DEBUG:boto:Params: {} 2023-09-22T22:11:19.208 DEBUG:boto:Token: None 2023-09-22T22:11:19.208 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/ 2023-09-22T22:11:19.208 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:11:19.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:11:19.211 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000864eb46c1915c086-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.211 DEBUG:boto:path=/ffzdqu-64/obj17 2023-09-22T22:11:19.211 DEBUG:boto:auth_path=/ffzdqu-64/obj17 2023-09-22T22:11:19.211 DEBUG:boto:Method: PUT 2023-09-22T22:11:19.211 DEBUG:boto:Path: /ffzdqu-64/obj17 2023-09-22T22:11:19.212 DEBUG:boto:Data: 2023-09-22T22:11:19.212 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:19.212 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.212 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.212 DEBUG:boto:Params: {} 2023-09-22T22:11:19.212 DEBUG:boto:Token: None 2023-09-22T22:11:19.212 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/obj17 2023-09-22T22:11:19.212 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:51G4MNs9+OzGbgzOmOvL1XUTpLE= 2023-09-22T22:11:19.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:51G4MNs9+OzGbgzOmOvL1XUTpLE='} 2023-09-22T22:11:19.259 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf91fd22b4bf38df-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.259 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:19.259 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:19.259 DEBUG:boto:Method: HEAD 2023-09-22T22:11:19.259 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:19.259 DEBUG:boto:Data: 2023-09-22T22:11:19.260 DEBUG:boto:Headers: {} 2023-09-22T22:11:19.260 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.260 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.260 DEBUG:boto:Params: {} 2023-09-22T22:11:19.260 DEBUG:boto:Token: None 2023-09-22T22:11:19.260 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/ 2023-09-22T22:11:19.260 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:11:19.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:11:19.263 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000088e3f00435b806c-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.264 DEBUG:boto:path=/ffzdqu-64/obj34 2023-09-22T22:11:19.264 DEBUG:boto:auth_path=/ffzdqu-64/obj34 2023-09-22T22:11:19.264 DEBUG:boto:Method: PUT 2023-09-22T22:11:19.264 DEBUG:boto:Path: /ffzdqu-64/obj34 2023-09-22T22:11:19.264 DEBUG:boto:Data: 2023-09-22T22:11:19.264 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:19.264 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.264 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.264 DEBUG:boto:Params: {} 2023-09-22T22:11:19.265 DEBUG:boto:Token: None 2023-09-22T22:11:19.265 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/obj34 2023-09-22T22:11:19.265 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:W3TunPRVqW0DlVdCaHyK2EI581A= 2023-09-22T22:11:19.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:W3TunPRVqW0DlVdCaHyK2EI581A='} 2023-09-22T22:11:19.275 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007dc371995da22c1d-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.275 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:19.276 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:19.276 DEBUG:boto:Method: HEAD 2023-09-22T22:11:19.276 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:19.276 DEBUG:boto:Data: 2023-09-22T22:11:19.276 DEBUG:boto:Headers: {} 2023-09-22T22:11:19.277 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.277 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.277 DEBUG:boto:Params: {} 2023-09-22T22:11:19.277 DEBUG:boto:Token: None 2023-09-22T22:11:19.277 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/ 2023-09-22T22:11:19.278 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:11:19.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:11:19.283 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083529939ba4b095d-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.283 DEBUG:boto:path=/ffzdqu-64/obj51 2023-09-22T22:11:19.283 DEBUG:boto:auth_path=/ffzdqu-64/obj51 2023-09-22T22:11:19.283 DEBUG:boto:Method: PUT 2023-09-22T22:11:19.284 DEBUG:boto:Path: /ffzdqu-64/obj51 2023-09-22T22:11:19.284 DEBUG:boto:Data: 2023-09-22T22:11:19.284 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:19.284 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.284 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.284 DEBUG:boto:Params: {} 2023-09-22T22:11:19.285 DEBUG:boto:Token: None 2023-09-22T22:11:19.285 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/obj51 2023-09-22T22:11:19.285 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:hcHoliysQcpRPVLpcfQi1p99BnE= 2023-09-22T22:11:19.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hcHoliysQcpRPVLpcfQi1p99BnE='} 2023-09-22T22:11:19.330 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000405e1def44873cb4-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.331 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:19.331 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:19.331 DEBUG:boto:Method: HEAD 2023-09-22T22:11:19.331 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:19.331 DEBUG:boto:Data: 2023-09-22T22:11:19.331 DEBUG:boto:Headers: {} 2023-09-22T22:11:19.331 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.332 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.332 DEBUG:boto:Params: {} 2023-09-22T22:11:19.332 DEBUG:boto:Token: None 2023-09-22T22:11:19.332 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/ 2023-09-22T22:11:19.332 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:11:19.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:11:19.337 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc66fc8148fe6270-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.337 DEBUG:boto:path=/ffzdqu-64/obj68 2023-09-22T22:11:19.338 DEBUG:boto:auth_path=/ffzdqu-64/obj68 2023-09-22T22:11:19.338 DEBUG:boto:Method: PUT 2023-09-22T22:11:19.338 DEBUG:boto:Path: /ffzdqu-64/obj68 2023-09-22T22:11:19.338 DEBUG:boto:Data: 2023-09-22T22:11:19.338 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:19.338 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.338 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.338 DEBUG:boto:Params: {} 2023-09-22T22:11:19.338 DEBUG:boto:Token: None 2023-09-22T22:11:19.339 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/obj68 2023-09-22T22:11:19.339 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:gxqClw28bKlBLOow19ycMImRyuI= 2023-09-22T22:11:19.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gxqClw28bKlBLOow19ycMImRyuI='} 2023-09-22T22:11:19.351 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f87bea913d048ad-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.352 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:19.352 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:19.352 DEBUG:boto:Method: HEAD 2023-09-22T22:11:19.352 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:19.352 DEBUG:boto:Data: 2023-09-22T22:11:19.352 DEBUG:boto:Headers: {} 2023-09-22T22:11:19.352 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.352 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.352 DEBUG:boto:Params: {} 2023-09-22T22:11:19.353 DEBUG:boto:Token: None 2023-09-22T22:11:19.353 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/ 2023-09-22T22:11:19.353 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:11:19.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:11:19.358 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000012d4da905d630b5-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.358 DEBUG:boto:path=/ffzdqu-64/obj85 2023-09-22T22:11:19.359 DEBUG:boto:auth_path=/ffzdqu-64/obj85 2023-09-22T22:11:19.359 DEBUG:boto:Method: PUT 2023-09-22T22:11:19.359 DEBUG:boto:Path: /ffzdqu-64/obj85 2023-09-22T22:11:19.359 DEBUG:boto:Data: 2023-09-22T22:11:19.359 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:19.359 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.359 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.359 DEBUG:boto:Params: {} 2023-09-22T22:11:19.359 DEBUG:boto:Token: None 2023-09-22T22:11:19.360 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/obj85 2023-09-22T22:11:19.360 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:duzIIbxW2hlats4YvsHvdDyhbV0= 2023-09-22T22:11:19.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:duzIIbxW2hlats4YvsHvdDyhbV0='} 2023-09-22T22:11:19.403 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4fde8efff9b9056-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:11:19.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:11:19.403 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-64 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:11:19.451 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:11:19.452 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:11:19.510 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T22:11:19.510 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-64 2023-09-22T22:11:19.521 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 10 2023-09-22T22:11:19.581 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:19.581 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:19.581 DEBUG:boto:Method: HEAD 2023-09-22T22:11:19.581 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:19.581 DEBUG:boto:Data: 2023-09-22T22:11:19.582 DEBUG:boto:Headers: {} 2023-09-22T22:11:19.582 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.582 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.582 DEBUG:boto:Params: {} 2023-09-22T22:11:19.582 DEBUG:boto:Token: None 2023-09-22T22:11:19.582 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/ 2023-09-22T22:11:19.582 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:11:19.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:11:19.586 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4c000295f98c39f-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.586 DEBUG:boto:path=/ffzdqu-64/obj19 2023-09-22T22:11:19.586 DEBUG:boto:auth_path=/ffzdqu-64/obj19 2023-09-22T22:11:19.586 DEBUG:boto:Method: PUT 2023-09-22T22:11:19.586 DEBUG:boto:Path: /ffzdqu-64/obj19 2023-09-22T22:11:19.587 DEBUG:boto:Data: 2023-09-22T22:11:19.587 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:19.587 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.587 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.587 DEBUG:boto:Params: {} 2023-09-22T22:11:19.587 DEBUG:boto:Token: None 2023-09-22T22:11:19.587 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/obj19 2023-09-22T22:11:19.587 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:bEXEWPjjZg3gs0rhY0WV2HZjOHg= 2023-09-22T22:11:19.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bEXEWPjjZg3gs0rhY0WV2HZjOHg='} 2023-09-22T22:11:19.594 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000351b61a2dce15be3-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.595 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:19.595 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:19.595 DEBUG:boto:Method: HEAD 2023-09-22T22:11:19.595 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:19.595 DEBUG:boto:Data: 2023-09-22T22:11:19.595 DEBUG:boto:Headers: {} 2023-09-22T22:11:19.595 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.595 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.596 DEBUG:boto:Params: {} 2023-09-22T22:11:19.596 DEBUG:boto:Token: None 2023-09-22T22:11:19.596 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/ 2023-09-22T22:11:19.596 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:11:19.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:11:19.599 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000316c4f87b60209a8-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.600 DEBUG:boto:path=/ffzdqu-64/obj38 2023-09-22T22:11:19.600 DEBUG:boto:auth_path=/ffzdqu-64/obj38 2023-09-22T22:11:19.600 DEBUG:boto:Method: PUT 2023-09-22T22:11:19.600 DEBUG:boto:Path: /ffzdqu-64/obj38 2023-09-22T22:11:19.600 DEBUG:boto:Data: 2023-09-22T22:11:19.600 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:19.600 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.601 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.601 DEBUG:boto:Params: {} 2023-09-22T22:11:19.601 DEBUG:boto:Token: None 2023-09-22T22:11:19.601 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/obj38 2023-09-22T22:11:19.601 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:OW1u814/jS6GQDR1MV4eezb4N/Y= 2023-09-22T22:11:19.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OW1u814/jS6GQDR1MV4eezb4N/Y='} 2023-09-22T22:11:19.606 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008624fce5341c9e55-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.606 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:19.606 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:19.606 DEBUG:boto:Method: HEAD 2023-09-22T22:11:19.607 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:19.607 DEBUG:boto:Data: 2023-09-22T22:11:19.607 DEBUG:boto:Headers: {} 2023-09-22T22:11:19.607 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.607 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.607 DEBUG:boto:Params: {} 2023-09-22T22:11:19.607 DEBUG:boto:Token: None 2023-09-22T22:11:19.607 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/ 2023-09-22T22:11:19.608 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:11:19.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:11:19.610 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b764dcfc203f9e25-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.611 DEBUG:boto:path=/ffzdqu-64/obj57 2023-09-22T22:11:19.611 DEBUG:boto:auth_path=/ffzdqu-64/obj57 2023-09-22T22:11:19.611 DEBUG:boto:Method: PUT 2023-09-22T22:11:19.611 DEBUG:boto:Path: /ffzdqu-64/obj57 2023-09-22T22:11:19.611 DEBUG:boto:Data: 2023-09-22T22:11:19.611 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:19.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.611 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.612 DEBUG:boto:Params: {} 2023-09-22T22:11:19.612 DEBUG:boto:Token: None 2023-09-22T22:11:19.612 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/obj57 2023-09-22T22:11:19.612 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:w3NizSev4uGnw37FgQOnCmfEFyM= 2023-09-22T22:11:19.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:w3NizSev4uGnw37FgQOnCmfEFyM='} 2023-09-22T22:11:19.659 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d8e6063267c904b-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.659 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:19.659 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:19.659 DEBUG:boto:Method: HEAD 2023-09-22T22:11:19.659 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:19.659 DEBUG:boto:Data: 2023-09-22T22:11:19.659 DEBUG:boto:Headers: {} 2023-09-22T22:11:19.660 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.660 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.660 DEBUG:boto:Params: {} 2023-09-22T22:11:19.660 DEBUG:boto:Token: None 2023-09-22T22:11:19.660 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/ 2023-09-22T22:11:19.660 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:11:19.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:11:19.665 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a06beaec186b5df5-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.665 DEBUG:boto:path=/ffzdqu-64/obj76 2023-09-22T22:11:19.665 DEBUG:boto:auth_path=/ffzdqu-64/obj76 2023-09-22T22:11:19.665 DEBUG:boto:Method: PUT 2023-09-22T22:11:19.665 DEBUG:boto:Path: /ffzdqu-64/obj76 2023-09-22T22:11:19.666 DEBUG:boto:Data: 2023-09-22T22:11:19.666 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:19.666 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.666 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.666 DEBUG:boto:Params: {} 2023-09-22T22:11:19.666 DEBUG:boto:Token: None 2023-09-22T22:11:19.666 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/obj76 2023-09-22T22:11:19.666 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:f5JUBsV7GuxYpUHTc6a2AmvawDs= 2023-09-22T22:11:19.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f5JUBsV7GuxYpUHTc6a2AmvawDs='} 2023-09-22T22:11:19.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed1bc2e33ec443f6-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.678 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:19.678 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:19.678 DEBUG:boto:Method: HEAD 2023-09-22T22:11:19.678 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:19.678 DEBUG:boto:Data: 2023-09-22T22:11:19.679 DEBUG:boto:Headers: {} 2023-09-22T22:11:19.679 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.679 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.679 DEBUG:boto:Params: {} 2023-09-22T22:11:19.679 DEBUG:boto:Token: None 2023-09-22T22:11:19.679 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/ 2023-09-22T22:11:19.679 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:11:19.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:11:19.685 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c705ed9b90557d6-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.685 DEBUG:boto:path=/ffzdqu-64/obj95 2023-09-22T22:11:19.685 DEBUG:boto:auth_path=/ffzdqu-64/obj95 2023-09-22T22:11:19.685 DEBUG:boto:Method: PUT 2023-09-22T22:11:19.685 DEBUG:boto:Path: /ffzdqu-64/obj95 2023-09-22T22:11:19.685 DEBUG:boto:Data: 2023-09-22T22:11:19.686 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:19.686 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.686 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.686 DEBUG:boto:Params: {} 2023-09-22T22:11:19.686 DEBUG:boto:Token: None 2023-09-22T22:11:19.686 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/obj95 2023-09-22T22:11:19.686 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:M88i8El7nKMgLfKYmB+L411DuV4= 2023-09-22T22:11:19.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M88i8El7nKMgLfKYmB+L411DuV4='} 2023-09-22T22:11:19.734 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a9110a0c9734991-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:11:19.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:11:19.735 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-64 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:11:19.782 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:11:19.783 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:11:19.844 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T22:11:19.844 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-64 2023-09-22T22:11:19.855 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 15 2023-09-22T22:11:19.920 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:19.921 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:19.921 DEBUG:boto:Method: HEAD 2023-09-22T22:11:19.921 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:19.921 DEBUG:boto:Data: 2023-09-22T22:11:19.921 DEBUG:boto:Headers: {} 2023-09-22T22:11:19.921 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.921 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.921 DEBUG:boto:Params: {} 2023-09-22T22:11:19.922 DEBUG:boto:Token: None 2023-09-22T22:11:19.922 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/ 2023-09-22T22:11:19.922 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:11:19.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:11:19.925 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006cf2e82e4b085f4a-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.926 DEBUG:boto:path=/ffzdqu-64/obj23 2023-09-22T22:11:19.926 DEBUG:boto:auth_path=/ffzdqu-64/obj23 2023-09-22T22:11:19.926 DEBUG:boto:Method: PUT 2023-09-22T22:11:19.926 DEBUG:boto:Path: /ffzdqu-64/obj23 2023-09-22T22:11:19.926 DEBUG:boto:Data: 2023-09-22T22:11:19.926 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:19.926 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.926 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.927 DEBUG:boto:Params: {} 2023-09-22T22:11:19.927 DEBUG:boto:Token: None 2023-09-22T22:11:19.927 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/obj23 2023-09-22T22:11:19.927 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:myc2/IT0K5WD8RbZ9jzBHD6mVaQ= 2023-09-22T22:11:19.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:myc2/IT0K5WD8RbZ9jzBHD6mVaQ='} 2023-09-22T22:11:19.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a77873d90c1c40b2-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.934 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:19.934 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:19.934 DEBUG:boto:Method: HEAD 2023-09-22T22:11:19.934 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:19.934 DEBUG:boto:Data: 2023-09-22T22:11:19.934 DEBUG:boto:Headers: {} 2023-09-22T22:11:19.934 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.935 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.935 DEBUG:boto:Params: {} 2023-09-22T22:11:19.935 DEBUG:boto:Token: None 2023-09-22T22:11:19.935 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/ 2023-09-22T22:11:19.935 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:11:19.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:11:19.940 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060b28c2ed159fc5e-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.940 DEBUG:boto:path=/ffzdqu-64/obj46 2023-09-22T22:11:19.941 DEBUG:boto:auth_path=/ffzdqu-64/obj46 2023-09-22T22:11:19.941 DEBUG:boto:Method: PUT 2023-09-22T22:11:19.941 DEBUG:boto:Path: /ffzdqu-64/obj46 2023-09-22T22:11:19.941 DEBUG:boto:Data: 2023-09-22T22:11:19.941 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:19.941 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.941 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.941 DEBUG:boto:Params: {} 2023-09-22T22:11:19.941 DEBUG:boto:Token: None 2023-09-22T22:11:19.942 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/obj46 2023-09-22T22:11:19.942 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:3CioJa78pcejkhCmkdl52hUwsgo= 2023-09-22T22:11:19.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3CioJa78pcejkhCmkdl52hUwsgo='} 2023-09-22T22:11:19.948 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5e2e1684cf0bb87-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.948 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:19.948 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:19.948 DEBUG:boto:Method: HEAD 2023-09-22T22:11:19.948 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:19.949 DEBUG:boto:Data: 2023-09-22T22:11:19.949 DEBUG:boto:Headers: {} 2023-09-22T22:11:19.949 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.949 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.949 DEBUG:boto:Params: {} 2023-09-22T22:11:19.949 DEBUG:boto:Token: None 2023-09-22T22:11:19.949 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/ 2023-09-22T22:11:19.949 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:11:19.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:11:19.952 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008cc678f64147350c-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.952 DEBUG:boto:path=/ffzdqu-64/obj69 2023-09-22T22:11:19.953 DEBUG:boto:auth_path=/ffzdqu-64/obj69 2023-09-22T22:11:19.953 DEBUG:boto:Method: PUT 2023-09-22T22:11:19.953 DEBUG:boto:Path: /ffzdqu-64/obj69 2023-09-22T22:11:19.953 DEBUG:boto:Data: 2023-09-22T22:11:19.953 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:19.953 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:19.953 DEBUG:boto:Port: 8000 2023-09-22T22:11:19.953 DEBUG:boto:Params: {} 2023-09-22T22:11:19.954 DEBUG:boto:Token: None 2023-09-22T22:11:19.954 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:19 GMT /ffzdqu-64/obj69 2023-09-22T22:11:19.955 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:cpWSblEFkwFoIAIKskB5tWcPoHc= 2023-09-22T22:11:19.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cpWSblEFkwFoIAIKskB5tWcPoHc='} 2023-09-22T22:11:19.999 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eea9cb61e02bb745-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:19.999 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:19.999 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:19.999 DEBUG:boto:Method: HEAD 2023-09-22T22:11:19.999 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:19.999 DEBUG:boto:Data: 2023-09-22T22:11:20.000 DEBUG:boto:Headers: {} 2023-09-22T22:11:20.000 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.000 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.000 DEBUG:boto:Params: {} 2023-09-22T22:11:20.000 DEBUG:boto:Token: None 2023-09-22T22:11:20.000 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/ 2023-09-22T22:11:20.000 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:11:20.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:11:20.005 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7cb73e51d20687f-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.005 DEBUG:boto:path=/ffzdqu-64/obj92 2023-09-22T22:11:20.006 DEBUG:boto:auth_path=/ffzdqu-64/obj92 2023-09-22T22:11:20.006 DEBUG:boto:Method: PUT 2023-09-22T22:11:20.006 DEBUG:boto:Path: /ffzdqu-64/obj92 2023-09-22T22:11:20.006 DEBUG:boto:Data: 2023-09-22T22:11:20.006 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:20.007 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.007 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.007 DEBUG:boto:Params: {} 2023-09-22T22:11:20.007 DEBUG:boto:Token: None 2023-09-22T22:11:20.007 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/obj92 2023-09-22T22:11:20.007 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:YhunsMRnF/ODwoEAUMwU399wz/Y= 2023-09-22T22:11:20.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YhunsMRnF/ODwoEAUMwU399wz/Y='} 2023-09-22T22:11:20.017 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d9c5f2443e34852b-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.017 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:20.018 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:20.018 DEBUG:boto:Method: HEAD 2023-09-22T22:11:20.018 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:20.018 DEBUG:boto:Data: 2023-09-22T22:11:20.018 DEBUG:boto:Headers: {} 2023-09-22T22:11:20.018 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.018 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.018 DEBUG:boto:Params: {} 2023-09-22T22:11:20.019 DEBUG:boto:Token: None 2023-09-22T22:11:20.019 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/ 2023-09-22T22:11:20.019 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:11:20.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:11:20.023 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee2b3bdc7824e9ac-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.023 DEBUG:boto:path=/ffzdqu-64/obj115 2023-09-22T22:11:20.023 DEBUG:boto:auth_path=/ffzdqu-64/obj115 2023-09-22T22:11:20.024 DEBUG:boto:Method: PUT 2023-09-22T22:11:20.024 DEBUG:boto:Path: /ffzdqu-64/obj115 2023-09-22T22:11:20.024 DEBUG:boto:Data: 2023-09-22T22:11:20.024 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:20.024 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.025 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.025 DEBUG:boto:Params: {} 2023-09-22T22:11:20.025 DEBUG:boto:Token: None 2023-09-22T22:11:20.025 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/obj115 2023-09-22T22:11:20.025 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RPUpmq1Asiq8XNxUdq7WmMZB7xw= 2023-09-22T22:11:20.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RPUpmq1Asiq8XNxUdq7WmMZB7xw='} 2023-09-22T22:11:20.075 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d84974711bd4ea01-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:11:20.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:11:20.076 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-64 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:11:20.123 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:11:20.123 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:11:20.186 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T22:11:20.187 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-64 2023-09-22T22:11:20.202 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 20 2023-09-22T22:11:20.273 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:20.273 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:20.273 DEBUG:boto:Method: HEAD 2023-09-22T22:11:20.273 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:20.273 DEBUG:boto:Data: 2023-09-22T22:11:20.273 DEBUG:boto:Headers: {} 2023-09-22T22:11:20.273 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.273 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.274 DEBUG:boto:Params: {} 2023-09-22T22:11:20.274 DEBUG:boto:Token: None 2023-09-22T22:11:20.274 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/ 2023-09-22T22:11:20.274 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:11:20.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:11:20.279 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000da16c860b32405d-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.279 DEBUG:boto:path=/ffzdqu-64/obj29 2023-09-22T22:11:20.279 DEBUG:boto:auth_path=/ffzdqu-64/obj29 2023-09-22T22:11:20.279 DEBUG:boto:Method: PUT 2023-09-22T22:11:20.280 DEBUG:boto:Path: /ffzdqu-64/obj29 2023-09-22T22:11:20.280 DEBUG:boto:Data: 2023-09-22T22:11:20.280 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:20.280 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.280 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.280 DEBUG:boto:Params: {} 2023-09-22T22:11:20.280 DEBUG:boto:Token: None 2023-09-22T22:11:20.280 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/obj29 2023-09-22T22:11:20.281 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:WQhxZ97zygogSjGZgjPYuGitakA= 2023-09-22T22:11:20.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WQhxZ97zygogSjGZgjPYuGitakA='} 2023-09-22T22:11:20.287 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd1adfb02a3f6dce-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.287 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:20.287 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:20.288 DEBUG:boto:Method: HEAD 2023-09-22T22:11:20.288 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:20.288 DEBUG:boto:Data: 2023-09-22T22:11:20.288 DEBUG:boto:Headers: {} 2023-09-22T22:11:20.288 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.288 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.288 DEBUG:boto:Params: {} 2023-09-22T22:11:20.288 DEBUG:boto:Token: None 2023-09-22T22:11:20.289 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/ 2023-09-22T22:11:20.289 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:11:20.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:11:20.294 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034cf21b1421e4b46-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.294 DEBUG:boto:path=/ffzdqu-64/obj58 2023-09-22T22:11:20.295 DEBUG:boto:auth_path=/ffzdqu-64/obj58 2023-09-22T22:11:20.297 DEBUG:boto:Method: PUT 2023-09-22T22:11:20.297 DEBUG:boto:Path: /ffzdqu-64/obj58 2023-09-22T22:11:20.297 DEBUG:boto:Data: 2023-09-22T22:11:20.298 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:20.298 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.298 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.299 DEBUG:boto:Params: {} 2023-09-22T22:11:20.299 DEBUG:boto:Token: None 2023-09-22T22:11:20.299 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/obj58 2023-09-22T22:11:20.300 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:nVYbdUuYDCfAF0gl6j9IaVLN+oU= 2023-09-22T22:11:20.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nVYbdUuYDCfAF0gl6j9IaVLN+oU='} 2023-09-22T22:11:20.347 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6d1d288f7666370-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.347 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:20.347 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:20.347 DEBUG:boto:Method: HEAD 2023-09-22T22:11:20.347 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:20.347 DEBUG:boto:Data: 2023-09-22T22:11:20.347 DEBUG:boto:Headers: {} 2023-09-22T22:11:20.348 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.348 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.348 DEBUG:boto:Params: {} 2023-09-22T22:11:20.348 DEBUG:boto:Token: None 2023-09-22T22:11:20.348 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/ 2023-09-22T22:11:20.348 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:11:20.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:11:20.354 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cae232a1e54ee96c-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.355 DEBUG:boto:path=/ffzdqu-64/obj87 2023-09-22T22:11:20.355 DEBUG:boto:auth_path=/ffzdqu-64/obj87 2023-09-22T22:11:20.355 DEBUG:boto:Method: PUT 2023-09-22T22:11:20.355 DEBUG:boto:Path: /ffzdqu-64/obj87 2023-09-22T22:11:20.355 DEBUG:boto:Data: 2023-09-22T22:11:20.355 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:20.355 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.355 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.355 DEBUG:boto:Params: {} 2023-09-22T22:11:20.356 DEBUG:boto:Token: None 2023-09-22T22:11:20.356 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/obj87 2023-09-22T22:11:20.356 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:RbrG9jNqwJoMKo6vUmik8F4t1fE= 2023-09-22T22:11:20.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RbrG9jNqwJoMKo6vUmik8F4t1fE='} 2023-09-22T22:11:20.367 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000007f301ffa6449c28-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.367 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:20.367 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:20.367 DEBUG:boto:Method: HEAD 2023-09-22T22:11:20.367 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:20.367 DEBUG:boto:Data: 2023-09-22T22:11:20.367 DEBUG:boto:Headers: {} 2023-09-22T22:11:20.368 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.368 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.368 DEBUG:boto:Params: {} 2023-09-22T22:11:20.368 DEBUG:boto:Token: None 2023-09-22T22:11:20.368 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/ 2023-09-22T22:11:20.368 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:11:20.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:11:20.374 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001fb732e46a0005bb-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.375 DEBUG:boto:path=/ffzdqu-64/obj116 2023-09-22T22:11:20.375 DEBUG:boto:auth_path=/ffzdqu-64/obj116 2023-09-22T22:11:20.375 DEBUG:boto:Method: PUT 2023-09-22T22:11:20.375 DEBUG:boto:Path: /ffzdqu-64/obj116 2023-09-22T22:11:20.375 DEBUG:boto:Data: 2023-09-22T22:11:20.375 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:20.375 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.375 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.375 DEBUG:boto:Params: {} 2023-09-22T22:11:20.376 DEBUG:boto:Token: None 2023-09-22T22:11:20.376 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/obj116 2023-09-22T22:11:20.376 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:8k5gHVVbpx5zQ2PcvlrajB8YvMA= 2023-09-22T22:11:20.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8k5gHVVbpx5zQ2PcvlrajB8YvMA='} 2023-09-22T22:11:20.423 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8a0490983b85786-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.423 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:20.423 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:20.423 DEBUG:boto:Method: HEAD 2023-09-22T22:11:20.423 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:20.423 DEBUG:boto:Data: 2023-09-22T22:11:20.423 DEBUG:boto:Headers: {} 2023-09-22T22:11:20.424 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.424 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.424 DEBUG:boto:Params: {} 2023-09-22T22:11:20.424 DEBUG:boto:Token: None 2023-09-22T22:11:20.424 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/ 2023-09-22T22:11:20.424 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:11:20.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:11:20.431 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c7593cf449ff543-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.432 DEBUG:boto:path=/ffzdqu-64/obj145 2023-09-22T22:11:20.432 DEBUG:boto:auth_path=/ffzdqu-64/obj145 2023-09-22T22:11:20.432 DEBUG:boto:Method: PUT 2023-09-22T22:11:20.432 DEBUG:boto:Path: /ffzdqu-64/obj145 2023-09-22T22:11:20.432 DEBUG:boto:Data: 2023-09-22T22:11:20.432 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:20.432 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.432 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.432 DEBUG:boto:Params: {} 2023-09-22T22:11:20.433 DEBUG:boto:Token: None 2023-09-22T22:11:20.433 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/obj145 2023-09-22T22:11:20.433 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Rx3nyXHuIX1M0FeAt3zbF1qp0Og= 2023-09-22T22:11:20.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rx3nyXHuIX1M0FeAt3zbF1qp0Og='} 2023-09-22T22:11:20.445 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093b6d87b43a42942-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:11:20.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:11:20.446 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-64 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:11:20.491 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:11:20.491 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:11:20.555 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T22:11:20.555 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-64 2023-09-22T22:11:20.573 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 25 2023-09-22T22:11:20.655 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:20.655 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:20.655 DEBUG:boto:Method: HEAD 2023-09-22T22:11:20.656 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:20.656 DEBUG:boto:Data: 2023-09-22T22:11:20.656 DEBUG:boto:Headers: {} 2023-09-22T22:11:20.656 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.656 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.656 DEBUG:boto:Params: {} 2023-09-22T22:11:20.656 DEBUG:boto:Token: None 2023-09-22T22:11:20.656 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/ 2023-09-22T22:11:20.657 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:11:20.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:11:20.661 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bae4091c369704a1-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.661 DEBUG:boto:path=/ffzdqu-64/obj31 2023-09-22T22:11:20.661 DEBUG:boto:auth_path=/ffzdqu-64/obj31 2023-09-22T22:11:20.661 DEBUG:boto:Method: PUT 2023-09-22T22:11:20.661 DEBUG:boto:Path: /ffzdqu-64/obj31 2023-09-22T22:11:20.661 DEBUG:boto:Data: 2023-09-22T22:11:20.662 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:20.662 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.662 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.662 DEBUG:boto:Params: {} 2023-09-22T22:11:20.662 DEBUG:boto:Token: None 2023-09-22T22:11:20.662 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/obj31 2023-09-22T22:11:20.662 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:2l2I8YLS6znPpkzDGcW64sfw+hI= 2023-09-22T22:11:20.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2l2I8YLS6znPpkzDGcW64sfw+hI='} 2023-09-22T22:11:20.707 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029e20422c37387ba-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.707 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:20.707 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:20.707 DEBUG:boto:Method: HEAD 2023-09-22T22:11:20.707 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:20.707 DEBUG:boto:Data: 2023-09-22T22:11:20.707 DEBUG:boto:Headers: {} 2023-09-22T22:11:20.707 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.709 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.709 DEBUG:boto:Params: {} 2023-09-22T22:11:20.709 DEBUG:boto:Token: None 2023-09-22T22:11:20.709 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/ 2023-09-22T22:11:20.709 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:11:20.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:11:20.714 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034fd207658c8bf67-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.715 DEBUG:boto:path=/ffzdqu-64/obj62 2023-09-22T22:11:20.715 DEBUG:boto:auth_path=/ffzdqu-64/obj62 2023-09-22T22:11:20.715 DEBUG:boto:Method: PUT 2023-09-22T22:11:20.715 DEBUG:boto:Path: /ffzdqu-64/obj62 2023-09-22T22:11:20.715 DEBUG:boto:Data: 2023-09-22T22:11:20.715 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:20.715 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.716 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.716 DEBUG:boto:Params: {} 2023-09-22T22:11:20.716 DEBUG:boto:Token: None 2023-09-22T22:11:20.716 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/obj62 2023-09-22T22:11:20.716 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:1MmzyMz1YoIk8nB2BlThSI8G1ko= 2023-09-22T22:11:20.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1MmzyMz1YoIk8nB2BlThSI8G1ko='} 2023-09-22T22:11:20.726 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fcd231d1a444b22b-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.726 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:20.727 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:20.727 DEBUG:boto:Method: HEAD 2023-09-22T22:11:20.727 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:20.727 DEBUG:boto:Data: 2023-09-22T22:11:20.727 DEBUG:boto:Headers: {} 2023-09-22T22:11:20.727 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.727 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.727 DEBUG:boto:Params: {} 2023-09-22T22:11:20.728 DEBUG:boto:Token: None 2023-09-22T22:11:20.728 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/ 2023-09-22T22:11:20.728 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:11:20.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:11:20.734 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ab566e2ac165413-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.735 DEBUG:boto:path=/ffzdqu-64/obj93 2023-09-22T22:11:20.735 DEBUG:boto:auth_path=/ffzdqu-64/obj93 2023-09-22T22:11:20.735 DEBUG:boto:Method: PUT 2023-09-22T22:11:20.735 DEBUG:boto:Path: /ffzdqu-64/obj93 2023-09-22T22:11:20.735 DEBUG:boto:Data: 2023-09-22T22:11:20.735 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:20.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.736 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.736 DEBUG:boto:Params: {} 2023-09-22T22:11:20.736 DEBUG:boto:Token: None 2023-09-22T22:11:20.736 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/obj93 2023-09-22T22:11:20.736 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:mBaF4+khmml0bx8FgI7x2bxqoaI= 2023-09-22T22:11:20.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mBaF4+khmml0bx8FgI7x2bxqoaI='} 2023-09-22T22:11:20.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000302406f1d9f88ca1-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.783 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:20.783 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:20.783 DEBUG:boto:Method: HEAD 2023-09-22T22:11:20.783 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:20.783 DEBUG:boto:Data: 2023-09-22T22:11:20.783 DEBUG:boto:Headers: {} 2023-09-22T22:11:20.784 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.784 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.784 DEBUG:boto:Params: {} 2023-09-22T22:11:20.784 DEBUG:boto:Token: None 2023-09-22T22:11:20.784 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/ 2023-09-22T22:11:20.784 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:11:20.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:11:20.791 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d200fc283554c375-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.792 DEBUG:boto:path=/ffzdqu-64/obj124 2023-09-22T22:11:20.792 DEBUG:boto:auth_path=/ffzdqu-64/obj124 2023-09-22T22:11:20.792 DEBUG:boto:Method: PUT 2023-09-22T22:11:20.792 DEBUG:boto:Path: /ffzdqu-64/obj124 2023-09-22T22:11:20.793 DEBUG:boto:Data: 2023-09-22T22:11:20.793 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:20.793 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.793 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.793 DEBUG:boto:Params: {} 2023-09-22T22:11:20.794 DEBUG:boto:Token: None 2023-09-22T22:11:20.794 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/obj124 2023-09-22T22:11:20.794 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:eQPuXu5tGbUeA8ckR3vPWyYiM28= 2023-09-22T22:11:20.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eQPuXu5tGbUeA8ckR3vPWyYiM28='} 2023-09-22T22:11:20.808 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001fb4be8727becbd7-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.808 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:20.809 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:20.809 DEBUG:boto:Method: HEAD 2023-09-22T22:11:20.809 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:20.809 DEBUG:boto:Data: 2023-09-22T22:11:20.809 DEBUG:boto:Headers: {} 2023-09-22T22:11:20.809 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.809 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.809 DEBUG:boto:Params: {} 2023-09-22T22:11:20.809 DEBUG:boto:Token: None 2023-09-22T22:11:20.810 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/ 2023-09-22T22:11:20.810 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:11:20.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:11:20.817 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083a818cc1349e18e-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.817 DEBUG:boto:path=/ffzdqu-64/obj155 2023-09-22T22:11:20.818 DEBUG:boto:auth_path=/ffzdqu-64/obj155 2023-09-22T22:11:20.818 DEBUG:boto:Method: PUT 2023-09-22T22:11:20.818 DEBUG:boto:Path: /ffzdqu-64/obj155 2023-09-22T22:11:20.818 DEBUG:boto:Data: 2023-09-22T22:11:20.818 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:20.818 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:20.818 DEBUG:boto:Port: 8000 2023-09-22T22:11:20.818 DEBUG:boto:Params: {} 2023-09-22T22:11:20.818 DEBUG:boto:Token: None 2023-09-22T22:11:20.819 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:20 GMT /ffzdqu-64/obj155 2023-09-22T22:11:20.819 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:T79vyoVpyG+hR/RGg3NElqmiCmE= 2023-09-22T22:11:20.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T79vyoVpyG+hR/RGg3NElqmiCmE='} 2023-09-22T22:11:20.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b135cd71c1c2ec0-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:20.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:11:20.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:11:20.867 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-64 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:11:20.913 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:11:20.914 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:11:20.980 INFO:teuthology.orchestra.run.smithi160.stdout:tenant: 2023-09-22T22:11:20.981 INFO:teuthology.orchestra.run.smithi160.stdout:bucket name: ffzdqu-64 2023-09-22T22:11:21.000 INFO:teuthology.orchestra.run.smithi160.stdout:total entries: 30 2023-09-22T22:11:21.081 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:21.195 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:21.195 DEBUG:boto:Method: HEAD 2023-09-22T22:11:21.195 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:21.195 DEBUG:boto:Data: 2023-09-22T22:11:21.196 DEBUG:boto:Headers: {} 2023-09-22T22:11:21.196 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:21.196 DEBUG:boto:Port: 8000 2023-09-22T22:11:21.196 DEBUG:boto:Params: {} 2023-09-22T22:11:21.196 DEBUG:boto:Token: None 2023-09-22T22:11:21.196 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:21 GMT /ffzdqu-64/ 2023-09-22T22:11:21.196 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jz5mny6Cgmzz+yJfi986d2+CwSM= 2023-09-22T22:11:21.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jz5mny6Cgmzz+yJfi986d2+CwSM=', 'Content-Length': '0'} 2023-09-22T22:11:21.204 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f7f8c8d9fcc0f7a-00650e1109-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:21.205 DEBUG:boto:path=/ffzdqu-64/obj37 2023-09-22T22:11:21.205 DEBUG:boto:auth_path=/ffzdqu-64/obj37 2023-09-22T22:11:21.205 DEBUG:boto:Method: PUT 2023-09-22T22:11:21.205 DEBUG:boto:Path: /ffzdqu-64/obj37 2023-09-22T22:11:21.205 DEBUG:boto:Data: 2023-09-22T22:11:21.205 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:21.205 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:21.206 DEBUG:boto:Port: 8000 2023-09-22T22:11:21.206 DEBUG:boto:Params: {} 2023-09-22T22:11:21.206 DEBUG:boto:Token: None 2023-09-22T22:11:21.206 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:21 GMT /ffzdqu-64/obj37 2023-09-22T22:11:21.206 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:BUEIpfRSnv6tJTjf1yKGk7A77/0= 2023-09-22T22:11:21.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BUEIpfRSnv6tJTjf1yKGk7A77/0='} 2023-09-22T22:11:21.219 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000028601734d7a73f62-00650e1109-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:21.219 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:21.219 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:21.220 DEBUG:boto:Method: HEAD 2023-09-22T22:11:21.220 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:21.220 DEBUG:boto:Data: 2023-09-22T22:11:21.220 DEBUG:boto:Headers: {} 2023-09-22T22:11:21.220 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:21.220 DEBUG:boto:Port: 8000 2023-09-22T22:11:21.220 DEBUG:boto:Params: {} 2023-09-22T22:11:21.220 DEBUG:boto:Token: None 2023-09-22T22:11:21.221 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:21 GMT /ffzdqu-64/ 2023-09-22T22:11:21.221 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jz5mny6Cgmzz+yJfi986d2+CwSM= 2023-09-22T22:11:21.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jz5mny6Cgmzz+yJfi986d2+CwSM=', 'Content-Length': '0'} 2023-09-22T22:11:21.228 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1cc1ef158d949df-00650e1109-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:21.229 DEBUG:boto:path=/ffzdqu-64/obj74 2023-09-22T22:11:21.229 DEBUG:boto:auth_path=/ffzdqu-64/obj74 2023-09-22T22:11:21.229 DEBUG:boto:Method: PUT 2023-09-22T22:11:21.229 DEBUG:boto:Path: /ffzdqu-64/obj74 2023-09-22T22:11:21.229 DEBUG:boto:Data: 2023-09-22T22:11:21.229 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:21.229 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:21.230 DEBUG:boto:Port: 8000 2023-09-22T22:11:21.230 DEBUG:boto:Params: {} 2023-09-22T22:11:21.230 DEBUG:boto:Token: None 2023-09-22T22:11:21.230 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:21 GMT /ffzdqu-64/obj74 2023-09-22T22:11:21.230 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:ESet2+5FgbAgNhA3pEw1HyTFz4g= 2023-09-22T22:11:21.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ESet2+5FgbAgNhA3pEw1HyTFz4g='} 2023-09-22T22:11:21.278 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003eb43a1dd9d6ea49-00650e1109-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:21.279 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:21.279 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:21.279 DEBUG:boto:Method: HEAD 2023-09-22T22:11:21.279 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:21.279 DEBUG:boto:Data: 2023-09-22T22:11:21.279 DEBUG:boto:Headers: {} 2023-09-22T22:11:21.280 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:21.280 DEBUG:boto:Port: 8000 2023-09-22T22:11:21.280 DEBUG:boto:Params: {} 2023-09-22T22:11:21.280 DEBUG:boto:Token: None 2023-09-22T22:11:21.280 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:21 GMT /ffzdqu-64/ 2023-09-22T22:11:21.280 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jz5mny6Cgmzz+yJfi986d2+CwSM= 2023-09-22T22:11:21.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jz5mny6Cgmzz+yJfi986d2+CwSM=', 'Content-Length': '0'} 2023-09-22T22:11:21.289 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ced8f719512cde88-00650e1109-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:21.290 DEBUG:boto:path=/ffzdqu-64/obj111 2023-09-22T22:11:21.290 DEBUG:boto:auth_path=/ffzdqu-64/obj111 2023-09-22T22:11:21.290 DEBUG:boto:Method: PUT 2023-09-22T22:11:21.290 DEBUG:boto:Path: /ffzdqu-64/obj111 2023-09-22T22:11:21.290 DEBUG:boto:Data: 2023-09-22T22:11:21.291 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:21.291 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:21.291 DEBUG:boto:Port: 8000 2023-09-22T22:11:21.291 DEBUG:boto:Params: {} 2023-09-22T22:11:21.291 DEBUG:boto:Token: None 2023-09-22T22:11:21.291 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:21 GMT /ffzdqu-64/obj111 2023-09-22T22:11:21.291 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:G8kPo6okSKLGQm4i5rorEdhkzOw= 2023-09-22T22:11:21.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G8kPo6okSKLGQm4i5rorEdhkzOw='} 2023-09-22T22:11:21.305 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009ca111ebb2bda01-00650e1109-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:21.305 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:21.305 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:21.305 DEBUG:boto:Method: HEAD 2023-09-22T22:11:21.305 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:21.305 DEBUG:boto:Data: 2023-09-22T22:11:21.306 DEBUG:boto:Headers: {} 2023-09-22T22:11:21.306 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:21.306 DEBUG:boto:Port: 8000 2023-09-22T22:11:21.306 DEBUG:boto:Params: {} 2023-09-22T22:11:21.306 DEBUG:boto:Token: None 2023-09-22T22:11:21.306 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:21 GMT /ffzdqu-64/ 2023-09-22T22:11:21.306 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jz5mny6Cgmzz+yJfi986d2+CwSM= 2023-09-22T22:11:21.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jz5mny6Cgmzz+yJfi986d2+CwSM=', 'Content-Length': '0'} 2023-09-22T22:11:21.314 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065b6dbda795e0d3c-00650e1109-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:21.314 DEBUG:boto:path=/ffzdqu-64/obj148 2023-09-22T22:11:21.314 DEBUG:boto:auth_path=/ffzdqu-64/obj148 2023-09-22T22:11:21.314 DEBUG:boto:Method: PUT 2023-09-22T22:11:21.314 DEBUG:boto:Path: /ffzdqu-64/obj148 2023-09-22T22:11:21.315 DEBUG:boto:Data: 2023-09-22T22:11:21.315 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:21.315 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:21.315 DEBUG:boto:Port: 8000 2023-09-22T22:11:21.315 DEBUG:boto:Params: {} 2023-09-22T22:11:21.315 DEBUG:boto:Token: None 2023-09-22T22:11:21.315 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:21 GMT /ffzdqu-64/obj148 2023-09-22T22:11:21.317 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:wvMQRq3NZizALfWVOhk1ILKVzlE= 2023-09-22T22:11:21.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wvMQRq3NZizALfWVOhk1ILKVzlE='} 2023-09-22T22:11:21.367 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2dd13c2b506f1f7-00650e1109-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:21.399 DEBUG:boto:path=/ffzdqu-64/ 2023-09-22T22:11:21.400 DEBUG:boto:auth_path=/ffzdqu-64/ 2023-09-22T22:11:21.400 DEBUG:boto:Method: HEAD 2023-09-22T22:11:21.400 DEBUG:boto:Path: /ffzdqu-64/ 2023-09-22T22:11:21.400 DEBUG:boto:Data: 2023-09-22T22:11:21.400 DEBUG:boto:Headers: {} 2023-09-22T22:11:21.400 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:21.401 DEBUG:boto:Port: 8000 2023-09-22T22:11:21.401 DEBUG:boto:Params: {} 2023-09-22T22:11:21.401 DEBUG:boto:Token: None 2023-09-22T22:11:21.401 DEBUG:boto:StringToSign: HEAD Fri, 22 Sep 2023 22:11:21 GMT /ffzdqu-64/ 2023-09-22T22:11:21.401 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:Jz5mny6Cgmzz+yJfi986d2+CwSM= 2023-09-22T22:11:21.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jz5mny6Cgmzz+yJfi986d2+CwSM=', 'Content-Length': '0'} 2023-09-22T22:11:21.409 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bacc5a93916e75ee-00650e1109-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:21.410 DEBUG:boto:path=/ffzdqu-64/obj185 2023-09-22T22:11:21.410 DEBUG:boto:auth_path=/ffzdqu-64/obj185 2023-09-22T22:11:21.410 DEBUG:boto:Method: PUT 2023-09-22T22:11:21.410 DEBUG:boto:Path: /ffzdqu-64/obj185 2023-09-22T22:11:21.410 DEBUG:boto:Data: 2023-09-22T22:11:21.411 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:11:21.411 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:11:21.411 DEBUG:boto:Port: 8000 2023-09-22T22:11:21.411 DEBUG:boto:Params: {} 2023-09-22T22:11:21.411 DEBUG:boto:Token: None 2023-09-22T22:11:21.411 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 22 Sep 2023 22:11:21 GMT /ffzdqu-64/obj185 2023-09-22T22:11:21.411 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:LvF4grXy/L76db1DndtUEd47W9k= 2023-09-22T22:11:21.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LvF4grXy/L76db1DndtUEd47W9k='} 2023-09-22T22:11:21.424 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002022c99d173a42a4-00650e1109-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:11:21.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:11:21.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:11:21.424 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:11:21.471 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:11:21.471 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:11:21.513 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:11:21.510+0000 7fcb56676a40 5 RGW-SYNC:data: waiting on data sync status lease 2023-09-22T22:11:21.514 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:11:21.514+0000 7fcb56676a40 5 RGW-SYNC:data: acquired data sync status lease 2023-09-22T22:11:21.544 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-22T22:11:21.544 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-22T22:11:21.546 INFO:tasks.rgw.c1.client.1:Started 2023-09-22T22:11:21.546 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:21.555 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:11:21.555 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T22:11:21.555 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T22:11:21.557 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T22:11:21.557 INFO:teuthology.orchestra.run.smithi160.stderr:curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T22:11:22.558 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:22.570 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:11:22.570 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T22:11:22.570 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T22:11:22.571 INFO:teuthology.orchestra.run.smithi160.stderr:curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T22:11:22.571 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T22:11:24.573 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:24.584 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:11:24.584 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T22:11:24.585 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T22:11:24.585 INFO:teuthology.orchestra.run.smithi160.stderr:curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T22:11:24.585 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T22:11:28.587 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:28.599 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:11:28.599 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T22:11:28.599 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T22:11:28.599 INFO:teuthology.orchestra.run.smithi160.stderr:curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T22:11:28.600 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T22:11:36.601 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:36.613 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:11:36.613 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T22:11:36.613 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T22:11:36.613 INFO:teuthology.orchestra.run.smithi160.stderr:curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T22:11:36.614 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T22:11:48.436 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T22:11:48.434+0000 7fce80ac7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T22:11:52.615 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:11:52.627 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:11:52.627 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2023-09-22T22:11:52.628 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 141k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-22T22:11:52.629 INFO:teuthology.orchestra.run.smithi160.stdout:anonymous 2023-09-22T22:11:52.630 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-64 2023-09-22T22:11:52.630 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:11:52.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:11:52.630 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T22:11:52.726 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:11:52.726 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:11:52.774 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:11:52.770+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:11:57.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:11:57.773+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:12:02.776 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:12:02.773+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:12:07.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:12:07.773+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:12:12.778 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:12:12.777+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:12:17.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:12:17.781+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:12:22.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:12:22.781+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:12:27.781 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:12:27.781+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:12:32.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:12:32.781+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:12:37.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:12:37.785+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:12:42.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:12:42.785+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:12:47.786 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:12:47.789+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:12:52.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:12:52.789+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:12:57.788 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:12:57.789+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:13:02.789 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:13:02.789+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:13:07.790 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:13:07.793+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:13:12.791 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:13:12.792+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:13:17.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:13:17.792+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:13:22.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:13:22.792+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:13:27.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:13:27.796+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:13:32.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:13:32.796+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:13:37.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:13:37.796+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:13:42.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:13:42.796+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:13:47.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:13:47.800+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:13:52.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:13:52.800+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:13:57.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:13:57.800+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:14:02.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:14:02.804+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:14:07.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:14:07.804+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:14:12.805 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:14:12.804+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:14:17.806 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:14:17.808+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:14:22.807 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:14:22.808+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:14:27.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:14:27.807+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:14:32.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:14:32.807+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:14:37.810 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:14:37.811+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:14:42.811 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:14:42.811+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:14:47.812 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:14:47.811+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:14:52.813 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:14:52.811+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:14:57.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:14:57.815+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:15:02.816 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:15:02.815+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:15:07.817 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:15:07.819+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:15:12.818 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:15:12.819+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:15:17.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:15:17.819+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:15:22.820 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:15:22.819+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:15:27.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:15:27.819+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:15:32.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:15:32.823+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:15:37.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:15:37.823+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:15:42.825 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:15:42.822+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:15:47.826 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:15:47.826+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:15:52.827 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:15:52.826+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:15:57.828 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:15:57.826+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:16:02.830 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:16:02.830+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:16:07.831 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:16:07.830+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:16:12.832 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:16:12.830+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:16:17.833 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:16:17.830+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:16:22.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:16:22.834+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:16:27.835 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:16:27.834+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:16:32.836 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:16:32.834+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:16:37.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:16:37.838+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:16:42.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:16:42.838+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:16:47.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:16:47.837+0000 7f1885df7a40 -1 bucket checkpoint timed out waiting to reach latest generation 6 2023-09-22T22:16:47.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:16:47.837+0000 7f1885df7a40 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-22T22:16:47.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:16:47.837+0000 7f1885df7a40 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-22T22:16:47.877 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-09-22T22:16:47.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... FAIL 2023-09-22T22:16:47.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:16:47.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:16:47.881 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:16:47.925 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:47.925 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:47.966 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T22:16:47.986 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"1_1695419384.000133_332.1","last_update":"2023-09-22T21:49:44.000133Z"},{"marker":"1_1695420681.013221_682.1","last_update":"2023-09-22T22:11:21.013221Z"},{"marker":"1_1695419383.971498_240.1","last_update":"2023-09-22T21:49:43.971498Z"}] 2023-09-22T22:16:47.986 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:16:47.986 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T22:16:47.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:16:47.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:16:47.987 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:16:48.070 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:48.070 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:48.108 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:16:48.105+0000 7f6f5868ba40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:16:48.109 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:16:48.105+0000 7f6f5868ba40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:16:48.127 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:16:48Z"} 2023-09-22T22:16:48.128 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:16:48Z"} 2023-09-22T22:16:48.128 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:16:48.128 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:16:48.128 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:16:48.128 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-22T22:16:48.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:48.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:48.129 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:16:48.202 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:48.203 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:48.261 INFO:teuthology.orchestra.run.smithi160.stdout:{"groups":[]} 2023-09-22T22:16:48.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:48.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:48.262 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:16:48.345 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:48.346 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:48.407 INFO:teuthology.orchestra.run.smithi160.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-22T22:16:48.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:48.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:48.408 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:16:48.448 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T22:16:48.445+0000 7fce80ac7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T22:16:48.484 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:48.484 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:48.541 INFO:teuthology.orchestra.run.smithi160.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-22T22:16:48.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:48.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:48.542 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:16:48.620 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:48.620 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:48.682 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"} 2023-09-22T22:16:48.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:48.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:48.683 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:16:48.758 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:48.759 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:48.814 INFO:teuthology.orchestra.run.smithi160.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-22T22:16:48.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:48.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:48.815 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:16:48.906 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:48.907 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:48.966 INFO:teuthology.orchestra.run.smithi160.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-09-22T22:16:48.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:48.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:48.967 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:16:49.006 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:49.006 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:49.058 INFO:teuthology.orchestra.run.smithi160.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-09-22T22:16:49.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:49.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:49.059 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:16:49.141 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:49.141 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:49.197 INFO:teuthology.orchestra.run.smithi160.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-22T22:16:49.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:49.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:49.198 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:16:49.280 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:49.280 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:49.334 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2023-09-22T22:16:49.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:49.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:49.335 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:16:49.410 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:49.410 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:49.472 INFO:tasks.rgw.c1.client.1.smithi160.stdout:./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7fce80ac7640 time 2023-09-22T22:16:49.476380+0000 2023-09-22T22:16:49.472 INFO:tasks.rgw.c1.client.1.smithi160.stdout:./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T22:16:49.474 INFO:tasks.rgw.c1.client.1.smithi160.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:16:49.474 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7fcea9675883] 2023-09-22T22:16:49.474 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7fcea9675a37] 2023-09-22T22:16:49.474 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x5557ee27e46c] 2023-09-22T22:16:49.474 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5557ee660910] 2023-09-22T22:16:49.474 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x5557ee28a56e] 2023-09-22T22:16:49.474 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x5557ee28be7b] 2023-09-22T22:16:49.475 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x5557ee28cced] 2023-09-22T22:16:49.475 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x5557ee693712] 2023-09-22T22:16:49.475 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 9: radosgw(+0x836db8) [0x5557ee43edb8] 2023-09-22T22:16:49.475 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x5557ee43f753] 2023-09-22T22:16:49.475 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fcea8344b43] 2023-09-22T22:16:49.475 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fcea83d6a00] 2023-09-22T22:16:49.475 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T22:16:49.473+0000 7fce80ac7640 -1 ./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7fce80ac7640 time 2023-09-22T22:16:49.476380+0000 2023-09-22T22:16:49.475 INFO:tasks.rgw.c1.client.1.smithi160.stdout:./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T22:16:49.475 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T22:16:49.476 INFO:tasks.rgw.c1.client.1.smithi160.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:16:49.476 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7fcea9675883] 2023-09-22T22:16:49.476 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7fcea9675a37] 2023-09-22T22:16:49.476 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x5557ee27e46c] 2023-09-22T22:16:49.476 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5557ee660910] 2023-09-22T22:16:49.476 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x5557ee28a56e] 2023-09-22T22:16:49.476 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x5557ee28be7b] 2023-09-22T22:16:49.476 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x5557ee28cced] 2023-09-22T22:16:49.476 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x5557ee693712] 2023-09-22T22:16:49.477 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 9: radosgw(+0x836db8) [0x5557ee43edb8] 2023-09-22T22:16:49.477 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x5557ee43f753] 2023-09-22T22:16:49.477 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fcea8344b43] 2023-09-22T22:16:49.477 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fcea83d6a00] 2023-09-22T22:16:49.477 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T22:16:49.477 INFO:tasks.rgw.c1.client.1.smithi160.stdout:*** Caught signal (Aborted) ** 2023-09-22T22:16:49.477 INFO:tasks.rgw.c1.client.1.smithi160.stdout: in thread 7fce80ac7640 thread_name:data-sync 2023-09-22T22:16:49.478 INFO:tasks.rgw.c1.client.1.smithi160.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:16:49.478 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fcea82f2520] 2023-09-22T22:16:49.478 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2: pthread_kill() 2023-09-22T22:16:49.478 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 3: raise() 2023-09-22T22:16:49.478 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 4: abort() 2023-09-22T22:16:49.478 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fcea96758dd] 2023-09-22T22:16:49.478 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7fcea9675a37] 2023-09-22T22:16:49.479 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x5557ee27e46c] 2023-09-22T22:16:49.479 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5557ee660910] 2023-09-22T22:16:49.479 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x5557ee28a56e] 2023-09-22T22:16:49.479 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x5557ee28be7b] 2023-09-22T22:16:49.479 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x5557ee28cced] 2023-09-22T22:16:49.479 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x5557ee693712] 2023-09-22T22:16:49.479 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 13: radosgw(+0x836db8) [0x5557ee43edb8] 2023-09-22T22:16:49.479 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x5557ee43f753] 2023-09-22T22:16:49.480 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fcea8344b43] 2023-09-22T22:16:49.480 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fcea83d6a00] 2023-09-22T22:16:49.480 INFO:tasks.rgw.c1.client.1.smithi160.stdout:2023-09-22T22:16:49.473+0000 7fce80ac7640 -1 *** Caught signal (Aborted) ** 2023-09-22T22:16:49.480 INFO:tasks.rgw.c1.client.1.smithi160.stdout: in thread 7fce80ac7640 thread_name:data-sync 2023-09-22T22:16:49.480 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T22:16:49.480 INFO:tasks.rgw.c1.client.1.smithi160.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:16:49.480 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fcea82f2520] 2023-09-22T22:16:49.480 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2: pthread_kill() 2023-09-22T22:16:49.480 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 3: raise() 2023-09-22T22:16:49.481 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 4: abort() 2023-09-22T22:16:49.481 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fcea96758dd] 2023-09-22T22:16:49.481 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7fcea9675a37] 2023-09-22T22:16:49.481 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x5557ee27e46c] 2023-09-22T22:16:49.481 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5557ee660910] 2023-09-22T22:16:49.481 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x5557ee28a56e] 2023-09-22T22:16:49.481 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x5557ee28be7b] 2023-09-22T22:16:49.481 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x5557ee28cced] 2023-09-22T22:16:49.482 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x5557ee693712] 2023-09-22T22:16:49.482 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 13: radosgw(+0x836db8) [0x5557ee43edb8] 2023-09-22T22:16:49.482 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x5557ee43f753] 2023-09-22T22:16:49.482 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fcea8344b43] 2023-09-22T22:16:49.482 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fcea83d6a00] 2023-09-22T22:16:49.482 INFO:tasks.rgw.c1.client.1.smithi160.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-09-22T22:16:49.482 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T22:16:49.489 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","epoch":13,"predecessor_uuid":"a6099785-5160-4a69-b202-552a1b9cb770","sync_status":[],"period_map":{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","zonegroups":[{"id":"a","name":"a","api_name":"","is_master":true,"endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","zones":[{"id":"2b721fee-a149-45da-925b-502d5a3d6853","name":"a2","endpoints":["http://smithi160.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"f132bdb9-20a7-4073-a932-8f99da20691e","name":"a1","endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["2b721fee-a149-45da-925b-502d5a3d6853","f132bdb9-20a7-4073-a932-8f99da20691e"]}],"directional":[{"source_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","dest_zone":"2b721fee-a149-45da-925b-502d5a3d6853"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["2b721fee-a149-45da-925b-502d5a3d6853","f132bdb9-20a7-4073-a932-8f99da20691e"]},"dest":{"bucket":"*","zones":["2b721fee-a149-45da-925b-502d5a3d6853","f132bdb9-20a7-4073-a932-8f99da20691e"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]},"enabled_features":["resharding"]},{"id":"b","name":"b","api_name":"","is_master":false,"endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","zones":[{"id":"087d4b41-8f81-4835-9e99-026ca13a7b60","name":"b2","endpoints":["http://smithi204.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","name":"b1","endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"087d4b41-8f81-4835-9e99-026ca13a7b60","val":2135214481},{"key":"2b721fee-a149-45da-925b-502d5a3d6853","val":3586193783},{"key":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","val":4184768938},{"key":"f132bdb9-20a7-4073-a932-8f99da20691e","val":3405200286}]},"master_zonegroup":"a","master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","realm_name":"test-realm","realm_epoch":2} 2023-09-22T22:16:49.489 INFO:rgw_multi.tests:create bucket zone=a1 name=ffzdqu-65 2023-09-22T22:16:49.490 DEBUG:boto:path=/ffzdqu-65/ 2023-09-22T22:16:49.490 DEBUG:boto:auth_path=/ffzdqu-65/ 2023-09-22T22:16:49.490 DEBUG:boto:Method: PUT 2023-09-22T22:16:49.490 DEBUG:boto:Path: /ffzdqu-65/ 2023-09-22T22:16:49.490 DEBUG:boto:Data: 2023-09-22T22:16:49.490 DEBUG:boto:Headers: {} 2023-09-22T22:16:49.490 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:16:49.490 DEBUG:boto:Port: 8000 2023-09-22T22:16:49.491 DEBUG:boto:Params: {} 2023-09-22T22:16:49.491 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T22:16:49.491 DEBUG:boto:Token: None 2023-09-22T22:16:49.491 DEBUG:boto:StringToSign: PUT Fri, 22 Sep 2023 22:16:49 GMT /ffzdqu-65/ 2023-09-22T22:16:49.491 DEBUG:boto:Signature: AWS RX9T9XC1DLDGURC46TDQ:4/5Tr3qDW+XNfGY4adXhQ1sJOUQ= 2023-09-22T22:16:49.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:16:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4/5Tr3qDW+XNfGY4adXhQ1sJOUQ=', 'Content-Length': '0'} 2023-09-22T22:16:49.493 INFO:tasks.rgw.c1.client.1.smithi160.stdout: -883> 2023-09-22T22:16:48.445+0000 7fce80ac7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T22:16:49.493 INFO:tasks.rgw.c1.client.1.smithi160.stdout: -1> 2023-09-22T22:16:49.473+0000 7fce80ac7640 -1 ./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7fce80ac7640 time 2023-09-22T22:16:49.476380+0000 2023-09-22T22:16:49.493 INFO:tasks.rgw.c1.client.1.smithi160.stdout:./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T22:16:49.493 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T22:16:49.493 INFO:tasks.rgw.c1.client.1.smithi160.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:16:49.493 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7fcea9675883] 2023-09-22T22:16:49.493 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7fcea9675a37] 2023-09-22T22:16:49.494 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x5557ee27e46c] 2023-09-22T22:16:49.494 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5557ee660910] 2023-09-22T22:16:49.494 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x5557ee28a56e] 2023-09-22T22:16:49.494 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x5557ee28be7b] 2023-09-22T22:16:49.494 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x5557ee28cced] 2023-09-22T22:16:49.494 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x5557ee693712] 2023-09-22T22:16:49.494 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 9: radosgw(+0x836db8) [0x5557ee43edb8] 2023-09-22T22:16:49.494 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x5557ee43f753] 2023-09-22T22:16:49.494 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fcea8344b43] 2023-09-22T22:16:49.495 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fcea83d6a00] 2023-09-22T22:16:49.495 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T22:16:49.495 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 0> 2023-09-22T22:16:49.473+0000 7fce80ac7640 -1 *** Caught signal (Aborted) ** 2023-09-22T22:16:49.495 INFO:tasks.rgw.c1.client.1.smithi160.stdout: in thread 7fce80ac7640 thread_name:data-sync 2023-09-22T22:16:49.495 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T22:16:49.495 INFO:tasks.rgw.c1.client.1.smithi160.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:16:49.495 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fcea82f2520] 2023-09-22T22:16:49.495 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2: pthread_kill() 2023-09-22T22:16:49.495 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 3: raise() 2023-09-22T22:16:49.495 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 4: abort() 2023-09-22T22:16:49.496 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fcea96758dd] 2023-09-22T22:16:49.496 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7fcea9675a37] 2023-09-22T22:16:49.496 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x5557ee27e46c] 2023-09-22T22:16:49.496 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5557ee660910] 2023-09-22T22:16:49.496 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x5557ee28a56e] 2023-09-22T22:16:49.496 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x5557ee28be7b] 2023-09-22T22:16:49.496 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x5557ee28cced] 2023-09-22T22:16:49.496 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x5557ee693712] 2023-09-22T22:16:49.496 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 13: radosgw(+0x836db8) [0x5557ee43edb8] 2023-09-22T22:16:49.496 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x5557ee43f753] 2023-09-22T22:16:49.497 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fcea8344b43] 2023-09-22T22:16:49.497 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fcea83d6a00] 2023-09-22T22:16:49.497 INFO:tasks.rgw.c1.client.1.smithi160.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-09-22T22:16:49.497 INFO:tasks.rgw.c1.client.1.smithi160.stdout: 2023-09-22T22:16:49.670 INFO:tasks.rgw.c1.client.1.smithi160.stderr:daemon-helper: command crashed with signal 11 2023-09-22T22:16:49.686 INFO:tasks.rgw.c2.client.0.smithi204.stdout:./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7f11298ba640 time 2023-09-22T22:16:49.589063+0000 2023-09-22T22:16:49.687 INFO:tasks.rgw.c2.client.0.smithi204.stdout:./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T22:16:49.691 INFO:tasks.rgw.c2.client.0.smithi204.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:16:49.691 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f1152468883] 2023-09-22T22:16:49.691 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f1152468a37] 2023-09-22T22:16:49.691 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x56533621a46c] 2023-09-22T22:16:49.691 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5653365fc910] 2023-09-22T22:16:49.691 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x56533622656e] 2023-09-22T22:16:49.691 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x565336227e7b] 2023-09-22T22:16:49.691 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x565336228ced] 2023-09-22T22:16:49.691 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x56533662f712] 2023-09-22T22:16:49.692 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 9: radosgw(+0x836db8) [0x5653363dadb8] 2023-09-22T22:16:49.692 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x5653363db753] 2023-09-22T22:16:49.692 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f1151137b43] 2023-09-22T22:16:49.692 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f11511c9a00] 2023-09-22T22:16:49.692 INFO:tasks.rgw.c2.client.0.smithi204.stdout:2023-09-22T22:16:49.583+0000 7f11298ba640 -1 ./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7f11298ba640 time 2023-09-22T22:16:49.589063+0000 2023-09-22T22:16:49.692 INFO:tasks.rgw.c2.client.0.smithi204.stdout:./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T22:16:49.692 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 2023-09-22T22:16:49.692 INFO:tasks.rgw.c2.client.0.smithi204.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:16:49.692 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f1152468883] 2023-09-22T22:16:49.693 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f1152468a37] 2023-09-22T22:16:49.693 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x56533621a46c] 2023-09-22T22:16:49.693 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5653365fc910] 2023-09-22T22:16:49.693 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x56533622656e] 2023-09-22T22:16:49.693 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x565336227e7b] 2023-09-22T22:16:49.693 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x565336228ced] 2023-09-22T22:16:49.693 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x56533662f712] 2023-09-22T22:16:49.693 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 9: radosgw(+0x836db8) [0x5653363dadb8] 2023-09-22T22:16:49.693 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x5653363db753] 2023-09-22T22:16:49.693 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f1151137b43] 2023-09-22T22:16:49.694 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f11511c9a00] 2023-09-22T22:16:49.694 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 2023-09-22T22:16:49.694 INFO:tasks.rgw.c2.client.0.smithi204.stdout:*** Caught signal (Aborted) ** 2023-09-22T22:16:49.694 INFO:tasks.rgw.c2.client.0.smithi204.stdout: in thread 7f11298ba640 thread_name:data-sync 2023-09-22T22:16:49.694 INFO:tasks.rgw.c2.client.0.smithi204.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:16:49.694 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f11510e5520] 2023-09-22T22:16:49.694 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 2: pthread_kill() 2023-09-22T22:16:49.694 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 3: raise() 2023-09-22T22:16:49.694 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 4: abort() 2023-09-22T22:16:49.695 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f11524688dd] 2023-09-22T22:16:49.695 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f1152468a37] 2023-09-22T22:16:49.695 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x56533621a46c] 2023-09-22T22:16:49.695 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5653365fc910] 2023-09-22T22:16:49.695 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x56533622656e] 2023-09-22T22:16:49.695 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x565336227e7b] 2023-09-22T22:16:49.695 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x565336228ced] 2023-09-22T22:16:49.695 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x56533662f712] 2023-09-22T22:16:49.695 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 13: radosgw(+0x836db8) [0x5653363dadb8] 2023-09-22T22:16:49.695 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x5653363db753] 2023-09-22T22:16:49.695 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f1151137b43] 2023-09-22T22:16:49.696 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f11511c9a00] 2023-09-22T22:16:49.696 INFO:tasks.rgw.c2.client.0.smithi204.stdout:2023-09-22T22:16:49.587+0000 7f11298ba640 -1 *** Caught signal (Aborted) ** 2023-09-22T22:16:49.696 INFO:tasks.rgw.c2.client.0.smithi204.stdout: in thread 7f11298ba640 thread_name:data-sync 2023-09-22T22:16:49.696 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 2023-09-22T22:16:49.696 INFO:tasks.rgw.c2.client.0.smithi204.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:16:49.696 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f11510e5520] 2023-09-22T22:16:49.696 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 2: pthread_kill() 2023-09-22T22:16:49.696 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 3: raise() 2023-09-22T22:16:49.696 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 4: abort() 2023-09-22T22:16:49.696 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f11524688dd] 2023-09-22T22:16:49.697 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f1152468a37] 2023-09-22T22:16:49.697 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x56533621a46c] 2023-09-22T22:16:49.697 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5653365fc910] 2023-09-22T22:16:49.697 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x56533622656e] 2023-09-22T22:16:49.697 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x565336227e7b] 2023-09-22T22:16:49.697 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x565336228ced] 2023-09-22T22:16:49.697 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x56533662f712] 2023-09-22T22:16:49.697 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 13: radosgw(+0x836db8) [0x5653363dadb8] 2023-09-22T22:16:49.697 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x5653363db753] 2023-09-22T22:16:49.697 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f1151137b43] 2023-09-22T22:16:49.698 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f11511c9a00] 2023-09-22T22:16:49.698 INFO:tasks.rgw.c2.client.0.smithi204.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-09-22T22:16:49.698 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 2023-09-22T22:16:49.698 INFO:tasks.rgw.c2.client.0.smithi204.stdout: -1> 2023-09-22T22:16:49.583+0000 7f11298ba640 -1 ./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7f11298ba640 time 2023-09-22T22:16:49.589063+0000 2023-09-22T22:16:49.698 INFO:tasks.rgw.c2.client.0.smithi204.stdout:./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T22:16:49.698 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 2023-09-22T22:16:49.698 INFO:tasks.rgw.c2.client.0.smithi204.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:16:49.698 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f1152468883] 2023-09-22T22:16:49.698 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f1152468a37] 2023-09-22T22:16:49.698 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x56533621a46c] 2023-09-22T22:16:49.699 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5653365fc910] 2023-09-22T22:16:49.699 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x56533622656e] 2023-09-22T22:16:49.699 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x565336227e7b] 2023-09-22T22:16:49.699 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x565336228ced] 2023-09-22T22:16:49.699 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x56533662f712] 2023-09-22T22:16:49.699 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 9: radosgw(+0x836db8) [0x5653363dadb8] 2023-09-22T22:16:49.699 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x5653363db753] 2023-09-22T22:16:49.699 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f1151137b43] 2023-09-22T22:16:49.699 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f11511c9a00] 2023-09-22T22:16:49.699 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 2023-09-22T22:16:49.700 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 0> 2023-09-22T22:16:49.587+0000 7f11298ba640 -1 *** Caught signal (Aborted) ** 2023-09-22T22:16:49.700 INFO:tasks.rgw.c2.client.0.smithi204.stdout: in thread 7f11298ba640 thread_name:data-sync 2023-09-22T22:16:49.700 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 2023-09-22T22:16:49.700 INFO:tasks.rgw.c2.client.0.smithi204.stdout: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:16:49.700 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f11510e5520] 2023-09-22T22:16:49.700 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 2: pthread_kill() 2023-09-22T22:16:49.700 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 3: raise() 2023-09-22T22:16:49.700 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 4: abort() 2023-09-22T22:16:49.700 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f11524688dd] 2023-09-22T22:16:49.700 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f1152468a37] 2023-09-22T22:16:49.701 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x56533621a46c] 2023-09-22T22:16:49.701 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5653365fc910] 2023-09-22T22:16:49.701 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x56533622656e] 2023-09-22T22:16:49.701 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x565336227e7b] 2023-09-22T22:16:49.701 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x565336228ced] 2023-09-22T22:16:49.701 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x56533662f712] 2023-09-22T22:16:49.701 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 13: radosgw(+0x836db8) [0x5653363dadb8] 2023-09-22T22:16:49.701 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x5653363db753] 2023-09-22T22:16:49.701 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f1151137b43] 2023-09-22T22:16:49.701 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f11511c9a00] 2023-09-22T22:16:49.701 INFO:tasks.rgw.c2.client.0.smithi204.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-09-22T22:16:49.702 INFO:tasks.rgw.c2.client.0.smithi204.stdout: 2023-09-22T22:16:49.702 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000ead651f78deaa4f-00650e1251-208a-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:16:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:16:49.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:49.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:49.703 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket ffzdqu-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:16:49.767 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:49.767 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:49.853 INFO:tasks.rgw.c2.client.0.smithi204.stderr:daemon-helper: command crashed with signal 11 2023-09-22T22:16:49.863 INFO:teuthology.orchestra.run.smithi160.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-22T22:16:49.863 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:49.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:49.864 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket ffzdqu-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:16:49.913 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:49.914 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:49.988 INFO:teuthology.orchestra.run.smithi160.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-22T22:16:49.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:49.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:49.989 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket ffzdqu-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:16:50.025 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:50.025 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:50.085 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"} 2023-09-22T22:16:50.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:50.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:50.086 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket ffzdqu-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:16:50.159 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:50.159 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:50.212 INFO:teuthology.orchestra.run.smithi160.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-22T22:16:50.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:50.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:50.213 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket ffzdqu-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:16:50.295 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:50.295 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:50.350 INFO:teuthology.orchestra.run.smithi160.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-09-22T22:16:50.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:50.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:50.351 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket ffzdqu-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:16:50.430 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:50.430 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:50.484 INFO:teuthology.orchestra.run.smithi160.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-09-22T22:16:50.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:50.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:50.485 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=ffzdqu-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:16:50.569 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:50.570 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:50.623 INFO:teuthology.orchestra.run.smithi160.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-09-22T22:16:50.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:50.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:16:50.624 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket ffzdqu-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:16:50.701 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:50.701 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:50.751 INFO:teuthology.orchestra.run.smithi160.stdout:{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"} 2023-09-22T22:16:50.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:16:50.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:16:50.751 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:16:50.827 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:50.828 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:50.863 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T22:16:50.879 INFO:teuthology.orchestra.run.smithi160.stdout:[{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"1_1695419384.000133_332.1","last_update":"2023-09-22T21:49:44.000133Z"},{"marker":"1_1695421010.611730_706.1","last_update":"2023-09-22T22:16:50.611730Z"},{"marker":"1_1695419383.971498_240.1","last_update":"2023-09-22T21:49:43.971498Z"}] 2023-09-22T22:16:50.880 DEBUG:rgw_multi.tests:master meta markers={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:16:50.880 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-22T22:16:50.880 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:16:50.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:16:50.880 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:16:50.960 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:50.960 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:50.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:16:50.993+0000 7f8106e97a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:16:50.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:16:50.993+0000 7f8106e97a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:16:51.011 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:16:51Z"} 2023-09-22T22:16:51.011 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:16:51Z"} 2023-09-22T22:16:51.011 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:16:51.012 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:16:51.012 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:16:51.012 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:16:52.969 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-09-22T22:16:53.361 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-09-22T22:16:56.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:16:56.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:16:56.013 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:16:56.058 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:16:56.058 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:16:56.098 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:16:56.097+0000 7fa573143a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:16:56.099 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:16:56.097+0000 7fa573143a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:16:56.119 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:16:56Z"} 2023-09-22T22:16:56.119 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:16:56Z"} 2023-09-22T22:16:56.120 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:16:56.120 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:16:56.120 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:16:56.121 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:16:58.575 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~6s 2023-09-22T22:16:58.967 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~6s 2023-09-22T22:17:01.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:01.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:01.122 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:17:01.166 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:17:01.166 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:17:01.205 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:01.201+0000 7f595c171a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:01.206 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:01.205+0000 7f595c171a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:01.230 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:01Z"} 2023-09-22T22:17:01.231 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:01Z"} 2023-09-22T22:17:01.231 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:17:01.231 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:01.231 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:01.231 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:17:04.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~11s 2023-09-22T22:17:04.572 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~11s 2023-09-22T22:17:06.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:06.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:06.233 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:17:06.275 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:17:06.276 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:17:06.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:06.317+0000 7fe6c0939a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:06.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:06.317+0000 7fe6c0939a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:06.333 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:06Z"} 2023-09-22T22:17:06.333 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:06Z"} 2023-09-22T22:17:06.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:17:06.334 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:06.334 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:06.334 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:17:09.785 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~17s 2023-09-22T22:17:10.176 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~17s 2023-09-22T22:17:11.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:11.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:11.335 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:17:11.379 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:17:11.379 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:17:11.419 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:11.421+0000 7f5a04cb1a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:11.420 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:11.421+0000 7f5a04cb1a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:11.439 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:11Z"} 2023-09-22T22:17:11.439 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:11Z"} 2023-09-22T22:17:11.439 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:17:11.439 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:11.440 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:11.440 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:17:15.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~22s 2023-09-22T22:17:15.782 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~22s 2023-09-22T22:17:16.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:16.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:16.442 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:17:16.485 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:17:16.641 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:17:16.642 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:16.525+0000 7fddbbf26a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:16.642 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:16.525+0000 7fddbbf26a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:16.643 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:16Z"} 2023-09-22T22:17:16.643 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:16Z"} 2023-09-22T22:17:16.644 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:17:16.644 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:16.644 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:16.644 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:17:20.996 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~28s 2023-09-22T22:17:21.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~28s 2023-09-22T22:17:21.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:21.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:21.645 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:17:21.688 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:17:21.689 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:17:21.727 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:21.729+0000 7f19cc582a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:21.728 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:21.729+0000 7f19cc582a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:21.747 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:21Z"} 2023-09-22T22:17:21.747 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:21Z"} 2023-09-22T22:17:21.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:17:21.747 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:21.747 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:21.748 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:17:26.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~34s 2023-09-22T22:17:26.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:26.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:26.749 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:17:26.793 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:17:26.793 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:17:26.832 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:26.833+0000 7f486bee1a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:26.833 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:26.833+0000 7f486bee1a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:26.851 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:26Z"} 2023-09-22T22:17:26.852 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:26Z"} 2023-09-22T22:17:26.852 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:17:26.852 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:26.852 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:26.852 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:17:26.992 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~34s 2023-09-22T22:17:31.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:31.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:31.854 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:17:31.896 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:17:31.897 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:17:31.934 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:31.937+0000 7fe8224a9a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:31.935 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:31.937+0000 7fe8224a9a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:31.956 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:31Z"} 2023-09-22T22:17:31.957 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:31Z"} 2023-09-22T22:17:31.957 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:17:31.957 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:31.957 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:31.957 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:17:32.207 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~39s 2023-09-22T22:17:32.598 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~39s 2023-09-22T22:17:36.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:36.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:36.959 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:17:37.003 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:17:37.004 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:17:37.041 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:37.041+0000 7f334fa27a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:37.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:37.045+0000 7f334fa27a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:37.060 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:37Z"} 2023-09-22T22:17:37.060 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:37Z"} 2023-09-22T22:17:37.061 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:17:37.061 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:37.061 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:37.061 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:17:37.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~45s 2023-09-22T22:17:38.204 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~45s 2023-09-22T22:17:42.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:42.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:42.063 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:17:42.106 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:17:42.106 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:17:42.146 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:42.149+0000 7f8297921a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:42.147 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:42.149+0000 7f8297921a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:42.166 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:42Z"} 2023-09-22T22:17:42.166 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:42Z"} 2023-09-22T22:17:42.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:17:42.166 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:42.166 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:42.167 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:17:43.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~50s 2023-09-22T22:17:43.809 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~50s 2023-09-22T22:17:47.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:47.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:47.168 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:17:47.212 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:17:47.213 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:17:47.251 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:47.253+0000 7f9a65de2a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:47.252 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:47.253+0000 7f9a65de2a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:47.270 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:47Z"} 2023-09-22T22:17:47.271 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:47Z"} 2023-09-22T22:17:47.271 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:17:47.271 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:47.271 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:47.271 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:17:49.034 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~56s 2023-09-22T22:17:49.415 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~56s 2023-09-22T22:17:52.272 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:52.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:52.273 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:17:52.316 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:17:52.317 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:17:52.354 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:52.357+0000 7f5168a56a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:52.355 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:52.357+0000 7f5168a56a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:52.374 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:52Z"} 2023-09-22T22:17:52.374 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:52Z"} 2023-09-22T22:17:52.375 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:17:52.375 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:52.375 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:52.375 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:17:54.640 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~62s 2023-09-22T22:17:55.021 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~62s 2023-09-22T22:17:57.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:57.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:17:57.377 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:17:57.421 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:17:57.421 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:17:57.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:57.460+0000 7fcbdb384a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:57.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:17:57.460+0000 7fcbdb384a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:17:57.479 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:57Z"} 2023-09-22T22:17:57.479 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:57Z"} 2023-09-22T22:17:57.480 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:17:57.480 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:57.480 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:17:57.480 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:18:00.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~67s 2023-09-22T22:18:00.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~67s 2023-09-22T22:18:02.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:02.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:02.481 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:18:02.525 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:18:02.525 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:18:02.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:02.564+0000 7f73eeb93a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:02.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:02.564+0000 7f73eeb93a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:02.583 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:02Z"} 2023-09-22T22:18:02.583 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:02Z"} 2023-09-22T22:18:02.584 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:18:02.584 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:02.584 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:02.584 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:18:05.852 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~73s 2023-09-22T22:18:06.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~73s 2023-09-22T22:18:07.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:07.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:07.586 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:18:07.629 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:18:07.629 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:18:07.670 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:07.672+0000 7fa986733a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:07.671 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:07.672+0000 7fa986733a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:07.689 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:07Z"} 2023-09-22T22:18:07.689 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:07Z"} 2023-09-22T22:18:07.690 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:18:07.690 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:07.690 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:07.690 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:18:11.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~78s 2023-09-22T22:18:11.836 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~78s 2023-09-22T22:18:12.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:12.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:12.692 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:18:12.736 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:18:12.736 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:18:12.776 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:12.776+0000 7fa974c0aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:12.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:12.776+0000 7fa974c0aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:12.795 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:12Z"} 2023-09-22T22:18:12.796 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:12Z"} 2023-09-22T22:18:12.796 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:18:12.796 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:12.796 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:12.796 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:18:17.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~84s 2023-09-22T22:18:17.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~84s 2023-09-22T22:18:17.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:17.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:17.798 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:18:17.843 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:18:17.843 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:18:17.880 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:17.880+0000 7fb49d372a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:17.881 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:17.884+0000 7fb49d372a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:17.900 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:17Z"} 2023-09-22T22:18:17.900 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:17Z"} 2023-09-22T22:18:17.901 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:18:17.901 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:17.901 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:17.901 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:18:22.736 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~90s 2023-09-22T22:18:22.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:22.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:22.903 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:18:22.948 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:18:22.948 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:18:22.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:22.992+0000 7ffbe5ee0a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:22.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:22.992+0000 7ffbe5ee0a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:23.009 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:22Z"} 2023-09-22T22:18:23.009 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:22Z"} 2023-09-22T22:18:23.010 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:18:23.010 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:23.010 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:23.010 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:18:23.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~90s 2023-09-22T22:18:28.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:28.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:28.012 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:18:28.057 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:18:28.057 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:18:28.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:28.096+0000 7f4fe351ba40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:28.098 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:28.100+0000 7f4fe351ba40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:28.117 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:28Z"} 2023-09-22T22:18:28.117 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:28Z"} 2023-09-22T22:18:28.117 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:18:28.118 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:28.118 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:28.118 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:18:28.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~95s 2023-09-22T22:18:28.652 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~95s 2023-09-22T22:18:33.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:33.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:33.120 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:18:33.163 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:18:33.163 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:18:33.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:33.200+0000 7fe58fcdaa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:33.202 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:33.204+0000 7fe58fcdaa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:33.221 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:33Z"} 2023-09-22T22:18:33.221 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:33Z"} 2023-09-22T22:18:33.221 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:18:33.221 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:33.221 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:33.222 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:18:33.947 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~101s 2023-09-22T22:18:34.256 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~101s 2023-09-22T22:18:38.225 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:38.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:38.225 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:18:38.269 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:18:38.269 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:18:38.307 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:38.308+0000 7fa415f77a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:38.308 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:38.308+0000 7fa415f77a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:38.327 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:38Z"} 2023-09-22T22:18:38.327 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:38Z"} 2023-09-22T22:18:38.327 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:18:38.327 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:38.327 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:38.328 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:18:39.552 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~107s 2023-09-22T22:18:39.861 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~106s 2023-09-22T22:18:43.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:43.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:43.329 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:18:43.373 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:18:43.373 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:18:43.411 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:43.412+0000 7fca5eb2da40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:43.412 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:43.412+0000 7fca5eb2da40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:43.432 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:43Z"} 2023-09-22T22:18:43.432 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:43Z"} 2023-09-22T22:18:43.432 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:18:43.432 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:43.433 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:43.433 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:18:45.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~112s 2023-09-22T22:18:45.478 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~112s 2023-09-22T22:18:48.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:48.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:48.434 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:18:48.480 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:18:48.481 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:18:48.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:48.520+0000 7f6a9a875a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:48.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:48.520+0000 7f6a9a875a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:48.539 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:48Z"} 2023-09-22T22:18:48.540 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:48Z"} 2023-09-22T22:18:48.540 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:18:48.540 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:48.540 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:48.540 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:18:50.781 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~118s 2023-09-22T22:18:51.083 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~118s 2023-09-22T22:18:53.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:53.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:53.783 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:18:53.826 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:18:53.826 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:18:53.865 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:53.864+0000 7f7ac47eda40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:53.866 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:53.868+0000 7f7ac47eda40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:53.885 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:53Z"} 2023-09-22T22:18:53.885 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:53Z"} 2023-09-22T22:18:53.885 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:18:53.885 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:53.886 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:53.886 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:18:56.386 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~123s 2023-09-22T22:18:56.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~123s 2023-09-22T22:18:58.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:58.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:18:58.887 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:18:58.936 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:18:58.937 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:18:58.976 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:58.976+0000 7f5e4f631a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:58.977 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:18:58.976+0000 7f5e4f631a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:18:58.996 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:58Z"} 2023-09-22T22:18:58.997 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:58Z"} 2023-09-22T22:18:58.997 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:18:58.997 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:58.997 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:18:58.997 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:19:01.991 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~129s 2023-09-22T22:19:02.293 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~129s 2023-09-22T22:19:03.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:03.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:03.999 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:19:04.043 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:19:04.043 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:19:04.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:04.079+0000 7fc1c3fcda40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:04.082 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:04.083+0000 7fc1c3fcda40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:04.100 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:04Z"} 2023-09-22T22:19:04.100 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:04Z"} 2023-09-22T22:19:04.101 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:19:04.101 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:04.101 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:04.101 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:19:07.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~135s 2023-09-22T22:19:07.898 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~135s 2023-09-22T22:19:09.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:09.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:09.103 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:19:09.146 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:19:09.147 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:19:09.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:09.183+0000 7f90b893fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:09.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:09.187+0000 7f90b893fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:09.205 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:09Z"} 2023-09-22T22:19:09.206 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:09Z"} 2023-09-22T22:19:09.206 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:19:09.206 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:09.206 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:09.207 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:19:13.202 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~140s 2023-09-22T22:19:13.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~140s 2023-09-22T22:19:14.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:14.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:14.208 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:19:14.252 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:19:14.252 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:19:14.294 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:14.295+0000 7f6f84ef6a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:14.927 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:14.295+0000 7f6f84ef6a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:14.928 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:14Z"} 2023-09-22T22:19:14.928 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:14Z"} 2023-09-22T22:19:14.928 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:19:14.928 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:14.929 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:14.929 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:19:18.807 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~146s 2023-09-22T22:19:19.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~146s 2023-09-22T22:19:19.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:19.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:19.930 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:19:19.973 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:19:19.973 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:19:20.012 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:20.011+0000 7f17e90b4a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:20.013 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:20.011+0000 7f17e90b4a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:20.032 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:20Z"} 2023-09-22T22:19:20.032 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:20Z"} 2023-09-22T22:19:20.032 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:19:20.032 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:20.032 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:20.032 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:19:24.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~151s 2023-09-22T22:19:24.713 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~151s 2023-09-22T22:19:25.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:25.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:25.033 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:19:25.080 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:19:25.081 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:19:25.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:25.119+0000 7f890d712a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:25.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:25.119+0000 7f890d712a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:25.138 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:25Z"} 2023-09-22T22:19:25.139 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:25Z"} 2023-09-22T22:19:25.139 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:19:25.139 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:25.139 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:25.139 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:19:30.018 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~157s 2023-09-22T22:19:30.140 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:30.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:30.141 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:19:30.184 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:19:30.184 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:19:30.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:30.223+0000 7f739da34a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:30.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:30.227+0000 7f739da34a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:30.244 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:30Z"} 2023-09-22T22:19:30.245 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:30Z"} 2023-09-22T22:19:30.245 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:19:30.245 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:30.245 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:30.245 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:19:30.318 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~157s 2023-09-22T22:19:35.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:35.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:35.247 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:19:35.290 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:19:35.291 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:19:35.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:35.327+0000 7f17cb84fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:35.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:35.331+0000 7f17cb84fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:35.348 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:35Z"} 2023-09-22T22:19:35.349 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:35Z"} 2023-09-22T22:19:35.349 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:19:35.349 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:35.349 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:35.349 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:19:35.623 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~163s 2023-09-22T22:19:36.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~163s 2023-09-22T22:19:40.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:40.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:40.351 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:19:40.395 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:19:40.395 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:19:40.434 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:40.435+0000 7f000cc06a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:40.435 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:40.435+0000 7f000cc06a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:40.454 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:40Z"} 2023-09-22T22:19:40.454 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:40Z"} 2023-09-22T22:19:40.454 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:19:40.454 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:40.455 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:40.455 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:19:41.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~169s 2023-09-22T22:19:41.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~169s 2023-09-22T22:19:45.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:45.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:45.456 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:19:45.499 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:19:45.499 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:19:45.539 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:45.539+0000 7ff7d62eca40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:45.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:45.539+0000 7ff7d62eca40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:45.559 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:45Z"} 2023-09-22T22:19:45.560 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:45Z"} 2023-09-22T22:19:45.560 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:19:45.560 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:45.560 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:45.560 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:19:47.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~174s 2023-09-22T22:19:47.471 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~174s 2023-09-22T22:19:50.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:50.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:50.562 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:19:50.606 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:19:50.607 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:19:50.646 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:50.647+0000 7fea2e3aea40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:50.647 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:50.647+0000 7fea2e3aea40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:50.671 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:50Z"} 2023-09-22T22:19:50.672 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:50Z"} 2023-09-22T22:19:50.672 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:19:50.672 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:50.672 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:50.672 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:19:52.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~180s 2023-09-22T22:19:53.078 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~180s 2023-09-22T22:19:55.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:55.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:19:55.674 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:19:55.718 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:19:55.718 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:19:55.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:55.755+0000 7f4116fd3a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:55.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:19:55.755+0000 7f4116fd3a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:19:55.775 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:55Z"} 2023-09-22T22:19:55.775 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:55Z"} 2023-09-22T22:19:55.776 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:19:55.776 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:55.776 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:19:55.776 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:19:58.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~186s 2023-09-22T22:19:58.684 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~185s 2023-09-22T22:20:00.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:00.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:00.777 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:20:00.821 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:20:00.822 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:20:00.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:00.859+0000 7fe9e10c1a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:00.861 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:00.859+0000 7fe9e10c1a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:00.880 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:00Z"} 2023-09-22T22:20:00.880 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:00Z"} 2023-09-22T22:20:00.880 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:20:00.881 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:00.881 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:00.881 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:20:04.089 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~191s 2023-09-22T22:20:04.305 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~191s 2023-09-22T22:20:05.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:05.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:05.882 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:20:05.928 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:20:05.928 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:20:05.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:05.966+0000 7f575aeaba40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:05.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:05.966+0000 7f575aeaba40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:05.985 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:05Z"} 2023-09-22T22:20:05.985 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:05Z"} 2023-09-22T22:20:05.986 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:20:05.986 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:05.986 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:05.986 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:20:09.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~197s 2023-09-22T22:20:10.342 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~197s 2023-09-22T22:20:10.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:10.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:10.988 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:20:11.030 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:20:11.030 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:20:11.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:11.070+0000 7fa5733cda40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:11.072 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:11.070+0000 7fa5733cda40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:11.091 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:11Z"} 2023-09-22T22:20:11.091 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:11Z"} 2023-09-22T22:20:11.092 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:20:11.092 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:11.092 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:11.092 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:20:15.415 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~202s 2023-09-22T22:20:15.947 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~203s 2023-09-22T22:20:16.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:16.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:16.094 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:20:16.138 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:20:16.138 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:20:16.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:16.174+0000 7f759fec0a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:16.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:16.174+0000 7f759fec0a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:16.199 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:16Z"} 2023-09-22T22:20:16.199 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:16Z"} 2023-09-22T22:20:16.199 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:20:16.200 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:16.200 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:16.200 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:20:21.021 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~208s 2023-09-22T22:20:21.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:21.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:21.201 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:20:21.245 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:20:21.245 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:20:21.284 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:21.282+0000 7faa2099ca40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:21.285 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:21.282+0000 7faa2099ca40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:21.303 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:21Z"} 2023-09-22T22:20:21.304 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:21Z"} 2023-09-22T22:20:21.304 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:20:21.304 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:21.304 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:21.304 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:20:21.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~208s 2023-09-22T22:20:26.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:26.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:26.305 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:20:26.348 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:20:26.348 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:20:26.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:26.386+0000 7fe84702da40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:26.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:26.386+0000 7fe84702da40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:26.406 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:26Z"} 2023-09-22T22:20:26.407 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:26Z"} 2023-09-22T22:20:26.407 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:20:26.407 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:26.407 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:26.407 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:20:26.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~214s 2023-09-22T22:20:27.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~214s 2023-09-22T22:20:31.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:31.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:31.409 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:20:31.452 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:20:31.452 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:20:31.490 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:31.490+0000 7f6a4c9daa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:31.491 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:31.490+0000 7f6a4c9daa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:31.509 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:31Z"} 2023-09-22T22:20:31.510 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:31Z"} 2023-09-22T22:20:31.510 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:20:31.510 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:31.510 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:31.510 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:20:32.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~219s 2023-09-22T22:20:32.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~219s 2023-09-22T22:20:36.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:36.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:36.511 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:20:36.554 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:20:36.554 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:20:36.593 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:36.590+0000 7f8bd2901a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:36.594 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:36.594+0000 7f8bd2901a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:36.612 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:36Z"} 2023-09-22T22:20:36.612 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:36Z"} 2023-09-22T22:20:36.612 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:20:36.612 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:36.612 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:36.613 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:20:37.837 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~225s 2023-09-22T22:20:38.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~225s 2023-09-22T22:20:41.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:41.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:41.615 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:20:41.659 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:20:41.659 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:20:41.699 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:41.698+0000 7fedb4eb0a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:41.700 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:41.698+0000 7fedb4eb0a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:41.719 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:41Z"} 2023-09-22T22:20:41.719 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:41Z"} 2023-09-22T22:20:41.720 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:20:41.720 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:41.720 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:41.720 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:20:43.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~230s 2023-09-22T22:20:43.969 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~231s 2023-09-22T22:20:46.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:46.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:46.721 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:20:46.765 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:20:46.765 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:20:46.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:46.802+0000 7fc472373a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:46.804 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:46.802+0000 7fc472373a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:46.822 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:46Z"} 2023-09-22T22:20:46.822 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:46Z"} 2023-09-22T22:20:46.822 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:20:46.823 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:46.823 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:46.823 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:20:49.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~236s 2023-09-22T22:20:49.575 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~236s 2023-09-22T22:20:51.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:51.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:51.824 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:20:51.867 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:20:51.867 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:20:51.905 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:51.902+0000 7fcda0c26a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:51.906 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:51.906+0000 7fcda0c26a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:51.925 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:51Z"} 2023-09-22T22:20:51.925 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:51Z"} 2023-09-22T22:20:51.925 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:20:51.925 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:51.925 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:51.926 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:20:54.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~242s 2023-09-22T22:20:55.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~242s 2023-09-22T22:20:56.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:56.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:20:56.927 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:20:56.972 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:20:56.972 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:20:57.013 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:57.010+0000 7fb8f5369a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:57.014 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:20:57.014+0000 7fb8f5369a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:20:57.033 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:57Z"} 2023-09-22T22:20:57.034 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:57Z"} 2023-09-22T22:20:57.034 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:20:57.034 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:57.034 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:20:57.034 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:21:00.258 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~247s 2023-09-22T22:21:00.782 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~247s 2023-09-22T22:21:02.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:02.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:02.036 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:21:02.080 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:21:02.080 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:21:02.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:02.118+0000 7f652d971a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:02.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:02.118+0000 7f652d971a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:02.138 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:02Z"} 2023-09-22T22:21:02.138 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:02Z"} 2023-09-22T22:21:02.138 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:21:02.139 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:02.139 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:02.139 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:21:05.863 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~253s 2023-09-22T22:21:06.386 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~253s 2023-09-22T22:21:07.140 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:07.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:07.140 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:21:07.184 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:21:07.184 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:21:07.223 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:07.222+0000 7fa22e0b7a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:07.224 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:07.222+0000 7fa22e0b7a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:07.242 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:07Z"} 2023-09-22T22:21:07.242 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:07Z"} 2023-09-22T22:21:07.242 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:21:07.242 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:07.243 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:07.243 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:21:11.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~259s 2023-09-22T22:21:11.991 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~259s 2023-09-22T22:21:12.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:12.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:12.244 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:21:12.288 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:21:12.288 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:21:12.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:12.326+0000 7f6f5660ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:12.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:12.326+0000 7f6f5660ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:12.347 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:12Z"} 2023-09-22T22:21:12.347 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:12Z"} 2023-09-22T22:21:12.347 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:21:12.347 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:12.348 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:12.348 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:21:17.073 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~264s 2023-09-22T22:21:17.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:17.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:17.349 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:21:17.393 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:21:17.393 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:21:17.430 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:17.429+0000 7f782078aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:17.431 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:17.429+0000 7f782078aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:17.449 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:17Z"} 2023-09-22T22:21:17.449 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:17Z"} 2023-09-22T22:21:17.450 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:21:17.450 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:17.450 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:17.450 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:21:17.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~264s 2023-09-22T22:21:22.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:22.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:22.452 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:21:22.496 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:21:22.496 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:21:22.535 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:22.533+0000 7f1cb8dbfa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:22.535 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:22.533+0000 7f1cb8dbfa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:22.554 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:22Z"} 2023-09-22T22:21:22.554 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:22Z"} 2023-09-22T22:21:22.554 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:21:22.554 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:22.555 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:22.555 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:21:22.679 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~270s 2023-09-22T22:21:23.201 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~270s 2023-09-22T22:21:27.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:27.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:27.556 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:21:27.600 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:21:27.600 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:21:27.639 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:27.637+0000 7f9bfdc96a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:27.640 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:27.637+0000 7f9bfdc96a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:27.658 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:27Z"} 2023-09-22T22:21:27.658 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:27Z"} 2023-09-22T22:21:27.659 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:21:27.659 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:27.659 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:27.659 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:21:28.284 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~275s 2023-09-22T22:21:28.806 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~275s 2023-09-22T22:21:32.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:32.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:32.661 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:21:32.703 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:21:32.703 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:21:32.743 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:32.745+0000 7fba3b903a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:32.744 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:32.745+0000 7fba3b903a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:32.763 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:32Z"} 2023-09-22T22:21:32.763 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:32Z"} 2023-09-22T22:21:32.763 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:21:32.764 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:32.764 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:32.764 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:21:33.889 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~281s 2023-09-22T22:21:34.410 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~281s 2023-09-22T22:21:37.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:37.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:37.765 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:21:37.809 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:21:37.809 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:21:37.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:37.849+0000 7f7a5aa8fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:37.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:37.849+0000 7f7a5aa8fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:37.867 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:37Z"} 2023-09-22T22:21:37.867 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:37Z"} 2023-09-22T22:21:37.867 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:21:37.867 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:37.868 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:37.868 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:21:39.495 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~287s 2023-09-22T22:21:40.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~287s 2023-09-22T22:21:42.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:42.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:42.869 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:21:42.913 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:21:42.913 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:21:42.956 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:42.957+0000 7f71354c8a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:42.957 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:42.957+0000 7f71354c8a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:42.978 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:42Z"} 2023-09-22T22:21:42.978 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:42Z"} 2023-09-22T22:21:42.979 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:21:42.979 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:42.979 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:42.979 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:21:45.100 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~292s 2023-09-22T22:21:45.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~292s 2023-09-22T22:21:47.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:47.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:47.981 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:21:48.024 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:21:48.024 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:21:48.065 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:48.065+0000 7fd88e498a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:48.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:48.069+0000 7fd88e498a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:48.085 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:48Z"} 2023-09-22T22:21:48.085 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:48Z"} 2023-09-22T22:21:48.085 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:21:48.085 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:48.085 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:48.086 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:21:50.704 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~298s 2023-09-22T22:21:51.224 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~298s 2023-09-22T22:21:53.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:53.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:53.088 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:21:53.130 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:21:53.131 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:21:53.168 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:53.169+0000 7faa854eaa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:53.169 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:21:53.169+0000 7faa854eaa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:21:53.188 INFO:teuthology.orchestra.run.smithi160.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:53Z"} 2023-09-22T22:21:53.188 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:53Z"} 2023-09-22T22:21:53.188 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:21:53.188 DEBUG:rgw_multi.tests:log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:53.188 DEBUG:rgw_multi.tests:sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:21:53.188 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:21:56.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~303s 2023-09-22T22:21:56.310 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-09-22T22:21:56.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~303s 2023-09-22T22:21:56.830 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-09-22T22:21:56.914 INFO:tasks.ceph.c1.osd.0:Sent signal 15 2023-09-22T22:21:56.914 INFO:tasks.ceph.c1.osd.1:Sent signal 15 2023-09-22T22:21:56.915 INFO:tasks.ceph.c1.osd.2:Sent signal 15 2023-09-22T22:21:56.915 INFO:tasks.ceph.c1.mon.a:Sent signal 15 2023-09-22T22:21:56.915 INFO:tasks.rgw.c1.client.0:Sent signal 15 2023-09-22T22:21:56.915 INFO:tasks.ceph.c1.mgr.x:Sent signal 15 2023-09-22T22:21:56.916 INFO:tasks.ceph.c1.osd.0.smithi160.stderr:2023-09-22T22:21:56.917+0000 7fe09a3fc640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c1 -i 0 (PID: 33372) UID: 0 2023-09-22T22:21:56.916 INFO:tasks.ceph.c1.osd.0.smithi160.stderr:2023-09-22T22:21:56.917+0000 7fe09a3fc640 -1 osd.0 66 *** Got signal Terminated *** 2023-09-22T22:21:56.916 INFO:tasks.ceph.c1.osd.0.smithi160.stderr:2023-09-22T22:21:56.917+0000 7fe09a3fc640 -1 osd.0 66 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-22T22:21:56.917 INFO:tasks.ceph.c1.osd.1.smithi160.stderr:2023-09-22T22:21:56.917+0000 7f1dfcffd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c1 -i 1 (PID: 33375) UID: 0 2023-09-22T22:21:56.917 INFO:tasks.ceph.c1.osd.1.smithi160.stderr:2023-09-22T22:21:56.917+0000 7f1dfcffd640 -1 osd.1 66 *** Got signal Terminated *** 2023-09-22T22:21:56.917 INFO:tasks.ceph.c1.osd.1.smithi160.stderr:2023-09-22T22:21:56.917+0000 7f1dfcffd640 -1 osd.1 66 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-22T22:21:56.917 INFO:tasks.ceph.c1.osd.2.smithi160.stderr:2023-09-22T22:21:56.917+0000 7fefff281640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c1 -i 2 (PID: 33374) UID: 0 2023-09-22T22:21:56.917 INFO:tasks.ceph.c1.osd.2.smithi160.stderr:2023-09-22T22:21:56.917+0000 7fefff281640 -1 osd.2 66 *** Got signal Terminated *** 2023-09-22T22:21:56.917 INFO:tasks.ceph.c1.osd.2.smithi160.stderr:2023-09-22T22:21:56.917+0000 7fefff281640 -1 osd.2 66 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-22T22:21:56.918 INFO:tasks.ceph.c1.mon.a.smithi160.stderr:2023-09-22T22:21:56.917+0000 7f7f3c01f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster c1 -i a (PID: 33199) UID: 0 2023-09-22T22:21:56.918 INFO:tasks.ceph.c1.mon.a.smithi160.stderr:2023-09-22T22:21:56.917+0000 7f7f3c01f640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-09-22T22:21:56.918 INFO:tasks.rgw.c1.client.0.smithi160.stdout:2023-09-22T22:21:56.917+0000 7fa9e36b9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm (PID: 39446) UID: 0 2023-09-22T22:21:56.919 INFO:tasks.rgw.c1.client.0.smithi160.stdout:2023-09-22T22:21:56.917+0000 7fa9e6f00a40 -1 shutting down 2023-09-22T22:21:56.919 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T22:21:56.917+0000 7f3fe34a2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster c1 -i x (PID: 33201) UID: 0 2023-09-22T22:21:56.919 INFO:tasks.ceph.c1.mgr.x.smithi160.stderr:2023-09-22T22:21:56.917+0000 7f3fe34a2640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-09-22T22:21:57.435 INFO:tasks.ceph.c2.osd.0:Sent signal 15 2023-09-22T22:21:57.435 INFO:tasks.ceph.c2.osd.1:Sent signal 15 2023-09-22T22:21:57.435 INFO:tasks.ceph.c2.osd.2:Sent signal 15 2023-09-22T22:21:57.435 INFO:tasks.ceph.c2.mon.a:Sent signal 15 2023-09-22T22:21:57.436 INFO:tasks.rgw.c2.client.1:Sent signal 15 2023-09-22T22:21:57.436 INFO:tasks.ceph.c2.mgr.x:Sent signal 15 2023-09-22T22:21:57.437 INFO:tasks.ceph.c2.osd.0.smithi204.stderr:2023-09-22T22:21:57.431+0000 7f49831e0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c2 -i 0 (PID: 33354) UID: 0 2023-09-22T22:21:57.437 INFO:tasks.ceph.c2.osd.0.smithi204.stderr:2023-09-22T22:21:57.431+0000 7f49831e0640 -1 osd.0 55 *** Got signal Terminated *** 2023-09-22T22:21:57.437 INFO:tasks.ceph.c2.osd.0.smithi204.stderr:2023-09-22T22:21:57.431+0000 7f49831e0640 -1 osd.0 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-22T22:21:57.437 INFO:tasks.ceph.c2.osd.2.smithi204.stderr:2023-09-22T22:21:57.435+0000 7f22e957d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c2 -i 2 (PID: 33357) UID: 0 2023-09-22T22:21:57.437 INFO:tasks.ceph.c2.osd.1.smithi204.stderr:2023-09-22T22:21:57.435+0000 7f3efa06f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c2 -i 1 (PID: 33358) UID: 0 2023-09-22T22:21:57.438 INFO:tasks.ceph.c2.osd.1.smithi204.stderr:2023-09-22T22:21:57.435+0000 7f3efa06f640 -1 osd.1 55 *** Got signal Terminated *** 2023-09-22T22:21:57.438 INFO:tasks.ceph.c2.osd.1.smithi204.stderr:2023-09-22T22:21:57.435+0000 7f3efa06f640 -1 osd.1 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-22T22:21:57.438 INFO:tasks.ceph.c2.osd.2.smithi204.stderr:2023-09-22T22:21:57.435+0000 7f22e957d640 -1 osd.2 55 *** Got signal Terminated *** 2023-09-22T22:21:57.438 INFO:tasks.ceph.c2.osd.2.smithi204.stderr:2023-09-22T22:21:57.435+0000 7f22e957d640 -1 osd.2 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-22T22:21:57.439 INFO:tasks.ceph.c2.mon.a.smithi204.stderr:2023-09-22T22:21:57.435+0000 7efe0545a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster c2 -i a (PID: 33182) UID: 0 2023-09-22T22:21:57.439 INFO:tasks.ceph.c2.mon.a.smithi204.stderr:2023-09-22T22:21:57.435+0000 7efe0545a640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-09-22T22:21:57.439 INFO:tasks.rgw.c2.client.1.smithi204.stdout:2023-09-22T22:21:57.435+0000 7fb94ea33640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm (PID: 40448) UID: 0 2023-09-22T22:21:57.439 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T22:21:57.435+0000 7fc33b2af640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster c2 -i x (PID: 33184) UID: 0 2023-09-22T22:21:57.440 INFO:tasks.ceph.c2.mgr.x.smithi204.stderr:2023-09-22T22:21:57.435+0000 7fc33b2af640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-09-22T22:21:57.524 INFO:tasks.rgw.c2.client.1.smithi204.stdout:2023-09-22T22:21:57.435+0000 7fb95227aa40 -1 shutting down 2023-09-22T22:21:58.191 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... FAIL 2023-09-22T22:21:58.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:58.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:21:58.193 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:21:58.235 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:21:58.236 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:26:58.238 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:26:58.240+0000 7f9a722e4a40 0 monclient(hunting): authenticate timed out after 300 2023-09-22T22:26:58.239 INFO:teuthology.orchestra.run.smithi160.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-09-22T22:26:58.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T22:26:58.243 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... FAIL 2023-09-22T22:26:58.244 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2023-09-22T22:26:58.245 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2023-09-22T22:26:58.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:26:58.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:26:58.246 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:26:58.286 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:26:58.286 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:31:58.288 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:31:58.288+0000 7f1171518a40 0 monclient(hunting): authenticate timed out after 300 2023-09-22T22:31:58.288 INFO:teuthology.orchestra.run.smithi160.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-09-22T22:31:58.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T22:31:58.291 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... FAIL 2023-09-22T22:31:58.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:31:58.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:31:58.292 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:31:58.376 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:31:58.376 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:36:58.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:36:58.379+0000 7f722ad25a40 0 monclient(hunting): authenticate timed out after 300 2023-09-22T22:36:58.404 INFO:teuthology.orchestra.run.smithi160.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-09-22T22:36:58.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T22:36:58.406 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... FAIL 2023-09-22T22:36:58.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:36:58.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:36:58.407 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:36:58.488 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:36:58.488 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:41:58.490 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:41:58.491+0000 7ffa565c3a40 0 monclient(hunting): authenticate timed out after 300 2023-09-22T22:41:58.501 INFO:teuthology.orchestra.run.smithi160.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-09-22T22:41:58.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T22:41:58.503 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... FAIL 2023-09-22T22:41:58.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:41:58.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:41:58.504 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:41:58.589 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:41:58.589 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:46:58.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:46:58.590+0000 7f96eae72a40 0 monclient(hunting): authenticate timed out after 300 2023-09-22T22:46:58.614 INFO:teuthology.orchestra.run.smithi160.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-09-22T22:46:58.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T22:46:58.616 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... FAIL 2023-09-22T22:46:58.616 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.617 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-22T22:46:58.617 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init 2023-09-22T22:46:58.617 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-22T22:46:58.617 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-22T22:46:58.617 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-22T22:46:58.617 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-22T22:46:58.618 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 1813, in test_bucket_incremental_sync_after_data_sync_init 2023-09-22T22:46:58.618 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-09-22T22:46:58.618 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-09-22T22:46:58.618 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-09-22T22:46:58.618 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-09-22T22:46:58.618 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-09-22T22:46:58.618 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-22T22:46:58.618 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-22T22:46:58.618 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-22T22:46:58.618 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-22T22:46:58.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/ 2023-09-22T22:46:58.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/ 2023-09-22T22:46:58.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/ 2023-09-22T22:46:58.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.619 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.619 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.620 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.620 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:15 GMT 2023-09-22T22:46:58.620 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/ 2023-09-22T22:46:58.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.620 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:KIjkGkLx+Aoy4Z2hwb2ldHKgSNw= 2023-09-22T22:46:58.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KIjkGkLx+Aoy4Z2hwb2ldHKgSNw=', 'Content-Length': '0'} 2023-09-22T22:46:58.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000072964fee0a5ba293-00650e0e33-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=ffzdqu-62 2023-09-22T22:46:58.620 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:46:58.620 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:46:58.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:46:58.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:46:58.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:46:58.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T22:46:58.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: [{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"1_1695419384.000133_332.1","last_update":"2023-09-22T21:49:44.000133Z"},{"marker":"1_1695419955.520713_604.1","last_update":"2023-09-22T21:59:15.520713Z"},{"marker":"1_1695419383.971498_240.1","last_update":"2023-09-22T21:49:43.971498Z"}] 2023-09-22T22:46:58.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419955.520713_604.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:46:58.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-09-22T22:46:58.621 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:46:58.621 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:46:58.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:46:58.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:46:58.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:46:58.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T21:59:15.728+0000 7fcb262e4a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:46:58.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T21:59:15.728+0000 7fcb262e4a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:46:58.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419955.520713_604.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:59:15.520713Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:59:15Z"} 2023-09-22T22:46:58.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695419955.520713_604.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:59:15.520713Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T21:59:15Z"} 2023-09-22T22:46:58.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695419955.520713_604.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:59:15.520713Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:46:58.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419955.520713_604.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:46:58.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695419955.520713_604.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:46:58.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-09-22T22:46:58.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/ 2023-09-22T22:46:58.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/ 2023-09-22T22:46:58.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/ 2023-09-22T22:46:58.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.623 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.623 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.623 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.623 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:15 GMT 2023-09-22T22:46:58.623 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/ 2023-09-22T22:46:58.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.624 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:wQNcODYrvpGbytqbF0wr7kur69g= 2023-09-22T22:46:58.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wQNcODYrvpGbytqbF0wr7kur69g=', 'Content-Length': '0'} 2023-09-22T22:46:58.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034dfe618db7807c2-00650e0e33-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/dummy 2023-09-22T22:46:58.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/dummy 2023-09-22T22:46:58.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/dummy 2023-09-22T22:46:58.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.625 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.625 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.625 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.625 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:15 GMT 2023-09-22T22:46:58.625 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/dummy 2023-09-22T22:46:58.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.625 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:ZhGAl+27u2pexLtw8H4f0bHB9C0= 2023-09-22T22:46:58.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:59:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZhGAl+27u2pexLtw8H4f0bHB9C0='} 2023-09-22T22:46:58.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3ff3a170283450e-00650e0e33-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-62 2023-09-22T22:46:58.626 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:46:58.626 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:46:58.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T22:46:58.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:46:58.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:46:58.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T21:59:15.872+0000 7f28598d8a40 1 waiting to reach incremental sync.. 2023-09-22T22:46:58.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T21:59:20.876+0000 7f28598d8a40 1 waiting to reach incremental sync.. 2023-09-22T22:46:58.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T21:59:25.876+0000 7f28598d8a40 1 bucket sync caught up with source: 2023-09-22T22:46:58.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:46:58.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:46:58.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T21:59:25.876+0000 7f28598d8a40 0 bucket checkpoint complete 2023-09-22T22:46:58.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-62 2023-09-22T22:46:58.627 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:46:58.627 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:46:58.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T22:46:58.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:46:58.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:46:58.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T21:59:26.000+0000 7f4b643d9a40 1 waiting to reach incremental sync.. 2023-09-22T22:46:58.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T21:59:31.000+0000 7f4b643d9a40 1 waiting to reach incremental sync.. 2023-09-22T22:46:58.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T21:59:36.000+0000 7f4b643d9a40 1 waiting to reach incremental sync.. 2023-09-22T22:46:58.627 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi160.stdout: INFO: 2023-09-22T21:59:36.004+0000 7fa9c1675640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T22:46:58.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T21:59:41.004+0000 7f4b643d9a40 1 bucket sync caught up with source: 2023-09-22T22:46:58.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.362.6, , , , , ] 2023-09-22T22:46:58.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.362.6, , , , , ] 2023-09-22T22:46:58.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T21:59:41.004+0000 7f4b643d9a40 0 bucket checkpoint complete 2023-09-22T22:46:58.628 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=ffzdqu-62 zones={a2, a1} 2023-09-22T22:46:58.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/ 2023-09-22T22:46:58.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/ 2023-09-22T22:46:58.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/ 2023-09-22T22:46:58.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.629 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.629 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.629 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.629 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.629 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/ 2023-09-22T22:46:58.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.629 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI= 2023-09-22T22:46:58.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI=', 'Content-Length': '0'} 2023-09-22T22:46:58.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005841c00c28494a3d-00650e0e4d-1edb-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/ 2023-09-22T22:46:58.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/ 2023-09-22T22:46:58.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/ 2023-09-22T22:46:58.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.630 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.630 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.631 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.631 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.631 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/ 2023-09-22T22:46:58.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.631 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI= 2023-09-22T22:46:58.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI=', 'Content-Length': '0'} 2023-09-22T22:46:58.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c18b68f049027c63-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.631 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-09-22T22:46:58.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/ 2023-09-22T22:46:58.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/ 2023-09-22T22:46:58.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/?versions&max-keys=999 2023-09-22T22:46:58.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/?versions&max-keys=999 2023-09-22T22:46:58.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/?versions&max-keys=999 2023-09-22T22:46:58.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.632 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.632 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.632 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.632 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.633 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/?versions 2023-09-22T22:46:58.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.633 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:whFfJb2kG8JY8Y2809KJdFShzdY= 2023-09-22T22:46:58.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:whFfJb2kG8JY8Y2809KJdFShzdY=', 'Content-Length': '0'} 2023-09-22T22:46:58.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af6689742bfcd8a2-00650e0e4d-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-62999falsedummynulltrue2023-09-22T21:59:15.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.633 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=dummy 2023-09-22T22:46:58.633 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-09-22T22:46:58.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/ 2023-09-22T22:46:58.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/ 2023-09-22T22:46:58.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/?versions&max-keys=999 2023-09-22T22:46:58.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/?versions&max-keys=999 2023-09-22T22:46:58.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/?versions&max-keys=999 2023-09-22T22:46:58.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.634 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.634 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.635 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.635 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.635 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/?versions 2023-09-22T22:46:58.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.635 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:whFfJb2kG8JY8Y2809KJdFShzdY= 2023-09-22T22:46:58.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:whFfJb2kG8JY8Y2809KJdFShzdY=', 'Content-Length': '0'} 2023-09-22T22:46:58.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc3a124040236878-00650e0e4d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-62999falsedummynulltrue2023-09-22T21:59:15.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.635 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=dummy 2023-09-22T22:46:58.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/ 2023-09-22T22:46:58.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/ 2023-09-22T22:46:58.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/?versions&max-keys=999 2023-09-22T22:46:58.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/?versions&max-keys=999 2023-09-22T22:46:58.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/?versions&max-keys=999 2023-09-22T22:46:58.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.636 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.636 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.637 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.637 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.637 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/?versions 2023-09-22T22:46:58.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.637 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:whFfJb2kG8JY8Y2809KJdFShzdY= 2023-09-22T22:46:58.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:whFfJb2kG8JY8Y2809KJdFShzdY=', 'Content-Length': '0'} 2023-09-22T22:46:58.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdeff3a8510f0bef-00650e0e4d-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-62999falsedummynulltrue2023-09-22T21:59:15.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/ 2023-09-22T22:46:58.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/ 2023-09-22T22:46:58.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/?versions&max-keys=999 2023-09-22T22:46:58.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/?versions&max-keys=999 2023-09-22T22:46:58.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/?versions&max-keys=999 2023-09-22T22:46:58.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.638 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.638 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.638 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.639 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.639 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/?versions 2023-09-22T22:46:58.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.639 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:whFfJb2kG8JY8Y2809KJdFShzdY= 2023-09-22T22:46:58.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:whFfJb2kG8JY8Y2809KJdFShzdY=', 'Content-Length': '0'} 2023-09-22T22:46:58.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001361a8be13945673-00650e0e4d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-62999falsedummynulltrue2023-09-22T21:59:15.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.639 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=dummy 2023-09-22T22:46:58.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/dummy 2023-09-22T22:46:58.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/dummy 2023-09-22T22:46:58.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.640 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.640 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.640 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.641 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.641 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.641 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:XK8T/qDhiV47hG/1jddhZE/IwdE= 2023-09-22T22:46:58.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XK8T/qDhiV47hG/1jddhZE/IwdE=', 'Content-Length': '0'} 2023-09-22T22:46:58.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-62:f132bdb9-20a7-4073-a932-8f99da20691e.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bd328ee38fd55a3-00650e0e4d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/dummy 2023-09-22T22:46:58.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/dummy 2023-09-22T22:46:58.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.642 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.642 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.642 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.642 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.642 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.643 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:XK8T/qDhiV47hG/1jddhZE/IwdE= 2023-09-22T22:46:58.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XK8T/qDhiV47hG/1jddhZE/IwdE=', 'Content-Length': '0'} 2023-09-22T22:46:58.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078520e28c4f8d683-00650e0e4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/dummy 2023-09-22T22:46:58.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/dummy 2023-09-22T22:46:58.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.644 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.644 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.644 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.644 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.644 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.644 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:BShKvrVw3lQWzqreBuyS1KfnsGY= 2023-09-22T22:46:58.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BShKvrVw3lQWzqreBuyS1KfnsGY=', 'Content-Length': '0'} 2023-09-22T22:46:58.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-62:f132bdb9-20a7-4073-a932-8f99da20691e.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff34d390cd8c32d3-00650e0e4d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/dummy 2023-09-22T22:46:58.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/dummy 2023-09-22T22:46:58.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.646 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.646 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.646 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.646 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.646 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.646 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:BShKvrVw3lQWzqreBuyS1KfnsGY= 2023-09-22T22:46:58.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BShKvrVw3lQWzqreBuyS1KfnsGY=', 'Content-Length': '0'} 2023-09-22T22:46:58.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081994114629c5c41-00650e0e4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.647 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=dummy 2023-09-22T22:46:58.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/dummy 2023-09-22T22:46:58.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/dummy 2023-09-22T22:46:58.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.648 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.648 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.648 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.648 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.648 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.648 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:XK8T/qDhiV47hG/1jddhZE/IwdE= 2023-09-22T22:46:58.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XK8T/qDhiV47hG/1jddhZE/IwdE=', 'Content-Length': '0'} 2023-09-22T22:46:58.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-62:f132bdb9-20a7-4073-a932-8f99da20691e.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e59c2afd494104f-00650e0e4d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/dummy 2023-09-22T22:46:58.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/dummy 2023-09-22T22:46:58.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.649 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.650 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.650 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.650 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.650 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/dummy?versionId=null 2023-09-22T22:46:58.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.650 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:XK8T/qDhiV47hG/1jddhZE/IwdE= 2023-09-22T22:46:58.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XK8T/qDhiV47hG/1jddhZE/IwdE=', 'Content-Length': '0'} 2023-09-22T22:46:58.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000031a2b72c9cc6b6d-00650e0e4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/ 2023-09-22T22:46:58.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/ 2023-09-22T22:46:58.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/?prefix=dummy 2023-09-22T22:46:58.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/?prefix=dummy 2023-09-22T22:46:58.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/?prefix=dummy 2023-09-22T22:46:58.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.651 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.651 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.651 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.652 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.652 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/ 2023-09-22T22:46:58.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.652 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:HyhWzt8jfbBeBMV01XtFDFD3y28= 2023-09-22T22:46:58.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HyhWzt8jfbBeBMV01XtFDFD3y28=', 'Content-Length': '0'} 2023-09-22T22:46:58.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef0709acb1ffd8ad-00650e0e4d-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-62dummy1000falsedummy2023-09-22T21:59:15.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/ 2023-09-22T22:46:58.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/ 2023-09-22T22:46:58.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/?prefix=dummy 2023-09-22T22:46:58.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/?prefix=dummy 2023-09-22T22:46:58.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/?prefix=dummy 2023-09-22T22:46:58.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.653 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.653 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.653 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.653 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.654 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/ 2023-09-22T22:46:58.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.654 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:HyhWzt8jfbBeBMV01XtFDFD3y28= 2023-09-22T22:46:58.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HyhWzt8jfbBeBMV01XtFDFD3y28=', 'Content-Length': '0'} 2023-09-22T22:46:58.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004adc563163cd9231-00650e0e4d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-62dummy1000falsedummy2023-09-22T21:59:15.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/dummy 2023-09-22T22:46:58.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/dummy 2023-09-22T22:46:58.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/dummy 2023-09-22T22:46:58.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.655 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.655 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.655 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.655 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.655 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/dummy 2023-09-22T22:46:58.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.655 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:+2ONW0AV9/gDBySOy0YU9MxEp6s= 2023-09-22T22:46:58.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+2ONW0AV9/gDBySOy0YU9MxEp6s=', 'Content-Length': '0'} 2023-09-22T22:46:58.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-62:f132bdb9-20a7-4073-a932-8f99da20691e.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000422c5817d1ea1d3-00650e0e4d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/dummy 2023-09-22T22:46:58.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/dummy 2023-09-22T22:46:58.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/dummy 2023-09-22T22:46:58.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.657 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.657 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.657 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.657 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.657 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/dummy 2023-09-22T22:46:58.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.657 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:+2ONW0AV9/gDBySOy0YU9MxEp6s= 2023-09-22T22:46:58.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+2ONW0AV9/gDBySOy0YU9MxEp6s=', 'Content-Length': '0'} 2023-09-22T22:46:58.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086406a2144efe5c9-00650e0e4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.657 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=dummy 2023-09-22T22:46:58.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/dummy 2023-09-22T22:46:58.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/dummy 2023-09-22T22:46:58.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/dummy 2023-09-22T22:46:58.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.658 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.659 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.659 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.659 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.659 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/dummy 2023-09-22T22:46:58.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.659 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:FdjZcrUTxfvgqZgBlkXN4zVMr7A= 2023-09-22T22:46:58.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FdjZcrUTxfvgqZgBlkXN4zVMr7A=', 'Content-Length': '0'} 2023-09-22T22:46:58.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-62:f132bdb9-20a7-4073-a932-8f99da20691e.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2c000c86bea9e97-00650e0e4d-1edb-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/dummy 2023-09-22T22:46:58.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/dummy 2023-09-22T22:46:58.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/dummy 2023-09-22T22:46:58.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.660 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.660 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.660 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.660 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.660 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/dummy 2023-09-22T22:46:58.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.661 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:FdjZcrUTxfvgqZgBlkXN4zVMr7A= 2023-09-22T22:46:58.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FdjZcrUTxfvgqZgBlkXN4zVMr7A=', 'Content-Length': '0'} 2023-09-22T22:46:58.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 21:59:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4f8a7da0dc8e06d-00650e0e4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/ 2023-09-22T22:46:58.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/ 2023-09-22T22:46:58.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/?prefix=dummy 2023-09-22T22:46:58.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/?prefix=dummy 2023-09-22T22:46:58.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/?prefix=dummy 2023-09-22T22:46:58.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.662 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.662 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.662 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.662 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.662 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/ 2023-09-22T22:46:58.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.662 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:HyhWzt8jfbBeBMV01XtFDFD3y28= 2023-09-22T22:46:58.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HyhWzt8jfbBeBMV01XtFDFD3y28=', 'Content-Length': '0'} 2023-09-22T22:46:58.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047374d626296d3f8-00650e0e4d-1edb-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-62dummy1000falsedummy2023-09-22T21:59:15.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/ 2023-09-22T22:46:58.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/ 2023-09-22T22:46:58.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/?prefix=dummy 2023-09-22T22:46:58.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/?prefix=dummy 2023-09-22T22:46:58.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/?prefix=dummy 2023-09-22T22:46:58.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.664 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.664 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.664 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.664 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.664 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/ 2023-09-22T22:46:58.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.664 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:HyhWzt8jfbBeBMV01XtFDFD3y28= 2023-09-22T22:46:58.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HyhWzt8jfbBeBMV01XtFDFD3y28=', 'Content-Length': '0'} 2023-09-22T22:46:58.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1a64e6a54359ba3-00650e0e4d-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-62dummy1000falsedummy2023-09-22T21:59:15.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.665 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=ffzdqu-62 zones={a2, a1} 2023-09-22T22:46:58.665 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: DEBUG: waiting for process to exit 2023-09-22T22:46:58.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2023-09-22T22:46:58.665 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T21:59:41.336+0000 7ff66bbbc640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 82185) UID: 0 2023-09-22T22:46:58.665 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T21:59:41.336+0000 7ff66f403a40 -1 shutting down 2023-09-22T22:46:58.665 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7ff64a379640 time 2023-09-22T21:59:41.362087+0000 2023-09-22T22:46:58.665 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T22:46:58.665 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:46:58.665 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7ff672f27883] 2023-09-22T22:46:58.665 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7ff672f27a37] 2023-09-22T22:46:58.665 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x559da71b446c] 2023-09-22T22:46:58.666 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x559da7596910] 2023-09-22T22:46:58.666 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x559da71c056e] 2023-09-22T22:46:58.666 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x559da71c1e7b] 2023-09-22T22:46:58.666 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x559da71c2ced] 2023-09-22T22:46:58.666 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x559da75c9712] 2023-09-22T22:46:58.666 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 9: radosgw(+0x836db8) [0x559da7374db8] 2023-09-22T22:46:58.666 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x559da7375753] 2023-09-22T22:46:58.666 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff671bf6b43] 2023-09-22T22:46:58.666 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff671c88a00] 2023-09-22T22:46:58.666 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T21:59:41.360+0000 7ff64a379640 -1 ./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7ff64a379640 time 2023-09-22T21:59:41.362087+0000 2023-09-22T22:46:58.666 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T22:46:58.666 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:46:58.667 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:46:58.667 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7ff672f27883] 2023-09-22T22:46:58.667 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7ff672f27a37] 2023-09-22T22:46:58.667 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x559da71b446c] 2023-09-22T22:46:58.667 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x559da7596910] 2023-09-22T22:46:58.667 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x559da71c056e] 2023-09-22T22:46:58.667 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x559da71c1e7b] 2023-09-22T22:46:58.667 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x559da71c2ced] 2023-09-22T22:46:58.667 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x559da75c9712] 2023-09-22T22:46:58.667 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 9: radosgw(+0x836db8) [0x559da7374db8] 2023-09-22T22:46:58.667 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x559da7375753] 2023-09-22T22:46:58.667 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff671bf6b43] 2023-09-22T22:46:58.668 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff671c88a00] 2023-09-22T22:46:58.668 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:46:58.668 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: *** Caught signal (Aborted) ** 2023-09-22T22:46:58.668 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: in thread 7ff64a379640 thread_name:data-sync 2023-09-22T22:46:58.668 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:46:58.668 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff671ba4520] 2023-09-22T22:46:58.668 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2: pthread_kill() 2023-09-22T22:46:58.668 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 3: raise() 2023-09-22T22:46:58.668 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 4: abort() 2023-09-22T22:46:58.668 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7ff672f278dd] 2023-09-22T22:46:58.668 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7ff672f27a37] 2023-09-22T22:46:58.668 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x559da71b446c] 2023-09-22T22:46:58.669 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x559da7596910] 2023-09-22T22:46:58.669 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x559da71c056e] 2023-09-22T22:46:58.669 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x559da71c1e7b] 2023-09-22T22:46:58.669 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x559da71c2ced] 2023-09-22T22:46:58.669 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x559da75c9712] 2023-09-22T22:46:58.669 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 13: radosgw(+0x836db8) [0x559da7374db8] 2023-09-22T22:46:58.669 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x559da7375753] 2023-09-22T22:46:58.669 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff671bf6b43] 2023-09-22T22:46:58.669 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff671c88a00] 2023-09-22T22:46:58.669 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T21:59:41.364+0000 7ff64a379640 -1 *** Caught signal (Aborted) ** 2023-09-22T22:46:58.669 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: in thread 7ff64a379640 thread_name:data-sync 2023-09-22T22:46:58.669 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:46:58.670 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:46:58.670 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff671ba4520] 2023-09-22T22:46:58.670 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2: pthread_kill() 2023-09-22T22:46:58.670 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 3: raise() 2023-09-22T22:46:58.670 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 4: abort() 2023-09-22T22:46:58.670 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7ff672f278dd] 2023-09-22T22:46:58.670 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7ff672f27a37] 2023-09-22T22:46:58.670 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x559da71b446c] 2023-09-22T22:46:58.670 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x559da7596910] 2023-09-22T22:46:58.670 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x559da71c056e] 2023-09-22T22:46:58.670 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x559da71c1e7b] 2023-09-22T22:46:58.670 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x559da71c2ced] 2023-09-22T22:46:58.671 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x559da75c9712] 2023-09-22T22:46:58.671 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 13: radosgw(+0x836db8) [0x559da7374db8] 2023-09-22T22:46:58.671 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x559da7375753] 2023-09-22T22:46:58.671 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff671bf6b43] 2023-09-22T22:46:58.671 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff671c88a00] 2023-09-22T22:46:58.671 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-09-22T22:46:58.671 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:46:58.671 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: -87> 2023-09-22T21:59:41.336+0000 7ff66bbbc640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 82185) UID: 0 2023-09-22T22:46:58.671 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: -84> 2023-09-22T21:59:41.336+0000 7ff66f403a40 -1 shutting down 2023-09-22T22:46:58.671 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: -1> 2023-09-22T21:59:41.360+0000 7ff64a379640 -1 ./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7ff64a379640 time 2023-09-22T21:59:41.362087+0000 2023-09-22T22:46:58.671 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T22:46:58.671 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:46:58.672 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:46:58.672 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7ff672f27883] 2023-09-22T22:46:58.672 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7ff672f27a37] 2023-09-22T22:46:58.672 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x559da71b446c] 2023-09-22T22:46:58.672 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x559da7596910] 2023-09-22T22:46:58.672 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x559da71c056e] 2023-09-22T22:46:58.672 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x559da71c1e7b] 2023-09-22T22:46:58.672 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x559da71c2ced] 2023-09-22T22:46:58.672 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x559da75c9712] 2023-09-22T22:46:58.672 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 9: radosgw(+0x836db8) [0x559da7374db8] 2023-09-22T22:46:58.672 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x559da7375753] 2023-09-22T22:46:58.672 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff671bf6b43] 2023-09-22T22:46:58.673 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff671c88a00] 2023-09-22T22:46:58.673 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:46:58.673 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 0> 2023-09-22T21:59:41.364+0000 7ff64a379640 -1 *** Caught signal (Aborted) ** 2023-09-22T22:46:58.673 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: in thread 7ff64a379640 thread_name:data-sync 2023-09-22T22:46:58.673 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:46:58.673 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:46:58.673 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff671ba4520] 2023-09-22T22:46:58.673 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2: pthread_kill() 2023-09-22T22:46:58.673 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 3: raise() 2023-09-22T22:46:58.673 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 4: abort() 2023-09-22T22:46:58.673 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7ff672f278dd] 2023-09-22T22:46:58.673 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7ff672f27a37] 2023-09-22T22:46:58.674 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x559da71b446c] 2023-09-22T22:46:58.674 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x559da7596910] 2023-09-22T22:46:58.674 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x559da71c056e] 2023-09-22T22:46:58.674 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x559da71c1e7b] 2023-09-22T22:46:58.674 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x559da71c2ced] 2023-09-22T22:46:58.674 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x559da75c9712] 2023-09-22T22:46:58.674 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 13: radosgw(+0x836db8) [0x559da7374db8] 2023-09-22T22:46:58.674 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x559da7375753] 2023-09-22T22:46:58.674 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff671bf6b43] 2023-09-22T22:46:58.674 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff671c88a00] 2023-09-22T22:46:58.674 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-09-22T22:46:58.674 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:46:58.675 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stderr: INFO: daemon-helper: command crashed with signal 11 2023-09-22T22:46:58.675 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Stopped 2023-09-22T22:46:58.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/ 2023-09-22T22:46:58.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/ 2023-09-22T22:46:58.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/ 2023-09-22T22:46:58.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.676 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.676 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.676 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.676 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.676 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/ 2023-09-22T22:46:58.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.676 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI= 2023-09-22T22:46:58.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI=', 'Content-Length': '0'} 2023-09-22T22:46:58.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000180c83bddb08a93d-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/obj11 2023-09-22T22:46:58.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/obj11 2023-09-22T22:46:58.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/obj11 2023-09-22T22:46:58.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.677 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.677 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.677 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.677 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.678 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/obj11 2023-09-22T22:46:58.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.678 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:4gLRkTN0hXrI7R6APS7/ZMe+U+A= 2023-09-22T22:46:58.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4gLRkTN0hXrI7R6APS7/ZMe+U+A='} 2023-09-22T22:46:58.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd6caa301c837722-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/ 2023-09-22T22:46:58.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/ 2023-09-22T22:46:58.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/ 2023-09-22T22:46:58.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.679 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.679 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.679 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.679 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.679 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/ 2023-09-22T22:46:58.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.679 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI= 2023-09-22T22:46:58.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI=', 'Content-Length': '0'} 2023-09-22T22:46:58.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4c856d9265de288-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/obj22 2023-09-22T22:46:58.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/obj22 2023-09-22T22:46:58.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/obj22 2023-09-22T22:46:58.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.681 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.681 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.681 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.681 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.681 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/obj22 2023-09-22T22:46:58.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.681 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:G1sCdLXauStwzYcFXcA69scYsU8= 2023-09-22T22:46:58.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G1sCdLXauStwzYcFXcA69scYsU8='} 2023-09-22T22:46:58.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa346b8ca634108a-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/ 2023-09-22T22:46:58.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/ 2023-09-22T22:46:58.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/ 2023-09-22T22:46:58.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.682 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.682 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.682 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.682 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.683 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/ 2023-09-22T22:46:58.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.683 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI= 2023-09-22T22:46:58.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI=', 'Content-Length': '0'} 2023-09-22T22:46:58.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023576e4b74b72bdf-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/obj33 2023-09-22T22:46:58.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/obj33 2023-09-22T22:46:58.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/obj33 2023-09-22T22:46:58.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.684 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.684 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.684 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.684 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.684 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/obj33 2023-09-22T22:46:58.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.684 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:5r7YCnwtALBZ8g6teOUDmzpKZSM= 2023-09-22T22:46:58.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5r7YCnwtALBZ8g6teOUDmzpKZSM='} 2023-09-22T22:46:58.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4878125c6453179-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/ 2023-09-22T22:46:58.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/ 2023-09-22T22:46:58.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/ 2023-09-22T22:46:58.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.686 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.686 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.686 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.686 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.686 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/ 2023-09-22T22:46:58.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.686 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI= 2023-09-22T22:46:58.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI=', 'Content-Length': '0'} 2023-09-22T22:46:58.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fda0651de41f6af6-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/obj44 2023-09-22T22:46:58.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/obj44 2023-09-22T22:46:58.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/obj44 2023-09-22T22:46:58.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.687 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.687 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.687 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.688 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.688 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/obj44 2023-09-22T22:46:58.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.688 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:8xoiE6mm+GiSZ9zpX4igRXA7LGE= 2023-09-22T22:46:58.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8xoiE6mm+GiSZ9zpX4igRXA7LGE='} 2023-09-22T22:46:58.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008538c06a8ce0727a-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/ 2023-09-22T22:46:58.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/ 2023-09-22T22:46:58.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/ 2023-09-22T22:46:58.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.689 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.689 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.689 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.689 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.689 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/ 2023-09-22T22:46:58.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.689 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI= 2023-09-22T22:46:58.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lfxyS6s+4wGMo4XrjgXNTrB6GJI=', 'Content-Length': '0'} 2023-09-22T22:46:58.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1df83734ae2dd87-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-62/obj55 2023-09-22T22:46:58.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-62/obj55 2023-09-22T22:46:58.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-62/obj55 2023-09-22T22:46:58.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.691 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.691 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.691 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.691 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 21:59:41 GMT 2023-09-22T22:46:58.691 INFO:tasks.rgw_multisite_tests:/ffzdqu-62/obj55 2023-09-22T22:46:58.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.691 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:ZXktj0CnX8k04WHPiK/Rm8H+I2c= 2023-09-22T22:46:58.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 21:59:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZXktj0CnX8k04WHPiK/Rm8H+I2c='} 2023-09-22T22:46:58.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ffa4b8fb304a038f-00650e0e4d-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.691 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:46:58.692 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:46:58.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:46:58.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:46:58.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:46:58.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T21:59:41.704+0000 7f4898cafa40 5 RGW-SYNC:data: waiting on data sync status lease 2023-09-22T22:46:58.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T21:59:41.704+0000 7f4898cafa40 5 RGW-SYNC:data: acquired data sync status lease 2023-09-22T22:46:58.692 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Restarting daemon 2023-09-22T22:46:58.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-22T22:46:58.692 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Started 2023-09-22T22:46:58.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:46:58.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-22T22:46:58.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T22:46:58.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-09-22T22:46:58.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T22:46:58.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:46:58.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-22T22:46:58.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T22:46:58.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-09-22T22:46:58.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T22:46:58.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:46:58.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-22T22:46:58.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-09-22T22:46:58.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T22:46:58.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T22:46:58.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:46:58.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-22T22:46:58.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T22:46:58.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-09-22T22:46:58.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T22:46:58.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:46:58.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-22T22:46:58.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T22:46:58.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-09-22T22:46:58.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T22:46:58.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:46:58.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-22T22:46:58.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 136k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-22T22:46:58.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: anonymous 2023-09-22T22:46:58.695 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-62 2023-09-22T22:46:58.695 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:46:58.695 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:46:58.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T22:46:58.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:46:58.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:46:58.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:00:13.111+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:46:58.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.696 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:00:14.147+0000 7f7bd38e3640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T22:46:58.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:00:18.115+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:46:58.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:00:23.115+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:46:58.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:00:28.119+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:46:58.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:00:33.123+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:46:58.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:00:38.123+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:46:58.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:00:43.127+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:46:58.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:00:48.127+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:46:58.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:00:53.135+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:46:58.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:00:58.135+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:46:58.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:01:03.138+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:46:58.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:01:08.142+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:46:58.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:01:13.142+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:46:58.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:01:18.146+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:46:58.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:01:23.146+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:46:58.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:01:28.150+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:46:58.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:01:33.154+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:46:58.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:01:38.154+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:46:58.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:01:43.158+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , , ] 2023-09-22T22:46:58.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:01:48.162+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:46:58.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:01:53.162+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:46:58.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:01:58.166+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:46:58.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:02:03.166+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:46:58.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:02:08.170+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:46:58.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:02:13.173+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:46:58.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:02:18.173+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:46:58.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:02:23.181+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:46:58.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:02:28.185+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:46:58.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:02:33.189+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:46:58.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:02:38.193+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:46:58.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:02:43.197+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000001.96.6] 2023-09-22T22:46:58.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:02:48.197+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:02:53.201+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:02:58.205+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:03:03.205+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:03:08.209+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:03:13.209+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:03:18.213+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:03:23.217+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:03:28.216+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:03:33.220+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:03:38.220+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:03:43.224+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:03:48.228+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:03:53.228+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:03:58.232+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:04:03.232+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:04:08.236+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:04:13.240+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:04:18.240+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:04:23.244+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:04:28.244+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:04:33.248+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:04:38.251+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:04:43.251+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:04:48.255+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:04:53.255+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:04:58.259+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:05:03.259+0000 7fc33889ba40 1 waiting for incremental sync to catch up: 2023-09-22T22:46:58.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , 00000000001.96.6, , , , 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , 00000000001.96.6, , , 00000000001.51.6, 00000000002.112.6, 00000000002.98.6] 2023-09-22T22:46:58.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:05:08.263+0000 7fc33889ba40 0 bucket checkpoint timed out waiting for incremental sync to catch up 2023-09-22T22:46:58.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:05:08.263+0000 7fc33889ba40 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-22T22:46:58.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:05:08.263+0000 7fc33889ba40 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-22T22:46:58.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-09-22T22:46:58.711 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-22T22:46:58.711 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.711 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-22T22:46:58.711 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-09-22T22:46:58.711 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-22T22:46:58.713 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-22T22:46:58.713 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-22T22:46:58.713 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-22T22:46:58.713 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 1861, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-09-22T22:46:58.713 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-09-22T22:46:58.713 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-09-22T22:46:58.714 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-09-22T22:46:58.714 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-09-22T22:46:58.714 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-09-22T22:46:58.714 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-22T22:46:58.714 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-22T22:46:58.714 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-22T22:46:58.714 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-22T22:46:58.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T22:46:58.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.715 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.715 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.715 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.715 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:08 GMT 2023-09-22T22:46:58.715 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.716 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:0wh2zKc2uZo0OGrGQmNMw9Iw/GY= 2023-09-22T22:46:58.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:08 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0wh2zKc2uZo0OGrGQmNMw9Iw/GY=', 'Content-Length': '0'} 2023-09-22T22:46:58.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000000f82e8cc78f9cc1-00650e0f94-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=ffzdqu-63 2023-09-22T22:46:58.716 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:46:58.716 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:46:58.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:46:58.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:46:58.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:46:58.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T22:46:58.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: [{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"1_1695419384.000133_332.1","last_update":"2023-09-22T21:49:44.000133Z"},{"marker":"1_1695420308.359160_609.1","last_update":"2023-09-22T22:05:08.359160Z"},{"marker":"1_1695419383.971498_240.1","last_update":"2023-09-22T21:49:43.971498Z"}] 2023-09-22T22:46:58.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420308.359160_609.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:46:58.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-09-22T22:46:58.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:46:58.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:46:58.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:46:58.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:46:58.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:46:58.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:05:08.583+0000 7fdebc370a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:46:58.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:05:08.583+0000 7fdebc370a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:46:58.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420308.344032_606.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:05:08.344032Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:05:08Z"} 2023-09-22T22:46:58.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420308.344032_606.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:05:08.344032Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:05:08Z"} 2023-09-22T22:46:58.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420308.344032_606.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:05:08.344032Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:46:58.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420308.359160_609.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:46:58.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420308.344032_606.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:46:58.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695420308.359160_609.1 target=1_1695420308.344032_606.1 2023-09-22T22:46:58.718 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:46:58.718 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:46:58.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:46:58.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:46:58.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:46:58.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:05:13.687+0000 7f4c4f92ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:46:58.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:05:13.691+0000 7f4c4f92ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:46:58.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420308.359160_609.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:05:08.359160Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:05:13Z"} 2023-09-22T22:46:58.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420308.359160_609.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:05:08.359160Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:05:13Z"} 2023-09-22T22:46:58.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420308.359160_609.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:05:08.359160Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:46:58.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420308.359160_609.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:46:58.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420308.359160_609.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:46:58.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-09-22T22:46:58.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.720 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.720 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.720 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.720 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:13 GMT 2023-09-22T22:46:58.720 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.720 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Y8MimPEpPTm4vni54PayPyzDfeA= 2023-09-22T22:46:58.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y8MimPEpPTm4vni54PayPyzDfeA=', 'Content-Length': '0'} 2023-09-22T22:46:58.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006432f24b88b85b58-00650e0f99-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj11 2023-09-22T22:46:58.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj11 2023-09-22T22:46:58.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj11 2023-09-22T22:46:58.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.722 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.722 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.722 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.722 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:13 GMT 2023-09-22T22:46:58.722 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj11 2023-09-22T22:46:58.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.722 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Mqp0cA7RwXh1w24OznCpY4mbNiw= 2023-09-22T22:46:58.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Mqp0cA7RwXh1w24OznCpY4mbNiw='} 2023-09-22T22:46:58.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002d9325dd78e9fc92-00650e0f99-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.723 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.723 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.723 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.724 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:13 GMT 2023-09-22T22:46:58.724 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.724 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Y8MimPEpPTm4vni54PayPyzDfeA= 2023-09-22T22:46:58.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y8MimPEpPTm4vni54PayPyzDfeA=', 'Content-Length': '0'} 2023-09-22T22:46:58.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2740b101737ba92-00650e0f99-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj22 2023-09-22T22:46:58.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj22 2023-09-22T22:46:58.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj22 2023-09-22T22:46:58.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.725 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.725 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.725 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.725 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:13 GMT 2023-09-22T22:46:58.725 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj22 2023-09-22T22:46:58.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.725 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:4PO+0BWdlhmkonqM/RkKBEcJeYo= 2023-09-22T22:46:58.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4PO+0BWdlhmkonqM/RkKBEcJeYo='} 2023-09-22T22:46:58.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000065fa785ca600546-00650e0f99-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.727 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.727 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.727 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.727 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:13 GMT 2023-09-22T22:46:58.727 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.727 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Y8MimPEpPTm4vni54PayPyzDfeA= 2023-09-22T22:46:58.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y8MimPEpPTm4vni54PayPyzDfeA=', 'Content-Length': '0'} 2023-09-22T22:46:58.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000917309ca6945d4c5-00650e0f99-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj33 2023-09-22T22:46:58.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj33 2023-09-22T22:46:58.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj33 2023-09-22T22:46:58.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.728 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.728 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.728 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.728 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:13 GMT 2023-09-22T22:46:58.729 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj33 2023-09-22T22:46:58.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.729 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:g8aZfu6MYIHYbI6jD1NpzQvPNM4= 2023-09-22T22:46:58.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:g8aZfu6MYIHYbI6jD1NpzQvPNM4='} 2023-09-22T22:46:58.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025e8c2ef910991e9-00650e0f99-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.730 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.730 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.730 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.730 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:13 GMT 2023-09-22T22:46:58.730 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.730 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Y8MimPEpPTm4vni54PayPyzDfeA= 2023-09-22T22:46:58.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y8MimPEpPTm4vni54PayPyzDfeA=', 'Content-Length': '0'} 2023-09-22T22:46:58.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001354c5e777c700e5-00650e0f99-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj44 2023-09-22T22:46:58.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj44 2023-09-22T22:46:58.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj44 2023-09-22T22:46:58.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.732 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.732 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.732 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.732 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:13 GMT 2023-09-22T22:46:58.732 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj44 2023-09-22T22:46:58.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.732 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:jQLzJCUjG0g15NpdomH6h+WjAp4= 2023-09-22T22:46:58.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jQLzJCUjG0g15NpdomH6h+WjAp4='} 2023-09-22T22:46:58.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f778c8ffd7c8e9fa-00650e0f99-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.733 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.733 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.734 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.734 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:13 GMT 2023-09-22T22:46:58.734 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.734 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Y8MimPEpPTm4vni54PayPyzDfeA= 2023-09-22T22:46:58.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Y8MimPEpPTm4vni54PayPyzDfeA=', 'Content-Length': '0'} 2023-09-22T22:46:58.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ab6e84cd1beea4f-00650e0f99-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj55 2023-09-22T22:46:58.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj55 2023-09-22T22:46:58.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj55 2023-09-22T22:46:58.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.735 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.735 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.735 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.735 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:13 GMT 2023-09-22T22:46:58.735 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj55 2023-09-22T22:46:58.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.736 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:yFPyanA5+4dDAjR2a3uYRWqFUUc= 2023-09-22T22:46:58.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yFPyanA5+4dDAjR2a3uYRWqFUUc='} 2023-09-22T22:46:58.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000672ef00ed86a81c5-00650e0f99-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.736 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:46:58.736 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:46:58.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:46:58.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:46:58.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:46:58.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: tenant: 2023-09-22T22:46:58.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: bucket name: ffzdqu-63 2023-09-22T22:46:58.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: total entries: 5 2023-09-22T22:46:58.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.737 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.738 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.738 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.738 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.738 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.738 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:46:58.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:46:58.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000865f3d0993c5d69e-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj17 2023-09-22T22:46:58.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj17 2023-09-22T22:46:58.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj17 2023-09-22T22:46:58.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.739 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.739 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.739 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.739 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.739 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj17 2023-09-22T22:46:58.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.740 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:lXNddYeGDSb63G5cVg4qzyLAF70= 2023-09-22T22:46:58.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lXNddYeGDSb63G5cVg4qzyLAF70='} 2023-09-22T22:46:58.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c43503f5585cdd90-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.741 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.741 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.741 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.741 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.741 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.741 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:46:58.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:46:58.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4558d2007f58520-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj34 2023-09-22T22:46:58.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj34 2023-09-22T22:46:58.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj34 2023-09-22T22:46:58.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.742 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.743 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.743 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.743 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.743 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj34 2023-09-22T22:46:58.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.743 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:AG/R2Wjh/SaEsZLn6PywH6ODNt0= 2023-09-22T22:46:58.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AG/R2Wjh/SaEsZLn6PywH6ODNt0='} 2023-09-22T22:46:58.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ea01f3a2244014fe-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.744 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.744 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.744 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.744 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.744 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.745 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:46:58.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:46:58.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f49f091017a70b7-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj51 2023-09-22T22:46:58.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj51 2023-09-22T22:46:58.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj51 2023-09-22T22:46:58.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.746 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.746 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.746 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.746 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.746 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj51 2023-09-22T22:46:58.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.746 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:IH50w5lC5jH2AgFH0DMPn3+e+yM= 2023-09-22T22:46:58.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IH50w5lC5jH2AgFH0DMPn3+e+yM='} 2023-09-22T22:46:58.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e5e51708af1c50f-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.747 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.747 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.748 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.748 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.748 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.748 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:46:58.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:46:58.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d7af7dab2627f42-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj68 2023-09-22T22:46:58.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj68 2023-09-22T22:46:58.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj68 2023-09-22T22:46:58.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.749 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.749 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.749 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.749 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.749 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj68 2023-09-22T22:46:58.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.750 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:IXu8VRNR+NJWTbIl0pMZf/XpQGk= 2023-09-22T22:46:58.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IXu8VRNR+NJWTbIl0pMZf/XpQGk='} 2023-09-22T22:46:58.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c1667f795b06156-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.751 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.751 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.751 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.751 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.751 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.751 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:46:58.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:46:58.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b597f27b3f0411a8-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj85 2023-09-22T22:46:58.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj85 2023-09-22T22:46:58.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj85 2023-09-22T22:46:58.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.752 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.752 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.753 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.753 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.753 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj85 2023-09-22T22:46:58.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.753 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:s9touLmomUYRfyAKA8YWGv1Jrx4= 2023-09-22T22:46:58.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:s9touLmomUYRfyAKA8YWGv1Jrx4='} 2023-09-22T22:46:58.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033af09fa02b48893-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.753 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:46:58.753 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:46:58.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:46:58.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:46:58.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:46:58.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: tenant: 2023-09-22T22:46:58.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: bucket name: ffzdqu-63 2023-09-22T22:46:58.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: total entries: 10 2023-09-22T22:46:58.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.755 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.755 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.755 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.755 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.755 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.755 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:46:58.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:46:58.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013eff33f8cd624f4-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj19 2023-09-22T22:46:58.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj19 2023-09-22T22:46:58.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj19 2023-09-22T22:46:58.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.756 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.756 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.757 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.757 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.757 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj19 2023-09-22T22:46:58.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.757 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:c0T76fAEzGZH1VwiIdFiU7YnOhg= 2023-09-22T22:46:58.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:c0T76fAEzGZH1VwiIdFiU7YnOhg='} 2023-09-22T22:46:58.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9994d9cf90a8189-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.758 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.758 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.758 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.758 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.758 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.759 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:46:58.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:46:58.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006281ecd2f5bfd9a1-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj38 2023-09-22T22:46:58.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj38 2023-09-22T22:46:58.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj38 2023-09-22T22:46:58.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.760 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.760 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.760 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.760 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.760 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj38 2023-09-22T22:46:58.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.760 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:HwUSNQqnTDqfjat88BxMKEZY5zc= 2023-09-22T22:46:58.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HwUSNQqnTDqfjat88BxMKEZY5zc='} 2023-09-22T22:46:58.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca9537e3eabf3431-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.761 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.761 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.762 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.762 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.762 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.762 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:46:58.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:46:58.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ed655c3d7d8da68-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj57 2023-09-22T22:46:58.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj57 2023-09-22T22:46:58.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj57 2023-09-22T22:46:58.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.763 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.763 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.763 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.763 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.763 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj57 2023-09-22T22:46:58.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.763 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:stdWr3aP9yy8Ke87dKHHrTL7Q8U= 2023-09-22T22:46:58.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:stdWr3aP9yy8Ke87dKHHrTL7Q8U='} 2023-09-22T22:46:58.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2830b6b48bc65ef-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.765 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.765 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.765 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.765 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.765 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.765 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:46:58.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:46:58.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e049e49825599422-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj76 2023-09-22T22:46:58.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj76 2023-09-22T22:46:58.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj76 2023-09-22T22:46:58.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.766 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.766 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.766 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.767 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.767 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj76 2023-09-22T22:46:58.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.767 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:llzLD2d0/sGxl1olFiWU4THw58M= 2023-09-22T22:46:58.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:llzLD2d0/sGxl1olFiWU4THw58M='} 2023-09-22T22:46:58.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aa16129cf5ea7409-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.768 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.768 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.768 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.768 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.768 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.768 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:46:58.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:46:58.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3571597473b3761-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj95 2023-09-22T22:46:58.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj95 2023-09-22T22:46:58.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj95 2023-09-22T22:46:58.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.770 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.770 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.770 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.770 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.770 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj95 2023-09-22T22:46:58.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.770 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:VYXVWdh4OESFZcQStNAho8zZ0kQ= 2023-09-22T22:46:58.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VYXVWdh4OESFZcQStNAho8zZ0kQ='} 2023-09-22T22:46:58.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000080ffc7252cd9cf8e-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.771 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:46:58.771 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:46:58.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:46:58.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:46:58.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:46:58.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: tenant: 2023-09-22T22:46:58.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: bucket name: ffzdqu-63 2023-09-22T22:46:58.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: total entries: 15 2023-09-22T22:46:58.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.772 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.772 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.772 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.772 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.773 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.773 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:46:58.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:46:58.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006aa1ce981f7f358e-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj23 2023-09-22T22:46:58.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj23 2023-09-22T22:46:58.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj23 2023-09-22T22:46:58.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.774 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.774 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.774 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.774 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.774 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj23 2023-09-22T22:46:58.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.774 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:3tGrlYdc+EP0GZQWRw6nfS5t27w= 2023-09-22T22:46:58.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3tGrlYdc+EP0GZQWRw6nfS5t27w='} 2023-09-22T22:46:58.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000004f9c3ae3f54e10c-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.776 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.776 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.776 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.776 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.776 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.776 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:46:58.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:46:58.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dab71441e84d11e9-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj46 2023-09-22T22:46:58.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj46 2023-09-22T22:46:58.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj46 2023-09-22T22:46:58.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.777 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.777 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.778 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.778 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.778 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj46 2023-09-22T22:46:58.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.778 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:eMPrMyqdKcrlbhrhS9WvhBusD6I= 2023-09-22T22:46:58.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eMPrMyqdKcrlbhrhS9WvhBusD6I='} 2023-09-22T22:46:58.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001317e00bbc8d3cb4-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.779 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.779 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.779 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.779 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.779 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.780 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:46:58.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:46:58.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f20f29a2bab43c8b-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj69 2023-09-22T22:46:58.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj69 2023-09-22T22:46:58.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj69 2023-09-22T22:46:58.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.781 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.781 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.781 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.781 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.781 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj69 2023-09-22T22:46:58.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.781 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:+hWMoOsiPMnOTOmpzj+etVLzPeQ= 2023-09-22T22:46:58.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+hWMoOsiPMnOTOmpzj+etVLzPeQ='} 2023-09-22T22:46:58.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009471ab76139b55ea-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.782 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.782 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.783 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.783 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.783 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.783 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:46:58.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:46:58.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d1c15fbe9600701-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj92 2023-09-22T22:46:58.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj92 2023-09-22T22:46:58.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj92 2023-09-22T22:46:58.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.784 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.784 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.784 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.784 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.784 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj92 2023-09-22T22:46:58.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.785 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:i6cbNbg0Z7NPXhvCWrP3vg1+dU0= 2023-09-22T22:46:58.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:i6cbNbg0Z7NPXhvCWrP3vg1+dU0='} 2023-09-22T22:46:58.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000933c5a4a71a9a23-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.786 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.786 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.786 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.786 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.786 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.786 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w= 2023-09-22T22:46:58.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:S9WnjIa9yRYbc3ZcRVjhhWf5Z5w=', 'Content-Length': '0'} 2023-09-22T22:46:58.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bad6d89b72f4fc49-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj115 2023-09-22T22:46:58.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj115 2023-09-22T22:46:58.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj115 2023-09-22T22:46:58.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.787 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.788 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.788 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.788 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:14 GMT 2023-09-22T22:46:58.788 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj115 2023-09-22T22:46:58.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.788 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:GLTkCkeu56p5fjGmIuRWLw3hC20= 2023-09-22T22:46:58.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GLTkCkeu56p5fjGmIuRWLw3hC20='} 2023-09-22T22:46:58.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ccf4c3747ec87fe5-00650e0f9a-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.788 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:46:58.788 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:46:58.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:46:58.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:46:58.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:46:58.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: tenant: 2023-09-22T22:46:58.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: bucket name: ffzdqu-63 2023-09-22T22:46:58.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: total entries: 20 2023-09-22T22:46:58.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.790 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.790 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.790 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.790 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.790 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.790 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:46:58.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:46:58.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002832cb6d1cc84109-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj29 2023-09-22T22:46:58.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj29 2023-09-22T22:46:58.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj29 2023-09-22T22:46:58.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.792 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.792 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.792 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.792 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.792 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj29 2023-09-22T22:46:58.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.792 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kIp1rDDxoKSTTYsulvt8L/NwYD8= 2023-09-22T22:46:58.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kIp1rDDxoKSTTYsulvt8L/NwYD8='} 2023-09-22T22:46:58.792 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:05:15.123+0000 7f7bd38e3640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T22:46:58.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000051ecbc9bd2f4ba5-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.793 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.793 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.793 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.794 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.794 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.794 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:46:58.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:46:58.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051b67d5900e62123-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj58 2023-09-22T22:46:58.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj58 2023-09-22T22:46:58.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj58 2023-09-22T22:46:58.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.795 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.795 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.795 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.795 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.795 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj58 2023-09-22T22:46:58.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.795 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:aEMCz98dlJp+nOIUAQofTt1yj68= 2023-09-22T22:46:58.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aEMCz98dlJp+nOIUAQofTt1yj68='} 2023-09-22T22:46:58.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fed1b496f80b5029-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.797 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.797 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.797 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.797 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.797 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.797 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:46:58.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:46:58.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004599ea8d618541be-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj87 2023-09-22T22:46:58.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj87 2023-09-22T22:46:58.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj87 2023-09-22T22:46:58.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.798 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.798 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.798 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.799 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.799 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj87 2023-09-22T22:46:58.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.799 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:0suG7HiA5ukwIJkzgsXt/p1xYu0= 2023-09-22T22:46:58.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0suG7HiA5ukwIJkzgsXt/p1xYu0='} 2023-09-22T22:46:58.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077cd2d471f7c2ea6-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.800 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.800 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.800 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.800 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.800 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.800 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:46:58.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:46:58.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089176a389c21f5fe-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj116 2023-09-22T22:46:58.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj116 2023-09-22T22:46:58.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj116 2023-09-22T22:46:58.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.802 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.802 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.802 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.802 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.802 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj116 2023-09-22T22:46:58.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.802 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sP36Z/dhzdVuZapG8/K0J+xlWwo= 2023-09-22T22:46:58.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sP36Z/dhzdVuZapG8/K0J+xlWwo='} 2023-09-22T22:46:58.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c7c2944c503a268-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.803 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.803 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.803 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.803 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.804 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.804 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:46:58.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:46:58.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000896a6a1c2d115896-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj145 2023-09-22T22:46:58.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj145 2023-09-22T22:46:58.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj145 2023-09-22T22:46:58.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.805 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.805 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.805 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.805 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.805 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj145 2023-09-22T22:46:58.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.805 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:vMRgIH1FUTDDKCRAbfaIZxncqJs= 2023-09-22T22:46:58.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vMRgIH1FUTDDKCRAbfaIZxncqJs='} 2023-09-22T22:46:58.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a9d69cc3ea6a4c0-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.806 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:46:58.806 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:46:58.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:46:58.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:46:58.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:46:58.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: tenant: 2023-09-22T22:46:58.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: bucket name: ffzdqu-63 2023-09-22T22:46:58.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: total entries: 25 2023-09-22T22:46:58.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.807 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.807 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.807 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.808 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.808 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.808 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:46:58.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:46:58.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000129595926ac7888f-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj31 2023-09-22T22:46:58.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj31 2023-09-22T22:46:58.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj31 2023-09-22T22:46:58.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.809 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.809 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.809 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.809 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.809 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj31 2023-09-22T22:46:58.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.809 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:iNbxX/mL1V2KR/2syAASToA/DBk= 2023-09-22T22:46:58.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iNbxX/mL1V2KR/2syAASToA/DBk='} 2023-09-22T22:46:58.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a6bafeb9e27af81-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.811 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.811 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.811 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.811 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.811 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.811 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:46:58.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:46:58.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ef2590a159f94ef-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj62 2023-09-22T22:46:58.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj62 2023-09-22T22:46:58.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj62 2023-09-22T22:46:58.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.812 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.812 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.812 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.813 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.813 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj62 2023-09-22T22:46:58.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.813 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:D2jo28niSCXgi7HYsXd9/haHhsU= 2023-09-22T22:46:58.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:D2jo28niSCXgi7HYsXd9/haHhsU='} 2023-09-22T22:46:58.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029923cb4d956979c-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.814 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.814 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.814 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.814 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.814 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.814 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:46:58.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:46:58.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094d9e25885d11b4e-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj93 2023-09-22T22:46:58.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj93 2023-09-22T22:46:58.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj93 2023-09-22T22:46:58.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.816 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.816 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.816 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.816 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.816 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj93 2023-09-22T22:46:58.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.816 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:dYgkq02BfzT4wkkcCxlDQ+GtYGc= 2023-09-22T22:46:58.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dYgkq02BfzT4wkkcCxlDQ+GtYGc='} 2023-09-22T22:46:58.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000651b6193bcda5f16-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.817 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.817 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.817 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.818 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.818 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.818 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:46:58.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:46:58.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f46f35ed90e6561-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj124 2023-09-22T22:46:58.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj124 2023-09-22T22:46:58.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj124 2023-09-22T22:46:58.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.819 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.819 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.819 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.819 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.819 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj124 2023-09-22T22:46:58.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.819 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:JH0S79ijGJEsCfgXk4bwynZwoe4= 2023-09-22T22:46:58.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JH0S79ijGJEsCfgXk4bwynZwoe4='} 2023-09-22T22:46:58.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000afc5e19c91a90e39-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.821 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.821 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.821 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.821 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.821 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.821 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:46:58.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:46:58.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b6f6ac4d1d9c4dd-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj155 2023-09-22T22:46:58.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj155 2023-09-22T22:46:58.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj155 2023-09-22T22:46:58.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.822 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.822 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.822 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.823 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.823 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj155 2023-09-22T22:46:58.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.823 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:7v9Y2stHJ4mejDL9AhsPSn6FmfA= 2023-09-22T22:46:58.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7v9Y2stHJ4mejDL9AhsPSn6FmfA='} 2023-09-22T22:46:58.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043bc08fe4cf5df7e-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.823 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:46:58.823 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:46:58.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:46:58.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:46:58.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:46:58.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: tenant: 2023-09-22T22:46:58.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: bucket name: ffzdqu-63 2023-09-22T22:46:58.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: total entries: 30 2023-09-22T22:46:58.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.825 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.825 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.825 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.825 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.825 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.825 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:46:58.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:46:58.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056db82d9ef6ac0b4-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj37 2023-09-22T22:46:58.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj37 2023-09-22T22:46:58.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj37 2023-09-22T22:46:58.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.826 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.826 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.826 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.827 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.827 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj37 2023-09-22T22:46:58.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.827 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:hdnzWO0AxYLo9l10XUBQmoXyCSM= 2023-09-22T22:46:58.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hdnzWO0AxYLo9l10XUBQmoXyCSM='} 2023-09-22T22:46:58.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002319d1a35bbda96a-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.828 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.828 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.828 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.828 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.828 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.828 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:46:58.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:46:58.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e11f48cd954fecf4-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj74 2023-09-22T22:46:58.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj74 2023-09-22T22:46:58.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj74 2023-09-22T22:46:58.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.830 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.830 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.830 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.830 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.830 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj74 2023-09-22T22:46:58.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.830 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:WWNgAXfi7Cbj3xBBHPG/uynL4wM= 2023-09-22T22:46:58.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WWNgAXfi7Cbj3xBBHPG/uynL4wM='} 2023-09-22T22:46:58.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036ebdd69127b76fb-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.831 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.831 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.831 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.831 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.832 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.832 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:46:58.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:46:58.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7f915c4c42735e8-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj111 2023-09-22T22:46:58.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj111 2023-09-22T22:46:58.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj111 2023-09-22T22:46:58.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.833 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.833 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.833 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.833 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.833 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj111 2023-09-22T22:46:58.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.833 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:7XvmPlMh+LB4HYDZ7Rk/Zq2mGRo= 2023-09-22T22:46:58.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7XvmPlMh+LB4HYDZ7Rk/Zq2mGRo='} 2023-09-22T22:46:58.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000023401e58001074e4-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.835 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.835 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.835 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.835 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.835 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.835 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:46:58.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:46:58.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008dc4b5f3a3fa8fd8-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj148 2023-09-22T22:46:58.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj148 2023-09-22T22:46:58.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj148 2023-09-22T22:46:58.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.836 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.836 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.836 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.837 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.837 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj148 2023-09-22T22:46:58.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.837 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Iddyv3VPHpH5c8+rKlIbp6xr6v4= 2023-09-22T22:46:58.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Iddyv3VPHpH5c8+rKlIbp6xr6v4='} 2023-09-22T22:46:58.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c864a3356cb227a-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.838 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.838 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.838 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.838 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:15 GMT 2023-09-22T22:46:58.838 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.839 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g= 2023-09-22T22:46:58.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:15 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KGna6Bf1YZFwquFt3m3o4oLni0g=', 'Content-Length': '0'} 2023-09-22T22:46:58.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005247631bd05b5516-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj185 2023-09-22T22:46:58.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj185 2023-09-22T22:46:58.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:58.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj185 2023-09-22T22:46:58.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:58.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.840 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:46:58.840 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:46:58.840 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:46:58.840 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:16 GMT 2023-09-22T22:46:58.840 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj185 2023-09-22T22:46:58.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.840 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:GelrMoTGZdYolxxpOaE99j2cTwk= 2023-09-22T22:46:58.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:16 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GelrMoTGZdYolxxpOaE99j2cTwk='} 2023-09-22T22:46:58.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004fe5c955ae744bc3-00650e0f9b-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.841 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-63 2023-09-22T22:46:58.841 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:46:58.841 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:46:58.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T22:46:58.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:46:58.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:46:58.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:05:16.103+0000 7fa098d9ca40 1 waiting to reach incremental sync.. 2023-09-22T22:46:58.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:05:21.103+0000 7fa098d9ca40 1 waiting to reach incremental sync.. 2023-09-22T22:46:58.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:05:26.111+0000 7fa098d9ca40 1 bucket sync caught up with source: 2023-09-22T22:46:58.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:46:58.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:46:58.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:05:26.111+0000 7fa098d9ca40 0 bucket checkpoint complete 2023-09-22T22:46:58.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-63 2023-09-22T22:46:58.842 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:46:58.842 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:46:58.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-63 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T22:46:58.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:46:58.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:46:58.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:05:26.279+0000 7fd3ea06da40 1 waiting to reach incremental sync.. 2023-09-22T22:46:58.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:05:31.279+0000 7fd3ea06da40 1 waiting to reach incremental sync.. 2023-09-22T22:46:58.842 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi160.stdout: INFO: 2023-09-22T22:05:31.415+0000 7fa9c1675640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T22:46:58.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:05:36.283+0000 7fd3ea06da40 1 bucket sync caught up with source: 2023-09-22T22:46:58.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, 00000000003.192.6, 00000000002.259.6, 00000000002.426.6, 00000000005.288.6, 00000000004.263.6, 00000000001.298.6, 00000000001.186.6, 00000000004.161.6, 00000000007.358.6, 00000000006.281.6] 2023-09-22T22:46:58.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, 00000000003.192.6, 00000000002.259.6, 00000000002.426.6, 00000000005.288.6, 00000000004.263.6, 00000000001.298.6, 00000000001.186.6, 00000000004.161.6, 00000000007.358.6, 00000000006.281.6] 2023-09-22T22:46:58.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:05:36.283+0000 7fd3ea06da40 0 bucket checkpoint complete 2023-09-22T22:46:58.843 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=ffzdqu-63 zones={a2, a1} 2023-09-22T22:46:58.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T22:46:58.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.844 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.844 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.844 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.844 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.844 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.844 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:VXStreklr320+CMWJc/abq+53FA= 2023-09-22T22:46:58.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VXStreklr320+CMWJc/abq+53FA=', 'Content-Length': '0'} 2023-09-22T22:46:58.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6d33b4290658527-00650e0fb0-1f3d-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:58.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.845 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.846 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.846 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.846 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.846 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.846 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:VXStreklr320+CMWJc/abq+53FA= 2023-09-22T22:46:58.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VXStreklr320+CMWJc/abq+53FA=', 'Content-Length': '0'} 2023-09-22T22:46:58.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d74f39d3c7811d5-00650e0fb0-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.846 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-09-22T22:46:58.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?versions&max-keys=999 2023-09-22T22:46:58.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?versions&max-keys=999 2023-09-22T22:46:58.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?versions&max-keys=999 2023-09-22T22:46:58.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.847 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.848 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.848 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.848 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.848 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/?versions 2023-09-22T22:46:58.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.848 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:O7ggq92aKlvvTpA9x89lddg1c2o= 2023-09-22T22:46:58.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:O7ggq92aKlvvTpA9x89lddg1c2o=', 'Content-Length': '0'} 2023-09-22T22:46:58.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009eb2c1fcd8f2f875-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63999falseobj11nulltrue2023-09-22T22:05:13.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T22:05:15.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T22:05:15.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-22T22:05:15.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T22:05:15.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T22:05:14.042Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-22T22:05:16.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T22:05:14.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T22:05:13.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T22:05:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T22:05:15.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T22:05:15.503Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T22:05:13.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T22:05:14.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-22T22:05:15.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T22:05:14.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T22:05:13.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T22:05:14.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T22:05:14.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T22:05:13.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T22:05:14.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T22:05:15.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T22:05:15.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T22:05:14.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T22:05:14.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-22T22:05:15.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T22:05:14.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T22:05:14.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T22:05:15.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T22:05:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T22:05:15.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T22:05:14.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.848 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-09-22T22:46:58.849 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj111 2023-09-22T22:46:58.849 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj115 2023-09-22T22:46:58.849 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj116 2023-09-22T22:46:58.849 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj124 2023-09-22T22:46:58.849 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj145 2023-09-22T22:46:58.849 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj148 2023-09-22T22:46:58.849 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj155 2023-09-22T22:46:58.849 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj17 2023-09-22T22:46:58.849 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj185 2023-09-22T22:46:58.849 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj19 2023-09-22T22:46:58.849 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-09-22T22:46:58.849 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj23 2023-09-22T22:46:58.849 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj29 2023-09-22T22:46:58.850 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj31 2023-09-22T22:46:58.850 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-09-22T22:46:58.850 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj34 2023-09-22T22:46:58.850 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj37 2023-09-22T22:46:58.850 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj38 2023-09-22T22:46:58.850 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-09-22T22:46:58.850 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj46 2023-09-22T22:46:58.850 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj51 2023-09-22T22:46:58.850 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-09-22T22:46:58.850 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj57 2023-09-22T22:46:58.850 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj58 2023-09-22T22:46:58.851 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj62 2023-09-22T22:46:58.851 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj68 2023-09-22T22:46:58.851 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj69 2023-09-22T22:46:58.851 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj74 2023-09-22T22:46:58.851 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj76 2023-09-22T22:46:58.851 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj85 2023-09-22T22:46:58.851 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj87 2023-09-22T22:46:58.851 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj92 2023-09-22T22:46:58.851 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj93 2023-09-22T22:46:58.851 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj95 2023-09-22T22:46:58.851 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-09-22T22:46:58.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?versions&max-keys=999 2023-09-22T22:46:58.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?versions&max-keys=999 2023-09-22T22:46:58.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?versions&max-keys=999 2023-09-22T22:46:58.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.853 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.853 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.853 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.853 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.853 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/?versions 2023-09-22T22:46:58.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.853 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:O7ggq92aKlvvTpA9x89lddg1c2o= 2023-09-22T22:46:58.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:O7ggq92aKlvvTpA9x89lddg1c2o=', 'Content-Length': '0'} 2023-09-22T22:46:58.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013ed9518d5f93a51-00650e0fb0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63999falseobj11nulltrue2023-09-22T22:05:13.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T22:05:15.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T22:05:15.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-22T22:05:15.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T22:05:15.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T22:05:14.042Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-22T22:05:16.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T22:05:14.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T22:05:13.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T22:05:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T22:05:15.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T22:05:15.503Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T22:05:13.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T22:05:14.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-22T22:05:15.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T22:05:14.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T22:05:13.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T22:05:14.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T22:05:14.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T22:05:13.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T22:05:14.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T22:05:15.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T22:05:15.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T22:05:14.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T22:05:14.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-22T22:05:15.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T22:05:14.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T22:05:14.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T22:05:15.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T22:05:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T22:05:15.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T22:05:14.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.853 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-09-22T22:46:58.854 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj111 2023-09-22T22:46:58.854 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj115 2023-09-22T22:46:58.854 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj116 2023-09-22T22:46:58.854 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj124 2023-09-22T22:46:58.854 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj145 2023-09-22T22:46:58.854 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj148 2023-09-22T22:46:58.854 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj155 2023-09-22T22:46:58.854 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj17 2023-09-22T22:46:58.854 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj185 2023-09-22T22:46:58.854 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj19 2023-09-22T22:46:58.854 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-09-22T22:46:58.854 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj23 2023-09-22T22:46:58.855 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj29 2023-09-22T22:46:58.855 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj31 2023-09-22T22:46:58.855 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-09-22T22:46:58.855 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj34 2023-09-22T22:46:58.855 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj37 2023-09-22T22:46:58.855 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj38 2023-09-22T22:46:58.855 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-09-22T22:46:58.855 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj46 2023-09-22T22:46:58.855 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj51 2023-09-22T22:46:58.855 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-09-22T22:46:58.855 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj57 2023-09-22T22:46:58.855 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj58 2023-09-22T22:46:58.856 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj62 2023-09-22T22:46:58.856 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj68 2023-09-22T22:46:58.856 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj69 2023-09-22T22:46:58.856 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj74 2023-09-22T22:46:58.856 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj76 2023-09-22T22:46:58.856 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj85 2023-09-22T22:46:58.856 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj87 2023-09-22T22:46:58.856 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj92 2023-09-22T22:46:58.856 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj93 2023-09-22T22:46:58.856 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj95 2023-09-22T22:46:58.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?versions&max-keys=999 2023-09-22T22:46:58.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?versions&max-keys=999 2023-09-22T22:46:58.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?versions&max-keys=999 2023-09-22T22:46:58.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.857 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.857 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.858 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.858 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.858 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/?versions 2023-09-22T22:46:58.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.858 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:O7ggq92aKlvvTpA9x89lddg1c2o= 2023-09-22T22:46:58.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:O7ggq92aKlvvTpA9x89lddg1c2o=', 'Content-Length': '0'} 2023-09-22T22:46:58.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e572f4a226894df-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63999falseobj11nulltrue2023-09-22T22:05:13.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T22:05:15.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T22:05:15.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-22T22:05:15.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T22:05:15.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T22:05:14.042Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-22T22:05:16.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T22:05:14.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T22:05:13.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T22:05:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T22:05:15.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T22:05:15.503Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T22:05:13.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T22:05:14.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-22T22:05:15.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T22:05:14.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T22:05:13.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T22:05:14.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T22:05:14.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T22:05:13.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T22:05:14.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T22:05:15.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T22:05:15.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T22:05:14.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T22:05:14.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-22T22:05:15.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T22:05:14.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T22:05:14.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T22:05:15.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T22:05:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T22:05:15.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T22:05:14.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?versions&max-keys=999 2023-09-22T22:46:58.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?versions&max-keys=999 2023-09-22T22:46:58.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?versions&max-keys=999 2023-09-22T22:46:58.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.859 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.859 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.860 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.860 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.860 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/?versions 2023-09-22T22:46:58.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.860 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:O7ggq92aKlvvTpA9x89lddg1c2o= 2023-09-22T22:46:58.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:O7ggq92aKlvvTpA9x89lddg1c2o=', 'Content-Length': '0'} 2023-09-22T22:46:58.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a85d29cc575a5a6-00650e0fb0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63999falseobj11nulltrue2023-09-22T22:05:13.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-22T22:05:15.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-22T22:05:15.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-22T22:05:15.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-22T22:05:15.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-22T22:05:14.042Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-22T22:05:16.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-22T22:05:14.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T22:05:13.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-22T22:05:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-22T22:05:15.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-22T22:05:15.503Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T22:05:13.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-22T22:05:14.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-22T22:05:15.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-22T22:05:14.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T22:05:13.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-22T22:05:14.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-22T22:05:14.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T22:05:13.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-22T22:05:14.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-22T22:05:15.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-22T22:05:15.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-22T22:05:14.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-22T22:05:14.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-22T22:05:15.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-22T22:05:14.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-22T22:05:14.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-22T22:05:15.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-22T22:05:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-22T22:05:15.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-22T22:05:14.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.860 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-09-22T22:46:58.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj11 2023-09-22T22:46:58.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj11 2023-09-22T22:46:58.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.861 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.862 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.862 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.862 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.862 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.862 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:gK6tup572M7PH0hYpjVUVvYT3p4= 2023-09-22T22:46:58.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gK6tup572M7PH0hYpjVUVvYT3p4=', 'Content-Length': '0'} 2023-09-22T22:46:58.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a4a6a060dc61120-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj11 2023-09-22T22:46:58.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj11 2023-09-22T22:46:58.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.863 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.863 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.863 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.864 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.864 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.864 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:gK6tup572M7PH0hYpjVUVvYT3p4= 2023-09-22T22:46:58.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gK6tup572M7PH0hYpjVUVvYT3p4=', 'Content-Length': '0'} 2023-09-22T22:46:58.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec201df7b0a21461-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj11 2023-09-22T22:46:58.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj11 2023-09-22T22:46:58.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.865 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.865 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.865 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.865 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.865 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.866 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:2Lz8XfQ7iVz2YjcPc7hdSq3HDfo= 2023-09-22T22:46:58.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2Lz8XfQ7iVz2YjcPc7hdSq3HDfo=', 'Content-Length': '0'} 2023-09-22T22:46:58.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070ede2a4dfbfaec6-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj11 2023-09-22T22:46:58.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj11 2023-09-22T22:46:58.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.867 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.867 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.867 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.867 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.867 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.867 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:2Lz8XfQ7iVz2YjcPc7hdSq3HDfo= 2023-09-22T22:46:58.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2Lz8XfQ7iVz2YjcPc7hdSq3HDfo=', 'Content-Length': '0'} 2023-09-22T22:46:58.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063814989ec024c8d-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.868 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-09-22T22:46:58.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj11 2023-09-22T22:46:58.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj11 2023-09-22T22:46:58.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.869 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.869 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.869 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.869 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.869 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.869 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:gK6tup572M7PH0hYpjVUVvYT3p4= 2023-09-22T22:46:58.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gK6tup572M7PH0hYpjVUVvYT3p4=', 'Content-Length': '0'} 2023-09-22T22:46:58.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a2e033f9cf70d01-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj11 2023-09-22T22:46:58.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj11 2023-09-22T22:46:58.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.871 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.871 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.871 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.871 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.871 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj11?versionId=null 2023-09-22T22:46:58.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.871 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:gK6tup572M7PH0hYpjVUVvYT3p4= 2023-09-22T22:46:58.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gK6tup572M7PH0hYpjVUVvYT3p4=', 'Content-Length': '0'} 2023-09-22T22:46:58.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e61fc05ec4a4c8c-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj11 2023-09-22T22:46:58.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj11 2023-09-22T22:46:58.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj11 2023-09-22T22:46:58.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.873 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.873 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.873 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.873 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.873 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.873 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:46:58.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:46:58.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060480d56b3a8c981-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj111000falseobj112023-09-22T22:05:13.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj11 2023-09-22T22:46:58.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj11 2023-09-22T22:46:58.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj11 2023-09-22T22:46:58.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.875 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.875 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.875 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.875 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.875 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.875 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:46:58.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:46:58.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006137252ed1f936e6-00650e0fb0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj111000falseobj112023-09-22T22:05:13.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj11 2023-09-22T22:46:58.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj11 2023-09-22T22:46:58.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj11 2023-09-22T22:46:58.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.876 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.876 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.876 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.877 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.877 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj11 2023-09-22T22:46:58.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.877 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:KHeCs1xe8XbuG/JRm7kTYcqUOP4= 2023-09-22T22:46:58.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KHeCs1xe8XbuG/JRm7kTYcqUOP4=', 'Content-Length': '0'} 2023-09-22T22:46:58.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d36fd6f5d9576e7a-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj11 2023-09-22T22:46:58.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj11 2023-09-22T22:46:58.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj11 2023-09-22T22:46:58.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.878 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.878 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.878 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.878 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.878 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj11 2023-09-22T22:46:58.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.878 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:KHeCs1xe8XbuG/JRm7kTYcqUOP4= 2023-09-22T22:46:58.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:KHeCs1xe8XbuG/JRm7kTYcqUOP4=', 'Content-Length': '0'} 2023-09-22T22:46:58.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000887eb3f3cab7f66f-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.879 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-09-22T22:46:58.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj11 2023-09-22T22:46:58.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj11 2023-09-22T22:46:58.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj11 2023-09-22T22:46:58.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.880 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.880 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.880 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.880 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.880 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj11 2023-09-22T22:46:58.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.880 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:09IrybW+1pJkZWTqnf6rYKBAfbQ= 2023-09-22T22:46:58.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:09IrybW+1pJkZWTqnf6rYKBAfbQ=', 'Content-Length': '0'} 2023-09-22T22:46:58.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc62df9cad051adf-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj11 2023-09-22T22:46:58.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj11 2023-09-22T22:46:58.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj11 2023-09-22T22:46:58.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.881 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.881 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.881 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.882 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.882 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj11 2023-09-22T22:46:58.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.882 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:09IrybW+1pJkZWTqnf6rYKBAfbQ= 2023-09-22T22:46:58.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:09IrybW+1pJkZWTqnf6rYKBAfbQ=', 'Content-Length': '0'} 2023-09-22T22:46:58.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006838e2b63c765dc1-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj11 2023-09-22T22:46:58.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj11 2023-09-22T22:46:58.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj11 2023-09-22T22:46:58.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.883 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.883 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.883 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.883 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.883 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.884 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:46:58.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:46:58.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc9789d92f5b4612-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj111000falseobj112023-09-22T22:05:13.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj11 2023-09-22T22:46:58.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj11 2023-09-22T22:46:58.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj11 2023-09-22T22:46:58.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.885 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.885 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.885 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.885 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.885 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.886 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:46:58.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:46:58.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000832e403a963ed282-00650e0fb0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj111000falseobj112023-09-22T22:05:13.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.886 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj111 2023-09-22T22:46:58.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj111 2023-09-22T22:46:58.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj111 2023-09-22T22:46:58.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.887 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.887 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.887 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.887 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.887 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.888 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:rf6f7mlJjp4Txk+CGqyquLg2aFE= 2023-09-22T22:46:58.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rf6f7mlJjp4Txk+CGqyquLg2aFE=', 'Content-Length': '0'} 2023-09-22T22:46:58.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f39031c3b12938fa-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj111 2023-09-22T22:46:58.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj111 2023-09-22T22:46:58.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.889 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.889 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.889 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.889 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.889 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.889 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:rf6f7mlJjp4Txk+CGqyquLg2aFE= 2023-09-22T22:46:58.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rf6f7mlJjp4Txk+CGqyquLg2aFE=', 'Content-Length': '0'} 2023-09-22T22:46:58.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007bc36bd8c6b14dc-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj111 2023-09-22T22:46:58.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj111 2023-09-22T22:46:58.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.891 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.891 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.891 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.891 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.891 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.891 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:0nkDBBO1MT4EbIS/IaRKV8sonRY= 2023-09-22T22:46:58.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0nkDBBO1MT4EbIS/IaRKV8sonRY=', 'Content-Length': '0'} 2023-09-22T22:46:58.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006fe431921d5790c-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj111 2023-09-22T22:46:58.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj111 2023-09-22T22:46:58.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.893 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.893 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.893 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.893 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.893 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.893 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:0nkDBBO1MT4EbIS/IaRKV8sonRY= 2023-09-22T22:46:58.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0nkDBBO1MT4EbIS/IaRKV8sonRY=', 'Content-Length': '0'} 2023-09-22T22:46:58.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bb8cac9933267aa-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.893 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj111 2023-09-22T22:46:58.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj111 2023-09-22T22:46:58.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj111 2023-09-22T22:46:58.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.895 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.895 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.895 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.895 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.895 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.895 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:rf6f7mlJjp4Txk+CGqyquLg2aFE= 2023-09-22T22:46:58.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rf6f7mlJjp4Txk+CGqyquLg2aFE=', 'Content-Length': '0'} 2023-09-22T22:46:58.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd245a3cc6b21bd5-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj111 2023-09-22T22:46:58.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj111 2023-09-22T22:46:58.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.896 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.896 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.897 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.897 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.897 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj111?versionId=null 2023-09-22T22:46:58.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.897 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:rf6f7mlJjp4Txk+CGqyquLg2aFE= 2023-09-22T22:46:58.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rf6f7mlJjp4Txk+CGqyquLg2aFE=', 'Content-Length': '0'} 2023-09-22T22:46:58.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b581b5f7d7c32fd-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj111 2023-09-22T22:46:58.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj111 2023-09-22T22:46:58.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj111 2023-09-22T22:46:58.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.898 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.898 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.898 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.899 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.899 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.899 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:46:58.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:46:58.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000312fb36ea09c1f30-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1111000falseobj1112023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj111 2023-09-22T22:46:58.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj111 2023-09-22T22:46:58.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj111 2023-09-22T22:46:58.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.900 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.900 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.900 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.900 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.901 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.901 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:46:58.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:46:58.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d201db9727eb268c-00650e0fb0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1111000falseobj1112023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj111 2023-09-22T22:46:58.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj111 2023-09-22T22:46:58.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj111 2023-09-22T22:46:58.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.902 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.902 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.902 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.902 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.902 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj111 2023-09-22T22:46:58.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.902 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:z0a/mZkGiedAfDJkcNmi99Zhlac= 2023-09-22T22:46:58.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:z0a/mZkGiedAfDJkcNmi99Zhlac=', 'Content-Length': '0'} 2023-09-22T22:46:58.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0e428a662c4bcdb-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj111 2023-09-22T22:46:58.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj111 2023-09-22T22:46:58.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj111 2023-09-22T22:46:58.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.904 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.904 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.904 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.904 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.904 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj111 2023-09-22T22:46:58.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.904 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:z0a/mZkGiedAfDJkcNmi99Zhlac= 2023-09-22T22:46:58.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:z0a/mZkGiedAfDJkcNmi99Zhlac=', 'Content-Length': '0'} 2023-09-22T22:46:58.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bb1a73c69efb1a3-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.904 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj111 2023-09-22T22:46:58.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj111 2023-09-22T22:46:58.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj111 2023-09-22T22:46:58.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj111 2023-09-22T22:46:58.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.905 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.905 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.906 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.906 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.906 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj111 2023-09-22T22:46:58.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.906 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:LSISURKMbShRzKpzgeowsJPP900= 2023-09-22T22:46:58.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LSISURKMbShRzKpzgeowsJPP900=', 'Content-Length': '0'} 2023-09-22T22:46:58.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000534929e4039723ee-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj111 2023-09-22T22:46:58.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj111 2023-09-22T22:46:58.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj111 2023-09-22T22:46:58.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.907 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.907 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.907 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.907 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.907 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj111 2023-09-22T22:46:58.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.908 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:LSISURKMbShRzKpzgeowsJPP900= 2023-09-22T22:46:58.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LSISURKMbShRzKpzgeowsJPP900=', 'Content-Length': '0'} 2023-09-22T22:46:58.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a38c933f35fe8228-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj111 2023-09-22T22:46:58.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj111 2023-09-22T22:46:58.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj111 2023-09-22T22:46:58.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.909 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.909 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.909 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.909 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.909 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.909 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:46:58.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:46:58.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab5b478bbe01cdc7-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1111000falseobj1112023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj111 2023-09-22T22:46:58.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj111 2023-09-22T22:46:58.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj111 2023-09-22T22:46:58.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.911 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.911 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.911 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.911 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.911 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.911 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:46:58.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:46:58.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000646fb577b2284a4e-00650e0fb0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1111000falseobj1112023-09-22T22:05:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.912 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj115 2023-09-22T22:46:58.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj115 2023-09-22T22:46:58.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj115 2023-09-22T22:46:58.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.913 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.913 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.913 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.913 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.913 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.913 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bAtWTfki/8437idRxzK0O7aHA20= 2023-09-22T22:46:58.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bAtWTfki/8437idRxzK0O7aHA20=', 'Content-Length': '0'} 2023-09-22T22:46:58.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dbb1a219675b672-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj115 2023-09-22T22:46:58.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj115 2023-09-22T22:46:58.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.915 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.915 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.915 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.915 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.915 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.915 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bAtWTfki/8437idRxzK0O7aHA20= 2023-09-22T22:46:58.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bAtWTfki/8437idRxzK0O7aHA20=', 'Content-Length': '0'} 2023-09-22T22:46:58.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d5e936ba2c8aa0f-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj115 2023-09-22T22:46:58.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj115 2023-09-22T22:46:58.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.917 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.917 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.917 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.917 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.917 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.917 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:EbiCdk54o041V0Sn0jb2Rhz/yKI= 2023-09-22T22:46:58.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EbiCdk54o041V0Sn0jb2Rhz/yKI=', 'Content-Length': '0'} 2023-09-22T22:46:58.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004067b6ca30857d0c-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj115 2023-09-22T22:46:58.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj115 2023-09-22T22:46:58.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.918 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.918 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.919 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.919 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.919 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.919 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:EbiCdk54o041V0Sn0jb2Rhz/yKI= 2023-09-22T22:46:58.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EbiCdk54o041V0Sn0jb2Rhz/yKI=', 'Content-Length': '0'} 2023-09-22T22:46:58.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1612045599955aa-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.919 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj115 2023-09-22T22:46:58.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj115 2023-09-22T22:46:58.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj115 2023-09-22T22:46:58.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.920 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.920 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.920 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.920 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.921 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.921 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bAtWTfki/8437idRxzK0O7aHA20= 2023-09-22T22:46:58.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bAtWTfki/8437idRxzK0O7aHA20=', 'Content-Length': '0'} 2023-09-22T22:46:58.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023b7aded51193570-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj115 2023-09-22T22:46:58.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj115 2023-09-22T22:46:58.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.922 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.922 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.922 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.922 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.922 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj115?versionId=null 2023-09-22T22:46:58.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.923 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bAtWTfki/8437idRxzK0O7aHA20= 2023-09-22T22:46:58.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bAtWTfki/8437idRxzK0O7aHA20=', 'Content-Length': '0'} 2023-09-22T22:46:58.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9a2e34fe36b66e9-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj115 2023-09-22T22:46:58.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj115 2023-09-22T22:46:58.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj115 2023-09-22T22:46:58.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.924 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.924 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.924 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.924 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.924 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.924 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:46:58.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:46:58.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058493da9ffbb324b-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1151000falseobj1152023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj115 2023-09-22T22:46:58.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj115 2023-09-22T22:46:58.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj115 2023-09-22T22:46:58.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.926 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.926 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.926 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.926 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.926 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.926 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:46:58.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:46:58.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2e9dfe7853187d6-00650e0fb0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1151000falseobj1152023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj115 2023-09-22T22:46:58.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj115 2023-09-22T22:46:58.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj115 2023-09-22T22:46:58.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.928 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.928 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.928 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.928 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.928 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj115 2023-09-22T22:46:58.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.928 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sz80IZbeOjCbhlSRMQcTgciJEo0= 2023-09-22T22:46:58.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sz80IZbeOjCbhlSRMQcTgciJEo0=', 'Content-Length': '0'} 2023-09-22T22:46:58.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef1e227c0a5f3340-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj115 2023-09-22T22:46:58.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj115 2023-09-22T22:46:58.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj115 2023-09-22T22:46:58.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.929 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.929 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.929 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.929 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.930 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj115 2023-09-22T22:46:58.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.930 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sz80IZbeOjCbhlSRMQcTgciJEo0= 2023-09-22T22:46:58.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sz80IZbeOjCbhlSRMQcTgciJEo0=', 'Content-Length': '0'} 2023-09-22T22:46:58.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ba44dba5145a7e2-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.930 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj115 2023-09-22T22:46:58.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj115 2023-09-22T22:46:58.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj115 2023-09-22T22:46:58.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj115 2023-09-22T22:46:58.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.931 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.931 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.931 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.931 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.931 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj115 2023-09-22T22:46:58.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.931 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:qXk8SOzcDL/NA19AdctX3W0Le98= 2023-09-22T22:46:58.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qXk8SOzcDL/NA19AdctX3W0Le98=', 'Content-Length': '0'} 2023-09-22T22:46:58.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039f10b229860dfd9-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj115 2023-09-22T22:46:58.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj115 2023-09-22T22:46:58.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj115 2023-09-22T22:46:58.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.933 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.933 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.933 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.933 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.933 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj115 2023-09-22T22:46:58.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.933 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:qXk8SOzcDL/NA19AdctX3W0Le98= 2023-09-22T22:46:58.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qXk8SOzcDL/NA19AdctX3W0Le98=', 'Content-Length': '0'} 2023-09-22T22:46:58.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c35f51444026a28b-00650e0fb0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj115 2023-09-22T22:46:58.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj115 2023-09-22T22:46:58.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj115 2023-09-22T22:46:58.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.935 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.935 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.935 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.935 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.935 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.935 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:46:58.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:46:58.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4cd6b7da165b58b-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1151000falseobj1152023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj115 2023-09-22T22:46:58.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj115 2023-09-22T22:46:58.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj115 2023-09-22T22:46:58.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.936 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.936 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.937 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.937 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.937 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.937 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc= 2023-09-22T22:46:58.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:n0hs5ctRaxIWoCDKL/T3mXfevOc=', 'Content-Length': '0'} 2023-09-22T22:46:58.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004d89ba1be40e7cd-00650e0fb0-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1151000falseobj1152023-09-22T22:05:14.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.937 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj116 2023-09-22T22:46:58.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj116 2023-09-22T22:46:58.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj116 2023-09-22T22:46:58.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.939 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.939 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.939 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.939 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:36 GMT 2023-09-22T22:46:58.939 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.939 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:DgPoQzSTMvcsJnb95MaKCSnRZOk= 2023-09-22T22:46:58.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:36 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DgPoQzSTMvcsJnb95MaKCSnRZOk=', 'Content-Length': '0'} 2023-09-22T22:46:58.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5fc3a0d8ca16e71-00650e0fb0-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj116 2023-09-22T22:46:58.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj116 2023-09-22T22:46:58.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.940 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.940 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.941 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.941 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:58.941 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.941 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:ywoI8NscNvthzFZ95UP+gtA+u/4= 2023-09-22T22:46:58.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ywoI8NscNvthzFZ95UP+gtA+u/4=', 'Content-Length': '0'} 2023-09-22T22:46:58.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ebcc4955b29a47b-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj116 2023-09-22T22:46:58.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj116 2023-09-22T22:46:58.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.942 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.942 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.942 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.942 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:58.943 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.943 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:ItjSYSfu6Ro5dVzLCkfK7ZAjJI8= 2023-09-22T22:46:58.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ItjSYSfu6Ro5dVzLCkfK7ZAjJI8=', 'Content-Length': '0'} 2023-09-22T22:46:58.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078fca83f4322e533-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj116 2023-09-22T22:46:58.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj116 2023-09-22T22:46:58.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.944 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.944 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.944 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.944 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:58.944 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.945 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:ItjSYSfu6Ro5dVzLCkfK7ZAjJI8= 2023-09-22T22:46:58.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ItjSYSfu6Ro5dVzLCkfK7ZAjJI8=', 'Content-Length': '0'} 2023-09-22T22:46:58.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000249cfb996a20ae13-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.945 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj116 2023-09-22T22:46:58.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj116 2023-09-22T22:46:58.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj116 2023-09-22T22:46:58.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.946 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.946 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.946 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.946 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:58.946 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.946 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:ywoI8NscNvthzFZ95UP+gtA+u/4= 2023-09-22T22:46:58.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ywoI8NscNvthzFZ95UP+gtA+u/4=', 'Content-Length': '0'} 2023-09-22T22:46:58.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd99a9d1f0545c5f-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj116 2023-09-22T22:46:58.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj116 2023-09-22T22:46:58.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.948 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.948 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.948 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.948 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:58.948 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj116?versionId=null 2023-09-22T22:46:58.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.948 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:ywoI8NscNvthzFZ95UP+gtA+u/4= 2023-09-22T22:46:58.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ywoI8NscNvthzFZ95UP+gtA+u/4=', 'Content-Length': '0'} 2023-09-22T22:46:58.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042d8c148bd91439e-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj116 2023-09-22T22:46:58.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj116 2023-09-22T22:46:58.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj116 2023-09-22T22:46:58.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.950 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.950 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.950 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.950 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:58.950 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.950 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:46:58.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:46:58.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005998ffb3b1146888-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1161000falseobj1162023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj116 2023-09-22T22:46:58.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj116 2023-09-22T22:46:58.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj116 2023-09-22T22:46:58.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.952 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.952 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.952 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.952 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:58.952 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.952 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:46:58.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:46:58.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092f57873179a0c86-00650e0fb1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1161000falseobj1162023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj116 2023-09-22T22:46:58.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj116 2023-09-22T22:46:58.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj116 2023-09-22T22:46:58.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.953 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.953 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.954 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.954 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:58.954 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj116 2023-09-22T22:46:58.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.954 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:SNy564q59PqpevZU7YqZw2Ow2Ok= 2023-09-22T22:46:58.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNy564q59PqpevZU7YqZw2Ow2Ok=', 'Content-Length': '0'} 2023-09-22T22:46:58.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eabf64d9cbf2364b-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj116 2023-09-22T22:46:58.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj116 2023-09-22T22:46:58.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj116 2023-09-22T22:46:58.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.955 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.955 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.955 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.955 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:58.955 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj116 2023-09-22T22:46:58.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.956 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:SNy564q59PqpevZU7YqZw2Ow2Ok= 2023-09-22T22:46:58.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:SNy564q59PqpevZU7YqZw2Ow2Ok=', 'Content-Length': '0'} 2023-09-22T22:46:58.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000897c0b22b7903809-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.956 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj116 2023-09-22T22:46:58.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj116 2023-09-22T22:46:58.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj116 2023-09-22T22:46:58.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj116 2023-09-22T22:46:58.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.957 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.957 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.957 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.957 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:58.957 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj116 2023-09-22T22:46:58.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.957 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Iwm/YA0lAjBkQN7d1/db24gxRtI= 2023-09-22T22:46:58.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Iwm/YA0lAjBkQN7d1/db24gxRtI=', 'Content-Length': '0'} 2023-09-22T22:46:58.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008873557ff6198e81-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj116 2023-09-22T22:46:58.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj116 2023-09-22T22:46:58.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj116 2023-09-22T22:46:58.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.958 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.958 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.959 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.959 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:58.959 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj116 2023-09-22T22:46:58.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.959 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Iwm/YA0lAjBkQN7d1/db24gxRtI= 2023-09-22T22:46:58.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Iwm/YA0lAjBkQN7d1/db24gxRtI=', 'Content-Length': '0'} 2023-09-22T22:46:58.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f5868f14d4352b4-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj116 2023-09-22T22:46:58.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj116 2023-09-22T22:46:58.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj116 2023-09-22T22:46:58.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.960 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.960 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.960 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.960 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:58.961 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.961 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:46:58.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:46:58.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052929a4ce3575c8d-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1161000falseobj1162023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:58.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:58.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj116 2023-09-22T22:46:58.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj116 2023-09-22T22:46:58.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj116 2023-09-22T22:46:58.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.962 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.962 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.962 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.962 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:58.963 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:58.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.963 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:46:58.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:46:58.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fe1c90601233434-00650e0fb1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1161000falseobj1162023-09-22T22:05:15.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:58.963 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj124 2023-09-22T22:46:58.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj124 2023-09-22T22:46:58.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj124 2023-09-22T22:46:58.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:58.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:58.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:58.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.964 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.964 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.964 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.965 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:58.965 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:58.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.965 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:fimquc5/yzLqcG+GY6rBy4s5Jtg= 2023-09-22T22:46:58.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fimquc5/yzLqcG+GY6rBy4s5Jtg=', 'Content-Length': '0'} 2023-09-22T22:46:58.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f34ac2c295499f69-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj124 2023-09-22T22:46:58.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj124 2023-09-22T22:46:58.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:58.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:58.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:58.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:58.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.966 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:58.966 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.966 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.966 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:58.966 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:58.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.967 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:fimquc5/yzLqcG+GY6rBy4s5Jtg= 2023-09-22T22:46:58.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fimquc5/yzLqcG+GY6rBy4s5Jtg=', 'Content-Length': '0'} 2023-09-22T22:46:58.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039b2d95eb2ef30f1-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj124 2023-09-22T22:46:58.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj124 2023-09-22T22:46:58.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:58.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:58.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:58.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:58.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:58.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.968 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.968 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.968 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.968 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:58.968 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:58.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:58.968 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:vGDXcV6Xpa8jP6j3wzrovUDwdl4= 2023-09-22T22:46:58.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vGDXcV6Xpa8jP6j3wzrovUDwdl4=', 'Content-Length': '0'} 2023-09-22T22:46:58.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b3431009229c932-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:58.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj124 2023-09-22T22:46:58.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj124 2023-09-22T22:46:58.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:58.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:58.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:58.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:58.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:58.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:58.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:58.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:58.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:58.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:58.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:58.970 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:58.970 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.970 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:58.970 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:58.970 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:58.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.131 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:vGDXcV6Xpa8jP6j3wzrovUDwdl4= 2023-09-22T22:46:59.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:vGDXcV6Xpa8jP6j3wzrovUDwdl4=', 'Content-Length': '0'} 2023-09-22T22:46:59.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000639c8b1b6670b538-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.131 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj124 2023-09-22T22:46:59.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj124 2023-09-22T22:46:59.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj124 2023-09-22T22:46:59.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:59.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:59.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:59.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.132 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.133 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.133 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.133 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.133 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:59.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.133 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:fimquc5/yzLqcG+GY6rBy4s5Jtg= 2023-09-22T22:46:59.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fimquc5/yzLqcG+GY6rBy4s5Jtg=', 'Content-Length': '0'} 2023-09-22T22:46:59.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fd54ef6bb1355d4-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj124 2023-09-22T22:46:59.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj124 2023-09-22T22:46:59.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:59.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:59.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:59.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.134 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.134 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.135 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.135 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.135 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj124?versionId=null 2023-09-22T22:46:59.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.135 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:fimquc5/yzLqcG+GY6rBy4s5Jtg= 2023-09-22T22:46:59.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fimquc5/yzLqcG+GY6rBy4s5Jtg=', 'Content-Length': '0'} 2023-09-22T22:46:59.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a13e859d1a74a8f7-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj124 2023-09-22T22:46:59.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj124 2023-09-22T22:46:59.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj124 2023-09-22T22:46:59.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.137 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.137 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.137 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.137 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.137 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.137 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:46:59.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:46:59.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dec6b3e31380df43-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1241000falseobj1242023-09-22T22:05:15.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj124 2023-09-22T22:46:59.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj124 2023-09-22T22:46:59.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj124 2023-09-22T22:46:59.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.139 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.139 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.139 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.139 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.139 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.139 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:46:59.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:46:59.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e410f379998149b1-00650e0fb1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1241000falseobj1242023-09-22T22:05:15.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj124 2023-09-22T22:46:59.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj124 2023-09-22T22:46:59.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj124 2023-09-22T22:46:59.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.140 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.141 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.141 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.141 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.141 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj124 2023-09-22T22:46:59.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.141 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:14s102Q24wkER37WtUb5udNTZf0= 2023-09-22T22:46:59.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:14s102Q24wkER37WtUb5udNTZf0=', 'Content-Length': '0'} 2023-09-22T22:46:59.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee28d331bbe3273b-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj124 2023-09-22T22:46:59.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj124 2023-09-22T22:46:59.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj124 2023-09-22T22:46:59.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.142 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.142 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.142 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.142 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.142 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj124 2023-09-22T22:46:59.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.143 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:14s102Q24wkER37WtUb5udNTZf0= 2023-09-22T22:46:59.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:14s102Q24wkER37WtUb5udNTZf0=', 'Content-Length': '0'} 2023-09-22T22:46:59.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000101acc8dde6c1d37-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.143 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj124 2023-09-22T22:46:59.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj124 2023-09-22T22:46:59.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj124 2023-09-22T22:46:59.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj124 2023-09-22T22:46:59.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.144 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.144 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.144 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.144 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.144 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj124 2023-09-22T22:46:59.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.144 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:IlWMbrbMFvShKjznecvygLxWe4Q= 2023-09-22T22:46:59.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IlWMbrbMFvShKjznecvygLxWe4Q=', 'Content-Length': '0'} 2023-09-22T22:46:59.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a8d69d7aa0a8dbc-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj124 2023-09-22T22:46:59.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj124 2023-09-22T22:46:59.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj124 2023-09-22T22:46:59.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.146 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.146 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.146 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.146 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.146 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj124 2023-09-22T22:46:59.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.146 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:IlWMbrbMFvShKjznecvygLxWe4Q= 2023-09-22T22:46:59.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:IlWMbrbMFvShKjznecvygLxWe4Q=', 'Content-Length': '0'} 2023-09-22T22:46:59.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceef9f7a6fe32da9-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj124 2023-09-22T22:46:59.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj124 2023-09-22T22:46:59.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj124 2023-09-22T22:46:59.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.148 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.148 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.148 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.148 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.148 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.148 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:46:59.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:46:59.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c6e3ff48298e893-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1241000falseobj1242023-09-22T22:05:15.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj124 2023-09-22T22:46:59.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj124 2023-09-22T22:46:59.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj124 2023-09-22T22:46:59.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.150 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.150 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.150 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.150 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.150 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.150 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:46:59.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:46:59.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3d884c2c889f5a0-00650e0fb1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1241000falseobj1242023-09-22T22:05:15.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.151 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj145 2023-09-22T22:46:59.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj145 2023-09-22T22:46:59.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj145 2023-09-22T22:46:59.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.152 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.152 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.152 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.153 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.153 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.153 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:rpXSdE1Jf+8b/bv3qzZXP+ATqTE= 2023-09-22T22:46:59.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rpXSdE1Jf+8b/bv3qzZXP+ATqTE=', 'Content-Length': '0'} 2023-09-22T22:46:59.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5b6aa8ce015e947-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj145 2023-09-22T22:46:59.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj145 2023-09-22T22:46:59.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.154 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.154 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.154 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.154 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.154 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.155 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:rpXSdE1Jf+8b/bv3qzZXP+ATqTE= 2023-09-22T22:46:59.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rpXSdE1Jf+8b/bv3qzZXP+ATqTE=', 'Content-Length': '0'} 2023-09-22T22:46:59.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005241ad99e2d3577c-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj145 2023-09-22T22:46:59.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj145 2023-09-22T22:46:59.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.156 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.156 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.156 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.156 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.156 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.157 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Snte4OVVpw1kbAUX7eGjtLpjL1M= 2023-09-22T22:46:59.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Snte4OVVpw1kbAUX7eGjtLpjL1M=', 'Content-Length': '0'} 2023-09-22T22:46:59.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094c4ebd2459b467d-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj145 2023-09-22T22:46:59.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj145 2023-09-22T22:46:59.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.158 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.158 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.158 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.158 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.158 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.158 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Snte4OVVpw1kbAUX7eGjtLpjL1M= 2023-09-22T22:46:59.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Snte4OVVpw1kbAUX7eGjtLpjL1M=', 'Content-Length': '0'} 2023-09-22T22:46:59.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e02a0201cb2f1100-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.159 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj145 2023-09-22T22:46:59.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj145 2023-09-22T22:46:59.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj145 2023-09-22T22:46:59.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.160 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.160 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.160 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.160 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.160 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.160 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:rpXSdE1Jf+8b/bv3qzZXP+ATqTE= 2023-09-22T22:46:59.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rpXSdE1Jf+8b/bv3qzZXP+ATqTE=', 'Content-Length': '0'} 2023-09-22T22:46:59.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003212822771f48c93-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj145 2023-09-22T22:46:59.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj145 2023-09-22T22:46:59.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.162 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.162 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.162 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.162 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.162 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj145?versionId=null 2023-09-22T22:46:59.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.162 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:rpXSdE1Jf+8b/bv3qzZXP+ATqTE= 2023-09-22T22:46:59.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rpXSdE1Jf+8b/bv3qzZXP+ATqTE=', 'Content-Length': '0'} 2023-09-22T22:46:59.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a4b1a056808af7c-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj145 2023-09-22T22:46:59.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj145 2023-09-22T22:46:59.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj145 2023-09-22T22:46:59.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.164 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.164 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.164 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.164 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.164 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.164 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:46:59.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:46:59.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd6bf72a20b35356-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1451000falseobj1452023-09-22T22:05:15.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj145 2023-09-22T22:46:59.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj145 2023-09-22T22:46:59.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj145 2023-09-22T22:46:59.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.166 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.166 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.166 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.166 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.166 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.166 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:46:59.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:46:59.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adc7438b332c669e-00650e0fb1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1451000falseobj1452023-09-22T22:05:15.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj145 2023-09-22T22:46:59.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj145 2023-09-22T22:46:59.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj145 2023-09-22T22:46:59.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.167 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.167 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.167 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.168 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.168 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj145 2023-09-22T22:46:59.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.168 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:rAIDCTDNYgh8n/gnx/w/dhAy1wQ= 2023-09-22T22:46:59.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rAIDCTDNYgh8n/gnx/w/dhAy1wQ=', 'Content-Length': '0'} 2023-09-22T22:46:59.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df5fd033994040a5-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj145 2023-09-22T22:46:59.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj145 2023-09-22T22:46:59.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj145 2023-09-22T22:46:59.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.169 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.169 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.169 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.169 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.169 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj145 2023-09-22T22:46:59.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.170 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:rAIDCTDNYgh8n/gnx/w/dhAy1wQ= 2023-09-22T22:46:59.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rAIDCTDNYgh8n/gnx/w/dhAy1wQ=', 'Content-Length': '0'} 2023-09-22T22:46:59.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e845f370efe5417-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.170 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj145 2023-09-22T22:46:59.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj145 2023-09-22T22:46:59.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj145 2023-09-22T22:46:59.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj145 2023-09-22T22:46:59.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.171 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.171 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.171 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.171 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.172 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj145 2023-09-22T22:46:59.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.172 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:WK1Uq3ChfGjO0Z5q8sKXj6rPtss= 2023-09-22T22:46:59.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WK1Uq3ChfGjO0Z5q8sKXj6rPtss=', 'Content-Length': '0'} 2023-09-22T22:46:59.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5fee5be6b42a29e-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj145 2023-09-22T22:46:59.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj145 2023-09-22T22:46:59.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj145 2023-09-22T22:46:59.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.173 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.173 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.174 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.174 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.174 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj145 2023-09-22T22:46:59.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.174 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:WK1Uq3ChfGjO0Z5q8sKXj6rPtss= 2023-09-22T22:46:59.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WK1Uq3ChfGjO0Z5q8sKXj6rPtss=', 'Content-Length': '0'} 2023-09-22T22:46:59.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000cbabe1f3d45625-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj145 2023-09-22T22:46:59.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj145 2023-09-22T22:46:59.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj145 2023-09-22T22:46:59.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.176 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.176 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.176 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.176 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.176 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.176 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:46:59.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:46:59.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052e1d00682bad0fd-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1451000falseobj1452023-09-22T22:05:15.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj145 2023-09-22T22:46:59.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj145 2023-09-22T22:46:59.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj145 2023-09-22T22:46:59.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.178 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.178 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.179 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.179 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.179 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.179 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:46:59.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:46:59.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8237ca2630cff22-00650e0fb1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1451000falseobj1452023-09-22T22:05:15.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.180 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj148 2023-09-22T22:46:59.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj148 2023-09-22T22:46:59.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj148 2023-09-22T22:46:59.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.181 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.181 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.181 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.181 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.181 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.181 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:1FsSUUWdrIzchAa0x6aIrJfSDPM= 2023-09-22T22:46:59.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1FsSUUWdrIzchAa0x6aIrJfSDPM=', 'Content-Length': '0'} 2023-09-22T22:46:59.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac41978b5b84eeb4-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj148 2023-09-22T22:46:59.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj148 2023-09-22T22:46:59.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.183 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.183 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.183 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.183 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.183 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.183 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:1FsSUUWdrIzchAa0x6aIrJfSDPM= 2023-09-22T22:46:59.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1FsSUUWdrIzchAa0x6aIrJfSDPM=', 'Content-Length': '0'} 2023-09-22T22:46:59.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f283182eaf4893d4-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj148 2023-09-22T22:46:59.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj148 2023-09-22T22:46:59.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.185 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.185 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.185 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.185 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.185 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.185 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:MQcJNQcHn8mqm2qbMpf8G386OUA= 2023-09-22T22:46:59.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MQcJNQcHn8mqm2qbMpf8G386OUA=', 'Content-Length': '0'} 2023-09-22T22:46:59.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f7c3d7ced628824-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj148 2023-09-22T22:46:59.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj148 2023-09-22T22:46:59.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.187 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.187 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.187 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.187 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.187 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.187 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:MQcJNQcHn8mqm2qbMpf8G386OUA= 2023-09-22T22:46:59.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MQcJNQcHn8mqm2qbMpf8G386OUA=', 'Content-Length': '0'} 2023-09-22T22:46:59.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f3f0f034ceb9066-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.187 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj148 2023-09-22T22:46:59.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj148 2023-09-22T22:46:59.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj148 2023-09-22T22:46:59.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.189 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.189 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.189 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.189 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.189 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.189 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:1FsSUUWdrIzchAa0x6aIrJfSDPM= 2023-09-22T22:46:59.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1FsSUUWdrIzchAa0x6aIrJfSDPM=', 'Content-Length': '0'} 2023-09-22T22:46:59.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000761cc8075eaf6a10-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj148 2023-09-22T22:46:59.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj148 2023-09-22T22:46:59.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.190 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.190 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.191 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.191 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.191 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj148?versionId=null 2023-09-22T22:46:59.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.191 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:1FsSUUWdrIzchAa0x6aIrJfSDPM= 2023-09-22T22:46:59.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1FsSUUWdrIzchAa0x6aIrJfSDPM=', 'Content-Length': '0'} 2023-09-22T22:46:59.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fe0a5448b84326c-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj148 2023-09-22T22:46:59.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj148 2023-09-22T22:46:59.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj148 2023-09-22T22:46:59.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.192 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.192 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.192 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.193 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.193 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.193 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:46:59.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:46:59.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb3082faef0f1ac0-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1481000falseobj1482023-09-22T22:05:15.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj148 2023-09-22T22:46:59.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj148 2023-09-22T22:46:59.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj148 2023-09-22T22:46:59.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.194 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.194 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.194 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.194 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.195 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.195 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:46:59.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:46:59.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b5a980b03fe545b-00650e0fb1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1481000falseobj1482023-09-22T22:05:15.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj148 2023-09-22T22:46:59.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj148 2023-09-22T22:46:59.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj148 2023-09-22T22:46:59.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.196 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.196 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.196 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.196 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.196 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj148 2023-09-22T22:46:59.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.197 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:gysKSRyvW8oABzgqf+Pc4iOLASk= 2023-09-22T22:46:59.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gysKSRyvW8oABzgqf+Pc4iOLASk=', 'Content-Length': '0'} 2023-09-22T22:46:59.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ca4c6679929f7b0-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj148 2023-09-22T22:46:59.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj148 2023-09-22T22:46:59.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj148 2023-09-22T22:46:59.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.198 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.198 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.198 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.198 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.198 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj148 2023-09-22T22:46:59.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.198 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:gysKSRyvW8oABzgqf+Pc4iOLASk= 2023-09-22T22:46:59.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gysKSRyvW8oABzgqf+Pc4iOLASk=', 'Content-Length': '0'} 2023-09-22T22:46:59.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004534251f3012e58a-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.199 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj148 2023-09-22T22:46:59.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj148 2023-09-22T22:46:59.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj148 2023-09-22T22:46:59.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj148 2023-09-22T22:46:59.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.200 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.200 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.200 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.200 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.200 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj148 2023-09-22T22:46:59.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.200 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Hd1vwzwyq55uyeFNUjYJyCoj5oE= 2023-09-22T22:46:59.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Hd1vwzwyq55uyeFNUjYJyCoj5oE=', 'Content-Length': '0'} 2023-09-22T22:46:59.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a419ddbcabf44380-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj148 2023-09-22T22:46:59.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj148 2023-09-22T22:46:59.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj148 2023-09-22T22:46:59.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.201 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.201 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.202 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.202 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.202 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj148 2023-09-22T22:46:59.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.202 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Hd1vwzwyq55uyeFNUjYJyCoj5oE= 2023-09-22T22:46:59.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Hd1vwzwyq55uyeFNUjYJyCoj5oE=', 'Content-Length': '0'} 2023-09-22T22:46:59.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006548f7309d96f5a7-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj148 2023-09-22T22:46:59.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj148 2023-09-22T22:46:59.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj148 2023-09-22T22:46:59.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.203 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.203 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.203 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.204 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.204 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.204 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:46:59.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:46:59.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000826cf2c3f3ba8855-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1481000falseobj1482023-09-22T22:05:15.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj148 2023-09-22T22:46:59.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj148 2023-09-22T22:46:59.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj148 2023-09-22T22:46:59.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.205 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.205 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.205 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.206 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.206 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.206 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:46:59.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:46:59.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3c5f1c4c1f37e21-00650e0fb1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1481000falseobj1482023-09-22T22:05:15.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.206 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj155 2023-09-22T22:46:59.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj155 2023-09-22T22:46:59.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj155 2023-09-22T22:46:59.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.207 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.207 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.208 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.208 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.208 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.208 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:/r3b2nZoGCN4rZRqE+a3kMF52tI= 2023-09-22T22:46:59.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/r3b2nZoGCN4rZRqE+a3kMF52tI=', 'Content-Length': '0'} 2023-09-22T22:46:59.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ebdfff93cea22cf-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj155 2023-09-22T22:46:59.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj155 2023-09-22T22:46:59.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.209 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.209 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.209 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.210 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.210 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.210 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:/r3b2nZoGCN4rZRqE+a3kMF52tI= 2023-09-22T22:46:59.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/r3b2nZoGCN4rZRqE+a3kMF52tI=', 'Content-Length': '0'} 2023-09-22T22:46:59.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000166675c4e7e37ad-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj155 2023-09-22T22:46:59.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj155 2023-09-22T22:46:59.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.211 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.211 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.212 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.212 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.212 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.212 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:HGvCOXD6BzC7MWgjACb9EiKznek= 2023-09-22T22:46:59.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HGvCOXD6BzC7MWgjACb9EiKznek=', 'Content-Length': '0'} 2023-09-22T22:46:59.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004028e195f741fe1e-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj155 2023-09-22T22:46:59.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj155 2023-09-22T22:46:59.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.214 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.214 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.214 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.215 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.215 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.215 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:HGvCOXD6BzC7MWgjACb9EiKznek= 2023-09-22T22:46:59.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HGvCOXD6BzC7MWgjACb9EiKznek=', 'Content-Length': '0'} 2023-09-22T22:46:59.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1441b98f7d2bcc7-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.215 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj155 2023-09-22T22:46:59.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj155 2023-09-22T22:46:59.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj155 2023-09-22T22:46:59.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.217 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.217 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.217 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.218 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.218 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.218 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:/r3b2nZoGCN4rZRqE+a3kMF52tI= 2023-09-22T22:46:59.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/r3b2nZoGCN4rZRqE+a3kMF52tI=', 'Content-Length': '0'} 2023-09-22T22:46:59.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063cddd83b1de0120-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj155 2023-09-22T22:46:59.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj155 2023-09-22T22:46:59.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.220 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.220 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.220 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.220 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.220 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj155?versionId=null 2023-09-22T22:46:59.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.220 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:/r3b2nZoGCN4rZRqE+a3kMF52tI= 2023-09-22T22:46:59.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/r3b2nZoGCN4rZRqE+a3kMF52tI=', 'Content-Length': '0'} 2023-09-22T22:46:59.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e0d08267af0c129-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj155 2023-09-22T22:46:59.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj155 2023-09-22T22:46:59.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj155 2023-09-22T22:46:59.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.222 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.222 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.222 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.222 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.223 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.223 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:46:59.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:46:59.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007622abdf63face86-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1551000falseobj1552023-09-22T22:05:15.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj155 2023-09-22T22:46:59.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj155 2023-09-22T22:46:59.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj155 2023-09-22T22:46:59.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.224 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.225 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.225 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.225 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.225 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.225 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM= 2023-09-22T22:46:59.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sJ93ctSYjH+U9st+jmU57Nu+XdM=', 'Content-Length': '0'} 2023-09-22T22:46:59.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae12186613b9ff39-00650e0fb1-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1551000falseobj1552023-09-22T22:05:15.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj155 2023-09-22T22:46:59.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj155 2023-09-22T22:46:59.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj155 2023-09-22T22:46:59.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.227 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.227 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.227 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.227 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.227 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj155 2023-09-22T22:46:59.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.227 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:GjtWMwyl58/OW+4yn5UHOt+TGaY= 2023-09-22T22:46:59.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GjtWMwyl58/OW+4yn5UHOt+TGaY=', 'Content-Length': '0'} 2023-09-22T22:46:59.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000626186fb584522b5-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj155 2023-09-22T22:46:59.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj155 2023-09-22T22:46:59.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj155 2023-09-22T22:46:59.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.229 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.229 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.229 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.229 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.230 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj155 2023-09-22T22:46:59.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.230 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:GjtWMwyl58/OW+4yn5UHOt+TGaY= 2023-09-22T22:46:59.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GjtWMwyl58/OW+4yn5UHOt+TGaY=', 'Content-Length': '0'} 2023-09-22T22:46:59.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bdf64349edbdda0-00650e0fb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.230 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj155 2023-09-22T22:46:59.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj155 2023-09-22T22:46:59.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj155 2023-09-22T22:46:59.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj155 2023-09-22T22:46:59.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.231 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.232 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.232 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.232 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:37 GMT 2023-09-22T22:46:59.232 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj155 2023-09-22T22:46:59.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.232 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:az/I6PpZgmMxAwxehRm7Q9ZcfvU= 2023-09-22T22:46:59.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:37 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:az/I6PpZgmMxAwxehRm7Q9ZcfvU=', 'Content-Length': '0'} 2023-09-22T22:46:59.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071c0e85a18aaf4f0-00650e0fb1-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj155 2023-09-22T22:46:59.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj155 2023-09-22T22:46:59.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj155 2023-09-22T22:46:59.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.234 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.234 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.234 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.234 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.234 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj155 2023-09-22T22:46:59.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.235 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:dZcYrQQQgYXtouD5NHQgD66YT14= 2023-09-22T22:46:59.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dZcYrQQQgYXtouD5NHQgD66YT14=', 'Content-Length': '0'} 2023-09-22T22:46:59.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f3df1447a5e8f77-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj155 2023-09-22T22:46:59.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj155 2023-09-22T22:46:59.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj155 2023-09-22T22:46:59.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.237 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.237 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.237 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.237 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.237 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.237 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:46:59.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:46:59.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000787f3eab71dba3ea-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1551000falseobj1552023-09-22T22:05:15.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj155 2023-09-22T22:46:59.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj155 2023-09-22T22:46:59.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj155 2023-09-22T22:46:59.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.239 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.239 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.239 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.239 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.239 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.240 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:46:59.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:46:59.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cad3fff41d453911-00650e0fb2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1551000falseobj1552023-09-22T22:05:15.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.240 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj17 2023-09-22T22:46:59.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj17 2023-09-22T22:46:59.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj17 2023-09-22T22:46:59.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.242 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.242 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.242 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.242 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.242 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.242 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:FRf/wY7RmkbHhpXzpcfDUiUdQQQ= 2023-09-22T22:46:59.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FRf/wY7RmkbHhpXzpcfDUiUdQQQ=', 'Content-Length': '0'} 2023-09-22T22:46:59.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa710e449efde951-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj17 2023-09-22T22:46:59.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj17 2023-09-22T22:46:59.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.246 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.246 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.246 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.246 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.247 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.247 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:FRf/wY7RmkbHhpXzpcfDUiUdQQQ= 2023-09-22T22:46:59.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FRf/wY7RmkbHhpXzpcfDUiUdQQQ=', 'Content-Length': '0'} 2023-09-22T22:46:59.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007961c31d9d1c856c-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj17 2023-09-22T22:46:59.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj17 2023-09-22T22:46:59.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.248 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.248 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.248 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.249 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.249 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.249 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:pUrmJMf7HZPsWZTeFGNt4v63WcY= 2023-09-22T22:46:59.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pUrmJMf7HZPsWZTeFGNt4v63WcY=', 'Content-Length': '0'} 2023-09-22T22:46:59.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090464193a7dfceaa-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj17 2023-09-22T22:46:59.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj17 2023-09-22T22:46:59.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.251 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.251 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.251 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.251 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.252 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.252 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:pUrmJMf7HZPsWZTeFGNt4v63WcY= 2023-09-22T22:46:59.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pUrmJMf7HZPsWZTeFGNt4v63WcY=', 'Content-Length': '0'} 2023-09-22T22:46:59.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008217709c9968f3b8-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.252 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj17 2023-09-22T22:46:59.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj17 2023-09-22T22:46:59.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj17 2023-09-22T22:46:59.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.253 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.254 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.254 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.254 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.254 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.254 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:FRf/wY7RmkbHhpXzpcfDUiUdQQQ= 2023-09-22T22:46:59.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FRf/wY7RmkbHhpXzpcfDUiUdQQQ=', 'Content-Length': '0'} 2023-09-22T22:46:59.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dbcb1a587756744-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj17 2023-09-22T22:46:59.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj17 2023-09-22T22:46:59.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.256 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.256 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.257 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.257 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.257 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj17?versionId=null 2023-09-22T22:46:59.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.257 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:FRf/wY7RmkbHhpXzpcfDUiUdQQQ= 2023-09-22T22:46:59.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FRf/wY7RmkbHhpXzpcfDUiUdQQQ=', 'Content-Length': '0'} 2023-09-22T22:46:59.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4afffda0acfedaa-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj17 2023-09-22T22:46:59.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj17 2023-09-22T22:46:59.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj17 2023-09-22T22:46:59.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.259 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.260 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.260 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.260 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.260 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.260 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:46:59.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:46:59.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000309628ae6239c313-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj171000falseobj172023-09-22T22:05:14.042Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj17 2023-09-22T22:46:59.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj17 2023-09-22T22:46:59.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj17 2023-09-22T22:46:59.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.262 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.262 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.262 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.262 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.263 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.263 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:46:59.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:46:59.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024391218acc26d08-00650e0fb2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj171000falseobj172023-09-22T22:05:14.042Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj17 2023-09-22T22:46:59.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj17 2023-09-22T22:46:59.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj17 2023-09-22T22:46:59.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.265 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.265 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.265 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.265 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.265 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj17 2023-09-22T22:46:59.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.265 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:JURMpqWay8zeev7BdMS8rF2HLj4= 2023-09-22T22:46:59.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JURMpqWay8zeev7BdMS8rF2HLj4=', 'Content-Length': '0'} 2023-09-22T22:46:59.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b3c8f8cbcf7b7e1-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj17 2023-09-22T22:46:59.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj17 2023-09-22T22:46:59.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj17 2023-09-22T22:46:59.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.267 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.268 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.268 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.268 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.268 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj17 2023-09-22T22:46:59.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.268 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:JURMpqWay8zeev7BdMS8rF2HLj4= 2023-09-22T22:46:59.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JURMpqWay8zeev7BdMS8rF2HLj4=', 'Content-Length': '0'} 2023-09-22T22:46:59.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7a153910268140c-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.268 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj17 2023-09-22T22:46:59.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj17 2023-09-22T22:46:59.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj17 2023-09-22T22:46:59.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj17 2023-09-22T22:46:59.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.270 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.270 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.270 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.270 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.270 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj17 2023-09-22T22:46:59.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.271 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:N4a+iMde9IqqVErDRW47vwIAn90= 2023-09-22T22:46:59.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:N4a+iMde9IqqVErDRW47vwIAn90=', 'Content-Length': '0'} 2023-09-22T22:46:59.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cf6c31b452be838-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj17 2023-09-22T22:46:59.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj17 2023-09-22T22:46:59.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj17 2023-09-22T22:46:59.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.273 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.273 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.273 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.273 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.273 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj17 2023-09-22T22:46:59.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.273 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:N4a+iMde9IqqVErDRW47vwIAn90= 2023-09-22T22:46:59.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:N4a+iMde9IqqVErDRW47vwIAn90=', 'Content-Length': '0'} 2023-09-22T22:46:59.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000374c37bb7c8dcab7-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj17 2023-09-22T22:46:59.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj17 2023-09-22T22:46:59.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj17 2023-09-22T22:46:59.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.275 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.275 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.276 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.276 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.276 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.276 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:46:59.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:46:59.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008482153ede7b7306-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj171000falseobj172023-09-22T22:05:14.042Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj17 2023-09-22T22:46:59.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj17 2023-09-22T22:46:59.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj17 2023-09-22T22:46:59.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.279 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.279 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.279 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.279 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.279 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.279 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:46:59.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:46:59.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051a9847759cacb67-00650e0fb2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj171000falseobj172023-09-22T22:05:14.042Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.280 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj185 2023-09-22T22:46:59.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj185 2023-09-22T22:46:59.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj185 2023-09-22T22:46:59.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.282 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.282 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.282 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.282 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.282 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.282 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cw136yPN3EsoInc2teF951FLeag= 2023-09-22T22:46:59.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cw136yPN3EsoInc2teF951FLeag=', 'Content-Length': '0'} 2023-09-22T22:46:59.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074e96ed73f3410f5-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj185 2023-09-22T22:46:59.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj185 2023-09-22T22:46:59.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.284 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.284 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.284 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.284 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.284 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.285 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cw136yPN3EsoInc2teF951FLeag= 2023-09-22T22:46:59.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cw136yPN3EsoInc2teF951FLeag=', 'Content-Length': '0'} 2023-09-22T22:46:59.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007619a835c48002fe-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj185 2023-09-22T22:46:59.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj185 2023-09-22T22:46:59.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.286 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.287 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.287 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.287 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.287 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.287 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:TgKshhcY7np0AHaA0O2nEx92YTk= 2023-09-22T22:46:59.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TgKshhcY7np0AHaA0O2nEx92YTk=', 'Content-Length': '0'} 2023-09-22T22:46:59.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015d4abed2374518c-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj185 2023-09-22T22:46:59.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj185 2023-09-22T22:46:59.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.290 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.290 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.290 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.290 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.290 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.291 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:TgKshhcY7np0AHaA0O2nEx92YTk= 2023-09-22T22:46:59.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TgKshhcY7np0AHaA0O2nEx92YTk=', 'Content-Length': '0'} 2023-09-22T22:46:59.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f45c400ed775fe1f-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.291 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj185 2023-09-22T22:46:59.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj185 2023-09-22T22:46:59.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj185 2023-09-22T22:46:59.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.293 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.293 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.293 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.293 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.293 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.294 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cw136yPN3EsoInc2teF951FLeag= 2023-09-22T22:46:59.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cw136yPN3EsoInc2teF951FLeag=', 'Content-Length': '0'} 2023-09-22T22:46:59.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af733d122f929e5b-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj185 2023-09-22T22:46:59.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj185 2023-09-22T22:46:59.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.296 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.296 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.296 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.296 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.296 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj185?versionId=null 2023-09-22T22:46:59.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.296 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cw136yPN3EsoInc2teF951FLeag= 2023-09-22T22:46:59.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cw136yPN3EsoInc2teF951FLeag=', 'Content-Length': '0'} 2023-09-22T22:46:59.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1b70c448a34635f-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj185 2023-09-22T22:46:59.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj185 2023-09-22T22:46:59.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj185 2023-09-22T22:46:59.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.298 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.298 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.299 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.299 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.299 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.299 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:46:59.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:46:59.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2ea3bdbb459f85e-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1851000falseobj1852023-09-22T22:05:16.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj185 2023-09-22T22:46:59.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj185 2023-09-22T22:46:59.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj185 2023-09-22T22:46:59.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.301 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.301 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.301 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.301 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.301 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.302 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:46:59.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:46:59.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fb4b6d083cae21f-00650e0fb2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1851000falseobj1852023-09-22T22:05:16.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj185 2023-09-22T22:46:59.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj185 2023-09-22T22:46:59.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj185 2023-09-22T22:46:59.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.303 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.303 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.303 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.303 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.303 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj185 2023-09-22T22:46:59.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.304 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:pMg3V+2j+fAJ1vliynRgm8oFlUs= 2023-09-22T22:46:59.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pMg3V+2j+fAJ1vliynRgm8oFlUs=', 'Content-Length': '0'} 2023-09-22T22:46:59.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006720f8912ebbbef0-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj185 2023-09-22T22:46:59.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj185 2023-09-22T22:46:59.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj185 2023-09-22T22:46:59.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.305 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.305 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.305 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.305 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.305 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj185 2023-09-22T22:46:59.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.305 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:pMg3V+2j+fAJ1vliynRgm8oFlUs= 2023-09-22T22:46:59.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pMg3V+2j+fAJ1vliynRgm8oFlUs=', 'Content-Length': '0'} 2023-09-22T22:46:59.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f615d8c8d04353a8-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.306 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj185 2023-09-22T22:46:59.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj185 2023-09-22T22:46:59.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj185 2023-09-22T22:46:59.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj185 2023-09-22T22:46:59.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.307 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.307 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.307 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.307 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.307 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj185 2023-09-22T22:46:59.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.308 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uLkr6f+CxMIS95Ctz0AKfhpOxOk= 2023-09-22T22:46:59.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uLkr6f+CxMIS95Ctz0AKfhpOxOk=', 'Content-Length': '0'} 2023-09-22T22:46:59.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5569765fa8460cd-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj185 2023-09-22T22:46:59.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj185 2023-09-22T22:46:59.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj185 2023-09-22T22:46:59.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.309 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.309 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.309 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.309 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.309 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj185 2023-09-22T22:46:59.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.309 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uLkr6f+CxMIS95Ctz0AKfhpOxOk= 2023-09-22T22:46:59.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uLkr6f+CxMIS95Ctz0AKfhpOxOk=', 'Content-Length': '0'} 2023-09-22T22:46:59.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047093f47440eeefb-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj185 2023-09-22T22:46:59.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj185 2023-09-22T22:46:59.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj185 2023-09-22T22:46:59.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.311 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.311 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.311 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.311 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.311 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.311 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:46:59.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:46:59.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac8a4e422609233f-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1851000falseobj1852023-09-22T22:05:16.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj185 2023-09-22T22:46:59.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj185 2023-09-22T22:46:59.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj185 2023-09-22T22:46:59.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.313 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.313 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.313 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.314 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.314 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.314 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:46:59.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:46:59.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087cd61095a07fe70-00650e0fb2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj1851000falseobj1852023-09-22T22:05:16.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.314 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj19 2023-09-22T22:46:59.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj19 2023-09-22T22:46:59.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj19 2023-09-22T22:46:59.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.316 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.316 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.316 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.316 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.316 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.316 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:YP7fbermt89LPn4/xK1q24DFRMs= 2023-09-22T22:46:59.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YP7fbermt89LPn4/xK1q24DFRMs=', 'Content-Length': '0'} 2023-09-22T22:46:59.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006c0d8d322b2ee91-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj19 2023-09-22T22:46:59.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj19 2023-09-22T22:46:59.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.318 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.318 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.318 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.318 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.318 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.319 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:YP7fbermt89LPn4/xK1q24DFRMs= 2023-09-22T22:46:59.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YP7fbermt89LPn4/xK1q24DFRMs=', 'Content-Length': '0'} 2023-09-22T22:46:59.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002aad02e3de7e30fa-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj19 2023-09-22T22:46:59.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj19 2023-09-22T22:46:59.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.320 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.321 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.321 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.321 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.321 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.321 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:4igrPJ24sXQIlAPRW8xDu1FZp6s= 2023-09-22T22:46:59.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4igrPJ24sXQIlAPRW8xDu1FZp6s=', 'Content-Length': '0'} 2023-09-22T22:46:59.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000477c565dca241f72-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj19 2023-09-22T22:46:59.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj19 2023-09-22T22:46:59.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.323 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.323 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.323 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.323 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.323 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.323 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:4igrPJ24sXQIlAPRW8xDu1FZp6s= 2023-09-22T22:46:59.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4igrPJ24sXQIlAPRW8xDu1FZp6s=', 'Content-Length': '0'} 2023-09-22T22:46:59.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002163748f8b65c586-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.324 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj19 2023-09-22T22:46:59.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj19 2023-09-22T22:46:59.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj19 2023-09-22T22:46:59.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.325 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.325 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.325 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.325 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.325 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.325 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:YP7fbermt89LPn4/xK1q24DFRMs= 2023-09-22T22:46:59.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YP7fbermt89LPn4/xK1q24DFRMs=', 'Content-Length': '0'} 2023-09-22T22:46:59.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068fb07adcd7fb493-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj19 2023-09-22T22:46:59.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj19 2023-09-22T22:46:59.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.327 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.327 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.327 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.327 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.327 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj19?versionId=null 2023-09-22T22:46:59.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.327 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:YP7fbermt89LPn4/xK1q24DFRMs= 2023-09-22T22:46:59.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YP7fbermt89LPn4/xK1q24DFRMs=', 'Content-Length': '0'} 2023-09-22T22:46:59.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d49be1a739a4bea-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj19 2023-09-22T22:46:59.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj19 2023-09-22T22:46:59.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj19 2023-09-22T22:46:59.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.329 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.329 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.329 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.329 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.329 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.329 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:46:59.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:46:59.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8d0a53470dd0d8f-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj191000falseobj192023-09-22T22:05:14.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj19 2023-09-22T22:46:59.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj19 2023-09-22T22:46:59.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj19 2023-09-22T22:46:59.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.331 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.331 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.331 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.331 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.331 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.331 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:46:59.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:46:59.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000877baa40f1e0d04d-00650e0fb2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj191000falseobj192023-09-22T22:05:14.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj19 2023-09-22T22:46:59.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj19 2023-09-22T22:46:59.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj19 2023-09-22T22:46:59.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.332 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.333 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.333 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.333 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.333 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj19 2023-09-22T22:46:59.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.333 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:jsJlU8zOjTlocXj8qntbg0oX9hk= 2023-09-22T22:46:59.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jsJlU8zOjTlocXj8qntbg0oX9hk=', 'Content-Length': '0'} 2023-09-22T22:46:59.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a1630da456aff9a-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj19 2023-09-22T22:46:59.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj19 2023-09-22T22:46:59.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj19 2023-09-22T22:46:59.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.334 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.334 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.334 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.335 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.335 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj19 2023-09-22T22:46:59.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.335 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:jsJlU8zOjTlocXj8qntbg0oX9hk= 2023-09-22T22:46:59.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jsJlU8zOjTlocXj8qntbg0oX9hk=', 'Content-Length': '0'} 2023-09-22T22:46:59.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007060795459fb0b70-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.335 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj19 2023-09-22T22:46:59.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj19 2023-09-22T22:46:59.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj19 2023-09-22T22:46:59.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj19 2023-09-22T22:46:59.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.336 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.336 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.336 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.336 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.336 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj19 2023-09-22T22:46:59.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.337 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:U3a7Gj49WDb/5DJDVH+bLe1jSqs= 2023-09-22T22:46:59.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:U3a7Gj49WDb/5DJDVH+bLe1jSqs=', 'Content-Length': '0'} 2023-09-22T22:46:59.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005337e483281872d2-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj19 2023-09-22T22:46:59.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj19 2023-09-22T22:46:59.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj19 2023-09-22T22:46:59.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.338 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.338 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.338 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.338 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.338 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj19 2023-09-22T22:46:59.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.338 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:U3a7Gj49WDb/5DJDVH+bLe1jSqs= 2023-09-22T22:46:59.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:U3a7Gj49WDb/5DJDVH+bLe1jSqs=', 'Content-Length': '0'} 2023-09-22T22:46:59.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000192d55f07a15447e-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj19 2023-09-22T22:46:59.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj19 2023-09-22T22:46:59.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj19 2023-09-22T22:46:59.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.340 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.340 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.340 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.340 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.340 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.340 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:46:59.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:46:59.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f0879996eae6d01-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj191000falseobj192023-09-22T22:05:14.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj19 2023-09-22T22:46:59.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj19 2023-09-22T22:46:59.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj19 2023-09-22T22:46:59.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.342 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.342 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.342 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.342 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.342 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.342 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:46:59.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:46:59.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080973d1f9001190c-00650e0fb2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj191000falseobj192023-09-22T22:05:14.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.343 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-09-22T22:46:59.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj22 2023-09-22T22:46:59.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj22 2023-09-22T22:46:59.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.344 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.344 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.344 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.344 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.344 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.344 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uKUdqjUmyu3eKy806idSG+PkgVg= 2023-09-22T22:46:59.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKUdqjUmyu3eKy806idSG+PkgVg=', 'Content-Length': '0'} 2023-09-22T22:46:59.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005494c9e69654ad7b-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj22 2023-09-22T22:46:59.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj22 2023-09-22T22:46:59.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.346 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.346 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.346 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.346 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.346 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.346 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uKUdqjUmyu3eKy806idSG+PkgVg= 2023-09-22T22:46:59.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKUdqjUmyu3eKy806idSG+PkgVg=', 'Content-Length': '0'} 2023-09-22T22:46:59.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd697dbffbdb0d19-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj22 2023-09-22T22:46:59.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj22 2023-09-22T22:46:59.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.347 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.348 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.348 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.348 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.348 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.348 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:5pjWoqz+Io6DTZ0ox/6cWXkKUl8= 2023-09-22T22:46:59.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5pjWoqz+Io6DTZ0ox/6cWXkKUl8=', 'Content-Length': '0'} 2023-09-22T22:46:59.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ec95bff7049be2d-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj22 2023-09-22T22:46:59.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj22 2023-09-22T22:46:59.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.349 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.349 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.349 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.350 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.350 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.350 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:5pjWoqz+Io6DTZ0ox/6cWXkKUl8= 2023-09-22T22:46:59.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:5pjWoqz+Io6DTZ0ox/6cWXkKUl8=', 'Content-Length': '0'} 2023-09-22T22:46:59.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b45d0d6140c188e1-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.350 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-09-22T22:46:59.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj22 2023-09-22T22:46:59.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj22 2023-09-22T22:46:59.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.351 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.351 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.351 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.352 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.352 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.352 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uKUdqjUmyu3eKy806idSG+PkgVg= 2023-09-22T22:46:59.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKUdqjUmyu3eKy806idSG+PkgVg=', 'Content-Length': '0'} 2023-09-22T22:46:59.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000453b478858be92eb-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj22 2023-09-22T22:46:59.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj22 2023-09-22T22:46:59.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.353 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.353 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.353 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.353 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.353 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj22?versionId=null 2023-09-22T22:46:59.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.354 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uKUdqjUmyu3eKy806idSG+PkgVg= 2023-09-22T22:46:59.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uKUdqjUmyu3eKy806idSG+PkgVg=', 'Content-Length': '0'} 2023-09-22T22:46:59.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c77a6ee1502bbb4-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj22 2023-09-22T22:46:59.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj22 2023-09-22T22:46:59.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj22 2023-09-22T22:46:59.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.355 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.355 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.355 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.355 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.355 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.355 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:46:59.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:46:59.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005da0e44ce556cbe4-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj221000falseobj222023-09-22T22:05:13.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj22 2023-09-22T22:46:59.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj22 2023-09-22T22:46:59.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj22 2023-09-22T22:46:59.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.357 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.357 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.357 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.357 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.357 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.357 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:46:59.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:46:59.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029b9e6cfb0d99493-00650e0fb2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj221000falseobj222023-09-22T22:05:13.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj22 2023-09-22T22:46:59.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj22 2023-09-22T22:46:59.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj22 2023-09-22T22:46:59.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.359 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.359 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.359 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.359 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.359 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj22 2023-09-22T22:46:59.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.359 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:6/8iwinmk9X8X5xu2R82CFUr+ao= 2023-09-22T22:46:59.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6/8iwinmk9X8X5xu2R82CFUr+ao=', 'Content-Length': '0'} 2023-09-22T22:46:59.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be68dcc1b349e694-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj22 2023-09-22T22:46:59.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj22 2023-09-22T22:46:59.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj22 2023-09-22T22:46:59.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.360 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.360 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.360 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.360 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.361 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj22 2023-09-22T22:46:59.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.361 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:6/8iwinmk9X8X5xu2R82CFUr+ao= 2023-09-22T22:46:59.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6/8iwinmk9X8X5xu2R82CFUr+ao=', 'Content-Length': '0'} 2023-09-22T22:46:59.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6140c99c441ef01-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.361 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-09-22T22:46:59.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj22 2023-09-22T22:46:59.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj22 2023-09-22T22:46:59.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj22 2023-09-22T22:46:59.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.362 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.362 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.362 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.362 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.362 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj22 2023-09-22T22:46:59.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.362 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:UFltGioGQCl7b08erZxcTqoxwY0= 2023-09-22T22:46:59.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UFltGioGQCl7b08erZxcTqoxwY0=', 'Content-Length': '0'} 2023-09-22T22:46:59.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae9bc5452efc153c-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj22 2023-09-22T22:46:59.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj22 2023-09-22T22:46:59.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj22 2023-09-22T22:46:59.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.364 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.364 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.364 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.364 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.364 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj22 2023-09-22T22:46:59.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.364 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:UFltGioGQCl7b08erZxcTqoxwY0= 2023-09-22T22:46:59.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UFltGioGQCl7b08erZxcTqoxwY0=', 'Content-Length': '0'} 2023-09-22T22:46:59.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8e1f06e0f5b9b38-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj22 2023-09-22T22:46:59.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj22 2023-09-22T22:46:59.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj22 2023-09-22T22:46:59.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.365 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.366 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.366 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.366 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.366 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.366 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:46:59.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:46:59.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087068558de212b1f-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj221000falseobj222023-09-22T22:05:13.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj22 2023-09-22T22:46:59.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj22 2023-09-22T22:46:59.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj22 2023-09-22T22:46:59.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.367 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.367 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.368 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.368 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.368 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.368 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:46:59.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:46:59.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c23f3d6d02b9e03e-00650e0fb2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj221000falseobj222023-09-22T22:05:13.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.368 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj23 2023-09-22T22:46:59.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj23 2023-09-22T22:46:59.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj23 2023-09-22T22:46:59.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.369 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.369 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.370 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.370 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.370 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.370 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oxli9BcBB93LqGHCI14abQMgJDw= 2023-09-22T22:46:59.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oxli9BcBB93LqGHCI14abQMgJDw=', 'Content-Length': '0'} 2023-09-22T22:46:59.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000359e1ff76a119e56-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj23 2023-09-22T22:46:59.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj23 2023-09-22T22:46:59.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.371 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.371 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.371 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.371 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.372 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.372 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oxli9BcBB93LqGHCI14abQMgJDw= 2023-09-22T22:46:59.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oxli9BcBB93LqGHCI14abQMgJDw=', 'Content-Length': '0'} 2023-09-22T22:46:59.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb21d030c552c2d6-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj23 2023-09-22T22:46:59.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj23 2023-09-22T22:46:59.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.373 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.373 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.373 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.373 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.373 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.374 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Bpitv75Q5PWZCZdaBSdJqnF3f2g= 2023-09-22T22:46:59.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Bpitv75Q5PWZCZdaBSdJqnF3f2g=', 'Content-Length': '0'} 2023-09-22T22:46:59.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5249dd1995a9a0d-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj23 2023-09-22T22:46:59.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj23 2023-09-22T22:46:59.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.375 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.375 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.375 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.375 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.375 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.376 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Bpitv75Q5PWZCZdaBSdJqnF3f2g= 2023-09-22T22:46:59.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Bpitv75Q5PWZCZdaBSdJqnF3f2g=', 'Content-Length': '0'} 2023-09-22T22:46:59.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077403d8f5813fc8b-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.376 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj23 2023-09-22T22:46:59.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj23 2023-09-22T22:46:59.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj23 2023-09-22T22:46:59.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.377 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.377 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.377 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.377 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.377 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.378 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oxli9BcBB93LqGHCI14abQMgJDw= 2023-09-22T22:46:59.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oxli9BcBB93LqGHCI14abQMgJDw=', 'Content-Length': '0'} 2023-09-22T22:46:59.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2864726713c5b00-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj23 2023-09-22T22:46:59.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj23 2023-09-22T22:46:59.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.379 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.379 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.379 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.379 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.379 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj23?versionId=null 2023-09-22T22:46:59.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.379 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oxli9BcBB93LqGHCI14abQMgJDw= 2023-09-22T22:46:59.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oxli9BcBB93LqGHCI14abQMgJDw=', 'Content-Length': '0'} 2023-09-22T22:46:59.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee26cfcbb390f9bd-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj23 2023-09-22T22:46:59.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj23 2023-09-22T22:46:59.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj23 2023-09-22T22:46:59.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.381 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.381 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.381 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.381 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.381 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.381 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:46:59.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:46:59.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079a090f9529b599f-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj231000falseobj232023-09-22T22:05:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj23 2023-09-22T22:46:59.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj23 2023-09-22T22:46:59.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj23 2023-09-22T22:46:59.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.383 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.383 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.383 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.383 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.383 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.383 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:46:59.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:46:59.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000575672fafbd50dbc-00650e0fb2-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj231000falseobj232023-09-22T22:05:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj23 2023-09-22T22:46:59.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj23 2023-09-22T22:46:59.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj23 2023-09-22T22:46:59.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.385 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.385 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.385 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.385 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.385 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj23 2023-09-22T22:46:59.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.385 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Iq2aVAou8qHmjEvrP9XuDx3pDuw= 2023-09-22T22:46:59.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Iq2aVAou8qHmjEvrP9XuDx3pDuw=', 'Content-Length': '0'} 2023-09-22T22:46:59.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076b3176df4c3d1cb-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj23 2023-09-22T22:46:59.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj23 2023-09-22T22:46:59.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj23 2023-09-22T22:46:59.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.386 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.387 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.387 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.387 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.387 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj23 2023-09-22T22:46:59.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.387 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Iq2aVAou8qHmjEvrP9XuDx3pDuw= 2023-09-22T22:46:59.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Iq2aVAou8qHmjEvrP9XuDx3pDuw=', 'Content-Length': '0'} 2023-09-22T22:46:59.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b67265d6c7313d7b-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.387 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj23 2023-09-22T22:46:59.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj23 2023-09-22T22:46:59.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj23 2023-09-22T22:46:59.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj23 2023-09-22T22:46:59.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.388 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.388 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.388 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.388 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.389 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj23 2023-09-22T22:46:59.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.389 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:aVkXjSK7EvlH4eOfLgIiMk8Qo6k= 2023-09-22T22:46:59.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aVkXjSK7EvlH4eOfLgIiMk8Qo6k=', 'Content-Length': '0'} 2023-09-22T22:46:59.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b320d99b16dbfcbb-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj23 2023-09-22T22:46:59.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj23 2023-09-22T22:46:59.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj23 2023-09-22T22:46:59.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.390 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.390 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.390 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.390 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.390 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj23 2023-09-22T22:46:59.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.390 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:aVkXjSK7EvlH4eOfLgIiMk8Qo6k= 2023-09-22T22:46:59.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aVkXjSK7EvlH4eOfLgIiMk8Qo6k=', 'Content-Length': '0'} 2023-09-22T22:46:59.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4bb4c4af63124d7-00650e0fb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj23 2023-09-22T22:46:59.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj23 2023-09-22T22:46:59.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj23 2023-09-22T22:46:59.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.392 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.392 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.392 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.392 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:38 GMT 2023-09-22T22:46:59.392 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.392 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M= 2023-09-22T22:46:59.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:38 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kbX1dpTs2Qp78irng71QVtdkG5M=', 'Content-Length': '0'} 2023-09-22T22:46:59.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000592bbdb4a8b9d762-00650e0fb2-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj231000falseobj232023-09-22T22:05:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj23 2023-09-22T22:46:59.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj23 2023-09-22T22:46:59.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj23 2023-09-22T22:46:59.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.394 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.394 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.394 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.394 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.394 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.394 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:46:59.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:46:59.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042b120036a9614de-00650e0fb3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj231000falseobj232023-09-22T22:05:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.395 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj29 2023-09-22T22:46:59.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj29 2023-09-22T22:46:59.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj29 2023-09-22T22:46:59.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.396 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.396 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.396 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.396 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.396 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.396 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:AwMeS/DjBkFg9H83verViX3M0XA= 2023-09-22T22:46:59.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AwMeS/DjBkFg9H83verViX3M0XA=', 'Content-Length': '0'} 2023-09-22T22:46:59.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff3f40fc268dd671-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj29 2023-09-22T22:46:59.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj29 2023-09-22T22:46:59.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.398 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.398 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.398 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.398 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.398 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.398 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:AwMeS/DjBkFg9H83verViX3M0XA= 2023-09-22T22:46:59.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AwMeS/DjBkFg9H83verViX3M0XA=', 'Content-Length': '0'} 2023-09-22T22:46:59.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b24d6ac3281b773-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj29 2023-09-22T22:46:59.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj29 2023-09-22T22:46:59.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.400 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.400 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.400 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.400 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.400 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.400 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:3o8gKiOp6CwJw3znoVfqAQsZ+fg= 2023-09-22T22:46:59.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3o8gKiOp6CwJw3znoVfqAQsZ+fg=', 'Content-Length': '0'} 2023-09-22T22:46:59.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068b9912fbaf0f9e4-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj29 2023-09-22T22:46:59.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj29 2023-09-22T22:46:59.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.401 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.402 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.402 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.402 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.402 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.402 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:3o8gKiOp6CwJw3znoVfqAQsZ+fg= 2023-09-22T22:46:59.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3o8gKiOp6CwJw3znoVfqAQsZ+fg=', 'Content-Length': '0'} 2023-09-22T22:46:59.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a33ee75060918ef-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.402 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj29 2023-09-22T22:46:59.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj29 2023-09-22T22:46:59.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj29 2023-09-22T22:46:59.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.403 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.404 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.404 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.404 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.404 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.404 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:AwMeS/DjBkFg9H83verViX3M0XA= 2023-09-22T22:46:59.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AwMeS/DjBkFg9H83verViX3M0XA=', 'Content-Length': '0'} 2023-09-22T22:46:59.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003398f0422d27f14c-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj29 2023-09-22T22:46:59.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj29 2023-09-22T22:46:59.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.405 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.405 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.405 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.406 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.406 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj29?versionId=null 2023-09-22T22:46:59.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.406 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:AwMeS/DjBkFg9H83verViX3M0XA= 2023-09-22T22:46:59.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AwMeS/DjBkFg9H83verViX3M0XA=', 'Content-Length': '0'} 2023-09-22T22:46:59.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af4463cca8d93f94-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj29 2023-09-22T22:46:59.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj29 2023-09-22T22:46:59.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj29 2023-09-22T22:46:59.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.407 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.407 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.407 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.408 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.408 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.408 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:46:59.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:46:59.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088b74ae85052b6d6-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj291000falseobj292023-09-22T22:05:15.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj29 2023-09-22T22:46:59.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj29 2023-09-22T22:46:59.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj29 2023-09-22T22:46:59.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.409 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.409 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.410 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.410 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.410 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.410 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:46:59.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:46:59.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e7d73912268547d-00650e0fb3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj291000falseobj292023-09-22T22:05:15.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj29 2023-09-22T22:46:59.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj29 2023-09-22T22:46:59.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj29 2023-09-22T22:46:59.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.411 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.411 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.412 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.412 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.412 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj29 2023-09-22T22:46:59.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.412 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:JWub2c7eUqWLLfjqOnwpDNVJuSc= 2023-09-22T22:46:59.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JWub2c7eUqWLLfjqOnwpDNVJuSc=', 'Content-Length': '0'} 2023-09-22T22:46:59.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f40a6caf60a505b0-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj29 2023-09-22T22:46:59.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj29 2023-09-22T22:46:59.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj29 2023-09-22T22:46:59.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.413 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.413 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.414 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.414 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.414 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj29 2023-09-22T22:46:59.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.414 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:JWub2c7eUqWLLfjqOnwpDNVJuSc= 2023-09-22T22:46:59.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:JWub2c7eUqWLLfjqOnwpDNVJuSc=', 'Content-Length': '0'} 2023-09-22T22:46:59.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000516a5ee4774fb5a8-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.414 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj29 2023-09-22T22:46:59.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj29 2023-09-22T22:46:59.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj29 2023-09-22T22:46:59.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj29 2023-09-22T22:46:59.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.415 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.415 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.415 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.416 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.416 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj29 2023-09-22T22:46:59.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.416 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:DWJz/hx/zSxNkzLJ0MKYyBWTgkA= 2023-09-22T22:46:59.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DWJz/hx/zSxNkzLJ0MKYyBWTgkA=', 'Content-Length': '0'} 2023-09-22T22:46:59.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008668e39986cbac48-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj29 2023-09-22T22:46:59.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj29 2023-09-22T22:46:59.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj29 2023-09-22T22:46:59.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.417 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.417 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.417 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.417 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.417 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj29 2023-09-22T22:46:59.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.418 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:DWJz/hx/zSxNkzLJ0MKYyBWTgkA= 2023-09-22T22:46:59.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DWJz/hx/zSxNkzLJ0MKYyBWTgkA=', 'Content-Length': '0'} 2023-09-22T22:46:59.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2754a5ddaf90189-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj29 2023-09-22T22:46:59.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj29 2023-09-22T22:46:59.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj29 2023-09-22T22:46:59.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.419 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.419 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.419 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.420 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.420 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.420 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:46:59.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:46:59.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd520a49b4fda6c6-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj291000falseobj292023-09-22T22:05:15.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj29 2023-09-22T22:46:59.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj29 2023-09-22T22:46:59.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj29 2023-09-22T22:46:59.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.421 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.422 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.422 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.422 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.422 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.422 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:46:59.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:46:59.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008064abf1a118978c-00650e0fb3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj291000falseobj292023-09-22T22:05:15.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.423 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj31 2023-09-22T22:46:59.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj31 2023-09-22T22:46:59.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj31 2023-09-22T22:46:59.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.424 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.424 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.424 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.424 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.424 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.425 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:HDwtIZ5FCc+AzD4PeMoL/WzVD1c= 2023-09-22T22:46:59.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HDwtIZ5FCc+AzD4PeMoL/WzVD1c=', 'Content-Length': '0'} 2023-09-22T22:46:59.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f96eecc85d3edd72-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj31 2023-09-22T22:46:59.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj31 2023-09-22T22:46:59.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.426 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.426 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.426 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.427 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.427 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.427 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:HDwtIZ5FCc+AzD4PeMoL/WzVD1c= 2023-09-22T22:46:59.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HDwtIZ5FCc+AzD4PeMoL/WzVD1c=', 'Content-Length': '0'} 2023-09-22T22:46:59.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8a42d40429f8eab-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj31 2023-09-22T22:46:59.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj31 2023-09-22T22:46:59.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.428 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.428 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.429 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.429 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.429 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.429 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:DkC61oRYFxQorw6xprFQppMYyMU= 2023-09-22T22:46:59.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DkC61oRYFxQorw6xprFQppMYyMU=', 'Content-Length': '0'} 2023-09-22T22:46:59.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002817a9ecb2cbd048-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj31 2023-09-22T22:46:59.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj31 2023-09-22T22:46:59.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.431 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.431 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.431 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.431 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.431 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.431 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:DkC61oRYFxQorw6xprFQppMYyMU= 2023-09-22T22:46:59.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DkC61oRYFxQorw6xprFQppMYyMU=', 'Content-Length': '0'} 2023-09-22T22:46:59.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bebf9f7d320423a0-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.431 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj31 2023-09-22T22:46:59.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj31 2023-09-22T22:46:59.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj31 2023-09-22T22:46:59.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.433 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.433 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.433 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.433 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.433 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.433 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:HDwtIZ5FCc+AzD4PeMoL/WzVD1c= 2023-09-22T22:46:59.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HDwtIZ5FCc+AzD4PeMoL/WzVD1c=', 'Content-Length': '0'} 2023-09-22T22:46:59.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097138f34adf1ead4-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj31 2023-09-22T22:46:59.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj31 2023-09-22T22:46:59.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.435 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.435 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.435 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.435 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.436 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj31?versionId=null 2023-09-22T22:46:59.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.436 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:HDwtIZ5FCc+AzD4PeMoL/WzVD1c= 2023-09-22T22:46:59.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HDwtIZ5FCc+AzD4PeMoL/WzVD1c=', 'Content-Length': '0'} 2023-09-22T22:46:59.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d827460594327c4b-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj31 2023-09-22T22:46:59.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj31 2023-09-22T22:46:59.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj31 2023-09-22T22:46:59.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.437 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.437 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.438 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.438 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.438 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.438 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:46:59.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:46:59.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf0494438f3478c8-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj311000falseobj312023-09-22T22:05:15.503Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj31 2023-09-22T22:46:59.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj31 2023-09-22T22:46:59.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj31 2023-09-22T22:46:59.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.440 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.440 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.440 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.440 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.440 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.440 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:46:59.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:46:59.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d912145cffbd6ce-00650e0fb3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj311000falseobj312023-09-22T22:05:15.503Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj31 2023-09-22T22:46:59.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj31 2023-09-22T22:46:59.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj31 2023-09-22T22:46:59.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.442 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.442 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.442 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.442 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.442 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj31 2023-09-22T22:46:59.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.442 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:LIpvjZrJylbKg/l1We+79JQ7i2g= 2023-09-22T22:46:59.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LIpvjZrJylbKg/l1We+79JQ7i2g=', 'Content-Length': '0'} 2023-09-22T22:46:59.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e17a3e2abb29713-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj31 2023-09-22T22:46:59.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj31 2023-09-22T22:46:59.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj31 2023-09-22T22:46:59.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.444 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.444 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.444 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.444 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.444 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj31 2023-09-22T22:46:59.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.444 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:LIpvjZrJylbKg/l1We+79JQ7i2g= 2023-09-22T22:46:59.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LIpvjZrJylbKg/l1We+79JQ7i2g=', 'Content-Length': '0'} 2023-09-22T22:46:59.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032f025391c77a505-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.444 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj31 2023-09-22T22:46:59.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj31 2023-09-22T22:46:59.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj31 2023-09-22T22:46:59.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj31 2023-09-22T22:46:59.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.445 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.446 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.446 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.446 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.446 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj31 2023-09-22T22:46:59.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.446 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:UtVR3iphNz55cxcYasbdnV4SvQg= 2023-09-22T22:46:59.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UtVR3iphNz55cxcYasbdnV4SvQg=', 'Content-Length': '0'} 2023-09-22T22:46:59.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d80fc0aa04281d40-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj31 2023-09-22T22:46:59.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj31 2023-09-22T22:46:59.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj31 2023-09-22T22:46:59.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.447 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.447 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.447 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.447 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.448 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj31 2023-09-22T22:46:59.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.448 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:UtVR3iphNz55cxcYasbdnV4SvQg= 2023-09-22T22:46:59.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UtVR3iphNz55cxcYasbdnV4SvQg=', 'Content-Length': '0'} 2023-09-22T22:46:59.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055f982dd88188b4f-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj31 2023-09-22T22:46:59.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj31 2023-09-22T22:46:59.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj31 2023-09-22T22:46:59.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.449 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.449 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.449 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.449 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.450 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.450 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:46:59.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:46:59.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a7c062ebf0443d0-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj311000falseobj312023-09-22T22:05:15.503Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj31 2023-09-22T22:46:59.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj31 2023-09-22T22:46:59.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj31 2023-09-22T22:46:59.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.451 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.451 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.451 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.451 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.451 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.452 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:46:59.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:46:59.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d33d447e69c5f13b-00650e0fb3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj311000falseobj312023-09-22T22:05:15.503Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.452 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-09-22T22:46:59.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj33 2023-09-22T22:46:59.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj33 2023-09-22T22:46:59.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.453 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.453 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.453 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.453 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.454 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.454 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:x29jAnhEnsxmC8n6NBw2mHN3zbk= 2023-09-22T22:46:59.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x29jAnhEnsxmC8n6NBw2mHN3zbk=', 'Content-Length': '0'} 2023-09-22T22:46:59.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7e326dd848ff20c-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj33 2023-09-22T22:46:59.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj33 2023-09-22T22:46:59.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.455 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.455 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.455 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.455 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.456 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.456 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:x29jAnhEnsxmC8n6NBw2mHN3zbk= 2023-09-22T22:46:59.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x29jAnhEnsxmC8n6NBw2mHN3zbk=', 'Content-Length': '0'} 2023-09-22T22:46:59.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e82fe82121a5bcd6-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj33 2023-09-22T22:46:59.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj33 2023-09-22T22:46:59.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.457 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.457 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.457 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.457 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.457 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.457 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:6I8SE0rvDP48Lm246yzuZ8INmgA= 2023-09-22T22:46:59.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6I8SE0rvDP48Lm246yzuZ8INmgA=', 'Content-Length': '0'} 2023-09-22T22:46:59.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000833fca8f951a4e26-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj33 2023-09-22T22:46:59.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj33 2023-09-22T22:46:59.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.459 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.459 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.459 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.459 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.459 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.459 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:6I8SE0rvDP48Lm246yzuZ8INmgA= 2023-09-22T22:46:59.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6I8SE0rvDP48Lm246yzuZ8INmgA=', 'Content-Length': '0'} 2023-09-22T22:46:59.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df4a68159ec8cde3-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.460 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-09-22T22:46:59.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj33 2023-09-22T22:46:59.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj33 2023-09-22T22:46:59.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.461 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.461 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.461 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.461 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.461 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.461 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:x29jAnhEnsxmC8n6NBw2mHN3zbk= 2023-09-22T22:46:59.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x29jAnhEnsxmC8n6NBw2mHN3zbk=', 'Content-Length': '0'} 2023-09-22T22:46:59.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fad5a05dd2d79e0-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj33 2023-09-22T22:46:59.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj33 2023-09-22T22:46:59.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.463 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.463 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.463 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.463 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.463 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj33?versionId=null 2023-09-22T22:46:59.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.463 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:x29jAnhEnsxmC8n6NBw2mHN3zbk= 2023-09-22T22:46:59.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:x29jAnhEnsxmC8n6NBw2mHN3zbk=', 'Content-Length': '0'} 2023-09-22T22:46:59.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaa3f517b9d3a21b-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj33 2023-09-22T22:46:59.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj33 2023-09-22T22:46:59.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj33 2023-09-22T22:46:59.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.465 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.465 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.465 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.465 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.465 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.465 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:46:59.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:46:59.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002baac6a38303c5d9-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj331000falseobj332023-09-22T22:05:13.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj33 2023-09-22T22:46:59.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj33 2023-09-22T22:46:59.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj33 2023-09-22T22:46:59.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.467 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.467 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.467 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.467 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.467 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.467 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:46:59.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:46:59.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017b7cf8b6a56bacd-00650e0fb3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj331000falseobj332023-09-22T22:05:13.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj33 2023-09-22T22:46:59.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj33 2023-09-22T22:46:59.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj33 2023-09-22T22:46:59.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.468 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.468 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.469 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.469 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.469 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj33 2023-09-22T22:46:59.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.469 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:N/IVs3J91C0KhLuGK4kYR6utcfU= 2023-09-22T22:46:59.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:N/IVs3J91C0KhLuGK4kYR6utcfU=', 'Content-Length': '0'} 2023-09-22T22:46:59.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019884ccea0b47cad-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj33 2023-09-22T22:46:59.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj33 2023-09-22T22:46:59.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj33 2023-09-22T22:46:59.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.470 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.470 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.470 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.470 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.470 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj33 2023-09-22T22:46:59.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.471 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:N/IVs3J91C0KhLuGK4kYR6utcfU= 2023-09-22T22:46:59.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:N/IVs3J91C0KhLuGK4kYR6utcfU=', 'Content-Length': '0'} 2023-09-22T22:46:59.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc1ada81bd96cfc7-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.471 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-09-22T22:46:59.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj33 2023-09-22T22:46:59.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj33 2023-09-22T22:46:59.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj33 2023-09-22T22:46:59.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.472 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.472 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.472 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.472 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.472 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj33 2023-09-22T22:46:59.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.472 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:+uGjrCnLehXGrMFsWcxXMAknNC8= 2023-09-22T22:46:59.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+uGjrCnLehXGrMFsWcxXMAknNC8=', 'Content-Length': '0'} 2023-09-22T22:46:59.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f1e002538c05171-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj33 2023-09-22T22:46:59.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj33 2023-09-22T22:46:59.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj33 2023-09-22T22:46:59.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.474 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.474 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.474 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.474 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.474 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj33 2023-09-22T22:46:59.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.474 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:+uGjrCnLehXGrMFsWcxXMAknNC8= 2023-09-22T22:46:59.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:+uGjrCnLehXGrMFsWcxXMAknNC8=', 'Content-Length': '0'} 2023-09-22T22:46:59.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d225002f2dcd82a0-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj33 2023-09-22T22:46:59.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj33 2023-09-22T22:46:59.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj33 2023-09-22T22:46:59.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.476 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.476 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.476 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.476 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.476 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.476 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:46:59.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:46:59.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096bfcbebb8cd1c64-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj331000falseobj332023-09-22T22:05:13.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj33 2023-09-22T22:46:59.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj33 2023-09-22T22:46:59.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj33 2023-09-22T22:46:59.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.478 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.478 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.478 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.478 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.478 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.478 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:46:59.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:46:59.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d7117af21d17025-00650e0fb3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj331000falseobj332023-09-22T22:05:13.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.479 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj34 2023-09-22T22:46:59.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj34 2023-09-22T22:46:59.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj34 2023-09-22T22:46:59.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.480 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.480 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.480 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.480 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.480 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.480 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:akqqJFTLpznzrfYm6gSxDkWWKmM= 2023-09-22T22:46:59.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:akqqJFTLpznzrfYm6gSxDkWWKmM=', 'Content-Length': '0'} 2023-09-22T22:46:59.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001170bd4d3b75b471-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj34 2023-09-22T22:46:59.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj34 2023-09-22T22:46:59.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.482 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.482 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.482 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.482 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.482 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.482 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:akqqJFTLpznzrfYm6gSxDkWWKmM= 2023-09-22T22:46:59.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:akqqJFTLpznzrfYm6gSxDkWWKmM=', 'Content-Length': '0'} 2023-09-22T22:46:59.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059ff2cac97a09245-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj34 2023-09-22T22:46:59.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj34 2023-09-22T22:46:59.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.484 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.484 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.484 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.484 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.484 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.484 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:j2cm0uxtNtIwGktDoULu3X6YM34= 2023-09-22T22:46:59.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:j2cm0uxtNtIwGktDoULu3X6YM34=', 'Content-Length': '0'} 2023-09-22T22:46:59.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f6fa505f7b990b1-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj34 2023-09-22T22:46:59.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj34 2023-09-22T22:46:59.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.486 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.486 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.486 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.486 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.486 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.486 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:j2cm0uxtNtIwGktDoULu3X6YM34= 2023-09-22T22:46:59.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:j2cm0uxtNtIwGktDoULu3X6YM34=', 'Content-Length': '0'} 2023-09-22T22:46:59.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb488d1e2332e825-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.487 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj34 2023-09-22T22:46:59.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj34 2023-09-22T22:46:59.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj34 2023-09-22T22:46:59.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.488 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.488 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.488 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.488 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.488 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.488 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:akqqJFTLpznzrfYm6gSxDkWWKmM= 2023-09-22T22:46:59.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:akqqJFTLpznzrfYm6gSxDkWWKmM=', 'Content-Length': '0'} 2023-09-22T22:46:59.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000991cc56ffb5bcaf8-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj34 2023-09-22T22:46:59.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj34 2023-09-22T22:46:59.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.490 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.490 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.490 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.490 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.490 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj34?versionId=null 2023-09-22T22:46:59.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.490 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:akqqJFTLpznzrfYm6gSxDkWWKmM= 2023-09-22T22:46:59.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:akqqJFTLpznzrfYm6gSxDkWWKmM=', 'Content-Length': '0'} 2023-09-22T22:46:59.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df55a959e46f7796-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj34 2023-09-22T22:46:59.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj34 2023-09-22T22:46:59.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj34 2023-09-22T22:46:59.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.492 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.492 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.492 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.492 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.492 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.492 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:46:59.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:46:59.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078d6f8516f374040-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj341000falseobj342023-09-22T22:05:14.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj34 2023-09-22T22:46:59.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj34 2023-09-22T22:46:59.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj34 2023-09-22T22:46:59.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.494 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.494 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.494 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.494 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.494 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.494 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:46:59.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:46:59.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088f1caebd0bd8295-00650e0fb3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj341000falseobj342023-09-22T22:05:14.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj34 2023-09-22T22:46:59.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj34 2023-09-22T22:46:59.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj34 2023-09-22T22:46:59.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.495 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.496 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.496 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.496 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.496 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj34 2023-09-22T22:46:59.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.496 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:ynEgdWlacHXX3fqI8nl8YBSAtrA= 2023-09-22T22:46:59.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ynEgdWlacHXX3fqI8nl8YBSAtrA=', 'Content-Length': '0'} 2023-09-22T22:46:59.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000225eced0f33d2051-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj34 2023-09-22T22:46:59.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj34 2023-09-22T22:46:59.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj34 2023-09-22T22:46:59.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.497 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.497 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.497 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.498 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.498 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj34 2023-09-22T22:46:59.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.498 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:ynEgdWlacHXX3fqI8nl8YBSAtrA= 2023-09-22T22:46:59.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ynEgdWlacHXX3fqI8nl8YBSAtrA=', 'Content-Length': '0'} 2023-09-22T22:46:59.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae30d85fd8da6b9d-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.498 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj34 2023-09-22T22:46:59.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj34 2023-09-22T22:46:59.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj34 2023-09-22T22:46:59.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj34 2023-09-22T22:46:59.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.499 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.499 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.499 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.499 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.499 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj34 2023-09-22T22:46:59.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.500 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oj/WlJNtJ5n3m1hE/jJL/5tQeds= 2023-09-22T22:46:59.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oj/WlJNtJ5n3m1hE/jJL/5tQeds=', 'Content-Length': '0'} 2023-09-22T22:46:59.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6b5b6002cac5ac8-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj34 2023-09-22T22:46:59.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj34 2023-09-22T22:46:59.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj34 2023-09-22T22:46:59.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.501 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.501 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.501 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.501 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.501 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj34 2023-09-22T22:46:59.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.501 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oj/WlJNtJ5n3m1hE/jJL/5tQeds= 2023-09-22T22:46:59.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oj/WlJNtJ5n3m1hE/jJL/5tQeds=', 'Content-Length': '0'} 2023-09-22T22:46:59.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dc8f3a6d0c74040-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj34 2023-09-22T22:46:59.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj34 2023-09-22T22:46:59.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj34 2023-09-22T22:46:59.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.503 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.503 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.503 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.503 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.503 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.503 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:46:59.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:46:59.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eacf51c4f6c934ef-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj341000falseobj342023-09-22T22:05:14.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj34 2023-09-22T22:46:59.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj34 2023-09-22T22:46:59.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj34 2023-09-22T22:46:59.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.505 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.505 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.505 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.505 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.505 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.505 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8= 2023-09-22T22:46:59.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bCo1ucnFtWf6fL+0tGB0R5pJ3j8=', 'Content-Length': '0'} 2023-09-22T22:46:59.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8aba1039505164e-00650e0fb3-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj341000falseobj342023-09-22T22:05:14.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.506 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj37 2023-09-22T22:46:59.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj37 2023-09-22T22:46:59.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj37 2023-09-22T22:46:59.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.507 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.507 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.507 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.507 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.507 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.507 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:tMMV6l4vg0+L8hoMus+8oWv1N1U= 2023-09-22T22:46:59.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tMMV6l4vg0+L8hoMus+8oWv1N1U=', 'Content-Length': '0'} 2023-09-22T22:46:59.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da2d9eef6987f003-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj37 2023-09-22T22:46:59.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj37 2023-09-22T22:46:59.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.509 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.509 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.509 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.509 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.509 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.509 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:tMMV6l4vg0+L8hoMus+8oWv1N1U= 2023-09-22T22:46:59.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tMMV6l4vg0+L8hoMus+8oWv1N1U=', 'Content-Length': '0'} 2023-09-22T22:46:59.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af78e68916ed7fc7-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj37 2023-09-22T22:46:59.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj37 2023-09-22T22:46:59.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.511 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.511 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.511 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.511 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.511 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.511 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:8WRoQzi1CkUkYQIzpuA6lbdMmRc= 2023-09-22T22:46:59.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8WRoQzi1CkUkYQIzpuA6lbdMmRc=', 'Content-Length': '0'} 2023-09-22T22:46:59.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000396c13d5e764fc0e-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj37 2023-09-22T22:46:59.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj37 2023-09-22T22:46:59.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.513 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.513 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.513 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.513 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.513 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.513 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:8WRoQzi1CkUkYQIzpuA6lbdMmRc= 2023-09-22T22:46:59.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8WRoQzi1CkUkYQIzpuA6lbdMmRc=', 'Content-Length': '0'} 2023-09-22T22:46:59.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000776fe2ea47775a0a-00650e0fb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.513 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj37 2023-09-22T22:46:59.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj37 2023-09-22T22:46:59.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj37 2023-09-22T22:46:59.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.515 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.515 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.515 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.515 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:39 GMT 2023-09-22T22:46:59.515 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.515 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:tMMV6l4vg0+L8hoMus+8oWv1N1U= 2023-09-22T22:46:59.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:39 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tMMV6l4vg0+L8hoMus+8oWv1N1U=', 'Content-Length': '0'} 2023-09-22T22:46:59.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dae22ee003619dc1-00650e0fb3-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj37 2023-09-22T22:46:59.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj37 2023-09-22T22:46:59.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.516 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.517 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.517 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.517 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.517 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj37?versionId=null 2023-09-22T22:46:59.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.517 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:RRgZJRYs+Zpq3Z33XrYHN1Uix4I= 2023-09-22T22:46:59.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RRgZJRYs+Zpq3Z33XrYHN1Uix4I=', 'Content-Length': '0'} 2023-09-22T22:46:59.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000415545d9856815d-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj37 2023-09-22T22:46:59.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj37 2023-09-22T22:46:59.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj37 2023-09-22T22:46:59.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.518 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.519 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.519 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.519 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.519 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.519 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d066638a228bca24-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj371000falseobj372023-09-22T22:05:15.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj37 2023-09-22T22:46:59.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj37 2023-09-22T22:46:59.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj37 2023-09-22T22:46:59.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.520 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.521 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.521 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.521 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.521 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.521 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000367a57655a16ef04-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj371000falseobj372023-09-22T22:05:15.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj37 2023-09-22T22:46:59.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj37 2023-09-22T22:46:59.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj37 2023-09-22T22:46:59.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.522 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.522 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.523 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.523 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.523 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj37 2023-09-22T22:46:59.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.523 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:1CIbB0EUuvhCN0rZPCh+C8Rtm1g= 2023-09-22T22:46:59.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1CIbB0EUuvhCN0rZPCh+C8Rtm1g=', 'Content-Length': '0'} 2023-09-22T22:46:59.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012930c9b1df54a81-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj37 2023-09-22T22:46:59.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj37 2023-09-22T22:46:59.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj37 2023-09-22T22:46:59.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.524 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.524 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.524 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.524 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.525 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj37 2023-09-22T22:46:59.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.525 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:1CIbB0EUuvhCN0rZPCh+C8Rtm1g= 2023-09-22T22:46:59.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1CIbB0EUuvhCN0rZPCh+C8Rtm1g=', 'Content-Length': '0'} 2023-09-22T22:46:59.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9ccc51ede129352-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.525 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj37 2023-09-22T22:46:59.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj37 2023-09-22T22:46:59.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj37 2023-09-22T22:46:59.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj37 2023-09-22T22:46:59.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.526 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.526 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.526 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.526 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.526 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj37 2023-09-22T22:46:59.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.527 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:1gkWHUmBkAXSOau12UA+Pa/tTVQ= 2023-09-22T22:46:59.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1gkWHUmBkAXSOau12UA+Pa/tTVQ=', 'Content-Length': '0'} 2023-09-22T22:46:59.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e4bfbe063dc7a2f-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj37 2023-09-22T22:46:59.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj37 2023-09-22T22:46:59.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj37 2023-09-22T22:46:59.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.528 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.528 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.528 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.528 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.528 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj37 2023-09-22T22:46:59.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.528 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:1gkWHUmBkAXSOau12UA+Pa/tTVQ= 2023-09-22T22:46:59.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1gkWHUmBkAXSOau12UA+Pa/tTVQ=', 'Content-Length': '0'} 2023-09-22T22:46:59.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d1bacb6d3635a78-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj37 2023-09-22T22:46:59.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj37 2023-09-22T22:46:59.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj37 2023-09-22T22:46:59.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.530 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.530 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.530 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.530 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.530 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.530 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d732a3651e466b3-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj371000falseobj372023-09-22T22:05:15.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj37 2023-09-22T22:46:59.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj37 2023-09-22T22:46:59.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj37 2023-09-22T22:46:59.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.532 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.532 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.532 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.532 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.532 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.532 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b892a9ba676902d-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj371000falseobj372023-09-22T22:05:15.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.533 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj38 2023-09-22T22:46:59.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj38 2023-09-22T22:46:59.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj38 2023-09-22T22:46:59.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.534 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.534 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.534 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.534 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.534 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.535 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:fPNN++EmkVj7/Pyc745/p+ve9xs= 2023-09-22T22:46:59.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fPNN++EmkVj7/Pyc745/p+ve9xs=', 'Content-Length': '0'} 2023-09-22T22:46:59.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e1088a566b4f34c-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj38 2023-09-22T22:46:59.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj38 2023-09-22T22:46:59.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.536 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.536 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.536 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.536 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.536 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.536 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:fPNN++EmkVj7/Pyc745/p+ve9xs= 2023-09-22T22:46:59.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fPNN++EmkVj7/Pyc745/p+ve9xs=', 'Content-Length': '0'} 2023-09-22T22:46:59.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c822640d7327322d-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj38 2023-09-22T22:46:59.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj38 2023-09-22T22:46:59.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.538 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.538 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.538 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.538 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.538 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.538 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:/rKp9M7/U2mw7ynEHzUc6eZkAJE= 2023-09-22T22:46:59.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/rKp9M7/U2mw7ynEHzUc6eZkAJE=', 'Content-Length': '0'} 2023-09-22T22:46:59.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d91caefe6fda4041-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj38 2023-09-22T22:46:59.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj38 2023-09-22T22:46:59.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.540 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.540 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.540 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.540 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.540 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.540 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:/rKp9M7/U2mw7ynEHzUc6eZkAJE= 2023-09-22T22:46:59.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/rKp9M7/U2mw7ynEHzUc6eZkAJE=', 'Content-Length': '0'} 2023-09-22T22:46:59.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007da552b6eb1f5705-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.541 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj38 2023-09-22T22:46:59.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj38 2023-09-22T22:46:59.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj38 2023-09-22T22:46:59.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.542 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.542 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.542 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.542 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.542 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.542 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:fPNN++EmkVj7/Pyc745/p+ve9xs= 2023-09-22T22:46:59.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fPNN++EmkVj7/Pyc745/p+ve9xs=', 'Content-Length': '0'} 2023-09-22T22:46:59.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042f2a0e10276ef4e-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj38 2023-09-22T22:46:59.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj38 2023-09-22T22:46:59.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.544 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.544 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.544 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.544 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.544 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj38?versionId=null 2023-09-22T22:46:59.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.544 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:fPNN++EmkVj7/Pyc745/p+ve9xs= 2023-09-22T22:46:59.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fPNN++EmkVj7/Pyc745/p+ve9xs=', 'Content-Length': '0'} 2023-09-22T22:46:59.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e15f32fcdcb3291-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj38 2023-09-22T22:46:59.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj38 2023-09-22T22:46:59.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj38 2023-09-22T22:46:59.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.546 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.546 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.546 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.546 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.546 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.546 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090eb80a566ea4088-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj381000falseobj382023-09-22T22:05:14.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj38 2023-09-22T22:46:59.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj38 2023-09-22T22:46:59.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj38 2023-09-22T22:46:59.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.548 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.548 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.548 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.548 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.548 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.548 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052fb9ef93f16fc56-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj381000falseobj382023-09-22T22:05:14.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj38 2023-09-22T22:46:59.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj38 2023-09-22T22:46:59.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj38 2023-09-22T22:46:59.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.549 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.549 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.550 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.550 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.550 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj38 2023-09-22T22:46:59.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.550 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:RhXSwvM/7Vc6nh8WPji+Uqw47/w= 2023-09-22T22:46:59.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RhXSwvM/7Vc6nh8WPji+Uqw47/w=', 'Content-Length': '0'} 2023-09-22T22:46:59.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ce304f7052a33ad-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj38 2023-09-22T22:46:59.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj38 2023-09-22T22:46:59.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj38 2023-09-22T22:46:59.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.551 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.551 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.551 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.551 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.551 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj38 2023-09-22T22:46:59.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.552 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:RhXSwvM/7Vc6nh8WPji+Uqw47/w= 2023-09-22T22:46:59.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RhXSwvM/7Vc6nh8WPji+Uqw47/w=', 'Content-Length': '0'} 2023-09-22T22:46:59.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f75c05b19793157-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.552 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj38 2023-09-22T22:46:59.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj38 2023-09-22T22:46:59.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj38 2023-09-22T22:46:59.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj38 2023-09-22T22:46:59.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.553 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.553 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.553 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.553 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.553 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj38 2023-09-22T22:46:59.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.553 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:J6qmxcd842GC8frB3hrMRAusR00= 2023-09-22T22:46:59.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J6qmxcd842GC8frB3hrMRAusR00=', 'Content-Length': '0'} 2023-09-22T22:46:59.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c86fc0e5200a69a-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj38 2023-09-22T22:46:59.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj38 2023-09-22T22:46:59.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj38 2023-09-22T22:46:59.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.555 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.555 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.555 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.555 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.555 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj38 2023-09-22T22:46:59.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.555 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:J6qmxcd842GC8frB3hrMRAusR00= 2023-09-22T22:46:59.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:J6qmxcd842GC8frB3hrMRAusR00=', 'Content-Length': '0'} 2023-09-22T22:46:59.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005abd6d04298c3479-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj38 2023-09-22T22:46:59.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj38 2023-09-22T22:46:59.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj38 2023-09-22T22:46:59.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.557 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.557 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.557 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.557 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.557 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.557 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0ccaa28b63fa1ed-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj381000falseobj382023-09-22T22:05:14.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj38 2023-09-22T22:46:59.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj38 2023-09-22T22:46:59.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj38 2023-09-22T22:46:59.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.559 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.559 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.559 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.559 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.559 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.559 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb00e72b24913425-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj381000falseobj382023-09-22T22:05:14.428Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.560 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-09-22T22:46:59.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj44 2023-09-22T22:46:59.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj44 2023-09-22T22:46:59.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.561 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.561 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.561 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.561 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.561 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.561 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:AlajPbA0VfaL+rN7XEawl/36qdg= 2023-09-22T22:46:59.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AlajPbA0VfaL+rN7XEawl/36qdg=', 'Content-Length': '0'} 2023-09-22T22:46:59.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cd6aa3a0f2b3ec5-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj44 2023-09-22T22:46:59.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj44 2023-09-22T22:46:59.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.563 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.563 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.563 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.563 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.563 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.563 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:AlajPbA0VfaL+rN7XEawl/36qdg= 2023-09-22T22:46:59.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AlajPbA0VfaL+rN7XEawl/36qdg=', 'Content-Length': '0'} 2023-09-22T22:46:59.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee7c53020e6dcb56-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj44 2023-09-22T22:46:59.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj44 2023-09-22T22:46:59.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.565 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.565 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.565 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.565 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.565 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.565 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:0J2r0OAitL6gcisFFJPLjCpGF3k= 2023-09-22T22:46:59.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0J2r0OAitL6gcisFFJPLjCpGF3k=', 'Content-Length': '0'} 2023-09-22T22:46:59.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000654bc0358b42a01c-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj44 2023-09-22T22:46:59.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj44 2023-09-22T22:46:59.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.566 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.567 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.567 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.567 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.567 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.567 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:0J2r0OAitL6gcisFFJPLjCpGF3k= 2023-09-22T22:46:59.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0J2r0OAitL6gcisFFJPLjCpGF3k=', 'Content-Length': '0'} 2023-09-22T22:46:59.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051070ed008a011d5-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.567 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-09-22T22:46:59.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj44 2023-09-22T22:46:59.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj44 2023-09-22T22:46:59.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.568 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.569 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.569 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.569 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.569 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.569 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:AlajPbA0VfaL+rN7XEawl/36qdg= 2023-09-22T22:46:59.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AlajPbA0VfaL+rN7XEawl/36qdg=', 'Content-Length': '0'} 2023-09-22T22:46:59.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b4535e10c947621-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj44 2023-09-22T22:46:59.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj44 2023-09-22T22:46:59.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.570 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.570 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.571 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.571 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.571 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj44?versionId=null 2023-09-22T22:46:59.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.571 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:AlajPbA0VfaL+rN7XEawl/36qdg= 2023-09-22T22:46:59.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AlajPbA0VfaL+rN7XEawl/36qdg=', 'Content-Length': '0'} 2023-09-22T22:46:59.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f58da9067887741-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj44 2023-09-22T22:46:59.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj44 2023-09-22T22:46:59.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj44 2023-09-22T22:46:59.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.572 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.572 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.572 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.573 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.573 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.573 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000539fbad568c89e96-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj441000falseobj442023-09-22T22:05:13.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj44 2023-09-22T22:46:59.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj44 2023-09-22T22:46:59.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj44 2023-09-22T22:46:59.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.574 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.574 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.575 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.575 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.575 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.575 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4086c4779db3cbc-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj441000falseobj442023-09-22T22:05:13.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj44 2023-09-22T22:46:59.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj44 2023-09-22T22:46:59.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj44 2023-09-22T22:46:59.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.576 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.576 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.576 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.577 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.577 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj44 2023-09-22T22:46:59.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.577 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kcwDutLOF3sQt2VKad4KG1HjJUM= 2023-09-22T22:46:59.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kcwDutLOF3sQt2VKad4KG1HjJUM=', 'Content-Length': '0'} 2023-09-22T22:46:59.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9bde5035d17c939-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj44 2023-09-22T22:46:59.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj44 2023-09-22T22:46:59.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj44 2023-09-22T22:46:59.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.578 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.578 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.578 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.578 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.578 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj44 2023-09-22T22:46:59.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.579 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kcwDutLOF3sQt2VKad4KG1HjJUM= 2023-09-22T22:46:59.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kcwDutLOF3sQt2VKad4KG1HjJUM=', 'Content-Length': '0'} 2023-09-22T22:46:59.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a5d9731e36d9d54-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.579 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-09-22T22:46:59.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj44 2023-09-22T22:46:59.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj44 2023-09-22T22:46:59.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj44 2023-09-22T22:46:59.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.580 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.580 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.580 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.580 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.580 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj44 2023-09-22T22:46:59.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.580 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:7C8xyDVQwG31/49Siciixifpay4= 2023-09-22T22:46:59.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7C8xyDVQwG31/49Siciixifpay4=', 'Content-Length': '0'} 2023-09-22T22:46:59.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000569d1b39ee59ec34-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj44 2023-09-22T22:46:59.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj44 2023-09-22T22:46:59.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj44 2023-09-22T22:46:59.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.582 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.582 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.582 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.582 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.582 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj44 2023-09-22T22:46:59.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.582 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:7C8xyDVQwG31/49Siciixifpay4= 2023-09-22T22:46:59.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7C8xyDVQwG31/49Siciixifpay4=', 'Content-Length': '0'} 2023-09-22T22:46:59.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f9b984cc1ebdc84-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj44 2023-09-22T22:46:59.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj44 2023-09-22T22:46:59.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj44 2023-09-22T22:46:59.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.584 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.584 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.584 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.584 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.584 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.584 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006780df58dd8c8b7d-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj441000falseobj442023-09-22T22:05:13.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj44 2023-09-22T22:46:59.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj44 2023-09-22T22:46:59.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj44 2023-09-22T22:46:59.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.586 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.586 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.586 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.586 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.586 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.586 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000798a5a6fee473949-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj441000falseobj442023-09-22T22:05:13.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.587 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj46 2023-09-22T22:46:59.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj46 2023-09-22T22:46:59.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj46 2023-09-22T22:46:59.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.588 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.588 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.588 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.588 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.588 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.588 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sU1gAwvZdmrCJZ9hGDFtZYU1UEM= 2023-09-22T22:46:59.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sU1gAwvZdmrCJZ9hGDFtZYU1UEM=', 'Content-Length': '0'} 2023-09-22T22:46:59.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7126b2c582d8956-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj46 2023-09-22T22:46:59.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj46 2023-09-22T22:46:59.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.590 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.590 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.590 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.590 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.590 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.590 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sU1gAwvZdmrCJZ9hGDFtZYU1UEM= 2023-09-22T22:46:59.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sU1gAwvZdmrCJZ9hGDFtZYU1UEM=', 'Content-Length': '0'} 2023-09-22T22:46:59.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e2fd840357ff1f2-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj46 2023-09-22T22:46:59.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj46 2023-09-22T22:46:59.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.591 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.592 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.592 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.592 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.592 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.592 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:rCWjQak7U9auKqBxhpzCkYnuc4o= 2023-09-22T22:46:59.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rCWjQak7U9auKqBxhpzCkYnuc4o=', 'Content-Length': '0'} 2023-09-22T22:46:59.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014cbf5d661c22a7c-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj46 2023-09-22T22:46:59.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj46 2023-09-22T22:46:59.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.593 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.593 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.594 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.594 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.594 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.594 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:rCWjQak7U9auKqBxhpzCkYnuc4o= 2023-09-22T22:46:59.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rCWjQak7U9auKqBxhpzCkYnuc4o=', 'Content-Length': '0'} 2023-09-22T22:46:59.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000087f9dcbc2a309b0-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.594 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj46 2023-09-22T22:46:59.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj46 2023-09-22T22:46:59.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj46 2023-09-22T22:46:59.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.595 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.595 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.596 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.596 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.596 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.596 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sU1gAwvZdmrCJZ9hGDFtZYU1UEM= 2023-09-22T22:46:59.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sU1gAwvZdmrCJZ9hGDFtZYU1UEM=', 'Content-Length': '0'} 2023-09-22T22:46:59.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd5b6f0e06461a0f-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj46 2023-09-22T22:46:59.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj46 2023-09-22T22:46:59.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.597 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.597 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.597 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.598 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.598 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj46?versionId=null 2023-09-22T22:46:59.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.598 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:sU1gAwvZdmrCJZ9hGDFtZYU1UEM= 2023-09-22T22:46:59.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:sU1gAwvZdmrCJZ9hGDFtZYU1UEM=', 'Content-Length': '0'} 2023-09-22T22:46:59.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029c9bc8dcfcd978f-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj46 2023-09-22T22:46:59.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj46 2023-09-22T22:46:59.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj46 2023-09-22T22:46:59.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.599 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.599 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.599 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.600 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.600 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.600 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9dc419cef4220cf-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj461000falseobj462023-09-22T22:05:14.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj46 2023-09-22T22:46:59.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj46 2023-09-22T22:46:59.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj46 2023-09-22T22:46:59.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.601 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.601 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.601 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.602 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.602 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.602 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003424316997bffdc6-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj461000falseobj462023-09-22T22:05:14.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj46 2023-09-22T22:46:59.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj46 2023-09-22T22:46:59.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj46 2023-09-22T22:46:59.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.603 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.603 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.603 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.604 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.604 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj46 2023-09-22T22:46:59.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.604 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:zyJvmpDY8irsZn3g9bhDv+qYLJQ= 2023-09-22T22:46:59.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zyJvmpDY8irsZn3g9bhDv+qYLJQ=', 'Content-Length': '0'} 2023-09-22T22:46:59.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c18221e987028d7-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj46 2023-09-22T22:46:59.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj46 2023-09-22T22:46:59.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj46 2023-09-22T22:46:59.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.605 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.605 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.605 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.605 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.605 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj46 2023-09-22T22:46:59.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.606 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:zyJvmpDY8irsZn3g9bhDv+qYLJQ= 2023-09-22T22:46:59.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zyJvmpDY8irsZn3g9bhDv+qYLJQ=', 'Content-Length': '0'} 2023-09-22T22:46:59.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000963026ba8dbf0701-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.606 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj46 2023-09-22T22:46:59.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj46 2023-09-22T22:46:59.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj46 2023-09-22T22:46:59.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj46 2023-09-22T22:46:59.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.607 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.607 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.607 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.607 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.607 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj46 2023-09-22T22:46:59.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.607 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:VOD2KbsFlyTKPzaXrDb8UvqNDI8= 2023-09-22T22:46:59.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VOD2KbsFlyTKPzaXrDb8UvqNDI8=', 'Content-Length': '0'} 2023-09-22T22:46:59.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c452f974b5a6130a-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj46 2023-09-22T22:46:59.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj46 2023-09-22T22:46:59.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj46 2023-09-22T22:46:59.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.609 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.609 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.609 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.609 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.609 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj46 2023-09-22T22:46:59.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.609 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:VOD2KbsFlyTKPzaXrDb8UvqNDI8= 2023-09-22T22:46:59.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VOD2KbsFlyTKPzaXrDb8UvqNDI8=', 'Content-Length': '0'} 2023-09-22T22:46:59.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fd289c8cf5f0793-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj46 2023-09-22T22:46:59.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj46 2023-09-22T22:46:59.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj46 2023-09-22T22:46:59.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.611 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.611 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.611 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.611 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.611 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.611 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acb4f2a08479023c-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj461000falseobj462023-09-22T22:05:14.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj46 2023-09-22T22:46:59.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj46 2023-09-22T22:46:59.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj46 2023-09-22T22:46:59.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.613 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.613 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.613 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.613 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.613 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.613 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c01980ba945dee9-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj461000falseobj462023-09-22T22:05:14.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.614 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj51 2023-09-22T22:46:59.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj51 2023-09-22T22:46:59.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj51 2023-09-22T22:46:59.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.615 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.615 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.615 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.615 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.615 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.616 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:DcFBQbb0OaJX99+jaf+cusiuYKU= 2023-09-22T22:46:59.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DcFBQbb0OaJX99+jaf+cusiuYKU=', 'Content-Length': '0'} 2023-09-22T22:46:59.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eefa6e42bad3adfc-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj51 2023-09-22T22:46:59.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj51 2023-09-22T22:46:59.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.617 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.617 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.617 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.617 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.617 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.618 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:DcFBQbb0OaJX99+jaf+cusiuYKU= 2023-09-22T22:46:59.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DcFBQbb0OaJX99+jaf+cusiuYKU=', 'Content-Length': '0'} 2023-09-22T22:46:59.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc8541b3cf277c4e-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj51 2023-09-22T22:46:59.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj51 2023-09-22T22:46:59.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.619 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.619 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.619 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.619 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.619 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.620 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:CFG5ela/Bcg3ZsO3/J7LOU8DOE4= 2023-09-22T22:46:59.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CFG5ela/Bcg3ZsO3/J7LOU8DOE4=', 'Content-Length': '0'} 2023-09-22T22:46:59.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070f700ea0cdfbdc1-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj51 2023-09-22T22:46:59.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj51 2023-09-22T22:46:59.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.621 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.621 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.621 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.621 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.621 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.622 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:CFG5ela/Bcg3ZsO3/J7LOU8DOE4= 2023-09-22T22:46:59.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CFG5ela/Bcg3ZsO3/J7LOU8DOE4=', 'Content-Length': '0'} 2023-09-22T22:46:59.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ceee6ec1d3c3261-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.622 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj51 2023-09-22T22:46:59.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj51 2023-09-22T22:46:59.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj51 2023-09-22T22:46:59.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.623 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.623 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.623 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.623 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.623 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.624 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:DcFBQbb0OaJX99+jaf+cusiuYKU= 2023-09-22T22:46:59.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DcFBQbb0OaJX99+jaf+cusiuYKU=', 'Content-Length': '0'} 2023-09-22T22:46:59.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007df33980debd6bb9-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj51 2023-09-22T22:46:59.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj51 2023-09-22T22:46:59.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.625 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.625 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.625 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.625 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.625 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj51?versionId=null 2023-09-22T22:46:59.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.626 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:DcFBQbb0OaJX99+jaf+cusiuYKU= 2023-09-22T22:46:59.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:DcFBQbb0OaJX99+jaf+cusiuYKU=', 'Content-Length': '0'} 2023-09-22T22:46:59.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045e51e66f2cacd10-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj51 2023-09-22T22:46:59.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj51 2023-09-22T22:46:59.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj51 2023-09-22T22:46:59.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.627 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.627 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.627 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.627 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.627 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.628 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000834516f94cbc612a-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj511000falseobj512023-09-22T22:05:14.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj51 2023-09-22T22:46:59.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj51 2023-09-22T22:46:59.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj51 2023-09-22T22:46:59.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.629 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.629 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.629 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.629 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.629 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.630 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002487477748ad3629-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj511000falseobj512023-09-22T22:05:14.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj51 2023-09-22T22:46:59.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj51 2023-09-22T22:46:59.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj51 2023-09-22T22:46:59.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.631 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.631 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.631 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.631 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.631 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj51 2023-09-22T22:46:59.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.631 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:WKOEbnuB8JJ42i6P2knu+nTfxQg= 2023-09-22T22:46:59.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WKOEbnuB8JJ42i6P2knu+nTfxQg=', 'Content-Length': '0'} 2023-09-22T22:46:59.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043cbb3165587527e-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj51 2023-09-22T22:46:59.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj51 2023-09-22T22:46:59.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj51 2023-09-22T22:46:59.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.633 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.633 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.633 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.633 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.633 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj51 2023-09-22T22:46:59.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.633 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:WKOEbnuB8JJ42i6P2knu+nTfxQg= 2023-09-22T22:46:59.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WKOEbnuB8JJ42i6P2knu+nTfxQg=', 'Content-Length': '0'} 2023-09-22T22:46:59.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006295efec391ecb95-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.634 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj51 2023-09-22T22:46:59.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj51 2023-09-22T22:46:59.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj51 2023-09-22T22:46:59.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj51 2023-09-22T22:46:59.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.635 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.635 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.635 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.635 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.635 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj51 2023-09-22T22:46:59.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.635 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Ms/4XGScL/CGCtr5A8umD+CySoA= 2023-09-22T22:46:59.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ms/4XGScL/CGCtr5A8umD+CySoA=', 'Content-Length': '0'} 2023-09-22T22:46:59.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000642e74f8bb4aa855-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj51 2023-09-22T22:46:59.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj51 2023-09-22T22:46:59.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj51 2023-09-22T22:46:59.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.636 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.637 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.637 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.637 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.637 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj51 2023-09-22T22:46:59.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.637 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Ms/4XGScL/CGCtr5A8umD+CySoA= 2023-09-22T22:46:59.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ms/4XGScL/CGCtr5A8umD+CySoA=', 'Content-Length': '0'} 2023-09-22T22:46:59.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008985ba2faef078d5-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj51 2023-09-22T22:46:59.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj51 2023-09-22T22:46:59.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj51 2023-09-22T22:46:59.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.638 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.639 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.639 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.639 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.639 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.639 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ab3b155ebb20af6-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj511000falseobj512023-09-22T22:05:14.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj51 2023-09-22T22:46:59.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj51 2023-09-22T22:46:59.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj51 2023-09-22T22:46:59.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.640 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.640 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.641 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.641 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.641 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.641 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd6872bfb2ec3af9-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj511000falseobj512023-09-22T22:05:14.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.641 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-09-22T22:46:59.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj55 2023-09-22T22:46:59.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj55 2023-09-22T22:46:59.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.643 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.643 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.643 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.643 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.643 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.643 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:3ptq3BCD3fgbA9BKIyI3LINdX3c= 2023-09-22T22:46:59.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3ptq3BCD3fgbA9BKIyI3LINdX3c=', 'Content-Length': '0'} 2023-09-22T22:46:59.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1cf258526174f24-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj55 2023-09-22T22:46:59.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj55 2023-09-22T22:46:59.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.645 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.645 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.645 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.645 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.645 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.645 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:3ptq3BCD3fgbA9BKIyI3LINdX3c= 2023-09-22T22:46:59.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3ptq3BCD3fgbA9BKIyI3LINdX3c=', 'Content-Length': '0'} 2023-09-22T22:46:59.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067b6ea8a14e61c84-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj55 2023-09-22T22:46:59.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj55 2023-09-22T22:46:59.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.646 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.647 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.647 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.647 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.647 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.647 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:eRBy4rjXgmJYBbmSRVHNtZHjmDQ= 2023-09-22T22:46:59.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eRBy4rjXgmJYBbmSRVHNtZHjmDQ=', 'Content-Length': '0'} 2023-09-22T22:46:59.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f5ce8751bd1c179-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj55 2023-09-22T22:46:59.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj55 2023-09-22T22:46:59.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.648 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.648 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.649 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.649 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.649 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.649 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:eRBy4rjXgmJYBbmSRVHNtZHjmDQ= 2023-09-22T22:46:59.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eRBy4rjXgmJYBbmSRVHNtZHjmDQ=', 'Content-Length': '0'} 2023-09-22T22:46:59.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f69bb510f6bf077-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.649 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-09-22T22:46:59.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj55 2023-09-22T22:46:59.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj55 2023-09-22T22:46:59.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.650 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.651 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.651 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.651 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.651 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.651 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:3ptq3BCD3fgbA9BKIyI3LINdX3c= 2023-09-22T22:46:59.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3ptq3BCD3fgbA9BKIyI3LINdX3c=', 'Content-Length': '0'} 2023-09-22T22:46:59.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afb572b9a42ad78a-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj55 2023-09-22T22:46:59.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj55 2023-09-22T22:46:59.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.652 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.652 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.653 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.653 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.653 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj55?versionId=null 2023-09-22T22:46:59.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.653 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:3ptq3BCD3fgbA9BKIyI3LINdX3c= 2023-09-22T22:46:59.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3ptq3BCD3fgbA9BKIyI3LINdX3c=', 'Content-Length': '0'} 2023-09-22T22:46:59.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8762b2a4a430ac7-00650e0fb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj55 2023-09-22T22:46:59.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj55 2023-09-22T22:46:59.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj55 2023-09-22T22:46:59.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.654 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.654 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.655 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.655 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.655 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.655 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e8913117861d887-00650e0fb4-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj551000falseobj552023-09-22T22:05:13.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj55 2023-09-22T22:46:59.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj55 2023-09-22T22:46:59.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj55 2023-09-22T22:46:59.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.658 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.658 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.658 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.658 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:40 GMT 2023-09-22T22:46:59.658 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.658 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM= 2023-09-22T22:46:59.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:40 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cVddQ+ftcMbZ1wqsinxp14lTDnM=', 'Content-Length': '0'} 2023-09-22T22:46:59.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c189833f4ff5bf8-00650e0fb4-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj551000falseobj552023-09-22T22:05:13.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj55 2023-09-22T22:46:59.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj55 2023-09-22T22:46:59.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj55 2023-09-22T22:46:59.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.660 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.660 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.660 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.660 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.660 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj55 2023-09-22T22:46:59.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.660 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:p1yxpdoeBeMXiDXoFgdM1QisM9E= 2023-09-22T22:46:59.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:p1yxpdoeBeMXiDXoFgdM1QisM9E=', 'Content-Length': '0'} 2023-09-22T22:46:59.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042d25124f0526d2f-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj55 2023-09-22T22:46:59.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj55 2023-09-22T22:46:59.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj55 2023-09-22T22:46:59.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.661 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.661 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.662 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.662 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.662 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj55 2023-09-22T22:46:59.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.662 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:p1yxpdoeBeMXiDXoFgdM1QisM9E= 2023-09-22T22:46:59.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:p1yxpdoeBeMXiDXoFgdM1QisM9E=', 'Content-Length': '0'} 2023-09-22T22:46:59.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092f7dde3d2343233-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.662 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-09-22T22:46:59.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj55 2023-09-22T22:46:59.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj55 2023-09-22T22:46:59.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj55 2023-09-22T22:46:59.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.663 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.663 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.663 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.664 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.664 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj55 2023-09-22T22:46:59.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.664 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:u7VlXbGC0TAhSa9bnJSCMsX68m8= 2023-09-22T22:46:59.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:u7VlXbGC0TAhSa9bnJSCMsX68m8=', 'Content-Length': '0'} 2023-09-22T22:46:59.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000611f96be3d4678c9-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj55 2023-09-22T22:46:59.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj55 2023-09-22T22:46:59.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj55 2023-09-22T22:46:59.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.665 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.665 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.665 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.665 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.665 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj55 2023-09-22T22:46:59.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.666 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:u7VlXbGC0TAhSa9bnJSCMsX68m8= 2023-09-22T22:46:59.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:u7VlXbGC0TAhSa9bnJSCMsX68m8=', 'Content-Length': '0'} 2023-09-22T22:46:59.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be109c05f0a6fbe1-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj55 2023-09-22T22:46:59.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj55 2023-09-22T22:46:59.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj55 2023-09-22T22:46:59.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.667 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.667 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.667 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.667 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.667 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.667 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:46:59.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:46:59.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8a05e608eaa07ac-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj551000falseobj552023-09-22T22:05:13.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj55 2023-09-22T22:46:59.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj55 2023-09-22T22:46:59.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj55 2023-09-22T22:46:59.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.669 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.669 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.669 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.669 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.669 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.670 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:46:59.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:46:59.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009eddf353ffaec0c7-00650e0fb5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj551000falseobj552023-09-22T22:05:13.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.670 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj57 2023-09-22T22:46:59.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj57 2023-09-22T22:46:59.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj57 2023-09-22T22:46:59.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.671 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.671 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.671 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.671 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.671 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.672 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:8CsBBhFi5YgAfSzFr/cJdELMHdo= 2023-09-22T22:46:59.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8CsBBhFi5YgAfSzFr/cJdELMHdo=', 'Content-Length': '0'} 2023-09-22T22:46:59.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008255446dceac2161-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj57 2023-09-22T22:46:59.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj57 2023-09-22T22:46:59.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.673 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.673 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.673 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.673 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.673 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.674 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:8CsBBhFi5YgAfSzFr/cJdELMHdo= 2023-09-22T22:46:59.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8CsBBhFi5YgAfSzFr/cJdELMHdo=', 'Content-Length': '0'} 2023-09-22T22:46:59.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000613b7fe38da60e3d-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj57 2023-09-22T22:46:59.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj57 2023-09-22T22:46:59.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.675 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.675 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.675 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.675 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.676 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.676 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:fPqGlTXpzWHIs305gl/MLFDk4pE= 2023-09-22T22:46:59.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fPqGlTXpzWHIs305gl/MLFDk4pE=', 'Content-Length': '0'} 2023-09-22T22:46:59.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c4cc9e1f9836d31-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj57 2023-09-22T22:46:59.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj57 2023-09-22T22:46:59.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.677 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.677 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.677 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.677 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.677 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.678 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:fPqGlTXpzWHIs305gl/MLFDk4pE= 2023-09-22T22:46:59.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:fPqGlTXpzWHIs305gl/MLFDk4pE=', 'Content-Length': '0'} 2023-09-22T22:46:59.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a53fdde1f3038005-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.678 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj57 2023-09-22T22:46:59.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj57 2023-09-22T22:46:59.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj57 2023-09-22T22:46:59.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.679 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.679 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.679 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.679 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.680 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.680 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:8CsBBhFi5YgAfSzFr/cJdELMHdo= 2023-09-22T22:46:59.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8CsBBhFi5YgAfSzFr/cJdELMHdo=', 'Content-Length': '0'} 2023-09-22T22:46:59.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065c94b3f6db2ef81-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj57 2023-09-22T22:46:59.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj57 2023-09-22T22:46:59.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.681 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.681 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.681 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.681 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.681 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj57?versionId=null 2023-09-22T22:46:59.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.682 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:8CsBBhFi5YgAfSzFr/cJdELMHdo= 2023-09-22T22:46:59.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8CsBBhFi5YgAfSzFr/cJdELMHdo=', 'Content-Length': '0'} 2023-09-22T22:46:59.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035064f8b0e87d8dd-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj57 2023-09-22T22:46:59.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj57 2023-09-22T22:46:59.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj57 2023-09-22T22:46:59.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.683 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.683 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.683 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.683 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.683 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.684 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:46:59.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:46:59.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bff49cfe290e6f27-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj571000falseobj572023-09-22T22:05:14.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj57 2023-09-22T22:46:59.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj57 2023-09-22T22:46:59.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj57 2023-09-22T22:46:59.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.685 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.685 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.685 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.685 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.686 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.686 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:46:59.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:46:59.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000262169911eae6152-00650e0fb5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj571000falseobj572023-09-22T22:05:14.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj57 2023-09-22T22:46:59.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj57 2023-09-22T22:46:59.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj57 2023-09-22T22:46:59.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.687 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.687 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.687 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.687 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.687 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj57 2023-09-22T22:46:59.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.688 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uK5xPtcDCpkqD3lfImiha3CLhVg= 2023-09-22T22:46:59.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uK5xPtcDCpkqD3lfImiha3CLhVg=', 'Content-Length': '0'} 2023-09-22T22:46:59.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec7691b086d73334-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj57 2023-09-22T22:46:59.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj57 2023-09-22T22:46:59.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj57 2023-09-22T22:46:59.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.689 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.689 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.689 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.689 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.689 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj57 2023-09-22T22:46:59.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.689 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uK5xPtcDCpkqD3lfImiha3CLhVg= 2023-09-22T22:46:59.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uK5xPtcDCpkqD3lfImiha3CLhVg=', 'Content-Length': '0'} 2023-09-22T22:46:59.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002da948e1533b27d1-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.689 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj57 2023-09-22T22:46:59.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj57 2023-09-22T22:46:59.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj57 2023-09-22T22:46:59.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj57 2023-09-22T22:46:59.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.691 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.691 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.691 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.691 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.691 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj57 2023-09-22T22:46:59.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.691 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:xq2IXsMfuiLnRPNx5OYeGCZbTLY= 2023-09-22T22:46:59.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xq2IXsMfuiLnRPNx5OYeGCZbTLY=', 'Content-Length': '0'} 2023-09-22T22:46:59.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ecba7049d573e47-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj57 2023-09-22T22:46:59.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj57 2023-09-22T22:46:59.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj57 2023-09-22T22:46:59.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.692 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.692 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.693 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.693 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.693 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj57 2023-09-22T22:46:59.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.693 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:xq2IXsMfuiLnRPNx5OYeGCZbTLY= 2023-09-22T22:46:59.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:xq2IXsMfuiLnRPNx5OYeGCZbTLY=', 'Content-Length': '0'} 2023-09-22T22:46:59.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d9448956e03e8e4-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj57 2023-09-22T22:46:59.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj57 2023-09-22T22:46:59.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj57 2023-09-22T22:46:59.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.694 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.694 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.695 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.695 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.695 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.695 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:46:59.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:46:59.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dd883514bca0081-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj571000falseobj572023-09-22T22:05:14.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj57 2023-09-22T22:46:59.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj57 2023-09-22T22:46:59.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj57 2023-09-22T22:46:59.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.696 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.696 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.696 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.697 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.697 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.697 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:46:59.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:46:59.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fec6d0a1e23f0c4c-00650e0fb5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj571000falseobj572023-09-22T22:05:14.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.697 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj58 2023-09-22T22:46:59.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj58 2023-09-22T22:46:59.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj58 2023-09-22T22:46:59.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.698 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.699 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.699 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.699 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.699 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.699 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:HbElvGukCjTOKLFNPpD6yC9cRwc= 2023-09-22T22:46:59.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HbElvGukCjTOKLFNPpD6yC9cRwc=', 'Content-Length': '0'} 2023-09-22T22:46:59.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e6c2321e13e57bf-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj58 2023-09-22T22:46:59.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj58 2023-09-22T22:46:59.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.700 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.700 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.700 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.701 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.701 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.701 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:HbElvGukCjTOKLFNPpD6yC9cRwc= 2023-09-22T22:46:59.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HbElvGukCjTOKLFNPpD6yC9cRwc=', 'Content-Length': '0'} 2023-09-22T22:46:59.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b716b4e98b3e6a4-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj58 2023-09-22T22:46:59.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj58 2023-09-22T22:46:59.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.702 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.702 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.702 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.703 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.703 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.703 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:iYqbM+TCxQfw3D9Y04x8aeqNjzE= 2023-09-22T22:46:59.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYqbM+TCxQfw3D9Y04x8aeqNjzE=', 'Content-Length': '0'} 2023-09-22T22:46:59.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009907954b1d9932d6-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj58 2023-09-22T22:46:59.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj58 2023-09-22T22:46:59.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.704 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.704 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.704 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.705 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.705 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.705 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:iYqbM+TCxQfw3D9Y04x8aeqNjzE= 2023-09-22T22:46:59.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iYqbM+TCxQfw3D9Y04x8aeqNjzE=', 'Content-Length': '0'} 2023-09-22T22:46:59.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000998df9d409e41c54-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.705 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj58 2023-09-22T22:46:59.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj58 2023-09-22T22:46:59.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj58 2023-09-22T22:46:59.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.706 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.706 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.706 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.707 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.707 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.707 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:HbElvGukCjTOKLFNPpD6yC9cRwc= 2023-09-22T22:46:59.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HbElvGukCjTOKLFNPpD6yC9cRwc=', 'Content-Length': '0'} 2023-09-22T22:46:59.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072e8047b715e3c4e-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj58 2023-09-22T22:46:59.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj58 2023-09-22T22:46:59.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.708 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.708 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.708 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.708 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.709 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj58?versionId=null 2023-09-22T22:46:59.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.709 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:HbElvGukCjTOKLFNPpD6yC9cRwc= 2023-09-22T22:46:59.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:HbElvGukCjTOKLFNPpD6yC9cRwc=', 'Content-Length': '0'} 2023-09-22T22:46:59.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4e6446e5f97be4f-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj58 2023-09-22T22:46:59.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj58 2023-09-22T22:46:59.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj58 2023-09-22T22:46:59.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.710 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.710 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.710 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.710 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.711 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.711 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:46:59.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:46:59.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e74436b4c1be076-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj581000falseobj582023-09-22T22:05:15.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj58 2023-09-22T22:46:59.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj58 2023-09-22T22:46:59.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj58 2023-09-22T22:46:59.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.712 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.712 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.712 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.712 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.713 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.713 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:46:59.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:46:59.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcd014ef0edf6a98-00650e0fb5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj581000falseobj582023-09-22T22:05:15.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj58 2023-09-22T22:46:59.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj58 2023-09-22T22:46:59.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj58 2023-09-22T22:46:59.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.714 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.714 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.714 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.714 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.714 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj58 2023-09-22T22:46:59.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.715 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:h0+BDr4RhYNH8kxT8vIN6L410WI= 2023-09-22T22:46:59.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h0+BDr4RhYNH8kxT8vIN6L410WI=', 'Content-Length': '0'} 2023-09-22T22:46:59.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b1baaa9fe56bdc8-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj58 2023-09-22T22:46:59.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj58 2023-09-22T22:46:59.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj58 2023-09-22T22:46:59.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.716 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.716 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.716 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.716 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.716 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj58 2023-09-22T22:46:59.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.716 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:h0+BDr4RhYNH8kxT8vIN6L410WI= 2023-09-22T22:46:59.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h0+BDr4RhYNH8kxT8vIN6L410WI=', 'Content-Length': '0'} 2023-09-22T22:46:59.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007536c4fae8a32207-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.716 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj58 2023-09-22T22:46:59.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj58 2023-09-22T22:46:59.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj58 2023-09-22T22:46:59.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj58 2023-09-22T22:46:59.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.717 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.717 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.718 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.718 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.718 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj58 2023-09-22T22:46:59.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.718 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:f6ieXfrzUXpJFKJBmm4QnzeNHD8= 2023-09-22T22:46:59.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f6ieXfrzUXpJFKJBmm4QnzeNHD8=', 'Content-Length': '0'} 2023-09-22T22:46:59.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006edf53d85477c00d-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj58 2023-09-22T22:46:59.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj58 2023-09-22T22:46:59.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj58 2023-09-22T22:46:59.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.719 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.719 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.719 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.719 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.720 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj58 2023-09-22T22:46:59.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.720 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:f6ieXfrzUXpJFKJBmm4QnzeNHD8= 2023-09-22T22:46:59.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f6ieXfrzUXpJFKJBmm4QnzeNHD8=', 'Content-Length': '0'} 2023-09-22T22:46:59.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045570e9517a1c3ca-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj58 2023-09-22T22:46:59.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj58 2023-09-22T22:46:59.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj58 2023-09-22T22:46:59.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.721 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.721 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.721 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.721 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.721 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.721 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:46:59.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:46:59.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000122a84c4823b4295-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj581000falseobj582023-09-22T22:05:15.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj58 2023-09-22T22:46:59.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj58 2023-09-22T22:46:59.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj58 2023-09-22T22:46:59.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.723 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.723 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.723 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.723 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.723 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.724 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:46:59.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:46:59.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030d81954336e03ad-00650e0fb5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj581000falseobj582023-09-22T22:05:15.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.724 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj62 2023-09-22T22:46:59.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj62 2023-09-22T22:46:59.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj62 2023-09-22T22:46:59.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.725 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.725 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.725 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.725 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.725 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.726 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:wbDUsltvuvF9UTqNaGPyZLId0VU= 2023-09-22T22:46:59.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wbDUsltvuvF9UTqNaGPyZLId0VU=', 'Content-Length': '0'} 2023-09-22T22:46:59.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a24e759710dcf652-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj62 2023-09-22T22:46:59.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj62 2023-09-22T22:46:59.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.727 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.727 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.727 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.727 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.727 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.727 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:wbDUsltvuvF9UTqNaGPyZLId0VU= 2023-09-22T22:46:59.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wbDUsltvuvF9UTqNaGPyZLId0VU=', 'Content-Length': '0'} 2023-09-22T22:46:59.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034283780a8078570-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj62 2023-09-22T22:46:59.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj62 2023-09-22T22:46:59.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.729 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.729 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.729 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.729 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.729 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.729 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:hSqI/+W2+leER6HP7uWrcJCMklo= 2023-09-22T22:46:59.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hSqI/+W2+leER6HP7uWrcJCMklo=', 'Content-Length': '0'} 2023-09-22T22:46:59.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089f0b8fc0e40c5e0-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj62 2023-09-22T22:46:59.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj62 2023-09-22T22:46:59.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.731 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.731 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.731 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.731 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.731 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.731 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:hSqI/+W2+leER6HP7uWrcJCMklo= 2023-09-22T22:46:59.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hSqI/+W2+leER6HP7uWrcJCMklo=', 'Content-Length': '0'} 2023-09-22T22:46:59.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005282fd0ae15b2169-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.731 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj62 2023-09-22T22:46:59.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj62 2023-09-22T22:46:59.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj62 2023-09-22T22:46:59.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.732 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.733 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.733 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.733 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.733 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.733 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:wbDUsltvuvF9UTqNaGPyZLId0VU= 2023-09-22T22:46:59.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wbDUsltvuvF9UTqNaGPyZLId0VU=', 'Content-Length': '0'} 2023-09-22T22:46:59.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b01a68f56b5ab94a-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj62 2023-09-22T22:46:59.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj62 2023-09-22T22:46:59.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.734 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.734 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.735 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.735 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.735 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj62?versionId=null 2023-09-22T22:46:59.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.735 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:wbDUsltvuvF9UTqNaGPyZLId0VU= 2023-09-22T22:46:59.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wbDUsltvuvF9UTqNaGPyZLId0VU=', 'Content-Length': '0'} 2023-09-22T22:46:59.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012bbc9898269f31c-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj62 2023-09-22T22:46:59.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj62 2023-09-22T22:46:59.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj62 2023-09-22T22:46:59.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.736 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.736 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.736 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.736 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.737 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.737 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:46:59.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:46:59.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dd73cf56469f1c8-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj621000falseobj622023-09-22T22:05:15.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj62 2023-09-22T22:46:59.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj62 2023-09-22T22:46:59.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj62 2023-09-22T22:46:59.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.738 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.738 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.738 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.738 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.738 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.739 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:46:59.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:46:59.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060e0bbcb811116f0-00650e0fb5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj621000falseobj622023-09-22T22:05:15.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj62 2023-09-22T22:46:59.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj62 2023-09-22T22:46:59.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj62 2023-09-22T22:46:59.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.740 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.740 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.740 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.740 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.740 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj62 2023-09-22T22:46:59.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.740 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:OAIpxDfV7JfYRtQO4F4/8FNuutU= 2023-09-22T22:46:59.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OAIpxDfV7JfYRtQO4F4/8FNuutU=', 'Content-Length': '0'} 2023-09-22T22:46:59.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022a7133c83242bd4-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj62 2023-09-22T22:46:59.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj62 2023-09-22T22:46:59.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj62 2023-09-22T22:46:59.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.741 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.742 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.742 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.742 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.742 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj62 2023-09-22T22:46:59.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.742 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:OAIpxDfV7JfYRtQO4F4/8FNuutU= 2023-09-22T22:46:59.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OAIpxDfV7JfYRtQO4F4/8FNuutU=', 'Content-Length': '0'} 2023-09-22T22:46:59.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa41ac04520480fb-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.742 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj62 2023-09-22T22:46:59.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj62 2023-09-22T22:46:59.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj62 2023-09-22T22:46:59.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj62 2023-09-22T22:46:59.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.743 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.743 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.743 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.744 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.744 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj62 2023-09-22T22:46:59.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.744 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:AiDD/6aEAw13Up4dF71VPs9J60A= 2023-09-22T22:46:59.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AiDD/6aEAw13Up4dF71VPs9J60A=', 'Content-Length': '0'} 2023-09-22T22:46:59.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006d89a03516c5689-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj62 2023-09-22T22:46:59.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj62 2023-09-22T22:46:59.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj62 2023-09-22T22:46:59.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.745 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.745 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.745 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.745 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.745 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj62 2023-09-22T22:46:59.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.746 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:AiDD/6aEAw13Up4dF71VPs9J60A= 2023-09-22T22:46:59.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AiDD/6aEAw13Up4dF71VPs9J60A=', 'Content-Length': '0'} 2023-09-22T22:46:59.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a40a82cfcbfcb3ee-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj62 2023-09-22T22:46:59.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj62 2023-09-22T22:46:59.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj62 2023-09-22T22:46:59.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.747 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.747 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.747 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.747 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.747 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.747 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:46:59.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:46:59.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cef660f9e6db5b0b-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj621000falseobj622023-09-22T22:05:15.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj62 2023-09-22T22:46:59.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj62 2023-09-22T22:46:59.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj62 2023-09-22T22:46:59.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.749 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.749 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.749 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.749 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.749 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.749 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:46:59.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:46:59.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e685ec332aad672-00650e0fb5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj621000falseobj622023-09-22T22:05:15.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.750 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj68 2023-09-22T22:46:59.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj68 2023-09-22T22:46:59.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj68 2023-09-22T22:46:59.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.751 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.751 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.751 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.751 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.751 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.751 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:FvITgYTqoW0R9xO0TwNclNZrNFM= 2023-09-22T22:46:59.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FvITgYTqoW0R9xO0TwNclNZrNFM=', 'Content-Length': '0'} 2023-09-22T22:46:59.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b6e27662c02f00b-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj68 2023-09-22T22:46:59.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj68 2023-09-22T22:46:59.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.752 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.752 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.753 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.753 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.753 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.753 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:FvITgYTqoW0R9xO0TwNclNZrNFM= 2023-09-22T22:46:59.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FvITgYTqoW0R9xO0TwNclNZrNFM=', 'Content-Length': '0'} 2023-09-22T22:46:59.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000989f29a810556aa7-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj68 2023-09-22T22:46:59.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj68 2023-09-22T22:46:59.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.754 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.754 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.754 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.755 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.755 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.755 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:jCI47/21CweQP6sNI/kiuQzkkWQ= 2023-09-22T22:46:59.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jCI47/21CweQP6sNI/kiuQzkkWQ=', 'Content-Length': '0'} 2023-09-22T22:46:59.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4e0688f1e6333a6-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj68 2023-09-22T22:46:59.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj68 2023-09-22T22:46:59.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.756 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.756 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.756 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.756 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.757 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.757 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:jCI47/21CweQP6sNI/kiuQzkkWQ= 2023-09-22T22:46:59.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jCI47/21CweQP6sNI/kiuQzkkWQ=', 'Content-Length': '0'} 2023-09-22T22:46:59.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001814f87731355365-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.757 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj68 2023-09-22T22:46:59.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj68 2023-09-22T22:46:59.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj68 2023-09-22T22:46:59.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.758 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.758 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.758 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.758 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.759 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.759 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:FvITgYTqoW0R9xO0TwNclNZrNFM= 2023-09-22T22:46:59.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FvITgYTqoW0R9xO0TwNclNZrNFM=', 'Content-Length': '0'} 2023-09-22T22:46:59.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaa4341dda2526e5-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj68 2023-09-22T22:46:59.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj68 2023-09-22T22:46:59.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.760 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.760 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.760 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.760 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.760 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj68?versionId=null 2023-09-22T22:46:59.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.761 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:FvITgYTqoW0R9xO0TwNclNZrNFM= 2023-09-22T22:46:59.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:FvITgYTqoW0R9xO0TwNclNZrNFM=', 'Content-Length': '0'} 2023-09-22T22:46:59.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b335ae60c5c2d00d-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj68 2023-09-22T22:46:59.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj68 2023-09-22T22:46:59.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj68 2023-09-22T22:46:59.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.762 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.762 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.762 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.762 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.762 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.762 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:46:59.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:46:59.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029cb7e95b699c5e1-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj681000falseobj682023-09-22T22:05:14.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj68 2023-09-22T22:46:59.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj68 2023-09-22T22:46:59.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj68 2023-09-22T22:46:59.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.764 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.764 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.764 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.764 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.764 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.765 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:46:59.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:46:59.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000243fb6129496c153-00650e0fb5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj681000falseobj682023-09-22T22:05:14.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj68 2023-09-22T22:46:59.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj68 2023-09-22T22:46:59.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj68 2023-09-22T22:46:59.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.766 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.766 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.766 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.766 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.766 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj68 2023-09-22T22:46:59.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.766 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:WAx3PX9RhdxC2kGJX4HxiNImGRs= 2023-09-22T22:46:59.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WAx3PX9RhdxC2kGJX4HxiNImGRs=', 'Content-Length': '0'} 2023-09-22T22:46:59.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000118af6acae65e701-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj68 2023-09-22T22:46:59.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj68 2023-09-22T22:46:59.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj68 2023-09-22T22:46:59.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.768 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.768 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.768 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.768 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.768 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj68 2023-09-22T22:46:59.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.768 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:WAx3PX9RhdxC2kGJX4HxiNImGRs= 2023-09-22T22:46:59.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WAx3PX9RhdxC2kGJX4HxiNImGRs=', 'Content-Length': '0'} 2023-09-22T22:46:59.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003042b73f170eea02-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.768 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj68 2023-09-22T22:46:59.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj68 2023-09-22T22:46:59.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj68 2023-09-22T22:46:59.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj68 2023-09-22T22:46:59.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.769 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.769 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.770 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.770 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.770 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj68 2023-09-22T22:46:59.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.770 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:owXkfr9fm1zG6i3JCGUj3vvou1Q= 2023-09-22T22:46:59.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:owXkfr9fm1zG6i3JCGUj3vvou1Q=', 'Content-Length': '0'} 2023-09-22T22:46:59.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000320ea172f4d93ae2-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj68 2023-09-22T22:46:59.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj68 2023-09-22T22:46:59.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj68 2023-09-22T22:46:59.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.771 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.771 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.771 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.771 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.771 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj68 2023-09-22T22:46:59.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.772 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:owXkfr9fm1zG6i3JCGUj3vvou1Q= 2023-09-22T22:46:59.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:owXkfr9fm1zG6i3JCGUj3vvou1Q=', 'Content-Length': '0'} 2023-09-22T22:46:59.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2f69cbd8d819fbf-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj68 2023-09-22T22:46:59.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj68 2023-09-22T22:46:59.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj68 2023-09-22T22:46:59.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.773 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.773 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.773 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.773 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.773 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.774 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:46:59.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:46:59.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7dcdeafabcf38a9-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj681000falseobj682023-09-22T22:05:14.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj68 2023-09-22T22:46:59.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj68 2023-09-22T22:46:59.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj68 2023-09-22T22:46:59.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.775 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.775 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.775 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.775 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.775 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.776 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE= 2023-09-22T22:46:59.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oImRdL87wOqjZ1olKovZcm5LGHE=', 'Content-Length': '0'} 2023-09-22T22:46:59.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af3dd1b84a32c9c6-00650e0fb5-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj681000falseobj682023-09-22T22:05:14.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.776 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj69 2023-09-22T22:46:59.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj69 2023-09-22T22:46:59.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj69 2023-09-22T22:46:59.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.777 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.777 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.777 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.777 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.777 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.778 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:7TUY0Fp5mj5/UaiaSX8JIZVGOrg= 2023-09-22T22:46:59.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7TUY0Fp5mj5/UaiaSX8JIZVGOrg=', 'Content-Length': '0'} 2023-09-22T22:46:59.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3836e6d25837e6c-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj69 2023-09-22T22:46:59.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj69 2023-09-22T22:46:59.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.779 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.779 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.779 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.779 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.779 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.780 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:7TUY0Fp5mj5/UaiaSX8JIZVGOrg= 2023-09-22T22:46:59.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7TUY0Fp5mj5/UaiaSX8JIZVGOrg=', 'Content-Length': '0'} 2023-09-22T22:46:59.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000191b24d825b7629-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj69 2023-09-22T22:46:59.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj69 2023-09-22T22:46:59.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.781 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.781 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.781 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.781 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.781 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.781 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:GRhT5FWesUOzCSxYZVCzfPcOSzQ= 2023-09-22T22:46:59.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GRhT5FWesUOzCSxYZVCzfPcOSzQ=', 'Content-Length': '0'} 2023-09-22T22:46:59.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b08ffedab928b3e-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj69 2023-09-22T22:46:59.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj69 2023-09-22T22:46:59.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.783 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.783 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.783 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.783 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.783 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.783 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:GRhT5FWesUOzCSxYZVCzfPcOSzQ= 2023-09-22T22:46:59.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:GRhT5FWesUOzCSxYZVCzfPcOSzQ=', 'Content-Length': '0'} 2023-09-22T22:46:59.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b82628b2c1f4349-00650e0fb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.784 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj69 2023-09-22T22:46:59.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj69 2023-09-22T22:46:59.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj69 2023-09-22T22:46:59.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.785 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.785 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.785 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.785 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:41 GMT 2023-09-22T22:46:59.785 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.785 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:7TUY0Fp5mj5/UaiaSX8JIZVGOrg= 2023-09-22T22:46:59.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:41 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7TUY0Fp5mj5/UaiaSX8JIZVGOrg=', 'Content-Length': '0'} 2023-09-22T22:46:59.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd354ff5395730db-00650e0fb5-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj69 2023-09-22T22:46:59.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj69 2023-09-22T22:46:59.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.787 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.787 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.787 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.787 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.787 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj69?versionId=null 2023-09-22T22:46:59.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.787 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:PguPhR+3E82NW/SdNY4vTsfm8wA= 2023-09-22T22:46:59.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:PguPhR+3E82NW/SdNY4vTsfm8wA=', 'Content-Length': '0'} 2023-09-22T22:46:59.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058e90bedf4799e97-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj69 2023-09-22T22:46:59.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj69 2023-09-22T22:46:59.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj69 2023-09-22T22:46:59.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.789 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.789 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.789 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.789 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.789 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.789 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:46:59.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:46:59.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046d2941ebaabbb8e-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj691000falseobj692023-09-22T22:05:14.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj69 2023-09-22T22:46:59.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj69 2023-09-22T22:46:59.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj69 2023-09-22T22:46:59.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.791 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.791 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.791 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.791 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.791 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.791 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:46:59.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:46:59.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000215fc50e2785c593-00650e0fb6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj691000falseobj692023-09-22T22:05:14.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj69 2023-09-22T22:46:59.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj69 2023-09-22T22:46:59.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj69 2023-09-22T22:46:59.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.792 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.792 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.793 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.793 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.793 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj69 2023-09-22T22:46:59.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.793 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:XuZpWUYn0ag7tSzsAEQNYwdaRcQ= 2023-09-22T22:46:59.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XuZpWUYn0ag7tSzsAEQNYwdaRcQ=', 'Content-Length': '0'} 2023-09-22T22:46:59.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c970e6d100ea051-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj69 2023-09-22T22:46:59.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj69 2023-09-22T22:46:59.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj69 2023-09-22T22:46:59.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.794 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.794 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.794 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.794 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.794 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj69 2023-09-22T22:46:59.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.795 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:XuZpWUYn0ag7tSzsAEQNYwdaRcQ= 2023-09-22T22:46:59.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XuZpWUYn0ag7tSzsAEQNYwdaRcQ=', 'Content-Length': '0'} 2023-09-22T22:46:59.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e7f5053d6e98b65-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.795 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj69 2023-09-22T22:46:59.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj69 2023-09-22T22:46:59.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj69 2023-09-22T22:46:59.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj69 2023-09-22T22:46:59.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.796 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.796 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.796 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.796 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.796 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj69 2023-09-22T22:46:59.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.796 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:ZIdMz94KnwG7/RbWGfTl2jQr12E= 2023-09-22T22:46:59.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZIdMz94KnwG7/RbWGfTl2jQr12E=', 'Content-Length': '0'} 2023-09-22T22:46:59.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087b76c4e17738f4c-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj69 2023-09-22T22:46:59.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj69 2023-09-22T22:46:59.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj69 2023-09-22T22:46:59.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.798 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.798 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.798 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.798 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.798 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj69 2023-09-22T22:46:59.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.798 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:ZIdMz94KnwG7/RbWGfTl2jQr12E= 2023-09-22T22:46:59.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ZIdMz94KnwG7/RbWGfTl2jQr12E=', 'Content-Length': '0'} 2023-09-22T22:46:59.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a9fc905960e9b1b-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj69 2023-09-22T22:46:59.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj69 2023-09-22T22:46:59.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj69 2023-09-22T22:46:59.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.800 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.800 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.800 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.800 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.800 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.800 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:46:59.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:46:59.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e7efd0b0c33d1d2-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj691000falseobj692023-09-22T22:05:14.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj69 2023-09-22T22:46:59.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj69 2023-09-22T22:46:59.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj69 2023-09-22T22:46:59.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.802 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.802 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.802 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.802 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.802 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.802 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:46:59.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:46:59.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d98b473427b5e24f-00650e0fb6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj691000falseobj692023-09-22T22:05:14.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.803 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj74 2023-09-22T22:46:59.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj74 2023-09-22T22:46:59.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj74 2023-09-22T22:46:59.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.804 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.804 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.804 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.804 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.804 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.804 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:tPNM71+HlmzSKA7c15maruaCOY8= 2023-09-22T22:46:59.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tPNM71+HlmzSKA7c15maruaCOY8=', 'Content-Length': '0'} 2023-09-22T22:46:59.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000423db6dd3a6b3d6c-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj74 2023-09-22T22:46:59.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj74 2023-09-22T22:46:59.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.806 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.806 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.806 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.806 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.806 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.806 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:tPNM71+HlmzSKA7c15maruaCOY8= 2023-09-22T22:46:59.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tPNM71+HlmzSKA7c15maruaCOY8=', 'Content-Length': '0'} 2023-09-22T22:46:59.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2e1ddc7c9ca387d-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj74 2023-09-22T22:46:59.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj74 2023-09-22T22:46:59.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.808 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.808 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.808 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.808 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.808 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.808 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:/v7h9/hPrOGDciltnsOJT2B7UJo= 2023-09-22T22:46:59.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/v7h9/hPrOGDciltnsOJT2B7UJo=', 'Content-Length': '0'} 2023-09-22T22:46:59.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000954fa1fa68ea669b-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj74 2023-09-22T22:46:59.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj74 2023-09-22T22:46:59.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.810 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.810 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.810 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.810 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.810 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.810 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:/v7h9/hPrOGDciltnsOJT2B7UJo= 2023-09-22T22:46:59.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/v7h9/hPrOGDciltnsOJT2B7UJo=', 'Content-Length': '0'} 2023-09-22T22:46:59.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feb9b46e29068adc-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.811 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj74 2023-09-22T22:46:59.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj74 2023-09-22T22:46:59.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj74 2023-09-22T22:46:59.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.812 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.812 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.812 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.812 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.812 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.812 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:tPNM71+HlmzSKA7c15maruaCOY8= 2023-09-22T22:46:59.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tPNM71+HlmzSKA7c15maruaCOY8=', 'Content-Length': '0'} 2023-09-22T22:46:59.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033528eba14e727b4-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj74 2023-09-22T22:46:59.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj74 2023-09-22T22:46:59.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.814 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.814 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.814 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.814 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.814 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj74?versionId=null 2023-09-22T22:46:59.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.814 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:tPNM71+HlmzSKA7c15maruaCOY8= 2023-09-22T22:46:59.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:tPNM71+HlmzSKA7c15maruaCOY8=', 'Content-Length': '0'} 2023-09-22T22:46:59.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b4931cbd7a633cb-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj74 2023-09-22T22:46:59.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj74 2023-09-22T22:46:59.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj74 2023-09-22T22:46:59.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.816 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.816 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.816 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.816 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.816 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.816 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:46:59.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:46:59.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f548e7e535e5b0a-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj741000falseobj742023-09-22T22:05:15.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj74 2023-09-22T22:46:59.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj74 2023-09-22T22:46:59.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj74 2023-09-22T22:46:59.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.818 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.818 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.818 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.818 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.818 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.818 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:46:59.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:46:59.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006846d15f619e174f-00650e0fb6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj741000falseobj742023-09-22T22:05:15.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj74 2023-09-22T22:46:59.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj74 2023-09-22T22:46:59.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj74 2023-09-22T22:46:59.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.820 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.820 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.820 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.820 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.820 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj74 2023-09-22T22:46:59.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.820 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:flTxT3yLAOliRRvPQBmCso8P5IY= 2023-09-22T22:46:59.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:flTxT3yLAOliRRvPQBmCso8P5IY=', 'Content-Length': '0'} 2023-09-22T22:46:59.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067b073d144472003-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj74 2023-09-22T22:46:59.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj74 2023-09-22T22:46:59.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj74 2023-09-22T22:46:59.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.821 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.821 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.821 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.822 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.822 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj74 2023-09-22T22:46:59.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.822 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:flTxT3yLAOliRRvPQBmCso8P5IY= 2023-09-22T22:46:59.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:flTxT3yLAOliRRvPQBmCso8P5IY=', 'Content-Length': '0'} 2023-09-22T22:46:59.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086ac682da727bb55-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.822 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj74 2023-09-22T22:46:59.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj74 2023-09-22T22:46:59.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj74 2023-09-22T22:46:59.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj74 2023-09-22T22:46:59.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.823 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.823 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.823 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.823 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.824 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj74 2023-09-22T22:46:59.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.824 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:In1MH66XZQ53RNE9LoyKTYvUUYw= 2023-09-22T22:46:59.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:In1MH66XZQ53RNE9LoyKTYvUUYw=', 'Content-Length': '0'} 2023-09-22T22:46:59.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4204d5b18eeb341-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj74 2023-09-22T22:46:59.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj74 2023-09-22T22:46:59.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj74 2023-09-22T22:46:59.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.825 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.825 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.825 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.825 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.825 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj74 2023-09-22T22:46:59.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.825 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:In1MH66XZQ53RNE9LoyKTYvUUYw= 2023-09-22T22:46:59.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:In1MH66XZQ53RNE9LoyKTYvUUYw=', 'Content-Length': '0'} 2023-09-22T22:46:59.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003d75cf950d49090-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj74 2023-09-22T22:46:59.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj74 2023-09-22T22:46:59.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj74 2023-09-22T22:46:59.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.827 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.827 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.827 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.827 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.827 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.827 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:46:59.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:46:59.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd4c5ac36a87aa15-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj741000falseobj742023-09-22T22:05:15.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj74 2023-09-22T22:46:59.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj74 2023-09-22T22:46:59.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj74 2023-09-22T22:46:59.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.829 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.829 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.829 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.829 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.829 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.829 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:46:59.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:46:59.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0adfe3f1b698438-00650e0fb6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj741000falseobj742023-09-22T22:05:15.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.830 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj76 2023-09-22T22:46:59.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj76 2023-09-22T22:46:59.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj76 2023-09-22T22:46:59.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.831 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.831 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.831 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.831 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.831 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.831 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:XhtPes5kmI6PMA9ufpMHuL+MZ3g= 2023-09-22T22:46:59.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XhtPes5kmI6PMA9ufpMHuL+MZ3g=', 'Content-Length': '0'} 2023-09-22T22:46:59.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001aa8584759d67829-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj76 2023-09-22T22:46:59.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj76 2023-09-22T22:46:59.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.833 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.833 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.833 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.833 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.833 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.833 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:XhtPes5kmI6PMA9ufpMHuL+MZ3g= 2023-09-22T22:46:59.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XhtPes5kmI6PMA9ufpMHuL+MZ3g=', 'Content-Length': '0'} 2023-09-22T22:46:59.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a59b793eedff0045-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj76 2023-09-22T22:46:59.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj76 2023-09-22T22:46:59.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.835 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.835 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.835 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.835 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.835 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.835 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:iOSpS1HmLIA1O66s6b3jyn4GP/I= 2023-09-22T22:46:59.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iOSpS1HmLIA1O66s6b3jyn4GP/I=', 'Content-Length': '0'} 2023-09-22T22:46:59.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000370ffb98985e7f5b-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj76 2023-09-22T22:46:59.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj76 2023-09-22T22:46:59.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.837 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.837 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.837 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.837 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.837 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.837 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:iOSpS1HmLIA1O66s6b3jyn4GP/I= 2023-09-22T22:46:59.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:iOSpS1HmLIA1O66s6b3jyn4GP/I=', 'Content-Length': '0'} 2023-09-22T22:46:59.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077b4ac220e50eadb-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.837 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj76 2023-09-22T22:46:59.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj76 2023-09-22T22:46:59.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj76 2023-09-22T22:46:59.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.839 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.839 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.839 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.839 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.839 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.839 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:XhtPes5kmI6PMA9ufpMHuL+MZ3g= 2023-09-22T22:46:59.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XhtPes5kmI6PMA9ufpMHuL+MZ3g=', 'Content-Length': '0'} 2023-09-22T22:46:59.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d13defd02da2293d-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj76 2023-09-22T22:46:59.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj76 2023-09-22T22:46:59.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.841 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.841 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.841 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.841 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.841 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj76?versionId=null 2023-09-22T22:46:59.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.841 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:XhtPes5kmI6PMA9ufpMHuL+MZ3g= 2023-09-22T22:46:59.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XhtPes5kmI6PMA9ufpMHuL+MZ3g=', 'Content-Length': '0'} 2023-09-22T22:46:59.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fc2f81b735439dd-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj76 2023-09-22T22:46:59.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj76 2023-09-22T22:46:59.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj76 2023-09-22T22:46:59.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.843 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.843 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.843 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.843 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.843 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.843 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:46:59.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:46:59.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e3eaabaef7bbdc9-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj761000falseobj762023-09-22T22:05:14.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj76 2023-09-22T22:46:59.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj76 2023-09-22T22:46:59.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj76 2023-09-22T22:46:59.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.845 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.845 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.845 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.845 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.845 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.845 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:46:59.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:46:59.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5fd959e7ee0b6e4-00650e0fb6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj761000falseobj762023-09-22T22:05:14.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj76 2023-09-22T22:46:59.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj76 2023-09-22T22:46:59.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj76 2023-09-22T22:46:59.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.847 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.847 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.847 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.847 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.847 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj76 2023-09-22T22:46:59.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.847 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:RlF3qM95snPe8w86eFgpkp8eTts= 2023-09-22T22:46:59.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RlF3qM95snPe8w86eFgpkp8eTts=', 'Content-Length': '0'} 2023-09-22T22:46:59.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000829df133941764e8-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj76 2023-09-22T22:46:59.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj76 2023-09-22T22:46:59.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj76 2023-09-22T22:46:59.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.848 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.848 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.848 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.849 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.849 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj76 2023-09-22T22:46:59.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.849 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:RlF3qM95snPe8w86eFgpkp8eTts= 2023-09-22T22:46:59.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RlF3qM95snPe8w86eFgpkp8eTts=', 'Content-Length': '0'} 2023-09-22T22:46:59.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055765f2fc6ee7222-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.849 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj76 2023-09-22T22:46:59.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj76 2023-09-22T22:46:59.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj76 2023-09-22T22:46:59.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj76 2023-09-22T22:46:59.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.850 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.850 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.850 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.850 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.850 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj76 2023-09-22T22:46:59.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.851 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:UmkDJ7t6hLn0Uubp2FtPINP5hCg= 2023-09-22T22:46:59.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UmkDJ7t6hLn0Uubp2FtPINP5hCg=', 'Content-Length': '0'} 2023-09-22T22:46:59.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d72383635c8e594-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj76 2023-09-22T22:46:59.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj76 2023-09-22T22:46:59.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj76 2023-09-22T22:46:59.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.852 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.852 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.852 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.852 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.852 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj76 2023-09-22T22:46:59.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.852 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:UmkDJ7t6hLn0Uubp2FtPINP5hCg= 2023-09-22T22:46:59.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:UmkDJ7t6hLn0Uubp2FtPINP5hCg=', 'Content-Length': '0'} 2023-09-22T22:46:59.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b2f54e1541fb448-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj76 2023-09-22T22:46:59.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj76 2023-09-22T22:46:59.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj76 2023-09-22T22:46:59.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.854 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.854 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.854 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.854 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.854 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.854 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:46:59.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:46:59.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f22fc0de71a74d2-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj761000falseobj762023-09-22T22:05:14.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj76 2023-09-22T22:46:59.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj76 2023-09-22T22:46:59.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj76 2023-09-22T22:46:59.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.856 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.856 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.856 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.856 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.856 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.856 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:46:59.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:46:59.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d71e533cb37437a-00650e0fb6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj761000falseobj762023-09-22T22:05:14.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.857 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj85 2023-09-22T22:46:59.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj85 2023-09-22T22:46:59.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj85 2023-09-22T22:46:59.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.858 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.858 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.858 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.858 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.858 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.858 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:XFiKSnT2fI/EJYAJdbjDlH7zFxc= 2023-09-22T22:46:59.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XFiKSnT2fI/EJYAJdbjDlH7zFxc=', 'Content-Length': '0'} 2023-09-22T22:46:59.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a6859daa8257774-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj85 2023-09-22T22:46:59.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj85 2023-09-22T22:46:59.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.861 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.861 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.861 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.861 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.861 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.862 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:XFiKSnT2fI/EJYAJdbjDlH7zFxc= 2023-09-22T22:46:59.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XFiKSnT2fI/EJYAJdbjDlH7zFxc=', 'Content-Length': '0'} 2023-09-22T22:46:59.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d953d6a8b92f8ee-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj85 2023-09-22T22:46:59.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj85 2023-09-22T22:46:59.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.863 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.863 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.863 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.863 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.863 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.864 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:lEVWTlsW//gSpxH99f5L91P5uF4= 2023-09-22T22:46:59.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lEVWTlsW//gSpxH99f5L91P5uF4=', 'Content-Length': '0'} 2023-09-22T22:46:59.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c4f9a920b1d1b8b-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj85 2023-09-22T22:46:59.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj85 2023-09-22T22:46:59.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.865 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.865 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.866 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.866 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.866 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.866 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:lEVWTlsW//gSpxH99f5L91P5uF4= 2023-09-22T22:46:59.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:lEVWTlsW//gSpxH99f5L91P5uF4=', 'Content-Length': '0'} 2023-09-22T22:46:59.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000961e4470e1c80130-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.867 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj85 2023-09-22T22:46:59.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj85 2023-09-22T22:46:59.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj85 2023-09-22T22:46:59.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.868 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.868 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.868 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.868 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.868 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.868 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:XFiKSnT2fI/EJYAJdbjDlH7zFxc= 2023-09-22T22:46:59.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XFiKSnT2fI/EJYAJdbjDlH7zFxc=', 'Content-Length': '0'} 2023-09-22T22:46:59.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000318e6f3436147728-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj85 2023-09-22T22:46:59.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj85 2023-09-22T22:46:59.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.870 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.870 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.870 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.870 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.870 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj85?versionId=null 2023-09-22T22:46:59.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.870 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:XFiKSnT2fI/EJYAJdbjDlH7zFxc= 2023-09-22T22:46:59.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XFiKSnT2fI/EJYAJdbjDlH7zFxc=', 'Content-Length': '0'} 2023-09-22T22:46:59.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7ac7096d579a0d5-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj85 2023-09-22T22:46:59.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj85 2023-09-22T22:46:59.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj85 2023-09-22T22:46:59.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.872 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.872 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.872 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.872 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.872 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.872 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:46:59.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:46:59.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053f1f08f920a54ae-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj851000falseobj852023-09-22T22:05:14.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj85 2023-09-22T22:46:59.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj85 2023-09-22T22:46:59.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj85 2023-09-22T22:46:59.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.874 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.874 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.874 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.874 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.874 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.874 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:46:59.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:46:59.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b731c1dd6abe5aa-00650e0fb6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj851000falseobj852023-09-22T22:05:14.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj85 2023-09-22T22:46:59.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj85 2023-09-22T22:46:59.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj85 2023-09-22T22:46:59.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.876 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.876 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.876 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.876 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.876 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj85 2023-09-22T22:46:59.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.876 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:QwbVYsdT6+OATixnENPz646WkV0= 2023-09-22T22:46:59.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QwbVYsdT6+OATixnENPz646WkV0=', 'Content-Length': '0'} 2023-09-22T22:46:59.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df54bcd96b2558f0-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj85 2023-09-22T22:46:59.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj85 2023-09-22T22:46:59.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj85 2023-09-22T22:46:59.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.878 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.878 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.878 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.878 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.878 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj85 2023-09-22T22:46:59.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.878 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:QwbVYsdT6+OATixnENPz646WkV0= 2023-09-22T22:46:59.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QwbVYsdT6+OATixnENPz646WkV0=', 'Content-Length': '0'} 2023-09-22T22:46:59.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b183eab5d148f93-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.878 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj85 2023-09-22T22:46:59.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj85 2023-09-22T22:46:59.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj85 2023-09-22T22:46:59.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj85 2023-09-22T22:46:59.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.879 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.880 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.880 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.880 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.880 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj85 2023-09-22T22:46:59.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.880 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:hLkorFe6avqqyK+UdVJFyaKzO5Q= 2023-09-22T22:46:59.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hLkorFe6avqqyK+UdVJFyaKzO5Q=', 'Content-Length': '0'} 2023-09-22T22:46:59.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f44d381b220bd587-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj85 2023-09-22T22:46:59.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj85 2023-09-22T22:46:59.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj85 2023-09-22T22:46:59.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.881 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.881 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.881 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.882 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.882 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj85 2023-09-22T22:46:59.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.882 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:hLkorFe6avqqyK+UdVJFyaKzO5Q= 2023-09-22T22:46:59.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hLkorFe6avqqyK+UdVJFyaKzO5Q=', 'Content-Length': '0'} 2023-09-22T22:46:59.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008515c7831d6608eb-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj85 2023-09-22T22:46:59.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj85 2023-09-22T22:46:59.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj85 2023-09-22T22:46:59.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.883 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.884 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.884 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.884 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.884 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.884 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:46:59.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:46:59.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb55de93f7bbd64f-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj851000falseobj852023-09-22T22:05:14.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj85 2023-09-22T22:46:59.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj85 2023-09-22T22:46:59.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj85 2023-09-22T22:46:59.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.886 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.886 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.886 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.886 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.886 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.886 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:46:59.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:46:59.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b9dc39856b34a36-00650e0fb6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj851000falseobj852023-09-22T22:05:14.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.887 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj87 2023-09-22T22:46:59.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj87 2023-09-22T22:46:59.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj87 2023-09-22T22:46:59.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.888 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.888 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.888 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.888 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.888 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.888 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:CKfJelByYNISJSHaqmuJh6qp0Nk= 2023-09-22T22:46:59.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CKfJelByYNISJSHaqmuJh6qp0Nk=', 'Content-Length': '0'} 2023-09-22T22:46:59.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7109c6feadd5157-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj87 2023-09-22T22:46:59.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj87 2023-09-22T22:46:59.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.890 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.890 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.890 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.890 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.890 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.891 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:CKfJelByYNISJSHaqmuJh6qp0Nk= 2023-09-22T22:46:59.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CKfJelByYNISJSHaqmuJh6qp0Nk=', 'Content-Length': '0'} 2023-09-22T22:46:59.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056d7a94cf5fab6ee-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj87 2023-09-22T22:46:59.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj87 2023-09-22T22:46:59.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.892 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.892 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.892 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.892 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.892 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.893 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:wtj2Nxk9QvK/yIRVIwi57snoMlY= 2023-09-22T22:46:59.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wtj2Nxk9QvK/yIRVIwi57snoMlY=', 'Content-Length': '0'} 2023-09-22T22:46:59.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000602e4d46feef14c6-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj87 2023-09-22T22:46:59.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj87 2023-09-22T22:46:59.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.894 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.894 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.894 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.894 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.894 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.895 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:wtj2Nxk9QvK/yIRVIwi57snoMlY= 2023-09-22T22:46:59.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wtj2Nxk9QvK/yIRVIwi57snoMlY=', 'Content-Length': '0'} 2023-09-22T22:46:59.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dcc488c062df2dd-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.895 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj87 2023-09-22T22:46:59.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj87 2023-09-22T22:46:59.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj87 2023-09-22T22:46:59.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.896 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.896 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.896 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.896 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.896 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.897 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:CKfJelByYNISJSHaqmuJh6qp0Nk= 2023-09-22T22:46:59.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CKfJelByYNISJSHaqmuJh6qp0Nk=', 'Content-Length': '0'} 2023-09-22T22:46:59.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003685b716af586cdf-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj87 2023-09-22T22:46:59.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj87 2023-09-22T22:46:59.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.898 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.898 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.898 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.898 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.899 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj87?versionId=null 2023-09-22T22:46:59.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.899 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:CKfJelByYNISJSHaqmuJh6qp0Nk= 2023-09-22T22:46:59.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CKfJelByYNISJSHaqmuJh6qp0Nk=', 'Content-Length': '0'} 2023-09-22T22:46:59.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000117554cd964fff75-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj87 2023-09-22T22:46:59.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj87 2023-09-22T22:46:59.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj87 2023-09-22T22:46:59.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.900 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.900 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.900 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.900 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.901 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.901 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:46:59.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:46:59.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000440ce94247197f79-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj871000falseobj872023-09-22T22:05:15.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj87 2023-09-22T22:46:59.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj87 2023-09-22T22:46:59.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj87 2023-09-22T22:46:59.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.902 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.902 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.902 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.903 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.903 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.903 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:46:59.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:46:59.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa858d4c41ecd0b8-00650e0fb6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj871000falseobj872023-09-22T22:05:15.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj87 2023-09-22T22:46:59.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj87 2023-09-22T22:46:59.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj87 2023-09-22T22:46:59.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.904 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.904 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.904 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.904 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.904 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj87 2023-09-22T22:46:59.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.905 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Z5bmL+vtCiDJZ2LH5VR1VOZhMlk= 2023-09-22T22:46:59.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Z5bmL+vtCiDJZ2LH5VR1VOZhMlk=', 'Content-Length': '0'} 2023-09-22T22:46:59.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d5e1af4a5009030-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj87 2023-09-22T22:46:59.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj87 2023-09-22T22:46:59.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj87 2023-09-22T22:46:59.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.906 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.906 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.906 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.906 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.906 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj87 2023-09-22T22:46:59.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.906 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Z5bmL+vtCiDJZ2LH5VR1VOZhMlk= 2023-09-22T22:46:59.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Z5bmL+vtCiDJZ2LH5VR1VOZhMlk=', 'Content-Length': '0'} 2023-09-22T22:46:59.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c9d2f0634825791-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.907 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj87 2023-09-22T22:46:59.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj87 2023-09-22T22:46:59.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj87 2023-09-22T22:46:59.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj87 2023-09-22T22:46:59.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.908 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.908 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.908 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.908 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.908 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj87 2023-09-22T22:46:59.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.908 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:gWJjvC4faFx7WeQ/iK9x/p+vD8M= 2023-09-22T22:46:59.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gWJjvC4faFx7WeQ/iK9x/p+vD8M=', 'Content-Length': '0'} 2023-09-22T22:46:59.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cfefe4d1dd7ecc4-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj87 2023-09-22T22:46:59.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj87 2023-09-22T22:46:59.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj87 2023-09-22T22:46:59.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.910 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.910 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.910 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.910 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.910 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj87 2023-09-22T22:46:59.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.910 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:gWJjvC4faFx7WeQ/iK9x/p+vD8M= 2023-09-22T22:46:59.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gWJjvC4faFx7WeQ/iK9x/p+vD8M=', 'Content-Length': '0'} 2023-09-22T22:46:59.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff812c1974eba416-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj87 2023-09-22T22:46:59.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj87 2023-09-22T22:46:59.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj87 2023-09-22T22:46:59.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.912 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.912 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.912 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.912 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.912 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.912 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:46:59.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:46:59.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea88e8fadacf8c43-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj871000falseobj872023-09-22T22:05:15.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj87 2023-09-22T22:46:59.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj87 2023-09-22T22:46:59.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj87 2023-09-22T22:46:59.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.914 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.914 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.914 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.914 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.914 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.914 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0= 2023-09-22T22:46:59.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:uWUfhHAL9VBF9RlAL4xnchw/vt0=', 'Content-Length': '0'} 2023-09-22T22:46:59.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fd33b2d4a58cc8a-00650e0fb6-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj871000falseobj872023-09-22T22:05:15.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.915 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj92 2023-09-22T22:46:59.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj92 2023-09-22T22:46:59.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj92 2023-09-22T22:46:59.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.916 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.916 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.916 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.916 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.916 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.916 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Ek75pf0KD5sk02d9jIDkruZC414= 2023-09-22T22:46:59.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ek75pf0KD5sk02d9jIDkruZC414=', 'Content-Length': '0'} 2023-09-22T22:46:59.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d13a2bf83f54466f-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj92 2023-09-22T22:46:59.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj92 2023-09-22T22:46:59.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.918 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.918 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.918 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.918 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.918 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.918 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Ek75pf0KD5sk02d9jIDkruZC414= 2023-09-22T22:46:59.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ek75pf0KD5sk02d9jIDkruZC414=', 'Content-Length': '0'} 2023-09-22T22:46:59.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050f6fe454b846a3c-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj92 2023-09-22T22:46:59.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj92 2023-09-22T22:46:59.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.920 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.920 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.920 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.920 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.920 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.920 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kUlk563M3++sOPTb0DANqmhZhZA= 2023-09-22T22:46:59.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kUlk563M3++sOPTb0DANqmhZhZA=', 'Content-Length': '0'} 2023-09-22T22:46:59.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a15dc3bda061267-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj92 2023-09-22T22:46:59.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj92 2023-09-22T22:46:59.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.921 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.922 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.922 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.922 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.922 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.922 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kUlk563M3++sOPTb0DANqmhZhZA= 2023-09-22T22:46:59.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kUlk563M3++sOPTb0DANqmhZhZA=', 'Content-Length': '0'} 2023-09-22T22:46:59.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b42eb422fbb0c120-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.922 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj92 2023-09-22T22:46:59.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj92 2023-09-22T22:46:59.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj92 2023-09-22T22:46:59.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.924 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.924 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.924 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.924 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.924 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.924 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Ek75pf0KD5sk02d9jIDkruZC414= 2023-09-22T22:46:59.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ek75pf0KD5sk02d9jIDkruZC414=', 'Content-Length': '0'} 2023-09-22T22:46:59.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aae509fc9a53935-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj92 2023-09-22T22:46:59.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj92 2023-09-22T22:46:59.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.926 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.926 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.926 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.926 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:42 GMT 2023-09-22T22:46:59.926 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj92?versionId=null 2023-09-22T22:46:59.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.926 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Ek75pf0KD5sk02d9jIDkruZC414= 2023-09-22T22:46:59.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:42 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ek75pf0KD5sk02d9jIDkruZC414=', 'Content-Length': '0'} 2023-09-22T22:46:59.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c04738da06b152bb-00650e0fb6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj92 2023-09-22T22:46:59.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj92 2023-09-22T22:46:59.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj92 2023-09-22T22:46:59.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.928 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.928 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.928 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.928 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.928 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.928 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:46:59.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:46:59.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5a8769997658bfb-00650e0fb6-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj921000falseobj922023-09-22T22:05:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj92 2023-09-22T22:46:59.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj92 2023-09-22T22:46:59.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj92 2023-09-22T22:46:59.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.930 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.930 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.930 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.930 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.930 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.930 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:46:59.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:46:59.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e61ebc2ebf31ef1e-00650e0fb7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj921000falseobj922023-09-22T22:05:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj92 2023-09-22T22:46:59.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj92 2023-09-22T22:46:59.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj92 2023-09-22T22:46:59.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.931 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.932 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.932 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.932 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.932 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj92 2023-09-22T22:46:59.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.932 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Xoawnu6ZJYh3bmipyFQPfw86qto= 2023-09-22T22:46:59.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Xoawnu6ZJYh3bmipyFQPfw86qto=', 'Content-Length': '0'} 2023-09-22T22:46:59.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd750d17298b9ed1-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj92 2023-09-22T22:46:59.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj92 2023-09-22T22:46:59.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj92 2023-09-22T22:46:59.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.933 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.933 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.933 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.933 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.934 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj92 2023-09-22T22:46:59.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.934 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Xoawnu6ZJYh3bmipyFQPfw86qto= 2023-09-22T22:46:59.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Xoawnu6ZJYh3bmipyFQPfw86qto=', 'Content-Length': '0'} 2023-09-22T22:46:59.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dae871acb95fd40a-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.934 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj92 2023-09-22T22:46:59.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj92 2023-09-22T22:46:59.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj92 2023-09-22T22:46:59.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj92 2023-09-22T22:46:59.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.935 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.935 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.935 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.935 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.935 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj92 2023-09-22T22:46:59.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.936 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:zT9I0avIVi8tvstlZYsM3LrbNTw= 2023-09-22T22:46:59.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zT9I0avIVi8tvstlZYsM3LrbNTw=', 'Content-Length': '0'} 2023-09-22T22:46:59.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eba8c361fac76430-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj92 2023-09-22T22:46:59.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj92 2023-09-22T22:46:59.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj92 2023-09-22T22:46:59.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.937 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.937 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.937 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.937 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.937 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj92 2023-09-22T22:46:59.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.937 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:zT9I0avIVi8tvstlZYsM3LrbNTw= 2023-09-22T22:46:59.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:zT9I0avIVi8tvstlZYsM3LrbNTw=', 'Content-Length': '0'} 2023-09-22T22:46:59.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feaf4b64d2445192-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj92 2023-09-22T22:46:59.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj92 2023-09-22T22:46:59.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj92 2023-09-22T22:46:59.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.939 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.939 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.939 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.939 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.939 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.939 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:46:59.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:46:59.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081ea4d0a7328b3b0-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj921000falseobj922023-09-22T22:05:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj92 2023-09-22T22:46:59.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj92 2023-09-22T22:46:59.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj92 2023-09-22T22:46:59.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.941 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.941 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.941 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.941 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.941 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.941 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:46:59.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:46:59.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af92349e91331298-00650e0fb7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj921000falseobj922023-09-22T22:05:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.942 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj93 2023-09-22T22:46:59.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj93 2023-09-22T22:46:59.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj93 2023-09-22T22:46:59.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.943 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.943 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.943 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.943 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.943 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.944 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:MJHMR+UXtiWxykgr1S7QRov7Opw= 2023-09-22T22:46:59.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MJHMR+UXtiWxykgr1S7QRov7Opw=', 'Content-Length': '0'} 2023-09-22T22:46:59.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000df549528d6cb950-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj93 2023-09-22T22:46:59.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj93 2023-09-22T22:46:59.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.945 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.945 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.945 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.945 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.945 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.946 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:MJHMR+UXtiWxykgr1S7QRov7Opw= 2023-09-22T22:46:59.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MJHMR+UXtiWxykgr1S7QRov7Opw=', 'Content-Length': '0'} 2023-09-22T22:46:59.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b931e11a0e9e2aaa-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj93 2023-09-22T22:46:59.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj93 2023-09-22T22:46:59.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.947 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.947 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.947 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.947 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.947 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.947 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:y99FCKWsNpvW/KadOooWXZJ1Xlc= 2023-09-22T22:46:59.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y99FCKWsNpvW/KadOooWXZJ1Xlc=', 'Content-Length': '0'} 2023-09-22T22:46:59.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032801624a45bcb25-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj93 2023-09-22T22:46:59.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj93 2023-09-22T22:46:59.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.949 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.949 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.949 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.949 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.949 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.949 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:y99FCKWsNpvW/KadOooWXZJ1Xlc= 2023-09-22T22:46:59.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:y99FCKWsNpvW/KadOooWXZJ1Xlc=', 'Content-Length': '0'} 2023-09-22T22:46:59.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085b7c989b343f2f2-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.950 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj93 2023-09-22T22:46:59.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj93 2023-09-22T22:46:59.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj93 2023-09-22T22:46:59.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.951 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.951 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.951 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.951 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.951 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.951 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:MJHMR+UXtiWxykgr1S7QRov7Opw= 2023-09-22T22:46:59.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MJHMR+UXtiWxykgr1S7QRov7Opw=', 'Content-Length': '0'} 2023-09-22T22:46:59.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045b2d3c928126130-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj93 2023-09-22T22:46:59.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj93 2023-09-22T22:46:59.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.953 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.953 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.953 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.953 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.953 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj93?versionId=null 2023-09-22T22:46:59.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.953 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:MJHMR+UXtiWxykgr1S7QRov7Opw= 2023-09-22T22:46:59.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MJHMR+UXtiWxykgr1S7QRov7Opw=', 'Content-Length': '0'} 2023-09-22T22:46:59.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d2789ade8e39eea-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj93 2023-09-22T22:46:59.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj93 2023-09-22T22:46:59.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj93 2023-09-22T22:46:59.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.955 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.955 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.955 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.955 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.955 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.955 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:46:59.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:46:59.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f9db05a74d3fdcc-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj931000falseobj932023-09-22T22:05:15.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj93 2023-09-22T22:46:59.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj93 2023-09-22T22:46:59.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj93 2023-09-22T22:46:59.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.957 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.957 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.957 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.957 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.957 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.957 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:46:59.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:46:59.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067de43cc42f2afc0-00650e0fb7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj931000falseobj932023-09-22T22:05:15.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj93 2023-09-22T22:46:59.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj93 2023-09-22T22:46:59.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj93 2023-09-22T22:46:59.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.959 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.959 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.959 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.959 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.959 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj93 2023-09-22T22:46:59.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.959 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:dtMbyFsRW47rjHEhOFQwuXz9hpM= 2023-09-22T22:46:59.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dtMbyFsRW47rjHEhOFQwuXz9hpM=', 'Content-Length': '0'} 2023-09-22T22:46:59.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054bfdb49acad5d45-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj93 2023-09-22T22:46:59.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj93 2023-09-22T22:46:59.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj93 2023-09-22T22:46:59.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.960 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.960 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.961 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.961 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.961 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj93 2023-09-22T22:46:59.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.961 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:dtMbyFsRW47rjHEhOFQwuXz9hpM= 2023-09-22T22:46:59.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:dtMbyFsRW47rjHEhOFQwuXz9hpM=', 'Content-Length': '0'} 2023-09-22T22:46:59.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6a2058ab3ae83b4-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.961 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj93 2023-09-22T22:46:59.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj93 2023-09-22T22:46:59.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj93 2023-09-22T22:46:59.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj93 2023-09-22T22:46:59.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.962 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.962 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.962 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.962 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.963 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj93 2023-09-22T22:46:59.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.963 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:27ZdIEbfvvbN/C67Eg/86x5tT2w= 2023-09-22T22:46:59.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:27ZdIEbfvvbN/C67Eg/86x5tT2w=', 'Content-Length': '0'} 2023-09-22T22:46:59.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027c319405451b894-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj93 2023-09-22T22:46:59.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj93 2023-09-22T22:46:59.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj93 2023-09-22T22:46:59.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.964 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.964 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.964 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.964 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.964 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj93 2023-09-22T22:46:59.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.964 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:27ZdIEbfvvbN/C67Eg/86x5tT2w= 2023-09-22T22:46:59.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:27ZdIEbfvvbN/C67Eg/86x5tT2w=', 'Content-Length': '0'} 2023-09-22T22:46:59.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a4251d6470d2a8f-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj93 2023-09-22T22:46:59.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj93 2023-09-22T22:46:59.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj93 2023-09-22T22:46:59.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.966 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.966 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.966 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.966 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.966 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.966 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:46:59.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:46:59.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e46988d20788bf21-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj931000falseobj932023-09-22T22:05:15.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj93 2023-09-22T22:46:59.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj93 2023-09-22T22:46:59.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj93 2023-09-22T22:46:59.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.968 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.968 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.968 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.968 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.968 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.968 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:46:59.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:46:59.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036e922a8ac213b79-00650e0fb7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj931000falseobj932023-09-22T22:05:15.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.969 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj95 2023-09-22T22:46:59.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj95 2023-09-22T22:46:59.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj95 2023-09-22T22:46:59.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.970 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.970 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.970 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.970 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.970 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.971 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Ia92LecqngRIpRSDl1q0G510zaE= 2023-09-22T22:46:59.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ia92LecqngRIpRSDl1q0G510zaE=', 'Content-Length': '0'} 2023-09-22T22:46:59.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceda11c762259ed5-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj95 2023-09-22T22:46:59.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj95 2023-09-22T22:46:59.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.972 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.972 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.972 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.972 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.972 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.973 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Ia92LecqngRIpRSDl1q0G510zaE= 2023-09-22T22:46:59.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ia92LecqngRIpRSDl1q0G510zaE=', 'Content-Length': '0'} 2023-09-22T22:46:59.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000631c134403fbd08-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj95 2023-09-22T22:46:59.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj95 2023-09-22T22:46:59.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.974 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.974 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.974 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.974 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.974 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.975 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bDTtTPCVZIaTFzn65i84LVySOwc= 2023-09-22T22:46:59.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bDTtTPCVZIaTFzn65i84LVySOwc=', 'Content-Length': '0'} 2023-09-22T22:46:59.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003502a5a32d87456f-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj95 2023-09-22T22:46:59.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj95 2023-09-22T22:46:59.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.976 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.976 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.976 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.976 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.976 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.976 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bDTtTPCVZIaTFzn65i84LVySOwc= 2023-09-22T22:46:59.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bDTtTPCVZIaTFzn65i84LVySOwc=', 'Content-Length': '0'} 2023-09-22T22:46:59.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff7e5b1f832b8f75-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.977 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj95 2023-09-22T22:46:59.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj95 2023-09-22T22:46:59.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj95 2023-09-22T22:46:59.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.978 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.978 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.978 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.978 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.978 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.979 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Ia92LecqngRIpRSDl1q0G510zaE= 2023-09-22T22:46:59.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ia92LecqngRIpRSDl1q0G510zaE=', 'Content-Length': '0'} 2023-09-22T22:46:59.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b01abe5e973224e-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj95 2023-09-22T22:46:59.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj95 2023-09-22T22:46:59.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.980 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.980 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.980 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.980 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.980 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj95?versionId=null 2023-09-22T22:46:59.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.980 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Ia92LecqngRIpRSDl1q0G510zaE= 2023-09-22T22:46:59.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Ia92LecqngRIpRSDl1q0G510zaE=', 'Content-Length': '0'} 2023-09-22T22:46:59.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c83833b1a2b623ab-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj95 2023-09-22T22:46:59.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj95 2023-09-22T22:46:59.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj95 2023-09-22T22:46:59.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.982 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.982 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.982 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.982 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.982 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.982 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:46:59.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:46:59.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdbb106311797da9-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj951000falseobj952023-09-22T22:05:14.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj95 2023-09-22T22:46:59.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj95 2023-09-22T22:46:59.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj95 2023-09-22T22:46:59.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.984 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.984 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.984 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.984 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.984 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.985 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:46:59.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:46:59.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b52cb48c3f52dd24-00650e0fb7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj951000falseobj952023-09-22T22:05:14.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj95 2023-09-22T22:46:59.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj95 2023-09-22T22:46:59.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj95 2023-09-22T22:46:59.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.986 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.986 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.986 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.986 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.986 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj95 2023-09-22T22:46:59.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.986 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:a0jh7PUag7FVXT6w50icyHMXdqE= 2023-09-22T22:46:59.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:a0jh7PUag7FVXT6w50icyHMXdqE=', 'Content-Length': '0'} 2023-09-22T22:46:59.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079fdf798b91ea29a-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj95 2023-09-22T22:46:59.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj95 2023-09-22T22:46:59.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj95 2023-09-22T22:46:59.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.988 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.988 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.988 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.988 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.988 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj95 2023-09-22T22:46:59.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.988 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:a0jh7PUag7FVXT6w50icyHMXdqE= 2023-09-22T22:46:59.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:a0jh7PUag7FVXT6w50icyHMXdqE=', 'Content-Length': '0'} 2023-09-22T22:46:59.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f531060cf65234a-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.988 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj95 2023-09-22T22:46:59.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj95 2023-09-22T22:46:59.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj95 2023-09-22T22:46:59.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj95 2023-09-22T22:46:59.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.989 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.990 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.990 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.990 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.990 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj95 2023-09-22T22:46:59.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.990 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oGIKI8L+3KtzhKk95AkGl66lFxc= 2023-09-22T22:46:59.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oGIKI8L+3KtzhKk95AkGl66lFxc=', 'Content-Length': '0'} 2023-09-22T22:46:59.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-63:f132bdb9-20a7-4073-a932-8f99da20691e.5729.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c0ea2c93989a02e-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj95 2023-09-22T22:46:59.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj95 2023-09-22T22:46:59.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj95 2023-09-22T22:46:59.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.991 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.991 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.991 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.991 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.992 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj95 2023-09-22T22:46:59.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.992 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:oGIKI8L+3KtzhKk95AkGl66lFxc= 2023-09-22T22:46:59.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:oGIKI8L+3KtzhKk95AkGl66lFxc=', 'Content-Length': '0'} 2023-09-22T22:46:59.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:05:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af3fa6b1bab5c4c1-00650e0fb7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj95 2023-09-22T22:46:59.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj95 2023-09-22T22:46:59.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj95 2023-09-22T22:46:59.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:46:59.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:46:59.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.993 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.993 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.993 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.993 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.994 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.994 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:46:59.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:46:59.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005470b2c4940f9782-00650e0fb7-1f3d-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj951000falseobj952023-09-22T22:05:14.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/?prefix=obj95 2023-09-22T22:46:59.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/?prefix=obj95 2023-09-22T22:46:59.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:46:59.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/?prefix=obj95 2023-09-22T22:46:59.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.995 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:46:59.995 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.995 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.995 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:43 GMT 2023-09-22T22:46:59.996 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.996 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc= 2023-09-22T22:46:59.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:43 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:aoPVaRJV8VVgtqj/DqEtkvAbTFc=', 'Content-Length': '0'} 2023-09-22T22:46:59.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b816606b29d1ab97-00650e0fb7-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-63obj951000falseobj952023-09-22T22:05:14.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:46:59.996 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=ffzdqu-63 zones={a2, a1} 2023-09-22T22:46:59.996 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: DEBUG: waiting for process to exit 2023-09-22T22:46:59.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2023-09-22T22:46:59.996 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:05:43.607+0000 7f7bf5126640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 83780) UID: 0 2023-09-22T22:46:59.996 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:05:43.607+0000 7f7bf896da40 -1 shutting down 2023-09-22T22:46:59.997 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2023-09-22T22:46:59.997 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Stopped 2023-09-22T22:46:59.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:46:59.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:46:59.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:46:59.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:46:59.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:46:59.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:46:59.998 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:46:59.998 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.998 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:46:59.998 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:49 GMT 2023-09-22T22:46:59.998 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:46:59.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:46:59.998 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:rT0I7C+K3yRSjyCLsgXGysOO0b8= 2023-09-22T22:46:59.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rT0I7C+K3yRSjyCLsgXGysOO0b8=', 'Content-Length': '0'} 2023-09-22T22:46:59.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000968603b916a7246a-00650e0fbd-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:46:59.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj37 2023-09-22T22:46:59.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj37 2023-09-22T22:46:59.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:46:59.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj37 2023-09-22T22:46:59.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:46:59.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:46:59.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:46:59.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:46:59.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:46:59.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:46:59.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.000 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.000 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.000 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.000 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:49 GMT 2023-09-22T22:47:00.000 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj37 2023-09-22T22:47:00.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.000 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:MBOmitfRUaiQ7ahB1puT6IgiQgg= 2023-09-22T22:47:00.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MBOmitfRUaiQ7ahB1puT6IgiQgg='} 2023-09-22T22:47:00.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003aea74f3019ee6f8-00650e0fbd-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:47:00.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:47:00.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:47:00.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.001 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.001 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.001 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.002 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:49 GMT 2023-09-22T22:47:00.002 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:47:00.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.002 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:rT0I7C+K3yRSjyCLsgXGysOO0b8= 2023-09-22T22:47:00.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rT0I7C+K3yRSjyCLsgXGysOO0b8=', 'Content-Length': '0'} 2023-09-22T22:47:00.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037d8d61a9a3e2fbd-00650e0fbd-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj74 2023-09-22T22:47:00.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj74 2023-09-22T22:47:00.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj74 2023-09-22T22:47:00.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.003 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.003 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.003 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.003 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:49 GMT 2023-09-22T22:47:00.003 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj74 2023-09-22T22:47:00.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.004 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:d43Ajibp+jx8/SOOBEYtaS65C4w= 2023-09-22T22:47:00.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:d43Ajibp+jx8/SOOBEYtaS65C4w='} 2023-09-22T22:47:00.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000097b4c4595abcf586-00650e0fbd-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:47:00.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:47:00.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:47:00.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.005 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.005 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.005 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.005 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:49 GMT 2023-09-22T22:47:00.005 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:47:00.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.005 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:rT0I7C+K3yRSjyCLsgXGysOO0b8= 2023-09-22T22:47:00.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rT0I7C+K3yRSjyCLsgXGysOO0b8=', 'Content-Length': '0'} 2023-09-22T22:47:00.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b24205224fb8e782-00650e0fbd-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj111 2023-09-22T22:47:00.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj111 2023-09-22T22:47:00.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj111 2023-09-22T22:47:00.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.007 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.007 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.007 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.007 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:49 GMT 2023-09-22T22:47:00.007 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj111 2023-09-22T22:47:00.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.007 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:79RWPyslkGteYGbPoiGUZX1JTh4= 2023-09-22T22:47:00.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:79RWPyslkGteYGbPoiGUZX1JTh4='} 2023-09-22T22:47:00.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006e3f0f09a2edee92-00650e0fbd-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:47:00.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:47:00.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:47:00.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.008 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.008 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.009 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.009 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:49 GMT 2023-09-22T22:47:00.009 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:47:00.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.009 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:rT0I7C+K3yRSjyCLsgXGysOO0b8= 2023-09-22T22:47:00.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rT0I7C+K3yRSjyCLsgXGysOO0b8=', 'Content-Length': '0'} 2023-09-22T22:47:00.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092cafe344ec4baca-00650e0fbd-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj148 2023-09-22T22:47:00.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj148 2023-09-22T22:47:00.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj148 2023-09-22T22:47:00.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.010 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.010 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.010 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.010 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:49 GMT 2023-09-22T22:47:00.011 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj148 2023-09-22T22:47:00.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.011 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:XSMuiUUrf93SZDCqnYPHpvX/Vlo= 2023-09-22T22:47:00.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:XSMuiUUrf93SZDCqnYPHpvX/Vlo='} 2023-09-22T22:47:00.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005907c3d0b573ef55-00650e0fbd-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/ 2023-09-22T22:47:00.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/ 2023-09-22T22:47:00.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/ 2023-09-22T22:47:00.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.012 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.012 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.012 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.012 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:49 GMT 2023-09-22T22:47:00.012 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/ 2023-09-22T22:47:00.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.012 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:rT0I7C+K3yRSjyCLsgXGysOO0b8= 2023-09-22T22:47:00.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:05:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:rT0I7C+K3yRSjyCLsgXGysOO0b8=', 'Content-Length': '0'} 2023-09-22T22:47:00.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc95a2fa55aabea9-00650e0fbd-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-63/obj185 2023-09-22T22:47:00.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-63/obj185 2023-09-22T22:47:00.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-63/obj185 2023-09-22T22:47:00.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.014 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.014 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.014 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.014 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:05:49 GMT 2023-09-22T22:47:00.014 INFO:tasks.rgw_multisite_tests:/ffzdqu-63/obj185 2023-09-22T22:47:00.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.014 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:G02xL4IoMYhyvGUMTA0QEA0D3mk= 2023-09-22T22:47:00.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:05:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G02xL4IoMYhyvGUMTA0QEA0D3mk='} 2023-09-22T22:47:00.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055c526d52dddd8ca-00650e0fbd-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.015 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.015 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:05:49.982+0000 7f6149764a40 5 RGW-SYNC:data: waiting on data sync status lease 2023-09-22T22:47:00.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:05:49.986+0000 7f6149764a40 5 RGW-SYNC:data: acquired data sync status lease 2023-09-22T22:47:00.015 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Restarting daemon 2023-09-22T22:47:00.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-22T22:47:00.015 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Started 2023-09-22T22:47:00.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:47:00.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-22T22:47:00.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T22:47:00.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-09-22T22:47:00.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T22:47:00.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:47:00.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-22T22:47:00.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 60554 0 --:--:-- --:--:-- --:--:-- 71333 2023-09-22T22:47:00.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: anonymous 2023-09-22T22:47:00.016 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-63 2023-09-22T22:47:00.017 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:47:00.017 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:47:00.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T22:47:00.017 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 has been restored 2023-09-22T22:47:00.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:05:51.222+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.017 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:05:53.122+0000 7f2d4a614640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T22:47:00.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:05:56.230+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:06:01.234+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:06:06.242+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:06:11.246+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:06:16.250+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:06:21.258+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:06:26.262+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:06:31.270+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:06:36.274+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:06:41.278+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:06:46.286+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:06:51.290+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:06:56.298+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:07:01.302+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:07:06.305+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:07:11.313+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:07:16.317+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:07:21.329+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:07:26.333+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:07:31.341+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:07:36.345+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:07:41.349+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:07:46.357+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:07:51.361+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:07:56.369+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:08:01.377+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:08:06.381+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:08:11.389+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:08:16.392+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:08:21.400+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:08:26.408+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:08:31.412+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:08:36.416+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:08:41.424+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:08:46.428+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:08:51.436+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:08:56.440+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:09:01.448+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:09:06.452+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:09:11.460+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:09:16.464+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:09:21.468+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:09:26.476+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:09:31.479+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:09:36.487+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:09:41.495+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:09:46.499+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:09:51.507+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:09:56.511+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:10:01.519+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:10:06.523+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:10:11.527+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:10:16.535+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:10:21.539+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:10:26.547+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:10:31.551+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:10:36.559+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:10:41.563+0000 7f34ca72ca40 1 waiting for incremental sync to catch up: 2023-09-22T22:47:00.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.110.6, , , , , 00000000002.119.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.132.6] 2023-09-22T22:47:00.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , 00000000003.138.6, , , , , , 00000000003.112.6, , , , , 00000000003.121.6, , , , , , , , , , , , , , , 00000000002.118.6, , , , 00000000002.136.6] 2023-09-22T22:47:00.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:10:46.566+0000 7f34ca72ca40 0 bucket checkpoint timed out waiting for incremental sync to catch up 2023-09-22T22:47:00.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:10:46.566+0000 7f34ca72ca40 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-22T22:47:00.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:10:46.566+0000 7f34ca72ca40 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-22T22:47:00.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-09-22T22:47:00.034 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-22T22:47:00.034 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.035 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-22T22:47:00.035 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2023-09-22T22:47:00.035 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-22T22:47:00.035 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-22T22:47:00.035 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-22T22:47:00.036 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-22T22:47:00.036 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 1904, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2023-09-22T22:47:00.036 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-09-22T22:47:00.036 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-09-22T22:47:00.036 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-09-22T22:47:00.036 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-09-22T22:47:00.036 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-09-22T22:47:00.036 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-22T22:47:00.036 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-22T22:47:00.036 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-22T22:47:00.036 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-22T22:47:00.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T22:47:00.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.038 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.038 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.038 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.038 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:10:46 GMT 2023-09-22T22:47:00.038 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.038 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:CVrBAGWAaMdXd+MXHcUiAhbZSNw= 2023-09-22T22:47:00.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:10:46 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CVrBAGWAaMdXd+MXHcUiAhbZSNw=', 'Content-Length': '0'} 2023-09-22T22:47:00.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000f9dd883816c1da9e-00650e10e6-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:10:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=ffzdqu-64 2023-09-22T22:47:00.039 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.039 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T22:47:00.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: [{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"1_1695419384.000133_332.1","last_update":"2023-09-22T21:49:44.000133Z"},{"marker":"1_1695420646.644925_651.1","last_update":"2023-09-22T22:10:46.644925Z"},{"marker":"1_1695419383.971498_240.1","last_update":"2023-09-22T21:49:43.971498Z"}] 2023-09-22T22:47:00.039 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420646.644925_651.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.039 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-09-22T22:47:00.039 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.039 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:10:46.862+0000 7f97e2059a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:10:46.862+0000 7f97e2059a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420646.633184_648.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:10:46.633184Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:10:46Z"} 2023-09-22T22:47:00.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420646.633184_648.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:10:46.633184Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:10:46Z"} 2023-09-22T22:47:00.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420646.633184_648.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:10:46.633184Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420646.644925_651.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420646.633184_648.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695420646.644925_651.1 target=1_1695420646.633184_648.1 2023-09-22T22:47:00.041 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.041 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:10:51.974+0000 7faef7ccba40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:10:51.974+0000 7faef7ccba40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420646.644925_651.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:10:46.644925Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:10:51Z"} 2023-09-22T22:47:00.041 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420646.644925_651.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:10:46.644925Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:10:51Z"} 2023-09-22T22:47:00.041 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420646.644925_651.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:10:46.644925Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.041 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420646.644925_651.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.042 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420646.644925_651.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.042 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-09-22T22:47:00.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.043 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.043 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.043 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.043 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:10:51 GMT 2023-09-22T22:47:00.043 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.043 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Z6slShZpeKfl90AtKNO/Ej/+b/I= 2023-09-22T22:47:00.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:10:51 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Z6slShZpeKfl90AtKNO/Ej/+b/I=', 'Content-Length': '0'} 2023-09-22T22:47:00.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfdee417852908f0-00650e10eb-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:10:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj11 2023-09-22T22:47:00.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj11 2023-09-22T22:47:00.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj11 2023-09-22T22:47:00.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.045 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.045 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.045 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.045 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:10:52 GMT 2023-09-22T22:47:00.045 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj11 2023-09-22T22:47:00.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.045 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:AfUO7EktB8KLz4WJe2mYlBkZ22g= 2023-09-22T22:47:00.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:10:52 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:AfUO7EktB8KLz4WJe2mYlBkZ22g='} 2023-09-22T22:47:00.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000166a03fd3ab0386b-00650e10ec-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.046 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.046 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.047 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.047 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:10:52 GMT 2023-09-22T22:47:00.047 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.047 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:3/A/+jKsD2yNrKEJV2T0CSQSx6A= 2023-09-22T22:47:00.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:10:52 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3/A/+jKsD2yNrKEJV2T0CSQSx6A=', 'Content-Length': '0'} 2023-09-22T22:47:00.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f4a5859f6b3e959-00650e10ec-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj22 2023-09-22T22:47:00.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj22 2023-09-22T22:47:00.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj22 2023-09-22T22:47:00.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.048 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.048 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.048 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.048 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:10:52 GMT 2023-09-22T22:47:00.048 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj22 2023-09-22T22:47:00.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.049 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:/jSv1AmuZr4EQkURslnSHbyqFBQ= 2023-09-22T22:47:00.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:10:52 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:/jSv1AmuZr4EQkURslnSHbyqFBQ='} 2023-09-22T22:47:00.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c47d6e4efbb7a9df-00650e10ec-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.050 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.050 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.050 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.050 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:10:52 GMT 2023-09-22T22:47:00.050 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.051 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:3/A/+jKsD2yNrKEJV2T0CSQSx6A= 2023-09-22T22:47:00.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:10:52 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3/A/+jKsD2yNrKEJV2T0CSQSx6A=', 'Content-Length': '0'} 2023-09-22T22:47:00.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000220629b97d22a251-00650e10ec-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj33 2023-09-22T22:47:00.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj33 2023-09-22T22:47:00.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj33 2023-09-22T22:47:00.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.052 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.052 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.052 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.052 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:10:52 GMT 2023-09-22T22:47:00.052 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj33 2023-09-22T22:47:00.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.052 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:VpMRv+3K5JePX/aPQNYw8OQfGFQ= 2023-09-22T22:47:00.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:10:52 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VpMRv+3K5JePX/aPQNYw8OQfGFQ='} 2023-09-22T22:47:00.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091899caa7cf4e99c-00650e10ec-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.054 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.054 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.054 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.054 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:10:52 GMT 2023-09-22T22:47:00.054 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.054 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:3/A/+jKsD2yNrKEJV2T0CSQSx6A= 2023-09-22T22:47:00.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:10:52 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3/A/+jKsD2yNrKEJV2T0CSQSx6A=', 'Content-Length': '0'} 2023-09-22T22:47:00.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a89f8c1ae2d30af5-00650e10ec-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj44 2023-09-22T22:47:00.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj44 2023-09-22T22:47:00.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj44 2023-09-22T22:47:00.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.055 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.055 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.056 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.056 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:10:52 GMT 2023-09-22T22:47:00.056 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj44 2023-09-22T22:47:00.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.056 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:0ASaUJR6s6qUhGad9kceN3r3QK8= 2023-09-22T22:47:00.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:10:52 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0ASaUJR6s6qUhGad9kceN3r3QK8='} 2023-09-22T22:47:00.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008cc88a2a50ae0bbc-00650e10ec-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.057 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.057 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.057 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.057 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:10:52 GMT 2023-09-22T22:47:00.057 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.058 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:3/A/+jKsD2yNrKEJV2T0CSQSx6A= 2023-09-22T22:47:00.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:10:52 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3/A/+jKsD2yNrKEJV2T0CSQSx6A=', 'Content-Length': '0'} 2023-09-22T22:47:00.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe6e7f39170f0a3d-00650e10ec-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj55 2023-09-22T22:47:00.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj55 2023-09-22T22:47:00.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj55 2023-09-22T22:47:00.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.059 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.059 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.059 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.059 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:10:52 GMT 2023-09-22T22:47:00.059 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj55 2023-09-22T22:47:00.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.059 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bIMgps7X0pCjRAepVC8u6tKLNXM= 2023-09-22T22:47:00.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:10:52 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bIMgps7X0pCjRAepVC8u6tKLNXM='} 2023-09-22T22:47:00.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b170bbfffadf91e-00650e10ec-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.060 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-64 2023-09-22T22:47:00.060 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:47:00.060 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:47:00.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T22:47:00.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:10:52.238+0000 7f43099e3a40 1 waiting to reach incremental sync.. 2023-09-22T22:47:00.060 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:10:53.126+0000 7f2d4a614640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T22:47:00.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:10:57.242+0000 7f43099e3a40 1 bucket sync caught up with source: 2023-09-22T22:47:00.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , , , 00000000001.107.6, 00000000002.84.6, 00000000002.115.6] 2023-09-22T22:47:00.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , , , 00000000001.107.6, 00000000002.84.6, 00000000002.115.6] 2023-09-22T22:47:00.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:10:57.242+0000 7f43099e3a40 0 bucket checkpoint complete 2023-09-22T22:47:00.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=ffzdqu-64 2023-09-22T22:47:00.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:47:00.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:47:00.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T22:47:00.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:10:57.370+0000 7f63e8855a40 1 waiting to reach incremental sync.. 2023-09-22T22:47:00.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:11:02.370+0000 7f63e8855a40 1 waiting to reach incremental sync.. 2023-09-22T22:47:00.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:11:07.370+0000 7f63e8855a40 1 waiting to reach incremental sync.. 2023-09-22T22:47:00.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:11:12.370+0000 7f63e8855a40 1 waiting to reach incremental sync.. 2023-09-22T22:47:00.062 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi160.stdout: INFO: 2023-09-22T22:11:12.546+0000 7fa9c1675640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T22:47:00.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:11:17.374+0000 7f63e8855a40 1 bucket sync caught up with source: 2023-09-22T22:47:00.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: local status: [, , , , , , , , 00000000001.367.6, 00000000002.283.6, 00000000002.239.6] 2023-09-22T22:47:00.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: remote markers: [, , , , , , , , 00000000001.367.6, 00000000002.283.6, 00000000002.239.6] 2023-09-22T22:47:00.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:11:17.374+0000 7f63e8855a40 0 bucket checkpoint complete 2023-09-22T22:47:00.062 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=ffzdqu-64 zones={a2, a1} 2023-09-22T22:47:00.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-22T22:47:00.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.063 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.063 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.063 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.063 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.064 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.064 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:MzzFtzOyR23+7IZDebU2fVumaUk= 2023-09-22T22:47:00.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MzzFtzOyR23+7IZDebU2fVumaUk=', 'Content-Length': '0'} 2023-09-22T22:47:00.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000546f4f365db72675-00650e1105-1fae-a2'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.065 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.065 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.065 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.065 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.065 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.065 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:MzzFtzOyR23+7IZDebU2fVumaUk= 2023-09-22T22:47:00.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MzzFtzOyR23+7IZDebU2fVumaUk=', 'Content-Length': '0'} 2023-09-22T22:47:00.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6fdaf47d6099dc4-00650e1105-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.066 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-09-22T22:47:00.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?versions&max-keys=999 2023-09-22T22:47:00.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?versions&max-keys=999 2023-09-22T22:47:00.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?versions&max-keys=999 2023-09-22T22:47:00.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.067 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.067 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.067 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.069 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.069 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/?versions 2023-09-22T22:47:00.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.069 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:6tEvQMrNLoSeNyxhnQYIMaZBat8= 2023-09-22T22:47:00.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6tEvQMrNLoSeNyxhnQYIMaZBat8=', 'Content-Length': '0'} 2023-09-22T22:47:00.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018be0ef4e8980cd9-00650e1105-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64999falseobj11nulltrue2023-09-22T22:10:52.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T22:10:52.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T22:10:52.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T22:10:52.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T22:10:52.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.069 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-09-22T22:47:00.070 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-09-22T22:47:00.070 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-09-22T22:47:00.070 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-09-22T22:47:00.070 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-09-22T22:47:00.070 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-09-22T22:47:00.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?versions&max-keys=999 2023-09-22T22:47:00.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?versions&max-keys=999 2023-09-22T22:47:00.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?versions&max-keys=999 2023-09-22T22:47:00.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.071 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.071 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.071 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.071 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.072 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/?versions 2023-09-22T22:47:00.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.072 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:6tEvQMrNLoSeNyxhnQYIMaZBat8= 2023-09-22T22:47:00.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6tEvQMrNLoSeNyxhnQYIMaZBat8=', 'Content-Length': '0'} 2023-09-22T22:47:00.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6f98e2897e5a0ac-00650e1105-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64999falseobj11nulltrue2023-09-22T22:10:52.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T22:10:52.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T22:10:52.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T22:10:52.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T22:10:52.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.072 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-09-22T22:47:00.072 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-09-22T22:47:00.072 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-09-22T22:47:00.072 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-09-22T22:47:00.072 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-09-22T22:47:00.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?versions&max-keys=999 2023-09-22T22:47:00.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?versions&max-keys=999 2023-09-22T22:47:00.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?versions&max-keys=999 2023-09-22T22:47:00.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.074 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.074 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.074 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.074 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.074 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/?versions 2023-09-22T22:47:00.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.074 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:6tEvQMrNLoSeNyxhnQYIMaZBat8= 2023-09-22T22:47:00.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6tEvQMrNLoSeNyxhnQYIMaZBat8=', 'Content-Length': '0'} 2023-09-22T22:47:00.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b33931ac92f66663-00650e1105-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64999falseobj11nulltrue2023-09-22T22:10:52.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T22:10:52.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T22:10:52.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T22:10:52.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T22:10:52.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?versions&max-keys=999 2023-09-22T22:47:00.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?versions&max-keys=999 2023-09-22T22:47:00.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?versions&max-keys=999 2023-09-22T22:47:00.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.076 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.076 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.076 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.076 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.076 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/?versions 2023-09-22T22:47:00.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.076 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:6tEvQMrNLoSeNyxhnQYIMaZBat8= 2023-09-22T22:47:00.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:6tEvQMrNLoSeNyxhnQYIMaZBat8=', 'Content-Length': '0'} 2023-09-22T22:47:00.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f397ed7ea8762e10-00650e1105-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64999falseobj11nulltrue2023-09-22T22:10:52.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-22T22:10:52.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-22T22:10:52.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-22T22:10:52.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-22T22:10:52.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.077 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-09-22T22:47:00.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj11 2023-09-22T22:47:00.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj11 2023-09-22T22:47:00.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.078 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.078 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.078 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.078 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.078 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.079 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:MAGCo+HTDcx7bvFqY8x75Ofvj6s= 2023-09-22T22:47:00.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MAGCo+HTDcx7bvFqY8x75Ofvj6s=', 'Content-Length': '0'} 2023-09-22T22:47:00.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000611778ff93b56430-00650e1105-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj11 2023-09-22T22:47:00.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj11 2023-09-22T22:47:00.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.080 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.080 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.080 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.080 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.080 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.080 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:MAGCo+HTDcx7bvFqY8x75Ofvj6s= 2023-09-22T22:47:00.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MAGCo+HTDcx7bvFqY8x75Ofvj6s=', 'Content-Length': '0'} 2023-09-22T22:47:00.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd7cc4c09ec378f0-00650e1105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj11 2023-09-22T22:47:00.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj11 2023-09-22T22:47:00.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.082 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.082 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.082 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.082 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.082 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.082 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:B8qQKr2P7CwCAj23Dz0W4CjdEM0= 2023-09-22T22:47:00.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:B8qQKr2P7CwCAj23Dz0W4CjdEM0=', 'Content-Length': '0'} 2023-09-22T22:47:00.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006663de5db3a9d0cb-00650e1105-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj11 2023-09-22T22:47:00.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj11 2023-09-22T22:47:00.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.084 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.084 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.084 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.084 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.084 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.084 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:B8qQKr2P7CwCAj23Dz0W4CjdEM0= 2023-09-22T22:47:00.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:B8qQKr2P7CwCAj23Dz0W4CjdEM0=', 'Content-Length': '0'} 2023-09-22T22:47:00.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000791e414cfe2b0bbc-00650e1105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.085 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-09-22T22:47:00.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj11 2023-09-22T22:47:00.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj11 2023-09-22T22:47:00.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.086 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.086 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.086 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.086 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.086 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.087 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:MAGCo+HTDcx7bvFqY8x75Ofvj6s= 2023-09-22T22:47:00.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MAGCo+HTDcx7bvFqY8x75Ofvj6s=', 'Content-Length': '0'} 2023-09-22T22:47:00.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b02322b564edaad3-00650e1105-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj11 2023-09-22T22:47:00.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj11 2023-09-22T22:47:00.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.088 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.088 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.088 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.088 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.088 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj11?versionId=null 2023-09-22T22:47:00.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.088 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:MAGCo+HTDcx7bvFqY8x75Ofvj6s= 2023-09-22T22:47:00.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:MAGCo+HTDcx7bvFqY8x75Ofvj6s=', 'Content-Length': '0'} 2023-09-22T22:47:00.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c31a133abd63d705-00650e1105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?prefix=obj11 2023-09-22T22:47:00.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?prefix=obj11 2023-09-22T22:47:00.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?prefix=obj11 2023-09-22T22:47:00.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.090 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.090 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.090 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.090 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.090 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.090 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs= 2023-09-22T22:47:00.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs=', 'Content-Length': '0'} 2023-09-22T22:47:00.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000796307c7439d1aec-00650e1105-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64obj111000falseobj112023-09-22T22:10:52.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?prefix=obj11 2023-09-22T22:47:00.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?prefix=obj11 2023-09-22T22:47:00.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?prefix=obj11 2023-09-22T22:47:00.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.092 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.092 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.092 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.092 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.092 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.092 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs= 2023-09-22T22:47:00.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs=', 'Content-Length': '0'} 2023-09-22T22:47:00.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fed5ff75ab9911a4-00650e1105-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64obj111000falseobj112023-09-22T22:10:52.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj11 2023-09-22T22:47:00.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj11 2023-09-22T22:47:00.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj11 2023-09-22T22:47:00.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.094 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.094 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.094 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.094 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.094 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj11 2023-09-22T22:47:00.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.094 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:7k87VKrrWndsR5w/94krjYTTNC0= 2023-09-22T22:47:00.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7k87VKrrWndsR5w/94krjYTTNC0=', 'Content-Length': '0'} 2023-09-22T22:47:00.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeca03ade649dac6-00650e1105-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj11 2023-09-22T22:47:00.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj11 2023-09-22T22:47:00.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj11 2023-09-22T22:47:00.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.096 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.096 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.096 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.096 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.096 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj11 2023-09-22T22:47:00.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.096 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:7k87VKrrWndsR5w/94krjYTTNC0= 2023-09-22T22:47:00.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:7k87VKrrWndsR5w/94krjYTTNC0=', 'Content-Length': '0'} 2023-09-22T22:47:00.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd0f457f96fdb82e-00650e1105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.096 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-09-22T22:47:00.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj11 2023-09-22T22:47:00.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj11 2023-09-22T22:47:00.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj11 2023-09-22T22:47:00.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.097 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.098 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.098 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.098 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.098 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj11 2023-09-22T22:47:00.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.098 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:QylKQUMlzPV7IrCtK3iyx3XSMRs= 2023-09-22T22:47:00.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QylKQUMlzPV7IrCtK3iyx3XSMRs=', 'Content-Length': '0'} 2023-09-22T22:47:00.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007657fcf73be387ad-00650e1105-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj11 2023-09-22T22:47:00.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj11 2023-09-22T22:47:00.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj11 2023-09-22T22:47:00.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.099 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.099 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.099 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.100 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.100 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj11 2023-09-22T22:47:00.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.100 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:QylKQUMlzPV7IrCtK3iyx3XSMRs= 2023-09-22T22:47:00.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:QylKQUMlzPV7IrCtK3iyx3XSMRs=', 'Content-Length': '0'} 2023-09-22T22:47:00.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f72abdc5d17afce6-00650e1105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?prefix=obj11 2023-09-22T22:47:00.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?prefix=obj11 2023-09-22T22:47:00.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?prefix=obj11 2023-09-22T22:47:00.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.101 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.101 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.101 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.101 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.101 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.102 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs= 2023-09-22T22:47:00.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs=', 'Content-Length': '0'} 2023-09-22T22:47:00.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006693fe62e1f5a16e-00650e1105-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64obj111000falseobj112023-09-22T22:10:52.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?prefix=obj11 2023-09-22T22:47:00.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?prefix=obj11 2023-09-22T22:47:00.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?prefix=obj11 2023-09-22T22:47:00.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.103 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.103 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.103 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.104 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.104 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.104 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs= 2023-09-22T22:47:00.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs=', 'Content-Length': '0'} 2023-09-22T22:47:00.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0ebe320f96c9f44-00650e1105-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64obj111000falseobj112023-09-22T22:10:52.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.104 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-09-22T22:47:00.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj22 2023-09-22T22:47:00.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj22 2023-09-22T22:47:00.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.105 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.106 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.106 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.106 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.106 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.106 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:ROBOZpWv40872mMlG8N3bfx+Z6c= 2023-09-22T22:47:00.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ROBOZpWv40872mMlG8N3bfx+Z6c=', 'Content-Length': '0'} 2023-09-22T22:47:00.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004558cb9f7bdec855-00650e1105-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj22 2023-09-22T22:47:00.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj22 2023-09-22T22:47:00.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.107 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.108 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.108 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.108 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.108 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.108 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:ROBOZpWv40872mMlG8N3bfx+Z6c= 2023-09-22T22:47:00.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ROBOZpWv40872mMlG8N3bfx+Z6c=', 'Content-Length': '0'} 2023-09-22T22:47:00.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067c1d0c82bde647b-00650e1105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj22 2023-09-22T22:47:00.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj22 2023-09-22T22:47:00.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.109 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.109 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.110 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.110 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.110 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.110 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:jPzYE2Z7psQmD1uzQMxqQR/NBHs= 2023-09-22T22:47:00.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jPzYE2Z7psQmD1uzQMxqQR/NBHs=', 'Content-Length': '0'} 2023-09-22T22:47:00.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5200ff1037059c3-00650e1105-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj22 2023-09-22T22:47:00.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj22 2023-09-22T22:47:00.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.111 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.111 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.112 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.112 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.112 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.112 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:jPzYE2Z7psQmD1uzQMxqQR/NBHs= 2023-09-22T22:47:00.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jPzYE2Z7psQmD1uzQMxqQR/NBHs=', 'Content-Length': '0'} 2023-09-22T22:47:00.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d9c7d09a33ab01a-00650e1105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.112 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-09-22T22:47:00.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj22 2023-09-22T22:47:00.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj22 2023-09-22T22:47:00.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.113 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.113 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.114 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.114 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.114 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.114 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:ROBOZpWv40872mMlG8N3bfx+Z6c= 2023-09-22T22:47:00.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ROBOZpWv40872mMlG8N3bfx+Z6c=', 'Content-Length': '0'} 2023-09-22T22:47:00.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bcf2a0a8c66eb4d-00650e1105-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj22 2023-09-22T22:47:00.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj22 2023-09-22T22:47:00.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.115 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.116 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.116 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.116 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.116 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj22?versionId=null 2023-09-22T22:47:00.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.116 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:ROBOZpWv40872mMlG8N3bfx+Z6c= 2023-09-22T22:47:00.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ROBOZpWv40872mMlG8N3bfx+Z6c=', 'Content-Length': '0'} 2023-09-22T22:47:00.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091876ed8dee1768b-00650e1105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?prefix=obj22 2023-09-22T22:47:00.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?prefix=obj22 2023-09-22T22:47:00.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?prefix=obj22 2023-09-22T22:47:00.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.117 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.118 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.118 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.118 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.118 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.118 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs= 2023-09-22T22:47:00.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs=', 'Content-Length': '0'} 2023-09-22T22:47:00.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000863f58de38cb9f6a-00650e1105-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64obj221000falseobj222023-09-22T22:10:52.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?prefix=obj22 2023-09-22T22:47:00.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?prefix=obj22 2023-09-22T22:47:00.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?prefix=obj22 2023-09-22T22:47:00.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.120 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.120 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.120 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.120 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.120 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.120 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs= 2023-09-22T22:47:00.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs=', 'Content-Length': '0'} 2023-09-22T22:47:00.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d818191acbd380f-00650e1105-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64obj221000falseobj222023-09-22T22:10:52.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj22 2023-09-22T22:47:00.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj22 2023-09-22T22:47:00.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj22 2023-09-22T22:47:00.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.121 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.121 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.122 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.122 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.122 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj22 2023-09-22T22:47:00.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.122 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:L0y0XpyXOV5C7XhRCyMgsrwiqd8= 2023-09-22T22:47:00.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:L0y0XpyXOV5C7XhRCyMgsrwiqd8=', 'Content-Length': '0'} 2023-09-22T22:47:00.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6d7ce0bc8d47c35-00650e1105-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj22 2023-09-22T22:47:00.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj22 2023-09-22T22:47:00.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj22 2023-09-22T22:47:00.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.123 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.123 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.123 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.123 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.123 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj22 2023-09-22T22:47:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.124 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:L0y0XpyXOV5C7XhRCyMgsrwiqd8= 2023-09-22T22:47:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:L0y0XpyXOV5C7XhRCyMgsrwiqd8=', 'Content-Length': '0'} 2023-09-22T22:47:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f56aea8ea7166d5-00650e1105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.124 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-09-22T22:47:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj22 2023-09-22T22:47:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj22 2023-09-22T22:47:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj22 2023-09-22T22:47:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.125 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.125 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.125 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.125 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.125 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj22 2023-09-22T22:47:00.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.126 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:VRM6bIR0b/eaR4udd9WdLZ14Vv0= 2023-09-22T22:47:00.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VRM6bIR0b/eaR4udd9WdLZ14Vv0=', 'Content-Length': '0'} 2023-09-22T22:47:00.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000823eea96d83d2424-00650e1105-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj22 2023-09-22T22:47:00.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj22 2023-09-22T22:47:00.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj22 2023-09-22T22:47:00.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.127 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.127 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.127 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.127 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.127 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj22 2023-09-22T22:47:00.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.127 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:VRM6bIR0b/eaR4udd9WdLZ14Vv0= 2023-09-22T22:47:00.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:VRM6bIR0b/eaR4udd9WdLZ14Vv0=', 'Content-Length': '0'} 2023-09-22T22:47:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fde57c1cd2dc8df3-00650e1105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?prefix=obj22 2023-09-22T22:47:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?prefix=obj22 2023-09-22T22:47:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?prefix=obj22 2023-09-22T22:47:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.129 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.129 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.129 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.129 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:17 GMT 2023-09-22T22:47:00.129 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.129 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs= 2023-09-22T22:47:00.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:17 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rlk9u5HO5f9UxBC/IbmhAKTYHIs=', 'Content-Length': '0'} 2023-09-22T22:47:00.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4a36318d2f336b4-00650e1105-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64obj221000falseobj222023-09-22T22:10:52.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?prefix=obj22 2023-09-22T22:47:00.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?prefix=obj22 2023-09-22T22:47:00.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?prefix=obj22 2023-09-22T22:47:00.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.131 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.131 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.131 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.131 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.131 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.131 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:47:00.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:47:00.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a17d1d72343e8bc9-00650e1106-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64obj221000falseobj222023-09-22T22:10:52.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.132 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-09-22T22:47:00.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj33 2023-09-22T22:47:00.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj33 2023-09-22T22:47:00.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.133 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.133 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.133 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.133 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.133 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.133 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:h/sE5s8+uqC0h7kQymg9eoylTJQ= 2023-09-22T22:47:00.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h/sE5s8+uqC0h7kQymg9eoylTJQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e132d24b1a50e3cf-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj33 2023-09-22T22:47:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj33 2023-09-22T22:47:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.135 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.135 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.135 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.135 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.135 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.135 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:h/sE5s8+uqC0h7kQymg9eoylTJQ= 2023-09-22T22:47:00.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h/sE5s8+uqC0h7kQymg9eoylTJQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f240b5af23d4815-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj33 2023-09-22T22:47:00.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj33 2023-09-22T22:47:00.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.137 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.137 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.137 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.137 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.137 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.137 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:gS8ZgG27cZOVxGRWfVbj0tuZ7Nc= 2023-09-22T22:47:00.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gS8ZgG27cZOVxGRWfVbj0tuZ7Nc=', 'Content-Length': '0'} 2023-09-22T22:47:00.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d874b3b249b3f94-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj33 2023-09-22T22:47:00.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj33 2023-09-22T22:47:00.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.139 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.139 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.139 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.139 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.139 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.139 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:gS8ZgG27cZOVxGRWfVbj0tuZ7Nc= 2023-09-22T22:47:00.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gS8ZgG27cZOVxGRWfVbj0tuZ7Nc=', 'Content-Length': '0'} 2023-09-22T22:47:00.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acb5eb1d03e88bba-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.139 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-09-22T22:47:00.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj33 2023-09-22T22:47:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj33 2023-09-22T22:47:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.141 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.141 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.141 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.141 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.141 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.141 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:h/sE5s8+uqC0h7kQymg9eoylTJQ= 2023-09-22T22:47:00.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h/sE5s8+uqC0h7kQymg9eoylTJQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008b4ae1ef9547c8a-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj33 2023-09-22T22:47:00.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj33 2023-09-22T22:47:00.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.143 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.143 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.143 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.143 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.143 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj33?versionId=null 2023-09-22T22:47:00.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.143 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:h/sE5s8+uqC0h7kQymg9eoylTJQ= 2023-09-22T22:47:00.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:h/sE5s8+uqC0h7kQymg9eoylTJQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbae06ba4dc22f1c-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?prefix=obj33 2023-09-22T22:47:00.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?prefix=obj33 2023-09-22T22:47:00.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?prefix=obj33 2023-09-22T22:47:00.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.145 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.145 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.145 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.145 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.145 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.145 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:47:00.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:47:00.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e601812d41a057b-00650e1106-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64obj331000falseobj332023-09-22T22:10:52.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?prefix=obj33 2023-09-22T22:47:00.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?prefix=obj33 2023-09-22T22:47:00.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?prefix=obj33 2023-09-22T22:47:00.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.147 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.147 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.147 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.147 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.147 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.147 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:47:00.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:47:00.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c41d94e5d3c5ad3-00650e1106-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64obj331000falseobj332023-09-22T22:10:52.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj33 2023-09-22T22:47:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj33 2023-09-22T22:47:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj33 2023-09-22T22:47:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.149 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.149 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.149 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.149 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.149 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj33 2023-09-22T22:47:00.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.149 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:0p86mvaEnhR35P9nBEliaWuLsTQ= 2023-09-22T22:47:00.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0p86mvaEnhR35P9nBEliaWuLsTQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000936931d4f0eac951-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj33 2023-09-22T22:47:00.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj33 2023-09-22T22:47:00.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj33 2023-09-22T22:47:00.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.150 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.150 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.151 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.151 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.151 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj33 2023-09-22T22:47:00.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.151 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:0p86mvaEnhR35P9nBEliaWuLsTQ= 2023-09-22T22:47:00.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:0p86mvaEnhR35P9nBEliaWuLsTQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3ecdfa7b8ee644f-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.151 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-09-22T22:47:00.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj33 2023-09-22T22:47:00.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj33 2023-09-22T22:47:00.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj33 2023-09-22T22:47:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.152 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.152 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.152 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.152 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.153 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj33 2023-09-22T22:47:00.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.153 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:TXAsBiDixehk/DPj8QFyPlMmXLo= 2023-09-22T22:47:00.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TXAsBiDixehk/DPj8QFyPlMmXLo=', 'Content-Length': '0'} 2023-09-22T22:47:00.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000227c7baa2b4463cf-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj33 2023-09-22T22:47:00.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj33 2023-09-22T22:47:00.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj33 2023-09-22T22:47:00.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.154 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.154 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.154 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.154 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.154 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj33 2023-09-22T22:47:00.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.155 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:TXAsBiDixehk/DPj8QFyPlMmXLo= 2023-09-22T22:47:00.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:TXAsBiDixehk/DPj8QFyPlMmXLo=', 'Content-Length': '0'} 2023-09-22T22:47:00.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001db047f2c9c68558-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?prefix=obj33 2023-09-22T22:47:00.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?prefix=obj33 2023-09-22T22:47:00.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?prefix=obj33 2023-09-22T22:47:00.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.156 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.156 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.156 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.156 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.156 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.156 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:47:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:47:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000981ab63a22fb03f9-00650e1106-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64obj331000falseobj332023-09-22T22:10:52.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?prefix=obj33 2023-09-22T22:47:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?prefix=obj33 2023-09-22T22:47:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?prefix=obj33 2023-09-22T22:47:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.158 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.158 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.158 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.158 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.158 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.158 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:47:00.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:47:00.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b40a942474e045ea-00650e1106-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64obj331000falseobj332023-09-22T22:10:52.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.159 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-09-22T22:47:00.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj44 2023-09-22T22:47:00.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj44 2023-09-22T22:47:00.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.160 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.160 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.160 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.160 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.160 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.161 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:yXb8zIL4PX0AVgFTslA41mui64s= 2023-09-22T22:47:00.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yXb8zIL4PX0AVgFTslA41mui64s=', 'Content-Length': '0'} 2023-09-22T22:47:00.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec77702b78f073c3-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj44 2023-09-22T22:47:00.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj44 2023-09-22T22:47:00.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.162 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.162 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.162 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.162 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.162 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.162 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:yXb8zIL4PX0AVgFTslA41mui64s= 2023-09-22T22:47:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yXb8zIL4PX0AVgFTslA41mui64s=', 'Content-Length': '0'} 2023-09-22T22:47:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b563afac71d444f-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj44 2023-09-22T22:47:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj44 2023-09-22T22:47:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.164 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.164 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.164 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.164 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.164 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.164 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:EFrz8+M7D/L5kogGoKGkduN5PAE= 2023-09-22T22:47:00.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EFrz8+M7D/L5kogGoKGkduN5PAE=', 'Content-Length': '0'} 2023-09-22T22:47:00.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000477d7a94e000bc59-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj44 2023-09-22T22:47:00.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj44 2023-09-22T22:47:00.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.166 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.166 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.166 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.166 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.166 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.166 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:EFrz8+M7D/L5kogGoKGkduN5PAE= 2023-09-22T22:47:00.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:EFrz8+M7D/L5kogGoKGkduN5PAE=', 'Content-Length': '0'} 2023-09-22T22:47:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e768442f3f1ebdb-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.167 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-09-22T22:47:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj44 2023-09-22T22:47:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj44 2023-09-22T22:47:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.168 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.168 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.168 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.168 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.168 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.168 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:yXb8zIL4PX0AVgFTslA41mui64s= 2023-09-22T22:47:00.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yXb8zIL4PX0AVgFTslA41mui64s=', 'Content-Length': '0'} 2023-09-22T22:47:00.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005abf7f3f3979aa89-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj44 2023-09-22T22:47:00.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj44 2023-09-22T22:47:00.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.170 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.170 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.170 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.170 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.170 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj44?versionId=null 2023-09-22T22:47:00.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.170 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:yXb8zIL4PX0AVgFTslA41mui64s= 2023-09-22T22:47:00.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:yXb8zIL4PX0AVgFTslA41mui64s=', 'Content-Length': '0'} 2023-09-22T22:47:00.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000566ab893974e345f-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?prefix=obj44 2023-09-22T22:47:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?prefix=obj44 2023-09-22T22:47:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?prefix=obj44 2023-09-22T22:47:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.172 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.172 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.172 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.172 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.172 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.172 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:47:00.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:47:00.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba935293d1da6759-00650e1106-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64obj441000falseobj442023-09-22T22:10:52.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?prefix=obj44 2023-09-22T22:47:00.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?prefix=obj44 2023-09-22T22:47:00.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?prefix=obj44 2023-09-22T22:47:00.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.174 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.174 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.174 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.174 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.174 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.174 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:47:00.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:47:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ed3458085a37b8f-00650e1106-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64obj441000falseobj442023-09-22T22:10:52.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj44 2023-09-22T22:47:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj44 2023-09-22T22:47:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj44 2023-09-22T22:47:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.176 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.176 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.176 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.176 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.176 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj44 2023-09-22T22:47:00.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.176 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:jeDa/cqB1y93s2UXJDHALNQSh8U= 2023-09-22T22:47:00.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jeDa/cqB1y93s2UXJDHALNQSh8U=', 'Content-Length': '0'} 2023-09-22T22:47:00.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001995f0bd9f892cd1-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj44 2023-09-22T22:47:00.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj44 2023-09-22T22:47:00.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj44 2023-09-22T22:47:00.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.177 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.178 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.178 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.178 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.178 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj44 2023-09-22T22:47:00.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.178 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:jeDa/cqB1y93s2UXJDHALNQSh8U= 2023-09-22T22:47:00.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:jeDa/cqB1y93s2UXJDHALNQSh8U=', 'Content-Length': '0'} 2023-09-22T22:47:00.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d12dfdfd0047492f-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.178 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-09-22T22:47:00.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj44 2023-09-22T22:47:00.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj44 2023-09-22T22:47:00.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj44 2023-09-22T22:47:00.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.179 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.179 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.179 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.180 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.180 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj44 2023-09-22T22:47:00.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.180 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:osifHDik/HLbMKBiy7FXtBP1jxI= 2023-09-22T22:47:00.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:osifHDik/HLbMKBiy7FXtBP1jxI=', 'Content-Length': '0'} 2023-09-22T22:47:00.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048c66e3d93c6a785-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj44 2023-09-22T22:47:00.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj44 2023-09-22T22:47:00.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj44 2023-09-22T22:47:00.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.181 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.181 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.181 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.181 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.181 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj44 2023-09-22T22:47:00.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.182 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:osifHDik/HLbMKBiy7FXtBP1jxI= 2023-09-22T22:47:00.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:osifHDik/HLbMKBiy7FXtBP1jxI=', 'Content-Length': '0'} 2023-09-22T22:47:00.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000824e71dc4ef2cecb-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?prefix=obj44 2023-09-22T22:47:00.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?prefix=obj44 2023-09-22T22:47:00.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?prefix=obj44 2023-09-22T22:47:00.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.183 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.183 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.183 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.183 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.183 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.183 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:47:00.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:47:00.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000505d4c4cf904ce49-00650e1106-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64obj441000falseobj442023-09-22T22:10:52.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?prefix=obj44 2023-09-22T22:47:00.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?prefix=obj44 2023-09-22T22:47:00.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?prefix=obj44 2023-09-22T22:47:00.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.185 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.185 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.185 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.185 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.185 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.185 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:47:00.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:47:00.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a47f2c7b20888279-00650e1106-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64obj441000falseobj442023-09-22T22:10:52.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.186 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-09-22T22:47:00.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj55 2023-09-22T22:47:00.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj55 2023-09-22T22:47:00.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.187 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.187 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.187 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.187 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.187 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.188 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:qfw5ymVdzxsE7OnZlBe3N/gB3PM= 2023-09-22T22:47:00.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qfw5ymVdzxsE7OnZlBe3N/gB3PM=', 'Content-Length': '0'} 2023-09-22T22:47:00.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f592ec8c2b881425-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj55 2023-09-22T22:47:00.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj55 2023-09-22T22:47:00.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.189 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.189 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.189 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.189 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.189 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.189 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:qfw5ymVdzxsE7OnZlBe3N/gB3PM= 2023-09-22T22:47:00.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qfw5ymVdzxsE7OnZlBe3N/gB3PM=', 'Content-Length': '0'} 2023-09-22T22:47:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ede9fd95026a523e-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj55 2023-09-22T22:47:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj55 2023-09-22T22:47:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.191 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.191 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.191 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.191 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.191 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.191 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kygRCK9S92gFrL9ECLO/EiV5Lg0= 2023-09-22T22:47:00.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kygRCK9S92gFrL9ECLO/EiV5Lg0=', 'Content-Length': '0'} 2023-09-22T22:47:00.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b62a92c5f16c8747-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj55 2023-09-22T22:47:00.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj55 2023-09-22T22:47:00.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.193 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.193 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.193 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.193 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.193 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.193 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:kygRCK9S92gFrL9ECLO/EiV5Lg0= 2023-09-22T22:47:00.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:kygRCK9S92gFrL9ECLO/EiV5Lg0=', 'Content-Length': '0'} 2023-09-22T22:47:00.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057f94b80ffb4371f-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.194 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-09-22T22:47:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj55 2023-09-22T22:47:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj55 2023-09-22T22:47:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.195 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.195 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.195 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.195 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.195 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.195 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:qfw5ymVdzxsE7OnZlBe3N/gB3PM= 2023-09-22T22:47:00.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qfw5ymVdzxsE7OnZlBe3N/gB3PM=', 'Content-Length': '0'} 2023-09-22T22:47:00.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083be85d1138ef417-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj55 2023-09-22T22:47:00.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj55 2023-09-22T22:47:00.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.197 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.197 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.197 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.197 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.197 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj55?versionId=null 2023-09-22T22:47:00.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.197 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:qfw5ymVdzxsE7OnZlBe3N/gB3PM= 2023-09-22T22:47:00.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:qfw5ymVdzxsE7OnZlBe3N/gB3PM=', 'Content-Length': '0'} 2023-09-22T22:47:00.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed46ca68500dc13a-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?prefix=obj55 2023-09-22T22:47:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?prefix=obj55 2023-09-22T22:47:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?prefix=obj55 2023-09-22T22:47:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.199 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.199 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.199 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.199 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.199 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.199 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:47:00.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:47:00.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007720670c841af18c-00650e1106-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64obj551000falseobj552023-09-22T22:10:52.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?prefix=obj55 2023-09-22T22:47:00.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?prefix=obj55 2023-09-22T22:47:00.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?prefix=obj55 2023-09-22T22:47:00.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.201 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.201 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.201 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.201 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.201 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.201 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:47:00.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:47:00.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b41754d97f2c9949-00650e1106-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64obj551000falseobj552023-09-22T22:10:52.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj55 2023-09-22T22:47:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj55 2023-09-22T22:47:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj55 2023-09-22T22:47:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.203 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.203 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.203 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.203 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.203 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj55 2023-09-22T22:47:00.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.203 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:pecKVGx4oXRPPnAi9iK1/tbmTdk= 2023-09-22T22:47:00.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pecKVGx4oXRPPnAi9iK1/tbmTdk=', 'Content-Length': '0'} 2023-09-22T22:47:00.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2404bb52d56e978-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj55 2023-09-22T22:47:00.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj55 2023-09-22T22:47:00.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj55 2023-09-22T22:47:00.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.204 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.204 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.205 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.205 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.205 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj55 2023-09-22T22:47:00.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.205 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:pecKVGx4oXRPPnAi9iK1/tbmTdk= 2023-09-22T22:47:00.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:pecKVGx4oXRPPnAi9iK1/tbmTdk=', 'Content-Length': '0'} 2023-09-22T22:47:00.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004aa0076d7610814b-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.205 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-09-22T22:47:00.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj55 2023-09-22T22:47:00.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj55 2023-09-22T22:47:00.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj55 2023-09-22T22:47:00.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.206 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.206 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.206 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.206 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.207 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj55 2023-09-22T22:47:00.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.207 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:wFqrtU41sGomtABbuMFt4q8MokQ= 2023-09-22T22:47:00.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wFqrtU41sGomtABbuMFt4q8MokQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f132bdb9-20a7-4073-a932-8f99da20691e:ffzdqu-64:f132bdb9-20a7-4073-a932-8f99da20691e.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d23acbc3db479ac9-00650e1106-1fae-a2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj55 2023-09-22T22:47:00.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj55 2023-09-22T22:47:00.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj55 2023-09-22T22:47:00.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.208 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.208 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.208 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.208 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.208 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj55 2023-09-22T22:47:00.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.208 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:wFqrtU41sGomtABbuMFt4q8MokQ= 2023-09-22T22:47:00.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wFqrtU41sGomtABbuMFt4q8MokQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 22 Sep 2023 22:10:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d441ebca96ba323e-00650e1106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?prefix=obj55 2023-09-22T22:47:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?prefix=obj55 2023-09-22T22:47:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?prefix=obj55 2023-09-22T22:47:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-22T22:47:00.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.210 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.210 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.210 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.210 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.210 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.210 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:47:00.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:47:00.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b709c4d1688ce1ff-00650e1106-1fae-a2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64obj551000falseobj552023-09-22T22:10:52.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/?prefix=obj55 2023-09-22T22:47:00.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/?prefix=obj55 2023-09-22T22:47:00.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-22T22:47:00.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/?prefix=obj55 2023-09-22T22:47:00.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.212 INFO:tasks.rgw_multisite_tests:GET 2023-09-22T22:47:00.212 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.212 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.212 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:18 GMT 2023-09-22T22:47:00.212 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.212 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA= 2023-09-22T22:47:00.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:18 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:q9jOM8bisuepqZE6Syua7m9dkyA=', 'Content-Length': '0'} 2023-09-22T22:47:00.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005398374f8e4f07d2-00650e1106-165f-a1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ffzdqu-64obj551000falseobj552023-09-22T22:10:52.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-22T22:47:00.213 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=ffzdqu-64 zones={a2, a1} 2023-09-22T22:47:00.213 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: DEBUG: waiting for process to exit 2023-09-22T22:47:00.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2023-09-22T22:47:00.213 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:11:18.922+0000 7f2d6be57640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 85618) UID: 0 2023-09-22T22:47:00.213 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:11:18.922+0000 7f2d6f69ea40 -1 shutting down 2023-09-22T22:47:00.213 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7f2d4a614640 time 2023-09-22T22:11:18.952159+0000 2023-09-22T22:47:00.213 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T22:47:00.213 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:47:00.213 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f2d731c2883] 2023-09-22T22:47:00.214 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f2d731c2a37] 2023-09-22T22:47:00.214 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x55698763b46c] 2023-09-22T22:47:00.214 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x556987a1d910] 2023-09-22T22:47:00.214 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x55698764756e] 2023-09-22T22:47:00.214 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x556987648e7b] 2023-09-22T22:47:00.214 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x556987649ced] 2023-09-22T22:47:00.214 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x556987a50712] 2023-09-22T22:47:00.214 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 9: radosgw(+0x836db8) [0x5569877fbdb8] 2023-09-22T22:47:00.214 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x5569877fc753] 2023-09-22T22:47:00.214 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2d71e91b43] 2023-09-22T22:47:00.214 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2d71f23a00] 2023-09-22T22:47:00.215 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:11:18.950+0000 7f2d4a614640 -1 ./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7f2d4a614640 time 2023-09-22T22:11:18.952159+0000 2023-09-22T22:47:00.215 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T22:47:00.215 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:47:00.215 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:47:00.215 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f2d731c2883] 2023-09-22T22:47:00.215 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f2d731c2a37] 2023-09-22T22:47:00.215 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x55698763b46c] 2023-09-22T22:47:00.215 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x556987a1d910] 2023-09-22T22:47:00.215 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x55698764756e] 2023-09-22T22:47:00.215 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x556987648e7b] 2023-09-22T22:47:00.215 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x556987649ced] 2023-09-22T22:47:00.215 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x556987a50712] 2023-09-22T22:47:00.216 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 9: radosgw(+0x836db8) [0x5569877fbdb8] 2023-09-22T22:47:00.216 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x5569877fc753] 2023-09-22T22:47:00.216 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2d71e91b43] 2023-09-22T22:47:00.216 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2d71f23a00] 2023-09-22T22:47:00.216 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:47:00.216 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: *** Caught signal (Aborted) ** 2023-09-22T22:47:00.216 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: in thread 7f2d4a614640 thread_name:data-sync 2023-09-22T22:47:00.216 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:47:00.216 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2d71e3f520] 2023-09-22T22:47:00.216 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2: pthread_kill() 2023-09-22T22:47:00.216 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 3: raise() 2023-09-22T22:47:00.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 4: abort() 2023-09-22T22:47:00.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f2d731c28dd] 2023-09-22T22:47:00.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f2d731c2a37] 2023-09-22T22:47:00.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x55698763b46c] 2023-09-22T22:47:00.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x556987a1d910] 2023-09-22T22:47:00.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x55698764756e] 2023-09-22T22:47:00.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x556987648e7b] 2023-09-22T22:47:00.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x556987649ced] 2023-09-22T22:47:00.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x556987a50712] 2023-09-22T22:47:00.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 13: radosgw(+0x836db8) [0x5569877fbdb8] 2023-09-22T22:47:00.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x5569877fc753] 2023-09-22T22:47:00.218 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2d71e91b43] 2023-09-22T22:47:00.218 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2d71f23a00] 2023-09-22T22:47:00.218 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:11:18.954+0000 7f2d4a614640 -1 *** Caught signal (Aborted) ** 2023-09-22T22:47:00.218 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: in thread 7f2d4a614640 thread_name:data-sync 2023-09-22T22:47:00.218 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:47:00.218 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:47:00.218 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2d71e3f520] 2023-09-22T22:47:00.218 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2: pthread_kill() 2023-09-22T22:47:00.218 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 3: raise() 2023-09-22T22:47:00.218 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 4: abort() 2023-09-22T22:47:00.218 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f2d731c28dd] 2023-09-22T22:47:00.219 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f2d731c2a37] 2023-09-22T22:47:00.219 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x55698763b46c] 2023-09-22T22:47:00.219 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x556987a1d910] 2023-09-22T22:47:00.219 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x55698764756e] 2023-09-22T22:47:00.219 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x556987648e7b] 2023-09-22T22:47:00.219 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x556987649ced] 2023-09-22T22:47:00.219 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x556987a50712] 2023-09-22T22:47:00.219 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 13: radosgw(+0x836db8) [0x5569877fbdb8] 2023-09-22T22:47:00.219 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x5569877fc753] 2023-09-22T22:47:00.219 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2d71e91b43] 2023-09-22T22:47:00.219 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2d71f23a00] 2023-09-22T22:47:00.220 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-09-22T22:47:00.220 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:47:00.220 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: -89> 2023-09-22T22:11:18.922+0000 7f2d6be57640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 85618) UID: 0 2023-09-22T22:47:00.220 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: -86> 2023-09-22T22:11:18.922+0000 7f2d6f69ea40 -1 shutting down 2023-09-22T22:47:00.220 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: -1> 2023-09-22T22:11:18.950+0000 7f2d4a614640 -1 ./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7f2d4a614640 time 2023-09-22T22:11:18.952159+0000 2023-09-22T22:47:00.220 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T22:47:00.220 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:47:00.220 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:47:00.220 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f2d731c2883] 2023-09-22T22:47:00.220 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f2d731c2a37] 2023-09-22T22:47:00.220 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x55698763b46c] 2023-09-22T22:47:00.220 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x556987a1d910] 2023-09-22T22:47:00.221 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x55698764756e] 2023-09-22T22:47:00.221 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x556987648e7b] 2023-09-22T22:47:00.221 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x556987649ced] 2023-09-22T22:47:00.221 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x556987a50712] 2023-09-22T22:47:00.221 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 9: radosgw(+0x836db8) [0x5569877fbdb8] 2023-09-22T22:47:00.221 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x5569877fc753] 2023-09-22T22:47:00.221 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2d71e91b43] 2023-09-22T22:47:00.221 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2d71f23a00] 2023-09-22T22:47:00.221 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:47:00.221 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 0> 2023-09-22T22:11:18.954+0000 7f2d4a614640 -1 *** Caught signal (Aborted) ** 2023-09-22T22:47:00.221 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: in thread 7f2d4a614640 thread_name:data-sync 2023-09-22T22:47:00.222 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:47:00.222 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:47:00.222 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2d71e3f520] 2023-09-22T22:47:00.222 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2: pthread_kill() 2023-09-22T22:47:00.222 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 3: raise() 2023-09-22T22:47:00.222 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 4: abort() 2023-09-22T22:47:00.222 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f2d731c28dd] 2023-09-22T22:47:00.222 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f2d731c2a37] 2023-09-22T22:47:00.222 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x55698763b46c] 2023-09-22T22:47:00.222 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x556987a1d910] 2023-09-22T22:47:00.222 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x55698764756e] 2023-09-22T22:47:00.222 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x556987648e7b] 2023-09-22T22:47:00.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x556987649ced] 2023-09-22T22:47:00.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x556987a50712] 2023-09-22T22:47:00.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 13: radosgw(+0x836db8) [0x5569877fbdb8] 2023-09-22T22:47:00.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x5569877fc753] 2023-09-22T22:47:00.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2d71e91b43] 2023-09-22T22:47:00.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2d71f23a00] 2023-09-22T22:47:00.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-09-22T22:47:00.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:47:00.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stderr: INFO: daemon-helper: command crashed with signal 11 2023-09-22T22:47:00.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Stopped 2023-09-22T22:47:00.223 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.224 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-64 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: tenant: 2023-09-22T22:47:00.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: bucket name: ffzdqu-64 2023-09-22T22:47:00.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: total entries: 5 2023-09-22T22:47:00.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.225 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.225 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.225 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.225 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.225 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.226 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:47:00.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000864eb46c1915c086-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj17 2023-09-22T22:47:00.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj17 2023-09-22T22:47:00.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj17 2023-09-22T22:47:00.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.227 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.227 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.227 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.227 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.227 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj17 2023-09-22T22:47:00.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.227 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:51G4MNs9+OzGbgzOmOvL1XUTpLE= 2023-09-22T22:47:00.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:51G4MNs9+OzGbgzOmOvL1XUTpLE='} 2023-09-22T22:47:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf91fd22b4bf38df-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.229 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.229 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.229 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.229 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.229 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.229 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:47:00.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000088e3f00435b806c-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj34 2023-09-22T22:47:00.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj34 2023-09-22T22:47:00.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj34 2023-09-22T22:47:00.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.230 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.230 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.231 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.231 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.231 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj34 2023-09-22T22:47:00.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.231 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:W3TunPRVqW0DlVdCaHyK2EI581A= 2023-09-22T22:47:00.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:W3TunPRVqW0DlVdCaHyK2EI581A='} 2023-09-22T22:47:00.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007dc371995da22c1d-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.232 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.232 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.232 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.232 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.232 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.233 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:47:00.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083529939ba4b095d-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj51 2023-09-22T22:47:00.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj51 2023-09-22T22:47:00.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj51 2023-09-22T22:47:00.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.234 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.234 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.234 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.234 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.234 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj51 2023-09-22T22:47:00.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.234 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:hcHoliysQcpRPVLpcfQi1p99BnE= 2023-09-22T22:47:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:hcHoliysQcpRPVLpcfQi1p99BnE='} 2023-09-22T22:47:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000405e1def44873cb4-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.236 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.236 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.236 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.236 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.236 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.236 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:47:00.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc66fc8148fe6270-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj68 2023-09-22T22:47:00.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj68 2023-09-22T22:47:00.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj68 2023-09-22T22:47:00.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.237 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.237 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.238 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.238 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.238 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj68 2023-09-22T22:47:00.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.238 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:gxqClw28bKlBLOow19ycMImRyuI= 2023-09-22T22:47:00.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:gxqClw28bKlBLOow19ycMImRyuI='} 2023-09-22T22:47:00.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f87bea913d048ad-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.239 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.239 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.239 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.239 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.239 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.240 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:47:00.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000012d4da905d630b5-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj85 2023-09-22T22:47:00.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj85 2023-09-22T22:47:00.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj85 2023-09-22T22:47:00.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.241 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.241 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.241 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.241 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.241 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj85 2023-09-22T22:47:00.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.241 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:duzIIbxW2hlats4YvsHvdDyhbV0= 2023-09-22T22:47:00.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:duzIIbxW2hlats4YvsHvdDyhbV0='} 2023-09-22T22:47:00.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4fde8efff9b9056-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.242 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.242 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-64 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: tenant: 2023-09-22T22:47:00.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: bucket name: ffzdqu-64 2023-09-22T22:47:00.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: total entries: 10 2023-09-22T22:47:00.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.243 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.243 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.244 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.244 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.244 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.244 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:47:00.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4c000295f98c39f-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj19 2023-09-22T22:47:00.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj19 2023-09-22T22:47:00.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj19 2023-09-22T22:47:00.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.245 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.245 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.245 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.245 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.245 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj19 2023-09-22T22:47:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.246 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:bEXEWPjjZg3gs0rhY0WV2HZjOHg= 2023-09-22T22:47:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:bEXEWPjjZg3gs0rhY0WV2HZjOHg='} 2023-09-22T22:47:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000351b61a2dce15be3-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.247 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.247 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.247 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.247 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.247 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.247 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:47:00.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000316c4f87b60209a8-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj38 2023-09-22T22:47:00.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj38 2023-09-22T22:47:00.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj38 2023-09-22T22:47:00.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.249 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.249 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.249 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.249 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.249 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj38 2023-09-22T22:47:00.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.249 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:OW1u814/jS6GQDR1MV4eezb4N/Y= 2023-09-22T22:47:00.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:OW1u814/jS6GQDR1MV4eezb4N/Y='} 2023-09-22T22:47:00.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008624fce5341c9e55-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.250 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.251 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.251 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.251 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.251 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.251 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:47:00.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b764dcfc203f9e25-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj57 2023-09-22T22:47:00.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj57 2023-09-22T22:47:00.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj57 2023-09-22T22:47:00.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.252 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.252 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.252 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.252 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.252 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj57 2023-09-22T22:47:00.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.253 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:w3NizSev4uGnw37FgQOnCmfEFyM= 2023-09-22T22:47:00.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:w3NizSev4uGnw37FgQOnCmfEFyM='} 2023-09-22T22:47:00.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d8e6063267c904b-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.254 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.254 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.254 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.254 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.254 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.254 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:47:00.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a06beaec186b5df5-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj76 2023-09-22T22:47:00.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj76 2023-09-22T22:47:00.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj76 2023-09-22T22:47:00.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.256 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.256 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.256 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.256 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.256 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj76 2023-09-22T22:47:00.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.256 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:f5JUBsV7GuxYpUHTc6a2AmvawDs= 2023-09-22T22:47:00.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:f5JUBsV7GuxYpUHTc6a2AmvawDs='} 2023-09-22T22:47:00.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed1bc2e33ec443f6-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.257 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.258 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.258 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.258 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.258 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.258 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:47:00.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c705ed9b90557d6-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj95 2023-09-22T22:47:00.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj95 2023-09-22T22:47:00.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj95 2023-09-22T22:47:00.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.259 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.259 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.259 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.259 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.260 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj95 2023-09-22T22:47:00.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.260 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:M88i8El7nKMgLfKYmB+L411DuV4= 2023-09-22T22:47:00.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:M88i8El7nKMgLfKYmB+L411DuV4='} 2023-09-22T22:47:00.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a9110a0c9734991-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.260 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.260 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-64 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: tenant: 2023-09-22T22:47:00.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: bucket name: ffzdqu-64 2023-09-22T22:47:00.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: total entries: 15 2023-09-22T22:47:00.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.262 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.262 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.262 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.262 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.262 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.262 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:47:00.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006cf2e82e4b085f4a-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj23 2023-09-22T22:47:00.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj23 2023-09-22T22:47:00.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj23 2023-09-22T22:47:00.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.263 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.263 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.264 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.264 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.264 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj23 2023-09-22T22:47:00.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.264 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:myc2/IT0K5WD8RbZ9jzBHD6mVaQ= 2023-09-22T22:47:00.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:myc2/IT0K5WD8RbZ9jzBHD6mVaQ='} 2023-09-22T22:47:00.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a77873d90c1c40b2-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.265 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.265 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.265 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.265 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.265 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.266 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:47:00.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060b28c2ed159fc5e-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj46 2023-09-22T22:47:00.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj46 2023-09-22T22:47:00.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj46 2023-09-22T22:47:00.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.267 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.267 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.267 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.267 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.267 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj46 2023-09-22T22:47:00.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.267 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:3CioJa78pcejkhCmkdl52hUwsgo= 2023-09-22T22:47:00.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:3CioJa78pcejkhCmkdl52hUwsgo='} 2023-09-22T22:47:00.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5e2e1684cf0bb87-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.270 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.270 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.270 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.270 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.270 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.270 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ= 2023-09-22T22:47:00.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:t08Y5ocFkF1NOpvg79zMkDRLAJQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008cc678f64147350c-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj69 2023-09-22T22:47:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj69 2023-09-22T22:47:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj69 2023-09-22T22:47:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.272 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.272 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.272 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.272 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:19 GMT 2023-09-22T22:47:00.272 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj69 2023-09-22T22:47:00.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.272 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:cpWSblEFkwFoIAIKskB5tWcPoHc= 2023-09-22T22:47:00.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:19 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:cpWSblEFkwFoIAIKskB5tWcPoHc='} 2023-09-22T22:47:00.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eea9cb61e02bb745-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.273 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.273 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.274 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.274 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.274 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.274 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:47:00.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:47:00.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7cb73e51d20687f-00650e1107-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj92 2023-09-22T22:47:00.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj92 2023-09-22T22:47:00.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj92 2023-09-22T22:47:00.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.275 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.275 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.275 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.275 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.276 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj92 2023-09-22T22:47:00.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.276 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:YhunsMRnF/ODwoEAUMwU399wz/Y= 2023-09-22T22:47:00.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:YhunsMRnF/ODwoEAUMwU399wz/Y='} 2023-09-22T22:47:00.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d9c5f2443e34852b-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.277 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.277 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.277 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.277 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.277 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.277 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:47:00.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:47:00.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee2b3bdc7824e9ac-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj115 2023-09-22T22:47:00.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj115 2023-09-22T22:47:00.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj115 2023-09-22T22:47:00.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.279 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.279 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.279 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.279 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.279 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj115 2023-09-22T22:47:00.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.279 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:RPUpmq1Asiq8XNxUdq7WmMZB7xw= 2023-09-22T22:47:00.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RPUpmq1Asiq8XNxUdq7WmMZB7xw='} 2023-09-22T22:47:00.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d84974711bd4ea01-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.279 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.280 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-64 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: tenant: 2023-09-22T22:47:00.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: bucket name: ffzdqu-64 2023-09-22T22:47:00.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: total entries: 20 2023-09-22T22:47:00.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.281 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.281 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.281 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.281 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.281 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.282 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:47:00.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:47:00.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000da16c860b32405d-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj29 2023-09-22T22:47:00.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj29 2023-09-22T22:47:00.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj29 2023-09-22T22:47:00.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.283 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.283 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.283 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.283 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.283 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj29 2023-09-22T22:47:00.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.283 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:WQhxZ97zygogSjGZgjPYuGitakA= 2023-09-22T22:47:00.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:WQhxZ97zygogSjGZgjPYuGitakA='} 2023-09-22T22:47:00.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd1adfb02a3f6dce-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.285 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.285 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.285 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.285 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.285 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.285 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:47:00.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:47:00.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034cf21b1421e4b46-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj58 2023-09-22T22:47:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj58 2023-09-22T22:47:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj58 2023-09-22T22:47:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.286 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.286 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.287 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.287 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.287 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj58 2023-09-22T22:47:00.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.287 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:nVYbdUuYDCfAF0gl6j9IaVLN+oU= 2023-09-22T22:47:00.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:nVYbdUuYDCfAF0gl6j9IaVLN+oU='} 2023-09-22T22:47:00.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6d1d288f7666370-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.288 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.288 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.288 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.288 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.288 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.289 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:47:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:47:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cae232a1e54ee96c-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj87 2023-09-22T22:47:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj87 2023-09-22T22:47:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj87 2023-09-22T22:47:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.290 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.290 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.290 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.290 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.290 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj87 2023-09-22T22:47:00.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.290 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:RbrG9jNqwJoMKo6vUmik8F4t1fE= 2023-09-22T22:47:00.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:RbrG9jNqwJoMKo6vUmik8F4t1fE='} 2023-09-22T22:47:00.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000007f301ffa6449c28-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.292 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.292 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.292 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.292 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.292 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.292 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:47:00.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:47:00.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001fb732e46a0005bb-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj116 2023-09-22T22:47:00.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj116 2023-09-22T22:47:00.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj116 2023-09-22T22:47:00.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.293 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.294 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.294 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.294 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.294 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj116 2023-09-22T22:47:00.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.294 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:8k5gHVVbpx5zQ2PcvlrajB8YvMA= 2023-09-22T22:47:00.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:8k5gHVVbpx5zQ2PcvlrajB8YvMA='} 2023-09-22T22:47:00.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8a0490983b85786-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.295 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.295 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.295 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.295 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.296 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.296 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:47:00.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:47:00.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c7593cf449ff543-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj145 2023-09-22T22:47:00.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj145 2023-09-22T22:47:00.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj145 2023-09-22T22:47:00.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.297 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.297 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.297 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.297 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.297 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj145 2023-09-22T22:47:00.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.297 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Rx3nyXHuIX1M0FeAt3zbF1qp0Og= 2023-09-22T22:47:00.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Rx3nyXHuIX1M0FeAt3zbF1qp0Og='} 2023-09-22T22:47:00.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093b6d87b43a42942-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.298 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.298 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-64 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: tenant: 2023-09-22T22:47:00.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: bucket name: ffzdqu-64 2023-09-22T22:47:00.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: total entries: 25 2023-09-22T22:47:00.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.299 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.299 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.300 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.300 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.300 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.300 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:47:00.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:47:00.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bae4091c369704a1-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj31 2023-09-22T22:47:00.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj31 2023-09-22T22:47:00.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj31 2023-09-22T22:47:00.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.301 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.301 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.301 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.301 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.302 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj31 2023-09-22T22:47:00.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.302 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:2l2I8YLS6znPpkzDGcW64sfw+hI= 2023-09-22T22:47:00.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:2l2I8YLS6znPpkzDGcW64sfw+hI='} 2023-09-22T22:47:00.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029e20422c37387ba-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.303 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.303 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.303 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.303 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.303 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.303 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:47:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:47:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034fd207658c8bf67-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj62 2023-09-22T22:47:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj62 2023-09-22T22:47:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj62 2023-09-22T22:47:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.305 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.305 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.305 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.305 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.305 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj62 2023-09-22T22:47:00.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.305 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:1MmzyMz1YoIk8nB2BlThSI8G1ko= 2023-09-22T22:47:00.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:1MmzyMz1YoIk8nB2BlThSI8G1ko='} 2023-09-22T22:47:00.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fcd231d1a444b22b-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.306 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.307 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.307 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.307 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.307 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.307 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:47:00.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:47:00.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ab566e2ac165413-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj93 2023-09-22T22:47:00.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj93 2023-09-22T22:47:00.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj93 2023-09-22T22:47:00.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.308 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.308 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.308 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.308 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.309 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj93 2023-09-22T22:47:00.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.309 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:mBaF4+khmml0bx8FgI7x2bxqoaI= 2023-09-22T22:47:00.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:mBaF4+khmml0bx8FgI7x2bxqoaI='} 2023-09-22T22:47:00.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000302406f1d9f88ca1-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.310 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.310 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.310 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.310 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.310 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.310 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:47:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:47:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d200fc283554c375-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj124 2023-09-22T22:47:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj124 2023-09-22T22:47:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj124 2023-09-22T22:47:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.312 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.312 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.312 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.312 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.312 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj124 2023-09-22T22:47:00.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.312 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:eQPuXu5tGbUeA8ckR3vPWyYiM28= 2023-09-22T22:47:00.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:eQPuXu5tGbUeA8ckR3vPWyYiM28='} 2023-09-22T22:47:00.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001fb4be8727becbd7-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.313 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.313 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.314 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.314 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.314 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.314 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s= 2023-09-22T22:47:00.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:CRWtPPHw820XzvEj8rEMtnSTc7s=', 'Content-Length': '0'} 2023-09-22T22:47:00.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083a818cc1349e18e-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj155 2023-09-22T22:47:00.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj155 2023-09-22T22:47:00.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj155 2023-09-22T22:47:00.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.315 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.315 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.315 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.315 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:20 GMT 2023-09-22T22:47:00.315 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj155 2023-09-22T22:47:00.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.316 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:T79vyoVpyG+hR/RGg3NElqmiCmE= 2023-09-22T22:47:00.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:20 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:T79vyoVpyG+hR/RGg3NElqmiCmE='} 2023-09-22T22:47:00.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b135cd71c1c2ec0-00650e1108-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.316 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.316 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ffzdqu-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ffzdqu-64 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: tenant: 2023-09-22T22:47:00.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: bucket name: ffzdqu-64 2023-09-22T22:47:00.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: total entries: 30 2023-09-22T22:47:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.318 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.318 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.318 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.318 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:21 GMT 2023-09-22T22:47:00.318 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.318 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Jz5mny6Cgmzz+yJfi986d2+CwSM= 2023-09-22T22:47:00.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jz5mny6Cgmzz+yJfi986d2+CwSM=', 'Content-Length': '0'} 2023-09-22T22:47:00.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f7f8c8d9fcc0f7a-00650e1109-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj37 2023-09-22T22:47:00.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj37 2023-09-22T22:47:00.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj37 2023-09-22T22:47:00.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.319 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.319 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.320 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.320 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:21 GMT 2023-09-22T22:47:00.320 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj37 2023-09-22T22:47:00.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.320 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:BUEIpfRSnv6tJTjf1yKGk7A77/0= 2023-09-22T22:47:00.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:BUEIpfRSnv6tJTjf1yKGk7A77/0='} 2023-09-22T22:47:00.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000028601734d7a73f62-00650e1109-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.321 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.321 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.321 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.321 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:21 GMT 2023-09-22T22:47:00.321 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.322 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Jz5mny6Cgmzz+yJfi986d2+CwSM= 2023-09-22T22:47:00.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jz5mny6Cgmzz+yJfi986d2+CwSM=', 'Content-Length': '0'} 2023-09-22T22:47:00.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1cc1ef158d949df-00650e1109-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj74 2023-09-22T22:47:00.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj74 2023-09-22T22:47:00.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj74 2023-09-22T22:47:00.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.323 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.323 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.323 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.323 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:21 GMT 2023-09-22T22:47:00.323 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj74 2023-09-22T22:47:00.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.323 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:ESet2+5FgbAgNhA3pEw1HyTFz4g= 2023-09-22T22:47:00.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:ESet2+5FgbAgNhA3pEw1HyTFz4g='} 2023-09-22T22:47:00.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003eb43a1dd9d6ea49-00650e1109-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.325 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.325 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.325 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.325 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:21 GMT 2023-09-22T22:47:00.325 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.325 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Jz5mny6Cgmzz+yJfi986d2+CwSM= 2023-09-22T22:47:00.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jz5mny6Cgmzz+yJfi986d2+CwSM=', 'Content-Length': '0'} 2023-09-22T22:47:00.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ced8f719512cde88-00650e1109-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj111 2023-09-22T22:47:00.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj111 2023-09-22T22:47:00.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj111 2023-09-22T22:47:00.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.327 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.327 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.327 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.327 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:21 GMT 2023-09-22T22:47:00.327 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj111 2023-09-22T22:47:00.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.327 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:G8kPo6okSKLGQm4i5rorEdhkzOw= 2023-09-22T22:47:00.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:G8kPo6okSKLGQm4i5rorEdhkzOw='} 2023-09-22T22:47:00.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009ca111ebb2bda01-00650e1109-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.328 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.328 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.328 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.329 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:21 GMT 2023-09-22T22:47:00.329 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.329 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Jz5mny6Cgmzz+yJfi986d2+CwSM= 2023-09-22T22:47:00.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jz5mny6Cgmzz+yJfi986d2+CwSM=', 'Content-Length': '0'} 2023-09-22T22:47:00.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065b6dbda795e0d3c-00650e1109-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj148 2023-09-22T22:47:00.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj148 2023-09-22T22:47:00.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj148 2023-09-22T22:47:00.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.330 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.330 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.330 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.330 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:21 GMT 2023-09-22T22:47:00.330 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj148 2023-09-22T22:47:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.331 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:wvMQRq3NZizALfWVOhk1ILKVzlE= 2023-09-22T22:47:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:wvMQRq3NZizALfWVOhk1ILKVzlE='} 2023-09-22T22:47:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2dd13c2b506f1f7-00650e1109-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/ 2023-09-22T22:47:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/ 2023-09-22T22:47:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-22T22:47:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/ 2023-09-22T22:47:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.332 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-22T22:47:00.332 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.332 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.332 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:21 GMT 2023-09-22T22:47:00.332 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/ 2023-09-22T22:47:00.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.332 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:Jz5mny6Cgmzz+yJfi986d2+CwSM= 2023-09-22T22:47:00.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:11:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:Jz5mny6Cgmzz+yJfi986d2+CwSM=', 'Content-Length': '0'} 2023-09-22T22:47:00.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bacc5a93916e75ee-00650e1109-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:11:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-64/obj185 2023-09-22T22:47:00.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-64/obj185 2023-09-22T22:47:00.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-64/obj185 2023-09-22T22:47:00.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-22T22:47:00.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.334 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.334 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-22T22:47:00.334 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-22T22:47:00.334 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:11:21 GMT 2023-09-22T22:47:00.334 INFO:tasks.rgw_multisite_tests:/ffzdqu-64/obj185 2023-09-22T22:47:00.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.334 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:LvF4grXy/L76db1DndtUEd47W9k= 2023-09-22T22:47:00.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 22 Sep 2023 22:11:21 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:LvF4grXy/L76db1DndtUEd47W9k='} 2023-09-22T22:47:00.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002022c99d173a42a4-00650e1109-165f-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Date', 'Fri, 22 Sep 2023 22:11:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.335 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.335 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:11:21.510+0000 7fcb56676a40 5 RGW-SYNC:data: waiting on data sync status lease 2023-09-22T22:47:00.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:11:21.514+0000 7fcb56676a40 5 RGW-SYNC:data: acquired data sync status lease 2023-09-22T22:47:00.335 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Restarting daemon 2023-09-22T22:47:00.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-22T22:47:00.335 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Started 2023-09-22T22:47:00.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:47:00.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-22T22:47:00.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T22:47:00.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-09-22T22:47:00.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T22:47:00.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:47:00.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-22T22:47:00.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T22:47:00.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T22:47:00.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-09-22T22:47:00.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:47:00.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-22T22:47:00.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T22:47:00.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T22:47:00.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-09-22T22:47:00.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:47:00.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-22T22:47:00.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T22:47:00.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T22:47:00.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-09-22T22:47:00.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:47:00.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-22T22:47:00.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-22T22:47:00.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2023-09-22T22:47:00.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-09-22T22:47:00.338 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:11:48.434+0000 7fce80ac7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T22:47:00.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > curl http://smithi160.front.sepia.ceph.com:8001 2023-09-22T22:47:00.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T22:47:00.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-22T22:47:00.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 141k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-22T22:47:00.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: anonymous 2023-09-22T22:47:00.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=ffzdqu-64 2023-09-22T22:47:00.339 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:47:00.339 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ffzdqu-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-22T22:47:00.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ffzdqu-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-22T22:47:00.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:11:52.770+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:11:57.773+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:12:02.773+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:12:07.773+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:12:12.777+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:12:17.781+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:12:22.781+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:12:27.781+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:12:32.781+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:12:37.785+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:12:42.785+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:12:47.789+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:12:52.789+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:12:57.789+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:13:02.789+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:13:07.793+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:13:12.792+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:13:17.792+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:13:22.792+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:13:27.796+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:13:32.796+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:13:37.796+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:13:42.796+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:13:47.800+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:13:52.800+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:13:57.800+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:14:02.804+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:14:07.804+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:14:12.804+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:14:17.808+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:14:22.808+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:14:27.807+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:14:32.807+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:14:37.811+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:14:42.811+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:14:47.811+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:14:52.811+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:14:57.815+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:15:02.815+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:15:07.819+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:15:12.819+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:15:17.819+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:15:22.819+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:15:27.819+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:15:32.823+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:15:37.823+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:15:42.822+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:15:47.826+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:15:52.826+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:15:57.826+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:16:02.830+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:16:07.830+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:16:12.830+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:16:17.830+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:16:22.834+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:16:27.834+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:16:32.834+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:16:37.838+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:16:42.838+0000 7f1885df7a40 1 waiting to reach latest gen 6, on 0.. 2023-09-22T22:47:00.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:16:47.837+0000 7f1885df7a40 -1 bucket checkpoint timed out waiting to reach latest generation 6 2023-09-22T22:47:00.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:16:47.837+0000 7f1885df7a40 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-22T22:47:00.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:16:47.837+0000 7f1885df7a40 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-22T22:47:00.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-09-22T22:47:00.345 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-22T22:47:00.345 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.345 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-22T22:47:00.345 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_policy_config_zonegroup: 2023-09-22T22:47:00.346 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-22T22:47:00.346 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-22T22:47:00.346 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-22T22:47:00.346 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-22T22:47:00.346 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 2106, in test_sync_policy_config_zonegroup 2023-09-22T22:47:00.346 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-09-22T22:47:00.346 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2023-09-22T22:47:00.346 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2023-09-22T22:47:00.347 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2023-09-22T22:47:00.347 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2023-09-22T22:47:00.347 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2023-09-22T22:47:00.347 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-22T22:47:00.347 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.347 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T22:47:00.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: [{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"1_1695419384.000133_332.1","last_update":"2023-09-22T21:49:44.000133Z"},{"marker":"1_1695420681.013221_682.1","last_update":"2023-09-22T22:11:21.013221Z"},{"marker":"1_1695419383.971498_240.1","last_update":"2023-09-22T21:49:43.971498Z"}] 2023-09-22T22:47:00.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-09-22T22:47:00.348 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.348 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:16:48.105+0000 7f6f5868ba40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:16:48.105+0000 7f6f5868ba40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:16:48Z"} 2023-09-22T22:47:00.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:16:48Z"} 2023-09-22T22:47:00.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-09-22T22:47:00.349 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.349 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"groups":[]} 2023-09-22T22:47:00.350 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.350 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-22T22:47:00.350 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.350 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.350 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:16:48.445+0000 7fce80ac7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T22:47:00.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-22T22:47:00.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"} 2023-09-22T22:47:00.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-22T22:47:00.352 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.352 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-09-22T22:47:00.352 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.352 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-09-22T22:47:00.353 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.353 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-22T22:47:00.353 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2023-09-22T22:47:00.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.354 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7fce80ac7640 time 2023-09-22T22:16:49.476380+0000 2023-09-22T22:47:00.355 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T22:47:00.355 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:47:00.355 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7fcea9675883] 2023-09-22T22:47:00.355 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7fcea9675a37] 2023-09-22T22:47:00.355 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x5557ee27e46c] 2023-09-22T22:47:00.355 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5557ee660910] 2023-09-22T22:47:00.355 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x5557ee28a56e] 2023-09-22T22:47:00.355 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x5557ee28be7b] 2023-09-22T22:47:00.355 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x5557ee28cced] 2023-09-22T22:47:00.355 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x5557ee693712] 2023-09-22T22:47:00.355 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 9: radosgw(+0x836db8) [0x5557ee43edb8] 2023-09-22T22:47:00.356 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x5557ee43f753] 2023-09-22T22:47:00.356 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fcea8344b43] 2023-09-22T22:47:00.356 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fcea83d6a00] 2023-09-22T22:47:00.356 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:16:49.473+0000 7fce80ac7640 -1 ./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7fce80ac7640 time 2023-09-22T22:16:49.476380+0000 2023-09-22T22:47:00.356 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T22:47:00.356 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:47:00.356 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:47:00.356 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7fcea9675883] 2023-09-22T22:47:00.356 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7fcea9675a37] 2023-09-22T22:47:00.356 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x5557ee27e46c] 2023-09-22T22:47:00.356 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5557ee660910] 2023-09-22T22:47:00.357 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x5557ee28a56e] 2023-09-22T22:47:00.357 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x5557ee28be7b] 2023-09-22T22:47:00.357 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x5557ee28cced] 2023-09-22T22:47:00.357 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x5557ee693712] 2023-09-22T22:47:00.357 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 9: radosgw(+0x836db8) [0x5557ee43edb8] 2023-09-22T22:47:00.357 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x5557ee43f753] 2023-09-22T22:47:00.357 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fcea8344b43] 2023-09-22T22:47:00.357 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fcea83d6a00] 2023-09-22T22:47:00.357 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:47:00.357 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: *** Caught signal (Aborted) ** 2023-09-22T22:47:00.357 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: in thread 7fce80ac7640 thread_name:data-sync 2023-09-22T22:47:00.357 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:47:00.358 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fcea82f2520] 2023-09-22T22:47:00.358 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2: pthread_kill() 2023-09-22T22:47:00.358 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 3: raise() 2023-09-22T22:47:00.358 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 4: abort() 2023-09-22T22:47:00.358 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fcea96758dd] 2023-09-22T22:47:00.358 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7fcea9675a37] 2023-09-22T22:47:00.358 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x5557ee27e46c] 2023-09-22T22:47:00.358 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5557ee660910] 2023-09-22T22:47:00.358 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x5557ee28a56e] 2023-09-22T22:47:00.358 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x5557ee28be7b] 2023-09-22T22:47:00.359 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x5557ee28cced] 2023-09-22T22:47:00.359 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x5557ee693712] 2023-09-22T22:47:00.359 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 13: radosgw(+0x836db8) [0x5557ee43edb8] 2023-09-22T22:47:00.359 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x5557ee43f753] 2023-09-22T22:47:00.359 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fcea8344b43] 2023-09-22T22:47:00.359 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fcea83d6a00] 2023-09-22T22:47:00.359 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:16:49.473+0000 7fce80ac7640 -1 *** Caught signal (Aborted) ** 2023-09-22T22:47:00.359 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: in thread 7fce80ac7640 thread_name:data-sync 2023-09-22T22:47:00.359 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:47:00.359 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:47:00.359 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fcea82f2520] 2023-09-22T22:47:00.359 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2: pthread_kill() 2023-09-22T22:47:00.360 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 3: raise() 2023-09-22T22:47:00.360 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 4: abort() 2023-09-22T22:47:00.360 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fcea96758dd] 2023-09-22T22:47:00.360 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7fcea9675a37] 2023-09-22T22:47:00.360 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x5557ee27e46c] 2023-09-22T22:47:00.360 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5557ee660910] 2023-09-22T22:47:00.360 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x5557ee28a56e] 2023-09-22T22:47:00.360 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x5557ee28be7b] 2023-09-22T22:47:00.360 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x5557ee28cced] 2023-09-22T22:47:00.360 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x5557ee693712] 2023-09-22T22:47:00.360 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 13: radosgw(+0x836db8) [0x5557ee43edb8] 2023-09-22T22:47:00.361 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x5557ee43f753] 2023-09-22T22:47:00.361 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fcea8344b43] 2023-09-22T22:47:00.361 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fcea83d6a00] 2023-09-22T22:47:00.361 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-09-22T22:47:00.361 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:47:00.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","epoch":13,"predecessor_uuid":"a6099785-5160-4a69-b202-552a1b9cb770","sync_status":[],"period_map":{"id":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","zonegroups":[{"id":"a","name":"a","api_name":"","is_master":true,"endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","zones":[{"id":"2b721fee-a149-45da-925b-502d5a3d6853","name":"a2","endpoints":["http://smithi160.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"f132bdb9-20a7-4073-a932-8f99da20691e","name":"a1","endpoints":["http://smithi160.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["2b721fee-a149-45da-925b-502d5a3d6853","f132bdb9-20a7-4073-a932-8f99da20691e"]}],"directional":[{"source_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","dest_zone":"2b721fee-a149-45da-925b-502d5a3d6853"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["2b721fee-a149-45da-925b-502d5a3d6853","f132bdb9-20a7-4073-a932-8f99da20691e"]},"dest":{"bucket":"*","zones":["2b721fee-a149-45da-925b-502d5a3d6853","f132bdb9-20a7-4073-a932-8f99da20691e"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]},"enabled_features":["resharding"]},{"id":"b","name":"b","api_name":"","is_master":false,"endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","zones":[{"id":"087d4b41-8f81-4835-9e99-026ca13a7b60","name":"b2","endpoints":["http://smithi204.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","name":"b1","endpoints":["http://smithi204.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"087d4b41-8f81-4835-9e99-026ca13a7b60","val":2135214481},{"key":"2b721fee-a149-45da-925b-502d5a3d6853","val":3586193783},{"key":"bfe1d114-67f9-4e0e-8e45-c23380ed8feb","val":4184768938},{"key":"f132bdb9-20a7-4073-a932-8f99da20691e","val":3405200286}]},"master_zonegroup":"a","master_zone":"f132bdb9-20a7-4073-a932-8f99da20691e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"9e7cb63e-d555-4d20-9c48-393b1c24d1ff","realm_name":"test-realm","realm_epoch":2} 2023-09-22T22:47:00.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=ffzdqu-65 2023-09-22T22:47:00.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ffzdqu-65/ 2023-09-22T22:47:00.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ffzdqu-65/ 2023-09-22T22:47:00.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-22T22:47:00.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ffzdqu-65/ 2023-09-22T22:47:00.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-22T22:47:00.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-22T22:47:00.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi160.front.sepia.ceph.com:8000 2023-09-22T22:47:00.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-22T22:47:00.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-22T22:47:00.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-22T22:47:00.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-22T22:47:00.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-22T22:47:00.362 INFO:tasks.rgw_multisite_tests:PUT 2023-09-22T22:47:00.362 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.362 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.363 INFO:tasks.rgw_multisite_tests:Fri, 22 Sep 2023 22:16:49 GMT 2023-09-22T22:47:00.363 INFO:tasks.rgw_multisite_tests:/ffzdqu-65/ 2023-09-22T22:47:00.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-22T22:47:00.363 INFO:tasks.rgw_multisite_tests:AWS RX9T9XC1DLDGURC46TDQ:4/5Tr3qDW+XNfGY4adXhQ1sJOUQ= 2023-09-22T22:47:00.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 22 Sep 2023 22:16:49 GMT', 'Authorization': 'AWS RX9T9XC1DLDGURC46TDQ:4/5Tr3qDW+XNfGY4adXhQ1sJOUQ=', 'Content-Length': '0'} 2023-09-22T22:47:00.363 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: -883> 2023-09-22T22:16:48.445+0000 7fce80ac7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-22T22:47:00.363 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: -1> 2023-09-22T22:16:49.473+0000 7fce80ac7640 -1 ./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7fce80ac7640 time 2023-09-22T22:16:49.476380+0000 2023-09-22T22:47:00.363 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T22:47:00.363 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:47:00.363 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:47:00.363 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7fcea9675883] 2023-09-22T22:47:00.364 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7fcea9675a37] 2023-09-22T22:47:00.364 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x5557ee27e46c] 2023-09-22T22:47:00.364 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5557ee660910] 2023-09-22T22:47:00.364 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x5557ee28a56e] 2023-09-22T22:47:00.364 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x5557ee28be7b] 2023-09-22T22:47:00.364 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x5557ee28cced] 2023-09-22T22:47:00.364 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x5557ee693712] 2023-09-22T22:47:00.364 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 9: radosgw(+0x836db8) [0x5557ee43edb8] 2023-09-22T22:47:00.364 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x5557ee43f753] 2023-09-22T22:47:00.364 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fcea8344b43] 2023-09-22T22:47:00.364 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fcea83d6a00] 2023-09-22T22:47:00.365 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:47:00.365 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 0> 2023-09-22T22:16:49.473+0000 7fce80ac7640 -1 *** Caught signal (Aborted) ** 2023-09-22T22:47:00.365 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: in thread 7fce80ac7640 thread_name:data-sync 2023-09-22T22:47:00.365 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:47:00.365 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:47:00.365 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fcea82f2520] 2023-09-22T22:47:00.365 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2: pthread_kill() 2023-09-22T22:47:00.365 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 3: raise() 2023-09-22T22:47:00.365 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 4: abort() 2023-09-22T22:47:00.365 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fcea96758dd] 2023-09-22T22:47:00.365 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7fcea9675a37] 2023-09-22T22:47:00.366 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x5557ee27e46c] 2023-09-22T22:47:00.366 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5557ee660910] 2023-09-22T22:47:00.366 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x5557ee28a56e] 2023-09-22T22:47:00.366 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x5557ee28be7b] 2023-09-22T22:47:00.366 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x5557ee28cced] 2023-09-22T22:47:00.366 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x5557ee693712] 2023-09-22T22:47:00.366 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 13: radosgw(+0x836db8) [0x5557ee43edb8] 2023-09-22T22:47:00.366 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x5557ee43f753] 2023-09-22T22:47:00.366 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fcea8344b43] 2023-09-22T22:47:00.366 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fcea83d6a00] 2023-09-22T22:47:00.366 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-09-22T22:47:00.367 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stdout: INFO: 2023-09-22T22:47:00.367 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi160.stderr: INFO: daemon-helper: command crashed with signal 11 2023-09-22T22:47:00.367 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: ./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7f11298ba640 time 2023-09-22T22:16:49.589063+0000 2023-09-22T22:47:00.367 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: ./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T22:47:00.367 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:47:00.367 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f1152468883] 2023-09-22T22:47:00.367 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f1152468a37] 2023-09-22T22:47:00.367 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x56533621a46c] 2023-09-22T22:47:00.367 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5653365fc910] 2023-09-22T22:47:00.367 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x56533622656e] 2023-09-22T22:47:00.367 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x565336227e7b] 2023-09-22T22:47:00.368 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x565336228ced] 2023-09-22T22:47:00.368 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x56533662f712] 2023-09-22T22:47:00.368 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 9: radosgw(+0x836db8) [0x5653363dadb8] 2023-09-22T22:47:00.368 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x5653363db753] 2023-09-22T22:47:00.368 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f1151137b43] 2023-09-22T22:47:00.368 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f11511c9a00] 2023-09-22T22:47:00.368 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 2023-09-22T22:16:49.583+0000 7f11298ba640 -1 ./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7f11298ba640 time 2023-09-22T22:16:49.589063+0000 2023-09-22T22:47:00.368 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: ./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T22:47:00.368 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 2023-09-22T22:47:00.368 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:47:00.368 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f1152468883] 2023-09-22T22:47:00.369 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f1152468a37] 2023-09-22T22:47:00.369 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x56533621a46c] 2023-09-22T22:47:00.369 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5653365fc910] 2023-09-22T22:47:00.369 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x56533622656e] 2023-09-22T22:47:00.369 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x565336227e7b] 2023-09-22T22:47:00.369 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x565336228ced] 2023-09-22T22:47:00.369 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x56533662f712] 2023-09-22T22:47:00.369 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 9: radosgw(+0x836db8) [0x5653363dadb8] 2023-09-22T22:47:00.369 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x5653363db753] 2023-09-22T22:47:00.369 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f1151137b43] 2023-09-22T22:47:00.369 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f11511c9a00] 2023-09-22T22:47:00.370 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 2023-09-22T22:47:00.370 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: *** Caught signal (Aborted) ** 2023-09-22T22:47:00.370 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: in thread 7f11298ba640 thread_name:data-sync 2023-09-22T22:47:00.370 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:47:00.370 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f11510e5520] 2023-09-22T22:47:00.370 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 2: pthread_kill() 2023-09-22T22:47:00.370 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 3: raise() 2023-09-22T22:47:00.370 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 4: abort() 2023-09-22T22:47:00.370 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f11524688dd] 2023-09-22T22:47:00.370 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f1152468a37] 2023-09-22T22:47:00.370 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x56533621a46c] 2023-09-22T22:47:00.371 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5653365fc910] 2023-09-22T22:47:00.371 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x56533622656e] 2023-09-22T22:47:00.371 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x565336227e7b] 2023-09-22T22:47:00.371 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x565336228ced] 2023-09-22T22:47:00.371 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x56533662f712] 2023-09-22T22:47:00.371 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 13: radosgw(+0x836db8) [0x5653363dadb8] 2023-09-22T22:47:00.371 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x5653363db753] 2023-09-22T22:47:00.371 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f1151137b43] 2023-09-22T22:47:00.371 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f11511c9a00] 2023-09-22T22:47:00.371 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 2023-09-22T22:16:49.587+0000 7f11298ba640 -1 *** Caught signal (Aborted) ** 2023-09-22T22:47:00.371 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: in thread 7f11298ba640 thread_name:data-sync 2023-09-22T22:47:00.371 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 2023-09-22T22:47:00.372 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:47:00.372 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f11510e5520] 2023-09-22T22:47:00.372 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 2: pthread_kill() 2023-09-22T22:47:00.372 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 3: raise() 2023-09-22T22:47:00.372 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 4: abort() 2023-09-22T22:47:00.372 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f11524688dd] 2023-09-22T22:47:00.372 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f1152468a37] 2023-09-22T22:47:00.372 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x56533621a46c] 2023-09-22T22:47:00.372 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5653365fc910] 2023-09-22T22:47:00.372 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x56533622656e] 2023-09-22T22:47:00.372 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x565336227e7b] 2023-09-22T22:47:00.373 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x565336228ced] 2023-09-22T22:47:00.373 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x56533662f712] 2023-09-22T22:47:00.373 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 13: radosgw(+0x836db8) [0x5653363dadb8] 2023-09-22T22:47:00.373 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x5653363db753] 2023-09-22T22:47:00.373 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f1151137b43] 2023-09-22T22:47:00.373 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f11511c9a00] 2023-09-22T22:47:00.373 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-09-22T22:47:00.373 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 2023-09-22T22:47:00.373 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: -1> 2023-09-22T22:16:49.583+0000 7f11298ba640 -1 ./src/rgw/rgw_coroutine.cc: In function 'void RGWCompletionManager::wait_interval(void*, const utime_t&, void*)' thread 7f11298ba640 time 2023-09-22T22:16:49.589063+0000 2023-09-22T22:47:00.373 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: ./src/rgw/rgw_coroutine.cc: 117: FAILED ceph_assert(waiters.find(opaque) == waiters.end()) 2023-09-22T22:47:00.373 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 2023-09-22T22:47:00.374 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:47:00.374 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f1152468883] 2023-09-22T22:47:00.374 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 2: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f1152468a37] 2023-09-22T22:47:00.374 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 3: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x56533621a46c] 2023-09-22T22:47:00.374 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 4: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5653365fc910] 2023-09-22T22:47:00.374 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x56533622656e] 2023-09-22T22:47:00.374 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x565336227e7b] 2023-09-22T22:47:00.374 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x565336228ced] 2023-09-22T22:47:00.374 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x56533662f712] 2023-09-22T22:47:00.374 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 9: radosgw(+0x836db8) [0x5653363dadb8] 2023-09-22T22:47:00.374 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x5653363db753] 2023-09-22T22:47:00.375 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f1151137b43] 2023-09-22T22:47:00.375 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f11511c9a00] 2023-09-22T22:47:00.375 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 2023-09-22T22:47:00.375 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 0> 2023-09-22T22:16:49.587+0000 7f11298ba640 -1 *** Caught signal (Aborted) ** 2023-09-22T22:47:00.375 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: in thread 7f11298ba640 thread_name:data-sync 2023-09-22T22:47:00.375 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 2023-09-22T22:47:00.375 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev) 2023-09-22T22:47:00.375 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f11510e5520] 2023-09-22T22:47:00.375 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 2: pthread_kill() 2023-09-22T22:47:00.375 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 3: raise() 2023-09-22T22:47:00.375 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 4: abort() 2023-09-22T22:47:00.376 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f11524688dd] 2023-09-22T22:47:00.376 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f1152468a37] 2023-09-22T22:47:00.376 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 7: (RGWCompletionManager::wait_interval(void*, utime_t const&, void*)+0x2ec) [0x56533621a46c] 2023-09-22T22:47:00.376 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 8: (RGWContinuousLeaseCR::operate(DoutPrefixProvider const*)+0x300) [0x5653365fc910] 2023-09-22T22:47:00.376 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x13e) [0x56533622656e] 2023-09-22T22:47:00.376 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x35b) [0x565336227e7b] 2023-09-22T22:47:00.376 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 11: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x565336228ced] 2023-09-22T22:47:00.376 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 12: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4a2) [0x56533662f712] 2023-09-22T22:47:00.376 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 13: radosgw(+0x836db8) [0x5653363dadb8] 2023-09-22T22:47:00.376 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 14: (RGWRadosThread::Worker::entry()+0xb3) [0x5653363db753] 2023-09-22T22:47:00.376 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f1151137b43] 2023-09-22T22:47:00.377 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f11511c9a00] 2023-09-22T22:47:00.377 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-09-22T22:47:00.377 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stdout: INFO: 2023-09-22T22:47:00.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000000ead651f78deaa4f-00650e1251-208a-a1'), ('Server', 'Ceph Object Gateway (reef)'), ('Content-Length', '0'), ('Date', 'Fri, 22 Sep 2023 22:16:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-22T22:47:00.377 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.377 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket ffzdqu-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.377 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi204.stderr: INFO: daemon-helper: command crashed with signal 11 2023-09-22T22:47:00.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-22T22:47:00.378 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.378 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket ffzdqu-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-22T22:47:00.378 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.378 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket ffzdqu-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"} 2023-09-22T22:47:00.379 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.379 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket ffzdqu-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-22T22:47:00.379 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.379 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket ffzdqu-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-09-22T22:47:00.380 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.380 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket ffzdqu-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-09-22T22:47:00.380 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.380 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=ffzdqu-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-09-22T22:47:00.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'ffzdqu-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-22T22:47:00.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket ffzdqu-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-22T22:47:00.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"} 2023-09-22T22:47:00.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: No --period given, using current period=03d00cc5-7ad7-4abf-a475-d9ffc8e37106 2023-09-22T22:47:00.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: [{"marker":"1_1695416545.459338_27.1","last_update":"2023-09-22T21:02:25.459338Z"},{"marker":"1_1695419384.000133_332.1","last_update":"2023-09-22T21:49:44.000133Z"},{"marker":"1_1695421010.611730_706.1","last_update":"2023-09-22T22:16:50.611730Z"},{"marker":"1_1695419383.971498_240.1","last_update":"2023-09-22T21:49:43.971498Z"}] 2023-09-22T22:47:00.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-09-22T22:47:00.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:16:50.993+0000 7f8106e97a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:16:50.993+0000 7f8106e97a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:16:51Z"} 2023-09-22T22:47:00.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:16:51Z"} 2023-09-22T22:47:00.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.383 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2023-09-22T22:47:00.384 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~0s 2023-09-22T22:47:00.384 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.384 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:16:56.097+0000 7fa573143a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:16:56.097+0000 7fa573143a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:16:56Z"} 2023-09-22T22:47:00.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:16:56Z"} 2023-09-22T22:47:00.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.385 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~6s 2023-09-22T22:47:00.385 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~6s 2023-09-22T22:47:00.385 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.385 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:01.201+0000 7f595c171a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:01.205+0000 7f595c171a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:01Z"} 2023-09-22T22:47:00.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:01Z"} 2023-09-22T22:47:00.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.386 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~11s 2023-09-22T22:47:00.386 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~11s 2023-09-22T22:47:00.386 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.386 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:06.317+0000 7fe6c0939a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:06.317+0000 7fe6c0939a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:06Z"} 2023-09-22T22:47:00.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:06Z"} 2023-09-22T22:47:00.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.388 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~17s 2023-09-22T22:47:00.388 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~17s 2023-09-22T22:47:00.388 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.388 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:11.421+0000 7f5a04cb1a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:11.421+0000 7f5a04cb1a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:11Z"} 2023-09-22T22:47:00.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:11Z"} 2023-09-22T22:47:00.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.389 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~22s 2023-09-22T22:47:00.389 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~22s 2023-09-22T22:47:00.389 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.389 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:16.525+0000 7fddbbf26a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:16.525+0000 7fddbbf26a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:16Z"} 2023-09-22T22:47:00.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:16Z"} 2023-09-22T22:47:00.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.390 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~28s 2023-09-22T22:47:00.390 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~28s 2023-09-22T22:47:00.390 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.391 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:21.729+0000 7f19cc582a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:21.729+0000 7f19cc582a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:21Z"} 2023-09-22T22:47:00.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:21Z"} 2023-09-22T22:47:00.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.392 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~34s 2023-09-22T22:47:00.392 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.392 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:26.833+0000 7f486bee1a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:26.833+0000 7f486bee1a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:26Z"} 2023-09-22T22:47:00.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:26Z"} 2023-09-22T22:47:00.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.393 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~34s 2023-09-22T22:47:00.393 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.393 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:31.937+0000 7fe8224a9a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:31.937+0000 7fe8224a9a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:31Z"} 2023-09-22T22:47:00.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:31Z"} 2023-09-22T22:47:00.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.394 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~39s 2023-09-22T22:47:00.394 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~39s 2023-09-22T22:47:00.394 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.395 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:37.041+0000 7f334fa27a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:37.045+0000 7f334fa27a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:37Z"} 2023-09-22T22:47:00.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:37Z"} 2023-09-22T22:47:00.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.396 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~45s 2023-09-22T22:47:00.396 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~45s 2023-09-22T22:47:00.396 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.396 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:42.149+0000 7f8297921a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:42.149+0000 7f8297921a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:42Z"} 2023-09-22T22:47:00.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:42Z"} 2023-09-22T22:47:00.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.397 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~50s 2023-09-22T22:47:00.397 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~50s 2023-09-22T22:47:00.397 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.397 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:47.253+0000 7f9a65de2a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:47.253+0000 7f9a65de2a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:47Z"} 2023-09-22T22:47:00.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:47Z"} 2023-09-22T22:47:00.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.398 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~56s 2023-09-22T22:47:00.399 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~56s 2023-09-22T22:47:00.399 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.399 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:52.357+0000 7f5168a56a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:52.357+0000 7f5168a56a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:52Z"} 2023-09-22T22:47:00.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:52Z"} 2023-09-22T22:47:00.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.400 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~62s 2023-09-22T22:47:00.400 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~62s 2023-09-22T22:47:00.400 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.400 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:57.460+0000 7fcbdb384a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:17:57.460+0000 7fcbdb384a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:57Z"} 2023-09-22T22:47:00.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:17:57Z"} 2023-09-22T22:47:00.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.401 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~67s 2023-09-22T22:47:00.401 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~67s 2023-09-22T22:47:00.401 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.401 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:02.564+0000 7f73eeb93a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:02.564+0000 7f73eeb93a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:02Z"} 2023-09-22T22:47:00.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:02Z"} 2023-09-22T22:47:00.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.403 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~73s 2023-09-22T22:47:00.403 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~73s 2023-09-22T22:47:00.403 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.403 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:07.672+0000 7fa986733a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:07.672+0000 7fa986733a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:07Z"} 2023-09-22T22:47:00.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:07Z"} 2023-09-22T22:47:00.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.404 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~78s 2023-09-22T22:47:00.404 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~78s 2023-09-22T22:47:00.404 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.404 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:12.776+0000 7fa974c0aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:12.776+0000 7fa974c0aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:12Z"} 2023-09-22T22:47:00.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:12Z"} 2023-09-22T22:47:00.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.405 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~84s 2023-09-22T22:47:00.406 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~84s 2023-09-22T22:47:00.406 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.406 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:17.880+0000 7fb49d372a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:17.884+0000 7fb49d372a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:17Z"} 2023-09-22T22:47:00.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:17Z"} 2023-09-22T22:47:00.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.407 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~90s 2023-09-22T22:47:00.407 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.407 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:22.992+0000 7ffbe5ee0a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:22.992+0000 7ffbe5ee0a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:22Z"} 2023-09-22T22:47:00.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:22Z"} 2023-09-22T22:47:00.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.408 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~90s 2023-09-22T22:47:00.408 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.408 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:28.096+0000 7f4fe351ba40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:28.100+0000 7f4fe351ba40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:28Z"} 2023-09-22T22:47:00.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:28Z"} 2023-09-22T22:47:00.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.409 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~95s 2023-09-22T22:47:00.410 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~95s 2023-09-22T22:47:00.410 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.410 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:33.200+0000 7fe58fcdaa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:33.204+0000 7fe58fcdaa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:33Z"} 2023-09-22T22:47:00.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:33Z"} 2023-09-22T22:47:00.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.411 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~101s 2023-09-22T22:47:00.411 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~101s 2023-09-22T22:47:00.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:38.308+0000 7fa415f77a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:38.308+0000 7fa415f77a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:38Z"} 2023-09-22T22:47:00.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:38Z"} 2023-09-22T22:47:00.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.412 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~107s 2023-09-22T22:47:00.413 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~106s 2023-09-22T22:47:00.413 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.413 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:43.412+0000 7fca5eb2da40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:43.412+0000 7fca5eb2da40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:43Z"} 2023-09-22T22:47:00.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:43Z"} 2023-09-22T22:47:00.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.414 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~112s 2023-09-22T22:47:00.414 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~112s 2023-09-22T22:47:00.414 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.414 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:48.520+0000 7f6a9a875a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:48.520+0000 7f6a9a875a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:48Z"} 2023-09-22T22:47:00.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:48Z"} 2023-09-22T22:47:00.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.415 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~118s 2023-09-22T22:47:00.415 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~118s 2023-09-22T22:47:00.415 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.416 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:53.864+0000 7f7ac47eda40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:53.868+0000 7f7ac47eda40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:53Z"} 2023-09-22T22:47:00.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:53Z"} 2023-09-22T22:47:00.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.417 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~123s 2023-09-22T22:47:00.417 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~123s 2023-09-22T22:47:00.417 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.417 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:58.976+0000 7f5e4f631a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:18:58.976+0000 7f5e4f631a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:58Z"} 2023-09-22T22:47:00.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:18:58Z"} 2023-09-22T22:47:00.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.418 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~129s 2023-09-22T22:47:00.418 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~129s 2023-09-22T22:47:00.418 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.418 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:04.079+0000 7fc1c3fcda40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:04.083+0000 7fc1c3fcda40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:04Z"} 2023-09-22T22:47:00.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:04Z"} 2023-09-22T22:47:00.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.419 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~135s 2023-09-22T22:47:00.420 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~135s 2023-09-22T22:47:00.420 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.420 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:09.183+0000 7f90b893fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:09.187+0000 7f90b893fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:09Z"} 2023-09-22T22:47:00.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:09Z"} 2023-09-22T22:47:00.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.421 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~140s 2023-09-22T22:47:00.421 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~140s 2023-09-22T22:47:00.421 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.421 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:14.295+0000 7f6f84ef6a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:14.295+0000 7f6f84ef6a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:14Z"} 2023-09-22T22:47:00.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:14Z"} 2023-09-22T22:47:00.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.422 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~146s 2023-09-22T22:47:00.422 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~146s 2023-09-22T22:47:00.422 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.422 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:20.011+0000 7f17e90b4a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:20.011+0000 7f17e90b4a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:20Z"} 2023-09-22T22:47:00.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:20Z"} 2023-09-22T22:47:00.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.424 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~151s 2023-09-22T22:47:00.424 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~151s 2023-09-22T22:47:00.424 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.424 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:25.119+0000 7f890d712a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:25.119+0000 7f890d712a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:25Z"} 2023-09-22T22:47:00.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:25Z"} 2023-09-22T22:47:00.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.425 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~157s 2023-09-22T22:47:00.425 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.425 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:30.223+0000 7f739da34a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:30.227+0000 7f739da34a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:30Z"} 2023-09-22T22:47:00.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:30Z"} 2023-09-22T22:47:00.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.426 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~157s 2023-09-22T22:47:00.426 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.427 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:35.327+0000 7f17cb84fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:35.331+0000 7f17cb84fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:35Z"} 2023-09-22T22:47:00.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:35Z"} 2023-09-22T22:47:00.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.428 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.428 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.428 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~163s 2023-09-22T22:47:00.428 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~163s 2023-09-22T22:47:00.428 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.428 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:40.435+0000 7f000cc06a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:40.435+0000 7f000cc06a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:40Z"} 2023-09-22T22:47:00.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:40Z"} 2023-09-22T22:47:00.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.429 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~169s 2023-09-22T22:47:00.429 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~169s 2023-09-22T22:47:00.429 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.429 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:45.539+0000 7ff7d62eca40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:45.539+0000 7ff7d62eca40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:45Z"} 2023-09-22T22:47:00.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:45Z"} 2023-09-22T22:47:00.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.431 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~174s 2023-09-22T22:47:00.431 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~174s 2023-09-22T22:47:00.431 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.431 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:50.647+0000 7fea2e3aea40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:50.647+0000 7fea2e3aea40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:50Z"} 2023-09-22T22:47:00.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:50Z"} 2023-09-22T22:47:00.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.432 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~180s 2023-09-22T22:47:00.432 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~180s 2023-09-22T22:47:00.432 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.432 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:55.755+0000 7f4116fd3a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:19:55.755+0000 7f4116fd3a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:55Z"} 2023-09-22T22:47:00.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:19:55Z"} 2023-09-22T22:47:00.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.433 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~186s 2023-09-22T22:47:00.433 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~185s 2023-09-22T22:47:00.433 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.434 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:00.859+0000 7fe9e10c1a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:00.859+0000 7fe9e10c1a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:00Z"} 2023-09-22T22:47:00.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:00Z"} 2023-09-22T22:47:00.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.435 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~191s 2023-09-22T22:47:00.435 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~191s 2023-09-22T22:47:00.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:05.966+0000 7f575aeaba40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:05.966+0000 7f575aeaba40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:05Z"} 2023-09-22T22:47:00.436 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:05Z"} 2023-09-22T22:47:00.436 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.436 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.436 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.436 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.436 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~197s 2023-09-22T22:47:00.436 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~197s 2023-09-22T22:47:00.436 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.436 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:11.070+0000 7fa5733cda40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:11.070+0000 7fa5733cda40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:11Z"} 2023-09-22T22:47:00.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:11Z"} 2023-09-22T22:47:00.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.437 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~202s 2023-09-22T22:47:00.438 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~203s 2023-09-22T22:47:00.438 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.438 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:16.174+0000 7f759fec0a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:16.174+0000 7f759fec0a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:16Z"} 2023-09-22T22:47:00.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:16Z"} 2023-09-22T22:47:00.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.439 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~208s 2023-09-22T22:47:00.439 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.439 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:21.282+0000 7faa2099ca40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:21.282+0000 7faa2099ca40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:21Z"} 2023-09-22T22:47:00.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:21Z"} 2023-09-22T22:47:00.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.440 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~208s 2023-09-22T22:47:00.440 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.440 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:26.386+0000 7fe84702da40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:26.386+0000 7fe84702da40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:26Z"} 2023-09-22T22:47:00.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:26Z"} 2023-09-22T22:47:00.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.441 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~214s 2023-09-22T22:47:00.441 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~214s 2023-09-22T22:47:00.442 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.442 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:31.490+0000 7f6a4c9daa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:31.490+0000 7f6a4c9daa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:31Z"} 2023-09-22T22:47:00.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:31Z"} 2023-09-22T22:47:00.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.443 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~219s 2023-09-22T22:47:00.443 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~219s 2023-09-22T22:47:00.443 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.443 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:36.590+0000 7f8bd2901a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:36.594+0000 7f8bd2901a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:36Z"} 2023-09-22T22:47:00.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:36Z"} 2023-09-22T22:47:00.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.444 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~225s 2023-09-22T22:47:00.444 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~225s 2023-09-22T22:47:00.444 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.444 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:41.698+0000 7fedb4eb0a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:41.698+0000 7fedb4eb0a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:41Z"} 2023-09-22T22:47:00.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:41Z"} 2023-09-22T22:47:00.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.446 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~230s 2023-09-22T22:47:00.446 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~231s 2023-09-22T22:47:00.446 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.446 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:46.802+0000 7fc472373a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:46.802+0000 7fc472373a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:46Z"} 2023-09-22T22:47:00.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:46Z"} 2023-09-22T22:47:00.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.447 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~236s 2023-09-22T22:47:00.447 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~236s 2023-09-22T22:47:00.447 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.447 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:51.902+0000 7fcda0c26a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:51.906+0000 7fcda0c26a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:51Z"} 2023-09-22T22:47:00.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:51Z"} 2023-09-22T22:47:00.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.448 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~242s 2023-09-22T22:47:00.448 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~242s 2023-09-22T22:47:00.449 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.449 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:57.010+0000 7fb8f5369a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:20:57.014+0000 7fb8f5369a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:57Z"} 2023-09-22T22:47:00.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:20:57Z"} 2023-09-22T22:47:00.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.450 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~247s 2023-09-22T22:47:00.450 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~247s 2023-09-22T22:47:00.450 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.450 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:02.118+0000 7f652d971a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:02.118+0000 7f652d971a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:02Z"} 2023-09-22T22:47:00.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:02Z"} 2023-09-22T22:47:00.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.451 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~253s 2023-09-22T22:47:00.451 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~253s 2023-09-22T22:47:00.451 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.451 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:07.222+0000 7fa22e0b7a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:07.222+0000 7fa22e0b7a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:07Z"} 2023-09-22T22:47:00.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:07Z"} 2023-09-22T22:47:00.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.452 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~259s 2023-09-22T22:47:00.452 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~259s 2023-09-22T22:47:00.453 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.453 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:12.326+0000 7f6f5660ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:12.326+0000 7f6f5660ea40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:12Z"} 2023-09-22T22:47:00.453 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:12Z"} 2023-09-22T22:47:00.453 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.454 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~264s 2023-09-22T22:47:00.454 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.454 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:17.429+0000 7f782078aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:17.429+0000 7f782078aa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:17Z"} 2023-09-22T22:47:00.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:17Z"} 2023-09-22T22:47:00.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.455 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~264s 2023-09-22T22:47:00.455 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.455 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:22.533+0000 7f1cb8dbfa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:22.533+0000 7f1cb8dbfa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:22Z"} 2023-09-22T22:47:00.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:22Z"} 2023-09-22T22:47:00.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.456 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~270s 2023-09-22T22:47:00.456 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~270s 2023-09-22T22:47:00.457 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.457 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:27.637+0000 7f9bfdc96a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:27.637+0000 7f9bfdc96a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:27Z"} 2023-09-22T22:47:00.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:27Z"} 2023-09-22T22:47:00.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.458 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.458 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.458 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~275s 2023-09-22T22:47:00.458 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~275s 2023-09-22T22:47:00.458 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.458 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:32.745+0000 7fba3b903a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:32.745+0000 7fba3b903a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:32Z"} 2023-09-22T22:47:00.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:32Z"} 2023-09-22T22:47:00.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.459 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~281s 2023-09-22T22:47:00.459 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~281s 2023-09-22T22:47:00.459 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.459 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:37.849+0000 7f7a5aa8fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:37.849+0000 7f7a5aa8fa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:37Z"} 2023-09-22T22:47:00.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:37Z"} 2023-09-22T22:47:00.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.461 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~287s 2023-09-22T22:47:00.461 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~287s 2023-09-22T22:47:00.461 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.461 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:42.957+0000 7f71354c8a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:42.957+0000 7f71354c8a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:42Z"} 2023-09-22T22:47:00.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:42Z"} 2023-09-22T22:47:00.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.462 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~292s 2023-09-22T22:47:00.462 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~292s 2023-09-22T22:47:00.462 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.462 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:48.065+0000 7fd88e498a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:48.069+0000 7fd88e498a40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:48Z"} 2023-09-22T22:47:00.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:48Z"} 2023-09-22T22:47:00.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.463 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~298s 2023-09-22T22:47:00.463 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~298s 2023-09-22T22:47:00.464 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.464 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:53.169+0000 7faa854eaa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:21:53.169+0000 7faa854eaa40 20 RGW-SYNC:meta: read sync status 2023-09-22T22:47:00.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:53Z"} 2023-09-22T22:47:00.464 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03d00cc5-7ad7-4abf-a475-d9ffc8e37106","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1695416545.459338_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2023-09-22T21:02:25.459338Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1695419384.000133_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:44.000133Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1695420681.013221_682.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T22:11:21.013221Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1695419383.971498_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2023-09-22T21:49:43.971498Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2023-09-22T22:21:53Z"} 2023-09-22T22:47:00.464 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1695416545.459338_27.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-22T21:02:25.459338Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1695419384.000133_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:44.000133Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1695420681.013221_682.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T22:11:21.013221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1695419383.971498_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-22T21:49:43.971498Z', 'realm_epoch': 2}}] 2023-09-22T22:47:00.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695421010.611730_706.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1695416545.459338_27.1', 1: '1_1695419384.000133_332.1', 2: '1_1695420681.013221_682.1', 3: '1_1695419383.971498_240.1'} 2023-09-22T22:47:00.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1695421010.611730_706.1 target=1_1695420681.013221_682.1 2023-09-22T22:47:00.465 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~303s 2023-09-22T22:47:00.465 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: BARK! unmounting mounts and killing all daemons 2023-09-22T22:47:00.465 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~303s 2023-09-22T22:47:00.465 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: BARK! unmounting mounts and killing all daemons 2023-09-22T22:47:00.465 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.0: INFO: Sent signal 15 2023-09-22T22:47:00.465 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.1: INFO: Sent signal 15 2023-09-22T22:47:00.465 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.2: INFO: Sent signal 15 2023-09-22T22:47:00.465 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.mon.a: INFO: Sent signal 15 2023-09-22T22:47:00.465 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0: INFO: Sent signal 15 2023-09-22T22:47:00.466 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.mgr.x: INFO: Sent signal 15 2023-09-22T22:47:00.466 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.0.smithi160.stderr: INFO: 2023-09-22T22:21:56.917+0000 7fe09a3fc640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c1 -i 0 (PID: 33372) UID: 0 2023-09-22T22:47:00.466 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.0.smithi160.stderr: INFO: 2023-09-22T22:21:56.917+0000 7fe09a3fc640 -1 osd.0 66 *** Got signal Terminated *** 2023-09-22T22:47:00.466 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.0.smithi160.stderr: INFO: 2023-09-22T22:21:56.917+0000 7fe09a3fc640 -1 osd.0 66 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-22T22:47:00.466 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.1.smithi160.stderr: INFO: 2023-09-22T22:21:56.917+0000 7f1dfcffd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c1 -i 1 (PID: 33375) UID: 0 2023-09-22T22:47:00.466 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.1.smithi160.stderr: INFO: 2023-09-22T22:21:56.917+0000 7f1dfcffd640 -1 osd.1 66 *** Got signal Terminated *** 2023-09-22T22:47:00.466 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.1.smithi160.stderr: INFO: 2023-09-22T22:21:56.917+0000 7f1dfcffd640 -1 osd.1 66 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-22T22:47:00.466 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.2.smithi160.stderr: INFO: 2023-09-22T22:21:56.917+0000 7fefff281640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c1 -i 2 (PID: 33374) UID: 0 2023-09-22T22:47:00.466 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.2.smithi160.stderr: INFO: 2023-09-22T22:21:56.917+0000 7fefff281640 -1 osd.2 66 *** Got signal Terminated *** 2023-09-22T22:47:00.466 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.2.smithi160.stderr: INFO: 2023-09-22T22:21:56.917+0000 7fefff281640 -1 osd.2 66 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-22T22:47:00.466 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.mon.a.smithi160.stderr: INFO: 2023-09-22T22:21:56.917+0000 7f7f3c01f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster c1 -i a (PID: 33199) UID: 0 2023-09-22T22:47:00.467 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.mon.a.smithi160.stderr: INFO: 2023-09-22T22:21:56.917+0000 7f7f3c01f640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-09-22T22:47:00.467 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi160.stdout: INFO: 2023-09-22T22:21:56.917+0000 7fa9e36b9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm (PID: 39446) UID: 0 2023-09-22T22:47:00.467 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi160.stdout: INFO: 2023-09-22T22:21:56.917+0000 7fa9e6f00a40 -1 shutting down 2023-09-22T22:47:00.467 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.mgr.x.smithi160.stderr: INFO: 2023-09-22T22:21:56.917+0000 7f3fe34a2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster c1 -i x (PID: 33201) UID: 0 2023-09-22T22:47:00.467 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.mgr.x.smithi160.stderr: INFO: 2023-09-22T22:21:56.917+0000 7f3fe34a2640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-09-22T22:47:00.467 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.0: INFO: Sent signal 15 2023-09-22T22:47:00.467 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.1: INFO: Sent signal 15 2023-09-22T22:47:00.467 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.2: INFO: Sent signal 15 2023-09-22T22:47:00.467 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mon.a: INFO: Sent signal 15 2023-09-22T22:47:00.467 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.1: INFO: Sent signal 15 2023-09-22T22:47:00.467 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mgr.x: INFO: Sent signal 15 2023-09-22T22:47:00.468 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.0.smithi204.stderr: INFO: 2023-09-22T22:21:57.431+0000 7f49831e0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c2 -i 0 (PID: 33354) UID: 0 2023-09-22T22:47:00.468 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.0.smithi204.stderr: INFO: 2023-09-22T22:21:57.431+0000 7f49831e0640 -1 osd.0 55 *** Got signal Terminated *** 2023-09-22T22:47:00.468 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.0.smithi204.stderr: INFO: 2023-09-22T22:21:57.431+0000 7f49831e0640 -1 osd.0 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-22T22:47:00.468 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.2.smithi204.stderr: INFO: 2023-09-22T22:21:57.435+0000 7f22e957d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c2 -i 2 (PID: 33357) UID: 0 2023-09-22T22:47:00.468 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.1.smithi204.stderr: INFO: 2023-09-22T22:21:57.435+0000 7f3efa06f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c2 -i 1 (PID: 33358) UID: 0 2023-09-22T22:47:00.468 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.1.smithi204.stderr: INFO: 2023-09-22T22:21:57.435+0000 7f3efa06f640 -1 osd.1 55 *** Got signal Terminated *** 2023-09-22T22:47:00.468 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.1.smithi204.stderr: INFO: 2023-09-22T22:21:57.435+0000 7f3efa06f640 -1 osd.1 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-22T22:47:00.468 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.2.smithi204.stderr: INFO: 2023-09-22T22:21:57.435+0000 7f22e957d640 -1 osd.2 55 *** Got signal Terminated *** 2023-09-22T22:47:00.468 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.2.smithi204.stderr: INFO: 2023-09-22T22:21:57.435+0000 7f22e957d640 -1 osd.2 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-22T22:47:00.468 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mon.a.smithi204.stderr: INFO: 2023-09-22T22:21:57.435+0000 7efe0545a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster c2 -i a (PID: 33182) UID: 0 2023-09-22T22:47:00.468 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mon.a.smithi204.stderr: INFO: 2023-09-22T22:21:57.435+0000 7efe0545a640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-09-22T22:47:00.469 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.1.smithi204.stdout: INFO: 2023-09-22T22:21:57.435+0000 7fb94ea33640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm (PID: 40448) UID: 0 2023-09-22T22:47:00.469 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mgr.x.smithi204.stderr: INFO: 2023-09-22T22:21:57.435+0000 7fc33b2af640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster c2 -i x (PID: 33184) UID: 0 2023-09-22T22:47:00.469 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mgr.x.smithi204.stderr: INFO: 2023-09-22T22:21:57.435+0000 7fc33b2af640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-09-22T22:47:00.469 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.1.smithi204.stdout: INFO: 2023-09-22T22:21:57.435+0000 7fb95227aa40 -1 shutting down 2023-09-22T22:47:00.469 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-22T22:47:00.469 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.469 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-22T22:47:00.469 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_symmetrical_zonegroup_all: 2023-09-22T22:47:00.469 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-22T22:47:00.470 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-22T22:47:00.470 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-22T22:47:00.470 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-22T22:47:00.470 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 2136, in test_sync_flow_symmetrical_zonegroup_all 2023-09-22T22:47:00.470 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-09-22T22:47:00.470 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 203, in zonegroup_meta_checkpoint 2023-09-22T22:47:00.470 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2023-09-22T22:47:00.470 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2023-09-22T22:47:00.470 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2023-09-22T22:47:00.470 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-22T22:47:00.471 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-22T22:47:00.471 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-22T22:47:00.471 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-22T22:47:00.471 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.471 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:26:58.240+0000 7f9a722e4a40 0 monclient(hunting): authenticate timed out after 300 2023-09-22T22:47:00.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2023-09-22T22:47:00.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2023-09-22T22:47:00.473 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-22T22:47:00.473 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.473 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-22T22:47:00.473 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_single_bucket: 2023-09-22T22:47:00.474 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-22T22:47:00.474 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-22T22:47:00.474 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-22T22:47:00.474 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-22T22:47:00.474 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 2382, in test_sync_single_bucket 2023-09-22T22:47:00.474 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-09-22T22:47:00.474 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 203, in zonegroup_meta_checkpoint 2023-09-22T22:47:00.474 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2023-09-22T22:47:00.475 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2023-09-22T22:47:00.475 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2023-09-22T22:47:00.475 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-22T22:47:00.475 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-22T22:47:00.475 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-22T22:47:00.475 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-22T22:47:00.475 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.475 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:31:58.288+0000 7f1171518a40 0 monclient(hunting): authenticate timed out after 300 2023-09-22T22:47:00.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2023-09-22T22:47:00.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2023-09-22T22:47:00.476 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-22T22:47:00.476 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.477 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-22T22:47:00.477 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_different_buckets: 2023-09-22T22:47:00.477 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-22T22:47:00.477 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-22T22:47:00.477 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-22T22:47:00.477 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-22T22:47:00.477 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 2499, in test_sync_different_buckets 2023-09-22T22:47:00.478 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-09-22T22:47:00.478 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 203, in zonegroup_meta_checkpoint 2023-09-22T22:47:00.478 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2023-09-22T22:47:00.478 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2023-09-22T22:47:00.478 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2023-09-22T22:47:00.478 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-22T22:47:00.478 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-22T22:47:00.478 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-22T22:47:00.478 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-22T22:47:00.479 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.479 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:36:58.379+0000 7f722ad25a40 0 monclient(hunting): authenticate timed out after 300 2023-09-22T22:47:00.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2023-09-22T22:47:00.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2023-09-22T22:47:00.479 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-22T22:47:00.480 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.480 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-22T22:47:00.480 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_multiple_buckets_to_single: 2023-09-22T22:47:00.480 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-22T22:47:00.480 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-22T22:47:00.480 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-22T22:47:00.480 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-22T22:47:00.480 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 2645, in test_sync_multiple_buckets_to_single 2023-09-22T22:47:00.480 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-09-22T22:47:00.481 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 203, in zonegroup_meta_checkpoint 2023-09-22T22:47:00.481 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2023-09-22T22:47:00.481 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2023-09-22T22:47:00.481 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2023-09-22T22:47:00.481 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-22T22:47:00.481 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-22T22:47:00.481 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-22T22:47:00.481 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-22T22:47:00.481 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.482 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:41:58.491+0000 7ffa565c3a40 0 monclient(hunting): authenticate timed out after 300 2023-09-22T22:47:00.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2023-09-22T22:47:00.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2023-09-22T22:47:00.483 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-22T22:47:00.483 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.483 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-22T22:47:00.483 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_single_bucket_to_multiple: 2023-09-22T22:47:00.483 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-22T22:47:00.483 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-22T22:47:00.484 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-22T22:47:00.484 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-22T22:47:00.484 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 2766, in test_sync_single_bucket_to_multiple 2023-09-22T22:47:00.484 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-09-22T22:47:00.484 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 203, in zonegroup_meta_checkpoint 2023-09-22T22:47:00.484 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2023-09-22T22:47:00.484 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2023-09-22T22:47:00.484 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2023-09-22T22:47:00.484 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-22T22:47:00.485 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-22T22:47:00.485 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-22T22:47:00.485 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-22T22:47:00.485 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.485 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-22T22:47:00.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-22T22:47:00.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-22T22:47:00.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-22T22:47:00.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: 2023-09-22T22:46:58.590+0000 7f96eae72a40 0 monclient(hunting): authenticate timed out after 300 2023-09-22T22:47:00.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi160.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2023-09-22T22:47:00.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2023-09-22T22:47:00.486 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-22T22:47:00.486 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.486 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-22T22:47:00.486 INFO:tasks.rgw_multisite_tests:Ran 47 tests in 6154.566s 2023-09-22T22:47:00.487 INFO:tasks.rgw_multisite_tests: 2023-09-22T22:47:00.487 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=4, failures=9) 2023-09-22T22:47:00.489 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-09-22T22:47:00.556 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=eedbdffa67a84becb428ca0f94d6ebce Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-09-22T22:47:00.558 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2023-09-22T22:47:00.578 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2023-09-22T22:47:00.710 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-09-22T22:47:00.982 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-09-22T22:47:00.982 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T22:47:00.984 INFO:tasks.rgw.c1.client.0:Stopped 2023-09-22T22:47:00.984 DEBUG:teuthology.orchestra.run.smithi160:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2023-09-22T22:47:00.995 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f /etc/ceph/vault-root-token 2023-09-22T22:47:01.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['gc', 'process', '--include-all'] 2023-09-22T22:47:01.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'gc', 'process', '--include-all'] 2023-09-22T22:47:01.052 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 gc process --include-all 2023-09-22T22:47:01.130 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2023-09-22T22:47:01.130 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T22:52:01.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-09-22T22:52:01.134+0000 7fd3b0590a40 0 monclient(hunting): authenticate timed out after 300 2023-09-22T22:52:01.152 INFO:teuthology.orchestra.run.smithi160.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-09-22T22:52:01.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T22:52:01.154 ERROR:teuthology.run_tasks:Manager failed: rgw Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw.py", line 473, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw.py", line 300, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw.py", line 242, in start_rgw rgwadmin(ctx, client, cmd=['gc', 'process', '--include-all'], check_status=True) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/util/rgw.py", line 34, in rgwadmin proc = remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 gc process --include-all' 2023-09-22T22:52:01.296 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-22T22:52:01.306 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2023-09-22T22:52:01.307 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-09-22T22:54:01.509 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-09-22T22:54:01.510 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw.py", line 473, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw.py", line 300, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw.py", line 242, in start_rgw rgwadmin(ctx, client, cmd=['gc', 'process', '--include-all'], check_status=True) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/util/rgw.py", line 34, in rgwadmin proc = remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 gc process --include-all' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph_manager.py", line 2742, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph_manager.py", line 2574, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph_manager.py", line 2340, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph_manager.py", line 1595, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph_manager.py", line 1586, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi204 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json' 2023-09-22T22:54:01.596 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-22T22:54:01.597 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-22T22:54:01.597 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2023-09-22T22:54:01.597 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T22:54:01.601 INFO:tasks.ceph.c2.osd.0:Stopped 2023-09-22T22:54:01.601 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2023-09-22T22:54:01.601 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T22:54:01.601 INFO:tasks.ceph.c2.osd.1:Stopped 2023-09-22T22:54:01.602 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2023-09-22T22:54:01.602 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T22:54:01.602 INFO:tasks.ceph.c2.osd.2:Stopped 2023-09-22T22:54:01.602 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-22T22:54:01.602 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2023-09-22T22:54:01.602 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T22:54:01.603 INFO:tasks.ceph.c2.mgr.x:Stopped 2023-09-22T22:54:01.603 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-22T22:54:01.603 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2023-09-22T22:54:01.603 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T22:54:01.603 INFO:tasks.ceph.c2.mon.a:Stopped 2023-09-22T22:54:01.604 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-22T22:54:01.604 DEBUG:teuthology.orchestra.run.smithi204:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2023-09-22T22:54:01.704 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi204.front.sepia.ceph.com 2023-09-22T22:54:01.705 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2023-09-22T22:54:01.896 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi204.front.sepia.ceph.com 2023-09-22T22:54:01.896 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2023-09-22T22:54:02.084 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi204.front.sepia.ceph.com 2023-09-22T22:54:02.084 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2023-09-22T22:54:02.273 INFO:tasks.ceph:Archiving mon data... 2023-09-22T22:54:02.730 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/mkogan-2023-09-22_17:38:01-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7401166/data/c2.mon.a.tgz 2023-09-22T22:54:02.730 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2023-09-22T22:54:02.754 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.iUCLIcJP35 2023-09-22T22:54:02.755 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.iUCLIcJP35 2023-09-22T22:54:02.940 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.iUCLIcJP35 2023-09-22T22:54:03.035 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.iUCLIcJP35 is 443KB 2023-09-22T22:54:03.101 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.iUCLIcJP35 2023-09-22T22:54:03.108 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-22T22:54:03.108 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-09-22T22:54:03.110 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-09-22T22:54:03.190 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw.py", line 473, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw.py", line 300, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw.py", line 242, in start_rgw rgwadmin(ctx, client, cmd=['gc', 'process', '--include-all'], check_status=True) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/util/rgw.py", line 34, in rgwadmin proc = remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 gc process --include-all' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph_manager.py", line 2742, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph_manager.py", line 2574, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph_manager.py", line 2340, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph_manager.py", line 1595, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph_manager.py", line 1586, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi204 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json' 2023-09-22T22:54:03.305 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-22T22:54:03.316 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2023-09-22T22:54:03.316 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-09-22T22:56:03.338 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-09-22T22:56:03.342 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw.py", line 473, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw.py", line 300, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw.py", line 242, in start_rgw rgwadmin(ctx, client, cmd=['gc', 'process', '--include-all'], check_status=True) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/util/rgw.py", line 34, in rgwadmin proc = remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 gc process --include-all' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph_manager.py", line 2742, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph_manager.py", line 2574, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph_manager.py", line 2340, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph_manager.py", line 1595, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph_manager.py", line 1586, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json' 2023-09-22T22:56:03.357 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-22T22:56:03.358 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-22T22:56:03.358 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2023-09-22T22:56:03.358 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T22:56:03.362 INFO:tasks.ceph.c1.osd.0:Stopped 2023-09-22T22:56:03.362 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2023-09-22T22:56:03.362 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T22:56:03.362 INFO:tasks.ceph.c1.osd.1:Stopped 2023-09-22T22:56:03.363 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2023-09-22T22:56:03.363 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T22:56:03.363 INFO:tasks.ceph.c1.osd.2:Stopped 2023-09-22T22:56:03.363 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-22T22:56:03.363 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2023-09-22T22:56:03.364 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T22:56:03.364 INFO:tasks.ceph.c1.mgr.x:Stopped 2023-09-22T22:56:03.364 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-22T22:56:03.364 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2023-09-22T22:56:03.364 INFO:teuthology.orchestra.run:waiting for 300 2023-09-22T22:56:03.365 INFO:tasks.ceph.c1.mon.a:Stopped 2023-09-22T22:56:03.365 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-22T22:56:03.365 DEBUG:teuthology.orchestra.run.smithi160:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2023-09-22T22:56:03.391 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi160.front.sepia.ceph.com 2023-09-22T22:56:03.392 DEBUG:teuthology.orchestra.run.smithi160:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2023-09-22T22:56:03.556 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi160.front.sepia.ceph.com 2023-09-22T22:56:03.557 DEBUG:teuthology.orchestra.run.smithi160:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2023-09-22T22:56:03.730 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi160.front.sepia.ceph.com 2023-09-22T22:56:03.731 DEBUG:teuthology.orchestra.run.smithi160:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2023-09-22T22:56:03.882 INFO:tasks.ceph:Archiving mon data... 2023-09-22T22:56:03.883 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/mkogan-2023-09-22_17:38:01-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7401166/data/c1.mon.a.tgz 2023-09-22T22:56:03.883 DEBUG:teuthology.orchestra.run.smithi160:> mktemp 2023-09-22T22:56:03.888 INFO:teuthology.orchestra.run.smithi160.stdout:/tmp/tmp.ItBzOx7noQ 2023-09-22T22:56:03.889 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.ItBzOx7noQ 2023-09-22T22:56:04.050 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0666 /tmp/tmp.ItBzOx7noQ 2023-09-22T22:56:04.111 DEBUG:teuthology.orchestra.remote:smithi160:/tmp/tmp.ItBzOx7noQ is 404KB 2023-09-22T22:56:04.176 DEBUG:teuthology.orchestra.run.smithi160:> rm -fr /tmp/tmp.ItBzOx7noQ 2023-09-22T22:56:04.181 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-22T22:56:04.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-09-22T22:56:04.227 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-09-22T22:56:04.242 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-22T22:56:04.243 DEBUG:teuthology.orchestra.run.smithi160:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-22T22:56:04.283 DEBUG:teuthology.orchestra.run.smithi204:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-22T22:56:04.304 INFO:teuthology.orchestra.run.smithi204.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-22T22:56:04.305 INFO:teuthology.orchestra.run.smithi160.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-22T22:56:04.309 INFO:tasks.ceph:Archiving crash dumps... 2023-09-22T22:56:04.311 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/lib/ceph/crash to /home/teuthworker/archive/mkogan-2023-09-22_17:38:01-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7401166/remote/smithi160/crash 2023-09-22T22:56:04.312 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-22T22:56:04.367 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/crash to /home/teuthworker/archive/mkogan-2023-09-22_17:38:01-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7401166/remote/smithi204/crash 2023-09-22T22:56:04.368 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-22T22:56:04.383 INFO:tasks.ceph:Compressing logs... 2023-09-22T22:56:04.383 DEBUG:teuthology.orchestra.run.smithi160:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-09-22T22:56:04.411 DEBUG:teuthology.orchestra.run.smithi204:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-09-22T22:56:04.420 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.43013.log 2023-09-22T22:56:04.422 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.50005.log 2023-09-22T22:56:04.422 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.43013.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.43013.log.gz 2023-09-22T22:56:04.423 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.60625.log 2023-09-22T22:56:04.423 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.78878.log 2023-09-22T22:56:04.423 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.50005.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.50005.log.gz 2023-09-22T22:56:04.423 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.60625.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.60625.log.gz 2023-09-22T22:56:04.423 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.77058.log 2023-09-22T22:56:04.423 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.78878.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.87018.log 2023-09-22T22:56:04.424 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.78878.log.gz 2023-09-22T22:56:04.491 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.33304.log 2023-09-22T22:56:04.492 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.37247.log 2023-09-22T22:56:04.492 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.33304.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.40063.log 2023-09-22T22:56:04.492 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.33304.log.gz 2023-09-22T22:56:04.492 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.37247.log: gzip -5 --verbose -- /var/log/ceph/c2-client.1.37545.log 2023-09-22T22:56:04.492 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.37247.log.gz 2023-09-22T22:56:04.492 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.1.37462.log 2023-09-22T22:56:04.492 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.0.40063.log: /var/log/ceph/c2-client.1.37545.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.39170.log 2023-09-22T22:56:04.493 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.40063.log.gz 2023-09-22T22:56:04.493 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.1.37462.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36572.log 2023-09-22T22:56:04.493 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.0.39170.log: gzip -5 --verbose -- /var/log/ceph/c2-osd.1.log 2023-09-22T22:56:04.493 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.39170.log.gz 2023-09-22T22:56:04.493 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.36572.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.37175.log 2023-09-22T22:56:04.493 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36572.log.gz 2023-09-22T22:56:04.493 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36789.log 2023-09-22T22:56:04.494 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.37175.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.41365.log 2023-09-22T22:56:04.494 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.37175.log.gz 2023-09-22T22:56:04.494 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.36789.log: gzip -5 --verbose -- /var/log/ceph/rgw.c2.client.0.log 2023-09-22T22:56:04.494 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36789.log.gz 2023-09-22T22:56:04.494 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.0.41365.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.37294.log 2023-09-22T22:56:04.494 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/rgw.c2.client.0.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36599.log 2023-09-22T22:56:04.495 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.0.37294.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30707.log 2023-09-22T22:56:04.495 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.36599.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.33425.log 2023-09-22T22:56:04.495 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36599.log.gz 2023-09-22T22:56:04.495 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph.tmp-client.admin.30707.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.40267.log 2023-09-22T22:56:04.495 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30707.log.gz 2023-09-22T22:56:04.495 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.33425.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36719.log 2023-09-22T22:56:04.495 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.33425.log.gz 2023-09-22T22:56:04.496 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.0.40267.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.40167.log 2023-09-22T22:56:04.496 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.36719.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.39247.log 2023-09-22T22:56:04.496 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36719.log.gz 2023-09-22T22:56:04.496 INFO:teuthology.orchestra.run.smithi204.stderr: 44.4% -- replaced with /var/log/ceph/c2-client.1.37545.log.gz 2023-09-22T22:56:04.496 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.40167.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36867.log 2023-09-22T22:56:04.496 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.40167.log.gz 2023-09-22T22:56:04.496 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.0.39247.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.33186.log 2023-09-22T22:56:04.497 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.36867.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.37125.log 2023-09-22T22:56:04.497 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36867.log.gz 2023-09-22T22:56:04.497 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.33186.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.33328.log 2023-09-22T22:56:04.497 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.33186.log.gz 2023-09-22T22:56:04.497 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.37125.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.37125.log.gz 2023-09-22T22:56:04.497 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.41279.log 2023-09-22T22:56:04.497 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.33328.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.41107.log 2023-09-22T22:56:04.498 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.33328.log.gz 2023-09-22T22:56:04.498 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.0.41279.log: gzip -5 --verbose -- /var/log/ceph/ops-log-c2-client.1.log 2023-09-22T22:56:04.498 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.0.41107.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36519.log 2023-09-22T22:56:04.498 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ops-log-c2-client.1.log: gzip -5 --verbose -- /var/log/ceph/c2.log 2023-09-22T22:56:04.498 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.36519.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.38888.log 2023-09-22T22:56:04.498 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36519.log.gz 2023-09-22T22:56:04.498 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.41193.log 2023-09-22T22:56:04.499 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.0.38888.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/c2-client.0.38888.log.gz 2023-09-22T22:56:04.499 INFO:teuthology.orchestra.run.smithi204.stderr: /var/log/ceph/c2-client.admin.40141.log 2023-09-22T22:56:04.499 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.0.41193.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.39147.log 2023-09-22T22:56:04.499 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.40141.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36796.log 2023-09-22T22:56:04.499 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.40141.log.gz 2023-09-22T22:56:04.499 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.39147.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.39147.log.gz 2023-09-22T22:56:04.499 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36650.log 2023-09-22T22:56:04.500 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36544.log 2023-09-22T22:56:04.500 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.36796.log: 0.0%/var/log/ceph/c2-client.admin.36650.log: -- replaced with /var/log/ceph/c2-client.admin.36796.log.gz 2023-09-22T22:56:04.500 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36650.log.gz 2023-09-22T22:56:04.500 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36874.log 2023-09-22T22:56:04.500 INFO:teuthology.orchestra.run.smithi204.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.41365.log.gz 2023-09-22T22:56:04.500 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.36544.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36718.log 2023-09-22T22:56:04.500 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36544.log.gz 2023-09-22T22:56:04.501 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.36874.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36874.log.gz 2023-09-22T22:56:04.501 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/rgw.c2.client.1.log 2023-09-22T22:56:04.501 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.37223.log 2023-09-22T22:56:04.501 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.36718.log: /var/log/ceph/rgw.c2.client.1.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36718.log.gz 2023-09-22T22:56:04.501 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.40190.log 2023-09-22T22:56:04.501 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.37223.log: gzip -5 --verbose -- /var/log/ceph/ops-log-c2-client.0.log 2023-09-22T22:56:04.502 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.37223.log.gz 2023-09-22T22:56:04.502 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.0.40190.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.40190.log.gzgzip 2023-09-22T22:56:04.502 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/c2-client.admin.33363.log 2023-09-22T22:56:04.502 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36797.log 2023-09-22T22:56:04.502 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ops-log-c2-client.0.log: /var/log/ceph/c2-client.admin.33363.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.33363.log.gz 2023-09-22T22:56:04.502 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.37271.log 2023-09-22T22:56:04.503 INFO:teuthology.orchestra.run.smithi204.stderr: 88.7% -- replaced with /var/log/ceph/c2.log.gz 2023-09-22T22:56:04.503 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.39985.log 2023-09-22T22:56:04.503 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.36797.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36797.log.gz/var/log/ceph/c2-client.admin.37271.log: 2023-09-22T22:56:04.503 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.37271.log.gz 2023-09-22T22:56:04.503 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36685.log 2023-09-22T22:56:04.503 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.0.39985.log: gzip -5 --verbose -- 0.0% /var/log/ceph/c2-client.admin.39121.log -- replaced with /var/log/ceph/c2-client.0.39985.log.gz 2023-09-22T22:56:04.503 INFO:teuthology.orchestra.run.smithi204.stderr: 2023-09-22T22:56:04.504 INFO:teuthology.orchestra.run.smithi204.stderr: 19.7% -- replaced with /var/log/ceph/c2-client.0.39247.log.gz 2023-09-22T22:56:04.504 INFO:teuthology.orchestra.run.smithi204.stderr: 19.1% -- replaced with /var/log/ceph/c2-client.0.40267.log.gz 2023-09-22T22:56:04.504 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.36685.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36685.log.gz 2023-09-22T22:56:04.504 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/c2.audit.log 2023-09-22T22:56:04.504 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.39121.log: gzip -5 --verbose -- /var/log/ceph/c2-mgr.x.log 2023-09-22T22:56:04.504 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.39121.log.gz 2023-09-22T22:56:04.504 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2.audit.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.37149.log 2023-09-22T22:56:04.505 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.37099.log 2023-09-22T22:56:04.505 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.37149.log: gzip -5 --verbose -- /var/log/ceph/c2-mon.a.log 2023-09-22T22:56:04.505 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.37149.log.gz 2023-09-22T22:56:04.505 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.37099.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.33280.log 2023-09-22T22:56:04.505 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.37099.log.gz 2023-09-22T22:56:04.505 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.37199.log 2023-09-22T22:56:04.505 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.33280.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.43709.log 2023-09-22T22:56:04.506 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.33280.log.gz 2023-09-22T22:56:04.506 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.37199.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.37049.log 2023-09-22T22:56:04.506 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.37199.log.gz 2023-09-22T22:56:04.506 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/c2-osd.2.log 2023-09-22T22:56:04.506 INFO:teuthology.orchestra.run.smithi204.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.41107.log.gz 2023-09-22T22:56:04.506 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.43709.log: /var/log/ceph/c2-client.admin.37049.log: 84.3% -- replaced with /var/log/ceph/c2-client.0.37294.log.gz 2023-09-22T22:56:04.507 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36715.log 2023-09-22T22:56:04.507 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/c2-client.admin.37049.log.gz 2023-09-22T22:56:04.507 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/c2-client.admin.43709.log.gz 2023-09-22T22:56:04.507 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36949.log 2023-09-22T22:56:04.507 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.36715.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36954.log 2023-09-22T22:56:04.507 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36715.log.gz 2023-09-22T22:56:04.507 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.36949.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36945.log 2023-09-22T22:56:04.508 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36949.log.gz 2023-09-22T22:56:04.508 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.36954.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.37075.log 2023-09-22T22:56:04.508 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36954.log.gz 2023-09-22T22:56:04.508 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/c2-osd.0.log 2023-09-22T22:56:04.508 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.36945.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36945.log.gz 2023-09-22T22:56:04.508 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.37075.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.37023.log 2023-09-22T22:56:04.508 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.37075.log.gz 2023-09-22T22:56:04.509 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36875.log 2023-09-22T22:56:04.509 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.37385.log 2023-09-22T22:56:04.509 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.37023.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.37023.log.gz 2023-09-22T22:56:04.509 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.admin.36875.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.39043.log 2023-09-22T22:56:04.509 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36875.log.gz 2023-09-22T22:56:04.509 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.38966.log 2023-09-22T22:56:04.509 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.0.37385.log: /var/log/ceph/c2-client.0.39043.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.39043.log.gz 2023-09-22T22:56:04.510 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c2-client.0.38966.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.38966.log.gz 2023-09-22T22:56:04.553 INFO:teuthology.orchestra.run.smithi204.stderr: 44.0% -- replaced with /var/log/ceph/c2-client.0.37385.log.gz 2023-09-22T22:56:04.554 INFO:teuthology.orchestra.run.smithi204.stderr: 83.0% -- replaced with /var/log/ceph/c2-client.1.37462.log.gz 2023-09-22T22:56:04.554 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/ops-log-c2-client.0.log.gz 2023-09-22T22:56:04.554 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/ops-log-c2-client.1.log.gz 2023-09-22T22:56:04.584 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.77058.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.58421.log 2023-09-22T22:56:04.584 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.77058.log.gz 2023-09-22T22:56:04.584 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.87018.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.87018.log.gz 2023-09-22T22:56:04.584 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.93589.log 2023-09-22T22:56:04.585 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.58421.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.48908.log 2023-09-22T22:56:04.585 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.58421.log.gz 2023-09-22T22:56:04.585 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.93589.log: gzip -5 --verbose -- 47.4% /var/log/ceph/c1-client.0.94795.log 2023-09-22T22:56:04.585 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.93589.log.gz 2023-09-22T22:56:04.585 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.48908.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.88190.log 2023-09-22T22:56:04.585 INFO:teuthology.orchestra.run.smithi160.stderr: 48.7% -- replaced with /var/log/ceph/c1-client.0.48908.log.gz 2023-09-22T22:56:04.585 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.94795.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.94795.log.gz 2023-09-22T22:56:04.586 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.56276.log 2023-09-22T22:56:04.586 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.88190.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.88190.log.gz 2023-09-22T22:56:04.586 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.45152.log 2023-09-22T22:56:04.586 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.56276.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36658.log 2023-09-22T22:56:04.586 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.56276.log.gz 2023-09-22T22:56:04.586 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.45152.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.49922.log 2023-09-22T22:56:04.587 INFO:teuthology.orchestra.run.smithi160.stderr: 70.3% -- replaced with /var/log/ceph/c1-client.0.45152.log.gz 2023-09-22T22:56:04.587 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.36658.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36658.log.gz 2023-09-22T22:56:04.587 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.70517.log 2023-09-22T22:56:04.587 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.79050.log 2023-09-22T22:56:04.587 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.49922.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.49922.log.gz 2023-09-22T22:56:04.587 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.70517.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.72346.log 2023-09-22T22:56:04.587 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.70517.log.gz 2023-09-22T22:56:04.588 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.79050.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.47283.log 2023-09-22T22:56:04.588 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.79050.log.gz 2023-09-22T22:56:04.588 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.72346.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.45412.log 2023-09-22T22:56:04.588 INFO:teuthology.orchestra.run.smithi160.stderr: 44.8% -- replaced with /var/log/ceph/c1-client.0.72346.log.gz 2023-09-22T22:56:04.588 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.47283.log: 49.7%gzip -- replaced with /var/log/ceph/c1-client.0.47283.log.gz 2023-09-22T22:56:04.588 INFO:teuthology.orchestra.run.smithi160.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.73231.log 2023-09-22T22:56:04.588 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.45412.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.81596.log 2023-09-22T22:56:04.589 INFO:teuthology.orchestra.run.smithi160.stderr: 68.4% -- replaced with /var/log/ceph/c1-client.0.45412.log.gz 2023-09-22T22:56:04.589 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.73231.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.73702.log 2023-09-22T22:56:04.589 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.73231.log.gz 2023-09-22T22:56:04.589 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.81596.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.57560.log 2023-09-22T22:56:04.589 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.81596.log.gz 2023-09-22T22:56:04.589 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.73702.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.50943.log 2023-09-22T22:56:04.589 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9% -- replaced with /var/log/ceph/c1-client.0.73702.log.gz 2023-09-22T22:56:04.590 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.52034.log 2023-09-22T22:56:04.590 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.57560.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.57560.log.gz 2023-09-22T22:56:04.590 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.50943.log: gzip -5 --verbose -- /var/log/ceph/c1-mgr.x.log 2023-09-22T22:56:04.590 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.50943.log.gz 2023-09-22T22:56:04.590 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.52034.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.52034.log.gz 2023-09-22T22:56:04.590 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.93159.log 2023-09-22T22:56:04.591 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.95246.log 2023-09-22T22:56:04.591 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.93159.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.66235.log 2023-09-22T22:56:04.591 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.93159.log.gz 2023-09-22T22:56:04.591 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.95246.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.89611.log 2023-09-22T22:56:04.591 INFO:teuthology.orchestra.run.smithi160.stderr: 9.2% -- replaced with /var/log/ceph/c1-client.0.95246.log.gz 2023-09-22T22:56:04.591 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.66235.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.50434.log 2023-09-22T22:56:04.591 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.66235.log.gz 2023-09-22T22:56:04.592 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.89611.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/c1-client.0.89611.log.gz -- 2023-09-22T22:56:04.592 INFO:teuthology.orchestra.run.smithi160.stderr: /var/log/ceph/c1-client.0.74738.log 2023-09-22T22:56:04.592 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.50434.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.88857.log 2023-09-22T22:56:04.592 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.50434.log.gz 2023-09-22T22:56:04.592 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.74738.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.83519.log 2023-09-22T22:56:04.592 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.74738.log.gz 2023-09-22T22:56:04.592 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.88857.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.58768.log 2023-09-22T22:56:04.593 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.88857.log.gz 2023-09-22T22:56:04.593 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.83519.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.37109.log 2023-09-22T22:56:04.593 INFO:teuthology.orchestra.run.smithi160.stderr: 58.3% -- replaced with /var/log/ceph/c1-client.0.83519.log.gz 2023-09-22T22:56:04.593 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.58768.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.61046.log 2023-09-22T22:56:04.593 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.58768.log.gz 2023-09-22T22:56:04.593 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.37109.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.63493.log 2023-09-22T22:56:04.593 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.37109.log.gz 2023-09-22T22:56:04.593 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.61046.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.72871.log 2023-09-22T22:56:04.594 INFO:teuthology.orchestra.run.smithi160.stderr: 90.0% -- replaced with /var/log/ceph/c1-client.0.61046.log.gz 2023-09-22T22:56:04.594 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.63493.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.58335.log 2023-09-22T22:56:04.594 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.63493.log.gz 2023-09-22T22:56:04.594 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.72871.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.47197.log 2023-09-22T22:56:04.594 INFO:teuthology.orchestra.run.smithi160.stderr: 86.3% -- replaced with /var/log/ceph/c1-client.0.72871.log.gz 2023-09-22T22:56:04.594 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.58335.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.81003.log 2023-09-22T22:56:04.595 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.58335.log.gz 2023-09-22T22:56:04.595 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.47197.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.55879.log 2023-09-22T22:56:04.595 INFO:teuthology.orchestra.run.smithi160.stderr: 82.3% -- replaced with /var/log/ceph/c1-client.0.47197.log.gz 2023-09-22T22:56:04.595 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.81003.log: 54.1% -- replaced with /var/log/ceph/c1-client.0.81003.log.gz 2023-09-22T22:56:04.595 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.63661.log 2023-09-22T22:56:04.595 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.55879.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.37350.log 2023-09-22T22:56:04.595 INFO:teuthology.orchestra.run.smithi160.stderr: 41.9% -- replaced with /var/log/ceph/c1-client.0.55879.log.gz 2023-09-22T22:56:04.596 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.63661.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.63661.log.gz 2023-09-22T22:56:04.596 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.49586.log 2023-09-22T22:56:04.596 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.37350.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.67181.log 2023-09-22T22:56:04.596 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.37350.log.gz 2023-09-22T22:56:04.596 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.49586.log: 68.4% -- replaced with /var/log/ceph/c1-client.0.49586.log.gz 2023-09-22T22:56:04.596 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.76818.log 2023-09-22T22:56:04.596 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.67181.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.67181.log.gz 2023-09-22T22:56:04.597 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.45494.log 2023-09-22T22:56:04.597 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.76818.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.76818.log.gz 2023-09-22T22:56:04.597 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.75218.log 2023-09-22T22:56:04.597 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.45494.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.43704.log 2023-09-22T22:56:04.597 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.45494.log.gz 2023-09-22T22:56:04.597 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.75218.log: gzip 0.0% -- replaced with /var/log/ceph/c1-client.0.75218.log.gz -5 2023-09-22T22:56:04.597 INFO:teuthology.orchestra.run.smithi160.stderr: --verbose -- /var/log/ceph/c1-client.0.76978.log 2023-09-22T22:56:04.598 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.43704.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.59694.log 2023-09-22T22:56:04.598 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.43704.log.gz 2023-09-22T22:56:04.598 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.76978.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.76978.log.gz 2023-09-22T22:56:04.598 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.55791.log 2023-09-22T22:56:04.598 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.59694.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.59694.log.gz 2023-09-22T22:56:04.598 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.47030.log 2023-09-22T22:56:04.598 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.55791.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.55791.log.gz 2023-09-22T22:56:04.599 INFO:teuthology.orchestra.run.smithi160.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.92270.log 2023-09-22T22:56:04.599 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.47030.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.54311.log 2023-09-22T22:56:04.599 INFO:teuthology.orchestra.run.smithi160.stderr: 64.9% -- replaced with /var/log/ceph/c1-client.0.47030.log.gz 2023-09-22T22:56:04.599 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.92270.log: gzip -5 44.8% --verbose -- replaced with /var/log/ceph/c1-client.0.92270.log.gz -- 2023-09-22T22:56:04.599 INFO:teuthology.orchestra.run.smithi160.stderr: /var/log/ceph/c1-client.admin.33203.log 2023-09-22T22:56:04.599 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.54311.log: gzip 0.0% -- replaced with /var/log/ceph/c1-client.0.54311.log.gz 2023-09-22T22:56:04.599 INFO:teuthology.orchestra.run.smithi160.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.46692.log 2023-09-22T22:56:04.600 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.33203.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.33203.log.gzgzip 2023-09-22T22:56:04.600 INFO:teuthology.orchestra.run.smithi160.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.59195.log 2023-09-22T22:56:04.600 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.46692.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.66484.log 2023-09-22T22:56:04.600 INFO:teuthology.orchestra.run.smithi160.stderr: 53.9% -- replaced with /var/log/ceph/c1-client.0.46692.log.gz 2023-09-22T22:56:04.600 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.59195.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.59195.log.gz 2023-09-22T22:56:04.600 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.84599.log 2023-09-22T22:56:04.600 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.66484.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.65642.log 2023-09-22T22:56:04.601 INFO:teuthology.orchestra.run.smithi160.stderr: 90.9% -- replaced with /var/log/ceph/c1-client.0.66484.log.gz 2023-09-22T22:56:04.601 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.84599.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.84599.log.gz 2023-09-22T22:56:04.601 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.55999.log 2023-09-22T22:56:04.601 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.65642.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.65642.log.gz 2023-09-22T22:56:04.601 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.48555.log 2023-09-22T22:56:04.601 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.55999.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.84855.log 2023-09-22T22:56:04.601 INFO:teuthology.orchestra.run.smithi160.stderr: 4.8% -- replaced with /var/log/ceph/c1-client.0.55999.log.gz 2023-09-22T22:56:04.602 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.48555.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.51194.log 2023-09-22T22:56:04.602 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.48555.log.gz 2023-09-22T22:56:04.602 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.84855.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.84855.log.gz 2023-09-22T22:56:04.602 INFO:teuthology.orchestra.run.smithi160.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.66069.log 2023-09-22T22:56:04.602 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.51194.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.51194.log.gz 2023-09-22T22:56:04.602 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.83602.log 2023-09-22T22:56:04.603 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.66069.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.66069.log.gz 2023-09-22T22:56:04.603 INFO:teuthology.orchestra.run.smithi160.stderr: -5 --verbose -- /var/log/ceph/c1-osd.2.log 2023-09-22T22:56:04.603 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.83602.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.47369.log 2023-09-22T22:56:04.603 INFO:teuthology.orchestra.run.smithi160.stderr: 64.9% -- replaced with /var/log/ceph/c1-client.0.83602.log.gz 2023-09-22T22:56:04.603 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.60708.log 2023-09-22T22:56:04.603 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.47369.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.47369.log.gz 2023-09-22T22:56:04.603 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.79309.log 2023-09-22T22:56:04.604 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.60708.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.70185.log 2023-09-22T22:56:04.604 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.60708.log.gz 2023-09-22T22:56:04.604 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.79309.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.79309.log.gz 2023-09-22T22:56:04.604 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.37397.log 2023-09-22T22:56:04.604 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.70185.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.70185.log.gz 2023-09-22T22:56:04.604 INFO:teuthology.orchestra.run.smithi160.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.90632.log 2023-09-22T22:56:04.604 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.37397.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.88691.log 2023-09-22T22:56:04.605 INFO:teuthology.orchestra.run.smithi160.stderr: 84.2% -- replaced with /var/log/ceph/c1-client.0.37397.log.gz 2023-09-22T22:56:04.605 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.90632.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.90632.log.gz 2023-09-22T22:56:04.605 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.69421.log 2023-09-22T22:56:04.605 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.88691.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.93245.log 2023-09-22T22:56:04.605 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.88691.log.gz 2023-09-22T22:56:04.605 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.69421.log: 88.1% -- replaced with /var/log/ceph/c1-client.0.69421.log.gz 2023-09-22T22:56:04.605 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.66665.log 2023-09-22T22:56:04.606 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.93245.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.75138.log 2023-09-22T22:56:04.606 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.93245.log.gz 2023-09-22T22:56:04.606 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.66665.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.66665.log.gz 2023-09-22T22:56:04.606 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.54646.log 2023-09-22T22:56:04.606 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.75138.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.52614.log 2023-09-22T22:56:04.606 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.75138.log.gz 2023-09-22T22:56:04.606 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.54646.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.54646.log.gz 2023-09-22T22:56:04.607 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.33321.log 2023-09-22T22:56:04.607 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.52614.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.63573.log 2023-09-22T22:56:04.607 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.52614.log.gz 2023-09-22T22:56:04.607 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.33321.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/c1-client.admin.33321.log.gz -- 2023-09-22T22:56:04.607 INFO:teuthology.orchestra.run.smithi160.stderr: /var/log/ceph/c1-client.0.83260.log 2023-09-22T22:56:04.607 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.63573.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.63573.log.gz 2023-09-22T22:56:04.607 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.47960.log 2023-09-22T22:56:04.608 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.83260.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.83260.log.gz 2023-09-22T22:56:04.608 INFO:teuthology.orchestra.run.smithi160.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.74818.log 2023-09-22T22:56:04.608 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.47960.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.91666.log 2023-09-22T22:56:04.608 INFO:teuthology.orchestra.run.smithi160.stderr: 50.4% -- replaced with /var/log/ceph/c1-client.0.47960.log.gz 2023-09-22T22:56:04.608 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.74818.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.74818.log.gz 2023-09-22T22:56:04.608 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.67278.log 2023-09-22T22:56:04.608 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.91666.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.79481.log 2023-09-22T22:56:04.609 INFO:teuthology.orchestra.run.smithi160.stderr: 44.2% -- replaced with /var/log/ceph/c1-client.0.91666.log.gz 2023-09-22T22:56:04.609 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.67278.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.67278.log.gz 2023-09-22T22:56:04.609 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/rgw.c1.client.1.log 2023-09-22T22:56:04.609 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.79481.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.77218.log 2023-09-22T22:56:04.609 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.79481.log.gz 2023-09-22T22:56:04.609 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/rgw.c1.client.1.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.47874.log 2023-09-22T22:56:04.609 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.77218.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.77218.log.gz 2023-09-22T22:56:04.610 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.61385.log 2023-09-22T22:56:04.610 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.47874.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.70018.log 2023-09-22T22:56:04.610 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.47874.log.gz 2023-09-22T22:56:04.610 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.61385.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.61385.log.gz 2023-09-22T22:56:04.610 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.89694.log 2023-09-22T22:56:04.610 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.70018.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.59524.log 2023-09-22T22:56:04.611 INFO:teuthology.orchestra.run.smithi160.stderr: 68.4% -- replaced with /var/log/ceph/c1-client.0.70018.log.gz 2023-09-22T22:56:04.611 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.89694.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.89694.log.gz 2023-09-22T22:56:04.611 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.95354.log 2023-09-22T22:56:04.611 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.74978.log 2023-09-22T22:56:04.611 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.59524.log: /var/log/ceph/c1-client.0.95354.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.59524.log.gz 2023-09-22T22:56:04.611 INFO:teuthology.orchestra.run.smithi160.stderr: 10.2% -- replaced with /var/log/ceph/c1-client.0.95354.log.gz 2023-09-22T22:56:04.611 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.43790.log 2023-09-22T22:56:04.612 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.74978.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.89106.log 2023-09-22T22:56:04.612 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.74978.log.gz 2023-09-22T22:56:04.612 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.43790.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.43790.log.gz 2023-09-22T22:56:04.612 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.52700.log 2023-09-22T22:56:04.612 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.81256.log 2023-09-22T22:56:04.612 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.89106.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.89106.log.gz 2023-09-22T22:56:04.612 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.52700.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.73794.log 2023-09-22T22:56:04.612 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.52700.log.gz 2023-09-22T22:56:04.613 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.81256.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.61809.log 2023-09-22T22:56:04.613 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.81256.log.gz 2023-09-22T22:56:04.613 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.73794.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.73794.log.gz 2023-09-22T22:56:04.613 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.59945.log 2023-09-22T22:56:04.613 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.61809.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94881.log 2023-09-22T22:56:04.613 INFO:teuthology.orchestra.run.smithi160.stderr: 44.8% -- replaced with /var/log/ceph/c1-client.0.61809.log.gz 2023-09-22T22:56:04.613 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.59945.log: gzip -5 --verbose -- 0.0% /var/log/ceph/c1-client.0.91494.log 2023-09-22T22:56:04.614 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.59945.log.gz 2023-09-22T22:56:04.614 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.94881.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.94881.log.gz 2023-09-22T22:56:04.614 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.88774.log 2023-09-22T22:56:04.614 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.39274.log 2023-09-22T22:56:04.614 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.91494.log: 45.5% -- replaced with /var/log/ceph/c1-client.0.91494.log.gz 2023-09-22T22:56:04.614 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.88774.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.76418.log 2023-09-22T22:56:04.615 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.88774.log.gz 2023-09-22T22:56:04.615 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.39274.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.39274.log.gz 2023-09-22T22:56:04.615 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.49080.log 2023-09-22T22:56:04.615 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.66748.log 2023-09-22T22:56:04.615 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.76418.log: /var/log/ceph/c1-client.0.49080.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.76418.log.gz 2023-09-22T22:56:04.615 INFO:teuthology.orchestra.run.smithi160.stderr: 63.1% -- replaced with /var/log/ceph/c1-client.0.49080.log.gz 2023-09-22T22:56:04.615 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.93331.log 2023-09-22T22:56:04.616 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.88940.log 2023-09-22T22:56:04.616 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.66748.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.66748.log.gz 2023-09-22T22:56:04.616 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.93331.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36734.log 2023-09-22T22:56:04.616 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.93331.log.gz 2023-09-22T22:56:04.616 INFO:teuthology.orchestra.run.smithi160.stderr:gzip/var/log/ceph/c1-client.0.88940.log: -5 --verbose -- /var/log/ceph/c1-client.0.93417.log 2023-09-22T22:56:04.616 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.88940.log.gz 2023-09-22T22:56:04.616 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.36734.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36734.log.gz 2023-09-22T22:56:04.617 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.63751.log 2023-09-22T22:56:04.617 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.93417.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.63325.log 2023-09-22T22:56:04.617 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.93417.log.gz 2023-09-22T22:56:04.617 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.63751.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.37326.log 2023-09-22T22:56:04.617 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.63751.log.gz 2023-09-22T22:56:04.617 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.63325.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.75698.log 2023-09-22T22:56:04.617 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.63325.log.gz 2023-09-22T22:56:04.618 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.37326.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.37326.log.gz 2023-09-22T22:56:04.618 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.63408.log 2023-09-22T22:56:04.618 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.75698.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.75938.log 2023-09-22T22:56:04.618 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.75698.log.gz 2023-09-22T22:56:04.618 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.63408.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.63408.log.gz 2023-09-22T22:56:04.618 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.74418.log 2023-09-22T22:56:04.618 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.95338.log 2023-09-22T22:56:04.619 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.75938.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.75938.log.gz 2023-09-22T22:56:04.619 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.74418.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.95283.log 2023-09-22T22:56:04.619 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.74418.log.gz 2023-09-22T22:56:04.619 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.51448.log 2023-09-22T22:56:04.619 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.95338.log: 9.2% -- replaced with /var/log/ceph/c1-client.0.95338.log.gz 2023-09-22T22:56:04.619 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.95283.log: 8.2% -- replaced with /var/log/ceph/c1-client.0.95283.log.gz 2023-09-22T22:56:04.619 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.91322.log 2023-09-22T22:56:04.620 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.66915.log 2023-09-22T22:56:04.620 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.51448.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.51448.log.gz 2023-09-22T22:56:04.620 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.91322.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36883.log 2023-09-22T22:56:04.620 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.91322.log.gz 2023-09-22T22:56:04.620 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.66915.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.46009.log 2023-09-22T22:56:04.620 INFO:teuthology.orchestra.run.smithi160.stderr: 58.8% -- replaced with /var/log/ceph/c1-client.0.66915.log.gz 2023-09-22T22:56:04.620 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.36883.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36961.log 2023-09-22T22:56:04.620 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36883.log.gz 2023-09-22T22:56:04.621 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.46009.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.74067.log 2023-09-22T22:56:04.621 INFO:teuthology.orchestra.run.smithi160.stderr: 63.5% -- replaced with /var/log/ceph/c1-client.0.46009.log.gz 2023-09-22T22:56:04.621 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.36961.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36961.log.gz 2023-09-22T22:56:04.621 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.73429.log 2023-09-22T22:56:04.621 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.86594.log 2023-09-22T22:56:04.621 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.74067.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.74067.log.gz 2023-09-22T22:56:04.621 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.73429.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.70434.log 2023-09-22T22:56:04.622 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9% -- replaced with /var/log/ceph/c1-client.0.73429.log.gz 2023-09-22T22:56:04.622 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.86594.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.86594.log.gz 2023-09-22T22:56:04.622 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.52527.log 2023-09-22T22:56:04.622 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.95143.log 2023-09-22T22:56:04.622 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.70434.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.70434.log.gz 2023-09-22T22:56:04.622 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.52527.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.62479.log 2023-09-22T22:56:04.622 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.52527.log.gz 2023-09-22T22:56:04.623 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.95143.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.74150.log 47.4% 2023-09-22T22:56:04.623 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.95143.log.gz 2023-09-22T22:56:04.623 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.62479.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.78964.log 2023-09-22T22:56:04.623 INFO:teuthology.orchestra.run.smithi160.stderr: 45.5% -- replaced with /var/log/ceph/c1-client.0.62479.log.gz 2023-09-22T22:56:04.623 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.74150.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.43969.log 2023-09-22T22:56:04.623 INFO:teuthology.orchestra.run.smithi160.stderr: 94.3% -- replaced with /var/log/ceph/c1-client.0.74150.log.gz 2023-09-22T22:56:04.623 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.65725.log 2023-09-22T22:56:04.624 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.78964.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.78964.log.gz 2023-09-22T22:56:04.624 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.43969.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.43969.log.gz 2023-09-22T22:56:04.624 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.55471.log 2023-09-22T22:56:04.624 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.37302.log 2023-09-22T22:56:04.624 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.65725.log: 45.5% -- replaced with /var/log/ceph/c1-client.0.65725.log.gz 2023-09-22T22:56:04.624 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.55471.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.55471.log.gz 2023-09-22T22:56:04.624 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.50178.log 2023-09-22T22:56:04.625 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.90976.log 2023-09-22T22:56:04.625 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.37302.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.37302.log.gz 2023-09-22T22:56:04.625 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.50178.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.46264.log 2023-09-22T22:56:04.625 INFO:teuthology.orchestra.run.smithi160.stderr: 45.5% -- replaced with /var/log/ceph/c1-client.0.50178.log.gz 2023-09-22T22:56:04.625 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.51697.log 2023-09-22T22:56:04.625 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.90976.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.90976.log.gz 2023-09-22T22:56:04.625 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.46264.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.38993.log 2023-09-22T22:56:04.625 INFO:teuthology.orchestra.run.smithi160.stderr: 84.1% -- replaced with /var/log/ceph/c1-client.0.46264.log.gz 2023-09-22T22:56:04.626 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.51697.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.63158.log 2023-09-22T22:56:04.626 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.51697.log.gz 2023-09-22T22:56:04.626 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.38993.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.38993.log.gz 2023-09-22T22:56:04.626 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.57819.log 2023-09-22T22:56:04.626 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.93847.log 2023-09-22T22:56:04.626 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.63158.log: 69.7% -- replaced with /var/log/ceph/c1-client.0.63158.log.gz 2023-09-22T22:56:04.627 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.57819.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.75058.log 2023-09-22T22:56:04.627 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.57819.log.gz 2023-09-22T22:56:04.627 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.93847.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.61895.log 2023-09-22T22:56:04.627 INFO:teuthology.orchestra.run.smithi160.stderr: 45.5% -- replaced with /var/log/ceph/c1-client.0.93847.log.gz 2023-09-22T22:56:04.627 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.75058.log: gzip -5 --verbose -- /var/log/ceph/ops-log-c1-client.1.log 2023-09-22T22:56:04.627 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.75058.log.gz 2023-09-22T22:56:04.627 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.61895.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.77138.log 2023-09-22T22:56:04.627 INFO:teuthology.orchestra.run.smithi160.stderr: 57.7% -- replaced with /var/log/ceph/c1-client.0.61895.log.gz 2023-09-22T22:56:04.628 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.84445.log 2023-09-22T22:56:04.628 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ops-log-c1-client.1.log: /var/log/ceph/c1-client.0.77138.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.77138.log.gz 2023-09-22T22:56:04.628 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.85353.log 2023-09-22T22:56:04.628 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.79136.log 2023-09-22T22:56:04.628 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.84445.log: /var/log/ceph/c1-client.0.85353.log: 96.2%gzip -5 --verbose -- /var/log/ceph/c1-client.admin.37161.log 2023-09-22T22:56:04.628 INFO:teuthology.orchestra.run.smithi160.stderr: 69.4% -- replaced with /var/log/ceph/c1-client.0.85353.log.gz 2023-09-22T22:56:04.629 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.84445.log.gz 2023-09-22T22:56:04.629 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.77298.log 2023-09-22T22:56:04.629 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.79136.log: 45.5%/var/log/ceph/c1-client.admin.37161.log: -- replaced with /var/log/ceph/c1-client.0.79136.log.gz 2023-09-22T22:56:04.629 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.37161.log.gz 2023-09-22T22:56:04.629 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.61217.log 2023-09-22T22:56:04.629 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.90718.log 2023-09-22T22:56:04.629 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.77298.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.77298.log.gz 2023-09-22T22:56:04.630 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.61217.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.56614.log 2023-09-22T22:56:04.630 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.61217.log.gz 2023-09-22T22:56:04.630 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30722.log 2023-09-22T22:56:04.630 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.90718.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.90718.log.gz 2023-09-22T22:56:04.630 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.56614.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.50264.log 2023-09-22T22:56:04.630 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.56614.log.gz 2023-09-22T22:56:04.630 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph.tmp-client.admin.30722.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.75778.log 0.0% 2023-09-22T22:56:04.631 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/ceph.tmp-client.admin.30722.log.gz 2023-09-22T22:56:04.631 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.64179.log 2023-09-22T22:56:04.631 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.50264.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.50264.log.gz 2023-09-22T22:56:04.631 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.75778.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.56531.log 2023-09-22T22:56:04.631 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.75778.log.gz 2023-09-22T22:56:04.631 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.64179.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.64179.log.gz -5 2023-09-22T22:56:04.631 INFO:teuthology.orchestra.run.smithi160.stderr: --verbose -- /var/log/ceph/c1-client.0.44900.log 2023-09-22T22:56:04.631 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36802.log 2023-09-22T22:56:04.632 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.56531.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.56531.log.gz 2023-09-22T22:56:04.632 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.44900.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.44307.log 2023-09-22T22:56:04.632 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.44900.log.gz 2023-09-22T22:56:04.632 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.85021.log 2023-09-22T22:56:04.632 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.36802.log: 0.0%/var/log/ceph/c1-client.0.44307.log: -- replaced with /var/log/ceph/c1-client.admin.36802.log.gz 2023-09-22T22:56:04.632 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.49836.log 2023-09-22T22:56:04.632 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.44307.log.gz 2023-09-22T22:56:04.633 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.85021.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.87184.log 2023-09-22T22:56:04.633 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.85021.log.gz 2023-09-22T22:56:04.633 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.49836.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.37252.log 2023-09-22T22:56:04.633 INFO:teuthology.orchestra.run.smithi160.stderr: 46.7% -- replaced with /var/log/ceph/c1-client.0.49836.log.gz 2023-09-22T22:56:04.633 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.87184.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.62063.log 2023-09-22T22:56:04.633 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.87184.log.gz 2023-09-22T22:56:04.633 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.37252.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.54815.log 2023-09-22T22:56:04.634 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.37252.log.gz 2023-09-22T22:56:04.634 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.62063.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.72691.log 2023-09-22T22:56:04.634 INFO:teuthology.orchestra.run.smithi160.stderr: 57.4% -- replaced with /var/log/ceph/c1-client.0.62063.log.gz 2023-09-22T22:56:04.634 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.54815.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.59609.log 2023-09-22T22:56:04.634 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.54815.log.gz 2023-09-22T22:56:04.634 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.72691.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36605.log 2023-09-22T22:56:04.634 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.72691.log.gz 2023-09-22T22:56:04.635 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.59609.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.58854.log 2023-09-22T22:56:04.635 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.59609.log.gz 2023-09-22T22:56:04.635 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.58249.log 2023-09-22T22:56:04.635 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.36605.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36605.log.gz 2023-09-22T22:56:04.635 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.58854.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.78790.log 2023-09-22T22:56:04.635 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.58854.log.gz 2023-09-22T22:56:04.635 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.58249.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.89278.log 2023-09-22T22:56:04.635 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.58249.log.gz 2023-09-22T22:56:04.636 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.78790.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.49332.log 2023-09-22T22:56:04.636 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.78790.log.gz 2023-09-22T22:56:04.636 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.89278.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.89278.log.gz -5 2023-09-22T22:56:04.636 INFO:teuthology.orchestra.run.smithi160.stderr: --verbose -- /var/log/ceph/c1-client.0.48641.log 2023-09-22T22:56:04.636 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.61300.log 2023-09-22T22:56:04.636 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.49332.log: 47.1% -- replaced with /var/log/ceph/c1-client.0.49332.log.gz 2023-09-22T22:56:04.636 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.48641.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.86765.log 2023-09-22T22:56:04.637 INFO:teuthology.orchestra.run.smithi160.stderr: 80.7% -- replaced with /var/log/ceph/c1-client.0.48641.log.gz 2023-09-22T22:56:04.637 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.61300.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.91580.log 2023-09-22T22:56:04.637 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.61300.log.gz 2023-09-22T22:56:04.637 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.72166.log 2023-09-22T22:56:04.637 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.86765.log: 70.0% -- replaced with /var/log/ceph/c1-client.0.86765.log.gz 2023-09-22T22:56:04.637 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.91580.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.87101.log 2023-09-22T22:56:04.637 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.91580.log.gz 2023-09-22T22:56:04.638 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.72166.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.55306.log 2023-09-22T22:56:04.638 INFO:teuthology.orchestra.run.smithi160.stderr: 94.6% -- replaced with /var/log/ceph/c1-client.0.72166.log.gz 2023-09-22T22:56:04.638 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.87101.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.42755.log 2023-09-22T22:56:04.638 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.87101.log.gz 2023-09-22T22:56:04.638 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.55306.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.84938.log 2023-09-22T22:56:04.638 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.55306.log.gz 2023-09-22T22:56:04.639 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.42755.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.64012.log 2023-09-22T22:56:04.639 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.42755.log.gz 2023-09-22T22:56:04.639 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.79395.log 2023-09-22T22:56:04.639 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.84938.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.84938.log.gz 2023-09-22T22:56:04.639 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.64012.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.59278.log 2023-09-22T22:56:04.639 INFO:teuthology.orchestra.run.smithi160.stderr: 68.2% -- replaced with /var/log/ceph/c1-client.0.64012.log.gz 2023-09-22T22:56:04.639 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.67732.log 2023-09-22T22:56:04.640 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.79395.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.79395.log.gz/var/log/ceph/c1-client.0.59278.log: 2023-09-22T22:56:04.640 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/c1-client.0.91236.log 2023-09-22T22:56:04.640 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.59278.log.gz 2023-09-22T22:56:04.640 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5/var/log/ceph/c1-client.0.67732.log: --verbose -- /var/log/ceph/c1-client.0.88273.log 2023-09-22T22:56:04.640 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.67732.log.gz 2023-09-22T22:56:04.640 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.91236.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.54984.log 2023-09-22T22:56:04.640 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.91236.log.gz 2023-09-22T22:56:04.641 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.79995.log 2023-09-22T22:56:04.641 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.88273.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.88273.log.gz/var/log/ceph/c1-client.0.54984.log: 2023-09-22T22:56:04.641 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/c1-client.0.85270.log 2023-09-22T22:56:04.641 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.54984.log.gz 2023-09-22T22:56:04.641 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.62145.log 2023-09-22T22:56:04.641 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.79995.log: 44.2% -- replaced with /var/log/ceph/c1-client.0.79995.log.gz 2023-09-22T22:56:04.641 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.85270.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.77466.log 2023-09-22T22:56:04.642 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.85270.log.gz 2023-09-22T22:56:04.642 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.62145.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.39351.log 2023-09-22T22:56:04.642 INFO:teuthology.orchestra.run.smithi160.stderr: 57.5% -- replaced with /var/log/ceph/c1-client.0.62145.log.gz 2023-09-22T22:56:04.642 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.77466.log: gzip -5 --verbose -- /var/log/ceph/rgw.c1.client.0.log 2023-09-22T22:56:04.642 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.77466.log.gz 2023-09-22T22:56:04.642 INFO:teuthology.orchestra.run.smithi160.stderr:gzip/var/log/ceph/c1-client.0.39351.log: -5 --verbose -- /var/log/ceph/c1-client.0.89445.log 2023-09-22T22:56:04.643 INFO:teuthology.orchestra.run.smithi160.stderr: 19.1% -- replaced with /var/log/ceph/c1-client.0.39351.log.gz 2023-09-22T22:56:04.643 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/rgw.c1.client.0.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.85104.log 2023-09-22T22:56:04.643 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.89445.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.88442.log 2023-09-22T22:56:04.643 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.89445.log.gz 2023-09-22T22:56:04.643 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.85104.log: gzip 0.0% -- replaced with /var/log/ceph/c1-client.0.85104.log.gz -5 2023-09-22T22:56:04.643 INFO:teuthology.orchestra.run.smithi160.stderr: --verbose -- /var/log/ceph/c1-client.0.45753.log 2023-09-22T22:56:04.643 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.88442.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.91064.log 2023-09-22T22:56:04.644 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.88442.log.gz 2023-09-22T22:56:04.644 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.44731.log 2023-09-22T22:56:04.644 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.45753.log: /var/log/ceph/c1-client.0.91064.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.82094.log 2023-09-22T22:56:04.644 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% 0.0% -- replaced with /var/log/ceph/c1-client.0.45753.log.gz 2023-09-22T22:56:04.644 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.91064.log.gz 2023-09-22T22:56:04.644 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.44731.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.44731.log.gz 2023-09-22T22:56:04.644 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.65559.log 2023-09-22T22:56:04.645 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36880.log 2023-09-22T22:56:04.645 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.82094.log: /var/log/ceph/c1-client.0.65559.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.60290.log 2023-09-22T22:56:04.645 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% 43.0% -- replaced with /var/log/ceph/c1-client.0.82094.log.gz 2023-09-22T22:56:04.645 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.65559.log.gz 2023-09-22T22:56:04.645 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.36880.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36880.log.gz 2023-09-22T22:56:04.645 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.1.37565.log 2023-09-22T22:56:04.645 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.80908.log 2023-09-22T22:56:04.646 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.60290.log: /var/log/ceph/c1-client.1.37565.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.76098.log 2023-09-22T22:56:04.646 INFO:teuthology.orchestra.run.smithi160.stderr: 70.5% 82.7% -- replaced with /var/log/ceph/c1-client.1.37565.log.gz 2023-09-22T22:56:04.646 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.60290.log.gz/var/log/ceph/c1-client.0.80908.log: 2023-09-22T22:56:04.646 INFO:teuthology.orchestra.run.smithi160.stderr: 91.8% -- replaced with /var/log/ceph/c1-client.0.80908.log.gz 2023-09-22T22:56:04.646 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.92872.log 2023-09-22T22:56:04.646 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.63926.log 2023-09-22T22:56:04.646 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.76098.log: /var/log/ceph/c1-client.0.92872.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.69679.log 2023-09-22T22:56:04.647 INFO:teuthology.orchestra.run.smithi160.stderr: 48.7% -- replaced with /var/log/ceph/c1-client.0.92872.log.gz 0.0% -- replaced with /var/log/ceph/c1-client.0.76098.log.gz 2023-09-22T22:56:04.647 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-09-22T22:56:04.647 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.63926.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.63926.log.gz 2023-09-22T22:56:04.647 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.89528.log 2023-09-22T22:56:04.647 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.46942.log 2023-09-22T22:56:04.647 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.69679.log: /var/log/ceph/c1-client.0.89528.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.86266.log 2023-09-22T22:56:04.647 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.69679.log.gz 2023-09-22T22:56:04.648 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.89528.log.gz 2023-09-22T22:56:04.648 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.46942.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.46942.log.gz 2023-09-22T22:56:04.648 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.81171.log 2023-09-22T22:56:04.648 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.59780.log 2023-09-22T22:56:04.648 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.86266.log: /var/log/ceph/c1-client.0.81171.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.50858.log 2023-09-22T22:56:04.648 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.81171.log.gz 2023-09-22T22:56:04.649 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.59780.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.59780.log.gz 2023-09-22T22:56:04.649 INFO:teuthology.orchestra.run.smithi160.stderr: 97.4% -- replaced with /var/log/ceph/c1-client.0.86266.log.gz 2023-09-22T22:56:04.649 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.58077.log 2023-09-22T22:56:04.649 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.70100.log 2023-09-22T22:56:04.649 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.50858.log: /var/log/ceph/c1-client.0.58077.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.61554.log 2023-09-22T22:56:04.649 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% 0.0% -- replaced with /var/log/ceph/c1-client.0.50858.log.gz 2023-09-22T22:56:04.649 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.58077.log.gz 2023-09-22T22:56:04.649 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.70100.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.45923.log 2023-09-22T22:56:04.650 INFO:teuthology.orchestra.run.smithi160.stderr: 71.1% -- replaced with /var/log/ceph/c1-client.0.70100.log.gz 2023-09-22T22:56:04.650 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.61554.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.61554.log.gz 2023-09-22T22:56:04.650 INFO:teuthology.orchestra.run.smithi160.stderr: 94.8% -- replaced with /var/log/ceph/ops-log-c1-client.1.log.gz 2023-09-22T22:56:04.650 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.62906.log 2023-09-22T22:56:04.650 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.86935.log 2023-09-22T22:56:04.650 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.45923.log: /var/log/ceph/c1-client.0.62906.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.60204.log 2023-09-22T22:56:04.651 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.62906.log.gz 2023-09-22T22:56:04.651 INFO:teuthology.orchestra.run.smithi160.stderr: 48.7% -- replaced with /var/log/ceph/c1-client.0.45923.log.gz 2023-09-22T22:56:04.651 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.86935.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.88098.log 2023-09-22T22:56:04.651 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.86935.log.gz 2023-09-22T22:56:04.651 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.60204.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.77549.log 2023-09-22T22:56:04.651 INFO:teuthology.orchestra.run.smithi160.stderr: 71.0% -- replaced with /var/log/ceph/c1-client.0.60204.log.gz 2023-09-22T22:56:04.651 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.88098.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.53563.log 2023-09-22T22:56:04.652 INFO:teuthology.orchestra.run.smithi160.stderr: 92.5% -- replaced with /var/log/ceph/c1-client.0.88098.log.gz 2023-09-22T22:56:04.652 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.77549.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.46606.log 2023-09-22T22:56:04.652 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.77549.log.gz 2023-09-22T22:56:04.652 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.53563.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.45665.log 2023-09-22T22:56:04.652 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.53563.log.gz 2023-09-22T22:56:04.652 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.46606.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94279.log 2023-09-22T22:56:04.652 INFO:teuthology.orchestra.run.smithi160.stderr: 67.3% -- replaced with /var/log/ceph/c1-client.0.46606.log.gz 2023-09-22T22:56:04.652 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.45665.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.64515.log 2023-09-22T22:56:04.653 INFO:teuthology.orchestra.run.smithi160.stderr: 77.3% -- replaced with /var/log/ceph/c1-client.0.45665.log.gz 2023-09-22T22:56:04.653 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.94279.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.48298.log 2023-09-22T22:56:04.653 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.94279.log.gz 2023-09-22T22:56:04.653 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.64515.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.37374.log 2023-09-22T22:56:04.653 INFO:teuthology.orchestra.run.smithi160.stderr: 44.8% -- replaced with /var/log/ceph/c1-client.0.64515.log.gz 2023-09-22T22:56:04.653 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.48298.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.42669.log 2023-09-22T22:56:04.653 INFO:teuthology.orchestra.run.smithi160.stderr: 76.9% -- replaced with /var/log/ceph/c1-client.0.48298.log.gz 2023-09-22T22:56:04.654 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.37374.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.61465.log 2023-09-22T22:56:04.654 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.37374.log.gz 2023-09-22T22:56:04.654 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.42669.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.81336.log 2023-09-22T22:56:04.654 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.42669.log.gz 2023-09-22T22:56:04.654 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.61465.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.78362.log 2023-09-22T22:56:04.654 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.61465.log.gz 2023-09-22T22:56:04.655 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.81336.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.51023.log 2023-09-22T22:56:04.655 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.81336.log.gz 2023-09-22T22:56:04.655 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.78362.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.79567.log 2023-09-22T22:56:04.655 INFO:teuthology.orchestra.run.smithi160.stderr: 76.9% -- replaced with /var/log/ceph/c1-client.0.78362.log.gz 2023-09-22T22:56:04.655 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.51023.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.64262.log 2023-09-22T22:56:04.655 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.51023.log.gz 2023-09-22T22:56:04.655 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.79567.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.57647.log 2023-09-22T22:56:04.656 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.79567.log.gz 2023-09-22T22:56:04.656 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.64262.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.42411.log 2023-09-22T22:56:04.656 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.64262.log.gz 2023-09-22T22:56:04.656 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.57647.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.76738.log 2023-09-22T22:56:04.656 INFO:teuthology.orchestra.run.smithi160.stderr: 45.5% -- replaced with /var/log/ceph/c1-client.0.57647.log.gz 2023-09-22T22:56:04.656 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.42411.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.53044.log 2023-09-22T22:56:04.656 INFO:teuthology.orchestra.run.smithi160.stderr: 45.5% -- replaced with /var/log/ceph/c1-client.0.42411.log.gz 2023-09-22T22:56:04.657 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.76738.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.51365.log 2023-09-22T22:56:04.657 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.76738.log.gz 2023-09-22T22:56:04.657 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.53044.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.90374.log 2023-09-22T22:56:04.657 INFO:teuthology.orchestra.run.smithi160.stderr: 45.5% -- replaced with /var/log/ceph/c1-client.0.53044.log.gz 2023-09-22T22:56:04.657 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.51365.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.76578.log 2023-09-22T22:56:04.657 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.51365.log.gz 2023-09-22T22:56:04.657 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.90374.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.81845.log 2023-09-22T22:56:04.658 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.90374.log.gz 2023-09-22T22:56:04.658 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.76578.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.43185.log 2023-09-22T22:56:04.658 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.76578.log.gz 2023-09-22T22:56:04.658 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.81845.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.41981.log 2023-09-22T22:56:04.658 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.81845.log.gz 2023-09-22T22:56:04.658 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.43185.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.93675.log 2023-09-22T22:56:04.658 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.43185.log.gz 2023-09-22T22:56:04.659 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.41981.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.41981.log.gz 2023-09-22T22:56:04.659 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.69849.log 2023-09-22T22:56:04.659 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.93675.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.40171.log 2023-09-22T22:56:04.659 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.93675.log.gz 2023-09-22T22:56:04.659 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.69849.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.64769.log 2023-09-22T22:56:04.659 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.69849.log.gz 2023-09-22T22:56:04.660 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.40171.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36875.log 2023-09-22T22:56:04.660 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.40171.log.gz 2023-09-22T22:56:04.660 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.64769.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.73314.log 2023-09-22T22:56:04.660 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.64769.log.gz 2023-09-22T22:56:04.660 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.36875.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.42841.log 2023-09-22T22:56:04.660 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36875.log.gz 2023-09-22T22:56:04.660 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.73314.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.65811.log 2023-09-22T22:56:04.661 INFO:teuthology.orchestra.run.smithi160.stderr: 94.6% -- replaced with /var/log/ceph/c1-client.0.73314.log.gz 2023-09-22T22:56:04.661 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.42841.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.59026.log 2023-09-22T22:56:04.661 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.42841.log.gz 2023-09-22T22:56:04.661 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.65811.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.73145.log 2023-09-22T22:56:04.661 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.65811.log.gz 2023-09-22T22:56:04.661 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.59026.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.79653.log 2023-09-22T22:56:04.661 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.59026.log.gz 2023-09-22T22:56:04.662 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.73145.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.72606.log 2023-09-22T22:56:04.662 INFO:teuthology.orchestra.run.smithi160.stderr: 75.9% -- replaced with /var/log/ceph/c1-client.0.73145.log.gz 2023-09-22T22:56:04.662 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.79653.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.52443.log 2023-09-22T22:56:04.662 INFO:teuthology.orchestra.run.smithi160.stderr: 44.8% -- replaced with /var/log/ceph/c1-client.0.79653.log.gz 2023-09-22T22:56:04.662 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.72606.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.66318.log 2023-09-22T22:56:04.662 INFO:teuthology.orchestra.run.smithi160.stderr: 54.4% -- replaced with /var/log/ceph/c1-client.0.72606.log.gz 2023-09-22T22:56:04.662 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.52443.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.92528.log 2023-09-22T22:56:04.663 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.52443.log.gz 2023-09-22T22:56:04.663 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.66318.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.76658.log 2023-09-22T22:56:04.663 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.66318.log.gz 2023-09-22T22:56:04.663 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.92528.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.76898.log 2023-09-22T22:56:04.663 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.92528.log.gz 2023-09-22T22:56:04.663 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.76658.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.53649.log 2023-09-22T22:56:04.663 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.76658.log.gz 2023-09-22T22:56:04.664 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.76898.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.70603.log 2023-09-22T22:56:04.664 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.76898.log.gz 2023-09-22T22:56:04.664 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.53649.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.64937.log 2023-09-22T22:56:04.664 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.53649.log.gz 2023-09-22T22:56:04.664 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.70603.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94019.log 2023-09-22T22:56:04.664 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.70603.log.gz 2023-09-22T22:56:04.664 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.64937.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.42067.log 2023-09-22T22:56:04.665 INFO:teuthology.orchestra.run.smithi160.stderr: 44.8% -- replaced with /var/log/ceph/c1-client.0.64937.log.gz 2023-09-22T22:56:04.665 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.94019.log: gzip -5 --verbose -- /var/log/ceph/c1-osd.0.log 2023-09-22T22:56:04.665 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.94019.log.gz 2023-09-22T22:56:04.665 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.42067.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.72084.log 2023-09-22T22:56:04.665 INFO:teuthology.orchestra.run.smithi160.stderr: 44.8% -- replaced with /var/log/ceph/c1-client.0.42067.log.gz 2023-09-22T22:56:04.665 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.87350.log 2023-09-22T22:56:04.665 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.72084.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.50773.log 2023-09-22T22:56:04.665 INFO:teuthology.orchestra.run.smithi160.stderr: 82.8% -- replaced with /var/log/ceph/c1-client.0.72084.log.gz 2023-09-22T22:56:04.666 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.87350.log: gzip -5 --verbose -- /var/log/ceph/c1.log 2023-09-22T22:56:04.666 INFO:teuthology.orchestra.run.smithi160.stderr: 41.4% -- replaced with /var/log/ceph/c1-client.0.87350.log.gz 2023-09-22T22:56:04.666 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.48129.log 2023-09-22T22:56:04.666 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.50773.log: /var/log/ceph/c1.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.41557.log 2023-09-22T22:56:04.666 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.50773.log.gz 2023-09-22T22:56:04.666 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.48129.log: 0.0%gzip -5 --verbose -- /var/log/ceph/c1-client.0.92010.log 2023-09-22T22:56:04.667 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.48129.log.gz 2023-09-22T22:56:04.667 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.41557.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.41557.log.gz 2023-09-22T22:56:04.667 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.84769.log 2023-09-22T22:56:04.667 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.73877.log 2023-09-22T22:56:04.667 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.92010.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.92010.log.gz 2023-09-22T22:56:04.667 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.84769.log: 44.8% -- replaced with /var/log/ceph/c1-client.0.84769.log.gz 2023-09-22T22:56:04.667 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.91150.log 2023-09-22T22:56:04.668 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.73877.log: 94.2% -- replaced with /var/log/ceph/c1-client.0.73877.log.gz 2023-09-22T22:56:04.668 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.70935.log 2023-09-22T22:56:04.668 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.66580.log 2023-09-22T22:56:04.668 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.91150.log: /var/log/ceph/c1-client.0.70935.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.91150.log.gz 2023-09-22T22:56:04.668 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.70935.log.gz 2023-09-22T22:56:04.668 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.47452.log 2023-09-22T22:56:04.668 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.46436.log 2023-09-22T22:56:04.669 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.66580.log: /var/log/ceph/c1-client.0.47452.log: 59.1% -- replaced with /var/log/ceph/c1-client.0.66580.log.gz 2023-09-22T22:56:04.669 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.47452.log.gz 2023-09-22T22:56:04.669 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.89777.log 2023-09-22T22:56:04.669 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.70268.log 2023-09-22T22:56:04.669 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.89777.log: /var/log/ceph/c1-client.0.46436.log: 53.3% -- replaced with /var/log/ceph/c1-client.0.46436.log.gz 2023-09-22T22:56:04.669 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.89777.log.gz 2023-09-22T22:56:04.669 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.42927.log 2023-09-22T22:56:04.670 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.72433.log 2023-09-22T22:56:04.670 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.70268.log: /var/log/ceph/c1-client.0.42927.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.70268.log.gz 2023-09-22T22:56:04.670 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.42927.log.gz 2023-09-22T22:56:04.670 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.42497.log 2023-09-22T22:56:04.670 INFO:teuthology.orchestra.run.smithi160.stderr: 88.8% -- replaced with /var/log/ceph/c1.log.gz 2023-09-22T22:56:04.670 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.60375.log 2023-09-22T22:56:04.670 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.72433.log: /var/log/ceph/c1-client.0.42497.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.52786.log 2023-09-22T22:56:04.671 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.42497.log.gz 2023-09-22T22:56:04.671 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.72433.log.gz 2023-09-22T22:56:04.671 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.60375.log: 57.9% -- replaced with /var/log/ceph/c1-client.0.60375.log.gz 2023-09-22T22:56:04.671 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.91838.log 2023-09-22T22:56:04.671 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.46774.log 2023-09-22T22:56:04.671 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.52786.log: /var/log/ceph/c1-client.0.91838.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.64683.log 2023-09-22T22:56:04.671 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.52786.log.gz 2023-09-22T22:56:04.672 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.91838.log.gz 2023-09-22T22:56:04.672 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.46774.log: 53.0% -- replaced with /var/log/ceph/c1-client.0.46774.log.gz 2023-09-22T22:56:04.672 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.64094.log 2023-09-22T22:56:04.672 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.47537.log 2023-09-22T22:56:04.672 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.64683.log: /var/log/ceph/c1-client.0.64094.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.82833.log 2023-09-22T22:56:04.672 INFO:teuthology.orchestra.run.smithi160.stderr: 68.5% -- replaced with /var/log/ceph/c1-client.0.64094.log.gz 2023-09-22T22:56:04.672 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.47537.log: 57.8% -- replaced with /var/log/ceph/c1-client.0.64683.log.gz 2023-09-22T22:56:04.672 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.47537.log.gz 2023-09-22T22:56:04.673 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.65983.log 2023-09-22T22:56:04.673 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.53475.log 2023-09-22T22:56:04.673 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.82833.log: /var/log/ceph/c1-client.0.65983.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.53389.log 2023-09-22T22:56:04.673 INFO:teuthology.orchestra.run.smithi160.stderr: 57.0% -- replaced with /var/log/ceph/c1-client.0.65983.log.gz 2023-09-22T22:56:04.673 INFO:teuthology.orchestra.run.smithi160.stderr: 89.9%/var/log/ceph/c1-client.0.53475.log: -- replaced with /var/log/ceph/c1-client.0.82833.log.gz 2023-09-22T22:56:04.673 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.53475.log.gz 2023-09-22T22:56:04.673 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.59441.log 2023-09-22T22:56:04.674 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.48383.log 2023-09-22T22:56:04.674 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.53389.log: /var/log/ceph/c1-client.0.59441.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.73043.log 2023-09-22T22:56:04.674 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% 44.8% -- replaced with /var/log/ceph/c1-client.0.53389.log.gz 2023-09-22T22:56:04.674 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.59441.log.gz 2023-09-22T22:56:04.674 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.48383.log: 81.2% -- replaced with /var/log/ceph/c1-client.0.48383.log.gz 2023-09-22T22:56:04.674 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.74578.log 2023-09-22T22:56:04.674 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.92958.log 2023-09-22T22:56:04.675 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.73043.log: /var/log/ceph/c1-client.0.74578.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.60874.log 2023-09-22T22:56:04.675 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.74578.log.gz 92.5% -- replaced with /var/log/ceph/c1-client.0.73043.log.gz 2023-09-22T22:56:04.675 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-09-22T22:56:04.675 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.92958.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.92958.log.gz 2023-09-22T22:56:04.675 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.40375.log 2023-09-22T22:56:04.675 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.62230.log 2023-09-22T22:56:04.675 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.60874.log: /var/log/ceph/c1-client.0.40375.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.53980.log 2023-09-22T22:56:04.676 INFO:teuthology.orchestra.run.smithi160.stderr: 19.7% 88.6% -- replaced with /var/log/ceph/c1-client.0.60874.log.gz 2023-09-22T22:56:04.676 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.40375.log.gz/var/log/ceph/c1-client.0.62230.log: 2023-09-22T22:56:04.676 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.62230.log.gz 2023-09-22T22:56:04.676 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.53130.log 2023-09-22T22:56:04.676 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.75618.log 2023-09-22T22:56:04.676 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.53980.log: /var/log/ceph/c1-client.0.53130.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.45326.log 2023-09-22T22:56:04.676 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.53980.log.gz 2023-09-22T22:56:04.676 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.53130.log.gz 2023-09-22T22:56:04.677 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.75618.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.75618.log.gz 2023-09-22T22:56:04.677 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.67361.log 2023-09-22T22:56:04.677 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.67098.log 2023-09-22T22:56:04.677 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.45326.log: /var/log/ceph/c1-client.0.67361.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.85527.log 2023-09-22T22:56:04.677 INFO:teuthology.orchestra.run.smithi160.stderr: 44.8% -- replaced with /var/log/ceph/c1-client.0.45326.log.gz 2023-09-22T22:56:04.677 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.67361.log.gz 2023-09-22T22:56:04.677 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.67098.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.67098.log.gz 2023-09-22T22:56:04.678 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.44393.log 2023-09-22T22:56:04.678 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.85527.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.49754.log 2023-09-22T22:56:04.678 INFO:teuthology.orchestra.run.smithi160.stderr: 41.4% -- replaced with /var/log/ceph/c1-client.0.85527.log.gz 2023-09-22T22:56:04.678 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.44393.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.51868.log 2023-09-22T22:56:04.678 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.44393.log.gz 2023-09-22T22:56:04.678 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.44562.log 2023-09-22T22:56:04.678 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.49754.log: /var/log/ceph/c1-client.0.51868.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.81679.log 2023-09-22T22:56:04.679 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% 67.8% -- replaced with /var/log/ceph/c1-client.0.51868.log.gz -- replaced with /var/log/ceph/c1-client.0.49754.log.gz 2023-09-22T22:56:04.679 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-09-22T22:56:04.679 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.44562.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.44562.log.gz 2023-09-22T22:56:04.679 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.62396.log 2023-09-22T22:56:04.679 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.63240.log 2023-09-22T22:56:04.679 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.81679.log: /var/log/ceph/c1-client.0.62396.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36804.log 2023-09-22T22:56:04.679 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.81679.log.gz 2023-09-22T22:56:04.679 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.62396.log.gz 2023-09-22T22:56:04.680 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.63240.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.75298.log 2023-09-22T22:56:04.680 INFO:teuthology.orchestra.run.smithi160.stderr: 70.1% -- replaced with /var/log/ceph/c1-client.0.63240.log.gz 2023-09-22T22:56:04.680 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.36804.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36804.log.gz 2023-09-22T22:56:04.680 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.75858.log 2023-09-22T22:56:04.680 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.65444.log 2023-09-22T22:56:04.680 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.75298.log: /var/log/ceph/c1-client.0.75858.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.52958.log 2023-09-22T22:56:04.681 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.75298.log.gz 2023-09-22T22:56:04.681 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.75858.log.gz 2023-09-22T22:56:04.681 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.65444.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.65444.log.gz 2023-09-22T22:56:04.681 INFO:teuthology.orchestra.run.smithi160.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.57302.log 2023-09-22T22:56:04.681 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.89023.log 2023-09-22T22:56:04.681 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.52958.log: /var/log/ceph/c1-client.0.57302.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.45069.log 2023-09-22T22:56:04.681 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.52958.log.gz 2023-09-22T22:56:04.682 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.57302.log.gz/var/log/ceph/c1-client.0.89023.log: 2023-09-22T22:56:04.682 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.89023.log.gz 2023-09-22T22:56:04.682 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.56786.log 2023-09-22T22:56:04.682 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.84683.log 2023-09-22T22:56:04.682 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.45069.log: /var/log/ceph/c1-client.0.56786.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.68153.log 2023-09-22T22:56:04.682 INFO:teuthology.orchestra.run.smithi160.stderr: 57.9% -- replaced with /var/log/ceph/c1-client.0.45069.log.gz 2023-09-22T22:56:04.682 INFO:teuthology.orchestra.run.smithi160.stderr: 44.8% -- replaced with /var/log/ceph/c1-client.0.56786.log.gz/var/log/ceph/c1-client.0.84683.log: 2023-09-22T22:56:04.683 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.84683.log.gz 2023-09-22T22:56:04.683 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.47788.log 2023-09-22T22:56:04.683 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.51614.log 2023-09-22T22:56:04.683 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.68153.log: /var/log/ceph/c1-client.0.47788.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.79739.log 2023-09-22T22:56:04.683 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.68153.log.gz 2023-09-22T22:56:04.683 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.47788.log.gz 2023-09-22T22:56:04.684 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.51614.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.51614.log.gz 2023-09-22T22:56:04.684 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.53726.log 2023-09-22T22:56:04.684 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.91924.log 2023-09-22T22:56:04.684 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.79739.log: /var/log/ceph/c1-client.0.53726.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.60791.log 2023-09-22T22:56:04.684 INFO:teuthology.orchestra.run.smithi160.stderr: 48.7% -- replaced with /var/log/ceph/c1-client.0.79739.log.gz 2023-09-22T22:56:04.684 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.53726.log.gz 2023-09-22T22:56:04.684 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.91924.log: 47.4%gzip -- replaced with /var/log/ceph/c1-client.0.91924.log.gz 2023-09-22T22:56:04.685 INFO:teuthology.orchestra.run.smithi160.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.57044.log 2023-09-22T22:56:04.685 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.71102.log 2023-09-22T22:56:04.685 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.60791.log: /var/log/ceph/c1-client.0.57044.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.40275.log 2023-09-22T22:56:04.685 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.60791.log.gz 2023-09-22T22:56:04.685 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.57044.log.gz/var/log/ceph/c1-client.0.71102.log: 2023-09-22T22:56:04.685 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.71102.log.gz 2023-09-22T22:56:04.685 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.42583.log 2023-09-22T22:56:04.686 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.74338.log 2023-09-22T22:56:04.686 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.40275.log: /var/log/ceph/c1-client.0.42583.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.48820.log 2023-09-22T22:56:04.686 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.40275.log.gz 2023-09-22T22:56:04.686 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.42583.log.gz/var/log/ceph/c1-client.0.74338.log: 2023-09-22T22:56:04.686 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.74338.log.gz 2023-09-22T22:56:04.686 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.71270.log 2023-09-22T22:56:04.687 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.33442.log 2023-09-22T22:56:04.687 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.48820.log: /var/log/ceph/c1-client.0.71270.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.42153.log 2023-09-22T22:56:04.687 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.48820.log.gz 2023-09-22T22:56:04.687 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.71270.log.gz 2023-09-22T22:56:04.687 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.33442.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.93761.log 0.0% 2023-09-22T22:56:04.687 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.admin.33442.log.gz 2023-09-22T22:56:04.687 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36630.log 2023-09-22T22:56:04.688 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.42153.log: /var/log/ceph/c1-client.0.93761.log: 47.4%gzip -5 --verbose -- /var/log/ceph/c1-client.admin.33345.log 2023-09-22T22:56:04.688 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.93761.log.gz 2023-09-22T22:56:04.688 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.42153.log.gz 2023-09-22T22:56:04.688 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.36630.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36630.log.gz 2023-09-22T22:56:04.688 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.81762.log 2023-09-22T22:56:04.688 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.88359.log 2023-09-22T22:56:04.688 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.33345.log: /var/log/ceph/c1-client.0.81762.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.46859.log 2023-09-22T22:56:04.689 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.81762.log.gz 2023-09-22T22:56:04.689 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.33345.log.gz 2023-09-22T22:56:04.689 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.88359.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.88359.log.gz 2023-09-22T22:56:04.689 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.1.37648.log 2023-09-22T22:56:04.689 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.65361.log 2023-09-22T22:56:04.689 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.46859.log: /var/log/ceph/c1-client.1.37648.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.90460.log 2023-09-22T22:56:04.690 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.46859.log.gz 2023-09-22T22:56:04.690 INFO:teuthology.orchestra.run.smithi160.stderr: 43.7% -- replaced with /var/log/ceph/c1-client.1.37648.log.gz/var/log/ceph/c1-client.0.65361.log: 2023-09-22T22:56:04.690 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.65361.log.gz 2023-09-22T22:56:04.690 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.94193.log 2023-09-22T22:56:04.690 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.44476.log 2023-09-22T22:56:04.690 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.90460.log: /var/log/ceph/c1-client.0.94193.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.48994.log 2023-09-22T22:56:04.690 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.90460.log.gz 2023-09-22T22:56:04.691 INFO:teuthology.orchestra.run.smithi160.stderr: 45.5% -- replaced with /var/log/ceph/c1-client.0.94193.log.gz/var/log/ceph/c1-client.0.44476.log: 2023-09-22T22:56:04.691 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.44476.log.gz 2023-09-22T22:56:04.691 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.52872.log 2023-09-22T22:56:04.691 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.67460.log 2023-09-22T22:56:04.691 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.48994.log: /var/log/ceph/c1-client.0.52872.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.60961.log 2023-09-22T22:56:04.691 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.52872.log.gz 45.5% -- replaced with /var/log/ceph/c1-client.0.48994.log.gz 2023-09-22T22:56:04.692 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-09-22T22:56:04.692 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.67460.log: 92.8% -- replaced with /var/log/ceph/c1-client.0.67460.log.gz 2023-09-22T22:56:04.692 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.58163.log 2023-09-22T22:56:04.692 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.37488.log 2023-09-22T22:56:04.692 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.60961.log: /var/log/ceph/c1-client.0.58163.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.47705.log 2023-09-22T22:56:04.692 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% 57.8% -- replaced with /var/log/ceph/c1-client.0.60961.log.gz 2023-09-22T22:56:04.692 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.58163.log.gz 2023-09-22T22:56:04.692 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.37488.log: 43.0% -- replaced with /var/log/ceph/c1-client.0.37488.log.gz 2023-09-22T22:56:04.693 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.62313.log 2023-09-22T22:56:04.693 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.60457.log 2023-09-22T22:56:04.693 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.47705.log: /var/log/ceph/c1-client.0.62313.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.89860.log 2023-09-22T22:56:04.693 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.47705.log.gz 2023-09-22T22:56:04.693 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.62313.log.gz 2023-09-22T22:56:04.693 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.60457.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.62823.log 2023-09-22T22:56:04.693 INFO:teuthology.orchestra.run.smithi160.stderr: 58.0% -- replaced with /var/log/ceph/c1-client.0.60457.log.gz 2023-09-22T22:56:04.694 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.37035.log 2023-09-22T22:56:04.694 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.89860.log: /var/log/ceph/c1-client.0.62823.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.70686.log 2023-09-22T22:56:04.694 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.89860.log.gz 2023-09-22T22:56:04.694 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.62823.log.gz 2023-09-22T22:56:04.694 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.37035.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.37035.log.gzgzip 2023-09-22T22:56:04.694 INFO:teuthology.orchestra.run.smithi160.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.74658.log 2023-09-22T22:56:04.694 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.56700.log 2023-09-22T22:56:04.695 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.70686.log: /var/log/ceph/c1-client.0.74658.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.50603.log 2023-09-22T22:56:04.695 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.70686.log.gz 2023-09-22T22:56:04.695 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.74658.log.gz/var/log/ceph/c1-client.0.56700.log: 2023-09-22T22:56:04.695 INFO:teuthology.orchestra.run.smithi160.stderr: 45.5% -- replaced with /var/log/ceph/c1-client.0.56700.log.gz 2023-09-22T22:56:04.695 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.59361.log 2023-09-22T22:56:04.695 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.92786.log 2023-09-22T22:56:04.695 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.50603.log: /var/log/ceph/c1-client.0.59361.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.61637.log 2023-09-22T22:56:04.696 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.50603.log.gz 2023-09-22T22:56:04.696 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.59361.log.gz 2023-09-22T22:56:04.696 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.92786.log: gzip 44.8% -5 -- replaced with /var/log/ceph/c1-client.0.92786.log.gz --verbose 2023-09-22T22:56:04.696 INFO:teuthology.orchestra.run.smithi160.stderr: -- /var/log/ceph/c1-client.admin.37031.log 2023-09-22T22:56:04.696 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.89362.log 2023-09-22T22:56:04.696 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.61637.log: /var/log/ceph/c1-client.admin.37031.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.83095.log 2023-09-22T22:56:04.696 INFO:teuthology.orchestra.run.smithi160.stderr: 45.5% -- replaced with /var/log/ceph/c1-client.0.61637.log.gz 2023-09-22T22:56:04.697 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.37031.log.gz 2023-09-22T22:56:04.697 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.89362.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.89362.log.gz 2023-09-22T22:56:04.697 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.80082.log 2023-09-22T22:56:04.697 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.82011.log 2023-09-22T22:56:04.697 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.83095.log: /var/log/ceph/c1-client.0.80082.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.78532.log 2023-09-22T22:56:04.697 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.83095.log.gz 2023-09-22T22:56:04.697 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.80082.log.gz/var/log/ceph/c1-client.0.82011.log: 2023-09-22T22:56:04.698 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.82011.log.gz 2023-09-22T22:56:04.698 INFO:teuthology.orchestra.run.smithi160.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.70769.log 2023-09-22T22:56:04.698 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.72519.log 2023-09-22T22:56:04.698 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.78532.log: /var/log/ceph/c1-client.0.70769.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.42239.log 2023-09-22T22:56:04.698 INFO:teuthology.orchestra.run.smithi160.stderr: 45.5% -- replaced with /var/log/ceph/c1-client.0.78532.log.gz 2023-09-22T22:56:04.698 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.70769.log.gz 2023-09-22T22:56:04.698 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.72519.log: gzip -5 65.9% --verbose -- -- replaced with /var/log/ceph/c1-client.0.72519.log.gz 2023-09-22T22:56:04.699 INFO:teuthology.orchestra.run.smithi160.stderr: /var/log/ceph/c1-client.0.52117.log 2023-09-22T22:56:04.699 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.95298.log 2023-09-22T22:56:04.699 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.42239.log: /var/log/ceph/c1-client.0.52117.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94537.log 2023-09-22T22:56:04.699 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.52117.log.gz 47.4% -- replaced with /var/log/ceph/c1-client.0.42239.log.gz 2023-09-22T22:56:04.699 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-09-22T22:56:04.699 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.95298.log: 6.1% -- replaced with /var/log/ceph/c1-client.0.95298.log.gz 2023-09-22T22:56:04.699 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.71353.log 2023-09-22T22:56:04.700 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.67647.log 2023-09-22T22:56:04.700 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.94537.log: /var/log/ceph/c1-client.0.71353.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.72775.log 2023-09-22T22:56:04.700 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% 47.4% -- replaced with /var/log/ceph/c1-client.0.94537.log.gz 2023-09-22T22:56:04.700 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.71353.log.gz 2023-09-22T22:56:04.700 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.67647.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.67647.log.gz 2023-09-22T22:56:04.700 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.95053.log 2023-09-22T22:56:04.700 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.68431.log 2023-09-22T22:56:04.701 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.72775.log: /var/log/ceph/c1-client.0.95053.log: gzip -5 --verbose -- /var/log/ceph/c1-osd.1.log 2023-09-22T22:56:04.701 INFO:teuthology.orchestra.run.smithi160.stderr: 92.8% -- replaced with /var/log/ceph/c1-client.0.72775.log.gz 2023-09-22T22:56:04.701 INFO:teuthology.orchestra.run.smithi160.stderr: 45.5% -- replaced with /var/log/ceph/c1-client.0.95053.log.gz/var/log/ceph/c1-client.0.68431.log: 2023-09-22T22:56:04.701 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.68431.log.gz 2023-09-22T22:56:04.701 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.43271.log 2023-09-22T22:56:04.701 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.64601.log 2023-09-22T22:56:04.701 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-osd.1.log: /var/log/ceph/c1-client.0.43271.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.55383.log 2023-09-22T22:56:04.702 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.43271.log.gz 2023-09-22T22:56:04.702 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.64601.log: 56.8% -- replaced with /var/log/ceph/c1-client.0.64601.log.gz 2023-09-22T22:56:04.769 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.91752.log 2023-09-22T22:56:04.769 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.55383.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.66401.log 2023-09-22T22:56:04.770 INFO:teuthology.orchestra.run.smithi160.stderr: 14.6% -- replaced with /var/log/ceph/c1-client.0.55383.log.gz 2023-09-22T22:56:04.770 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.91752.log: 45.5%gzip -5 --verbose -- /var/log/ceph/c1-client.0.49500.log 2023-09-22T22:56:04.770 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.91752.log.gz 2023-09-22T22:56:04.770 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.66401.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.66401.log.gz 2023-09-22T22:56:04.770 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.55602.log 2023-09-22T22:56:04.770 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.49500.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.86852.log 2023-09-22T22:56:04.770 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4%/var/log/ceph/c1-client.0.55602.log: -- replaced with /var/log/ceph/c1-client.0.49500.log.gz 2023-09-22T22:56:04.771 INFO:teuthology.orchestra.run.smithi160.stderr: 6.3% -- replaced with /var/log/ceph/c1-client.0.55602.log.gz 2023-09-22T22:56:04.771 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.67559.log 2023-09-22T22:56:04.771 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.65278.log 2023-09-22T22:56:04.771 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.86852.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.86852.log.gz 2023-09-22T22:56:04.771 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.67559.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.62565.log 2023-09-22T22:56:04.772 INFO:teuthology.orchestra.run.smithi160.stderr: 86.3% -- replaced with /var/log/ceph/c1-client.0.67559.log.gz 2023-09-22T22:56:04.772 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.65278.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.44983.log 2023-09-22T22:56:04.772 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.65278.log.gz 2023-09-22T22:56:04.772 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.62565.log: 44.8% -- replaced with /var/log/ceph/c1-client.0.62565.log.gzgzip -5 --verbose -- /var/log/ceph/c1-client.0.64431.log 2023-09-22T22:56:04.772 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-09-22T22:56:04.772 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.44983.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.44983.log.gz 2023-09-22T22:56:04.772 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.78280.log 2023-09-22T22:56:04.772 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.88525.log 2023-09-22T22:56:04.773 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.64431.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.64431.log.gz 2023-09-22T22:56:04.773 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.78280.log: 66.6%gzip -5 --verbose -- /var/log/ceph/c1-client.0.83433.log 2023-09-22T22:56:04.773 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.78280.log.gz 2023-09-22T22:56:04.773 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.88525.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.88525.log.gz 2023-09-22T22:56:04.773 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.79826.log 2023-09-22T22:56:04.773 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.39225.log 2023-09-22T22:56:04.773 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.83433.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.83433.log.gz 2023-09-22T22:56:04.774 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.79826.log: 47.4%gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36805.log 2023-09-22T22:56:04.774 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.79826.log.gz 2023-09-22T22:56:04.774 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.39225.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.39225.log.gz 2023-09-22T22:56:04.774 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.86507.log 2023-09-22T22:56:04.774 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.60542.log 2023-09-22T22:56:04.774 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.36805.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36805.log.gz 2023-09-22T22:56:04.774 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.86507.log: 47.4%gzip -5 --verbose -- /var/log/ceph/c1-client.0.74898.log 2023-09-22T22:56:04.775 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.86507.log.gz 2023-09-22T22:56:04.775 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.60542.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.60542.log.gz 2023-09-22T22:56:04.775 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.81424.log 2023-09-22T22:56:04.775 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.43099.log 2023-09-22T22:56:04.775 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.74898.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.74898.log.gz 2023-09-22T22:56:04.775 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.81424.log: 0.0%gzip -5 --verbose -- /var/log/ceph/c1-client.admin.37135.log 2023-09-22T22:56:04.775 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.81424.log.gz 2023-09-22T22:56:04.776 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.43099.log: 44.8% -- replaced with /var/log/ceph/c1-client.0.43099.log.gz 2023-09-22T22:56:04.776 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.57216.log 2023-09-22T22:56:04.776 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.77380.log 2023-09-22T22:56:04.776 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.37135.log: /var/log/ceph/c1-client.0.57216.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.37135.log.gz 2023-09-22T22:56:04.776 INFO:teuthology.orchestra.run.smithi160.stderr: 45.5% -- replaced with /var/log/ceph/c1-client.0.57216.log.gz 2023-09-22T22:56:04.776 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.52194.log 2023-09-22T22:56:04.776 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.40088.log 2023-09-22T22:56:04.777 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.77380.log: /var/log/ceph/c1-client.0.52194.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.52194.log.gz 2023-09-22T22:56:04.777 INFO:teuthology.orchestra.run.smithi160.stderr: 97.0% -- replaced with /var/log/ceph/c1-client.0.77380.log.gz 2023-09-22T22:56:04.777 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.40249.log 2023-09-22T22:56:04.777 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.61978.log 2023-09-22T22:56:04.777 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.40088.log: /var/log/ceph/c1-client.admin.40249.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.40088.log.gz 2023-09-22T22:56:04.777 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.40249.log.gz 2023-09-22T22:56:04.777 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.90026.log 2023-09-22T22:56:04.777 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.94451.log 2023-09-22T22:56:04.778 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.61978.log: 58.0% -- replaced with /var/log/ceph/c1-client.0.61978.log.gz 2023-09-22T22:56:04.778 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.90026.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.90026.log.gzgzip -5 --verbose -- /var/log/ceph/c1-client.0.44224.log 2023-09-22T22:56:04.778 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-09-22T22:56:04.778 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.94451.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.94451.log.gz 2023-09-22T22:56:04.778 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.64348.log 2023-09-22T22:56:04.778 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.39251.log 2023-09-22T22:56:04.779 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.44224.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.44224.log.gz 2023-09-22T22:56:04.779 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.64348.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.56170.log 2023-09-22T22:56:04.779 INFO:teuthology.orchestra.run.smithi160.stderr: 64.6% -- replaced with /var/log/ceph/c1-client.0.64348.log.gz 2023-09-22T22:56:04.779 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.39251.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.39251.log.gz 2023-09-22T22:56:04.779 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.53216.log 2023-09-22T22:56:04.779 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.60118.log 2023-09-22T22:56:04.779 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.56170.log: /var/log/ceph/c1-client.0.53216.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.56170.log.gz 2023-09-22T22:56:04.781 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.53216.log.gz 2023-09-22T22:56:04.781 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.61132.log 2023-09-22T22:56:04.781 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.60118.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.60118.log.gz 2023-09-22T22:56:04.781 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36685.log 2023-09-22T22:56:04.781 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.86680.log 2023-09-22T22:56:04.782 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.61132.log: /var/log/ceph/c1-client.admin.36685.log: 58.3% -- replaced with /var/log/ceph/c1-client.0.61132.log.gz 2023-09-22T22:56:04.782 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36685.log.gz 2023-09-22T22:56:04.782 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.41809.log 2023-09-22T22:56:04.782 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.68236.log 2023-09-22T22:56:04.782 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.86680.log: /var/log/ceph/c1-client.0.41809.log: 55.1% -- replaced with /var/log/ceph/c1-client.0.86680.log.gz 2023-09-22T22:56:04.782 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.41809.log.gz 2023-09-22T22:56:04.782 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.90890.log 2023-09-22T22:56:04.783 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.46091.log 2023-09-22T22:56:04.783 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.68236.log: 92.6% -- replaced with /var/log/ceph/c1-client.0.68236.log.gz 2023-09-22T22:56:04.783 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.90890.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.90890.log.gz 2023-09-22T22:56:04.783 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.48043.log 2023-09-22T22:56:04.783 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.46091.log: 68.6% -- replaced with /var/log/ceph/c1-client.0.46091.log.gz 2023-09-22T22:56:04.783 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ops-log-c1-client.0.log 2023-09-22T22:56:04.783 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.92184.log 2023-09-22T22:56:04.783 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.48043.log: 51.0% -- replaced with /var/log/ceph/c1-client.0.48043.log.gz 2023-09-22T22:56:04.784 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ops-log-c1-client.0.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.49417.log 2023-09-22T22:56:04.784 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.92184.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.92184.log.gz 2023-09-22T22:56:04.784 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.58681.log 2023-09-22T22:56:04.784 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.49417.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.49417.log.gz 2023-09-22T22:56:04.784 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.83348.log 2023-09-22T22:56:04.784 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.58681.log: 45.5% -- replaced with /var/log/ceph/c1-client.0.58681.log.gz 2023-09-22T22:56:04.784 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.37039.log 2023-09-22T22:56:04.785 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.83348.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.83348.log.gz 2023-09-22T22:56:04.785 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.90804.log 2023-09-22T22:56:04.785 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.37039.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.37039.log.gz 2023-09-22T22:56:04.785 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.94709.log 2023-09-22T22:56:04.785 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.90804.log: 45.5% -- replaced with /var/log/ceph/c1-client.0.90804.log.gz 2023-09-22T22:56:04.785 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.92442.log 2023-09-22T22:56:04.785 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.94709.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.94709.log.gz 2023-09-22T22:56:04.786 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.47617.log 2023-09-22T22:56:04.786 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.92442.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.92442.log.gz 2023-09-22T22:56:04.786 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.44645.log 2023-09-22T22:56:04.786 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.47617.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.47617.log.gz 2023-09-22T22:56:04.786 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.41723.log 2023-09-22T22:56:04.786 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.44645.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.44645.log.gz 2023-09-22T22:56:04.786 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.51785.log 2023-09-22T22:56:04.787 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.41723.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.41723.log.gz 2023-09-22T22:56:04.787 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.76018.log 2023-09-22T22:56:04.787 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.51785.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.51785.log.gz 2023-09-22T22:56:04.787 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.58940.log 2023-09-22T22:56:04.787 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.76018.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.76018.log.gz 2023-09-22T22:56:04.787 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.56872.log 2023-09-22T22:56:04.787 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.58940.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.58940.log.gz 2023-09-22T22:56:04.788 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.45243.log 2023-09-22T22:56:04.788 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.56872.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.56872.log.gz 2023-09-22T22:56:04.788 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.54729.log 2023-09-22T22:56:04.788 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.45243.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.45243.log.gz 2023-09-22T22:56:04.788 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.65023.log 2023-09-22T22:56:04.788 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.54729.log: 44.8% -- replaced with /var/log/ceph/c1-client.0.54729.log.gz 2023-09-22T22:56:04.788 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.57733.log 2023-09-22T22:56:04.788 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.65023.log: 67.2% -- replaced with /var/log/ceph/c1-client.0.65023.log.gz 2023-09-22T22:56:04.789 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.46354.log 2023-09-22T22:56:04.789 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.57733.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.57733.log.gz 2023-09-22T22:56:04.789 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.90287.log 2023-09-22T22:56:04.789 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.46354.log: 53.6% -- replaced with /var/log/ceph/c1-client.0.46354.log.gz 2023-09-22T22:56:04.789 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.75538.log 2023-09-22T22:56:04.789 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.90287.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.90287.log.gz 2023-09-22T22:56:04.789 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.95229.log 2023-09-22T22:56:04.790 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.75538.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.75538.log.gz 2023-09-22T22:56:04.790 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.53303.log 2023-09-22T22:56:04.790 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.95229.log: 8.2% -- replaced with /var/log/ceph/c1-client.0.95229.log.gz 2023-09-22T22:56:04.790 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.65106.log 2023-09-22T22:56:04.790 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.53303.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.53303.log.gz 2023-09-22T22:56:04.790 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.39147.log 2023-09-22T22:56:04.790 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.65106.log: 57.0% -- replaced with /var/log/ceph/c1-client.0.65106.log.gz 2023-09-22T22:56:04.791 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36953.log 2023-09-22T22:56:04.791 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.39147.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.39147.log.gz 2023-09-22T22:56:04.791 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.58593.log 2023-09-22T22:56:04.791 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.36953.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36953.log.gz 2023-09-22T22:56:04.791 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.40298.log 2023-09-22T22:56:04.791 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.58593.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.58593.log.gz 2023-09-22T22:56:04.791 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.51531.log 2023-09-22T22:56:04.792 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.40298.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.40298.log.gz 2023-09-22T22:56:04.792 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.92700.log 2023-09-22T22:56:04.792 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.51531.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.51531.log.gz 2023-09-22T22:56:04.792 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.33297.log 2023-09-22T22:56:04.792 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.92700.log: 45.5% -- replaced with /var/log/ceph/c1-client.0.92700.log.gz 2023-09-22T22:56:04.792 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.50520.log 2023-09-22T22:56:04.792 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.33297.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.33297.log.gz 2023-09-22T22:56:04.793 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.59865.log 2023-09-22T22:56:04.793 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.50520.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.50520.log.gz 2023-09-22T22:56:04.793 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.72263.log 2023-09-22T22:56:04.793 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.59865.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.59865.log.gz 2023-09-22T22:56:04.793 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.94623.log 2023-09-22T22:56:04.793 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.72263.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.72263.log.gz 2023-09-22T22:56:04.793 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.69596.log 2023-09-22T22:56:04.794 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.94623.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.94623.log.gz 2023-09-22T22:56:04.794 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.82927.log 2023-09-22T22:56:04.794 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.69596.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.69596.log.gz 2023-09-22T22:56:04.794 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.62737.log 2023-09-22T22:56:04.794 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.82927.log: 68.0% -- replaced with /var/log/ceph/c1-client.0.82927.log.gz 2023-09-22T22:56:04.794 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.94967.log 2023-09-22T22:56:04.794 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.62737.log: 45.5% -- replaced with /var/log/ceph/c1-client.0.62737.log.gz 2023-09-22T22:56:04.794 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.93933.log 2023-09-22T22:56:04.795 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.94967.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.94967.log.gz 2023-09-22T22:56:04.795 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.90201.log 2023-09-22T22:56:04.795 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.93933.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.93933.log.gz 2023-09-22T22:56:04.795 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.56958.log 2023-09-22T22:56:04.795 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.90201.log: 45.5% -- replaced with /var/log/ceph/c1-client.0.90201.log.gz 2023-09-22T22:56:04.795 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.43615.log 2023-09-22T22:56:04.795 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.56958.log: 45.5% -- replaced with /var/log/ceph/c1-client.0.56958.log.gz 2023-09-22T22:56:04.796 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.57130.log 2023-09-22T22:56:04.796 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.43615.log: 45.5% -- replaced with /var/log/ceph/c1-client.0.43615.log.gz 2023-09-22T22:56:04.796 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.54057.log 2023-09-22T22:56:04.796 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.57130.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.57130.log.gz 2023-09-22T22:56:04.796 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.79222.log 2023-09-22T22:56:04.796 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.54057.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.54057.log.gz 2023-09-22T22:56:04.796 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.91408.log 2023-09-22T22:56:04.797 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.79222.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.79222.log.gz 2023-09-22T22:56:04.797 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.43529.log 2023-09-22T22:56:04.797 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.91408.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.91408.log.gz 2023-09-22T22:56:04.797 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.49669.log 2023-09-22T22:56:04.797 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.43529.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.43529.log.gz 2023-09-22T22:56:04.797 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.49250.log 2023-09-22T22:56:04.797 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.49669.log: 46.3% -- replaced with /var/log/ceph/c1-client.0.49669.log.gz 2023-09-22T22:56:04.797 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.43443.log 2023-09-22T22:56:04.798 INFO:teuthology.orchestra.run.smithi160.stderr: 94.8% -- replaced with /var/log/ceph/ops-log-c1-client.0.log.gz 2023-09-22T22:56:04.798 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.49250.log: 68.9% -- replaced with /var/log/ceph/c1-client.0.49250.log.gz 2023-09-22T22:56:04.798 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.93073.log 2023-09-22T22:56:04.798 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.86424.log 2023-09-22T22:56:04.798 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.43443.log: /var/log/ceph/c1-client.0.93073.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.43443.log.gz 2023-09-22T22:56:04.798 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.93073.log.gz 2023-09-22T22:56:04.798 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.41895.log 2023-09-22T22:56:04.799 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.48472.log 2023-09-22T22:56:04.799 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.86424.log: /var/log/ceph/c1-client.0.41895.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.86424.log.gz 2023-09-22T22:56:04.799 INFO:teuthology.orchestra.run.smithi160.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.41895.log.gz 2023-09-22T22:56:04.799 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.75378.log 2023-09-22T22:56:04.799 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.46521.log 2023-09-22T22:56:04.799 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.48472.log: /var/log/ceph/c1-client.0.75378.log: 0.0% 0.0% -- replaced with /var/log/ceph/c1-client.0.48472.log.gz 2023-09-22T22:56:04.799 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.75378.log.gz 2023-09-22T22:56:04.800 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.70351.log 2023-09-22T22:56:04.800 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.62651.log 2023-09-22T22:56:04.800 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.46521.log: /var/log/ceph/c1-client.0.70351.log: 81.1% -- replaced with /var/log/ceph/c1-client.0.46521.log.gz 2023-09-22T22:56:04.800 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.70351.log.gz 2023-09-22T22:56:04.800 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.51111.log 2023-09-22T22:56:04.800 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.94365.log 2023-09-22T22:56:04.800 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.62651.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.62651.log.gz 2023-09-22T22:56:04.801 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.51111.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.51111.log.gzgzip -5 --verbose -- /var/log/ceph/c1-client.0.39070.log 2023-09-22T22:56:04.801 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-09-22T22:56:04.801 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.94365.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.94365.log.gz 2023-09-22T22:56:04.801 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.81508.log 2023-09-22T22:56:04.801 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.90112.log 2023-09-22T22:56:04.801 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.39070.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.39070.log.gz 2023-09-22T22:56:04.801 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.81508.log: 47.4%gzip -5 --verbose -- /var/log/ceph/c1-client.0.67985.log 2023-09-22T22:56:04.802 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.81508.log.gz 2023-09-22T22:56:04.802 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.90112.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.90112.log.gz 2023-09-22T22:56:04.802 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.83688.log 2023-09-22T22:56:04.802 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.68070.log 2023-09-22T22:56:04.802 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.67985.log: /var/log/ceph/c1-client.0.83688.log: 56.7% -- replaced with /var/log/ceph/c1-client.0.67985.log.gz 2023-09-22T22:56:04.802 INFO:teuthology.orchestra.run.smithi160.stderr: 43.0% -- replaced with /var/log/ceph/c1-client.0.83688.log.gz 2023-09-22T22:56:04.802 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.48212.log 2023-09-22T22:56:04.803 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.55191.log 2023-09-22T22:56:04.803 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.68070.log: /var/log/ceph/c1-client.0.48212.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.68070.log.gz 2023-09-22T22:56:04.803 INFO:teuthology.orchestra.run.smithi160.stderr: 44.8% -- replaced with /var/log/ceph/c1-client.0.48212.log.gz 2023-09-22T22:56:04.803 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.59112.log 2023-09-22T22:56:04.803 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.68601.log 2023-09-22T22:56:04.803 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.55191.log: /var/log/ceph/c1-client.0.59112.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.55191.log.gz 2023-09-22T22:56:04.804 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.59112.log.gz 2023-09-22T22:56:04.804 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.81928.log 2023-09-22T22:56:04.804 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.48727.log 2023-09-22T22:56:04.804 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.68601.log: /var/log/ceph/c1-client.0.81928.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.68601.log.gz 2023-09-22T22:56:04.804 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.81928.log.gz 2023-09-22T22:56:04.804 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.93503.log 2023-09-22T22:56:04.804 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.83180.log 2023-09-22T22:56:04.804 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.48727.log: 88.2% -- replaced with /var/log/ceph/c1-client.0.48727.log.gz 2023-09-22T22:56:04.805 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.93503.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.93503.log.gzgzip -5 --verbose -- /var/log/ceph/c1-client.0.73978.log 2023-09-22T22:56:04.805 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-09-22T22:56:04.805 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.83180.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.83180.log.gz 2023-09-22T22:56:04.805 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.64854.log 2023-09-22T22:56:04.805 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.41640.log 2023-09-22T22:56:04.805 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.73978.log: 87.2% -- replaced with /var/log/ceph/c1-client.0.73978.log.gz 2023-09-22T22:56:04.805 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.64854.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.64854.log.gz 2023-09-22T22:56:04.806 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.45836.log 2023-09-22T22:56:04.806 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.41640.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.41640.log.gz 2023-09-22T22:56:04.806 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.50347.log 2023-09-22T22:56:04.806 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.44138.log 2023-09-22T22:56:04.806 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.45836.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.45836.log.gz 2023-09-22T22:56:04.806 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.50347.log: 47.4%gzip -5 --verbose -- /var/log/ceph/c1-client.0.60035.log 2023-09-22T22:56:04.806 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.50347.log.gz 2023-09-22T22:56:04.807 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.44138.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.44138.log.gz 2023-09-22T22:56:04.807 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.61723.log 2023-09-22T22:56:04.807 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.90546.log 2023-09-22T22:56:04.807 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.60035.log: /var/log/ceph/c1-client.0.61723.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.60035.log.gz 2023-09-22T22:56:04.807 INFO:teuthology.orchestra.run.smithi160.stderr: 44.8% -- replaced with /var/log/ceph/c1-client.0.61723.log.gz 2023-09-22T22:56:04.807 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.73600.log 2023-09-22T22:56:04.807 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.87267.log 2023-09-22T22:56:04.807 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.90546.log: /var/log/ceph/c1-client.0.73600.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.90546.log.gz 93.6% -- replaced with /var/log/ceph/c1-client.0.73600.log.gz 2023-09-22T22:56:04.808 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-09-22T22:56:04.808 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.78448.log 2023-09-22T22:56:04.808 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.81088.log 2023-09-22T22:56:04.808 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.87267.log: /var/log/ceph/c1-client.0.78448.log: 0.0% 0.0% -- replaced with /var/log/ceph/c1-client.0.87267.log.gz 2023-09-22T22:56:04.808 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/c1-client.0.78448.log.gz 2023-09-22T22:56:04.808 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.33380.log 2023-09-22T22:56:04.808 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.57474.log 2023-09-22T22:56:04.809 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.81088.log: /var/log/ceph/c1-client.admin.33380.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.81088.log.gz 2023-09-22T22:56:04.809 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.33380.log.gz 2023-09-22T22:56:04.809 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.47112.log 2023-09-22T22:56:04.809 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.76498.log 2023-09-22T22:56:04.809 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.57474.log: /var/log/ceph/c1-client.0.47112.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.57474.log.gz 2023-09-22T22:56:04.809 INFO:teuthology.orchestra.run.smithi160.stderr: 68.8% -- replaced with /var/log/ceph/c1-client.0.47112.log.gz 2023-09-22T22:56:04.809 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.89943.log 2023-09-22T22:56:04.810 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.44052.log 2023-09-22T22:56:04.810 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.76498.log: /var/log/ceph/c1-client.0.89943.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.76498.log.gz 2023-09-22T22:56:04.810 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.89943.log.gz 2023-09-22T22:56:04.810 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.46177.log 2023-09-22T22:56:04.810 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.75458.log 2023-09-22T22:56:04.810 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.46177.log: /var/log/ceph/c1-client.0.44052.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.44052.log.gz 2023-09-22T22:56:04.810 INFO:teuthology.orchestra.run.smithi160.stderr: 82.1% -- replaced with /var/log/ceph/c1-client.0.46177.log.gz 2023-09-22T22:56:04.811 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.92356.log 2023-09-22T22:56:04.811 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-mon.a.log 2023-09-22T22:56:04.811 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.75458.log: /var/log/ceph/c1-client.0.92356.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.75458.log.gz 2023-09-22T22:56:04.811 INFO:teuthology.orchestra.run.smithi160.stderr: 45.5% -- replaced with /var/log/ceph/c1-client.0.92356.log.gz 2023-09-22T22:56:04.811 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.94107.log 2023-09-22T22:56:04.811 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.71019.log 2023-09-22T22:56:04.811 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.94107.log: 45.5% -- replaced with /var/log/ceph/c1-client.0.94107.log.gz 2023-09-22T22:56:04.811 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.42325.log 2023-09-22T22:56:04.812 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.71019.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.71019.log.gz 2023-09-22T22:56:04.812 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.85187.log 2023-09-22T22:56:04.812 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.42325.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.42325.log.gz 2023-09-22T22:56:04.812 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.37202.log 2023-09-22T22:56:04.812 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.85187.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.85187.log.gz 2023-09-22T22:56:04.812 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.92614.log 2023-09-22T22:56:04.812 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.37202.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.37202.log.gz 2023-09-22T22:56:04.813 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.95399.log 2023-09-22T22:56:04.813 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.92614.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.92614.log.gz 2023-09-22T22:56:04.813 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.50092.log 2023-09-22T22:56:04.813 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.95399.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.95399.log.gz 2023-09-22T22:56:04.813 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.74498.log 2023-09-22T22:56:04.813 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.50092.log: 45.5% -- replaced with /var/log/ceph/c1-client.0.50092.log.gz 2023-09-22T22:56:04.813 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.92098.log 2023-09-22T22:56:04.813 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.74498.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.74498.log.gz 2023-09-22T22:56:04.814 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.54388.log 2023-09-22T22:56:04.814 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.92098.log: 45.5% -- replaced with /var/log/ceph/c1-client.0.92098.log.gz 2023-09-22T22:56:04.814 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.62989.log 2023-09-22T22:56:04.814 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.54388.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.54388.log.gz 2023-09-22T22:56:04.814 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.51951.log 2023-09-22T22:56:04.814 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.62989.log: 69.3% -- replaced with /var/log/ceph/c1-client.0.62989.log.gz 2023-09-22T22:56:04.814 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36771.log 2023-09-22T22:56:04.815 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.51951.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.51951.log.gz 2023-09-22T22:56:04.815 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1.audit.log 2023-09-22T22:56:04.815 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.36771.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36771.log.gz 2023-09-22T22:56:04.815 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.68514.log 2023-09-22T22:56:04.815 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1.audit.log: 89.4% -- replaced with /var/log/ceph/c1.audit.log.gz 2023-09-22T22:56:04.815 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.74250.log 2023-09-22T22:56:04.815 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.68514.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.68514.log.gz 2023-09-22T22:56:04.816 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.72960.log 2023-09-22T22:56:04.816 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.74250.log: 84.9% -- replaced with /var/log/ceph/c1-client.0.74250.log.gz 2023-09-22T22:56:04.816 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.69935.log 2023-09-22T22:56:04.816 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.72960.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.72960.log.gz 2023-09-22T22:56:04.816 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.65191.log 2023-09-22T22:56:04.816 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.69935.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.69935.log.gz 2023-09-22T22:56:04.816 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.57388.log 2023-09-22T22:56:04.817 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.65191.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.65191.log.gz 2023-09-22T22:56:04.817 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.78704.log 2023-09-22T22:56:04.817 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.57388.log: 49.4% -- replaced with /var/log/ceph/c1-client.0.57388.log.gz 2023-09-22T22:56:04.817 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.85437.log 2023-09-22T22:56:04.817 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.78704.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.78704.log.gz 2023-09-22T22:56:04.817 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.68687.log 2023-09-22T22:56:04.817 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.85437.log: 69.6% -- replaced with /var/log/ceph/c1-client.0.85437.log.gz 2023-09-22T22:56:04.818 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.65897.log 2023-09-22T22:56:04.818 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.68687.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.68687.log.gz 2023-09-22T22:56:04.818 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.80169.log 2023-09-22T22:56:04.818 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.65897.log: 66.6% -- replaced with /var/log/ceph/c1-client.0.65897.log.gz 2023-09-22T22:56:04.818 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.57905.log 2023-09-22T22:56:04.818 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.80169.log: 43.0% -- replaced with /var/log/ceph/c1-client.0.80169.log.gz 2023-09-22T22:56:04.818 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.67815.log 2023-09-22T22:56:04.819 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.57905.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.57905.log.gz 2023-09-22T22:56:04.819 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.88608.log 2023-09-22T22:56:04.819 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.67815.log: 48.7% -- replaced with /var/log/ceph/c1-client.0.67815.log.gz 2023-09-22T22:56:04.819 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.69765.log 2023-09-22T22:56:04.819 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.88608.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.88608.log.gz 2023-09-22T22:56:04.819 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.37228.log 2023-09-22T22:56:04.819 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.69765.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.69765.log.gz 2023-09-22T22:56:04.819 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.63071.log 2023-09-22T22:56:04.820 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.37228.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.37228.log.gz 2023-09-22T22:56:04.820 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.76178.log 2023-09-22T22:56:04.820 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.63071.log: 71.9% -- replaced with /var/log/ceph/c1-client.0.63071.log.gz 2023-09-22T22:56:04.820 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.43873.log 2023-09-22T22:56:04.820 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.76178.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.76178.log.gz 2023-09-22T22:56:04.820 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.58507.log 2023-09-22T22:56:04.820 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.43873.log: 45.5% -- replaced with /var/log/ceph/c1-client.0.43873.log.gz 2023-09-22T22:56:04.821 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.66152.log 2023-09-22T22:56:04.821 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.58507.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.58507.log.gz 2023-09-22T22:56:04.821 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.69511.log 2023-09-22T22:56:04.821 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.66152.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.66152.log.gz 2023-09-22T22:56:04.821 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.44814.log 2023-09-22T22:56:04.821 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.69511.log: 58.8% -- replaced with /var/log/ceph/c1-client.0.69511.log.gz 2023-09-22T22:56:04.821 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.76338.log 2023-09-22T22:56:04.822 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.44814.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.44814.log.gz 2023-09-22T22:56:04.822 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.43357.log 2023-09-22T22:56:04.822 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.76338.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.76338.log.gz 2023-09-22T22:56:04.822 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.50688.log 2023-09-22T22:56:04.822 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.43357.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.43357.log.gz 2023-09-22T22:56:04.822 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.67901.log 2023-09-22T22:56:04.822 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.50688.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.50688.log.gz 2023-09-22T22:56:04.823 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.54892.log 2023-09-22T22:56:04.823 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.67901.log: 70.1% -- replaced with /var/log/ceph/c1-client.0.67901.log.gz 2023-09-22T22:56:04.823 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36957.log 2023-09-22T22:56:04.823 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.54892.log: 19.1% -- replaced with /var/log/ceph/c1-client.0.54892.log.gz 2023-09-22T22:56:04.823 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.83012.log 2023-09-22T22:56:04.823 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.36957.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36957.log.gz 2023-09-22T22:56:04.823 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.67000.log 2023-09-22T22:56:04.824 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.83012.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.83012.log.gz 2023-09-22T22:56:04.824 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.78618.log 2023-09-22T22:56:04.824 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.67000.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.67000.log.gz 2023-09-22T22:56:04.824 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.70852.log 2023-09-22T22:56:04.824 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.78618.log: 45.5% -- replaced with /var/log/ceph/c1-client.0.78618.log.gz 2023-09-22T22:56:04.824 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.37278.log 2023-09-22T22:56:04.824 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.70852.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.70852.log.gz 2023-09-22T22:56:04.825 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.66831.log 2023-09-22T22:56:04.825 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.admin.37278.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.37278.log.gz 2023-09-22T22:56:04.825 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.76258.log 2023-09-22T22:56:04.825 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.66831.log: 64.5% -- replaced with /var/log/ceph/c1-client.0.66831.log.gz 2023-09-22T22:56:04.825 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.68340.log 2023-09-22T22:56:04.825 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.76258.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.76258.log.gz 2023-09-22T22:56:04.825 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.79912.log 2023-09-22T22:56:04.826 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.68340.log: 84.8% -- replaced with /var/log/ceph/c1-client.0.68340.log.gz 2023-09-22T22:56:04.826 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.57991.log 2023-09-22T22:56:04.826 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.79912.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.79912.log.gz 2023-09-22T22:56:04.826 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.73517.log 2023-09-22T22:56:04.826 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.57991.log: 45.5% -- replaced with /var/log/ceph/c1-client.0.57991.log.gz 2023-09-22T22:56:04.826 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.63843.log 2023-09-22T22:56:04.826 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.73517.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.73517.log.gz 2023-09-22T22:56:04.827 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.45579.log 2023-09-22T22:56:04.827 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.63843.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.63843.log.gz 2023-09-22T22:56:04.827 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.49165.log 2023-09-22T22:56:04.827 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.45579.log: 81.2% -- replaced with /var/log/ceph/c1-client.0.45579.log.gz 2023-09-22T22:56:04.827 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.51282.log 2023-09-22T22:56:04.827 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.49165.log: 46.7% -- replaced with /var/log/ceph/c1-client.0.49165.log.gz 2023-09-22T22:56:04.827 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.71185.log 2023-09-22T22:56:04.828 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.51282.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.51282.log.gz 2023-09-22T22:56:04.828 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/c1-client.0.71185.log: 96.4% -- replaced with /var/log/ceph/c1-client.0.71185.log.gz 2023-09-22T22:56:04.942 INFO:teuthology.orchestra.run.smithi204.stderr: 89.6% -- replaced with /var/log/ceph/c2.audit.log.gz 2023-09-22T22:56:04.947 INFO:teuthology.orchestra.run.smithi204.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.41193.log.gz 2023-09-22T22:56:04.948 INFO:teuthology.orchestra.run.smithi204.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.41279.log.gz 2023-09-22T22:56:05.109 INFO:teuthology.orchestra.run.smithi160.stderr: 91.8% -- replaced with /var/log/ceph/c1-mgr.x.log.gz 2023-09-22T22:56:07.935 INFO:teuthology.orchestra.run.smithi160.stderr: 92.5% -- replaced with /var/log/ceph/c1-mon.a.log.gz 2023-09-22T22:56:09.090 INFO:teuthology.orchestra.run.smithi160.stderr: 93.9% -- replaced with /var/log/ceph/rgw.c1.client.1.log.gz 2023-09-22T22:56:18.824 INFO:teuthology.orchestra.run.smithi204.stderr: 91.8% -- replaced with /var/log/ceph/c2-mgr.x.log.gz 2023-09-22T22:56:27.733 INFO:teuthology.orchestra.run.smithi160.stderr: 94.1% -- replaced with /var/log/ceph/rgw.c1.client.0.log.gz 2023-09-22T22:56:34.602 INFO:teuthology.orchestra.run.smithi160.stderr: 94.1% -- replaced with /var/log/ceph/c1-osd.1.log.gz 2023-09-22T22:56:35.046 INFO:teuthology.orchestra.run.smithi160.stderr: 94.0% -- replaced with /var/log/ceph/c1-osd.2.log.gz 2023-09-22T22:56:35.396 INFO:teuthology.orchestra.run.smithi160.stderr: 94.3% -- replaced with /var/log/ceph/c1-osd.0.log.gz 2023-09-22T22:56:35.397 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-09-22T22:56:35.397 INFO:teuthology.orchestra.run.smithi160.stderr:real 0m30.985s 2023-09-22T22:56:35.398 INFO:teuthology.orchestra.run.smithi160.stderr:user 1m59.364s 2023-09-22T22:56:35.398 INFO:teuthology.orchestra.run.smithi160.stderr:sys 0m4.570s 2023-09-22T22:56:40.478 INFO:teuthology.orchestra.run.smithi204.stderr: 92.1% -- replaced with /var/log/ceph/c2-mon.a.log.gz 2023-09-22T22:59:41.535 INFO:teuthology.orchestra.run.smithi204.stderr: 94.5% -- replaced with /var/log/ceph/rgw.c2.client.0.log.gz 2023-09-22T22:59:47.860 INFO:teuthology.orchestra.run.smithi204.stderr: 94.5% -- replaced with /var/log/ceph/rgw.c2.client.1.log.gz 2023-09-22T23:35:26.609 INFO:teuthology.orchestra.run.smithi204.stderr: 93.5% -- replaced with /var/log/ceph/c2-osd.1.log.gz 2023-09-22T23:36:14.338 INFO:teuthology.orchestra.run.smithi204.stderr: 93.4% -- replaced with /var/log/ceph/c2-osd.2.log.gz 2023-09-22T23:38:04.777 INFO:teuthology.orchestra.run.smithi204.stderr: 93.4% -- replaced with /var/log/ceph/c2-osd.0.log.gz 2023-09-22T23:38:04.779 INFO:teuthology.orchestra.run.smithi204.stderr: 2023-09-22T23:38:04.779 INFO:teuthology.orchestra.run.smithi204.stderr:real 42m0.342s 2023-09-22T23:38:04.780 INFO:teuthology.orchestra.run.smithi204.stderr:user 37m33.456s 2023-09-22T23:38:04.780 INFO:teuthology.orchestra.run.smithi204.stderr:sys 1m48.576s 2023-09-22T23:38:04.781 INFO:tasks.ceph:Archiving logs... 2023-09-22T23:38:04.781 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/log/ceph to /home/teuthworker/archive/mkogan-2023-09-22_17:38:01-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7401166/remote/smithi160/log 2023-09-22T23:38:04.782 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-22T23:38:18.704 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/log/ceph to /home/teuthworker/archive/mkogan-2023-09-22_17:38:01-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7401166/remote/smithi204/log 2023-09-22T23:38:18.705 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-22T23:39:54.639 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw.py", line 473, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw.py", line 300, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw.py", line 242, in start_rgw rgwadmin(ctx, client, cmd=['gc', 'process', '--include-all'], check_status=True) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/util/rgw.py", line 34, in rgwadmin proc = remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 gc process --include-all' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph_manager.py", line 2742, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph_manager.py", line 2574, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph_manager.py", line 2340, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph_manager.py", line 1595, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/ceph_manager.py", line 1586, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json' 2023-09-22T23:39:54.646 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-22T23:39:54.655 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw.py", line 473, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw.py", line 300, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw.py", line 242, in start_rgw rgwadmin(ctx, client, cmd=['gc', 'process', '--include-all'], check_status=True) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/util/rgw.py", line 34, in rgwadmin proc = remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 gc process --include-all' 2023-09-22T23:39:54.668 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-22T23:39:54.668 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-22T23:39:54.691 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-22T23:39:54.822 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-22T23:39:54.832 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-22T23:39:54.833 DEBUG:teuthology.orchestra.run.smithi160:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-22T23:39:54.835 DEBUG:teuthology.orchestra.run.smithi204:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-22T23:39:54.850 INFO:teuthology.orchestra.run.smithi160.stdout: remote refid st t when poll reach delay offset jitter 2023-09-22T23:39:54.850 INFO:teuthology.orchestra.run.smithi160.stdout:============================================================================== 2023-09-22T23:39:54.850 INFO:teuthology.orchestra.run.smithi160.stdout:*hv01.front.sepi 44.4.53.2 2 u 292 512 377 0.174 -4.093 1.282 2023-09-22T23:39:54.850 INFO:teuthology.orchestra.run.smithi160.stdout:+hv02.front.sepi 63.231.80.2 3 u 360 512 377 0.156 -2.764 0.683 2023-09-22T23:39:54.850 INFO:teuthology.orchestra.run.smithi160.stdout:+hv03.front.sepi 74.6.168.72 3 u 377 512 377 0.223 -3.513 0.859 2023-09-22T23:39:54.851 INFO:teuthology.orchestra.run.smithi160.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-09-22T23:39:54.886 INFO:teuthology.orchestra.run.smithi204.stdout: remote refid st t when poll reach delay offset jitter 2023-09-22T23:39:54.887 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================== 2023-09-22T23:39:54.887 INFO:teuthology.orchestra.run.smithi204.stdout:*hv01.front.sepi 44.4.53.2 2 u 483 512 377 0.256 -3.038 0.664 2023-09-22T23:39:54.887 INFO:teuthology.orchestra.run.smithi204.stdout:+hv02.front.sepi 63.231.80.2 3 u 192 512 377 0.192 -0.993 1.193 2023-09-22T23:39:54.887 INFO:teuthology.orchestra.run.smithi204.stdout:+hv03.front.sepi 74.6.168.72 3 u 474 512 377 0.193 -2.804 0.873 2023-09-22T23:39:54.887 INFO:teuthology.orchestra.run.smithi204.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-09-22T23:39:54.888 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-22T23:39:54.898 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-22T23:39:54.898 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-22T23:39:54.909 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-22T23:39:54.919 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-22T23:39:54.928 INFO:teuthology.task.internal:Duration was 10173.758372 seconds 2023-09-22T23:39:54.929 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-22T23:39:54.940 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-22T23:39:54.940 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-22T23:39:54.943 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-22T23:39:55.206 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-22T23:39:55.206 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi160.front.sepia.ceph.com 2023-09-22T23:39:55.206 DEBUG:teuthology.orchestra.run.smithi160:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-22T23:39:55.217 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi204.front.sepia.ceph.com 2023-09-22T23:39:55.217 DEBUG:teuthology.orchestra.run.smithi204:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-22T23:39:55.266 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-22T23:39:55.266 DEBUG:teuthology.orchestra.run.smithi160:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-22T23:39:55.268 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-22T23:39:55.351 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-22T23:39:55.351 DEBUG:teuthology.orchestra.run.smithi160:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-22T23:39:55.353 DEBUG:teuthology.orchestra.run.smithi204:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-22T23:39:55.645 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-22T23:39:55.656 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-22T23:39:55.656 DEBUG:teuthology.orchestra.run.smithi160:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-22T23:39:55.671 DEBUG:teuthology.orchestra.run.smithi204:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-22T23:39:55.717 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-22T23:39:55.727 DEBUG:teuthology.orchestra.run.smithi160:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-22T23:39:55.730 DEBUG:teuthology.orchestra.run.smithi204:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-22T23:39:55.740 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = core 2023-09-22T23:39:55.788 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = core 2023-09-22T23:39:55.811 DEBUG:teuthology.orchestra.run.smithi160:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-22T23:39:55.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T23:39:55.815 DEBUG:teuthology.orchestra.run.smithi204:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-22T23:39:55.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T23:39:55.861 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-22T23:39:55.872 INFO:teuthology.task.internal:Transferring archived files... 2023-09-22T23:39:55.873 DEBUG:teuthology.misc:Transferring archived files from smithi160:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mkogan-2023-09-22_17:38:01-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7401166/remote/smithi160 2023-09-22T23:39:55.873 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-22T23:39:55.891 DEBUG:teuthology.misc:Transferring archived files from smithi204:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mkogan-2023-09-22_17:38:01-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7401166/remote/smithi204 2023-09-22T23:39:55.891 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-22T23:39:55.924 INFO:teuthology.task.internal:Removing archive directory... 2023-09-22T23:39:55.924 DEBUG:teuthology.orchestra.run.smithi160:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-22T23:39:55.930 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-22T23:39:55.965 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-22T23:39:55.977 INFO:teuthology.task.internal:Not uploading archives. 2023-09-22T23:39:55.977 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-22T23:39:55.988 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-22T23:39:55.988 DEBUG:teuthology.orchestra.run.smithi160:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-22T23:39:55.990 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-22T23:39:55.993 INFO:teuthology.orchestra.run.smithi160.stdout: 655472 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 22 23:39 /home/ubuntu/cephtest 2023-09-22T23:39:55.994 INFO:teuthology.orchestra.run.smithi160.stdout: 673687 0 srwxr-xr-x 1 root root 0 Sep 22 22:11 /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2023-09-22T23:39:55.994 INFO:teuthology.orchestra.run.smithi160.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-09-22T23:39:56.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T23:39:56.010 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw.py", line 473, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw.py", line 300, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/rgw.py", line 242, in start_rgw rgwadmin(ctx, client, cmd=['gc', 'process', '--include-all'], check_status=True) File "/home/teuthworker/src/github.com_mkogan1_ceph_691e03976a4d019c58bad99761055f3266556055/qa/tasks/util/rgw.py", line 34, in rgwadmin proc = remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 gc process --include-all' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-09-22T23:39:56.022 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-22T23:39:56.032 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-22T23:39:56.060 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi160 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 gc process --include-all' 2023-09-22T23:39:56.060 INFO:teuthology.nuke:Checking targets against current locks 2023-09-22T23:39:56.067 INFO:teuthology.orchestra.run.smithi204.stdout: 655472 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 22 23:39 /home/ubuntu/cephtest 2023-09-22T23:39:56.067 INFO:teuthology.orchestra.run.smithi204.stdout: 673687 0 srwxr-xr-x 1 root root 0 Sep 22 21:02 /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2023-09-22T23:39:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: 673686 0 srwxr-xr-x 1 root root 0 Sep 22 21:02 /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2023-09-22T23:39:56.068 INFO:teuthology.orchestra.run.smithi204.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-09-22T23:39:56.092 DEBUG:teuthology.nuke:shortname: smithi160 2023-09-22T23:39:56.092 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-22T23:39:56.115 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi160.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-09-22_17:38:01-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7401166', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-22 20:41:35.308549', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJc+XzPRS2TrfwsLPPiLo/tNCa2RGt6AcCR/nSya29UpF+GTgm+TxeVeHmZSk/cnLRECsVqSxIMEtf26RoWCMWI='} 2023-09-22T23:39:56.118 DEBUG:teuthology.nuke:shortname: smithi204 2023-09-22T23:39:56.119 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-22T23:39:56.139 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-09-22_17:38:01-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7401166', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-22 20:41:35.306891', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEqnq9KNsYblkwRNcJhKaEyCeQLeszT9O0Xr2v9A/EES2mM7cdfy9fws30xx5mDF1ikl+kZ3V+/mYuCmhT9Lpvw='} 2023-09-22T23:39:56.181 INFO:teuthology.orchestra.console.smithi204:Power off 2023-09-22T23:39:56.182 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-22T23:39:56.196 INFO:teuthology.orchestra.console.smithi160:Power off 2023-09-22T23:39:56.197 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-22T23:39:56.207 DEBUG:teuthology.orchestra.console.smithi204:power off output: Chassis Power Control: Down/Off 2023-09-22T23:39:56.207 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-22T23:39:56.218 DEBUG:teuthology.orchestra.console.smithi160:power off output: Chassis Power Control: Down/Off 2023-09-22T23:39:56.219 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-22T23:39:56.229 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2023-09-22T23:39:56.241 DEBUG:teuthology.orchestra.console.smithi160:check power output: Chassis Power is on 2023-09-22T23:40:00.230 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-22T23:40:00.242 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-22T23:40:00.342 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2023-09-22T23:40:00.353 DEBUG:teuthology.orchestra.console.smithi160:check power output: Chassis Power is on 2023-09-22T23:40:04.346 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-22T23:40:04.356 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-22T23:40:04.458 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2023-09-22T23:40:04.466 DEBUG:teuthology.orchestra.console.smithi160:check power output: Chassis Power is off 2023-09-22T23:40:04.567 INFO:teuthology.orchestra.console.smithi160:Power off completed 2023-09-22T23:40:08.463 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-22T23:40:08.575 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is off 2023-09-22T23:40:08.675 INFO:teuthology.orchestra.console.smithi204:Power off completed 2023-09-22T23:40:08.780 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi} duration: 10173.758372306824 failure_reason: rgw multisite test failures flavor: default owner: scheduled_mkogan@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=eedbdffa67a84becb428ca0f94d6ebce status: fail success: false 2023-09-22T23:40:08.780 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-22T23:40:08.855 INFO:teuthology.run:FAIL